Robocup

INTRODUCTION

1 1. INTRODUCTION Robocup is short for Robots Soccer Cup is an international scientific initiative invented in 1993 with the idea of robots playing soccer. After several years of preparation, the first event was held in 1997 for both real and simulation robots with the participation of over 40 teams. So far, there have been more than 70 teams coming from more than 25 countries competing in this annually international cup. The aim of Robocup is to promote robotics and artificial intelligence research under challenging and adverse conditions. FIGURE 1 ROBOCUP LOGO 1 Robocup provides a standard problem for testing, evaluating theories and algorithms by offering challenging problem in which various of researching areas have to deal with such as strategy acquisition, machine learning, real-time recognition, planning, and reasoning, reasoning and action in dynamics environment, multi-agent systems, context recognition, vision, strategic decision-making. One the most challenging issues in Robocup is the integration those technologies into one system for achieving a specific goal 1 . Variety of difficult, but interesting problems make Robocup attractive to researchers. Many academic institutions have been taking Robocup a vehicle to advance the state of the art of intelligent robots All of our team members are interested in Robocup, we are on the way to build a Robocup simulation team. The team is based on agent2D – the base code of HELIOS team from Japan. Agent2D has been used for qualifying teams which want to compete in annual Robocup competition event. From the base code, by improving both offensive and defensive strategies we have made significant enhancement in team performance. For 1 www.robocup.org 2 offensive strategy, we did improve low both level skills such as through pass, scoring skills and high level skills. For defensive strategy, we made change on the way fullback players marking. We are now can defeat the base team. 7/11 teams in 2011 world final event cannot win over base team for in qualification round. This paper consists of six chapters. The first chapter is dedicated for the introduction about Robocup and our work. The second chapter will talk about a Robocup 2D simulation, one of the oldest sub-leagues in Robocup. In chapter 3, we will review several work and research on Robocup 2D Simulation. In the fourth chapter, the improvement we have made on the base code will be discussed in detailed. The next chapter is the evaluation of our work. And in the last chapter, we will discuss our work and the future improvement for our team. 2. ROBOCUP 2D SIMULATION In this chapter, we will introduce about Robocup Soccer 2D simulation 2 , its architecture, the game environment as well as the players or the agents 3 . 2.1. Architecture The architecture of Robocup 2D simulation consists of four main parts. The first one is server which is in charge of controlling the game, updating information to players. The second one is agents which are independent connect to server to receiver information about the game. Agents are independent i.e. no share memory and players do not permission to access to other players to get information. The third one is monitor that is used to visualize the game for audience to see what is happening in the game and the last part is coach. Coach is simulation of coach manager in real soccer game who can make substitution, change strategy in game 2.2. Environment In 2D simulation league, server simulates the real world soccer environment in 2 dimensions i.e. no highball, the ball always runs on ground. Agents play on a plane with 2D Cartesian coordinate which represents the soccer field. Each position in the soccer field is characterized by two real number (x,y). x belong to the range [-54,54], y belongs 3 to the range [-32,32]. Timing system is different from the real world. A match kickoff fulltime is 6000 cycles (approximate 6 minutes), each half lasts for 3000 cycle. The clock stops when the ball is dead, therefore, no extra time in a match. Golden goal policy is applied in case two teams draw in the 2 halves of the game. The environment is noisy because actions are performed might be a little different from intention. For example, a player wants to kick the ball to the position(x,y), the ball may run to the position (x+dx, y+dy) with dx, dy added by server. All noises are simulated in server. Another characteristic of the environment is partly-observed. An agent can only see a specific region depends on the position on where he stands. Besides, an agent sees objects that are far from less exact than the nearby objects. 2.3. Agents Agents are computer software. They play roles as players on a match. Whenever an agent wants to do an action (for example, move, kick, tackle…) he sends the command to the server. The server updates the game environment and sends back to every agent the current state of the match. Agents are characterized by many parameters such as running speed, acceleration, dash rate, kick margin, stamina…There are total of 18 types of agents, they are different in the value of those above mentioned parameters. Every agent has the below primitive skills: - Turn neck: Turn neck to a direction - Turn body: Turn body to a direction - Kick: Kick the ball to a direction - Tackle: Try to get the ball from opponent From those primitive skills, many more complex and advanced skills are build based on those primitive skills such as: Dribble, direct pass, through pass, shoot. The soccer server supports agents with some built-in models. Sensor model enables agents to hear, to see and to get information about them. Movement model helps them to move on the soccer field. Action Model which give agents abilities to catch the ball (goalie only), to dash and to kick the ball. .3.1 Sensor Model A robocup agent has three different sensors, namely the aural, the visual and the body sensor. 4 2.3.2.1. Aural Sensor Model Aural sensor messages are sent when a client or a coach sends a say command. All messages are received immediately. The server parameters that affect the aural sensor are described in the following table: Figure ABC * Range of communication: A message said by a player is transmitted only to players within audio_cut_dist meters from that player. Messages from the referee can be heard by all players. 2.3.2.2. Visual Sensor Model The visual sensor reports the object currently seen by the player. The information is automatically sent to the player every sense_step, currently 150, milli-seconds. Range of view The server parameters sense_step and visible_angle determine the basic time step between visual information and how many degrees the player’s normal view cone is. The player can also influence the frequency and quality of the information by changing ViewWidth and ViewQuality. To calculate the view_frequency and view_angle of the agent use these equations: view_frequency = sense_step * view_quality_factor * view_width_factor where view_quality_factor is 1 iff ViewQuality is high and 0.5 iff ViewQuality is low; view_width_factor is 2 iff ViewWidth is narrow, 1 iff ViewWidth is normal, and 0.5 iff ViewWidth is wide. view_angle = visible_angle * view_width_factor 5 where view_width_factor is 0.5 iff ViewWidth is narrow, 1 iff ViewWidth is normal, and 2 iff ViewWidth is wide. 2.3.2.3. Visual Sensor Noise Model In order to introduce noise in the visual sensor data the values sent from the server is quantized. This means that player can not know the exact positions of very far objects. Body Sensor Model The body sensor reports the current “physical” status of the player. The information is automaticall sent to the player every sense_body_step, currently 100, milli-seconds. .3.1 Movement Model In each simulation step, movement of each object is calculated as following manner: where , and are respectively position and velocity of the object in timestep t. 2.3.2.1. Movement Noise Model In order to reflect unexpected movements of objects in real world, server adds noise to the movements of objects and parameters of commands 2.3.2.2. Collision Model If at the end of the simulation cycle, two objects overlap, then the objects are moved back until they do not overlap. Then the velocities are multiplied by -0.1. Note that it is possible for the ball to go through a player as long as the ball and the player never overlap at the end of the cycle. .3.1 Action Model There are 7 types of action. They are: 6 Type of Action Description Catch The goalie is the only player with the ability to catch the ball. The goalie can catch the ball if the ball is within the catchable area and the goalie is inside the penalty area Dash This command is used to accelerate the player in direction of its body. Dash take the acceleration power as the parameter. Kick Kick command takes 2 parameters, the kick power and the angle the player kicks the ball to. Once the kick command arrived at the server, the kick will be executed if the ball is kick-able for the player and the player is not offside. The ball is kickable for the player if the distance between the player and the ball is between 0 and kickable_margin. Move Move command is used to place a player directly onto a desired position on the field, it does not work during normal play and is available at the beginning of each half and after the goal has been scored. The second purpose of move command is to move the goalie within the penalty area after the goalie caught the ball. Say Used by players to broadcast messages to other players. Turn Used to change the players body direction. The argument for the turn command is the moment. If the player does not move, the moment is equal to the angle the player will turn. TurnNeck With turn_neck, a player can turn its neck somewhat independently of its body. Turn_neck command can be executed during the same cycle as turn, dash and kick commands. 3. RELATED WORK 3.1. Literature Review Researching on Robocup 2D simulation is now a very active area in despite of its challenges. More and more universities and institutions are conducting research on this branch and strengthen their teams for compete in Robocup 2D. Japanese team HELIOS the champion of 2010 competition is developed by Fukuoka University, National Institute of Advanced Industrial Science and Technology (AIST) and Osaka Prefecture University. The champion of 2011 – WriteEagle 4 team is owned by University of Science and 7 Technology of China. Other academic institutes that are worth mentioned for their high ranks in the competitions and contribution for Robocup Community are University of Freiburg from Germany, University of Oxford from England, and University of Amsterdam from the Netherlands. Amongst the team participated in the annual event competitions, there is some develop their own team from scratch, however, many of those make improvement base on the released resources of others’ team which is also called base team. There are many ways to improve a base team, in the following; we will list some of them that are related to our improvement. They are passing and defensive strategies. Passing plays an important role in Robocup 2D simulation. Whatever offense or defense, passing has great impact on the team performance. For attacking strategies, passing is the main mean of deliver ball to the area near to the opponent goal. Many affiliations attempted for improving the passing strategy for applying in their teams. Guangdong University of Technology tried to combine the advantages of several machine learning methods (Nash-Q 5 , CE-Q and WoLF-PHC 6 ) into a new reinforcement learning 7 . This method allows ball holder to better choosing teammate to pass ball. HELIOS 2010 used game tree searching for a chain of actions including passing, dribbling and shooting. However, they stated in Team Description Paper that its performance is weak because the uncertainty and partly-observed environment of Robocup 2D simulation makes the evaluation and generation chain of actions get low accuracy. Agent2D – HELIOS base code using hand-coded passing generator mixing of direct pass, lead pass and through pass. Beside offense, defense also plays a very important role in a match. If the defense is not strong enough then however strong the offense is, the team could never win. However, defense is not easy to be improved. In fact, while a lot of papers on RoboCup’s robotic 2D soccer simulation have focused on the players’ offensive behavior, there are only a few papers that specifically address a team’s defensive problems. In the past, there were few papers dealt with defensive problem such as a paper of Thomas Gabel, Martin Riedmiller and Florian Trost from Germany discussing about “Defense behavior in Soccer Simulation 2D: The NeuroHassle Approach” 8 . Their method requires multiple trainings against different training opponents to get better result. We have briefly described others related work that we have known. In the next section, we will describe agent2d which we use as a framework for our team and 8 3.2. Helios base code Creating a Robocup 2d team from scratch takes much time and effort since there are many matters that need to be considered, for example: connection between server and agents, agents design… Therefore, many team releases their code without high-level decision making section as a framework in order to help other teams. Those frameworks are often called base code. In our team, we use the Helios team’s base code. Today, Helios is one of the best teams in Robocup 2d simulation league. They are the champion in Robocup 2010 and runner-up in 2009 and 2011. Their base code is called agent2d 9 . Agent2d also comes with a library package which supports the latest soccer server and provides plenty of helpful functions. Although the high-level decision making has been removed almost entirely, they are still pretty strong and in recent year, they have been used as a benchmark for teams who want to participate in Robocup. As in 2011, 7 teams among 20 qualified teams could not defeat the base team and 2 teams lost all matches against agent2d. Beside, agent2d is coded in C++and supports the latest GNU C++ compiler which makes it very convenient to use. Moreover, we have found out that they have a very well-designed structure and they also provides necessary basic skills like pass, dribble, shoot, etc. 3.2.1. Formation In agent2d, they have implemented a very fantastic formation for the team. The formation is defined by some specifics ball position in the field. Agent’s position will be interpolated with Delaunay triangulation from those specific positions. Helios also provides a tool for editing team’s formation 10 . 3.2.2. Low-level skill In this project, we use the name low-level skills to call skills which composed mostly of a series of basic actions of an agent without any hard decision making problems. For example, currently, dribble is called a basic skill because it is as simple as kick the ball toward a position then dash along it. But if we decided to make a more complex dribbling skill which involved trick the opponent, we may classify it as high- level skill. This section will describe some basic information about low-level skills in agent2d. 9 3.2.2.1. Passing skill We have decided to rewrite the passing skill since agent2d’s passing skill doesn’t meet our expectation. Our passing skill will be described in later section. 3.2.2.2. Dribbling skill Dribbling skill is the skill of moving with the ball to a specific position without losing the ball to the opponent. As I have mentioned above, dribbling skill mostly consists of simple kick and dash sequences. They also have a function to verify whether the agent can catch up with the ball before opponents and a function to estimate the number of dash commands will be used. 3.2.2.3. Shooting skill In order to shoot, the Helios base does the following steps: - Step 1: Create and evaluate possible shoots - Step 2: Execute the one with highest score - Step 3: Create and evaluate possible shoots Figure <?>: Shoot model Dividing the opponent goal line to m points. The first point is the goal left column and the last point is the goal right column. The distance between any pair of adjacent points are equal. Only shoot to one of these points will be considered. Using opponent goal keeper body angle, speed and acceleration, check whether the ball can be caught or not. The ball can be caught if it is inside goal keeper CATCHABLE_AREA which is defined by the simulation server. Any shoot which can be caught will be discarded. The rest will be evaluated based on the ball’s travelling distance, speed and distance to the goal keeper. 10 3.2.2.4. Others low level skills Agent2d also provide many helpful skills like: Intercept ball: this is the skill used to receive the ball from a pass. The agent will use a table to keep track of ball positions and use that table to choose the best ball positions to receive. Hold ball: this skill is basically staying in the same position while trying to maintain ball possession. Tackling skill: this is the skill used to take away opponent ball possession. However, there will be chance for the agent to be booked (i.e. receive a card). Since those skills are currently not our interest, we haven’t explored them yet. 3.1.1 High Level strategy Although have been almost entirely removed, the remaining high-level strategy of agent2d is still pretty strong. The remaining is part of their chain-action which as described in their TDP [?] consists of: - Generate a chain of actions from basic action like dribble, pass, shoot - Estimate the final state of the action chain Their performance can be described as simple wing play strategy. The ball is intensively passed to the wing forward and he will try to dribble along the touch line until reaching the corner area. From that position, he will keep dribbling along the goal line and find a way to cross the ball in or score directly. That simple strategy works surprisingly well against agent2d itself. However, after inspecting the movement of their defenders, we found out that their defenders practically do nothing. In fact, they just follow the team formation. 4. IMPROVEMENT In this chapter, we will described in detailed what we have done to achieve better performance in compared with the agent2D. The overall actions and strategies of Agent2D team can be described as below. [...]... tested our approach in this case yet Currently, our approach does not perform well in this situation We will try to improve in near future 6 CONCLUSION In this report, we have introduced about the Robocup and Robocup 2D simulation league which we’re interested in In our team, we use Helios’ base code named agent2d We have built our own strategies and also improved some basic skills in agent2d Now, our... further improving our team performance in order to take part in future Robocup competitions 7 REFERENCES 1 Layered Learning in Multi-AgentSystem[D] Ph.D Thesis Carnegie Mellon University, CMU-CS-98- 187,1998 2 Noda, I., Matsubara, H.: Soccer server and researches on multi-agent systems In Kitano, H., ed.: Proceedings of IROS-96 Workshop on RoboCup (Nov 1996) 1–7 3 MaoChen, EhsanForoughi, FredrikHeintz, ZhanXiangHuang,... pre-defined constants and both is greater than 1.0 16 4.1.3.4 Move: In this skill, agents without ball have to go to a strategic position in order to collaborate with his teammate We use the tools from Helios robocup team called fedit to define the positions for which agents have to get to There are only several main strategic positions are pre-defined in the formation file When the agents in the situation... find_best_position() model 18 Action involved: dribble, shoot Case 2: When our strikers are near the opponent goal: In this case, approaching the goal is very difficult since dribbling pass defenders is a hard task in robocup 2d simulation Currently, our approach can be described as a greedy approach, but we intends to improve it in near future can_shoot(): Helios base’s function, return true if there are chances to... determine whether or not the incoming message is from opponents or teammates Second, if opponents say ceaselessly, agents will be not able to receiving any message due to threshold hearing defined by the Robocup 2D simulation Through pass is generated different from lead pass and direct pass, however, they use the same evaluation method This leads to the domination of direct pass action in the match As... inspected from the source code of Helios base team, we decided to improve it to better our team performance Our approach is mainly focus on the communication between pass taker and pass receiver In the Robocup 2D server version 8.0, there is a useful action that an agent can employ to collaborate with his teammates It is “point to” action, i.e point agent’s arm to a desired position In the task of improving... on RoboCup (Nov 1996) 1–7 3 MaoChen, EhsanForoughi, FredrikHeintz, ZhanXiangHuang, SpirosKapetanakis, KostasKostiadis, JohanKummeneje, ItsukiNoda, OliverObst, PatRiley, TimoSteens, YiWang and XiangYin: RoboCupSoccerServer 4 Aijun Bai, Guanghui Lu, Haochong Zhang and Xiaoping Chen: WrightEagle 2D Soccer Simulation Team Description 2011 5 Hu J,WellmanMP.Nashq-learningfor general-sumstochastic games Journal... Multi-agent’s Q-Learning in Multi-AUV.[J].Applied Science and Technology.Jan.2008,Vol.35,No.1:57-60 7 Li Xiong Chen Wei Guo Jing Zhai Zhenkun Huang Zekai: A New Passing Strategy Based on QLearning Algorithm in RoboCup, Computer Science and Software Engineering, 2008 International Conference 8 Thomas Gabel, Martin Riedmiller, Florian Trost : Defense behavior in Soccer Simulation 2D: The NeuroHassle Approach 9 . about Robocup and our work. The second chapter will talk about a Robocup 2D simulation, one of the oldest sub-leagues in Robocup. In chapter 3, we will review several work and research on Robocup. institutions have been taking Robocup a vehicle to advance the state of the art of intelligent robots All of our team members are interested in Robocup, we are on the way to build a Robocup simulation. international cup. The aim of Robocup is to promote robotics and artificial intelligence research under challenging and adverse conditions. FIGURE 1 ROBOCUP LOGO 1 Robocup provides a standard problem