Tối ưu hóa viễn thông và thích nghi Kỹ thuật Heuristic P16 pptx

16 Evolutionary Game Theory Applied to Service Selection and Network Ecologies Sverrir Olafsson 16.1 Introduction Today an ever increasing number of social and financial services can be provided over data networks. Individuals and businesses alike initiate and complete a large number of commercial transactions over the Internet or other specially constructed networks. Home banking, home shopping, video on demand, buying and selling of stocks and other financial securities can now be undertaken from almost any part of the world where an access to a local network can be achieved. New services are being offered all the time, and when successful face an immediate competition from a large number of companies and individuals with sufficient knowledge base, financial resources and access to the Internet. The increasing competition in service provision on the Internet provides individuals with an increasing choice, lower prices and previously unknown opportunities. But this also creates greater difficulties in making the right choices at the right time. The problem is not only that of making the right choice, but also to acquire the necessary information so that educated decisions can be made. Clearly, all network users or agents browsing the network can in principle acquire a complete information on all competing network services. However, acquiring complete information can be excessively expensive, relative to the only marginal improvements it may provide the holder of that information. The problem therefore is to strike the balance between sacrificing information and the penalties one has to pay for making decisions when equipped with only limited information. Telecommunications Optimization: Heuristic and Adaptive Techniques, edited by D. Corne, M.J. Oates and G.D. Smith © 2000 John Wiley & Sons, Ltd Telecommunications Optimization: Heuristic and Adaptive Techniques. Edited by David W. Corne, Martin J. Oates, George D. Smith Copyright © 2000 John Wiley & Sons Ltd ISBNs: 0-471-98855-3 (Hardback); 0-470-84163X (Electronic) Telecommunications Optimization: Heuristic and Adaptive Techniques284 Also, the increasing speed with which services are introduced requires an increasing complexity of networks as well as the need for efficient distributed control mechanism. The supply of a multitude of continuously changing and interacting services requires the simultaneous or sequential use of many network resources, possibly distributed over locations large geographical distances apart. That requires reliable algorithms, which identify the required sources and support their efficient utilization. It is in principle possible to burden every network and network user with the communication overheads, which make knowledge of all network resources available. Such an overhead, on the other hand, would severely limit the processing capacity of the network and be very time consuming for the user. In principle, the most appropriate resource can be found for the execution of every task, but the computational expense and the time required for achieving that might well exceed the benefits from using that resource as compared with another, slightly less effective one. This problem is of a very general nature and relates to a number of optimization problems, as well as many management issues where performance has always to be compared with the cost of achieving it. As a result of these problems soft, biologically based, approaches to resource management have become increasingly popular in recent years (Huberman, 1988). Predominantly these approaches have been based on a neural network approach and various implementations of genetic algorithms. For example, neural networks have been developed for resource allocation (Bousono and Manning, 1995) and for the switching of network traffic (Amin et al., 1994). Genetic algorithms have been applied to resource allocation (Oates and Corne, 1998b), routing or file management in distributed systems (Bilchev and Olafsson, 1998a), just to mention a few. In this paper we take a different approach, based on evolutionary game theory. Game theoretic approach to network utilisation has been considered by a number of authors in recent years (Olafsson, 1995). Common to most of these approaches is the notion of an agent utility, which dictates the dynamics of the allocation of service requirements to network resources. The system performance resulting from this dynamic therefore depends strongly on the selected agent utility function, i.e. with what rationality criteria the agents have been programmed. A potential problem with agent based approaches is the fact that agents sometimes consider only their own demand for processing requirements. Where each agent is guided only by its own interests, the performance of the whole system can still be very low. In view of this, it seems essential to design systems whose agents are concerned, not only for their own service requirements, but also those of the whole community of agents. This may requires some co-operation between agents, which is aimed at improving the mean efficiency of the whole system. Occasionally, this can be detrimental to the interests of a few agents, but the whole community of agents is likely to benefit from the co-operation. In fact, this problem is very similar to that of the prisoners dilemma (Axelrod, 1984) or the tragedy of the commons (Hardin, 1984), which are representative for many competitive situations in areas as diverse as, politics, commerce, marketing and real biological systems. The approach taken here is based on evolutionary game theory (Maynard Smith, 1989). Agents distribute their service requirements, i.e. they select strategies on the basis of their personal utility function and their present knowledge of the available services. This knowledge includes present availability, price, quality and possible time delays. Generally, the agents are in the possession of only limited information and may therefore have to test the different service options. The service options are considered to be the strategies Evolutionary Game Theory Applied to Service Selection and Network Ecologies 285 available to the agents. On the basis of the utility (fitness value) achieved by applying any set of strategies the agents try to evolve their service selection towards higher utility and lower cost. This process of strategy selection, leading to more efficient resource utilization, resembles the strategic behaviour of an animal community, which on the basis of what it learns evolves its strategies to a more efficient performance. Learning and the updating of strategies based on new information is hence an essential part of the approach shown here. This approach allows us to monitor the system evolution in terms of game theoretic concepts such as Nash equilibrium. In the evolutionary game theory, the concept of Nash equilibrium has been extended to that of an Evolutionarily Stable Strategy (ESS) (Maynard Smith and Price, 1973) which is a strongly biologically motivated concept. The ESS gives a formalised definition of a strategy, which has the ability to resist the invasion of new strategies (phenotypes). A population or an ecosystem, which plays the evolutionarily stable strategy, can therefore not be invaded by opponents playing different strategies. It is useful to comment on a few important differences between the evolutionary game theory approach discussed here and other evolutionary models, such as genetic algorithms (Holland, 1975; Goldberg, 1989), evolutionary strategies (Schwefel, 1981) and genetic programming (Koza, 1993). Evolutionary game theory is a local theory, whereas both genetic algorithms and evolutionary strategies are essentially centralized models requiring a simultaneous comparison of the fitness of all individuals existing at the same time and competing in the same environment. Even though evolutionary computing models have some parallel attributes, this mutual comparison of all individuals presents a potentially limiting bottleneck in their implementation. Recenetly there has been considerable work on more distributed implementations of genetic algorithms (Blickle, 1993), but the absence of a theoretical framework has been very noticeable. In evolutionary game theory the mentioned bottleneck can be avoided as each individual’s fitness is compared against the aggregated mean fitness of all coexisting individuals (phenotypes). The information required for strategy update is therefore local because no centralized procedure is needed. Evolutionary game theory could thus provide some theoretical basis for analyzing distributed systems. This chapter is organized as follows. In section 15.2 we give a brief description of genetic algorithms and how by applying schema theory these can be formulated as a dynamic replicator algorithm. In section 15.3 we provide a brief discussion of evolutionary game theory, followed by a section on basic replicator dynamics. Section 15.5 formulates evolutionary game theory as a replicator dynamic, and in section 15.6 we discuss in general terms how evolutionary game theory can be applied to resource allocation on networks. Section 15.7 looks at the links between evolutionary game theory and dynamic models driven by general agent expectations. In section 15.8 we discuss how load distribution on server networks emerges from an expectation dynamic, and give sufficient conditions for this load distribution being stable. In this section we also discuss two examples, and give the conditions for stable load distributions in both cases. We end with general conclusions and comments on the outlook for this approach to service selection on networks. 16.2 Connecting Genetic Algorithms and Evolutionary Game Theory In spite of implementation differences genetic algorithms and evolutionary game theory share some very fundamental analogies, based essentially on the principles of a replicator Telecommunications Optimization: Heuristic and Adaptive Techniques286 dynamic. These similarities are in fact so fundamental that both models can be viewed as two different dynamic implementations of a general replicator dynamic. Of course this is not surprising as both models have their roots in two biological domains, one considering the molecular basis of evolution whereas the other studies the competition between species (phenotypes) in an ecological context. In this section we will discuss these similarities and demonstrate how both algorithms lead to a mean fitness increasing dynamics. Genetic Algorithms (GA) (Holland, 1975; Goldberg, 1989) draw upon principles from the theory of natural selection and were initially introduced as a possible model of real evolution. It was only later realized that genetic algorithms could successfully be applied to a number of different optimisation problems (De Jong, 1975). The basis for the application of GA to optimisation is the fact that they define a search strategy that is biased towards search points of high fitness, i.e. points, which tend to improve the value of the cost function. GAs have similarities with simulated annealing (Geman and Geman, 1984) in that they employ random search strategies. The main difference between the two methods, however, is the fact that GAs have intrinsic parallel features which render them powerful in solving a number of important optimisation problems. GAs have turned out to be particularly valuable in dealing with problems where the external circumstances cannot be modeled precisely. Within the framework of GAs and evolutionary models in general, evolution can broadly be viewed as a two-step process of random structure modification followed by selection. Random mutations or cross over processes generate new structures, or modify existing genes, whose phenotypic manifestations are then tested against the requirements of the environment. However, in the language of optimisation each randomly generated structure offers a solution to the defined cost function. Depending on how good that solution is, when compared with other already known solutions, it is accepted or rejected by the selection process. GAs distinguish themselves from other optimization algorithms in that at any moment in time they maintain a collection of generated structures, or points in search space, rather that simply one point. All these points or structures are competing solutions to the optimization problem. It is the implementation of the selection algorithm that decides which structures to keep and which ones to get rid of. The fundamental entities in most implementations of GA are strings of variable or fixed length N. In the case of fixed length strings the state space can be presented as the set: {} N N 1,0=Ω of all binary strings of length N. Within a given problem framework each element in this space can be associated with a given fitness; hence there is a fitness function: ℜ→Ω N f : where ℜ is the set of real numbers. The objective of a GA is therefore to find the optimal value of this fitness function over the space of binary strings: () sff N s m Ω∈ = max Evolutionary Game Theory Applied to Service Selection and Network Ecologies 287 The collection of all strings kept at any moment in time defines the present population of strings. All members of this population compete to be accepted as a solution to the given problem. Generally, it is not known which regions in N Ω are likely to provide good or even acceptable solutions to a given problem. In the absence of any such knowledge the best one can do is to generate randomly an initial population of strings. Assume that the initial population size is M then the initial population at time t=0 is presented by a MN × matrix: () () ()() 0, ,00 1 M ssS = In this notation each position () 0 i s presents an N component binary string: () () ()() 0, ,00 ,1, Niii sss = From this initial population subsequent populations are generated by the use of mutation and/or crossover followed by a selection. Formally we write for this process: () () ( ) 10 →→→→ TSSS (16.1) The competitive pressure in the evolutionary process is based on the fact that the population size is usually kept constant at each generation. Of the generally large number of offspring produced by each generation, only a few are selected to survive and to produce their own offspring. Assume that at some time t we have the population ( ) tS still of size M. After mutations and crossovers, within this population, offspring are generated and the population increases in size. From the new enlarged population only M are allowed to survive to produce new offspring. The selection of survivors to be kept for further breeding generally depends, in a stochastic manner, on the fitness of the candidates. Each member of the population, say ( ) ,ts i has some fitness ( ) tf i and his survival into the next generation depends upon this fitness value. Essentially the chance of survival to the next generation is given by the expression p f f i i j j = ∑ (16.2) Therefore, the fitter an individual is the more likely it is to survive to produce offsprings by a mutation or a crossover with other fit individuals, which were also selected for reproduction. 16.2.1 Schema Theory Consider a binary string where some positions are fixed, whereas the remaining positions can take arbitrary values, i.e. either 0 or 1. To follow a general convention we take the symbol ∗ to represent 0 or 1. An example of a string in this notation is { } 01110 ∗∗∗=s . A Telecommunications Optimization: Heuristic and Adaptive Techniques288 schema H is defined as a subset of strings in the space of strings N Ω . The structure of the schema is provided by the fact that certain positions in the strings that belong to H are fixed. The remaining positions, indicated by ∗ are variable. An example of a set of four bit strings belonging to the same schemata ( ) 011∗∗=H is given by the set: ()()()(){} 11110,11100,01110,01100=S (16.3) { } 011∗∗=H is therefore a compact notation for the set S. As the strings belonging to given schema are presented by three symbols { } 0 1 ,,∗ there are in total 3 N different possible schemata. This is in fact much larger than N 2 the number of possible strings. Therefore, any given string will belong to N 2 different schemata. To develop a dynamic based on preferential selection of high fitness let ( ) tHm , be the number of strings, which at time t belong to schema H. Remember, ( ) tS presents the whole set of strings in existence at time t. Therefore the number of strings ( ) tHm , has to be in the intersection ( ) tSH ∩ . Each set of strings can be presented in terms of a smaller set of schemata classes. Formally, one can decompose any set of strings into a set of disjoint schemata classes: U i i HS = Now, let () tHf , be the average fitness of strings in () tSH ∩ and )(tf the average fitness of all strings in the population ( ) tS at time t. It can be shown (Goldberg, 1989) that the number of strings still belonging to schema H at time t + 1 is given by: ()() () () tHm tf tHf mtHmtHm , )( , ,1, =∆+=+ (16.4) As we are mainly interested in the change in the number of strings belonging to the schema H, we rewrite this expression as: () () () tHm tf tftHf tHm , )( )(, ,         − =∆ (16.5) From this it is obvious that as long as the fitness of strings in H is larger than the mean fitness of all strings the change is positive, i.e. ( ) 0, >∆ tHm . Otherwise, the population of strings belonging to H is reduced. There is, however, a small caveat to this argument as the genetic operations can remove some of the strings from the schema H. It is possible to provide some analysis on the type of operations, which leave the schema structure H invariant. For that purpose one defines two new quantities. The order of a schema, ( ) Ho is simply the number of fixed position on the string. The defining length of a schema ( ) H δ is the distance between the first and the last fixed positions on the schema. It can be shown (Goldberg, 1989) that a schema is invariant with respect to a cross over when the cross over position is chosen outside the strings Evolutionary Game Theory Applied to Service Selection and Network Ecologies 289 defining length. If c P denotes the probability with which the crossover operator is applied, then: () 1 1 , − −= N HP p c cs δ (16.6) is the probability that the string survives the crossover operation. Similarly, if m P is the probability of a mutation then: () () Ho mms Pp −= 1 , is the probability that a string will still belong to the schema H after the effects of a mutation. From this we find that the expected number of strings belonging to the schema H after the application of crossover and mutation is given by: () () () tHmtHmpptHm mscs mc ,,, ,, , <= (16.7) For small values of m P we have () () () m Ho m PHoP −=− 11 and therefore: () ()() m c mscs pH N Hp pp ο δ −       − −= 1 1 1 ,, Multiplying out the bracket and ignoring the term containing mc PP we have: () () pp pH N Hp sc sm c m,, =− − −1 1 δ ο (16.8) Equation 16.5 has to be modified slightly to accommodate for the effects of these probabilities. One finds: () () () () ( ) tHmPHo N HP tf tHf tHm m c , 1 1 )( , 1,       − − −≥+ δ (16.9) If ( ) tM is the total population at time t then we can look at: () () () tM tHm tHp , , = as the probability that a randomly selected string from the total population belongs to the schema H. In terms of this probability we rewrite equation 16.9 as: Telecommunications Optimization: Heuristic and Adaptive Techniques290 () () () () () ( ) pHt fHt ft PH N oHP pHt c m , , ,+≥ − − −       11 1 δ (16.10) The interpretation of this equation is as follows. The expression ( ) tHp , gives the probability that a string selected at time t belongs to the schema H. The expression in the large bracket is the probability that the children of this selected string will also belong to the schema H. The probability that these children belong to H is further biased by: () )( , tf tHf which measures the ratio of their fitness to the mean fitness of the whole population of strings. Finally, the change in probability of a string belonging to schemata H from one generation to another can be written as () () () () () () ( )         −       − − −≥∆ tHfpH N Hp tHf tHf tHp tHp m c , 1 1, , , , ο δ (16.11) If the probability that a string leaves a schema as a result of crossover or mutation can be neglected the above expression simply reads: ()() () () ()         − =∆ tHf tHftHf tHptHp , ,, ,, (16.12) Equations 16.11 and 16.12 demonstrate the basic principle of a replicator dynamic which is to be interpreted in the following way. The fitness of each schema is to be compared with the mean fitness of the whole community of strings. If the schema fitness turns out to be larger than the mean fitness of all strings then the number of strings belonging to that schema class is increased, i.e. it is rewarded by a higher number of offspring. Structurally the equations 16.11 and 16.12 are very similar to the dynamic replicator equations considered in evolutionary game theory as will be explained in the next section. 16.3 Evolutionary Game Theory The application of game theory to evolution goes back to the works of Lewontin (1960) and later Slobodkin and Rapoport (1974). These authors used game theory mainly to analyse the struggle of a species against nature. In the game, a species seeks to construct strategies, which maximise its fitness (probability of surviving and producing offspring) under given environmental conditions. Later, Maynard Smith and Price (1973) extended the theory and applied it to animal conflict situations as they occur in territorial claims or competition for mates or dominance rights. Central to their approach to evolutionary game theory is the Evolutionary Game Theory Applied to Service Selection and Network Ecologies 291 analysis of pairwise contests between opponents. In every contest each opponent can apply only a limited number of strategies or different combinations of strategies. An essential feature of evolutionary game theory is its dynamic nature. The probability distribution on the set of available strategies evolves as a result of how successful the strategies are. Maynard Smith and Price (1973) introduced the concept of evolutionarily stable strategy as an uninvadable strategy. They demonstrated that, in a number of cases, the strategy distribution eventually evolved towards an evolutionarily stable strategy. This concept has been immensely popular amongst evolutionary biologists. It has also been successful at explaining some evolutionarily stable conditions such as the sex ratio in many species (Hamilton, 1967). More recently evolutionary game theory has also received some interest from game theorists and economists (Weibull, 1995). The application of conventional game theory to economics and decision theory has not been as successful as some of the field’s initiators had hoped for. It was indeed noted by von Neumann that some of game theory’s most serious limitations were due to its lacking of dynamics. Clearly, some economists and game theorists have, therefore, watched with interest the emergence of the dynamic evolutionary game theory. Also, game theory has been applied to resource allocation and routing on communication and data networks. In view of the rapid changes that are taking place in the provision of networked services in recent years it appears that dynamic modeling is more appropriate for dynamically changing service environment. In this environment, users have to make decisions as to what services are most suitable to satisfy their needs. This is why it was suggested in Olafsson (1996) that evolutionary game theory may be well suited for the modeling of modern service provision. But also evolutionary game theory is not entirely without problems. In spite of its dynamic nature the concept of equilibrium plays a fundamental role. Even though evolutionarily stable strategy has a well defined meaning in ecological systems (Maynard Smith, 1989) its meaning in competitive markets and service provision systems is far from clear. Also, what decision-making processes in human, market, social or service selection scenarios are responsible for the strategy distribution moving towards an evolutionarily stable strategy? Or, in more basic terms, under what general conditions can one expect market competitors and/or service selectors to play the evolutionarily stable strategy? When this criticism is considered, one has to be aware that the concept of equilibrium (evolutionarily stable or not) is, even in evolutionary theory a difficult one as evolution demonstrates a process of continuous change. It does nevertheless supply the evolutionary biologist with valuable tools for the study of animal to animal or animal to environment conflict situations. Even though the system trajectories are generally far more difficult to analyse than are the equilibria, they contain important information on the dynamic of the learning process. Some studies on the learning process leading to evolutionarily stable strategies have been undertaken in recent years (Zeeman, 1981). The user service selection process is naturally modeled as a learning process. With an increasing speed with which products and services are designed and then brought to market, accompanied by an ever shorter life expectancy, the continuous consumer learning process becomes an ever more important issue. This dynamic and volatile environment calls for a dynamic and adaptive modeling. In Olafsson (1995a; 1996) we have argued that these scenarios are appropriately modeled by evolutionary game theory, where both the transitory learning dynamics and the resulting equilibria may be of considerable interest. Telecommunications Optimization: Heuristic and Adaptive Techniques292 There is one final point worth making. Whereas conventional game theory has a clear notion of a player which when facing other players applies some given strategy, evolutionary game theory puts the main emphasis on the strategy itself. There is no clear definition of a ‘player’. The player concept is replaced by a large population of players, which can select from a finite number of strategies. At any time, when the whole population of players is in a certain state of a ‘mixed strategy’, fitness can be associated with this mixed strategy, and therefore the whole population, as well as to individual strategies. A player, if one wants to hold on to the concept, becomes in principle only an anonymous member of the population, which applies any of the strategies with some probability. 16.4 Replicator Dynamics In this section we give a brief description of the concepts, problems and ideas behind the application of evolutionary game theory to the analysis of strategy evolution in ecological systems, as developed by Maynard Smith and Price (1973). As in most evolutionary systems the model discussed here describes an evolutionary system driven by increased fitness expectations. Generally fitness is not simply some kind of a deterministic function but a probabilistic linear or non-linear combination of some options available to an individual. We demonstrate in this section that evolutionary game theory is simply one instance of an application of competitive replicator dynamics, where the fitness of each individual is compared against the mean fitness of the community it belongs to. A dynamic of this kind is generally (not always) mean fitness increasing, i.e. the mean fitness of the community tends to increase under the evolutionary dynamics. We start with a set of possible strategies presented by a set { } N sssS , ,, 21 = . We assume that individuals have to choose from this set of possible strategies. Assume that the fitness of strategy i s is given by i f and that the vector ( ) N x x x X , ,, 21 = gives the distribution of the population over the different strategies, i.e. x i is the fraction, which selects strategy . i s Generally, the fitness depends upon this distribution, ( ) Xff ii = . The mean fitness of all individuals is given by: ∑ = = N i ii fxXf 1 )( (16.13) Each individual’s fitness ( ) Xff ii = will be measured against this mean fitness. The difference: ff ii −=∆ gives the fitness individuals selecting i s have in excess of the mean fitness of all individuals. Only if 0>∆ i will selecting i s be more rewarding than following the selection mixture as presented by the vector ( ) N x x x X , ,, 21 = . On the other hand, if 0<∆ i then the pure strategy i s is weaker than the mixed strategy ( ) N x x x X , ,, 21 = . In evolutionary terms i ∆ can be viewed as the growth rate, positive or negative, of the ith strategy or phenotype. The selection dynamic itself can then be written as: [...]... i, j = 1, , N is the payoff to an individual applying strategy s i against an individual using strategy s j Note that this definition does not imply that the gain 294 Telecommunications Optimization: Heuristic and Adaptive Techniques matrix is symmetric as in general, U ij ≠ U ji for i ≠ j Essentially, the introduction of the utility matrix allows for the mixing of strategies Assuming that the population... similar quantity can be written down Assuming that M servers have been contacted, the information available to the user can be written as: I = (I 1 , , I M ) (16.22) 296 Telecommunications Optimization: Heuristic and Adaptive Techniques A user could well have clearly defined criteria, which would enable him to make a straightforward server selection Generally, the situation is not that simple as the selection... this server as Ei (ci , X ) = ci − xi and the mean utility as E (C , X ) = ∑k =1 x k (c k − x k ) N The replicator dynamics for the server system is given by: (16.30) 298 Telecommunications Optimization: Heuristic and Adaptive Techniques N   x i = xi  c i − x i − x k (c k − x k ) &   k =1   ∑ (16.31) Either one of the two following conditions is sufficient for the dynamics to reach an equilibrium:... then equation 16.41 takes on the much simpler form, Ω = ∆ For a more thorough discussion of the stability properties of the expectation dynamic, see Olafsson (1996) 300 Telecommunications Optimization: Heuristic and Adaptive Techniques Figure 16.2 The utility function (equation 16.43) The surface is parameterized by the two variables xi and xj Figure 16.3 The figure displays the conditions for stable... 300 350 400 450 500 Figure 16.4 Load distribution based on the fixed utility matrix () The large variance in the load distribution implies poor resource utilization 302 Telecommunications Optimization: Heuristic and Adaptive Techniques The jiggery load curves result from the system noise, which is used to model the limited information available to the agents The noise which is normally distributed is... considerably lower variance in the load distribution, a standard deviation of std (x m ) = 0.049 as opposed to 0.15, when the distribution is controlled by H0 alone 304 Telecommunications Optimization: Heuristic and Adaptive Techniques 0.18 Fixed utility Standard deviation in fitness 0.17 0.16 0.15 0.14 0.13 0.12 Adaptive utility 0.11 0.1 0 100 200 300 400 500 Time 600 700 800 900 1000 Figure 16.6 The . information. Telecommunications Optimization: Heuristic and Adaptive Techniques, edited by D. Corne, M.J. Oates and G.D. Smith © 2000 John Wiley & Sons, Ltd Telecommunications Optimization: Heuristic and Adaptive. Ltd ISBNs: 0-471-98855-3 (Hardback); 0-470-84163X (Electronic) Telecommunications Optimization: Heuristic and Adaptive Techniques284 Also, the increasing speed with which services are introduced. analogies, based essentially on the principles of a replicator Telecommunications Optimization: Heuristic and Adaptive Techniques286 dynamic. These similarities are in fact so fundamental that