A Stochastic Evolutionary Game Approach to Energy Management in a Distributed Aloha Network

Transcription

1 A Stochastic Evolutionary Game Aroach to Energy Management in a Distributed Aloha Network Eitan Altman, Yezekael Hayel INRIA, 2004 Route des Lucioles, Sohia Antiolis, France altman@sohiainriafr LIA, Université d Avignon 339, chemin des Meinajaries, Agroarc BP 228, 849 Avignon Cede 9, France yezekaelhayel@univ-avignonfr Abstract A major contribution of biology to cometitive decision making is the area of evolutionary games It describes the evolution of sizes of large oulations as a result of many local interactions, each involving a small number of randomly selected individuals An individual lays only once; it lays in a one shot game against another randomly selected layer with the goal of maimizing its utility (fitness) in that game We introduce here a new more general tye of games: a Stochastic Evolutionary Game where each layer may be in different states; the layer may be involved in several local interactions during his life time and his actions determine not only the utilities but also the transition robabilities and his life duration This is used to study a large oulation of mobiles forming a sarse ad-hoc network, where mobiles comete with their neighbors on the access to a radio channel We study the imact of the level of energy in the battery on the aggressiveness of the access olicy of mobiles at equilibrium We obtain roerties of the ESS (Evolutionary Stable Strategy) equilibrium which, Unlike the Nash equilibrium concet, is robust against deviations of a whole ositive fraction of the oulation We further study dynamical roerties of the system when it is not in equilibrium I INTRODUCTION The evolutionary games formalism is a central mathematical tools develoed by biologists for redicting oulation dynamics in the contet of interactions between oulations This formalism identifies and studies two concets: The ESS (for Evolutionary Stable Strategy), and the Relicator Dynamics [0] The ESS is characterized by a roerty of robustness against invaders (mutations) More secifically, (i) if an ESS is reached, then the roortions of each oulation do not change in time (ii) at ESS, the oulations are immune from being invaded by other small oulations This notion is stronger than Nash equilibrium in which it is only requested that a single user would not benefit by a change (mutation) of its behavior ESS has first been defined in 972 by M Smith [3], who further develoed it in his seminal tet Evolution and the Theory of Games [4], followed shortly by Aelrod s famous work [] Although ESS has been defined in the contet of biological systems, it is highly relevant to engineering as well (see [5]) In the biological contet, the relicator dynamics is a model for the change of the size of the oulation(s) as biologist observe, where as in engineering, we can go beyond characterizing and modelling eisting evolution The evolution of rotocols can be engineered by roviding guidelines or regulations for the way to ugrade eisting ones and in determining arameters related to deloyment of new rotocols and services In doing so we may wish to achieve adatability to changing environments (growth of traffic in networks, increase of seeds or of congestion) and yet to avoid instabilities that could otherwise revent the system to reach an ESS Evolutionary Stable Strategies Consider a large oulation of layers Each individual needs occasionally to take some action (such as ower control decisions, or forwarding decision) We focus on some (arbitrary) tagged individual Occasionally, the action of some N (ossibly random number of) other individuals interact with the action of that individual (eg other neighboring nodes transmit at the same time) In order to make use of the wealth of tools and theory develoed in the biology literature, we shall often restrict, as they do, to interactions that are limited to airwise, ie to N = This will corresond to networks oerating at light loads, such as sensor networks that need to track some rare events such as the arrival at the vicinity of a sensor of some tagged animal We define by J(,q) the eected ayoff for our tagged individual if it uses a strategy (also called olicy) when meeting another individual who adots the strategy q This ayoff is called fitness and strategies with larger fitness are eected to roagate faster in a oulation and q belong to a set K of available strategies In the standard framework for evolutionary games there are a finite number of so called ure strategies, and a general strategy of an individual is a robability distribution over the ure strategies An equivalent interretation of strategies is obtained by assuming that individuals choose ure strategies and then the robability distribution reresents the fraction of individuals in the oulation that choose each strategy Note that J is linear in and q

2 Suose that the whole oulation uses a strategy q and that a small fraction ǫ (called mutations ) adots another strategy Evolutionary forces are eected to select against if J(q,ǫ + ( ǫ)q) > J(,ǫ + ( ǫ)q) () Definition : A strategy q is said to be ESS if for every q there eists some ǫ y > 0 such that () holds for all ǫ (0,ǫ y ) In fact, we eect that if for all q, J(q,q) > J(,q) (2) then the mutations fraction in the oulation will tend to decrease (as it has a lower reward, meaning a lower growth rate) q is then immune to mutations If it does not but if still the following holds, for all q, J(q,q) = J(,q) and J(q,) > J(,) (3) then a oulation using q are weakly immune against a mutation using since if the mutant s oulation grows, then we shall frequently have individuals with strategy q cometing with mutants; in such cases, the condition J(q,) > J(,) ensures that the growth rate of the original oulation eceeds that of the mutants We shall need the following characterization: Theorem : [9, Proosition 2] or [2, Theorem 64, age 63] A strategy q is ESS if and only if it satisfies (2) or (3) Corollary : (2) is a sufficient condition for q to be an ESS A necessary condition for it to be an ESS is for all q, J(q,q) J(,q) (4) The conditions on ESS can be related to an interreted in terms of Nash equilibrium in a matri game The situation in which an individual, say layer, is faced with a member of a oulation in which a fraction chooses strategy A is then translated to laying the matri game against a second layer who uses mied strategies (randomizes) with robabilities and, res The central model that we shall use to investigate rotocol and service evolution is introduced in the net Subsection along with its matri game reresentation We can learn and adot notions from biology not only through the concet of evolutionary game, but also in alications related to energy issues that have a central role both in biology as well as in mobile networking The long term animal survival is directly related to its energy strategies (cometition over food etc), and a oulation of animals that have good strategies for avoiding starvation is more fit and is eected to survive [], [2], [3] By analogy, we may eect mobile terminals which adot efficient energy strategies to live longer Networks with more efficient nodes then have more chances to survive longer [4] Another element of evolutionary games is the relicator dynamics It has been used for describing the evolution of road traffic congestion in which the fitness is determined by the strategies chosen by all drivers [7] It has also been studied in the contet of the association roblem in wireless networks in [8] We shall roose a relicator dynamics that etends the standnard one and study its convergence The structure of the aer is as follows In the net section we resent the model and rovide some notation and definitions We then comute in Section III the ESS as a function of the system s arameters, and then study its roerties in Section IV In Section V we comare the erformance at ESS which characterizes the non-coooerative behavior to that obtained by a cooerative aroach We then roose and study in Section VI the relicator dynamics and end with a section on numerical eamles II MODEL We consider a large oulation of mobile terminals We assume that the density of the network is low, so that if a terminal attemts transmission one can neglect the robability of interference from more than one other mobile (called neighbor ) Interference occurs as in the Aloha rotocol: if more than one neighbors transmit a acket at the same time then there is a collision To avoid loosing both ackets at a collision, we assume that a mobile whose battery energy level is high can transmit at a high ower This allows to recover the transmission whenever a collision occurs with a acket transmitted at a low energy level More recisely, we consider non-cooerative game among a large oulation of users of mobile terminals A terminal i attemts transmissions during some sequence {t i n} of times, called time oortunities or time slots At each attemt, it has to take a decision on the transmission ower based on his battery energy state To simlify, we assume that the state can take three values: {F,A,E} for Full, Almost emty or Emty The transmission signal ower of a terminal can be High (h) or Low (l) We call these the actions that a terminal can take Transmission at high ower is ossible only when the mobile is in state F At state A only action l is available, and at E transmission is not ossible any more The lifetime of the mobile is defined as the number of slots during which its battery is nonemty We consider an Aloha-tye game where a mobile transmits a acket with success during a slot if: the mobile is the only one to transmit during this slot the mobile transmits with high ower and all others transmitting nodes use low ower A Some notation We introduce the following notations: is the robability for a mobile to be the only transmitter during a slot Q i (a) is the robability of remaining at energy level i when using action a Since state A only action l is available, we write Q A instead of Q A (l) α is the fraction of the oulation who use the action h at any given time Note that for a given α, the roortion of the oulation that choose action h at state F is higher than α, since at state A a mobile cannot choose h

3 B Policies In standard evolutionary games, a layer takes only one action The result of the action is reresented using a matri R whose entry (i,j) reresent the fitness of the individual if it lays action i when it meets an individual that lays action j Then, if a fraction α j of the oulation lays action j, then the eected fitness of and individual laying action i is j α jr ij The game we face here is different since an individual can take several actions during its lifetime Hence the fitness cannot be directly reresented as an entry of a matri whose rows corresonds to the layers actions only Instead, an entry would corresond to the total eected fitness that the layer gets in his lifetime as a function of its olicy A general olicy u is a sequence u = (u,u 2,) where u i is the robability of choosing h if at time i the state is F A stationary olicy is a olicy for which at any time that the individual is at state F, the robability u i of choosing h is the same We shall use a single real number β to denote a stationary olicy of an individual β stands for the robability that a mobile uses h when at state F A stationary olicy is called ure if it does not use randomization (in our case there eist two ure stationary olicies: the one that always choose h at state F and the one that always choose l) Other stationary olicies are called randomized stationary olicies (They lay the role of mied olicies in standard evolutionary games) C Fitness Choose some individual terminal and let R t denote the number of ackets (zero or one) successfully transmitted at time slot t by this terminal We define the fitness of the terminal to be given by t= R t, ie the total number of ackets successfully transmitted during its lifetime Assume that α is fied and does not change in time (we shall consider later the case in which it may vary in time) Then the eected otimal fitness of an individual starting at a given initial state can be comuted using the standard theory of total-cost dynamic rogramming, that states in articular that there eist otimal stationary olicy (ie a olicy for which at any time that the individual is at state F, the robability u i of choosing h is the same) We shall therefore restrict to stationary olicies unless stated otherwise Some more definitions: V β (i,α) is the total eected fitness (ie reward or valuation) of a user given that it uses olicy β, that it is in state i and given the arameter α We define with some abuse of notation V h (i,α) (resectively V l (i,α)) to be the total eected fitness of a user given that it is in state i, that it uses action h when at state F (action l, resectively) and given the arameter α III COMPUTING FITNESS AND EXPECTED SOJOURN TIMES We roceed by comuting the individual s eected total utility and remaining lifetime that corresond to a given initial state and a stationary olicy β We note however that only at state F there will be a deendence on β State E When the level of energy is in state E, the valuation is equal to V (E) = 0 State A When the state is A, the valuation is which gives that V (A) = + Q A V (A), V (A) = Q A The eected time during which a mobile sends in state A is given by T(A) = + Q A T(A) which imlies that T(A) = Q A State F Define the dynamic rogramming oerator Y (v,a,α) to be the total eected fitness of an individual starting at state F, if It takes action a at time, If at time 2 the state is F then the total sum of eected fitness from time 2 onwards is v At each time the mobile attemts transmission, the robability that another interfering mobile uses action h, given that there is a simultaneous transmission with another mobile, interferes with another transmission, is α Below, α is omitted for the case a = l since the deendence on α aears only with the action a = h We have and Y (v,l) = + Q F (l)v + ( Q F (l))v (A), (5) = + Q F (l)v + Q F(l) Q A Y (v,h,α) = α( + Q F (h)v + ( Q F (h))v (A)) (6) +( α)( + Q F (h)v + ( Q F (h))v (A)), = α + ( α) + Q F (h)v + Q F(h) Q A Assume that a mobile uses h w β at state F Then the eected time it sends at state F is T β (F) = + βq F (h)t β (F) + ( β)q F (l)t β (F) which gives T β (F) = βq F (h) ( β)q F (l)

4 The fraction of time that the mobile uses action h is then T(F) α(β) = β T(F) + T(A) Q A = β (7) 2 Q A βq F (h) ( β)q F (l) Denote by V β (F,α) the total eected utility the mobile gains starting from state F Then V β (F,α) is the unique solution of This gives v = ( β)y (v,l) + βy (v,h,α) V β (F,α) = ( Q A )[( β)( Q F (l) + β( Q F (h)))] ( β)q F (l) βq F (h) ( β) + β(α( ) + ) + ( β)q F (l) βq F (h) = ( β) + β(α( ) + ) + Q A ( β)q F (l) βq F (h) = + β( )( α) V (A) + Q F (l) + β(q F (l) Q F (h)) In the secial case of the aggressive olicy β =, we obtain α, V (F,α) = V (A) + and for β = 0, we obtain α, V 0 (F,α) = V (A) + α( ) Q F (h), (8) Q F (l) (9) Remark : Differentiating (8) with resect to β, we obtain dv β (F,α) dβ = (0) ( )( α)( Q F (l)) (Q F (l) Q F (h)) ( Q F (l) + β(q F (l) Q F (h))) 2 This is either strictly ositive for all β [0,] or strictly negative for all β [0,] or equals zero over all the interval We conclude that V β (F,α) is either constant or strictly monotone in β over the whole interval [0,] IV PROPERTIES AND CHARACTERIZATION OF THE ESS A Preliminaries As already mentioned, our game is different and has a more comle structure than a standard evolutionary game In articular, the fitness that is maimized is not the outcome of a single interaction but of the sum of fitnesses obtained during all the oortunities in the mobile s lifetime In site of this difference, we shall still use the definition or the equivalent conditions (2) or (3) for the ESS but with the following changes: β relaces the strategy q in the initial definition, β relaces the strategy in the initial definition, We use for J(q,) the total eected fitness V β (F,α), where α = α(β ) is given in (7) We obtain the following characterizations of ESS Corollary 2: A necessary condition for β to be an ESS is for all β β, V β (F, α(β )) V β (F, α(β )) () A sufficient condition for β to be an ESS is that () holds with strict inequality for all β β We resent net a structural roerty of ESS It is an adatation to our setting of the fact that in standard evolutionary games, only ure olicies can be ESS satisfying (2); where as a non-ure stationary olicy can be an ESS only if it satisfies (3), that is Theorem 2: A necessary condition for a non-ure stationary olicy β to be an ESS is that for all β β, V β (F, α(β )) = V β (F, α(β )) (2) Equivalently, assume that for some stationary olicies β and β β we have V β (F, α(β )) V β (F, α(β )) Then a necessary condition for β to be an ESS is that it is a ure olicy Proof: If V β (F, α(β )) < V β (F, α(β )) then β is not ESS due to Corollary 2 Assume that β is not ure (it is neither 0 nor ) and that V β (F, α(β )) > V β (F, α(β )) for some β β Alying Remark with α = α(β ) and β = β we conclude that V (F, α(β )) > V β (F, α(β )) or V 0(F, α(β )) > V β (F, α(β )) Hence the necessary condition for β to be an ESS (see Corollary 2) does not hold, which establishes the roof We shall call an ESS olicy β that satisfies (2) weakly immuned ESS, and one satisfying (4) with strict inequality for all β β a strongly immuned ESS Remark imlies that an ESS is either strongly immuned or weakly immuned In other words, there does not eist an ESS β for which (4) holds with equality for some β β and with strict inequality for some other β We roceed by identifying range of arameters for which various ESS structures are obtained B Pure equilibrium: high ower at F We define an aggressive multi-olicy to be the one in which all mobiles use high ower each slot at state F A mobile that always uses high ower at state F sends an eected amount of time of T(F) = Q F (h) at state F The fraction of time it sends at state F is thus α() = Theorem 3: Define T(F) T(F) + T(A) = Q A 2 Q A Q F (h) Q F (h) h := 2 Q A Q F (h) ( ) Q F(l) Q F (h) Q F (l) (3)

5 Let u be the ure aggressive strategy that uses always h at state F (i) h > 0 is a sufficient condition for u to be an ESS (ii) h 0 is a necessary condition for u to be an ESS Proof: We substitute the value of α from eq (3) in (8) to obtain V (F, α()) = V (A) (4) ( ) Q A + ( ) Q F(h) 2 Q A Q F(h) To rove (i), we have to check that if h > 0 then for all β we have V (F,α()) > V β (F,α()) (5) For the case β = 0, we obtain when taking the difference between (9) and (8) that V (F,α()) V 0 (F,α()) = h Q F (h) so (5) indeed holds for β = 0 if h > 0 We show net that this holds for all other β (0,) as well It follows from Remark that V (F,α()) > V β (F,α()) either holds for every β [0,) or for no β in this interval Since we showed that if h > 0 then it holds for β = 0 then we conclude that h > 0 is a sufficient condition for (5) to hold for all β The necessary condition is roved by showing that h 0 imlies that V (F,α()) V β (F,α()) 0 for all β This is done in the same way as the sufficient condition is established, and then we aly Corollary 2 Conclusions We can draw many qualitative conclusions from the Theorem; here are some of them Note that we have Q F (l) > Q F (h) because using less ower, the mobile has more robability to stay in state full than using high ower If, for any reason, those robabilities are the same, ie Q F (l) = Q F (h), then the strategy high ower is obviously an ESS If = 0 (ie the robability of having another simultaneous transmission is null) then there is no benefit from transmission with high ower Indeed we see that in this case < 0 In fact, we can conclude that there is a threshold h given by ( Q F(h))( Q F(l)) ( Q F(h))( Q F(l)) + (2 Q A Q F(h))(Q F(l) Q F(h)) so that u is an ESS for > h and is not ESS for < h h > 0 is a sufficient and necessary condition for u to be a strongly immune ESS C Pure equilibrium: low ower at F We investigate conditions for which in the full state, each user decides to transmit always with low ower Theorem 4: Define l := ( Q F (h)) ( Q F (l)) Let v be the ure strategy that uses always l at state F (i) l > 0 is a sufficient condition for v to be an ESS (ii) l 0 is a necessary condition for v to be an ESS Proof: Note that α(0) = 0 We have V 0 (F,0) V (F,0) = l ( Q F (H))( Q F (l)) Hence l > 0 imlies that V 0 (F,0) > V β (F,0) for β = As in the roof of Theorem 3, we can use Remark to show that this imlies that V 0 (F,0) > V β (F,0) for all β 0 Alying Corollary 2, we conclude that l > 0 is a sufficient condition for the olicy v (ie for β = 0) to be ESS Conclusions We can draw many qualitative conclusions from the Theorem; here are some of them Assume that Q F (l) < or that < If Q F (h) = Q F (l) then l < 0 so v is not ESS Define l = Q F(l) Q F (h) Then it is seen that for all < l, the olicy v is an ESS, and for all > l it is not an ESS Note that the conditions for the olicy u (in Theorem 3) to be ESS deended on Q A where as the conditions for the olicy v (Theorem 4) to be ESS does not The reason is that the condition for ESS of a olicy β involves α(β) which deends on Q A for all β s ecet β = 0 l > 0 is a sufficient and necessary condition for v to be a strongly immune ESS D Weakly Immune ESS and Mied Equilibrium In the revious Subsections we characterized all strongly immune ESS In view of Theorem 2, a necessary condition for β to be a weakly immune ESS is that V β (F, α(β )) be indeendent of β Equivalently, we need β to be such that dv β (F,α)/dβ = 0 where α = α(β ) Using (0) this condition rovides: which yields α = (Q F(l) Q F (h)) ( )( Q F (l)) α = ( Q F(l)) ( Q F (h)) (6) ( )( Q F (l)) For a real number ζ we denote below ζ := ζ Theorem 5: (a) Each one of the following conditions is necessary for there to eist a Weakly Immune ESS: Condition (i): l 0, Condition (ii): h 0, (b) Assume that Condition (i) and (ii) hold Then there eists a unique weakly immune ESS given by β (Q A + Q F(l))[Q F(l) Q F(h)] = Q AQ F(l) (Q F(l) Q F(h))(Q F(l) Q F(h)) (7)

6 Proof: A necessary condition for α in (6) to corresond to an ESS is that 0 α α() First, we observe that the enumerator of (6) is l This rovides that l 0 imlies α 0 Second, the condition α α() is ( Q F (l)) ( Q F (h)) ( )( Q F (l)) This condition is equivalent to m := But we have: Q A 2 Q A Q F (h) Q A ( QF(l)) ( QF(h)) 0 2 Q A Q F(h) ( )( Q F(l)) ( ) m = ( )( QA) ( QF(l) QF(h) + 2 Q A Q F(h) ( Q F(l) Q, F(l) = QF(h) QF(h) + ( QA), Q F(l) 2 Q A Q F(h) = h We can deduce that if h 0 then α α() This roves (a) Net we establish (b) Assume conditions (i) and (ii) hold Inverting (7) and substituting there (6), we conclude that if β is a Weakly Immune ESS then it is given by eression (7) This rovides the uniqueness of a Weakly Immune ESS It remains to show that β, as defined in (7), is indeed a Weakly Immune ESS We first observe that for all β β, V β (F, α(β )) = V β (F, α(β )) so to conclude it suffices to show that β β, V β (F, α(β)) > V β (F, α(β)) (8) Define H(β) = V β (F, α(β)) V β (F, α(β)) and after some algebras we have the following eression: H(β) = (β β)(q F (l)( )( α(β)) ), (Q F (l) + β )(Q F (l) + β) with = Q F (l) Q F (h) Thus (8) is satisfied if the following two conditions are satisfied: ) β > β, α(β) > ( ) y ( ), 2) β < β, α(β) < ( ) y ( ) After some calculous, we observe that α(β) < ( ) y ( ) β < β Then we deduce that β β we have H(β) > 0 so that (8) holds We conclude that β is indeed a weakly immune ESS, which establishes (b) We finally mention the following useful roerty whose roof is immediate: V β (F, α(γ)) is strictly monotone decreasing in γ (9) We net resent a last result about eistence and unicity of the ESS Theorem 6: For all Q A, Q F (l), Q F (h) and, the ESS β of the stochastic evolutionary game eists and is unique Proof: The eression of the ESS of the stochastic evolutionary game deends on the sign of l and h If both are negative, we have the weakly immune ESS β defined in theorem 5 (a) and is unique If l or h is ositive, we have a strongly immune ESS β = 0 or β = We rove now that we cannot have both l and h ositive We assume that l strictly ositive, that is QF (l) > Q F (h) Now, we have h strictly ositive if and only if min Q A [0,] Q F(h) QF(l) QF(h) ( ) >, 2 Q A Q F(h) Q F(l) QF(l) QF(h) ( ) >, Q F(h) 2 Q A Q F(h) Q F(l) > QF(l) QF(h), Q F(l) l < 0 We then have roved that there is always one weakly immune ESS β or one of the two strongly immune ESS (eclusively) V THE GLOBALLY OPTIMAL SOLUTION AND THE PRICE OF ANARCHY We net comute the cooerative otimal strategy for users and comare it with the ESS β is globally otimal if it maimizes V β (F, α(β)) Substituting (7) into (8) this amounts to maimize with α(β) = Z(β) = V (A) + + β( α(β)) Q F (l) + β(q F (l) Q F (h)), βq A We have Q A+Q F (l)+β(q F (l) Q F (h)) Z (β) = [QF(l) + β(qf(l) QF(h))][( α(β)) β α (β)] (Q F(l) + β(q F(l) Q F(h))) 2 [ + β( α(β))][(qf(l) QF(h))] (Q F(l) + β(q F(l) Q F(h))) 2 In order to find the otimum oint, we have to solve Z (β) = 0, that is Q F (l)( α(β)) β α (β)[q F (l) + β(q F (l) Q F (h))] We introduce the following notations: (Q F (l) Q F (h)) = 0 = Q F (l) Q F (h) Moreover, after some calculus we obtain α (β) = Q A (Q A + Q F (l)) (Q A + Q F (l) + β(q F (l) Q F (h))) 2 > 0 The equation Z (β) = 0 is equivalent to ( α(β))q F (l) = β α (β)q F (l) + β 2 α (β) +,

7 and given the derivative of α we obtain after some algebras the following second order equation: ( ) β 2 Q F(l)( Q A) Q A(Q A + Q F(l)) 3 ( ) +2β Q F(l)(Q A + Q F(l))( Q A) 2 (Q A + Q F(l)) The discriminant of this equation is +(Q F(l) )(QA + QF(l))2 = 0 = 4(Q A + Q F (l)) 2 Q A 2 (QF (l) 2 + Q F (l) 2 ) We observe that deending on Q F (l), Q F (h) and, the sign of the discriminant is changing We shall use the following equivalence: > 0 < u + u, where u = Q F (l)q F (h) (Q F (l) Q F (h)) 2 (20) Theorem 7: If > u +u, where u is given in (20), then the global otimal strategy is Proof: If > u +u β = 0 then after some calculus we obtain: ( ) 2 Q F (l) > Q F(l) + Also the discriminant is strictly negative, then the sign of the derivative is the same for all β [0,] and we have Z (0) = Q F(l) (Q F (l) Q F (h)) Q F (l) 2 Observing that + ( ) 2 imlies leads to Z (0) < 0 Then we conclude that the global otimal strategy is obtain with β = 0 because the function Z is strictly decreasing when > u u QF (l) > +u QF (l) > QF (l) When < +u, the solutions β+ and β of the second order equation are given by and β = Q F(l)( Q A ) + 2 Q A Q F (l)( Q A)+ 2 Q F (l)+q A Q A β + = Q F(l)( z) Q A Q F (l)( Q A)+ 2 Q F (l)+q A Q A,, with = Q F (l) 2 + Q F (l) 2 The global otimal strategy in the cooerative case is then given by the following theorem Theorem 8: Let β and β + be defined as above Then the global otimal strategy β is β = arg ma β {0,β,β +,} Z(β) Proof: The derivative of the global objective function Z is a second order olynomial function deending in β After some algebra, we obtain β and β + as the two roots of this function Then, deending on those values are in the interval [0,] and the sign of Z, the otimal global strategy is either 0, or one of the roots of the function Net, we obtain some relations between the ESS and the globally otimal solution Theorem 9: ESS strategy is more aggressive than the social otimum strategy, ie β β Proof: If β = 0 a strong immune ESS, then we have for all β > 0: V 0 (F, α(0)) > V β (F, α(0) V β (F, α(β), because α(β) is strictly increasing in β Then we have roved that β = 0 is the global otimum,ie β = 0 and β = β If β is weakly immune then β ]0,[ and V 0 (F, α(0)) = V 0 (F, α(β )) = V β (F, α(β )) = V (F, α(β )) > V (F, α()) (the last inequality follows from (9)) From theorem 7, we have: Z (0) = Q F(l) (Q F (l) Q F (h)) Q F (l) 2 = l Q F (l) 2 As β is weakly immune, l 0 and thus Z (0) 0 Moreover, as V 0 (F, α(0)) = V β (F, α(β )), there eists β ]0,β [ which maimizes V over ]0,β [ Assuming that β ]β,[, then there are necessary three local otima, which is not ossible because V is a quadratic function in β and has two local otima Thus, β = β < β VI DYNAMICS We etend the definition of the well known relicator dynamics [9] to the contet of stochastic evolutionary game, and study its convergence to the various ESS obtained in the last section In the biological contet, a relicator dynamics is a differential equation that describes the way strategies change in time as a function of the fitness Roughly seaking they are based on the idea that the average growth rate er individual that uses a given action is roortional to the ecess of fitness of that action with resect to the average fitness In engineering, the relicator dynamics could be viewed as a rule for udating olicies by individuals It is a decentralized rule since it only requires knowing the average utility of the oulation rather than the action of each individual Although each individual terminal i attemts transmission at some distinct times t i n, these times are assumed to be sufficiently variable from one terminal to another so that the udate rule of the whole oulation can be written as

8 the following continuous time differential equation Define β = (β, β) and set for a = h and a = l: [ ] dβ(a) = Kβ(a) V a(f, α(β)) β(b)v b (F, α(β)), (2) := W( β) Note that when summing the above over a we get simly d β(h) + d β(l) = 0 and hence β(h)+ β(l) = at any time, as eected In order to rove the convergence and relation between equilibria ad rest oints of the dynamics, we consider the roerty of ositive correlation Following [6], we define the roerty of ositive correlation or net monotonicity [8] of our dynamic equation dβ = W( β) as V b (F, α(β))w b ( β) > 0 whenever W( β) 0 By a straightforward adatation of the argument in [6] we conclude that the ositive correlation insures that all equilibria of our game are the stationary oints of the relicator dynamics Thus a non-stationary oint of the dynamics cannot be an equilibrium We therefore rove the following: Proosition : The relicator dynamics given by d β = W( β) described in (2) satisfies the roerty of ositive correlation Proof: We have V b (F, α(β))w b ( β) = V b (F, α(β)) d β = ( [ ]) V b (F, α(β)) Kβ(b) V b (F, α(β)) β(b)v b (F, α(β)), ( ) 2 = K β(b)v b (F, α(β)) 2 K β(b)v b (F, α(β)) Using Jensen s inequality, we have that this eression is strictly ositive and thus the relicator dynamics, in this stochastic contet, has the roerty of ositive correlation We show net that the relicator dynamics converges almost globally Theorem 0: For all interior starting oints β 0 ]0,[, the relicator dynamics defined be Equation (2) converges to the ESS β Proof: We rewrite the relicator dynamics for the high ower level as: dβ(h) = Kβ(h)( β(h))[v h (F, α(β)) V l (F, α(β))] Hence the sign of the derivative of the evolution of the h strategy is defined by the sign of V h (F, α(β)) V l (F, α(β)) Q F (l), Notice that V l (F, α(β)) = V 0 (F, α(β)) = V (A) + which does not deend on β and so is constant Then only V h (F, α(β)) = V (F, α(β)) deends on β and we have already roved that α(β) is strictly increasing in β, thus V (F, α(β)) is strictly decreasing in β Then we have just to comare V (F, α(0)) (V (A) + Q F (l) ) and V (F, α()) (V (A)+ Q F (l)) in order to know the sign of the difference for all β ]0,[ First, we have after some calculous: V (F, α(0)) V (A) + Q F (l), V (A) + V (A) + Q F (h) Q F (l), which is equivalent to l 0 Then, we rove that if β = 0, using theorem 4 we have l 0, which imlies that for all β ]0,[, V (F, α(β)) < V (F, α(0)) V (A) + Q F (l), that leads d β(h) < 0 and the relicator dynamics converge to β = 0 Second, after some analog calculus we obtain: V (F, α()) V (A) + Q F (l), A Q ( ) 2 Q V (A) + A Q F (h) Q F (h) V (A) + Q F (l), which is equivalent to h 0, which is a necessary condition to have β = Then, from the same arguments that in the revious case, the relicator dynamics converge here to the ESS, β =, because d β(h) > 0 for all β ]0,[ Finally, if we have a mied ESS, from theorem 5 we have l 0 and h 0, then V (F, α(0)) V (A) + Q F (l), V (F, α()) V (A) + Q F (l) and V (F, α(β )) = V (A) + Q F (l) Thus, for all β ]0,β [, we have V (F, α(β)) > V 0 (F, α(β)), ie d β(h) > 0 And for all β ]β [,, we have dβ(h) V (F, α(β)) < V 0 (F, α(β)), ie < 0 Thus, we have also convergence of the relicator dynamics to the unique ESS β VII NUMERICAL ILLUSTRATIONS In this section we resent several numerical results illustrating theorems of revious sections For all numerical alications we use the following variables: Q A = 05, Q F (l) = 09 and Q F (h) = 07 First, we lot on figures (a) and (b) the valuation function V (β) deending on the strategy β for different value of the robability We observe that the global otimal strategy is equal to β = 0408 and becomes β = 0 as the robability grows from 0 to 04 On figures 2(a) and 2(b) we show both equilibria: noncooerative one β and the global otimum cooerative one β We observe on both figures that the result of the theorem 9, saying that the ESS is always more aggressive than the social otimum, is verified Also, we describe numerically both

9 β β β V(β) (a) = 0 β Fig Cooerative valuation function V(β) β (b) = 04 equilibria deending on the robability on figure 2(a) and we observe that when the robability for a mobile to be the only transmitter is small, the ESS is β = That is the strategy is every one uses high ower because the robability to have cometition with other mobiles is high We observe also the different equilibria deending on the battery quality, that is the difference Q F (l) Q F (h), on figure 2(b) Hence, this difference gives difference in robability to stay in the state F using high or low ower We observe that both equilibria are less aggressive as the difference increases because the robability to stay in state F using low ower becomes very imortant comare to the one using high ower oulations β social otimum β * ESS (a) Deending on the robability Fig 2 oulations β social otimum β * ESS Q F (l) Q F (h) (b) Deending on the difference Q F (l) Q F (h) ESS and global otimum Finally, on figures 3(a), 3(b) and 3(c), we show the convergence of our relicator dynamics with two initial oints β 0 = 09 and β 0 = 0 We consider the three ossible ESS, the two ure and the mied one; we take K = We observe the convergence roerty defined in theorem 0 and we can also have an idea of the seed of convergence (after 5 iterations, the system attains the equilibrium) iterations (a) = 0 Fig iterations (c) = iterations (b) = 025 Convergence of the relicator dynamics VIII CONCLUSIONS AND PERSPECTIVES Biological models and tools have insired a growing number of studies and of designs of decentralized wireless ad-hoc networks In various ways, this aer falls in this category of biology-insired research First, the roblem we have studied, concerning cometitive energy management in wireless terminals, is inherent to biological models as well where animals comete over food A second biology-insired feature of this work is the evolutionary game aradigm that we have adoted and etended Our convergence results concerning the relicator dynamics renders the ESS equilibrium concet more relevant and meaningful than the standard Nash equilibrium, as the ESS is shown to be actually achieved as a limit of a relicator dynamics, and if achieved, this equilibrium oint turns out to be more stable than the Nash equilibrium We have resented a roblem that etends the standard evolutionary games by (i) modeling the ossibility of individuals to take several actions during their lifetime, (ii) allowing these actions to have an imact not only on the instantaneous fitness but also on the future individual s state The state deendence allows in turn to distinguish between individuals In our case, the state of an individual gave an indication on its eected time to live and on its available set of actions We lan to etract the generic features resent in our roblem in order to develo a generic theory of stochastic evolutionary game REFERENCES [] Aelrod, R The Evolution of Cooeration New York: Basic Books, 984 [2] Josef Hofbauer and Karl Sigmund, Evolutionary Games and Poulation Dynamics, Cambridge University Press 998 [3] M Smith, Game Theory and the Evolution of Fighting In John Maynard Smith, On Evolution (Edinburgh: Edinburgh University Press), 8-28, 972 [4] M Smith, Evolution and the Theory of Games, Cambridge University Press, UK, 982 [5] T L Vincent and T L S Vincent, Evolution and control system design, IEEE Control Systems Magazine, Vol 20 No 5, Oct 2000 [6] L Samuelson, Evolution and Game Theory, The journal of Economics Persectives, Vol 6, No 2, 2002 [7] WH Sandholm, Potential games with continuous layer sets, Journal of Economic Theory, vol 97, 200 [8] S Shakkottai, E Altman and A Kumar, The Case for Noncooerative Multihoming of Users to Access Points in IEEE 802 WLANs, roceedings of IEEE Infocom, 2006 [9] JW Weibull, Evolutionary Game Theory, Cambridge, MA: MIT Press, 995 [0] J Hofbauer and K Sigmund, Evolutionary game dynamics, American Mathematical Society, Vol 40, No 4, 2003 [] A I Houston and J M McNamara, Evolutionarily stable strategies in the reeated hawk-dove game, Behavioral Ecology, 99 [2] J M McNamara, The olicy which maimizes long-term survival of an animal faced with the risks of starvation and redation, Advances of Alied Probability, vol 22, 990 [3] J M McNamara, S Merad and E J Collins, The hawk-dove game as an average cost roblem, Advances of Alied Probability, vol 23, 99 [4] V Mhatre and C Rosenberg, Energy and cost otimizations in wireless sensor networks: A survey, Kluwer Academic Publishers, Jan 2004 [5] M Nowak, Five Rules for the Evolution of Cooeration Science, vol 34, 2006 [6] W Sandholm, Evolutionary Imlementation and Congestion Pricing, Review of Economic Studies, vol 69, 2002