Uncoupled automata and pure Nash equilibria

Transcription

1 Int J Gae Theory (200) 39: DOI 0.007/s ORIGINAL PAPER Uncoupled autoata and pure Nash equilibria Yakov Babichenko Accepted: 2 February 200 / Published online: 20 March 200 Springer-Verlag 200 Abstract We study the proble of reaching a pure Nash equilibriu in ulti-person gaes that are repeatedly played, under the assuption of uncoupledness: EVERY player knows only his own payoff function. We consider strategies that can be ipleented by finite-state autoata, and characterize the inial nuber of states needed in order to guarantee that a pure Nash equilibriu is reached in every gae where such an equilibriu exists. Keywords Autoaton Nash equilibriu Uncoupledness Introduction We study the proble of reaching Nash equilibria in ulti-person gaes, where the players play the sae gae repeatedly. The ain assuption, called uncoupledness (see Hart and Mas-Colell 2003), is that every player knows only his own utility function. The resulting play of the gae yields an uncoupled dynaic. Hart and Mas-Colell show in 2003 that if the gae is played in continuous tie, and the oves of every player are deterinistic, then uncoupled dynaics cannot always lead to Nash equilibria. In Hart and Mas-Colell (2006) they show that the situation is different when stochastic oves are allowed and the gae is played in discrete tie: if the players know the history of play, then there are uncoupled strategies that lead to a Nash equilibriu. The question is whether it is necessary to know the whole history in order to reach a Nash equilibriu. The answer is no. It was proved in Hart I.e., the past actions of all the players. Y. Babichenko (B) Center for Study of Rationality and The Institute of Matheatics, The Hebrew University of Jerusale, 9904 Jerusale, Israel e-ail: yak@ath.huji.ac.il

2 484 Y. Babichenko and Mas-Colell (2006), Theores 4 and 5, that under the assuption of uncoupledness, convergence of the long-run epirical distribution of play to a (pure or ixed) Nash equilibriu can be guaranteed by using only the history of the last R periods of play, for soe finite R. This is called a finite-recall strategy. Although finite-recall uncoupled strategies can guarantee convergence of the distribution of play to a Nash equilibriu, it is shown in Hart and Mas-Colell (2006), Theore 6, that this cannot hold for the period-by-period behavior probabilities. If however, instead of finite recall one uses finite eory (e.g., finitely any periods of history but not necessarily the last ones), then the convergence of the behavior can be guaranteed as well (Hart and Mas-Colell 2006, Theore 7). This leads us to the study of uncoupled strategies with finite eory, i.e., finitestate autoata. In this paper, we deal with convergence to pure Nash equilibria in gaes which have such equilibria. In Hart and Mas-Colell (2006), Theore 3, it is shown that in order to guarantee convergence to pure Nash equilibria one needs recall of size R = 2. Since finite recall is a special case of finite autoata, the question we address here concerns the iniu nuber of states required for uncoupled finite autoata to reach a pure Nash equilibriu. There are four classes of finite-state autoata: the actions in every state can be deterinistic (pure) or stochastic (ixed), and the transitions between states can be deterinistic or stochastic. We will analyze each of the four classes in turn. Section 2 presents the odel, defines the relevant concepts and present the total results of the paper. Since the results are different for two-player gaes than for gaes with ore than two players, we consider two-player gaes in Sect. 3 and n-player gaes for n 3 in Sect. 4. In Sects. 3 and 4, we discuss each of the four autoata classes separately. Appendix A and Appendix B containing the proofs of Theores 6 and 7. 2 The odel 2. The gae A basic static (one-shot) gae Ɣ is given in strategic (or noral) for as follows. There are n 2 players, denoted i =, 2,...,n. Each player i has a finite set of pure actions A i =a i,...,ai };leta := A A 2 A n be the set of action cobinations. The payoff function i (or utility function) of player i is a real-valued function u i : A R. The set of ixed (or randoized) actions of player i is the probability siplex over A i, i.e., (A i ) = x i = ( x i( a i )) j j=,..., i : i j= xi (a i j ) = and x i (a i j ) 0for j =,...,i} ; payoff functions u i are ultilinearly extended, and so u i : (A ) (A 2 ) (A n ) R. We fix the set of players n and the action sets A i, and identify a gae by its n-tuple of payoff functions U = (u, u 2,...,u n ).LetU i be the set of payoff functions of player i, and U := U U n. Denote the actions of all the players except player i by a i, i.e., a i = (a j,..., a i j i, a i+ j i+,...,a n j n ), and denote the set of actions of all the players except player i by A i = A A i A i+ A n. An action a i j Ai will be called a best

3 Uncoupled autoata and pure Nash equilibria 485 reply to a i if u i (a i j, a i ) u i (ak i, a i ) for every ak i Ai. A pure Nash equilibriu is an action cobination a = (a j, a 2 j 2,...,a n j n ) A, such that a i j i is a best reply to a i for all i. For every gae U,letŨ = (ũ, ũ 2,...,ũ n ) denote the resulting best-reply gae, which is defined by ũ i, if a (a) = i is a best reply to a i 0, otherwise Note that a is a pure Nash equilibriu of U if and only if it is a pure Nash equilibriu of Ũ. 2.2 The dynaic setup The dynaic setup consists of the repeated play, at discrete-tie periods t =, 2,..., of the static gae U. Leta i (t) A i denote the action of player i at tie t, and put a(t) = (a (t), a 2 (t),..., a n (t)) A for the cobination of actions at t. We assue that there is standard onitoring: at the end of period t each player i observes everyone s action, i.e., a(t); when the choices are rando, the players observe only the realized actions a(t). 2.3 Autoata An autoaton 2 for player i is a 4-tuple i := i, s i 0, f i, g i. i is the set of states; s i 0 i is the starting state; f i : i (A i ) is the action function; and g i : A i ( i ) is the transition function.leta i denote the set of all autoata of player i. An autoaton i A i will be called a pure-action autoaton if the actions in all states are pure, i.e., 3 I( f i ) A i. Otherwise it will be called a ixedaction autoaton. An autoaton i A i will be called a deterinistic-transition autoaton if all the transitions are deterinistic, i.e., I(g i ) i. Otherwise it will be called a stochastic-transition autoaton. An autoaton i A i will be called a k i - autoaton if it has k i states, i.e., i =k i. Let (, 2,..., n ) be n autoata, where i is a k i -autoaton for player i.the play proceeds as follows. At tie t = every player i is at his starting state s i 0, and plays an action a i () according to the probability distribution f i (s i 0 ). Let the realized actions of all the players be a() := (a (),...,a n ()). Then every player i oves to a new state according to the transition probabilities g i (a(), s i 0 ). Now assue that at tie t player i is in state s i i, and hence at tie t + player i plays an action a i (t) according to the probability distribution f i (s i ). The actions of all the players are a(t + ), and every player i then oves to a new state according to the transition probabilities g i (a(t + ), s i ). 2 This is short for a strategy ipleented by an autoaton. 3 We identify A i with the unit vectors in (A i ).

4 486 Y. Babichenko 2.4 Strategy appings Let ϕ : U A A n be a apping that associates to every gae U = (u,...,u n ) U an n-tuple of autoaton strategies ϕ(u) = (ϕ (U),...,ϕ n (U)) (with ϕ i (U) the autoaton of player i). We will call the apping ϕ uncoupled if, for each player i, the ith coordinate ϕ i of ϕ depends only on u i, i.e., ϕ i : U i A i (rather than ϕ i : U A i ). That is, ϕ i associates to each payoff function u i U i of player i an autoaton ϕ i (u i ) A i, and ϕ(u) = ϕ(u, u 2,...,u n ) = (ϕ (u ), ϕ 2 (u 2 ),..., ϕ n (u n )). We will refer to ϕ i : U i A i as an uncoupled strategy apping to autoata for player i; thus, ϕ i constructs an autoaton for player i by considering u i only. 4 If ϕ i (u i ) A i is an autoaton of size (at ost) k i for every payoff function u i U i, we will say that ϕ i is an uncoupled strategy apping to k i -autoata. Finally, we will say that the apping ϕ is a Pure Nash apping, or PN-apping for short, if the strategies ϕ(u) yield alost sure convergence of play to a pure Nash equilibriu in every gae U U where such an equilibriu exists. 2.5 The results Clearly, every finite-recall strategy is in particular a finite-autoaton strategy. Indeed, a strategy with recall of size R can be ipleented by an autoaton of size A R = ( n i= i) R (i.e. one state for each possible recall). Therefore, by Theore 3inHart and Mas-Colell (2006), there is uncoupled PN-apping to autoata of size ( ni= i) 2. The question we address here is whether there is uncoupled PN-apping to autoata with fewer states. Our purpose is thus to characterize inial nubers k,...,k n such that there exists uncoupled PN-apping where, for each i, the range is k i -autoata. We will analyze each of the four cases (pure or ixed-action autoata, and deterinistic or stochastic-transition autoata) separately. The results are the following: For two-player gaes (n = 2): There exists uncoupled PN-apping to autoata of sizes: ixed actions pure actions stochastic transitions deterinistic transitions or or O() O() There is no uncoupled PN-apping to autoata of sizes, 2. 4 We assue that every player i knows his index i.

5 Uncoupled autoata and pure Nash equilibria 487 For n-player gaes (n 3): There exists uncoupled PN-apping to autoata of sizes: ixed actions pure actions stochastic transitions 2 i 2 i deterinistic transitions 2 i + 3 O( i + n log n) Let k, k 2,...,k n such that i =,...,n : k i < 2 i. Then there is no uncoupled PN-apping to autoata of sizes k, k 2,...,k n. 3 Two-player gaes 3. Stochastic transitions and ixed actions We will show that there exists an uncoupled PN-apping where the range for player is( + )-autoata and the range for player 2 is 2 -autoata or, syetrically, the range for player is -autoata and the range for player 2 is ( 2 + )-autoata. On the other hand, we will show that there is no PN-apping where the ranges of the players are saller. Theore Let k and k Then, for each player i =, 2, there exists an uncoupled strategy apping to k i -autoata with stochastic transitions and ixed actions that guarantees alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. Proof We define the apping ϕ as follows: GivenagaeU = (u, u 2 ), the autoaton ϕ (u ) = A is constructed as follows : Denote ϕ (u ) = =, s 0, f, g when is a -autoaton. We denote the states of by =s,...,s }. s 0 :=s. f (si ) :=a i (0,...,0,, i 0,...,0). g (a, si ) = g ((ai, a2 ), si ) s := i (0,...,0, i ( ), 0,...,0),..., if a i is a best reply to a 2 otherwise In state s i player plays action a i. He stays in this state if a i is a best reply to the action of player 2; otherwise he oves randoly to any one of the states with equal probability. Note that whether an action of player is a best reply or not depends only on his payoff function; therefore, depends on u only.

6 488 Y. Babichenko Now we construct the autoaton ϕ 2 (u 2 ) = 2 A 2 as follows: Denote 2 = 2, s 2 0, f 2, g 2 when 2 is a ( 2 + )-autoaton. We denote the states of 2 by 2 =s 2 0, s2,...,s2 }. s 2 0 :=s2 0 (. ) f 2 (s 2 j ) :=,..., 2 2 j = 0 a 2 j j g 2 (a, s 2 j ) = g2 ((a, a 2 j ), s2 j ( ) ) 2 +,..., 2 + j = 0 := s ( 2 j and a j ) 2 j 2 +,..., j and a 2 2 j + is a best reply to a is not a best reply to a ( In the state s0 2, player 2 plays the ixed action,...,, and oves to any of 2 the 2 + states with probability 2 +. In the states si 2, i, player 2 plays action a2 i. He stays in this state if a2 i is a best 2 ) reply to the action of player ; otherwise he oves to the state s 2 0. Now we will prove that (ϕ,ϕ 2 ) is a PN-apping. We partition the space 2 of the autoata states into four regions: P := (si, s2 j ), i, j 2 : ũ (ai, a2 j ) =, ũ2 (ai, a2 j ) = }; i.e., in this case (ai, a2 j ) is a pure Nash equilibriu. P 2 := (si, s2 0 ), i }. P 3 := (si, s2 j ), i, j 2 : ũ 2 (ai, a2 j ) = 0}. P 4 := (si, s2 j ), i, j 2 : ũ (ai, a2 j ) = 0, ũ2 (ai, a2 j ) = }. These four regions clearly cover the space 2. In fact, player 2 can be in the state s0 2 (P 2) or in any other state (P P 3 P 4 ). If player 2 is not in the state s0 2, then the action of player 2 can be a best reply (P P 4 ) or not (P 3 ). If it is a best reply, then the action of player can be a best reply (P ) or not (P 4 )). Players and 2 stay at the sae state if their action is a best reply; i.e., each state in P is absorbing. We will prove that there is a positive probability of reaching a state fro P, in finitely any periods, fro any other state s 2. Definition 2 An action a i j A i of player i will be called doinant if for every a i A i a i j is a best reply to a i. s = (si, s2 0 ) P 2: The actions are ( ai, (,..., )) 2 = ( f (s 2 i ), f 2 (s0 2)). Ifa i is a doinant action, then denote by al 2 an action that is a best reply to ai. Player 2 oves to sl 2 with probability 2 +. Then (s i, s2 l ) P.Ifai is not a doinant action, then denote by ak 2 an action such that a i is not a best reply to it, and then with probability player 2 plays action a 2 2 k. Now both players ove randoly over all their states and with positive probability they will get to P.

7 Uncoupled autoata and pure Nash equilibria 489 s = (si, s2 j ) P 3: The actions are (ai, a2 j ); a2 j is not a best reply. Therefore, player 2 oves to s0 2. Denote the state to which player oves by s k. Then (s k, s2 0 ) P 2. s = (si, s2 j ) P 4: The actions are (ai, a2 j ); a2 j is a best reply, and a i is not. Therefore, player 2 stays in s 2 j, whereas player ove to s k with probability, where a k is a best reply of player to a 2 j. Now either (s k, s2 j ) P or (sk, s2 j ) P 3, depending on whether a 2 j is a best reply to a k. In each of the above cases there is positive probability of reaching an absorbing state in P in at ost 3 steps. Definition 3 AgaeU will be called a full gae if, for every action a i j A i of every player i, there exists a i A i such that a i j is a best reply to a i. We prove a general result about n-player full gaes which will be useful in the sequel. Lea 4 Let ϕ = (ϕ,...,ϕ n ) be an uncoupled strategy apping that guarantees alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. Then for every full gae U = (u,...,u n ) and for every player i, there exist i nonepty sets of states B i,...,bi in i (the i set of states of the autoaton ϕ i (u i ) = i ) such that in every state sk i Bi j player i plays a i j (with probability ), and stays in Bi j (with probability ) if his action is a best reply to the actions of the other players. Proof (By contradiction). Assue that there exists a full gae U s.t. for player the set B j does not exist (or is epty). U is a full gae, and so there exists a A such that a j is a best reply to it. Consider the gae U = (u, u 2,...,u n ) when u := u, ifa = (a and u i (a) := j, a ). The only Nash equilibriu of U is (a 0 otherwise j, a ).By uncoupledness we get = ϕ (u ) = ϕ (u ) =.If(a j, a ) has been played, the next period player will not play ak with probability (otherwise the set B j could not be epty), and the equilibriu in the gae U will never be reached with probability (in contradiction to the assuption). Theore 5 Let k = and k 2 = 2. Then there are no uncoupled strategy appings to k i -autoata with stochastic transitions and ixed actions, that guarantee alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. Proof (By contradiction). Let U be a full gae. Consider the sets B i,...,bi in i i (see Lea 3). By assuption i i. On the one hand, B i j, and, on the other hand, B i j i = i, and so B i j = and B i j j j = i. In other words, every B i j includes exactly one state in which player i plays ai j and stay there if ai j is a

8 490 Y. Babichenko best reply, and there are no other states. Therefore, the strategy of player i is such that if his action is a best reply to the action of the other player, then in the next step he plays the sae action. In Hart and Mas-Colell (2006), Proof of Theore, Hart and Mas-Colell show that such a strategy cannot always lead to a pure Nash equilibriu, contradicting our assuption. 3.2 Stochastic transitions and pure actions We will show the result of Theore continues to hold when the autoata are restricted to be pure-actions autoata. As was shown in Theore 5, however, there is no PN-apping where the ranges of the players are saller. Theore 6 Let k and k Then, for each player i =, 2, there exists an uncoupled strategy apping to k i -autoata with stochastic transitions and pure actions, that guarantees alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. The proof is relegated to Appendix A. 3.3 Deterinistic transitions and ixed actions We will show that there exists an uncoupled PN-apping where the range for player i is ( i + 2)-autoata. Clearly, every deterinistic-transition autoaton is a particular case of stochastic-transition autoaton, and so Theore 5 holds here as well. Theore 7 Let k i i +2. Then for each player i there exists an uncoupled strategy apping to k i -autoata with deterinistic transitions and ixed actions, that guarantee alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. The proof is relegated to Appendix B. 3.4 Deterinistic transitions and pure actions In Theore 3, we will show for general n-player gaes that there exists an uncoupled PN-apping where the range for player i is (O( i + n log n))-autoata, where i is the nuber of actions of player i and n is the nuber of players. In the case of 2 players, the construction of the autoata in the proof of Theore 3 proves the existence of uncoupled PN-apping where the range for player i is (4 i + O())- autoata. Other uncoupled PN-apping, specifically for 2 players, has a range of ( )-autoata for player, and ( )-autoata for player 2. We will not show this construction here but the idea is to go through all the possible actions (ai, a2 j ) in soe econoical way.

9 Uncoupled autoata and pure Nash equilibria 49 4 n-player gaes (n 3) 4. Stochastic transitions, and pure or ixed actions We will show that there exists an uncoupled PN-apping where the range for player i is 2 i -autoata. On the other hand, we will show that there is no uncoupled PN-apping whose range is saller for all players. Theore 8 Let k i 2 i. Then for each player i there exists an uncoupled strategy apping to k i -autoata with stochastic transitions and pure actions that guarantee alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. Proof Let us introduce the appings ϕ i (u i ) = i given a payoff function U = (u,...,u n ). Denote the states of i by i =s,0 i, si,, si 2,0, si 2,...,si i,0, si }. The states i, s i j,0 will be called 0-states; the states si j,, -states. Definition 9 Given a state s = (s,...,s n ) n we will say that player i is fit at s if player i is at a 0-state and player i + (od n) is at a state s i j,k for k 0, } and j =, or player i is at a -state and player i + (od n) is at a state s,k i for k 0, }. In every state s i j,l player i plays action ai j.ifai j is a best reply to what the other players played, and player i fits player i + (od n), player i stays in s i j,l. Otherwise he oves to any one of the 2 i states with equal probability. 2 i Let the starting states be s i 0 := si,0. To prove that these autoata reach a pure Nash equilibriu we partition the space n of the autoata states into n + 2 regions: P := (sk,l,...,sk n n,l n ), k i i, l i = 0, : (ak,...,ak n n ) is a pure Nash equilibriu and all the players are fit}. Note that for every pure Nash equilibriu (ak,...,ak n n ) there is a state s P where the players play (ak,...,ak n n ).Takesk i i,l i with l i = when k i+ = and l i = 0 otherwise. For 0 r n P 2,r := (sk,l,...,sk n n,l n ): there exist exactly r players that are fit}. P 3 := ((sk,l,...,sk n n,l n ): all the players are fit, but (ak,...,ak n n ) is not a pure Nash equilibriu}. Clearly each state in P is absorbing. Next we clai that a state in P is reached with positive probability, in finitely any periods, fro any other state s n. s P 2,0 : all the players are not fit, and so all the players ove randoly over all their states, and there is a positive probability of reaching P.

10 492 Y. Babichenko For r n : s = (sk,l,...,sk n n,l n ) P 2,r. Assue player i is fit, but player 5 i + is not. Such i exist, because we have a circle of players of which soe are fit, and soe are not. There is a positive probability that all the players except i + will stay at their states, and player i + (who oves randoly because he is not fit) will ove in the following way: if k i+ = then he oves to s2,l i+ i+, and if k i+ 2, then he oves to s,l i+ i+. Now all the players except i and i + reain fit/not fit, as they were before, because neither they nor the next player change their state. Player i + does not change his l i+, player i + 2 does not change his state, and so player i + stays not fit, as he was before. Player i was fit but after the ove of player i +, he is not fit. The only player that changes his fitness is player i, and they get to P 2,r. By induction, with positive probability they get to P 2,0 in r steps. s = (sk,l,...,sk n n,l n ) P 3 : The action is a = (ak,...,ak n n ) and it is not a pure Nash equilibriu; therefore, there exists player i s.t. ak i i is not a best reply to a i, and player i oves randoly over all the states. Hence, there is a positive probability that all the players except i will stay at their states and player i will ove to sk i i, l i. Now all the players except player i stay fitted, but player i do not fit. And so (sk,l,...sk i i, l i,...,sk n n,l n ) P 2,n. Fro any state s there is a positive probability of reaching an absorbing state in P in at ost n + 2 steps. Theore 0 Let n 4, and let k,...,k n satisfy k i < 2 i for all i =,...,n, (except, perhaps, for one of the). Then there is no uncoupled strategy apping to k i -autoata with stochastic transitions and ixed actions, that guarantees alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. Proof Assue on the contrary that such a strategy apping exists, and that k i < 2 i for all i = 2,...,n. By Lea (2) in i there exist i nonepty sets of states B i,...,bi s.t. in every state s i j Bk i player i plays ak i. i < 2 i, and by the pigeon hole principle there exists k(i) s.t. Bk(i) i =. Therefore, every player i has a state sk(i) i where he plays ai k(i), and he stays there if it is a best reply. Consider a four-players gae where every player has 2 actions. Consider the following utility function of players 2, 3, 4: u 2 : a 3 a 3 2 a 4 }} a 2 a2 2 a 0 a2 0 a 2 a a 2 a2 4 }} a 2 a2 2 a a2 a 2 a Fro here till the end of the proof, we will write i + instead of i + (od n).

11 Uncoupled autoata and pure Nash equilibria 493 u 3 : a 3 a 3 2 a 4 }} a 2 a2 2 a a2 a 2 a 0 a 2 0 a2 4 }} a 2 a2 2 a 0 a2 0 a 2 a 2 u 4 : a 3 a 3 2 a 4 }} a 2 a2 2 a a2 a 2 a 0 a 2 0 a2 4 }} a 2 a2 2 a 0 a2 0 a 2 a 2 Player i = 2, 3, 4 gets if he plays the sae action as one of the players 2, 3, 4 (except hiself). Otherwise he gets 0. The strategy apping ϕ i : u i A i constructs an autoaton. As entioned, there exists a state sk(i) i where player i plays ai k(i), and he stays there if it is a best reply. There are 2 actions for every player, and 3 actions ak(i) i i = 2, 3, 4. So there exist 2 players i, j whohavethesaeactionak(i) i, a j k( j), where k(i) = k( j). Because of the syetry of the functions u 2, u 3, u 4, assue k(i) = k( j) =, and assue that the two players are players 3 and 4. Let us now consider the following gae: Ɣ : a 3 a 3 2 a 4 }} a 2 a2 2 a, 0,, 0,,, a2 0,,,, 0,, a 2 a 0, 0, 0, 0, 0,, 0 a 2 0, 0, 0, 0, 0,, 0 a2 4 }} a 2 a2 2 a 0, 0,, 0 0, 0, 0, a2 0, 0,, 0 0, 0, 0, a 2 a 2,,,,,,,,,,,, Players 3 and player 4 have the utility functions u 3 and u 4, respectively. Therefore, the autoaton that their strategy apping constructs include states s 3, s4, where they play action a 3, a4, respectively, and stay there if it is a best reply. If players 3 and 4 get to the states sk(3) 3 and s4 k(4), then the pure Nash equilibriu will never be reached. For a larger nuber of actions the sae proof works, if we take all the actions a2 i,...,ai to be identical to the action a i i 2 in this proof.

12 494 Y. Babichenko For a larger nuber of players we take the utility functions of players 2, 3, and 4 to be the sae as in the case of 4 players and independent of the actions of the other player (, 5,...,n).AndinthegaeƔ the utility functions of players 5,...,n will be if players 3 and 4 played the sae action, and 0 otherwise. 4.2 Deterinistic transitions and ixed actions We will show that there exists an uncoupled PN-apping such that the range for player i is (2 i + 3)-autoata. Clearly, every deterinistic-transition autoaton is a particular case of stochastic-transition autoaton, and so Theore 0 holds here as well. Theore Let k i 2 i + 3. Then for each player i there exists an uncoupled strategy apping to k i -autoata with deterinistic transitions and ixed actions, that guarantees alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. Proof We introduce the appings ϕ i (u i ) = i given a payoff function U = (u,...,u n ): As in Theore 8, we will use the states s,0 i, si,, si 2.0, si 2,...,si i,0, si }. These i, are the sae states, exapt that their transitions are deterinistic. Denote the states of i by i =s i, si 2, si 3 } si,0, si,, si 2.0, si 2,...,si i,0, si i, }. The states s i j,l are siilar to the states si j,l in Theore 8. In every state si j,l player i plays action a i j.ifai j is a best reply to what the other players played, and player i fits, then player i stays on it (exactly as ( before). Otherwise he oves to s i. In the state s i player i plays,..., ). If he played a i i i he stays in si.ifhe played a2 i he oves to si 2. If he played ai j j =, 2 he oves to s i ( j,0. In the state s2 i player i plays,..., ). If he played a i i i 3 he stays in si.ifhe played a4 i he oves to si 3. If he played ai j j = 3, 4 he oves to s i ( j,. In the state s3 i player i plays,..., ). If he played a i i i he oves to si,0.ifhe played a2 i he oves to si 2,0. If he played ai 3 he oves to si 3,. If he played ai 4 he oves to s4, i. If he played ai 5 he stays in si 3. If he played ai j j 6, he oves to s i. Let the starting states be s i 0 := si. The proof of the clai that this apping is a PN-apping is proven siilarly to Theore 7 with 2 players. 4.3 Deterinistic transitions and pure actions We will show that there exists an uncoupled PN-apping such that the range for player i is O( i + n log n)-autoata. Clearly, every deterinistic-transition autoaton is a particular case of a stochastic-transition autoaton, and so Theore 0 holds here as well.

13 Uncoupled autoata and pure Nash equilibria 495 Lea 2 For every, n N there exist n different prie nubers p,...,p n, p i for every i, such that p i = O( + n log n) for every i. Proof We know that in, 2,...,} there exist a axiu of α prie nubers (α is a constant). } We also know that in, 2,...,β(n + α ) log(n + α ) there exist a iniu of n + α prie nubers (β is a constant). } Therefore, in +, + 2,...,β(n + α ) log(n + α ) there exist a iniu of n prie( nubers, and ) we( can take different ) prie nubers p,...,p n s.t. i : < p i <β n + α log n + α. ( ) ( ) To coplete the proof we will show that β n + α log n + α = O( + n log n). If n then ( β n+α ) ( log n+α ) = O((+α)n log((+α)n)) = O(n log n) = O( + n log n). Ifn < n log n, then ( log n log n 2n = 2n 2n β n + α ) ( log n + α ) 2log 2 = O(( + 2α)n log(( + 2α)n)) = O(n log n) = O( + n log n). Ifn log n < n 2, then Ifn 2 <, then ( β n+α n log n log n 2 log n 2 ( β n + α ) ( log n + α ) ( ( )) = O (2 + α) (2 + α) ( ) = O ( log ) = O() = O( + n log n). = O ) ( log ( ( log ) n+α ) ( ( )) =O (+α) (+α) ) = O() = O( + n log n).

14 496 Y. Babichenko ( ) ( ) In any case β n + α log n + α = O( + n log n). Theore 3 Let k i O( + n log n), where = ax i }. Then for each player i there exists an uncoupled strategy apping to k i -autoata with deterinistic transitions and pure actions, that guarantees alost sure convergence of play to a pure Nash equilibriu of the stage gae in every gae where such an equilibriu exists. Proof Let p,...,p n be different prie nubers s.t. i p i > i. 6 By Lea 2 we can take p,...,p n s.t. p i = O( + n log n). We will show that there exists a PN-apping (ϕ,...,ϕ n ) such that the range of ϕ i is (4p i + 3)-autoata, and in doing so we will have concluded our proof. We introduce the appings ϕ i (u i ) = i, given a payoff function U = (u,...,u n ): The autoaton i consists of, starting state, a state after that and p i regions. The first region has five states. The other regions have the sae structure of four states. Denote the p i regions of i by Q i =si j,, si j,2,...,si j,5 }, and Qi j =si j,,...,si j,4 } for j = 2,...,p i. Denote the starting state by s i 0 := si and the state after it by si 2. s i : player i plays ai.if(a,...,an ) was played he oves to si, (the starting state of Q i ). Otherwise, if ai is a best reply, then he stays at si ;ifai is not a best reply, then he oves to s2 i. s2 i : player i plays ai 2 and in any case oves to si, (the starting state of Qi ). These two starting states guarantee us two things: First, if the equilibriu is (a,...,an ) then the players will stay there. Second: if it is not, then all the players get to s, i siultaneously. We check the actions (a,...,an ) separately, because in the continuation of the construction of the autoata we will use the fact that (a,...,an ) is not an equilibriu. The regions Q i, Qi 2,...,Qi p i are arranged in a circle when the players ove to the region Q i j+(od p i ) fro the previous region Qi j. For j = the construction of Q i j is the following: The starting state s i j, : player i plays ai j. If player i +(od n) played7 a i+, player i oves to s i j,2 ; otherwise he oves to si j,3.if(a,...,an ) was played, he oves to8 s i j+,. S i j,2 : player i plays ai j. If it is a best reply, and also player i + played ai+ (the sae action as before) then player i stays there. Otherwise he oves to s i j,4.if(a,...,an ) was played, he oves to s i j+,. 6 Every player i has to choose his nuber p i to construct his autoaton. But he doesn t know the nubers of the other players: p,...,p i, p i+,...,p n. Yet we have to ensure that every player will choose a different nuber. Therefore, we have to define a choice function: χ : N n (PRIME) n, known to the players. The choice function chooses for every (,..., n ) n prie nubers: χ(,..., n ), and then every player i will choose the nuber p i := (χ(,..., n )) i. As an exaple of choice function, let w k } k= be the sequence of all the prie nubers in increasing order. wk(i) For every i let k(i) be the inial nuber such that i k(i) = i(od n), and define p i := w k(i). 7 Fro here till the end of the proof, we will write i ± instead of i ± (od n). 8 Fro here till the end of the proof, we will write j + instead of j + (od p i ).

15 Uncoupled autoata and pure Nash equilibria 497 s i j,3 : player i plays ai j. If it is a best reply, and also player i + played ai+ k k 2(the sae action as before), player i stays there. Otherwise he oves to s i j,4.if(a,...,an ) was played, he oves to s i j+,. s i j,4 : player i plays ai.if(a,...,an ) was played, he oves to si j+, ; otherwise he stays at s i j,4. For j = the construction of Q i is quite siilar. The only difference is that the state s i j,4, changed by two si,4, si,5 : s,4 i : player i plays ai 2. In any case, he oves to si,5. s,5 i : player i plays ai.if(a,...,an ) was played, he oves to si j+,. Otherwise he stays in s,5 i. For i < j p i let Q i j := Q i denoted that the act in every s i i j,k i < j p i, k =,...,4, is identical to the act in s i i,k. In the state s i j, player i infors player i what he played. If the players located at Q k,...,q n k n, and (ak,...,ak n n ) is an equilibriu, then all the players will stay at s i j,2, si j,3. In the state s i j,4 player i plays the opposite action than he played before and infors player i that it is not an equilibriu. In the state s i j,4 for j =, and si,5 for j = player i waits until all the are infored that it is not an equilibriu. Then all the players can ove to their the next region siultaneously. To suarize, if the players are located at Q k,...,q n k n, and (ak,...,ak n n ) is an equilibriu, then all the players stay at the equilibriu all the tie. Otherwise they all will ove to Q k +,...,Qn k n + siultaneously. Since the players ove fro Q i j to Qi j+ siultaneously, by the Chinese Reainder theore it follows that they will visit every (Q k,...,q n k n ): k i i, until they becoe stuck in soe (Q k,...,q n k n ) for which (ak,...,ak n n ) is an equilibriu. Therefore, if a pure equilibriu exists, the autoata will eventually reach it. Appendix A. Proof of Theore 6 Proof We define the apping ϕ as follows: GivenagaeU = (u, u 2 ), the autoaton ϕ (u ) = A is constructed as follows: Denote ϕ (u ) = =, s 0, f, g when is a -autoaton. We denote the states of by =s,...,s }. s 0 := s. In state si player plays action ai. He stays in this state if a i is a best reply to the action of player 2; otherwise he oves randoly to any one of the states with equal probability. In order to define the apping ϕ 2, we start by considering the following action b 2 of player 2: for every action a 2 j of player 2, let #BR(a2 j ) be the nuber of s in the colun

16 498 Y. Babichenko a 2 j in the table ũ2 ; i.e., #BR(a 2 j ) := a i ũ2 (ai, a2 j ) = } = a i a2 j is a best reply to ai }. Consider an action a2 k with a axial nuber of s in its colun: #BR(a2 k ) = ax#br(a 2 j ) a2 j A 2 }. Without loss of generality assue k = ; i.e., the first colun of ũ 2 has no fewer s than any other colun. Denote this action by b 2 := a 2. Now we construct the autoaton ϕ 2 (u 2 ) = 2 A 2 as follows: Denote 2 =< 2, s 2 0, f 2, g 2 > when 2 is a ( 2 + )-autoaton. We denote the states of 2 by 2 =s0 2, s2,...,s2 }. s 2 0 := s2 0. In the state s0 2, player 2 plays the action b2, and oves to any of the 2 + states with probability 2 +. In the states si 2, i, player 2 plays action a2 i. He stays in this state if a2 i is a best reply to the action of player ; otherwise he oves randoly to any one of the 2 + states with equal probability 2 +. Now we will prove that (ϕ,ϕ 2 ) is PN-apping. Let us consider two cases: Case For every i =,..., : ũ 2 (a i, b2 ) = ; i.e., b 2 is a doinant action. We partition the space 2 of the autoata states into four regions: P := (si, s2 j ), i, j 2 : ũ (ai, a2 j ) =, ũ2 (ai, a2 j ) = }; i.e., in this case (ai, a2 j ) is a pure Nash equilibriu. P 2 := (si, s2 0 ), i } P 3 := (si, s2 j ), i, j 2 : ũ 2 (ai, a2 j ) = 0} P 4 := (si, s2 j ), i, j 2 : ũ (ai, a2 j ) = 0, ũ2 (ai, a2 j ) = } These four regions clearly cover the space 2 (because player 2 can be in the state s0 2 (P 2) or in any other state (P P 3 P 4 ). The action of player 2 can be a best reply (P P 4 )ornot(p 3 ). If it is a best reply, then the action of player can be a best reply (P ) or not (P 4 ).) Players and 2 stay at the sae state if their action is a best reply; i.e., each state in P is absorbing. We will prove that there is a positive probability of reaching a state fro P, in finitely any periods, fro any other state s 2. s = (si, s2 0 ) P 2: The actions are (ai, b2 ) = ( f (si ), f 2 (s0 2)). Wether a i is a best reply or not, player has a positive probability ( or correspondingly) to ove to sk, where a k is a best reply of player to b2. Player 2 will ove to the state s 2 with probability 2 +, where b2 is a best reply to every action of player, in particular to ak.now(s k, s2 ) P. s = (si, s2 j ) P 3: The actions are (ai, a2 j ); a2 j is not a best reply. Therefore, player 2 oves to s0 2 with probability 2 +. Denote the state to which player oves by s k. Then (sk, s2 0 ) P 2. s = (si, s2 j ) P 4: The actions are (ai, a2 j ); a2 j is a best reply, a i is not. Therefore, with probability is a best player 2 stays in s 2 j, and player ove to s k, where a k reply of player to a 2 j. Now either (s k, s2 j ) P or (sk, s2 j ) P 3, depending on whether a 2 j is a best reply to a k.

17 Uncoupled autoata and pure Nash equilibria 499 In all the cases there is a positive probability of reaching an absorbing state in P in at ost 3 steps. Case 2 There exists an action of player, say a l, such that b2 is not a best reply to it; i.e., there exists l =,..., such that ũ 2 (a l, b2 ) = 0. Before proving the Theore, we will prove the following siple clai about the configuration of 0s and s in ũ 2. This clai will be useful later. Clai If there exists l =,...,, such that ũ 2 (a l, b2 ) = 0, then: (a) Let a 2 j A2. Then there exists ai A such that ũ 2 (ai, a2 j ) = 0. (b) Let ai A. Then there exists a 2 j A2 such that ũ 2 (ai, a2 j ) =. (c) Let ai A, a 2 j A 2, such that such that ũ2 (ak, a2 j ) = 0 ũ 2 (ak, b2 ) =. ũ2 (a i, a2 j ) = ũ 2 (a i, b2 ) = 0. Then there exists a k A Proof (a) Otherwise there would be a colun in ũ 2 that includes s only. By the assuption in this case there exist i =,...,, such that ũ 2 (ai, b2 ) = 0, which contradicts the fact that b 2 is the colun with the axial nuber of s. (b) There is soe action a 2 j A 2 that is a best reply to the action ai A.This action satisfies ũ 2 (ai, a2 j ) =. (c) Otherwise the nuber of s in the j-th colun would be bigger than in the first colun. We partition the space 2 of the autoata states into regions: For every x, y, z 0, } put: and P xyz := (s i, s2 j ), i, j 2 : ũ (a i, a2 j ) = x, ũ2 (a i, a2 j ) = y, ũ 2 (a i, b2 ) = z} Q xy := (s i, s2 0 ), i : ũ (a i, b2 ) = x, ũ 2 (a i, b2 ) = y}. In words Q for a region we ean that player 2 is in the state s0 2, and by P we ean that he is not. There are three indices for region P, and two Boolean indices for region Q. The first index in P, Q corresponds to player and indicates whether his action is a best reply () or not (0). The second index in P, Q corresponds siilarly to player 2. The third index in P indicates whether action b 2 is a best reply to the action that player played () or not (0). Clearly ( P xyz) = ( 2 \s 2 x,y,z 0,} 0 }) and ( Q xy) = s 2 x,y 0,} 0 }, since we are considering all the possibilities. Therefore, ( P xyz) ( Q xy) = x,y,z 0,} x,y 0,} 2.

18 500 Y. Babichenko Consider the region P. By the defn of P xyz we can see that P =(si, s2 j ), i, j 2 : ũ (ai, a2 j ) = ũ2 (ai, a2 j ) = } =(s i, s2 j ), i, j 2 : (ai, a2 j ) is a pure Nash equilibriu}. Players and 2 stay at the sae state if their action is a best reply; i.e., each state in P is absorbing. If we can show that a state fro P is reached with positive probability, in finitely any periods, fro all the other regions that we have already defined, then we will have concluded our proof. P 00 : This is the region of all the states where the actions of both players are not a best reply, and so both players ove randoly over all their states, therefore, they reach a pure Nash equilibriu with probability 2 + ; i.e., they reach P. Q 0 : Player s action is not a best reply, and player 2 is in state s0 2. Therefore, as before both players ove randoly over all their states, and they reach P with probability, as before. 2 + Q : Player s action is a best reply and so he stays in the sae state. Player 2 is in state s0 2 and so he oves randoly over all his states. Player 2 will ove to s2 with probability 2 +, and they get to P. P 0 : Player stays, and player 2 randoizes. Since ũ 2 (ai, b2 ) = player 2 will ove to s0 2 with probability 2 +, and they get to Q 0 or Q. P 00 : Player randoizes and player 2 stays. By clai (c) there exists ak A such that ũ2 (ak, a2 j ) = 0 ũ 2 (ak, b2 ) =. Player will ove to s k get to P 00 or P 0. P 00 : Player stays and player 2 randoizes. By clai (b) there exists a 2 j A 2 with probability, and they such that ũ 2 (ai, a2 j ) =. Player 2 will ove to s2 j with probability 2 +. Note that a i does not change and so still ũ 2 (ai, b2 ) = 0 and they get to P 0,,0 or P 0 (i.e., P 0 ). P 0 : Player randoizes and player 2 stays. By clai (a) there exists ak A such that ũ 2 (ak, a2 j ) = 0. Player will ove to s k with probability, and they get to P 00 P 00 P 0 = P 0. Q 0 : Player stays and player 2 randoizes. Player 2 oves to s 2 with probability 2 +, and they get to P 00. Thus, we have covered all the regions and shown that in at ost 5 periods there is a positive probability of reaching the absorbing state P. The regions cover all the space 2, and so the autoata will reach a pure Nash equilibriu when such an equilibriu exists with probability. Appendix B. Proof of Theore 7 Proof We define the apping ϕ as follows: GivenagaeU = (u, u 2 ), the autoaton ϕ (u ) = A is constructed as follows: Denote the states of i by i =s00 i, si 0, si,...,si }. In the state s i j player i plays action ai j. He stays in this state if ai j is the best reply to the action of player 2; otherwise he oves to s00 i.

19 Uncoupled autoata and pure Nash equilibria 50 ( In the state s00 i player i plays,..., ). If he played a i i i, he stays in si 00.Ifhe played a2 i, he oves to si 0. If he ( played ai j, j 3, he oves to si j. In the state s0 i player i plays,..., ). If he played a i i i, he oves to si.ifhe played a2 i, he oves to si 2. If he played ai 3, he stays in si 0. If he played ai j j 4, he oves to s00 i. Let the starting state be s i 0 := si 00. The proof that these apping is PN-apping requires consideration of four cases separate: if a pure Nash equilibriu is (ak, a2 l ) then we will consider the cases k 2, l 2}, k 2, l > 2}, k > 2, l 2}, and k > 2, l > 2}. We will prove only ona case, says k 2, l > 2}, since the proofs for the other cases is siilar. We partition the space 2 of the autoata states into regions: P := (si, s2 j ), i, j 2 : (ai, a2 j ) is a pure Nash equilibriu} P 2 := (s0, s2 00 )} P 3 := (s00, s2 00 ) (s 00, s2 0 ) (s 0, s2 0 )} For every x 00, 0}, y,>} put: Q x,y := (sx, s2 j ), j 2 : there exist best reply to ai } For every x,>}, y 00, 0} put: Q x,y := (si, s2 y ), i : there exist i 2ify is i > 2ify is > such that a2 j is not a j 2ifx is j > 2ifx is > such that a i is not a best reply to a 2 j } P 4 := (si, s2 j ), i, j 2 : (ai, a2 j ) is not a pure Nash equilibriu} Clearly, P 2 P 3 =s00, s 0 } s2 00, s2 0 }, P P 4 =s,...,s } s 2,...,s2 }, 2 Q x,y =s x 00,0},y,>} 00, s 0 } s2,...,s2 }, Q 2 x,y =s x,>},y 00,0},..., s } s 2 00, s2 0 }. Therefore, the union of all the regions is 2. Players and 2 stay at the sae state if their action is a best reply; i.e., each state in P is absorbing. If we can show that a state fro P is reached with positive probability, in finitely any periods, fro all the other regions that we have already defined, then we will have concluded our proof. P 2 : Both players randoize. They play (ak, a2 l ) with positive probability, and then they get to P. P 3 : Both players randoize. Fro every one of the states there is a positive probability to get to P 2. For exaple, if they are in the states (s00, s2 00 ), then if they play (a2, a2 ) they ove to (s 0, s2 00 ) P 2. Q 00, : Player randoizes. With positive probability the actions are (ai, a2 j ) such that a 2 j is not a best reply to a i, so player stays at s 00, s 0 } and the action of player 2 is not a best reply. Therefore, player 2 oves to s 2 00, and so (s 00, s2 00 ), (s 0, s2 00 ) P 2 P 3. Q 0, : Player randoizes. With positive probability the actions are (a 4, a2 j ).If a 2 j is a best reply to a 4, they ove to (s 00, s2 j ) Q 00,. Otherwise they ove to (s 00, s2 00 ).

20 502 Y. Babichenko Q 0,>, Q 00,> : Siilar to Q 00,, Q 0, Q,00, Q,0, Q >,0, Q >,00 : Syetric to Q 00,, Q 0,, Q 0,>, Q 00,> P 4 : The action of one of the players is not a best reply to the action of the other, and so one of the will ove to the state s 00 (i.e., s00 or s2 00 ). Hence, they get to one of the previous regions. We have covered all the regions and shown that in at ost 5 periods there is positive probability of reaching the absorbing state P. The regions cover all the space 2 ; therefore, the apping is PN-apping. In the other cases where the pure Nash equilibriu (ak, a2 l ) satisfies k 2, l 2}, k > 2, l 2}, or k > 2, l > 2} the only difference is in how the regions P 2 and P 3 are defined. For exaple, for the case k 2, l 2}, P 2 and P 3 will be defined by P 2 := (s0, s2 0 )} and P 3 := (s00, s2 00 ) (s 00, s2 0 ) (s 0, s2 00 )}. Acknowledgeents discussions I wish to thank Sergiu Hart for his support and guidance, and Noa Nisan for helpful References Hart S, Mas-Colell A (2003) Uncoupled dynaics do not lead to nash equilibriu. A Econ Rev 93: Hart S, Mas-Colell A (2006) Stochastic uncoupled dynaics and nash equilibriu. Gaes Econ Behav 57: