Regret Matching with Finite Memory

Transcription

1 Regret Matching with Finite Meory Rene Saran Maastricht University, Maastricht, The Netherlands Roberto Serrano Brown University, Providence, U.S.A. IMDEA Social Sciences Institute, Madrid, Spain This version: May 2011 Abstract We consider the regret atching process with finite eory. For general gaes in noral for, it is shown that any recurrent class of the dynaics ust be such that the action profiles that appear in it constitute a closed set under the sae or better reply correspondence (CUSOBR set) that does not contain a saller product set that is closed under sae or better replies, i.e., a saller PCUSOBR set. Two characterizations of the recurrent classes are offered. First, for the class of weakly acyclic gaes under better replies, each recurrent class is onoorphic and corresponds to each pure Nash equilibriu. Second, for a odified process with rando sapling, if the saple size is sufficiently sall with respect to the eory bound, the recurrent classes consist of action profiles that are inial PCUSOBR sets. Our results are used in a robust exaple that shows that the liiting epirical distribution of play can be arbitrarily far fro correlated equilibria for any large but finite choice of the eory bound. Keywords: Regret Matching; Nash Equilibria; Closed Sets under Sae or Better Replies; Correlated Equilibria. JEL: C72; C73; D83. We thank Willia Sandhol and an anonyous referee for very helpful coents. We also thank Antonio Cabrales, Sergiu Hart, David Levine, Andreu Mas-Colell, Karl Schlag, Andriy Zapechelnyuk and audience at Fall 2010 Midwest Econoic Theory Meeting (Madison) for their coents and encourageent. Eail address: r.saran@aastrichtuniversity.nl; Tel: ; Fax: Eail address: roberto serrano@brown.edu 1

2 1 Introduction We consider the regret atching process based on finite eory. Each player reebers the last action profiles used in the gae. Regret is calculated with respect to the average payoff obtained in those periods. With respect to the last action chosen, the player calculates his or her regret fro not having used other actions, when those actions replace the last action each tie it was used in the periods that the player recalls. The player switches with positive probability to those actions associated with positive regret, but continues to play the sae action also with positive probability. This process corresponds to the regret atching of Hart and Mas-Colell (2000), except that our players eory is not unbounded. A typical state of the -period eory regret learning process is a list of action profiles. It is shown that any recurrent class of the dynaics ust be such that the action profiles that appear in it constitute a closed set under the sae or better reply correspondence (CUSOBR set) that does not contain a saller product set that is closed under sae or better replies, i.e., a saller PCUSOBR set. We shall refer to such a set as an ω-set, for short. We discuss below the relationships between our CUSOBR sets, PCUSOBR sets and ω-sets and previous concepts in the literature, such as closed sets under rational behavior (curb sets) or the set of rationalizable action profiles. The investigation of all the properties of ω-sets is beyond our scope here, but they see to be sets that are distinct fro the previous solution concepts entioned. Since our first finding is only a necessary condition, in general gaes we are not able to offer a characterization of the recurrent classes of the -period eory regret learning process. However, we offer two possible ways out that yield a characterization. First, for the class of weakly acyclic gaes under better replies, each recurrent class is onoorphic and corresponds to each pure Nash equilibriu of the gae. Second, for a odified process in which agents saple at rando fro their bounded eory, if the saple size is sufficiently sall with respect to the eory bound, the recurrent classes consist of action profiles that are inial PCUSOBR sets. Foster and Vohra (1997) and Fudenberg and Levine (1999) obtain learning procedures for which the epirical distribution of play converges to the set of correlated equilibria. These procedures are coplex, requiring sophisticated updating of beliefs. Hart and Mas-Colell s regret atching was the first particularly siple and intuitive 2

3 procedure leading to the set of correlated equilibria in all gaes, a striking result. It turns out, however, that the regret atching process with finite eory ay give a long-run prediction far fro the set of correlated equilibria, in the following sense. We use our first results to show that for any finite the eory bound, there exists a gae such that the liiting epirical distribution of play concentrates with probability arbitrarily close to 1 on an action profile that is not in the support of the gae s unique correlated equilibriu. Thus, the ain result in Hart and Mas-Colell (2000) sees to depend crucially on the unbounded eory assuption. Nevertheless, we do not know whether our finding will be robust to the opposite order of liits. That is, we do not know whether, for a given gae, the liit as of the epirical distribution of play in the corresponding liiting process will converge to the set of correlated equilibria. This reains an iportant open question. On the other hand, our findings are robust to the eory bounds being player dependent. 1.1 Related Literature Our process is part of the no-regret learning literature (e.g., Hannan (1957), Fudenberg and Levine (1995), Foster and Vohra (1998), Hart and Mas-Colell (2000)). 1 related processes, Young (1993) shows that if players have bounded recall and play a yopic best reply to a saple drawn fro their eory, where the saple is sufficiently sall copared to the eory, then in gaes that are weakly acyclic (under single best reply ), per-period play converges alost surely to a pure Nash equilibriu. 2 Young (1998) proves that this learning dynaics converges alost surely to a inial curb set for generic finite N-player gaes. In contrast, we show that if the players calculate their regrets with respect to a randoly drawn saple fro their eory and the saple size is sall enough, then the regret atching dynaics converges alost surely in any finite N-player gae to a inial PCUSOBR set. Siilar connections between liits of various learning dynaics and set-valued solution concepts have been discovered in the literature. For instance, Hurkens (1995) provides a learning dynaics that converges alost surely to a inial curb set in all finite N-player gaes. In Hurkens dynaics, players have bounded recall of periods and play yopic best replies to their beliefs, where the belief of player i about 1 See Fudenberg and Levine (1998), Hart (2005), Young (2004) or Sandhol (2009) for surveys of learning and related areas. 2 Single eans that only one player is allowed to change his or her action at a tie. In 3

4 player j is any distribution with its support in the set of actions played by player j during the last periods. Instead of yopic best reply, players in Josephson and Matros (2004) use an iitation dynaics. That is, players have bounded recall, and out of her eory, each player saples all past actions and the corresponding payoffs. She then plays the action that had the highest average payoff in her saple. The recurrent classes of this dynaics in all finite N-player gaes are onoorphic states and the ain result is that the set of stochastically stable onoorphic states is a union of sets that are inial closed sets under single better replies. Ritzberger and Weibull (1995) prove that the face of a product set (i.e., set of all ixed strategies with support in the set) is asyptotically stable under any sign-preserving selection dynaics (in continuous tie) if and only if the set is closed under weakly better replies. Each such set always contains an essential coponent of Nash equilibria that is strategically stable. There are other better-reply dynaics that yield convergence to pure Nash equilibria in soe gaes. Young (2004) considers a better-reply process with finite eory and inertia, in which each player repeats her last action with an exogenous probability, and with the rest of the probability, she chooses an action according to soe distribution over the set of actions with positive unconditional regret over the finite nuber of periods she reebers. Young (2004) proves that the better-reply process converges alost surely in per-period play to a pure Nash equilibriu in gaes that are weakly acyclic under single-better replies. Friedan and Mezzetti (2001) obtain the sae convergence result for their better-reply dynaics in which a randoly selected player randoly saples her other actions and switches to the sapled action if it is a better reply to the status quo action profile. Marden et al (2007) study a regret based dynaics with fading instead of finite eory and inertia. That is, with a positive probability each player repeats her last period s action and with the rest of the probability she updates her action as a function of her unconditional regrets where past regrets are exponentially discounted. Their result is that if players use this learning rule, then in gaes that are weakly acyclic under single-better replies and in which no player is indifferent between distinct strategies, per-period play converges to a pure Nash equilibriu alost surely. We argue in Section 4 that any gae that is weakly acyclic under single-better replies is also weakly acyclic under better replies but the converse is not true. Thus, for regret atching with bounded recall, we provide the result of convergence to pure Nash equilibria for a larger class 4

5 of gaes. A couple of other papers in the literature show that evolutionary dynaics need not converge to the set of correlated equilibriu distributions. Viossat (2007) gives an exaple of a 4 4 gae in which fro an open set of initial conditions, the single action profile in the support of the gae s unique correlated equilibriu is eliinated under the replicator dynaics. Viossat (2008) generalizes this result to an open set of gaes and and any other deterinistic evolutionary dynaics defined on continuous state spaces. However, the unique correlated equilibriu in these gaes is a strict Nash equilibriu and hence reains asyptotically stable under those evolutionary dynaics. In contrast, we have a stochastic dynaics on a discrete state space. As a result, we are able to show that for any initial condition, the liiting epirical distribution of play is concentrated on an action profile that is not in the support of the gae s unique correlated equilibriu distribution. In Zapechelnyuk (2008), an agent is playing against nature. The agent has recall and her adaptive behavior is a function of her unconditional regrets over the last periods. He assues that the agent plays according to a better-reply rule, which is defined by the following weak requireent: whenever there exists an action with positive unconditional regret, the agent does not play any action with non-positive unconditional regrets. Unconditional regret atching of Hart and Mas-Colell (2000) is a particular better-reply rule. He provides a 2 3 gae exaple, where the agent is the row player and nature is the colun player. He assues that nature plays according to fictitious play with recall, i.e., in every period, it plays a best reply to the agent s average play over the last periods. Under this assuption, he proves that for any better-reply rule and for any large enough recall, there exists an initial history and period T such that for all t T, the probability that the agent s axiu unconditional regret over the last periods is bounded away fro 0 is bounded below by a positive constant. That is, any better-reply rule of the agent with large enough bounded recall is not universally consistent with nature s strategy. Apart fro adaptive play being a function of unconditional regrets, the difference with respect to our exaple below is that nature s adaptive behavior (although a better-reply rule) is not the sae as the agent s. In related work to Zapechelnyuk s, Lehrer and Solan (2009) find an adaptive rule with bounded recall that converges to the set of correlated equilibria by restarting the eory. 5

6 1.2 Plan of the Paper The rest of the paper is organized as follows. Section 2 describes regret atching with finite eory. Section 3 defines CUSOBR, PCUSOBR and ω-sets. We provide the results in Section 4. In Section 5, we discuss the connections with Hart and Mas-Colell (2000). Finally, Section 6 collects the proofs. 2 Regret Matching with Finite Meory Consider a N-person gae in noral for G, with a finite set of actions A i for each player i N (we use N to denote both the set and the nuber of players). Call A = i N A i. Let π i (a i, a i ) be player i s payoff when she chooses a i and the other players choose a i. Suppose that the players reeber the last 1 action profiles. At the beginning of period t + 1, where t, let (a t +1,..., a t ) be the history of action profiles played during the last periods (the initial history, when t <, is fored arbitrarily). Player i s average payoff over these periods is given by Π i = 1 t π i (a k ). k=t +1 Let a t i be the action played by player i in period t. For all a i a t i, let Π i (a i) be the average payoff over the last periods that player i would have obtained had she played action a i every tie she played action a t i during the last periods. That is, Π i (a i) = 1 t υi k (a i), where k=t +1 υ k i (a i) = { π i (a i, a k i) if a k i = a t i π i (a k i, a k i) if a k i a t i. Define R i (a i) = Π i (a i) Π i. Then, player i switches to action a i in period t + 1 with probability q(r i (a i)) > 0 if and only if R i (a i) > 0, whereas she does not switch with the rest of probability, which we assue is positive, i.e., q(r i (a i)) < 1. 3 This adaptive behavior is regret atching à la Hart and Mas-Colell (2000) but with finite a i at i 3 Several different specifications of q( ) are possible. For instance, let A be the axiu nuber ( ( of actions that any player has and = ax i ax (ai,a i) axa π i(a i i, a i) π i (a i, a i ) )). Then any q( ) such that q(r) [ ) 1 0, A and q(r) > 0 r > 0 will fulfill these properties. In our analysis, we fix q( ) to be one such function. (1) 6

7 recall. 4 Define a state in a period to be the history of last action profiles. Hence, the set of states is H = A. Given G, for fixed q( ), regret atching with bounded recall describes an aperiodic Markov process M G (q) on the state space H. We identify its recurrent classes. A recurrent class is a set of states such that if the process reaches one of the, it will never leave the set, and such that it does not adit a proper subset of states with the sae property. The recurrent classes of M G (q) are closely related to the sets of action profiles in G that are closed under sae or better replies of the players. We define these sets in the next section. 3 CUSOBR, PCUSOBR and ω-sets For any (a i, a i ) A, the set of sae-or-better replies for player i is B i (a i, a i ) = {a i A i either a i = a i or π i (a i, a i ) > π i (a i, a i )}. Let B G : A A be the sae-or-better-reply correspondence of the gae G, i.e., B G (a 1,..., a N ) = i N B i (a i, a i ). Definition 3.1. A set of action profiles  A in G is closed under sae-or-better replies (CUSOBR set) if for all (a 1,..., a N ) Â, we have B G(a 1,..., a N ) Â. A inial CUSOBR set is a CUSOBR set that does not contain a proper subset that is a CUSOBR set. For any nonepty  A, define B G (Â) = (a 1,...,a N )  ( ) B i (a i, a i ). 4 Our specification of the switching probabilities is ore general as it allows for several signpreserving functional fors. It is only in Section 5, where we copare our results with Hart and Mas-Colell (2000), that we consider the particular proportional functional representation used by Hart and Mas-Colell (2000). i N 7

8 Equivalently,  is a CUSOBR set if and only if  is a fixed point of B G, i.e., B G (Â) = Â. Sets that are closed under weakly better replies (Ritzberger and Weibull (1995)) are CUSOBR sets. However, the converse is not true, i.e., a CUSOBR set need not be closed under weakly better replies since a player can have weakly better replies that have the sae payoff and hence, do not belong to the set of sae-or-better replies. For the sae reason, CUSOBR sets are not closed under rational behavior (curb) sets (Basu and Weibull (1991)). Definition 3.2.  A is a product set of action profiles that is closed under saeor-better replies (PCUSOBR set) if  is a product set, i.e.,  = i N Âi, where = Âi A i, i N, and for all (a 1,..., a N ) Â, we have B G(a 1,..., a N ) Â. A inial PCUSOBR set is a PCUSOBR set that does not contain a proper subset that is a PCUSOBR set. For any nonepty  A, define ˆB G (Â) = i N (a i,a i )  B i (a i, a i ). Note that  is a PCUSOBR set if and only if  is a fixed point of ˆB G, i.e., ˆB G (Â) = Â. Since G has a finite nuber of action profiles, there exist both inial CUSOBR and inial PCUSOBR sets in G. It is also easy to see that (a 1,..., a N ) is a pure Nash equilibriu of G if and only if {(a 1,..., a N )} is a singleton inial CUSOBR set and a singleton inial PCUSOBR set. Every inial CUSOBR set that is a product set is a inial PCUSOBR set. Moreover, every inial PCUSOBR set contains a inial CUSOBR set. Hence, the set of inial CUSOBR sets and inial PCUSOBR sets coincide in gaes where all inial CUSOBR sets are product sets. However, in soe gaes, the set of inial CUSOBR sets is a refineent of the set of inial PCUSOBR sets. Gae (a) in Figure 1 has a unique inial CUSOBR set {(U, X), (M, X), (M, Y ), (D, X), (D, Y )}, which is a refineent of its unique inial PCUSOBR set that is equal to the set of all action profiles. On the other hand, it is also possible that there exists a inial CUSOBR set that is not a subset of any inial PCUSOBR set of the gae. For exaple, Gae (b) in Figure 1 has a unique inial PCUSOBR set {(Û, Z)} but it has two inial CUSOBR sets, {(Û, Z)} and {(U, X), (M, X), (M, Y ), (D, X), (D, Y )}. 8

9 X Y U 1, 0 0, 0 M 0, 1 2, 0 D 2, 0 0, 1 (a) Û X Y Z 0, 0 0, 0 1, 1 U 1, 0 0, 1 0, 0 M 0, 1 2, 0 0, 0 D 2, 0 0, 1 0, 0 (b) Figure 1 X Y Z U 1, 3 3, 1 2, 2.5 D 2, 0 3, 2 1, 2 (c) We introduce one ore class of sets before we present our results. Definition 3.3. A set of action profiles  A in G is an ω-set if  is a CUSOBR set that does not contain a proper subset that is a PCUSOBR set. Clearly, every inial CUSOBR set or inial PCUSOBR set is an ω-set. Thus, every gae has at least one ω-set. It is also possible that a gae has an ω-set that is neither a inial CUSOBR set nor a inial PCUSOBR set (see Exaple 4.3). Every singleton ω-set is a singleton inial CUSOBR set and a singleton inial PCUSOBR set. Hence, a pure Nash equilibriu is a singleton ω-set and vice versa. Thus, the set of pure Nash equilibria of any gae G lies in the intersection of the set of its inial CUSOBR sets and the set of its inial PCUSOBR sets, and this intersection in turn lies in the set of its ω-sets. In contrast, a pure Nash equilibriu is a curb set or a set that is closed under weakly better replies only if it is strict. Another solution concept that is weaker than Nash equilibriu is that of rationalizability (Bernhei (1984) and Pearce (1984)). However, there is no logical relation between the sets of actions of a player that are supported in inial CUSOBR sets, inial PCUSOBR sets and ω-sets, and the set of her rationalizable actions. Consider Gae (a) in Figure 1. It has two ω-sets, one equal to its unique inial CUSOBR set and the other equal to its unique inial PCUSOBR set. Thus, action U of the row player is supported in the gae s unique inial CUSOBR set, unique inial PCUSOBR set and both ω-sets. However, U is not rationalizable for the row player since it is strictly doinated by the ixed strategy in which the row player plays M with probability 0.25 and D with probability On the other hand, consider Gae (c) in Figure 1. This gae has a unique ω-set (and hence, unique inial CUSOBR set and unique inial PCUSOBR set) equal to {(D, Y )}. However, all actions of both players are rationalizable in this gae. Since action Y of the colun 9

10 player is weakly doinated by action Z, this exaple further shows that the sets of actions of a player that are supported in inial CUSOBR sets, inial PCUSOBR sets and ω-sets need not intersect with the set of her undoinated actions. As a final exaple, consider the gae in Figure 2, which was originally given by Shapley (1964) to prove that fictitious play need not converge to Nash equilibriu. Young (1998) calls it the Fashion Gae. To give his description, let X, Y and Z stand for red, yellow and blue, respectively. The row player is a fashion follower who likes to iitate the colun player. On the other hand, the colun player is a fashion leader who prefers to wear a color that contrasts with the other player s choice. Thus, the colun player prefers blue when the row player wears red; the colun player prefers red when the row player wears yellow, and the colun player prefers yellow when the row player wears blue. X Y Z X 1, 0 0, 0 0, 1 Y 0, 1 1, 0 0, 0 Z 0, 0 0, 1 1, 0 Figure 2 In this gae, {(X, X), (X, Z), (Z, Z), (Z, Y ), (Y, Y ), (Y, X)} is the unique inial CUSOBR set, and therefore, an ω-set of the gae. This set corresponds to a fashion cycle of red blue yellow red in which the colun player keeps defining new fashion and the row player keeps catching up to the forer. This gae has one ore ω-set, which is the set of all action profiles (this is also the gae s unique inial PCUSOBR set). The largest ω-set coincides with the set of all rationalizable action profiles. 4 Results For any set of states Ĥ H, let A(Ĥ) A be the set of all action profiles that are played in soe state in Ĥ. Proposition 4.1. (a) If  is a inial PCUSOBR set of G, then there exists a recurrent class Ĥ of M G (q) such that A(Ĥ) Â. 10

11 (b) Ĥ is a recurrent class of M G (q) only if A(Ĥ) is an ω-set. Due to inertia in the dynaics, for any (a 1,..., a N ) A(Ĥ), there exists a onoorphic state (a 1,..., a ) Ĥ such that ak = (a 1,..., a N ) for all k = 1,...,. Hence, if Ĥ and Ĥ are two recurrent classes of M G (q), then A(Ĥ) A(Ĥ ) =. Therefore, according to the necessary condition in part (b) of Proposition 4.1, each recurrent class of the regret atching with bounded recall dynaics corresponds to a distinct ω-set of G. Saran and Serrano (2010) provide a coplete characterization of the recurrent classes of the dynaics when players have only one-period eory. Proposition 4.2 (Saran and Serrano (2010)). Suppose = 1. Ĥ is a recurrent class of M G (q) if and only if A(Ĥ) is a inial CUSOBR set of G. Thus, each recurrent class of the regret atching dynaics with one-period eory corresponds to a distinct inial CUSOBR set of G and vice versa. However, as shown in the following exaple, for longer eory lengths, there can exist a recurrent class Ĥ such that A(Ĥ) is not a inial CUSOBR set. Exaple 4.3. Suppose = 100. Consider the gae in Figure 3. Let the row and colun players be denoted by i and j, respectively. X Y Z U 1, 50 0, 0 400, 0 M 0, , 0 0, 100 D 2, 0 0, 10 0, 0 Figure 3 There are three CUSOBR sets in this gae (1) the set of all action profiles A (2) A \ {(D, Z)} and (3) A \ {(U, Y ), (D, Z)}. The last set A \ {(U, Y ), (D, Z)} is therefore the unique inial CUSOBR set of the gae. Notice that A \ {(D, Z)} is an ω-set that is neither a inial CUSOBR set nor a inial PCUSOBR set. Since (D, X) belongs to all CUSOBR sets, Proposition 4.1 iplies that the regret atching process with eory of 100 periods has a unique recurrent class Ĥ such that (D, X) A(Ĥ). Due to inertia, the onoorphic state ( (D, X),..., (D, X) ) Ĥ. 11

12 Suppose that the dynaics is in this onoorphic state in soe period t 100. We argue that in finite tie, the dynaics will reach a state in which (U, Y ) is played. Period t: State is ((D, X),..., (D, X) ). }{{} 100 In this state, R j (Y ) > 0 since player j s average payoff over the last 100 periods is Π j = 0 but her average payoff had she played Y every tie she played X in the last 100 periods is Π j (Y ) = 10. Thus, with a positive probability, player j switches to Y and player i continues to play D in period t + 1. Period t + 1: State is ((D, X),..., (D, X), (D, Y )). }{{} 99 In this state, R i (M) > 0 since Π i = 1.98 and Π i (M) = 2. Thus, with a positive probability, player i switches to M and player j continues to play Y in period t + 2. Period t + 2: State is ((D, X),..., (D, X), (D, Y ), (M, Y )). }{{} 98 In this state, R j (Z) > 0 since Π j = 0.1 and Π j (Z) = 1. Thus, with a positive probability, player j switches to Z and player i continues to play M in period t + 3. Period t + 3: State is ((D, X),..., (D, X), (D, Y ), (M, Y ), (M, Z)). }{{} 97 In this state, R i (U) > 0 since Π i = 3.94 and Π i (U) = Thus, with a positive probability, player i switches to U and player j continues to play Z in period t + 4. Period t + 4: State is ((D, X),..., (D, X), (D, Y ), (M, Y ), (M, Z), (U, Z)). }{{} 96 In this state, R j (X) > 0 since Π j = 1.1 and Π j (X) = 1.6. Thus, with a positive probability, player j switches to X and player i continues to play U in period t + 5. Period t + 5: State is ((D, X),..., (D, X), (D, Y ), (M, Y ), (M, Z), (U, Z), (U, X)). }{{} 95 In this state, R j (Y ) > 0 since Π j = 1.6 and Π j (Y ) = Thus, with a positive probability, player j switches to Y and player i continues to play U in period t + 6. Period t+6: State is ((D, X),..., (D, X), (D, Y ), (M, Y ), (M, Z), (U, Z), (U, X), (U, Y )). }{{} 94 Therefore, (U, Y ) A(Ĥ) but, as already shown, (U, Y ) does not belong to the unique inial CUSOBR set of the gae. 12

13 4.1 Weakly Acyclic Gaes A stronger result can be established if G is weakly acyclic under better replies. A better-reply graph is defined as follows: each action profile of G is a vertex of the graph and there exists a directed edge fro vertex (a 1,..., a N ) to vertex (a 1,..., a N ) if and only if (a 1,..., a N ) (a 1,..., a N ) and (a 1,..., a N ) B G(a 1,..., a N ). sink is a vertex with no outgoing edges. A better-reply path is a sequence of vertices (a 1 1,..., a 1 N ),..., (al 1,..., a L N ) such that there exists a directed edge fro each (a l 1,..., a l N ) to (al+1 1,..., a l+1 N ). The gae G is weakly acyclic under better replies if fro any action profile, there exists at least one better-reply path to a sink. Clearly, an action profile is a sink if and only if it is a pure Nash equilibriu of G. Thus, the gae G is weakly acyclic under better replies if fro any action profile there exists at least one better-reply path to a pure Nash equilibriu. If G is weakly acyclic under better replies, then every CUSOBR set contains a pure Nash equilibriu, which is a singleton PCUSOBR set. Hence, we easily obtain the following corollary fro Proposition 4.1: Corollary 4.4. Suppose G is weakly acyclic under better replies. Then, Ĥ is a recurrent class of M G (q) if and only if A(Ĥ) is a pure Nash equilibriu of G. Given any directed edge fro (a 1,..., a N ) to (a 1,..., a N ) in the better-reply graph, we have a i B i (a 1,..., a N ) for all i N. Therefore, ore than one players can switch their actions to better replies along any directed edge in the better-reply graph. A single-better-reply graph is a spanning subgraph of the better-reply graph (i.e., has the sae set of vertices) with only those directed edges such that exactly one player switches her action to a better reply along that edge. A gae is weakly acyclic under single-better replies if in the single-better-reply graph, there exists a path fro any action profile to a pure Nash equilibriu. 5 Since a single-better-reply graph is a subgraph of the better-reply graph, the class of gaes that are weakly acyclic under single-better replies is a subset of the class of gaes that are weakly acyclic under better replies. As entioned in the introduction, Friedan and Mezzetti (2001), Young (2004) and Marden et al (2007) have studied better-reply dynaics that converge alost surely to pure Nash equilibria in gaes that are weakly acyclic under 5 Friedan and Mezzetti (2001) call this the weak finite iproveent property. A stronger condition called the finite iproveent property (Monderer and Shapley (1996)) is that there are no directed cycles in the single-better-reply graph. A 13

14 single-better replies. Corollary 4.4 iplies that per-period play under regret atching with bounded recall converges alost surely to pure Nash equilibria in gaes that are weakly acyclic under better replies. Thus, we show convergence to pure Nash equilibria in a larger class of gaes. Although it is possible to construct exaples of several gaes that are weakly acyclic under better replies but not weakly acyclic under single-better replies (see Saran and Serrano (2010) for one such gae), it is nevertheless unclear how uch larger the forer class is. Soe well-known gaes that are weakly acyclic under single-better replies, and hence also weakly acyclic under better replies and covered by our result, include ordinal potential gaes (Monderer and Shapley (1996)), superodular gaes (Friedan and Mezzetti (2001)), second-price and first-price auctions, and Bertrand duopolies (Saran and Serrano (2010)). 4.2 Rando Sapling We are not able to strengthen Proposition 4.1 to an if and only if stateent for gaes that are not weakly acyclic under better replies, which is the reason to turn to a rando sapling version of the process next. That is, we thus far have assued that players consider all the past periods in the -period history. Instead, suppose that each player i independently draws a rando saple of s periods (a 1,..., a s ) fro the -period history (a t +1,..., a t ) and calculates her regrets relative to the latest action in her saple, a s i (unlike earlier, where the regrets are calculated relative to the latest action a t i). Forally, let Π s i = 1 s π i (a k ) be player i s average payoff over her s-period s k=1 saple. For all a i a s i, let Π s i (a i) be the average payoff over these s periods that player i would have obtained had she played action a i every tie she played action a s i during these s periods. That is, Π s i (a i) = 1 s υi k (a s i), where { υi k (a π i (a i) = i, a k i) if a k i = a s i π i (a k i, a k i) if a k i a s i. Define Ri s (a i) = Π s i (a i) Π s i. Then, player i plays action a i in period t + 1 with probability q(ri s (a i)) > 0 if and only if Ri s (a i) > 0, whereas she does not switch with k=1 14

15 probability 1 a q(r i as i s (a i)) > 0. 6 This adaptive behavior is regret atching with i bounded recall and rando sapling. As before, a state in a period is the history of last action profiles. Hence, the set of states is still H. Given G, for fixed q( ), regret atching with bounded recall and rando sapling describes an aperiodic Markov process M G (q) on the state space H. Proposition 4.5. If s/ is sufficiently sall, then Ĥ is a recurrent class of M if and only if A(Ĥ) is a inial PCUSOBR set of G. G (q) Thus, if the saple size is sall enough, then per-period play under regret atching with bounded recall and rando sapling will alost surely in finite tie enter a inial PCUSOBR set and then stay inside this set forever with every action profile in the set being played infinitely often. 5 Connections with Hart and Mas-Colell s Regret Matching Hart and Mas-Colell (2000) study the long-run behavior when the players use regret atching but, in contrast to our odel, have unbounded eory. Regret atching with unbounded recall is defined as follows: at the beginning of period t + 1, let (a 1,..., a t ) be the history of action profiles played. The average payoff of player i over this history is given by Π t i = 1 t π i (a k ). Let a t i be the action played by player t k=1 i in period t. For all a i a t i, let Π t i(a i) be the average payoff that player i would have obtained had she played action a i every tie she played action a t i in the history. That is, Π t i(a i) = 1 t υi k (a t i), where υi k (a i) is as in (1). Define Ri(a t i) = Π t i(a i) Π t i. k=1 Then, player i switches to action a i in period t + 1 with probability 1 c ax{rt i(a i), 0}, 6 Again, several specifications of q( ) are possible. See Footnote 3. 15

16 whereas she does not switch with probability 1 1 c a i at i ax{r t i(a i), 0}, which is positive for a sufficiently large constant c. Let µ t be the epirical distribution of play up to period t, i.e., for every (a 1,..., a N ), µ t (a 1,..., a N ) = 1 t {1 k t ak = (a 1,..., a N )}. The ain theore in Hart and Mas-Colell (2000) states the following: If the players use regret atching with unbounded recall, then the epirical distribution of play µ t converges alost surely as t to the set of correlated equilibriu distributions of G. Nevertheless, as Hart and Mas-Colell (2000, p. 1132) theselves point out, we know little about additional convergence properties of µ t under regret atching with unbounded recall. In particular, it is not known whether µ t converges to a point, i.e., a distribution over the set of action profiles. We know that if there exists a finite tie T such that for all t > T, µ t lies in the set of correlated equilibria, then µ t ust converge to a pure Nash equilibriu (because the action profile does not change whenever µ t is a correlated equilibriu as all regrets are zero). Hence, if µ t does not converge to a pure Nash equilibriu, then the sequence {µ t } t 1 ust lie infinitely often outside the set of correlated equilibria. Therefore, if µ t converges to a point, then it can either converge to a pure Nash equilibriu or a correlated equilibriu on the boundary of the set of correlated equilibria. 7 To facilitate the coparison with regret atching with bounded recall (but no sapling), let s fix q( ) to be such that player i switches to action a i in period t + 1 with probability 1 ax{r c i(a i), 0}, whereas she does not switch with probability 1 1 ax{r i (a c i), 0}, where c is sufficiently large to ensure that the latter is positive. a i at i In contrast to Hart and Mas-Colell (2000), we have precise results about per-period play in the long run under regret atching with bounded recall. If G is weakly acyclic under better replies, Corollary 4.4 tells us that per period play a t will alost surely in finite tie be a pure Nash equilibriu the particular equilibriu depends on the 7 Hart and Mas-Colell (2000) did not consider specific gaes and it ight well be that ore precise convergence results are obtained for soe classes of gaes. 16

17 initial history. In general, for any gae, pointwise convergence of per-period play can only happen to pure Nash equilibria. But in addition, Proposition 4.1 tells us that under regret atching with bounded recall, per period play a t will alost surely in finite tie enter soe ω-set again, the particular set depends on the initial history and after that tie, each of the action profiles that belong to this set, and only this set, will be played infinitely often. Thus, µ t will converge alost surely as t to a distribution with support over soe ω-set. However, as the following exaple illustrates, the epirical distribution of play need not converge to the set of correlated equilibriu distributions when players have bounded recall. Exaple 5.1. Suppose there are two players who repeatedly play the gae in Figure 4, where ɛ 0. X Y Z U 0, 20 50, 15 60, 20 D 10, 30 40, ɛ, 25 Figure 4 Fix 1, and let M(, ɛ) be the transition atrix of the Markov process when the players use regret atching with bounded recall of. Let M hh (, ɛ) be the hh entry in this atrix, i.e., the probability of transition fro state h = (a 1,..., a ) to state h = (a 1,..., a ) in one period. Note that a k = a k+1 for all k = 1,..., 1. Let i and j denote, respectively, the row and colun players. Since the players choose their actions independently, M hh (, ɛ) = i hh (, ɛ)j hh (, ɛ), where i hh (, ɛ) and j hh (, ɛ) are the probabilities that, respectively, the row player plays action a i and the colun player plays action a j during the next period conditional on state h. Since R j ( ) does not depend on ɛ, j hh (, ɛ) does not depend on ɛ. Thus, j hh (, ɛ) = j hh (, 0) for all ɛ. Siilarly, if h is such that a k j Z for all k = 1,...,, then i hh (, ɛ) = i hh (, 0) for all ɛ. So suppose h is such that player i played a i in ˆ periods and in those ˆ periods, player j played X and Z, respectively, x and z ties. First, let a i = U. Then, conditional on state h, R i (D) = 10 (2x + z ˆ) + ɛ z. Therefore, if ɛ < in{ 10, c 10} (note that c > 10 to ensure positive probability of inertia in the process when ɛ = 0), then the probability that the row player switches 17

18 to D the next period is 10 z (2x + z ˆ) + ɛ c c if 10 (2x + z ˆ) 0 0, if 10 (2x + z ˆ) < 0. Hence, for all ɛ < in{ 10, c 10}, we have: if a i if a i = D, then i hh (, ɛ) = = U, then i hh (, ɛ) = { { i hh (, 0) + ɛ z c if 10 (2x + z ˆ) 0 i hh (, 0), otherwise. i hh (, 0) ɛ z c if 10 (2x + z ˆ) 0 i hh (, 0), otherwise. Next, let a i = D. Then, conditional on state h, R i (U) = 10 ( ˆ 2x z) ɛ z. Therefore, if ɛ < 10 and c > 10, then the probability that the row player switches to U the next period is 10 z ( ˆ 2x z) ɛ c c if 10 ( ˆ 2x z) > 0 0, if 10 ( ˆ 2x z) 0. Hence, for all ɛ < 10, we have: if a i if a i = U, then i hh (, ɛ) = = D, then i hh (, ɛ) = { { i hh (, 0) ɛ z c if 10 ( ˆ 2x z) > 0 i hh (, 0), otherwise. i hh (, 0) + ɛ z c if 10 ( ˆ 2x z) > 0 i hh (, 0), otherwise. Thus, whenever ɛ < in{ 10, c 10}, there exists a Q() such that M(, ɛ) = M(, 0) + ɛq(). 18

19 If ɛ > 0, then the set of all action profiles is the gae s unique ω-set. Hence, it follows fro Proposition 4.1 that the Markov process defined by M(, ɛ) has a unique recurrent class and hence, a unique invariant distribution, µ(, ɛ). Then, µ(, ɛ) = µ(, ɛ)m(, ɛ) = µ(, ɛ)m(, 0) + ɛµ(, ɛ)q(). There exists a subsequence where µ(, ɛ) converges pointwise to say µ() as ɛ 0. Hence, along this subsequence, we have ( ) µ() = li µ(, ɛ) = li µ(, ɛ) M(, 0) = µ()m(, 0). ɛ 0 ɛ 0 That is, µ() is an invariant distribution of the Markov process defined by M(, 0). But if ɛ = 0, then {(U, Z)} is the gae s unique ω-set; the other CUSOBR sets are A and A \ {(D, Z)}. Therefore, the Markov process defined by M(, 0) has a unique invariant distribution, with support on the onoorphic state in which (U, Z) is played. Thus, we conclude that for any eory, there exists an ɛ > 0 such that the unique invariant distribution of the Markov process defined by M(, ɛ ) puts probability close to 1 on the onoorphic state in which (U, Z) is played. For any ɛ > 0, the gae has a unique correlated equilibriu, in which each of the action profiles (U, X), (U, Y ), (D, X) and (D, Y ) has probability equal to Thus, it follows fro the above arguents that fixing a finite as large as one wishes, there exists a sall enough ɛ such that the liiting epirical distribution of play in the corresponding gae is concentrated on the outcoe (U, Z) a proportion of tie close to 1: this is very far fro the unique correlated equilibriu distribution of the gae. On the other hand, for any ɛ > 0 and taking =, it follows fro the result in Hart and Mas-Colell (2000) that the liiting epirical distribution of play ust approxiate the unique correlated equilibriu. Our analysis shows that, in obtaining this result, the infinite tail of eory is crucial. 6 Proofs Proof of Proposition 4.1: Suppose  is a PCUSOBR set. Let Pi(Â) be the projection of  on A i. Pick any action profile (a 1,..., a N ) Â. Suppose that in period t, the dynaics is in state (a t +1,..., a t ) H such that a k = (a 1,..., a N ), k = 19

20 t + 1,..., t. We argue by induction that for all t t, the state in period t, (a t +1,..., a t ) is such that a k Â, k = t + 1,..., t. This is clearly true for t = t. Now, suppose this is true for t t. Consider a t +1. It ust be that for all i, either a t +1 i = a t i or there exists a a k, where t +1 k t, such that a k i = a t i π i (a t +1 i, a k i) > π i (a k i, a k i). If a t +1 i = a t i, then obviously a t +1 i P i (Â). On the other hand, since a k  (follows fro the induction hypothesis) and  is a PCUSOBR set, we again have a t +1 i P i (Â). Since this is true for all i, +1 at i N Pi(Â) = Â, where the equality follows since  is a product set, which copletes the induction arguent. This iplies that starting fro period t, any action profile that does not belong to  is played with zero probability. Hence, there exists a recurrent class Ĥ such that A(Ĥ) Â. The first stateent in the proposition follows fro this fact. Next, suppose Ĥ is a recurrent class of M G (q). We first argue that a CUSOBR set. Pick any action profile (a 1,..., a N ) and A(Ĥ) is A(Ĥ). There exists a (a 1,..., a ) Ĥ such that ak = (a 1,..., a N ), k = 1,..., (because there is inertia in the dynaics and Ĥ is a recurrent class). Let (a 1,..., a N ) B G(a 1,..., a N ). Fro state (a 1,..., a ), there is a positive probability that the dynaics will ove to the new state (a 2,..., a, a +1 ), where a +1 = (a 1,..., a N ). Since Ĥ is a recurrent class, it ust be that (a 2,..., a, a +1 ) Ĥ and hence, (a 1,..., a N ) A(Ĥ). Now, suppose A(Ĥ) is a CUSOBR set that contains a saller PCUSOBR set Â. Then there exists a recurrent class of M G (q), H such that A(H )  A(Ĥ), a contradiction. This copletes the proof of the second stateent in the proposition. Proof of Proposition 4.5: As in the previous proof, we can argue that if  is a PCUSOBR set, then there exists a recurrent class Ĥ of M G (q) such that A(Ĥ) Â. We argue that if Ĥ is a recurrent class of M G (q), then A(Ĥ) contains a PCUSOBR set. Pick any action profile (a 1,..., a N ) A(Ĥ). Recall the definition of ˆB G and to siplify notation, we instead write ˆB. Consider the iteration ˆB({(a 1,..., a N )}) ˆB 2 ({(a 1,..., a N )})... ˆB l ({(a 1,..., a N )})... Since the set of action profiles is finite, there exists a finite l such that for all l l, ˆB l ({(a 1,..., a N )}) = ˆB l+1 ({(a 1,..., a N )}) = Ã. By construction, à is a PCUSOBR 20

21 set. Let s A <. Since Ĥ is a recurrent class, starting at any state in Ĥ, the action profile (a 1,..., a N ) will be played after finite tie. repeatedly draw a saple in which a s Then each player can = (a 1,..., a N ) and therefore, this action profile will be played for the next periods due to inertia, i.e., there exists a (a 1,..., a ) Ĥ such that ak = (a 1,..., a N ), k = 1,...,. Let (a 1,..., a N ) ˆB({(a 1,..., a N )}) \ {(a 1,..., a N )}. Starting with state (a 1,..., a ) in period t, there is a positive probability that (a 1,..., a N ) is played for the next s periods. This is because in each t + k period, where 1 k s, each player can draw a s-period saple in which only (a 1,..., a N ) is played. Let (a 1,..., a N ) ˆB({(a 1,..., a N )}) \ {(a 1,..., a N ), (a 1,..., a N )}. Starting with period t + s, there is a positive probability that (a 1,..., a N ) is played for the next s periods. This is because in each t + s + k period, where 1 k s, each player can again draw a s-period saple in which only (a 1,..., a N ) is played. It is clear that in finite tie, we will obtain a history h in which each action profile in ˆB({(a 1,..., a N )} is played for at least s periods. Let (ã 1,..., ã N ) ˆB 2 ({(a 1,..., a N )}) \ ˆB({(a 1,..., a N )}). Hence, for all i, there exists a (ã i, ã i) ˆB({(a 1,..., a N )}) such that ã i B i (ã i, ã i). In each of the s periods following history h, there is a positive probability that player i will draw a s-period saple in which only (ã i, ã i) is played. Hence, there is a positive probability that (ã 1,..., ã N ) will be played during these s periods. Continuing the arguent, we see that we will obtain a history h in which all action profiles in à are played at least s ties. Since Ĥ is a recurrent class, history h Ĥ. Hence, à A(Ĥ). So far we have argued that: (i) if  is a PCUSOBR set, then there exists a recurrent class Ĥ such that A(Ĥ) Â, and (ii) if Ĥ is a recurrent class, then A(Ĥ) contains a PCUSOBR set. It follows fro these stateents that a inial PCUSOBR set  contains a A(Ĥ), where Ĥ is a recurrent class, which in turn contains a PCUSOBR set Ã. Since  is a inial PCUSOBR set, it ust be that  = A(Ĥ). On the other hand, if Ĥ is a recurrent class, then A(Ĥ) contains a PCUSOBR set and hence a inial PCUSOBR set Ã, which in turn contains a A( H), where H is a recurrent class. But H = Ĥ and hence, A(Ĥ) = Ã. Thus, the proposition is established. 21

22 References Basu, K., Weibull, J.W., Strategy Subsets Closed under Rational Behavior. Econoics Letters 36, Bernhei, B.D., Rationalizable Strategic Behavior. Econoetrica 52, Foster, D.P., Vohra, R.V., Calibrated Learning and Correlated Equilibriu. Gaes and Econoic Behavior 21, Foster, D.P., Vohra, R.V., Asyptotic Calibration. Bioetrika 85, Friedan, J.W., Mezzetti, C., Learning in Gaes by Rando Sapling. Journal of Econoic Theory 98, Fudenberg, D., Levine, D.K., Universal Consistency and Cautious Fictitious Play. Journal of Econoic Dynaics and Control 19, Fudenberg, D., Levine, D.K., The Theory of Learning in Gaes. Cabridge: MIT Press. Fudenberg, D., Levine, D.K., Conditional Universal Consistency. Gaes and Econoic Behavior 29, Hannan, J., Approxiation to Bayes Risk in Repeated Play. In: Dresher, M., et al (Eds.). Contributions to the Theory of Gaes III. Princeton: Princeton University Press, Hart, S., Adaptive Heuristics. Econoetrica 73, Hart, S., Mas-Colell, A., A Siple Adaptive Procedure Leading to Correlated Equilibriu. Econoetrica 68, Hurkens, S., Learning by Forgetful Players. Gaes and Econoic Behavior 11, Josephson, J., Matros, A., Stochastic Iitation in Finite Gaes. Gaes and Econoic Behavior 49,

23 Lehrer, E., Solan, E., Approachability with Bounded Meory. Gaes and Econoic Behavior 66, Marden, J.R., Arslan, G., Shaa, J.S., Regret Based Dynaics: Convergence in Weakly Acyclic Gaes. AAMAS 07: Proceedings of the 6th International Joint Conference on Autonoous Agents and Multiagent Systes. New York: ACM, Monderer, D., Shapley, L.S., Potential Gaes. Gaes and Econoic Behavior 14, Pearce, D.G., Rationalizable Strategic Behavior and the Proble of Perfection. Econoetrica 52, Ritzberger, K., Weibull, J.W., Evolutionary Selection in Noral-For Gaes. Econoetrica 63, Sandhol, W.H., Evolutionary Gae Theory. In: Meyers, R. (Ed.). Encyclopedia of Coplexity and Systes Science. New York: Springer, Saran, R., Serrano, R., Ex-Post Regret Learning in Gaes with Fixed and Rando Matching: The Case of Private Values. Working Paper, Brown University. URL: Shapley, L.S., Soe Topics in Two-Person Gaes. In: Dresher, M., et al (Eds.). Advances in Gae Theory. Annals of Matheatical Studies 52. Princeton: Princeton University Press, Viossat, Y., The Replicator Dynaics does not Lead to Correlated Equilibria. Gaes and Econoic Behavior 59, Viossat, Y., Evolutionary Dynaics ay Eliinate All Strategies Used in Correlated Equilibriu. Matheatical Social Sciences 56, Young, H.P., The Evolution of Conventions. Econoetrica 61, Young, H.P., Individual Strategy and Social Structure. Princeton: Princeton University Press. 23

24 Young, H.P., Strategic Learning and its Liits. Oxford: Oxford University Press. Zapechelnyuk, A., Better-Reply Dynaics with Bounded Recall. Matheatics of Operations Research 33,