Folk Theorem in Stotchastic Games with Private State and Private Monitoring Preliminary: Please do not circulate without permission

Transcription

1 Folk Theorem n Stotchastc Games wth Prvate State and Prvate Montorng Prelmnary: Please do not crculate wthout permsson Takuo Sugaya Stanford Graduate School of Busness December 9, 202 Abstract We show the folk theorem n stochastc games, where each player has prvate nformaton about the state and the other players actons. The punshment payo s characterzed analogously to a statc Nash equlbrum n repeated games.

2 Introducton One of the key nsghts n the lterature on dynamc games s that enttes wth ther own objectves can sustan cooperaton n the long term. The drvng force of cooperaton s recprocty: f a player devates today, she wll be punshed n future. Past papers n repeated games has shown that recprocty can mantan cooperaton n more and more general settngs: Fudenberg and Maskn (986) establsh the folk theorem under perfect montorng, Fudenberg, Levne, and Maskn (994) extend the folk theorem to mperfect publc montorng, Hörner and Olszewsk (2006) and Hörner and Olszewsk (2009) show the robustness of these results to almost perfect and almost publc montorng respectvely, and a seres of papers, Sugaya (202d), Sugaya (202a), Sugaya (202b) and Sugaya (202c) show that the folk theorem holds wth general prvate montorng. All of these papers assume that the strategc envronment s statonary: the expected utlty functon and jont condtonal dstrbuton of sgnals gven the actons played are the same across perods. In some cases, ths statonary assumpton seems too strong: magne two rms competng aganst each other. Ther utlty functons may depend on stochastc state of Nature such as busness cycles. Motvated by these stochastc features of strategc envronments, a seres of papers consder stochastc games, where state of Nature n the next perod s stochastcally determned gven current state and current actons: Dutta (995) shows that, f actons are perfectly observable, then the folk theorem holds n stochastc games. Fudenberg and Yamamoto (20a) and? show that the folk theorem holds wth publc montorng. These papers assume that the state s publcally observable and that there s no uncertanly about the current strategc envronment. However, ths may not be approprate n certan examples. Consder agan the two rms. One rm may have d erent nformaton about busness cycles than the other rm. Hence, now the state of Nature s the pro le of each rm s prvate nformaton. Further, suppose these rms are competng aganst each other n terms of prces (Bertrand competton) and, snce demands come from large ndustral buyers, prces are set n one- 2

3 to-one con dental negotaton between a rm and ts own customer (See Stgler (964)). Hence, each rm cannot observe the other rm s acton (prce) drectly. Instead, ts own sales can serve as an mperfect sgnal about the other rm s prce-settng behavor. Snce the sales are also determned n con dental meetngs, one rm s sales level s ts own prvate nformaton. Hence, we can see that each rm has a prvate sgnal about the other rm s acton. In addton, snce relatonshps between the expected sales and prces are a ected by busness cycles, the jont dstrbuton of each rm s prvate sgnals about the other rm s acton also depends on the current state. Fnally, snce rms can take actons to enhance future demands, t s natural to assume that the dstrbuton of the next state s determned by the current state and actons. In total, we consder the followng stochastc games wth asymmetrc nformaton. Let 3 be the set of possble states for player, and Q 2I 3 ( ) 2I be the set of possble state pro les (state of busness cycles n the above example). In addton, let Y 3 y be the set of possble prvate sgnals for player about players s actons, and Y Q 2I Y 3 y (y ) 2I be the set of possble sgnal pro les. Gven current state t and acton pro le a t, the jont condtonal dstrbuton of the next state t+ and prvate sgnal pro le y s determned. Each player s expected utlty gven acton pro le a t depends on t. Our man result s the folk theorem: wth perfect and publc cheap talk communcaton devce, any payo pro le v, whch s n (a properly de ned convex superset of) the Pareto fronter and domnates a (properly de ned) perfect Bayesan equlbrum payo n ntely repeated verson of the above game (wth a properly de ned termnal payo ), can be approxmately attaned n a perfect Bayesan equlbrum n ntely repeated games f the players are su cently patent. The basc dea of the proof follows from Mezzett (2004) n the dynamc mechansm desgn lterature. Suppose we want to construct an equlbrum whose payo maxmzes the socal welfare wth Pareto weght ( ) 2I. Let (a(; )) 2 be the Markov strategy that We use the same set of Escobar and Tokka (202). 3

4 maxmzes the socal welfare P t= t P n2i n~u n. 2;3 If players collect nformaton about ther ex post utltes by cheap talk and gve P n n2 ~u n;t n each perod, 4 then each player wants to maxmze P t= ~u t ;t + P n n2 ~u n;t, equvalently, P t= t P n2i n~u n;t. The problem s that ths s not feasble after some hstory: suppose now the players are at the very best equlbrum payo. Then, players payo by transtng to the better equlbrum. cannot ncrease player s equlbrum To overcome ths d culty, see the stochastc game as repettons of long T -perod blocks. Wthn a block, players aggregate nformaton and gve the followng reward as the P contnuaton payo : t:wthn the block t P n n2 ~u n;t mnus T tmes the expectaton of P n n2 ~u n;t + " usng the statonary dstrbuton gven (a(; )) 2. If " > 0 and the state transton s rreducble, then by the central lmt theorem, wth a hgh probablty, the reward s negatve. Snce players can decrease player s equlbrum payo by transtng to the worse equlbrum from the best equlbrum, t s feasble to mplement the negatve reward as the contnuaton payo. However, snce ths s true only wth a hgh probablty, not wth probablty one, there are rare hstores after whch the reward becomes postve. In the context of repeated games, a seres of papers by Sugaya 5 deals wth ths ssue. Intutvely, we apply Sugaya (202d) to stochastc games. To ths end, there are four man d cultes specal n stochastc games: Frst, snce the feasblty s not sats ed, after a rare hstory, the players cannot use the same reward. Hence, after a rare hstory, the players changes how they determne the contnuaton payo s d erently. Suppose the players are now n one perod before ths change. Then, even though P n2i n~u n;t s the margnal e ect of the nformaton n the current perod on the current payo, ths does not ncentvze the players to collaboratvely take (a(; )) 2 snce (a(; )) 2 s de ned takng the e ect on the contnuaton payo nto account, but the margnal e ect on the contnuaton payo s no longer P n2i n~u n;t. We wll see that, wth 2 Here, s a dscount factor, I s the set of all players, and ~u n;t s a realzaton of the ex post payo for player n n perod t. 3 For smple notaton, we omt the dependence of (a(; )) 2 on dscount factor. 4 Here, s the set of players except for player. 5 Namely Sugaya (202d), Sugaya (202a), Sugaya (202b) and Sugaya (202c). 4

5 rreducble state transton, ths e ect s su cently small f the current perods s su cently far from the change of the contnuaton payo and does not a ect the long-run equlbrum payo. Second, the players take mxed strateges n Sugaya (202d). On the other hand, f the players per perod payo s are equvalent to P n2i n~u n;t, then each player has the strct ncentve to take (a(; )) 2. In addton, to ncentvze player to take a mxed strategy by makng player nd erent between more than one actons, players need to know the state pro le. However, f players use player s report about player s state ;t to make player nd erent, then t generally creates the ncentve for player to tell a le. We deal wth ths problem by usng publc randomzaton devce. 6 Intutvely, after player reports ;t, wth some small probablty by publc randomzaton, players make player nd erent. Then, from player s perspectve who s about to report ;t, wth a hgh probablty, the players should maxmze P n2i n~u n;t by collaboratvely take a(; )) 2 dependng on the states, whch means player should tell the truth about ;t. Snce we also need to ncentvze player to take a mxed strategy wth respect to reports about ;t, the actual procedure s more complcated and the explanaton s deferred to Secton 0.2. Thrd, player j 2 need to nfer player s mxed strategy from her own hstory. In the repeated games, as long as player takes a mxed strategy ndependently across tme, player j can treat a sgnal n a d erent perod ndependently. However, n stochastc games, to nfer player s acton n perod t, player j needs to nfer the state n perod t. In turn, to nfer the state n perod t, t s mportant to nfer the state n perod t and acton pro le n perod t. Agan, to nfer the state n perod t, player j need to nfer the state n perod t 2 and acton pro le n perod t 2, and so on. Therefore, the nference s now tme-dependent and gettng more and more complcated as tme goes on. We wll show that, wth rreducble state transton, as long as players j take Markov strateges (and t s true wth a hgh probablty on equlbrum), the nformaton far away from the current perod s not useful to nfer the current acton and state. Usng ths fact, we can deal wth 6 Wth cheap talk communcaton, t s nnocuous to assume the extence of publc randomzaton devce. 5

6 ths problem. Fourth, n Sugaya (202d), player controls player s contnuaton payo by transtng between a general strategy to gve her rewards n terms of contnuaton payo s and a harsh strategy to gve her punshments. However, now, the e ect of the swtch of player s strategy on player s contnuaton payo depends on the current state. As n the rst d culty, we can show that, wth rreducble state transton, ths e ect s su cently small so that we can adjust ths e ect wthout a ectng players ncentves and payo s. Let us revew the related lterature. If the state transton s..d. across perods, then Fudenberg, Levne, and Maskn (994) o ers the folk theorem wth publc montorng. On the other hand, n the case where the state s perfectly persstent (that s, the state s xed once and for all at the begnnng of the game), Aumann and Maschler (995), Athey, Bagwell, and Sanchrco (2004), Fudenberg and Yamamoto (20b), Hörner and Lovo (2009), Hörner, Lovo, and Tomala (20), Pesk (2008) and Watson (2002), among others, provde a seres of research. 7 If the state follows the rreducble Markov chan, then Athey and Bagwell (2008) analyze colluson n a Bertrand olgopoly wth prvately known costs. Escobar and Tokka (202) shows the folk theorem. Man d erences from ths paper s that they consder prvate values and assume that the state transtons are ndependent across players and also ndependent of prevous actons, whle we consder (possbly) nterdependent values and the state transtons can be nterdependent across players and also depend on prevous actons. In the context of dynamc mechansm desgn, Athey and Segal (2007) prove an e cency result wth transferable utltes. In ths paper, we do not assume transfers and all the rewards mentoned above should be mplemented by the movement of the contnuaton payo n stochastc games. The rest of the paper s organzed as follows. In Secton 2, we ntroduce the model. Snce the tmng of the arrval of nformaton and players move s complcated, we o er the explanaton of the tme lne n Secton 3. After statng the assumptons n Secton 4, we 7 See also the lterature on reputaton such as Kreps and Wlson (982) and Mlgrom and Roberts (982). 6

7 state the theorem n Secton 5. Secton 6 relates the n ntely repeated stochastc game to a ntely repeated stochastc games wth auxlary scenaro (reward) and derves su cent condtons on the ntely repeated stochastc game to show the folk theorem n the n ntely repeated stochastc game. The rest of the paper s devoted to the proof of the su cent condtons. Secton 7 explans the basc structure of the ntely repeated stochastc game. The next three sectons deal wth the four d cultes mentoned above: Secton 8 deal wth the fourth d culty; Secton 9 consder the rst d culty; Secton 0 solves the second and thrd d cultes. In Secton 0, whle we are dealng wth the second d culty, we ntroduce a spec c form of cheap talk n addton to the perfect and publc communcaton devce. Secton shows that ths specal cheap talk can be replaced wth communcatons va actons and the perfect and publc communcaton devce. That s, the specal form of the cheap talk s ntroduced solely for smple exposton. 2 Model We consder N-player stochastc games wth prvate states. Let I = f; :::; Ng be the set of players, = f; :::; j jg be the set of player s prvate states, A be the set of player s actons, Y be the set of player s prvate sgnals, and U be the set of player s ex post utlty realzatons. The ntal state pro le ( ; ) 2I wth ; 2 for all 2 I s drawn from a dstrbuton q. We assume q s common knowledge. 8 Suppose player takes a ;t 2 A n perod t, whch determnes the current acton pro le a t. Player observes player s prvate sgnal y ;t 2 Y, ex post utlty ~u ;t and state n the next perod ;t+. We assume that the sgnal pro le n perod t, the ex post utlty pro le n perod t and the state pro le n perod t + are determned by a jont dstrbuton functon q(y t ; ~u t ; t+ j t ; a t ), dependng on a t and t. Wthout loss, we see (y ;t ; ~u ;t ) as player s prvate sgnal. Therefore, from now on, we assume that U = ; for all 2 I and the ex post utlty s a functon of a ;t and y ;t. The 8 In our equlbrum, snce each player has the ncentve to tell the truth about ; ex post, ths assumpton s not mportant. 7

8 mplct assumpton here s that player can observe her ex post utlty every perod. Usng the dstrbuton q(y t ; t+ j t ; a t ), we can calculate player s ex ante utlty u (a t ; t ). Let u(a t ; t ) = (u (a t ; t )) 2I be the ex ante payo pro le. 3 Cheap Talk and Tme Lne Throughout the paper, we assume that the players communcate va (perfect and publc) cheap talk. Wth ths assumpton, t s wthout loss to assume the exstence of publc randomzaton devce. Gven cheap talk and publc randomzaton devce, we consder the followng tme lne of the game: n each perod t,. the players draw the rst publc randomzaton devce; 2. the players send the rst messages; 3. the players draw the second publc randomzaton devce; 4. the players send the second messages; 5. the players draw the thrd publc randomzaton devce; 6. the players take actons. The detals of each step wll be spec ed later. 4 Assumptons As mentoned n Secton 2, we assume that the players observe ther own ex post utltes: Assumpton The players observe ther own ex post utltes. Wth ths assumpton, players can make player a resdual clamant: f players gve the summaton of ther ex post utltes ~u ;t P n2 8 n ~u n;t to player as a

9 contnuaton payo, then player s objectve functon wll be ~u ;t + P n n2i ~u n;t. By the a ne transformaton, player wants to maxmze u(a t ; t ) P n2i n~u n;t. Therefore, each player s objectve s algned. Here, 2 = 2 R N ++ : kk = () s a Pareto weght. Throughout the paper, we use Eucldean norm kk = q P n2i 2 n. Next, we assume two assumptons wth regard to the full-support condton. Frst, let q (y ;t ; t+ j t ; a t ) be the margnal dstrbuton of player s sgnal and next state pro le gven the current state pro le and acton pro le. We assume the full support for ths margnal dstrbuton: Assumpton 2 For any 2 I, y ;t 2 Y, t+ ; t 2 and a t 2 A, q (y ;t ; t+ j t ; a t ) > 0. In partcular, ths assumpton mples that the state transton s rreducble. Second, we assume that player j 2 cannot dentfy player s acton completely after knowng player s sgnal: Assumpton 3 Suppose player j has (a j;t ; y j;t ) and knows that the state pro les n perods t and t + are t and t+, that players (; j) take acton (;j) ( t ) dependng on the current state pro le t, and that player observes y ;t. Player j cannot dentfy player s acton wth probablty one: for any ; j 2 I wth 6= j, there exsts a (;j) 2 A (;j) such that, for any (a j;t ; y j;t ) 2 A j Y j, t ; t+ 2, and a ;t ; a 0 ;t 2 A, we have q(y ;t ; y j;t ; y (;j);t ; t+ j t ; a ;t ; a j;t ; a (;j) ) > 0 for some y (;j);t 2 Y (;j) f and only f q(y ;t ; y j;t ; y (;j);t ; t+ j t ; a 0 ;t; a j;t ; a (;j) ) > 0 for some y (;j);t 2 Y (;j) : 9

10 One su cent condton for Assumpton 3 s that the montorng s state ndependent, q(y t ; t+ j t ; a t ) = q ( t+ j t ; a t )q y (y t j a t ), and that q y (y t j a t ) s publc montorng or prvate montorng wth full support. Intutvely speakng, n our equlbrum, the players take the prvate strateges. Assumptons 2 and 3 mply that, on the equlbrum path, each player keeps some nformaton prvate. The next two assumptons are about the dent ablty of actons. Frst, we assume that, f the players knew the state pro le t, then ndvdual dent ablty of actons s sats ed: Assumpton 4 For any t 2, 2 I and a ;t 2 A, a jy j j j-dmensonal vector (q (y ;t ; ;t+ j t ; a ;t ; a ;t )) y ;t ; ;t+ 2Y s lnearly ndependent wth respect to a ;t 2 A. Fnally, consder the stuaton that, for some arbtrarly xed and acton pro le a, we want to gve player a strct ncentve to take a ;t = a gven t = and a ;t = a. We assume that, for each player, there exsts a statstcs calculated from the other players nformaton such that () the expected value of the statstcs s the same for all the players regardless of the current state pro le and acton pro le, and that () player has a strct ncentve to coordnately take a t = a gven t = : Assumpton 5 For any 2 and a 2 A, there exsts x [; a] : A Y! R such that. the expected value of the statstcs s the same for all the players: E [x [; a]( ;t ; a ;t ; y ;t ; ;t+ ) j t ; a t ] s ndependent of for all a t 2 A t and t 2 t ; 0

11 2. player has a strct ncentve to coordnately take a t = a gven ;t = : for all a 0 6= a, E [x [; a](; a ; y ;t ; ;t+ ) j ( t ; a t ) = (; a)] > E [x [; a](; a ; y ;t ; ;t+ ) j ( t ; a t ) = (; a 0 )] + : Intutvely speakng, f x [; a]( ;t ; a ;t ; y ;t ; ;t+ ) s the contnuaton payo that player gets n perod t, then Condton mples that each player receves the same expected contnuaton payo regardless of the current state pro le and acton pro le, and t ncentvzes the players to collaboratvely take a() n the case where the current state s. There are three mportant examples that satsfy ths assumpton:. suppose that the montorng s publc and that, for each, gven t =, the publc sgnal can statstcally dentfy the acton pro le. Then, for each a 2 A, there exsts x[; a] : Y! R such that a = arg max a2a E [x[; a](y t ) j ( t ; a t ) = (; a)]. 9 If we de ne x [; a]( ;t ; a ;t ; y ;t ; ;t+ ) = x (; a) [y t ] for all, then ths sats es two condtons n Assumpton 5 after a proper a ne transformaton; 2. suppose that the montorng s ndependent of the states, q(y t ; t+ j t ; a t ) = q ( t+ j t ; a t )q y (y t j a t ), and that, for 2 I, players s sgnals can statstcally dentfy the acton pro les. Then, for each a 2 A, there exsts x [a] : Y! R such that E [x [a](y ;t ) j t ; a t ] = E [x [a](y ;t ) j a t ] = E [x j [a](y j;t ) j a t ] = E [x j [a](y j;t ) j ( t ; a t ) = (; a)] for each ; j 2 I, and that a = arg max a2a E [x [a](y ;t ) j ( t ; a t ) = (; a)]. If we de- ne x [; a]( ;t ; a ;t ; y ;t ; ;t+ ) = x (; a) [y ;t ] for all, then ths sats es two condtons n Assumpton 5 after a proper a ne transformaton; 9 Here, we de ne Y 3 y as the support of publc sgnals.

12 3. suppose that, for each ;t 2 and a ;t 2 A, gven ;t and a ;t, f players s nformaton (y ;t ; ;t+ ) can statstcally dent es player s nformaton ( ;t ; a ;t ), then Condtons and 2 are sats ed. 5 (Partal) Folk Theorem Consder any pure Markov strategy (a()) 2, assumng that each player knows the state pro le. Let v((a()) 2 ) be the lmt payo pro le where the dscount factor goes to one: v((a()) 2 ) lm E h( )! t= t u(a t ; t ) j ; (a()) 2 : By Assumpton 2, v((a()) 2 ) s ndependent of the ntal state. Let V be the convex full of these values: V co(fv((a()) 2 )g (a())2 ); (2) and V Pareto be the Pareto fronter of V : V Pareto fv 2 V 0 2 V wth v 0 v for all and v 0 j > v j for some jg: (3) Ths s the upper bound of the equlbrum payo s. Now, we wll characterze the punshment payo v. Let v () ( : ) be the expected value of the sum of all the players stage game payo wth Pareto weght = (; :::; ) gven the ntal state : v () ( : ) = max E h( ) (a()) 2 t= t u (a t ; t ) j ; (a()) 2 : (4) 2I In addton, let (A () ( : )) 2 be the set of maxmzer: (A () ( : )) 2 = arg max E h( ) (a()) 2 t= t u (a t ; t ) j ; (a()) 2 : (5) 2I 2

13 Suppose that the players agree that they wll take (a () ( : )) 2 2 (A () ( : )) 2, dependng on each player s report about the state. Wthout loss, we assume that f player wth state 2 tells a le that her state s 0, then there exsts 2 such that f the states of players are, then tellng a le nduces a suboptmal strategy: a () ( 0 ; : ) =2 A () ( : ): (6) Otherwse, we consder the followng adjustment. In perod, we assume that the players draw publc randomzaton devce. Wth probablty p, the players ncentve s the same as (5). However, wth probablty p, for each player, player gves the reward () player to ncentvze player to tell the truth about. Now, we de ne (). For each, take a strct () such that, for each 2 I and 2, there exsts 2 such that a strct () 6= a strct ( 0 ; ): (7) on By Assumpton 2, v () ( : ) v () ( 0 : ) max ; ; 0 K strct max a; 0 ju (a; ) u (a 0 ; 0 )j " full (8) wth " full = mn t;a t; t+ q( t+ j t ; a t ): Gven (a strct ()) 2 and K strct, () s the summaton of () (): () () () s de ned as = P () (), and ;t () = K strct ()x [; a strct ()]( ;t ; a ;t ; y ;t ; ;t+ ): 3

14 Now, order the state pro les arbtrarly: ; :::; jj. If we take 8 >< >: K strct ( jj K strct ( jj ) > K strct 0 K strct ; ) > K strct K strct + 2 K strct 0 ;. K strct ( ) > Kjj strct K strct + P jj n 0 =0 2 K n strct 0 wth max jx [; a]( ;t ; a ;t ; y ;t ; ;t+ )j ; ;;a; ;t ;a ;t ;y ;t ; ;t+ then, n each state, by (8), each player has the ncentve to collaboratvely take a strct (). Ths means, by (7), each player has the strct ncentve to tell the truth about. If we take p su cently close to zero, the ex ante value does not change. In addton, as we wll see, ths reward s mplementable as the change n the contnuaton payo n the n ntely repeated game. 0 Therefore, we can drectly assume (6). Now, consder the followng L + T -perod ntely repeated game, where L and T wll be spec ed later: the sgnals and states are dstrbuted by q(y t ; t+ j t ; a t ) as n the orgnal game. For each perod t = ; :::; L + T,. rst, each player sends a message m ;t 2. Intutvely, player sends the message about her state; 2. second, each player takes an acton a ;t 2 A. The payo s from each perod t depend on whch perod the players are n:. n perod t = ; :::; L, player s payo s n2i t u n (a t ; t ) + ;t ; (9) 0 As wll be seen, player wll obtan the nformaton own by players ( ; ) n the repeated game so that player can calculate x [; a]( ;t ; a ;t ; y ;t ; ;t+ ). 4

15 that s, as f player yelded the summaton of all the players payo s plus some random varable ;t, whch wll be de ned below; 2. n perod t = L + ; :::; L + T, player s payo s t (u (a t ; t ) + x ( ;t ; a ;t ; y ;t ; ;t+ )) : (0) Note that player s payo s augmented by a varable x, whch depends on the nformaton by players ; 3. nally, when ths ntely repeated game s over, player yelds L+T v () ( L+T + : ) L+T v () ( : ) () wth some xed. In the n ntely repeated game, f the players trgger the punshment, they play the punshment phase for L + T perods. The players know that, after the punshment phase s over, then they wll take an acton pro le to mplement the average payo v () ( : ). What matters for the ncentves untl perod L + T s whch state L+T + realzes n perod L + T +, whch s represented by (). Snce we want to characterze the punshment payo v, we wll nd (preferably low) v such that there exst L, T, (( ;t ) L t=) 2I and (x ) 2I so that the average payo from the last T perods n the above L + T -perod ntely repeated game s approxmately equal to v for su cently large. Frst, we pn down ( ;t ) L t= and strategy pro le n the rst L perods: Let h t t ;l ; (m n;l ) n2i ; a ;l ; y ;l ; l= ;t be player s hstory n perod t wth h = and H t be the set of possble hstores of player, H t. Player s strategy L spec es what message player should send when player sends 5

16 m ;t 2 n perod t: L : [ L t=h t! ( ), and what acton player should take when player takes an acton gven h t and the message pro le n perod t, m t (m n;t ) n2i 2 : L : [ L t=(h t )! (A ). Note that, snce player s ntal hstory contans the state pro le, the equlbrum s ex post regardng the ntal state, and that snce players the truth about ; as wll be seen, h t contans ; = m ;. tell For a moment, we assume that the gan from one-shot devaton n perod t = ; :::; L from L for the contnuaton payo from perod L+ to perod T +L s bounded by ( " full ) L t K, rrespectvely of L, T and. For example, f the equlbrum strategy from perod L + only uses the nte past hstory, ths condton s sats ed. Especally, the exstence of Markov perfect equlbra from perod L + guarantees ths condton. After we pn down the strategy pro le n the rst L perods, L, we wll derve the condton on (x ) such that we can nd a contnuaton strategy for each player from perod L+;T +L L + to perod T + L, such that the entre strategy pro le for T + L perods L L+;T +L ; s an equlbrum and the one-shot devaton gan from L n perod t on 2I the contnuaton payo from perod L + s bounded by ( " full ) L t K. Remember that player s payo n the rst L perods s (9). We de ne ;t and player s strategy n perod t recursvely by backward nducton for t = L; L purpose, we order the state pro les arbtrarly: ; :::; jj. ; :::;. For that In perod t = L, ;t s the summaton of ;t (): ;t = P ;t(). Now, we de ne ;t () nductvely wth respect to :. we de ne ;t ( ) = K t ( )x [ ; a () t ( )]( ;t ; a ;t ; y ;t ; ;t+ ) (2) wth arbtrarly xed a () t ( ). Snce we assume that the devaton gan for the contnuaton payo from perod L + s bounded by K, from Assumpton 5, wth su cently large K t ( ), f the state pro le s, then each player has the ncentve to collaboratvely take a () t ( ); 6

17 2. suppose that the current state pro le s 2 and that player s objectve s to maxmze n2i t u n (a t ; t ) + ;t ( 2 ): (3) By Assumpton 5, there exsts a () t ( 2 ) that maxmzes (3) for all 2 I. De ne ;t ( 2 ) = K t ( 2 )x [ 2 ; a () t ( 2 )]( ;t ; a ;t ; y ;t ; ;t+ ) so that, wth su cently large K t ( 2 ), f the state s 2, then each player has the ncentve to collaboratvely take a () t ( 2 ), even after takng the devaton gan for the contnuaton payo from perod L + nto account; 3. recursvely, suppose that the current state pro le s n and that player s objectve s to maxmze n2i t u n (a t ; t ) + n 0 <n ;t ( n0 ) and de ne the maxmzer a () t ( n ) whch s common for all 2 I. De ne ;t ( n ) = K t ( n )x [ n ; a () t ( n )]( ;t ; a ;t ; y ;t ; ;t+ ): If we take 8>< >: K t ( jj ) = K 0 K; K t ( jj ) = K K + 2 K 0 ;. K t ( jj n ) = K n K + P n n 0 =0 2 K n 0;. K t ( ) = K jj K + P jj n 0 =0 2 K n 0 (4) wth max jx [; a]( ;t ; a ;t ; y ;t ; ;t+ )j ; ;;a; ;t ;a ;t ;y ;t ; ;t+ 7

18 then, n each state, each player has the ncentve to collaboratvely take a () t (). In perod t = L ;t () nductvely:, ;t s the summaton of ;t (): ;t = P ;t(). Now, we de ne. suppose that the current state pro le s and that player s objectve s to maxmze n2i t u n (a t ; t ) + n2i t+ u n (a t+ ; t+ ) + ;t+ : {z } payo from the next perod Note that the contnuaton payo from perod L + s gnored. By Assumpton 5, there exsts a maxmzer a () t ( 2 ) whch s common for each 2 I. We de ne ;t ( ) = K t ( )x [ ; a () t ( )]( ;t ; a ;t ; y ;t ; ;t+ ) so that, f the state s, then each player has the ncentve to collaboratvely take a () t ( ) even after takng the devaton gan for the contnuaton payo from perod L + nto account; 2. recursvely, suppose that the current state pro le s n and that player s objectve s to maxmze n2i t u n (a t ; t ) + n 0 <n ;t ( n0 ) + n2i t+ u n (a t+ ; t+ ) + ;t+ and de ne the maxmzer a () t ( n ) whch s common for all 2 I. De ne ;t ( n ) = K t ( n )x [ n ; a () t ( n )]( ;t ; a ;t ; y ;t ; ;t+ ): By backward nducton, n perod t, ;t = P ;t() and ;t () s de ned as follows: 8

19 suppose that the current state pro le s n and that player s objectve s to maxmze n2i t u n (a t ; t ) + L ;t ( n0 ) + n2i t0 u n (a t 0; t 0) + ;t 0 n 0 <n t 0 =t+ {z } payo from the next perod to perod L and de ne the maxmzer a () t ( n ) whch s common for all 2 I. De ne ;t ( n ) = K t ( n )x [ n ; a () t ( n )]( ;t ; a ;t ; y ;t ; ;t+ ): By our assumpton that the gan from one-shot devaton n perod t = ; :::; L from L for the contnuaton payo from perod L + to perod T + L s bounded by ( " full ) L t K, we can take K t () satsfyng (4) wth K replaced wth ( " full ) L t K. Let us de ne the equlbrum strategy L n perod t. Let ; ; :::; ;N(;t) be the partton of so that, for any n 2 f; :::; N(; t)g, for any two sgnals ; 0 of player n the same partton ;n, for any realzaton of the other players states 2, a () ;t () s the same: a () ;t ( ; ) = a () ;t (0 ; ). In perod t, the message space s whch partton her state s n. Then, each player tells the truth and takes a () ;t ( t). Note that f each player j 2 nforms player of whch partton j;n player j s state j;l3 s n, then player can gure out a () ;t ( t) snce f a () ;t ( ;t; j;t ; (;j) ) 6= a () ;t ( ;t; 0 j;t; (;j) ) for some (;j), then a () j;t ( j;t; j ) = a () j;t (0 j;t; j ) for some j, whch means j;t and 0 j;t are n the d erent parttons. Snce L s Markov, the ncentve n perod s convergng to (4) wth rate ( " full ) L. By (6), for su cently large L, each player has the ncentve to tell the truth about the ntal state, that s, n perod, the partton s nest: j ;n j = for all n = ; :::; N(; t). Now, we derve the constrant on (x ) 2I. The rst condton s that x ( ;t ; a ;t ; y ;t ; ;t+ ) 0 (5) for all 2 I, ;t ; ;t+ 2, a ;t 2 A and y ;t 2 Y. As wll be seen n Secton 6, x 9

20 corresponds to the margnal e ect of players s hstory on player s contnuaton payo. When we consder the punshment payo, we consder the case where players are n the worst equlbrum strategy aganst player. Hence, players cannot decrease player s contnuaton payo by gong to a worse equlbrum. The second condton s about the equlbrum condton n the last T perods n the L + T -perod ntely repeated game. For any K <, we say (x ) 2I 2 ( K) f, n addton to the rst constrant (5), gven ( L ) 2I de ned above gven K, for su cently large and any 2 I, L and T, there exsts a strategy pro le ( n) n2i from perod L + to T + L satsfyng the followng three constrants:. suppose the players would know t for t = L + ; :::; T + L. There exsts a Markov strategy (a()) such that the average payo for these T perods s ncluded n the feasble payo set V de ned n (2). Otherwse, the punshment payo de ned from the average equlbrum payo n these T perods s not feasble; 2. on equlbrum path untl perod L, that s, after player takng L, gven player s hstory h L+ L ;t ; (m n;t ) n2i ; a ;t ; y ;t ; t= ;L+, the other players equlbrum strategy untl perod L, ( L n) n2 perod L + to perod T, ( n) n2 2 arg max 2 E 6 4, and the other players equlbrum strategy from, s player s best response: P T +L t=l+ t (u (a t ; t ) + x ( ;t ; a ;t ; y ;t ; ;t+ )) + L+T v() ( L+T + :) v () ( :) j h L+ ; ( L n) n2i ; ; ( n) n ; 3. for each perod t = ; :::; L, for any hstory h t, player cannot mprove her value from perod L + to perod T + L by one shot devaton n perod t by more than ( " full ) L t K: Gven h t, suppose that player sends m ;t and takes a (m t ) dependng on the other players message and that, from the next perod, player takes the contnuaton strategy accordng to L : L j h t ; m ; a (m t ); y ;t ; ;t+. Then, the expected 20

21 contnuaton payo from perod L + s 2 E 6 4 P T +L t=l+ t (u (a t ; t ) + x ( ;t ; a ;t ; y ;t ; ;t+ )) + L+T v() ( L+T + :) v () ( :) j h t ; m ;t ; a (m t ); ( L n) n2i ; ( n) n2i : We assume that the contnuaton payo s not mproved by more than ( " full ) L t K: 2 E E 6 4 P T +L t=l+ t (u (a t ; t ) + x ( ;t ; a ;t ; y ;t ; ;t+ )) ( " full ) L t K: + L+T v() ( L+T + :) v () ( :) j h t ; m ;t ; a (m t ); ( L n) n2i ; ( n) n2i P T +L t=l+ t (u (a t ; t ) + x ( ;t ; a ;t ; y ;t ; ;t+ )) + L+T v() ( L+T + :) v () ( :) j h t ; ( L n) n2i ; ( n) n2i Here, the second lne represents the payo n the case where player obeys L after h t. Note that, gven these constrants, ( L n) n2i s an equlbrum strategy, takng the e ect on the contnuaton payo from perod L + nto account; 4. let v L;T 2 = E 6 4 ; (x ) 2I ; ( L n; n) n2i ( ) P L t= t (u (a t ; t ) + ;t ) + ( ) P T +L t=l+ t (u (a t ; t ) + x ( ;t ; a ;t ; y ;t ; ;t+ )) + L+T v () ( L+T + : ) L+T v () ( : ) 3 j ; ( L n; n) n2i 7 5 be player s equlbrum average payo for the above T + L-perod ntely repeated 2

22 game. Ths does not depend on the ntal state by more than ( ) K: v L;T ; (x ) 2I ; ( L n; n) n2i v L;T 0 ; (x ) 2I ; ( L n; n) n2i ( ) K: Snce ( L n) n2i s Markov, ths s sats ed f ( n) n2i depends on the nte past of the hstory observed before perod L. Now, let us de ne the punshment payo v. Now we consder the order of takng lmt carefully. Frst, gven K and L, we can de ne ( L ) L = ndependently of T and ( ) 2I. Gven ths, ( K) s well de ned. For (x ) 2I 2 ( K), for each and T, we can nd ( n) n2i satsfyng Condtons 2 and 3 above. Gven ( n) n2i for each and T, take the lmt where converges to one rst, then T dverges to, and nally take L dverges to : v((x ) 2I ) = lm L! lm T! lm! vl;t ; (x ) 2I ; ( L n; n) n2i : Snce ( K) s bounded by Condton, takng a convergence subsequence f necessary, ths lmt s well de ned. In addton, Condton 4 mples that v L ((x ) 2I ) s ndependent of the ntal state. Snce the perods n Let V fv : v = v ((x ) 2I ) for some K and (x ) 2I 2 ( K)g be the set of all possble punshment payo s, changng (x ) 2I satsfyng the condtons (x ) 2I 2 ( K) for some K. Fnally, let F = fv 2 R N : v 2 co(v Pareto ) and there exsts v 2 V such that v v for all g be the set of payo s n the convex hull of the Pareto fronter such that all the players yeld the payo s no less than the punshment payo v. We can show that F s ncluded n the set of equlbrum payo s: Theorem Suppose Assumptons, 2, 3, 4 and 5 are sats ed. Then, for any v 2 nt(f ) and " > 0, there exsts such that there exsts a sequental equlbrum payo whose equlbrum payo pro le v SE sats es kv v SE k < " f >. 22

23 From now on, we x v 2 nt(f ), v 2 V such that v v for all, and " > 0 arbtrarly. Before proceedng to the proof, let us comment on the de nton of v. Snce we do not know the structure of the stochastc games wth prvate state, t s hard to attan a general characterzaton of the punshment payo. Gven ths d culty, we leave the attempt to smplfy the de nton of v n a general settng for future research. Yet there are at least four examples where we can calculate at least one v easly and use ths v as a punshment payo. Frst, n repeated games, that s, f there s only one state pro le (jj = ), then any statc Nash equlbrum payo v sats es the condtons by we takng x ( ;t ; a ;t ; y ;t ; ;t+ ) = 0 for all ;t, a ;t, y ;t, ;t+. Second, suppose that the set of possble states s the same for all the players: = for all. Moreover, suppose ( ) 2I s drawn as follows: gven acton pro le a t and state pro le t, the prvate sgnal pro le (y ;t ) 2I and underlyng publc state t+ 2 s drawn by the condtonal jont dstrbuton functon q: q (y ;t ) 2I ; t+ j a t ; t : Gven t+, each player s state s determned so that ;t+ s equal to t+ wth a hgh probablty: for each, ndependent of all the other varables, 8 < t+ wth probablty "; ;t+ = : t+ wth probablty jj " : The ntuton s that t+ s the true state that s the same for all the players, and each player observes a slghtly nosy sgnal of t+. In ths model, for any q such that the margnal dstrbuton over y ;t and t+ has full support, for su cently small ", we can take v as the value of a Markov perfect equlbrum n the game where the state 2 s publcally observable and the state transton s determned by q (y ;t+ ) 2I ; t+ j a t ; t by takng x ( ;t ; a ;t ; y ;t ; ;t+ ) so that the payo of acton 23

24 a gven a n the stochastc game wth prvate states s the same as that n the stochastc game wth publcally observable states. Thrd, f state transton s ndependent of acton pro les and there exsts an ex post equlbrum n the one-shot game, then we can take v as the value of repeatng the ex post equlbrum calculated by the statonary dstrbuton of the state transton and x ( ;t ; a ;t ; y ;t ; ;t+ ) = 0 for all ;t, a ;t, y ;t, ;t+. Fourth, agan suppose that the state transton s ndependent of acton pro les. Further, assume that the montorng of actons s ndependent of the state: q (y t ; t+ j a t ; t ) = q ( t+ j t ) q (y t j a t ). Fx any pure acton pro le a. By Assumpton 4 and the fact that the montorng of actons s state ndependent, there exsts x (y ) such that x (y ) 0 for all y 2 Y ; E [u (a; ) + x (y ) j a ; ] max a 2A E u (a; ) + x (y ) j a ; a ; for all 2 : Then, we can take v as the expected value of E [u (a; ) + x (y ) j a ; ] by usng the statonary dstrbuton of gven a. In partcular, f the montorng q (y t j a t ) s almost perfect, then E [u (a; ) + x (y ) j a ; ] s at most u (a ; ) + max a 2A ; 0 2 u a ; a ; 0 u (a ; 0 ) : (6) Note that, snce the state transton s acton ndependent and players acton, there s no ncentve for player to devate to change players state dstrbuton n the next perod. take a constant s nformaton or the In partcular, n the two-player repeated game wth jij = 2 and jj =, by takng a such that a n s the pure mnmaxng strategy for each player n, (6) s equal to player s mnmax payo u (a ) + max a 2A u a ; a u (a ) = max a 2A u a ; a. To satsfy (5), the expected value of x can be strctly postve and the value wth prvate states s slghtly d erent from that wth publc states. However, for su cently small ", ths adjustment s also su cently small. 24

25 6 Phase Belef Free We modfy the phase-belef-free equlbrum n repeated games so that we can apply t to the stochastc games. To show Theorem, t su ces to nd T P such that there exst the phase-belef-free equlbrum strategy and the reward functon for T P -perod revew phases, satsfyng the followng four constrants: lettng h T P + be player s hstory untl the end of T P (the formal de nton wll be o ered below), for each 2 I and 2,. condtonal on the ntal state pro le, both G and B are optmal: G ; B PTP 2 arg max E t= t u (a t ; t ) + x (h T P + ) + T P (vx ; TP v x + ; ) j ; ; x ; where v; x = max PTP E T P t= t u (a t ; t ) + x (h T P + ) + T P (vx ; TP v x + ; ) j ; ; x s the value for player wth the ntal state. The last term does not exst n repeated games. Ths term represents the change n the value at the begnnng of the next phase based on the state realzaton; 2. player s state solely controls player s value ndependently of : (a) player s value s ndependent of x ( ) : there exsts v x ; x ( ) ; x 0 ( ) 2 fg; BgN and 2, such that, for each v x ; v x ;x ( ) ; = v x ;x 0 ( ) ; ; (b) the d erence of player s value v x ; exsts such that, for all 2, s ndependent of the ntal state: there = v G ; v B ; : 25

26 Intutvely, player controls player s contnuaton payo by swtchng x = G and x = B. Ths condton means that the e ect of player s state changes on player s payo s ndependent of the state pro le. That s, wthout knowng, player can mplement any change of player s contnuaton payo as long as the change sats es Condton 4 below (feasblty); (c) there exsts p G such that p G v G ; + ( p B ) v B ; v < " (7) for all 2. That s, f player determnes the ntal x such that x = G wth probablty p G, then the equlbrum payo s close to the targeted payo v; 3. the feasblty condton s sats ed: for all h T P +, 8 >< >: lm TP! lm! G (h T P + ) 0; B (h T P + ) 0; T P T P x (h T P ) < for all x. Let us call these four condtons Su cent Condtons, 2, 3 and 4. For the rest of the paper, we wll show that these su cent condtons are sats ed. 7 Heurstc De nton of the Reward Functon Now, we start our attempt to de ne x of x (h T P ), whch needs to be mod ed later. (h T P ). In ths secton, we gve a heurstc de nton Snce v 2 nt(co(v Pareto )), for su cently large, for each x 2 fg; Bg N, there exst x 2 + f 2 R N ++ : kk = g, 2 (v x ( : )) 2 and (a x ( : )) 2 such that 2 Snce we consder the strct Pareto optmalty n (3), we have x > 0 for all. 26

27 . (v x ( : )) 2 s the socal welfare wth current state pro le and Pareto weght x, where (v x ( : )) 2 s maxmzed by Markov strategy (a x ( : )) 2 : v x ( : ) = max a E [( ) x u(a ; ) + v x ( 2 : ) j a ; ] ; a x ( : ) = arg max a E [( ) x u(a ; ) + v x ( 2 : ) j a ; ] (8) for all. Here, wthout loss, we assume that a x ( : ) s the unque maxmzer of (8): otherwse, by changng the utlty functon by a reward smlar to ;t but wth a very small K t ( n ), we can break a te wthout a ectng the equlbrum payo s. 2. f player s state s G, that s, f player s equlbrum payo needs to be hgh, then for su cently large L, P v < E L L t= t u (a t ; t ) j (a x ( : )) 2 ; (9) for all ; 3. f player s state s B, that s, f player s equlbrum payo needs to be low, then for su cently large L, P v > E L L t= t u (a t ; t ) j (a x ( : )) 2 ; (20) for all. Let us call these three condtons Condtons, 2 and 3 for Pareto weght x. From these observatons, we try to satsfy the condtons n Secton 6. Frst, Condton for Pareto weght x mples that, f player s nstantaneous utltes were u (a t ; t ) + n2 x n x ~u n;t c x wth some constant c x forever, players told the truth about ;t and took a x (^ ;t ; ;t ) 27

28 wth ^ ;t beng player s message about ;t, then t would be optmal for player to tell the truth about ;t and take a x ( ;t ; ^ ;t ). Now, take c x be the expectaton of P x n n2 ~u x n;t under (a x ( : )) 2. Then, Condtons 2 and 3 for Pareto weght x mply that, for su cently large L, for all 2,. f player s state s G, then " P v < E L L l= t u (a t ; t ) + n2 x n x ~u n;t c x! # j (a x ( : )) 2 ; ; 2. f player s state s B, then " P v > E L L l= t u (a t ; t ) + n2 x n x ~u n;t c x! # j (a x ( : )) 2 ; : Suppose, for a moment, that T P would be T wth 8 < x (h T + ) = : 4uT B 4uT B + P T t= t P n2 + P T t= t P n2 x n x ~u n;t c x x n ~u x n;t c x f x = G; f x = B (2) wth u > 0. Let us check Su cent Condtons n Secton 6: assumng that x and take take a x ( ;t ; ^ ;t : )) 2 n each perod t, tell the truth. player wants to maxmze " max E PT T t= t u (a t ; t ) + n2 By a ne transformaton, we have x n x ~u n;t c x! # + T (vx ; T + v; x ) j ; ; x : x v; x = max E PT T t= t x u(a t ; t ) + T (x v; x T + x v; x ) j ; ; x : (22) Therefore, by Condton for Pareto weght x, x s optmal, as desred; 28

29 2. wth (9) and (20) mply that, for su cently large B, v B;x ( ) ; < v < v G;x ( ) ; all x ( ) 2 fg; Bg N. In addton, by Assumpton 2, there exsts K such that v; x v x ; < K 0 ( ). Therefore, f player wth x = G subtracts a reward based on x ( ) and wth x = B adds a reward based on x ( ), we can make sure that, for each 2 I, for (a) player s value s almost ndependent of x ( ) : for any x ( ) ; x 0 ( ) 2 fg; BgN and 2, (b) the d erence of player s value v x ; v x ;x ( ) ; v x ;x 0 ( ) ; < K ( ); s almost ndependent of the ntal state: there exsts > 0 such that, for all 2 and x ( ) ; x 0 ( ) 2 fg; BgN v G;x ( ) ; v B;x0 ( ) ; < K ( ); (c) f we take su cently large, Su cent Condton 2-(c) s sats ed. 3. lm T! lm! T T x expectaton, (h T + ) = 0, as desred. In addton, by de nton of cx, n E G (h T + ) j x ut E B (h T + ) j x ut. Hence, by the central lmt theorem, wth a hgh probablty, G (h T + ) 0 and B (h T + ) 0. However, there could be a rare hstory wth whch G (h T + ) > 0 or B (h T + ) < 0. We wll modfy x (h T + ) so that Su cent Condtons 2-(a) and 2-(b) are exactly sats ed and Su cent Condton 3 s sats ed after any hstory. 29

30 8 State Independence We rst x the problems n Su cent Condton 2. Suppose we nsert the followng L perods before the T -perod phase: n perod t = ; :::; L, player gves the reward ~u n;t + ;t ; (23) where ;t s analogously de ned as ;t n Secton 5 wth K replaced wth n2 v; x L+ (L) v x ;L+(L) K > lm max 0 ; (24)! ;x; L+ ; 0 L+ where v x ; L+ (L) s player s value from perod L + on wth the state n perod L + beng L+. Snce we change the reward between perod to perod L, ths value v x ; L+ (L) may be d erent from v x ; L+. As n Secton 5, j ;t j can be bounded by ( " full ) t L K. Hence, n perod t = ; :::; L, the followng strategy s an equlbrum: n perod t, the message space s whch partton her state s n, as n Secton 5. Each player has the strct ncentve to tell the truth and takes a ;t ( t ). For su cently large L, by (6), from perod to perod L=2, the partton s nest, j ;n j = for all n = ; :::; N(; t), and a ;t ( t ) = a ( t : ). Now, consder the perods close to the last perod of the current phase t = T + L. For su cently large L, the players play the strategy a ( t : ) at shortest for L=2 perods n the next phase. Therefore, the e ect of the state pro le n perod T + L + (the rst perod of the next phase) s not vx ; T +L+ v x ; but close to v() ( L+T + :) v () ( :). Let v; x () v x ;() K 2 > lm max 0! ;x; ; 0 be the e ect of the state pro le n perod T + L + on the ncentve, where v x ; () s the value n the rst perod. To deal wth ths problem, we add addtonal L perods at the end of the current phase. (25) 30

31 In perod t = L + T + ; :::; L + T + L, player gves the reward n2 t x n x ~u n;t + x ;t; whch s smlar to (23) but s adjusted based on x so that player s ncentve s close to that n (8). Spec cally, we de ne x ;t for t = L + T + ; :::; L + T + L as follows. Agan, we order the state pro les arbtrarly: ; :::; jj. In perod t = L + T + L, x ;t s the summaton of x ;t(): x ;t = P x ;t(). Now, we de ne x ;t() nductvely:. we de ne x ;t( ) = x Kt x ( )x [ ; a x t ( )]( ;t ; a ;t ; y ;t ; ;t+ ) wth arbtrarly xed a x t ( ). From Assumpton 5, wth su cently large Kt x ( ), f the state s, then each player has the ncentve to collaboratvely take a x t ( ). 2. suppose that the current state pro le s 2 and that player s objectve s to maxmze t u (a t ; t ) + n2 t x n x u n (a t ; t ) + x ;t( 2 ): By the a ne transformaton, ths s equvalent to maxmzng n2i t x nu n (a t ; t ) + x x ;t( 2 ) = n2i t x nu n (a t ; t ) + K x t ( )x [ ; a x t ( )]( ;t ; a ;t ; y ;t ; ;t+ ): (26) By Assumpton 5, there exsts a x t ( 2 ) that maxmzes (26) for all 2 I. De ne x ;t( 2 ) = x K x t ( 2 )x [ 2 ; a x t ( 2 )]( ;t ; a ;t ; y ;t ; ;t+ ) so that, wth su cently large K x t ( 2 ), f the state s 2, then each player has the ncentve to collaboratvely take a x t ( 2 ), even after takng the contnuaton payo from the next phase nto account; 3

32 3. recursvely, suppose that the current state pro le s n and that player s objectve s to maxmze n2i t x nu n (a t ; t ) + x ;t( n0 ) n 0 <n and de ne the maxmzer a x t ( n ) whch s common for all 2 I. De ne x ;t( n ) = x K x t ( n )x [ n ; a x t ( n )]( ;t ; a ;t ; y ;t ; ;t+ ): By backward nducton, n perod t, x ;t s the summaton of x ;t(): x ;t = P x ;t() and x ;t() s de ned as follows: suppose that the current state pro le s n and that player s objectve s to maxmze n2i t x nu n (a t ; t ) + n 0 <n x ;t( n0 ) + L+T +L t 0 =t+ n2i t0 x nu n (a t 0; t 0) + ;t 0: Then, by Assumpton 5, there exsts a maxmzer a x t ( n ) whch s common for each 2 I. We de ne x ;t( n ) = x K x t ( n )x [ n ; a x t ( n )]( ;t ; a ;t ; y ;t ; ;t+ ): Now, consder how large K x t () should be. By Assumpton 2 and (25), the e ect of the contnuaton payo from the next phase on the ncentve n perod t s bounded by ( " full ) L+T +L t K2 Hence, Kt x () should satsfy (4) wth K replaced wth ( " full ) L+T +L t K2. Therefore, we can show that the ncentve n perod L + T s convergng to (8) wth rate ( " full ) L. Let us de ne the equlbrum strategy n perod t. Let ; ; :::; ;N(;t) be the partton of so that, for any n 2 f; :::; N(; t)g, for any two sgnals ; 0 of player n the same partton ;n, for any realzaton of the other players states 2, a ;t () s the same: a ;t ( ; ) = a ;t ( 0 ; ). In perod t, the message space s whch partton her state s n. Then, each player tells the truth and takes a ;t ( t ). 32

33 For su cently large L, snce between perod L + and perod L + T, each player has the ncentve to take a x ( t : ), as desred. Fnally, let us verfy that we can take T, L, K and K 2 consstently wth (24) and (25) snce these values depend on L, T, K and K 2 : for any " > 0, there exst L, K and K 2 such that, for su cently large T, lm lm max! ; 0 2;x2fG;Bg N ;2I lm max! ; 0 2;x2fG;Bg N ;2I max! ; 0 2;x;x 0 2fG;Bg N ;2I v x ; 0(L : T; L; K ; K 2 ) v;(l x : T; L; K ; K 2 ) K (27) ; v x ; 0( : T; L; K ; K 2 ) v;( x : T; L; K ; K 2 ) K (28) 2 ; v x ; ( : T; L; K ; K 2 ) v; x0 ( : T; L; K ; K 2 ) v x ; 0( : T; L; K ; K 2 ) v x0 ; 0( : T; L; K ; K < " (29) 2 ) Here, we explct wrte down the dependence of the value on T, L, K and K 2. Frst, guess the polcy functon such that the players take (a x ( t : )) for perod t 2 fl + ; :::; L + T g. Second, snce ths strategy s Markov, for any L, K and K 2, for su cently large T, Assumpton 2 mples that lm max! ; 0 2;x2fG;Bg N ;2I v x ; 0(L : T; L; K ; K 2 ) " full max ;a;a 0 ;; 0 ju (a; ) u (a 0 ; 0 )j + ( " full ) T L K + K 2 : v x ;(L : T; L; K ; K 2 ) Therefore, there exsts K such that, for su cently large T, (27) s sats ed. Fx such K. Thrd, gven (27), the players take (a () t ( t )) for perod t 2 f; :::; Lg. Snce ths strategy s Markov, for any K 2, for su cently large T and L, Assumpton 2 mples that lm max! ; 0 2;x2fG;Bg N ;2I v x ; 0( : T; L; K ; K 2 ) v x ;( : T; L; K ; K 2 ) " full max ;a;a 0 ;; 0 ju (a; ) u (a 0 ; 0 )j + L K + ( " full ) T L K + K 2 : Therefore, for any L, there exsts K 2 such that, for su cently large T, (27) s sats ed. 33

34 Fnally, for su cently large L, snce (a () t ( t )) and ;t n perod t 2 f; :::; Lg s ndependent of x, lm max! ; 0 2;x;x 0 2fG;Bg N ;2I v x ; ( : T; L; K ; K 2 ) v; x0 ( : T; L; K ; K 2 ) v x ; 0( : T; L; K ; K 2 ) v x0 ; 0( : T; L; K ; K 2 ) ( " full) L " full max ;a;a 0 ;; 0 ju (a; ) u (a 0 ; 0 )j + ( " full ) T L K + K 2 : Hence, for any K and K 2, for su cently large L and T, (29) s sats ed. Let us change the structure of the revew phase more. Before the rst perod, we nsert the followng L + L + L 4 perods:. from perod to perod L(2) L, the players tell the truth about the parttons about t and take a () t ( t ); 2. from perod L(2) + to perod L(3) L(2) + T, the players take (a x ( : )) ; 3. from perod L(3) + to perod L(4) L(3) + L, the players tell the truth about the parttons about t and the players take (a x t ()). Let x The reward s be the strategy above. 8 < : L(2) + + t= 4uT B f x = G 4uT B f x = B L(4) t=l(3)+ t n2 n2 ~u n;t + ;t! + t x n x ~u n;t + x ;t L(3) t=l(2)+! t n2 x n x ~u n;t c x! 34