The Folk Theorem for Games with Private Almost-Perfect Monitoring

Transcription

1 The Folk Theorem for Games wth Prvate Almost-Perfect Montorng Johannes Hörner y Wojcech Olszewsk z October 2005 Abstract We prove the folk theorem for dscounted repeated games under prvate, almost-perfect montorng. Our result covers all nte, n-player games satsfyng the usual full-dmensonalty condton. Mxed strateges are allowed n determnng the ndvdually ratonal payo s. We assume no cheap-talk communcaton between players and no publc randomzaton devce. KEYWORDS: Repeated games, prvate montorng, folk theorem. We thank semnar audences at the Unversty of Chcago, the London School of Economcs, New York Unversty, Northwestern Unversty, Penn State Unversty, Prnceton Unversty, Rutgers Unversty, Unversty College London, Unversté de Toulouse and WZB, Berln, the 15th Summer Festval on Game Theory at Stony Brook, the 2nd World Congress of the Game Theory Socety n Marselle, and Paul Hedhues for hs dscusson. We would also lke to thank two anonymous referees and Davd Levne for ther comments that led to substantal mprovements n the exposton. y Department of Manageral Economcs and Decson Scences, Kellogg School of Management and HEC School of Management, Pars and CEPR. j-horner@kellogg.northwestern.edu. z Department of Economcs, Northwestern Unversty. wo@northwestern.edu. 1

2 1 Introducton The central result of the lterature on dscounted repeated games s probably the folk theorem (Fudenberg and Maskn (1986)): wth only two players, or when a full dmensonalty condton holds, any feasble payo vector Pareto-domnatng the mnmax pont of the stage game s acheved by some subgame-perfect equlbrum of the n ntely repeated game provded that the players are su cently patent. Under some dent ablty condtons, ths result has been subsequently generalzed by Fudenberg, Levne and Maskn (1994) to the case n whch players do not observe the chosen acton pro le, but only a publc sgnal that s a stochastc functon of the acton pro le. For that purpose t su ces to consder a restrcted class of sequental equlbra. In perfect publc equlbra (PPE), players contnuaton strategy only depends on the publc hstory, that s, on the hstory of publc sgnals. The analyss of PPE s tractable because after any hstory the contnuaton strateges correspond to an equlbrum n the orgnal game, so that the set of PPE payo s can be characterzed by technques borrowed from dynamc programmng (see Abreu, Pearce and Stacchett (1990)). 1 Thus, common knowledge of relevant aspects of players hstores plays an essental role n the proofs of the folk theorem so far. Ths sort of common knowledge s mssng n games wth prvate montorng. In such games, each player only observes a prvate sgnal that s a stochastc functon of the acton pro le. If, for each acton pro le, the sgnals of all players are perfectly correlated, then the montorng s publc, and f moreover the sgnals are perfectly correlated wth the acton pro le, the montorng s perfect. Yet, n general, sgnals are nether perfect nor publc, so that players share no publc nformaton to coordnate contnuaton play. Ths paper shows that the folk theorem s robust. It remans vald under the standard full-dmensonalty assumpton, provded only that the prvate sgnals are su cently close to perfect. In partcular, sgnals are not restrcted to be almost-publc or condtonally ndependent. 1 In general, however, the set of sequental equlbrum payo s s strctly larger than the set of PPE payo s. See Kandor and Obara (2004) for detals. 2

3 More spec cally, take any nte n-player game whose set of feasble, ndvdually ratonal payo s has non-empty nteror V, where the ndvdually ratonal payo s are determned by consderng (ndependent) mxed strateges. Consder the canoncal sgnal space, n whch a player s set of sgnals s the set of acton pro les of ts opponents. More general sgnal spaces are dscussed n Secton 5. Montorng s "-perfect f, for any player, under any acton pro le a, player obtans sgnal = a wth probablty at least 1 ". The parameter " s the nose level. When " = 0, montorng s perfect. Payo s are dscounted at common factor 2 (0; 1). No publc randomzaton or communcaton devce s assumed. Gven dscount factor, denote by E (; ") the set of average payo vectors n the repeated game that are sequental equlbrum payo s for all "-perfect montorng structures. Ths paper shows that: 8 v2v 9 <1;">0 8 2( ;1) v 2 E (; "). Observe that the result does not post any partcular order of lmts, as t holds for a jont neghborhood of dscount factors and nose levels. In addton, the result states that the payo vector v s exactly acheved, not only approxmated. There are several related contrbutons. Lehrer (1990) obtans e cent equlbra whle consderng tme-average payo s, whle Fudenberg and Levne (1991) requre approxmate optmzaton. The equlbrum strateges proposed n these papers are no longer equlbrum strateges once dscountng and exact optmzaton are ntroduced. Compte (1998), Kandor and Matsushma (1998), Aoyag (2002) and Fudenberg and Levne (2002) prove versons of the folk theorem whle allowng players to communcate. Whle a realstc assumpton n many applcatons, communcaton rentroduces an element of publc nformaton that s somewhat at odds wth the motvaton of prvate montorng as a robustness test outlned above. Malath and Morrs (2002) prove a folk theorem for almost-perfect montorng, assumng n addton that montorng s also almost-publc. 3

4 Sekguch (1997) acheves the e cent outcome and Bhaskar and Obara (2002) establshes the folk theorem, under almost-perfect montorng, for the specal case of the two-player prsoner s dlemma. They solate a set of (contnuaton) strateges closed under best-response: for any relevant belef a player may have about hs opponent s contnuaton strategy (wthn that set), some strategy wthn the set s a best-response. Usng a d erent approach, Ely and Välmäk (2002) and Pccone (2002) prove the folk theorem under almost-perfect montorng for the twoplayer prsoner s dlemma. They solate a set of (contnuaton) strateges satsfyng a stronger property: for any belef a player may have about hs opponent s contnuaton strategy (wthn that set), any strategy wthn that set s a best-response. Ths approach has been further used by Matsushma (2004) to extend the two-player prsoner dlemma s folk theorem from the case of almost-perfect montorng to the case of condtonally ndependent, but not necessarly almost perfect, montorng. Fnally, Yamamoto (2004) shows, by modfyng the constructon of Ely and Välmäk (2002) and Matsushma (2004), that the e cent outcome can be acheved n a class of N-player games, smlar n structure to the prsoner s dlemma, under almost-perfect as well as condtonally ndependent montorng. Whle the rst, belef-based, approach s more general than the second belef-free approach, t appears less tractable and has not been generalzed so far to other stage games. The beleffree approach has been studed more generally by Ely, Hörner and Olszewsk (2004), whch characterzes the set of payo s that can be acheved usng sequental equlbra satsfyng ths property. For many stage games, ths set of payo s s larger than the convex hull of statc Nash equlbrum payo s, but for almost all stage games, t fals to yeld the folk theorem even under almost-perfect montorng. Although the equlbra studed n ths paper are not belef-free, they retan some features of belef-free equlbra. To get some nsght nto the constructon, consder the case of two players. In each consecutve block of T perods, players use one of two strateges of the T - ntely repeated game. The length T s chosen so that the average payo over the horzon T of each of the four resultng strategy pro les surrounds the average payo vector v to be acheved overall: f player 4

5 uses one of hs two strateges, hs opponent receves at least v whchever of the two strateges he uses hmself; f he uses the other strategy, hs opponent gets no more than v, no matter whch strategy he uses n the T - ntely repeated game. In ths case, player may be called upon to mnmax hs opponent. There are thus two knds of punshments : from one block to the next, player may use the strategy that gves hs opponent a hgh or a low payo ; wthn each block n whch he chooses the strategy gvng hs opponent a low payo, player mnmax hm. may need to Therefore, a player s not nd erent over hs opponent s choce of strategy n a block. However, by a sutable choce of the probablty wth whch a player uses each strategy wthn each non-ntal block (the transton probabltes), as a functon of hs recent hstory and of hs recent strategy (.e. wthn the prevous block), players are nd erent over ther two strateges, and weakly prefer them to all others, at the begnnng of each block. Further, by sutably choosng the probablty wth whch each strategy s used n the ntal block, the payo vector v s acheved. Ths guarantees that belefs are rrelevant at the begnnng of each block. Belef-free equlbra obtan for T = 1. 2 In ths sense, we show that the specal features of the prsoner s dlemma ensurng that the folk theorem obtans wth T = 1 obtan for any stage game, provded one chooses approprately T 1. 3 Sequental ratonalty poses several d cultes when T > 1. After recent hstores that are consstent wth both players havng only observed correct sgnals, a player s belef about hs opponent s recent hstory has a tractable structure: when the nose level s small enough, he assgns probablty almost one to hs opponent havng observed the same recent hstory. Ths s not true for the other recent, erroneous hstores, as hs posteror may then dramatcally vary wth small d erences n the relatve lkelhood of ncorrect sgnals. As a player s best-reply n 2 More precsely, ths s the case for belef-free equlbra usng a constant regme (see Ely, Hörner and Olszewsk (2004)). 3 Thus, T -perod blocks do not serve the purpose of statstcal dscrmnaton between actons, as n Radner (1986) or Matsushma (2004). 5

6 non-ntal perods may depend on hs opponent s recent hstory, specfyng best-responses after such hstores s less tractable. Worse, a player s belef about hs opponent s recent hstory - and thus hs best-response- a pror depends on hs belef about the recent strategy used by hs opponent, whch n turn depends on hs own entre prvate hstory, rather than hs recent one. Ths means that best-reples may depend on a player s entre hstory, destroyng the recursve structure of our constructon. Ths problem s solved as follows. For one of hs two strateges used n each block, the correspondng transton probabltes are chosen so that a player s opponent s nd erent over all strateges wthn the block, not only over the two strategy he actually chooses from. Thus, to compute hs best-reply, a player may always condton on hs opponent usng the other strategy, ndependently of the belefs he actually holds about the recent strategy used by hs opponent. As ths best-reply depends on what ths other strategy spec es, for each possble recent hstory, as well as on the correspondng transton probabltes, ths strategy and the transton probabltes that go along must be determned jontly, by applyng Kakutan s xed pont theorem. Ths guarantees that optmal play after recent hstores s ndeed a functon of that recent hstory only. It leaves the play not explctly spec ed after erroneous hstores (n partcular, we do not know the payo s contngent on such hstores), but t does not pose any major problem. Roughly because, by choosng transton probabltes that yeld lower payo s contngent on erroneous hstores, players can be gven ncentves not to trgger such hstores, and snce such hstores appear wth small probablty, total payo s are not a ected much by the play on erroneous hstores. The case wth more than two players creates addtonal challenges, related to coordnaton and dmensonalty ssues. The constructon n that case s more ntrcate, but we postpone dscusson of t to the relevant secton. Whle the constructon for n > 2 also works for n = 2, t s more natural to ntroduce ths constructon by rst consderng the case of two players. Secton 2 ntroduces the notaton and states the results. Secton 3 presents the constructon for two players, rst under perfect montorng, and then under mperfect prvate montorng. 6

7 Secton 4 presents the constructon for n > 2 players, followng the same two steps as for n = 2. Fnally, Secton 5 relaxes the restrcton on the sgnal set and o ers concludng comments. 2 Notaton and result Consder the followng nte n-person game. Each player = 1; :::; n has a ( nte) acton set A and a ( nte) set of sgnals. Wthout loss of generalty, assume that A contans at least two elements, for all. Throughout Sectons 2 to 4, we mantan the assumpton that = A, where A := A 1 A 1 A +1 A n. Ths assumpton s convenent to measure the dstance of a partcular montorng structure from perfect montorng. For each acton pro le a 2 A := A 1 A n, m ( j a) spec es a probablty dstrbuton over := 1 n. The collecton of probablty dstrbutons over sgnal pro les fm ( j a) : a 2 Ag de nes the montorng structure. For each acton pro le a 2 A, m ( j a) denotes the margnal dstrbuton of m ( j a) over. Thus, m ( j a) s the probablty that player receves sgnal 2 under acton pro le a 2 A. We focus attenton on the case n whch the montorng structure s close to perfect montorng. Followng Ely and Välmäk (2002), we formalze ths noton as follows: for " 0, the montorng structure fm ( j a) : a 2 Ag s "-perfect f for each player and each acton pro le a 2 A, m ( = a j a) 1 ". That s, under any acton pro le, the probablty that a player observes an erroneous sgnal does not exceed ". The perfect montorng structure s a specal case that obtans for " = 0. Observe that ths de nton s stated n terms of margnal dstrbutons only. Therefore, whle ths de nton s consstent wth almost-publc or condtonally ndependent sgnals, t does not mpose any such restrcton. We do not mpose any full-support restrcton ether. Mxed actons are unobservable. For any nte set W, let 4W denote the set of probablty dstrbutons over W. Wth some abuse of notaton, we use 4A := 4A 1 4A n to denote 7

8 the set of (ndependent) mxed acton pro les. Smlarly, 4A := 4A 1 4A 1 4A +1 4A n. No publc randomzaton devce s assumed. Player s realzed payo n the stage game, u : A! R, s a functon of hs acton and sgnal alone, so that hs expected payo g : A! R s gven by g (a) = X m ( j a) u (a ; ). 2 The doman of g s extended to mxtures 2 4A n the usual manner: g () = X (a) g (a), a2a where (a) denotes the probablty assgned to acton pro le a by the mxture 2 4A. Observe that repeated games wth publc montorng are specal cases of ths formulaton. If sgnals are perfectly correlated wth each other, we obtan a game wth mperfect publc montorng, whle under the perfect montorng structure, we obtan a standard game wth perfect montorng. Players share a common dscount factor < 1. All repeated game payo s, both n nte and nte, are dscounted, and ther doman s extended to mxed strateges n the usual fashon; unless explctly mentoned otherwse (as wll occur), all payo s are normalzed by a factor 1, sometmes referred to as the average, or normalzed payo s. Total, or unnormalzed payo s are payo s that are dscounted, but not normalzed. For each, the mnmax payo v v := of player (n mxed strateges) s de ned as mn max g (a ; ). 24A a 2A Choose 2 4A so that v = max a 2A g a ;. The acton s the (not necessarly unque) mnmax acton aganst player, and v smallest payo that the other players can keep player below n the statc game. 4 s the 4 Under some mperfect montorng structures, t may be possble to keep player s payo even lower f n 3, as sgnals may allow players to correlate ther actons wthout beng observed by player. 8

9 Let: U := f(v 1 ; : : : ; v n ) j 9a 2 A, 8; g (a) = v g ; V := Convex Hull of U, and V := Interor of f(v 1 ; : : : ; v n ) 2 V j 8; v > v g. The set V conssts of the feasble payo s, and V s the set of payo s n the nteror of V that strctly Pareto-domnate the mnmax pont v := (v 1; : : : ; v n). We assume throughout that V s non-empty. Gven dscount factor, recall that E (; ") s the set of average payo vectors n the repeated game that are sequental equlbrum payo s for all "-perfect montorng structures. We can now state our man result. Theorem 1: 5 (The Folk Theorem) For any (v 1 ; : : : ; v n ) 2 V, f players dscount the future su cently lttle and the nose level s su cently small, there exsts a sequental equlbrum of the n ntely repeated game where, for all, player s average payo s v. That s, 8 v2v 9 <1;">0 8 2( ;1) v 2 E (; "). The proof uses the followng notatons. A t-length (prvate) hstory for player s an element of H t := (A ) t. A par of t-length hstores s denoted h t. Such a par s also referred to as a hstory. As (prvate) hstores are always ndexed by the relevant player, no confuson should arse. A player s ntal hstory s the null hstory ;. Let H t denote the set of all t-length hstores, H t the set of s (prvate) t-length hstores, H = [ t H t the set of hstores, and H = [ t H t the set of (prvate) hstores for. A repeated-game (behavor) strategy for player s a mappng s : H! 4A. The mxed acton prescrbed by strategy s, gven prvate hstory h t s denoted s [h t ], whle the probablty assgned to acton any a by s [h t ] s denoted s [h t ] (a ). The set of 5 Theorem 1 does not rule out equlbrum payo s outsde V (see footnote 4), but t s possble to show that player s mnmax payo n the repeated game tends to hs stage game mnmax payo v as "! 0. Also, we do not know whether the full dmensonalty condton can be relaxed, or dropped wth only two players. 9

10 all strateges of player n the n ntely repeated game s S, and a strategy pro le s denoted s 2 S := S 1 S n. For any hstory h t 2 H, let s jh t denote the contnuaton strategy derved from s after hstory h t, and s jh 0 the restrcton of s to the set of hstores H 0 H. For T 1, we shall also consder the game repeated T tmes (henceforth smply referred to as the ntely repeated game). The set of all t-length (prvate) hstores of player n the T - ntely repeated game s denoted by H t, the set of all hstores by H T = [ tt H t and the set of (behavor) strateges n the ntely repeated game by S T. For t T, we use the same notaton for contnuaton strateges as n the case of the n ntely repeated game. Three types of repeated game payo s are consdered. Gven strategy pro le s 2 S, player s payo s denoted U (s) n the n ntely repeated game. Gven strategy pro le s 2 S T := S1 T Sn T, player s payo s denoted U T (s) n the ntely repeated game. Fnally, we shall consder the ntely repeated game augmented by a transfer : H+1 T! R at the end of the last perod (dentfyng 1 and n + 1). Gven := ( 1 ; : : : ; n ) and some hstory h T +1, player s payo n ths auxlary scenaro s de ned as U A h T +1; := U T h+1 T + (1 ) T h+1 T, and ts de nton extended to strateges s 2 S T n the usual fashon. Contnuaton payo s gven some prvate hstory h t are denoted U (s j h t ), U T (s j h t ). Gven some strategy pro le s 2 S T := S1 T S T 1 S+1 T Sn T and transfer, let B (s ; ) denote the set of auxlary scenaro best-responses of player. Fnally, gven a set of hstores H E H T, a strategy s 2 S T, a strategy s 2 S T and transfer, let B (s ; j s ) denote the set of strateges that maxmze player s auxlary-scenaro payo aganst s, among all strateges s 2 S T such that s j H E = s j H E. By B(v; ), we mean the ball around payo vector v of radus ; by co W, the convex hull of a set W, and by #W, the cardnalty of the nte set W. 10

11 3 Two-player Games 3.1 From belef-free equlbra to block equlbra As mentoned n the ntroducton, the folk theorem under prvate almost-perfect montorng s already known to hold for the two-player prsoner s dlemma (G; L 0): C D C (1; 1) ( L; 1 + G) D (1 + G; L) (0; 0) The key observaton that makes the analyss tractable n ths case s that each player can ensure, through cooperaton, that hs opponent gets at least one, and through defecton, that he gets at most zero, whether hs opponent cooperates or not. More formally, for each payo vector v 2 (0; 1) 2, there exsts a subset of actons A A, and two elements G ; B 2 4A, all, such that: mn g a ; G > v > max g a ; B : A A To see ths, pck A = A and G = C; B = D, = 1; 2. The acton G s the Good acton that secures one to player zero. Consder now a strategy by player, whle B s the Bad acton that keeps hs payo below that ether plays G or B. Such a strategy may provde ncentves for cooperaton f observatons pontng towards defecton are more lkely to trgger play of B than observatons pontng towards cooperaton. In fact, by sutably choosng the probablty wth whch player stcks to or changes hs acton as a functon of hs last sgnal only, he ensures that player s nd erent across all hs actons n A, ndependently of hs prvate hstory, provded only that nose and dscountng are low enough. In turn, because player s nd erent across hs actons, t s optmal for hm to condton hs play on hs prvate sgnal so as to make hs opponent nd erent. The payo v s then exactly acheved by specfyng approprately the probablty that player plays G n the ntal perod. 11

12 Ths constructon can be generalzed. In dong so, one can extend the prevous argument to all feasble and ndvdually ratonal payo s of the prsoner s dlemma. The set A may be a proper subset of A, and t may depend on calendar tme (although t cannot depend on the spec c hstory). One obtans thereby the set of all belef-free equlbra. Gven a belef-free equlbrum s = (s 1 ; s 2 ), there exsts a sequence of subsets fa t g 1 t=0 of A such that any strategy s 0 of player that adheres to ths sequence from perod t on, that s, for whch 8 rt ; 8 h r s 0 (h r ) 2 A r, s an optmal contnuaton strategy, ndependently of player s hstory h t. In many games, the set of belef-free equlbrum payo s s larger than the convex hull of the statc Nash equlbrum payo s. But the prsoner s dlemma s exceptonal: n most games, ths set does not converge to V as nose and dscountng vansh. If v s a belef-free equlbrum payo close to player s mnmax level v, then player must be able to use an acton close to the mnmax acton most of the tme, mplyng that, n any such perod, A t ncludes the mnmax acton s support. Ths support may nclude actons yeldng player low stage-game payo s -say, below hs own mnmax payo -, ndependently of s acton. Ths s mpossble, as t must be ndvdually ratonal for player hs own hstory. to use ths acton n these perods, ndependently of We clam that the payo structure descrbed above may be recovered, for any game, and any feasble and ndvdually ratonal payo, provded that the stage game s replaced by the normal form of the ntely repeated game. That s, gven v 2 V, there exsts T; S ; s G ; s B T 2 N, S S T, and s G, s B 2 S such that, for all close enough to one, mn U T s ; s G > v > max U T s ; s B S : S T =1;2, wth Strategy s G s the Good strategy that secures player at least v on average over T perods, provded only player uses some strategy wthn S. Strategy s B s the Bad strategy that keeps player s average payo below v, ndependently of s strategy s 2 S T. In each block of 12

13 the supergame, player player uses ether s G or s B. By sutably choosng the probablty wth whch stcks to or changes hs ntely-repeated game strategy from one block to the next, as a functon of hs observatons n the last block only, he ensures that player s nd erent across all the elements n S at the begnnng of each block, ndependently of hs prvate hstory, provded only that nose and dscountng are low enough. In turn, because player s nd erent across these elements, t s optmal for hm to condton hs choce of s G or s B wthn each block on hs observatons n the last block. The payo v s then exactly acheved by specfyng approprately the probablty that player plays s G n the ntal block. Thus, the horzon of the game s dvded nto T -perod blocks. We construct equlbra such that any strategy of player that adheres wthn each future block to an element of S s optmal, ndependently of player s hstory. More precsely, let s 0n to the (n + 1)-st block. Gven s, any strategy s 0 such that j h nt denote the restrcton of s 0 j h nt 8 mn ; 8 h mt s 0m j h mt 2 S, for all hstores h mt ndependently of h nt. followng hstory h nt, yelds an optmal contnuaton strategy s j h nt, In the prsoner s dlemma, t s enough to pck T = 1. In general, T depends both on the stage game and the payo vector v. When T > 1, a block equlbrum need not be belef-free, as a player s set of optmal actons wthn a block may depend on hs prvate hstory. However, ths dependence s lmted to the recent hstory - the nte, termnal segment of the player s prvate hstory of those actons taken and sgnals observed wthn the current block. Because block equlbra need not be belef-free, sequental ratonalty wthn each block rases d cultes under mperfect prvate montorng, a ectng the way S, s G, s B and T are de ned. 3.2 Perfect Montorng In ths subsecton, montorng s perfect. Therefore, s sgnal s s actual acton, and h t equals h t up to the orderng of sgnals and actons. Tme t refers to the number of perods elapsed n 13

14 a block, not n the supergame. Thus, h t denotes a recent hstory, or hstory for short. Fx a stage game and a payo vector v 2 V throughout Payo s and Actons Pck four acton pro les a XY X;Y 2fG;Bg, and correspondng payo vectors w XY X;Y 2fG;Bg : w XY = g a XY, = 1; 2, X; Y 2 fg; Bg : (1) The th superscrpt (G for Good, B for Bad ) refers to th payo, and ndcates whether ths payo s strctly above or below v. See Fgure 1. Formally: w GG > v > w BB, = 1; 2; w1 GB > v 1 > w1 BG and w2 BG > v 2 > w2 GB : Therefore, there exsts v, v, wth v < v < v < v, such that: [v 1 ; v 1 ] [v 2 ; v 2 ] Interor of co w GG ; w GB ; w BG ; w BB : player 2 s payo 6 V w BG r rw GG 2 v 2 2 w BB r rv r w GB v 1 Fgure 1: Payo s v 1 v 1 - player 1 s payo 14

15 Typcally, pure acton pro les satsfyng the desred nequaltes do not exst. However, there always exst an nteger m and four nte sequence a XY 1 ; :::; a XY m such that each X;Y 2fB;Gg vector w XY, the average dscounted payo vector over the sequence a XY 1 ; :::; a XY m, sats es the approprate nequaltes, provded s close enough to one. The constructon that follows must then be mod ed, by replacng each acton pro le a XY a XY 1 ; :::; a XY m. Detals are omtted. by the nte sequence of acton pro les We wll show that each payo n the set [v 1 ; v 1 ][v 2 ; v 2 ] s acheved by some block equlbrum The set of strateges S In the rst perod of a block, actons are used to coordnate contnuaton play. To ths end, partton the set of s actons nto two non-empty subsets, G and B. Player sends message M = G f he pcks an acton n G n the rst perod; otherwse, player sends message M = B. As we shall see, players control each other s payo. That s, s message refers to s average payo, and the acton pro le that corresponds to the par of messages (M 1 ; M 2 ) s thus a M 2M 1. Thus, messages refer to the rst perod s acton choces. The set of strateges S restrcts player s acton only when he observes M = G; even then, the restrcton only apples as long as nether player has devated from the acton pro le correspondng to the observed par of messages (M 1 ; M 2 ). In that case, player must use acton a M 2M 1. Conversely, any strategy n S T that sats es ths restrcton s an element of S. Formally, s 2 S T s n S S T f and only f: ( 2) 8h t = a; a M 2M 1 ; a M 2M 1 ; : : : ; a M 2 M 1 ; a M 2M 1, a 2 M G, t 1: s [h t ] = a M 2M 1 : Observe that the set S mposes no restrcton on the acton spec ed by strategy s 2 S n the ntal perod, after any hstory along whch player observed M = B, and after any hstory along whch some player has devated from a M 2M 1 ; a M 2M 1. In fact, gven any hstory h t, the set S mposes ether no restrcton on s [h t ], or t restrcts 15

16 s [h t ] to a sngle acton. That s, for each hstory h t, de ne A h t := a 2 A : 9 s 2S s h t (a ) > 0. Then A (h t ), the set of actons prescrbed by S, s ether A or a sngleton a M 2M 1. Wth some abuse of notaton, we say that S prescrbes acton a M 2M 1 f A (h t ) = a M 2M 1. It s useful to consder strateges that assgn postve probablty to any acton that s not ruled out by ( 2). To ths end, we de ne, for > 0, the set S := That s, strategy s 2 S s n S n s 2 S : 8 h t 8 a 2A (h t ) o s h t (a ). f after every hstory h t, t assgns probablty at least to every acton n A (h t ). Gven (s ; s ) 2 S S, the set of hstores that are on- and o the equlbrum path s ndependent of the partcular choce (s ; s ). We de ne an erroneous hstory h t as any (recent) hstory that s o the equlbrum path for some (and therefore, every) strategy pro le n S 1 S 2. Otherwse, h t s a regular hstory. Wrte H E;t t-length hstores, and let H R;t = H t nh E;t for the set of all erroneous denote the complement of H E;t. We de ne: H R := [ tt H R;t, H E := [ H E;t. tt The strateges s B, s G 2 S and T We de ne the strateges s B, s G n two steps. Frst, de ne s g (for now, on some hstores) so that: s g [;] 2 4G and 8h t = a; a M 2M 1 ; a M 2M 1 ; : : : ; a M 2 M 1 ; a M 2M 1, a 2 M M, t 1 : s g h t = a M 2 M 1 : That s, s g sends message G and spec es then the acton determned by the par of messages, as long as player observes no devaton from ths acton pro le. Smlarly, de ne s b as follows: s b [;] 2 4B and 8h t = a; a M 2M 1 ; a M 2M 1 ; : : : ; a M 2 M 1 ; a M 2M 1, a 2 M M, t 1 : s b h t = a M 2 M 1 ; 16

17 moreover, let s b h t = for every hstory h t whch s a contnuaton of a hstory That s, s b d ers from s g h r = a; a M 2M 1 ; a M 2M 1 ; : : : ; a M 2 M 1 ; a M 2M 1 ; a M 2 M 1 ; a 0, a 2 B M, a 0 6= a M 2M 1, t r > 1. n two respects. Frst, t sends message B rather than G. Second, t spec es mnmaxng whenever s opponent was the rst to devate from the acton pro le determned by the par of messages - provded chose message B. So far, strateges s b and s g are only de ned for some hstores. On all other hstores, de ne them arbtrarly. Clearly, s b and s g are n S. Observe that, f player uses strategy s b, player gets a stage game payo strctly below v n every perod but at most two: the ntal perod, and the one n whch he devates from a M 2M 1 ; a M 2M 1. Ths s true regardless of the strategy s 2 S T he may be usng. Smlarly, f player uses strategy s g, player gets strctly more than v n every perod but the rst, for any s 2 S used by player. We may thus pck T such that, for some < 1 and all, s average payo (relatve to ) wthn the block from any strategy s 2 S aganst s g strctly exceeds v, whle hs average payo from any strategy s 2 S T aganst s b s strctly below v. From now on, assume. Pck now some small > 0 and perturb slghtly s g ; sb so as to get a par of strateges s G ; s B n S. By choosng small enough, we may assume: mn U T s ; s G > v > v > v > max U T s ; s B S. Under mperfect montorng, the small perturbaton should be thought of as su cently large relatve to the montorng nose. 6 6 Ths perturbaton s unnecessary under perfect montorng. See Remark 2. S T 17

18 3.2.4 The result Besdes statng and provng the result, t s useful to de ne two further strateges, r G 2 S and r B 2 S T and functons, or transfers, G : H T! R, and B : H T! R +. Let r G be a strategy s 2 S such that, for every hstory h t 2 H t, the strategy s jh t yelds the lowest payo aganst s G among all strateges s 2 S. Smlarly, let r B be a strategy s 2 S T such that, for every hstory h t 2 H t, the strategy s jh t yelds the hghest payo aganst s B among all strateges s 2 S T. Wthout loss of generalty, we may take r G and r B to be pure strateges. By enlargng f necessary the rectangle [v 1 ; v 1 ] [v 2 ; v 2 ], we may assume wthout loss of generalty that: U T (r G ; s G ) = v and U T (r B ; s B ) = v. Gven h T, for all t = 1; : : : ; T, let B t denote the d erence between s unnormalzed contnuaton payo (wthn a block) from playng r B from perod t on, and s unnormalzed contnuaton payo from playng s acton - as observed by player n perod t along h T - followed by reverson to r B from perod t + 1 on. By de nton of r B, B t 0. Set B h T := T P T t=1 t 1 B t. Gven h T, for all t = 1; : : : ; T, let t denote the d erence between s unnormalzed contnuaton payo (wthn the block) from playng r G from perod t on and s unnormalzed contnuaton payo from choosng s acton - as observed by player n perod t along h T - followed by reverson to r G from perod t + 1 on. By de nton of r G, t 0 whenever the observed acton s n A (h t ), where h t s player s t-hstory that corresponds to h T. De ne G t := mnf0; t g 0 and set G h T := T P T t=1 t 1 G t. Snce T s xed, we may restrct attenton to close enough to one so that (1 ) B h T < v v and (1 ) G h T > v v : Theorem 1 (n = 2, perfect montorng): Under perfect montorng, for any (v 1 ; v 2 ) 2 V, f players dscount the future su cently lttle, there exsts a block equlbrum of the n ntely repeated game where, for all, player s average payo s v. 18

19 Proof: The strateges we specfy can be descrbed by automata, whch revses states and actons at the begnnng of every block (that s, at the begnnng of perods 1, T + 1; 2T + 1; : : :). An acton of the automaton s the ntely repeated game strategy to be used by the player n the block. The automaton can be descrbed as follows. For each = 1; 2: State space: The state u of player s automaton s an element of [v ; v ], player s contnuaton payo n the repeated game. Intal state: Player starts n state u = v, the payo to be acheved. Actons: In state u, player pcks strategy s G wth probablty q and strategy s B otherwse, where q 2 [0; 1] solves u = qv + (1 q)v. Thus, n each block, player performs an ntal randomzaton, and then stcks to the resultng strategy s G or s B throughout the block. Transtons: If the acton of the automaton s s B, and player s hstory (n the block) s h T, then, at the end of the block, player transts to state: v + (1 ) B h T, (2) whch s n [v ; v ]. If the state of the automaton s s G, and player end of the block, player transts to state: s hstory (n the block) s h T, then, at the v + (1 ) G h T, (3) whch s n [v ; v ]. We clam that these strateges form a subgame-perfect equlbrum. It follows from equatons (7)-(8) and the one-shot devaton property that, gven s strategy, any strategy s such that ts restrcton to any gven block s n S s a best-reply. Player s payo s equal to the weghted average of the payo of playng r G aganst s G and the payo of playng r B aganst s B, wth respectve weghts q and 1 q. Both the average payo wthn the block and the contnuaton payo from playng r G aganst s G are equal to v, and both the average payo wthn the block and the contnuaton payo from playng r B aganst s B are equal to v. Thus, at the begnnng of a block, player s payo when player s state s u s qv + (1 q)v = u. Q.E.D. 19

20 We conclude ths secton wth a few remarks pontng out those features that play an essental role under mperfect montorng. Remark 1: If player uses s B, and computes therefore transfers accordng to B, then player s nd erent across all strateges n S T. Ths mples that, for the sake of computng best-reples, player can always condton on player usng s G (and G ). Remark 2: Snce S mposes no restrcton on s acton n the rst perod and n any ensung perod after message B, and snce both s G and s B are n S, the probablty that player s recent hstory h t 2 H T s regular provded he uses s B s (almost) one, even f player devates to any strategy s 2 S T. Ths would not be true f s G or s B were merely requred to be n S. Ths s why t s possble to specfy explctly s B after (some) regular hstores only, yet guarantee that s payo aganst s B does not exceed v ndependently of s strategy. Ths feature s especally mportant under mperfect montorng, as t s then not possble to specfy play after erroneous hstores. Ths s the motvaton for consderng S and s G ; s B 2 S. Remark 3: Recall that s G and s B are de ned arbtrarly after some hstores, ncludng all erroneous ones. Whle the spec caton on erroneous hstores a ects r G and r B as well as G and B, t s rrelevant for the argument: the transfers B that s suboptmal aganst s B n the block, whle transfers G compensate player for any course of acton penalze player for any course of acton that mproves upon r G aganst s G. We could nstead start by settng G h T 0 arbtrarly on all erroneous hstores h T ; de ne then s G 1 ; s G 2 on erroneous hstores H E 1 = H2 E as any subgame-perfect equlbrum of the game restrcted to erroneous hstores 7, n whch transfers G are added to the payo s of the ntely-repeated game; set s B = s G on any such hstory; gven s B, let B 0 be the transfers that make player nd erent across all strateges n S T aganst s B ; B. In the rst case, play on erroneous hstores s gven and sutable transfers are derved. In the second, transfers are gven and play s derved. Nether approach generalzes to mperfect prvate montorng. The transfers for whch the strateges n S are optmal after regular hstores 7 One can thnk about the restrcted game as a game n whch actons on regular hstores are gven, and players choose ther actons only on erroneous hstores. 20

21 then depend on strateges on erroneous hstores, and conversely. Therefore, we de ne strateges on such hstores and transfers jontly, by applyng Kakutan s xed-pont theorem. 3.3 Imperfect Prvate Montorng In ths subsecton, we prove the two-player folk theorem for almost-perfect montorng, constructng block equlbra smlar to those from the perfect-montorng case. The generalzaton to mperfect montorng must overcome a sgn cant d culty. Note rst that, under the canoncal sgnal structure, the de ntons of S and S are stll vald, as the doman of prvate hstores s the same under perfect and mperfect montorng. So are the de ntons of regular and erroneous (recent) hstores. 8 When the montorng s almost perfect, condtonal on a regular hstory h t, player s almost sure that hs opponent has observed the correspondng hstory, along whch s sgnals concde wth s actons along h t, and s actons concde wth s sgnals along h t. For such hstores h t, t s relatvely straghtforward to gve player ncentves to play the prescrbed acton condtonal on h t. Loosely speakng, ths can (almost) be done by penalzng or rewardng player after a block when s sgnal n perod t, along the hstory correspondng to h t, d ers from, or concdes wth the prescrbed acton. On the other hand, condtonal on an erroneous hstory, player can be sure that at least one player, n at least one perod, has observed an ncorrect, or erroneous sgnal n the block. Ths may or may not be player. For nstance, t could be that s sgnal n the prevous perod was erroneous, leadng hm to pck an acton consstent wth hs own strategy gven that sgnal, but d erent from the one he should have chosen f hs sgnal had been correct. Whch scenaro s more lkely depends on the ne detals of the montorng structure fm ( j a) : a 2 Ag: merely assumng that the nose level s small hardly restrcts s belefs over hstores h t 1 condtonal on erroneous hstores. It s then d cult to prescrbe actons through penaltes or rewards. 8 Observe, however, that gven a strategy pro le (s 1 ; s 2 ) 2 S 1 S 2, erroneous hstores need no longer be o the equlbrum path. 21

22 Yet t s essental that there exsts some best-reply after erroneous hstores h t that s ndependent of s 2 s G ; s B, as otherwse, s best-reply after h t would depend on hs belef about s current strategy, and thus, on hs own hstory n the entre supergame, destroyng thereby the recursve structure of block equlbra. To handle ths d culty, Lemma 1 proves that, f player uses s B (more precsely, plays s B on regular hstores), transfers B (and so transton probabltes n the supergame) can be de ned so that player s nd erent across all actons, condtonal on any hstory, yet holdng the payo of player below the target level v. Ths guarantees that any best-reply after h t condtonal on s G s also a best-reply condtonal on s B. Thus, player may always assume, for the sake of computng best-reples, that hs opponent s playng s G, ndependently of hs own hstory. Recall the de nton of the auxlary scenaro, ntroduced n Secton 2. Gven and h T, player s payo n the auxlary scenaro s de ned as U A h T ; := U T h T +(1 ) T h T. Ths de nton s extended to s 2 S T n the usual fashon, and denoted U A (s; ). Gven (s ; ), the set of s best-responses n the auxlary scenaro s B (s ; ). In the statement of the followng lemmata, U T (s) refers to player s average payo n the ntely repeated game under perfect montorng, gven strategy pro le s 2 S1 T S2 T, whle U A (s; ) denotes player s average payo gven transfer and strategy pro le s 2 S1 T S2 T under mperfect prvate montorng. Lemma 1 For every strategy s j H E, there exsts " > 0 such that for all " < ": There exst a non-negatve transfer B : H T! R + such that S T = B (s B ; B ), (4) where s B j H R = s B j H R and s B j H E = s j H E, and for every s 2 B (s B ; B ); lm U A (s ; s B ; B ) = max U T "!0 es 2S T (es ; s B ). (5) 22

23 Proof: Gven a hstory h T, let obtaned by player for some functon (; ) to be spec ed. h t ; a denote the truncaton of h T to h t and the sgnal n perod t. The transfer wll have the form: " TX # B (h T ) = 1 t 1 (h t 1 T ; a ), t=1 We wll acheve (4) de nng the values, or transfers (; ), by backward nducton. Frst, for every hstory h T 1, we pck transfers (h T 1 ; a ) that make player nd erent between all hs actons condtonal on the event that h T 1 s the hstory of player at the begnnng of perod T. To see that t s possble, consder, for every acton a 2 A, as a row vector the probabltes m ( j ) of sgnals 2 = A observed by player when he plays = s B [h T 1 ] and player plays a (.e. assgns probablty 1 to a ). Construct a matrx D T 1 by stackng the row vectors for every acton a 2 A. When montorng s almost perfect, ths matrx s nvertble (t s actually almost an dentty matrx f, for every a, the row correspondng to the acton a has the same number as the column correspondng to the sgnal a ). Therefore the system of equatons wth the coe cent matrx D T 1, the column vector of unknowns (h T 1 ; a ) and the rght-hand sde vector of g (s B [h T 1 ]; a ) g (s B [h T 1 ]; a ), where a s a stage-game best response to s B [h T 1 ], must have a soluton. Suppose now that all transfers (h ; a ) for > t are already de ned, so that player s nd erent across all hs strateges from perod t + 1 on (condtonal on each hstory h t+1 player ). Then, for every hstory h t 1 2 H t 1, the s auxlary-scenaro contnuaton payo, de ned as the sum of the repeated-game contnuaton payo and the expected value of the transfer 1 T " TX s=t s 1 (h s 1 ; a ) condtonal of the event that h t 1 s s hstory at the begnnng of perod t, depends only on s acton n perod t (.e. t does not depend on hs strategy from perod t + 1 on). For every acton a 2 A, consder as a row vector the probabltes m ( j ) of sgnals 2 = A observed 23 #, of

24 by player when he plays = s B [h t 1 ] and player plays a. Construct a matrx D t 1 by stackng the row vectors for every acton a 2 A. De ne transfers (h t 1 ; a ) as the unque soluton of the system of equatons wth the coe cent matrx D t 1, and the rght-hand sde vector consstng of the d erences between the auxlary-scenaro contnuaton payo of player when he plays an acton a contnuaton payo when he plays acton a. maxmzng ths contnuaton payo and the auxlary-scenaro In ths way, (4) s acheved, as h 0 1 = h 0 2 = ;, but two d cultes reman. Frst, transfers must be non-negatve. To ths end, notce that, as montorng becomes perfect, the matrces D t 1 tend to the dentty matrx, and the rght-hand sde vectors n the systems of equatons determnng transfers (h t ; a ) are non-negatve. Therefore, the lmts of transfers (h t ; a ), as nose vanshes, must be non-negatve. Thus, we can make all transfers (h t ; a ) non-negatve by addng to all of them a postve constant, and we can assume that ths constant tends to 0 as nose vanshes. Second, (5) must be acheved. To ths end, consder the strategy r B of player such that (8h t 1 2 H t 1 ) r B [h t 1 ] s hs stage-game best reply to s B [h t 1 1 ], where ht denotes the hstory of player correspondng to the hstory h t 1. [Note that ths strategy r B typcally d ers from the strategy denoted by the same symbol n Secton 3.2. The two strateges play, however, the same role, and for ths reason, we denote them by the same symbol.] As montorng becomes perfect: (a) by constructon, the expected value of the transfer B (h T ) tends to 0, f players play r B and s B respectvely, and (b) by the de nton of r B, the repeated game payo of player tends to the rght-hand sde of (5); obvously, (a) and (b) yeld (5). player Q.E.D. The second lemma asserts that, for any xed par of strateges on erroneous hstores, f plays s G on hs regular hstores, then he can pck transfers such that any strategy of player that stcks to some strategy n S on hs regular hstores s optmal. Gven H E H T, (s ; s ) 2 S T S T and, B (s ; j s ) denotes the set of strateges maxmzng s auxlaryscenaro payo aganst (s ; ) among all strateges s 2 S T such that s j H E = s j H E. 24

25 Lemma 2 For every strategy s j H E, there exsts " > 0 such that for " < ": There exsts a non-postve transfer G : H T! R such that fs 2 S T : s j H R = es j H R for some es 2 S and s j H E = s j H E g B (s G ; G j s ); (3) where s G j H R = s G j H R and s G j H E = s j H E. Proof: The transfer wll agan have the form: " TX # G (h T ) = 1 t 1 (h t 1 T ; a ), Pck r G 2 S T to be a strategy that sats es: (a) r G j H R = s j H R for some s 2 S ; (b) r G j H E = s j H E ; (c) 8h t 1 2 H R;t 1, r G jh t 1 game, among all strateges wth propertes (a) and (b). t=1 yelds the lowest payo aganst s G, n the T -perod repeated Wthout loss of generalty, assume that r G j H R s a pure strategy. [Note agan that ths strategy r G formally d ers from the strategy denoted by the same symbol n Secton 3.2, but the two strateges play the same role, and for ths reason, we denote them by the same symbol.] We now de ne (h t 1 ; a ), 8h t 1 2 H t 1, 8a 2 = A by backward nducton wth respect to t. To satsfy (3), t su ces to pck non-postve values for (h t 1 ; a ) such that (gven (h 1 ; a ), > t), the followng constrants, or propertes, are sats ed: 1. For every hstory h t 1 2 H R;t 1 such that S prescrbes A, player s nd erent under the auxlary scenaro between playng all actons a 2 A, each followed by swtchng to r G from perod t + 1 on; 2. For every hstory h t 1 2 H R;t 1 such that S prescrbes a = a GG or a BG f = 2, or a GB = 1, the payo of player under the auxlary scenaro to playng a (weakly) exceeds the payo to playng any other acton, both followed by swtchng to r G f from perod t + 1 on. 25

26 [Note that the auxlary-scenaro payo d erence across contnuaton strateges s j h t 1 player s ndependent of (h e t 1 ; a ) for et < t as those values cannot be a ected by actons taken n perods et t. The d erences, even the preference orderng over contnuaton strateges, may of course depend on the values of (h e t 1 ; a ) for et > t, but these values are already determned by backward nducton.] For every > 0, observe that there exsts "= small enough such that: for any hstory h t 1 2 H R;t 1, player assgns, condtonal on observng h t 1, probablty at least 1 to the event that player 1 observed the correspondng hstory h t 1 2 H R;t 1, along whch the sgnals of player concde wth the actons of player n h t 1 and the actons of player along h t 1 concde wth the sgnals of player n h t 1. Note that, gven two dstnct hstores h t 1 ; h 0t 1 2 H R;t 1, the correspondng hstores h t 1 and h 0t 1 h t 1 2 H R;t 1 of are dstnct. Gven any hstory and any acton a 2 A, consder as a row vector the probabltes assgned by player, condtonal on hstory h t 1 and on acton a taken by player n perod t, to the d erent hstores h t 1 2 H t 1 and sgnals a 2 = A observed by player n perod t. Construct a matrx D t 1 by stackng the row vectors for all regular hstores h t 1 2 H R;t 1 actons a 2 A. By the prevous paragraph, the matrx D t and 1 has full row rank, provded "= s small enough. Therefore, there exst values (h t 1 ; a ) satsfyng constrants 1 and 2. Indeed the number of columns (rows) of D t 1 exceeds the number of lnear equalty (or nequalty) constrants that 1-2 mposes on (h t 1 ; a ) by k, where k := #H R;t 1 s the number of (t 1)-length regular hstores h t 1. [Say, A = fa 1 ; :::; a l g conssts of l actons and suppose that, gven a hstory h t 1 2 H R;t 1, constrant 1 must be sats ed (the argument for constrant 2 s analogous). That s, l 1 equatons have to be sats ed: player must be nd erent between playng a k and a k+1 for k = 1; :::; l 1. Snce there are l actons a, there are l rows of D t 1 correspondng to each h t 1 2 H R;t 1, but only l 1 constrants.] Further, we can assume that these values (h t 1 ; a ) are all non-postve snce propertes 1-2 de ne the values (h t 1 ; a ) up to a constant. Q.E.D. 26

27 The thrd lemma shows that, provded the nose level s su cently small, the appled transfer G (from Lemma 2) s arbtrarly close to zero, gven s G, f player plays r G. Lemma 3 In Lemma 2, the non-postve transfer G : H T! R can be chosen so that, for every s 2 B (s G ; G j s ), () lm "!0 U A (s ; s G ; G ) = mn es 2S U T (es ; s G ); (6) () G depends contnuously on s, and () G s bounded away from 1,.e. there exsts (ndependent of s) such that G. Proof: To guarantee (6), the transfers (; ) from the proof of Lemma 1 must be further spec ed. In the backward nducton wth respect to t, we can assume, n addton to propertes 1-2, that (h t 1 ; a ) = 0 whenever h t 1 2 H t 1 s the hstory correspondng to some hstory h t 1 2 H R;t 1 and a = r G (h t 1 ) or h t 1 s a hstory not correspondng to any h t 1 2 H R;t 1 and a 2 = A. Indeed, remember that the number of constrants mposed on (h t 1 ; a ) by propertes 1-2 falls below the rank of D t 1 by k). Snce (h t 1 ; a ) = 0 for every h t 1 correspondng to h t 1 2 H R;t 1 and a = r G (h t 1 ), the expected value of the transfer G (h T ) f he uses the strategy r G tends to 0 as "! 0. Ths yelds (), except that the values (h t 1 ; a ) may be postve. We shall now show that the transfers can be pcked n such a way that, for any a 2 A and any hstory h t 1 correspondng to some h t 1 2 H R;t 1, (h t 1 ; a ) tends to a non-postve value as "! 0. Ths wll guarantee that all values (h t 1 ; a ) can be made non-postve, by subtractng a constant from all of them, and ths wll not a ect the requred propertes snce the constant can be tendng 0 for "! 0. Ths follows from property 1 for every hstory h t 1 2 H R;t 1 such that S prescrbes A on the hstory h t 1 correspondng to h t 1 2 H R;t 1. Indeed, the value (h t 1 ; a ) s n the lmt equal to the d erence between the expected value of the contnuaton transfer (from perod t on),.e. 27