The Folk Theorem in Repeated Games with Individual Learning

Transcription

1 The Folk Theorem n Repeated Games wth Indvdual Learnng Takuo Sugaya and Yuch Yamamoto Frst Draft: December 10, 2012 Ths Verson: July 24, 2013 Ths s a prelmnary draft. Please do not crculate wthout permsson. Abstract We study repeated games where players observe nosy prvate sgnals about the unknown state of the world n every perod. We fnd a suffcent condton under whch the folk theorem obtans by ex-post equlbra. Our condton s satsfed for generc sgnal dstrbutons as long as each player has at least two possble prvate sgnals. Journal of Economc Lterature Classfcaton Numbers: C72, C73. Keywords: repeated game, prvate montorng, ncomplete nformaton, ex-post equlbrum, ndvdual learnng. Stanford Graduate School of Busness. Emal: tsugaya@gsb.stanford.edu Department of Economcs, Unversty of Pennsylvana. Emal: yyam@sas.upenn.edu 1

2 1 Introducton In many economc actvtes, agents face uncertanty about the underlyng payoff structure, and expermentaton s an mportant nformaton source to resolve such an uncertanty. Suppose that two frms enter a new market. The frms are not famlar wth the structure of the market and hence do not know how proftable the market s. These frms nteract repeatedly n the market; n every perod, each frm chooses ts prce and prvately observes ts sales level, whch s stochastc due to unobservable demand shocks. In ths stuaton, the frms can eventually learn the true proftablty of the market through expermentaton. For example, they can conjecture that the market s good f they observe hgh sales levels frequently. However, snce sales levels are prvate nformaton, a frm may not have precse nformaton about what the rval frm has learned n the past play, and hence t s unsure about what the rval frm wll do next. Also, n ths setup, a frm can ntentonally manpulate the belef of the rval frm; for example, f frm A commts to choose some aggressve behavor n every perod, then frm B may thnk that frm A s very optmstc about the market proftablty, and as a result frm B may update ts posteror toward a postve drecton. How do these features nfluence the frms ncentves? Can the frms sustan a colluson n such a stuaton? More generally, when players learn the unknown state from prvate sgnals, does the long-run relatonshp help to provde ncentves to cooperate? To address these questons, we develop a general model of repeated games wth ndvdual learnng. In our model, Nature moves frst and chooses the state of the world, whch s fxed throughout the game. The state s not observable to players. Then players play an nfntely repeated game, and n each perod, players observe prvate sgnals, the dstrbuton of whch depends on the state of the world. Actons are perfectly observable, and a player s stage-game payoff depends both on actons and on her prvate sgnal. In ths setup, players update ther belefs about the state n every perod through realzed actons and sgnals. Snce sgnals are prvate nformaton, players posteror belefs about the true state need not concde n later perods. In partcular, whle each player may prvately learn the state from observed sgnals, ths learnng process may not lead to common learnng n the sense of Crpps, Ely, Malath, and Samuelson (2008), that s, a player may not learn that the opponent learns the state, or a player may not learn 2

3 that each player learns that each player learns the state, or... Ths paper nvestgates how the evoluton of these hgher-order belefs mpacts players ncentves to coordnate ther play. The man fndng of the paper s that the folk theorem obtans under mld dentfablty condtons. That s, we show that f players are patent, there are equlbra n whch players eventually obtan payoffs as f they knew the true state and played an equlbrum for that state. Ths fndng s new to the lterature, as most of the past work assumes that players learn the true state from publc sgnals. The equlbrum analyss n our model s substantally dfferent from that of publc learnng manly for two reasons. Frst, n games wth ndvdual learnng, players do not know what the opponents have learned n the past play, and hence they face uncertanty about what the opponents wll do. Ths feature can potentally have a negatve mpact on players wllngness to coordnate. Second, players may strategcally pretend as f they observed sgnals whch dffer from actual observatons, n order to manpulate the opponents posteror belefs. Nonetheless, our result shows that there stll exsts an equlbrum where players do coordnate ther play to approxmate a Pareto-effcent outcome state by state. Another appealng feature of our result s that we use ex-post equlbra as a soluton concept, and hence our equlbrum strateges are robust to perturbatons of players ntal pror. That s, our equlbrum strategy s stll an equlbrum even f players ntal pror changes. The followng two condtons are crucal to establsh our folk theorem. Frst, we requre the statewse full-rank condton, whch says that there s an acton profle such that sgnal dstrbutons are dfferent at dfferent states even f someone unlaterally devates. Ths condton ensures that each player can ndvdually learn the true state from prvate sgnals n the long run, and nobody can nterrupt the opponents state learnng. Second, we need the correlated learnng condton. Roughly, ths condton says that players prvate sgnals are (possbly slghtly) correlated so that f a player observed too many unlkely sgnals and made a wrong nference about the true state, then she beleves that the opponents sgnals are also dstorted and ts emprcal dstrbuton s dfferent from the theoretcal dstrbuton. These two condtons are satsfed for generc sgnal dstrbutons as long as each player has at least two possble prvate sgnals. Our proof of the folk theorem s constructve. In our equlbrum strategy, the 3

4 nfnte horzon s dvded nto a sequence of block games wth length T b > 1. In each block game, players start wth learnng the true state from prvate sgnals; the statewse full-rank condton ensures that ths s ndeed possble. Players spend suffcently many perods on ths learnng process so that they can make a correct nference about the state wth a very hgh probablty. Then players report ther prvate nferences through ther actons, and dependng on the reported nformaton, they effectvely adjust actons n the remanng perods of the block game. Ths learnng and adjustment behavor allows to approxmate desred payoffs state by state. An advantage of these block-game strateges s that even f someone fals to learn the true state by accdent or by a manpulaton by the opponents, ts mpact on the long-run payoff s small, because players redo learnng n the next block game. The dea here may look smlar to that of Wseman (2012), who also uses block-game strateges. However, he assumes that players observe both publc and prvate sgnals about the state, and focuses on equlbra n whch players learn from publc sgnals only and gnore prvate sgnals; ths means that players contnuaton play s always common knowledge among players on the equlbrum path. In our equlbra, players learn from prvate sgnals, and hence a player s belef about what the opponents learned s relevant to her ncentves. Accordngly, the ncentve compatblty constrant n our analyss s sgnfcantly dfferent from that of Wseman (2012). Also, he assumes that mxed actons are observable to establsh a folk theorem wth mxed-strategy mnmax payoffs. We do not need such an assumpton. The key step n our equlbrum constructon s to provde players wth ncentves to report the truth about the result of ther state learnng. In partcular, we need to show that a player s wllng to report her nference truthfully even when there were too many unlucky draws n the learnng round and she made a wrong nference. Here the correlated learnng condton plays an mportant role, and the dea s roughly as follows. When a player was unlucky and made a wrong nference, the correlated learnng condton says that she beleves that the opponents sgnals durng the learnng round are also (possbly slghtly) dstorted. Ths means that she beleves that the opponents do not expect her to learn the state successfully and hence they wll not punsh her even f she reports a wrong nference. Ths provdes the desred truth tellng ncentve. 4

5 There s a rapdly growng lterature on learnng n repeated games. Most of the exstng work studes the case n whch players observe publc sgnals about the state, and focuses on equlbrum strateges where players condton ther play only on publc nformaton on the equlbrum path (Wseman (2005), Wseman (2012), Fudenberg and Yamamoto (2010), Fudenberg and Yamamoto (2011a)). In ths context, players hgher-order belefs are rrelevant to the equlbrum analyss, as what players wll do next s common knowledge. An excepton s Yamamoto (2014), who assumes that players learn the state from prvate sgnals only, as n our paper. However, he looks at a specal class of sequental equlbra, called belef-free equlbra, n whch a player s best reply s always ndependent of the opponent s past hstory. Ths means that a player does not care what the opponent wll do next, and hence hgher-order belefs are payoff-rrelevant nformaton. In contrast, ths paper consders equlbra where a player s best reply depends on the opponents hstory, so we need to keep track of the evoluton of hgher-order belefs to verfy the ncentve compatblty constrants. In ths sense, our analyss s sgnfcantly dfferent from the exstng work, at both conceptual and techncal levels. Ths dfference s ndeed mportant, because t s well-known among experts that focusng on belef-free equlbra s loss of generalty and ts equlbrum payoff set s often bounded away from the effcent fronter. We show that a larger payoff set s supported by consderng general sequental equlbra. 1 As mentoned, we use ex-post equlbra as our soluton concept. Ths greatly smplfes our analyss, snce we do not need to compute a player s frst-order belef about the state to check the ncentve compatblty constrants. Some recent papers use ths ex-post equlbrum approach n dfferent settngs of repeated games, such as perfect montorng and fxed states (Hörner and Lovo (2009) and Hörner, Lovo, and Tomala (2011)), publc montorng and fxed states (Fudenberg and Yamamoto (2010) and Fudenberg and Yamamoto (2011a)), prvate montorng and fxed states (Yamamoto (2014)), and changng states wth an..d. dstrbuton (Mller (2012)). Note also that there are many papers workng on ex-post equlbra n undscounted repeated games; see Koren (1992) and Shalev (1994), for example. 1 Whle our equlbrum strategy s not belef-free, t s perodcally belef-free n the sense that a player s best reply s ndependent of the opponent s past hstory at the begnnng of each block game. Hörner and Olszewsk (2006) s the frst to use ths dea to prove the folk theorem for games wth general payoff functons. 5

6 Our model s deeply related to repeated games wth prvate montorng, because a player s belef about the opponent s hstory nfluences her ncentves n both models. Past work has shown that a long-term relatonshp helps provde ncentves to cooperate under prvate montorng. For example, effcency can be approxmately acheved n the prsoner s dlemma, when observatons are nearly perfect (Sekguch (1997), Bhaskar and Obara (2002), Pccone (2002), Ely and Välmäk (2002), Yamamoto (2007), Yamamoto (2009), Hörner and Olszewsk (2006), Chen (2010), and Malath and Olszewsk (2011)), nearly publc (Malath and Morrs (2002), Malath and Morrs (2006), and Hörner and Olszewsk (2009)), statstcally ndependent (Matsushma (2004), Yamamoto (2012)), or even fully nosy and correlated (Fong, Gossner, Hörner and Sannkov (2011) and Sugaya (2012)). Kandor (2002) and Malath and Samuelson (2006) are excellent surveys. See also Lehrer (1990) for the case of no dscountng, and Fudenberg and Levne (1991) for the study of approxmate equlbra wth dscountng. 2 Repeated Games wth Indvdual Learnng Gven a fnte set X, let X be the set of probablty dstrbutons over X. Gven a subset W of n, let cow denote the convex hull of W. We consder N-player nfntely repeated games, where the set of players s denoted by I = {1,,N}. At the begnnng of the game, Nature chooses the state of the world ω from a fnte set Ω = {ω 1,,ω o }. Assume that players cannot observe the true state ω, and let µ Ω denote ther common pror over ω. 2 Throughout the paper, we assume that the game begns wth symmetrc nformaton: Each player s belefs about ω correspond to the pror. But t s straghtforward to extend our analyss to the case wth asymmetrc nformaton as n Fudenberg and Yamamoto (2011a). 3 2 Because our arguments deal only wth ex-post ncentves, they extend to games wthout a common pror. However, as Dekel, Fudenberg, and Levne (2004) argue, the combnaton of equlbrum analyss and a non-common pror s hard to justfy. 3 Specfcally, all the results n ths paper extend to the case n whch each player has ntal prvate nformaton θ about the true state ω, where the set Θ of player s possble prvate nformaton s a partton of Ω. Gven the true state ω Ω, player observes θ ω Θ, where θ ω denotes θ Θ such that ω θ. In ths setup, prvate nformaton θ ω allows player to narrow down the set of possble states; for example, player knows the state f Θ = {(ω 1 ),,(ω o )}. 6

7 Each perod, players move smultaneously, and player I chooses an acton a from a fnte set A. The chosen acton profle a A I A s perfectly observable to players, and n addton, each player observes a prvate sgnal z from a fnte set Z. The dstrbuton of a sgnal profle z Z I Z depends on the state of the world ω and on an acton profle a A, and s denoted by π ω ( a) Z. Let π ω ( a) denote the margnal dstrbuton of z Z at state ω condtonal on a A, that s, π ω (z a) = z Z π ω (z a). Also let π ω ( a) be the margnal dstrbuton of z. Player s realzed payoff s u ω(a,z ), so that her expected payoff at state ω gven an acton profle a s g ω (a) = z Z π ω (z a)u ω (a,z ). We wrte π ω (α) and g ω (α) for the sgnal dstrbuton and expected payoff when players play a mxed acton profle α I A. Smlarly, we wrte π ω (a,α ) and g ω (a,α ) for the sgnal dstrbuton and expected payoff when players play a mxed acton α A. Let g ω (a) denote the vector of expected payoffs at state ω gven an acton profle a. We do not need the full support assumpton. 4 As emphaszed n the ntroducton, uncertanty about the payoff functons s common n applcatons. Examples that ft our model nclude the followngs: Olgopoly market wth unknown demand functon. I s the set of frms n an olgopoly market, a s frm s prce, and z s frm s sales level. The frms do not have precse nformaton about the demand functon, and hence they do not know the dstrbuton π of sales levels; ths scenaro s captured by assumng that π depends on ω. Team producton and prvate beneft. I s the set of agents workng n a jont project, a s agent s effort level, and z s agent s prvate proft from the project. Agents do not know the proftablty of the project, so π depends on ω. In the nfntely repeated game, players have a common dscount factor δ (0,1). Let (a τ,z τ ) A Z be player s prvate observaton n perod τ, and we denote player s prvate hstory from perod one to perod t 1 by h t = (aτ,z τ )t τ=1. Let h 0 = /0, and for each t 0, and let H t be the set of all prvate hstores h t. 4 If there are ω Ω and ω ω such that u ω (a,z ) u ω (a,z ) for some a A and z Z, then t mght be natural to assume that player does not observe the realzed value of u as the game s played; otherwse players mght learn the true state from observng ther realzed payoffs. Snce we consder ex-post equlbra, we do not need to mpose such a restrcton. 7

8 Also, we denote a profle of t-perod hstores by h t = (h t ) I, and let H t be the set of all hstory profles h t. A strategy for player s defned to be a mappng s : t=0 H t A. Let S be the set of all strateges for player, and let S = I S. We defne the feasble payoff set for a gven state ω to be V (ω) co{g ω (a) a A}, that s, V (ω) s the set of the convex hull of possble stage-game payoff vectors gven ω. Then we defne the feasble payoff set for the overall game to be V ω Ω V (ω). Thus a vector v V specfes payoffs for each player and for each state,.e., v = ((v1 ω,,vω N )) ω Ω. Note that a gven v V may be generated usng dfferent acton dstrbutons n each state ω. If players observe ω at the start of the game and are very patent, then any payoff n V can be obtaned by a state-contngent strategy of the nfntely repeated game. Lookng ahead, there wll be equlbra that approxmate payoffs n V f the state s dentfed by the sgnals, so that players learn t over tme. We defne player s mnmax payoff for a gven state ω to be m ω mn α max a g ω (a,α ). Let α ω () denote the (possbly mxed) mnmax acton profle aganst player condtonal on ω. We denote the set of feasble and ndvdually ratonal payoffs by V, that s, V {v V v ω m ω ω}. Here the ndvdual ratonalty s mposed state by state;.e., V s the set of feasble payoffs such that each player obtans at least her mnmax payoff for each state ω. Throughout the paper, we assume that the set V s full dmensonal: Condton 1. (Full Dmenson) dmv = I Ω. 3 The Folk Theorem wth Indvdual Learnng Our man result of the paper s the folk theorem; that s, we show that f players are patent, then there are equlbra n whch players eventually obtan payoffs as 8

9 f they knew the true state and played an equlbrum for that state. Ths means that even when players learn the state from prvate sgnals, there stll exsts an equlbrum where players are wllng to coordnate ther play to approxmate a Pareto-effcent outcome state by state. We gve a set of condtons under whch the folk theorem obtans. Our frst condton s the statewse full-rank condton of Yamamoto (2014), whch requres that there be an acton profle such that each player can learn the true state ω from her prvate sgnal z : Condton 2. (Statewse Full Rank) There s an acton profle a G A such that π ω ( a j,a G j ) π ω ( a j,a G j ) for each, j, a j A j, and for each (ω, ω) wth ω ω, Intutvely, the statewse full rank mples that player can statstcally dstngush ω from ω through her prvate sgnal z, even f someone else unlaterally devates from a G. 5 We fx a G throughout the paper. Note that Condton 2 s satsfed for generc sgnal structures π f Z 2 for each. Our next condton s about the correlaton of players prvate sgnals. To gve a formal statement, the followng notaton s useful. Let π ω (z a,z ) denote the condtonal probablty of z gven that the true state s ω, players play an acton profle a, and player observes z ;.e., π ω (z a,z ) = πω (z a) π ω (z a). Let π ω (z a,z ) = 0 f π ω (z a) = 0. Then let C ω (a) be the matrx such that the rows are ndexed by the elements of Z, the columns are ndexed by the elements of Z, and the (z,z )-component s π ω (z a,z ). Intutvely, C ω (a) maps player s observatons to her estmate (expectaton) of the opponents observatons condtonal on the true state ω and the acton profle a. To get the precse meanng, suppose that players played an acton a for T perods, and player observed a sgnal sequence (z 1,,zT ). Let f Z be a column vector whch 5 Ths condton s stronger than necessary. For example, our equlbrum constructon extends wth no dffculty f for each (,ω, ω) wth ω ω, there s an acton profle a A such that π ω ( a j,a j) π ω ( a j,a j) for each j and a j A j. That s, each player may use dfferent acton profles to dstngush dfferent pars of states. But t sgnfcantly complcates our notaton wth no addtonal nsghts. Also, whle Condton 2 requres that all players can learn the state from prvate sgnals, t s easy to see that our equlbrum constructon s vald as long as there are at least two players who can dstngush the state. 9

10 represents the correspondng sgnal frequency,.e., let f = ( f [z ]) z Z where f [z ] = {t {1,,T } zt =z } T for each z. Then the emprcal posteror dstrbuton of the opponents sgnals z durng these T perods (.e., the estmaton of the opponents sgnal frequences condtonal on the true state ω and player s observaton f ) s represented by C ω (a) f. So the matrx C ω (a) converts player s sgnal frequency f to her estmate of the opponents sgnal frequences. We mpose the followng condton: Condton 3. (Correlated Learnng) For each and for each (ω, ω) wth ω ω, we have C ω (a G )π ω (a G ) π ω (ag ). Very roughly speakng, ths condton says that prvate sgnals are correlated across players so that, f player s (cumulatve) observaton s too dfferent from the theoretcal dstrbuton, then she should expect that the opponents sgnals are also dstorted. To be precse, suppose that there are two possble states, ω and ω, and that players are tryng to dentfy the true state usng prvate sgnals durng T perods. Players play a G durng these perods, so Condton 2 mples that ndvdual learnng s possble. Suppose that the emprcal dstrbuton of player s sgnals n these perods s exactly equal to the theoretcal dstrbuton at state ω,.e., f = π ω (a G ). Ths ndcates that the true state s lkely to be ω, but snce sgnals are nosy nformaton, player cannot rule out the possblty that the true state s ω rather than ω, and that havng f = π ω (a G ) s just due to bad luck. Condton 3 mples that, f ths s such a bad luck case, then player should expect that the opponents sgnals are also dstorted. More precsely, the condton says that, f the true state s ω but nonetheless player s sgnal frequency s equal to the theoretcal dstrbuton at the other state ω, then her estmate about the opponents sgnal frequency, whch s represented by C ω (a G )π ω (a G ), s dfferent from the theoretcal dstrbuton π ω (ag ) at the true state ω. Note that Condton 3 s satsfed for generc sgnal structures π, snce t can be satsfed by (almost all) small perturbatons of the matrx C ω (a G ). However, t rules out the case n whch prvate sgnals are condtonally ndependent. Indeed, f sgnals are ndependently drawn condtonal on a G, then π ω (z a G,z ) = π ω (z a G ) and hence C ω (a G ) f = π ω (ag ) for any f, whch means that Condton 3 fals. In Secton 6, we wll nvestgate what happens when sgnals are condtonally ndependent. 10

11 Condton 3 plays an mportant role n our equlbrum constructon, because a player s belef about the opponents belefs about ω (.e., a player s second order belef about ω) nfluences her wllngness to coordnate. For example, after some hstory, a player may beleve that the true state s ω but that the opponents does not beleve that the true state s ω; we need to carefully treat her ncentves after such a hstory. As wll be seen, Condton 3 s helpful when we handle ths sort of problem. 6 The followng s the man result of ths paper: Proposton 1. Under Condtons 1 through 3, the folk theorem obtans,.e., for any v ntv, there s δ (0,1) such that for any δ (δ,1), there s an ex-post equlbrum wth payoff v. Ths proposton says that there are ex-post equlbra n whch players eventually obtan payoffs as f they knew the true state and played an equlbrum for that state, even when players learn the state from nosy prvate sgnals. The proof s gven n the next secton. 4 Proof of Proposton Notaton and Overvew Fx an arbtrary payoff vector v ntv. We wll construct an ex-post equlbrum wth payoff v for suffcently large δ, by extendng the dea of block strateges of Hörner and Olszewsk (2006). Take v ω and v ω for each and ω so that v ω < v ω < v ω for each and ω and that the product set I ω Ω [v ω,vω ] s n the nteror of the set V. In our equlbrum, the nfnte horzon s dvded nto a sequence of block games wth T b perods, and our equlbrum strategy can be descrbed by an automaton. At the begnnng of each block game, each player prvately chooses an automaton state, whch s denoted by x = (x ω ) ω Ω {G,B} Ω. Note that an automaton state x s a vector wth Ω components, and each component x ω s ether a good state x ω = G or a bad state x ω = B. Ths automaton state x completely 6 Condton 3 s not mposed n Yamamoto (2014), because he restrcts attenton to belef-free equlbra n whch a player s belef about the opponent s hstory (and thereby her second-order belef) does not nfluence her best reply. 11

12 determnes player s strategy n the current block game. (So the automaton state x changes perodcally. That s, x s fxed untl the end of the current block game, then at the begnnng of the next block game, player wll choose a new automaton state x.) Roughly, beng n the good state x ω = G means that player plans to reward player +1 durng the current block game f the true state s ω. (Here player + 1 refers to player 1 f = N.) Lkewse, beng n the bad state x ω = B means that player plans to punsh player + 1 durng the block game f the true state s ω. So player s choce of her automaton state x ω reflects her ntenton about whether to punsh or reward player + 1 when the state s ω. In what follows, we wll construct the automaton strategy carefully so that the followng propertes are satsfed; Player s contnuaton payoff from the current block game s v ω f the state s ω and the current ntenton profle s x = (x j ω ) ( j, ω) such that x 1 ω = G. Player s contnuaton payoff from the current block game s v ω f the state s ω and the current ntenton profle s x = (x j ω ) ( j, ω) such that x 1 ω = B. That s, player s payoff gven ω s hgh f player ( 1) s current ntenton for the state ω s good, and s low f the ntenton s bad. Note that player s payoff gven ω s solely determned by x 1 ω and does not depends on the other components of the ntenton profle x. In partcular, player s ntenton does not nfluence her own payoffs. As we wll explan, n our equlbrum, player 1 chooses the bad ntenton x 1 ω = B wth a larger probablty when player devated n the last block game; ths effectvely deters player s devaton. Now we descrbe the structure of the block games. Each block game s further dvded nto a Learnng Round, an Announcement Round, a Man Round, and a Report Round. The role of each round s as follows: Learnng Round: The frst N Ω ( Ω 1) 2 T perods of each block game are regarded as a learnng round, n whch players take turns and try to learn the true state from observed prvate sgnals. That s, the learnng round s further dvded nto N rounds wth the equal length, and player makes an nference about the true state based on observatons n the th round. We call ths th round player s learnng round, and let T () be the set of perods ncluded n ths round. At the end of player s learnng round, she summarzes the observed sgnals and makes 12

13 an nference ω() Ω {/0}; here, ω() = ω means that player beleves that the true state s ω, whle ω() = /0 means that the learnng fals and player s not sure about the true state. The way she makes ths nference ω() wll be descrbed later, but we would lke to stress that each player s state learnng s almost perfect n our equlbrum, snce we take T suffcently large. That s, when the true state s ω, the probablty of player havng ω() = ω s close to one. Announcement Round: The next K perods are regarded as an announcement round, where K s the smallest nteger satsfyng K log A ( Ω + 1) for each. In ths round, each player announces what she learned n the learnng round,.e., she reports ω() through her choce of actons. By dong so, players can check whether they agree on the true state or not. Note that each player can ndeed represent ω() Ω {/0} by a sequence of actons wth length K, snce K log A ( Ω + 1). Man Round: The next T 2 perods are regarded as a man round. Snce the man round s much longer than any other round n the block game, the average payoff of the block game s approxmated by the average payoff of the man round when δ s close to one. Players behavor n the man round s dependent on the nformaton released n the announcement round. Very roughly speakng, f they could agree on the true state n the announcement round, then they reveal ther prvate automaton states (the ntentons about whether to punsh or reward the opponent) n the ntal perod of the man round. Then n the remanng perods, they choose actons consstent wth the revealed ntentons; that s, they choose actons whch gve hgh payoffs to those who should be rewarded and low payoffs to those who should be punshed. Detals wll be gven later. Report Round: The remanng KNT Ω ( Ω 1) 2 perods of the block game are regarded as a report round round, where K log A Z for each. In the report round, each player reports the sequence of her prvate sgnals n the learnng round, through her choce of actons. The man queston n the proof s how to provde ncentves to follow the equlbrum strategy. Our goal s to construct an ex-post equlbrum, so n the 13

14 followng dscusson, we fx ω and consder ncentves gven ths state ω. To smplfy our dscusson, assume that there are only two players for now. As explaned, n our equlbrum, player s payoff gven ω s solely determned by the opponent s ntenton for that state, x ω ; the good ntenton xω = G gves hgh payoffs, whle the bad ntenton x ω = B yelds low payoffs. At the begnnng of every block game, player mxes these two ntentons x ω = G and x ω = B, dependng on her prvate hstory ht b n the prevous block game. Let ρ ω (ht b G) denote the probablty that player changes the ntenton n the next block game when her current ntenton s x ω = G. (See Fgure 1.) In the proof, we wll choose ths mxture probablty carefully so that player has no ncentve to devate durng the current block game gven that the true state s ω and the opponent s current ntenton s x ω = G. The dea s to punsh player by ncreasng the probablty of the bad ntenton, ρ ω (ht b G), f player devates n the current block game. Lkewse, let ρ ω (ht b B) denote the probablty that player changes the ntenton n the next block game when her current ntenton s x ω = B. We wll choose ths mxture probablty carefully so that player has no ncentve to devate gven that the true state s ω and the opponent s current ntenton s x ω = B. If such mxture probabltes exst, then player has no ncentve to devate after every hstory gven the state ω; ths means that ex-post ncentves are satsfed, as desred. 1 ρ ω (ht b G) xω = G Reward ρ ω (ht b G) ρ ω (ht b B) x ω = B Punsh 1 ρω (ht b B) Fgure 1: Automaton The way we choose the mxture probablty ρ ω (ht b B) s qute smlar to that of Hörner and Olszewsk (2006). On the other hand, we need a novel dea when we choose the mxture probablty ρ ω (ht b G); ndeed, ths s exactly the place n whch Condton 3 plays a role. So, here we wll focus on how to fnd the 14

15 mxture probablty ρ ω (ht b G) whch provdes approprate ncentves to player condtonal on ω and x ω = G. Frst of all, consder ncentves n the learnng round. Recall that players are asked to play a G durng ths round. Snce actons are observable, f player devates from a G then t wll be mmedately detected by the opponent. In such a case, we ask the opponent to ncrease the probablty of the bad ntenton x ω = B n the next block game, whch lowers player s contnuaton payoff. Ths effectvely deters player s devaton n the learnng round. A smlar logc apples to ncentve problems n the man round. Note that n second or later perods of the man round, any devaton s mmedately detected, snce actons n these perods are dependent only on past publc nformaton (specfcally, t depends only on actons n the announcement round and n the frst perod of the man round). Therefore, n order to deter player s devaton n these perods, we only need to ncrease the probablty of the opponent havng the bad ntenton n the next block game. Also, n the frst perod of the man round, players reports the ntenton x ω truthfully, as her payoff does not depend on her own ntenton and thus she s ndfferent between reportng x ω = G and x ω = B. Now, consder ncentves n the announcement round. Recall that n ths round, player s asked to report her nference ω(). Snce ths nference ω() s prvate nformaton, the opponent cannot dstngush whether player s report s truthful or not, and thus punshng player s devaton s a non-trval task. In order to solve ths problem, we use Condton 3, whch ensures the correlaton of prvate sgnals across players. The dea s smple: Snce sgnals are correlated, the opponent s sgnals durng player s learnng round contan (mperfect) nformaton about player s nference ω(), and thus we ask the opponent to punsh player f player s report s dfferent from what the opponent expected to see. In the proof, we use ths dea and construct a punshment mechansm under whch player s always wllng to tell the truth. More precsely, our constructon of the punshment mechansm conssts of two steps. In the frst step, we construct a smple mechansm under whch the truthtellng s ε-optmal for player, n the ex-post sense; that s, under ths mechansm, reportng ω() truthfully s ε-optmal gven any nference ω() and gven any state of the world ω (but not gven the opponent sgnals n the learnng round). Then n the second step, we modfy the mechansm to ensure the exact optmalty of the 15

16 truthful report. In ths second step, nformaton released n the report round plays an mportant role. Very roughly speakng, the mechansm we consder n the frst step s as follows. To smplfy the dscusson, assume that there are only two states, ω and ω. Supoose that the true state s ω, and consder player s ncentve gven ths ω. Let f be the opponent s sgnal frequency n player s learnng round, and suppose that t was very close to the theoretcal dstrbuton π ω (ag ). Note that such a hstory s very lkely gven the true state ω. In ths case, we ask the opponent to punsh player who reports the wrong nference ω() = ω; also we ask the opponent to make player ndfferent between reportng ω() = ω and ω() = /0. (Ths can be done by choosng the probablty of x ω = B approprately.) The dea behnd ths mechansm s qute smple: When the opponent s sgnal frequency f was close to the theoretcal dstrbuton π ω (ag ), the opponent expects that player s sgnal frequency was also close to the theoretcal dstrbuton π ω (a G ). 7 Ths means that the opponent should expect player to report ω() = ω (or at least ω() = /0), but not ω() = ω. Hence, f player reports ω, then the opponent wll punsh t. Now suppose that the opponent s observaton f was not close to π ω (ag ). Ths s a very unlkely event gven the true state ω, and n ths case we ask the opponent to make player ndfferent over all reports n the announcement round. Table 1 summarzes player s ncentves gven the mechansm proposed here. If f s close to π ω (ag ) If f s not close to π ω (ag ) Reportng ω() = ω Optmal Optmal Reportng ω() = ω Not optmal Optmal Reportng ω() = /0 Optmal Optmal Table 1: Incentves gven the true state ω 7 Note that C ω (ag )π ω (ag ) = π ω (a G ), snce each component of C ω (ag )π ω (ag ) s π ω (z a G ) z π ω (z a G ) πω (z a G ) = π ω (z a G ). Then by the contnuty, f f s close to π ω (ag ), the opponent s estmate about player s observaton, C ω (ag ) f, must be close to π ω (a G ). Ths mples that the opponent should beleve that player s sgnal frequency s close to the theoretcal dstrbuton π ω (a G ). 16

17 We clam that the truthful report s ε-optmal for player gven the true state ω under ths punshment mechansm. Frst, consder player wth nference ω() = ω,.e., consder player whose nference s correct. In ths case, the truth-tellng s exactly optmal, as reportng ω() = ω s optmal regardless of the opponent s observaton f. Smlarly, f player nference s ω() = /0, the truthful report s optmal. Now, consder player wth nference ω() = ω,.e., consder player who made a wrong nference. In ths case, the truthful report s not optmal, as t would be punshed f the opponent s observaton f were close to π ω (ag ). However, when player s nference was ω() = ω, her observaton n the learnng round should be close to π ω (a G ), and thus accordng to Condton 3, she should beleve that the opponent s observaton f was not close to π ω (ag ) almost surely. Therefore, the fear of the punshment when player reports ω() = ω truthfully s close to zero, and hence the truthful report s ε-optmal. As argued above, we need to further modfy ths mechansm to provde the exact truth-tellng ncentves, usng the nformaton dsclosed n the report round. How to modfy the mechansm, along wth the ncentves n the report round, wll be descrbed n Secton Here are a few remarks about our punshment mechansm. Frst, correlaton of prvate sgnals (Condton 3) s essental for the mechansm to work. If sgnals are ndependent across players, the opponents sgnals have no nformaton about player s sgnals, and thus punshng player contngent on the opponents observaton cannot provde effectve ncentves. Specfcally, when sgnals are ndependent and player s nference s ω() = ω, she beleves that the opponent s observaton f was close to the theoretcal dstrbuton π ω (ag ) gven the true state ω; thus player expects that the truthful report of ω() = ω wll be punshed and hence wants to msreport. Second, reportng ω() = /0 s a safe opton n the above mechansm, n the sense that t s optmal regardless of the true state ω and of the opponents sgnals. (Note that reportng ω() = ω s suboptmal f the true state s ω, and f f s close to π ω (ag )). To see the role of ths safe opton, consder player whose observaton was not close to π ω (a G ) for all ω;.e., consder player who s unsure about the true state. In such a case, we ask player to take the safe opton ω() = /0 n the equlbrum. Ths guarantees that player s play s ndeed optmal gven any state ω, although Condton 3 tells us nothng about her belef about the 17

18 opponents sgnals. Thrd, n our punshment mechansm, we consder ncentves va value burnng only; we cannot use ncentves va utlty transfers because each player s contnuaton payoff s determned ndependently by the ntenton of player 1. 8 Therefore, n order to approxmate effcent outcomes, we have to mantan the probablty of value burnng close to zero. Ths property s ndeed satsfed n our mechansm; we wll construct the mechansm n such a way that the value destructon occurs only when the realzed sgnal frequency of some player s not close to the theoretcal dstrbuton, whch s a very rare event on the equlbrum path. See Remark 2 for more detals. 4.2 Learnng Round As explaned, players tres to learn the true state n the learnng round. Here we gve more detals about how they learn the state. In our equlbra, the learnng round s regarded as a sequence of T -perod ntervals; snce the learnng round conssts of N Ω ( Ω 1) 2 T perods, there are N Ω ( Ω 1) 2 ntervals. Each of these ntervals s labeled by (,ω, ω) wth ω ω. (Here we dentfy (,ω, ω) and (, ω,ω), so the order of two states do not matter.) Roughly, n the T -perod nterval labeled by (,ω, ω), player compares ω and ω and determnes whch s more lkely to be the true state, by aggregatng T prvate sgnals. Let T (,ω, ω) denote the set of perods ncluded n ths T -perod nterval, and let T () be the unon of T (,ω, ω) over all possble pars (ω, ω). Intutvely, T () s the set of perods n whch player tres to learn the true state. Durng the learnng round, players are asked to play a G, so Condton 2 ensures that each player can dstngush the true state from her prvate sgnals. Some remarks are n order. Frst, snce we take T suffcently large, each player can make a correct nference wth a probablty close to one. That s, f the true state were ω or ω, player s nference n the T -perod nterval T (,ω, ω) would concde wth the true state almost surely. Second, players make statstcal nferences sequentally, that s, n the T -perod nterval T (,ω, ω), only player 8 Essentally, ths requrement comes from the fact that the equlbrum we construct s (perodcally) belef-free, as n Hörner and Olszewsk (2006). It s well known that the analyss of belef-based equlbra s ntractable when we consder the case n whch players observe prvate sgnals. 18

19 tres to make an nference whle the opponents do not. Thrd, f there are more than two states, each player makes multple statstcal nferences. For example, f there are three states ω 1, ω 2, and ω 3, player conducts three statstcal tests n the learnng round; one compares ω 1 and ω 2 usng sgnals n T (,ω 1,ω 2 ), one compares ω 2 and ω 3 usng sgnals n T (,ω 2,ω 3 ), and one compares ω 1 and ω 3 usng sgnals n T (,ω 1,ω 3 ). Now we consder what players do n the T -perod nterval T (,ω, ω) n more detals. As argued, player makes an nference n ths nterval, and we denote her nference by r (ω, ω) {ω, ω, /0}. Roughly, r (ω, ω) = ω means that player thnks that ω s lkely to be the true state; r (ω, ω) = ω means that player thnks that ω s lkely to be the true state; and r (ω, ω) = /0 means that the learnng fals and player s not sure about the true state. Formally, r (ω, ω) s a random varable whose dstrbuton s a functon of player s prvate hstory h T durng the nterval T (,ω, ω). Let P( h T ) be the dstrbuton of r (ω, ω) when h T s gven. Then, for each state ω and for each sequence of acton profles (a 1,,a T ), let ˆP( ω,a 1,,a T ) denote the condtonal dstrbuton of r (ω, ω) nduced by P gven that the true state s ω and players follow (a 1,,a T ) durng the T -perod nterval. That s, ˆP( ω,a 1,,a T ) = Pr(h T ω,a 1,,a T )P( h T ) h T where Pr(h T ω,a 1,,a T ) s the probablty of h T when the true state s ω and players follow (a 1,,a T ). Lkewse, for each j, t {1,,T 1} and h t j, let ˆP( ω,h t j,at+1,,a T,) be the condtonal dstrbuton of r (ω, ω) gven that the true state s ω, player j s hstory up to the tth perod of the nterval s h t j = (aτ,z τ j )t τ=1, and players follow the acton sequence (at+1,,a T ) thereafter. Gven h T, let f (h T ) Z be player s sgnal frequency correspondng to h T. The followng lemma shows that there s a dstrbuton P( h T ) whch satsfes some mportant propertes. The proof s smlar to Fong, Gossner, Hörner and Sannkov (2011) and Sugaya (2012), and can be found n Appendx. Lemma 1. Suppose that Condton 2 holds. Then there are ε and T such that for any ε (0,ε) and T > T, we can choose P( h T ) {ω, ω, /0} for each ht that all the followng condtons are satsfed: so 19

20 () If the true state s ether ω or ω and players play a G n the T -perod nterval, the nference r (ω, ω) concdes wth the true state almost surely: For each ω {ω, ω} ˆP(r (ω, ω) = ω ω,a G,,a G ) 1 exp(t 2 1 ). () Regardless of the past hstory, player j s devaton cannot manpulate player s nference almost surely: For each ω {ω, ω}, j, t {1,,T 1}, h t j, (aτ ) T τ=t+1, and (ãτ ) τ=t+1 T such that aτ j = ãτ j = ag j for all τ, ˆP( ω,h t j,a t+1,,a T,) ˆP( ω,h t j,ã t+1,,ã T,) exp(t 1 2 ). () Whenever player s nference s r (ω, ω) = ω, her sgnal frequency durng the nterval s close to the theoretcal dstrbuton π ω (a G ) at ω : For all h T = (a t,z t )T t=1 such that at = a G for all t and such that P(r (ω, ω) = ω h T ) > 0, π ω (a G ) f (h T ) < ε. Clause () means that player s state learnng s almost perfect, and clause () mples that player j can earn almost no gan even f she devates n the nterval T (,ω, ω). Note that both clauses () and () are natural consequences of Condton 2, whch guarantees that player can learn the true state even f someone else unlaterally devates. Clause () mples that player forms the nference r (ω, ω) = ω only f her sgnal frequency s close to the theoretcal dstrbuton π ω (a G ) at ω. So f her sgnal frequency s not close to π ω (a G ) or π ω (a G ), she forms the nference r (ω, ω) = /0. Clause () has an mportant mplcaton about player s belef about the opponents sgnal frequency. To see ths, the followng lemma s useful: Lemma 2. Suppose that Condton 3 holds. If ε s small enough, then for any ω and f Z such that π ω (a G ) f < ε for some ω ω, we have π ω (ag ) C ω (a G ) f ε. Proof. Snce ε s small, we have C ω (a G )π ω (a G ) C ω (a G ) f ε for all (a G ) f < ε for some ω ω. Also, from Cond- f Z such that π ω ton 3 and contnuty, we have π ω (ag ) C ω (a G )π ω (a G ) 2 ε for small ε. Combnng these two, we obtan the desred nequalty. Q.E.D. 20

21 Ths lemma, together wth clause (), mples that whenever player s nference s r (ω, ω) = ω, we have π (a ω G ) C ω (a G ) f (h T ) ε. (1) In words, f the true state was ω but nonetheless player made a wrong nference r (ω, ω) = ω, then she would beleve that the opponents sgnal dstrbuton would be also dstorted and not close to the theoretcal dstrbuton π ω (ag ) at ω. Lkewse, whenever player s nference s r (ω, ω) = ω, we have π (a ω G ) C ω (a G ) f (h T ) ε. That s, f the true state was ω but nonetheless player made a wrong nference r (ω, ω) = ω, then she would beleve that the opponents sgnal dstrbuton would be not close to the theoretcal dstrbuton π ω (ag ). These propertes are essental when we consder ncentves n the announcement round. Detals wll be gven later. In ths way, durng the learnng round, player makes an nference r (ω, ω) for each par (ω, ω). Then at the end of the learnng round, she summarzes all the nferences r (ω, ω) and make a fnal nference ω() Ω {/0}. The dea s that she selects ω as a fnal nference f t beats all the other states ω ω n the relevant comparsons. Formally, we set ω() = ω f r (ω, ω) = ω for all ω ω. If such ω does not exst, then we set ω() = / Announcement, Man, and Report Rounds To smplfy our dscusson, we assume that K = 1 n the rest of the proof, that s, we assume that there are suffcently many actons so that the length of the announcement round s one. 9 As explaned, n the announcement round, each player reports ω() Ω {/0} through her actons. That s, we consder a partton {A (n)} n Ω {/0} of A and player wth an nference ω() chooses an acton a A (ω()) n the announcement round. 9 There s no dffculty n extendng our result to the case wth K 2. When K 2, each player can get the opponents prvate nformaton through ther actons n the mddle of the announcement round. Specfcally, player can learn what player j observed durng the T -perod nterval for ( j,ω, ω). However, player cannot get any nformaton about what player j observed durng the T -perod nterval T (,ω, ω). Hence Lemma 5 remans true for any perod of the announcement round,.e., the truthful report s stll almost optmal n the announcement round. 21

22 The communcaton n the announcement round allows players to check whether they can agree on the true state, and t nfluences the contnuaton play n the man round. If players could agree that the true state s ω, then each player would reveal her relevant automaton state, x ω {G,B}, at the begnnng of the man round. (She does not reveal x ω for ω ω.) Recall that player ( 1) s automaton state represents her ntenton about whether to punsh or reward player ; so through ths communcaton, players can make sure who should be punshed and who should be rewarded. Specfcally, player should be punshed f player 1 expresses the bad ntenton x ω = B, and should be rewarded f player 1 expresses the good ntenton x ω = G. Then n the remanng perods of the man round, players choose approprate actons contngent on the revealed ntentons. Formally, player s strategy n the man round s descrbed as follows. For each state ω and each ntenton profle x ω = (x ω) I, let a ω,xω be an acton profle such that g ω(aω,xω ) > v ω for each wth x 1 ω = G and gω (aω,xω ) < v ω for each wth x 1 ω = B. That s, the acton profle aω,xω yelds hgh payoffs to those who should be rewarded accordng to the ntenton profle x ω, and low payoffs to those who should be punshed. See Fgure 2 for how to choose actons a ω,xω n two-player games. 10 If all players reported the same state ω n the announcement round, then player reports her ntenton x ω n the frst perod of the man round. Specfcally, she chooses a G f her automaton state s x ω = G, whle she chooses some other actons f x ω = B. Note that she reports the ntenton for state ω only; that s, she does not report her ntenton for other states, lke x ω. Then n the remanng perods of the man round, she plays the acton a ω,xω, where x ω = (x ω) I s the ntentons revealed n the frst perod of the man round. If player j unlaterally devates from a ω,xω, then player plays the mnmax acton α ω ( j) n the remanng perods of the man round. If there s j I (possbly j = ) such that each player l j reported the same state ω whle player j reported /0, then play s the same as above. If there s j I (possbly j = ) such that each player l j reported the same 10 Dependng on the payoff functon, such acton profles a ω,xω may not exst. In ths case, as n Hörner and Olszewsk (2006), we take acton sequences (a ω,xω (1),,a ω,xω (n)) nstead of acton profles; the rest of the proof extends to ths case wth no dffculty. 22

23 state ω whle player j reported a dfferent state ω ω, then player reports x ω n the frst perod, and plays the mnmax acton α ω ( j) n the remanng perods. Otherwse, the play s the same as n the case n whch all players reported ω 1. Player 2 s payoffs w GB w GG V (ω) I = {1,2} x ω {GG,GB,BG,BB} v ω 2 v ω 2 w BB v ω w BG w xω = (w xω 1,wxω 2 ) w xω = g ω (aω,xω ) v ω 1 v ω 1 Fgure 2: Actons Player 1 s payoffs To get a more concrete dea about the man round, suppose that the true state s ω and players automaton state s x n the current block game. When T s suffcently large, players can successfully learn ω n the learnng round wth a very hgh probablty, and they report t n the announcement round. In ths hstory, players agree that the true state s ω; thus n the man round, they behave as we descrbed n the frst bullet pont. That s, they reveal ther ntentons x ω = (x ω) and then play a ω,xω untl the end of the man round. By the defnton of a ω,xω, t s easy to see that player s average payoff n the man round s hgher than v ω f x 1 ω = G, and s lower than vω f x 1 ω = B. Also, snce the average payoff of the block game s approxmated by that of the man round, player s average block-game payoff s hgher than v ω f x 1 ω = G, and s lower than vω f x 1 ω = B. That s, player s ndeed rewarded f player ( 1) s ntenton for state ω s good, and she s punshed f player ( 1) s ntenton s bad. 23

24 In the report round, players reports ther prvate hstores n the learnng round. In the frst KT Ω ( Ω 1) 2 perods of the report round, each player reports all the sgnals n the perods n the set T (), that s, each player reports her observatons durng the perods n whch she tred to learn the true state. Then n the remanng K(N 1)T Ω ( Ω 1) 2 perods, each player reports the sgnals n the perods n the set T ( j) for each j. As we wll explan, the nformaton revealed n the report round s used when we determne the transton rules of players automaton states. 4.4 Auxlary Scenaro So far we have explaned how each player behaves n the block game when her automaton state x s gven. Thus the descrpton of the equlbrum strategy of the nfntely repeated game s completed by specfyng how the automaton state x s determned at the begnnng of each block game. But before dong so, t s convenent to consder the auxlary scenaro game n whch () the true state ω and players automaton states x are gven and common knowledge, and () after the block game wth T b perods, the game ends and each player obtans a transfer U ω,xω 1 (h T b 1 ). In ths auxlary scenaro gven (ω,x), player s (unnormalzed) expected payoff s T b t=1 δ t 1 g ω (a t ) + δ T b U ω,xω 1 (h T b 1 ). There are two remarks on the transfer functon U ω,xω 1. Frst, the amount of the transfer, U ω,xω 1 (h T b 1 ), s dependent on the play n the block game only through player ( 1) s prvate hstory h T b 1 ; that s, the amount of the transfer does not depend on prvate nformaton of player j 1. Second, the transfer functon U ω,xω 1 depends on the true state ω and on player ( 1) s automaton state x ω 1 {G,B} for that ω. Because ω s not observable to players n our orgnal model, one may wonder why we should be nterested n the auxlary scenaro game n whch ω s common knowledge. The reason s that we wll construct an ex-post equlbrum n the nfntely repeated game; that s, our equlbrum strategy profle must be an equlbrum even f the state ω were revealed to players. Hence we need to consder players ncentves when ω s common knowledge, whch motvates the study of the auxlary scenaro game. 24