We Can Cooperate Even When the Monitoring Structure Will Never Be Known

Transcription

1 We Can Cooperate Even When the Montorng Structure Wll Never Be Known Yuch Yamamoto Frst Draft: March 29, 2014 Ths Verson: Aprl 8, 2017 Abstract Ths paper consders nfnte-horzon stochastc games wth hdden states and hdden actons. The state changes over tme, players observe only a nosy publc sgnal about the state each perod, and actons are prvate nformaton. In ths model, uncertanty about the montorng structure does not dsappear. We show how to construct an approxmately effcent equlbrum n a repeated Cournot game. Then we extend t to a general case and obtan the folk theorem usng ex-post equlbra under a mld condton. Journal of Economc Lterature Classfcaton Numbers: C72, C73. Keywords: stochastc game, hdden state, publc montorng, pseudoergodc strategy, folk theorem, ex-post equlbrum. The author thanks George Malath and Takuo Sugaya for helpful conversatons, and semnar partcpants at varous places. Department of Economcs, Unversty of Pennsylvana. Emal: yyam@sas.upenn.edu 1

2 Contents 1 Introducton Overvew of the Argument Lterature Revew Setup Stochastc Games wth Hdden States Belef Convergence Theorem Example: Stochastc Cournot Competton Model Feasble Payoff Set and Optmal Polces Pseudo-Ergodc Strateges Random Blocks and Self-Generaton Regular Drectons Coordnate Drectons and Value Burnng Comments and Remarks State-Specfc Punshment vs Unform Punshment Perfect Ex-Post IC vs Perodc Ex-Post IC Pseudo-Ergodc Strateges Feasble Payoff Set and Pseudo-Ergodc Strateges Mnmax Payoffs and Pseudo-Ergodc Strateges The Folk Theorem 44 Appendx A: Cournot Example 46 Appendx B: Proofs 48 B.1 Proof of Proposton B.2 Proof of Proposton B.2.1 Step 1: Optmal Polcy for Interor Belefs B.2.2 Step 2: Optmal Polcy for General Belefs B.2.3 Step 3: Contnuaton Strateges B.3 Proof of Proposton

3 B.3.1 Step 1: Almost Flat Convex Curve B.3.2 Step 2: Some Convex Curve Approxmates the Maxmal Value B.3.3 Step 3: All Convex Curves Approxmate the Maxmal Value B.4 Proof of Proposton B.5 Proof of Proposton B.5.1 Step 1: Bound on Devaton Payoffs B.5.2 Step 2: Strateges wth Cross-State Full-Rank Condtons. 69 B.5.3 Step 3: Enforceablty for Regular Drectons B.5.4 Step 4: Enforceablty for Postve Coordnate Drecton. 84 B.5.5 Step 5: Enforceablty for Negatve Coordnate Drecton. 89 B.5.6 Step 6: Unform Enforceablty Appendx C: Dspensablty of Publc Randomzaton 91 Appendx D: Relaxng the Full Support Assumpton 99 D.1 Unform Connectedness and Feasble Payoff Set D.2 Robust Connectedness and Mnmax Payoffs D.3 Belef Convergence Theorem D.4 Folk Theorem D.5 Ex-Post Equlbra to Sequental Equlbra D.6 Proof of Proposton D D.6.1 Step 1: Mnmax Payoff for µ D.6.2 Step 2: Convex Curves when Ω(µ,h t ) s Robustly Accessble D.6.3 Step 3: Convex Curves when Ω(µ,h t ) s Transent D.7 Proof of Proposton D D.8 Proof of Proposton D

4 1 Introducton The theory of repeated games provdes a framework to study the role of long-term relatonshps n facltatng cooperaton. Past work has shown that recprocaton can lead to more cooperatve equlbrum outcomes even f there s mperfect publc montorng so that each perod, players observe only a nosy publc sgnal about the actons played (Abreu, Pearce, and Stacchett (1990) and Fudenberg, Levne, and Maskn (1994, hereafter FLM)). Ths work has covered a range of applcatons, from olgopoly prcng (e.g. Green and Porter (1984) and Athey and Bagwell (2001)), repeated partnershps (Radner, Myerson, and Maskn (1986)), and relatonal contracts (Levn (2003)). A common assumpton n the lterature s that players know the montorng structure,.e., they know the dstrbuton of publc sgnals as a functon of the actons played. Fudenberg and Yamamoto (2010) relax ths assumpton and consder players who ntally do not know the montorng structure. However, n ther model, the true montorng structure s fxed over tme, and thus players can learn t from observed sgnals n the long run. In partcular, ther analyss reles on the fact that patent players care only about payoffs n a dstant future n whch uncertanty about the montorng structure vanshes. Assumng (asymptotcally) perfect knowledge of the montorng structure s restrctve. To address ths concern, ths paper consders a model n whch uncertanty about the montorng structure never dsappears. Specfcally, we consder a model wth unknown, perpetually changng montorng structure. In ths model, players may obtan some nformaton about the current montorng structure from the sgnal today, after they choose actons. But then n the next perod, the montorng structure wll stochastcally change, so players wll contnue to face new uncertanty. Changng montorng structures naturally arse when the underlyng economc condtons change over tme. One example s a repeated Cournot model wth hdden correlated demand shocks. Suppose that the state of the economy ω, whch nfluences the dstrbuton of the market prce today, s hdden nformaton and postvely correlated over tme. As n Green and Porter (1984), the realzed prce s regarded as a nosy publc sgnal about the current actons (quanttes), because hgher quanttes nduce low prces more lkely. In ths model, the sgnal (prce) dstrbutons are dfferent for dfferent perods, snce the state ω s not..d.. Also 4

5 the true sgnal dstrbuton s unknown each perod, as the state ω s not observable. Hence the montorng structure s unknown and changng. Another example s a repeated prncpal-agent problem. If the agent s productvty s unobservable, and s changng due to experence, then the true dstrbuton of the output s unknown and changng. Ths paper shows that long-run relatonshps facltate cooperaton even n such stuatons. In partcular, we show that the folk theorem obtans usng publc-strategy equlbra. Formally, we consder a new class of stochastc games n whch a hdden, changng state nfluences the montorng structure (the sgnal dstrbuton) n the current stage game. Actons can nfluence the state transton. Each player s stagegame payoff depends on her own acton and the publc sgnal, so the payoff does not contan more nformaton than the sgnal. In ths setup, the hdden state ndrectly nfluences the stage-game payoffs through the dstrbuton of the publc sgnal. For example, n a repeated olgopoly, frms have hgher expected payoffs at a state n whch hgh prces are more lkely. So uncertanty about the probablty of hgh prces leads to uncertanty about the expected payoffs of the stage game. Snce we assume that actons are prvate nformaton, even f players have an ntal common pror about the state, ther posteror belefs can potentally dverge n later perods. For example, f a player chooses a mxed acton, the realzed acton s her prvate nformaton, and she updates her posteror gven ths nformaton. Smlarly, f a player devates from an equlbrum strategy, she wll update her posteror gven her devaton, whle the opponents wll update wthout knowng t. A common technque n the lterature s to allow cheap-talk communcatons to resolve conflctng nformaton (e.g., Kandor and Matsushma (1998)), but t does not seem to easly apply to our setup. 1 Dvergng posteror belefs can cause a mscoordnaton problem. Suppose 1 Kandor and Matsushma (1998) consder repeated games wth prvate montorng and communcaton, n whch there s no payoff-relevant state and each player reports prvate sgnals about the opponents actons. They focus on publc equlbra n whch the play depends only on publc reports. Then a player s contnuaton payoff s a functon of the past publc reports, whch allows them to use recursve tools to characterze the equlbrum payoff set. In contrast, n ths paper, each player has a prvate belef µ about the payoff-relevant state, and ths prvate belef drectly nfluences her contnuaton payoffs. That s, the contnuaton payoff depends on the true belef µ and s not a functon of publc hstores. Hörner, Takahash, and Velle (2015) argue that the equlbrum analyss becomes sgnfcantly harder n such a case. They show that t s stll possble to provde truthful ncentves f some assumptons are satsfed (e.g., ndependent prvate values); but unfortunately, these assumptons do not hold n our setup. 5

6 that there are two players who want to reward each other (.e., they each want to gve hgh payoffs to the other). If they have the same belef µ about the state, ths can be done by playng a welfare-maxmzng strategy profle, say s eff (µ). Note that ths profle depends on the belef µ, because the stage-game payoffs depend on the hdden state ω. On the other hand, when players have dfferent belefs and these belefs are prvate nformaton, t s less clear what they should do. Indeed, f each player smply chooses the welfare-maxmzng strategy correspondng to her own belef, the resultng strategy profle may not maxmze the welfare due to mscoordnaton. A smlar problem arses when we consder a player who wants to punsh the opponent. If the opponent s belef s prvate nformaton, t s unclear how to punsh the opponent, as the effectve punshment depends on the opponent s belef n general. To overcome ths problem, ths paper ntroduces the dea of pseudo-ergodc strateges. In general, gven a strategy profle n the nfnte-horzon game, dfferent ntal prors nduce dfferent payoff streams, whch result n dfferent average payoffs. Pseudo-ergodc strateges are a specal class of strategy profles n whch all ntal prors yeld approxmately the same average payoffs. Such a property may sound demandng, but t turns out that n our settng, for an arbtrarly fxed belef µ, the correspondng welfare-maxmzng strategy profle s eff ( µ) s a pseudo-ergodc strategy whch approxmates the welfare-maxmzng payoff regardless of the true belef µ. A rough dea s that when players play ths strategy profle s eff ( µ), the resultng payoff stream n the nfnte-horzon game has a flavor of ergodcty n the sense that the ntal belef does not nfluence the contnuaton payoff after a long tme. Ths ndeed mples that the strategy profle s eff ( µ) yelds almost the same payoff for all ntal belefs, as patent players care only about payoffs n a dstant future. See Secton 4.1 for more detals. So f players want to reward each other, they may gnore ther prvate belefs and smply play the above profle s eff ( µ). That s, they may form a dummy publc belef µ and play the correspondng strategy. Ths approxmates the effcent payoff regardless of ther ntal prvate belefs. Smlarly, f a player plays the mnmax strategy for some dummy belef µ, t approxmates the mnmax payoff regardless of the true belef µ. In ths way, players can reward or punsh ther opponent wthout fne-tunng the strategy dependng on ther prvate belefs. The next queston s whether we can actually construct an equlbrum by as- 6

7 semblng these pseudo-ergodc strateges: We need to fnd an effectve punshment mechansm when players have dvergng belefs about the true montorng structure. To solve ths problem, we consder a punshment mechansm n whch a devaton today wll lower contnuaton payoffs regardless of the current hdden state ω. Under ths mechansm, ex-post ncentve compatblty s satsfed n that any devaton today s prevented regardless of the current hdden state ω; hence ths mechansm works even f there s uncertanty about the state ω. Of course, ex-post ncentve compatblty s more demandng than Bayesan ncentve compatblty, and n general, the set of ex-post equlbrum payoffs s smaller than the set of sequental equlbrum payoffs. However, n our envronment, t s possble for ex-post equlbra to approxmate the Pareto-effcent fronter. Indeed, our man result s the folk theorem: We show that any feasble and ndvdually ratonal payoff can be approxmated by a publc ex-post equlbrum, f players are patent and f cross-state ndvdual full rank and cross-state parwse full rank hold. The cross-state full-rank condton s an extenson of ndvdual full rank and parwse full rank of FLM, and requres that a publc sgnal can statstcally dstngush the current state and the chosen acton profle. Fudenberg and Yamamoto (2010) also consder ex-post equlbra when players face uncertanty about the montorng structure. However, there are mportant dfferences between ther work and ths paper. As noted, n Fudenberg and Yamamoto (2010), players can learn the true montorng structure n the long run. Then players ncentve problems can be decomposed state by state; ths s because t s possble to nfluence players ncentves n some state ω wthout affectng ncentves n other states, by changng players contnuaton play n a dstant future n whch players have learned the true state ω. Ths property helps to provde ex-post ncentves. 2 On the other hand, n our model, the state today s never revealed to players, and thus the above dea does not apply. Accordngly, ncentve problems for dfferent states are entangled n a non-trval way, and provdng ex-post ncentve compatblty becomes qute delcate. In partcular, the state-specfc punshment of Fudenberg and Yamamoto (2010) do not work effectvely n our envronment. 2 Ther analyss s more complcated than the dscusson here, because ex-post ncentves must be provded each perod. They develop a useful recursve method and show that t s ndeed possble to provde such ncentves. 7

8 More detaled dscussons wll be gven n Secton 3.7. The contrbutons of ths paper are two-fold. Frst, we provde a general dea on how to construct an equlbrum n a new envronment, at least for hgh dscount factors δ. In partcular, we llustrate how dynamc ncentves can be effectvely and smply provded va publc pseudo-ergodc strateges. Second, we show that ex-post equlbrum can approxmate effcent outcomes, even f the state changes over tme so that state learnng s mpossble. As wll be explaned n the next subsecton, utlty transfer across players of FLM cannot provde approprate ncentves n our envronment, and we construct a new punshment mechansm whch works out even when there s a hdden changng state. 1.1 Overvew of the Argument To understand the crtcal steps n our proof, t s useful to revew the deas of FLM, who prove the folk theorem for repeated games wth publc montorng. Ther man fndng s that when players are patent, any ball W n the nteror of the feasble and ndvdually ratonal payoff set (see Fgure 1) s self-generatng. That s, each payoff v n the ball W s achevable by (some acton profle today and) contnuaton payoffs n the ball W tself. As shown by Abreu, Pearce, and Stacchett (1990), such a ball W s attaned by publc equlbra, and hence the folk theorem ndeed follows. How do they prove that the ball W s self-generatng? As a frst step, they show that each payoff v on the boundary of the ball W can be acheved usng contnuaton payoffs w on a translate of the tangent lne. For example, take the target payoff v as n Fgure 1. (As we wll soon see, ths s the most dffcult case n our proof.) FLM show that ths payoff v s achevable by the acton profle a X whch yelds the payoff X n the fgure, and by some contnuaton payoffs on the horzontal lne L. Here, the contnuaton payoffs take dfferent values for dfferent sgnals, so that player 1 s devaton today s deterred. Also, snce player 2 s contnuaton payoff s constant on the lne L and the acton profle a X acheves the best payoff X for player 2, she has no ncentve to devate ether. So approprate ncentves are ndeed provded by the contnuaton payoffs on the lne L. Wthout loss of generalty, we can assume that the varaton n contnuaton payoffs (the dstance between w and w n the fgure) s of order O(1 δ); such contnuaton 8

9 payoffs can ndeed deter player 1 s devaton today, because her gan by devatng s of order O(1 δ). Also, the length D n the fgure, whch measures the dstance from the payoff v to the lne L, s of order O(1 δ). Ths s so because v must be exactly acheved as the weghted average of today s payoff X and the expected contnuaton payoff on the lne L, where the weght on today s payoff s 1 δ. X Player 2 s payoff D v w w w L W V Player 1 s payoff Fgure 1: Contnuaton payoffs w, w, and w are on the lne L. Then as a second step, FLM show that f contnuaton payoffs w move only on the lne L (and f the varaton s of order O(1 δ)), they stay n the nteror of the ball W. The proof dea s llustrated n the left panel of Fgure 2; as one can see, the dstance to the boundary of the ball W s of order O( 1 δ), whch s much larger than the varaton n the contnuaton payoffs w, and thus w never goes to the outsde of the ball W. Ths result mples that the contnuaton payoffs constructed n the frst step are n the ball W, so any boundary pont v of the ball W s achevable by contnuaton payoffs n W. They also show that the same result holds even f v s an nteror pont of W, so n sum, any payoff v n the ball W s achevable by contnuaton payoffs n W tself. Hence W s ndeed self-generatng. To summarze, the key technque of FLM s to decompose the target payoff v nto two parts: The one-shot acton profle a X and the contnuaton payoffs on the lne L (whch are always n the ball W). Our proof extends ths technque to the case n whch the montorng structure s unknown and changng. Snce there s a hdden changng state n our model, new complcatons arse, and we need to modfy the proof accordngly. Specfcally, we make the followng changes: We replace the acton profle a X above wth a pseudo-ergodc block strategy, whch approxmates the payoff X regardless of players prvate belefs. 9

10 We allow the contnuaton payoffs to move vertcally, so they are not on the lne L. (But they are stll n the ball W, so the ball s self-generatng. See Fgure 2.) In what follows, we wll explan why we need such changes, and how they work. To begn wth, note that the defnton of the feasble payoff set n our envronment s dfferent from the one n the standard repeated game; snce the stage game payoffs are nfluenced by a hdden, changng state ω, they are not feasble payoffs n the nfnte-horzon game. Instead, gven the ntal pror µ about the state and the dscount factor δ, we defne the feasble payoff set V µ (δ) as the set of all possble payoff vectors n the nfnte-horzon game. Let V µ denote the lmt of the feasble payoff set as δ 1; ntutvely, ths s the feasble payoff set when players are patent. In the specal case n whch the state s observable and follows a Markov process, ths lmt feasble payoff set V µ does not depend on the ntal pror µ, because the state eventually converges to the statonary dstrbuton regardless of the ntal state. Our Proposton 2 shows that under a mld condton, the same result holds even for our general model n whch the state s unobservable and nfluenced by actons. So we denote ths lmt feasble payoff set by V, as n Fgure 1. Snce the feasble payoff set s qute dfferent from the stage-game payoffs, each extreme pont of the feasble payoff set V may not be attaned by any oneshot acton profle n our model. For example, n order to decompose the payoff v n Fgure 1, FLM use the acton profle a X, whch yelds the best payoff X for player 2 wthn the feasble payoff set. In our model, such an acton profle may not exst, as the payoff X s a payoff n the nfnte-horzon game, rather than a FLM Ths paper v v w w D L W O(1 δ) O( 1 δ) L W Fgure 2: Vertcal move of w must be less than D = O(1 δ). Ths s more restrctve than the bound on the horzontal move, whch s of order O( 1 δ). 10

11 stage-game payoff. To fx ths problem, we regard the nfnte horzon as a seres of blocks, and treat each block as a bg stage game. The pont s that when the block s suffcently long, each extreme payoff of the feasble payoff set V s approxmated by the average payoff n the block (.e., the payoff n the bg stage game). That s, the dfference between the stage-game payoffs and the feasble payoff set dsappears, f we regard a long block as a bg stage game. In partcular, we use a pseudo-ergodc strategy n each block, so that players prvate belefs about the state have almost no mpact on the block payoff. For example, nstead of the acton profle a X n FLM, we use a pseudo-ergodc block strategy whose block payoff approxmates the payoff X regardless of players belefs. To see how to fnd such a pseudo-ergodc strategy, pck a dummy belef µ arbtrarly, and let s X be the block strategy whch would maxmze player 2 s block payoff f players ntal common pror was µ. In general, ths strategy s X needs not maxmze player 2 s payoff when the true belef µ dffers from µ. However, our Proposton 3 shows that t approxmates the best payoff X regardless of the true belef µ. Ths s X s the pseudo-ergodc strategy we use. 3 Ths explans why we need the change stated n the frst bullet pont: By replacng one-shot acton profles n FLM wth pseudo-ergodc block strateges, we can approxmate each extreme pont of the feasble payoff set, regardless of players belefs. However, ths s not the only change we must make: As noted n the second bullet pont, we consder contnuaton payoffs whch are not on a translate of the tangent lne. We make ths change because we need to construct a publc equlbrum n the presence of the hdden changng state, whch requres contnuaton payoffs to satsfy a more demandng condton than n the standard repeated game. Movng contnuaton payoffs only on a translate of the tangent lne s too restrctve to satsfy ths new condton. To be more specfc, take an arbtrary ball W n the feasble payoff set V. Our goal s to show that ths ball W s acheved by publc equlbra. For ths, t s suffcent to show that the ball W s self-generatng; but the defnton of self-generaton here s slghtly dfferent from the one n FLM, due to the hdden 3 The dea of the block tself s not new: Dutta (1995) uses the same technque n stochastc games wth observable states. The novelty here s to use a pseudo-ergodc strategy, whch allows players to approxmate a desred payoff even though ther block strategy cannot depend on the hdden state. In Dutta (1995), the state s observable, and thus players can use dfferent block strateges for dfferent ntal states. 11

12 changng state. To llustrate the dfference, take the payoff v as n Fgure 1. For the ball W to be self-generatng n our sense, we need to fnd contnuaton payoffs w n the ball W such that regardless of players belefs µ, () the payoff v s exactly acheved as the sum of the block payoff by the pseudo-ergodc strategy s X (whch approxmates the payoff X) and the contnuaton payoff w, and () any devaton from the strategy s X durng the block s not proftable. Ths condton s more demandng than that n FLM, because the choce of w must be ndependent of players ntal belef µ, that s, our contnuaton payoffs must work for all belefs µ. Note n partcular that the condton () above requres s X to be an ex-post equlbrum, n that playng s X s optmal for each player even f the ntal state ω s revealed. 4 To satsfy ths condton, we consder contnuaton payoffs whch move vertcally, as n the rght panel of Fgure 2. Allowng vertcal move s useful for two reasons: (a) The block strategy s X approxmates the payoff X, but does not exactly acheve t. In partcular, dfferent ntal belefs µ yeld (slghtly) dfferent block payoffs to player 2. Ths payoff dfference must be offset by contnuaton payoffs, as we want the same payoff v to be acheved for all belefs. So player 2 s contnuaton payoff cannot be constant, and thus w must move vertcally. (b) When player 2 s belef dffers from the dummy belef µ, the block strategy s X does not maxmze her block payoff, so she can earn a postve proft by devatng from s X. We need to punsh such a devaton va a varaton n contnuaton payoffs. That s, we need to burn player 2 s contnuaton value (relatve to the lne L) after some sgnals. 5 These ssues (a) and (b) could be easly handled f we could choose contnuaton payoffs n an arbtrarly way, but unfortunately, there s a constrant; we must choose the contnuaton payoffs from the ball W. (Otherwse, the ball W s not 4 But ths condton s weaker than perfect publc ex-post equlbra of Fudenberg and Yamamoto (2010), whch requres that n each perod t, the contnuaton strategy s a Nash equlbrum even f the state ω t n that perod s revealed. See Secton 3.7 for more detaled dscussons. 5 The problem (b) here s relevant only when the tangent at the pont v s a coordnate vector. Indeed, when we consder v whose tangent s not a coordnate vector (ths s the case of regular drectons n FLM), we can ncentvze both players by movng contnuaton payoffs on a translate of the tangent lne. See Secton 3.5 for more detals. 12

13 self-generatng.) Ths n partcular mples that the vertcal move of the contnuaton payoffs cannot be greater than the length D n Fgure 2. It turns out that ths constrant s qute restrctve, and makes our problem substantally dfferent from the one n FLM, n the followng sense. Recall that FLM consder the horzontal move only, n whch case the dstance to the boundary of the ball W s of order O( 1 δ). Ths constrant s loose n that contnuaton payoffs are always n the ball W as long as the varaton s of order O(1 δ). In contrast, the bound D on the vertcal move s of order O(1 δ). Hence the contnuaton payoff may go to the outsde of the ball, even f the varaton toward the vertcal drecton s of order O(1 δ). So n sum, we need to solve the above problems (a) and (b), subject to the constrant that the vertcal move s suffcently small. Note that ths constrant s deeply related to the neffcency result of Radner, Myerson, and Maskn (1986), who show that the use of huge value burnng causes huge neffcency. In order to construct an approxmately effcent equlbrum, we must avod such neffcency, so we need to mnmze the amount of value burnng. It s relatvely easy to show that small value burnng s ndeed enough to solve the problem (a). Snce the block strategy s X yelds almost the same block payoff for all belefs µ, only a small perturbaton of the contnuaton payoffs s enough to offset ths payoff dfference. The problem (b) s more delcate. Snce the block strategy s X approxmates the best payoff X for player 2, her ex-ante expected gan by devatng from s X s small; that s, devatng from s X cannot mprove the block payoff by much, f we evaluate payoffs by takng expectatons over the future states and the future hstores. However, ths property needs not mply that small value burnng s enough to solve the problem (b). To see why, suppose that we are now n perod t > 1 of the block and the hstory wthn the block so far s h t 1. If we want to deter player 2 s devaton n the current perod t va small value burnng, we have to show that her gan by such a devaton s small condtonal on the current hstory h t 1. Obvously, ths condton needs not be satsfed even f the ex-ante gan (whch takes the expectaton over h t 1 ) s small. So n order to solve the problem (b) wth small value burnng, we need to carefully evaluate player 2 s gan when she devates n later perods of the block. To do so, t s useful to examne how her posteror belef about the state evolves over 13

14 tme. Snce the state s changng, the belef evoluton n our model s complex, and keepng track of t over a long block s computatonally demandng. Nonetheless, we fnd that under a mld condton, the belef convergence theorem holds, so that the mpact of the current belef on the posteror n a dstant future s almost neglgble. In other words, after a long hstory, all ntal belefs µ nduce almost the same posteror. An mportant consequence of the belef convergence theorem s that even f the true belef µ s qute dfferent from the dummy belef µ n perod one, after a long tme, they nduce asymptotcally the same posterors, µ t and µ t. (See Fgure 3.) Ths result s useful to obtan an effectve bound on player 2 s gan by devatng n a later perod t of the block: Recall that the block strategy s X maxmzes player 2 s payoff gven the dummy ntal belef µ. So after every hstory h t 1, devatng from s X n the contnuaton game s not proftable f player 2 s true posteror µ t equals the dummy posteror µ t. Of course, these posterors µ t and µ t need not be equal, f the ntal belef µ dffers from the dummy belef µ; but as noted above, the belef convergence theorem ensures that the posterors µ t and µ t are asymptotcally the same for large t, even f the ntal belef µ s qute dfferent from µ. Hence, player 2 s gan by devatng n a later perod t s small, and converges to zero as t ncreases. Ths property (n partcular the fact that the gan converges to zero) s useful to fnd an effectve bound on the amount of value burnng whch deters player 2 s devaton n all perods of the block. See Secton 3.6 for more detals. 1.2 Lterature Revew The framework of stochastc games was proposed by Shapley (1953). Dutta (1995) proved the folk theorem for the case of observable actons, and Fudenberg and Yamamoto (2011b) and Hörner, Sugaya, Takahash, and Velle (2011) extend t to games wth publc montorng. All these papers assume that the state of the world s publcly observable at the begnnng of each perod. Yamamoto (2016) consders hdden states, but assumes that actons are observable. Accordngly, the belef s always common across players, and the model reduces to the stochastc game n whch players belef s a common state varable. In ths paper, players belefs are prvate nformaton and there s no common state varable. 14

15 Perod 1 Perod 2 Perod 3 Perod t 0 µ 1 µ 1 µ t µ t Observe y 1 Observe (y 1,y 2 ) Observe (y 1,,y t 1 ) Fgure 3: Belef evoluton when there are only two states. The whole belef space s [0,1]. Each thck lne s the set of all possble posterors gven the past hstory. It shrnks over tme, so eventually all ntal prors nduce the same posteror. Athey and Bagwell (2008), Escobar and Tokka (2013), and Hörner, Takahash, and Velle (2015) consder repeated Bayesan games n whch the state changes as tme goes and players have prvate nformaton about the state each perod. They assume hat the state of the world s a collecton of players prvate nformaton, so f players report ther nformaton truthfully, the state s perfectly revealed before they choose actons. 6 In contrast, n ths paper, the state s not perfectly revealed. Wseman (2005), Fudenberg and Yamamoto (2010), Fudenberg and Yamamoto (2011a), and Wseman (2012) study repeated games wth unknown states. They assume that the state does not change over tme, so that players can (almost) perfectly learn the true state by aggregatng all the past publc sgnals. In our model, the state changes as tme goes and players never learn t perfectly. Ex-post equlbra have been recently used n varous dynamc models, such as Hörner and Lovo (2009), Fudenberg and Yamamoto (2010), Fudenberg and Yamamoto (2011a), Hörner, Lovo, and Tomala (2011), and Yamamoto (2014). They consder the case n whch the state s fxed at the begnnng. Agan ths paper dffers from ther work, because we consder changng states. 7 6 Sectons 4 and 5 of Hörner, Takahash, and Velle (2015) consder equlbra n whch some players do not reveal nformaton, but ther analyss reles on the ndependent prvate value assumpton. 7 There are many papers that dscuss ex-post equlbra n undscounted repeated games; see Koren (1992) and Shalev (1994), for example. 15

16 2 Setup 2.1 Stochastc Games wth Hdden States Let I = {1,,N} be the set of players. At the begnnng of the game, Nature chooses the state of the world ω 1 from a fnte set Ω. The state may change as tme passes, and the state n perod t = 1,2, s denoted by ω t Ω. The state ω t s not observable to players, so they have an ntal common pror µ Ω about ω 1. In each perod t, players move smultaneously, wth player I choosng an acton a from a fnte set A. Let A I A be the set of acton profles a = (a ) I. Actons are not observable, and nstead players observe a publc sgnal y from a fnte set Y. Then players go to the next perod t + 1, wth a new (hdden) state ω t+1. The dstrbuton of y and ω t+1 depends on the current state ω t and the current acton profle a A; let π ω (y, ω a) denote the probablty that players observe a sgnal y and the next state becomes ω t+1 = ω, gven ω t = ω and a. In ths setup, a publc sgnal y can be nformatve about the current state ω and the next state ω. Ths s so because the dstrbuton of y may depend on ω, and y may be correlated wth ω. Let πy ω (y a) denote the margnal probablty of y. Player s payoff n perod t s a functon of her current acton a and the current publc sgnal y, and s denoted by u (a,y). Her expected stage-game payoff condtonal on the current state ω and the current acton profle a s g ω (a) = y Y π ω Y (y a)u (a,y). Here the hdden state ω nfluences a player s expected payoff through the dstrbuton of y. Let g ω (a) = (g ω (a)) I be the vector of expected payoffs. Let g = max ω,a 2g ω (a), and let g = I g. Also let π be the mnmum of π ω (y, ω a) over all (ω, ω,a,y) such that π ω (y, ω a) > 0. Our formulaton encompasses the followng examples: Stochastc games wth observable states. Let Y = Ω Ω Y A, and suppose that π ω (y, ω a) = 0 for y = (y 1,y 2,y A ) such that y 1 ω or y 2 ω. That s, the frst component of the sgnal y reveals the current state, and the second component reveals the next state. The thrd component s a nosy sgnal about actons. Snce the sgnal n the prevous perod perfectly reveals the current state, players know the state ω t before they choose an acton profle a t. Also, the stage-game payoff u (a,y) drectly depends on the current 16

17 state through the frst component y 1 of the sgnal. Ths s exactly the standard stochastc games studed n the lterature. Delayed observaton. Let Y = Ω Y A, and assume that πy ω (y a) = 1 for each y = (y Ω,y A ) such that y Ω ω. That s, the frst component of the current sgnal reveals the current state. The second component s a nosy sgnal about actons. Ths s the case n whch players observe the current state after they choose ther actons. In the nfnte-horzon stochastc game, players have a common dscount factor δ (0,1). Let (ω τ,a τ,y τ ) be the state, the acton profle, and the publc sgnal n perod τ. Player s hstory up to perod t 1 s h t = (aτ,yτ ) t τ=1. Let Ht denote the set of all h t, and let H 0 = {/0}. A publc hstory up to perod t 1 s denoted by h t = (y τ ) t τ=1. Let Ht denote the set of all h t, let H 0 = {/0}, and let H = t=0 H t be the set of all publc hstores. A strategy for player s a mappng s : t=0 H t A. Let S be the set of all strateges for player, and let S = I S. For each strategy s, let s h t be the contnuaton strategy nduced by s after hstory h t. A strategy s s publc f t depends only on publc nformaton,.e., s (h t ) = s ( h t ) for t, ht, and h t such that yτ = ỹ τ for all τ. A strategy profle s s publc f s s publc for all. For each publc strategy s, let s h t be the contnuaton strategy nduced by s after publc hstory h t. Smlarly, s h t denotes the contnuaton strategy profle after publc hstory h t. Let v µ (δ,s) denote player s average payoff n the stochastc game when the ntal pror s µ, the dscount factor s δ, and players play the strategy profle s. Let v µ (δ,s) = (v µ (δ,s)) I be the payoff vector acheved by the strategy profle s, gven µ and δ. We wrte v ω (δ,s) and vω (δ,s) nstead of v µ (δ,s) and vµ (δ,s), when the ntal pror µ puts probablty one on the state ω. As Yamamoto (2016) shows, for each ntal pror µ, dscount factor δ, and publc strategy s, player s best reply s exsts. A strategy profle s s a Nash equlbrum for an ntal pror µ f v µ (δ,s) vµ (δ, s,s ) for all and s. Also, a strategy profle s an expost equlbrum f t s a Nash equlbrum for all µ. As Sekguch (1997) shows, under the full support assumpton (whch wll be stated n the next subsecton), the dfference between Nash and sequental equlbra s not essental. Indeed, gven an ntal pror µ, f a payoff v s acheved by some ex-post equlbrum s, there s 17

18 a sequental equlbrum s whch acheves the same payoff v. In what follows, we assume that the functon π has a full support: Defnton 1. The full support assumpton holds f π ω (y, ω a) > 0 for all ω, ω, a, and y. The full support assumpton requres that regardless of the current state ω and the current acton profle a, any sgnal y can be observed and any state ω can realze tomorrow. Under ths assumpton, any publc hstory h t can happen wth postve probablty. Also, snce any state can happen wth postve probablty, n any perod t > 1, a player s posteror belef about the state s always nteror,.e., she assgns at least probablty π on any state ω. The full support assumpton s mposed only for the sake of exposton. In Appendx D, we show that our result remans vald even f the full support assumpton s replaced wth a weaker condton. In partcular, we show that the folk theorem holds n the examples presented above. 2.2 Belef Convergence Theorem As noted n the ntroducton, snce actons are prvate nformaton, players belefs can possbly dverge n our model. In ths subsecton, we present the belef convergence theorem, whch shows that the current belef has only a neglgble mpact on the posteror belef after a suffcently long hstory h t. Ths result mples that f players play pure strateges and do not devate, ther prvate belefs wll eventually merge. Formally, gven a pure publc strategy s, a publc strategy s, and an ntal pror µ, let µ (h t µ,s) Ω denote player s belef about the state ω t+1 n perod t + 1 after the publc hstory h t. That s, µ (h t µ,s) s the posteror belef when no one devates from the strategy profle s. Ths belef s well-defned under the full support assumpton, because all publc hstores can appear wth postve probablty on the path. The followng s the belef convergence theorem; the proof reles on weak ergodcty of nhomogeneous Markov matrces, see Appendx B. Proposton 1. Suppose that the full support assumpton holds, and let β = 1 π Ω (0,1). Then for each, pure publc strategy s, publc strategy s, t 0, h t, 18

19 µ, and µ, we have µ (h t µ,s) µ (h t µ,s) β t. To nterpret ths result, pck a strategy profle s as stated, and pck an arbtrary publc hstory h t. In general, gven ths hstory h t, dfferent ntal prors µ and µ nduce dfferent posteror belefs, µ (h t µ,s) and µ (h t µ,s). However, the above proposton ensures that these two posteror belefs get closer as t ncreases, at a rate at least geometrc wth parameter β. So the mpact of player s current belef on her posteror belef n a dstant future s almost neglgble, as shown n Fgure 3 n Secton 1.1. Ths ensures that even f the opponents do not know player s current belef, after a long tme, they wll eventually obtan very precse nformaton about player s posteror belef. The result above reles on the assumpton that player chooses a pure strategy s and does not devate. Indeed, f s s a mxed strategy, player s belef n perod t crucally depends on her prvate nformaton about what actons are actually chosen, and hence the opponents cannot obtan precse nformaton about her posteror belef. Smlarly, f player devates to other strategy s s. then her posteror belef s µ (h t µ, s,s ), whch can be qute dfferent from the opponents expectaton µ (h t µ,s). In other words, player can always possess new prvate nformaton about the true state by devatng from a prescrbed strategy s. The belef convergence theorem does not ensure that two posteror belefs nduced by dfferent publc hstory h t and h t wll merge. That s, dfferent publc hstores may yeld qute dfferent belefs even after a long tme. In ths sense, the belef evoluton s path-dependent, and state learnng never ends. 3 Example: Stochastc Cournot Competton To llustrate the key deas of the paper, n ths secton, we consder a Cournot example and show how to construct an approxmately effcent equlbrum. 3.1 Model There are two frms, and each frm produces product. We consder dfferentated products, but these products are qute smlar; so the prces of the two products 19

20 are hghly correlated. 8 (For example, thnk about the prce of coffee beans from Brazl and the one from Kenya.) In each perod, each frm chooses quantty a. There are three possble values of a : H = 20 (hgh), M = 10 (mddle), or L = 0 (low). There s a persstent demand shock and an..d. demand shock. The persstent demand shock s represented by a hdden state ω, whch follows a Markov process. Specfcally, the state s ether a boom (ω = ω G ) or a slump (ω = ω B ), and after each perod, the state stays at the current state wth probablty 0.8. Actons (quanttes) do not nfluence the state evoluton. Let µ (0,1) be the probablty of ω G n perod one. Due to the..d. demand shock, the market prce s stochastcally dstrbuted, condtonal on the current economc condton ω and the quantty a = (a 1,a 2 ). Let y {0,10,20,30,40,50} denote the prce of product, and let y = (y 1,y 2 ). For each state ω and each quantty a, let πy ω ( a) denote the dstrbuton of the prce vector y over Y = {0,10,20,30,40,50} 2. We assume that both y 1 and y 2 are publcly observable. The precse specfcaton of π Y wll be gven n Appendx A, and here we lst only the key propertes of π Y : πy ω (y a) > 0 for each ω, a, and y, so the full support assumpton holds. The dstrbutons {(π ω Y (y a)) y Y } (ω,a) are lnearly ndependent. Ths mples that the frms can statstcally dstngush (ω,a) through y. The expected prce E[y ω,a] = y Y π ω Y (y a)y condtonal on the state ω = ω G s as n the left table below: For each cell, the frst component represents the expectaton of y 1, and the second s of y 2. Smlarly, the expected prce condtonal on the state ω = ω B s as n the rght table. L M H L 42, 42 41, 40 23, 22 M 40, 41 23, 23 16, 15 H 22, 23 15, 16 13, 13 L M H L 36, 36 35, 34 17, 16 M 34, 35 17, 17 10, 9 H 16, 17 9, 10 7, 7 8 Even when the frms produce homogeneous products, as n Green and Porter (1984), our folk theorem (Proposton 6) apples so that we can construct an approxmately effcent equlbrum. However, the equlbrum strategy becomes a bt more complcated n that case. The reason s that when the products are homogeneous, symmetrc acton profles never have parwse full rank, and thus we need to perturb the optmal polcy for the sgnal to be nformatve about the dentty of the devator. See pages of FLM for more detals. 20

21 Snce the two frms produce smlar products, the expected prces are (almost) determned by the total producton a 1 + a 2. For example, the three acton profles (H,L), (M,M), and (L,H) nduce the same producton level a 1 + a 2 = 20, and hence result n smlar expected prces, 22 or 23, at the good state ω G (the left table). Also, as one can see from the rght table, when the state changes to the bad state ω B, the expected prce drops by 6, compared to the case wth the good state ω G. For smplcty, we assume that the margnal cost of producton s C = 0 for each frm. Hence frm s actual proft s a y, and ts expected payoff gven ω and a s a E[y ω,a]. We normalze the payoff (subtract 200 and then dvde by 10) and denote t by g ω(a). That s, let gω = a E[y ω,a] These payoffs g ω (a) are summarzed as follows; the left table descrbes the payoffs for the good state ω G, and the rght table descrbes the ones for the bad state ω B. L M H L 20, 20 20, 20 20, 24 M 20, 20 3, 3 4, 10 H 24, 20 10, 4 6, 6 L M H L 20, 20 20, 14 20, 12 M 14, 20 3, 3 10, 2 H 12, 20 2, 10 6, 6 As one can see, (H,H) s a Nash equlbrum of the stage game, regardless of ω. Also, Always (H, H) s a sequental equlbrum n the nfnte-horzon game regardless of the ntal pror µ, snce the state transton does not depend on actons. The payoff of ths equlbrum s 1 2 (6,6)+ 1 2 ( 6, 6) = (0,0) n the lmt as δ 1, because the tme average of the hdden state ω s n the long run. In ths game, the effcent acton profle (.e., the acton profle whch maxmzes the total proft of the frms) s (H,H) at the state ω G, but s (M,M) at the state ω B. So n order to maxmze the total proft, the frms should produce less than the Nash equlbrum quantty when they are pessmstc about the current state of the economy. The next subsecton studes ths ssue n more detals. 3.2 Feasble Payoff Set and Optmal Polces Gven the ntal pror µ and the dscount factor δ, dfferent strategy profles s yeld dfferent payoffs n the nfnte-horzon game. The set of all such payoff vectors s the feasble payoff set n our envronment. That s, the feasble payoff 21

22 set gven the ntal pror µ and the dscount factor δ s defned as V µ (δ) = co{v ω (δ,s) s S}. The welfare-maxmzng pont n ths set V µ (δ) can be computed by dynamc programmng. To see ths, note that the welfare-maxmzng pont must be achevable by a pure strategy, as mxed strateges can acheve only a convex combnaton of pure-strategy payoffs. When the frms use a pure strategy profle and do not devate, they do not have prvate nformaton, so the posteror belef s common after each perod. Thus the maxmal welfare must be acheved by a pure Markovan strategy profle n whch the posteror belef µ t s a common state varable. Ths mples that the maxmal welfare must solve the followng Bellman equaton: Let f (µ) be the maxmal welfare gven the ntal pror µ, and let µ(y µ,a) be the posteror belef n perod two gven that the ntal pror s µ and the outcome n perod one s (a,y). Then the functon f must solve [ ] f (µ) = max (1 δ)(g µ a A 1 (a) + gµ 2 (a)) + δ π µ Y (y a) f ( µ(y µ,a)) y Y Intutvely, (1) asserts that the maxmal welfare f (µ) s the sum of today s proft g µ 1 (a) + gµ 2 (a) and the expectaton of the future profts f ( µ(y µ,a)). The current acton should maxmze ths sum, and hence we take the maxmum wth respect to a. (1) delta=0.95 delta=0.99 delta= Fgure 4: Value Functons for Hgh δ x-axs: belef µ. y-axs: payoffs. Fgure 5: Optmal Polcy x-axs: belef µ. y-axs: actons. 22

23 For each dscount factor δ (0,1), we can derve an approxmate soluton to (1) by value functon teraton wth a dscretzed belef space. Fgure 4 llustrates how the value functon f changes when the frms become more patent; t descrbes the value functons for δ = 0.95, δ = 0.99, and δ = As one can see, the value functon becomes almost flat as the dscount factor approaches one, that s, the frms ntal pror has almost no mpact on the effcent payoff. For δ close to one, the value functon (the maxmal welfare f (µ)) approxmates 0.70 regardless of the ntal pror µ. The optmal polcy for δ = 0.95 s descrbed n Fgure 5, where 1 n the vertcal axs means a = (M,M), and 0 means a = (H,H). It shows that the optmal polcy s a smple cut-off strategy, whch chooses (M,M) when the current belef µ s less than 1 2, and (H,H) otherwse.9 In what follows, let s eff (δ,µ) denote the optmal polcy gven δ and µ. That s, s eff (δ,µ) s the strategy for the nfnte-horzon game whch acheves the effcent payoff wthn the feasble payoff set, gven the dscount factor δ and the ntal pror µ. Wthout loss of generalty, we assume that the optmal polcy s eff (δ,µ) s a pure publc strategy profle. Usng a smlar technque, we can compute other extreme ponts of the feasble payoff set V µ (δ). For example, the hghest payoff for frm 1 wthn the feasble payoff set can be computed by solvng [ f (µ) = max a A (1 δ)g µ 1 (a) + δ π µ Y (y a) f ( µ(y µ,a)) y Y Agan, we can derve an approxmate soluton usng value functon teraton. It turns out that when δ s close to one, the value functon s almost flat and approxmates 18.2 regardless of the ntal pror µ. The optmal polcy s agan a cut-off strategy; t chooses (M,L) when µ 1 3, and (H,L) when µ > 3 2, regardless of δ. Let s 1 (δ,µ) denote the optmal polcy gven δ and µ. That s, s 1 (δ, µ) s the strategy for the nfnte-horzon game whch maxmzes frm 1 s payoff. Smlarly, let s 2 (δ, µ) denote the strategy whch maxmzes frm 2 s payoff. To summarze, when δ s close to one, the ntal pror does not nfluence the maxmal welfare or the hghest payoff for each frm. More generally, our 9 Note that ths optmal polcy s dentcal wth the myopc polcy, whch maxmzes the stagegame payoff each perod. Ths follows from the fact that n ths example, the dstrbuton of the belef tomorrow does not depend on the current acton profle a, as explaned n Appendx A. Our equlbrum constructon does not rely on ths property, and s vald n more general envronments. 23 ].

24 Proposton 2 shows that the feasble payoff set does not depend on the ntal pror µ, n the lmt as δ goes to one. Let V denote ths lmt feasble payoff set, and let V be the set of all payoffs n V whch Pareto-domnates the trval equlbrum payoff, (0, 0). Fgure 6 descrbes (a subset of) the lmt feasble payoff set. Here the proft maxmzng pont s (0.35, 0.35), because the value functon f approxmates 0.70 as δ goes to one. Also the down-rght corner s (18.2, 20) because the value functon f approxmates 18.2, and the optmal polcy s 1 (δ, µ) asks frm 2 to play L forever, whch yelds 20 each perod. Fgure 6 s only a subset of V, because we have not computed other extreme ponts of V. Fgure 7 descrbes (a subset of) the feasble and ndvdually ratonal payoff set V. In what follows, we wll construct an equlbrum whch approxmates the effcent payoff vector (0.35,0.35). ( 20, 18.2) (0.35, 0.35) (0.35, 0.35) (18.2, 20) Fgure 6: Subset of V Fgure 7: Subset of V 3.3 Pseudo-Ergodc Strateges As explaned, when the frms have a common belef µ, the effcent payoff vector (0.35,0.35) s (approxmately) acheved f the frms coordnate and play the optmal polcy s eff (δ,µ). However, n our model, the frms may not have a common belef, whch causes a possble mscoordnaton problem n the followng sense: Suppose that each frm s current belef s µ where µ 1 µ 2, and that these belefs are prvate nformaton. If each frm chooses the optmal polcy correspondng to ts own belef, then the resultng profle (s eff 1 (δ, µ 1),s eff 2 (δ,µ 2)) s qute dfferent from the optmal polcy. 24

25 To avod ths mscoordnaton problem, n our equlbrum, the frms play pseudo-ergodc strateges whch do not depend on the prvate belefs. For example, f the frms want to cooperate and approxmate (0.35,0.35) from now on, they form a dummy publc belef µ = 1 2 and play the correspondng optmal polcy s eff (δ, 1 2 ). By the defnton, ths strategy seff (δ, 1 2 ) s not optmal unless the frms current belefs are µ 1 = µ 2 = 1 2. However, as Proposton 3 shows, t s approxmately optmal for all ntal prors µ, that s, t approxmates the effcent payoff (0.35,0.35) regardless of the true belef. So f the frms want to cooperate, they may gnore ther prvate belefs and smply play s eff (δ, 1 2 ), the optmal polcy for the dummy publc belef µ = 1 2. Proposton 3 also shows that the same result holds for other optmal polces. For example, the optmal polcy s 1 (δ, 1 2 ), whch acheves the best payoff for frm 1 gven the dummy belef µ = 1 2, approxmates the down-rght corner (18.2, 20) of the feasble payoff set regardless of the ntal pror µ. So the frms may play t f they want to reward frm 1 (by gvng 18.2) whle punshng frm 2 (by gvng 20). Smlarly, the optmal polcy s 2 (δ, 1 2 ) for the dummy belef 1 2 approxmates ( 20,18.2) regardless of the ntal pror µ. Ths strategy can be used when the frms want to reward frm 2 whle punshng frm 1. Also, any constant acton profle s a pseudo-ergodc strategy n that t acheves approxmately the same payoff regardless of the ntal belef. For example, f the frms always play (H,H), the payoff (0,0) s acheved n the lmt as δ 1, regardless of the ntal belef. As wll be explaned, the frms use ths Always (H,H) when they want to punsh each other. 3.4 Random Blocks and Self-Generaton Now we construct an equlbrum approxmatng the effcent payoff (0.35,0.35), by assemblng the pseudo-ergodc strateges n the prevous subsecton. In what follows, we assume that publc randomzaton z, whch follows the unform dstrbuton on [0,1], s avalable at the end of each perod. As n Yamamoto (2016), we regard the nfnte horzon as a sequence of random blocks, the length of whch s determned by publc randomzaton. Specfcally, at the end of each perod t, the frms check the publc randomzaton z t. If z t p for some fxed number p [0,1], then the current random block termnates 25