A Reinforcement Procedure Leading to Correlated Equilibrium

Transcription

1 A Renforcemen Procedure Leadng o Correlaed Equlbrum Sergu Har and Andreu Mas-Colell 2 Cener for Raonaly and Ineracve Decson Theory; Deparmen of Mahemacs; and Deparmen of Economcs, The Hebrew Unversy of Jerusalem, Feldman Buldng, Gva-Ram, 9904 Jerusalem, Israel 2 Deparmen of Economcs and Busness and CREI, Unversa Pompeu Fabra, Ramon Tras Fargas 25-27, Barcelona, Span Absrac. We consder repeaed games where a any perod each player knows only hs se of acons and he sream of payoffs ha he has receved n he pas. He knows neher hs own payoff funcon, nor he characerscs of he oher players how many here are, her sraeges and payoffs. In hs conex, we presen an adapve procedure for play called modfed-regre-machng whch s nerpreable as a smulus-response or renforcemen procedure, and whch has he propery ha any lm pon of he emprcal dsrbuon of play s a correlaed equlbrum of he sage game. Inroducon Werner Hldenbrand has repeaedly emphaszed he usefulness, concepual and echncal, of carryng ou equlbrum analyss by means of dsrbuonal noons. For socal suaons modelled as N-person games, he concep of correlaed equlbrum, an exenson of Nash equlbrum nroduced by Aumann, can be vewed n a mos naural way as mposng equlbrum condons on he dsrbuon of acon combnaons of he dfferen players. Correlaed equlbrum wll be he subjec of he curren paper, and he syle of our analyss should no surprse anybody famlar wh Hldenbrand s research. Ths paper connues he sudy of Har and Mas-Colell [2000], where he noon of correlaed equlbrum and ha of approachably by means of smple rules of humb or adapve procedures of play were lnked. We showed here ha f a game s repeaedly played and he players deermne her sage probables of play accordng o a procedure called regre-machng, hen he emprcal dsrbuon of play converges wh probably one o he se Dedcaed wh grea admraon o Werner Hldenbrand on hs 65h brhday. Prevous versons of hese resuls were ncluded n he Cener for Raonaly Dscusson Papers #26 December 996 and #66 March 998. We hank Dean Foser for suggesng he use of modfed regres. The research s parally suppored by grans of he Israel Academy of Scences and Humanes; he Spansh Mnsry of Educaon; he Generala de Caalunya; CREI; and he EU-TMR Research Nework.

2 82 Sergu Har and Andreu Mas-Colell of correlaed equlbra of he gven game. Regre-machng means ha f a perod player has aken acon j, hen he probably of playng anoher acon k a me + wll be proporonal o he regre for no havng played k nsead of j, whch s defned smply as he ncrease, f any, n he average payoff ha would resul f all pas plays of acon j were replaced by acon k and everyhng else remaned unalered. The mplemenaon of regre-machng by a player requres ha player o observe he acons aken by all players n he pas. Bu ha s no all. The player should also be able o carry ou he hough expermen of compung wha hs payoffs would have been, had hs acons n he pas been dfferen from wha hey really were. For hs he needs o know wha game he s playng or, a he very leas, hs own payoff funcon. In hs paper we wll show ha hs level of sophscaon and knowledge s no really necessary o oban correlaed equlbra. We shall modfy he regre-machng procedure n such a way ha he play probables are deermned from he acual realzaons only. Specfcally, each player only needs o know he payoffs he receved n pas perods. He need no know he game he s playng neher hs own payoff funcon nor he ohers players payoffs; n fac, he may well be oblvous of he fac ha here s a game a all. The procedure ha we shall examne s a smulus-response or renforcemen procedure, n he sense ha a relavely hgh payoff a perod wll end o ncrease he probably of playng a perod + he same acon ha was played a. In Secon 2 we presen he basc model and sae our resuls. Secon 3 conans some furher dscussons, and he proofs are gven n Secon 4. To summarze: We consder repeaed games where each player knows only hs se of avalable choces bu nohng abou he oher players, and observes only hs own acually realzed payoffs. We exhb smple sraeges of play whereby he long-run frequences of he varous acon combnaons beng played are anamoun o a correlaed equlbrum. Of course, hs sophscaed macro -pcure ha emerges canno be seen a he ndvdual level. Ths s a concluson, we rus, wh whch Werner Hldenbrand can ruly sympahze. 2 Model and resuls We consder fne N-person games n sraegc or normal form. The se of players s a fne se N, he acon se of each player N s a fne se S, and he payoff funcon of s u : S R, where 2 S := l N Sl. We wll denoe hs game N,S N, u N by Γ. To avod confuson, we wll use he erm acon for he one-sho game, and sraegy for he mul-sage game. 2 R denoes he real lne.

3 Renforcemen Procedure Leadng o Correlaed Equlbrum 83 A correlaed equlbrum a concep nroduced by Aumann [974] s nohng oher han a Nash equlbrum where each player may receve a prvae sgnal before he game s played he sgnals do no affec he payoffs; and he players may base her choces on he sgnals receved. Ths may be equvalenly descrbed as follows: Assume ha he sgnal of each player s n fac a play recommendaon s S, where he N-uple s = s N s seleced by a devce or medaor accordng o a commonly known probably dsrbuon. A correlaed equlbrum resuls f each player realzes ha he bes he can do s o follow he recommendaon, provded ha all he oher players do lkewse. Equvalenly, a probably dsrbuon ψ on he se S of acon N-uples.e., ψ s 0 for all s S and s S ψ s = s a correlaed equlbrum of he game Γ f, for every player N and every wo acons j, k S of, we have ψ s u k, s u s 0, s S:s =j where s S := l Sl denoes he acon combnaon of all players excep hus s = s,s. The nequaly afer dvdng he expresson here by s S:s =j ψ s means ha when he recommendaon o player s o choose acon j, hen choosng k nsead of j canno yeld a hgher expeced payoff o. Fnally, a correlaed ε-equlbrum obans when 0 s replaced by ε on he rgh-hand sde of. Now assume ha he game Γ s played repeaedly n dscree me =, 2,... Denoe by s S he realzed choce of player a me, and pu s =s N S. The payoff of player n perod s denoed U := u s. The basc seup of hs paper assumes ha he nformaon of each player consss jus of hs se of avalable choces S. He does no know he game Γ ; n fac, he does no know how many players here are besdes hmself, 3 wha her choces are and wha he payoff funcons hs own as well as he ohers are. The only hng player observes s hs own realzed payoffs and acons. Tha s, n deermnng hs play probables a me +, player s nformaon consss of hs own realzed payoffs U,U 2,..., U n he prevous perods, as well as hs acual acons s,s 2,..., s here. In Har and Mas-Colell [2000] henceforh [HM] we nroduced regremachng procedures. These are smple adapve sraeges where he play probables are proporonal o he regres for no havng played oher acons. Specfcally, for any wo dsnc acons j k n S and every me, he regre of a from j o k s R j, k := [ D j, k ] + max { D j, k, 0 }, 2 3 Perhaps none: could be a game agans Naure.

4 84 Sergu Har and Andreu Mas-Colell where D j, k := τ :s τ =j u k, s τ u s τ. 3 Ths s he change n he average payoff of ha would have resuled f he had played acon k every me n he pas ha he acually chose j. Rewrng hs as D j, k = u k, s τ u s τ τ :s τ =j τ :s τ =j shows ha player can compue he second erm: s jus / τ :s τ =j U τ. Bu canno compue he frs erm, snce he knows neher hs own payoff funcon u nor he choces s τ of he oher players. Therefore we replace he frs erm by an esmae ha can be compued on he bass of he avalable nformaon; namely, we defne C j, k := p τ :s τ =k τ j p τ k U τ Uτ, 4 τ :s τ =j where p τ denoes he play probables a me hus p τ j s he probably ha chose j a perod τ; hese probables p τ are defned below hey wll ndeed depend only on U,..., U. As n 2, he modfed regres are hen Q j, k := [ C j, k ] + max { C j, k, 0 }. 5 In words, he modfed regre for no havng used k nsead of j measures he dfference srcly speakng, s posve par of he average payoff over he perods when k was used and he perods when j was used. In addon, he payoffs of hese perods are normalzed n a manner ha, nuvely speakng, makes he lengh of he respecve perods comparable. 4 Nex we defne he play probables, based on hese modfed regres. Recall he regre-machng procedure of [HM]: If j := s s he acon chosen by a me, hen he probably of swchng a me + o anoher acon k j s proporonal wh a fxed facor /µ o he regre from j o k; wh he remanng probably, he same acon j s chosen agan. Here we shall make wo changes relave o [HM]. Frs, we need o guaranee ha he sum of he proposed probables does no exceed one; mulplyng he modfed regres by a facor /µ whch we sll do does no suffce 4 For nsance, f he probably of k beng chosen s always wce ha of j, hen n he long run k wll be played wce as ofen as j. So, n order o compare he sum of he payoffs over hose perods when j s played wh he parallel sum when k s played, one needs o mulply he k-sum by /2. When he probables change over me, Formula 4 uses he correc normalzaon: he dfference beween he regres and he modfed regres has condonal expecaon equal o 0 see Sep 0 and he Proof of Sep n Secon 4.

5 Renforcemen Procedure Leadng o Correlaed Equlbrum 85 because he modfed regres are no bounded. 5 Second, we need every acon o be played wh some mnmal frequency. 6 Thus we shall requre ha he ransons remble over every acon. Le herefore µ>0beasuffcenly large number; as n [HM], suffces o ake µ so ha µ>2m m for all, where m := S s 7 he number of acons of, and M s an upper bound on u s for all s S. Le 0 <δ<, and defne he play probables of player a me +by p + k := δ mn p + j := k S :k j { µ Q j, k, p + k, m } + δ, for k j; and m 6 where j := s s he choce of n he prevous perod. As for he frs perod, he play probables a = are gven by an arbrary srcly posve vecor p S ; for smplcy, assume ha p j δ/m for all j n S. Formula 6 says ha p + s a weghed average of wo probably vecors noe ha p + j = δ k j mn { Q j, k /µ, / m } + δ/m. The frs, wh wegh δ, s gven by he modfed regres n a manner smlar o [HM, Formula 2.2] noe ha akng he mnmum wh / m guaranees ha he resulng sum does no exceed. The second erm, wh wegh δ, s jus he unform dsrbuon over S : each k S has equal probably /m. Ths unform remble guaranees ha all probables a me + are a leas δ/m > 0. Pung 4, 5 and 6 ogeher mples ha p + depends only on he prevously realzed payoffs U,..., U and play probables p,..., p. Therefore, by recurson, p + does ndeed depend only on U,..., U. Le 8 z S be he emprcal dsrbuon of play up o ; ha s, for every s S, z s := {τ : s τ = s} s he relave frequency ha he N-uple of acons s has been played n he frs sages. 5 Noce ha here s no a pror lower bound on he probables appearng n he denomnaors. 6 Roughly speakng, one needs hs exogenous sascal nose o be able o esmae he conngen payoffs usng only he realzed payoffs recall he frs erm n he modfed regre. In fac, hs s que delcae, snce he acons of he oher players are never observed. 7 We wre A for he number of elemens of a fne se A. 8 We wre A := { p R A + : a A pa =} for he se of probably dsrbuons over he fne se A.

6 86 Sergu Har and Andreu Mas-Colell Our frs resul s: Theorem. For every ε>0 here s a δ 0 δ 0 ε > 0 such ha for all δ<δ 0, f every player plays accordng o adapve procedure 6, hen he emprcal dsrbuons of play z converge almos surely as o he se of correlaed ε-equlbra of he game Γ. Thus: for almos every rajecory, z s close o a correlaed ε-equlbrum for all large enough ; equvalenly, z s a correlaed 2ε-equlbrum from some me on hs me may depend on he rajecory; he heorem saes ha, wh probably one, s fne. As n he Man Theorem of [HM], noe ha he convergence s no o a pon, bu o a se; ha s, he dsance o he se goes o 0. Fnally, we remark ha boh µ and δ can be aken o be dfferen for dfferen players. To oban n he lm correlaed equlbra raher han correlaed approxmae equlbra we need δ o decrease o 0 as ncreases. A smple way o do hs s o replace δ n 6 by δ/ γ, where γ s a number srcly beween 0 and 9 /4. Thus, we now defne p + k := p + j := δ { } γ mn µ Q j, k, m + δ γ m, for k j; and 7 k S :k j p + k, where, agan, j := s s he choce of n perod. A =, le p be an arbrary vecor n S wh p j δ/m for all j n S. The second resul s: Theorem 2. If every player plays accordng o adapve procedure 7, hen he emprcal dsrbuons of play z converge almos surely as o he se of correlaed equlbra of he game Γ. We have chosen he parcular ype of sequence δ := δ/ γ for smplcy and convenence only. Wha maers s, frs, ha δ converge o 0, and second, ha do so suffcenly slowly oherwse he modfed regres C may become oo large; recall ha he probables p, whch are bounded from below by δ /m, appear n he denomnaors. Ths explans he need for an upper bound on γ urns ou ha γ</4 suffces; see he proof n Secon 4. Moreover, we noe ha one may well ake dfferen sequences δ for dfferen players cf. he Remark a he end of Secon 4. 9 The reason for hs resrcon wll be explaned below.

7 3 Dscusson Renforcemen Procedure Leadng o Correlaed Equlbrum 87 a Renforcemen and smulus-response The play procedures of hs paper can be nerpreed as renforcemen or smulus-response procedures see, for example, Bush and Moseller [955], Roh and Erev [995], Borgers and Sarn [995], Erev and Roh [998], Fudenberg and Levne [998, Secon 4.8]. Indeed, he behavor of our players s very far from he deal of a sophscaed decson-maker ha makes opmal decsons gven hs more or less well-formed belefs abou he envronmen. The behavor we posulae s, n fac, much closer o he model of a reflexorened ndvdual ha, from a very lmed concepon of he world n whch he lves, smply reacs o smul by renforcng hose behavors wh pleasurable consequences. In order o be specfc and o llusrae he above pon furher, le us assume ha he payoffs are posve and focus on he lm case of our procedure where probables are chosen n exac proporon o he modfed regres 5. Suppose ha player has played acon j a perod. We hen have for every k j: f Q j, k > 0 hen Q +j, k <Q j, k; and f Q j, k = 0 hen Q +j, k =0. Thus all he modfed regres decrease or say null from o +. Hence he probably of choosng j a + ges renforced.e., ncreases relave o he same probably a by he occurrence of j a, whle he probables of he oher acons k j decrease. Moreover, as can easly be seen from Defnons 4 and 5, he hgher he payoff obaned a me when j was played, he greaer hs renforcemen. Fnally, all hese effecs decrease wh me snce we average over n 4. b Relaed work There s by now a subsanve body of work on no fully opmal behavor n repeaed games see for nsance he book of Fudenberg and Levne [998]. In parcular, sraeges ha lead o he se of correlaed equlbra have been proposed by Foser and Vohra [997] and by Fudenberg and Levne [999]. There s also an older radon, begnnng wh Hannan [957], ha focuses on anoher form of regres of an uncondonal knd see c below and on sraeges ha asympocally ake hem down o zero. Clearly, our work here and n Har and Mas-Colell [2000, 200] belongs o hese lnes of research. Snce he man dfference beween [HM] and he curren paper s ha here he players do no know he game, we wan o pon ou ha hs unknown game case has already been suded. Specfcally, n he conex of Hannan regres, see Foser and Vohra [993, 998], Auer e al. [995] and Fudenberg and Levne [998, Secon 4.8] and also Baños [968] and Megddo [980] for relaed work.

8 88 Sergu Har and Andreu Mas-Colell c Hannan-conssency We say ha a sraegy of a player s Hannan-conssen f guaranees ha hs long-run average payoff s as large as he hghes payoff ha can be obaned by playng a consan acon; ha s, s no less han he onesho bes-reply payoff agans he emprcal dsrbuon of play of he oher players. Formally, he Hannan regre of player a me for acon k S s defned as DH k := τ= u k, s τ τ= Uτ = u k, z Uτ, where z S s he emprcal dsrbuon of he acons chosen by he oher players n he pas. 0, A sraegy of player s hen called Hannanconssen f, as ncreases, all Hannan-regres are guaraneed o become almos surely non-posve n he lm, no maer wha he oher players do; ha s, wh probably one, lm sup DHk 0 for all k S. The reader s referred o Har and Mas-Colell [200] for dealed dscussons and resuls. In he seup of he curren paper, he modfed-regre-machng approach leads o a smple renforcemen sraegy ha s Hannan-conssen recall [HM, Theorem B]: For every k S defne CHk := p τ :s τ =k τ k U τ Uτ, τ= τ= and p +k := δ [ CH k ] + [ j S CH j ] + δ m. 8 + Here δ = δ/ γ for some δ>0 and 0 <γ</2; we ake p + S obe arbrary for = 0 and whenever he denomnaor vanshes. We have Theorem 3. The sraegy 8 s Hannan-conssen. The proof of hs heorem s parallel o, and smpler han, he proof of Theorem 2 below, and herefore omed. 0 I.e., z s := {τ : sτ = s } / for each s S. Noe ha DH k = j k D j, k; we can hus refer o DHk as he uncondonal regre for k, and o D j, k as he regre for k, condonal on j.

9 4 Proof Renforcemen Procedure Leadng o Correlaed Equlbrum 89 In hs secon we wll prove Theorems and 2 of Secon 2 ogeher. Le δ := δ γ where δ>0 and 0 γ</4. For Theorem ake γ = 0, and for Theorem 2, γ>0. We nroduce some noaons, n addon o hose of he prevous secons. Fx player n N; for smplcy, we drop reference o he ndex whenever hs does no creae confuson hus we wre C nsead of C, and so on. Recall ha m := S s he number of sraeges of, and M s an upper bound on he payoffs: M u s for all s S. Denoe L := { j, k S S : j k } ; hen R L s he m m Eucldean space wh coordnaes ndexed by L. For each =, 2,... and each j, k nl, denoe 2 Thus, we have Z j, k := p j p k {s =k} {s =j}; B j, k :=Z j, k u s ; A j, k := {s =j} u k, s u s. C j, k = B τ j, k and τ D j, k = A τ j, k. τ We shall wre B for he vecor B j, k j,k L R L ; and smlarly for he oher vecors A,C, and so on. Nex, defne { } δ mn µ C+ j, k, m + δ m, f k j, Π j, k := δ { } mn µ C+ j, k, m + δ m, f k = j. k j Noe ha Π j, S for all j S ;husπ s a ranson probably marx on S. Boh procedures 6 and 7 sasfy p + k =Π s,k for all k where, agan, γ = 0 corresponds o 6 and γ>0 o 7. Le ρ := ds C, R L 2 C + j, k 2 2 We wre E for he ndcaor of he even E.e., E =fe occurs and = 0 oherwse. j k

10 90 Sergu Har and Andreu Mas-Colell be he squared Eucldean dsance n R L of he vecor C from he nonposve orhan R L. We wll use he sandard O noaon: For wo real-valued funcons f and g defned on a doman X, weake f x =O g x o mean ha here exss a consan K< such ha fx Kg x for all x n X. From now on,, v, w wll denoe posve negers; h =s τ τ wll be hsores of lengh ; j, k, and s wll be sraeges of.e., elemens of S ; s and s wll be elemens of S and S, respecvely. Unless saed oherwse, all saemens should be undersood o hold for all, v, h,j,k, ec. ; where hsores h are concerned, only hose ha occur wh posve probably are consdered. Fnally, P sands for Probably, E for Expecaon and Var for Varance. The proof of Theorems and 2 wll be dvded no 4 seps. Sep 4 shows ha he regres are small n he lm; he Proposon of Secon 3 of [HM] hen mples ha he emprcal dsrbuons are correlaed approxmae equlbra. We noe ha Seps hold for any non-ncreasng sequence δ > 0, whereas Seps 2 4 make use of he specal form δ = δ/ γ. A gude o he proof follows he saemen of he seps. Sep : ] E [ + v 2 ρ +v h 2 ρ +2 v w= + v 2 ρ +v 2 v + v 2 ρ = O. Defne 3 β +w j := k j δ 2 +v C + E [B +w h ]+O µ C+ k, j p +w k µ C+ j, k p +w j. k j Sep 2: C + E [B +w h ]=µe u j, s +w β+w j h. j S v 2 δ 2 +v. Sep 3: v C +v j, k C j, k =O. δ +v v Π +v j, k Π j, k =O +δ δ +v. δ +v 3 Noe ha β +w j s measurable wh respec o h +w acually, depends only on h +w and s +w, bu no on s +w.

11 Renforcemen Procedure Leadng o Correlaed Equlbrum 9 Defne Z j, k := Z τ j, k ; and τ= Y := Z j, k. j,k L Sep 4: Π j, k µ C+ j, k =O δ + Y for all j k. C + Sep 5: 4 E [B +w h ]=µe u j, s s +w Π 2 Π +w,j h j S +O δ + Y + w δ +w For each >0 and each hsory h, we defne an auxlary sochasc process ŝ +w w=0,,2,... wh values n S as follows: The nal sae s ŝ = s, and he ranson probables are 5. P [ŝ +w = s ŝ,..., ŝ +w ]:= Π l ŝl +w,s l. l N Tha s, he ŝ-process s saonary: I uses he ranson probables of perod a each perod +,+2,..., + w,... Sep 6: w 2 P [s +w = s h ] P [ŝ +w = s h ]=O + w δ δ +w. δ +w C + Sep 7: E [B +w h ]=µe u j, ŝ +w Π 2 ŝ Π +w,j h j S +O δ + Y + w2 + w δ δ +w. δ +w 4 Π 2 s he second power of he marx Π.e., Π Π, and Π 2 Π k, j s he k, j-elemen of he marx Π 2 Π. 5 We wre Π l for he ranson probably marx of player l hus Π s Π.

12 92 Sergu Har and Andreu Mas-Colell Sep 8: E u j, ŝ +w Π 2 ŝ Π +w,j h j S = s S P [ŝ +w = s ] h Π w+ Π w s,j = O w Sep 9: ] E [ + v 2 ρ +v h 2 ρ + O vδ + vy + v3 + v 2 δ δ +v + v + δ +v Sep 0: Sep : E [Z j, k h ]=0. Var [Z j, k] = O δ. lm Z j, k = 0 a.s. lm Y = 0 a.s. lm C j, k D j, k = 0 a.s. v2 δ 2 +v Le ξ sasfy 6 { } 2 <ξ<mn +γ, ; 9 4γ such a ξ exss snce 0 γ</4. For each n =, 2,..., le n := n ξ be he larges neger no exceedng n ξ. Sep 2: There exss η<2ξ such ha ] E [ 2 n+ρ n+ h n 2 nρ n + O δn 2ξ ξγ + Y n n 2ξ + n η. Sep 3: 6 When γ =0, 9 s <ξ<2. If γ = 0 hen lm sup ρ n = O δ a.s. n If γ>0hen lm ρ n n = 0 a.s..

13 Renforcemen Procedure Leadng o Correlaed Equlbrum 93 Sep 4: If γ = 0 hen lm sup If γ>0hen R j, k =O δ lm R j, k = 0 a.s. We now provde a shor nuve overvew of he seps of he proof. The proof s based on he Proof of he Man Theorem of [HM] see Seps M M n he Appendx here whch n urn s nspred by he Approachably Theorem of Blackwell [956] wh a number of addonal seps needed o ake care of he modfcaons. Mos of he proof s devoed o showng ha he modfed regres Q C + are small. From hs one readly ges n Sep 4 ha he acual regres R D + are also small, snce he dfference C D s a marngale convergng almos surely o 0 see Sep. The man seps n he proof are as follows: We sar wh he basc recurson equaon n Sep smlar o [HM, Sep M]. Nex, we esmae he mddle erm on he rgh-hand sde of by approxmang he s-process wh he ŝ-process, whch s ndependen across players Seps 2 7; parallel o [HM, Seps M2 M6]. Ths leads o a formula smlar o [HM, Formula 3.4], excep ha he nvaran dsrbuon q λ n [HM, 3.4b] s replaced here by he ransons afer w and w + perods, whch are close o one anoher Sep 8; compare wh [HM, Sep M7]. Fnally, we oban he recurson formula n Sep 9. On comparng hs formula wh he parallel one n [HM, Sep M8], we see ha n Sep 9 here are addonal erms; one of hem, whch comes abou because he modfed regres are no bounded, conans a random varable Y. Sep 0, Sep and he Proof of Sep 3 show ha hs erm also goes o zero. Seps 2 and 3 complee he proof for he modfed regres, n a manner ha s smlar o [HM, Seps M9 M] hough more complcaed, parly because of he Y erms. As we have ndcaed above, Sep 4 yelds he needed resul for he acual regres. We now proceed o he proofs of Seps 4. Proof of Sep : Because C R L we have ρ +v C +w C 2 = + v C + 2 v B +w C + v w= 2 = + v 2 C C 2 2 v + + v 2 B+w C C C w= 2 + v2 v + v 2 B +w C v w= 2 + v 2 ρ 2 v + + v 2 B +w C + + v2 + v 2 m m 4M 2 m 2. w= a.s. δ 2 +v

14 94 Sergu Har and Andreu Mas-Colell Indeed: For he second erm, noe ha C C C = C C + =0. As for he hrd erm, we have u s M and Z +w j, k m/δ +w m/δ +v for w v snce he sequence δ s non-ncreasng; herefore B +w j, k and C j, k are each bounded by 2Mm/δ +v. Ths yelds. To ge, ake he condonal expecaon gven h hus ρ and C are fxed. Proof of Sep 2: Noe ha B +w j, k vanshes excep when s +w = j, k. We condon on h +w and s +w.e., on he whole hsory h +w excep player s choce a me + w: E [ B +w j, k h +w,s +w = s ] Hence = p +w k p +w j p +w k u k, s p +w j u j, s = p +w j u k, s u j, s C + E [ B +w h +w,s +w = s ] = C + j, k p +w j u k, s u j, s j k j = u j, s C + k, j p +w k C + j, k p +w j j k j k j we have colleced ogeher all erms conanng u j, s. Condonng now on h yelds he resul. Proof of Sep 3: We have + v C +v j, k C j, k v B +w j, k C j, k. Snce boh B +w j, k and C j, k are O /δ +v, we ge C +v j, k C j, k =O v/ δ +v. For j k, he dfference beween Π +v j, k and Π j, k s herefore a mos δ O v/ δ +v +δ δ +v / m. For j = k, s a mos m mes hs amoun. Proof of Sep 4: We dsngush wo cases. Frs, when /µ C + j, k / m, we have Π j, k µ C+ j, k =δ m µ C+ j, k, w= and hs s O δ les beween 0 and δ /m.

15 Renforcemen Procedure Leadng o Correlaed Equlbrum 95 Second, when /µ C + j, k / m, we have Π j, k = δ m + δ m m µ C+ j, k. For he oppose nequaly, noe ha Z τ j, k 2+Z τ j, k snce he only possble negave value of Z τ j, k s, hus µ C+ j, k µ C j, k Z τ j, k u s τ µ τ= 2M µ + M µ Z j, k < m + M µ Z j, k = Π j, k+ recall ha µ>2m m. δ m m + M µ Z j, k Proof of Sep 5: Denoe s +w by r; hen β +w j = k j µ C+ k, j Π +w r, k µ C+ j, k Π +w r, j. k j Also, Π 2 Π r, j =Π 2 r, j Π r, j = k S Π k, j Π r, k Π r, j k S Π j, k = Π k, j Π r, k Π j, k Π r, j k j k j we have subraced he j-erm from boh sums. Comparng he las expresson wh β +w j, we see ha Π 2 Π r, j s obaned by replacng each C + /µ and each Π +w n β +w j byπ.thus β +w j Π 2 Π r, j = µ C+ k, j Π k, j Π +w r, k k j + Π k, jπ +w r, k Π r, k k j µ C+ j, k Π j, k k j Π +w r, j Π j, kπ +w r, j Π r, j k j = O δ + Y + w δ +w,

16 96 Sergu Har and Andreu Mas-Colell by he esmaes of Seps 3 and 4. I only remans o subsue hs no he formula of Sep 2. Proof of Sep 6: We use he Lemma of [HM, Sep M4], whch mples ha f he -sep ranson probables of wo Markov processes on S dffer by a mos β, hen he w-sep ranson probables dffer by a mos S wβ. Applyng hs o he ŝ- and s-processes, wh β gven by Sep 3, yelds he resul. Proof of Sep 7: Replacng s +w w by ŝ +w w n he formula of Sep 5 gves an addonal error ha s esmaed n Sep 6 noe ha he wo processes s +w w and ŝ +w w sar from he same hsory h. Proof of Sep 8: Gven h, he random varables ŝ +w and ŝ +w are ndependen, snce he ranson probables of he ŝ-process are all deermned a me, and he players randomze ndependenly. Hence: E u j, ŝ +w Π 2 ŝ Π +w,j h j S = [ŝ P +w = s ] [ŝ ] h P +w = r h Π 2 Π r, j s S r S = P [ ŝ +w = s ] h Π w s,r Π 2 Π r, j s = s r P [ ŝ +w = s h ] Π w+ Π w s,j. The esmae of O / w s obaned by he Lemma of [HM, Sep M7]. 7 Proof of Sep 9: Pung ogeher he esmaes of Seps 7 and 8 and recallng ha v w= wλ = O v λ+ for λ yelds 2 v w= C + E [B +w h ]=O vδ + vy + v3 + v 2 δ δ +v + v. δ +v Recallng he formula of Sep complees he proof. 7 Whch s based on a Cenral Lm Theorem esmae. Noe ha here unlke he Man Theorem of [HM] Π s a srcly posve sochasc marx: all s enres are δ /m > 0. I can hen be shown ha Π w+ k, j Π w k, j δ /m w. Ths alernave esmae can be used nsead of O w /2 bu we hen need γ</5 raher han γ</4.

17 Renforcemen Procedure Leadng o Correlaed Equlbrum 97 Proof of Sep 0: Par follows mmedaely from he defnon of Z j, k : E [Z j, k h ]= p j p k p k p j =0. Therefore E [Z j, k] = 0, and Var [Z j, k] = E [ Z 2 j, k ] = E [ E [ Z 2 ]] j, k h [ p = E j ] 2 [ ] p k 2 p k+ 2 p j E p m, k δ whch gves. Proof of Sep : We wll use he followng Srong Law of Large Numbers for Dependen Random Varables; see Loève [978, Theorem 32..E]: Theorem 4. Le X n n=,2,... be a sequence of random varables and b n n=,2,... a sequence of real numbers ncreasng o, such ha he seres n= Var X n /b 2 n converges.then lm n b n ν= In our case, we have by Sep 0 = n X ν E [X ν X,..., X ν ] = 0 a.s. 2 Var [Z j, k] = m 2 δ = Ths seres converges, snce γ < /4 <. Therefore = m δ 2 γ. Z τ j, k E [Z τ j, k Z j, k,..., Z τ j, k] 0 a.s. τ and hus, by Sep 0, Z j, k 0 a.s. Ths yelds and. To ge, noe ha {s =k} u s = {s =k} u k, s, so B j, k A j, k p = j p k {s =k} {s =j} u s {s =j} u k, s u s p = j p k {s =k} {s =j} u k, s = Z j, k u k, s.

18 98 Sergu Har and Andreu Mas-Colell Bu s and s are ndependen gven h, herefore E [B j, k A j, k h ]=E [Z j, k h ] E [ u k, s snce he frs erm s 0 by Sep 0. Moreover, [ Var [B j, k A j, k] = E Z 2 j, k u k, s 2 ] M 2 E [ Z 2 j, k ] = O. δ h ] =0, I follows ha he seres Var [B j, k A j, k] / 2 converges, mplyng ha 8 C j, k D j, k =/ τ B τ j, k A τ j, k 0 a.s. argumen as n he proof of above. Proof of Sep 2: Apply he nequaly of Sep 9 wh = n = n ξ and v = n+ n. Then: v = O n ξ ; δ δn ξγ ; δ +v δ n + ξγ = O n ξγ ; and 9 δ δ +v = O n ξγ. Therefore ] E [ 2 n+ρ n+ h n 2 nρ n + O δn 2ξ ξγ + Y n n 2ξ +O n 3ξ+ξγ 3 + n 3ξ ξγ 3 + n 3ξ /2 + n 2ξ+2ξγ 2. To complee he proof, noe ha he defnon 9 of ξ mples ha: 3ξ ξγ 3 3ξ + ξγ 3 < 2ξ snce ξ<2/ + γ; 3ξ /2 < 2ξ snce ξ>; and 2ξ +2ξγ 2 < 2ξ snce ξ</ 4γ / 2γ. Therefore we ake η := max {3ξ + ξγ 3, 3ξ /2, 2ξ +2ξγ 2} < 2ξ. Proof of Sep 3: Le b n := 2 n n 2ξ and X n := b n ρ n b n ρ n = n 2 nρ n 2 n ρ n. By Sep we have X n = O v n + vn 2 /δ 2 n+ = O n 2ξ+2ξγ ; hus n Var X n /b 2 n = n O n 4ξ+4ξγ 2 /n 4ξ = n O n 4ξγ 2 converges snce 4ξγ < by he choce of ξ. Nex, consder /b n ν n E [X ν X,..., X ν ]. The nequaly of Sep 2 yelds hree erms: The frs s O n 2ξ δ ν n ν2ξ ξγ = O δn ξγ ; he second one converges o 0 a.s. as, snce Y n 0 by Sep 0 and Lemma below wh y n = Y n and a n = n 2ξ ; and he hrd one s O n η 2ξ 0 snce η<2ξ. Alogeher, we ge O δ when γ =0, and 0 when γ>0. 8 I s neresng o noe ha, whle he regres are nvaran o a change of orgn for he uly funcon.e., addng a consan o all payoffs, hs s no so for he modfed regres. Noneheless, Sep 3 shows ha he resulng dfference s jus a marngale convergng a.s. o 0. 9 When γ = 0 and hus δ δ +v = 0, hs yelds an overesmae of O n, whch wll urn ou however no o maer.

19 Renforcemen Procedure Leadng o Correlaed Equlbrum 99 The proof s compleed by applyng agan he Srong Law of Large Numbers for Dependen Random Varables Theorem 4 and nong ha 0 ρ n =/b n ν n X ν. Lemma. Assume: y n 0 as n ; a n > 0 for all n; and n= a n =.Then c n := n ν= a νy ν / n ν= a ν 0 as n. Proof. Gven ε>0, le n 0 be such ha y n <εfor all n>n 0. Then c n = ν n 0 a ν y ν / ν n a ν + n a 0<ν n νy ν / ν n a ν. The frs erm converges o 0 snce he numeraor s fxed, and he second s bounded by ε. Proof of Sep 4: When n n+, we have by Sep 3: C j, k C n j, k =O v n / n δ n+ = O n ξγ 0, for all j k. Thus lm sup ρ = lm sup n ρ n. Recallng Sep complees he proof. Remark. For smplcy, we have assumed ha all players use he same sequence δ. In he case of dfferen sequences δ l for he dfferen players l N, wh δ l = δ l / γl for some δ l > 0 and 0 γ l < /4, s sraghforward o check ha he esmae of Sep 6 becomes now l N O w 2 / δ l +w + w δ l δ l +w. Choosng ξ o sasfy < ξ < mn {2/ + γ, / 4γ}, where γ := max l N γ l, yelds, on he rgh-hand sde of Sep 2, 2 nρ n + O δ n 2ξ ξγ + Y n n 2ξ + O n 2ξ, and Seps 3 and 4 go hrough wh δ and γ nsead of δ and γ, respecvely. The fnal resul n Sep 4 becomes: If δ = δ for all, hen lm sup R j, k =O δ a.s. If δ 0.e. f γ > 0, hen lm R j, k = 0 a.s. References. Auer P., N. Cesa-Banch, Y. Freund and R. E. Schapre [995], Gamblng n a Rgged Casno: The Adversaral Mul-Armed Band Problem, Proceedngs of he 36h Annual Symposum on Foundaons of Compuer Scence, Aumann, R. J. [974], Subjecvy and Correlaon n Randomzed Sraeges, Journal of Mahemacal Economcs, Baños, A. [968], On Pseudo-Games, The Annals of Mahemacal Sascs 39, Blackwell, D. [956], An Analog of he Mnmax Theorem for Vecor Payoffs, Pacfc Journal of Mahemacs 6, Borgers, T. and R. Sarn [995], Nave Renforcemen Learnng wh Endogenous Aspraons, Unversy College London mmeo. 6. Bush, R. and F. Moseller [955], Sochasc Models for Learnng, Wley.

20 200 Sergu Har and Andreu Mas-Colell 7. Erev, I. and A. E. Roh [998], Predcng How People Play Games: Renforcemen Learnng n Expermenal Games wh Unque, Mxed Sraeges, Amercan Economc Revew 88, Foser, D. and R. V. Vohra [993], A Randomzed Rule for Selecng Forecass, Operaons Research 4, Foser, D. and R. V. Vohra [997], Calbraed Learnng and Correlaed Equlbrum, Games and Economc Behavor 2, Foser, D. and R. V. Vohra [998], Asympoc Calbraon, Bomerka 85, Fudenberg, D. and D. K. Levne [998], Theory of Learnng n Games, MIT Press. 2. Fudenberg, D. and D. K. Levne [999], Condonal Unversal Conssency, Games and Economc Behavor 29, Hannan, J. [957], Approxmaon o Bayes Rsk n Repeaed Play, n Conrbuons o he Theory of Games, Vol. III Annals of Mahemacs Sudes 39, M. Dresher, A. W. Tucker and P. Wolfe eds., Prnceon Unversy Press, Har, S. and A. Mas-Colell [2000], A Smple Adapve Procedure Leadng o Correlaed Equlbrum, Economerca. 5. Har, S. and A. Mas-Colell [200], A General Class of Adapve Sraeges, Journal of Economc Theory. 6. Loève, M. [978], Probably Theory, Vol. II, 4h Edon, Sprnger-Verlag. 7. Megddo, N. [980], On Repeaed Games wh Incomplee Informaon Played by Non-Bayesan Players, Inernaonal Journal of Game Theory 9, Roh, A. E. and I. Erev [995], Learnng n Exensve-Form Games: Expermenal Daa and Smple Dynamc Models n he Inermedae Term, Games and Economc Behavor 8,