Discussion Paper Series Number 186

Transcription

1 Ednburgh School of Economcs Dscusson Paper Seres Number 86 The El Farol Bar Problem Revsed: Renforcemen Learnng n a Poenal Game Duncan Whehead Ednburgh Unversy Dae Sepember 2008 Publshed by School of Economcs Unversy of Ednburgh 30 3 Buccleuch Place Ednburgh EH8 9JT +44 (0) hp://edn.ac/6ja6a6

2 The El Farol Bar Problem Revsed: Renforcemen Learnng n a Poenal Game Duncan Whehead Unversy of Ednburgh Sepember 7, 2008 Absrac We revs he El Farol bar problem developed by Bran W. Arhur (994) o nvesgae how one mgh bes model bounded raonaly n economcs. We begn by modellng he El Farol bar problem as a marke enry game and descrbng s Nash equlbra. Then, assumng agens are boundedly raonal n accordance wh a renforcemen learnng model, we analyse long-run behavour n he repeaed game. We hen sae our man resul. In a sngle populaon of ndvduals playng he El Farol game, learnng heory predcs ha he populaon s evenually subdvded no wo dsnc groups: hose who nvarably go o he bar and hose who almos never do. In dong so we demonsrae ha learnng heory predcs sorng n he El Farol bar problem. Inroducon The El Farol bar problem was nroduced by Bran W. Arhur (994) as a framework o nvesgae how one models bounded raonaly n economcs. I was nspred by he El Farol bar n Sana Fe, New Mexco, whch o ered Irsh musc on Thursday nghs. The orgnal problem was consruced as follows: N people decde ndependenly each week wheher o go o a bar ha o ers eneranmen on a ceran ngh. For correcness, le us se N a 00. Space s lmed, and he evenng s enjoyable f hngs are no oo crowded spec cally, f fewer han 60 percen of he possble 00 are presen. There s no sure way o ell he numbers comng n advance; herefore a person or an agen goes (deems worh gong) f he expecs fewer han 60 o show up or says home f he expecs more han 60 o go. Arhur s (994) prelmnary resuls from he eld of compuaonal economcs show ha he number of people aendng he bar converges quckly and hen hovers around he capacy level of he resource. Our conrbuon o he leraure on he El Farol bar problem and heory of learnng n games s fourfold. Frs, we apply he Erev and Roh (998) model of renforcemen learnng o he El Farol framework. We beleve he Arhur (994), pp 409.

3 Erev and Roh (998) model of renforcemen learnng s he mos approprae ndvdual learnng model o apply n hs nsance, because n general people who say a home do no know wha payo hey would have receved f hey had gone o he bar. We hen prove analycally ha long-run behavour wll converge asympocally o he se of pure sraegy Nash equlbra of he El Farol sage game. 2 In oher words he number of people aendng he bar converges and hen hovers around he capacy level of he resource. Furhermore, learnng heory predcs sorng n he El Farol bar problem; ha s, n a sngle populaon of ndvduals playng he El Farol game, learnng heory predcs ha he populaon s evenually subdvded no wo dsnc groups: hose who nvarably go o he bar and hose who almos never do. Second, we demonsrae ha he El Farol bar problem may be modelled as a marke enry game wh boundedly raonal renforcemen learners. We buld upon he work of Du y and Hopkns (2005), who have proved ha n marke enry games, where payo s are decreasng n a connuous manner wh respec o he number of oher marke enrans, he only asympocally sable Nash equlbra are hose correspondng o pure Nash pro les. Our man resul also proves asympoc convergence o hose equlbra correspondng o pure Nash pro les n he marke enry game. In addon our resul also proves ha hs s he case when payo s are decreasng n a dsconnuous way wh respec o he number of oher marke enrans. Thrd, Sandholm (200) has proved ha, under a broad class of evoluonary dynamcs, behavour convergences o Nash equlbrum from all nal condons n poenal games wh connuous player ses. Sandholm s (200) convergence resuls assume ha ndvdual behavour adjusmens should sasfy wha was ermed posve correlaon; meanng any myopc adjusmen dynamc ha exhbs a posve relaonshp beween growh raes and payo s n each populaon. Our resul conrbues o hs leraure by provng ha, for he evoluonary dynamcs assocaed wh Erev and Roh s (998) model of renforcemen learnng, long-run behavour converges n poenal games wh ne ses of players. Fnally, here s a conrbuon o be made o he exensve leraure on he El Farol bar problem and s assocaed problem, he Mnory Game n he eld of complex sysems. 3 Currenly, would appear ha he opporuny o apply convergence resuls from models of ndvdual learnng o suaons lke hose represened by he El Farol bar problem has been overlooked. We wll begn by usng he ools of game heory o model he El Farol bar problem as a non-cooperave coordnaon game n whch payo s are deermned by negave exernales. We hen model he El Farol bar problem as a repeaed marke-enry game wh boundedly raonal agens. Analyss of he sage game wll show ha here are a large number of Nash equlbra. Therefore, equlbra re nemen/coordnaon becomes problemac. In order o re ne he equlbra se, we allow players o learn from experence. The analycal ools developed n Du y and Hopkns (2005), Hopkns and Posch (2005) and Monderer and Shapley (996) wll be employed o sudy he predced oucome of play under he Erev and Roh (998) model of renforcemen learnng. 2 Ths s n conras wh Franke s (2003) use of numercal smulaons of renforcemen learnng appled o he El Farol bar problem. 3 See hp:// for research on he Mnory Game. 2

4 Renforcemen learnng assumes ha ndvduals only have access o he aendance gures of he bar for each week ha hey aend. 4 The long-run behavour of agens under hs adapve rule wll hen be consdered, and wll be shown ha under hs learnng process, play wll converge o he se of pure sraegy Nash equlbra wh probably one. The nuon behnd our man resul s ha n he El Farol bar problem renforcemen learners who do no regularly aend are more ofen han no dsapponed when hey do choose o do so. Smlarly, hose who regularly aend always seem o have a good me, and hus are more lkely o aend n he fuure. A good way o hnk abou hs oucome s o magne ha all players n one week play a mxed sraegy. I s que lkely ha he bar acually urns ou o be busy. Therefore, all agens who aended wll be renforced wh he lower payo. Ths wll reduce her propensy o aend n he fuure. The followng week he probably of he bar beng overcrowded wll be dmnshed. Those who do aend wll mos lkely receve hgh payo renforcemen from aendng and her propensy o aend n he fuure wll ncrease agan whle ha of he players who sayed away wll be reduced. Therefore, we have wo posve feedback loops. One causes hose who aend regularly o do so more ofen. The oher leads hose who say a home o be more lkely o do so n he fuure. We can herefore see ha any mxed sraegy Nash equlbrum s asympocally unsable under he dynamcs of Erev and Roh (998) renforcemen learnng. In Secon 2 we revew he El Farol bar problem as nroduced by Arhur (994). We se ou hs modellng approach o bounded raonaly n he El Farol bar problem and summarse he nal resuls from hs compuaonal expermens. We dscuss he use of he nducve hnkng approach o modellng bounded raonaly, boh n he El Farol bar problem and s closely relaed problem, he Mnory Game. We hen oulne our movaon for he applcaon of he ndvdual learnng approach o capurng he bounded raonaly of decson makers and sugges a renforcemen learnng model for he El Farol framework. In Secon 3 we nroduce our model of he El Farol bar problem, de ne he El Farol sage game and characerse he se of Nash equlbra, se ou n deal he Erev and Roh (998) model of renforcemen learnng whn he El Farol framework, and wre down an expresson for player s expeced sraegy adjusmen. In Secon 4 we sae and prove our man resul; ha n he El Farol bar problem a populaon of boundedly raonal agens who behave n accordance wh he Erev and Roh (998) renforcemen learnng model are sored no hose who always aend he El Farol bar and hose who always say a home. Fnally, we provde some concludng remarks n Secon 5. 2 The El Farol Bar Problem The El Farol bar problem was creaed by Arhur (994) as a devce o nvesgae how one mgh bes model bounded raonaly n economcs. I was nspred by he El Farol bar n Sana Fe, New Mexco, whch o ered Irsh musc on Thursday nghs. The problem s se ou as follows: here s a ne populaon 4 However, he resuls presened here whn are easly exended o allow for he more generc se-up where all ndvduals learn aendance gures wheher hey aend or no. Ths s ofen referred o as hypohecal renforcemen or cous play learnng. 3

5 of people and every Thursday ngh all of he hem wan o go o he El Farol bar. However, he El Farol bar s que small, and s no enjoyable o go here f s oo crowded. So much so, n fac, ha he followng rules are n place: If less han 60% of he populaon go o he bar, hose who go have a more enjoyable evenng a he bar han hey would have had had hey sayed a home. If 60% or more of he populaon go o he bar, hose who go have a worse evenng a he bar han hey would have had had hey sayed a home. Unforunaely, s necessary for everyone o decde a he same me wheher hey wll go o he bar or no. They canno wa and see how many ohers go on a parcular Thursday before decdng o go hemselves on ha Thursday. The mporan characersc of he El Farol bar problem s ha f here was an obvous mehod ha all ndvduals could use o base her decsons on, hen would be possble o nd a deducve soluon o he problem. However, no maer wha mehod each ndvdual uses o decde f hey wll go o he bar or no, f everyone uses he same mehod s guaraneed o fal. Therefore, from he pon of vew of he ndvdual, he problem s ll-de ned and no deducve raonal soluon exss. Suaons lke hose represened by he El Farol bar problem hghlgh wo spec c reasons why perfec deducve reasonng mgh fal o provde clear soluons o some heorecal problems. The rs s smply a queson of he cognve lmaons of he mnd. Beyond a ceran level of complexy, logcal capacy fals o cope. The second s ha n complex sraegc suaons ndvduals canno always rely on persons hey are neracng wh o behave under assumpons of perfec raonaly. In suaons lke he El Farol bar problem, ndvduals are forced no a world where hey mus choose her sraeges based on guesses of her opponens lkely behavour. Whou objecve, well-de ned, shared assumpons, hese ypes of problems become ll-de ned and canno be solved raonally. The queson ha arses s how does one bes model bounded raonaly n economcs when perfec raonaly fals? Gven he de nng characersc of he El Farol bar problem, namely ha ndng a deducve raonal soluon s mpossble, follows ha he problem self could provde a useful framework o explore models of bounded raonaly n general. 2. Inducve Reasonng n he El Farol Framework Arhur (994) noes ha here s a consensus among psychologss ha n suaons ha are eher complcaed and/or ll-de ned, humans end o look for paerns n order o develop nernal models on whch hey can base her decsons. These mehods are nherenly nducve. In he El Farol bar problem, Arhur (994) follows hs lne of hough and posulaes ha ndvduals decde wheher hey wll go o he bar or no by employng menal models o predc expeced fuure aendance. In oher words hey creae forecasng models. If an ndvdual usng a spec c forecasng model predcs aendance o be low hen, based on ha model, ha ndvdual would aend and vce-versa f aendance s predced o be hgh. 4

6 As prevously dscussed, and dervng from he ll-de ned naure of he El Farol bar problem self, we can conclude ha no forecasng model can be employed by all ndvduals and be accurae a he same me. We can easly demonsrae hs fac by assumng ha a forecasng model exss ha predcs ha he aendance n he comng week, gven aendance n pas weeks, s gong o be hgh. If all ndvduals use hs forecasng model o base her decsons on, hen nobody wll go o he bar. 5 Ths hen renders he forecas nvald and mples ha here exss no sngle forecasng model ha all ndvduals can use upon whch o base her aendance decsons. No deducve soluon exss o hs problem. 2.. The Inducve Thnkng Approach Arhur s (994) approach o modellng bounded raonaly n he El Farol bar problem s o assume ha each ndvdual has access o a number of forecasng models whch hey use o make her decsons. Furhermore, hey score and rank hese models a he end of each week accordng o her accuracy n order o deermne whch parcular model hey should base her decson on. Formally, Arhur (994) magnes ha each ndvdual ulses a number of forecasng models, denoed s k, o predc aendance n he comng week. Each model forecass aendance for he comng week gven he hsory of aendance over he las d weeks, denoed d (h ) 2 D, where D s he se of all possble aendance pro les for he las d weeks and d s an exogenously xed parameer. Then, followng he dsclosure of he number of ndvduals who aended he El Farol bar on he mos recen Thursday ngh, a score s assocaed wh each forecasng model. Spec cally, he score, denoed U s k, s calculaed by compung he weghed average of he score of he same model n he prevous week and he absolue d erence beween he forecasng model s las predcon, denoed s k (d (h )), and he mos recen realsed urnou, denoed y. Equaon () formulses hs calculaon. 6 U s k = U s k + ( ) s k (d (h ) y ) () In each week he forecasng model wh he hghes score s referred o as he acve predcor. On each Thursday ndvduals underake he acon of eher aendng he El Farol bar or no n accordance wh her acve predcor. If an ndvdual s acve predcor forecass he aendance on he comng evenng o be hgh, hen ha ndvdual wll choose no o go o he bar. Conversely, f he acve predcor forecass aendance o be low, hen ha ndvdual wll deem worhwhle gong o he bar and hey wll ancpae an enjoyable evenng of Irsh musc. Once all ndvduals have made her decsons,.e. wheher o aend he El Farol bar or no, hey are hen nformed of he acual urnou a he bar. Ths nformaon s made know publcly o all ndvduals. Each ndvdual hen realses her payo s, updaes he score for all her avalable forecasng models, and con rms her acve predcor for nex Thursday s decson. 5 Ths s remnscen of Yog Berra s famous commen, "Oh, ha place. I s so crowded nobody goes here anymore." 6 I should be noed ha I have aken spec c care o oulne he El Farol bar problem and Arhur s proposed model of he problem as he orgnally formulaed. Ths has been possble due o he work of Zambrano (2004) who re-analysed Arhur s orgnal code. 5

7 2..2 Agen-Based Compuer Smulaons Arhur (994) nvesgaed hs model of he El Farol bar problem hrough he use of compuaonal expermens. He desgned ar cal agens and smulaed her dynamc neracon over me. In Arhur s (994) compuer smulaons, as n he orgnal formulaon of he problem, he sze of he populaon, N, s se o 00 and he enjoyable capacy of he El Farol bar, C, s se o 60. Arhur (994) hen creaes a ne se of dverse forecasng models, or predcors, whch map aendance hsores o a predced bar aendance for he comng week. These models were doled ou unformly and randomly, such ha each agen was endowed wh a non-ransferable se of K forecasng models. 7 Each smulaon expermen was hen run for 00 perods wh he combned runs oallng o 0; 000 perods. Fgure : Aendance Accordng o Arhur s (994) Smulaons. The rs hng o noe abou he resuls of hese compuer expermens s ha, gven he sarng condons and he xed se of predcors avalable o each smulaed agen, he dynamcs are compleely deermnsc. Neverheless, he smulaons produce some neresng resuls. Two observaons become mmedaely apparen. Frs, mean aendance always converges o he capacy of he bar. Second, on average 40% of he acve predcors forecased aendance o be hgher han he capacy level and 60% below. Arhur (994) expands on hese observaons by nong ha, he predcors self organse no an equlbrum paern or ecology. 8 An example of he aendance raes from a ypcal run of 00 perods can be seen n Fgure The Mnory Game There has been much neres n he El Farol bar problem as a sysem o sudy agens n marke-lke neracons. Ths has led o he de non of a smlar problem called he Mnory Game whch embodes some basc marke mechansms, whle keepng mahemacal complexy o a mnmum. 7 Ths dd no preclude he possbly ha he agens predcor ses mgh overlap. 8 Arhur (994), pp

8 The Mnory Game s a repeaed game where N agens have o decde beween wo acons, such as buy or sell or aend or no. Wh N odd hs procedure den es a mnory acon as ha chosen by he mnory. Agens who ake he mnory acon are rewarded wh one payo un. Agens canno communcae wh one anoher and hey have access o publcly avalable nformaon on he hsory of pas oucomes for a xed number of perods. As n he El Farol bar problem, he se up requres a prohbve compuaonal ask and, from a sraegc pon of vew, he problem s ll-de ned. Agan s posulaed ha n such complex sraegc neracons, agens may prefer o smplfy her decson asks by seekng ou behavour rules, or heurscs, ha allocae an acon for each possble observed hsory of oucomes. The leraure on he Mnory Game concludes, hrough boh agen-based and analycal models, ha here exss a cooperave phase of play when he rao of he number of unque possble hsores o he number of agens, N, s large enough. Tha s, wh respec o he so-called random agen sae, n whch each agen chooses her acon by ppng a con, agens are beer o because he sysem moves o a sor of coordnaed sae. The analycal research on he Mnory Game employs echnques borrowed from sascal physcs n order o descrbe he game as a spn sysem, hus enablng he sysem s properes o be oulned. I should be noed ha hs avenue of nvesgaon does no enable he sudy of ndvdual behavour, bu only he sysem as a whole. One aspec of hs approach, and ndeed Arhur s (994) orgnal nvesgaons, o he El Farol bar problem and bounded raonaly s ha he heory does no explcly deal he predcors ha should/would be avalable o each ndvdual/agen. In realy here mos lkely exss an evoluonary process ha regulaes he se of predcors as a whole and her avalably o each ndvdual agen. Arhur (994) draws on he followng meaphor o make he pon: Jus as speces, o survve and reproduce, mus prove hemselves by compeng and beng adaped whn he envronmen creaed by oher speces, n hs world hypohess, o be accurae and herefore aced upon, mus prove hemselves by compeng and beng adaped whn and envronmen creaed by oher agens hypohess Indvdual Learnng n he El Farol Framework The El Farol bar problem represens a complex sraegc envronmen where raonal deducve hnkng fals o provde any clear soluons. The queson we wsh o address s wha we should pu n place of perfec raonaly. In he prevous secon, we revewed he leraure reporng work ha has been dreced a achevng hs goal whn he El Farol framework hrough he use of nducve reasonng. Suppose nsead ha ndvduals n he El Farol bar problem can nd her way o an opmal soluon by ral and error,.e. learnng. 0 In e ec we propose ha hs s he role ha, loosely speakng, he predcors ful l n Arhur s (994) orgnal paper on nducve reasonng and bounded raonaly n he El Farol bar problem. Recall ha f a predcor correcly forecass aendance, s more lkely o be used as an acve predcor. 9 Arhur, (994), pp A player canno adap o suaons ha are only encounered once. Wh hs n mnd, we mus consder players learnng equlbra n an dencally repeaed game envronmen. 7

9 If no, wll no be used. Followng hs argumen seems reasonable o consder he El Farol bar problem as one wh boundedly raonal agens who gradually adjus her behavour over me, unl here s no longer any room for mprovemen n her payo s. In game heory he echnques for modellng hs ype of adapaon process are closely relaed o replcaor dynamcs. The dea of replcaor dynamcs was nroduced by Smh (974) o model dynamc processes n he bologcal scences. Essenally, replcaor dynamcs says ha f an ndvdual of a ceran ype earns an above average payo, hen ha ndvdual ype s frequency n he populaon rses. When modellng an ndvdual learnng process n a repeaed game, we modfy hs nerpreaon of replcaor dynamcs o he followng: f an ndvdual who has a propensy o use a parcular sraegy earns an above average payo from ha sraegy, hen he propensy o use ha sraegy n he fuure ncreases. The El Farol bar problem wll now be modelled as a repeaed marke enry game where players adhere o a pre-spec ed learnng process. The manner n whch ndvdual learnng s modelled n repeaed games s smple and que nuve. Essenally, ndvdual learnng s an algorhm ha each player follows n each perod of play. Imagne ha each ndvdual n he El Farol bar problem, wheher hey go o he bar or no, keeps an urn by her sde. In he urn here are a number of balls coloured eher green or red. We can consder hese balls o be replacng he funcon of Arhur s (994) predcors n he El Farol bar problem. Insead of each ndvdual makng her acon choce dependen on he forecas of her acve predcor, players wll choose a ball from her urn and obey s colour codng. In oher words f a green ball s seleced ha ndvdual wll go o he bar and f a red ball s chosen hey wll say a home. Once a ball s drawn and he correspondng acon s aken, he ball s hen placed back no he urn. The learnng model s hen spec ed by an updang rule. Ths s he se of nsrucons ha dcaes how many balls and of wha colours should be added o he urn afer each round of play. Usng hs framework we can descrbe each player as havng propenses for each acon. The propensy o underake a ceran acon s a funcon of he number of correspondngly coloured balls n he urn. 2 The probably ha a ball of a ceran colour wll be chosen from a parcular ndvdual s urn s deermned by he choce rule, whch s a mappng from propenses o a number n he un nerval. To nd he equlbrum, we calculae n he lm, as he number of repeons of he game ends o n ny, he probably ha each acon wll be aken. Le us now recall n deal he movaon for employng an ndvdual learnng model of bounded raonaly n he El Farol bar problem. As prevously saed, he complexy of he problem makes reasonable o assume ha ndvduals su er from cognve lmaons. Furhermore, we have already demonsraed ha he complexy of belefs means ha, from a sraegc vewpon, ndvduals are unable o employ deducve reasonng o denfy opmal/coordnaed sraeges. Gven hese consrans we suppose ha ndvduals nd her opmal sraeges n he El Farol bar problem hrough Ths s no o be aken lerally, bu hey wll provde he same decson funcon as he predcors do n Arhur s formulaon. 2 I s also dependen on he choce rule spec ed n he learnng model whch shall be expanded on laer n he paper. 8

10 repeaed neracon and he applcaon of an adapve algorhm. I wll be assumed ha any adapve algorhm wll adhere o some basc prncples of ndvdual learnng. Frs, he law of e ec: choces ha have led o good oucomes n he pas are more lkely o be repeaed n he fuure. Second, he power law of pracce: learnng curves should nally be seep and hen laer hey should be aer. Ths s paramoun o assumng ha n any adapve process he adjusmens become smaller over me. Fnally, choce behavor should be probablsc. Ths s a basc assumpon n mos mahemacal learnng heores proposed n psychology. Erev and Roh (998) have developed a robus model of renforcemen learnng whch ncorporaes all hese prncples ha shall be appled o our model of he El Farol bar problem. 3 A Model of he El Farol Bar Problem The El Farol bar problem s essenally a repeaed smulaneous move game. There are N players wh dencal preferences who aemp o coordnae her acons of eher gong o he bar or sayng a home n such a way as o maxmse her ndvdual payo s, subjec o he crowdng exernaly from gong o he bar. Players need o coordnae her acons, ndependenly and whou pror communcaon, such ha: when a player decdes o go o he bar,.e. deems worhy of gong o he bar, hey can look forward o a payo ha s greaer han wha hey would have receved had hey sayed a home and when a player decdes o say a home,.e. deems no worhy of gong o he bar, hey can look forward o a payo ha s greaer han wha hey would have receved had hey no sayed a home. The El Farol bar problem can be nerpreed as a marke enry game (Franke 2003). In general marke enry games are nerpreed as runcaed wo-sage games (Selen and Güh 982). In he rs sage, players smulaneously choose eher o ener or say ou of he marke. Then, n he second sage, he payo s of he enrans are deermned from her marke acons. Usually hese payo s are negavely relaed o he number of marke enrans n a connuous way. However, n he El Farol bar problem, payo s o players enerng he bar are relaed o he number of bar aendans n a dsconnuous manner. Alernavely, he El Farol bar problem may be vewed as a congeson model and hus can be modelled, a la Rosenhal (973), as a congeson game. 3 I s a congeson game, because each player s payo depends on he number of oher players who choose o ulse he same resource, namely he El Farol bar. Ths nerpreaon has been referred o n many sudes of he El Farol bar problem n he leraure (e.g. Greenwald, Mshra, and Parkh 998, Bell and Sehares 999, Bell and Sehares 200, Bell, Sehares, and Bucklew 2003, Farago, Greenwald, and Hall 2002, Zambrano 2004), bu has rarely been developed. In hs paper we shall nally nerpre he El Farol bar problem as a marke enry game. Laer on n our dscussons we shall reurn o he dea of congeson 3 Clearly marke enry games are a subse of he larger class of congeson games. 9

11 games, because hey have mporan properes ha are useful n undersandng he long-run behavour of boundedly raonal agens learnng n accordance wh a renforcemen model n he El Farol bar problem. 3. The El Farol Sage Game Le C, a posve no-zero neger, represen he capacy of he bar. If less han C players choose o go o he bar, hen he payo hey receve s alled wh he noon ha ex pos hose players deemed worhwhle gong. They receve a payo srcly greaer han he payo hey would have receved had hey sayed a home. On he oher hand, f C or more players choose o go o he bar, hen he payo he bar enrans receve s alled wh he noon ha, ex pos, hose players dd no deem worhwhle gong o he bar. In oher words hey receve a payo srcly less han he payo hey would have receved had hey sayed a home. Sae Uncrowded Crowded Player Go o he Bar G B Say a Home S S where G > S > B Fgure 2: Sae Dependen Payo for Player n he El Farol sage game. The payo funcon for each player consss of an uncondonal payo for sayng a home, denoed by S, and a condonal payo, denoed by G or B, dependen on he sae of he bar. There are wo saes of he bar, crowded or no crowded, and he sae s deermned by he remanng N players. To ensure he sraegc form of he game, he payo s mus be srcly ordered such ha G > S > B. The payo srucure for represenave player for an solaed Thursday n he El Farol bar problem can be represened by he followng payo marx (see Fgure 2). Gven he above prelmnares, we can now de ne he El Farol sage game as a sngle-sage marke enry game wh dsconnuous, bu weakly monoonc, payo s n oher players acons. De non De ne he El Farol sage game as he one sho sraegc game =< N; ; > conssng of, N players ndexed by 2 f; 2; ; :::; Ng, 0

12 a ne se of acons = f0; g ndexed by, where = denoes player s acon go o he bar and = 0 denoes player s acon say a home and 4 a payo funcon :! R = fs; B; Gg, such ha G > S > B, where = Q 6=j j de nes he sae of he bar. Formally we can wre he payo funcon as, 8 < G f = and P j6= j < C = B f = and P : j6= j C S f = 0 where C 2 Z. 3.. Nash Equlbra n he El Farol Sage Game Le us now characerse he equlbra of he El Farol sage game. The rs hng o noe s ha he number of Nash equlbra n he El Farol sage game s large and rses quckly as N ncreases. Furhermore, he number of Nash equlbra s maxmsed for any gven N when C N=2. There are essenally hree ypes of Nash equlbra, namely: Pure Sraegy Nash Equlbra Nash equlbra where all players play a pure sraegy. Symmerc Mxed Sraegy Nash Equlbra Nash equlbra where all players play a mxed sraegy. Asymmerc Mxed Sraegy Nash Equlbra Nash equlbra where some players play a pure sraegy and he remanng play a mxed sraegy. Le Y denoe he se of Nash equlbra of he El Farol sage game. I can be shown ha Y conans a ne number of elemens. In Proposon we sae he number of pure sraegy Nash equlbra, denoed Y P. Nex, we show va Proposons 2 and 3 ha here exss a unque symmerc mxed sraegy Nash equlbrum, denoed Y S. And nally n Proposon 4, we show ha he number of asymmerc mxed sraegy Nash equlbra, denoed Y A, s counable. Therefore, he number of Nash equlbra n he El Farol sage game s ne. 5 Proposon The number of pure sraegy Nash equlbra n he El Farol sage game wh N 2 N players and a capacy of C 2 N s, N N! = (2) C C! (N C)! 4 I should be noed ha alhough we employ he noaon o denoe he se of only wo acons avalable o each player, we do so only o ndcae how he renforcemen learnng model would be exended o games wh more han wo dsnc acons. 5 Noe ha Y = Y P [ Y S [ Y A.

13 Proof See Secon A. n Appendx A. The followng wo proposons ogeher demonsrae ha a symmerc mxed sraegy Nash equlbrum exss and s unque. In Proposon 2 we prove ha here s a symmerc mxed sraegy Nash equlbrum where all players play he same mxed sraegy and ha s unque. In Proposon 3 we hen prove ha f all players are playng a mxed sraegy hey mus be playng he same mxed sraegy. Therefore, we have a unque symmerc mxed sraegy Nash equlbrum n he El Farol sage game. 6 Proposon 2 In he El Farol sage game here s a symmerc mxed sraegy equlbrum where all players play he same mxed sraegy de ned by he sraegy uple (; [ ]), where denoes he probably of gong o he bar and [ ] denoes he probably of sayng a home. Furhermore, s unquely de ned by he followng relaonshp: S G B = B CX m=0 N m Proof See Secon A.2 n Appendx A. m [ ] N m (3) Proposon 3 In a Nash equlbra n he El Farol sage game where all players employ a mxed sraegy, all agens mus play he same mxed sraegy. Proof See secon A.3 n Appendx A. Le us now consder he asymmerc mxed sraegy Nash equlbra. Gven ha we can calculae he number of pure sraegy Nash equlbra from (2) and ha here s a unque symmerc mxed sraegy Nash equlbrum, an approach can be abled o demonsrae ha he number of asymmerc mxed sraegy Nash equlbra s ne. Proposon 4 The number of asymmerc mxed sraegy Nash equlbra n he El Farol sage game s counable. Proof See Secon A.4 n Appendx A. We have now characersed he Nash equlbra of he El Farol sage game. Furhermore, we have shown ha he number of Nash equlbra s ne. Ths ndng wll be employed laer n provng our man resul. 3.2 The El Farol Game For compleeness we de ne he El Farol bar problem as he repeaed El Farol sage game wh boundedly raonal agens who learn n accordance wh he Erev and Roh (998) renforcemen learnng model. Le us begn by de nng he El Farol game. De non 2 The El Farol game s he n nely repeaed El Farol sage game. 6 A smlar resul has been proved by Cheng (997). 2

14 3.3 Erev and Roh (998) Renforcemen Learnng We now se ou he procedure for he Erev and Roh (998) renforcemen learnng model n deal. In hs learnng model, each player has a propensy o underake each acon n each perod, denoed q (). The melne of he learnng procedure s ha n each perod each player chooses o underake one of her avalable acons 2 = f0; g n accordance wh a mappng from he propenses o he un nerval [0; ]. Ths mappng s de ned by he choce rule. The player hen underakes he acon dcaed by he choce rule and receves a payo n ha perod assocaed wh ha acon. Player hen updaes hs propenses. The updang procedure s deermned by he updang rule. In he Erev and Roh (998) renforcemen learnng model, he only propenses o be updaed are hose correspondng o he acual acon aken. We can now de ne he model formally. The learnng procedure comprses of hree componens: he nal condons, a choce rule and an updang rule Inal Condons Le q () be player s propensy o play acon 2 n perod. In he nal perod, = 0, we assume ha all players have posve propenses for all possble acons. Tha s, q () > 0 for = 0 and for all 2 N and 2 (4) Ths assumpon, along wh posve payo s, wll also ensure ha q () > 0 for all and Choce Rule Each player has posve a propensy, q (), o ake acon 2 = f0; g n perod. In models of renforcemen learnng, he choce rule provdes a mappng from propenses o sraeges. Le y; y represen player s mxed sraegy n perod wh wo possble acons 2 = f0; g, where y s he probably placed by agen on acon = n perod and y s he probably placed by agen on acon = 0 n perod. The choce rule employed n he Erev and Roh (998) renforcemen learnng model s ofen referred o as he smple choce rule. I s a sraghforward probably mappng from propenses o he un nerval [0; ]. Tha s, Pr ( = ) = y q () = P 2 q () = q () Q where Q = P 2 q (). 7 (5) Updang Rule Le ; m denoe he realsed ncremen o player s propensy n perod from akng acon 2 = f0; g gven he aggregae acons aken by he 7 Noe ha snce here are only wo possble acons for each player we can wre Pr ( = 0) = y q = P (0) 2 q () = q (0) Q 3

15 remanng N, denoed by m where m = P j6= j. To complee, and mos crucal o, our renforcemen learnng model, we mus sae he means by whch players updae her propenses. Spec cally, n he Erev and Roh (998) renforcemen learnng model, akes he form ha f agen akes acon n perod, hen he agen s h propensy s ncreased by an ncremen equal o agen s realsed payo n ha perod. All oher propenses reman unchanged. In oher words only realsed payo s ac as renforcers. We hus have he followng updang rule, 8 q + () = q () + ; m for all 2 = f0; g (6) 3.4 Renforcemen Learnng n he El Farol Game We wll now model he El Farol bar problem as he El Farol game wh boundedly raonal agens who learn accordng o he Erev and Roh (998) renforcemen learnng model. To sudy he long-run dynamcs of he El Farol game wh bounded raonal agens learnng n accordance wh he Erev and Roh (998) renforcemen model, we need o rs wre he expeced moon of he h player s = sraegy adjusmen. In order o accomplsh hs ask, we mus rs de ne player s expeced payo ncremen. Le ^ ; y denoe he expeced ncremen o player s propensy n perod from akng acon gven he aggregae acons aken by he remanng N players, denoed by y, where y s a vecor sraegy pro le. Noe ha he updang rule n he Erev and Roh (998) renforcemen learnng model s a funcon of realsed payo s. However, he expeced moon of he h player s = sraegy adjusmen wll be a funcon of expeced payo ncremens. Ths s quanavely and qualavely d eren from realsed payo ncremens Expeced Sraegy Adjusmen n he El Farol Game To oban analycal resuls from he applcaon of Erev and Roh (998) renforcemen learnng model o he El Farol game, we make use of resuls from he heory of sochasc approxmaon. In essence we nvesgae he behavour of he sochasc learnng model by evaluang s expeced moon as!. In he case of he Erev and Roh (998) learnng model de ned by he choce rule (5) and updang rule (6), we can wre down he expeced moon of he h player s = sraegy adjusmen hrough he followng proposon: Proposon 5 Gven he choce rule (5) and he updang rule (6), he expeced moon of he h player s = sraegy adjusmen n he repeaed El Farol game s: E y+jy y = Q y + O! Q 2 y ^ ; y ^ 0; y (7) Proof See Secon B. n Appendx B. 8 Noe ha hs updang rule reveals why n hs model of renforcemen learnng all payo s mus be posve. Oherwse, here would be a possbly of propenses becomng negave and hus leadng o choce probables ha are unde ned. 4

16 4 Long-run Behavour n he El Farol Game We now arrve a our man resul. We consder he behavour of he expeced moon of he players = sraegy adjusmen as!. We begn by sang he man resul. Theorem (Man Resul) If agens n he repeaed El Farol game as de ned employ he choce rule (5) and renforcemen updang rule (6) for all of N 2 N and C 2 N such ha C N and payo s such ha G > S > B 0, wh probably one he Erev and Roh (998) renforcemen learnng process converges o a pure Nash equlbrum of he one-sho El Farol game. Tha s, Pr lm! y 2 Y P =, where y = y ; y 2 ; :::; y N, y 2 Y, s a sraegy pro le for he N agens and Y P s he se of pure Nash equlbrum pro les. Now we prove he man resul, Theorem. In he El Farol game wh dencal boundedly raonal agens, learnng accordng o he Erev and Roh (998) renforcemen learnng model, long-run behavour converges asympocally o he se of pure sraegy Nash equlbra of he El Farol sage game. Ths resul s esablshed by sudyng he convergen behavour of he dscree me sochasc process (7) descrbng he expeced sraegy adjusmen of player s acon of gong o he bar. In essence we wsh o nvesgae he lm of hs process as!. We accomplsh hs ask n wo man sages: a posve convergence saemen and a negave one. Drawng hese wo resuls ogeher we prove our man resul. Each sage employs resuls from he leraure on sochasc approxmaon. Frs, a resul of Benaïm (999, Corollary 6.6) s employed o demonsrae ha he sochasc process wll, n he lm as!, converge asympocally o one of he xed pons of he adjused replcaor dynamcs. Second, wo resuls of Hopkns and Posch (2005, Proposon 2 and 3) are ulsed o demonsrae ha he sochasc process descrbng he expeced sraegy adjusmen of player s acon of gong o he bar wll no converge asympocally o any xed pons ha do no correspond o a Nash equlbra of he El Farol sage game or o any correspondng Nash equlbra ha are unsable under he adjused replcaor dynamcs. These wo sages combned wll mply ha he dscree me sochasc process descrbng he expeced sraegy adjusmen of player s acon of gong o he bar converges asympocally o he se of pure sraegy equlbra of he El Farol sage game. 4. Proof of Man Resul: Frs Sage In he rs sage of he proof, we show ha he dscree me sochasc process (7) converges wh probably one o one of he xed pons of he sandard replcaor dynamcs. Consder for a momen he behavour of he followng sochasc process (Benvense, Méver, and Proure 990): x = f (x ) + (x ) + O [ ] 2 (8) x + 5

17 where x les n R N, E [ (x ) jx ] = 0 and de nes he naure of he gan n hs adapve process. For our purposes s nerpreed as he sep sze of he learnng algorhm. In our analyss we wsh o sudy he generc convergence properes of sochasc processes of hs form as!. I urns ou ha he naure of he gan s mporan n deermnng wha nferences can be made abou he behavour of (8) n he lm. In fac he sronger resuls from he heory of sochasc approxmaon apply o adapve algorhms wh decreasng gan, ha s sochasc processes wh decreasng sep sze. De non 3 The sochasc process (8) s sad o have decreasng gan f X ( ) < for some > where X = + For example a common sep sze of = = would ensure ha (8) has decreasng gan. I emerges ha as! here s a close relaonshp beween he behavour of sochasc processes (8) wh a decreasng gan and he mean or averaged ordnary d erenal equaon of he sochasc process. _x = f (x) (9) In parcular can be shown va Benaïm (999, Corollary 6.6) ha f (9) mees ceran crera, he sochasc process (8) mus converge wh probably one o one of he xed pons of he mean or averaged ordnary d erenal equaon (9). Theorem 2 (Benaïm (999, Corollary 6.6)) If he dynamc process (9) adms a src Lyapunov funcon and processes a ne number of xed pons, hen wh probably one he sochasc process (8) converges o one of hese xed pons. We now have a mehod of llusrang ha he long-run behavour of boundedly raonal agens, adjusng her sraeges accordng o he Erev and Roh (998) renforcemen learnng model, n he El Farol game converges o one of he xed pons of mean or averaged d erenal equaon (9) assocaed wh he vecor of player s expeced sraegy adjusmens. In order o apply hs general resul, we mus rs denfy he mean or averaged d erenal sysem assocaed wh players expeced sraegy adjusmen. Furhermore, mus be shown ha he mean or averaged d erenal sysem adms a src Lyapunov funcon. And nally, we mus esablsh ha he mean or averaged d erenal sysem possesses a ne number of solaed xed pons. In he nex hree subsecons we purpor o demonsrae jus ha. 4.. The Jon Dynamc Sysem One mgh hope ha he sandard replcaor dynamcs represen he mean or averaged d erenal sysem derved from he dscree me sochasc process (7). _y = y y ^ ; y ^ 0; y (0) 6

18 Unforunaely, he sandard replcaor dynamcs (0) do no for wo smple reasons. Frs, n he Erev and Roh (998) model, he sep sze s endogenous; ha s, s deermned by he accumulaon of payo s, and hus s no exogenously xed. Second, he sep sze s no a scalar. In order o accoun for hese dscrepances, le us nroduce a common sep sze of = = and N new varables, such ha: = Q We can now subsue for =Q n our dscree me sochasc process (7) and arrve a he followng correced expeced moon of he h player s sraegy adjusmen of gong o he bar: E y+jy y = y + y ; y 0; y y ^ ^ + O [ ] 2 () Snce we have assumed ha all payo s n he El Farol game are posve o ensure ha choce probables are well de ned, follows ha s bounded away from zero. Furhermore, snce = =Q equals he nverse of he average payo n he lm as! ; follows ha he assocaed mean or averaged d erenal equaon (9) assocaed wh he correced dscree me sochasc process () s: _y = y y ^ ; y ^ 0; y (2) In equlbrum hs amouns o he sandard de non of he adjused replcaor dynamcs. Ths s exremely useful because here are many resuls n he leraure on he equlbrum behavour of he adjused replcaor dynamcs (see Fudenberg and Levne 998, Hopkns 2002). We shall revs some of hese ndngs laer n hs proof of he man resul. Because each vares over me, we requre a furher se of N equaons descrbng he dscree me sochasc process of. Usng he mehod we prevously employed o wre player s expeced sraegy adjusmen of gong o he bar, we now nd he expeced change player s sep sze. Lemma Gven he choce rule (5) and he updang rule (6), he expeced moon of he h player s sep sze n he El Farol game s: E +j = 2 ^ 0; y + 2 y ^ 0; y ^ + y ; y + O [ ] 2 (3) Proof Frs, magne ha player chooses o aend he bar n perod. The expeced change n he player sep sze can be wren as: + + () = Q + ^ ; y Q = ^ ; y + O ( ) 7

19 Now consder he expeced change n sep sze f player says a home. + + () = Q + ^ 0; y Q = ^ 0; y + O ( ) The expeced moon of each player s sep sze gven can now be wren as he expeced moon n he sep sze gven y mes he sep sze n perod. E +j () () = y ^ ; y + O ( ) + y ^ 0; y + O ( ) and afer some more algebrac manpulaon we arrve a (3). The mean or averaged d erenal equaon derved from he dscree me sochasc process (7) has now been correced for he endogenous and non-scalar sep sze. Therefore, we have he followng mean or averaged d erenal sysem conssng of 2N d erenal equaons wh 2N endogenous varables: _y = y y ^ ; y ^ 0; y (4a) _ = ^ 0; y + y ^ 0; y ^ ; y (4b) Le us refer o hs as he jon dynamc sysem descrbng he evoluon of player s sraegy adjusmen of gong o he bar n he El Farol game Admsson of a Src Lyapunov Funcon We mus show ha he assocaed mean or averaged ordnary d erenal sysem, he jon dynamc sysem (4), adms a src Lyapunov funcon. Le us begn wh some de nons. De non 4 Le (9) be an ordnary d erenal equaon de ned on some subse Y of R N. Le V : Y! R be a connuously d erenable funcon. Furhermore, le y be a xed pon of V (y). V (y) s a Lyapunov funcon f, _V (y) 0; 8 y 2 Y and (5a) _V (y) = 0; 8 y 2 (5b) where s he se of xed pons of (9). De non 5 A src Lyapunov funcon s a Lyapunov funcon V (y) such ha: _V (y) > 0; 8 y =2 (6) In general can be d cul and me consumng o denfy a suable Lyapunov funcon for a parcular sysem. I s ofen a process of ral and error. An approach o hs aspec of he problem developed n he exsng leraure on he convergence of learnng models n games (see Du y and Hopkns 2005) 8

20 has been o explcly derve a suable funcon for V (y) and hen show ha adms a src Lyapunov funcon. In heory, bu no always n pracce, hs can be accomplshed by rs assumng ha V (y) ndeed adms a src Lyapunov funcon. If hs s he case, hen he paral (y) =@y represens he expeced payo ncremen o player from gong o he = ^ ; y ^ 0; y (7) I should hen jus be a queson of (y) =@y wh respec o y n order o nd a suable V (y) and checkng ha boh condons (5) and (6) de nng src Lyapunov funcons are me. The d culy wh hs approach s n explcly ndng a funcon V (y). Expressng ^ ; y ^ 0; y n a compac form s no as sraghforward as one mgh rs hope. Ths can be demonsraed by (y) =@y furher. Noe ha (7) can be expressed as: = E j = E j = 0 = CX j=0 = [B S] + [G B] Pr m = j (8) s he probably ha C players or less of he remanng N players choose o go o he El Farol bar. I s wrng ou hs laer probably expresson (8) for ha s unforunaely problemac and can ge cumbersome very quckly. Therefore, hs urns ou o be an nracable mehod of demonsrang ha he jon dynamc sysem (4) adms a src Lyapunov funcon. An alernave approach s o employ a resul of Monderer and Shapley (996, Theorem 3.) from he heory of poenal games o demonsrae ha he jon sysem (4) adms a src Lyapunov funcon. The argumen s as follows: he El Farol game s a congeson game herefore s a poenal game and hus adms a poenal funcon. The properes of poenal funcons are smlar o hose of src Lyapunov funcons and herefore, follows ha he jon dynamc sysem (4) adms a src Lyapunov funcon. Le us now begn wh some de nons and a resang of Monderer and Shapley (996, Theorem 3.). De non 6 Le N; Y ; be a game n sraegc form. s called a poenal game f adms a poenal funcon. De non 7 A funcon P : Y! R s a poenal funcon for 2 N and for every y 2 Y, f for every x; y x 0 ; y = P x; y P x 0 ; y 8 x; x 0 2 Y Theorem 3 (Monderer and Shapley (996, Theorem 3.)) Every congeson game s a poenal game. 9

21 Now we can show ha he jon dynamc sysem (4) adms a src Lyapunov funcon. Lemma 2 The jon dynamc sysem (4) adms a src Lyapunov funcon. Proof The El Farol sage game s a congeson game and herefore by Theorem 3, Monderer and Shapley (996, Theorem 3.), s a poenal game. Thus, here exss a funcon P :! R for every 2 N and for every 2 such ha: ; 0; = P ; P 0; 8 2 = f0; g Gven ha here s a connuous se of mxed sraeges, we can wre he poenal funcon P (y) as a smooh funcon wh respec o he sraegy space y 2 [0; ] N. P (y) s herefore connuously d erenable. Therefore, for every 2 N and for every x 2 [0; ] N, x; y x 0 ; y = P x; y P x 0 ; y 8 x; x 0 2 [0; ] Now choose x and x 0 equal o 0 and respecvely and ake expecaons of boh sdes. I follows ha for every 2 N and for every y 2 [0; ] N, ^ ; y ^ 0; y = P ; y P 0; y Or oherwse = ^ ; y ^ 0; y (9) Furhermore, P _ dp (y) (y) = dy _y = ^ ; y ^ 0; y _y = y y ^ ; y ^ 0; y 2 0 By assumpon, > 0 and y 2 [0; ]. Snce ^ ; y ^ 0; y 2 0 we have ha P _ (y) s non negave. Addonally, a any xed pon y 2 eher y = 0, y = 0 or ^ ; y ^ 0; y = 0. Thus P (y) adms a Lyapunov funcon. A any y =2, _y 6= 0. I should be obvous ha: _ P (y) = y y ^ ; y ^ 0; y 2 > 0. Therefore, P (y) adms a src Lyapunov funcon. I follows ha he jon dynamc sysem (4) adms a src Lyapunov funcon Fxed Pons of he Jon Dynamc Sysem De non 8 The xed pons of he jon dynamc sysem (4) are de ned as x = (y; ) such ha _y = 0 and _ = 0. 20

22 Lemma 3 The jon dynamc sysem (4) possesses a ne number of solaed xed pons. Proof Consder he jon dynamc sysem (4). The xed pons of he N equaons descrbng he evoluon of he sep sze occur when eher: = 0; ^ 0; y ^ + y 0; y ^ ; y By assumpon, all payo s are posve herefore s bounded away from zero. Ths means ha he xed pons of he jon dynamc sysem (4) wh = 0 are always unsable (see Hopkns 2002, Du y and Hopkns 2005) and herefore are never asympoc oucomes. We can hus concenrae on he laer case. Consder he rs N equaons of he jon dynamc sysem (4). Once we subsue for and mulply boh sdes by he denomnaor we have: y y ^ ; y ^ 0; y = 0 In oher words he xed pons of he jon dynamc sysem (4) are exacly he same as hose under he adjused replcaor dynamcs (2) and, consequenly, he sandard replcaor dynamcs (0). The characersaon of he xed pon of he sandard replcaor dynamcs (0) s well known (see Webull 995) and consss of he unon of all pure saes and Nash equlbra of he underlyng game. The number of pure saes s obvously ne and, as proved n Proposons 2-4, he number of Nash equlbra n he underlyng El Farol game s counable. Therefore, he jon dynamc sysem (4) possesses a ne number of xed pons pons. Jus o be absoluely clear, he xed pons of he jon dynamc sysem (4) conss of he followng: Pure sraegy Nash equlbra These are he pure saes of he jon dynamc sysem (4) ha correspond o he pure sraegy Nash equlbra of he underlyng game. Symmerc mxed sraegy Nash equlbrum Ths s he full neror sae of he jon dynamc sysem (4) ha corresponds o he symmerc mxed sraegy Nash equlbra of he underlyng game. Tha s, he Nash equlbrum where all players play a unque mxed sraegy bes response. Asymmerc mxed sraegy Nash equlbra These are boundary saes of he jon dynamc sysem (4) ha correspond o asymmerc mxed sraegy Nash equlbra of he underlyng game. By boundary saes we mean hose where a subse of he N players play a unque mxed sraegy bes response whle he remander play a pure sraegy. Fxed pons ha are no Nash equlbra No all xed pons of he jon dynamc sysem (4) correspond o Nash equlbra of he underlyng game. There are pure saes of he 2

23 jon dynamc sysem (4) ha do no correspond o pure sraegy Nash equlbra of he underlyng game. Noe ha s no possble o have neror xed pons or xed pons on some boundary of he sae space of he jon dynamc sysem (4) ha do no correspond o Nash equlbra of he underlyng game Posve Convergence Resul Proposon 6 The dscree me sochasc process (7) converges wh probably one o one of he xed pons of he sandard replcaor dynamcs (0). Proof By Lemma 2 he jon dynamc sysem (4) adms a src Lyapunov funcon. By Lemma 3 he jon dynamc sysem (4) possesses a ne number of xed pons whch are dencal o hose of he sandard replcaor dynamcs (0). Therefore, by Theorem 2, Benaïm (999, Corollary 6.6), he dscree me sochasc process (7) converges o one of he xed pons of he sandard replcaor dynamcs (0). 4.2 Proof of Man Resul: Second Sage In he second par of he proof of he man resul, we show ha he dscree me sochasc process (7) does no converge o any equlbra correspondng o Nash equlbra of he underlyng game whch are unsable under he adjused replcaor dynamcs (2) or equlbra ha do no correspondng o a Nash of he underlyng game. We ackle hs n wo seps. Frs, we show ha he sably properes of a xed pon of he jon dynamc sysem (4) are enrely deermned by he sably properes of he correspondng xed pon under he adjused replcaor dynamcs (2). We hen deermne he sably properes of he Nash equlbra under he adjused replcaor dynamcs (2). We conclude ha only he pure sraegy Nash equlbra are sable under he adjused replcaor dynamcs (2). Fnally, we employ Hopkns and Posch (2005, Proposon 2) o show ha he dscree me sochasc process (7) canno converge o any xed pons unsable under he adjused replcaor dynamcs (2). Second, we employ Hopkns and Posch (2005, Proposon 3) o demonsrae ha he dscree me sochasc process (7) canno converge o any xed pon no correspondng o a Nash equlbra under he underlyng game. Therefore, we have our negave convergence resul Unsable Equlbra n he Adjused Replcaor Dynamcs De non 9 A xed pon x = (y; ) of he jon dynamc sysem (4) s unsable f s lnearsaon evaluaed a x has a leas one egenvalue wh a posve real par. Theorem 4 (Hopkns and Posch (2005, Proposon 2)) Le x be a Nash equlbrum ha s lnearly unsable under he adjused replcaor dynamcs (2). Then he Erev and Roh (998) renforcemen learnng process de ned by he choce rule (5) and he updang rule (6) asympocally converges o one of hese pons wh probably zero. 22