Learning in Games. Teck H. Ho. University of California, Berkeley. Joint work with Colin Camerer and Juin-Kuan Chong.

Transcription

1 Unversy of Calforna, erkeley Jon work wh Coln Camerer and Jun-Kuan Chong Sepember,

2 Oulne Research Queson Crera of a Good Model Experence Weghed Aracon EWA) Learnng Model Choce Renforcemen Weghed Fcous Play Self-runnng EWA Learnng Sepember,

3 Medan-Effor Game Sepember,

4 Medan-Effor Game Fgure 1a: Acual choce frequences Perod Sraegy Van Huyck, aalo, and el 1990) Sepember,

5 The learnng seng In games where each player s aware of he payoff able Player s sraegy space consss of dscree choces ndexed by j e.g., 1, 2,, 7) The game s repeaed for several rounds A each round, all players observed: Sraegy or acon hsory of all oher players Own payoff hsory Sepember,

6 Research queson To develop a good descrpve model of adapve learnng o predc he probably of player 1,,n) choosng sraegy j a round n any game P j ) Sepember,

7 Crera of a good model Use poenally) all avalable nformaon subjecs receve n a sensble way Sasfes plausble prncples of behavor.e., conformy wh oher scences such as psychology) Fs and predcs choce behavor well Ably o generae new nsghs As smple as he daa allow Sepember,

8 Models of Learnng Inrospecon P j ): Requres oo much human cognon Nash equlbrum Nash, 1950) Quanal response equlbrum Nash-λ) McKelvey and Palfrey, 1995) Evoluon P j ) ): Players are pre-programmed Replcaor dynamcs Fredman, 1991) Genec algorhm Ho, 1996) Learnng P j ) ): Uses abou he rgh level of cognon Experence-weghed aracon learnng Camerer and Ho, 1999) Renforcemen Roh and Erev, 1995) elef-based learnng Courno bes-response dynamcs Courno, 1838) Smple Fcous Play rown, 1951) Weghed Fcous Play Fudenberg and Levne, 1998) Dreconal learnng Selen, 1991) Sepember,

9 Informaon Usage n Learnng Choce renforcemen learnng Thorndke, 1911; ush and Moseller, 1955; Herrnsen, 1970; Arhur, 1991; Erev and Roh, 1998): successful sraeges played agan elef-based Learnng Courno, 1838; rown, 1951; Fudenberg and Levne 1998): form belefs based on opponens acon hsory and choose accordng o expeced payoffs The nformaon used by renforcemen learnng s own payoff hsory and by belef-based models s opponens acon hsory EWA uses boh knds of nformaon Sepember,

10 Laws of Effecs n Learnng T M L R 8 10 Row player s payoff able 9 Coln chose and receved 4 Teck chose M and receved 5 Ther opponens chose L Law of acual effec: successes ncrease he probably of chosen sraeges Teck s more lkely han Coln o sck o hs prevous choce oher hngs beng equal) Law of smulaed effec: sraeges wh smulaed successes wll be chosen more ofen Coln s more lkely o swch o T han M Law of dmnshng effec: Incremenal effec of renforcemen dmnshes over me $1 has more mpac n round 2 han n round 7. Sepember,

11 Assumpons of Renforcemen and elef Learnng Renforcemen learnng gnores smulaed effec elef learnng predcs acual and smulaed effecs are equally srong EWA learnng allows for a posve and smaller han acual) smulaed effec Sepember,

12 Sepember, The EWA Model 0) 0), N A j Inal aracons and experence.e., ) Updang rules Choce probables 1 1 ) ) ) )), 1) 1) ) ) )), 1) 1) ) + ) + + Ν N s s N s s A N s s N s s A N A j j j j j j j ρ π δ φ π φ + k j m k A A j e e P 1 ) ) 1) λ λ Camerer and Ho Economerca, 1999)

13 EWA Model and Laws of Effecs A j ) φ N φ N 1) 1) A A j j 1) + π s N ) 1) + δ π s N ) j, s j )), s )) s s j j s s ) ) N ) ρ Ν 1) + 1 Law of acual effec: successes ncrease he probably of chosen sraeges posve ncremenal renforcemen ncreases aracon and hence probably) Law of smulaed effec: sraeges wh smulaed successes wll be chosen more ofen δ > 0 ) Law of dmnshng effec: Incremenal effec of renforcemen dmnshes over me N) > N-1) ) Sepember,

14 The EWA model: An Example L R T Hsory: Perod 1,L) Row player s payoff able Perod 0: A T 0), A 0) Perod 1: A T 1) φ A T 0 ) ρ N N 0 ) + 0 ) + 1 δ 8 A 1) φ A 0 ) N ρ N 0 ) 0 ) Sepember,

15 Renforcemen Model: An Example L R T Hsory: Perod 1,L) Row player s payoff able Perod 0: R T 0), R 0) Perod 1: R R T 1) 1) φ R φ R T 0 ) 0 ) + 4 If δ 0, ρ EWA RF 0, A A T 1 ) φ 1 ) φ N 0) 1, A T 0 ) ρ N N 0 ) + 0 ) + 1 A 0 ) N 0 ) + ρ N 0 ) + 1 δ 4 8 Sepember,

16 elef-based) model: An Example T L R 8 9 Perod 0: L N 0 ) N 0 ) + Perod 1: Sepember, 2006 N R 0 ) E E E E L T L T 0) 0) 0) 1) 1) 1) 4 L N 0) N 0) ρ ρ 4 L L N N R 0) + 9 0) ) R ayesan Learnng wh Drchle prors L L 0) R R N 0) N 0) 0) N L N 0) 8 + N N 0) L R 0) 4 + N N 0) 0) 9 R 0) 10 L R 0 ) + 1 R ρ N 0 ) + 0 1) 0 ) + 1 ρ N 0 ) + 1 T R ρ E 0 ) N 0 ) + 8 1) + 9 1) ρ N 0 ) + 1 R ρ E 0 ) N 0 ) + 4 1) ) ρ N 0 )

17 Relaonshp beween elef-based ) and EWA Learnng Models EWA L 1) ρ N ρ N 0) + 1 0) + 1 L R 1) ρ N ρ N R 0) + 0 0) + 1 E T 1) 8 L 1) + 9 R 1) T ρ E 0) N 0) ρ N 0) A T 1) φ A T 0 ) ρ N N 0 ) + 0 ) + 1 δ 8 E 1) 4 L 1) + 10 R 1) ρ E 0) N ρ N 0) 0) A 1) φ A 0 ) N ρ N 0 ) 0 ) If δ EWA 1, ρ φ Sepember,

18 Model Inerpreaon Smulaon or aenon parameer δ ) : measures he degree of sensvy o foregone payoffs φ Exploaon parameer κ ): measures how rapdly φ players lock-n o a sraegy average versus cumulave) ρ Saonary or moon parameer φ ): measures players percepon of he degree of saonary of he envronmen Sepember,

19 Model Inerpreaon Weghed Fcous Play Courno Fcous Play Cumulave Renforcemen Average Renforcemen Sepember,

20 New Insgh Renforcemen and belef learnng were hough o be fundamenal dfferen for 50 years. For nsance,.n roe [renforcemen] learnng success and falure drecly nfluence he choce probables. elef learnng s very dfferen. Here experences srenghen or weaken belefs. elef learnng has only an ndrec nfluence on behavor. Selen, 1991) EWA shows ha belef and renforcemen learnng are relaed and specal knds of EWA learnng Sepember,

21 Acual versus elef-ased Model Frequences Fgure 1a: Acual choce frequences Fgure 1b: e le f-base d Mode l frequences Sraegy Perod Sraegy Perod Sepember,

22 Acual versus Renforcemen Model Frequences Fgure 1a: Acual choce frequences Fgure 1c: Choce renforcemen model frequences Perod Perod Sraegy Sraegy Sepember,

23 Acual versus EWA Model Frequences Fgure 1a: Acual choce frequences Fgure 1d: EWA model frequences Perod Perod Sraegy Sraegy Sepember,

24 Esmaon and Resuls Game Calbraon Valdaon Model No. of Parameers LL AIC IC ρ 2 LL MSD Medan AconM378) 1-segmen Random Choce Choce Renforcemen elef-based EWA * Segmen Random Choce Renforcemen elef-based EWA * * * * Sepember,

25 Sepember,

26 Sepember,

27 Exensons Heerogeney JMP, Camerer and Ho, 1999) Payoff learnng EJ, Camerer, Ho, and Wang, 2006) Sophscaon and sraegc eachng Sophscaed learnng JET, Camerer, Ho, and Chong, 2002) Repuaon buldng GE, Chong, Camerer, and Ho, 2006) EWA Le Self-unng EWA learnng) JET, Ho, Camerer, and Chong, forhcomng) Applcaons: Aucon markes ook Chaper, Camerer, Ho, and Hsa, 2000) Produc Choce a Supermarkes JMR, Ho and Chong, 2004) Sepember,

28 Research Queson Crera of a Good Model Adapve Experence Weghed Aracon EWA) Learnng Model Choce Renforcemen Weghed Fcous Play Self-runnng EWA Learnng Sepember,

29 Two Open Quesons A heory o explan why parameers vary across games Mercs for judgng model performance Sascal Measures Economc value Sepember,

30 EWA Cube Weghed Fcous Play Courno Fcous Play Cumulave Renforcemen Average Renforcemen Sepember,

31 Sepember, The EWA Model 0) 0), N A j Inal aracons and experence.e., ) Updang rules Choce probables 1 1 ) ) ) )), 1) 1) ) ) )), 1) 1) ) + ) + + Ν N s s N s s A N s s N s s A N A j j j j j j j ρ π δ φ π φ + k j m k A A j e e P 1 ) ) 1) λ λ

32 Self-Tunng EWA Model: Inalzaon Inal aracons deermned by CH Model wh τ 1.5; Camerer, Ho, Chong, 2004, QJE) N0)1 fades away quckly wh experence anyway) Se κ 0.e., φ ρ) Sepember,

33 Sepember, Self-unng EWA Model: Change-deecor funcon φ ), >1) The core of he change-deecor funcon s he surprse ndex, S ) Change-deecor funcon s: ) ) )), ) )), ) )] ) [ ) S s s I r s s I h r h S k k k k m k k k φ τ τ Dsance beween hsory and curren round hsory curren round

34 Change-deecor funcon φ ), >1): Examples If he oher player chooses he sraegy she has always chosen before, hen S ) 0, and φ ) 1. If he oher player chooses a new sraegy whch was never chosen before n a very long run of hsory, S ) 2 and φ ) 0. If a player chose he same sraegy for each of nne perods and a new sraegy n perod 10, hen S ) ) ) and φ ) If unque sraeges have been played n every perod up o -1, and anoher unque sraegy s played n perod, hen φ ) /2 ) Sepember,

35 Self-unng EWA Model: Aenon funcon δ j ), >1) Aenon funcon s: δ j ) k 1 f π s, s )) > π ) 0 oherwse Pay aenon o only sraeges ha gve srcly beer ex-pos payoffs because of lmed aenon Sepember,

36 Aenon funcon δ j ), >1): Examples If subjecs are srcly bes-respondng ex pos), hen no oher sraeges have a hgher ex-pos payoff so δ j ) 0, whch reduces he model o choce renforcemen If subjecs always choose he wors sraegy, hen δ j 1) 1, whch corresponds o weghed fcous play A naural way o formalze learnng drecon heory Creae exploraon-exploaon shf over me: sar wh poor choces and hen lock n o he bes choce Sepember,

37 Sepember,

38 Example 2: P-eauy Cones n players Every player smulaneously chooses a number from 0 o 100 Compue he group average Defne Targe Number o be 0.7 mes he group average The wnner s he player whose number s he close o he Targe Number The prze o he wnner s US$20 Sepember,

39 P-beauy coness: Acual Choces of experenced Subjecs Ho, Camerer, and Wegel AER, 1998) Sepember,

40 P-beauy coness : Predcon of EWA Sepember,

41 P-beauy coness : Predcon of Self-unng EWA Sepember,

42 Two Open Quesons A heory o explan why parameers vary across games Mercs for judgng model performance Sascal Measures Economc value Sepember,

43 Economc Value Evaluae models based on her value-added raher han sascal f Camerer and Ho, 2000) Trea models lke consulans and seek advce on wha opponens wll do If players were o hre eher Mr. Nash or Ms. EWA as consulan and lsen o hs or her advce a he begnnng of each round, would hey have made a hgher payoff? Sepember,

44 Economc Value: Defnon and Movaon Economc value of a model how much more players would earn f hey use he model o forecas wha ohers wll do, and bes-respond gven ha forecas, compared o how much hey acually earn A measure of degree of dsequlbrum, n dollar erms. If players are n equlbrum, hen an equlbrum heory wll advse hem o make he same choces hey would make anyway, and hence wll have zero economc value If players are no n equlbrum, hen players are ms-forecasng wha ohers wll do. A heory wh more accurae belefs wll have posve economc value and an equlbrum heory can have negave economc value f msleads players) Sepember,

45 Economc Value: Example L R T 8,8 6,8 8,6 12, Column's Choce R L Row's Choce T T - Acual Payoff 8 8 Model 1's Predcon of ProbR) Model 1's Recommendaon - Improvemen n Payoff 4-2 Model 2's Predcon of ProbR) Model 2's Recommendaon T - Improvemen n Payoff 4 0 Sepember,

46 Economc Value of Models Sepember,

47 Takeaways EWA cube provdes a smple bu useful framework for sudyng learnng n games. EWA model fs and predcs beer han renforcemen and belef learnng n many classes of games because allows for a posve and smaller han acual) smulaed effec. Self-unng EWA can approxmae EWA reasonably well. Economc value s an alernave measure for judgng models. Sepember,

48 Sepember,