The Importance of Action History in Decision Making and Reinforcement Learning

Transcription

1 Th Importanc of Action History in Dcision Making and Rinforcmnt Larning Yongjia Wang Univrsity of Michigan, 2260 Hayward Strt Ann Arbor, MI John E. Laird Univrsity of Michigan, 2260 Hayward Strt Ann Arbor, MI Abstract W invstigat th hypothsis that historical information plays an important rol in larning action slction via rinforcmnt larning. In particular, w considr th valu of th history of prior actions in th classic T maz of Tolman and Honzik (Tolman & Honzik W show that including a squnc of actions in th stat maks it possibl to larn th task using rinforcmnt larning. Morovr w show that larning ovr squncs of lngth 0 ~ 4 is ncssary to modl rat bhavior. This bhavior is modld in Soar-RL and compard to an arlir modl cratd in ACT-R. Introduction In many task immdiat snsory data is insufficint for dcision making. Enriching th stat with information about prvious actions or prvious situations can disambiguat btwn situations that would othrwis appar idntical, which maks it possibl not only to mak corrct dcisions but also to larn th corrct dcision. Morovr, knowldg of th past can rplac th nd for unralistic snsor such as knowing th xact location in a maz. Using historical information as part of th stat rprsntation poss som challngs. For th tasks w dscrib hr, w us a simplifid vrsion of history a squnc of prior actions. This lavs opn th lngth of squnc, and how to modl th rlation btwn similar squncs to achiv propr lvl of gnralization and spcialization during larning. W dmonstrat how ths issus can b addrssd in Soar-RL (Nason, & Laird, 2005 by proposing a simpl modl on an animal basd xprimnt. W analyz th task and compar rsults to a rcnt ACT-R modl (Fu & Andrson Th T Maz Task Th task w will xplor is th T maz task of Tolman and Honzik (Tolman & Hoznik 1930 in which a rat is put at th start location and it is rwardd if it gts to th nd location. As shown in Figur 1, th T maz contains 14 numbrd blinds (dad-nds, ach corrsponds to a binary choic point (th task is dsignd to prohibit th rats from going back at T-junctions. Whnvr th rat turns into a dadnd, that is considrd an rror. In such a maz, thr ar fw if any salint faturs. Rats ar abl to maintain a sns of dirction, so that would provid th ability to crat for diffrnt classs of T s. Th only othr salint faturs appar to b a history of th rat s bhavior that is th squnc of turns it mad bfor coming to a T at which it must mak a dcision. Figur 1. T maz usd in Tolman and Honzik (1930 To cast this as a problm conduciv to rinforcmnt larning, w us th sam convntions as a rcnt ACT-R modl on this task (Fu & Andrson Moving into dad-nds and turning back rsults in immdiat ngativ rward, whil raching th final goal rsults in positiv rward. Figur 2 shows a pictur of th actual nvironmnt, whr th maz is mbddd in a grid world and th subjct movs on unit at a tim. Dark boxs rprsnt pnalti and th light box rprsnts th final rward.

2 Figur 2. T-maz modl Qualitativ Analysis of Task Constraints Givn th darth of faturs in th nvironmnt, th only xtrnal faturs availabl to th rat ar its prior movs. Thu w assum th rprsntation of th stat includs a squnc of prvious movs. Th movs could b ncodd rlativ to th currnt hading: lft, right, forward, backward; howvr, as pointd out in (Fu & Andrson 2006, th rats hav strong dirctional bia and thus w assum thy hav knowldg of absolut dirction and hav availabl th absolut dirctions of thir movmnt To dscrib th modl, w us north, ast, south and wst as labls for ths dirctions. For xampl, at choic point 6, th stat includs th squnc of [ast, north, wst, ] ordrd lft-to-right by rcncy, so that th first itm in th squnc is th currnt dirction. Figur 3 shows th rlationships among th choic points associatd with ach numbrd dad-nd basd on th squnc rprsntation dscribd arlir. Choic points that ar groupd togthr hav th sam prvious input squnc and fac with th sam st of choics. Within th sam group, points ar furthr dividd basd on what is th corrct choic. Dcision point for which moving north (2, 4, 6 or moving wst (3, 11 ar corrct, ar colord in light numbr with dark background; othr points ar colord in dark numbr with light background. Points in th sam group but with diffrnt color ar compting points in that larning to rduc th rror for on typ of points will simultanously incras th rror for th othr typ of points. Intrfrnc is most intns for th most gnral lvl (Sq 0, and disappars at th most spcific lvl (Sq 4, whr th corrct dcision can b larnd for ach choic point. Th tr structur in Figur 3 thrfor capturs all such constraints in th task modl. Figur 3. Rlations among choic points Our hypothsis is that choic points with similar stat rprsntations (in this cas th squnc of prior movs will appar similar to th rat and it will larn to mak th sam dcisions in thos stats. Choic points with diffrnt corrct dirctions but similar stat rprsntations will intrfr with ach othr during larning. According to Figur 3, if th agnt maks dcision basd on Sq 0, for xampl, it will tnd to mov south mor than north and ast mor than wst at ach choic point whr thos options ar availabl, sinc south and ast corrspond to th corrct choic for th majority of th choic points within ach group (4 south vs. 3 north and 5 ast vs. 2 wst. At Sq 4 (th most spcific lvl, all choic points ar compltly discriminatd and th corrct dcision can b mad at ach choic point. Our assumption is that squncs of prior actions ar maintaind and availabl for dcision making. Figur 3 provids th information ncssary to dtrmin what impact ach squnc can hav on larning. Rlying solly on squncs of lngth 0, a rat should tnd to mak mor rrors at points 2, 4, 6, 3, and 11. Rlying solly on squncs of lngth 1, point 4 should involv lss rror than point 2 and 6, sinc point 4 is discriminatd from majority of conflicting points (spcially th strongst point 14 but only intrfr with point 12. Point 4 will b corrctly larnd at th nxt spcificity lvl, whil point 2 and 6 ar still confusd with point 8. Point 3 will involv mor rrors than point 11 sinc it is not discriminatd from point 7 until squnc lngth of 4. On important proprty of most approachs to larning ths discriminations is that larning is quickr for mor gnral lvls bcaus thy ar xposd to mor xampls. For xampl, thr ar 4 diffrnt ruls (diffrnt combinations of stats and lgal actions at th lvl of sq 0, ach of thm will rciv a quartr of th total training instanc whil at th most spcific lvl of Sq 4, thr ar

3 28 diffrnt rul ach of thm only rcivs lss than 4% of total training instancs. This suggst thr is an advantag to including slction knowldg basd on all lvls of th squncs so that som rough knowldg can com into play arly, but mor and mor spcific knowldg is larnd ovr tim. No dlibrat mchanism is rquird to achiv this ffct. Ths conclusions ar largly consistnt with th xprimntal data from th T-maz task as shown in Figur 4. Figur 4. Prcntag rror in Honznik (1930 Soar Rinforcmnt Larning Modl As mntiond abov, our hypothsis is that th modl must considr th spctrum of spcificity lvls of th stat rprsntations and that ths will influnc larning and bhavior. In Soar, this ffct can b radily modld bcaus Soar allows knowldg for slction of an action to b ncodd in multipl ruls that fir togthr in paralll, ach providing its own prdiction of th xpctd utility of th oprator. Th xpctd utilitis for th sam oprator ar combind, producing a singl, joint xpctd probability. Thu whn making a dcision, ruls match and fir for ach of th lvl for ach of th possibl actions. Thu w can captur all of th lvls of spcificity in Figur 3. Onc a dcision is mad, all th ruls that contributd to th slctd action updat thir xpctd utility valus. Th ffct is that gnral ruls will hav th most influnc for dcisions at novl situations whr spcific rul hasn t bn larnd yt. In ths situation th xpctd valus cratd by th spcific ruls will b rlativly wak with valus still clos to th initial valu of 0. As larning progrss mor and mor of th spcific ruls will hav sufficint xampls so that thir larning stabilizs and thir valu combind with corrsponding gnral rul rflct th xpctd utility of thos situations. Soar-RL Soar rinforcmnt larning implmnts th gnral tmporal-diffrnc larning. Th larnd policy is rprsntd as a Q valu function as in standard Q larning. A Q valu rflcts th utility of taking a particular action in a particular stat. In Soar-RL, a Q valu is associatd with ach stat-action pair rprsntd as a Soar RL production rul. Th updat function in th cas of multipl ruls firing is as th following. A tmporal diffrnc is computd basd on th sum of Q valus for all ruls that match th currnt condition, and is vnly distributd to updat ach rul. Sinc mor gnral rinforcmnt larning ruls fir mor oftn, and a spcific rul will always fir with th sam gnral rul (thr is a strict hirarchy in this task, th rsult is that th gnral rul quickly larns gnralizd Q valu with rlativly fwr training whil spcific ruls will fin tun th total Q valu for spcific situations and stabiliz aftr rciving mor training xampls. Without gnral rul th modl has to dirctly larn th spcific ruls without usful initial bias in novl situations whr mor gnral rul could hav hlpd. Without spcific rul on th othr hand, it cannot larn th prcis policy. Th probability of making a particular choic is calculatd basd on th Boltzmann distribution (quation 1. In th binary choic cas of this task modl, it can b rwrittn as quation 2, thrfor th probability of making th wrong choic P wrong is a monotonic function of th Q valu diffrnc quantity Q( a wrong Q( a corrct. Hr th Q valu rprsntd as a function of a stat-action pair, whr a wrong stands for th wrong action and a corrct stands for th corrct action. P i = Q( ai i Q( ai Tmpratur Tmpratur Q( a1 Q( a2 Tmpratur P 1 Q( a1 Q( a2 Tmpratur 1+ Equation 1 = Equation 2

4 Figur 5. Effcts of rinforcmnt larning ruls with stat rprsntation at diffrnt spcificity lvls Figur 5 plots th Q valu diffrnc = Q( a wrong Q( a corrct at ach choic point for rinforcmnt larning ruls with diffrnt spcificity lvl (from Sq 0 to Sq 4. Th Q valus ar larnd sparatly and ach is an avrag from 10 indpndnt simulations for 17 trials. Ths Q valu diffrnc curvs show th convrgncs trnds for ruls at diffrnt spcificity lvl. Th plot qualitativly illustrats how ruls at ach spcificity lvl will affct th rlativ rror rat shown in Figur 4. Th initial rror rat distribution should b similar to th curv Sq 0, but as mor and mor spcific dcisions ar larnd it vntually convrgs to th curv of Sq 4, th most spcific lvl, as xplaind in th analysis prsntd in th prvious sction. Th plot can b viwd approximatly as a contour of Q valu diffrnc updating dynamic sinc whn all lvls of ruls ar usd in Soar, th total Q valu diffrnc will gradually convrg following th path which is consistnt with our mpirical rsults (data not shown. On spcific intrprtation from Figur 5 is that initial rror for point 4 is rlativly highr than point 3, but it larns fastr and rsults in lowr total rror rat. Qualitativly, th avrag Q valu diffrnc across all spcificity lvl which is shown as a bold curv in Figur 5 approximats th rlativ total rror rats for ach dad-nd. This can b confirmd by comparing with Figur 4. Figur 5 only shows th qualitativly analysis basd on sparat simulations of ach individual lvl. It is mor informativ to xamin th combind Q valu diffrnc of all ruls during larning. (a (b Figur 6. Chang of combind Q valu diffrnc during larning. Th numbrs in Figur 6 rfr to trial with 20 trial intrvals. For xampl, th curv with 1 rprsnts th Q valu diffrnc aftr trial 1, 3 rprsnts aftr 21 trials. Thr ar totally 81 trials shown in th plot to dmonstrat th Q valu dynamic although th actual rat xprimnt only taks 17 trials. (a is larning with only th most gnral ruls and th most spcific ruls. (b is larning with all lvls of ruls. On of th main diffrncs btwn (a and (b is point 3 is larnd rlativly slowly whn using all lvls of ruls. Th dynamics of larning is consistnt with Figur 5 and th abov analysis. Rsults

5 Figur 7 compars obsrvd data with prdiction using all 4 lvls of ruls. Th paramtrs ar pnalty for turning back -20, rward for raching th goal +100, larning rat 0.1, linar discount of 10, on-policy larning with Boltzmann xploration tmpratur 3. Linar discount is usd bcaus it givs bttr rsults than standard xponntial discount. Figur 5 and Figur 6 ar gnratd using standard xponntial discount and thy ar for illustration purposs. Th most important paramtr is th larning rat, and th rsults ar not vry snsitiv to othr paramtrs. (a Figur 7. Soar modl prdiction Comparison with ACT-R An ACT-R modl (Fu & Andrson 2006 was dvlopd to modl th Tolman and Honzik (1930 xprimnt, rlying on ACT-R s nativ rinforcmnt larning componnt. In ACT-R, thr ar wights associatd with ruls. Larning adjusts thos wight which ar usd in slction. Each rul corrsponds to on action and thr is no xplicit combination of valus or joint updating of ruls that ar for th sam action. Th ACT-R modl uss two sts of ruls: a st of twnty-ight spcific rul two for ach choic point; and a st of four gnral rul on for ach absolut dirction. Th spcific rul st is quivalnt to th lvl of sq. 4 (th modl dos not us squnc assuming a rat knows its position in th maz, and th gnral rul st is quivalnt to sq. 0 in Figur 3. (b Figur 8. (a Prdiction using ACT-R modl. (b Prdiction using Soar modl with quivalnt ruls. Figur 8 (a shows th ACT-R prdiction with only th most gnral ruls and most spcific rul which is quivalnt to using only Sq 0 and Sq 4 in th Soar modl. Figur 8 (b shows th prdiction using Soar 0, 4 modl, which is similar to th ACT-R modl spcially for blind 3. Tabl 1. Corrlation Matrix comparing all modls Obsrvd Soar0~4 Soar 0,4 ACT-R Obsrvd Soar 0~ Soar 0, ACT-R

6 Tabl 1 compars th corrlations of th ACT-R modl and Soar modl. Th Soar 0, 4 modl prdicts th ACT-R modl vry wll (corrlation 0.95, whil Soar 0~4 modl prdicts th xprimntal data bttr (corrlation 0.91 than th othr modls. Th diffrncs btwn th corrlation cofficints ar statistically significant. For 0.86 vrsus 0.91, th p valu is < assuming th original rat data and ACT-R data both hav th sam varianc as our simulation data. This (wakly suggsts that th rats larn to mak dcisions using a history of prior dcisions. Tabl 2. Corrlation with partial obsrvd data Soar0~4 Soar 0,4 ACT-R Blinds 1~ Blinds 10~ Taking a closr look at th rsult Soar 0~4 matchs th blinds closr to th bginning much bttr whil th Soar 0,4 modl matchs wll for thos closr to th nd. Tabl 2 compars th corrlation with partial xprimntal data. On hypothsis could b that whn th rat is at th choic points clos to th nd, th gnral ruls giv good rsult so that it lss dpnds on lowr lvl ruls. Whil at th bginning, whr th gnral ruls giv bad rsult it also uss th mor spcific ruls. This hypothsis suggsts adjusting th wights of ruls at diffrnt lvls for diffrnt choic point but that introducs mor paramtrs and is probably byond what can b confirmd from th availabl data. Tabl 3. Comparison btwn th modls Modl Lvl Architctural Lvl Soar Us action history Paralll rul firing ACT-R No action history Singl rul firing Tabl 3 compars th two lvls of diffrnc btwn th ACT-R modl and our Soar modl. Clarly th most important diffrnc is our modl s larning ovr multipl squncs of past actions (th modl lvl diffrnc. It is rasonabl to assum that rprsnting that information in th stat and incrasing th numbr of ruls in ACT-R would improv th ACT-R modl s match to th obsrvd data, spcially for th arly choic points. A dtaild comparison btwn rinforcmnt larning in ACT-R and Soar has alrady bn mad by Nason (Nason & Laird Howvr, this task modl highlights an important diffrnc btwn th two approachs. In Soar, for a singl dcision, multipl rinforcmnt larning ruls ar allowd to contribut to th dcision making and thn ar updatd by larning. In ACT-R, although multipl ruls contribut to making a dcision through comptition, only on is pickd and updatd. Soar spds larning with multipl rinforcmnt larning ruls in trms of rquiring fwr xtrnal action although th asymptotic bhavior of th two approachs should b similar. This architctural lvl diffrnc is scondary for th rsults prsntd hr it is th action history rprsntation (modl lvl diffrnc that maks th qualitativly diffrnt prdictions in our hypothsis. Howvr, it may b worthwhil to xplor th importanc of this architctural lvl diffrnc in othr applications. Anothr diffrnc is th rward discount functions usd in Soar and ACT-R. Th dfault option in Soar is to multiply futur xpctd rward with a discount factor γ (0 < γ <1 in th stp-wis updat function, which rsults in xponntial dcay of rwards. W xprimntd with linar discount (constant discount btwn stps which gnrats slightly bttr rsults. Th compard ACT-R modl uss a hyprbolic discount function, which might hlp our modl to mak bttr prdictions spcially for thos latr choic points. In gnral, it s flxibl to xprimnt with diffrnt options in th Soar architctur. Discussions Th major contributions of this papr ar to xamin th contribution of squncs of action histori to dcisionmaking and larning. Th scond major contribution was to valuat th approach to rprsntation and updating of xpctd valus in Soar-RL and discovring that thy provid an accurat modl of larning dynamics by having ovrlapping ruls at diffrnt spcificity lvls. Functionally, larning progrsss from gnraliz to spcific. Th ACT-R modl providd a usful bnchmark for comparison. W can also ask whr our modl falls short. Our modl dos not mak a good prdiction at blind 12. This could b du to xprimntal data noi but it s mor likly that thr is mor structur in th task that is not capturd by our modl. On possibility can b that instad of always using absolut dirction th rats may actually us combinations of absolut and rlativ dirction such as turning lft and right as th stat ncoding stratgy. Rfrncs Fu, Wai-Tat & Andrson, J. R (2006. From Rcurrnt Choic to Skill Larning: A Rinforcmnt-Larning Modl. Journal of Exprimntal Psychology: Gnral, 135( Nason, S., Laird, J. E., (2005 Soar-RL: Intgrating Rinforcmnt Larning with Soar, Cognitiv Systm Volum 6, Issu 1, pp Nuxoll A.& Laird J.E. (2004. A Cognitiv Modl of Episodic Mmory Intgratd with a Gnral Cognitiv Architctur, Intrnational Confrnc on Cognitiv Modling. Tolman E. C. & Honzik, C. H. (1930. Dgrs of hungr, rward and non-rward, and maz larning in rat Univrsity of California Publications in Psuchology, 4(16,