Algorithmic models of human decision making in Gaussian multi-armed bandit problems

Transcription

1 Algorhmc models of human decson makng n Gaussan mul-armed band problems Paul Reverdy, Vabhav Srvasava and Naom E. Leonard Absrac We consder a heursc Bayesan algorhm as a model of human decson makng n mul-armed band problems wh Gaussan rewards. We derve a novel upper bound on he Gaussan nverse cumulave dsrbuon funcon and use o show ha he algorhm acheves logarhmc regre. We exend he algorhm o allow for sochasc decson makng usng Bolzmann acon selecon wh a dynamc emperaure parameer and provde a feedback rule for unng he emperaure parameer such ha he sochasc algorhm acheves logarhmc regre. The sochasc algorhm encodes many of he observed feaures of human decson makng. I. INTRODUCTION The mul-armed band problem has been exensvely suded n he machne learnng and conrols communy [3, [4, [6, [9. I s a canoncal model of decson makng under uncerany where he explore-explo radeoff s cenral. A each of a sequence of mes, he decson maker chooses one among fne opons (arms, wh unceran assocaed rewards, amng o maxmze accumulaed reward over he whole sequence. When he decson maker chooses he mos rewardng among known opons, he sraegy s called exploaon, and when he decson maker chooses a poorly known bu poenally revealng opon, he sraegy s called exploraon. Good sraeges balance exploraon o reduce uncerany and exploaon o accumulae hgh reward. In he conrols leraure, he mul-armed band problem s a model problem for adapve conrol [, [9 and has been appled o a varey of problems, ncludng mul-agen ask assgnmen [ and channel allocaon for neworks [. The performance of algorhms solvng he mul-armed band problem can be characerzed n erms of regre, whch s he accumulaed dfference beween he hghes avalable reward and he expeced reward of he algorhm. La and Robbns [0 proved ha any algorhm solvng he mularmed band problem mus ncur regre ha grows logarhmcally wh me, and hey provded an algorhm ha asympocally acheves ha bound. Snce hen, a sgnfcan lne of research has focused on provdng algorhms ha unformly acheve he La-Robbns bound. One such class of algorhms s he so-called Upper Confdence Bound (UCB algorhms, frs nroduced by Auer e al. [3. For each decson me, hese algorhms compue a heursc value for each opon whch provdes Ths research has been suppored n par by ONR gran N and ARO gran W9NG P. Reverdy s suppored hrough a NDSEG Fellowshp. Deparmen of Mechancal and Aerospace Engneerng, Prnceon Unversy, Prnceon, NJ 08544, USA {preverdy,vabhavs,naom}@prnceon.edu an upper bound for he expeced reward o be ganed by selecng ha opon: Q = µ + C, ( where µ s he expeced reward and C s a measure of uncerany n he reward of opon a me. The algorhm s decson a me s o pck he opon ha maxmzes Q. UCB, he man algorhm nroduced n [3, s desgned for he case where rewards are drawn from a dsrbuon wh bounded suppor. In hs case, Auer e al. proved ha UCB acheves logarhmc regre. They also consdered he case where rewards are drawn from a Gaussan dsrbuon wh unknown varance and nroduced an algorhm hey call UCB-Normal o solve. They analyzed he performance of UCB-Normal and showed ha acheves logarhmc regre, bu her proof reles on several conjecures abou Suden and χ random varables ha hey only verfy numercally. Lu and Zhao [ suded mul-armed band problems where he rewards are drawn from a lgh-aled dsrbuon, whch ncludes Gaussan dsrbuons wh known varance as a specal case. For such lgh-aled rewards, hey exended UCB o acheve logarhmc regre. UCB and s varans rely on frequens esmaors, and herefore canno ncorporae pror knowledge abou he rewards. Recen work n neuroscence [5 showed ha human decson makng n mul-armed band problems s conssen wh a sochasc heursc smlar o (. In he presen paper we consruc an algorhmc model of human decson makng ha formalzes he connecon beween he heurscs used by humans and he UCB algorhms. Our model uses sochasc decson makng and Bayesan esmaors o ncorporae pror knowledge abou he rewards. In he presen work, we provde a Bayesan algorhm for he Gaussan mul-armed band problem and prove ha acheves logarhmc regre n ceran cases. The algorhm s derved by applyng he deas n [7 o he case of bands wh Gaussan rewards, and addng a nose model o he decson process. Raher han followng he analyss n [7, our analyss follows Auer e al. [3 and faclaes exensons o he case of sochasc decson makng as well as o he case of he mul-armed band wh ranson coss and he graphcal mul-armed band, consdered n [4. We make use of a novel upper bound on he nverse cumulave dsrbuon funcon for he sandard Gaussan dsrbuon, whch we presen n Theorem. The bound s gher han he one used by Lu and Zhao [, and allows us o acheve a smaller leadng facor n he case of Gaussan rewards.

2 The conrbuons of hs paper are hreefold. Frs, we provde a deermnsc Bayesan algorhm ha provably acheves unform logarhmc regre n he case of Gaussan bands. Second, we show how o exend hs algorhm o employ sochasc polces whle sll achevng logarhmc regre. Thrd, we show how he sochasc algorhm can be used as a model of human decson makng n mul-armed band problems. The remander of he paper s organzed as follows. In Secon II we descrbe he mul-armed band problem. In Secon III we revew he algorhm of [7 and apply o he case of Gaussan bands. In Secon IV we analyze he fne-me properes of he model and prove ha acheves logarhmc regre n he case of deermnsc decson makng. We exend he model o he case of sochasc decson makng usng Bolzmann acon selecon wh a dynamc emperaure parameer and provde a feedback rule ha unes he emperaure parameer such ha he model agan acheves logarhmc regre. Fnally, we conclude n Secon V. II. GAUSSIAN BANDITS PROBLEM Consder a se of N opons, ermed arms n analogy wh he lever of a slo machne. A sngle-levered slo machne s ermed a one-armed band, so he case of N > opons s ofen called a mul-armed band. In he mul-armed band problem, he decson-makng agen mus choose, a each of a sequence of mes, one among N arms. Each arm has an assocaed mean reward m, whch s unknown o he agen and remans fxed for he duraon of he problem. The agen collecs rewards by choosng arm a each me =,,..., T and recevng reward r, whch s he mean reward assocaed wh he arm plus Gaussan nose: r N (m, σ r. The nose varance σ r s assumed known, e.g. from prevous observaons or known characerscs of he reward generaon process. The agen s objecve s o maxmze cumulave expeced reward by choosng a sequence of arms { }: [ T max J, J = E r = m. ( { } In hs conex exploaon refers o pckng arm whch appears o have he hghes mean a me, and exploraon refers o pckng any oher arm. Equvalenly, defnng m = max m and R = m m as he expeced regre a me, he objecve can be formulaed as mnmzng he cumulave expeced regre N J R = R = E [ n T, = where n T s he cumulave number of mes arm has been chosen up o me T and = m m s he expeced regre due o pckng arm nsead of arm. A. Bound on opmal performance La and Robbns [0 showed ha any algorhm solvng he mul-armed band problem mus choose subopmal arms a a rae ha s a leas logarhmc n me: E [ ( n T D(p p + o( log T, (3 where o( 0 as T + and D(p p := p (r log p(r p (r dr s he Kullback-Lebler dvergence beween he reward densy p of any subopmal arm and he reward densy p of he opmal arm. The bound on E [ n T mples a bound on cumulave regre J R, showng ha mus grow a leas logarhmcally wh me. In he presen case where r N (m, σ r, he Kullback- Lebler dvergence s equal o so he bound s E [ n T D(p p = σr, (4 ( σ r + o( log T. (5 The nuon s ha for a fxed value of σ r, a subopmal arm wh hgher s easer o denfy snce yelds a lower average reward. Conversely, for a fxed value of, hgher values of σ r mean ha he observed rewards are more varable, makng more dffcul o dsngush he opmal arm from he subopmal ones. B. Bayes-UCB For every probably dsrbuon f(x wh assocaed cumulave dsrbuon funcon (cdf F (x, he quanle funcon F (p nvers he cdf o provde an upper bound for he value of he random varable X f(x: Pr [ X F (p = p. (6 In hs sense, F (p s an upper confdence bound, an upper bound ha holds wh probably, or confdence level, p. The auhors of [7 consdered he mul-armed band problem from a Bayesan perspecve and suggesed usng F (p of he poseror reward dsrbuon as he heursc funcon (. The nuon s ha Q = F (p gves a bound such ha Pr [m > Q = p, so ha f p < s large, hen p s small and s unlkely ha he rue mean reward for arm s hgher han he bound. In order o be ncreasngly sure of choosng he opmal arm as me goes on, he algorhm n [7 ses p = α as a funcon of me wh α = /((log T c, so ha p s of order /. The auhors erm he resulng algorhm Bayes-UCB, and n he case ha he rewards are Bernoull dsrbued hey proved ha wh c 5 Bayes-UCB acheves he bound (3. III. THE UPPER CREDIBLE LIMIT ALGORITHM We apply Bayes-UCB o he case of bands wh Gaussan rewards of known varance σ r and erm he resulng algorhm he Upper Credble Lm algorhm, or UCL. We hen consder an exenson of UCL o a sochasc polcy by usng Bolzmann acon selecon and erm he resulng algorhm sochasc UCL.

3 A. Inference algorhm We begn by assumng ha he agen s pror dsrbuon of m (.e. he agen s nal belefs abou he mean reward values m and her covarance Σ s mulvarae Gaussan wh mean µ 0 and covarance Σ 0 : m N (µ 0, Σ 0, where µ 0 R N and Σ 0 R N N s a posve-defne marx. Noe ha hs does no assume ha he rewards are ruly descrbed by hese sascs, smply ha hese are he agen s nal belefs, nformed perhaps by prevous measuremens of he mean value and covarance. Wh hs pror, he poseror dsrbuon s also Gaussan, so he Bayesan opmal nference algorhm s lnear and can be wren down as follows. A each me, he agen selecs arm and receves a reward r. Recall ha n s defned as he number of mes he agen has seleced arm up o me, and le m be he emprcal mean reward observed for arm. Le n and m be he correspondng vecors wh componens n, m, respecvely. Then he belef sae (µ, Σ updaes as follows: Λ = dag(n σ r + Λ 0, Σ = Λ (7 dag(n µ = µ 0 + Σ σr ( m µ 0, (8 where Λ = Σ s he precson marx. As noed above, hs assumes ha he samplng nose σ r s known, e.g. from prevous observaons or known sensor characerscs. The above holds for general Σ 0 > 0, bu for smplcy of exposon we wll specalze n he followng o he case where Σ 0 = σ0i, so he agen beleves he mean rewards o be ndependen. In hs case he belef sae updae equaons smplfy o Var(m m = ( σ σ = r E [ m m where δ = σ r/σ 0. B. Quanle funcon δ + n = µ = δ µ 0 + n m δ + n, Wh he assumpon of ndependence made above, he poseror dsrbuon of he mean m a me s ( m N µ σ r, δ + n, so he ( α h quanle of he dsrbuon s gven by F ( α = µ + σ r δ + n Φ ( α, (9 where Φ (p s he nverse of he cdf of he normal dsrbuon, also known as he prob funcon. C. Decson heursc In he case of deermnsc decson makng, he decson a me s gven by maxmzng he heursc: = arg max Q, (0 where he heursc funcon s defned by he quanle (9, Q = µ σ r + Φ ( α δ + n, ( wh α = /K and K > s a consan. We exend he algorhm o he case of sochasc decson makng usng Bolzmann acon selecon, as s used n smulaed annealng [3, [8. The choce of arm s made sochascally usng a Bolzmann dsrbuon wh emperaure υ, so he probably P of pckng arm a me s gven by exp(q P = /υ N j= exp(q j /υ. In he case υ 0 + hs scheme reduces o he deermnsc scheme (0, and as υ ncreases he probably of selecng any oher arm ncreases. In hs way, Bolzmann selecon generalzes he maxmum operaon and s somemes known as he sof maxmum (or sofmax rule. In he conex of smulaed annealng, he choce of υ s known as a coolng schedule. In her classc work, Mra e al. [3 showed ha good coolng schedules for smulaed annealng ake he form υ = ν log, so we sudy coolng schedules of hs form. We choose ν usng a feedback rule on he values of he heursc funcon Q and defne he coolng schedule as υ = Q mn D, ( log where Q mn = mn j Q Q j s he mnmum gap beween he heursc funcon value for any wo pars of arms and D > 0 s a consan. We defne = 0, so ha Q mn = 0 f wo arms have nfne heursc values, and defne 0/0 =. Much of he bands leraure consders only deermnsc maxmzaon rules; for example, Bayes-UCB, as presened n [7, s a deermnsc decson rule. However, several auhors have consdered sochasc decson rules n adversaral conexs, where s advanageous o avod makng predcable decsons. See Chaper 3 of he recen revew [4 and references heren. D. Applcaon o human decson makng Human decson makng n mul-armed band problems s well modeled by a heursc smlar o ha of UCL ( and humans are sensve o he parameers of he problem [5. In parcular, boh he uncerany measure and he level of decson nose ncrease wh problem horzon T. Sochasc UCL can be used o model human decson makng. By choosng he parameers K and D as ncreasng

4 Quanle value Fg.. Depcon of he normal quanle funcon Φ ( α (sold lne and he bounds (3 and (4 (dashed lnes. funcons of he horzon T, he sochasc UCL algorhm presened here capures he effec of he horzon and oher mporan feaures of human decson makng n mul-armed band problems, as suded n more deal n [4. IV. REGRET ANALYSIS In hs secon we frs consder UCL and bound s cumulave expeced regre. We show he bound s logarhmc n horzon lengh T wh proporonaly consan whn a consan facor of he bes possble bound σr/ (cf. (5. We hen consder he case of sochasc UCL where he coolng schedule follows ( and show ha he regre s agan bounded by a logarhmc funcon of he horzon lengh T. A. Deermnsc decson makng Before analyzng he regre of our model n he case of deermnsc decson makng, we sae he followng bounds on he values of he normal quanle funcon Φ ( α. Theorem (Bounds on he Gaussan nverse cdf: The followng bounds hold when α < / π and β.0: α Φ ( α < β log( (πα log(πα (3 Φ ( α > log(πα ( log(πα. (4 Fan [5 posed hese bounds (whou he facor β n (3 as conjecures whou proof. In fac, he facor β s necessary o ge a correc upper bound, as we prove n he Appendx. See Fgure for a vsual depcon of he bounds. Turnng o he regre analyss of he UCL algorhm, we consder he case of an unnformave pror,.e., σ0 +. In he case of an unnformave pror and seng K =, he followng performance bound holds wh β =.0: Theorem (Regre for deermnsc decson makng: Le β =.0. The expeced number of draws of any sub-opmal arm s bounded by E [ n T ( 8β σ r + log T + 4β σr ( log log log T + +. Proof: In he spr of [3, we bound n T n T = ( = ( Q > Q η + ( Q > Q, n( η, as follows: where η s some posve neger and (x s he ndcaor funcon, wh (x = f x s a rue saemen and 0 oherwse. A me, he agen pcks arm over only f Q Q. Ths s rue when a leas one of he followng holds: where C = µ m C (5 µ m + C (6 m < m + C (7 σr Φ ( α. Oherwse, f none of he δ +n equaons (5-(7 holds, Q = µ + C > m m + C > µ + C = Q, and arm s pcked over arm a me. We proceed by analyzng he probably ha Equaons (5 and (6 hold. Noe ha he emprcal mean m s a normal random varable wh mean m and varance σr/n, so, condonal on n, µ s a normal random varable dsrbued as ( δ µ µ 0 N + n m δ + n, Equaon (5 holds f n σ r (δ + n. m µ + σ r δ + n Φ ( α m µ σ r δ + n Φ ( α z n + δ n Φ ( α + δ m σ r n where z N (0, s a sandard normal random varable and m = m µ 0. For an unnformave pror δ 0 +, and consequenly Equaon (5 holds f and only f z Φ( α. Therefore, for an unnformave pror, P(Equaon (5 holds = α = K =. Smlarly, Equaon (6 holds f m µ σ r Φ ( α δ + n µ m z σ r δ + n Φ ( α n + δ n Φ ( α + δ m, σ r n,

5 where z N (0, s a sandard normal random varable and m = m µ 0. The analogous argumen o ha for he above case shows ha, for an unnformave pror, P(Equaon (6 holds = α = K =. Equaon (7 holds f m < m + < σ r δ + n Φ ( α σ r δ + n Φ ( α 4β σr (δ + n < log( πα log(πα (8 = 4β σr (δ + n < + log T log log log T where = m m and he nequaly (8 follows from he bound (3. Therefore, for an unnformave pror, nequaly (7 never holds f n 4β σr ( + log T log log log T. Wh η = 4β σ r ( + log T log log log T, we ge E [ n T T η + P(Q > Q, n( η = η + + P(Equaon (5 holds, n ( η P(Equaon (6 holds, n ( η < 4β σr ( + log T log log log T + + T. The sum can be bounded by he negral T + yeldng he desred bound E [ n T ( 8β σ r + log T d = + log T, + 4β σr ( log log log T + +. Thus, we have shown ha n he case of deermnsc decson makng, he model acheves logarhmc regre unformly n T wh a consan whch agrees wh he bes possble one (5 up o a consan facor. As he followng secon shows, he analyss exends o he case of sochasc decson makng n a sraghforward way. B. Sochasc decson makng In he case where υ s defned by (, a smlar analyss holds. Agan consderng he case of an unnformave pror and seng he parameers K = and D =, he followng performance bound holds. Theorem 3 (Regre for sochasc decson makng: The expeced number of draws of a subopmal arm sasfes E [ n T ( 8β σ r + 4β σ r + log T + π 6 ( log log log T + +. Proof: See Appendx. Noe ha he bound on regre of he sochasc decsonmakng algorhm only dffers from ha of he deermnsc decson-makng algorhm by a consan equal o π /6. Therefore, by usng he dynamc feedback rule ( n he coolng schedule, he algorhm only pays a small performance penaly for he use of a sochasc maxmzaon n he decson sep. Human decson makng s nherenly sochasc. Whle s unlkely humans are usng hs specfc form of feedback rule, Theorem 3 shows ha a sochasc decson rule can acheve near-opmal performance. V. CONCLUSION In concluson, we propose he UCL algorhm for mularmed Gaussan band problems, and we analyze s performance n erms of expeced regre. We show ha, usng an unnformave pror, acheves logarhmc regre. We exend he algorhm o ncorporae sochasc polces usng Bolzmann acon selecon and develop a feedback law o dynamcally une he emperaure parameer of he selecon rule such ha he sochasc algorhm acheves logarhmc regre. As shown furher n [4, wh approprae choces of parameer values, sochasc UCL s a model for human decson makng n mul-armed band problems. APPENDIX Proof: [Proof of Theorem Snce he cdf for he sandard normal random varable s a connuous and monooncally ncreasng funcon, suffces o show ha Φ(β log( πα log(πα + α 0, (9 for each α (0,. Equaon (9 can be equvalenly wren as f(x 0, where x = πα and f s defned by f(x = Φ(β x log( x log(x +. π Noe ha lm x 0 + f(x = 0 and lm x f(x = / π. Therefore, o esablsh he heorem, suffces o esablsh ha f s a monooncally ncreasng funcon. I follows ha g(x := πf (x = x + β( x log(xβ / ( + log(x log( x log(x. Noe ha lm x 0 + g(x = + and lm x g(x =. Therefore, o esablsh ha f s monooncally ncreasng,

6 suffces o show ha g s non-negave for x (0,. Ths s he case f he followng nequaly holds: g(x = x + β( x log(xβ / ( + log(x log( x log(x 0, whch holds f log( x log(x β x( + log(x ( x log(x β = β x( + log(x + (log(x ( x log(x β. Leng = log(x, he above nequaly s ransformed o log(e β e ( + (e β, whch holds f log β β ( + e (β, whch s rue f nf log [, max [, β β 3 ( + e (β. (0 The exrema can be calculaed analycally, so we have for he lef hand sde and nf log = [, e max [, β β 3 ( + e (β for he rgh hand sde of (0, so (0 holds. Therefore, g(x s non-negave for x (0,, f(x s a monooncally ncreasng funcon, and he heorem holds. Proof: [Proof of Theorem 3 We begn by boundng E[n T as follows E [ n T T = E [P η + E [ P ( n η, ( where η s a posve neger. Now, decompose E [P as E[P = E [ P Q Q P(Q Q + E [ P Q > Q P(Q > Q E [ P Q Q + P(Q > Q. ( The probably P can self be bounded as P = exp(q /υ N j= exp(q j /υ exp(q /υ exp(q /υ. (3 Subsung he expresson for he coolng schedule n nequaly (3, we oban ( P exp (Q Q Q log = (Q Q Q mn. (4 mn Snce Q mn 0, wh equaly only f wo arms have dencal heursc values, condoned on Q Q he exponen on can ake he followng magnudes: 0 Q Q 0 =, f Q = Q, Q = +, f Q mn Q and Q mn = 0, x, f Q mn 0, where x [, +. The sgn of he exponen s deermned by he sgn of Q Q. Once each arm has been pcked once, he probably of es beween any par of he Q s s zero,.e., Q mn = 0 s zero. Consequenly, follows from nequaly ( ha E[P Q Q I follows from nequaly (4 ha E[P π T 6 + P(Q > Q = = π 6 + ( 8β σ r π 6. + log T + 4β σr ( log log log T + +, where he las nequaly follows from Theorem. REFERENCES [ J. A and A.A. Abouzed. Opporunsc specrum access based on a consraned mul-armed band formulaon. J. of Communcaons and Neworks, (:34 47, 009. [ M. Asawa and D. Tenekezs. Mul-armed bands wh swchng penales. IEEE Trans. on Auomac Conrol, 4(3:38 348, 996. [3 P. Auer, N. Cesa-Banch, and P. Fscher. Fne-me analyss of he mularmed band problem. Machne Learnng, 47(:35 56, 00. [4 S. Bubeck and N. Cesa-Banch. Regre analyss of sochasc and nonsochasc mul-armed band problems. Foundaons and Trends n Machne Learnng, 5(:, 0. [5 P. Fan. New nequales of Mll s rao and s applcaon o he nverse Q-funcon approxmaon. arxv:.4899, 0. [6 J. Gns, K. Glazebrook, and R. Weber. Mul-armed Band Allocaon Indces. Wley, 0. [7 E. Kaufmann, O. Cappé, and A. Garver. On Bayesan upper confdence bounds for band problems. In In. Conf. on Arfcal Inellgence and Sascs, pages , 0. [8 S. Krkparck, C. D. Gela Jr., and M. P. Vecch. Opmzaon by smulaed annealng. Scence, 0(4598:67 680, 983. [9 P. Kumar. A survey of some resuls n sochasc adapve conrol. SIAM Journal on Conrol and Opmzaon, 3(3:39 380, 985. [0 T. L. La and H. Robbns. Asympocally effcen adapve allocaon rules. Advances n Appled Mahemacs, 6(:4, 985. [ J. Le Ny, M. Dahleh, and E. Feron. Mul-agen ask assgnmen n he band framework. In IEEE Conf. on Decson and Conrol, pages , 006. [ K. Lu and Q. Zhao. Exended UCB polcy for mul-armed band wh lgh-aled reward dsrbuons. arxv:.768, 0. [3 D. Mra, F. Romeo, and A. Sangovann-Vncenell. Convergence and fne-me behavor of smulaed annealng. Advances n Appled Probably, 8(3:747 77, 986. [4 P. Reverdy, V. Srvasava, and N. E. Leonard. Modelng human decson-makng n generalzed Gaussan mul-armed bands. Proc. IEEE, 0(4:544 57, 04. [5 R. C. Wlson, A. Geana, J. M. Whe, E. A. Ludvg, and J. D. Cohen. Why he grass s greener on he oher sde: Behavoral evdence for an ambguy bonus n human exploraory decson-makng. In Neuroscence 0 Absracs, Washngon, DC, 0. Socey for Neuroscence.