Use of variance estimation in the multi-armed bandit problem

Transcription

1 Ue of variance eimaion in he muli-armed bi problem Jean Yve Audiber CERTIS - Ecole de Pon 19, rue Alfred Nobel - Cié Decare Marne-la-Vallée - France audiber@cerienpcfr Rémi Muno INRIA Fuur, Grappa Univerié Lille, France remimuno@inriafr Caba Szepevári Compuer Auomaion Reearch Iniue of he Hungarian Academy of Science Kende u 1-17, Budape 1111, Hungary zcaba@zaihu Abrac An imporan apec of mo deciion maing problem concern he appropriae balance beween exploiaion (acing opimally according o he parial nowledge acquired o far exploraion of he environmen (acing ub-opimally in order o refine he curren nowledge improve fuure deciion A ypical example of hi o-called exploraion veru exploiaion dilemma i he muli-armed bi problem, for which many raegie have been developed Here we inveigae policie baed he choice of he arm having he highe upper-confidence bound, where he bound ae ino accoun he empirical variance of he differen arm Such an algorihm wa found earlier o ouperform i peer in a erie of numerical experimen The main conribuion of hi paper i he heoreical inveigaion of hi algorihm Our conribuion here i wofold Fir, we prove ha wih probabiliy a lea 1 β, he regre afer n play of a varian of he UCB algorihm (called β-ucb i upper-bounded by a conan, ha cale linearly wih log(1/β, bu which i independen from n We alo analye a varian which i cloer o he algorihm uggeed earlier We prove a logarihmic bound on he expeced regre of hi algorihm argue ha he bound cale favourably wih he variance of he ubopimal arm 1 Inroducion noaion A K-armed bi problem (K i defined by rom variable X, (1 K, N +, where each i he index of an arm of he bi repreen ime Succeive play of arm yield reward X,1, X,, which are independen idenically diribued (iid according o an unnown diribuion Independence alo hold for reward acro he differen arm, ie for any N + 1 < K, (X,1,, X, (X,1,, X, are independen Le µ σ be repecively he (unnown expecaion variance of he reward coming from arm For any {1,, K} N, le X, V, be heir repecive empirical eimae: X, 1 i=1 X,i

2 V, 1 i=1 (X,i X,, where by convenion X,0 0 V,0 0 An opimal arm i an arm having he be expeced reward argmax µ {1,,K} For he ae of impliciy we aume ha here i a ingle opimal arm The proof hence he reul hold when here are muliple uch arm We denoe quaniie relaed o he opimal arm by puing in he upper index In paricular, µ = max µ The expeced regre of an arm i µ µ A policy i a way of chooing he nex arm o play baed on he equence of pa play obained reward More formally, i i a mapping from N {1,, K} R ino {1,, K} Le T ( be he number of ime arm i choen by he policy during he fir play Le I denoe he arm played by he policy a ime We define he cumulaive regre of he policy up o ime n a We alo define he cumulaive peudo-regre ˆR n nµ n =1 X,T I ( R n = K =1 T (n The expeced cumulaive regre of he policy up o ime n i The β-ucb policy 1 The algorihm R n nµ E [ n =1 X,T I (] = K =1 E[T (n] Aume ha he reward are bounded Then, wihou lo of generaliy, we may aume ha all he reward are almo urely in [0, 1] Le 0 < β < 1 be ome fixed confidence level Conider he ub-confidence level β defined a Le B, wih he convenion 1/0 = + ( X, + β β 4K(+1 (1 V, log(β log(β 1 1 β-ucb policy: A ime, play an arm maximizing B,T ( 1 Le u now explain he choice of B,T ( 1 The quaniy eenially come from he following heorem Theorem 1 Le X 1,, X be iid rom variable aing heir value in [0; 1] Le µ = EX 1 be heir common expeced value Conider he empirical expecaion µ ard deviaion σ 0 defined repecively a i=1 µ = X i σ i=1 = (X i µ Wih probabiliy a lea 1 β, we have log(β µ µ + σ 1 log(β 1 ( µ µ σ log(β 1 16 log(β 1 (

3 Proof 1 See Secion A1 Noe ha ( i uele for 5 ince i rh i larger han 1 We may apply Theorem 1 o he reward X,1,, X, he confidence level β Since 1 K; 1 β, = β/4, i give ha wih probabiliy a lea 1 β/4 1 β, for any N {1,, K}, we have µ B, I mean ha wih confidence level β, for any ime 1, afer he fir 1 play, he uer of he policy now ha he expeced reward of arm i upper bounded by B,T ( 1 The uer of he β-ucb policy chooe hi play only hrough hee upper confidence bound (UCB Properie of he β-ucb policy We ar wih a deviaion inequaliy for he number of play of non-opimal arm Theorem For any non-opimal arm (ie > 0, conider u he malle ineger uch ha u log[4ku (u +1β 1 ] > 8σ (4 Wih probabiliy a lea 1 β, he β-ucb policy play any non-opimal arm a mo u ime Proof See Secion A Thi mean ha wih high probabiliy, he number of play of non-opimal arm i bounded by ome quaniy independen of he oal number of play Theorem direcly lead o upper bound on he cumulaive regre of he policy up o ime n on i expeced value Since u depend on he parameer β, we will now wrie i u,β The following lemma give more explici bound on u,β Lemma 1 Le w = 8σ We have u,β 5w log(w Kβ 1 u,β w log(4kβ 1 + w log { 6w log(w Kβ 1 } The fir bound i he imple bu he lea accurae In he econd one, he leading erm i he fir one (when β goe o 0 Proof See Secion A Theorem Wih probabiliy a lea 1 β, for any ime n, he cumulaive regre of he β-ucb policy aifie K =1 T (n K [u,β n] (5 Beide for any poiive ineger n, he expeced cumulaive regre of he 1/n-UCB up o ime n aifie K =1 E[T (n] K C 1 K for ome univeral conan C 1 C [ (1 + u,1/n n ] { [( σ + 1 log(kn ] [ ] } n C log(n ( 1 + σ (6 Proof 4 The fir aerion i a direc conequence of Theorem For he econd aerion, he fir inequaliy come from T (n n ET (n P[T (n > u ]n + P[T (n u ]u The econd inequaliy ue Lemma 1 The hird inequaliy follow by conidering wo cae: eiher K > n (ie no enough ime o explore all he arm, hen he propery i rivial, or K n which implie log(kn log(n for any n 1

4 Bound for he expeced regre 1 Adapive β-ucb A far a reul in expecaion are concerned (ee econd par of Theorem, we need o ae β dependen on he ime horizon (ie he oal number of arm o be drawn o obain a regre bound of order C log n Schemaically an algorihm which need o now i ime horizon o have a (log n- cumulaive regre bound up o ime n can be modified ino an adapive algorihm having he ame cumulaive regre bound (up o a muliplicaive conan cloe o 1 Thi i done by he doubling ric (ee [, p] reference wihin in which we cu he ime pace ino inerval of lengh For each of hi epoch, we launch he algorihm independenly of wha happen in he oher epoch The policy now i ime horizon lead o a cumulaive regre for epoch of order Summing hee regre up o ome ime horizon T = K, we obain a cumulaive regre of order K i=1 i = K+1 1, hence of order log T For he β-ucb policy, we need neiher o rear a each epoch he policy nor o cu he ime pace in epoch Indeed, i uffice o ae β = 1/ a ime To decreae β when he number of arm already drawn increae i naural: when he ime increae, he exploiaion of an almo opimal arm become more more derimenal o he qualiy of he policy, o we wan o be a bi more ure ha he opimal one i no in oher arm Conequenly, we need o have beer upper confidence bound, which mean ha β hould be aen maller For hi adapive policy, one can how uing he ime cuing argumen preened above ha he reul given in (6 ill hold UCB-uned policy Define he confidence equence of arm c ( V, log(4, p The following figure decribe he UCB-uned policy: log(4p UCB-uned policy: A ime, play an arm maximizing ( X,T ( 1 + c (,T ( 1 1 A ligh variaion (wih differen confidence equence wa propoed in Secion 4 of [1] In heir experimenal udy hee auhor have found ha hi algorihm ouperform mo previou algorihm under a wide range of condiion However, no heoreical analyi of hi algorihm ha been aemped o far (Theorem 4 of [1], which i a cloely relaed reul ha applie o normally diribued payoff only, i no a complee proof ince i relie on ome conjecure The nex heorem how ha wih p >, he regre of he above algorihm cale wih he logarihm of he number of ep A crucial feaure of hi reul i ha inead of caling wih 1/ j, he regre cale wih σj / j Thi how ha he performance of UCB-uned i le eniive o wheher he aumed payoff range i a good mach o he rue range of payoff Theorem 4 Le p > For any ime n, he expeced regre of he UCB-uned policy i bounded by R n 16 [ log(4 + p log(n ] ( ( 1 + σ p 4

5 A Proof of he reul A1 Proof of Theorem 1 The following inequaliy i a direc conequence of Bernein inequaliy (ee eg [, p14] Lemma Le W 1,, W be iid rom variable aing heir value in [0; 1] Le E = EW 1 V = E(W 1 E be he expecaion variance of hee rom variable For any ɛ > 0, wih probabiliy a lea 1 ɛ, he empirical mean Ē = ( i=1 X i/ aifie V log(ɛ Ē < E log(ɛ 1 (7 To prove Theorem 1, we apply Lemma for ɛ = β/ hree differen iid rom variable: W i = X i, W i = 1 X i W i = 1 (X i EX 1 Le σ denoe he ard deviaion of X 1 : σ E(X i EX 1 Inroduce V Var [ (X 1 EX 1 ] We obain ha wih probabiliy a lea 1 β, we imulaneouly have ( µ µ σ 1 (8 log(β 1 + log(β 1 σ σ + (µ µ V log(β log(β 1 (9 Le δ be he rh of (8 L log(β 1 / Noing ha V σ, we have hence ucceively σ σ + δ + δ σ + δ, σ σ L 4L/ σ 0, σ L + σ + 10L/ σ + ( + 10/ L Plugging hi inequaliy in (8, we obain µ µ + σ L + [ + 0/ + / ] L µ + σ L L/ The revere inequaliy i obained in a imilar way A Proof of Theorem Le l log(β 1 (remember ha β = β/(4( + 1 Conider he even E on which X N +, µ σ < l + l {1,, K} (10 V, σ < Le u how ha hi even hold wih probabiliy a lea 1 β 8σ l + 4l Proof 5 We apply Lemma wih ɛ = β differen iid rom variable: W i = X,i, W i = 1 X,i, W i = (X,i µ W i = 1 (X,i µ We ue ha he variance of he la wo rom variable i bounded by E[(X,1 µ 4 ] σ ha he empirical expecaion of (X,i µ i 1 i=1 (X,i µ = V, + (X, µ We obain ha for any N + {1,, K}, wih probabiliy a lea 1 β X, µ σ < l + l V, + (X, µ σ σ < l + l 5

6 Since X, µ 1, hi la inequaliy lead o V, σ < X, µ + σ l + l, which give he econd inequaliy of (10 Uing an union bound, all hee inequaliie hold imulaneouly wih probabiliy a lea 1 4 K =1 1 β = 1 β Now le u prove ha on he even E, we have µ B, B, µ + σ 8l + 8l (11 Proof 6 For ae of impliciy, le u emporarily drop he indice: eg B,, µ, σ V, repecively become B, µ, σ V Inroduce L = l By (10, σ V < 8σ L + 4L Le q(σ = σ σ L + ( V 4L Since q(σ i negaive only beween i wo roo, he large roo give a bound on he value σ can ae when q(σ < 0 (he quare roo ric : σ < L + V + 10L V + (1 + 5/ L Plugging hi inequaliy in he fir inequaliy of (10, we obain µ µ < V L L, in paricular µ B For he econd par of he aerion, we ue { µ µ + σ L + L V σ + σ 8L + 4L ( σ + L obain B µ + σ L + 8L, which i he announced reul Le Ǩ be he e of non-opimal arm: Ǩ = { K : > 0 } For any ineger u where Ǩ, we have P[ Ǩ T ( > u ] = P[ Ǩ T ( > u ; E] + P[ Ǩ T ( > u ; E c ] P [ Ǩ < T ( = u I +1 = ; E ] + P(E c P [ Ǩ < T ( = u B,T ( B,T ( ; E ] + β P [ Ǩ < B,u µ or B,T ( < µ ; E ] + β P [ Ǩ B,u µ ; E ] + P [ r < B,r < µ ; E ] + β ( P Ǩ µ + 8σ l,u u + 8l,u u µ β The probabiliy in hi la rh i equal o zero provided ha he u are large enough Preciely, we wan u uch ha = l,u /u aifie 8 + σ < 0, equivalenly dropping he indice: < ( σ + 8 σ /8 We ge u 64 l u > ( σ +8 = ( 1 σ σ σ Thi inequaliy i a lea aified when which end he proof u l u > 8σ, (1 6

7 A Proof of Lemma 1 u Proof 7 By he definiion of u, we have 1 log[4ku (u 1β 1 ] w Thi implie u 1 + w log[4ku β 1 ] (1 Baic compuaion lead o u w Kβ 1 Thi very rough upper bound can be ued o have a igh upper bound of u Indeed, afer ome imple compuaion uing ha w 16, he fir wo recurive ue of (1 give u 5w log(w Kβ 1 u w log(4kβ 1 + w log { 6w log(w Kβ 1 }, which are he announced reul B Proof of Theorem 4 (UCB-uned regre The choice of hi confidence equence i uch ha, for any fixed arm index, any pair (, aifying 1, he following hold: Here P(µ X, + c (, 1 p, d (, 8σ log(4 p P(X, + c (, µ + d (, 1 p ( log(4p Indeed, following he proof of Theorem, we derive ha wih probabiliy 1 β, we have for any fixed, X, µ σ < log(4β 1 + log(4β 1 (15 V, σ 8σ < log(4β log(4β 1 From hi we deduce, imilarly o wha i done in he proof of Theorem, ha for any 1, wih probabiliy 1 β, we have he wo inequaliie V, log(4β µ X, + 1 log(4β 1 8σ µ + log(4β log(4β 1 (14 follow when chooing β = p Now, pic a ubopimal arm, (ie, µ < µ Then defining u,n 1 an ineger-valued equence ha will be eleced laer, we have: T (n = =1 I{I =} u,n 1 + u,n 1 + u,n =1 u,n =1 =1 I{I =,T ( u,n } I { X T ( +c,t ( X,T (+c (,T (,T ( u,n } I { } X T =1 ( +c,t ( µ I { } + X,T (+c (,T ( d(,t ( µ =1 =1 =1 =1 I {X +c, µ } I { } + X, +c (, d(, µ =1 =1 I { µ <µ +d (,T (,T ( u,n } (16 =u,n I { d (, > } (17 7

8 where (16 follow a follow: Aume ha X T ( + c,t ( > µ X,T ( + c (,T ( d (,T ( < µ, X T ( + c,t ( X,T ( + c (,T ( Then µ < X T ( + c,t ( X,T (+c (,T ( < µ +d (,T ( o under hee condiion we mu have µ < µ +d (,T ( The wo fir um in he expreion (17 are upper-bounded, in expecaion, by 1 =1 P(X + c, µ + P(X, + c (, d (, µ which, from (14, i upper-bounded by 1 =1 p = 1 p+1 (1 + 1/(p (bounding he um by an inegral for p > Now, he la um in (17 equal zero for ome appropriae value of u,n Indeed, han o he increaing monooniciy of d (, he decreaing monooniciy of d (,, he even d (, > for any 1 u,n n implie he even d ( n,u,n > Bu hi la even never hold for a large enough value of u,n Indeed, uing he ame argumen a in he proof of Theorem, ie he fac ha 8σ l u + 8l u < whenever u/l > 8σ / /, we deduce ha for we have d ( n,u,n < hu d (, he um in (17 i zero u,n log(4n p > 8σ < for all 1 u,n n, he hird erm of Thu, defining u,n 1 + log(4n p ( 8σ he logarihmic expeced number of ime a ubopimal arm i choen: E[T (n] log(4n p ( 8σ umming he hree erm of (17 yield he logarihmic bound on he expeced regre: R n [ log(4 + p log(n ] ( 8σ Reference + + p, + ( + p [1] P Auer, N Cea-Bianchi, P Ficher Finie-ime analyi of he muliarmed bi problem Mach Learn, 47(-:5 56, 00 [] L Györfi L Devroye G Lugoi A Probabiliic Theory of Paern Recogniion Springer-Verlag, 1996 [] G Solz Incomplee informaion inernal regre in predicion of individual equence PhD hei, Univerié Pari 11, 005 8