arxiv: v1 [cs.lg] 9 Oct 2018

Transcription

1 Bridgig he gap bewee regre miimizaio ad bes arm ideificaio, wih applicaio o A/B ess Rémy Degee Thomas Nedelec Cléme Calauzèes Viaey erche LSM, Uiversié aris Didero, Crieo AI Lab Crieo AI Lab CMLA, ENS aris Saclay, CMLA, ENS aris Saclay CMLA, ENS aris Saclay, Crieo AI Lab arxiv:800088v [cslg] 9 Oc 08 Absrac Sae of he ar olie learig procedures focus eiher o selecig he bes aleraive bes arm ideificaio or o miimizig he cos he regre We merge hese wo objecives by providig he heoreical aalysis of cos miimizig algorihms ha are also -AC wih a prove guaraeed boud o he decisio ime, hece fulfillig a he same ime regre miimizaio ad bes arm ideificaio This aalysis sheds ligh o he commo observaio ha ill-callibraed UCB-algorihms miimize regre while sill ideifyig quickly he bes arm We also exed hese resuls o he o-iid case faced by may praciioers This provides a echique o make cos versus decisio ime compromise whe doig adapive ess wih applicaios ragig from websie A/B esig o cliical rials Iroducio Wih he growig use of persoalizaio ad machie learig echiques o user-facig sysems, radomized experimes or A/B ess have become a sadard ool o evaluae he performaces of differe versios of he sysems Two of he mai drawbacks of such experime are is possible legh ad is cos, as i commoly akes few weeks or eve mohs o esure a saisically fouded decisio Thus, los of aeio have bee paid o he complexiy of he uderlyig saisical ess [8, 3,, 0, 3, ] These approaches have idubiable pracical ieres, bu are ofe limied o quie simple A/B frameworks because of heir resricive assumpios The firs resricive oe is he assumpio ha a good procedure should miimize he ime relimiary work eeded o ake a saisically sigifica decisio [3, ] I may siuaios, he mai objecive of praciioers is o miimize he overall cos of A/B esig wihou preveig him o ake he righ decisio give a cerai ime budge Aoher radiioal assumpio i he olie learig lieraure is ha oucomes arrivig over ime are idepede [3, ] Numerous commo scearii do o saisfy his ad praciioers cao beefi from he saisically efficie mehods available i he iid seig For isace, whe a olie reailer was o A/B es wo versios of is mobile applicaio, i may o be able o cosider he purchases as idepedes whe cusomers are buyig recurrely or clickig repeaedly i he CTR opimizaio problem To address he firs limiaio, we exhibi a well-ued varia of he UCB algorihm [] ha is boh able o ake a -AC decisio i fiie ime ad reaches a low regre This algorihm ca be ake as a ool for praciioers o ierpolae bewee he asks of bes arm ideificaio ad regre miimizaio Eve if his objecive has bee briefly meioed ad/or observed empirically [9], we provide he firs heoreical aalysis of such algorihm To hadle he secod limiaio, we exed he ideas developed i he iid case o a more complex seig ha ca hadle uis arrivig hrough ime ad deliverig rewards coiuously durig he es We provide sample efficie saisical decisio rules ad guaraees o ake decisios i fiie ime i his seig, highlighig he clear rade-off bewee regre miimizaio ad bes arm ideificaio, eve o more complex seigs Regre vs Bes-Arm Ideificaio i iid seig Framework We cosider he classical muli-armed badi problem wih K arms or populaio A each ime sep, he age chooses a arm i [K] :=,, K ad observes a reward draw from a ukow disribuio r i wih expecaio r i We assume ha each r i are σ -subgaussia, where he variace proxy σ is kow We deoe by π [K] he sequece of radom variable idicaig which arm o pull a ime N

2 Mauscrip uder review by AISTATS 09 Objecive We cosider boh aural objecives of badi problems The firs oe correspods o regre miimizaio I cosiss i miimizig he cumulaive regre RT = T max r i ; i [K] E T = r π I A/B esig, whe K =, miimizig he regre is he same as miimizig he cos of esig a ew echology or he impac of a cliical rial o paies The secod objecive, machig he problem of bes arm ideificaio wih fixed cofidece, is o desig a algorihm for a give cofidece level, ha miimizes he wors-case umber of sample T eeded for he algorihm o fiish ad o reur he opimal arm wih probabiliy Usig a algorihm for bes arm ideificaio i a A/B es gives a guaraee o he amou ime ecessary before beig able o o ake a saisically sigifica decisio Iuiively, a algorihm ha is opimal for regre miimizaio is sub-opimal for bes arm ideificaio because is exploraio is oo slow The opposie is also rue sice he exploraio of opimal bes arm ideificaio algorihms is oo aggressive for regre miimizaio We aim a sudyig a family of algorihms ha ierpolae bewee hese wo objecives Iformally, wih [0, ], our objecive is o desig algorihms for which wih probabiliy, for all badi problems i some class, he worse arm is discarded afer T sages ad we have boh T f, ad RT g, The values f, ad g, characerize he performaces of he algorihm ad should be as small as possible Lieraure review Regre miimizaio This objecive has bee exesively sudied i he badi lieraure sice he semial paper of [] We meio wo paricular classes of well-kow algorihms ha we will use hroughou he paper The Upper Cofidece Boud UCB algorihm iroduced i [, ] decides which arms o cosider depedig o he respecive empirical meas ad a error erm depedig o he umber of pulls of each arm Is regre, i he case of wo arms wih Gaussia rewards, is equal o logt / where = r A r B is he gap bewee he mea of he wo arms [] The oher class of algorihms we cosider is kow as Explore The Commi ETC[0, 7] They are firs cosiderig sages of pure exploraio before exploiig he arm wih he highes empirical mea The algorihm cosiss i uig he swichig imes bewee he sages Is regre i he case of wo arms wih Gaussia rewards is of order logt / We recall ha ETC is ecessarily sub-opimal for regre miimizaio, as i he case of Gaussia rewards here exiss a sub-opimal addiioal ad muliplicaive facor [7] Bes arm ideificaio The problem of bes arm ideificaio [5], ca be cas i wo mai seigs depedig o he cosrai imposed o he sysem: fixed budge [, ] where a oal umber of samples T N is give ad he goal is o miimize he error probabiliy a ime T; fixed cofidece [5, 5] where he goal is o miimize he oal umber of sages used o reur he bes arm wih probabiliy I he fixed cofidece seig here are wo mai ways o evaluae he sample complexiy of he algorihm : he average sample complexiy sudied i [5, 5, 3, ] where he goal is o miimize he expeced ime of decisio ad he wors case sample complexiy sudied i [6,, 9] where he objecive is o have a quaiy T N as low as possible such ha wih probabiliy, he algorihm makes o misake ad he ime of decisio τ d is below T I he case of wo arms, he opimal samplig sraegy is o sample each arm uiformly ad sop wih a crierio similar o he oe used i ETC [3] A/B esig Mos of he saisical lieraure o A/B esig [3, ] has focused o he objecive of miimizig he ime ecessary o ake a saisical sufficie decisio ad o he bes of our kowledge, here exiss o work heoreically ierpolaig bewee he objecives of bes arm ideificaio ad regre miimizaio Simulaeous Bes-arm Ideificaio ad Regre Miimizaio I his secio, we cosruc a family of algorihms ha miimizes regre while beig -AC wih a prove guaraeed boud o he decisio ime, hece fulfillig a he same ime he regre miimizaio ad bes-arm ideificaio For he sake of clariy, we are goig o assume ha here are oly k = populaios, as i A/B esig, deoed by A ad B The resuls for he geeral case K > ca acually be deduced almos immediaely from hose whe K = Le us firs recall ha he algorihm wih lowes decisio ime[], give he variace of arms are ideical, is ETC, which pulls boh arms uiformly ad, afer pullig each arm imes, reurs he arm eg A wih highes empirical average ˆr A, if ˆr A ˆr B σ log log I he saeme of our heorems, he usual Ladau oaio O sads for ay fucio whose limi is, as goes o 0, equal o 0 Similarly, we will use 3 log isead of

3 Mauscrip uder review by AISTATS 09 for he sake of clariy Exac values of, precise saemes ad proofs are mosly delayed o Appedix B The performaces of ETC are ow well udersood roposiio [9] Wih probabiliy greaer ha, ETC reurs he bes arm a a sage τ d T where T 3σ log + O So he regre of ETC a he ime of decisio verifies Rτ d 6σ log + O O he oher had, he semial algorihm which is opimal for regre miimizaio is called UCB; i sequeially pulls he arm wih he highes score he sum of he empirical average plus some error erm while is decisio rule is o saisfied Alhough is regre is small, i is o guaraeed ha he bes arm will be ideified i a shor ime roposiio [] Wih probabiliy a leas, he regre of UCB is bouded as Rτ d 8σ log + O There is o guaraee ha τ d is uiformly bouded Our family of algorihms ierpolaes bewee UCB ad ETC by iroducig a sigle parameer α [, + ] whose exreme values correspod respecively o focusig solely o regre miimizaio ie, our algorihm ideifies wih UCB or bes-arm ideificaio i ideifies wih ETC To each value of α [, + ] correspods a differe rade-off: Ideed, he bigger he α, he bigger he regre bu he smaller he decisio ime More precisely, we iroduce ad sudy a coiuum of algorihms UCB α For N σ, defie ε = log 3 log For α, R +, UCB α allocaes he user a he populaio wih he highes score argmax i ˆr i + αε i i I reurs a arm i A, B if i domiaes he oher arm j i he sese ha ˆr i ε i i ˆr j + ε j j By cosrucio, he UCB α algorihm will keep boh idexes aroud he same level Bu due o he facor α >, he iervals of decisio wih widh ε A ad ε B wihou he facor α ha is oly used i he samplig policy, o i he decisio rule will eveually become disjoi Thus, while behavig like a UCB-ype algorihm o miimize regre, UCB α ca sill reur he ideiy of he bes arm The followig heorem makes precise his rade-off bewee ime of decisio ad regre Theorem 3 Wih probabiliy greaer ha, UCB α has a regre Rτ d a is ime of decisio saisfyig 8σ Rτ d c α + log + O Algorihm UCB α Ipu: α, : repea over : for each populaio i A, B do σ log 3 log i i 3: ε i = : ed for 5: Assig ex user o populaio i = argmax i A,B ˆr i + αε i 6: i = argmax i A,B ˆr i 7: uil ˆr i ε i > ˆr j + ε j for j i 8: reur i where he cosa c α [, 7] is defied by c α = mi α + α, ad c α = ; c = O ha eve, he ime of decisio saisfies τ d T wih T α + 6σ α c α + log + O Whe α goes o, he leadig erm of he regre goes o 8σ log / bu he ime of decisio becomes ifiie Whe α, he leadig erm becomes 3σ log / ad he ime of decisio is of order 6σ log / The fac ha he cosa c α is o moooic i α, is a arifac of he proof, ie, a byproduc of wo differe aalyses of he same problem for eiher small or large values of α Ideed, Figure acually idicaes ha regre of UCB α is ceraily moooic wih respec o α as expeced ETC has a regre bouded by a expressio of order 6σ log This is wice bigger ha he regre of UCB α for small α These are oly oe-sided bouds ad do o allow o coclude o which algorihm ges a lower regre, bu experimes show ha UCB α for small α ideed preses a advaage i erms of regre versus ETC, a he cos of a higher decisio ime See Figure Whe α goes o ifiiy, he exploraio erm is domia for UCB α, ad i will always pull he leas pulled arm As a cosequece, i becomes a varia of ETC wih a subopimal decisio crierio We deoe i by ETC I allocaes o he populaios A ad B uiformly ad selecs A if ˆr A ˆr B 8σ log 3 log Is cofidece ierval widh is imes larger ha he oe of ETC because of he differe coceraio argumes used i desigig he algorihms: ETC uses a coceraio lemma o he differece ˆr A ˆr B, while UCB α ad is limi ETC deals separaely wih arm A ad B sice heir umber of samples migh be differe Noe ha he differece bewee ETC ad ETC is a specificiy of he wo-armed badi case,

4 Mauscrip uder review by AISTATS 09 ad is regre Rτ d saisfies Rτ d K 8σ c α + log K + O k k= Wih K > populaio, ETC would sop samplig populaio k as sooas here exiss aoher populaio i such ha ˆr ˆr i k σ log 3K log As a cosequece, is decisio ime will be upper-bouded by 3σ mi + K k= ad is regre smaller ha K k= 3σ log K k + O 3σ logk + O As a cosequece, eve if UCB α ierpolaes bewee UCB ad ETC i erm of regre, i acually ouperforms ETC i erms of decisio ime whe α How does iflaig he exploraio erm lead o a fiie decisio ime? A proof skech Figure : Compariso of ETC ad UCB α Lef: regre a selecio Righ: ime of selecio The curves average 000 experimes wih wo Gaussia arms of meas 0 ad ad variace as he geeralizaio of ETC o more ha wo arms [9] cosiders per-arm cofidece iervals O he oher had, UCB α geeralizes immediaely whe he umber of populaios is greaer ha I ha case, we assume ha he idex of he opimal populaio is ad we deoe by k = r r k he gap associaed o he subobimal populaio k The we ca derive he followig corollary from Theorem 3 Corollary Wih K differe arms, ad wih probabiliy greaer ha, UCB α has a decisio ime smaller ha α + 8σ α c α + mi + K k= log K 8σ c α + K k + O We cosider a algorihm which pulls arg max i A,B ˆr i + i αε i wih α >, ad sops if r A r B A > ε B A + ε B ad reurs he arm wih highes mea a ha poi Recall ha ε is a quaiy close o /, up o logarihmic erms i ad muliplicaive cosas, hece /ε c for some cosa c If we ca prove ha as log as o decisio is ake, /ε A ad /ε B are bouded from above, he we obai a upper boud o he ime = A + B before a decisio is ake Suppose ha he bes arm is A Boudig B is doe hrough classical badi argumes: sice B is he worse arm, a UCB-ype algorihm does o pull i ofe The challege is o show ha our algorihm also corols A We ca make use of a coceraio resul of he form: wih probabiliy, ˆr A A + ε A r A ad ˆr B B ε B r B The decisio crierio esures ha if a decisio is ake ad hese coceraio iequaliies hold, he he arm reured is he correc oe Ideed uder his coceraio eve, ˆr B B ˆr A A r B r A + ε B + ε A, which is sricly smaller ha ε B + ε A Hece B cao be reured: if he algorihm sops, i is correc The algorihm will keep boh idexes roughly equal, hece ˆr A + αε A A ˆr B + αε B B ad he pullig cofidece iervals wih widh αε will ever ge disjoi Bu as ε A ad ε B ge small, he smaller decisio cofidece iervals wih widh ε will eveually separae, as see i Figure

5 Mauscrip uder review by AISTATS 09 More formally, if A is pulled ad o decisio was ake ye, he he idex of A is big, ˆr A A + αε A > ˆr B B + αε B, bu o so big ha he algorihm sops, ie ˆr A A ε A ˆr B B + ε B By combiig he wo, we obai he relaio A ε A α+ α ε B α+ α B, where is o be read as he iformal saeme ha he quaiies are roughly proporioal To sum-up he idea of he proof: B will o be pulled much sice i is he wors arm ad we employ a UCB-ype algorihm A will be pulled less ha a facor depedig o α imes B Thus as log as o decisio is ake, = A + B is bouded, by a quaiy T We coclude ha he decisio ime is smaller ha T ˆr A ˆr B A αε B B ε B o resriced o olie markeig Cliical rials classically measure he survival ime or qualiy of life of paies over ime ad adapive esig is a key challege i his coex, however i is so far eiher heurisic [] or uder he resricive assumpio o observe he rewards before he ex decisio [3, 8] More recely, [7] sudied a similar seig of reward arrivig over ime, however wih a differe objecive, amely fidig a adapive way o sop he es for a paie akig io accou a cos of esig The seig of muli-armed badi preseed i he previous secio has bee applied o A/B ess [3, 3, ] wih idepede rewards The goal of his secio is o show ha we ca exed algorihms preseed i he firs secio i a more complex seig ad provide a framework o praciioers for ierpolaig bewee bes arm ideificaio ad regre miimizaio i oher seigs ha he iid case To model heses aspecs, we show ha decisios of he muliarmed badi algorihm ca be ake a a ui level eg users, paies, servers Oce allocaed o a populaio A or B, a ui u ieracs wih he sysem durig he whole A/B es Whe ime icreases, he sysem gahers more ad more sigal o a ui arrived early i he A/B es We will also assume i order o be able o ake a causal decisio a he ed of he A/B es ha uis already exposed o oe reame ca o be swiched from populaio A ui says i he same populaio durig he whole A/B es Iuiively, he sysem will esimae he performace of oe echology by averagig is performace o he differe uis ˆr A ˆr B A Figure : Cofidece iervals used i UCB α Top: Before he ime of decisio, he idexes ˆr+αε are aliged, he igher ε iervals also overlap Boom: ime of selecio: he igh ε cofidece iervals become disjoi Exesios o o-iid seigs Whe a iere plaform was o A/B es wo versios of is websie, purchases from cusomers ha are buyig recurrely ca o be cosidered as idepede Whe a echical A/B es is ru, i is also usual o spli servers i wo populaios o A/B es he busiess impac of he laecy of a ew code versio I his case, observaios from he same server are o idepede This seig is B Noaios We eed o differeiae he uis eg users radomly assiged o populaios A ad B from heir associaed rewards I he iid seig, he reward ru A associaed o a ui u i populaio A was assumed o be observed isaly Now, we assume ha, for each ui u we have bee seeig so far, we observe oisy versio of his reward over ime ad ru A is oly he ukow expecaio of his process opulaio-specific oaio is symmeric, hus, for he sake of readabiliy, we oly deail oaios for he corol A ad assume he correspodig oe for he reame B More formally, we assume he uis u are iid samples from a ukow disribuio ad arrivig i he es dyamically over ime To each ui u is associaed a ukow reward r A u which is a iid sub-gaussia rv wih expeced value r A ad variace σ r, as well as a arrival ime T u A deoes he se of uis i A of cardialiy A a ime The, give a ui u ad sarig a ime T u, we observe over ime radom oucomes r A u, = r A u +ε u, where ε u, is a zero-mea sub-gaussia variable wih variace σ ε A ime, he ui u has geeraed T u + oucomes r A u, u r A u, A each ime sep, he algorihm has access o all he rewards geeraed by all he users already prese i he A/B es ad

6 Mauscrip uder review by AISTATS 09 badi algorihms wih good properies Wih a differe model o he per-user radom oucomes as mea reverig process or cyclic process, our proof echiques could be used as log as i is possible o cosruc a esimaor of he expecaio ha cocerae well eough recise saemes ad proofs of resuls preseed i his secio ca be foud i Appedix C uis A B Figure 3: The ui allocaio problem A each sage N, a ew ui is allocaed o eiher populaio A or B ad a sample from every ui arrived before is observed he precisio o r A u will icrease wih ime as more samples r A u, are gahered I his seig, a aural esimaor o cosider is ˆr A := A u A T u +? s=t u r A u,s We call his esimaor he Mea of meas esimaor Wih his esimaor, we ca desig algorihms ha ca reach a rade-off bewee regre miimizaio ad bes arm ideificaio ad see how his regre deped o σ r ad σ ε We could have cosidered oher esimaors such ha he Toal Mea esimaor defied as ˆR A := u A T u + ru,s A u A s=t u which is also a ubiased esimaor of r Ye uforuaely, his esimaor pus more weighs o older uis ha o he more rece oe I he case where all uis have more or less he same oise, his is o really a issue bu his is more or less he oly case where he oal mea esimaor has good behavior Uforuaely, wih adapive samplig algorihm such as UCB, he i could be he case ha a ew ui is allocaed o a populaio afer a expoeial log ime I ha case, he differe weighs pu o differe uis ca be of differe order of magiude, preveig fas covergece of he esimae o he esimaed mea A secod moivaio for choosig he Mea of meas esimaor is because i opes doors for geeralizig o more complex models o he sochasic processes uderlyig he uis behavior Ideed, i ca be see as a average over he uis of he per-ui sochasic process expeced values Here, we assume he radom oucomes r i u, are iid wih expecaio r i u echically our resuls hold for marigale differece sequece The he echical par shows how o combie coceraio resuls o each of he uis o derive ETC The ETC algorihm allocaes uis aleraively o A ad B, choosig possibly he firs populaio a radom To simplify he aalysis, we are acually goig o assume ha uis arrive a each sage, oe of each beig allocaed o each populaio So ha if ETC sops a sage N, he boh populaios have = uis, ad regre is The soppig rule crierio of ETC is simply he followig: σr+ σ ε loge ˆr A ˆr B π > log 3 Theorem 5 Wih probabiliy a leas, he ETC algorihm oupus he bes populaio ad sops afer havig allocaig a oal of a mos τ d T uis, where T = 3σ r log + σ ε +O σr log log +O Is regre a τ is equal o τ 3 UCB-MM The varia of UCB we cosider is defied wih respec o he followig idex of performace of populaio i A, B defied by ˆr i σr + σ ε logei + i i i log max, i σr Usig his idex, UCB-MM is described i Algorihm We meio here ha havig radom umbers of uis wih radom umbers of occurreces preves us for derivig he more or less sadard coceraio iequaliies derived for he oher algorihm Ideed, i acually requires o combie several ypes of differe iequaliies, which explais he o-sadard erm i he log par of he idex Theorem 6 Wih probabiliy a leas, he regre of UCB-MM is bouded a sage d as Rτ d 8σ r log + O + σ r log log + O There is o guaraee ha τ d is uiformly bouded

7 Mauscrip uder review by AISTATS 09 Algorihm UCB-MM α Ipu: α, : repea over : for each populaio i A, B do 3: ε i σr = + σ ε logei i i i : log max, i σ r σ ε 5: ed for 6: Assig ex user o populaio i = argmax i A,B ˆr i + αε i 7: i = argmax i A,B ˆr i 8: uil ˆr i ε i > ˆr j + ε j for j i 9: reur i We emphasize here ha he depedecy of he oal regre wih respec o he oise variace is egligible compared o is depedecy wih respec o he variace of he ui performace σr Ideed, regre has a log facor i fro of σr a erm which is uavoidable, eve wihou exra oise, ie, if = 0 O he oher had, he muliplicaive facor of is oly double logarihmic, i log log Moreover, he addiioal umber of uis required o fid he bes populaio is, asympoically, idepede of, he proximiy measure of he wo populaios I he idex defiiio of UCB-MM Equaio, leig σ ε goes o 0 gives a void boud he idex is basically always + This is a arefac of he proof eeded for havig oly a exra log log erm I is also possible o use he followig aleraive error erm for UCB-MM σr 9 log i i log + loge i i i log This error erm coverges, as σ ε goes o 0, o he usual error erm of UCB Uforuaely, he regre depedecy i σ ε deerioraes as i scales wih 8σ r log + σ r log + O hus wih a log exra erm isead of a log log oe 3 Ierpolaig bewee regre miimizaio ad bes arm ideificaio As i he iid case, i is possible, wih ui, o defie UCB- MM α o ierpolae bewee UCB-MM ad ETC-MM by muliplyig he error erm of UCB-MM by a facor α [, + ] Theorem 7 Wih probabiliy a leas, he regre of UCB-MM α is bouded a sage T as Rτ d 8σ r c α + log + O + σ r log log + O Moreover, he ime of decisio is upper-bouded by τ d α + 6σ r α c α + log + O + σ r log log + O The proof of his resul proceeds as i he iid case see he skech of proof i secio, assumig ha we ca provide a suiable coceraio iequaliy o boud he deviaio of he mea-of-meas esimaor The mai differece is he coceraio argumes used The aalysis is adapive i ha i gives a boud o he decisio ime for ay model, as log as we are able o provide coceraio iequaliies for he populaio meas The coceraio iequaliy is used o obai a boud o he umber of allocaios o he subopimal populaio, he he iflaed cofidece iervals mechaically provide a boud o he umber of pulls of he bes populaio wih respec o he sub-opimal oe As i he iid case, we have saed our resuls for K = populaios, bu i s sraighforward o geeralize hem for K > differe populaios racical remark I pracice, aribuig users dyamically o populaios could be hard o hadle i producio for isace, he populaio of he user mus be sored o ierac wih him whe he is comig back o he plaform This is why we also provide aoher ayime boud i appedix i a simpler seig where we assume ha all he uis are already prese a he begiig of he es, hus allowig o allocae hem o populaios usig a simple hash o heir ideifiers Based o his boud, he praciioer ca sop he es as early as possible such ha he decisio is saically sufficie However, his boud ca o help o do a radeoff bewee regre miimizaio ad bes arm ideificaio O he oher had, we assume ha he reward ad oise were subgaussia radom variable, wih kow variace proxy σ r ad σ ε Our resuls ad echiques ca be geeralized o he case where he radom variables r i ad ε, are bouded say, i [0, ] wih ukow variace Oe jus eed o use empirical Bersei coceraio iequaliies [6] 3 Experimes O a simple iid seig, we show he performace proved i Secio o Figure There are wo Gaussia arms wih same variace ad meas 0 ad The wo graphs o

8 Mauscrip uder review by AISTATS 09 he figure show he decisio ime ad he regre a he ime of decisio of several algorihms for a rage of values of log/ The algorihms show are ETC, four isaces of UCB α for alpha i 5,,, 3 ad ETC, he varia of ETC o which UCB α eds o whe α We highligh a few coclusios from hese plos, which are all i agreeme wih he heoreical resuls The algorihm wih lowes decisio ime is ETC, he algorihm wih lowes regre is UCB α wih small α For α, UCB α has lower regre ad higher decisio ime ha ETC, bu is worse ha ETC o boh crieria For α equal o 5 or, UCB α has lower regre ad higher decisio ime ha ETC UCB α is see empirically o realize a rade-off bewee is wo limiig algorihms UCB ad ETC, ad here is a ierval for α i which UCB α is a rade-off bewee UCB ad ETC The umerical relaios bewee he bouds ca also be verified: he regre of UCB 5 is almos wice smaller ha he regre of ETC, which is wice smaller ha he regre of ETC We he show o Figure empirical resuls o he ui seig preseed i Secio A each ime sep, a user arrives ad he geerae a reward accordig o r u for every ime sep uil he ed of he game I our simulaio, r u is sampled from a ormal disribuio wih mea 0 for populaio A respecively wih mea for populaio B ad variace σ r equal o The oise ɛ a each ime sep is also Gaussia of mea 0 ad variace σ ɛ equal o The daa ca be see as a riagular marix cf Fig 3 We compare performaces bewee UCB α for differe values of α ad ETC i erms of regre ad imes of decisio We see ha we are able o reproduce wha we observed i he iid seig i he ui seig We observe agai a facor bewee he regre of ETC ad UCB Wih UCB α, we ca realize a rade-off bewee ETC ad UCB, boh i erms of regre ad decisio ime Coclusio We sudied A/B ess i he fixed cofidece seig from he wo perspecives of regre miimizaio ad bes arms ideificaio We iroduced a class of algorihms ha opimizes a he same ime boh objecives I ierpolaes bewee opimal algorihms desiged for each case Our sudy also shed ligh o a effec ofe see i pracice: he UCB algorihm o oly miimizes regre bu also ideifies he bes arm i fiie ime, as soo as is exploraio erm is slighly iflaed This iflaio is early always prese i pracice Ideed, for UCB o be a valid algorihm for a oise ad he cofidece ierval o o be bigger ha ecessary, i would eed o be ru wih a cosa machig exacly he ukow sub-gaussia orm of he oise Figure : Firs: Regre i he ui seig Secod: Decisio ime for he ui seig The curves are averages of 000 experimes wih wo Gaussia arms wih meas 0 ad ad σ r =, σ ɛ = We exeded our sudy o a o-iid seig by derivig adaped coceraio resuls for he mea-of-meas esimaor, agai obaiig algorihms which ierpolae bewee he wo objecives We would however like o war praciioers o a iesive use of badi algorihms for A/B ess Eve if daa are colleced hrough ime, i ca prove o be difficul o defie a ime of arrival for uis o correlaed wih he daa o esure he iid assumpio holds as badi algorihms ha eglec his aspec could behave quie poorly Hadlig such problem would require o model he ui sochasic process codiioally o he ime of arrival which would be a use case for geeralizig our resuls o more complex sochasic processes for ui modellig

9 Mauscrip uder review by AISTATS 09 Refereces [] J-Y Audiber ad S Bubeck Bes arm ideificaio i muli-armed badis I Coferece o Learig Theory, 00 [] Auer, N Cesa-Biachi, ad Fischer Fiie-ime aalysis of he muliarmed badi problem Joural of Machie Learig Research, 7-3, 00 [3] D A Berry ad S G Eick Adapive assigme versus balaced radomizaio i cliical rials: A decisio aalysis Saisics i Medicie, 3, 995 [] S Bubeck, R Muos, ad G Solz ure exploraio i fiiely-armed ad coiuous-armed badis Theoreical Compuer Sciece,, 0 [5] A Carpeier ad A Locaelli Tigh lower bouds for he fixed budge bes arm ideificaio badi problem I Coferece o Learig Theory, 06 [6] E Eve-Dar, S Maor, ad Y Masour Acio elimiaio ad soppig codiios for he muli-armed badi ad reiforceme learig problems Joural of Machie Learig Research, 7Ju, 006 [7] A Garivier, T Laimore, ad E Kaufma O explore-he-commi sraegies I Advaces i Neural Iformaio rocessig Sysems, 06 [8] F Hu ad W F Roseberger The Theory of Respose- Adapive Radomizaio i Cliical Trials Joh Wiley & Sos, Ic, Apr 006 [9] K Jamieso, M Malloy, R Nowak, ad S Bubeck lil ucb: A opimal exploraio algorihm for muliarmed badis I Coferece o Learig Theory, 0 [0] R Johari, Koome, L ekelis, ad D Walsh eekig a a/b ess: Why i maers, ad wha o do abou i I roceedigs of he 3rd ACM SIGKDD Ieraioal Coferece o Kowledge Discovery ad Daa Miig, KDD 7 ACM, 07 [] Z Kari, T Kore, ad O Somekh Almos opimal exploraio i muli-armed badis I Ieraioal Coferece o Machie Learig, 03 [] M N Kaehakis ad H Robbis Sequeial choice from several populaios roceedigs of he Naioal Academy of Scieces of he Uied Saes of America, 99, 995 [3] E Kaufma, O Cappé, ad A Garivier O he Complexiy of A/B Tesig I Coferece o Learig Theory, roceedigs of The 7h Coferece o Learig Theory, 0 [] E Kaufma, O Cappé, ad A Garivier O he complexiy of bes-arm ideificaio i muli-armed badi models Joural of Machie Learig Research, 7, 06 [5] S Maor ad J N Tsisiklis The sample complexiy of exploraio i he muli-armed badi problem Joural of Machie Learig Research, 5Ju, 00 [6] A Maurer ad M oil Empirical Bersei Bouds ad Sample-Variace ealizaio Coferece o Learig Theory, 009 [7] D M Negoescu, K Bimpikis, M L Bradeau, ad D A Iacu Dyamic learig of paie respose ypes: A applicaio o reaig chroic diseases Maageme Sciece, 68, 08 [8] L ekelis, D Walsh, ad R Johari The ew sas egie Iere Rerieved December, 6, 05 [9] V erche ad Rigolle The muli-armed badi problem wih covariaes The Aals of Saisics, 03 [0] V erche, Rigolle, S Chassag, E Sowberg, e al Bached badi problems The Aals of Saisics,, 06 [] W H ress Badi soluios provide uified ehical models for radomized cliical rials ad comparaive effeciveess research roc Nal Acad Sci U S A, 06 5, Dec 009 [] W R Thompso O he likelihood ha oe ukow probabiliy exceeds aoher i view of he evidece of wo samples Biomerika, 53/, 933 [3] F Yag, A Ramdas, K G Jamieso, ad M J Waiwrigh A framework for muli-armed/badi esig wih olie fdr corol I Advaces i Neural Iformaio rocessig Sysems Curra Associaes, Ic, 07 [] S Zhao, E Zhou, A Sabharwal, ad S Ermo Adapive coceraio iequaliies for sequeial decisio problems I Advaces i Neural Iformaio rocessig Sysems 06

10 Mauscrip uder review by AISTATS 09 A Coceraio Iequaliies We provide here required resuls of ayime bouds ha hold i he iid seig for he ieresed reader Hoeffdig o iervals Lemma 8 Le Z be a σ -subgaussia marigale differece sequece he for every > 0 ad every iegers T T N σ [T ; T ], Z T log φ T where he mappig φ is defied by φx = + x + x x ad i holds ha x 6 φx roof For ay λ > 0, i holds ha ] E [e λz F = E [ ] e λz F e λ Z e λ σ e λ Z sice Z is σ -subgaussia Thus X = e λz λ σ is a supermarigale ad we derive from Markov s iequaliy ha: [T, T ], λz λ σ + ε e ε As is cocave o [T, T ], we ca fid β such ha β λ σ + ε wih equaliy i T ad T From hese wo equaliies, we choose λ ad β as fucio of ε: ε T + T λ = σ β = ε T T T T Combiig his wih, we fially obai: his yields he resul Ayime Hoeffdig [T, T ], λz β = [T, T ], Z σ ε T + T e ε, T T Lemma 9 Le Z be a σ -sub-gaussia marigale differece sequece The, for ay α > 0 ad > 0 saisfyig i holds ha σ N, Z + log log < α < log log 8 log logα c α log log, e α + log +, where c α = e α log 8α α α log log e α log α 8α 6/5 α 6/5, so ha c 6 ad log is he aural logarihm i basis, wih he exra assumpio ha log =

11 Mauscrip uder review by AISTATS 09 If α, > 0 are such ha he σ N, Z + log α log log logα c α log α + + 8α log/, so ha c 7 where c α = e α/ζ α e α/ζ 3α α α log/ α, roof We are goig o use he fac ha wih probabiliy a leas, for all s [T, T ], s Z s E[Z] Defie ε = σ log logα so ha wih γ = + η >, m= log γ N, Z ε m=0 σ sφ T T log [γ m, γ m+ ], Z ε The case m = 0 correspodig o = will be hadled separaly a he cos of a exra erm Noe ha σ [γ m, γ m+ ], Z = m= log γ m= log γ σ [γ m, γ m+ ], Z [γ m, γ m+ ], Z σ log logα γ m log logα γ m φγ η 6 φγm+ γ m log log α γ m We ca ow apply Lemma 8 Ad his gives, assumig γ ie, η < for he mome, N, Z ε = m= log γ log α γ log α log α η 6 log α γ m + η 6 η 6 η 6 m= log γ + + η mα 6 α η 6 log γ + α η 6 + η The choice of η = 8α/ loglog/, which esure ha η < as log as α < 8 N, Z ε log α e log log α 8α log log α α log α e α 8α log log η 6 + log log gives + + 6/5 α e α 6/5 + log α +

12 Mauscrip uder review by AISTATS 09 We ow cosider he case where γ migh be bigger ha, bu le us assume for ow ha γ < 5 ie, η ad he exac same argume wih he choice of η = 8α/ log/ gives N, Z ε m= log α γ m η 6 + γ η 6 log α ζα η + γ + η 6 η αζα η 6 + γ η 6 = e α log/ α/ζ α α α log/ = log α + + e α/ζ α α + + α log/ 8α log/ 8α log/ Corollary 0 Le Z be a σ -sub-gaussia marigale differece sequece The for > 0 small eough σ where is defied by eiher or, depedig o he rage of, by, N, Z log log, = c log log log3 + log e log +, = c log loglog + 5 log I he firs case, is of he order of log ad, i he secod case, of he order of log The followig coceraio iequaliy will be useful for he o-iid case I ha framework, a some sage, he oal umber of users i he populaio is radom, deoed by We recall ha r deoes he mea of he -h user of he populaio ad ε,s is he radom whie oise for ha user afer he is i he populaio for s sages I he remaiig, we assume ha he expecaio of r u is equal o r ad ha his radom variable is σr-subgaussia, O he oher had, he expecaio of ε is aurally 0 ad his radom variable is -subgaussia A algorihm is herefore a samplig policy A ha idicaes afer seeig he firs values of r u plus some empirical average oise ε u, a ime N wheher o add a ew user or o We deoe by A 0, he decisio o iclude a ew user or o a sage We deoe by T N he ime where he -h user is added, by τ = T + T he umber of sages wih exacly users ad by τ m: = s=m τ s he umber of sages bewee he arrival of he m-h user ad he -h oe We also deoe by + he umber of user a sage N, roposiio For ay algorihm ad > 0, i holds N, σr + σ ε loge r u + ε u, Tu+ r u= log 6 max, σr roof We rewrie he saeme of he proposiio ad oice ha we jus eed o prove ha, for ay N, s τ, r + ε,s+τ : + + ε,s σr + σ ε loge 36 r log max, σ r 3 3/

13 Mauscrip uder review by AISTATS 09 The expoe 3/ will come from he fac ha is of he order of log ad o We will eve prove he followig s <, r + ε,s+τ : + + ε,s σr + σ ε loge 36 r log max, σ r 3 3/ We will decompose he cosidered eve defied o s < i wo, depedig wheher s 6 max, σ r σ or ε s > 6 max, σ r σ ε I he firs case, we aim a provig ha for all s 6 max, σ r σ, ε r + ε,s+τ : + + ε,s σr + σ ε loge r As usual, ake η > 0 ad le us ry o upper boud 36 log max, σ r 8 r + + r r + ε,s+τ: + + ε,s η 7/ max, σ r Iroduce λ > 0 ad Markov iequaliy yields r + + r r + ε,s+τ: + + ε,s η E[e λr++r r+ε,s+τ : ++ε,s ]e λη Noe ha he realizaios of ε s ad µ are idepede of he oher average oce we have ake he decisio o iclude ha user, ie, codiioally o T E[e λr++r r+ε,s+τ : ++ε,s ] = E [E [ e λr r+ε,s e λr++r r+ε,s+τ ] : ++ε,s+τ : T ] = E [E [ e λr r+ε,s ] [ T E e λr ++r +ε,s+τ: ++ε,s+τ : ] T ] = E [E [ e r r+ε,s] E [ e λr++r r+ε,s+τ ] : ++ε,s+τ : T ] Le us rewrie, for clariy, he las expecaio of he rhs as = E [ e r r+ε,s] E [ e λr++r r+ε,s+τ : ++ε,s+τ : ] E [ e λr++r r+ε,s+τ : ++ε,s+τ : ] = E [ e λr++r r+ε,s+τ +τ : ++ε,s+τ ] If we codiio similarly o T, we ca focus o upper-boudig Choosig λ = [ E e ] λr µ+ε,s+τ = η σ r +σ ε loge gives ha j=s+ j=s+ [ ] E e λr µ+ε,j s + τ = j e λ σ [ ] r + σ ε j E s + τ = j e λ σ r r + + r r + ε,s+τ: + + ε,s η e η σ r + loge + σ ε s+ σ ε So, for all N ad all sage s 6 max, σ r σ ad wih probabiliy a leas ε, i holds r + ε,s+τ : + + ε,s σr + σ ε loge µ 36 log max, σ r

14 Mauscrip uder review by AISTATS 09 We ow focus o he sages where s > 6 max, σ r σ Bu firs, oice ha Lemma 9 implies ha: ε N, r r σ r log log, ad also, similarly, i N, s N, ε i,s σ ε log 36i log s s This implies ha, wih probabiliy a leas, for every N ad s s := 6 max, σ r I oly remais o oice ha σ r r + ε,s+τ : + + ε,s log 8 log + r σ r 36 s log log s log log + σr + σ ε loge σ ε s 36 s log log 36 log This iequaliy is a cosequece of he fac ha a + λb a + b as soo as λ resul 6 max,a/b max, σ r, ad his gives he B UCB α ad ETC: proofs Our exac heorems are he wo followig Theorem Wih probabiliy greaer ha, UCB α wih α > reurs he bes arm a a sage d wih τ d C + C log + C log logc maxlog, logc + C log logc maxlog, logc, wih C = σ miα +, 6α α + ad C = α+ α C + Wih he same probabiliy, he regre Rτ d of UCB α a he ime of decisio verifies σ Rτ d miα + 6α, α + log + log logc maxlog, logc Theorem 3 Wih probabiliy greaer ha, ETC reurs he bes arm a a sage τ d wih τ d 3σ log + log log3σ maxlog, log3σ Wih he same probabiliy, he regre R ET C of ETC a he ime of decisio verifies Rτ d 6σ log + log log3σ maxlog, log3σ We firs prove a geeric lemma, which will also be useful i he o-iid case Lemma Cosider he wo arms problem where A is he bes arm Le 0, ] ad ε N be a sequece such ha wih probabiliy, for all A, B N we have he coceraio iequaliies ˆr A A + ε A µ A ad ˆr B B ε B µ B Suppose ha for all 0 N, ε ε C +

15 Mauscrip uder review by AISTATS 09 Le a algorihm be such ha i pulls each arm 0 imes, he pulls arg max i A,B ˆr i i + αε i for α >, ad akes a decisio if for some i, j A, B, ˆr i i ε i > ˆr i j + ε j The wih probabiliy, he algorihm correcly reurs A ad a all imes afer 0 ad prior o he decisio ε B miα +, α + ε A α 6α α + C, miα +, 6α α + C + C roof We prove ha he umber of pulls of he bes arm A is bouded by a fucio of he umber of pulls of B, which is iself bouded sice i is he worse arm Relaio bewee A ad B whe A is pulled No decisio was ake ye: ˆr A A ε A ˆr B B + ε B, A is pulled: ˆr A A + αε A ˆr B B + αε B Subrac from o obai α + ε A α ε B Equivalely, ε A oly whe A is chose ad grows a mos by C, we have ha for all sages, α + ε A α ε + C B α+ α ε B Sice he lef-had-side grows Upper boud o B Whe B is pulled, ˆr A A + αε A ˆr B B + αε B Wih probabiliy, for all A ad A we also have he coceraio iequaliies ˆr A A + ε A µ A ad ˆr B B ε B µ B Hece From his iequaliy we ca ge ha ε B mos by, we have for all sages µ A + α ε A µ B + α + ε B α+ Sice he lef-had-side grows oly whe B is pulled ad grows a α + ε B + C I order o ge aoher boud, releva whe α is big, we wrie ha whe o decisio is ake ad coceraio holds, we have µ A ε A ˆr A A ε A ˆr B B + ε B µ B + ε B This leads o ε B + ε A Whe B is pulled, we also have he iequaliy α + ε B α ε A, such ha This gives a secod iequaliy for B For all sages, Boud o he decisio ime ε B + α + α = ε B ε B 6α α + C α α We have he followig iequaliies for all sages prior o he decisio, ε B miα +, α + ε A α 6α α + C, miα +, 6α α + C + C

16 roof of Theorem We apply Lemma wih ε = ad for which we ca ake 0 = 3 ad C = prior o he decisio, B log log B σ A log log A Mauscrip uder review by AISTATS 09 σ σ log/ < miα +, σ α + α log log, for which coceraio holds wih probabiliy σ We obai he followig iequaliies for all sages 6α α +, miα +, 6α α + + We iroduce he oaios C = σ miα + 6α, α + ad C = α+ α C + A he ime of decisio, B C log + log logc maxlog, logc A C log + log logc maxlog, logc where L regroups he doubly logarihmic erms = A + B C + C log + L, The regre is B Thus wih probabiliy greaer ha, σ R d miα + 6α, α + log + log logc maxlog, logc roof of Theorem 3 Le = / We cosider oly eve sages Le ε = ake we have ˆr A ˆr B ε σ Wih probabiliy, for all, we have he coceraio iequaliy ˆr A ˆr B ε Combiig he wo iequaliies we obai ha as log as o decisio is ake, ε Tha is, log log 6σ 6σ log + log log3σ This boud o gives boh a boud o he regre ad o he decisio ime log log As log as o decisio is maxlog, log3σ Lemma 5 Le a, b > e ad a b If a + b log logmaxa, b logb, he a + b log log roof For b, he fucio b log log is icreasig Le x be he soluio of x b log log x = a We will show ha he proposed is bigger ha x Case : a b log logx The x = a + b log logx a ad a a + b log loga, such ha a + b log loga a + b log loga + b log loga, from which we coclude ha x a + b log loga Sice is bigger ha he laer, i is bigger ha x Case : a b log logx The x b log log x If x > b log b he x b log b This implies ha hece x a + b log logb logb x log log x > b log b log logb log b a + b log logb logb a + b log loga + b log logb logb, I boh cases, he proposed is bigger ha x, hece i verifies he waed iequaliy b We obai ha

17 Mauscrip uder review by AISTATS 09 C UCB-MM α ad ETC-MM: proofs Lemma 6 Assume ha σ r ad γ >, ad le be defied by he followig equaio σr + σ ε loge 36 log max, σ r = γ 3 he γσ r log + O + σ ε σr log log + O roof Cosider firs he followig equaio Σ log5 = C ad deoe by 0 is soluio I follows from sraighforward compuaios ha, 0 Σ C log + 5 logσ C + 0log log Σ + 5 log = Σ C C log +, O where logx = max5, logx We ow go back o Equaio 3, ad assume for he mome ha he soluio is such ha σ r σ ε Moreover, i is clear ha γσ r log 36 σr γσ 5 r =: As a cosequece, if we deoe by Σ = σr + σ ε log, he is such ha Σ log 5 γ, where = 36σ r σ ε So a he ed, we have proved ha which gives he resul = γσ log + O = γσ r log + + σ ε O σr log log + O Corollary 7 The decisio ime of UCB ad ETC correspods respecively o he soluio of Equaio 3 wih γ = for UCB ad γ = 8 for ETC ad γ = 6 for ETC Theorem 8 Give > 0 ad α, he decisio ime of UCB-MM α is such ha, wih probabiliy a leas, d σ r α + mi α + 6α α, + α + log + O + σ r log log + O Moreover, o he same eve, he regre of UCB-MM α a decisio ime saisfies R d 8σr mi α +, α + α log + O + σ r log log + O roof The proof is almos ideical o he iid case The mai differece is he chage i error erms Ideed, for N, we defie σr + σ ε loge 36 ε = log max, σ r

18 Mauscrip uder review by AISTATS 09 We apply Lemma wih his ε, 0 = 0 ad C = I yields ha for all sages prior o he decisio, ε B miα +, α + ε A α 6α α +, miα +, 6α α + + We ow iroduce he oaios γ = miα +, 6α α + ad γ = α+ α γ + A he ime of decisio, B = γ σ r A = γ σ r d = A + B = γ + γ σ r As a cosequece, o he same eve, R d log + O + σ r log log + O log + O + σ r log log + O log + O + σ r log log + O = σ r α + α mi α +, 8σr mi α +, + σ r log log + O α + α 6α α + + log + O log + + σ r O log log + O D Saic populaio I he saic seig, all users are allocaed o populaios A ad B from he begiig of he es, ad we oly cosider he case where he size of boh populaios are equal, eve hough he geeralizaio o differe populaio size is almos sraighforwaed Fiie fixed horizo: boud o he uplif The firs baselie is o wai uil some horizo T ad perform a saisical es based o a cofidece roposiio 9 I he saic seig, he followig holds wih probabiliy a leas, 8 σ ˆr A ˆr T B r + σ ε T log Therefore, he procedure waiig uil he horizo T o selec he B if ˆ T is greaer ha he rhs erm has a liear regre of RT = / ad is guaraeed o be, T -AC 3σ r + σ ε T log The proof is a direc cosequece of sadard coceraio iequaliies Adapive decisio ime Isead of waiig for a fixed arbirary horizo T, he decisio ca ofe be ake before, a he cos of usig maximal coceraio iequaliies, ha are valid a all sages

19 Mauscrip uder review by AISTATS 09 Theorem 0 I he saic seig, i holds ha, for all N ad wih probabiliy a leas, ˆr A ˆr B 8σr log 8 log 3 log + As a cosequece, ETC ca ake a correc decisio wih probabiliy a leas if he umber of users is greaer ha 3σ r log ad he, if we deoe by η := 3σr log, he decisio will be ake before he ime sep 3 η log + O