Bandits : optimality in exponential families

Transcription

1 Bandits : optimality in exponential families Odalric-Ambrym Maillard IHES, January 2016 Odalric-Ambrym Maillard Bandits 1 / 40

2 Introduction 1 Stochastic multi-armed bandits 2 Boundary crossing probabilities 3 Analysis for exponential families Odalric-Ambrym Maillard Bandits 2 / 40

3 Stochastic multi-armed bandits

4 Application examples Clinical trials K possible treatments (unknown effects) What treatment allocate to each patient, depending on observed effects on previous patients? Online advertising K ads that can be displayed What ad show to each user, depending on observed clicks from past users? Odalric-Ambrym Maillard Bandits 4 / 40

5 The stochastic Multi-Armed Bandit setting Setting Set of choices A Each a A is associated with an unknown probability distribution ν a D with mean µ a Game The game is sequential: At each round t 1, Goal the player first picks an arm A t A based on previous observations then receives (and sees) a stochastic payoff Y t ν At [ T ] Maximize the cumulative payoff E Y t or equivalently t=1 minimize the regret at round T : [ T ] def R T = T µ E Y t where t=1 µ = max{ µ a ; a A }, a argmax{ µ a ; a A } Odalric-Ambrym Maillard Bandits 5 / 40

6 Applications of Bandits What book to display? (+Few interactions) What drug is the most efficient? (+Economic/Human cost) What power to inject in the electrical network? (+Fast decisions) What news to display? (+Short lifespan) What action to choose? (+Risk Aversion) How to organize future cities? (+Long-term decisions) Odalric-Ambrym Maillard Bandits 6 / 40

7 Core problem Decision Making under Uncertainty Exploration versus Exploitation Sample more information or trust current estimates? Optimism in face of (modeled) uncertainty Build empirical confidence bounds on value Choose point with highest value Odalric-Ambrym Maillard Bandits 7 / 40

8 Some historical facts 1933: Introduction by Thompson ("Bayesian" way) 1952: Frequentist formalization by Robbins 1985: Lai and Robbins first lower-performance bounds 1996: Burnetas and Katehakis get general lower-performance bounds 2002: Upper-Confidence Bound (UCB) algorithm gets first non-asymptotic guarantee 2010+: Provably optimal strategies KL-UCB, D-MED, Thompson Sampling for specific distributions This talk: Extension to general exponential families Odalric-Ambrym Maillard Bandits 8 / 40

9 Typical performance guarantees Lower bound (Burnetas and Katehakis 96) For all "consistent" algorithm, for arbitrary D P([0, 1]) lim inf T R T log(t ) a A µ µ a K inf (ν a, µ ), where K inf (ν a, µ ) = inf{kl(ν a, ν) : ν D, E ν [Y ] > µ } Upper bounds (Cappe, Garivier, M, Munos, Stoltz 2013) KL-UCB achieves for specific D (eg 1-dim exponential families): R T a A µ µ a K inf (ν a, µ ) log(t ) + log(t )2/3 Odalric-Ambrym Maillard Bandits 9 / 40

10 The KL-UCB algorithm Pull arm A t+1 that maximizes the upper bound where { ( ) sup E ν [Y ]: ν D, KL Π D ( νa,na(t),ν ) f (t) } N a (t) ν a,n = 1 n nm=1 δ Xa,m, N a (t) = t s=1 I{A s = a} Π D : M 1 (R) D projection on D f : N R non-decreasing Build empirical distribution for arm a Odalric-Ambrym Maillard Bandits 10 / 40

11 The KL-UCB algorithm Pull arm A t+1 that maximizes the upper bound where { ( ) sup E ν [Y ]: ν D, KL Π D ( νa,na(t),ν ) f (t) } N a (t) ν a,n = 1 n nm=1 δ Xa,m, N a (t) = t s=1 I{A s = a} Π D : M 1 (R) D projection on D f : N R non-decreasing Project it on D Odalric-Ambrym Maillard Bandits 10 / 40

12 The KL-UCB algorithm Pull arm A t+1 that maximizes the upper bound where { ( ) sup E ν [Y ]: ν D, KL Π D ( νa,na(t),ν ) f (t) } N a (t) ν a,n = 1 n nm=1 δ Xa,m, N a (t) = t s=1 I{A s = a} Π D : M 1 (R) D projection on D f : N R non-decreasing Threshold Consider all ν D close to this distribution Odalric-Ambrym Maillard Bandits 10 / 40

13 The KL-UCB algorithm Pull arm A t+1 that maximizes the upper bound where { ( ) sup E ν [Y ]: ν D, KL Π D ( νa,na(t),ν ) f (t) } N a (t) ν a,n = 1 n nm=1 δ Xa,m, N a (t) = t s=1 I{A s = a} Π D : M 1 (R) D projection on D f : N R non-decreasing Compute the maximal mean of these candidate distributions Odalric-Ambrym Maillard Bandits 10 / 40

14 The KL-UCB+ algorithm Pull arm A t+1 that maximizes the upper bound ( ) { ( ) ) f t/n a (t) } sup E ν [Y ]: ν D, KL Π D ( νa,na(t),ν N a (t) where ν a,n = 1 n nm=1 δ Xa,m, N a (t) = t s=1 I{A s = a} Π D : M 1 (R) D projection on D f : N R non-decreasing Odalric-Ambrym Maillard Bandits 11 / 40

15 From Regret to Boundary Crossing Probabilities (a) The regret is given by R T = (µ µ a )E[N a (T )] {}}{ (b) {}}{ a A a A(µ log(t ) µ a ) K inf (ν a, µ ) + log(t ) 2/3 Theorem Let 0 < ε < min{µ µ a, a A} Then, the number of pulls of a sub-optimal arm a A by Algorithm KL-UCB satisfies E [ N T (a) ] T 1 T { } P K inf (Π D ( ν a,n ), µ ε) f (T )/n n=1 }{{} (a) Easy term, via Sanov } { ( P K inf ΠD ( ν a,n a (t)), µ ε ) f (t)/n a (t) t= A } {{ } (b) Difficult: Boundary Crossing Probability Odalric-Ambrym Maillard Bandits 12 / 40

16 From Regret to Boundary Crossing Probabilities (a) The regret is given by R T = (µ µ a )E[N a (T )] {}}{ (b) {}}{ a A a A(µ log(t ) µ a ) K inf (ν a, µ ) + log(t ) 2/3 Theorem Let 0 < ε < min{µ µ a, a A} Likewise, the number of pulls of a sub-optimal arm a A by Algorithm KL-UCB + satisfies E [ N T (a) ] T 1 T { } P K inf (Π D ( ν a,n ), µ ε) f (T /n)/n n=1 } {{ } (a) Easy term, via Sanov } { ( P K inf ΠD ( ν a,n a (t)), µ ε ) f (t/n a (t))/n a (t) t= A } {{ } (b) Difficult: Boundary Crossing Probability Odalric-Ambrym Maillard Bandits 12 / 40

17 Exponential families E parametric family of distributions of the form where ν θ (x) = exp ( θ, F (x) ψ(θ) ) ν 0 (x), F : X R K, ν 0 P(X ): reference function/measure { } def θ Θ E = θ R K ; ψ(θ) < natural parameter ψ(θ) def ( ) = log exp θ, F (x) ν 0 (dx): log-partition function Example X Bernoulli: X = {0, 1}, F : x x (K = 1), Θ D = R, ψ(θ) = log(1 + e θ ) B(µ): parameter θ = log(µ/(1 µ)) Gaussian: X = R, F : x (x, x 2 )(K = 2), Θ D = R R, ( ) ψ(θ) = θ2 1 4θ log π θ 2 N (µ, σ 2 ): parameter θ = ( µ, 1 ) σ 2 2σ 2 Odalric-Ambrym Maillard Bandits 13 / 40

18 Basic properties of regular exponential families E νθ (F (X)) = ψ(θ) Odalric-Ambrym Maillard Bandits 14 / 40

19 Basic properties of regular exponential families E νθ (F (X)) = ψ(θ) KL(ν θ, ν θ ) = B ψ (θ, θ ) = θ θ, ψ(θ) ψ(θ) + ψ(θ }{{}}{{} ) }{{} Bregman div natural dual Odalric-Ambrym Maillard Bandits 14 / 40

20 Basic properties of regular exponential families E νθ (F (X)) = ψ(θ) KL(ν θ, ν θ ) = B ψ (θ, θ ) = θ θ, ψ(θ) ψ(θ) + ψ(θ }{{}}{{} ) }{{} Bregman div natural dual Estimation for regular family: Odalric-Ambrym Maillard Bandits 14 / 40

21 Basic properties of regular exponential families E νθ (F (X)) = ψ(θ) KL(ν θ, ν θ ) = B ψ (θ, θ ) = θ θ, ψ(θ) ψ(θ) + ψ(θ }{{}}{{} ) }{{} Bregman div natural dual Estimation for regular family: Let {Xi } i n ν θ, ν n = 1 n n i=1 δ X i Odalric-Ambrym Maillard Bandits 14 / 40

22 Basic properties of regular exponential families E νθ (F (X)) = ψ(θ) KL(ν θ, ν θ ) = B ψ (θ, θ ) = θ θ, ψ(θ) ψ(θ) + ψ(θ }{{}}{{} ) }{{} Bregman div natural dual Estimation for regular family: Let {Xi } i n ν θ, ν n = 1 n n i=1 δ X i def Fn = 1 n n i=1 F (X i) = (F (X)) ψ(θ E νn D ) In the sequel, we would like to have Π E ( ν a,n) = ν θn for some θ n Θ E Odalric-Ambrym Maillard Bandits 14 / 40

23 Ṭechnical assumption: Well-defined empirical parameter θ n We assume that θ Θ and F n ψ(θ ρ ) where Θ is convex Neighborhood Θ 2ρ Θ I where Θ2ρ = {θ : inf θ Θ θ θ < 2ρ} Θ I def = { θ Θ E ; 0 < λ MIN ( 2 ψ(θ)) λ MAX ( 2 ψ(θ)) < with λ MIN/MAX (M): min/max eigenvalues of a sdp matrix M 2ρ < ρ = d(θ, Θ c I ) }, θ Θ Θ 2ρ Θ I Θ E Then we can form Π E ( ν a,n) = ν θn with θ n Θ ρ and K inf ( ΠD ( ν a,n), µ ε ) = inf{b ψ ( θ n, θ) : E νθ [X] µ ε} Odalric-Ambrym Maillard Bandits 15 / 40

24 Boundary crossing probabilities for general exponential families

25 What it is about { ( P θ K inf ΠE ( ν a,n a (t)), µ ε ) } f (t/n a (t))/n a (t) P θ { t n=1 } inf{b ψ ( θ n, θ) : E νθ [X] µ ε} f (t/n)/n Eg for canonical exp family of dimension 1: Unique θ = θ ε R such that E νθ [X] = µ ε, thus: P θ { t n=1 } B ψ ( θ n, θε) f (t/n)/n We want this to be o(1/t) for well-chosen f Odalric-Ambrym Maillard Bandits 17 / 40

26 Known results Theorem (Cappé, Garivier, M, Munos, Stoltz 2013) For canonical (F (x) = x X ) exp families of dimension K = 1: P θ { t n=1 B ψ ( θ n, θ ) f ( t ) } /n e f (t) log(t) e f (t) For f (x) = log(x) + ξ log log(x), the bound is O( log2 ξ (t) t ) A similar result is shown for discrete distributions Limitations Requires ξ > 2, while experiments shows ξ 0 still works (and even better) Only for threshold f (t)/n, does not work for f (t/n)/n Consider θ : does not take advantage of ε > 0 Odalric-Ambrym Maillard Bandits 18 / 40

27 Forgetted result Theorem (Lai, 1988; exponential family of dimension K) For γ (0, 1) let C γ (θ) = {θ R K : θ, θ γ θ θ } Then, for f (x) = log(x) + ξ log log(x) it holds for all θ Θ ρ such that θ θ 2 δ t, where δ t 0, tδ t as t, { t P θ n=1 ( θ n Θ ρ B ψ ( θ n, θ ) f ) } t/n /n ψ( θ n ) ψ(θ ) C γ (θ θ ) ( t θ θ 2 ) = O log ξ 1+K/2 (t θ θ 2 ) t Pros ξ > K/2 1 is enough For f (t/n) Cons Blue term Asymptotic "Boundary crossing problems for sample means", 1988 Odalric-Ambrym Maillard Bandits 19 / 40

28 Illustrative experiment on Bernoulli distributions Regret as function of time for KL-UCB and KL-UCB+ (mean and quantiles) Odalric-Ambrym Maillard Bandits 20 / 40

29 This talk Explain the proof technique for this result: 1 Peeling (standard) and covering 2 Change of measure argument 3 Localization + change of measure 4 Concentration (standard) 5 Fine tuning (gain log 2 factor) Show that Lai s theorem can be made non-asymptotic, and used to control the general case Odalric-Ambrym Maillard Bandits 21 / 40

30 Analysis and proof technique (up to some details for presentation purpose) P θ { θ } 1 n t n Θ ρ K inf (Π E ( ν a,n), µ ε) f (t/n)/n }{{} E n

31 I Peeling Let n 0 = 1 < n 1 < < n It = t + 1 for some I t N { } I t 1 { } P θ E n P θ E n 1 n t i=0 n i n<n i+1 For some b > 1 and β (0, 1), use: b i def if i < I n i = t = log b (βt + β) t + 1 if i = I t, (valid sequence since n It 1 b log b (βt+β) < t + 1 = n It ) Odalric-Ambrym Maillard Bandits 23 / 40

32 II Cone covering 1 First: 2 Then K inf (Π E ( ν a,n), µ ε) = inf{b ψ ( θ n, θ) : θ Θ E, µ θ µ ε} = inf{b ψ ( θ n, θ ) : θ Θ E, µ θ µ ε} B ψ ( θ n, θ ) = B ψ ( θ n, θ ) B ψ (θ, θ ), ψ(θ ) ψ( θ n ) }{{} shift Odalric-Ambrym Maillard Bandits 24 / 40

33 II Cone covering { } Consider half-cone C γ ( ) = θ R K : θ, γ θ, then set { c } 1 c Cγ,K of directions st R K = C γ,k c=1 C γ ( c ) C γ ( ) invariant by rescaling of : We can choose c small enough so that θ c Θ ρ C γ,1 = 2, C γ,k when γ 1 Odalric-Ambrym Maillard Bandits 25 / 40

34 II Cone covering K inf (Π E ( ν a,n), µ ε) = inf{b ψ ( θ n, θ ) : θ Θ E, µ θ µ ε} B ψ ( θ n, θ c ) θ c Θ ρ, µ θ c µ ε, c [C γ,k ] Thus P θ { where E n,c,c = n i n<n i+1 E n } C γ,k min c c =1 P θ { n i n<n i+1 E n,c,c { θ n Θ ρ ψ(θ c ) F n C γ ( c) B ψ ( θ n, θ c ) f (t/n) } n } E n,c,c is almost the event appearing in Lai s theorem Odalric-Ambrym Maillard Bandits 26 / 40

35 Recap of steps I II { } I t 1 P θ E n 1 n t i=0 min c C γ,k P θ c =1 { n i n<n i+1 E n,c,c } Odalric-Ambrym Maillard Bandits 27 / 40

36 III Change of measure Lemma (Change of measure) If n nf (t/n) is non-decreasing and θ c Θ ρ, then P θ { n i n<n i+1 E n,c,c } ( C exp n ) i 2 v ρδc 2 2ni f (t/n i ) γδ c v ρ V ρ { } P θ c E n,c,c, n i n<n i+1 where v ρ = inf θ Θρ λ MIN ( 2 ψ(θ)), V ρ = sup θ Θρ λ MAX ( 2 ψ(θ)), δ c = c Concentration of B ψ ( θ n, θ c ) for distribution ν θ vs concentration of B ψ ( θ n, θ c ) for distribution ν θ c Odalric-Ambrym Maillard Bandits 28 / 40

37 III Change of measure: proof details for n iid variables look at the ratio dp θ Π n k=1 = ν θ (X k) dp θ c Π n k=1 ν θ c (X k ) ( = exp n c, F ( ) ) a,n n ψ(θ ) ψ(θ c ) ( = exp n c, ψ(θ c ) F a,n }{{} (a) ) n B ψ (θ c, θ ) }{{} (b) Odalric-Ambrym Maillard Bandits 29 / 40

38 III Change of measure: proof details for n iid variables look at the ratio ( dp θ = exp dp θ c For (b): n c, ψ(θ c ) F a,n }{{} (a) B ψ (θ c, θ ) 1 2 v ρδ 2 c ) n B ψ (θ c, θ ) }{{} (b) Odalric-Ambrym Maillard Bandits 29 / 40

39 III Change of measure: proof details for n iid variables look at the ratio ( dp θ = exp dp θ c For (b): n c, ψ(θ c ) F a,n }{{} (a) B ψ (θ c, θ ) 1 2 v ρδc 2 For (a): (up to going from c to c) ) n B ψ (θ c, θ ) }{{} (b) c, ψ(θ c ) F n γ c ψ(θ c ) F n γδ c v ρ θ n θ + c γδ c v ρ 2 V ρ B ψ ( θ n, θ c ) γδ c v ρ 2f (t/n) V ρ n Odalric-Ambrym Maillard Bandits 29 / 40

40 IV Localization By construction: ψ(θ c ) F n C p ( c ) c, ψ(θ c ) F n γ c ψ(θ c ) F n, Thus, we look at ψ(θ c ) F n Decomposition, for some constant ε t,i,c > 0 { } P θ c n i n<n i+1 E n,c P θ c { E n,c ψ(θ c ) F n < ε t,i,c n i n<n i+1 }{{} (a) Requires some care { + P θ c E n,c ψ(θ c ) F } n ε t,i,c n i n<n i+1 }{{} (b) "Standard" concentration } Odalric-Ambrym Maillard Bandits 30 / 40

41 V Localized change of measure and concentration Lemma (Concentration (admitted)) Under some technical assumption, ( (b) 2K exp n2 i ε2 ) t,i,c 2KV 2ρ n i+1 2KV For ε t,i,c = 2ρ n i+1 f (t/n i+1 ), then (b) 2K exp( f (t/n ni 2 i+1 )) Lemma (Localized change of measure) For the ε t,i,c considered, we have ( ) (a) 2C b,ρ exp f (t/n i+1 ) f (t/n i+1 ) K/2, for an explicit constant C b,ρ = max { 2KbV2ρ ρ 2 v 2 ρ, 4, 8K 2 b 2 V2ρ 2 } K/2 (K+2)vρ 2 Odalric-Ambrym Maillard Bandits 31 / 40

42 Recap { P θ 1 n t } θ n Θ ρ K inf (Π E ( ν a,n), µ ε) f (t/n)/n End of the proof: C γ,k I t 1 c=1 i=0 }{{} Peeling+Covering [ e f (t/n i+1) Ce n i 2 vρδ2 2n c γδ cv i f (t/n i ) ρ Vρ }{{} Change of measure ] 2CC b,ρ f (t/n i+1 ) K/2 + 2K } {{ } Localized change of measure + Concentration "Change of measure" term kills n i δc 2 / log(tδc 2 ) terms Remains e f (tδ2 c ) f (tδc 2 ) K/2 / log(tδc 2 ) 1 log(tδ 2 tδc 2 c ) ξ+k/2 1 Odalric-Ambrym Maillard Bandits 32 / 40

43 Illustrative experiment on Bernoulli distributions Regret as function of time for KL-UCB and KL-UCB+ (mean and quantiles) Odalric-Ambrym Maillard Bandits 33 / 40

44 Back to V(a) Localized change of measure P θ c { Let θ c n i n<n i+1 E n,c ψ(θ c ) F n < ε t,i,c def = θ c Θ ρ E n,c { θ n Θ ρ B ψ ( θ n, θ c) f (t/n)/n} } Odalric-Ambrym Maillard Bandits 34 / 40

45 Back to V(a) Localized change of measure: idea Change of measure from P θ c to the mixture Q B ( ) = θ B P θ ( ) where B = Θ 2ρ Ball(θ c, 2ε t,i,c v ρ ) We have Ball( θ n, min{ ε t,i,c v ρ, ρ}) B Θ 2ρ Study for n iid points the ratio dp θ dq B = = [ [ θ B θ B Π n k=1 ν θ (X ] 1 k) Π n k=1 ν θ(x k ) dθ ( exp n θ θ, F a,n n ( ψ(θ ) ψ(θ) ) ] 1 )dθ }{{} h(θ ) Odalric-Ambrym Maillard Bandits 35 / 40

46 Back to V(a) Localized change of measure: idea Studying h, it comes for θ Θ 2ρ h(θ ) nb ψ ( θ n, θ) n 2 (θ θ n ) 2 ψ( θ)(θ θ n ) f (t/n) n 2 θ θ n 2 V 2ρ (convexity of Θ 2ρ ) Thus using that Ball( θ n, min{ ε t,i,c v ρ, ρ}) B dp θ dq B [ θ B e f (t/n i+1) exp (f (t/n) n ] 1 2 θ θ n 2 V 2ρ )dθ [ ( exp n ) 1 i+1 x Ball(0,min{ρ, ε t,i,c vρ }) 2 x 2 V 2ρ dx], Odalric-Ambrym Maillard Bandits 36 / 40

47 Back to V(a) Localized change of measure: idea Thus for our event of interest Ω, [ } ] 1 P θ c {Ω e f (t/n i+1) e n { } i+1 x Ball(0,min{ρ, ε 2 2 V 2ρ dx P t,i,c θ Ω dθ vρ }{{ }) B }{{}} B Volume of a Gaussian ball And after some easy computations } ( ) { P θ c {Ω 2 exp f (t/n i+1 ) min ρ 2 vρ 2, ε 2 t,i,c, (K + 2)v ρ 2 } K/2 2 K ε K Kn i+1 V t,i,c 2ρ 2KV We conclude using ε t,i,c = 2ρ n i+1 f (t/n i+1 ) ni 2, Odalric-Ambrym Maillard Bandits 37 / 40

48 Conclusion For a general exponential family E, known, we can* Exhibit a concentration of measure for the Bregman divergence induced by the family: in finite time for all dimension K powerful proof technique Get fine understanding of the threshold f (x) = log(x) + ξ log log(x) improve over existing work: ξ > K/2 1 is enough + hold for all K + KL-UCB+ as well closely follows what is observed in experiments show this was known 30 years ago by Lai but we still need to know the family E * we skipped some regularity restrictions on E Odalric-Ambrym Maillard Bandits 38 / 40

49 Teaser Regret as function of time for KL-UCB, KL-UCB+ and BESA (mean and quantiles) BESA does not know the family E "Sub-sampling for the multi-armed bandit", A Baransi et al 2014 Odalric-Ambrym Maillard Bandits 39 / 40

50 Looking for Interns, PhD candidates, collaborations Thank you