On the Complexity of Best Arm Identification in Multi-Armed Bandit Models

Transcription

1 On the Complexity of Best Arm Identification in Multi-Armed Bandit Models Aurélien Garivier Institut de Mathématiques de Toulouse Information Theory, Learning and Big Data Simons Institute, Berkeley, March 2015

2 Roadmap Simple Multi-Armed Bandit Model 1 Simple Multi-Armed Bandit Model 2 Complexity of Best Arm Identification Lower bounds on the complexities Gaussian Feedback Binary Feedback

3 Simple Multi-Armed Bandit Model The (stochastic) Multi-Armed Bandit Model Environment K arms with parameters θ = (θ 1,..., θ K ) such that for any possible choice of arm a t {1,..., K} at time t, one receives the reward X t = X at,t where, for any 1 a K and s 1, X a,s ν a, and the (X a,s ) a,s are independent. Reward distributions ν a F a parametric family, or not: canonical exponential family, general bounded rewards Example Bernoulli rewards: θ [0, 1] K, ν a = B(θ a ) Strategy The agent s actions follow a dynamical strategy π = (π 1, π 2,... ) such that A t = π t (X 1,..., X t 1 )

4 Simple Multi-Armed Bandit Model Real challenges Randomized clinical trials original motivation since the 1930 s dynamic strategies can save resources Recommender systems: advertisement website optimization news, blog posts,... Computer experiments large systems can be simulated in order to optimize some criterion over a set of parameters but the simulation cost may be high, so that only few choices are possible for the parameters Games and planning (tree-structured options)

5 Simple Multi-Armed Bandit Model Performance Evaluation: Cumulated Regret Cumulated Reward: S T = T t=1 X t Goal: Choose π so as to maximize E [S T ] = = T t=1 a=1 K E [ E [X t 1{A t = a} X 1,..., X t 1 ] ] K µ a E [Na π (T)] a=1 where Na π (T) = t T 1{A t = a} is the number of draws of arm a up to time T, and µ a = E(ν a ). Regret Minimization: maximizing E [S T ] minimizing R T = Tµ E [S T ] = (µ µ a )E [Na π (T)] a:µ a<µ where µ max{µ a : 1 a K}

6 Simple Multi-Armed Bandit Model Upper Confidence Bound Strategies UCB [Lai&Robins 85; Agrawal 95; Auer&al 02] Construct an upper confidence bound for the expected reward of each arm: S a (t) log(t) + N a (t) 2N }{{} a (t) }{{} estimated reward exploration bonus Choose the arm with the highest UCB It is an index strategy [Gittins 79] Its behavior is easily interpretable and intuitively appealing Listen to Robert Nowak s talk tomorrow!

7 Optimality? Simple Multi-Armed Bandit Model Generalization of [Lai&Robbins 85] Theorem [Burnetas and Katehakis, 96] If π is a uniformly efficient strategy, then for any θ [0, 1] K, E [ N a (T) ] 1 lim inf T log(t) K inf (ν a, µ ) where δ 12 K inf (ν a, µ ) = inf { K(ν a, ν ) : ν F a, E(ν ) µ } νa Idea: change of distribution δ 0 K inf (νa, µ ) ν µ δ 1

8 Simple Multi-Armed Bandit Model Reaching Optimality: Empirical Likelihood The KL-UCB Algorithm, AoS 2013 joint work with O. Cappé, O-A. Maillard, R. Munos, G. Stoltz Parameters: An operator Π F : M 1 (S) F; a non-decreasing function f : N R Initialization: Pull each arm of {1,..., K} once for t = K to T 1 do compute for each arm a the quantity { ( U a (t) = sup E(ν) : ν F and KL Π F (ˆνa (t) ) ), ν f (t) } N a (t) end for pick an arm A t+1 arg max U a (t) a {1,...,K}

9 Regret bound Simple Multi-Armed Bandit Model Theorem: Assume that F is the set of finitely supported probability distributions over S = [0, 1], that µ a > 0 for all arms a and that µ < 1. There exists a constant M(ν a, µ ) > 0 only depending on ν a and µ such that, with the choice f (t) = log(t) + log ( log(t) ) for t 2, for all T 3: E [ N a (T) ] log(t) 36 ( ) ( 4/5 ( ) K inf νa, µ ) + log(t) log log(t) (µ ) 4 ( ) 72 + (µ ) 4 + 2µ (log(t) ) 4/5 ( (1 µ ) K inf νa, µ ) 2 + (1 µ ) 2 M(ν a, µ ) 2(µ ) 2 ( log(t) ) 2/5 + log( log(t) ) ( K inf νa, µ ) + 2µ ( (1 µ ) K inf νa, µ )

10 Regret bound Simple Multi-Armed Bandit Model Theorem: Assume that F is the set of finitely supported probability distributions over S = [0, 1], that µ a > 0 for all arms a and that µ < 1. There exists a constant M(ν a, µ ) > 0 only depending on ν a and µ such that, with the choice f (t) = log(t) + log ( log(t) ) for t 2, for all T 3: E [ N a (T) ] log(t) 36 ( ) ( 4/5 ( ) K inf νa, µ ) + log(t) log log(t) (µ ) 4 ( ) 72 + (µ ) 4 + 2µ (log(t) ) 4/5 ( (1 µ ) K inf νa, µ ) 2 + (1 µ ) 2 M(ν a, µ ) 2(µ ) 2 ( log(t) ) 2/5 + log( log(t) ) ( K inf νa, µ ) + 2µ ( (1 µ ) K inf νa, µ )

11 Roadmap Complexity of Best Arm Identification 1 Simple Multi-Armed Bandit Model 2 Complexity of Best Arm Identification Lower bounds on the complexities Gaussian Feedback Binary Feedback

12 Best Arm Identification Strategies A two-armed bandit model is a pair ν = (ν 1, ν 2 ) of probability distributions ( arms ) with respective means µ 1 and µ 2 a = argmax a µ a is the (unknown) best arm Strategy = a sampling rule (A t ) t N where A t {1, 2} is the arm chosen at time t (based on past observations) a sample Z t ν At is observed a stopping rule τ indicating when he stops sampling the arms a recommendation rule â τ {1, 2} indicating which arm he thinks is best (at the end of the interaction) In classical A/B Testing, the sampling rule A t is uniform on {1, 2} and the stopping rule τ = t is fixed in advance.

13 Best Arm Identification Joint work with Emilie Kaufmann and Olivier Cappé (Telecom ParisTech) Goal: design a strategy A = ((A t ), τ, â τ ) such that: Fixed-budget setting τ = t p t (ν) := P ν (â t a ) as small as possible Fixed-confidence setting P ν (â τ a ) δ E ν [τ] as small as possible See also: [Mannor&Tsitsiklis 04], [Even-Dar&al. 06], [Audibert&al. 10], [Bubeck&al. 11, 13], [Kalyanakrishnan&al. 12], [Karnin&al. 13], [Jamieson&al. 14]...

14 Two possible goals Goal: design a strategy A = ((A t ), τ, â τ ) such that: Fixed-budget setting τ = t p t (ν) := P ν (â t a ) as small as possible Fixed-confidence setting P ν (â τ a ) δ E ν [τ] as small as possible In the particular case of uniform sampling : Fixed-budget setting Fixed-confidence setting classical test of sequential test of (µ 1 > µ 2 ) against (µ 1 < µ 2 ) (µ 1 > µ 2 ) against (µ 1 < µ 2 ) based on t samples with probability of error uniformly bounded by δ [Siegmund 85]: sequential tests can save samples!

15 The complexities of best-arm identification For a class M bandit models, algorithm A = ((A t ), τ, â τ ) is... Fixed-budget setting consistent on M if ν M, p t (ν) = P ν (â t a ) 0 t From the literature ( p t (ν) exp ) Fixed-confidence setting δ-pac on M if ν M, P ν (â τ a ) δ t CH(ν) E ν [τ] C H (ν) log(1/δ) [Audibert&al. 10],[Bubeck&al 11] [Mannor&Tsitsiklis 04],[Even-Dar&al. 06] [Bubeck&al 13],... = two complexities ( κ B (ν) = inf A cons. [Kalanakrishnan&al 12],... 1 lim sup 1 t t(ν)) log p κ C (ν) = inf lim sup t A δ PAC δ 0 E ν[τ] log(1/δ for a probability of error δ, for a probability of error δ, budget t κ B (ν) log(1/δ) E ν [τ] κ C (ν) log(1/δ)

16 Changes of distribution Lower bounds on the complexities Theorem: how to use (and hide) the change of distribution Let ν and ν be two bandit models with K arms such that for all a, the distributions ν a and ν a are mutually absolutely continuous. For any almost-surely finite stopping time σ with respect to (F t ), K a=1 E ν [N a (σ)] KL(ν a, ν a) sup E F σ kl ( P ν (E), P ν (E) ), where kl(x, y) = x log(x/y) + (1 x) log ( (1 x)/(1 y) ). Useful remark: δ [0, 1], kl ( δ, 1 δ ) log δ,

17 General lower bounds Lower bounds on the complexities Theorem 1 Let M be a class of two armed bandit models that are continuously parametrized by their means. Let ν = (ν 1, ν 2 ) M. Fixed-budget setting Fixed-confidence setting any consistent algorithm satisfies any δ-pac algorithm satisfies lim sup t 1 t log p t(ν) K 1 (ν 1, ν 2 ) E ν [τ] K (ν 1,ν 2 ) log ( ) 1 2.4δ with K (ν 1, ν 2 ) with K (ν 1, ν 2 ) = KL(ν, ν 1 ) = KL(ν, ν 2 ) = KL(ν 1, ν ) = KL(ν 2, ν ) Thus, κ B (ν) 1 K (ν 1,ν 2 ) Thus, κ C (ν) 1 K (ν 1,ν 2 )

18 Gaussian Feedback Gaussian Rewards: Fixed-Budget Setting For fixed (known) values σ 1, σ 2, we consider Gaussian bandit models M = { ν = ( N ( µ 1, σ1) 2 (, N µ2, σ2 2 )) : (µ1, µ 2 ) R 2 }, µ 1 µ 2 Theorem 1: κ B (ν) 2(σ 1 + σ 2 ) 2 (µ 1 µ 2 ) 2 A strategy allocating t 1 = σ1 samples to arm 1 and σ 1 +σ 2 t t 2 = t t 1 samples to arm 1, and recommending the empirical best satisfies lim inf 1 t t log p t(ν) (µ 1 µ 2 ) 2 2(σ 1 + σ 2 ) 2 κ B (ν) = 2(σ 1 + σ 2 ) 2 (µ 1 µ 2 ) 2

19 Gaussian Feedback Gaussian Rewards: Fixed-confidence setting The α-elimination algorithm with exploration rate β(t, δ) chooses A t in order to keep a proportion N 1 (t)/t α if ˆµ a (t) is the empirical mean of rewards obtained from a up to time t, σt 2(α) { = σ2 1 / αt + σ2 2 /(t αt ), } τ = inf t N : ˆµ 1 (t) ˆµ 2 (t) > 2σ 2t (α)β(t, δ) recommends the empirical best arm â τ = argmax a ˆµ a (τ)

20 Gaussian Feedback Gaussian Rewards: Fixed-confidence setting From Theorem 1: E ν [τ] 2(σ 1 + σ 2 ) 2 ( ) 1 (µ 1 µ 2 ) 2 log 2.4δ σ 1 σ 1 +σ 2 -Elimination with β(t, δ) = log t δ + 2 log log(6t) is δ-pac and ɛ > 0, E ν [τ] (1 + ɛ) 2(σ 1 + σ 2 ) 2 (µ 1 µ 2 ) 2 log ( ) ( 1 + o ɛ log 1 ) 2.4δ δ 0 δ κ C (ν) = 2(σ 1 + σ 2 ) 2 (µ 1 µ 2 ) 2

21 Gaussian Rewards: Conclusion Gaussian Feedback For any two fixed values of σ 1 and σ 2, κ B (ν) = κ C (ν) = 2(σ 1 + σ 2 ) 2 (µ 1 µ 2 ) 2 If the variances are equal, σ 1 = σ 2 = σ, κ B (ν) = κ C (ν) = 8σ 2 (µ 1 µ 2 ) 2 uniform sampling is optimal only when σ 1 = σ 2 1/2-Elimination is δ-pac for a smaller exploration rate β(t, δ) log(log(t)/δ)

22 Binary Rewards: Lower Bounds Binary Feedback M = {ν = (B(µ 1 ), B(µ 2 )) : (µ 1, µ 2 ) ]0; 1[ 2, µ 1 µ 2 }, shorthand: K(µ, µ ) = KL (B(µ), B(µ )). Fixed-budget setting Fixed-confidence setting any consistent algorithm satisfies any δ-pac algorithm satisfies lim sup t 1 t log p t(ν) K 1 (µ 1, µ 2 ) E ν [τ] K (µ 1,µ 2 ) log ( ) 1 2δ (Chernoff information) K (µ 1, µ 2 ) > K (µ 1, µ 2 )

23 Binary Feedback Binary Rewards: Uniform Sampling For any consistent... For any δ-pac algorithm p t (ν) e K (µ 1,µ 2 )t E ν[τ] log(1/δ) 1 K (µ 1,µ 2 )... algorithm using p t (ν) e K(µ,µ 1)+K(µ,µ 2 ) 2 t E ν[τ] log(1/δ) 2 K(µ 1,µ)+K(µ 2,µ) uniform sampling with µ = f (µ 1, µ 2 ) with µ = µ 1+µ 2 2 Remark: Quantities in the same column appear to be close from one another Binary rewards: uniform sampling close to optimal

24 Binary Feedback Binary Rewards: Uniform Sampling For any consistent... For any δ-pac algorithm p t (ν) e K (µ 1,µ 2 )t E ν[τ] log(1/δ) 1 K (µ 1,µ 2 )... algorithm using p t (ν) e K(µ,µ 1)+K(µ,µ 2 ) 2 t E ν[τ] log(1/δ) 2 K(µ 1,µ)+K(µ 2,µ) uniform sampling with µ = f (µ 1, µ 2 ) with µ = µ 1+µ 2 2 Remark: Quantities in the same column appear to be close from one another Binary rewards: uniform sampling close to optimal

25 Binary Feedback Binary Rewards: Fixed-Budget Setting In fact, κ B (ν) = 1 K (µ 1, µ 2 ) The algorithm using uniform sampling and recommending the empirical best arm is very close to optimal

26 Binary Feedback Binary Rewards: Fixed-Confidence Setting δ-pac algorithms using uniform sampling satisfy E ν [τ] log(1/δ) 1 I (ν) with I (ν) = K ( µ1, µ 1+µ 2 2 ) ( + K µ2, µ 1+µ 2 ) 2. 2 The algorithm using uniform sampling and { τ = inf t 2N : ˆµ 1 (t) ˆµ 2 (t) > log log(t) + 1 } δ E[τ] is δ-pac but not optimal: log(1/δ) 2 > 1 (µ 1 µ 2 ) 2 I. (ν) A better stopping rule NOT based on the difference of empirical means { τ = inf t 2N : t I (ˆµ 1 (t), ˆµ 2 (t)) > log log(t) + 1 } δ

27 Binary Feedback Binary Rewards: Conclusion Regarding the complexities: κ B (ν) = 1 K (µ 1,µ 2 ) κ C (ν) 1 K (µ 1,µ 2 ) > 1 K (µ 1,µ 2 ) Thus κ C (ν) > κ B (ν) Regarding the algorithms There is not much to gain by departing from uniform sampling In the fixed-confidence setting, a sequential test based on the difference of the empirical means is no longer optimal

28 Conclusion Complexity of Best Arm Identification Binary Feedback the complexities κ B (ν) and κ C (ν) are not always equal (and feature some different informational quantities) strategies using random stopping do not necessarily lead to a saving in terms of the number of sample used for Bernoulli distributions and Gaussian with similar variances, strategies using uniform sampling are (almost) optimal Generalization to m best arms identification among K arms

29 Binary Feedback Elements of Bibliography (see references therein!) 1 [Lai&Robins 85] T.L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1) :4-22, [Agrawal 95] R. Agrawal. Sample mean based index policies with O(log n) regret for the multi-armed bandit problem. Advances in Applied Probability, 27(4) : , [Auer&al 02] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2) : , [Even-Dar&al 06] Action elimination and stopping conditions for multi-armed bandit and reinforcement leraning problems, JMLR 7: , [Audibert&al 09] J-Y. Audibert, R. Munos, and Cs. Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science, 410(19), [Filippi &al 10] S. Filippi, O. Cappé, and A. Garivier. Optimism in reinforcement learning and Kullback-Leibler divergence. In Allerton Conf. on Communication, Control, and Computing, Monticello, US, [Cappé&al 13] O. Cappé, A. Garivier, O-A. Maillard, R Munos, G. Stoltz. Kullback-Leibler Upper Confidence Bounds for Optimal Sequential Allocation. Annals of Statistics (41:3) Jun pp [Abbasi-Yadkori&al 11] Yasin Abbasi-Yadkori, Dávid Pál, Csaba Szepesvári: Online Least Squares Estimation with Self-Normalized Processes: An Application to Bandit Problems. CoRR abs/ : (2011) 9 [Bubeck&Cesa-Bianchi 12] S. Bubeck and N. Cesa-Bianchi, Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems. Foundations and Trends in Machine Learning 5(1): (2012) 10 [Cappé&al 13] O. Cappé, A. Garivier, O-A. Maillard, R Munos, G. Stoltz. Kullback-Leibler Upper Confidence Bounds for Optimal Sequential Allocation. Annals of Statistics (41:3) Jun pp [Jamieson&al 14] K. Jamieson, M. Malloy, R. Nowak and S. Bubeck. lil UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits. COLT 2014 : [Kaufmann&al 15] E. Kaufmann, O. Cappé, A. Garivier, On the Complexity of Best Arm Identification in Multi-Armed Bandit Models, ArXiv: