Bayesian and Frequentist Methods in Bandit Models

Transcription

1 Bayesian and Frequentist Methods in Bandit Models Emilie Kaufmann, Telecom ParisTech Bayes In Paris, ENSAE, October 24th, 2013 Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 1 / 48

2 1 The multiarmed bandit problem 2 Bayesian bandits, frequentist bandits 3 Two Bayesian bandit algorithms Bayes-UCB Thompson Sampling 4 Bayesian algorithms for pure exploration? 5 Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 2 / 48

3 The multiarmed bandit problem 1 The multiarmed bandit problem 2 Bayesian bandits, frequentist bandits 3 Two Bayesian bandit algorithms Bayes-UCB Thompson Sampling 4 Bayesian algorithms for pure exploration? 5 Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 3 / 48

4 The multiarmed bandit problem Bandit model Bandit model A multi-armed bandit model is a set of K arms where Each arm a is a probability distribution ν a of mean µ a Drawing arm a is observing a realization of ν a Arms are assumed to be independent In a bandit game, at round t, a forecaster chooses arm A t to draw based on past observations, according to its sampling strategy observes reward X t ν At Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 4 / 48

5 The multiarmed bandit problem The regret minimization problem Our bandit problem: regret minimization µ = max µ a and a = argmax µ a a a The forecaster wants to maximize the reward accumulated during learning or equivalentely minimize its regret: [ n ] R n = nµ E X t He has to find a sampling strategy (or bandit algorithm) that t=1 realizes a tradeoff between exploration and exploitation Applications (with arms beeing Bernoulli random variables) Finding the best slot machine in a casino (just for the name!) Initial motivation: Sequential allocation of medical treatments Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 5 / 48

6 The multiarmed bandit problem The regret minimization problem A recent motivation for bandits: Online advertisement Yahoo!(c) has to choose between K different advertisements the one to display on its webpage for each user (indexed by t N). Ad number a unknown probability of click p a Unknown best advertisement a = argmax a p a X t,a B(p a ): (X t,a = 1)=(user t has clicked on ad a) Yahoo!(c): chooses ad A t to display for user number t observes whether the user has clicked or not: X t,at wants to maximize the click-through-rate How should Yahoo!(c) choose ad A t to display depending on the previous clicks of the (t 1) first users? Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 6 / 48

7 Bayesian bandits, frequentist bandits 1 The multiarmed bandit problem 2 Bayesian bandits, frequentist bandits 3 Two Bayesian bandit algorithms Bayes-UCB Thompson Sampling 4 Bayesian algorithms for pure exploration? 5 Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 7 / 48

8 Bayesian bandits, frequentist bandits Two probabilistic modellings Two probabilistic models K independent arms. µ = µ a Frequentist : θ 1,..., θ K unknown parameters (Y a,t ) t is i.i.d. with distribution ν θa with mean µ a highest expectation of reward. Bayesian : i.i.d. θ a π a (Y a,t ) t is i.i.d. conditionally to θ a with distribution ν θa At time t, arm A t is chosen and reward X t = Y At,t is observed Two measures of performance Minimize regret R n (θ) = E θ [ n t=1 µ µ At ] Minimize Bayes risk [ n ] Risk n = E µ µ At = t=1 R n (θ)dπ(θ) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 8 / 48

9 Bayesian bandits, frequentist bandits Two probabilistic models Frequentist tools, Bayesian tools Bandit algorithms based on frequentist tools use: MLE for the mean parameter of each arm confidence intervals for the parameter of each arm Bandit algorithms based on Bayesian tools use: Π t = (π1 t,..., πt K ) the current posterior over (θ 1,..., θ K ) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 9 / 48

10 Bayesian bandits, frequentist bandits Frequentist tools, Bayesian tools Two probabilistic models Bandit algorithms based on frequentist tools use: MLE for the mean parameter of each arm confidence intervals for the parameter of each arm Bandit algorithms based on Bayesian tools use: Π t = (π t 1,..., πt K ) the current posterior over (θ 1,..., θ K ) One can separate tools and objectives: Objective Frequentist Bayesian algorithms algorithms Regret?? Bayes risk?? Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 9 / 48

11 Bayesian bandits, frequentist bandits Bayesian algorithm and Bayes risk Bayesian algorithms minimizing Bayes risk Objective Frequentist Bayesian algorithms algorithms Regret?? Bayes risk?? Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 10 / 48

12 Bayesian bandits, frequentist bandits Bayesian algorithm and Bayes risk MDP formulation of the Bernoulli bandit game Benoulli bandits with uniform prior on the means: θ a = µ a i.i.d θ a U([0, 1]) = Beta(1, 1) πa t = Beta(S a (t) + 1, N a (t) S a (t) + 1) Matrix S t M K,2 summarizes the game : Line a gives the parameters of the Beta posterior over arm a, π t a S t can be seen as a state in a Markov Decision Process, and the optimal policy is (depending on the criterion) [ ] [ n ] arg max E γ t 1 X t or arg max E X t (A t) (A t) t=1 Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 11 / 48 t=1

13 Bayesian bandits, frequentist bandits The Finite-Horizon Gittins algorithm Bayesian algorithm and Bayes risk Gittins ([1979]) shows the optimal policy in the discounted case is an index policy: A t = argmax ν Disc (π t (a)). a Similarly, in the finite-horizon case (our setting), the optimal policy has an explicit formulation A t = argmax a The Finite-Horizon Gittins algorithm But... ν F H (π t (a), n t) minimizes minimizes the Bayes risk Risk n and display very good performance on frequentist problems! FH-Gittins indices are hard to compute the algorithm is heavily horizon-dependent Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 12 / 48

14 Bayesian bandits, frequentist bandits Frequentist algorithms and regret Frequentist algorithms minimizing regret Objective Frequentist Bayesian algorithms algorithms Regret?? Bayes risk? Finite-Horizon Gittins algorithm Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 13 / 48

15 Bayesian bandits, frequentist bandits Frequentist optimality Asymptotically optimal algorithms in the frequentist setting N a (t) the number of draws of arm a up to time t K R n (θ) = (µ µ a )E θ [N a (n)] a=1 [Lai and Robbins,1985]: every consistent policy satisfies µ a < µ lim inf n E θ [N a (n)] ln n 1 KL(ν θa, ν θ ) A bandit algorithm is asymptotically optimal if µ a < µ lim sup n E θ [N a (n)] ln n 1 KL(ν θa, ν θ ) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 14 / 48

16 Bayesian bandits, frequentist bandits The optimism principle A family of frequentist algorithms The following heuristic defines a family of optimistic index policies: For each arm a, compute a confidence interval on the unknown parameter µ a : µ a UCB a (t) w.h.p Use the optimism-in-face-of-uncertainty principle: act as if the best possible model was the true model The algorithm chooses at time t A t = arg max a UCB a (t) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 15 / 48

17 Bayesian bandits, frequentist bandits UCB Towards optimal algorithms Example for Bernoulli rewards: UCB [Auer et al. 02] uses Hoeffding bounds: UCB a (t) = S a(t) N a (t) + α log(t) 2N a (t) and one has: E[N a (n)] K 1 2(µ a µ ) 2 ln n + K 2, with K 1 > 1. Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 16 / 48

18 Bayesian bandits, frequentist bandits KL-UCB KL-UCB: and asymptotically optimal frequentist algorithm Example for Bernoulli rewards: KL-UCB [Cappé et al. 2013] uses the index: { ( ) } Sa (t) u a (t) = argmax K x> Sa(t) N a (t), x ln(t) + c ln ln(t) N a (t) Na(t) β(t,δ)/n a (t) with and one has S a (t)/n a (t) K(p, q) = KL (B(p), B(q)) = p log E[N a (n)] ( p q 1 K(µ a, µ ln n + C. ) ) + (1 p) log ( ) 1 p 1 q Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 17 / 48

19 Bayesian bandits, frequentist bandits Frequentist algorithms and Bayes risk Frequentist algorithms optimal minimizing Bayes risk Objective Frequentist Bayesian algorithms algorithms Regret KL-UCB? Bayes risk KL-UCB Finite-Horizon Gittins algorithm (at least in an asymptotic sense, see [Lai 1987]) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 18 / 48

20 Bayesian bandits, frequentist bandits Bayesian algorithms and regret Bayesian algorithms minimizing regret Objective Frequentist Bayesian algorithms algorithms Regret KL-UCB? Bayes risk KL-UCB Finite-Horizon Gittins algorithm We want to design Bayesian algorithm that are optimal with respect to the frequentist regret Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 19 / 48

21 Two Bayesian bandit algorithms 1 The multiarmed bandit problem 2 Bayesian bandits, frequentist bandits 3 Two Bayesian bandit algorithms Bayes-UCB Thompson Sampling 4 Bayesian algorithms for pure exploration? 5 Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 20 / 48

22 Two Bayesian bandit algorithms UCBs versus Bayesian algorithms Bayesian algorithms versus UCBs Figure: Confidence intervals on the arms means after t rounds of a bandit game Figure: Posterior over the means of the arms after t rounds of a bandit game Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 21 / 48

23 Two Bayesian bandit algorithms UCBs versus Bayesian algorithms Bayesian algorithms versus UCBs Figure: Confidence intervals on the arms means after t rounds of a bandit game Figure: Posterior over the means of the arms after t rounds of a bandit game How do we exploit the posterior in a Bayesian bandit algorithm? Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 21 / 48

24 Two Bayesian bandit algorithms Bayes-UCB 1 The multiarmed bandit problem 2 Bayesian bandits, frequentist bandits 3 Two Bayesian bandit algorithms Bayes-UCB Thompson Sampling 4 Bayesian algorithms for pure exploration? 5 Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 22 / 48

25 Two Bayesian bandit algorithms The Bayes-UCB algorithm Bayes-UCB Let : Π 0 = (π 0 1,..., π0 K ) be a prior distribution over (θ 1,..., θ K ) Λ t = (λ t 1,..., λt K ) be the posterior over the means (µ 1,..., µ K ) a the end of round t The Bayes-UCB algorithm chooses at time t ( Q 1 A t = argmax a 1 t(log t) c, λt 1 a where Q(α, π) is the quantile of order α of the distribution π. ) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 23 / 48

26 Two Bayesian bandit algorithms The Bayes-UCB algorithm Bayes-UCB Let : Π 0 = (π 0 1,..., π0 K ) be a prior distribution over (θ 1,..., θ K ) Λ t = (λ t 1,..., λt K ) be the posterior over the means (µ 1,..., µ K ) a the end of round t The Bayes-UCB algorithm chooses at time t ( Q 1 A t = argmax a 1 t(log t) c, λt 1 a where Q(α, π) is the quantile of order α of the distribution π. Bernoulli reward with uniform prior: θ = µ and Π t = Λ t ( Q 1 A t = argmax a 1 t(log t) c, Beta(S a(t) + 1, N a (t) S a (t) + 1) ) ) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 23 / 48

27 Two Bayesian bandit algorithms Bayes-UCB Theoretical results for the Bernoulli case Bayes-UCB is asymptotically optimal in this case Theorem [K.,Cappé,Garivier 2012] Let ɛ > 0. The Bayes-UCB algorithm using a uniform prior over the arms and with parameter c 5 satisfies 1 + ɛ E θ [N a (n)] KL(B(µ a ), B(µ )) log(n) + o ɛ,c (log(n)). Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 24 / 48

28 Two Bayesian bandit algorithms Link to a frequentist algorithm Bayes-UCB Bayes-UCB index is close to KL-UCB index: ũ a (t) q a (t) u a (t) with: { ( ) } Sa (t) u a (t) = argmax K x> Sa(t) N a (t), x log(t) + c log(log(t)) N a (t) Na(t) ( ) ( ) t ũ a (t) = argmax K Sa (t) log x> Sa(t) N a (t) + 1, x N a(t)+2 + c log(log(t)) (N a (t) + 1) Na(t)+1 Bayes-UCB appears to build automatically confidence intervals based on Kullback-Leibler divergence, that are adapted to the geometry of the problem in this specific case. Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 25 / 48

29 Two Bayesian bandit algorithms Bayes-UCB Where does it come from? We have a tight bound on the tail of posterior distributions (Beta distributions) First element: link between Beta and Binomial distribution: P(X a,b x) = P(S a+b 1,1 x b) Second element: Sanov inequality: for k > nx, e nd( k n,x) n + 1 P(S n,x k) e nd( k n,x) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 26 / 48

30 Two Bayesian bandit algorithms Thompson Sampling 1 The multiarmed bandit problem 2 Bayesian bandits, frequentist bandits 3 Two Bayesian bandit algorithms Bayes-UCB Thompson Sampling 4 Bayesian algorithms for pure exploration? 5 Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 27 / 48

31 Thompson Sampling Two Bayesian bandit algorithms Thompson Sampling A randomized Bayesian algorithm: a {1..K}, θ a (t) λ t a A t = argmax a µ(θ a (t)) (Recent) interest for this algorithm: a very old algorithm [Thompson 1933] partial analysis proposed [Granmo 2010][May, Korda, Lee, Leslie 2012] extensive numerical study beyond the Bernoulli case [Chapelle, Li 2011] first logarithmic upper bound on the regret [Agrawal,Goyal 2012] Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 28 / 48

32 Two Bayesian bandit algorithms Thompson Sampling An optimal regret bound for Bernoulli bandits Assume the first arm is the unique optimal and a = µ 1 µ a. Known result : [Agrawal,Goyal 2012] ( K ) 1 E[R n ] C ln(n) + o µ (ln(n)) a=2 a Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 29 / 48

33 Two Bayesian bandit algorithms Thompson Sampling An optimal regret bound for Bernoulli bandits Assume the first arm is the unique optimal and a = µ 1 µ a. Known result : [Agrawal,Goyal 2012] ( K ) 1 E[R n ] C ln(n) + o µ (ln(n)) a=2 a Our improvement : [K.,Korda,Munos 2012] Theorem ɛ > 0, E[R n ] (1 + ɛ) ( K a=2 a KL(B(µ a ), B(µ )) ) ln(n) + o µ,ɛ (ln(n)) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 29 / 48

34 Two Bayesian bandit algorithms Thompson Sampling Two key elements in the proof Introduce a quantile to replace the sample: ( q a (t) := Q 1 1 ) t ln(n), πt a such that n P (θ a (t) > q a (t)) 2 t=1 and use what we know about quantiles (cf. Bayes-UCB) Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 30 / 48

35 Two Bayesian bandit algorithms Two key elements in the proof Thompson Sampling Introduce a quantile to replace the sample: ( q a (t) := Q 1 1 ) t ln(n), πt a such that n P (θ a (t) > q a (t)) 2 t=1 and use what we know about quantiles (cf. Bayes-UCB) Proove separately that the optimal arm has to be drawn a lot Proposition There exists constants b = b(µ) (0, 1) and C b < such that ( P N 1 (t) t b) C b. t=1 Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 30 / 48

36 Two Bayesian bandit algorithms Thompson Sampling Thompson Sampling in Exponential Family bandits Arm a has distribution ν θa with density f(x θ a ) = A(x) exp(t (x)θ a F (θ a )). The Jeffreys prior on arm a is π J (θ) F (θ). Practical implementation of TS with Jeffreys prior Name Distribution Prior on λ Posterior on λ B(λ) λ x (1 λ) 1 x δ 0,1 B ( 1 2, 1 ) 2 B ( s, n s) N (λ, σ 2 (x λ)2 ( ) 1 ) e 2σ 2 1 N s 2πσ 2 n, σ2 n Γ(k, λ) λ k Γ(k) xk 1 e λx 1 [0,+ [ 1 λ Γ(kn, s) Pareto(x m, λ) λx λ m 1 x λ+1 [xm,+ [ 1 λ Γ (n + 1, s n log x m ) Posterior after n observations y 1,..., y n, with s = n s=1 T (y s). Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 31 / 48

37 Two Bayesian bandit algorithms Thompson Sampling Thompson Sampling in Exponential Family bandits Theorem [Korda,K.,Munos 13] If the rewards distributions belong to a 1-dimensional canonical exponential family, Thompson sampling with Jeffreys prior π J satisfies E[N a (n)] lim = n ln n 1 KL(ν θa, ν θa ). Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 32 / 48

38 Two Bayesian bandit algorithms Thompson Sampling Thompson Sampling in Exponential Family bandits An idea of the proof θ a (t) be a sample of the posterior π a (t) on θ a. TS samples at round t A t = arg max a θ a (t). E a,t = 1 N a (t) N a(t) s=1 T (Y a,s ) F (θ a ) δ a, Ea,t θ = (µ(θ a (t)) µ a + a ) E[N a (n)] = n n P(A t = a, E a,t, Ea,t) θ + P(A t = a, E a,t, (Ea,t) θ c ) t=1 + n P(A t = a, Ea,t) c t=1 t=1 Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 33 / 48

39 Two Bayesian bandit algorithms Thompson Sampling Thompson Sampling in Exponential Family bandits Two key ingredients Theorem (posterior concentration) There exists two constants C 1,a, C 2,a such that P((E θ a,t) c F t )1 Ea,t C 1,a N a,t e Na,t(1 δac 2,a)KL(ν θa,ν µ 1 (µa+ a) ) Proposition (number of draws of the optimal arm) For every b ]0, 1[, there exists C b < such that ( P N 1 (t) t b) C b. t=1 Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 34 / 48

40 Two Bayesian bandit algorithms Thompson Sampling Understanding the deviation result Recall the result For every b ]0, 1[ there exists a constant C b < such that t=1 ( P N 1 (t) t b) C b. Where does it come from? { N 1 (t) t b} = {there exists a time range of length at least t 1 b 1 with no draw of arm 1} Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 35 / 48

41 Two Bayesian bandit algorithms Thompson Sampling µ 2 µ 2 + δ µ Assume that : on I j = [τ j, τ j + t 1 b 1 ] there is no draw of arm 1 there exists J j I j such that s J j, a 1, µ(θ a (s)) µ 2 + δ Then : s J j, µ(θ 1 (s)) µ 2 + δ This only happens with small probability Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 36 / 48

42 Two Bayesian bandit algorithms Go Bayesian! Why using Bayesian algorithm in the frequentist setting? UCB KLUCB regret time time BayesUCB Thompson time time Regret as a function of time in a ten arms Bernoulli bandit problem with low rewards, horizon n = 20000, average over N = trials. Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 37 / 48

43 Two Bayesian bandit algorithms Go Bayesian! Why using Bayesian algorithm in the frequentist setting? In the Bernoulli case, for each arm, KL-UCB requires to solve an optimization problem: { ( ) } Sa (t) u a (t) = argmax K x> Sa(t) N a (t), x ln(t) + c ln ln(t) N a (t) Na(t) Bayes-UCB requires to compute one quantile of a Beta distribution Thompson Sampling requires to compute one sample of a Beta distribution Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 38 / 48

44 Two Bayesian bandit algorithms Go Bayesian! Why using Bayesian algorithm in the frequentist setting? In the Bernoulli case, for each arm, KL-UCB requires to solve an optimization problem: { ( ) } Sa (t) u a (t) = argmax K x> Sa(t) N a (t), x ln(t) + c ln ln(t) N a (t) Na(t) Bayes-UCB requires to compute one quantile of a Beta distribution Thompson Sampling requires to compute one sample of a Beta distribution Other advantages of Bayesian algorithms: they easily generalize to more complex models......even when the posterior is not directly computable (using MCMC) the prior can incorporate correlation between arms Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 38 / 48

45 Bayesian algorithms for pure exploration? 1 The multiarmed bandit problem 2 Bayesian bandits, frequentist bandits 3 Two Bayesian bandit algorithms Bayes-UCB Thompson Sampling 4 Bayesian algorithms for pure exploration? 5 Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 39 / 48

46 Bayesian algorithms for pure exploration? Setup and notations One bandit model, two bandit problems Recall a bandit model is simply a set of K unknown distributions. Assume µ 1 µ }{{ m > µ } m+1 µ K. Sm We have seen sofar one bandit problem: regret minimization We introduce here another bandit problem: pure-exploration: The forecaster has to find the set of m best arms, using as few observations of the arms as possible, but without suffering a loss when drawing a bad arm. The forecaster: draws the arms according to an exploration strategy stops at time τ (stopping strategy) and recommends a set S of m arms His goal: P(S = S m) 1 δ and E[τ] as small as possible Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 40 / 48

47 Bayesian algorithms for pure exploration? Algorithms for finding the m best KL-UCB: an algorithm for finding the m best arms At round t, the KL-LUCB algorithm ([K., Kalyanakrishnan, 13]) draws two well-chosen arms: u t and l t stops when CI for arms in Ŝm(t) and (Ŝm) c (t) are separated recommends the set Ŝm(τ) of m empirical best arms K=6,m=3. Set Ŝm(t), arm l t in bold Set (Ŝm(t)) c, arm u t in bold Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 41 / 48

48 Bayesian algorithms for pure exploration? Algorithms for finding the m best Bayesian algorithms for finding the m best? KL-LUCB uses KL-confidence intervals: L a (t) = min {q ˆp a (t) : N a (t)k(ˆp a (t), q) β(t, δ)}, U a (t) = max {q ˆp a (t) : N a (t)k(ˆp a (t), q) β(t, δ)}. ( ) We use β(t, δ) = log k1 Kt α to make sure P(S = Sm) 1 δ. δ Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 42 / 48

49 Bayesian algorithms for pure exploration? Algorithms for finding the m best Bayesian algorithms for finding the m best? KL-LUCB uses KL-confidence intervals: L a (t) = min {q ˆp a (t) : N a (t)k(ˆp a (t), q) β(t, δ)}, U a (t) = max {q ˆp a (t) : N a (t)k(ˆp a (t), q) β(t, δ)}. ( ) We use β(t, δ) = log k1 Kt α to make sure P(S = Sm) 1 δ. δ How to propose a Bayesian algorithm that adapts to δ? Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 42 / 48

50 Conclusion 1 The multiarmed bandit problem 2 Bayesian bandits, frequentist bandits 3 Two Bayesian bandit algorithms Bayes-UCB Thompson Sampling 4 Bayesian algorithms for pure exploration? 5 Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 43 / 48

51 Conclusion Conclusion for regret minimization Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 44 / 48

52 Work in progress Conclusion Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 45 / 48

53 Conclusion Conclusion Regret minimization: Go Bayesian! Bayes-UCB show striking similarities with KL-UCB Thompson Sampling is an easy-to-implement alternative to the optimistic approach both algorithms are asymptotically optimal towards frequentist regret (and more efficient in practise) in the Bernoulli case Thompson Sampling with Jeffreys prior is asymptotically optimal when rewards belong to a one-dimensional exponential family, which matches the guarantees of the KL-UCB algorithm Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 46 / 48

54 Conclusion Conclusion Regret minimization: Go Bayesian! Bayes-UCB show striking similarities with KL-UCB Thompson Sampling is an easy-to-implement alternative to the optimistic approach both algorithms are asymptotically optimal towards frequentist regret (and more efficient in practise) in the Bernoulli case Thompson Sampling with Jeffreys prior is asymptotically optimal when rewards belong to a one-dimensional exponential family, which matches the guarantees of the KL-UCB algorithm Natural open question: Can Bayesian tools be used to build efficient algorithms for the pure-exploration objective? Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 46 / 48

55 References of the talk References (1/2) S.Agrawal and N.Goyal, Analysis of Thompson Sampling for the multi-armed bandit problem, COLT 2012 O.Cappé, A.Garivier, O. Maillard, R. Munos, G. Stoltz. Kullback-Leibler upper confidence bound for optimal sequential allocation, Annals of Statistics, 2013 J.C. Gittins. Bandit processes and dynamic allocation indices, Journal of the Royal Statistical Society, Serie B, 1979 E. Kaufmann, O. Cappé, and A. Garivier. On Bayesian upper confidence bounds for bandit problems. AISTATS 2012 E.Kaufmann, N.Korda, and R.Munos. Thompson Sampling: an asymptotically optimal finite-time analysis. ALT 2012 Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 47 / 48

56 References of the talk References (2/2) E.Kaufmann and S.Kalyanakrishnan. Information Complexity in Bandit Subset Selection, COLT 2013 N.Korda, E.Kaufmann, R.Munos. Thompson Sampling for one-dimensional exponential families, to appear in NIPS 2013 T.Lai, Adaptive treatment allocation and the multi-armed bandit problem, Annals of Statistics, 1987 T.Lai and H.Robbins. Asymptotically efficient adaptive allocation rules, Advances in applied mathematics, 1985 W. Thompson, On the likelihood that one unknown probability exceeds another in view of the evidence of two samples, Biometrika, 1933 Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP, 24/10/13 48 / 48