Online Learning and Sequential Decision Making
|
|
- Scarlett Eaton
- 5 years ago
- Views:
Transcription
1 Online Learning and Sequential Decision Making Emilie Kaufmann CNRS & CRIStAL, Inria SequeL, Research School, ENS Lyon, Novembre 12-13th 2018 Emilie Kaufmann Sequential Decision Making 1 / 59
2 Learning make predictions (in this class) make decisions based on examples / history use observed data to adapt one s behavior on new data Emilie Kaufmann Sequential Decision Making 2 / 59
3 Part II: Sequential Decision Making Sequential Ressource Allocation Stochastic Multi-Armed Bandit Models Bandit Algorithms for Maximizing Rewards First Ideas UCB algorithms Bayesian Bandit Algorithms Beyond Simple Bandit Models Bandit algorithms for Optimal Exploration Emilie Kaufmann Sequential Decision Making 3 / 59
4 Part II: Sequential Decision Making Sequential Ressource Allocation Stochastic Multi-Armed Bandit Models Bandit Algorithms for Maximizing Rewards First Ideas UCB algorithms Bayesian Bandit Algorithms Beyond Simple Bandit Models Bandit algorithms for Optimal Exploration Emilie Kaufmann Sequential Decision Making 4 / 59
5 Examples Clinical trials K treatment for a given symptom (with unknown effect) What treatment should be allocated to the next patient based on responses observed on previous patients? Emilie Kaufmann Sequential Decision Making 5 / 59
6 Examples Clinical trials K treatment for a given symptom (with unknown effect) What treatment should be allocated to the next patient based on responses observed on previous patients? Online advertisement K adds that can be displayed Which add should be displayed for a user, based on the previous clicks of previous (similar) users? Emilie Kaufmann Sequential Decision Making 5 / 59
7 Dynamic allocation of computational resource Numerical experiments: where to evaluate a costly function in order to find its maximum quickly? Emilie Kaufmann Sequential Decision Making 6 / 59
8 Dynamic allocation of computational resource Numerical experiments: where to evaluate a costly function in order to find its maximum quickly? Artificial intelligence for games: where to choose the next evaluation to perform in order to find the best move to play next? Emilie Kaufmann Sequential Decision Making 6 / 59
9 Multi-Armed Bandit one agent K possibles actions, which provide rewards Goal: maximize the sum of the rewards obtained. Emilie Kaufmann Sequential Decision Making 7 / 59
10 Multi-Armed Bandit A each time step t = 1,..., T, the agent chooses an action A t {1,..., K} observes a reward R t associated to the chosen action. Sequential strategy: A t depends on A 1, R 1,..., A t 1, R t 1. Emilie Kaufmann Sequential Decision Making 8 / 59
11 How to model the rewards? As for predicting individual sequences, we can first consider arbitrary rewards. action k: (arbitrary) sequence of rewards r k,1, r k,2..., r k,t reward at time t: reward of the chosen action R t = r At,t. we don t observe r k,t for k A t! Goal: maximize the sum of the rewards minimize regret R T = T max r k,t k {1,...,K} t=1 }{{} cumulative rewards of the best action T t=1 R t }{{} cumulative rewards of our strategy Emilie Kaufmann Sequential Decision Making 9 / 59
12 EXP3 for rewards maximization Hypothesis: r k,t [0, 1] pour tout t Parameter: η > 0. Initialization: for all k {1,..., K}, w k,1 = 1 K. Pour t = 1,..., n 1. Compute the probability vector p t = (p 1,t,..., p K,t ) where p k,t = w k,t K i=1 w i,t 2. Choose an action A t p t, i.e. P (A t = k) = p k,t 3. Observe the reward R t = r At,t. 4. Compute the (re-weighted) loss of the chosen action: L t = 1 R t p At,t 5. Update the weight of the chosen action ) w At,t+1 = w At,t exp ( η L t Emilie Kaufmann Sequential Decision Making 10 / 59
13 Analysis of EXP3 Theorem [Auer et al., 02] For the choice log(k) η T = KT EXP3(η T ) satisfies E[R T ] 2 ln(k) KT Leveraging some stochastic assumption on the rewards, one can obtain better regret guarantees. Emilie Kaufmann Sequential Decision Making 11 / 59
14 Outline Sequential Ressource Allocation Stochastic Multi-Armed Bandit Models Bandit Algorithms for Maximizing Rewards First Ideas UCB algorithms Bayesian Bandit Algorithms Beyond Simple Bandit Models Bandit algorithms for Optimal Exploration Emilie Kaufmann Sequential Decision Making 12 / 59
15 Stochastic Multi-Armed Bandit model A simple stochastic model: k = 1,..., K, (X k,t ) t N is i.i.d. with distribution ν k K arms K (unknown) probability distribution ν 1 ν 2 ν 3 ν 4 ν 5 At round t, an agent: chooses an arm A t receives a reward R t = X At,t ν At The sampling strategy (or bandit algorithm) (A t ) is sequential: A t+1 = F t (A 1, R 1,..., A t, R t ). [ T ] Goal: Maximize E t=1 R t. Emilie Kaufmann Sequential Decision Making 13 / 59
16 Stochastic Multi-Armed Bandit model A simple stochastic model: k = 1,..., K, (X k,t ) t N is i.i.d. with mean µ k K arms K (unknown) probability distribution µ 1 µ 2 µ 3 µ 4 µ 5 At round t, an agent: chooses an arm A t receives a reward R t = X At,t ν At The sampling strategy (or bandit algorithm) (A t ) is sequential: A t+1 = F t (A 1, R 1,..., A t, R t ). [ T ] Goal: Maximize E t=1 R t. Emilie Kaufmann Sequential Decision Making 13 / 59
17 Why Stochastic Bandits? ν 1 ν 2 ν 3 ν 4 ν 5 Goal: maximize ones gains in a casino? (HOPELESS) Emilie Kaufmann Sequential Decision Making 14 / 59
18 Clinical trials Historical motivation [Thompson 1933] B(µ 1 ) B(µ 2 ) B(µ 3 ) B(µ 4 ) B(µ 5 ) For the t-th patient in a clinical study, chooses a treatment A t observes a response R t {0, 1} : P(R t = 1 A t = a) = µ a Goal: maximize the number of patients healed Emilie Kaufmann Sequential Decision Making 15 / 59
19 Online content optimization $$ Modern motivation (recommender systems, online advertisement, A/B Testing...) ν 1 ν 2 ν 3 ν 4 ν 5 For the t-th visitor of a website, recommend a movie A t observe a rating R t ν At (e.g. R t {1,..., 5}) Goal: maximize the sum of ratings Emilie Kaufmann Sequential Decision Making 16 / 59
20 Cognitive radios Agent: a smart radio device Arms: radio channels (frequency bands) streams indicating channel availabilities Channel 1 X 1,1 X 1,2... X 1,t... X 1,T Channel 2 X 2,1 X 2,2... X 2,t... X 2,T Channel K X K,1 X K,2... X K,t... X K,T At round t, the device: selects channel A t observes the channel availability R t = X At,t = 0 or 1 Goal: Maximize the number of successful transmissions Emilie Kaufmann Sequential Decision Making 17 / 59
21 Cognitive radios Agent: a smart radio device Arms: radio channels (frequency bands) streams indicating channel availabilities Arm 1 X 1,1 X 1,2... X 1,t... X 1,T Arm 2 X 2,1 X 2,2... X 2,t... X 2,T Arm K X K,1 X K,2... X K,t... X K,T At round t, the device: selects arm A t observes the channel availability R t = X At,t = 0 or 1 Goal: Maximize the number of successful transmissions Emilie Kaufmann Sequential Decision Making 17 / 59
22 Objective Goal: find a strategy maximizing [ T ] E R t. t=1 Oracle: always play the arm k = argmax µ k with mean µ = max µ k. k {1,...,K} k {1,...,K} Can we be almost as good as the oracle? [ T E t=1 R t ] T µ? Emilie Kaufmann Sequential Decision Making 18 / 59
23 Performance measure: the regret Maximizing rewards minimizing regret [ T ] R T := T µ E X t t=1 K = (µ µ k )E[N k (T )], k=1 N k (t): number of draws of arm k up to round t. Need for an Exploration/Exploitation tradeoff Emilie Kaufmann Sequential Decision Making 19 / 59
24 Performance measure: the regret Maximizing rewards minimizing regret [ T ] R T := T µ E R t We want the regret to grow sub-linearly: R T T t=1 0 (consistency) T what rate of regret can we expect? Emilie Kaufmann Sequential Decision Making 19 / 59
25 A lower bound on the regret Bernoulli bandit model, µ = (µ 1,..., µ K ) R T (µ) = K (µ µ k )E µ [N k (T )] k=1 When T grows, all the arms should be drawn infinitely many! [Lai & Robbins, 1985]: for any uniformly good strategy, µ k < µ lim inf T where E µ [N k (T )] log T d(p, p ) = KL(B(p), B(p )) the regret is at least logarithmic 1 d(µ k, µ ), = p log p 1 p + (1 p) log p 1 p. Emilie Kaufmann Sequential Decision Making 20 / 59
26 A lower bound on the regret Bernoulli bandit model, µ = (µ 1,..., µ K ) R T (µ) = K (µ µ k )E µ [N k (T )] k=1 When T grows, all the arms should be drawn infinitely many! [Lai & Robbins, 1985]: for any uniformly good strategy, µ k < µ lim inf T E µ [N k (T )] log T where d(p, p ) = KL(B(p), B(p )) 1 d(µ k, µ ), = p log p 1 p + (1 p) log p 1 p. can we find asymptotically optimal algorithm, i.e. algorithms matching the lower bound? Emilie Kaufmann Sequential Decision Making 20 / 59
27 Outline Sequential Ressource Allocation Stochastic Multi-Armed Bandit Models Bandit Algorithms for Maximizing Rewards First Ideas UCB algorithms Bayesian Bandit Algorithms Beyond Simple Bandit Models Bandit algorithms for Optimal Exploration Emilie Kaufmann Sequential Decision Making 21 / 59
28 Some (naive) strategies Idea 1 : Draw each arm T /K times EXPLORATION ( 1 R(T ) = K ) K (µ 1 µ k ) T k=2 Emilie Kaufmann Sequential Decision Making 22 / 59
29 Some (naive) strategies Idea 1 : Draw each arm T /K times EXPLORATION ( 1 R(T ) = K ) K (µ 1 µ k ) T k=2 Idea 2 : Always trust the empirical best arm where A t+1 = argmax ˆµ k (t) k {1,...,K} ˆµ k (t) = 1 t R s 1 N k (t) (As=k) s=1 is an estimate of the unknown mean µ k. EXPLOITATION R(T ) (1 µ 1 ) µ 2 (µ 1 µ 2 )T Emilie Kaufmann Sequential Decision Making 22 / 59
30 A better idea: Explore-Then-Exploit Given m {1,..., T /K}, draw each arm m times compute the empirical best arm ˆk = argmax k ˆµ k (Km) keep playing this arm until round T A t+1 = ˆk for t Km EXPLORATION followed by EXPLOITATION Emilie Kaufmann Sequential Decision Making 23 / 59
31 A better idea: Explore-Then-Exploit Given m {1,..., T /K}, draw each arm m times compute the empirical best arm ˆk = argmax k ˆµ k (Km) keep playing this arm until round T A t+1 = ˆk for t Km EXPLORATION followed by EXPLOITATION Analysis for two arms. µ 1 > µ 2, := µ 1 µ 2. Assumption: the rewards are bounded in [0, 1]. ) R(T ) m + T exp ( m 2 2 Proof: on the board, uses Hoeffding s inequality Emilie Kaufmann Sequential Decision Making 23 / 59
32 A better idea: Explore-Then-Exploit Given m {1,..., T /K}, draw each arm m times compute the empirical best arm ˆk = argmax k ˆµ k (Km) keep playing this arm until round T A t+1 = ˆk for t Km EXPLORATION followed by EXPLOITATION Analysis for two arms. µ 1 > µ 2, := µ 1 µ 2. Assumption: the rewards are bounded in [0, 1]. ) R(T ) m + T exp ( m 2 2 How to choose m? Emilie Kaufmann Sequential Decision Making 24 / 59
33 A better idea: Explore-Then-Exploit Given m {1,..., T /K}, draw each arm m times compute the empirical best arm ˆk = argmax k ˆµ k (Km) keep playing this arm until round T A t+1 = ˆk for t Km EXPLORATION followed by EXPLOITATION Analysis for two arms. µ 1 > µ 2, := µ 1 µ 2. Assumption: the rewards are bounded in [0, 1]. with m = 2 2 log R(T ) 2 ( T 2 2 ) [ log ( T 2 2 ) ] + 1 Emilie Kaufmann Sequential Decision Making 24 / 59
34 A better idea: Explore-Then-Exploit Given m {1,..., T /K}, draw each arm m times compute the empirical best arm ˆk = argmax k ˆµ k (Km) keep playing this arm until round T A t+1 = ˆk for t Km EXPLORATION followed by EXPLOITATION Analysis for two arms. µ 1 > µ 2, := µ 1 µ 2. Assumption: the rewards are bounded in [0, 1]. R(T ) 2 ( ) T 2 log with m = 2 2 log ( T 2 Problem: requires the knowledge of T and 2 ) Emilie Kaufmann Sequential Decision Making 25 / 59
35 Sequential Explore-Then-Exploit (2 arms) explore uniformly { until the random time } 4 log(t τ = inf t N : ˆµ 1 (t) ˆµ 2 (t) > 2 ) t ˆk = argmax k ˆµ k (τ) and (A t+1 = ˆk) for t {τ,..., T } R T 16 log (T ) + C( ). logarithmic regret without knowing Emilie Kaufmann Sequential Decision Making 26 / 59
36 Sequential Explore-Then-Exploit (2 arms) explore uniformly { until the random time } 4 log(t /t) τ = inf t N : ˆµ 1 (t) ˆµ 2 (t) > t ˆk = argmaxk ˆµ k (τ) and (A t+1 = ˆk) for t {τ,..., T } R T 2 log (T ) + C log(t ). same regret rate, without knowing [Garivier et al. On Explore-Then-Commit Strategies, 2016] Emilie Kaufmann Sequential Decision Making 27 / 59
37 Sequential Explore-Then-Exploit (2 arms) explore uniformly { until the random time } 4 log(t /t) τ = inf t N : ˆµ 1 (t) ˆµ 2 (t) > t ˆk = argmaxk ˆµ k (τ) and (A t+1 = ˆk) for t {τ,..., T } R T 2 log (T ) + C log(t ). still requires the knowledge of T... [Garivier et al. On Explore-Then-Commit Strategies, 2016] Emilie Kaufmann Sequential Decision Making 27 / 59
38 Outline Sequential Ressource Allocation Stochastic Multi-Armed Bandit Models Bandit Algorithms for Maximizing Rewards First Ideas UCB algorithms Bayesian Bandit Algorithms Beyond Simple Bandit Models Bandit algorithms for Optimal Exploration Emilie Kaufmann Sequential Decision Making 28 / 59
39 The optimism principle For each arm k, build a confidence interval on the mean µ k : I k (t) = [LCB k (t), UCB k (t)] LCB = Lower Confidence Bound UCB = Upper Confidence Bound Figure: Confidence intervals on the means after t rounds Emilie Kaufmann Sequential Decision Making 29 / 59
40 The optimism principle We apply the following principle: act as if the best possible model was the true model (optimism in face of uncertainty) Figure: Confidence intervals on the means after t rounds Thus, one selects at time t + 1 the arm A t+1 = argmax k=1,...,k [Lai and Robbins 1985] [Agrawal 1995] UCB k (t) Emilie Kaufmann Sequential Decision Making 30 / 59
41 How to build confidence intervals? We need UCB a (t) such that P (µ a UCB a (t)) 1 1 t. use concentration inequalities Assumption: the rewards are bounded in [0, 1]. Hoeffding inequality Z i i.i.d. with mean µ and Z i [0, 1]. For all s 1 ( ) Z1 + + Z s P µ + x e 2x 2 s s We can show that ( P µ a ˆµ a (t) + ) α log(t) 1 1 N a (t) t 2α 1. Emilie Kaufmann Sequential Decision Making 31 / 59
42 How to build confidence intervals? We need UCB a (t) such that P (µ a UCB a (t)) 1 1 t. use concentration inequalities Assumption: the rewards are bounded in [0, 1]. Hoeffding inequality Z i i.i.d. with mean µ and Z i [0, 1]. For all s 1 ( ) Z1 + + Z s P µ x e 2x 2 s s We can show that ( P µ a ˆµ a (t) + ) α log(t) 1 1 N a (t) t 2α 1. Emilie Kaufmann Sequential Decision Making 31 / 59
43 A UCB algorithm UCB(α) selects A t+1 = argmax a UCB a (t) where α log(t) UCB a (t) = ˆµ a (t) +. }{{} N a (t) exploitation term }{{} exploration bonus popularised by [Auer et al. 02] : UCB1, α = Emilie Kaufmann Sequential Decision Making 32 / 59
44 UCB in action Emilie Kaufmann Sequential Decision Making 33 / 59
45 Regret analysis of a UCB algorithm Theorem The UCB(α) algorithm satisfies, for α = 3/2, 6 π2 E[N a (T )] log(t ) (µ 1 µ a ) arms : R(T ) 6 log(t ) + C. Proof: on the board Emilie Kaufmann Sequential Decision Making 34 / 59
46 An improved analysis of UCB(α) Theorem [Bubeck 11],[Cappé et al. 13] For α > 1/2, the UCB(α) algorithm satisfies E[N a (T )] α (µ 1 µ a ) 2 log(t ) + O( log(t )). 2 arms : R(T ) 1 2 log(t ) + C log(t ). UCB combines exploration and exploitation and has a smaller regret than Explore-Then-Exploit approaches order-optimal for Bernoulli distributions [ Pinsker s inequality: d(µa, µ 1 ) 2(µ 1 µ a ) 2] Emilie Kaufmann Sequential Decision Making 35 / 59
47 The kl-ucb algorithm (for Bernoulli bandits, or other simple parametric families) A UCB-type algorithm: A t+1 = argmax u k (t)... associated to the right upper confidence bound: { u k (t) = max q : d (ˆµ k (t), q) log(t) }, d(x, y) = KL(B(x), B(y)). N k (t) k d(µ a (t),q) log(t)/n a (t) [Cappé et al. 13] : 0 µ a (t) [ ] u a (t) q E µ [N k (T )] 1 d(µ k, µ ) log T + O( log(t )). Emilie Kaufmann Sequential Decision Making 36 / 59
48 The kl-ucb algorithm (for Bernoulli bandits, or other simple parametric families) A UCB-type algorithm: A t+1 = argmax u k (t)... associated to the right upper confidence bound: { u k (t) = max q : d (ˆµ k (t), q) log(t) }, d(x, y) = KL(B(x), B(y)). N k (t) k d(µ a (t),q) log(t)/n a (t) µ a (t) [ ] u a (t) q kl-ucb is asymptotically optimal for Bernoulli bandits! Emilie Kaufmann Sequential Decision Making 36 / 59
49 Outline Sequential Ressource Allocation Stochastic Multi-Armed Bandit Models Bandit Algorithms for Maximizing Rewards First Ideas UCB algorithms Bayesian Bandit Algorithms Beyond Simple Bandit Models Bandit algorithms for Optimal Exploration Emilie Kaufmann Sequential Decision Making 37 / 59
50 The Bayesian choice Bernoulli bandit model µ = (µ 1,..., µ K ) frequentist view: µ 1,..., µ K are unknown parameters tools: estimators, confidence intervals Emilie Kaufmann Sequential Decision Making 38 / 59
51 The Bayesian choice Bernoulli bandit model µ = (µ 1,..., µ K ) Bayesian view: µ 1,..., µ K are random variables prior distribution : µ a U([0, 1]) tool: posterior distribution π k (t) = L (µ k R 1,..., R t ) = Beta(S k (t) + 1, N k (t) S k (t) + 1) π a (t) π a (t+1) si X t+1 = 0 π a (t) π a (t+1) si X t+1 = π S k (t) = t s=1 R s1 (As=k) sum of the rewards from arm k Emilie Kaufmann Sequential Decision Making 38 / 59
52 Bayesian algorithm A Bayesian bandit algorithm exploits the posterior distributions of the means to decide which arm to select Emilie Kaufmann Sequential Decision Making 39 / 59
53 The Bayes-UCB algorithm π k (t) the posterior distribution over µ k at the end of round t. Bayes-UCB [K., Cappé, Garivier 2012] selects A t+1 = argmax Q k {1,...,K} ( 1 1 t, π k(t) where Q(α, ν) is the quantile of order α of the distribution ν. Properties: P X ν (X Q(α, ν)) = α. easy to implement (quantiles of Beta distributions) also asymptotically optimal for Bernoulli bandits! ( q k (t) = Q 1 1 ) t, π k(t) u k (t) efficient in practice and easy to generalize ) Emilie Kaufmann Sequential Decision Making 40 / 59
54 Bayes-UCB in practice Emilie Kaufmann Sequential Decision Making 41 / 59
55 Thompson Sampling { a {1..K}, θa (t) π a (t) A t+1 = argmax θ a (t). a=1...k π 1 (t) π 2 (t) θ 0 1 (t) µ µ 2 θ 2 (t) Figure: TS selects arm 2 as θ 2 (t) θ 1 (t) the first bandit algorithm! [Thompson 1933] very efficient, beyond Bernoulli bandits matches the Lai and Robbins bound for Bernoulli bandits [K., Korda and Munos, 2012], [Agrawal and Goyal 2013] Emilie Kaufmann Sequential Decision Making 42 / 59
56 Thompson Sampling, a General Principle Draw a possible model from your posterior distribution and act optimally in this sampled model Emilie Kaufmann Sequential Decision Making 43 / 59
57 Outline Sequential Ressource Allocation Stochastic Multi-Armed Bandit Models Bandit Algorithms for Maximizing Rewards First Ideas UCB algorithms Bayesian Bandit Algorithms Beyond Simple Bandit Models Bandit algorithms for Optimal Exploration Emilie Kaufmann Sequential Decision Making 44 / 59
58 Contextual Bandits Example: movie recommendation What movie should Netflix recommend to a particular user, given the ratings provided by previous users? Ü to make good recommendation, we should take into account the characteristics of the movies / users Contextual bandit problem: at time t I a context xt is observed I an arm At is chosen I a reward Rt that depends on xt, At is received. Emilie Kaufmann Sequential Decision Making 45 / 59
59 Contextual Bandits Example: movie recommendation What movie should Netflix recommend to a particular user, given the ratings provided by previous users? Ü to make good recommendation, we should take into account the characteristics of the movies / users Example: linear model Rt = hxt,at, θi + t I xa,t Rd is a feature vector built for user t and movie a I θ Rd unknown parameter Emilie Kaufmann Sequential Decision Making 46 / 59
60 Contextual Bandits Example: movie recommendation What movie should Netflix recommend to a particular user, given the ratings provided by previous users? Ü to make good recommendation, we should take into account the characteristics of the movies / users Example: logistic model P(Rt = + At = a) = 1 T 1 + e xt,a θ I xa,t Rd is a feature vector built for user t and movie a I θ Rd unknown parameter Emilie Kaufmann Sequential Decision Making 47 / 59
61 Tools for (linear/logistic) contextual bandits In essence, the tools are the same: Explore-Then-Exploit [Rusmeviechtentong and Tsitsiklis, Linearly Parameterized Bandits, 2010] Upper Confidence Bounds: build a confidence region for the regression parameter θ R d (Lin-UCB) [Abbasi-Yadkori et al., Improved algorithms for linear bandits, 2011] [Filippi et al., Generalized Linear Bandits, 2010] Thompson Sampling: Bayesian model with a prior distribution on the regression parameter θ R d. [Agrawal and Goyal, Thompson Sampling for Contextual Bandits with linear payoffs, 2013] Emilie Kaufmann Sequential Decision Making 48 / 59
62 Example: Thompson Sampling for Linear Bandits Bayesian model: R t = x T t,a t θ + ɛ t, θ N ( 0, κ 2 I d ), ɛt N ( 0, σ 2).. Explicit posterior distribution: ) p(θ x 1,A1, R 1,..., x t,at, R t ) = N (ˆθ(t), Σt (multivariate Gaussian distribution) Thompson Sampling θ(t) N A t+1 = argmax a=1,...,k (ˆθ(t), Σ t ), x T a,t+1 θ(t). Emilie Kaufmann Sequential Decision Making 49 / 59
63 Reinforcement Learning A more general setup in which one repeatedly take decisions that generate rewards and evolution of some state. ex. management of an electric grid Formalization: an agent in a state s t chooses an action a t, receives a reward r t+1 and ends up in a new state s t+1. modelling tool: Markov Decision Process (MDP) Emilie Kaufmann Sequential Decision Making 50 / 59
64 Reinforcement Learning A more general setup in which one repeatedly take decisions that generate rewards and evolution of some state. ex. management of an electric grid Formalization: an agent in a state s t chooses an action a t, receives a reward r t+1 and ends up in a new state s t+1. Reinforcement Learning = How to solve an unknwon MDP? Emilie Kaufmann Sequential Decision Making 50 / 59
65 Bandit Tools for building AI for Games To decide the next move to play: successively chose trajectories in the game tree perform (random) evaluations of certain positions How to sequentially select trajectories in the game tree? Algorithme UCT [Kocsis & Szepesvari 06]: UCB for Trees! Emilie Kaufmann Sequential Decision Making 51 / 59
66 Outline Sequential Ressource Allocation Stochastic Multi-Armed Bandit Models Bandit Algorithms for Maximizing Rewards First Ideas UCB algorithms Bayesian Bandit Algorithms Beyond Simple Bandit Models Bandit algorithms for Optimal Exploration Emilie Kaufmann Sequential Decision Making 52 / 59
67 A motivation: A/B Testing Which version of a webpage generates more revenue? For the t-th visitor: display version A t {1, 2,..., K} observe R t = 1 if a conversion occurs this is a bandit problem, but the company does not want to alternate forever between different versions... Emilie Kaufmann Sequential Decision Making 53 / 59
68 A motivation: A/B Testing Which version of a webpage generates more revenue? For the t-th visitor: display version A t {1, 2,..., K} observe R t = 1 if a conversion occurs possibly stop the test and identify the best version Emilie Kaufmann Sequential Decision Making 54 / 59
69 A pure-exploration problem The agent has to identify the arm with highest mean a (no loss when drawing bad arms) The agent uses a sampling strategy (A t ) stops at some (random) time τ upon stopping, recommends an arm â τ Goal: given a risk parameter δ identify the best version with high-probability P (â τ = a ) 1 δ using as few samples as possible (short testing phase) [Even-Dar et al. 06] E [τ] is small. Emilie Kaufmann Sequential Decision Making 55 / 59
70 An algorithm based on confidence intervals LUCB [Kalyanakrishnan et al. 12] relies on Upper And Lower confidence bounds. For KL-LUCB: u a (t) = max {q : N a (t)d(ˆµ a (t), q) log(ct/δ)} l a (t) = min {q : N a (t)d(ˆµ a (t), q) log(ct/δ)} sampling rule: B t+1 = argmax ˆµ a (t), C t+1 = argmax a b A t+1 stopping rule: τ = inf{t N : l Bt (t) > u Ct (t)} u b (t) Emilie Kaufmann Sequential Decision Making 56 / 59
71 KL-LUCB in action Emilie Kaufmann Sequential Decision Making 57 / 59
72 µ 1 > µ 2 µ K. Theoretical garantees Theorem [Kalyanakrishan et al.] For well-chosen confidence ([ intervals, LUCB ] is (δ)-correct and 1 K ( ) ) 1 1 E [τ δ ] = O + log δ a=2 a Remark: KL-UCB versus KL-LUCB behave quite differently Emilie Kaufmann Sequential Decision Making 58 / 59
73 Theoretical garantees µ 1 > µ 2 µ K. Theorem [Kalyanakrishan et al.] For well-chosen confidence ([ intervals, LUCB ] is (δ)-correct and 1 K ( ) ) 1 1 E [τ δ ] = O + log δ a=2 a Remark: KL-UCB versus KL-LUCB behave quite differently for further comments on this and optimal best arm identification algorithms [Kaufmann and Garivier, Finding the arm with largest mean: Two bandit frameworks, 2017] Emilie Kaufmann Sequential Decision Making 58 / 59
74 Reference The Bandit Book (follow the link) Tor Lattimore and Csaba Szepesvari, to be released in 2019 Emilie Kaufmann Sequential Decision Making 59 / 59
Multi-armed bandit models: a tutorial
Multi-armed bandit models: a tutorial CERMICS seminar, March 30th, 2016 Multi-Armed Bandit model: general setting K arms: for a {1,..., K}, (X a,t ) t N is a stochastic process. (unknown distributions)
More informationStratégies bayésiennes et fréquentistes dans un modèle de bandit
Stratégies bayésiennes et fréquentistes dans un modèle de bandit thèse effectuée à Telecom ParisTech, co-dirigée par Olivier Cappé, Aurélien Garivier et Rémi Munos Journées MAS, Grenoble, 30 août 2016
More informationBandit models: a tutorial
Gdt COS, December 3rd, 2015 Multi-Armed Bandit model: general setting K arms: for a {1,..., K}, (X a,t ) t N is a stochastic process. (unknown distributions) Bandit game: a each round t, an agent chooses
More informationTwo optimization problems in a stochastic bandit model
Two optimization problems in a stochastic bandit model Emilie Kaufmann joint work with Olivier Cappé, Aurélien Garivier and Shivaram Kalyanakrishnan Journées MAS 204, Toulouse Outline From stochastic optimization
More informationThe information complexity of sequential resource allocation
The information complexity of sequential resource allocation Emilie Kaufmann, joint work with Olivier Cappé, Aurélien Garivier and Shivaram Kalyanakrishan SMILE Seminar, ENS, June 8th, 205 Sequential allocation
More informationRevisiting the Exploration-Exploitation Tradeoff in Bandit Models
Revisiting the Exploration-Exploitation Tradeoff in Bandit Models joint work with Aurélien Garivier (IMT, Toulouse) and Tor Lattimore (University of Alberta) Workshop on Optimization and Decision-Making
More informationBayesian and Frequentist Methods in Bandit Models
Bayesian and Frequentist Methods in Bandit Models Emilie Kaufmann, Telecom ParisTech Bayes In Paris, ENSAE, October 24th, 2013 Emilie Kaufmann (Telecom ParisTech) Bayesian and Frequentist Bandits BIP,
More informationOn the Complexity of Best Arm Identification in Multi-Armed Bandit Models
On the Complexity of Best Arm Identification in Multi-Armed Bandit Models Aurélien Garivier Institut de Mathématiques de Toulouse Information Theory, Learning and Big Data Simons Institute, Berkeley, March
More informationThe information complexity of best-arm identification
The information complexity of best-arm identification Emilie Kaufmann, joint work with Olivier Cappé and Aurélien Garivier MAB workshop, Lancaster, January th, 206 Context: the multi-armed bandit model
More informationOn the Complexity of Best Arm Identification with Fixed Confidence
On the Complexity of Best Arm Identification with Fixed Confidence Discrete Optimization with Noise Aurélien Garivier, Emilie Kaufmann COLT, June 23 th 2016, New York Institut de Mathématiques de Toulouse
More informationOn Bayesian bandit algorithms
On Bayesian bandit algorithms Emilie Kaufmann joint work with Olivier Cappé, Aurélien Garivier, Nathaniel Korda and Rémi Munos July 1st, 2012 Emilie Kaufmann (Telecom ParisTech) On Bayesian bandit algorithms
More informationOn the Complexity of Best Arm Identification with Fixed Confidence
On the Complexity of Best Arm Identification with Fixed Confidence Discrete Optimization with Noise Aurélien Garivier, joint work with Emilie Kaufmann CNRS, CRIStAL) to be presented at COLT 16, New York
More informationBandits : optimality in exponential families
Bandits : optimality in exponential families Odalric-Ambrym Maillard IHES, January 2016 Odalric-Ambrym Maillard Bandits 1 / 40 Introduction 1 Stochastic multi-armed bandits 2 Boundary crossing probabilities
More informationRegret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems, Part I. Sébastien Bubeck Theory Group
Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems, Part I Sébastien Bubeck Theory Group i.i.d. multi-armed bandit, Robbins [1952] i.i.d. multi-armed bandit, Robbins [1952] Known
More informationTwo generic principles in modern bandits: the optimistic principle and Thompson sampling
Two generic principles in modern bandits: the optimistic principle and Thompson sampling Rémi Munos INRIA Lille, France CSML Lunch Seminars, September 12, 2014 Outline Two principles: The optimistic principle
More informationReinforcement Learning
Reinforcement Learning Lecture 5: Bandit optimisation Alexandre Proutiere, Sadegh Talebi, Jungseul Ok KTH, The Royal Institute of Technology Objectives of this lecture Introduce bandit optimisation: the
More informationAdvanced Machine Learning
Advanced Machine Learning Bandit Problems MEHRYAR MOHRI MOHRI@ COURANT INSTITUTE & GOOGLE RESEARCH. Multi-Armed Bandit Problem Problem: which arm of a K-slot machine should a gambler pull to maximize his
More informationOnline Learning and Sequential Decision Making
Online Learning and Sequential Decision Making Emilie Kaufmann CNRS & CRIStAL, Inria SequeL, emilie.kaufmann@univ-lille.fr Research School, ENS Lyon, Novembre 12-13th 2018 Emilie Kaufmann Online Learning
More informationSparse Linear Contextual Bandits via Relevance Vector Machines
Sparse Linear Contextual Bandits via Relevance Vector Machines Davis Gilton and Rebecca Willett Electrical and Computer Engineering University of Wisconsin-Madison Madison, WI 53706 Email: gilton@wisc.edu,
More informationarxiv: v2 [stat.ml] 19 Jul 2012
Thompson Sampling: An Asymptotically Optimal Finite Time Analysis Emilie Kaufmann, Nathaniel Korda and Rémi Munos arxiv:105.417v [stat.ml] 19 Jul 01 Telecom Paristech UMR CNRS 5141 & INRIA Lille - Nord
More informationBandit Algorithms. Zhifeng Wang ... Department of Statistics Florida State University
Bandit Algorithms Zhifeng Wang Department of Statistics Florida State University Outline Multi-Armed Bandits (MAB) Exploration-First Epsilon-Greedy Softmax UCB Thompson Sampling Adversarial Bandits Exp3
More informationInformational Confidence Bounds for Self-Normalized Averages and Applications
Informational Confidence Bounds for Self-Normalized Averages and Applications Aurélien Garivier Institut de Mathématiques de Toulouse - Université Paul Sabatier Thursday, September 12th 2013 Context Tree
More informationMonte-Carlo Tree Search by. MCTS by Best Arm Identification
Monte-Carlo Tree Search by Best Arm Identification and Wouter M. Koolen Inria Lille SequeL team CWI Machine Learning Group Inria-CWI workshop Amsterdam, September 20th, 2017 Part of...... a new Associate
More informationThe Multi-Arm Bandit Framework
The Multi-Arm Bandit Framework A. LAZARIC (SequeL Team @INRIA-Lille) ENS Cachan - Master 2 MVA SequeL INRIA Lille MVA-RL Course In This Lecture A. LAZARIC Reinforcement Learning Algorithms Oct 29th, 2013-2/94
More informationBandits and Exploration: How do we (optimally) gather information? Sham M. Kakade
Bandits and Exploration: How do we (optimally) gather information? Sham M. Kakade Machine Learning for Big Data CSE547/STAT548 University of Washington S. M. Kakade (UW) Optimization for Big data 1 / 22
More informationBandit Algorithms for Pure Exploration: Best Arm Identification and Game Tree Search. Wouter M. Koolen
Bandit Algorithms for Pure Exploration: Best Arm Identification and Game Tree Search Wouter M. Koolen Machine Learning and Statistics for Structures Friday 23 rd February, 2018 Outline 1 Intro 2 Model
More informationarxiv: v2 [stat.ml] 14 Nov 2016
Journal of Machine Learning Research 6 06-4 Submitted 7/4; Revised /5; Published /6 On the Complexity of Best-Arm Identification in Multi-Armed Bandit Models arxiv:407.4443v [stat.ml] 4 Nov 06 Emilie Kaufmann
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Uncertainty & Probabilities & Bandits Daniel Hennes 16.11.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Uncertainty Probability
More informationBandit Algorithms. Tor Lattimore & Csaba Szepesvári
Bandit Algorithms Tor Lattimore & Csaba Szepesvári Bandits Time 1 2 3 4 5 6 7 8 9 10 11 12 Left arm $1 $0 $1 $1 $0 Right arm $1 $0 Five rounds to go. Which arm would you play next? Overview What are bandits,
More informationLecture 5: Regret Bounds for Thompson Sampling
CMSC 858G: Bandits, Experts and Games 09/2/6 Lecture 5: Regret Bounds for Thompson Sampling Instructor: Alex Slivkins Scribed by: Yancy Liao Regret Bounds for Thompson Sampling For each round t, we defined
More informationarxiv: v1 [cs.lg] 15 Oct 2014
THOMPSON SAMPLING WITH THE ONLINE BOOTSTRAP By Dean Eckles and Maurits Kaptein Facebook, Inc., and Radboud University, Nijmegen arxiv:141.49v1 [cs.lg] 15 Oct 214 Thompson sampling provides a solution to
More informationEvaluation of multi armed bandit algorithms and empirical algorithm
Acta Technica 62, No. 2B/2017, 639 656 c 2017 Institute of Thermomechanics CAS, v.v.i. Evaluation of multi armed bandit algorithms and empirical algorithm Zhang Hong 2,3, Cao Xiushan 1, Pu Qiumei 1,4 Abstract.
More informationAdministration. CSCI567 Machine Learning (Fall 2018) Outline. Outline. HW5 is available, due on 11/18. Practice final will also be available soon.
Administration CSCI567 Machine Learning Fall 2018 Prof. Haipeng Luo U of Southern California Nov 7, 2018 HW5 is available, due on 11/18. Practice final will also be available soon. Remaining weeks: 11/14,
More informationDynamic resource allocation: Bandit problems and extensions
Dynamic resource allocation: Bandit problems and extensions Aurélien Garivier Institut de Mathématiques de Toulouse MAD Seminar, Université Toulouse 1 October 3rd, 2014 The Bandit Model Roadmap 1 The Bandit
More informationMulti-Armed Bandits. Credit: David Silver. Google DeepMind. Presenter: Tianlu Wang
Multi-Armed Bandits Credit: David Silver Google DeepMind Presenter: Tianlu Wang Credit: David Silver (DeepMind) Multi-Armed Bandits Presenter: Tianlu Wang 1 / 27 Outline 1 Introduction Exploration vs.
More informationCsaba Szepesvári 1. University of Alberta. Machine Learning Summer School, Ile de Re, France, 2008
LEARNING THEORY OF OPTIMAL DECISION MAKING PART I: ON-LINE LEARNING IN STOCHASTIC ENVIRONMENTS Csaba Szepesvári 1 1 Department of Computing Science University of Alberta Machine Learning Summer School,
More informationIntroduction to Bandit Algorithms. Introduction to Bandit Algorithms
Stochastic K-Arm Bandit Problem Formulation Consider K arms (actions) each correspond to an unknown distribution {ν k } K k=1 with values bounded in [0, 1]. At each time t, the agent pulls an arm I t {1,...,
More informationStat 260/CS Learning in Sequential Decision Problems. Peter Bartlett
Stat 260/CS 294-102. Learning in Sequential Decision Problems. Peter Bartlett 1. Thompson sampling Bernoulli strategy Regret bounds Extensions the flexibility of Bayesian strategies 1 Bayesian bandit strategies
More informationOn the Complexity of A/B Testing
JMLR: Workshop and Conference Proceedings vol 35:1 3, 014 On the Complexity of A/B Testing Emilie Kaufmann LTCI, Télécom ParisTech & CNRS KAUFMANN@TELECOM-PARISTECH.FR Olivier Cappé CAPPE@TELECOM-PARISTECH.FR
More informationLecture 4: Lower Bounds (ending); Thompson Sampling
CMSC 858G: Bandits, Experts and Games 09/12/16 Lecture 4: Lower Bounds (ending); Thompson Sampling Instructor: Alex Slivkins Scribed by: Guowei Sun,Cheng Jie 1 Lower bounds on regret (ending) Recap from
More informationBandit View on Continuous Stochastic Optimization
Bandit View on Continuous Stochastic Optimization Sébastien Bubeck 1 joint work with Rémi Munos 1 & Gilles Stoltz 2 & Csaba Szepesvari 3 1 INRIA Lille, SequeL team 2 CNRS/ENS/HEC 3 University of Alberta
More informationMulti-Armed Bandit: Learning in Dynamic Systems with Unknown Models
c Qing Zhao, UC Davis. Talk at Xidian Univ., September, 2011. 1 Multi-Armed Bandit: Learning in Dynamic Systems with Unknown Models Qing Zhao Department of Electrical and Computer Engineering University
More informationComplexity of stochastic branch and bound methods for belief tree search in Bayesian reinforcement learning
Complexity of stochastic branch and bound methods for belief tree search in Bayesian reinforcement learning Christos Dimitrakakis Informatics Institute, University of Amsterdam, Amsterdam, The Netherlands
More informationReinforcement Learning
Reinforcement Learning Lecture 6: RL algorithms 2.0 Alexandre Proutiere, Sadegh Talebi, Jungseul Ok KTH, The Royal Institute of Technology Objectives of this lecture Present and analyse two online algorithms
More informationCOS 402 Machine Learning and Artificial Intelligence Fall Lecture 22. Exploration & Exploitation in Reinforcement Learning: MAB, UCB, Exp3
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 22 Exploration & Exploitation in Reinforcement Learning: MAB, UCB, Exp3 How to balance exploration and exploitation in reinforcement
More informationStochastic bandits: Explore-First and UCB
CSE599s, Spring 2014, Online Learning Lecture 15-2/19/2014 Stochastic bandits: Explore-First and UCB Lecturer: Brendan McMahan or Ofer Dekel Scribe: Javad Hosseini In this lecture, we like to answer this
More informationMultiple Identifications in Multi-Armed Bandits
Multiple Identifications in Multi-Armed Bandits arxiv:05.38v [cs.lg] 4 May 0 Sébastien Bubeck Department of Operations Research and Financial Engineering, Princeton University sbubeck@princeton.edu Tengyao
More informationStochastic Contextual Bandits with Known. Reward Functions
Stochastic Contextual Bandits with nown 1 Reward Functions Pranav Sakulkar and Bhaskar rishnamachari Ming Hsieh Department of Electrical Engineering Viterbi School of Engineering University of Southern
More informationStat 260/CS Learning in Sequential Decision Problems. Peter Bartlett
Stat 260/CS 294-102. Learning in Sequential Decision Problems. Peter Bartlett 1. Multi-armed bandit algorithms. Concentration inequalities. P(X ǫ) exp( ψ (ǫ))). Cumulant generating function bounds. Hoeffding
More informationKULLBACK-LEIBLER UPPER CONFIDENCE BOUNDS FOR OPTIMAL SEQUENTIAL ALLOCATION
Submitted to the Annals of Statistics arxiv: math.pr/0000000 KULLBACK-LEIBLER UPPER CONFIDENCE BOUNDS FOR OPTIMAL SEQUENTIAL ALLOCATION By Olivier Cappé 1, Aurélien Garivier 2, Odalric-Ambrym Maillard
More informationProfile-Based Bandit with Unknown Profiles
Journal of Machine Learning Research 9 (208) -40 Submitted /7; Revised 6/8; Published 9/8 Profile-Based Bandit with Unknown Profiles Sylvain Lamprier sylvain.lamprier@lip6.fr Sorbonne Universités, UPMC
More informationarxiv: v3 [cs.lg] 7 Nov 2017
ESAIM: PROCEEDINGS AND SURVEYS, Vol.?, 2017, 1-10 Editors: Will be set by the publisher arxiv:1702.00001v3 [cs.lg] 7 Nov 2017 LEARNING THE DISTRIBUTION WITH LARGEST MEAN: TWO BANDIT FRAMEWORKS Emilie Kaufmann
More informationRegret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems. Sébastien Bubeck Theory Group
Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems Sébastien Bubeck Theory Group Part 1: i.i.d., adversarial, and Bayesian bandit models i.i.d. multi-armed bandit, Robbins [1952]
More informationThompson Sampling for the non-stationary Corrupt Multi-Armed Bandit
European Worshop on Reinforcement Learning 14 (2018 October 2018, Lille, France. Thompson Sampling for the non-stationary Corrupt Multi-Armed Bandit Réda Alami Orange Labs 2 Avenue Pierre Marzin 22300,
More informationLearning to play K-armed bandit problems
Learning to play K-armed bandit problems Francis Maes 1, Louis Wehenkel 1 and Damien Ernst 1 1 University of Liège Dept. of Electrical Engineering and Computer Science Institut Montefiore, B28, B-4000,
More informationOnline Learning with Gaussian Payoffs and Side Observations
Online Learning with Gaussian Payoffs and Side Observations Yifan Wu 1 András György 2 Csaba Szepesvári 1 1 Department of Computing Science University of Alberta 2 Department of Electrical and Electronic
More informationAnalysis of Thompson Sampling for the multi-armed bandit problem
Analysis of Thompson Sampling for the multi-armed bandit problem Shipra Agrawal Microsoft Research India shipra@microsoft.com avin Goyal Microsoft Research India navingo@microsoft.com Abstract We show
More informationComplex Bandit Problems and Thompson Sampling
Complex Bandit Problems and Aditya Gopalan Department of Electrical Engineering Technion, Israel aditya@ee.technion.ac.il Shie Mannor Department of Electrical Engineering Technion, Israel shie@ee.technion.ac.il
More informationAn Estimation Based Allocation Rule with Super-linear Regret and Finite Lock-on Time for Time-dependent Multi-armed Bandit Processes
An Estimation Based Allocation Rule with Super-linear Regret and Finite Lock-on Time for Time-dependent Multi-armed Bandit Processes Prokopis C. Prokopiou, Peter E. Caines, and Aditya Mahajan McGill University
More informationMulti-Armed Bandit Formulations for Identification and Control
Multi-Armed Bandit Formulations for Identification and Control Cristian R. Rojas Joint work with Matías I. Müller and Alexandre Proutiere KTH Royal Institute of Technology, Sweden ERNSI, September 24-27,
More informationRacing Thompson: an Efficient Algorithm for Thompson Sampling with Non-conjugate Priors
Racing Thompson: an Efficient Algorithm for Thompson Sampling with Non-conugate Priors Yichi Zhou 1 Jun Zhu 1 Jingwe Zhuo 1 Abstract Thompson sampling has impressive empirical performance for many multi-armed
More informationLecture 4 January 23
STAT 263/363: Experimental Design Winter 2016/17 Lecture 4 January 23 Lecturer: Art B. Owen Scribe: Zachary del Rosario 4.1 Bandits Bandits are a form of online (adaptive) experiments; i.e. samples are
More informationAnalysis of Thompson Sampling for the multi-armed bandit problem
Analysis of Thompson Sampling for the multi-armed bandit problem Shipra Agrawal Microsoft Research India shipra@microsoft.com Navin Goyal Microsoft Research India navingo@microsoft.com Abstract The multi-armed
More informationValue Directed Exploration in Multi-Armed Bandits with Structured Priors
Value Directed Exploration in Multi-Armed Bandits with Structured Priors Bence Cserna Marek Petrik Reazul Hasan Russel Wheeler Ruml University of New Hampshire Durham, NH 03824 USA bence, mpetrik, rrussel,
More informationTHE first formalization of the multi-armed bandit problem
EDIC RESEARCH PROPOSAL 1 Multi-armed Bandits in a Network Farnood Salehi I&C, EPFL Abstract The multi-armed bandit problem is a sequential decision problem in which we have several options (arms). We can
More informationYevgeny Seldin. University of Copenhagen
Yevgeny Seldin University of Copenhagen Classical (Batch) Machine Learning Collect Data Data Assumption The samples are independent identically distributed (i.i.d.) Machine Learning Prediction rule New
More information1 MDP Value Iteration Algorithm
CS 0. - Active Learning Problem Set Handed out: 4 Jan 009 Due: 9 Jan 009 MDP Value Iteration Algorithm. Implement the value iteration algorithm given in the lecture. That is, solve Bellman s equation using
More informationAn Information-Theoretic Analysis of Thompson Sampling
Journal of Machine Learning Research (2015) Submitted ; Published An Information-Theoretic Analysis of Thompson Sampling Daniel Russo Department of Management Science and Engineering Stanford University
More informationThe multi armed-bandit problem
The multi armed-bandit problem (with covariates if we have time) Vianney Perchet & Philippe Rigollet LPMA Université Paris Diderot ORFE Princeton University Algorithms and Dynamics for Games and Optimization
More informationThe Multi-Armed Bandit Problem
The Multi-Armed Bandit Problem Electrical and Computer Engineering December 7, 2013 Outline 1 2 Mathematical 3 Algorithm Upper Confidence Bound Algorithm A/B Testing Exploration vs. Exploitation Scientist
More informationImproved Algorithms for Linear Stochastic Bandits
Improved Algorithms for Linear Stochastic Bandits Yasin Abbasi-Yadkori abbasiya@ualberta.ca Dept. of Computing Science University of Alberta Dávid Pál dpal@google.com Dept. of Computing Science University
More informationCorrupt Bandits. Abstract
Corrupt Bandits Pratik Gajane Orange labs/inria SequeL Tanguy Urvoy Orange labs Emilie Kaufmann INRIA SequeL pratik.gajane@inria.fr tanguy.urvoy@orange.com emilie.kaufmann@inria.fr Editor: Abstract We
More informationThe geometry of Gaussian processes and Bayesian optimization. Contal CMLA, ENS Cachan
The geometry of Gaussian processes and Bayesian optimization. Contal CMLA, ENS Cachan Background: Global Optimization and Gaussian Processes The Geometry of Gaussian Processes and the Chaining Trick Algorithm
More informationarxiv: v1 [cs.lg] 12 Sep 2017
Adaptive Exploration-Exploitation Tradeoff for Opportunistic Bandits Huasen Wu, Xueying Guo,, Xin Liu University of California, Davis, CA, USA huasenwu@gmail.com guoxueying@outlook.com xinliu@ucdavis.edu
More informationBasics of reinforcement learning
Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system
More informationOptimality of Thompson Sampling for Gaussian Bandits Depends on Priors
Optimality of Thompson Sampling for Gaussian Bandits Depends on Priors Junya Honda Akimichi Takemura The University of Tokyo {honda, takemura}@stat.t.u-tokyo.ac.jp Abstract In stochastic bandit problems,
More informationMulti-armed Bandits in the Presence of Side Observations in Social Networks
52nd IEEE Conference on Decision and Control December 0-3, 203. Florence, Italy Multi-armed Bandits in the Presence of Side Observations in Social Networks Swapna Buccapatnam, Atilla Eryilmaz, and Ness
More informationarxiv: v4 [cs.lg] 22 Jul 2014
Learning to Optimize Via Information-Directed Sampling Daniel Russo and Benjamin Van Roy July 23, 2014 arxiv:1403.5556v4 cs.lg] 22 Jul 2014 Abstract We propose information-directed sampling a new algorithm
More informationThompson Sampling for Contextual Bandits with Linear Payoffs
Thompson Sampling for Contextual Bandits with Linear Payoffs Shipra Agrawal Microsoft Research Navin Goyal Microsoft Research Abstract Thompson Sampling is one of the oldest heuristics for multi-armed
More informationExploration and exploitation of scratch games
Mach Learn (2013) 92:377 401 DOI 10.1007/s10994-013-5359-2 Exploration and exploitation of scratch games Raphaël Féraud Tanguy Urvoy Received: 10 January 2013 / Accepted: 12 April 2013 / Published online:
More informationThe Multi-Armed Bandit Problem
Università degli Studi di Milano The bandit problem [Robbins, 1952]... K slot machines Rewards X i,1, X i,2,... of machine i are i.i.d. [0, 1]-valued random variables An allocation policy prescribes which
More informationPure Exploration in Finitely Armed and Continuous Armed Bandits
Pure Exploration in Finitely Armed and Continuous Armed Bandits Sébastien Bubeck INRIA Lille Nord Europe, SequeL project, 40 avenue Halley, 59650 Villeneuve d Ascq, France Rémi Munos INRIA Lille Nord Europe,
More informationOptimistic Bayesian Sampling in Contextual-Bandit Problems
Journal of Machine Learning Research volume (2012) 2069-2106 Submitted 7/11; Revised 5/12; Published 6/12 Optimistic Bayesian Sampling in Contextual-Bandit Problems Benedict C. May School of Mathematics
More informationReinforcement Learning
Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value
More informationFrom Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning
From Bandits to Monte-Carlo Tree Search: The Optimistic Principle Applied to Optimization and Planning Rémi Munos To cite this version: Rémi Munos. From Bandits to Monte-Carlo Tree Search: The Optimistic
More informationReinforcement Learning
Reinforcement Learning Lecture 3: RL problems, sample complexity and regret Alexandre Proutiere, Sadegh Talebi, Jungseul Ok KTH, The Royal Institute of Technology Objectives of this lecture Introduce the
More informationSubsampling, Concentration and Multi-armed bandits
Subsampling, Concentration and Multi-armed bandits Odalric-Ambrym Maillard, R. Bardenet, S. Mannor, A. Baransi, N. Galichet, J. Pineau, A. Durand Toulouse, November 09, 2015 O-A. Maillard Subsampling and
More informationarxiv: v1 [cs.ds] 4 Mar 2016
Sequential ranking under random semi-bandit feedback arxiv:1603.01450v1 [cs.ds] 4 Mar 2016 Hossein Vahabi, Paul Lagrée, Claire Vernade, Olivier Cappé March 7, 2016 Abstract In many web applications, a
More informationarxiv: v3 [stat.ml] 6 Nov 2017
On Bayesian index policies for sequential resource allocation Emilie Kaufmann arxiv:60.090v3 [stat.ml] 6 Nov 07 CNRS & Univ. Lille, UMR 989 CRIStAL, Inria SequeL emilie.kaufmann@univ-lille.fr Abstract
More informationMulti-armed bandit based policies for cognitive radio s decision making issues
Multi-armed bandit based policies for cognitive radio s decision making issues Wassim Jouini SUPELEC/IETR wassim.jouini@supelec.fr Damien Ernst University of Liège dernst@ulg.ac.be Christophe Moy SUPELEC/IETR
More informationarxiv: v1 [cs.lg] 7 Sep 2018
Analysis of Thompson Sampling for Combinatorial Multi-armed Bandit with Probabilistically Triggered Arms Alihan Hüyük Bilkent University Cem Tekin Bilkent University arxiv:809.02707v [cs.lg] 7 Sep 208
More informationExploiting Correlation in Finite-Armed Structured Bandits
Exploiting Correlation in Finite-Armed Structured Bandits Samarth Gupta Carnegie Mellon University Pittsburgh, PA 1513 Gauri Joshi Carnegie Mellon University Pittsburgh, PA 1513 Osman Yağan Carnegie Mellon
More informationAdaptive Learning with Unknown Information Flows
Adaptive Learning with Unknown Information Flows Yonatan Gur Stanford University Ahmadreza Momeni Stanford University June 8, 018 Abstract An agent facing sequential decisions that are characterized by
More informationMechanisms with Learning for Stochastic Multi-armed Bandit Problems
Mechanisms with Learning for Stochastic Multi-armed Bandit Problems Shweta Jain 1, Satyanath Bhat 1, Ganesh Ghalme 1, Divya Padmanabhan 1, and Y. Narahari 1 Department of Computer Science and Automation,
More informationONLINE ADVERTISEMENTS AND MULTI-ARMED BANDITS CHONG JIANG DISSERTATION
c 2015 Chong Jiang ONLINE ADVERTISEMENTS AND MULTI-ARMED BANDITS BY CHONG JIANG DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical and
More informationReward Maximization Under Uncertainty: Leveraging Side-Observations on Networks
Reward Maximization Under Uncertainty: Leveraging Side-Observations Reward Maximization Under Uncertainty: Leveraging Side-Observations on Networks Swapna Buccapatnam AT&T Labs Research, Middletown, NJ
More informationIntroduction to Multi-Armed Bandits
Introduction to Multi-Armed Bandits (preliminary and incomplete draft) Aleksandrs Slivkins Microsoft Research NYC https://www.microsoft.com/en-us/research/people/slivkins/ First draft: January 2017 This
More informationCOMP3702/7702 Artificial Intelligence Lecture 11: Introduction to Machine Learning and Reinforcement Learning. Hanna Kurniawati
COMP3702/7702 Artificial Intelligence Lecture 11: Introduction to Machine Learning and Reinforcement Learning Hanna Kurniawati Today } What is machine learning? } Where is it used? } Types of machine learning
More informationAnnealing-Pareto Multi-Objective Multi-Armed Bandit Algorithm
Annealing-Pareto Multi-Objective Multi-Armed Bandit Algorithm Saba Q. Yahyaa, Madalina M. Drugan and Bernard Manderick Vrije Universiteit Brussel, Department of Computer Science, Pleinlaan 2, 1050 Brussels,
More informationAnytime optimal algorithms in stochastic multi-armed bandits
Rémy Degenne LPMA, Université Paris Diderot Vianney Perchet CREST, ENSAE REMYDEGENNE@MATHUNIV-PARIS-DIDEROTFR VIANNEYPERCHET@NORMALESUPORG Abstract We introduce an anytime algorithm for stochastic multi-armed
More information