Stochastic Primal-Dual Methods for Reinforcement Learning

Size: px

Start display at page:

Download "Stochastic Primal-Dual Methods for Reinforcement Learning"

Adam Barnett
5 years ago
Views:

1 Stochastic Primal-Dual Methods for Reinforcement Learning Alireza Askarian 1 Amber Srivastava 1 1 Department of Mechanical Engineering University of Illinois at Urbana Champaign Big Data Optimization, 2018 Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

2 Outline Markov Decision Process Infinite Horizon MDP Primal-Dual Formulation of Bellman Equation Saddle Point Formulation Stochastic Primal Dual Algorithm Sample Complexity for ɛ optimal solution Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

P a P is the transition matrix where P a (i, j) is the transition probability from state i to state j under the action a A.

3 Markov Decision Process - A brief overview An Infinite Horizon (Discounted) MDP is a tuple M = (S, A, P, r, γ), where S is a finite state space. A is a finite action space. γ (0, 1) is a discount factor. P a P is the transition matrix where P a (i, j) is the transition probability from state i to state j under the action a A. ˆr ija [0, σ] (with Eˆr ija = r ija ) is a random reward incurred in going from state i to j under the action a. lireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

4 Infinite Horizon Markov Decision Process Let π : S A be a policy that defines π(i) A for each i S. Denote P π P as the transition probability matrix corresponding to policy π. Denote the transitional reward vector by r π, i.e. r π (i) = j S P π(i) (i, j)r ijπ(i) Definition 1 Value vector v π R S for fixed policy π is defined as [ ] v π (i) = E γ k ˆr ik i k+1 π(i k ) i 0 = i k=0 v (i) = max v π (i), π (i) = arg max v π (i) π π Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

5 Bellman Equation and Primal-Dual Formulation for MDP Bellman Equation v (i) = max a A Equivalent Linear Program { γ P a (i, j)v (j) + P a (i, j)r ija }, i S j S j S minimize ξ T v subject to (I γp a )v r a 0, a A Dual to the above LP maximize a A λ T a r a subject to a A(I γp T a )λ a = ξ, λ a 0, a A Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

6 Bellman Equation and Primal-Dual Formulation for MDP Equivalence between optimal policy π and optimal dual variable λ. Let (v, λ ) be a unique solution to the primal-dual problem then complementarity slackness condition gives λ a,i.(v i γp a,i v r a,i ) = 0 i S, a A From the definition of value function vi γp π (i),iv r π (i),i = 0. Hence we have that Recover the optimal policy as follows λ a,i = 0 a π π (i) = a, if λ a,i > 0 Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

7 Saddle Point Formulation of Discounted MDP Equivalent Saddle Point problem for the LP program min max L(v, λ) = v R S λ 0 ξt v + λ T a ((γp a I)v + r a ) a A To develop an efficient algorithm we modify the above as where Ξ = min v V max L(v, λ) = λ Ξ ξt v + λ T a ((γp a I)v + r a ) a A V = { v v 0, v σ }, 1 γ { λ } { λ a,i ξ i, i S, = λ λ 0, λ 1 = a A ξ } 1 γ Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

8 Stochastic Primal Dual Methods for RL S, A, σ, γ are known. P, r are unknown. Sample oracle SO(i, a) = ( j, P a (i, j), ˆr ija, r ija ) Input : SO, n = S, m = A, γ (0, 1), σ (0, ) Initialize v (0), λ (0) arbitrarily. Set ξ = σ n e for k = 1, 2,..., T do Sample i S, a A. Generate j and ˆr ija from SO, v k+1 = [ V v k β ( v L(v (k), λ (k) )] ) + ɛ k λ k+1 = [ Ξ λ k + β ( λ L(v (k), λ (k) )] ) + e k end Output: ˆλ = 1 T T k=1 λ(k) and ˆπ where P( ˆ π(i) = a) = ˆλ a(i) a Aˆλa(i) Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

9 Analysis of the Algorithm Theorem ( ( )) For sample size/ iteration number Ω S 3 A 2 σ 4 ln 1 (1 γ) 4 ɛ 2 δ the duality gap is a A (ˆλ a ) T (v γp a v r a ) ɛ with probability at least 1 δ. Theorem ( ( )) For sample size/ iteration number Ω S 4 A 2 σ 2 ln 1 (1 γ) 6 ɛ 2 δ the output policy ˆπ is such that max i S v ˆπ (i) v (i) ɛ Theorem ( ( )) For sample size/ iteration number Ω S 4 A 4 σ 2 ln 1 d 2 (1 γ) 4 δ, ˆπ(i) = arg max a A ˆλa,i for all i S. Then P(ˆπ = π ) 1 δ. Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Learning Data Optimization, / 11

10 Proof Sketch Define ɛ k := v k v 2 + a A λ k a λ a 2 G k := a A(λ k a) T (v γp a v r a ) Obtain that ɛ k decreases by a factor of duality gap G k. Construct a martingale using ɛ k and Eɛ k and apply the Bernstein inequality given by ( 1 P n n i=1 ) ( X i ɛ exp nɛ2 ) 2(1 + ɛ 3 Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Data Learning Optimization, / 11

11 Thank You. Alireza Askarian, Amber Srivastava (University Stochastic of IllinoisPrimal-Dual at Urbana Champaign) Methods for Reinforcement Big Data Learning Optimization, / 11

Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015

Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)