Finite-Sample Analysis in Reinforcement Learning

Transcription

1 Finite-Sample Analysis in Reinforcement Learning Mohammad Ghavamzadeh INRIA Lille Nord Europe, Team SequeL

2 Outline 1 Introduction to RL and DP 2 Approximate Dynamic Programming (AVI & API) 3 How does Statistical Learning Theory come to the picture? 4 Error Propagation (AVI & API Error Propagation) 5 An AVI Algorithm (Fitted Q-Iteration) FQI: error at each iteration Final performance bound of FQI 6 An API Algorithm (Least-Squares Policy Iteration) 7 Discussion Error at each iteration (LSTD error) Final performance bound of LSPI

3 Sequential Decision-Making under Uncertainty? How Can I...? Move around in the physical world (e.g. driving, navigation) Play and win a game Retrieve information over the web Medical diagnosis and treatment Maximize the throughput of a factory Optimize the performance of a rescue team

4 Reinforcement Learning (RL) RL: A class of learning problems in which an agent interacts with a dynamic, stochastic, and incompletely known environment Goal: Learn an action-selection strategy, or policy, to optimize some measure of its long-term performance Interaction: Modeled as a MDP or a POMDP

5 Markov Decision Process MDP An MDP M is a tuple X,A, r, p,γ. The state space X is a bounded closed subset of R d. The set of actions A is finite ( A < ). The reward function r : X A R is bounded by R max. The transition model p( x, a) is a distribution over X. γ (0, 1) is a discount factor. Policy: a mapping from states to actions π(x) A

6 Value Function For a policy π Value function V π : X R [ V π (x) = E γ t r ( X t,π(x t ) ) ] X 0 = x t=0 Action-value function Q π : X A R [ ] Q π (x, a) = E γ t r(x t, A t ) X 0 = x, A 0 = a t=0

7 Notation Bellman Operator Bellman operator for policy π T π : B V (X; V max ) B V (X; V max ) V π is the unique fixed-point of the Bellman operator (T π V)(x) = r ( x,π(x) ) +γ p ( dy x,π(x) ) V(y) The action-value function Q π is defined as Q π (x, a) = r(x, a)+γ p(dy x, a)v π (y) X X

8 Optimal Value Function and Optimal Policy Optimal value function V (x) = sup V π (x) π x X Optimal action-value function Q (x, a) = sup Q π (x, a) π x X, a A A policy π is optimal if V π (x) = V (x) x X

9 Notation Bellman Optimality Operator Bellman optimality operator T : B V (X; V max ) B V (X; V max ) V is the unique fixed-point of the Bellman optimality operator [ ] (T V)(x) = max r(x, a)+γ p(dy x, a)v(y) a A X Optimal action-value function Q is defined as Q (x, a) = r(x, a)+γ p(dy x, a)v (y) X

10 Properties of Bellman Operators Monotonicity: if V 1 V 2 component-wise T π V 1 T π V 2 and T V 1 T V 2 Max-Norm Contraction: V 1, V 2 B V (X; V max ) T π V 1 T π V 2 γ V 1 V 2 T V 1 T V 2 γ V 1 V 2

11 Dynamic Programming Algorithms Value Iteration start with an arbitrary action-value function Q 0 at each iteration k Q k+1 = T Q k Convergence lim k V k = V. V V k+1 = T V T V k γ V V k γ k+1 V V 0 k 0

12 Dynamic Programming Algorithms Policy Iteration start with an arbitrary policy π 0 at each iteration k Convergence Policy Evaluation: Compute Q π k Policy Improvement: Compute the greedy policy w.r.t. Q π k π k+1 (x) = (Gπ k )(x) = arg max Q π k (x, a) a A PI generates a sequence of policies with increasing performance (V π k+1 V π k ) and stops after a finite number of iterations with the optimal policy π. V π k = T π k V π k T V π k = T π k+1 V π k lim n (T π k+1 ) n V π k = V π k+1

13 Approximate Dynamic Programming

14 Approximate Dynamic Programming Algorithms Value Iteration start with an arbitrary action-value function Q 0 at each iteration k Q k+1 = T Q k What if Q k+1 T Q k? Q Q k+1? γ Q Q k

15 Approximate Dynamic Programming Algorithms Policy Iteration start with an arbitrary policy π 0 at each iteration k Policy Evaluation: Compute Q π k Policy Improvement: Compute the greedy policy w.r.t. Q π k π k+1 (x) = (Gπ k )(x) = arg max Q π k (x, a) a A What if we cannot compute Q π k exactly? (Compute Qπ k Q π k instead) π k+1 (x) = arg max a A Q π k (x, a) (Gπ k )(x) V π k+1? V π k

16 Statistical Learning Theory in RL & ADP Approximate Value Iteration (AVI) Q k+1 T Q k finding a function that best approximatestq k Q = min f f TQ k µ only noisy observations of T Q k are available T Qk Target Function = T Q k Noisy Observation = T Q k we minimize the empirical error Q k+1 = Q = min f f T Q k µ with the target of minimizing the true error Q = min f f TQ k µ Objective: Q TQ k µ Q Q µ + Q TQ k µ }{{}}{{} estimation error approximation error to be small regression

17 Statistical Learning Theory in RL & ADP Approximate Value Iteration (AVI) Q k+1 T Q k finding a function that best approximatestq k Q = min f f TQ k µ only noisy observations of T Q k are available T Qk Target Function = T Q k Noisy Observation = T Q k we minimize the empirical error Q k+1 = Q = min f f T Q k µ with the target of minimizing the true error Q = min f f TQ k µ Objective: Q TQ k µ Q Q µ + Q TQ k µ }{{}}{{} estimation error approximation error to be small regression

18 Statistical Learning Theory in RL & ADP Approximate Value Iteration (AVI) Q k+1 T Q k finding a function that best approximatestq k Q = min f f TQ k µ only noisy observations of T Q k are available T Qk Target Function = T Q k Noisy Observation = T Q k we minimize the empirical error Q k+1 = Q = min f f T Q k µ with the target of minimizing the true error Q = min f f TQ k µ Objective: Q TQ k µ Q Q µ + Q TQ k µ }{{}}{{} estimation error approximation error to be small regression

19 Statistical Learning Theory in RL & ADP Approximate Value Iteration (AVI) Q k+1 T Q k finding a function that best approximatestq k Q = min f f TQ k µ only noisy observations of T Q k are available T Qk Target Function = T Q k Noisy Observation = T Q k we minimize the empirical error Q k+1 = Q = min f f T Q k µ with the target of minimizing the true error Q = min f f TQ k µ Objective: Q TQ k µ Q Q µ + Q TQ k µ }{{}}{{} estimation error approximation error to be small regression

20 Statistical Learning Theory in RL & ADP Approximate Value Iteration (AVI) Q k+1 T Q k finding a function that best approximatestq k Q = min f f TQ k µ only noisy observations of T Q k are available T Qk Target Function = T Q k Noisy Observation = T Q k we minimize the empirical error Q k+1 = Q = min f f T Q k µ with the target of minimizing the true error Q = min f f TQ k µ Objective: Q TQ k µ Q Q µ + Q TQ k µ }{{}}{{} estimation error approximation error to be small regression

21 Statistical Learning Theory in RL & ADP Approximate Value Iteration (AVI) Q k+1 T Q k finding a function that best approximatestq k Q = min f f TQ k µ only noisy observations of T Q k are available T Qk Target Function = T Q k Noisy Observation = T Q k we minimize the empirical error Q k+1 = Q = min f f T Q k µ with the target of minimizing the true error Q = min f f TQ k µ Objective: Q TQ k µ Q Q µ + Q TQ k µ }{{}}{{} estimation error approximation error to be small regression

22 Statistical Learning Theory in RL & ADP Approximate Policy Iteration (API) - policy evaluation finding a function that best approximates Q π k Q = min f f Q π k µ only noisy observations of Q π k are available Qπ k Target Function= Q π k Noisy Observation = Q π k we minimize the empirical error Q = minf f Q π k µ with the target of minimizing the true error Q = min f f Q π k µ Objective: Q Q π k µ Q Q µ + Q Q π k µ }{{}}{{} estimation error approximation error to be small regression

23 Statistical Learning Theory in RL & ADP Approximate Policy Iteration (API) π k+1 Gπ k finding a policy that best approximatesgπ k π = min f L(f,π k ;µ) we minimize the empirical error π k+1 = π = min f L(f,πk ; µ) with the target of minimizing the true error π = min f L(f,π k ;µ) Objective: L( π,π k ;µ) L( π,π;µ) + L(π,π k ;µ) }{{}}{{} estimation error approximation error to be small classification (we do not discuss it in this talk)

24 Statistical Learning Theory in RL & ADP Approximate Policy Iteration (API) - policy evaluation finding the fixed-point of T π k only noisy observations of T π k are available T π k a fixed-point problem

25 SLT in RL & ADP supervised learning methods (regression, classification) appear in the inner-loop of ADP algorithms (performance at each iteration) tools from SLT that are used to analyze supervised learning methods can be used in RL and ADP (e.g., how many samples are required to achieve a certain performance) What makes RL more challenging? the objective is not always to recover a target function from its noisy observations (fixed-point vs. regression) the target sometimes has to be approximated given sample trajectories (non i.i.d. samples) propagation of error (control problem) is there any hope?

26 Approximate Value Iteration (AVI) V k+1 = T V k +ǫ k or V k+1 T V k = ǫ k Proposition (AVI Error Propagation) We run AVI for K iterations and π K = GV K Proof V V π K 2γ (1 γ) 2 max ǫ k + 2γK+1 0 k<k 1 γ V V 0. V V k+1 T V T V k + T V k V k+1 = γ V V k +ǫ k so K 1 V V K γ K 1 k ǫ k +γ K V V γ max ǫ k +γ K V V 0 0 k<k k=0 the result follows by the fact that V V π K 2γ 1 γ V V K.

27 Approximate Policy Iteration (API) 1 V k = V π k +ǫk or V k V π k = ǫ k (Policy Evaluation Error) 2 V k = T π k Vk +ǫ k or V k T π k Vk = ǫ k (Bellman Residual) Proposition (API Asymptotic Performance) (1) lim sup V V π k k 2γ (1 γ) lim sup V 2 k V π k k }{{} ǫ k (2) lim sup V V π k k 2γ (1 γ) lim sup V 2 k T π k V k k }{{} ǫ k

28 Approximate Dynamic Programming (ADP) Proposition (AVI Asymptotic Performance) lim sup V V π k k 2γ (1 γ) lim sup V 2 k+1 T V k k }{{} ǫ k Proposition (API Asymptotic Performance) (1) lim sup V V π k k (2) lim sup V V π k k 2γ (1 γ) 2γ (1 γ) lim sup V 2 k V π k k }{{} ǫ k lim sup V 2 k T π k V k k }{{} ǫ k

29 Error Propagation

30 AVI Error Propagation Error at each iteration k: ǫ k = T V k V k+1 π K is a greedy policy w.r.t. V K 1 π K = G(V K 1 ) Proposition (AVI Pointwise Error Bound) V V π K (I γp π K ) 1{ K 1 γ K k[ (P π ) K k + P π K P π K 1...P ] π k+1 ǫ k k=0 +γ K+1[ (P π ) K+1 +(P π K P π K 1...P π 0 ) ] } V V 0

31 AVI Error Propagation Proposition (AVI L p Error Bound) ǫ k = T V k V k+1 V V π K p,ρ V V π K [ ] 2γ C 1/p (1 γ) 2 ρ,µ max ǫ k p,µ + 2γ K/p V max 0 k<k [ ] 2γ C 1/p (1 γ) 2 µ max ǫ k p,µ + 2γ K/p V max 0 k<k (A1) (A2)

32 AVI Error Propagation Proposition (AVI L p Error Bound) ǫ k = T V k V k+1 V V π K p,ρ V V π K [ ] 2γ C 1/p (1 γ) 2 ρ,µ max ǫ k p,µ + 2γ K/p V max 0 k<k [ ] 2γ C 1/p (1 γ) 2 µ max ǫ k p,µ + 2γ K/p V max 0 k<k (A1) (A2) ǫ k p,µ : error at each iteration k, note that ǫ k = T V k V k+1

33 AVI Error Propagation Proposition (AVI L p Error Bound) ǫ k = T V k V k+1 V V π K p,ρ V V π K [ ] 2γ C 1/p (1 γ) 2 ρ,µ max ǫ k p,µ + 2γ K/p V max 0 k<k [ ] 2γ C 1/p (1 γ) 2 µ max ǫ k p,µ + 2γ K/p V max 0 k<k (A1) (A2) ǫ k p,µ : error at each iteration k, note that ǫ k = T V k V k+1 2γ K/p V max : initialization error V V 0

34 AVI Error Propagation Proposition (AVI L p Error Bound) ǫ k = T V k V k+1 V V π K p,ρ V V π K [ ] 2γ C 1/p (1 γ) 2 ρ,µ max ǫ k p,µ + 2γ K/p V max 0 k<k [ ] 2γ C 1/p (1 γ) 2 µ max ǫ k p,µ + 2γ K/p V max 0 k<k (A1) (A2) ǫ k p,µ : error at each iteration k, note that ǫ k = T V k V k+1 2γ K/p V max : initialization error V V 0 C ρ,µ, C µ : final performance is evaluated w.r.t. a measure ρ µ

35 AVI Error Propagation (Concentrability Coefficients) Final performance is evaluated w.r.t. a measure ρ µ, V V π K p,ρ Assumption 1. (Uniformly Stochastic Transitions) For all x X and a A, there exists a constant C µ < such that P( x, a) C µ µ( ). Assumption 2. (Discounted-Average Concentrability of Future-State Distribution) For any sequence of policies {π m } m 1, there exists a constant c ρ,µ (m) < such that ρp π 1 P π 2...P π m c ρ,µ (m)µ. We define C ρ,µ = (1 γ) 2 m 1 mγ m 1 c ρ,µ (m) Note that C ρ,µ C µ.

36 API Error Propagation Error at each iteration k: ǫ k = V k T π k V k π K is a greedy policy w.r.t. V K 1 π K = G(V K 1 ) Proposition (API Pointwise Error Bound) V V π K K 1 γ (γp π ) K k 1 E k ǫ k +(γp π ) K V V π 0 k=0 where E k = P π k+1 (I γp π k+1 ) 1 P π (I γp π k ) 1

37 API Error Propagation Proposition (API L p Error Bound) V V π K p,ρ [ ] 2γ C 1/p (1 γ) 2 ρ,µ max ǫ k p,µ + 2γ K/p V max 0 k<k ǫ k = V k T π k V k (A1) V V π K [ ] 2γ C 1/p (1 γ) 2 µ max ǫ k p,µ + 2γ K/p V max 0 k<k (A2)

38 API Error Propagation Proposition (API L p Error Bound) V V π K p,ρ [ ] 2γ C 1/p (1 γ) 2 ρ,µ max ǫ k p,µ + 2γ K/p V max 0 k<k ǫ k = V k T π k V k (A1) V V π K [ ] 2γ C 1/p (1 γ) 2 µ max ǫ k p,µ + 2γ K/p V max 0 k<k (A2) ǫ k p,µ : error at each iteration k, note that ǫ k = V k T π k Vk 2γ K/p V max : initialization error V V π 0 C ρ,µ, C µ : final performance is evaluated w.r.t. a measure ρ µ

39 API Error Propagation (Concentrability Coefficients) Final performance is evaluated w.r.t. a measure ρ µ, V V π K p,ρ Assumption 1. (Uniformly Stochastic Transitions) For all x X and a A, there exists a constant C µ < such that P( x, a) C µ µ( ). Assumption 2. (Discounted-Average Concentrability of Future-State Distribution) For any policy π and any non-negative integers s and t, there exists a constant c ρ,µ (s, t) < such that ρ(p ) s (P π ) t c ρ,µ (s, t)µ. We define C ρ,µ = (1 γ) 2 γ s+t c ρ,µ (s, t) s=0 t=0 Note that C ρ,µ C µ.

40 Finite-Sample Performance Bound of an AVI Algorithm

41 Approximate Value Iteration (AVI) if F is a function space, then V k+1 can be defined as V k+1 = inf V F V T V k? = Π? T V k (projection of T V k into F according to the norm L? ) if V k+1 = Π T V k then AVI converges to the unique fixed-point of Π T, i.e. V F : V = Π T V (T is a contraction in L -norm and Π is non-expansive) if we consider another norm, e.g. L 2 (µ), then AVI does not necessarily converge (Π 2,µ T is not necessarily a contraction)

42 An Approximate Value Iteration Algorithm Linear function space F = { f : f( ) = d j=1 α jϕ j ( ) } {ϕ j } d j=1 B ( (X,A); L ), φ : (X,A) R d, φ( ) = ( ϕ 1 ( ),...,ϕ d ( ) ) Fitted Q-Iteration (FQI) At each iteration k: Generate N samples of the form (X i, A i, X i, R i), where (X i, A i ) µ, X i p( X i, A i ), R i r(x i, A i ) { ((Xi Build the training set D k =, A i ), T Q k (X i, A i ) )} N, where i=1 T Q k (X i, A i ) = R i +γ max a A Q k (X i, a) Q k+1 = arg min f F f T Q k 2 N = arg min f F 1 N [ N i=1 f(xi, A i ) T Q k (X i, A i ) ] 2 (regression)

43 FQI - Error at Each Iteration Theorem (FQI - Error at Iteration k) Let F be a d-dim linear space, D k = { (X i, A i, X i, R i ) } N, i=1 (X i, A i ) iid µ, p( X i, A i ), R i = r(x i, A i ), and Q be the training set and the truncated X i solution at the k th iteration of FQI. Then with probability 1 δ, we have ( ) ( ) Q T Q k µ 4 inf f T Q k µ+o α log(1/δ) d log(n/δ) k +O. f F N N Note that Q k+1 = Q.

44 FQI - Error at Each Iteration Theorem (FQI - Error at Iteration k) Let F be a d-dim linear space, D k = { (X i, A i, X i, R i ) } N, i=1 (X i, A i ) iid µ, p( X i, A i ), R i = r(x i, A i ), and Q be the training set and the truncated X i solution at the k th iteration of FQI. Then with probability 1 δ, we have ( ) ( ) Q T Q k µ 4 inf f T Q k µ+o α log(1/δ) d log(n/δ) k +O. f F N N Note that Q k+1 = Q. N = # of samples, d = dimension of the linear function space F

45 FQI - Error at Each Iteration Theorem (FQI - Error at Iteration k) Let F be a d-dim linear space, D k = { (X i, A i, X i, R i ) } N, i=1 (X i, A i ) iid µ, p( X i, A i ), R i = r(x i, A i ), and Q be the training set and the truncated X i solution at the k th iteration of FQI. Then with probability 1 δ, we have ( ) ( ) Q T Q k µ 4 inf f T Q k µ+o α log(1/δ) d log(n/δ) k +O. f F N N Note that Q k+1 = Q. N = # of samples, d = dimension of the linear function space F α k f α k = Π 2,µ T Q k : the best approximation of T Q k in F w.r.t. µ

46 FQI - Error at Each Iteration FQI - Error at Iteration k ( ) ( ) log(1/δ) d log(n/δ) Q T Q k µ 4 inf f TQ k µ + O α k + O f F N N }{{}}{{} approximation error estimation error Approximation error: it depends on how well the function space F can approximatetq k Estimation error: it depends on the number of samples N, the dim of the function space d, and α k

47 FQI Error Bound Theorem (FQI Error Bound) Let Q 1 F be an arbitrary initial value function, Q 0,..., Q K 1 be the sequence of truncated action-value functions generated by FQI after K iterations, and π K be the greedy policy w.r.t. Q K 1. Then with probability 1 δ, we have V V π K ρ { ) 2γ log(k/δ) (1 γ) 2 C ρ,µ[ d µ(t F,F)+O (Q max N ν µ + O ( } d log(nk/δ) )]+2γ K/2 Q max N

48 FQI Error Bound Theorem (FQI Error Bound) Let Q 1 F be an arbitrary initial value function, Q 0,..., Q K 1 be the sequence of truncated action-value functions generated by FQI after K iterations, and π K be the greedy policy w.r.t. Q K 1. Then with probability 1 δ, we have V V π K ρ { ) 2γ log(k/δ) (1 γ) 2 C ρ,µ[ d µ(t F,F)+O (Q max N ν µ + O ( } d log(nk/δ) )]+2γ K/2 Q max N Approximation error: d µ(t F,F) = sup f F inf g F g Tf µ

49 FQI Error Bound Theorem (FQI Error Bound) Let Q 1 F be an arbitrary initial value function, Q 0,..., Q K 1 be the sequence of truncated action-value functions generated by FQI after K iterations, and π K be the greedy policy w.r.t. Q K 1. Then with probability 1 δ, we have { ) V V π 2γ log(k/δ) K ρ (1 γ) 2 C ρ,µ[ d µ(t F,F)+O (Q max N ν µ ( } d log(nk/δ) + O )]+2γ K/2 Q max N Approximation error: d µ(t F,F) = sup f F inf g F g Tf µ Estimation error: depends on N, d,ν µ, K. ν µ = the smallest eigenvalue of the Gram matrix ( ϕ i ϕ j dµ) i,j Note that α k Qmax ν µ

50 FQI Error Bound Theorem (FQI Error Bound) Let Q 1 F be an arbitrary initial value function, Q 0,..., Q K 1 be the sequence of truncated action-value functions generated by FQI after K iterations, and π K be the greedy policy w.r.t. Q K 1. Then with probability 1 δ, we have { ) V V π 2γ log(k/δ) K ρ (1 γ) 2 C ρ,µ[ d µ(t F,F)+O (Q max N ν µ ( } d log(nk/δ) + O )]+2γ K/2 Q max N Approximation error: d µ(t F,F) = sup f F inf g F g Tf µ Estimation error: depends on N, d,ν µ, K. ν µ = the smallest eigenvalue of the Gram matrix ( ϕ i ϕ j dµ) i,j Note that α k Qmax ν µ Initialization error: error due to the choice of the initial action-value function Q Q 0

51 Finite-Sample Performance Bound of an API Algorithm

52 Least-Squares Temporal-Difference Learning (LSTD) Linear function space F = { f : f( ) = d j=1 α jϕ j ( ) } {ϕ j } d j=1 B(X; L ), φ : X R d, φ( ) = ( ϕ 1 ( ),...,ϕ d ( ) ) V π is the fixed-point of T π V π may not belong to F T π V π = V π V π / F LSTD searches for the fixed-point of Π? T π instead (Π? is a projection into F w.r.t. L? -norm) Π T π is a contraction in L -norm L -projection is numerically expensive when the number of states is large or infinite LSTD searches for the fixed-point of Π 2,µ T π Π 2,µ g = arg min f F f g 2,µ

53 Least-Squares Temporal-Difference Learning (LSTD) When the fixed-point of Π µ T π exists, we call it the LSTD solution V TD = Π µ T π V TD T π V π T π V TD T π F Π µv π V TD = Π µ T π V TD r π,ϕ i µ }{{} b i T π V TD V TD,ϕ i µ = 0, i = 1,...,d r π +γp π V TD V TD,ϕ i µ = 0 d ϕ j γp π (j) ϕ j,ϕ i µ αtd }{{} = 0 i=1 A ij A α TD = b In general,π µt π is not a contraction and does not have a fixed-point. If µ = µ π, the stationary dist. of π, thenπ µ πt π has a unique fixed-point.

54 LSTD Algorithm Proposition (LSTD Performance) V π V TD µ π 1 inf 1 γ 2 V F Vπ V µ π LSTD Algorithm We observe a trajectory generated by following the policy π (X 0, R 0, X 1, R 1,...,X N ) where X t+1 P ( X t,π(x t ) ) and R t = r ( X t,π(x t ) ) We build estimators of the matrix A and vector b  ij = 1 N 1 ϕ i (X t ) [ ϕ j (X t ) γϕ j (X t+1 ) ], bi = 1 N 1 ϕ i (X t )R t N N t=0 t=0  α TD = b, VTD ( ) = φ( ) α TD when n then  A and b b, and thus, α TD α TD and V TD V TD.

55 LSTD Error Bound When the Markov chain induced by the policy under evaluation π has a stationary distribution µ π (Markov chain is ergodic - e.g.β-mixing), then Theorem (LSTD Error Bound) Let Ṽ be the truncated LSTD solution computed using n samples along a trajectory generated by following the policy π. Then with probability 1 δ, we have ( ) V π Ṽ µ π c d log(d/δ) inf 1 γ 2 f F Vπ f µ π + O n ν n = # of samples, d = dimension of the linear function space F ν = the smallest eigenvalue of the Gram matrix ( ϕ i ϕ j dµ π ) i,j (Assume: eigenvalues of the Gram matrix are strictly positive - existence of the model-based LSTD solution) β-mixing coefficients are hidden in O notation

56 LSTD Error Bound LSTD Error Bound V π Ṽ µ π c 1 γ 2 inf f F Vπ f µ π + O }{{} approximation error ( ) d log(d/δ) n ν } {{ } estimation error Approximation error: it depends on how well the function space F can approximate the value function V π Estimation error: it depends on the number of samples n, the dim of the function space d, the smallest eigenvalue of the Gram matrix ν, the mixing properties of the Markov chain (hidden in O)

57 LSPI Error Bound Theorem (LSPI Error Bound) Let V 1 F be an arbitrary initial value function, Ṽ0,...,ṼK 1 be the sequence of truncated value functions generated by LSPI after K iterations, and π K be the greedy policy w.r.t. ṼK 1. Then with probability 1 δ, we have V V π K ρ { ( )] } 4γ d log(dk/δ) (1 γ) 2 CC ρ,µ[ ce 0 (F)+O +γ K 1 2 R max n ν µ

58 LSPI Error Bound Theorem (LSPI Error Bound) Let V 1 F be an arbitrary initial value function, Ṽ0,...,ṼK 1 be the sequence of truncated value functions generated by LSPI after K iterations, and π K be the greedy policy w.r.t. ṼK 1. Then with probability 1 δ, we have V V π K ρ { ( )] } 4γ d log(dk/δ) (1 γ) 2 CC ρ,µ[ ce 0 (F)+O +γ K 1 2 R max n ν µ Approximation error: E 0 (F) = sup π G( F) inf f F V π f µ π

59 LSPI Error Bound Theorem (LSPI Error Bound) Let V 1 F be an arbitrary initial value function, Ṽ0,...,ṼK 1 be the sequence of truncated value functions generated by LSPI after K iterations, and π K be the greedy policy w.r.t. ṼK 1. Then with probability 1 δ, we have V V π K ρ { ( )] } 4γ d log(dk/δ) (1 γ) 2 CC ρ,µ[ ce 0 (F)+O +γ K 1 2 R max n ν µ Approximation error: E 0 (F) = sup π G( F) inf f F V π f µ π Estimation error: depends on n, d,ν µ, K

60 LSPI Error Bound Theorem (LSPI Error Bound) Let V 1 F be an arbitrary initial value function, Ṽ0,...,ṼK 1 be the sequence of truncated value functions generated by LSPI after K iterations, and π K be the greedy policy w.r.t. ṼK 1. Then with probability 1 δ, we have V V π K ρ { ( )] } 4γ d log(dk/δ) (1 γ) 2 CC ρ,µ[ ce 0 (F)+O +γ K 1 2 R max n ν µ Approximation error: E 0 (F) = sup π G( F) inf f F V π f µ π Estimation error: depends on n, d,ν µ, K Initialization error: error due to the choice of the initial value function or initial policy V V π 0

61 LSPI Error Bound LSPI Error Bound V V π K ρ { ( )] } 4γ d log(dk/δ) (1 γ) 2 CC ρ,µ[ ce 0 (F)+O +γ K 1 2 R max n ν µ Lower-Bounding Distribution There exists a distributionµ such that for any policy π G( F), we have µ Cµ π, where C < is a constant and µ π is the stationary distribution of π. Furthermore, we can define the concentrability coefficient C ρ,µ as before.

62 LSPI Error Bound LSPI Error Bound V V π K ρ { ( )] } 4γ d log(dk/δ) (1 γ) 2 CC ρ,µ[ ce 0 (F)+O +γ K 1 2 R max n ν µ Lower-Bounding Distribution There exists a distributionµ such that for any policy π G( F), we have µ Cµ π, where C < is a constant and µ π is the stationary distribution of π. Furthermore, we can define the concentrability coefficient C ρ,µ as before. ν µ = the smallest eigenvalue of the Gram matrix ( ϕ i ϕ j dµ) i,j

63 Discussion we obtain the optimal rate of regression and classification for RL (ADP) algorithms What makes RL more challenging then? the propagation of error (control problem) the approximation error is more complex the sampling problem (how to choose µ - exploration problem)

64 Other Finite-Sample Analysis Results in RL Approximate Value Iteration [MS08] Approximate Policy Iteration LSTD and LSPI [LGM10, LGM11] Bellman Residual Minimization [MMLG10] Modified Bellman Residual Minimization [ASM08] Classification-based Policy Iteration [FYG06, LGM10, GLGS11] Regularized Approximate Dynamic Programming L 2 -Regularization L 2 -Regularized Policy Iteration [FGSM08] L 2 -Regularized Fitted Q-Iteration [FGSM09] L 1 -Regularization and High-Dimensional RL Lasso-TD [GLMH11] LSTD (LSPI) with Random Projections [GLMM10]

65 Bibliography I Antos, A., Szepesvári, Cs., and Munos, R. Learning Near-Optimal Policies with Bellman Residual Minimization-based Fitted Policy Iteration and a Single Sample Path. Machine Learning Journal, 71:89 129, Farahmand, A., Ghavamzadeh, M., Szepesvári Cs., and Mannor, S. Regularized Policy Iteration. Proceedings of Advances in Neural Information Processing Systems 21, pp , Farahmand, A., Ghavamzadeh, M., Szepesvári Cs., and Mannor, S. Regularized Fitted Q-iteration for Planning in Continuous-Space Markovian Decision Problems. Proceedings of the American Control Conference, pp , Fern, A., Yoon, S., and Givan, R. Approximate Policy Iteration with a Policy Language Bias: Solving Relational Markov Decision Processes. Journal of Artificial Intelligence Research, 25:85 118, Gabillon, V., Lazaric, A., Ghavamzadeh, M., and Scherrer, B. Classification-based Policy Iteration with a Critic. Proceedings of the Twenty-Eighth International Conference on Machine Learning, pp , Ghavamzadeh, M., Lazaric A., Munos, R., and Hoffman, M. Finite-Sample Analysis of Lasso-TD. Proceedings of the Twenty-Eighth International Conference on Machine Learning, pp , Ghavamzadeh, M., Lazaric, A., Maillard, O., and Munos, R. LSTD with Random Projections. Proceedings of Advances in Neural Information Processing Systems 23, pp , 2010.

66 Bibliography II Lazaric A., Ghavamzadeh, M., and Munos, R. Analysis of a Classification-based Policy Iteration Algorithm. Proceedings of the Twenty-Seventh International Conference on Machine Learning, pp , Lazaric A., Ghavamzadeh, M., and Munos, R. Finite-Sample Analysis of LSTD. Proceedings of the Twenty-Seventh International Conference on Machine Learning, pp , Lazaric A., Ghavamzadeh, M., and Munos, R. Finite-Sample Analysis of Least-Squares Policy Iteration. Accepted at the Journal of Machine Learning Research, Maillard, O., Munos, R., Lazaric A., and Ghavamzadeh, M. Finite-Sample Analysis of Bellman Residual Minimization. Proceedings of the Second Asian Conference on Machine Learning, pp , Munos, R. and Szepesvári, Cs. Finite-Time Bounds for Fitted Value Iteration. Journal of Machine Learning Research, 9: , Munos, R. Performance Bounds in Lp-norm for Approximate Value Iteration. SIAM Journal of Control and Optimization, Munos, R. Error Bounds for Approximate Policy Iteration. Proceedings of the Nineteenth International Conference on Machine Learning, pp , 2003.