Difference of Convex Functions Programming for Reinforcement Learning (Supplementary File)

Transcription

1 Difference of Convex Functions Programming for Reinforcement Learning Sulementary File Bilal Piot 1,, Matthieu Geist 1, Olivier Pietquin,3 1 MaLIS research grou SUPELEC - UMI 958 GeorgiaTech-CRS, France LIFL UMR 80 CRS/Lille 1 - SequeL team, Lille, France 3 University Lille 1 - IUF Institut Universitaire de France, France bilal.iot@lifl.fr, matthieu.geist@suelec.fr, olivier.ietquin@univ-lille1.fr This sulementary material rovides the roofs of the results given in the main aer. 1 Proof of Th. 1 Theorem 1. Let ν S A, µ S A, ˆπ A S and C 1 ν, µ, ˆπ [1, + [ {+ } the smallest constant verifying ν t 0 γt Pπ ṱ C 1ν, µ, ˆπµ, then: Q R S A, Q Q π,ν C1 ν, µ, π + C 1 ν, µ, π where π is greedy with resect to Q and π any otimal olicy. 1 T Q Q,µ, 1 Proof. This roof is widely insired by the roof of a similar result for value functions Th. 5.3 in [1]. First, we show that, comonentwise: Q R S A, Q Q π [I γp π 1 + I γp π 1 ] T Q Q. To do so, we use rincially the fact that T Q T π Q and T π Q = T Q as π is greedy with resect to Q. Thus: Q Q π = a T π Q T Q + T Q T π Q π, T π Q T π Q + T π Q T π Q π, b = γp π Q Q + γp π Q Q π, = γp π Q Q π + Q π Q + γp π Q Q π, where equality a comes from the fact that Q = T π Q and Q π = T π Q π and inequality b from the fact that T Q T π Q and T π Q = T Q. Hence I γp π Q Q π γp π P π Q π Q. In addition as I γp π 1 = t 0 γt Pπ t is a matrix with only ositive elements, we can multily by I γp π 1 on each side of the matricial inequality and conserve the order: Moreover, we have: Q Q π γi γp π 1 P π P π Q π Q. I γp π Q π Q = γp π Q Q π + Q π Q, = T π Q T π Q π + Q π Q = c T Q Q, 1

2 where equality c comes from the fact that Q π = T π Q π and T π Q = T Q. Therefore, we have: Q Q π γi γp π 1 P π P π Q π Q, = I γp π 1 γp π γp π I γp π 1 T Q Q, = I γp π 1 I γp π I γp π I γp π 1 T Q Q, = [I γp π 1 I γp π 1 ]T Q Q, [I γp π 1 + I γp π 1 ] T Q Q. ow, as it is mentioned in [1], in order to obtain an L,µ bound, we remark that if u R S A and v R S A are two vectors with ositive elements and is stochastic matrix of size S A S A such that u v and if ν S A and µ S A are two distributions such that ν Cµ where C is a constant greater to 1, then : Indeed: u,ν = s,a S A [us, a] νs, a u,ν C 1 v,µ. 3 d s,a S A e f s,a S A s,a S A s,a S A C s,a S A s, a, s, a vs, a νs, a, s, a, s, a [vs, a ] νs, a, µs, a [vs, a ] = C v,µ, where inequality d is true because u v, inequality e is true using Jensen s Inequality and inequality f comes from ν Cµ. To establish our bound Eq. 1, it is sufficient to remark that the inequality Eq. can be written: Q Q π A T Q Q, where A = 1 γ [I γp π 1 + I γp π 1 ] is a stochastic matrix. Moreover by definition of C 1 ν, µ, π and C 1 ν, µ, π we have: C1 ν, µ, π + C 1 ν, µ, π νa µ. Thus, if we rewrite Eq. 3, where Q Q π lays the role of u, 1 γ T Q Q lays the role of v, A lays the role of, and C1ν,µ,π+C1ν,µ,π lays the role of C, then we have: Q Q π,ν C1 ν, µ, π + C 1 ν, µ, π 1 T Q Q,µ. Proof of Th. Theorem. Let η ]0, 1[ and M be a finite deterministic MDP, with robability at least 1 η, we have: Q Q, T Q Q,µ T Q Q,µ + R ε,

3 where ε = hln h +1+ln 4 η and h = A d + 1. Moreover, with robability at least 1 η: ɛ OBRM = T Q Q,µ ɛ B + R 1 ε ln1/η +, where ɛ B = min Q Q T Q Q,µ is the error due to the choice of features. Proof. Here, we work with finite deterministic MDPs. This means that for each state-action coule s, a, there exists a unique next state s. Let us note l S S A the function that mas each state-action coule s, a to its next state s. Then, we have: Q R S A, s, a S A, T Qs, a = Rs, a + γ max Qls, a, b. The result is based on Th 5.3 of [], briefly recalled here. Let F R X be a set of measurable bounded real-valued functions where X is a measurable set. In articular, we have f F, x X, a fx b where a, b R. Let x i be indeendent and identically distributed random variables taking their values in X and such that x i F where F is a distribution over X. If F has a finite VC-dimension Vanik-Chervonenkis dimension vf h and η ]0, 1[ then with robability at least 1 η, we have: f F, fxf dx 1 x X fx i + b a ε, where ε = hln h +1+ln 4 η. And with robability at least 1 η: ε ln1/η min fxf dx min fx i + b a +. f F x X f F Our result has exactly the same form where X = S A, the random variables x i are relaced by S i, A i, the distribution F = µ S A, the sace F is relaced by Q = { T Q Q, where Q Q}, a = 0 and b = R 1 γ. The only thing left to rove our result is to show that the VC-dimension of Q, v Q, is such that v Q A d + 1. First, let recall some definitions relative to the VC-dimension of a set of functions. Let f R X be a real-valued function where X is a set and x i, t i X R a sequence of coules of one element of X and one real value, mf, x i, t i = 1 {fxi t i} is a boolean which says if fx i is greater than t i or not. Moreover, Mf, x i, t i = mf, x i, t i can be seen as a boolean vector of size and we call it the message relative to both the function f and the sequence x i, t i. Let F RX, F, x i, t i is the number of ossible messages Mf, x i, t i obtained when f F: F, x i, t i = Card{Mf, x i, t i, f F}, where Card denotes the cardinal of a given set. As Mf, x i, t i is a boolean vector of size, we have F, x i, t i. In addition, we define F, = su F, x xi,t i i, t X R i the maximum number of ossible messages when f F that a given sequence x i, t i can roduce. Finally, the VC-dimension of F is defined by: vf = inf {F, < }. In our roof, the followings roerties relative to VC-dimensions of functions sets are needed. Proerty 1. Let F k K k=1 be a sequence of set of functions where F k R X and vf k is finite. Then, the set of functions F = {max k [ 1:K ] f k, where k [ 1 : K ], f k F k } has a finite VC-dimension lower that K k=1 vf k. 3

4 Let x i, t i X R, f k F k K k=1 and f = max k [ 1:K ] f k, then: Mf, x i, t i = Mf 1, x i, t i Mf, x i, t i Mf K, x i, t i, where is the boolean disjunction the inclusive or. Thus, the number of ossible messages F, x i, t i is such that: This imlies that: F, x i, t i F, F k, x i, t i. F k,. ow, if we choose such that > max k [ 1:K ] vf k, we have: F, vfk K = vfk k=1. So, F has a finite VC-dimension lower that K k=1 vf k. A second interesting result is a corollary of the revious roosition. Proerty. Let F R X be a set of functions with a finite VC-dimension, then the set of functions F. = { f, f F} has a VC-dimension lower than vf. Indeed, to rove the result, we remark that f = maxf, f and as the set F = { f, f F} has the same VC-dimension than F, we aly the revious result to conclude. The last result needed is the following. Proerty 3. Let F R X + be a set of functions with a finite VC-dimension, then the set F = {f, f F}, where 1, has the same VC-dimension. To show this roerty, let f F and x i, t i X R, then: Mf, x i, t i = Mf, x i, sgnt i t i, where sgn is the sign function, thus F, x i, t i = F, x i, sgnt i t i. As the function t sgnt t is a bijection, then: su F, x i, t i = su F, x i, sgnt i t i. x i,t i X R x i,t i X R So, F, = F, which imlies that vf = vf. ow, let show that the VC-dimension of Q is such that v Q A d + 1. To do so, we are going to roceed in several stes. The first ste is to remark that: d T Q θ s, a Q θ s, a = max θ k [γφ k ls, a, b φ k s, a] + Rs, a. k=1 Thus, if we note b A, k [ 1 : d ], ψ b k s, a = γφ kls, a, b φ k s, a, we have: d T Q θ s, a Q θ s, a = max [ θ k ψks, b a + θ 0 Rs, a]. where θ 0 = 1. Let b A, the set of functions F b = {f θ = d k=1 θ kψk b + θ 0R, θ R d } has a finite VC-dimension lower than d + 1 as the functions f θ F b deends linearly on d + 1 arameters [] where one of them θ 0 is fixed. ow, we want to show that the set F = {f θ = max [ d k=1 θ kψk b +R], θ Rd } has a finite VC-dimension lower than A d+1. To do so, we remark that F = {f = max f b, where b A, f b F b }, thus by alying roerty. 1 to F we obtain that its VC-dimension is lower than A d+1. ow, let define the k=1 4

5 set of functions F. = {f θ = max [ d k=1 θ kψ b k + R], θ Rd } = { T Q θ Q θ, θ R d }. We remark that, F. = { f, f F}, thus, by using roerty, the VC-dimension of F. is lower than A d+1. Finally, we define the set F = { T Q θ Q θ, θ R d }, where 1. We have F = {f, f F. }, thus, by alying roerty 3, we have that the VC-dimension of F is lower than A d + 1. As Q F, we have v Q A d Proof of Th.4 Theorem 3. There exists exlicit olyhedral decomositions of J,µ when = 1 and =. For = 1: J 1,µ = G 1,µ H 1,µ, where G 1,µ = 1 maxg i, h i and H 1,µ = 1 g i+h i, with g i = φs i, A i,. + RS i, A i and h i = γ s S P s S i, A i max a A φs, a,.. For = : J,µ = G,µ H,µ, where G,µ = 1 [g i + h i ] and H,µ = 1 [g i + h i ] with: g i = maxg i, h i + g i h i = maxg i, h i + h i φs i, A i + γ s S P s S i, A i φs, a 1,. RS i, A i φs i, A i + γ s S P s S i, A i φs, a 1,. RS i, A i,. Proof. When = 1, it is sufficient to remark that for two functions g, h R E, g h = maxg, h g +h. Thus, let G 1,µ = 1 maxg i, h i and H 1,µ = 1 g i +h i which are convex and continuous as a finite maximum of convex and continuous functions and a ositively weighted sum of convex and continuous functions are convex and continuous, then J 1,µ = G 1,µ H 1,µ. When =, the decomosition is less straightforward. An imortant roerty that we use is the fact that f is a convex and continuous functions if f is a ositive and continuous convex function. The first thing to do is to find a decomosition of f i = g i h i such that g i and h i are ositive and continuous convex functions. To do so, it is sufficient to remark that: g i = maxg i, h i + g i h i = maxg i, h i + h i φs i, A i + γ s S P s S i, A i φs, a 1,. RS i, A i φs i, A i + γ s S P s S i, A i φs, a 1,. RS i, A i are ositive and continuous convex functions. Thus: J,µ = 1 [g i h i ] = 1 [g i + h i ] 1 [g i + h i ]. As g i and h i are convex, continuous and ositive then g i + h i and [g i + h i ] are convex and continuous. So, if we note G,µ = 1 [g i + h i ] and H,µ = 1 [g i + h i ] which are convex and continuous, we have J,µ = G,µ H,µ. We also remark that G,µ, H,µ, G 1,µ and H,µ are olyhedral and J,µ is under bounded by 0, thus DCA has better convergence roerties than in the classical case. References [1] R. Munos. Performance bounds in L -norm for aroximate value iteration. SIAM journal on control and otimization, 007. [] V. Vanik. Statistical learning theory. Wiley, 1998.,. 5