On Ergodic Impulse Control with Constraint

Transcription

1 On Ergodic Impulse Control with Constraint Maurice Robin Based on joint papers with J.L. Menaldi University Paris-Sanclay 9119 Saint-Aubin, France ( IMA, Minneapolis, MN May 7 11, 218 MR (UP-S) Ergodic Impulse Control with Constraint 1 / 19

2 Content - Statement of the problem - HJB equation - Solution of the HJB equation - Existence of an optimal control - Extensions - References MR (UP-S) Ergodic Impulse Control with Constraint 2 / 19

3 Introduction Statement of the Problem Statement of the Problem (as in JL Menaldi s talk except for the cost and ergodic assumptions) - The uncontrolled state is described by a Markov-Feller process x t (values in E metric compact) - impulse control ν = (θ i, ξ i ) i 1, θ i increasing sequence of stopping times, ξ i E valued random variable - constraint on impulse controls: θ i > and θ i is a jump time of the signal process y t y τn =, y t = t τ n for τ n t τ n+1, n 1, T n = τ n+1 τ n, conditionally to x t as IID random variables with intensity λ(x, y) - ξ i Γ(x θi ), Γ(x) closed set of E and ξ Γ(x), Γ(ξ) Γ(x) MR (UP-S) Ergodic Impulse Control with Constraint 3 / 19

4 Introduction Statement of the Problem (2) Statement of the Problem (2) - running cost f (x, y) and impulse cost c(x, ξ), both positive bounded and continuous, c(x, ξ) c > and c(x, ξ) + c(ξ, ξ ) c(x, ξ ) - Mg(x) inf ξ Γ(x) {c(x, ξ) + g(x)} is assumed to be continuous if g is continuous and there exists a measurable selector ˆξ(x, g). V will denote the set of admissible controls V = {(θ i, ξ i ), i 1, θ 1 >, y θi = } V the set of admissible controls satisfying the constraint, but θ 1 = is allowed MR (UP-S) Ergodic Impulse Control with Constraint 4 / 19

5 Introduction Statement of the Problem (3) Statement of the Problem (3) - Controlled process The controlled process for a control ν is defined on the product space Ω, Ω = D(R + ; E R + ) by a probability P ν xy, and (x ν t, y ν t ) = (x i t, y i t ) for θ i 1 t < θ i evolves as the uncontrolled process between impulses instants. MR (UP-S) Ergodic Impulse Control with Constraint 5 / 19

6 Introduction Statement of the Problem (4) Statement of the Problem (4) - The average cost to be minimized: J(x, y, ν) = lim inf T 1 { T T Eν xy f (xs ν, ys ν )ds + i } 1 θi T c(x i 1 θ i, ξ i ) µ(x, y) = inf{j(x, y, ν) : ν V} We will use an auxiliary problem J(x, 1 { y, ν) = lim inf n E ν E ν τ n xy f (xs ν, ys ν )ds + xyτ n i } 1 θi T c(x i 1 θ i, ξ i ) µ (x, y) = inf{j(x, y, ν) : ν V } MR (UP-S) Ergodic Impulse Control with Constraint 6 / 19

7 Introduction Additional Assumptions Additional Assumptions λ(x, y) is bounded and continuous and Ergodicity assumption: < a 1 E x (τ 1 ) a 2 P(x, B) = E x 1 B (x τ1 ) B B(E) satisfies: there exists a positive measure m on E s.t. < m(e) 1 and P(x, B) m(b) B B(E) Example: reflected diffusions and reflected diffusions with jumps for which the transition density satisfies p(x, t, x ) k(ε) on E [ε, [ E MR (UP-S) Ergodic Impulse Control with Constraint 7 / 19

8 Solution HJB Equation HJB Equation Heuristic argument with the discounted problem { { u α (x, ) = min Mu α τ 1 } } (x, ), E x e αt f dt + e ατ 1 u α (x τ1, ) Let m α = inf u α(x, ), w α = uα m α, { { w α (x, ) = min Mw α τ 1 } } (x, ), E x e αt (f αm α )dt+e ατ 1 w α (x τ1, ) Assuming w α w a function, and αm α µ a constant, { { τ 1 } } w (x, ) = min Mw (x, ), E x (f µ )dt + w (x τ1, ) One can also use heuristic argument on { u T (t, x, y) = inf E ν T t x f dt + i } 1 θi T tc(x i 1 θ i, ξ i ). MR (UP-S) Ergodic Impulse Control with Constraint 8 / 19

9 Solution HJB Equation HJB Equation (2) - For the auxiliary problem: Find (w, µ ), µ constant, such that { { τ 1 } } w (x, ) = min Mw (x, ), E x [f µ ]ds + w (x τ1, ) then w (x, y), for y >, is given by w (x, y) = E xy { τ 1 } [f µ ]ds + w (x τ1, ) - For the initial problem: (w, µ ) gives w(x, y) as w(x, y) = E xy { τ 1 } [f µ ]ds + w (x τ1, ) MR (UP-S) Ergodic Impulse Control with Constraint 9 / 19

10 Solution Solution (µ, w ) Solution (µ, w ) - A discrete time HJB equation for ( µ, w (x, ) ) : define l(x) = E x { τ 1 } f (x s, y s )ds, Pg(x) = E x g(x τ1 ) τ(x) = E x τ 1, w (x) w (x, ), then { w (x) = min inf ξ Γ(x) { c(x, ξ) + w (ξ) }, l(x) µ τ(x) + Pw (x)} is equivalent to the previous HJB equation Proposition There exists a solution (µ, w ) in R + C(E) of the HJB equation. Remark: If w is solution, w + constant is also solution. The uniqueness of µ will come from the stochastic interpretation. MR (UP-S) Ergodic Impulse Control with Constraint 1 / 19

11 Solution Solution (µ, w ) Arguments The proof uses the following equivalent form of the HJB equation w (x) = inf ξ Γ(x) {x} { l(ξ) + 1ξ x c(x, ξ) µ τ(ξ) + Pw (ξ) } = Rw and the fact that P(x, β) τ(x)γ(β) for a positive measure γ on E satisfying γ(e) > 1 β τ(x), < β < 1. Then R is a contraction on C(E). w is the unique fixed point and µ = E w (x)γ(dx) MR (UP-S) Ergodic Impulse Control with Constraint 11 / 19

12 Solution Existence of an Optimal Control Existence of an Optimal Control Additional assumptions: the (uncontrolled) Markov process (x t, y t ) has a unique invariant measure ζ and there exists a continuous function h(x, y) s.t. { τ } E xy h(x τ, y τ ) = h(x, y) E xy [f (x t, y t ) f ]dt, for any finite stopping time τ, with f = f (x, y)dζ. E R + Remark: if f (x, y) = f (x), then it is sufficient to assume that the Poisson equation for x t, i.e. A x h = f (x) f has a continuous solution. MR (UP-S) Ergodic Impulse Control with Constraint 12 / 19

13 Solution Existence of an Optimal Control (2) Existence of an Optimal Control (2) Theorem With the additional assumption, we have µ = inf ( J(x,, ν), ν V ) and there exists an optimal control ˆν of the auxiliary problem case 1. µ = f : then it is optimal to do nothing case 2. µ < f : one can rewrite the HJB equation w(x) = min { ψ(x), l(x) + P w(x) } with w = w h(x, ), ψ = Mw h(x, ), l(x) = ( f µ )E x τ 1. This is the HJB equation of a discrete optimal stopping problem which has an optimal control ˆη = inf { n : w(x n ) = ψ(x n ) }, i.e., ˆη = inf { n : w (x n ) = Mw (x n ) } where x n is the Markov chain x τn. From this, we deduce an optimal control with θ 1 = τˆη, and θ i = θˆηi with ˆη i = inf { n ˆη i : w (x n ) = Mw (x n ) }. MR (UP-S) Ergodic Impulse Control with Constraint 13 / 19

14 Solution Existence of an Optimal Control (3) Existence of an Optimal Control (3) Corollary µ = inf { J(x, y, ν), v V } and the optimal control ˆν obtained by translation by τ 1 of the control ˆν. The final result is given by Theorem µ = inf{j(x, y, ν) : ν V} = J(x, y, ˆν) MR (UP-S) Ergodic Impulse Control with Constraint 14 / 19

15 Solution Existence of an Optimal Control (4) Existence of an Optimal Control (4) A first step is to prove: Proposition (µ, w ) being the solution previously obtained and recalling that (µ, w) is solution of w(x, y) = E x { τ 1 } [f µ ]ds + w (x τ1, ) A xy w(x, y) + λ(x, y)[w(x, ) Mw(x, )] + = f µ. where A xy is the (weak) infinitesimal generator of the uncontrolled process A xy ϕ = A x ϕ + ϕ y + λ(x, y)[ ϕ(x, ) ϕ(x, y) ]. MR (UP-S) Ergodic Impulse Control with Constraint 15 / 19

16 Solution Existence of an Optimal Control (5) Existence of an Optimal Control (5) To prove the proposition, one first shows which gives w (x) = min{w(x, ), Mw(x, )} { τ 1 w(x, y) = E xy [f µ ]dt + w(x τ1, ) [w(x τ1, ) Mw] +} from which we deduce that equation. Next, the proposition allows us to show that M T = T [f (x t, y t ) µ ]ds + w(x T, y T ) is a submartingale. This gives µ J(x, y, ν), ν V, and, from the first expression of w(x, y), on obtains µ = J(x, y, ν). MR (UP-S) Ergodic Impulse Control with Constraint 16 / 19

17 Solution Existence of an Optimal Control (5) Extensions E locally compact. If we assume Φ(t)C (E) C (E) all other assumptions being unchanged, one can still obtain the results on the HJB equation. However, the additional assumption is no longer sufficient to get the result on the existence of an optimal cost. adding h(x, ) bounded would be sufficient (e.g., if A xy h = f f has a bounded solution), but more general assumptions would require further work. When λ is independent of x and f independent of y, one can obtain an optimal control without an additional assumption. MR (UP-S) Ergodic Impulse Control with Constraint 17 / 19

18 Solution Existence of an Optimal Control (5) Extensions (2) P(x, B) m(b) is also restrictive for E locally compact Replacing by a localized condition like P(x, B) α1 K (x)m(b), allows to obtains the results on the HJB equations and, if h(x, ) is bounded, the existence of an optimal control. MR (UP-S) Ergodic Impulse Control with Constraint 18 / 19

19 Solution References (among others) References (among others) Besides the classical works on impulse control, A. Bensoussan and J.L. Lions (book 1982), etc,..., For discrete time control: A. Bensoussan (book, 211), O. Hernandez-Lerma and J. Lasserre (books 1996, 1999),... For average control, ergodicity: Bensoussan (book 1988), M. Kurano (paper 1989), F. Luque-Vasquez and O. Hernandez-Lerma (paper 1999), L. Stettner (paper 1986), D. Gatarek and L. Stettner (paper 199), A. Arapostathis et al. (survey paper 1993), M.G. Garroni and J.L. Menaldi (book 22), Several papers with J.L. Menaldi (1997, 213,... ), Papers to appear with J.L. Menaldi, H. Jasso-Fuentes, T. Prieto-Rumeau. MR (UP-S) Ergodic Impulse Control with Constraint 19 / 19