On Stopping Times and Impulse Control with Constraint

Transcription

1 On Stopping Times and Impulse Control with Constraint Jose Luis Menaldi Based on joint papers with M. Robin (216, 217) Department of Mathematics Wayne State University Detroit, Michigan 4822, USA ( IMA, Minneapolis, MN May 7 11, 218 Wed May 9, 218, 1:-1:5 JLM (WSU) Control Problems with Constraint 1 / 24

2 1.- Introduction Table of Content Content 1.- Introduction: Stopping, Impulse, and Signal 2.- Stopping with Constraint 3.- Impulse Control with Constraint - Setting, Assumptions Formal Analysis, Dynamic Programming Solving HJB Equation Extensions 7.- Hybrid Control, Stopping, Impulse, Switching JLM (WSU) Control Problems with Constraint 2 / 24

3 1.- Introduction Stopping & Impulse Optimal Stopping Time & Impulse Control Problem Control θ = stop evolution of a Markov-Feller x t on a Polish space Optimal Problem = maximize/minimize the reward/cost functional { θ J x (θ, ψ) = E x e αt f (x t )dt + e αθ ψ(x θ ), (1) θ stopping time, f, ψ (running & terminal), α > (discount). Optimal cost: v(x) = inf { J x (θ, ψ) : θ stopping time. Cost functional (with intervention) { J x (ν) = E x e αt f (xt ν )dt + e αϑ i c(xϑ ν i, ξ i ), (2) for any ˆν impulse/switching control, and the optimal cost function v(x) = inf ν i= J x(ν). JLM (WSU) Control Problems with Constraint 3 / 24

4 1.- Introduction Stopping & Impulse - Constraint VI & QVI - Stopping & Impulse - Constraint The dynamic programming gives a HJB equation (VI or QVI) of the form min { Av αv + f, ψ v = or min { Av αv + f, Mv v =, with A = infinitesimal generator of Markov-Feller process {x t, t [, [ (with states in E, Polish space), and Mu(x) = inf ξ {c(x, ξ) + u(ξ). This QVI can be solved by using the sequence of variational inequalities min { Av n αv n + f, Mv n 1 v n =, where v n is the optimal cost for the problem with stopping cost Mv n 1. Bensoussan and Lions (book 1978), Robin (thesis 1978), Menaldi (thesis 198), etc, etc, Peskir and Shiryaev (book 26), etc, etc, etc. (extremely long list, with variations) Constraint: Wait for a signal. For instance, the system evolves according to a Wiener process w t (with drift b and diffusion σ) and the signal is the jumps of a Poisson process N t, independent of the Wiener process. Dupuis and Wang (22) [also, Lempa (212), Liang and Wei (214)] studied this case for a geometric Brownian process. JLM (WSU) Control Problems with Constraint 4 / 24

5 1.- Introduction Signal Constraint: Wait for a Signal The dynamic programming equation (or HJB equation) takes the form Au αu = λ[u ψ] +, with f =, where the verification theorem is based on an explicit solution of the HJB equation and on the solution of a discrete time problem. Objective: To generalize the above problem in two directions: - to replace the 1-d Wiener process with a Markov-Feller process, - to replace the Poisson process by a more general counting process. The simplest model {τ n : n 1 are the jumps of a Poisson process, i.e., the time between two consecutive jumps {T 1 = τ 1, T 2 = τ 2 τ 1, T 3 = τ 3 τ 2,... is necessarily an IID sequence, exponentially distributed. However, we focus first on {T n : n 1 being a sequence of IID random variables with common law π (not exponential in general, i.e., not memoryless). JLM (WSU) Control Problems with Constraint 5 / 24

6 1.- Introduction Time Elapsed Since Last Signal Signal as a Process Let us introduce an homogeneous Markov process {y t : t, y t [, [, independent of {x t : t time elapsed since the last signal. So that almost surely, the cad-lag paths t y t are piecewise differentiable ẏ t = 1, with jumps only back to zero, and expected infinitesimal generator A 1 ϕ(y) = y ϕ(t) + λ(y)[ϕ() ϕ(y)], y, (3) where the intensity λ(y) is a Borel measurable function. Thus, the signals are defined as functionals on {y t : t, namely, τ =, τ n = inf { t > τ n 1 : y t =, n 1. (4) The couple {(x t, y t ) : t is an homogeneous Markov process in continuous time, and a signal arrives at a random instant τ if and only if y τ = (and τ > ). Stopping with Constraint JLM (WSU) Control Problems with Constraint 6 / 24

7 2.- Stopping Problem Setting Stopping Two Optimal Costs - Stopping Problem C b = C b (E [, [) continuous & bounded functions on E [, [, {(x t, y t ) : t Markov-Feller process, i.e., for any ϕ(x, y) in C b, The cost function is (x, y, t) E x,y { ϕ(xt, y t ) is continuous, and (5) α > and f, ψ, C b (E [, [), (6) J x,y (θ, ψ) = E x,y { θ which yields an optimal cost e αt f (x t, y t )dt + e αθ ψ(x θ, y θ ), v(x, y) = inf { J x,y (θ, ψ) : θ >, y θ =, (7) i.e., θ is any admissible stopping time, and an auxiliary optimal cost is defined as v (x, y) = inf { J x,y (θ, ψ) : y θ =, (8) which is an homogeneous Markov model. JLM (WSU) Control Problems with Constraint 7 / 24

8 2.- Stopping Problem HJB Stopping Dynamic Programming - HJB Equations - Optimal Stopping Cost If τ = inf { t > : y t = then HJB equations and { u (x, ) = min {ψ, τ E x,y e αt f (x t, y t )dt + e ατ u (x τ, y τ ), (9) u(x, y) = E x,y { τ e αt f (x t, y t )dt + e ατ min{ψ, u(x τ, y τ ), (1) are the corresponding HJB equations, and both problems are related by the equation u(x, y) = E x,y { τ e αt f (x t, y t )dt + e ατ u (x τ, y τ ). (11) Note that y τ = and u (x, y) = u(x, y) for any y >. JLM (WSU) Control Problems with Constraint 8 / 24

9 2.- Stopping Problem Solving Stopping Solving Optimal Stopping Problems Theorem Assume (5) & (6). Then VI (9) & (1) have each a unique solution in C b (E [, [), which are the optimal costs (7) & (8). Moreover, the first exit time from the continuation region [u < ψ] is optimal (discrete s.t.) ˆθ = inf { t > : u(x t, y t ) ψ(x t, y t ), y t =, ˆθ = inf { t : u (x t, y t ) = ψ(x t, y t ), y t = (12) are optimal, namely, u(x, y) = J x,y (ˆθ, ψ) and u (x, y) = J x,y (ˆθ, ψ). Furthermore, the relation (11) holds. Also u = min{u, ψ, which may D(A x,y ) C b (E [, [). However, the optimal cost u given by (7) belongs to D(A x,y ) and for any (x, y) in E [, [, (HJB) A x,y u(x, y) + αu(x, y) + λ(x, y)[u(x, ) ψ(x, y)] + = f (x, y). (13) IC w/constraint: Setting, Assumptions JLM (WSU) Control Problems with Constraint 9 / 24

10 3.- Impulse Control Assumptions Impulse Control Setting - Evolution If x t represents the state of the uncontrolled system then a sequence ν = {(ϑ i, ξ i ) : i 1 of stopping times {ϑ 1 ϑ 2 and E-valued random variables {ξ 1, ξ 2,... is called an impulse control. The controlled system {xt ν : t behaves likes x t for t within [ϑ i 1, ϑ i [, i 1, and xϑ ν i = ξ i, in short, instantaneously moves from x ν ϑ i (xϑ ν i ) to ξ i. A cost-per-impulse c : E Γ [c, [, c is continuous and c >, (14) with a closed subset Γ of E (= set where to jump / admissible jumps). {(x t, y t ) : t is a time-homogeneous Markov-Feller process, with infinitesimal generator, x E, y, A x,y ϕ(x, y) = A x ϕ(x, y) + y ϕ(x, y) + λ(x, y)[ϕ(x, ) ϕ(x, y)], (15) i.e., now the intensity λ depends also on x. JLM (WSU) Control Problems with Constraint 1 / 24

11 3.- Impulse Control Setting Cost per Interventions / Impulse costs First: admissible impulse control ν = {(ϑ i, ξ i ) : i 1 if y ϑi =, i 1, and ϑ 1 >. If ϑ 1 = is allowed then ν is zero-admissible. 1st decision: choose F-st ϑ 1, a F ϑ1 -meas. rv ξ 1 : Ω Γ E, cost up to ϑ 1 { J x,y (ν ϑ 1 ) = E ν ϑ ϑ 1 x,y f (x t, y t )e αt dt + e αϑ 1 c(x ϑ1, ξ 1 ), ϑ =, ϑ 2nd decision: choose a F-st ϑ 2, a F ϑ2 -meas. rv ξ 2 : Ω Γ E, cost up to ϑ 2 { J x,y (ν ϑ 2 ) = J x,y (ν ϑ 1 ) + E ν ϑ ϑ 2 1 x,y f (x t, y t )e αt dt + e αϑ 2 c(x ϑ2, ξ 2 ) ϑ 1 Iterating, cost upto ϑ k+1 (ϑ k+1 < means intervention ) { J x,y (ν ϑ k+1 ) = J x,y (ν ϑ k ) + E ν ϑ ϑ k+1 k x,y f (x t, y t )e αt dt + ϑ k + e αϑ k+1 c(x ϑk+1, ξ k+1 ). (16) JLM (WSU) Control Problems with Constraint 11 / 24

12 3.- Impulse Control Setting (cont. 1) Optimal Impulse Costs Total cost: limit J x,y (ν) = lim k J x,y (ν ϑ k+1 ). However, it is convenient to construct (in an copy-space) a probability P ν x,y such that { J x,y (ν) = E ν x,y e αt f (xt ν, yt ν )dt + e αϑ i c(x i 1,ν ϑ i, ξ i ), (17) V (or V ) = admissible (or zero-admissible) impulse controls, all relative to the initial condition (x, y ) = (x, y). Optimal cost: v(x, y) = inf { J x,y (ν) : ν V, (x, y) E [, [, (18) and also, the time-homogeneous impulse control, with an optimal cost: v (x, y) = inf { J x,y (ν) : ν V, (x, y) E [, [, (19) which is actually needed only for y =. Formal Analysis, Dynamic Programming JLM (WSU) Control Problems with Constraint 12 / 24 i=1

13 4.- Formal Analysis Dynamic Programming - IC Dynamic Programming - State Model - Time Homogeneous Two models: constraint wait to intervene until a signal arrive : (a) v(x, y) translated as intervene only when y = and t > ; (b) (Markov/stationary) v (x, y) translated as intervene only when y =. (1) Time homogeneous? v(x, y) v(x, y, s), v(x, y, ) = v(x, y) and for s >, { J x,y,s (ν) = E ν x,y e α(t s) f (xs+t)dt ν + e α(ϑ i s) c(x i 1,ν s+ϑ i, ξ i ), i v(x, y, s) = inf { J x,y,s (ν) : ν any admissible impulse control, where now admissible means intervene only when y = and s. (2) Multiple impulses? Are excluded from the optimal decision if c(x, ξ 1 ) + c(ξ 1, ξ 2 ) c(x, ξ 2 ), x E, ξ 1, ξ 2 Γ. (>?) (2) and v(x, y) v (x, y), x E, y, with = if y >, hold true. v (x, ) = min { v(x, ), inf ξ Γ {v(ξ, ) + c(x, ξ), x E. (21) JLM (WSU) Control Problems with Constraint 13 / 24

14 4.- Formal Analysis Dynamic Programming - Weak DP Equation Dynamic Programming - Weak DP/HJB Equation The relations u(x, y) = E x,y { τ 1 An equation with u alone { τ 1 u(x, y) = E x,y e αt f (x t )dt + or equivalently Also e αt f (x t )dt + e ατ1 u (x τ1, ). (22) + e ατ1 min { u(x τ1, ), inf ξ 1 Γ {u(ξ 1, ) + c(x τ1, ξ 1 ), u(x, y) = ( R(min{u(, ), (Mu(, )) ) (x, y), x, y, (23) u (x, ) = min { u(x, ), (Mu(, ))(x) = { = min {(Mu τ 1 (, ))(x), E x, e αt f (x t )dt + e ατ1 u (x τ1, ). Solving HJB Equation JLM (WSU) Control Problems with Constraint 14 / 24

15 6.- Solving HJB Equation Existence and Uniqueness HJB Equation - Existence and Uniqueness Either Au + αu = f or u (x) = E x e αt f (x t )dt, x E, (24) v, v C(u ) = {ϕ C(E) : ϕ(x) u (x), x E (25) HJB equation with u (x) = u (x, ), either u = min{mu, Ru or { u (x) = min inf {u (ξ) + c(x, ξ), ξ Γ { τ 1, E x, e αt f (x t )dt + e ατ1 u (x τ1 ) (26) Consider the scheme u n = min{mun 1, Ru n, with u = u, n 1. This equation has a unique solution in C(E), and u n = v n (, ) with { θ v n (x, y) = inf E x,y e αt f (x t )dt + e αθ Mv n 1 (x θ, ), (27) θ where the minimization is over all zero-admissible stopping times θ, i.e., θ = τ η for some discrete stopping time η. JLM (WSU) Control Problems with Constraint 15 / 24

16 6.- Solving HJB Equation Existence and Uniqueness (Thm 1) Existence and Uniqueness (Thm 1: u ) Theorem Under the previous assumptions, the decreasing sequence {u n : n 1 converges to u, and there are two constants C > and ρ in ], 1[ such that u n (x) u (x) Cρ n, x E, n 1. (28) Moreover, the HJB equation (26) has one and only one solution u within the interval C(u ), and the representation { θ u (x) = inf E x, e αt f (x t )dt + e αθ Mu (x θ ), (29) θ holds true, where the minimization is over all zero-admissible stopping times θ, i.e., θ = τ η for some discrete stopping time η. JLM (WSU) Control Problems with Constraint 16 / 24

17 6.- Solving HJB Equation Existence and Uniqueness (Thm 2) Existence and Uniqueness (Thm 2: u) Theorem Under previous assumptions, u n u, and C > and ρ in ], 1[ such that u n (x) u(x) Cρ n, x E, n 1. (3) Moreover, the HJB equation (23) has one and only one solution u within the interval C(u ), and { θ u(x) = inf E x, e αt f (x t )dt + e αθ Mu(x θ ), (31) θ θ admissible, i.e., θ = τ η for some discrete stopping time η 1. Furthermore, u n and u belong to D(A x,y ) C b (E [, [) and A x,y u(x, y) + αu(x, y) + λ(x, y) [ u(x, ) Mu(, )(x) ] + = f (x) (32) A x,y u n (x, y) + αu(x, y) + λ [ u n (x, ) Mu n 1 (, )(x) ] + = f (x) (33) JLM (WSU) Control Problems with Constraint 17 / 24

18 6.- Solving HJB Equation Existence and Uniqueness (Thm 3 & 4) Solution: Optimal Cost and Feedback (Thm 3 & 4) Theorem Under the previous assumptions, the unique solution of the HJB equation (23) is the optimal cost (18), i.e., u(x, y) = inf { J x,y (ν) : ν any admissible impulse control, (34) for every (x, y) in E [, [. Theorem Under previous assumptions, the first exit time of the continuation region [u < Mu] provides an optimal admissible impulse control. Extensions JLM (WSU) Control Problems with Constraint 18 / 24

19 6.- Extensions Other Impulse Control Problems Extensions - Other Impulse Control Problems Instead of Γ, may use variable Γ(x), Γ : E 2 E, with Γ(x) closed x E. Impulse intervention operator Mϕ(x) = inf { ϕ(x) + c(x, ξ) : ξ Γ(x), x E, should have the properties { (a) M maps continuously Cb (E) into itself, (b) there exists a Borel measurable minimizer for M, (35) where a minimizer means a Borel function ˆξ : E C b (E) E satisfying ˆξ(x, ϕ) Γ(x), Mϕ(x) = ϕ(x) + c(x, ˆξ(x, ϕ)), x, ϕ. Sometimes, the space C b (E) is replaced by another suitable space. JLM (WSU) Control Problems with Constraint 19 / 24

20 6.- Extensions Unbounded & Other Signals Unbounded & Other Signals Initially E compact, but E = R d, or in general, E locally compact f with polynomial growth, in C p (R d ), C p (H), H = Hilbert Signal with a sequence {T 1, T 2,... of independent identically distributed (IID conditionally to {x t : t ), or pure jump Markov processes, semi Markov processes, piecewise-deterministic Markov processes and diffusion processes with jumps. Essentially, it suffices to add a new variable to the initial process {x t : t to be reduced to the current situation. Instead of assuming y t in R +, we can consider the case y t in [, y [, for some finite y >, and therefore π([, y [) = 1 In the case of IID variables independent of {x t : t, we can consider a general distribution π, without a density. Hybrid Control, Stopping, Impulse, Switching JLM (WSU) Control Problems with Constraint 2 / 24

21 7. Hybrid Control Abstract Model Hybrid Abstract Model a lot of references, many models,... a book in preparation with Héctor (Jasso-Fuentes) & Maurice (Robin) time t in [, [, state variable (x, n) in S R d R m trajectories piecewise continuous in x, piecewise constant in n the continuous-type component x may include some discrete evolution (i.e., taking discrete values) needed to trigger a discrete transition (or event), produced when x hits the set-interface D n = {x : (x, n) D n is sometime called mode/regime, example of evolution equation: (ẋ(t), ṅ(t) ) = ( g(x(t), n(ti ), v(t)), ), for t [t i, t i+1 [, (x(t i ), n(t i )) = ( X (x(t i ), n(t i ), k i ), N(x(t i ), n(t i ), k i ) ), i =, 1, t i+1 T(x( ), n(t i ), t i, D) := inf { t t i : (x(t), n(t i )) D, if t i <. with initial condition (x(), n()) = (x, n ) S, t = JLM (WSU) Control Problems with Constraint 21 / 24

22 7. Hybrid Control Automaton Case Automaton Case Theorem Under the assumptions inf k K (X, N) : D K S D, uniformly continuous. g : S V R d, continuous g(x, n, v) C(1 + x ), x, n, v g(x, n, v) g(x, n, v) M x x, x, x, n, v inf { ξ X (x, n, k) + η N(x, n, k) c > (x,n),(ξ,η) D the trajectories are well defined. Stochastic Dynamics, SDEs, SPDEs, general Markov-Feller processes,... JLM (WSU) Control Problems with Constraint 22 / 24

23 7. Hybrid Control General Case General Case The data is as follows: state-space S R d R m, open or closed, minimal set-interface D S, closed, maximal set-interface D S, closed, D D, control-spaces V K R p R q, compact, set-constraints R V S V and R K S K, closed. The dynamics follows the rule: (1) a mandatory jump or switching is applied on D, (2) a continuous evolution takes place on S D, (3) the controller chooses either (1) or (2) on D D. JLM (WSU) Control Problems with Constraint 23 / 24

24 7. Hybrid Control Stopping/Impulse/Switching Stopping/Impulse/Switching with Constraint Particular cases: (x, y) S = E [, [, D =, D = E {, i.e., the control is allowed only when y = (the signal has arrived). In our Optimal Stopping Time Problem (with constraint) we have two costs, u and u related by u(x, y) = E x,y { e ατ 1 u (x τ1, ), (x, y) E [, [. (36) The cost u corresponds to the hybrid problem control only when y =, but the cost that we are interested in is u. Current Works (besides with M. Robin): in hybrid control (with Héctor Jasso-Fuentes) Relaxation and linear programs in hybrid control problems,... (with Héctor Jasso-Fuentes and Tomás Prieto-Rumeau) Discrete-Time Hybrid Control in Borel Spaces,... T H A N K S! JLM (WSU) Control Problems with Constraint 24 / 24