HJ equations. Reachability analysis. Optimal control problems

Transcription

1 HJ equations. Reachability analysis. Optimal control problems Hasnaa Zidani 1 1 ENSTA Paris-Tech & INRIA-Saclay Graz, 8-11 September 2014 H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

2 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

3 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

4 Some steady equations λv(x) + H(x, Dv(x)) = 0 x R d { H(x, Dv(x)) = 0 x Ω v(x) = 0 x Ω where Ω R d Time-dependant equations { t v(t, x) + H(t, x, Dv(t, x)) = 0 x R d, t > 0 v(0, x) = v o (x) x R d { min ( t v(t, x) + H(t, x, Dv(t, x)), v(t, x) g(t, x)) = 0 v(0, x) = v o (x) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

5 Hamiltonian H H : R d R d R is a Lipschitz continuous function In the sequel, we will consider the particular case where the Hamiltonian H is defined by H(x, p) := sup ( q p l(x, q)) q F (x) where F : R d R d is a multi-valued function, and l : R d R d R. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

6 Some examples Eikonal equation: { t v(t, x) + c(x) Dv(t, x) = 0 x R d, t > 0 v(0, x) = v o (x) x R d Here the velocity c( ) 0, and the Hamiltonian is defined by H(x, p) := c(x) p = max ( q p). q B(0,1) Advection equation with an obstacle: { min ( t v(t, x) a(x) Dv(t, x), v(t, x) g(t, x) ) = 0 v(0, x) = v o (x) x R d the Hamiltonian is defined by H(x, p) := a(x) p. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

7 Hamilton-Jacobi-Bellman (HJB for short) equations Notion of solutions for HJB equations Existence of solutions: Value function of optimal control problems Uniqueness results: Comparison principle H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

8 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

9 Definition. Let Ω R M, and F : Ω R R M R be continuous. Consider the HJB equation: F(x, u(x), Du(x)) = 0 in Ω. (1) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if for any ϕ C 1 (Ω) and any local maximum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 )) 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

10 Definition. Let Ω R M, and F : Ω R R M R be continuous. Consider the HJB equation: F(x, u(x), Du(x)) = 0 in Ω. (1) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if for any ϕ C 1 (Ω) and any local maximum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 )) 0. (ii) A function u LSC(Ω)) is a viscosity super-solution of (7) if for any ϕ C 1 (Ω) and any local minimum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 )) 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

11 Definition. Let Ω R M, and F : Ω R R M R be continuous. Consider the HJB equation: F(x, u(x), Du(x)) = 0 in Ω. (1) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if for any ϕ C 1 (Ω) and any local maximum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 )) 0. (ii) A function u LSC(Ω)) is a viscosity super-solution of (7) if for any ϕ C 1 (Ω) and any local minimum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 )) 0. (iii) A function u C(Ω) is a viscosity solution of (7) if it is a sub- and super-solution. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - Lectures 1& Graz, september / 107

12 Vanishing Viscosity Limits Lemma Let u ε be a sequence of smooth solutions to the viscous equations: F(x, u ɛ (x), Du ɛ (x)) ε u ɛ (x) = 0 x Ω. If u ε u uniformly on Ω, when ε 0+, then u is a viscosity solution of F(x, u(x), Du(x)) = 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

13 Idea of the proof - 1: u ε is a viscosity sub-solution to the viscous equation. Indeed, let ψ C 1 (Ω), and let x Ω a local maximum of u ε ψ. Then Du ε (x) = Dψ(x), u ε (x) ψ(x). Therefore, F(x, u ε (x), Dψ(x)) ε ψ(x) F(x, u ɛ (x), Du ɛ (x)) ε u ɛ (x) = 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

14 Idea of the proof - 1: u ε is a viscosity sub-solution to the viscous equation. Indeed, let ψ C 1 (Ω), and let x Ω a local maximum of u ε ψ. Then Du ε (x) = Dψ(x), u ε (x) ψ(x). Therefore, F(x, u ε (x), Dψ(x)) ε ψ(x) 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

15 Idea of the proof - 1: u ε is a viscosity sub-solution to the viscous equation. Indeed, let ψ C 1 (Ω), and let x Ω a local maximum of u ε ψ. Then Du ε (x) = Dψ(x), u ε (x) ψ(x). Therefore, F(x, u ε (x), Dψ(x)) ε ψ(x) 0. Let ϕ C 1 (Ω), x Ω is a local maximum of u ϕ. ρ > 0, δ ρ and there exists ψ C 1 (Ω) s.t. ψ ϕ C 1 (Ω) δ u ε ψ has a local maximum x ε inside the ball B(x, ρ). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

16 Idea of the proof - 2: Extract a convergent subsequence x ε x B(x, ρ). By passing to the limit in the inequality F(x ε, u ε (x ε ), Dψ(x ε )) ε ψ(x ε ) 0, we get: F( x, u( x), Dψ( x)) 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

17 Idea of the proof - 2: Extract a convergent subsequence x ε x B(x, ρ). By passing to the limit in the inequality F(x ε, u ε (x ε ), Dψ(x ε )) ε ψ(x ε ) 0, we get: F( x, u( x), Dψ( x)) 0. Choosing ρ small enough, we have x x and Dψ( x) Dϕ(x) as small as we want. So by continuity of F, we finally obtain: F(x, u(x), Dϕ(x)) 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

18 Idea of the proof - 2: Extract a convergent subsequence x ε x B(x, ρ). By passing to the limit in the inequality F(x ε, u ε (x ε ), Dψ(x ε )) ε ψ(x ε ) 0, we get: F( x, u( x), Dψ( x)) 0. Choosing ρ small enough, we have x x and Dψ( x) Dϕ(x) as small as we want. So by continuity of F, we finally obtain: F(x, u(x), Dϕ(x)) 0. The supersolution can be proved in the same way. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

19 A more general stability result A general stability theorem holds for viscosity solutions, without additional requirements on the derivatives. Theorem Consider a sequence of continuous functions (u n ) n 1, where u n is a viscosity sub- solutions (resp. super-solutions) to F n (x, u n (x), Du n (x)) = 0, x Ω. Assume F n F uniformly on compact subsets of Ω R d R d, when n + and u n u uniformly on Ω. Then u is a viscosity sub-solution (a super-solution) of F(x, u(x), Du(x)) = 0, x Ω. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

20 The stability results are based on the following "key ingredient": Lemma Let v : Ω R be a upper semi-continuous function that achieves a strict local maximum at x Ω. Let (v n ) n be a sequence of upper semi-continuous function on Ω. Assume that lim sup v n (z) = v(x 0 ). Then there exists a sequence (x n ) n z x n + in Ω such that for every n 1, x n is a local maximum of v n and lim x n = x, n + lim v(x n) = v( x). n + H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

21 Semi-relaxed limits Let (u n ) n be a sequence of functions on Ω. Assume that (u n ) n are locally uniformly bounded from above Define the upper relaxed limit by: u (x) := lim sup u n (x n ). x n x Assume that (u n ) n are locally uniformly bounded from below. Define the lower relaxed limit by: u (x) := lim inf x n x u n(x n )). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

22 Theorem (Stability by semi-relaxed limits) Consider a sequence of usc (resp. lsc) functions (u n ) n 1, where u n is a viscosity sub- solutions (resp. super-solutions) to F n (x, u n (x), Du n (x)) = 0, x Ω. Assume F n F uniformly on compact subsets of Ω R d R d, when n + and (u n ) n is locally uniformly bounded from above (resp. from below) on Ω. Then u is a viscosity sub-solution (resp u is a super-solution) of F(x, u(x), Du(x)) = 0, x Ω. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

23 Viscosity notion: An equivalent definition. Let Ω R M, and F : Ω R R M R be continuous. Consider the HJB equation: F(x, u(x), Du(x)) = 0 in Ω, (2) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if F(x, u(x), q) 0 q D + u(x). q D + u(x) u(y) u(x) + q (y x) + o( y x ), q D u(x) u(y) u(x) + q (y x) + o( y x ). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

24 Viscosity notion: An equivalent definition. Let Ω R M, and F : Ω R R M R be continuous. Consider the HJB equation: F(x, u(x), Du(x)) = 0 in Ω, (2) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if F(x, u(x), q) 0 q D + u(x). (ii) A function u LSC(Ω)) is a viscosity super-solution of (7) if F(x, u(x), q) 0 q D u(x). q D + u(x) u(y) u(x) + q (y x) + o( y x ), q D u(x) u(y) u(x) + q (y x) + o( y x ). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

25 Viscosity notion: An equivalent definition. Let Ω R M, and F : Ω R R M R be continuous. Consider the HJB equation: F(x, u(x), Du(x)) = 0 in Ω, (2) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if F(x, u(x), q) 0 q D + u(x). (ii) A function u LSC(Ω)) is a viscosity super-solution of (7) if F(x, u(x), q) 0 q D u(x). (iii) A function u C(Ω) is a viscosity solution of (7) if it is a sub- and super-solution. q D + u(x) u(y) u(x) + q (y x) + o( y x ), q D u(x) u(y) u(x) + q (y x) + o( y x ). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

26 Lemma Suppose u C(Ω). (i) q D + u(x) if and only if there exists ϕ C 1 (Ω) such that Dϕ(x) = q and x is a local minimum of u ϕ. (ii) q D u(x) if and only if there exists ϕ C 1 (Ω) such that Dϕ(x) = q and x is a local maximum of u ϕ. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

27 Some remarks If u is differentiable at x Ω, then D u(x) = D + u(x) = { u(x)}. A viscosity solution u of the HJ equation satisfies the equality F(x, u(x), u(x)) = 0 in the classical sense whenever u is differentiable at x. A Lipschitz continuous viscosity solution u is also a solution in the "almost everywhere" sense. (Indeed, by Rademacher s theorem, a Lipschitz continuous function is differentiable a.e.) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

28 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

29 Controlled Systems { ẏx (s) F(y x (s)), s (0, 1), y x (0) = x, (H1) F : R d R d is a usc set-valued function x 0 R d, ε > 0, η > 0, x x 0 η = F(x) F(x 0 ) + εb. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

30 Controlled Systems { ẏx (s) F(y x (s)), s (0, 1), y x (0) = x, (H1) F : R d R d is a usc set-valued function, with convex compact nonempty images, and F has a linear growth k > 0, x R d, sup v k(1 + x ). v F (x) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

31 Controlled Systems { ẏx (s) F(y x (s)), s (0, 1), y x (0) = x, (H1) F : R d R d is a usc set-valued function, with convex compact nonempty images, and F has a linear growth (H2) F : R d R d is Lipschitz continuous C > 0, s.t. F(y 2 ) F(y 1 ) + C y 2 y 1 B, y 1, y 2 R d H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

32 Controlled Systems { ẏx (s) F(y x (s)), s (0, 1), y x (0) = x, (H1) F : R d R d is a usc set-valued function, with convex compact nonempty images, and F has a linear growth (H2) F : R d R d is Lipschitz continuous Example: F(x) := f (x, U); where U R m compact, f : R d U R d is Lipschitz, bounded, and f (x, U) is convex for any x R d. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

33 Consider the set of admissible trajectories S [0,τ] (x) := {y x ẏ x (s) F (y x (s)) on(0, τ), y x (0) = x} Under (H1), S [0,τ] (x) is a compact subset of C([0, τ]). Under (H1)-(H2), the set-valued function x S [0,τ] (x) is Lipschitz continuous, L > 0, S [0,τ] (x) S [0,τ] (z) + L x z B W 1,1 x, z R d. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

34 Reachable (or Attainable) set The reachable set R(x; t) from x at time t is the set of all points of the form y x (τ), where y x S [0,t] (x): R(x; t) := {y x (τ) y x S [0,t] (x), τ [0, t]}. The reachable set from X is defined by: R X (t) := x X R(x; t). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

35 Some properties of the reachable sets If the set-valued F is usc with non-empty compact images (not necessarily convex images) then R C (t) is not necessarily closed. Assume (H1), then for every t 0, R C (t) is a closed set. In all the sequel, we will assume that F satisfies at least (H1). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

36 Let C be a closed target set (in our examples, C is safe) Capture Basin (or Backward reachable set) The Capture Basin C t C, at time t, is the set of all initial positions x from which a trajectory y x S [0,t] (x) can reach the target C. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

37 Let C be a closed target set (in our examples, C is safe) Capture Basin (or Backward reachable set) The Capture Basin R C t, before time t, is the set of all initial positions x from which a trajectory y x S [0,t] (x) can reach the target C before t. { } R C t := x R d, τ [0, t], y x S [0,τ] (x), y x (τ) C H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

38 Optimization-based controller design Minimum-time control problem: T (x) = inf { t; y x (t) C, y x S [0,t] (x) }. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

39 Optimization-based controller design Minimum-time control problem: T (x) = inf { t; y x (t) C, y x S [0,t] (x) }. The sublevels of the minimum function T correspond to the Capture Basins of the target C: C C t = {x R d T (x) = t} (3) R C t = {x R d T (x) t}. (4) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

40 Zermelo problem H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

41 Distance function. Level-set appraoch Consider the function Φ(x) = d C (x). A Mayer s problem: V (x, t) = inf Φ(y x(t)) y x S [0,t] (x) A Mayer s problem with extended trajectories V (x, t) = inf Φ(y x (t)) y x Ŝ[0,t](x) where Ŝ[0,t](x) is the set of trajectories satisfying: with F(x) = [0, 1] F(x). ẏ F(y(t)), y(0) = x, H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

42 Level set approach V and V are Lipschitz continuous functions H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

43 Level set approach V and V are Lipschitz continuous functions For every t 0, R C t = {x Rd ; V (x, t) 0}; R C t = {x R d ; V (x, t) 0} H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

44 Level set approach V and V are Lipschitz continuous functions For every t 0, R C t = {x Rd ; V (x, t) 0}; R C t = {x R d ; V (x, t) 0} The minimum time function T : R d R + {+ } is lsc. Moreover, we have: T (x) = inf{t 0; V (x, t) 0} = min{t 0; V (x, t) 0}. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

45 Level set approach V and V are Lipschitz continuous functions For every t 0, R C t = {x Rd ; V (x, t) 0}; R C t = {x R d ; V (x, t) 0} The minimum time function T : R d R + {+ } is lsc. Moreover, we have: T (x) = inf{t 0; V (x, t) 0} = min{t 0; V (x, t) 0}. Φ can be any function satisfying Φ(x) 0 x C. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

46 Van der Pol Problem : ẏ 1 (t) = y 2 ẏ 2 (t) = y 1 + y 2 (1 y1 2) + a(t) a(t) [ 1, 1] Φ(y) = d C (x) x x1 H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

47 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

48 Now, consider the following control problems: Mayer s problem: V (x, t) = inf Φ(y x(t)) y x S [0,t] (x) Time minimum problem (C closed set in R d ): T (x) = inf { t; y x (t) C, y x S [0,t] (x) } Supremum cost: V (x, t) = inf Φ(y x(t)) sup g(y x (θ)) y x S [0,t] (x) θ [0,t] H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

49 Assume (H1)-(H2) hold, and g : R d R is Lipschitz continuous. If Φ : R d R is lsc (resp. Lipschitz), then V and V are lsc (resp. Lipschitz). When the target C is closed, the minimum time function T is lsc. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

50 Mayer Problem V (x, t) = min y x S [0,h] (x) V (y x(h), t h) V (x, 0) = Φ(x) h (0, t), H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

51 Mayer Problem V (x, t) = min y x S [0,h] (x) V (y x(h), t h) V (x, 0) = Φ(x) h (0, t), Minimum time problem: T (x) = min y x S [0,h] (x) T (y x(h)) + h h < T (x), x C, T (x) = 0 x C; H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

52 Mayer Problem V (x, t) = min y x S [0,h] (x) V (y x(h), t h) V (x, 0) = Φ(x) h (0, t), Minimum time problem: T (x) = min y x S [0,h] (x) T (y x(h)) + h h < T (x), x C, T (x) = 0 x C; Supremum cost V (x, t) = min V (y x (h), t h) sup g(y x (θ)) y x S [0,h] (x) θ [0,h] V (x, 0) = Φ(x) g(x); H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

53 V (x, t) = min Φ(y x(t)) y x S [0,t] (x) V (x, t) = min y x S [0,h] (x) V (y x(h), t h) V (x, 0) = Φ(x) h (0, t). Suboptimality: y x S [0,t] (x), s V (y x (s), t s) is increasing, Superoptimality y x S [0,t] (x), s V (y x (s), t s) is constant H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

54 Next, we derive the Hamilton-Jacobi-Bellman equation (HJB), which is an infinitesimal version of the DPP. If the value function is C 1 in a neighborhood of (x, t), then t V (x, t) + H(x, D x V (x, t)) = 0, x R d, t > 0; H(x, DT (x)) = 1, x C, T (x) < + ; min( t V (x, t) + H(x, DV (x, t)), V (x, t) g(x)) = 0, x R d, t > 0; where H(x, q) := max p F (x) ( p.q). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

55 Next, we derive the Hamilton-Jacobi-Bellman equation (HJB), which is an infinitesimal version of the DPP. If the value function is C 1 in a neighborhood of (x, t), then t V (x, t) + H(x, D x V (x, t)) = 0, x R d, t > 0; V (x, 0) = Φ(x) Time-dependent HJB equation H(x, DT (x)) = 1, T (x) = 0 on C x C, T (x) < + ; Steady HJB equation min( t V (x, t) + H(x, DV (x, t)), V (x, t) g(x)) = 0, V (x, 0) = Φ(x) g(x) HJB-VI ineqation where H(x, q) := max p F (x) ( p.q). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

56 Let u C 1 b (Rd [0, T ]). We say that u is a classical verification function if it satisfies t u(x, t) + H(x, D x u(x, t)) 0 x R d, t [0, T ], u(x, 0) = Φ(x); where H(x, q) = max p F (x) ( p q). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

57 Let u C 1 b (Rd [0, T ]). We say that u is a classical verification function if it satisfies t u(x, t) + H(x, D x u(x, t)) 0 x R d, t [0, T ], u(x, 0) = Φ(x); Proposition Let y x S [0,T ] (x) be a feasible arc. Assume that there exists u a classical verification function, s.t. then y x is optimal for x on [0, T ]. u(x, T ) = Φ(y x (T )), H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

58 Proof. We take any y x S [0,T ] (x), we have: d ds [u(y x(s), T s)] = t u(y x (s), T s) + ẏ x (s) Du(y x (s), T s) 0. Which means that s u(y x (s), T s) is increasing: Therefore, u(x, T ) V (x, T ). u(x, T ) = Φ(y x (T )) V (x, T ). u(x, T ) u(y x (T ), 0) = Φ(y x (T )). Therefore u(x, T ) = Φ(y x (T )) = V (x, T ), and y x is optimal. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

59 Assume F = f (x, A), A is a compact set in R m. Proposition Let y x S [0,T ] (x) be a feasible arc, and let a ( ) A be the corresponding control strategy: ẏ x (t) = f (y x (t), a (t)) a.e in [0, T ]. Assume that there exists u a classical verification function, s.t. t u(y x (t), t) f (y x (t), a (t)).du(y x (t), t) = 0 for a.e t [0, T ], then y x is optimal for x on [0, T ]. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

60 Proof. We know that s u(y x (s), T s) is increasing: Therefore, u(x, T ) V (x, T ). u(x, T ) u(y x (T ), 0) = Φ(y x (T )). u(y x (s), T s) = u(x, T ) = Cst = u(y x (T ), 0) = Φ(y x (T )) V (x, T ). Therefore u(x, T ) = Φ(y x (T )) = V (x, T ), and y x is optimal. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

61 If we take the value function V itself as a verification function (i.e, if V is smooth), then a necessary and sufficient condition of optimality is: f (y x (t), a (t)) DV (y x (t), t) = max a A ( f (y x (t), a) DV (y x (t), t)) for a.e 0 < t < T. = H(y x (t), DV (y x (t), t)) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

62 Under the (irrealistic!) assumption that V is smooth, we consider the multivalued function: Ψ(x, t) := argmax a A ( f (x, a) DV (x, t)). Let y x S [0,t] (x). y x is optimal for x on [0, T ] if and only if y x (0) = x, and ẏ x (t) f (y x (t), Ψ(y x (t), t)) t (0, T ). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

63 Van der Pol Problem : ẏ 1 (t) = y 2 ẏ 2 (t) = y 1 + y 2 (1 y1 2) + a(t) a(t) [ 1, 1] x2 0 Φ(y) = y r x1 H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

64 The value function satisfies a Hamilton-Jacobi-Bellman equation (HJB) in a viscosity sense. t V (x, t) + H(x, D x V (x, t)) = 0, x R d, t > 0; H(x, DT (x)) = 1, x C, T (x) < + ; min( t V (x, t) + H(x, DV (x, t)), V (x, t) g(x)) = 0, x R d, t > 0; where H(x, q) := max p F (x) ( p.q). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

65 The value function satisfies a Hamilton-Jacobi-Bellman equation (HJB) in a viscosity sense. t V (x, t) + H(x, D x V (x, t)) = 0, x R d, t > 0; V (x, 0) = Φ(x) Time-dependent HJB equation H(x, DT (x)) = 1, T (x) = 0 on C x C, T (x) < + ; Steady HJB equation min( t V (x, t) + H(x, DV (x, t)), V (x, t) g(x)) = 0, V (x, 0) = Φ(x) g(x) HJB-VI ineqation where H(x, q) := max p F (x) ( p.q). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

66 Theorem Assume that the value function is continuous. Then it is a viscosity solution of the corresponding HJB equation. Proof for the Minimum time problem. Let ϕ C 1 is such that T ϕ has a local maximum at x C, then for any y x S [0,s] (x) we have: for small s. Therefore, ϕ(x) ϕ(y x (s)) T (x) T (y x (s)) s sup ( p Dϕ(x)) = H(x, Dϕ) 1, p F (x) so T is a subsolution. With similar arguments, we prove that T is a supersolution as well. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

67 The boundedness assumption on V and V is not restrictive. It is verified whenever Φ and g are bounded. The continuity of V and V holds whenever Φ and g are continuous. The uniqueness results for the HJB equation associated to the minimal time problem is not a trivial task. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

68 Assume C is a closure of smooth domain, and let η x be the normal to C. If min p η x < 0, x C, p F (x) then the minimal time function T is continuous. The continuity of T requires a controllability assumption of the system around the target. This important property is not satisfied in several examples, such as the Zermelo problem. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

69 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

70 { t V (x, t) + H(x, DV (x, t)) = 0 x R d, t > 0 V (x, 0) = Φ(x) x R d where H(x, p) = max ( q p). q F (x) A viscosity solution is given by the value function: V (x, t) = min Φ(y x(t)). y x S [0,t] (x) Regularity and growth properties of V can be derived directly from the definition of V. Moreover, V satisfies: Sub-optimality: V (x, t) V (y x (h), t h), y x S [0,h] (x). Super-optimality: y x S [0,t] (x), V (x, t) = V (y x (h), t h). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

71 Comparison Principle: A geometric argument (nonsmooth analysis) Proposition Let u : R d [0, T ] R. (i) Assume u is usc. Then u is a viscosity sub-solution iff u satisfies the sub-optimality principle. (ii) Assume u is lsc. Then, u is a viscosity super-solution iff u satisfies the super-optimality principle. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

72 Idea of the proof Let U : (0, T ) R a usc viscosity sub-solution of: U (t) 0 t (0, T ), then U(t) U(t s) for every s [0, t], for every 0 t T. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

73 Idea of the proof Let U : (0, T ) R a usc viscosity sub-solution of: U (t) 0 t (0, T ), then U(t) U(t s) for every s [0, t], for every 0 t T. For every p D + U(t), we have p 0 H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

74 Idea of the proof Let U : (0, T ) R a usc viscosity sub-solution of: U (t) 0 t (0, T ), then U(t) U(t s) for every s [0, t], for every 0 t T. For every p D + U(t), we have p 0 Since u is usc: p D + U(t) (p, 1) N Hp(U) (t, U(t)). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

75 Idea of the proof Let U : (0, T ) R a usc viscosity sub-solution of: U (t) 0 t (0, T ), then U(t) U(t s) for every s [0, t], for every 0 t T. For every p D + U(t), we have p 0 Since u is usc: p D + U(t) (p, 1) N Hp(U) (t, U(t)). We conclude (by Rockafellar s Horizontal theorem) that: ( ) 1 (p, q) 0, (p, q) N 0 Hp(U) (t, U(t)). (5) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

76 Idea of the proof Let U : (0, T ) R a usc viscosity sub-solution of: U (t) 0 t (0, T ), then U(t) U(t s) for every s [0, t], for every 0 t T. For every p D + U(t), we have p 0 Since u is usc: p D + U(t) (p, 1) N Hp(U) (t, U(t)). We conclude (by Rockafellar s Horizontal theorem) that: ( ) 1 (p, q) 0, (p, q) N 0 Hp(U) (t, U(t)). (5) For a given t > 0, consider the ODE: ( ) 1 Ż (s) = s (0, t), Z (0) = 0 ( ) t. U(t) By (5) and the invariance principle, we conclude that Z (s) Hp(U) s (0, t). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

77 Comparison principle: Doubling variable techniques (Kruzhkov s method for conservation laws) Theorem Let Ω be a bounded open set of R d. Let u 1, u 2 BUC(Ω) be respectively, viscosity sub- and super-solutions of u + H(x, Du(x)) = 0 on Ω, and u 1 u 2 on Ω. Moreover, assume that H : R d R d R is uniformly continuous w.r.t the x-variable H(x 1, p) H(x 2, p) ω ( x 1 x 2 (1 + p )), where ω : R + R + is a continous non-deacring function satisfying ω(0) = 0. Then u 1 (x) u 2 (x) for every x Ω. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

78 Proof - 1: Case where u 1, u 2 are smooth (C 1 functions) Assume u 1 u 2 attains a maximum at x 0 Ω. Then p := Du 1 (x 0 ) = Du 2 (x 0 ) and: u 1 (x 0 ) + H(x 0, p) 0 u 2 (x 0 ) + H(x 0, p). We conclude that u 1 (x 0 ) u 2 (x 0 ). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

79 Proof - 2: the general case Assume u 1 u 2 attains a positive maximum at x 0 Ω. It may happen that D + u 1 (x 0 ) D u 2 (x 0 ) =. In this case, the HJB equation doesn t give any information at x 0. The doubling variables idea consists of building two sequences (x ɛ ), (y ɛ ) in a neighbourhood of x 0, such that D + u 1 (x ɛ ) D u 2 (y ɛ ) a p ɛ D + u 1 (x ɛ ) p ɛ D u 2 (y ɛ ) x ɛ y ɛ p ɛ D + u 1 (x ɛ ) D u 2 (y ɛ ) u 1 u b H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

80 Proof - 2: the general case Assume u 1 u 2 attains a positive maximum at x 0 Ω. Set δ := u 1 (x 0 ) u 2 (x 0 ) > 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

81 Proof - 2: the general case Assume u 1 u 2 attains a positive maximum at x 0 Ω. Set δ := u 1 (x 0 ) u 2 (x 0 ) > 0. Define the function of two variables: Ψ ε (x, y) := u 1 (x) u 2 (y) x y 2. 2ε H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

82 Proof - 2: the general case Assume u 1 u 2 attains a positive maximum at x 0 Ω. Set δ := u 1 (x 0 ) u 2 (x 0 ) > 0. Define the function of two variables: Ψ ε (x, y) := u 1 (x) u 2 (y) x y 2. 2ε Ψ attains its global maximum (x ε, y ε ) on Ω Ω, and we have: Ψ(x ε, y ε ) Ψ(x 0, x 0 ) = Ψ(x ε, y ε ) δ > 0. (6) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

83 Let M > 0 s.t u 1 + u 2 M. We get: 0 < Ψ(x ε, y ε ) M x ε y ε 2, 2ε wich implies that: x ε y ε 2Mε. Moreover, x ε y ε ε 0, when ε 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

84 Let M > 0 s.t u 1 + u 2 M. We get: 0 < Ψ(x ε, y ε ) M x ε y ε 2, 2ε wich implies that: x ε y ε 2Mε. Moreover, x ε y ε ε 0, when ε 0. Since u 2 BUC(Ω), there exists ε > 0 s.t.: u 2 (x) u 2 (y) δ/2 whenever x y 2Mε. For every ε < ε, we have: x ε, y ε Ω. Indeed, if x ε Ω, then we would have: Ψ(x ε, y ε ) (u 1 (x ε ) u 2 (x ε )) + u 2 (x ε ) u 2 (y ε ) x ε y ε δ/2 + 0 in contradiction with (6). 2ε H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

85 We can check that p ε := x ε y ε ε D + u 1 (x ε ) D u 2 (y ε ) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

86 We can check that p ε := x ε y ε ε From the definition of u 1, u 2, we obtain: D + u 1 (x ε ) D u 2 (y ε ) u 1 (x ε ) + H(x ε, p ε ) 0 u 2 (y ε ) + H(y ε, p ε ). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

87 We can check that p ε := x ε y ε ε From the definition of u 1, u 2, we obtain: D + u 1 (x ε ) D u 2 (y ε ) u 1 (x ε ) + H(x ε, p ε ) 0 u 2 (y ε ) + H(y ε, p ε ). Finally, by using the assumption on H, it comes: δ Ψ(x ε, y ε ) u 1 (x ε ) u 2 (y ε ) ω( x ε y ε (1 + p ε ) ). This yields a contradiction when ε goes to 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

88 Time dependent case. Theorem Let H : R d R d R is Lipschitz continuous: H(x, p) H(y, p) C( x y (1 + p )), H(x, p 1 ) H(x, p 2 ) C p 1 p 2. Let u 1, u 2 are resp., a sub- and super-solution of t u + H(x, Du) = 0 x R d, t (0, T ] Assume that u 1 (x, 0) u 2 (x, 0), then u 1 u 2 on R d [0, T ]. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

89 Idea of the proof. A new difficulty: the domain is unbounded. For this, we define the doubling-variable function as: Ψ ε (x, t, y, s) := u 1 (x, t) u 2 (y, s) ε( x 2 + y 2 ) ( t s 2 + x y 2 ) 2ε 2. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

90 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

91 { t V (x, t) + H(x, DV (x, t), D 2 V (x, t)) = 0 x R d, t > 0 V (x, 0) = Φ(x) x R d where H(x, p, Q) = max a A ( l(x, a) b(x, a) p 1 2 Tr(σ(x, a)σt (x, a) Q)). σ( ) is a mapping R d A into the space of n r matrices, b : R d A R d. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

92 Definition. Let Ω R m, and F : Ω R R m S m m R be continuous and satisfying F(x, r, p, X) F(x, r, p, Y ) when X Y. Consider the HJB equation: F(x, u(x), Du(x), D 2 u(x)) = 0 in Ω. (7) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if for any ϕ C 2 (Ω) and any local maximum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 ), D 2 ϕ(x 0 )) 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

93 Definition. Let Ω R m, and F : Ω R R m S m m R be continuous and satisfying F(x, r, p, X) F(x, r, p, Y ) when X Y. Consider the HJB equation: F(x, u(x), Du(x), D 2 u(x)) = 0 in Ω. (7) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if for any ϕ C 2 (Ω) and any local maximum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 ), D 2 ϕ(x 0 )) 0. (ii) A function u LSC(Ω)) is a viscosity super-solution of (7) if for any ϕ C 2 (Ω) and any local minimum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 ), D 2 ϕ(x 0 )) 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

94 Definition. Let Ω R m, and F : Ω R R m S m m R be continuous and satisfying F(x, r, p, X) F(x, r, p, Y ) when X Y. Consider the HJB equation: F(x, u(x), Du(x), D 2 u(x)) = 0 in Ω. (7) (i) A function u USC(Ω) is a viscosity sub-solution of (7) if for any ϕ C 2 (Ω) and any local maximum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 ), D 2 ϕ(x 0 )) 0. (ii) A function u LSC(Ω)) is a viscosity super-solution of (7) if for any ϕ C 2 (Ω) and any local minimum point x 0 Ω of u ϕ, F(x 0, u(x 0 ), Dϕ(x 0 ), D 2 ϕ(x 0 )) 0. (iii) A function u C(Ω) is a viscosity solution of (7) if it is a sub- and super-solution. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

95 The notion of viscosity can also be defined by an adequate concept of "super-" and "sub-" derivatives (of order 2), called "semi-jets". The viscosity solution is the limit of classical solutions of viscous equations. As for first order equations, a general stability property holds, without any requirement about the convergence of derivatives. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

96 TIme-dependent HJ equation The associated control problem: [ V (x, t) = Min E Φ(y(T )) + V satisfies a DPP: V (x, t) = min α(]0,h[) A E V (x, 0) = Φ(x). t 0 ] l(y(s), α(s)) ds ; dy(s) = b(y(s), α(s))ds + σ(y(s), α(s))dw s, y(0) = x, α(s) A, s [0, t]. [ V (y α x (h), t h) + h 0 l(y α x (s), α(s)) ds A general comparison principle can be stated, as for first order equations, under Lipschitz continuity of the Hamiltonian. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107 ],

97 H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107 J. Yong and X.Y. Zhou, Stochastic Controls. Hamiltonian Systems and References on viscosity notion for HJ equations G. Barles, Solutions de Viscosité des Equations de Hamilton-Jacobi, Springer, M. Bardi and I. Capuzzo Dolcetta, Optimal Control and Viscosity Solutions of Hamilton- Jacobi-Bellman Equations, Birkhäuser, M. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobi equations, Trans. Amer. Math. Soc., 277(1983), M. Crandall, H. Ishii and P. L. Lions, User s guide to viscosity solutions of second order partial differential equations, Bull. Amer. Math. Soc., 27(1992), W.H. Fleming and H.M. Soner, Controlled Markov Processes and Viscosity Solutions. Springer-Verlag, New-York, S. N. Kruskov, First order quasilinear equations in several independant variables. Math. USSR-Sbornik 10(1970), P.L. Lions, Generalized solution of Hamilton-Jacobi equations, Pitman, London, 1982.

98 Extensions and active fields The comparison principle has been extended to unbounded viscosity solutions, with polynomial or exponential growth (ref: Ishii 92) Boundary conditions can be also considered in the viscosity framework (ref: Barles, Capuzzo-Dolceta, Lions, Ishii,... ) Some existence and uniqueness results have been extended to Hamilton-Jacobi equations with L p data (ref: Cafarrelli, Cardaliaguet, Souganidis,... ) Hamilton-Jacobi equations in infinite dimension setting have been investigated by (ref: Barbu, Cannarsa, Gozzi, Soner,... ) Homogenization and Ergodic control problems (ref: Alvarez, Bardi, Achdou, Cardaliaguet, Siconolfi, Souganidis... )... H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

99 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

100 A junction condition on O x J x J x branch J i x J N 8 >< >: u i t + H i (u i x)=0, x > 0, i =1,...,N u i = u j =: u, x =0, u t + F (u 1 x,...,u N x )=0, x =0 H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

101 Junction conditions on multi-domains R d is divided into several open disjoint subdomains (Ω i ) i=1,...,m with R d = Ω 1 Ω m. y Ω 2 Γ 1 Ω 1 Γ 2 Γ 4 0 Γ 5 x Ω 3 Ω 4 Γ 3 H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

102 Junction conditions on multi-domains R d is divided into several open disjoint subdomains (Ω i ) i=1,...,m with R d = Ω 1 Ω m. y Ω 2 Γ 1 Ω 1 Γ 2 Γ 0 Γ 4 5 x Ω 3 Γ Ω 3 4 t u i (t, x) + H i (x, D x u i (t, x)) = 0 t (0, T ), x Ω i, u i (0, x) = ϕ(x) x R d, H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

103 Junction conditions on multi-domains R d is divided into several open disjoint subdomains (Ω i ) i=1,...,m with R d = Ω 1 Ω m. y Ω 2 Γ 1 Ω 1 Γ 2 Γ 0 Γ 4 5 x Ω 3 Γ Ω 3 4 t u i (t, x) + H i (x, D x u i (t, x)) = 0 t (0, T ), x Ω i, u i (0, x) = ϕ(x) x R d, u = u 1,..., u m t u(t, x) + F(x, D x u 1 (t, x),..., D x u m (t, x)) = 0 on j Γ j H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

104 Motivation & Position of the problem data transmission and traffic management (Garavello-Piccoli 2006, Engel et al. 08, Lebacque 05 blood circulation supply chains (Göttlich, Herty, Klar 2005) gas pipelines multiprocess systems without transition cost multiple thermostatic hybrid systems (Bagagiolo et al. 2012) Numerical purpose: decomposition of domains H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

105 Position of the problem R d = ( m i=1 Ω i ) ( l j=1 Γ j ), y Ω 2 Γ 1 Ω 1 Γ 2 Γ 0 Γ 4 5 x Ω 3 Γ Ω 3 4 H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

106 For each i = 1,..., m, consider the system of HJB equations: { t u i (t, x) + H i (x, D x u i (t, x)) = 0 t (0, T ), x Ω i, u(0, x) = ϕ(x) x R d, (8) where T > 0 is a fixed final time and ϕ is a given regular function. Each H i is a Lipschitz continuous Hamiltonian of Bellman form: H i (x, p) := sup { v p L i (x, v)}. v F i (x) where F i is a Lipscitz set-valued function on Ω i. Junction conditions Question: what conditions of transition between the subdomains Ω i should be considered to ensure the existence and uniqueness of solution u for the system (8)? H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

107 For each i = 1,..., m, consider the system of HJB equations: { t u i (t, x) + H i (x, D x u i (t, x)) = 0 t (0, T ), x Ω i, u(0, x) = ϕ(x) x R d, (8) where T > 0 is a fixed final time and ϕ is a given regular function. Each H i is a Lipschitz continuous Hamiltonian of Bellman form: H i (x, p) := sup { v p }. v F i (x) where F i is a Lipscitz set-valued function on Ω i. For simplicity, we shall take L 0. Junction conditions Question: what conditions of transition between the subdomains Ω i should be considered to ensure the existence and uniqueness of solution u for the system (8)? H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

108 Outline 1 Introduction 2 The viscosity Notion 3 Controlled systems. Optimal control problems 4 Value functions. HJB equations 5 Comparison principle 6 Parabolic HJB equations. Stochastic control problems 7 HJ equation on multidomains. Transmission conditions 8 Link between HJB equations and optimal control problems H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

109 Consider the equation t u(t, x) + H F (x, D x u(t, x)) = 0, u(0, x) = ϕ(x), x Ω = R d where H F (x, q) = sup p F (x) { p q}. Definition: viscosity solution A continuous function u is subsolution if for every Φ C 1 ((0, T ) Ω) such that u Φ (resp. u Φ) with equality at (t o, x o ) Ω, then: t Φ(t o, x o ) + H F (x o, D x Φ(t o, x o )) 0 (resp. 0). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

110 Mayer s optimal control problems Assume: R d = Ω Let ϕ : R d R be a regular function and F : R d R d be a Lipschitz continuous multi-function with closed convex images. For an initial data (t, x) [0, T ] R d, consider the differential inclusion: { ẏ(s) F(y(s)) s (0, t), y(0) = x. The set S [0,t] (x) of all trajectories satisfying (9) is closed in W 1,1 (0, t). And the value function v : [0, T ] R d R is defined by v(t, x) := inf{ϕ(y(t)) : y( ) satisfies (9)}. (9) H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

111 Classical Dynamical Programming Principle For any t [0, T ], x R d, h [0, t], we have More precisely, v(t, x) = min {v(t h, y(h))}. y S [0,t] (x) The super-optimality: There exists an optimal trajectory ȳ S [0,t] (x) such that v(t, x) (=) v(t h, ȳ(h)); The sub-optimality: For any trajectory y S [0,t] (x), we have: v(t, x) v(t h, y(h)). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

112 Characterization In the classical case where F is Lipschitz continuous, consider the Hamiltonian H F (x, p) = sup { q p}. q F (x) For any Lipschitz continuous function u : [0, T ] R d R, we have u satisfies the super-optimality u satisfies t u(t, x) + H F (x, Du(t, x)) 0; u satisfies the sub-optimality u satisfies t u(t, x) + H F (x, Du(t, x)) 0. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

113 Classical case The value function v defined by: v(t, x) := is the unique viscosity solution of min ϕ(y(t)), y S [0,t] (x) t v(t, x) + H F (x, Dv(t, x)) = 0 v(0, x) = ϕ(x). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

114 Consider the cross structure in R 2 t u(t, x) + H i (x, Du(t, x)) = 0 u(0, x) = ϕ(x), (0, T ) Ω i with H i (x, p) = sup v Fi (x)( v p). Idea: introduce a global optimal control problem and investigate the equation satisfied by the value function on the interfaces. Main difficulty: How to define the dynamics on the interfaces which allow that the trajectories can either transit between the domains or stay on the interfaces? A global Lipschitz continuous Hamiltonian could not be expected due to the singularity of the domain. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

115 Literature on non-lipschitz Hamiltonians Hamilton-Jacobi equations with discontinuous Hamiltonians: Ishii 89. Hamilton-Jacobi-Bellman equations with discontinuous Lagrangians: Soravia 02. Hamilton-Jacobi equations with measurable Hamiltonians: Camilli-Siconolfi 03. Infinite horizon optimal control problem on two-domains: Barles-Briani-Chasseigne 12. Hamilton-Jacobi approach to junction problems on network: Achdou-Camilli-Cutri-Tchou 12, Imbert-Monneau-Zidani 12. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

116 Assumptions We are interested in the Hamiltonians H i of the following Bellman form: H i (x, q) = sup { p q}, p F i (x) where F i : Ω i R 2 are multifunctions satisfying the following: for any x Ω i, F i (x) is a nonempty, convex and compact set; F i is Lipschitz continuous with respect to Hausdorff metric; A controllability assumption. For example: δ > 0 so that B(0, δ) F i (x). H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

117 Filippov regularization for the dynamics To define the global dynamics, we consider the following Filippov regularization of the multi-functions (F i ) i=1,...,4 : { Fi (x) if x Ω FC(x) := i, co(f i (x) : x Ω i ) otherwise. Then consider the differential inclusion { ẏ(s) FC(y(s)) s (0, t), y(0) = x. (10) Remark: FC is the smallest upper semi-continuous envelope of (F i ) i=1,...,4. Moreover, FC(x) = ε>0 co { F i (y) : y x < ε, y Ω i }. H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107

118 Global optimal control problem For any t [0, T ], x R 2, we set S [0,t] (x) := {y( ) y(t) = x, ẏ(s) FC(y(s)) for s (0, t)} as the set of trajectories. Consider the value function v(t, x) := inf {ϕ(y(t))}. y( ) S [0,t] (x) The upper semi-continuity and the convexity of FC imply that S [0,t] (x) is compact, then the "inf" is in fact "min". Remark: Here, v is the value function of a control problem of hybrid systems without transition cost = Zeno effect H. Zidani (ENSTA & Inria) HJ equations. Reachability analysis - LecturesGraz, 1& september / 107