Viscosity solutions of Hamilton-Jacobi equations - summer course at Universidade Federal Fluminense, Niterói. Diogo Aguiar Gomes

Transcription

1 Viscosiy soluions of Hamilon-Jacobi equaions - summer course a Universidade Federal Fluminense, Nierói Diogo Aguiar Gomes

2

3 Conens 1. Inroducion 7 2. Classical calculus of variaions 9 1. Euler-Lagrange Equaions 9 2. Furher necessary condiions Hamilonian dynamics Bibliographical noes Classical opimal conrol Opimal Conrol Elemenary properies Dynamic programming principle Opimal conrols - bounded conrol space Ponryagin maximum principle The Hamilon-Jacobi equaion Verificaion heorem Bibliographical noes Viscosiy soluions Viscosiy Soluions Sub and superdifferenials Calculus of variaions seing Viscosiy soluions - he calculus of variaions seing Uniqueness of viscosiy soluions Bibliographical noes Differenial Games Dynamic programming principle Viscosiy soluions Bibliographical noes An inroducion o mean field games The N + 1 player game A mean field model Convergence 85 3

4 4 CONTENTS 4. Bibliographical noes 95. Bibliography 97

5 CONTENTS 5

6

7 1 Inroducion These lecure noes are an abridged version of he book wrien for a course in he 27h Brazilian Mahemaics Colloquium, and correspond o he lecures given a he Universidade Federal Fluminense, Nierói, Brazil in This course covered basic noions in viscosiy soluions and is applicaions o deerminisic opimal conrol and differenial games. This books is parially based on a course on Calculus of Variaions and Parial Differenial Equaions ha I have augh over he years a he Mahemaics Deparmen of Insiuo Superior Técnico. I would like o hank my sudens: Tiago Alcaria, Parícia Engrácia, Sílvia Guerra, Igor Kravchenko, Anabela Pelicano, Ana Ria Pires, Verónica Quíalo, Lucian Radu, Joana Sanos, Ana Sanos, and Vior Saraiva, which ook my courses and suggesed me several correcions and improvemens. Also my pos-docs Andrey Byriuk, Filippo Cagnei, and Milena Chermisi, and my colleagues Pedro Girão, Cláudia Nunes Philipar, and Anónio Serra have suggesed numerous improvemens on he original ex. I would like o hank Arur Lopes ha challenged me o presen he proposal of he original course a IMPA. I would like o hank Max Souza for he inviaion o give hese lecures o a very ineresed and engaging audience. The srucure of his ex is he following: we sar wih a survey of classical mechanics and classical calculus of variaions. Then we presen he basic ools in classical opimal conrol. We coninue wih a discussion of viscosiy soluions for he erminal value problem. We follow wih a brief discussion of zero sum differenial games. We ended he course wih an inroducion o games based upon he auhor s join work wih Joana Mohr and Rafael Souza from he Universidade Federal do Rio Grande do Sul. For addiional maerial, he reader should consul he bibliographical references. In each chaper we have a secion on bibliographical noes ha liss he main references on he maerial of ha chaper. 7

8

9 2 Classical calculus of variaions This chaper is dedicaed o he sudy of classical mechanics and calculus of variaions. We sar by discussing he minimum acion principle, Euler-Lagrange equaions and some applicaions o Classical Mechanics. In secion 2 we esablish furher necessary condiions for minimizers. The following secion is dedicaed o he Hamilonian formalism. We end he chaper wih some bibliographical noes. 1. Euler-Lagrange Equaions In classical mechanics, he rajecories x : [0, T ] R n of a mechanical sysem are deermined by a variaional principle called he minimal acion principle. This principle assers ha he rajecories are minimizers (or a leas criical poins) of an inegral funcional. In his secion we sudy his problem and discuss several examples. Consider a mechanical sysem on R n wih kineic energy K(x, v) and poenial energy U(x, v). We define he Lagrangian, L(x, v) : R n R n R o be difference beween he kineic energy K and poenial energy U of he sysem, ha is, L = K U. The variaional formulaion of classical mechanics assers ha rajecories of his mechanical sysem minimize (or are a leas criical poins) of he acion funcional S[x] = 0 L(x(), ẋ())d, under fixed boundary condiions. More precisely, a C 1 rajecory x : [0, T ] R n is a minimizer S under fixed boundary condiions if for any C 1 rajecory y : [0, T ] R n such ha x(0) = y(0) and x(t ) = y(t ) we have S[x] S[y]. In paricular, for any C 1 funcion ϕ : [0, T ] R n wih compac suppor in (0, T ), and any ɛ R we have i(ɛ) = S[x + ɛϕ] S[x] = i(0). 9

10 10 2. CLASSICAL CALCULUS OF VARIATIONS Thus i(ɛ) has a minimum a ɛ = 0. So, if i is differeniable, i (0) = 0. A rajecory x is a criical poin of S, if for any C 1 funcion ϕ : [0, T ] R n wih compac suppor in (0, T ) we have i (0) = d dɛ S[x + ɛϕ] ɛ=0 = 0. The criical poins of he acion which are of class C 2 are soluions o an ordinary differenial equaion, he Euler-Lagrange equaion, ha we derive in wha follows. Any minimizer of he acion funcional saisfies furher necessary condiions which will be discussed in secion 2. Theorem 1 (Euler-Lagrange equaion). Le L(x, v) : R n R n R be a C 2 funcion. Suppose ha x : [0, T ] R n is a C 2 criical poin of he acion S under fixed boundary condiions x(0) and x(t ). Then (1) d d D vl(x, ẋ) D x L(x, ẋ) = 0. Proof. Le x be as in he saemen. compac suppor on (0, T ), he funcion Then for any ϕ : [0, T ] R n wih has a minimum a ɛ = 0. Thus ha is, 0 i(ɛ) = S[x + ɛϕ] i (0) = 0, D x L(x, ẋ)ϕ + D v L(x, ẋ) ϕ = 0. Inegraing by pars, we conclude ha [ ] d d D vl(x, ẋ) D x L(x, ẋ) ϕ = 0, 0 for all ϕ : [0, T ] R n wih compac suppor in (0, T ). This implies (1) and ends he proof of he heorem. Example 1. In classical mechanics, he kineic energy K of a paricle wih mass m wih rajecory x() is: K = m ẋ 2 2. Suppose ha he poenial energy U(x) depends only on he posiion x. Assume also ha U is smooh. Then he Lagrangian for his mechanical sysem is hen L = K U. and he corresponding Euler-Lagrange equaion is mẍ = U (x),

11 1. EULER-LAGRANGE EQUATIONS 11 which is he Newon s law. Exercise 1. Le P R n, and consider he Lagrangian L(x, v) : R n R n R defined by L(x, v) = g(x) v 2 + P v U(x), where g and U are C 2 funcions. Deermine he Euler-Lagrange equaion and show ha i does no depend on P. Exercise 2. Suppose we form a surface of revoluion by connecing a poin (x 0, y 0 ) wih a poin (x 1, y 1 ) by a curve (x, y(x)), x [0, 1], and hen revolving i around he y axis. The area of his surface is x1 x 0 x 1 + ẏ 2 dx. Compue he Euler-Lagrange equaion and sudy is soluions. To undersand he behavior of he Euler-Lagrange equaion i is someimes useful o change coordinaes. The following proposiion shows how his is achieved: Proposiion 2. Le x : [0, T ] R n be a criical poin of he acion 0 L(x, ẋ)d. Le g : R n R n be a C 2 diffeomorphism and ˆL given by Then y = g 1 x is a criical poin of ˆL(y, w) = L(g(y), Dg(y)w). 0 ˆL(y, ẏ)d. Proof. This is a simple compuaion and is lef as an exercise o he reader. Before proceeding, we will discuss some applicaions of variaional mehods o classical mechanics. As menioned before, he rajecories of a mechanical sysem wih kineic energy K and poenial energy U are criical poins of he acion corresponding o he Lagrangian L = K U. In he following examples we use his variaional principle o sudy he moion of a paricle in a cenral field, and he planar wo body problem. Example 2 (Cenral field moion). Consider he Lagrangian of a paricle in he plane subjeced o a radial poenial field. L(x, y, ẋ, ẏ) = ẋ2 + ẏ 2 U( x y 2 ).

12 12 2. CLASSICAL CALCULUS OF VARIATIONS Consider polar coordinaes, (r, θ), ha is (x, y) = (r cos θ, r sin θ) = g(r, θ), We can change coordinaes (see proposiion 2) and obain he Lagragian in hese new coordinaes ˆL(r, θ, ṙ, θ) = r2 θ 2 + ṙ 2 U(r). 2 Then he Euler-Lagrange equaions can be wrien as d d r2 θ = 0 d dṙ = U (r) + r θ 2. The firs equaion implies ha r 2 θ η is conserved. Therefore, r θ 2 = η2 r. Muliplying he second equaion by ṙ we 3 ge Consequenly d d [ṙ2 2 + U(r) + η2 2r 2 ] = 0. E η = ṙ2 η2 + U(r) + 2 2r 2 is a conserved quaniy. Thus, we can solve for ṙ as a funcion of r (given he values of he conserved quaniies E η and η) and so obain a firs-order differenial equaion for he rajecories. Example 3 (Planar wo-body problem). Consider now he problem of wo poin bodies in he plane, wih rajecories (x 1, y 1 ) and (x 2, y 2 ). Suppose ha he ineracion poenial energy U depends only on he disance (x 1 x 2 ) 2 + (y 1 y 2 ) 2 beween hem. We will show how o reduce his problem o he one of a single body under a radial field. The Lagrangian of his sysem is L = m 1 ẋ ẏ m 2 ẋ ẏ U( (x 1 x 2 ) 2 + (y 1 y 2 ) 2 ). Consider new coordinaes (X, Y, x, y), where (X, Y ) is he cener of mass X = m 1x 1 + m 2 x 2 m 1 + m 2, Y = m 1y 1 + m 2 y 2 m 1 + m 2, and (x, y) he relaive posiion of he wo bodies x = x 1 x 2, y = y 1 y 2. In hese new coordinaes he Lagrangian, using proposiion 2, is ˆL = ˆL 1 (Ẋ, Ẏ) + ˆL 2 (x, y, ẋ, ẏ). Therefore, he equaions for he variables X and Y are decoupled from he ones for x, y. Elemenary compuaions show ha d 2 d 2 X = d2 d 2 Y = 0.

13 1. EULER-LAGRANGE EQUATIONS 13 Thus X() = X 0 + V X and Y() = Y 0 + V Y, for suiable consans X 0, Y 0, V X and V Y. Since L 2 = m 1m 2 ẋ 2 + ẏ 2 U( x m 1 + m y 2 ), he problem now is reduced o he previous example. Exercise 3 (Two body problem). Consider a sysem of wo poin bodies in R 3 wih masses m 1 and m 2, whose relaive locaion is given by he vecor r R 3. Assume ha he ineracion depends only on he disance beween he bodies. Show ha by choosing appropriae coordinaes, he moion can be reduced o he one of a single poin paricle wih mass M = m1m2 m 1+m 2 under a radial poenial. Show, by proving ha r ṙ is conserved, ha he orbi of a paricle under a radial field lies in a fixed plane for all imes. Exercise 4. Le x : [0, T ] R n be a soluion o he Euler-Lagrange equaion associaed o a C 2 Lagrangian L : R n R n R. Show ha E() = L(x, ẋ) + ẋ D v L(x, ẋ) is consan in ime. For mechanical sysems his is simply he conservaion of energy. Occasionally, he ideniy d de() = 0 is also called he Belrami ideniy. Exercise 5. Consider a sysem of n poin bodies of mass m i, and posiions r i R 3, 1 i n. Suppose he kineic energy is T = m i i 2 ṙ 2 and he poenial energy is U = m im j i,j i 2 r. Le I = i r j i m i r i 2. Show ha d 2 I = 4T + 2U, d2 which is sricly posiive if he energy T + U is posiive. Wha implicaions does his ideniy have for he sabiliy of planeary sysems? Exercise 6 (Jacobi meric). Le L(x, v) : R n R n R be a C 2 Lagrangian. Le x : [0, T ] R n be a soluion o he corresponding Euler-Lagrange (2) for he Lagrangian Le E() = ẋ() V (x()). d d D vl D x L = 0, L(x, v) = v 2 2 V (x). 1. Show ha Ė = 0.

14 14 2. CLASSICAL CALCULUS OF VARIATIONS (3) 2. Le E 0 = E(0). Show ha x is a soluion o he Euler-Lagrange equaion d d D vl J D x L J = 0 associaed o L J = E 0 V (x) ẋ. 3. Show ha any reparamerizaion in ime of x is also a soluion o (3) and observe ha he funcional E0 V (x) ẋ 0 represens he lengh of he pah beween x(0) and x(t ) using he Jacobi meric g = E 0 V (x). 4. Show ha he soluions o he Euler-Lagrange (3) when reparamerized in ime in such a way ha he energy of he reparamerized rajecory is E 0 saisfy (2). Exercise 7 (Braquisochrone problem). Le (x 1, y 1 ) be a poin in a (verical) plane. Show ha he curve y = u(x) ha connecs (0, 0) o (x 1, y 1 ) in such a way ha a paricle wih uni mass moving under he influence a uni graviy field reaches (x 1, y 1 ) in he minimum amoun of ime minimizes x1 1 + u 2 2u dx. 0 Hin: use he fac ha he sum of kineic and poenial energy is consan. 4. Deermine he Euler-Lagrange equaion and sudy is soluions, using exercise Exercise 8. Consider a second-order variaional problem: (4) min x 0 L(x, ẋ, ẍ) where he minimum is aken over all rajecories x : [0, T ] R n wih fixed boundary daa x(0), x(t ), ẋ(0), ẋ(t ). Deermine he Euler-Lagrange equaion corresponding o. 2. Furher necessary condiions A classical sraegy in he sudy of variaional problems consiss in esablishing necessary condiions for minimizers. If here exiss a minimizer and if he necessary condiions have a unique soluion, hen his soluion has o be he unique minimizer and hus he problem is solved. In addiion o Euler-Lagrange equaions, several oher necessary condiions can be derived. In his secion we discuss boundary

15 2. FURTHER NECESSARY CONDITIONS 15 condiions which arise, for insance when he end-poins are no fixed, and secondorder condiions Boundary condiions. In cerain problems, he boundary condiions, such as end poin values are no prescribed a-priori. In his case, i is possible o prove ha he minimizers saisfy cerain boundary condiions auomaically. These are called naural boundary condiions. (5) Example 4. Consider he problem of minimizing he inegral 0 L(x, ẋ)d, over all C 2 curves x : [0, T ] R n. Noe ha he boundary values for he rajecory x a = 0, T are no prescribed a-priori. Le x be a minimizer of (5) (wih free endpoins). Then for all ϕ : [0, T ] R n, no necessarily compacly suppored, 0 D x L(x, ẋ)ϕ + D v L(x, ẋ) ϕd = 0. Inegraing by pars and using he fac ha x is a soluion o he Euler-Lagrange equaion, we conclude ha D v L(x(0), ẋ(0)) = D v L(x(T ), ẋ(t )) = 0. Exercise 9. Consider he problem of minimizing he inegral 0 L(x, ẋ)d, over all C 2 curves x : [0, T ] R n such ha x(0) = x(t ). Deduce ha D v L(x(0), ẋ(0)) = D v L(x(T ), ẋ(t )). Use he previous ideniy o show ha any periodic (smooh) minimizer is in fac a periodic soluions o he Euler-Lagrange equaions. Exercise 10. Consider he problem of minimizing 0 L(x, ẋ)d + ψ(x(t )), wih x(0) fixed and x(t ) free. Derive a boundary condiion a = T for he minimizers. Exercise 11 (Free boundary).

16 16 2. CLASSICAL CALCULUS OF VARIATIONS Consider he problem of minimizing 0 L(x, ẋ), over all erminal imes T and all C 2 curves x : [0, T ] R n. Show ha x is a soluion o he Euler-Lagrange equaion and ha L(x(T ), ẋ(t )) = 0, D x L(x(T ), ẋ(t ))ẋ(t ) + D v L(x(T ), ẋ(t ))ẍ(t ) 0, D v L(x(T ), ẋ(t )) = 0. Le q R and L : R 2 R given by (v q)2 L(x, v) = + x If possible, deermine T and x : [0, T ] R ha are (local) minimizers of wih x(0) = 0. 0 L(x, ẋ)ds, 2.2. Second-order condiions. If f : R R is a C 2 funcion which has a minimum a a poin x 0 hen f (x 0 ) = 0 and f (x 0 ) 0. For he minimal acion problem, he analog of he vanishing of he firs derivaive is he Euler-Lagrange equaion. We will now consider he analog o he second derivaive being nonnegaive. The nex heorem concerns second-order condiions for minimizers: Theorem 3 (Jacobi s es). Le L(x, v) : R n R n R be a C 2 Lagrangian. Le x : [0, T ] R n be a C 1 minimizer of he acion under fixed boundary condiions. Then, for each η : [0, T ] R n, wih compac suppor in (0, T ), we have (6) ηt D 2 xxl(x, ẋ)η + η T D 2 xvl(x, ẋ) η ηt D 2 vvl(x, ẋ) η 0. Proof. If x is a minimizer, he funcion ɛ I[x+ɛη] has a minimum a ɛ = 0. By compuing d2 dɛ I[x + ɛη] a ɛ = 0 we obain (6). 2 A corollary of he previous heorem is Lagrange s es ha we sae nex: Corollary 4 (Lagrange s es). Le L(x, v) : R n R n R be a C 2 Lagrangian. Suppose x : [0, T ] R n is a C 1 minimizer of he acion under fixed boundary condiions. Then D 2 vvl(x, ẋ) 0.

17 3. HAMILTONIAN DYNAMICS 17 Proof. Use Theorem 3 wih η = ɛξ() sin ɛ, for ξ : [0, T ] Rn, wih compac suppor in (0, T ), and le ɛ 0. Exercise 12. Le L : R 2n R be a coninuous Lagrangian and le x : [0, T ] R n be a coninuous piecewise C 1 rajecory. Show ha for each δ > 0 here exiss a rajecory y δ : [0, T ] R n of class C 1 such ha 0 L(x, ẋ) 0 L(y δ, ẏ δ ) < δ. As a corollary, show ha he value of he infimum of he acion over piecewise C 1 rajecories is he same as he infimum over rajecories globally C 1. Noe, however, ha a minimizer may no be C 1. Exercise 13 (Weiersrass es). Le x : [0, T ] R n be a C 1 minimum of he acion corresponding o a Lagrangian L. Le v, w R n and 0 λ 1 be such ha λv + (1 λ)w = 0. Show ha λl(x, ẋ + v) + (1 λ)l(x, ẋ + w) L(x, ẋ). Hin: To prove he inequaliy a a poin 0, choose η such ha v if 0 + λɛ η() = w if + λɛ < 0 + ɛ 0 oherwise and consider I[x + η], as ɛ Hamilonian dynamics In his secion we inroduce he Hamilonian formalism of Classical Mechanics. We sar by discussing he main properies of he Legendre ransform. Then we derive Hamilon s equaions. Aferwards we discuss briefly he classical heory of canonical ransformaions. The secion ends wih a discussion of addiional variaional principles Legendre ransform. Before we proceed, we need o discuss he Legendre ransform of convex funcions. The Legendre ransform is used o define he Hamilonian of a mechanical sysem and i plays an essenial role in many problems in calculus of variaions. Addiionally, i illusraes many of he ools associaed wih convexiy.

18 18 2. CLASSICAL CALCULUS OF VARIATIONS Le L(v) : R n R be a convex funcion, saisfying he following superlinear growh condiion: L(v) lim = +. v v The Legendre ransform L of L is L (p) = sup v R n [ v p L(v)]. This is he usual definiion of Legendre ransform in opimal conrol, see [FS06] or [BCD97]. However, i differs by a sign from he Legendre ransform radiionally used in classical mechanics: L (p) = sup v R n [v p L(v)], as i is defined, for insance, in [AKN97] or [Eva98]. They are relaed by he elemenary ideniy L (p) = L ( p). We will frequenly denoe L (p) by H(p). The Legendre ransform of H is denoed by H and is H (v) = sup p R n [ p v H(p)]. In classical mechanics, he Lagrangian L can depend also on a posiion coordinae x R n, L(x, v), bu for purposes of he Legendre ransform x is aken as a fixed parameer. In his case we wrie also H(p, x) = L (p, x). Proposiion 5. Le L(x, v) be a C 2 funcion, which for each x fixed is sricly convex and superlinear in v. Le H = L. Then 1. H(p, x) is convex in p; 2. H = L; 3. for each x H(p, x) lim = ; p p 4. le v be defined by p = D v L(x, v ), hen H(p, x) = v p L(x, v ); 5. in a similar way, le p be given by v = D p H(p, x), hen L(x, v) = v p H(p, x); 6. if p = D v L(x, v) or v = D p H(p, x), hen D x L(x, v) = D x H(p, x).

19 3. HAMILTONIAN DYNAMICS 19 Proof. The firs saemen follows from he fac ha he supremum of convex funcions is a convex funcion. To prove he second poin, observe ha For v = w we conclude ha H (x, w) = sup [ w p H(p, x)] p = sup p inf [(v w) p + L(x, v)]. v H (x, w) L(x, w). The opposie inequaliy is obained by observing, since L is convex in v, ha for each w R n here exiss s R n such ha Therefore, by leing p = s. H (x, w) sup p L(x, v) L(x, w) + s (v w). To prove he hird poin observe ha inf [(p + s) (v w) + L(x, w)] L(x, w), v H(p, x) p by choosing v = λ p p. Thus, we conclude Since λ is arbirary, we have p L(x, λ λ p H(p, x) lim inf λ. p p H(p, x) lim inf =. p p p ) To esablish he fourh poin, noe ha for fixed p he funcion v v p + L(x, v) is differeniable and sricly convex. Consequenly, is minimum, which exiss by coerciviy and is unique by he sric convexiy, is achieved for p D v L(x, v) = 0. Noe also ha v as funcion of p is a differeniable funcion by he inverse funcion heorem. The proof of he fifh poin is similar. Finally, o prove he las iem, observe ha for p(x, v) = D v L(x, v),,

20 20 2. CLASSICAL CALCULUS OF VARIATIONS we have H(p(x, v), x) = v p(x, v) L(x, v). Differeniaing his las equaion wih respec o x and using v = D p H(p(x, v), x), we obain D x H = D x L. Exercise 14. Compue he Legendre ransform of he following funcions: 1. L(x, v) = 1 2 a ij(x)v i v j + h i (x)v i U(x), where a ij is a posiive definie marix and h(x) an arbirary vecor field. L(x, v) = where a ij is a posiive definie marix. a ij (x)v i v j, L(x, v) = 1 2 v λ U(x), wih λ > 1. Exercise 15. By allowing he Lagrangian and is Legendre ransform o assume he values ± comue he Legendre ransforms of 1. for ω R n 0 if v = ω L(v) = + oherwise. 2. for ω R n se L(v) = ω v. 3. for R > 0 0 if v R L(v) = + oherwise.

21 3. HAMILTONIAN DYNAMICS Hamilonian formalism. To moivae he Hamilonian formalism, we consider he following alernaive problem. Raher han looking for curves x : [0, T ] R n, which minimize he acion 0 L(x, ẋ)d we can consider exended curves (x, v) : [0, T ] R 2n which minimize he acion (7) 0 L(x, v)d and ha saisfy he addiional consrain ẋ = v. Obviously, his problem is equivalen o he original one, however i moivaes he inroducion of a Lagrange muliplier p in order o enforce he consrain. Therefore, we will look for criical poins of (8) 0 L(x, v) + p (v ẋ)d. Proposiion 6. Le L : R n R n R be a smooh Lagrangian. Le (x, v) : [0, T ] R 2n be a criical poin of (7) under fixed boundary condiions and under he consrain ẋ = v (he choice of p is irrelevan since he corresponding erm always vanishes). Le p = D v L(x, v). Then he curve (x, v, p) is a criical poin of (8) under fixed boundary condiions. Addiionally, any criical poin (x, v, p) of (8) saisfies ẋ = v p = D v L(x, v) ṗ = D x L(x, v). In paricular, x is a criical poin of (7). Furhermore, he Euler-Lagrange equaion can be rewrien as ṗ = D x H(p, x) ẋ = D p H(p, x). Proof. Le φ, ψ and η be C 2 ([0, T ], R n ) wih compac suppor in (0, T ). Then, a ɛ = 0 d dɛ 0 L(x + ɛφ, v + ɛψ) + (p + ɛη) (v ẋ) + ɛ(p + ɛη) (ψ φ) = = 0 0 D x L(x, ẋ)φ + D v Lψ + p (ψ φ) + η (v ẋ) [D x L(x, ẋ) + ṗ] φ = 0.

22 22 2. CLASSICAL CALCULUS OF VARIATIONS If p = D v L(x, v), hen v maximizes p v L(x, v). Le H(p, x) = max [ p v L(x, v)]. v By proposiion 5 we have D x H(p, x) = D x L(x, v) whenever Addiionally, we also have p = D v L(x, v). v = D p H(p, x). Therefore, he Euler-Lagrange equaion can be rewrien as ṗ = D x H(p, x) ẋ = D p H(p, x). These are he Hamilon equaions. Exercise 16. Suppose H(p, x) : R n R n R is a C 1 funcion. Show ha he energy, which coincides wih H, is conserved by he Hamilonian flow since d H(p, x) = 0. d 4. Bibliographical noes There is a very large lieraure on he opics of his chaper. The main references we have used were [Arn95] and [AKN97]. Two classical physics books on his subjec are [Gol80] and [LL76]. On he more geomerical perspecive, he reader may wan o look a [dc92] (see also [dc79]) and [Oli02]. Addiional maerial on classical calculus of variaions can be found in [Dac09] and he classical book [Bol61]. A very good reference in Poruguese is [Lop06].

23 4. BIBLIOGRAPHICAL NOTES 23

24

25 3 Classical opimal conrol In his chaper we consider deerminisic opimal conrol problems and is connecion wih Hamilon-Jacobi equaions. We sar he discussion, in he nex secion, wih he se up of he problem. Then we presen some elemenary properies and examples. The dynamic programming principle and Ponryangin maximum principles are discussed in secions 3 and 5, respecively. The Ponryagin maximum principle is he analog of he Euler-Lagrange equaion for opimal conrol problems. Then, in secion 6 we will show ha if he value funcion V is differeniable, i saisfies he Hamilon-Jacobi parial differenial equaion V + H(D x V, x, ) = 0, in which H(p, x), he Hamilonian, is he (generalized) Legendre ransform of he Lagrangian L (9) H(p, x, ) = sup p f(x, v) L(x, v, ). v U We end his chaper wih a verificaion heorem, secion 7 ha esablishes ha a sufficienly smooh soluion o he Hamilon-Jacobi equaion is he value funcion. 1. Opimal Conrol A ypical problem in opimal conrol, whose sudy we begin now is he erminal value opimal conrol problem. For ha le he conrol space be a closed convex subse U of R m. A conrol on an inerval I R is a measurable funcion u : I U. Le f : R n U R n be a coninuous funcion, Lipschiz in x. For each conrol u we can consider he conrolled dynamics (10) ẋ = f(x, u). We could of course consider conrol laws depending on ime wihou any problem whasoever. We consider inegral soluions of (10), i.e., x is a soluion of (10) wih iniial condiion x() = x 0 if x(t ) = x f(x(s), u(s))ds,

26 26 3. CLASSICAL OPTIMAL CONTROL for all T >. I is well known from ODE heory ha, a leas locally in ime, equaion (10) admis a unique inegral soluion, for any (bounded) conrol u. We are given a running cos L : R n U R and a erminal cos ψ : R n R. To avoid echnical problems we assume ha boh L and ψ are bounded below and, wihou loss of generaliy (by adding suiable consans), we suppose for definieness, L, ψ 0. Furhermore, we require ψ L and L o saisfy he following bound: here exiss u 0 U and a consan C such ha (11) L(x, u 0 ) C. Given a erminal ime T, he erminal value opimal conrol problem consiss in deermining he opimal rajecories x( ) which minimize J[u; x, ] = L(x, u)ds + ψ(x( 1 )), among all bounded conrols u( ) : [, 1 ] R n and all soluions x of (10) saisfying he iniial condiion x() = x. The value funcion V is (12) V (x, ) = inf J[u; x, ] in which he infimum is aken over all conrols on [, T ]. An imporan example is he calculus of variaions seing, where, f(x, u) = u, and he opimal rajecories x( ), as we have shown, are soluions o he Euler- Lagrange equaion d L L (x, ẋ) (x, ẋ) = 0. d v x Furhermore, p = D v L(x, ẋ) is a soluion of Hamilon s equaions: ẋ = D p H(p, x), ṗ = D x H(p, x). In he nex chaper we will consider his problem under he ligh of opimal conrol and generalize he previous resuls. In secion 4, before considering he calculus of variaions seing, we sudy a simpler bu imporan siuaion, he bounded conrol case. In his he conrol space U is a compac convex se. Furhermore, in ha secion we suppose addiionally ha L(x, u) is a bounded coninuous funcion, convex in u. We assume furher ha he funcion f(x, u) saisfies he following Lipschiz condiion f(x, u) f(y, u) C x y.

27 2. ELEMENTARY PROPERTIES 27 To esablish exisence of opimal soluions we simplify even more by assuming ha f(x, u) has he form (13) f(x, u) = A(x)u + B(x), where A and B are Lipschiz coninuous funcions. 2. Elemenary properies In his secion we esablish some elemenary properies of he erminal value problem. Proposiion 7. The value funcion V saisfies he following inequaliies ψ V c 1 T + ψ. Proof. The firs inequaliy follows from L 0. To obain he second inequaliy i is enough o observe ha V J(x, ; 0) c 1 T + ψ. Example 5 (Lax-Hopf formula). Suppose ha L(x, v) L(v), L convex in v and coercive. Assume furher ha f(x, v) = v. By Jensen s inequaliy ( 1 T ) 1 T ( ) y x L(ẋ(s)) L ẋ(s) = L, T T T where y = x(t ). Therefore, o solve he erminal value opimal conrol problem, i is enough o consider consan conrols of he form u(s) = y x T. Thus [ ( ) ] y x V (x, ) = inf (T )L + ψ(y), y R n T and, consequenly, he infimum is a minimum. Thus Lax-Hopf formula gives an explici soluion o he opimal conrol problem. Exercise 17. Le Q and A be n n consan, posiive definie, marices. Le L(v) = 1 2 vt Qv and ψ(y) = 1 2 yt Ay. Use Lax-Hopf formula o deermine V (x, ). Proposiion 8. Le ψ 1 (x) and ψ 2 (x) be coninuous funcions such ha ψ 1 ψ 2. Le V 1 (x, ) and V 2 (x, ) be he corresponding value funcions. Then V 1 (x, ) V 2 (x, ).

28 28 3. CLASSICAL OPTIMAL CONTROL Proof. Fix ɛ > 0. Then here exiss an almos opimal conrol u ɛ and corresponding rajecory x ɛ such ha Clearly and herefore V 2 (x, ) > V 1 (x, ) L(x ɛ (s), u ɛ (s), s)ds + ψ 2 (x ɛ (T )) ɛ. L(x ɛ (s), u ɛ (s), s)ds + ψ 1 (x ɛ (T )), V 1 (x, ) V 2 (x, ) ψ 1 (x ɛ ( 1 )) ψ 2 (x ɛ ( 1 )) + ɛ ɛ. Since ɛ is arbirary, his ends he proof. An imporan corollary is he coninuiy of he value funcion on he erminal value, wih respec o he L norm. Corollary 9. Le ψ 1 (x) and ψ 2 (x) be coninuous funcions and V 1 (x, ) and V 2 (x, ) he corresponding value funcions. Then sup x V 1 (x, ) V 2 (x, ) sup ψ 1 (x) ψ 2 (x). x Proof. Noe ha ψ 1 ψ 2 ψ 2 + sup ψ 1 (y) ψ 2 (y). y Le Ṽ2 be he value funcion corresponding o ψ 2. Clearly, Ṽ 2 = V 2 + sup ψ 1 (y) ψ 2 (y). y By he previous proposiion, V 1 Ṽ2 0, which implies V 1 V 2 sup ψ 1 (y) ψ 2 (y). y By revering he roles of V 1 and V 2 we obain he oher inequaliy. 3. Dynamic programming principle The dynamic programming principle, ha we prove in he nex heorem, is simply a semigroup propery ha he evoluion of he value funcion saisfies.

29 3. DYNAMIC PROGRAMMING PRINCIPLE 29 Theorem 10 (Dynamic programming principle). Suppose ha T. Then [ ] (14) V (x, ) = inf u where x() = x and ẋ = f(x, u). L(x(s), u(s), s)ds + V (y, ), Proof. Denoe by Ṽ (x, ) he righ hand side of (14). For fixed ɛ > 0, le uɛ be an almos opimal conrol for V (x, ). Le x ɛ (s) be he corresponding rajecory rajecory, i.e., assume ha J(x, ; u ɛ ) V (x, ) + ɛ. We claim ha Ṽ (x, ) V (x, ) + ɛ. To check his, le x( ) = xɛ ( ) and y = x ɛ ( ). Then Addiionally, Therefore Ṽ (x, ) L(x ɛ (s), u ɛ (s), s)ds + V (y, ). V (y, ) J(y, ; u ɛ ). Ṽ (x, ) J(x, ; u ɛ ) V (x, ) + ɛ, and, since ɛ is arbirary, Ṽ (x, ) V (x, ). To prove he opposie inequaliy, we will proceed by conradicion. Therefore, if Ṽ (x, ) < V (x, ), we could choose ɛ > 0 and a conrol u such ha L(x (s), u (s), s)ds + V (y, ) < V (x, ) ɛ, where ẋ = f(x, u ), x () = x, and y = x ( ). Choose u such ha J(y, ; u ) V (y, ) + ɛ 2 Define u as u (s) = u (s) for s < u (s) = u (s) for < s. So, we would have V (x, ) ɛ > L(x (s), u (s), s)ds + V (y, ) L(x (s), u (s), s)ds + J(y, ; u ) ɛ 2 = = J(x, ; u ) ɛ 2 V (x, ) ɛ 2, which is a conradicion.

30 30 3. CLASSICAL OPTIMAL CONTROL 4. Opimal conrols - bounded conrol space We now give a proof of he exisence of opimal conrols for bounded conrol space. The unbounded case will be addressed in 3. Lemma 11. Le f is as in (13) a linear conrol law. Then J is weakly lower semiconinuous, wih respec o weak-* convergence in L. Proof. Le u n be a sequence of conrols such ha u n u in L [, 1 ]. Then, by using Ascoli-Arzela heorem, we can exrac a subsequence of x n ( ) converging uniformly o x( ). Furhermore, because he conrol law (13) is linear we have We have 1 J(x, ; u n ) = 1 + ẋ = f(x, u). [L(x n (s), u n (s), s) L(x(s), u n (s), s)] ds+ L(x(s), u n (s), s)ds + ψ(x n ( 1 )). The firs erm, 1 [L(x n (s), u n (s), s) L(x(s), u n (s), s)] ds, converges o zero. Similarly, ψ(x n ( 1 )) ψ(x( 1 )). Finally, he convexiy of L implies L(x(s), u n (s), s) L(x(s), u(s), s) + D v L(x(s), u(s), s)(u n (s) u(s)). Since u n u, Hence 1 D v L(x(s), u(s), s)(u n (s) u(s))ds 0. lim inf J(x, ; u n ) J(x, ; u), ha is, J is weakly lower semiconinuous. Using he previous resul we can now sae and prove our firs exisence resul. Lemma 12. Suppose he conrol se U is bounded, closed and convex. There exiss a minimizer u of J. Proof. Le u n be a minimizing sequence, ha is, such ha J(x, ; u n ) inf u U R J(x, ; u). Because his sequence is bounded in L, by Banach-Alaoglu heorem we can exrac a sequence u n u. Clearly, we have u U, by closeness and convexiy. We claim now ha J(x, ; u ) = inf J(x, ; u). u

31 5. PONTRYAGIN MAXIMUM PRINCIPLE 31 This jus follows from he weak lower semiconinuiy: inf J(x, ; u) J(x, ; u u ) lim inf J(x, ; u n ) = inf J(x, ; u), u which ends he proof. Example 6 (Bang-Bang principle). Consider he case of a bounded closed convex conrol space U and suppose he Lagrangian L is consan. Suppose f(x, u) = Au + B, for suiable consan marices A and B, and ha he erminal value ψ is convex. In his seing we firs observe ha he se of all opimal conrols is convex. As such i admis an exreme poin u. We claim ha u akes values on U. To see his, choose a ime r and suppose ha for some ɛ here is a se of posiive measure in [r, r + ɛ] for which u is in he inerior of U. Then here exiss an L funcion ν suppored on his se such ha r+ɛ dν = 0, and such ha u ± ν is r an admissible conrol. By our assumpions i is also an opimal conrol. I is clear hen ha u is no an exreme poin, which is a conradicion. 5. Ponryagin maximum principle In his secion we assume he conrol space U is bounded and ha we can apply he resuls of he previous secion o esablish exisence of an opimal conrol u and corresponding opimal rajecory x. We assume also ha he erminal daa ψ is differeniable. Le r [, 1 ) be a poin where u is srongly approximaely coninuous, i.e., ϕ(u 1 (r)) = lim δ 0 δ r+δ r ϕ(u (s))ds, for all coninuous funcions ϕ (noe ha in he limi δ can ake boh posiive and negaive values). Noe ha almos any r is a poin of approximae coninuiy, see [EG92]. Denoe by Ξ 0 he fundamenal soluion of (15) ξ0 = D x f(x, u )ξ 0, wih Ξ 0 (r) = I. Le p be given by (16) p (r) = D x ψ(x R ( 1 ))Ξ 0 ( 1 ) + 1 r D x L(x (s), u (s), s)ξ 0 (s)ds.

32 32 3. CLASSICAL OPTIMAL CONTROL Lemma 13 (Ponryagin maximum principle). Suppose ha ψ is differeniable. Le u be an opimal conrol for iniial daa (x, ) and x he corresponding opimal rajecory. Then, for almos all r (, T ), (17) f(x (r), u (r)) p (r) + L(x (r), u (r), r) = min v U [f(x, v) p (r) + L(x (r), v, r)]. Proof. Le v U. For almos all r (, T ) u is srongly approximaely coninuous (see [EG92]). Le r be one of hese poins. Le δ 0 Define v if r δ < s < r u δ (s) = u (s) oherwise. Le where x (s) if < s < r δ x δ (s) = x (r) + s r δ f(x δ, v) if r δ < s < r x (s) + δξ δ (s) if r δ < s < T, ξ δ (r) = 1 δ r r δ and y δ x (s) + δξ δ (s) solves, for r < s < 1, Observe ha [f(x δ(s), v) f(x (s), u (s))] ds, ẏ δ = f(y δ, u ). (18) ξ 0 (r) = lim δ 0 ξ δ (r + δ) = f(x (r), v) f(x (r), u (r)). By a sandard ODE resul we have ha ξ δ converges, as δ 0, o a soluion ξ 0 of (15) wih iniial daa given by (18). Thus ξ 0 (s) = Ξ 0 (s) (f(x (r), v) f(x (r), u (r))). Clearly J(, x; u ) This las inequaliy implies 1 δ + 1 δ r r δ r L(x δ (s), u δ (s), s)ds + ψ(x (T ) + δξ δ ). [L(x δ (s), v, s) L(x (s), u (s), s)] ds+ [L(x (s) + δξ δ, u (s), s) L(x (s), u (s), s)] ds+ + 1 δ [ψ(x (T ) + δξ δ ) ψ(x (T ))] 0. When δ 0, he firs erm converges o L(x (r), v, r) L(x (r), u (r), r),

33 6. THE HAMILTON-JACOBI EQUATION 33 since u is srongly approximaely coninuous. The second erm converges o r D x L(x (s), u (s), s)ξ 0 (s)ds, whereas he hird one has he following limi: D x ψ(x R (T )) ξ 0 (T )). This implies ha for almos all r (, T ), L(x (r), v, r) L(x (r), u (r), r) + p (r) (f(x (r), v) f(x (r), u (r))) 0. Consequenly as required. f(x (r), u (r)) p (r) + L(x (r), u (r), r) = min v U [f(x (r), v) p (r) + L(x R (r), v, r)], 6. The Hamilon-Jacobi equaion We now show ha if he value funcion is differeniable hen i is a soluion o he Hamilon-Jacobi equaion. Theorem 14. Le V be he value funcion o he erminal value problem. Suppose V is C 1. Then i solves he Hamilon-Jacobi equaion V + H(D x V, x) = 0. Proof. Fix any consan conrol u. principle V (x, ) +h Then, by he dynamic programming L(x(s), u ) + V (x( + h), + h). By using Taylor s formula, dividing by h we obain, as h 0, ha is 0 V + L(x, u ) + f(x, u ) D x V V H(D x V, x), V + H(D x V, x) 0. Suppose now ha in fac he previous inequaliy were sric a a poin (x 0, 0 ), ha is V (x 0, 0 ) + H(D x V (x 0, 0 ), x 0 ) = δ < 0.

34 34 3. CLASSICAL OPTIMAL CONTROL Then in a neighborhood N of (x 0, 0 ) we have V (x, ) + H(D x V (x, ), x) < δ 2. Le u be an opimal conrol, and le τ be he exi ime of N of he corresponding rajecory. Then, by Taylor s formula, V (x(τ), τ) V (x 0, 0 ) = and, by he dynamic programming principle Thus V (x 0, 0 ) = 0 = which is a conradicion. τ 0 τ τ 0 τ 0 V + f D x V L(x, u )d + V (x(τ), τ). L(x, u ) + V + f D x V d 0 V H(D x V (x, ), x) δ 2 (τ 0), Exercise 18. Le M(), N() be n n marices wih ime-differeniable coefficiens. Suppose ha is N inverible. Le D be a n n consan marix. Consider he Lagrangian L(x, v) = 1 2 xt M()x vt N()v and he erminal condiion ψ = 1 2 xt Dx. Show ha here exiss a soluion o he Hamilon-Jacobi wih erminal condiion ψ a = T (a leas for close o T ) of he form where P () saisfies he Ricai equaion and P (T ) = D. V = 1 2 xt P ()x, P = P T N 1 P M 7. Verificaion heorem Now we will show ha any sufficienly smooh soluion o he Hamilon-Jacobi equaion is he value funcion and i can be used o compue an opimal conrol. Theorem 15. Le L(x, v) be a C 1 Lagrangian, sricly convex in v, and le f(x, u) be a linear conrol law as in (13), and H be he generalized Legendre ransform (9) of L. Le Φ(x, ) be a classical soluion o he Hamilon-Jacobi equaion (19) Φ + H(D x Φ, x) = 0

35 7. VERIFICATION THEOREM 35 on he ime inerval [0, T ], wih erminal daa Φ(x, T ) = ψ(x). Then, for all 0 T, where V is he value funcion. Φ(x, ) = V (x, ), Proof. Le u be a conrol on [, T ] and x be he corresponding soluion o ẋ = f(x, u), wih x() = x. Then, using Φ(x(T ), T ) = ψ(x(t )) we have ψ(x(t )) Φ(x(), ) = = d Φ(x(s), s)ds ds D x Φ(x(s), s) f(x, u) + Φ s (x(s), s)ds. Adding L(x(s), u(s))ds+φ(x(), ) o he above equaliy and aking he infimum over all conrols u, we obain ( inf L(x(s), u(s))ds + ψ ( x(t ) )) = Φ(x(), ) ( ) T + inf Φ s (x(s), s) + L(x(s), u(s)) + D x Φ(x(s), s) f(x, u)ds. Now recall ha for any v, H(p, x) L(x, v) + p f(x, v), herefore ( V (x, ) = inf L(x(s), ẋ(s))ds + ϕ ( x(t ) )) ( ( Φ(x(), ) + inf Φs (x(s), s) H(D x Φ(x(s), s), x(s)) ) ) ds = Φ(x(), ). Le r(x, ) be uniquely defined (uniqueness follows from convexiy) as (20) r(x, ) argmin v U L(x, v) + D x Φ(x, ) f(x, v). A simple argumen shows ha r is a coninuous funcion. Now consider he rajecory x obained by solving he following differenial equaion ẋ(s) = f(x, r(x(s), s)),

36 36 3. CLASSICAL OPTIMAL CONTROL wih iniial condiion x() = x. Noe ha since he righ-hand side is coninuous here is a soluion, alhough i may no be unique. Then ( V (x, ) = inf L(x(s), ẋ(s))ds + ψ ( x(t ) )) Φ(x(), ) + = Φ(x(), ), ( Φ s ( x(s), s ) H ( Dx Φ(x(s), s), x(s) )) ds which ends he proof. We should observe from he proof ha (20) gives an opimal feedback law for he opimal conrol, provided we can find a soluion o he Hamilon-Jacobi equaion (19). 8. Bibliographical noes The main references we have used on opimal conrol are [BCD97], [FS06], [Lio82], [Bar94], and [Eva98].

37 4 Viscosiy soluions In his chaper we build upon he heory developed previously o sudy he erminal value problem. In secion 2 we give some echnical resuls concerning subdifferenials and semiconcaviy. Then, in secion 3 we consider he issue of exisence of conrols and regulariy of he value funcion in he calculus of variaions seing. I is well known ha firs order parial differenial equaions such as he Hamilon-Jacobi equaion may no admi classical soluions. Using he mehod of characerisics, he nex exercise gives an example of non-exisence of smooh soluions: Exercise 19. Solve, using he mehod of characerisics, he equaion u + u 2 x = 0 x R, > 0 u(x, 0) = ±x 2. I is herefore necessary o consider weak soluions o he Hamilon-Jacobi equaion: viscosiy soluions. In secion 4 we develop he heory of viscosiy soluions for Hamilon-Jacobi equaions, and show ha he value funcion is he unique viscosiy soluion of he Hamilon-Jacobi equaion. 1. Viscosiy Soluions A bounded uniformly coninuous funcion V is a viscosiy subsoluion (resp. supersoluion) of he Hamilon-Jacobi equaion (27) if for any C 1 funcion φ and any inerior poin (x, ) argmax V φ (resp. argmin) hen D φ + H(D x φ, x, ) 0 (resp. 0) a (x, ). A bounded uniformly coninuous funcion V is a viscosiy soluion of he Hamilon-Jacobi equaion if i is boh a sub and supersoluion. The value funcion is a viscosiy soluion of (27), alhough i may no be a classical soluion. The moivaion behind he definiion of viscosiy soluion is he following: if V is differeniable and (x, ) argmaxv φ (or argmin) hen 37

38 38 4. VISCOSITY SOLUTIONS D x V = D x φ and D V = D φ, herefore we should have boh inequaliies. The specific choice of inequaliies is relaed wih he following parabolic approximaion of he Hamilon-Jacobi equaion (21) D u ɛ + H(D x u ɛ, x, ) = ɛ u ɛ. This equaion arises naurally in opimal sochasic conrol. The limi ɛ 0 corresponds o he case in which he diffusion coefficien vanishes. Proposiion 16. Le u ɛ be a family of soluions of (21) such ha, as ɛ 0, he sequence u ɛ u uniformly. Then u is a viscosiy soluion of (27). Proof. Suppose ha φ(x, ) is a C 2 funcion such ha u φ has a sric local maximum a (x, ). We mus show ha D φ + H(D x φ, x, ) 0. By hypohesis, u ɛ u uniformly. Therefore we can find sequences (x ɛ, ɛ ) (x, ) such ha u ɛ φ has a local maximum a (x ɛ, ɛ ). Therefore, Du ɛ (x ɛ, ɛ ) = Dφ(x ɛ, ɛ ) and Consequenly, u ɛ (x ɛ, ɛ ) φ(x ɛ, ɛ ). D φ(x ɛ, ɛ ) + H(D x φ(x ɛ, ɛ ), x ɛ, ɛ ) ɛ φ(x ɛ, ɛ ). I is herefore enough o ake ɛ 0 o end he proof. Theorem 17. Assume we are in he bounded conrol case. The value funcion is a viscosiy soluion o V + H(DV, x) = 0. Proof. Le ϕ(x, ) be a smooh funcion and le (x 0, 0 ) argmin V ϕ. Wihou loss of generaliy we may assume V (x 0, 0 ) = ϕ(x 0, 0 ), and so V (x, ) ϕ(x, ) for all (x, ). By he dynamic programming principle, here exiss u h and x h such ha ϕ(x 0, 0 ) = V (x 0, 0 ) 0+h 0+h 0 L(x h, u h )d + V (x h ( 0 + h), 0 + h) h 2 0 L(x h, u h )d + ϕ(x h ( 0 + h), 0 + h) h 2.

39 2. SUB AND SUPERDIFFERENTIALS 39 Using Taylor s formula we hen conclude ha 0 0+h 0 (L(x, u) + ϕ + fd x ϕ) d h 2 0+h 0 (ϕ H(Dϕ(x), x)) d h 2. Dividing by h and sending h 0, we conclude ha ϕ (x 0, 0 ) + H(D x ϕ(x 0, 0 ), x 0 ) 0, since sup 0 s 0+h x h (s) x 0 0, since we are in he bounded conrol seing. To obain he second inequaliy, suppose (x 0, 0 ) argmax V ϕ. As before, wihou loss of generaliy we may assume V (x 0, 0 ) = ϕ(x 0, 0 ), and so V (x, ) ϕ(x, ) for all (x, ). Le u be a consan conrol. Then, by he dynamic programming principle ϕ(x 0, 0 ) = V (x 0, 0 ) 0+h 0+h This hen implies, by sending h 0, 0 0 L(x, u )d + V (x( 0 + h), 0 + h) L(x, u )d + ϕ(x( 0 + h), 0 + h). 0 L(x 0, u ) + ϕ + fd x ϕd, and so ϕ (x 0, 0 ) + H(D x ϕ(x 0, 0 ), x 0 ) Sub and superdifferenials Before proceeding wih he general case of unbounded conrol spaces we will need o discuss some echnical resuls concerning sub-differenials and semiconcaviy. Le ψ : R n R be a coninuous funcion. The superdifferenial D + x ψ(x) of ψ a x is he se of vecors p R n such ha lim sup v 0 Consequenly, p D + x ψ(x) if and only if ψ(x + v) ψ(x) p v v 0. ψ(x + v) ψ(x) + p v + o( v ),

40 40 4. VISCOSITY SOLUTIONS as v 0. Similarly, he subdifferenial, Dx ψ(x), of ψ a x is he se of vecors p such ha ψ(x + v) ψ(x) p v lim inf 0. v 0 v Exercise 20. Show ha if u : R n R has a maximum (resp. minimum) a x 0 hen 0 D + u(x 0 ) (resp. D u(x 0 )). We can regard hese ses as one-sided derivaives. In fac, ψ is differeniable hen D x ψ(x) = D + x ψ(x) = {D x ψ(x)}. More precisely, Proposiion 18. If boh D x ψ(x) and D + x ψ(x) are non-empy hen D x ψ(x) = D + x ψ(x) = {p}, furhermore ψ is differeniable a x wih D x ψ = p. Conversely, if ψ is differeniable a x hen D x ψ(x) = D + x ψ(x) = {D x ψ(x)}. Proof. Suppose ha Dx ψ(x) and D x + ψ(x) are boh non-empy. Then we claim ha hese wo ses agree and have a single poin p. To see his, ake p Dx ψ(x) and p + D x + ψ(x). Then lim inf v 0 lim sup v 0 By subracing hese wo ideniies ψ(x + v) ψ(x) p v v ψ(x + v) ψ(x) p + v v lim inf v 0 (p + p ) v v 0. In paricular, by choosing v = ɛ p+ p p p +, we obain p p + 0, which implies p = p + p. Consequenly and so D x ψ = p ψ(x + v) ψ(x) p v lim = 0, v 0 v To prove he converse i suffices o observe ha if ψ is differeniable hen ψ(x + v) = ψ(x) + D x ψ(x) v + o( v ).

41 2. SUB AND SUPERDIFFERENTIALS 41 Exercise 21. Le ψ be a coninuous funcion. Show ha if x 0 is a local maximum of ψ hen 0 D + ψ(x 0 ). Proposiion 19. Le be a coninuous funcion. Then, if ψ : R n R p D x + ψ(x 0 ) (resp. p Dx ψ(x 0 )), here exiss a C 1 funcion φ such ha ψ(x) φ(x) has a local sric maximum (resp. minimum) a x 0 and such ha p = D x φ(x 0 ). On he oher hand, if φ is a C 1 funcion such ha ψ(x) φ(x) has a local maximum (resp. minimum) a x 0 hen D x φ(x 0 ) D x + ψ(x 0 ) (resp. Dx ψ(x 0 )). Proof. By subracing p (x x 0 ) + ψ(x 0 ) o ψ, we can assume, wihou loss of generaliy, ha ψ(x 0 ) = 0 and p = 0. By changing coordinaes, if necessary, we can also assume ha x 0 = 0. Because 0 D + x ψ(0) we have lim sup x 0 ψ(x) x 0. Therefore here exiss a coninuous funcion ρ(x), wih ρ(0) = 0, such ha More precisely we can choose ψ(x) x ρ(x). ρ(x) = max{ ψ x, 0}. Le η(r) = max x r {ρ(x)}. Then η is coninuous, non decreasing and η(0) = 0. Le φ(x) = 2 x x η(r)dr + x 2. The funcion φ is C 1 and saisfies φ(0) = D x φ(0) = 0. Addiionally, if x 0, ψ(x) φ(x) x ρ(x) 2 x Thus ψ φ has a sric local maximum a 0. x η(r)dr x 2 < 0.

42 42 4. VISCOSITY SOLUTIONS To prove he second par of he proposiion, suppose ha he difference ψ(x) φ(x) has a sric local maximum a 0. Wihou loss of generaliy, we can assume ψ(0) φ(0) = 0 and φ(0) = 0. Then ψ(x) φ(x) 0 or, equivalenly, ψ(x) p x + (φ(x) p x). Thus, by seing p = D x φ(0), and using he fac ha φ(x) p x lim = 0, x 0 x we conclude ha D x φ(0) D + x ψ(0). The case of a minimum is similar. A coninuous funcion f is semiconcave if here exiss C such ha f(x + y) + f(x y) 2f (x) C y 2. Similarly, a funcion f is semiconvex if here exiss a consan such ha f(x + y) + f(x y) 2f (x) C y 2. Proposiion 20. The following saemens are equivalen: 1. f is semiconcave; 2. f(x) = f(x) C 2 x 2 is concave; 3. for all λ, 0 λ 1, and any y, z such ha λy + (1 λ)z = 0 we have λf(x + y) + (1 λ)f(x + z) f(x) C 2 (λ y 2 + (1 λ) z 2 ). Addiionally, if f is semiconcave, hen a. D + x f(x) ; b. if D x f(x) hen f is differeniable a x; c. here exiss C such ha, for each p i D + x f(x i ) (i = 0, 1), (x 0 x 1 ) (p 0 p 1 ) C x 0 x 1 2. Remark. Of course analogous resuls hold for semiconvex funcions. Proof. Clearly 2 = 3 = 1. Therefore, o prove he equivalence, i is enough o show ha 1 = 2. Subracing C x 2 o f, we may assume C = 0. Also, by changing coordinaes if necessary, i suffices o prove ha for all x, y such ha λx + (1 λ)y = 0, for some λ [0, 1], we have: (22) λf(x) + (1 λ)f(y) f(0) 0. We claim now ha he previous equaion holds for each λ = k 2 j, wih 0 k 2 j. Clearly (22) holds for j = 1. We will proceed by inducion on j. Suppose ha

43 2. SUB AND SUPERDIFFERENTIALS 43 (22) if valid for λ = k 2. We will show ha i also holds for λ = k j 2. If k is even, j+1 we can reduce he fracion and, herefore, we assume ha k is odd, λ = k 2 and j+1 λx + (1 λ)y = 0. Noe ha 0 = 1 [ ( k j+1 x + 1 k 1 ) ] 2 j+1 y + 1 [ ( k j+1 x + 1 k + 1 )] 2 j+1 y. consequenly, f(0) 1 ( ( k 1 2 f 2 j+1 x + 1 k 1 2 j f ( k j+1 x + ) ) y + ( 1 k j+1 ) ) y bu, since k 1 and k + 1 are even, k 0 = k 1 2 and k 1 = k+1 2 are inegers. Therefore ( ( f(0) 1 k0 2 f 2 j x + 1 k ) ) ( ( 0 2 j y + 1 k1 2 f 2 j x + 1 k ) ) 1 2 j y Bu his implies From k 0 + k 1 = k we obain f(0) k 0 + k 1 f(x) + 2j+1 f(0) ( 1 k 0 + k 1 2 j+1 ) f(y). k ( f(x) + 1 k ) 2j+1 2 j+1 f(y). Since he funcion f is coninuous and he raionals of he form k 2 j are dense in R, we conclude ha for each real λ, wih 0 λ 1. f(0) λf(x) + (1 λ)f(y), To prove he second par of he proposiion, observe ha by proposiion 18, a = b. To check a, i.e., ha D + x f(x), i is enough o observe ha if f is concave hen D + x f(x). By subracing C x 2 o f, we can reduce he problem o concave funcions. Finally, if p i D + x f(x i ) (i = 0, 1) hen f(x 0 ) C 2 x 0 2 f(x 1 ) C 2 x (p 1 Cx 1 ) (x 0 x 1 ), and f(x 1 ) C 2 x 1 2 f(x 0 ) C 2 x (p 0 Cx 0 ) (x 1 x 0 ). Therefore, 0 (p 1 p 0 ) (x 0 x 1 ) + C x 0 x 1 2, and so (p 0 p 1 ) (x 0 x 1 ) C x 0 x 1 2. Exercise 22. Le f : R n R be a coninuous funcion. Show ha if x 0 is a local maximum hen 0 D + f(x 0 ).

44 44 4. VISCOSITY SOLUTIONS 3. Calculus of variaions seing We now consider he calculus of variaions seing and prove he exisence of opimal conrols. The main echnical issue is he fac ha he conrol space U = R n is unbounded and herefore compacness argumens do no work direcly. We will consider he calculus of variaions seing, ha is f(x, u) = u and we will work under he following assumpions: L(x, v) : R 2n R, x R n, v R n, is a C funcion, sricly convex em v, i.e., D 2 vvl is posiive definie, and saisfying he coerciviy condiion L(x, v, ) lim =, v v for each (x, ); wihou loss of generaliy, we may also assume ha L(x, v, ) 0, by adding a consan if necessary. We will also assume ha L(x, 0, ) c 1, D x L c 2 L + c 3, for suiable consans c 1, c 2 and c 3 ; finally we assume ha here exiss a funcion C(R) such ha D 2 xxl C(R), D v L C(R) whenever v R. The erminal cos, ψ, is assumed o be a bounded Lipschiz funcion. Example 7. Alhough he condiions on L are quie echnical, hey are fulfilled by a wide class of Lagrangians, for insance L(x, v) = 1 2 vt A(x)v V (x), where A and V are C,bounded wih bounded derivaives, and A(x) is posiive definie. Forunaely, he coerciviy of he Lagrangian is enough o esablish he exisence of a-priori bounds on opimal conrols. Theorem 21. Le x R n and 0 1. Suppose ha he Lagrangian L(x, v) saisfies: A. L is C, sricly convex in v (i.e., D 2 vvl is posiive definie), and saisfying he coerciviy condiion uniformly in (x, ); L(x, v) lim =, v v

45 3. CALCULUS OF VARIATIONS SETTING 45 B. L is bounded by below (wihou loss of generaliy we assume L(x, v) 0); C. L saisfies he inequaliies L(x, 0) c 1, D x L c 2 L + c 3 for suiable c 1, c 2, and c 3 ; D. here exis funcions C 0 (R), C 1 (R) : R + R + such ha whenever v R. D v L C 0 (R), D 2 xxl C 1 (R) Then, if ψ is a bounded Lipschiz funcion, (23) 1. There exiss u L [, 1 ] such ha is corresponding rajecory x, given by is opimal, ha is i saisfies V (x, ) = ẋ (s) = u(s) x () = x, 1 L(x (s), ẋ (s))ds + ψ(x ( 1 )). 2. There exiss C, depending only on L, ψ and 1 bu no on x or such ha u(s) < C for s 1. The opimal rajecory x ( ) is a C 2 [, 1 ] soluion of he Euler-Lagrange equaion d d D vl D x L = 0 wih iniial condiion x () = x. 3. The adjoin variable p, defined by (24) p() = D v L(x, ẋ ), saisfies he differenial equaion ṗ(s) = D x H(p(s), x (s)) ẋ (s) = D p H(p(s), x (s)) wih erminal condiion Addiionally, p( 1 ) D x ψ(x ( 1 )). (p(s), H(p(s), x (s))) D V (x (s), s) for < s The value funcion V is Lipschiz, and so almos everywhere differeniable. 5. If D 2 vvl is uniformly bounded, hen for each < 1, V (x, ) is semiconcave in x.

46 46 4. VISCOSITY SOLUTIONS 6. For s < 1 (p(s), H(p(s), x (s))) D + V (x (s), s) and, herefore, DV (x (s), s) exiss for < s < 1. Proof. We will divide he proof ino several auxiliary lemmas. For R > 0, define U R = {u U : u R}. From lemma 12 here exiss a minimizer u R of J in U R. Then we will show ha he minimizer u R saisfies uniform esimaes in R. Finally, we will le R. Le p R be he adjoin variable given by he Ponryagin maximum principle. We now will ry o esimae he opimal conrol u R uniformly in R, in order o send R. Lemma 22. Suppose ψ is differeniable. Then here exiss a consan C, independen on R, such ha p R C. Proof. Since ψ is Lipschiz and differeniable we have Therefore p R (s) 1 s D x ψ D x ψ <. D x L(x R (r), u R (r) dr + D x ψ. Le V R be he value funcion for he erminal value problem wih he addiional consrain of bounded conrol: v R. From D x L c 2 L + c 3, i follows p R (s) C(V R (, x) + 1), for an appropriae consan C. Proposiion 7, shows ha here exiss a consan C, which does no depend on R, such ha V R C. Therefore p R C. As we will see, he uniform esimaes for p R yield uniform esimaes for u R. R, Lemma 23. Le ψ be differeniable. Then here exiss R 1 > 0 such ha, for all u R R 1.