Systems & Control: Foundations & Applications

Transcription

1

2 Systems & Control: Foundations & Applications Founding Editor Christopher I. Byrnes, Washington University

3 Martino Bardi Italo Capuzzo-Dolcetta Optimal Control and Viscosity Solutions of Hamilton-1acobi-Bellman Equations Springer Science+Business Media, LLC

4 Maitino Bardi Italo Capuzzo-Dolcetta Dipartimento di Matematica P. ed A. Dipartimento di Matematica Università di Padova Université di Roma "La Sapienza" Padova Roma Italy Italy Library of Congress Cataloging-in-Publication Data Bardi, M. (Martino) Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations / Martino Bardi, Italo Capuzzo Dolcetta. p. cm. - (Sytems and control) Includes bibliographical references. 1. Viscosity solutions. 2. Control theory. 3. Differential games. I. Capuzzo Dolcetta, I. (Italo), II. Title. in. Series: Systems & control. QA316.B '.64-dc CIP AMS Classifications: 49L20,49L25,35F20,90D25 Printed on acid-free paper Springer Science+Business Media New York 1997 Originally published by Birkhäuser Boston in 1997 Softcover reprint of the hardcover 1st edition 1997 Birkhäuser Copyright is not claimed for works of U.S. Government employees. Allrightsreserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without prior permission of the copyright owner. Permission to photocopy for internal or personal use of specific clients is granted by Birkhäuser Boston for libraries and other users registered with the Copyright Clearance Center (CCC), provided that the base fee of $6.00 per copy, plus $0.20 per page is paid directly to CCC, 222 Rosewood Drive, Danvers, MA 01923, U.S.A. Special requests should be addressed directly to Springer Science+Business Media, LLC ISBN ISBN (ebook) DOI / Camera-ready text provided by the authors in I^TfeX

5 Contents Preface Basic notations xi xv Chapter I, Outline of the main ideas on a model problem 1 1. The infinite horizon discounted regulator The Dynamic Programming Principle The Hamilton-Jacobi-Bellman equation in the viscosity sense 3 4. Comparison, uniqueness and stability of viscosity solutions Synthesis of optimal controls and verification theorems.., Pontryagin Maximum Principle as a necessary and sufficient condition of optimality Discrete time Dynamic Programming and convergence of approximations The viscosity approximation and stochastic control Bibliographical notes Chapter II, Continuous viscosity solutions of Hamilton-Jacobi equations Definitions and basic properties Some calculus and further properties of viscosity solutions Some comparison and uniqueness results Lipschitz continuity and semiconcavity Lipschitz continuity Semiconcavity Special results for convex Hamiltonians Semiconcave generalized solutions and bilateral supersolutions Differentiability of solutions.. " A comparison theorem Solutions in the extended sense.. 84

6 vi CONTENTS 5.5. Differential inequalities in the viscosity sense Monotonicity of value functions along trajectories Bibliographical notes Chapter III. Optimal control problems with continuous value functions: unrestricted state space The controlled dynamical system The infinite horizon problem Dynamic Programming and the Hamilton-Jacobi-Bellman equation Some simple applications: verification theorems, relaxation, stability Backward Dynamic Programming, sub- and superoptimality principles, bilateral solutions Generalized directional derivatives and equivalent notions of solution Necessary and sufficient conditions of optimality, minimum principles, and multivalued optimal feedbacks The finite horizon problem The HJB equation Local comparison and unbounded value functions Equivalent notions of solution Necessary and sufficient conditions of optimality and the Pontryagin Maximum Principle Problems whose HJB equation is a variational or quasivariational inequality The monotone control problem Optimal stopping Impulse control Optimal switching Appendix: Some results on ordinary differential equations Bibliographical notes Chapter IV. Optimal control problems with continuous value functions: restricted state space Small-time controllability and minimal time functions HJB equations and boundary value problems for the minimal time function: basic theory Problems with exit times and non-zero terminal cost Compatible terminal cost and continuity of the value function The HJB equation and a superoptimality principle Free boundaries and local comparison results for undiscounted problems with exit times Problems with state constraints Bibliographical notes

7 CONTENTS vii Chapter V. Discontinuous viscosity solutions and applications Semicontinuous sub- and supersolutions, weak limits, and stability Non-continuous solutions Definitions, basic properties, and examples Existence of solutions by Perron's method Envelope solutions of Dirichlet problems Existence and uniqueness of e-solutions Time-optimal problems lacking controllability Boundary conditions in the viscosity sense Motivations and basic properties Comparison results and applications to exit-time problems and stability Uniqueness and complete solution for time-optimal control Bilateral supersolutions Problems with exit times and general targets Finite horizon problems with constraints on the endpoint of the trajectories Bibliographical notes Chapter VI. Approximation and perturbation problems Semidiscrete approximation and e-optimal feedbacks Approximation of the value function and construction of optimal controls A first result on the rate of convergence Improving the rate of convergence Regular perturbations Stochastic control with small noise and vanishing viscosity Appendix: Dynamic Programming for Discrete Time Systems Bibliographical notes Chapter VII. Asymptotic problems Ergodic problems Vanishing discount in the state constrained problem Vanishing discount in the unconstrained case: optimal stopping Vanishing switching costs Penalization Penalization of stopping costs Penalization of state constraints Singular perturbation problems The infinite horizon problem for systems with fast components Asymptotics for the monotone control problem Bibliographical notes

8 viii CONTENTS Chapter VIII. Differential Games Dynamic Programming for lower and upper values Existence of a value, relaxation, verification theorems Comparison with other information patterns and other notions of value Feedback strategies Approximation by discrete time games Bibliographical notes Appendix A. Numerical Solution of Dynamic Programming Equations by Maurizio Falcone The infinite horizon problem The Dynamic Programming equation Synthesis of feedback controls Numerical tests Problems with state constraints Minimum time problems and pursuit-evasion games Time-optimal control Pursuit-evasion games Numerical tests Some hints for the construction of the algorithms Bibliographical notes Appendix B. Nonlinear 'Hoc control by Pierpaolo Somvia Definitions Linear systems 'Hoc control and differential games Dynamic Programming equation On the partial information problem Solving the problem Exercises Bibliographical notes 531 Bibliography 533 Index 565

9 To Alessandra and Patrizia

10 Preface The purpose of the present book is to offer an up-to-date account of the theory of viscosity solutions of first order partial differential equations of Hamilton-Jacobi type and its applications to optimal deterministic control and differential games. The theory of viscosity solutions, initiated in the early 80's by the papers of M.G. Crandall and P.L. Lions [CL8I, CL83], M.G. Crandall, L.C. Evans and P.L. Lions [CEL84] and P.L. Lions' influential monograph [L82], provides an extremely convenient PDE framework for dealing with the lack of smoothness of the value functions arising in dynamic optimization problems. The leading theme of this book is a description of the implementation of the viscosity solutions approach to a number of significant model problems in optimal deterministic control and differential games. We have tried to emphasize the advantages offered by this approach in establishing the well-posedness of the corresponding Hamilton-Jacobi equations and to point out its role (when combined with various techniques from optimal control theory and nonsmooth analysis) in the important issue of feedback synthesis. The main ideas are introduced in Chapter I where the infinite horizon discounted regulator problem is taken as a model. After the derivation of the Hamilton Jacobi Bellman equation from the Dynamic Programming optimality principle we cover, in a rather informal way, such topics as uniqueness, stability and necessary and sufficient conditions for optimality. A quick review of discrete time and stochastic approximations to the value function is given in the last two sections. Chapter II is devoted to the basic theory of continuous viscosity solutions. A long section of this chapter deals with the case of Hamilton-Jacobi-Bellman equations, corresponding to Hamiltonians which are convex with respect to the gradient variable. This is the relevant case in connection with optimal control problems. In particular we discuss the connections between viscosity solutions and some different notions, such as Lipschitz continuous functions solving the equation almost everywhere, Barron and Jensen's bilateral supersolutions, and solutions in the extended sense of Clarke. In the next two chapters the basic theory of Chapter II is specialized and developed with reference to various optimal control problems with continuous value

11 xii PREFACE function. We made the choice to present in Chapter III some problems with unrestricted state space, corresponding to the simpler case of Hamilton-Jacobi equations without boundary conditions. We note that Section 4 deals with some quasivariational inequalities and systems arising in connection with stopping times, impulse control, or switching costs. On the other hand, Chapter IV is dedicated to problems involving exit times from a given domain or constraints on the state variables for systems having suitable controllability properties, leading to boundary value problems. Chapter V is dedicated to the case of discontinuous value functions, a typical motivation coming from minimum time problems lacking controllability. Various notions of discontinuous viscosity solutions, including those of Barles and Perthame, Ishii, Barron and Jensen, and Subbotin, are discussed and compared. Section 1 of this chapter is of independent interest since there we develop the so-called weak limits technique of Barles and Perthame, a very useful tool which we adopt to tackle the various limit problems in Chapters VI and VII. In particular, in Chapter VI we consider an approximation scheme for value functions based on discrete time Dynamic Programming. This is an important issue in the applications because it also provides a method to construct almost optimal feedback controls. The convergence of the scheme and some estimates on the rate of convergence are proved by viscosity solutions methods. A section is dedicated to regular perturbation problems where similar techniques can be employed. Chapter VII deals with the analysis of some asymptotic problems such as singular perturbations, penalization of state constraints, vanishing discount and vanishing switching costs. The limiting behavior of the associated Hamilton-Jacobi equations is analyzed in a simple way by the viscosity solutions approach and the weak limit technique. The last chapter is intended as an introduction to the theory of two-person zero sum differential games. Different notions of value are discussed together with the derivation of the relevant Hamilton-Jacobi-Isaacs equations. It is worth noting that the viscosity solutions method appears to be highly privileged for the treatment of this kind of nonconvex, nonlinear PDE's. Finally, two appendices deal with some additional topics which are important for the applications. The first one, by M. Falcone, describes in some detail a computational method based on the approximation theory developed in Chapter VI. The second, by P. Soravia, gives a fairly complete account of some recent results on the viscosity solutions approach to 'Hoc control Our main goal in planning this work was to provide a self-contained but, at the same time, rather comprehensive presentation of the topic to scientists in the areas of optimal control, system theory, and partial differential equations. To this end, each chapter is enriched with a section of bibliographical and historical notes. As is often the case, the present book originated from lecture notes of courses taught by the authors. We believe that the style of presentation (in particular, the set of exercises proposed at the end of each section) reflects this pedagogically oriented origin, and so selected parts could easily be used for a graduate course in

12 PREFACE xiii optimal control. In this regard, a possible path for a one semester course, which we have tested in lectures at Paris-Dauphine, Rome, Padua, and Naples, might include the first three sections of Chapter II, a choice of a few sections from Chapters III and IV (for example, selected parts of sections 1-3 of Chapter III and the first two sections of Chapter IV) and the initial sections of Chapters V and VI. The prerequisites for reading most of the book are just advanced calculus, some very basic functional analysis (uniform convergence and the Ascoli-Arzela theorem) as well as, of course, the fundamental facts about nonlinear ordinary differential equations which are recalled in the appendix to Chapter III. These are also the only prerequisites for the course outlined above. More sophisticated mathematical tools such as relaxed controls, 2nd order POE's, stochastic control and the geometric theory of controllability appear in the book but are not essential for understanding its core. We are happy to thank L.C. Evans and P.L. Lions who introduced us to viscosity solutions in the very early stages of the theory and also the other colleagues and friends with whom we have collaborated, in particular H. Ishii, B. Perthame, M. Falcone, P. Soravia. We had the occasion to discuss the overall project of the book with W.H. Fleming, M.H. Soner and G. Barles. Additional thanks are due to M. Falcone and P. Soravia for contributing the Appendices. Several people read parts of the manuscript. We are particularly grateful to o. Alvarez, S. Bortoletto and A. Cutd for reading the book from cover to cover and for stimulating remarks, and to F. Da Lio, P. Goatin, F. Gozzi, S. Mirica, M. Motta, F. Rampazzo, C. Sartori, F. Sullivan. We also thank A.I. Subbotin, N.N. Subbotina, and A.l\t Tarasyev for translating some of their papers in Russian into English for us. Finally, thanks are due to G. Bertin for skilled and sensitive typing. April 1997 Martino Bardi, [talo Capuzzo-Dolcetta

13 Basic notations x y span{... } Ixl B(xo,r) B(xo,r) 8E inte E C coe coe P(X) 0' cc 0 lei = mease d(x, E) = dist(x, E) diame dh(e,s) n(x) "lao' "(VO' "(+,,,(- [rj the euclidean N-dimensional space N the scalar product E XiYi of vectors x = (Xl,., X N) and Y = (YI:..., YN) i=l the vector space generated by the vectors {... } the euclidean norm of X E JRN, Ixl = (x. x)i/2 the open ball {x E JRN : Ix - xol < r} the closed ball {x E JRN : Ix - xol $ r} the boundary of the set E the interior of the set E the closure of the set E the complement of the set E the convex hull of the set E the closed convex hull of the set E the set of all subsets of X means 0' ~ 0 the Lebesgue measure of the set E the distance from x to E (i.e., d(x, E) = infyee Ix - yl) the diameter of the set E (Le., diam E = sup{lx - yl : x, Y E E}) the Hausdorff distance between the sets E and S ( III.2.2) the outward normal unit vector to a set E at x E 8E minh, O'} for,,(,0' E R maxh, O'} for "1,0' E R the positive and negative part of"( E R (Le., "(+ = "I V 0, "1- = "I A 0) the integer part of r E JR

14 xvi BASIC NOTATIONS sgnr argmineu liulloo Un '\. U Un./' U Un::; U suppu Xx w oct) as t -+ a Du(x) D+u(x), D-u(x) 8cu(x) 8u(x) D*u(x) &(Xjq) a+u(xj q), 8-u(xj q) uo(xj q), uo(xj q) Au l!(x) = liminf.u,,(x),,->0+ u(x) = limsup*ue(x) 10->0+ u., u A the sign of r E R (1 if r > 0, -1 if r < 0, 0 if r = 0) the set of minimum points of u : E -+ R the supremum norm supxee lu(x)1 of a function u: E -+ R the sequence of functions Un is nonincreasing and tends to u the same as the preceding, but Un is nondecreasing uniform convergence of the functions Un to u the support of the function u, i.e., the closure of the set {x: u(x) t= O} the characteristic (or indicator) function of the set X (1 in X, 0 outside) a modulus, i.e., a function w : [0, +oo[ -+ [0, +oo[ continuous, nondecreasing, and such that w(o) = OJ or, more generally, w : [0, +00[2 -+ [0, +oo[ such that for all R > 0 w(.,r) has the preceding properties a function u such that limt->a u(t)jt = 0 the gradient of the function U at x, i.e., Du(x) = ( 8U 8u) 8Xl (x),..., 8Xl (x) the super- and subdifferential of u at x ( 1.3) the subdifferential of the convex function of u at x ( II.1) the Clarke's gradient of u at x ( II.4.1) {p ERN: p = limn->+oo Du(xn ), Xn -+ x} ( II.4.1) the (one-sided) directional derivative of u at x in the direction q ( II.4.1) the (generalized) Dini directional derivatives ( 1I.4.1 and III.2.4) the regularized directional derivatives ( 1I.4.1) the laplacian of the function u, i.e., Au = i ~ : : ~ the lower weak limit of U E at x as e ( V.l) the upper weak limit of U E at x as e ( V.l) the lower and the upper semicontinuous envelopes of u ( V.2) the set of controls, i.e., (Lebesgue) measurable functions 0: [O,+oo[ -+ A the set of relaxed controls ( III.2.2) the state at time t of a control system ( III.l) the first entry time of the state in some given closed set T ( 1I.5.6, 1II.2.3, and JV.l) the first entry time of the state in int T ( IV.l)

15 BASIC NOTATIONS xvii f(x,a) Dxf B(E) C(E) BC(E) UC(E) BUC(E) Lip(E) USC(E), LSC(E) BUSC(E), BLSC(E) BC8(0) LI (Ito, tl]) Loo([to, tl]) LI(O,T;E) Loo(O, T;E) STCT STLC o STCT r,6 :F, 9... <l {q E JRN : q = f(x,a) for some a E A} the Jacobian matrix of f with respect to the x variable the space of functions u : E -+ JR with liulioo < +00 the space of continuous functions u : E -+ JR the space B(E) n C(E) the space of uniformly continuous functions u : E -+ JR the space B(E) n UC(E) the space of Lipschitz continuous functions u : E -+ JR, i.e., such that for some L ~ lu(x) - u(y)1 :5 L Ix - yl for all x,ye E the space of locally Lipschitz continuous functions u : E -+ JR, i.e., functions whose restriction to any compact subset of E is Lipschitz continuous the space of "Y-Holder continuous functions u : E -+ JR, "Y E. lu(x) - u(y)1 10,1[, I.e., sup I I < +00 x,yee X - Y "I for k ~ 1 and n open subset of JRN, the subspace of C(n) of functions with continuous partial derivatives in n up to order k for E ~ JRN, the space of functions that are the restrictions to E of some u E Ck(n) for some open set n ;2 E the spaces of lower and upper semi continuous functions u : E-+JR the spaces USC(E) n B(E) and LSC(E) n B(E) the subspace of B(O) of the functions continuous at all points of an the Lebesgue space of integrable function [to, til -+ lr. the Lebesgue space of essentially bounded function [to, til -+ JR the space of integrable function [0, T] -+ E the space of essentially bounded function [0, T] -+ E small-time controllability on T ( IV.l) small-time local controllability ( IV.l) small-time controllability on int T ( IV.l) nonanticipating strategies for the first and the second player ( VIII.1) feedback strategies for the first and the second player ( VIII.3.1) end of a proof end of a definition, end of a remark, end of an example