Numerical Methods for Differential Games based on Partial Differential Equations

Transcription

1 Numerical Methods for Differential Games based on Partial Differential Equations M. Falcone 29th April 25 Abstract In this paper we present some numerical methods for the solution of two-persons zero-sum deterministic differential games. The methods are based on the dnamic programming approach. We first solve the Isaacs equation associated to the game to get an approimate value function and then we use it to reconstruct approimate optimal feedback controls and optimal trajectories. The approimation schemes also have an interesting control interpretation since the time-discrete scheme stems from a dnamic programming principle for the associated discrete time dnamical sstem. The general framework for convergence results to the value function is the theor of viscosit solutions. Numerical eperiments are presented solving some classical pursuit-evasion games. This paper is based on the lectures given at the Summer School on Differential Games and Applications, held at GERAD, Montreal (June 14-18, 24). Kewords. AMS Subject Classification. Primar 65M12; Secondar 49N25, 49L2. Pursuit-evasion games, numerical methods, dnamic programming, Isaacs equation 1 Introduction In this paper we present a class of numerical methods for two-persons zero-sum deterministic differential games. These methods are strongl connected to Dnamic Programming (DP in the sequel) for two main reasons. The first is that we solve the Isaacs equation related to the game and compute an approimate value function from which we derive all the informations and the approimations for optimal feedbacks and optimal trajectories. The second is that the schemes are derived from a discrete version of the Dnamic Programming Principle which gives a nice control interpretation and helps in the analsis of the properties of the schemes. This paper is intended to be a tutorial on the subject so we will present the main ideas and results tring to avoid technicalities. The interested reader can find all the details and recent developments in the list of references. It is worth to note that DP is one of the most important tools in the analsis of twopersons zero-sum differential games. It has its roots in the classical work on calculus of variations and was applied etensivel to optimal control problems b Bellman. Work partiall supported b MIUR Project 23 Metodologie Numeriche Avanzate per il Calcolo Scientifico. We gratefull acknowledge the technical support given b CASPUR to the development of this research. 1

2 The central object of investigation for this method is the value function v() of the problem, which is, roughl speaking, the outcome of the game with initial state if the plaers behave optimall. In the 6s man control problems in discrete-time were investigated via the DP method showing that v satisfies a functional equation and that one can derive from its knowledge optimal controls in feedback form. For discrete-time sstems, DP leads to a difference equation for v, we refer the interested reader to the classical books [Ber] and [BeS] for an etended presentation of these results. For continuous-time sstems, the situation is more delicate since the DP equation is a nonlinear partial differential equation (PDE in the sequel) and one has to show first that the equation has a solution and that this solution is unique, so it coincides with the value function. The main difficult is the fact that, in general, the value functions of deterministic optimal control problems and games are not differentiable (the are not even continuous for some problems) so the are not classical solutions of the DP equation. Moreover, for a deterministic optimal control/game problem the DP equation is of first order and it is well-known that such equations do not have global classical solutions. In general, one does not even know how to interpret the DP equation at points where v is not differentiable. One wa to circumvent the problem is to tr to solve the equation eplicitl but this can be done onl for sstems with low state-space dimensions (tpicall 1 or 2 dimensions). Isaacs used etensivel the DP principle and the first order PDE which is now associated to his name in his book [I] on two-persons zero-sum differential games, but he worked mainl on the eplicit solution of several eamples where the value function is regular ecept some smooth surfaces. Onl at the beginning of the eighties M.C. Crandall and P.L. Lions [CL] introduced a new notion of weak solution for a class of first-order PDEs including Isaacs equations, and proved their eistence, uniqueness and stabilit for the main boundar value problems. These solutions are called viscosit solutions because the coincide with the limits of the approimations obtained adding a vanishing second order term to the equation (a vanishing artificial viscosit in phsical terms, which eplains the name). The theor was reformulated b Crandall, L.C. Evans and Lions [CEL] and P.L. Lions proved in [L] that the value functions of optimal control problems are viscosit solutions of the corresponding DP equations as soon as the are continuous. The same tpe of results was obtained for the value function of some zero-sum differential games b Barron, Evans, Jensen and Souganidis in [BEJ], [ES], [So] for various definitions of upper and lower value. Independentl, Subbotin [Su 1, Su 2] found that the value functions in the Krassovski-Subbotin [KS] sense of some differential games satisf certain inequalities for the directional derivatives which reduce to the Isaacs equation at points of differentiabilit. Moreover, he introduced a different notion of weak solution for first order nonlinear PDEs, the minma solution. The book [Su 6] presents this theor with a special emphasis on the solution of differential games which motivated his stud and the name for these solutions (since in games the DP equation has a min-ma operator, see Section 2). It is important to note that several proofs were given of the equivalence of this notion of weak solution with viscosit solutions (see [EI, LS, SuT] so that nowadas the two theories are essentiall unified, cfr.[su 3, Su 4, Su 6]. The theor of viscosit solutions has received man contributions in the last twent ears so that it now covers several boundar and Cauch problems for general first and second order nonlinear PDEs. Moreover, the theor has been developed to deal with discontinuous solutions at least for some classes of Hamilton-Jacobi equations, which include DP equations for control problems (see [Ba 1], [BJ 2], [S 3]). An etensive presentation of the theor of 2

3 viscosit solutions can be found in the surve paper [CIL] and in the book [Ba 2]. For the applications to control problems and games the interested reader should refer to the books [FS] and [BCD]. The numerical approimation of viscosit solutions has grown in parallel with the above mentioned theoretical results and in [CL 2] Crandall and Lions have investigated a number of monotone schemes proving a general convergence result. As we said, the theor has shown to be successful in man fields of application so that numerical schemes are now available for nonlinear equations arising e.g. in control, games, image processing, phase-transitions, economics. The theor of numerical approimation offers some general convergence results mainl for first order schemes and for conve Hamiltonians, see [BSo], [LT]. A general results for high-order semi-lagrangian approimation schemes has been recentl proved b Ferretti [Fe]. The approimation of the value function in the framework of viscosit solutions has been investigated b several authors starting from Capuzzo Dolcetta [CD] and [F 1]. Several convergence results as well as a-priori error estimates are now available for the approimation of classical control problems and games, see e.g. [BF 1], [BF 2], [CDF], [A 1], [A 2], [BS 3]. The interested reader will find in the surve papers [F 1] and [BFS2] a comprehensive presentation of this theor respectivel for control problems and games. We will present in the sequel the main results of this approach which is based on a discretization in time of the original control/game problem followed b a discretization in space which result in a fied point problem. This approach is natural for control problems since at ever discretization we keep the meaning of the approimate solutions in terms of the control problem and we have a-priori error estimates which just depend on the data of the problem and on the discretization steps. Thus the algorithms based on this approach produce approimate solutions which are close to the eact solution within a given tolerance. Moreover, b the approimate value function one can easil compute approimate feedback controls and optimal trajectories. For the snthesis of feedback controls we have some error estimates in the case of control problems [F 2] but the problem is still open for games. Before starting our presentation let us quote other numerical approaches related to the approimation of games. The theor of minma solutions has also a numerical counterpart which is based on the construction of generalized gradients adapted to finite difference operators which approimate the value function. This approach has been developed b the russian school (see [TUU], [PT]) and has also produced an approimation of optimal feedback controls [T]. Another approimation for the value and for the optimal policies of dnamic zero-sum stochastic games has been proposed in [TA], [TPA] and it is based on the approimation of the game b a finite state approimation (see also [TG 1] for a numerical approimation of zero-sum differential games with stopping time). The theor of viabilit [A] gives a different characterization of the value function of control/game problems: the value function is the boundar of the viabilit kernel. This approach is based on set-valued analsis and allows to deal easil with lower semicontinuous solutions of the DP equation [F]. The numerical counterpart of this approach is based on the approimation of the viabilit kernel and can be found in [CQS 1] and [CQS 2]. Finall, let us mention that other numerical methods based on the approimation of open-loop control have been proposed. The advantage is of course to replace the approimation of the DP equation (which can be difficult or impossible to solve for high-dimensional problems) b a large sstem of ordinar differential equations eploiting the necessar conditions for the optimal polic and trajector. The interested 3

4 reader can find in [P] a general presentation and in [LBP 1], [LBP 2] some eamples of the effectiveness of the method. The outline of the paper is the following. In Section 1 we briefl present the general framework of the theor of viscosit solutions which allows to characterize the value function of control/game problems as the unique solution of the DP equation, at the end we give some informations about the etension to discontinuous viscosit solutions. The first part of this section partl follows the presentation in [B 2]. Section 3 is devoted to the numerical approimation, here we analze the time and the space discretization giving the basic results on convergence and error estimates. A reader alread skilled on viscosit solutions can start from there. In Section 4 we shortl present the algorithm to compute a numerical snthesis of optimal controls and give some informations on recent developments. Finall, Section 5 is devoted to the numerical solution of some classical pursuit-evasion games b the methods presented in this paper. 2 Dnamic Programming for games and viscosit solutions Let us consider the nonlinear sstem { ẏ(t) = f((t), a(t), b(t)), t >, () = (D) where (t) R N is the state a( ) A is the control of plaer 1 (plaer a) b( ) B is the control of plaer 2 (plaer b), A{ a : [, + [ A, measurable } (1) B = { b : [, + [ B, measurable }, (2) A, B R M are given compact sets. A tpical choice is to take as admissible control function for the two plaers piecewise constant functions respectivel with values in A or B. Assume f is continuous and f(, a, b) f(, a, b) L, R N, a A, b B. B Caratheodor s theorem the choice of measurable controls guaratees that for an given a( ) A and b( ) B, there is a unique trajector of (D) which we will denote b (t; a, b). The paoff of the game is t (a( ), b( )) = min{ t : (t; a, b) T } +, (3) where T R N is a given closed target. Naturall, t {a( ), b( )} will be finite onl under additional assumptions on the target and on the dnamics. The two plaers are opponents since plaer a wants to minimize the paoff (he is called the pursuer) whereas plaer b wants to maimize the paoff (he is called the evader). Let us give some eamples. 4

5 Eample 1: Minimum time problem This is a classical control problem, here we have just one plaer: { ẏ = a, A = { a R N : a = 1 }, () =. Since the maimum speed is 1, t (a ) is equal to the length of the optimal trajector joining and the point (t (a )), thus t (a ) = min a A t (a) = dist(, T ). Note that an optimal trajector is a straight line. Eample 2: Pursuit-Evasion games We have two plaers, each one controlling its own dnamics { ẏ1 = f 1 ( 1, a), i R N/2, i = 1, 2 ẏ 2 = f 2 ( 2, b) (PEG) The target is T ɛ { 1 2 ɛ }, for ɛ >, or T { ( 1, 2 ) : 1 = 2 }. Then, t (a( ), b( )) is the capture time corresponding to the strategies a( ) and b( ). 2.1 Dnamic Programming for a single plaer Let us consider first the case of a single plaer. So in this section we assume B = { b } which allow us to write the dnamics (D) as { ẏ = f(, a), t >, Define the value function () =. T () inf t (a). a( ) A T ( ) is the minimum-time function, it is the best possible outcome of the game for plaer a, as a function of the initial position of the sstem. Definition 2.1 The reachable set is R { R N : T () < + }, i.e.it is the set of starting points from which it is possible to reach the target. The reachable set depends on the target, on the dnamics and on the set of admissible controls in a rather complicated wa, it is not a datum in our problem. Lemma 1 (Dnamic Programming Principle) For all R, t < T () (so that / T ), T () = inf { t + T ( (t; a)) }. (DPP) a( ) A 5

6 Proof The inequalit follows from the intuitive fact that a( ) T () t + T ( (t; a)). The proof of the opposite inequalit is based on the fact that the equalit holds if a( ) is optimal for. To prove rigorousl the above inequalities the following two properties of A are crucial: 1. a( ) A s R the function t a(t + s) is in A; 2. a 1, a 2 A and then a( ) A, s >. a(t) { a1 (t) t s, a 2 (t) t > s. Note that the DPP works for A = { piecewise constants functions into A } but not for A = { continuous functions into A } because joining together two continuous controls we are not guaranteed that the resulting control is continuous. Let us derive the Hamilton-Jacobi-Bellman equation from the DPP. Rewrite (DPP) as T () inf a( ) T ( (t; a)) = t and divide b t >, { } T () T ( (t; a)) sup = 1 t < T (). a( ) t We want to pass to the limit as t +. Assume T is differentiable at and lim t + commute with sup a( ). Then, if ẏ (; a) eists, sup { T () ẏ (, a) } = 1, a( ) A so that, if lim t + a(t) = a, we get sup { T () f(, a ) } = 1. (HJB) a A This is the Hamilton-Jacobi-Bellman partial differential equation associated to the minimum time problem. It is a first order, full nonlinear PDE. Note that in (HJB) the supremum is taken over the A and not on the set of measurable controls A. Let us define the Hamiltonian, we can rewrite (HJB) in short as H 1 (, p) sup{ p f(, a) } 1, a A H 1 (, T ()) = in R \ T. 6

7 Note that H 1 (, ) is conve since is the sup of linear operators. condition on T is T () = for T A natural boundar Let us prove that if T is regular then it is a classical solution of (HJB) (i.e.t C 1 (R \ T )) and (HJB) is satisfied pointwise). Proposition 2 If T ( ) is C 1 in a neighborhood of R \ T, then T ( ) satisfies (HJB) at. Proof We first prove the inequalit. Fi ā(t) a t, and set (t) = (t; ā). (DPP) gives T () T ( (t)) t t < T (). We divide b t > and let t + to get T () ẏ () 1, where ẏ () = f(, a ) (since ā(t) a ). Then, T () f(, a ) 1 a A and we get sup{ T () f(, a) } 1. a A Net we prove the inequalit. Fi ε >. For all t ], T ()[, b (DPP) there eists α ε A such that which implies T () t + T ( (t; α ε )) εt, t 1 ε T () T ( (t; α ε )) = 1 t t s T ( (s; α ε )) ds = 1 T ( (s)) ẏ (s; α ε ) ds. t (4) Adding and subtracting the term T () f( (s; α ε )) b the Lipschitz continuit of f we get from (4) 1 ε 1 t so for t +, ε + we finall get t t T () f(, α(s)) ds + o(1) (5) sup{ T () f(, a) } 1. a A Unfortunatel T is not regular even for simple dnamics as the following eample shows. Consider Eample 1 where T () = dist(, T ) it is eas to see that T is not 7

8 differentiable at if there eist two distinct points of minimal distance. N = 1, f(, a) = a, A = B(, 1) and In fact, for we have T = ], 1] [1, + [. T () = 1 which is not differentiable at =. We must observe also that in this eample the Bellman equation is the eikonal equation Du() = 1 (6) which has infinitel man a.e. solutions also when we fi the values on the boundar T, u( 1) = u(1) = Also the continuit of T is, in general, not guarateed. Take the previous eample and set A = [ 1, ], then we have T (1) = lim T () = 2 (7) 1 However, the continuit of T ( ) is equivalent to the propert of Small-Time Local Controllabilit (STLC) around T. Definition 2.2 Assume T smooth. We sa that the STLC is satisfied if where η() is the eteriour normal to T at. T â A : f(, â) η() <. (STLC) The STLC guarantees that R is an open subset of R N and that lim T () = +, R We want to interpret the HJB equation in a weak sense so that T ( ) is a solution (non-classical), unique under suitable boundar conditions. Let s go back to the proof of Proposition 2. We showed that 1. T () T ( (t)) t, t small and T C 1 implies H(, T ()) ; 2. T () T ( (t)) t(1 ε), t, ε small and T C 1 implies H(, T ()). Idea: If φ C 1 and T φ has a maimum at then T () φ() T ( (t)) φ( (t)) t, thus φ() φ( (t)) T () T ( (t)) t, so we can replace T b φ in the proof of Proposition 2 and get H(, φ()). 8

9 Similarl, if φ C 1 and T φ has a minimum at, then T () φ() T ( (t)) φ( (t)), t. thus and, b the proof of Proposition 2, φ() φ( (t)) T () T ( (t)) t(1 ε) H(, φ()). Thus, the classical proof can be fied when T is not C 1 replacing T with a test function φ C 1. Definition 2.3 (Crandall-Evans-Lions [CEL]) Let F : R N R R N R be continuous, Ω R N open. We sa that u C(Ω) is a viscosit subsolution of F (, u, u) = in Ω if φ C 1, local maimum point of u φ, F (, u( ), φ( )). It is a viscosit supersolution if φ C 1, local minimum point of u φ, F (, u( ), φ( )). A viscosit solution is a sub- and supersolution. Theorem 3 If R \ T is open and T ( ) is continuous, then T ( ) is a viscosit solution of the Hamilton-Jacobi-Bellman equation (HJB). Proof. The proof is the argument before the definition. The following result (see e.g. [BCD] for the proof) shows the link between viscosit and classical solutions. Corollar 4 1. If u is a classical solution of F (, u, u) = in Ω then u is a viscosit solution; 2. if u is a viscosit solution of F (, u, u) = in Ω and if u is differentiable at then the equation is satisfied in the classical sense at, i.e. F (, u( ), u( )) =. 9

10 It is clear that the set of viscosit solutions contains that of classical solutions. The main issue in this theor of weak solutions is to prove uniqueness results. This point is ver important also for numerical purposes since the fact that we have a unique solution allow to prove convergence results for the approimation schemes. To this end, let us consider the Dirichlet boundar value problem { u + H(, u) = in Ω (BVP) u = g on Ω and prove a uniqueness result under assumptions on H including Bellman s Hamiltonian H 1. This new boundar value problem is connected to (HJB) because the new solution of (BVP) is a rescaling of T. In fact, introducing the new variable V () { 1 e T () if T () < +, i.e. R 1 if T () = +, ( / R) (8) it is eas to check that, b the DPP, V () = inf J(, a) a( ) A where Moreover, V is a solution of J(, a) t(a) e t dt. { V + ma { V f(, a) 1 } = a A in RN \ T V = on T, (BVP-B) which is a special case of (BVP), with H(, p) = H 1 (, p) ma a A { p f(, a) 1 } and Ω = T c R N \ T. The change of variable (8) is called Kružkov transformation and has several advantages. First of all V takes values in [, 1] whereas T is generall unbounded and this helps in the numerical approimation. Moreover, one can alwas reconstruct T and R from V b the relations T () = log(1 V ()), R = { : V () < 1 }. Lemma 5 The Mininimum Time Hamiltonian H 1 satisfies the structural condition H(, p) H(, q) K(1 + ) p q + q L,, p, q, (SH) where K and L are two positive constants. Theorem 6 (Crandall-Lions [CL 1]) Assume H satisfies (SH), u, w BUC( Ω), u subsolution, w supersolution of v + H(, v) = in Ω (open), u w on Ω. Then, u w in Ω. Definition 2.4 We call subsolution (respectivel supersolution) of (BVP-B) a subsolution (respectivel supersolution) u of the differential equation such that u on Ω (respectivel on Ω). 1

11 Corollar 7 If the value function V ( ) BUC(T c ), then V is the maimal subsolution and the minimal supersolution of (BVP-B) (we sa it is the complete solution). Thus V is the unique viscosit solution. If the sstem is STLC around T (i.e. T ( ) is continuous at each point of T ) then V BUC(T c ) and we can appl the Corollar. 2.2 Dnamic Programming for games We now go back to our original problem to develop the same approach. The first question is: how can we define the value function for the 2-plaers game? Certainl it is not inf sup J(, a, b) a A b B because a would choose his control function with the information of the whole future response of plaer b to an control function a( ) and this will give him a big advantage. A more unbiased information pattern can be modeled b means of the notion of nonanticipating strategies (see [EK] and the references therein), { α : B A : b(t) = b(t) t t α[b](t) = α[ b](t) t t }, (9) Γ { β : A B : a(t) = ã(t) t t β[a](t) = β[ã](t) t t }. (1) The above definition is fair with respect to the two plaers. In fact, if plaer a chooses his control in he will not be influenced b the future choices of plaer b (Γ has the same role for plaer b). Now we can define the lower value of the game or T () inf V () inf sup α b B sup α b B when the paoff is J(, a, b) = t (a,b) e t dt. Similarl the upper value of the game is or t (α[b], b), J(, α[b], b) T () sup inf t (a, β[a]), a A β Γ Ṽ () sup inf J(, a, β[a]). β Γ a A We sa that the game has a value if the upper and lower values coincide, i.e.if T = T or V = Ṽ. Lemma 8 (DPP for games) For all t < T () and T () = inf V () = inf α sup b B sup α b B { t { t + T ( (t; α[b], b)) }, R \ T, } e s ds + e t V ( (t; α[b], b)), T c. 11

12 The proof is similar to the 1-plaer case but more technical due to the use of nonanticipating strategies. Note that the upper values T and Ṽ satisf a similar DPP. Let us introduce the two Hamiltonians for games Isaacs Lower Hamiltonian H(, p) min b B ma { p f(, a, b) } 1. a A Isaacs Upper Hamiltonian H(, p) ma a A min b B { p f(, a, b) } 1. Theorem 9 (Evans-Souganidis [ES] ) 1. then T ( ) is a viscosit solution of If R \ T is open and T ( ) is continuous, 2. If V ( ) is continuous, then it is a viscosit solution of H(, T ) = in R \ T. (HJI-L) V + H(, V ) = in T c. The structural condition (SH) plas an important role for uniqueness. Lemma 1 Isaacs Hamiltonians H, H satisf the structural condition (SH). Then Comparison Theorem 6 applies and we get Theorem 11 If the lower value function V ( ) BUC(T c ), then V is the complete solution (maimal subsolution and minimal supersolution) of { u + H(, Du) = in T c,. (BVP-I-L) u = on T. Thus V is the unique viscosit solution. Note that for the upper value functions T and W the same results are valid with H = H. We can give capturabilit conditions on the sstem ensuring V, Ṽ BUC(T c ). However, those conditions are less studied for games because there are important pursuit-evasion games with discontinuous value, the games with barriers (cfr.[i]). It is important to note that in general the upper and the lower values are different. However, the Isaacs condition H(, p) = H(, p), p, (11) guarantees that the coincide. Corollar 12 If V, Ṽ BUC(T c ), then V Ṽ, T T. If Isaacs condition holds then V = Ṽ and T = T, (i.e.the game has a value). 12

13 Proof Immediate from the comparison and uniqueness for (BVP-I-L). For numerical purposes, one can decide to write down an approimation scheme for either the upper or the lower value using the techniques of the net section. Before going to it, let us give some informations about the characterization of discontinuous value functions for games. This is an important issue because discontinuities appear even in classical pursuit-evasion games (e.g. in the homicidal chauffeur game that we will present and solve in Section 5). We will denote b B(Ω) the set of bounded real functions defined on Ω. Let us start with a definition which has been successfull applied to conve Hamiltonians. Definition 2.5 (Discontinuous Viscosit Solutions) Let H(, u, ) be conve. We define, u () = lim inf u(), u () = lim sup u() We sa that u B(Ω) is a viscosit solution if φ C 1 (Ω), the following conditions are satisfied: 1. at ever local maimum point for u φ, H(, u ( ), φ( )). 2. at ever local minimum point for u φ, H(, u ( ), φ( )). As we have seen, to obtain uniqueness one should prove that a comparison principle holds, i.e.for ever subsolution w and supersolution W we have w W Although this is sufficient to get uniqueness in the conve case the above definition will not guarantee uniqueness for nonconve hamiltonians (e.g.min-ma Hamiltonians). Two new definitions have been proposed. Let us denote b S the set of subsolutions of our equation and b Z the set of supersolutions alwas satisfing the Dirichlet boundar condition on Ω. Definition 2.6 (minma solutions [Su 6]) u is a minma solution if there eists two sequences w n S and W n Z such that w n = W n = on Ω (12) w n is continuous on Ω (13) and lim n w n () = u() = lim n W n (), Ω. (14) Definition 2.7 (e-solutions, see e.g.[bcd]) u is an e-solution (envelope solution) if there eists two non empt subsets S(u) S Z(u) Z 13

14 such that Ω u() = sup w() = inf W () w S(u) W Z(u) B the comparison Lemma there eists a unique e-solution. In fact, if u and v are two e-solutions, and also u() = v() = sup w() inf W () = v() w S(u) W Z(v) sup w() inf W () = u() w S(v) W Z(u) It is interesting to note that in our problem the two definitions coincide. Theorem 13 Under our hpotheses, u is a minma solution if and onl if u is an e- solution. 3 Numerical approimation We will describe a method to construct approimation schemes for the Isaacs equation where we tr to keep the essential informations of the game/control problem which is behind it. In this approach, the numerical approimation of the first order PDE is based on a time-discretization of the original control problem via discrete DP principle. Then, the functional equation for the time-discrete problem is projected on a grid to derive a finite dimensional fied point problem. Naturall, one can also choose to construct directl an approimation scheme for the Isaacs equation based on classical methods for hperbolic PDE, e.g.using a Finite Difference (FD) scheme. However this choice is not simpler from the point of view of implementation since it is well known that an up-wind correction is needed in the scheme to keep stabilit and obtain a converging scheme (cfr.[str]). Moreover, proving convergence of the scheme b onl PDE arguments is sometimes more complicated. The dnamic programming schemes which we present in this section have a built-in up-wind correction and convergence can be proved also using control arguments. 3.1 Time discretization Let us start b the time discretization of the minimum time problem. The fact that there is onl one plaer makes easier to describe the scheme and to introduce the basic ideas which are behind this approach. In the previous section, we have seen how one can obtain b the Kružkov change of variable (8) a characterization of the (rescaled) value function as the unique viscosit solution of (BVP-B). To build the approimation let us choose a time step h = t > for the dnamical sstem and define the discrete times t m = mh, m N. We can obtain a discrete dnamical sstem associated to (D) just using an onestep scheme for the Cauch problem. A well known eample is the eplicit Euler scheme which corresponds to the following discrete dnamical sstem { m+1 = m + hf( m, a m ) (D = h ) 14

15 We will denote b (n; {a m }) the state at time nh of the discrete time trajector verifing (D h ). Define the discrete analogue of the reachable set R h { R N : a sequence {a m } and m N such that m T } (15) and n h ({a m }, ) = { + / Rh min{m N : m T } R h (16) N h () = min {a m} n h({a m }, ), (17) Thus the discrete analogue of the minimum time function T ( ) is N h ( )h. Lemma 14 (Discrete Dnamic Programming Principle) Let h > be fied. For all R h, n < N h () (so that / T ), hn h () = inf {a m} { hn + N h( (n; {a m })) }. (DDPP) Sketch of the proof. The inequalit is eas since hn h is clearl lower than an choice which makes n steps on the dnamics and then is optimal starting from n. For the reverse inequalit, b definition, for ever ɛ > there eists a sequence {a ɛ n} such that hn h () > n h ({a ɛ m }, ) ɛ > hn + N h( (n; a ɛ m )) ɛ (18) Since ɛ is arbitrar, this concludes proof. As in the continuous problem, we appl the Kružkov change of variable v h () = 1 e h N h(). (19) Note that, b definition, v h 1 and v h has constant values on the set of initial points which can be driven to T b the discrete dnamical sstem in the same number of steps (of constant width h). Writing the discrete Discrete Dnamic Programming Principle for n = 1, and changing variable we get the following characterization of v h where v h () = S(v h )() on R N \ T (HJB h ) v h () = on T (BC h ) [ ] S(v h )() min e h v h ( + hf(, a)) + 1 e h (2) a A In fact, note that R c h RN \ R implies + hf(, a) R c h so we can easil etend v h to R c h just defining v h () = 1 on R c h and get rid of R h finall setting (HJB h ) on R N \ T. 15

16 Theorem 15 v h is the unique bounded solution of (HJB h ) (BC h ). Sketch of the proof. The proof directl follows from the fact that S defined in (2) is a contraction map in L (R N ). In fact, one can easil prove (see [BF 1] for details) that for ever R N S(u)() S(w)() e h u w, for an u, w L (R N ). (21) In order to prove an error bound for the approimation we need to introduce some assumptions which are the discrete analogue of the local controllabilit assumptions of Section 1. Let us define the δ-neighbourhood of T T δ T + δb(, 1), and d() dist (, T ) We are now able to prove the following upper bounds for our approimations Lemma 16 Under our assumptions on f and STLC, there eist some positive constants h, δ such that v h () C d() + h, h < h, T δ Theorem 17 Let the assumptions of Lemma 16 be satisfied and let T be compact with nonempt interiour. Then, v h converges to v locall uniforml in R N for h + Sketch of the proof. Since v h is a discontinuous function we define the two semicontinuous envelopes v = lim inf h + v h (), v = lim sup v h () h + Note that v (respectivel v) is lower (respectivel upper) semicontinuous. The first step is to show that 1. v is a viscosit subsolution for (HJB) 2. v is a viscosit supersolution for (HJB) Then, we want show that both the envelopes satisf the boundar condition on T. In fact, b Lemma 16, v h () C d() + h which implies v C d() (22) v C d() (23) 16

17 so we have v = v = on T Since the two envelopes coincide on T we can appl the comparison theorem for semicontinuous sub and supersolutions in [BP] and obtain v = v = v on R N. We now want to prove an error estimate for our discrete time approimation. Let us assume Q is a compact subset of R where the following condition holds: C > : Q there is a time optimal control with total variation less than C bringing the sstem to T. (BV) Theorem 18 ([BF 2]) Let the assumptions of Lemma 16 be verified and let Q be a compact subset of R where (BV) holds. Then there eists two positive constants h and C such that v() v h () Ch Q, h h (E) The above results show that the rate of convergence for the scheme based on the Euler scheme is 1, which is eactl what we epected. Now let us go back to games. The same time discretization can be written for the dnamics and natural etensions of N h and v h are easil obtained. The crucial point is to prove that the discrete dnamic programming principle holds true and that the upper value of the discrete game is the unique solution in L (R N ) of the eternal Dirichlet problem v h () = S(v h )() on R N \ T (HJI h ) v h () = on T (BC h ) where the fied point operator now is [ ] S(v h )() ma min e h v h ( + hf(, a, b)) + 1 e h b B a A The net step is to show that the discretization (HJI h ) (BC h ) is convergent to the upper value of the game. A detailed presentation goes beond the purposes of this paper, the interested reader will find these results in [BS 3]. We just give the main convergence result for the continuous case. Theorem 19 Let v h be the solution of (HJI h ) (BC h ), Let T be compact with nonempt interiour, the assumptions on f be verified, v be continuous.then, v h converges to v locall uniforml in R N for h +. 17

18 3.2 Space discretization (1 plaer) In order to solve the problem numericall we need a (finite) grid so we have to restrict our problem to a compact subdomain. A tpical choice is to replace the whole space with a hpercube containing T. We consider a triangular mesh of Q made b triangles S j, j J denoting b k the size of the mesh (this means that k = ma j {diam(s j )}. Let us just remark that one can alwas decide to build a structured grid for Q as is the case for FD schemes, although for dnamic programming schemes this is not compulsor. Moreover, the use of triangles instead of uniform rectangular cells can be a clever and better choice when the boundar of T is rather complicated. We will denote b i, the nodes of the mesh (the vertices of the triangles S j ), tpicall i is a multi-inde, i = (i 1,..., i N ) where N is the dimension of the state space of the problem. We denote b L the global number of nodes. To simplif the presentation let us take a rectangle Q in R 2, Q T and present the scheme for N = 2. Moreover, we map the matri of the values at the nodes (where v i1,i 2 is the value corresponding to the node i1,i 2 ) on to a vector V of dimension L b the usual representation b rows where the element v i1,i 2 goes into V m for m = (i 1 1)N columns + i 2. This ordering allows us to locate the nodes and their values b a single inde so from now on i I {1,..., L}. We will divide the nodes into three subsets, the algorithm will perfom different operations in the three subsets. Let us introduce the sets of indices I T {i I : i T } (24) I out {i I : i + hf( i, a) / Q a A} (25) I in {i I : i + hf( i, a) Q} (26) We can describe the full discrete scheme simpl writing (HJB h ) at ever node of the grid such that i I in adding the boundar conditions on T and on Q (more precisel, on the part of the boundar where the vectorfield points outward for ever control). The full discrete scheme is v( i ) = min a A [βv( i + hf( i, a)] + 1 β, for i I in (27) v( i ) = for i I T (28) v( i ) = 1 for i I out (29) where β = e h. Note that the condition on I out assigns to those nodes a value greater than the maimum value inside Q \ T. It is like saing that once the trajector leaves Q it will never come back to T (which is obviousl false). Nonetheless the condition is reasonable since we will never get the information that the real trajector (living in the whole space) can get back to the target unless we compute the solution in a larger domain containing Q. The solution we compute in Q is correct onl if I out is empt, if this is not the case the solution is correct in a subdomain of Q and is greater than the real solution everwhere in Q. This means that the reachable set is approimated from the inside. We look for a solution of the above problem in the space of piecewise linear functions W k {w : Q [, 1] : w C(Q), w = constant in S j } (3) 18

19 For an i I in, there eists at least one control such that z i (a) i + hf( i, a) Q. Associated to z i (a) Q there is a unique vector of coefficients, λ i j (z i(a)), i, j I such that L λ i j (a) 1, λ i j (z i(a)) = 1 and z i (a) = j=1 L λ i j (z i(a)) j The coefficients λ i j (z i(a)) are the local (baricentric) coordinates of the point z i (a) with respect to the vertices of the triangle containing z i (a). The above conditions just sa that z can be written in a unique wa as a conve combination of the nodes. Since we are looking for a solution in W k, it is important to note that for an w W k, w(z i (a)) = j λi j w j where w j = w( j ), j = 1,..., L. For z / Q we set w(z) = 1. Let us define componentwise the operator S : R L R L corresponding to the full discrete scheme [S(U)] i j=1 min a A [βλi (a) U] + 1 β, i I in i I T (31) 1 i I out where Λ i (a) (λ i 1 (z i(a)),..., λ i L (z i(a))) The following theorem shows that S has a unique fied point. Theorem 2 The operator S defined in (31) has the following properties: i) S is monotone, i.e.u V implies S(U) S(V ); ii) S : [, 1] L [, 1] L ; iii) S is a contraction mapping in the ma norm W = ma i W i, Sketch of the proof. S(U) S(V ) β U V i) To prove that S is monotone it suffices to show that for U V, S(U) i S(V ) i, for i I in. In fact, for an i I in, we have S(U) i S(V ) i βλ i (â) (U V ) where â is the control where the minimum for S(V ) is achieved. The proof just follows b the fact that all the λ i j are nonnegative. ii) Then, for an U [, 1] L 1 β = S i () S i (U) S i (1) = 1, i I in 19

20 where 1 (1, 1,..., 1). This concludes the proof. iii). For an i I in S i (U) S i (V ) βλ i (â)(u V ) and Λ i (â) 1, a A which implies S i (U) S(V ) β U V. Although one can compute the fied point starting from an initial guess U it is more efficient to start from an initial guess in the set of discrete supersolutions U +, U + {U [, 1] L : U S(U)} This will guarantee monotone convergence. In fact, let us consider the fied point sequence Taking U U +, the monotonicit of S implies U n+1 S(U n ) (32) U S(U ) = U 1, U 1 U 2, U 2 U 3... and U n converges to U monotonicall decreasing b the fied point argument. A tpical choice for U U + is { Ui i IT = 1 elsewhere It is also interesting to remark that in the algorithm the information flows from the target to the other nodes of the grid. In fact, on the nodes in Q \ T we have Ui = 1 but these values immediatel decrease in a neighbourhood of T since, b the local controllabilit assumption, the Euler scheme drives them to the target in just one step. At the net iteration other values, in a larger neighbourhood of T, will decrease due to the same mechanism and so on. 3.3 Full discrete scheme for games Let us get back to zero-sum games. Using the change of variable v() 1 e T () we can set the Isaacs equation in R N obtaining { v() + min v() = b B ma a A [ f(, a, b) v()] = 1 in RN \ T for T (HJI) Assume we want to solve the equation in Q, an hpercube in R N. As we have seen, in the case of a single plaer we need to impose boundar conditions on Q or, at least, on I out. However, the situation for games is much more complicated. In fact, setting the value of the solution outside Q equal to 1 (as in the single plaer case) will impl that the pursuer looses ever time the evader drives the dnamics outside Q. On the contrar, setting the value to outside Q will give a great advantage to the pursuer. One wa to define more unbiased boundar conditions is the following. Assume that Q = Q 1 Q 2, 2

21 where Q i, i = 1, 2 are subsets of R N/2 which can be interpreted as the set of constraints for the i-th plaer. For eample, in R 2 we can consider as Q 1 a vertical strip and as Q 2 an horizontal strip and compute in the rectangle Q which is the intersection of those strips. According to this construction, we penalize the pursuer if the dnamics eits Q 1 and the evader if the dnamics eits Q 2. When the dnamics eits Q 1 and Q 2 we have assign a value, e.g.giving an advantage to one of them (in the following scheme we are giving advantage to the evader). The discretization in time and space leads to a full discrete scheme where β e h and w( i ) = ma min[βw( i + hf( i, a, b))] + 1 β for i I b a in (33) w( i ) = 1 for i I out2 (34) w( i ) = for i I T I out1 (35) I in = {i : i + hf( i, a, b) Q \ T for an a A, b B} (36) I T = {i : i T Q} (37) I out1 = {i : i / Q 2 } (38) I out2 = {i : i / Q 2 \ Q} (39) Theorem 21 The operator S defined in (33) has the following properties: i) S is monotone, i.e.u V implies S(U) S(V ); ii) S : [, 1] L [, 1] L ; iii) S is a contraction mapping in the ma norm, S(U) S(V ) β U V The proof of the above theorem is a generalization of that of Theorem 2 and can be found in [BFS1]. The above result guarantees that there is a unique fied point U for S. Naturall the numerical solution w will be obtained etending b linear interpolation the values of U and it will depend on the discretization steps h = t and k =. Let us state the first convergence result for continuous value functions Theorem 22 Let T be the closure of an open set with Lipschitz boundar, diam Q + and v be continuous. Then, for h + and k h +, w h,k converges to v locall uniforml on the compact sets of R N. Note that the requirement diam Q + is just a technical trick to avoid to deal with boundar conditions on Q (a similar statement can be written in the whole space just working on an infinite mesh). We conclude this section quoting a convergence result which holds also in presence of discontinuities (barriers) for the values function. Let w ɛ n be the sequence generated b the numerical scheme with target T ɛ = { : d(, T ) ɛ}. 21

22 Theorem 23 For all there eists the limit w() = lim w ε + n ɛ () n + n n(ɛ) and it coincides with the lower value V of the game with target T, i.e.w = V. Convergence is uniform on ever compact set where V is continuous. Can we know a-priori what is the accurac of the method in the approimation of the value function? This result is necessar to understand how far we are from the real solution when we compute our approimate solution. To simplif, let us assume that the Lipschitz constant for f L f 1 and that v is Lipschitz continuous. Then, ( ) ) k 2 w h,k v Ch (1 1/2 + h The proof of the above error estimate is rather technical and can be found in [S 4]. 4 Approimation of optimal feedback and trajectories One of the goals of ever approimation for control problems and games is to compute discrete snthesis of feedback controls. It is interesting to note that the algorithm proposed and analzed in the previous section computes an approimate optimal control at ever point of the grid. Since the numerical solution w has been etended to Q b interpolation we can also compute an approimate optimal feedback at ever point of Q, i.e.for the control problem we can define the feedback map F : Q A. In order to construct this map, let us introduce the notation I k (, a) e h w( + hf(, a)) + 1 e h. (4) where we indicate b the inde k the fact that the above function also depends on the space discretization. Note that I k (, ) has a minimum over A, but the minimum point ma be not unique. We want to construct a selection, e.g. take a strictl conve φ and define A k = {â A : I k (, â) = min A Ik (, a)} (41) The selection is a = arg min A k φ(a) (42) In this wa we are able to compute our approimate optimal trajectories, we define the piecewise constant control a k (s) = a m,h s [mh, (m + 1)h[ (43) where m,h is the state of the Euler scheme, at the iteration m. Error estimates of the approimation of feedbacks and optimal trajectories are available for control problems in [BCD] [F 1, F 2]. 22

23 For games, the algorithm computes an approimate optimal control couple (a, b ) at ever point of the grid. Again b w we can also compute an approimate optimal feedback at ever point Q. (a (), b ()) argminma{e h w( + hf(, a, b))} + 1 e h (44) If that control is not unique then we can select a unique couple, e.g.minimizing two conve functionals. A tpical choice is to introduce an inertial criterium to stabilize the trajectories, i.e.if at step n + 1 the set of optimal couples contains (a n, b n ) we keep it. We end this section giving some informations on recent developments which we cannot include in this presentation. Some acceleration methods have been implemented to reduce the amount of floating point operation needed to compute the fied point: Gauss-Seidel iterations, monotone acceleration methods [St] and approimation in polic space [TG 2, St]. Moreover, an effort has been made to reduce the size of the problem b means of the domain decomposition technique, see [QV] for a general presentation of the method and several applications to PDEs. This approach has produced parallel codes for the Isaacs equation [FLM, St, FSt]. 5 Numerical eperiments Let us eamine some classical games and look at their numerical solutions. We will focus our attention to the accurac in the approimation of the value function as well as to the accurac in the approimation of optimal feedbacks and trajectories. In the previous sections we alwas assumed that the sets of controls A and B were compact. In the algorithm and in the numerical tests we have used a discrete finite approimation for those sets which allows to compute the min-ma b comparison. For eample, we will consider the following discrete sets { A = a 1 + j a 2 a } 1, j =,..., c 1; { B = c 1 b 1 + j b 2 b 1 c 1 }, j =,..., c 1; where [a 1, a 2 ] and [b 1, b 2 ] represent the control sets respectivel for the pursuer P and the evader E. Finall, note that all the value functions represented in the pictures have values in [, 1] because we have computed the fied point after the Kružkov change of variable. 5.1 The Tag-Chase Game Two bos P and E are running one after the other in the plane R 2. P wants to catch E in minimal time whereas E wants to avoid the capture. Both of them are running with constant velocit and can change their direction instantaneousl. This means that the dnamics of the sstem is f P (, a, b) = v P a f E (, a, b) = v E b where v P and v E are two scalars representing the maimum speed for P and E and the admissible controls are taken in the sets A = B = B(, 1). 23

24 Let us give a more eplicit version of the dnamics which is useful for the discretization. Let us denote b ( P, P ) the position of P and b ( E, E ) the position of E, we can write the dnamics as ẋ P = v P sin P ẏ P = v P cos P ẋ E = v E sin E ẏ E = v E cos E (45) where P [a 1, a 2 ] [ π, π] is the control for P and E [b 1, b 2 ] [ π, π] is the control for E, P and E are the angles between the ais and the velocities for P and E (see Figure 5.1). We sa that E has been captured b P if their distance in the plane is lower than a given threshold ɛ >. Introducing z ( P, P, E, E ) we can sa that the capture occurs whenever z T where { T z R 4 : } ( P E ) 2 + ( P E ) 2 < ɛ. (46) The Isaacs equation is set in R 4 since ever plaer belongs to R 2. However the result of the game just depends on the relative positions of P and E, since their dnamics are homogeneous. In order to reduce the amount of computations needed to compute the value function we describe the game in a new coordinate sstem introducing the variables = ( E P ) cos ( E P ) sin (47) ỹ = ( E P ) sin ( E P ) cos (48) The new sstem (called relative coordinates sstem) has the origin fied on the position of P and moves with this plaer (see Figure 5.1). Note that the ais is oriented from P to E. In the new coordinates the dnamics (45) becomes { = v E sin E v P sin P = v E cos E v P cos P (5) (49) and the target (46) is T { (, ) : } < ɛ. It is important to note that the above change of variables greatl simplifies the numerical solution of the problem for three different reasons. The first is that we now solve the Isaacs equation in R 2 and we need a grid of just M 2 nodes instead of M 4 nodes (here M denotes the number of nodes in one dimension). The second reason is that we now have a compact target in R 2 whereas the original target (46) is unbounded. This is a major advantage since we can choose a fied rectangular domain Q in R 2 such that it contains T and compute the solution in it. Finall, we get rid of the boundar conditions on Q (see Section 3). It is easil seen that the game has alwas a value and that the onl interesting case is v P > v E (if the opposite inequalit holds true capture is impossible if E plas optimall). In this situation the best strateg for E is to run at maimal velocit in the direction 24

25 opposite to P along the line passing through the initial positions of P and E. The optimal strateg for P is to run after E at maimal velocit. The corresponding minimal time of capture is (E P ) T ( P, P, E, E ) = 2 + ( E P ) 2 v P v E or, in relative coordinates, T (, ) = v P v E. Let us comment some numerical eperiments. We have chosen Q = [ 1, 1] 2 v P = 2, v E = 1, A = B = [ π, π]. Figures 2, 3 correspond to the following discretization # Nodes t ɛ # Controls P=41 E=41 The value function is represented in the relative coordinate sstem, so P is fied at the origin and the value at ever point is the minimal time of capture (after Kružkov transform). As one can see in Figure 2, the behaviour is correct since it correspond to a (rescaled) distance function. The optimal trajectories for the initial positions P = (.3,.3), E = (.6,.3) are represented in Figure The Tag-Chase game with constraints on the directions This game has the dnamics (45). The onl difference with respect to the Tag-Chase game is that now the pursuer P has a constraint on his displacement directions. He can choose his control in the set P [a 1, a 2 ] [ 3/4π, 3/4π]. The evader can still choose his control as E [b 1, b 2 ] = [ π, π], i.e. A = [ 1, 2 ] and B = [ π, π]. In the numerical eperiment below we have chosen v P = 2, v E = 1, A = [ 3 4 π, 3 4π] and B = [ π, π]. As one can see in Figure 4 the time of capture at points which are below the origin and which cannot be reached b P in a direct wa have a value bigger than at the smmetric points (above the origin). This is clearl due to the fact that P has to zig-zag to those points because the directions pointing directl to them are not allowed (Figure 5). 5.3 Zermelo navigation problem A boat B moves with constant velocit in a river and it can change its direction istantaneousl. The water of the river flows with a velocit σ and the boat tries to reach an island in the middle of the river (the target) maneuvering against water current. We choose a sstem of coordinates such that the velocit of the current is (σ, ) (see Figure 6). In the new sstem, the dnamics of the boat is described b { ẋ = σ + v B cos a, ẏ = v B sin a, 25