c 2011 Society for Industrial and Applied Mathematics

SIAM J. CONTROL OPTIM. Vol. 49, No. 4, pp. 1829 1856 c 211 Society for Inustrial an Applie Mathematics DISCRETE HAMILTON JACOBI THEORY TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK Abstract. We evelop a iscrete analogue of Hamilton Jacobi theory in the framewor of iscrete Hamiltonian mechanics. The resulting iscrete Hamilton Jacobi equation is iscrete only in time. We escribe a iscrete analogue of Jacobi s solution an also prove a iscrete version of the geometric Hamilton Jacobi theorem. The theory applie to iscrete linear Hamiltonian systems yiels the iscrete Riccati equation as a special case of the iscrete Hamilton Jacobi equation. We also apply the theory to iscrete optimal control problems, an recover some well-nown results, such as the Bellman equation iscrete-time HJB equation) of ynamic programming an its relation to the costate variable in the Pontryagin maximum principle. This relationship between the iscrete Hamilton Jacobi equation an Bellman equation is exploite to erive a generalize form of the Bellman equation that has controls at internal stages. Key wors. Hamilton Jacobi equation, Bellman equation, ynamic programming, iscrete mechanics, iscrete-time optimal control AMS subject classifications. 7H2, 49L2, 93C55, 49J15, 7H5, 7H25 DOI. 1.1137/9776822 1. Introuction. 1.1. Discrete mechanics. Discrete mechanics is a reformulation of Lagrangian an Hamiltonian mechanics with iscrete time, as oppose to a iscretization of the equations in the continuous-time theory. It not only provies a systematic view of structure-preserving integrators but also has interesting theoretical aspects analogous to continuous-time Lagrangian an Hamiltonian mechanics see, e.g., Marsen an West [3] an Suris [34, 35]). The main feature of iscrete mechanics is its use of iscrete versions of variational principles. Namely, iscrete mechanics assumes that the ynamics is efine at iscrete times from the outset, formulates a iscrete variational principle for such ynamics, an then erives a iscrete analogue of the Euler Lagrange or Hamilton s equations from it. The avantage of this construction is that it naturally gives rise to iscrete analogues of the concepts an ieas in continuous time that have the same or similar properties, such as symplectic forms, the Legenre transformation, momentum maps, an Noether s theorem [3]. This in turn provies us with the iscrete ingreients that facilitate further theoretical evelopments, such as iscrete analogues of the theories of complete integrability see, e.g., Moser an Veselov [31] an Suris [34, 35]) an also those of reuction an connections [2, 26, 25]. Whereas the main topic in iscrete mechanics is the evelopment of structure-preserving algorithms for Lagrangian an Hamiltonian systems see, e.g., [3]), the theoretical aspects of it are interesting in their own right, an furthermore provie insight into the numerical aspects as well. Another notable feature of iscrete mechanics, especially on the Hamiltonian sie, is that it is a generalization of regular) iscrete optimal control problems. In fact, Receive by the eitors November 11, 29; accepte for publication in revise form) May 16, 211; publishe electronically August 11, 211. This wor was partially supporte by NSF grants DMS-6437, DMS-726263, DMS-97949, an DMS-11687. http://www.siam.org/journals/sicon/49-4/77682.html Department of Mathematics, University of California, San Diego, 95 Gilman Drive, La Jolla, California 9293 112 tohsawa@ucs.eu, mleo@math.ucs.eu). Department of Mathematics, University of Michigan, 53 Church Street, Ann Arbor, Michigan 4819 143 abloch@umich.eu). 1829 Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

183 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK as state in [3], iscrete mechanics is inspire by iscrete formulations of optimal control problems see, e.g., Joran an Pola [21] an Cazow [9]). 1.2. Hamilton Jacobi theory. In classical mechanics see, e.g., Lanczos [24], Arnol [3], Marsen an Ratiu [29], an Golstein, Poole, an Safo [16]), the Hamilton Jacobi equation is first introuce as a partial ifferential equation that the action integral satisfies. Specifically, let Q be a configuration space an T Q be its cotangent bunle; let q Q an t> be arbitrary an suppose that ˆqs), ˆps)) T Q is a solution of Hamilton s equations 1.1) q = H p, ṗ = H q with the enpoint conition ˆqt) = q. Then, calculate the action integral along the solution over the time interval [,t], i.e., 1.2) Sq, t) := t [ ˆps) ˆqs) ] Hˆqs), ˆps)) s, where we regar the resulting integral as a function of the enpoint q, t) Q R + with R + being the set of positive real numbers. Then, by taing variation of the enpoint q, t), one obtains a partial ifferential equation satisfie by Sq, t): S 1.3) t + H q, S ) =. q This is the Hamilton Jacobi H-J) equation. Conversely, it is shown that if Sq, t) is a solution of the Hamilton Jacobi equation, then Sq, t) is a generating function for the family of canonical transformations or symplectic flows) that escribe the ynamics efine by Hamilton s equations. This result is the theoretical basis for the powerful technique of exact integration calle separation of variables. 1.3. Connection with optimal control an the Hamilton Jacobi Bellman equation. The iea of Hamilton Jacobi theory is also useful in optimal control theory see, e.g., Jurjevic [22, Chapter 13] an Bertseas [6, Section 3.2]). Consier a typical optimal control problem T min Cq, u) t, u ) subject to the constraints, q = fq, u), an q) = q an qt )=q T. We efine the augmente cost functional: T T ] Ŝ[u] := {Cq, u)+p[ q fq, u)]} t = [p q Ĥq, p, u) t, where we introuce the costate p, an also efine the control Hamiltonian, Ĥq, p, u) := p fq, u) Cq, u). Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1831 Assuming that Ĥ q, p, u) = u uniquely efines the optimal control u = u q, p), we set Hq, p) := maxĥq, p, u) =Ĥq, p, u u q, p)). We also efine the optimal cost-to-go function T { ]} Jq, t) := Cˆq,u )+p[ ˆq fˆq, u ) s t T [ = ˆp ˆq ] Hˆq, ˆp) s = S Sq, t), t where ˆqs), ˆps)) for s [,T] is the solution of Hamilton s equations with the above H such that ˆqt) =q; S is the optimal cost S := T [ ˆp ˆq ] Hˆq, ˆp) s = an the function Sq, t) is efine by Sq, t) := t T [ [ ˆp ˆq ] Ĥˆq, ˆp, u ˆq, ˆp)) s = Ŝ[u ], ˆp ˆq ] Hˆq, ˆp) s. Since this efinition coincies with 1.2), the function Sq, t) =S Jq, t) satisfies the H-J equation 1.3); this reuces to the Hamilton Jacobi Bellman HJB) equation for the optimal cost-to-go function Jq, t): [ ] J J 1.4) t +min fq, u)+cq, u) =. u q It can also be shown that the costate p of the optimal solution is relate to the solution of the HJB equation. 1.4. Discrete Hamilton Jacobi theory. The main objective of this paper is to present a iscrete analogue of Hamilton Jacobi theory within the framewor of iscrete Hamiltonian mechanics of Lall an West [23], an also to apply the theory to iscrete optimal control problems. There are some previous wors on iscrete-time analogues of the Hamilton Jacobi equation, such as Elnatanov an Schiff [13] an Lall an West [23]. Specifically, Elnatanov an Schiff [13] erive an equation for a generating function of a coorinate transformation that trivializes the ynamics. This erivation is a iscrete analogue of the conventional erivation of the continuous-time Hamilton Jacobi equation see, e.g., Lanczos [24]). Lall an West [23] formulate a iscrete Lagrangian analogue of the Hamilton Jacobi equation as a separable optimization problem. 1.5. Main results. Our wor was inspire by the result of Elnatanov an Schiff [13] an starts from a reinterpretation of their result in the language of iscrete mechanics. This paper further extens the result by eveloping iscrete analogues Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1832 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK Fig. 1. Discrete evolution equations left) an corresponing iscrete Hamilton Jacobi-type equations right). Dashe lines are the lins establishe in the paper. of results in continuous-time) Hamilton Jacobi theory. Namely, we formulate a iscrete analogue of Jacobi s solution, which relates the iscrete action sum to a solution of the iscrete Hamilton Jacobi equation. This also provies a very simple erivation of the iscrete Hamilton Jacobi equation an exhibits a natural corresponence with the continuous-time theory. Another important result in this paper is a iscrete analogue of the Hamilton Jacobi theorem, which relates the solution of the iscrete Hamilton Jacobi equation with the solution of the iscrete Hamilton s equations. We also show that the iscrete Hamilton Jacobi equation is a generalization of the iscrete Riccati equation an the Bellman equation see Figure 1). Specifically, we show that the iscrete Hamilton Jacobi equation applie to linear iscrete Hamiltonian systems an iscrete optimal control problems reuces to the iscrete Riccati an Bellman equations, respectively. This is again a iscrete analogue of the well-nown results that the Hamilton Jacobi equation applie to linear Hamiltonian systems an optimal control problems reuces to the Riccati see, e.g., Jurjevic [22, p. 421]) an HJB equations see section 1.3 above), respectively. The lin between the iscrete Hamilton Jacobi equation an the Bellman equation turns out to be useful in eriving a class of generalize Bellman equations that are higher-orer approximations of the original continuous-time problem. Specifically, we use the iea of the Galerin Hamiltonian variational integrator of Leo an Zhang [28] to erive iscrete control Hamiltonians that yiel higher-orer approximations, an then show that the corresponing iscrete Hamilton Jacobi equation gives a class of Bellman equations with controls at internal stages. 1.6. Outline of the paper. We first present a brief review of iscrete Lagrangian an Hamiltonian mechanics in section 2. In section 3, we escribe a reinterpretation of the result of Elnatanov an Schiff [13] in the language of iscrete mechanics an a iscrete analogue of Jacobi s solution to the iscrete Hamilton Jacobi equation. The remainer of section 3 is evote to more etaile stuies of the iscrete Hamilton Jacobi equation: its left an right variants, more explicit forms of them, an also a igression on the Lagrangian sie. In section 4, we prove a iscrete version of the Hamilton Jacobi theorem. In section 5, we apply the theory to linear iscrete Hamiltonian systems, an show that the iscrete Riccati equation follows from the iscrete Hamilton Jacobi equation. Section 6 establishes the lin with Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1833 iscrete-time optimal control an interprets the results of the preceing sections in this setting. Section 7 further extens this iea to erive a class of Bellman equations with controls at internal stages. 2. Discrete mechanics. This section briefly reviews some ey results of iscrete mechanics following Marsen an West [3] an Lall an West [23]. 2.1. Discrete Lagrangian mechanics. A iscrete Lagrangian flow {q } N =,on an n-imensional ifferentiable manifol Q, can be escribe by the following iscrete variational principle: Let S N be the following action sum of the iscrete Lagrangian L : Q Q R: 2.1) S N {q } N =) := N 1 = L q,q +1 ) tn Lqt), qt)) t, which is an approximation of the action integral as shown above. Consier iscrete variations q q +εδq,for =, 1,...,N, with δq = δq N =. Then, the iscrete variational principle δs N = gives the iscrete Euler Lagrange equations: 2.2) D 2 L q 1,q )+D 1 L q,q +1 )=. This etermines the iscrete flow F L : Q Q Q Q: 2.3) F L :q 1,q ) q,q +1 ). Let us efine the iscrete Lagrangian symplectic one-forms Θ ± L : Q Q T Q Q) by 2.4a) 2.4b) Θ + L :q,q +1 ) D 2 L q,q +1 ) q +1, Θ L :q,q +1 ) D 1 L q,q +1 ) q. Then, the iscrete flow F L preserves the iscrete Lagrangian symplectic form 2.5) Ω L q,q +1 ) := Θ ± L = D 1 D 2 L q,q +1 ) q q +1. Specifically, we have F L ) Ω L =Ω L. 2.2. Discrete Hamiltonian mechanics. Introuce the right an left iscrete Legenre transforms FL ± : Q Q T Q by 2.6a) 2.6b) FL + :q,q +1 ) q +1,D 2 L q,q +1 )), FL :q,q +1 ) q, D 1 L q,q +1 )), respectively. Then, we fin that the iscrete Lagrangian symplectic forms in 2.4) an 2.5) are pullbacs by these maps of the stanar symplectic forms on T Q: Let us efine the momenta Θ ± L =FL ± ) Θ, Ω ± L =FL ± ) Ω. p,+1 := D 1L q,q +1 ), p +,+1 := D 2L q,q +1 ). Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1834 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK Then, the iscrete Euler Lagrange equations 2.2) become simply p + 1, = p,+1. So, efining p := p + 1, = p,+1, one can rewrite the iscrete Euler Lagrange equations 2.2) as follows: 2.7) p = D 1 L q,q +1 ), p +1 = D 2 L q,q +1 ). Furthermore, efine the iscrete Hamiltonian map F L : T Q T Q by 2.8) FL :q,p ) q +1,p +1 ). Then, one may relate this map with the iscrete Legenre transforms in 2.6) as follows: 2.9) FL = FL + FL ) 1. Furthermore, one can also show that this map is symplectic, i.e., F L ) Ω=Ω. This is the Hamiltonian escription of the ynamics efine by the iscrete Euler Lagrange equation 2.2) introuce by Marsen an West [3]. Notice, however, that no iscrete analogue of Hamilton s equations is introuce here, although the flow is now on the cotangent bunle T Q. Lall an West [23] pushe this iea further to give iscrete analogues of Hamilton s equations: From the point of view that a iscrete Lagrangian is essentially a generating function of type one [16], we can apply Legenre transforms to the iscrete Lagrangian to fin the corresponing generating functions of type two or three [16]. In fact, they turn out to be a natural Hamiltonian counterpart to the iscrete Lagrangian mechanics escribe above. Specifically, with the right iscrete Legenre transform 2.1) p +1 = FL + q,q +1 )=D 2 L q,q +1 ), we can efine the following right iscrete Hamiltonian: 2.11) H + q,p +1 )=p +1 q +1 L q,q +1 ). Then, the iscrete Hamiltonian map F L :q,p ) q +1,p +1 ) is efine implicitly by the right iscrete Hamilton s equations 2.12a) 2.12b) q +1 = D 2 H + q,p +1 ), p = D 1 H + q,p +1 ), which are precisely the characterization of a symplectic map in terms of a generating function, H +, of type two. Similarly, with the left iscrete Legenre transform 2.13) p = FL q,q +1 )= D 1 L q,q +1 ), we can efine the following left iscrete Hamiltonian: 2.14) H p,q +1 )= p q L q,q +1 ). Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1835 Then, we have the left iscrete Hamilton s equations 2.15a) 2.15b) q = D 1 H p,q +1 ), p +1 = D 2 H p,q +1 ), which correspon to a symplectic map expresse in terms of a generation function, H, of type three. On the other han, Leo an Zhang [28] emonstrate that iscrete Hamiltonian mechanics can be obtaine as a irect variational iscretization of continuous Hamiltonian mechanics, instea of having to go via iscrete Lagrangian mechanics. 3. Discrete Hamilton Jacobi equation. 3.1. Derivation by Elnatanov an Schiff. Elnatanov an Schiff [13] erive a iscrete Hamilton Jacobi equation base on the iea that the Hamilton Jacobi equation is an equation for a symplectic change of coorinates uner which the ynamics becomes trivial. In this section, we woul lie to reinterpret their erivation in the framewor of iscrete Hamiltonian mechanics reviewe above. Theorem 3.1. Suppose that the iscrete ynamics {q,p )} N = is governe by the right iscrete Hamilton s equations 2.12). Consier the symplectic coorinate transformation q,p ) ˆq, ˆp ) that satisfies the following: i) The ol an new coorinates are relate by the type-one generating function 1 S : R n R n R: 3.1) ˆp = D 1 S ˆq,q ), p = D 2 S ˆq,q ); ii) the ynamics in the new coorinates {ˆq, ˆp )} N = is renere trivial, i.e., ˆq +1, ˆp +1 )=ˆq, ˆp ). Then, the set of functions {S } N =1 satisfies the iscrete Hamilton Jacobi equation: 3.2) S +1 ˆq,q +1 ) S ˆq,q ) D 2 S +1 ˆq,q +1 ) q +1 + H + q,d 2 S +1 ˆq,q +1 ) ) =, or, with the shorthan notation S q ) := S ˆq,q ), 3.3) S +1 q +1 ) Sq ) DS +1 q +1 ) q +1 + H + q,ds +1 q +1 ) ) =. Proof. The ey ingreient in the proof is the right iscrete Hamiltonian in the new coorinates, i.e., a function Ĥ+ ˆq, ˆp +1 )thatsatisfies 3.4) ˆq +1 = D 2 Ĥ + ˆq, ˆp +1 ), ˆp = D 1 Ĥ + ˆq, ˆp +1 ), or equivalently, 3.5) ˆp ˆq +ˆq +1 ˆp +1 = Ĥ+ ˆq, ˆp +1 ). 1 This is essentially the same as 2.7) in the sense that they are both transformations efine by generating functions of type one: Replace q,p,q +1,p +1,L )byˆq, ˆp,q,p,S ). However, they have ifferent interpretations: 2.7) escribes the ynamics or time evolution, whereas 3.1) is a change of coorinates. Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1836 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK LetusfirstwriteĤ+ in terms of the original right iscrete Hamiltonian H+ an the generating function S. For that purpose, first rewrite 2.12) an 3.1) as follows: an p q = q +1 p +1 + H + q,p +1 ) ˆp ˆq = p q S ˆq,q ), respectively. Then, using the above relations, we have ˆp ˆq +ˆq +1 ˆp +1 =ˆp ˆq + ˆp +1 ˆq +1 ) ˆp +1 ˆq +1 = p q S ˆq,q )+ˆp +1 ˆq +1 ) p +1 q +1 + S +1 ˆq +1,q +1 ) = q +1 p +1 + H + q,p +1 ) S ˆq,q ) + ˆp +1 ˆq +1 ) p +1 q +1 + S +1 ˆq +1,q +1 ) = H + q,p +1 )+ˆp +1 ˆq +1 p +1 q +1 + S +1 ˆq +1,q +1 ) S ˆq,q ) ). Thus, in view of 3.5), we obtain 3.6) Ĥ + ˆq, ˆp +1 )=H + q,p +1 )+ˆp +1 ˆq +1 p +1 q +1 +S +1 ˆq +1,q +1 ) S ˆq,q ). Now consier the choice of the new right iscrete Hamiltonian Ĥ+ that reners the ynamics trivial, i.e., ˆq +1, ˆp +1 )=ˆq, ˆp ). It is clear from 3.4) that we can set Ĥ + ˆq, ˆp +1 )=ˆp +1 ˆq. Then, 3.6) becomes ˆp +1 ˆq = H + q,p +1 )+ˆp +1 ˆq +1 p +1 q +1 + S +1 ˆq +1,q +1 ) S ˆq,q ), an since ˆq +1 =ˆq = =ˆq,wehave =H + q,p +1 ) p +1 q +1 + S +1 ˆq,q +1 ) S ˆq,q ). Eliminating p +1 by using 3.1), we obtain 3.2). Remar 3.2. What Elnatanov an Schiff [13] refer to as the Hamilton Jacobi ifference equation is the following: 3.7) S +1 ˆq,q +1 ) S ˆq,q ) D 2 S +1 ˆq,q +1 ) D 2 H + q,p +1 )+H + q,p +1 )=. It is clear that this is equivalent to 3.2) in view of 2.12) 3.2. Discrete analogue of Jacobi s solution. This section presents a iscrete analogue of Jacobi s solution. This also gives an alternative erivation of the iscrete Hamilton Jacobi equation that is much simpler than the one shown above. Theorem 3.3. Consier the action sums, 2.1), written in terms of the right iscrete Hamiltonian, 2.11): 1 3.8) S q [ ) := pl+1 q l+1 H + q l,p l+1 ) ] l= Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1837 evaluate along a solution of the right iscrete Hamilton s equations 2.12); each S q ) is seen as a function of the enpoint coorinates q an the iscrete en time. Then, these action sums satisfy the iscrete Hamilton Jacobi equation 3.3). Proof. From 3.8), we have 3.9) S +1 q +1 ) S q )=p +1 q +1 H + q,p +1 ), where p +1 is consiere to be a function of q an q +1, i.e., p +1 = p +1 q,q +1 ). Taing the erivative of both sies with respect to q +1,wehave DS +1 q +1 )=p +1 + p +1 [q +1 D 2 H + q q,p +1 ) ]. +1 However, the terms in the bracets vanish because the right iscrete Hamilton s equations 2.12) are assume to be satisfie. Thus, we have 3.1) p +1 = DS +1 q +1 ). Substituting this into 3.9) gives 3.3). Remar 3.4. Recall that, in the erivation of the continuous Hamilton Jacobi equation see, e.g., [15, Section 23]), we consier the variation of the action integral, 1.2), with respect to the enpoint q, t) an fin 3.11) S = pq Hq, p) t. This gives S = Hq, p), t p an hence the Hamilton Jacobi equation S t + H q, S q = S q, ) =. In the above erivation of the iscrete Hamilton Jacobi equation 3.3), the ifference of two action sums, 3.9), is a natural iscrete analogue of the variation S in 3.11). Notice also that 3.9) plays the same essential role as 3.11) oes in eriving the Hamilton Jacobi equation. Table 1 summarizes the corresponence between the ingreients in the continuous an iscrete theories. 3.3. The right an left iscrete Hamilton Jacobi equations. Recall that, in 3.8), we wrote the action sum, 2.1), in terms of the right iscrete Hamiltonian, 2.11). We can also write it in terms of the left iscrete Hamiltonian, 2.14), as follows: 1 3.12) Sq [ )= pl q l H p l,q l+1 ) ]. l= Then, we can procee as in the proof of Theorem 3.3: First, we have 3.13) S +1 q +1 ) S q )= p q H p,q +1 ), where p is consiere to be a function of q an q +1, i.e., p = p q,q +1 ). Taing the erivative of both sies with respect to q,wehave DS q )= p p q [q + D 1 H p,q +1 ) ]. Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1838 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK Table 1 Corresponence between ingreients in continuous an iscrete theories; R is the set of nonnegative real numbers an N is the set of nonnegative integers see also Remar 3.4). Continuous Discrete q, t) Q R q,) Q N q = H/ p, q +1 = D 2 H + q,p +1 ), ṗ = H/ q p = D 1 H + q,p +1 ) t 1 Sq, t) := [ps) qs) Hqs),ps))] s S q [ ] ) := p l+1 q l+1 H + q l,p l+1 ) l= S = S S q + q t t S+1 q +1 ) S q ) pq Hq, p) t p +1 q +1 H + q,p +1 ) S t + H q, S ) = q S +1 q +1 ) S q ) DS +1 q +1 ) q ) +1 +H + q,ds +1 q +1 ) = However, the terms in the bracets vanish because the left iscrete Hamilton s equations 2.15) are assume to be satisfie. Thus, we have 3.14) p = DS q ). Substituting this into 3.13) gives the iscrete Hamilton Jacobi equation with the left iscrete Hamiltonian: 3.15) S +1 q +1 ) Sq )+DSq ) q + H ) DS q ),q +1 =. We refer to 3.3) an 3.15) as the right an left iscrete Hamilton Jacobi equations, respectively. As mentione above, 3.8) an 3.12) are the same action sum, 2.1), expresse in ifferent ways. Therefore, we may summarize the above argument as follows. Proposition 3.5. The action sums, 3.8) or equivalently 3.12), satisfy both the right an left iscrete Hamilton Jacobi equations, 3.3) an 3.15), respectively. 3.4. Explicit forms of the iscrete Hamilton Jacobi equations. The expressions for the right an left iscrete Hamilton Jacobi equations in 3.3) an 3.15) are implicit in the sense that they contain two spatial variables q an q +1 ;Theorem 3.3 suggests that one may consier q an q +1 to be relate by the iscrete Hamiltonian ynamics efine by either the right or left iscrete Hamilton s equations 2.12) or 2.15), or equivalently, the iscrete Hamiltonian map F L :q,p ) q +1,p +1 ) efine in 2.8). More specifically, we may write q +1 in terms of q. This results in explicit forms of the iscrete Hamilton Jacobi equations, an we shall efine the iscrete Hamilton Jacobi equations by the resulting explicit forms. We will see later in section 6 that the explicit form is compatible with the formulation of the Bellman equation. For the right iscrete Hamilton Jacobi equation 3.3), we first efine the map f + : Q Q as follows: Replace p +1 in 2.12a) by DS +1 q +1 ) as suggeste by 3.1): q +1 = D 2 H + q,ds +1 q +1 ) ). Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1839 Assuming this equation is solvable for q +1, we efine f + : Q Q by f q )=q +1 ; i.e., f + is implicitly efine by 3.16) f + q )=D 2 H + q,ds +1 f + q )) ). We may now ientify q +1 with f + q ) in the implicit form of the right Hamilton Jacobi equation 3.3): 3.17) S +1 f + q)) S q) DS +1 f + q)) f + q)+h+ q, DS +1 f + q))) =, where we suppresse the subscript of q since it is now clear that q is an inepenent variable as oppose to a function of the iscrete time. We efine 3.17) to be the right iscrete Hamilton Jacobi equation. Notice that these are ifferential-ifferencefunctional equations efine on Q N, with the spatial variable q an the iscrete time. For the left iscrete Hamilton Jacobi equation 3.15), we efine the map f : Q Q as follows: 3.18) f q ) := π Q F L S q ) ), where π Q : T Q Q is the cotangent bunle projection; equivalently, f so that the iagram below commutes. is efine T Q F L T Q S q ) F L S q ) ) S π Q Q Q f q f q ) Notice also that, since the map F L :q,p ) q +1,p +1 ) is efine by 2.15), f is efine implicitly by 3.19) q = D 1 H DS q ),f q ) ). In other wors, replace p in 2.15a) by DS q ) as suggeste by 3.14), an efine f q )astheq +1 in the resulting equation. We may now ientify q +1 with f q ) in 3.15): 3.2) S +1 f q)) S q)+dsq) q + H DS q),f q)) =, where we again suppresse the subscript of q. We efine 3.17) an 3.2) to be the right an left iscrete Hamilton Jacobi equations, respectively. Remar 3.6. Notice that the right iscrete Hamilton Jacobi equation 3.17) is more complicate than the left one 3.2), particularly because the map f + appears more often than f oes in the latter; notice here that, as shown in 3.18), the maps f ± in the iscrete Hamilton Jacobi equations 3.17) an 3.2) epen on the function S, which is the unnown one has to solve for. However, it is possible to efine an equally simple variant of the right iscrete Hamilton Jacobi equation by writing q 1 in terms of q : Let us first efine g : Q Q by g q ) := π Q F 1 L S q ) ), Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

184 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK or so that the iagram below commutes. T Q F 1 L T Q F 1 L S q ) ) S q ) π Q Q g Q S g q ) q Now, in 3.3), change the inices from, +1)to 1,)anientifyq 1 with g q )toobtain Sq) S 1 g q)) DSq) q + H + g q),dsq) ) =, where we again suppresse the subscript of q. This is as simple as the left iscrete Hamilton Jacobi equation 3.2). However, the map g is, being bacwar in time, rather unnatural compare to f ±. Furthermore, as we shall see in section 6, in the iscrete optimal control setting, the map f ± is efine by a given function an thus the formulation with f ± will turn out to be more convenient. 3.5. The iscrete Hamilton Jacobi equation on the Lagrangian sie. First, notice that 2.1) gives 3.21) S +1 q +1 ) S q )=L q,q +1 ). This is essentially the Lagrangian equivalent of the iscrete Hamilton Jacobi equation 3.17) as Lall an West [23] suggest. Let us apply the same argument as above to obtain the explicit form for 3.21). Taing the erivative of the above equation with respect to q,wehave D 1 L q,q +1 ) q = S q ), an hence from the efinition of the left iscrete Legenre transform, 2.6b), Assuming that FL FL q,q +1 )=S q ). is invertible, we have q,q +1 )=FL ) 1 S q ) ) =: q,f L q )), where we efine the map f L below): : Q Q as follows see the commutative iagram 3.22) f L q ) := pr 2 FL ) 1 S q ) ), where pr 2 : Q Q Q is the projection to the secon factor, i.e., pr 2 q 1,q 2 )=q 2. Thus, eliminating q +1 from 3.21) an then replacing q by q, we obtain the iscrete Hamilton Jacobi equation on the Lagrangian sie: 3.23) S +1 f L q)) S q) =L q, f L q) ). Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1841 efine above in 3.18) as the com- The map f L efine in 3.22) is ientical to f mutative iagram below emonstrates. F L T Q T Q FL 33333333 ) 1 FL + 3 333333333 Q Q pr 1 pr 2 3 Q Q S f L,f π Q S q ) F L S q ) ) 3333333 3 q,f L 333333333 q )) 3 q f Lq ) The commutativity of the square in the iagram efines the f as we saw earlier, whereas that of the right-angle triangle on the lower left efines the f L in 3.22); note the relation F L = FL + FL ) 1 from 2.9). The map f L being ientical to f implies that the iscrete Hamilton Jacobi equations on the Hamiltonian an Lagrangian sies, 3.2) an 3.23), are equivalent. 4. Discrete Hamilton Jacobi theorem. The following gives a iscrete analogue of the geometric Hamilton Jacobi theorem by Abraham an Marsen [1, Theorem 5.2.4]. Theorem 4.1 iscrete Hamilton Jacobi). Suppose that S satisfies the right iscrete Hamilton Jacobi equation 3.17), anlet{c } N = Q be a set of points such that 4.1) c +1 = f + c ) for =, 1,...,N 1. Then,thesetofpoints{c,p )} N = T Q with 4.2) p := DS c ) is a solution of the right iscrete Hamilton s equations 2.12). Similarly, suppose that S satisfies the left iscrete Hamilton Jacobi equation 3.2), anlet{c } N = Q be a set of points that satisfy 4.3) c +1 = f c ) for =, 1,...,N 1. Furthermore, assume that the Jacobian Df set of points {c,p )} N = T Q with is invertible at each point c.then,the 4.4) p := DS c ) is a solution of the left iscrete Hamilton s equations 2.15). Proof. To prove the first assertion, first recall the implicit efinition of f + in 3.16): 4.5) f + q) =D 2H + q, DS +1 f + q))). In particular, for q = c,wehave 4.6) c +1 = D 2 H + c,p ), Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1842 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK where we use 4.1) an 4.2). On the other han, taing the erivative of 3.17) with respect to q, DS +1 f + q)) Df+ q) DS q) Df+ q) D2 S +1 f + q)) f + q) DS +1 f + q)) Df+ q)+d 1H + q, DS +1 f + q))) + D 2 H + q, DS +1 f + q))) D 2 S +1 f + q)) Df+ q) =, which reuces to DS q)+d 1 H + q, DS +1 f + q))) =, ue to 4.5). Then, substituting q = c gives DS c )+D 1 H + c,ds +1 f + c )) ) =. Using 4.1) an 4.2), we obtain 4.7) p = D 1 H + c,p +1 ). Equations 4.6) an 4.7) show that the sequence {c,p )} satisfies the right iscrete Hamilton s equations 2.12). Now, let us prove the latter assertion. First, recall the implicit efinition of f in 3.19): 4.8) q = D 1 H DS q),f q)). In particular, for q = c,wehave 4.9) c = D 1 H p,c +1 ), where we use 4.3) an 4.4). On the other han, taing the erivative of 3.17) with respect to q yiels DS +1 f q)) Df q) DS q)+d2 S q) q + DS q) + D 1 H DS q),f q)) D 2 S q)+d 2H DS q),f q)) Df q) =, which reuces to [ DS +1 f q)) + D 2H DS q),f q))] Df q) =, ue to 4.8). Then, substituting q = c gives DS +1 f c )) = D 2 H DS c ),f c ) ), since Df c ) is invertible by assumption. Then, using 4.3) an 4.4), we obtain 4.1) p +1 = D 2 H p,c +1 ). Equations 4.9) an 4.1) show that the sequence {c,p )} satisfies the left iscrete Hamilton s equations 2.15). Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1843 5. Application to iscrete linear Hamiltonian systems. 5.1. Discrete linear Hamiltonian systems an matrix Riccati equation. Example 5.1 quaratic iscrete Hamiltonian iscrete linear Hamiltonian systems). Consier a iscrete Hamiltonian system on T R n = R n R n the configuration space is Q = R n ) efine by the quaratic left iscrete Hamiltonian 5.1) H p,q +1 )= 1 2 pt M 1 p + p T Lq +1 + 1 2 qt +1Kq +1, where M, K, anl are real n n matrices; we assume that M an L are invertible an also that M an K are symmetric. The left iscrete Hamilton s equations 2.15) are q = M 1 p + Lq +1 ), p +1 = L T p + Kq +1 ), or 5.2) ) ) ) q+1 L 1 L = 1 M 1 q p +1 KL 1 KL 1 M 1 L T, p an hence are a iscrete linear Hamiltonian system see section A.1). Now, let us solve the left iscrete Hamilton Jacobi equation 3.2) for this system. For that purpose, we first generalize the problem to that with a set of initial points instea of a single initial point q,p ). More specifically, consier the set of initial points that is a Lagrangian affine space L z see Definition A.2), which contains the point z :=q,p ). Then, the ynamics is formally written as, for any iscrete time N, L ) := F L ) ) Lz = F L F ) L Lz, }{{} where F L : T Q T Q is the iscrete Hamiltonian map efine in 2.8). Since F L is a symplectic map, Proposition A.4 implies that L ) is a Lagrangian affine space. Then, assuming that L ) is transversal to {} Q, Corollary A.6 implies that there exists a set of functions S of the form 5.3) S q) =1 2 qt A q + b T q + c, such that L ) =graphs ;herea are symmetric n n matrices, b are elements in R n,anc are in R. Now that we now the form of the solution, we substitute the above expression into the iscrete Hamilton Jacobi equation to fin the equations for A, b,anc. Notice first that the map f is given by the first half of 5.2) with p replace by DS q): 5.4) f q) = L 1 q + M 1 DS q)) = L 1 I + M 1 A )q L 1 M 1 b. Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1844 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK Then, substituting 5.3) into the left-han sie of the left iscrete Hamilton Jacobi equation 3.2) yiels the following recurrence relations for A, b,anc : 5.5a) 5.5b) 5.5c) A +1 = L T I + A M 1 ) 1 A L K, b +1 = L T I + A M 1 ) 1 b, c +1 = c 1 2 bt M + A ) 1 b, whereweassumethati + A M 1 is invertible. Remar 5.2. For the A +1 efine by 5.5a) to be symmetric, it is sufficient that A is invertible; for if it is, then 5.5a) becomes A +1 = L T A 1 + M 1 ) 1 L K, an so A, M, ank being symmetric implies that A +1 is as well. Remar 5.3. We can rewrite 5.5a) as follows: 5.6) A +1 = [ KL 1 +KL 1 M 1 L T )A ] L 1 L 1 M 1 A ) 1. Notice the exact corresponence between the coefficients in the above equation an the matrix entries in the iscrete linear Hamiltonian equations 5.2). In fact, this is the iscrete Riccati equation that correspons to the iteration efine by 5.2). See Ammar an Martin [2] for etails on this corresponence. To summarize the above observation, we have the following proposition. Proposition 5.4. The iscrete Hamilton Jacobi equation 3.2) applie to the iscrete linear Hamiltonian system 5.2) yiels the iscrete Riccati equation 5.6). In other wors, the iscrete Hamilton Jacobi equation is a nonlinear generalization of the iscrete Riccati equation. 6. Relation to the Bellman equation. In this section, we apply the above results to the optimal control setting. We will show that the right) iscrete Hamilton Jacobi equation 3.17) gives the Bellman equation iscrete-time HJB equation) as a special case. This result gives a iscrete analogue of the relationship between the H-J an HJB equations iscusse in section 1.3. 6.1. Discrete optimal control problem. Let q := {q } N = be the state variables in a vector space V = R n with q an q N fixe an u := {u } N 1 = be controls in the set U R m. With a given function C : V U R, efine the iscrete cost functional J := N 1 = C q,u ). Then, we formulate the stanar iscrete optimal control problem as follows see, e.g., [21, 9, 4, 17]). Problem 6.1 stanar iscrete optimal control problem). Minimize the iscrete cost functional, i.e., 6.1) min u subject to the constraint, N 1 J =min u = C q,u ), 6.2) q +1 = f q,u ). Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1845 6.2. Necessary conition for optimality an the Bellman equation. We woul lie to formulate the necessary conition for optimality. First, introuce the augmente iscrete cost functional: N 1 Ŝ q,p,u ) := {C q,u )+p +1 [q +1 f q,u )]} = N 1 [ ] = p +1 q +1 Ĥ+ q,p +1,u ), = where we introuce the costate p := {p } N =1 with p V, an also efine the iscrete control Hamiltonian 6.3) Ĥ + q,p +1,u ) := p +1 f q,u ) C q,u ). Then, the optimality conition 6.1) is restate as min q,p,u Ŝ q,p,u )= min q,p,u N 1 [ ] p +1 q +1 Ĥ+ q,p +1,u ). = In particular, extremality with respect to the control u implies 6.4) D 3 Ĥ + q,p +1,u )=, =, 1,...,N 1. Now, we assume that Ĥ+ is sufficiently regular so that this equation uniquely etermines the optimal control u := {u }N 1 = ; an therefore, u is a function of q an p +1, i.e., u = u q,p +1 ). We then efine 6.5) H + q,p +1 ) := maxĥ + u q,p +1,u ) =max[p +1 f q,u ) C q,u )] u = p +1 f q,u ) C q,u ), an also the optimal iscrete cost-to-go function 6.6) where S an N 1 J q ) := {C q l,u l )+p l+1 [q l+1 f q l,u l )]} l= N 1 [ = pl+1 q l+1 H + q l,p l+1 ) ] l= = S Sq ), is the optimal iscrete cost functional, i.e., N 1 S := Ŝq,p,u [ )= p+1 q +1 H + q,p +1 ) ] = 1 S q [ ) := pl+1 q l+1 H + q l,p l+1 ) ]. l= Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1846 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK The above action sum has exactly the same form as 3.8) formulate in the framewor of iscrete Hamiltonian mechanics. Therefore, our theory now irectly applies to this case: The corresponing right iscrete Hamilton s equations 2.12) are, using the expression in 6.5), q +1 = f q,u ), p = p +1 D 1 f q,u ) D 1C q,u ). Therefore, 3.16) gives the implicit efinition of f + as follows: 6.7) f + q )=f q,u q,ds +1 f + q )) )). Hence, the right) iscrete Hamilton Jacobi equation 3.17) applie to this case gives S +1 f q,u )) S q ) DS +1 f q,u )) f q,u ) + H + q,ds +1 f q,u )) ) =, an again using the expression for the Hamiltonian in 6.5), this becomes [ max S +1 u f q,u )) C q,u ) ] = S q ). Since S q )=S J q ), we obtain [ 6.8) min J +1 u f q,u )) + C q,u ) ] = J q ), which is the Bellman equation see, e.g., Bellman [4, Chapter 2], [5] an Bertseas [6, Chapter 1]). Remar 6.2. Notice that the Bellman equation 6.8) is much simpler than the iscrete Hamilton Jacobi equations 3.17) an 3.2) because of the special form of the control Hamiltonian equation 6.5). Also, notice that, as shown in 6.7), the term f + q ) is written in terms of the given function f. See Remar 3.6 for comparison. 6.3. Relation between the iscrete H-J an Bellman equations an its consequences. Summarizing the observation mae above, we have the following proposition. Proposition 6.3. The right iscrete Hamilton Jacobi equation 3.17) applie to the Hamiltonian formulation of the stanar iscrete optimal control problem 6.1 gives the Bellman equation 6.8). This observation leas to the following well-nown fact. Proposition 6.4. Let J q ) be a solution to the Bellman equation 6.8). Then, the costate p in the iscrete maximum principle is given as follows: p = DJ c ), where c +1 = f c,u ) with the optimal control u. Proof. The proposition follows from a reinterpretation of Theorem 4.1 through Proposition 6.3 with the relation S q )=S J q ). 7. Generalize Bellman equation with internal-stage controls. In the previous section, we showe that the iscrete Hamilton Jacobi equation recovers the Bellman equation if we apply our theory to the Hamiltonian formulation of the stanar iscrete optimal control problem 6.1. In this section, we generalize the approach Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1847 to erive what may be consiere as higher-orer iscrete-time approximations of the HJB equation 1.4). Namely, we erive a class of iscrete control Hamiltonians that use higher-orer approximations a more general version of 6.3)) by employing the technique of Galerin Hamiltonian variational integrators introuce by Leo an Zhang [28]; an then, we apply our theory to obtain a class of generalize Bellman equations that have controls at internal stages. 7.1. Continuous-time optimal control problem. Letusfirstbrieflyreview the stanar formulation of continuous-time optimal control problems. Let q be the state variable in a vector space V = R n, q an q T fixe in V,anu be the control in the set U R m. With a given function C : V U R, efine the cost functional J := T Cqt),ut)) t. Then, we formulate the stanar continuous-time optimal control problem as follows. Problem 7.1 stanar continuous-time optimal control problem). Minimize the cost functional, i.e., T min J =min Cqt),ut)) t, u ) u ) subject to the constraints, q = fq, u), an q) = q an qt )=q T. A Hamiltonian structure comes into play with the introuction of the augmente cost functional: Ŝ := = T T {Cqt),ut)) + pt)[ qt) fqt),ut))]} t [pt) qt) Ĥqt),pt),ut)) ] t, where we introuce the costate pt) V, an also efine the control Hamiltonian, 7.1) Ĥq, p, u) := p fq, u) Cq, u). 7.2. Galerin Hamiltonian variational integrator. Recall, from [28, Section 2.2], that the exact right iscrete Hamiltonian is a type-two generating function for the original continuous-time Hamiltonian flow, efine by { } h 7.2) H +,ex q,p 1 )= ext p 1 q 1 [pt) qt) Hqt),pt))] t, q,p) C 1 [,h],t Q) q)=q,ph)=p 1 where h is the time step; C 1 [,h],t Q) is the set of continuously ifferentiable curves on T Q overthetimeinterval[,h]; an extremum is achieve for the exact solution of Hamilton s equations 1.1) that satisfy the specifie bounary conitions. Therefore, it requires the exact solution qt),pt)) to evaluate the the above integral, an so the exact iscrete Hamiltonian cannot be practically compute in general. The ey iea of Galerin Hamiltonian variational integrators [28] is to replace the set of curves C 1 [,h],t Q) by a certain finite-imensional space so as to obtain a computable expression for a iscrete Hamiltonian. Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1848 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK 7.3. Galerin iscrete control Hamiltonian. Here, we woul lie to apply the above iea to the control Hamiltonian 7.1) to obtain a iscrete control Hamiltonian. Let C s V ) be a finite-imensional space of curves efine by { } CV s ) := t w i ψ i t/h) t [,h], w i V for each i {1, 2,...,s}, with the basis functions {ψ i :[, 1] R} s. 1. Use the basis functions ψ i to approximate the velocity q over the interval [,h], qτh) q τh)= w i ψ i τ), where τ [, 1] an w i V for each i =1,...,s. 2. Integrate q t) over[,τh], to obtain the approximation for the position q, i.e., τh τ q τh)=q ) + w i ψ i t/h) t = q + h w i ψ i ρ) ρ, where we applie the bounary conition q ) = q. Applying the bounary conition q h) =q 1 at the other enpoint yiels 1 q 1 = q h) =q + h w i ψ i ρ) ρ = q + h B i w i, where B i := 1 ψ iτ) τ. Furthermore, we introuce the internal stages, c i 7.3) Q i w) := q c i h)=q + h w j ψ j τ) τ = q + h A i jw j, j=1 where A i j := c i ψ jτ) τ. 3. The exact iscrete control Hamiltonian Ĥ+,ex is efine as in 7.2): { } h ] Ĥ +,ex q,p 1 ; u )) := p 1 q 1 [p q Ĥq, p, u) t. ext q,p) C 1 [,h],v V ) qt )=q,pt 1)=p 1 Again this is practically not computable, an so we employ the following approximation: Use the numerical quarature formula 1 fρ) ρ b i fc i ) with constants b i,c i ) an the finite-imensional function space C s V )toconstruct the iscrete control Hamiltonian Ĥ+ q,p 1,U) as follows: Ĥ + q,p 1,U) := ext q C s V ) P i V { p 1 q h) h =ext w,p Kq,w,P,U,p 1 ), j=1 b i [ pc i h) q c i h) Ĥ Q i w),p i,u i)]} Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1849 wherewesetp i := pc i h)anu i := uc i h) an efine ) Kq,w,P,U,p 1 ) := p 1 q + h B i w i h b i P i w j ψ j c i ) Ĥ Q i w),p i,u i) j=1 ) = p 1 q + h B i w i h b i P i Mjw i j fq i w),u i ) + CQ i w),u i ), j=1 whereweefinemj i := ψ jc i ) an use the expression for the control Hamiltonian in 7.1); note that P i V an w i V for each i =1,...,s,anthatf taes values in V. In orer to obtain an expression for H + q,p 1,U), we first compute the stationarity conitions for Kq,w,P,U,p 1 ) uner the fixe bounary conition q,p 1 ): 7.4a) = Kq,w,P,U,p 1 ) w j 7.4b) = Kq,w,P,U,p 1 ) P i = hp 1 B j h b i [Mj i P i ha i j D 1 Ĥ q i w),p i,u i)], = hb i Mj i wj fq i w),u i ), j=1 for j =1,...,s. 4. By solving the 2ns stationarity conitions 7.4), we can express the parameters w an P in terms of q, p 1,anU, i.e., w = wq,u)anp = P q,p 1,U): In particular, assuming b j foreachj =1,...,s, 7.4b) gives w j Mj i = fqi w),u i ); this gives a set of ns nonlinear equations 2 satisfie by w = wq,u). 3 Therefore, we have Kq, wq,u),p,u,p 1 )=p 1 q + h B j w j q,u) h b i C Q i wq,u)),u i). j=1 Notice that the internal-stage momenta P i isappear when we substitute w = wq,u). Therefore, we obtain the following Galerin iscrete control Hamiltonian: Ĥ + q,p 1,U) := K q, wq,u), P ) q,p 1,U),U,p 1 7.5) = p 1 q + h B j w j q,u) h b i C Q i wq,u)),u i). j=1 2 Recall that w j V an f taes values in V. 3 Note from 7.3) that Q i is written in terms of q an w. Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

185 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK Fig. 2. Internal-stage controls {U i }s in the iscrete-time intervals between an +1. 7.4. The Bellman equation with internal-stage controls. The Galerin iscrete control Hamiltonian 7.5) gives 7.6) Ĥ + q,p +1,U 1,...,Us ) := p +1 f q,u 1,...,Us ) C q,u 1,...,Us ), with 7.7) f q,u 1,...,U s ) := q + h an 7.8) C q,u 1,...,U s ) := h w i q,u,...,u 1 ) s B i, b i C Q i wq,u 1,...,U)),U s ) i. This is a generalize version of 6.3) with internal-stage controls {U i}s as oppose to a single control u per time step see Figure 2). Now assume that 7.9) U i Ĥ + q,p +1,U 1,...,Us )=, i =1,...,s, is solvable for {U i}s,i to give the optimal internal-stage controls {U }s. Then, we may apply the same argument as in section 6: In particular, the right iscrete Hamilton Jacobi equation 3.17) applie to this case gives the following Bellman equation with internal-stage controls: [ 7.1) min J +1 U 1,...,U s f q,u 1,...,U)) s + C q,u 1,...,U) s ] = J q ). The following example shows that the stanar Bellman equation 6.8) follows as a special case. Example 7.2 the stanar Bellman equation). Let s = 1, an select ψ 1 τ) =1, b 1 =1, c 1 =. Then, we have B 1 =1,A 1 1 =,anm 1 1 = 1. Hence, 7.3) gives Q1 = q we set the enpoints q,q 1 )tobeq,q +1 ) here), an 7.4b) gives w 1 = fq,u 1 ). However, the control U 1 to [t,t + h] here): is efine as follows we shift the time intervals from [,h] U 1 := ut + c 1 h)=ut )=: u. Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1851 So, we have w 1 = fq,u ), an thus, 7.7) an 7.8) give an f q,u )=q + hfq,u ) C q,u )=hcq,u ), respectively. Notice that this approximation gives the forwar-euler iscretization of the stanar continuous-time optimal control problem 7.1 to yiel the stanar iscrete optimal control problem 6.1. In fact, the Bellman equation with internalstage controls, 7.1), reuces to the stanar Bellman equation 6.8): 7.11) min u [ J +1 f q,u )) + C q,u ) ] = J q ). Remar 7.3. Higher-orer approximations with any number of internal stages s are possible as long as 7.4b) is solvable for w. See [28] for various ifferent choices of iscretizations. 7.5. Application to the Heisenberg system. Let us now apply the above results to a simple optimal control problem to illustrate the result. Example 7.4 the Heisenberg system; see, e.g., Brocett [8] an Bloch [7]). Consier the following optimal control problem: For a fixe time T>, T 1 min u ),v ) 2 u2 + v 2 ) t, subject to the constraint, ẋ = u, ẏ = v, ż = uy vx. This is the stanar continuous-time optimal control problem 7.1 with V = R 3, U = R 2, q =x, y, z), an fx, y, z, u, v) = u v, Cx, y, z, u, v) = 1 2 u2 + v 2 ). uy vx If we apply the choice of the iscretization in Example 7.2, we have the stanar Bellman equation [ min S +1 u,v f q,u,v )) C q,u,v ) ] S q )=, with q :=x,y,z ), where f q,u,v )= x + hu y + hv z + h u y v x ), C q,u,v )= h 2 u2 + v 2 ). Now, if we choose s = 2, an select ψ 1 τ),ψ 2 τ)) = 1, cosπτ)), b =b 1,b 2 )= 1 2, 1 ), c =c 1,c 2 )=, 1). 2 Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

1852 TOMOKI OHSAWA, ANTHONY M. BLOCH, AND MELVIN LEOK Then, we have min u 1,v1,u2,v2 B =B 1,B 2 )=1, ), A = ) ) 1 1, M =. 1 1 1 Leo an Zhang [28, Example 4.4] show that this choice of iscretization correspons to the Störmer Verlet metho see, e.g., Marsen an West [3]). The Bellman equation with internal-stage controls, 7.1), then becomes [ J +1 f q,u 1 )),v1,u2,v2 + C q,u 1 )],v1,u2,v2 = J q ), where an f q,u 1 ),v1,u2,v2 = C q,u 1,v 1,u 2,v 2 z + h u 1 + u 2 2 x + h u 1 + u2 )/2 y + h v 1 + v2 )/2 y v1 + v2 2 x + u2 v1 u1 v2 4 [ ) h u 1 = ) 2 +u 2 )2 + v1 )2 +v 2 ] )2. 2 2 2 ) 8. Conclusion an future wor. We evelope a iscrete-time analogue of Hamilton Jacobi theory starting from the iscrete variational Hamiltonian mechanics formulate by Lall an West [23]. We reinterprete an extene the iscrete Hamilton Jacobi equation given by Elnatanov an Schiff [13] in the language of iscrete mechanics. Furthermore, we showe that the iscrete Hamilton Jacobi equation reuces to the iscrete Riccati equation with a quaratic Hamiltonian, an also that it specializes to the Bellman equation of ynamic programming if applie to stanar iscrete optimal control problems. These results are iscrete analogues of the corresponing nown results in the continuous-time theory. Application to iscrete optimal control also reveale that the iscrete Hamilton Jacobi theorem 4.1 specializes to a well-nown result in iscrete optimal control theory. We also use a Galerin-type approximation to erive Galerin iscrete control Hamiltonians. This technique gave an explicit formula for iscrete control Hamiltonians in terms of the constructs in the original continuous-time optimal control problem. By viewing the Bellman equation as a special case of the iscrete Hamilton Jacobi equation, we coul introuce the iscretization technique for iscrete Hamiltonian mechanics into the iscrete optimal control setting; this lea us to a class of Bellman equations with controls at internal stages. We are intereste in the following topics for future wor: Application to integrable iscrete systems: Theorem 4.1 gives a iscrete analogue of the theory behin the technique of solution by separation of variables, i.e., the theorem relates a solution of the iscrete Hamilton Jacobi equations with that of the iscrete Hamilton s equations. An interesting question then is whether or not separation of variables applies to integrable iscrete systems, e.g., iscrete rigi boies of Moser an Veselov [31] an various others iscusse by Suris [34, 35]. Development of numerical methos base on the iscrete Hamilton Jacobi equation: Hamilton Jacobi equation has been use to evelop structure integrators for Hamiltonian systems. Ge an Marsen [14] evelope a numerical Copyright by SIAM. Unauthorize reprouction of this article is prohibite.

DISCRETE HAMILTON JACOBI THEORY 1853 metho that preserves momentum maps an Poisson bracets of Lie Poisson systems by solving the Lie Poisson Hamilton Jacobi equation approximately. See also Channell an Scovel [11] an references therein) for a survey of structure integrators base on the Hamilton Jacobi equation. The present theory, being inherently iscrete in time, potentially provies a variant of such numerical methos. Extension to iscrete nonholonomic an Dirac mechanics: The present wor is concerne only with unconstraine systems. Extensions to nonholonomic an Dirac mechanics, more specifically iscrete-time versions of the nonholonomic Hamilton Jacobi theory [19, 12, 32, 1, 33] an Dirac Hamilton Jacobi theory [27], are another irection for future research. Relation to the power metho an iterations on the Grassmannian manifol: Ammar an Martin [2] establishe lins between the power metho, iterations on the Grassmannian manifol, an the Riccati equation. The iscussion on iterations of Lagrangian subspaces an its relation to the Riccati equation in sections 5.1 an A.2 is a special case of such lins. On the other han, Proposition 5.4 suggests that the iscrete Hamilton Jacobi equation is a generalization of the Riccati equation. We are intereste in exploring possible further lins implie by the generalization. Galerin iscrete optimal control problems: The Galerin iscrete control Hamiltonians may be consiere to be a means of formulating iscrete optimal control problems with higher-orer of approximation to a continuoustime optimal control problem. This iea generalizes the Runge Kutta iscretizations of optimal control problems see, e.g., Hager [18] an references therein). In fact, Leo an Zhang [28] showe that their metho recovers the symplectic-partitione Runge Kutta metho. Therefore, this approach is expecte to provie higher-orer structure-preserving numerical methos for optimal control problems. Appenix A. Discrete linear Hamiltonian systems. A.1. Discrete linear Hamiltonian systems. Suppose that the configuration space Q is an n-imensional vector space, an that the iscrete Hamiltonian H + or H is quaratic as in 5.1). Also assume that the corresponing iscrete Hamiltonian map F L :q,p ) q +1,p +1 ) is invertible. Then, the iscrete Hamilton s equations 2.12) or 2.15) reuce to the iscrete linear Hamiltonian system A.1) z +1 = A L z, where z R 2n is a coorinate expression for q,p ) Q Q an A L : Q Q Q Q is the matrix representation of the map F L in the same basis. Since F L is symplectic, it follows that A L is an 2n 2n symplectic matrix, i.e., A.2) A T L JA L = J, where the matrix J is efine by J := ) I I with I the n n ientity matrix. Copyright by SIAM. Unauthorize reprouction of this article is prohibite.