Numerical methods for nonlinear optimal control problems

Size: px

Start display at page:

Download "Numerical methods for nonlinear optimal control problems"

Julia Powers
6 years ago
Views:

1 Title: Name: Affil./Addr.: Numerical metods for nonlinear optimal control problems Lars Grüne Matematical Institute, University of Bayreut, Bayreut, Germany ( Numerical metods for nonlinear optimal control problems Summary. In tis article we describe te tree most common approaces for numerically solving nonlinear optimal control problems governed by ordinary differential equations. For computing approximations to optimal value functions and optimal feedback laws we present te Hamilton-Jacobi- Bellman approac. For computing approximately optimal open loop control functions and trajectories for a single initial value, we outline te indirect approac based on Pontryagin s Maximum Principles and te approac via direct discretization. Introduction Tis article concerns optimal control problems governed by nonlinear ordinary differential equations ẋ(t) = f(x(t), u(t)) (1) wit f : R R n R m R n. We assume tat for eac initial value x R n and measurable control function u( ) L (R, R m ) tere exists a unique solution x(t) = x(t, x, u( )) of (1) satisfying x(0, x, u( )) = x. Given a state constraint set X R n and a control constraint set U R m, a running cost g : X U R, a terminal cost F : X U and a discount rate δ 0, we consider te optimal control problem minimize u( ) U T (x) J T (x, u( )) (2)

2 2 were and J T (x, u( )) := T 0 e δs g(x(s, x, u( )), u(s))ds + e δt F (x(t, x, u( ))) x(s, x, u( )) X U T (x) := u( ) L (R, U) (4) for all s [0, T ] In addition to tis finite orizon optimal control problem, we also consider te (3) infinite orizon problem in wic T is replaced by, i.e., minimize J (x, u( )) (5) u( ) U (x) were and respectively. J (x, u( )) := 0 e δs g(x(s, x, u( )), u(s))ds (6) x(s, x, u( )) X U (x) := u( ) L (R, U) for all s 0, (7) Te term solving (2) (4) or (5) (7) can ave various meanings. First, te optimal value functions or V T (x) = inf u( ) U T (x) J T (x, u( )) V (x) = inf J (x, u( )) u( ) U (x) may be of interest. Second, and often more importantly, one would like to know te optimal control policy. Tis can be expressed in open loop form u : R U, in wic te function u depends on te initial value x and on te initial time wic we set to 0 ere. Alternatively, te optimal control can be computed in state and time dependent closed loop form, in wic a feedback law µ : R X U is sougt. Via u (t) = µ (t, x(t)), tis feedback law can ten be used in order to generate te time dependent optimal control function for all possible initial values. Since te feedback law is evaluated along

3 3 te trajectory, it is able to react to perturbations and uncertainties wic may make x(t) deviate from te predicted pat. Finally, knowing u or µ one can reconstruct te corresponding optimal trajectory by solving ẋ(t) = f(x(t), u (t)) or ẋ(t) = f(x(t), µ (t, x(t))). Hamilton-Jacobi-Bellman approac In tis section we describe te numerical approac to solving optimal control problems via Hamilton-Jacobi-Bellman equations. We first describe ow tis approac can be used in order to compute approximations to te optimal value function V T and V, respectively, and afterwards ow te optimal control can be syntesized using tese approximations. In order to formulate tis approac for finite orizon T, we interpret V T (x) as a function in T and x. We denote differentiation w.r.t. T and x wit subscript T and x, i.e., V T x (x) = dv T (x)/dx, V T T (x) = dv T (x)/dt etc. We define te Hamiltonian of te optimal control problem as H(x, p) := max{ g(x, u) p f(x, u)}, u U wit x, p R n, f from (1), g from (3) or (6) and denoting te inner product in R n. Ten, under appropriate regularity conditions on te problem data, te optimal value functions V T and V satisfy te first order partial differential equations (PDEs) V T T (x) + δv T (x) + H(x, V T x (x)) = 0 and δv (x) + H(x, V x (x)) = 0 in te viscosity solution sense. In te case of V T, te equation olds for all T 0 wit te boundary condition V 0 (x) = F (x).

4 4 Te framework of viscosity solutions is needed because in general te optimal value functions will not be smoot, tus a generalized solution concept for PDEs must be employed, see Bardi and Capuzzo Dolcetta (1997). Of course, appropriate boundary conditions are needed at te boundary of te state constraint set X. Once te Hamilton-Jacobi-Bellman caracterization is establised, one can compute numerical approximations to V T or V by solving tese PDEs numerically. To tis end, various numerical scemes ave been suggested, including various types of finite element and finite difference scemes. Among tose, semi-lagrangian scemes (Falcone (1997) or Falcone and Ferretti (2013)) allow for a particularly elegant interpretation in terms of optimal control syntesis, wic we explain for te infinite orizon case. In te semi-lagrangian approac, one takes advantage of te fact tat by te cain rule for p = Vx (x) and constant control functions u te identity δv (x) p f(x, u) = d dt (1 δt)v (x(t, x, u)) t=0 olds. Hence, te left and side of tis equality can be approximated by by te difference quotient V (x) (1 δ)v (x(, x, u)) for small > 0. Inserting tis approximation into te Hamilton-Jacobi-Bellman equation, replacing x(, x, u) by a numerical approximation x(, x, u) (in te simplest case te Euler metod x(, x, u) = x + f(x, u)), multiplying by and rearranging terms, one arrives at te equation V (x) = min{g(x, u) + (1 δ)v ( x(, x, u))} u U defining an approximation V V. Tis is now a purely algebraic dynamic programming type equation wic can be solved numerically, e.g., by using a finite element approac. Te equation is typically solved iteratively using a suitable minimization

5 5 routine for computing te min in eac iteration (in te simplest case U is discretized wit finitely many values and te minimum is determined by direct comparison). We denote te resulting approximation of V by Ṽ. Here, approximation is usually understood in te L sense, see Falcone (1997) or Falcone and Ferretti (2013). Te semi-lagrangian sceme is appealing for syntesis of an approximately optimal feedback because V is te optimal value function of te auxiliary discrete time problem defined by x. Tis implies tat te expression µ (x) := argmin{g(x, u) + (1 δ)v ( x(, x, u))}, u U is an optimal feedback control value for tis discrete time problem for te next time step, i.e., on te time interval [t, t + ) if x = x(t). Tis feedback law will be approximately optimal for te continuous time control system wen applied as a discrete time feedback law and tis approximate optimality remains true if we replace V in te definition of µ by its numerically computable approximation Ṽ. A similar construction can be made based on any oter numerical approximation Ṽ V, but te explicit correspondence of te semi-lagrangian sceme to a discrete time auxiliary system facilitates te interpretation and error analysis of te resulting control law. Te main advantage of te Hamilton-Jacobi-approac is tat it directly computes an approximately optimal feedback law. Its main disadvantage is tat te number of grid nodes needed for maintaining a given accuracy in a finite element approac to compute Ṽ in general grows exponentially wit te state dimension n. Tis fact known as te curse of dimensionality restricts tis metod to low dimensional state spaces. Unless special structure is available wic can be exploited, as, e.g., in te maxplus approac, see McEneaney (2006), it is currently almost impossible to go beyond state dimensions of about n = 10, typically less for strongly nonlinear problems.

6 Maximum Principle approac 6 In contrast to te Hamilton-Jacobi-Bellman approac, te approac via Pontryagin s Maximum Principle does not compute a feedback law. Instead, it yields an approximately open loop optimal control u togeter wit an approximation to te optimal trajectory x for a fixed initial value. We explain te approac for te finite orizon problem. For simplicity of presentation, we omit state constraints in our presentation, i.e., we set X = R n and refer to, e.g., Vinter (2000), Bryson and Ho (1975) or Grass et al (2008) for more general formulations as well as for rigorous versions of te following statements. In order to state te Maximum Principle (wic, since we are considering a minimization problem ere, could also be called Minimum Principle) we define te non-minimized Hamiltonian as H(x, p, u) = g(x, u) + p f(x, u). Ten, under appropriate regularity assumptions tere exists an absolutely continuous function p : [0, T ] R n suc tat te optimal trajectory x and te corresponding optimal control function u for (2) (4) satisfy ṗ(t) = δp(t) H x (x (t), p(t), u (t)) (8) wit terminal or transversality condition p(t ) = F x (x (T )) (9) and u (t) = argmin H(x (t), p(t), u), (10) u U for almost all t [0, T ], see Grass et al (2008), Teorem 3.4. Te variable p is referred to as te adjoint or costate variable.

7 For a given initial value x 0 R n, te numerical approac now consists of finding functions x : [0, T ] R n, u : [0, T ] U and p : [0, T ] R n satisfying 7 ẋ(t) = f(x(t), u(t)) (11) ṗ(t) = δp(t) H x (x(t), p(t), u(t)) (12) u(t) = argmin H(x(t), p(t), u) (13) u U x(0) = x 0, p(t ) = F x (x(t )) (14) for t [0, T ]. Depending on te regularity of te underlying data te conditions (11) (14) may only be necessary but not sufficient for x and u being an optimal trajectory x and control function u, respectively. However, usually x and u satisfying tese conditions are good candidates for te optimal trajectory and control, tus justifying te use of tese conditions for te numerical approac. If needed, optimality of te candidates can be cecked using suitable sufficient optimality conditions for wic we refer to, e.g., Maurer (1981) or Malanowski et al (2004). Due to te fact tat in te Maximum Principle approac first optimality conditions are derived wic are ten discretized for numerical simulation, it is also termed first optimize ten discretize. Solving (11) (14) numerically amounts to solving a boundary value problem, because te condition x (0) = x 0 is posed at te beginning of te time interval [0, T ] wile te condition p(t ) = F x (x (T )) is required at te end. In order to solve suc a problem, te simplest approac is te single sooting metod wic proceeds as follows: We select a numerical sceme for solving te ordinary differential equations (11) and (12) for t [0, T ] wit initial conditions x(0) = x 0, p(0) = p 0 and control function u(t). Ten, we proceed iteratively as follows: (0) Find initial guesses p 0 0 R n and u 0 (t) for te initial costate and te control, fix ε > 0 and set k := 0

8 (1) Solve (11) and (12) numerically wit initial values x 0 and p k 0 and control function u k. Denote te resulting trajectories by x k (t) and p k (t). (2) Apply one step of an iterative metod for solving te zero finding problem G(p) = 0 wit G(p k 0) := p k (T ) F x ( x k (T )) for computing p k+1 0. For instance, in case of te Newton metod we get 8 p k+1 0 := p k 0 DG(p k 0) 1 G(p k 0). If p k+1 0 p k 0 < ε stop; else compute set k := k + 1 and go to (1). u k+1 (t) := argmin H(x k (t), p k (t), u), u U Te procedure described in tis algoritm is called single sooting because te iteration is performed on te single initial value p k 0. For an implementable sceme, several details still need to be made precise, e.g., ow to parameterize te function u(t) (e.g., piecewise constant, piecewise linear or polynomial), ow to compute te derivative DG and its inverse (or an approximation tereof) and te argmin in (2). Te last task considerably simplifies if te structure of te optimal control, e.g., te number of switcings in case of a bang-bang control, is known. However, even if all tese points are settled, te set of initial guesses p 0 0 and u 0 for wic te metod is going to converge to a solution of (11) (14) tends to be very small. One reason for tis is tat te solutions of (11) and (12) typically depend very sensitively on p 0 0 and u 0. In order to circumvent tis problem, multiple sooting can be used. To tis end, one selects a time grid 0 = t 0 < t 1 < t 2 <... < t N = T and in addition to p k 0 introduces variables x k 1,..., x k N 1, pk 1,..., p k N 1 Rn. Ten, starting from initial guesses p 0 0, u 0 and x 0 1,..., x 0 N 1, p0 1,..., p 0 N 1, in eac iteration te equations

9 9 (11) (14) are solved numerically on te intervals [t j, t j+1 ] wit initial values x k j and p k j, respectively. We denote te respective solutions in te k-t iteration by x k j and p k j. In order to enforce tat te trajectory pieces computed on te individual intervals [t j, t j+1 ] fit togeter continuously, te map G is redefined as G(x k 1,..., x k N 1, pk 0, p k 1,..., p k N 1 ) = x k 0(t 1 ) x k 1. x k N 2 (t 1) x k N 1 p k 0(t 1 ) p k. 1. p k N 2 (t 1) p k N 1 p k N 1 (T ) F x( x k N 1 (T )) Te benefit of tis approac is tat te solutions on te sortened time intervals depend muc less sensitively on te initial values and te control, tus making te problem numerically muc better conditioned. Te obvious disadvantage is tat te problem becomes larger as te function G is now defined on a muc iger dimensional space but tis additional effort usually pays off. Wile te convergence beavior for te multiple sooting metod is considerably better tan for single sooting, it is still a difficult task to select good initial guesses x 0 j, p 0 j and u 0. In order to accomplis tis, omotopy metods can be used, see, e.g., Pesc (1994) or te result of a direct approac as presented in te next section can be used as an initial guess. Te latter can be reasonable as te Maximum Principle based approac can yield approximations of iger accuracy tan te direct metod. In te presence of state constraints or mixed state and control constraints te conditions (12) (14) become considerably more tecnical and tus more difficult to be implemented numerically, cf. Pesc (1994).

10 Direct discretization 10 Despite being te most straigtforward and simple of te approaces described in tis article, te direct discretization approac is currently te most widely used approac for computing single finite orizon optimal trajectories. In te direct approac we first discretize te problem and ten solve a finite dimensional nonlinear optimization problem (NLP), i.e., we first discretize, ten optimize. Te main reason for te popularity of tis approac are te simplicity wit wic constraints can be andled and te numerical efficiency due to te availability of fast and reliable NLP solvers. Te direct approac again applies to te finite orizon problem and computes an approximation to a single optimal trajectory x (t) and control function u (t) for a given initial value x 0 X. To tis end, a time grid 0 = t 0 < t 1 < t 2 <... < t N = T and a set U d of control functions wic are parametrized by finitely many values are selected. Te simplest way to do so is to coose u(t) u j U for all t [t i, t i+1 ]. However, oter approaces like piecewise linear or piecewise polynomial control functions are possible, too. We use a numerical algoritm for ordinary differential equations in order to approximately solve te initial value problems ẋ(t) = f(x(t), u i ), x(t i ) = x i (15) for i = 0,..., N 1 on [t i, t i+1 ]. We denote te exact and numerical solution of (15) by x(t, t i, x i, u i ) and x(t, t i, x i, u i ), respectively. Finally, we coose a numerical integration rule in order to compute an approximation I(t i, t i+1, x i, u i ) ti+1 t i e δt g(x(t, t i, x i, u), u(t))dt. In te simplest case, one migt coose x as te Euler sceme and I as te rectangle rule, leading to x(t i+1, t i, x i, u i ) = x i + (t i+1 t i )f(x i, u i )

11 11 and I(t i, t i+1, x i, u i ) = (t i+1 t i )e δt i g(x i, u i ). Introducing te optimization variables u 0,..., u N 1 R m and x 1,..., x N R n, te discretized version of (2) (4) reads subject to te constraints N 1 minimize I(t i, t i+1, x i, u) + e δt F (x N ) x j R n,u j R m i=0 u j U, j = 0,..., N 1 x j X, x j+1 = x(t j+1, t j, x j, u), j = 1,..., N j = 0,..., N Tis way, we ave converted te optimal control problem (2) (4) into a finite dimensional nonlinear optimization problem (NLP). As suc, it can be solved wit any numerical metod for solving suc problems. Popular metods are, for instance, sequential quadratic programming (SQP) or interior point (IP) algoritms. Te convergence of tis approac was proved in Malanowski et al (1998), for an up to date account on teory and practice of te metod see Gerdts (2012) and Betts (2010). Tese references also explain ow information about te costates p(t) can be extracted from a direct discretization, tus linking te approac to te Maximum Principle. Te direct metod sketced ere is again a multiple sooting metod and te benefit of tis approac is te same as for solving boundary problems: tanks to te sort intervals [t i, t i+1 ] te solutions depend muc less sensitively on te data tan te solution on te wole interval [0, T ], tus making te iterative solution of te resulting discretized NLP muc easier. Te price to pay is again te increase of te number of optimization variables. However, due to te particular structure of te constraints guaranteeing continuity of te solution, te resulting matrices in te NLP ave a par-

12 12 ticular structure wic can be exploited numerically by a metod called condensing, see Bock and Plitt (1984). An alternative to multiple sooting metods are collocation metods, in wic te internal variables of te numerical algoritm for solving (15) are also optimization variables. However, nowadays te multiple sooting approac as described above is usually preferred. For a more detailed description of various direct approaces see also Binder et al (2001), Section 5. Furter approaces for infinite orizon problems Te last two approaces only apply to finite orizon problems. Wile te Maximum Principle approac can be generalized to infinite orizon problems, te necessary conditions become weaker and te numerical solution becomes considerably more involved, see Grass et al (2008). Bot te Maximum Principle and te direct approac can, owever, be applied in a receding orizon fasion, in wic an infinite orizon problem is approximated by te iterative solution of finite orizon problems. Te resulting control tecnique is known under te name of Model Predictive control (MPC, see Grüne and Pannek (2011)) and under suitable assumptions a rigorous approximation result can be establised. Summary and Future Directions Te tree main numerical approaces to optimal control are te Hamilton-Jacobi-Bellman approac, wic provides a global solution in feedback form but is computationally expensive for iger dimensional systems te Pontryagin Maximum Principle approac wic computes single optimal trajectories wit ig accuracy but needs good initial guesses for te iteration

13 13 te direct approac wic also computes single optimal trajectories but is less demanding in terms of te initial guesses at te expense of a somewat lower accuracy Currently, te main trends in numerical optimal control lie in te areas of Hamilton- Jacobi-Bellman equations and direct discretization. For te former, te development of discretization scemes suitable for increasingly iger dimensional problems are in te focus. For te latter, te popularity of tese metods in online applications like MPC triggers continuing effort to make tis approac faster and more reliable. Beyond ordinary differential equations, te development of numerical algoritms for te optimal control of partial differential equations (PDEs) as attracted considerable attention during te last years. Wile many of tese metods are still restricted to linear systems, in te near future we can expect to see many extensions to (classes of) nonlinear PDEs. It is wort noting tat for PDEs Maximum Principle-like approaces are more popular tan for ordinary differential equations. Cross References <<link to "Optimal control and Pontryagin s maximum principle", Ricard Vinter>> <<link to "Optimal control and te dynamic programming principle", Maurizio Falcone>> <<link to "Optimization algoritms for Model predictive control", Moritz Diel>> <<link to "Nominal model predictive control", Lars Grüne>> <<link to "Economic model predictive control", David Angeli>> <<link to "Discrete optimal control", David Martin De Diego>>

14 References 14 Bardi M, Capuzzo Dolcetta I (1997) Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman equations. Birkäuser, Boston Betts JT (2010) Practical metods for optimal control and estimation using nonlinear programming, 2nd edn. SIAM, Piladelpia Binder T, Blank L, Bock HG, Bulirsc R, Damen W, Diel M, Kronseder T, Marquardt W, Sclöder JP, von Stryk O (2001) Introduction to Model Based Optimization of Cemical Processes on Moving Horizons. In: Grötscel M, Krumke SO, Rambau J (eds) Online Optimization of Large Scale Systems: State of te Art, Springer-Verlag, Heidelberg, pp Bock HG, Plitt K (1984) A multiple sooting algoritm for direct solution of optimal control problems. In: Proceedings of te 9t IFAC World Congress Budapest, Pergamon, Oxford, pp Bryson AE, Ho YC (1975) Applied optimal control. Hemispere Publising Corp. Wasington, D.C., revised printing Falcone M (1997) Numerical solution of dynamic programming equations. Appendix A in Bardi, M. and Capuzzo Dolcetta, I., Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations, Birkäuser, Boston Falcone M, Ferretti R (2013) Semi-Lagrangian approximation scemes for linear and Hamilton-Jacobi equations. SIAM, Piladelpia Gerdts M (2012) Optimal control of ODEs and DAEs. de Gruyter Textbook, Walter de Gruyter & Co., Berlin Grass D, Caulkins JP, Feictinger G, Tragler G, Berens DA (2008) Optimal control of nonlinear processes. Springer-Verlag, Berlin Grüne L, Pannek J (2011) Nonlinear Model Predictive Control. Teory and Algoritms. Springer-Verlag, London

15 15 Malanowski K, Büskens C, Maurer H (1998) Convergence of approximations to nonlinear optimal control problems. In: Matematical programming wit data perturbations, Lecture Notes in Pure and Appl. Mat., vol 195, Dekker, New York, pp Malanowski K, Maurer H, Pickenain S (2004) Second-order sufficient conditions for state-constrained optimal control problems. J Optim Teory Appl 123(3): Maurer H (1981) First and second order sufficient optimality conditions in matematical programming and optimal control. Mat Programming Stud 14: McEneaney WM (2006) Max-plus metods for nonlinear control and estimation. Systems & Control: Foundations & Applications, Birkäuser, Boston Pesc HJ (1994) A practical guide to te solution of real-life optimal control problems. Control Cybernet 23(1-2):7 60 Vinter R (2000) Optimal control. Systems & Control: Foundations & Applications, Birkäuser, Boston

Numerical Optimal Control Overview. Moritz Diehl

Numerical Optimal Control Overview Moritz Diehl Simplified Optimal Control Problem in ODE path constraints h(x, u) 0 initial value x0 states x(t) terminal constraint r(x(t )) 0 controls u(t) 0 t T minimize