The Linear Quadratic Regulator

Transcription

1 10 The Linear Qadratic Reglator 10.1 Problem formlation This chapter concerns optimal control of dynamical systems. Most of this development concerns linear models with a particlarly simple notion of optimality. However, to nderstand the most basic concepts in optimal control, and not become lost in complex notation, it is most convenient to consider first the general model given in nonlinear state space form ẋ = f(x,,t), x( ) = x 0 (10.1) The cost V of a particlar control inpt is defined by the following integral where V () = l(x,,τ)dτ + m(x(t 1 )) (a) t 1 is the final time of the control problem. (b) l is a scalar-valed fnction of x,, and t (c) m is a fnction of x. It is called the terminal penalty fnction. We assme that x 0,, and t 1 are known, fixed vales, and x(t 1 ) is free. Or goal is to choose the control [t0,t 1 ] to minimize V. A case which is typical in applications is where m and l are qadratic fnctions of their argments. For an LTV model, the system description and cost are then given by ẋ = A(t)x + B(t), x( ) = x 0 V () = (x T Q(t)x + T R(t))dt + x T (10.2) (t 1 )Mx(t 1 ) where M, Q and R are positive semidefinite matrix-valed fnctions of time. These matrices can be chosen by the designer to obtain desirable closed loop response. The minimization of the qadratic cost V for a linear system is known as the linear qadratic reglator (LQR) problem. We will stdy the LQR problem in detail, bt first we develop some general reslts for the nonlinear state space model (10.1).

2 The Linear Qadratic Reglator 10.2 Hamilton-Jacobi-Bellman eqations The vale fnction V = V (x 0, ) is defined to be the minimm vale of V over all controls. This is a fnction of the two variables x and t which can be written explicitly as [ t 1 ] V (x,t) = min [t,t1 ] t l(x(τ),(τ),τ)dτ + m(x(t 1 )) Under very general conditions, the vale fnction satisfies a partial differential eqation known as the Hamilton-Jacobi-Bellman (HJB) eqation. To derive this reslt, let x and t be an arbitrary initial time and initial state, and let t m be an intermediate time, t < t m < t 1. Assming that x(τ), t τ t 1, is a soltion to the state eqations with x(t) = x, we mst have [ V t m ] (x,t) = min l(x(τ),(τ),τ)dτ + l(x(τ),(τ),τ)dτ + m(x(t 1 )) [t,t1 ] t t m [ tm = min [t,tm] t This gives the fnctional eqation ( t 1 )] l(x(τ),(τ),τ)dτ + min l(x(τ),(τ),τ)dτ + m(x(t 1 )) [t,t1 ] t } m {{} V (x(t m),t m) [ V tm ] (x,t) = min l(x(τ),(τ),τ)dτ + V (x(t m ),t m ) [t,tm] t (10.3) As a conseqence, the optimal control over the whole interval has the property illstrated in Figre 10.1: If the optimal trajectory passes throgh the state x m at time x(t m ) sing the control, then the control [t m,t 1 ] mst be optimal for the system starting at x m at time t m. If a better existed on [t m,t 1 ], we wold have chosen it. This concept is called the principle of optimality. Optimal trajectory starting from x at time t Optimal trajectory starting from x m at time t m x(t) x(t ) = x m m x(t ) 1 Fig If a better control existed on [t m, t 1 ], we wold have chosen it. By letting t m approach t, we can derive a partial differential eqation for the vale fnction V. Let t denote a small positive nmber, and define t m x m = t + t = x(t m ) = x(t + t) = x(t) + x

3 10.2 Hamilton-Jacobi-Bellman eqations 147 Assming that the vale fnction is sfficiently smooth, we may perform a Taylor series expansion on V sing the optimality eqation (10.3) to obtain V (x,t) = min {l(x(t),(t),t) t + V (x,t) + V } V (x(t),t) x + [t,tm] x t (x(t),t) t Dividing throgh by t and recalling that x(t) = x then gives { 0 = min l(x,(t),t) t [t,tm] t + V x (x,t) x t + V } t (x,t) t t Letting t 0, the ratio x/ t can be replaced by a derivative to give 0 = min [l(x,,t) + V ] x (x,t)ẋ(t) + V t where V [ V x =... V ] = ( x V ) T x 1 x n Ths, we have obtained the following partial differential eqation which the vale fnction mst satisfy if it is smooth. The reslting eqation (10.4) is the Hamilton Jacobi Bellman (HJB) eqation, V t (x,t) = min [l(x,,t) + V ] x (x,t)f(x,,t) The terminal penalty term gives a bondary condition for the HJB eqation V (x(t 1 ),t 1 ) = m(x(t 1 )). The term in brackets in (10.4) is called the Hamiltonian, and is denoted H: (10.4) H(x,p,,t) := l(x,,t) + p T f(x,,t), (10.5) where p = x V. We ths arrive at the following Theorem If the vale fnction V has continos partial derivatives, then it satisfies the following partial differential eqation V t (x,t) = min H(x, xv (x,t),,t), and the optimal control (t) and corresponding state trajectory x (t) mst satisfy min H(x (t), x V (x (t),t),,t) = H(x (t), x V (x (t),t), (t),t). (10.6) An important conseqence of Theorem is that the optimal control can be written in state feedback form (t) = ū(x (t),t), where the fnction ū is defined throgh the minimization in (10.6).

4 The Linear Qadratic Reglator Example Consider the simple integrator model, with the polynomial cost criterion ẋ = V () = T 0 [ 2 + x 4 ]dt Here we have f(x,,t) =, l(x,,t) = 2 + x 4, and m(x,t) 0. The Hamiltonian is ths H(x,p,,t) = p x 4, and the HJB eqation becomes V t Minimizing with respect to gives = min{ V x x 4 }. = 1 V 2 x (x,t), which is a form of state feedback. The closed loop system has the appealing form ẋ (t) = 1 V 2 x (x (t),t) This eqation shows that the control forces the state to move in the direction in which the cost to go V decreases. Sbstitting the formla for back into the HJB eqation gives the PDE V ( V ) 2 t (x,t) = 1 4 x (x,t) + x 4, with the bondary condition V (x,t) = 0. This is as far as we can go, since we do not have available methods to solve a PDE of this form. If a soltion is reqired, it may be fond nmerically. However, a simpler set of eqations is obtained in the limit as T. This simpler problem is treated in Exercise 1 below. We now leave the general nonlinear model and concentrate on linear systems with qadratic cost. We will retrn to the more general problem in Chapter A soltion to the LQR problem For the remainder of this chapter we consider the LQR problem whose system description and cost are given in (10.2). For this control problem, x 0, and t 1 are given, and x(t 1 ) is free. To ensre that this problem has a soltion we assme that R is strictly positive definite (R > 0). To begin, we now compte the optimal control by solving the HJB partial differential eqation. The Hamiltonian for this problem is given by H(x,p,,t) = l + p T f = x T Qx + T R + p T (Ax + B) To minimize H with respect to we compte the derivative

5 11 Introdction to the Minimm Principle We now retrn to the general nonlinear state space model, with general cost criterion of the form ẋ(t) = f(x(t),(t),t), V () = x( ) = x 0 R n l(x(t),(t),t)dt + m(x(t 1 )) (11.1) We have already shown nder certain conditions that if the inpt minimizes V (), then a partial differential eqation known as the HJB eqation mst be satisfied. One conseqence of this reslt is that the optimal control can be written in state feedback form throgh the derivative of the vale fnction V. The biggest drawback to this reslt is that one mst solve a PDE which in general can be very complex. Even for the relatively simple LQR problem, the soltion of the HJB eqations reqired significant ingenity. The minimm principle is again a method for obtaining necessary conditions nder which a control is optimal. This reslt is based pon the soltion to an ordinary differential eqation in 2n dimensions. Becase it is an ordinary differential eqation, in many instances it may be solved even thogh the HJB eqation is intractable. Unfortnately, this simplicity comes with a price. The soltion to this ordinary differential eqation is in open-loop form, rather than state space form. The two view points, both the Minimm Principle and the HJB eqations, each have vale in nonlinear optimization, and neither approach can tell the whole story Minimm Principle and the HJB eqations There is no space in a book this size or a corse at this level to give a complete proof of the Minimm Principle. We can however give some heristic argments to make the reslt seem plasible, and to gain some insight. We initially consider the optimization problem (11.1) where x 0,, t 1 are fixed, and x(t 1 ) is free. Or first approach is throgh the HJB eqation

6 11.1 Minimm Principle and the HJB eqations 175 t V (x,t) = min (l(x,,t) + V x f(x,,t)) (t) = argmin (l(x,,t) + V x f(x,,t)) For x,t fixed, let ū(x,t) denote the vale of which attains the minimm above, so that = ū(x (t),t). In the derivation below we assme that ū is a smooth fnction of x. This assmption is false in many models, which indicates that another derivation of the Minimm Principle which does not rely on the HJB eqation is reqired to create a general theory. With ū so defined, the HJB eqation becomes t V (x,t) = l(x,ū(x,t),t) + V x (x,t)f(x,ū(x,t),t). Taking partial derivatives of both sides with respect to x gives the term V x sides of the reslting eqation: on both 2 x t V (x,t) = l x (x,ū(x,t),t) + 2 V 2 x (x,t)f(x,ū(x,t),t) + V x (x,t) f x (x,ū(x,t),t) + (l + V ) x f ū(x, t) (x,ū(x,t),t) x }{{} derivative vanishes at ū As indicated above, since ū is the nconstrained minimm of H = l + V x f, the partial derivative with respect to mst vanish. This PDE holds for any state x and time t - they are treated as independent variables. Consider now the optimal trajectory x (t) with optimal inpt (t) = ū(x (t),t). By a simple sbstittion we obtain 0 = 2 x t V (x (t),t) + l x (x (t), (t),t) + 2 V x 2 (x (t),t)f(x (t), (t),t) + V x (x (t),t) f x (x (t), (t),t). (11.2) We can now convert this PDE into an ODE. The main trick is to define p(t) := V x (x (t),t) This fnction is interpreted as the sensitivity of the cost with respect to crrent state, and takes vales in R n. The derivative of p with respect to t can be compted as follows:

7 Introdction to the Minimm Principle d dt p(t) = 2 V x 2 (x (t),t)ẋ (t) + 2 V x t (x (t),t) = 2 V x 2 (x (t),t)f(x (t), (t),t) + 2 V x t (x (t),t). The two mixed partial terms in this eqation are also inclded in (11.2). Combining these two eqations, we thereby eliminate these terms to obtain 0 = d l p(t) + dt x (x (t), (t),t) + V x (x (t),t) }{{} p T (t) f x (x (t), (t),t). (11.3) From the form of the Hamiltonian H(x,p,,t) = p T f(x,,t) + l(x,,t), the differential eqation (11.3) may be written with the bondary condition ṗ(t) = x H(x (t),p(t), (t),t) p(t 1 ) = x V (x (t 1 ),t 1 ) = x m(x (t 1 ),t 1 ). This is not a proof, since for example we do not know if the vale fnction V will be smooth. However, it does make the following reslt seem plasible. For a proof see [8]. Theorem (Minimm Principle). Sppose that t 1 is fixed, x(t 1 ) is free, and sppose that is a soltion to the optimal control problem (11.1). Then (a) There exists a costate vector p(t) sch that for t t 1, (t) = arg min H(x (t),p(t),,t) (b) The pair (p,x ) satisfy the 2-point bondary vale problem: ẋ (t) = p H(x (t),p(t), (t),t) ṗ(t) = x H(x (t),p(t), (t),t) ( ) = f(x (t), (t),t) (11.4) with the two bondary conditions x( ) = x 0 ; p(t 1 ) = x m(x(t 1 ),t 1 ).

8 11.2 Minimm Principle and Lagrange mltipliers Minimm Principle and Lagrange mltipliers Optimal control involves a fnctional minimization which is similar in form to ordinary optimization in R m as described in a second year calcls corse. For nconstrained optimization problems in R m, the main idea is to look at the derivative of the fnction V to be minimized, and find points in R m at which the derivative is zero, so that V (x) = ϑ. Sch an x is called a stationary point of the optimization problem. By examining all stationary points of the fnction to be minimized one can freqently find among these the optimal soltion. The calcls of variations is the infinite dimensional generalization of ordinary optimization in R m. Conceptally, it is no more complex than its finite dimensional conterpart. In more advanced calcls corses, soltions to constrained optimization problems in R m are addressed sing Lagrange mltipliers. Sppose for example that one desires to minimize the fnction V (x) sbject to the constraint g(x) = ϑ, x R m, where g: R m R d. Consider the new cost fnction V (x,p) = V (x) + p T g(x), where p R d. The vector p is known as a Lagrange mltiplier. The point of extending the state space in this way is that we can now solve an nconstrained optimization problem, and very often this gives a soltion to the constrained problem. Indeed, if x 0,p 0 is a stationary point for V then we mst have ϑ = x V (x 0,p 0 ) = V (x 0 ) + g(x 0 )p 0 (11.5) ϑ = p V (x 0,p 0 ) = g(x 0 ) (11.6) Eqation (11.5) implies that the gradient of V is a linear combination of the gradients of the components of g at x 0, which is a necessary condition for optimality nder very general conditions. Figre 11.1 illstrates this with m = 2 and d = 1. Eqation (11.6) is simply a restatement of the constraint g = ϑ. This Lagrange mltiplier approach can also be generalized to infinite dimensional problems of the form (11.1), and this then gives a direct derivation of the Minimm Principle which does not rely on the HJB eqations. To generalize the Lagrange mltiplier approach we mst first generalize the concept of a stationary point. Sppose that F is a fnctional on D r [,t 1 ]. That is, for any fnction z D r [,t 1 ], F(z) is a real nmber. For any η D r [,t 1 ] we can define a directional derivative as follows: F(z + ǫη) F(z) D η F (z) = lim, ǫ 0 ǫ whenever the limit exists. The fnction z(t,ǫ) = z(t) + ǫη(t) may be viewed as a pertrbation of z, as shown in Figre 11.2 We call z 0 a stationary point of F if D η F (z 0 ) = 0 for any η D r [,t 1 ]. If z 0 is a minimm of F, then a pertrbation cannot decrease the cost, and hence we mst have for any η, F(z 0 + ǫη) F(z 0 ), ǫ R. From the definition of the derivative it then follows that an optimal soltion mst be a stationary point, jst as in the finite dimensional case!

9 Introdction to the Minimm Principle g(x ) 1 x g(x 2 ) V(x ) 1 1 x 2 V(x ) 2 {x : V(x) = -2} {x : V(x) = -1} {x : g(x) = 0} {x : V(x) = 0} {x : V(x) = 1} Fig An optimization problem on R 2 with a single constraint g = 0. The point x 1 is not optimal: By moving to the right along the constraint set, the fnction V (x) will decrease. The point x 2 is a minimm of V sbject to the constraint g(x) = 0 since at this point we have V (x 2 ) = 1. V can get no lower - for example, V (x) is never eqal to 2 since the level set {x : V (x) = 2} does not intersect the constraint set {x : g(x) = 0}. At x 2, the gradient of V and the gradient of g are parallel. The problem at hand can be cast as a constrained fnctional minimization problem, Minimize V (x,) = ldτ + m Sbject to ẋ f = ϑ, x D n [,t 1 ], D m [,t 1 ]. To obtain necessary conditions for optimality for constrained problems of this form we can extend the Lagrange mltiplier method. We otline this approach in five steps: Step 1. Append the state eqations to obtain the new cost fnctional V (x,) = ldt + m(x(t 1 )) + p T (f ẋ)dt The Lagrange mltiplier vector p lies in D n [,t 1 ]. The prpose of this is to gain an expression for the cost in which x and can be varied independently. Step 2. of V : Use integration by parts to eliminate the derivative of x in the definition p T ẋdt = p T dx = ( p T x ) t 1 ṗ T xdt

10 11.2 Minimm Principle and Lagrange mltipliers 179 z(t) z(t,ε) = z(t) + ε η(t) Fig A pertrbation of the fnction z D[, t 1 ]. Step 3. Recall the definition of the Hamiltonian, H(x,p,,t) := l(x,,t) + p T f(x,,t). Combining this with the formlas given in the previos steps gives V (x,) = H(x,p,,t)dt + ṗ T xdt +p T ( )x( ) p T (t 1 )x(t 1 ) + m(x(t 1 )) (11.7) Step 4. Sppose that is an optimal control, and that x is the corresponding optimal state trajectory. The Lagrange mltiplier theorem asserts that the pair (x, ) is a stationary point of V for some p. Perform pertrbations of the optimal control and state trajectories to form (t,δ) = (t) + δψ(t) x(t,ǫ) = x (t) + ǫη(t) (11.8) Since we are insisting that x( ) = x 0, we may assme that η( ) = ϑ. Consider first variations in ǫ, with δ set eqal to zero. Letting V (ǫ) = V (x + ǫη, ), we mst have d dǫ V (ǫ) = 0. Using (11.7) to compte the derivative gives t1 0 = x H(x,p,,t)η(t)dt + ṗ T η dt p T (t 1 )η(t 1 ) + p T ( )η( ) + x m(x (t 1 ))η(t 1 ). (11.9) Similarly, by considering pertrbations in we obtain for any ψ D m [,t 1 ], ϑ = H(x,p,,t)ψ(t)dt (11.10) This simpler expression is obtained becase only the first term in (11.7) depends pon.

11 Introdction to the Minimm Principle Step 5. We can choose η(t) freely in (11.9). From this it follows that and since η( ) = ϑ, and η(t 1 ) is free, Similarly, by (11.10) we have H x + ṗt = ϑ T ṗ = x H p T (t 1 ) + m x (t 1) = ϑ T p(t 1 ) = x m(x(t 1 )) H = ϑt H = ϑ In fact, if is to be a minimm of ˆV, then in fact it mst minimize H pointwise. These final eqations then give the Minimm Principle Theorem From this proof it is clear that many generalizations of the Minimm Principle are possible. Sppose for instance that the final state x(t 1 ) = x 1 is specified. Then the pertrbation η will satisfy η(t 1 ) = ϑ, and hence sing (11.9), it is impossible to find a bondary condition for p. None is needed in this case, since to solve the 2n-dimensional copled state and costate eqations, it is enogh to know the initial and final conditions of x The penalty approach A third heristic approach to the Minimm Principle involves relaxing the hard constraint ẋ f = ϑ, bt instead impose a large, yet soft constraint by defining the cost V (x,) = l(x(t),(t),t)dt + k 2 ẋ(t) f(x(t),(t),t) 2 dt + m(x(t 1 )) The constant k in this eqation is assmed large, so that ẋ(t) f(x(t),(t),t) ϑ. We assme that a pair (x k, k ) exists which minimizes V k. Letting (x, ) denote a soltion to the original optimization problem, we have by optimality, V (x k, k ) V (x, ) = V. Assming l and m are positive, this gives the following niform bond ẋ(t) f(x(t),(t),t) 2 dt 2 k V Hence, for large k, the pair (x k, k ) will indeed approximately satisfy the differential eqation ẋ = f. If we pertrb x k to form x k + ǫη and define V (ǫ) = V (x k + ǫη, k ) then we mst have d/dǫ V (ǫ) = 0 when ǫ = 0. Using the definition of ˆV gives