LINEAR QUADRATIC REGULATOR PROBLEM WITH POSITIVE CONTROLS W.P.M.H. Heemels, S.J.L. van Eijndhoven y, A.A. Stoorvogel y Technical University of Eindhoven y Dept. of Electrical Engineering Dept. of Mathematics and Computing Science P.O. Box 53 5600 MB Eindhoven Fax: +3 40 43458, e-mail: w.p.m.h.heemels@ele.tue.nl Keywords: Optimal control, linear systems, stability. Abstract In this paper, the Linear Quadratic Regulator Problem with a positivity constraint on the admissible control set is addressed. Necessary and sucient conditions for optimality are presented in terms of inner products, projections on closed convex sets, Pontryagin's maximum principle and dynamic programming. Sucient and sometimes necessary conditions for the existence of positive stabilizing controls are incorporated. Convergence properties between the nite and innite horizon case are presented. Besides these analytical methods, we describe briey a method for the approximation of the optimal controls for the nite and innite horizon problem. Introduction In literature (e.g. []), the Linear Quadratic Regulator Problem has been solved by using Riccati equations. In this article, the same problem will be treated with an additional positivity constraint: we require all components of the control to stay positive. In many real-life problems, the inuence we haveonthe system can be used only in one direction. One example is the controlling of the temperature in a room: the heating element can only put energy into the room, but it cannot extract energy. In process industry, the ow of a certain uid or gas into a reactor is regulated by closing or opening avalve, resulting in one-way streams. Other examples of positive inputs can be found in economical systems, where quantities like investments and taxes are always positive. Mainly, there are two classical approaches in optimal control theory. The rst approach is the maximum principle, initiated by Pontryagin et al. ([3]). This original maximum principle has been used and extended by many others. See for instance, [0], [4], [], [8], to mention a few. We will use the convergence results between the nite and innite horizon problem to derive a maximum principle on innite horizon for the \positive Linear Quadratic Regulator problem." The second approach, Dynamic Programming, was originally conceived by R. Bellman ([]) as a fruitful numerical method to compute optimal controls in discrete time processes. Later, people realized that the same ideas can be used for optimal control problems in continuous time. For continuous time problems, Dynamic Programming leads to a partial dierential equation, the so-called Hamilton- Jacobi-Bellman equation, which has the value function among its solutions ([4]). In a lot of problems, the value function does not behave smoothly, which causes analytical and numerical problems. In case of nonsmoothness, the value function does not satisfy the HJB-equation in a classical sense, but in a more general and more complicated way: it are so-called \viscosity solutions" (see [5]). In this paper, it will be shown that the value function in our problem is continuously dierentiable and satises the HJB-equation in a classical way. This simplies Dynamic Programming in both analytical and numerical aspects. In the formulation of the innite horizon problem, we have to impose some boundedness requirements on the state trajectory: we allow only positive controls that result in square integrable state trajectories. This leads to the introduction of the concept of positive stabilizability. In [7] and [3], the problem of controllability of linear systems with positive controls is addressed. The result of these papers is used to derive sucient and sometimes also necessary conditions for the existence of positive stabilizing controls. Problem Formulation We consider a linear system with input or control u : [t; T ]! IR m, state x :[t; T ]! IR n and output z :[t; T ]! IR p, given by
_x = Ax + Bu () z = Cx+ Du; () where A; B; C and D are matrices of appropriate dimensions and T is a xed end time. For every input function u L [t; T ] m (Lebesgue space of square integrable functions) and initial condition (t; x 0 ), i.e. x(t) = x 0, the solution of () is an absolutely continuous state trajectory, denoted by x t;x0;u. The corresponding output can be written as {z } z t;x0;u(s) =Ce A(s t) x 0 + (M t;t xz 0)(s) s + Ce A(s ) Bu()d + Du(s) ; (3) t {z } (L t;t u)(s) where s [t; T ], with M t;t a bounded linear operator from IR n to L [t; T ] p and L t;t is a bounded linear operator from L [t; T ] m to L [t; T ] p. Both operators are dened in (3). The closed convex cone of positive functions in L [t; T ] m is dened by P [t; T ]:=ful [t; T ] m j u(s) a.e. on [t; T ]g; where := f IR m j i 0g. First, we consider the nite horizon case (T <). The innite horizon version (T = ) will be postponed to section 6. Our objective is to determine for every initial condition (t; x 0 ) [0;T]IR n a control input u P [t; T ], an optimal control, such that J(t; x 0 ;u):=kz t;x0;uk = km t;t x 0 + L t;t uk (4) is minimal. R kk denotes the standard L -norm, i.e. kzk T = z > (s)z(s)ds. In fact, this is a minimum-norm t problem for a closed convex cone. The optimal value of J for all considered initial conditions is described by the value function. Denition (Value function) The value function V is a function from [0;T] IR n to IR and is dened for every (t; x 0 ) [0;T]IR n by 3 Example V (t; x 0 ):= inf J(t; x 0 ;u) (5) up [t;t ] To illustrate that there is no simple connection between the optimal unconstrained LQ-feedback (u= Fx) and the optimal control with positivity constraint, we consider a pendulum given (after linearizing) by the dynamics _x = x _x = x u with u 0: it is allowed only R to push from one side. Our objective is to minimize 0 fx (s) +u (s)gds. We computed J(0;x 0 );x 0 =(;0) > for controls of the form u = g max(0;fx);g 0. The results are plotted in gure. We observe that the cost function is not optimal for g =, because u :3 performs better. However, we can compute a nonnegative control achieving even 5:08, the value indicated by the dashed line in gure. We conclude that there is no simple connection between constrained and unconstrained optimal controls. cost function 5.7 5.6 5.5 5.4 5.3 5. 5. 5. 0.9 0.95.05..5..5.3 values of g Figure : Cost function for various gains g. 4 Mathematical Preliminaries A starting point is the fundamental theorem in Hilbert space theory concerning the minimum distance to a convex set. For a proof, see chapter 3 in []. Theorem Let x be avector in a Hilbert space H with inner product (j)and let K be a closed convex subset of H. Then there is exactly one k 0 K such that kx k 0 k kx kk for all k K. Furthermore, a necessary and sucient condition that k 0 is the unique minimizing vector is that (x k 0 j k k 0 ) 0 for all k K. Theorem can be used to generalize orthogonal projections on closed subspaces in Hilbert spaces. Let K be a closed convex set of the Hilbert space H. We introduce the projection P K onto K as P K x = k 0 K ()kx k 0 kkx kk8kk (6) for x H. Theorem justies this denition and gives an equivalent characterisation of P K : P K x = k 0 K () (x k 0 j k k 0 ) 0 8k K: (7) We now state some simple properties of projections on closed convex sets. Lemma 3 P K = P K. kp K x P K ykkx ykfor all x; y H:
Besides K being closed and convex, assume it is a cone. Then P K (x) = P K x for all x H and 0. 5 Optimal Controls 5. Existence and Uniqueness A standing assumption in the remainder of the paper will be the injectivity ofd. In this case, L t;t has a bounded left inverse, as we will see. Consider a z in the range of L t;t, i.e. z = L t;t u for some u L [t; T ] m. In that case z satises _x(s) = Ax(s)+Bu(s); x(t) =0 z(s) = Cx(s)+Du(s) for s [t; T ] and we can solve u by putting u(s) = (D > D) D > fz(s) Cx(s)g. Substituting this in the differential equation gives _x(s) = (A B(D > D) D > C)x(s)+ +B(D > D) D > z(s); x(t)=0 u(s) = (D > D) D > fz(s) Cx(s)g: This is a prescription in state space representation of a bounded left inverse of L t;t. We denote this particular left inverse by ~ L t;t. Theorem 4 Let t; T be nite times with t<t and x 0 IR n. There is a unique control u t;t;x0 P [t; T ] such that km t;t x 0 + L t;t u t;t;x0 k km t;t x 0 + L t;t uk for all u P [t; T ]. A necessary and sucient condition for u? P [t; T ] to be the unique minimizing control is that (M t;t x 0 + L t;t u? jl t;t u L t;t u? ) 0 (8) for all u P [t; T ]. Proof. If we realize that our problem consists of minimizing kv M t;t x 0 k over v L t;t (P [t; T ]), the result is an easy translation of Theorem, where we use K = L t;t (P [t; T ]) and k 0 = L t;t (u? ). The optimal control, the optimal trajectory and the optimal output with initial conditions (t; x 0 ) and nal time T will be denoted by u t;t;x0, x t;t;x0 and z t;t;x0, respectively. Considering the proof above, it is not hard to see that in terms of projections, we can write u t;t;x0 = ~L t;t P ( M t;t x 0 ); (9) where P is the projection on the closed convex cone L t;t (P [t; T ]). From the last expression and Lemma 3, it is clear that for 0 and hence, u t;t;x0 = u t;t;x0 (0) V (t; x 0 )= V(t; x 0 ): () 5. Maximum Principle The results in this subsection are classical (see e.g. [3]). They are stated here, because in section 6 this theory is used to derive convergence results between nite and innite horizon optimal controls. Consider the Hamiltonian H(x; ; ) =x > A > + > B > +x > C > Cx + +x > C > D + > D > D () for (x; ; ) IR n IR m IR n. The adjoint or costate equation reads @H(x; u; ) _ = = @x A > C > Cx C > Du = A > C > z; (3) with terminal condition (T )=0 (4) and z = Cx+ Du. The optimal control u 0;T;x0, for shortness denoted by u opt, satises for all s [0;T] u opt (s) arg min H(x opt (s);; opt (s)); (5) where x opt is the optimal trajectory and opt the solution to the adjoint equation (3) with z equal to the optimal output z opt = z 0;T;x0.=fIR m j i 0g. Weintroduce kk D > D as the norm induced by the inner product (x j y) D > D = x > D > Dy for x; y IR n in the Hilbert space IR n.furthermore, in this Hilbert space, P denotes the projection on, which is continuous according to Lemma 3. Lemma 5 Let the nal time T>0and initial state x 0 IR n be xed. The optimal control u opt satises u opt (s) =P ( (D> D) fb > (s)+d > Cx(s)g) (6) for all s [0;T], where the functions x and are given by _x = Ax + Bu opt x(0) = x 0 _ = A > C > Cx C > Du opt (T ) = 0 (7) Hence, the optimal control is continuous in time. Proof. This lemma is a straightforward application of the theory above. By (5), u opt (s) is the unique pointwise minimizer of the costs > D > D + > g(s) = k+ (D> D) g(s)k D> D 4 g> (s)(d > D) g(s); (8)
taken over all, where g(s) := B > opt (s) + D > Cx opt (s). Substituting (6) in (7) gives a two-point boundary value problem. Its solutions provide us with a set of candidates containing the optimal control, because the maximum principle is a necessary condition for optimality. In case D > D = I, the expression P ( (D> D) fb > (s)+d > Cx(s)g) simplies to maxf0; B > (s) D > Cx(s)g; where \max" means taking the maximum componentwise. 5.3 Dynamic Programming In the introduction, we discussed the problems with nonsmooth value functions and announced to show that in the positive LQ-problem, the value function is continuously dierentiable and satises the Hamilton-Jacobi-Bellman equation in classical sense. The proof of this fact, being rather technical, is not included, but can be found in [6]. We start with a short overview of the technique of dynamic programming. To do so, we introduce the function L for (x; ) IR n by L(x; ) =z > z= =x > C > Cx +x > C > D + > D > D: (9) The HJB-equation is given by where ~ H is given by V t (t; x)+ ~ H(x; V x (t; x)) = 0; (0) ~H(x; p) = inf fp > Ax + p > B + L(t; x; )g () for (x; p) IR n IR n. V t means the time-derivative of the value function and V x the gradient ofv. By manipulating (), we get ~H(x; p) =x > C > Cx+ + p > Ax 4 g> (x; p)(d > D) g(x; p)+ + inf k + (D> D) g(x; p)k D> D () with g(x; p) = B > p +D > Cx and kk D > D as in the previous subsection. The minimizing equals P ( (D> D) g(x; p)), where P denotes the projection on in the Hilbert space IR n with inner product (j) D > D.IfVis continuously differentiable and the optimal controls are continuous, the value function satises the HJB-equation (see [5]). The Maximum Principle shows that the optimal controls in our problem are continuous. Theorem 6 (Dierentiability of V w.r.t. x) The value function V is continuously dierentiable with respect to x and its directional derivative in the point (t; x 0 ) [t; T ) IR n with increment h IR n is given by or, explicitly, (V x (t; x 0 ) j h) =(z t;t;x0 jm t;t h) (3) V x (t; x 0 )= Z T e A> (s t) C > z t;t;x0 (s)ds: (4) t z t;t;x0 is the output corresponding to initial condition (t; x 0 ) with optimal control u t;t;x0. Notice that in (3) the left inner product is the Euclidean inner product in IR n and the right inner product is the L -inner product. Theorem 7 (Dierentiability of V w.r.t. t) The value function V is dierentiable with respect to t and the partial derivative in the point (t; x 0 ) [t; T )IR n equals V t (t; x 0 )= (V x (t; x 0 ) j Ax + Bu t;t;x0 (t)) + z > t;t;x 0 (t)z t;t;x0 (t) (5) In [6], it is shown also that both partial derivatives are continuous in both arguments. We can conclude now that V is a classical solution to the HJB-equation with the boundary condition V (T;x) = 0 for all x IR n : The socalled verication theorem in [5] gives now that the optimal control u t;t;x0 is the pointwise minimizer of () along the optimal trajectory, i.e. u t;t;x0 (s) = P ( (D> D) g(x t;t;x0 (s);v x (s; x t;t;x0 (s)))); (6) This denes a time-varying optimal feedback. In fact, by using (6), we see that (5) is the HJB-equation. Comparing (4) with the adjoint equation of the Maximum Principle yields a connection between the adjoint variable and the gradient of V: t;t;x0 (s) =V x (s; x t;t;x0 (s)); (7) where t;t;x0 is the solution to the adjoint equation corresponding to u t;t;x0. Moreover, we can even conclude that both dynamic programming and the maximum principle are necessary and sucient conditions for optimality. This follows from the link (7) between the two methods. Necessity of the maximum principle in terms of Lemma 5 translates via (7) into necessity of dynamic programming in terms of (6). Vice versa, the suciency of dynamic programming leads to suciency of the maximum principle along the same lines.
6 Innite Horizon 6. Problem Formulation In this section we consider the innite horizon case (T = ). The time-invariance of the problem guarantees that we can take t = 0 without loss of generality. The problem is to minimize Z J(x 0 ;u):= z > x 0;u(s)z x0 ;u(s)ds (8) 0 over u P [0; ) subject to the relations (), (), x(0) = x 0 and the additional constraint that the state trajectory x should be contained in L [0; ) n. Notice that x L [0; ) n implies x(s)! 0(s!), because x is necessarily absolutely continuous. Denition 8 A control is said to be stabilizing for initial state x 0, if the corresponding state trajectory x satises x L [0; ) n. (A; B) is said to be positive stabilizable, if for every x 0 there exists a stabilizing control u P [0; ). Notice that u L [0; ) m and x L [0; ) n imply z L [0; ) p and thus the niteness of J. So the positive stabilizability of(a; B) guarantees the existence of positive controls, which keep the cost function nite. This is necessary to formulate a descent innite horizon problem. 6. Positive Stabilizability In this subsection we present sucient conditions for the positive stabilizability of(a; B). In the single input case (m = ), they turn out to be sucient and necessary. The main results are stated without proofs. The proofs can be found in [6]. Theorem 9 Consider the system given by (A; B; C; D) with control restraint set =f( ;::: ; m ) > IR m j i 0 ;8 i=;::: ;m g: If (A; B) is stabilizable and all real eigenvectors v of A > corresponding to a nonnegative eigenvalue of A > have the property that B > v has positive components, then (A; B) is positively stabilizable. Notice that a real eigenvector v of A > corresponding to a nonnegative eigenvalue B > v should also have negative components, because v is an eigenvector too. For m = the conditions are also necessary. Corollary Consider the system (A; B; C; D) with m = and control restraint set =firj0g=[0;): (A; B) is positively stabilizable if and only if (A; B) stabilizable and (A) \ [0; )=;. 6.3 Connection between Finite and Innite Horizon In this subsection, we present only the results. The proofs are rather technical and therefore, omitted. However, they can be found in [6]. In the remainder of this paper, we concentrate on systems with the following properties.. (A; B) is positively stabilizable;. D is injective; 3. (A; B; C; D) is minimum phase. Items. and 3. can be replaced by item 3. and (C + DF ;A+BF ) is detectable, where F is given by (C+ DF ) > D =0. Under these assumptions, the optimal controls with in- nite horizon exist and are unique. We dene the operators T : L [0; ) m! L [0;T] m for s [0;T]by( T u)(s) =u(s) and T : L [0;T] m! L [0; ) m for all u L m[0;t]by ( T u)(s) = u(s) st 0 s>t In the sequel, we denote by u T ;x T and z T the optimal control,the optimal state trajectory and the optimal output on [0;T] with xed initial condition (0;x 0 ) for T>0 or T =. Theorem 0 For T!, we have the convergences T u T! u; T z T! z and T x T! x in the L -norm. Also the optimal costs of the nite horizon converge to the optimal costs of the innite horizon, if the horizon tends to innity. Pontryagin's Maximum Principle for the nite horizon converges to a maximum principle for the innite horizon, which implies the continuity of the optimal control on innite horizon. Theorem Fix initial state x 0. The corresponding optimal control, denoted by u, satises u(s) =P ((D > D) f B> (s) D > Cx(s)g); (9) where the continuous function L [0; ) n is initially given by (0) and the dierential equation _ = A > C > z: (30) (0) is the limit of a converging sequence f Ti (0)g iin, where ft i g iin is a sequence of positive numbers, which tend to innity. By using the maximum principle in nite and innite horizon, we can extend the convergence results of Theorem 0 to pointwise convergence.
Theorem for all s [0; ). 6.4 Approximation u T (s)! u(s) (T!); It is obvious that for all s 0 u t;x0 (s + t) = u x0 (s). From this it follows that that there exists a time-invariant optimal feedback u fdb dened by u fdb (x 0 ):=u t;x0 (t) =u 0;x0 (0) = lim T! u 0;T;x 0 (0): (3) The limit in this equation provides us with a method to approximate the optimal feedback. Discretization in time and state is a method to approximate the nite horizon optimal controller. This technique is based on discrete Dynamic Programming, as initiated by R. Bellman ([]). Such techniques can be found in e.g. [9] and [8]. If T is large enough we can use u 0;T;x0 as an approximation for u fdb (x 0 ). In [6], some numerical result are presented. 7 Conclusions In this paper, we addressed three approaches to Linear Quadratic Regulator problem with a positivity constraint on the admissible controls. As main contributions, we consider the necessary and sucient conditions in terms of inner products, projections, the maximum principle and dynamic programming. The maximum principle results in atwo-point boundary value problem, dynamic programming in a partial dierential equation. If one of these equations are solved, their solutions lead analytically to the optimal control. The conditions for positive stability are veried easily and can be used to check the wellposedness of the innite horizon problem. Another essential result is the maximum principle for the innite horizon case and the convergence results between nite horizon optimal controls and the innite horizon control which result in a method to approximate the optimal positive feedback in the considered problem. The results hold for LQ-problems with arbitrary closed convex cones as control restraint set as well. [4] Feichtinger G., Hartl R.F., \ Okonomischer Prozesse: Anwendung des Maximumprinzips in den Wirtschaftswissenschaften," De Gruyter, (986). [5] Fleming W.H., Soner H.M., \Controlled Markov Processes and Viscosity Solutions", Springer-Verlag, (993). [6] Heemels W.P.M.H., \Optimal Positive Control & Optimal Control with Positive State Entries", Master's thesis Technical University of Eindhoven, (995). [7] Heymann M., Stern R.J., \Controllability of Linear Systems with Positive Controls: Geometric Considerations", Journal of Mathematical Analysis and Applications, 5, pages 36-4, (975). [8] Kirk D.E., \Optimal Control Theory, An Introduction", Prentice-Hall Inc., (970). [9] Kushner H.J., and Dupuis P.G., \Numerical Methods for Stochastic Control Problems in Continuous Time", Springer-Verlag, (99). [0] Lee E.B., Markus L., \Foundations of Optimal Control Theory", John Wiley & Sons, Inc., (967). [] Luenberger D.G., \Optimization by Vector Space Methods." John Wiley & Sons, Inc., (969). [] Macki J., Strauss A., \Introduction to Optimal Control Theory", Springer-Verlag Inc., (98). [3] Pontryagin L.S., Boltyanskii V.G., Gamkrelidze R.V., Mishchenko E.F., \The Mathematical Theory of Optimal Processes." Interscience Publishers, John Wiley & Sons, (96). [4] Fleming W.H., Rishel R.W., \Deterministic and Stochastic Optimal Control", Springer-Verlag, (975). References [] Anderson B.D.O., Moore J.B., \Optimal Control - Linear Quadratic Methods", Prentice Hall, (990). [] Bellman R., \Introduction to the Mathematical Theory of Control Processes", Academic Press, (967). [3] Brammer R.F., \Controllability in Linear Autonomous Systems with Positive Controllers", Siam J. of Control, 0-, (97).