Systems & Control Letters 25 ( 1995) Finite-dimensional quasi-linear risk-sensitive control. Received 8 October 1993; revised 10 July 1994

Transcription

1 Systems & Control Letters 25 ( 1995) Sxsiiwis & CONTROL Lrlrims Finite-dimensional quasi-linear risk-sensitive control Lakhdar Aggoun a, Alain Bensoussan b, Robert J. Elliott c, John B. Moored a Department of Statistics, University of A ucklarui, private Bag 92019, A uck[und, New Zealand b 1.N. R. I. A. Rocquencourt, Le Chesnay Cedex, France c Department of Mathematical Sciences, Universi~y of A Iberia, Edmonton, A B, Canada ~ De~arlmenl of SYstems Engineering and Cooperative Research Centre for Robust and Adaptive Syrwn.f, Reward scho~l o~ F hysica[ Sciences and bngineerinq, Australian National University, Canberra, ACT 0200, A uslra[ia Received 8 October 1993; revised 10 July 1994 Abstract A discrete-time partially observed stochastic control problem with exponential running cost is considered. The dynamics are linear and the running cost quadratic in the state variable, but the control may enter nonlinearly. Explicit solutions for a forward Zakai equation and a backward adjoint equation are derived in terms of finite-dimensional dynamics. This enables the partially observed problem to be expressed in finite-dimensional terms and a separation principle applied. Keywords; Risk-sensitive partially observed stochastic control; Finite-dimensional information states 1. Introduction Recent interest in risk-sensitive stochastic control problems is due in part to connections with Hm or robust control problems and dynamic games. The solution of a risk-sensitive problem leads to a conservative optimal policy, corresponding to the controller s aversion to risk. For linear/quadratic risk-sensitive problems with fill state information, Jacobson [6] established the connection with dynamic games. The analogous nonlinear problem was studied recently, and a dynamic game is obtained as a small noise limit [3, 5, 7, 10]. A risk-neutral stochastic control problem obtains as a small risk limit [3, 7]. Whittle [9] solved the discrete-time linear/quadratic risk-sensitive stochastic control problem with incomplete state information, and characterized the solution in terms of a certainty equivalence principle. The analogous continuous-time problem was solved by Bensoussan and van Schuppen [2], where the problem was converted to an equivalent one with full state information. The general nonlinear exponential cost with incomplete information in discrete time is treated in [8]. Recently Bensoussan and Elliott [1] discussed a finite-dimensional risk sensitive control problem with quasilinear dynamics and quadratic cost; the results of this paper are a discrete-time version of [1]. 2. Dynamics Suppose {Q, %, P} is a probability space with a complete filtration {%~ }, k c N!, on which are given two sequences of independent random variables xk and ~k, having normal densities ~ k = N(O, ~k ) and ($k = N(O, rk ) where ~k and rk are n x n and m x m positive definite matrices for all k G N. *Corresponding author. Elsevier Science B.V. SSDI (94)

2 152 L. Aqqoun et al, I Systems & Control Letters 25 (1995) 15! 15? U is a nonempty subset of RP and f~k = a{ yl, 1<k } is the complete filtration generated by y. The admissible controls u are the set of (1-valued {?~k} adapted processes. Write Uk,I for the set of such control processes defined for k,..., 1. Consider the following functions.4~(u): U x N ~ -L(R, W), (the space of n x n matrices), G~(u): U X N - R, Ck(u) : U X ~ - ~(~m,~m). For any control u, define Ao,o = 1 and Aok+, = fi i/+ I(x/+1 ~~(u)x[ - Gk(u))f)/+l(J [+1 - C I(U)XI+I(U)) )d/+l(y/+1 ) Then AO,kis an %k martingale and EIAO,k] = 1. A new probability measure can be defined by putting dp/dpix, = AO,k. Define the processes vk = u! and wk = w:, Vk+l = X~+I /tk(u)xk Gk(U), Wk = Yk Ck(tt)xk. Then under, ~, ok and wk are two sequences of independent normally distributed random variables with densities $!i and If!Jk,respectively. Therefore under ~, Xk+l = /tk(tf)xk + Gk(U) + ~k+l, (1) yk = Ck(u)xk + wk. (2) 3. cost Consider the following mappings A4(. ):u + R, m(.): U+ R, N(.):u + R, q.): w - R. The results extend immediately to the case where M, m, and N are time dependent. For any admissible control u and real number 8 we consider the expected exponential risk sensitive cost T 1 J(u) = Oi exp 6 ~ [(M(u)xl, x1) + (m(u), xl) + [{ I=o }1 = (IE AoTexp O ~ [(M(u)xI,xI) + (m(u), x/) +N(u)I [ ~ {:: }1 Write Do,o = 1, k 1 ) D~,~ = D; = exp~ ~[(M(u)xI,xI) + (m(u), x/) + N(u)]. [ /=0 (3) (4) 4. Finite-dimensional information states Notation 4.1. For any admissible control u consider the measure C#(X)dX := E[A;D#(xk 6 dx)lwk]. (5)

3 L. Aqqoun et u1.i System.y & Control Letters 25 ( 1W5) Note z{(x) = ~o(x), Then Y; satisfies the following forward recursion ~;+,(x)= A+lbk+l - ck+l(~)x) exp O((M(U)X,X) + (m(u), x) + N(u)) dk+l(yk+l) x #+I(x-4+I(u)z-G ~+I(u))m(z)&. J The linearity of the dynamics and the fact that uk and wk are independent and normally distributed implies that U;(X) is an unnormalized normal density which we write as (6) #(X) = z,,(u)exp-~(x ~k(u)) ~~ (u)(x /fk(u)). (7) R; (u), ~k (u) and.zk(u ) are given by the following algebraic recursions: and ~k(u) = R /&(~). The recursions for ~, R and Z are algebraic and involve no integration. For this partially observed stochastic control problem the information state u;(x) (which is, in general, a measure valued process), is determined by the three finite-dimensional parameters #k(u), Rk(u) and Zk(u ). These parameters can be considered as the state L of the process: <; = (,Uk(u), Rk(u), zk(u)) and we can write xl(x) = f%(<;,x) = -zk(~)exp -+(x ~k(u)) R; (U)(X ~k(u)). For integrable ~(x), (!xk(&),f) = / CXL(~;,X)f(X)dX, which in fact equds ~[i i~,k~{,kf(xk ) I lk]. W 5. A separation principle For any admissible control u, we saw that the expected total cost is J(u) = A: ~ exp 6 fi[(a4(u)x/,x/) + (m(u),x~) + N(u)]+ [ ~ {/=1 exp O@(x~)] = OEIEIA~,~D~,r exp t9@(x~)14y~]]

4 154 L. Aqgoun et al. I Systems & Control Letters 25 ( 1995] = (IE exp 6@(x) wr(x)dx = OE[(ct~(<~), exp 00)], [./ R 1 is the terminal cost as in Eq. (3). 6. Adjoint process Now for any k, O < k < T, Write fi(xk ) = E[A;+,,~Dj!+,,~ ) Ixk, IT] where, using the Markov property of X, the conditioning involves only Xk. Note that fl~(x~ ) = ). Therefore Note this decomposition is independent of k, so Lemma 6.1. We have the,fbllowing backward recursion for the process /3: fl;(xk ) = h+l(yk+l - C~+I(U)X) ik+i(x ~k(u)xk - Gk(~)) / x exp fl{x kf(u)x + (m(u), X) + ~(~)}P~+I (x) dx. (9) Proof. DE I(Yk+I - ck+l(u~k+l )tk+l(xk+l 14k(~)xk - Gk(u)) k+l )~k+l(xk+l) DE dk+l(yk+l - ck+l(u)xk+l )~k+l(xk+l ~k(u)xk - Gk(u)) [ dk+l(~k+l )tk+l(xk+l ) x exp o{xi+l~(u)xk+l + (m(u), xk+l) + N(u)} flf+l(xk+l )lxk,~~t. 1 Integrating with respect to the density ofxk+l and using the independence assumption under P gives the result. U Consider the unnormalized Gaussian densities ~~(x, z ) given by ~~(x,z)= ;exp-i(x - ~k(z)) ~~ ((x - Yk(z)).

5 L. Agqoun et al. I Systems & Conlrol Leiter.Y 25 (1995i We put ~~(x,z) = 6(x z), ST = O and y~ = z. Then Yk,Sk- and ~k are given by the following backward recursions. Sk- =~j[~;:, - k+l a- ~~+!l]~k jk = ~:~;;la- [c;+,~;:l~k+l + ZI:IGL. + ~m + jk+, ~Gk] ~k = ik+,l~k+,l -1/21 a112e~p-+{b-qfiyk} (lo) Here Furthermore, H(x) = / w Remark 6.2. /J ;is the adjoint process and again it is determined by the finite-dimensional parameters y, S and z which satisfy the reverse-time, algebraic, recursions (10 ). 7. Dynamic programming We have noted that the information state!xk(x) is determined by the finite-dimensional parameters t; = (pk(u), &( U), zk(u)). Given t; a control Uk and the new observation Yk+1, Eqs. (8) determine the next value ~i+l = tk+l(t:>uk, yk+l). Suppose at some intermediate time k, O < k < T, the information state <k is ~ = (p, R, Z) The value function for this control problem is Theorem 7.1. The value function.sutisjies the recursion: V(~, k) = &f,e[~(~k+,(&u, J k+l),k + 1)] (11) and V(<, T) = (ct~(<), exp O@). Proof. Now ~(<,k) = J&:_, ~[(a;(t)>fl) Ih = L].

6 156 L. Aggoun et (il./ Systems & Control Leiier.~ 25 (1995) From ( 10) flk is given by a backward recursion from flk+l, that is we can write flk = [~~(fl~+,). Therefore The interchange of conditional expectation and minimization is justified by application of the lattice property of the controls (see [4] ). Write U;,, for the set of control processes on the time interval k,..., 1 which are adapted to the filtration rr{t,: k <js 1}. We call such controls separated. Theorem 7.2 (Verification). Suppose u* ~ U&, is a control which,jtir each k = O,..., T 1, u; ( (k ) achieves the minimum in (11 ). Then U* E UO,T ~ and is an optimal control, Proof. Write ~(<,k,u) = ~[(~;(<)>pf)l(k = <1. We shall show that V(~, k) = p((, k,u ), foreachk=o,..., T. (12) For k = T ( 12) is clearly satisfied, Suppose ( 12) is true fork + 1,..., T. Then ~(~, k,u )=~[~[(clk+ l(t~;,), ~;;,)l<k = t>~~k+l]ltk = L] = EIV(Lt:l(L),k + l), u Uk.tl,r-1)] = EIV(&~l((), k + 1)1 = V(i, k). This gives ( 12). Putting k = Owe see V(<, o,u ) = V(g,o)<v(t,(),zl) for any u G Uo,r-,. That is, u* is an optimal control. Remark 7.3. This result shows the optimal policy u* for the risk sensitive control problem is a separated policy, in that it is a function of the information state ~. Acknowledgments The authors wish to acknowledge support of the Natural Sciences and Engineering Research Council of Canada, grant A7964, INRIA, Rocquencourt France and the hospitality of the Department of Systems Engineering at the Australian National University.

7 L. Aggoun et al. I Systems & Control Letters 25 (1995) References [I] A, Bensoussan and R.J. Elliott, A finite dimensional risk sensitive control problem, to appear in SIAM J Control Opzirn [2] A, Bensoussan and J,H. van Schuppen, Optimal control of partially observable stochastic systems with an exponential-of-integral performance index, SIAM J. Control Opfinr. 23 ( 1985) [3] C. Campi and M.R, James, Discrete-time nonlinear risk-sensitive control, Preprint, [4] R.J, Elliott, Stochastic calculus and applications, in: Applications of A4a/hematics, Vol. 18 (Springer, Berlin, 1982). [5] W.H. Fleming and W.M. McEneaney, Risk sensitive optimal control and differential games, in: Proc. Conf on Adupiice and Siochasric Corrrra/, Univ. of Kansas (Springer, Berlin, 1991). [6] D.H, Jacobson, Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games, IEEE Trans. Automat. Control AC-18 (2) ( 1973) [7] M.R. James, Asymptotic analysis of nonlinear stochastic risk-sensitive control and differential games, Math. Control Signak SYsferns 5 (4) (1992) 401 JI17. [8] MR. James, J. Baras and R.J. Elliott, Risk sensitive control and dynamic games for partially observed discrete-time nonlinear systems, IEEE Trans. Automat. Conrrol AC-39 ( 1994) [9] P. Whittle, Risk-sensitive linear/quadratic/gaussian control, Adu. Appl. Probab. 13 ( 198 I ) 76L777. [10] P. Whittle, A risk-sensitive maximum principle, Sy.$lems Control Lett. 15 ( 1990)