Time-Inconsistent Mean-Field Stochastic Linear-Quadratic Optimal Control

Transcription

1 Time-Inconsistent Mean-Fied Stochastic Linear-Quadratic Optima Contro I Yuan-Hua 1 1. Department of Mathematics Schoo of Sciences Tianjin Poytechnic University Tianjin P. R. China E-mai: yhni@amss.ac.cn Abstract: This paper is concerned with a ind of time-consistent soution of the time-inconsistent mean-fied stochastic inearquadratic optima contro. Different from standard stochastic inear-quadratic probems both the system matrices and the weighting matrices are dependent on initia time and the conditiona expectations of the contro and state enter quadraticay into the cost functiona. Such features wi ruin the Beman optimaity principe and resut in the time-inconsistency of the optima contro. Due to the dynamica nature of contro probems a ind of open-oop time-consistent equiibrium contro is thoroughy investigated in this paper. It is shown that the existence of an open-oop time-consistent equiibrium contro for a fixed initia pair is equivaent to the sovabiity of a set of forward-bacward stochastic difference equations with stationary conditions and convexity conditions. By decouping the forward-bacward stochastic difference equations the existence of the open-oop e- quiibrium contro for a the initia pairs is shown to be equivaent to the sovabiity of a set of couped constraint generaized difference Riccati equations and a set of couped constraint inear difference equations. Key Words: Time-inconsistency mean-fied contro stochastic inear-quadratic optima contro 1 Introduction The Beman optimaity principe is an essentia property in optima contro theory which provides the theoretica foundation of the dynamic programming approach. From the Beman optimaity principe an optima contro for an initia pair is aso optima aong the optima trajectory. Such a phenomenon is referred as the time-consistency of the optima contro which ensures that one needs ony to sove an optima contro probem for a given initia pair and the obtained optima contro is aso optima aong the optima trajectory. However in reaity the time-consistency fais quite often. For instance when the initia time or initia state enters into the system dynamics or cost functiona expicity or even more the conditiona expectations of the state or contro enters nonineary into the cost functiona the corresponding probems are time-inconsistent. See exampes in 13] and 5] about the hyperboic discounting and quasi-geometric discounting. The probem with noninear term of the conditiona expectation in cost functiona is caed as mean-fied stochastic optima contro. In this case the smoothing property of the conditiona expectation wi not be in effect to ensure the time-consistency of the optima contro. A particuar exampe of this case is the nown mean-variance utiity 3] and 5]. Due to the dynamica nature of the contro probems it is reasonabe to hande the time-inconsistency in a dynamic manner. Instead of seeing an optima contro some ind of equiibrium contros are concerned with. This is mainy motivated by practica appications such as in mathematica finance and economics and has recenty attracted considerabe interest and efforts. The mathematica formuation of the time-inconsistency was first reported by 22] and a quaitative anaysis was given in 21]. Foowing 22] the wors 10] 13] 14] and 19] are for discrete dynamic system- Yuan-Hua i is supported by the ationa atura Science Foundation of China the China Postdoctora Science Foundation 2014M T s or simpe ordinary differentia equations ODEs. Subsequenty 6] and 7] studied the non-exponentia discounting probems both for simpe ODEs and stochastic differentia equations and introduced the notion of time-consistent contro. 5] discussed the probems of genera Marovian time-inconsistent stochastic optima contro. 24] and 25] addressed the deterministic continuous-time inear-quadratic LQ optima contro by an essentiay cooperative game approach. 27] considered the stochastic LQ probem of meanfied case which is caed the cosed-oop formuation there. Different from 24] and 25] 12] studied another ind of time-consistent equiibrium contro which is an infinitesimay open-oop optima contro. Roughy speaing the controer given by 12] can commit to the equiibrium contro in an infinitesima manner. 27] thoroughy investigated both the open-oop and the cosed-oop time-consistent soutions for genera mean-fied stochastic LQ probems. It is shown 27] that the existence of the open-oop equiibrium contro and the cosed-oop equiibrium strategy is ensured via the sovabiity of certain systems of Riccati-type differentia equations. However a these existing resuts focus on the definite LQ probems. In this paper we sha investigate a time-inconsistent mean-fied stochastic LQ optima contro probem whose system dynamics and cost functiona are aso dependent on the initia time. It is worth noting that the definiteness constraints are not posed on the state and contro weighting matrices. Here we adopt a dynamic manner to attac the time-inconsistency and intend seeing a ind of open-oop equiibrium contro of Probem LQ. After giving the definition of open-oop equiibrium pair its existence for a fixed initia pair is shown to be equivaent to the sovabiity of a set of forward-bacward stochastic difference equations FBS Es with stationary conditions and convexity conditions. If for a fixed initia pair Probem LQ admits an openoop equiibrium pair then a set of constrained inear difference equations LDEs is sovabe and the open-oop equiibrium contro admits a cosed-oop representation. Here the cosed-oop representation is a inear feedbac of the cur-

2 rent vaue of equiibrium state whose gains are computed via the soutions of the constraint LDEs 11 and a set of generaized difference Riccati equations GDREs 14. For any initia pair Probem LQ admitting an open-oop equiibrium pair is shown to be equivaent to that the constraint LDEs 15 and a set of constraint GDREs 16 are sovabe. Different from is of constrained GDREs. Interestingy if sovabe the set of GDREs 16 does not have symmetric structure i.e. its soution is not symmetric. In 16] a simpified version of Probem LQ is considered where the system dynamics and the cost functiona do not contain mean-fied terms. Hence this paper is a continuation of 16]. If the system dynamics and cost functiona are independent of the initia time the corresponding LQ probem is a dynamic version of that considered in 17] where a static version is studied with the conditiona expectation operator being repaced by the expectation operator. For detais on mean-fied stochastic optima contro and reated meanfied games we refer to for exampe 8] 17] 18] 26] 4] 11] 15] and the references therein. The rest of the paper is organized as foows. Section 2 presents the resuts mentioned above and Section 3 gives some concuding remars. 2 Open-oop time-consistent equiibrium contro Consider the foowing controed stochastic difference e- quation S E X+1 t A t X t + ĀtE t X t + B t u + B t E t u + C t X t + C t E t X t 1 + D t u + D t E t u w Xt t x T t t T where T {0... 1} T t {t 1} and A t Āt C t C t R n n B t B t D t D t R n m are deterministic matrices; {X t T t } X t and {u T t } u with T t being {0... } are the state process and contro process respectivey. Here the initia time t is parameterized in the matrices and state to emphasize that the matrices and state may change according to the initia time t. Simiar notations wi be used throughout the paper. In 1 E t is the conditiona mathematica expectation E F t ] with respect to F t {x 0 w 0 1 t 1} and F 1 is understood as { Ω}. The noise {w T} is assumed to be a martingae difference sequence defined on a probabiity space Ω F P with E +1 w +1 2 ] 1 T. 2 The cost functiona associated with system 1 is Jt x; u t E t X t T Q t X t + E t X t T Qt E t X t + u T R t u + E t u T Rt E t u ] + E t X t T G t X t + E t X t T Ḡ t E t X t 3 where Q t Q t R t R t T t G t Ḡt are deterministic symmetric matrices of appropriate dimensions. In 1 x is in L 2 F t; Rn which is a set of random variabes such that any ξ L 2 F t; Rn is F t -measurabe and E ξ 2 <. Let L 2 F T t; H be a set of H-vaued processes such that for any ν {ν T t } L 2 F T t; H ν is F -measurabe and t E ν 2 <. Then we pose the foowing optima contro probem. Probem LQ. Concerned with 13 and the initia pair t x find a u L 2 F T t; R m such that Jt x; u inf Jt x; u. 4 u L 2 F Tt;Rm Instead of soving Probem LQ for the static precommitted optima contro we adopt the concept of dynamic equiibrium contro which is optima in an infinitesima sense and consistent with the dynamica nature of Probem LQ. Definition 2.1 Given t T and x L 2 F t; Rn a statecontro pair X u with u L 2 F T t; R m is caed an open-oop equiibrium pair of Probem LQ for the initia pair t x if X t x and J X ; u T J X ; u u T+1 5 hods for any u L 2 F ; Rm and T t. Here u T and u T+1 are the restrictions of u on T and T +1 respectivey. Furthermore such a u is caed an openoop equiibrium contro for the initia pair t x. For a u L 2 F T t; R m the requirement that u is F 1 - measurabe is parae to the standard statement on the admissibe contros of continuous-time stochastic optima contro; see 9] 23] for detais. In other words u is determined from { x 0 w 0... w 1 } ony irrespective of how the state process deveops. From this and the standard arguments about the open-oop contro 2] u L 2 F T t; R m can be viewed as an open-oop contro. Hence we ca u an open-oop equiibrium contro. Furthermore noting that u T u u T+1 the contro u u T+1 on the right hand of the inequaity of 5 differs from u T ony at stage. Therefore 5 is viewed as a oca optimaity condition. Simiary to 16] we can show that {u t... u } is the ash equiibrium of a muti-person game with hierarchica structure. By its definition an open-oop equiibrium contro u is timeconsistent in the sense that for any T t u T is an open-oop equiibrium contro for the initia pair X. In other words u is time-consistent aong the trajectory of the equiibrium state X. By a formua of the difference of cost functionas we can derive the foowing necessary and sufficient condition to the existence of the open-oop equiibrium pair for a given initia pair whose proof is omitted due to space imitations. Theorem 2.1 Given t T and x L 2 F t; Rn the foowing statements are equivaent. i There exists an open-oop equiibrium pair of Probem LQ for the initia pair t x. ii There exists a u L 2 F T t; R m such that for any T t the foowing FBS E admits a soution

3 X tx Z tx X tx +1 A X tx Z tx X tx Z tx T + B u + C X tx + D u + ĀE X tx + B E u + C E X tx + D E u w A T E Z tx +1 + ĀT E Z tx +1 + C T E Z tx +1 w + C T E Z tx +1 w + Q X tx X G X tx with the stationary condition 0 R + R u + Q E X tx + ḠE X tx + B + B T E Z tx +1 + D + D T E Z tx +1 w 7 and the convexity condition 0 E u T R + R u ] + + E Y ū + E Y ū E Y ū T Q Y ū ] T Q E Y ū T G Y ū 6 ] + E Y ū T Ḡ E Y ū.8 In the above Y ū and X are given respectivey by Y ū +1 A Y ū + ĀE Y ū + C Y ū + C E Y ū w T +1 Y ū +1 B + B ū + D + D ū w Y ū 0 and X ĀX +1 A + + C + C X X t x T t. + B + B u + D + D u ] w In this case X u given in ii is an open-oop equiibrium pair. Though the means of the state and contro appear in both the system dynamics and the cost functiona the stationary condition 7 does not contain the means. This is different from the case of static mean-fied LQ optima contro 17]. Here the static mean-fied LQ optima contro is a variant of Probem LQ where the conditiona expectation operator E t is repaced by the expectation operator E and the matrices in the system and cost functiona are independent of the initia time. Another interesting phenomenon is that the system 9 for the equiibrium pair doesn t contain the mean-fied terms too. To simpify the notations et A A + Ā B B + B C C + C D D + D Q Q + Q R R + R G G + Ḡ T t T. 9 Reca the pseudo-inverse of a matrix. By 20] for a given matrix M R n m there exists a unique matrix in R m n denoted by M such that { MM M M M MM M MM T MM M M T M 10 M. This M is caed the Moore-Penrose inverse of M. The foowing emma is from 1]. Lemma 2.1 Let matrices L M and be given with appropriate size. Then LXM has a soution X if and ony if LL MM. Moreover the soution of LXM can be expressed as X L M + Y L LY MM where Y is a matrix with appropriate size. The foowing theorem is concerned with the necessary condition on the existence of the open-oop equiibrium pair for a fixed initia pair. Theorem 2.2 Let Probem LQ for the initia pair t x admit an open-oop equiibrium pair. Then the foowing set of LDEs P Q + A T P +1A + C T P +1C P Q + A T P +1A + C T P +1C P G P G T R + B T P +1B + D T P +1D 0 T t 11 is sovabe in the sense that the foowing inequaities hod { R + B T P +1B + D T P +1D 0 12 T t. Furthermore we have Z tx P X t T + T X E X t + P E X tx E X + T E X and an open-oop equiibrium contro is given by u W H X T t. 13 Here {T T T t } {T T T t } {W T t } and {H T t } are given respectivey by T A T T +1A + C T T +1C A T P +1B + A T T +1B + C T P +1D + C T T +1D W H T A T T +1A + C T T +1C A T P +1B + A T T +1B + C T P +1D + C T T +1D W H T 0 T 0 T T t 14

4 and W R + B T P+1 + T +1 B + D T P+1 + T +1 D H B T P+1 + T +1 A + D T P+1 + T +1 C. Proof. Denote the righthand side of 8 by Ĵ 0; ū. We then have Ĵ 0; ū E Y ū T Q Y ū + ū T R ū + E Y ū + E Y ū T Ḡ E Y ū E Y ū T Q Y ū + Y ū +1 T P +1 Y ū +1 Y ū + E Y ū +1 T P+1 E Y ū + ū T R ū +1 + E Y ū T G Y ū ] T Q E Y ū ] + E Y ū T Q E Y ū T P Y ū +1 E Y ū E E Y ū T Q + A T P +1 A + C T P +1 C P E Y ū + Y ū E Y ū T Q + A T P +1 A + C T P +1 C P Y ū E Y ū ] T P E Y ū ] + ū T R + B T P +1 B + D T P +1 D ū ū T R + B T P +1 B + D T P +1 D ū. By this and 8 we have the sovabiity of 11. Let u be an open-oop equiibrium contro. Then from Theorem 2.1 for any T t the FBS E 6 admits a soution and 7 hods. As Z tx we have by 7 G X tx + ḠE X tx 0 R u + B T G E X ] + g + DG T E X w. Substituting X and by Lemma 2.1 we have u W H X Ψ X where W R + B T G B + D T G D H B T G A + D T G C. Simiary we have Z 2tx P 2 + T 2 X From 7 we have for 2 + P 2 + T 2 E 2 X. 0 R 2 2 u 2 + B 2 2 T P 2 + T 2 E 2 X ] + D 2 2 T P 2 + T 2 E 2 X w 2. Substituting X 2 and by Lemma 2.1 we have u 2 W 2 H 2X 2 Ψ 2X 2 where W 2 R B T 2 2 P 2 + T 2 B D 2 2 T P 2 + T 2 D 2 2 H 2 B 2 2 T P 2 + T 2 A D T 2 2 P 2 + T 2 C 2 2. Bacwardy repeating above procedure we can get 14 and 15. This competes the proof. Theorem 2.3 For any t T and any x L 2 F t; Rn Probem LQ for the initia pair t x admits an open-oop equiibrium pair if and ony if P Q + A T P +1A + C T P +1C P Q + A T P +1A + C T P +1C P G P G T R + B T P +1B + D T P +1D 0 T and the set of GDREs T A T T +1A + C T T +1C A T P +1B + A T T +1B + C T P +1D + C T T +1D W H T A T T +1A + C T T +1C A T P +1B + A T T +1B + C T P +1D + C T T +1D W H T 0 T 0 T W W H H 0 T are sovabe where W R + B T P+1 + T +1 B + D T P+1 + T +1 D H B T P+1 + T +1 A + D T P+1 + T +1 C

5 In this case an open-oop equiibrium contro for the initia pair t x is given in u W H X T t. 18 Proof. Sufficiency. Introduce a dynamics X +1 A B W H X + A B W H and a contro X t x T t ũ W H X w 19 X T t. 20 Then by reversing the first part of the proof of Theorem 2.2 we can show that for any T t the foowing FBS E admits an adapted soution X tx +1 A Xtx tx X tx T + B ũ + C Xtx + D ũ A T E with property and tx 0 R ũ + C T E + Q Xtx + ĀE Xtx + B E ũ + C E Xtx + D E ũ tx +1 + ĀT E tx X tx P + T X tx tx +1 w +1 w + C T E + Q E Xtx X G Xtx tx +1 w + ḠE Xtx E Xtx + P E Xtx E X + T E X + BE T tx +1 + DT tx E +1 w. Furthermore by 12 and 15 we have 8. From Theorem 2.1 Probem LQ for the initia pair t x admits an openoop equiibrium pair and X ũ is an equiibrium pair. ecessity. By Theorem 2.2 we need ony to prove W W H H 0 T. 21 Consider Probem LQ for the initia pair 1 x with x L 2 F 1; Rn. By the proof of Theorem 2.2 we have 0 W u x + H X x. 22 Let e i be a R n -vaued vector with the i-th entry being 1 and other entries 0. Then we have 0 W u e 1... u en + H e1... e n. oting that e 1... e n is the identity matrix and by Lemma 2.1 we have W W H H 0. Considering Probem LQ for the initia pair 2 x with x L 2 F 2; Rn we can simiary prove W 2 W 2 H 2 H 2 0. Continuing above procedure we then achieve the concusion. ote that P P T T +1 are symmetric. If Q Q R R are seected such that Q Q + Q R R + R 0 T T then 15 is sovabe. Furthermore Θ {P P T T π } are used to express Z tx. {P P } is then caed the symmetric part of Θ and {T T } are viewed as the nonsymmetric part. 3 Concusion In this paper the open-oop time-consistent equiibrium contro is investigated for a ind of mean-fied stochastic LQ probem. ecessary sufficient conditions are presented for the cases with a fixed initia pair and a the initia pairs. Furthermore the GDREs and LDEs are introduced to characterize the open-oop equiibrium contro. For future research the cosed-oop time-consistent soution shoud be studied. References 1] M. Ait Rami X. Chen and X. Y. Zhou. Discrete-time indefinite LQ contro with state and contro dependent noises. Journa of Goba Optimization 2002 vo ] K. J. Astrom. Introduction to Stochastic Contro Theory. A- cademic Press Inc. ew Yor Mathematics in Science and Engineering 1970 vo.70. 3] S. Basa and G. Chabaauri. Dynamic mean-variance asset aocation. Rev. Financia Stud vo ] A. Bensoussan J. Frehse and P. Yam. Mean fied games and mean fied type contro theory. Spriger Briefs in Mathematics ] T. Bjor A. Murgoci. A genera theory of Marovian time inconsisiten stochastic contro probem. Woring paper. 6] I. Eeand A. Lazra. Being serious about non-commitment: subgame perfect equiibrium in continuous time. Preprint U- niversity of University British Coumbia ] I. Eeand T. Privu. Investment and consumption without commitment. Preprint University of University British Coumbia ] R.J. Eiott X. Li and Y.H. i. Discrete time mean-fied s- tochastic inear-quadratic optima contro probems. Automatica 2013 vo.49 no ] W.H. Feming H.M. Soner. Controed Marov Processes and Viscosity Soutions Second Edition. Springer Science+Business Media Inc ] S. M. Godman. Consistent pan. Review of Economic Studies 1980 vo ] M. Huang P.E. Caines and R.P. Mahame. Largepopuation cost-couped LQG probems with nonuniform a- gents: individua-mass behavior and decentraized ε-ash equiibria. IEEE Transactions on Automatic Contro 2007 vo ] Y. Hu H. Jin and X.Y. Zhou. Time-inconsistent stochastic inear-quadratic contro. SIAM Journa on Contro and Optimization 2012 vo ] P. Kruse A. A. Smith. Consumption and savings decisions with quasi-georetric discounting. Econometrica 2003 vo.71 no

6 14] D. Laibson. Goden eggs and hyperboic discounting. The Quartery Journa of Economics 1997 vo ] J.M. Lasry and P.L. Lions. Mean-fied games Japanese Journa of Mathematics 2007 vo ] X. Li Y.H. i and J.F. Zhang. On time-consistent soution to time-inconsistent inear-quadratic optima contro of discretetime stochastic systems. Submitted to IEEE Transactions on Automatic Contro ] Y.H. i J.F. Zhang and X. Li. Indefinite mean-fied stochastic inear-quadratic optima contro. IEEE Transactions on Automatic Contro 2015 vo.60 no ] Y.H. i X. Li and J.F. Zhang. Indefinite mean-fied stochastic inear-quadratic optima contro: from finite horizon to infinite horizon. IEEE Transactions on Automatic Contro DOI: /TAC ] I. Paacios-Huerta. Time-inconsistent preferences in Adam Smith and Davis Hume. History of Poitica Economy 2003 Vo ] R. Penrose. A generaized inverse of matrices. Mathematica Proceedings of the Cambridge Phiosophica Society 1955 vo ] A. Smith. The theory of mora sentiments. First Edition 1759; Reprint Oxford University Press ] R.H. Strotz. Myopia and inconsistency in dynamic utiity maximization. Review of Economic Studies 1955 vo ] J.M. Yong and X.Y. Zhou. Stochastic contros: Hamitonian systems and HJB equations. Springer ew Yor ] J.M. Yong. A deterministic inear quadratic time-inconsitent optima contro probem. Mathematica Contro and Reated Rieds 2011 vo.1 no ] J.M. Yong. Deterministic time-inconsistent optima contro probems an essentiay cooperative approach. Acta App. Math. Sinica 2012 vo ] J.M. Yong. A inear-quadratic optima contro probem for mean-fied stochastic differentia equations. SIAM Journa on Contro Optimization 2013 vo ] J.M. Yong. Linear-quadratic optima contro probems for mean-fied stochastic differentia equations time-consistent soutions. Eectronicay pubished by Transactions of the American Mathematica Society DOI: /tran/