Nonlinear n-th Cost Cumulant Control and Hamilton-Jacobi-Bellman Equations for Markov Diffusion Process

Transcription

1 Nonlnear n-th Cost Cumulant Control and Hamlton-Jacob-Bellman Equatons for Markov Dffuson Process Chang-Hee Won, IEEE Member Abstract A general nonlnear stochastc system wth nonquadratc cost functon s consdered for cost cumulant control of a Markov dffuson problem The Hamlton-Jacob-Bellman equaton for the n-th cost moment case s derved as a necessary condton for optmalty The n-th cost cumulant Hamlton- Jacob-Bellman equaton dervaton procedure s gven Second, thrd, and fourth cost cumulant Hamlton-Jacob-Bellman equatons are derved usng the proposed procedure The solutons of the nonlnear cost cumulant control problem s dscussed usng the state dependent Rccat equaton method I INTRODUCTION We propose to mnmze a nonlnear Markov dffuson process usng the cumulants of a non-quadratc cost functon The fnal objectve of cost cumulant control, whch s also known as statstcal control or cost densty control, s to manpulate the cost dstrbuton by controllng each cumulant The cost cumulant control started wth San An open-loop mnmum cost varance problem was solved by hm n [8] San and Lberty contnued to study the characterstcs of cost cumulant control n [0] Lberty studed the quadratc nature of the mnmal cost cumulant control n [5] However ther study was restrcted to the lnear system and quadratc cost case The relatonshp between the cost cumulant control and rsk-senstve control was establshed n [] San et al publshed results for the second cost cumulant control n the full-state-feedback case [] Why cumulants? Cumulants have more ntutve meanngs than moments The frst cumulant s the mean, the second cumulant s the varance, and the thrd cumulant s the skewness and the fourth cumulant s the kurtoss The second cumulant shows the varaton of the dstrbuton around the mean, the thrd cumulant mples the departure from symmetry, and the fourth cumulant shows the flatness of the dstrbuton Moreover, hgher order cumulants have decreasng sgnfcance, thus appromatng usng frst few cumulants are a good appromaton to the general problem Secton II dscusses the necessary mathematcal prelmnares, Secton III presents the Hamlton-Jacob-Bellman HJB equaton for the n-th moment case, Secton IV dscusses the n-th cumulant HJB equaton, and Secton V dscusses the soluton procedure for the cost cumulant control problem Fnally, conclusons are gven n the last secton Ths materal s supported n part by the Natonal Scence Foundaton under grant number ECS and by the Army under the grant number W9NF Chang-Hee Won s wth the Department of Electrcal and Computer Engneerng, Temple Unversty, Phladelpha, PA 9, USA cwon@templeedu II MATHEMATICAL PRELIMINARIES Consder the nonlnear stochastc dfferental equaton: dt = ft, t, utdt + σt, tdwt, where t [t 0, t F ], t 0 = 0, t IR n s the state, ut U s the control acton, and dwt s a Gaussan random process of dmenson d wth zero mean and covarance of Wtdt A memoryless feedback control law s ntroduced as ut = kt, t, where k s a nonrandom functon wth random arguments Also, consder a non-quadratc cost functon tf [ ] Jt, t; k = Ls, s, ks, s ds + ψt F t 3 Let Q 0 = [t 0, t F IR n and Q 0 denote the closure of Q 0 Assume that L s contnuously dfferentable and satsfes the polynomal growth condton We assume that f, σ, and k satsfy the Lpschtz condton and the lnear growth condton The moments are defned as M t, ; k = E J t, ; k t = Ok s the backward evoluton operator gven by Ok = ft, +, kt,, + tr σt, Wtσ t, where suppressng arguments n f, = f = O k = tr σwσ n = σwσ j = O k j,j= Now we defne a moment around an arbtrary pont, a, usng Steltjes ntegral M = a df where = 0,,, and M 0 = by defnton The moments are a set of descrptve constants of a dstrbuton 4 5

2 that are useful for measurng ts propertes Formally, the cumulants V, V,,V are defned by the dentty n t t t ep V = M!! = =0 Note that V 0 s not defned Furthermore, we have the followng moment characterstcs functon and cumulant generatng functons, φt = e jt df and ψt = log φt, where j s the comple operator, j = Now, we present the moment-cumulant relatonshp from [4] r π π M M V r = r!!! = π π M ρ ρ! 6! π! π! where π +π + +π = ρ, π +π + +π = r, and π s are non-negatve ntegers r π π V V M r =!! = π π V r!! π! π! 7 where π + π + + π = r and π s are non-negatve ntegers III N-TH MOMENT HAMILTON-JACOBI-BELLMAN EQUATION Here we derve the Hamlton-Jacob-Bellman HJB equaton for n-th moment of the cost functon, whch s a necessary condton for the optmalty We wll assume the estence of an optmal controller The followng algebrac dentty wll be used n the sequel Also, for brevty we have removed all the proofs Proofs are avalable by contactng the author Algebrac Identty: a+b n = n n k a k b n k where n k = k=0 n! k!n k! Theorem 3: Let M be an admssble mean cost functon and K M be the correspondng class of control laws Assume M t, C, p Q 0 and the estence of an optmal controller km M, where =, 3, Then km M and M t, satsfy the partal dfferental equaton Ok M M [M t, ] + M t, Lt,, k M M = 0 8 for t T, IR n, where Ok M M [M t, ] + M t, Lt,, k M M = mn k K M Ok[M t, ] + M t, Lt,, k, 9 along wth the boundary condton M t F, = M t F, = ψ t F, =,, 3, 0 Theorem 3 gves the HJB Equaton for any n-th moment of the cost functon We wll utlze ths theorem to fnd n-th cumulant HJB equaton n the net secton IV N-TH CUMULANT HAMILTON-JACOBI-BELLMAN EQUATION In order to fnd n-th cumulant HJB equaton, we wll utlze the moment-cumulant relatonshp and n-th moment HJB equaton Lemma 4: The -th cost moment, M t, ; k s related to -th cost moment, M t, ; k by the followng partal dfferental equaton Ok[M t, ; k] + M t, ; klt,, k = 0 wth the boundary condton M t f, ; k = ψ t F where =,, Note that frst, second, and thrd moment HJB equatons are gven respectvely as 0 = OkM + L 0 = OkM + M L 0 = OkM 3 + 3M L Lemma 4: The powers of cost moments M are related by the followng partal dfferental equaton Ok[M p t, ; k] = pmp t, ; klt,, k pp + M p t, ; k M σwσ Lemma 43: The powers of the cost moments M M j are related by the followng partal dfferental equaton Ok[M p Mq j ] = pmp M q j M L qjm p Mq j + pp +qpm p M j L M q j M p M q j tr M σwσ σwσ Mj M M j σwσ qq + M p Mq j Now we propose a procedure to fnd the n-th order cumulant HJB equaton: Use Equaton 6 or 7 to fnd the relatonshp between n-th moment and n-th cumulant Substtute M n nto Equaton 9 and fnd the relatonshp between the lower order moments and cumulants Prove that the above condton n step two s vald and determne the n-th cumulant HJB equaton Usng ths procedure t s possble to determne any n-th cumulant HJB equaton As eamples, we wll fnd second, thrd and fourth cumulant HJB equatons

3 Theorem 4: Let M Cp, Q 0 be an admssble mean cost functon, and let M nduce a nonempty class K M of admssble control laws Assume the estence of an optmal control law k = kv M and an optmum value functon V C, p Q 0 Then the mnmal second cumulant varance functon V satsfes the followng HJB equaton mn Ok[V t, ] + V t, k K M = 0 3 σwσ for t, Q 0, together wth the termnal condton, V t F, = 0 Here, we present the thrd cumulant HJB equaton Theorem 4: Let M Cp, Q 0 be an admssble mean cost functon, and let M nduce a nonempty class K M of admssble control laws Assume the estence of an optmal control law k = kv 3 M and an optmum value functon V3 Cp, Q 0 Then the mnmal thrd cost cumulant skewness functon V3 satsfes the followng HJB equaton mn Ok[V3 ] + 3tr σwσ V V = 0 k K M 4 for t, Q 0, together wth the termnal condton, V3 t F, = 0 The mnmzaton of second cumulant varance depends on the defnton of the frst cumulant mean value functon V as can be seen from Equaton 3 The mnmzaton of thrd cumulant skewness depends on the defnton of both frst and second cumulants value functons V and V as can be seen from Equaton 4 Here, we present the fourth cumulant kurtoss HJB equaton Theorem 43: Let M Cp, Q 0 be an admssble mean cost functon, and let M nduce a nonempty class K M of admssble control laws Assume the estence of an optmal control law k = kv 4 M and an optmum value functon V4 Cp, Q 0 Then the mnmal fourth cost cumulant kurtoss functon V4 satsfes the followng HJB equaton mn Ok[V4 ] + tr σwσ V V3 k K M + V + 3 V V V = 05 σwσ σwσ for t, Q 0, together wth the termnal condton, V4 t F, = 0 Usng the presented method the frst to ffty order cumulant HJB equatons can be determned To fnd the hgher order 0 > n 6, we requre more partal dfferental equaton lemmas such as the one s for Ok[M p Mq j Mr k ] The necessary number of partal dfferental equatons are summarzed n the followng table Table : Necessary Equatons for n-th Cumulant Problem No Necessary Theorem Order of Cumulants Ok[M p ] 3 > n Ok[M p Mq j ] 6 > n 3 3 Ok[M p Mq j Mr k ] 0 > n 6 m Ok[M p Mq j Mm η ] m+ = > n m = V SOLUTIONS OF COST CUMULANT CONTROL The general procedure for fndng the solutons of cost cumulant control of a nonlnear system s dscussed n ths secton The solutons of frst cumulant optmzaton LQGof a lnear system s well known [3] The second cumulant mnmal cost varance case s gven n [] Here, we derve the optmal controller for frst two cumulants and frst three cumulants of a general nonlnear system The procedure to fnd the optmal controller s gven as follows Decde on the number of cost cumulants to mnmze Derve the HJB partal dfferental equatons for each of the cost cumulants Create the partal dfferental equaton to mnmze usng Lagrange multplers Solve for the optmal controller Substtute the optmal controller back to the HJB equaton and solve the HJB equaton There are a number of approaches that one can take to obtan the solutons of the HJB equatons: Assume a lnear system wth quadratc cost and solve for the optmal controller, fnd solutons of the HJB equaton numercally [], [6], or 3 utlze the state dependent Rccat equaton SDRE approach [] In the sequel, frst and thrd methods are nvestgated Clouter and others have ntroduced the state-dependent Rccat equaton SDRE approach to solve varous nonlnear regulator problems [] Ther approach s however for the tme-nvarant, nfnte-tme-horzon, nonlnear, and determnstc systems We epand on ther dea to the tmevaryng, nonlnear, fnte-tme-horzon, and stochastc systems Then we wll assume a lnear system wth quadratc cost and solve the cost cumulant problem and verfy wth the estng results A Optmzng Solutons of the Frst Two Cumulants Here we fnd the controller, k, that wll mnmze the value functon, V t, + γ tv t,, where γ t s a tme varyng Lagrange multpler From the prevous secton, we have the followng two partal dfferental equatons as the necessary condtons for optmalty V +g V +k B V + tr σwσ V +h+k Rk = 0 6

4 and V + V g + k B V + tr σwσ V +tr σwσ V V = 0 7 Now we use the Lagrange multpler method Introduce a tme varyng Lagrange multpler, γ t, and optmze the HJB equaton for the frst cumulant plus the Lagrange multpler tmes the HJB equaton of the second cumulant 0 = mn k + tr V + g V + k B V + h + k Rk + γ t σwσ V [ V + g V + k B V +tr σwσ V ] V + tr σwσ V The mnmzng controller s obtaned as k = [ R t, B V t, + γ t V ] 8 Second order necessary condton, R > 0, s satsfed also Therefore the mnmum s guaranteed, and the controller 8 s a canddate for an optmal frst and second cost cumulant controller Now we wll fnd the solutons to the nonlnear frst and second cost cumulant optmzaton problem usng the SDRE method Nonlnear System Frst and Second Cumulant Mnmzaton: Assume that and ht, = Qt, gt, = At, Now, we fnd the two partal dfferental equatons from the necessary condtons of optmalty, Equatons 6 and 7 To fnd the frst partal dfferental equaton, we substtute the optmal controller, Equaton 8, and the above assumptons nto the Equaton 6 Suppressng arguments for smplcty, we have 0 = V + A V V BR V B + Q V γ BR V B + tr σwσ V + V BR B V 4 + γ V BR B V 4 + γ 4 V BR B V + γ 4 V BR B V, whch can be smplfed to 0 = V + A V 4 + tr σwσ V + γ 4 V V BR B V + Q BR B V 9 Ths s one of the necessary partal dfferental equatons for the frst two cumulant problem To fnd the other equaton, we substtute the optmal controller 8 nto the Equaton 7, whch smplfes to V BR B V 0 = V + A V 4 V BR B V 4 + tr V + σwσ V + γ V σwσ V BR B V 0 Assume that V t, and V t, are symmetrc nonnegatve defnte matrces V m t, V m V m V m = V m t, + v m t = Vm t, + v m t = V m t, + vec V m = V m t, + vec V m Vm +vec + vec V m where m =, and vecz = [z, z,, z n ] The soluton must satsfy the followng symmetry condtons n k= V mk j k = n k= V mjk k for m =,, =,,,n and j = +,,n Now, the SDRE method s utlzed Substtute the above equatons nto Equaton 9 and reduce t to 0 = V + Q + v V BR B V vec V BR B V V BR B vec V BR B vec 4 vec V +γ V BR B V + γ V BR B vec V V

5 + γ vec + γ 4 vec BR B V BR B vec V V +tr σwσ V + tr σwσ vec +tr σwσ V vec + σwσ tr vec V + A vec V V V + A V From the above equaton, we obtan the followng necessary condtons for optmalty and v = trσwσ V, 0 = V + Q V BR B V + γ V BR B V +A V + V A, 0 = vec V BR B V V BR B vec V BR B vec 4 vec V + γ V BR B vec V + γ vec V BR B V + γ 4 vec V BR B vec +tr σwσ vec V +tr σwσ V vec + σwσ tr vec V + A vec V V V 3 The last equaton s smlar to Clouter s necessary condton form optmalty n determnstc LQR case [] To fnd the second Rccat-type equaton, we let V t, be a symmetrc nonnegatve defnte matr Then we substtute Equatons wth m = nto Equaton 0 to obtan the followng condtons v = trσwσ V 0 = V V BR B V V BR B V γ V BR B V + 4V σwσ V + A V + V A 4 and 0 = A vec V V BR B vec vec V 4 vec V V BR B vec vec V 4 vec V γ V BR B vec γ vec V V BR B V BR B vec V BR B V BR B vec V BR B V BR B vec γ vec V +tr σwσ vec V +tr σwσ V vec + σwσ tr vec V + V σwσ vec V +vec V σwσ V σwσ vec +vec V V V V V 5 Note that the suppressed arguments for V, V, A, B, Q, and R are t, Thus, above equatons are state dependent In summary, the nonlnear, non-quadratc frst two cost cumulant optmzaton problem has the optmal controller gven by k = [ R t, B t, V + vec + γ t V + vec V ], V where V s must satsfy two coupled Rccat-type Equatons and 4 Furthermore, two necessary condtons gven by Equatons 3 and 5 must be satsfed In general these necessary condtons are dffcult to satsfy for a gven At, n multvarable case Thus, n the net secton, we assume a lnear system wth a quadratc cost functon to present the complete soluton of the frst two cost cumulant optmzaton problem

6 Frst and Second Cost Cumulant Mnmzaton: Lnear System: Now we assume a lnear system and a quadratc cost functon We verfy that ths soluton s equvalent to the mnmal cost varance soluton of [], [7] Assume that ht, = Qt, gt, = At, and σt, = Et, thus we have Lt,, kt, ψt F ft,, kt, = Qt + k Rtk, = t F Q F t F, and = At + Btkt, Furthermore, we assume quadratc form solutons V = V + v V = V + v V = V V + v = V + v V = V V = V V = V V = V 6 Substtute the above equatons nto Equatons 8 to obtan the optmal controller, k = R B V + γ V Substtutng Assumptons n 6 nto Equatons 6 and 7 we obtan the followng coupled Rccat-type equatons 0 = V + Q V BR B V + γ V BR B V +A V + V A 0 = V V BR B V V BR B V γ V BR B V + 4V σwσ V + A V + V A wth the boundary condtons V t F = Q F and V t F = 0 And the followng equatons: v = trσwσ V v = trσwσ V The suppressed arguments n the above equatons are t The above equatons are equvalent to the mnmal cost varance soluton obtaned n [] If we were to perform cost moment mnmzaton, we wll obtan nonlnear controller even for the lnear, quadratc cost case To see ths, consder two partal dfferental equatons for the frst two moments from Equaton 8: 0 = M + h + k Rk + tr σwσ M +g M + k B M 0 = M + M h + k Rk + tr σwσ M +g M + k B M After multplyng a Lagrange multpler and takng dervatves wth respect to k, we fnd the optmal frst two cumulant controller as k = R B M + 4γM + γ M If we let M t, = M + m for =,, we obtan a nonlnear controller VI CONCLUSIONS Ths paper presented the necessary condton for n-th cost moment optmzaton problem n the form of HJB equaton The n-th cost cumulant optmzaton procedure s also dscussed n detal Second, thrd, and fourth cost cumulant HJB equatons are derved utlzng the proposed procedure Thus the necessary condton for any cost moment and cumulant can be derved The solutons for any cost cumulant problem s also nvestgated n ths paper We proposed to use the Lagrange multpler method wth the HJB equatons Optmzng solutons for the frst two cumulants are presented For a nonlnear system wth nonquadratc cost problem, we used the state dependent Rccat equaton technque to fnd an optmal soluton REFERENCES [] R W Beard, G N Sards, and J T Wen, Appromate Solutons to the Tme-Invarant Hamlton-Jacob-Bellman Equaton, Journal of Optmzaton Theory and Applcatons, 998 [] J R Clouter, C N D Souza, and C P Mracek, Nonlnear Regulaton and Nonlnear H Control Va the State-Dependent Rccat Equaton Technque: Part I, Theory; Prat II, Eamples Proceedngs of the Internatonal Conference on Nonlnear Problems n Avaton and Aerospace, pp 7 4, May 996 [3] W H Flemng and R W Rshel, Determnstc and Stochastc Optmal Control New York: Sprnger-Verlag, 975 [4] A Stuart and J K Ord, Kendall s Advanced Theory of Statstcs, Volume Dstrbuton Theory, Ffth Edton, Oford Unversty Press, New York, 987 [5] S R Lberty and R C Hartwg, On the Essental Quadratc Nature of LQG Control-Performance Measure Cumulants, Informaton and Control, Volume 3, Number 3, pp , 976 [6] C L Navasca and A J Krener, Soluton of Hamlton Jacob Bellman Equatons, Proceedngs of the 39th IEEE Conference on Decson and Control, Sydney, Australa, pp , December 000 [7] K D Pham, S R Lberty, and M K San, Lnear Optmal Cost Cumulant Control: A k-cumulant Problem Class, Proceedngs of the Thrty-Sth Annual Allerton Conference on Communcaton, Control, and Computng, Urbana-Champagn, 998 [8] M K San, Control of Lnear Systems Accordng to the Mnmal Varance Crteron A New Approach to the Dsturbance Problem, IEEE Transactons on Automatc Control, Volume AC-, Number, pp 8, January 966 [9] M K San, Performance Moment Recursons, wth Applcaton to Equalzer Control Laws, Proceedngs of 5th Allerton Conference, pp , 967 [0] M K San and S R Lberty, Performance Measure Denstes for a Class of LQG Control Systems, IEEE Transactons on Automatc Control, Volume AC-6, Number 5, pp , October 97 [] M K San, C-H Won, and B F Spencer, Jr Cumulant Mnmzaton and Robust Control, Stochastc Theory and Adaptve Control, Lecture Notes n Control and Informaton Scences 84, T E Duncan and B Pask-Duncan Eds, Sprnger-Verlag, pp 4 45, 99 [] M K San, C-H Won, B F Spencer, Jr, and Stanley R Lberty, Cumulants and Rsk-Senstve Control: A Cost Mean and Varance Theory wth Applcaton to Sesmc Protecton of Structures, Advances n Dynamc Games and Applcatons, Annals of the Internatonal Socety of Dynamc Games, Volume 5, pp , Jerzy A Flar, Vladmr Gatsgory, and Koch Mzukam, Edtors Boston: Brkhauser, 000