Off-policy Reinforcement Learning for Robust Control of Discrete-time Uncertain Linear Systems

Transcription

1 Off-polcy Renforcement Learnng for Robust Control of Dscrete-tme Uncertan Lnear Systems Yonglang Yang 1 Zhshan Guo 2 Donald Wunsch 3 Yxn Yn 1 1 School of Automatc and Electrcal Engneerng Unversty of Scence and echnology Bejng Bejng P R Chna E-mal: yyang2016@eeeorgyyx@esustbeducn 2 Department of Computer Scence Mssour Unversty of Scence and echnology Rolla MO USA E-mal: guozh@mstedu 3 Department of Electrcal and Computer Engneerng Mssour Unversty of Scence and echnology Rolla MO USA E-mal: wunsch@eeeorg Abstract: In ths paper an off-polcy renforcement learnng method s developed for the robust stablzng controller desgn of dscrete-tme uncertan lnear systems he proposed robust control desgn conssts of two steps Frst the robust control problem s transformed to an optmal control problem Second the off-polcy RL method s used to desgn the optmal control polcy whch guarantees the robust stablty of the orgnal system wth uncertanty he condton for the equvalence between the robust control problem and the optmal control problem s dscussed he off-polcy does not requre any knowledge of the system knowledge and effcently utlze the data collected from on-lne to mprove the performance of approxmate optmal control polcy n each teraton successvely Fnally a smulaton example s carred out to verfy the effectveness of the presented algorthm for the robust control problem of dscrete-tme lnear system wth uncertanty Key Words: system uncertanty robust control optmal control off-polcy trnforcement learnng model-free 1 Introducton he models of real world physcal systems are usually coupled wth model uncertanty whch s challengng to the feedback control desgn Robust control s desgned to deal wth the uncertan parameters and structures wthn a certan bound n order to acheve the guaranteed performance In the early tme robust control desgn was based on frequency doman analyss 1 Snce then tme doman based approaches were also developed to nvestgate the robust stablzaton problem for both lnear and nonlnear systems 2 3 he adaptve control technques have been appled successfully to deal wth mult-agent systems 4 tme delay systems 5 7 and so on Another knd of robust control approach s proposed based on the optmal control desgn method In 8 the robust control problem s transformed to an optmal control problem wth a modfed system dynamcs and a performance functon he robust control problem s equvalent to the optmal control problem n the sense that the unque optmal control of the transformed optmal control problem s able to robustly stablze the orgnal system wth uncertanty In ths way the robust control problem reduces to the soluton of the Hamlton-Jacob-Bellman HJB equaton for a general nonlnear system and the Rccat equaton for the lnear system hs dea has been successfully appled to guaranteed cost regulaton problem n 9 guaranteed cost trackng problem n 10 robust control of uncertan constraned systems n 11 robust optmal control desgn n It s necessary to develop an effcent method for the transformed optmal control problem whch s able to equvalently tackle the robust control problem hs s the motvaton for ths hs work was supported n part by the Mary K Fnley Mssour Endowment the Mssour S& Intellgent Systems Center the Natonal Scence Foundaton the Natonal Natural Scence Foundaton of Chna NSFC Grant No and the Chna Scholarshp Councl CSC No paper An effcent method referred to as approxmate/adaptve dynamc programmng ADP or renforcement learnng RL was proposed n 14 he core dea of ADP as the name ndcates s to fnd the soluton that satsfes the HJB equaton For lnear systems the soluton of Rccat equaton was approxmated successvely by solvng a sequence of Lyapunov equatons n 15 hs dea was extended to nonlnear contnuous-tme systems n 16 Snce then ADP developed from off-lne 17 to on-lne from modelbased 20 to unknown systems 21 A novel RL approach called off-polcy RL was presented to approxmate the optmal control polcy n an on-lne manner by a novel polcy teraton algorthm for the contnuous-tme lnear systems wthout any knowledge of the system dynamcs 22 hen the off-polcy RL method was extended to H control problem n trackng control problem n 25 and the output synchronzaton problem of mult-agent system n 26 o the authors knowledge the off-polcy RL method has not been appled to the robust control problem for dscretetme uncertan lnear systems yet he man contrbuton of ths paper s to utlze the off-polcy RL approach to solve the robust control problem of dscrete-tme uncertan lnear systems wthout requrng any knowledge of the systems dynamcs he remander of ths paper s organzed as follows Secton 2 descrbes the robust control problem of dscrete-tme lnear system wth uncertanty In Secton 3 the robust control problem s transformed to the optmal control problem of a modfed system he condton for the equvalence that the optmal control polcy of the modfed system can robustly stablze the orgn uncertan system s also gven n Secton 3 he model-free off-polcy RL method to solve the optmal control problem s descrbed n Secton 4 In Secton 5 a smulaton s conducted to demonstrate the valdty of the proposed approach Fnally concludng remarks

2 and future works are presented n Secton 5 2 Problem Statement Consder a class of dscrete-tme lnear systems wth uncertanty x k+1 = A + p x k + Bu k 1 where x k R n s the system state u k R m1 s the control nput A + R n n represents the drft dynamcs and B R n m1 s the nput dynamcs p wth p Ω p represents the bounded perturbaton of the nomnal system x k+1 = Ax k + Bu k 2 Wthout loss of generalty t s assumed that A B 1 s stablzable and x = 0 s an equlbrum state of system 1 he system uncertanty can be classfed nto two types the matched and msmatched uncertanty 8 he class of matched uncertanty satsfes p = Bφ p 3 hat s the matched uncertanty p n 3 can only descrbe the class of uncertanty n the space spanned by the columns of the nput matrx B 1 herefore another type of uncertanty unmatched uncertanty s consdered p = BB p + I BB p 4 where B 1 s the pseudo-nverse of B 1 In 4 the uncertanty p s composed of the matched component B 1 B 1 p and the unmatched component I B 1 B 1 p Assume that the uncertanty p s upper bounded by the followng nequalty ε 1 p p F p Ω p 5 where ε 0 s a desgn parameter to be determned he robust control problem of system 1 s formulated as the followng Robust Control Problemo fnd a state feedback control law u k = Kx k such that the close-loop system x k+1 = A + BK x k + p x k = A c x k + p x k 6 s asymptotcally stable for p Ω p In order to solve the robust control problem the feedback gan can be obtaned by an optmal control desgn method descrbed as followng Optmal Control Problem Consder the modfed nomnal system x k+1 = Ax k + Bu k + Dv k 7 wth the performance descrbed as J x k u k = 1 2 x j Qx j +x j F x j + β 2 x j x j j=k + u j R 1 u j + v j R 2 v j 8 where D = α I B 1 B 1 R n r and r s the rank of B V x k s also referred to as the value functon 27 he scalars α > 0 β > 0 and the matrces Q 0 R 1 0 and R 2 0 are parameters to be determned For smplcty denote the utlty functon as r x k u k v k = x k Qx k + x k F x k + β 2 x k x k + u k R 1 u k + v k R 2 v k 9 he optmal control problem of system 7 wth respect to the performance 8 s to fnd the optmal state feedback control law u k = K x k and v k = L x k such that the value functon V x k n 8 s mnmzed Remark 1 As shown later n Secton 3 under some specfc condtons K s able to stablze the uncertan system 1 e the optmal soluton of the correspondng optmal control problem can stablze the uncertan system Note that the control nput v k only appears n the modfed nomnal system 7 herefore the feedback gan L does not affect the system 1 drectly However L helps to desgn K to stablze the system 1 ndrectly In the next secton t wll be proved that the optmal feedback gan of system 7 wth respect to the performance 8 K s a stablzng feedback gan for the uncertan system 1 hen the model-free ADP technque can be further employed to the correspondng optmal control problem 3 Optmal Control Desgn Approach for the Robust Control Problem In ths secton the optmal control desgn based approach for the robust control problem s consdered Frst the robust control problem s transformed to an optmal control problem for the system 7 wth respect to the performance 8 he optmalty condton for the optmal control and the expresson of the optmal feedback gan K and L are derved hen the condtons that guarantee the asymptotc stablty of the closed-loop system 6 when the feedback gan K s gven Frst the defnton of admssble control s requred Defnton 1 Admssble Control For the modfed nomnal system 7 the control mappngs u x and v x are sad to be admssble wth respect to performance 8 f 1 u x and v x are contnuous; 2 u 0 = v 0 = 0; 3 u x and v x stablze the modfed nomnal system 7; 4 the value functon V x k s fnte for x k In the optmal control problem of system 7 wth respect to 8 the objectve s to fnd the optmal control u k such that V x k = mn u k J x k u k 10 he Bellman equaton for the value functon V x k n 8 s V x k = V x k+1 + r x k u k v k 11 Defne the Hamltonan as H x k u k v k = x k Qx k + x k F x k + β 2 x k x k +u k R 1 u k + vk R 2 v k + V x k+1 V x k 12

3 For lnear systems the value functon n 8 can be denoted as V x k = x k P x k 13 Based on 27 the necessary condtons for optmal control u k and v k s gven by H x k u k v k u k = 0 H x k u k v k v k = 0 14 Consderng 12 and s equvalent to: R1 + B P B B P D u k B D P B R 2 + D P D vk = P A D P A Denote E = B P A G = D P A and M11 M the block matrx M = 12 = M 21 M 22 R1 + B P B B P D D P B R 2 + D hen the optmal P D control u k and vk can be expressed as u k E vk = M 1 x G k Let M 1 = N be parttoned nto the block form as N N11 N = 12 Based on the matrx nverson lemma n N 21 N N can be expressed as: N 11 = M 11 M 12 M 1 N 12 = M 11 M 12 M22 1 M 1M12 21 M 1 N 21 = M 22 M 21 M11 1 M 1M21 12 M 1 22 M N 22 = M 22 M 21 M 1 11 M Fnally the optmal control can be expressed u k = K x k and v k = L x k wth K = N 11 E + N 12 G 16 L = N 21 E + N 22 G 17 akng E G M and N nto 16 and 17 then the followng can be obtaned K = B P B B P D R 2 + D P D 1 D P B +R 1 1 B P A B P D R 2 + D P D 1 D P A L = D P D D P B R 1 + B P B 1 B P D +R 2 1 D P A D P B R 1 + B P B 1 B P A where P s the soluton of the algebrac Rccat euqaton ARE E M 1 E + A P A P + G G Q=0 18 As mentoned n Remark 1 the optmal soluton to the optmal control problem n 7 and 8 s able to solve the robust stablzaton problem only under some specfc condtons he condtons that guarantee the feedback gan K n 16 asymptotcally stablzes system 1 s provded as the followng theorem heorem 1 Suppose that there exst a scalar ε 0 satsfes 5 and such that ε 1 I P 0 19 hen the state feedback control u k = K x k wth K satsfyng 16 can asymptotcally stablze system 1 f t satsfes the followng nequalty P 1 εi 1 Ac M P 1 M A c +K R 1 K + L R 2 L + Q + β 2 I 20 where M = P 1 + B R 1 1 B + D R 1 2 D 1 A and L satsfes 17 Proof When the feedback gan K n 16 s appled to system 1 t can be shown that the functon V x k s a Lyapunov functon of system 1 f 19 and 20 are satsfed Frst V x k = x k P x k 0 x k 0 snce P s the postve defnte soluton of the ARE 18 Now t remans to show that the tme dfference V x k = V x k+1 V x k 0 x k 0 Insertng the feedback gan K n 16 nto the uncertan closed-loop dynamcs 6 x k+1 = A c + x k 21 where A c = A + BK he tme dfference of V x k along the state trajectory of 21 s V x k = x k A c P A c + P A c + A c P + P x k x k P x k 22 Based on 19 the followng s true ε 1 I P Usng the nequalty a 2 + b 2 2ab and the fact that P and ε 1 I P 1 are postve defnte one can obtan: A c P ε 1 I P 1 P Ac + ε 1 A A P = A c P ε 1 I P 1 P Ac + ε 1 I P A c P + P A c 24 By rearrangng tems n 24 the followng s obtaned A c P + P A c + P A c P ε 1 I P P Ac + ε 1 25 Combnng 22 wth 25 one can obtan V x k x k A c P ε 1 I P P Ac + A c P A c + ε 1 P x k 26 Based on the matrx nverson lemma n 28 the followng s true P ε 1 I P P + P = P 1 εi 1 27 Consderng s equvalent to V x k x k A c P 1 εi 1 Ac + ε 1 P x k 28

4 Consderng P n 18 then 28 s equvalent to V x k x k A c P 1 εi 1 Ac A P 1 + B R1 1 B + D R2 1 D 1 A + ε 1 Q x k 29 Let N = P 1 + B R 1 1 B + D R 1 2 D M = N 1 A then A 1A P 1 + B1 R B2 R2 1 2 = M P 1 M + K R 1 K + L R 2 L 30 Insertng 30 nto 29 yelds V x k = x k A c P 1 εi 1 Ac + ε 1 F M P 1 M K R 1 K L R 2 L Q β 2 I x k 31 Based on 5 V x k 0 f 20 s satsfed hs completes the proof Remark 2 Condtons 19 and 20 guarantee the asymptotc stablty of system 1 when K s appled Note that the optmal feedback gans K and L depend on P the soluton of the ARE 18 Even f P s known the knowledge of the modfed system dynamcs n 7 s stll requred for the optmal feedback gan computaton 4 Off-polcy Renforcement Learnng he condton for the robust stablzaton of the optmal feedback gan K has been derved n 16 In order to solve the ARE 18 the off-polcy renforcement learnng method s developed to derve the optmal feedback gan K L n ths secton he dervaton of the off-polcy renforcement learnng algorthm for the dscrete-tme lnear dynamc system 7 s also derved hrough the dervaton t can be seen that the off-polcy RL algorthm has the mert that the optmal control problem could be solved wthout the requrement of the system knowledge Suppose the admssble polces u k = ux k and u k = ux k are appled to the system 7 he modfed nomnal system 7 can be rewrtten as: x k+1 = A x k + B u k K x k + D vk L x k = where A = A + BK + DL u k = K x k vk = L x k and u 0 x k = K 0 x k v 0 x k = L 0 x k are admssble polces hen the Bellman equaton 11 can be rewrtten as: V x k V x k+1 = r x k u k vk = x k Qx k + x k F x k + β 2 x k x k + u k R1 u k + v k R1 v k 33 where V x k = x k P x k he aylor seres expanson of the value functon Λ x at the state a should be: Λ a Λ x = Λ a + x a a x a 2 Λ a a 2 x a 34 Consderng that V x k = x k P x k then 34 s equvalent to: V x k V x k+1 = 2x k+1p x k x k+1 +x k x k+1 P x k x k+1 35 By takng 32 nto 35 one can obtan: V x k V x k+1 = x k P x k x k A P A x k v k L x k D P x k+1 v k L x k D P A x k u k K x k B P x k+1 u k K x k B P A x k 36 By usng 11 the followng dscrete tme Lyapunov equaton holds: P = Q + F + β 2 I + K R 1 K + L R 2 L + A P A herefore the followng holds: x k P x k x k A P A x k = x k Qx k + x k F x k +β 2 x k x k + x k K R 1 K x k + x k L R 2 L x k 37 Insertng 37 nto 36 gves the off-polcy Bellman equaton: V x k V x k+1 = x k Qx k + x k F x k v k L x k D P x k+1 + x k K R 1 K x k v k L x k D P A x k + x k L R 2 L x k u k K x k B P x k+1 + β 2 x k x k u k K x k B P A x k 38 By usng the Kronecker product the off-polcy Bellman equaton 38 can be rewrtten as: x k x k vec P x k+1 x k+1 vec P vk + 2 L k vec D P A vk + L x k uk + K x k vec D P B vk + L x k vk + L x k vec D P D uk + 2 K k vec B P A uk + K x k uk + K x k vec B P B uk + K x k vk + L x k vec B P D = x k Qx k + x k F x k + β 2 x k x k + x k K R 1 K x k + x k L R 2 L x k 39 where the unknown varables collected as X X = 1 X 2 X 3 X 4 X 5 X 6 X 7 40 wth X 1 = vec P X 2 = vec D P A X 3 = vec D P B X 4 = vec D P D X 5 = vec B P A X 6 = vec B P B X 7 = vec B P D

5 he data collected onlne n compact form s denoted as: Hk = Hxx k Hvx k Hvu k Hvv k Hux k Huu k Huv k wth xx = x k x k x k+1 x k+1 vk Hvx k = 2 L k vu = v k L x k uk + K x k vv = v k L x k vk + L x k uk Hux k = 2 K k uu = u k K x k uk + K x k uv = u k K x k vk + L x k Furthermore denote the onlne measured utlty functon r k = x k Qx k + x k F x k + β 2 x k x k + x k K R 1 K x k + x k L R 2 L x k 41 he Kronecker product based off-polcy Bellman equaton 39 can be rewrtten n compact form as: H kx = r k 42 Note that n 39 there are N = 3n 2 + m 2 + 3nm unknown components herefore at least N data are requred to collected n order to solve 39 or 42 by least squares methods Assumed that N 1 N data are collected as H 1:N1 X = H 1 H 2 H N 1 X = r 1 r 1 r N1 herefore the least squares soluton of 43 = r 1:N 1 43 ˆX = H 1:N 1 H 1:N1 1H 1:N1 r 1:N1 44 Based on the least squares soluton ˆX n 44 the feedback gan K and L are updated as K +1 = L +1 = 5 Smulaton Study R 1 + X3 + X6 X 1X R 2 5 X2 X6 X 1X 7 + R 2 4 R 2 + X7 X5 R1 + X3 1X 1 6 X4 + X5 R1 + X3 1X 2 In ths secton n order to demonstrate the effectveness of the presented algorthm n the prevous secton a dscretetme rotatng nverted pendulum n 29 s consdered he samplng tme of the lnear dscrete-tme rotatng pendulum model s = 0:005s he system dynamcs s gven as x k+1 = A + x k + Bu k 45 where A = B = = sn k he uncertanty bound satsfyng 5 s gven as F = and ε = 0001 For the correspondng optmal control problem the weght matrx s selected as Q = I 4 4 and R 1 = R 2 = 1 he ntal state s x 0 = he desgn parameters α and β that satsfy 20 s selected as α = 001 and β = 2 o begn the off-polcy renforcement learnng the ntal admssble feedback gans are chosen as K = L = Fnally the soluton of the ARE n 18 s P = and the optmal feedback gan s K = L = When takng the optmal feedback gan n 46 back to the orgnal system wth uncertanty n 45 the system state trajectores s shown n Fgure 1 It can be seen from Fgure 1 that wth the presented optmal control desgn based method the robust control problem of the lnear dynamc system wth bounded uncertanty s solved 6 Concluson hs artcle presents a model-free soluton to the robust control problem of the dscrete-tme lnear systems wth bounded uncertanty Inspred by the dea of 12 for contnuous-tme systems robust control problems the dea that translatng robust control problem to the optmal control problem s adopted n ths paper for dscrete-tme systems he equvalence that the optmal control law s able to stablze the orgnal uncertan system s provded Off-polcy RL method s then appled to the transformed optmal control problem whch has two merts Frst off-polcy RL method can be used to fnd the optmal control feedback gan wthout requrng any knowledge of the system dynamcs Second the data collected from on-lne can be utlzed effcently A smulaton s conducted to valdate the robust stablty of the proposed algorthm

6 References Fg 1: he systems trajectores x k 1 J Cruz J Freudenberg and D Looze A relatonshp between senstvty and stablty of multvarable feedback systems IEEE ransactons on Automatc Control vol 26 no 1 pp F Ln R D Brandt and J Sun Robust control of nonlnear systems: compensatng for uncertanty Internatonal Journal of Control vol 56 no 6 pp B Chen and C Wong Robust lnear controller desgn: tme doman approach IEEE transactons on automatc control vol 32 no 2 pp H Ma Z Wang D Wang D Lu P Yan and Q We Neural-network-based dstrbuted adaptve robust control for a class of nonlnear multagent systems wth tme delays and external noses IEEE ransactons on Systems Man and Cybernetcs: Systems vol 46 no 6 pp D ong Q Zhu W Zhou Y Xu and J Fang Adaptve synchronzaton for stochastc t s fuzzy neural networks wth tme-delay and markovan jumpng parameters Neurocomputng vol 117 pp D ong W Zhou X Zhou J Yang L Zhang and Y Xu Exponental synchronzaton for stochastc neural networks wth mult-delayed and markovan swtchng va adaptve feedback control Communcatons n Nonlnear Scence and Numercal Smulaton vol 29 no 1 pp D ong L Zhang W Zhou J Zhou and Y Xu Asymptotcal synchronzaton for delayed stochastc neural networks wth uncertanty va adaptve control Internatonal Journal of Control Automaton and Systems vol 14 no 3 pp F Ln Robust control desgn: an optmal control approach John Wley & Sons 2007 vol 18 9 D Lu D Wang F-Y Wang H L and X Yang Neuralnetwork-based onlne HJB soluton for optmal robust guaranteed cost control of contnuous-tme uncertan nonlnear systems IEEE transactons on cybernetcs vol 44 no 12 pp X Yang D Lu Q We and D Wang Guaranteed cost neural trackng control for a class of uncertan nonlnear systems usng adaptve dynamc programmng Neurocomputng vol 198 pp D Lu X Yang D Wang and Q We Renforcementlearnng-based robust controller desgn for contnuous-tme uncertan nonlnear systems subject to nput constrants IEEE transactons on cybernetcs vol 45 no 7 pp D Wang D Lu H L and H Ma Neural-network-based robust optmal control desgn for a class of uncertan nonlnear systems va adaptve dynamc programmng Informaton Scences vol 282 pp D Wang C L D Lu and C Mu Data-based robust optmal control of contnuous-tme affne nonlnear systems wth matched uncertantes Informaton Scences vol 366 pp P J Werbos Approxmate dynamc programmng for realtme control and neural modelng Handbook of ntellgent control: Neural fuzzy and adaptve approaches vol 15 pp D Klenman On an teratve technque for rccat equaton computatons IEEE ransactons on Automatc Control vol 13 no 1 pp G N Sards and C-S G Lee An approxmaton theory of optmal control for tranable manpulators IEEE ransactons on systems Man and Cybernetcs vol 9 no 3 pp M Abu-Khalaf and F L Lews Nearly optmal control laws for nonlnear systems wth saturatng actuators usng a neural network hjb approach Automatca vol 41 no 5 pp D Vrabe and F Lews Neural network approach to contnuous-tme drect adaptve optmal control for partally unknown nonlnear systems Neural Networks vol 22 no 3 pp D Vrabe O Pastravanu M Abu-Khalaf and F Lews Adaptve optmal control for contnuous-tme lnear systems based on polcy teraton Automatca vol 45 no 2 pp F Slva and R F Stengel Model-based adaptve crtc desgns Handbook of learnng and approxmate dynamc programmng vol 2 p D Wang D Lu Q We D Zhao and N Jn Optmal control of unknown nonaffne nonlnear dscrete-tme systems based on adaptve dynamc programmng Automatca vol 48 no 8 pp Y Jang and Z Jang Computatonal adaptve optmal control for contnuous-tme lnear systems wth completely unknown dynamcs Automatca vol 48 no 10 pp B Luo H Wu and Huang Off-polcy renforcement learnng for control desgn IEEE transactons on cybernetcs vol 45 no 1 pp B Kumars F L Lews and Z-P Jang H control of lnear dscrete-tme systems: Off-polcy renforcement learnng Automatca vol 78 pp H Modares F L Lews and Z-P Jang rackng control of completely unknown contnuous-tme systems va off-polcy renforcement learnng IEEE transactons on neural networks and learnng systems vol 26 no 10 pp H Modares S P Nageshrao G A D Lopes R Babuška and F L Lews Optmal model-free output synchronzaton of heterogeneous systems usng off-polcy renforcement learnng Automatca vol 71 pp F L Lews and V L Syrmos Optmal control John Wley & Sons R A Horn and C R Johnson Matrx analyss Cambrdge unversty press N S rpathy I N Kar and K Paul Stablzaton of uncertan dscrete-tme lnear system wth lmted communcaton IEEE ransactons on Automatc Control 2017 n Press