Optimal Control Scheme for Nonlinear Systems with Saturating Actuator Using ε-iterative Adaptive Dynamic Programming

Transcription

1 UKACC Internatonal Conference on Control Cardff, UK, 3-5 Septeber Optal Control Schee for Nonlnear Systes wth Saturatng Actuator Usng -Iteratve Adaptve Dynac Prograng Xaofeng Ln, Yuanjun Huang and Nuyun Cao School of Electrcal Engneerng, Guangx Unversty Nannng, Chna Yuzhang Ln Departent of Electrcal Engneerng, snghua Unversty Bejng, Chna Abstract In ths paper, a fnte-horzon optal control schee for a class of nonlnear systes wth saturatng actuator s proposed by an proved teratve adaptve dynac prograng (ADP) algorth he Halton-Jacob-Bellan (HJB) equaton correspondng to constraned control s forulated usng a sutable nonquadratc functon hen atheatcal analyss of the convergence s presented, by provng that the perforance ndex functon can reach the optu usng the adaptve teraton Fnally the fnte-horzon optal control law can be obtaned by the -teratve adaptve algorth he exaples are gven to deonstrate the effectveness of the above ethods Keywords-Adaptve dynac prograng(adp); Saturatng actuators; nonlnear syste; Fnte te optal control I INRODUCION In a practcal control syste, the saturatng actuators wll reduce the syste's dynac perforance, and even affect the stablty of the syste herefore seeng a better way to desgn control systes wth saturatng actuator has attached consderable attenton by any researchers n recent years he stablty of dscrete-te lnear systes subject to actuator saturaton was analyzed usng a saturaton-dependent Lyapunov functon based on the soluton of an LMI optzaton proble[] Saber (996) and Sussann(994) proposed several processes to control saturaton probles, but they dd not consder non-lnear systes and optal probles (see [],[3]) A gan-scheduled output control desgn for systes copng wth nonlnear te-varyng paraeter dependent systes subject to saturated actuators was proposed n [4] Pontryagn's Mnu Prncple s a way to solve optal control proble wth Saturatng Actuator However, ths needs to solve dfferental equatons wth boundary, and the result we get s an open-loop control by ths way Lyshevs desgned an optal control for a closed-loop feedbac syste usng a non-quadratc perforance ndex functon proble to deal wth the control constrants based on dynac prograng prncple n [5], but the dffculty les n the HJB equatons Adaptve Dynac Prograng (ADP) s a powerful tool proposed by the dea of adaptve crtc and renforceent learnng wth dynac prograng[6] It s solved by teratve algorth to get an approxate soluton of HJB equaton, ADP has becoe an effectve tool for optal control probles and has acheved any results[7],[8-] A greedy teratve adaptve algorth was proposed to solve the nonlnear dscrete-te systes HJB equaton n [8] Iteratve ADP was used to get an nfnte horzon optal control schee for nonlnear systes wth saturatng actuator n [9] However, for practcal systes, a lted perod of te s requred to acheve control Fnte te optal control proble could be bacward solved by dynac prograng, when facng the ult-densonal nonlnear characterstcs of coplex systes, the calculaton wll be very large, that s the proble of the curse of densonalty -error bounds of adaptve algorth was proposed to deal wth fnte te optal control n [],[] o the best of our nowledge, qute few research has been presented to deal wth fnte-horzon optal control wth saturatng actuator hs otvates our research hs paper as to solve fnte te optal control proble for nonlnear systes wth saturatng actuator It s organzed as follows In Secton II, the proble s ntroduced and HJB equaton correspondng to constraned control s presented In Secton III, the teratve ADP algorth and ts convergence for fnte-horzon optal control proble are derved In Secton IV, -optal control algorth s developed wth the defnton of fnte teraton steps and -optal control In Secton V, an exaple s gven to deonstrate the effectveness of the algorth In Secton VI, the concluson s drawn II HJB EQUAION CORRESPONDING O CONSRAINED CONROL A Deal wth saturated proble Consder the followng class of dscrete-te nonlnear systes ( ) x( + ) = F x( ), u ( ) =,,, () n Where x( ) s the state, assue the syste F( x(, ) u ( ) ) s Lpschtz contnuous controllable on a set Ω hs wor s supported by Natonal Natural Scence Foundaton of Chna (6964, 634) //$3 IEEE 58

2 contanng the orgn he control u ( ) Ω u, and ( ) [ u( ),u( ),,u( )] : u( ) u Ωu = u = R, where u s the -th executon controller saturaton boundary, =,, On the other hand, the constant dagonal atrx s A = dag [ u, u,, u ],A F (, ) =, hence x = s an equlbru state of syste () under the control u = For the ntal state x (), to nze the perforance ndex functon defned n () wth fnte control sequences as N u = u, u,, u N - ( () () ( )) ( N- N (), ) = ( ) Q ( ) + ( ( )) J x u x x W u () = u( ) Where - W( u( )) = ( Atanh ( s A)) R ds he length N s deterned wth ternal te, ths nd of optal control probles has been called fnte-horzon probles wth unspecfed ternal te [] If we set W( u( )) = u( ) Ru( ) n (), the quadratc perforance ndex functon wth unconstraned control s used to desgn the optal control for control syste wth saturated actuator, however, ths syste could not ensure the optal syste perforance, even ay lead to syste nstablty Accordng to [], let Where u( ) W( u( )) = Aφ ( ) Rds A - s φ ( u ( )) = [ φ ( u( )), φ ( u( )), φ ( u ( ))] - u A,s, φ,let R be a dagonal postve defnte, φ() s a bounded onotoncally ncreasng odd functon p belongs to ( p ) and L ( Ω) wth φ(), the frst dervatve s a bounded constant M, such functon as φ () = tanh() Fgure shows a well approachng saturaton when u 5 wth functon φ() herefore, we can guarantee the control output sgnal wthn the range of actuator saturaton usng the perforance ndex functon n () Saturated output of the actuator 5-5 φ () Molde saturaton Coed Sgnal sat() ure saturaton Fgure he odel of saturaton when u 5 B HJB equaton and soluton Defnton : he correspondng fnte-horzon adssble (3) + N control sequence of perforance ndex functon n () u s n defned as follows: For x ( ), there exsts a control + N f sequence u satsfy x ( x( ), u + N- N- ) = and Jx ( ( ), u ) s f + N- fnte Where N > s a postve nteger, x ( x( ), u ) s the ternal state Let N + N f +N- +N- x = u x ( x( ), u ), u = = N C : be all the adssble control sets wth length N Assue a state x( ) s a fnte-horzon adssble control By Defnton, the optal perforance ndex functon at the fnte-horzon adssble control could be wrtten as N- N- N J ( x( )) = nf J( x( ), u ): u C (4) x N- u Accordng to the perforance ndex functon defned by equaton () and Bellan prncple of optalty, J ( x ( )) under dscrete te HJB equaton can be wrtten as J ( x( )) = n x( ) Qx( )+ W( u( ))+ J ( F( x( ), u ( ))) (5) u( ) Defne the optal control sequence startng at wth length of N by N N N x + + ( ( ))= ( ( ), ): C u + N u x nf J x u u (6) and defne the one step optal control vector by u( ) u ( x( )) = arg n x( ) Qx( )+ W( u( ))+ J ( F( x( ), u ( ))) (7) Dynac prograng s used to solvng optal control sequence n equaton (6), whle the frst step s to deterne u ( x ( + N )) by equaton(6) n ternal state x( + N ) u x argn x x W u u( N-) ( ( + N )) = ( + N ) Q ( + N )+ ( ( + N )) st F ( x( + N ), u ( + N ))= (8) Puttng u ( x ( + N )) nto (5), the optal perforance ndex functon J ( x( + N )) = x( + N ) Qx( + N )+ W( u ( + N )) (9) After u ( x( + N )) and J ( x( + N )) s obtaned, u ( x( + N )) and J ((+ x N )),, could be deterned by equaton (5) and (7), fnally u ( x( )) and J ( x ( )) s solved hen solvng u ( ) = u ( x( )) wth x( ), and puttng u ( ) nto equaton () so as to get x( + ) = F( x( ), u ( )) By the sae way, solvng u ( + ) by x( + ) and u ( x( + )), puttng u ( x( + )) nto equaton (), then we can easly get x( + ) = F( x( + ), u ( + )), repeatng ths process [9], the optal control sequence wll be obtaned as u ( ) = u ( ), u ( + ),, u ( + N ) However, the perforance ndex functon n ths paper s non- quadratc, and the syste s non-lnear herefore, t s 59

3 hard to get an analytcal soluton of optal control law by solvng HJB equaton On the other hand, when dealng wth dscrete te dynac prograng by bacward ethod, one has to calculate and save all J ( x( )) and u ( x( )) of the sequence Hence, t wll eet dffcultes when calculatng fnte-horzon optal control probles by dynac prograng III FINIE IME IERAIVE ADP ALGORIHM A Forula derved for teratve ADP For any state x( ), the perforance ndex functon V and control polcy υ n the teratve ADP algorth are updated by recursve teratons he teratve starts wth = and the ntal perforance ndex functonv ( x ( ))=, the perforance ndex functon for = s coputed as u( ) u( ) V ( x( ) ) = n x x W u V F x u ( ) Q ( )+ ( ( ))+ ( ( ( ), ( ))) st F( x( ), u( )) = = n x( ) Qx( )+ W( u( )) st F( x( ), u( ) ) = = x( ) Qx( )+ W( u ( x ( ))) () Where F( x( ), u ( ) ) be wrtten as V =, let u ( ) = ( x( )), then () can ( x( )) = n x( ) Qx( )+ W( u( ))+ V ( F( x( ), u( ))) u( ) where st F( x( ), u( )) = = x x W υ x () ( ) Q ( )+ ( ( ( ))) υ = argn ( x( )) x( ) Qx( )+ W( u( )) u( ) υ st F( x( ), u ( )) = () For = 34,,, the teratve ADP algorth s updated as follows V ( x ( )) = n x( ) Qx( ) + W( u( )) + V ( F( x( ), u( ))) where u( ) = x x + W υ x + V F x υ x (3) ( ) Q ( ) ( ( ( ))) ( ( ( ), ( ( )))) υ (4) ( x( )) = arg n x( ) Qx( )+ W( u( )) + V ( F( x( ), u( ))) u( ) After steps teraton, the perforance ndex functon sequence s obtaned as V = ( V,V,,V) and control polcy sequence as υ = ( υ, υ, υ ) Note that n ths case, each control sequence u() wll obey wth dfferent control polcy, that s, for =,, N, control sequence u() s obtaned by υ respectvely However, one could prove that V ( x ( )) s lt to J ( x( )) when herefore, the perforance ndex functon J ( x ( )) n HJB equaton wll be replaced by the teratve perforance ndex functon V ( x( )) whle control polcy ux ( ) wll be replaced by the teratve control polcy B Convergence of teraton ADP Rear : Accordng to () and (3), we have + + ( + ) V+ ( x( )) = n J( x( ), u ): u C (5) + u Prove: Accordng to (3), we have hus, V x x x W u x ( ( +)) = n ( + ) Q ( + ) + ( ( + ))) u( + ) st F(( x + ),( u + )) = (6) V x x x W u V x + ( ( ))= n ( ) Q ( )+ ( ( ))+ ( (+) u( ) = n x( ) Q x( )+ W( u( )) u( ) ( ( +-)) + V ( x( +)) hen (7) could be wrtten as V + x = x Qx + W u (()) n() () (()) + u + n x( + ) Qx( + )+ W( u( + )) u( + ) + n x( +) Qx( +)+ W( u( +))+ u( + ) + n x( +-) Qx( +-)+ Wu u( + ) + x( + ) Qx( + ) + W( u( + )) + + x + x + + W u + ( ) Q ( ) ( ( )) st F( x( + ), u( + ))= + + ( + ) ( ( ), ): + C x u (7) = n Jx u u (8) hus (5) holds hree theores as well as a corollary are gven n the followng For the proof n detal, see [],[] heore : Gvng an arbtrary state vector x ( ), assue () there exst an ntegral such that C x, then ( + C ) x, the perforance ndex functon V ( x( )) s a onotoncally nonncreasng sequence e, V+ ( x( )) V( x( )) heore : Gvng an arbtrary state vector x (),Defne the perforance ndex functon V ( x( )) as the lt of the 6

4 teratve functon V ( x ( )), e, hus, V ( x( )) = l V( x( )) (9) V x = x x W u + V x + () ( ( )) n ( ) Q ( ) + ( ( )) ( ( )) u( ) heore 3: Defne the perforance ndex functon V ( x( )) as (3), assue the state x( ) s controllable, then the perforance ndex functon V ( x( )) equals the optal perforance ndex functon J ( x ( )), e l V x J x () ( ( )) = ( ( )) Corollary : Assue the syste state x( ) s adssble controllable and the perforance ndex functon s defned n (3), and heore 3 holds, then the teratve control law υ ( x( )) converges to the optal control law u ( x ( )) IV -OPIMAL CONROL ALGORIHM A -Optal Control In order to get the lt of the perforance ndex functon V ( x ( )), teratve ADP algorth () s need to perfor, optal control law u ( x( )) won t obtan untl However, s usually nubered n practcal, J ( x ( )) = V ( ( )) x could not hold for any fnte herefore, error bound s ntroduced for V ( x( )) and J ( x( )) n teratve ADP algorth, so that the perforance ndex functon V ( x( )) approxates to the optal perforance ndex functon J ( x( )) n fnte nuber of steps Defnton : let Γ be a controllable state set, x ( ) Γ, >, then the nuber of teraton steps K ( x( )) for optal control s defned as K( x ( )) = n :V( x( )) J ( x( )) () K ( x( )) refers to the length of control sequence reachng equlbru pont fro x( ), snce x( ) Γ,thus l V ( x( )) = J ( x( )) herefore, there exsts a fnte such that V ( x( )) J ( x( )) (3) holds hus : V ( x( )) J ( x( )),so that K ( x( )) s defned Defnton 3: let x() Γ be a controllable state vector, for any >, f V ( x( )) J ( x( )) holds for the teratve control law υ ( x( )), then the υ ( x( )) s defned as a -Optal Control μ ( x( )), e () Q ()+ (()) x x W u μ (()) x = υ (()) x = argn (4) u() + V ( Fx ( ( ), ux ( ( )))) B Suary of the -Optal Control Algorth Step A Gvng an ntal state x( ) and an error bound Step A Set =, V ( x( )) = and K ( x( )) = Step A3 Calculate υ ( x( )) = u ( ) by () Step A4 For =, calculatev ( x( )) by () Step A5 Set = + and K ( x( )) = Step A6 For = 3,,, calculate υ ( x( )) by (4), calculate V ( x( )) by (3) Step A7 If V ( x ( )) V ( x + ( )), go to step A8; then K ( x( )) = s the nuber of optal control steps, -optal control law s μ ( x( )) = υ ( x( )) otherwse, go to step A5 Step A8 Stop V SIMULAION SUDY A -Iteratve ADP Algorth for Saturatng Actuator Consder the followng nonlnear syste x( + ) = f( x( )) + g( x( ) ) u( ) (5) where 5 x ( ) f( x( )) = sn( 8 x( ) x( )) + 8 x( ) x ( ) g( x ( ) ) =, u 5 A = =, Q=R=I 8x( ) u, he perforance ndex functon s + N ( ( ), ( )) ( ) Q ( )+ ( ( )) J x u = x x W u (6) = Neural networs are used to pleent the teratve ADP algorth n [7],[9],[] wth ts good functon approxatng characterstcs he structure dagra of the teratve ADP algorth usng Neural-Networ approxate functon s shown n fgure In the dagra, crtc neural networ s used to approxate functon V ( x ), acton neural networ s used to control law υ ( x( )), gradent descent algorth s used to adjust the weght by neural networ tranng rule, the approxate proof and forula dervaton can be checed n [7],[9],[] he crtc networ and the acton networ are chosen as three-layer bac-propagaton (BP) neural networs wth the structures of and Accordng to 3 -optal control algorth, let =, the ntal state s Τ chosen as x ( ) = [, ], learnng rate α = 5 For each teratve step, the crtc networ and the acton networ are traned for teraton steps so as to guarantee the neural 6 networ tranng error s less than 6

5 x( ) Sgnal lne Bac-propagatng Path Weght ranssson û ( ) + u ( ) e V ( x( )) = u () x( ) x( + ) V ˆ ( x ( )) + + V ˆ ( x( + )) Utlty x x W u + + ( ) Q ( ) + ( ( )) Fgure he structure dagra of the teratve ADP algorth usng V ( ) Neural-Networ approxate x and υ ( x) perforance ndex functon Iteraton Steps Fgure 3 he convergence process of the cost functon error of the V (x) Iteraton Steps Fgure 4 he error of neural-networ approxate cost functon Sulaton result n Fgure 3 shows the teraton convergence process of the cost functon V ( x( )) When = 3, V ( x( )) J ( x( )) holds, thus -optal control law μ s obtaned n fnte step K (() x ) = = 3 Besdes, Fgure 3 also shows that V ( x( )) satsfy the onotoncally nonncreasng and constrngency property n heore snce V ( x ( )) V ( x + ( )) Fgure 4 shows the error of neural-networ approxate cost functon In order to verfy the control law μ obtaned by -teraton algorth, the state trajectores and the optal control trajectores are shows n Τ Fgure 5 and Fgure 6 for the state x ( ) = [, ], Fgure 7 shows the state trajectores n 3d space perforance Notce that the state has been control n stable wthn fnte step f = 3 showed n Fgure 6, whch satsfes the theory, on the other hand, the control output u u = 5, u u = n Fgure 6 shows that control output sgnal actuator eeps under the constrants u A e States rajectory 5-5 Control utput - 5 X X e Steps Fgure 5 he state trajectores u u e Steps Fgure 6 he optal control trajectores x - 5 x e Steps Fgure 7 he state trajectores n 3d space perforance B Coparson of sulaton In order to contrast wth the controller wth saturatng actuator, the state trajectores wthout actuators saturaton s showed n Fgure 8 and the optal control trajectores wthout actuators saturaton s showed n Fgure 9 In Fgure 9, control output satsfy u 5, u, thus the control sgnal wll be dstorted wth saturatng actuator, whle t eeps under the constrants n Fgure 6 Coparng results wth Fgure 6 and Fgure 9, result shows that -Iteratve adaptve dynac prograng wors effectve for fnte-horzon optal control schee for nonlnear systes wth saturatng actuator States rajectores e steps Fgure 8 he state trajectores wthout actuators saturaton 3 x x 6

6 Control output 5 5 u u e steps Fgure 9 he optal control trajectores wthout actuators saturaton VI CONCLUSION In ths paper, a functonal perforance ndex functon was used to deal wth saturatng actuator control wth constrants effectvely Furtherore, dscrete HJB equaton of nonlnear systes was derved Fnte horzon teraton ADP algorth for control wth saturaton was developed va atheatcal analyss Fnally the fnte-horzon optal control was obtaned by the -teratve adaptve algorth REFERENCES [] Yong-Yan Cao, Zongl Ln, Stablty analyss of dscrete-te systes wth actuator saturaton by a saturaton-dependent Lyapunov functon, Autoatca,Vol39, 3, pp 35 4 [] Saber, A Z,Ln,A eel, Control of Lnear Systes wth Saturatng Actuators, IEEE ransactons on Autoatc Control, Vol 4, NO3, March 996, pp [3] Sussann H,E D Sontag,Y Yang, A General Result on the Stablzaton of Lnear Systes Usng Bounded Controls, IEEE rans Autoatc Control, Vol39, No, Deceber 994, pp 4 45 [4] Marc Jungers a, Eugêno B Castelan, Gan-scheduled output control desgn for a class of dscrete-te nonlnear systes wth saturatng actuators, Systes & Control Letters, Vol6,, [5] Lyshevs, SE, Optal Control of Nonlnear Contnuou e Systes: Desgn of Bounded Controllers VaGen-ralzed Nonquadratc Functonals, Aercan Control Conference, June 998, pp5 9 [6] Werbos P J, Approxate dynac prograng for reall controlll and neural odelnghandboo of Intellgent Control, Neural Fuzzy and Adaptve ApproachesNew Yor:Van No strand Renhold, 99 [7] We Qng-la, Zhang Huang-guang, Lu De-rong, Zhao Yan, An Optal Control Schee for a Class of Dscrete-te Nonlnear Systes wth e Delays Usng Adaptve Dynac Prograng, Acta Autoatca Snca, Vol 36, NO,, pp 9 [8] Al-a, F L Lews, and M Abu-Khalaf, Model Free Q-learnng desgns for lnear dscrete-te zero-su gaes wth applcaton to H-nfnty control, Autoatca, Vol 43, NO3, Mar 7, pp [9] Luo Yan-Hong, Research on Adaptve and Optal Control for Nonlnear Systes Based on Neural Networs, Northerestern Unversty, 8, pp33 59 [] Wang Fe-Yue, Jn Nng, Lu Derong, We Qng-La, Adaptve Dynac Prograng for Fnte-Horzon Optal Control of Dscrete-e Nonlnear Systes Wth -Error Bound, ransactons on Neural Networs, Vol,, pp 4 36 [] Wang Fe-Yue, Jn Nng, Lu Derong, Hou Zeng-Guang, adaptve Dynac Prograng wth epslon-error Bound for Nonlnear Dscrete-e Systes Usng Neural Networs, ransactons on Neural Networs, 8 [] Abu-Khalaf M, Lews F L, Nearly optal control laws for nonlnear systes wth saturatng actuators usng a neural networ HJB approach, Autoatca 5, Vol 4, NO5, pp