Optimal Control of Nonlinear Systems using RBF Neural Network and Adaptive Extended Kalman Filter

Transcription

1 9 American Control Conerence Hyatt Regency Riverront, St. Louis, MO, USA June -, 9 WeA. Optimal Control o Nonlinear Systems using RBF Neural Network and Adaptive Extended Kalman Filter Peda V. Medagam, Student Member, IEEE, and Farad Pourboghrat, Senior Member, IEEE Abstract his paper presents a nonlinear optimal control technique based on approximating the solution to the Hamilton-Jacobi-Bellman (HJB equation. he HJB solution (value unction is approximated as the output o a radial basis unction neural network (RBFNN with unknown parameters (weights, centers, and widths whose inputs are the system s states. he problem o solving the HJB equation is thereore converted to estimating the parameters o the RBFNN. he RBFNN s parameters estimation is then recognied as an associated state estimation problem. An adaptive extended Kalman ilter (AEKF algorithm is developed or estimating the associated states (parameters o the RBFNN. Numerical examples illustrate the merits o the proposed approach. I. INRODUCION Optimal control is an important aspect o control theory because o guaranteed closed loop perormance or a given system. Optimal control o linear time invariant (LI systems has been well studied and practiced with much success []. he optimal control design or LI systems generally involves the solution o algebraic Riccati equation (ARE. One special extension to nonlinear systems involves state dependent Riccati equation (SDRE technique, which provides high perormance control. his method consists o changing the nonlinear system into a state dependent pseudo-linear orm and involves the solution o an algebraic SDRE along the system s trajectory to obtain a nonlinear eedback controller []. For most nonlinear systems, however, the optimal control design requires the solution o Hamilton-Jacobi-Bellman (HJB equation. he HJB equation in general cannot be solved analytically. here have been a number o approaches to solve the HJB equation. hese approaches, generally, involve approximation techniques. One method o solution is based on power series approximation o HJB equation. he basic idea o this method is to approximate the value unction as a truncated power series and to ind the corresponding terms o the series by itting it in the HJB equation [], [4]. here are also other approximation techniques. Saridis et al. [5] developed a recursive approximation technique which starts with a stabiliing controller or a given plant and P. V. Medagam is with the Department o Electrical and Computer Engineering, Southern Illinois University Carbondale, Carbondale, IL USA ( pmedagam@phasetechnologies.com. F. Pourboghrat is with the Department o Electrical and Computer Engineering, Southern Illinois University Carbondale, Carbondale, IL USA ( pour@siu.edu. converges point-wise to the optimal control. Based on this technique Beard [6] proposed a successive Galerkin approximation or generalied Hamilton-Jacobi-Bellman (GHJB equation and showed the convergence o the successive approximation or optimal control solution. he diiculty with this and other similar methods is the selection o basis unctions, which are important or the convergence o the solution to optimal control. his diiculty may be resolved by employing wavelets [7] or neural networks [8] as basis unctions. Neural networks have been widely used in the identiication, estimation and control o nonlinear systems [9], []. Oline estimation o the value unction using neural networks has been studied in [] where the neural network was trained using least square technique. In addition, nonlinear H control using radial basis unction (RBF neural networks has been reported in []. his method is based on the estimation o the value unction using nonlinear RBF neural networks (RBFNN where the network is trained, oline, using gradient method. In the present paper, we solve the HJB equation or the optimal value unction. he value unction is needed to solve or an optimal control design. RBFNNs are used to estimate the value unction as the solution to the generalied HJB (GHJB equation. he perormance o the RBFNNs depends on their weights, centers and widths. he common training approach is to irst select the RBF centers using unsupervised K-means algorithm [], or randomly rom input data, and then determine the weights using least square or gradient descent method []. However, here, the weights as well as the centers and widths o the RBFNN are unknown and appear nonlinearly. An adaptive extended Kalman ilter (AEKF method is thereore developed to train the neural network weights, centers, and widths, online, with good accuracy. Fast digital signal processors (DSP can be used or real-time implementation. II. SAEMEN OF HE PROBLEM Consider a nonlinear time invariant system described by x = ( x + g( x u ( n m n n where x Ω R, u R, : R R, and n n m g : R R R. Without loss o generality, it is assumed that x = is the equilibrium state and that ( x =. Moreover the system is Lipschit continuous on a set Ω /9/$5. 9 AACC 55

2 n inr. Also consider the cost unction o the orm V( x = [ x Qx+ u Ru] dt ( n n m m where Q R and R R are positive deinite matrices. he optimal eedback control problem is to ind the admissible control u so that the perormance index ( is minimied. When an optimal control exists, it is given by * u = R g( x V ( x ( where = is the gradient operator, and that V ( x x is the value unction and it satisies the ollowing Hamilton- Jacobi-Bellman (HJB equation with boundary conditions V ( = ; i.e., * 4 v( V = V + x Qx V gr g V = (4 he above HJB equation is a two point boundary-value problem in terms o the value unction V ( x, which in general is impossible to solve, analytically. he above equation can be also written in the orm o generalied Hamilton-Jacobi-Bellman (GHJB equation [], i.e., * * * * v( V, u = V ( + gu + x Qx+ u Ru = (5 * which is a unction o both V and u *. Unlike the HJB equation, the above GHJB equation is linear in terms o V and, together with equation (, can be solved using approximation techniques. III. NEURAL NEWORK APPROXIMAION OF HE VALUE FUNCION Similar to the HJB equation, in general, the GHJB equation cannot be solved analytically. Hence, we approximate the value unction with radial basis unction neural networks. he RBF neural network has a eedorward structure with one nonlinear hidden layer with N nodes and a linear output layer. Fig. shows the schematic diagram o the RBF neural network. Fig. - Schematic diagram o the RBF neural network he outputs o the hidden nodes are speciied as = x c ; σ (6 ( i i i where i=,..,n, x is the input vector to the neural network, and that c i s and σ i s are, respectively, the adjustable centers and widths o the RBF unctions. he activation unctions, s, are chosen as i ( / ( x = x c + σ (7 i i i he output o the RBF neural network (RBFNN is given by yx ( = wxcσ (,, (8 where w = [ w,, w N ], c = [ c,, c N ] and σ = [ σ,, σ N ] are the vectors o adjustable weights, centers, and widths o the RBFNN, and that ( x = [,, N ] is the vector o basis unctions. Realiing that the value unction must be positive deinite, we approximate the unknown value unction using the output o the RBF neural network as V( x = ( x x P( x x (9 + ( y y ( y y where y = w ( x, c, σ, x is the equilibrium point and P is the positive deinite solution o the Riccati equation corresponding to the linearied approximation o the nonlinear system (. It should be noted that y y = w ϕ( x, c, σ ( where ϕ( xc,, σ = xc (,, σ x (, c, σ. More speciically, noting that x = and using (, the value unction (9 can be expressed as V( x = x Px ( + ϕ ( xc,, σ wwϕ( xc,, σ Selecting V( x as above, it can be veriied that V( x > n or all x R, x, and that V ( =. Clearly, i the RBFNN parameter vectors w, c, and σ can be ound so that the proposed approximation o the value unction in ( satisies the GHJB equation (5, then V( x will satisy the cost unction ( and its time derivative will be given by V = x Qx u Ru ( which is negative deinite. But, since V( x is positive deinite and radially unbounded, this indicates the stability o the closed-loop system. A proper parameter estimation technique, however, is required to ind the unknown parameters o the RBFNN or correct approximation o the value unction V( x. he gradient o the value unction ( can be written as V( x V = = x P+ ϕ ( x, c, σ( ww ϕ( x, c, σ ( x ϕ ( xc,, σ where ϕ( xc,, σ =. Substituting or V, rom x the above, in the GHJB equation (5, we get v( V, u = ( x P+ ϕ ( x, c, σ( ww ϕ( x, c, σ (4 ( ( x + g( x u( x + x Qx+ u ( x Ru( x he above equation can be equivalently written as 56

3 = ht (, θ + v (5 where = x Qx u( x Ru( x x P( ( x + g( x u( x is a known measurable unction o x and u, ht (, θ = ϕ ( xc,, σ( ww ϕ( xc,, σ ( ( x + g( xux ( is a known unction o the unknown RBFNN parameter vector θ = [ w, c, σ ], and v= v( V, u is the equation error due to approximation o the value unction. Clearly, equation (5 is nonlinear in terms o the unknown parameter vector θ. Hence, in order to estimate the unknown RBFNN parameter vector θ, one must use nonlinear parameter estimation techniques. his will be explained in more detail in the ollowing section. IV. NEURAL NEWORK RAINING Estimating the RBFNN parameters rom equation (5 can be viewed as state estimation problem or an associated parameter system where the RBFNN parameters (weights, centers and widths are the unknown states to be estimated. It is known that Kalman ilters (KF can be successully used in the state estimation problem or linear systems. However, because o the nonlinearities in the RBFNN ormulation, instead o the KF method an extended Kalman ilter (EKF must be used or state (parameter estimation. Realiing that the unknown state vector (parameter vector θ is constant, and using equation (5, the associated parameter dynamics is given as [5] θ = ω (6 = ht (, θ + v where θ and are, respectively, the unknown states and measurable outputs o this system, and ω and ν are white noise disturbances, with covariance matrices Q and R, aecting the states and outputs. he EKF algorithm is to estimate the unknown states θ o the system and is given as ˆ θ = K ˆ ( h (, t θ (7 where the ilter gain matrix K is given as K = SH R (8 H ( = h θ θ θ = θ where is the observation matrix or the linearied model o the system (6 and S > is a symmetric positive deinite matrix, which satisies the ollowing ilter dierential Riccati equation [], i.e., S = α S + Q SHR H S (9 where St ( = S = S >, Q, R >, and α >. he convergence properties o the EKF are explained in [6]. However, the region o convergence or the EKF algorithm in this case may be small. Here, to improve the region o convergence, an adaptive extended Kalman ilter (AEKF is proposed as ˆ θ = K ( where = ˆ, ˆ = ht (, ˆ θ, and the ilter gain matrix K is given by ˆ K = SC R ( Also, the corresponding adaptive output matrix Ĉ is adjusted as C ˆ = λ + ( ˆ γsign θ ( t = + γ γ γ ( ˆ ( where λ >, ˆ( ˆ h C = H( = ˆ θ, γ, γ >, α >, sign( sat and ε <<. Moreover, the symmetric ( ε positive deinite matrix, S >, satisies a ilter dierential Riccati equation (FDRE [6], given as ˆ S = α S SC R ˆ CS+ Q (4 where S( = S ( = S >. he convergence o the RBFNN parameters (weights, centers, and widths is shown in the ollowing lemma. Lemma: Consider the dynamics o the RBFNN parameters (6. I these unknown parameters are estimated according to the proposed AEKF algorithm (-(4, then the output and the parameter errors,, θ, converge to ero, asymptotically. Proo: Let θ = θ ˆ θ. hen θ = K + ω (5 Using power series expansion, let us express h= h(, t θ as h= h + C(, t θ θ (6 where h = h( t,, and Ctθ (, is a nonlinear vector. he estimate o htθ (, can then be written as hˆ = h ˆ ˆ ˆ + C(, t θ θ (7 where h ˆ = h( t, ˆ θ. Now deine h = h hˆ = Cθ Cˆˆ θ. hen ˆ ˆˆ h = h h = C θ + Cθ Cθ Cˆˆ θ (8 ˆ ˆ = + ˆ Cω Cθ CSC R ˆ Cθ hen, noting that = h + v and using ˆC orm (-(, one can write ˆ ˆ = μ CSC R ( + ( ˆ λ γsign θ (9 where μ = + + Cω Cθ v is bounded. Now, using the Lyapunov unction = and its derivative V η, V with ˆ η η = λ θ +, η > and γ ˆ θ min min μ η, one can show that converges to ero. However, since Cθ + v is bounded and = h + v = Cθ Cˆˆ θ + v goes to ero, ˆˆ Cθ must be bounded and must converge to Cθ + v. 57

4 Moreover, since ˆ ˆ ˆ V = C = CC is lower bounded and its Cˆ derivative ˆˆ V ˆˆ ˆ = CC = Cθ λ+ γsign C ( converges to ero, then V Ĉ and hence Ĉ must be bounded. But, since ˆˆ Cθ is bounded, then ˆ θ must also be bounded, Now consider a Lyapunov unction candidate V = θ Π θ + ( where Π = Π > such that S. S =Π and Π= Π Π ake the time-derivative o V( x, apply the Young s inequality to the terms θ Πω and ˆ θ C R, add and subtract the term ˆ Π + αθ θ θ C R Cˆ θ, note that Cˆ S ˆ ˆ C R C R θ θ θ Π θ, and substitute or rom (9. hen, we get = Π + V θ θ θ Π θ + = Π Π + Π ˆ θ θ θ ω θ + θ Π Π θ + αθ Π θ + ω S C R Π S α ˆ ˆ CSC α θ θ 4α R + Π + + Π ˆ ˆ θ α θ ω α Cˆ S + α+ α + α R θ Π θ (4 ˆ ˆ α R + CSC 4 ( + ( ˆ μ λ γsign θ α R Π S+ S SC R CS Π + ( Now substitute or S rom (4 in the above, and note that since Q and R >, the solution S to the FDRE (4 is bounded both rom above and below. Let α >, α, α α + α +. hen, we get 4R ˆ C S R Π ˆ V θ Π Q + ˆ Π + SC R CS θ ω α ( + ( + ( ˆ μ λ γsign θ However, as long as, γ will increase and, since ˆ θ, ater a inite time we have some η >. Now, let Also let = min { ΠQ Π, } γ ˆ θ μ η or min ˆ η min η = λ θ + with η >. λ η and choose α > such Π min that α δ ω or some δ >. hen, the derivative o the Lyapunov unction will become V λ θ + + δ ( ( his proves the boundedness o the Lyapunov unction V as well as θ and. Moreover, let us deine the residual set = {(, ( + } δ λ r θ θ ε where ε = >. hen, it is easy to show that the total estimation error θ + must converge to the small residual set r. Furthermore, in case ω =, we will have ε = and r = {}, and hence V and both θ and must converge to ero, asymptotically. Given Q and R >, and assuming that the associated parameter system is observable, the FDRE (4 can be solved or the covariance matrix S > continuously and the time varying gain K can be determined rom equation (. he RBFNN parameters are then estimated using equation ( with the knowledge o K and measurement o system states x. Assuming the nonlinear system is controllable, with the estimated RBFNN parameters, the nonlinear system can be controlled using the optimal control ( via the solution o the GHJB equation (5. It can be shown that the closed-loop system, consisting o the AEKF parameter estimator and the GHJB-based optimal control working together, simultaneously, is globally stable and that both the estimation and the control errors will converge to a small residual set. V. SIMULAION RESULS In this section, SISO nonlinear systems are considered or regulation problem. he proposed technique is applied or the optimal control design. wo examples are presented. Example : Consider the ollowing irst order nonlinear system x = x + u (4 he objective is to design the optimal control or this system so as to minimie the cost 4 ( J = x + x + u dt. In this example we compare our proposed neural network based optimal control with the exact optimal control. he exact optimal control can be ound analytically by solving the HJB equation [], which results in ( x ( x l( x u ( x = sign( bx + (5 b b r For the above example, the optimal control is given by u ( x = x x (6 For our proposed method the number o basis unctions and, hence, the number o centers, widths and weights were selected to be 5 and the neural network is trained by the adaptive EKF algorithm. Fig. shows the system states. Fig. shows a comparison o the value unctions corresponding to the exact optimal control ound in equation (6 and that o our proposed method. he comparison establishes that our proposed method is nearly optimal. Also, Fig.4 shows the norm o the RBFNN parameter vectors ŵ, 58

5 ĉ and ˆ σ. It can be seen that these parameters converge to some constant values, quickly. Fig.- System state Fig. - Perormance cost Fig. 4- Norms o the RBFNN parameter vectors Example : Consider the ollowing nd order SISO nonlinear system x x x = ( x + g( x u = + u (7 x+ x where the state vector x is assumed to be completely known. he goal is to ind the optimal control so as to minimie the perormance cost J = [ x Qx + u Ru] dt (8 where Q and R are the positive deinite matrices and chosen as R= and Q = I or this problem. It can be veriied that the system has an equilibrium point at x = (,. Again, the value unction V( x is approximated, using the output o an RBF neural network, as V ( x = x Px + ϕ ( x w w ϕ ( x (9 he positive deinite matrix P o the above value unction is obtained by lineariing the system (7, around its equilibrium point x=, and by solving the ollowing Riccati equation, i.e., A P + PA + Q PBR B P = (4 where A = F x x =, F B = u x=. u = u = he number o radial basis unctions selected or this problem is 8. he weights, centers, and widths o the RBFNN are estimated, orm the GHJB equation, using the proposed AEKF algorithm explained in the previous section. he optimal control law was then ound rom the estimated value unction, based on the proposed method. In addition, or comparative study, two other approximation techniques, namely the exact eedback lineariation (FL and successive Galerkin approximation (SGA were used to ind the optimal control. Consequently, the proposed optimal control, the FL control and the SGA control were each applied to the system, independently, under the same conditions. he exact eedback lineariation control law u FL or this SISO nonlinear system was adopted rom [4], which is given as 5 u FL = x + x x x +.44x.5( x + x (4 he control law obtained rom the SGA method, using 8 basis unctions, is given as [4] usga =.45x.5x.4784x (4 +.79xx xx +.588x All the simulations were carried out using MALAB/ SIMULINK. Fig.5 shows the system states x and x or the proposed control. A comparison o the value unction ound using these control methods with the initial conditions ( x, x [,] is shown in Fig.6. In this igure V FL is the value unction associated with the control law u FL, V SGA is the value unction associated with the control law u SGA and V is the value unction associated with the proposed control law based on the RBF neural network. From these igures it is clear that the proposed control method provides a better approximation o the optimal control as compared to those ound in [7] or the same number o basis unctions. It is shown in [7] that the controller with 5 basis unctions perorms better than the controller with 8 basis unctions. With 5 basis unctions, the proposed method perorms slightly better than 8 basis unctions and is similar to [7] with 5 basis unctions. Fig.7 59

6 shows the norm o the RBFNN parameter vectors ŵ, ĉ and ˆ σ, where they all converge to some constant values. neural networks (RBFNN. he neural network is trained by an adaptive extended Kalman ilter (AEKF, which estimates the RBFNN parameters, online. he proposed nonlinear optimal control method was applied to a st and a nd order SISO nonlinear system. In the case o st order nonlinear system the proposed approximation o the optimal control was compared with the exact optimal control to show that the proposed method indeed generates a nearly optimal control. Acknowledgement Partial support by the NSF grant No. 557 is grateully acknowledged. Fig. 5- System states Fig. 6- Perormance cost Fig. 7- Norms o the RBFNN parameter vectors VI. CONCLUSION his paper presents an approach to inding the optimal control law or nonlinear systems using neural networks. he design procedure approximately solves the generalied Hamilton-Jacobi-Bellman (GHJB equation by estimating the value unction using nonlinear radial basis unction REFERENCES [] Peter Dorato. Linear quadratic control, an introduction. Krieger publishing company: Florida, [] Curtis P. Mracek and James R. Cloutier. Control designs or the nonlinear benchmark problem via the state dependent Riccati equation method. International journal o robust and nonlinear control, 998; 8(4-5: 4-4 [] W. L. Garrard and J.M.Jordan. Design o nonlinear automatic light control systems. Automatica, 977; : [4] W.L.Garrard, D.F.Ennis and S.A. Snell. Nonlinear eedback control o highly maneuverable aircrat International journal o control, 99; 56(4: [5] G.Saridis, C.S.Lee. An approximation theory o optimal control or trainable manipulators. IEEE ransactions on Systems, Man, Cybernetics, 979; 9(:5-59. [6] R.Beard. Improving the closed-loop perormance o nonlinear systems. PhD hesis, Rensselaer Polytechnic Institute, roy, New York, 995. [7] C Park, P siotras. Sub-optimal eedback control using a successive- Galerkin algorithm. American control conerence ; : 96-9 [8] S. Kumar, R. Padhi and L. Behra Continuous-time Single Network Adaptive Critic or Regulator Design o Nonlinear Control Aine Systems Proceedings o 7 th World Congress o the International Federation o Automatic Control, 8, Seol, S. Korea. [9] F. Pourboghrat, H.Pongpairoj, Ziqian Liu, F.Farid. Dynamic neural Networks or modeling and control o nonlinear systems. Intelligent Automation and sot computing, ; 9(: 6-7. [] Chen, F.C., Liu C. Adaptively controlling nonlinear continuous-time systems using multilayer neural networks. IEEE ransactions on Automatic Control, 994; 9(6: 6-. [] FL Lewis, M Abu-Khala. A Hamilton-Jacobi setup or constrained neural network control. Proceedings o the IEEE International Symposium on Intelligent Control : -5. [] M.S.Ahmed Neural controllers or nonlinear state eedback L - gain control IEE Proceedings o Control heory Applications, 47(. [] J. K. Sing., D. K. Basu, and M. Nasipuri Improved K- means algorithm in the design o radial basis unction network. Proceedings o Convergent echnologies or Asia-Paciic Region, encon., [4] J. J. Murray, C. J. Cox, G. G. Lendaris and R. Saeks Adaptive Dynamic programming IEEE rans. On Systems, Man and Cybernetics, Part C., Vol., No., May. [5] Dan Simon. raining radial basis neural networks with the extended kalman ilter. Neurocomputing. [6] K. Rei, F. Sonnemann and R. Unbehauen. An EKF-based nonlinear observer with a prescribed degree o stability. Automatica 998 4(9 :9-. [7] R. Beard, G. Saridis and J. Wen. Galerkin approximation o the generalied Hamilton-Jacobi-Bellman equation. Automatica 997 (: