RESEARCHES on the coordination control of multiagent

Transcription

1 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS Dsrbued Opmal Consensus Conrol for Nonlnear Mulagen Sysem Wh Unnown Dynamc Jle Zhang, Huaguang Zhang, Fellow, IEEE, and Tao Feng Absrac Ths paper focuses on he dsrbued opmal cooperave conrol for connuous-me nonlnear mulagen sysems MASs) wh compleely unnown dynamcs va adapve dynamc programmng ADP) echnology. By nroducng predesgned exra compensaors, he augmened neghborhood error sysems are derved, whch successfully crcumvens he sysem nowledge requremen for ADP. I s revealed ha he opmal consensus proocols acually wor as he soluons of he MAS dfferenal game. Polcy eraon algorhm s adoped, and s heorecally proved ha he erave value funcon sequence srcly converges o he soluon of he coupled Hamlon Jacob Bellman equaon. Based on hs pon, a novel onlne erave scheme s proposed, whch runs based on he daa sampled from he augmened sysem and he graden of he value funcon. Neural newors are employed o mplemen he algorhm and he weghs are updaed, n he leas-square sense, o he deal value, whch yelds approxmaed opmal consensus proocols. Fnally, a numercal example s gven o llusrae he effecveness of he proposed scheme. Index Terms Adapve dynamc programmng ADP), compensaor, dsrbued conrol, mulagen sysem MAS), opmal consensus conrol. I. INTRODUCTION RESEARCHES on he coordnaon conrol of mulagen sysem MAS) should dae bac o he lae 980s, nally begnnng n he feld of moble robocs see [] for a more dealed hsory). In he lae 990s and early 2000s, he coordnaon conrol of MASs becomes a hghly acve research area, spurrng furher advances. Over he las decade, hs research area has been boomng, wh many new sysems beng proposed n applcaon areas rangng from mlary bale sysems o moble sensors newors, o commercal hghway and ar ransporaon sysems. In recen years, owng Manuscrp receved February 29, 206; revsed Ocober 27, 206; acceped July 9, 207. Ths wor was suppored n par by he Naonal Naural Scence Foundaon of Chna under Gran , Gran , Gran , and Gran , n par by he Naonal Hgh Technology Research and Developmen Program of Chna under Gran 202AA04004, n par by he IAPI Fundamenal Research Funds under Gran 203ZCX4, and n par by he Fundamenal Research Funds for he Cenral Unverses under Gran CX098 and Gran ZDPY0. Correspondng auhor: Huaguang Zhang.) J. Zhang and T. Feng are wh he School of Informaon Scence and Technology, Souhwes Jaoong Unversy, Chengdu 6756, Chna e-mal: jle0226@63.com; sunnyfengao@63.com). H. Zhang s wh he School of Informaon Scence and Engneerng, Norheasern Unversy, Shenyang 089, Chna e-mal: hgzhang@ eee.org). Color versons of one or more of he fgures n hs paper are avalable onlne a hp://eeexplore.eee.org. Dgal Objec Idenfer 0.09/TNNLS o he broad applcaons of consensus conrol of MASs, such as saelle formaon flyng [2], sensor newors [3], cooperave unmanned ar vehcles [4], newored mechancal sprng-mass sysems [5], and baery conrol [6], has receved compellng aenon from varous scenfc communes [7] []. Wh he research on MASs becomng a ho opc, he dsrbued opmal consensus conrol has been he very challengng problems. The desred conrol no only maes he MAS reach a consensus on behavor, bu also meanwhle opmzes her performance ndexes. From a praccal sense, he dsrbued opmal consensus conrol seers agens behavor o reach a consensus wh he lowes possble cos. For he sgnfcance of he opmal consensus conrol for MASs, n praccal applcaon, goes whou sayng. Therefore, a dsrbued consensus conrol s requred o mnmze every agen s cos by acng on self, accordng o he oucomes of s neghborhood agens. In essence, every agen depends on he acons of all neghbors besdes self. The coordnaon mechansm s very smlar o he wors [3], [4]. A presen, some excellen scholars have gven varous resuls on he opmal consensus problem of MAS, such as he lnear quadrac regulaor echnology [5], [6], he adapve learnng mehod [7], [8], and he model predcve conrol echnology [9]. The mehods n [5] [7] and [9] have desgned he opmal coordnaon conrol for lnear MAS. However, hey are very dffcul o be appled o nonlnear MAS. Alhough he mehod n [8] has presened a good scheme for nonlnear MAS by fuzzy adapve dynamc programmng ADP), he model of MAS mus be requred. Neverheless, he accurae nowledge of model s hard o be obaned n general, and hs pon movaes he research on he opmal consensus conrol for MAS wh unnown dynamc. In hs paper, he dsrbued opmal consensus conrol for MASs s presened, whch s fully no dependen on he dynamc of agens by ADP. ADP s a mehod, whch can oban he nearly opmal conrol for conrolled sysems. For he sae-of-he-ar developmens of ADP, see [23] [25] for deal. The opmal problem for sysems s ubquy n many dfferen felds, ncludng aerospace, process conrol, robocs, boengneerng, economcs, and managemen scence such as [26] and [38]), and connues o be an acve research area whn conrol heory. Before he arrval of he dgal compuer, only smple opmal conrol problems could be solved. Wh he emergence of dgal compuer, ADP echnology has been used o deal wh some complex problems ha s, he X 207 IEEE. Personal use s permed, bu republcaon/redsrbuon requres IEEE permsson. See hp:// for more nformaon.

2 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. 2 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS opmal conrol for nonlnear sysems) wh he help of opmal conrol heory and mehods, such as [27] [32]. However, hese conrol schemes are su for sysems whose dynamcs can be characerzed precsely. When he plan s dynamcs are poorly modeled, he sasfacory responses canno be obaned under hose desgned conrols. In addon o he above menoned resuls, some ADP mehods, whch do no requre a compleed dynamc of he sysems, have also been developed, such as [20], [33], [34], and [36] [39]. Inspred by he resuls [7], [20] [22], we presen a scheme o desgn he dsrbued opmal consensus conrol for mulagen dfferenal games. Here, he exra compensaor s employed o elmnae he dependence on he dynamc of sysems. Then, n lgh of he mulagen algorhm n [7] and [8], we gve he mplemenaon algorhm, whch brngs ogeher he leas-square mehod and adapve algorhm wh polcy eraon PI) [20]. Owng o he emergence of compuer, mos connuous-me sysems can be addressed n dscree verson. Based on hese deas and nspraons, he dsrbued opmal consensus conrol for connuous-me nonlnear MAS s proposed by denfyng one crc newor n he leas square sense, raher han by denfyng wo neuralnewor NN) srucures, such as [7]. Frs, he augmened sysem s consruced by he addonal compensaor. Then, he correspondng coupled Hamlon Jacob Bellman CHJB) equaons n dscree verson subjec o he augmened sysem are solved by he ADP algorhm. The man dea of our mehod s o sample he sae, sae dervave, and npu of he augmened sysems, and hen updae he weghs of NNs by leas squared echnque. The updang process s mplemened n he framewor of PI algorhm. The conrbuons of hs paper are exraced, as follows. ) Ths paper mproves and reduces he srucure of he algorhm n [20]. Specally, he cos funcon V does no requre o be solved as n [20, Fg. ]. 2) The dsrbued consensus conrol for nonlnear MASs s solved by ADP echnology wh he exra compensaors, crcumvenng he requremen of he sysem dynamcs. 3) In pracce, he dscrezed verson CHJB equaons and sampled dea are more convenen o synhesze he opmal consensus conrol by compuer, for connuous MASs. The res of hs paper s organzed as follows. In Secon II, some fundamenal conceps are nroduced, such as game heory and consensus for newors of agens. Secon III gves he scheme on he dsrbued opmal consensus conrol for MASs by exra compensaors, and opmzes her performance ndexes by ADP echnology. The mplemenaon of opmal consensus conrol s shown by he NN approxmaor n Secon IV. Fnally, a numercal example s gven o llusrae he effecveness of our mehod. Noaons: The Kronnecer produc s denoed by. The ransposon of marx A s denoed by A T. R n m denoes he n m-dmensonal marx n Eucldean space, and I n denoes he n-dmensonal deny marx n R n n. R n s he vecor wh all elemens. A denoes he 2-norm of marx A. dag{a, a 2,...,a n } denoes he bloc dagonal marx wh dagonal elemens a,,...,n. II. PRELIMINARIES Here, graph heory s used o descrbe he MAS as a vald mahemacal ool. The opology of a communcaon newor can be expressed by a weghed marx wheher he undreced or he dreced newor. Before dscussng he consensus problem on he MASs, we frs revew some basc conceps: graph heory, MAS, and consensus. A. Graph Theory Le G V, E, A) be a weghed graph, whch s used o descrbe he nformaon communcaon beween N agens. V s he nonempy fne se of nodes {v,...,v N }. An edge se E {e j v,v j )} V V and a weghed adjacency marx A [a j ] wh nonnegave adjacency elemens a j. Moreover, a j > 0fv,v j ) E, a j 0fv,v j ) E, and a 0forall,...n. v,v j ) E f and only f h agen can receve nformaon from jh agen drecly. N {v j V : v,v j ) E} s defned as he se of neghbors of node v. The node ndex belongs o he neger se I {, 2,...,N}. In assocaon wh G, he degree marx D dag{d, d 2,...d n } s dagonal marx, whose dagonal elemens are gven as d n j a j. The Laplacan marx L D A [l j ] s defned by l j a j and l nj a j. Laplacan marx has all row sums equal o zero. A dreced pah s a sequence of edges n a dreced graph of he form v,v 2 ), v 2,v 3 ),..., where v V. In hs paper, only smple graph s consdered. I has a spannng ree, f here s a node v, whch has a dreced pah from a node o any oher nodes n he graph. Whle he dgraph s srongly conneced, here s a dreced pah from every node o any oher nodes. The srong connecvy s an unnecessary and suffcen condon o a graph ha has a spannng ree. Here, he srongly conneced communcaon dgraph wh fxed opology s focused. B. Mulagen Sysem An MAS consss of mulple agens by a newor and can be descrbed by algebrac graph. Every node n newor graph represens an agen. Le x R n denoe he sae of node v. We call G x G, x) wh he sae x R Nn ) a newor or algebrac graph), where x [x T,...,x N T ]T. C. Consensus for Mulagen Sysem For he consensus problem, all agens are o reach he same sae,.e., x x j as,forall, j I, j. For he consensus problem wh a leader, all nodes of he MAS synchronze o he sae rajecory of he leader node,.e., x x 0 as, I, wherex 0 s he sae rajecory of he leader. III. PROBLEM FORMULATION Here, he MAS wh N agens n he form of communcaon dgraph G x s consdered. The dynamc of h agen s ẋ f x ) + g x )u )

3 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. ZHANG e al.: DISTRIBUTED OPTIMAL CONSENSUS CONTROL FOR NONLINEAR MAS 3 where x R n s he sae of agen v, u R m s s npu conrol, f x ) R n,andg x ) R n m. The global newor dynamc s ẋ f x) + gx)u 2) where x [x T, x T 2,...,x T N ]T R Nn s he global sae vecor, f x) [f T x ), f T x 2 ),..., f T x N )] T R Nn s he global dynamc vecor, gx) dag{g x )} R Nn Nm, I s he global npu vecor, and u [u T, ut 2,...,uT N ]T R Nm s he global conrol npu. The rajecory of he leader x 0 sasfes he dynamc ẋ 0 f x 0 ) 3) where x 0 R n and f x 0 ) s dfferenable. By he newor graph of MAS, he local neghborhood consensus error e for h agen s descrbed as e j N a j x x j ) + b x x 0 ) 4) where e [e, e 2,...,e n ] T e R n )andb 0. Noe ha he h agen s conneced o a leader f and only f b > 0. Remar : From he above-menoned expresson, we can see he consensus nformaon of MAS can be represened by he local neghborhood consensus error e. The MAS wll reach a consensus f e 0, as. The global error vecor for he graph G x s e Lx x 0 ) 5) wh L L + B) I n, e [e T, et 2,...,eT N ]T R Nn, and x 0 I x 0 R Nn wh I I n R Nn n and B dag{b, b 2,...,b N } R N N by nong ha b b and b j 0, j. Dfferenang 4) or 5), he dynamc of he local neghborhood consensus error for newor G x s gven by ė L f e ) + L gx)u 6) where f e ) f x) f x 0 ) wh f x 0 ) I f x 0 ) and L L + B ) I n. L s denoed as a row vecor, whch s he h row vecor of Laplacan marx L, has,l [l,...,l,...,l N ]. Smlarly, B [b,...,b,...,b N ]. Remar 2: Snce a j s zero when he node v j s no he neghbor of node v, he expresson 6) only conans conrol npus of all he neghbors of node v and self n newor G x. In fac, denoes ha he local neghborhood consensus error depends on he saes and he conrol npus from node v and all s neghbors. Here, hs paper s o desgn an opmal consensus conrol, whch s no dependence on model of sysem, o mae agens reach a consensus. Nex, we wll solve he problem by addng he exra compensaors. A. Augmened Sysem Wh he Exra Compensaor To crcumven ha ADP algorhm depends on he sysem model of every agen, he precompensaon echnque n [2] and [42] s used o augmen an exend error sysem. Frs, we desgn he exra compensaor, whch can be defned by any desred conrollable) npu affne dfferenal equaon 7) u a u ) + b u )w 7) where he sae vecor s he npu vecor u of he gven h agen, wh a sngulary a u 0andw 0. w R m. The augmened error sysem can be obaned by combnng he compensaor 7) wh sysem 6), as follows: ē F ē ) + G ē )w 8) wh he sae vecor ē [e T, ut ]T χ, χ R n+m,anda sngulary a ē 0,w 0). w becomes he conrol npu of he augmened sysem [ ] L f F ē ) e ) + L gx)u : χ R a u ) n+m and [ ] 0 G ē ) : χ R b u ) n+m) m. Defne he connuously dfferenable performance ndex cos funcon), whch requres o be opmzed as J ē ), w )) Ū ē τ), w τ))dτ 9) where Ū ē,w ) Q ē ) + w T R w, Q ) 0, R > 0. Problem : The problem requred o be solved s o desgn he dsrbued opmal consensus conrol w whou he nowledge of dynamc for MAS. The conrol no only mnmzes he local performance ndexes 9) subjec o 8), bu also maes all nodes agens) reach a consensus on he leader 3). Noe ha here he desgn problem on u s ransformed no he desgn problem on new consensus conrol w.nex, we wll presen he desgn scheme. B. ADP Algorhm on Mulagen Sysem Defnon Admssble Consensus Conrol): Gven sysem 8), a conrol w s defned o be admssble wh respec o he sae uly funcon Ū, ) on, wren w w, f he followng hold. ) w s connuous on w and w 0) 0. 2) w sablzes sysem 8) on. 3) 0 Ū ē,w )dτ <, ē. Under he gven admssble consensus conrol w, he local value funcon V ē ) for h agen s defned by V ē )) Ū ē,w )dτ Q ē ) + w T R w )dτ 0) and he correspondng local CHJB or nonlnear Lyapunov equaon) subjec o 8) s 0 H ē, Vē,w ) Ū ē,w ) + Vē T F ē ) + G ē )w ) Q ē ) + w T R w + Vē T F ē ) + G ē )w ). )

4 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. 4 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS Vē s he graden of he value funcon V ē ) wh respec o ē. The local coupled Hamlonan of Problem can be readly obaned as H ē, Vē,w ) Q ē ) + w T R w +V T ē F ē )+G ē )w ). 2) By he necessary condon of opmaly prncple, he followng conrol can be obaned: w 2 R G T ē )Vē. 3) If V ē ) s he local opmal value funcon, hen sasfes he followng CHJB equaon: mn w H ē, V ē,w ) 0 4) hen he correspondng local opmal consensus conrol s w 2 R G T ē )V ē. 5) When w n 4) s replaced by 5), we can oban Q ē ) 4 V T ē G ē )R G T ē )Vē + Vē T F ē ) 0. 6) Noe ha he opmal consensus conrol 5) can be obaned by V ē ). Acually, he opmal value funcon V ē ) [ha s, he soluon of 6)] s he opmal soluon o Problem. Nex, he defnon of opmal soluon of Problem wll be gven. Defnon 2 Opmal Soluon): A conrol w s referred o as an opmal consensus conrol of Problem f J J w ) J w ), w w. The performance value J s nown as he opmal soluon for Problem. Nex, we ge wo mporan conclusons [.e., ) and 2)] n Theorem. Theorem : Le V ē )>0 be a soluon o CHJB equaon 6), and he opmal consensus conrol w be gven by 5) n erm of he soluon V ē ). Then, he followng hold. ) The local neghborhood consensus error sysem 6) s asympocally sable. 2) The local performance value J ē 0), w ) s equal o V ē 0)); w s he opmal conrol. Proof: We frs prove ). Under he gven condons, V ē ) > 0 and V 0) 0. Furhermore, snce V ē ) sasfes 6), also sasfes ). Therefore, V ē ) can be aen as a Lyapunov canddae funcon for sysem 8). Tae he dervave of V ē ) wh respec o V ē ) Vē T ē Vē T F ē ) + G ē )w ) Q ē ) w T R w. Snce Q ē ) 0, R > 0, V ē )<0. Therefore, he local neghborhood consensus error sysem 6) s asympocally sable. Concluson 2) s very easy o be obaned by 9) and 0) and Defnon 2. I ndcaes ha he soluon of 6) s he opmal soluon o Problem. In order o solve he opmal consensus conrol problem, only needs o oban he soluon o he CHJB equaon 6) for he value funcon. However, s generally dffcul o solve he CHJB equaon. Therefore, PI algorhm s nroduced o solve he CHJB equaons [40], [4]. Acually, value funcon 0) can be wren as he dscrezed form by samplng, such as ha n [20], [42], and [43], as follows: V ē )) +T Ū ē τ), w τ))dτ T s he samplng perod. I can also be rewren as + V ē )) V ē + T )) +T +T Ū ē τ), w τ))dτ. 7) Ū ē τ), w τ))dτ 8) hen ae he dervaon on boh sdes of 8) wh respec o me V T ē ē ) V T ē ē + T ) Ū ē + T ), w + T )) Ū ē ), w )). 9) Lemma : Equaon 9) s equvalen o he CHJB equaon ). Proof [) 9)]: Nong ha w w, hen he cos funcon V ē )) along he correspondng error sae drven by w follows ha V ē )) 0, and by negrang ) over [, ], wege: and V T ē ē τ)dτ +T V T ē ē τ)dτ V ē τ))dτ Ū ē τ), w τ))dτ V ē )) V ē τ + T ))dτ V T ē ē τ + T )dτ + T ) +T V ē + T )) hen we can oban ha [ V Tē ē τ + T ) Vē T τ) ē ) ] dτ [ V ē τ + T )) V ē τ))]dτ V ē )) V ē + T )) +T +T Ū ē τ), w τ))dτ + Ū ē τ), w τ))dτ. Ū ē τ), w τ))dτ Ū ē τ), w τ))dτ

5 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. ZHANG e al.: DISTRIBUTED OPTIMAL CONSENSUS CONTROL FOR NONLINEAR MAS 5 Afer ang he dervave of he above-menoned equaons wh respec o me, holds ha V T ē ē ) V T ē ē + T ) Ū ē + T ), w + T )) Ū ē ), w )) whch s defnely 9). [9) )]: From 9), we oban ha V T ē ē ) V T ē ē + T ) + Ū ē + T ), w + T )) Ū ē ), w )). By nong ha T R, he above-menoned equaon always holds; hen, le T, and hen, holds ha V ē )) Ū ē ), w )) whch s equvalen o ). Ths complees he proof. Noe ha he admssble conrol w sasfes he opmal srucure 3) wh respec o he cos funcon V ē ); hus, he opmal consensus conrol can be obaned n Algorhm. Algorhm PI Algorhm Sar wh admssble nal polces w 0,...,w0 N. Sep : Polcy Evaluaon) Gven he N-uple of polces w,...,w N,solveforN-uple of coss V,...,V N usng ) or equvalenly H ē, Vē,w ) 0,,...,N 20) V T ē ) ē ) V T ē +T ) ē + T ) Ū ē + T ), w + T )) Ū ē ), w )),...,N. 2) Sep 2: Polcy Improvemen) Updae he N-uple of conrol polces usng 3) w + 2 R G T ē )V ē,,...,n. 22) Go o sep. I does no sop unl w converges o w, for. Nex, nspred by [30], we presen wo resuls on he convergence of he PI algorhm and he sably for he augmened nonlnear MAS. Theorem 2: Assumng he conrol wh erave sep be obaned by 22), V > 0, sasfyng he CHJB equaon 20). Then, he erave cos funcon s nonncreasng,.e., V + V. Thus, he every sep erave cos funcon s convergen. Defne he lm as V,.e., lm V V. Proof: Please see Appendx A. Theorem 2 ndcaes ha he erave cos funcon s convergen. The followng heorem wll prove ha sysem 8) under each of he erave conrols s asympocally sable. Theorem 3: Assumng he conrol wh erave sep be obaned by 22), V + > 0, sasfyng he CHJB equaon 20). Then, he every sep erave conrol w maes sysem 8) asympocally sable. Proof: Please see Appendx B. In wha follows, he NNs are used o solve he approxmae opmal soluon o he CHJB equaon by he leas-square mehod. Ths scheme s mplemened under he framewor of PI algorhm. IV. NEURAL NETWORK APPROXIMATION AND ALGORITHM IMPLEMENTATION A. Neural Newor Approxmaon As s nown o all, as a resul of he unversal approxmaon propery [44], NNs are naural canddaes o approxmae smooh funcons on compac ses. Therefore, here, n order o solve 4), we approxmae he local value funcon by he followng NN: M V ē ) c j φ j ē ) C T ē ) 23) j where c j s he weghs of he oupu layer, φ j ē ) s he acvaon funcons, and M s he number of neurons on he hdden layer. C and ē ) are he vecors combnng of c j and φ j ē ), respecvely j,...,m), such as C [c, c 2,...,c M ] T and ē ) [φ ē ), φ 2 ē ),...,φ M ē )] T. Noe ha he se {φ j ē )} M mus be lnearly ndependen. Snce Lemma holds, we subsue 23) no 9) C T ē )) ē ) ē + T )) ē + T )) Ū ē + T ), w + T )) Ū ē ), w )) 24) wh ē )) beng he graden of ē )) wh respec o ē ). Obvously, he algorhm mplemenaon 24) based on NN ulzes 9). Whle 9) s equvalen o ) by Lemma, herefore, he opmzaon problem, whch requres o be solved, has no been changed. Le z ē 0 + )T )) ē 0 + )T ) ē 0 + T)) ē 0 +T) and y Ū ē 0 +T), w 0 + T)) Ū ē 0 )T ), w 0 )T )). 0 s he nal samplng me. Z [z, z 2,...,z ] and Y [y, y 2,...,y ] s he samplng sep). By he sampled daa and he leas squares echnque [45], he wegh C T can be obaned by C T Y Z. 25) If here exss a soluon o 25), Z mus be nverble or here exss he pseudo nverse). Therefore, he samplng number s a leas M,.e., M. The followng lemma shows Z s nverble. Lemma 2: Le u u, such ha f + g u s asympocally sable. Gven ha he se {φ j } N s lnearly ndependen, hen T > 0, such ha x ) {0}, and he se {φ j x)) φ j x + T ))} N s also lnearly ndependen. Proof: The proof s smlar o [20, Lemma 3].

6 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. 6 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS Theorem 4: Le w w, such ha F +G w s asympocally sable. Gven ha he se {φ j } N s lnearly ndependen, hen T > 0, such ha ē {0}, and he se { φ j ē + T )) ē +T ) φ j ē )) ē )} N s also lnearly ndependen. Proof: Snce μ w, hen he cos funcon V ē )) s a Lyapunov funcon for he sysem ē F + G w and sasfes V ē )) V ē + T )) +T V T ē F + G w )dτ 26) over he me nerval [, + T ]. Tang he dervaon on boh sdes of he equaon wh respec o me and subsung 23) no 26), 26) can be wren as C T ē )) C T ē + T )) C T ē + T )) ē + T ) C T ē )) ē ). 27) Suppose ha Theorem 4 s no rue, hen T > 0, and here exss a nonzero consan vecor C R N, such ha C T [ ē )) ē + T ))] 0 ē ). 28) Ths mples ha we can oban he followng equaon from he me negraon of 28): C T [ ē )) ē + T ))] 0. 29) Ths means ha {φ j ē )) φ j ē +T ))} N s no lnearly ndependen, conradcng Lemma 2; hus T,> 0suchha ē 0 ), and he se { φ j ē)) φ j ē + T ))} N s lnearly ndependen. By 27), { φ j ē + T )) ē + T ) φ j ē )) ē )} N s also lnearly ndependen. Remar 3: Theorem 4 properly apples o solve 24) or 25) assocaed wh he augmened sysem 8). B. Algorhm Implemenaon In he mplemenaon process based on NNs, he PI algorhm s rewren as Algorhm 2. Algorhm 2 PI Algorhm Wh NN Le w 0 wh he nal weghs C 0 ) be an nal admssble polcy, hen he eraon beween ) Polcy evaluaon) solve for V se ) usng C st Y s Z s,,...,n 30) and 2) Polcy mprovemen) updae he conrol polcy usng w s+ 2 R G T ē ) ē )C st,,...,n 3) unl w converges o he opmal conrol w wh he correspondng wegh C. Remar 4: Accordng o Lemma 2, he above-menoned PI algorhm can be used for obanng he approxmae Fg.. Srucure of he sysem wh adapve conroller. opmal soluons. Obvously, he algorhm successfully crcumvens he requremen for he nowledge of sysem ), ha s, he sysem dynamc f x ) and he conrol marx g x ) can be unnown. The srucure of he sysem wh adapve conroller s shown n Fg.. From Fg., we frs sample he daa error, error dervave, and conrol) a every perod. Then, we oban he daa y and z y s a scalar and z s M vecor) by calculaon a every me momen. In order o guaranee Z s nverble, collec such daa no wo sorages.e., Y and Z ), respecvely, unl M. Noe ha Z s an M marx and Y s an vecor. I s valuable o emphasze ha he process of solvng he cos funcon V s elmnaed compared wh he mehod see [20, Fg. ]) n [20]. The proposed opmal algorhm requres only he sample daa of sae, sae dervave, and he correspondng conrol along he augmened sysem rajecores. The number of samples s over me T, and he samplng me sequence s { 0, 0 + T,..., 0 + T}. Then, z and y are calculaed by hose daa and held by he zero-order holder unl M. Furhermore, he daa Z and Y are used o updae he wegh C. Repea he updang process under he srucure of PI algorhm along he augmened sysem rajecores unl C s C s <ε ε s an deal parameer as he ermnal condon of he eraon algorhm). The algorhm flowchar s shown n Fg. 2. Remar 5: We need o sress ha he opmal consensus conrol w s he soluon o he Lyapunov equaon 6) [see 5) and 6)]; herefore, he opmal consensus conrol w s admssble sablzng conrol). In hs paper, Theorem 3 also saes ha under PI algorhm, he obaned conrol w s admssble sablzng conrol) n every eraon sep. In addon, he proof for he sably analyss of onlne mplemenaon has been proved n [20, Corollary 2]. Therefore, he proof s omed here. See [20, Sec. IV-B] for he more dealed explanaon.

7 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. ZHANG e al.: DISTRIBUTED OPTIMAL CONSENSUS CONTROL FOR NONLINEAR MAS 7 Fg. 4. Evoluon of he weghs for agen node). Fg. 2. Flowchar of PI algorhm. Fg. 3. Srucure of hree-node dgraph wh leader node. Fg. 5. Evoluon of he weghs for agen node) 2. V. SIMULATION In hs secon, we llusrae he effecveness of our scheme by a numercal example, and desgn he local opmal consensus conrol for MASs. Here, we consder he hree-node dgraph srucure wh leader node conneced o node, as shown n Fg. 3. The edge weghs and he pnnng gan n 4) are chosen as one. For he srucure n Fg. 3, he sae rajecory of he leader node s ẋ 0 f x 0 ) and each node dynamc s consdered as [ ] x + x 2 + 2x2 3 f x ) ) 0.5x + x 2 ) + 0.5x 2 + 2x 2 2 sn 2 x ) [ ] 0 g x ), 2, 3). snx ) Le W I, R 0.; Q ē ) ē T W ē, 2, 3). Tae T 0.2s, ε 0.0 and ē ) [e 2, e e 2, e u, e 2 2, e 2u, u 2 ]T,andM 6. Desgn he hree exra compensaors as a u ) 2u and b u ) sn 2 u ) for sysem 6). Then, we use he mehod n Secon IV o oban he deal weghs of NN approxmaor for every newor, respecvely, as n Fgs Obvously, afer fve eraon seps every eraon spends 2 s), all he weghs of hree NNs can converge o he deal value. Fg. 6. Evoluon of he weghs for agen node) 3. Fg. 7 shows he rajecores of he agens saes under he opmal consensus conrol w ). Afer abou 2.5 s, all saes reach a consensus. In he process of smulaon, our mehod does no need o solve he cos funcon V as ha see Fg. ) n [20]. Algorhm s much smpler. In addon, comparng wh our prevous wor [8], hs paper solves he opmal consensus problem on nonlnear MAS whose dynamc s unnown.

8 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. 8 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS From 22), holds Then, one has V + V Vē T G 2w +T R. [ w + T R w + w T R w 2w +T R w + w )] dτ. 34) Snce R s symmerc posve defne, hen here exss an orhogonal marx P, such ha R P T P,where dagλ,λ 2,...,λ m ), λ j > 0, j,..., m denoes he egenvalue of R. Thus, 34) s equvalen o Fg. 7. Trajecores of he agens saes. VI. CONCLUSION In hs paper, a model-free opmal consensus conrol algorhm has been proposed wh he exra compensaors. The HJB equaons have been solved by ADP, whch consss of he leas-squared echnque, NN approxmaor, and PI algorhm. The man dea of our mehod s o sample he nformaon of sae, sae dervave, and npu of he augmened sysem, and hen updae he weghs of NN by he leas-squared echnque. The updang process s mplemened under he framewor of PI. Fnally, an example has verfed he effecveness of our scheme. In he fuure, we wll sudy he opmal problem on swched sysems wh delay [46] n he lgh of he mehod n [47] and [48]. APPENDIX A. Proof of Theorem 2 Accordng o he CHJB equaon H ē, Vē,w ) 0, one can readly oban ha V T ē whch ndcaes ha V T ē F ē T Q ē w T F + V T ē G w + ē T Q ē w T R w V T ē G w 32) R w Vē T G w + Vē T G w +. 33) Consder he sae rajecory drven by he erave conrol w +,whchs ē F + G w +. The correspondng CHJB equaon s H ē, V + ) 0, whch yelds V +T ē hen one ge V + ē,w + F + Vē +T G w + ē T Q ē w +T R w + V V +T ē ē dτ + V T ē ē dτ [ w + T R w + w T R w Vē T G w + Vē T G w + ] dτ. V + V [ w + T For readably, le ϒ V + V P T P w + w T P T P w 2w +T P T P w + w )] dτ. 35) m P w ; hen, one has [ ϒ + T ϒ + m j λ j j ϒ T ϒ 2ϒ +T ϒ + ϒ λ j ϒ + T ϒ + ϒ T ϒ 2ϒ +T ϒ + ϒ ϒ + ϒ ) T ϒ + )] dτ )) dτ ϒ ) dτ 0. 36) Therefore, can be concluded V + V. Snce V s posve defne, he erave cos funcon V converges o as accordng o he Weersrass heorem. V B. Proof of Theorem 3 Consderng he sae rajecory drven by he erave conrol w +,.e., ē F +G w +, and ang he dervave of V wh respec o me along he foremenoned rajecory, holds ha V V T ē F + V T ē G w +. Smlar o he dervaon of 33), one can oban ha V ē T Q ē w T From 22), one has R w Vē T G w + Vē T G w +. 37) Vē T G 2w +T R. 38)

9 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. ZHANG e al.: DISTRIBUTED OPTIMAL CONSENSUS CONTROL FOR NONLINEAR MAS 9 Usng 37) and 38), hen followng he smlar operaons o he proof of Theorem 2, holds: V ē T Q ē w T R w + 2w +T R w w + ) ē T Q ē w T + 2w +T P T ē T Q ē w T + 2w +T P T P w P w w + P T P w P T P w 2w +T P T P w + ē T Q ē ϒ T ϒ + 2ϒ +T ϒ 2ϒ +T ϒ + m j λ j [ 2ϒ + T ϒ + ē T Q ē 0. ) + 2ϒ +T ϒ ϒ T ϒ ] Therefore, he closed-loop sysem drven by w + asympocally sable. REFERENCES [] L. E. Parer, Curren sae of he ar n dsrbued aunomous moble robocs, n Proc. In. Symp. Dsrb. Auo. Robo. Sys. DARS), 2000, pp [2] Z. Ln, B. Francs, and M. Maggore, Necessary and suffcen graphcal condons for formaon conrol of uncycles, IEEE Trans. Auom. Conrol, vol. 50, no., pp. 2 27, Jan [3] L. Xao, S. Boyd, and S. Lall, A scheme for robus dsrbued sensor fuson based on average consensus, n Proc. 4h In. Symp. Inf. Process. Sensor New., Los Angeles, CA, USA, 2005, pp [4] R. W. Beard, T. W. McLan, M. A. Goodrch, and E. P. Anderson, Coordnaed arge assgnmen and nercep for unmanned ar vehcles, IEEE Trans. Robo. Auom., vol. 8, no. 6, pp , Dec [5] H. Yan, F. Qan, H. Zhang, F. Yang, and G. Guo, H faul deecon for newored mechancal sprng-mass sysems wh ncomplee nformaon, IEEE Trans. Ind. Elecron., vol. 63, no. 9, pp , Sep [6] Q. We, D. Lu, G. Sh, and Y. Lu, Mulbaery opmal coordnaon conrol for home energy managemen sysems va dsrbued erave adapve dynamc programmng, IEEE Trans. Ind. Elecron., vol. 62, no. 7, pp , Jul [7] Y. Zheng and L. Wang, Fne-me consensus of heerogeneous mulagen sysems wh and whou velocy measuremens, Sys. Conrol Le., vol. 6, no. 8, pp , 202. [8] Y. Zheng and L. Wang, Consensus of swched mulagen sysems, IEEE Trans. Crcus Sys. II, Exp. Brefs, vol. 63, no. 3, pp , Mar [9] H. Zhang, G. Feng, H. Yan, and Q. Chen, Observer-based oupu feedbac even-rggered conrol for consensus of mul-agen sysems, IEEE Trans. Ind. Elecron., vol. 6, no. 9, pp , Mar [0] H. L, P. Sh, D. Yao, and L. Wu, Observer-based adapve sldng mode conrol for nonlnear Marovan jump sysems, Auomaca, vol. 64, pp , Feb [] H. L, P. Sh, and D. Yao, Adapve sldng-mode conrol of Marov jump nonlnear sysems wh acuaor fauls, IEEE Trans. Auom. Conrol, vol. 62, no. 4, pp , Apr [2] W. Ren, R. W. Beard, and E. M. Ans, A survey of consensus problems n mul-agen coordnaon, n Proc. Amer. Conrol Conf., vol. 3. Porland, OR, USA, Jun. 2005, pp [3] Q. We, D. Lu, and F. L. Lews, Opmal dsrbued synchronzaon conrol for connuous-me heerogeneous mul-agen dfferenal graphcal games, Inf. Sc., vol. 37, pp. 96 3, Oc [4] K. G. Vamvoudas and F. L. Lews, Mul-player non-zero-sum games: Onlne adapve learnng soluon of coupled Hamlon Jacob equaons, Auomaca, vol. 47, no. 8, pp , 20. [5] H. Zhang, F. L. Lews, and Z. Qu, Lyapunov, adapve, and opmal desgn echnques for cooperave sysems on dreced communcaon graphs, IEEE Trans. Ind. Elecron., vol. 59, no. 7, pp , Jul s [6] J. Ma, Y. Zheng, and L. Wang, LQR-based opmal opology of leaderfollowng consensus, In. J. Robus Nonlnear Conrol, vol. 25, no. 7, pp , 205. [7] K. G. Vamvoudas, F. L. Lews, and G. R. Hudas, Mul-agen dfferenal graphcal games: Onlne adapve learnng soluon for synchronzaon wh opmaly, Auomaca, vol. 48, no. 8, pp , 202. [8] H. Zhang, J. Zhang, G.-H. Yang, and Y. Luo, Leader-based opmal coordnaon conrol for he consensus problem of mulagen dfferenal games va fuzzy adapve dynamc programmng, IEEE Trans. Fuzzy Sys., vol. 23, no., pp , Feb [9] G. Ferrar-Trecae, L. Galbusera, M. P. E. Marcand, and R. Scaoln, Model predcve conrol schemes for consensus n mul-agen sysems wh sngle- and double-negraor dynamcs, IEEE Trans. Auom. Conrol, vol. 54, no., pp , Nov [20] D. Vrabe and F. Lews, Neural newor approach o connuous-me drec adapve opmal conrol for parally unnown nonlnear sysems, Neural New., vol. 22, no. 3, pp , [2] R. Saes and C. Cox, Adapve crc conrol and funconal ln neural newors, n Proc. IEEE Conf. Sys., Man, Cybern., Oc. 998, pp [22] C. Cox, S. Sepnews, C. Jorgensen, R. Saes, and C. Lews, On he desgn of a neural newor auolander, In. J. Robus Nonlnear Conrol, vol. 9, no. 4, pp , 999. [23] J.S,A.G.Baro,W.B.Powell,andD.Wunsch,Handboo of Learnng and Approxmae Dynamc Programmng. NewYor,NY,USA:Wley, [24] F.-Y. Wang, H. Zhang, and D. Lu, Adapve dynamc programmng: An nroducon, IEEE Compu. Inell. Mag., vol. 4, no. 2, pp , May [25] F. L. Lews and D. Vrabe, Renforcemen learnng and adapve dynamc programmng for feedbac conrol, IEEE Crcus Sys. Mag., vol. 9, no. 3, pp , Aug [26] R.-J. Wa and M.-C. Lee, Inellgen opmal conrol of sngleln flexble robo arm, IEEE Trans. Ind. Elecron., vol. 5, no., pp , Feb [27] H. Zhang, Q. We, and D. Lu, An erave adapve dynamc programmng mehod for solvng a class of nonlnear zero-sum dfferenal games, Auomaca, vol. 47, no., pp , Jan. 20. [28] H. Zhang, L. Cu, and Y. Luo, Near-opmal conrol for nonzero-sum dfferenal games of connuous-me nonlnear sysems usng snglenewor ADP, IEEE Trans. Cybern., vol. 43, no., pp , Feb [29] J.-L. Zhang, H.-G. Zhang, Y.-H. Luo, and H.-J. Lang, Nearly opmal conrol scheme usng adapve dynamc programmng based on generalzed fuzzy hyperbolc model, Aca Auom. Snca, vol. 39, no. 2, pp , 203. [30] M. Abu-Khalaf and F. L. Lews, Nearly opmal conrol laws for nonlnear sysems wh saurang acuaors usng a neural newor HJB approach, Auomaca, vol. 4, no. 5, pp , May [3] H. Zhang, Q. We, and Y. Luo, A novel nfne-me opmal racng conrol scheme for a class of dscree-me nonlnear sysems va he greedy HDP eraon algorhm, IEEE Trans. Sys. Man, Cybern. B, Cybern., vol. 38, no. 4, pp , Aug [32] Q. We, D. Lu, and H. Ln, Value eraon adapve dynamc programmng for opmal conrol of dscree-me nonlnear sysems, IEEE Trans. Cybern., vol. 46, no. 3, pp , Mar [33] D. Vrabe, O. Pasravanu, M. Abu-Khalaf, and F. L. Lews, Adapve opmal conrol for connuous-me lnear sysems based on polcy eraon, Auomaca, vol. 45, no. 2, pp , Feb [34] F. L. Lews and K. G. Vamvoudas, Renforcemen learnng for parally observable dynamc processes: Adapve dynamc programmng usng measured oupu daa, IEEE Trans. Sys. Man, Cybern. B, Cybern., vol. 4, no., pp. 4 23, Feb. 20. [35] Y. Cao, W. Yu, W. Ren, and G. Chen, An overvew of recen progress n he sudy of dsrbued mul-agen coordnaon, IEEE Trans. Ind. Informa., vol. 9, no., pp , Feb [36] H. Zhang, L. Cu, X. Zhang, and Y. Luo, Daa-drven robus approxmae opmal racng conrol for unnown general nonlnear sysems usng adapve dynamc programmng mehod, IEEE Trans. Neural New., vol. 22, no. 2, pp , Dec. 20. [37] Z. N, H. He, X. Zhong, and D. V. Prohorov, Model-free dual heursc dynamc programmng, IEEE Trans. Neural New. Learn. Sys., vol. 26, no. 8, pp , Aug. 203.

10 Ths arcle has been acceped for ncluson n a fuure ssue of hs journal. Conen s fnal as presened, wh he excepon of pagnaon. 0 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS [38] X. Zhong, H. He, H. Zhang, and Z. Wang, Opmal conrol for unnown dscree-me nonlnear Marov jump sysems usng adapve dynamc programmng, IEEE Trans. Neural New. Learn. Sys., vol. 25, no. 2, pp , Dec [39] X. Zhong and H. He, An even-rggered ADP conrol approach for connuous-me sysem wh unnown nernal saes, IEEE Trans. Cybern., vol. 47, no. 3, pp , Mar [40] H. Modares, M.-B. N. Ssan, and F. L. Lews, A polcy eraon approach o onlne opmal conrol of connuous-me consraned-npu sysems, ISA Trans., vol. 52, no. 5, pp. 6 62, 203. [4] S. J. Brade, B. E. Ydse, and A. G. Baro, Adapve lnear quadrac conrol usng polcy eraon, n Proc. IEEE Amer. Conrol Conf., vol. 3. Jul. 994, pp [42] J. J. Murray, C. J. Cox, G. G. Lendars, and R. Saes, Adapve dynamc programmng, IEEE Trans. Sys., Man, Cybern. C, Appl. Rev., vol. 32, no. 2, pp , May [43] A. Jadbabae, Recedng horzon conrol of nonlnear sysems: A conrol Lyapunov funcon approach, M.S. hess, Dep. Conrol Dyn. Sys., Calforna Ins. Technol., Pasadena, CA, USA, [44] K. Horn, M. Snchcombe, and H. Whe, Unversal approxmaon of an unnown mappng and s dervaves usng mullayer feedforward newors, Neural New., vol. 3, no. 5, pp , 990. [45] G.-B. Huang and C.-K. Sew, Exreme learnng machne: RBF newor case, n Proc. 8h Conrol, Auom., Robo. Vs. Conf., Kunmng, Chna, Dec. 2004, pp [46] Y.-E. Wang, X.-M. Sun, and F. Mazenc, Sably of swched nonlnear sysems wh delay and dsurbance, Auomaca, vol. 69, pp , Jul [47] L. Ba, Q. Zhou, L. Wang, and Z. Yu, Observer-based adapve conrol for sochasc nonsrc-feedbac sysems wh unnown baclash-le hyseress, In. J. Adap. Conrol Sgnal Process., o be publshed, do: 0.002/acs [48] L. Wang, H. L, Q. Zhou, and R. Lu, Adapve fuzzy conrol for nonsrc feedbac sysems wh unmodeled dynamcs and fuzzy dead zone va oupu feedbac, IEEE Trans. Cybern., o be publshed, do: 0.09/TCYB Huaguang Zhang M 03 SM 04 F 4) receved he B.S. and M.S. degrees n conrol engneerng from he Norheas Danl Unversy, Jln Cy, Chna, n 982 and 985, respecvely, and he Ph.D. degree n hermal power engneerng and auomaon from Souheas Unversy, Nanjng, Chna, n 99. He joned he Deparmen of Auomac Conrol, Norheasern Unversy, Shenyang, Chna, n 992, as a Pos-Docoral Fellow for wo years. Snce 994, he has been a Professor and he Head of he School of Informaon Scence and Engneerng, Insue of Elecrc Auomaon, Norheasern Unversy. He has auhored or co-auhored over 280 journal and conference papers and sx monographs and co-nvened 90 paens. Hs curren research neress nclude fuzzy conrol, sochasc sysem conrol, neural newors based conrol, nonlnear conrol, and her applcaons. Dr. Zhang was a recpen of he Ousandng Youh Scence Foundaon Award from he Naonal Naural Scence Foundaon Commee of Chna n He was also a recpen of he IEEE TRANSACTIONS ON NEURAL NETWORKS 202 Ousandng Paper Award and he Andrew P. Sage Bes Transacons Paper Award 205. He was named he Cheung Kong Scholar by he Educaon Mnsry of Chna n He s he E-Leer Char of he IEEE CIS Socey and he former Char of he Adapve Dynamc Programmng & Renforcemen Learnng Techncal Commee on he IEEE Compuaonal Inellgence Socey. He was an Assocae Edor of he IEEE TRANSACTIONS ON FUZZY SYSTEMS from 2008 o 203. He s an Assocae Edor of Auomaca, he IEEE TRANSACTIONS ON NEURAL NETWORKS, he IEEE TRANSACTIONS ON CYBERNETICS, andneurocompung. Jle Zhang receved he B.S. degree n auomaon from he Laonng Unversy of Technology, Jnzhou, Chna, n 2007, he M.S. degree n conrol heory and conrol engneerng from he Kunmng Unversy of Scence and Technology, Kunmng, Chna, n 200, and he Ph.D. degree n conrol heory and conrol engneerng from Norheasern Unversy, Shenyang, Chna, n 204. He joned he School of Informaon Scence and Technology, Souhwes Jaoong Unversy, Chengdu, Chna, n 204, as a Lecurer. Hs curren research neress nclude mulagen sysem, approxmae dynamc programmng, renforcemen learnng, and game heory. Tao Feng receved he B.S. degree n mahemacs and appled mahemacs from he Chna Unversy of Peroleum, Bejng, Chna, n 2008, and he M.S. degree n fundamenal mahemacs and he Ph.D. degree from he School of Informaon Scence and Engneerng, Norheasern Unversy, Shenyang, Chna, n 20 and 206, respecvely. He s currenly a Lecurer wh he College of Informaon Scence and Technology, Souhwes Jaoong Unversy, Chengdu, Chna. Hs curren research neress nclude approxmae dynamc programmng, nverse opmal conrol, and mulagen sysems.