One-Sided Adaptation for Infinite-Horizon Linear Quadratic N-person Nonzero-sum Dynamic Games

Transcription

1 Proceedngs of the European Control Conference 7 Kos, Greece, July -5, 7 TuB45 One-Sded Adaptaton for Infnte-Horzon Lnear Quadratc N-person Nonzero-sum Dynamc Games Xaohuan Tan and Jose B Cruz, Jr, Lfe Fellow, IEEE Abstract In ths paper, we consder a class of nfntehorzon dscrete-tme lnear quadratc N-person games, n whch one of the players lac the complete nformaton of the game Wth the assumptons on perfect state nformaton pattern and steady state feedbac strateges, we convert the orgnal game problem nto a multvarable adaptve control problem by tang use of the concept of fcttous play and the scheme of adaptve control For the proposed adjustment procedure, we prove that each element of the estmates converges to ts correspondng true value under the condton of persstent exctaton I INTRODUCTION Most research n game theory has been devoted to the class of games wth complete nformaton, n whch all decson maers have the nowledge of the mathematcal structures as well as the exact values of the parameters of games In ths paper, however, we are more nterested n the class of games wth ncomplete nformaton For ths class of games, Harsany [], [], [3] ntroduced the noton of types and proposed to formulate the problem based on Bayes rules Although mathematcally sold, how to obtan the nformaton of the probablty densty dstrbuton over compettors types remans a bg challenge n applcatons As a wdely used model of learnng, fcttous play [4] s another method to deal wth the games wth uncertantes, n whch decson maers behave as f they thn that they are facng a set of statonary but unnown strateges dstrbuton of compettors In a dynamc case, the decson maers may change ther belefs about the compettors strateges after each round of play, whch leaves whether or not play converges a ey queston If t does not, then t not plausble that players wll mantan that assumpton about compettors strateges Fudenberg and Kreps [5] examned some suffcent condtons under whch fcttous play converges Adaptve control [6], as a mature branch of control theores, provdes decson maers the ablty of self-adjustment or self-modfcaton n accordance wth changng envronment or structure The convergence analyss of adaptve control also gves us lots of nsght on how to deal wth the convergence problem concerned n fcttous play In ths paper, we propose to ncorporate the concept of fcttous play wth the technque of adaptve control to solve a class of game problems wth ncomplete nformaton To ths end, we frst consder a class of nfnte-horzon dscrete-tme lnear quadratc N-person game and complete The authors are wth the Department of Electrcal and Computer Engneerng, The Oho State Unversty, Columbus, OH 43, USA emal: tan4@osuedu; jbcruz@eeeorg) nformaton, gvng out the state feedbac Nash equlbrum solutons under perfect state nformaton pattern n Secton II For such games wth one player who lacs the complete nformaton of the game, we propose an adaptaton scheme for the player n Secton III Persstent exctaton and the analyss of parameter convergence are also gven Smulatons n Secton V numercally verfy our theoretcal results and the conclusons are drawn n the last secton II INFINITE-HORIZON DISCRETE-TIME LQ N -PERSON NONZERO-SUM DYNAMIC GAME Let N {,,,N} be the player s set and K be the maxmum number of stages n a dscrete-tme game such that K {,,,K} Denote the strategy sets by {Γ, N, K} and the decson control) varables by {u U, N, K} wth no chance moves allowed Defne the system dynamcs and the cost functonal)s J x KQ Kx K + x + Ax + NB u ) K x Q x + j Nu j Rj uj ) where x R n, u U, A R n n, B Rm R n m, Q Rn n, R j and R j Rm m >, Q > for all,j N, j, K For any gven control strateges {γ Γ, N, K}, the decsons of the players are completely determned by the relatons u γη ) 3) where η N, K) denotes the nformaton set of P N) [7] Under the perfect state nformaton pattern γ {x }, the feedbac Nash equlbrum can be obtaned by followng the steps of dynamc programmng It s gven n [7] that the game defned by ) and ) admts a unque state feedbac Nash equlbrum strateges γ x ) N, K) u : arg mn J γx ) L x 4) u U where L N, K) are matrces of approprate dmensons, satsfyng the set of lnear matrx equatons L R + B S+B ) B S+ A + B j L j 5) j N,j ISBN:

2 TuB45 and S N, K) are obtaned recursvely from S A + j N +Q + j N B j L j S + A + j N B j L j L j Rj Lj 6) wth SK Q K The game defned above becomes a game wth nfnte horzon when K We drop the tme ndces from the weghtng matrces n ) and let S N) denote the lmt of S K) as K for fxed assumng that ths lmt exsts and s ndependent of Lewse, let L N) be the lmt of L K) as K Then these lmtng matrces necessarly satsfy the followng two algebrac matrx equatons L R + B S B ) B S F 7) S Q + L R L + F + B L ) S F + B L ) 8) where F, Q N) are defned by F A + B j L j 9) j N,j Q Q + j N,j L j R j L j ) The correspondng steady state feedbac strategy for P N) s then u : arg mn J γ x ) L x,,, ) u U Remar So far there s no exstence proof for an nfntehorzon games n general Obtanng the condtons on the parameters of the game that wll guarantee the exstence of a soluton set to 7) and 8) s qute a challenge and beyond the scope of ths research Hereby we wll assume that the soluton set exsts In the nfnte-horzon case, t s natural to assume that the problem s well-defned, n the sense that there exsts at least one set of control sequences that renders t a fnte cost Condtons whch ensure ths are stablzablty of the matrx par F,B ) N) and detectablty of the par F,C ) N) where C s a matrx such that C C Q The latter condton s also referred to detectablty of the par F, Q ) N) When players strateges are fxed at γ x ) L x and requrng that the resultng nfntehorzon game problem s well-defned, S s a soluton to a dscrete-tme algebrac Rccat equaton, as concluded n the followng lemma Lemma For N, f the par A, B ) s stablzable, A, Q ) s detectable and B B s nvertble, then S s the unque postve sem-defnte soluton to the dscrete-tme algebrac Rccat equaton DARE) S Q + F S F F S B R +B S B ) B S F Proof: Tang expanson of 8) and pluggng 7) nto t, we obtan S Q + L R + B S B )L + F S F + F S B L + L B S F Q + F S F + F S B R + B S B ) R + B S B )R + B S B ) B S F F S B R + B S B ) B S F F S B R + B S B ) B S F Q + F S F F S B R + B S B ) B S F ) It can be found n any textboo on lnear control systems that the unque nonnegatve-defnte soluton to ) exsts when the par F,B ) s stablzable and the par F, Q ) s detectable We wll show next that when B B s nvertble, the stablzablty on A,B ) mples the stablzablty on F,B ), and the detectablty on A, Q ) mples the detectablty on F, Q ) When the par A,B ) s stablzable, there exsts some L such that A + B L s Hurwtz When B B s nvertble, we may construct L L B B ) B B L B j L j 3) j N,j such that F + B L s also Hurwtz, snce F + B L A + B L 4) e, F,B ) s also stablzable Smlarly, the detectablty of A, Q ) means that there exsts some M such that A + C M s Hurwtz, where C C Q Snce Q >, C >, such that there also exsts M M C ) C M B j L j ) 5) satsfyng j N,j F + C M A + C M 6) whch mples that F, Q ) s detectable Ths completes the proof of the lemma III ONE-SIDED ADAPTATION FOR N -PERSON GAMES In many practcal cases such as strategc bddng problems n the deregulated electrcty maret, although all players commt to Nash equlbrum strateges [8], not all players are able to access the complete nformaton of the game, especally the weghtng parameters of compettors cost functons as these are prvate and essental nformaton The lac of complete nformaton of a game maes t mpossble for the decson maers to determne ther strateges In ths case, one may tae use of the concept of fcttous play that each player starts wth belevng that the compettors are playng stable strateges and then each player chooses the best response to the belefs The process s repeated 734

3 TuB45 wth updated observatons In the sequel, we adopt the dea and propose to tae use of avalable adaptve control technque to estmate other decson maers strateges before mang each player s own decson We wll frst propose an adjustment procedure n Secton III-A, then dscuss the parameter convergence problem n Secton III-B A Adaptaton Mechansm Desgn Consder an nfnte-horzon dscrete-tme LQ N-person nonzero-sum game defned n Secton II Suppose all players commt to play the steady state feedbac Nash equlbrum strateges u γ x ) L x N,,,), such that the system dynamcs s x + Ax + B u + B j L j x 7) j N,j where for N and,, we assume that x R n, A R n n, B R n m, u U Rm, and L R m n Also suppose P N) s the only decson maer who s unable to access the complete nformaton of the game, and thus has to estmate all other players control strateges before determne hs/her own controls Defne a subset of players by N {,,,+,,N} Let ˆL j j N,,,) be the estmate of L j at the th tme step, and let ˆx be the estmate of x by P Wthout loss of generosty, we let The state estmate calculated by P s then ˆx + Ax + B u + B j ˆLj x 8) j N Denote L j ˆL j Lj, the dfference between the system dynamcs 7) and the state estmate 8) s ˆx + x + [ L B B N] x 9) Let m θ N m and suppose m θ to be such that m θ n to ensure that the column vectors of the n m θ matrx [B B N ] are lnear ndependent, e, there s no redundant control varables If the m θ m θ matrx [B B N ] [B B N ] s nvertble, then we defne the estmaton error vector e + R m θ as [B e + B N] [ B B N]) [ B B N] ˆx+ x + ) ) Also defne the unnown parameter matrx Θ + R n m θ as Θ + [ˆL ] ˆLN ) and ntroduce the regressor vector φ + z [x + ] x R n, where z s a delay operator, and the nomnal L N vector Θ + [L L N ] R n m θ, such that 9) s reduced to e Θ φ, Θ Θ Θ ) whch s ndeed a multvarable lnear parametrc model Let the ntal guess to be Θ, and choose the normalzed gradent adaptve algorthm for updatng Θ to be Θ + Θ Γφ e m ) 3) where < Γ Γ < I n s a gan matrx, and m κ + φ φ wth κ > beng a desgn parameter Lemma The adaptve law 3) guarantees that lm tr[ Θ Θ ] exsts and s fnte Proof: Introduce the postve defnte functon V Θ ) tr[ Θ Γ Θ ], where tr[ ] s the trace operator for a matrx The tme ncrement of V along 3) s V Θ + ) V Θ ) [ tr m ) [ tr m ) e Θ φ e + e φ Θ e φ Γφ e m ) φ Γφ ) ] m ) e Let λ max [Γ], ) be the maxmum egenvalue of Γ whch satsfes the condton < Γ Γ < I n By lnear algebra theory, we have the nequalty < φ Γφ λ max [Γ]φ φ Wth the desgn parameter κ >, we have < φ Γφ m ) φ Γφ κ + φ φ Denote α λ max [Γ] >, then λ max[γ]φ φ φ φ e e V Θ + ) V Θ ) αtr[ ] m ) λ max [Γ] )] 4) whch mples that lm V Θ ) lm tr[ Θ ΓΘ ] exsts and s fnte, snce we have shown that V Θ ) s nonncreasng and bounded from below V Θ ) > ) Furthermore, matrx Γ > can be decomposed nto Γ DΛD, where Λ s the dagonal matrx consstng of the egenvalue of Γ, and D s the orthogonal matrx consstng of correspondng egenvectors, such that V Θ ) tr[ Θ DΛD Θ ] tr[λ Θ Θ ] The exstence of lm V Θ ) thus mples the exstence of lm tr[ Θ Θ ] snce < λ[γ ] < Ths completes the proof of Lemma 735

4 TuB45 B Persstent Exctaton and Parameter Convergence We observe from the estmaton error equaton ) that when e, t s suffcent but not necessary that Θ To mae Θ Θ Θ small when e s small, the regressor φ should be suffcently rch, whch s crucal for ensurng parameter convergence wth the gradent algorthm We thus requre that condton of persstent exctaton [9] s satsfed for the regressor φ, upon whch we prove next n Theorem that each element n the matrx ˆL j j N,,, ) wll converge to ts correspondng true value exponentally Note that we let n the prevous secton for clarfcaton purpose Here we wll leave t as a varable snce any player who s dong estmaton may adopt the same adjustment mechansm as 3) wth approprately defned e and Θ Theorem If the sgnal ψ φ m s persstently exctng over the tme sequence set {,,}, the adaptve law 3) ensures that lm ˆL j ) ql L j ) ql exponentally, where ˆL j ) ql and L j ) ql are the q,l) th element of the matrces ˆL j ) ql and L j ) ql, respectvely j N, q, l n) Proof: Consder the adaptve law 3) wth Γ I, whch can be rewrtten as Θ + Θ φ φ m ) Θ I ψ ψ ) Θ 5) where ψ φ m can be vewed as the observaton vector For a postve defnte functon V Θ ) tr[ Θ Θ ], V Θ + ) V Θ ) tr[ Θ I ψ ψ ) I ψ ψ ) Θ ] tr[ Θ Θ ] tr[ Θ ψ ψ + ψ ψ ) ψ ψ ) Θ ] tr[ Θ ψ ψ Θ ] 6) Denote the state transton matrx as Φσ+δ,σ) σ+δ σ I ψ ψ ) such that Θ σ+δ Φσ +δ,σ) Θ σ Then for arbtrary δ >, σ > t we have [ σ+δ ] V Θ σ+δ ) V Θ σ ) tr Θ ψ ψ Θ where σ V Θ σ ) tr[ Θ σ S Θ σ ] 7) σ+δ S Φ,σ)) ψ ψ Φ, σ) 8) σ If ψ s persstently excted over the nterval σ,σ + δ), the matrx S s stll postve defnte because the observablty property of the system s the same Decompose S > nto Γ DΛ S D, such that tr[ Θ σ S Θ σ ] tr[ Θ σ DΛ S D Θσ ] tr[λ S Θ σ Θσ ] Denote the mnmum egenvalue of S as λ mn [S] >, then tr[ Θ σ S Θ σ ] tr[λ S Θ σ Θσ ] λ mn [S]tr[ Θ σ Θ σ ] whch results n the nequalty 7) to be reduced to V Θ σ+δ ) λ mn [S])V Θ σ ) βv Θ σ ) 9) where β,) as V Θ σ ) > for Θ σ Furthermore, note that the d th d N ) dagonal bloc of the m θ m θ matrx Θ Θ, L j j L j N ), s n L j ) l n L j j L L j ) l n L j ) l where L j ) ql s the q,l) th element of the n matrx L j j N, q, l n,,,) Therefore tr[ Θ Θ ] and 9) s equvalent to j N q l j N q l n L j σ+δ ) ql β It follows that m j n L j t +Tδ ) ql β T j N q l whch mples that lm T + j N q l n L j ) ql 3) j N q l j N q l exponentally Ths s equvalent to say that lm + j N q l n L j σ) ql 3) n L j t ) ql 3) n L j t +Tδ ) ql 33) n L j ) ql 34) snce we have nown from Lemma that the lmt exsts and s fnte By the defnton of lmt, 34) means that ε >, K ε such that > K ε, we have j N q l n L j ) ql < ε 35) Snce L j ) ql for all j N, q, l n,,,, the worst case s when all but one element, say L j ) ql, are zeros In ths case, 35) turns out to be L j ) ql < ε 36) whch agan by defnton mples that lm + L j ) ql, e, lm + ˆL j ) ql L j ) ql Ths completes the proof of Theorem To satsfy the condton on PE, we may ntroduce reference sgnals r and u r N,,,) to be traced by the 736

5 TuB45 state and the controls respectvely, such that the cost functon J N) s revsed to J + [x r ) Q x r ) + u j ujr ) R j uj ujr )] 37) j N where r s a nown trajectory by all players Snce the problem s well-defned, we need to restrct the control references u r N) to be B u r r + Ar 38) N such that n steady state ) when x r and u u r, the system can stll be controlled at a fnte cost The Nash equlbrum soluton to the game wth cost functon 37) can stll be obtaned after tang a coordnate transformaton, e, denote y x r and v u ur The state feedbac Nash soluton for N n the new coordnates s v : arg mn J γ y ) L y 39) v U whch s equvalent to u : arg mn J γ x ) L x + u r u L r ) 4) U n the orgnal coordnates, where L s defned n 7) Gven the form of control n 4) and the restrcton 38), the state equaton ) can be expressed as x + A + N L ) x + r + A + N L ) r 4) whch shows that PE condton can be satsfed by choosng r properly At the every step of the game, wth all estmates ˆL j j N ), P N) can calculate hs/her best response strategy ˆL from 7) and 8), whch n ths adaptaton s ˆL R + B ŜB ) B Ŝ ˆ F 4) Ŝ ˆ Q + ˆ F Ŝ ˆ F ˆ F ŜB R + B ŜB ) B Ŝ ˆ F 43) where ˆ F, ˆ Q N,,,) are defned by ˆ F A + B j ˆLj 44) j N,j ˆ Q Q + j N,j IV NUMERICAL SIMULATIONS ˆL j Rj ˆLj 45) Now we consder a three-person game wth vector control, n whch P lacs the complete nformaton of the costs of other players and thus employs the adaptaton mechansm proposed to estmate other players strateges before mang hs/her own decsons The parameters of the state equaton are supposed to be A B B B The weghtng matrces n the cost functonals are 5 6 Q 5 Q Q R 8 R 5 R 3 5 R 5 5 R 5 5 R 3 4 R R 3 3 R By choosng the desgn parameters κ 4, Γ I 4 and the tracng sgnal r [7sn5), 5cos9), 5cos9 ), 7sn35)], we see from Fg and Fg 3 that each element n ˆL and ˆL 3 matrces converges well to ther correspondng true values n L and L 3, whch maes the elements n ˆL also converge to ther true values n L as shown n Fg V CONCLUSIONS In ths paper, we proposed to solve a class of nfntehorzon LQ Nash games wth ncomplete nformaton by tang use of the concept of fcttous play and the scheme of adaptve control We frst consdered a class of dscrete-tme lnear quadratc N-person game wth complete nformaton, gvng out the state feedbac Nash equlbrum solutons under perfect state nformaton pattern We proved that under condtons of stablzablty and detectablty, the matrx S s actually the unque postve sem-defnte soluton to the assocated coupled dscrete-tme algebrac equaton, whch lad out the base on whch a decson maer can determne hs/her best response strateges aganst the estmates to the compettors strateges For such a class of N-person games wth vector-valued decson varables, we assumed that all players commt to play the steady state feedbac equlbrum strateges, n whch an arbtrary player P lacs the complete nformaton of the game Under some condtons, we converted the game 737

6 TuB45 3 L ) and ˆL ) L ) 3 and ˆL ) L ) and ˆL ) 4 L ) 4 and ˆL ) L 3 ) 3 and ˆL 3 ) 3 5 L 3 ) and ˆL 3 ) L 3 ) 4 and ˆL 3 ) L 3 ) and ˆL 3 ) 8 L ) and ˆL ) 4 L ) and ˆL ) 4 5 L 3 ) and ˆL 3 ) 5 L 3 ) and ˆL 3 ) L ) 3 and ˆL ) 3 6 tme step L ) 4 and ˆL ) 4 tme step 5 L 3 ) 3 and ˆL 3 ) tme step 4 L 3 ) 4 and ˆL 3 ) tme step Fg Control gan L dashed) and ts estmates ˆL L ) ql or ˆL ) ql) s the q, l) th element of L or ˆL ) sold), where Fg 3 Control gan L 3 dashed) and ts estmates ˆL 3 L 3 ) ql or ˆL 3 ) ql) s the q, l) th element of L 3 or ˆL 3 ) sold), where 5 5 L ) and ˆL ) L ) 3 and ˆL ) L ) and ˆL ) 8 4 L ) 3 and ˆL ) tme step 5 L ) and ˆL ) 5 L ) 4 and ˆL ) L ) and ˆL ) 5 5 L ) 4 and ˆL ) tme step Fg Control gan L dashed) and ts estmates ˆL sold), where L ) ql or ˆL ) ql) s the q, l) th element of L or ˆL ) problem nto a mult-nput mult-output adaptve control problem, for whch P employs an adjustment procedure usng normalzed gradent algorthm to estmate the compettors strateges at every tme step before mang hs/her own decsons We proved n theory that each element of P s estmates converges to ts correspondng equlbrum value under the condton of persstent exctaton Numercal smulatons of a three-person game verfed our clams REFERENCES [] J C Harsany, Games wth ncomplete nformaton played by Bayesan player, Part I: The basc model, Management Scence, vol 4, no 3, pp 59 8, Nov 967 [], Games wth ncomplete nformaton played by Bayesan player, Part II: Bayesan equlbrum ponts, Management Scence, vol 4, no 5, pp 3 334, Jan 968 [3], Games wth ncomplete nformaton played by Bayesan player, Part III: The basc probablty dstrbuton of the game, Management Scence, vol 4, no 7, pp 486 5, Mar 968 [4] G W Brown, Iteratve soluton of games by fcttous play, n Actvty Analyss of Producton and Allocaton, T C Koopmans, Ed New Yor: Wley, 95, pp [5] D Fudenberg and D Kreps, Lectures on learnng and equlbrum n stratgc-from games, n CORE Lecture Seres Mmeo, 99 [6] K J Åström, Theory and applcaton of adaptve control - a survey, Automatca, vol 9, pp , 983 [7] T Başar and G J Olsder, Dynamc Noncoopertve Game Theory, revsed ed New Yor, NY: Academc Press, 998 [8] R W Ferrero, V C Ramesh, and S M Shahdehpour, Applcaton games wth ncomplete nformaton for prcng electrcty n deregulated power pools, IEEE Trans on Power Systems, vol 3, no, pp 84 89, Feb 998 [9] G Tao, Adaptve Control Desgn and Analyss Hoboen, NY: Wley- Interscence, 3 738