Dynamic Team Decision Theory

Transcription

1 Dynamc Team Decson Theory EECS 558 Proec Repor Shruvandana Sharma and Davd Shuman December, 005

2 I. Inroducon Whle he sochasc conrol problem feaures one decson maker acng over me, many complex conrolled sysems nvolve mulple decson makers (DMs. In general, hese decson makers dffer n hree ways: hey conrol dfferen decson varables; hey base her decsons on dfferen nformaon; and 3 hey have dfferen goals or obecves. A eam s an organzaon n whch here s a sngle goal or payoff common o all decson makers. Accordngly, n he general eam decson problem, he decson makers dffer only n he frs wo respecs descrbed above. The eam decson problem was nally examned by Marschak and Radner n he conex of organzaon heory, and has snce found applcaons n a wde range of engneerng and economc sysems, ncludng elecrc power, raffc, and communcaons sysems. Despe s applcably, however, very few general resuls are known abou he eam decson problem due o s nheren dffcules. The purpose of hs repor s o provde a bref nroducon o dynamc eam decson heory, ncludng an overvew of some of he key resuls. Secon II of hs paper formulaes he eam decson problem. Secon III summarzes hree suaons n whch a dynamc eam problem can be decomposed no several smaller dynamc eam problems. Secon IV proceeds o descrbe one example of such decomposon he one sep communcaon delay problem. Fnally, Secon V summarzes Wsenhausen s mporan proof ha all dscree varable and mos connuous varable dynamc eam problems can be mapped no equvalen sac eam problems. II. Formulaon of he Dynamc Team Decson Problem The general dynamc eam decson problem s formulaed as follows: The eam consss of decson makers, DM, =,,, The decson makers ac n a fxed sequence (w.o.l.g. DM, DM,, DM. Whou hs assumpon, he problem becomes a non-sequenal eam problem ha falls no a more general (and more dffcul class of problems The basc random varables are represened by he vecor ξ ξ has a known probably dsrbuon, F(ξ, and s ndependen of he conrol law Informaon z avalable o each DM s gven by z = η (ξ,u, =,,, η:=(η, η,, η s called he nformaon srucure of he eam Admssble decson rules are of he form u = γ (z The obecve s o fnd a decson rule γ:=(γ, γ,, γ o mnmze J(γ =E γ {w(ξ,u}, where w( s he oal cos funcon. For a fxed polcy γ, he above expecaon s wh respec o he basc random varables ξ w(, η(, and F(ξ are assumed o be common nformaon, known a pror o all decson makers

3 By way of defnon, a eam s sad o be sac f η, =,,,, s ndependen of u,.e., z = η (ξ, and dynamc oherwse. The model descrbed on page 558 of [] wh observaons made a M poss and conrol npus appled a k saons a each dscree sep of a lengh T me horzon can easly be convered no he above formulaon by re-labelng he T*k conrollers DM, DM,, DM. If we ake k= and assume perfec recall n he model of [], hen we have he sochasc conrol problem. In general, he nformaon avalable o each decson maker can be any subse of prevous observaons and conrol npus. Much of he dffculy of he dynamc eam decson problem orgnaes from he funconal dependences beween he conrollers nformaon, and, herefore, analyss of he nformaon srucure plays a crcal role n solvng any eam decson problem. For more on specfc nformaon srucures (alernavely called nformaon paerns ncludng he classcal, n-sep delayed sharng, and amnesc paerns, see [] and [5]. III. Decomposon of he Dynamc Team Decson Problem Due o he nheren dffculy of he eam decson problem, a group of auhors posed he queson when can a dynamc eam problem be decomposed no several smaller dynamc eam problems? Three such scenaros are dscussed below. I s worh nong ha even when he eam decson problem can be decomposed, here s no guaranee ha he resulng problems wll be racable. For hs reason, hese resuls are bes vewed as more phlosophcal resuls ha can be of help n ryng o gan furher nsgh no specfc examples whn hs class of problems. Prelmnary: Precedence Dagrams o Represen Informaon Srucures In [4], Yoshkawa refnes slghly he noaon of precedence dagrams, orgnally presened n [] by Ho and Chu. See hese papers for examples of precedence dagrams. The hree mporan defnons follow: u affecs z, wren R, f here exs ξ, u a :=[u,u,,u -,u a,u+,,u ] T, and u b :=[u,u,,u -,u b, u+,,u ] T such ha η (ξ,u a η (ξ,u b DM s a preceden of DM, wren DM DM, f R, or here exs r, s,, such ha Rr, rrs,, R Informaon z of DM s nesed n nformaon z of DM, wren DM DM, f here γ γ exss a measurable funcon f such ha for any ξ, z = f (z oe ha for causaly o hold, f DM s a preceden of DM, hen DM canno be a preceden of DM. Ye, s possble for DM o be a preceden of DM, and for he nformaon of DM o be nesed n he nformaon of DM a he same me. See Example n [4] for such a sysem. Decomposon Scenaro I: Independen Paron Informally, Yoshkawa [4] defnes an ndependen paron o be a paron for whch here s no precedence relaon beween any par of groups and he oal cos funcon s gven by he sum of 3

4 he cos funcon for each group. More formally, a paron (H,H,,H K s an -paron of H={DM : =,,,} f : ( (, {,,..., K}, : DM ' H and DM ' H DM ' DM ' w( ξ, u = K = w ( ξ, u H Theorem of [4] essenally says ha f a eam H has an ndependen paron, hen he opmal soluon can be found by opmzng each sub-eam accordng o s sub-cos funcon w (ξ,u H. Combnng hese opmal sub-soluons wll resul n he opmal soluon of he orgnal problem. In [4] Eq. 4, Yoshkawa also relaxes condon ( above o allow for monooncally ncreasng cos funcons wh respec o every w. Decomposon Scenaro II: Sequenal Paron A sequenal paron (s-paron s also defned by wo condons. [4] Frs, he decsons of a group do no affec he nformaon of any prevous group. Second, he nformaon of all members of a group s nesed n he nformaon of he followng group. More formally, a paron (H,H,,H K s an s-paron of H f : ( ( {,,..., K } { +,..., K} DM ' H DM ' H : DM ' DM ' {,,..., K } DM ' H DM ' H+ : DM ' DM ' oe ha boh condons above can be deermned from a precedence dagram alone. The sequenal nesedness of nformaon leads o Theorem of [4], whch bascally says ha he dynamc programmng echnque can be appled o a eam wh an s-paron o fnd he opmal soluon. For example, consder he precedence dagram n Fgure below. Suppose we fx he acons for all bu he las group, H 3. Snce he acons of he las group do no affec ha of any of he prevous groups, he acons of H 3 can be opmzed ndependenly of he prevous groups. Wh knowledge of H 3 s opmal acons correspondng o every possble se of prevous acons, we are lef wh a new wo group s-paron and an updaed cos funcon ha depends only on he acons of H and H. The acons of H can hen be opmzed, and so forh. Once he frs group s opmal polcy s deermned, he followng groups opmal polces can be chosen accordngly based on he complee descrpon of he opmal polces under all fxed bu arbrary prevous acons, and hs complees he deermnaon of he eam s opmal polcy. H H H 3 Fgure. Example of an S-Paron 4

5 Decomposon Scenaro III: Paral esedness We have descrbed wo forms of nformaon srucures for whch he opmal soluons, f hey exs, can be found by solvng subproblems of he orgnal problem. In [] Ho and Chu dscuss a hrd form of nformaon srucure, paral nesedness, n combnaon wh a se of addonal assumpons. Under hs scenaro, a unque opmal soluon s guaraneed, and s form s known o be affne n z ; however, due o he addonal assumpons, he doman of applcably s more lmed. In [], Ho and Chu assume he basc random varables are ndependen, zero mean, and only Gaussan; he cos funcon s quadrac, and he nformaon srucure akes on he lnear form: z = H ξ + D where he marces H and D are known o all DMs. (A [] leverages he heorem of Radner ha for a sac eam wh he se of assumpons (A, he opmal conrol law s gven by: u z Az b b b b c Q Q A H XH S XH T T T T T T = γ ( = +, (,,..., =, ( = u An nformaon srucure s sad o be parally nesed f DM DM mples DM he follower can always deduce he acons of s precedens for a gven polcy, γ. DM,.e. The key resul n [] s for a dynamc eam sasfyng (A and havng parally nesed nformaon srucure z = H ξ + D DM DM I s shown ha he above nformaon srucure s equvalen o an nformaon srucure n sac form for any fxed se of conrol laws. I hen follows from Radner s resul ha he opmal conrol law for each member of such a dynamc eam exss, s unque, and s lnear n z. As a fnal remark on paral nesedness, noe ha whle an s-paron s smlar concepually o paral nesedness on he group level, paral nesedness s neher a necessary nor suffcen condon for he exsence of an s-paron. IV. The One Sep Communcaon Delay Problem The decomposon mehods descrbed n Secon III have found applcaons n some mporan decenralzed conrol problems. One such applcaon s he one sep communcaon delay problem. Ths problem feaures a dscree me sochasc sysem wh wo conrollers and he followng sysem dynamcs: X + = f ( X, U, U,, =,,..., T u Y = h ( X, M, =, 5

6 where X,,,, T, M, M,, M T are muually ndependen. Addonally, he nformaon srucure s one sep delay sharng,.e. Z = ( Y, Y,..., Y, Y, U, U,..., U, =, The cos funcon s descrbed by w = T = c ( X +, X, U and he precedence dagram for hs sysem s shown below n Fgure. I s easy o see from hs dagram ha H = {H, H, H T } represens a sequenal paron, and he nformaon srucure s also parally nesed. In he case ha he sysem s lnear, he nose s Gaussan, and he cos funcon s quadrac (LQG, Theorem of Ho and Chu [] for eams wh parally nesed nformaon srucure can be nvoked o show ha he opmal soluon s affne n he nformaon avalable o each conroller. H H H 3 u u u 3 u u u 3 H T- H T u T u T u T u T Fgure. Precedence Dagram for One Sep Communcaon Delay Problem Varaya and Walrand prove n [3] ha even n he one sep communcaon delay problem whou he LQG assumpons, here s no loss of opmaly n resrcng he search for he opmal γ * o separaed conrol laws of he form φ (Y,F, where F s he condonal dsrbuon of X - gven (Y,,Y -,U,,U -. As n he sochasc conrol problem wh mperfec nformaon, F, he resul of a flerng problem, s ndependen of he conrol law and he cos funcon. I was orgnally conecured n [], Asserons 8 and 9, ha analogous separaon resuls [wh and whou LQG assumpons] hold n he case of n-sep delay sharng. However, Varaya and Walrand also show n [3] ha hs s no he case by provdng an LQG counerexample ha has an opmal soluon ousde he space of separaed laws. Fnally, [] and [4] also dscuss he applcaon of decomposon mehods o oher classes of decenralzed conrol problems ncludng perodc sharng paerns and herarchcal conrol sysems. 6

7 V. Equvalence of Sac and Dynamc Teams As we noed earler, much of he dffculy n dealng wh dynamc eams s due o he nerdependences among he sysem varables. Because of hs fac, was hypoheszed for a long me ha dynamc eam problems are more dffcul han sac eam problems. However, n a maor breakhrough from 988, Wsenhausen [5] shows ha all dscree varable and mos connuous varable dynamc eam problems can be mapped no equvalen sac eam problems. Wsenhausen s basc mehodology s o shf he dependence amongs he decson makers nformaon varables no he cos funcon. The problem does no become any less complcaed n dong so, bu he resul s a sac eam problem. In he followng, we presen a summary of he key deas used o prove he equvalence of dynamc and sac eams. Alernave Descrpon of he Inrnsc Model The dynamc eam model dscussed n prevous secons s known as he nrnsc model [wh a sequenal orderng]. To faclae he proof, Wsenhausen defnes a new model for he dynamc eam problem, whch he shows o be equvalen o he nrnsc model. The key elemens of Wsenhausen s model follow: The eam consss of decson makers, DM, =,,, whch are assumed o ake acons sequenally n me whou volang causaly The basc random varables are ω = ( ω0, ω,..., ω They are generaed from he underlyng probably space and assumed o be ndependen;.e. ( Ω, B, P dp( ω = = dp( 0 ω Observaon a me s generaed as y = g ( ω, ω, u, u,..., u, 0 For each =,,, he observaons avalable o DM are descrbed by k {,,..., } Admssble decson rules are of he form where u = γ ({ y τ } τ k γ Γ c and Γ c s he subse of all measurable funcons from τ k ( Yτ, Yτ ( U, U 7

8 The common obecve s o fnd a decson rule γ = [ γ, γ,..., γ ] o mnmze he expeced oal cos γ J( γ : = E { V( ω, y, y,..., y, u, u,..., u } 0 All decson makers know a pror V(, g (, k and dp( ω Key Observaons Leadng o Sac Reducon We now observe some key feaures nheren n hs formulaon ha lead owards an approach for sac reducon: For a gven polcy γ, he on pdf of all he random varables n he above formulaon s compleely deermned by he on pdf dp( ω of ω ( ω, ω,..., ω do no explcly appear n V( ω0, y, y,..., y, u, u,..., u ( u, u,..., u are compleely deermned by Therefore, J ( γ s compleely deermned by he on pdf ω0, y, y,..., y. We now consder he dfference beween a sac and a dynamc eam. In a dynamc eam, are dependen on each oher, and hs dependence s refleced n her on pdf. ( In a sac eam, y depends only on ω 0 and ω, and hence, he dependence amongs he y s s much weaker. Furher, f for every, y depends only on ω, hen he y s are ndependen and her on pdf can be wren as he produc of he margnal pdfs. ( Is here a way o separae he dependence amongs he y s from he dynamc eam probably measure ( o ge a probably measure of he form of a sac eam (? (3 The answer s gven by he Common denomnaor condon [5], Eq. (4., under whch a condonal probably measure on some space can be wren as a produc of an uncondonal probably measure on he same space, and a measurable funcon of he varables nvolved, whereby he funcon absorbs he dependences from he orgnal probably measure. Thus, f f ( denoes he measurable funcon and Q( dy denoes he uncondonal probably measure, one can wre, for all, Py ( S ω, u, u,..., u f( y, ω, u, u,..., u Qdy ( = ( y, y,..., y 0 0 S (4 { y } = 8

9 Example : Gaussan Sgnalng To llusrae hs condon, we consder he Gaussan Sgnalng example dscussed n [5]. The sysem consss of wo agens (= conrollng decson varables u and u. Ther respecve observaons are gven by y = ωand y = ω + u, whereω 0 s degenerae; ω ~ (0, ; ω ~ (0, ; and ω0, ω, ω are ndependen. The cos funcon s V = ( u y + k ( u. The decson rules o be consdered are of he form, u = γ( y and u = γ ( y. Thus we have, y Pdy ( ω0 = exp dy and, π ( y u ( u yu y P( dy ω0, u = exp dy = exp exp dy, π π from whch one can denfy he funcons f ( and Q( dy, =, as, f( y, ω 0 =, ( u yu f( y, ω0, u = exp, y Q( d y = exp dy and, π y Q( dy = exp dy π We wll reurn o hs example below as we dscuss he sac reducon. The Sac Reducon Havng llusraed he common denomnaor condon, we wll now see how helps n answerng he queson n (3. I s proved n [5], Lemma 5., ha for dsrbuons sasfyng he common denomnaor condon, he on dsrbuon of ω0, y, y,..., y can be wren as f ( y, ω, u, u,..., u Q( dy P( dω = = (5 As can be seen, he erm n he square bracke defnes a probably dsrbuon on ω0, y, y,..., y whch s he produc of ndvdual dsrbuons, and hus, of he form expeced n (. Therefore, n he presen form, he dsrbuon does correspond o a sac eam. However, he on dsrbuon sll conans a erm (he frs erm ha reflecs he dependences among he sysem varables. I s mporan o noe a hs sage ha he goal s o desgn a polcy γ ha mnmzes he obecve funcon J ( γ and hence o a desgner, wo sysems wll be equvalen f hey acheve he same goal va he same se of desgns. Thus, n he presen conex, he equvalence beween a dynamc and a sac eam requres ha we can fnd a sac sysem ha acheves he same obecve wh he probably dsrbuon of observaon varables sasfyng he condons saed n (. The dealed condons defnng he equvalence of he wo sysems are gven n 9

10 [5]. Wh hs concep of equvalence n mnd, le us now see wha mpac he new form of probably dsrbuon (5 has on he obecve funcon. We can wre he orgnal obecve as, J ( γ = L V ( ω0, y, y,..., y, u, u,..., u f( y, ω0, u, u,..., u P0( dω0 Q( dy (6 = = ω y,..., y, 0 = V( ω0, y, y,..., y, u, u,..., u f( y, ω0, u, u,..., u P0( dω0 Q( dy y,..., y ω = = 0 = Vˆ( y, y,..., y, u, u,..., u Q ( dy y,..., y = (7 As can be seen, by absorbng he dependency par no he cos funcon, he probably dsrbuon has been brough exacly no he form of an ndependen se of observaons. Thus, f a new sysem s defned n whch he prmve random varables { ω * } = are generaed accordng o ( Y,,, and he observaons are generaed as Y Q = * * * * * * * * y = g ( ω0, ω, u, u,..., u = ω,, hen he sysem n (7 wll correspond o a sac eam. The wo sysems n (6 and (7 acheve exacly he same goals hrough exacly he same se of decsons, and are herefore equvalen. The dfference s ha n he frs sysem, he dependence les amongs he sysem varables and he complexy s conaned n he probably dsrbuon; n he second sysem, however, he dependence s absorbed n he cos funcon and he dsrbuon has a smpler form. The above proof by Wsenhausen esablshes ha o every dynamc eam problem ha sasfes he common denomnaor condon, here corresponds a sac eam problem wh he same complexy as he orgnal dynamc eam problem. To llusrae how he complexy shfs o he cos funcon n he sac reducon, we wll now complee Example. Snceω 0 s degenerae, we can fnd he new cos funcon correspondng o he sac eam drecly from he funcons found n Example. Thus, V ˆ( y, y, u, u = V ( ω0, y, y, u, u f( y, ω0, u, u,..., u = u yu ( = ( u y + k ( u exp I s clear ha he new cos funcon s much more complcaed han he orgnal one and hence he new sysem s no easer o analyze han he orgnal sysem. In summary, s mporan o noe ha by esablshng he equvalence beween dynamc and sac eams, Wsenhausen has no proved ha he reducon of dynamc eam problems o sac eam problems makes hem easer o solve. Raher, he conrbuon of he paper s n dsprovng he conecure ha sac eams are less complcaed han dynamc eams. Thus, he soluon of complex eam problems, wheher dynamc or sac, sll remans an acve area of nvesgaon. 0

11 VI. Concluson The decenralzed naure of nformaon n he dynamc eam heory problem poses que a challenge n fndng opmal conrol laws. Ths paper has presened an overvew of wo of he more general resuls avalable regardng hs class of problems: scenaros under whch he problem can be decomposed no several smaller problems, and he equvalence of dynamc and sac eams. Whle hese resuls may provde some nsgh no specfc problems, seems o us ha even afer decades of research n he area, he eam heory landscape remans surprsngly unplowed consderng he wdespread applcably of he framework. I s our hope ha furher progress on such eam heorec problems wll be forhcomng. VII. References [] H.S. Wsenhausen, Separaon of esmaon and conrol for dscree me sysems, Proc. IEEE, vol. 59 (97, [] Y.C. Ho and K.C. Chu, Team decson heory and nformaon srucures n opmal conrol problems Par I, IEEE Trans. Auoma. Conrol, vol.7 (97, 5-. [3] P. Varaya and J. Walrand, On delayed sharng paerns, IEEE Trans. Auoma. Conrol, vol. 3 (978, [4] T. Yoshkawa, Decomposon of dynamc eam decson problems, IEEE Trans. Auoma. Conrol, vol. 3 (978, [5] H.S. Wsenhausen, Equvalen sochasc conrol problems, Mahemacs of Conrol, Sgnals, and Sysems, vol. (988, 3-.