Abstract. (Under the direction of Dr. Russell E. King and Dr. Thom J. Hodgson)

Transcription

1 Abtact WEI, WENBIN. Quantfyng Shaed Infomaton Value n a Supply Chan Ung Decentalzed Makov Decon Pocee wth Retcted Obevaton. (Unde the decton of D. Ruell E. Kng and D. Thom J. Hodgon) Infomaton hang n a two-tage and thee-tage upply chan tuded. Aumng the cutome demand dtbuton known along the upply chan, the nfomaton to be haed the nventoy level of each upply chan membe. In ode to tudy the value of haed nfomaton, the upply chan examned unde dffeent nfomaton hang cheme. A Makov decon poce (MDP) appoach ued to model the upply chan, and the optmal polcy gven each cheme detemned. By compang thee cheme, the value of haed nfomaton can be quantfed. Snce the optmal polcy maxmze the total poft wthn a upply chan, allocaton of the poft among upply chan membe, o tanfe cot/pce negotaton, alo dcued. The nfomaton hang cheme nclude full nfomaton hang, patal nfomaton hang and no nfomaton hang. In the cae of full nfomaton hang, the upply chan poblem modeled a a ngle agent Makov decon poce wth complete obevaton (a tadtonal MDP) whch can be olved baed on the polcy teaton method of Howad (960). In the cae of patal nfomaton hang o no nfomaton hang, the upply chan poblem modeled a a decentalzed Makov decon poce wth etcted obevaton (DEC-ROMDP). Each agent may have complete obevaton of the poce, o may have only etcted obevaton of the poce. In ode to olve the DEC-ROMDP, an evolutonay coodnaton algothm ntoduced, whch pove to be effectve f coupled wth polcy petubaton and multple tat tatege.

2

3 Bogaphy Wenbn We wa bon n Wuhan, Hube povnce, Chna, and gew up n Shyan, an automoble cty famou fo t well-known Chnee tuck Dongfeng. Befoe he came to US n 200 fo h Ph.D. n the Indutal Engneeng Depatment at Noth Caolna State Unvety, he tuded n Bejng fo x and a half yea n Behang Unvety (alo known a Bejng Unvety of Aeonautc and Atonautc), whee he eceved h B.S. degee and M.S. degee fom Mechancal Engneeng and Automaton School. Dung the umme of 2005, he woked a a techncal tudent n SAS Inttute Inc., Cay, NC.

4 Acknowledgement I codally thank my co-advo, D. Kng and D. Hodgon. The content uppot and patent gudance wll be teaued fo good. My pecal thank to my colleague Lauen Dav, whoe detaton eeach wa cloely elated wth mne. We pent plenty of tme hang pogammng code and exchangng eeach nght. I alo thank my colleague Kent Mahall, a obot expet, who ha bought much joy to me. I owe many thank to my Mom, my Dad and my bothe fo the love and encouagement dung o many yea. My thank go to all my fend fo the nvaluable fendhp and paye. Epecally thoe fend whom I made when I joned Wayne Ly bble tudy goup dung The tme wa exctng, fun and memoable. Dung I joned Intenatonal Bble Study (alo called IBS) goup nea the NC State Unvety campu. It ha been enjoyable to hae evey Fday evenng at IBS gettng away fom wok and wohpng God.

5 Table of Content Lt of Fgue... v Lt of Table... v Chapte Ovevew.... Intoducton....2 Refeence... 3 Chapte 2 Makov Decon Pocee wth Retcted Obevaton 5 2. Intoducton Mathematcal Model fo a ROMDP ROMDP Notaton LP Model fo a MDP Poblem NLP Model fo a ROMDP Poblem Methodology fo Solvng a ROMDP Petubaton Polcy Petubaton P Petubaton Expementaton Genec Poblem Supply Chan Poblem Concluon Refeence... 2 Chapte 3 Decentalzed Makov Decon Pocee wth Retcted Obevaton Intoducton Model Sngle agent MDP and ROMDP DEC-ROMDP (Mult-agent) DEC-ROMDP Algothm Cae Study: A Two-Agent DEC-ROMDP poblem Geneal two-agent DEC-ROMDP Model DEC-ROMDP Applcaton to a Supply Chan Poblem Concluon Refeence Chapte 4 Quantfyng the Value of Infomaton and Tanfe Pce Negotaton n a Supply Chan Intoducton Backgound Lteatue Revew Infomaton hang n a 2-tage Supply Chan Modelng Aumpton Model Paamete v

6 Makov Decon Poce Appoach Infomaton Flow Methodology Expementaton Degn of Expement I Degn of Expement II Infomaton hang n a 3-tage Supply Chan Modelng and Methodology Expementaton Degn of Expement III Degn of Expement IV Tanfe Cot Negotaton Detemnaton of Tanfe Cot n a 2-tage Supply Chan Detemnaton of Tanfe Cot n a 3-Stage Supply Chan Concluon Refeence Appendx Chapte 5 Summay and Futue Reeach Summay and Futue Reeach Refeence... 7 v

7 Lt of Fgue Fgue 2.: the pecentage of poblem olved optmally (Supply Chan ROMDP) Fgue 2.2: The Aveage eo of Poblem Unolved Optmally (Supply Chan ROMDP) Fgue 2.3: Max Eo of Poblem Unolved Optmally (Supply Chan ROMDP)... 2 Fgue 3.: Geneal Two-Agent DEC-ROMDP Model wth Dffeent Obevaton Fgue 3.2: Pefomance fo the Geneal Model I Statng wth a Jont Myopc Polcy... 3 Fgue 3.3: Pefomance fo Geneal Model II wth Multple Stat Fgue 3.4: Pefomance fo Geneal Model III wth Multple Stat Fgue 3.5: Pefomance fo Geneal Model IV wth Multple Stat Fgue 3.6: Dffeent Infomaton Shang Scheme fo Supply Chan Poblem Fgue 3.7: Solvng Supply Chan Poblem wth Multple Stat and Petubaton Fgue 4.: Infomaton Flow n Each Model Fgue 4.2: Relatve Infomaton Value when Mean Demand = Fgue 4.3: g 3 and g 4 v. C, wth Mean Demand = 5 and Cov = Fgue 4.4: Infomaton Flow n Each 3-Stage Model Fgue 4.5: RIV fo the Ft 32 Poblem n DOE3 (all holdng cot = ) Fgue 4.6: RIV3 Change wth Cm, C and Cov Fgue 4.7: Tanfe cot between upple and etale when CI = Fgue 4.8: Tanfe cot between upple and etale, when CI chaged to the upple Fgue 4.9: Poft Tangle befoe Infomaton Shang... 6 Fgue 4.0: Infomaton Shang Tangle Fgue 4.: Infomaton Shang Tangle Shnk When CI chaged to the Retale. 62 Fgue 4.2: Wothy Cuve of Infomaton Shang fo 96 Poblem Intance n DOE2. 65 v

8 Lt of Table Table 2.: Pecentage of Poblem Solved Optmally (Genec ROMDP)... 8 Table 2.2: Aveage eo of Poblem Not Solved Optmally (Genec ROMDP)... 8 Table 2.3: Max Eo of Poblem Not olved Optmally (Genec ROMDP)... 8 Table 2.4: Aveage Executon Tme (econd) (Genec ROMDP)... 8 Table 3.: Aveage Executon Tme (Second) (Geneal Model I) Table 4.: Degn of Expement I Table 4.2: Degn of Expement II Table 4.3: Degn of Expement III Table 4.4: Summay of Mean Compaon fo RIV Table 4.5: Degn of Expement IV Table 4.6: Wothy Cuve Analy fo 96 Poblem Intance n DOE2 when CI= Table 4.7: Wothy Cuve Analy fo 96 Poblem Intance n DOE2 when CI=2%*Sale v

9 Chapte Ovevew. Intoducton If nfomaton n a upply chan not haed among the ndvdual chan element (e.g., demand nfomaton), actual demand nfomaton (fom downteam to upteam of the upply chan) may be dtoted (th alo temed the bullwhp effect, Lee et al. 997) and caue unneceay cot. It ha been epoted that nfomaton hang benefcal to a upply chan, epecally n educng the bullwhp effect (Lee et al. 997, 2000, Cachon and Fhe 2000) and upply chan cot (Gavnen et al. 999, Swamnathan et al. 997, Tan 999). Howeve, t may not be benefcal to a upply chan f the cot of adoptng the nte-oganzatonal nfomaton ytem too hgh (Swamnathan et al. 997, Cohen 2000). In tem of nfomaton hang, the concen uually whch poducton nfomaton to hae and how to hae t to maxmze the mutual beneft n a upply chan (Huang et al. 2003). The objectve of th detaton to quantfy the value of hang nventoy nfomaton n a make-to-tock envonment and optmze the opeatonal contol fo a two-tage and thee-tage upply chan though appopate nfomaton hang. Th detaton an extenon of Dav (2004) wok on a two-tage upply chan wth a ngle capactated upple and a ngle etale. Dav fnd the upple optmal polcy by aumng the etale ue a fxed polcy, uch a a bae tock polcy o (, S) polcy. Dav wok ha ome lmtaton. Ft, the etale polcy fxed. A moe flexble polcy could pobly acheve bette ytem pefomance. Second, only the value of hang etale nventoy nfomaton examned. Th detaton allow the etale to ue a flexble polcy, and examne the value of hang upple nventoy nfomaton. Howeve, the poblem become much moe complcated nce the upple and etale need coodnaton when makng the eplenhment decon n ode to optmze the upply chan. Fou dffeent nfomaton hang model ae examned n a two-tage upply chan poblem, whle eght dffeent model ae examned n a theetage upply chan. Solvng upply chan model fo the optmal eplenhment polcy a key to quantfyng nfomaton value. Due to the dffculty of detemnng optmal polce fo a

10 mult-echelon nventoy ytem, eeache uually aet that a cetan type of polcy, lke a bae-tock polcy, optmal fo one tage (Gavnen et al. 999, Gavnen 2002, Smch-Lev and Zhao 2002ab, Dav 2004) o both tage (Lee et al. 2000), and then fnd the pecfc polcy fo each tage. Thoe aumpton do not puue ytem-wde optmalty, nce the aeton come fom the eult of a ngle-tage nventoy ytem, and the poblty of coodnaton between membe gnoed. To ovecome th dawback, we model a mult-tage upply chan a a Makov Decon Poce (MDP). In the context of a MDP, an agent wth full obevaton (due to nfomaton hang) actually face a common MDP poblem (alo called a completely obevable MDP, COMDP), whle an agent wth etcted obevaton (lack of nfomaton hang) face a MDP wth etcted obevaton (called ROMDP). Dav (2004) olve a ngle agent ROMDP. A an extenon, th detaton povde a oluton fo multagent (decentalzed) MDP o ROMDP poblem (called DEC-ROMDP), whee upply chan membe need to be coodnated n ode to maxmze poft. Th detaton popoe and analyze an nfnte hozon ROMDP wth an aveage cot cteon, wth an objectve to maxmze the aveage ewad. A computatonally effcent algothm developed to fnd optmal polce baed on the polcy teaton method of Howad (960) fo the nfnte hozon undcounted cot cae. Fomally, a ROMDP can be epeented a a mxed ntege nonlnea pogammng (MINLP) poblem, fo whch t dffcult to fnd the global optmal oluton. The bac heutc popoed hee nclude two tep: value detemnaton and polcy mpovement. It poven that the polcy mpovement eache fo an optmal oluton by followng a teepet acent decton. We alo popoe petubaton method, uch a polcy petubaton and Π petubaton ( Π the ytem teady tate pobablty vecto), to mpove local optma towad a global optmal polcy. Succeve appoxmaton ued to educe computatonal effot. In addton, Dng encapulaton evoluton method (985) can be ued to futhe educe computatonal effot fo pecally tuctued upply chan poblem (Dav 2004). A mult-agent model vewed a a decentalzed Makov decon poce wth etcted obevaton (DEC-ROMDP), whch can be vewed a a pecal cae of a decentalzed POMDP (DEC-POMDP) (Benten et al., 2000) and a mult-agent team 2

11 decon poblem (MTDP) (Pynadath and Tambe, 2002). An evolutonay coodnaton algothm ued to make a jont polcy evolve to a locally optmal oluton, and then petubaton method and a multple etat tategy ae ued to mpove the polcy. By ung the tool fo olvng DEC-ROMDP model, a wde ange of upply chan poblem wth dffeent nfomaton hang cheme ae olved. Chapte 2 popoe the mathematcal model fo an nfnte hozon ROMDP wth an aveage cot cteon and ntoduce heutc algothm fo oluton. Chapte 3 gve the defnton of DEC-ROMDP and popoe an evolutonay coodnaton algothm to olve the mult-agent decon poblem. Chapte 4 apple the evolutonay coodnaton algothm to two-tage and thee tage upply chan poblem, and elaboate on nfomaton hang and tanfe cot negotaton wthn the upply chan. Chapte 5 outlne the futue eeach to be pefomed..2 Refeence Benten, D., S. Zlbeten, and N. Immeman, The complexty of decentalzed contol of MDP. In Poceedng of the Sxteenth Confeence on Uncetanty n Atfcal Intellgence. Cachon, G.P., and M. Fhe Supply chan nventoy management and the value of haed nfomaton. Management Scence, 45, Cohen, S.L Aymmetc nfomaton n vendo managed nventoy ytem. PhD The, Stanfod Unvety. Dav, L.B., 2004, State Cluteng n Makov Decon Pocee wth an Applcaton n Infomaton Shang, unpublhed Ph.D. detaton, Indutal Engneeng Depatment, N.C. State Unvety. Dng, F.Y., T. Hodgon and R. Kng, 988. A methodology fo computaton educton fo pecally tuctued lage cale Makov decon poblem. Euopean Jounal of Opeatonal Reeach. Volume 34:05-2. Gavnen, S., R. Kapucn, and S. Tayu Value of nfomaton n capactated upply chan. Management Scence, 46, no.: Gavnen, S Infomaton flow n capactated upply chan wth fxed odeng cot. Management Scence, 48, Howad, R Dynamc Pogammng and Makov Pocee. MIT Pe, Cambdge, MA Huang, G.Q., J.S.K. Lau, and K.L. Mak The mpact of hang poducton nfomaton on upply chan dynamc: a evew of the lteatue. Intenatonal Jounal of Poducton Reeach. 4, no.7:

12 Lee, H., Padmanabhan, V. and Whang, S Infomaton dtoton n a upply chan: the bullwhp effect. Management Scence, 43, Lee, H.L., K.C. So, and C.S. Tang The value of nfomaton hang n a two-level upply chan. Management Scence, 46, Pynadath, D., and M. Tambe, The communcatve multagent team decon poblem: Analyzng teamwok theoe and model. Jounal of Atfcal Intellgence Reeach, 6, Smch-Lev, D., and Y. Zhao. 2002a. The value of nfomaton hang n a two-tage upply chan wth poducton capacty contant. Wokng pape, Nothweten Unvety, Evanton, IL b. The value of nfomaton hang n a two-tage upply chan wth poducton capacty contant: the nfnte hozon cae. Manufactung and Sevce Opeaton Management, 4, no.: Swamnathan, J. M., N.M. Sadeh. and S.F. Smth Effect of hang upple capacty nfomaton. Haa School of Bune, Unvety of Calfona, Bekeley. 4

13 Chapte 2 Makov Decon Pocee wth Retcted Obevaton 2. Intoducton Th chapte peent a computatonally effcent pocedue to detemne contol polce fo an nfnte hozon, undcounted Makov decon poce (MDP) wth etcted obevaton (ROMDP). In the MDP famewok, t uually aumed that an agent nteact ynchonouly wth a wold (Kaelblng, Lttman, and Caanda 998). A Makov decon poce can be defned a a tuple < S, A, T, R >. S a fnte et of S wold tate; A a et of A acton; T: S A S [0, ] the tate-tanton model, whee T(, a, ) epeent the pobablty of tanton fom tate to, gven that the agent take acton a; R: S A R the ewad model, whee R(, a) epeent the expected ewad fo takng acton a n tate (aumed to be bounded n th chapte). In a common MDP, the wold tate aumed a completely obevable to the agent, o th poce alo called COMDP (completely obevable Makov decon poce) n th detaton. Snce th poce condeed obevable and the tate of the ytem obevable to the agent, the tatonay polcy a functon of the tate pace. If the wold tate not completely obevable to the agent, th poce a patally obevable Makov decon poce (POMDP), whch can be defned a a tuple <S, A, T, R, Z, O>, whee S, A, T, and R ae the ame a thoe n a COMDP. Z a fnte et of Z obevaton; O: S A Z [0, ] the obevaton pobablty dtbuton model, whee O (z, a, ) epeent the pobablty that the agent obeve z gven that t took acton a and then the wold tate eached. A the agent cannot obeve the tate dectly, a POMDP polcy, dffeent fom a COMDP polcy, not a functon of the tate pace, but the functon of belef tate,.e., the teady tate pobablty dtbuton. Snce the belef tate contnuou, t not ealtc to fnd a polcy baed upon evey poble belef tate. Howeve, an optmal polcy can be baed only upon fnte patton of belef Th detaton attempt to fnd the optmal tatonay detemntc polcy fo an nfnte hozon undcounted ROMDP poblem. A ROMDP polcy pace a ubet of a common MDP polcy pace. A common MDP polcy can be categozed a detemntc o andomzed, Makovan o htoy-dependent. A tatonay polcy geneally fo an nfnte hozon MDP poblem. Refe to Puteman (994) fo detal of thee polcy type. 5

14 tate pace (Smallwood and Sondk 973) 2. Seveal algothm have been developed to effcently detemne the patton (Sondk 97, Cheng 988, Lttman 994, and Zhang and Lu 996). A Makov decon poce wth etcted obevaton a pecal POMDP, and t can be epeented by a tuple < S, A, T, R, Z, G >, whee S, A, T, R, and Z ae the ame a thoe n a POMDP. G: S Z epeent the mappng functon fom a tate to a ngle obevaton fo the agent. If a tate output an obevaton z, t can be denoted a G() =z. A ROMDP polcy epeented a a functon of the obevaton pace. In th polcy, f an acton a appled gven an obevaton z, th acton a would apply to any poble tate atfyng G() = z, that, the acton a mut be mplementable/admble to all thee tate. Hence, a ROMDP polcy alo called an mplementable polcy (Sen and Kuln 995) o admble polcy (Smth 97). Although an ROMDP a pecal POMDP, t tll ntactable to olve. Sen and Kuln (995) develop an algothm that fnd local optmal tatonay andomzed polce fo the nfnte hozon dcounted ewad cae, wth the objectve to optmze the total dcounted ewad. Sen and Ava (997) ntoduce a mla algothm fo the fnte hozon dcounted ewad cae, and pove a detemntc optmal polcy ext n th cae. Smth (97), Hodjk and Loeve (994), and Hatng and Sadjad (979) peent algothm that detemne detemntc polce fo nfnte hozon undcounted ewad poblem, wth the objectve to optmze the aveage ewad. The algothm developed by Hatng and Sadjad (979) enumeatve baed and thu ntactable fo lage poblem. The algothm developed by Smth (97) a polcy teaton type of algothm contanng enumeatve component when a bette polcy cannot be detemned. None of the above algothm have addeed to an nfnte hozon lage cale ROMDP poblem. Th chapte ntoduce a computatonally effcent algothm that alo fnd optmal detemntc polce, baed on the polcy teaton method developed by Howad (960) fo the nfnte hozon undcounted cot cae. Th chapte demontate empcally that the algothm fnd the optmal detemntc polcy fo ove 99% of the geneal ROMDP poblem ntance geneated. In the ntance whee the optmal polcy cannot be detemned, the aveage eo n the objectve functon 2 Th apple to a fnte hozon POMDP poblem. It may not be the cae fo an nfnte hozon poblem. 6

15 le than %. Th algothm acheve bette pefomance fo upply chan ROMDP poblem. 2.2 Mathematcal Model fo a ROMDP 2.2. ROMDP Notaton The poce beng analyzed a Makov Decon Poce wth tate pace S and acton pace A. The tate of the ytem cannot be obeved, howeve ome output of the ytem obevable. Baed on thoe output, one can nfe the tate o poble tate of the ytem. Th chapte fnd an optmal contol polcy defned ove the obevaton poce that maxmze the long tem aveage ewad. The optmal polcy ha the popety that each tate wthn a gven obevaton et take the ame acton. A ummay of the poblem notaton peented below. S: The et of poble tate { N}. A: The et of avalable acton { M}. X n : A andom vaable that defne the tate at tme n. Y n : A andom vaable that defne the acton by the agent at tme n. a p j : The one tep tanton pobablty fom tate to j gven an acton a. p = P{ X = j X =, Y a},, j S, a A, a j n n n = c a : The expected mmedate ewad aocated wth tantonng to tate gven acton a. Z: The et of obevable output {...K}. G(): A functon mappng a tate to a ngle obevable output n the et Z. S k : A patton of the tate pace S atfyng {: G() = k}. Wthout lo of genealty, t aumed any tate pace patton ha the ame numbe (ay L) of tate. Obvouly, K*L = N, and S k ={(k-)*l, (k-)*l2 k*l}, k Z. A(k): The et of admble acton fo the obevaton et S k. Obvouly, A(k) A. Wthout lo of genealty, t aumed A(k) = A fo a genec ROMDP. Z n : A andom vaable that defne the obevaton by the agent at tme n. 7

16 2.2.2 LP Model fo a MDP Poblem Befoe peentng the mathematcal model fo a ROMDP wth undcounted ewad and nfnte hozon, th ecton tat fom the lnea pogammng model (LP) fo an undelyng common MDP poblem (Wolf and Dantzg, 962). max M a= M x a x N a a= = M a= = = x a N ubject to M c a N x a= j= = a x ja p a j 0 S, a A, S Th pmal poblem am to maxmze the aveage ewad. It decon vaable x a can be ntepeted a the teady tate pobablty that tate wll be vted at a typcal tanton and acton a wll be appled. The contant can be atfed wth ome feable teady tate pobablte aocated wth a cetan andomzed tatonay polcy, that, a polcy that chooe at tate the acton a wth pobablty x a. LP theoy mple that the optmal oluton alway obtaned wth detemntc tatonay polce. Indeed, f α* an optmal (detemntc) tatonay polcy that un-chan ( α *( ) denote the acton fo the tate ), and x * the coepondng teady * * x f a = α * ( ) tate pobablty of tate, then xa = an optmal oluton of the 0 othewe. pmal poblem. It alo nghtful to nvetgate the followng dual fomulaton fo the poblem. mn g ubject to g v p v j N j= a j v fee, j S j c a S, a A Howad (960) dynamc pogammng algothm olve a COMDP poblem fom a dual pepectve. Th chapte olve the aocated ROMDP fom a pmal pepectve. 8

17 2.2.3 NLP Model fo a ROMDP Poblem By addng obevablty contant to the above pmal MDP poblem, the nonlnea pogammng (NLP) model fo a coepondng ROMDP poblem obtaned. Befoe fomulatng the NLP model, the followng defnton ae ntoduced. α = ( α, α2,..., αm α 2,..., α 2M... α K,..., α KM ) a ROMDP polcy, n whch α denote the pobablty that acton a appled gven obevaton k. Hee xa xa α = =, S k, a A. M x x x M a= = x a= a a, S. The NLP model fo the ROMDP gven a max x M a= x N = = x N j= a= N M = a= ubject to α M = c a x α =, k Z 0 S j x α G( ) a G( j) a p a j, S Let P(α) be the matx defned wth ente p j (α), whee ( ) = M p j α α p, a= G( ) a a j and M c ( α) = α c. a a= The NLP can be wtten n matx notaton a 9

18 max Φ ubject to x x ( α ) = xc( α ) [ I P( α )] N = M a= x α = = 0 =, k Z 0, S The matx [I-P(α)] not nvetble nce t contan a edundant contant. To educe th edundant contant, eplace the N th column of th matx wth all one and defne an nvetble matx Q(α). p( α) p2 ( α) p2( α) p22 ( α) Q ( α) = p N ( α) p N 2 ( α) Futhemoe, defne an n-element vecto b =(0,0 ). Then tanfom the NLP nto max Φ ubject to xq( α) = b M a= x α ( α ) = xc( α ) =, k Z 0 S By emovng the vaable x, t become max Φ ubject to M a= α ( α ) = bq( α) c( α ) =, k Z Note the optmal oluton to th NLP poblem may be a andomzed tatonay polcy α. That, α, a component of an optmal polcy may be a numbe othe than 0 and. It ntutve that an optmal andomzed polcy wll be bette than an optmal detemntc polcy. A only the optmal detemntc polcy n concen, the NLP 0

19 tanfomed nto a mxed ntege nonlnea pogammng poblem (MINLP) by addng the ntege contant. max Φ( α) = bq( α) ubject to M a= α =, k Z c ( α ) α {0,}, k Z, a A 2.3 Methodology fo Solvng a ROMDP Defnton : β = β, β,..., β β,..., β... β,..., β ) a feable decton at ( 2 M 2 2M K KM a feable polcy α f and only f αθβ alo a feable polcy fo ome θ>0. Clealy, M a= β = 0, k Z and β 0 fo α = 0 and β 0 fo α =. Wthout lo of genealty, the nomalzaton etcton β = aumed on the feable K M k= a= decton (Sen and Kuln 995). Defnton 2: A feable decton β an acent decton at a feable polcy α f Φ ( α θβ ) > Φ( α), fo all θ (0,δ) fo ome δ>0. Lemma : If a feable decton β atfe that Φ( α) T β potve, then β an acent decton at polcy α. Hee, Φ(α ) the gadent of the objectve functon at α. Poof: It Obvou accodng to the defnton of the gadent. (Q.E.D) Theoem : Let v = v... v ] be the oluton to Q(α)v= c(α), x = x... x ] be the oluton [ n [ n to x Q(α ) = b, and P the matx wth p j a pj = 0 f G( ) = k, and j N. If Othewe a * k = ag max( a A x ( c a j= S k N p j v )), then β atfyng β j = / 2, f / 2, f * α = 0,and a = ak α =, and a a 0, othewe a teepet acent decton at the cuent polcy α, whch maxmze the dectonal * k devatve Φ( α) T β. Poof:

20 Ft deve the gadent of the objectve functon at α,.e. Φ(α ). To compute the gadent, t aumed that the objectve functon contnuou and dffeental at evey pont of feable egon,.e., α can be andomzed. Φ α c α ( α ) c ( α ) ( α ) Φ So α Let x α = = ( α ) x α α = α S k c x = α a a x x α c ( α ) ca f G( ) = k caα = 0 othewe x x c ( α ) α x2, α Q( α) Q( α) x = 0, α x N... α Defne P Q( α) =. Clealy, ( α ) P the matx wth p a pj = 0 f G( ) = k, and Othewe j N j. Then x α Theefoe, Q(α) = xp Φ α ( α ) = S k. Snce Q( α) ext, c a x xp Q( α) c( α) x α = xp Q( α). Let v = v... v ] be the oluton to Q(α)v= c(α), Then ( α ) [ n Φ = x ca pj (α ) v j. α S k j In ode to fnd the teepet acent decton at the cuent polcy α, Φ( α) T hould be maxmzed. Note the cuent polcy α mut be detemntc, fo ntance, f α, f a = b ue acton b fo a tate et S k, thee mut beα =. It obvou fo a tate 0, othewe et S k, β,..., β ) mut have a ngle negatve component, ay β kb. Note ( k km β 2

21 β 0 fo α = 0, β 0 fo α =, and β = 0, k Z. In ode to maxmze Φ( α) T β, t obvou to chooe a component β * among β,..., β ), M a= ( k km * Φ( α) whee a = ag max( ), and et β * to be -β kb, othe component zeo. a A α K Condeng the nomalzaton etcton β =, o β * = /2, and β kb = -/2. If M k= a= a * = b, evey component of β,..., β ) zeo. (Q.E.D) ( k km If a teepet acent decton β at a cuent polcy α zeo, the polcy α condeed a a local optmal polcy. Othewe, f β not zeo, thee ext a tep ze θ>0 uch that αθβ (t mght be a andomzed polcy) bette thanα,.e. Φ(αθβ)>Φ(α). Snce α detemntc and the teepet acent decton β defned a Theoem, t neceay to make θ = 2 uch that α =αθβ alo detemntc. Note a tep ze θ = 2 may not atfy Φ(αθβ)>Φ(α). Howeve, t uffcent to keep th tep ze and enue polcy move between detemntc polce. Th movement ha a dawback that may not keep Φ(αθβ)>Φ(α), but f that a cae, the cuent polcy α alo condeed a a local optmal polcy. Baed on th gudance, a heutc algothm peented below. Defnton 3: If tatng fom a cuent polcy α, β the teepet acent decton found accodng to Theoem, then the opeaton of gettng a new polcy α =α2β called a polcy mpovement. Defnton 4: A polcy α a local optmal oluton, f afte a polcy mpovement the polcy change fom α to α, and Φ ( α') Φ( α). Lemma 2: Let v = v... v ] be the oluton to Q(α)v= c(α), then g = v N the gan [ N aocated wth the polcy α. Poof: Fo a polcy α, the gan g = xc(α ), whee x the teady tate pobablty. Snce x = bq( α), the gan g = b Q( α ) c( α) = bv. Hence g = v N a b = [0,0,,]. (Q.E.D) 3

22 A detemntc polce ae of nteet, t uffcent to only cay the nfomaton needed n tem of the acton taken fo a gven obevaton et S k, and th dpel the necety to contuct a K*M-element decon vecto α to epeent an mplementable polcy. Theefoe, a K*M-element ognal decon vecto α can be epeented by K-element polcy vecto δ = [ δ... δ K ], whee δ k = a f α =. Then Q(α) ha ente p j (α) whee p j a δg ( ) ( α ) α p = p = a G( ), a Smlaly, the vecto c(α) ha ente c (α) whee ( α ) = c α = c a G( ), a c, δg ( ) a j j Thee ubttuton wll be denoted a Q(α/δ) and c(α/δ). To mply notaton, a polcy alway condeed a K-element vecto notaton n th chapte, but eade hould keep n mnd that the polcy can have two epeentaton. Ue α k to epeent the acton ued fo obevaton et k. Heutc Algothm: The algothm fo fndng an mplementable polcy a below. Step 0. Intalzaton Geneate an ntal admble polcy α. Set g* = -. Step. Value Detemnaton Detemne elatve value v, teady tate pobablte x, and the gan g δ. x = bq(α) -, v = Q(α) - c(α), (a). If α * g > g, et g * = g and poceed to Step 2. (b) If g α * g, the cuent oluton a local maxmum, and top. Step 2. Polcy Impovement Fo all k Z fnd an actonα k = ag max a A x ( ca S k N j= p j v j ), and go to tep. Lemma 3: The algothm defned above wll temnate at a local optmal oluton afte a fnte numbe of teaton. 4

23 Poof: By aumng the ewad fo the Makov decon poce bounded, the gan aocated wth any polcy would be bounded. Snce the polcy mpovement tep wll not nceae the gan ndefntely, thee mut be a fnte numbe of teaton uch that the algothm temnate. (Q.E.D) 2.4 Petubaton The above heutc, called Nomal Convegence, doe not guaantee to obtan the global optmum unle only a ngle local optmum ext. Theefoe, the nomal convegence augmented by a local mpovement pocedue, called petubaton, to nceae the pobablty of fndng the global optmum. In ode to mpove the heutc, th chapte alo povde two petubaton method. One called Polcy Petubaton, and the othe called Π Petubaton Polcy Petubaton Polcy petubaton caed out baed on the polcy fom Nomal Convegence. The bac dea to petub the bet polcy obtaned fom nomal convegence, and fom a neghbong polcy. By tatng fom th new polcy, epeat value detemnaton and polcy mpovement cycle. Once a bette polcy found, contnue petubng th polcy untl no bette polcy can be found. Obvouly, how a polcy petubed and how many petubaton ae pefomed mpact the effect of polcy petubaton. Two appoache fo polcy petubaton ae developed. The ft one (denoted a PP) modfe only one element n the polcy vecto, and the numbe of the petubaton nceae popotonally to the length of the polcy vecto. The econd one (denoted a PP2) an extenon of the ft one. Afte modfyng one element n the polcy vecto, t te to modfy any adjacent two element n the polcy vecto. Obvouly, PP2 ha moe petubaton than PP. How to modfy the element to obtan a new polcy? Dung the polcy mpovement tep, the tet quantte fo dffeent altenatve ae computed, and the bet altenatve detemned, whch maxmze the tet quantty, ay, S k N x ( c p v ). Actually, the econd bet altenatve can eve a the canddate a j= j j fo petubaton. The example below demontate the appoach. 5

24 The ognal polcy α = α, α,..., α α,..., α,... α,..., α ) not ( 2 M 2 2M K KM convenent to epeent. Accodng to the chaactetc of detemntc polcy α, th chapte mplfe the epeentaton of th polcy wth K-element polcy vecto, of whch each element coepond to the elected acton ndex fo one of the K obevaton. Fo ntance, fo a ROMDP poblem wth N = 6, M = 4, K = 4, and L = 4,, f a bet polcy (4, 3, 3, 2) afte Nomal Convegence, the ognal epeentaton of polcy actually α=(0,0,0, 0,0,,0 0,0,,0 0,,0,0). Aume the econd bet altenatve ae (3, 4,, 3). Polcy Petubaton I (denoted a PP) eult n the followng polce afte petubaton: (3,3,3,2), (4,4,3,2), (4,3,,2), and (4,3,3,3). The Polcy Petubaton II (denoted a PP2) eult n the followng polce afte petubaton: (3,3,3,2), (4,4,3,2), (4,3,,2), (4,3,3,3), (3,4,3,2), (4,4,,2), (4,3,,3), and (3,3,3,3). Notce that the ft element and the lat element n a vecto ae teated a adjacent P Petubaton P Petubaton ( Π Petubaton) mla to the polcy petubaton, except that t petub a teady tate pobablty vecto x ntead of a polcy vecto. Although the polcy not modfed, modfcaton of vecto x may lead to a bette polcy by epeatng the value detemnaton and polcy mpovement cycle. If a bette polcy found, agan petub the vecto x aocated wth th polcy. The poce contnued untl no bette polcy can be found. Unlke a polcy vecto, vecto x ha a contnuou pace. Dffeent fom polcy petubaton, Π petubaton pefomed by andomzng the x vecto unde the expectaton that th modfed vecto x wll eventually lead to a bette polcy dung the value detemnaton and polcy mpovement cycle. Two type of Π petubaton ae developed (et ε = /N). () Π Petubaton I (denoted a PP) ) x = x ε ) x = N x = x (2) Π Petubaton II (denoted a PP2) 6

25 ) x max(0, x = x ε ε ) f 0.5,, f > 0.5, whee andomly geneated value between 0 and. ) x = N x = x 2.5 Expementaton 2.5. Genec Poblem To evaluate the effectvene of an algothm, dffeent ze of genec ROMDP poblem ae geneated, and each poblem ha 000 andom ntance. The heutc oluton of olvng thee ntance wa compaed wth the optmal oluton though bute foce enumeaton. Let K epeent the numbe of patton, L the numbe of tate n each patton, N=K*L the numbe of total tate n the ytem, and M the ze of acton pace. Table 2. gve the pecentage of poblem olved optmally. By applyng Nomal Convegence (NC), Polcy Petubaton I and II (PP and PP2), and Π Petubaton I and II (PP and PP2), 88.3%, 98.5%, 99.2%, 98.8%, and 99.% of,000 poblem ntance ae optmally olved, epectvely. By combnng PP and PP2, 99.7% of,000 poblem ntance wee olved optmally. Table 2.2 gve the aveage eo fom the optmal oluton fo thoe poblem that ae not olved optmally. Table 2.3 gve the maxmum eo fom the optmal oluton fo thoe poblem that ae not olved optmally. Table 2.4 gve the aveage executon tme (econd) fo each poblem. The eult how that NC can optmally olve at leat 85% of genec ROMDP poblem. Wth polcy petubaton o Π petubaton, moe than 96% of thee poblem ae olved optmally. Among thoe poblem not olved optmally, the aveage eo ae le than 2% and the maxmum eo ae le than 0%; wth petubaton, the eo ae much malle. A the ze of the acton pace nceae, the polcy pace nceae exponentally and t pohbtve to ue bute foce enumeaton to obtan the optmal oluton fo lage poblem. Ou algothm appea effectve and fat to the genec ROMDP poblem. 7

26 Table 2.: Pecentage of Poblem Solved Optmally (Genec ROMDP) % of Poblem Solved Optmally Among 000 Intance KXL, N, M Polcy pace NC PP PP2 PP PP2 PP&PP2 3X3, 9, = % 97.40% 98.30% 96.20% 98.0% 98.90% 4X4, 6, = % 98.30% 98.90% 97.70% 98.40% 99.50% 5X5, 25, = % 98.50% 99.20% 98.80% 99.0% 99.70% 6X6, 36, = % 99.6% 99.7% 98.7% 98.4% 99.6% Table 2.2: Aveage eo of Poblem Not Solved Optmally (Genec ROMDP) Aveage Eo of Poblem Not Solved Optmally KXL, N, M NC PP PP2 PP PP2 PP&PP2 3X3, 9, 3.54%.33%.8%.39% 0.79% 0.68% 4X4, 6, % 0.43% 0.45% 0.57% 0.40% 0.55% 5X5, 25, % 0.9% 0.25% 0.6% 0.2% 0.28% 6X6, 36, % 0.09% 0.09% 0.09% 0.09% 0.09% Table 2.3: Max Eo of Poblem Not olved Optmally (Genec ROMDP) Max Eo of Poblem Not Solved Optmally KXL, N, M NC PP PP2 PP PP2 PP&PP2 3X3, 9, % 5.20% 4.79% 7.23% 2.29%.8% 4X4, 6, %.44%.44%.86%.03%.44% 5X5, 25, 5.65% 0.64% 0.64% 0.77% 0.70% 0.64% 6X6, 36, % 0.6% 0.6% 0.23% 0.23% 0.6% Table 2.4: Aveage Executon Tme (econd) (Genec ROMDP) Aveage Executon Tme (econd 3 ) fo 000 Poblem Intance KXL, N, M NC PP PP2 PP PP2 PP&PP2 Enumeaton 3X3, 9, X4, 6, X5, 25, X6, 36, Supply Chan Poblem The ROMDP algothm alo appled to a two-tage upply chan ROMDP poblem (maxmzaton poblem), n whch the etale ue a fxed ode-up-to polcy, and the upple am to optmze the ytem wthout knowng the etale nventoy nfomaton. 3 The expement wee pefomed on a compute wth Intel Pentum 2.2GHz CPU. 8

27 The aumpton nclude: Thee a cutome demand dtbuton that etale mut atfy. The upple poducton and the etale ode hpment ae ynchonou, and the lead-tme a typcal peod. Each ha a maxmum nventoy capacty. The upple poducton capacty lmted by t nventoy capacty, nce t cannot poduce moe than can be accommodated n h waehoue. The etale apple ode-up-to polcy, and the ode-up-to level t nventoy capacty. Note that the exce demand fom a cutome o the etale lot. The cot tuctue nclude poducton/ode etup cot ( F and F ), holdng cot ( H and H ), vaable poducton/puchae cot ( W and W ), and a tock out penalty cot ( L and L ). Hee the ubcpton of tand fo the upple and fo the etale. The typcal paamete fo the upply chan ae a follow. C : The nventoy capacty fo the upple. C : The nventoy capacty fo the etale. V: The ellng pce to the cutome. d : The demand fom the cutome, d = 0, D, aumng : The nventoy level of the upple, D = C. = 0,,2,..., C. The upple obevaton on h own nventoy z =. : The nventoy level of the upple, on he own nventoy z =. = 0,,2,..., C. The etale obevaton k : The poducton ode quantty placed by the upple. The poble ode quantty depend on the upple nventoy capacty and cuent nventoy level,.e., k = 0,,2,..., C. The objectve to fnd the optmal polcy fo the upple, who only obeve h own nventoy, uch that the upply chan total poft maxmze. Obvouly, th a typcal ROMDP poblem. The ytem tate can be epeented by the nventoe of both the upple and the etale,.e., = ( ) C, and the acton can be epeented by the ode quantty of the upple,.e., k. Snce the upple ha the capacty etcton, the polcy pace not a lage a the genec ROMDP wth the ame acton pace. Suppoe the cuent tate (the upple and the etale nventoe ae 9

28 coepondngly and epectvely), unde an acton k and a cutome demand d, then the total poft of th upply chan would be: P(, k, d ) = V * mn( d, mn(, mn( ) [ H k, * )) * F H L * W * ( d ) * k ] mn(, k ) * F dffeent Snce C and C and C detemne the poblem ze, 000 poblem ntance fo C ae geneated. It appea that the pefomance bette than genec poblem (ee Fgue 2., Fgue 2.2, and Fgue 2.3). Note that C = K and C = L. Wthout any petubaton, NC method ha acheved moe than 93% of poblem olved optmally. Wth petubaton, almot olve all the poblem ae olved; even fo thoe poblem that ae not olved optmally, the aveage eo ae cloe to zeo % Facton Optmal Found 98.00% 96.00% 94.00% 92.00% 90.00% 88.00% 3X3 4X4 5X5 6X6 7X7 8X8 NC PP PP2 PP PP2 PP&PP2 Fgue 2.: the pecentage of poblem olved optmally (Supply Chan ROMDP) Aveage Relatve Eo 2.00% 0.00% 8.00% 6.00% 4.00% 2.00% 0.00% NC PP PP2 PP PP2 PP&PP2 3X3 4X4 5X5 6X6 7X7 8X8 Fgue 2.2: The Aveage eo of Poblem Unolved Optmally (Supply Chan ROMDP) 20

29 Maxmum Relatve Eo 8.00% 6.00% 4.00% 2.00% 0.00% 8.00% 6.00% 4.00% 2.00% 0.00% NC PP PP2 PP PP2 PP&PP2 3X3 4X4 5X5 6X6 7X7 8X8 Fgue 2.3: Max Eo of Poblem Unolved Optmally (Supply Chan ROMDP) Table 2.5: Aveage Executon Tme (econd) (Supply Chan ROMDP) Aveage Executon Tme (econd) fo 000 Poblem Intance KXL, N, M NC PP PP2 PP PP2 PP&PP2 Enumeaton 3X3, 9, 3 8e e e-005 4X4, 6, X5, 25, X6, 36, X7, 49, X8, 64, Concluon Expemental eult demontate that the heutc appoach to olvng ROMDP poblem vey effectve and effcent. Fo pactcal upply chan poblem, t ha bette pefomance. The heutc appoach can be ued fo olvng lage-cale ROMDP poblem (Dav 2004). 2.7 Refeence Cheng, Hen-Te Algothm fo Patally obevable Makov Decon Pocee. PhD the, Unvety of Bth Comuba, Bth Coumba, Canada. Dav, L.B., 2004, State Cluteng n Makov Decon Pocee wth an Applcaton n Infomaton Shang, unpublhed Ph.D. detaton, Indutal Engneeng Depatment, N.C. State Unvety. Hatng, N., and D. Sadjad Shot communcaton: Makov pogammng wth polcy contant. Euopean Jounal of Opeatonal Reeach, 3:

30 Hodjk, A., and J. Loeve Undcounted Makov decon chan wth patal nfomaton: an algothm fo computng a locally optmal peodc polcy. Mathematcal Method of Opeaton Reeach, 40:63-8. Howad, R Dynamc Pogammng and Makov Pocee. MIT Pe, Cambdge, MA Kaelblng, L.P., M.L. Lttman, and A.R. Caanda Plannng and actng n patally obevable tochatc doman. Techncal Repot CS-96-08, Bown Unvety, Povdence, RI. Lttmann, M The wtne algothm: Solvng patally obevable Makov decon pocee. Techncal epot CS-94-40, Depatment of Compute Scence, Bown Unvety. Sen, Y., and Z. Ava Makov decon pocee wth etcted obevaton: fnte hozon cae. Naval Reeach Logtc, 44: Sen, Y., and V.G. Kuln Implementable polce: dcounted cot cae n W.J. Stewad (Ed.), Computaton wth Makov Chan. Kluwe Academc Publhe, Dodecht. Smth, J.L. 97. Makov decon on a pattoned tate pace. IEEE tanacton on ytem, man and cybenetc SMC-, no.: Sondk, E.J. 97. The Optmal Contol of Patally Obevable Makov Pocee. PhD the, Stanfod Unvety, Stanfod, Calfona. Wolfe, P., and G.B. Dantzg Lnea pogammng n a Makov chan. Opeaton Reeach 0: Zhang, N.L., W. Lu Plannng n tochatc doman: Poblem chaactetc and appoxmaton. Techncal epot HKUST-CS

31 Chapte 3 Decentalzed Makov Decon Pocee wth Retcted Obevaton 3. Intoducton Th chapte peent a computatonally effcent algothm to olve a dtbuted mult-agent decon poce poblem. It aumed that a goup of agent ae fully coopeatve, and that the objectve to deve optmal jont polce fo the agent that maxmze the jont ewad ove an nfnte hozon. Geneally, a Makov Decon Poce o MDP (Howad, 960) can be ued to model a ngle agent decon poblem whee the agent ha full obevablty of the poce. Wthn a mult-agent famewok, the global tate may not be obevable by evey agent. It aumed that agent ae only able to obeve the local tate whch ae the obevable patton of the global tate pace. Due to the patal obevablty, each agent face a Retcted Obevable Makov Decon Poce o ROMDP (Chapte 2). It ntuctonal to note that a ROMDP a pecal cae of a patally obevable Makov decon poce o POMDP (Sondk, 97). In a POMDP, fo each global tate thee a pobablty dtbuton aocated wth the eultng obevaton wheea n a ROMDP thee a ngle obevaton aocated wth each global tate (although multple global tate may yeld the ame obevaton). Thu, the mult-agent poblem can be vewed a a Decentalzed ROMDP (DEC-ROMDP). A DEC-ROMDP can be vewed a a pecal cae of a decentalzed POMDP (DEC-POMDP) (Benten et al., 2000) and a multagent team decon poblem (MTDP) (Pynadath and Tambe, 2002). Note that wthn a DEC-ROMDP famewok, f evey agent ha full obevablty of the global tate, the DEC-ROMDP degeneate nto a Mult-agent MDP (MMDP) (Boutle, 999) o a Decentalzed MDP (DEC-MDP) (Benten et al., 2000), whee evey agent a MDP decon make that collectvely act to acheve a common objectve. Solvng a decentalzed Makov decon poblem extemely dffcult. The computatonal complexty of a DEC-POMDP wth at leat two agent o a DEC-MDP wth at leat thee agent complete fo the complexty cla nondetemntc exponental tme (Benten et al., 2000). One appoach to ccumventng th complexty bae to explot the tuctue of decentalzed poblem. Fo example, Becke et al. 23

32 (2002) peent a coveage et algothm to olve a geneal cla of decentalzed MDP that exhbt tanton ndependence wthout ewad ndependence. Anothe appoach to mplfy the natue of decentalzed decon poblem. Fo example, Chade et al. (2002) convet a DEC-POMDP nto a MMDP (Boutle, 999) by appoxmatng the ewad functon and tanton functon ove obevaton ntead of ove tate. Howeve, the conveon fom olvng a DEC-POMDP to olvng a MMDP can be qute complex and the oluton to the MMDP appoxmate to the DEC-POMDP nce t gnoe the nontatonay popety of the tanton and ewad functon ove obevaton. Reeache have been explotng algothm wthn the famewok of fnte hozon DEC-POMDP and DEC-MDP (fo example, Becke et al., 2002, Na et al. 2003, Chade et al. 2003, Xuan et al., 200). Chapte 2 peent an effectve appoach fo olvng ngle agent ROMDP poblem. Howeve, a DEC-ROMDP cannot be teated a epaate ROMDP becaue the tanton and ewad functon geneally depend on the jont polcy, athe than a ngle agent polcy. To the bet of the autho' knowledge thee no effcent algothm fo DEC-ROMDP n the lteatue. Th chapte peent an evolutonay coodnaton mechanm to evolve a jont polcy to a locally optmal polcy fo nfnte hozon DEC-ROMDP. In the coodnaton mechanm, each agent teatvely update the local polcy whle keepng the othe agent polce fxed. Each update attempt to nceae the jont ewad untl no mpovement can be made. Smla coodnaton mechanm ae tuded by Na et al. (2003) and Chade et al. (2002) fo fnte hozon DEC-POMDP. Fo example, Na et al. (2003) peent a mla coodnaton mechanm called JESP (jont equlbum-baed each fo polcy) whch ue ethe exhautve each o dynamc pogammng to fnd the bet polcy fo each agent. Ou expementaton ndcate that the evolutonay coodnaton algothm, coupled wth a multple tat tategy and polcy petubaton, effectvely olve geneal DEC-ROMDP. Addtonal expementaton how that fo pecally tuctued upply chan poblem modeled a nfnte hozon DEC-ROMDP, 00% of poblem teted ae olved optmally. Ung ucceve appoxmaton (Whte, 960) to educe computaton 24

33 effot, th algothm ha been ued to olve lage-cale upply chan poblem, and appea to be effectve and effcent. 3.2 Model 3.2. Sngle agent MDP and ROMDP A ngle agent Makov decon poce can be defned a a tuple < S, A, T, R >. S the fnte et of global tate; A the et of acton; T: S A S [0,] the tatetanton model, whee p, ' epeent the pobablty of endng at a tate gven that a the poce n tate and the agent take acton a; R: S A R the ewad model, whee a 4 epeent the expected ewad when takng acton a n tate. In a common MDP (efeed hencefoth a a completely obevable Makov decon poce o COMDP), the global tate aumed a completely obevable to the agent. If a global tate not completely obevable to the agent, th poce a patally obevable Makov decon poce (POMDP), whch can be defned a a tuple <S, A, T, R, Z, O>, whee S, A, T, and R ae the ame a thoe n a COMDP. Z the fnte et of obevaton; O: S A Z [0, ] an obevaton pobablty dtbuton model, whee a o, z ' epeent the pobablty that the agent obeve z gven that t took acton a and then the global tate changed to. If the obevaton pobablty dtbuton O mplfed a a mappng functon uch that G()=z, the POMDP degeneate nto a ROMDP. Thu, a ROMDP can be epeented by a tuple < S, A, T, R, Z, G >, whee S, A, T, R, and Z ae the ame a thoe n a POMDP. G: S Z epeent the mappng functon fom a tate to a ngle obevaton fo the agent. Note that the mappng elatonhp enue the pattonng of the tate pace by obevaton. Specally, f G()=, the ROMDP degeneate nto a COMDP. Th chapte fnd the optmal tatonay detemntc polcy 5 to maxmze aveage ewad fo nfnte hozon decon poblem. A COMDP polcy can be epeented a a functon of the tate pace, and a ROMDP polcy a functon of the obevaton pace. Unde a RODMP polcy, f an acton a appled gven an 4 Aumed bounded n th detaton. 5 A common MDP polcy can be categozed a detemntc o andomzed, Makovan o htoydependent. A tatonay polcy geneally ought fo an nfnte hozon MDP poblem. Refe to Puteman (994) fo detal of thee polcy type. 25

34 obevaton z, th acton a apple to any poble tate atfyng G()=z, that, the acton a mut be mplementable/admble to all thee tate. Hence, a ROMDP polcy alo called an mplementable polcy (Sen and Kuln, 995) o admble polcy (Smth, 97). Obvouly, a ROMDP polcy pace a ubet of a common COMDP polcy pace DEC-ROMDP (Mult-agent) Defnton. An n-agent DEC-ROMDP defned a a tuple <S, A, T, R, Z, G, Λ>, whee S a fnte et of global tate; A = A... An a fnte et of jont acton, wth A ndcatng the ndvdual acton et by agent ; T: S A S [0,] a tate-tanton model, whee p, epeent the pobablty of endng at tate, gven that the ytem tate and each agent follow the ndvdual acton a. The collecton of ndvdual acton, (a,, a n ), fom a jont acton a; R: S A R a ewad model, whee a ' a epeent the mmedate expected ewad fo takng jont acton a=(a,, a n ) when the ytem tate ; Z = Z,..., Z } a fnte et of obevaton, wth Z ndcatng the ndvdual { n obevaton et of agent ; G = G,..., G } a et of mappng functon, wth G : S Z ndcatng an { n ndvdual mappng functon fom a tate to an obevaton by agent ; and Λ ={ n} a et of n agent. Defnton 2: Gven an n-agent DEC-ROMDP, a tatonay ndvdual polcy fo an agent defned a δ : Z A, o δ : S A (due to the mappng functon between a global tate and an obevaton by the agent,.e. G : S Z ). Th chapte tend to ue the epeentaton of δ : S A uch that a tatonay jont polcy fo thee agent can be defned a δ: S A... A. Note δ equvalent to (δ, δ 2,,δ n ). n Defnton 3: An agent ha full obevablty f t can obeve the global ytem tate. Othewe, t ha etcted obevablty whee thee ext a mappng functon G : S 26

35 Z. It aumed that only thee two type of obevablty ext wthn an n-agent DEC-ROMDP. Defnton 4: Gven an n-agent DEC-ROMDP and a cuent jont polcy δ = (δ, δ 2 δ n ), δ δ δ δ the teady tate pobablte can be defned a x = ( x, x,..., x ), whee epeent the long un pobablty that the ytem tate k and S the cadnalty of et S. Defnton 5: Gven an n-agent DEC-ROMDP and a cuent jont polcy δ = (δ, δ 2 δ n ), δ = δ S the elatve value can be defned a v ( v,..., v ) whee δ 2 S δ x k g δ δ δ v = p S δ v δ S. Refe to Howad (960) fo moe detal on elatve value. Defnton 6: Gven an n-agent DEC-ROMDP and a cuent jont polcy δ = (δ, δ 2 δ n ), the aocated expected ewad defned a Φ(δ), whch alo called the gan, denoted a δ g. Defnton 7: Gven an n-agent DEC-ROMDP and a cuent jont polcy δ = (δ, δ 2 δ n ), the followng opeaton called an ndvdual polcy update by agent. If agent ha full obevablty, fnd a new ndvdual polcy δ ' whch atfe δ ( ),..., δ ( ), a, δ ( ),..., δn ( )) ( δ( ),..., δ ( ), a, δ δ '( ) = ag max( p S. a A ( ( ),..., δn ( )), ' ' S If agent ha etcted obevablty, fnd a new ndvdual polcy δ ' whch atfe δ ( δ( ),..., δ ( ), a, δ ( ),..., δn ( )) ( δ( ),..., δ ( ), a, δ δ '( ) = ag max[ x ( p S G ( ) = z a A, z Z. G ( S ) = z ' S, ' v δ ' ) ( ),..., δn ( )) Note the above update keep ndvdual polce unchanged fo evey agent except agent,.e. the new jont polcy afte the update = ( δ,..., δ, δ ', δ,..., δ ) δ ' n. v δ ' )] 27

36 Lemma : Gven an n-agent DEC-ROMDP and a cuent jont polcy δ = (δ, δ 2 δ n ), afte a polcy update by agent, the jont polcy become = ( δ,..., δ, δ ', δ,..., δ ) δ ' n. If agent ha full obevablty, t guaanteed that Φ ( δ') Φ( δ). If agent ha etcted obevablty, t not guaanteed that Φ ( δ') Φ( δ). Poof: If agent ha full obevablty, the agent face a COMDP poblem by fxng the othe agent' polce. A polcy update can be teated a a polcy mpovement tep n Howad' (960) pocedue whch guaantee Φ ( δ') Φ( δ). If agent ha etcted obevablty, the agent face a ROMDP poblem by fxng othe agent polce. A polcy update can be teated a a polcy mpovement tep n the heutc algothm fo olvng a ROMDP poblem. Accodng to Chapte 2, th doe not guaantee Φ ( δ') Φ( δ). Howeve, f th happen, the agent ha found a local optmum fo that ROMDP poblem. (Q.E.D) Defnton 8: A polcy update by agent fom δ = (δ, δ 2 δ n ), to δ' = (δ,...,δ,δ,δ,...,δ ) called a polcy mpovement f Φ ( δ') Φ( δ). n Defnton 9: A jont polcy δ = (δ, δ 2 δ n ) called local optmal polcy f no polcy mpovement ext fom any agent whle fxng the othe agent ndvdual polce. The gan aocated wth the local optmal polcy called local optmal gan. Defnton 0: A jont polcy δ = (δ, δ 2 δ n ) called a jont myopc polcy f a A a δ( ) = ag max, S. That, a jont myopc polcy chooe an acton whch maxmze the mmedate expected ewad fo each tate. 3.3 DEC-ROMDP Algothm Th chapte ntoduce an evolutonay coodnaton algothm that update one agent polcy whle keepng othe agent polce unchanged. Thee ext two vaaton of the algothm. The ft update one agent polcy only once and then pefom a polcy update on the next agent. The econd keep updatng one agent polcy untl no mpovement can be made befoe pefomng a polcy update on the next agent. Both temnate when no polcy update avalable at any agent. The detal of the algothm ae a follow. Algothm I 28

37 δ δ Intalze jont polcy, the gan g Φ (δ), and fal 0 whle fal < n do fo = to n polcy update fom δ = (δ, δ 2 δ n ) to δ ' = ( δ,..., δ, δ ', δ,..., δn ) f th polcy update a polcy mpovement then δ g Φ (δ'), δ δ', fal 0 ele fal fal f fal = n then beak δ etun g and δ. Algothm II δ δ Intalze jont polcy, the gan g Φ (δ), and fal 0 whle fal < n do fo = to n mpoved 0 whle tue do polcy update fom δ = (δ, δ 2 δ n ) to δ ' = ( δ,..., δ, δ ', δ,..., δn ) f th polcy update a polcy mpovement then δ g Φ (δ'), δ δ', mpoved ele beak f mpoved = then fal 0 ele fal fal f fal = n then beak δ etun g and δ. Theoem : The above algothm monotoncally nceae expect ewad, and eventually wll temnate at a local optmal polcy afte a fnte numbe of teaton. Poof: Both of the algothm pefom a polcy update on an agent. If th polcy update doe not mpove the cuent polcy, the next agent elected to pefom the polcy update. Hence, the polcy monotoncally nceang. A the expected ewad aumed bounded, the algothm eventually temnate afte a fnte numbe of teaton. Accodng to Defnton 9, t temnate at a local optmal polcy (Q.E.D.) 29