Optimal control of infinite horizon partially observable decision processes modelled as generators of probabilistic regular languages

Transcription

1 Intnational Jounal of Contol Vol. 83, No. 3, Mach 2, Optimal contol of infinit hoizon patially obsvabl dcision pocsss modlld as gnatos of pobabilistic gula languags Ishanu Chattopadhyay* and Asok Ray Dpatmnt of Mchanical Engining, Th Pnnsylvania Stat Univsity, Univsity Pak, PA, USA (Rcivd 22 Dcmb 28; final vsion civd 2 July 29) Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 Dcision pocsss with incomplt stat fdback hav bn taditionally modlld as patially obsvabl Makov dcision pocsss. In this aticl, w psnt an altnativ fomulation basd on pobabilistic gula languags. Th poposd appoach gnaliss th cntly potd wok on languag masu thotic optimal contol fo pfctly obsvabl situations and shows that such a famwok is fa mo computationally tactabl to th classical altnativ. In paticula, w show that th infinit hoizon dcision poblm und patial obsvation, modlld in th poposd famwok, is -appoximabl and, in gnal, is not had to solv compad to th fully obsvabl cas. Th appoach is illustatd via two simpl xampls. Kywods: POMDP; fomal languag thoy; patial obsvation; languag masu; disct vnt systms. Intoduction and motivation Planning und unctainty is on of th oldst and most studid poblms in sach litatu ptaining to automatd dcision making and atificial intllignc. Oftn, th cntal objctiv is to squntially choos contol actions fo on o mo agnts intacting with th opating nvionmnt such that som associatd wad function is maximisd fo a p-spcifid finit futu (finit hoizon poblms) o fo all possibl futus (infinit hoizon poblms). Th contol poblm bcoms immnsly difficult if th ffct of such contol actions is not pfctly obsvabl to th contoll. Absnc of pfct obsvability also maks it had to tak optimal coctiv actions in spons to uncontollabl xognous distubancs fom th intacting nvionmnt. Such scnaios a of immns pactical significanc; th loss of snsos and communication links oftn cannot b avoidd in modn multi-componnt and oftn non-collocatd ngind systms. Und ths cicumstancs, an vnt may concivably b obsvabl at on plant stat whil bing unobsvabl at anoth; vnt obsvability may vn bcom dpndnt on th histoy of vnt occuncs. Among th vaious mathmatical fomalisms studid to modl and solv such contol poblms, Makov dcision pocsss (MDPs) hav civd significant attntion. A bif ovviw of th cunt stat-of-at in MDP-basd dcision thotic planning is ncssay to plac this wok in appopiat contxt.. Makov dcision pocsss MDP modls (Putman 99; Whit 993) xtnd th classical planning famwok (Mcallst and Rosnblitt 99; Pnbthy and Wld 992; Png and Williams 993; Kushmick, Hanks, and Wld 995) to accommodat unctain ffcts of agnt actions with th associatd contol algoithms attmpting to maximis th xpctd wad and is capabl, in thoy, of handling alistic dcision scnaios aising in opations sach, optimal contol thoy and, mo cntly, autonomous mission planning in pobabilistic obotics (Atash and Konig 2). In bif, an MDP consists of stats and actions with a st of action-spcific pobability tansition matics allowing on to comput th distibution ov modl stats sulting fom th xcution of a paticula action squnc. Thus, th ndstat sulting fom an action is not known uniquly a pioi. Howv, th agnt is assumd to occupy on and only on stat at any givn tim, which is coctly obsvd, onc th action squnc is complt. Futhmo, ach stat is associatd with a wad valu and th pfomanc of a contolld MDP is th intgatd wad ov spcifid opation tim (which can b infinit). A patially obsvabl Makov dcision pocss (POMDP) is a gnalisation of MDPs which assums actions to b nondtministic as in an MDP, but laxs th assumption of pfct knowldg of th cunt modl stat. A policy fo an MDP is a mapping fom th st of stats to th st of actions. If both sts a assumd to *Cosponding autho. ixc28@psu.du ISSN pint/issn onlin ß 2 Taylo & Fancis DOI:.8/

2 458 I. Chattopadhyay and A. Ray Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 b finit, th numb of possibl mappings is also finit implying that an optimal policy can b found by conducting sach ov this finit st. In a POMDP, on th oth hand, th cunt stat can b stimatd only as a distibution ov undlying modl stats as a function of opation and obsvation histoy. Th spac of all such stimations o blif stats is a continuous spac although th undlying modl has only a finit numb of stats. In contast to MDPs, a POMDP policy is a mapping fom th blif spac to th st of actions implying that th computation of optimal policy dmands a sach ov a continuum making th poblm dastically mo difficult to solv..2 Ngativ sults ptaining to POMDP solution As statd abov, an optimal solution to a POMDP is a policy which spcifis actions to xcut in spons to stat fdback with th objctiv of maximising pfomanc. Policis may b dtministic with a singl action spcifid at ach blif stat o stochastic which spcify an allowabl choic of actions at ach stat. Policis can also b catgoisd as stationay, tim-dpndnt o histoy-dpndnt; stationay policis dpnd only on th cunt blif stat, tim-dpndnt policis may vay with th opation tim and histoydpndnt policis vay with th stat histoy. Th cunt stat-of-at in POMDP solution algoithms (Zhang and Golin 2; Cassanda 998) a all vaiations of Sondick s oiginal wok (Sondik 978) on valu itation basd on dynamic pogamming (DP). Valu itations, in gnal, a quid to solv a lag numb of lina pogams at ach DP updat and consquntly suff fom xponntial wost cas complxity. Givn that it is had to find an optimal policy, it is natual to ty to sk th on that is good nough. Idally, on would b asonably satisfid to hav an algoithm guaantd to b fast which poducs a policy that is asonably clos (-appoximation) to th optimal solution. Unfotunatly, th xistnc of such algoithms is unlikly o, in som cass, impossibl. Complxity sults show that POMDP solutions a nonappoximabl (Buago, d Rougmont, and Slissnko 996; Madani, Hanks, and Condon 999; Lusna, Goldsmith, and Mundhnk 2) with th abov-statd guaant xisting in gnal only if ctain complxity classs collaps. Fo xampl, th optimal stationay policy fo POMDPs of finit-stat spac can b -appoximatd if and only if P ¼ NP. Tabl poducd fom Lusna t al. (2) summaiss th known complxity sults in this contxt. Thus, finding th histoy-dpndnt optimal policy fo vn a finit hoizon POMDP is PSPACE-complt. Sinc this is a boad poblm Tabl. -appoximability of optimal POMDP solutions. Policy Hoizon Appoximability Stationay K Not unlss P¼NP Tim-dpndnt K Not unlss P¼NP Histoy-dpndnt K Not unlss P¼PSPACE Stationay Not unlss P¼NP Tim-dpndnt Uncomputabl class than NP, th sult suggsts that POMDP poblms a vn had than NP-complt poblms. Claly, infinit hoizon POMDPs can b no asi to solv than finit hoizon POMDPs. In spit of th cnt dvlopmnt of nw xact and appoximat algoithms to fficintly comput optimal solutions (Cassanda 998) and machin-laning appoachs to cop with unctainty (Hansn 998), th most fficint algoithms to dat a abl to comput na optimal solutions only fo POMDPs of lativly small stat spacs..3 Pobabilistic gula languag basd modls This wok invstigats dcision-thotic planning und patial obsvation in a famwok distinct fom th MDP philosophy (Figu ). Dcision pocsss a modlld as pobabilistic finit-stat automata (PFSA) which act as gnatos of pobabilistic gula languags (Chattopadhyay and Ray 28b). Not: It is impotant to not that th PFSA modl usd in this aticl is concptually vy diffnt fom th notion of pobabilistic automata intoducd by Rabin (963), Paz (97), tc., and ssntially follows th fomulation of p-languag thotic analysis fist potd by Gag (992a, 992b). Th ky diffncs btwn th MDP famwok and PFSA-basd modlling (Figu ) can b numatd bifly as follows: () In both MDP and PFSA fomalisms, w hav th notion of stats. Th notion of actions in th fom is analogous to that of vnts in th latt. Howv, unlik actions in th MDP famwok, which can b xcutd at will (if dfind at th cunt stat), th gnation of vnts in th contxt of PFSA modls is pobabilistic. Also, such vnts a catgoisd as bing contollabl o uncontollabl. A contollabl vnt can b disabld so that th stat chang du to th gnation of that paticula vnt is inhibitd; uncontollabl

3 Intnational Jounal of Contol 459 oth hand, hav only on tansition pobability matix computd fom th stat-basd vnt gnation pobabilitis. (4) It is cla that MDPs mphasis stats and stat-squncs; whil PFSA modls mphasis vnts and vnt-squncs. Fo xampl, in POMDPs, th obsvations a stats; whil thos in th obsvability modl fo PFSAs (as adoptd in this aticl) a vnts. (5) In oth wods, patial obsvability in MDP dictly sults in not knowing th cunt stat; in PFSA modls patial obsvability sults in not knowing tanspid vnts which as an ffct causs confusion in th dtmination of th cunt stat. Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 Figu. Compaison of modlling smantics fo MDPs and PFSA. vnts, on th oth hand, cannot b disabld in this sns. (2) Fo an MDP, givn a stat and an action slctd fo xcution, w can only comput th pobability distibution ov modl stats sulting fom th action; although th agnt nds up in an uniqu stat du to xcution of th chosn action, this ndstat cannot b dtmind a pioi. Fo a PFSA, on th oth hand, givn a stat, w only know th pobability of occunc of ach alphabt symbol as th nxt to-b gnatd vnt ach of which causs a tansition to an a pioi known uniqu ndstat; howv, th nxt stat is still unctain du to th possibl xcution of uncontollabl vnts dfind at th cunt stat. Thus, both th fomalisms aim to captu th unctain ffcts of agnt dcisions; albit via diffnt mchanisms. (3) Tansition pobabilitis in MDPs a, in gnal, functions of both th cunt stat and th action xcutd; i.. th a m tansition pobability matics wh m is th cadinality of th st of actions. PFSA modls, on th This aticl psnts an fficint algoithm fo computing th histoy-dpndnt (Lusna t al. 2) optimal supvision policy fo infinit hoizon dcision poblms modlld in th PFSA famwok. Th ky tool usd is th cntly potd concpt of a igoous languag masu fo pobabilistic finit-stat languag gnatos (Chattopadhyay and Ray 26). This is a gnalisation of th wok on languag masuthotic optimal contol fo th fully obsvabl cas (Chattopadhyay and Ray 27) and w show in this aticl that th patially obsvabl scnaio is not had to solv in this modlling famwok. Th st of this aticl is oganisd in fiv additional sctions and two bif appndics. Sction 2 intoducs th pliminay concpts and lvant sults fom th potd litatu. Sction 3 psnts an onlin implmntation of th languag masu-thotic supvision policy fo pfctly obsvabl plants which lays th famwok fo th subsqunt dvlopmnt of th poposd optimal contol policy fo patially obsvabl systms in Sction 4. Th thotical dvlopmnt is vifid and validatd in two simulatd xampls in Sction 5. This aticl is summaisd and concludd in Sction 6 with commndations fo futu wok. 2. Pliminay concpts and latd wok This sction psnts th fomal dfinition of th PFSA modl and summaiss th concpt of signd al masu of gula languags; th dtails a potd in Ray (25), Ray, Phoha, and Phoha (25) and Chattopadhyay and Ray (26b). Also, w bifly viw th computation of th uniqu maximally pmissiv optimal contol policy fo PFSA (Chattopadhyay and Ray 27b) via th maximisation of th languag masu. In th squl, this masuthotic appoach will b gnalisd to addss

4 46 I. Chattopadhyay and A. Ray Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 patially obsvabl cass and is thus citical to th dvlopmnt psntd in this aticl. 2. Th PFSA modl Lt G i ¼ (Q,,, q i, Q m ) b a finit-stat automaton modl that ncods all possibl volutions of th disct-vnt dynamics of a physical plant, wh Q ¼ {q k : k 2I Q } is th indx st of stats and I Q {, 2,..., n} is th indx st of stats; th automaton stats with th initial stat q i ; th alphabt of vnts is ¼ { k : k 2I }, having T I Q ¼; and I {, 2,..., } is th indx st of vnts; : Q! Q is th (possibly patial) function of stat tansitions; and Q m {q m, q m2,..., q ml } Q is th st of makd (i.. accptd) stats with q mk ¼ q j fo som j 2I Q. Lt * b th Kln closu of, i.. th st of all finit-lngth stings mad of th vnts blonging to as wll as th mpty sting that is viwd as th idntity of th monoid * und th opation of sting concatnation, i.. s ¼ s ¼ s. Th stat tansition map is cusivly xtndd to its flxiv and tansitiv closu : Q *! Q by dfining 8q j 2 Q, ðq j, Þ ¼q j ðaþ 8q j 2 Q, 2, s 2?, ðq i, sþ ¼ððq i, Þ, sþ ðbþ Dfinition 2.: Th languag L(q i ) gnatd by a dtministic finit-stat automata (DFSA) G initialisd at th stat q i 2 Q is dfind as: Lðq i Þ¼fs 2 j ðq i, sþ2qg Th languag L m (q i ) makd by th DFSA G initialisd at th stat q i 2 Q is dfind as L m ðq i Þ¼fs 2 j ðq i, sþ2q m g Dfinition 2.2: Fo vy q j 2 Q, lt L(q i, q j ) dnot th st of all stings that, stating fom th stat q i, tminat at th stat q j, i.. L i,j ¼fs 2 j ðq i, sþ ¼q j 2 Qg To complt th spcification of a PFSA, w nd to spcify th vnt gnation pobabilitis and th stat chaactistic wight vcto, which w dfin nxt. Dfinition 2.3: Th vnt gnation pobabilitis a spcifid by th function ~ : Q?!½, Š such that 8q j 2 Q, 8 k 2, 8s 2?, () ~ðq j, k Þ¼ 4 ~ jk 2½,Þ; P k ~ jk ¼, with2ð,þ; (2) ~ðq j, Þ ¼if(q j, ) is undfind; ~ðq j, Þ ¼; (3) ~ðq j, k sþ¼ ~ðq j, k Þ ~ððq j, k Þ, sþ: ð2þ ð3þ ð4þ Notation 2.: Th n vnt cost matix is dfind as: j ij ¼ ~ðq i, j Þ Dfinition 2.4: Th stat tansition pobability : Q Q! [, ), of th DFSA G i is dfind as follows: X 8q i, q j 2 Q, ij ¼ ~ðq i, Þ ð5þ 2 s:t: ðq i, Þ¼q j Notation 2.2: Th n n stat tansition pobability matix is dfind as j ij ¼ (q i, q j ) Th st Q m of makd stats is patitiond into Q þ m and Q m, i.. Q m ¼ Q þ m [ Q m and Qþ m \ Q m ¼;, wh Q þ m contains all good makd stats that w dsi to ach, and Q m contains all bad makd stats that w want to avoid, although it may not always b possibl to compltly avoid th bad stats whil attmpting to ach th good stats. To chaactis this, ach makd stat is assignd a al valu basd on th dsign s pcption of its impact on th systm pfomanc. Dfinition 2.5: Th chaactistic function : Q! [, ] that assigns a signd al wight to stat-basd sublanguags L(q i, q) is dfind as 8 >< ½, Þ, q 2 Q m 8q 2 Q, ðqþ2 fg, q 2= Q m ð6þ >: ð, Š, q 2 Q þ m Th stat wighting vcto, dnotd by ¼ [ 2... n ] T, wh j (q j ) 8j 2I Q, is calld th -vcto. Th j-th lmnt j of -vcto is th wight assignd to th cosponding tminal stat q j. Rmak 2.: Th stat chaactistic function : Q! [, ] o quivalntly th chaactistic vcto s is analogous to th notion of th wad function in MDP analysis. Howv, unlik MDP modls, wh th wad (o pnalty) is put on individual stat-basd actions, in ou modl, th chaactistic is put on th stat itslf. Th similaity of th two notions is claifid by noting that just as MDP pfomanc can b valuatd as th total wad gand as actions a xcutd squntially, th pfomanc of a PFSA can b computd by summing th chaactistics of th stats visitd du to tanspid vnt squncs. Plant modls considd in this aticl a DFSA (plant) with wll-dfind vnt occunc pobabilitis. In oth wods, th occunc of vnts is pobabilistic, but th stat at which th plant nds up, givn a paticula vnt has occud, is dtministic. No mphasis is laid on th initial stat of th plant, i.. w allow fo th fact that th plant may stat fom

5 Intnational Jounal of Contol 46 Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 any stat. Futhmo, having dfind th chaactistic stat wight vcto s, it is not ncssay to spcify th st of makd stats, bcaus if i ¼, thn q i is not makd and if i 6¼, thn q i is makd. Dfinition 2.6 (Contol philosophy): If q i! q k and th vnt is disabld at stat q i, thn th supvisoy action is to pvnt th plant fom making a tansition to th stat q k, by focing it to stay at th oiginal stat q i. Thus, disabling any tansition at a givn stat q sults in dltion of oiginal tansition and th appaanc of slf-loop (q, ) ¼ q with th occunc pobability of fom th stat q maining unchangd in th supvisd and unsupvisd plants. Dfinition 2.7 (Contollabl tansitions): Fo a givn plant, tansitions that can b disabld in th sns of Dfinition 2.6 a dfind to b contollabl tansitions. Th st of contollabl tansitions in a plant is dnotd by C. Not that contollability is stat-basd. It follows that plant modls can b spcifid by th following sxtuplt: G ¼ðQ,,,,, CÞ 2.2 Fomal languag masu fo tminating plants Th fomal languag masu is fist dfind fo tminating plants (Gag 992b) with sub-stochastic vnt gnation pobabilitis, i.. th vnt gnation pobabilitis at ach stat summing to stictly lss than unity. In gnal, th makd languag L m (q i ) consists of both good and bad vnt stings that, stating fom th initial stat q i, lad to Q þ m and Q m, spctivly. Any vnt sting blonging to th languag L ðq i Þ¼ Lðq i Þ L m ðq i Þ lads to on of th non-makd stats blonging to Q Q m and L dos not contain any on of th good o bad stings. Basd on th quivalnc classs dfind in th Myhill Nod Thom (Hopcoft, Motwani, and Ullman 2), th gula languags L(q i ) and L m (q i ) can b xpssd as Lðq i Þ¼ [ L i,k ð8þ q k 2 Q L m ðq i Þ¼ [ L i,k ¼ L þ m [ L m ð9þ q k 2 Q m wh th sublanguag L i,k L(q i ) having th initial stat q i is uniquly lablld by th tminal stat q k, k 2I Q and L i,j \ L i,k ¼; 8j 6¼ k; and L þ m S q k 2 Q þ L i,k m and L m S q k 2 Q L i,k a good and bad sublanguags m of L m (q i ), spctivly. Thn, L ¼ S q k 2= Q m L i,k and Lðq i Þ¼L [ L þ m [ L m. A signd al masu i :2 L(qi)! R (, þ) is constuctd on th -algba 2 L(qi) fo any i 2I Q and ð7þ intstd ads a fd to Ray (25) and Ray t al. (25) fo th dtails of masu-thotic dfinitions and sults. With th choic of this -algba, vy singlton st mad of an vnt sting s 2 L(q i ) is a masuabl st. By Hahn Dcomposition Thom (Rudin 988), ach of ths masuabl sts qualifis itslf to hav a numical valu basd on th abov stat-basd dcomposition of L(q i ) into L (null), L þ (positiv) and L (ngativ) sublanguags. Dfinition 2.8: Lt! 2 L(q i, q j ) 2 L(qi). Th signd al masu i of vy singlton sting st! is dfind as i ðf!gþ ¼ ~ðq i,!þðq j Þ ðþ Th signd al masu of a sublanguag L i,j L(q i ) is dfind as! X i,j ¼ i ðlðq i, q j ÞÞ ¼ ~ðq i,!þ ðþ! 2 Lðq i,q j Þ Thfo, th signd al masu of th languag of a DFSA G i initialisd at q i 2 Q, is dfind as i ¼ i ðlðq i ÞÞ ¼ X m i ðl i,j Þ ð2þ j 2I Q It is shown in Ray (25) and Ray t al. (25) that th languag masu in Equation (2) can b xpssd as i ¼ X ij j þ i ð3þ j 2I Q Th languag masu vcto, dnotd as ¼ [ 2... n ] T, is calld th -vcto. In a vcto fom, Equation (3) bcoms whos solution is givn by k ¼ k þ v k ¼ðI Þ v j ð4þ ð5þ Th invs in Equation (5) xists fo tminating plant modls (Gag 992a, 992b) bcaus is a contaction opato (Ray 25a; Ray t al. 25) du to th stict inquality P j ij 5. Th sidual i ¼ P j ij is fd to as th tmination pobability fo stat q i 2 Q. W xtnd th analysis to nontminating plants (Gag 992a, 992b) with stochastic tansition pobability matics (i.. with i ¼, 8q i 2 Q) by nomalising th languag masu (Chattopadhyay and Ray 26) with spct to th unifom tmination pobability of a limiting tminating modl as dscibd futh.

6 462 I. Chattopadhyay and A. Ray Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 Lt and b th stochastic vnt gnation and tansition pobability matics fo a non-tminating plant G i ¼ (Q,,, q i, Q m ). W consid th tminating plant G i () with th sam DFSA stuctu (Q,,, q i, Q m ) such that th vnt gnation pobability matix is givn by ð Þ with 2 (, ) implying that th stat tansition pobability matix is ( ). Dfinition 2.9 (Rnomalisd masu): Th nomalisd masu i : 2LðqiÞ!½, Š fo th -paamtisd tminating plant G i () is dfind as: 8! 2 Lðq i Þ, i ðf!gþ ¼ i ðf!gþ ð6þ Th cosponding matix fom is givn by l ¼ k ¼ ½I ð ÞŠ v with 2ð, Þ ð7þ W not that th vcto psntation allows fo th following notational simplification: i ðlðq iþþ ¼ l j i ð8þ Th nomalisd masu fo th non-tminating plant G i is dfind to b lim! þ i. Th following sults a taind fo th sak of compltnss. Complt poofs can b found in Chattopadhyay (26) and Chattopadhyay and Ray (26b). Poposition 2.: Th limiting masu vcto l ¼ 4 lim! þ l xists and kl k. Poposition 2.2: Lt b th stochastic tansition matix of a non-tminating PFSA (Gag 992a, 992b). Thn, as th paamt! þ, th limiting masu vcto is obtaind as: l ¼C()s wh th matix opato C¼ 4 P lim k k! k j¼ j is th Csao limit (Bapat and Raghavan 997; Bman and Plmmons 979) of th stochastic tansition matix. Coollay 2. (to Poposition 2.2): Th xpssion C()l is indpndnt of. Spcifically, th following idntity holds fo all 2 (, ): CðÞl ¼CðÞv ð9þ Notation 2.3: Th linaly indpndnt othogonal st fv i 2 R CadðQÞ : v i j ¼ ijg is dnotd as B wh ij dnots th Ko nck dlta function. W not that th is a on-to-on onto mapping btwn th stats q i 2 Q and th lmnts of B, namly: q i () k ¼ if k ¼ i ð2þ othwis Dfinition 2.: Fo any non-zo vcto v 2 R CARD(Q), th nomalising function N : R CARD(Q) n! R CARD(Q) is dfind as NðvÞ ¼P v. i v i 2.3 Th optimal supvision poblm: fomulation and solution A supviso disabls a subst of th st C of contollabl tansitions and hnc th is a bijction btwn th st of all possibl supvision policis and th pow st 2 C. That is, th xist 2 jcj possibl supvisos and ach supviso is uniquly idntifiabl with a subst of C and th cosponding languag masu l allows a quantitativ compaison of diffnt policis. Dfinition 2.: Fo an unsupvisd plant G ¼ðQ,,,,, CÞ, lt G y and G z b th supvisd plants with sts of disabld tansitions, D y C and D z C, spctivly, whos masus a l y and l z. Thn, th supviso that disabls D y is dfind to b supio to th supviso that disabls D z if l y = ELEMENTWISE l z and stictly supio if l y 4 ELEMENTWISE l z. Dfinition 2.2 (Optimal supvision poblm): Givn a (non-tminating) plant G ¼ðQ,,,,, CÞ, th poblm is to comput a supviso that disabls a subst D? C, such that 8D y C, l? = ELEMENTWISE l y wh l? and l y a th masu vctos of th supvisd plants G? and G y und D? and D y, spctivly. Rmak 2.2: Th solution to th optimal supvision poblm is obtaind in Chattopadhyay and Ray (27) and Chattopadhyay and Ray (27b) by dsigning an optimal policy fo a tminating plant (Gag 992a, 992b) with a substochastic tansition pobability matix ð Þ with 2 (, ). To nsu that th computd optimal policy coincids with th on fo ¼, th suggstd algoithm chooss a small valu fo in ach itation stp of th dsign algoithm. Howv, choosing too small may caus numical poblms in convgnc. Algoithm B.2 (Appndix B) computs th citical low bound? (i.. how small a is actually quid). In conjunction with Algoithm B.2, th optimal supvision poblm is solvd by using of Algoithm B. fo a gnic PFSA as potd in Chattopadhyay (27) and Chattopadhyay and Ray (27b). Th following sults in Poposition 2.3 a citical to dvlopmnt in th squl and hnc a psntd h without poof. Th complt poofs a availabl in Chattopadhyay (27) and Chattopadhyay and Ray (27b). Poposition 2.3 () (Monotonicity) Lt l [k] b th languag masu vcto computd in th kth itation of Algoithm B.. Th masu vctos computd by th

7 Intnational Jounal of Contol 463 Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 algoithm fom an lmntwis non-dcasing squnc, i.. l [kþ] = ELEMENTWISE l [k] 8k. (2) (Effctivnss) Algoithm B. is an ffctiv pocdu (Hopcoft t al. 2), i.. it is guaantd to tminat. (3) (Optimality) Th supvision policy computd by Algoithm B. is optimal in th sns of Dfinition 2.2. (4) (Uniqunss) Givn an unsupvisd plant G, th optimal supviso G?, computd by Algoithm B., is uniqu in th sns that it is maximally pmissiv among all possibl supvision policis with optimal pfomanc. That is, if D? and D y a th disabld tansition sts, and l? and l y a th languag masu vctos fo G? and an abitaily supvisd plant G y, spctivly, thn l? ELEMENTWISE l y ) D? D y C. Dfinition 2.3: Following Rmak 2.2, w not that Algoithm B.2 computs a low bound fo th citical tmination pobability fo ach itation of Algoithm B. such that th disabling/nabling dcisions fo th tminating plant coincid with th givn non-tminating modl. W dfin min ¼ min? ½kŠ k ð2þ wh? ½kŠ is th tmination pobability computd by Algoithm B.2 in th kth itation of Algoithm B.. Dfinition 2.4: If G and G? a th unsupvisd and optimally supvisd PFSA, spctivly, thn w dnot th nomalisd masu of th tminating plant G? ( min )as i? : 2LðqiÞ!½, Š (Dfinition 2.9). Hnc, in vcto notation w hav l? ¼ l min ¼ min ½I ð min Þ? Š v ð22þ wh? is th tansition pobability matix of th supvisd plant G?. Rmak 2.3: Rfing to Algoithm B., it is notd that l? ¼ [k] wh K is th total numb of itations fo Algoithm B Th patial obsvability modl Th obsvation modl usd in this aticl is dfind by th so-calld unobsvability maps dvlopd in Chattopadhyay and Ray (27a) as a gnalisation of natual pojctions in disct vnt systms. It is impotant to mntion that whil som authos f to unobsvability as th cas wh no tansitions a obsvabl in th systm, w us th tms unobsvabl and patially obsvabl intchangably in th squl. Th lvant concpts dvlopd in Chattopadhyay and Ray (27a) a numatd in this sction fo th sak of compltnss Assumptions and notations W mak two ky assumptions:. Th unobsvability situation in th modl is spcifid by a boundd mmoy unobsvability map p which is availabl to th supviso.. Unobsvabl tansitions a uncontollabl. Dfinition 2.5: An unobsvability map p : Q?!? fo a givn modl G ¼ðQ,,,, v, CÞ is dfind cusivly as follows: 8q i 2 Q, j 2 and j! 2 L(q i ), pðq i, j Þ¼, if j is unobsvabl fom q i j, othwis pðq i, j!þ¼pðq i, j Þpððq i, Þ,!Þ ð23aþ ð23bþ W can indicat tansitions to b unobsvabl in th gaph fo th automaton G ¼ðQ,,,,, CÞ as unobsvabl and this would suffic fo a complt spcification of th unobsvability map acting on th plant. Th assumption of boundd mmoy of th unobsvability maps imply that although w may nd to unfold th automaton gaph to unambiguously indicat th unobsvabl tansitions, th xists a finit unfolding that suffics fo ou pupos. Such unobsvability maps w fd to as gula in Chattopadhyay and Ray (27a). Rmak 2.4: Th unobsvability maps considd in this aticl a stat-basd as opposd to bing vntbasd obsvability considd in Ramadg and Wonham (987). Dfinition 2.6: A sting! 2? is calld unobsvabl at th supvisoy lvl if at last on of th vnts in! is unobsvabl, i.. p(q i,!) 6¼! Similaly, a sting! 2? is calld compltly unobsvabl if ach of th vnts in! is unobsvabl, i.. p(q i,!) ¼. Also, if th a no unobsvabl stings, w dnot th unobsvability map p as tivial. Th subsqunt analysis quis th notion of th phantom automaton intoducd in Chattopadhyay and Ray (26a). Th following dfinition is includd fo th sak of compltion. Dfinition 2.7: Givn a modl G ¼ðQ,,,, v, CÞ and an unobsvability map p, th phantom automaton PðGÞ ¼ðQ,, PðÞ, PðÞ, v, PðCÞÞ is dfind as

8 464 I. Chattopadhyay and A. Ray Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 follows: PðÞðq i, j Þ¼ ðq i, j Þ if pðq i, j Þ¼ Undfind othwis ( ðq PðÞðq i, j Þ¼ i, j Þ if pðq i, j Þ¼ othwis PðCÞ ¼ ð24aþ ð24bþ ð24cþ Rmak 2.5: Th phantom automata in th sns of Dfinition 2.7 is a finit-stat machin dsciption of th languag of compltly unobsvabl stings sulting fom th unobsvability map p acting on th modl G ¼ðQ,,,, v, CÞ. Not that Equation (24c) is a consqunc of th assumption that unobsvabl tansitions a uncontollabl. Thus, no tansition in th phantom automaton is contollabl. Algoithm B.3 (Appndix B) computs th tansition pobability matix fo th phantom automaton of a givn plant G und a spcifid unobsvability map p by dlting all obsvabl tansitions fom G Th Pti nt obsv Fo a givn modl G ¼ðQ,,,, v, CÞ and a nontivial unobsvability map p, it is, in gnal, impossibl to pinpoint th cunt stat fom an obsvd vnt squnc at th supvisoy lvl. Howv, it is possibl to stimat th st of plausibl stats fom a knowldg of th phantom automaton P(G). Dfinition 2.8 (Instantanous stat dsciption): Fo a givn plant G ¼ðQ,,,, v, CÞ initialisd at stat q 2 Q and a non-tivial unobsvability map p, th instantanous stat dsciption is dfind to b th imag of an obsvd vnt squnc! 2? und th map Q : pðlðg ÞÞ! 2 Q as follows: n ^ o Qð!Þ¼ q j 2 Q : 9s 2? s:t: ðq, sþ¼q j pðq, sþ¼! Rmak 2.6: Not that fo a tivial unobsvability map p with 8! 2?, p(!) ¼!, w hav Qð!Þ ¼ðq,!Þ wh q is th initial stat of th plant. Th instantanous stat dsciption Qð!Þ can b stimatd on-lin by constucting a Pti nt obsv with flush-out acs (Moody and Antsaklis 998; Gibaudo, Sno, Hovath, and Bobbio 2). Th advantag of using a Pti nt dsciption is th compactnss of psntation and th simplicity of th on-lin xcution algoithm that w psnt nxt. Ou pfnc of a Pti nt dsciption ov a subst constuction fo finit-stat machins is motivatd by th following: th Pti nt fomalism is natual, du to its ability to modl tansitions of th typ q!j % q 2 & q3, which flcts th condition th plant can possibly b in stats q 2 o q 3 aft an obsvd tansition fom q. On can avoid intoducing an xponntially lag numb of combind stats of th fom [q 2, q 3 ] as involvd in th subst constuction and mo impotantly psv th stat dsciption of th undlying plant. Flush-out acs w intoducd by Gibaudo t al. (2) in th contxt of fluid stochastic Pti nts. W apply this notion to odinay nts with simila maning: a flush-out ac is connctd to a lablld tansition, which, on fiing, movs a tokn fom th input plac (if th ac wight is on). Instantanous dsciptions can b computd on-lin fficintly du to th following sult: Poposition 2.4 () Algoithm B.4 has polynomial complxity. (2) Onc th Pti nt obsv has bn computd offlin, th cunt possibl stats fo any obsvd squnc can b computd by xcuting Algoithm B.5 onlin. Poof: Poof is givn in Chattopadhyay and Ray (27a). 3. Onlin implmntation of masu-thotic optimal contol und pfct obsvation This sction dviss an onlin implmntation schm fo th languag masu-thotic optimal contol algoithm which will b lat xtndd to handl plants with non-tivial unobsvability maps. Fomally, a supvision policy S fo a givn plant G ¼ðQ,,,,, CÞ spcifis th contol in th tms of disabld contollabl tansitions at ach stat q i 2 Q, i.. S ¼ (G, ) wh Cad ðþ : Q! f, g ð25þ Th map is fd in th litatu as th stat fdback map (Ramadg and Wonham 987) and it spcifis th st of disabld tansitions as follows: if at stat q i 2 Q and vnts i, i a disabld by th paticula supvision policy, thn (q i ) is a binay squnc on {, } of lngth qual to th cadinality of th vnt alphabt such that?? y i th lmnt EEE y i th lmnt ðq i Þ¼ Rmak 3.: If it is possibl to patition th alphabt as ¼ c F uc, wh c is th st of contollabl tansitions and uc is th st of uncontollabl

9 Intnational Jounal of Contol 465 Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 tansitions, thn it suffics to consid as a map : Q! {, } Cad(c). Howv, sinc w consid contollability to b stat dpndnt (i.. th possibility that an vnt is contollabl if gnatd at a stat q i and uncontollabl if gnatd at som oth stat q j ), such a patitioning schm is not fasibl. Und pfct obsvation, a computd supviso (G, ) sponds to th pot of a gnatd vnt as follows:. Th cunt stat of th plant modl is computd as q cunt ¼ (q last, ), wh is th potd vnt and q last is th stat of th plant modl bfo th vnt is potd.. All vnts spcifid by (q cunt ) is disabld. Not that such an appoach quis th supviso to mmb (q i )8q i 2 Q, which is quivalnt to kping in mmoy an n m matix, wh n is th numb of plant stats and m is th cadinality of th vnt alphabt. W show that th is a altnativ simpl implmntation. Lmma 3.: Fo a givn finit-stat plant G ¼ðQ,,,,, CÞ and th cosponding optimal languag masu l?, th pai (G, l? ) compltly spcifis th optimal supvision policy. Poof: Th optimal configuation G? is chaactisd as follows (Chattopadhyay 26; Chattopadhyay and Ray 27a):. if fo stats q i, q j 2 Q, l i? 4 l j?, thn all contollabl tansitions q i! q j a disabld. qi. if fo stats q i, q j 2 Q, l i? 5 l j?, thn all contollabl tansitions q i! q j a nabld. qi It follows that if th supviso has accss to th unsupvisd plant modl G and th languag masu vcto l?, thn th optimal policy can b implmntd by th following pocdu: () Comput th cunt stat of th plant modl as q cunt ¼ (q last, ), wh is th potd vnt and q old is th stat of th plant modl bfo th vnt is potd. Lt q cunt ¼ q i. (2) Disabl all contollabl tansitions q i! q k if j l i? 4 l k? fo all q k 2 Q. This complts th poof. Th pocdu is summaisd in Algoithm 3.. Th appoach givn in Lmma 3. is impotant fom th pspctiv that it foms th intuitiv basis fo xtnding th optimal contol algoithm divd und th assumption of pfct obsvation to situations wh on o mo tansitions a unobsvabl at th supvisoy lvl. 4. Optimal contol und non-tivial unobsvability This sction maks us of th unobsvability analysis psntd in Sction 2.4 to div a modifid onlinimplmntabl contol algoithm fo patially obsvabl pobabilistic finit-stat plant modls. 4. Th faction nt obsv In Sction 2.4, th notion of instantanous dsciption of was intoducd as a map Q : pðlðg i ÞÞ! 2 Q fom th st of obsvd vnt tacs to th pow st of th stat st Q, such that givn an obsvd vnt tac!, Qð!Þ Q is th st of stats that th undlying dtministic finit stat plant can possibly occupy at th givn instant. W constuctd a Pti nt obsv (Algoithm B.4) and showd that th instantanous dsciption can b computd onlin with polynomial complxity. Howv, fo a plant modlld by a pobabilistic gula languag, th knowldg of th vnt occunc pobabilitis allows us to comput not only th st of possibl cunt stats (i.. th instantanous dsciption) but also th pobabilistic cost of nding up in ach stat in th instantanous dsciption. To achiv this objctiv, w modify th Pti nt obsv intoducd in Sction by assigning (possibly) factional wights computd as functions of th vnt occunc pobabilitis to th input acs. Th output acs a still givn unity wights. In th squl, th Pti nt obsv with possibly factional ac wights is fd to as th faction nt

10 466 I. Chattopadhyay and A. Ray Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 obsv (FNO). Fist w nd to fomalis th notation fo th FNO. Dfinition 4.: Givn a finit-stat tminating plant modl GðÞ ¼ðQ,,, ð Þ,, CÞ, and an unobsvability map p, th FNO, dnotd as F (G,p), is a lablld Pti nt (Q,, A I, A O, w I, x ) with factional ac wights and possibly factional makings, wh Q is th st of placs, is th vnt labl alphabt, A I j Q Q and A O j Q a th sts of input and output acs, w I is th input wight assignmnt function and x 2B (Notation 2.3) is th initial making. Th output acs a dfind to hav unity wights. Th algoithmic constuction of an FNO is divd nxt. W assum that th Pti nt obsv has alady bn computd (by Algoithm B.4) with Q th st of placs, th st of tansition labls, A I j Q Q th st of input acs and A O j Q th st of output acs. Dfinition 4.2: Th input wight assigning function w I : A I! (, ) fo th FNO is dfind as 8q i 2 Q, 8 j 2, 8q k 2 Q, ðq i, j Þ¼q ¼)w I ðq i, j, q k Þ X ¼ ð Þ j!j ~ðq,!þ! 2? Vs.t.? ðq,!þ¼q k pðq,!þ¼ wh : Q! Q is th tansition map of th undlying DFSA and p is th givn unobsvability map and ~ is th vnt cost (i.. th occunc pobability) function (Ray 25). It follows that th wight on an input ac fom tansition j (having an output ac fom plac q i ) to plac q k is th sum of th total conditional pobabilitis of all compltly unobsvabl paths by which th undlying plant can ach th stat q k fom stat q wh q ¼ (q i, j ). Computation of th input ac wights fo th FNO quis th notion of th phantom automaton (Dfinition 2.7). Th computation of th ac wights fo th FNO is summaisd in Algoithm 4.. Poposition 4.: Givn a Pti nt obsv (Q,, A I, A O ), th vnt occunc pobability matix and th tansition pobability matix fo th phantom automaton P(), Algoithm 4. computs th ac wights fo th FNO as statd in Dfinition 4.2. Poof: Algoithm 4. mploys th following idntity to comput input ac wights: 8q i 2Q, 8 j 2, 8q k 2Q, w I ðq i, j,q k Þ 8h < I ð ÞPðÞi if ðq i, j,q k Þ2A I ^ ðq i, j Þ¼q ¼ k :, othwis which follows fom th following agumnt. Assum that fo th givn unobsvability map p, G P is th phantom automaton fo th undlying plant G. W obsv that th masu of th languag of all stings initiating fom stat q and tminating at stat q k in th phantom automaton G P is givn by I PðÞ k. Sinc vy sting gnatd by th phantom automaton is compltly unobsvabl (in th sns of Dfinition 2.7), w conclud h X I ð ÞPðÞi ¼ ð Þ j!j ~ðq,!þ k!2? s.t.? ðq,!þ¼q k Vpðq,!Þ¼ ð26þ This complts th poof. In Sction 2.4.2, w psntd Algoithm B.5 to comput th instantanous stat dsciption Qð!Þ onlin without fing to th tansition pobabilitis. Th appoach consistd of fiing all nabld tansitions (in th Pti nt obsv) lablld by j on obsving th vnt j in th undlying plant. Th st of possibl cunt stats thn consistd of all stats which cospondd to placs with on o mo tokns. Fo th FNO, w us a slightly diffnt appoach which involvs th computation of a st of vnt-indxd stat tansition matics. Dfinition 4.3: Fo an FNO (Q,, A I, A O, w I, x ), th st of vnt-indxd stat tansition matics! ¼ { j : j 2 } is a st of m matics ach of dimnsion n n (wh m is th cadinality of th vnt alphabt and n is th numb of placs), such that on obsving vnt j in th undlying plant, th updatd making x [kþ] fo th FNO (du to fiing of all nabld j -lablld tansitions in th nt) can b obtaind fom th xisting making x [k] as follows: x ½kþŠ ¼ x ½kŠ j ð27þ Th pocdu fo computing! is psntd in Algoithm 4.2. Not that th only inputs to th

11 Intnational Jounal of Contol 467 Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 algoithm a th tansition matix fo th phantom automaton, th unobsvability map p and th tansition map fo th undlying plant modl. Th nxt poposition shows that th algoithm is coct. Poposition 4.2: Algoithm 4.2 coctly computs th st of vnt-indxd tansition matics! ¼ { j: j 2 } fo a givn FNO (Q,, A I, w I, x ) in th sns statd in Dfinition 4.3. Poof: Lt th cunt making of th FNO spcifid as (Q,, A I, A O, w O, w I ) b dnotd by x [k] wh x [k] 2 [, ) n with n ¼ Cad(Q). Assum that vnt j 2 is obsvd in th undlying plant modl. To obtain th updatd making of th FNO, w nd to fi all tansitions lablld by j in th FNO. Sinc th gaph of th FNO is idntical with th gaph of th Pti nt obsv constuctd by Algoithm B.4, it follows that if (q i, j ) is undfind o th vnt j is unobsvabl fom th stat q i in th undlying plant, thn th is a flush-out ac to a tansition lablld j fom th plac q i in th gaph of th FNO. This implis that th contnt of plac q i will b flushd out and hnc will not contibut to any plac in th updatd making x [kþ], i.. x ½kŠ i j i ¼ 8 i 2f,..., ng ð28þ implying that th ith column of th matix j is [,...,] T. This justifis Lin 5 of Algoithm 4.2. If j is dfind and obsvabl fom th stat q i in th undlying plant, thn w not that th contnts of th plac q i nd up in all placs q 2 Q such that th xists an input ac (q i, j, q ) in th FNO. Moov, th contibution to th plac q coming fom plac q i is wightd by w I (q i, j, q ). Dnot this contibution by c i. Thn w hav c i ¼ w I ðq i, j, q Þx ½kŠ i ¼) X c i ¼ X w I ðq i, j, q Þx ½kŠ i i i ¼)x ½kþŠ ¼ X w I ðq i, j, q Þx ½kŠ i i ð29þ Not that P i c i ¼ x ½kþŠ sinc contibutions fom all placs to q sum to th valu of th updatd making in th plac q. Rcalling fom Poposition 4. that h w I ðq i, j, q Þ¼ I ð ÞPðÞi ð3þ wh q ¼ (q i, j ) in th undlying plant, th sult follows. Poposition 4.2 allows an altnat computation of th instantanous stat dsciption. W assum that th initial stat of th undlying plant is known and hnc th initial making fo th FNO is assignd as follows: x ½Š i ¼ if q i is th initial stat ð3þ othwis It is impotant to not that sinc th undlying plant is a DFSA having only on initial stat, th initial making of th FNO has only on plac with valu and all maining placs a mpty. It follows fom Poposition 4.2 that fo a givn initial making x [] of th FNO, th making aft obsving a sting! ¼ k wh j 2 is obtaind as: x ½kŠ ¼ x ½Š Yj¼ k j¼ j ð32þ Rfing to th notation fo instantanous dsciption intoducd in Dfinition 2.8, w hav Qð!Þ ¼ q i 2 Q: x ½j!jŠ i 4 ð33þ Rmak 4.: W obsv that to solv th stat dtminacy poblm, w only nd to know if th individual making valus a non-zo. Th spcific valus of th ntis in th making x [k], howv, allow us to stimat th cost of occupying individual stats in th instantanous dsciption Qð!Þ. 4.2 Stat ntanglmnt du to patial obsvability Th makings of th FNO F (G,p) fo th plant GðÞ ¼ðQ,,, ð Þ,, CÞ in th cas of pfct obsvation is of th following fom: 8k2N, x ½kŠ ¼½Š T,i:: x ½kŠ 2BðNotation 2:3Þ It follows that fo a pfctly obsvabl systm, B is an numation of th stat st Q in th sns x ½kŠ i ¼ implis that th cunt stat is q i 2 Q. Und a nontivial unobsvability map p, th st of all possibl FNO makings polifats and w can intpt x [k] aft th kth obsvation instanc as th cunt stats of th obsvd dynamics. This follows fom th fact that no pvious knowldg byond that of th cunt FNO making x [k] is quid to dfin th futu volution of x [k]. Th ffct of patial obsvation can

12 468 I. Chattopadhyay and A. Ray Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 thn b intptd as adding nw stats to th modl with ach nw stat a lina combination of th undlying stats numatd in B. Dawing an analogy with th phnomnon of stat ntanglmnt in quantum mchanics, w f to B as th st of pu stats; whil all oth occupancy stimats that may appa a fd to as mixd o ntangld stats. Evn fo a finit stat plant modl, th cadinality of th st of all possibl ntangld stats is not guaantd to b finit. Lmma 4.: Lt F (G,p) with initial making x [] 2Bb th FNO fo th undlying tminating plant GðÞ ¼ðQ,,, ð Þ,, CÞ with unifom tmination pobability. Thn fo any obsvd sting! ¼ s of lngth s 2 N with j 2 8 j 2 {,..., k}, th occupancy stimat x [k], aft th occunc of th kth obsvabl tansition, satisfis: x ½kŠ 2, CARD ðþ ð34aþ Poof:. Lt th initial making x [] 2B b givn by ½ Š ð35þ ðith lmntþ " Elmntwis non-ngativity of x [k] fo all k 2 N follows fom th fact that x [] 2B is lmntwis non-ngativ and ach is a non-ngativ matix fo all 2. W also nd to show that x [k] cannot b th zo vcto. Th agumnt is as follows: assum that if possibl x [ ] ¼ wh x [ ] 6¼ and 2 is th cunt obsvd vnt. It follows fom th constuction of th tansition matics that 8q i 2 Q, x ½ Š i 6¼ implis that ith (q i, ) is undfind o p(q i, ) ¼ ". In ith cas, it is impossibl to obsv th vnt with th cunt occupancy stimat x [ ], which is a contadiction. Finally, w nd to pov th lmntwis upp bound of on x[k]. W not that that x ½kŠ j is th sum of th total conditional pobabilitis of all stings u 2? initiating fom stat q i 2 Q (sinc 8j, x ½Š j ¼ ij ) that tminat on th stat q j 2 Q and satisfy pðuþ ¼! ð36þ It follows that x ½kŠ j 5 x ½Š ½I ð ÞŠ j sinc th ight-hand sid is th sum of conditional pobabilitis of all stings that go to q j fom q i, ispctiv of th obsvability. Hnc w conclud: kx ½kŠ k 5 kx ½Š ½I ð ÞŠ k 5 which complts th poof. Rmak 4.2: It follows fom Lmma 4. that th ntangld stats blong to a compact subst of R CARD (Q). Dfinition 4.4 (Entangld stat st): Fo a givn G ¼ðQ,,,,, CÞ and p, th ntangld stat st Q F R CARD (Q) n is th st of all possibl makings of th FNO initiatd at any of th pu stats x [] 2B. 4.3 An illustativ xampl of stat ntanglmnt W consid th plant modl as psntd in th lfthand plat of Figu 2. Th finit-stat plant modl with th unobsvabl tansition (makd in d dashd) along with th constuctd Pti nt obsv is shown in Figu 2. Th vnt occunc pobabilitis assumd a shown in Tabl 2 and th tansition pobability matix P is shown in Tabl 3. Givn ¼., w apply Algoithm B.3 to obtain: 2 3 :2 I ð ÞPðÞ ¼ a MODEL a a, a.2.2 a FRACTION NET OBSERVER a ð37þ Figu 2. Undlying plant and Pti nt obsv. Availabl in colou onlin. Tabl 2. Evnt occunc pobabilitis. a Tabl 3. Tansition pobability matix

13 Intnational Jounal of Contol 469 Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 Th ac wights a thn computd fo th FNO and th sult is shown in th ight-hand plat of Figu 2. Not that th acs in bold d a th ons with factional wights in this cas and all oth ac wights a unity. Th st of tansitions matics! a now computd fom Algoithm 4.2 as ¼ 6 4 7, ¼ a ¼ W consid th diffnt obsvation squncs,, a assuming that th initial stat in th undlying plant is in ach cas (i.. th initial making of th FNO is givn by ¼ [] T. Th final makings (i.. th ntangld stats) a givn by :2 :24 ¼ , :2 ¼ , a ¼ :2 5 ð38þ Not that whil in th cas of th Pti nt obsv, w could only say that QðÞ ¼fq, q 2 g, fo th FNO, w hav an stimat of th cost of occupying ach stat (.2 and.24, spctivly, fo th fist cas). Nxt w consid a slightly modifid undlying plant with th vnt occunc pobabilitis as tabulatd in Tabl 4. Th modifid plant (dnotd as Modl 2) is shown in th ight-hand plat of Figu 3. Th two modls a simulatd with th initial pu stat st to [ ] in ach cas. W not that th numb of ntangld stats in th cous of simulatd opation mo than doubls fom 6 fo Modl to 25 fo Modl 2 (Figu 4). In th simulation, ntangld stat vctos w distinguishd with a tolanc of on th max nom. Tabl 4. Evnt occunc pobabilitis fo Modl 2. a Maximisation of intgatd instantanous masu Dfinition 4.5 Instantanous chaactistic: Givn a plant GðÞ ¼ðQ,,, ð Þ,, CÞ, th instantanous chaactistic ^ðtþ is dfind as a function of plant opation tim t 2 [, ) as follows: ^ðtþ ¼v i ð39þ wh q i 2 Q is th stat occupid at tim t Dfinition 4.6 Instantanous masu fo pfctly obsvabl plants: Givn a plant GðÞ ¼ðQ,,, ð Þ,,, CÞ, th instantanous masu ( ^ ðtþ) is dfind as a function of plant opation tim t 2 [, ) as follows: ^ ðtþ ¼hðtÞ, l i ð4þ wh 2Bcosponds to th stat that G is obsvd to occupy at tim t (f to Equation (2)) and l is th nomalisd languag masu vcto fo th undlying plant G with unifom tmination pobability. Nxt, w show that th optimal contol algoithms psntd in Sction 3, fo pfctly obsvabl No. of ntangld stats ncountd a a Modl Modl 2 Figu 3. Undlying modls to illustat ffct of unobsvability on th cadinality of th ntangld stat st. Modl Modl No. of obsvd vnts, a Figu 4. Total numb of distinct ntangld stats ncountd as a function of th numb of obsvation ticks, i.. th numb of obsvd vnts. a a a,

14 47 I. Chattopadhyay and A. Ray Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 situations, can b intptd as maximising th xpctation of th tim-intgatd instantanous masu fo th finit-stat plant modl und considation (Figu 5). Poposition 4.3: Fo th unsupvisd plant G ¼ðQ,,,,, CÞ with all tansitions obsvabl at th supvisoy lvl, lt G? b th optimally supvisd plant and G # b obtaind by abitaily disabling contollabl tansitions. Dnoting th instantanous masus fo G? and G # by ^? ðtþ and ^# ðtþ fo som unifom tmination pobability 2 (, ) spctivly, w hav E ^? ðþd = E ^ # ðþd 8t 2½, Þ, 8 2ð, Þ ð4þ wh t is th plant opation tim and E() dnots th xpctd valu of th xpssion within bacs. Poof: Assum that th stochastic tansition pobability matix fo an abitay finit-stat plant modl b dnotd by P and dnot th Csao limit as: C¼lim k k! j¼ j. Dnoting th final stabl stat k pobability vcto as p i, wh th plant is assumd to initiat opation in stat q i, w claim that p i j ¼C ij which follows immdiatly fom noting that if th initiating stat is q i thn ð p i Þ T ¼ " ith lmnt P lim k k! k j¼ j i.. (p i ) T is th ith ow of C(). Hnc, w hav E ^ðþd ¼ Eð^ðÞ Þd ¼ thp i, vi ¼tl i ðnot : ¼ Þ wh finit numb of stats guaant that th xpctation opato and th intgal can b xchangd. Rcalling that optimal supvision lmntwis maximiss th languag masu vcto l, w conclud that E ^? ðþd = E ^ # ðþd 8t 2½, Þ ð42þ wh th ^ðtþ fo th plant configuations G? and G # is dnotd as ^? and ^ #, spctivly. Noting that th constuction of th Pti nt obsv (Algoithm B.4) implis that in th cas of pfct obsvation, ach tansition lads to xactly on plac, w conclud that th instantanous masu is givn by ^ ðtþ ¼l i wh th cunt stat at tim t is q i ð43þ Figu 5. Tim intgals of instantanous masu and instantanous chaactistic vs. opation tim. Futhmo, w call fom Coollay 2. that Cl ¼ Cv ¼) Eð^ ðtþþ ¼ Eð^ðtÞ Þ8t 2½, Þ ð44þ which lads to th following agumnt: E ^? ðþd =E ^ # ðþd 8t2½, Þ ¼) Eð^? ðþþd = E ^ # ðþ d 8t2½,Þ ¼) ¼)E E ^? ðþ d = E ^ # ðþ d 8t2½,Þ,82ð,Þ ^ # ðþd 8t2½,Þ,82ð,Þ ^? ðþd =E This complts th poof. Nxt w fomalis a pocdu of implmnting an optimal supvision policy fom a knowldg of th optimal languag masu vcto fo th undlying plant. 4.5 Th optimal contol algoithm Fo any finit-stat undlying plant GðÞ ¼ðQ,,, ð Þ,, CÞ and a spcifid unobsvability map p, it is possibl to dfin a pobabilistic tansition systm as a possibly infinit-stat gnalisation of PFSA which w dnot as th ntangld tansition systm cosponding to th undlying plant and th spcifid unobsvability map. In dfining th ntangld tansition systm (Dfinition 4.7), w us a simila fomalism as statd in Sction 2., with th xcption of dopping th last agumnt fo contollability spcification in Equation (7). Contollability nds to handld spaatly to addss th issus of

15 Intnational Jounal of Contol 47 Downloadd By: [Chattopadhyay, Ishanu] At: 6:24 8 Fbuay 2 patial contollability aising as a sult of patial obsvation. Dfinition 4.7 (Entangld tansition systm): Fo a givn plant GðÞ ¼ðQ,,, ð Þ,, CÞ and an unobsvability map p, th ntangld tansition systm E ðg,pþ ¼ðQ F,, D, ~ E, E Þ is dfind as: () Th tansition map D : Q F?! Q F is dfind as: 8 2 Q F, Dð,!Þ ¼ Y m i wh! ¼ m (2) Th vnt gnation pobabilitis ~ E : Q F?!½, Š a spcifid as: i¼card ~ E ð, Þ ¼ X ðqþ ð ÞN ð i Þ ~ðq i, Þ i¼ (3) Th chaactistic function E : Q F! [, ] is dfind as: E () ¼h, si Rmak 4.3: Th dfinition of ~ E is consistnt in th sns: X 8 2 Q F, ~ E ð; Þ ¼¼ X Nð i Þð Þ ¼ 2 i implying that if Q F is finit thn E (G, p) is a pfctly obsvabl tminating modl with unifom tmination pobability. Poposition 4.4: Th nomalisd languag masu E ðþ fo th stat 2 Q F of th ntangld tansition systm E ðg,pþ ¼ðQ F,, D, ~ E, E Þ can b computd as follows: E ðþ ¼h, l i ð45þ wh l is th languag masu vcto fo th undlying tminating plant GðÞ ¼ðQ,,, ð Þ,, CÞ with unifom tmination pobability. Poof: W fist comput th masu of th pu stats BQ F of E ðg,pþ ¼ðQ F,, D, ~ E, E Þ dnotd by th vcto l E. Sinc vy sting gnatd by th phantom automaton is compltly unobsvabl, it follows that th masu of th mpty sting " fom any stat 2B is givn by [I ( )p()] s. Lt cospond to th stat q i 2 Q in th undlying plant. Thn th masu of th st of all stings gnatd fom 2B having at last on obsvabl tansition in th undlying plant is givn by X ð Þ I ð ÞPðÞ PðÞ l E j ij j ð46þ which is simply th masu of th st of all stings of th fom!! 2 wh p(!! 2 ) ¼ p(! 2 ). It thfo follows fom th additivity of masus that l E I ¼ð Þ ð ÞPðÞ PðÞ þ I ð ÞPðÞ v ) l E "I # ¼ ð Þ I ð ÞPðÞ PðÞ I ð ÞPðÞ v ) l E ¼ I ð Þ v ¼ l l E ð47þ which implis that fo any pu stat 2B, w hav E ðþ ¼h, l i. Th gnal sult thn follows fom th following lina lation aising fom th dfinitions of ~ E and E: 8 2B, 8k 2 R, E ðkþ ¼kE ðþ ð48þ This complts th poof. Dfinition 4.8 (Instantanous chaactistic fo ntangld tansition systms): Givn an undlying plant GðÞ ¼ðQ,,, ð Þ,, CÞ and an unobsvability map p, th instantanous chaactistic ^ E ðtþ fo th cosponding ntangld tansition systm E ðg,pþ ¼ðQ F,, D, ~ E, E Þ is dfind as a function of plant opation tim t 2 [, ) as follows: ^ E ðtþ ¼hðtÞ, vi ð49þ wh (t) is th ntangld stat occupid at tim t Dfinition 4.9 (Instantanous masu fo patially obsvabl plants): Givn an undlying plant GðÞ ¼ðQ,,, ð Þ,, CÞ and an unobsvability map p, th instantanous masu ( ^ ðtþ) is dfind as a function of plant opation tim t 2 [, ) as follows: ^ ðtþ ¼hðtÞ, l E i ð5þ wh 2 Q E is th ntangld stat at tim t and l E is th nomalisd languag masu vcto fo th cosponding ntangld tansition systm E ðg,pþ ¼ðQ F,, D, ~ E, E Þ. Coollay 4. (Coollay to Poposition 4.4): Fo a givn plant GðÞ ¼ðQ,,, ð Þ,, CÞ and an unobsvability map p, th instantanous masu ^ : ½, Þ! ½, Š is givn by ^ ðtþ ¼hðtÞ, l i ð5þ