Linear Program for Partially Observable Markov Decision Processes. MS&E 339B June 9th, 2004 Erick Delage

Transcription

1 Linea Pogam fo Patiall Obsevable Makov Decision Pocesses MS&E 339B June 9th 2004 Eick Delage

2 Intoduction Patiall Obsevable Makov Decision Pocesses Etension of the Makov Decision Pocess to a wold with uncetaint about the cuent state Double objective decisions: Confim state vs. Reduce epected immediate cost Fomulation as a continuous state space Makov Decision Pocess Methods to appoimate the Value function Value function appoimation using Linea Pogamming Etension of LP appoach to continuous state-space MDP Application to POMDP Implementation fo small poblems

3 Patiall Obsevable Makov Decision Pocess Descibed b: a set of states a set of actions a set of obsevations a set of tansition pobabilities a set of obsevation pobabilities a set of costs a discount facto an initial belief the hbid DP Value function S { s1 s2... ss} A { a1 a2... aa} Z { z1 z2... zz} si a sj p sj si a O zi a sj p zi sj R gs:s A α 01 [ ] p0 s J p min{ Ε [ α g k µ p µ p U 0 ~p0 k 0 k k k+ 1 ]}

4 POMDP 2

5 POMDP 3 In bief: MDP with uncetaint about the state MDP is a specific case of POMDP Applications: Robot Navigation A obot gathes infomation about its position and decides of movements to get to destination A obot gathes infomation about a moving objects position and decides of pope movements to get close to it. Dialogue Modeling A machine needs to ask the ight questions to lean what is a use s intentions Limits of Scalabilit: Cuse of dimensionalit : state space is the S -1 dimension pobabilit simple Cuse of histo : the value function gets moe comple with depth of hoizon

6 POMDP as a Continuous state Makov Decision Pocess POMDP can be fomulated : a conve state space B b s p s a set of actions A { a1 a2... aa} a set of tansition pobabilities bi a bj p bjbi a a cost function gb: B R a discount facto α 01 [ ] an initial belief b0 Belief Space: Discete pobabilit distibution ove the set of envionment state S S -dimensional pobabilit simple which vetices ae beliefs whee the state is known with 100% cetaint Cost function on the belief space is the epected immediate cost g b b a Ε[ gs s a] b s gs s a s S Continuous state-space Bellman s equation J b min{ Ε[ g b µ b + αj b]} π

7 Beliefs ansition pobabilities bi a bj p bjbi a p zibi a if bj s p s a s~b i s Using Bae s ule: p za p zs p s a s S p a s S p zs p s a p zs a O z a s p s a p s s a p s s S p z s a p zs a p s a p s a z p za p za

8 Some Solutions - Heuistics Most likel state Dnamic Pogam Noubakhsh et al J Ε 0 min k s s α g sk µ sk π k0 Polic will be satisfied with making most pobable the lowest cost state In fact tue immediate cost is which penalizes uncetaint Bi-Objective algoithm Cetaint vs. Rewad Cassanda et al Limits the entop level on the wa to the lowest cost state H p s N k 1 p slog2 p s aps sstem in states that ae suounded b high uncetaint J gb b Ε[ gs s] π p s s p s Jsagma p s p s agmin Ε[ H p sa z] if H p s > κ π p s a othewise

9 Some Solutions - Appoimations Gid Based Appoimaton Hauskecht 2000 N discete points in the belief space the value function evaluated at these points an intepolation function fo all othe points Vˆ B ˆ { bˆˆ 1b2...ˆ bn} Vˆ b: Bˆ R V b Int b Vˆ 1 b min{ gb b a + E[ Int b Vˆ k]} b Bˆ k+ α π b PCA & Eponential Famil PCA compession of state space Ro 2003 Assumes tajectoies fo most lage eal wold POMDP lie nea a low dimensional manifold embedded in the belief space Finds an m-dimension manifold m<<n ove which to poject the belief space Discetise the m-dimension manifold Use Gid based Appoimation ove the manifold

10 Linea Pogam Appoimation Appoach J min Ε[ g µ + αj ] µ U J min{ g µ + αε[ J ]} In the case of POMDP if we assume Z {z1z2 } is a finite set then: owads fomulating the Linea Pogam: Minimize Subject to µ U J min { g µ + α p µ J d } J µ U min { g µ + α µ U c J g µ + p Y µ p Y µ µ J } J J µ µ

11 LP Appoimation Appoach 2 Minimize Subject to ] [ 2 1 J k d c p g Y µ µ µ µ c d c d c d c d c. 2 1 Σ Y Y Y p p p µ µ µ µ µ µ µ

12 LP Appoimation Appoach 3 Minimize Subject to c [ Σ. ] g µ µ µ Computing the integals Could be computed analticall Becomes moe pactical to use c as a distibution to appoimate the integals Sampling the constaints Discete ALP concepts suggests to use c as the andom distibution of samples Numbe of constaints equied popotional to numbe of featue functions Choosing the featue functions Integal must make sense ove the pobabilit simple Since value function is concave intuition suggests using a quadatic function

13 Choosing c C is a measue of whee the LP should concentate its effots Poblem dependent fo best esults In POMDP not all beliefs ae visited C should eflect this

14 Implementation Small poblems Sstem had ~10 states ~10-D belief space Featue functions: 1000 linea constaints + C distibution of beliefs when behaving andoml Pocessing time: less than 1 minute [ 2 2L ] 0 1 s 2 s S s S S

15 Conclusion Appoimate Linea Pogam etends easil to the continuous state space Simple paametes to deal with : featue functions & c Most of pocessing done b the linea pogam solve Computation should gow lineal with numbe of featue functions Numbe of featue functions should gow linea with numbe of states S

16 Futue Wok Appl Continuous ALP to lage scale POMDPs Compae pefomances with othe techniques Confim bounds on compleit gowth Stud bounds on optimal polic pefomance Stud acceptable class of featue functions

17 Refeences Cassanda A. R. Kaelbling L. and Kuien J. A Acting unde uncetaint: Discete Baesian models fo mobile-obot navigation. In Poceedings of the IEEE/RSJ Intenational Confeence on Intelligent Robotic Sstems IROS. De Faias D.P. Van Ro B he Linea Pogamming Appoach to Appoimate Dnamic Pogamming. In Opeations Reseach 51:6. Hauskecht M Value-function appoimations fo patiall obsevable Makov decision pocesses. Jounal of Atificial Intelligence Reseach 13: Hauskecht M. and Kveton B Linea pogam appoimations fo factoed continuous-state Makov decision pocesses. In Neual Infomation Pocessing Sstems. Pineau J. Godon G. hun S Point-based value iteation: An antime algoithm fo POMDPs. Intenational Joint Confeence on Atificial Intelligence Noubakhsh I. Powes R. and Bichfield S.1995 Devish an office-navigation obot. AI Magazine 162: Ro N Finding Appoimate POMDP Solutions hough Belief Compession. Ph.D. thesis Canegie Mellon Univesit Pittsbugh PA.