A Partition-Based First-Order Probabilistic Logic to Represent Interactive Beliefs

Transcription

1 A Partition-Based First-Order Probabilistic Logic to Represent Interactive Beliefs Alessandro Panella and Piotr Gmytrasiewicz Fifth International Conference on Scalable Uncertainty Management Dayton, OH October 10, 2011 Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

2 Outline 1 Quick Look 2 Introduction The Problem Related Work 3 Proposed Formalization 0-th Level Beliefs 1st Level Beliefs n-th Level Beliefs 4 Conclusion 5 Bibliography Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

3 Contribution Quick Look Formalization of a theoretical framework that allows to compactly represent interactive beliefs Probability theory (First-Order) Logic Maximum Entropy Main idea: recursive partitioning of the belief simplices... (a) (b) (c) (d) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

4 Introduction The Problem Stochastic Planning The need for compact representations: Use of first-order logic: Describe sets of states Capture regularities Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

5 Introduction The Problem Stochastic Planning The need for compact representations: Use of first-order logic: Describe sets of states Capture regularities Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

6 Introduction The Problem Stochastic Planning An Example n n grid world Actions: UP, DOWN, LEFT, RIGHT Probabilistic transition function: For every location, P(succeed) =.9 GO RIGHT GO DOWN Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

7 Introduction The Problem Stochastic Planning An Example n n grid world Actions: UP, DOWN, LEFT, RIGHT Probabilistic transition function: For every location, P(succeed) =.9 GO RIGHT GO DOWN Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

8 Introduction The Problem Stochastic Planning An Example n n grid world Actions: UP, DOWN, LEFT, RIGHT Probabilistic transition function: For every location, P(succeed) =.9 GO RIGHT GO DOWN Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

9 Interactive Settings Introduction The Problem Representation needs even more stringent Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

10 Interactive Settings Introduction The Problem Representation needs even more stringent Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

11 Introduction The Problem Interactive Settings Representation needs even more stringent Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

12 Introduction The Problem Interactive Settings Representation needs even more stringent Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

13 Introduction The Problem (Finitely Nested) Interactive POMDPs Level-n belief b i,n (IS i,n ), where IS i,0 = S IS i,1 = S (IS j,0 ). IS i,n = S (IS j,n 1 ) Value function of (I-)POMDPs is Piecewise linear and convex. Divides the simplex into behavior-equivalent partitions. (0, 1) s 1 (1, 0) (0, 0) s 2 From Kaelbling et al. (1998) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

14 Introduction The Problem (Finitely Nested) Interactive POMDPs Level-n belief b i,n (IS i,n ), where IS i,0 = S IS i,1 = S (IS j,0 ). IS i,n = S (IS j,n 1 ) Value function of (I-)POMDPs is Piecewise linear and convex. Divides the simplex into behavior-equivalent partitions. (0, 1) s 1 (1, 0) (0, 0) s 2 From Kaelbling et al. (1998) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

15 Introduction Related Work Related Work First Order Probabilistic Languages: Seminal theoretical work (Nilsson, 1986; Halpern, 1989) Recent practical approaches: BLOG (Milch et al., 2005), Markov Logic (Richardson and Domingos, 2006),... Relational stochastic planning: Relational MDPs (Boutilier et al., 2001; Sanner and Boutilier, 2009) Relational POMDPs (Sanner and Kersting, 2010; Wang and Khardon, 2010) Belief hierarchies Extensive treatment in Game Theory, starting from Bayesian Games (Harsanyi, 1967; Aumann, 1999) Probabilistic modal logics (Fagin and Halpern, 1994; Shirazi and Amir, 2008) Interactive POMDPs (Gmytrasiewicz and Doshi, 2005) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

16 Introduction Related Work Related Work First Order Probabilistic Languages: Seminal theoretical work (Nilsson, 1986; Halpern, 1989) Recent practical approaches: BLOG (Milch et al., 2005), Markov Logic (Richardson and Domingos, 2006),... Relational stochastic planning: Relational MDPs (Boutilier et al., 2001; Sanner and Boutilier, 2009) Relational POMDPs (Sanner and Kersting, 2010; Wang and Khardon, 2010) Belief hierarchies Extensive treatment in Game Theory, starting from Bayesian Games (Harsanyi, 1967; Aumann, 1999) Probabilistic modal logics (Fagin and Halpern, 1994; Shirazi and Amir, 2008) Interactive POMDPs (Gmytrasiewicz and Doshi, 2005) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

17 Introduction Related Work Related Work First Order Probabilistic Languages: Seminal theoretical work (Nilsson, 1986; Halpern, 1989) Recent practical approaches: BLOG (Milch et al., 2005), Markov Logic (Richardson and Domingos, 2006),... Relational stochastic planning: Relational MDPs (Boutilier et al., 2001; Sanner and Boutilier, 2009) Relational POMDPs (Sanner and Kersting, 2010; Wang and Khardon, 2010) Belief hierarchies Extensive treatment in Game Theory, starting from Bayesian Games (Harsanyi, 1967; Aumann, 1999) Probabilistic modal logics (Fagin and Halpern, 1994; Shirazi and Amir, 2008) Interactive POMDPs (Gmytrasiewicz and Doshi, 2005) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

18 Grid World Example Proposed Formalization 0-th Level Beliefs n n grid Agent i tagging a moving target j Uncertainty about target s position: predicate jpos(x, y) Auxiliary deterministic predicates: geq(x, k) x k leq(x, k) x k Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

19 Level-0 Belief Base Proposed Formalization 0-th Level Beliefs i s belief about the state of the world B i,0 = φ 1, α 1 : : φ m, α m φ 0 S ψ 0 ψ 1 ψ 2 φ 1 φ k s are arbitrary sentences in predicate logic, and α k [0, 1]; ψ s are the induced partitions (Ψ, 2 Ψ, p i,0 ) Only partial specification of distribution To obtain unique distribution: Maximum Entropy (max-ent): ( max ) p i,0 (S(ψ)) log p i,0 (S(ψ)) p i,0 ψ Ψ B Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

20 Proposed Formalization Grid World Example (cont d) 0-th Level Beliefs Assume agent interested in horizontal position of target w.r.t. center: B i,0 = x, y(jpos(x, y) leq(x, n/2 )), 0.8 x, y(jpos(x, y) geq(x, n/2 )), 0.5 S ψ 2 ψ 1 ψ 0 φ 0 φ 1 In this case, unique distribution p i,0 = (0.5, 0.3, 0.2). Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

21 Level-1 Belief Base Proposed Formalization 1st Level Beliefs i s belief about j s belief B i,1 = φ j,0 1, α1 : : φ j,0 m, α m (1) φ j,0 k is of the form P j (φ) β, {<,, =,, >}, β [0, 1]; The sentences φ j,0 k induce a partitioning on j s L-0 belief simplex; Ψ B Ψ j,0 B pi,1 ψ Ψ j,0 B Φ B Φ j,0 B S (Ψ B) i (a) (b) (c) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

22 Proposed Formalization Grid World Example (cont d) 1st Level Beliefs Target j models agent i s beliefs about j s position B j,1 = Pi(φ0) 0.4, 0.4 P i(φ 1) > 0.5, 0.7 State of the world: S i's L0 simplex: ψ 2 ψ 2 p i(φ 0)=0.4 ψ i,0 2 ψ 1 ψ 0 ψ i,0 0 ψ i,0 1 φ 0 φ1 ψ p 0 i(φ 1)=0.5 ψ 1 φ i,0 0 φ i,0 1 Unique consistent distribution p j,1 = (0.3, 0.1, 0.6). Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

23 Level-n Belief Base Proposed Formalization n-th Level Beliefs Generalize the procedure to an arbitrary nesting level: i s beliefs about j s beliefs about i s beliefs about... B n i = φ j,n 1 1, α 1 : : φ j,n 1 m α m φ j,n 1 k is of the form P j (φ i,n 2 ) β; The core idea is that the partitions at level l 1 constitute the vertices of the simplex at level l... (a) (b) (c) (d) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

24 Level-n Belief Base Proposed Formalization n-th Level Beliefs Generalize the procedure to an arbitrary nesting level: i s beliefs about j s beliefs about i s beliefs about... B n i = φ j,n 1 1, α 1 : : φ j,n 1 m α m φ j,n 1 k is of the form P j (φ i,n 2 ) β; The core idea is that the partitions at level l 1 constitute the vertices of the simplex at level l... (a) (b) (c) (d) Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

25 Proposed Formalization Grid World Example (cont d) n-th Level Beliefs Agent i models target j s beliefs about i s beliefs about j s position B i,2 = Pj(Pi(φ0) 0.4) < 0.4, 0.2 P j(p i(φ 1) > 0.5) < 0.7, 0.6 State of the world: i's L0 simplex: ψ 2 j's L1 simplex: S ψ i,1 2 ψ 0 ψ 1 ψ 2 pi(φ0) =0.4 ψ i,0 0 ψ i,0 2 ψ i,0 1 pi(φ i,0 0 )=0.6 ψ j,1 3 ψ j,1 0 ψ j,1 2 ψ j,1 1 φ 0 φ1 ψ pi(φ1) =0.5 0 ψ 1 ψ i,0 pi(φ i,0 0 ψ i,0 0 )=0.7 1 φ i,0 0 φ i,0 1 φ j,1 0 φ j,1 1 Max-ent optimization gives p i,2 = (0.48, 0.32, 0.08, 0.12). Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

26 Conclusion Conclusion and Future Work Contribution: Formalization of a FOPL that allows to represent interactive beliefs through recursively partitioning the belief simplices. Future Work: Practical implementation of (subset of) the proposed formalism; Extension of existing approaches to interactive beliefs semantics; E.g. BLOG, Markov Logic, Relational Probabilistic Models,... Embed the formalism in a (multi-agent) stochastic planning algorithm Piecewise linearity of value functions in (I-)POMDPs makes the proposed approach promising; Extend approaches like Sanner et al., 2010 (Sanner and Kersting, 2010). Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

27 Conclusion Conclusion and Future Work Contribution: Formalization of a FOPL that allows to represent interactive beliefs through recursively partitioning the belief simplices. Future Work: Practical implementation of (subset of) the proposed formalism; Extension of existing approaches to interactive beliefs semantics; E.g. BLOG, Markov Logic, Relational Probabilistic Models,... Embed the formalism in a (multi-agent) stochastic planning algorithm Piecewise linearity of value functions in (I-)POMDPs makes the proposed approach promising; Extend approaches like Sanner et al., 2010 (Sanner and Kersting, 2010). Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

28 Conclusion The end. Any questions? Thank you! Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18

29 Bibliography Bibliography R. J. Aumann. Interactive epistemology II: Probability. International Journal of Game Theory, 28(3): , C. Boutilier, R. Reiter, and B. Price. Symbolic Dynamic Programming for First-Order MDPs. In IJCAI 01, pages Morgan Kaufmann, R. Fagin and J. Y. Halpern. Reasoning about knowledge and probability. J. ACM, 41: , March ISSN P. J. Gmytrasiewicz and P. Doshi. A framework for sequential planning in multi-agent settings. Journal of Artificial Intelligence Research, 24:24 49, J. Y. Halpern. An analysis of first-order logics of probability. In Proceedings of the 11th international joint conference on Artificial intelligence - Volume 2, pages , San Francisco, CA, USA, Morgan Kaufmann Publishers Inc. J. C. Harsanyi. Games with Incomplete Information Played by Bayesian Players, I-III. Part I. The Basic Model. Management Science, 14(3): , doi: / L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101:99 134, B. Milch, B. Marthi, S. J. Russell, D. Sontag, D. L. Ong, and A. Kolobov. Blog: Probabilistic models with unknown objects. In Probabilistic, Logical and Relational Learning, N. J. Nilsson. Probabilistic logic. Artif. Intell., 28:71 88, February ISSN doi: / (86) M. Richardson and P. Domingos. Markov logic networks. Mach. Learn., 62: , February ISSN doi: /s S. Sanner and C. Boutilier. Practical solution techniques for first-order MDPs. Artif. Intell., 173: , April ISSN doi: /j.artint S. Sanner and K. Kersting. Symbolic dynamic programming for first-order POMDPs. In AAAI 10, pages , A. Shirazi and E. Amir. Factored models for probabilistic modal logic. In Proceedings of the 23rd national conference on Artificial intelligence - Volume 1, pages AAAI Press, ISBN C. Wang and R. Khardon. Relational partially observable MDPs. In AAAI 10, pages , Panella and Gmytrasiewicz (CS Dept. - UIC) A FOPL for Interactive Beliefs October 10, / 18