Chapter 17 Planning Based on Model Checking

Transcription

1 Lecture slides for Automated Planning: Theory and Practice Chapter 17 Planning Based on Model Checking Dana S. Nau University of Maryland 1:19 PM February 9, 01 1

2 Motivation c a b Actions with multiple possible outcomes Action failures» e.g., gripper drops its load Exogenous events» e.g., road closed Nondeterministic systems are like Markov Decision Processes (MDPs), but without probabilities attached to the outcomes Useful if accurate probabilities aren t available, or if probability calculations would introduce inaccuracies a c b grasp(c) Intended outcome a b Unintended outcome

3 Nondeterministic Systems Nondeterministic system: a triple Σ = (S, A, γ) S = finite set of states A = finite set of actions γ: S A s Like in the previous chapter, the book doesn t commit to any particular representation It only deals with the underlying semantics Draw the state-transition graph explicitly Like in the previous chapter, a policy is a function from states into actions π: S A Notation: S π = {s (s,a) π} In some algorithms, we ll temporarily have nondeterministic policies» Ambiguous: multiple actions for some states π: S A, or equivalently, π S A We ll always make these policies deterministic before the algorithm terminates 3

4 Example Robot r1 starts at location l1 Objective is to get r1 to location l4 π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 4

5 Example Robot r1 starts at location l1 Objective is to get r1 to location l4 π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 5

6 Example Robot r1 starts at location l1 Objective is to get r1 to location l4 π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 6

7 Execution Structures Execution structure for a policy π: s The graph of all of π s execution paths Notation: Σ π = (Q,T) Q S T S S π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 7

8 Execution Structures Execution structure for a policy π: The graph of all of π s execution paths Notation: Σ π = (Q,T) Q S T S S s π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 8

9 Execution Structures Execution structure for a policy π: s The graph of all of π s execution paths Notation: Σ π = (Q,T) Q S T S S π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 9

10 Execution Structures Execution structure for a policy π: The graph of all of π s execution paths Notation: Σ π = (Q,T) Q S T S S s π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 10

11 Execution Structures Execution structure for a policy π: The graph of all of π s execution paths Notation: Σ π = (Q,T) Q S T S S π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 11

12 Execution Structures Execution structure for a policy π: The graph of all of π s execution paths Notation: Σ π = (Q,T) Q S T S S π 1 = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4))} π = {(, move(r1,l1,l)), (s, move(r1,l,l3)), (, move(r1,l3,l4)), (, move(r1,l3,l4))} π 3 = {(, move(r1,l1,l4))} 1

13 Types of Solutions Weak solution: at least one execution path reaches a goal s0 a0 a1 s a Strong solution: every execution path reaches a goal s0 a0 a1 s a a3 Strong-cyclic solution: every fair execution path reaches a goal Don t stay in a cycle forever if there s a state-transition out of it s0 a0 a1 a3 s a 13

14 Backward breadth-first search Finding Strong Solutions StrongPreImg(S) = {(s,a) : γ(s,a), γ(s,a) S} all state-action pairs for which all of the successors are in S PruneStates(π,S) = {(s,a) π : s S} S is the set of states we ve already solved keep only the state-action pairs for other states MkDet(π') π' is a policy that may be nondeterministic remove some state-action pairs if necessary, to get a deterministic policy 14

15 Example π = failure π' = S π' = S g S π' = {} 15

16 Example π = failure π' = S π' = S g S π' = {} π'' PreImage = {(,move(r1,l3,l4)), (,move(r1,l5,l4))} 16

17 Example π = failure π' = S π' = S g S π' = {} π'' PreImage = {(,move(r1,l3,l4)), (,move(r1,l5,l4))} π π' = π' π' U π'' = {(,move(r1,l3,l4)), (,move(r1,l5,l4))} 17

18 π = Example π' = {(,move(r1,l3,l4)), (,move(r1,l5,l4))} S π' = {,} S g S π' = {,,} 18

19 π = Example π' = {(,move(r1,l3,l4)), (,move(r1,l5,l4))} s S π' = {,} S g S π' = {,,} PreImage {(s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4)), (,move(r1,l4,l3)), (,move(r1,l4,l5))} π'' {(s,move(r1,l,l3))} π π' = {(,move(r1,l3,l4)), (,move(r1,l5,l4))} π' {(s,move(r1,l,l3), (,move(r1,l3,l4)), (,move(r1,l5,l4))} 19

20 Example π = {(,move(r1,l3,l4)), (,move(r1,l5,l4))} π' = {(s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} S π' = {s,,} S g S π' = {s,,,} s 0

21 Example π = {(,move(r1,l3,l4)), (,move(r1,l5,l4))} π' = {(s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} s S π' = {s,,} S g S π' = {s,,,} π'' {(,move(r1,l1,l))} π π' = {(s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} π' {(,move(r1,l1,l)), (s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} 1

22 Example π = {(s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} s π' = {(,move(r1,l1,l)), (s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} S π' = {,s,,} S g S π' = {,s,,,}

23 Example π = {(s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} s π' = {(,move(r1,l1,l)), (s,move(r1,l,l3)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} S π' = {,s,,} S g S π' = {,s,,,} S 0 S g S π' MkDet(π') = π' 3

24 Finding Weak Solutions Weak-Plan is just like Strong-Plan except for this: WeakPreImg(S) = {(s,a) : γ(s,a) i S } at least one successor is in S Weak Weak 4

25 π = failure π' = S π' = S g S π' = {} Example Weak Weak 5

26 Example π = failure π' = S π' = S g S π' = {} π'' = PreImage = {(,move(r1,l1,l4)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} π π' = π' π' U π'' = {(,move(r1,l1,l4)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} Weak Weak 6

27 Example π = π' = {(,move(r1,l1,l4)), (,move(r1,l3,l4)), (,move(r1,l5,l4))} S π' = {,,} S g S π' = {,,,} S 0 S g S π' MkDet(π') = π' Weak Weak 7

28 Finding Strong-Cyclic Solutions Begin with a universal policy π' that contains all state-action pairs Repeatedly, eliminate state-action pairs that take us to bad states PruneOutgoing removes state-action pairs that go to states not in S g S π» PruneOutgoing(π,S) = π {(s,a) π : γ(s,a) S S π PruneUnconnected removes states from which it is impossible to get to S g» with π' =, compute fixpoint of π' π WeakPreImg(S g S π ) 8

29 Finding Strong-Cyclic Solutions s Once the policy stops changing, If it s not a solution, return failure RemoveNonProgress removes state-action pairs that don t go toward the goal implement as backward search from the goal s6 at(r1,l6) MkDet makes sure there s only one action for each state 9

30 Example 1 π π' {(s,a) : a is applicable to s} s s6 at(r1,l6) 30

31 Example 1 π π' {(s,a) : a is applicable to s} π {(s,a) : a is applicable to s} s PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = π' RemoveNonProgress(π') =? s6 at(r1,l6) 31

32 Example 1 π π' {(s,a) : a is applicable to s} π {(s,a) : a is applicable to s} s PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = π' RemoveNonProgress(π') = as shown s6 at(r1,l6) 3

33 π Example 1 π' {(s,a) : a is applicable to s} π {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = π' RemoveNonProgress(π') = as shown MkDet( ) = either {(,move(r1,l1,l4), (s,move(r1,l,l3)), (,move(r1,l3,l4), (,move(r1,l4,l6), (,move(r1,l5,l4)} or {(,move(r1,l1,l), (s,move(r1,l,l3)), (,move(r1,l3,l4), (,move(r1,l4,l6), (,move(r1,l5,l4)} s s6 at(r1,l6) 33

34 Example : no applicable actions at π π' {(s,a) : a is applicable to s} s s6 at(r1,l6) 34

35 Example : no applicable actions at π π' {(s,a) : a is applicable to s} s π {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = s6 at(r1,l6) 35

36 Example : no applicable actions at π π' {(s,a) : a is applicable to s} s π {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' s6 at(r1,l6) 36

37 Example : no applicable actions at π π' {(s,a) : a is applicable to s} s π {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = as shown s6 at(r1,l6) 37

38 Example : no applicable actions at π π' {(s,a) : a is applicable to s} s π {(s,a) : a is applicable to s} PruneOutgoing(π',S g ) = π' PruneUnconnected(π',S g ) = as shown π' as shown s6 at(r1,l6) 38

39 Example : no applicable actions at π' as hown π π' PruneOutgoing(π',S g ) = π' s PruneUnconnected(π',S g ) = π' so π = π' RemoveNonProgress(π') = s6 at(r1,l6) 39

40 Example : no applicable actions at π' shown π π' PruneOutgoing(π',S g ) = π' s PruneUnconnected(π'',S g ) = π' so π = π' RemoveNonProgress(π') = as shown MkDet(shown) = no change s6 at(r1,l6) 40

41 Planning for Extended s Here, extended means temporally extended Constraints that apply to some sequence of states Examples: want to move to l3, and then to l5 want to keep going back and forth between l3 and l5 wait wait wait move(r1,l,l1) wait 41

42 Planning for Extended s Context: the internal state of the controller Plan: (C, c 0, act, ctxt) C = a set of execution contexts c 0 is the initial context act: S C A ctxt: S C S C Sections 17.3 extends the ideas in Sections 17.1 and 17. to deal with extended goals We ll skip the details 4