Duality in Probabilistic Automata

Transcription

1 Duality in Probabilistic Automata Chris Hundt Prakash Panangaden Joelle Pineau Doina Precup Gavin Seal McGill University MFPS May 2006 Genoa p.1/40

2 Overview We have discovered an - apparently - new kind of duality for automata. MFPS May 2006 Genoa p.2/40

3 Overview We have discovered an - apparently - new kind of duality for automata. Special case of this construction known since 1962 to Brzozowski. MFPS May 2006 Genoa p.2/40

4 Overview We have discovered an - apparently - new kind of duality for automata. Special case of this construction known since 1962 to Brzozowski. Works for probabilistic automata. MFPS May 2006 Genoa p.2/40

5 Overview We have discovered an - apparently - new kind of duality for automata. Special case of this construction known since 1962 to Brzozowski. Works for probabilistic automata. Seems interesting for learning and planning. MFPS May 2006 Genoa p.2/40

6 Overview We have discovered an - apparently - new kind of duality for automata. Special case of this construction known since 1962 to Brzozowski. Works for probabilistic automata. Seems interesting for learning and planning. Could be connected to duality in control theory, Pontryagin duality or general concrete dualities. MFPS May 2006 Genoa p.2/40

7 Overview We have discovered an - apparently - new kind of duality for automata. Special case of this construction known since 1962 to Brzozowski. Works for probabilistic automata. Seems interesting for learning and planning. Could be connected to duality in control theory, Pontryagin duality or general concrete dualities. We are not sure about the right categorical setting. MFPS May 2006 Genoa p.2/40

8 Deterministic Automata A = (Q, Σ, P, δ, γ): a deterministic finite automaton. Q is the set of states, Σ an input alphabet (actions), P is a set of propositions. MFPS May 2006 Genoa p.3/40

9 Deterministic Automata A = (Q, Σ, P, δ, γ): a deterministic finite automaton. Q is the set of states, Σ an input alphabet (actions), P is a set of propositions. δ : Q Σ Q is the state transition function. MFPS May 2006 Genoa p.3/40

10 Deterministic Automata A = (Q, Σ, P, δ, γ): a deterministic finite automaton. Q is the set of states, Σ an input alphabet (actions), P is a set of propositions. δ : Q Σ Q is the state transition function. γ : Q 2 P or γ : Q P 2 is a labeling function. MFPS May 2006 Genoa p.3/40

11 Deterministic Automata A = (Q, Σ, P, δ, γ): a deterministic finite automaton. Q is the set of states, Σ an input alphabet (actions), P is a set of propositions. δ : Q Σ Q is the state transition function. γ : Q 2 P or γ : Q P 2 is a labeling function. If P = {accept} we have ordinary deterministic finite automata. MFPS May 2006 Genoa p.3/40

12 A Simple Modal Logic Thinking of the elements of P as formulas we can use them to define a simple modal logic. We define a formula ϕ according to the following grammar: where a Σ. ϕ ::== p P (a)ϕ MFPS May 2006 Genoa p.4/40

13 A Simple Modal Logic Thinking of the elements of P as formulas we can use them to define a simple modal logic. We define a formula ϕ according to the following grammar: where a Σ. ϕ ::== p P (a)ϕ We say s = p, if p γ(s) (or γ(s, p) = T ). We say s = (a)ϕ if δ(s, a) = ϕ. MFPS May 2006 Genoa p.4/40

14 A Simple Modal Logic Thinking of the elements of P as formulas we can use them to define a simple modal logic. We define a formula ϕ according to the following grammar: where a Σ. ϕ ::== p P (a)ϕ We say s = p, if p γ(s) (or γ(s, p) = T ). We say s = (a)ϕ if δ(s, a) = ϕ. Now we define [ϕ] A = {s Q s = ϕ}. MFPS May 2006 Genoa p.4/40

15 An Equivalence Relation on Formulas We write sa as shorthand for δ(s, a). MFPS May 2006 Genoa p.5/40

16 An Equivalence Relation on Formulas We write sa as shorthand for δ(s, a). Define A between formulas as ϕ A ψ if [ϕ] A = [ψ] A. MFPS May 2006 Genoa p.5/40

17 An Equivalence Relation on Formulas We write sa as shorthand for δ(s, a). Define A between formulas as ϕ A ψ if [ϕ] A = [ψ] A. Note that this allows us to identify an equivalence class for ϕ with the set of states [ϕ] A that satisfy ϕ. MFPS May 2006 Genoa p.5/40

18 An Equivalence Relation on Formulas We write sa as shorthand for δ(s, a). Define A between formulas as ϕ A ψ if [ϕ] A = [ψ] A. Note that this allows us to identify an equivalence class for ϕ with the set of states [ϕ] A that satisfy ϕ. Note that another way of defining this equivalence relations is ϕ A ϕ := s Q.s = ϕ s = ϕ. MFPS May 2006 Genoa p.5/40

19 An Equivalence Relation on States We also define an equivalence between states in A as s 1 s 2 if for all formulas ϕ on A, s 1 = ϕ s 2 = ϕ. MFPS May 2006 Genoa p.6/40

20 An Equivalence Relation on States We also define an equivalence between states in A as s 1 s 2 if for all formulas ϕ on A, s 1 = ϕ s 2 = ϕ. The equivalence relations and are clearly closely related: they are the hinge of the duality between states and observations. MFPS May 2006 Genoa p.6/40

21 An Equivalence Relation on States We also define an equivalence between states in A as s 1 s 2 if for all formulas ϕ on A, s 1 = ϕ s 2 = ϕ. The equivalence relations and are clearly closely related: they are the hinge of the duality between states and observations. We say that A is reduced if the -equivalence classes are singletons. MFPS May 2006 Genoa p.6/40

22 An Equivalence Relation on States We also define an equivalence between states in A as s 1 s 2 if for all formulas ϕ on A, s 1 = ϕ s 2 = ϕ. The equivalence relations and are clearly closely related: they are the hinge of the duality between states and observations. We say that A is reduced if the -equivalence classes are singletons. Since there is more than just one proposition in general the relation is finer than the usual equivalence of automata theory. MFPS May 2006 Genoa p.6/40

23 A Dual Automaton Given a finite automaton A = (Q, Σ, P, δ, γ). Let T be the set of A -equivalence classes of formulas on A. MFPS May 2006 Genoa p.7/40

24 A Dual Automaton Given a finite automaton A = (Q, Σ, P, δ, γ). Let T be the set of A -equivalence classes of formulas on A. We define A = (Q, Σ, P, δ, γ ) as follows: MFPS May 2006 Genoa p.7/40

25 A Dual Automaton Given a finite automaton A = (Q, Σ, P, δ, γ). Let T be the set of A -equivalence classes of formulas on A. We define A = (Q, Σ, P, δ, γ ) as follows: Q = T = {[ϕ] A } MFPS May 2006 Genoa p.7/40

26 A Dual Automaton Given a finite automaton A = (Q, Σ, P, δ, γ). Let T be the set of A -equivalence classes of formulas on A. We define A = (Q, Σ, P, δ, γ ) as follows: Q = T = {[ϕ] A } P = Q MFPS May 2006 Genoa p.7/40

27 A Dual Automaton Given a finite automaton A = (Q, Σ, P, δ, γ). Let T be the set of A -equivalence classes of formulas on A. We define A = (Q, Σ, P, δ, γ ) as follows: Q = T = {[ϕ] A } P = Q δ ([ϕ] A, a) = [(a)ϕ] A MFPS May 2006 Genoa p.7/40

28 A Dual Automaton Given a finite automaton A = (Q, Σ, P, δ, γ). Let T be the set of A -equivalence classes of formulas on A. We define A = (Q, Σ, P, δ, γ ) as follows: Q = T = {[ϕ] A } P = Q δ ([ϕ] A, a) = [(a)ϕ] A γ ([ϕ] A, p) = [ϕ] A MFPS May 2006 Genoa p.7/40

29 The intuition We have interchanged the states and the observations or propositions; more precisely we have interchanged equivalence classes of formulas - based on the observations - with the states. We have made the states of the old machine the observations of the dual machine. MFPS May 2006 Genoa p.8/40

30 The Double Dual Now consider A = (A ), the dual of the dual. MFPS May 2006 Genoa p.9/40

31 The Double Dual Now consider A = (A ), the dual of the dual. Its states are equivalence classes of A -formulas. MFPS May 2006 Genoa p.9/40

32 The Double Dual Now consider A = (A ), the dual of the dual. Its states are equivalence classes of A -formulas. Each such class is identified with a set [ϕ ] A of A -states by which formulas in that class are satisfied, and MFPS May 2006 Genoa p.9/40

33 The Double Dual Now consider A = (A ), the dual of the dual. Its states are equivalence classes of A -formulas. Each such class is identified with a set [ϕ ] A of A -states by which formulas in that class are satisfied, and each A -state is an equivalence class of A-formulas. MFPS May 2006 Genoa p.9/40

34 The Double Dual Now consider A = (A ), the dual of the dual. Its states are equivalence classes of A -formulas. Each such class is identified with a set [ϕ ] A of A -states by which formulas in that class are satisfied, and each A -state is an equivalence class of A-formulas. Thus we can look at states in A as collections of S-formula equivalence classes. MFPS May 2006 Genoa p.9/40

35 The Double Dual 2 Let A be the double dual, and for any state s Q in the original automaton we define Sat(s) = {[ϕ] A : s = ϕ}. MFPS May 2006 Genoa p.10/40

36 The Double Dual 2 Let A be the double dual, and for any state s Q in the original automaton we define Sat(s) = {[ϕ] A : s = ϕ}. Lemma: For any s Q, Sat(s) is a state in A. MFPS May 2006 Genoa p.10/40

37 The Double Dual 2 Let A be the double dual, and for any state s Q in the original automaton we define Sat(s) = {[ϕ] A : s = ϕ}. Lemma: For any s Q, Sat(s) is a state in A. In fact all the states of the double dual have this form. MFPS May 2006 Genoa p.10/40

38 The Double Dual 2 Let A be the double dual, and for any state s Q in the original automaton we define Sat(s) = {[ϕ] A : s = ϕ}. Lemma: For any s Q, Sat(s) is a state in A. In fact all the states of the double dual have this form. Lemma: Let s = [ϕ] A Q be any state in A. Then s = Sat(s ϕ ) for some state s ϕ Q. MFPS May 2006 Genoa p.10/40

39 The Double Dual 2 Let A be the double dual, and for any state s Q in the original automaton we define Sat(s) = {[ϕ] A : s = ϕ}. Lemma: For any s Q, Sat(s) is a state in A. In fact all the states of the double dual have this form. Lemma: Let s = [ϕ] A Q be any state in A. Then s = Sat(s ϕ ) for some state s ϕ Q. The proof is by an easy induction on ϕ. MFPS May 2006 Genoa p.10/40

40 Minimality Properties If A is reduced then Sat is a bijection from Q to Q. MFPS May 2006 Genoa p.11/40

41 Minimality Properties If A is reduced then Sat is a bijection from Q to Q. The statement above can be strengthened to show that we actually have an isomorphism of automata. MFPS May 2006 Genoa p.11/40

42 Minimality Properties If A is reduced then Sat is a bijection from Q to Q. The statement above can be strengthened to show that we actually have an isomorphism of automata. If we define a notion of bisimulation we can show that a machine and its double dual are bisimilar. MFPS May 2006 Genoa p.11/40

43 Minimality Properties If A is reduced then Sat is a bijection from Q to Q. The statement above can be strengthened to show that we actually have an isomorphism of automata. If we define a notion of bisimulation we can show that a machine and its double dual are bisimilar. The minimality is, of course, due to the use of the equivalence relations in the duality. MFPS May 2006 Genoa p.11/40

44 The Nondeterministic Case Here we consider automata of the type A = (Q, Σ, P, δ : Q Σ 2 Q, γ : Q 2 P ). MFPS May 2006 Genoa p.12/40

45 The Nondeterministic Case Here we consider automata of the type A = (Q, Σ, P, δ : Q Σ 2 Q, γ : Q 2 P ). We use the same formulas but we have a different notion of satisfaction: S Q S = p s S : p γ(s) S = (a)ϕ δ(s, a) = ϕ. MFPS May 2006 Genoa p.12/40

46 The Nondeterministic Case Here we consider automata of the type A = (Q, Σ, P, δ : Q Σ 2 Q, γ : Q 2 P ). We use the same formulas but we have a different notion of satisfaction: S Q S = p s S : p γ(s) S = (a)ϕ δ(s, a) = ϕ. We define an appropriate notion of simulation and prove: A is simulated by A. MFPS May 2006 Genoa p.12/40

47 The Nondeterministic Case Here we consider automata of the type A = (Q, Σ, P, δ : Q Σ 2 Q, γ : Q 2 P ). We use the same formulas but we have a different notion of satisfaction: S Q S = p s S : p γ(s) S = (a)ϕ δ(s, a) = ϕ. We define an appropriate notion of simulation and prove: A is simulated by A. The double dual is always deterministic. MFPS May 2006 Genoa p.12/40

48 Brzozowski s Algorithm 1962 Take a NFA and just reverse all the transitions and interchange initial and final states. MFPS May 2006 Genoa p.13/40

49 Brzozowski s Algorithm 1962 Take a NFA and just reverse all the transitions and interchange initial and final states. Determinize the result. MFPS May 2006 Genoa p.13/40

50 Brzozowski s Algorithm 1962 Take a NFA and just reverse all the transitions and interchange initial and final states. Determinize the result. Reverse all the transitions again and interchange initial and final states. MFPS May 2006 Genoa p.13/40

51 Brzozowski s Algorithm 1962 Take a NFA and just reverse all the transitions and interchange initial and final states. Determinize the result. Reverse all the transitions again and interchange initial and final states. Determinize the result. MFPS May 2006 Genoa p.13/40

52 Brzozowski s Algorithm 1962 Take a NFA and just reverse all the transitions and interchange initial and final states. Determinize the result. Reverse all the transitions again and interchange initial and final states. Determinize the result. This gives the minimal DFA recognizing the same language. The intermediate step can blow up the size of the automaton exponentially before minimizing it. MFPS May 2006 Genoa p.13/40

53 Probabilistic systems Everything is discrete. MFPS May 2006 Genoa p.14/40

54 Probabilistic systems Everything is discrete. Markov Decision Processes aka Labelled Markov Processes: M = (S, A, a Aτ a : S S [0, 1]). The τ a are transition probability functions (matrices). MFPS May 2006 Genoa p.14/40

55 Probabilistic systems Everything is discrete. Markov Decision Processes aka Labelled Markov Processes: M = (S, A, a Aτ a : S S [0, 1]). The τ a are transition probability functions (matrices). Usually MDPs have rewards but I will not consider them for now. MFPS May 2006 Genoa p.14/40

56 Probabilistic systems Everything is discrete. Markov Decision Processes aka Labelled Markov Processes: M = (S, A, a Aτ a : S S [0, 1]). The τ a are transition probability functions (matrices). Usually MDPs have rewards but I will not consider them for now. We could make things continuous but that is orthogonal. MFPS May 2006 Genoa p.14/40

57 Partial Observations Partially Observable Markov Decision Processes (POMDPs). We cannot see the entire state but we can see something. MFPS May 2006 Genoa p.15/40

58 Partial Observations Partially Observable Markov Decision Processes (POMDPs). We cannot see the entire state but we can see something. In process algebra we typically take actions as not always being enabled and we observe whether actions are accepted or rejected. MFPS May 2006 Genoa p.15/40

59 Partial Observations Partially Observable Markov Decision Processes (POMDPs). We cannot see the entire state but we can see something. In process algebra we typically take actions as not always being enabled and we observe whether actions are accepted or rejected. In POMDPs we assume actions are always accepted but with each transition some propositions are true, or some boolean observables are on. MFPS May 2006 Genoa p.15/40

60 Partial Observations Partially Observable Markov Decision Processes (POMDPs). We cannot see the entire state but we can see something. In process algebra we typically take actions as not always being enabled and we observe whether actions are accepted or rejected. In POMDPs we assume actions are always accepted but with each transition some propositions are true, or some boolean observables are on. Note that the observations can depend probabilistically on the action taken and the final state. Many variations are possible. MFPS May 2006 Genoa p.15/40

61 Formal Definition of a POMDP M = (S, Σ, O, δ : S Σ S [0, 1], γ : S Σ O [0, 1]), MFPS May 2006 Genoa p.16/40

62 Formal Definition of a POMDP M = (S, Σ, O, δ : S Σ S [0, 1], γ : S Σ O [0, 1]), where S is a set of states, O is a set of observations, Σ is a set of actions, δ is the transition probability function and γ gives the observation probabilities. MFPS May 2006 Genoa p.16/40

63 Formal Definition of a POMDP M = (S, Σ, O, δ : S Σ S [0, 1], γ : S Σ O [0, 1]), where S is a set of states, O is a set of observations, Σ is a set of actions, δ is the transition probability function and γ gives the observation probabilities. MFPS May 2006 Genoa p.16/40

64 Automata with State-based Observations An automaton with stochastic observations is a quintuple E = (S, Σ, O, δ : S Σ S, γ : S O [0, 1]). Note that this has deterministic transitions and stochastic observations. MFPS May 2006 Genoa p.17/40

65 Automata with State-based Observations An automaton with stochastic observations is a quintuple E = (S, Σ, O, δ : S Σ S, γ : S O [0, 1]). Note that this has deterministic transitions and stochastic observations. A probabilistic automaton with stochastic observations is F = (S, Σ, O, δ : S Σ S [0, 1], γ : S O [0, 1]). MFPS May 2006 Genoa p.17/40

66 Simple Tests Rather than thinking of propositions and formulas we will think of observations and tests. I will look at state-based notions of observations. MFPS May 2006 Genoa p.18/40

67 Simple Tests Rather than thinking of propositions and formulas we will think of observations and tests. I will look at state-based notions of observations. Recall probabilistic automata E = (S, Σ, O, δ, γ), MFPS May 2006 Genoa p.18/40

68 Simple Tests Rather than thinking of propositions and formulas we will think of observations and tests. I will look at state-based notions of observations. Recall probabilistic automata E = (S, Σ, O, δ, γ), where δ : S Σ S [0, 1] is the transition function MFPS May 2006 Genoa p.18/40

69 Simple Tests Rather than thinking of propositions and formulas we will think of observations and tests. I will look at state-based notions of observations. Recall probabilistic automata E = (S, Σ, O, δ, γ), where δ : S Σ S [0, 1] is the transition function and γ : S O [0, 1] is the observation function. MFPS May 2006 Genoa p.18/40

70 Simple Tests 2 We use the same logic as before except that we give a probabilistic semantics and call the formulas tests. I will a.t or at rather than (a)ϕ. MFPS May 2006 Genoa p.19/40

71 Simple Tests 2 We use the same logic as before except that we give a probabilistic semantics and call the formulas tests. I will a.t or at rather than (a)ϕ. Tests define functions from states to [0, 1]. If they define the same function they are equivalent. MFPS May 2006 Genoa p.19/40

72 Simple Tests 2 We use the same logic as before except that we give a probabilistic semantics and call the formulas tests. I will a.t or at rather than (a)ϕ. Tests define functions from states to [0, 1]. If they define the same function they are equivalent. The explicit definition of these functions are: [o] E (s) = γ(s, o) [at] E (s) = s δ(s, a, s )[t] E (s ). MFPS May 2006 Genoa p.19/40

73 Simple Tests 2 We use the same logic as before except that we give a probabilistic semantics and call the formulas tests. I will a.t or at rather than (a)ϕ. Tests define functions from states to [0, 1]. If they define the same function they are equivalent. The explicit definition of these functions are: [o] E (s) = γ(s, o) [at] E (s) = s δ(s, a, s )[t] E (s ). In AI these are called e-tests. MFPS May 2006 Genoa p.19/40

74 Duality with e-tests S = {[t] E } MFPS May 2006 Genoa p.20/40

75 Duality with e-tests S = {[t] E } O = S MFPS May 2006 Genoa p.20/40

76 Duality with e-tests S = {[t] E } O = S γ ([t] E, s) = [t] E (s) MFPS May 2006 Genoa p.20/40

77 Duality with e-tests S = {[t] E } O = S γ ([t] E, s) = [t] E (s) δ ([t] E, a, [at] E ) = 1; 0 otherwise. MFPS May 2006 Genoa p.20/40

78 Duality with e-tests S = {[t] E } O = S γ ([t] E, s) = [t] E (s) δ ([t] E, a, [at] E ) = 1; 0 otherwise. This machine has deterministic transitions and γ is just the transpose of γ. MFPS May 2006 Genoa p.20/40

79 Duality with e-tests S = {[t] E } O = S γ ([t] E, s) = [t] E (s) δ ([t] E, a, [at] E ) = 1; 0 otherwise. This machine has deterministic transitions and γ is just the transpose of γ. MFPS May 2006 Genoa p.20/40

80 The Double Dual If E is the primal and E is the dual then the states of the double dual, E are E -equivalence classes of tests. MFPS May 2006 Genoa p.21/40

81 The Double Dual If E is the primal and E is the dual then the states of the double dual, E are E -equivalence classes of tests. An atomic test is just an observation of E, which is just a state of E so it has the form [s] E for some s. MFPS May 2006 Genoa p.21/40

82 The Double Dual If E is the primal and E is the dual then the states of the double dual, E are E -equivalence classes of tests. An atomic test is just an observation of E, which is just a state of E so it has the form [s] E for some s. We see that γ ([s] E, [o] E ) = [s] E ([o] E ) = γ ([o] E, s) = [o] E (s) = γ(s, o). MFPS May 2006 Genoa p.21/40

83 The Double Dual If E is the primal and E is the dual then the states of the double dual, E are E -equivalence classes of tests. An atomic test is just an observation of E, which is just a state of E so it has the form [s] E for some s. We see that γ ([s] E, [o] E ) = [s] E ([o] E ) = γ ([o] E, s) = [o] E (s) = γ(s, o). An easy calculation shows: [a 1 a 2 a k o] E ([s] E ) = [a 1 a 2 a k o] E (s). MFPS May 2006 Genoa p.21/40

84 Inadequacy of e-tests There is a loss of information in the previous construction. MFPS May 2006 Genoa p.22/40

85 Inadequacy of e-tests There is a loss of information in the previous construction. The double dual behaves just like the primal with respect to e-tests but not with respect to more refined kinds of observations. MFPS May 2006 Genoa p.22/40

86 Inadequacy of e-tests There is a loss of information in the previous construction. The double dual behaves just like the primal with respect to e-tests but not with respect to more refined kinds of observations. [o 1 a 1 o 2 a 2 o 3 ] E ([s] E ) = [o 1 ] E ([s] E ) [a 1 o 2 ] E ([s] E ) [a 1 a 2 o 3 ] E ([s] E ). This does not hold in the primal. MFPS May 2006 Genoa p.22/40

87 Inadequacy of e-tests There is a loss of information in the previous construction. The double dual behaves just like the primal with respect to e-tests but not with respect to more refined kinds of observations. [o 1 a 1 o 2 a 2 o 3 ] E ([s] E ) = [o 1 ] E ([s] E ) [a 1 o 2 ] E ([s] E ) [a 1 a 2 o 3 ] E ([s] E ). This does not hold in the primal. The double dual does not conditionalize with respect to intermediate observations. MFPS May 2006 Genoa p.22/40

88 More General Tests Recall the definition of a POMDP M = (S, Σ, O, δ a : S S [0, 1], γ a : S O [0, 1]). MFPS May 2006 Genoa p.23/40

89 More General Tests Recall the definition of a POMDP M = (S, Σ, O, δ a : S S [0, 1], γ a : S O [0, 1]). A test t is a non-empty sequence of actions followed by an observation, i.e. t = a 1 a n o, with n 1. MFPS May 2006 Genoa p.23/40

90 More General Tests Recall the definition of a POMDP M = (S, Σ, O, δ a : S S [0, 1], γ a : S O [0, 1]). A test t is a non-empty sequence of actions followed by an observation, i.e. t = a 1 a n o, with n 1. An experiment is a non-empty sequence of tests e = t 1 t m with m 1. MFPS May 2006 Genoa p.23/40

91 More General Tests Recall the definition of a POMDP M = (S, Σ, O, δ a : S S [0, 1], γ a : S O [0, 1]). A test t is a non-empty sequence of actions followed by an observation, i.e. t = a 1 a n o, with n 1. An experiment is a non-empty sequence of tests e = t 1 t m with m 1. MFPS May 2006 Genoa p.23/40

92 Some Notation We need to generalize the transition function to keep track of the final state. δ ɛ (s, s ) = 1 s=s s, s S δ aα (s, s ) = s δ a (s, s )δ α (s, s ) s, s S. MFPS May 2006 Genoa p.24/40

93 Some Notation We need to generalize the transition function to keep track of the final state. δ ɛ (s, s ) = 1 s=s s, s S δ aα (s, s ) = s δ a (s, s )δ α (s, s ) s, s S. We have written 1 s=s for the indicator function. MFPS May 2006 Genoa p.24/40

94 Some Notation We need to generalize the transition function to keep track of the final state. δ ɛ (s, s ) = 1 s=s s, s S δ aα (s, s ) = s δ a (s, s )δ α (s, s ) s, s S. We have written 1 s=s for the indicator function. We define the symbol s t s which gives the probability that the system starts in s, is subjected to the test t and ends up in the state s ; similarly s e s. MFPS May 2006 Genoa p.24/40

95 Notation continued We have s a 1 a n o s = δ α (s, s )γ an (s, o). MFPS May 2006 Genoa p.25/40

96 Notation continued We have s a 1 a n o s = δ α (s, s )γ an (s, o). We define s e = s s e s. MFPS May 2006 Genoa p.25/40

97 Equivalence on Experiments For experiments e 1, e 2, we say e 1 M e 2 s e 1 = s e 2 s S. MFPS May 2006 Genoa p.26/40

98 Equivalence on Experiments For experiments e 1, e 2, we say e 1 M e 2 s e 1 = s e 2 s S. Then [e] M is the M -equivalence class of e. MFPS May 2006 Genoa p.26/40

99 Equivalence on Experiments For experiments e 1, e 2, we say e 1 M e 2 s e 1 = s e 2 s S. Then [e] M is the M -equivalence class of e. The construction of the dual proceeds as before by making equivalence classes of experiments the states of the dual machine and MFPS May 2006 Genoa p.26/40

100 Equivalence on Experiments For experiments e 1, e 2, we say e 1 M e 2 s e 1 = s e 2 s S. Then [e] M is the M -equivalence class of e. The construction of the dual proceeds as before by making equivalence classes of experiments the states of the dual machine and the states of the primal machine become the observations of the dual machine. MFPS May 2006 Genoa p.26/40

101 The Dual Machine We define the dual as M = (S, Σ, O, δ : S Σ S, γ : S O [0, 1]), MFPS May 2006 Genoa p.27/40

102 The Dual Machine We define the dual as M = (S, Σ, O, δ : S Σ S, γ : S O [0, 1]), where S = {[e] M }, O = S MFPS May 2006 Genoa p.27/40

103 The Dual Machine We define the dual as M = (S, Σ, O, δ : S Σ S, γ : S O [0, 1]), where S = {[e] M }, O = S δ ([e] M, a 0 ) = [a 0 e] M and MFPS May 2006 Genoa p.27/40

104 The Dual Machine We define the dual as M = (S, Σ, O, δ : S Σ S, γ : S O [0, 1]), where S = {[e] M }, O = S δ ([e] M, a 0 ) = [a 0 e] M and γ ([e] M, s) = s e. MFPS May 2006 Genoa p.27/40

105 The Double Dual We use the e-test construction to go from the dual to the double dual. MFPS May 2006 Genoa p.28/40

106 The Double Dual We use the e-test construction to go from the dual to the double dual. The double dual is where M = (S, A, O, δ, γ ), MFPS May 2006 Genoa p.28/40

107 The Double Dual We use the e-test construction to go from the dual to the double dual. The double dual is where S = {[t] M }, O = S, M = (S, A, O, δ, γ ), MFPS May 2006 Genoa p.28/40

108 The Double Dual We use the e-test construction to go from the dual to the double dual. The double dual is where S = {[t] M }, O = S, δ ([t] M, a 0 ) = [a 0 e] M and M = (S, A, O, δ, γ ), MFPS May 2006 Genoa p.28/40

109 The Double Dual We use the e-test construction to go from the dual to the double dual. The double dual is M = (S, A, O, δ, γ ), where S = {[t] M }, O = S, δ ([t] M, a 0 ) = [a 0 e] M and γ ([t] M, [t] M ) = [t] M e = s α R t (e = αs). MFPS May 2006 Genoa p.28/40

110 The Main Theorem One has to check that everything is well defined. MFPS May 2006 Genoa p.29/40

111 The Main Theorem One has to check that everything is well defined. The main result is: The probability of a state s in the primal satisfying a experiment e, i.e. s e is given by [s] M [e] M = γ ([s] M ) [e] M, where [s] indicates the equivalence class of the e-test on the dual which has s as an observation and an empty sequence of actions. MFPS May 2006 Genoa p.29/40

112 AI Motivation One can plan when one has the model: value iteration etc., but quite often one does not have the model. MFPS May 2006 Genoa p.30/40

113 AI Motivation One can plan when one has the model: value iteration etc., but quite often one does not have the model. In the absence of a model, one is forced to learn from data. MFPS May 2006 Genoa p.30/40

114 AI Motivation One can plan when one has the model: value iteration etc., but quite often one does not have the model. In the absence of a model, one is forced to learn from data. Learning is hopeless when one has no idea what the state space is. MFPS May 2006 Genoa p.30/40

115 AI Motivation One can plan when one has the model: value iteration etc., but quite often one does not have the model. In the absence of a model, one is forced to learn from data. Learning is hopeless when one has no idea what the state space is. There should be no such thing as absolute state! State is just a summary of past observations that can be used to make predictions. MFPS May 2006 Genoa p.30/40

116 AI Motivation One can plan when one has the model: value iteration etc., but quite often one does not have the model. In the absence of a model, one is forced to learn from data. Learning is hopeless when one has no idea what the state space is. There should be no such thing as absolute state! State is just a summary of past observations that can be used to make predictions. The double dual shows that the state can be regarded as just the summary of the outcomes of experiments. MFPS May 2006 Genoa p.30/40

117 We have a paper in the upcoming AAAI conference showing how to use the double-dual to represent systems with hidden state. MFPS May 2006 Genoa p.31/40

118 Machines Categorically A a set and T : Set Set is the functor T S = S A. MFPS May 2006 Genoa p.32/40

119 Machines Categorically A a set and T : Set Set is the functor T S = S A. A machine M is a pair (δ, γ) where δ : S A S is a T -algebra and γ : S P 2 is a relation in Set. MFPS May 2006 Genoa p.32/40

120 Machines Categorically A a set and T : Set Set is the functor T S = S A. A machine M is a pair (δ, γ) where δ : S A S is a T -algebra and γ : S P 2 is a relation in Set. S is the set of states, A the actions and P the propositions. MFPS May 2006 Genoa p.32/40

121 Morphisms of Machines A morphism m from M 1 = (δ 1 : S 1 A S 1, γ 1 : S 1 P 1 2) to M 2 = (δ21 : S 2 A S 2, γ 2 : S 2 P 2 2) is a pair m = (f : S 1 S 2, g : P 2 P 1 ) making the following diagrams commute MFPS May 2006 Genoa p.33/40

122 Morphisms of Machines A morphism m from M 1 = (δ 1 : S 1 A S 1, γ 1 : S 1 P 1 2) to M 2 = (δ21 : S 2 A S 2, γ 2 : S 2 P 2 2) is a pair m = (f : S 1 S 2, g : P 2 P 1 ) making the following diagrams commute S 1 A f id A S 2 A and S 1 P 2 f id P2 S 2 P 2 δ 1 S 1 f S 2 δ 2 id S1 g S 1 γ 1 S 2 γ 2 MFPS May 2006 Genoa p.33/40

123 The Dual Machine The category of machines is written Mch. MFPS May 2006 Genoa p.34/40

124 The Dual Machine The category of machines is written Mch. Given a machine M we define the formulas of M, F M, to be the set A P. If φ = (w, p) we will write aφ for (aw, p). MFPS May 2006 Genoa p.34/40

125 The Dual Machine The category of machines is written Mch. Given a machine M we define the formulas of M, F M, to be the set A P. If φ = (w, p) we will write aφ for (aw, p). We define satisfaction by s = (w, p) δ (s, w)γp. MFPS May 2006 Genoa p.34/40

126 The Dual Machine The category of machines is written Mch. Given a machine M we define the formulas of M, F M, to be the set A P. If φ = (w, p) we will write aφ for (aw, p). We define satisfaction by s = (w, p) δ (s, w)γp. The contravariant functor sends M to M, the dual defined before, and the morphism (f, g) : M 1 M 2 to (g, f) where g ([(w, p)] M2 ) = [(w, g(p))] M1. MFPS May 2006 Genoa p.34/40

127 The Reduction Functor A machine is state reduced if s 1 s 2 φ such that s 1 = φ and s 2 = φ or vice versa. MFPS May 2006 Genoa p.35/40

128 The Reduction Functor A machine is state reduced if s 1 s 2 φ such that s 1 = φ and s 2 = φ or vice versa. A machine is proposition reduced if p 1, p 2 ( w 1, w 2 A [(w 1, p 1 )] M = [(w 2, p 2 )] M ) p 1 = p 2. MFPS May 2006 Genoa p.35/40

129 The Reduction Functor A machine is state reduced if s 1 s 2 φ such that s 1 = φ and s 2 = φ or vice versa. A machine is proposition reduced if p 1, p 2 ( w 1, w 2 A [(w 1, p 1 )] M = [(w 2, p 2 )] M ) p 1 = p 2. We define the reduction functor to be composed with itself i.e.. MFPS May 2006 Genoa p.35/40

130 The Reduction Functor A machine is state reduced if s 1 s 2 φ such that s 1 = φ and s 2 = φ or vice versa. A machine is proposition reduced if p 1, p 2 ( w 1, w 2 A [(w 1, p 1 )] M = [(w 2, p 2 )] M ) p 1 = p 2. We define the reduction functor to be composed with itself i.e.. if M = M we say that it is completely reduced. MFPS May 2006 Genoa p.35/40

131 The Disappointment It would be very pleasant if we took Q : Mch Mch op and R : Mch op Mch to be the two (covariant) functors that represent and get Q R. MFPS May 2006 Genoa p.36/40

132 The Disappointment It would be very pleasant if we took Q : Mch Mch op and R : Mch op Mch to be the two (covariant) functors that represent and get Q R. But this is not possible the way we have set things up! MFPS May 2006 Genoa p.36/40

133 The Disappointment It would be very pleasant if we took Q : Mch Mch op and R : Mch op Mch to be the two (covariant) functors that represent and get Q R. But this is not possible the way we have set things up! The unit of the adjunction would have to be a morphism η M : M M which would then require MFPS May 2006 Genoa p.36/40

134 The Disappointment It would be very pleasant if we took Q : Mch Mch op and R : Mch op Mch to be the two (covariant) functors that represent and get Q R. But this is not possible the way we have set things up! The unit of the adjunction would have to be a morphism η M : M M which would then require a map g : [F M ] P 2. MFPS May 2006 Genoa p.36/40

135 The Disappointment It would be very pleasant if we took Q : Mch Mch op and R : Mch op Mch to be the two (covariant) functors that represent and get Q R. But this is not possible the way we have set things up! The unit of the adjunction would have to be a morphism η M : M M which would then require a map g : [F M ] P 2. Unless M is proposition reduced there is no reason at all for such a thing to exist. MFPS May 2006 Genoa p.36/40

136 But what did we prove before? We did not quite use the construction of the last two slides. MFPS May 2006 Genoa p.37/40

137 But what did we prove before? We did not quite use the construction of the last two slides. M = (δ, γ : [F ] M P 2). MFPS May 2006 Genoa p.37/40

138 But what did we prove before? We did not quite use the construction of the last two slides. M = (δ, γ : [F ] M P 2). We proved that this machine was state reduced. MFPS May 2006 Genoa p.37/40

139 But what did we prove before? We did not quite use the construction of the last two slides. M = (δ, γ : [F ] M P 2). We proved that this machine was state reduced. We quietly ignored the extra propositions in the double dual. MFPS May 2006 Genoa p.37/40

140 Purgatory I There is another way of decomposing into a pair of (covariant) functors F and G. F modifies only the propositions and G modifies only the states. MFPS May 2006 Genoa p.38/40

141 Purgatory I There is another way of decomposing into a pair of (covariant) functors F and G. F modifies only the propositions and G modifies only the states. F M = (δ : S A S, γ : S [F ] M 2) where s γ [φ] M [φ] M γ s s [φ] M. MFPS May 2006 Genoa p.38/40

142 Purgatory I There is another way of decomposing into a pair of (covariant) functors F and G. F modifies only the propositions and G modifies only the states. F M = (δ : S A S, γ : S [F ] M 2) where s γ [φ] M [φ] M γ s s [φ] M. GM = (δ : [S] M A [S] M, γ : [S] M P 2); where MFPS May 2006 Genoa p.38/40

143 Purgatory I There is another way of decomposing into a pair of (covariant) functors F and G. F modifies only the propositions and G modifies only the states. F M = (δ : S A S, γ : S [F ] M 2) where s γ [φ] M [φ] M γ s s [φ] M. GM = (δ : [S] M A [S] M, γ : [S] M P 2); where [s] M := {s S φ F, s = φ s = φ} and δ([s] M, a) := [δ(s, a)] M and [s] M γp sγp. MFPS May 2006 Genoa p.38/40

144 Purgatory II The following natural isos hold: F 2 = F, G 2 = G, QF = Q, and GF = F G = RQ. MFPS May 2006 Genoa p.39/40

145 Purgatory II The following natural isos hold: F 2 = F, G 2 = G, QF = Q, and GF = F G = RQ. For any machine M the following diagram is a pullback MFPS May 2006 Genoa p.39/40

146 Purgatory II The following natural isos hold: F 2 = F, G 2 = G, QF = Q, and GF = F G = RQ. For any machine M the following diagram is a pullback and a pushout at the same time id S π P F M (π S,id [F] ) F GM M (π S,id P ) GM (id [S],π P ) MFPS May 2006 Genoa p.39/40

147 Purgatory II The following natural isos hold: F 2 = F, G 2 = G, QF = Q, and GF = F G = RQ. For any machine M the following diagram is a pullback and a pushout at the same time id S π P F M (π S,id [F] ) F GM M (π S,id P ) GM (id [S],π P ) MFPS May 2006 Genoa p.39/40

148 Conclusions We need to understand the general framework in which this fits. How is it related to dualities in control theory? [Alexander Kurz, Jan Rutten] MFPS May 2006 Genoa p.40/40

149 Conclusions We need to understand the general framework in which this fits. How is it related to dualities in control theory? [Alexander Kurz, Jan Rutten] We are experimenting with these ideas for practical problems in the RL Lab at McGill; joint with Doina Precup and Joelle Pineau. MFPS May 2006 Genoa p.40/40

150 Conclusions We need to understand the general framework in which this fits. How is it related to dualities in control theory? [Alexander Kurz, Jan Rutten] We are experimenting with these ideas for practical problems in the RL Lab at McGill; joint with Doina Precup and Joelle Pineau. Extension to continuous observation and continuous state spaces. MFPS May 2006 Genoa p.40/40

151 Conclusions We need to understand the general framework in which this fits. How is it related to dualities in control theory? [Alexander Kurz, Jan Rutten] We are experimenting with these ideas for practical problems in the RL Lab at McGill; joint with Doina Precup and Joelle Pineau. Extension to continuous observation and continuous state spaces. It is possible to eliminate state completely in favour of histories; when can this representation be compressed and made tractable? MFPS May 2006 Genoa p.40/40