Learning cover context-free grammars from structural data

Transcription

1 Learning cover context-free grammars from structural data Mircea Marin Gabriel Istrate West University of Timişoara, Romania 11th International Colloquium on Theoretical Aspects of Computing ICTAC 2014

2 Outline 1 Introduction Learning languages from queries 2 Context-free languages Structural data 3 Our problem Motivation Preliminary remark 4 Our contributions Algorithm LA l Complexity analysis

3 Learning languages from queries The general learning problem Learn a correct specification of a language from a finite number of positive and negative examples. Well-established means to specify languages of interest: For regular word languages: DFA (hopefully minimum), regular expressions For context-free languages: CFGs, push-down automata For regular tree languages: DFTA (hopefully minimum), regular tree expressions Learning methods: From an arbitrary set of positive and negative examples: Learning a regular language is NP-hard [Gold, 1967] From answers to queries posed to a teacher

4 Learning languages from queries Learning languages from queries Well-known results Learning a language L from answers to 2 kinds of queries: Membership query: w L? ANSWER: yes/no Equivalence query: L = L(A)? where A is the language specification guessed by the learner. ANSWER: yes/no+counterexample w L L(A). R1. For a regular language L, algorithm L builds a minimum DFA with n states in time polynomial in n and m [Angluin, 1987] R2. If L is the regular tree language of structural descriptions of a CFL, algorithm LA builds a minimum DFTA with n states in time polynomial in n and m [Sakakibara, 1990] Note: m =size of largest counterexample provided by the teacher.

5 Learning languages from queries Learning languages from queries An interesting variation (1) In several practical applications, we are interested in the correct specification of a finite subset of L, e.g., only those words of length l: Learn a minimal cover DFA A, i.e., DFA with minimum number of states, such that L(A) Σ l = L Σ l The learning protocol is slightly modified: Membership query: w L? for some w Σ l ANSWER: yes/no Equivalence query: L Σ l = L(A) Σ l? ANSWER: yes/no+counterexample w (L L(A)) Σ l. Note: Often, the size of a minimal cover DFA w.r.t. l is the size of minimum DFA.

6 Learning languages from queries Learning languages from queries An interesting variation (2) Let n := number of states of a minimal cover DFA of L w.r.t. l. A minimal cover DFA of L w.r.t. l can be constructed with algorithm L l in time polynomial in n and m, where m is the size of the largest counterexample returned by the teacher. [Ipate 2012] L l is a nontrivial adjustment of Angluin s algorithm L.

7 Learning languages from queries Learning languages from queries An interesting variation (2) Let n := number of states of a minimal cover DFA of L w.r.t. l. A minimal cover DFA of L w.r.t. l can be constructed with algorithm L l in time polynomial in n and m, where m is the size of the largest counterexample returned by the teacher. [Ipate 2012] L l is a nontrivial adjustment of Angluin s algorithm L. Questions: 1 Can Sakakibara s algorithm LA be adjusted, like L was adjusted to L l, to learn a cover automaton for the structural descriptions of a CFL? 2 Does this notion of cover automaton have practical significance? 3 What is the complexity of such an algorithm? (Is it tractable?)

8 Structural data Skeletons of derivation trees [Sakakibara 1990] ASSUMPTIONS: G = (N, Σ, P, S): ɛ-free CFG. D(G) : set of derivation trees of G Definition (skeleton) The skeleton, or structural description of t D(G) is the labeled tree sk(t) obtained from t by replacing all labels of interior nodes with σ. Example (Structural description of derivation trees) G = ({S, A}, {a, b}, {S A, A aab, A ab}, S) is a CFG t = a a S A A b b D(G) sk(t) = a a σ σ σ b b T ({σ, a, b})

9 Structural data Notation, definitions FOR WE DEFINE G, G U : CFGs with terminals from Σ A : DFTA for trees from T ({σ} Σ) M D(G), B T ({σ} Σ), l : positive integer K (M) := {sk(s) s M} K (D(G)) is called set of structural descriptions of G L(A) : language accepted by A B [l] := {t B d(t) l} where d(t) is the depth of t. G is a cover CFG of G U w.r.t. l if K (D(G)) [l] = K (D(G U )) [l] A is a cover DFTA of G w.r.t. l if L(A) [l] = K (D(G)) [l]

10 Structural data DFTAs for structural descriptions A skeletal signature for CFG G = (N, Σ, P, S) is {σ} Σ together with ar : {σ} Σ P(N) defined by ar(a) = 0 for all a Σ ar(σ) = {m (X U 1... U m ) P}. Definition (DFTA for skeletal signature) A = (Q, {σ} Σ, Q f, {δ i i {0} ar(σ)}) where Q : set of states, Q f Q: set of final states, δ 0 = id Σ and δ i : (Σ Q) i Q for all i ar(σ) Define recursively δ : T ({σ} Σ) Q Σ: δ (a) := a for all a Σ, δ (σ(t 1,..., t i )) := δ i (δ (t 1 ),..., δ (t i )) Q if i > 0. and L(A) := {t δ (t) Q f }.

11 The problem For a positive integer l, and an unknown CFG G U learn a cover CFG G of G U w.r.t. l by posing two kinds of questions: Structural membership query: t K (D(G U )) [l]? ANSWER: yes/no Structural equivalence query: K (D(G)) [l] = K (D(G U )) [l]? ANSWER: yes cover CFG G was learned no counterexample s K (D(G)) [l] K (D(G U )) [l]. ASSUMPTION: The learner and teacher share the following knowledge: σ: set of terminals of G U d := max{m X U 1... U m is a production of G U } l and σ.

12 Motivation Significance of the problem By learning the structural descriptions of a CFL instead of the CFL itself, we learn how to parse and interpret its expressions. In natural language understanding, the bound memory restriction on human comprehension seems to limit the recursion depth of such parse trees to a constant. the restriction to structural descriptions of derivation trees with depth a given constant is reasonable. E.g., the L A T E X system limits the number of nestings of itemized environments to a small constant.

13 Preliminary remark From DFTAs to CFGs Every cover DFTA of G U can be transformed easily into a cover CFG of G U : where A = (Q, {σ} Σ, Q f, {δ i i {0} ar(σ)}) G = (Q {S}, Σ, P, S) P := {q r 1... r m δ m (r 1,..., r m ) = q} {S r 1... r m δ m (r 1,..., r m ) Q f }.

14 Preliminary remark From DFTAs to CFGs Every cover DFTA of G U can be transformed easily into a cover CFG of G U : where A = (Q, {σ} Σ, Q f, {δ i i {0} ar(σ)}) G = (Q {S}, Σ, P, S) P := {q r 1... r m δ m (r 1,..., r m ) = q} {S r 1... r m δ m (r 1,..., r m ) Q f }. it suffices to learn a cover DFTA of K (D(G U )) instead of a cover CFG of K (D(G U )).

15 Algorithm LA l Main result 1 Algorithm LA l which learns a minimal DCTA A for K (D(G U )) w.r.t. l 2 The running time of LA l is T (LA l ) = polynomial in m and n where n =number of states of A m = max size of counterexample provided by teacher

16 Algorithm LA l Auxiliary notions S T ({σ} Σ) \ Σ is subterm-closed if s S whenever s S and s is a subterm if s with d(s) > 0. C C({σ} Σ) if C T ({σ} Σ { }) and appears exactly once in C, at leaf position. If S is subterm-closed then σ S is the set of contexts with hole-depth 1, whose subterms below root are from S Σ: σ S := m ar(σ) i=1 m {σ(s 1,..., s m )[ ] i s 1,..., s m S Σ} E C({σ} Σ) is -prefix closed if C E whenever C = C 1 [C ] with C E and C 1 σ S.

17 Algorithm LA l The observation table (1) LA l =generalisation of algorithms L l and LA Based on the construction of an observation table T = (S, E, T, l) for K (D(G U )) T ({σ} Σ), where S = subterm-closed set of trees from T ({σ} Σ) [l] \ Σ E = -prefix-closed set of contexts from C({σ} Σ) with depth l and hole depth l 1 Rows are labeled with elements from S X(S) [l] where X(S) := {C 1 [s] C 1 σ S, s S Σ} \ S Columns are labeled with elements from E Builds DFTA A = (Q, {σ} Σ, Q f, δ) consistent with T, i.e. C E, s S X(S) such that d(c[s]) l: δ (C[s]) Q f iff T (C[s]) = 1.

18 Algorithm LA l The observation table (2) The entry at position (s, C) (S X(S) [l] ) E in the table is labeled with T (C[s]) {1, 0, 1} where C[s] =term produced by replacing with s in C, and 1 if t K (D(G U )) [l], T (t) := 0 if t T (Sk Σ) [l] \ K (D(G U )), 1 if t T (Sk Σ) [l]. Initial observation table E = { }, S = X(S) [l] has 1 + Σ Σ d = Σ d+1 1 Σ 1 entries, where d=max. rank of σ.

19 Algorithm LA l Observation tables Example For G U := ({S, A}, {a, b}, {S Ab, A Ab, A ab}, S), l = 2, S = {σ(a, b)}, E = {, σ(, b), σ(a, σ(, b))}. Then σ (S) = {σ(, s), σ(s, ) s {a, b, σ(a, b)}}. The observation table is σ(, b) σ(a, b) 0 1 σ(a, a) 0 0 σ(b, a) 0 0 σ(σ(a, b), a) 0 1 σ(b, b) 0 0 σ(σ(a, b), b) 1 1 σ(a, σ(a, b)) 0 1 σ(b, σ(a, b)) 0 1 σ(σ(a, b), σ(a, b)) 0 1

20 Algorithm LA l The automaton construction Main ideas (1) We wish to identify the states of a minimal DCTA A for K (D(G U )) [l] with certain rows of s S: even if row of s 1 row of s 2, they may represent the same state of A, if C[s 1 ] = C[s 2 ] whenever C E such that C[s 1 ], C[s 2 ] T ({σ} Σ) [l] auxiliary notions: k-similarity for 1 k l: s k t : T (C[s]) = T (C[t]) for all C E k max{d(s),d(t)} similarity: := l A total order T on T ({σ} Σ) induced by a total order on Σ (Defn. 6)

21 Algorithm LA l The automaton construction Main ideas (1) We wish to identify the states of a minimal DCTA A for K (D(G U )) [l] with certain rows of s S: even if row of s 1 row of s 2, they may represent the same state of A, if C[s 1 ] = C[s 2 ] whenever C E such that C[s 1 ], C[s 2 ] T ({σ} Σ) [l] auxiliary notions: k-similarity for 1 k l: s k t : T (C[s]) = T (C[t]) for all C E k max{d(s),d(t)} similarity: := l A total order T on T ({σ} Σ) induced by a total order on Σ (Defn. 6) the representative of x S X(S): r(x) := min T {s S x S}

22 Algorithm LA l The automaton construction Main ideas (2) where Q := {r(s) s S} Observation table T = (S, E, T, l) DFTA A(T) := (Q, {σ} Σ, Q f, δ) Q f := {q Q T (q) = 1} δ(q 1,..., q m ) := r(σ(q 1,..., q m )) for all m ar(σ) REMARK: If T is closed and consistent then A(T) is well-defined.

23 Algorithm LA l Relevant properties of observation tables T = (S, E, T, l) is Consistent if k {1,..., l}, s 1, s 2 S, C 1 σ S : If s 1 k s 2 then C 1 [s 1 ] k C 1 [s 2 ] Closed if x X(S), s S : d(s) d(x) x s

24 Algorithm LA l Relevant properties of observation tables T = (S, E, T, l) is Consistent if k {1,..., l}, s 1, s 2 S, C 1 σ S : If s 1 k s 2 then C 1 [s 1 ] k C 1 [s 2 ] Closed if x X(S), s S : d(s) d(x) x s Fact (Corr. 2) If T: closed and consistent observation table of K (D(G U )) n = # of states of A(T) N = # of states of minimal DFCA of K (D(G U )) n N then n = N and A(T) is minimal DFCA of K (D(G U ))

25 Algorithm LA l Relevant properties of observation tables Example is Consistent Not closed σ(, b) σ(a, b) 0 1 σ(a, a) 0 0 σ(b, a) 0 0 σ(σ(a, b), a) 0 1 σ(b, b) 0 0 σ(σ(a, b), b) 1 1 σ(a, σ(a, b)) 0 1 σ(b, σ(a, b)) 0 1 σ(σ(a, b), (a, b)) 0 1

26 Algorithm LA l Learning strategy Create initial observation table T 0 for S = and E =

27 Algorithm LA l Learning strategy Create initial observation table T 0 for S = and E = Repeat building sound and consistent observation tables of K (D(G U )), with more and more states Tables are produced incrementally, by adding more and more rows and columns. 1 If T(t) is not consistent T(t + 1) extends T(t) with new column 2 If T t is not closed T(t + 1) extends T(t) with new row 3 Otherwise, ask structural equivalence query: K (D(G U )) [l] = L(A(T(t))) [l]? 1 If yes, stop with learned answer A(T t) 2 Otherwise, use counterexample s K (D(G U )) L(A(T(t))) [l] to extend T(t) with all missing rows of subterms of s, including s itself.

28 Algorithm LA l Learning strategy Create initial observation table T 0 for S = and E = Repeat building sound and consistent observation tables of K (D(G U )), with more and more states Tables are produced incrementally, by adding more and more rows and columns. 1 If T(t) is not consistent T(t + 1) extends T(t) with new column 2 If T t is not closed T(t + 1) extends T(t) with new row 3 Otherwise, ask structural equivalence query: K (D(G U )) [l] = L(A(T(t))) [l]? 1 If yes, stop with learned answer A(T t) 2 Otherwise, use counterexample s K (D(G U )) L(A(T(t))) [l] to extend T(t) with all missing rows of subterms of s, including s itself. By Corr. 2, a minimal DFCA of K (D(G U )) will be eventually constructed.

29 Complexity analysis Complexity analysis Failed checks T(t) = (S t, E t, T, l) is extended when Failed closeness check: Find C 1 σ S t and s S t such that C 1 [s] t for all t S t with d(t) d(c 1 [s]) Failed consistency check: Find C E t with d (C) = i, s 1, s 2 S t with d(s 1 ), d(s 2 ) l i 1, and C 1 σ (S t ) s.t. C[C 1 [s 1 ]], C[C 1 [s 2 ]] T ({σ} Σ) [l], s 1 k s 2, where k = max{d(s 1 ), d(s 2 )} + i + 1 and T (C[C 1 [s 1 ]]) T (C[C 1 [s 2 ]]) Failed structural equivalence query: T(t) is closed and consistent, but we know a counterexample t K (D(G U )) [l] L(A(T(t))) [l]

30 Complexity analysis Complexity analysis If n := # of states of minimal DFCA of K (D(G U )) w.r.t. l, then 1 total # of failed closedness checks: n(n + 1)/2 [Theorem 3] 2 total # of failed consistency checks: n(n 1)/2 [Theorem 5] 3 total # of failed structural equivalence queries: n [Theorem 6]

31 Complexity analysis Complexity analysis Parameters: n := # of states of minimal DFCA of K (D(G U )) w.r.t. l m := max. size of counterexample returned by a failed structural equivalence query p := Σ d := max. rank of skeleton symbol σ R1. Total space occupied by the observation table at any time: O ( n 2 (m n + n 2 + p) d (m + 2 n + 1)d m+2 n+1). R2. Consistency check: O(n 5 d (m n + n 2 + p) d+2 ). R3. Closedness check: O((m n + n 2 ) 2 d (m n + n 2 + p) d n 2 ) = O(n 2 d (m n + n 2 + p) d+2 ).

32 Complexity analysis Complexity analysis R4. There are at most n(n + 1)/2 + n m elements added to S by failed closeness checks and failed structural equivalence checks. R5. Extend the table with new element in S t+1 : O(d (m n + n 2 + p) d ) membership queries total time spent to extend the S-component of the observation table: O(n 2 d (2 d d) (m n + n 2 )(m n + n 2 + p) d ) R6. Only failed consistency checks extend the E-component of the observation table There are at most n(n 1)/2 failed consistency checks Total time to insert context in E: O(d (m n + n 2 + p) d ) Total time to insert all contexts in E: O(n 2 d (m n + n 2 + p) d )