Congruence based approaches

Transcription

1 Congruence based approaches Learnable representations for languages Alexander Clark Department of Computer Science Royal Holloway, University of London August 00 ESSLLI, 00

2 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

3 Context free grammar Tuple Σ, V, P, S V is a set of non-terminals S V is a start symbol P is a set of productions of the form V (V Σ), which we write as N α

4 Context free grammar Tuple Σ, V, P, S V is a set of non-terminals S V is a start symbol P is a set of productions of the form V (V Σ), which we write as N α Derivation relation: βnγ βαγ when N α P L(G, N){w N w} L(G) = {w S w}

5 Almost Chomsky normal form All rule are of the form N PQ N a N λ Multiple start symbols: G = Σ, V, P, I I V is a set of initial symbols. L(G) = S I L(G, S)

6 Distributional learning Several reasons to take distributional learning seriously: Cognitively plausible (Saffran et al,, Mintz, 00) It works in practice: large scale lexical induction (Curran, J. 00) Linguists use it as a constituent structure test (Carnie, A, 00) Historically, PSGs were intended to be the output from distributional learning algorithms: Chomsky (/00) The concept of "phrase structure grammar" was explicitly designed to express the richest system that could reasonable be expected to result from the application of Harris-type procedures to a corpus.

7 Distributional Learning Zellig Harris (4, ) Here as throughout these procedures X and Y are substitutable if for every utterance which includes X we can find (or gain native acceptance for) an utterance which is identical except for having Y in the place of X

8 Empirical work on Distributional Learning Real corpora Sample is not just of grammatical sentences Also semantically well-formed Also true in some non-technical sense Empirical distribution is very complex Distributional similarity in real corpora often reflects semantic relatedness.

9 Various notions of context Local syntactic context Immediately preceding and following word If the candidate has an outstanding examination result Word has context (candidate, an) Wide bag-of-words context Skip stop words Set of words occurring in the same sentence/discourse. Probabilistic models Vector of counts of frequent words Schütze

10 Distribution Full context Context (or environment) A context is just a pair of strings (l, r) Σ Σ. (l, r) u = lur f = (l, r). Special context (λ, λ) Given a language L Σ. Distribution of a string C L (u) = {(l, r) lur L} = {f f u L} Distributional Learning models/exploits the distribution of strings;

11 Simple CFL L = {a n b n n 0} L = {λ, ab, aabb... } Distribution of aab (λ, b) aab = aabb L Example (a, bb) aab = aaabbb L (λ, abb) aab = aababb L

12 Simple CFL L = {a n b n n 0} L = {λ, ab, aabb... } Distribution of aab (λ, b) aab = aabb L Example (a, bb) aab = aaabbb L (λ, abb) aab = aababb L C L (aab) = {(λ, b), (a, bb),... (a i, b i+ )... } C L (aaabb) = C L (aab) C L (a) = {(λ, b), (a, bb), (λ, abb)... }

13 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

14 Congruence classes and the syntactic monoid Congruence classes u v iff C L (u) = C L (v) Write [u] for class of u Two strings are congruent if they are perfectly substitutable in every context. Words: Tuesday/Wednesday, cat/dog, man/student

15 Congruence classes of a regular language Example L = (ab) [λ] = {λ} [a] = {a, aba, ababa,... } [b] = {b, bab, babab,... } [ab] = {ab, abab,... } [ba] = {ba, baba,... } [bb] every other string with C L (u) =

16 Regular language Theorem A regular languages has finitely many congruence classes. Proof Later. Number of congruence classes may be exponential in size of minimal DFA: Q Q.

17 Example blowup S0 a b b S a b S a a b S a b S4 a b S

18 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

19 Monoid Simple algebraic structure Definition Associative operation with an element u = u = u Examples Strings under concatenation Numbers under addition Matrices under multiplication

20 Syntactic monoid Concatenation If u u and v v then uv u v [u][v] [uv] Example L = (ab) [b][ab] = {b, bab,... }{ab, abab... } = {bab, babab... } {b, bab... } = [b] Syntactic monoid Σ / L [u] [v] = [uv] Associative and [λ] is identity

21 Zero Suppose Sub(L) is not equal to Σ There are strings that do not occur as a substring of any string in L. Example L = {(ab) } bb Sub(L) C L (bb) = Fact: if u Sub(L) then uv Sub(L) So we may have a congruence class 0; 0 X = 0 = X 0

22 Example L = {(ab) } [λ] [bb] [a] [b] [ab] [ba] [λ] [λ] [bb] [a] [b] [ab] [ba] [bb] [bb] [bb] [bb] [bb] [bb] [bb] [a] [a] [bb] [bb] [ab] [bb] [a] [b] [b] [bb] [ba] [bb] [b] [bb] [ab] [ab] [bb] [a] [bb] [ab] [bb] [ba] [ba] [bb] [bb] [b] [bb] [ba]

23 Exercise Language {w w a = w b } Language of equal numbers of a s and b s Question What is the syntactic monoid of this? What is it isomorphic too?

24 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

25 Context free grammar Suppose we have a grammar with non-terminals N, P, Q We have a rule N PQ This means that Y (N) Y (P)Y (Q).

26 Context free grammar Suppose we have a grammar with non-terminals N, P, Q We have a rule N PQ This means that Y (N) Y (P)Y (Q). Backwards Given a collection of sets of strings X, Y, Z Suppose X YZ Then we add a rule X YZ.

27 Congruence classes Partition of the strings Congruence classes have nice properties! [u][v] [uv] [uv] [u][v] [u] [v] [u][v] L = {a n b n n 0} [a] = {a} [abb] = {abb, aabbb,... } [a][abb] = {aabb, aaabbb... } [aabb] = [ab] So we have a rule [ab] [a][aab]

28 Constructing CFG from congruence classes One non-terminal per congruence class [uv] [u][v] [a] a [λ] λ I = {[u] [u] L} Multiple start symbols.

29 Example L = {a n b n n 0} Grammar I = {[ab], [λ]}

30 Example L = {a n b n n 0} Grammar I = {[ab], [λ]} [a] a, [b] b, [λ] λ

31 Example L = {a n b n n 0} Grammar I = {[ab], [λ]} [a] a, [b] b, [λ] λ [ab] [aab][b], [ab] [a][b], [ab] [a][abb] [aab] [a][ab], [abb] [ab][b]

32 Example L = {a n b n n 0} Grammar I = {[ab], [λ]} [a] a, [b] b, [λ] λ [ab] [aab][b], [ab] [a][b], [ab] [a][abb] [aab] [a][ab], [abb] [ab][b] Plus [ba] [b][ba]... Plus [a] [λ][a]...

33 Problems If we can figure out whether u L v then we can write down a grammar. Problem A In general we will have an infinite set of congruence classes Pick some finite subset of them This does not give us every CFL {a n b m n < m} Problem B Hard to tell whether u L v We need some way of testing this.

34 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

35 Three algorithms All based on the same representational idea:. Substitutable languages from positive data. Congruential languages with queries. NTS languages with stochastic positive examples

36 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

37 Two ideas Chomsky review of Greenberg, let us say that two units A and B are substitutable if there are expressions X and Y such that XAY and XBY are sentences of L.; substitutable if whenever XAY is a sentence of L then so is XBY and whenever XBY is a sentence of L so is XAY (i.e. A and B are completely mutually substitutable). These are the simplest and most basic notions. Problem: we need substitutability but what we observe is substitutability

38 Old concept John Myhill, 0 commenting on Bar-Hillel I shall call a system regular if the following holds for all expressions µ, ν and all wffs φ, ψ each of which contains an occurrence of ν: If the result of writing µ for some occurrence of ν in φ is a wff, so is the result of writing µ for any occurrence of ν in ψ. Nearly all formal systems so far constructed are regular; ordinary word-languages are conspicuously not so. Clark and Eyraud, 00 A language is substitutable if lur, lvr, l ur L means that l vr L.

39 substitutable and reversible Clark and Eyraud, 00 A language is substitutable if lur, lvr, l ur L means that l vr L. Angluin, A language is reversible if ur, vr, ur L means that vr L.

40 A Bad Intuition One context in common in enough The cat died The dog died So cat and dog are congruent

41 A Bad Intuition One context in common in enough The cat died The dog died So cat and dog are congruent He is an Englishman He is thin So thin and an Englishman are congruent.

42 Result Clark and Eyraud, 00/00 Polynomial result The class of substitutable context free languages is polynomially identifiable in the limit from positive data only. Polynomial characteristic set Polynomial update time Why the delay?

43 A Simple Algorithm Non-technical description Given a sample of strings W = {w,..., w n }. Define a graph G = N, E N is the set of all non-empty substrings (factors) of W. E = {(u, v) (l, r), lur W lvr W }. Define a grammar in Chomsky normal form The set of non-terminals is the set of components of the graph G Have productions for: [a] a Add rules for non terminals: [w] [u][v] iff [w] = [uv].

44 Simple linguistically motivated example the man who is hungry died. the man ordered dinner. the man died. the man is hungry. is the man hungry? the man is ordering dinner. is the man who is hungry ordering dinner? is the man who hungry is ordering dinner?

45 Substitution graph Auxiliary fronting example

46 Derivations [is the man hungry?] [is the man hungry] [?] [is the man] [hungry] [?] [is] [the man] [hungry] [?] [is] [the man] [who is hungry] [hungry] [?] [is] [the man] [who is hungry] [ordering dinner] [?]. Note that this will not work for all data since English is not substitutable (Berwick, Coen and Niyogi, p.c.)

47 Substitution graph Learnable example p p p p e j d a m g p e j d a m g n p e j d a m g n n p e j d a m g n n o m p e j d a m g n n o m h p e j d a m g n n o m h e p e j d a m g n n o m h e r p e j d a m g o 0 p e j d a m g o b 4 p e j d a m g o b j 4 p e j d a m g o b j r p e j d a m g o b j r t p e j d a m g o b j p e j d a m g o b j o p n 4 p e g b t s j r i s p s p s h p m t p r l q o p d 0 p r l q o p d i p r n c c m p r n c c m t p c p p c n j a i q o j d p n p j a p o p p o 4 p o m c 4 p k 4 e e l b p e l b p m e l b p b 0 e s 0 e s n i s h b t f e n p p q q e j d a m g n n o m h e r e j d a m g o b j r t e j d a m g o b j o p n 4 e g b t s j r i s e o 44 e o s e o c 4 e o g s i j t 4 t l p t b 4 t l p t b c 0 t s 0 t s p e j d a m g 4 t s p e j d a m g o 4 t s p e j d a m g o b 4 t s p e j d a m g o b j 4 t s p e j d a m g o b j r t s p e j d a m g o b j c t s q l j a d i d o 0 t s q l j a d i d o b 0 t s q l j a d i d o b j 0 0 t s s h l o 0 t s s h l o b 0 t s s h l o b j 0 t s d 0 t s d j o 0 0 t s d j o b 0 0 t s d j o b j 0 0 t s d j o b j r t s d j o b j c t s d k o b s j j s k o t s d k o b s j j s k o b t s d k o b s j j s k o b j t s d k o b s j j s k o b j r t s o n m e d k d c o 0 0 t s o n m e d k d c o b 0 0 t s o n m e d k d c o b j 0 0 t s f i 0 0 t s f i o t s f i o b t s f i o b j t s f i o b j r t s a s j e i o t s a s j e i o b t s a s j e i o b j t h 4 t r b n r f r b t n n l t j k q i e 0 t f k i i e l b p i t b g i m d l a 4 i t b g i m d l a c i i p j a p o p i i n r o i d d h h i d o g h q i m 00 i n 4 i n n i n n o m h e r i j i o t m p e t 4 i a n m i a n m d 4 i a n m d d i a n m f l 4 l i l i q n 0 l i q n j 0 l i q n j l l l n g t r l n g t r t l n g t r t m l n g t r t b l g d s q i 0 l g d s q i o 0 l g d s q i o f 0 l g d s q i o f i 0 l g d s q i o f i n l b p m q q p q p p q p k q e e i s j q i k q r n k q l j a d q l j a d i q l j a d i d q l j a d i d n q l j a d i d n n q l j a d i d n n o m q l j a d i d n n o m h q l j a d i d n n o m q l j a d i d n n o m h e r q l j a d i d o 0 q l j a d i d o b 0 q l j a d i d o b j 0 q s q m h n q r k c q b n t l t q b n t l t p 0 q b n t l t k q o r t b s m q o r t b s m i q f s 0 q f s c q a f l a c t 4 s p e j d a m g 4 s p e j d a m g o 4 s p e j d a m g o b j 4 s p e j d a m g o b j r s p e j d a m g o b j c s t j k q i e 0 s l s q 4 s q l j a d i d 0 s q l j a d i d o 0 s q l j a d i d o b j 0 s s h l 0 s s h l o 0 s s h l o b j 0 s d t m h a s d j 0 0 s d j o 0 0 s d j o b j 0 0 s d j o b j r s d j o b j c s d k o b s j j s k s d k o b s j j s k o s d k o b s j j s k o b j s d k o b s j j s k o b j r s h 4 s h p m t s h p m t d s h p m t d d 44 s h p m t f 0 s h l 0 s h l n s h l n n s h l n n o m s h l n n o m h s h l n n o m h e s h l n n o m h e r s h l o 0 s h l o b 0 s h l o b j 0 s m k 4 s n i s h b t f s n f a e s n f a e j s b i e i q l 0 s o s o n m e d k d c 0 0 s o n m e d k d c o 0 0 s o n m e d k d c o b j 0 0 s o g g 0 s f s f i s f i o s f i o b j s f i o b j r s f q s a s a s j e 0 s a s j e i s a s j e i o s a s j e i o b j s a m j c s b h d e i d 4 0 d e l b p d t g d l n g t r t d c n j a i q o j d d j d j n d j n n d j n n o m d j n n o m h d j n n o m h e d j n n o m h e r d j o d j o b 0 0 d j o b j 0 0 d j o b j r d j o b j r t d o g h q 0 d k o b s j j s k 0 4 d k o b s j j s k n d k o b s j j s k n n d k o b s j j s k n n o m d k o b s j j s k n n o m h d k o b s j j s k n n o d k o b s j j s k n n o m h e r d k o b s j j s k o 4 d k o b s j j s k o b d k o b s j j s k o b j d k o b s j j s k o b j r d k o b s j j s k o b j r t d f c h p m t h p m t d h p m t f 0 h t s p e j d a m g 4 h t s p e j d a m g o 4 h t s p e j d a m g o b 4 h t s p e j d a m g o b j 4 h t s p e j d a m g o b j r h t s p e j d a m g o b j c h t s q l j a d i d 0 h t s q l j a d i d o 0 h t s q l j a d i d o b j 0 h t s s h l 0 h t s s h l o 0 h 0 h t s s h l o b j 0 h t s d j 0 0 h t s d j o 0 0 h t s d j o b 0 h t s d j o b j 0 0 h t s d j o b j r h t s d j o b j c h t s d k o b s j j s k h t s d k o b s j j s k o h t s d k o b s j j s k o b j h t s d k o b s j j s k o b j r h t s o n m e d k d c 0 0 h t s o n m e d k d c o 0 0 h t s o n m e d k d c o b j 0 0 h t s f 0 h t s f i h t s f i o h t s f i o b 0 0 h t s f i o b j h t s f i o b j r 0 h t s a s j e i h t s a s j e i o h t s a s j e i o b j h l n n o m h e r h d s c h h t s p e j d a m g o 4 h h t s p e j d a m g o b 4 h h t s p e j d a m g o b j h h t s p e j d a m g o b j r t 4 h h t s p e j d a m g o b j c h h t s p e j d a m g o b j o p n 4 h h t s q l j a d i d o b 0 h h t s s h l o b 0 0 h h t s d j o b 0 0 h h t s d j o b j r t h h t s d j o b j c h h t s d k o b s j j s k o b h h t s d k o b s j j s k o b j r t h h t s o n m e d k d c o b 0 0 h h t s f 0 h h t s f h h t s f i o b h h t s h h t s f i o b j r t h h t s a s j e 0 h h t s a s j e i o b h r a c j o h f h n e t j e p g h n e t j e p g n h j l t m q b h g j q c n c b h g j q c n c b j h f i t g a n p 0 h f i t g a n p p 0 h f i t g a n p p c 0 m 4 m l m h l b a m h g m m m 4 0 m m m c m n l c m n l c j 4 m k 0 m f o 0 m f o a 0 m a i a r 4 4 r p n i q o d p q r p n i q o d p q d r p n i q o d p q d d r p n i q o d p q f 4 0 r t 4 r t l l 4 c p c l i q n j l l c q e e i s c q e e i s j c q e e i s j l l c s d t m h a 0 0 c m c m f o 0 c m f o a c m f o a p 0 4 c m f o a k c r t l l 4 c c q e e i s j l l c c a 4 c n j a i q o j d c n j a i q o j d o 0 c n j a i q o j d f c j c j a 40 c o p n l l c f j c f k d i i l e s e l l n p r l q o p d 0 n e t j e p g n n e t j e p g n i 0 n i s k n i o t m p e t 4 n l 0 4 n l h b d q j e 4 n h r a c j o h f 4 4 n m e d k d c n n o m h e r n r b e b m b o 4 n r o 4 4 n j a i q o j d n j a i q o j d s n j a i q o j d o 0 n j a i q o j d k n f t s p i n f a e j j 4 j e l b p j t e l j i d d h h j l t m q b j l t m q b i j l i q n j j s b i e i q l 4 j d e i d j d a m g o b j r t j d a m g o b j h e j h r q e c m q j m h l b j m h l b a j r p n i q o d p q d j r t j r g h j c q e e i s j j c s d t m h j c s d t m h a j n e o k c i i c 4 0 j n n j n n o j n n o m j n n o m h j n n o m h e j g t j o j o p n j o m c d 0 j o b 4 j o b j 44 j o b j r t j o b j c q e e i s j 4 j f k d i i l e s e j a j a p o p j a l g d s q i o f i 0 j a l g d s q i o f i i 0 j a s j a h n e t j e p g n j a h n e t j e p g n i j a h j l t m q b j a h j l t m q b i j a n p r l q o p d 4 j a n p r l q o p d i 4 j a a e n p p q q 4 4 j a a e n p p q q i 4 4 g l f h h 4 g r f k j 0 g r f k j i 0 g r f k j i d 0 4 g b t s j r i s g o b j r t g o b j c q e e i s j g o b j o p n 4 b c h a f 04 b c h a f d 4 b c h a f d d 40 b c h a f f b c c 4 4 b c c a b n t l t b n t l t p 0 b j i d d h h b j l i q n j b j s b i e i q l 4 b j d e i d b j m h l b a b j r t b j r g h b j r g h p 4 b j r g h k b j c q e e i s j b j c s d t m h a b j g t b j g t p 4 b j g t k b j o p n b j f k d i i l e s e o o p n o p n l l o e o g s i o e o g s i j o i o s o m c o m c d 4 o m c d d 0 4 o m c f o n m e d k d c o n m e d k d c n o n m e d k d c n n o n m e d k d c n n o m o n m e d k d c n n o m h o n m e d k d c n n o m h e o n m e d k d c n n o m h e r o n m e d k d c o 0 0 o n m e d k d c o b 0 0 o n m e d k d c o b j 0 0 o g h q o g h q o 4 o g g 0 o b j i d d h h o b j l i q n j o b j s o b j d e i d o b j m h l b o b j m h l b a o b j c s d t m h o b j c s d t m h a o b j o p n o b j f k d i i l e s e o o j j e l l f s 4 o a k o a k m 0 k t t k t t p 0 4 k t t p c 0 k t f k 0 0 k t f k d 4 k t f k d d 0 4 k l g d s q i o f i 0 k s m k k s o g g 0 k d t g 0 0 k d t g d k d t g d d 4 0 k c p 0 k c n j a i q o j d 0 k n 0 k n t 4 k n t m k n l k n l h b d q j e k o e o g s i j k o g h q k o b s j j s k 0 4 k o b s j j s k n n k o b s j j s k n n o k o b s j j s k n n o m k o b s j j s k n n o m h k o b s j j s k n n o m h e k o b s j j s k o 4 k o b s j j s k o b k o b s j j s k o b j k o b s j j s k o b j r t k f j j b r k k f j j b r k p k f j j b r k k k a k a o 0 k a f f f i f i n 0 f i n n f i n n o m f i n n o m h f i n n o m h e f i n n o m h e r f i o f i o b f i o b j f i o b j r 4 f i o b j r t 4 f i f p 0 f q f c f c i k 4 f c i k a f c i k a p f c i k a k f o l 4 4 f o l p f k d i i l e s e f k d i i l e s e l l a 0 a p a e n p p q a e n p p q q 4 a e n p p q q n a e o g d s 0 4 a i a l g d s q i o f a l g d s q i o f i i 0 a q o r t b s m 0 4 a s a s j e a s j e i a s j e i n a s j e i n n a s j e i n n o m a s j e i n n o m h a s j e i n n o m h e a s j e i n n o m h e r a s j e i o a s j e i o b a s j e i o b j a h n e t j e p g a h j l t m q b i a m j c s b h a m g o b j r t a m g o b j o p n 4 a n p r l q o p d i 4 a n r b e b m b o 4 4 a k a k m a f l a c t 4 a f s a j j f e a f s a j j f e d 4 a f s a j j f e d d 4 a f s a j j f e f 4 4 4

48 Substitution graph Unlearnable languages

49 Substitutable languages Some very basic languages are not substitutable: L = {a, aa} L = {a n b n n > 0} Dyck language The very strict requirement for contexts to be disjoint is unrealistic. If we move to a probabilistic learning approach, we can test for congruence with Ĉ(u) Ĉ(v).

50 Congruence class results Positive data alone lur L and lvr L implies u L v Polynomial result from positive data. (Clark and Eyraud, 00) k-l substitutable languages, (Yoshinaka 00) Stochastic data If data is generated from a PCFG PAC-learn unambiguous NTS languages, (Clark, 00) Membership queries An efficient query-learning result (Clark, 00) Pick a finite set of contexts F Test if C L (u) F = C L (v) F

51 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

52 ICGI 00 Use query learning just as with LSTAR the MAT model. Membership queries Equivalence queries counterexamples Efficient learning but not for all CFLs.

53 Algorithm Maintain two sets: A set of strings K ; includes Σ and λ A set of contexts F ; includes (λ, λ) We have rows for K And also for KK every pair.

54 Observation table K a set of strings and F a set of contexts f f f f 4 f f f f f f 0 k k k k k 4 k k k

55 Observation table K a set of strings and F a set of contexts (λ, λ) (a, λ) (λ, b) λ a b ab

56 bcc aab bbcc bc abc aabb ab c b a λ Observation table K a set of strings and F a set of contexts (aaabb, bccc) (λ, abbccc) (aa, bbc) (abb, cc) (λ, λ) (aaabbc, λ) (aaab, bccc) (aa, bbbc) (abbb, cc)

57 bcc aab abc aabb ab λ bbcc bc c b a Observation table K a set of strings and F a set of contexts (λ, λ) (aaabb, bccc) (λ, abbccc) (aa, bbbc) (aa, bbc) (abb, cc) (aaabbc, λ) (aaab, bccc) (abbb, cc)

58 a Substitutable L = {a n cb n n 0} (a, b) (ac, b) (λ, b) (aac, b) (λ, λ) (λ, cb) (a, λ) (ac, λ) aacb ac acbb cb aacbb acb c b

59 Example Artificial f f f f 4 f f f f f f 0 k k k k k 4 k k k

60 Example Artificial f f f f 4 f f f f f f 0 k k k k k 4 k k k

61 Example Artificial f f f f 4 f f f f f f 0 k k k k k 4 k k k

62 Example Artificial f f f f 4 f f f f f f 0 k k k k k 4 k k k

63 Example Artificial f f f f 4 f f f f f f 0 k k k k k 4 k k k

64 Example Artificial f f f f 4 f f f f f f 0 k k k k k 4 k k k

65 Constructing CFG Given the observation table: Non-terminals are equivalence classes of rows: w N if N is [w] Add N PQ if there is a u P, v Q and uv N. Initial non-terminals are those rows with (λ, λ): S N iff N L

66 (λ, λ) (a, λ) (λ, b) λ 0 0 a 0 0 b 0 0 ab 0 0 aab 0 0 abb 0 0 aa ba bb bab 0 0 aba 0 0 abab 0 0 Example Dyck language

67 Example Dyck language (λ, λ) (a, λ) (λ, b) λ 0 0 ab 0 0 abab 0 0 a 0 0 aab 0 0 aba 0 0 b 0 0 abb 0 0 bab 0 0 aa ba bb Discard rows with no element in K.

68 Example Dyck language Three classes {λ, ab, abab} Call this S {a, aab, aba} A {b, bab, abb} B. Add rules S AB, S SS A AS, SA, a B BS, SB, b Note that there is no rule X AA.

69 Closure LSTAR In the LSTAR algorithm, the table needed to be closed. For every transition we needed to represent the state at the end. For every u K, a Σ, we needed a v K such that (ua) L = v L. This algorithm: non-regular language There are an infinite number of congruence classes We cannot have a closed table Doesn t matter With a DFA unique derivation With a CFG lots of different trees

70 Consistency The example is consistent: If u F u and v F v, then uv F u v. If it is not consistent then we know something is wrong and we can make it consistent by adding a feature. If (l, r) is a context that distinguishes uv and u v One of (lu, r), (l, vr), (lu, r), (l, v r) will split u, u or v, v. But it might take exponential time each context could be twice the size of the previous one. Not essential so leave it out.

71 Undergenerating If we get a counter-example w L, then we add Sub(w) to K. Lemma Each positive counter-example will give us at least one of the non-terminals in the target. Thus we will get at most n positive counterexamples for a CFG of size n

72 Overgeneralising If we have enough contexts then we won t overgeneralise. If we overgeneralise, we don t have enough contexts. Problem S w, S I, but w L. Triple w, N, (l, r) N w All elements of N should have (l, r) (l, r) C L (w)

73 FindContext Recursive Goal: find a set of strings N and a context (l, r) that splits N. N w N PQ uv = w Assume all elements of N have the feature (l, r) So we must have u v N, u P, v Q.

74 Table Possibilities luvr NO lu vr lu v r YES luv r Query the two gaps

75 Table luvr NO lu vr NO luv r YES lu v r YES v cannot be congruent to v as (lu, r) and (lu, r) C L (v ) but not C L (v).

76 Table luvr NO lu vr YES luv r NO lu v r YES u cannot be congruent to u as (l, v r) and (l, vr) C L (u ) but not C L (u).

77 Table luvr NO lu vr YES luv r YES lu v r YES u cannot be congruent to u : (l, vr) v cannot be congruent to v : (lu, r)

78 Table 4 luvr NO lu vr NO luv r NO lu v r YES u cannot be congruent to u : (l, v r) v cannot be congruent to v : (lu, r)

79 Recursion w, N, (l, r) Start at root with (λ, λ) Go down through the parse tree: At each step we have a triple such that at least one element of N has the context (l, r), but w does not. Terminate when we can split N If we get down to a leaf we will always terminate, as w N

80 4 4 Algorithm Result: A CFG G K Σ {λ}, K = K ; F {(λ, λ)}; D = L {λ} ; G = K, D, F ; while true do if Equiv(G) returns correct then return G ; w Equiv(G) ; if w is not in L(G) then K K Sub(w) ; else F AddContexts(G,w); G MakeGrammar(K, D, F) ; Algorithm : LearnCFG

81 Example Step 0 (λ, λ) λ

82 Step (λ, λ) λ a 0 b 0 ab aa 0 ba 0 bb 0 abb 0 aba 0 aab 0 bab 0 abab Example

83 Step (λ, λ) (a, λ) λ 0 a 0 0 b 0 ab 0 aa 0 0 ba 0 0 bb 0 0 abb 0 aba 0 0 aab 0 0 bab 0 abab 0 Example

84 Step (λ, λ) (a, λ) (λ, b) λ 0 0 a 0 0 b 0 0 ab 0 0 aa ba bb abb 0 0 aba 0 0 aab 0 0 bab 0 0 abab 0 0 Example

85 Result Clark, ICGI 00 Theorem This algorithm polynomially learns the class of congruential context free languages from MQs and EQs Language class A CFG is congruential if N u, N v implies u L v Observation The same approach works for positive data and membership queries alone

86 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

87 NTS Languages Boasson and Senizergues Up to now we have relied on undecidable language theoretic properties. NTS is a decidable syntactic property. Definition A grammar G is non terminally separated (NTS) iff for every N, M if N α and M uαv then M unv.

88 NTS Languages Boasson and Senizergues Up to now we have relied on undecidable language theoretic properties. NTS is a decidable syntactic property. Definition A grammar G is non terminally separated (NTS) iff for every N, M if N α and M uαv then M unv. For we like sheep have been led astray.

89 NTS Languages Boasson and Senizergues Up to now we have relied on undecidable language theoretic properties. NTS is a decidable syntactic property. Definition A grammar G is non terminally separated (NTS) iff for every N, M if N α and M uαv then M unv. For we like sheep have been led astray. Informally If there is a string like We like sheep that can be a sentence, whenever you see an occurrence of We like sheep it can be an S.

90 Congruence classes of NTS languages Congruence classes of non-terminals Suppose G is an NTS grammar, and N is a non-terminal, then if N u and N v then u L v.

91 Congruence classes of NTS languages Congruence classes of non-terminals Suppose G is an NTS grammar, and N is a non-terminal, then if N u and N v then u L v. Thus we have a partial equivalence between the congruence classes of the language, and the non-terminals. Decidable property of CFG. Decidable equivalence The MAT algorithm can learn all NTS languages.

92 PCFG Suppose we have a PCFG which defines a probability distribution D. Context distribution C D (u)[l, r] = P D(lur) E D (u) Note l,r P D(lur) = E D (u), which could be greater than.

93 Unambiguous NTS languages If we have an unambiguous NTS language then: Any two strings generated from the same non-terminal will have the same probabilistic distribution. Ambiguous NTS languages Some strings in {w N w} could have different sets of derivations. Add production N w with large parameter.

94 PAC-learning Unambiguous NTS grammars If we restrict the distributions to those generated by a PCFG with the same structure then we can learn.

95 PAC-learning Unambiguous NTS grammars If we restrict the distributions to those generated by a PCFG with the same structure then we can learn. Parameters µ -Distinguishability: same as with PDFAs

96 PAC-learning Unambiguous NTS grammars If we restrict the distributions to those generated by a PCFG with the same structure then we can learn. Parameters µ -Distinguishability: same as with PDFAs Separability: contexts are sufficiently far apart: A PCFG is ν-separable for some ν > 0 if for every pair of strings u, v in Sub(L(G)) such that u v, it is the case that L (C u C v ) ν min(l (C u ), L (C v ))

97 PAC-learning Unambiguous NTS grammars If we restrict the distributions to those generated by a PCFG with the same structure then we can learn. Parameters µ -Distinguishability: same as with PDFAs Separability: contexts are sufficiently far apart: A PCFG is ν-separable for some ν > 0 if for every pair of strings u, v in Sub(L(G)) such that u v, it is the case that L (C u C v ) ν min(l (C u ), L (C v )) Reachability: A PCFG is µ -reachable, if for every non-terminal N V there is a string u such that N G u and L (C u ) > µ.

98 Result Theorem The class of unambiguous NTS grammars is PAC-learnable, with parameters µ, ν, µ, when the distributions are generated by a PCFG with the same structure as the NTS grammar.

99 Result Theorem The class of unambiguous NTS grammars is PAC-learnable, with parameters µ, ν, µ, when the distributions are generated by a PCFG with the same structure as the NTS grammar. Requirement for unambiguity is very strong with NTS grammars. Too many stratificational parameters?

100 Outline Distributional learning Congruence classes Syntactic monoid Canonical CFG Algorithms Substitutable languages MAT learner NTS languages Conclusion

101 Includes Language class Still limited All regular languages (syntactic monoid is finite) Dyck language {a n b n n 0}... Many simple languages are not in this class: Palindromes over {a, b} L = {a n b n n 0} {a n b n n 0} L = {a n b n c m n, m 0} {a m b n c n n, m 0} (Some subtle differences in classes but we conjecture that they define the same class)

102 Linguistic modelling This seems close to the base model that is used in a lot of empirical work. Limitations Inadequate for natural language if Σ is a set of words: Exact substitutability is too strict Scholz and Pullum (00): fond versus proud cat is not perfectly substitutable with dog Words are almost never perfectly substitutable; phrases sometimes. The classes are far too small they are treated as completely unrelated

103 NP DT N company companies associate furniture sheep oil the a an this those some

104 NP DT N company companies associate furniture sheep oil the a an this those some Every determiner and noun is in a different congruence class Different rule for each combination

105 Regular language Theorem A regular languages has finitely many congruence classes. Proof Take a DFA for the language L with state set q Define f w : Q Q such that f w (q) = δ(q, w) There are finitely many functions and if f u = f v then u L v. Number of congruence classes may be exponential in size of minimal DFA: Q Q.

106 Monoid of regular languages f uv = f u f v If u has δ(q, u) = q and v has δ(q, v) = q then uv has δ(q, uv) = q There is some structure in the congruence classes; we can view the function as being composed of simpler basis functions.

107 Summary Context free Congruence based approach Define a set of primitives Congruence classes Derivation relation Syntactic monoid Language class Subclass of CF languages Inference algorithms Testing congruence We have made the change from regular to CF but results are too weak.