Context-Free Parsing: CKY & Earley Algorithms and Probabilistic Parsing

Transcription

1 Context-Free Parsing: CKY & Earley Algorithms and Probabilistic Parsing Natural Language Processing CS 4120/6120 Spring 2016 Northeastern University David Smith with some slides from Jason Eisner & Andrew McCallum

2 Language structure and meaning We want to know how meaning is mapped onto what language structures. Commonly in English in ways like this: [Thing The dog] is [Place in the garden] [Thing The dog] is [Property fierce] [Action [Thing The dog] is chasing [Thing the cat]] [State [Thing The dog] was sitting [Place in the garden] [Time yesterday]] [Action [Thing We] ran [Path out into the water]] [Action [Thing The dog] barked [Property/Manner loudly]] [Action [Thing The dog] barked [Property/Amount nonstop for five hours]]

3 Language structure and meaning We want to know how meaning is mapped onto what language structures. Commonly in English in ways like this: [Thing The dog] is [Place in the garden] [Thing The dog] is [Property fierce] [Action [Thing The dog] is chasing [Thing the cat]] [State [Thing The dog] was sitting [Place in the garden] [Time yesterday]] [Action [Thing We] ran [Path out into the water]] [Action [Thing The dog] barked [Property/Manner loudly]] [Action [Thing The dog] barked [Property/Amount nonstop for five hours]]

4 Part of speech Substitution Test The {sad, intelligent, green, fat,...} one is in the corner.

5 Constituency The idea: Groups of words may behave as a single unit or phrase, called a consituent. E.g. Noun Phrase Kermit the frog they December twenty-sixth the reason he is running for president

6 Constituency Sentences have parts, some of which appear to have subparts. groupings of words that go together we will call constituents. These (How do we know they go together? Coming in a few slides...) I hit the man with a cleaver I hit [the man with a cleaver] I hit [the man] with a cleaver You could not go to her party You [could not] go to her party You could [not go] to her party

7 Constituent Phrases For constituents, we usually name them as phrases based on the word that heads the constituent: the man from Amherst extremely clever down the river killed the rabbit is a Noun Phrase (NP) because the head man is a noun is an Adjective Phrase (AP) because the head clever is an adjective is a Prepositional Phrase (PP) because the head down is a preposition is a Verb Phrase (VP) because the head killed is a verb Note that a word is a constituent (a little one). Sometimes words also act as phrases. In: Joe grew potatoes. Joe and potatoes are both nouns and noun phrases. Compare with: The man from Amherst grew beautiful russet potatoes. We say Joe counts as a noun phrase because it appears in a place that a larger noun phrase could have been.

8 Evidence constituency exists 1. They appear in similar environments (before a verb) Kermit the frog comes on stage They come to Massachusetts every summer December twenty-sixth comes after Christmas The reason he is running for president comes out only now. But not each individual word in the consituent *The comes our... *is comes out... *for comes out The constituent can be placed in a number of di erent locations Consituent = Prepositional phrase: On December twenty-sixth On December twenty-sixth I d like to fly to Florida. I d like to fly on December twenty-sixth to Florida. I d like to fly to Florida on December twenty-sixth. But not split apart *On December I d like to fly twenty-sixth to Florida. *On I d like to fly December twenty-sixth to Florida.

9 Context-free grammar The most common way of modeling constituency. CFG = Context-Free Grammar = Phrase Structure Grammar = BNF = Backus-Naur Form The idea of basing a grammar on constituent structure dates back to Wilhem Wundt (1890), but not formalized until Chomsky (1956), and, independently, by Backus (1959).

10 Context-free grammar G = T,N,S,R T is set of terminals (lexicon) N is set of non-terminals For NLP, we usually distinguish out a set P N of preterminals which always rewrite as terminals. S is start symbol (one of the nonterminals) R is rules/productions of the form X, where X is a nonterminal and is a sequence of terminals and nonterminals (may be empty). A grammar G generates a language L.

11 An example context-free grammar G = T,N,S,R T = {that, this, a, the, man, book, flight, meal, include, read, does} N = {S, NP, NOM, VP, Det, Noun, Verb, Aux} S = S R = { S NP VP S Aux NP VP S VP NP Det NOM NOM Noun NOM Noun NOM VP Verb VP Verb NP Det that this a the Noun book flight meal man Verb book include read Aux does }

12 Application of grammar rewrite rules S NP VP Det that this a the S Aux NP VP Noun book flight meal man S VP Verb book include read NP Det NOM Aux does NOM Noun NOM Noun NOM VP Verb VP Verb NP S NP VP Det NOM VP The NOM VP The Noun VP The man VP The man Verb NP The man read NP The man read Det NOM The man read this NOM The man read this Noun The man read this book

13 Parse tree Det The NP NOM Noun S Verb read VP Det NP NOM man this Noun book

14 CFGs can capture recursion Example of seemingly endless recursion of embedded prepositional phrases: PP Prep NP NP Noun PP [ S The mailman ate his [ NP lunch [ PP with his friend [ PP from the cleaning sta [ PP of the building [ PP at the intersection [ PP on the north end [ PP of town]]]]]]]. (Bracket notation)

15 Grammaticality A CFG defines a formal language = the set of all sentences (strings of words) that can be derived by the grammar. Sentences in this set said to be grammatical. Sentences outside this set said to be ungrammatical.

16 The Chomsky hierarchy Type 0 Languages / Grammars Rewrite rules where and are any string of terminals and nonterminals Context-sensitive Languages / Grammars Rewrite rules X where X is a non-terminal, and,, are any string of terminals and nonterminals, ( must be non-empty). Context-free Languages / Grammars Rewrite rules X where X is a nonterminal and is any string of terminals and nonterminals Regular Languages / Grammars Rewrite rules X Y where X, Y are single nonterminals, and might be missing. is a string of terminals; Y

17 Parsing regular grammars (Languages that can be generated by finite-state automata.) Finite state automaton regular expression regular grammar Space needed to parse: constant Time needed to parse: linear (in the length of the input string) Cannot do embedded recursion, e.g. a n b n. (Context-free grammars can.) In the language: ab, aaabbb; not in the language: aabbb The cat likes tuna fish. The cat the dog chased likes tuna fish The cat the dog the boy loves chased likes tuna fish. John, always early to rise, even after a sleepless night filled with the cries of the neighbor s baby, goes running every morning. John and Mary, always early to rise, even after a sleepless night filled with the cries of the neighbor s baby, go running every morning.

18 Parsing context-free grammars (Languages that can be generated by pushdown automata.) Widely used for surface syntax description (correct word order specification) in natural languages. Space needed to parse: stack (sometimes a stack of stacks) In general, proportional to the number of levels of recursion in the data. Time needed to parse: in general O(n 3 ). Can to a n b n, but cannot do a n b n c n. Chomsky Normal Form All rules of the form X YZ or X a or S. (S is the only non-terminal that can go to.) Any CFG can be converted into this form. How would you convert the rule W XY az to Chomsky Normal Form?

19 Chomsky Normal Form Conversion These steps are used in the conversion: 1. Make S non-recursive 2. Eliminate all epsilon except the one in S (if there is one) 3. Eliminate all chain rules 4. Remove useless symbols (the ones not used in any production). How would you convert the following grammar? S ABS S A A xyz B wb B v

20 Parsing context-sensitive grammars (Languages that can be recognized by a non-deterministic Turing machine whose tape is bounded by a constant times the length of the input.) Natural languages are really not context-free: e.g. pronouns more likely in Object rather than Subject of a sentence. But parsing is PSPACE-complete! (Recognized by a Turing machine using a polynomial amount of memory, and unlimited time.) Often work with mildly context-sensitive grammars. More on this next week. E.g. Tree-adjoining grammars. Time needed to parse, e.g. O(n 6 ) or O(n 5 )...

21 Bottom-up versus Top-down science empiricist Britain: Francis Bacon, John Locke Knowledge is induced and reasoning proceeds based on data from the real world. rationalist Continental Europe: Descartes Learning and reasoning is guided by prior knowledge and innate ideas.

22 What is parsing? We want to run the grammar backwards to find the structure. Parsing can be viewed as a search problem. We search through the legal rewritings of the grammar. We want to find all structures matching an input string of words (for the moment) We can do this bottom-up or top-down This distinction is independent of depth-first versus breadth-first; we can do either both ways. Doing this we build a search tree which is di erent from the parse tree.

23 Recognizers and parsers A recognizer is a program for which a given grammar and a given sentence returns YES if the sentence is accepted by the grammar (i.e., the sentence is in the language), and NO otherwise. A parser in addition to doing the work of a recognizer also returns the set of parse trees for the string.

24 Soundness and completeness A parser is sound if every parse it returns is valid/correct. A parser terminates if it is guaranteed not to go o into an infinite loop. A parser is complete if for any given grammar and sentence it is sound, produces every valid parse for that sentence, and terminates. (For many cases, we settle for sound but incomplete parsers: probabilistic parsers that return a k-best list.) e.g.

25 Top-down parsing is goal-directed. Top-down parsing A top-down parser starts with a list of constituents to be built. It rewrites the goals in the goal list by matching one against the LHS of the grammar rules, and expanding it with the RHS,...attempting to match the sentence to be derived. If a goal can be rewritten in several ways, then there is a choice of which rule to apply (search problem) Can use depth-first or breadth-first search, and goal ordering.

26 Top-down parsing example (Breadth-first) S NP VP Det that this a the S Aux NP VP Noun book flight meal man S VP Verb book include read NP Det NOM Aux does NOM Noun NOM Noun NOM VP Verb VP Verb NP Book that flight. (Work out top-down, breadth-first search on the board...)

27 Top-down parsing example (Breadth-first) S NP S VP S Aux NP VP S VP Det S NP VP NOM Verb Det NP NOM S VP Verb NP... S VP Verb S VP Verb VP... S Verb book VP NP Det NOM that Noun flight

28 Problems with top-down parsing Left recursive rules... e.g. NP NP PP... lead to infinite recursion Will do badly if there are many di erent rules for the same LHS. Consider if there are 600 rules for S, 599 of which start with NP, but one of which starts with a V, and the sentence starts with a V. Useless work: expands things that are possible top-down but not there (no bottom-up evidence for them). Top-down parsers do well if there is useful grammar-driven control: search is directed by the grammar. Top-down is hopeless for rewriting parts of speech (preterminals) with words (terminals). In practice that is always done bottom-up as lexical lookup. Repeated work: anywhere there is common substructure.

29 Top-down parsing is data-directed. Bottom-up parsing The initial goal list of a bottom-up parser is the string to be parsed. If a sequence in the goal list matches the RHS of a rule, then this sequence may be replaced by the LHS of the rule. Parsing is finished when the goal list contains just the start symbol. If the RHS of several rules match the goal list, then there is a choice of which rule to apply (search problem) Can use depth-first or breadth-first search, and goal ordering. The standard presentation is as shift-reduce parsing.

30 Shift Reduce Parser Start with the sentence to be parsed in an input bu er. a shift action correponds to pushing the next input symbol from the bu er onto the stack a reduce action occurrs when we have a rule s RHS on top of the stack. To perform the reduction, we pop the rule s RHS o the stack and replace it with the terminal on the LHS of the corresponding rule. (When either shift or reduce is possible, choose one arbitrarily.) If you end up with only the Start symbol on the stack, then success! If you don t, and you cannot and no shift or reduce actions are possible, backtrack.

31 Bottom-up parsing example S NP VP Det that this a the S Aux NP VP Noun book flight meal man S VP Verb book include read NP Det NOM Aux does NOM Noun NOM Noun NOM VP Verb VP Verb NP Book that flight. (Work out bottom-up search on the board...)

32 Shift-reduce parsing Stack Input remaining Action () Book that flight shift (Book) that flight reduce, Verb book, (Choice #1 of 2) (Verb) that flight shift (Verb that) flight reduce, Det that (Verb Det) flight shift (Verb Det flight) reduce, Noun flight (Verb Det Noun) reduce, NOM Noun (Verb Det NOM) reduce, NP Det NOM (Verb NP) reduce, VP Verb NP (Verb) reduce, S V (S) SUCCESS! Ambiguity may lead to the need for backtracking.

33 Shift Reduce Parser In a top-down parser, the main decision was which production rule to pick. In a bottom-up shift-reduce parser there are two decisions: 1. Should we shift another symbol, or reduce by some rule? 2. If reduce, then reduce by which rule? both of which can lead to the need to backtrack

34 Problems with bottom-up parsing Unable to deal with empty categories: termination problem, unless rewriting empties as constituents is somehow restricted (but then it s generally incomplete) Useless work: locally possible, but globally impossible. Ine cient when there is great lexical ambiguity (grammar-driven control might help here). Conversely, it is data-directed: it attempts to parse the words that are there. Repeated work: anywhere there is common substructure. Both Top-down (LL) and Bottom-up (LR) parsers can (and frequently do) do work exponential in the sentence length on NLP problems.

35 Principles for success Left recursive structures must be found, not predicted. Empty categories must be predicted, not found. Don t waste e ort re-working what was previously parsed before backtracking. An alternative way to fix things: Grammar transformations can fix both left-recursion and epsilon productions. Then you parse the same language but with di erent trees. BUT linguists tend to hate you, because the structure of the re-written grammar isn t what they wanted.

36 From Shift-Reduce to CKY Shift-reduce parsing can make wrong turns, needs backtracking Shift-reduce must pop the top of the stack, but how many items to pop? Time-space tradeoff Chomsky normal form

37 Chomsky Normal Form Any CFL can be generated by an equivalent grammar in CNF Rules of three types X Y Z X, Y, Z nonterminals X a X nonterminal, a terminal S ε S the start symbol NB: the derivation of a given string may change

38 CNF Conversion Create new start symbol Remove NTs that can generate epsilon Remove NTs that can generate each other, (unary rule cycles) Chain rules with RHS > 2 Related topic: rule Markovization (later)

39 CKY Algorithm Input: string of n words Output (of recognizer): grammatical or not Dynamic programming in a chart: rows labeled 0 to n-1 columns labeled 1 to n cell [i,j] lists possible constituents spanning words between i and j

40 CKY Algorithm! for i := 1 to n! Add to [i-1,i] all (part-of-speech) categories for the i th word! for width := 2 to n! for start := 0 to n-width! Define end := start + width! for mid := start+1 to end-1! for every constituent X in [start,mid]! for every constituent Y in [mid,end]! for all ways of combining X and Y (if any)! Add the resulting constituent to [start,end] if it s not already there.

41 CKY Algorithm! for i := 1 to n! Add to [i-1,i] all (part-of-speech) categories for the i th word! for width := 2 to n! for start := 0 to n-width! Define end := start + width! for mid := start+1 to end-1! for every constituent X in [start,mid]! for every constituent Y in [mid,end]! for all ways of combining X and Y (if any)! Add the resulting constituent to [start,end] if it s not already there. Time complexity?

42 CKY Algorithm! for i := 1 to n! Add to [i-1,i] all (part-of-speech) categories for the i th word! for width := 2 to n! for start := 0 to n-width! Define end := start + width! for mid := start+1 to end-1! for every constituent X in [start,mid]! for every constituent Y in [mid,end]! for all ways of combining X and Y (if any)! Add the resulting constituent to [start,end] if it s not already there. Time complexity? O(Gn 3 )

43 CKY Algorithm! for i := 1 to n! Add to [i-1,i] all (part-of-speech) categories for the i th word! for width := 2 to n! for start := 0 to n-width! Define end := start + width! for mid := start+1 to end-1! for every constituent X in [start,mid]! for every constituent Y in [mid,end]! for all ways of combining X and Y (if any)! Add the resulting constituent to [start,end] if it s not already there. Time complexity? O(Gn 3 ) Space complexity?

44 CKY Algorithm! for i := 1 to n! Add to [i-1,i] all (part-of-speech) categories for the i th word! for width := 2 to n! for start := 0 to n-width! Define end := start + width! for mid := start+1 to end-1! for every constituent X in [start,mid]! for every constituent Y in [mid,end]! for all ways of combining X and Y (if any)! Add the resulting constituent to [start,end] if it s not already there. Time complexity? O(Gn 3 ) Space complexity? O(Tn 2 )

45 time 1 flies 2 like 3 an 4 arrow 5 NP 3 Vst 3 0 NP! time Vst! time NP! flies VP! flies P! like V! like Det! an N! arrow NP 4 VP 4 P 2 V 5 Det 1 N 8 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

46 time 1 flies 2 like 3 an 4 arrow 5 NP 3 Vst NP 4 VP 4 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 Det 1 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

47 time 1 flies 2 like 3 an 4 arrow 5 NP 3 Vst 3 NP NP 4 VP 4 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 Det 1 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

48 time 1 flies 2 like 3 an 4 arrow 5 NP 3 Vst 3 NP 10 S NP 4 VP 4 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 Det 1 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

49 time 1 flies 2 like 3 an 4 arrow 5 0 NP 3 Vst 3 NP 10 S 8 S 13 1 NP 4 VP 4 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 Det 1 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

50 time 1 flies 2 like 3 an 4 arrow 5 0 NP 3 Vst 3 NP 10 S 8 S 13 1 NP 4 VP 4 _ 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

51 time 1 flies 2 like 3 an 4 arrow 5 0 NP 3 Vst 3 NP 10 S 8 S 13 1 NP 4 VP 4 _ 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

52 time 1 flies 2 like 3 an 4 arrow 5 0 NP 3 Vst 3 NP 10 S 8 S 13 1 NP 4 VP 4 _ 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

53 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 _ Vst 3 S 8 S NP 4 VP 4 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

54 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 _ Vst 3 S 8 S NP 4 VP 4 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

55 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 _ Vst 3 S 8 S NP 4 VP 4 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

56 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S NP 4 VP 4 NP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

57 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S NP 4 VP 4 NP 18 S 21 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

58 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S NP 4 VP 4 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

59 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 Vst 3 S 8 S NP 4 VP 4 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

60 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S NP 4 VP 4 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

61 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S NP 4 VP 4 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

62 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S 13 S NP 4 VP 4 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

63 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S 13 S NP 4 VP 4 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

64 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S 13 S 27 0 NP 24 1 NP 4 VP 4 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

65 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S 13 S 27 0 NP 24 1 NP 4 VP 4 S 27 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

66 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S 13 S 27 0 NP 24 1 NP 4 VP 4 S 27 S 22 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

67 time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S 13 S 27 0 NP 24 1 NP 4 VP 4 S 27 S 22 S 27 NP 18 S 21 VP 18 1 S! NP VP 6 S! Vst NP 2 S! S PP 1 VP! V NP 2 VP! VP PP P 2 V 5 _ Det 1 PP 12 VP 16 NP 10 N 8 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

68 Follow backpointers S time 1 flies 2 like 3 an 4 arrow 5 NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S 13 S 27 0 NP 24 S 27 S 22 1 NP 4 VP 4 S 27 NP 18 S 21 1 S! NP VP 6 S! Vst NP 2 S! S PP P 2 V 5 _ Det 1 VP 18 PP 12 VP 16 NP 10 N 8 1 VP! V NP 2 VP! VP PP 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

69 S time 1 flies 2 like 3 an 4 arrow 5 NP VP NP 3 NP 10 NP 24 Vst 3 S 8 S 22 S 13 S 27 0 NP 24 S 27 S 22 1 NP 4 VP 4 S 27 NP 18 S 21 1 S! NP VP 6 S! Vst NP 2 S! S PP P 2 V 5 _ Det 1 VP 18 PP 12 VP 16 NP 10 N 8 1 VP! V NP 2 VP! VP PP 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

70 S time 1 flies 2 like 3 an 4 arrow 5 NP VP NP 3 Vst 3 NP 10 S 8 NP 24 S 22 VP PP S 13 S 27 0 NP 24 S 27 S 22 1 NP 4 VP 4 S 27 NP 18 S 21 1 S! NP VP 6 S! Vst NP 2 S! S PP P 2 V 5 _ Det 1 VP 18 PP 12 VP 16 NP 10 N 8 1 VP! V NP 2 VP! VP PP 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

71 S time 1 flies 2 like 3 an 4 arrow 5 NP VP NP 3 Vst 3 NP 10 S 8 NP 24 S 22 VP PP 0 S 13 S 27 NP 24 P NP S 27 S 22 1 NP 4 VP 4 S 27 NP 18 S 21 1 S! NP VP 6 S! Vst NP 2 S! S PP P 2 V 5 _ Det 1 VP 18 PP 12 VP 16 NP 10 N 8 1 VP! V NP 2 VP! VP PP 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

72 S time 1 flies 2 like 3 an 4 arrow 5 NP VP NP 3 Vst 3 NP 10 S 8 NP 24 S 22 VP PP 0 S 13 S 27 NP 24 P NP S 27 S 22 Det N 1 NP 4 VP 4 S 27 NP 18 S 21 1 S! NP VP 6 S! Vst NP 2 S! S PP P 2 V 5 _ Det 1 VP 18 PP 12 VP 16 NP 10 N 8 1 VP! V NP 2 VP! VP PP 1 NP! Det N 2 NP! NP PP 3 NP! NP NP 0 PP! P NP

73 Treebank Grammars What rules would you extract from this tree? What probabilities would you assign them? Det The NP NOM Noun S Verb read VP Det NP NOM man this Noun book

74 Treebank Grammars Penn Treebank Lots of rules have high fanout (flat phrases) Lots of unary cycles How should we evaluate? What are the consequences of CNF conversion?

75 Parsing as Deduction CKY as inference rules CKY as Prolog program But Prolog is top-down with backtracking i.e., backward chaining, CKY is forward chaining Inference rules as Boolean semiring

76 Probabilistic CFGs Generative process (already familiar) It s context free: Rules are applied independently, therefore we multiply their probabilities How to estimate probabilities? Supervised and unsupervised

77 Questions for PCFGs What is the most likely parse for a sentence? (parsing) What is the probability of a sentence? (language modeling) What rule probabilities maximize the probability of a sentence? (parameter estimation)

78 Algorithms for PCFGs Exact analogues to HMM algorithms Parsing: Viterbi CKY Language modeling: inside probabilities Parameter estimation: inside-outside probabilities with EM

79 Parsing as Deduction A, B, C N,W V,0 i, j, k m constit(b, i, j) constit(c, j, k) A BC constit(a, i, k) word(w, i) A W constit(a, i, i + 1) In Prolog: constit(a, I1, I) :- word(i, W), (A ---> [W]), I1 is I - 1. constit(a, I, K) :- constit(b, I, J), constit(c, J, K), (A ---> [B, C]). But Prolog uses top-down search with backtracking...

80 Parsing as Deduction A, B, C N,W V,0 i, j, k m constit(b, i, j) constit(c, j, k) A BC constit(a, i, k) word(w, i) A W constit(a, i, i + 1) constit(a, i, k) = _ B,C,j constit(b,i,j) ^ constit(c, j, k) ^ A! BC constit(a, i, j) = _ W word(w, i, j) ^ A! W

81 Parsing as Deduction constit(a, i, k) = _ B,C,j constit(b,i,j) ^ constit(c, j, k) ^ A! BC constit(a, i, j) = _ W word(w, i, j) ^ A! W score(constit(a, i, k)) = max B,C,j score(constit(b,i,j)) score(constit(a, i, j)) = max W score(constit(c, j, k)) score(a! BC) score(word(w, i, j)) score(a! W ) And how about the inside algorithm?

82 Inside & Viterbi Algorithms Let A(i, j) = p(constit(a, i, j)) NB: index between words; M&S index words = p(w ij nonterminal A from i to j) A(i, k) = B,C,j B(i, j) C(j, k) p(a BC) Let A(i, j) = p best (constit(a, i, j)) A(i, k) = max B,C,j B (i, j)) C(j, k) p(a BC) S(0,n) =? S(0,n) =?

83 Forward-Backward Algorithm NN αnn(2) a[vb NN]b[interest VB] a[nn VB]b[rates NN] βnn(4) NNS αnns(2) a[vb NNS]b[interest VB] a[nns VB]b[rates NNS] βnns(4) NNP αnnp(2) a[vb NNP]b[interest VB] a[nnp VB]b[rates NNP] βnnp(4) VB αvb(2) a[vb VB]b[interest VB] a[vb VB]b[rates VB] βvb(4) VBZ αvbz(2) a[vb VBZ]b[interest VB] a[vbz VB]b[rates VBZ] βvbz(4) Fed raises interest rates

84 Inside & Outside constit(a, i, j) p(words 0-i, words j-n, constit) w(0, 1) w(i-1, i) w(j, j+1) w(n-1, n)

85 Inside & Outside constit(a, i, j) p(words 0-i, words j-n, constit) Inside: p(words i-j constit) w(0, 1) w(i-1, i) w(j, j+1) w(n-1, n)

86 Inside & Outside constit(a, i, j) p(words 0-i, words j-n, constit) Inside: p(words i-j constit) w(0, 1) w(i-1, i) w(j, j+1) w(n-1, n)

87 Inside & Outside constit(a, i, j) Inside x Outside = Inside probability of the whole tree p(words 0-i, words j-n, constit) Inside: p(words i-j constit) w(0, 1) w(i-1, i) w(j, j+1) w(n-1, n)

88 Outside Algorithm A(i, j) =p(w 0,i,A i,j,w j,n ) Uses inside probs. A(i, j) = S(0,n) =? PP(0,n) =? n B,C,k=j + i B,C,k=0 B(i, k) C (j, k) p(b AC) B(k, j) C (k, i) p(b CA) Some resemblance to derivative product rule

89 Problems with Inside-Outside EM Each sentence at each iteration takes O(m 3 n 3 ) Local maxima even more problematic than for HMMs: Charniak (1993) found a different maximum for each of 300 trials More NTs needed to learn a good model NTs don t correspond to intuitions: HMMs are easier to constrain with tag dictionaries

90 Top-Down/Bottom-Up Top-down parsers Can get caught in infinite loops Take exponential time backtracking CKY Needs Chomsky normal form Builds all possible constituents

91 Earley Parser (1970) Nice combination of dynamic programming incremental interpretation avoids infinite loops no restrictions on the form of the context-free grammar. A! B C the D of causes no problems O(n 3 ) worst case, but faster for many grammars Uses left context and optionally right context to constrain search.

92 Earley s Overview Finds constituents and partial constituents in input A! B C. D E is partial: only the first half of the A A B C D E D + = A B C D E A! B C. D E A! B C D. E

93 Earley s Overview Proceeds incrementally left-to-right Before it reads word 5, it has already built all hypotheses that are consistent with first 4 words Reads word 5 & attaches it to immediately preceding hypotheses. Might yield new constituents that are then attached to hypotheses immediately preceding them E.g., attaching D to A! B C. D E gives A! B C D. E Attaching E to that gives A! B C D E. Now we have a complete A that we can attach to hypotheses immediately preceding the A, etc.

94 The Parse Table Columns 0 through n corresponding to the gaps between words Entries in column 5 look like (3, NP! NP. PP) (but we ll omit the! etc. to save space) Built while processing word 5 Means that the input substring from 3 to 5 matches the initial NP portion of a NP! NP PP rule Dot shows how much we ve matched as of column 5 Perfectly fine to have entries like (3, VP! is it. true that S)

95 The Parse Table Entries in column 5 look like (3, NP! NP. PP) What will it mean that we have this entry? Unknown right context: Doesn t mean we ll necessarily be able to find a VP starting at column 5 to complete the S. Known left context: Does mean that some dotted rule back in column 3 is looking for an S that starts at 3. So if we actually do find a VP starting at column 5, allowing us to complete the S, then we ll be able to attach the S to something. And when that something is complete, it too will have a customer to its left In short, a top-down (i.e., goal-directed) parser: it chooses to start building a constituent not because of the input but because that s what the left context needs. In the spoon, won t build spoon as a verb because there s no way to use a verb there. So any hypothesis in column 5 could get used in the correct parse, if words 1-5 are continued in just the right way by words 6-n.

96 Earley s as a Recognizer Add ROOT!. S to column 0. For each j from 0 to n: For each dotted rule in column j, (including those we add as we go!) look at what s after the dot: If it s a word w, SCAN: If w matches the input word between j and j+1, advance the dot and add the resulting rule to column j+1 If it s a non-terminal X, PREDICT: Add all rules for X to the bottom of column j, wth the dot at the start: e.g. X!. Y Z If there s nothing after the dot, ATTACH: We ve finished some constituent, A, that started in column I<j. So for each rule in column j that has A after the dot: Advance the dot and add the result to the bottom of column j. Output yes just if last column has ROOT! S. NOTE: Don t add an entry to a column if it s already there!

97 Earley s Summary Process all hypotheses one at a time in order. (Current hypothesis is shown in blue.) This may add new hypotheses to the end of the to-do list, or try to add old hypotheses again. Process a hypothesis according to what follows the dot: If a word, scan input and see if it matches If a nonterminal, predict ways to match it (we ll predict blindly, but could reduce # of predictions by looking ahead k symbols in the input and only making predictions that are compatible with this limited right context) If nothing, then we have a complete constituent, so attach it to all its customers

98 A (Whimsical) Grammar S! NP VP NP! Papa NP! Det N N! caviar NP! NP PP N! spoon VP! V NP V! ate VP! VP PP P! with PP! P NP Det! the Det! a An Input Sentence Papa ate the caviar with a spoon.

99 0 initialize 0 ROOT. S Remember this stands for (0, ROOT!. S)

100 0 0 ROOT. S 0 S. NP VP predict the kind of S we are looking for Remember this stands for (0, S!. NP VP)

101 0 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa predict the kind of NP we are looking for (actually we ll look for 3 kinds: any of the 3 will do)

102 0 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a predict the kind of Det we are looking for (2 kinds)

103 0 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a predict the kind of NP we re looking for but we were already looking for these so don t add duplicate goals! Note that this happened when we were processing a left-recursive rule.

104 0 Papa 1 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a 0 NP Papa. scan: the desired word is in the input!

105 0 Papa 1 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a 0 NP Papa. scan: failure

106 0 Papa 1 0 ROOT. S 0 S. NP VP 0 NP. Det N 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a 0 NP Papa. scan: failure

107 0 Papa 1 0 ROOT. S 0 NP Papa. 0 S. NP VP 0 S NP. VP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 0 NP. Papa 0 Det. the 0 Det. a attach the newly created NP (which starts at 0) to its customers (incomplete constituents that end at 0 and have NP after the dot)

108 0 Papa 1 0 ROOT. S 0 NP Papa. 0 S. NP VP 0 S NP. VP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 1 VP. V NP 0 NP. Papa 1 VP. VP PP 0 Det. the 0 Det. a predict

109 0 Papa 1 0 ROOT. S 0 NP Papa. 0 S. NP VP 0 S NP. VP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 1 VP. V NP 0 NP. Papa 1 VP. VP PP 0 Det. the 1 PP. P NP 0 Det. a predict

110 0 Papa 1 0 ROOT. S 0 NP Papa. 0 S. NP VP 0 S NP. VP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 1 VP. V NP 0 NP. Papa 1 VP. VP PP 0 Det. the 1 PP. P NP 0 Det. a 1 V. ate predict

111 0 Papa 1 0 ROOT. S 0 NP Papa. 0 S. NP VP 0 S NP. VP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 1 VP. V NP 0 NP. Papa 1 VP. VP PP 0 Det. the 1 PP. P NP 0 Det. a 1 V. ate predict

112 0 Papa 1 0 ROOT. S 0 NP Papa. 0 S. NP VP 0 S NP. VP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 1 VP. V NP 0 NP. Papa 1 VP. VP PP 0 Det. the 1 PP. P NP 0 Det. a 1 V. ate 1 P. with predict

113 0 Papa 1 ate 2 0 ROOT. S 0 NP Papa. 1 V ate. 0 S. NP VP 0 S NP. VP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 1 VP. V NP 0 NP. Papa 1 VP. VP PP 0 Det. the 1 PP. P NP 0 Det. a 1 V. ate 1 P. with scan: success!

114 0 Papa 1 ate 2 0 ROOT. S 0 NP Papa. 1 V ate. 0 S. NP VP 0 S NP. VP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 1 VP. V NP 0 NP. Papa 1 VP. VP PP 0 Det. the 1 PP. P NP 0 Det. a 1 V. ate 1 P. with scan: failure

115 0 Papa 1 ate 2 0 ROOT. S 0 NP Papa. 1 V ate. 0 S. NP VP 0 S NP. VP 1 VP V. NP 0 NP. Det N 0 NP NP. PP 0 NP. NP PP 1 VP. V NP 0 NP. Papa 1 VP. VP PP 0 Det. the 1 PP. P NP 0 Det. a 1 V. ate 1 P. with attach

116 0 Papa 1 ate 2 0 ROOT. S 0 NP Papa. 1 V ate. 0 S. NP VP 0 S NP. VP 1 VP V. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 0 Det. a 1 V. ate 1 P. with predict

117 0 Papa 1 ate 2 0 ROOT. S 0 NP Papa. 1 V ate. 0 S. NP VP 0 S NP. VP 1 VP V. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with predict (these next few steps should look familiar)

118 0 Papa 1 ate 2 0 ROOT. S 0 NP Papa. 1 V ate. 0 S. NP VP 0 S NP. VP 1 VP V. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with predict

119 0 Papa 1 ate 2 0 ROOT. S 0 NP Papa. 1 V ate. 0 S. NP VP 0 S NP. VP 1 VP V. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with scan (this time we fail since Papa is not the next word)

120 0 Papa 1 ate 2 the 3 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 0 S. NP VP 0 S NP. VP 1 VP V. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with scan: success!

121 0 Papa 1 ate 2 the 3 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 0 S. NP VP 0 S NP. VP 1 VP V. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with

122 0 Papa 1 ate 2 the 3 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 0 NP. Det N 0 NP NP. PP 2 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with

123 0 Papa 1 ate 2 the 3 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with

124 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with

125 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with

126 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with attach

127 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 0 Det. a 1 V. ate 2 Det. a 1 P. with attach (again!)

128 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 1 P. with attach (again!)

129 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with

130 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. attach (again!)

131 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S.

132 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with

133 0 Papa 1 ate 2 the 3 caviar 4 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with

134 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 4 P with. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with

135 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 4 P with. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with

136 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 4 P with. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 5 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with

137 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 4 P with. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 5 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 5 Det. the 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 5 Det. a 1 P. with 0 ROOT S. 4 P. with

138 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 4 P with. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 5 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 5 Det. the 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 5 Det. a 1 P. with 0 ROOT S. 4 P. with

139 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 4 P with. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 5 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 5 Det. the 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 5 Det. a 1 P. with 0 ROOT S. 4 P. with

140 0 Papa 1 ate 2 the 3 caviar 4 with 5 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 4 P with. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 5 NP. Papa 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 5 Det. the 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 5 Det. a 1 P. with 0 ROOT S. 4 P. with

141 e 2 the 3 caviar 4 with 5 a 6 1 V ate. 2 Det the. 3 N caviar. 4 P with. 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP P 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP P 2 NP. Papa 0 S NP VP. 5 NP. Papa 2 Det. the 1 VP VP. PP 5 Det. the 2 Det. a 4 PP. P NP 5 Det. a 0 ROOT S. 4 P. with 5 Det a.

142 e 2 the 3 caviar 4 with 5 a 6 1 V ate. 2 Det the. 3 N caviar. 4 P with. 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP P 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP P 2 NP. Papa 0 S NP VP. 5 NP. Papa 2 Det. the 1 VP VP. PP 5 Det. the 2 Det. a 4 PP. P NP 5 Det. a 0 ROOT S. 4 P. with 5 Det a. 5 NP Det. N

143 e 2 the 3 caviar 4 with 5 a 6 1 V ate. 2 Det the. 3 N caviar. 4 P with. 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP P 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP P 2 NP. Papa 0 S NP VP. 5 NP. Papa 2 Det. the 1 VP VP. PP 5 Det. the 2 Det. a 4 PP. P NP 5 Det. a 0 ROOT S. 4 P. with 5 Det a. 5 NP Det. N 6 N. caviar 6 N. spoon

144 e 2 the 3 caviar 4 with 5 a 6 1 V ate. 2 Det the. 3 N caviar. 4 P with. 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP P 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP P 2 NP. Papa 0 S NP VP. 5 NP. Papa 2 Det. the 1 VP VP. PP 5 Det. the 2 Det. a 4 PP. P NP 5 Det. a 0 ROOT S. 4 P. with 5 Det a. 5 NP Det. N 6 N. caviar 6 N. spoon

145 e 2 the 3 caviar 4 with 5 a 6 spoon 7 1 V ate. 2 Det the. 3 N caviar. 4 P with. 5 Det a. 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 5 NP Det. N P 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 6 N. caviar 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP 6 N. spoon P 2 NP. Papa 0 S NP VP. 5 NP. Papa 2 Det. the 1 VP VP. PP 5 Det. the 2 Det. a 4 PP. P NP 5 Det. a 0 ROOT S. 4 P. with 6 N spoon.

146 e 2 the 3 caviar 4 with 5 a 6 spoon 7 1 V ate. 2 Det the. 3 N caviar. 4 P with. 5 Det a. 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 5 NP Det. N P 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 6 N. caviar 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP 6 N. spoon P 2 NP. Papa 0 S NP VP. 5 NP. Papa 2 Det. the 1 VP VP. PP 5 Det. the 2 Det. a 4 PP. P NP 5 Det. a 0 ROOT S. 4 P. with 6 N spoon. 5 NP Det N.

147 e 2 the 3 caviar 4 with 5 a 6 spoon 7 1 V ate. 2 Det the. 3 N caviar. 4 P with. 5 Det a. 1 VP V. NP 2 NP Det. N 2 NP Det N. 4 PP P. NP 5 NP Det. N P 2 NP. Det N 3 N. caviar 1 VP V NP. 5 NP. Det N 6 N. caviar 2 NP. NP PP 3 N. spoon 2 NP NP. PP 5 NP. NP PP 6 N. spoon P 2 NP. Papa 0 S NP VP. 5 NP. Papa 2 Det. the 1 VP VP. PP 5 Det. the 2 Det. a 4 PP. P NP 5 Det. a 0 ROOT S. 4 P. with 6 N spoon. 5 NP Det N. 4 PP P NP. 5 NP NP. PP

148 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with 6 N spoon. 5 NP Det N. 4 PP P NP. 5 NP NP. PP 2 NP NP PP. 1 VP VP PP.

149 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with 6 N spoon. 5 NP Det N. 4 PP P NP. 5 NP NP. PP 2 NP NP PP. 1 VP VP PP. 7 PP. P NP

150 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with 6 N spoon. 5 NP Det N. 4 PP P NP. 5 NP NP. PP 2 NP NP PP. 1 VP VP PP. 7 PP. P NP 1 VP V NP. 2 NP NP. PP

151 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with 6 N spoon. 5 NP Det N. 4 PP P NP. 5 NP NP. PP 2 NP NP PP. 1 VP VP PP. 7 PP. P NP 1 VP V NP. 2 NP NP. PP 0 S NP VP. 1 VP VP. PP

152 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with 6 N spoon. 5 NP Det N. 4 PP P NP. 5 NP NP. PP 2 NP NP PP. 1 VP VP PP. 7 PP. P NP 1 VP V NP. 2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with

153 0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7 0 ROOT. S 0 NP Papa. 1 V ate. 2 Det the. 3 N caviar. 0 S. NP VP 0 S NP. VP 1 VP V. NP 2 NP Det. N 2 NP Det N. 0 NP. Det N 0 NP NP. PP 2 NP. Det N 3 N. caviar 1 VP V NP. 0 NP. NP PP 1 VP. V NP 2 NP. NP PP 3 N. spoon 2 NP NP. PP 0 NP. Papa 1 VP. VP PP 2 NP. Papa 0 S NP VP. 0 Det. the 1 PP. P NP 2 Det. the 1 VP VP. PP 0 Det. a 1 V. ate 2 Det. a 4 PP. P NP 1 P. with 0 ROOT S. 4 P. with 6 N spoon. 5 NP Det N. 4 PP P NP. 5 NP NP. PP 2 NP NP PP. 1 VP VP PP. 7 PP. P NP 1 VP V NP. 2 NP NP. PP 0 S NP VP. 1 VP VP. PP 7 P. with