Cross-lingual Semantic Parsing

Transcription

1 Cross-lingual Semantic Parsing Part I: 11 Dimensions of Semantic Parsing Kilian Evang University of Düsseldorf 1 / 94

2 Abbreviations NL natural language e.g., English, Bulgarian NLU natural language utterance e.g., a sentence, a question, a command MRL meaning representation language (a formal language) e.g., first-order logic, SQL MR meaning representation e.g., a first-order logical formula, an SQL query 2 / 94

3 What Is a Semantic Parser? A program that translates NLUs to MRs, which a computer can execute. Example (1) a. Give me the cities in Virginia. b. answer(a, (city(a), loc(a, B), const(b, stateid(virginia)))) 3 / 94

4 Semantic Parsing Why? Virtual assistants Wake me up at seven. Natural-language database interfaces List all 1990s French comedy movies. Machine reading Extract structured information from text 4 / 94

5 Canonicalization many ways to say the same thing in NL semantic parsers need to canonicalize Example (2) a. What s the population of Alabama? b. How many people live in Alabama? c. How many citizens in Alabama? d. answer(a, (population(b, A), const(b, stateid(alabama))) 5 / 94

6 Subproblems of Semantic Parsing 1. Predict the meaning of each word 2. Predict how to assemble word meanings into NLU meanings 6 / 94

7 Rule-based vs. Learning-based Methods Rule-based semantic parsing high precision, low recall requires expert effort for each new construction suitable for small domains, controlled NL Learning-based semantic parsing learns from examples important for scaling to large domains, variable NL focus of this course 7 / 94

8 Subproblems of Semantic Parser Learning 1. lexicon learning: which words mean what? 2. parser learning: how to assemble word meanings into sentence meanings? 8 / 94

9 11 Questions for Learning a Semantic Parser 1. What is the input? (NL) 2. What is the output? (MRL) 3. What kind of supervision is available? 4. How do we structure the lexicon? 5. What parsing algorithm do we use? 6. What features do we use? 7. What kind of model do we use? 8. What learning algorithm do we use? 9. How do we learn the lexicon? 10. What experimental setup do we use? 11. How do we evaluate the semantic parser? 9 / 94

10 Example Semantic Parser GeoPar simple semantic parser for illustration parses questions about United States geography into GeoQuery MRs inspired by Zelle and Mooney (1996) optional exercise: implement GeoPar starter code: solution for peeking: texttheater/geopar/tree/solution 10 / 94

11 What Is the Input? Some Possible Answers List of words (GeoPar) e.g., [ what, is, the, capital, of, the, state, with, the, largest, population,? ] List of words + part-of-speech tags List of words + word vectors 11 / 94

12 What Is the Output? depends on purpose Some Possible Answers MRs in a special-purpose MRL e.g., GeoQuery (GeoPar) MRs in a query language for databases/knowlege bases e.g., SQL, SPARQL, description logics MRs in a wide-coverage MRL e.g., DRT, AMR 12 / 94

13 GeoQuery Example (3) a. What is the capital of the state with the largest population? b. answer(c, (capital(c), loc(c, S), largest(p, (state(s), population(s, P))))) Zelle and Mooney (1996) 13 / 94

14 GeoQuery (cont.) supports queries about U.S. states, cities, rivers, etc. Prolog subset datasets Geo880: 880 NLU-MRL pairs, English, widely used as a benchmark in early work on learning-based semantic parsing (c ), ml/nldata/geoquery.html Geo250: subset with three additional NLs (Turkish, Spanish, Japanese), 14 / 94

15 GeoQuery Predicates Concepts capital( ) city( ) lake( ) major( ) mountain( ) place( ) state( ) river( ) Properties area(, ) density(, ) elevation(, ) high point(, ) len(, ) population(, ) size(, ) Relations capital(, ) loc(, ) next to(, ) traverse(, ) 15 / 94

16 GeoQuery Predicates (cont.) Constants const(, countryid(usa)) const(, stateid(mississippi)) const(, cityid( los angeles, ca)) const(, riverid(mississippi)) const(, placeid( mount whitney ))... Aggregation count(,, ) fewest(,, ) higher(, ) highest(, ) largest(, ) longer(, ) lowest(, ) most(,, ) shortest(, ) smallest(, ) sum(,, ) Negation \+ 16 / 94

17 Description Logics (DL) Example (4) a. What college did Obama go to? b. Type.University Education.BarackObama Berant et al. (2013) 17 / 94

18 Description Logics (cont.) DL expressions describe concepts Application-dependent set of concept symbols and relation symbols.: join a relation with a concept : intersection of concepts translated to SQL or SPARQL for querying databases such as Freebase datasets Free917 (917 NLU-MR pairs) WebQuestions (5810 NLU-answer pairs) 18 / 94

19 Discourse Representation Structures (DRS) Example (5) a. All equipment will be completely manufactured. b. x1 equipment.n.01(x1) e1 s1 manufacture.v.01(e1) Manner(e1, s1) Result(e1, x1) complete.r.01(s1) Bjerva et al. (2016) 19 / 94

20 Discourse Representation Structures (DRS) (cont.) Discourse Representation Theory (DRT) (Kamp, 1984) general-purpose meaning representation language uses discourse referents, event semantics, logical connectives WordNet sense inventory, VerbNet frame inventory datasets Groningen Meaning Bank ( English Parallel Meaning Bank ( English, Dutch, Italian, German 20 / 94

21 Example Abstract Meaning Representations (AMR) (6) a. All equipment will be completely manufactured. b. Bjerva et al. (2016) 21 / 94

22 Abstract Meaning Representations (AMR) (cont.) graph-based general-purpose meaning representation (Banarescu et al., 2013) somewhat limited expressive power compared to DRS (Bos, 2016) no word sense annotation, PropBank frame inventory datasets 22 / 94

23 What Kind of Supervision is Available? Some Possible Answers NLU-MR pairs (GeoPar) question-answer pairs + database text + database NLU-MR pairs in another language (cross-lingually supervised) 23 / 94

24 Supervised Learning from NLU-MR Pairs Example (7) a. What is the capital of the state with the largest population? b. answer(c, (capital(c), loc(c, S), largest(p, (state(s), population(s, P))))) Zelle and Mooney (1996) 24 / 94

25 Supervised Learning from NLU-MR Pairs (cont.) parser trained to produce gold-standard MRs requires expert annotation 25 / 94

26 Supervised Learning from Question-Answer Pairs + Database Liang et al. (2011) 26 / 94

27 Supervised Learning from Question-Answer Pairs + Database (cont.) parser trained to produce any MR, as long as executing it on DB gives gold-standard answer enables annotation by non-experts 27 / 94

28 Weakly Supervised Learning from Text + Database Reddy et al. (2014) 28 / 94

29 Weakly Supervised Learning from Text + Database (cont.) convert output of syntactic parser to ungrounded MR learn to map ungrounded MRs to grounded MRs (database subgraphs) so that facts found in Web text correspond to facts found in database 29 / 94

30 Cross-lingually Supervised Learning from NLU-MR Pairs in another Language Evang and Bos (2016) 30 / 94

31 Cross-lingually Supervised Learning from NLU-MR Pairs in another Language (cont.) have: semantic parser for language A want: semantic parser for language B use parallel data to project parses from A to B, train semantic parser on projected parses 31 / 94

32 How do We Structure the Lexicon? Some Possible Answers Lexicon = a mapping from (multi)words to predicates (GeoPar) Lexicon = a mapping from (multi)words to lambda expressions often with CCG categories (adding syntactic type and word order information) 32 / 94

33 The GeoPar Lexicon Format Example NLU-MR Pair (8) a. What states border Texas? b. answer(s, (state(s), next to(s, T), const(t, stateid(texas)))) Corresponding Lexicon Entries states state(, ) border next to(, ) Texas const(, stateid(texas)) 33 / 94

34 A λ-based Lexicon Format Example NLU-MR Pair (9) a. What states border Texas? b. λs.state(s) next to(s, texas) Corresponding Lexicon Entries What λf.λg.λx.(f (x) g(x)) states λx.state(x) border λx.λy.next to(y, x) Texas texas Zettlemoyer and Collins (2005) 34 / 94

35 A λ-based Lexicon Format with CCG Categories Example NLU-MR Pair (9) a. What states border Texas? b. λs.state(s) next to(s, texas) Corresponding Lexicon Entries What λf.λg.λx.(f (x) g(x)) states λx.state(x) border λx.λy.next to(y, x) Texas texas Zettlemoyer and Collins (2005) 34 / 94

36 A λ-based Lexicon Format with CCG Categories Example NLU-MR Pair (9) a. What states border Texas? b. λs.state(s) next to(s, texas) Corresponding Lexicon Entries What λf.λg.λx.(f (x) g(x)) states N : λx.state(x) border λx.λy.next to(y, x) Texas texas Zettlemoyer and Collins (2005) 34 / 94

37 A λ-based Lexicon Format with CCG Categories Example NLU-MR Pair (9) a. What states border Texas? b. λs.state(s) next to(s, texas) Corresponding Lexicon Entries What λf.λg.λx.(f (x) g(x)) states N : λx.state(x) border λx.λy.next to(y, x) Texas NP : texas Zettlemoyer and Collins (2005) 34 / 94

38 A λ-based Lexicon Format with CCG Categories Example NLU-MR Pair (9) a. What states border Texas? b. λs.state(s) next to(s, texas) Corresponding Lexicon Entries What λf.λg.λx.(f (x) g(x)) states N : λx.state(x) border (S \ NP)/ NP : λx.λy.next to(y, x) Texas NP : texas Zettlemoyer and Collins (2005) 34 / 94

39 A λ-based Lexicon Format with CCG Categories Example NLU-MR Pair (9) a. What states border Texas? b. λs.state(s) next to(s, texas) Corresponding Lexicon Entries What (S /(S \ NP))/ N : λf.λg.λx.(f (x) g(x)) states N : λx.state(x) border (S \ NP)/ NP : λx.λy.next to(y, x) Texas NP : texas Zettlemoyer and Collins (2005) 34 / 94

40 What Parsing Algorithm Do We Use? Some Possible Answers CYK variant Shift-reduce variant (GeoPar) 35 / 94

41 CYK Parsing for a long time the most popular algorithm for syntactic and semantic parsing each cell stands for one subspan of the input cells filled with all possible constituents/arcs/mrs over that span shorter spans first, longer spans later quadratic running time even with beam search 36 / 94

42 Shift-reduce Parsing a transition-based family of algorithms more popular for syntactic and semantic parsing recently processes input from left to right builds constituents/arcs/mrs from left to right advantage: linear running time (when greedy or using beam search) advantage: psychological plausbility 37 / 94

43 A Shift-reduce Algorithm for GeoQuery a parser state (aka parse item, configuration) consists of a stack and a queue in the initial state, the stack contains 1 term: answer(, ), and the queue contains the whole input NLU in a final state, the stack contains 1 term (the output MR) and the queue is empty actions (aka transitions) take the parser from one state (the predecessor) to the next state (its successor) 38 / 94

44 We Need Actions to... Example (10) a. What is the capital of the state with the largest population? b. answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 39 / 94

45 We Need Actions to... Example (10) a. What is the capital of the state with the largest population? b. answer(c, ( capital (S, C), largest(p, (state(s), population(s, P)))))...introduce predicate instances 39 / 94

46 We Need Actions to... Example (10) a. What is the capital of the state with the largest population? b. answer(c, (capital( S, C), largest(p, (state( S ), population(s, P)))))...introduce predicate instances...coreference variables 39 / 94

47 We Need Actions to... Example (10) a. What is the capital of the state with the largest population? b. answer(c, (capital(s, C), largest (P, (state(s), population (S, P)))))...introduce predicate instances...coreference variables...embed one predicate instance under another 39 / 94

48 We Need Actions to... Example (10) a. What is the capital of the state with the largest population? b. answer(c, (capital(s, C), largest(p, ( state (S), population (S, P)))))...introduce predicate instances...coreference variables...embed one predicate instance under another...conjoin two predicate instances 39 / 94

49 Notation q[0]: leftmost element of queue q[:n]: first n elements of queue s[0]: topmost element of stack s[1]: second-topmost element of stack Each stack element s[i] has a secondary stack s[i].sec which is initially empty but may later contain subterms of s[i]. When s[i].sec is empty, then s[i].sec[0] refers to s[i]. j marks the subterm found at s[i].sec[j] t.args: the arguments of the term t 40 / 94

50 Action Types shift-n-p: remove q[:n] from queue and append P to stack such that q[:n] P in lexicon skip: remove q[0] coref- i -j- k -l: unify the variables at s[1].sec[i].args[j - 1] and s[0].sec[k].args[l - 1] drop-i: move s[0] into s[1].sec[0].args[i - 1], push it onto s[1].sec precondition: s[0].sec is empty lift-i: move s[1] into s[0].sec[0].args[i - 1], push it onto s[0].sec precondition: s[1].sec is empty conj: move s[0] to form a conjunction with s[1].sec[0], push it onto s[1].sec pop: pop 1 element from s[0].sec idle: no-op for final items 41 / 94

51 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 42 / 94

52 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 42 / 94

53 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 42 / 94

54 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 42 / 94

55 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 42 / 94

56 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 42 / 94

57 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 42 / 94

58 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 42 / 94

59 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 42 / 94

60 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 42 / 94

61 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 42 / 94

62 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 42 / 94

63 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 42 / 94

64 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 42 / 94

65 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 42 / 94

66 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 42 / 94

67 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 42 / 94

68 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 42 / 94

69 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 42 / 94

70 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 42 / 94

71 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 42 / 94

72 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 42 / 94

73 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 42 / 94

74 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 42 / 94

75 Example Parse answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 24 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 42 / 94

76 Parser States as Python Objects class ParserState: def init (self, stack, queue, pred, action): self.stack = stack self.queue = queue self.pred = pred # predecessor parser state self.action = action # action leading from pred # to self def shift(self, n, term): queue = self.queue for i in range(n): queue = queue.pop() stack = self.stack.push(term) return ParserState(stack, queue, self, ( shift, n, term.to_string())) def skip(self): return ParserState(self.stack, self.queue.pop(), self, ( skip,)) # / 94

77 What Features Do We Use? parser must be decide what is the next action for each state represent state as feature vectors to make this easier Some Possible Answers Discrete representations of parser states (do feature engineering; GeoPar) Continuous representations of parser states (use neural network) 44 / 94

78 Predicting Actions using Features action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 45 / 94

79 Predicting Actions using Features action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 45 / 94

80 Predicting Actions using Features action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 45 / 94

81 Predicting Actions using Features action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 45 / 94

82 Predicting Actions using Features action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 46 / 94

83 Predicting Actions using Features action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 46 / 94

84 Predicting Actions using Features action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 46 / 94

85 Predicting Actions using Features action stack queue 0 answer(, ) what is the capital of the state with the largest population? 1 skip answer(, ) is the capital of the state with the largest population? 2 skip answer(, ) the capital of the state with the largest population? 3 skip answer(, ) capital of the state with the largest population? 4 shift-1-capital(, ) answer(, ) capital(, ) of the state with the largest population? 5 coref answer(c, ) capital(, C) of the state with the largest population? 6 drop-2 answer(c, 0 capital(, C)) of the state with the largest population? 7 skip answer(c, 0 capital(, C)) the state with the largest population? 8 skip answer(c, 0 capital(, C)) state with the largest population? 9 shift-1-state( ) answer(c, 0 capital(, C)) state( ) with the largest population? 10 coref answer(c, 0 capital(s, C)) state(s) with the largest population? 11 skip answer(c, 0 capital(s, C)) state(s) the largest population? 12 skip answer(c, 0 capital(s, C)) state(s) largest population? 13 shift-1-largest(, ) answer(c, 0 capital(s, C)) state(s) largest(, ) population? 14 lift-2 answer(c, 0 capital(s, C)) largest(, 0 state(s)) population? 15 shift-1-population(, ) answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(, )? 16 coref answer(c, 0 capital(s, C)) largest(, 0 state(s)) population(s, )? 17 coref answer(c, 0 capital(s, C)) largest(p, 0 state(s)) population(s, P)? 18 conj answer(c, 0 capital(s, C)) largest(p, 1 (state(s), 0 population(s, P)))? 19 pop answer(c, 0 capital(s, C)) largest(p, 0 (state(s), population(s, P)))? 20 pop answer(c, 0 capital(s, C)) largest(p, (state(s), population(s, P)))? 21 conj answer(c, 1 (capital(s, C), 0 largest(p, (state(s), population(s, P)))))? 22 pop answer(c, 0 (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 pop answer(c, (capital(s, C), largest(p, (state(s), population(s, P)))))? 23 skip answer(c, (capital(s, C), largest(p, (state(s), population(s, P))))) 46 / 94

86 Feature Extraction Code class ParserState: #... def local_features(self): def get_stack_terms(): for sp, ssp in ((0, 0), (0, 1), (1, 0), (1, 1), (1, 2)): try: yield self.stack[sp].sec[ssp] except IndexError: yield None (s00, s01, s10, s11, s12) = get_stack_terms() # / 94

87 Feature Extraction Code (cont.) class ParserState: #... def local_features(self): #... def get_words(): for i in (-4, -3, -2, -1, 0, 1, 2, 3): try: yield self.words[self.offset + i] except IndexError: yield None (W4, W3, W2, W1, w0, w1, w2, w3) = get_words() # / 94

88 Feature Extraction Code (cont.) class ParserState: #... def local_features(self): #... tf = {} # map from templates to features # feature group 1: stack predicates tf[ s00p ] = s00.pred if s00 is not None else None tf[ s01p ] = s00.pred if s01 is not None else None tf[ s10p ] = s10.pred if s10 is not None else None tf[ s11p ] = s11.pred if s10 is not None else None tf[ s12p ] = s12.pred if s12 is not None else None # / 94

89 Feature Extraction Code (cont.) class ParserState: #... def local_features(self): #... # feature group 2: combinations of stack predicates for s0ip in ( s00p, s01p ): for s1jp in ( s10p, s11p, s12p ): tf[(s0ip, s1jp)] = (tf[s0ip], tf[s1jp]) # / 94

90 Feature Extraction Code (cont.) class ParserState: #... def local_features(self): #... # feature group 3: unigrams tf[ W4 ] = W4 tf[ W3 ] = W3 tf[ W2 ] = W2 tf[ W1 ] = W1 tf[ w0 ] = w0 tf[ w1 ] = w1 tf[ w2 ] = w2 tf[ w3 ] = w3 # / 94

91 Feature Extraction Code (cont.) class ParserState: #... def local_features(self): #... # feature group 4: bigrams unigram_templates = ( W4, W3, W2, W1, w0, w1, w2, w3 ) for (wi, wj) in zip(unigram_templates, unigram_templates[1:]): tf[(wi, wj)] = (tf[wi], tf[wj]) # feature group 5: trigrams for (wi, wj, wj) in zip(unigram.templates, unigram_templates[1:], unigram.templates[2:]): tf[(wi, wj, wk)] = (tf[wi], tf[wj], tf[wk]) # / 94

92 Feature Extraction Code (cont.) class ParserState: #... def local_features(self): #... # Yield all template features as strings: for (template, value) in tf.items(): if isinstance(template, tuple): template =.join(template) if isinstance(value, tuple): value =.join(str(x) for x in value) yield template + = + value # Also yield a bias feature: yield bias 53 / 94

93 Feature Extraction Code (cont.) class ParserState: #... def features(self): if self._features is None: # memoize if self.pred is None: self._features = collections.counter() else: self._features = collections.counter( self.pred.features()) action =.join( str(x) for x in self.action) for f in self.pred.local_features(): f = f + : + action self._features[f] += 1 return self._features 54 / 94

94 Some Caveats in practice, use numpy arrays rather than collections.counter for efficiency use a vector compression method such as hash kernels (Bohnet, 2010) to limit vector size need more complex features for good performance, e.g., combinations of queue words and stack predicates stack predicate classes, e.g. aggregate vs. concept previous actions various combinations thereof 55 / 94

95 Example Features Features Usually Indicative of Good Parsing Decisions w0 = capital : shift 1 capital s10p W3 W2 W1 = answer what is the : coref Features Usually Indicative of Bad Parsing Decisions w0 = capital : skip W1 = the : skip 56 / 94

96 What Kind of Model Do We Use? Some Possible Answers Linear model (GeoPar) Log-linear model Neural network 57 / 94

97 Linear Model associates every feature with a real-valued weight positive weights for features indicative of good parsing decisions negative weights for features indicative of bad parsing decisions weights close to 0 for features not very indicative the score of a parser state is the sum of the counts of each of its features multiplied by its weight parser will choose actions that lead to parser states with higher scores 58 / 94

98 Model Code class LinearModel: def init (self): self.weights = collections.counter() def score(self, features): result = 0 for feature in features: result += self.weights[feature] return result 59 / 94

99 Parsing with Beam Search def parse(words, model, beam_size=10): agenda = [initial_state(words)] while any(not st.is_final() for st in agenda): agenda = [su for st in agenda for su in st.successors()] agenda.sort( key=lambda st: model.score(st.features()), reverse=true) beam = agenda[:min(beam_size, len(agenda))] agenda = beam if not agenda: raise ValueError( no parse found ) return agenda[0].stack[0] 60 / 94

100 What Learning Algorithm Do We Use? How do we set the weights? Need to train our model on the training data Some Possible Answers Perceptron (GeoPar) Some variant of stochastic gradient descent 61 / 94

101 Perceptron Update class LinearModel: #... def update(self, gold_features, predicted_features): update = collections.counter() for gf in gold_features: update[gf] += 1 for pf in predicted_features: update[pf] -= 1 for feature, delta in update.items(): if delta!= 0: self.weights[feature] += delta 62 / 94

102 Perceptron Training def train_one_epoch(train_data, model, beam_size=10): for words, gold_action_sequence in train_data: def is_correct(state): # a state is considered correct if the action sequence leading up # to it is a prefix of the complete gold action sequence, or vice # versa: return all(g == i for g, i in zip(gold_action_sequence, state.action_sequence())) agenda = [initial_state(words)] while any(not st.is_final() for st in agenda): agenda = [su for st in agenda for su in st.successors()] agenda.sort( key=lambda st: model.score(st.features()), reverse=true) beam = agenda[:min(beam_size, len(agenda))] if not any(is_correct(st) for st in beam): break # early update agenda = beam best = agenda[0] best_correct = next(st for st in agenda if is_correct(st)) if best!= best_correct: model.update(best_correct.features(), best.features()) 63 / 94

103 Perceptron Training for Multiple Epochs def train(train_data, num_epochs): train_data = list(train_data) model = LinearModel() for t in range(num_epochs): random.shuffle(train_data) train_one_epoch(train_data, model) return model 64 / 94

104 Some Caveats again, model.weights should be a numpy array in practice use averaged perceptron: keep track of weights after each update, average them at the end rather than fixing number of epochs, use validation data to stop training once further epochs do not improve accuracy 65 / 94

105 How Do We Learn the Lexicon? Some Possible Answers specify lexicon manually (GeoPar) use some combination of unsupervised word-predicate alignment extension-based lexicon induction lexical templates splitting via higher-order unification iterative lexicon refinement cross-lingual lexicon induction 66 / 94

106 Unsupervised Word-predicate Alignment What states border Texas answer(s, ( state(s), next to(s, T) const(t, stateid(texas))))

107 Unsupervised Word-predicate Alignment What states border Texas answer(s, ( state(s), next to(s, T) const(t, stateid(texas)))) 67 / 94

108 Unsupervised Word-predicate Alignment (cont.) widely used technique in machine translation use, e.g., GIZA++ or fast align alignments are uncertain possible solution: generate multiple lexical entries; let the learning algorithm figure out which ones are good/bad 68 / 94

109 Extension-based Lexicon Induction Berant et al. (2013) 69 / 94

110 Extension-based Lexicon Induction (cont.) use predicates in a DB, e.g., PlaceOfBirth the extension of a predicate is the set of pairs of entities it is true of, e.g., (BarackObama, Honolulu), (MichelleObama, Chicago) find phrases in raw text, e.g., lived in the extension of a phrase is the set of pairs of entities mentioned in the same sentence, e.g. (BarackObama, Honolulu), (BarackObama, Chicago) map entity mentions to entities based on string similarity heuristics learn to map phrases to predicates based on the similarity of their extensions 70 / 94

111 Lexical Templates Zettlemoyer and Collins (2005) 71 / 94