Introduction to Computational Linguistics

Transcription

1 Introduction to Computational Linguistics Olga Zamaraeva (2018) Based on Bender (prev. years) University of Washington May 3, / 101

2 Midterm Project Milestone 2: due Friday Assgnments 4& 5 due dates moved Start as soon as you can though 2 / 101

3 Overview We will focus on the concept of parsing and why naive parsing is inefficient idea Then will briefly cover and then Assignment 4 asks you to study it in detail next time: probabilistic parsing (also need for Assignment 4) 3 / 101

4 Parsing Recognizing string as input and assigning structure to it Syntactic parsing: assigning syntactic structure Semantic parsing: assigning semantic structure 4 / 101

5 Parsing: Making explicit structure that is inherent (implicit) in natural language strings What is that structure? Why would we need it? 5 / 101

6 Bottom Up vs. Top Down parsing S / \??? a b c d d f 6 / 101

7 Recursive Top-Down Parsing slide from: 7 / 101

8 Recursive Top-Down Parsing slide from: 8 / 101

9 Recursive Top-Down Parsing slide from: 9 / 101

10 Recursive Top-Down Parsing slide from: 10 / 101

11 Recursive Top-Down Parsing slide from: 11 / 101

12 Recursive Top-Down Parsing What is the time complexity of what we just saw? 12 / 101

13 Recursive Top-Down Parsing What is the time complexity of what we just saw? Exponential in n! (Meaning, as we increase the length of input, the time to do parsing increases exponentially) (Which is very very bad) 13 / 101

14 **Optional exercise: Recursive Parsing Running Time S A A a A b a A c ɛ Consider a string a n c n Convince yourself that you will expand A 2 n times 14 / 101

15 Example 15 / 101

16 Bottom-up parsing Book the flight through Houston 16 / 101

17 Bottom-up parsing N D N P PN Book the flight through Houston 17 / 101

18 Bottom-up parsing D P PN N the N through Houston Book flight 18 / 101

19 Bottom-up parsing NP PP N D P PN Book the N through Houston flight No more possibilities! Backtrack / 101

20 Bottom-up parsing D PP N the N P PN Book flight through Houston 20 / 101

21 Bottom-up parsing D N the PP Book N P PN flight through Houston 21 / 101

22 Bottom-up parsing NP N D Book the PP N P PN flight through Houston No more possibilities! Backtrack... Up to where?.. 22 / 101

23 Bottom-up parsing Backtrack to the very beginning, actually! Book the flight through Houston 23 / 101

24 Bottom-up parsing V D N P PN Book the flight through Houston 24 / 101

25 Bottom-up parsing V D P PN Book the N through Houston flight 25 / 101

26 Bottom-up parsing V NP PP Book D P PN the N through Houston flight 26 / 101

27 Bottom-up parsing VP PP V NP P PN Book D through Houston the N flight 27 / 101

28 Bottom-up parsing VP VP PP V NP P PN Book D through Houston the N flight 28 / 101

29 Bottom-up parsing S VP VP PP V NP P PN Book D through Houston the N flight 29 / 101

30 Bottom-up parsing Or, we could have instead done: V D PP Book the N P PN flight through Houston 30 / 101

31 Bottom-up parsing V D Book the PP N P PN flight through Houston 31 / 101

32 Bottom-up parsing V NP Book D the PP N P PN flight through Houston 32 / 101

33 Bottom-up parsing VP V NP Book D the PP N P PN flight through Houston 33 / 101

34 Bottom-up parsing S VP V NP Book D the PP N P PN flight through Houston 34 / 101

35 How do we make sure we get both trees? Go through all possibilities for productions 35 / 101

36 Top-Down Parsing S 36 / 101

37 Top-Down Parsing S NP VP 37 / 101

38 Top-Down Parsing S NP VP Pronoun 38 / 101

39 Top-Down Parsing S NP VP 39 / 101

40 Top-Down Parsing S NP VP PN 40 / 101

41 Top-Down Parsing S NP VP 41 / 101

42 Top-Down Parsing S NP VP D 42 / 101

43 Top-Down Parsing S NP VP 43 / 101

44 Top-Down Parsing S 44 / 101

45 Top-Down Parsing S Aux NP VP 45 / 101

46 Top-Down Parsing S 46 / 101

47 Top-Down Parsing S VP 47 / 101

48 Top-Down Parsing S VP V 48 / 101

49 Top-Down Parsing S VP V Book Yes, but we have more input still / 101

50 Top-Down Parsing S VP V NP 50 / 101

51 Top-Down Parsing S VP V NP Book 51 / 101

52 Top-Down Parsing S VP V NP Book Pronoun 52 / 101

53 Top-Down Parsing S VP V NP Book 53 / 101

54 Top-Down Parsing S VP V NP Book PN 54 / 101

55 Top-Down Parsing S VP V NP Book 55 / 101

56 Top-Down Parsing S VP V NP Book D 56 / 101

57 Top-Down Parsing S VP V NP Book D the 57 / 101

58 Top-Down Parsing S VP V NP Book D the N 58 / 101

59 Top-Down Parsing S VP V NP Book D the N flight Yes, but we have more input still / 101

60 Top-Down Parsing S VP V NP Book D the N 60 / 101

61 Top-Down Parsing S VP V NP Book D the N N 61 / 101

62 Top-Down Parsing S VP V NP Book D the N N flight Nope... Backtrack again / 101

63 Top-Down Parsing S VP V NP Book D the 63 / 101

64 Top-Down Parsing S VP V NP Book D the PP 64 / 101

65 Top-Down Parsing S VP V NP Book D the PP N 65 / 101

66 Top-Down Parsing S VP V NP Book D the PP N flight 66 / 101

67 Top-Down Parsing S VP V NP Book D the PP N P NP flight 67 / 101

68 Top-Down Parsing S VP V NP Book D the PP N P NP flight through 68 / 101

69 Top-Down Parsing S VP V NP Book D the PP N P NP flight through Pronoun 69 / 101

70 Top-Down Parsing S VP V NP Book D the PP N P NP flight through 70 / 101

71 Top-Down Parsing S VP V NP Book D the PP N P NP flight through PN 71 / 101

72 Top-Down Parsing S V VP NP Book D the PP N P NP flight through PN Houston 72 / 101

73 Top-Down Parsing Could we have gotten the second tree by top-down parsing? 73 / 101

74 Top-Down Parsing Could we have gotten the second tree by top-down parsing? Yes; it is a matter of which rule happened to be on the top of the stack We grabbed VP V NP But the option VP VP PP is also on the stack somewhere Thus the returned parse is subject to an arbitrary listing of rules in the grammar 74 / 101

75 Bottom Up vs. Top Down parsing Top-down parsers do not waste time exploring hypotheses not leading to S...but do waste time exploring hypotheses not matching the input Bottom-up parsers do not waste time exploring hypotheses not matching input...but do waste time exploring hypotheses not leading to S Both can take exponential time (in the worst case, easier shown on abstract CFG) Some recursive parsers are O(n 4 ) An answer to poor time complexity: dynamic O(n 3 ) 75 / 101

76 Example: The Fibonacci numbers Recursive definition: f(0) = 0; f(1) = 1 f(n) = f(n-1) + f(n-2) f(100) = / 101

77 The Fibonacci numbers: naive implementation Since we have a recursive definition, let s implement the Fibonacci numbers printer recursively! def fibonacci(n): if n in [0,1]: return n return fibonacci(n-1) + fibonacci(n-2) What s the problem with this? 77 / 101

78 The Fibonacci numbers: better implementation def fibonacci(n): return fibonacci_helper(n,{}) def fibonacci_helper(n,memo): if n in [0,1]: return n if not n in memo: memo[n] = fibonacci_helper(n-1,memo) + fibonacci_helper(n-2,memo) return memo[n] 78 / 101

79 Fill in a table with solutions to subproblems Then can just look up momentarily the precomputed solution No need to perform the same computation many times 79 / 101

80 Fibonacci numbers f(0) = 0 f(1) = 1 f(2) = f(1) + f(0) what s f(1)? what s f(0)? 80 / 101

81 Fibonacci numbers f(3) = f(2) + f(1) f(2) = f(1) + f(0) f(1) = 1 f(0) = 0 81 / 101

82 Fibonacci numbers f(4) = f(3) + f(2) f(3) = f(2) + f(1) f(2) = f(1) + f(0) f(1) = 1 f(0) = 0 82 / 101

83 Fibonacci numbers f(5) = f(4) + f(3) f(4) = f(3) + f(2) f(3) = f(2) + f(1) f(2) = f(1) + f(0) f(1) = 1 f(0) = 0 etc... (deep recursion; slow; do the same computation again and again) 83 / 101

84 Fibonacci numbers f(0) =? / 101

85 Fibonacci numbers f(0) =? Not in the table, so compute: f(0)=0 (or rather, return the base case) fill in the cell / 101

86 Fibonacci numbers f(1) =? Not in the table, so compute: f(1)=1 (or rather, return the base case) fill in the cell / 101

87 Fibonacci numbers f(2) =? Not in the table, so compute: f(2)=f(2-1) + f(2-2) = f(1) + f(0) But both f(1) and f(0) are already in the table! No need to compute! Just look up! fill in the cell / 101

88 Fibonacci numbers f(3) =? Not in the table, so compute: f(3)=f(3-1) + f(3-2) = f(2) + f(1) But both f(2) and f(1) are already in the table! No need to compute! Just look up! fill in the cell / 101

89 Fibonacci numbers f(4) =? Not in the table, so compute: f(4)=f(4-1) + f(4-2) = f(3) + f(2) But both f(3) and f(2) are already in the table! No need to compute! Just look up! fill in the cell / 101

90 for parsing Once the constituent has been discovered, store the information Example: The algorithm (Cocke-Kazami-Younger) 90 / 101

91 Chomsky Normal Form All productions must conform to two forms: A BC A w i.e. only binary branching trees (and leaves) 91 / 101

92 Chomsky Normal Form To convert to CNF: copy all conforming rules as is Flatten unit productions N, N cat dog becomes: cat, dog Introduce dummy rules to get rid of mixed terminal-nonterminal RHS (right-hand side) INF to VP becomes: TO to and INF TO VP Introduce dummy rules to expand rules with RHS greater than 2 nonterminals A B C D becomes: A X D, X B C 92 / 101

93 Original grammar 93 / 101

94 Chomsky Normal Form 94 / 101

95 : Main idea A 2-dimensional array (aka a table) can encode the structure of the tree Each cell [i,j] contains all constituents that span positions i through j of the input string 0 Book 1 that 2 flight 3 Cell [0,n] must have the Start symbol if we have a parse...and can have more than one! 95 / 101

96 96 / 101

97 97 / 101

98 98 / 101

99 99 / 101

100 Limitations of classical It is a recognizer Turn a recognizer into a parser by storing all tree paths leading to S...but returning all possible trees is again exponential time! Also, we modified the grammar! 100 / 101

101 Limitations of classical : Solutions Probabilistic parsing Train a probabilistic grammar and then return the most probable parse Modify to be able to recover original grammar Employ e.g. partial parsing to get accommodate CFG directly (not in CNF) 101 / 101