Chart Parsing: the CYK Algorithm

Transcription

1 Chrt Prsing: the CYK lgorithm Informtics 2: Lecture 20 Shy Cohen 2 November / 43

2 Lst Clss Constituents nd Phrses: phrse inherits the ctegory of its hed. mbiguity: sentence cn hve multiple prse trees (or multiple POS nlyses) Recursive descent Prsing: top-down prser. Iterte through ll rules, in depth-first style, until prse tht mtches the sentence is found. Exhustive serch, essentilly. Shift-Reduce Prsing: Two opertions SHIFT: put word with its POS tg on the stck. REDUCE: Tke sequence of top symbols on the stck nd pop them if they mtch with right-hnd side of rule, nd then plce the left-hnd side on the top of the stck. Why not LL(1)? We hve mbiguous grmmrs! 2 / 43

3 Shift-Reduce Prsing Shift-Reduce prser tries to find sequences of words nd phrses tht correspond to the righthnd side of grmmr production nd replce them with the lefthnd side: Directionlity = bottom-up: strts with the words of the input nd tries to build trees from the words up. Serch strtegy = bredth-first: strts with the words, then pplies rules with mtching right hnd sides, nd so on until the whole sentence is reduced to n S. 3 / 43

4 lgorithm Sketch: Shift-Reduce Prsing Until the words in the sentences re substituted with S: Scn through the input until we recognise something tht corresponds to the RHS of one of the production rules (shift) pply production rule in reverse; i.e., replce the RHS of the rule which ppers in the sententil form with the LHS of the rule (reduce) shift-reduce prser implemented using stck: 1. strt with n empty stck 2. shift ction pushes the current input symbol onto the stck 3. reduce ction replces n items with single item 4 / 43

5 Shift-Reduce Prsing Stck Remining T my dog sw mn in the prk with s 5 / 43

6 Shift-Reduce Prsing Stck Det Remining T dog sw mn in the prk with s my 6 / 43

7 Shift-Reduce Prsing Stck Det N Remining T sw mn in the prk with s my dog 7 / 43

8 Shift-Reduce Prsing Stck NP Remining T sw mn in the prk with s Det my N dog 8 / 43

9 Shift-Reduce Prsing Stck Remining T NP V NP in the prk with s Det N sw Det N my dog mn 9 / 43

10 Shift-Reduce Prsing Stck Remining T NP V NP PP with s Det N sw Det N P NP my dog mn in Det N the prk 10 / 43

11 Shift-Reduce Prsing Stck NP V NP Det N sw NP PP my dog Det N P NP mn in Det N the prk 11 / 43

12 Shift-Reduce Prsing Stck NP VP Det N V NP my dog sw NP PP Det N P NP mn in Det N the prk 12 / 43

13 Shift-Reduce Prsing Stck S NP VP Det N V NP my dog sw NP PP Det N P NP mn in Det N the prk 13 / 43

14 Modern shift-reduce prsers Shift-reduce prsers re highly efficient, they re liner in the length of the string they prse, if they explore only one pth How to do tht? Lern from dt wht ctions to tke t ech point, nd try to mke the optiml decisions so tht the correct prse tree will be found This keeps the prser liner in the length of the string, but one smll error cn propgte through the whole prse, nd led to the wrong prse tree 14 / 43

15 Try it out Yourselves! Recursive Descent Prser >>> from nltk.pp import rdprser >>> rdprser() Shift-Reduce Prser >>> from nltk.pp import srprser >>> srprser() 15 / 43

16 Problems with Prsing s Serch Grmmr Restructuring Problems with Prsing s Serch The CYK lgorithm Prsing s Dynmic Progrmming The CYK lgorithm Properties of the lgorithm 16 / 43

17 Grmmr Restructuring Deterministic prsing (e.g., LL(1)) ims to ddress limited mount of locl mbiguity the problem of not being ble to decide uniquely which grmmr rule to use next in left-to-right nlysis of the input string. By re-structuring the grmmr, the prser cn mke unique decision, bsed on limited mount of look-hed. Recursive Descent prsing lso demnds grmmr restructuring, in order to eliminte left-recursive rules tht cn get it into hopeless loop. 17 / 43

18 Left Recursion But grmmrs for nturl humn lnguges should be reveling, re-structuring the grmmr my destroy this. (Indirectly) left-recursive rules re needed in English. NP DET N NP NPR DET NP s These rules generte NPs with possessive modifiers such s: John s sister John s mother s sister John s mother s uncle s sister John s mother s uncle s sister s niece 18 / 43

19 Left Recursion NP NP NP DET N DET N DET N NP NP NP NPR DET N DET N John s sister NP NPR mother s sister DET NP N uncle s sister John s NP mother s NPR John We don t wnt to re-structure our grmmr rules just to be ble to use prticulr pproch to prsing. Need n lterntive. s 19 / 43

20 How mny prses re there? If our grmmr is mbiguous (inherently, or by design) then how mny possible prses re there? 20 / 43

21 How mny prses re there? If our grmmr is mbiguous (inherently, or by design) then how mny possible prses re there? In generl: n infinite number, if we llow unry recursion. 20 / 43

22 How mny prses re there? If our grmmr is mbiguous (inherently, or by design) then how mny possible prses re there? In generl: n infinite number, if we llow unry recursion. More specific: suppose tht we hve grmmr in Chomsky norml form. How mny possible prses re there for sentence of n words? Imgine tht every nonterminl cn rewrite s every pir of nonterminls ( BC) nd every nonterminl ( ) 1. n 2. n 2 3. n log n 4. (2n)! (n+1)!n! 20 / 43

23 How mny prses re there? 21 / 43

24 How mny prses re there? 21 / 43

25 How mny prses re there? 21 / 43

26 How mny prses re there? 21 / 43

27 How mny prses re there? 21 / 43

28 How mny prses re there? Intution. Let C(n) be the number of binry trees over sentence of length n. The root of this tree hs two subtrees: one over k words (1 k < n), nd one over n k words. Hence, for ll vlues of k, we cn combine ny subtree over k words with ny subtree over n k words: n 1 C(n) = C(k) C(n k) k=1 22 / 43

29 How mny prses re there? Intution. Let C(n) be the number of binry trees over sentence of length n. The root of this tree hs two subtrees: one over k words (1 k < n), nd one over n k words. Hence, for ll vlues of k, we cn combine ny subtree over k words with ny subtree over n k words: n 1 C(n) = C(k) C(n k) k=1 C(n) = (2n)! (n + 1)!n! These numbers re clled the Ctln numbers. They re big numbers! n C(n) / 43

30 Problems with Prsing s Serch 1. recursive descent prser (top-down) will do bdly if there re mny different rules for the sme LHS. Hopeless for rewriting prts of speech (preterminls) with words (terminls). 2. shift-reduce prser (bottom-up) does lot of useless work: mny phrse structures will be loclly possible, but globlly impossible. lso inefficient when there is much lexicl mbiguity. 3. Both strtegies do repeted work by re-nlyzing the sme substring mny times. We will see how chrt prsing solves the re-prsing problem, nd lso copes well with mbiguity. 23 / 43

31 Dynmic Progrmming With CFG, prser should be ble to void re-nlyzing sub-strings becuse the nlysis of ny sub-string is independent of the rest of the prse. The dog sw mn in the prk NP The prser s explortion of its serch spce cn exploit this independence if the prser uses dynmic progrmming. NP Dynmic progrmming is the bsis for ll chrt prsing lgorithms. NP NP 24 / 43

32 Prsing s Dynmic Progrmming Given problem, systemticlly fill tble of solutions to sub-problems: this is clled memoiztion. Once solutions to ll sub-problems hve been ccumulted, solve the overll problem by composing them. For prsing, the sub-problems re nlyses of sub-strings nd correspond to constituents tht hve been found. Sub-trees re stored in chrt (k well-formed substring tble), which is record of ll the substructures tht hve ever been built during the prse. Solves re-prsing problem: sub-trees re looked up, not re-prsed! Solves mbiguity problem: chrt implicitly stores ll prses! 25 / 43

33 Depicting Chrt chrt cn be depicted s mtrix: Rows nd columns of the mtrix correspond to the strt nd end positions of spn (ie, strting right before the first word, ending right fter the finl one); cell in the mtrix corresponds to the sub-string tht strts t the row index nd ends t the column index. It cn contin informtion bout the type of constituent (or constituents) tht spn(s) the substring, pointers to its sub-constituents, nd/or predictions bout wht constituents might follow the substring. 26 / 43

34 CYK lgorithm CYK (Cocke, Younger, Ksmi) is n lgorithm for recognizing nd recording constituents in the chrt. ssumes tht the grmmr is in Chomsky Norml Form: rules ll hve form BC or w. Conversion to CNF cn be done utomticlly. NP Det Nom NP Det Nom Nom N OptP Nom Nom book ornge P Nom OptP ɛ Optdv P hevy ornge dv hevy ornge hevy ornge Det Det Optdv ɛ very dv very N book ornge 27 / 43

35 CYK: n exmple Let s look t simple exmple before we explin the generl cse. Grmmr Rules in CNF NP Det Nom Nom book ornge P Nom P hevy ornge dv hevy ornge Det dv very (N.B. Converting to CNF sometimes breeds dupliction!) Now let s prse: very hevy ornge book 28 / 43

36 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very 2 hevy 3 ornge 4 book very hevy ornge book 29 / 43

37 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det 1 very 2 hevy 3 ornge 4 book 29 / 43

38 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det 1 very dv 2 hevy 3 ornge 4 book 29 / 43

39 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det 1 very dv 2 hevy,p 3 ornge 4 book 29 / 43

40 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det 1 very dv P 2 hevy,p 3 ornge 4 book 29 / 43

41 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det 1 very dv P 2 hevy,p 3 ornge Nom,,P 4 book 29 / 43

42 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det 1 very dv P 2 hevy,p Nom 3 ornge Nom,,P 4 book 29 / 43

43 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det 1 very dv P Nom 2 hevy,p Nom 3 ornge Nom,,P 4 book 29 / 43

44 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det NP 1 very dv P Nom 2 hevy,p Nom 3 ornge Nom,,P 4 book 29 / 43

45 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det NP 1 very dv P Nom 2 hevy,p Nom 3 ornge Nom,,P 4 book Nom 29 / 43

46 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det NP 1 very dv P Nom 2 hevy,p Nom 3 ornge Nom,,P Nom 4 book Nom 29 / 43

47 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det NP 1 very dv P Nom 2 hevy,p Nom Nom 3 ornge Nom,,P Nom 4 book Nom 29 / 43

48 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det NP 1 very dv P Nom Nom 2 hevy,p Nom Nom 3 ornge Nom,,P Nom 4 book Nom 29 / 43

49 Filling out the CYK chrt 0 1 very 2 hevy 3 ornge 4 book very hevy ornge book 0 Det NP NP 1 very dv P Nom Nom 2 hevy,p Nom Nom 3 ornge Nom,,P Nom 4 book Nom 29 / 43

50 CYK: The generl lgorithm function CKY-Prse(words, grmmr) returns tble for j from 1 to Length(words) do tble[j 1, j] { words[j] grmmr} for i from j 2 downto 0 do for k i + 1 to j 1 do tble[i, j] tble[i, j] { BC grmmr, B tble[i, k] C tble[k, j]} 30 / 43

51 CYK: The generl lgorithm function CKY-Prse(words, grmmr) returns tble for j from 1 to Length(words) do loop over the columns tble[j 1, j] { words[j] grmmr} fill bottom cell for i from j 2 downto 0 do fill row i in column j for k i + 1 to j 1 do loop over split loctions tble[i, j] tble[i, j] between i nd j { BC grmmr, Check the grmmr B tble[i, k] for rules tht C tble[k, j]} link the constituents in [i, k] with those in [k, j]. For ech rule found store LHS in cell [i, j]. 31 / 43

52 succinct representtion of CKY We hve Boolen tble clled Chrt, such tht Chrt[, i, j] is true if there is sub-phrse ccording the grmmr tht domintes words i through words j Build this chrt recursively, similrly to the Viterbi lgorithm: For j > i + 1: Chrt[, i, j] = j 1 k=i+1 B C Chrt[B, i, k] Chrt[C, k, j] Seed the chrt, for i + 1 = j: Chrt[, i, i + 1] = True if there exists rule w i+1 where w i+1 is the (i + 1)th word in the string 32 / 43

53 From CYK Recognizer to CYK Prser So fr, we just hve chrt recognizer, wy of determining whether string belongs to the given lnguge. Chnging this to prser requires recording which existing constituents were combined to mke ech new constituent. This requires nother field to record the one or more wys in which constituent spnning (i,j) cn be mde from constituents spnning (i,k) nd (k,j). (More clerly displyed in grph representtion, see next lecture.) In ny cse, for fixed grmmr, the CYK lgorithm runs in time O(n 3 ) on n input string of n tokens. The lgorithm identifies ll possible prses. 33 / 43

54 CYK-style prse chrts Even without converting grmmr to CNF, we cn drw CYK-style prse chrts: very hevy ornge book 0 Det NP NP 1 very Optdv OptP Nom Nom 2 hevy,optp Nom Nom 3 ornge N,Nom,,P Nom 4 book N,Nom (We hven t ttempted to show ɛ-phrses here. Could in principle use cells below the min digonl for this... ) However, CYK-style prsing will hve run-time worse thn O(n 3 ) if e.g. the grmmr hs rules BCD. 34 / 43

55 Second exmple Grmmr Rules in CNF S NP VP Nominl book flight money S X 1 VP Nominl Nominl noun X 1 ux VP Nominl Nominl PP S book include prefer VP book include prefer S Verb NP VPVerb NP S X 2 VP X 2 PP S Verb PP X 2 Verb NP S VP PP VP Verb NP NP TW Houston VP VP PP NP Det Nominl PP Preposition NP Verb book include prefer Noun book flight money Let s prse Book the flight through Houston! 35 / 43

56 Second exmple Grmmr Rules in CNF S NP VP Nominl book flight money S X 1 VP Nominl Nominl noun X 1 ux VP Nominl Nominl PP S book include prefer VP book include prefer S Verb NP VPVerb NP S X 2 VP X 2 PP S Verb PP X 2 Verb NP S VP PP VP Verb NP NP TW Houston VP VP PP NP Det Nominl PP Preposition NP Verb book include prefer Noun book flight money Let s prse Book the flight through Houston! 35 / 43

57 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun [0, 1] 36 / 43

58 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun [0, 1] Det [1, 2] 36 / 43

59 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun [0, 1] Det [1, 2] Nominl, Noun [2, 3] 36 / 43

60 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun [0, 1] Det [1, 2] Nominl, Noun [2, 3] Prep [3, 4] 36 / 43

61 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun [0, 1] Det [1, 2] Nominl, Noun [2, 3] Prep [3, 4] NP, Proper- Noun [4, 5] 36 / 43

62 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun [0, 1] [0, 2] Det [1, 2] Nominl, Noun [2, 3] Prep [3, 4] NP, Proper- Noun [4, 5] 36 / 43

63 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun [0, 1] [0, 2] Det NP [1, 2] [1, 3] Nominl, Noun [2, 3] Prep [3, 4] NP, Proper- Noun [4, 5] 36 / 43

64 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun S, VP, X2 [0, 1] [0, 2] [0, 3] Det NP [1, 2] [1, 3] Nominl, Noun [2, 3] Prep [3, 4] NP, Proper- Noun [4, 5] 36 / 43

65 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun S, VP, X2 [0, 1] [0, 2] [0, 3] Det NP [1, 2] [1, 3] Nominl, Noun [2, 3] [2, 4] Prep [3, 4] NP, Proper- Noun [4, 5] 36 / 43

66 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun S, VP, X2 [0, 1] [0, 2] [0, 3] Det NP [1, 2] [1, 3] [1, 4] Nominl, Noun [2, 3] [2, 4] Prep [3, 4] NP, Proper- Noun [4, 5] 36 / 43

67 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun S, VP, X2 [0, 1] [0, 2] [0, 3] [0, 4] Det NP [1, 2] [1, 3] [1, 4] Nominl, Noun [2, 3] [2, 4] Prep [3, 4] NP, Proper- Noun [4, 5] 36 / 43

68 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun S, VP, X2 [0, 1] [0, 2] [0, 3] [0, 4] Det NP [1, 2] [1, 3] [1, 4] Nominl, Noun [2, 3] [2, 4] Prep PP [3, 4] [3, 5] NP, Proper- Noun [4, 5] 36 / 43

69 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun S, VP, X2 [0, 1] [0, 2] [0, 3] [0, 4] Det NP [1, 2] [1, 3] [1, 4] Nominl, Nominl Noun [2, 3] [2, 4] [2, 5] Prep PP [3, 4] [3, 5] NP, Proper- Noun [4, 5] 36 / 43

70 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun S, VP, X2 [0, 1] [0, 2] [0, 3] [0, 4] Det NP NP [1, 2] [1, 3] [1, 4] [1, 5] Nominl, Nominl Noun [2, 3] [2, 4] [2, 5] Prep PP [3, 4] [3, 5] NP, Proper- Noun [4, 5] 36 / 43

71 Second exmple Book the flight through Houston S, VP, Verb, Nominl, Noun S, VP, X2 S 1, VP, X2, S 2, VP, S 3 [0, 1] [0, 2] [0, 3] [0, 4] [0, 5] Det NP NP [1, 2] [1, 3] [1, 4] [1, 5] Nominl, Nominl Noun [2, 3] [2, 4] [2, 5] Prep PP [3, 4] [3, 5] NP, Proper- Noun [4, 5] 36 / 43

72 Visulizing the Chrt 37 / 43

73 Visulizing the Chrt 38 / 43

74 Dynmic Progrmming s problem-solving technique Given problem, systemticlly fill tble of solutions to sub-problems: this is clled memoiztion. Once solutions to ll sub-problems hve been ccumulted, solve the overll problem by composing them. For prsing, the sub-problems re nlyses of sub-strings nd correspond to constituents tht hve been found. Sub-trees re stored in chrt (k well-formed substring tble), which is record of ll the substructures tht hve ever been built during the prse. Solves re-prsing problem: sub-trees re looked up, not re-prsed! Solves mbiguity problem: chrt implicitly stores ll prses! 39 / 43

75 Tribute to CKY (prt 1/3) You, my CKY lgorithm, dictte every prser s rhythm, if Cocke, Younger nd Ksmi hdn t bothered, ll of our prsing drems would hve been shttered. You re so simple, yet so powerful, nd with the proper semiring nd time, you will be truthful, to return the best prse - nything less would be crime. With dynmic progrmming or memoiztion, you re one of kind, I relly don t need to mention, if it werent for you, ll syntx trees would be behind. 40 / 43

76 Tribute to CKY (prt 2/3) Filed ttempts hve been mde to show there re better, for exmple, by using mtrix multipliction, ll of these imprcticl lgorithms didn t mtter you cme out stronger, insisting on just using summtion. ll prsing lgorithms to you hil, t lest those with bckbones which re context-free, you will never become stle, s long s we need to hve syntx tree. It doesn t mtter tht the C is lwys in front, or tht the K nd Y cn swp, you re still on the sme hunt, mximizing nd summing, nonstop. 41 / 43

77 Tribute to CKY (prt 3/3) Every Informtics student knows you intimtely, they hve seen your vrints dozens of times, you hve erned tht respect legitimtely, nd you will follow them through their primes. CKY, going bckwrd nd forwrd, inside nd out, it is so strightforwrd - You re the best, there is no doubt. 42 / 43

78 Summry Prsing s serch is inefficient (typiclly exponentil time). lterntive: use dynmic progrmming nd memoize sub-nlysis in chrt to void duplicte work. The chrt cn be visulized s s mtrix. The CYK lgorithm builds chrt in O(n 3 ) time. The bsic version gives just recognizer, but it cn be mde into prser if more info is recorded in the chrt. Reding: J&M (2nd ed), Chpter. 13, Sections NLTK Book, Chpter. 8 (nlyzing Sentence Structure), Section 8.4 Next lecture: the Erley prser or dynmic progrmming for topdown prsing 43 / 43