f is a function that maps a structure (x, y) to a feature vector w is a parameter vector (also a member of R d )

Transcription

1 Three Components of Globl Ler Models f is function tht mps structure (x, y) to feture vector f(x, y) R d (Fll 2007) Globl Ler Models: Prt III GEN is function tht mps n put x to set of cndidtes GEN(x) w is prmeter vector (lso member of R d ) Trg dt is used to set the vlue of w 1 3 Overview Recp: globl ler models Dependency prsg GLMs for dependency prsg Eisner s prsg lgorithm Results from McDonld (2005) Puttg it ll Together X is set of sentences, Y is set of possible outputs (e.g. trees) Need to lern function F : X Y GEN, f, w defe F (x) = rg mx f(x, y) w y GEN(x) Choose the highest scorg cndidte s the most plusible structure Given exmples (x i, y i ), how to set w? 2 4

2 he nnounced to promote sfety trucks nd vns he nnounced to promote sfety trucks nd vns he nnounced he to promote nnounced sfety trucks nd vns to he nnounced promote to promote sfety sfety trucks nd vns trucks nd he nnounced vns to promote sfety trucks nd vns he nnounced to promote sfety trucks nd vns he nnounced to promote sfety trucks nd vns GEN A tgged sentence with n words hs n history/tg pirs Hispniol/N quickly/rb becme/vb n/dt importnt/jj bse/nn f f f f f f 1, 1, 3, 5 2, 0, 0, 5 1, 0, 1, 5 0, 0, 3, 0 0, 1, 0, 5 0, 0, 1, 5 f w f w f w f w f w f w rg mx History Tg t 2 t 1 w [1:n] i t * * Hispniol, quickly,..., 1 N * N Hispniol, quickly,..., 2 RB N RB Hispniol, quickly,..., 3 VB RB VB Hispniol, quickly,..., 4 DT DT Hispniol, quickly,..., 5 JJ DT JJ Hispniol, quickly,..., 6 NN Defe globl fetures through locl fetures: n f(t [1:n], w [1:n] ) = g(h i, t i ) i=1 where t i is the i th tg, h i is the i th history 5 7 A Vrt of the Perceptron Algorithm Globl nd Locl Fetures Inputs: Initiliztion: w = 0 Trg set (x i, y i ) for i = 1... n Typiclly, locl fetures re dictor functions, e.g., Defe: Algorithm: Output: F (x) = rgmx y GEN(x) f(x, y) w For t = 1... T, i = 1... n z i = F (x i ) If (z i y i ) w = w + f(x i, y i ) f(x i, z i ) Prmeters w g 101 (h, t) = { 1 if current word wi ends g nd t = VBG 0 otherwise nd globl fetures re then counts, f 101 (w [1:n], t [1:n] ) = Number of times word endg g is tgged s VBG (w [1:n], t [1:n] ) 6 8

3 Puttg it ll Together GEN(w [1:n] ) is the set of ll tgged sequences of length n GEN, f, w defe ome notes: F (w [1:n] ) = rg mx t [1:n] GEN(w [1:n] ) w f(w [1:n], t [1:n] ) = rg mx w n g(h i, t i ) t [1:n] GEN(w [1:n] ) = rg mx t [1:n] GEN(w [1:n] ) i=1 i=1 n w g(h i, t i ) core for tgged sequence is sum of locl scores Dynmic mg cn be used to fd the rgmx! (becuse history only considers the previous two tgs) Trg Tgger Usg the Perceptron Algorithm Inputs: Trg set (w[1:n i i ], ti [1:n i ]) for i = 1... n. Initiliztion: w = 0 Algorithm: For t = 1... T, i = 1... n z [1:ni ] = rg mx u [1:ni ] T n i w f(w i [1:n i ], u [1:ni ]) z [1:ni ] cn be computed with the dynmic mg (Viterbi) lgorithm If z [1:ni ] t i [1:n i ] then w = w + f(w i [1:n i ], t i [1:n i ]) f(w i [1:n i ], z [1:ni ]) Output: Prmeter vector w A Vrt of the Perceptron Algorithm Overview Inputs: Trg set (x i, y i ) for i = 1... n Recp: globl ler models Initiliztion: w = 0 Dependency prsg Defe: Algorithm: Output: F (x) = rgmx y GEN(x) f(x, y) w For t = 1... T, i = 1... n z i = F (x i ) If (z i y i ) w = w + f(x i, y i ) f(x i, z i ) Prmeters w GLMs for dependency prsg Eisner s prsg lgorithm Results from McDonld (2005) 10 12

4 Unlbeled Dependency Prses A More Complex Exmple root John sw movie tht he liked tody root John sw movie root is specil root symbol Ech dependency is pir (h, m) where h is the dex of hed word, m is the dex of modifier word. In the figures, we represent dependency (h, m) by directed edge from h to m. Dependencies the bove exmple re (0, 2), (2, 1), (2, 4), nd (4, 3). (We tke 0 to be the root symbol.) All Dependency Prses for John sw Mry Conditions on Dependency tructures root John sw Mry root John sw Mry root John sw Mry root John sw Mry root John sw Mry root John sw movie tht he liked tody The dependency rcs form directed tree, with the root symbol t the root of the tree. (Defition: A directed tree rooted t root is tree, where for every word w other thn the root, there is directed pth from root to w.) There re no crossg dependencies. Dependency structures with no crossg dependencies re sometimes referred to s projective structures

5 Lbeled Dependency Prses (told,v) (Hillry,N) (told,vbd) N Hillry V(told,VBD) (Clton,N) BAR(tht,COMP) imilr to unlbeled structures, but ech dependency is triple (h, m, l) where h is the dex of hed word, m is the dex of modifier word, nd l is lbel. In the figures, we represent dependency (h, m, l) by directed edge from h to m with lbel l. For most of this lecture we ll stick to unlbeled dependency structures. VBD told N Clton COMP tht (she,prp) PRP she Vt ws (ws,vt) (president,nn) NN president ( told VBD TOP PECIAL) (told VBD Hillry N LEFT) (told VBD Clton N VBD RIGHT) (told VBD tht COMP VBD BAR RIGHT) (tht COMP ws Vt BAR COMP RIGHT) (ws Vt she PRP LEFT) (ws Vt president Vt RIGHT) Extrctg Dependency Prses from Treebnks (told,v) There s recently been lot of terest dependency prsg. For exmple, the CoNLL 2006 conference hd shred tsk where 12 lnguges were volved (Arbic, Chese, Czech, Dnish, Dutch, Germn, Jpnese, Portuguese, lovene, pnish, wedish, Turkish). 19 different groups developed dependency prsg systems. CoNLL 2007 hd similr shred tsk. Google for conll 2006 shred tsk for more detils. For recent PhD thesis on the topic, see Ryn McDonld, Discrimtive Trg nd pnng Tree Algorithms for Dependency Prsg, University of Pennsylvni. (Hillry,N) N Hillry V(told,VBD) VBD told (told,vbd) (Clton,N) N Clton BAR(tht,COMP) COMP tht (she,prp) PRP she (ws,vt) Vt (president,nn) ws NN For some lnguges, e.g., Czech, there re dependency bnks vilble which cont trg dt the form of sentences pired with dependency structures For other lnguges, we hve treebnks from which we cn extrct dependency structures, usg lexiclized grmmrs described erlier the course (see Prsg nd yntx 2) Unlbeled Dependencies: (0,2) (for root told) (2,1) (for told Hillry) (2,3) (for told Clton) (2,4) (for told tht) (4,6) (for tht ws) (6,5) (for ws she) (6,7) (for ws president) president 18 20

6 Efficiency of Dependency Prsg PCFG prsg is O(n 3 G 3 ) where n is the length of the sentence, G is the number of non-termls the grmmr Lexiclized PCFG prsg is O(n 5 G 3 ) where n is the length of the sentence, G is the number of non-termls the grmmr. (With the lgorithms we ve seen it is possible to do little better thn this.) x is sentence GLMs for Dependency prsg GEN(x) is set of ll dependency structures for x f(x, y) is feture vector for sentence x pired with dependency prse y Unlbeled dependency prsg is O(n 3 ). (ee prt 4 of these slides for the lgorithm.) Overview Recp: globl ler models Dependency prsg Globl Ler Models (GLMs) for dependency prsg Eisner s prsg lgorithm Results from McDonld (2005) GLMs for Dependency prsg To run the perceptron lgorithm, we must be ble to efficiently clculte rg mx w f(x, y) y GEN(x) Locl feture vectors: defe f(x, y) = (h,m) y g(x, h, m) where g(x, h, m) mps sentence x nd dependency (h, m) to locl feture vector Cn then efficiently clculte rg mx y GEN(x) w f(x, y) = rg mx y GEN(x) (h,m) y w g(x, h, m) 22 24

7 Defition of Locl Feture Vectors g(x, h, m) mps sentence x nd dependency (h, m) to locl feture vector Fetures from McDonld et l. (2005): Note: defe w i to be the i th word the sentence, t i to be the prtof-speech (PO) tg for the i th word. Unigrm fetures: Identity of w h. Identity of w m. Identity of t h. Identity of t m. Bigrm fetures: Identity of the 4-tuple w h, w m, t h, t m. Identity of sub-sets of this 4-tuple, e.g., identity of the pir w h, w m. Contextul fetures: Identity of the 4-tuple t h, t h+1, t m 1, t m. imilr fetures which consider t h 1 nd t m+1, givg 4 possible feture types. In-between fetures: Identity of triples t h, t, t m for ny tg t seen between words h nd m. Eisner s Algorithm for Dependency Prsg Runs O(n 3 ) time for sentence of length n Algorithm is similr to the dynmic mg lgorithm for PCFGs, but represents constituents novel wy The problem: fd rg mx y GEN(x) (h,m) y (h, m) where x is sentence, GEN(x) is the set of ll dependency trees for x, nd (h, m) is the score of dependency (h, m). In our cse, (h, m) = w g(x, h, m) Overview Recp: globl ler models Dependency prsg Globl Ler Models (GLMs) for dependency prsg Eisner s prsg lgorithm Results from McDonld (2005) Complete Constituents A complete consituent with direction for words w s... w t is set of dependencies D such tht: Every word w s+1... w t is modifier to some word The dependencies D form well formed dependency sub-prse: i.e., there re no crossg dependencies, or cycles. No dependencies D volve words other thn w s is the hed of t lest one dependency. Note: this mens tht the dependencies D form directed tree tht spns ll words w s... w t, with w s t the root of the tree

8 Complete Constituents A complete consituent with direction for words w s... w t is set of dependencies D such tht: Every word w s... w t 1 is modifier to some word The dependencies D form well formed dependency sub-prse: i.e., there re no crossg dependencies, or cycles. No dependencies D volve words other thn w t is the hed of t lest one dependency. Note: this mens tht the dependencies D form directed tree tht spns ll words w s... w t, with w t t the root of the tree. 29 Incomplete Constituents An complete consituent with direction for words w s... w t is set of dependencies D such tht: Every word w s... w t 1 is modifier to some word The dependencies D form well formed dependency sub-prse: i.e., there re no crossg dependencies, or cycles. No dependencies D volve words other thn w t is the hed of t lest one dependency. A new condition: there is dependency (t, s) D. Note: ny complete constituent is lso complete constituent 31 Incomplete Constituents An complete consituent with direction for words w s... w t is set of dependencies D such tht: Every word w s+1... w t is modifier to some word The dependencies D form well formed dependency sub-prse: i.e., there re no crossg dependencies, or cycles. No dependencies D volve words other thn w s is the hed of t lest one dependency. A new condition: there is dependency (s, t) D. Note: ny complete constituent is lso complete constituent 30 The Dynmic Progrmmg Tble C[s][t][d][c] is the highest score for ny constituent tht: pns words w s... w t Hs direction d (either or ) Hs type c (c = 0 for complete constituents, c = 1 for complete constituents) Bse cse for the dynmic mg lgorithm: for s = 1... n, C[s][s][ ][1] = C[s][s][ ][1] =

9 Intuition: Cretg Incomplete Constituents We cn form n complete constituent spnng words w s... w t by combg two complete constituents. Intuition: Cretg Complete Constituents We cn form complete constituent spnng words w s... w t by combg n complete nd complete constituent Cretg Incomplete Constituents Cretg Complete Constituents First cse: for ny s, t such tht 1 s < t n, C[s][t][ ][0] = mx (C[s][r][ ][1] + C[r + 1][t][ ][1] + (t, s)) s r<t Intuition: combe two complete constituents to form n complete constituent econd cse: for ny s, t such tht 1 s < t n, C[s][t][ ][0] = mx (C[s][r][ ][1] + C[r + 1][t][ ][1] + (s, t)) s r<t First cse: for ny s, t such tht 1 s < t n, C[s][t][ ][1] = mx (C[s][r][ ][1] + C[r][t][ ][0]) s r<t Intuition: combe one complete constituent, one complete constituent, to form complete constituent econd cse: for ny s, t such tht 1 s < t n, C[s][t][ ][1] = mx (C[s][r][ ][0] + C[r][t][ ][1]) s<r t 34 36

10 Initiliztion: The Full Algorithm Results from McDonld (2005) for k = 1... n + 1 for s = 0... n t = s + k if t > n then brek for s = 0... n, C[s][s][ ][1] = C[s][s][ ][1] = 0.0 % First: crete complete items C[s][t][ ][0] = mx s r<t (C[s][r][ ][1] + C[r + 1][t][ ][1] + (t, s)) C[s][t][ ][0] = mx s r<t (C[s][r][ ][1] + C[r + 1][t][ ][1] + (s, t)) % econd: crete complete items C[s][t][ ][1] = mx s r<t (C[s][r][ ][1] + C[r][t][ ][0]) C[s][t][ ][1] = mx s<r t (C[s][r][ ][0] + C[r][t][ ][1]) Return C[0][n][ ][1] s the highest score for ny prse Method Accurcy Colls (1997) 91.4% 1st order dependency 90.7% 2nd order dependency 91.5% Accurcy is percentge of correct unlbeled dependencies Colls (1997) is result from lexiclized context-free prser, with dependencies extrcted from the prser s output 1st order dependency is the method just described. 2nd order dependency is model tht uses richer representtions. Advntges of the dependency prsg pproches: simplicity, efficiency (O(n 3 ) prsg time) Overview Extensions Recp: globl ler models Dependency prsg Globl Ler Models (GLMs) for dependency prsg 2nd-order dependency prsg Non-projective dependency structures Eisner s prsg lgorithm Results from McDonld (2005) * John sw movie tody tht he liked 38 40