f is a function that maps a structure (x, y) to a feature vector w is a parameter vector (also a member of R d )

Transcription

1 Three Components of Globl Ler Models f is function tht mps structure (x, y) to feture vector f(x, y) R d (Fll 2007) Globl Ler Models: Prt II GEN is function tht mps n put x to set of cndidtes GEN(x) w is prmeter vector (lso member of R d ) Trg dt is used to set the vlue of w 1 3 Overview Puttg it ll Together Recp: globl ler models X is set of sentences, Y is set of possible outputs (e.g. trees) Log-ler models for prmeter estimtion Need to lern function F : X Y Globl nd locl fetures The perceptron revisited Log-ler models revisited GEN, f, w defe F (x) = rg mx f(x, y) w y GEN(x) Choose the highest scorg cndidte s the most plusible structure Given exmples (x i, y i ), how to set w? 2 4

2 progrm to promote sfety trucks nd vns progrm to promote sfety trucks nd vns progrm to promote sfety trucks nd progrm vns to promote progrm to promote sfety sfety trucks nd vns trucks nd vns progrm to promote sfety trucks nd vns progrm to promote sfety trucks nd vns progrm to promote sfety trucks nd vns GEN Overview Recp: globl ler models f f f f f f 1, 1, 3, 5 2, 0, 0, 5 1, 0, 1, 5 0, 0, 3, 0 0, 1, 0, 5 0, 0, 1, 5 f w f w f w f w f w f w rg mx Log-ler models for prmeter estimtion Globl nd locl fetures The perceptron revisited Log-ler models revisited 5 7 A Vrt of the Perceptron Algorithm Inputs: Trg set (x i, y i ) for i = 1... n Initiliztion: w = 0 Defe: F (x) = rgmx y GEN(x) f(x, y) w Bck to Mximum Likelihood Estimtion [Johnson et. l 1999] We cn use the prmeters to defe probbility for ech prse: e f(x,y) w P (y x, w) = y GEN(x) e f(x,y ) w Algorithm: For t = 1... T, i = 1... n z i = F (x i ) If (z i y i ) w = w + f(x i, y i ) f(x i, z i ) Log-likelihood is then L(w) = i log P (y i x i, w) Output: Prmeters w A first estimtion method: tke mximum likelihood estimtes, i.e., w ML = rgmx w L(w) 6 8

3 Addg Guss Priors [Johnson et. l 1999] A first estimtion method: tke mximum likelihood estimtes, i.e., w ML = rgmx w L(w) Unfortuntely, very likely to overfit A wy of preventg overfittg: choose prmeters s ( w MAP = rgmx w L(w) C ) w 2 k k Overview Recp: globl ler models Log-ler models for prmeter estimtion Globl nd locl fetures The perceptron revisited Log-ler models revisited for some constnt C Intuition: dds penlty for lrge prmeter vlues 9 11 ummry Choose prmeters s: ( w MAP = rgmx w L(w) C k where L(w) = log P (y i x i, w) i w 2 k ) Globl nd Locl Fetures o fr: lgorithms hve depended on size of GEN trtegies for keepg the size of GEN mngeble: Rernkg methods: use bsele model to generte its top N nlyses = i log e f(x i,y i ) w y GEN(x i ) e f(x i,y ) w Cn use (conjugte) grdient scent (see previous lectures on log-ler models) 10 12

4 Globl nd Locl Fetures Tggg Globl ler models re globl couple of wys: Feture vectors re defed over entire structures Prmeter estimtion methods explicitly relted to errors on entire structures Next topic: globl trg methods with locl fetures Our globl fetures will be defed through locl fetures Prmeter estimtes will be globl GEN will be lrge! Dynmic progrmmg used for serch nd prmeter estimtion: this is possible for some combtions of GEN nd f Gog bck to tggg: Inputs x re sentences w [1:n] = {w 1... w n } GEN(w [1:n] ) = T n i.e. ll tg sequences of length n Note: GEN hs n exponentil number of members How do we defe f? Tggg Problems TAGGING: trgs to Tgged equences b e e f h j /C b/d e/c e/c /D f/c h/d j/c Exmple 1: Prt-of-speech tggg Profits/N sored/v t/p Boeg/N Co./N,/, esily/adv toppg/v forecsts/n on/p Wll/N treet/n,/, s/p their/po CEO/N Aln/N Mullly/N /V first/adj qurter/n results/n./. Exmple 2: Nmed Entity Recognition Profits/NA sored/na t/na Boeg/C Co./CC,/NA esily/na toppg/na forecsts/na on/na Wll/L treet/cl,/na s/na their/na CEO/NA Aln/P Mullly/CP /NA first/na qurter/na results/na./na Representtion: Histories A history is 4-tuple t 2, t 1, w [1:n], i t 2, t 1 re the previous two tgs. w [1:n] re the n words the put sentence. i is the dex of the word beg tgged Hispniol/N quickly/rb becme/vb n/dt importnt/jj bse/?? from which p expnded its empire to the rest of the Western Hemisphere. t 2, t 1 = DT, JJ w [1:n] = Hispniol, quickly, becme,..., Hemisphere,. i =

5 Locl Feture-Vector Representtions Tke history/tg pir (h, t). g s (h, t) for s = 1... d re locl fetures representg tggg decision t context h. Exmple: PO Tggg Word/tg fetures { 1 if current word wi is bse nd t = VB g 100 (h, t) = 0 otherwise { 1 if current word wi ends g nd t = VBG g 101 (h, t) = 0 otherwise Contextul Fetures { 1 if t 2, t g 103 (h, t) = 1, t = DT, JJ, VB 0 otherwise A tgged sentence with n words hs n history/tg pirs Hispniol/N quickly/rb becme/vb n/dt importnt/jj bse/nn History Tg t 2 t 1 w [1:n] i t * * Hispniol, quickly,..., 1 N * N Hispniol, quickly,..., 2 RB N RB Hispniol, quickly,..., 3 VB RB VB Hispniol, quickly,..., 4 DT DT Hispniol, quickly,..., 5 JJ DT JJ Hispniol, quickly,..., 6 NN Defe globl fetures through locl fetures: n f(t [1:n], w [1:n] ) = g(h i, t i ) where t i is the i th tg, h i is the i th history A tgged sentence with n words hs n history/tg pirs Globl nd Locl Fetures Hispniol/N quickly/rb becme/vb n/dt importnt/jj bse/nn History Tg t 2 t 1 w [1:n] i t * * Hispniol, quickly,..., 1 N * N Hispniol, quickly,..., 2 RB N RB Hispniol, quickly,..., 3 VB RB VB Hispniol, quickly,..., 4 DT DT Hispniol, quickly,..., 5 JJ DT JJ Hispniol, quickly,..., 6 NN Typiclly, locl fetures re dictor functions, e.g., { 1 if current word wi ends g nd t = VBG g 101 (h, t) = 0 otherwise nd globl fetures re then counts, f 101 (w [1:n], t [1:n] ) = Number of times word endg g is tgged s VBG (w [1:n], t [1:n] ) 18 20

6 Puttg it ll Together GEN(w [1:n] ) is the set of ll tgged sequences of length n GEN, f, w defe ome notes: F (w [1:n] ) = rg mx t [1:n] GEN(w [1:n] ) w f(w [1:n], t [1:n] ) = rg mx w n g(h i, t i ) t [1:n] GEN(w [1:n] ) = rg mx t [1:n] GEN(w [1:n] ) n w g(h i, t i ) core for tgged sequence is sum of locl scores Dynmic progrmmg cn be used to fd the rgmx! (becuse history only considers the previous two tgs) Trg Tgger Usg the Perceptron Algorithm Inputs: Trg set (w[1:n i i ], ti [1:n i ]) for i = 1... n. Initiliztion: w = 0 Algorithm: For t = 1... T, i = 1... n z [1:ni ] = rg mx u [1:ni ] T n i w f(w i [1:n i ], u [1:ni ]) z [1:ni ] cn be computed with the dynmic progrmmg (Viterbi) lgorithm If z [1:ni ] t i [1:n i ] then w = w + f(w i [1:n i ], t i [1:n i ]) f(w i [1:n i ], z [1:ni ]) Output: Prmeter vector w A Vrt of the Perceptron Algorithm Inputs: Trg set (x i, y i ) for i = 1... n Initiliztion: w = 0 Defe: F (x) = rgmx y GEN(x) f(x, y) w An Exmple y the correct tgs for i th sentence re the/dt mn/nn bit/vbd the/dt dog/nn Under current prmeters, output is the/dt mn/nn bit/nn the/dt dog/nn Algorithm: Output: For t = 1... T, i = 1... n z i = F (x i ) If (z i y i ) w = w + f(x i, y i ) f(x i, z i ) Prmeters w Assume lso tht fetures trck: (1) ll bigrms; (2) word/tg pirs Prmeters cremented: NN, VBD, VBD, DT, VBD bit Prmeters decremented: NN, NN, NN, DT, NN bit 22 24

7 Experiments Wll treet Journl prt-of-speech tggg dt Perceptron = 2.89%, Mx-ent = 3.28% (11.9% reltive error reduction) [Rmshw nd Mrcus, 1995] chunkg dt Perceptron = 93.63%, Mx-ent = 93.29% (5.1% reltive error reduction) Log-Ler Tggg Models Tke history/tg pir (h, t). g s (h, t) w s for s = 1... d re fetures for s = 1... d re prmeters Conditionl distribution: P (t h) = ew g(h,t) Z(h, w) where Z(h, w) = t T e w g(h,t ) Prmeters estimted usg mximum-likelihood 25 How Does this Differ from Log-Ler Tggers? Log-ler tggers ( n erlier lecture) used very similr locl representtions How does the perceptron model differ? Why might these differences be importnt? 27 Log-Ler Tggg Models Word sequence w [1:n] = [w 1, w 2... w n ] Tg sequence t [1:n] = [t 1, t 2... t n ] Histories h i = t i 1, t i 2, w [1:n], i log P (t [1:n] w [1:n] ) n n = log P (t i h i ) = w g(h i, t i ) }{{} Ler core n log Z(h i, w) }{{} Locl Normliztion Terms Compre this to the perceptron, where GEN, f, w defe F (w [1:n] ) = n rg mx w g(h i, t i ) t [1:n] GEN(w [1:n] ) }{{} Ler score 26 28

8 Problems with Loclly Normlized models Lbel bis problem [Lfferty, McCllum nd Pereir 2001] ee lso [Kle nd Mnng 2002] Exmple of conditionl distribution tht loclly normlized models cn t cpture (under bigrm tg representtion): b c A B C with P (A B C b c) = 1 b c b e A D E b e Impossible to fd prmeters tht stisfy with P (A D E b e) = 1 P (A ) P (B b, A) P (C c, B) = 1 P (A ) P (D b, A) P (E e, D) = 1 Globl Log-Ler Models We cn use the prmeters to defe probbility for ech tgged sequence: where P (t [1:n] w [1:n], w) = e i w g(h i,t i ) Z(w [1:n], w) Z(w [1:n], w) = is globl normliztion term t [1:n] GEN(w [1:n] ) This is globl log-ler model with f(w [1:n], t [1:n] ) = i e i w g(h i,t i ) g(h i, t i ) Overview Recp: globl ler models, nd boostg Log-ler models for prmeter estimtion An ppliction: LFG prsg Globl nd locl fetures The perceptron revisited Log-ler models revisited Now we hve: log P (t [1:n] w [1:n] ) n = w g(h i, t i ) } {{ } Ler core log Z(w [1:n], w) }{{} Globl Normliztion Term When fdg highest probbility tg sequence, the globl term is irrelevnt: n ( rgmx t[1:n] GEN(w [1:n] ) w g(hi, t i ) log Z(w [1:n], w) ) n = rgmx t[1:n] GEN(w [1:n] ) w g(h i, t i ) 30 32

9 Prmeter Estimtion For prmeter estimtion, we must clculte the grdient of n log P (t [1:n] w [1:n] ) = w g(h i, t i ) log e i w g(h i,t i ) t [1:n] GEN(w [1:n]) with respect to w Dynmic progrmmg is lso used trg: Perceptron requires highest-scorg tg sequence for ech trg exmple Globl log-ler model requires grdient, nd therefore expected counts Tkg derivtives gives dl n dw = n g(h i, t i ) P (t [1:n] w [1:n], w) g(h i, t i) t [1:n] GEN(w [1:n]) Cn be clculted usg dynmic progrmmg! ummry of Perceptron vs. Globl Log-Ler Model Both re globl ler models, where GEN(w [1:n] ) = the set of ll possible tg sequences for w [1:n] f(w [1:n], t [1:n] ) = g(h i, t i ) i In both cses, F (w [1:n] ) = rgmx t[1:n] GEN(w [1:n] )w f(w [1:n], t [1:n] ) = rgmx t[1:n] GEN(w [1:n] ) w g(h i, t i ) cn be computed usg dynmic progrmmg i From [h nd Pereir, 2003] Results Tsk = shllow prsg (bse noun-phrse recognition) Model Accurcy VM combtion 94.39% Conditionl rndom field 94.38% (globl log-ler model) Generlized wnow 93.89% Perceptron 94.09% Locl log-ler model 93.70% 34 36