Neural network LM. CS 690N, Spring 2018 Advanced Natural Language Processing

Transcription

1 Neurl network LM CS 690N, Spring 2018 Advnced Nturl Lnguge Processing Brendn O Connor College nformtion nd Computer Sciences Universy Msschusetts Amrst

2 Pper presenttions Groups 2-3 Aim for 20 minutes Choose full-length reserch pper in NLP, or computtionl linguistics Choose rself (nd get our pprovl >1 ek out), or choose from list Similr to t reding feedbck wring: Summriztion (wht did ty do? wht methods? wht dt), explntion (wht re t contributions?), syntsis nd crique (wht re t strengths/ knesses? reltionships to otr work or future work?) 2

3 Bengio et l 2003: N-grm multilyer perceptron i-th output = P(w t = i context) stmx most computtion re tnh f (w t,,w t n+1 )= ˆP(w t w t 1 1 ) ( ) Lern: C, W, U, H, d (chin rule) C(i) 2 R m Word embedding prmeters x =(C(w t 1 ),C(w t 2 ),,C(w t n+1 )) Lookup lyer wh conction: (kind) hidden lyer size (n-1)m shortcut liner lyer notr hidden lyer, size h C(w t n+1 ) C(w t 2 ) C(w t 1 ) y = b +Wx+U tnh(d + Hx) Tble look up in C Mtrix C shred prmeters cross words Vocb output: log-probs size V w t n+1 w t 2 index for index for index for w t 1 ˆP(w t w t 1, w t n+1 )= ey wt i e y i 3 Output lyer (stmx / log-liner)

4 Embedding lookup (C: dim (V,m)) equivlent to one-hot encoding (len V) + hidden lyer (C) 4

5 Why? Curse dimensionly: bottleneck informtion into K30 hidden dimensions (K<< V) NNs cn lern complicted functions don t relly hve good grip on wht s lernble beyond universl function pproximtion but seems better thn liner dim reduction (eg S+P) Non-plnr regions in embedding spce? Multilyer structures Mybe: deep models lern more bstrct concepts (clerly in vision; less cler for NLP, though cn lp) Definely: hierrchicl nd sequentil NNs to mtch hierrchicl/memory-ful structure in lnguge (recursive/ recurrent NNs) 5

6 Word/feture embeddings Lookup lyer : from discrete input fetures (words, ngrms, etc) to continuous vectors Any binry feture tht ws directly used in log-liner models, give vector Chrcter n-grms, prt--speech tgs, etc As model prmeters: lern tm like everything else Or, s externl informtion: use pretrined embeddings Common in prctice: use fster-to-trin model on very lrge, perhps dferent, dtset [eg word2vec, glove pretrined word vectors] Shred representtions for domin dpttion nd multsk lerning 6

7 Nonliner ctivtion functions sigmoid(x) = ex 1+e x tnh(x) =2 sgm(x) 1 (x) + = mx(0,x) posive prt k ReLU 7

8 Neurl Lnguge Models: Smpling w n w n 1, w n 2 ˆp n t ws nd ll r ct rdvrk ˆp n h n w n 2 w n 1 [Slide: Phil Blunsom]

9 Neurl Lnguge Models: Smpling w n w n 1, w n 2 ˆp n t ws nd ll r ct rdvrk ˆp 1 h 1 w 1 w 0 <s> <s> [Slide: Phil Blunsom]

10 Neurl Lnguge Models: Smpling w n w n 1, w n 2 ˆp n t ws nd ll r ct rdvrk ˆp 1 t ws nd ll r ct rdvrk ˆp 2 h 1 h 2 w 1 w 0 w 0 w 1 <s> <s> [Slide: Phil Blunsom]

11 t ws nd ll r ct rdvrk w 1 <s> w0 p 1 t ws nd ll r ct rdvrk h1 w0 wn wn w1 p 2 t ws nd ll r ct rdvrk Neurl Lnguge Models: Smpling 1, wn 2 p n h2 w1 p 3 h3 w2 <s> [Slide: Phil Blunsom]

12 t ws nd ll r ct rdvrk w 1 <s> w0 p 1 t ws nd ll r ct rdvrk h1 w0 w1 p 2 h2 w1 1, wn 2 w2 p 3 t ws nd ll r ct rdvrk wn wn t ws nd ll r ct rdvrk Neurl Lnguge Models: Smpling p n h3 w2 p 4 h4 w3 <s> [Slide: Phil Blunsom]

13 Neurl Lnguge Models: Trining T usul trining objective is t cross entropy t dt given t model (MLE): F = 1 N X n cost n (w n, ˆp n ) T cost function is simply t model s estimted log-probbily w n : w n cost n ˆp n h n cost(, b) = T log b (ssuming w i is one hot encoding t word) w n 2 w n 1 [Slide: Phil Blunsom]

14 Neurl Lnguge Models: Trining Clculting t grdients is strightforwrd wh bck propgtion: w n = = 1 N P ˆp n P ˆp ˆp w n 2 ˆp n h n w n 1 [Slide: Phil Blunsom]

15 Neurl Lnguge Models: Trining Clculting t grdients is strightforwrd wh bck propgtion: 4 4 X 1 p 1 p =, n=1 w n=1 w1 w2 cost1 1 F w3 w4 cost2 cost3 cost4 p 1 p 2 p 3 p 4 h1 h2 h3 h4 w0 w0 w1 w1 w2 w2 w3 Note tht clculting t grdients for ech time step n is independent ll otr timesteps, s such ty re clculted in prllel nd summed [Slide: Phil Blunsom]

16 Comprison wh Count Bsed N-Grm LMs Good Better generlistion on unseen n-grms, poorer on seen n-grms Solution: direct (liner) ngrm fetures Simple NLMs re n order mgnude smller in memory footprint thn tir vnill n-grm cousins (though not use t liner fetures suggested bove!) Bd T number prmeters in t model scles wh t n-grm size nd thus t length t history cptured T n-grm history is fine nd thus tre is lim on t longest dependencies tht n be cptured Mostly trined wh Mximum Likelihood bsed objectives which do not encode t expected frequencies words priori [Slide: Phil Blunsom]

17 Trining NNs Dropout (preferred regulriztion method) Minibtch (dptive) SGD Prlleliztion (CPUs, GPUs) whin minibtch Locl optim (?) 17

18 Boring old SGD x t+1 = x t g t, prms x, lerning rte η, minibtch timestep t, grdient gt (typiclly: lerning rte decy on fixed scdule or constnt lerning rte?) 18

19 Boring old SGD x t+1 = x t g t, prms x, lerning rte η, minibtch timestep t, grdient gt (typiclly: lerning rte decy on fixed scdule or constnt lerning rte?) p Adptive SGD AdGrd: simplest dptive SGD methods Hs per-prmeter, dptive lerning rtes x t+1,i = x t,i x t+1 = x t G 1/2 t q P q Pt g t,i, t 0 =1 g2 t 0,i g t f G ws t Hessin, nd clculted g nd G on t whole btch, this would be Newton-Rphson step Relted: (Nesterov) momentum Vrints wh tricks bout history decy, etc (eg Adm, RMSprop, Addelt) 19

20 Locl vs globl models Locl models Long-history models w t w t 2,w t 1 w t w 1,w t 1 Fully observed direct word models Lt-clss direct word models Log-liner models Mrkovin neurl LM Recurrent neurl LM 20