3 Noisy Channel model

Size: px

Start display at page:

Download "3 Noisy Channel model"

Grant Perkins
5 years ago
Views:

1 3 Noisy Channl modl W obsrv a distortd mssag R (forign string f). W hav a modl on how th mssag is distortd (translation modl t(f )) and also a modl on which original mssags ar probabl (languag modl p()). Our objct is to rcovr th original mssag S (English string ). Drivation of noisy channl modl in a probabilistic framwork using th Bays rul ê = arg max p( f) (1) = arg max = arg max p(f )p() p(f) p(f ) translation modl p() languag modl Advantags of gnrativ SMT modls: Th translation problm can b brokn up into simplr problms: translation modl and languag modl. Lik that, simplr problms can b solvd sparatly and stimation and modl dfinitions ar indpndnt. Not: Sourc in th nois channl modl should b rad as original. In trms of gnrativ SMT w calld th original th targt. 22

2 4 Phras-basd SMT Forign input is sgmntd into phrass. Phrass ar translatd into English. Phrass ar rordrd. This works alrady bttr thn word-basd translation modls (which ar still usd in alignmnt). Phras-basd SMT as gnrativ modl: ê = arg max p( f) (2) = arg max p(f ) translation modl p() languag modl 4.1 Rordring modl in Phras-basd SMT p( f I 1 ēi 1) = I i=1 φ( f i ē i ) phras translation probability d(start i nd i 1 1) (phras) distortion probability d is calld distortion probability, rordring probability or rathr distortion cost. It dscribs th numbr of words skippd to th right (+) or lft (-) whn taking forign words out of squnc. start i is th position of th first word of a forign phras corrsponding to th i th English phras (in th figur abov, start 4 = 4). nd i 1 is th position of th last word of a forign phras corrsponding to th prvious English phras (in th figur abov, nd 3 = 2). 23 (3)

3 phras translats movmnt distanc calculation start at bginning =0 2 6 skip ovr = mov back ovr = skip ovr =+1 Exampl (phras 2): d(start i nd i 1 1) = d(start 2 nd 1 1) = d(6 3 1) = 2 Exampl (phras 3): d(start i nd i 1 1) = d(start 3 nd 2 1) = d(4 6 1) = 3 Scoring function: d(x) = α x 4.2 Wightd modls in SMT With th knowldg w hav w can build th standard gnrativ modl consisting of 3 submodls / moduls: phras translation modl φ( f ē) distortion modl d languag modl p LM () Th gnrativ modl dfins joint probability by assuming th indpndnc of moduls product modl. Th standard modl was drivd (inspird) mor or lss from a wll-dfind optimization task (max. liklihood). Quit xpctd, liklihood is not what popl rport whn askd to valuat a translation w hav a modl-task discrpancy. Mor complx objctivs usually rquir mor flxibl modls. 24

4 From an nginring prspctiv, th dvlopr may notic that th translations nd to b mor flunt and that incrasing a languag modl s importanc is ncssary. This is not possibl within th noisy channl modl (violats th Bays rul). Morovr, h may argu that combining svral languag modls might b bnficial. To summariz, rasons for a mov from th product modl to mor gnral log-linar modls: including mor faturs, tuning th rlativ importanc of faturs. Whil w violat th assumptions of th initial modl w still want to land on som known modl Wightd modls Add wight to moduls: ê = arg max I φ( f i ē i ) λφ (4) i d(start i nd i 1 1) λd p LM ( j 1,..., j 1 ) λ LM j=1 Now w work with diffrnt, mor xprssiv probability modls log-linar modl log-linar modl n p(x) = xp λ i h i (x) (5) i=1 λ i = paramtr h i = faturs SMT as log-linar modl: 25

5 fatur-function h 1 = log φ φ = h 1 fatur-function h 2 = log d d = h 2 fatur-function h 3 = log p LM p LM = h 3 φ λφ d λd p λlm LM = h 1λ 1 h 2λ 2 h 3λ 3 (6) 3 = i=1 h iλ i = 3 i=1 h iλ i ê = arg max xp(λ φ I + λ d log d(start i nd i 1 1) i + λ LM log p LM ( j 1,..., j 1 )) j=1 I log φ( f i ē i ) (7) Although w obtaind log-linar modls as a gnralization of a product modl, it turns out that th following formulations ar dual: MaxEnt + momnt consrvation ML + Gibbs i Advantags of log-linar modl Standard moduls can b wightd. Additional fatur functions can b addd asily. 3 standard faturs: φ, d, p LM 26

6 Additional faturs: With th hlp of fatur w can now asily injct domain knowldg into th modl. Th final answr whthr ths nw fatur ar usful will b always givn by xprimnt, but having som intuition is ncssary to dsign thm. Word count: wc() = log ω, ω<1 prfrs fwr words ω>1 prfrs mor words Th wc fatur corrcts th bias of th languag modl towards short translations. Phras count: pc() = log I ρ, I=numbr of phrass ρ<1 prfrs fwr phrass, i.., longr phras ρ>1 prfrs shortr phrass, i.., mor phrass Th pc fatur fin-tuns fin or coars phras sgmntation. Trad-off: longr phrass ar mor grammatical but lss statistically rliabl; combinations of shortr phrass ar mor oftn disflunt but thir statistics can b stimatd on smallr corpora. Th task of this fatur is to automatically rsolv this trad-off for your particular cas (data). In contrast, choosing mor linguistically motivatd boundaris was not shown to b spcially hlpful. Multipl languag modls Multipl translation modls:.g., src-trg and trg-src translation modls Bidirctional alignmnt probabilitis Exampl: vry long English phras ē is xtractd along with a forign phras f. In rsults, φ( f ē) is high, LM will lik it as wll will b oftn usd. Add th rvrs dirction φ(ē f) (small) to prvnt this from happning, and wight th rlativ importanc of both modl. Lxically wightd phras translation probabilitis: Lxical wighting of phrass with word translation probabilitis for rar phrass with unrliabl phras translation probabilitis. W would lik 27

7 to back-off to th word lvl to compnsat for th missing information: lx(ē f, a) = ē i=1 1 {j (i, j) a} (i,j) a w( i f j ) Each English word i in a phras ē is gnratd indpndntly by an alignd forign word f j in f with th word translation probability w( i f j ) or avrag if multipl alignmnt is possibl. Again, w can us both lx(ē f) and lx( f ē). ght nicht davon aus NULL dos not assum Exampl: lx(ē f, a) = w(dos NULL) w(not nicht) 1 (w(assum ght) + w(assum davon) 3 + w(assum aus)) 4.3 Lxicalizd Rordring Rordring basd on distanc in words is not xprssiv nough. W want to b abl to distinguish th rordring cost for diffrnt lxical phrass. Lxical rordring larns 3 typs of rordring for ach lxical phras: orintation {monoton, swap, discontinuous} Larning orintation prfrnc p(orintation f, ē) is th lxical rordring probability distribution. During phras xtraction from alignmnt, chck: 28

8 if a word alignmnt point to th top lft xists monoton (m) lsif a word alignmnt point to th top right xists swap (s) ls discontinuous (d) Estimation of lxical rordring probability: unsmoothd stimat: ˆp(orintation ē, f) = count(orintation, ē, f) count(orintation, ē, f) o smoothd stimat: ˆp(orintation ē, f) = λp(orintation) + count(orintation, ē, f) λ 1 + o count(orintation, ē, f) whr p(orintation) = o f f ē ē count(orintation, ē, f) count(orintation, ē, f) linar intrpolation with unlxicalizd orintation modl. 4.4 Dirct training of phrass Why not try to dirctly larn phras-alignmnt with EM? Goal: Dirctly larn phrass using EM without huristic xtraction from word alignmnts. 29

9 1. Initializ: Start with uniform φ(ē, f) phras pair probabilitis. 2. E-Stp: Assign phras alignmnt probabilitis to all phrass in all sntnc pairs. 3. M-Stp: Collct counts for phras pairs (ē, f), wightd by th phras alignmnt probability. 4. Updat phras translation probabilitis φ(ē, f). Problms: Mthod ovrfits asily: Long phras pairs, oftn spanning ntir sntncs, ar prfrrd. Infficint, bcaus of larg spac of alignmnts. Solutions: Rstrict th phras lngth; disallow phrass or phras pairs that occur only onc. Us old-styl huristics: Word alignmnts rstricts possibl phrass. Nw-styl Baysian approach: Dfin prior that imposs a bias towards shortr phrass. 30

Data Assimilation 1. Alan O Neill National Centre for Earth Observation UK

Data Assimilation 1. Alan O Neill National Centre for Earth Observation UK Data Assimilation 1 Alan O Nill National Cntr for Earth Obsrvation UK Plan Motivation & basic idas Univariat (scalar) data assimilation Multivariat (vctor) data assimilation 3d-Variational Mthod (& optimal