INF5820/INF9820 LANGUAGE TECHNOLOGICAL APPLICATIONS. Jan Tore Lønning, Lecture 4, 10 Sep.

Transcription

1 INF5820/INF9820 LANGUAGE TECHNOLOGICAL ALICATIONS Jn Tor Lønning Lctur 4 0 Sp. tl@ii.uio.no

2 Tody 2 Sttisticl chin trnsltion: Th noisy chnnl odl Word-bsd Trining IBM odl

3 3

4 SMT xpl 4 En kokk lgd n rtt d bygg. 0.9 ch 0.6 d right 0.9 with 0.4 building 0.45 cook 0.3 crtd 0.25 stright 0.7 by 0.3 construction 0.33 prprd 0.5 court 0.2 o 0.2 brly 0. constructd 0.2 dish 0. cookd 0.05 cours 0.07 Siilrly or: pos 0-2 2x3 pos -3 pos 2-4 pos 3-5 4x5 pos 6-8 os4 pos 6 x3x3 ny os5 pos 7 5x3x3 ny right with 2.7x0-2 right with building.7x0-8 right o.5x0-0 right with construction 5.4x0-8 right by 9.7x0-2 right with brly 8.7x0-9 cours o.5x0-4 cours o brly.5x0-6

5 Sttisticl Mchin Trnsltion - SMT INF5820 Jn Tor Lønning Dprtnt o Inortics Univrsitty o Oslo Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

6 Sttisticl lrning Gol Find th bst ost probbl English trnsltion Ê o orign sntnc F. Ê = rg x E F E 3 stps coon to ny tsks A odl. W y not hv sn F bor. Th odl will dtrin wht to look or. 2 W ust lrn or stit th prtrs o th odl ro dt. 3 W ust hv thod or using th odl to ind th bst E givn F dcoding. Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

7 Noisy chnnl odls Applying Bys orul Ê = rg x E F E = rg x E F E E F = rg x F EE E Turning th pictur: considr F s trnsltion distortion o E nd sk which E? Why? Suitbl or pproxitions. Mks us o lngug odl E. c. K:SMT slid 34 Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

8 Noisy chnnls Th noisy chnnl odl Expl S distortion o th originl. Gol: guss th originl J&M Fig og 25.5 Spch rcognition: Sounds distortion o writing. Tgging: Word squnc distortion o tg squnc Trnsltion: Sourc lngug distortion o trgt lngug. Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

9 Sprting th odls Strting point: Ê = rg x F EE E Th odls W cn build nd trin two sprt odls: Th lngug odl: E Th trnsltion odl: F E Dcoding ust us both odls siultnously Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

10 Lngug odl Gol Estit th probbility E = 2... n o th string o words 2... n n-gr odl 2... n = n 2... n n n n = i+ i i= Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

11 Conts: Uss th incorrct Mrkov-ssuption Lst slid shows th bigr odl. Could ltrntivly us trigr qudgr... Trigr: 2... n = n i= i+ i i For ll n-grs : spcil sybols or strt nd nd: Wht is th probbility o bing th irst word o sntnc? Wht is th probbility o bing th lst word o sntnc? Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

12 Th trnsltion odl Svrl ltrntivs: Word bsd In prticulr th IBM-odls: hrs bsd rtr stition otn don on top o word-bsd odl. Syntx bsd Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

13 Word-bsd odls Suppos Sourc nd trgt sntnc lwys th s lngth Word-ordr is prsrvd. A on-to-on corrspondnc btwn words Th trnsltion would b lik HMM-tgging Trnsltion Tgging sourc lngug word word trgt lngug word tg n-grs or trg. lng. n-grs o tgs sourc sntnc sntnc to b tggd word trnsltion probs. probbility or word givn tg S sipliid SMT xpl on slids ro irst MT lctur. Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

14 Word-bsd trnsltion odls But trnsltion rordrs dlts dds gos ny-to-on on-to-ny nd ny-to-ny. W cnnot pply HMM dirctly Two prts to word-bsd trnsltion Wht is th probbility tht sourc word is trnsltd s trgt word b? 2 Alignnt: Which words in th trgt lngug sntnc is th trnsltion o which words in th sourc sntnc? J& M Figur Jn Tor Lønning Sttisticl Mchin Trnsltion - SMT

15 5

16 Alignnt 6 Lngth o English string: k =7 Lngth o orign string: =9 An lignnt is vctor o lngth ch ntry nubr btwn 0 nd k Th xpl: < 2 9 > = < >

17 Alignnt 7 Artiicil rstrictions: Svrl orign words y b lignd with th s E word A orign word cnnot b lignd to or thn on E word

18 IBM Modl Considr ll possibl lignnts : For ch lignnt us th gnrtiv odl: Sipliy th odl k ssuptions 8

19 9 Figur 25.23

20 Th gnrtiv odl: Choos th lngth o th orign string Which E word trnslts to th irst F word Wht is th trnsltion o this word? Which E word trnslts to th -th F word givn th choics so r Wht is th trnsltion o this word givn th choics so r 0

21 Assuptions pproxitions is constnt indpndnt o nd E ll lignnts th s probbility dds to th word trnsltion probbility only dpnds on sourc word k t

22 IBM odl Sipliis to is norlistion ctor Forul 4.7 in th SMT book Th book gos not t k t k 2

23 rtr stition 3 I th trining corpus ws lignd th odl could b lrnd by counting: t C C I w hd known th trnsltion probbilitis w could hv ound th ost probbl lignnt. W nithr know word probbilitis nor lignnt: Chickn nd gg probl EM-lgorith: w y lrn th two siultnously

24 Trining th id 4. Fro th trnsltion probbilitis w y stit lignnt probbilitis W do not choos only th bst lignnt 2. Fro lignnt probbilitis w y rclcult trnsltion probbilitis By ltrnting btwn nd 2 th nubrs convrg towrds bttr rsults For IBM Modl it y b provd tht thy convrg towrds globl optiu

25 5

26 6

27 7

28 8

29 9

30 20

31 Two wys to dscrib th lgorith 2 Intuitiv rocd. Trnsltion prob. Alignnt prob 2. Trnsltion prob 2. Alignnt prob 3. Trnsltion prob Etc J&M sc xpl Intrctbl in prctic Eicint Sidstp lignnt probs:. Trnsltion prob 2. Trnsltion prob 3. Trnsltion prob Etc K:SMT sc xpl How it gts iplntd

32 Trining th intuitiv pproch 22. Initliz th prtr vlus t or pirs o words nd. With no ino initliz th uniorly: Ech word in th orign lngug is n qully likly trnsltion o th word. 2. For ch pir o sntncs in th corpus us t to clcult th probbilitis to ll possibl lignnts o th two sntncs. Clld th xpcttion stp pply odl to dt

33 Trining th intuitiv pproch Collct rctionl counts tc : «How ny tis is trnsltd s». First clcult this c ; or ch sntnc whr w count: how ny tis is lignd to by ch lignnt wighd by th probbility o th lignnt. 2. Thn dd ovr ll sntncs to gt tc c ;

34 Trining th intuitiv pproch Clcult th nw trnsltion probbilitis t = tc tc Errors in orul 4.4 in K:SMT whr vris ovr ll orign words Clld th xiiztion stp stit odl ro counts 5. Rpt ro 2 s long s you lik

35 Assign probbilitis to lignnts Gol: coput Sinc w hv W know t k 25

36 Expl th intuitiv wy 26 Corpus : Dog brkd : Hund bt 2 : Dog bit dog 2 : Hund bt hund 3 English words: dog bit brkd 3 orign words: hund bt bt

37 Stp initiliztion 27 thunddog = /3 tbtdog = /3 tbtdog = /3 thundbit = /3 tbtbit = /3 tbtbit = /3 thundbrkd = /3 tbtbrkd = /3 tbtbrkd = /3 thund0 = /3 tbt0 = /3 tbt0 = /3 Unior Obsrv tht w includ th lst lin sinc n - word y b lignd to 0.

38 Stp 2: Alignnt probbilitis 28 : Dog brkd : Hund bt 2 : Dog bit dog 2 : Hund bt hund Sntnc pir : 9 possibl lignnts: <00> <0> <02> <0> <> <2><20><2> <22> Ech qully probbl: /9 cll this :.g. <0>=/27 Sntnc pir 2: 64 possibl lignnts: <000> <00> <333> Ech qully probbl: /64 cll this 2. Or th hrd wy nxt slid

39 Stp 2: Th hrd wy Sntnc pir 2: 64 possibl lignnts: <000> <00> <333> Ech trnsltion probbility: /27 2 : Dog bit dog 2 : Hund bt hund hund t bit bt t dog hund t t t t t t k 64 64*27 64* 64*

40 Stp 3.: Collct rctionl counts 30 Clcult c ; or ch sntnc : Expl: =hund = dog : Thr r 3 lignnts tht connct th: <0> <> <2> : Dog brkd : Hund bt chunddog; = <0>+ <>+ <2>=3*/9 = /3 chunddog; = /3 cbtdog; = /3 chundbrkd; = /3 cbtbrkd; = /3 chund0; = /3 cbt0; = /3

41 Stp 3.: Collct rc. counts ctd : =bt = bit 2 : Dog bit dog 2 : Hund bt hund 6 lignnts connct th: <x2z> or xz in {023} cbtbit; 2 2 = 6/64 = /4 =bt = dog ll lignnts <xz> nd <x3z> or xz in {023} cbtdog; 2 2 = 2*6/64 = /2 chunddog; 2 2 = cbtdog; 2 2 = /2 chundbit; 2 2 = /2 cbtbit; 2 2 = /4 chund0; 2 2 = /2 cbt0; 2 2 = /4

42 Stp 3.2: Totl counts 32 tc c ; tchunddog = +/3 tcbtdog = /2 tcbtdog = /3 tc*dog=4/3+/2+/3 =3/6 tchundbit = ½ tcbtbit = ¼ tcbtbit = 0 tc*bit=3/4 tchundbrkd = /3 tcbtbrkd = 0 tcbtbrkd = /3 tc*brkd =2/3 tchund0 = ½+/3 tcbt0 = /4 tcbt0 = /3 tc*0=7/2

43 Stp 4: nw trns. probbilitis 33 t = tc tc t xct dcil 0 hund 5/6/7/2 0/ bt /4/7/2 3/ bt /3/7/2 4/ dog hund 4/3/3/6 8/ dog bt /2/3/6 3/ dog bt /3/3/6 2/ bit hund /2/3/4 2/ bit bt /4/3/4 / brkd hund /3/2/3 /2 0.5 brkd bt /3/2/3 /2 0.5

44 Rpt: Stp 2 sntnc 34 9 dirnt lignnts = c = : Dog brkd : Hund bt = / <00> = thund0*tbt0= 0/7*3/7= <0>= thund0*tbtdog= 0/7*2/3= <02>= thund0*tbtbrkd= 0/7*/2= <0> = thunddog*tbt0= 8/3*3/7= <> = thunddog*tbtdog= 8/3*2/3= <2> = thunddog*tbtbrkd= 8/3*/2= <20> = thundbrkd*tbt0= /2*3/7= <2>= thundbrkd*tbtdog= /2*2/3= <22>= thundbrkd*tbtbrkd= /2*/2= Su o s

45 Rpt: Stp 2 sntnc dirnt lignnts Ho work til nxt wk! How ny lignnts i th sntncs r 0 words long? Tht s why w nd srtr wy. To b continud