Machine Translation Classical and Statistical Approaches

Transcription

1 Week : Overvie Machie Traslatio Classical ad Statistical Approaches Sessio 6: Statistical MT Itro ( Joas Kuh Uiversität des Saarlades, Saarbrücke The Uiversity of Texas at Austi joask@coli.ui-sb.de Data-drive, statistical approaches to MT The oisy chael model [Bro et al. 990, Kight 999] Laguage modelig Traslatio modelig Word aligmet Phrase aligmet [Koeh et al. 003] Decodig [Koeh 994] Lab exercise: buildig a phrase-based statistical MT system from parallel texts take from the Iteret Evaluatio methods Other uses of ord aligmets [Yarosky et al. 00] DGfS/CL Fall School 005, Ruhr-Uiversität Bochum, September 9-30, 005 Joas Kuh: MT Sessios 6/7: Statistical MT Itro Traslatio ithout uderstadig? Ackoledgemets: Some slides are borroed from Kevi Kight, Uiversity of Souther Califoria, from Coli Cherry, Alberta (see ad from Leila Kosseim ( Traslatio ithout uderstadig Very brief itroductio to probabilities The oisy chael model for traslatio Laguage modelig Traslatio modelig Decodig Traslatio is easy for (biligual people Process: Read the text i Frech Uderstad it Write it do i Eglish Joas Kuh: MT 3 Joas Kuh: MT 4

2 Traslatio ithout uderstadig? Traslatio is easy for (biligual people Process: Read the text i Frech Uderstad it Write it do i Eglish Hard for computers The huma process is ivisible, itagible Oe approach: Rule-based MT Compare eek Problems: Buildig a broad-coverage system is a eormous egieerig challege Addig e laguages/text domais is very costly May disambiguatio decisios caot be made ithout orld koledge/cotextual koledge Joas Kuh: MT 5 Joas Kuh: MT 6 Alterative Approach: Statistical MT Data-Drive Machie Traslatio Go back to Warre Weaver s idea of usig statistical techiques fid the most probable traslatio of a give setece We at to traslate from Frech to Eglish Task: give a Frech setece, hat is the most probable Eglish traslatio? Notatio: Fid E* arg max E E F Ma, this is so borig. Hmm, every time he sees baco, he either types bak or bech but if he sees baco de, he alays types bak, ever bech Traslated documets Joas Kuh: MT 7 Slide from Kevi Kight Joas Kuh: MT 8

3 Cetauri/Arctura [Kight, 997] Exercise: traslate this to Arctura: farok crrrok hihok yorok clok katok ok-yurp Recet Progress i Statistical MT a. ok-voo ororok sprok. b. at-voo bichat dat. a. ok-drubel ok-voo aok plok sprok. 7a. lalok farok ororok lalok sprok izok eemok. 7b. at jjat bichat at dat vat eeat. 8a. lalok brok aok plok ok isistet Wedesday may recurred her trips to Libya tomorro for flyig Cairo 6-4 ( AFP - a official aouced today i the Egyptia lies compay for flyig Tuesday is a compay " isistet for flyig " may resumed a cosideratio of a day Wedesday tomorro her trips to Libya of Security Coucil decisio trace iteratioal the imposed ba commet. Ad said the official " the istitutio set a speech to Miistry of Foreig Affairs of liftig o Libya air, a situatio her receivig replyig are so a trip ill pull to Libya a morig Wedesday ". Egyptair Has Tomorro to Resume Its Flights to Libya Cairo 4-6 (AFP - said a official at the Egyptia Aviatio Compay today that the compay egyptair may resume as of tomorro, Wedesday its flights to Libya after the Iteratioal Security Coucil resolutio to the suspesio of the embargo imposed o Libya. " The official said that the compay had set a letter to the Miistry of Foreig Affairs, iformatio o the liftig of the air embargo o Libya, here it had received a respose, the first take off a trip to Libya o Wedesday morig ". b. at-drubel at-voo pippat rrat dat. 3a. erok sprok izok hihok ghirok. 3b. totat dat arrat vat hilat. 4a. ok-voo aok drok brok jok. 4b. at-voo krat pippat sat lat. 5a. iok farok izok stok. 5b. totat jjat quat cat. 6a. lalok sprok izok jok stok. 6b. at dat krat quat cat. 8b. iat lat pippat rrat at. 9a. iok ok izok katok ok-yurp. 9b. totat at quat oloat at-yurp. 0a. lalok mok ok yorok ghirok clok. 0b. at at gat mat bat hilat. a. lalok ok crrrok hihok yorok zazaok. b. at at arrat mat zazaat. a. lalok rarok ok izok hihok mok. b. at at forat arrat vat gat. 9 Slide from C. Waye, DARPA Joas Kuh: MT 0 Very brief itro to probabilities Usig commo sese, e are pretty good at dealig ith the likelihood of (radom evets Probability fuctios assig a value betee 0 ad to the occurrece of a particular outcome of a radom evet Example: rollig a die / We eed some termiology ad otatio Pop star example Assume you are a photo reporter ad at to take a exclusive picture of a iteratioal pop star ho s o tour i Germay There are rumors that certai cocerts ill get cacelled You at to guess hat route the pop star ill take through Germay Each route has a certai probability Wait at a locatio alog the route ith the highest probability to take the picture Joas Kuh: MT Joas Kuh: MT

4 Probabilities Calculatios ith probabilities Simple probability (Prior probability A You call up the tour maager ad ask hether the cocert i Berli ill be cacelled or ot With 60% probability the cocert ill take place CiB CiB Coditioal probability (Posterior probability A B If the pop star has a cocert i Berli ho likely is it that she ill visit the Reichstagsgebäude? Oe out of four pop stars ho gives a cocert i Berli also visits Reichstagsgebäude Oly 0% of the pop stars ho do t give a cocert i Berli visit the Reichstagsgebäude Rtg CiB 0.5 Rtg CiB Joas Kuh: MT Ho likely is it that the pop star ill sho up at the Reichstag [ What is Rtg ]? All e have are coditioal probabilities for the pop star visitig the Reichstag, so e have to cosider both optios for the precoditio CiB CiB Rtg CiB Joit probability A,B Rtg CiB 0. Rtg, CiB CiB Rtg CiB Rtg, CiB CiB Rtg CiB Sice CiB ad CiB cover the full space of probabilities e get: Rtg Rtg, CiB + Rtg, CiB What s the use of a exact value like this? Compariso ith alterative optios, e.g., FRA_Airport 0.5 Joas Kuh: MT Calculatios ith probabilities Bayes La We just exploited the fact that joit probabilities [i.e., A,B] ca be calculated by multiplyig the prior probability for oe evet ith the coditioal probability for the other evet, give the first evet This is called the chai rule We ca go either ay (because A,B B,A: A,B A B A or A,B B A B So: B A B A B A Divide both sides of the equatio by B : A B B A A B Joas Kuh: MT 5 B A A A B B This is called Bayes La Importace: Ofte, traiig [i.e., statistical parameter estimatio from a sample of radom experimets] for oe of the to coditioal probabilities ca be doe much more reliably tha for the other oe Joas Kuh: MT 6

5 Bayes La Crime Scee Aalogy B A A A B B Whe e are oly lookig for the most likely outcome A* for a evet, give a fixed evet B, the deomiator does t play a role: A* arg max arg max arg max A A A A B B A A B B A A B is a crime scee. A is a perso ho may have committed the crime A B - look at the scee - ho did it? A - ho had a motive? (Profiler B A - could they have doe it? (trasportatio, access to eapos, alibi Some people might have great motives, but o meas - you eed both! Joas Kuh: MT 7 Joas Kuh: MT 8 Back to traslatio Why Bayes rule at all? We at to traslate from Frech to Eglish Task: give a Frech setece, hat is the most probable Eglish traslatio? Notatio: Fid E* arg max E E F With Bayes la e ca search the E that maximizes F E E Fid the Eglish strig E for hich the product of E [laguage model probability] times F E [traslatio model probability E F] is maximal Why ot model E F directly? F E E decompositio allos us to be sloppy E orries about good Eglish: Fluecy F E orries about Frech that matches Eglish: Faithfuless The to ca be traied idepedetly Joas Kuh: MT 9 Joas Kuh: MT 0

6 O voit Jo à la télévisio Fluecy vs. Faithfuless Jo appeared i TV. Appeared o Jo TV. good Eglish? E good match to Frech? F E Note that eve theoretically, it is sometimes impossible to have a traslatio that is maximally faithful to the source laguage, but also fluet i the target laguage I Jo appeared TV. Jo is happy today. Jo appeared o TV. TV appeared o Jo. TV i Jo appeared. Example Japaese: fukaku hasei shite orimasu Fluet traslatio: e apologize Faithful traslatio: e are deeply reflectig (o our past behaviour, ad hat e did rog, ad ho to avoid the problem ext time Jo as ot happy. Table borroed from Jaso Eiser Joas Kuh: MT Joas Kuh: MT The Noisy Chael Model Statistical MT is based o the oisy chael model Developed by Shao to model commuicatio (e.g., over a phoe lie The Noisy Chael Model Noisy chael model i SMT (ex. F E: Assume that the true text is i Eglish But he it as trasmitted over the oisy chael, it someho got corrupted ad came out i Frech i.e. the oisy chael has deformed/corrupted the origial Eglish iput ito Frech So really Frech is a form of oisy Eglish The task is to recover the origial Eglish setece (or to decode the Frech ito Eglish Joas Kuh: MT 3 Joas Kuh: MT 4

7 We eed three thigs (for FE. A Laguage Model of Eglish: E Measures fluecy Probability of a Eglish setece ~ Provides a set of fluet seteces to test for potetial traslatio. A Traslatio Model: F E Measures faithfuless Probability of a (Frech, Eglish pair (give Eglish setece ~Tests if a give fluet setece is a traslatio 3. A Decoder: arg max A effective ad efficiet search techique to fid E* The search space is ifiite ad rather ustructured, so heuristic search has to be applied Where ill e get E? Laguage modelig is a commo task i Natural Laguage Processig Applicatio cotexts (besides MT: Speech recogitio Had-ritig recogitio Augmetative commuicatio systems for the disabled Cotext-sesitive spellig error correctio (see example o ext slide Itroductio i chapter 6 of Jurafsky, D. ad J. H. Marti (000: Speech ad laguage processig: A Itroductio to Natural Laguage Processig, Computatioal Liguistics, ad Speech Recogitio, Upper Saddle River, NJ: Pretice-Hall. Joas Kuh: MT 5 Joas Kuh: MT 6 N-gram laguage models (Quick itro Give a sequece of ords, hat ill be the ext ord? Hard to guess but if e do t demad extremely high accuracy, it is ot that hard Probability of a sequece of ords To closely related problems: Guessig the ext ord Computig the probability of a sequece of ords I d like to make a collect call telephoe iteratioal Joas Kuh: MT 7 Joas Kuh: MT 8

8 Coutig ords i corpora To estimate probabilities, e eed to cout frequecies What do people cout? Word forms Lemmas The type/toke distictio Number of (ord form types: distict ords i a corpus (i.e., the size of the vocabulary Number of (ord form tokes: total umber of ruig ords Coutig ords i corpora Sitchboard corpus (spoke Eglish.4 millio ord form tokes c. 0,000 ord form types Shakespeare s complete ords 884,647 ord form tokes 9,066 ord form types Bro corpus millio ord form tokes 6,805 ord form types (37,85 lemma types Joas Kuh: MT 9 Joas Kuh: MT 30 Estimatig ord probabilities Ho probable is a Eglish ord (form as the ext ord i a sequece? Simplest model: every ord has the same probability of occurrig Assume vocabulary size 00,000 Sigle ord: the probability of fidig is 00,000 Word i a sequece, assumig coditioal idepedece from the cotext... 00,000 Estimatig ord probabilities Sequece of to ords, still assumig that each ord form is equally likely that are coditioally idepedet from each other that are coditioally idepedet from the cotext, 00,000 00,000 0,000,000, Joas Kuh: MT 3 Joas Kuh: MT 3

9 A slightly more complex model A slightly more complex model Still assume that ay ord ca follo ay other ord Take ito accout that differet ord forms occur ith differet frequecies the occurs 69,97 times i the,000,000 tokes of the Bro corpus rabbit occurs times i the Bro corpus Austi occurs 0 times liguist occurs 3 times Joas Kuh: MT 33 Estimatig probabilities based o relative frequecy Sample:,000,000 trials of producig a radom Eglish ord (N,000,000 Relative frequecy of outcome u: f f f f u i the i rabbit i Austi u N i the N i rabbit N i Austi N 69,97,000,000,000, ,000, Joas Kuh: MT 34 Coditioal probability of a ord Relative frequecies are ot a good model for the probability of ords i a give cotext Just the, the hite the.07 rabbit.0000 We should take the previous ords that have occurred ito accout We ill get: rabbit hite > rabbit rabbit, hite P ( rabbit hite hite Joas Kuh: MT 35 Probability of a strig of ords Usig the chai rule of probability: But ho ca e estimate such probabilities? [ ] P (,, 3...,, k k k If e ated to cout the frequecy of every ord appearig after a log sequece of other ords, e ould eed a far too large corpus as a sample Joas Kuh: MT

10 Chai of probabilities Probability of a strig of ords USA Berli/Tegel Frakfurt/M. Flughafe Berli City Köl Hamburg City Potsdam Dresde Approximate the probability We have to form equivalece classes over ord cotexts, so e get a larger sample from hich e estimate probabilities Müche Flughafe Müche City Düsseldorf Stuttgart USA, Ffm, Köl, Berli, Potsdam USA Ffm USA Köl USA, Ffm Berli USA, Ffm, Köl Potsdam USA, Ffm, Köl, Berli Joas Kuh: MT 37 Simple approximatio: look oly at oe precedig ord Joas Kuh: MT 38 Bigram model Approximate rabbit Just the other day I sa a by rabbit a Markov assumptio: predictig a future evet based o a limited ido of past evets Bigrams: first-order Markov model (lookig back oe toke ito the past N-gram models Bigram model: first-order Markov model lookig back oe toke Trigram model: secod-order Markov model lookig back to tokes P N-gram model: N-th order Markov model lookig back N- tokes [... ] N + N + N + ( Joas Kuh: MT 39 Joas Kuh: MT 40

11 Bigram approximatio of strig prob. Bigram laguage model example k k k Simplifyig assumptio: k k k... Joas Kuh: MT 4 3 Resultig equatio (bigram laguage model: Berkeley Restaurat Project (corpus of c. 0,000 seteces Most likely ords to follo eat eat o.6 eat some.06 eat luch.06 eat dier.05 eat at.04 eat a.04 eat Idia.04 eat today.03 eat Thai.03 eat breakfast.03 eat i.0 eat Chiese.0 eat Mexica.0 eat tomorro.0 eat dessert.007 eat British.00 Joas Kuh: MT 4 Bigram probabilities Computig the setece probability <s> I.5 I at.3 at to.65 <s> I d.06 I ould.9 at a.05 <s> Tell.04 I do t.08 at some.04 <s> I m.0 I have.04 at thai.0 to eat.6 British food.60 to have.4 British restaurat.5 to sped.09 British cuisie.0 to be.0 British luch.0 I at to eat British food I < s > at I to at eat to British eat food British Joas Kuh: MT 43 Joas Kuh: MT 44

12 Traiig N-gram models Coutig ad ormalizig Cout occurreces of a bigram (say, eat luch Divide by total cout of bigrams sharig the first ord (i.e., eat for some Joas Kuh: MT 45 Traiig N-gram models Geeral case of N-gram parameter estimatio N + N + N + Relative frequecy Example of Maximum Likelihood Estimatio (MLE techique Joas Kuh: MT 46 Relative frequecy: example Bigram couts from Berkeley Restaurat Project Relative frequecy: example Uigram couts from corpus I at to eat Chiese food luch I at to eat Chiese food luch I 3437 at 5 to 356 eat 938 Chiese 3 food 506 luch 459 Joas Kuh: MT 47 Joas Kuh: MT 48

13 Relative frequecy: example Bigram probabilities (after ormalizig, i.e., through dividig by uigram couts: I at to eat Chiese food luch I at to eat Chiese food luch Laguage modelig: ed of itro Plai relative frequecy estimatio is problematic Uobserved N-grams are assiged zero probability Problematic ith loer-frequecy ords Smoothig techiques reserve some probability mass for uobserved evets Build your o laguage model: CMU Statistical Laguage Modelig Toolkit Joas Kuh: MT 49 Joas Kuh: MT 50 Cetauri/Arctura [Kight, 997] Exercise: traslate this to Arctura: a. ok-voo ororok sprok. b. at-voo bichat dat. a. ok-drubel ok-voo aok plok sprok. b. at-drubel at-voo pippat rrat dat. 3a. erok sprok izok hihok ghirok. 3b. totat dat arrat vat hilat. 4a. ok-voo aok drok brok jok. 4b. at-voo krat pippat sat lat. 5a. iok farok izok stok. 5b. totat jjat quat cat. 6a. lalok sprok izok jok stok. 6b. at dat krat quat cat. farok crrrok hihok yorok clok katok ok-yurp 7a. lalok farok ororok lalok sprok izok eemok. 7b. at jjat bichat at dat vat eeat. 8a. lalok brok aok plok ok. 8b. iat lat pippat rrat at. 9a. iok ok izok katok ok-yurp. 9b. totat at quat oloat at-yurp. 0a. lalok mok ok yorok ghirok clok. 0b. at at gat mat bat hilat. a. lalok ok crrrok hihok yorok zazaok. b. at at arrat mat zazaat. a. lalok rarok ok izok hihok mok. b. at at forat arrat vat gat. 5