Overview (Fall 2007) Machine Translation Part III. Roadmap for the Next Few Lectures. Phrase-Based Models. Learning phrases from alignments

Size: px

Start display at page:

Download "Overview (Fall 2007) Machine Translation Part III. Roadmap for the Next Few Lectures. Phrase-Based Models. Learning phrases from alignments"

Daisy Gibson
5 years ago
Views:

1 Overview Learning phrases from alignments (Fall 2007) Machine Translation Part III A phrase-based model Decoding in phrase-based models (Thanks to Philipp Koehn for giving me slides from his EACL 2006 tutorial) 1 3 Roadmap for Next Few Lectures Lecture 1 (last time): IBM Models 1 and 2 Lecture 2 (today): phrase-based models Lecture 3: Syntax in statistical machine translation Phrase-Based Models First stage in training a phrase-based model is extraction of a phrase-based (PB) lexicon A PB lexicon pairs strings in one language with strings in anor language, e.g., nach Kanada in Canada zur Konferenz to conference Morgen tomorrow fliege will fly

2 An Example (from tutorial by Koehn and Knight) A training example (Spanish/English sentence pair): Spanish: Maria no daba una bofetada a la bruja verde English: Mary not slap Some (not all) phrase pairs extracted from this example: (Maria Mary), (bruja ), (verde ), (no not), (no daba una bofetada not slap), (daba una bofetada a la slap ) We ll see how to do this using alignments from IBM models (e.g., from IBM model 2) Representation as Alignment Matrix Mary not slap (Note: bof = bofetada ) In IBM model 2, each foreign (Spanish) word is aligned to exactly one English word. The matrix shows se alignments. 5 7 Recap: IBM Model 2 IBM model 2 defines a distribution P (a, f e) where f is foreign (French) sentence, e is an English sentence, a is an alignment A useful by-product: once we ve trained model, for any (f, e) pair, we can calculate Finding Alignment Matrices Step 1: train IBM model 2 for P (f e), and come up with most likely alignment for each (e, f) pair Step 2: train IBM model 4 for P (e f) and come up with most likely alignment for each (e, f) pair We now have two alignments: take intersection of two alignments as a starting point a = arg max a P (a f, e) = arg max P (a, f e) a under model. a is most likely alignment 6 8

3 Alignment from P (f e) model: Mary not slap Alignment from P (e f) model: Mary not slap Heuristics for Growing Alignments Only explore alignment in union of P (f e) and P (e f) alignments Add one alignment point at a time Only add alignment points which align a word that currently has no alignment At first, restrict ourselves to alignment points that are neighbors (adjacent or diagonal) of current alignment points Later, consider or alignment points 9 11 Intersection of two alignments: Mary not slap The intersection of two alignments has been found to be a very reliable starting point The final alignment, created by taking intersection of two alignments, n adding new points using growing heuristics: Mary not slap Note that alignment is no longer many-to-one: potentially multiple Spanish words can be aligned to a single English word, and vice versa

4 Extracting Phrase Pairs from Alignment Matrix Mary not slap A phrase-pair consists of a sequence of English words, e, paired with a sequence of foreign words, f A phrase-pair (e, f) is consistent if re are no words in f aligned to words outside e, and re are no words in e aligned to words outside f e.g., (Mary not, Maria no) is consistent. (Mary, Maria no) is not consistent: no is aligned to not, which is not in string Mary We extract all consistent phrase pairs from training example. Koehn, EACL 2006 tutorial, pages for illustration. See An Example Phrase Translation Table An example from Koehn, EACL 2006 tutorial. (Note that we have P (e f) not P (f e) in this example.) Phrase Translations for den Vorschlag English P (e f) English P (e f) proposal suggestions s proposal proposed a proposal motion idea idea of this proposal proposal, proposal its proposal of proposal it proposals Probabilities for Phrase Pairs For any phrase pair (f, e) extracted from training data, we can calculate Count(f, e) P (f e) = Count(e) e.g., Count(daba una bofetada, slap) P (daba una bofetada slap) = Count(slap) Overview Learning phrases from alignments A phrase-based model Decoding in phrase-based models 14 16

5 Phrase-Based Systems: A Sketch Heute werden wir uber die Wiedereroffnung des Mont-Blanc- Tunnels diskutieren Phrase-Based Systems: A Sketch Heute werden wir uber die Wiedereroffnung des Mont-Blanc-Tunnels diskutieren Score = log P (Today START) Language model + log P (Heute Today) Phrase model + log P ( ) Distortion model Phrase-Based Systems: A Sketch Heute werden wir uber die Wiedereroffnung des Mont-Blanc- Tunnels diskutieren Score = log P (we shall be today) Language model Phrase-Based Systems: A Sketch Heute werden wir uber die Wiedereroffnung des Mont-Blanc- Tunnels diskutieren + log P (werden wir we will be) Phrase model + log P ( ) Distortion model 18 20

6 Phrase-Based Systems: A Sketch Heute werden wir uber die Wiedereroffnung des Mont-Blanc-Tunnels diskutieren Phrase-Based Systems: Formal Definitions We n have Cost(E, F ) = P (E) l P (f i e i )d(a i b i 1 ) i=1 P (E) is language model score for string defined by E P (f i e i ) is phrase-table probability for i th phrase pair d(a i b i 1 ) is some probability/penalty for distance between i th phrase and (i 1) th phrase. Usually, we define d(a i b i 1 ) = α a i b i 1 1 for some α < 1. Note that this is not a coherent probability model Phrase-Based Systems: Formal Definitions (following notation in Jurafsky and Martin, chapter 25) We d like to translate a French string f E is a sequence of l English phrases, e 1, e 2,..., e l. For example, e 1 = Mary, e 2 = not, e 3 = slap, e 4 =, e 5 = E defines a possible translation, in this case e 1 e 2... e 5 = Mary not slap. F is a sequence of l foreign phrases, f 1, f 2,..., f l. For example, f 1 = Maria, f 2 = no, f 3 = dio una bofetada, f 4 = a la, f 5 = bruja verde An Example Position English Mary not slap Spanish Maria no dio una bofetada a la bruja verde In this case, Cost(E, F ) = P L (Mary not slap ) P (Maria Mary) d(1) P (no not) d(1) P (dio una bofetada slap) d(1) P (a la ) d(1) P (bruja verde ) d(1) P L is score from a language model a i for i = 1... l is position of first word of f i in f. b i for i = 1... l is position of last word of f i in f

7 Anor Example Position English Mary not slap Spanish Maria no dio una bofetada a la verde bruje The original Spanish string was Maria no dio una bofetada a la bruje verde, so notice that last two phrase pairs involve reordering In this case, Cost(E, F ) = P L (Mary not slap ) P (Maria Mary) d(1) P (no not) d(1) P (dio una bofetada slap) d(1) P (a la ) d(1) P (verde ) d(2) P (bruja ) d(1) The Decoding Problem For a given foreign string f, decoding problem is to find arg max (E,F ) Cost(E, F ) where arg max is over all (E, F ) pairs that are consistent with f See Koehn tutorial, EACL 2006, slides See Jurafsky and Martin, Chapter 25, Figure See Jurafsky and Martin, Chapter 25, section Overview Learning phrases from alignments A phrase-based model Decoding in phrase-based models 26

COMS 4705, Fall Machine Translation Part III

COMS 4705, Fall Machine Translation Part III COMS 4705, Fall 2011 Machine Translation Part III 1 Roadmap for the Next Few Lectures Lecture 1 (last time): IBM Models 1 and 2 Lecture 2 (today): phrase-based models Lecture 3: Syntax in statistical machine