Aspects of Tree-Based Statistical Machine Translation

Transcription

1 Aspects of Tree-Based Statistical Machine Translation Marcello Federico Human Language Technology FBK 2014

2 Outline Tree-based translation models: Synchronous context free grammars Hierarchical phrase-based model Decoding with SCFGs: Translation as Parsing DP-based chart decoding Integration of language model scores Credits: adapted from slides by Gabriele Musillo. 2 / 31

3 Tree-Based Translation Models Levels of Representation in Machine Translation: π π π π σ σ source target π σ: tree-to-string σ π: string-to-tree π π: tree-to-tree? Appropriate Levels of Representation? 3 / 31

4 Tree Structures NNP Pierre S NP NNP MD Vinken will VP VP VB NP join Det the NN board Syntactic Structures: rooted ordered trees internal nodes labeled with syntactic categories leaf nodes labeled with words linear and hierarchical relations between nodes 4 / 31

5 Tree-to-Tree Translation Models NNP Pierre NP Pierre NNP NNP Vinken NP S Vinken NNP MD wird S will MD VP NP Det dem join VB VP VP NN Vorstand the Det NP VP board NN VB beitreten syntactic generalizations over pairs of languages: isomorphic trees syntactically informed unbounded reordering formalized as derivations in synchronous grammars? Adequacy of Isomorphism Assumption? 5 / 31

6 Context-Free Grammars CFG (Chomsky, 1956): formal model of languages more expressive than Finite State Automata and Regular Expressions first used in linguistics to describe embedded and recursive structures CFG Rules: left-hand side nonterminal symbol right-hand side string of nonterminal or terminal symbols distinguished start nonterminal symbol { S 0S1 S ɛ S rewrites as 0S1 S rewrites as ɛ 6 / 31

7 CFG Examples G 1 : G 3 : R = {S NP VP, NP N DET N N PP, VP V NP V NP PP, PP P NP, DET the a, N Alice Bob trumpet, V chased, P with} R = {NP NP CONJ NP NP PP DET N, PP P NP, P of, DET the two three, N mother pianists singers, CONJ and}? derivations of the mother of three pianists and two singers? derivations of Alice chased Bob with the trumpet same parse tree can be derived in different ways ( order of rules) same sentence can have different parse trees ( choice of rules) 7 / 31

8 Transduction Grammars aka Synchronous Grammars TG (Lewis and Stearns, 1968; Aho and Ullman, 1969): two or more strings derived simultaneously more powerful than FSTs used in NLP to model alignments, unbounded reordering, and mappings from surface forms to logical forms E E [1] + E [2] / + E [1] E [2] Synchronous Rules: left-hand side nonterminal symbol associated with source and target right-hand sides bijection [] mapping nonterminals in source and target of right-hand sides infix to Polish notation E E [1] E [2] / E [1] E [2] E n / n n N 8 / 31

9 Synchronous CFG 1-to-1 correspondence between nodes isomorphic derivation trees uniquely determined word alignment 9 / 31

10 Hierarchical Phrase-Based Models HPBM (Chiang, 2007): formalized as SCFG first tree-to-tree approach to perform better than phrase-based systems in large-scale evaluations discontinuous phrases, i.e. phrases with gaps long-range reordering rules no syntactic rules: only two non-terminal symbols Example Chinese-English: original, transliteration, glosses, and translation 10 / 31

11 HPBM: Motivations Typical Phrase-Based Chinese-English Translation: Chinese VPs follow PPs / English VPs precede PPs yu X 1 you X 2 / have X 2 with X 1 Chinese NPs follow RCs / English NPs precede RCs X 1 de X 2 / the X 2 that X 1 translation of zhiyi construct in English word order X 1 zhiyi / one of X 1 11 / 31

12 HPBM: Example Rules S X 1 / X 1 (1) S S 1 X 2 / S 1 X 2 (2) X yu X 1 you X 2 / have X 2 with X 1 (3) X X 1 de X 2 / the X 2 that X 1 (4) X X 1 zhiyi / one of X 1 (5) X Aozhou / Australia (6) X Beihan / N. Korea (7) X she / is (8) X bangjiao / dipl.rels. (9) X shaoshu guojia / few countries (10) 12 / 31

13 Summary Synchronous Context-Free Grammars: Context-Free Grammars HPB recursive reordering model Next topics: Decoding SCFGs: Translation as Parsing DP-based chart decoding Integration of language model scores 13 / 31

14 Synchronous Context-Free Grammars SCFGs: CFGs in two dimensions synchronous derivation of isomorphic a trees unbounded reordering preserving hierarchy a excluding leafs VB PRP 1 VB1 2 VB2 3 / PRP 1 VB2 3 VB1 2 VB2 VB 1 TO 2 / TO 2 VB 1 ga TO TO 1 NN 2 / NN 2 TO 1 PRP he / kare ha VB listening / daisuki desu VB 1 PRP 2 he VB 1 VB1 3 VB2 4 adores VB 5 TO 6 listening TO 7 NN 8 PRP 2 kare ha VB2 4 TO 6 VB 5 NN 8 TO 7 kiku no ongaku wo ga VB1 3 daisuki desu to music 14 / 31

15 Weighted SCFGs rules A α / β associated with positive weights w A α/β derivation trees π = π 1, π 2 weighted as W(π) = A α/β G w c(a α/β;π) A α/β probabilistic SCFGs if the following conditions hold w A α/β [0, 1] and α,β W A α/β = 1 notice: SCFGs might well include rules of type A α/β 1... A α/β k 15 / 31

16 MAP Translation Problem Maximum A Posterior Translation: e = argmax e = argmax e p(e f ) π Π(f,e) p(e, π f ) Π(f, e) is the set of synchronous derivation trees yielding f, e Exact MAP decoding is NP-hard (Simaan, 1996; Satta and Peserico, 2005) 16 / 31

17 Viterbi Approximation Tractable Approximate Decoding: e = argmax e argmax e π Π(f,e) max π Π(f,e) = E(argmax p(π)) π Π(f ) p(e, π f ) p(e, π f ) Π(f ) is the set of synchronous derivations yielding f E(π) is the target string resulting from the synchronous derivation π 17 / 31

18 Translation as Parsing Parsing Solution: π = argmax p(π) π Π(f ) 1. compute the most probable derivation tree that generates f using the source dimension of the WSCFG 2. build the translation string e by applying the target dimension of the rules used in the most probable derivation most probable derivation computed in O(n 3 ) using dynamic programming algorithms for parsing weighted CFGs transfer of decoding algorithms developed for CFG to SMT 18 / 31

19 Weighted CFGs in Chomsky Normal Form WCFGs: rules A α associated with positive weights w A α derivation trees π weighted as W(π) = A α G w c(a α;π) A α probabilistic CFGs if the following conditions hold w A α [0, 1] and α w A α = 1 WCFGs in CNF: rules in CFGs in Chomsky Normal Form: A BC or A a equivalence between WCFGs and WCFGs in CNF no analogous equivalence holds for weighted SCFGs 19 / 31

20 Weighted CKY Parsing Dynamic Programming: recursive division of problems into subproblems optimal solutions compose optimal sub-solutions (Bellman s Principle) tabulation of subproblems and their solutions CKY Parsing: subproblems: parsing substrings of the input string u 1... u n bottom up algorithm starting with derivation of terminals solutions to subproblems tabulated using a chart O(n 3 G ) time complexity 20 / 31

21 Weighted CKY Parsing Q(A, i, k) = max {w A B C Q(B, i, j) Q(C, j, k)} B,C,i<j<k S A B C u i+1,j u j+1,k 21 / 31

22 Parsing SCFG and Language Modelling Viterbi Decoding of WSCFGs: focus on most probable derivation of source (ignoring different target sides associated with the same source side) derivation weights do not include language models scores? HOW TO EFFICIENTLY COMPUTE TARGET LANGUAGE MODEL SCORES FOR POSSIBLE DERIVATIONS? Approaches: 1. online: integrate target m-gram LM scores into dynamic programming parsing 2. cube pruning (Huang and Chiang, 2007): rescore k-best sub-translations at each node of the parse forest 22 / 31

23 Online Translation Online Translation: parsing of the source string and building of the corresponding subtranslations in parallel PP 1,3 : (w 1, t 1 ) VP 3,6 : (w 2, t 2 ) VP 1,6 : (w w 1 w 2, t 2 t 1 ) w 1, w 2 : weights of the two antecedents w: weight of the synchronous rule t 1, t 2 : translations 23 / 31

24 LM Online Integration (Wu, 1996) PP with Sharon 1,3 : (w 1, t 1 ) VP held talk 3,6 : (w 2, t 2 ) VP held Sharon 1,6 : (w w 1 w 2 p LM (with talk), t 2 t 1 ) Integrate LM information in the state: Q(A, i, j, pfx, sfx) O(n 3 E 4(m 1) ): recombine 4 prefixes/suffixes of (m-1) words 24 / 31

25 Cube Pruning (Huang and Chiang, 2007) Beam Search: at each step in the derivation, keep at most k items integrating target subtranslations in a beam enumerate all possible combinations of LM items extract the k-best combinations Cube Pruning: get k-best LM items without computing all combinations approximate search: in practice negligible search errors 25 / 31

26 Cube Pruning Heuristic Assumption: margin scores are -log-probs of the left/right spans best adjacent items lie towards the upper-left corner part of the grid can be pruned without computing its cells 26 / 31

27 Cube Pruning: Example 27 / 31

28 Cube Pruning: Example 28 / 31

29 Cube Pruning: Example 29 / 31

30 Cube Pruning: Example 30 / 31

31 Summary Translation As Parsing: Viterbi Approximation Weighted CKY Parsing Online LM Integration and Cube Pruning 31 / 31