Improving Decoding Generalization for Tree-to-String Translation

Size: px

Start display at page:

Download "Improving Decoding Generalization for Tree-to-String Translation"

Clarence Dorsey
5 years ago
Views:

1 Improving Dcoding Gnralization for Tr-to-String Translation Jingbo Zhu Natural Languag Procssing Laboratory Northastrn Univrsity, Shnyang, China Tong Xiao Natural Languag Procssing Laboratory Northastrn Univrsity, Shnyang, China Abstract To addrss th pars rror issu for tr-tostring translation, this papr proposs a similarity-basd dcoding gnration (SDG) solution by rconstructing similar sourc pars trs for dcoding at th dcoding tim instad of taking multipl sourc pars trs as input for dcoding. Exprimnts on Chins-English translation dmonstratd that our approach can achiv a significant improvmnt ovr th standard mthod, and has littl impact on dcoding spd in practic. Our approach is vry asy to implmnt, and can b applid to othr paradigms such as tr-to-tr modls. 1 Introduction Among linguistically syntax-basd statistical machin translation (SMT) approachs, th tr-tostring modl (Huang t al. 2006; Liu t al. 2006) is th simplst and fastst, in which pars trs on sourc sid ar usd for grammar xtraction and dcoding. Formally, givn a sourc (.g., Chins) string c and its auto-parsd tr T 1-bst, th goal of typical tr-to-string SMT is to find a targt (.g., English) string * by th following quation as * = arg max Pr( c, T1 bst ) (1) whr Pr( c,t 1-bst ) is th probability that is th translation of th givn sourc string c and its T 1-bst. A typical tr-to-string dcodr aims to sarch for th bst drivation among all consistnt drivations that convrt sourc tr into a targt-languag string. W call this st of consistnt drivations th tr-to-string sarch spac. Each drivation in th sarch spac rspcts th sourc pars tr. Parsing rrors on sourc pars trs would caus ngativ ffcts on tr-to-string translation du to dcoding on incorrct sourc pars trs. To addrss th pars rror issu in tr-to-string translation, a natural solution is to us n-bst pars trs instad of 1-bst pars tr as input for dcoding, which can b xprssd by * = arg max Pr( c, T ) (2) n bst whr <T n-bst > dnots a st of n-bst pars trs of c producd by a stat-of-th-art syntactic parsr. A simpl altrnativ (Xiao t al. 2010) to gnrat <T n-bst > is to utiliz multipl parsrs, which can improv th divrsity among sourc pars trs in <T n-bst >. In this solution, th most rprsntativ work is th forst-basd translation mthod (Mi t al. 2008; Mi and Huang 2008; Zhang t al. 2009) in which a packd forst (forst for short) structur is usd to ffctivly rprsnt <T n-bst > for dcoding. Forst-basd approachs can incras th trto-string sarch spac for dcoding, but fac a nontrivial problm of high dcoding tim complxity in practic. In this papr, w propos a nw solution by rconstructing nw similar sourc pars trs for dcoding, rfrrd to as similarity-basd dcoding gnration (SDG), which is xprssd as * = arg max Pr( c, T1 bst ) (3) arg max Pr( c,{ T, T }) 1 bst whr <T sim > dnots a st of similar pars trs of T 1-bst that ar dynamically rconstructd at th d- sim 418 Procdings of th 49th Annual Mting of th Association for Computational Linguistics:shortpaprs, pags , Portland, Orgon, Jun 19-24, c 2011 Association for Computational Linguistics

2 coding tim. Roughly spaking, <T n-bst > is a subst of {T 1-bst, <T sim >}. Along this lin of thinking, Equation (2) can b considrd as a spcial cas of Equation (3). In our SDG solution, givn a sourc pars tr T 1-bst, th ky is how to gnrat its <T sim > at th dcoding tim. In practic, it is almost intractabl to dirctly rconstructing <T sim > in advanc as input for dcoding du to too high computation complxity. To addrss this crucial challng, this papr prsnts a simpl and ffctiv tchniqu basd on similarity-basd matching constraints to construct nw similar sourc pars trs for dcoding at th dcoding tim. Our SDG approach can xplicitly incras th tr-to-string sarch spac for dcoding without changing any grammar xtraction and pruning sttings, and has littl impact on dcoding spd in practic. 2 Tr-to-String Drivation W choos th tr-to-string paradigm in our study bcaus this is th simplst and fastst among syntax-basd modls, and has bn shown to b on of th stat-of-th-art syntax-basd modls. Typically, by using th GHKM algorithm (Gally t al. 2004), translation ruls ar larnd from word-alignd bilingual txts whos sourc sid has bn parsd by using a syntactic parsr. Each rul consists of a syntax tr in th sourc languag having som words (trminals) or variabls (nontrminals) at lavs, and squnc words or variabls in th targt languag. With th hlp of ths larnd translation ruls, th goal of tr-to-string dcoding is to sarch for th bst drivation that convrts th sourc tr into a targt-languag string. A drivation is a squnc of translation stps (i.., th us of translation ruls). Figur 1 shows an xampl drivation d that prforms translation ovr a Chins sourc pars tr, and how this procss works. In th first stp, w can apply rul r 1 at th root nod that matchs a subtr {IP [1] (NP [2] VP [3] )}. Th corrsponding targt sid {x 1 x 2 } mans to prsrv th top-lvl word-ordr in th translation, and rsults in two unfinishd subtrs with root labls NP [2] and VP [3], rspctivly. Th rul r 2 finishs th translation on th subtr of NP [2], in which th Chins word 中方 is translatd into an English string th Chins sid. Th rul r 3 is applid to prform translation on th subtr of VP [3], and rsults in an An xampl tr-to-string drivation d consisting of fiv translation ruls is givn as follows: r 1 : IP [1] (x 1 :NP [2] x 2 :VP [3] ) x 1 x 2 r 2 : NP [2] (NN ( 中方 )) th Chins sid r 3 : VP [3] (ADVP(AD( 高度 )) VP(VV( 评价 ) AS( 了 ) x 1 :NP [4] )) highly apprciatd x 1 r 4 : NP [4] (DP(DT( 这 ) CLP(M( 次 ))) x 1 :NP [5] ) this x 1 r 5 : NP [5] (NN( 会谈 )) talk Translation rsults: Th Chins sid highly apprciatd this talk. Figur 1. An xampl drivation prforms translation ovr th Chins pars tr T. unfinishd subtr of NP [4]. Similarly, ruls r 4 and r 5 ar squntially usd to finish th translation on th rmaining. This procss is a dpth-first sarch ovr th whol sourc tr, and visits vry nod only onc. 3 Dcoding Gnralization 3.1 Similarity-basd Matching Constraints In typical tr-to-string dcoding, an ordrd squnc of ruls can b rassmbld to form a drivation d whos sourc sid matchs th givn sourc pars tr T. Th sourc sid of ach rul in d should match on of subtrs of T, rfrrd to as matching constraint. Bfor discussing how to apply our similarity-basd matching constraints to rconstruct nw similar sourc pars trs for dcoding at th dcoding tim, w first dfin th similarity btwn two tr-to-string ruls. Dfinition 1 Givn two tr-to-string ruls t and u, w say that t and u ar similar such that thir sourc sids t s and u s hav th sam root labl and frontir nods, writtn as t u, othrwis not. 419

3 Figur 2: Two similar tr-to-string ruls. (a) rul r 3 usd by th xampl drivation d in Figur 1, and (b) a similar rul τ 3 of r 3. Hr w us an xampl figur to xplain our similarity-basd matching constraint schm (similarity-basd schm for short). Figur 3: (a) a typical tr-to-string drivation d using rul t, and (b) a nw drivation d* is gnratd by th similarity-basd matching constraint schm by using rul t* instad of rul t, whr t* t. Givn a sourc-languag pars tr T, in typical tr-to-string matching constraint schm shown in Figur 3(a), rul t usd by th drivation d should match a substr ABC of T. In our similarity-basd schm, th similar rul t* ( t ) is usd to form a nw drivation d* that prforms translation ovr th sam sourc sntnc {w 1... w n }. In such a cas, this nw drivation d* can yild a nw similar pars tr T* of T. Sinc an incorrct sourc pars tr might filtr out good drivations during tr-to-string dcoding, our similarity-basd schm is much mor likly to rcovr th corrct tr for dcoding at th dcoding tim, and dos not rul out good (potntially corrct) translation choics. In our mthod, many nw sourc-languag trs T * that ar similar to but diffrnt from th original sourc tr T can b rconstructd at th dcoding tim. In thory our similarity-basd schm can incras th sarch spac of th tr-to-string dcodr, but w did not chang any rul xtraction and pruning sttings. In practic, our similarity-basd schm can ffctivly kp th advantag of fast dcoding for tr-to-string translation bcaus its implmntation is vry simpl. Lt s rvisit th xampl drivation d in Figur 1, i.., d=r 1 r 2 r 3 r 4 r 1 5. In such a cas, th dcodr can ffctivly produc a nw drivation d* by simply rplacing rul r 3 with its similar rul τ 3 ( r 3 ) shown in Figur 2, that is, d*=r 1 r 2 τ 3 r 4 r 5. With bam sarch, typical tr-to-string dcoding with an intgratd languag modl can run in tim 2 O(ncb 2 ) in practic (Huang 2007). For our dcoding tim complxity computation, only th paramtr c valu can b affctd by our similarity-basd schm. In othr words, our similaritybasd schm would rsult in a largr c valu at dcoding tim as many similar translation ruls might b matchd at ach nod. In practic, thr ar two fasibl optimization tchniqus to allviat this problm. Th first tchniqu is to limit th maximum numbr of similar translation ruls matchd at ach nod. Th scond on is to prdfin a similarity thrshold to filtr out lss similar translation ruls in advanc. In th implmntation, w add a nw fatur into th modl: similarity-basd matching counting fatur. This fatur counts th numbr of similar ruls usd to form th drivation. Th wight λ sim of this fatur is tund via minimal rror rat training (MERT) (Och 2003) with othr fatur wights. 3.2 Psudo-rul Gnration In th implmntation of tr-to-string dcoding, th first stp is to load all translation ruls matchd at ach nod of th sourc tr T. It is possibl that som nontrminal nods do not hav any matchd ruls whn dcoding som nw sntncs. If th root nod of th sourc tr has no any matchd ruls, it would caus dcoding failur. To tackl this problm, motivatd by glu ruls (Chiang 2005), for som nod S without any matchd ruls, w introduc a spcial psudo-rul which rassmbls all child nods with local rordring to form nw translation ruls for S to complt dcoding. 1 Th symbol dnots th composition (lftmost substitution) opration of two tr-to-string ruls. 2 Whr n is th numbr of words, b is th siz of th bam, and c is th numbr of translation ruls matchd at ach nod. 420

4 S S(x 1 :A x 2 :B x 3 :C x 4 :D) x 1 x 2 x 3 x 4 S(x 1 :A x 2 :B x 3 :C x 4 :D) x 2 x 1 x 3 x 4 S(x 1 :A x 2 :B x 3 :C x 4 :D) x 1 x 3 x 2 x 4 A B C D S(x 1 :A x 2 :B x 3 :C x 4 :D) x 1 x 2 x 4 x 3 (a) (b) Figur 4: (a) An xampl unsn substr, and (b) its four psudo-ruls. Figur 4 (a) dpicts an xampl unsn substr whr no any ruls is matchd at its root nod S. Its simplst psudo-rul is to simply combin a squnc of S s child nods. To giv th modl mor options to build partial translations, w utiliz a local rordring tchniqu in which any two adjacnt frontir (child) nods ar rordrd during dcoding. Figur 4(b) shows four psudo-ruls in total gnratd from this xampl unsn substr. In th implmntation, w add a nw fatur to th modl: psudo-rul counting fatur. This fatur counts th numbr of psudo-ruls usd to form th drivation. Th wight λ psudo of this fatur is tund via MERT with othr fatur wights. 4 Evaluation 4.1 Stup Our bilingual training data consists of 140K Chins-English sntnc pairs in th FBIS data st. For rul xtraction, th minimal GHKM ruls (Gally t al. 2004) wr xtractd from th bitxt, and th composd ruls wr gnratd by combining two or thr minimal GHKM ruls. A 5-gram languag modl was traind on th targt-sid of th bilingual data and th Xinhua portion of English Gigaword corpus. Th bam siz for bam sarch was st to 20. Th bas fatur st usd for all systms is similar to that usd in (Marcu t al. 2006), including 14 bas faturs in total such as 5-gram languag modl, bidirctional lxical and phrasbasd translation probabilitis. All faturs wr linarly combind and thir wights ar optimizd by using MERT. Th dvlopmnt data st usd for wight training in our approachs coms from NIST MT03 valuation st. To spd up MERT, sntncs with mor than 20 words wr rmovd from th dvlopmnt st (Dv st). Th tst sts ar th NIST MT04 and MT05 valuation sts. Th translation quality was valuatd in trms of casinsnsitiv NIST vrsion BLEU mtric. Statistical significanc tst was conductd by using th bootstrap r-sampling mthod (Kohn 2004). 4.2 Rsults DEV MT04 MT05 MT03 <=20 ALL <=20 ALL Baslin This work * (+1.68) (+0.45) * (+2.33) (+0.55) * (+2.52) Tabl 1. BLEU4 (%) scors of various mthods on Dv st (MT03) and two tst sts (MT04 and MT05). Each small tst st (<=20) was built by rmoving th sntncs with mor than 20 words from th full st (ALL). + and * indicat significantly bttr on prformanc comparison at p <.05 and p <.01, rspctivly. Tabl 1 dpicts th BLEU scors of various mthods on th Dv st and four tst sts. Compard to typical tr-to-string dcoding (th baslin), our mthod can achiv significant improvmnts on all datasts. It is notworthy that th improvmnt achivd by our approach on full tst sts is biggr than that on small tst sts. For xampl, our mthod rsults in an improvmnt of 2.52 BLEU points ovr th baslin on th MT05 full tst st, but only 0.55 points on th MT05 small tst st. As mntiond bfor, tr-to-string approachs ar mor vulnrabl to parsing rrors. In practic, th Brkly parsr (Ptrov t al. 2006) w usd yilds unsatisfactory parsing prformanc on som long sntncs in th full tst sts. In such a cas, it would rsult in ngativ ffcts on th prformanc of th baslin mthod on th full tst sts. Exprimntal rsults show that our SDG approach can ffctivly allviat this problm, and significantly improv tr-to-string translation. Anothr issu w ar intrstd in is th dcoding spd of our mthod in practic. To invstigat this issu, w valuat th avrag dcoding spd of our SDG mthod and th baslin on th Dv st and all tst sts. Dcoding Tim (sconds pr sntnc) <=20 ALL Baslin 0.43s 1.1s This work 0.50s 1.3s Tabl 2. Avrag dcoding spd of various mthods on small (<=20) and full (ALL) datasts in trms of sconds pr sntnc. Th parsing tim of ach sntnc is not includd. Th dcodrs wr implmntd in C++ cods on an X86-basd PC with two procssors of 2.4GHZ and 4GB physical mmory. 421

5 Tabl 2 shows that our approach only has littl impact on dcoding spd in practic, compard to th typical tr-to-string dcoding (baslin). Notic that in ths comparisons our mthod did not adopt any optimization tchniqus mntiond in Sction 3.1,.g., to limit th maximum numbr of similar ruls matchd at ach nod. It is obviously that th us of such an optimization tchniqu can ffctivly incras th dcoding spd of our mthod, but might hurt th prformanc in practic. Bsids, to spd up dcoding long sntncs, it sms a fasibl solution to first divid a long sntnc into multipl short sub-sntncs for dcoding,.g., basd on comma. In othr words, w can sgmnt a complx sourc-languag pars tr into multipl smallr subtrs for dcoding, and combin th translations of ths small subtrs to form th final translation. This practical solution can spd up th dcoding on long sntncs in ralworld MT applications, but might hurt th translation prformanc. For convninc, hr w call th rul τ 3 in Figur 2(b) similar-ruls. It is worth invstigating how many similar-ruls and psudo-ruls ar usd to form th bst drivations in our similarity-basd schm. To do it, w count th numbr of similarruls and psudo-ruls usd to form th bst drivations whn dcoding on th MT05 full st. Exprimntal rsults show that on avrag 13.97% of ruls usd to form th bst drivations ar similarruls, and on psudo-rul pr sntnc is usd. Roughly spaking, avrag fiv similar-ruls pr sntnc ar utilizd for dcoding gnralization. 5 Rlatd Work String-to-tr SMT approachs also utiliz th similarity-basd matching constraint on targt sid to gnrat targt translation. This papr applis it on sourc sid to rconstruct nw similar sourc pars trs for dcoding at th dcoding tim, which aims to incras th tr-to-string sarch spac for dcoding, and improv dcoding gnralization for tr-to-string translation. Th most rlatd work is th forst-basd translation mthod (Mi t al. 2008; Mi and Huang 2008; Zhang t al. 2009) in which rul xtraction and dcoding ar implmntd ovr k-bst pars trs (.g., in th form of packd forst) instad of on bst tr as translation input. Liu and Liu (2010) proposd a joint parsing and translation modl by casting tr-basd translation as parsing (Eisnr 2003), in which th dcodr dos not rspct th sourc tr. Ths mthods can incras th trto-string sarch spac. Howvr, th dcoding tim complxity of thir mthods is high, i.., mor than tn or svral dozn tims slowr than typical trto-string dcoding (Liu and Liu 2010). Som prvious fforts utilizd th tchniqus of soft syntactic constraints to incras th sarch spac in hirarchical phras-basd modls (Marton and Rsnik 2008; Chiang t al. 2009; Huang t al. 2010), string-to-tr modls (Vnugopal t al. 2009) or tr-to-tr (Chiang 2010) systms. Ths mthods focus on softning matching constraints on th root labl of ach rul rgardlss of its intrnal tr structur, and oftn gnrat many nw syntactic catgoris 3. It maks thm mor difficult to satisfy syntactic constraints for th tr-to-string dcoding. 6 Conclusion and Futur Work This papr addrsss th pars rror issu for trto-string translation, and proposs a similaritybasd dcoding gnration solution by rconstructing nw similar sourc pars trs for dcoding at th dcoding tim. It is notworthy that our SDG approach is vry asy to implmnt. In principl, forst-basd and tr squnc-basd approachs improv rul covrag by changing th rul xtraction sttings, and us xact tr-to-string matching constraints for dcoding. Sinc our SDG approach is indpndnt of any rul xtraction and pruning tchniqus, it is also applicabl to forst-basd approachs or othr tr-basd translation modls,.g., in th cas of casting tr-to-tr translation as tr parsing (Eisnr 2003). Acknowldgmnts W would lik to thank Filiang Rn, Muhua Zhu and Hao Zhang for discussions and th anonymous rviwrs for commnts. This rsarch was supportd in part by th National Scinc Foundation of China ( ; ), th Spcializd Rsarch Fund for th Doctoral Program of Highr Education ( ) and th Fundamntal Rsarch Funds for th Cntral Univrsitis in China. 3 Latnt syntactic catgoris wr introducd in th mthod of Huang t al. (2010). 422

6 Rfrncs Chiang David A hirarchical phras-basd modl for statistical machin translation. In Proc. of ACL2005, pp Chiang David Larning to translat with sourc and targt syntax. In Proc. of ACL2010, pp Chiang David, Kvin Knight and Wi Wang ,001 nw faturs for statistical machin translation. In Proc. of NAACL2009, pp Eisnr Jason Larning non-isomorphic tr mappings for machin translation. In Proc. of ACL 2003, pp Gally Michl, Mark Hopkins, Kvin Knight and Danil Marcu What's in a translation rul? In Proc. of HLT-NAACL 2004, pp Huang Liang Binarization, synchronous binarization and targt-sid binarization. In Proc. of NAACL Workshop on Syntax and Structur in Statistical Translation. Huang Liang and David Chiang Forst rscoring: Fastr dcoding with intgratd languag modls. In Proc. of ACL 2007, pp Huang Liang, Kvin Knight and Aravind Joshi Statistical syntax-dirctd translation with xtndd domain of locality. In Proc. of AMTA 2006, pp Huang Zhongqiang, Martin Cmjrk and Bown Zhou Soft syntactic constraints for hirarchical phras-basd translation using latnt syntactic distribution. In Proc. of EMNLP2010, pp Kohn Philipp Statistical Significanc Tsts for Machin Translation Evaluation. In Proc. of EMNLP 2004, pp Liu Yang and Qun Liu Joint parsing and translation. In Proc. of Coling2010, pp Liu Yang, Qun Liu and Shouxun Lin Tr-tostring alignmnt tmplat for statistical machin translation. In Proc. of COLING/ACL 2006, pp Marcu Danil, Wi Wang, Abdssamad Echihabi and Kvin Knight SPMT: Statistical machin translation with syntactifid targt languag phrass. In Proc. of EMNLP 2006, pp Marton Yuval and Philip Rsnik Soft syntactic constraints for hirarchical phras-basd translation. In Proc. of ACL08, pp Mi Haitao and Liang Huang Forst-basd Translation Rul Extraction. In Proc. of EMNLP 2008, pp Mi Haitao, Liang Huang and Qun Liu Forstbasd translation. In Proc. of ACL2008. Och Franz Josf Minimum rror rat training in statistical machin translation. In Proc. of ACL2003. Ptrov Slav, Lon Barrtt, Roman Thibaux and Dan Klin Larning accurat, compact, and intrprtabl tr annotation. In Proc. of ACL2006, pp Xiao Tong, Jingbo Zhu, Hao Zhang and Muhua Zhu An mpirical study of translation rul xtraction with multipl parsrs. In Proc. of Coling2010, pp Vnugopal Ashish, Andras Zollmann, Noah A. Smith and Stphan Vogl Prfrnc grammars: softning syntactic constraints to improv statistical machin translation. In Proc. of NAACL2009, pp Zhang Hui, Min Zhang, Haizhou Li, Aiti Aw and Chw Lim Tan Forst-basd tr squnc to string translation modl. In Proc. of ACL-IJCNLP2009, pp

2. Tree-to-String [6] (Phrase Based Machine Translation, PBMT)[1] [2] [7] (Targeted Self-Training) [7] Tree-to-String [3] Tree-to-String [4]

2. Tree-to-String [6] (Phrase Based Machine Translation, PBMT)[1] [2] [7] (Targeted Self-Training) [7] Tree-to-String [3] Tree-to-String [4] 1,a) 1,b) Graham Nubig 1,c) 1,d) 1,) 1. (Phras Basd Machin Translation, PBMT)[1] [2] Tr-to-String [3] Tr-to-String [4] [5] 1 Graduat School of Information Scinc, Nara Institut of Scinc and Tchnology a)