Learning to Jointly Translate and Predict Dropped Pronouns with a Shared Reconstruction Mechanism

Transcription

1 Learning o Joinly Translae and Predic Dropped Pronouns wih a Shared Reconsrucion Mechanism Longyue Wang Tencen AI Lab vinnylywang@encen.com Andy Way Dublin Ciy Universiy andy.way@adapcenre.ie Absrac Pronouns are frequenly omied in pro-drop languages, such as Chinese, generally leading o significan challenges wih respec o he producion of complee ranslaions. Recenly, Wang e al. (2018) proposed a novel reconsrucion-based approach o alleviaing dropped pronoun (DP) ranslaion problems for neural machine ranslaion models. In his work, we improve he original model from wo perspecives. Firs, we employ a shared reconsrucor o beer exploi encoder and decoder represenaions. Second, we joinly learn o ranslae and predic DPs in an end-o-end manner, o avoid he errors propagaed from an exernal DP predicion model. Experimenal resuls show ha our approach significanly improves boh ranslaion performance and DP predicion accuracy. 1 Inroducion Pronouns are imporan in naural languages as hey imply rich discourse informaion. However, in pro-drop languages such as Chinese and Japanese, pronouns are frequenly omied when heir referens can be pragmaically inferred from he conex. When ranslaing senences from a pro-drop language ino a non-pro-drop language (e.g. Chinese-o-English), ranslaion models generally fail o ranslae invisible dropped pronouns (DPs). This phenomenon leads o various ranslaion problems in erms of compleeness, synax and even semanics of ranslaions. A number of approaches have been invesigaed for DP ranslaion (Le Nagard and Koehn, 2010; Xiang e al., 2013; Wang e al., 2016, 2018). Wang e al. (2018) is a pioneering work o model DP ranslaion for neural machine rans- Zhaopeng Tu is he corresponding auhor of he paper. This work was conduced when Longyue Wang was sudying and Qun Liu was working a he ADAPT Cenre in he School of Compuing a Dublin Ciy Universiy. Zhaopeng Tu Tencen AI Lab zpu@encen.com Qun Liu Huawei Noah s Ark Lab qun.liu@huawei.com laion (NMT) models. They employ wo separae reconsrucors (Tu e al., 2017) o respecively reconsruc encoder and decoder represenaions back o he DP-annoaed source senence. The annoaion of DP is provided by an exernal predicion model, which is rained on he parallel corpus using auomaically learned alignmen informaion (Wang e al., 2016). Alhough his model achieved significan improvemens, here noneheless exis wo drawbacks: 1) here is no ineracion beween he wo separae reconsrucors, which misses he opporuniy o exploi useful relaions beween encoder and decoder represenaions; and 2) he exernal DP predicion model only has an accuracy of 66% in F1-score, which propagaes numerous errors o he ranslaion model. In his work, we propose o improve he original model from wo perspecives. Firs, we use a shared reconsrucor o read hidden saes from boh encoder and decoder. Second, we inegrae a DP predicor ino NMT o joinly learn o ranslae and predic DPs. Incorporaing hese as wo auxiliary loss erms can guide boh he encoder and decoder saes o learn criical informaion relevan o DPs. Experimenal resuls on a largescale Chinese English subile corpus show ha he wo modificaions can accumulaively improve ranslaion performance, and he bes resul is +1.5 BLEU poins beer han ha repored by Wang e al. (2018). In addiion, he joinly learned DP predicion model significanly ouperforms is exernal counerpar by 9% in F1-score. 2 Background As shown in Figure 1, Wang e al. (2018) inroduced wo independen reconsrucors wih heir own parameers, which reconsruc he DPannoaed source senence from he encoder and decoder hidden saes, respecively. The cenral 2997 Proceedings of he 2018 Conference on Empirical Mehods in Naural Language Processing, pages Brussels, Belgium, Ocober 31 - November 4, c 2018 Associaion for Compuaional Linguisics

2 y x decoder encoder Did you bake i???? ˆx encoderreconsrucor decoderreconsrucor Figure 1: Archiecure of separae reconsrucors. Predicion F1-score Example DP Posiion 88% 你烤的 #DP# 吗? DP Words 66% 你烤的它吗? Table 1: Evaluaion of exernal models on predicing he posiions of DPs ( DP Posiion ) and he exac words of DP ( DP Words ). idea underpinning heir approach is o guide he corresponding hidden saes o embed he recalled source-side DP informaion and subsequenly o help he NMT model generae he missing pronouns wih hese enhanced hidden represenaions. The DPs can be auomaically annoaed for raining and es daa using wo differen sraegies (Wang e al., 2016). In he raining phase, where he arge senence is available, we annoae DPs for he source senence using alignmen informaion. These annoaed source senences can be used o build a neural-based DP predicor, which can be used o annoae es senences since he arge senence is no available during he esing phase. As shown in Table 1, Wang e al. (2016, 2018) explored o predic he exac DP words 1, he accuracy of which is only 66% in F1-score. By analyzing he ranslaion oupus, we found ha 16.2% of errors are newly inroduced and caused by errors from he DP predicor. Forunaely, he accuracy of predicing DP posiions (DPPs) is much higher, which provides he chance o alleviae he error propagaion problem. Inuiively, we can learn o generae DPs a he prediced posiions using a joinly rained DP predicor, which is fed wih informaive represenaions in he reconsrucor. 1 Unless oherwise indicaed, in he paper, he erms DP and DP word are idenical. ˆx 3 Approach 3.1 Shared Reconsrucor Recen work shows ha NMT models can benefi from sharing a componen across differen asks and languages. Taking muli-language ranslaion as an example, Fira e al. (2016) share an aenion model across languages while Dong e al. (2015) share an encoder. Our work is mos similar o he work of Zoph and Knigh (2016) and Anasasopoulos and Chiang (2018), which share a decoder and wo separae aenion models o read from wo differen sources. In conras, we share informaion a he level of reconsruced frames. The archiecures of our proposed shared reconsrucion model are shown in Figure 2(a). Formally, he reconsrucor reads from boh he encoder and decoder hidden saes, as well as he DP-annoaed source senence, and oupus a reconsrucion score. I uses wo separae aenion models o reconsruc he annoaed source senence ˆx = {ˆx 1, ˆx 2,..., ˆx T } word by word, and he reconsrucion score is compued by R(ˆx h enc, h dec ) = T =1 g r (ˆx 1, h rec, ĉ enc, ĉ dec ) where h rec is he hidden sae in he reconsrucor, and compued by Equaion (1): h rec = f r (ˆx 1, h rec 1, ĉ enc, ĉ dec ) (1) Here g r ( ) and f r ( ) are respecively sofmax and acivaion funcions for he reconsrucor. The conex vecors ĉ enc and ĉ dec are he weighed sum of h enc and h dec, respecively, as in Equaion (2) and (3): ĉ enc ĉ dec = J j=1 ˆαenc,j h enc j (2) = I i=1 ˆαdec,i h dec i (3) Noe ha he weighs ˆα enc and ˆα dec are calculaed by wo separae aenion models. We propose wo aenion sraegies which differ as o wheher he wo aenion models have ineracions or no. Independen Aenion calculaes he wo weigh marices independenly, as in Equaion (4) and (5): ˆα enc = ATT enc (ˆx 1, h rec 1, h enc ) (4) ˆα dec = ATT dec (ˆx 1, h rec 1, h dec ) (5) where ATT enc ( ) and ATT dec ( ) are wo separae aenion models wih heir own parameers. 2998

3 y Did you bake i? y Did you bake i? decoder? ˆx shared reconsrucor decoder #DP#? ˆx shared reconsrucor encoder encoder dp join predicion x? x? (a) Shared reconsrucor. (b) Shared reconsrucor wih join predicion. Figure 2: Model archiecures in which he words in red are auomaically annoaed DPs and DPPs. Ineracive Aenion feeds he conex vecor produced by one aenion model o anoher aenion model. The inuiion behind his is ha he ineracion beween wo aenion models can lead o a beer exploiaion of he encoder and decoder represenaions. As he ineracive aenion is direcional, we have wo opions (Equaion (6) and (7)) which modify eiher ATT enc ( ) or ATT dec ( ) while leaving he oher one unchanged: enc dec: ˆα dec = ATT dec (ˆx 1, h rec 1, h dec, ĉ enc ) (6) dec enc: ˆα enc = ATT enc (ˆx 1, h rec 1, h enc, ĉ dec ) (7) 3.2 Join Predicion of Dropped Pronouns Inspired by recen successes of muli-ask learning (Dong e al., 2015; Luong e al., 2016), we propose o joinly learn o ranslae and predic DPs (as shown in Figure 2(b)). To ease he learning difficuly, we leverage he informaion of DPPs prediced by an exernal model, which can achieve an accuracy of 88% in F1-score. Accordingly, we ransform he original DP predicion problem o DP word generaion given he pre-prediced DP posiions. Since he DPP-annoaed source senence serves as he reconsruced inpu, we inroduce an addiional DP-generaion loss, which measures how well he DP is generaed from he corresponding hidden sae in he reconsrucor. Le dp = {dp 1, dp 2,..., dp D } be he lis of DPs in he annoaed source senence, and h rec = {h rec 1, hrec 2,..., hrec D } be he corresponding hidden saes in he reconsrucor. The generaion probabiliy is compued by P (dp h rec ) = = D d=1 D d=1 P (dp d h rec d ) g p (dp d h rec d ) where g p ( ) is sofmax for he DP predicor. 3.3 Training and Tesing (8) We rain boh he encoder-decoder and he shared reconsrucors ogeher in a single end-o-end process, and he raining objecive is J(θ, γ, ψ) = arg max θ,γ,ψ { log L(y x; θ) }{{} likelihood + log R(ˆx h enc, h dec ; θ, γ) }{{} reconsrucion } + log P (dp ĥrec ; θ, γ, ψ) }{{} predicion (9) where {θ, γ, ψ} are respecively he parameers associaed wih he encoder-decoder, shared reconsrucor and he DP predicion model. The auxiliary reconsrucion objecive R( ) guides he relaed par of he parameer marix θ o learn beer laen represenaions, which are used o reconsruc he DPP-annoaed source senence. The auxiliary predicion loss P ( ) guides he relaed par of boh he encoder-decoder and he reconsrucor o learn beer laen represenaions, which are used o predic he DPs in he source senence. Following Tu e al. (2017) and Wang e al. (2018), we use he reconsrucion score 2999

4 # Model #Params Speed Train Decode BLEU Exising sysem (Wang e al., 2018) 1 Baseline 86.7M 1.60K Baseline (+DPs) 86.7M 1.59K Separae-Recs (+DPs) +73.8M 0.57K Our sysem 4 Baseline (+DPPs) 86.7M 1.54K Shared-Rec independen (+DPPs) +86.6M 0.52K Shared-Rec independen (+DPPs) + join predicion +87.9M 0.51K Shared-Rec enc dec (+DPPs) + join predicion +91.9M 0.48K Shared-Rec dec enc (+DPPs) + join predicion +89.9M 0.49K Table 2: Evaluaion of ranslaion performance for Chinese English. Baseline is rained and evaluaed on he original daa, while Baseline (+DPs) and Baseline (+DPPs) are rained on he daa annoaed wih DPs and DPPs, respecively. Training and decoding (beam size is 10) speeds are measured in words/second. and indicae saisically significan difference (p < 0.01) from Baseline (+DDPs) and Separae-Recs (+DPs), respecively. as a reranking echnique o selec he bes ranslaion candidae from he generaed n-bes lis a esing ime. Differen from Wang e al. (2018), we reconsruc DPP-annoaed source senence, which is prediced by an exernal model. 4 Experimen 4.1 Seup To compare our work wih he resuls repored by previous work (Wang e al., 2018), we conduced experimens on heir released Chinese English TV Subile corpus. 2 The raining, validaion, and es ses conain 2.15M, 1.09K, and 1.15K senence pairs, respecively. We used case-insensiive 4-gram NIST BLEU merics (Papineni e al., 2002) for evaluaion, and sign-es (Collins e al., 2005) o es for saisical significance. We implemened our models on he code reposiory released by Wang e al. (2018). 3 We used he same configuraions (e.g. vocabulary size = 30K, hidden size = 1000) and reproduced heir repored resuls. I should be emphasized ha we did no use he pre-rain sraegy as done in Wang e al. (2018), since we found raining from scrach achieved a beer performance in he shared reconsrucor seing. 2 hps://gihub.com/longyuewangdcu/ vsub 3 hps://gihub.com/uzhaopeng/nm 4.2 Resuls Table 2 shows he ranslaion resuls. I is clear ha he proposed models significanly ouperform he baselines in all cases, alhough here are considerable differences among differen variaions. Baselines (Rows 1-4): The hree baselines (Rows 1, 2, and 4) differ regarding he raining daa used. Separae-Recs (+DPs) (Row 3) is he bes model repored in Wang e al. (2018), which we employed as anoher srong baseline. The baseline rained on he DPP-annoaed daa ( Baseline (+DPPs), Row 4) ouperforms he oher wo counerpars, indicaing ha he error propagaion problem does affec he performance of ranslaing DPs. I suggess he necessiy of joinly learning o ranslae and predic DPs. Our Models (Rows 5-8): Using our shared reconsrucor (Row 5) no only ouperforms he corresponding baseline (Row 4), bu also surpasses is separae reconsrucor counerpar (Row 3). Inroducing a join predicion objecive (Row 6) can achieve a furher improvemen of BLEU poins. These resuls verify ha shared reconsrucor and joinly predicing DPs can accumulaively improve ranslaion performance. Among he variaions of shared reconsrucors (Rows 6-8), we found ha an ineracion aenion from encoder o decoder (Row 7) achieves he bes performance, which is BLEU poins beer han our baseline (Row 4) and BLEU poins 3000

5 beer han he bes resul repored by Wang e al. (2018) (Row 3). We aribue he superior performance of Shared-Rec enc dec o he fac ha he aenion conex over encoder represenaions embeds useful DP informaion, which can help o beer aend o he represenaions of he corresponding pronouns in he decoder side. Similar o Wang e al. (2018), he proposed approach improves BLEU scores a he cos of decreased raining and decoding speed, which is due o he large number of newly inroduced parameers resuling from he incorporaion of reconsrucors ino he NMT model. 4.3 Analysis Models Precision Recall F1-score Exernal Join Table 3: Evaluaion of DP predicion accuracy. Exernal model is separaely rained on DP-annoaed daa wih exernal neural mehods (Wang e al., 2016), while Join model is joinly rained wih he NMT model (Secion 3.2). Model Auo. Man. Seperae-Recs (+DPs) Shared-Rec (+DPPs) Table 5: Translaion performance gap ( ) beween manually ( Man. ) and auomaically ( Auo. ) labelling DPs/DPPs for inpu senences in esing. Effec of DPP Labelling Accuracy For each senence in esing, he DPs and DPPs are labelled auomaically by wo separae exernal predicion models, he accuracy of which are respecively 66% and 88% measured in F1 score. We invesigae he bes performance he models can achieve wih manual labelling, which can be regarded as an Oracle, as shown in Table 5. As seen, here sill exiss a significan gap in performance, and his could be improved by improving he accuracy of our DPP generaor. In addiion, our models show a relaively smaller disance in performance from he oracle performance ( Man ), indicaing ha he error propagaion problem is alleviaed o some exen. DP Predicion Accuracy As shown in Table 3, he joinly learned model significanly ouperforms he exernal one by 9% in F1-score. We aribue his o he useful conexual informaion embedded in he reconsrucor represenaions, which are used o generae he exac DP words. Model Tes Baseline (+DPPs) Separae-Recs (+DPs) Shared-Rec (+DPPs) Table 4: Translaion resuls when reconsrucion is used in raining only while no used in esing. Conribuion Analysis Table 4 liss ranslaion resuls when he reconsrucion model is used in raining only. We can see ha he proposed model ouperforms boh he srong baseline and he bes model repored in Wang e al. (2018). This is encouraging since no exra resources and compuaion are inroduced o online decoding, which makes he approach highly pracical, for example for ranslaion in indusry applicaions. 5 Conclusion In his paper, we proposed effecive approaches of ranslaing DPs wih NMT models: shared reconsrucor and joinly learning o ranslae and predic DPs. Through experimens we verified ha 1) shared reconsrucion is helpful o share knowledge beween he encoder and decoder; and 2) join learning of he DP predicion model indeed alleviaes he error propagaion problem by improving predicion accuracy. The wo approaches accumulaively improve ranslaion performance. The mehod is no resriced o he DP ranslaion ask and could poenially be applied o oher sequence generaion problems where addiional source-side informaion could be incorporaed. In fuure work we plan o: 1) build a fully end-o-end NMT model for DP ranslaion, which does no depend on any exernal componen (i.e. DPP predicor); 2) exploi cross-senence conex (Wang e al., 2017) o furher improve DP ranslaion; 3) invesigae a new research srand ha adaps our model in an inverse ranslaion direcion by learning o drop pronouns insead of recovering DPs. 3001

6 Acknowledgmens The ADAPT Cenre for Digial Conen Technology is funded under he SFI Research Cenres Programme (Gran 13/RC/2106) and is co-funded under he European Regional Developmen Fund. We hank he anonymous reviewers for heir insighful commens. References Anonios Anasasopoulos and David Chiang Tied muliask learning for neural speech ranslaion. In Proceedings of he 2018 Conference of he Norh American Chaper of he Associaion for Compuaional Linguisics: Human Language Technologies, pages 82 91, New Orleans, Louisiana, USA. Michael Collins, Philipp Koehn, and Ivona Kucerova Clause resrucuring for saisical machine ranslaion. In Proceedings of he 43rd Annual Meeing of he Associaion for Compuaional Linguisics, pages , Ann Arbor, Michigan, USA. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang Muli-ask learning for muliple language ranslaion. In Proceedings of he 53rd Annual Meeing of he Associaion for Compuaional Linguisics, pages , Beijing, China. Orhan Fira, Kyunghyun Cho, and Yoshua Bengio Muli-way, mulilingual neural machine ranslaion wih a shared aenion mechanism. In Proceedings of he 2016 Conference of he Norh American Chaper of he Associaion for Compuaional Linguisics: Human Language Technologies, pages , San Diego, California, USA. Ronan Le Nagard and Philipp Koehn Aiding pronoun ranslaion wih co-reference resoluion. In Proceedings of he Join 5h Workshop on Saisical Machine Translaion and MericsMATR, pages , Uppsala, Sweden. Minh-Thang Luong, Quoc V Le, Ilya Suskever, Oriol Vinyals, and Lukasz Kaiser Muli-ask sequence o sequence learning. In Proceedings of he 27h Inernaional Conference on Compuaional Linguisics, pages , Sana Fe, New Mexico, USA. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu BLEU: A mehod for auomaic evaluaion of machine ranslaion. In Proceedings of he 40h Annual Meeing on Associaion for Compuaional Linguisics, pages , Philadelphia, Pennsylvania, USA. Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li Neural machine ranslaion wih reconsrucion. In Proceedings of he 31s AAAI Conference on Arificial Inelligence, pages , San Francisco, California, USA. Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvee Graham, and Qun Liu Translaing pro-drop languages wih reconsrucion models. In Proceedings of he 32nd AAAI Conference on Arificial Inelligence, pages , New Orleans, Louisiana, USA. Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu Exploiing cross-senence conex for neural machine ranslaion. In Proceedings of he 2017 Conference on Empirical Mehods in Naural Language Processing, pages , Copenhagen, Denmark. Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu A novel approach for dropped pronoun ranslaion. In Proceedings of he 2016 Conference of he Norh American Chaper of he Associaion for Compuaional Linguisics: Human Language Technologies, pages , San Diego, California, USA. Bing Xiang, Xiaoqiang Luo, and Bowen Zhou Enlising he ghos: Modeling empy caegories for machine ranslaion. In Proceedings of he 51s Annual Meeing of he Associaion for Compuaional Linguisics, pages , Sofia, Bulgaria. Barre Zoph and Knigh Knigh Muli-source neural ranslaion. In Proceedings of he 2016 Conference of he Norh American Chaper of he Associaion for Compuaional Linguisics: Human Language Technologies, pages 30 34, San Diego, California. 3002