Extract Domain-specific Paraphrase from Monolingual Corpus for Automatic Evaluation of Machine Translation

Transcription

1 Extract Doma-specfc Paraphrase from Moolgual Corpus for Automatc Evaluato of Mache Traslato Ll Zhag, Zhe Weg, Weya Xao, Jay Wa, Zhmg Che, Ymg Ta, Maox L, Mgwe Wag School of Computer Iformato Egeerg, Jagx Normal Uversty { , , , {qqchezhmg, tt_yymm, mosesl, Abstract Paraphrase ca help match syoyms or match phrases wth the same or smlar meag, thus t plays a mportat role automatc evaluato of mache traslato. The tradtoal approaches extract paraphrase geeral doma from blgual corpus. Because the WMT16 metrcs task cossts of three subtasks, amely ews doma, medcal doma, ad IT doma, we propose to extract domaspecfc paraphrase tables from moolgual corpus to replace the geeral paraphrase table. We utlze the M-L approach to flter the large scale geeral moolgual corpus to a doma-specfc sub-corpus, ad explot Markov Network model to extract paraphrase tables from the sub-corpus. The expermetal results o WMT15 Metrcs task show that METEOR metrc usg the doma-specfc paraphrase tables outperforms that usg the paraphrase table geeral doma extracted from the blgual corpus. 1 Itroducto Mache traslato (MT) automatc evaluato metrcs, such as BLEU (Pape et al., 00), NIST (Doddgto, 00), METEOR (Baeree et al., 005), TER (Sover et al., 006), MAXSIM (Cha et al., 008) etc., evaluate the qualty of the MT system output by calculatg the smlarty betwee the traslato output ad the huma referece. Accurately matchg words or phrases wth the same or smlar meag s crtcal to the performace of the automatc evaluato metrcs (L et al., 013; L et al., 016). Recetly, may works ehaced tradtoal metrcs by addg paraphrase match. For stace, the latest verso of METEOR package (Dekowsk ad Lave, 014), the paraphrase match was added after the stadard exact word match, stem match ad syoym match. Ad the latest verso of TER package (Baard et al., 005) relaxes the codto of word match or chuk shft by addg paraphrase match. Note that the paraphrase tables used latest METE- OR ad TER metrcs belog to the geeral doma ad they are extracted from blgual parallel corpus by the Pvot approach (Baard et al., 005). However, the WMT16 metrcs task cossts of sub-tasks o specfc domas volvg several dfferet laguages. Cofroted wth the chages, we propose a Moolgual Paraphrase Extracto method based o Doma Adaptato (MPEDA), ad use the ew doma-specfc paraphrase table to replace the tradtoal paraphrase tables the latest METEOR package. Related Work I statstcal atural laguage processg, both the scale ad the qualty of the trag data have a drect mpact o the performace of statstcal learg. Take statstcal MT for a example, f the sze of trag data s larger ad the more t covers -gram appeared the test set, the qualty of the MT outputs wll be better. To expad the scale of the exstg domaspecfc corpus, Moore ad Lews (010) traed models wth geeral corpus ad doma-specfc corpus, ad computed cross etropy of each setece the geeral corpus to extract a subcorpus much larger tha the exstg domaspecfc corpus. I ths way, a large scale doma-specfc trag corpus for statstcal MT was establshed. Alog ths approach, Amtta et al. (011) proposed a blgual parallel data selecto approach based o cross etropy to mprove the MT performace for spoke laguage traslato. Ad Jur et al. (015) fltered trag data for automatc extracto of paraphrase by usg Moore ad Lews approach to extract paraphrases from the fltered trag data va the Pvot approach. Automatcally extractg paraphrases from the large scale corpus s low cost. Barzlay ad McKeow (001) preseted a usupervsed learg approach to extract paraphrases of 511 Proceedgs of the Frst Coferece o Mache Traslato, Volume : Shared Task Papers, pages , Berl, Germay, August 11-1, 016. c 016 Assocato for Computatoal Lgustcs

2 words ad phrases from dfferet Eglsh traslatos of the detcal source laguage seteces. Baard ad Callso-Burch (005) employed the word algmet techque of statstcal MT to extract paraphrases from blgual parallel corpus. Shyama et al. (00) used the amed etty recogto features to extract paraphrases from moolgual comparable corpus. Barzlay ad Lee (003) used text strgs algmet algorthm to lear paraphrases at setece level from the uaotated comparable corpus. Yet, there are stll great restrctos of the latter two moolgual paraphrase extracto methods. Therefore, we adopt the Markov-based method proposed by Weg et al. (015) to extract paraphrases specfc doma from moolgual corpus because that t has o restrctos o moolgual corpus the target laguage as t ca extract paraphrase by costructg the Markov etworks of words. Pror to the paraphrase extracto, we frst flter large scale moolgual corpus to sub-corpus close to the doma of the huma referece. Compared wth geeral trag corpus, the fltered sub-corpus s smaller ad more related to the target doma, whch results the mprovemet o the qualty of paraphrase table as well as the performace whe the paraphrase table s appled automatc evaluato metrc. 3 MPEDA: Moolgual Paraphrase Extracto Based o Doma Adaptato We extract doma-specfc paraphrases from the moolgual corpus whch are the most related to the test data. Our approach ams at accurately matchg syoyms ad phrases wth the same or smlar meag MT outputs ad huma refereces wth the help of the doma-specfc paraphrase. We frst flter a sub-corpus from a large geeral corpus by the exteded M-L method, ad the extract paraphrases based o Markov Network model ad fally apply the paraphrase table to METEOR metrc. 3.1 Extractg paraphrases based o word chuks Accordg to the Markov Network model, we frst use the term co-occurrece the text set to calculate the correlato amog terms ad costruct a term Markov etwork where the correlato betwee two words the etwork (edge weght) s computed by the ot codtoal probablty of two terms the text set accordg to Formula (1) - (3), whch codtoal probablty P(t t ) ad P(t t ) are ot equal. P( t t ) P( t t ) R( t, t ) (1) C( t, t ) P( t t ) () Ct ( ) C( t, t ) P( t t) (3) Ct ( ) I Formula (1) (3), t ad t stad for two terms, C(t, t ) s the umber of documets that the whole trag data term t ad term t cooccur the same wdow, C(t ) ad C(t ) deote the umbers of documets that term t ad term t occur the whole trag data respectvely, R(t, t ) deotes the correlato betwee term t ad term t. The greater the R value, the hgher the correlato betwee the two terms. Extractg paraphrases from the costructed term Markov etwork s bult o the followg hypothess: the more word chuks co-occurrg betwee two terms, the more smlar ther sematc meags are, ad thus the two terms are a paraphrase par. Therefore, we eed to buld a -gram word chuk set for each term ad the calculate the rato betwee the umber of cooccurrg word chuks of two terms ad the total umber of word chuks wth oe term occurrg. The rato s cosdered as the possblty of the two terms costructg a paraphrase par, whch ca be obtaed by formula (4) - (6). Formula (6) s used to calculate the weght of -gram word chuk. W3 ( t, t ) pos( t, t ) (4) 1 ( W3 ( t ) W3 ( t )) W ( t, t ) w ( t, t, t ) (5) 3 3 k kk tk clque ( t, t, tk ) w { t, t,... t } 1, 1 1 R( t, t ) ( 1) (6) I the above formulas, pos(t,t ) s the paraphrase probablty of term t ad term t, W 3 (t,t ) s the sum of weghts of all the 3-gram word chuks cotag term t ad term t, W 3 (t ) s the sum of weghts of all the 3-gram word chuks cotag term t, W 3 (t ) deotes the sum of weghts of all the 3-gram word chuks cotag term t, deotes the umber of odes word chuk, R(t,t ) deotes the correlato betwee term t ad term t. 51

3 We use the terms co-occurrece to costruct a term Markov etwork ad extract phrases the corpus as a ode of Markov etwork. Fgure 1 shows a example of 3-gram word chuk, where t 1 stads for the term computer, t stads for the term Iteret, t 3 stads for the term calculatg mache, t 4 stads for the term electroc. I ths example, the 3-gram word chuk set for each term s S(C 3 (t 1 ))= {{ t 1,t,t 3 }, {t 1, t 3, t 4 }}, S(C 3 (t ))={ t 1, t, t 3 }, S(C 3 (t 3 )= {{ t 1,t,t 3 }, {t 1,t 3,t 4 }}, S(C 3 (t 4 ))={ t 1,t 3, t 4 }. It ca be observed that S(C 3 (t 1 ))= S(C 3 (t 3 )= {{ t 1,t,t 3 }, {t 1,t 3,t 4 }}, hece, there s a hgh correlato betwee the two terms of t 1 ad t 3. Based o the hypothess of ths paper, we thk term t 1, computer, ad term t 3, calculatg mache, ths example s a paraphrase par. computer t1 Fgure 1: 3-gram word chuk 3. Corpus flterg electroc t4 Iteret t 3..1 M-L corpus flterg The corpus flterg method s bult smlar to the M-L method proposed by Moore ad Lews (010). To extract a sub-corpus of target doma from the large geeral corpus, we frst select a doma-specfc corpus ad a geeral large scale corpus. To mprove the automatc MT metrc, we use the huma refereces of each sub-task the metrc tasks as the doma-specfc corpus, ad tra the laguage model of the two corpora respectvely, furthermore, we calculate the cross etropy of the two models. Fally, the smlarty betwee the seteces ad the huma refereces s measured by calculatg the dfferece of two cross etropy of the same setece accordg to Formula (7). Geerally, smaller value meas the setece s closer to the target doma. H ( S ) H ( S ) (7) s ref tra Calculatg mache I formula (7), S deotes the -th setece, H ref deotes the cross etropy of the laguage t3 model traed from the huma refereces, whle H tra deotes the cross etropy of the laguage model traed from the trag data. 3.. Documet sets flterg The Markov etwork-based automatc paraphrase extracto approach requres dvde a geeral moolgual corpus to dfferet documet sets. Weg et al. (015) dvded the text of a fxed legth to a documet wthout cosderg the correlato amog documets. Hece, we form the seteces the corpus to cluster va K-meas clusterg algorthm, ad the use the bag of word model to create a vector for each setece the corpus. Thus the dstace betwee two seteces ca be obtaed by calculatg the cose value of the two vectors. Each cluster s vewed as a documet. I the process of clusterg, dvdg documets va K-meas algorthm ca guaratee that the seteces a documet approxmately belog to the same doma. The, the M-L method s used to extract the sub-sets of documets whch are close to the target doma from the clustered geeral documet sets. Ths sgfes that t s the documet ot the setece that s regarded as the smallest flterg ut the process of corpus flterg. Ad we wat to detfy documets whch are smlar to our target doma by summg up the dfferece of cross etropy of each setece the documet. However, whe dvdg the large-scale corpus to documets va K-meas algorthm, the umber of seteces the documets vares, thus we calculate the mea after summg up the dfferece of cross etropy of each setece to obta the score of each documet by Formula (8), ( H ( ) ( )) 1 ref S Htra S D (8) where s the score of the -th documet, D H ref (S ) s the cross etropy of the -th setece the documet D derved from the laguage model of the refereces, H tra (S ) s the cross etropy of the -th setece the documet D derved from the laguage model of the trag data, s the umber of seteces the documet D. The we sort ascedg order. The lower D score mples the documet s more lke the huma refereces. D 513

4 4 Expermets To test the qualty of the doma-specfc paraphrase extracted from moolgual corpus by the proposed approach, we coducted expermets o WMT15 Metrcs task. The METEOR-Uversal metrc (Dekowsk ad Lave, 014) usg the paraphrase tables whch were extracted from the blgual parallel corpus was set as the basele metrc. We used the paraphrase tables geeral doma extracted by the Markov Network model, ad the domaspecfc paraphrase tables extracted by our approach substtuted for the orgal paraphrased tables, respectvely. The updated metrcs are called as METEOR-Markov ad METEOR- MPEDA. We compared the METEOR-MPEDA metrc wth the METEOR-Markov metrc ad METEOR-Uversal metrc to demostrate the qualty of the doma-specfc paraphrase table extracted by our approach. Besdes, we compared the METEOR-MPEDA wth METEOR metrc (Baeree et al., 005) whch oly uses the exact word match, stem match ad syoym match. Data e-cs e-de e-fr e-f e-ru cs-e de-e fr-e f-e ru-e T-corpus 1000k 190k 007k 196k 1074k 18k 18k 18k 18k 18k ref Table 1. The statstcs of the corpus Data e-cs e-de e-fr e-f e-ru cs-e de-e fr-e f-e ru-e D-corpus Table. The umber of documets trag data 4.1 Corpus The trag data ad the huma refereces we used the expermet are all provded WMT15 Traslato task ad Metrcs task (Boar et al., 015), every trag data has ts correspodg refereces. Table 1 shows the umber of seteces the corpora. The row T-corpus deotes the trag data, whle the row ref deotes the refereces. The trag data was processed by text clusterg. We used K-meas clusterg algorthm to gather the corpus seteces dfferet clusters, ad the adopted the bag of word to create a vector for each setece. By computg the cose value of the two vectors, we obtaed the dstace betwee two seteces. Each cluster was vewed as a documet. The -th documet trag data was amed D, ad the umber of seteces each documet was dfferet. Table s the umber of documets after trag data clusterg. The row D-corpus s the umber of documet used the trag data. 4. Expermets Settgs After dvdg the trag data to documets, we processed the corpus by the followg procedure: tokeze the trag data ad the refereces; delete the puctuatos; trasform the captalzed letters of words to lower case. The, we employed 4-gram laguage model wth Keser-Ney dscoutg to tra correspodg laguage models for trag data ad the refereces. The dfferece of cross etropy of each setece the trag data laguage model was calculated. The we summed up ad ormalzed the dfferece of the cross etropy of the documets seteces. Thus every documet the trag data receved a score. The smaller the value s, the closer the documet s to the referece. Later, we arraged the values a ascedg order, meawhle, a threshold value was set, ad the corpus beyod the threshold was abadoed. I ths way, we obtaed a smaller subcorpus wth the approxmately same doma wth the trag data. Fally, we gave dfferet threshold value to the dfferet sub-tasks, other words, we selected the top documets after orderg. We used the Markov etwork to buld a term Markov etwork model the sub-corpus, the we calculated the relato amog words accordg to words co-occurrece, ext, we extracted the word chuks the Markov etwork, ad computed the lkelhood that two words are a paraphrase par by comparg the two chuks smlarty. I ths work, we extracted te paraphrase tables for te sub-tasks sx laguages o WMT

5 4.3 Results The Pearso Coeffcet s used to compute the system-level correlato betwee automatc evaluato ad huma udgmets as follows: r 1 ( H H)( M M ) ( H ) ( ) 1 H M 1 M (9) where H ad M are the -th system scores of huma udgmet ad that of the automatc evaluato metrcs, respectvely. The system-level correlato for the three metrcs s gve Table 3 ad Table 4, from the tables, we foud that the system-level correlato of METEOR-MPEDA metrc s better tha ME- TEOR, METEOR-Uversal ad METEOR- Markov o average. Furthermore, Kedall s τ coeffcet was used to compute the correlato betwee automatc evaluato metrcs ad huma udgmets at segmet -level as follows: Cocordat Dscordat = Cocordat Dscordat (10) where Cocordat deotes the set where the huma udgmet ad the automatc evaluato metrcs score are cocordat, whle Dscordat deotes the set where they are dscordat. The segmet-level correlato s gve Table 5 ad 6. It ca be observed that the segmetlevel correlato of METEOR-MPEDA metrc o evaluato traslato to Eglsh tasks s better tha METEOR, METEOR-Uversal metrc ad METEOR-Markov metrc o average. However, whe evaluatg traslato out of Eglsh tasks, the performace of the METEOR- MPEDA metrc s slghtly lower tha METEOR- Uversal metrc. It ca be explaed that whe we have a large amout of blgual parallel trag data, the paraphrase table extracted from the blgual corpus s better tha that from moolgual corpus for automatc evaluato of MT. Metrcs de-e cs-e fr-e f-e ru-e Average METEOR METEOR-Uversal METEOR-Markov METEOR-MPEDA Table 3. The system-level correlato of metrcs o evaluato traslato to Eglsh o WMT15 Metrcs task Metrcs e-de e-cs e-fr e-f e-ru Average METEOR METEOR-Uversal METEOR-Markov METEOR-MPEDA Table 4. The system-level correlato of metrcs o evaluato traslato out of Eglsh o WMT15 Metrcs task Metrcs de-e cs-e fr-e f-e ru-e Average METEOR METEOR-Uversal METEOR-Markov METEOR-MPEDA Table 5. The segmet-level correlato of metrcs o evaluato traslato to Eglsh o WMT15 Metrcs task Metrcs e-de e-cs e-fr e-f e-ru Average METEOR METEOR-Uversal METEOR-Markov METEOR-MPEDA Table 6. The segmet-level correlato of metrcs o evaluato traslato out of Eglsh o WMT15 Metrcs task 515

6 5 Cocluso I ths paper, we descrbe the submssos of our metrc for WMT16 Metrcs task detal. We propose a approach to extract doma-specfc paraphrase table from moolgual corpus for automatc evaluato of MT, ad use t to replace the orgal paraphrase table METEOR metrc to mprove the correlato betwee huma udgmet ad automatc evaluato metrcs. The proposed approach s tested o the ewswre doma. I future work, we wll systematcally apply t to dfferet specfc domas such as the medcal doma, IT doma, etc. Ackowledgmets Ths research has bee fuded by the Natural Scece Foudato of Cha uder Grat No , , , ad , ad supported by the Natural Scece Foudato of Jagx Provcal Departmet of Scece ad Techology of Cha uder Grat No 0151BAB0705, ad also supported by the Natural Scece Foudato of Jagx Educatoal Commttee of Cha uder Grat No. GJJ Refereces Amtta Axelrod, Xaodog He ad Jafeg Gao, 011. Doma Adaptato va Pseudo I-Doma Data Selecto. Proceedgs of the 011 Coferece o Emprcal Methods Natural Laguage Processg, pages , Edburgh, Scotlad, UK. Sataeev Baeree ad Alo Lave, 005. METEOR: A Automatc Metrc for MT Evaluato wth Improved Correlato wth Huma Judgmets. Proceedgs of the ACL Workshop o Itrsc ad Extrsc Evaluato Measures for Mache Traslato ad/or Summarzato, pages 65-7, A Arbor. Col Baard ad Chrs Callso-Burch, 005. Paraphrasg wth Blgual Parallel Corpora. Proceedgs of the 43rd Aual Meetg of the Assocato for Computatoal Lgustcs, pages , A Arbor, Mchga. Rega Barzlay ad Kathlee R. McKeow, 001. Extractg Paraphrases from a Parallel Corpus. Proceedgs of 39th Aual Meetg of the Assocato for Computatoal Lgustcs, pages 50-57, Toulouse, Frace. Rega Barzlay ad Llla Lee, 003. Learg to Paraphrase: A Usupervsed Approach Usg Multple-Sequece Algmet. Proceedgs of the 003 Huma Laguage Techology Coferece of the North Amerca Chapter of the Assocato for Computatoal Lgustcs, pages Odre Boar, Rae Chatteree, Chrsta Federma, Barry Haddow, Matthas Huck, Chrs Hokamp, Phlpp Koeh, Varvara Logacheva, Chrstof Moz, Matteo Negr, Matt Post, Carola Scarto, Luca Speca ad Marco Turch, 015. Fdgs of the 015 Workshop o Statstcal Mache Traslato. Proceedgs of the Teth Workshop o Statstcal Mache Traslato, pages 1-46, Lsbo, Portugal. Yee Seg Cha ad Hwee Tou Ng, 008. MAXSIM: A Maxmum Smlarty Metrc for Mache Traslato Evaluato. Proceedgs of the 46th Aual Meetg of the Assocato for Computatoal Lgustcs, pages 55-6, Columbus, Oho. Mchael Dekowsk ad Alo Lave, 014. Meteor Uversal: Laguage Specfc Traslato Evaluato for Ay Target Laguage. Proceedgs of the Nth Workshop o Statstcal Mache Traslato, pages George Doddgto, 00. Automatc Evaluato of Mache Traslato Qualty Usg N-gram Cooccurrece Statstcs. Proceedgs of the secod teratoal coferece o Huma Laguage Techology Research, pages , Sa Dego, Calfora, CA, USA. Maox L, Awe Jag ad Mgwe Wag, 013. Lstwse Approach to Learg to Rak for Automatc Evaluato of Mache Traslato. Proceedgs of Mache Traslato Summt XIV, pages 51-59, Nce, Frace. Maox L, Mgwe Wag, Hax L, Fa Xu, 016. Modelg Moolgual Character Algmet for Automatc Evaluato of Chese Traslato. ACM Trasactos o Asa ad Low-Resource Laguage Iformato Processg, 15(3), pages Robert C. Moore ad Wllam Lews, 010. Itellget Selecto of Laguage Model Trag Data. Proceedgs of the ACL 010 Coferece (Short Papers), pages 0-4, Uppsala, Swede. Kshore Pape, Salm Roukos, Todd Ward ad We-Jg Zhu, 00. BLEU: a Method for Automatc Evaluato of Mache Traslato. Proceedgs of the 40th Aual Meetg o Assocato for Computatoal Lgustcs, pages , Phladelpha, Pesylvaa. Elle Pavlck, Jur Gatkevtch, Tsz Pg Cha, Xuche Yao, Beam Va Durme ad Chrs Callso-Burch, 015. Doma-Specfc Paraphrase Extracto. Proceedgs of the 53rd Aual Meetg of the Assocato for Computatoal Lgustcs ad the 7th Iteratoal Jot Coferece o Natural Laguage Processg, pages 57 6, Beg, Cha. Yusuke Shyama, Satosh Seke ad Kyosh Sudo, 00. Automatc Paraphrase Acqusto from News Artcles. Proceedgs of the secod teratoal coferece o Huma Laguage Techology Research, pages Matthew Sover, Boe Dorr, Rchard Schwartz, Joh Makhoul, Lea Mcculla ad Ralph Ma- 516

7 khoul, 006. A Study of Traslato Edt Rate wth Targeted Huma Aotato. Proceedgs of Assocato for Mache Traslato the Amercas, pages 3-31, Cambrdge. Zhe Weg, Maox L, Mgwe Wag, 015. Ehace Automatc Evaluato of Mache Traslato by Markov Network Based Paraphrases ( Chese). Joural of Chese Iformato Processg, 9(6), pages