Building a Bilingual Dictionary with Scarce Resources: A Genetic Algorithm Approach

Transcription

1 Buldng a Blngual Dctonary wth carce Resources: A Genetc Algorthm Approach Benamn Han Language Technologes Insttute, Carnege Mellon Unversty 5000 Forbes Avenue Pttsburgh, PA , UA benhd@cs.cmu.edu Abstract Current corpus-based machne translaton systems usually requre sgnfcant amount of parallel text to buld a useful blngual dctonary for translaton. To allevate ths data dependency I propose a novel approach based on genetc algorthms to mprove translatons by fusng dfferent lngustc hypotheses. A prelmnary evaluaton s also reported. Introducton Most of the current corpus-based machne translaton systems rely on statstcal methods to extract blngual dctonares. Whle these methods, such as [Kay and Röschesen 993], K-vec and DK-vec [Fung and McKeown 994, 997], and [Brown 997], presume lttle or no pror knowledge of source languages, they requre sgnfcant amount of parallel text to buld an accurate blngual dctonary [Jones and omers 995] [omers and Ward 996] [Haruno and Yamazak 996]. The requrement makes these approaches less desrable when lttle data of the source languages could be obtaned. The lack of parallel text presents one aspect of the data scarcty problem [Al-Onazan 2000]. Beng able to solve the problem has both theoretcal and practcal values. On the one hand an effectve approach could shed lght on the theoretcal framework requred for buldng general lexcal acquston systems, on the other t could facltate accessng nformaton expressed n mnorty or ndgenous languages. Obvously the problem cannot be remeded wthout help. As proposed n [Kumano and Hrakawa 994], [Utsuro et al. 994], [Haruno and Yamazak 996] and [Brown 999], ncorporatng more lngustc knowledge could be a promsng soluton. In ths paper t has motvated the proposal of a novel approach based on genetc algorthms (GA) [Holland 975, Goldberg 989] as a way to fuse dfferent lngustc hypotheses. The paper s organzed as follows: n ecton a vew of translatng by optmzaton together wth the two workng hypotheses are ntroduced. ecton 2 presents the detals of the algorthm, and ecton 3 reports the prelmnary evaluaton. The paper s then concluded wth Future Works. Translatng by Optmzaton Let f: W W 2 be a translaton mappng, where W /W 2 represents the words of the source/target language (L/L2), a good translaton f should maxmze an obectve (f). Vewng translaton as an optmzaton problem allows a straghtforward ncorporaton of dfferent nformaton. In ths paper the functon s determned based on the two proposed lngustc hypotheses, namely, the smlarty of localty and part-of-speech (PO) dstrbutons across the two languages. More specfcally, the functon over a translaton f s defned as f ) = λ ( f, s ) + ( λ) ( f, ) () ( L P s whch s the sum of a lnear combnaton of L (f, s ) and P (f, s ) the s contrbuted by the localty and the PO hypotheses, respectvely, over an L sentence s. It s worth notng that for dfferent language pars, e.g., from a casemarkng language (e.g. Russan) to a confguratonal language (e.g. Englsh), the λ s a constant set to 0.5 n the evaluaton.

2 hypotheses used here mght not be approprate and new hypotheses of dfferent types must be proposed due to the dstnct natures of the lngustc mappngs between the languages. The two hypotheses are descrbed below.. Localty Hypothess The localty hypothess stpulates that a good translaton f should map two words n an L sentence nto a par of words wth smlar word dstance n the L2 sentence. Let w, w W, a preferable f has the followng property: Dst( w, w Dst( f ( w ), f ( w ) )) (2) where Dst(.,.) denotes the dstance between two words. Note t does not presume a strct word order smlarty snce only the relatve dstances of words are taken nto account. For an L sentence wth n words Equ. (2) mples a O(n 2 ) tme for evaluatng f over the sentence. To save tme an approxmaton s adopted where only every k+-th word n s s consdered. The L s then formulated as 2 L ( f, s ) = s k = κ k Dst mn s k ( f ( w ), f ( w κ + k... (3) where Dst mn (.,.) returns the mnmum dstance between a par of words, κ = max(k, t -k) s a normalzng constant, and t s the translated sentence n L2. Ths s smply an average of the relatve dfferences of the word dstances between the L sentence s and ts translaton t n L2..2 Part-of-speech (PO) mlarty Hypothess The second hypothess adopted s that a good translaton should preserve the PO dstrbuton. Let the PO dstrbuton of a word w, D(w), be a vector of PO confdence values,.e., D(w) = {c t 0 c t s the confdence that w s of PO t} 2 For the current mplementaton k=. )) The PO dstrbuton of a sentence s can then be defned as D ( s) = D( w ) w s And the P s defned as where f, s ) = Dst ( D( f ( s )), D( t ))... (4) P ( P Dst ( D, D P 2 D D2 π ) = acos( ) s the D D 2 normalzed angle between the two dstrbuton vectors. Note snce both f(s ) and t are n L2, only the PO nformaton of L2 s requred. For translaton for assmlaton ths s usually not a problem snce n these scenaros the PO nformaton of L2 s usually avalable 3. 2 A Genetc Algorthm Approach The GA-based approach s proposed to solve the translaton optmzaton problem based on the followng observatons: (a) the search space s huge and not well understood, and (b) the problem satsfes the buldng block hypothess n that a good translaton possesses many useful smaller buldng blocks whch can be exchanged wth the others to potentally yeld a better translaton [Goldberg 989]. Untl recently GA has not been wdely used n the feld of computatonal lngustcs. everal works have been reported for grammar nducton, robust parsng, anaphora resoluton and morphologcal analyss [Losee 995] [Rosé 998] [Orasan et al 2000] [Kazakov and Manandhar 2000]. To the author s knowledge there has not been any report on usng GA-based technques to extract blngual dctonares. 2. Algorthm Outlne The algorthm apples varous GA operators on a populaton of solutons to maxmze the obectve functon (f). A soluton (translaton mappng) s encoded by a vector v where v = denotes the translaton f(w ) = x. A sketch of the 3 For the current mplementaton Brll s Transformaton-based PO Tagger [Brll 992] s used for L2 (Englsh). 2

3 proposed steady-state genetc algorthm wth least-ft-deleton strategy s descrbed below:. For each w W, teratvely compute a canddate set C(w ) contanng the possble translatons wth ther respectve confdence values (descrbed below). 2. Intalze a populaton of solutons by randomly pckng a canddate translaton for each w accordng to the confdence dstrbutons 4, and addng a soluton contanng the canddates wth the hghest confdence values 5. Evaluate all of the solutons accordng to Equ. (), (3) and (4) and lnearly scale the s nto the ftness values. 3. Randomly pck a GA operator accordng to the operator ftness dstrbuton. One or two solutons are then randomly selected accordng to the soluton ftness dstrbuton, and the GA operator s appled on them to generate new solutons. 4. The new solutons are evaluated agan by Equ. (), (3) and (4), and the operator ftness values are adapted accordng to the performances of the new solutons. The worst solutons are then replaced by the new ones. 5. A complete run of tep 3 and 4 s called an epoch. Run a certan number of epochs or stop when the of the best soluton exceeds a preset threshold. The followng sub-sectons gve the rest of the detals of the algorthm. 2.2 Iteratve Canddate et Computaton For a word w W and x W 2, we frst defne and to be the set of sentences where w w x and x occurs respectvely. The coverage of x wth respect to w and vce versa are then computed by 4 The same random selecton operaton, roulettewheel selecton, whch pcks a selectee randomly accordng to a gven dstrbuton, s adopted n the entre experment. 5 Ths s to ensure that the GA mproves upon the ntal best soluton. C( w, x ) = w w x x x w C( x, w ) =... (5) The n-th teratve confdence that x s the correct translaton of w s then defned recursvely as 6 cn ( w, x ) cn ( x, w ) c n ( w, x ) =, c ( w, x ) c ( x, w ) c ( w, x n cn ( w, xk ) 0 k n ) = C( w, x ) C( x, w )... (6) Fnally for w we rank x n descendng order accordng to c n (w,x ), and form the canddate set C(w ) by only takng the top k (cutoff value) L2 words n the rankng lst 7. The ntuton behnd Equ (5) & (6) s that a preferable canddate x for w should have a hgher rankng n C(w ) and vce versa. Ths prevents a hgh-frequency L2 word from beng a preferable translaton for too many L words. 2.3 Genetc Operators Three GA operators are adopted: crossover, mutaton and creep. Each of them has ts own ftness value, whch s then used n selectng one operator n each epoch so that an operator wth hgher ftness value wll on average be pcked more frequently. To acheve greater autonomy and more dynamc system response the ftness values are adapted based on the dea of [Davs 989], although the realzaton s somewhat dfferent. In each epoch after evaluatng the new solutons, a reward proportonal to ther mprovements over the best soluton n the populaton s credted to the responsble operator, and a proporton of the reward s propagated back to the operator generatng the parent(s), and to the operator generatng the grandparent(s), etc., untl we reach out of a preset hstory wndow. 6 For the experment results reported here n=2. 7 For the experment results reported here k=0. k

4 The crossover operator takes two solutons, randomly pcks a crossover pont and swaps the sub-solutons between the two up to the pont. The mutaton operator randomly pcks an L word and changes the current translaton to another L2 canddate accordng to the canddate confdence dstrbuton. These two operators are adopted n most of GA mplementatons, and contrbute to the search process by combnng the potentally useful buldng blocks and randomly explorng the search space, respectvely. The thrd operator, creep, locally optmzes a soluton over a randomly chosen sentence. The new soluton has the local alteraton ncorporated f t mproves the obectve over the sentence, and the result of ths perturbaton s subsequently measured by Equ. (), (3) and (4). The ntroducton of the creep operator s based on observng how a human tres to decode words usng a small parallel corpus [Al-Onazan 2000]. It s usually done by comng up wth a set of translaton hypotheses upon observng the correspondences between an L sentence and ts correspondng L2 sentence. The hypotheses are then tested aganst the rest of the corpus. The obectve functon to be optmzed by the creep operator s the translaton over a partcular sentence s, namely ( L P f, s ) = λ ( f, s ) + ( λ) ( f, s ). To make the search problem tractable ths s done by a lmted depth-frst beam search: for each word w, only 3 possble translatons randomly pcked from C(w ) are searched 8, and only the frst 500 or 50% of the total dfferent sentence translatons are searched Ftness calng To avod premature populaton convergence the ftness values are computed by lnearly scalng the soluton s followng the suggeston n [Goldberg 989]. The ftness value F(f) of a 8 Agan they are selected accordng to the canddate confdence dstrbuton. 9 Whchever smaller. soluton f s computed by F(f) = a (f) + b, where the scalng coeffcents a and b are computed by solvng the lnear equatons 0 = a + b c = a max + b hould they fal the coeffcents are then obtaned from solvng the followng equatons: = a + b F mn = a mn + b where F mn s a parameter. If these fal agan the populaton s fully converged and we gve up scalng by settng a =.0 and b = Experments and Results 2 In order to evaluate the effectveness of the approach wth lmted data, the experments were conducted on three small corpora of dfferent szes (corpus C C3), taken from the pansh and Englsh portons of the UN Multlngual Corpus [Graff and Fnch 994]. The statstcs of each corpus together wth the populaton sze used n the tranng sesson s shown below. # of sentence pars pansh lexcon sze Englsh lexcon sze Pop. sze C C C Table. tatstcs for corpus C, C2 and C3 In all of the three tranng sessons the ntal ftness values for the three GA operators were set equal (/3), and,000,000 epochs were run for each corpus. After the tranng sessons 400 pansh words were randomly pcked from C3 and ther translatons were gven accordng to the best soluton before and after tranng usng 0 Constant c s usually set n [.2,2]. For the current mplementaton c=.2. For the the current mplementaton F mn = All of the experments were run on a Lnux machne wth AMD 700Mhz CPU and 256MB RAM.

5 Int. Int. per sentence Fnal Fnal per sentence Int. Word Precson Int. PO Precson Fnal Word Precson Fnal PO Precson C % 60% 49% 62% C % 65% 50% 67% C % 63% 54% 68% Table 2. Translaton accuraces before and after tranng, wth recall=00%; the max. precson s shown n bold typeface each corpus. The results were then graded by natve pansh speakers, who were gven the translatons and the actual parallel sentence pars to udge the correctness of the translatons. The results are summarzed n Table 2. The ntal best obectve and the persentence for each corpus are shown n the st and the 2 nd column, whle the fnal cores are shown n the 3 rd and 4 th column. The ntal s represent the pre-ga, pure statstcal translaton precsons acheved by the teratve canddate set computaton (see ecton 2.2). Comparng the ntal and the fnal s ndcates that the GA-based approach dd perform better, although the advantages seemed to keep decreasng wth the corpus sze. Ths mght be due to the nsuffcent runnng epochs when the corpus sze grows larger. The rest of Table 2 shows the evaluaton results gven by the natve speaker graders. At recall level 00% for all corpora, both of the fnal word and PO precsons are consstently hgher than the pre-ga ones although the mprovements are not sgnfcant (the most sgnfcant mprovement s about 6.5%). In addton to the premature termnaton of the tranng processes mentoned earler, one possble explanaton for the relatvely modest mprovements s that the random selecton of a canddate n the mutaton and creep operators s based on the canddate confdence dstrbuton, whch bases strongly toward the ntal best soluton. Another possble reason mght be that the hypotheses were not reflectng perfectly the real lngustc smlartes between these two languages. It has become apparent durng the evaluaton process that the system could take advantage of the observed lngustc constrants between these two languages, e.g., renforcng the patterns where a pansh adectve s translated behnd a pansh nomnal head. Obvously any more specfc hypotheses would rsk the generalty of the approach. As a comparson wth a conventonal statstcal approach, the experment reported n [Brown 997] used the same pansh-englsh UN corpus, but nstead of usng only a small porton the author used the entre corpus for tranng (total 685,000 sentence pars wth 96,793 unque pansh words). The best precson acheved at recall = 4.92% was 7%, but at the hghest recall level (38.07%) the precson was 54%. Future Works In ths paper I addressed the problem of translatng wth scarce resources, and demonstrated that by vewng translaton as optmzaton, useful nformaton could be fused to yeld better translatons. In partcular, two lngustc hypotheses localty and PO smlarty were postulated, and a GA-based technque was developed to solve the optmzaton problem. A prelmnary evaluaton based on the three small pansh-englsh corpora s also reported However there are several problems that need to be addressed n order to make the technque fully practcal. nce the approach optmzes translaton mappngs sentence by sentence, as the sze of the corpus grows the runnng tme becomes unacceptable. The problem could be allevated by only optmzng over the most dscrmnatng sentences. Another lmtaton of the approach s the overly smplfed representaton used to represent lexcal entres. There are no phrasal terms, no polysemous words, and no lngustc constrants between

6 words are learned (e.g., verb subcategorzaton). The problem must be overcome n order to make the machne-generated lexcon more useful wthn a complete translaton system. Fnally, as cued n the prevous secton, t remans to be seen f a more accurate ftness functon and a set of more robust GA operators can be dscovered. In the realm of machne translaton ths mght translate nto fndng the commonaltes between a specfc par, or even any par of languages, and a set of more unversal operators for transformng word tokens of L nto those of L2. References Al-Onazan, Y., Germann, U., Hermakob, U., Knght, K., Koehn, P., Marcu, D. and Yamada, K. (2000) Translatng wth carce Resources. The 7 th Natonal Conference of the Amercan Assocaton for Artfcal Intellgence (AAAI-2000), Austn, Texas. Brll, E. (992) A smple rule-based part of speech tagger. In Proceedngs of the Thrd Conference on Appled Natural Language Processng, ACL.. Brown, R. (997) Automated Dctonary Extracton for Knowledge-Free Example-Based Translaton". In Proceedngs of the eventh Internatonal Conference on Theoretcal and Methodologcal Issues n Machne Translaton, pp. -8. anta Fe. Brown, R. (999) Addng Lngustc Knowledge to a Lexcal Example-Based Translaton ystem. In Proceedngs of the Eghth Internatonal Conference on Theoretcal and Methodologcal Issues n Machne Translaton, pp Chester, UK. Davs, L. (989) Adaptng Operator Probabltes n Genetc Algorthms. In Proceedngs of the Thrd Internatonal Conference on Genetc Algorthms, pp. 6-69, Morgan Kaufmann. Fung, P. and McKeown, K. (994) Algnng Nosy Parallel Corpora Across Language Groups: Word Par Feature Matchng by Dynamc Tme Warpng. In Proceedngs of the Conference on Theoretcal and Methodologcal Issues n Machne Translaton, pp Fung, P. and KcKeown, K. (997) A Techncal Word- and Term-Translaton Ad Usng Nosy Parallel Corpora Across Language Groups. Machne Translaton, Vol. 2, Nos. -2, pp Goldberg, D. E. (989) Genetc Algorthms n earch, Optmzaton, and Machne Learnng. Addson-Wesley. Graff, D. and Fnch, R. (994) Multlngual Text Resources at the Lngustc Data Consortum. In Proceedngs of the 994 ARPA Human Language Technology Workshop. Morgan Kaufmann. Holland, J. H. (975) Adaptaton n Natural and Artfcal ystems, Unversty of Mchgan Press. Haruno, M. and Yamazak, T. (996) Hgh-Precson Blngual Text Algnment Usng tatstcal and Dctonary Informaton. In Proceedngs of Annual Conference of the Assocaton for Computatonal Lngustcs, pp Jones, D. and omers, H. (995) Blngual Vocabulary Estmaton from Nosy Parallel Corpora Usng Varable Bag Estmaton. In JADT III Gornale Internazonale d Anals tatstca de Dat Testual, pp , Rome. Kay, M. and Röschesen, M. (993) Text-Translaton Algnment. Computatonal Lngustcs, Vol. 9, No, pp Kazakov, K. and Manandhar,. (2000) Unsupervsed Learnng of Word egmentaton Rules wth Genetc Algorthms and Inductve Logc Programmng. To appear n Journal of Machne Learnng. Kumano, A. and Hrakawa, H. (994) Buldng an MT Dctonary from Parallel Texts Based on Lngustc and tatstc Informaton. In Proceedngs of Internatonal Conference on Computatonal Lngustcs, pp. 76-8, Kyoto. Losee, R. M. (995) Learnng yntactc Rules and Tags wth Genetc Algorthms for Informaton Retreval and Flterng: An Emprcal Bass for Grammatcal Rules. In Informaton Processng and Management. Orasan, C., Evans, R. and Mtkov, R. (2000) Enhancng Preference-Based Anaphora Resoluton wth Genetc Algorthms. In Chrstodoulaks (ed.) Proceedngs of Natural Language Processng (NLP 2000), pp Rosé, C. P. and A. Lave (998) A Doman Independent Approach for Effcently Interpretng Extragrammatcal Utterances. In Journal of Natural Language Engneerng, () pp omers, H. and Ward, A. (996) ome More Experments n Blngual Text Algnments. In Oflazer, K. and omers, H. (eds) Proceedngs of the econdinternatonal Conference on New Methods n Language Methods n Language Processng, pp , Ankara. Utsuro, T. et al. (994) Blngual Text Matchng Usng Blngual Dctonary and tatstcs. In Proceedngs of Internatonal Conference on Computatonal Lngustcs, pp , Kyoto.