Automatic Measuring of English Language Proficiency using MT Evaluation Technology

Transcription

1 Aumaic Measuing f English Language Pficiency using MT Evaluain Technlgy Keiji Yasuda ATR Spken Language Tanslain Reseach Labaies Depamen f SLR Hikaidai, Keihanna Science Ciy Ky Japan keiji.yasuda@a.jp Fumiaki Sugaya KDDI R&D Labaies 2--5, Ohaa, Kamifukuka-ciy, Saiama, , Japan fsugaya@kddilabs.jp Eiichi Sumia ATR Spken Language Tanslain Reseach Labaies Depamen f NLR Hikaidai, Keihanna Science Ciy Ky Japan eiichi.sumia@a.jp Genichi Kikui ATR Spken Language Tanslain Reseach Labaies Depamen f SLR Hikaidai, Keihanna Science Ciy Ky Japan genichi.kikui@a.jp Absac Assising in feign language leaning is ne f he maj aeas in which naual language pcessing echnlgy can cnibue. This pape ppses a cmpueized mehd f measuing cmmunicaive skill in English as a feign language. The ppsed mehd cnsiss f w pas. The fis pa invlves a es senence selecin pa achieve pecise measuemen wih a small es se. The secnd pa is he acual measuemen, which has hee seps. Sep ne asks pficiency-knwn human subjecs anslae Japanese senences in English. Sep w gauges he mach beween he anslains f he subjecs and cec anslains based n he n-gam velap he edi disance beween anslains. Sep hee leans he elainship beween pficiency and mach. By egessin i finds a saigh-line fiing f he scae pl epesening he pficiency and maches f he subjecs. Then, i esimaes pficiency f pficiency-unknwn uses by using Tshiyuki Takezawa ATR Spken Language Tanslain Reseach Labaies Depamen f SLR Hikaidai, Keihanna Science Ciy Ky Japan shiyuki.akezawa@a.jp Seiichi Yamam ATR Spken Language Tanslain Reseach Labaies Hikaidai, Keihanna Science Ciy Ky Japan seiichi.yamam@a.jp he line and he mach. Based n his appach, we cnduced expeimens n esimaing he Tes f English f Inenainal Cmmunicain (TOEIC) sce. We clleced w ses f daa cnsising f English senences anslaed fm Japanese. The fis se cnsiss f 330 senences, each anslaed English by 29 subjecs wih vaied English pficiency. The secnd se cnsiss f 50 senences anslaed in a simila manne by a sepaae gup f 8 subjecs. We fund ha he esimaed sces celaed wih he acual sces. Inducin F effecive secnd language leaning, i is absluely necessay es pficiency in he secnd language. This esing can help in selecing educainal maeials befe leaning, checking leanes undesanding afe leaning, and s n. T make leaning efficien, i is impan achieve esing wih a sh unaund ime. Cmpue-based esing is ne sluin f his,

2 and seveal kinds f ess have been develped, including CASEC (CASEC, 2004) and TOEFL- CBT (TOEFL, 2004). Hweve, hese ess ae mainly based n clze esing muliple-chice quesins. Cnsequenly, hey equie labu css f expe examinain designes make he quesins and he alenaive deac answes. In his pape, we ppse a mehd f he aumaic measuemen f English language pficiency by applying aumaic evaluain echniques. The ppsed mehd selecs adequae es senences fm an exising cpus. Then, i aumaically evaluaes he anslains f es senences dne by uses. The ce echnlgy f he ppsed mehd, i.e., he aumaic evaluain f anslains, was develped in eseach aiming a he efficien develpmen f Machine Tanslain (MT) echnlgy (Su e al., 992; Papineni e al., 2002;, 2002). In he ppsed mehd, we apply hese MT evaluain echnlgies he measuemen f human English language pficiency. The ppsed mehd fcuses n measuing he cmmunicaive skill f sucuing senences, which is indispensable f wiing and speaking. I des n measue elemenay capabiliies including vcabulay gamma. This mehd als ppses a es senence selecin scheme enable efficien esing. Secin 2 descibes seveal aumaic evaluain mehds applied he ppsed mehd. Secin 3 induces he ppsed evaluain scheme. Secin 4 shws he evaluain esuls bained by he ppsed mehd. Secin 5 cncludes he pape. 2 MT Evaluain Technlgies In his secin, we biefly descibe aumaic evaluain mehds f anslain. These mehds wee ppsed evaluae MT upu, bu hey ae applicable anslain by humans. All f hese mehds ae based n he same idea, ha is, cmpae he age anslain f evaluain wih high-qualiy efeence anslains ha ae usually dne by skilled anslas. Theefe, hese mehds equie a cpus f high-qualiy human efeence anslains. We call hese anslains as efeences. 2. -based Mehd The sce beween a anslain upu and efeences can be calculaed by maching (Su e al., 992; Takezawa e al., 999). Fis, we define he sce beween senence (i.e., wd aay) W a and senence W b by he fllwing fmula. S (W a, W b ) = T S I D () T whee T is he al numbe f wds in W a, S is he numbe f subsiuin wds f cmpaing W a W b, I is he numbe f inseed wds f cmpaing W a W b, and D is he numbe f deleed wds f cmpaing W a W b. Using Equain, (S i (j)), ha is, he es senence uni -sce f he anslain f es senence j dne by subjec i, can be calculaed by he fllwing fmula. S i (j) = { } max S (W ef(k) (j), W sub(i) (j)), 0 k= N ef (2) whee N ef is he numbe f efeences, W ef(k) (j) is he k-h efeence f he es senence j, and W sub(i) (j) is he anslain f he es senence j dne by subjec i. Finally, S i, which is he es se uni sce f subjec i, can be calculaed by he fllwing fmula. S i = N sen S i (j) (3) N sen j= whee N sen is he numbe f es senences. 2.2 N-gam-based Mehd Papineni e al. (2002) ppsed, which is an aumaic mehd f evaluaing MT qualiy using N-gam maching. The Nainal Insiue f Sandads and Technlgy als ppsed an aumaic evaluain mehd called (2002), which is a mdified mehd f. In his eseach we use w kinds f unis apply and. One is a es senence uni and he he is a es se uni. The uni f ueance cespnds he uni f segmen in he iginal and sudies (Papineni e al., 2002;, 2002). Equain 4 is he es senence uni sce fmulain f he anslain f es senence j dne by subjec i. S i (j) = { N exp w n lg(p n ) max n= ( )} L ef, 0 L sys

3 whee (4) Cpus p n = C {Candidaes} w n = N and L ef C {Candidaes} n gam {C} Cun clip (n gam) n gam {C} Cun(n gam) = he numbe f wds in he efeence anslain ha is clses in lengh he anslain being sced L sys = he numbe f wds in he anslain being sced Equain 5 is he es senence uni sce fmulain f he anslain f es senence j dne by subjec i. S i (j) = whee Nn= { all w...wn in sys upu inf(w...w n) exp all w...wn in sys upu () [ ( )]} {β lg 2 L min sys, L ef inf(w... w n ) = ( ) lg he numbe f ccuence f w...w n 2 he numbe f ccuence f w...w n } (5) L ef = he aveage numbe f wds in a efeence anslain, aveaged ve all efeence anslains L sys = he numbe f wds in he anslain being sced and β is chsen make he beviy penaly fac=0.5 when he numbe f wds in he sysem anslain is 2/3 f he aveage numbe f wds in he efeence anslain. F Equains 4 and 7, N indicaes he maximum n-gam lengh. In his eseach we se N 4 f and 5 f. We may cnside he uni f he es se cespnding he uni f dcumen sysem in and. Hweve, we use fmulains f he es se uni sces ha ae diffeen fm hse f he iginal and. Japanese es se Refeences anslaed by bilinguals English wiing by pficiency-knwn human subjecs English senences by pficiency Aumaic evaluain (senence uni evaluain) Calculae celain beween TOEIC sce and senence uni aumaic sce Selec es senences based n celain Figue : Flw f Tes Se Selecin The es se uni sces f and ae calculaed by Equains 6 and 7. S i = N sen S i (j) (6) N sen j= S i = N sen S i (j) (7) N sen j= 3 The Ppsed Mehd The ppsed mehd descibed in his pape cnsiss f w pas. One is he es se selecin pa and he he is he acual measuemen pa. The measuemen pa is divided in w phases: a paamee-esimain phase and a esing phase. Hee, we use he em subjecs efe he human subjecs in he es se selecin pa and he paamee-esimain phase f he measuemen pa; we use uses efe he humans in he esing phase f he measuemen pa.

4 Cpus Japanese es se Refeences anslaed by bilinguals English wiing by pficiency-knwn human subjecs English wiing by a use English senences English senences by pficiency Aumaic evaluain Aumaic evaluain (Tes se uni evaluain) Aumaic sce Regessin analysis using pficiency and aumaic sces Regessin cefficien <Paamee-esimain phase> Esimain f English pficiency English pficiency <Tesing phase> Figue 2: Flw f English Pficiency Measumen We emply he Tes f English f Inenainal Cmmunicain (TOEIC, 2004) as an bjecive measue f English pficiency. 3. Tes Senence Selecin Mehd Figue shws he flw f he es senence selecin. We fis calculae he es senence uni aumaic sce by using Equain 2, 4 5 f each es senence and subjec. Secnd, f each es senence, we calculae he celain beween he aumaic sces and subjecs TOEIC sces. Finally, using he abve esuls, we chse he es senences ha give high celain. 3.2 Mehd f Measuing English Pficiency Figue 2 shws he flw f measuing English pficiency. In he paamee-esimain phase, f each subjec, we fis calculae he es se uni aumaic sce by using Equain 3, 6 7. Nex, we apply egessin analysis using he aumaic sces and subjecs TOEIC sces. In he esing phase, we calculae a use s TOEIC sce using he aumaic sce f he use and he egessin line calculaed in he paamee-esimain phase. 4 Expeimens 4. Expeimenal Cndiins 4.. Tes ses F he expeimens, we emply w diffeen es ses. One is BTEC (Basic Tavel Expessin Cpus) (Takezawa e al., 2002) and he he is SLTA (Takezawa, 999). Bh BTEC and SLTA ae pas f bilingual cpa ha have been clleced f eseach n speech anslain sysems. Hweve, hey have diffeen feaues. A deailed analysis f hese cpa was dne by Kikui e al. (2003). Hee, we biefly explain hese es ses. In his sudy, we use he Japanese side as a es se and he English side as a efeence f aumaic evaluain. BTEC BTEC was designed cve expessins f evey penial subjec in avel cnvesain. This es se was clleced by invesigaing

5 phasebks ha cnain Japanese/English senence pais ha expes cnside useful f uiss aveling abad. One senence cnains 8 wds n aveage. The es se f his expeimen cnsiss f 50 senences fm he BTEC cpus. The al numbe f examinees is 8, and he ange f hei TOEIC sces is beween he 400s and 900s. Evey hunded-pin ange has 3 examinees n 0.2 i la e C SLTA SLTA cnsiss f 330 senences in 23 cnvesains fm he ATR bilingual avel cnvesain daabase (Takezawa, 999). One senence cnains 3 wds n aveage. This cpus was clleced by simulaed dialgues beween Japanese and English speakes hugh a pfessinal inepee. The pics f he cnvesains ae mainly hel cnvesains, such as esevains, enquiies and s n. The al numbe f examinees is 29, and he ange f hei TOEIC sce is beween he 300s and 800s. Excluding he 600s, evey hundedpin ange has 5 examinees Refeence F he aumaic evaluain, we clleced 6 efeences f each es senence. One f hem is fm he English side f he es se, and he emaining 5 wee anslaed by 5 bilinguals (3 efeences by bilingual). 4.2 Expeimenal Resuls 4.2. Expeimenal Resuls f Tes Se Selecin Figues 3 and 4 shw he celain beween he es senence uni aumaic sce and he subjecs TOEIC sce. Hee, he aumaic sce is calculaed using Equain 2, 4 5. Figue 3 shws he esuls n BTEC, and Fig. 4 shws he esuls n SLTA. In hese figues, he dinae epesens he celain. The filled cicles indicae he esuls using he -based aumaic evaluain mehd. The gay cicles indicae he esuls using. The empy cicles indicae he esuls using. Lking a hese figues, we find ha he hee aumaic evaluain mehds shw a simila endency. Cmpaing BTEC and SLTA, BTEC cnains me cumbesme es senences. In BTEC, abu 20% f he es senences give a celain f less han 0. Meanwhile, in he SLTA, his pecenage is abu 0% Tes senence (sed by celain) Figue 3: Celain beween es senence uni aumaic sces and subjecs TOEIC sces (BTEC) Table shws examples f lw-celaed es senences. As shwn in he able, BTEC cnains me sh and fequenly used expessins han des SLTA. This kind f expessin is hugh be easy f esing, s his lw-celain phenmenn is hugh ccu. SLTA sill cnains a few senences f his kind ( Example f SLTA in he able). Addiinally, hee is anhe cnibuing fac explaining he lw celain in SLTA. Lking a Example 2 f SLTA in he able, his expessin is n vey easy anslae. F his es senence, seveal expessins can be pduced as an English anslain. Thus, aumaic evaluain mehds cann evaluae cecly due he insufficien vaiey f efeences. Cnsideing hese esuls, his mehd can emve inadequae es senences due n nly he easiness f he es senence bu als he difficuly f he aumaic evaluain. Figues 5 and 6 shw he elainship beween he numbe f es senences and celain. This celain is calculaed beween he es se uni aumaic sces and he subjecs TOEIC sces. Hee, he aumaic sce is calculaed using Equain 3, 6 7. Figue 5 shws he esuls n BTEC, and Fig. 6 shws he esuls n SLTA. In hese figues, he abscissa epesens he numbe f es senences, i.e., N sen in Equains 3, 6 and 7, and he dinae epesens he celain. Definiins f he cicles ae he same as hse in he pevius figue. Hee, he es senence selecin is based n he celain shwn in Figs. 3 and 4. Cmpaing Fig. 5 Fig. 6, in he case f

6 Table : Example f lw-celaed es senences C E T B Example おやすみなさい Example 2 メニューを見せてください Japanese Gd nigh. Can I see a menu, please? English A T L S Example はい マスターカードでお願いします Example 2 それでお願いしたいんですが 予算の都合もありますんでとりあえずスイートの料金を教えてください Yes, wih my Masecad please I wish I culd ake ha bu we have a limied budge s hw much will ha cs? n 0.2 i la e C n0.85 i la e 0.8 C Tes senence (sed by celain) Numbe f es senences Figue 4: Celain beween es senence uni aumaic sces and subjecs TOEIC sces (SLTA) Figue 6: Celain beween es se uni aumaic sces and subjecs TOEIC sces (SLTA) n i la e 0.8 C e d 200 a d n a S Numbe f es senences Figue 5: Celain beween es se uni aumaic sces and subjecs TOEIC sces (BTEC) using he full es se (50 es senences f BTEC, 330 es senences f SLTA), he celain f BTEC is lwe han ha f SLTA. As we menined abve, he ai f he lwcelaed es senences in BTEC is highe han ha f SLTA (See Figs. 3 and 4). This issue is hugh cause a decease in he celain shwn in Fig. 5. Hweve, by applying he se Numbe f es senences Figue 7: Sandad e (BTEC) lecin based n senence uni celain, hese bsucive es senences can be emved. This pemis he selecin f high-celaed smallsized es ses. In hese figues, he highes celains ae aund Expeimenal Resuls f English Pficiency Measuemen F he expeimens n English pficiency measuemen, we caied u a leave-ne-u css validain es. The leave-ne-u css valida-

7 e d 200 a d n a S Numbe f es senences Figue 8: Sandad e (SLTA) in es is cnduced n nly f he measuemen f he English pficiency bu als f he es se selecin. T evaluae he pficiency measuemen by he ppsed mehd, we calculae he sandad e f he esuls f a leave-ne-u css validain es. The fllwing fmula is he definiin f he sandad e. σ E = N use N use i= (T i A i ) 2 (8) whee N use is he numbe f uses, T i is he acual TOEIC sce f use i, and A i is use i s esimaed TOEIC sce by using he ppsed mehd. Figues 7 and 8 shw he elainship beween he numbe f es senences and he sandad e. In hese figues, he abscissa epesens he numbe f es senences, and he dinae epesens he sandad e. Definiins f he cicles ae he same as in he pevius figue. Hee, he es senence selecin is based n he celain shwn in Figs. 3 and 4. Lking a Figs. 7 and 8, we can bseve diffeences beween he sandad es f BTEC and SLTA. This is hugh be due he diffeence f he numbe f subjecs in he expeimens (f he leave-ne-u css validain es, 7 subjecs wih BTEC and 28 subjecs wih SLTA). Even hugh hese wee clsed expeimens, he esuls in Figs. 5 and 6 shw an even highe celain wih BTEC han wih SLTA a he highes pin. Theefe, hee is m f impvemen by inceasing he numbe f subjecs wih BTEC. In he es using es senences in Figs. 7 and 8, he sandad es ae much smalle han in he es using he full es se (50 es senences f BTEC, 330 es senences f SLTA). These esuls imply ha he es se selecin wks vey well and ha i enables pecise esing using a smalle size es se. 5 Cnclusin We ppsed an aumaic measuemen mehd f English language pficiency. The ppsed mehd applies aumaic MT evaluain measue human English language pficiency. This mehd fcuses n measuing he cmmunicaive skill f sucuing senences, which is indispensable in wiing and speaking. Hweve, i des n measue elemenay capabiliies such as vcabulay and gamma. The mehd als invlves a new es senence selecin scheme enable efficien esing. In he expeimens, we used TOEIC as an bjecive measue f English language pficiency. We hen applied sme cuenly available aumaic evaluain mehds:, and a -based mehd. We caied u expeimens n w es ses: BTEC and SLTA. Accding he expeimenal esuls, he ppsed mehd gave a gd measuemen esul n a small-sized es se. The sandad e f measuemen is aund 20 pins n he TOEIC sce wih BTEC and less han 00 TOEIC pins sce wih SLTA. In bh cases, he pimum size f he es se is es senences. The ppsed mehd sill needs human labu make he efeences. T bain highe pabiliy, we will apply an aumaic paaphase scheme (Finch e al., 2002; Shimhaa and Sumia, 2002) make he efeences aumaically. 6 Acknwledgemens The eseach eped hee was supped in pa by a cnac wih he Nainal Insiue f Infmain and Cmmunicains Technlgy eniled A sudy f speech dialgue anslain echnlgy based n a lage cpus. Refeences CASEC Cmpue Assessmen Sysem f English Cmmunicain. hp:// A. Finch, T. Waanabe, and E. Sumia Paaphasing by Saisical Machine Tanslain. In Pceedings f he s Fum n

8 Infmain Technlgy (FIT2002), vlume E-53, pages G. Kikui, E. Sumia, T. Takezawa, and S. Yamam Ceaing Cpa f Speech--Speech Tanslain. In Pceedings f EUROSPEECH, pages Aumaic Evaluain f Machine Tanslain Qualiy Using N-gam C-Occuence Saisics. hp:// /m200/esuce/. K. Papineni, S. Ruks, T. Wad, and W.- J. Zhu Bleu: a mehd f aumaic evaluain f machine anslain. In Pceedings f he 40h Annual Meeing f he Assciain f Cmpuainal Linguisics (ACL), pages M. Shimhaa and E. Sumia Aumaic Paaphasing Based n Paallel Cpus f Nmalizain. In Pceedings f Inenainal Cnfeence n Language Resuces and Evaluain (LREC), pages K.-Y. Su, M.-W. Wu, and J.-S. Chang A new quaniaive qualiy measue f machine anslain sysems. In Pceedings f he 4h Inenainal Cnfeence n Cmpuainal Linguisics(COLING), pages T. Takezawa, F. Sugaya, A. Yk, and S. Yamam A new evaluain mehd f speech anslain sysems and a case sudy n ATR-MATRIX fm Japanese English. In Pceeding f Machine Tanslain Summi (MT Summi), pages T. Takezawa, E. Sumia, F. Sugaya, H. Yamam, and S. Yamam Twad a Bad-Cveage Bilingual Cpus f Speech Tanslain f Tavel Cnvesains in he Real Wld. In Pceedings f Inenainal Cnfeence n Language Resuces and Evaluain (LREC), pages T. Takezawa Building a bilingual avel cnvesain daabase f speech anslain eseach. In Pceedings f he 2nd Inenainal Wkshp n Eas-Asian Language Resuces and Evaluain Oienal CO- COSDA Wkshp 99, pages TOEFL Tes f English as a Feign Language. hp:// TOEIC Tes f English f Inenainal Cmmunicain. hp://