Information Gain Ratio meets Maximal Marginal Relevance A method of Summarization for Multiple Documents

Transcription

1 Pocding of th Thid NTCIR Wokhop Infomation Gain Ratio mt Maximal Maginal Rlvanc A mthod of Summaization fo Multipl Documnt Tatunoi MORI and Takuo SASAKI Gaduat School of Envionmnt and Infomation Scinc, Yokohama National Univity 79-7 Tokiwadai, Hodogaya, Yokohama , Japan fmoi,takuog@fot.i.ynu.ac.jp Abtact In thi pap, w popo a mthod to mak a ummay fom multipl documnt with taking account of comphnibility and adability. A fo comphnibility, w how an intgation of MMR into th tmwighting mthod bad on IGR. A fo adability, w popo a mthod to gnat a ummay bad on cluting impotant ntnc accoding to ubtopic and making a kywod lit a a vy bif ummay fo ach clut. By th valuation in NTCIR3 TSC2, w how that th popod mthod wok wll to gnat comphniv ummai whn th lngth of ummay i hot and th tagt i a mall (7 o l) numb of documnt. 1 Intoduction Sinc a hug amount of documnt a availabl in digital fom, it i on of difficult poblm fo u to obtain ally ncay infomation fficintly. In od to dal with uch poblm, many ach a ngagd in tudi on Infomation Rtival (IR) and Txt Cluting. Automatd txt ummaization i alo on of th impotant topic a th mthod to duc th amount of documnt to b ad. Epcially, ummaization of multipl documnt i widly noticd a on of way to achiv mo fficint and ffctiv pntation of documnt. In multi-documnt ummaization, it hould b mo alitic to ummaiz not a lag t of htognou documnt dictly but a t of looly latd documnt. Fo xampl, lt u conid th ituation that a u u a ummaization ytm with a navigation intfac fo IR ult lik Scatt/Gatt[3]. Th navigation ytm claify th tivd documnt into om adquat numb of clut. Thn, th u lct om topic-latd clut to naow down th candidat documnt. Aft om itation of cluting and u lction, th u may obtain a mall numb of documnt to b ad. If th ummaization ytm can mak a ffctiv ummay, which can b a ubtitut of th oiginal documnt, th u can av th tim to ad. In thi pap, thfo, w will dicu a ummaization ytm und th following condition: ffl Tagt i a t of documnt that a oiginally tivd by an IR ytm on a ctain topic and naowd down to om mall numb of latd documnt. ffl Th output i an infomativ ummay, which can b a ubtitut of th oiginal (tagt) documnt. In od to atify th cond condition, th following point a impotant. Comphnibility Th ummay hould includ main contnt of tagt documnt xhautivly. Radability Th ummay hould b a lf-containd documnt and hould b adabl. Many of cnt ach put mphai on th comphnibility whil kping adability of ummai to om xtnt. Fo xampl, Radv t al.[12, 11] popod a mthod that claifi givn documnt into om clut and mak on ub-ummay fo ach clut, thn plac thm in an od. In th mthod of Stin t al.[13], fit of all, on ummay i mad fom ach documnt. Th t of uch ub-ummai a claifid into val clut, and on typical ummay i lctd fo ach clut a it ummay. Thn, th lctd ummai a lind up. Goldtin t al.[4] popod th mthod calld MMR-MD (Maximal Maginal Rlvanc Multi- Documnt), which collct paag latd to th quy fom nwpap aticl tivd by an IR ytm and aang thm into on ummay. Th main contibution of MMR-MD i a anking mchanim fo paag. Th mthod conid not only th imilaity btwn a paag and a quy, but alo th imilaity btwn th paag and ach of high-ankd paag. If a paag ha high imilaity with om of paag alady lctd, it hould b dundant. In uch a ca, om pnalty i givn to th paag and -ankd. Th mthod alo mak u of cluting to mak om clut of latd documnt. Th clut of documnt a ud in anking paag. All of th mthod dcibd abov utiliz om (non-hiachical) cluting mchanim to find goup of imila documnt. Sinc th main matt of concn i to xtact common infomation in a clut of documnt and thy tat ach clut paatly, thy m to hav th following poblm: ffl Thy cannot xplicitly tat th impotant pat that com fom diffnc among clut. ffl Thi ult of ummay havily dpnd on iz of ach clut, bcau common pat of documnt may vay accoding to clut iz National Intitut of Infomatic

2 Th Thid NTCIR Wokhop, Sp Oct W think that much contivanc i ndd to mak good ummai, which hav not only th common infomation of all clut but alo th impotant infomation pcific to ach clut. If w can obtain th dtail of imilaity tuctu of tagt documnt, w may tak account of not only th common pat of clut but alo th diffnc among clut. Such infomation would b a good clu to mak a ummay and impov th quality of xtacting impotant pat of documnt. Bd on th conidation dcibd abov, in thi pap, w will appoach to th multi-documnt ummaization in th following way. In th viwpoint of comphnibility, w popo an intgation of MMR into a tm-wighting mthod popod by Moi[7, 6]. Sinc th tm-wighting mthod giv a wight to ach wod accoding to th wod contibution to dtmining th hiachical tuctu of documnt clut, it can flct both commonality and diffnc among documnt into wight of tm. In th ca of ummaizing ach documnt in IR ult paatly, w nd a tm-wighting mthod and an impotant ntnc lction only, bcau w do not hav to ca about contolling dundancy. Each documnt can b uppod to hav no dundancy and to b a cohnt txt. Actually, Moi[7, 6] potd that th ntnc xtaction bad on thi tm-wighing mthod wok vy wll in uch a ituation. W, howv, hav to conid th dundancy contol in th ca of multipl documnt ummaization, bcau contnt of on documnt may ovlap with oth documnt. Thfo, w popo an intoduction of MMR into th famwok by Moi[7, 6]. Whil MMR i oiginally dignd to tat th lvanc of paag to a quy and dundancy among paag imultanouly, w modify it in od to dal with both impotanc and dundancy of ntnc. In th viwpoint of adability, w popo a mthod to gnat a ummay by cluting xtactd ntnc accoding to ub-topic and making a kywod lit a a vy bif ummay fo ach clut. Not that w aim at making not on vy cohiv and adabl txt of ummay but an infomativ txt, fom which u can obtain ncay infomation appopiatly. 2 Iu on Multipl Documnt Summaization In ummaization of multipl documnt, fitly, w hav to tak account of th following om xta point, which w do not conid in ingl documnt ummaization. Viwpoint 1 (Multi-documnt ummaization) 1. Finding impotant pat in ach documnt. 2. Elimination of dundant pat of documnt by dtcting common pat. 3. Finding diffnc among documnt and aanging thm. H, w uppo that tagt documnt a adquatly gathd though infomation tival poc fo a ctain topic and u lction in a navigation poc. Scondly, if th tagt documnt a th ult of infomation tival in a ctain topic, w alo hav to conid a quy, o infomation nd. Viwpoint 2 (Quy) Quy, which pnt th u infomation nd in th tival. Thidly, w hav to conid th chaactitic of tagt documnt. It may vay accoding to givn quy and documnt databa in th IR ytm. If th tagt a nwpap aticl lik NTCIR3 TSC2, th a, at lat, two typ of documnt a follow: Viwpoint 3 (Tagt t of documnt) 1. St of documnt that dcib on pcific topic in tim i lik aticl about on cim. 2. St of documnt that dcib on boad topic fom val diffnt viw. Thfo, th t may conit of val ub-topic. If w adopt th impotant ntnc xtaction a ummaization mthod, Viwpoint 1 i latd to th lction poc of impotant ntnc, on th oth hand, Viwpoint 3 ha a lation to th aangmnt poc of lctd impotant ntnc in od to mak on ummay txt. Viwpoint 2 may b in connction with both of tho poc. In thi pap, a hown in Figu 1, w tat th abov viwpoint with th following bai: 1. A fo Viwpoint 1-1, 1-3, and Viwpoint 2, w tat thm imultanouly with th mthod to map th imilaity tuctu of documnt into tm wight, which i popod by Moi[7, 6]. 2. A fo Viwpoint 1-2, w dal with it by a vaiant of MMR fo ingl ntnc. 3. A fo Viwpoint 3, w aliz it by claifying oiginal documnt into val clut by ingl link cluting algoithm, and giving a kywod lit to ach clut a a ub-titl. 3 Impotant Sntnc Extaction by Intgation of MMR and tm wighting bad on Infomation Gain Ratio In thi ction, w will popo an impotant ntnc xtaction mthod bad on intgation of th following chm: 1. Calculation of ntnc impotanc bad on Infomation Gain Ratio 2. Contol of dundancy in ummai by MMR. Thi xtaction mthod ha th following fatu: ffl Sinc imilaity tuctu among documnt i flctd into wight of tm, ntnc xtaction can b pfomd indpndntly of clut tuctu of documnt. Thfo, th mthod do not hav th tiction that th compion atio mut b pcify fo ach clut. ffl Sinc a kind of bia by quy i intgatd into th poc of mapping clut tuctu to tm wight,

3 Pocding of th Thid NTCIR Wokhop Doc. DB IDF Calc. Quy Sach Engin Hiachical Cluting IGR Calc. Rtivd Documnt (Input of TSC2 Tak B) D 1 D 2 D n TF Calc. Cluting D 2 1 D 2 3 D 2 D D7 2 C lu t S t u c tu D S 2 1 D S 2 3 S 2 D S7 2 K y w o d G n a tio n Summay fo ach doc S 1 S2 S n KW D S 2 1 KW D S 2 3 KW S 2 KW D S7 2 to D o c D i p a tc h A a n g m n t w4 w2 w5 w1 w3 w6 Final output of Summay Compion Ratio S4 S2 S5 S1 S3 S6 IDF(w) IGR(w,D) Intgation TF(w,D) wight(w,d) Tm Wight Sntnc Sgmntation S1 S2 S3 S4 S5 S6 Im S p o n t ta n n c c w1 w2 w3 w4 w5 w6 S1 S2 S3 S4 S5 S6 Ranking MMI-MS Nomaliz Figu 1. Ovviw of ou popod mthod fo multi-documnt ummaization th mthod can tak account of topic of tival unlik Stin mthod, and th mthod do not nd any qui xplicitly unlik MMR-MD. Moov, th mthod ha an ffct of quy xpanion in tm wighting, which MMR-MD do not hav. In od fo MMR to b ffctiv in dundancy limination, in thi ction, pcially w will dicu 1) th nomalization of ntnc wight and imilaity co among ntnc, and 2) choic of th valu of th paamt, which contol th dg of dundanc in MMR. 3.1 Tm wighting mthod bad on Infomation Gain Ratio In od to ummaiz ach of documnt in an IR ult, Moi[7, 6] popod a mthod of impotant ntnc xtaction by tm wighting bad on Infomation Gain Ratio(IGR). Thi mthod xtact th imilaity tuctu among a t of documnt though a hiachical cluting, thn giv high wight to wod that contibut to foming th tuctu. A an mau fo th contibution, it utiliz IGR[10] of pobabilitic ditibution of wod in a clut. Th IGR gain (w; C) of a wod w in a clut C i calculatd a follow: gain (w; C) = gain(w; C) plit info(c) (1) gain(w; C) = ntopy(w; C) ntopy p(w; C) ntopy(w; C) = p(wjc)log 2 p(wjc) (1 p(wjc)) log 2 (1 p(wjc)) p(wjc) = fq(c; w)=jcj ntopy p(w; X jc i j C) = i) jcj i plit info(c) = X i jc i j jcj log jc ij jcj wh fq(w; C), C i and jcj a th fquncy of th wod w in C, th i-th ub-clut of C, and th numb of wod in jcj, pctivly. Th valu (1) pnt th dg of conitncy btwn th pobabilitic ditibution of a wod and th patition of ub-clut. H, w hav to tak account of th following point: 1. If th tagt of ummaization i an IR ult, it i impotant to compa th tagt documnt with th t of documnt which a not tivd but xit in th documnt databa, in od to obtain th infomation what wod ally hav contibution in th tival. Thfo, a hown in Figu 2 w intoduc anoth lay of clut, which copond to th whol documnt databa. Th clut conit of two ub-clut. On ub-clut i th clut of tivd documnt, which i th tagt of futh cluting, and th oth on copond to th t of databa. Bcau of th contat in th intoducd clut, high wight a givn to wod pcific to th tivd documnt. It may includ th ffct of quy bia. 2. A hown in Figu 2, w obtain on IGR valu fo ach wod in ach clut. In od fo vy patition of clut to b flctd in tm wight, w hav to intgat all of IGR valu. In thi pap, w adopt th intgation IGR um givn by th ummation hown in (2): IGR um(w; D) = X C2Ct(D) gain (w; C);(2) wh Ct(D) i th t of all clut to which th documnt D blong.

4 Th Thid NTCIR Wokhop, Sp Oct IGR_um(w,D) gain_(w,c oot ) + gain_(w,c 1 ) + + gain_(w,c n-1 ) + gain_(w,c n ) Clut C oot Clut C 1 Clut C n Clut of all tivd documnt Clut of all documnt (Documnt DB) Clut of documnt not to b tivd wh A, Sim 1 and Sim 2 a th t of paag alady lctd, th imilaity btwn a quy and a paag, th imilaity btwn two paag. Aft initializing S to th mpty t and aigning an appopiat valu to th paamt, w may hav a anking of paag und th contol of dundancy by calculating (4) patdly. In (4), th fit tm and th cond tm of th ight hand id may b conidd a th tm fo xtacting th pat lvant to th quy and th tm fo liminating dundancy, pctivly. Th valu of paamt coodinat tho two ffct. 3.3 MMI-MS D Figu 2. Tm wight IGR um bad on Infomation Gain Ratio W dfin th final wight wight(w; D) of th wod w in th documnt D a th combination of th th typ of fundamntal wight, TF, IDF and IGR um. Sinc th wight TF, IDF and IGR um flct th indpndnt chaactitic, ach documnt, th documnt databa, and th clut tuctu pctivly, w adopt th following intgation: wight(w;d) = IGR um(w; D) TF(w; D) IDF(w) (3) W adopt th maximum ditanc algoithm a th cluting mthod fo diving IGR valu. Th algoithm ha on paamt, ff, which ha th ang [0.5,1] and contol th numb of ub-clut in a clut implicitly. Th mall th valu of th paamt i, th mo clut ach clut i dividd into. In th xpimnt dcibd blow, w t th paamt to MMR A dcibd in Sction 1, Goldtin t al. popod a multipl documnt ummaization mthod fo IR ult, calld MMR-MD (Maximal Maginal Rlvanc Multi-Documnt)[4]. To tat dundancy among tivd paag, paag a -ankd accoding to not only th lvanc of paag to a quy but alo th imilaity among paag. MMR-MD, thfo, can do both dtction of had pat of documnt (duction of dundancy) and xtaction of diffnt pat (impovmnt of comphnibility), imultanouly. Th dundancy contol of MMR-MD i bad on th notion of MMR(Maximal Maginal Rlvanc) popod by Cabonll t al.[2]. A dfind in (4), MMR lct th nxt paag in th t R of paag lvant to th quy Q. MMR(R; A) df = Ag max [ Sim 1 (D i ;Q) (1 ) max Sim 2(D i ;D j )] D i 2RnA D j 2A (4) In thi ction, w will dicu about intoduction of dundancy contol mchanim of MMR into th impotant ntnc xtaction bad on IGR. Th unit of MMR i oiginally paag o documnt, and it adoptd paag lvanc to a quy a initial anking Sim 1. Thfo, MMR may b incopoatd into th chm of impotant ntnc xtaction, if w chang th unit fom paag to ntnc and u th impotanc of ntnc a an initial anking mthod. In thi pap, w call th xtndd chm chm of impotant ntnc xtaction MMI-MS (Maximal Maginal Impotanc Multi-Sntnc) and dfin it a follow: MMI-MS df = Ag max [ Imp(S i ) (1 ) max Sim(S i;s j )] S i 2SSnA S j 2A (5) wh SS, Imp(S i ) and Sim a th t of all ntnc in tagt documnt, th impotanc of ntnc S i and a ctain mau of imilaity btwn ntnc, pctivly. A Imp(S i ) and Sim, w adopt th ntnc impotanc bad on IGR and th coin colation btwn ntnc vcto. 3.4 Nomalization of ntnc impotanc and imilaity btwn ntnc Th ang of Imp(S i ) and Sim hould b compaabl in od fo MMI-MS to wok appopiatly in limination of dundancy. Th ang of Sim i [0; 1]. On th oth hand, Imp(S i ) do not hav uch a fixd ang in ou impotanc calculation. Thfo, w pfom th following nomalization of Imp(S i ) and Sim. 1. Nomalization of ntnc impotanc in a documnt In documnt D, impotanc of ntnc i convtd to dviation co who avag i 0:5 a follow: Imp d (S i)=0:5 + Imp (S i ) Imp av (D) ff(d) wh Imp av (D) and ff(d) a th avag and tandad dviation of ntnc impotanc in documnt D. Thi nomalization qualiz th avag of ntnc impotanc in documnt und th hypothi that all documnt i qually impotant. (6)

5 Pocding of th Thid NTCIR Wokhop 2. Nomalization of ntnc impotanc and ntnc imilaity in all of documnt Thi nomalization i to mak u that MMI-MS tak ffct. Fitly, Imp (S i ) and Sim (S i ;S j ) a dividd by thi maximum valu, pctivly. Thn, thy a convtd to dviation co who avag i 0: Choic of th valu of paamt in MMR-MS Th paamt ha th ang [0; 1]. Th clo to 0 th paamt i, th mo ffctiv th limination of dundancy i. Sinc th adquat valu of may dpnd on th tagt t of documnt, th valu hould b lctd adquatly. Fo xampl, if th documnt t includ many diffnt ub-topic and cacly ha ovlap, hould b t to th valu clo to 1 to wakn th dundancy limination. On th contay, if th documnt t ha many ovlap lik aticl about on pcific topic, th valu hould b mall valu to pomot th limination of dundancy. In thi pap, w pay attntion to th avag of coin colation a th mau fo dundancy among ntnc. Fo xampl, th ituation that th avag ha high valu how th fact that th a many imila ntnc. In uch a ituation, th paamt i t to a mall valu in od to pomot limination of dundancy by MMI-MS. W adopt th quation (7) fo, which ha th valu ang [0:5; 1], in od fo th ntnc impotanc to b mo dominant than limination of dundancy. = 0:5 +0:5(1 Sim av ) (7) wh Sim av i th avag of Sim (S i ;S j ). 4 Gnation of Txt of Summay So fa, w dcib th mthod to xtact impotant ntnc fom multipl documnt. In thi ction, fom th viwpoint of adability w will dicu how th xtactd ntnc hould b aangd to b on ult ummay. In gnating ummay, w hav to pay attntion not only natualn a a txt but alo cohnc among xtactd ntnc. Each ntnc in a ummay hould hav an adquat contxt in od fo ad not to b mild. In th chm of ntnc xtaction fo ummay gnation, w nd th following tp to mak a good cohiv ummay. 1. Extact impotant ntnc with taking th cohnc into account. Fo xampl, xtaction unit may b a i of ntnc connctd to ach oth with fnc o lxical cohion. W imply call th unit of xtaction paag haft. Paaphaing fnc xpion into thi fnt would alo ffctiv to mak ntnc lfcontaind. 2. Plac xtactd paag in od accoding to lation among paag lik tim od, topical cohnc, and o foth. 3. Gnat xpion to claify lation btwn paag and int thm to th ummay, if ndd. 4. Finally, adjut th daft of ummay by paapha o implification to inca unifomity of xpion. Mani t al.[5] popod a mthod to impov infomativn and adability of oiginal ummay, which i pntd a a lit of yntactic t with cofnc infomation. Nanba t al.[8] dicud a mthod to gnat ummay ay to ad by witing xtactd impotant ntnc. Otuka t al.[9] impov th cohnc among xtactd ntnc by placing fnc xpion with thi antcdnt. In ingl documnt ummaization, w may achiv th minimum quimnt of cohion in a ummay by making an ffot to maintain th cohion which a tagt documnt oiginally ha. On th oth hand, in multi-documnt ummaization, w hav to find cohion aco tagt documnt, which do not appa xplicitly in th documnt. Moov, th tyl of documnt would diff fom ach oth. Howv, many of xiting ach do not tackl th poblm dictly, but mly plac ntnc o paag in om od by uing infomation in oiginal documnt, lik dat infomation, poition in documnt, and o foth. Fo xampl, mthod popod by Goldtin t al. [4], Stin t al. [13] and Radv t al. [12, 11] aang th xtactd unit in tim od of oiginal documnt. Stin mthod, moov, od th unit (ach ummay of ingl documnt) o a to inca imilaity btwn adjacnt unit. Th can b gadd a attmpt to mak cohion among unit in ay way. Not that th mthod dcibd abov aim to ummaiz documnt into on continuou ading. Thi typ of ummay would b uitabl fo documnt of on pcific topic lik aticl about on cim. In ummaization of IR ult, howv, w uually obtain documnt in val diffnt point of viw, namly ub-topic, vn if th quy i about on topic. W think that, in uch a ca, it would b pfabl to how th outlin of ub-topic xplicitly and pnt ub-ummai fo ub-topic along with th outlin. Bazilay t al.[1] potd th following finding by th xpinc in which ubjct aang th am t of xtactd impotant ntnc o a to maximiz adability of txt. ffl Fitly, th total od of ntnc dpnd on ubjct, and th a val poibiliti of od of ntnc. ffl Th od, howv, i not totally f, but val block of ntnc can b found. ffl Sntnc blonging to am block a latd to ach oth in tm of a ub-topic. Whil th od of block and th od of ntnc in ach block may dpnd on ach ubjct, th i gnal agmnt among ubjct that ach block conit of ntnc latd to on ub-topic. Bad on abov dicuion, w popo a mthod of ummay gnation with following th tp accoding to chaactitic of documnt t:

6 Th Thid NTCIR Wokhop, Sp Oct Mak ub-ummai by claifying ntnc accoding to ub-topic, 2. Gnat a ub-titl, o a lit of kywod, fo ach ub-ummay, 3. And aang ub-ummai in od of dat along with ub-titl. 4.1 Cluting impotant ntnc accoding to ub-topic A dcibd in Viwpoint 3, th a, at lat, two typ of tuctu of ub-topic in a documnt t: 1. St of documnt that dcib on pcific topic in tim i lik aticl about on cim. 2. St of documnt that dcib on boad topic fom val diffnt viw. Thfo, th t may conit of val ub-topic. Fo a documnt t lik 1, th tatgy to aang ntnc in tim i would b uitabl. On th oth hand, fo a documnt t lik 2, w nd to adopt om tatgy to claify ntnc into ub-topic, and aang thm in tim i, if ncay. W may dal with both of th ca by claifying th oiginal documnt into om clut accoding to lvanc to ub-topic. Now, what kind of cluting algoithm i uitabl fo ou pupo? Th focu of topic may gadually hift accoding to th pog of incidnt, vn if th tagt documnt a in th ca of 1, wh thy hav to b claifid into on clut. In uch a ca, it i impotant fo cluting algoithm to minimiz not th iz of ach clut but th ditanc btwn ach documnt and it nat documnt. Thfo, w adopt th ingl link mthod, in which ach documnt i claifid into th clut that ha th clot documnt. A fo th imilaity (o ditanc) btwn documnt, w u th imilaity of oiginal documnt. Similaity of ub-ummai of documnt i not uitabl fo th cluting, bcau in thi tag ou mthod ha alady liminatd th dundancy in ummai. Bad on th abov dicuion, w will mak clut of xtactd ntnc and lin thm up by th following poc: 1. Obtain th ummay of ach documnt by gathing xtactd impotant ntnc documnt by documnt. Th ummay of ach documnt can b gadd a ub-ummay and i th unit of th following pocing. 2. Mak clut of ub-ummai accoding to th imilaity among oiginal documnt (coin valu of documnt vcto), wh th cluting algoithm i th ingl link mthod, and th thhold imilaity fo mging i 0:5. 3. Lt th clut dat b th dat of th oldt documnt. Lin up clut in tim od of clut dat. 4. In ach clut, lin up ub-ummai in tim od accoding to documnt dat. 4.2 Kywod gnation fo bif ummai of clut Th vy bif ummay fo ach clut of ubummay may uful fo ad to undtand th outlin of th whol ummay. Th hot ummai wok a paato of ub-topic, and may contibut fo impoving th adability. A bif ummay of ach clut, in thi pap, w u kywod xtactd fom ach clut by th following poc. 1. Slct th mot impotant wod in ach documnt. Th mot impotant wod i dfind a th wod of hight wight in th mot impotant ntnc. If th mot impotant wod i th pat of long compound wod, th longt compound wod i lctd a kywod of documnt. Othwi, th mot impotant wod i th kywod of documnt. 2. Th bif ummay of clut i mad a a t of kywod of documnt that blong to th clut. Th aon to lct compound wod in kywod lction i that uch compound wod may cay mo pcific infomation. In th ummaization, w diplay th kywod at th bginning of ub-ummay of ach clut. 5 Expimnt W paticipatd in NTCIR3 TSC2 and valuatd ou popoal with th Tak B[14]. Each paticipant gnatd ummai accoding to topic infomation givn by th tak oganiz, and ubmittd a t of ummai to th tak oganiz. Each topic copond to on IR ult, which conit of th following infomation: ffl Topic ID ffl Lit of kywod fo quy in IR. ffl Bif dciption of th infomation nd. ffl St of documnt ID, which a tagt documnt of ummaization. Th numb of documnt vai fom 3 to 17 accoding to topic. ffl Lngth of ummay to b gnatd. Th a two lngth of ummay, Long and Shot. Th lngth i givn in chaact and th chaact Nwlin i not includd in th lngth. Whil th lngth may vai accoding to th topic, th lngth of Long i twic of Shot. To ach paticipant, th tak oganiz upplid th infomation about ytm valuation. In th t of thi ction, w will dcib th valuativ infomation. 5.1 Evaluation by Ranking Th tak oganiz ppad th following fou ummai fo ach topic and ach lngth: 1. Summay mad by a human ( Manual haft).

7 Pocding of th Thid NTCIR Wokhop 2. Summay gnatd by a ytm to b valuatd. 3. Summay gnatd by Lad mthod (Balin No.1. Lad haft). 4. Summay gnatd by Stin mthod (Balin No.2. Stin haft). Nxt, on of twnty human ao ad th oiginal t of documnt and fou ummai fo ach topic and ach ummay lngth, thn ank th fou ummai in th following valuation point of viw: Comphnibility Th ummay ha all of impotant contnt, o not. ( C haft) Radability Th ummay i ay to ad, o not ( R haft) Thfo, vy ytm will hav fou typ of anking, namly, C Shot, R Shot, C Long, and R Long, fo ach topic. Not that mall valu man th btt ytm in thi valuation. 5.2 Evaluation by Human Coction In thi valuation, on of human ao coct ummai of ytm output in tm of th valuation viwpoint dcibd abov. Th ach tp of coction hould b on of th opation, namly, intion, dltion, ubtitution of ting. If mo than 50% of ummay hould b coct, ao may giv it up. 6 Expimntal Rult 6.1 Rult of Evaluation by Ranking Tabl 1 and 2 how th valuation by avag ank. Not that, tictly paking, th avag of ank i not a valu uitabl fo compaion of ytm. Th anking i not an intval cal but an odinal cal. In oth wod, th numb of poition and th diffnc of poition in anking do not hav pcial maning, whil th od of itm i maningful. Thu, w hould conid th avag of ank a not a tict mau but a lax mau. Thfo, w compad ou ytm with ach of balin topic by topic. Th ult i hown in Tabl 3. W alo valuat only th topic that hav 7 documnt o l, bcau by xamination of ach topic w found out that ou mthod wok wll fo mall documnt t. Th ult i hown in Tabl 4 and Rult of Evaluation by Human Coction Tabl 6 how th avag amount of human coction. 7 Dicuion 7.1 Comphnibility Accoding to xpimntal ult, popod mthod i upio to th balin, namly th lad mthod and Stin mthod, in comphnibility, pcially und th following ituation: Tabl 1. Evaluation by Ranking fo Popod mthod (All of 30 topic) C Shot R Shot C Long R Long Avag Ranking(AR) Ranking of AR Avag Rank: Avag anking (AR) of ou ytm compad with th balin. Ranking of AR: Ranking of AR in all nin paticipating ytm Tabl 2. Avag Ranking compad with Balin (All of 30 topic) C Shot R Shot C Long R Long Popod Manual Lad Stin Tabl 3. Numb of Win and Lo compad with Balin (All of 30 topic) C Shot R Shot W L T W L T v.. Manual v.. Lad v.. Stin C Long R Long W L T W L T v.. Manual v.. Lad v.. Stin W: win, L: lo, T: ti Tabl 4. Avag Ranking compad with Balin (15 Topic with 7 documnt o l only) C Shot R Shot C Long R Long Popod Manual Lad Stin Tabl 5. Numb of Win and Lo compad with Balin (15 Topic with 7 documnt o l only) C Shot R Shot W L T W L T v.. Manual v.. Lad v.. Stin C Long R Long W L T W L T v.. Manual v.. Lad v.. Stin W: win, L: lo, T: ti

8 Th Thid NTCIR Wokhop, Sp Oct Tabl 6. Avag amount of human coction (in chaact p documnt) Shot Long Popod Avag Popod Avag Dltion C(%) R(%) Intion C(%) R(%) Subtitution C(Dl)(%) C(In)(%) R(Dl)(%) R(In)(%) Th output a hot ummai. 2. Th tagt i a mall numb of documnt. Th ult would how that MMR i ffctivly intgatd into th tm-wighting mthod bad on IGR. On th oth hand, ou mthod i not atd highly in th oth ituation. On of aon may b th incompltn of automatic adjutmnt fo paamt, which contol dundancy of ummay. A hown in (7), ha th bia 0.5, and w put mo t on impotant ntnc lction. That i, MMI-MS tnd to b uppd. Whn th numb of tagt documnt bcom lag, th mthod tnd to lct impotant ntnc fom vy documnt vnly if th ffct of MMR i wak. 7.2 Radability Accoding to human coction, th total amount of coction of ou ummai i lag than output of oth ytm. On of main aon i th fact that ou ytm i bad on impotant ntnc xtaction, and do not u any oth mthod uch a a tchniqu to inca cohion among ntnc, witing to hotn ntnc. W alo hav to not that th viwpoint of adability of TSC2 tak oganiz i diffnt fom ou. Th tak oganiz valuatd adability of ummai in tm of ingl documnt. Howv, w aim at adability in tm of infomation acc, and w do not tick to mak a ingl documnt. Actually, almot all of ubtitl in ou ummai a dltd in human coction. 8 Concluion In thi pap, w popod a mthod to mak a ummay fom multipl documnt with taking account of comphnibility and adability. A fo comphnibility, w howd an intgation of MMR into th tm-wighting mthod bad on IGR. A fo adability, w popo a mthod to gnat a ummay bad on cluting impotant ntnc accoding to ub-topic and making a kywod lit a a vy bif ummay fo ach clut. Th valuation in NTCIR3 TSC2 howd that th popod mthod wok wll to gnat comphniv ummai whn th lngth of ummay i hot and th tagt i a mall (7 o l) numb of documnt. In ou futu wok, w will conid th impovmnt of ummay gnation including impovmnt of adability, tatmnt of cohion, paapha. Rfnc [1] R. Bazilay, N. Elhadad, and K. R. MaKown. Sntnc oding in multidocumnt ummaization. In Pocding of th th fit Intnational Confnc on Human Languag Tchnology Rach (HLT 2001), pag , [2] J. Cabonll and J. Goldtin. Th U of MMR, Divity-Bad Ranking fo Roding Documnt and Poducing Summai. In Pocding of th 21t Annual Intnational ACM-SIGIR Confnc on Rach and Dvlopmnt in Infomation Rtival, pag , [3] D. R. Cutting, D. R. Kag, and J. O. P. J. W. Tuky. Scatt/gath: A clut-bad appoach to bowing lag documnt collction. In Pocding of SIGIR 92: 15th Annual Intnational ACM SIGIR Confnc on Rach and Dvlopmnt in Infomation Rtival, pag , [4] J. Goldtin, V. Mittal, J. Cabonll, and M. Kantowitz. Multi-Documnt Summaization by Sntnc Extaction. In Pocding of ANLP/NAACL Wokhop on Automatic Summaization, pag 40 48, [5] I. Mani, B. Gat, and E. Blodon. Impoving ummai by viing thm. In Pocding of th 37th Annual Mting of th Aociation of Computational Linguitic (ACL 99), pag , [6] T. Moi. Infomation gain atio a tm wight th ca of ummaization of i ult. In Pocding of th 19th Intnational Confnc on Computational Linguitic (COLING 02), pag , Aug [7] T. Moi. A tm wighting mthod bad on infomation gain atio fo ummaizing documnt tivd by i ytm. Jounal of Natual Languag Pocing, 9(4):3 32, (in Japan). [8] H. Nanba and M. Okumua. Poducing mo adabl xtact by viing thm. SIG Not NL-133-8, Infomation Pocing Socity of Japan, (In Japan). [9] T. Otuka, A. Utumi, and K. Hiota. Youyaku-bunii-ni-oku houou-hoi (anaphoa olution in ummay gnation). In Pocding of vnth annual mting of th Aociation fo Natual Languag Pocing (NLP 2001), pag , Ma (In Japan). [10] J. R. Quinlan. C4.5: pogam fo machin laning. Mogan Kaufmann Publih, May [11] D. R. Radv and W. Fan. Automatic ummaization of ach ngin hit lit. In Pocding of ACL Wokhop on Rcnt Advanc in NLP and IR, [12] D. R. Radv, H. Jing, and M. Budzikowka. Cntoidbad ummaization of multipl documnt: ntnc xtaction, utility-bad valuation, and u tudi. In Pocding of ANLP/NAACL Wokhop on Automatic Summaization, [13] G. C. Stin, T. Stazalkowki, and G. B. Wi. Summaizing Multipl Documnt uing Txt Extaction and Intactiv Cluting. In Pocding of th ixth Pacific Aociation fo Computational Linguitic (PACLING 99), pag , [14] TSC Committ. NTCIR 3 automatic txt ummaization tak/tsc 2(txt ummaization challng 2) wb pag. n.html, 2001.