Trainable Automatic Text Summarization Using Segmentation of Sentence

Transcription

1 Proceedings of the Third NTCIR Workshop Trainable Atomatic Text Smmarization Using Segmentation of Sentence Kai ISHIKAWA Shin-ichi ANDO Shin-ichi DOI Akitoshi OKUMURA Mltimedia Research Laboratories, NC Corporation Miyazaki Miyamae-k Kawasaki-shi Kanagawa , Japan k-ishikawadq, s-andocw, s-doiah, a-okmrabx.jp.nec.com Abstract In this paper, we propose an atomatic smmarization method combining conventional sentence extraction and trainable classifier based on Spport Vector Machine. To make extraction nit smaller than the original sentence extraction, we also introdce sentence segmentation process in or method. The evalation reslts show that or system achieves the best reslt among all the other systems with regard to contents, and closer to the hman constrcted smmaries (pper bond) at 20% smmary rate. On the other hand, the system needs to improve readability of its smmary otpt. Keywords: atomatic text smmarization, manally constrcted smmary, nit of extraction, machine learning, dividing sentences into clases, attribte vales, Spport Vector Machine. 1 Introdction One of the challenging isses of or smmarization stdy is the improvement of accracy in evalating importance of parts in articles to compose smmaries. The basic concepts of the extraction method are well known [10][2]. When we participated in the previos NTCIR Workshop, i.e. NTCIR-2 TSC task-a (sentence extraction task), we tried to improve the precision of sentence extraction by optimizing the combination of heristic rles. We tilized headline information, term freqencies, and sentence position as sefl information to calclate a sentence importance, and stdied the way of combining then into a score and obtained improvements. In or previos work, we have the following two problems left. The first one is the degradation of sentence score accracy by sing a linear combined score of mltiple score fnctions based on heristic rles. In this formlation, it is difficlt to make one heristic rle most credible among the heristics rles to jdge the importance of an extraction nit. We shold introdce some atomatic method to determine the copling coefficients which we optimized manally according to some experimental reslts in the previos work. The second one is that the minimm extracting nit is a sentence, which needs to be smaller when we have longer sentences in an article containing both important and nimportant parts. In this paper, we try to improve these problems in or previos smmarization work by introdcing the following two procedres: (1) Machine learning sing hman constrcted smmaries. (2) Using clases/sbsentence as extraction nit. Before going throgh the procedres (1) and (2) in details, we want to review some of related works in corps base extraction methods. In these types of methods, trainable classifiers are sed to classify sentences into important class and n-important class. Kpiec, et.al. [9] proposed a method sing statistical classifier based on Bayes rle. Nomoto, et.al. [13] and Okmra [14] applied decision tree learning (C4.5 [15]) to obtain a sentence classifier. Rhetorical Strctre Theory (RST) proposed by Mann and Thompson [11] is applied to atomatic smmarization by Marc [12]. An elemental nit of RST corresponds to clases/sb-sentence. In Marc s method, importance of each nit is calclated according to the hierarchical depth of a rhetorical strctre. Kobori et.al. [8]also applied RST in their smmarization method. His method ses phrase as an extraction nit, which is mch smaller than the nit of RST. They se morphological information beside rhetorical relations, and applied C4.5 to obtain a classifier for phrases from training data composed of 1000 paragraphs from newspapers and academic papers randomly selected. They report the problem of prodcing ngrammatical sentences in smmaries. Knight [7] proposed a statistical smmarization method based on noisy channel model, its mathematical formlation is almost the same with that of statistical machine translation. This method composes a smmary sentence composed of selected words from the word seqence of an inpt sentence, and generates 2003 National Institte of Informatics

2 The Third NTCIR Workshop, Sep Oct mch precise smmaries than by phrase extraction. However, large amont of parallel corps (word alignment of sorce sentences and smmary sentences is obtained) is necessary to constrct a statistical model, and the information of discorse level is not yet considered in the present formalism. As we can see from these related works, redcing the size of extraction nit entails increase of cost to constrct training data and risk to generate ngrammatical sentences as well as increases the preciseness of extraction to generate smmary sentences. Or method processes smmarization based on or conventional extraction method sing heristic rles, combined with atomatic trainable classifiers. A sentence or a clase is sed as an extracted nit. The advantage of this method is the se of manally smmarized data as its training data. The method also resolves the problem of smmarization particles being too coarse with long sentences contained in the text. 2 System Description 2.1 Approaches Or smmarization system ses a conventional key sentence extraction method, where we se a clase as a minimm nit of extraction as opposed to a sentence commonly sed as a minimm nit. The smmarizer operates segmentation of compond sentences based on ce words like connecting expressions, and obtains clases / sb-sentences as nits of extraction. ach segmented nit is repaired to improve its readablity by converting the form of declinable word at the end of the nits to a complete form. When a sentence is not segmented, it becomes the nit of extraction itself. The smmarizer is combined with classifiers obtained by machine learning sing Spport Vector Machine (SVM). Smmarization by sentence extraction is considered as a classification task, where each sentence is classified as relevant or non-relevant for the smmary extracted patterns. The system atomatically learns the manally annotated smmaries. Here, the vector attribte factors sed in the SVM learning are; position of the sentences in the text, tf scores, similarity with the headline sentence, occrrence of ce words sch as conjnctions, and docment genres. The SVM classifier assigns real vale to each nit as classification reslt. We consider this vale as a score of relevance for each nit, and combine it into the new score by calclating a linear combination of and the relevance score of conventional method. Those with highest scores are extracted and compose a smmary text. 2.2 Process flow The following process flow depicts or method, groped into (A) training process and (B) smmary generation process. (A) Training process (A-1) Apply morphological analysis to the sentences in the articles of training data. (A-2) xtract attribte vales from the sentences (refer to table 1). (A-3) Generate a classifier by training the sentences in the training data (decision of extracting each sentence while smmarization are mannally annotated) and the attribtes vales sing SVM. (B) Smmary generation process (B-1) Apply morphological analysis to the sentences in the inpt article. (B-2) Obtain the nits of extraction from the sentences sing segmentation of comond sentences at clase bondaries. (B-3) Repair the segmented nits by converting them to a complete end-form. (B-4) xtract attribte vales from the each nit of extraction (refer to table 1). (B-5) Calclate the scores of each nits based on a conventional extraction method. (B-6) Calclate the relevancy sing SVM classifier. (B-7) Combine the scores of conventional method and the SVM classifier. (B-8) Generate smmary text by extracting the higher ranked nits of extraction in their original order in the inpt article. 2.3 Segmentation of sentences into clases / sb-sentences It is known that the connecting expressions in Japanese, which connect clases, can be classified into some conjnction levels based on their clase connection strength to the main sentence [6] [1]. In or approach, we se the expressions of the highest connection level to detect the sentence bondaries. Segmented sb-sentences become independent sentences (or clases) as described in the followings.

3 P d a [ a 4 Ú Proceedings of the Third NTCIR Workshop Method: 1. Find connecting expressions in a sentence that indicate the break points for clases/sb-sentences. 2. valate the independency of each component in their contexts. Inhibit segmentations within the expression (if then ). Inhibit segmentations within the parenthesis. 3. Repair each segmented nit by converting it to a complete end-form to improve their readability. The next two examples show cases where (a) the segmentation is applied and (b) the segmentation is not applied. (a) An example of sentence where segmentation process IS APPLID: Before segmentation! # $ % & process: ' ( *,. / (Since major companies own 7 8 % movie ; = theaters & ' nder ( A C their, D direct F management, H I! 4 K L % )6 M N O P Q / Q R N U A N V 2 4 W X Y Z P M N [ F N U A \ ] & D b P 2 g [ 4 ^ _ a (it is cheaper to show the movies prodced by independent prodctions nder profitable contract than to make in-hose prodcts with high prodction h cost, i to maintain A the o p low-profit q r s * theaters, and pls,)6 k m n (they don t face any big risk.)6 After segmentation and repairment: 1st-part:! # $ % & ' ( *,. / (Major companies own movie theaters nder their direct management.) 2nd-part: 8 % ; = & ' ( A C, D F H I! 4 K L % M N O P Q Q R N U A N V 2 4 W X Y Z P M N [ F N U A \ ] & D b P d P 2 g ^ _ a (It is cheaper to show the movies prodced by independent prodctions nder profitable contract than to make in-hose prodcts with high prodction cost, to maintain the low-profit theaters.) 3rd-part: i A o p q r s * k m n (They don t face any big risk.) (b) An example of sentence which sentence segmentation process IS NOT APPLID: ~ ƒ Q = % & ' M N ƒ % L 4 M N ƒ % ˆ Š Œ % Ž X 4 % Ž 4 N U A i m š Ž œ % Ž 4 Ÿ % P ž (If there exists bdgetary spport by the federal government, sch as Cltral Society s fnd increase in movie-making, formation of long-term loan for prodcers, establishment of training instittes, and constrctions of pblic theaters 7 to provide ª opportnities «! for Q film = presentations,)6 V q (then several promoting measres are conceivable.)6 2.4 SVM classifier Or system ses ± ³ µ º (Ver.4.00) [5], which is an implementation of Vapnik s Spport Vector Machine. There are some other choices in the machine learning frameworks that are known to be effective in the related smmarization works, i.e.; probabilistic classifier based on Bayes rles [9], decision tree C4.5 [4] [13] [14] [8], and perceptron [4]. Aside from SVM method, we were inspired by these works and performed preliminary testing sing C4.5. Decision tree classifier proved effective in extracting most important and nimportant parts (10 to 20 % of the docment) from a docment. However, smmarization rate in this method seemed ncontrolable, becase we cold not observe the difference among the smmaries from three decision trees each trained with 10%, 30%, and 50% smmaries respectivelly. Observing the trees beeing trained we fond that the order of heristic rles applied to the decision trees seemed indifferent and casing a confsion while classifying sentences with medim relevancy. Therefore, we sed the SVM classifier here, which gives vales of -1 to 1 as that is convenient to controll the smmarization ratios of the otpts. The following attribtes are sed. Attribtes (16) ¼ ½ À Á, (17) à Ä, and (18) Ã Ä Æ È Ä, are assigned based on the following eqations, where we represent a docment as Ê, a nit of extraction (clase or sb-sentence) in the docment as, and a term as à respectively. ¼ ½ À Á Ì º Í Î Ñ Ó Ô µ Õ Ñ º Ö Î Ù º Û º 7 Ý º Î Ù Þ (1) here, Ú is the Kronecher s delta. This ¼ ½ À Á represents the rate of occrrence of headline words in a nit of extraction. Ã Ä is given as follows, where Ä à à is the nmber of times à appeared in the docment Ê : Ã Ä Ì º Î Ù Ã Ä Æ È Ä is given as follows: Ä à à (2)

4 Ú í The Third NTCIR Workshop, Sep Oct Table 1. The set of attribtes sed for the SVM. No. Featre name vale Definitions 1 top page 0/1 Genre of the article. The vale is 1 for 1st, 2nd, and 3rd page of the articles. 2 general 0/1 Genre of the article. The vale is 1 for general articles. 3 editorial 0/1 Genre of the article. The vale is 1 for editorial articles. 4 nsent Positive integer The nmber of sentences in the article. 5 dloc 0 to 1 Real Linear scale of sentence position in the article. 0 for the first, 1 for the last. 6 ploc 0 to 1 Real Linear scale of sentence position in the paragraph, 0 for the first, 1 for the last. 7 Conj-SB 0/ 1 Conjnction at the beginning of the sentence. Vale = 1 if present. 8 Demo-SB 0/1 Demonstrative pronon at the beginning of the sentence. Vale = 1 if present. 9 Mark-SB 0/1 Mark at the beginning of the sentence. Vale = 1 if present. 10 Unpnct-S 0/1 Vale = 1 if no pnctation mark at the end. 11 Conj-NSB 0/1 Conjnction at the beginning of the following next sentence. Vale = 1 if present. 12 Demo-NSB 0/1 Demonstrative pronon at the beginning of the following next sentence. Vale = 1 if present. 13 Mark-NSB 0/1 Mark at the beginning of the following next sentence. Vale = 1 if present. 14 Last-NS 0/1 Vale = 1 when the following next sentence is the last sentence in the article. 15 nchar Positive integer The nmber of characters in the sentence. 16 rhead Positive real Similarity with the headline. The vale is given by the eqation tf Positive real Normalized term freqency. The vale is given by the eqation tfisf Positive real Prodct of term freqency and inverted sentence freqency. The vale is given by the eqation 3. where, Æ È Ä Ã Ì Ã Ä Æ È Ä Ì æ é Ù Î à º Î Ù æ é Ä à Ã å Æ È Ä Ã Þ (3) æ é 2.5 Smmary generation Ù Î à º Í Î Ù º Û º Í Ù Î à í (4) In smmary generation process, the score for each nit of extraction, (a sentence or a clase obtained by segmentation process) is calclated and otpts higher ranked nits as a smmary otpt nder constraint of the assigned smmary rate. We se a combined score with the conventional score based on heristics ¼ ï ð Õ ò Ñ Õ º µ ð Õ Ó È and the otpt vale of the SVM classifier. The following is sed to calclate the conventional score; ¼ ï ð Õ ò Ñ Õ º µ ð Õ Ó È Ì Ã Ä Æ È Ä È ô ¼ ½ À Á È (5) This score is combined with the otpt vale SVM classifier into the score ¼ È ; ¼ È Ì of ¼ ï ð Õ ò Ñ Õ º µ ð Õ Ó È ô ø å È (6) Here, ø is a copling constant. If we have mltiple sets of training data in different smmarization rates, r1, r2, and r3, we shold obtain mltiple classifiers by training from them independently. In this case, mltiple otpts of classifiers, ù, ù 7, and ù h are combined into the score ¼ È : ¼ È Ì ¼ ï ð Õ ò Ñ Õ º µ ð Õ Ó È ô ø å ù È ô ù 7 È ô ù h È (7) 3 valation 3.1 Sbmitted reslts We implemented a smmarization system as described in the previos section and sbmitted the reslts (at 20% and 40% smmarization rate for 30 articles) of or system to TSC-2 task-a [3]. To carry ot

5 N! ý * i P * 4 Proceedings of the Third NTCIR Workshop Table 2. Ranking of smmaries for each system. Content Reliability 20% 40% Ave. 20% 40% Ave. System System System System System System System Or System TF Hman the training of SVM classifier, we tilized NTCIR-2 TSC (previos TSC) task-a test set collection of dryrn and formal-rn. This is composed of 60 newspaper articles with answer smmaries of each article at three smmarization rates, 10%, 30%, and 50%. We trained SVM classifier on this data set and obtained three classifiers corresponding to each smmarization rate. These classifiers are sed in the sentence score calclation described in the eqation (7). 3.2 TSC-2 evalation reslts The feasibility and the efficiency of or method are shown in the evalation reslts given to participants grops by TSC-2 task organizer. Table 2 shows the reslts of ranking evalation. In this evalation, the following for types of smmaries are compared and manally ranked. (1) xtraction base smmary constrcted by hman (2) Free style smmary by hman (pper bond) (3) Smmary reslt by a smmarization system (target) (4) Smmary reslt by lead method (baseline) The ranking of smmarization systems performed on each article is shown in the table 2. The ranking was based on: Content-Important contents of the original article are covered in smmary. Readability-Smmary remains to be readabile and meaningfl for comprehension. The reslts show that, or system achieved the best reslt among all the other systems with regard to contents, and performed closer to the hman process especially at 20% smmarization rate. The system didn t excel with regard to readability when compared to other top systems. Or smmarization system as more advantage in extracting key contents while more work Table 3. Correlation between the rankings according to content and readability. Content 20% smmaries 40% smmaries Reada bility is necessary to improve the readability. Table 3 shows the correlation of rankings on content and readability for each article at 20% and 40% smmarization rates. The reslts are pretty consistent for the content and the readability. Some smmaries at 20% rate have vale of 1 for the content while having different vales from 1 to 4 for the readability. To frther investigate the difference, we pick the two extreme smmary samples: the article with readability 4 shown in table 4, and the article with readability 1 shown in table 5. Both have the content ranking of 1. ; ý [ F ÿ Œ ý! I! ý (That old edcation system s Shonan Jnior High, I hated it.) Q 2 % ý g. F (I hated it becase it was all work, and becase of these ntrstworthy # " 4 snobby teachers that I mentioned.)! % # % ' % ( ) * P ý I + Q, r P - P % 0 2 g 3 s F (After the war, this high-ranking breacrat was invited as one of the first gradate of the Shonan Jnior High and introdced as '!, =! the A B most sccessfl almni.) 9 : ; P C D % g 3 s / F (Professor Okno taght s that the senses are most important to '!, G I 4 L M % * % s hmans.) 9 : ; J F = N A Q S 3 s A B % U 7 A / O F (Professor Okno also taght s the importance of the senses that we can even draw a rectangle a circle ' I W if it appeared to be rond to or senses.) ; (No teacher is better than that professor.)!, ý! g 3 s Y. F (In college, I felt the school isn t all that bad.) Table 4. A smmary reslt with 1 for contents and 4 for readability, generated from the article DOCID: By comparing the two smmaries, we fond that the smmary generated with better readability (article

6 Q 4! The Third NTCIR Workshop, Sep Oct Œ P Ž S = F %! 4 [ \ P [ I F % Ÿ ] _ A a [ F 2 [ / 4 c d % X A e 2 f [ D r s A g S F H g (The fixed session was introdced to avoid the majority party blldozing the law establishment by extending the congressional sessions.) *. s * i j Ÿ c 4 l m Ÿ % œ n! % p 4 q r Ÿ s s t Ÿ % œ n! p v w a ˆ i r s P x y 7 / (The same Diet law also permits the one time extension of reglar session and two times for the special and the extraordinary sessions.) [ Q [ 4 z { % q l 2 4 a ˆ % } ~ P 4 «A I D F H s ƒ q % 4 b % Ÿ * ˆ, D % (Bt as the government official says, if the session extension s main prpose is job creation and to qickly implement the economic measres, most Japanese are likely to spport the decision.) F g [ 4 a ˆ % P 4 g S r s * v F 4!. i 2 [ / (Bt it is clear that the prpose of the extension is not limited to these.) a ˆ Ÿ \ P 2 Œ Ž Š : i! v g v g (Politicians of the rling and opposition parties have a long way to go in the coming extended session.) Table 5. A smmary reslt with 1 for contents and 1 for readability, generated from the article DOCID: ) is more readable, meaningfl and natrally combined. The other smmary with lower readability ranking (article ) is not comprehensive enogh and often disrptive, with demonstrative pronon presented withot its precedent. One drawback of or system is lack of mechanism to deal with the discorse strctre of an article. 4 Conclsion In this paper, we propose an atomatic smmarization method combining conventional sentence extraction and trainable classifier based on Spport Vector Machine. To make extraction nit smaller than the original sentence extraction, we also introdce sentence segmentation process in or method. The feasibility and the efficiency of or method are shown in the evalation reslts given to participants grops by TSC-2 task organizer. The reslts show that, or system achieves the best reslts among all others with regard to contents, even close to the hman process (pper bond) at the smmary rate of 20%. On the other hand, or system didn t excel in readability improvement compared to other top systems. More detailed assessment will be necessary to evalate the efficiency of each procedre. References [1] S. Doi, K. Iwata, K. Mraki, and Y. Mitome. Pase control in japanese text-to-speech with lexical discorse grammar. In Proceedings of 3rd International Conference on Spoken Langage Processing (ICSLP 94), pages , [2] H. P. dmndson. New methods in atomatic extracting. ACM, 16(2): , [3] T. Fksima and M. Okmra. Text smmarization challenge: Text smmarization evalation at ntcir workshop2. In Proceedings of the Second NT- CIR Workshop Meeting on valation of Chinese & Japanese Text Retrieval and Text Smmarization., pages 45 50, [4]. Hovy and C.-Y. Lin. Atomated Text Smmarization in SUMMARIST. Chapter 8 in: Advances in Atomatic Text Smmarization, I. Mani and M. T. Maybry (ed.), The MIT Press, [5] T. Joachims. Making Large-Scale SVM Learning Practical. Chapter 11 in: Advances in Atomatic Text Smmarization, B. Schoelkopf and C. J. C. Brges and A. J. Smola (ed.), The MIT Press, [6] S. Kamei and K. Mraki. Proposal of lexical discorse grammer. IIC Technical Report, NLC86-7:1 5, [7] K. Knight and D. Marc. Statics-based smmarization - step one: Sentence compression. In Proceedings of the 17th National Conference on Artificial Intelligence, pages , [8] M. Kobori and N. Tamra. Atomatic smmarization based on the rhetorical strctre of each paragraph and the order of importance between paragraphs. IPSJ SIG Notes, NL-136(11):79 86, [9] J. Kpiec, J. Pedersen, and F. Chen. A trainable docment smmarizer. In Proceedings of the 18th ACM- SIGIR Conference, pages 68 73, [10] H. P. Lhn. The atomatic creation of literatre abstracts. IBM Jornal of Research and Development, 2(2): , [11] W. C. Mann and S. A. Thompson. Rhetorical strctre theory: Toward a fnctional theory of text organization. Text, 8(3): , [12] D. Marc. The Theory and Practice of Discorse Parsing and Smmarization. The MIT Press, [13] T. Nomoto and Y. Matsmoto. The reliability of hman coding and effects on atomatic abstrcting. IPSJ SIG Notes, NL-120(11):71 76, [14] M. Okmra, Y. Haragchi, and H. Mochizki. Some observations on atomatic text smmarization based on decision tree learning. IPSJ 59th Annal Convention, 2: , [15] J. R. Qinlan. C4.5: Programs for Machine Learning. Morgan Kafman Pblishers, 1993.