Complex Question Answering with ASQA at NTCIR 7 ACLIA

Transcription

1 Complex Queston Answerng wth ASQA at NTCIR 7 ACLIA Y-Hsun Lee 1, Cheng-We Lee 12, Cheng-Lung Sung 1, Mon-Tn Tzou 1, Chh-Chen Wang 1, Shh-Hung Lu 1, Cheng-We Shh 1, Pe-Yn Yang 1, Wen-Lan Hsu 1 1 Insttute of Informaton Scence, Academa Snca, Tawan, R.O.C 2 Department of Computer Scence, Natonal Tsng-Hua Unversty, Tawan, R.O.C correspondng author {rog,aska,clsung,mttzou,vncent,journey,dap,annabel,hsu}@s.snca.edu.tw Abstract At NTCIR 7, we mplemented the Academa Snca Queston Answerng (ASQA) system for complex questons. The system uses three methods to select answer strngs from a news corpus. (a) It uses syntactc patterns, whch are usually used by QA systems, to retreve more precse answer strngs than those derved by tradtonal IR. (b) Usng external knowledge, the system can fnd accurate answers to specfc questons that the tradtonal IR approach can not process. (c) Entropy-based and co-occurrence-based mnng methods are used to retreve relevant answer strngs for document retreval. In the NTCIR 7 CCLQA task, ASQA acheved 0.26 n the CT-CT task and 0.20 n the CS-CS task. 1. Introducton Because of the hgh level of nformaton overload on the Internet, research nto queston answerng, whch focuses on how to respond to users queres wth exact answers, s becomng ncreasngly mportant. In recent years, many nternatonal queston answerng contest-type evaluaton tasks have been held at conferences and workshops, such as TREC [3], CLEF [1], and NTCIR [2]. At NTCIR 7, the systems were requred to process complex questons related to defntons, bographes, relatonshps, and events. Ths year, our system focused on external knowledge, syntax patterns and some data mnng technques. We ntroduce the methods n Secton 5. The remander of ths paper s organzed as follows. Secton 2 provdes an overvew of the system archtecture. In Sectons 3 and 4, we ntroduce the queston analyss strategy and document retreval method, respectvely. In Secton 5, we dscuss the methods used at NTCIR 7. In Secton 6, we consder the redundancy removal module. Secton 7 evaluates the system performance, and Secton 8 contans a dscusson. Secton 9 summarzes our conclusons. 2. System Descrpton Our queston answerng system ASQA (Academa Snca Queston Answerng) for complex questons s dvded nto fve modules: queston processng, document retreval, sentence selecton, answer rankng, and redundancy removal. Questons are frst analyzed by the queston processng module to extract the keywords and queston topc, whch are then used to retreve documents from the corpus. In the next step, the sentence selecton module obtans the sentence scores, after whch the sentence rankng module determnes whether or not the sentences are relevant. Fnally, the redundancy removal module flters smlar sentences as answer strngs. 3. Queston Analyss The queston processng module uses smple surface patterns to classfy questons nto four types: (a) defnton, (b) bography, (c) relatonshp, and (d) event. For example, bography questons always start wth, whch we call a queston term. The man topcs of such questons are usually adjacent to the queston terms, so we use surface text patterns to retreve a queston s topc. In bography 70

2 questons, we consder that the queston topc and the queston keywords should be the same. In the other queston types, we use a keyword extractor to extract keywords from queston topcs. To ensure that a queston s not dvded nto meanngless small peces by the keyword extractor, we retreve Wkpeda ttles to address the problem. If the queston topc appears n the Wkpeda ttles, we set the keywords as the queston topc. 4. Document Retreval The document retreval module uses Lucene, an open source nformaton retreval engne, to ndex documents by character. In the retreval process, we use the AND operator to form a query strng comprsed of queston topcs, whch s then sent to the Lucene engne. If the number of retreved documents s larger than a threshold, we termnate the retreval process; otherwse, we use the keywords to form a new query strng. If the number of retreved documents s stll less than the threshold, we use the OR operator to combne the keywords as a query strng. After retrevng the documents, we splt each one nto several sentences. 5. Sentence Selecton The sentence selecton module uses co-occurrence-based and entropy-based methods, syntactc patterns, and external knowledge to fnd relevant sentences. We descrbe the process n detal n followng sub-sectons Co-occurrence-based Sentence Selecton Magnn et al. [5] consder that the number of documents n whch the queston terms and the answer co-occur s useful for QA. The hypothess s smlar to that of Clarke et al. [4], who use co-occurrence methods to measure the relevance of an answer to the gven queston based on Web search results. Although the method s used n factod QA, we thnk the hypothess could also be useful for complex QA. Our ratonale s that, n good qualty passages, the more often a term co-occurs wth the queston topc, the hgher the confdence that the passage wll be relevant. We regard a co-occurrence as an ndcaton that the passage can be taken as correct based on the co-occurrng queston topcs. Let the gven term be t and the gven queston be Q, where Q conssts of a set (QT) of queston terms {qt1, qt2, qt3,, qtn} created from the queston processng result of Q. We use the followng equaton to calculate each term s weght: p( t Q) frequency( t Q) / freuqency( Q), p( t ) frequency( t ) / N where P (t QT) denotes the probablty of the term when QT appears; N s the number of documents n the corpus; and P(t) s the probablty of the term occurrng n the news corpus. We consder that the probablty P (t QT) should be sgnfcantly dfferent to the probablty P(t). After calculatng the probabltes of the terms, we use the followng hypothess test to determne whether or not the term s relevant: H 0 : p( t Q) p( t ). H : p( t Q) p( t ) Entropy-based Sentence Retreval Inspred by the work of Xu and Croft [6], we use the concept of entropy, whch s a measure of uncertanty n nformaton theory, to perform sentence retreval n our queston answer system. Ths concept s smlar to the nverse document frequency (IDF) n nformaton retreval. If the entropy value of a term s large, the term s regarded as a stop word or keyword n some topc. For example, the term poltcal party should be mportant n poltcal news. The entropy value of each term we use s calculated by: n Entropy( t) p( t ) log p( (1) 1 10 t ) Let t and n denote the gven term and the number of retreved documents respectvely. Then, the probablty of t s calculated by frequency( t ) P( t ) (2) Total( t) where frequency( t ) represents the number of passages n whch the term t occurs, and Total (t) denotes the number of retreved passages n whch t occurs. Next, we select relevant sentences based on ther entropy values. We adopt the vector space model to calculate the smlarty between the retreved passages and an entropy vector, whch s constructed from the terms whose entropy values are greater than a threshold. In our system, the threshold s set at 0.6. In addton, we use the IDF to flter stop words. The sentence vector s constructed based on the segmentaton result. Each weght of the vector represents the term s entropy value or a default value. Because the entropy method s data-drven, we only use t f the queston topc appears n several documents. 71

3 5.3. External-Knowledge-based Sentence Retreval There are several knowledge databases, such as WordNet, HowNet, and Wkpeda. The knowledge they contan should be useful for QA systems. To support our ASQA, we use Wkpeda, an onlne encyclopeda whose content s provded by Internet users. If the keywords of a queston appear n a ttle n Wkpeda, the contents of the ttle should be the standard answers that also appear n the corpus. In addton, we consder the case where the queston does not appear on a Wkpeda page. To address ths problem, we employ the three methods descrbed n the followng sub-sectons Related Term-based Sentence Retreval In ths sentence retreval method, we also use entropy values to fnd related terms. A low entropy value ndcates a topc-specfc term, whch should be useful for dentfyng answer sentences. However, n our experence, terms wth small entropy values are not useful when the corpus s derved from news artcles or the Web, because Web data or corpus data contans too much nose. Therefore, we only apply the entropy method to the Wkpeda data because t contans less nosy nformaton. For example, n Wkpeda Lesley who s the daughter of Ma Yng-jeou only co-occurs wth Ma Yng-jeou, but on the Web, Lesley co-occurs wth many terms that are not of nterest to us. Obvously, ths sentence selecton method can only be used when the queston topc exsts n Wkpeda. Equaton (1) s used to calculate each term s entropy value. Then, we apply the followng equaton to calculate the sentence score: If entropy(term ) 0 m Score( s) 1 (3) entropy( term 1 ) The smaller a term s entropy value, the more relevant the term wll be. If a sentence contans many relevant terms, t wll be taken as the standard answer. Therefore, we aggregate the nverse of the terms entropy values as the score of s Defntonal Term-based Sentence Retreval Some terms or phrases provde useful hnts for dentfyng defntonal sentences. For example, the phrase s a knd of s a clear ndcaton that the sentence s about the type or category of an object or term. Snce such phrases or terms are used a great deal n Wkpeda, we collect the Wkpeda pages to extract such phrases or terms, whch we call defntonal terms. Ths sentence selecton method uses Equaton (1) to fnd defntonal terms. In contrast to the method descrbed n sub-secton 5.3.1, whch utlzes small entropy values to locate related terms, we want to fnd terms or phrases wth large entropy values that occur n most Wkpeda artcles. However, these knds of terms could also be stop words or defntonal terms, so they must be fltered out. We assume that stop words are evenly dstrbuted n any corpus. If a term has a hgh entropy value or a hgh IDF value n both the Wkpeda corpus and the News corpus, the term s probably a stop word. Fnally, we calculate the score of a sentence by aggregatng the scores of terms that have hgh entropy values as follows: 0 If entropy(term ) threshold m Score( s) (4) entropyterm ( ) Syntactc Surface Pattern-based Sentence Retreval Syntactc patterns are useful for fndng precse nformaton for bography questons. Next, we descrbe how we construct bographcal syntactc surface patterns (SPs). 72

4 Table 1. TREC BographyQueston Type Q Types Year 2003 Year 2004 Year 2005 Year 2006 Sum percentage Appellaton % Ancestral Home % Natonalty % Domcle % Brth/Death Day and Place % Race % Famly % Parentage % Occupaton % Educaton % Belef % Characterstcs % Honour % Contrbuton % Notables % Organzaton % Quotatons % Publcatons % Dsease % Dsablty % Msadventure % Scandal % Controversy % Others % Syntactc Pattern Generaton To construct SPs for bography questons, we need To construct SPs for bography questons, we need categores of bographcal nformaton. Snce 2008 was the frst year that TREC consdered bography questons, we had to use Englsh tranng data for our analyss. We took the test questons from TREC 2003 to TREC 2006 and categorzed them nto 24 types. Some of the queston types were combned based on ther smlarty, whch resulted n the 20 types shown n Table 1. Then, from the bographcal nformaton n the table, we created our own Chnese tranng questons and answer nuggets to tran the SPs. We created the SPs sem-automatcally from the followng sources. 1. Algnment-based Template Generaton: For relatonal nformaton, such as the relatonshp between a husband and wfe, we collected the relatonal term pars and appled our algnment-based template generaton method to obtan prelmnary sentence patterns. Then, hgh frequency patterns were rewrtten nto answer templates by our analysts. 2. Sentence makng: We tred to thnk of possble bographcal answers for each type ourselves and then wrote correspondng templates. 3. Internet: Defntonal sentences n bographcal artcles obtaned from the Internet, such as Wkpeda, were also referenced durng template formaton. 4. News Corpus: Our analysts used the Lucene search engne to extract artcles about the targeted person from the CIRB20 and CIRB40 corpora, and then selected relevant defntonal sentences n the artcles 73

5 Table 2. Bography Template Type Bography20Attrbutes Instance Rule TOTAL Rules Appellaton Belef Characterstcs Contrbuton Controversy Domcle DPBD&Natonalty Educaton Famly Honour IPC Msadventure Notables Org&Occ Others Parentage Publcatons Quotatons Race Scandal TOTAL to make correspondng templates Negaton Term To help screen out unwanted sentences whle keepng the good ones, we added two categores of keywords: Terms to be excluded and Terms that should not be excluded. Most of the content of the frst category s comprsed of negatve terms, whle the second contans double negatves. The followng are some examples of the two types of terms; the words n parentheses are the lteral meanngs of the Chnese terms: 1. Terms to be excluded: (reject)(ban) (cant)(wont)(should) (Dont thnk about)(questonng partcle) (rumor)(not necessary) 2. Terms that should not to be excluded: (cant help but)(cantwthout ) (no..not...) (no other than...) (none...not...) Punctuaton In Chnese, the sentence structures of declaratves and questons are often the same. If a queston does not have a questonng partcle, such as or, t would be very dffcult for a computer program to dstngush t from a declaratve. Therefore, a sentence that matches our templates may be just a queston, rather than a declaratve that we wsh to retreve. To avod ths problem, all sentences endng wth queston marks were deleted when retrevng sentences, unless the queston marks were part of the matched templates. 74

6 6. Redundancy Removal Rankng Canddates ALL Best IASL-CT-CT-01-T IASL-CS-CS-01-T X Term OverLap 8. Dscusson and Error Analyss N-gram Overlap VSM Model Our analyss of the experment results dentfed certan problems that need to be addressed. We dscuss them n the followng sub-sectons Segmentaton of NER problems Answer Strngs Fgure 1 Redundancy Removal FlowChart A news corpus may contan a great deal of redundant nformaton, especally f the news s hot, because smlar nformaton could be mentoned n dfferent news artcles. To handle ths problem, we need a module to flter out smlar sentences. We adopt a two-stage redundancy removal approach, as shown n Fgure 2. Frst, we use the concept of term overlap to remove smlar sentences. The sentence selecton module dentfes relevant terms, defntonal terms, and entropy terms. In ths stage, we set a new threshold to fnd good terms from the set of terms derved by the sentence selecton module. It s assumed that these terms would be the same n smlar sentences. In the second stage, f we have answer canddates that match some of the syntactc patterns descrbed n sub-secton 5.4, we use an N-gram overlap method to remove smlar sentences; otherwse, we use a vector space model to remove such sentences. 7. System Performance In the NTCIR 7 CCLQA Task, we acheved an F-score of n CT-CT and n CS-CS. Among the four queston types, our system acheved a hgher F-score for bography questons than for the other three types. The system dd not perform well on event questons because we only used IR technques to retreve relevant sentences. All of our system s modules are based on tradtonal Chnese; therefore we have to convert smplfed Chnese questons and corpora. However, as shown by the results, the process degrades the system s performance. Because we only used a character-based ndex for document retreval and dd not use an NER technque to flter out wrong documents wth dfferent people s names, we encountered some problems when nformaton about the wrong person was retreved. For example, there was a queston. However, there was another person called n the news corpus. Examples lke ths cause the performance to deterorate on some questons Dfferent Perspectves between Tranng and Test Data Accordng to the tranng data provded for the task, most of the answers were summarzaton-orented. The answers contaned a lot of related nformaton about the queston focus, and they were qute dfferent from the answers for factod questons. Unfortunately, we found a dscrepancy between the tranng data and test data because some of the test questons were judged as f they were factod questons. For example, there was a queston. In the tranng set, the answers to ths knd of queston would contan nformaton lke brthplace, brthday, famly or occupaton, but there s only a small porton of ths nformaton n the standard answers n the test data. The above-mentoned dscrepancy had a substantal mpact on the system s performance Inconsstency n Treatng People wth Same Name For questons lke and, there s more than one person wth the same name n the news corpus. We found that the task evaluators treated them n dfferent ways. For, there are at least fve people wth that name n the news corpus, but only the person who does gymnastcs was labeled as correct. However, for, all the answers related to dfferent were labeled as correct. Some errors 75

7 were caused by ths nconsstency Unmarked Correct Answers System responses judged as correct answers should be marked, yet some correct answers were unmarked. Ths degraded our system s performance substantally. For example, the standard answer, whch appeared n our system response, was unmarked. for Answer Valdaton, Proceedngs of the 40th Annual Meetng on Assocaton for Computatonal Lngustcs, 2001, pp [6] J. Xu and W. B. Croft, Query expanson usng local and global document analyss, Proceedngs of the 19th annual nternatonal ACM SIGIR conference, Zurch, Swtzerland, 1996, pp Concluson For NTCIR 7 CCLQA, we bult a complex queston answerng system. Snce the task s complex, we assume the answers are summarzaton-orented, whch means they contan varous types of nformaton that requres careful flterng. We used several syntax patterns to select precse answers when dealng wth bography questons, and used the entropy method to select defntonal terms and related terms from Wkpeda for defnton questons and relaton questons respectvely. Even though the pattern-based methods were not hghly accurate, we stll encountered the low coverage problem. Therefore, we adopted the co-occurrence and entropy-based methods to fnd relevant sentences. 10. Acknowledgments Ths research was supported n part by the Natonal Scence Councl of Tawan under Center of Excellence Grant NSC E PAE; and by the Research Center for Humantes and Socal Scences, Academa Snca, and the Thematc Program of Academa Snca under Grant AS 95ASIA02. We wsh to thank the Chnese Knowledge and Informaton Processng Group (CKIP) of Academa Snca for provdng us wth AutoTag for Chnese word segmentaton. 11. References [1] Cross Language Evaluaton Forum (CLEF), [2] NTCIR Workshop, [3] Text REtreval Conference (TREC), [4] C. L. A. Clarke, G. Cormack, G. Kemkes, M. Laszlo, T. Lynam, E. Terra and P. Tlker, Statstcal Selecton of Exact Answers (MultText Experments for TREC 2002), Proc. of TREC, 2002, pp [5] B. Magnn, M. N. R. Prevete and H. Tanev, Is t the rght answer?: explotng web redundancy 76