A Framework for Efficient Document Ranking Using Order and Non Order Based Fitness Function

Transcription

1 A Framework for Efficien Documen Ranking Using Order and Non Order Based Finess Funcion Hazra Imran, Adii Sharan Absrac One cenral problem of informaion rerieval is o deermine he relevance of documens wih respec o he user informaion needs. The choice of similariy measure is crucial for improving search effeciveness of a rerieval sysem. Differen similariy measures have been suggesed o mach he query and documens. This sudy invesigaes he use of Geneic Algorihm o increase he efficiency of informaion rerieval by defining a combined similariy measure. Geneic Algorihm has been used for learning weighs of he componens of he combined similariy measure. We have provided a weigh-learning algorihm for he same. We have considered order based and non-order based finess funcions o evaluae he goodness of he soluion. A non-order based finess funcion is based on recall-precision values only. However, i has been observed ha a beer finess funcion can be obained if we also consider he order in which relevan documens are rerieved. This leads o an idea of order based finess funcions. We evaluaed he efficacy of a geneic algorihm wih various finess funcions. Furher, we provide a framework for applying geneic algorihms o improve he rerieval efficiency by combing various similariy measures. The experimens have been carried ou on TREC daa collecion. The resuls have been compared wih various well-known similariy measures. Index Terms Documen rerieval, geneic algorihms, similariy measures, informaion rerieval, vecor space Model. I. INTRODUCTION Informaion rerieval sysem is devoed o finding relevan documens wih respec o users query. Sandard IR sysems are based on Boolean [], vecor [0], and probabilisic models []. Each model describes documens, queries and provides algorihms o compue similariy beween user's query and documens. The objecive of an informaion rerieval sysem is o provide is users wih saisfacory rerieval resuls. Toward his objecive a rerieval resul should be scienifically measured. Two essenial evaluaion crieria for rerieval success are recall and precision. Precision is defined as he raio of he number of relevan rerieved documens o he oal number of rerieved documens. Manuscrip received December 3, 009. Hazra Imran, is working as an Assisan Professor in Deparmen of Compuer Science, Jamia Hamdard,India. She is pursuing her PhD from Jawaharlal Nehru Universiy, India. (Phone hazrabano@ gmail.com). Dr Adii Sharan, is working as an Assisan Professor in School of Compuers and Sysem Sciences, Jawaharlal Nehru Universiy, India ( adiisharan@mail.jnu.ac.in). Precision = no. of relevan documens rerieved () no. of documens rerieved Recall is defined as raio of number of relevan rerieved documens o he oal number of relevan documens. Recall = no. of relevan documens rerieved () Toal no. of relevan documens Our proposed framework is based on Vecor Space Model (VSM). In VSM, boh documens and he user given query are represened as vecor of erms. Suppose here are index erms in a collecion of documens. Then documen D i and query Q can be represened as D i =(d i, d i,... d i ) Q=(w q,w q,...w q ) Where d ( o ) are erm weighs in documen D i and w qj ( o ) are erm weighs in he query Q. There are differen mehods of assigning weighs o d. Mos popular mehod being used is TFXIDF. Here TF (erm frequency) measures he number of imes a erm appears in a documen or query. The IDF (inverse documen frequency) is based on he inuiion ha a erm, which occurs in many documens, is no a good discriminaor and should be given less weigh han one, which occurs in few documens. One of he ways of calculaing IDF is log(n/df) where N is he oal number of documens in he collecion and DF is he number of documens in which erm has appeared in enire documen collecion. Oher measures can be found in [4]. For calculaing similariy of query and documen cosine and jaccard are he popularly used similariy measures. Cosine, jaccard and dice similariy measures are defined as follows: Cos (Q,D i ) = w qj d ( d ) ( w qj ) (3)

2 JACCARD(Q,D I )= ( d ) + wqj) ( w d qj w d qj (4) III. FRAMEWORK Figure presens overall framework of our proposed sysem. The inpu o he framework is vecor represenaions of documen and query and oupu is he opimized weighs for combined similariy measure.we will now discuss he working of each module. User Query Documen Collecion Dice = wd qj ( d ) ( wqj ) (5) Maching Module SM SM SM n Populaion Module W W W n In his paper we focus on applicabiliy of GA for improving relevance of rerieved documens wih respec o user query. The mehod is esed on real documen collecions TREC and resuls are very encouraging. The paper is organized as follows: Secion deal wih he inroducion of informaion rerieval sysem. Secion deals wih Combined Similariy measure using Geneic algorihm. Secion 3 discusses he framework and he algorihm in deail. Secion 4 focuses on he experimens and resuls. Secion 5 concludes he work. Combined similariy Module Finess Module II. COMBINED SIMILARITY MEASURE USING GENETIC ALGORITHM In our problem, we have defined a combined similariy measure consising of sandard similariy measures. We hen assign appropriae weighs o he componens of hese measures so as o achieve maximum rerieval efficiency. Our combined similariy measure is a weighed sum of he scores reurned by differen maching measures. In general a combined similariy measure is represened as:- combined _ similariy _ measure ( D, Q) = M i= ( w SM i i ( D, Q)) (6) Where SM i (D,Q) signifies he score of documen D for given query Q for i h similariy measure. M is oal number of sandard similariy measures considered. We have used wo similariy measures: cos(sm ) and jaccard(sm ) (Refer Eqn 3 and 4). w i is he weigh assigned o i h similariy measure. Weighs w and w range from 0.0 o.0. A less weigh signifies ha associaed similariy measure is less significan. We have used GA for finding opimal se of weighs. Geneic Algorihms (GA) comes under he inelligen search mehods. GA s apply naural evoluion process in arificial inelligence o find a globally opimal soluion o he opimizaion problem. Geneic Algorihms are learning algorihms as well as offer a domain independen search abiliy ha can be used in many learning asks. Due o his reason, he applicaion of GAs o IR has increased in he las decade [5,6,8, 9, ]. In nex secion we presen our framework for finding opimal weighs using combined similariy measure. Fig.. Framework of Proposed Sysem 3. Maching Module The inpus o he module are he vecor space represenaions for user query and he documens. Maching module calculaes he similariy of he documens and query wih respec o differen similariy measures given in above equaions and reurns hese values as SM,SM, SM n. 3. Populaion Module GA Module (selecion, crossover and muaion) The populaion module generaes chromosomes for iniial populaion of GA. The chromosome is represened in he following way: w w w n Where w i = weigh of he i h similariy measure. We have used a real-valued chromosome because i is more naural represenaion for our problem and i decreases dramaically he number of genes required o specify a design, hus making he soluion space easier o search. 3.3 Combined Similariy Module Considering similariy measures and weighs as inpu, combined similariy module calculaes similariy of he documen wih respec o combined similariy measure using equaion 6.

3 3.4 Finess Module Finess module is used o find he finess of he soluion Finess Funcion Finess funcion is a performance measure ha is used o evaluae how good each soluion is. Given a chromosome, he finess funcion mus reurn a numerical value ha represens he chromosome s uiliy. Previous work has been done in GA considering finess funcions based on recall and precision only. However i has been observed ha a beer finess funcion can be obained if we also consider he order of he rerieval of documens. In our work we have compared he order and non-order based finess funcions. Non order- based finess funcion is based on recall and precision only. We have used following non-order and order- based finess funcions [,7]. ( recall precision) Finess non order-based = recall + precision D D i D i= j Finess order-based = r( di ) Where D is he oal number of documens rerieved, and r(d) is he funcion ha reurns he relevance of documen d, giving if he documen is relevan and a 0 oherwise. Order-Based Finess funcion akes ino accoun he number of relevan and irrelevan documens along wih he order of heir appearance. Noe ha he documens rerieved earlier in order have higher uiliy in comparison o documens rerieved laer. The basic idea being ha i is no he same ha he relevan documens appear a he beginning or a he end of he lis of rerieved documens. The imporance of he order-based finess funcion can be jusified wih he following example. Le us assume ha wo similariy measures rerieve 6 relevan documens in he op 5 resul giving Case and Case, where Case = Case = The relevance informaion is coded as (if relevan) and 0 (if irrelevan). Here we observe ha he recall in boh he cases is same. However one can easily see ha Case has a beer performance as all he 6 documens are rerieved earlier in comparison o Case. If we consider he oal number of relevan documen as 0 hen he values for Finess non order-based is.48 for boh he cases whereas Finess order-based are calculaed as 0.45 for Case and 0.64 for Case. Therefore we can jusify ha Finess order-based is beer Working of Finess Module The finess module sors he populaion on he basis of he combined similariy measure obained. Furher precision and recall are calculaed and finally he finess is calculaed for each member of he populaion using equaion 7 and 8. Once (7) (8) he finess is calculaed nex modules applies sandard GA funcions: selecion, crossover and muaion and generaes new populaion. The whole process is repeaed ieraively unil populaion converges or maximum number of ieraions has been carried ou. 3.5 Geneic Algorihm Module This module applies he sandard GA funcions (selecion, crossover, muaion) o generae new populaion from he old populaion, which is discussed as follows. Selecion: - Selecion embodies he principle of survival of he fies. Chromosomes having higher finess are seleced for crossover. The roulee wheel reproducion process [3] was used o selec individuals. Crossover: - Crossover is he geneic operaor ha combines wo chromosomes ogeher o form new chromosome. We have used single poin crossover where a locus posiion is seleced wihin wo paren chromosomes and he genes are swapped from ha posiion o he end of paren. Muaion: - Muaion involves he modificaion of he value of each gene of a soluion wih some probabiliy (muaion probabiliy). Paren A Child A 3.6Algorihm Paren A Paren B Child A Child B We have used he following algorihm o find he weighs of erms in CSM using Geneic Algorihm. GA_combined_similariy_measure (query, documens) Begin Indexing (documens and query) Build TF-IDF erm documen marix For each documen find SM, SM,... SM n Generae iniial random populaion w, w... w n Repea For each member of populaion and for each documen Find combined_similariy_measure For each member of populaion

4 Finess_calculaion(curren populaion, combined_ similariy _measure for all documens) Perform finess based selecion, crossover, muaion Make rank based selecion and generae new populaion Unil finess value is sabled OR reaches o maximum number of generaions End Finess_calculaion ( populaion, combined similariy measure for all documens) Begin Sor he documens based on combined similariy measure Selec op N documens Find recall and precision values Calculae finess value using formula given End [ , , ] Finess : [ , , ] Finess : Below are he graphs o show he resuls of experimen. IV. EXPERIMENTS AND RESULTS For our experimens, we used volume of he TIPSTER documen collecion, a sandard es collecion in he IR communiy. Volume is a. Gbye collecion of full-ex aricles and absracs, divided ino seven main files. The documens came from he following sources.. WSJ -- Wall Sree Journal (986, 987, 988, 989,990,99 and 99). AP -- AP Newswire (988,989 and 990) 3.ZIFF -- Informaion from Compuer Selec disks (Ziff-Davis Publishing) 4. FR -- Federal Regiser (988) 5. DOE -- Shor absracs from Deparmen of Energy Fig.. Curve showing variaion of Average finess wih Generaion Number for boh he finess funcion The experimens were done for non-order-based (Finessnonorderbased) and order based (Finessorderbased) finess funcions. The conrol parameers were as follows: populaion size=00, probabiliy of crossover (Pc = 0.7), and probabiliy of muaion (Pm = 0.0).The documen cu off was 0. We have performed our experimen on 50 queries. Below is he example of applying Geneic Algorihm on TREC Query 50. Analyzing query: - Topic: Airbus Subsidies Descripion: Analyzing query: - Topic: Airbus Subsidies Descripion: Documen will discuss governmen assisance o Airbus Indusrie, or menion a rade dispue beween Airbus and a U.S. aircraf producer over he issue of subsidies. [ , , ] Finess : Fig. 3. Curve showing variaion of Precision wih Generaion number for boh he finess funcion [ , , ] Finess :

5 Fig. 4. Curve showing variaion of Recall wih generaion number for boh he finess funcions. Figure shows variaion of average finess wih generaion number for a specific query 50 of TREC daase. As shown he average finess increases wih generaion number. On he basis of our experimen we observed ha for 55% and 60% of he queries of TREC daase has increased average finess in successive generaions. Figure 3 shows curve showing variaion of Precision wih Generaion Number for boh he order and non order based finess funcions and Figure 4 shows curve showing variaion of Recall wih generaion Number for boh he finess funcion. We compared he resuls of our experimens wih he recall and precision values obained by all hree sandard similariy measure ie. Cos, Jaccard Dice and combined similariy measure. For almos all he queries our experimen gave beer resuls. Figure 5 shows recall precision curve for all he hree sandard measure for a specific query in TREC. Documen cuoff was 0. Figure 6 shows recall precision graph for GA based experimen for he same daase for he combined similariy measure. Recall-precision curve is a sandard curve for measuring efficiency of any rerieval algorihm. They are useful because hey allow us o evaluae quaniaively boh qualiy of overall answer se and breadh of rerieval algorihm. Good qualiy informaion rerieval algorihm gives high precision a low recall values. By comparing figure 5 wih figure 3 and 4 one can easily observe ha iniially all similariy measures sar wih high precision values. However sandard similariy measures show a sharp decline in he beginning whereas our combined similariy measure shows gradual decrease in precision. This indicaes ha combined similariy measure gives beer qualiy of resul. The breadh of combined similariy measure is maximum as i is giving a recall value. V. CONCLUSION Fig. 5. Curve showing Recall and Precision Graph of combined Similariy Measure In his paper we have focused on he applicaion of Geneic Algorihm for documen ranking. We have used a combined similariy measure consising of differen sandard similariy measures. GA was used for learning he weighs of he componens of he combined similariy measure. The paper provides a framework for learning opimized weighs used in combined similariy measure.. The algorihm was implemened for TREC daase wih he order and non order based finess funcions. The resuls were compared wih sandard similariy measures using sandard recall precision graphs. I was expeced ha order based finess funcion should gives beer resul. As expeced he empirical resul verified he same. Therefore we can conclude ha i is desirable o use finess funcions ha value no only wheher he possible soluion rerieves many relevan documens and few irrelevan documens, bu also wheher he relevan documens are a he op of he rerieval lis or a he end. REFERENCES Fig. 6. Curve showing Recall Precision Graph of cos,jaccard and Dice Coefficien [] A. Booksein, Probabiliy and fuzzy-se applicaions o informaion rerieval, Annual Review of Informaion Science and Technology, 0, pp: 7-5, 985. [] C.-H Chang, The design of an informaion sysem for hyperex rerieval and auomaic discovery on WWW, PhD hesis, Deparmen of CSIE, Naional Taiwan Universiy,999.

6 [3] D. E. Goldberg, Geneic Algorihms in Search, Opimizaion, and Machine Learning, Addison-Wesley, Reading, Mass,989. [4] G. Salon and C. Buckley, Term weighing approaches in auomaic ex rerieval, Informaion Processing and Managemen, vol. 4, no. 5, pp , 988. [5] R. José, Pérez-Agüera, Using Geneic Algorihms for Query Reformulaion, BCS IRSG Symposium: Fuure Direcions in Informaion Access (FDIA 007), 007. [6] D.H. Kraf, e. al. The Use of Geneic Programming o Build Queries for Informaion Rerieval, In Proceedings of he Firs IEEE Conference on Evoluional Compuaion. New York: IEEE Press., PP ,994. [7] K. L. Kwok, Comparing represenaions in Chinese informaion rerieval, In Proceedings of he ACM SIGIR, pages 34-4,997. [8] Lourdes Araujo, R Jose, Improving Query Expansion wih Semming Terms: A New Geneic Algorihm Approach Evoluionary Compuaion in Combinaorial Opimizaion,008 [9] M.J. Marin-Bauisa, e. al. An Approach o An Adapive Informaion Rerieval Agen using Geneic Algorihms wih Fuzzy Se Genes, In Proceeding of he Sixh Inernaional Conference on Fuzzy Sysems. New York: IEEE Press. PP.7-3.,997. [0] S. E. Roberson, The probabilisic characer of relevance, Informaion Processing & Managemen, 3, pp: 47-5,997. [] G. Salon, & M. McGill, "Inroducion o Modern Informaion Rerieval", New York: McGraw-Hill, 983. [] Z. Michalewicz, Geneic Algorihms +Daa Srucures=Evoluion Programs,Springer-Verlag, 996.