A Knowledge-Based Feature Selection Method for Text Categorization

Transcription

1 A Knowledge-Based Feature Selecton Method for Text Categorzaton Yan Xu,2, JnTao L, Bn Wang,ChunMng Sun,2 Insttute of Coputng Technology,Chnese Acadey of Scences No.6 Kexueyuan South Road, Zhongguancun,Hadan Dstrct, Beng,Chna {xuyan, tl, wangbn, 2 Dept.of coputer scence,ncepu(bj),beng Abstract. A aor dffculty of text categorzaton s the hgh densonalty of the orgnal feature space. Feature selecton plays an portant role n text categorzaton. Autoatc feature selecton ethods such as docuent frequency thresholdng (DF), nforaton gan (IG), utual nforaton (MI), and so on are coonly appled n text categorzaton. Many exstng experents show IG s one of the ost effectve ethods. In ths paper, a ethod s proposed to easure attrbute s portance based on Rough Set theory. Accordng to Rough set theory, knowledge about a unverse of obects ay be defned as classfcatons based on certan propertes of the obects,.e. Rough set theory assues that knowledge s an ablty to partton obects. We quantfy the ablty of partton obects, and call the aount of ths ablty as knowledge quantty, and than put forward a knowledge-based feature selecton ethod called KG. Experental results on NewsGroup and OHSUMED corpora show that KG perfors uch better than MI, DF, even than IG. Introducton Text categorzaton s the process of groupng texts nto one or ore predefned categores based on ther content. Due to the ncreased avalablty of docuents n dgtal for and the rapd growth of onlne nforaton, text categorzaton has becoe one of the key technques for handlng and organzng text data. A aor dffculty of text categorzaton s the hgh densonalty of the orgnal feature space. Consequently, feature selecton-reducng the orgnal feature space s serously proected and carefully nvestgated. In recent years, a growng nuber of statstcal classfcaton ethods and achne learnng technques have been appled n ths feld. Many feature selecton ethods such as docuent frequency thresholdng(df), nforaton gan easure(ig), utual nforaton easure(mi), and so on have been wdely used. Many exstng experents show IG s one of the ost effectve ethods[][2][3]. It s well known that the attrbutes are not portant equally n nforaton syste, but, whch attrbute s portant and whch attrbute s unportant, even redundant? For nforaton easureent, the feld of Inforaton Theory has ts orgn n Claude Shannon's 948 paper "A Matheatcal Theory of Councatons". Wrtten

2 nforaton has the property of reducng the uncertanty of a stuaton. The easureent of nforaton s thus the easureent of the uncertanty. Rough Set theory, whch s a very useful tool to descrbe vague and uncertan nforaton, regards knowledge as an ablty to partton obects. We quantfy the ablty of partton obects, and call the aount of ths ablty as knowledge quantty. Accordng to the knowledge quantty we propose a rough set feature selecton ethod naed knowledge-gan ethod(kg). Experental results on NewsGroup and OHSUMED corpora show that KG perfors uch better than MI, DF even than IG. 2 Feature selecton ethods and rough set theory In ths secton we reexane feature selecton ethods DF, IG and MI whch are coonly used n feature selecton for text categorzaton, and DF, IG both have good perforance for feature selecton n TC, wth IG beng respected to have the best perforance n any experents[][2][3]. The followng defntons of DF, IG and MI are taken fro []. 2. Docuent frequency thresholdng Docuent frequency s the nuber of docuents n whch a ter occurs. Only the ters that occur n a hgher nuber of docuents are retaned. DF thresholdng s the splest technque for vocabulary reducton. It easly scales to very large corpora wth a coputatonal coplexty approxately lnear n the nuber of tranng docuents. 2.2 Inforaton gan Inforaton gan s coonly used as a ter goodness crteron n achne learnng [4][5]. It easures the aount of nforaton obtaned for category predcton by knowng the presence or absence of a ter n a docuent. Let { c} = denote the set of categores n the target space. The nforaton gan of ter t s defned to be: G( = c )log c ) + c log c + = = + = c log c Gven a tranng corpus, for each unque ter the nforaton gan s coputed and those ters whose nforaton gan s less than soe predeterned threshold are reoved fro the feature space.

3 2.3 Mutual nforaton Mutual nforaton s a crteron coonly used n statstcal language odelng of word assocatons and related applcatons. Gven a category c and a ter t, the utual nforaton crteron between t and c s defned as: p ( t c) I( t, c) = log2 p ( p ( c) These category-specfc scores of a ter are then cobned to easure the goodness of the ter at a global level. Let } denote the set of categores n the { c = target space. Typcally t can be done n two alternate ways: I avg ( = = p ( c ) I( t, c ), Iax( = ax{ I( t, c )} After the coputaton of these crtera, thresholdng s perfored to acheve the desred degree of feature elnaton fro the full vocabulary of a docuent corpus. = 2.4 Basc concepts of rough set theory Rough set theory, ntroduced by Zdzslaw Pawlak n 982 [6][7][8], s a atheatcal tool to deal wth vagueness and uncertanty. At present t s wdely appled n any felds, such as achne learnng, knowledge acquston, decson analyss, knowledge dscovery fro databases, expert systes, pattern recognton, etc. In ths secton, we ntroduce soe basc concepts of rough set theory whch used n ths paper. Gven two sets U and A, where U ={x,..., x n } s a nonepty fnte set of obects called the unverse, and A = {a,, a k } s a nonepty fnte set of attrbutes, the attrbutes n A s further classfed nto two dsont subsets, condton attrbute set C and decson attrbute set D, A=C D and C D = Φ. Each attrbute a A, V s the doan of values of A, Va s the set of values of a, defnng an nforaton functon fa, : U Va, we call 4-tuple <U,A,V, f > as an nforaton syste. a(x) denotes the value of attrbute a for obect x. Any subset B A deternes a bnary relaton Ind(B) on U, called ndsceblty relaton: Ind(B)={ (x,y) U U a B, a(x) = a(y) } The faly of all equvalence classes of Ind(B), naely the partton deterned by B, wll be denoted by U/B. If ( x, y ) Ind(B), we wll call that x and y are B- ndscernble.equvalence classes of the relaton Ind(B) are referred to as B - eleentary sets. The eleentary sets are the basc blocks of our knowledge about realty, soetes called as concepts. 2.5 Relevant work Many text categorzaton ethods are presented and soe works are based on Rough Set. The paper[] locates a nal set of coordnate keywords frstly to dstngush

4 between classes of docuents, and then use rough set to reduce the densonalty of the keyword vectors. The paper[0] proposes a hybrd technque usng Latent Seantc Indexng (LSI) and Rough Sets theory. The paper[9] ntroduces a hybrd ethod to select features usng soe feature selecton ethod and rough set theory, t selects features frstly usng one of feature selecton ethods, such as utual nforaton, nforaton gan, and then further select features usng rough set. In text categorzaton, each docuent s descrbed by a vector of extreely hgh densonalty, so ost rough set-based ethods use rough set and other technque at the sae te, such as [9][0][]. Ths paper does not use other technque, but only accordng to rough set theory that knowledge s an ablty to partton obects. We quantfy the ablty of partton obects, and call the aount of ths ablty as knowledge quantty, then put forward a knowledge-based feature selecton ethod KG. 3 Knowledge easureent based on Rough Set 3. The ablty to dscern obects An exaple of an nforaton table s gven n Table [2]. Rows of table, labeled wth E, E2,,E8, are eleents(obects), the features are X and Y, where X=College Maor, Y=Lkes Gladator : Table. An nforaton table for text X Y E Math Yes E2 Hstory No E3 CS Yes E4 Math No E5 Math No E6 CS Yes E7 Hstory No E8 Math Yes The portant concept n rough set theory s ndscernblty relaton. For exaple, n table, (E, E 4 ) s X-ndscernble, (E, E 2 ) s not X-ndscernble. In table, X dvdes { E, E2,,E8} nto three equvalence classes { E, E4, E5,E8}, {E2, E7} and { E3, E6}. That s to say, T can dscern E2, E7 fro E, E4, E5, E8. Slarly, Y can dscern E, E3, E6, E8 fro E2, E4, E5, E7.

5 Now we quantfy the ablty of dscernng obects for a feature or a set of features P, we call the aount of the ablty of dscernng obects as knowledge quantty. When coputng knowledge quantty, we take the followng consderatons nto account: When each obect s dscernble fro the other by feature set P, P has the largest knowledge quantty; When all eleents can only be dvded nto one equvalence class by P, that s to say, P can t dstngush any obect fro the others, P has the sallest knowledge quantty. 3.2 Knowledge quantty Ths secton wll be dscussed on nforaton table (Let decson feature set D = Φ). Defnton. The obect doan set U s dvded nto equvalence classes by the set P (soe features n nforaton table), the probablty of eleents n each equvalence class s: p, p 2,, p, let W P denotes the knowledge quantty of P, W P =W(p,p 2,...,p ), and t satsfes the followng condtons: ) f = then W(p )=W() = 0 2) W(p,...,p,, p,, p )= W(p,,p,..., p,..., p ) 3) W(p,p 2,...,p )= W(p,p p )+ W(p 2,...,p ) 4) W(p,p 2 +p 3 )= W(p,p 2 )+ W(p,p 3 ) Ths can be explaned as: If soe P can t dscern any obect fro the other one n the doan U,.e. only one equvalence class s dvded by P n the doan U, then the ablty of dscern obects for P s 0,.e. knowledge quantty of P s 0. If the doan U s dvded nto equvalence classes by soe feature set P, for the dfferent orders of the sae equvalence classes, the sae feature set P should have the sae ablty of dscernng obects, so W(p,..., p,, p,, p )=W(p,, p,..., p,..., p ). If the doan U s dvded nto equvalence classes E, E 2,, E by soe feature set P, and the probablty of eleents n each equvalence class s p, p 2,, p, that s, E can be dscerned fro E2 E 3 E by P, and E2 E 3 E can be dvded nto - equvalence classes E 2, E 3,, E by P. So W(p,p 2,...,p )= W (p,p p )+ W(p 2,...,p ) If the doan U s dvded nto two equvalence classes E and E 2 by soe feature set P, and the probablty of eleents n E and E 2 s p and p 2 +p 3, then all eleents n E can be dscerned fro p 2 eleents n E 2 and also all eleents n E can be dscerned fro the other p 3 eleents n E 2 by P, that s, W(p,p 2 +p 3 )= W(p,p 2 )+ W(p,p 3 ). Theore. If the doan U s dvded nto equvalence classes by soe feature set P, and the eleent nuber of each equvalence class s n,n 2, n, then the knowledge quantty of P s: W (p,p 2,,p )=c (Here c s constant < p p paraeter,the nuber of eleents n U s n, c= n 2 W (, ) n n )

6 Proof. (o E.G. Fro table we estate W c p p = c c c = c X = < 3 W c p p = 0. c = Y 25 < Specfc condtonal knowledge quantty The easureent of nforaton called entropy s the easureent of the uncertanty, accordng to the entropy, a non-perodcal lterature [2] gets a seres of defnton, such as specfc condtonal entropy, condtonal entropy and nforaton gan. We follow ths way to get specfc condtonal knowledge quantty, condtonal knowledge quantty and knowledge gan. Defnton 2. The obect doan s U, set P and set D are two attrbute(feature) sets, v s a specfc value of P, defnton of specfc condtonal knowledge quantty as W D / P = v : W D / P= v =The knowledge quantty of D aong only those records n whch P has value v. E.G. Fro table we estate: W Y X = Math = c p p = c = 0.25 c / < 2 W Y / X =Hstory = W Y / X =CS = Condtonal knowledge quantty Defnton 3. The obect doan s U, set P and set D are two attrbute(feature) sets, v s a specfc value of P, defnton of condtonal knowledge quantty as W D / P : W D / P = prob( P = v D P v. ) W / = E.G. Fro table we estate: Table 2. v Prob(X= v ) W Y / X = v Math c Hstory CS = W X / Y = prob( X v ) WY / X = = 0.5*0.25c+0.25*0+0.25*0=0.25c v

7 3.5 Knowledge gan Defnton 4. The obect doan s U, set P and set D are two attrbute(feature) sets, defnton of knowledge gan as KG( D P) : KG ( D P) = W D W D / P E.G. Fro table we estate: W X = c W c Y / X = KG( Y X ) = WX WY / X = 0.325c 0.25c = c 4 A knowledge-based feature selecton ethod Knowledge gan easures the aount of knowledge obtaned for category predcton by knowng the presence or absence of a ter n a docuent. Let { c} = denote the set of categores n the target space. The knowledge gan of ter t s defned to be: KG ( = KG ( C T ) = c < p p ( < c c + < c c ) c Gven a tranng corpus, for each unque ter the knowledge gan s coputed and those ters whose knowledge gan s less than soe predeterned threshold are reoved fro the feature space. We call ths ethod as Knowledge-Gan ethod(kg). 5 Experent results Many feature selecton ethods such as DF, IG, MI, and so on have been wdely used n text categorzaton. Exstng experents show IG s one of the ost effectve ethods. Our obectve s to copare the DF, IG and MI ethod wth the KG ethod. A nuber of statstcal classfcaton and achne learnng technques have been appled to text categorzaton, we use two dfferent classfers, k-nearest-neghbor classfer (knn) and Naïve Bayes classfer. We use knn, whch s one of the topperforng classfers[3], evaluatons have shown that t outperfors nearly all the other systes, and we selected Naïve Bayes because t s also one of the ost effcent and effectve nductve learnng algorth for classfcaton [4]. Accordng to [5], cro-averagng precson was wdely used n Cross-Method coparsons, here we adopt t to evaluate the perforance of dfferent feature selecton ethods.

8 5. Data Collectons Two corpora are used n our experents: the NewsGroup collecton[7] and the OHSUMED collecton[] [6]. The 20 Newsgroups data set s a collecton of approxately 20,000 newsgroup docuents, parttoned (nearly) evenly across 20 dfferent newsgroups, each correspondng to a dfferent topc. The 20 newsgroups collecton s a popular data set for experents n text applcatons of achne learnng technques, such as text classfcaton and text clusterng. In ths experent, after we reove the unrelated data set, there are 5769 docuents as a tranng set and the 3837 docuents as the test set n ths study. There are 309 unque ters n the tranng set and 0 categores n ths docuent collecton. OHSUMED s a bblographcal docuent collecton. There are about 800 categores defned n MeSH, and 432 categores present n the OHSUMED docuent collecton. We used a subset of ths docuent collecton docuents as a tranng set and the 3729 docuents as the test set n ths study. There are 465 unque ters n the tranng set and 0 categores n ths docuent collecton. 5.2 Results (-a) (-b) Fg.. Average precson of KNN vs. Nuber of selected features on NewsGroup. (2-a) (2-b) Fg. 2. Average precson of Naïve Bayes vs. Nuber of selected features on NewsGroup.

9 Fgure and Fgure 2 exhbt the perforance curves of knn and Naïve Bayes on NewsGroup after feature selecton DF, IG, MI and KG. We can note that KG and IG ost effectve n our experents, n contrast, MI had relatvely poor perforance. Specally, KG perfors better than the IG ethod, on extreely aggressve reducton, t s notable that KG outperfor IG ((-a),(2-a)). (3-a) (3-b) Fg. 3. Average precson of KNN vs. Nuber of selected features on OHSUMED. (4-a) (4-b) Fg. 4. Average precson of Naïve Bayes vs. Nuber of selected features on OHSUMED Fgure 3 and Fgure 4 exhbt the perforance curves of knn and Naïve Bayes on OHSUMED after feature selecton DF, IG, MI and KG. We also can note that KG and IG ost effectve n our experents, n contrast, MI had relatvely poor perforance. Specally, KG perfors better than IG, on extreely aggressve reducton, t s notable that KG prevalently outperfor IG ((3-a),(4-a)). 6 Concluson Feature selecton plays an portant role n text categorzaton. Many feature selecton ethods such as DF, IG, MI, and so on have been wdely used. Many exstng experents show IG s one of the ost effectve ethods. n ths paper: Accordng to Rough set theory, we gve an nterpretaton for knowledge and

10 knowledge quantty. We put forward a knowledge-based feature selecton ethod called KG. Experental results on NewsGroup and OHSUMED corpora show that KG perfors uch better than MI, DF even than IG. Specally, on extreely aggressve reducton, t s notable that KG outperfors IG. Many text categorzaton ethods are presented and soe works are based on Rough Set. In text categorzaton, each docuent s descrbed by a vector of extreely hgh densonalty, so ost rough set-based ethods use rough set for feature selecton and other technque at the sae te. Ths paper does not use other technque, but only accordng to Rough set theory that knowledge s an ablty to partton obects, we quantfy the ablty of partton obects, and call the aount of ths ablty as knowledge quantty, then put forward a knowledge-based feature selecton ethod KG. References Yng Yang, Jan O. Pedersen A Coparatve Study on Feature Selecton n Text Categorzaton. Proceedngs of ICML-97, pp Yng Lu, A Coparatve Study on Feature Selecton Methods for Drug Dscovery, J. Che. Inf. Coput. Sc. 2004, 44, Stewart M.Yang, Xao-Bn Wu, Zh-Hong Deng, Mng Zhang, Dong-Qng Yang Modfcaton of Feature Selecton Methods Usng Relatve Ter Frequency. Proceedngs of ICMLC-2002, pp J.R. Qunlan. Inducton of decson trees. Machne Learnng, (): pp.8-06, To Mtchell. Machne Learnng. McCraw Hll, Pawlak Z. Rough Sets. Internatonal Journal of Coputer and Inforaton Scence, 982, (5): Koorowsk, J., Pawlak, Z., Polkowsk, L., Skowron, A Rough sets: A tutoral. A New Trend n Decson-Makng, Sprnger-Verlag, Sngapore, Pawlak Z. Grzyala-Busse J, Nelson D E, etal. Rough Sets, Councatons of the ACM, 995, 38(): A rough set-based hybrd feature selecton ethod for topc-specfc text flterng, Proceedngs of the Thrd Internatonal Conference on Machne Learnng and Cybernetcs, August, A Rough Set-Based Hybrd Method to Text Categorzaton. WISE () 200: Chouchoulas and Q. Shen. A Rough Set-Based Approach to Text Classfcaton. Proceedngs of the 7th Internatonal Workshop on Rough Sets (Lecture Notes n Artfcal Intellgence, No. 7), pages 8-27, Andrew Moore. Statstcal Data Mnng Tutorals. 3 Yng Yang, Xn Lu. A re-exanaton of text categorzaton ethods. (SIGIR 99), pp , H. Zhang. The optalty of nave Bayes. The 7th Internatonal FLAIRS conference, Ma Beach, May 7-9, Yng Yang. An evaluaton of statstcal approaches to text categorzaton. Journal of Inforaton Retreval, Vol, No. /2, pp 67 88, OHSUMED 7 NewsGroup