Refinement of Document Clustering by Using NMF *

Transcription

1 Refinemen of Documen Clusering by Using NMF * Hiroyuki Shinnou and Minoru Sasaki Deparmen of Compuer and Informaion Sciences, Ibaraki Universiy, Nakanarusawa, Hiachi, Ibaraki JAPAN {shinnou, msasaki}@mx.ibaraki.ac.jp Absrac. In his paper, we use non-negaive marix facorizaion (NMF) o refine he documen clusering resuls. NMF is a dimensional reducion mehod and effecive for documen clusering, because a erm-documen marix is high-dimensional and sparse. The iniial marix of he NMF algorihm is regarded as a clusering resul, herefore we can use NMF as a refinemen mehod. Firs we perform min-max cu (Mcu), which is a powerful specral clusering mehod, and hen refine he resul via NMF. Finally we should obain an accurae clusering resul. However, NMF ofen fails o improve he given clusering resul. To overcome his problem, we use he Mcu objec funcion o sop he ieraion of NMF. Keywords: documen clusering, Non-negaive Marix Facorizaion, specral clusering, iniial marix 1. Inroducion In his paper, we use non-negaive marix facorizaion (NMF) o improve he documen clusering resul generaed by a powerful documen clusering mehod. Using his sraegy, we can obain an accurae documen clusering resul. Documen clusering is a ask ha divides a given documen daa se ino a number of groups according o documen similariy. This is he basic inelligen procedure, and an imporan facor in ex-mining sysems, from Berry (2003). Relevan feedback in informaion rerieval (IR), where rerieved documens are clusered, is a specific applicaion ha is acively researched by Hears e al. (1996), Leuski (2001), Zeng e al. (2001) and Kummamuru (2004). NMF is a dimensional reducion mehod and an effecive documen clusering mehod, because a erm-documen marix is high-dimensional and sparse, from Xu e al. (2003). Le X o be a m n erm-documen marix, consising of m rows (erms) and n columns (documens). If he number of clusers is k, NMF decomposes X o he marices U and V as follows: X = UV where U is m k, V is n k and V is he ransposed marix of V. The marix U and V are nonnegaive. In NMF, each k dimensional column vecor in V corresponds o a documen. An acual clusering procedure is usually performed using hese reduced vecors. However, NMF does no need such a clusering procedure. The reduced vecor expresses is cluser by iself, because each column axis of V represens a opic of he cluser. Furhermore, he marices V and U are * This research was parially suppored by he Minisry of Educaion, Science, Spors and Culure, Granin-Aid for Scienific Research on Prioriy Areas, Japanese Corpus, , Copyrigh 2007 by Hiroyuki Shinnou and Minoru Sasaki. 430

2 obained by a simple ieraion, from Lee (2000), where he iniial marices and V are U 0 0 updaed. Therefore, we can regard NMF as a refinemen mehod for a given clusering resul, because he marix V represens a clusering resul. In his paper, we use NMF o improve clusering resuls. Providing NMF wih an accurae documen clusering resul, we can ensure a more accurae resul, because NMF is effecive for documen clusering. However, NMF ofen fails o improve he iniial clusering resul. The main reason for his is ha he objec funcion of NMF does no properly represen he goodness of clusering. To overcome his problem, we use anoher objec funcion. Afer each ieraion of NMF, he curren clusering resul is evaluaed by ha objec funcion. We firs need he iniial clusering resul. To obain his, we perform min-max cu (Mcu) proposed by Ding e al. (2001), which is a specral clusering mehod. Mcu is a very powerful clusering mehod, and we can obain an accurae clusering resul by improving he clusering resul generaed hrough Mcu, In he experimen, we used 19 daa se provided via he CLUTO websie. Our mehod improved he clusering resul generaed by Mcu. In addiion, he accuracy of he obained clusering resul was higher han hose of NMF, CLUTO and Mcu. 2. Refinemen using NMF 2.1. Feaures of NMF NMF decomposes he m n erm-documen marix X o he m k marix U and he ransposed marix V of he n k marix V, from Xu e al. (2003), where k is he number of clusers: X = UV. NMF aemps o find he axes corresponding o he opic of he clusers, and represens he documen vecor and he erm vecor as a linear combinaion of he found axes. NMF has following hree feaures: i. V and U are non-negaive. The elemen of V and U refers o he degree of relevance o he opic corresponding o he axis of is elemen. I is herefore naural o assign a non-negaive value o he elemen. SVD can also reduce dimensions, bu negaive values appear unlike wih NMF. ii. The marix V represens he clusering resul. The dimensional reducion ranslaes high-dimensional daa o lower-dimensional daa. Therefore, we usually mus perform acual clusering for he reduced daa. However, NMF does no require his, because he marix V represens he clusering resul. The i-h documen d i corresponds o he i-h row vecor of V, ha is, di = ( vi 1, vi2, L, v ). The cluser number is obained from arg maxv. j iii. V and U do no need o be an orhogonal marix. LSI consrucs orhogonal space from documen space. On he oher hand, in NMF, he axis in he reduced space corresponds o a opic, herefore, hese axes do no need o be orhogonal. As a resul, NMF aemps o find he axis corresponding o he cluser ha has documens conaining idenical words NMF algorihm For he given erm-documen marix X, we can obain U and V by he following ieraion, shown by Lee (2000). 431

3 u ( XV ) u (Eq.1) ( UV V ) v ( X U ) v (Eq.2) ( VU U ) Here, u, v and (X ) are he i-h row and he j-h column elemen of U, V and a marix X respecively. Afer each ieraion, U mus be normalized as follows: u u 2 u The ieraion sops by he fixed maximum ieraion number, or he disance J beween X and UV : J = X UV (Eq.3) Here, J is he decomposiion error. i 2.3. Clusering resul and iniial marices In general, he iniial marices and V are consruced using random values. In his paper, U 0 0 we consruc he and V hrough a clusering resul. U 0 0 In paricular, if he cluser number of he i-h daa is clusered ino he c-h cluser, he i-h row vecor of is consruced as follows: V 0 Here, is consruced via XV. U 0 0 v 1.0 ( j = c) = 0.1 ( j c) 2.4. Problem of he objec funcion of NMF We can use NMF as a refinemen mehod for a clusering resul, because he iniial marix of NMF corresponds o a clusering resul. However, NMF ofen fails o improve he given clusering resul. This is because he objec funcion of NMF, ha is, Eq. 3, does no properly represen he goodness of clusering. To confirm his problem, we performed NMF using he documen daa se ``r45'' which is a par of he daa se used in Secion 5. The iniial marix was consruced using he clusering resul obained by Mcu. Figure 1 shows he resuls of his experimen. LINE-1 and LINE-2 in Figure 1 show he change in J in each ieraion and he change in he clusering accuracy, respecively. From Figure 1, we can confirm ha a smaller J does no always mean a more accurae clusering. To overcome his problem, we evaluaed he curren clusering resul using anoher objec funcion afer each ieraion of NMF. Specifically, we used he objec funcion of Mcu. We calculaed he value of he objec funcion afer each ieraion of NMF. If he bes value was no improved for hree consecuive ieraions, we sopped NMF. 432

4 LINE-1 decomposion error LINE-2 clusering accuracy Figure 1: Decomposiion error and clusering accuracy 3. Mcu Nex, we needed he iniial clusering resul. To obain his, we used Mcu proposed by Ding e al. (2001) which is a ype of specral clusering. In his specral clusering mehod, he daa se is represened as a graph. Each daa poin is represened as a verex in he graph. If he similariy beween daa A and B is non-zero, he edge beween A and B is drawn and he similariy is used as he weigh of he edge. From his graph, clusering can be seen o correspond o he segmenaion of he graph ino a number of subgraphs by cuing he edges. The preferable cuing is such ha he sum of he weighs of he edges in he subgraph is large and he sum of weighs of he cu edges is small. To find he ideal cu, he objec funcion is used. The specral clusering mehod finds he desirable cu by using he fac ha an opimum soluion of he objec funcion corresponds o he soluion of an eigenvalue problem. Differen objec funcions are proposed. In his paper, we use he objec funcion of Mcu. Firs, we define he similariy cu(a,b) beween he subgraph A and B as follows: cu(a,b) = W(A,B). The funcion W(A,B) is he sum of he weighs of he edges beween A and B. We define W(A) as W(A,A). The objec funcion of Mcu is he following: 433

5 cu( A, B) cu( A, B) Mcu = + (Eq.4) W ( A) W ( B) The clusering ask is o find A and B o minimize he above equaion. Noe ha he specral clusering mehod divides he daa se ino wo groups. If he number of clusers is larger han wo, he above procedure is ieraed recursively. The minimizaion problem of Eq.4 is equivalen o he problem of finding he n dimensional discree vecor y o minimize he following equaion: y ( D W ) y J m = (Eq.5) y Wy where W is he similariy marix of daa, D = diag(we) and e = ( 1,1, L,1). Each elemen in he vecor y is a or -b, where d B d A a =, b =, d X = ( D) ii and d = d A + d B. If he d d d d A i-h elemen of he vecor y is a (or -b), he i-h daa elemen belongs o he cluser A (or B). We can solve Eq.5 by convering he discree vecor y o he coninuous vecor y. Finally, we can obain an approximae soluion o Eq.5 by solving he following eigenvalue problem: ( I D 1/ 2 WD 1/ 2 ) z = λz (Eq.6) We obain he eigenvecor z, ha is, Fielder vecor, corresponding o he second minimum eigenvalue by solving he eigenvalue problem represened by Eq.6. We can obain he soluion y 1/ 2 o Eq.5 from z = D y. By he sign of he i-h value of y, we can judge wheher he i-h daa elemen belongs o cluser A or B. Noe ha Eq.4 is he objec funcion when he number of clusers is wo. The objec funcion used in NMF is he following general objec funcion for k clusers }. where k B i X { G i i= 1: k cu( G1, G1 ) cu( G2, G2 ) cu( Gk, Gk ) Mcu K = + + L + (Eq.7) W ( G ) W ( G ) W ( G ) 1 G is he complemen of G. The smaller Mcu is, he beer i is. k 2 K k 4. Experimen In he experimen, we used he daa se provided via he CLUTO websie hp://glaros.dc.umn.edu/gkhome/cluo/cluo/download. In oal, 24 daa ses are available. We used daa ses ha had less han 5,000 daa elemens. As a resul, we used 19 daa ses, shown in Table 1. In each daa se, he documen vecor is no normalized. We normalize hem by TF-IDF. Table 1: Documen daa ses Daa # of documens # of erms # of non-zero elemens # of classes cacmcisi 4,663 41,681 83,181 2 cranmed 2,431 41, ,658 2 fbis 2,463 2, , hiech 2, , ,881 6 k1a 2,340 21, , k1b 2,340 21, ,792 6 la1 3,204 31, ,024 6 la2 3,075 31, ,

6 mm 2, , ,062 2 re0 1,504 2,886 77, re1 1,657 3,758 87, reviews 4, , ,635 5 r , ,613 9 r ,804 85,640 8 r ,832 78,609 6 r , ,903 7 r , , r , , wap 1,560 6, , Table 2 shows he resul. NMF-rfn in he able refers o our mehod. Tha is, we obained he iniial clusering resuls by Mcu and hen improved i by performing NMF. The NMF-rfn column in Table 2 shows he raio of values of Eq.7 obained using our mehod o hose obained using Mcu. As shown in Table 2, he value of Eq.7 of our mehod is less han (or equal o) Mcu absoluely. This means ha our mehod absoluely improves he clusering resuls considering Eq.7. Table 2: Comparison of he objec funcion value Daa NMF-rfn cacmcisi cranmed Fbis Hiech k1a k1b la la Mm re re reviews r r r r r r Wap Average

7 Nex, we checked he accuracy of our mehod. Table 3 and Figure 2 show he resuls. The column of NMF, CLUTO 1 and Mcu in Table 3 shows he accuracy of NMF, CLUTO and Mcu respecively. And he column of NMF-ref is he accuracy of our mehod. Table 3: Accuracy of each mehod Daa NMF CLUTO Mcu NMF-rfn cacmcisi cranmed fbis hiech k1a k1b la la mm re re reviews r r r r r r wap Average NMF CLUTO Mcu NMF-rfn Figure 2: Average accuracy of each mehod 1 CLUTO is a very powerful clusering ool. We can ge from he following websie. hp://glaros.dc.umn.edu/gkhome/views/cluo (version 2.1.2a) 436

8 Clusering accuracy is he mos rigorous evaluaion of he clusering resul. However, accuracy is difficul o measure. Firs of all, all daa mus be labeled. Forunaely he daa ses used saisfy his condiion. Nex, we mus map each obained cluser o he cluser label. This mapping is usually difficul. In his paper, we assigned he label o he cluser o assure he accuracy is high, by using dynamic programming. As a resul, we obain accurae clusering. The measure of similariy and he clusering mehod of CLUTO mus also be examined We can selec hese via he opional parameer of CLUTO. In our experimens, we conduced CLUTO wihou any opional parameers, ha is, by using he defaul seing. In his case, CLUTO uses he cosine similariy measure and he k-way clusering mehod, which akes a opdown approach o divide daa ino wo pariions and ieraes his division unil k pariions are obained. In general, he k-way clusering mehod is more powerful han k-means for documen clusering. There were six daa ses for which he accuracy was degraded by performing NMF afer Mcu. Bu in seven daa ses, he accuracy was improved by NMF. In he remaining six daa ses, he accuracy was no changed. Figure 2 shows ha he average accuracies of CLUTO, Mcu and NMF-rfn were 58.21%, 61.82% and 63.22% respecively. Tha is, our mehod showed he bes performance. 5. Discussions 5.1. Search for he opimum soluion The objec funcion value of he end clusering resul is never degraded from he value in Mcu. However, as shown in Table 2, here are some daa ses for which he clusering accuracy of NMF-rfn is worse han ha of Mcu. This is because he objec funcion used does no refer o he goodness of clusering in a precise sense. All objec funcions suffer from he same problem. Especially, he objec funcion J in Eq. 3 is no so good. In fac, we confirmed ha he Mcu objec funcion is beer han J in Eq.3 for NMF from anoher experimen. The clusering ask has wo pars: one is he objec funcion, and he oher is he search mehod for he opimum soluion o he objec funcion. Mcu-rfn uses Eq.7 as he objec funcion and combines he search mehods of Mcu and NMF as is search mehod. Recen heoreical analysis shows he equivalence beween specral clusering and oher clusering mehods. For example, Dhillon e al. (2005) show ha a search for an opimum soluion via specral clusering can be performed using he weighed kernel k-means. Addiionally, Ding e al. (2005) show he equivalence beween specral clusering and NMF. By using hese echniques, a search for an opimum soluion may be consruced in a consisen manner, unlike wih Mcu-rfn. However, such a consisen manner canno avoid falling ino a local opimum soluion. I is herefore helpful o add a mechanism o jump ou from a local opimum soluion. Our hybrid approach is an example of such a mehod. The ``local search'' proposed by Dhillon e al. (2002) is relevan o our approach. This mehod firs obains a soluion by k-means and hen improves i by he ``firs variaion'' and ieraes hese wo seps alernaely. Mcu-rfn firs obains a soluion by Mcu and hen improves i by NMF, bu i does no ierae hem, because he inpu of Mcu does no need o be a clusering soluion. Using he weighed kernel k-means, we can ake he ``ping-pong'' sraegy like he local search. 437

9 5.2. Iniial marices and accuracy of NMF In NMF, clusering accuracy depends on he iniial marices. This is because he local opimum soluion obained by NMF varies according o he iniial value. Therefore, deciding wha iniial marices should be used is a difficul problem, from Wild e al. (2004). Regarding he objec funcion, iniial accuracy mus be improved. Thus, we ook he approach o se he value ha had a high accuracy as he iniial value. However, even if NMF sars from iniial values ha have low accuracy, NMF can sill obain highly accurae resuls. For example, for he daa se ``k1a'' and ``r11'' in our experimens, CLUTO was beer han Mcu. Using he resul of CLUTO as he iniial value, accuracy was no improved by NMF. On he oher hand, in he case of Mcu, accuracy was improved by NMF, and he final accuracy was beer han ha of CLUTO. Finally, clusering is an NP-hard combinaorial opimizaion problem afer he objec funcion is fixed. I is impossible o find he opimal iniial value. Thus, he clusering algorihm mus ake an approach ha improves he soluion gradually. Under such a siuaion, our approach o se a feasible soluion o he iniial value is pracical Fuure works for documen clusering The clusering ask is a purely engineered problem once daa is ranslaed ino vecors. To ge more accurae clusering, we should acively use knowledge on daa a he pre-ranslaed sage. In he case of documen clusering, we should remember ha he daa is a documen. I may be imporan o ensure ha mea-informaion such as he publicaion place, auhor, aim of clusering is incorporaed ino he clusering process or vecor-ranslaion process. Clusering is unsupervised learning. The effecive way o raise accuracy is herefore o assign supervised labels o daa. Recenly, semi-supervised clusering using user-ineracion has been acively researched by Basu e al. (2002), Bilenko e al. (2004) and Xing e al. (2003). This semi-supervised clusering using mea-informaion shows promise. 6. Conclusion In his paper, we have shown ha NMF can be used o improve clusering resul. For pracical use, we used anoher objec funcion, and we evaluaed he curren clusering resul using ha objec funcion afer each ieraion of NMF. By performing Mcu o obain he iniial clusering resul, we can obain an accurae clusering resul. In he experimen, we used 19 daa se provided via he CLUTO websie. Our mehod improved he clusering resul obained by Mcu. In addiion, he accuracy of he obained clusering resul was higher han hose of NMF, CLUTO and Mcu. In fuure, we will research semi-supervised clusering using meainformaion. References Basu, S., A. Banerjee, and R. J. Mooney Semi-supervised Clusering by Seeding. Proceedings of ICML-2002, pp Berry, M. W Survey of Tex Mining: Clusering, Classificaion, and Rerieval. Springer. Bilenko, M., S. Basu and R. J. Mooney Inegraing Consrains and Meric Learning in Semi-Supervised Clusering. Proceedings of ICML-2004, pp Dhillon, I. S., Y. Guan and J. Kogan Ieraive Clusering of High Dimenional Tex Daa Augmened by Local Search. The 2002 IEEE Inernaional Conference on Daa Mining, Dhillon, I. S., Y. Guan and B. Kulis A Unified View of Kernel k-means, Specral 438

10 Clusering and Graph Cus. The Universiy of Texas a Ausin, Deparmen of Compuer Sciences. Technical Repor TR Ding, C., X. He and H. D. Simon On he Equivalence of Nonnegaive Marix Facorizaion and Specral Clusering. Proceedings of SDM Ding, C., X. He, H. Zha, M. Gu and H. Simon Specral Min-max Cu for Graph Pariioning and Daa Clusering. Lawrence Berkeley Naional Lab. Tech. repor Hears, M. A. and J. O. Pedersen Reexamining he Cluser Hypohesis: Scaer/gaher on Rerieval Resuls. Proceedings of SIGIR-96, pp Kummamuru, K., R. Lolikar, S. Roy, K. Singal and R. Krishnapuram A Hierarchical Monoheic Documen Clusering Algorihm for Summarizaion and Browsing Search Resuls. Proceedings of WWW-04, pp Lee, D. D. and H. S. Seung Algorihms for Non-negaive Marix Facorizaion. Proceedings of NIPS-2000, pp Leuski, A Evaluaing Documen Clusering for Ineracive Informaion Rerieval. Proceedings of CIKM-01, pp Wild, S., J. Curry and A. Doughery Improving Non-negaive Marix Facorizaions hrough Srucured Iniializaion. Paern Recogniion, Vol.37, No.11, Xing, E. P., A. Y. Ng, M. I. Jordan and S. Russell Disance Meric Learning, wih Applicaion o Clusering wih Side-informaion. Advances in Neural Informaion Processing Sysems 15, Xu, Wei., X. Liu and Y. Gong Documen Clusering Based on Non-negaive Marix Facorizaion. Proceedings of SIGIR-03, pp Zeng, H.-J., Q.-C. He, Z. Chen, W.-Y. Ma and J. Ma Learning o Cluser Web Search Resuls. Proceedings of SIGIR-04, pp