Non-negative Matrix Factorization for Filtering Chinese Document *

Transcription

1 No-egative Matrix Factorizatio for Filterig Chiese Documet * Jiaiag Lu,,3, Baowe Xu,, Jixiag Jiag, ad Dazhou Kag Departmet of Computer Sciece ad Egieerig, Southeast Uiversity, Naig, 0096, Chia Jiagsu Istitute of Software Quality, Naig, 0096, Chia 3 PLA Uiversity of Sciece ad Techology, Naig, 0007, Chia lu@seu.edu.c Abstract. There are two asty classical problems of syoymy ad polysemy i the filterig systems of Chiese documets. To deal with these two problems, we would ideally lie to represet documets ot by words, but by the sematic relatios betwee words. No-egative matrix factorizatio (NMF is applied to dimesioality reductio of the words space. NMF is distiguished from the latet sematic idexig (LSI by its o-egativity costraits. These costraits lead to a parts-based represetatio because they allow oly additive, ot subtractive, combiatios. Also, NMF computatio is based o the simple iterative algorithm; it is therefore advatageous for applicatios ivolvig large sparse matrices. The experimetal results show that, comparig with LSI, NMF method ot oly improves filterig precisio maredly, but also has the merits of fast computig speed ad less memory occupacy. Itroductio Automatic filterig of iformatio from documet sources has become icreasigly importat i recet years. Iformatio filterig systems are desiged to shift through large quatities of dyamically geerated documets ad display oly those which may be relevat to a user s iterests []. Two maor types of filterig systems have bee proposed: cotet-based filterig [] ad collaborative filterig. Collaborative filterig selects documets based o user s evaluatios of the documets. O the other had, cotet-based filterig selects documets based o the cotets of documets ad each user s preferece. There exist several types of cotet-based filterig systems. I the vector space model [3], user profiles ad documet profiles are represeted as weighted vectors of the words i the system. The relevace of each documet to each user is calculated accordig to the similarity betwee the user profile vector ad the documet profile vector. There are two asty classical problems of syoymy ad polysemy i the vector space model. To deal with these two ad other similar problems, we would * This wor was supported i part by the Youg Scietist's Fud of NSFC ( , , Natioal Grad Fudametal Research 973 Program of Chia (00CB3000, Natioal Research Foudatio for the Doctoral Program of Higher Educatio of Chia. M. Buba et al. (Eds.: ICCS 004, LNCS 3037, pp. 3 0, 004. Spriger-Verlag Berli Heidelberg 004

2 4 J. Lu et al. ideally lie to represet documets ot by words, but by the sematic relatios betwee words. Latet sematic idexig (LSI aalysis based o sigular-value decompositio (SVD [4, 5] is a iformatio retrieval method that attempts to capture the sematic relatios by usig techiques from liear algebra. LSI costructs a low-dimesioal sematic space wherei words ad documets that are closely associated are placed ear oe aother. SVD allows the arragemet of the space to reflect the maor associative patters i the data, ad igore the smaller, less importat iflueces. However, the cost of SVD computatio will be prohibitive whe matrices become large. I additio, SVD is lac of ituitive otio. I this paper, a method based o o-egative matrix factorizatio (NMF [6, 7] for costructig Chiese user profile is preseted. This method proposes to apply NMF to dimesioality reductio of the documet vectors. NMF ca decompose a o-egative matrix ito two o-egative matrices. Oe of the decomposed matrices ca be regarded as the basis vectors. The dimesioality reductio ca be performed by proectig the documet vectors oto the lower dimesioal space which is formed by these basis vectors. NMF is distiguished from LSI by its o-egativity costraits. These costraits lead to a parts-based represetatio because they allow oly additive, ot subtractive, combiatios. Also, NMF computatio is based o the simple iterative algorithm; it is therefore advatageous for applicatios ivolvig large sparse matrices. The remaider of this paper is orgaized as follows. I sectio, we briefly review how to represet a set of ustructured Chiese documets as a vector space model. I sectio 3, we itroduce o-egative matrix factorizatio. I sectio 4, a NMF method for costructig Chiese user profile is preseted. I sectio 5, the experimetal results of NMF method are compared with LSI. Fially, sectio 6 gives the coclusios. Vector Space Models for Documets Let D = { d, d, d } be a set of Chiese topic documets, let D {,,, } = d d d be a set of o-topic documets, + =, D = D D. We briefly review how to represet a set of ustructured Chiese documets as a vector space model. The preprocessig is as followig. ( Chiese documets are writte as characters strigs with o spaces betwee words, so we first use word segmetatio algorithm [8] to segmet the Chiese documets. ( After word segmetatio, we elimiate o-cotet-bearig stopwords. (3 Usig heuristic or iformatio-theoretic criteria, elimiate o-cotet-bearig high-frequecy ad low-frequecy words. Such words ad the stopwords are both ow as fuctio words. Elimiatig fuctio words removes little or o iformatio, while speedig up the computatio. (4 After above elimiatio, suppose m uique words remai, let be T = t, t, t }. We use ormalized word frequecy-iverse documet frequecy { m

3 No-egative Matrix Factorizatio for Filterig Chiese Documet 5 scheme [9] to obtai word-documet matrix X = ( x i m. The i th elemet x i of the T documet vector x = x, x,, x is give by ( m x = # ( x i, t log i h i where #(x, t i deotes the umber that the word t i appears i the documet x, h i deotes the umber of the documets i which the word t i appears, is the total documet umber. Documet vectors are usually ormalized to a uit vector, that is, x x i i =, m x i = i, i =,, m Ituitively, the effect of ormalizatio is to retai oly the directio of the documet vectors. This esures that documets dealig with the same subect matter (that is, usig similar words, but differig i legth lead to similar documet vectors. After the preprocessig, Chiese documets are represeted as m dimesioal documet vectors x,,, X = x x, x be topic documet vectors, { v v, v } =. Let { }, X =, be o-topic documet vectors, X = X X. These documet vectors mae up of word-documet matrix = (. X x i m 3 No-egative Matrix Factorizatio Give a o-egative matrix U = u i m r X = x i m (, NMF fids the o-egative m r matrix ( ad the o-egative r matrix V = ( v i r such that The r is geerally chose to satisfy ( m r < m regarded as a compressed form of the data i X. The equatio ( ca be rewritte colum by colums as X UV ( +, so that the product UV ca be x Uv ( where x ad v are the correspodig colums of X ad V. Each vector x is approximated by a liear combiatio of the colums of U, weighted by the compoets of v. Therefore, U ca be regarded as cotaiig a basis vector that is optimized for the liear approximatio of the vector i X. Sice relatively few basis vectors are used to represet may vectors, good approximatio ca oly be achieved if the basis vectors discover structure that is latet i the vectors. Here, we itroduce a algorithm based o iterative estimatio of U ad V. At each iteratio of the algorithm, the ew value of U ad V is foud by multiplyig the curret value by some factor that depeds o the quality of the approximatio i equatio (. Repeated iteratio of the update rules is guarateed to coverge to a locally optimal matrix factorizatio. The update rules give i the ext equatios [7].

4 6 J. Lu et al. v i v i u i x y i (3 u i u u i i x y i i ui u where U ad V are iitial stochastic matrices. The update rules maximize the followig obective fuctio: x i F ( X, Y = x + + i log xi yi α a i β b i, yi i, i v ii (4 (5 (6 where T VV,α, > 0 a i is the compoets of β are some costats, = UV = ( y i m. U T U, b ii is the diagoal compoets of Y 4 Costructig Chiese User Profile After word-documet matrix X is decomposed by the NMF i sectio 3, the m dimesioal documet vectors are proected ito the r dimesioal vectors. Let V = { v, v,, v } be proectig of the topic documet vectors, V = { v, v,, v } be proectig of the o-topic documet vectors. We ca compute the mea vector of topic documet vectors. x = O = ( o, o, om = I the same way, we ca compute the mea vector of proectig of the topic documet vectors. v = O = ( o, o, o r = Accordig to equatio (, we ca easily obtai. O UO = u o + u o + + u o (7 r r A simply way is to select UO as a user profile, but this user profile may be ieffective. Next, we defie the class discrimiative degree of the basis vectors i order to obtai a effective user profile.

5 No-egative Matrix Factorizatio for Filterig Chiese Documet 7 T T Defiitio. Let v = ( v, v, vr, =,,, v = ( v, v, vr, =,, Class discrimiative degree of the basis vector u s to topic documets is defied as follows: d s = vs vs = =, s =,, r (8 If the average weight ad the average weight v s = v s = of the basis vector u s i the topic documets is big, of the basis vector u s i the o-topic documets is small, the the class discrimiative degree d s is large. That is to say, the basis vector u s has strog discrimiative ability betwee topic documets ad the o-topic documets. We select basis vectors with big class discrimiative degrees, simply let be u, u,, u, r. Accordig to equatio (7, we use these basis vectors to costruct a m dimesioal vector of the words as follows. u o + u o + + u o (9 The, we sort the compoets of the m dimesioal vector by the value, ad select l compoets with big values as the user profile. User _ Profile = t, g >, < t, g >, < t l, g } { < l > where t i is a word, g i is the compoet value with respect to word t i, i =,, l. 5 Experimetal Results All documets i the experimet are dowloaded from The topic documets iclude: Chess (77, Gym (49, Badmito (0, Box (69, Pig- Pog (77, Volleyball (7, Racig (95, Swimmig (6, Teis ball (08, Baseball (60, Satig (55, Golf (, Trac ad field (47, Billiards (58 ad Martial art (43. I additio, there have 044 o-topic documets. I order to test 5 topic documets, we partitio each topic documets ito four groups, ad select oe group as testig documets, other groups as traiig documets. Whe recall was set to 0%, 0%, 0%, 30%,, 90%, 00%, the average precisio at poits of filterig systems based o NMF ad LSI [5] are compared. We select 843 words i the experimet, let r = 00, figure shows the experimetal results o the four topic documets. Figure (a ad figure (b show the average precisios of filterig systems based o NMF ad LSI with the selectig words respectively. I the filterig systems based o NMF, we select basis vectors that have strog discrimiative ability betwee topic documets ad the o-topic documets. Figure shows the average precisios of 5 topic documets. The experimetal results show that the average precisio of filterig systems based o NMF is better tha LSI. I additio, the memory occupacy of NMF is lesser tha LSI. For example, the memory occupacy of the left sigular matrix i LSI is

6 8 J. Lu et al. 6.3M, ad the memory occupacy of the right sigular matrix is 33.0M. Whereas the memory occupacy of the left o-egative matrix i NMF is 6.9M, ad the memory occupacy of the right o-egative matrix is 6.7M. Furthermore, NMF oly eeds iterative times, so NMF costs less computatio time tha LSI. I the experimet, usig the SVDPACK from NMF oly costs about half time of LSI. (a LSI (r=00 (b NMF (r=00 Fig.. Comparig the precisio

7 No-egative Matrix Factorizatio for Filterig Chiese Documet 9 Fig.. Comparig average precisio 6 Coclusio Automatic filterig of iformatio from documet sources has become icreasigly importat as the volume of electroically accessible documets has exploded i recet years. I this paper, a method based o NMF for costructig Chiese user profile is preseted. This method proposes to apply NMF to dimesioality reductio of the documet vectors i the word-documet matrices. NMF decomposes a o-egative matrix ito two o-egative matrices. Oe of the decomposed matrices ca be regarded as the basis vectors. The dimesioality reductio ca be performed by proectig the documet vectors oto the lower dimesioal space which is formed by these basis vectors. NMF is distiguished from LSI by its o-egativity costraits. These costraits lead to a parts-based represetatio because they allow oly additive, ot subtractive, combiatios. Also, NMF computatio is based o the simple iterative algorithm, it is therefore advatageous for applicatios ivolvig large sparse matrices. The experimetal results show that, comparig with LSI, NMF method ot oly improves filterig precisio maredly, but also has the merits of fast computig speed ad less memory occupacy. I the future wor, we will discuss how to use NMF i the Chiese documet clusterig ad classificatio.

8 0 J. Lu et al. Refereces. Beli, N.J., Croft, W. B.: Iformatio Filterig ad Iformatio Retrieval: two sides of the same coi. Commuicatio of ACM, 35(. ( Che, L., Katia, S.: WebMate: A Persoal Aget for Browsig ad Searchig. ACM AGENTS 98, Proceedigs of the Iteratioal Coferece o Autoomous Agets, Mieapolis ( Yart, T. W., Garcia-Molia, H.: Idex Structures for Iformatio Filterig uder the Vector Space Model. Proceedigs of the 0th Iteratioal Coferece o Data Egieerig, Alamitos, CA, IEEE ( Papadimitriou, C. H., Raghava, P., Tamai, H.: Latet Sematic Idexig: A Probabilistic Aalysis. Proceedigs of PODS 98, Seattle, WA ( Lu, Z., Lu, H., Li, Y.: FDS Expressive Method i Iformatio Filterig. Joural of Tsighua Uiversity (sciece ad techology, 39(9. ( Lee, D. D., Seug, H. S.: Learig the Parts of Obects by No-egative Matrix Factorizatio. Nature, 40. ( Li, S. Z., Hou, X. W., Zhag, H. J.: Learig Spatially Localized Parts-based Represetatio. Proceedigs of IEEE Iteratioal Coferece o Computer Visio ad Patter Recogitio, Hawaii ( Che, G. L., Wag, Y. C., HAN, K. S., Wag, G.: A Improved Fast Algorithm for Chiese Word Segmetatio. Joural of Computer Research ad Developmet, 37(4. ( Kolda, T. G.: Limited-Memory Matrix Methods with Applicatios. Ph.D. thesis, The Applied Mathematics Program, Uiversity of Marylad, College Par, Maylad (977