Latent Semantic Indexing Based on Factor Analysis

Transcription

1 Latet Sematc Idexg Based o Factor Aalyss Norak Kawamae Ceter for Advaced Research ad Techology The Uversty of Tokyo 4-6- Komaba Meguroku Tokyo JAPAN tel fax kawamae@mpeg.rcast.u-tokyo.ac.p Abstract The ma purpose of ths paper s to propose a ovel latet sematc dexg (LSI), statstcal approach to smultaeously mappg documets ad terms to a latet sematc space. Ths approach ca dex documets more effectvely tha the vector space model (VSM). Latet sematc dexg (LSI), whch s based o sgular value decomposto (SVD), ad probablstc latet sematc dexg (PLSI) have already bee proposed to overcome problems documet dexg, but crtcal problems rema. I cotrast to LSI ad PLSI, our method uses a more meagful, robust statstcal model based o factor aalyss ad formato theory. As a result, ths model ca solve the remag crtcal problems LSI ad PLSI. Expermetal results wth a test collecto showed that our method s superor to LSI ad PLSI from the vewpots of formato retreval ad classfcato. We also propose a ew term weghtg method based o etropy.. Itroducto Wth the advet of dgtal databases ad commucato etworks, t s easy to obta Web pages ad electroc documets. It s thus ecessary to develop formato retreval methods to smplfy the process of fdg relevat formato. Documet dexg s oe mportat formato retreval method. Tradtoally, documets have bee dexed ad labeled maually by humas. A mportat example s the dea of otoal famles the work of H. P. Luh [9]. The prmary goal s to dex documets wth the same precso acheved by humas. To develop such documet dexg methods, the followg problems must be solved: Ambvalece betwee terms Calculato cost Documet dexg ad keyword matchg methods Due to these problems, the retreval performace of dexg systems s poor. Amog prevous works, latet sematc dexg (LSI), based o sgular value decomposto (SVD), ad probablstc latet sematc dexg (PLSI) have bee developed to overcome these problems, but usolved problems rema. Our prmary goal ths paper s to preset a ovel statstcal approach, to smultaeously mappg documets ad terms to a latet sematc space. Ths approach ca dex documets better tha by usg dvdual dexg terms because the topcs a documet are more closely related to the cocepts descrbed there tha to the dex terms used the documet s descrpto. Our method uses a more meagful, robust statstcal model - called a code model - that assocates documets ad terms wth a latet sematc factor. Ths model s based o factor aalyss ad formato theory ad eables us to remove ths ose ad extract a better latet sematc space tha other methods. As a result, documets ca be dexed early as well as by humas. Ths s maly because factor aalyss s a better statstcal model tha SVD for capturg hdde factors. 2. Related work o documet dexg ad term selecto The vector space model (VSM) [] s a approach to mappg documets to a space assocated wth the terms the documets. The weghtg of the terms a documet provdes the documet s coordates space, ad the smlarty betwee documets s measured space. Latet sematc aalyss (LSA) [2] s a approach to mappg documets to a lower dmesoal space tha LSI. Ths s accomplshed by proectg term vectors to ths space by usg a model based o SVD. To date, several theoretcal results or explaatos have appeared, ad these studes have provded a better uderstadg of LSI. However, may fudametal problems rema uresolved, as follows. Iadequate use of statstcal methods: LSI uses SVD as a statstcal model. Therefore, LSI caot expla or extract the latet sematc factor as a hdde varable, because SVD s othg more tha the decomposto of the observed varable matrx [6]. Optmal decomposto dmeso: I geeral, dmesoalty reducto s ustfed by the statstcal sgfcace of the latet sematc vectors as measured by the lkelhood of the model based o SVD [3],[5]. The complexty of the model s ot cosdered terms of the dmeso. Term weghtg method: To date, may researchers o LSI have used tf/df [] or a smlar method. Ths type of method, however, s ot the best way to evaluate the usefuless of terms, because the weghtg of low-frequecy terms s uderestmated, whle that of hgh-frequecy terms s overestmated. 3. Latet sematc extracto ad term selecto Our method s a ovel statstcal approach to smultaeously mappg documets ad terms to a latet sematc space, ad t mproves documet retreval performace. Our method cossts of three compoets: () a ovel term weghtg

2 method, (2) a code model, ad (3) a statstcal model crtero. The ma dea ths approach s that a latet sematc factor s assocated wth each topc. A partcular topc a documet s more related to the cocepts descrbed the documet tha to the dex terms used the descrpto of the documet. Therefore, ths proposed dexg method eables us to retreve documets based o smlartes betwee cocepts. As a result, our proposed method evolves from keyword matchg to cocept matchg. Ths allows us to retreve documets eve f they are ot dexed by the terms a query, because oe documet shares cocepts wth aother documet dexed by the gve query. Therefore, the latet sematc space mproves documet retreval performace. 3. Term-documet matrx Morphologcal aalyss ca be used to covert a documet to a vector cosstg of the terms occurrg t. The vector space model s a method of geometrcally vsualzg the relatoshps betwee documets. I ths model, the relatoshps betwee terms ad documets are represeted as a term-documet matrx, whch cotas the values of the dex terms t occurrg each documet d, properly weghted by other factors [4][2][8]. We deote m as the umber of dex terms a collecto of documets ad as the total umber of documets. Formally, we let A deote a term-documet matrx wth rows ad m colums ad let w be a elemet (, ) of A. Each w s assged a weght assocated wth the term-documet par (d, t ), where d ( ) represets the -th documet ad t ( m) represets the -th term. For example, usg a tf/df represetato, we have w = tf(t -d )df(t ). Thus, gve the values of the w, the term-documet matrx represets the whole documet collecto. Therefore, each documet ca be expressed as a vector cosstg of the weghts of each term ad mapped a vector space: w w t. A = ( d ) d () wm wm tm There are two methods of term weghtg [8]: local ad global. Amog varous weghtg methods, those desgated as L3, G2, ad G3 are ovel ad have ot bee descrbed prevously [Toku 999]. 3.. Local weghtg Ths approach defes the weghts w each documet. Let P (, m) deote the occurrece probablty of t d. We ascrbe sgfcace to a term s occurrece, o the grouds that t represets a documet s topcs more tha other factors do. Therefore, we base w o the term occurrece probablty P each documet, ad we defe a local weghtg L as follows: L = P *log (+P ). (2) I cotrast to other local weghtg methods based o term frequecy, ths method reduces the effect of very hgh frequecy terms by multplyg P by a logarthm Global weghtg Ths approach defes the weghts w over all documets. Let P (, m) deote the relatve occurrece probablty of t d. We ascrbe sgfcace to the probablty of a term occurrg over all documets o the grouds that a gve term provdes formato for topc predcto ad affects global performace. Etropy s a coveet metrc for comparg the probablty dstrbuto of a term s occurrece. Therefore, we base w o the etropy of the relatve occurrece probablty, ad we defe a global weghtg G as follows: G = + p log p. log Because ths weghtg s based o etropy, f a term occurs wth equal probablty all documets, t s weghted as (the maxmum). To esure that t does ot exceed, ths weghtg s ormalzed through dvso by log, where s the total umber of documets. 3.2 Itroducto of factor aalyss to obta latet sematc space 3.2. Code model The ma theme obtag a latet sematc space s to capture the essetal, meagful sematc assocatos whle reducg the amout of redudat, osy formato. Here, we propose a code model to determe a documet s coordates the latet sematc space. The code model s based o the hypothess that terms documets are geerated from a latet sematc factor ad a gve documet has a probablty of belogg to some category. Ths model ca be used to determe the coordates of ot oly documets but also terms. The relatoshps betwee terms ad latet sematc factors are smlar to the relatoshp betwee ecoded data ad a formato source formato theory. Because of ths smlarty, we call our model a code model. We ext descrbe the key dea ths model. We thk that a latet sematc factor s a formato source ad s coded to a term a documet. Therefore, the relatoshp betwee terms ad latet sematc factors ca be defed as follows. (3) P(t l k ): Probablty of latet sematc factor l k geeratg term t, where m P ( t l k ). A term t = ca be geerated from ot oly = the latet sematc factor l k but also aother factor l k. The relatoshp betwee a documet ad a latet sematc factor ca be defed the followg way: P(l k d ): Probablty of documet d belogg to latet sematc factor l k,

3 where P ( l k d ) =. Documet d ca belog to ot oly latet sematc factor l k but also aother factor l k. The weghts w, whch represet the value of t d, are used to combe these deftos to a ot probablty model, resultg the expresso: w l ( d ) = P( t l ) P( l d ) + P ( t ) P ( d ) = P t k = k where P(t d ) deotes ot the emprcal occurrece but the statstcal probablty dstrbuto of t d, obtaed by multplyg the local ad global weghtgs; P (t ) deotes the uque probablty of t ; ad P (d ) deotes the uque probablty of d. The reaso for troducg these quattes s that P(t d ) caot actually be explaed terms of oly the ot probablty of P(t l k ) ad P(l k d ). The dfferece betwee the statstcal probablty dstrbuto P(t d ) ad the model dstrbuto based o oly the ot probablty of P(t l k ) ad P(l k d ) s cosdered as ose formato theory. Therefore, we call ths model a code model. Notce that multple documets ca belog to some latet sematc factor at the same tme. Ths s because the latet sematc factors are assocated wth the observed varaces as t, d. Moreover, the latet sematc factors do ot costra documets to be orthogoal to each other. Despte the smlarty, the fudametal dfferece betwee the aspect model [5] PLSI ad ths code model s the uque probablty. Ths dfferece affects the latet sematc factors obtaed Factor aalyss To calculate the occurrece probablty of terms the code model, we troduce factor aalyss, whch s a statstcal method that resolves a observed varace to a correspodg latet factor. Udoubtedly, SVD ca also calculate a varable correspodg to the latet sematc factor obtaed arthmetcally by our method. The varable SVD s a mxture of observed varables. I cotrast, the varable factor aalyss s a latet factor. Therefore, t fts the code model metoed earler. I utlzg factor aalyss, we defe the observed varace as P(t d ) ad the latet factor as a cocept. 3.3 Statstcal model selecto based o stochastc complexty (SC) A term-documet matrx s composed of oly observed varaces. The obect of factor aalyss s to estmate the factor loadg, meag P(l d) the code model, whle keepg the umber of uque factors as low as possble. We eed to defe the optmzed umber of factors advace. I ths paper, we determe ths umber by applyg stochastc complexty (SC) [Rssae 996] oe tme. Ths refers to a code legth that shortest the case of codg wth the umber of parameters fxed to m. SC s defed as follows to determe the optmzed umber of latet sematc factors: k ε ε, (4) ( Ak) [ 2( k + ) k( k ) ] ( m) SC log Pa ˆ ( A) + log * 2 (0) where P a (A) deotes the maxmum lkelhood estmator of term-documet matrx A. The mmum of SC(A k) gves the optmzed umber of latet sematc factors, k. Ths formulato allows us to solve the remag problem of determg the optmzed umber of factors both factor aalyss ad (P)LSI. The code model ot oly determes documets coordates a latet sematc space but also s a proper geeratve model for the observed data,.e., the probablty of t d. 4. Documet dexg latet sematc space 4. Expermetal desg Here, we assess the effectveess of OUR METHOD from three vewpots: () term weghtg methods, (2) Our method vs. (P)LSI, ad (3) statstcal model selecto based o SC. Both (P)LSI ad OUR METHOD use the term-documet matrx as a startg pot ad map documets to a latet sematc space. To compare the effectveess of these methods, we evaluated the retreval performace for latet sematc spaces based o each method. Or, terms of obtag a latet sematc space, we compared SVD ad factor aalyss. We evaluated term weghtg terms of the cotrbuto to the latet sematc factors, ad statstcal model selecto terms of whether t ca predct the optmal umber of latet sematc factors. For our expermets, we prepared 20 ews artcles to form a test collecto. These artcles covered seve topc categores: ecoomcs, etertamet, formato techology, poltcs, socety, sports, ad world ews. Each category cluded the same umber of artcles. We used these categores to udge retreval ad classfcato results terms of average precso ad recall rate. Frst, we decomposed the artcles to terms by usg morphologcal aalyss, whch exchages a setece for ts compoet terms. We used Chase s approach [] for ths step. We used oly terms that meet the codto that the part of speech s a ou or a ukow that s ot regstered the dctoary of morphologcal aalyss. Ths s because the terms, as used queres, are lmted to these parts of speech. We also omtted terms that dd ot appear (7) three or less dfferet documets. Geerally, terms wth a sze of (for example, a, etc.) were defed as stop words ad omtted, ad umbers were omtted as well. Sgle Japaese characters, however, do have meag, so we dd ot treat these as stop words. The documets cota 7863 dfferet terms, so a 7863 * 20 term-documet matrx was geerated. 4.2 Comparso of (P)LSI vs Our method ad term weghtg methods Table compares smlarty results amog the documets based o dfferet term weghtgs ad latet sematc spaces. Ths comparso was doe as follows. Frst, we calculated the documets smlarty matrx, as defed secto Secod, f the smlarty of two documets was above a

4 threshold, we udged these documets to be alke. Eve f oe of the documets dd ot clude a term used a query, we cluded both documets the search results. Fally, we evaluated the search results terms of the precso ad recall rates, as show Table. I ths expermet, we defed 0.7 as the threshold. The ormal vector space model cossted of 7863 term axes. The spaces decomposed by SVD or by factor aalyss both cossted of 7 axes. I both cases, the precso ad recall rate were hghest for a decomposed space wth ths umber of axes. Table dcates that L3 ad L4 were the most effectve term weghtg methods for local weghtg, whle G, G3, ad G4 were the most effectve for global weghtg. As for the decomposed space, both SVD ad factor aalyss acheved hgher values tha the ormal vector space model. These results show that the smlarty betwee documets measured a space decomposed by SVD or factor aalyss reflects a fudametal relatoshp betwee topcs the documets. They do ot, however, dstgush Our method from (P)LSI. The term weghtg methods used ths expermet are defed as follows. Local weghtg L: term occurrece probablty (=tf) Weghts w defed by occurrece a documet. L = P = C( t ) m = C( t ) L2: ormalzed term occurrece probablty Weghts w defed by a ormalzato of L based o a log fucto. L2 = log+ C( t ) C( t ) = L3: ormalzed etropy of documet Weghts d defed by a term s occurrece probablty dstrbuto a documet. L3 = log m m m = p log p m: Number of dfferet kds of terms documet d. P: Frequecy probablty of term t documet d. L4: proposed local weghtg method Global weghtg G: ormalzed etropy of term occurrece over all documets Weghts t defed by each term s occurrece over all documets. Ths method s smlar to our proposed global weghtg but dffers terms of P. G = + log p log p P: Occurrece probablty of term t over all documets. G2: ormalzed etropy of term occurrece over all documets Weghts t defed by the occurrece proporto over all documets. G2 ( p ) log( p ) = p log p P: Occurrece probablty of a documet cotag t over all documets. G3: proposed global weghtg method G4: documet frequecy (=df) Weghts d defed by the verse of the documet frequecy. H = + log C ( d ) C (t): Number of documets cotag t. G5: ormalzed etropy of relatve term occurrece Weghts d defed by the relatve probablty dstrbuto over all documets. H = + log p log p P: Relatve probablty of t occurrg d. Table.: Average precso ad recall rates for (P)LSI ad Our method Ths table compares ot oly the term weghtg methods but also thevsm, (P)LSI, ad Our method, for each term weghtg method. I For each same term weghtg method, the upper top row s the VSM, the mddle row s (P)LSI, ad the lower bottom row s Our method. Data lsted as N/P/R. TW ad N/P/R are defed bellow ths table. TW Ecoomcs eetertamet ITformato Poltcs ssports ssocetyal World ews aaverage LG.0/00/3.3.0/00/3.3.0/00/3.3.0/99.7/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /96.6/ /98.9/ /98.7/ /99.7/ /99.8/ /98.3/ /98.7/ /98.7/ /96.3/ /98.9/ /98.6/ /00/ /99.8/ /98./ /98.7/ /98.6/97.8

5 LG2.0/00/3.3.0/00/3.3.0/00/3.3.0/99.9/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /97.0/ /00/ /98.7/ /00/ /00/ /99.4/ /98.5/ /99./ /96.7/ /00/ /98.8/ /00/ /00/ /99.4/ /98.6/ /99./92. LG3.0/00/3.3.0/00/3.3.0/00/3.3.0/99.8/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /96.9/ /99.3/ /98.9/ /99.8/ /99.8/00 30./98.8/ /98.7/ /98.9/ /96.5/ /99.4/ /98.7/ /00/ /99.8/00 30./98.8/ /98.7/ /98.8/98. LG4.0/00/3.3.0/00/3.3.0/00/3.3.0/99.8/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /97.8/ /99.7/ /99.0/ /99.8/00 30./99.7/ /99.5/ /99.2/ /99.2/ /97.5/ /99.5/ /98.8/ /00/ /99.5/ /99.5/ /99./ /99./99.2 L2G.0/00/3.3.0/00/3.3.0/00/3.3.0/99.7/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /96.6/ /99.0/ /98.7/ /99.7/ /99.8/ /98.4/ /98.7/ /98.7/ /96.3/ /99.0/ /98.6/ /00/ /99.8/ /98.2/ /98.7/ /98.6/97.8 L2G2.0/00/3.3.0/00/3.3.0/00/3.3.0/99.9/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /97.0/ /00/ /98.8/ /00/ /00/ /99.4/ /98.6/ /99./ /96.8/ /00/ /98.8/ /00/ /00/ /99.4/ /98.6/ /99./92.2 L2G3.0/00/3.3.0/00/3.3.0/00/3.3.0/99.8/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /97.0/ /99.4/ /98.9/ /99.8/ /99.8/00 30./98.9/ /98.7/ /98.9/ /96.5/ /99.4/ /98.7/ /00/ /99.8/00 30./98.8/ /98.7/ /98.8/98. L2G4.0/00/3.3.0/00/3.3.0/00/3.3.0/99.8/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /97.8/ /99.7/ /99.0/ /99.8/00 30./99.7/ /99.5/ /99.2/ /99.2/ /97.6/ /99.5/ /98.8/ /00/ /99.5/ /99.5/ /99.2/ /99./99.2 L3G.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/3.4 L3G2.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /99.9/ /00/ /99.9/ /00/ /00/ /00/00 29./00/ /00/ /99.9/ /99.2/ /99.9/ /00/ /99.0/ /00/00 7.7/00/ /99.7/93.4 L3G3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/3.4 L3G4.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/3.4 L4G.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/3.4 L4G2.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/ /99.9/ /00/ /99.9/ /00/ /00/ /00/00 29./00/ /00/ /99.9/ /99.2/ /99.9/ /00/ /99.0/ /00/00 7.7/00/ /99.7/93.4 L4G3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/3.4 L4G4.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3.0/00/3.3./00/3.6.0/00/3.4

6 30.0/00/ /99.9/ /00/ /0.0/ /99.9/ /00/ /00/ /00/00 TW:term weghtg, N:the umber of documets retrevedaldocumets, P:precso, R:recall documets, we caot dstgush factor aalyss from SVD terms of precso. Whe we reduce the umber of terms by usg the value of global weghtg, however, we see the superorty of factor aalyss. As for the weghtg methods, the results dcate that t s effectve to select terms based o the value wth L4 as the local weghtg method. We should emphasze that the latet sematc space based o the combato of factor aalyss ad G3*L4 ca be used to classfy documets wth oly the half umber of terms ormally requred. The VSM, o the other had, does ot deped o the umber of terms or the weghtg method. hese results show that our proposed weghtg method s useful ot oly for dexg documets but also for selectg the mmum umber of terms. 4.4 Statstcal model selecto I ths expermet, the optmal umber of latet sematc factors was seve. Ths s the same as the umber of topcs used. Ths dcates that the obtaed latet sematc factors ca effectvely characterze documets statstcally; we ca say that these factors represet sematc meags. SC predcts e as the optmal umber of latet sematc factors, as show Table 2. Other obectve fuctos [3], [5], however, predct umbers far greater tha/less tha e, whch do ot appear Table 2. The value predcted by SC s thus approxmately correct. Table 2. Optmal umber of latet sematc factors ad SC FN 5 6 7* 8 9 Lkelhood SC * FN: Number of latet sematc factors 5. Coclusos We have proposed a ovel statstcal approach to smultaeously mappg documets ad terms to a latet sematc space. Our method cossts of three compoets: () a ovel term weghtg method, (2) a code model, ad (3) a statstcal model crtero. Retreval ad classfcato expermets o a test collecto dcated that Our method s superor to (P)LSI. I other words, the axes a latet sematc space obtaed by our method are closer to the geeral cocepts dexed by humas tha ay other method. Fally, we troduced a statstcal model selecto approach based o stochastc complexty (SC) to solve the remag problem (P)LSI: the problem of how to determe the umber of latet sematc factors. Our expermets showed that the formulato based o SC solves ths problem. We thus coclude that our method s a useful documet dexg method that ot oly solves the crtcal remag problems LSI ad PLSI but also mproves the retreval performace. REFERENCES [] Chase: [2] Deerwester, S., Dumas, S. T., Furas, G. W., Ladaure, T. K., ad Harshma, R.: Idexg by latet sematcs aalyss, Joural of the Amerca Socety for Iformato Scece, 990. [3] Dg, C. H. Q.: A Dual Probablstc Model for Latet Sematc Idexg Iformato Retreval ad Flterg, Proceedgs of the 22d Aual Coferece o Research ad Developmet Iformato Retreval (ACM SIGIR), 999. [4] Dumas, S.T.: Improvg the retreval of formato from exteral sources, Behavor Research Methods, Istrumets ad Computers, 23(2), [5] Hofma, T.: Probablstc latet sematc dexg, Proceedgs of the 22d Aual Coferece o Research ad Developmet Iformato Retreval (ACM SIGIR), 999. [6] Kawamae, N., Aok, T., Yasuda, H.: Iformato Retreval Based o the Iformato Theory Model, Techcal Report of IEICE, DE200-57, 200 ( Japaese). [7] Kawamae, N., Aok, T., Yasuda, H.: Documet Classfcato ad Retreval after Removg Word Nose, Techcal Report of IEICE, NLC200-48, 200 ( Japaese). [8] Kta, K.: Statstcal Laguage Model, The Uversty of Tokyo Press, 999. [9] Luh, H. P.: The automatc dervato of formato retreval ecodemet from mache readable text, Iformato Retreval ad Mache Traslato, 3(2), , 96. [0] Schutze, H. ad Pederse, J.: A vector model for sytagmatc ad paradgmatc relatedess, Proceedgs of the 9th Aual Coferece of the Uversty of Waterloo Ceter for the New OED ad Text Research, 993. [] Salto, G. ad McGll, M. J.: Itroducto to Moder Iformato Retreval, McGraw-Hll, 983. [2] Salto, G. ad Buckley, C.: Term-weghtg approaches automatc text retreval, Iformato Processg ad Maagemet, 24(5) , 988. [3] Saul, L., ad Perera, F.: Aggregate ad mxed order Markov models for statstcal laguage processg, Proceedgs of 2d Iteratoal Coferece o Emprcal Methods Natural Laguage Processg, 997. [4] Tohku, T.: Iformato Retreval ad Laguage Process, The Uversty of Tokyo Press, 999.