Instability of Relevance-Ranked Results Using Latent Semantic Indexing for Web Search

Transcription

1 Instability of Releance-Ranked Reslts Using Latent Semantic Indexing for Web Search Hossain Kettani ECECS Department Polytechnic Uniersity of Perto Rico San Jan, PR 0099 Gregory B. Newby Arctic Region Spercompting Center Uniersity of Alaska Fairbanks, AK Abstract he latent semantic indexing (LSI) methodology for information retrieal applies the singlar ale decomposition to identify an eigensystem for a large matrix, in which cells represent the occrrence of terms (words) within docments. his methodology is sed to rank text docments, sch as Web pages or abstracts, based on their releance to a topic. he LSI was introdced to address the isses of synonymy (different words with the same meaning) and polysemy (the same words with mltiple meanings), ths addressing the ambigity in hman langage by tilizing the statistical context of words. Rather than keeping all k possible eigenectors and eigenales from the singlar ale decomposition which approximates the original term by docment matrix, a smaller nmber is sed essentially allowing a fzzy match of a topic to the original term by docment matrix. In this paper, we show that the choice k impacts the resltant ranking and there is no ale of k that reslts in stability of ranked reslts for similarity of the topic to docments. his is a srprising reslt, becase prior literatre indicates that eigensystems based on sccessiely large ales of k shold approximate the complete (max k) eigensystems. he finding that docment-qery similarity rankings with larger ales of k do not, in fact, maintain consistency, makes it difficlt to assert that any particlar ale of k is optimal. his in trn renders LSI potentially ntrstworthy for se in ranking text docments, een for ales that differ by only 0 of the max k. his work was spported in part by the Arctic Region Spercompting Center at the Uniersity of Alaska Fairbanks, with additional spport from the US Department of Defense High Performance Compting Modernization Program Office nder contract G and others. his work was also spported in part by the United States Nclear Reglatory Commission nder grant nmber NRC Introdction Searching large collections of docments is challenging for comptational, practical and hmanistic reasons. he great sccess of today s online search engines has allowed s to oerlook many of these challenges, since reslts are, generally, satisfying. Yet, only a small fraction of aailable electronic data is aailable throgh sch search engines [8, 2]. From a practical standpoint, we wold like to make increased nmbers of items of all types (Web pages and other docments) aailable for search. In some cases, this will reqire interacting directly with a search serice on a specific site, rather than a centralized aggregation of content (as seen in Google & others). Sch distribted search might offer the only method for oercoming secrity and accessibility concerns for access to specific data sets [5]. If practical considerations for distribted search were oercome, there wold still be comptational and hmanistic concerns remaining. Hmanistically, consider that the role of search systems more generally called information retrieal or IR systems is to proide a ranked list of the most releant docments. his list is created based on matching the hman expression of information need the qery to a set of docments. he set of docments can be ery large, as we see with today s Web search engines. he challenge is that releance to a hman s information need is a difficlt concept [0]. he list of releant docments for a qery is sitational, and for a particlar hman information seeker the releance of a gien docment will depend on what he or she is really looking for, as well as prior knowledge and expertise in the sbject area. Expecting an IR system to make an effectie ranked list of docments based on jst a few qery words is ambitios, indeed! Sccessfl search engines today often present a ariety of different reslts in the first set of 0 or 20 docments, so that different information seekers can qickly find high /0 $ IEEE

2 qality docments of interest in the list. In this paper, we discss latent semantic indexing (LSI), which has been proposed as one approach to better match the conceptal meaning of a qery against the concepts fond in a docment, rather than being solely reliant on the (few) terms of a qery, and the often ambigos langage sed in qeries and docments [2]. Comptational challenges of search hae, to some extent, been addressed throgh heroic engineering ndertaken by database and Web search engine proiders, among others. Being able to respond to a qery in only a second or so is testament to the ability to rapidly prodce reslts by parallelizing search across a single large corps. In this paper, we consider the comptational challenges broght abot by distribted search and the merger of reslts from mltiple collections, which has not yet been the sbject of intense engineering efforts. We anticipate that the practical need to search across data sets, along with the hmanistic need to get search reslts which are more closely targeted for indiidal information needs, will proide impets to address sch problems. In this paper, we describe the LSI approach to information retrieal. We compare its sccessie application by incorporating different nmbers of eigenales k that differ by 0 of the maximm k. We deelop a statistical approach to compare the resltant ranking from each k ale. We obsere that the rankings differ significantly. he finding that docment-qery similarity rankings with larger ales of k do not, in fact, maintain consistency, makes it difficlt to assert that any particlar ale of k is optimal. his in trn renders LSI potentially ntrstworthy for se in ranking text docments, een for ales that differ by only 0 of the max k. he rest of the paper is organized as follows. he ector space model and latent semantic indexing algorithms are described in Sections 2 and 3. A discssion of arios performance challenges for LSI is laid ot in Section44. Accordingly, Section 5 presents an empirical ealation of the robstness of LSI with respect to the choice of k. Finally, conclding remarks are presented in Section Vector Space Model When addressing the isse of docment ranking, it is conenient to redce it to a mathematical problem. hs, [9] introdced the ector space model (VSM), in which keywords (terms) are the axis and docments are points in a mltidimensional space. For example, if we hae two docments d and d 2, the keywords are the set {cat, dog}, the word cat occrs fie and seen times in d and d 2, respectiely, and the word dog occrs two and three times in d and d 2, respectiely, then d and d 2 are represented by the points (5, 2) and (7, 3), respectiely, and the qery ector cold be (0, ), (, 0), or (, ). A pictorial illstration of this ector space model is gien in Figre. he ranking of the similarity of all docments to the qery is achieed by calclating the distance between the docments and the qery, then ranking all docments based on their distance measre. he closer a docment to a qery is, the higher its rank. Alternatiely, since calclating distances is time consming, we can se the angle θ i between each docment and the qery as a measre for ranking. More precisely, we can se the cosine of this angle for ranking. hs, we can define, q d i α i = cos θ i =. q d Frthermore, if the ectors are normalized to nit length, this measre can be redced to α i = q d i. hs, if all ectors are normalized, and since all the points lie in the nonnegatie hypercbe, minimizing the distance between d i and q is eqialent to maximizing α i, which is a simple dot prodct between the qery and the corresponding docment. Additional comptational saings are gained by the fact that any docment that does not contain any qery terms will hae a cosine of zero, ths immediately eliminating sch docment from frther consideration. he VSM was introdced as an alternatie to an een simpler approach called Boolean matching. With Boolean matching, any docment with all qery terms is considered eqialently releant to any other docment with all qery terms. While this approach was sitable in the past for small collections, and for long qeries, it is clearly inappropriate for today s collections of millions or billions of docments. he tendency of many qeries to hae ery few words makes the problem worse. With the VSM, docment length and qery term freqency are taken into accont, sch that the relatie importance of qery terms within each docment is taken into accont. Other techniqes, sch as pioted docment length normalization, enhanced the VSM frther. i 2

3 he ector space model and follow-on techniqes is an excellent approach to information retrieal, and scales well to larger collections. In fact, VSM is at the heart of most IR systems today. Howeer, the VSM does not adeqately address the isses of synonymy and polysemy of words in a langage, sch as car s. atomobile, or the se of can in I can erss can in tna can. Neertheless, its application is relatiely fast. Accordingly, obtaining each α i reqires O(n) mltiplications (where n is the nmber of docments from a corps that contain at least one qery term). herefore, obtaining the score ector α== [α, α 2,, α n ] reqires O(n 2 ), which is the time complexity of VSM. his can frther be redced to O(n) throgh parallelization. 3. Latent Semantic Indexing he latent semantic indexing (LSI) algorithm [2] was introdced as an enhancement to the ector space model to address the isses of synonymy and polysemy throgh statistical treatment of the relations among terms and docments. LSI applies the singlar ale decomposition (SVD) to redce linear redndancies in data. Accordingly, [2] points ot that a corps can be presented as a matrix C with non-negatie integer entries representing the nmber of occrrence of each term in each docment. he colmns of C represent docments and its rows represent the terms as follows: m docments c c2 cm c C = n terms cn cn2 c nm introdces the following approximation of the term-bydocment matrix C: Cˆ = US ˆ ˆVˆ n 2 n2 = k nk s s s k m 2 n2 k mk After normalizing the colmns of Ĉ, the ector space model approach of taking the cosine between qery ectors and docment ectors can be applied. An itemized description of the LSI algorithm is as follows. We start by sing VSM to represent the qery and corps by letting:. q represent an n qery ector with nit norm, and 2. C represent an n m term by docment matrix. We then apply LSI as follows: 3. C = USV is the SVD of C, 4. Obtain by keeping the first k diagonal elements of S, 5. Adjst U and V accordingly to get and, by keeping the first k colmns of each, 6. Approximate C by, 7. Normalize the colmns of,,,, 8. Define α i = q ĉ i, For a reasonably sized test collection, sch as the Web2 collection from the annal ext REtrieal Conference ( m is on the order of millions, while n is hndreds of thosands. Latent semantic indexing sggests first obtaining the corresponding singlar ale decomposition as follows: t 2 q C = USV = 0 2 m s m 2 s 2 0 n n2 nm sm m n2 mm hen, a rank redction is obtained by zeroing ot the singlar ales s i sch that s i << s i k, which θ i 0 t Figre. Pictorial illstration of the ector space model of docment ranking, where {t, t 2 } is the set of keywords, q is the qery, other points are the docments, θ i is the angle between a docment and the qery. c i 3

4 9. Let α = [α, α 2,, α n ] be the score ector, 0. he higher the score is, the higher the rank of the docment. Conseqently, the time complexity of LSI can be obtained as follows. he singlar ale decomposition of the n m matrix C in Step 3, reqires O(nm 2 +++n 2 m + n 3 ) mltiplications [3]. he mltiplication of an n m matrix A and an mn p matrix B reqires O(nmp) [3]. hs, obtaining the matrix in Step 6, reqires O(max{nk 2, nkm}) = O(nkm), since k < m. On the other hand, obtaining each α i in Step 8, reqires O(n), and therefore, obtaining the score ector α in Step 9, reqires O(n 2 ), which is the time complexity of VSM. Since LSI consists mainly of the aforementioned three parts, then it can readily be seen that SVD controls the complexity of LSI. Accordingly, the time complexity of LSI algorithm is in O(nm 2 + n 2 m + n 3 ). 4. Performance Isses For a ery large corps, LSI is intractable with crrent microprocessors and memory constraints. For example, the REC Million Qery rack collection [], reslts in a matrix C with n of abot 600,000 terms by m of abot 25 million docments. Storing this matrix is doable, since it is ery sparse with abot of its elements bring zero. (his is becase most terms do not occr in most docments.) Howeer, REC participants hae not yet prodced an SVD for this matrix, een on today s large spercompters. his motiated researchers sch as [4], to parallelize the problem to better tilize the rapidly adancing high performance compting infrastrctres. Singlar ale decomposition (SVD) prodces a diagonal matrix containing singlar ales of the matrix C in descending order. As mentioned earlier, in its attempt to approximate C, LSI zeroes ot the singlar ales s i sch that s i << s i k. Bt when is the condition s i << s satisfied? In his illstratie example, [2] picked the largest two singlar ales and zeroed ot the other seen. So the athors sppressed all singlar ales less than 7 of the largest singlar ale. Others, sch as [3], were more catios and arged that the condition s i << s corresponds to s i less than 0 of s. o empirically ealate the performance of a docment ranking algorithm, it is tempting to se random matrices to simlate the term-by-docment matrix C. his randomness, howeer, makes singlar ales decrease ery slowly een if we ensre that the matrix is ery sparse. For example, we generated sparse random matrix in MALAB with the following: C = ceil(r rand(m, n)), where 0 r. his reslts in abot 00r of the elements of C being nonzero. he slow decay of singlar ales will entail keeping most of them. For instance, if the 0 criterion is followed, we will need to keep 75 to 99 of the singlar ales. his in trn, defeats the rationale behind the se of LSI. Nonetheless, the strctre of hman langages makes the content and formation of a text docment far from being random. o the contrary, LSI is designed to make se of the non-randomness of hman langage. For or prposes, the REC Million Qery rack [] proides an excellent, yet challenging, datasets for ealation prpose. his dataset consists of millions of English Web docments and qeries gathered from the United States.go domains. We constrcted C from sb-collections of this dataset based on separate specific domains (sch as or each with thosands of niqe terms and docments. More than 99 of the elements of C in these sb-collections were zero. We obsered an exponential decrease in the magnitde of the corresponding singlar ales. For example, following the 0 criterion, we keep only 0.5 to 0.69 of the singlar ales. Finally, it is important to note that in a docment ranking algorithm, the order of docments in a corps shold not affect their ranking. Indeed, if SVM is applied alone, then the partitioning does not affect the ranking. Howeer, de to LSI, we generally hae C C C. hs, in the next section; we inestigate the stability and robstness of LSI sing the REC dataset. 5. Empirical analysis 5.. Real data o ealate the performance of LSI, we applied it to sb-collections of the corps obtained from the REC Million Qery rack []. REC is the ext REtrieal Conference, held annally at the National Institte for Standards and echnology (NIS) in Gaithersbrg, Maryland. REC is intended to proide a form for comparison of information retrieal techniqes across a ariety of test conditions. Different tasks are ealated as part of tracks, which typically last for at least a few years before being pdated, merged with other tracks, or discontined. In the past, REC focsed on relatiely strctred textal data, sch as abstracts, news articles and proceedings sch as the Federal Register. 4

5 he growth of the Web, and interest in Web-based search engines, has reslted in nmeros tracks that examined HML, often harested from the lie Web. hese harests reslted in fixed test collections, with the largest in 2008 holding oer 20 million niqe Web pages from oer 6,000 niqe hosts. By resing a test collection for mltiple years of REC experiments, a set of jdgments for the releance of series of qeries (known in REC as topics) against the collection (known as a corps) can help to bild knowledge of the collection, and compare reslts from one year to other years. At the Arctic Region Spercompting Center (ARSC), we hae participated in REC s ery large corps track (VLC), million qery track (MQ) and terabyte track (B) in past years. hese tracks hae proided tens of thosands of niqe REC topics for ealation against sbseqently larger collections of Web-based docments. In [4], we sed an early approach to the diide and conqer method described in [4]. his followed past experiments based on techniqes from [] and other methods for merging reslts from separate systems. For the 2008 REC million qery track, or approach was to portray collections (which were sb-collections of the entire corps, based on specific Web sites within.go) as ectors in an LSI space. For a gien qery, we determined the most similar collections, based on the cosine between the collection ectors and qery ectors. he fifty collections with the highest cosines were then searched by LSI sing the algorithm as described aboe, rather than the whole set of oer 6,000 collections. By applying LSI techniqes at the collection leel, to rank and select collections prior to LSI at the docment leel, we were able to obtain greater efficiency by selecting only those collections deemed to be more highly related to a qery. For practical prposes with distribted search, this is important since the incremental cost for adding more collections (in terms of latency and larger sets of docments to rank) can be aoided by only searching collections of greater interest Performance ealation From REC data, we selected 320 docments, 064 qeries, and 645,27 terms. he relatiely small nmber of docments made it feasible to perform the mltiple experiments described here. Ongoing research will address larger sets of docments. We applied the LSI algorithm to this collection and kept arios percentage of the singlar ales: k = 0, 20,, 80. We then compared the ranking reslt of consectie permtations of k sing the following k able. Aerage sample probability 95 confidence interals that the first 0 elements of the first ranking sing k singlar ales (rows) are contained in the first 0 of the elements of the second ranking sing k singlar ales (colmns). Sample Probability measrements: We arge that if LSI is stable, then the top 0 of each reslt needs to hae almost the same elements as another ranking with different ctoff of singlar ales. So, we calclate the sample probability that the first 0 elements of the first ranking are contained in the first 0 of the elements of the second ranking. his was done for each qery. We then took the aerage oer all 064 qeries. Ideally, we want this sample probability to be as close to one as possible. For example, for each qery we rn LSI with k = 0 of maximm k and with k = 20 of maximm k. We then obtain the ranking from the two ales of k. We constrct the sample probability that the raking from LSI 20 is contained in the ranking of LSI 0. We then take the aerage of the resltant sample probabilities from all 064 qeries. We repeat for LSI 0 s. LSI 30, etc. he resltant aerage sample probability and the corresponding 95 (sample) confidence interal are smmarized in able. 5

6 We obsere that the confidence interals are ery large, ranging between 0.4 and We also obsere that the sample probability is at its max when comparing performance of LSI k with LSI k + 0 of max k and decreases monotonically as the difference between the nmber of singlar ales incorporated in LSI increases. For example, this probability is 0.64, 0.50, 0.42, 0.37,,, 0.27, 0.26 for 0 s. 20, 30, 40, 50, 60, 70, 80, 90, respectiely. On the other hand, the maximm aerage sample probability achieed by comparing LSI k with LSI k + r increases generally as k increases, reaching as high as 0.92 when comparing LSI 80 with LSI 90. Howeer, rnning LSI in this case takes almost a week on a fast machine, and it defeats the rationale behind the se of LSI s. VSM. hs, it is clear from or analysis that LSI is not robst with respect to the choice of the nmber of singlar ales. In fact, the ranking reslts of LSI depend on the choice of k no matter how high the chosen nmber of singlar ales is. his alone brings the integrity of LSI reslts into qestion. Add to this the fact that LSI is orders of magnitde slower than VSM. Based on the eidence from this test, we mst conclde that LSI may be insfficiently robst as a ranking algorithm for information retrieal. 6. Conclsions In this paper, we hae inestigated the stability of the latent semantic indexing (LSI) algorithm. We applied it to sb-collections from REC Million Qery datasets sing different singlar ale ct-offs. he reslts of or analysis show that the LSI ranking is ery sensitie to the choice of the nmber of singlar ales retained. his pt the robstness of LSI into qestion. Or ongoing research will inclde larger collections, inclding the 2009 REC Web track dataset of oer billion docments. Frther analysis of other sb-collections, and comparison of ranking stability to REC ealation measres sch as mean aerage precision, are needed in order to better nderstand the impact of this ariability in LSI rankings for different nmbers of singlar ales. References [] L. Calé and J. Saoy, Database merging strategy based on logistic regression. Information Processing and Management, 2000, 36(3): [2] S. Deerwester, S.. Dmais,. K. Landaer, G. W. Frnas, and R. A. Harshman, Indexing by latent semantic analysis, Jornal of the Society for Information Science, 990, 4(6): [3] M. Efron, Eigenale-based model selection dring latent semantic indexing, Jornal of the American Society for Information Science and echnology, 2005, 56(9): [4] C. Fallen, G. B. Newby, and K. McCormick, Distribted mltisearch and resorce selection for the REC Million Qery track. Gaithersbrg: National Institte of Standards and echnology, [5] K. Gamiel, G. B. Newby, and N. Nassar, Grid information retrieal reqirements (GFD-I.027). Lamont, Illinois: Open Grid Form, [6] E. M. Garzón and I. García, Soling eigenproblems on mlticompters: two different approaches, International Jornal of Compters and Applications, 2004, 26(4): [7] M. R. Garracino, F. Perla, and P. Zanetti, A parallel block Lanczos algorithm and its implementation for the ealation of some eigenales of large sparse symmetric matrices on mlticompters, International Jornal on Applied Mathematics and Compter Science, 2006, 6(2): [8] S. Raghaan and H. Garcia-Molina, Crawling the hidden web, Proceedings of Very Large Databases (VLDB), 200. [9] G. Salton, A. Wong, and C. S. Yang, A Vector Space Model for Atomatic Indexing, Commnications of the ACM, 975, 8(): [0] L. Schamber, M. B. Eisernberg, and M. Nilan, A reexamination of releance: toward a dynamic, sitational definition, Information Processing and Management, 990, 26(6): [] E. Voorhees, Oeriew of REC 2007, Proceedings of the 6th Annal ext REtrieal Conference. Gaithersbrg: National Institte of Standards and echnology, [2] J. X, H. Chen, Z. Zho, and J. Qin, On the topology of the dark web of terrorist grops, Proceedings of the IEEE International Conference on Intelligence and Secrity Informatics (ISI), [3] G. Golb and A. an Loan, Matrix Comptations, Johns Hopkins Uniersity Press, 996. [4] H. Kettani and G. B. Newby, On the ranking of text docments from large corpses, proceedings of the 2009 International Conference on Data Mining (DMIN'09), Las Vegas, Neada, Jly