Lecture 5: Latent Semantic Indexing. Independence. Dealing with Topics. Latent Semantic Indexing. Linear Algebra

Transcription

1 Idepedece Iformatio Retrieval ad Web Search Egies Lecture 5: Latet Sematic Idexig November 26 th, 2013 Wolf-Tilo Balke ad Kida El Maarry Istitut für Iformatiossysteme Techische Uiversität Brauschweig May iformatio retrieval models assume idepedet (orthogoal) terms This is problematic (syoyms, ) What ca we do? Use idepedet topics istead of terms! What do we eed? How to relate sigle terms to topics? How to relate documets to topics? How to relate query terms to topics? 2 Dealig with Topics Latet Sematic Idexig Naïve approach: 1. Fid a libraria who kows the subject area of your documet collectio well eough 2. Let him/her idetify idepedet topics The difficult part 3. Let him/her assig documets to topics A documet about sports gets a weight of 1.1 with Ca respect it be to the automated? topic politics A documet about the vector space model gets a weight of 2.7 with respect to the topic iformatio retrieval 4. Fid a method to trasform queries over terms ito queries over The topics easy (e.g. by part exploitig term/topic assigmets provided by the libraria) Latet Sematic Idexig does the trick Proposed by Dumais et al. (1988) Pateted i 1988 (US Patet 4,839,853) Cetral idea: Represet each documet withi a latet space of topics Use sigular value decompositio (SVD) to derive the structure of this space The SVD is a importat result from liear algebra 3 4 Liear Algebra Lecture 5: Latet Sematic Idexig 1. Recap of Liear Algebra 2. Sigular Value Decompositio 3. Latet Sematic Idexig Liear algebra is the brach of mathematics cocered with the study of: systems of liear equatios, vectors, vector spaces, ad liear trasformatios (represeted by matrices). Importat tool i Iformatio retrieval Data compressio 5 6

2 Vectors Vectors represet poits i space There are: Row vectors: Matrices Every (m )-matrix A defies a liear map from R to R m by sedig the colum vector x R to the colum vector Ax R m : Colum vectors: Traspose All vectors (ad matrices) cosidered i this course will be real-valued 7 Row i Colum j 8 m Matrix Gallery Square matrix Diagoal matrix Idetity matrix m (a i, j ) = (a j, i ) 0 1 Liear Idepedece A set {x (1),, x (k) } of -dimesioal vectors is liearly depedet if there are real umbers λ 1,, λ k, ot all zero, such that Otherwise, this set is called liearly idepedet If k >, the set is liearly depedet Null vector Rectagular matrix Symmetric matrix 9 10 Liear Spa Liear Spa (2) Let {x (1),, x (k) } be a set of -dimesioal vectors The liear spa (aka liear hull) of this set is defied as: Liear combiatio Idea: The liear spa is the set of all poits i R that ca be expressed by liear combiatios of x (1),, x (k) The liear spa is a subspace of R with dimesio at most k 11 The spa of {x (1),, x (k) } ca be: A sigle poit (0-dimesioal) A lie (1-dimesioal) A plae (2-dimesioal) : spa{(1, 2, 3), (2, 4, 6), (3, 6, 9)} is a lie i R 3 : spa{(1, 0, 0), (0, 1, 0), (0, 0, 1)} = R 3 12

3 Basis Let {x (1),, x (k) } be a set of liearly idepedet -dimesioal vectors spa{x (1),, x (k) } is a k-dimesioal subspace of R Ay poit i spa{x (1),, x (k) } is geerated by a uique liear combiatio of x (1),, x (k) Two bases of R 2 : B 1 = {(1, 0), (0, 1)} (stadard basis) B 2 = {(1, 1), (2, 3)} What are the coordiates of stadard basis poit (3, 4) with respect to basis B 2? B 1 : 3 (1, 0) + 4 (0, 1) = (3, 4) B 2 : 1 (1, 1) + 1 (2, 3) = (3, 4) {x (1),, x (k) } is called a basis of the subset it spas No-Stadard Bases Chage of Basis Ofte it is useful to represet data usig a o-stadard basis: Let B 1 = {x (1),, x (k) } ad B 2 = {y (1),, y (k) } be two bases of the same subspacev R, i.e., spab 1 = V = spab 2 Deviatio Weight Height Size There is a uique trasformatio matrix T such that Tx (i) = y (i), for ay i = 1,, k T ca be used to trasform the coordiates of poits give with respect to base B 1 ito the correspodig coordiates with respect to base B Orthogoality Two bases of R 2 : B 1 = {(1, 1), (2, 3)} B 2 = {(0, 1), (3, 0)} Give a poit p with coordiates (1, 1) wrt. base B 1 What are p s coordiates wrt. base B 2? Scalar product (aka dot product) of vectors x, y R ad legth (orm) of a vector x R : Two vectors x, y R are orthogoal if x y= 0 x T (1, 1) T = (4, 1) 17 α y 18

4 Orthoormality Ay set of mutually orthogoal vectors is liearly idepedet A set of -dimesioal vectors is orthoormal if all vectors are of legth 1 ad are mutually orthogoal A matrix is colum-orthoormal if its set of colum vectors is orthoormal (row-orthoormality is defied aalogously) Rak of a Matrix The rak of a matrix is the umber of liearly idepedet rows i it (or colums; it s the same) The rak of a matrix A ca also be defied as the dimesio of the image of the liear map f(x) = Ax The rak of a diagoal matrix is equal to the umber of its ozero diagoal etries Eigevectors ad Eigevalues is row- ad colum-orthoormal; its rak is 4 is row-orthoormal; its rak is 3 Let A be a square ( )-matrix Let x R be a o-zero vector x is a eigevector of A if it satisfies the equatio Ax = λx, for some real umber λ The, λ is called a eigevalue of A correspodig to the eigevector x Idea: Eigevectors are mapped to itself (possibly scaled) Eigevalues are the correspodig scalig factors Mafred Eige Uit vector x Vector Ax (image of x) Eigevectors multiplied by eigevalues It could be useful to chage the basis to the set of eigevectors Lecture 5: Latet Sematic Idexig 1. Recap of Liear Algebra 2. Sigular Value Decompositio 3. Latet Sematic Idexig Source:

5 Sigular Value Decompositio Sigular Value Decompositio (2) Let A be a (m )-matrix (rectagular!) Let r be the rak of A A ca be decomposed such that A = U S V, where U is a colum-orthoormal (m r)-matrix V is a row-orthoormal (r )-matrix S is a diagoal matrix such that S = diag(s 1,, s r ) ad s 1 s 2 s r > 0 The colums of U are called left sigular vectors The rows of V are called right sigular vectors s i is referred to as A s i-thsigular value A m = m U r columorthoormal, left sigular vectors, rak r The liear map A ca be split ito three mappig steps: Give x R, it is Ax = USVx V maps x ito space R r, S scales the compoets of Vx U maps SVx ito space R m The same holds for y R m ; it is ya = yusv r S r diagoal, sigular values, rak r r V row-orthoormal, right sigular vectors, rak r (2) We measured the height ad weight of several persos: Perso 1 Perso 2 Perso 3 Perso 4 Perso 5 Height 170cm 175cm 182cm 183cm 190cm Weight 69kg 77kg 77kg 85kg 89kg Compute the SVD of this data matrix: Weight The colums of this product provide the ew basis 0.5 U S V 27 Note: Axes are orthogoal, but they do ot look like that (due to scalig) Iformatio Retrieval 2 ad Web Search Egies Wolf-Tilo Balke ad Kida El Maarry Techische Uiversität Brauschweig Height 28 Low Rak Approximatio A = USV U R m r : colum-orthoormal S R r r : diagoal V R r : row-orthoormal Sice S is diagoal, A ca be writte as a sum of matrices: Low Rak Approximatio (2) The i-th summad is scaled by s i Remember: s 1 s 2 s r > 0 The first summads are most importat The last oes have low impact o A (if their s i s are small) First sigular value First left sigular vector (colum vector) First right sigular vector (row vector) 29 Idea: Get a approximatio of A by removig some less importat summads This saves space ad could remove small oise i the data 30

6 Low Rak Approximatio (3) Rak-kapproximatio of A (for ay k = 0,, r): Low Rak Approximatio (4) Rak-kapproximatio of A (for ay k = 0,, r): Let U k deote the matrix U after removig the colums k + 1 to r Let S k deote the matrix S after removig both the rows ad colums k + 1 to r Let V k deote the matrix V after removig the rows k + 1 to r The it is A k = U k S k V k 31 How large is the approximatio error? The error ca be measured usig the Frobeius distace The Frobeius distace of two matrices A, B R m is: Roughly the same as the mea squared etry-wise error 32 Low Rak Approximatio (5) Low Rak Approximatio (6) For ay (m )-matrix B of rak at most k, it is d F (A, B) d F (A, A k ) It is Therefore, A k is a optimal rak-k approximatio of A If the sigular values startig at s k+1 are small eough, the approximatio A k is good eough (2) Let s igore the secod axis SVD: Idea: Project data poits ito a 1-dimesioal subspace of the origial 2-dimesioal space, while miimizig the error itroduced by this projectio. 0.5 Rak-1 approximatio:

7 Coectio to Eigevectors Let A be a (m )-matrix ad A = USV its SVD The: Vis row-orthoormal, i.e. VV T = I U s colums are the eigevectors of AA T, the matrix S 2 cotais the correspodig eigevalues Similarly, V s rows are the eigevectors of A T A, S 2 agai cotais the eigevalues S 2 is still diagoal (etries got squared) 37 Lecture 5: Latet Sematic Idexig 1. Recap of Liear Algebra 2. Sigular Value Decompositio 3. Latet Sematic Idexig 38 Latet Sematic Idexig Idea of Dumaiset al. (1988): Apply the SVD to a term documet matrix! The r itermediate dimesios correspod to topics Terms that usually occur together get budled (syoyms) Terms havig several meaigs get assiged to several topics (polysemes) Discardig dimesios havig small sigular values removes oise from the data Low rak approximatios ehace data quality! from (Berry et al., 1995): A small collectio of book titles (2) Term documet matrix (biary, sice o term occured more tha oce): (3) The first two dimesios of the SVD: Books ad terms are plotted usig the ew basis coordiates Similar terms have similar coordiates 41 42

8 Mappig ito Latet Space How to exactly map documets ad terms ito the latet space? Recall: A k = U k S k V k To get rid of the scalig factors (sigular values), S k usually is split up ad moved ito U k ad V k : Let S k 1/2 deote the matrix that results from extractig square roots from S k (etry-wise) Defie U k = U k S k 1/2 ad V k = S k 1/2 V k, which gives A k = U k V k The: The latet space coordiates of the j-th documet are give by the j-th colum of V k The i-th term s coordiates are give by the i-th row of U k 43 Processig Queries How does queryig work? Idea: Map the query vector q R m ito the latet space But: How to map ew documets/queries ito the latet space? Let q R k deote the query s (yet ukow) coordiates i latet space Assumig that q ad q are colum vectors, we kow that the followig must be true (by defiitio of the SVD): 44 Processig Queries (2) Now, let s solve this equatio with respect to q : Multiply by U kt o the left-had side: Multiply by S k 1/2 (the etry-wise reciprocal of S 1/2 ): Query = applicatio theory All books withi the shaded area have a cosie similarity to the query of at least 0.9 Thus, fially: Aother Aother (2) by Mark Girolami (Uiversity of Glasgow) Documets from a collectio of Useet postigs 47 48

9 Aother (3) Yet Aother Reuters collectio short ewswire messages from 1987 Top-3 results whe queryig for taxes reaga usig LSI: FITZWATER SAYS REAGAN STRONGLY AGAINST TAX HIKE WASHINGTON, March 9 - White House spokesma Marli Fitzwater said Presidet Reaga's record i opposig tax hikes is "log ad strog" ad ot about to chage. ROSTENKOWSKI SAYS WILL BACK U.S. TAX HIKE, BUT DOUBTS PASSAGE WITHOUT REAGAN SUPPORT WHITE HOUSE SAYS IT OPPOSED TO TAX INCREASE AS UNNECESSARY 49 The last documet does t metio the term reaga! 50 A Differet View o LSI A Differet View o LSI (2) Use a model similar to eural etworks : m = 3, = 4 SVD represetatio: rak(a) = Rows Rows Colums 3 4 For a give colum, its rows i V represet the colum s coectios stregth to the topics Colums A Differet View o LSI (3) A Differet View o LSI (4) Recostructio of A by multiplicatio: What does this mea for term documet matrices? Rows a 2, 1 = ( 0.7) = (roudig errors) Terms Colums Documets

10 A Differet View o LSI (5) What documets cotai term 2? A Differet View o LSI (6) The SVD itroduces a itermediate layer: Terms 3 Terms Documets Documets A Differet View o LSI (7) Computig the SVD Remove uimportat topics: Terms Documets Computig the SVD o large matrices is at least very difficult Traditioal algorithms require matrices to be kept i memory There are more specialized algorithm available, but computatios still takes a log time o large collectios We have ot bee able to fid ay LSI experimet ivolvig more tha 1,000,000 documets Alterative: Compute LSI o a subset of the data Recetly, quite simple approximatio algorithms have bee developed that require much less memory ad are relatively fast For example, based o gradiet descet Maybe those approaches will make LSI easier to use i the future 58 What s the k? A cetral questio remais: How may dimesios k should be used? It s a tradeoff: Too may dimesios make computatio expesive ad lead to performace degradatio i retrieval (o oise gets filtered out) Too few dimesios also lead to performace degradatio sice importat topics are left out The right k depeds o the collectio: How specialized is it? Are there special types of documets? What s the k? (2) Ladauer ad Dumais (1997) evaluated retrieval performace as a fuctio of k: 59 60

11 Pros ad Cos The Netflix Prize Pros Very good retrieval quality Reasoable mathematical foudatios Geeral tool for differet purposes Cos Latet dimesios foud might be difficult to iterpret High computatioal requiremets The right k is hard to fid Netflix: Large DVD retal service The Netflix Prize Wi $1,000,000 Dataset of customers DVD ratigs: 480,189 customers 17,700 movies 100,480,507 ratigs (scale: 1 5) Desity of ratig matrix: Task: Estimate 2,817,131 ratigs ot published by Netflix The Netflix Prize (2) The Netflix Prize (3) Computig a (sort of) SVD o the ratig matrix has bee proved to be highly successful Mai problem here: The matrix is very sparse! Sparse meas missig kowledge (i cotrast to LSI!) SVD o Ratig Data SVD o Ratig Data (2) Each movie ca be represeted as a poit i some k-dimesioal coordiate space May iterestig applicatios Fidig similar movies: Rocky (1976) Dirty Dacig (1987) The Birds (1963) Rocky II (1979) Pretty Woma (1990) Psycho (1960) Rocky III (1982) Footloose (1984) Vertigo (1958) Hoosiers (1986) Grease (1978) Rear Widow (1954) The Natural (1984) Ghost (1990) North By Northwest (1959) The Karate Kid (1984) Flashdace (1983) Dial M for Murder (1954) Automatically reweightig gere assigmets: Movie IMDb s geres Reweighted geres Back to the Future III (1990) Adveture Comedy Family Sci-Fi Wester Rocky (1976) Drama Romace Sport Star Trek (1979) Actio Adveture Mystery Sci-Fi Titaic (1997) Adveture Drama History Romace Adveture Comedy Family Sci-Fi Wester Drama Romace Sport Actio Adveture Mystery Sci-Fi Adveture Drama History Romace 65 66

12 Next Lecture Laguage models What is relevace? Evaluatio of retrieval quality 67