24 Multiple Eigenvectors; Latent Factor Analysis; Nearest Neighbors

Transcription

1 Multiple Eigenvectrs; Latent Factr Analysis; Nearest Neighbrs Multiple Eigenvectrs; Latent Factr Analysis; Nearest Neighbrs Clustering w/multiple Eigenvectrs [When we use the Fiedler vectr fr spectral graph clustering, it tells us hw t divide a graph int tw graphs. If we want mre than tw clusters, we can use divisive clustering: we repeatedly cut the subgraphs int smaller subgraphs by cmputing their Fiedler vectrs. Hwever, there are several ther methds t subdivide a graph int k clusters in ne sht that use multiple eigenvectrs rather than just the Fiedler vectr v 2. These methds are usually faster and smetimes give better results. They use k eigenvectrs in a natural way t cluster a graph int k subgraphs.] Fr k clusters, cmpute first k eigenvectrs v =, v 2,...,v k f generalized eigensystem Lv = Mv. V = V [V s clumns are the eigenvectrs with the k smallest eigenvalues.] v 2 [Yes, we d include the all- s vectr v as ne f = the clumns f V.] v k V n [Draw this by hand. eigenvectrs.pdf ] n k Rw V i is spectral vectr [my name] fr vertex i. [The rws are vectrs in a k-dimensinal space I ll call the spectral space. When we were using just ne eigenvectr, it made sense t cluster vertices tgether if their cmpnents were clse tgether. When we use mre than ne eigenvectr, it turns ut that it makes sense t cluster vertices tgether if their spectral vectrs pint in similar directins.] Nrmalize each rw V i t unit length. [Nw yu can think f the spectral vectrs as pints n a unit sphere centered at the rigin.] [Draw this by hand vectrclusters.png ] [A 2D example shwing tw clusters n a circle. If the graph has k cmpnents, the pints in each cluster will have identical spectral vectrs that are exactly rthgnal t all the ther cmpnents spectral vectrs (left). If we mdify the graph by cnnecting these cmpnents with small-weight edges, we get vectrs mre like thse at right nt exactly rthgnal, but still tending tward distinct clusters.] k-means cluster these vectrs. [Because all the spectral vectrs lie n the sphere, k-means clustering will cluster tgether vectrs that are separated by small angles.]

2 48 Jnathan Richard Shewchuk cmpkmeans.png, cmpspectral.png [Cmparisn f pint sets clustered by k-means just k-means by itself, that is vs. a spectral methd. T create a graph fr the spectral methd, we use an expnentially decaying functin t assign weights t pairs f pints, like we used fr image segmentatin but withut the brightnesses.] Invented by [ur wn] Prf. Michael Jrdan, Andrew Ng [when he was still a student at Berkeley], Yair Weiss. [This wasn t the first algrithm t use multiple eigenvectrs fr spectral clustering, but it has becme ne f the mst ppular.]

3 Multiple Eigenvectrs; Latent Factr Analysis; Nearest Neighbrs 49 LATENT FACTOR ANALYSIS [aka Latent Semantic Indexing] [Yu can think f this as dimensinality reductin fr matrices.] Suppse X is a term-dcument matrix: [aka bag-f-wrds mdel] rw i represents dcument i; clumn j represents term j. [Term = wrd.] [Term-dcument matrices are usually sparse, meaning mst entries are zer.] X ij = ccurrences f term j in dc i better: lg (+ ccurrences) [S frequent wrds dn t dminate.] [Better still is t weight the entries s rare wrds give big entries and cmmn wrds like the give small entries. T d that, yu need t knw hw frequently each wrd ccurs in general. I ll mit the details, but this is the cmmn practice.] Recall SVD X = UDV > = dx i= iu i v > i. Suppse i apple j fr i j. Unlike PCA, we usually dn t center X. Fr greatest i, each v i lists terms in a genre/cluster f dcuments each u i dcus in a genre using similar/related terms E.g. u might have large cmpnents fr the rmance nvels, v fr terms passin, ravish, bdice... [... and wuld give us an idea hw much bigger the rmance nvel market is than the markets fr every ther genre f bks.] [v and u tell us that there is a large subset f bks that tend t use the same large subset f wrds. We can read the wrds by lking at the larger cmpnents f v, and we can read the bks by lking at the larger cmpnents f u.] [The prperty f being a rmance nvel is an example f a latent factr. S is the prperty f being the srt f wrd used in rmance nvels. There s nthing in X that tells yu explicitly that rmance nvels exist, but the genre is a hidden cnnectin between them that gives them a large singular value. The vectr u reveals which bks have that genre, and v reveals which wrds are emphasized in that genre.] Like clustering, but clusters verlap: if u picks ut rmances & u 2 picks ut histries, they bth pick ut histrical rmances. [S yu can think f latent factr analysis as a srt f clustering that permits clusters t verlap. Anther way in which it di ers frm traditinal clustering is that the u-vectrs cntain real numbers, and s sme pints have strnger cluster membership than thers. One bk might be just a bit rmance, anther a lt.]

4 50 Jnathan Richard Shewchuk Applicatin in market research: identifying cnsumer types (hipster, sccer mm) & items bught tgether. [Fr applicatins like this, the first few singular vectrs are the mst useful. Mst f the singular vectrs are mstly nise, and they have small singular values t tell yu s. This mtivates apprximating a matrix by using nly sme f its singular vectrs.] Truncated sum X 0 = rx i= iu i v > i is a lw-rank apprximatin f X, f rank r. [We chse the singular vectrs with the largest singular values, because they carry the mst infrmatin.] X 0 = u... 0 v 0 v r r r r r d u r n d n r [Draw this by hand. truncate.pdf ] X 0 is the rank-r matrix that minimizes the [squared] Frbenius nrm X kx X 0 k 2 F = Xij Xij 0 2 Applicatins: i, j Fuzzy search. [Suppse yu want t find a dcument abut gasline prices, but the dcument yu want desn t have the wrd gasline ; it has the wrd petrl. One cl thing abut the reducedrank matrix X 0 is that it will prbably assciate that dcument with gasline, because the SVD tends t grup synnyms tgether.] Denising. [The idea is t assume that X is a nisy measurement f sme unknwn matrix that prbably has lw rank. If that assumptin is partly true, then the reduced-rank matrix X 0 might be better than the input X.] Matrix cmpressin. [As yu can see abve, if we use a lw-rank apprximatin with a small rank r, we can express the apprximate matrix as an SVD that takes up much less space than the riginal matrix. Often this rw-rank apprximatin supprts faster matrix cmputatins.] Cllabrative filtering: fills in unknwn values, e.g. user ratings. [Suppse the rws f X represents Netflix users and the clumns represent mvies. The entry X ij is the review scre that user i gave t mvie j. But mst users haven t reviewed mst mvies. We want t fill in the missing values. Just as the rank reductin will assciate petrl with gasline, it will tend t assciate users with similar tastes in mvies, s the reduced-rank matrix X 0 can predict ratings fr users wh didn t supply any. Yu ll try this ut in the last hmewrk.]

5 Multiple Eigenvectrs; Latent Factr Analysis; Nearest Neighbrs 5 NEAREST NEIGHBOR CLASSIFICATION [We re dne with unsupervised learning. Nw I m ging back t classifiers, and I saved the simplest fr the end f the semester.] Idea: Given query pint q, find the k sample pts nearest q. Distance metric f yur chice. Regressin: Return average label f the k pts. Classificatin: Return class with the mst vtes frm the k pts OR return histgram f class prbabilities. [The histgram f class prbabilities tries t estimate the psterir prbabilities f the classes. Obviusly, the histgram has limited precisin. If k = 3, then the nly prbabilities yu ll ever return are 0, /3, 2/3, r. Yu can imprve the precisin by making k larger, but yu might underfit. The histgram wrks best when yu have a huge amunt f data.] KNN: K=0 KNN: K= KNN: K=00 allnn.pdf (ISL, Figures 2.5, 2.6) [Examples f -NN, 0-NN, and 00-NN. A larger k smths ut the bundary. In this example, the -NN classifier is badly verfitting the data, and the 00-NN classifier is badly underfitting. The 0-NN classifier des well: it s reasnably clse t the Bayes decisin bundary. Generally, the ideal k depends n hw dense yur data is. As yur data gets denser, the best k increases.] [There are therems shwing that if yu have a lt f data, nearest neighbrs can wrk quite well.] Therem (Cver & Hart, 967): As n!, the -NN errr rate is < B(2 B) where B = Bayes risk. if nly 2 classes, apple 2B( B) [There are a few technical requirements f this therem. The mst imprtant is that the training pints and the test pints all have t be drawn independently frm the same prbability distributin just like in ur last lecture, n learning thery. The therem applies t any separable metric space, s it s nt just fr the Euclidean metric.] [By the way, this Cver is the same Thmas Cver f Cver s Therem in the last lecture. He s a prfessr in Electrical Engineering and Statistics at Stanfrd, and these are the first and third jurnal articles he published.] Therem (Fix & Hdges, 95): As n!, k!, k/n! 0, k-nn errr rate cnverges t B. [Which means ptimal.]

6 52 Jnathan Richard Shewchuk The Gemetry f High-Dimensinal Spaces Cnsider shell between spheres f radii r & r. [Draw this by hand cncentric.png ] [Cncentric balls. In high dimensins, almst every pint chsen unifrmly at randm in the uter ball lies utside the inner ball.] Vlume f uter ball / r d Vlume f inner ball / (r ) d Rati f inner ball vlume t uter = (r ) d r d = d exp r d r! which is small fr large d. E.g. if r = 0.&d = 00, inner ball has % f vlume. Randm pints frm unifrm distributin in ball: nearly all are in uter shell. Gaussian : nearly all are in sme shell. [If the dimensin is very high, the majrity f the randm pints generated frm an istrpic Gaussian distributin are apprximately at the same distance frm the center. S they lie in a thin shell. As the dimensin grws, the standard deviatin f a randm pint s distance t the center gets smaller and smaller cmpared t the distance itself. Yu can think f a pint frm a multivariate Gaussian distributin as a sample f d scalar values frm a univariate Gaussian. As d gets bigger, the mean f the squares f the cmpnents cnverges t the true mean fr the ppulatin.] [This is ne f the things that makes machine learning hard in high dimensins. Smetimes the nearest neighbr and the farthest neighbr aren t much di erent.] Exhaustive k-nn Alg. Given query pint q: Scan thrugh all n sample pts, cmputing (squared) distances t q. Maintain a max-heap with the k shrtest distances seen s far. [Whenever yu encunter a sample pint clser t q than the pint at the tp f the heap, yu remve the heap-tp pint and insert the better pint. Obviusly yu dn t need a heap if k = r even 3, but if k = 0 a heap will substantially speed up keeping track f the distance t beat.] Time t cnstruct classifier: 0 [This is the nly O(0)-time algrithm we ll learn this semester.] Query time: O(nd + n lg k) expected O(nd + k lg 2 k) if randm pt rder [It s a cute theretical bservatin that yu can slightly imprve the expected running time by randmizing the pint rder s that nly expected O(k lg k) heap peratins ccur. But in practice I dn t recmmend it; yu ll prbably lse mre frm cache misses than yu ll gain frm fewer heap peratins.]

7 Multiple Eigenvectrs; Latent Factr Analysis; Nearest Neighbrs 53 Randm Prjectin Prjectins int lw-d space like PCA speed up NN, but distances are apprximate. [Mst fast nearest-neighbr algrithms in mre than a few dimensins are apprximate nearest neighbr algrithms; we dn t necessarily expect t find the exact nearest neighbrs. Fr classificatin, that s usually su cient.] Randm prjectin is cheap alternative t PCA as preprcess fr NN r clustering. [Prjects nt a randm subspace instead f the best subspace, but takes a fractin f the time f PCA.] Pick a randm subspace S R d f dimensin k, where k = 2 ln(/ ) 2 /2 3 /3. q d Fr any pt q, let ˆq be k times [rthgnal] prjectin f q nt S. Fr any tw pts q, w 2 R d, ( ) qw 2 apple ˆqŵ 2 apple ( + ) qw 2 with prbability 2. [S the distance between the tw pints after prjecting is rarely much di erent than the distance befre. Fr reasnably reliable clustering r apprximate nearest neighbr search, it s custmary t chse apple /n 2. In practice, yu can experiment with k t find the best speed-accuracy trade. But the key bservatin is that yu need a subspace f dimensin (lg n). The hidden cnstant is large, thugh. Fr example, yu can bring,000,000 sample pints dwn t a 0,362-dimensinal space with a 0% errr in the distances.] [If yu want a prf f this, lk up references abut the Jhnsn Lindenstrauss Lemma.] 00000t000.pdf [Cmparisn f inter-pint distances befre and after prjecting pints in 00,000-dimensinal space dwn t,000 dimensins. This example suggests that the theretical bunds are a bit pessimistic cmpared t practice.] [Why des this wrk? A randm prjectin f a vectr is equivalent t taking a randm vectr and selecting k cmpnents. The mean f the squares f thse k sampled cmpnents apprximates the mean fr the whle ppulatin.] [Hw d yu get a unifrmly distributed randm prjectin directin? Yu can chse each cmpnent frm a univariate Gaussian distributin, then nrmalize the vectr t unit length. Hw d yu get a randm subspace? Yu can chse k randm vectrs, then use Gram-Schmidt rthgnalizatin t make them mutually rthnrmal. Interestingly, Indyk and Mtwani shw that if yu skip the expensive nrmalizatin and Gram-Schmidt steps, randm prjectin still wrks almst as well, because randm vectrs in a highdimensinal space are nearly equal in length and nearly rthgnal t each ther with high prbability.]