COMPSCI 514: Algorithms for Data Science

Transcription

1 COMPSCI 514: Algoritms for Data Science Arya Mazumdar University of Massacusetts at Amerst Fall 2018

2 Lecture 11 Locality Sensitive Hasing

3 Midterm exam Average out of 35 (82.8%). One exam is still pending grading (we received late).

4 Finding near-neigbors in a ig dimensional space Near neigbors: Points tat are only a small distance away Have to define distance or similarity For example, Jaccard distance/similarity

5 Minas signature Take n (say 100) random permutations Eac define a minas function: 1, 2,..., n Minas signature of S is a vector [ 1 (S), 2 (S),..., n (S)] T n is muc smaller tan te number of rows of te caracteristic matrix Tis constitute a signature matrix Intuition: Similarity between sets implies similarity between signature vectors

6 Row S 1 S 2 S 3 S 4 x +1 mod5 3x +1 mod Fast computation of signature matrix 84 CHAPTER 3. FINDING SIMILAR ITEMS Pick n random as functions 1, 2,..., n SIG(i, c) is te it row and ct column of te signature order. matrix k is large and tere are not too many collisions. We can maintain te fiction tat our as function permutes row r to position (r) in te permuted Tus, instead of picking n random permutations of rows, we pick n randomly cosen Initialization: as functionsset 1,SIG(i, 2,..., n c) on= te rows. for all We iconstruct and c te signature matrix by considering eac row in teir given order. Let SIG(i, c) be te element of te Forsignature te rtmatrix row: for te it as function and column c. Initially, set SIG(i, c) to for all i and c. We andle row r by doing te following: 1. Compute 1 (r), 2 (r),..., n (r). 2. For eac column c do te following: (a) If c as 0 in row r, do noting. (b) However, if c as 1 in row r, tenforeaci =1, 2,...,nset SIG(i, c) to te smaller of te current value of SIG(i, c) and i (r).

7 2. For eac column c do te following: Fast computation of signature matrix: Example (a) If c as 0 in row r, do noting. (b) However, if c as 1 in row r, tenforeaci =1, 2,...,nset SIG(i, c) to te smaller of te current value of SIG(i, c) and i (r). Row S 1 S 2 S 3 S 4 x +1 mod5 3x +1 mod Figure 3.4: Has functions computed for te matrix of Fig. 3.2 Two as functions: 1 (x) = x + 1 mod 5; 2 (x) = 3x + 1 mod 5. Example 3.8 : Let us reconsider te caracteristic matrix of Fig. 3.2, wic we reproduce wit some additional data as Fig We ave replaced te letters naming te rows by integers 0 troug 4. We ave also cosen twoas functions: 1 (x) =x+1 mod 5 and 2 (x) =3x+1 mod 5. Te values of tese two functions applied to te row numbers are given in te last two columns of

8 Figure 3.4: Has functions computed for te matrix of Fig. 3.2 (a) If c as 0 in row r, do noting. Fast computation of signature matrix: Initialization (b) However, if c as 1 in row r, tenforeaci =1, 2,...,nset SIG(i, c) xample 3.8 to : te Let smaller us reconsider of te current te caracteristic value of SIG(i, c) matrix and i of (r). Fig. 3.2, wic reproduce wit some additional data as Fig We ave replaced t tters naming te rows by integers 0 troug 4. We ave also cosen twoas nctions: Row S 1 S 2 S 3 S 4 x +1 mod5 3x +1 mod5 1 (x) =x+1 mod 5 and 2 (x) =3x+1 mod 5. Te values of tes o functions 0 applied 1 to 0 te 0 row numbers 1 are 1 given in te 1 last two columns o g Notice tat tese simple as functions are true permutations of t ws, but a true permutation is only possible because te number of rows, 5, i prime. In general, 4 0 tere 0 will 1 be 0 collisions, 0 were two rows 3 get te same as lue. Now, let us simulate te algoritm for computing te signature matrix itially, tis Figure matrix 3.4: Has consists functions of all computed s: for te matrix of Fig. 3.2 S 1 S 2 S 3 S 4 Example 3.8 : Let us reconsider te caracteristic matrix of Fig. 3.2, wic we reproduce wit some additional 1 data as Fig We ave replaced te letters naming te rows by integers 2 0 troug 4. We ave also cosen twoas functions: 1 (x) =x+1 mod 5 and 2 (x) =3x+1 mod 5. Te values of tese First, we consider row 0 of Fig We see tat te values of two functions applied to te row numbers are given in te last two columns 1 (0) an of (0) Fig. are 3.4. bot Notice 1. tat Tetese row numbered simple as0 functions as 1 s inare te true columns permutations for sets ofste 1 an

9 (a) If c as 0 in row r, do noting. Fast computation of signature matrix: Row 0 (b) However, if c as 1 in row r, tenforeaci =1, 2,...,nset SIG(i, c) to te smaller of te current value of SIG(i, c) and i (r). Row S 1 S 2 S 3 S 4 x +1 mod5 3x +1 mod SIMILARITY-PRESERVING SUMMARIES 3 OF SETS so only tese columns of te signature matrix can cange. As 1 is less t we do in fact cange bot values in te columns for S 1 and S 4.Tecurr mate offigure te signature 3.4: Hasmatrix functions iscomputed tus: for te matrix of Fig. 3.2 S 1 S 2 S 3 S 4 Example 3.8 : Let us reconsider te caracteristic matrix of Fig. 3.2, wic we reproduce wit some additional data as Fig We ave replaced te letters naming te rows by integers troug 4. We ave also cosen twoas Now, functions: we move 1 (x) =x+1 to te row mod numbered 5 and 2 (x) 1=3x+1 in Fig. mod Tis Te values row as of tese 1 only two functions applied to te row numbers are given in te last two columns of and its as values are Fig Notice tat tese simple 1 (1) = 2 and as functions 2 (1) = 4. Tus, we set SIG(1, 3) t are true permutations of te SIG(2, 3) to 4. All oter signature entries remain as tey are because t

10 . SIMILARITY-PRESERVING SUMMARIES OF SETS (a) If c as 0 in row r, do noting. Fast computation of signature matrix: Row 1 (b) However, if c as 1 in row r, tenforeaci =1, 2,...,nset SIG(i, c), so only tese columns of te signature matrix can cange. As 1 is less t to te smaller of te current value of SIG(i, c) and i (r)., we do in fact cange bot values in te columns for S 1 and S 4.Tecurre imate of te signature matrix is tus: Row S 1 S 2 S 3 4 x +1 mod5 3x +1 mod S 1 1 S 2 1 S 3 S Now, we move 4 to 0 te 0 row1 numbered 0 1 in 0 Fig Tis 3 row as 1 only, and its as values are 1 (1) = 2 and 2 (1) = 4. Tus, we set SIG(1, 3) to d SIG(2, 3) to 4. All oter signature entries remain as tey are because te lumns ave Figure 0 in 3.4: te Has rowfunctions numbered computed 1. Tefor new tesignature matrix of matrix: Fig. 3.2 S 1 S 2 S 3 S 4 Example 3.8 : Let us reconsider te caracteristic matrix of Fig. 3.2, wic we reproduce wit some additional data as Fig We ave replaced te letters naming te rows by integers troug 4. We ave also cosen twoas functions: Te row of 1 (x) Fig. =x numbered mod 5 and2 as 2 (x) 1 s =3x+1 in temod columns 5. Te for values S of tese 2 and S 4,an two as functions values are applied to te row numbers are given in te last two columns of 1 (2) = 3 and 2 (2) = 2. We could cange te values in t Fig Notice tat tese simple as functions are true permutations of te nature for S 4, but te values in tis column of te signature matrix, [1, 1], a

11 and its(a) as If cvalues as 0 inare row 1 r, (1) do= noting. 2 and 2 (1) = 4. Tus, we set SIG(1, 3) SIG(2, (b) 3) Fast to However, 4. computation Allifoter c as signature 1 in row ofr, signature tenforeaci entries remain matrix: =1, as2,...,nset tey Roware 2because SIG(i, c) t mns ave to 0 in te te smaller rowof numbered te current1. value Teof new SIG(i, signature c) and i (r). matrix: S 1 S 2 S 3 S 4 Row S 1 S 2 S 3 S 4 x +1 mod5 3x +1 mod e row of2fig numbered as 1 s3in te columns 2 for S 2 and S 4, as values are (2) = 3 and (2) = 2. We could cange te values in 0 3 ature for S 4, but te values in tis column of te signature matrix, [1, 1], less tan te corresponding as values [3, 2]. However, since te colu Figure 3.4: Has functions computed for te matrix of Fig still as s, we replace it by [3, 2], resulting in: S 1 S 2 S 3 S 4 Example 3.8 : Let us reconsider te caracteristic matrix of Fig. 3.2, wic we reproduce wit some additional data as Fig We ave replaced te letters naming te rows byintegers troug We1 ave also cosen twoas functions: 1 (x) =x+1 mod 5 and 2 (x) =3x+1 mod 5. Te values of tese ext comes te row numbered 3 in Fig Here, all columns but S two functions applied to te row numbers are given in te last two columns of 2 nd Fig. te 3.4. as Notice values tat are tese 1 simple (3) = as 4 andfunctions 2 (3) = are 0. true Tepermutations value 4 for of 1 te exce t is already in te signature matrix for all te columns, so we sall

12 r S 2 still (a) as If c s, as 0we inreplace row r, do it noting. by [3, 2], resulting in: Fast computation of signature matrix: Row 3 (b) However, if c as 1 in row r, tenforeaci =1, 2,...,nset SIG(i, c) S to te smaller of te current 1 S value 2 S 3 S of SIG(i, 4 c) and i (r) Row S 1 S 2 S 3 S 4 x +1 mod5 3x +1 mod5 Next comes 0 te 1row 0numbered 0 13 in Fig Here, all columns 1 but S 2 av and te as 1 values 0 are 0 1 1(3) = 04 and 2 (3) 2 = 0. Te value 4 4 for 1 exceed at is already 2 in 0te signature 1 0 1matrix for3 all te columns, 2 so we sall no ange any values 3 in 1 te0 first1 row1of te signature 4 matrix. 0However, te valu for 2 is less4 tan0wat 0 is already 1 0 present, 0so we lower SIG(2, 3 1), SIG(2, 3) and G(2, 4) to 0. Note tat we cannot lower SIG(2, 2) because te column for S 2 in g. 3.4 as Figure 0 in 3.4: te Has row we functions are currently computed considering. for te matrix Te of resulting Fig. 3.2 signatur atrix: S 1 S 2 S 3 S 4 Example 3.8 : Let us reconsider te caracteristic matrix of Fig. 3.2, wic we reproduce wit some additional data as Fig We ave replaced te letters naming te rows by integers troug We 0 ave also cosen twoas functions: Finally, consider 1 (x) =x+1 te row mod of 5 Fig. and (x) numbered =3x+1 4. mod 5. Te values of tese 1 (4) = 0 and 2 (4) = 3 two functions applied to te row numbers are given in te last two columns of nce row 4 as 1 only in te column for S Fig Notice tat tese simple as functions 3, we only compare te curren are true permutations of te nature column for tat set, [2, 0] wit te as values [0, 3]. Since 0 < 2, w

13 ange any values in te first row of te signature matrix. However, te valu (a) If c as 0 in row r, do noting. for 2 is less tan wat is already present, so we lower SIG(2, 1), SIG(2, 3) and (b) Fast However, computation if c as 1 in row ofr, signature tenforeaci matrix: =1, 2,...,nset Row 4 G(2, 4) to 0. Note tat we cannot lower SIG(2, 2) because te column SIG(i, for Sc) 2 in ig. 3.4 as to 0 in te te smaller row we of te are currently value considering. of SIG(i, c) Te andresulting i (r). signatur atrix: Row S 1 S 2 S 3 S 41 x S 2 +1Smod5 3 S 4 3x +1 mod Finally, consider 3 1 te0row 1of Fig numbered (4) 0= 0 and 2 (4) = 3 ince row 4 4as 10only0 in te 1 column 0 for0s 3, we only compare 3 te curren gnature column for tat set, [2, 0] wit te as values [0, 3]. Since 0 < 2, w ange SIG(1, Figure 3) 3.4: to 0, Has but functions since 3 computed > 0wedonotcange for te matrix of SIG(2, Fig. 3). 3.2 Te fina gnature matrix is: S Example 3.8 : Let us reconsider 1 S te caracteristic 2 S 3 S 4 matrix of Fig. 3.2, wic we reproduce wit some additional 1 1 data 3 as0fig We ave replaced te letters naming te rows by integers troug We 0 ave also cosen twoas functions: 1 (x) =x+1 mod 5 and 2 (x) =3x+1 mod 5. Te values of tese two functions applied to te row numbers are given in te last two columns of Fig Notice tat tese simple as functions are true permutations of te rows, but a true permutation is only possible because te number of rows, 5, is

14 Locality Sensitive Hasing Finding similarity for every pair is still difficult Find te closest pair/ Near neigbor searc Has items: suc tat similar items are ased into te same bucket Ceck pairs witin te buckets False positives: dissimilar items in te same buckets - make it small False negatives: similar items are in different buckets - make it small

15 Measure of Distance A distance satisfy tree conditions: 1) nonnegativity (is zero iff measuring to itself) 2) symmetry and 3) triangle inequality Different measures of distances are possible L p : d(x, y) = ( n i=1 x i y i p ) 1/p Wat is L? L : d(x, y) = max n i=1 x i y i Wat is L 0? L 0 or Hamming distance: d(x, y) = {i : x i y i } Jaccard: d(a, B) = 1 A B A B Cosine distance: d(x, y) = arccos x,y x 2 y 2

16 Measure of Distance Edit distance d(x, y) = number of insertions, deletions to cange a string to anoter d(abcde, acfdeg) = 3

17 Figure 3.9 illustrates wat we expect about te probability tat a given In many cases, te function f will as items, and te decision will be based on weter or not Locality te result is equal. sensitive Because itasing is convenient to use te notation f(x) =f(y) tomeantatf(x, y) is yes; make x and y a candidate pair, we sall use f(x) =f(y) as a sortand wit tis meaning. We also use f(x) f(y) tomean donotmakex and y a candidate pair unless some oter function concludes we sould do so. Suppose a distance measure is fixed. A collection of functions of tis form will be called a family of functions. We are given a as function f For example, te family of minas functions, eac based on one of te possible f is permutations called (d of rows 1, dof 2, apcaracteristic 1, p 2 )-LSH matrix, if form a family. Let d 1 <d 2 be two distances according to some distance measure d. A 1. family forf any of functions two x, is said y suc to be (dtat 1,d 2,pd(x, 1,p 2)-sensitive y) dif 1 for, Pr(f every f(x) in F: = f (y)) p for If d(x, any) two d 1, ten x, yte suc probability tat tat d(x, f(x) y) =f(y) dis 2, atpr(f least p(x) 1. = f (y)) p 2 2. If d(x, y) d 2, ten te probability tat f(x) =f(y) is at most p 2. p 1 Probabilty of being declared a candidate p 2 d 1 d 2 Distance Figure 3.9: Beavior of a (d 1,d 2,p 1,p 2)-sensitive function

18 LSH for Jaccard distance Minas wit a random permutation is (d 1, d 2, 1 d 1, 1 d 2 )-LSH for any d 1, d 2 Wy? Pr((A) = (B)) = Sim(A, B) = 1 d(a, B)

19 LSH Amplification Let f be a (d 1, d 2, p 1, p 2 )-LSH Consider independent copies of f : f 1, f 2, f 3,..., f r AND construction F AND (x) = F AND (y) iff f i (x) = f i (y) for all i = 1,..., r F AND is a (d 1, d 2, p r 1, pr 2 )-LSH OR construction F OR (x) = F OR (y) iff tere exists some i {1,..., r} wit f i (x) = f i (y) F OR is a (d 1, d 2, 1 (1 p 1 ) r, 1 (1 p 2 ) r )-LSH

20 LSH Amplification: Cascading AND and OR constructions Let f be a (d 1, d 2, 0.9, 0.2)-LSH Consider f wic is a 4-fold AND construction of f f is (d 1, d 2, , )-LSH Consider f wic is a 4-fold OR construction of f f is (d 1, d 2, , )-LSH

21 LSH for Hamming distance Let f is te value of randomly cosen coordinate from te all n coordinates Pr(f (x) = f (y)) = {i:x i y i } n = 1 d(x,y) n f is (d 1, d 2, 1 d 1 n, 1 d 2 n )-LSH for any d 1, d 2