Machine Learning. Measuring Distance. several slides from Bryan Pardo

Transcription

1 Machine Learning Measuring Distance several slides from Bran Pardo 1

2 Wh measure distance? Nearest neighbor requires a distance measure Also: Local search methods require a measure of localit (Frida) Clustering requires a distance measure (later) Search engines require a measure of similarit, etc.

3 Euclidean Distance What people intuitivel think of as distance Dimension 2 dd, = ( 1 1 ) Dimension 1

4 Generalized Euclidean Distance ), i( d i i n n n i i i R > =< > =< = = and,...,,,,...,, where ), ( / 1 2 n = the number of dimensions

5 L p norms L p norms are all special cases of this: d(, ) n = i= 1 i i p 1/ p p changes the norm 1 = L 1 norm = Manhattan Distance : p = 1 2 = L 2 norm = Euclidean Distance : p = 2 Hamming Distance : p = 1and i, i { 0,1}

6 Weighting Dimensions Put point in cluster with the closest center of gravit? Which cluster should the red point go in? How do I measure distance in a wa that gives the right answer for both situations?

7 Weighting Dimensions Think Start End Put point in cluster with the closest center of gravit? Which cluster should the red point go in? How do I measure distance in a wa that gives the right answer for both situations?

8 Weighting Dimensions Pair Start End Put point in cluster with the closest center of gravit? Which cluster should the red point go in? How do I measure distance in a wa that gives the right answer for both situations?

9 Weighting Dimensions Share Put point in cluster with the closest center of gravit? Which cluster should the red point go in? How do I measure distance in a wa that gives the right answer for both situations?

10 Weighted Norms You can compensate b weighting our dimensions. d n 1/ (, ) = w p i i i i= 1 p This lets ou turn our circle of equal-distance into an elipse with aes parallel to the dimensions of the vectors.

11 Mahalanobis distance The region of constant Mahalanobis distance around the mean of a distribution forms an ellipsoid. The aes of this ellipsiod don t have to be parallel to the dimensions describing the vector Images from: 11

12 Calculating Mahalanobis d (, ) = ( ) T S 1 ( ) This matri S is called the covariance matri and is calculated from the data distribution 12

13 Take-awa on Mahalanobis Is good for nonsphericall smmetric distributions. Accounts for scaling of coordinate aes Can reduce to Euclidean 13

14 What is a metric? A metric has these four qualities. otherwise, call it a measure (triangle inequalit) ), ( ), ( ), ( (smmetr) ), ( ), ( (non - negative) 0 ), ( (refleivit) iff 0 ), ( z d z d d d d d d + = = =

15 Metric, or not? Driving distance with 1-wa streets Categorical Stuff : Is distance (Jazz to Blues to Rock) no less than distance (Jazz to Rock)?

16 Categorical Variables Consider feature vectors for genre & vocals: Genre: {Blues, Jazz, Rock, Hip Hop} Vocals: {vocals,no vocals} s1 = {rock, vocals} s2 = {jazz, no vocals} s3 = { rock, no vocals} Which two songs are more similar?

17 One Solution:Hamming distance Blues Jazz Rock Hip Hop Vocals s1 = {rock, vocals} s2 = {jazz, no_vocals} s3 = { rock, no_vocals} Hamming Distance = number of different bits in two binar vectors

18 Hamming Distance {0,1}) ( and,...,,,,...,, where ), ( > =< > =< = = i i n n n i i i, i d

19 Defining our own distance (an eample) How often does artist quote artist? Quote Frequenc Beethoven Beatles Liz Phair Beethoven Beatles Liz Phair? 1 2 Let s build a distance measure!

20 Defining our own distance (an eample) Beethoven Beatles Liz Phair Beethoven Beatles Liz Phair? 1 2 Quote frequenc Q Distance d(, ) = 1 f (, ) = value in table Q f Q z Artists (, ) f (, z)

21 Missing data What if, for some categor, on some eamples, there is no value given? Approaches: Discard all eamples missing the categor Fill in the blanks with the mean value Onl use a categor in the distance measure if both eamples give a value

22 Dealing with missing data ww ii = 1, if both iiand ii are defined 0,otherwise dd, = nn nn ww ww ii φφ( ii, ii ) ii ii=1 nn ii=1

23 Edit Distance Quer = string from finite alphabet Target = string from finite alphabet Cost of Edits = Distance Target: C A G E D - - Quer: C E A E D

24 Semantic Relatedness d(portland, Hippies) << d(portland, Monster trucks) 24

25 Semantic Relatedness Several measures have been proposed One that works well: Milne-Witten SR MW (, ) fraction of Wikipedia in-links to either or that link to both 25

26

27

28

29

30

31 One more distance measure Kullback Leibler divergence Related to entrop & information gain not a metric, since it is not smmetric Take EECS 428:Information Theor to find out more 31