Visual matching: distance measures

Transcription

1 Visual matching: distance measures Metric and non-metric distances: what distance to use It is generally assumed that visual data may be thought of as vectors (e.g. histograms) that can be compared for similarity using the Euclidean distance or more generally metric distances: Given a S set of paaerns a distance d: S S R is metric if sabsfying: Self idenbty: x S, d(x,x) = 0 PosiBvity: x y S, d(x,y) > 0 Symmetry: x,y S, d(x,y) = d(y,x) Triangle inequality: x,y,z S, d(x,z) d(x,y) + d(y,z) However, this may not be a valid assumpbon. A number of approaches in computer vision compare images using measures of similarity that are not Euclidean nor even metric, in that they do not obey the triangle inequality or simmetry. 1

2 Most notable cases where non metric distances are suited are: RecogniBon systems that aaempt to faithfully reflect human judgments of similarity. Much research in psychology suggests that human similarity judgments are not metric and distances are not symmetric. Matching of subsets of the images while ignoring the most dissimilar parts. In this case non metric distances are less affected by extreme differences than the Euclidean distance, and more robust to outliers. Distance funcbons that are robust to outliers or to extremely noisy data will typically violate the triangle inequality. Comparison between data that are output of a complex algorithm, like image comparisons using deformable template matching scheme, has no obvious way of ensuring that the triangle inequality holds. Histogram based representabons Feature vectors are o\en in the form of histograms that collect the distribubon of salient features. Several distances can be defined between histograms. Choice is dependent on the goals of matching and type of histogram. If human perceptual similarity is accounted, non metric distances are preferred. Grey level, color histograms are the most frequently used Color histogram 2

3 Human judgements of similarity Symmetry does not always hold for human percepbon: d(a,b) < d(b,a) A B Part based representabons ParBal similarity is a non metric relabon: d(a,b) + d(b,c) < d(a,c) B A Am I human? Yes, I m parbally human. C I am a centaur. Am I equine? Yes, I m parbally equine. 3

4 Similarity from complex algorithms Shape deformabon similarity is non metric: similarity can be assessed by minimizing the energy of deformabon E spent while maximizing matching M between edges Metric distances Feature maching where image data are represented by vector data are well suited to work with metric distances. Many metric distance measures are possible. Among them: HeurisBc Minkowski form Geometric Cosine distance Working with distribubons (histograms) Euclidean L1 Hamming Weighted Mean Variance (WMV) 4

5 Minkowski distance L p metrics also called Minkowski distance defined for two feature vectors A = (x 1,,x n ) and B = (y 1,,y n ) : ( )1 n p ( ) i=1 d p ( A, B) = x i y i p L 1 : City Block or Manha:an d 1 = A-B = L 2 : Euclidean distance n ( ) d 2 = x i y i i=1 n x i y i i=1 2 L : max, Chess board distance d = max i x i y i The L 1 norm and the L 2 norm are mostly used because of their low computabonal cost. A circle is a set of points at a fixed distance from the center equal to a radius r. In ManhaAan geometry, distance is determined differently than in Euclidean geometry and the shape of circles changes into squares with sides at 45 to the coordinate axes. 5

6 Color histograms: L1 and Euclidean distance If comparing two color histograms with ManhaAan or Euclidean distance take care of: the L 1 and Euclidean distances result in many false negabves because neighboring bins are not considered Euclidean distance is only suited for Lab and Luv color spaces. L 1 and Euclidean distance d(h 1,H 2 ) Color histograms: Hamming distance Hamming distance hypothesizes that histograms are binary vectors. Can detect absence/presence of colors: n n d h (I q,i d ) = H j (I q ),H j (I d ) = (H j (I q ) XOR H j (I d )) j=1 j=1 Hamming distance d(h 1,H 2 )

7 Cosine Distance Cosine distance derives from the definibon of dot product between two vectors. Geometrically, dot product means that a and b are drawn with a common start point and then the length of a is mulbplied with the length of that component of b that points in the same direcbon as a. Cosine distance measures how much a is not aligned with b m x i y i d(a,b) = 1 cos(a,b) = 1 A B = i=1 A B m m Π( x 2 i y 2 i ) i = 1 i = 1 ProperBes: Metric F 1 = 2x 1 + 3x 2 + 5x 3 Only angle is relevant, not vector lengths x 3 Example: F 1 = 2x 1 + 3x 2 + 5x 3 F 2 = 3x 1 + 7x 2 + x 3 Q = 0x 1 + 0x 2 + 2x 3 Q is closer to F 1 than F 2 F 2 = 3x 1 + 7x 2 + x 3 x 2 Q = 0x 1 + 0x 2 + 2x 3 x 1 Weighted Mean Variance Weighted Mean Variance (WMV) distance includes some minimal informabon about the data distribubon: D r (I,J) = µ r (I) µ r (J) σ (µ r ) + σ r (I) σ r (J) σ (σ r ) WMV is parbcularly quick because the calculabon is quick and the values can be precomputed offline. 7

8 Non metric distances With vector data : HeurisBc Minkowski form p<1 Mahalanobis Working with distribubons (histograms) Nonparametric test stabsbcs Kolmogorov Smirnov (KS) Cramer/Von Mises (CvM) χ 2 (Chi Square) Ground distance measures Histogram intersecbon QuadraBc form (QF) Earth Movers Distance (EMD) InformaBon theory divergences Kullback Liebler (KL) Jeffrey divergence (JD) Effects of variance and covariance on Euclidean distance B A The ellipse shows the 50% contour of a hypothebcal populabon. Euclidean distance is not suited to account for differences in variance between the variables and to account for correlabons between variables. Points A and B have similar Euclidean distances from the mean, but point B is more different from the populabon than point A. This is parbcularly cribcal for effects connected to human percepbon in low level feature image matching. In this case the Malahanobis distance should be used. 8

9 Mahalanobis (QuadraBc) Distance QuadraBc Form distance accounts for correlabon between features : d 2 ( A, B) = m i=1 m x i y i w ij x j y j = A B T W A B j=1 where W is the covariance matrix and diagonal terms are variance in each dimension and off diagonal terms indicate the dependency between variables. ProperBes: Metric only if w ij = w ji and w ii =1 Non metric otherwise Geometric interpretabon of metric distances 9

10 Color histograms: Mahanalobis distance Mahanalobis distance is used for color histogram similarity as it closely resembles human percepbon: d h (A,B) = (H(I q ) H(I d )) T A (H(I q ) H(I d )) [ ] A = a ij being the similarity matrix denobng similarity between bins i and j of N feature vectors x 1,,x N, each of length n. The quadrabc form distance in image retrieval results in false posibves because it tends to overesbmate the mutual similarity of color distribubons without a pronounced mode: the same mass in a given bin of the first histogram is simultaneously made to correspond to masses contained in different bins of the other histogram Histogram intersecbon Histogram intersecbon helps to check occurrence of object in region H obj [j] < H reg [j]. Histogram intersecbon is not symmetric: n j=1 d h (I q,i d ) = 1 min(h j (I q ),H j (I d )) n j=1 H j (I d ) Histogram intersecmon is widely used because of its ability to handle parbal matches when the areas of the two histograms are different. 10

11 CumulaBve Difference distances Kolmogorov Smirnov distance (KS) D r (I,J) = max F r (i;i) F r (i;j) Cramer/von Mises distance (CvM) D r (I,J) = (F r (i;i) F r (i;j)) 2 i where F r (I;.) is the marginal histogram distribubon Both Kolmogorov Smirnov and Cramer/von Mises distance are stabsbcal measures that measure the underlying similarity of two unbinned distribubons. Work only for 1D data or cumulabve histograms. They are non symmetric distance funcbons. CumulaBve Histogram CumulaBve Histogram describes the probability that a random variable X with a certain pdf will be found at a value less than or equal to x. Normal Histogram CumulaBve Histogram 11

12 CumulaBve Difference Example Histogram 1 Histogram 2 Difference - = K-S = D r (I, J ) = max F r (i; I) F r (i; J ) CvM = D r (I,J) = (F r (i;i) F r (i;j)) 2 i χ 2 distance χ 2 distance measures the underlying similarity of two samples where differences are emphasized: ( ) = D I, J ( f( i;i ) ˆ f ( i) ) 2, ˆ f i ˆ f ( i) i ( ) = [ f( i;i ) + f( i; J) ] / 2 is the expected frequency χ 2 distance measures how unlikely it is that one distribubon was drawn from the populabon represented by the other. The major drawback of these measures is that it accounts only for the correspondence between bins with the same index and do not uses informabon across bins 12

13 Earth Mover s distance Earth Mover s distance (EMD) between two distribubons x and y represents the minimum work to morph one distribubon into the other. Informally the two distribubons represent different ways of amassing the same amount of material from a region D and the EMD is given by the amount of mass Bmes the distance by which it is moved. Region D f ij amount of mass from x i to y j d ij distance from x i to y j EMD = 0.23* * *316.3 = EMD opt = 0.23* * * *277 = Given feature vectors with their associated feature weights A = {(x i,w i )} and B = {(y j,u j )} and a funcbon f ij expressing the capability of flowing from x i to y j over a distance d ij : provided that: f ij 0, Σ j f ij w i, Σ i f ij u i, Σ ij f ij = min(w,u) min U f ij d ij i,j d h (A,B) = min (W, U) ProperBes Respects scaling Metric if d metric, and W = U If W U: No posibvity, surplus not taken into account, No triangle inequality It is the only measure that works on distribubons with a different number of bins. Widely used for color, edge, mobon vector histograms but has high computabonal cost. 13

14 Moving Earth with histograms Considering two histograms H 1 and H 2 as defined f.e. in a color space, pixels can be regarded as the unit of mass to be transported from one distribubon to the other. It has to be based on some metric of distance between individual features. Moving Earth with histograms 14

15 Moving Earth with histograms = CompuBng Moving Earth Distance (amount moved) = 15

16 CompuBng Moving Earth Distance (amount moved) * (distance moved) = With variable length representabons P (distance moved) * (amount moved) m clusters m i=1 n j=1 f ij d ij = work Q n clusters 16

17 Constraints 1. Move earth only from P to Q P m clusters P Q n clusters Q Constraints 2. Cannot send more earth than there is P m clusters P Q n clusters Q 17

18 Constraints 3. Q cannot receive more earth than it can hold P m clusters P Q n clusters Q Constraints 4. As much earth as possible must be moved P m clusters P Q n clusters Q 18

19 Kullback Leibler distance Kullback Leibler distance considers histograms as distribubons and measures their similarity by calculabng the relabve entropy. It measures the shared informabon between two variables.: i.e. the cost of encoding one distribubon as another. In other words it measures how well can one distribubon be coded using the other as a codebook Σ i H i [I q ] = Σ i H i [I d ] =1 H i [I q ], H i [I d ] 0 n d h (I q,i d ) = H i (I q )log H i(i q ) H i (I d ) i=1 The Kullback Leibler divergence is not symmetric. It can be used to determine how far away a probability distribubon P is from another distribubon Q i.e. as a distance measure between two documents. The Kullback Leibler divergence does not necessarily match perceptual similarity well and is sensibve to histogram binning. Jeffrey divergence Jeffrey divergence n d h (I q, I d ) = H i (I q )log H (I ) i q i=1 H i (I d ) + H (I )log H (I ) i q i q H i (I d ) The divergence is an empirical modificabon of the KL divergence that is numerically stable, symmetric and robust with respect to noise and the size of histogram bins 19

20 Distance properbes summary /- /- /- /- by Kein FolienBtel L p WMV χ 2 KS CvM KL JD QF EMD Minkowski form Weighted Mean Variance Chi Square Kolmogorov Smirnov Cramer/von Mises Kullback Liebler Jeffrey divergence QuadraBc form Earth Movers Distance Examples using Color CIE Lab L1 distance Using Color CIE Lab) Jeffrey divergence χ 2 stabsbcs QuadraBc form distance Earth Mover Distance 20

21 Image Lookup Merging SimilariBes In the case in which several features are considered, distances computed between each feature vector can be merged together to evauate the full similarity. CombinaBon of distances can be performed according to different policies: Linear weighbng: combine k different feature distances d i, e.g. color, texture and shape distances: Linear weighbng (weighted average) Non linear weighbng: α trimmed mean: weight only α percent highest of the k values 21

22 Distances for symbolic representabons In some cases features are represented as strings of symbols. This is the case of spabal relabons, temporal features, semanbc content.. In these cases edit distances can be used that compute the number of changes required to transform one string into the other: Edit distance operabons that are considered are: InserMon, where an extra character is inserted into the string DeleMon, where a character has been removed from the string TransposiMon, in which two characters are reversed in their sequence SubsMtuMon, which is an inserbon followed by a delebon Hamming and Levenshtein distances The Hamming distance (seen for histograms) is suited to compute edit distances between binary vectors. the Needleman Wunch distance (specializabon of Levenshtein edit distance) between components of the feature vectors: A B N W Distance: 6 = (4 2) + (4 2) + (4 2) 22