EE 6882 Visual Search Engine

Transcription

1 /0/0 EE 688 Visual Search Engine Prof. Shih Fu Chang, Jan. 0, 0 Lecture # Visual Features: Global features and matching Evaluation metrics (Many slides from. Efors, W. Freeman, C. Kambhamettu, L. Xie, and likely others) (Slides preparation assisted by Rong Rong Ji) Course Format Lectures + two hands on homeworks (due /, /7) Mid term proect Review and implement topics of interest, students each team Proposal due /5, narrated slides due /6 Selected proects presented and discussed in class (/6 4/9) Final proect Extension of mid term proects encouraged, students each team Proposal due 4/, narrated slides due 4/0 Selected proects presented and discussed in class (4/0 5/7) Grading: Class participation (0%), homework (0%), mid term (0%), final (40%) Late policy: a total budget of 4 days for late submissions. No other delays accepted.

2 /0/ Image Features Why features are needed? Finding similar images in database Classifying images to categories Tracking obects in video Creating panorama Stereo matching > D Desired properties Compact (~ dimensions) Easy to compute (0 fps for video) Robust (invariant to photometric, geometric, content variations) photoguides.net merl.com Desired Properties of Visual Features Invariance: Rotation, scaling, cropping, shift, etc. illumination, pose, clutter, occlusion, viewpoint

3 /0/0 Invariant Local Features Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters Features Descriptors (Slide of. Efros) (review) Imaging Formation irradiance Lens CCD Sensor dditive Noise R G R G R G G G R G R G R G G G R G R G R Demosaicking Filter Camera Response Function DSP (White alance, Contrast Enhancement etc) Image intensity 6

4 /0/0 Color Spaces and Color Order Systems Color Spaces RG cube in Euclidean space R G r g b R G RG RG Standard representation used in color displays Drawbacks RG basis not related to human color udgments Intensity should be one of the dimensions of color Important perceptual components of color are hue, saturation, and brightness Perceptual color spaces: HIS, HSV Understanding HSI from RG Turn the RG cube so that lack White axis is vertical Each plane containing the W axis and a color point contains all the colors of the same hue Hue represented as angle between the plane and a reference plane (e.g. Red) Saturation: distance to axis, less saturated by mixing more grey colors Intensity measured by intersection with the W axis. Cross section shape: triangle hexagon triangle 8 Images from Gonzalez and Woods 4

5 /0/0 Colors on the HSI color cone Cross section approximated by triangle or circle HSI values computed by various geometrical models, e.g., I / V / 6 V / 6 / / 6 / 6 / R / 6 G 0 V H tan ( ) V Chroma ( V V / ) More suitable for measuring perceptual distance Can be quantized unevenly, e.g., Columbia VisualSEEK System: 6M colors (in RG) quantized to 66 HSV colors (8 Hue, Sat, Val, 4 Gray) 9 Manipulations in the HSI space HSI values of primary/secondary colors HSI allows independent manipulations of colors Hue of Green & lue set to 0. Saturation of Cyan reduced by half. Intensity of White reduced by half. 0 5

6 /0/0 Color Histogram Feature extraction from color images h RG Choose GOOD color space Quantize color space to reduce number of colors [, r g, b] Invariance? m n if IR[ m, n] r, IG[ m, n] g, I[ m, n] b 0 otherwise Scale, shift, rotation, crop, view angle, illumination, clutter, occlusion dvantages Easy to compute and compare Cons Lack spatial information, dimension may be high Color Moments Is there a more compact representation than color histogram? Compute moment statistics in each color channel.? 6

7 /0/0 Localizing Color Layout Search Columbia VisualSEEk (Smith & Chang, 96) IM QIC (Flickner et al 95) Query results 7

8 /0/0 Color correlogram 8

9 /0/0 Color Coherence Vector (CCV) (Pass et al, 997) Region segmentation C C C D E E E E E Not ust count of colors, also check adacency regions color size 5 C D E 5 Coherent! Size threshold: CCV color 5 0 5,,...,,,,...,, G G I n n I n n n = = G i i i i H i i i i i i G H by triangular inequality n 9

10 /0/0 Distance Metrics between Feature Vectors Lp distance Quadratic distance D q ( x D p ( x ( x i ) i x ( i ) x T C ( x p / p ( i) x( i) ) x ( i ) C ( i, ) x ) ( ) x ( ) ) C(i,): color distance Histogram Intersection Mohalanobis distance Normalize distance in the maor/minor axes where C x is the covariance matrix Mohalanobis Metric T mah x D x x C x x covariance matrix C N k x c(,) c(, )... c(, d) d: dimension of features cd (,) cd (,)... cdd (, ) c(, i ) x k() i m() i x k( ) m( ) / N, N : number of samples x x x x x o o o o o oo o o o o o o o o o o oo o o o x i x i x i x i x c s i is c s i s c 0 c s i s c s is s i, s : std. deviation 0

11 /0/0 Mohalanobis Metric where C x is the covariance matrix T Cx e e... ed diag(,,..., d) e e... ed Cx e e... ed ( diag(,,..., d)) e e... ed T e e Normalize distance in the eigen vector axes Proect data to the eigen vectors, divide with the sd of each eigen dimension, and compute Euclidian distance Mohalanobis Metric (cont.) dvantages of Mahalanobis metric ccount for scaling of coordinate axes Invariant under linear transformation T If y x Cy Cx, Dy Dx Correct for correlation Produce curved as well as linear decision boundaries cm km x i m c m c c c Maha. Dist. Maha. Dist. Minimum Selector Selected class Potential issue Need enough training data to estimate Cov. Matrix

12 /0/0 Earth Mover s Distance (EMD) Rubner, Tomasi, Guibas 98 Mallow s distance in statistics in 950 s Transportation Problem [Dantzig 5] I: set of suppliers J: set of consumers c i : cost of shipping a unit of supply from i to I c i J Problem: find the optimal flows f i minimize c f s.t. ii ii i i fi 0, i I, J (No reverse shipping) f y, J (satisfy each consumer need /cacacity) ii J J i f x, i I (bounded by each supplier's limit) i i y x ( feasibility) ii i EMD of Color Histogram h h, h,..., h M, g= g, g,..., h N, assume g( ) h( i) EMD h, g M N Ci fi i M N fi i M N N i i / i Earth Hole = C f g Fill up each hole C i : distance between color i in color space h and color in color space g f : move f units of mass from color i in h to color in g i i Normalization by the denominator term void bias toward low mass distributions (i.e., small images) what s the difference if both h and g are normalized first? i

13 /0/0 Evaluation Ground truth " Relevant" V n Detection False larms Misses 0 " Irrelevant " Returned Correct Dismissals Relevant Results D C C D n 0 N - ( ( K V n0 n K n0 ( V ) N n0 n N n0 n V ) ( V )) n search D N Images K Returned Results Recall R /( C ) Precision P /( ) Fallout F /( D) P R Combined F ( P R) / Evaluation Measures Precision at depth K k Pk ( n0vn ) / K Precision Recall Curve P ( P vs R ) Receiver Operating Characteristic (ROC Curve) (hit) vs (false) R

14 /0/0 Evaluation Metric: verage Precision P approximates areas under PR curve K P [ P I ( D min( K, R ) Example: R : total # of Ground truth Precision relevant Ranked list of data in response to a query D D D D D 5 / Precision / data, 6 / /4 is correct I : indicator 0 /5 0 /6 0 s /7 Pi P )] function Evaluation Metric: verage Precision Observations (P) P depends on the rankings of relevant data and the size of the relevant data set. E.g., R=0 Case I: Pre: P= Case II: - + Pre: / Case II: Pre: / / / / / / / / / / / 0/0 P=/ P~0. 4

15 /0/0 Homework # Given a small image database and a few queries Implement codes to extract color histogram Implement codes to measure L image similarity Use image obect labels to measure precision/recall curves onus: dd new color features or similarity metrics to improve performance Design GUI for result browsing Reading List Rui, Y., T.S. Huang, and S. F. Chang, Image retrieval: current techniques, promising directions and open issues. Journal of Visual Communication and Image Representation, (4): p Smith, J.R. and S. F. Chang. VisualSEEk: a Fully utomated Content ased Image Query System. in CM International Conference on Multimedia oston, M. David G. Lowe, Distinctive Image Features from Scale Invariant Keypoints, International Journal of Computer Vision, 60(), 004, pp9 0. Randen, T. and J. Husoy, Filtering for texture classification: comparative study. Pattern nalysis and Machine Intelligence, IEEE Transactions on, 00. (4): p Mikolaczyk, K. and C. Schmid, performance evaluation of local descriptors. IEEE Transactions on Pattern nalysis and Machine Intelligence, 005: p rown, M., R. Szeliski, and S. Winder. Multi image matching using multi scale oriented patches. in IEEE CVPR