Lec 12 Review of Part I: (Hand-crafted) Features and Classifiers in Image Classification

Transcription

1 Image Analysis & Retrieval Spring 2017: Image Analysis Lec 12 Review of Part I: (Hand-crafted) Features and Classifiers in Image Classification Zhu Li Dept of CSEE, UMKC Office: FH560E, Ph: x p.1

2 Outline HW-2 Quiz-1 Review part I: Image Features & Classifiers Image Formation: Homography Color Space & Color Features Filtering and Edge Features Harris Detector SIFT Aggregation: BoW, VLAD, Fisher, and Supervector Image Retrieval System Metrics: TPR/Precision, FPR, Recall, map Classification: knn, GMM Bayesian, SVM p.2

3 Homework-2: Aggregation Data Set: CDVS data set Global Codebook for VLAD, FisherVector Compute SIFT and DenseSIFT from each image, sort them into two arrays: hw2_sift: hw2_dsift Random sample a subset of them: use randperm() Compute PCA: o indx = randperm(n_sift); indx = indx(1:20*2^10); % sample 20k o [A0, s, eigv]=princomp(hw2_sift(indx,: )); Project SIFT o Dimension reduction: x = sift*a0(:,1:kd) Compute Kmeans and GMM models for VLAD and FV aggregation: o vl_feat: vl_kmeans(), vl_gmm(); p.3

4 VLAD- Vector of Locally Aggregated Descriptors Aggregate feature difference from the codebook Hard assignment by finding the NN of feature {x k } to {μ k } Compute aggregated differences 1 assign descriptors x 2 v k = j,s.t.nn x j =μ k x j μ k 2 compute x- i 5 L2 normalize v k = v k / v k 2 Final feature: k x d 3 v i =sum x- i for cell i v 1 v 2 v 3 v 4 v 5 p.4

5 Fisher Vector Aggregation FV encoding Gradient on the mean, for GMM component k, j=1..d In the end, we have 2K x D aggregation on the derivation w.r.t. the means and variances FV = u 1, u 2,, u K, v 1, v 2,, v K T Richer characterization of the code book with GMM: {μ k, Σ k, π k } Normalize by Fisher Kernel p.5

6 HW-2 Deliverables Functions: [vlad]=getvladsift(im, km, opt); % opt = sift, or dsft [fv]=getfvsift(im, gmm, opt); % can use vl_fisher() Plots: For each image in mp_fid, nmp_fid, we have {vlad, fv} x {kd=[16,24]} x{n=[24, 48, 96]}, total 12 features per image, plot the true positive distance and true negative distance histograms. Hint: hist(d0, 30); hist(d1, 30); TPR-FPR: p.6

7 Quiz-1 Covers what we reviewed today Format: True or False Multiple Choices Problem Solving Extra Credit (25%) Close Book, 1-page A4 cheating sheet allowed. cheating sheet need to be turned in with name and id number cheating sheet will also be graded for up to 10 pts Relax Quiz is actually on me. p.7

8 Best Cheating Sheet Award Cheating sheet is also graded. Up to 10 pts extra A nice way to summarize and organize/indexing your knowledge base. Cheating sheet + HW = what you really get from the class. p.8

9 Outline HW-2 Quiz-1 Review part I: Image Features & Classifiers Image Formation: Homography Color Space & Color Features Filtering and Edge Features Harris Detector SIFT Aggregation: BoW, VLAD, Fisher, and Supervector Image Retrieval System Metrics: TPR/Precision, FPR, Recall, map Classification: knn, GMM Bayesian, SVM p.9

10 Classification in Image Retrieval An image retrieval pipeline (hand crafted features) Image Formation Feature Computing Feature Aggregation Classification Homography, Color space Color histogram Filtering, Edge Detection HoG, Harris Detector, SIFT Bow VLAD Fisher Vector Supervector Knowledge/ Data Base p.10

11 Image Formation - Geometry p.11 X 0 x K I z y x f f v u w K Slide Credit: GaTech Computer Vision class, Intrinsic Assumptions Unit aspect ratio Optical center at (0,0) No skew of pixels Extrinsic Assumptions No rotation Camera at (0,0,0) X x

12 Image Formation: Homography In general, homography H maps 2d points according to, x =Hx Up to a scale, as [x, y, w]=[sx, sy, sw], so H has 8 DoF Affine Transform: 6 DoF: Contains a translation [t 1, t 2 ], and invertable affine matrix A [2x2] Similarity Transform, 4 DoF: a rigid transform that preserves distance if s=1: H = x y w = h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 H = x y w a 1 a 2 t 1 a 3 a 4 t s cos θ s sin(θ) t 1 s sin θ s cos(θ) t p.12

13 VL_FEAT Implementation VL_FEAT has an implementation: p.13

14 Image Formation Color Space RGB Color Space: 8 bit Red, Green, Blue Channels Mixed together YUV/YCbCr Color Space HSV Color Space Hue: color in polar coord Saturation Value p.14

15 Color Histogram Fixed Codebook Color Histogram: Fixed code book to generate histogram Example: MPEG 7 Scalable Color Descriptor HSV Color Space, 16x4x4 partition for bins Scalability thru quantization and Haar Transform p.15

16 Color Histogram Adaptive Codebook Adaptive Color Histogram No fixed code book Example: MPEG 7 Dominant Color Descriptor Distance Metric Not index-able p.16

17 Image Filters Basic Filter Operations 1/9 1/9 1 1/9 1 1/9 1 1/9 1 1/9 1 * 1/9 1 1/9 1 1/9 1 Region of Support p.17

18 Filter Properties Linear Filters are: Shift-Invariant: f(m-k, n-j)*h = g(m-k, n-j), if f*h=g Associative: f*h 1 *h 2 = f*(h 1 *h 2 ) this can save a lot of complexity Distributive: f*h1 + f*h2 = f*(h1+h2) useful in SIFT s DoG filtering. p.18

19 Edge Detection The need of smoothing Gaussian smoothing Combine differentiation and Gaussian derivative of Gaussian (x) f p.19

20 Image Gradient from DoG Filtering Image Gradient (at a scale): The gradient points in the direction of most rapid increase in intensity The edge strength is given by the gradient magnitude: The gradient direction is given by: p.20

21 HOG Histogram of Oriented Gradients Generate local histogram per block Size of HoG: n*m*4*9=36nm, for n x m cells. f h h9, h1,..., h9, h1,..., h9, h1,...,,..., h 4 9 Angle Magnitude HOG Cell size 5 Binary voting Magnitude voting p.21

22 Harris Detector Smoothed 2 nd moment: 1. Image derivatives (optionally blur first) 2 I x ( D) I, D) g( I ) I xi y ( D) ( 2 I xi y ( D) I y ( D) I x I y det M trace M Square of derivatives I x 2 I y 2 I x I y 3. Gaussian filter g( I ) g(i x2 ) g(i y2 ) g(i x I y ) 4. Harris Function detect corners, no need to compute the eigen values Rharris 2 det[ (, )] [trace( (, )) ] g I D ( I x ) g( I y ) [ g( I xi y)] [ g( I x ) g( I y I D )] 2 har p.22

23 Harris Detector Steps Response function -> Threshold -> Local Maxima Harris Detector NOT scale invariant R(x,y) p.23

24 Scale Space Theory - Lindeberg Scale Space Response via Laplacian of Gaussian The scale is controlled by σ 2 g 2 g 2 x 2 g 2 y g = e x+y 2 2σ Characteristic Scale: r image σ = 0.8r σ = 1.2r σ = 2r characteristic scale p.24

25 SIFT Detector via Difference of Gaussian However, LoG computing is expensive nonseperatable filter SIFT: uses Difference of Gaussian filters to approximate LoG Separate-able, much faster Box Filter Approximation you will see later. LoG Scale space construction By Gaussian Filtering, and Image Difference p.25

26 Peak Strength & Edge Removal Peak Strength: Interpolate true DoG response and pixel location by Taylor expansion Edge Removal: Re-do Harris type detection to remove edge on much reduced pixel set p.26

27 Scale Invariance thru Dominant Orientation Coding Voting for the dominant orientation Weighted by a Gaussian window to give more emphasis to the gradients closer to the center p.27

28 SIFT Description Rotate to dominant direction, divide Image Patch (16x16 pixels) around SIFT point into 4x4 Cells For each cell, compute an 8-bin edge histogram 0 2 angle histogram Spatial-scale space extrema Overall, we have 4x4x8=128 dimension p.28

29 SIFT Matching and Repeatability Prediction SIFT Distance True Love Test d(s 1 1, s 2 k ) d(s 1 1, s 2 k ) θ Not all SIFT are created equal Peak strength (DoG response at interpolated position) Combined scale/peak strength pmf p.29

30 Outline HW-2 Quiz-1 Review part I: Image Features & Classifiers Image Formation: Homography Color Space & Color Features Filtering and Edge Features Harris Detector SIFT Aggregation: BoW, VLAD, Fisher, and Supervector Image Retrieval System Metrics: TPR/Precision, FPR, Recall, map Classification: knn, GMM Bayesian, SVM p.30

31 Why Aggregation? Curse of Dimensionality + Decision Boundary / Indexing p.31

32 Histogram Aggregation: Bag-of-Words Codebook: Feature space: R d, k-means to get k centroids, {μ 1, μ 2,, μ k } BoW Hard Encoding: For n feature points,{x 1, x 2,,x n } assignment matrix: kxn, with column only 1-non zero entry Aggregated dimension: k k n p.32

33 Histogram Aggregation: Kernel Code Book Soft Encoding Kernel Code Book Soft Encoding Kernel Affinity: K x j, μ k = e k x j μ k 2 Assignment Matrix: A j,k = K(x j, μ k )/ k K(x j, μ k ) Encoding: k-dimensional: X(k)= 1 n j A j,k p.33

34 VLAD- Vector of Locally Aggregated Descriptors Aggregate feature difference from the codebook Hard assignment by finding the NN of feature {x k } to {μ k } Compute aggregated differences v k = j,s.t.nn x j =μ k x j μ k 1 assign descriptors x compute x- i v k = v k / v k 2 L2 normalize 3 v i =sum x- i for cell i Final feature: k x d v 1 v 2 v 3 v 4 v 5 p.34

35 Fisher Vector Aggregation FV encoding Gradient on the mean, for GMM component k, j=1..d In the end, we have 2K x D aggregation on the derivation w.r.t. the means and variances FV = u 1, u 2,, u K, v 1, v 2,, v K T Richer characterization of the code book with GMM: {μ k, Σ k, π k } Normalize by Fisher Kernel p.35

36 Supervector Aggregation Assuming we have UBM GMM model λ UBM = {P k, μ k, Σ k }, with identical prior and covariance Then for two utterance samples a and b, with GMM models λ a = {P k, μ k a, Σ k }, λ b = {P k, μ k b, Σ k }, The SV distance is, K λ a, λ b = k P k Σ k ( 1 2 ) μ k a It means the means of two models need to be normalized by the UBM covariance induced Mahanolibis distance metric This is also a linear kernel function scaled by the UBM covariances T ( P k Σ k ( 1 2 ) μ k b ) p.36

37 AKULA Adaptive KLUster Aggregation AKULA Descriptor: No fixed codebook! Outer loop: subspace optimization Inner loop: cluster centroids + SIFT count A 1 ={yc 1 1, yc 1 2,, yc 1 k ; pc 1 1, pc 1 2,, pc 1 k } A 2 ={yc 2 1, yc 2 2,, yc 2 k ; pc 2 1, pc 2 2,, pc 2 k } Distance metric: Min centroids distance, weighted by SIFT count (kind of manifolds tangent distance) d A 1, A 2 = 1 k k j=0 1 d min 1 j w min (j) + 1 k k i=0 2 d min 2 i w min (i) 1 d min 2 d min j = min i i = min j d j,i d j,i 1 w min 2 w min j = w j,i, i = w j,i, i = arg min i j = arg min j d j,i d j,i p.37

38 Image Retrieval Performance Metrics True Positive Rate False Positive Rate Precision Recall F-Measure TPR = TP/(TP+FN) FPR = FP/(FP+TN) Precision =TP/(TP + FP), Recall = TP/(TP + FN) F-measure = 2*(precision*recall)/(precision + recall) p.38

39 Retrieval Performance: Mean Average Precision map measures the retrieval performance across all queries p.39

40 map example MAP is computed across all query results as the average precision over the recall p.40

41 Compute TPR-FPR ROC Curve (Receiver Operating Characteristics) Confusion/distance matrix: d(j,k) Ground Truth: m(j,k) m j, k = 1, doc j and k are match 0, not match p.41

42 Classification in Image Retrieval A typical image retrieval pipeline Image Formation Feature Computing Feature Aggregation Classification Knowledge/ Data Base p.42

43 knn Classifier knn is Nonlinear Classifier Operations: majority vote Performance bounds on k: not worse off by 2 times the error rate of Bayesian error rate As k increases, it improves, but not much Accelerate NN retrieval operation Kd-tree: o a space partition scheme, to limit the number of data points comparison operation Kd-tree supported knn search: o Approx KNN search o Kd-Forest supported Approx Search p.43

44 Gaussian Generative Model Classifier Shaped by the relative shape of Covariance Matrix Matlab: [rec_label, err]=classify(q, x, y, method); Method = quadratic p.44

45 Why called SVM? The Support Vector Solution K(x i, x) p.45

46 Support Vector Machine Matlab: % train svm svm = svmtrain(x, y, 'showplot', true); % svm classification for k=1:5 figure(41); hold on; grid on; axis([ ]); [u,v]=ginput(1); q=[u,v]; plot(u, v, '*b'); rec = svmclassify(svm, q); fprintf('\n recognized as %c', rec); end p.46

47 Summary Image Analysis & Retrieval Part I Image Formation Color and Texture Features Interest Points Detection Harris Detector Invariant Interest Point Detection SIFT Aggregation Classification tools: knn, gmm, svm Performance metrics: o tp, fp, tn, fn, tpr/precision, fpr, recall, map Part II: Holistic approach Just throw the pixels to the algorithm to figure out what are the good features. Subspace approach: Eigenface, Fisherface, Laplacian Embedding Deep Learning p.47