Lecture 13 Visual recognition

Transcription

1 Lecture 13 Visual recognition Announcements Silvio Savarese Lecture Feb-14

2 Lecture 13 Visual recognition Object classification bag of words models Discriminative methods Generative methods Object classification by PCA and FLD Silvio Savarese Lecture Feb-14

3 Challenges Variability due to: View point Illumination Occlusions Intra-class variability

4 Challenges: intra-class variation

5 Basic properties Representation How to represent an object category; which classification scheme? Learning How to learn the classifier, given training data Recognition How the classifier is to be used on novel data

6 definition of BoW Histogram of visual words (codewords) codewords dictionary

7 learning Representation recognition feature detection & representation codewords dictionary image representation category models (and/or) classifiers category decision

8 Classification Discriminative methods Nearest neighbors Linear classifier SVM Generative methods 20-Feb-14

9 SVM classification category models Model space Class 1 Class N w

10 SVM classification Query image Model space Winning class: pink w

11 Caltech 101

12 Caltech 101 BOW ~15%

13 Major drawback of BOW models Don t capture spatial information!

14 Spatial Pyramid Matching Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. S. Lazebnik, C. Schmid, and J. Ponce N i i h i h h h I )) ( ), ( min( ), ( ), ( 2 1 ), ( h h I h h I

15 Caltech 101

16 Caltech 101 Pyramid matching

17 Discriminative models Nearest neighbor Neural networks 10 6 examples Shakhnarovich, Viola, Darrell 2003 Berg, Berg, Malik LeCun, Bottou, Bengio, Haffner 1998 Rowley, Baluja, Kanade 1998 Support Vector Machines Latent SVM Structural SVM Boosting Guyon, Vapnik, Heisele, Serre, Poggio Random forests Felzenszwalb 00 Ramanan 03 Viola, Jones 2001, Torralba et al. 2004, Opelt et al. 2006, Courtesy of Vittorio Ferrari Slide credit: Kristen Grauman Slide adapted from Antonio Torralba

18 Lecture 13 Visual recognition Object classification bag of words models Discriminative methods Generative methods Object classification by PCA and FLD Silvio Savarese Lecture Feb-14

19 Image classification p( zebra image) vs. p( no zebra image) Bayes rule: p ( zebra image ) p ( no zebra image ) p ( image zebra ) p ( image no zebra ) p ( zebra ) p ( no zebra ) posterior ratio likelihood ratio prior ratio

20 p( zebra image) p( no zebra image) Discriminative methods Modeling the posterior ratio: Decision boundary Zebra Non-zebra

21 Generative methods p( zebra image) vs. p( no zebra image) Bayes rule: p ( zebra image ) p ( no zebra image ) p ( image zebra ) p ( image no zebra ) p ( zebra ) p ( no zebra ) posterior ratio likelihood ratio prior ratio

22 Generative models 1. Naïve Bayes classifier Csurka Bray, Dance & Fan, Hierarchical Bayesian text models (plsa and LDA) Background: Hoffman 2001, Blei, Ng & Jordan, 2004 Object categorization: Sivic et al. 2005, Sudderth et al Natural scene categorization: Fei-Fei et al. 2005

23 Some notations w: a collection of all N codewords in the image w = [w1,w2,,wn] c: category of the image

24 the Naïve Bayes model p(c w) ~ p(c)p(w c) p( c) p( w1,, w c) N Prior prob. of the object classes Image likelihood given the class c w N

25 the Naïve Bayes model p (c w) ~ p(c)p(w c) p( c) p( w1,, w c) Prior prob. of the object classes Image likelihood given the class p( c) N N n1 p( w n c) Likelihood of n th visual word given the class Assume that each feature (codewords) is conditionally independent given the class p(w,, 1 w N c) N i1 p(w c) i

26 the Naïve Bayes model p (c w) ~ p(c)p(w c) p( c) p( w1,, w c) Prior prob. of the object classes Image likelihood given the class p( c) N N n1 p( w n c) Likelihood of n th visual word given the class Example: p( wi c1 ) 2 classes: bananas vs oranges Histogram of colors Wi = number of pixels colored in yellow in the image p( wi c2) x-axis: percentage of pixel that are colored in yellow in the image 25% 50% 75%

27 the Naïve Bayes model p (c w) ~ p(c)p(w c) p( c) p( w1,, w c) Prior prob. of the object classes Image likelihood given the class p( c) N N n1 p( w n c) Likelihood of n th visual word given the class How do we learn P(w i c j )? From empirical frequencies of code words in images from a given class p( wi c2) p( wi c1 )

28 Classification/Recognition c arg max c p( c w) p( c) N n1 p( w n c) Object class decision Example: p( wi c1 ) 2 classes: bananas vs oranges Query image contains a banana Look at how many pixels are yellow: say 60% Look at corresponding likelihood values given the two class hypotheses banana! p( wi c2) 60%

29 Summary: Generative models Naïve Bayes Unigram models in document analysis Assumes conditional independence of words given class Parameter estimation: frequency counting

30 Csurka s dataset 7 classes Csurka et al. 2004

31 E = 28% E = 15%

32 Generative vs discriminative Discriminative methods Computationally efficient & fast Generative models Convenient for weakly- or un-supervised, incremental training Prior information Flexibility in modeling parameters

33 Weakness of BoW the models All have equal probability for bag-of-words methods Location information is important No rigorous geometric information of the object components Segmentation and localization unclear

34 Lecture 13 Visual recognition Object classification bag of words models Discriminative methods Generative methods Object classification by PCA and FLD Silvio Savarese Lecture Feb-14

35 Object classification by Principle Component Analysis (PCA) Linear Discriminant Analysis (LDA) Originally introduced for faces: Eigenfaces and Fisherfaces Turk & Penland, 91 Belhumeur et al.,

36 The Space of images or histograms An image (or histogram) H is a point in a high dimensional space An N x M image is a point in R NM [Thanks to Chuck Dyer, Steve Seitz, Nishino]

37 Key Idea H in the possible set {xˆ} are highly correlated. So, compress them to a low-dimensional subspace that captures key appearance characteristics of the visual DOFs. USE PCA for estimating the sub-space (dimensionality reduction) Compare two objects by projecting the images into the subspace and measuring the EUCLIDEAN distance between them. EIGENFACES: [Turk and Pentland 91]

38 Image space Face space Computes n-dim subspace such that the projection of the data points onto the subspace has the largest variance among all n-dim subspaces. Maximize the scatter of the training images in face space

39 USE PCA for estimating the sub-space x2 4 x X1 x1 x1 PCA projection Computes n-dim subspace such that the projection of the data points onto the subspace has the largest variance among all n-dim subspaces.

40 USE PCA for estimating the sub-space x 2 2rd principal component x 1 1 st principal component

41 PCA Mathematical Formulation PCA = eigenvalue decomposition of a data covariance matrix Define a transformation, W, y j W T x j j 1, 2... N ~ S T m-dimensional Orthonormal n-dimensional N T T (x j x)(x j x) j1 N T T (y j y)(y j y) W ST W j1 Measure data scatter S = Data Scatter matrix = Transf. data scatter matrix Eigenvectors of S T [ v1 v 2 v m ]

42 Image space v 1 Face space v 2 v 3 v 4

43 Projecting onto the Eigenfaces The eigenfaces v 1,..., v K span the space of faces A face is converted to eigenface coordinates by

44 Algorithm Training 1. Align training images x 1, x 2,, x N Note that each image is formulated into a long vector! 2. Compute average face x = 1/N Σ x i 3. Compute the difference image x i x

45 Algorithm 4. Compute the covariance matrix (total scatter matrix) S N T T (x j x)(x j x) j1 5. Compute the eigenvectors of the covariance matrix S T 6. Compute training projections a1, a2... a N Testing 1. Take query image X 2. Project X into Eigenface space (W = {eigenfaces}) and compute projection ω i 3. Compare projection ω i with all training N projections a i

46 Illustration of Eigenfaces The visualization of eigenvectors: These are the first 4 eigenvectors from a training set of 400 images (ORL Face Database).

47 Eigenfaces look somewhat like generic faces.

48 Reconstruction and Errors P = 4 P = 200 P = 400 Only selecting the top P eigenfaces reduces the dimensionality. Fewer eigenfaces result in more information loss, and hence less discrimination between faces.

49 Summary for Eigenface Pros Non-iterative, globally optimal solution Limitations PCA projection is optimal for reconstruction from a low dimensional basis, but may NOT be optimal for discrimination

50 Extensions Generalized PCA: R. Vidal, Y. Ma, and S. Sastry. Generalized Principal Component Analysis (GPCA). IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 27, number 12, pages 1-15, Tensor Faces: "Multilinear Analysis of Image Ensembles: TensorFaces," M.A.O. Vasilescu, D. Terzopoulos, Proc. 7th European Conference on Computer Vision (ECCV'02), Copenhagen, Denmark, May, 2002 PCA-SIFT PCA-SIFT: A More Distinctive Representation for Local Image Descriptors - Y Ke, R Sukthankar - IEEE CVPR 04

51 Linear Discriminant Analysis (LDA) Fisher s Linear Discriminant (FLD) Eigenfaces exploit the max scatter of the training images in face space Fisherfaces attempt to maximise the between class scatter, while minimising the within class scatter.

52 Illustration of the Projection Using two classes as example: x2 x2 x1 x1 Poor Projection Good

53 Variables N Sample images: c classes: x,, 1 x N 1,, c Average of each class: Total average: i 1 N 1 i x k x k i N x k N k1

54 Scatters Scatter of class i: T i k x i k i x x S i k c i S W S i 1 c i T i i i S B 1 B W T S S S Within class scatter: Between class scatter: Total scatter:

55 Illustration x2 S W S 1 S 2 Within class scatter S 1 S 2 S B x1 Between class scatter S i x k x x T i k i k i c S W S i i1 S B c i1 i i i T

56 Mathematical Formulation (1) After projection: y W k T x k Between class scatter (of y s): Within class scatter (of y s): ~ S ~ S B W W W T T S S B W W W

57 Illustration 2 ~ S 1 ~ S S ~ B 2 1 ~ ~ ~ S S S W x1 x2 k T k x y W W S W S B T B ~ W S W S W T W ~ c i S W S i 1 c i T i i i S B 1

58 Mathematical Formulation The desired projection: W opt ~ SB arg max ~ W S W arg max W W W T T S S B W W W How is it found? Generalized Eigenvectors If S w has full rank, the generalized eigenvectors are eigenvectors of S W -1 S B with largest eigen-values SB wi i SW wi i 1,, m

59 Results: Eigenface vs. Fisherface Input: Train: Test: 160 images of 16 people 159 images 1 image Variation in Facial Expression, Eyewear, and Lighting With glasses Without glasses 3 Lighting conditions 5 expressions

60 Results: Eigenface vs. Fisherface Error rate %

61 Object detection Next lecture