Image Analysis PCA and Eigenfaces Christophoros Nikou cnikou@cs.uoi.gr Images taken from: D. Forsyth and J. Ponce. Computer Vision: A Modern Approach, Prentice Hall, 2003. Computer Vision course by Svetlana Lazebnik, University of North Carolina at Chapel Hill. Computer Vision course by Michael Black, Brown University. Research page of Antonio orralba, MI. University of Ioannina - Department of Computer Science 2 Face detection and recognition 1
3 Face detection and recognition Detection Recognition Sally 4 Consumer application: iphoto 2009 http://www.apple.com/ilife/iphoto/ 2
5 Consumer application: iphoto 2009 It can be trained to recognize pets! http://www.maclife.com/article/news/iphotos_faces_recognizes_cats 6 Consumer application: iphoto 2009 iphoto decides that this is a face 3
7 Outline Face recognition Eigenfaces Face detection he Viola and Jones algorithm. M. urk and A. Pentland. "Eigenfaces for recognition". Journal of Cognitive Neuroscience, 3(1):71 86, 1991. Also in CVPR 1991. P. Viola and M. Jones. Robust real-time face detection. International Journal of Computer Vision, 57(2), 2004. Also in CVPR 2001. 8 he space of all face images When viewed as vectors of pixel values, face images are extremely high-dimensional 100x100 image = 10,000 dimensions However, relatively few 10,000-dimensional vectors correspond to valid face images. Is there a compact and effective representation of the subspace of face images? 4
9 he space of all face images Insight into principal component analysis (PCA). We want to construct a low-dimensional linear subspace that best explains the variation in the set of face images. 10 Principal Component Analysis Given: N data points x 1,,x N in R d We want to find a new set of features that are linear combinations of the original ones: w(x i ) = u (x i µ) (µ: mean of data points) What unit vector u in R d captures the most variance of the data? 5
11 Principal Component Analysis (cont.) he variance of the projected data: 1 1 var ( ) ( ) ( ) ( ) ( ) N N N N ( w x i ) = w x i w x i = u x i μ ( u x i μ ) i= 1 i= 1 Projection of data point N N 1 1 = u ( x i μ )( x i μ ) u= u ( i )( i ) N i= 1 N x μ x μ u i= 1 = u Σu Covariance matrix of data 12 Principal Component Analysis (cont.) We now estimate vector u maximizing the variance: subject to: u Σu 2 uu= u = 1 because any multiple of u maximizes the objective function. he Lagrangian is J ( u; λ) = u Σu + λ (1 u u ) Σu= λu leading to the solution: which is an eigenvector of Σ. he one maximizing J corresponds to the largest eigenvalue of Σ. 6
13 Principal Component Analysis (cont.) he direction that captures the maximum covariance of the data is the eigenvector corresponding to the largest eigenvalue of the data covariance matrix. he top k orthogonal directions that capture the most variance of the data are the k eigenvectors corresponding to the k largest eigenvalues. 14 Eigenfaces: Key idea Assume that most face images lie on a low-dimensional subspace determined by the first k (k<d) directions of maximum variance. Use PCA to determine the vectors or eigenfaces u 1, u k that span that subspace. Represent all face images in the dataset as linear combinations of eigenfaces. 7
15 Eigenfaces example raining images 16 Eigenfaces example (cont.) op eigenvectors Mean: μ 8
17 Eigenfaces example (cont.) u k μ -3λ k u k μ + 3λ k u k 18 Eigenfaces example (cont.) Face x in face space coordinates: x u 1( x μ), u2( x μ),..., uk ( x μ) = [,,..., ] w1 w2 w k Reconstruction: = + ˆx = μ + w1u1+ w2u2 +... + w k u k 9
19 Eigenfaces example (cont.) Any face of the training set may be expressed as a linear combination of a number of eigenfaces. In matrix-vector form: [ ] x = μ + u 1 u2 u k w1 w 2 w k Error: K i k 1 i E k = = + K λ i = 1 i λ = + 20 Eigenfaces example (cont.) Intra-personal subspace (variations in expression) Extra-personal subspace (variations between people) 10
21 Face Recognition with eigenfaces Process the training images: Find mean µ and covariance matrix Σ. Find k principal components (eigenvectors of Σ) u 1, uu k. Project each training image x i onto subspace spanned by principal components: (w i1,,w ik ) = (u 1 (x i µ),, u k (x i µ)) Given a new image x: Project onto subspace: (w 1,,w k ) = (u 1 (x µ),, u k (x µ)). Optional: check the reconstruction error x x^ to determine whether the new image is really a face. Classify as closest training face in k-dimensional subspace. M. urk and A. Pentland, Face Recognition using Eigenfaces, CVPR 1991 22 Computation of eigenvectors Given the d-dimensional vectorized images x i (i=1,2,n), we form matrix X=[x 1, x 2,, x N ], d >>N. Covariance matrix: Σ= 1 XX N Σ has dimensions dxd which, for example, for a 200x200 image is very large (40000x40000). How do we compute its eigenvalues and eigenvectors? 11
23 Computation of eigenvectors (cont.) We create the NxN matrix: = N 1 X X has N eigevalues (γ i ) and eigenvectors (e i ): e = γ e i i i 1 X Xei iei N =γ Σ ( Xe ) =γ ( Xe ) i i i 1 XX Xei ixei N =γ γ i is an eigenvalue of Σ and Xe i is the correspoding eigenvector. 24 he importance of the mean image and the variations What is this? Average of 100 of the images from the Caltech-101 dataset. Reproduced from the web page of A. orralba http://web.mit.edu/torralba/www/gallery/gallery11.html 12
25 he importance of the mean image and the variations (cont.) he context plays an important role. Is it a car or a person? Both blobs correspond to the same shape after a 90 degrees rotation. Reproduced from the web page of A. orralba. http://web.mit.edu/torralba/www/gallery/gallery11.html 26 he importance of the mean image and the variations (cont.) Average fashion model. Average of 60 aligned face images of fashion models. Reproduced from the Perception Lab, University of St Andrews in Fife, UK. http://perception.st-and.ac.uk 13
27 Application: MRF features for texture representation Each pixel is described by its 7x7 neighborhood in every Lab channel. his results in a 7x7x3 = 147-dimensional vector per pixel. Applying PCA to these vectors leads to a compact 6- to 8-dimensional representation of natural images. It captures more than 90% of the variation of the image pixels neighborhoods. 28 Application: MRF features for texture representation (cont.) 14
29 Application: view-based modeling PCA on various views of a 3D object. H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995. 30 Application: view-based modeling (cont.) Subspace of the first three PC (varying view angle). H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995. 15
31 Application: view-based modeling (cont.) View angle recognition. H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995. 32 Application: view-based modeling (cont.) Extension to more 3D objects with varying pose and illumination. H. Murase and S. Nayar. Visual Learning and Recognition of 3D objects from appearance, International Journal of Computer Vision, Vol. 14, pp. 5-24,1995. 16
33 Application: Eigenheads Principal components 1 and 2. 34 Application: Eigenheads (cont.) Principal components 3 and 4. 17
35 Limitations Global appearance method not robust to misalignment, background variation. 36 Limitations (cont.) Projection may suppress important detail he smallest variance directions may not be unimportant. PCA assumes Gaussian data. he shape of this dataset is not well described by its principal components 18
37 Limitations (cont.) PCA does not take the discriminative task into account ypically, we wish to compute features that allow good discrimination. Projection onto the major axis can not separate the green from the red data. he second principal component captures what is required for classification. 38 Canonical Variates Also called Linear Discriminant Analysis A labeled training set is necessary. We wish to choose linear functions of the features that allow good discrimination. Assume class-conditional covariances are the same. We seek for linear feature maximizing the spread of class means for a fixed within-class variance. 19
39 Canonical Variates (cont.) We have a set of vectorized images x i (i=1,2,n)., he images belong to C categories each having a mean μ j, ( j=1,2,,c ). he mean of the class means: 1 C μ = μ j C j= 1 40 Canonical Variates (cont.) Between class covariance: C ( )( ) S = N μ μ μ μ B i i i i= 1 with N i being the number of images in class i. Within class covariance: C N i ( μ )( μ ) S = x x W k,i i k,i i i= 1 k= 1 20
41 Canonical Variates (cont.) Using the same argument as in PCA (and graphbased segmentation) we seek vector u maximizing which is equivalent to usu B max u usu W max{ usu} s.t. usu= 1 he solution is the top eigenvector of the generalized eigenvalue problem: Su B u W B = λs u W 42 Canonical Variates (cont.) 71 views of 10 objects at a variety of poses at a black background. 60 images were used to determine a set of canonical variates. 11 images were used for testing. 21
43 Canonical Variates (cont.) he first two canonical variates. he clusters are tight and well separated (a different symbol per object is used). We could probably get a quite good classification. 22