COS 429: COMPUTER VISON Face Recognition

Transcription

1 COS 429: COMPUTER VISON Face Recognition Intro to recognition PCA and Eigenfaces LDA and Fisherfaces Face detection: Viola & Jones (Optional) generic object models for faces: the Constellation Model Reading: Turk & Pentland,???

2 Face Recognition Digital photography

3 Face Recognition Digital photography Surveillance

4 Face Recognition Digital photography Surveillance Album organization

5 Face Recognition Digital photography Surveillance Album organization Person tracking/id.

6 Face Recognition Digital photography Surveillance Album organization Person tracking/id. Emotions and expressions

7 Face Recognition Digital photography Surveillance Album organization Person tracking/id. Emotions and expressions Security/warfare Tele-conferencing Etc.

8 What s recognition? vs. Identification or Discrimination

9 What s recognition? vs. Identification or Discrimination Categorization or Classification

10 What s recognition? Yes, there are faces No localization Identification or Discrimination Categorization or Classification

11 What s recognition? Yes, there is John Lennon No localization Identification or Discrimination Categorization or Classification

12 What s recognition? Detection or Localizatoin John Lennon No localization Identification or Discrimination Categorization or Classification

13 What s recognition? No localization Detection or Localizatoin Identification or Discrimination Categorization or Classification

14 What s recognition? No localization Detection or Localizatoin Identification or Discrimination Categorization or Classification

15 Today s agenda Detection or Localizatoin 3. AdaBoost 4. Constellation model No localization 1. PCA & Eigenfaces 2. LDA & Fisherfaces Identification or Discrimination Categorization or Classification

16

17 Eigenfaces and Fishfaces Introduction Techniques Principle Component Analysis (PCA) Linear Discriminant Analysis (LDA) Experiments

18 The Space of Faces = + An image is a point in a high dimensional space An N x M image is a point in R NM We can define vectors in this space as we did in the 2D case [Thanks to Chuck Dyer, Steve Seitz, Nishino]

19 Key Idea χ = xˆ { P RL Images in the possible set are highly correlated. So, compress them to a low-dimensional subspace that captures key appearance characteristics of the visual DOFs. } EIGENFACES: [Turk and Pentland] USE PCA!

20 Two simple but useful techniques For example, a generative graphical model: P(identity,image) = P(identiy image) P(image) Preprocessing model (can be performed by PCA)

21 Principal Component Analysis (PCA) PCA is used to determine the most representing features among data points. It computes the p-dimensional subspace such that the projection of the data points onto the subspace has the largest variance among all p-dimensional subspaces.

22 Illustration of PCA x2 4 x X1 x1 x1 One projection PCA projection

23 Illustration of PCA x 2 2rd principal component x 1 1 st principal component

24 Eigenface for Face Recognition PCA has been used for face image representation/compression, face recognition and many others. Compare two faces by projecting the images into the subspace and measuring the EUCLIDEAN distance between them.

25 Mathematical Formulation Find a transformation, W, m-dimensional Orthonormal n-dimensional Total scatter matrix: W opt corresponds to m eigenvectors of S T

26 Eigenfaces PCA extracts the eigenvectors of A Gives a set of vectors v 1, v 2, v 3,... Each one of these vectors is a direction in face space what do these look like?

27 Projecting onto the Eigenfaces The eigenfaces v 1,..., v K span the space of faces A face is converted to eigenface coordinates by

28 Algorithm Training 1. Align training images x 1, x 2,, x N Note that each image is formulated into a long vector! 2. Compute average face u = 1/N Σ x i 3. Compute the difference image φ i = x i u

29 Algorithm 4. Compute the covariance matrix (total scatter matrix) S T = 1/NΣφ i φ it = BB T, B=[φ 1, φ 2 φ N ] 5. Compute the eigenvectors of the covariance matrix, W Testing 1. Projection in Eigenface Projection ω i = W (X u), W = {eigenfaces} 2. Compare projections

30 Illustration of Eigenfaces The visualization of eigenvectors: These are the first 4 eigenvectors from a training set of 400 images (ORL Face Database). They look like faces, hence called Eigenface.

31 Eigenfaces look somewhat like generic faces.

32 Eigenvalues

33 Reconstruction and Errors P = 4 P = 200 P = 400 Only selecting the top P eigenfaces reduces the dimensionality. Fewer eigenfaces result in more information loss, and hence less discrimination between faces.

34 Summary for PCA and Eigenface Non-iterative, globally optimal solution PCA projection is optimal for reconstruction from a low dimensional basis, but may NOT be optimal for discrimination

35 Linear Discriminant Analysis (LDA) Using Linear Discriminant Analysis (LDA) or Fisher s Linear Discriminant (FLD) Eigenfaces attempt to maximise the scatter of the training images in face space, while Fisherfaces attempt to maximise the between class scatter, while minimising the within class scatter.

36 Illustration of the Projection Using two classes as example: x2 x2 x1 x1 Poor Projection Good Projection

37 Comparing with PCA

38 Variables N Sample images: c classes: Average of each class: Total average: { x,, } μ 1 L x N { χ,, } i = μ = 1 L 1 N 1 i χ c x k x k χ i N x k N k= 1

39 Scatters Scatter of class i: ( )( ) T i k x i k i x x S i k μ μ χ = = = c i S W S i 1 ( )( ) = = c i T i i i S B 1 μ μ μ μ χ B W T S S S + = Within class scatter: Between class scatter: Total scatter:

40 Illustration x2 S W = S 1 + S 2 S 1 S B S 2 x1

41 Mathematical Formulation (1) After projection: y = W k T x k Between class scatter (of y s): Within class scatter (of y s): ~ S ~ S B = W = W W T T S S B W W W

42 Mathematical Formulation (2) The desired projection: W opt ~ SB = arg max ~ = W S W arg max W W W T T S S B W W W How is it found? Generalized Eigenvectors S w = λ S w B i i W i,, i = 1 K Data dimension is much larger than the number of samples n >> N The matrix is singular: Rank( SW ) N c S W m

43 Fisherface (PCA+FLD) Project with PCA to N c space z = W k T pca x k W pca = arg maxw W T S T W Project with FLD to c 1 space y = W k T fld z k W fld = arg max W W W T T W W T pca T pca S S B W W W pca pca W W

44 Illustration of FisherFace Fisherface

45 Results: Eigenface vs. Fisherface (1) Input: Train: Test: 160 images of 16 people 159 images 1 image Variation in Facial Expression, Eyewear, and Lighting With glasses Without glasses 3 Lighting conditions 5 expressions

46 Eigenface vs. Fisherface (2)

47 discussion Removing the first three principal components results in better performance under variable lighting conditions The Firsherface methods had error rates lower than the Eigenface method for the small datasets tested.

48 Today s agenda Detection or Localizatoin 3. AdaBoost 4. Constellation model No localization 1. PCA & Eigenfaces 2. LDA & Fisherfaces Identification or Discrimination Categorization or Classification

49 Robust Face Detection Using AdaBoost Brief intro on (Ada)Boosting Viola & Jones, 2001 Weak detectors: Haar wavelets Integral image Cascade Exp. & Res. Reference: P. Viola and M. Jones (2001) Robust Real-time Object Detection, IJCV.

50 Discriminative methods Object detection and recognition is formulated as a classification problem. The image is partitioned into a set of overlapping windows and a decision is taken at each window about if it contains a target object or not. Where are the screens? Background Decision boundary Bag of image patches Computer screen In some feature space

51 A simple object detector with Boosting Download Toolbox for manipulating dataset Code and dataset Matlab code Gentle boosting Object detector using a part based model Dataset with cars and computer monitors

52 Why boosting? A simple algorithm for learning robust classifiers Freund & Shapire, 1995 Friedman, Hastie, Tibshhirani, 1998 Provides efficient algorithm for sparse visual feature selection Tieu & Viola, 2000 Viola & Jones, 2003 Easy to implement, not requires external optimization tools.

53 Boosting Defines a classifier using an additive model: Strong classifier Features vector Weight Weak classifier

54 Boosting Defines a classifier using an additive model: Strong classifier Features vector Weight Weak classifier We need to define a family of weak classifiers from a family of weak classifiers

55 Boosting It is a sequential procedure: x t=1 x t=2 x t Each data point has a class label: y t = +1 ( ) -1 ( ) and a weight: w t =1

56 Toy example Weak learners from the family of lines Each data point has a class label: y t = +1 ( ) -1 ( ) and a weight: w t =1 h => p(error) = 0.5 it is at chance

57 Toy example Each data point has a class label: y t = +1 ( ) -1 ( ) and a weight: w t =1 This one seems to be the best This is a weak classifier : It performs slightly better than chance.

58 Toy example Each data point has a class label: +1 ( ) y t = -1 ( ) We update the weights: w t w t exp{-y t H t } We set a new problem for which the previous weak classifier performs at chance again

59 Toy example Each data point has a class label: +1 ( ) y t = -1 ( ) We update the weights: w t w t exp{-y t H t } We set a new problem for which the previous weak classifier performs at chance again

60 Toy example Each data point has a class label: +1 ( ) y t = -1 ( ) We update the weights: w t w t exp{-y t H t } We set a new problem for which the previous weak classifier performs at chance again

61 Toy example Each data point has a class label: +1 ( ) y t = -1 ( ) We update the weights: w t w t exp{-y t H t } We set a new problem for which the previous weak classifier performs at chance again

62 Toy example f 1 f 2 f 4 f 3 The strong (non- linear) classifier is built as the combination of all the weak (linear) classifiers.

63 Real-time Face Detection Integral Image New image representation to compute the features very quickly AdaBoost Selecting a small number of important feature Cascade A method for combining classifiers Focussing attention on promising regions of the image Implemented on 700MHz Intel Pentium Ⅲ, face detection proceeds at 15f/s. Working only with a single grey scale image

64 Features Three kinds of rectangle features The sum of the pixels which lie within the white rectangles are subtracted from the sum of pixels in the gray rectangles

65 Integral Image ii( x, y) = i( x, y ) x x, y y The sum within D=4-(2+3)+1

66 Learning Classification Function (1) Selecting a small number of important features α 1 α ( x, y ) n n ( 1,1) x ( x2,1) ( x3,0) x4 (,0) ( 5,0) x 6 ( x,0) 2 Initialize weights w 1, i = 1 1, for yi 1, 0 respectively 2l 2m = # of face # of nonface

67 Learning Classification Function (2) 3 For t=1,.,t a. Normalize the weights w ti, w ti, n j = 1 w t, j 1 if f j ( x) >θ j hj ( x) = 0 otherwise b. For each feature, j ε = w h ( x ) y i j i j i i c. Choose the classifier, h t with the lowest error εt d. Update the weights 1 or ( 0) t+ 1, i = t, i t h x y w w β t i i εt βt = 1 ε t

68 Learning Classification Function (3) 4 The final strong classifier is T 1 1 α h ( x) h( x) 2 0 otherwise = t= 1 t t t= 1 t T α α = t log 1 β t The final hypothesis is a weighted linear combination of the T hypotheses where the weights are inversely proportional to the training errors α 1 α 2

69 Learning Results 200 features Detection rate: 95% - false positive 1/14,804 The first and second features selected by AdaBoost Requiring 0.7 seconds to scan 384x288 pixel image

70 The Attentional Cascade Reject many of the negative sub-windows Two-feature strong classifier Detect: 100% False positive:40%

71 A Cascaded Detector f, d f, d f, d T T T F F F F target

72 Detector Cascade Discussion The speed of the cascaded classifier is almost 10 times faster

73 Experimental Results (1) Training dataset Face training set: 4916 hand labeled faces Scaled and aligned to a base resolution of 24ⅹ24 Structure of the detector cascade 32layer, 4297 feature Training time for the entire 32 layer detector on the order of weeks a single 466 MHz AlphaStation XP T T T T T F F F Reject about 60% of non-faces Correctly detecting close to 100% of faces F F

74 Face Image Databases Databases for face recognition can be best utilized as training sets Each image consists of an individual on a uniform and uncluttered background Test Sets for face detection MIT, CMU (frontal, profile), Kodak

75 Training dataset: 4916 images

76 Experimental Results Test dataset MIT+CMU frontal face test set 130 images with 507 labeled frontal faces False detection AdaBoost Neural-net > not 50 significantly different 110 accuracy > but the cascade class. almost times faster MIT test set: 23 images with 149 faces Sung & poggio: detection rate 79.9% with 5 false positive AdaBoost: detection rate 77.8% with 5 false positives

77

78

79

80

81

82

83

84

85 Today s agenda Detection or Localizatoin 3. AdaBoost 4. Constellation model No localization 1. PCA & Eigenfaces 2. LDA & Fisherfaces Identification or Discrimination Categorization or Classification

86 Parts and Structure Literature Fischler & Elschlager 1973 Yuille 91 Brunelli & Poggio 93 Lades, v.d. Malsburg et al. 93 Cootes, Lanitis, Taylor et al. 95 Amit & Geman 95, 99 et al. Perona 95, 96, 98, 00, 03 Huttenlocher et al. 00 Agarwal & Roth 02 etc

87 Deformations A B C D

88 Background clutter

89 Frontal faces

90 Face images

91 3D Object recognition Multiple mixture components

92 3D Orientation Tuning 100 Orientation Tuning % Correct % Correct angle in degrees Frontal Profile