Bilinear Discriminant Analysis for Face Recognition

Transcription

1 Bilinear Discriminant Analysis for Face Recognition Muriel Visani 1, Christophe Garcia 1 and Jean-Michel Jolion 2 1 France Telecom Division R&D, TECH/IRIS 2 Laoratoire LIRIS, INSA Lyon 4, rue du Clos Courtel 20, Avenue Alert Einstein Cesson-Sevigne, France Villeuranne, cedex, France {muriel.visani,christophe.garcia}@rd.francetelecom.com Jean-Michel.Jolion@insa-lyon.fr Astract In this paper, a new statistical projection method called Bilinear Discriminant Analysis (BDA) is presented. The proposed method efficiently comines two complementary versions of Two-Dimensional-Oriented Linear Discriminant Analysis (2DoLDA), namely Column-Oriented Linear Discriminant Analysis () and Row-Oriented Linear Discriminant Analysis (), through an iterative algorithm using a generalized ilinear projectionased Fisher criterion. A series of experiments was performed on various international face image dataases in order to evaluate and compare the effectiveness of BDA to and. The experimental results indicate that BDA is more efficient than, and 2DPCA for the task of face recognition, while leading to a significant dimensionality reduction. 1 Introduction In the eigenfaces [1] (resp. fisherfaces [2]) method, the 2D face images of size h w are first transformed into 1D image vectors of size h w, and then a Principal Component Analysis (PCA) (resp. Linear Discriminant Analysis (LDA)) is applied to the high-dimensional image vector space, where statistical analysis is costly and may e unstale. To overcome these drawacks, Yang et al. [3] proposed the Two Dimensional PCA () method that aims at performing PCA using directly the face image matrices. It has een shown that is more efficient [3] and roust [4] than the eigenfaces when dealing with face segmentation inaccuracies, low-quality images and partial occlusions. In [5], we proposed the Two-Dimensional-Oriented Linear Discriminant Analysis (2DoLDA) approach, that consists in applying LDA on image matrices. We have shown on various face image dataases that 2DoLDA is more efficient than oth and the Fisherfaces method for the task of face recognition, and that it is more roust to variations in lighting conditions, facial expressions and head poses. In this paper, we propose a novel supervised projection method called Bilinear Discriminant Analysis (BDA) that achieves etter recognition results than 2DoLDA while sustantially reducing the computational cost of the recognition step, using a generalized Fisher criterion relying on ilinear projections. The remainder of the paper is organized as follows. In section 2, we remind the theory and algorithm of 2DoLDA. In section 3, we descrie in details the principle and algorithm of the proposed BDA method, pointing out its advantages over previous methods. In section 4, a series of three experiments, on different international data sets, is presented to demonstrate the effectiveness and roustness of BDA and compare its performances with respect to, and 2DPCA. Finally, conclusions are drawn in section 5. 2 Two-Dimensional Oriented Linear Discriminant Analysis (2Do LDA) In this section, we remind the 2DoLDA theory and algorithm. For more details refer to [5]. 2DoLDA may e implemented in two different ways: Row-oriented LDA () and Column-oriented LDA (). Let us first present. The model is constructed from a training set containing n face images stored as h w matrices X i laelled y their corresponding identity Ω c. The set of images corresponding to one person is called a class. The training set contains C classes. Let us consider a projection matrix P, of size w k. The projection Xi P of X i with P is given y: Xi P = X i P (1) Using, the signature of X i is Xi P, of size h k. Our aim is to determine the matrix P jointly maximizing separation etween signatures from different classes and minimizing separation etween signatures from the same class. Under the assumptions that the rows of the images are multinormal and that rows from different classes have the same within-class covariance, P maximizes the generalized Fisher criterion [5]: J(P ) = P T S P P T S wp (2) S w and S eing respectively the generalized within-class and etween-class covariance matrices of the training

2 set: C S w = (X i X c ) T (X i X c ) and S = X i Ω c C n c ( X c X) T ( X c X) (3) and X c and X are respectively the mean of the n c samples of class Ω c and the mean of the whole training set. If S w is non-singular (which is generally verified as w << n), the k columns of P, named Row-Oriented Discriminant Components (RoDCs) and maximizing criterion (2) are the eigenvectors of Sw 1 S with largest eigenvalues. A numerically stale way to compute them is given in [6]. Analogeously, for Column-Oriented Linear Discriminant Analysis (), the considered projection is: X Q i = Q T X i, (4) where Q is a projection matrix of size h k. Under the assumptions of multinormality and homoscedasticity of the columns, we can consider the following generalized Fisher criterion: J(Q) = QT Σ Q Q T (5) Σ w Q where Σ w and Σ are respectively the within-class and etween-class covariance matrices of the (X T i ) i {1...n}: Σ w = C X i Ω c (X i X c )(X i X c ) T and Σ = C n c ( X c X)( X c X) T. (6) If Σ w is non-singular, the k columns of Q, named Column-Oriented Dicriminant Components and maximizing criterion (5) are the k eigenvectors of Σ 1 w Σ with largest eigenvalues. There are at most C 1 RoDCs and CoDCs corresponding to non-zero eigenvalues; their numer k can e selected using the Wilks Lamda criteria, which is also known as the stepwise discriminant analysis [7]. This analysis shows that the numer k of 2Do-DCs required y oth methods is comparale, generally inferior to 15, even if the numer of classes is large, as shown in Figure 2(a), reporting an experiment performed on 107 classes. Recognition is performed in the projection space defined y (resp. ): two faces images X a and X are compared using the Euclidean distance etween their projections Xa P and X P (resp. XQ a and X Q ). 3 Bilinear Discriminant Analysis (BDA) 3.1 Why Comining and? We conducted four experiments highlighting the complementarity of and. In the following, all the face images are centered and cropped to a size of h w = pixels. The first two experiments are performed on susets of the Asian Face Image Dataase PF01 [8] containing 107 persons; they illustrate the fact that depending on the training and test data, can significantly outperform or the contrary. In the first experiment, the training set (see Figure 1(a)) contains 5 near-frontal views per person (535 images). The test set (see Figure 1()) contains 4 views per person, with stronger non-frontal poses than in the training set. (a) () (c) (d) Figure 1: Susets of the IML dataase used for experiments 1-2. (a): Extract of the training set used for the first experiment, (): extract of the test set used for the first experiment, (c): Extract of the training set used for the second experiment, (d): extract of the test set used for the second experiment. Figure 2(a) shows that oth and provide more than 92% recognition rate, and outperform 2D PCA. However, is more efficient than, with 4,5% improvement of the recognition rate etween the respective maxima. In the second experiment, the training set (see Figure 1(c)) containing 3 views per person and the test set (see Figure 1(d)) containing 2 views per person are randomly taken from the suset of the IML dataase containing facial expressions: neutral, happy, surprised, irritated and closed eyes. Figure 2() shows that oth and outperform, and that is more efficient than (with 5,6% improvement of the recognition rate etween the maxima) on that particular and complex -the recognition rates are inferior to 60%- suset of the IML dataase. The third and fourth experiments provide further comparison of the performances of and ; they are performed on the Yale Face Dataase [2] (see Figure 3(a-)) containing 11 views of each of 15 persons.

3 1 0,65 0,95 0,9 0,85 0, Numer of projection vectors 0,6 0,55 0,5 0,45 0, Numer of projection vectors (a) () Figure 2: Compared recognition rates of, and on a suset of the IML dataase showing (a) head pose changes () facial expression changes, when varying the numer k of projection vectors. These views present occlusions ( glasses, noglasses ) and dissimilarities in the lighting conditions and facial expressions. In the third experiment, the Yale face dataase is randomly partitioned into a training set containing four views per person, and a test set containing six views per person. To ensure homoscedasticity, the views of each set are consistent among the classes, e.g. all the wink views are included in the test set, and all the neutral in the training set. This operation is repeated several times. From each partition we compute a confusion matrix. The confusion matrices from five different random partitions, with k = C 1 = 14, are shown in Tale 1. In each confusion matrix, the top left cell contains the numer of faces correctly classified y oth and. The top right element is the numer of faces correctly classified y, ut misclassified y. The ottom left element is the numer of faces correctly classified y, ut misclassified y. The ottom right cell contains the numer of faces misclassified y oth and (a) () (c) (d) (e) Tale 1: Confusion matrices on random partitions of the Yale Face Dataase. Tale 1(a) shows that, on the first random partition of the Yale dataase, the performances of and are comparale (the recognition rates are respectively = 70% and 71,1%). However, classification results are very different: 21 samples (23,3% of the test set) are correctly classified y one method, ut not y the other, and 82,2% of the test faces are recognized y at least one of the two methods. Tale 1(-c) illustrates the fact that generally outperforms. Tale 1(d-e) shows that in some configurations where the rate of misclassification y oth methods is high (respectively = 17, 8% and 20% for partitions (d) and (e)), outperforms. The fourth experiment provides further qualitative analysis. The training set (see Figure 3(a)) contains four views for all the 15 sujects, with variations in the lighting conditions and facial expressions. Then, seven test sets are uilt (see Figure 3()), corresponding to the remaining views. Figure 3(c) compares the performances of and on these test sets. Figure 3(c) shows that, for a fixed training set containing few variations, is generally more efficient than, ut in some cases drastically outperforms, especially when the test set contains dissimetries of the image following the vertical axis ( leftlight and rightlight ). can also slightly outperform when the test set shows strong facial expression changes, e.g. surprised. Normal Centerlight Happy Glasses / Noglasses (a) Wink Surprised Leftlight Rightlight Sad Sleepy Occlusion 1 0,9 0,8 0,6 0,5 0,4 0,3 0,2 0,1 0 w ink surprised leftlight rightlight sad sleepy occlusion () (c) Figure 3: (a): Extract of the training set used for the fourth experiment (): extract of the 7 test sets of the fourth experiment. A suject not wearing eyeglasses in the training set wears eyeglasses in the occlusion set, and vice-versa. (c): Compared recognition rates of and on partitions of the Yale Dataase.

4 Choosing etween and therefore requires a preliminary qualitative analysis of the training and test sets, which is a difficult task. As oth and are very performant ut give different recognition results, efficiently comining them can lead to a highly performant method. In 2DoLDA, considering image matrices instead of vectors (as in the Fisherfaces method) when performing LDA leads to a reduced computational cost when uilding the model, and to a reduced storage cost [5]. But the size of the matrices Xa P and X P (resp. XQ a and X Q ) is h k for (resp. k w for ) and may e large. As exposed in the following section, using BDA for recognition leads to a drastic reduction in the signatures size, and therefore reduces the computational cost during the recognition step, which is often online. 3.2 Description of the method Let us consider two projection matrices Q R h k and P R w k. The projected sample X Q,P i of X i on (Q, P ) is given y: X Q,P i = Q T X i P (7) Let us define the signature of X i y its ilinear projection X Q,P i (of size k k) onto (Q, P ). We are searching for the optimal pair of matrices (Q, P ) maximizing separation etween signatures from different classes while minimizing separation etween signatures from the same class: Q,P C (Q, P S ) = Argmax (Q,P ) R h k R w k Sw Q,P = Argmax n c(xc Q,P X Q,P ) T (Xc Q,P X Q,P ) (Q,P ) R h k R w k (8) C i Ω c (X Q,P i Xc Q,P ) T (X Q,P i Xc Q,P ) Sw Q,P and S Q,P eing the within-class and etween-class covariance matrices of the set (X Q,P i ) i {1,...,n}. This ojective function is iquadratic and has no analytical solution. We therefore propose an iterative procedure that we call Bilinear Discriminant Analysis. Let us expand the expression (8): (Q, P ) = Argmax (Q,P ) R h k R w k [ Σ C n c(p T (X c X) T QQ T (X c X)P ) Σ C Σ i Ω c (P T (X i X c) T QQ T (X i X c)p ) For any fixed Q R h k, using equation (9), the ojective function (8) can e rewritten: [ ] P = Argmax P R w k h i P T Σ C n c(xc Q X Q ) T (Xc Q X Q ) P T h Σ C Σ i Ω c (XQ i XQ c ) T (X Q i XQ c ) P i P = Argmax P R w k ] P T S Q P P T S Q w P Sw Q and S Q eing respectively the generalized within-class covariance matrix and the generalized etween-class covariance matrix of the set (X Q i ) i {1...n} (from equation (4)). Therefore the columns of the matrix P are the 1 S Q k eigenvectors of Sw Q with largest eigenvalues, otained y applying on the set of the projected samples X Q i. Let us denote A = P T (X c X) T Q, matrix of size k k. Given that, for every square matrix A, A T A = AA T, the ojective function (8) can e rewritten: [ ] (Q, P Σ ) = Argmax C n c(q T (X c X)P P T (X c X) T Q) (11) (Q,P ) R h k R [Σ C Σ w k i Ω c (QT (X i X c )P P T (X i X c ) T Q) For a fixed P R w k, using equation (11) the ojective function (8) can e rewritten Q = Argmax Q R h k Σ P w and Σ P eing the generalized within-class and etween-class covariance matrices of the set ((XP i )T ) i {1...n}. (9) (10) Q T Σ P Q Q T Σ P w Q, with largest eigenvalues, otained y applying Therefore, the columns of Q are the k eigenvectors of (Σ P w) 1 Σ P on the set of the projected samples (Xi P ) i {1...n}. Recognition is performed in the projection space defined y BDA: two face image X a and X are compared using the Euclidean distance etween their ilinear projections X (Q,P ) a and X (Q,P ). We can note that the computational cost of one comparison is o(k 2 ) for BDA, versus o(h k) for and 2D PCA, and o(w k) for ; therefore BDA drastically reduces the computational cost of the recognition step. 3.3 Algorithm of the BDA approach Let us initialize Q 0 = I h, I h eing the identity matrix of R h h, and k 0 =C 1. The proposed algorithm for BDA is: 1. For i {1,..., n}, compute X Q t i = (Q t ) T X i. 2. Apply on (X Qt i 3. For i {1,..., n}, compute X P t 4. Apply on (X P t i ) i {1,...,n} : compute Sw Qt, S Qt i = X i P t. ) i {1,...,n} : compute Σ P t w, Σ P t and, from (S Qt w ) 1 S Qt, compute P t, of size w k t ;, and, from (ΣP t w ) 1 Σ P t, compute Q t, of size h k t ;

5 5. Compute χ 2 = (n h+c 2 1)ln( C 1 j=k t λ j ). 6. if χ 2 < p value [ χ 2 ((h k t )(C k t 1)) ], then t t+1, k t k t 1 1, and return to step else k t k t 1, Q Q t 1 and P P t 1 The stopping criterion (steps 6-7) derives from the Wilks Lamda criterion,testing the discriminatory power of the C 1 k t eigenvectors of (Σ P t w ) 1 Σ Pt removed at step 4, when keeping only the k t first columns of Q t. We consider the following test: H 0 : at least one of the eigenvectors k t +1,... C 1 is discriminative, and H 1 : non H 0. Under H 0, it can e easily shown that (n h+c 2 1)ln( C 1 1 j=k t+1 1+λ j ) corresponds to a χ 2 distriution, with (h k t )(C k t 1) degrees of freedom. The p-value can e chosen at confidence level of 5%. If χ 2 < p value, the C k t 1 last eigenvectors (with smallest eigenvalue) can e removed and the stepwise analysis goes on. If χ 2 > p value, the eigenvector k t + 1 = k t 1 is discriminative and should e kept. 4 Experimental Results Three experiments are performed on the Asian Face Image Dataase PF01 [8], the FERET [9] 1 face dataase, and the ORL Dataase [10], to assess the effectiveness of BDA and compare it with, and 2D-PCA. The first experiment is performed on the suset of the IML dataase containing different facial expressions (see Figure 1(c-d)). Figure 4(d) shows that BDA outperforms, and with at most 10% recognition rate improvement when k = 14. The second experiment, performed on FERET, aims at testing the generalization power of BDA, and comparing it to, and. Indeed, LDA-ased methods are known to e more efficient when comparing faces of known people, ut provide worse generalization results than unsupervised methods. The training set (see Figure 4(a)) contains 818 images of 152 persons with at least 4 views / person, taken on different days, under different lighting conditions. Two test sets (see Figure 4(-c)) are compared, each one containing 200 persons with one view per person, taken from FERET ut not present in the training set. From one test set to the other, the facial expressions vary. From Figure 4(e), we can conclude that when the training set contains many classes and important variations inside the classes, BDA provides etter generalization than the other methods. IML - facial expressions 0,9 FERET - Comparison of the two test sets 0,65 (a) () (c) 0,6 0,55 0,5 0,45 BDA 0,85 0,8 5 BDA 0, Numer of projection vectors (d) Numer of projection vectors Figure 4: (a): Extract of the training set used for the second experiment. (-c): Extracts of the two test sets compared in the second experiment. (d): Compared recognition rates of BDA,, and, on the suset of IML with expression variations. (e): Compared classification rates of BDA,, and 2D PCA on the suset of the FERET dataase, when varying the numer k of projection vectors. For the third experiment, the ORL dataase has een randomly partitioned into a training set containing 5 views, and a test set containing the 5 remaining views, for each of the 40 persons. This operation has een repeated 7 times and BDA, and have een applied. Figure 5 shows that BDA provides etter recognition rates than and on all the random partitions, whenever outperforms (partitions (a-) and (d-g)) or outperforms (partition (c)). The results are computed from the optimal numer of projection vectors, which is k = 14 for the three methods. For further analysis, the contingency tale summed over partitions (a-g) is given in Tale 2. The total numer of tested faces is = The logical symol stands for not, i.e. the element corresponding to the second row and first column of the tale is the numer of faces recognized y, ut misclassified y. 1 Portions of the research in this paper use the FERET dataase of facial images collected under the FERET program. (e)

6 0,99 0,97 0,95 0,93 0,91 0,89 0,87 0,85 a c d e f g BDA Figure 5: Compared recognition rates of BDA, and on 7 random partitions of the ORL dataase. BDA Total Tale 2: Contingency tale summed over partitions (a-g) of the ORL dataase. The second row and second column element is the numer of samples correctly classified y and BDA, ut misclassified y. From tale 2 we can see that BDA correctly classifies = 99.6% of the samples that were correctly classified y oth and. Moreover, it recognizes the major part of the samples that were correctly classified y only one of the two methods (72.7% for and 63.6% for ). It also correctly classifies 35,4% of the samples that were misclassified y oth methods, which shows the efficiency of the BDA iterative algorithm. It should e noted that the size of the signature for one sample is = 1050 for, = 910 for, and only 14 2 = 196 for BDA. 5 Conclusion In this paper, we have proposed a new class-ased projection method, called Bilinear Discriminant Analysis, that can e successfully applied to face recognition. This technique, y maximizing a generalized Fisher criterion lying on a ilinear projection and computed directly from face image matrices, constructs a discriminant projection matrix. It has already een shown [3,4] that outperforms the traditional eigenfaces method. We have already highlighted [5] that 2DoLDA outperforms and the fisherfaces method. In this paper, we have shown on various international dataases that BDA is more efficient than 2DoLDA. Moreover, BDA, y providing reduced size image signatures, allows an important computational gain compared to 2DoLDA and during the recognition step. Aknowledgement This research was supported y the European Commission under contract FP acemedia. 6 References [1] M.A. Turk and A.D. Pentland, Eigenfaces for Recognition. Journal of Cognitive Neuroscience, 3(1), pages 71-86, [2] P.N. Belhumeur, J.P. Hespanha and D.J. Kriegman, Eigenfaces vs Fisherfaces : Recognition Using Class Specific Linear Projection. In IEEE Transactions on Pattern Analysis and Machine Intelligence, Special Issue on Face Recognition, 17(7), pages , [3] J. Yang, D. Zhang and A.F. Frangi, Two-Dimensional PCA: A New Approach to Appearance-Based Face Representation and Recognition. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(1), pages , Jan [4] M. Visani, C. Garcia and C. Laurent, Comparing Roustness of Two-Dimensional PCA and Eigenfaces for Face Recognition. In Proceedings of the 1 st International Conference on Image Analysis and Recognition (ICIAR 04), Springer Lecture Notes in Computer Science (LNCS 3211), A. Campilho, M. Kamel (eds), pages Sept [5] M. Visani, C. Garcia and J.M. Jolion, Two-Dimensional-Oriented Linear Discriminant Analysis for Face Recognition. In Proceedings of the International Conference on Computer Vision and Graphics (ICCVG 2004), to appear in Computational Imaging and Vision series, Warsaw, Poland, Sept [6] D.L. Swets and J. Weng, Using Discriminant Eigenfeatures for Image Retrieval. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8), pages , [7] R.L. Jenrich, Stepwise Discriminant Analysis, Standard Statistical Methods for Digital Computers, pages Ralston, A and Wilf, H.S. (eds), Wiley, New York, USA, [8] B.W. Hwang, M.C. Roh and S.W. Lee, Performance Evaluation of Face Recognition Algorithms on Asian Face Dataase. In IEEE International Conference on Automatic Face and Gesture Recognition, pages , Seoul, Korea, May [9] P.J. Phillips, H. Wechsler, J. Huang and P. Rauss, The FERET Dataase and Evaluation Procedure for Face Recognition

7 Algorithms. In Image and Vision Computing, 16(5), pages , [10] F. Samaria and A. Harter, Parameterisation of a Stochastic Model for Human Face Identification. In Proceedings of 2 nd IEEE Workshop on Applications of Computer Vision, Sarasota FL, Dec