Enhanced Fisher Linear Discriminant Models for Face Recognition

Transcription

1 Appears in the 14th International Conference on Pattern Recognition, ICPR 98, Queensland, Australia, August 17-2, 1998 Enhanced isher Linear Discriminant Models for ace Recognition Chengjun Liu and Harry Wechsler Department of Computer cience, George Mason University, 44 University Drive, airfax, VA , UA cliu, Abstract We introduce in this paper two Enhanced isher Linear Discriminant (LD) Models (EM) in order to improve the generalization ability of the standard LD based classifiers such as isherfaces imilar to isherfaces, both EM models apply first Principal Component Analysis (PCA) for dimensionality reduction before proceeding with LD type of analysis EM-1 implements the dimensionality reduction with the goal to balance between the need that the selected eigenvalues account for most of the spectral energy of the raw data and the requirement that the eigenvalues of the within-class scatter matrix in the reduced PCA subspace are not too small EM-2 implements the dimensionality reduction as isherfaces do It proceeds with the whitening of the within-class scatter matrix in the reduced PCA subspace and then chooses a small set of features (corresponding to the eigenvectors of the within-class scatter matrix) so that the smaller trailing eigenvalues are not included in further computation of the between-class scatter matrix Experimental data using a large set of faces 1,17 images drawn from 369 subjects and including duplicates acquired at a later time under different illumination from the ERE database shows that the EM models outperform the standard LD based methods 1 Introduction A successful face recognition methodology depends heavily on the particular choice of the features used by the (pattern) classifier [2], [5] eature selection in pattern recognition involves the derivation of salient features from the raw input data in order to reduce the amount of data used for classification and simultaneously provide enhanced discriminatory power wo popular techniques for selecting a subset of features are Principal Component Analysis (PCA) and the isher Linear Discriminant (LD) PCA is a standard decorrelation technique and following its application one derives an orthogonal projection basis which directly leads to dimensionality reduction, and possibly to feature selection Methods such as LD, possibly following methods such as PCA and operating then in a compressed subspace, seek for discriminatory features by taking into account within- and between-class scatter as the relevant information for pattern classification While PCA maximizes for all the scatter as appropriate for signal representation, LD differentiates between the within- and between-class scatter as appropriate for pattern classification PCA was first applied to represent human faces by Kirby and irovich [4] and to recognize faces by urk and Pentland [7] he recognition method, known as eigenfaces, defines a feature space, or face space, which drastically reduces the dimensionality of the original space, and face detection and identification are carried out in the reduced space LD has also been applied to solve face recognition problems By applying first PCA for dimensionality reduction and then LD for discriminant analysis Belhumire, Hespanha, and Kriegman [1] developed an approach called isherfaces for face recognition Using a similar approach, wets and Weng [6] have pointed out that the eigenfaces derived using PCA are only the most expressive features (ME) he ME are unrelated to actual face recognition, and in order to derive the most discriminating features (MD), one needs a subsequent LD projection Methods that combine PCA and standard LD, however, lack in their generalization ability as they overfit to the training data his paper explains the reasons why overfitting takes place, introduces two Enhanced LD Models (EM) to overcome problems associated with overfitting, and presents experimental data whose enhanced performance supports the EM models 2 he tandard LD Based Methods and Overfitting he standard LD based methods such as isherfaces apply first PCA for dimensionality reduction and then discriminant analysis Relevant questions concerning PCA are

2 = usually related to the range of Principal Components (PCs) used and how it affects performance Regarding discriminant analysis one has to understand the reasons for overfitting and how to avoid it he answers to those two questions are closely related One can actually show that using more PCs may lead to decreased performance (for recognition) he explanation for this behavior is that the trailing eigenvalues correspond to high-frequency components and usually encode noise As a result, when these trailing but small valued eigenvalues are used to define the reduced PCA subspace, the LD procedure has to fit for noise as well and as a consequence overfitting takes place he LD procedure involves the simultaneous diagonalization of the two within- and between-class scatter matrices and it is stepwise equivalent to two operations as pointed out by ukunaga [3]: first whitening the within-class scatter matrix, and second applying PCA on the between-class scatter matrix using the transformed data he purpose of the whitening step is to normalize (to unity) the within-class scatter matrix for uniform gain control he second operation maximizes then the between-class scatter to separate different classes as much as possible he robustness of the LD procedure depends on whether or not the within-class scatter captures reliable variations for a specific class Note that when PCA precedes LD for dimensionality reduction and more PCs are used, the trailing eigenvalues of the within-class scatter matrix tend to capture noise and their values are fairly small As during whitening the eigenvalues of the within-class scatter matrix appear in the denominator, the small (trailing) eigenvalues cause the whitening step to fit for misleading variations and thus generalize poorly when exposed to new data he Enhanced LD Models (EM) presented in the next section address those problems and display increased generalization ability 3 Enhanced LD Models (EM) wo Enhanced LD Models (EM-1 and EM-2) are introduced to improve on the generalization ability of the standard LD based methods such as isherfaces Both EM-1 and EM-2 apply first PCA for dimensionality reduction before proceeding with LD type of analysis EM- 1 implements the dimensionality reduction step with the goal to balance between the need that the selected eigenvalues (corresponding to the PCs for the original image space) account for most of the spectral energy of the raw data and the requirement that the eigenvalues of the withinclass scatter matrix (in the reduced PCA subspace) are not too small EM-2 implements dimensionality reduction as isherfaces do It proceeds with the whitening of the withinclass scatter matrix in the reduced PCA subspace and then chooses a small set of features (corresponding to the eigenvectors of the within-class scatter matrix) so that the smaller trailing eigenvalues are not included in further computation of the between-class scatter matrix!"# $&('*),+- PCA generates a set of orthonormal basis vectors, known as principal components (PCs), that maximize the scatter of all the projected samples Let /132 /,4657/98#5;:<:;:;57/>=@? be the sample set of the original images After normalizing the images to unity norm and subtracting the grand mean a new image set ABB2 AC4D57A8D5<:;:;:<57A=@? is derived Each AE represents a normalized image with dimensionality, AEGIHKJ#EMLN5OJ#EQPD5<:;:<:;5OJEQR, HKUVIW#5(X"5;:<:;:Y57Z he covariance matrix of the normalized image set is defined as []\ W Z EQ AÈ A E W Z AGA (1) and the eigenvector and eigenvalue matrices a, b are computed as [ \ acda b (2) Note that AVAV is an fe matrix while Ag A is an ZheiZ matrix If the sample size Z is much smaller than the dimensionality, then the following method saves some computation [7] H`A Agkjdljmbn4 (3) o Apj (4) where bn4nq#u rst6u4n5ku8#5<:;:;:<5ku=v o, and w2xan4d5(a 8#5;:;:<:Y5 a =@? If one assumes that the eigenvalues are sorted in decreasing order, uc46yzu8#y {<{;{Qyzu}=, then the first ~ leading eigenvectors define matrix 2 an4n5(a 8#5;:;:<:Y5(a z? (5) he new feature set with lower dimensionality ~ (~ ƒ ) is then computed as ` EM-1 A (6) or EM-1 model, our purpose is to choose ~, the number of PCs in Eq 5, such that proper balance is preserved between the need that the selected eigenvalues (corresponding to the PCs for the original image space) account for most of the spectral energy of the raw data and the requirement that the eigenvalues of the within-class scatter matrix (in the reduced PCA subspace) are not too small he eigenvalue spectrum of PCA is derived by Eq 3, and the relative magnitude of the eigenvalues for the face data used in our experiments (see ect 4) is shown in ig 1 he index for the eigenvalues ranges from 1 to ~ˆŠ, where L stands 2

3 W W I f k [ i [ i D D! &' () * H`uË "$# _$4 u for the number of (face) classes considered in our experiments One can see from ig 1 that the first 5 eigenvalues capture most of the energy and that the eigenvalues whose index is greater than 1 are fairly small and most likely capture noise o calculate the eigenvalue spectrum of the withinclass scatter matrix in the reduced PCA subspace one needs to decompose the LD procedure into two operations: whitening and diagonalization [3] In particular, let + 4N5,+$8#5;:<:;:Y5-+/ and 94657g8D5<:;:;:<57 denote the classes and the number of images within each class, respectively Let c4651 8D5<:;:<:Y523 and be the means of the classes and the grand mean in the reduced PCA subspace 4-5 r@z 2 a 4 5 a 8 5<:;:;:<5(a? We then have 6/7 ; 8 :9 8 5 <> W#5kX 5;:<:;: 57 (7) B >H=+ > (8) ; where?9 8 5@g W#5(X"5;:<:;: 5k, represents the sample images from class +, and >HA+ is the prior probability [CB of [CD + he within- and between-class scatter matrices and in the reduced PCA subspace are estimated as [EB >HA+ G H CI ; 8KJ 9 ML ; 8KJ 9 ML ON P Q (9) [ D >H=+ YHR J w;h J w (1) he standard LD procedure derives a projection matrix that maximizes the ratio U U>"WU z XU he ratio [ZY is maximized when consists of the eigenvectors of B 4 [VD corresponding to the leading eigenvalues he stepwise LD [ procedure derives the eigenvalues and eigenvectors of as the result of the simultaneous B Y 4 [VD diagonal- [ B [ D ization of and irst whiten the within-class scatter matrix [ B\[ [] ] Y 4>a78 [ r@zq [VD [EB `_ (11) [ B\[] Y 4>ak8 `_ (12) he eigenvalue spectrum of the within-class scatter matrix in the reduced PCA subspace can be derived by Eq 11, and different spectra are obtained corresponding to different number of PCs that are utilized ig 2 displays the relative magnitudes of the eigenvalues corresponding to the within-class scatter matrix Now one has to simultaneously optimize the behavior of the trailing eigenvalues in the reduced PCA space (ig 2) with the energy criteria for the original image space (ig 1) Note that different choices on the cutoff PC index for the original image space yield different within-class spectra as shown in ig 2 As one can see from ig 1 and 2, a suitable choice is to set ~ cbed, since this choice not only accounts for most of the spectral energy of the raw data (ig 1) but also meets the requirement that the eigenvalues of the within-class scatter matrix (in the reduced 5 dimensional PCA subspace) are not too small (ig 2) After the number of PCs is set, EM-1 model computes the between-class scatter matrix as ] Y 4>a78 [ [ D-[] Y 4>ak8 gf (13) Diagonalize the new between-class scatter matrix f D1h h:i r@zq j_ (14) he overall transformation matrix (after PCA, see Eq 6) for EM-1 model is now defined as K EM-2 [] Y 4-ak8 h (15) he EM-2 model uses the same number of PCs as utilized by isherfaces [1] and MD method [6] for dimensionality reduction, namely, mll~nlz J, where Z is the number of training samples and the number of classes 3

4 f ' f ' m = 3 m = 4 m = 5! &' ) (** e,3! -!j * ) : 1 *1! \ Z &' *! ~Šd ~ (d ~ bed '& W! ) * & "! (! # $ & he LD proceeds then by choosing a small set of features (corresponding to the eigenvectors of the within-class scatter matrix see Eq 11) after the whitening procedure so that the smaller trailing eigenvalues are not included Let 4 ( 4& ~ ) be the chosen in the whitening subspace according to the eigenvalue spectrum [EB of (see ig 3, a suitable choice is to set 4 W dd ), and the new feature matrix [(' is derived [ ' 2 ) 4 5*) 8 5;:;:<:;5*) +7? (16) where ) 4 5*) 8 5;:;:<:Y5"),+ are the eigenvectors of [CB corresponding to the leading eigenvalues -4D5-"8D5;:<:;:Y5"- + he new diagonal matrix is ] ' qurs t/- 4N5-"8#5<:;:;:;5- + v (17) he new whitening transformation matrix in the reduced s dimensional whitening subspace is [ ' ] ' H ;H Y 4>a78 (18) [WD he between-class scatter matrix in this subspace is transformed to f (19) Note that the new between-class scatter matrix f D is 4 e 4 now instead of ~ e,~ Diagonalize f D h ' h ' i ' h ' h ' r@zq `_ (2) [VD 21 X! &' m ) (*`! -!34! 6 *1 Hu 5? E " # 1 ) u he overall transformation matrix (after PCA, see Eq 6) for EM-2 model is k ' [ ' ] ' Y 4>a78 h ' (21) 4 Experimental Results and uture Work he experimental data, consisting of 1,17 facial images corresponding to 369 subjects, comes from the ERE database 6 out of the 1,17 images correspond to 2 subjects with each subject having three images two of them are the first and the second shot, while the third shot is taken under low illumination or the remaining 169 subjects there are also three images for each subject, but two out of the three images are duplicates taken at a different time wo images of each subject are used for training with the remaining one used for testing he images are cropped to the size of 64 x 96, once the eye coordinates are manually detected ig 4 displays the recognition performance for the isherfaces It confirms that after using a certain number of features (around 7) the performance peaks he graphs were obtained when ~ is optimally set to because lw~ lwz J and Zc XD he top 1 recognition rate records the accuracy rate for the top response being correct, while the top 3 recognition rate records the accuracy rate for the correct response being included among the first three ranked choices ig 5 shows the comparative performance for isherfaces and our new EM-1 and EM-2 models he EM 4

5 top 1 and top 3 recognition rate top 1 top 3 top 3 recognition rate LD EM-1 EM-2 & & '& - # O '& * ' e *: 1 * H ~Š & ' & - O - & 1 '&? * ' O (*! * &! * models increase the top 1 recognition rate by 1 to 15 compared to the isherfaces ig 6 plots the top 3 recognition rate, and again our EM models improve the performance by 1 to 15 top 1 recognition rate & ' & > # e, & ' &? * ' O *! * &! * LD EM-1 EM-2 We plan to expand the EM approach so that it seeks the best subset of PCs rather than the first ones corresponding to the leading eigenvalues While the first PCs are probably useful as they encode for global image characteristics, in analogy to the characteristics displayed by the human visual system, there is reason to believe that one should at least attenuate the very first PCs As PCA encodes 2nd order statistics and those are equivalent to the power spectrum, one possible choice would be to weight the PCs with a filter whose characteristics are similar to the Contrast ensitivity unction (C) Another possibility is to actually find the best subset of PCs using a greedy search type of algorithm Acknowledgments: his work was partially supported by the DoD Counterdrug echnology Development Program, with the U Army Research Laboratory as echnical Agent, under contract DAAL1-97-K-118 References [1] P Belhumeur, J Hespanha, and D Kriegman Eigenfaces vs fisherfaces: Recognition using class specific linear projection IEEE rans Pattern Analysis and Machine Intelligence, 19(7):711 72, 1997 [2] R Chellappa, C Wilson, and irohey Human and machine recognition of faces: A survey Proc IEEE, 83(5):75 74, 1995 [3] K ukunaga Introduction to tatistical Pattern Recognition Academic Press, second edition, 1991 [4] M Kirby and L irovich Application of the karhunen-loeve procedure for the characterization of human faces IEEE rans Pattern Analysis and Machine Intelligence, 12(1):13 18, 199 [5] A amal and P Iyengar Automatic recognition and analysis of human faces and facial expression: A survey Pattern Recognition, 25(1):65 77, 1992 [6] D L wets and J Weng Using discriminant eigenfeatures for image retrieval IEEE rans on PAMI, 18(8): , 1996 [7] M urk and A Pentland Eigenfaces for recognition Journal of Cognitive Neuroscience, 13(1):71 86,