A New Facial Expression Recognition Method Based on * Local Gabor Filter Bank and PCA plus LDA

Transcription

1 Hong-Bo Deng, Lan-Wen Jn, L-Xn Zhen, Jan-Cheng Huang A New Facal Expresson Recognton Method Based on Local Gabor Flter Bank and PCA plus LDA A New Facal Expresson Recognton Method Based on * Local Gabor Flter Bank and PCA plus LDA Hong-Bo Deng 1, Lan-Wen Jn 1, L-Xn Zhen, Jan-Cheng Huang 1 School of Electronc and Informaton Engneerng, South Chna Unversty of echnology, Guangzhou, , P.R.Chna Motorola Chna Research Center, Shangha, 10000, P.R.Chna {hbdeng, eelwn}@scut.edu.cn {L-Xn.Zhen, Jan-Cheng.Huang}@motorola.com Abstract hs paper proposes a facal expresson recognton system based on Gabor feature usng a novel local Gabor flter bank. radtonally, a global Gabor flter bank wth 5 frequences and 8 orentatons s often used to extract the Gabor feature. A lot of tme wll be nvolved to extract feature and the dmensons of such Gabor feature vector are prohbtvely hgh. A novel local Gabor flter bank wth part of frequency and orentaton parameters s proposed. In order to evaluate the performance of the local Gabor flter bank, we frst employed a two-stage feature compresson method PCA plus LDA to select and compress the Gabor feature, then adopted mnmum dstance classfer to recognze facal expresson. Expermental results show that the method s effectve for both dmenson reducton and good recognton performance n comparson wth tradtonal entre Gabor flter bank. he best average recognton rate acheves 97.33% for JAFFE facal expresson database. Keyword: Local Gabor flter bank, feature extracton, PCA, LDA, facal expresson recognton. I. Introducton Facal expressons delver rch nformaton about human emoton and play an essental role n human communcatons. In order to facltate a more ntellgent and natural human machne nterface of new multmeda products, automatc facal expresson recognton [1][18][0] had been studed world wde n the last ten years, whch has become a very actve research area n computer vson and pattern recognton. here are many approaches have been proposed for facal expresson analyss from both statc mages and mage sequences [1][18] n the lterature. In ths paper we focus on the recognton of facal expresson from sngle dgtal mages wth emphass on the feature extracton. A number of approaches have been developed for extractng * Proect sponsored by: Motorola Labs Research Foundaton (No.303D80437), NSFC (No ), GDNSF (No.003C50101, ). 86

2 Internatonal Journal of Informaton echnology Vol. 11 No features from stll mages. urk and Pentland [7] proposed Egenfaces employed prncpal component analyss (PCA). PCA [8][9] s an unsupervsed learnng method, whch treats samples of the dfferent classes n the same way. Fsherfaces proposed by Belhumeour and Hespanha [6] s a supervsed learnng method usng the category nformaton assocated wth each sample to extract the most dscrmnatory features. It has been shown that Fsherfaces performs well n many applcatons. Lyons [10] used Gabor wavelet [3][4][10][14] to code facal expressons. Nowadays varous researchers reported the model-based methods [1][18][0] for feature extracton, such as actve appearance model [19], pont dstrbuton model and labeled graphs. But those methods requre heavy computaton or manually detected feature nodes to construct the model, whch can hardly be mplemented n real-tme automatc facal expresson recognton (FER). Donato and Bartlett [1] compared varous methods of feature extracton for automatcally recognzng facal expresson, ncludng PCA, LDA, Gabor wavelet, etc. Best performances were obtaned usng the Gabor wavelet presentaton. But the computaton and memory requrement of such Gabor feature are very large and the dmenson s very hgh. he novelty of our method s to select a local Gabor flter bank, wth part of the entre m- frequency, n-orentaton set of Gabor flters, nstead of usng the entre global flter bank to extract feature. In contrast to the entre global Gabor flter bank, the use of the local Gabor flter bank can effectvely decrease the computaton and reduce the dmenson, even mprove the recognton capablty n some stuatons. For further dmensonalty reducton and good recognton performance, we adopt a two-phase framework PCA plus LDA for feature compresson and selecton n our facal expresson recognton system. he remander of the paper s organzed as follows: Secton descrbes the preprocessng procedure to get the pure expresson mage. Secton 3 presents the Gabor feature extracton and our novel local Gabor flter bank. Feature compresson based on PCA and LDA s dscussed n Secton 4. In Secton 5, experments are performed on the JAFFE facal expresson database [10] wth dfferent expermental condtons. Fnally, concluson s gven n Secton 6. II. Preprocessng Procedure Preprocessng procedure s very mportant step for facal expresson recognton. he deal output of processng s to obtan pure facal expresson mages, whch have normalzed ntensty, unform sze and shape. It also should elmnate the effect of llumnaton and lghtng. he preprocessng procedure of our FER system performs the followng fve steps n convertng a IFF JAFFE mage to a normalzed pure expresson mage for feature extracton: 1). detectng facal feature ponts manually ncludng eyes, nose and mouth; ). rotatng to lne up the eye coordnates; 3) locatng and croppng the face regon usng a rectangle accordng to face model [5] as shown n Fg.1. Suppose Fg. 1. Facal model Fg.. Example of pure facal expresson mages after preprocessng from JAFFE database. 87

3 Hong-Bo Deng, Lan-Wen Jn, L-Xn Zhen, Jan-Cheng Huang A New Facal Expresson Recognton Method Based on Local Gabor Flter Bank and PCA plus LDA the dstance between two eyes s d, the rectangle wll be.d 1.8d; 4). scalng the mage to fxed sze of 18 96, locatng the center poston of the two eyes to a fxed poston; 5). usng a hstogram equalzaton method to elmnate llumnaton effect. Fg. llustrates some examples of pure facal expresson mages after preprocessng from JAFFE database. Fg. 4. he magntudes of the Gabor feature representaton of the frst face mage n Fg. III. Gabor Feature Extracton he Gabor flters, whose kernels are smlar to the D receptve feld profles of the mammalan cortcal smple cells [3][4], have been consdered as a very useful tool n computer vson and mage analyss due to ts optmal localzaton propertes n both spatal analyss and frequency doman [][14][15][16][17]. A. Gabor Flters Fg. 3. he real part of the Gabor flters wth fve frequences and eght orentatons for ω max =π/, the row corresponds to dfferent frequency ω m, the column corresponds to dfferent orentaton θ n In the spatal doman, a Gabor flter s a complex exponental modulated by a Gaussan functon [4]. he Gabor flter can be defned as follows, x' + y' ϖ σ 1 ( ) σ ϖx' ψ ( x, y, ϖ, θ ) = e [ e e πσ x' = x cosθ + y snθ, y' = xsnθ + y cosθ ] (1) where (x, y) s the pxel poston n the spatal doman, ω the radal center frequency, θ the orentaton of Gabor flter, and σ the standard devaton of the round Gaussan functon along 88

4 Internatonal Journal of Informaton echnology Vol. 11 No ϖ the x- and y-axes. In addton, the second term of the Gabor flter, σ / e, compensates for the DC value because the cosne component has nonzero mean whle the sne component has zero mean. Accordng to [4], we set σ π/ω to defne the relatonshp between σ and ω n our experments. In most cases a Gabor flter bank wth fve frequences and eght orentatons [1][][18] s used to extract the Gabor feature for face representaton. Selectng the maxmum frequency ω max =π/, ω m =ω (max) λ -(m-1), m={1,,3,4,5}, λ =, θ n =(n-1)π/8, n={1,,,8}, the real part of the Gabor flters wth fve frequences and eght orentatons s shown n Fg.3. From Fg.3 t can be seen that the Gabor flters exhbt strong characterstcs of spatal localty and orentaton selectvty. B. Gabor Feature Representaton he Gabor feature representaton of an mage I(x, y) s the convoluton of the mage wth the Gabor flter bank ψ(x, y, ω m, θ n ) as gven by: O m,n ( m n x, y) = I( x, y) ψ ( x, y, ϖ, θ ) () where * denotes the convoluton operator. he magntude of the convoluton outputs of a sample mage (the frst mage n Fg.) correspondng to the flter bank n Fg.3 s shown n Fg.4. In practce, the tme for performng Gabor feature extracton s very long and the dmenson of Gabor feature vector s prohbtvely large. For example, f the sze of normalzed Fg. 5. Examples of several global and local Gabor flter bank, the black blocks are the selected flter for LG(mxn) mage s 18 96, the dmenson of the Gabor feature vector wth 40 flters wll result n ( ). C. Local Gabor Flter Bank It can be seen that the Gabor representatons are very smlar usng the flters wth the same orentaton, especally usng the flters wth the two neghborng frequences, such as the frst column n Fg.4. It s found that the Gabor feature vector wth all the 40 flters becomes very redundant and correlatve. For the global Gabor flter bank wth all the m frequences and n orentatons, we denoted t as G(mxn). In ths paper we proposed a novel local flter bank wth part of the entre m frequences and n orentatons, and we denoted t as LG(mxn). In order to select few Gabor flters to reduce the dmenson and computaton wthout degradng the recognton performance, t should cover all the frequences and orentatons, but only select one frequency for each orentaton or ncrease the nterval between the neghborng frequences wth the same orentaton. Several global and local Gabor flter banks are shown n Fg.5. he method of selectng the LG1(mxn) s that the parameter m of frequency ncreases repeatedly 89

5 Hong-Bo Deng, Lan-Wen Jn, L-Xn Zhen, Jan-Cheng Huang A New Facal Expresson Recognton Method Based on Local Gabor Flter Bank and PCA plus LDA from mn to max, and the parameter n of orentaton adds one for each tme. he dfference of LG(mxn) s that the parameter m of frequency decreases from max to mn. For LG3(mxn), t s selected wth an nterval between any two flters. he computaton and memory requred by dfferent global and local Gabor flter bank are gven n able1. By comparng the performance of G(mxn) wth LG(mxn), LG(mxn) has the advantages of shortenng the tme for feature extracton, reducng the dmenson, decreasng the computaton and storage. he recognton performance wll be depcted n Secton 5. Our experments were conducted usng a Pentum IV.8G PC wth 51MB memory. able 1. Computaton and memory requred by dfferent Gabor flter bank Gabor Flter Bank Number of Flters Orgnal Dmenson (D) Feature Extracton me(ms) Samplng Feature dmenson PCA Matrx G(5x8) G(4x8) G(3x8) LG1(3x8) LG3(3x8) o encompass the propertes of spatal localty and orentaton selectvty, t should concatenate [][13] all the outputs of Gabor flter bank and derve the Gabor feature vector. Before the concatenaton, we frst downsample each output of Gabor flter, and then normalze t to zero mean and unt varance. One smple scheme s to sample the facal feature n a regular grd [13], whch sample over the whole face regons wth regular nterval. he last columns of able1 show the dmenson and PCA matrx of samplng feature when the samplng nterval s 8 pxels. IV. Feature Compresson One approach to copng wth the problem of excessve dmensonalty s to reduce the dmensonalty by lnear combnng features [9]. In effect, lnear methods proect the hghdmensonal data onto a lower dmensonal space, we call t feature compresson. here are two classcal approaches to fndng effectve lnear transformatons, whch are Prncpal Component Analyss (PCA) [1][6][7][8][9] and Lnear Dscrmnant Analyss (LDA) [1][6][8][9]. PCA seeks a proecton that best represents the orgnal data n a least-squares sense, and LDA seeks a proecton that best separates the data n a least-squares sense. A. PCA Let us consder a set of N sample mages {x 1, x,, x N } represented by t-dmensonal Gabor feature vector. he PCA [8][9] can be used to fnd a lnear transformaton mappng the orgnal 90

6 Internatonal Journal of Informaton echnology Vol. 11 No t-dmensonal feature space nto an f-dmensonal feature subspace, where normally f << t. he f new feature vector y R are defned by y = W x ( = 1,,,N) (3) pca where W pca s the lnear transformatons matrx, s the number of sample mages. he columns of W pca are the f egenvectors assocated wth the f largest egenvalues of the scatter matrx S, whch s defned as S = N = 1 ( x µ )( x µ ) (4) t where µ R s the mean mage of all samples. he dsadvantage of PCA s that t may lose mportant nformaton for dscrmnaton between dfferent classes. B. LDA LDA [8][9] s a supervsed learnng method, whch utlzes the category nformaton assocated wth each sample. he goal of LDA s to maxmze the between-class scatter whle mnmzng the wthn-class scatter. Mathematcally speakng, the wthn-class scatter matrx S w and between-class scatter matrx S b are defned as S S w b = = c = 1 = 1 c = 1 N ( x µ )( x ( µ µ )( µ µ ) µ ) (5) where x s the th sample of class, µ s the mean of class, µ s the mean mage of all classes, c s the number of classes, and N s the number of samples of class. One way to select W lda s to maxmze the rato det S b /det S w. If S w s nonsngular matrx then ths rato s maxmzed, when the transformaton matrx W lda conssts of g generalzed egenvectors correspondng to the g largest egenvalues of S w -1 S b [1][6][8][9]. Note that there are at most c-1 nonzero generalzed egenvalues, and so an upper bound on g s c-1. In ths paper, we consder seven knds of facal expressons, so the dmenson of LDA s up to 6. C. PCA+LDA In order to guarantee S w not to become sngular, we requre at least t+c samples. In practce t s dffcult to obtan so many samples when the dmenson of feature s very hgh. o solve ths problem, a two-phase framework PCA plus LDA s proposed by [1][6][8][18]. It can be descrbed that PCA maps the orgnal t-dmensonal feature x to the f-dmensonal feature y as an ntermedate space, and then LDA proects the PCA output to a new g-dmensonal feature vectors z. More formally, t s gven by 91

7 Hong-Bo Deng, Lan-Wen Jn, L-Xn Zhen, Jan-Cheng Huang A New Facal Expresson Recognton Method Based on Local Gabor Flter Bank and PCA plus LDA z = W W x ( = 1,,,N) (6) lda pca o compare the performance of PCA wth PCA+LDA, recognton results usng PCA feature and PCA+LDA feature respectvely wll be reported n Secton 5. V. Experments and Results he proposed method s evaluated n terms of ts recognton performance usng the JAFFE female facal expresson database [10][11], whch ncludes 13 facal expresson mages correspondng to 10 persons. Every person posed 3 or 4 examples of each of the seven facal expressons (happness, sadness, surprse, anger, dsgust, fear, neural). wo facal expresson mages of each expresson of each subect were randomly selected as tranng samples, whle the remanng samples were used as test data. We have 138 tranng mages and 75 testng mages for each tral. Snce the sze of the JAFFE database s lmted, we perform the tral over 3 tmes to get the average recognton rate. In our experments, the nearest neghbor rule s then used to classfy the facal expresson mages. Experments were performed wth the PCA feature and PCA+LDA feature respectvely. wo dstance measures Eucldean (L) and ctyblock (L1) are employed by the classfer. A. Experment performance of dfferent Gabor Flter Bank he frst experment was desgned to compare the recognton performance usng dfferent Gabor flter bank. able shows the recognton results. he classfcaton performance shown n able suggested the followng conclusons: 1). he recognton rate of PCA feature s from 76.89% to 90.67%, L1 performs better than L wth PCA feature. When usng PCA+LDA feature to for FER, the recognton rate ncreases several percent and get up to 97.33%. L s slghtly better than L1 wth PCA+LDA feature. ). In comparson wth the performance of dfferent Gabor flter bank, t s found that G(5x8) s the best whle G(4x8) and G(3x8) decrease a lttle along wth fewer Gabor flters used. However local Gabor flter banks such as LG1(3x8), LG(3x8) and LG3(3x8) have almost the same classfcaton performance as global Gabor flter bank G(3x8). LG3(3x8) even outperforms than G(3x8) when usng PCA+LDA. able. Recognton rates usng PCA feature and PCA+LDA feature separately correspondng to dfferent Gabor flter bank Gabor Flter Bank Classfcaton Methods PCA PCA+LDA L L1 L L1 9

8 Internatonal Journal of Informaton echnology Vol. 11 No G(5x8) G(4x8) G(3x8) LG1(3x8) LG(3x8) LG3(3x8) LG3(4x8) B. Recognton performance aganst llumnaton normalzaton hs experment was desgned to test the effect of llumnaton normalzaton whch utlzed a hstogram equalzaton method. Selectng LG3(3x8) to extract feature, usng L1 for PCA feature and L for PCA+LDA feature, the recognton results aganst llumnaton normalzaton are gven by able3. able 3. Recognton rates wth and wthout llumnaton normalzaton Illumnaton Classfcaton Methods Normalzaton PCA PCA+LDA No (wthout) Yes (wth) From able3 t s obvous to see that llumnaton normalzaton s effectve for PCA feature to acheve hgh performance, whose recognton rate s mproved 4% or so. As to PCA+LDA feature, llumnaton normalzaton also works better, but t s not as clear as PCA feature. We can draw the concluson that PCA feature s senstve to varous llumnaton, but PCA+LDA feature may be less senstve to dfferent llumnaton. C. Elmnatng the frst several prncpal components Prevous studes n the feld of face recognton [6][7][8] reported that dscardng the frst one to three prncpal components (PCs) mproved performance. In ths paper, we dscarded the frst one to nne PCs to analyze the recognton effect for FER. Selectng G(3x8) and LG3(3x8) to extract feature, usng L1 for PCA feature and L for PCA+LDA feature, able4 depcts the results from ths experment. able 4. Recognton results by elmnatng the frst one to nne prncpal components Gabor Flter Bank Input Feature Number of Elmnated Prncpal Component

9 Hong-Bo Deng, Lan-Wen Jn, L-Xn Zhen, Jan-Cheng Huang A New Facal Expresson Recognton Method Based on Local Gabor Flter Bank and PCA plus LDA G(3x8) PCA PCA+LDA LG3(3x8) PCA PCA+LDA able4 demonstrated that elmnatng the frst one to three PCs wll reach to the best performance rates. Best performance for PCA feature, 91.11%, was obtaned by elmnatng the frst three PCs. Best performance for PCA+LDA feature, 97.33%, was obtaned by removng the frst two PCs. If dscardng more than three PCs, n general, the results wll become worse. VI. Conclusons In ths paper, we ntroduced our FER system based on Gabor feature and PCA+LDA. We proposed a novel local Gabor flter bank for feature extracton. A mnmum dstance classfer was employed to evaluate the recognton performance n dfferent experment condtons. he experments suggest the followng conclusons: 1). Local Gabor flter bank outperforms global Gabor flter bank n the aspects of shortenng the tme for feature extracton, reducng the hgh dmensonal feature, decreasng the requred computaton and storage, even achevng better performance n some stuatons. ). PCA can sgnfcantly reduce the dmensonalty of the orgnal feature wthout loss of much nformaton n the sense of representaton, but t may lose mportant nformaton for dscrmnaton between dfferent classes. When usng PCA feature to classfy, the L1 dstance measure performs better than L. Illumnaton normalzaton s effectve for PCA feature to acheve hgh performance. 3). When usng PCA+LDA method, the dmensonalty drastcally reduced to 6 dmensons and the recognton performance s mproved several percent compared wth PCA. Experments show that PCA+LDA feature may partly elmnate the senstvty of llumnaton. 4). Dscardng the frst one to three PCs wll reach to the best performance rates. If removng more than three PCs, t wll commonly worsen the results. he best performance, usng LG3(3x8) and PCA+LDA feature by elmnatng the frst two PCs, was 97.33% n our experments. References [1] G. Donato, M. S. Bartlett, J. C. Hager, Classfyng Facal Actons, IEEE rans. Pattern Analyss and Machne Intellgence, Vol. 1, 1999, pp [] C. Lu, H. Wechsler, Independent Component Analyss of Gabor Features for Face recognton, IEEE rans. Neural Networks, Vol. 14, 003, pp [3] J. G. Daugman, Complete Dscrete -D Gabor ransforms by Neural Networks for Image Analyss and Compresson, IEEE rans. Acoustc, speech and sgnal processng, Vol. 36, 1988, pp [4]. S. Lee, Image Representaton Usng D Gabor Wavelets, IEEE rans. Pattern Analyss and Machne Intellgence, Vol. 18, 1996, pp

10 Internatonal Journal of Informaton echnology Vol. 11 No [5] F. Y. Shh, C. Chuang, Automatc extracton of head and face boundares and facal features, Informaton Scences, Vol. 158, 004, pp [6] P. N. Belhumeour, J. P. Hespanha, D. J. Kregman, Egenfaces vs. Fsherfaces: Recognton Usng Class Specfc Lnear Proecton, IEEE rans. Pattern Analyss and Machne Intellgence, Vol. 19, 1997, pp [7] M. urk, A. Pentland, Egenfaces for Recognton, Journal Cogntve Neuro-scence, Vol. 3, 1991, pp [8] A. M. Martnez, A. C. Kak, PCA versus LDA, IEEE rans. Pattern Analyss and Machne Intellgence, Vol. 3, 001, pp.8-33 [9] R. O. Duda, P. E. Hart, D. G. Stork, Pattern Classfcaton. Wley, New York (001) [10] M. J. Lyons, S. Akamatsu, M. Kamach, J. Gyoba, Codng Facal Expressons wth Gabor Wavelets, In: Proceedngs of the 3th IEEE Internatonal Conference on Automatc Face and Gesture Recognton, Nara, Japan, 1998, pp [11] Z. Zhang, M. Lyons, M. Schuster, S. Akamatsu, Comparson Between Geometry-Based and Gabor-Wavelets-Based Facal Expresson Recognton Usng Mult-Layer Perceptron, In: Proceedngs of the hrd IEEE Internatonal Conference on Automatc Face and Gesture Recognton, Nara, Japan, 1998, pp [1] I. A. Essa, A. P. Pentland, Codng, Analyss, Interpretaton, and Recognton of Facal Expressons, IEEE rans. Pattern Analyss and Machne Intellgence, Vol. 19, 1997, pp [13] D. H. Lu, K. M. Lam, L. S. Shen, Optmal samplng of Gabor features for face recognton, Pattern Recognton Letters, Vol. 5, 004, pp [14] C. Lu, H. Wechsler, Gabor Feature Based Classfcaton Usng the Enhanced Fsher Lnear Dscrmnant Model for Face Recognton, IEEE rans. Image Processng, Vol. 11, 00, pp [15] I. R. Fasel, M. S. Bartlett, J. R. Movellan, A Comparson of Gabor Flter Methods for Automatc Detecton of Facal Landmarks, In: Proceedngs of Ffth IEEE Internatonal Conf. Automatc Face and Gesture Recognton, Washngton, USA, 00, pp [16] L. L. Huang, A. Shmzu, H. Kobatake, Classfcaton-Based Face Detecton Usng Gabor Flter Features, In: Proceedngs of Ffth IEEE Internatonal Conference on Automatc Face and Gesture Recognton, Seoul, Korea, 004, pp [17] V. Kyrk, J. K. Kamaranen, H. Kalvanen, Smple Gabor feature space for nvarant obect recognton, Pattern Recognton Letters, Vol. 5, 004, pp [18] B. Fasel, J. Luettn, Automatc facal expresson analyss: a survey, Pattern Recognton, Vol. 36, 003, pp [19]. Cootes, G. Edwards, C. aylor, Actve appearance models, IEEE rans. Pattern Analyss and Machne Intellgence, Vol. 3, 001, pp [0] M. Pantc, J. M. Rothkrantz, Automatc Analyss of Facal Expressons: he State of the Art, IEEE rans. Pattern Analyss and Machne Intellgence, Vol., 000, pp Hong-Bo Deng receved hs B.S. degree n Electroncal Engneerng from South Chna Unversty of echnology, Chna, n 003. He s currently a M.S. student n the School of Electronc and Informaton Engneerng, South Chna Unversty of echnology, Chna. Hs current nterests nclude Face Expresson Recognton, Image Processng, Computer Vson, Human-Computer Interface. 95

11 Hong-Bo Deng, Lan-Wen Jn, L-Xn Zhen, Jan-Cheng Huang A New Facal Expresson Recognton Method Based on Local Gabor Flter Bank and PCA plus LDA Lan-Wen Jn, an IEEE member, receved hs Ph.D degrees from South Chna Unversty of echnology, Chna, n He s currently a professor at School of Electronc and Informaton Engneerng, South Chna Unversty of echnology, Chna. He has research nterests n varous aspects of Character Recognton, Human-Computer Interface, Image Processng, Neural Network, and Computer Vson. Dr. L-Xn Zhen s currently a senor research manager of Motorola Labs Chna Research Center. Dr. Jan-Cheng Huang s currently the drector of Motorola Labs Chna Research Center. 96