A Novel Approach to Expression Recognition from Non-frontal Face Images

Transcription

1 A Novel Appoach to Expesson Recognton fom Non-fontal Face Images Wenmng Zheng 1,2, HaoTang 1, Zhouchen Ln 3, Thomas S. Huang 1 1 Beckman Insttute, Unvesty of Illnos at Ubana-Champagn, USA 2 Reseach Cente fo Leanng Scence, Southeast Unvesty, Nanjng , Chna. 3 Vson Computng Goup, Mcosoft Reseach Asa, Chna E-mal: wenmng zheng@seu.edu.cn Abstact Non-fontal vew facal expesson ecognton s mpotant n many scenaos whee the fontal vew face mages may not be avalable. Howeve, few wok on ths ssue has been done n the past seveal yeas because of ts techncal challenges and the lack of appopate databases. Recently, a 3D facal expesson database (BU-3DFE database) s collected by Yn et al. [10] and has attacted some eseaches to study ths ssue. Based on the BU-3DFE database, n ths pape we popose a novel appoach to expesson ecognton fom non-fontal vew facal mages. The novelty of the poposed method les n ecognzng the mult-vew expessons unde the unfed Bayes theoetcal famewok, whee the ecognton poblem can be fomulated as an optmzaton poblem of mnmzng an uppe bound of Bayes eo. We also popose a close-fom soluton method based on the powe teaton appoach and ank-one update (ROU) technque to fnd the optmal solutons of the poposed method. Extensve expements on BU-3DFE database wth 100 subjects and 5 yaw otaton vew angles demonstate the effectveness of ou method. 1. Intoducton Automatc facal expesson ecognton has become a vey hot eseach topc n compute vson and patten ecognton communty due to ts techncal challenge and the wde potental applcatons n many felds. A majo task of facal expesson ecognton s to classfy a gven facal mage nto sx categoes,.e. angy, dsgust, fea, happy, sad, and supse, based on some facal featues. Dung the past seveal yeas, many facal expesson ecognton methods have been poposed. Fo a lteatue suvey, see [3][8][11]. A majo lmtaton of the pevous facal expesson ecognton methods s that most of them focus on the fontal o neay fontal vew facal mages. In many scenaos, howeve, fontal o neay fontal vew face mages may not be avalable. Moeove, people have found that nonfontal facal mages may be moe nfomatve than fontal ones because n non-fontal facal mages the heghts of the nose and cheek may be avalable, whle fontal ones ae usually symmetc and edundant. Recently, a publcly avalable 3D facal expesson database, called BU-3DFE, was collected by Yn et al. [10] at State Unvesty of New Yok at Bnghamton. The BU- 3DFE facal expesson database attacted some eseaches to nvestgate the non-fontal vew facal expesson ecognton ssue and had obtaned some nteestng fndngs. Fo example, Wang et al. [9] found that the expesson ecognton pefomed pooy when the facal mages suffeed fom lage vew vaaton by compang the expesson ecognton pefomance unde the dffeent testng facal vew egons based on the classfe taned on the fontal-vew facal mages. Anothe nteestng fndng was conducted by Hu et al. [6] who conducted vaous facal expesson expements on fve yaw vews,.e., 0 o, 30 o, 45 o, 60 o, and 90 o. They found that usng the face mages unde the non-fontal vew can obtan bette ecognton pefomance than unde the fontal vew. The majo lmtaton of the non-fontal vew appoache s that they ae vew-dependent between the tanng and testng face mages,.e., they tan the algothm wth the face mages of some specfc vews. Hence, they need to know the vewng angles of the testng mages befoe the ecognton pocedue s pefomed, whch s stll a challengng wok n compute vson. In ths pape, we popose a novel appoach to the expesson ecognton fom non-fontal vew facal mages. The novelty of the poposed method les n ecognzng the mult-vew expessons unde the unfed Bayes theoetcal famewok. Moe specfcally, we fomulate the multvew facal expesson ecognton poblem as the optmzaton poblem of mnmzng an uppe bound of Bayes eo, whch bols down to solve a sees of pncpal egenvecto of matces, whch can be effcently solved va the powe teaton appoach [5] and ank-one update (ROU) technque. Moeove, to obtan bette ecognton pefomance, we use

2 the SIFT (Scale Invaant Featue Tansfom) featues [7] to epesent the face mages. The SIFT featues ae known to be nvaant to mage scale and otaton, and also obust acoss changes n llumnaton, nose, and a substantal ange of affne dstoton [7]. In ths pape, we extact the spase SIFT featues loacted 83 facal featue ponts (FPs) of each face mage to epesent the the face mages. The est of ths pape s oganzed as follows: In secton 2, we addess the BU-3DFE database and the SIFT featues epesentaton. In secton 3, we addess ou nonfontal vew facal expesson ecognton appoach. The expements ae pesented n secton 4. Fnally secton 5 concludes ou pape. 2. Descpton of the BU-3DFE Database and SIFT Featues Repesentaton The BU-3DFE database conssts of 100 subjects (56 female and 44 male) of dffeent ethnctes, each of whom elcts 6 unvesal facal expessons (ange, dsgust, fea, happness, sadness, and supse) wth 4 levels of ntenstes. The D facal expesson models ae descbed by both 3D geometcal shapes and colo textues. To facltate coespondences, 83 FPs ae dentfed on evey 3D model. A moe detaled descpton of the database can be found n [10]. We ende the 3D models wth OpenGL by selectng pope vewponts, esultng n 5-vew pojected face mages coespondng to 0 o, 30 o, 45 o, 60 o, and 90 o yaw angles. As a esult, the 83 FPs ae also pojected onto the same 2D faces. Fg. 1 shows some examples of the 2D face mages endeed fom the 3D face models. Fgue 1. Some facal mages of dffeent expessons obtaned by pojectng 3D face models n BU-3DFE onto dffeent vews. Fo each 2D face mage, we extact the spase SIFT featues located n 83 FPs to epesent that mage, whee the spase SIFT featues ae extacted as follows: 1. Fo each 2D face mage, we use the 83 FPs as the key ponts whee the SIFT featues to be computed. If the key ponts ae occluded by the face, we smply compute the SIFT featues n the coespondng postons of the mage. 2. In computng the SIFT featues, the same fxed hozontal oentaton s used fo all the 83 key ponts. 3. The oentaton hstogams of 4 x 4 sample egons of each key ponts ae used to calculate the SIFT featues. By computng the 128 dmensonal SIFT descptos at the 83 spase featue ponts, we obtan a dmensonal featue vecto to epesent each 2D facal mage. To educe the computatonal cost, all the SIFT featue vectos ae educed fom dmensonal vecto space to 500- dmensonal featue vecto space va a common pncpal component analyss (PCA) tansfomaton matx. 3. Ou Appoach to Expesson Recognton fo Non-fontal Vew Facal Images In ths secton, we wll deve ou new appoach to the non-fontal vew facal expesson ecognton. Fo the smplcty of the devaton, let C denote the numbe of the facal expesson classes and let K denote the numbe of the vews of each class Two-class Mult-vew Facal Expesson Recognton Unde Bayes Theoetcal Famewok Let denote the Bayes eo of classfyng the expessons between the th vew of the th class and the jth vew of the lth class, whee l. Then we have [4] = mn(p p (x),p jl p jl (x))dx. (1) P and P jl ae po pobabltes, and p (x) and p jl (x) ae class-condtonal pobablty densty functons. The basc dea of ou method s to handle the two-class facal expesson ecognton poblem,.e., classfyng the th class and the lth class, as a mult-class facal expesson ecognton poblem, whee the facal mages fom the same vew and class s vewed as an ndependent subclass. Howeve, t should be noted that the Bayes eo between any two subclasses that belong to the same basc expesson categoy should be zeo. The Gaph llustated n Fg. 2 shows the basc dea of ou method, whee the mult-class Bayes eo poblem was dvded nto seveal pas of twoclass Bayes eo poblem. Based on Fgue 2, we obtan that the Bayes eo between the th class and the lth class, denoted by E, satsfes the followng nequalty [2]: E =1 j=1. (2)

3 and the Bayes eo can be expessed as Fgue 2. The Gaph descbng the Bayes eo between the th class and the lth class, whee each class contans K dffeent vews. If we suppose that each subclass s Gaussan dstbuted,.e., p (x) =N(m, Σ ) (3) then applyng the followng nequalty mn(a, b) ab, a, b 0. (4) to the expesson n (1), we obtan that the Bayes eo can be expessed as: P P jl p (x)p jl (x)dx = P P jl exp( d j ), (5) whee d j s the Bhattachayya dstance defned by dj = 1 8 (m m jl ) T j ( Σ ) 1 (m m jl )+ 1 2 ln j Σ, Σ Σ jl j whee Σ = 1 2 (Σ + Σ jl ). Now pojectng the sample vectos to 1D featue by a vecto ω, the dstbuton of p (x) becomes p (x) =N(ω T m,ω T Σ ω), (6) and the Bhattachayya dstance becomes d j = 1 [ω T (m m jl )] 2 8 ω T Σ j ω ln ω T Σj ω (ωt Σ ω)(ω T Σ jl ω). (7) Let u = ω T Σj ω, v = ωt ΔΣ j ω, whee ΔΣj = 1 2 (Σ Σ jl ). Then d j can be wtten as d j = 1 8 [ω T (m m jl )] 2 ωt j Σ ω 1 4 ln(1 ( v u )2 ) 1 [ω T (m m jl )] 2 8 ωt j Σ ω ( v u )2, (8) P P jl exp( d j ) ( ) P P jl exp 1 [ω T (m m jl )] 2 8 ωt j Σ ω 1 4 ( v u )2 ( ) P P jl ω T ΔΣ j 2 P P jl ω 4 ωt j Σ ω P P jl [ω T (m m jl )] 2 8 ωt j Σ ω. (9) Wthout loss of genealty, we assume the equal po pobablty fo all subclasses,.e., P = P ( = 1,,K; =1,,C), then (9) can be smplfed as P P 8 [ω T (m m jl )] 2 ωt j Σ ω P 4 ( ω T ΔΣ j ω ω T j Σ ω 3.2. Mult-class Mult-vew Facal Expesson Recognton Unde Bayes Theoetcal Famewok Fo the C-class facal expesson ecognton poblem, we use Fg. 3 to llustate the components of the oveall Bayes eo, whee E and E l denotes the same Bayes eo between the th class and the lth class. Let E denote the oveall Bayes eo pobablty of classfyng the C classes expessons, then fom the mult-class Bayes eo theoy [2] and (10) wehave E C 1 C =1 l=+1 E = 1 2 C =1 C l =1, l E (11) Combnng (2), (9) and (11), we obtan that the Bayes eo n (11) can be expessed as E P 2 j P [ω T (m m jl )] 2 16 j ωt j Σ ω ( ) P ω T ΔΣ j 2 (12) 8 ωt j Σ j ω Recusvely applyng the followng nequaltes ( a ) 2 ( c ) ( ) 2 2 a + c +, a, c 0; b, d > 0 b d b + d (13) a b + c d a + c, a, c 0; b, d > 0 b + d (14) j ) 2. (10)

4 Fgue 3. The Bayes eo between any two dffeent classes, whee the numbe of the facal expessons s fxed at the 6 basc emotons. It should be noted that the Bayes eo E and E l means the same poblem and thus they should be equal. to the eo bound n (12), we have E 1 P 2 j P 3 j [ωt (m m jl )] 2 16 j P 2 ωt j Σ ω ( P j P 2 ω T ΔΣ j ω ) 2 8 j P 2 ωt j Σ ω.(15) Let B = j (m m j )(m m j ) T, and Σ = j P 2 Σj. Then we obtan that to mnmze the Bayes eo, we should mnmze ts uppe bound, whch bols down to maxmzng the followng dscmnant cteon: ( J(ω) = ωt Bω ω T Σω + 1 j P 2 ω T ΔΣ j ω 2P 2 ω T Σω (16) Fo the ease of computaton, we use the followng fomula as the dscmnant cteon: J(ω, μ) = = ωt Bω 4μ + ωt Σω 2P 2 ω T Bω ω T Σω + μ ) 2 j P 2 ω T ΔΣ j ω ω T Σω j ωt (Σ Σ jl )ω ω T Σω (17) whee 0 < 4μ < 1 0 < μ < 1 4 s a paamete fo compensate the change of the second tem n J(ω). On the othe hand, we note that ω T (Σ Σ jl )ω ω T (Σ Σ jl )ω j=1 = K ω T (Σ 1 K j=1 Σ jl )ω (18) j=1 Let Σ l = 1 K K j=1 Σ jl. Then, fom (17) and (18), we obtan J(ω, μ) ωt Bω + Kμ ωt (Σ Σ l )ω ωt Σω ω T Σω. (19) Let g(ω, μ) = ωt Bω ω T Σω +Kμ ωt (Σ Σ l )ω ω T Σω, then we obtan that maxmzng cteon g(ω, μ) wll also maxmze cteon J(ω, μ). To smplfy the computaton, we use the cteon g(ω, μ) as the dscmnant cteon to deve the optmal dscmnant vectos fo the dmensonalty educton of the facal expesson featue vectos. Based on the cteon g(ω, μ), we defne the followng optmal set of dscmnant vectos: ω 1 = ag max g(ω, μ), ω ω k = ag max ω T ω j =0, j =1,,k 1 g(ω, μ) (20)

5 Let ω = Σ 1 2 α, ˆΣ = Σ 1 2 Σ Σ 1 2, ˆ Σl = Σ 1 2 Σl Σ 1 2 and ˆB = Σ 1 2 B Σ 1 2. Then the dscmnant cteon g(ω, μ) can be expessed as ĝ(α, μ) = αt ˆBα + μ αt ( ˆΣ ˆ Σl )α Hence, solvng the optmzaton poblem n (20) s equvalent to solvng the followng optmzaton poblem: α 1 = ag max ĝ(α, μ) α α k = ag max α T U k 1 = 0 ĝ(α, μ) (21) whee U k 1 =[ Σ 1 α 1, Σ 1 α 2,, Σ 1 α k 1 ] Soluton method Let S =[s ] be a C C K tenso whose elements s {+1, 1}. Fo smplcty, we call any matx lke S as a sgn tenso. Now we defne a matx T(S,μ) assocated wth S and μ as: T(S,μ) = ˆB + μ s ( ˆΣ ˆ Σl ) Let Ω = {S S =[s ],s {+1, 1}} denote the sgn tenso set. Then we obtan that T(S,μ)α ĝ(α, μ) = max S Ω {αt α T } (22) α Fom (21) and (22), we obtan that the optmal vectos α ( =1, 2,,k) can be expessed as α 1 = ag max α k = ag max max S Ω α max S Ω α T U k 1 =0 α T T(S,μ)α α T T(S,μ)α Solvng the fst dscmnant vecto α 1 (23) To solve the fst vecto α 1 n (23), t s cucal to fnd the optmal sgn tenso S assocated wth α 1. If the sgn tenso S s fxed, then the optmal dscmnant vecto s the egenvecto assocated wth the lagest egenvalue of the matx T(S,μ). Solvng the pncpal egenvecto of T(S,μ) can be effcently ealzed va the powe teaton appoach [5]. Necessay Condton 1: Suppose that S s the optmal sgn tenso and α s the assocated pncpal egenvecto of T(S,μ). Lets be the element of S n the l poston. Then the followng holds because the sgn of both s and α T ( ˆΣ ˆ Σl )α ae the same: s α T ( ˆΣ ˆ Σl )α 0. Defnton 1: Let S 1 and S 2 be two sgn tensos and α 1 and α 2 ae the assocated pncpal egenvecto of T(S 1,μ) and T(S 2,μ), espectvely. If α T 1 T(S 1,μ)α 1 > α T 2 T(S 2,μ)α 2, then we say that S 1 s bette than S 2. Suppose that S 1 s the sgn tenso assocated wth the optmal vecto α 1. Then fom defnton 1 and (23), we obtan that S 1 s bette than any sgn tenso S Ω. Consequently, fndng the optmal sgn tenso S 1 s the pocess of fndng the best sgn tenso n Ω. In what follows, we popose a geedy appoach to fnd the suboptmal sgn tenso S and then fnd a suboptmal vecto α 1. The basc dea of ths appoach s to fx a pope value (+1 o 1) fo each element of S. We show ou appoach as the followng steps: 1. Set an ntal value fo S,e.g,s 1; 2. Solve the pncpal egenvecto of T(s,μ)α = λα va powe teaton method, and set λ 0 λ; 3. Fo = 1, 2,,C, l = 1, 2,,C, and = 1, 2,,K Do Set s s ; Solve the pncpal egenvecto of T(S,μ)α = λα va powe teaton method, and set λ 1 λ; If λ 1 λ 0, then s s,elseλ 0 λ 1 ; Afte pefomng the above steps, we can obtan a bette sgn tenso S. Moeove, we can epeat steps 2 to 3 untl the sgn paamete vecto S conveges Solvng the (k +1)-th dscmnant vecto α k+1 Suppose that we have obtaned the fst k vectos α 1,,α k. To solve the (k +1)-th vecto α k+1, we ntoduce the followng lemma and theoems [12]. Lemma 1. Let Q be a d p (p <d) matx wth othonomal columns. If α T Q = 0, then thee exsts a (non-unque) β IR d such that α =(I QQ T )β. Poof: We can fnd the complement bass Q such that the matx Q =(Q Q ) s an othogonal matx. Then we have Q (Q ) T = I d QQ T due to Q Q T = I d. Fom α T Q = 0, thee exsts a γ IR d p such that α = Q γ. On the othe hand, ank{(q ) T } = d p. Theefoe, the columns of (Q ) T fom a bass of IR d p. So thee exsts a β IR d, such that γ =(Q ) T β. Thus, we have α = Q (Q ) T β =(I d QQ T )β. Theoem 1. Let Q R be the QR decomposton of U, whee R s an uppe tangula matx. Then α +1 defned n (23) s the pncpal egenvecto coespondng to the lagest egenvalue of the followng matx (I d Q Q T )T(S,μ)(I d Q k Q T ). Poof: Snce α T U = 0, Q R s the QR decomposton of U, and R s non-sngula, we obtan that

6 α T Q = 0. Fom Lemma 1, thee exsts a β IR d such that α = (I Q Q T )β. Moeove, t should be noted that (I Q Q T )(I Q Q T ) = I Q Q T. Thus, we have α = (I Q Q T )β = (I Q Q T )α. Hence, we obtan that max αt U αt T(S,μ)α =0 = α max T (I Q Q T )T(S,μ)(I QQT )α α. Theefoe, α +1 s the pncpal egenvecto coespondng to the lagest egenvalue of the matx (I Q Q T )T(S,μ)(I Q Q T ). Theoem 2. Suppose that Q R s the QR decomposton of U. Let U +1 = ((U α +1 )), q = α +1 Q (Q T q α +1 ), and Q +1 = Q q. ( ) R Q Then Q T α s the QR decomposton 0 q of U +1. Poof: Fom q = α +1 Q (Q T α +1 ) and the fact that Q R be the QR decomposton of U, we obtan that Q T q = 0 and Q T Q = I, whee I s the dentty matx. Thus, we have ( ) Q T Q T Q Q T q q +1Q +1 = = I +1 (24) q T Q q 1 whee I +1 s the ( +1) ( +1)dentty matx. On the othe hand, ( )( ) q R Q Q T α +1 =(Q q 0 q R α +1 ) = (U α +1 )=U +1. (25) Fom (24) and (25), one can see that the theoem s tue. Based on the above two theoems, we can solve (23) n an effcent way: Theoem 1 makes t possble to use the powe method to solve (23) and Theoem 2 makes t possble to update Q +1 fom Q by addng a sngle column. Moeove, we have I d Q Q T =(I d Q 1 Q T 1)(I d q q T ) (26) whee q s the th column of Q. Equaton (26) makes t possble to use the ROU technque fo fast updatng the postve semdefnte matx (I d Q Q T )T(S +1,μ)(I d Q Q T ) fom (I d Q 1 Q T 1)T(S +1,μ)(I d Q 1 Q T 1). We gve the pseudo-code of solvng these k dscmnant vectos of the poposed method n Algothm Expements In ode to valdate ou algothm, we conduct facal expesson ecognton expements wth the mult-vew face mages of 100 subjects that we have geneated fom the BU- 3DFE database, as descbed n Secton 2. We compae ou algothm wth PCA [4] and Fshe s LDA [1] based on the Algothm 1: Solvng the optmal vectos ω ( = 1, 2,,k) Input: Data matx X, class label vecto L, and paamete μ. Intalzaton: 1. Compute the covaance matces Σ and Σ ( = 1,,C and =1,,K), Σ, and B; 2. Pefom SVD of Σ: Σ = UΛU T, compute Σ 1 2 = UΛ 1 2 U T and Σ 1 = UΛ 1 U T ; 3. Compute ˆΣ = Σ 1 2 Σ Σ 1 2 ( =1,,C; = 1,,K) and ˆB = Σ 1 2 B Σ 1 2, and let C = max(l); Fo =1, 2,,k,Do 1. Set S ones(c, C, K), S 1 S, and T(S,μ) ˆB + μ s ( ˆΣ ˆ Σl ); 2. Solve the pncpal egenvecto of T(S,μ)α = λα va powe teaton method, and set λ 0 λ; Whle S S 1,Do (a) Set S 1 S; (b) Fo, l =1, 2,,C; =1,,K,Do Set s s and T(S,μ) ˆB + μ s ( ˆΣ ˆ Σl ); Solve the pncpal egenvecto of T(S,μ)α = λα va powe teaton method, and set λ 1 λ; If λ 1 λ 0, then s s,elseλ 0 λ 1 ; (c) Compute T(S,μ) = ˆB + μ s ( ˆΣ ˆ Σl ) and solve the pncpal egenvecto of T(S,μ)α = λα va powe teaton method; 3. If =1, q α, q q / q, and Q 1 q ; else q α Q 1 (Q T 1 α ), q q / q, and Q (Q 1 q ); 4. Compute ˆ Σp ˆ Σp ( ˆ Σp q )q T q (q T ˆ Σp )+ q (q T ˆ Σp q )q T (p =1,,C); 5. Compute ˆ Σpq ˆ Σpq ( ˆ Σpq q )q T q (q T ˆ Σpq )+ q (q T ˆ Σpq q )q T (p =1,,C; q =1,,K ); 6. Compute ˆB ˆB ˆBq q T q (q T ˆB) + q (q T ˆBq )q T Output: ω = α T Σ 1 2 Σ 1 α α, =1, 2,,k.

7 same K-Neaest Neghbo (KNN) classfe. Moe specfcally, ou expements ae caed out as follows: Fo each expement, we un 10 ndependent tals. In each tal, we andomly patton the 100 subjects nto two goups. One goup contans face mages of 80 subjects of all 6 expessons, 4 ntenstes, and 5 vews and compses a tanng set of 9600 face mages. The othe goup contans faces mages of 20 subjects of all 6 expessons, 4 ntenstes, and 5 vews and compses a test set of 2400 face mages. Each tal of an expement nvolves a dffeent andom patton of the 100 subjects nto a tanng set and a test set, and the esults of the 10 ndependent tals ae aveaged. Table 1 shows the oveall eo ates of the thee methods and Fg. 4 shows the oveall confuson matx of ecognzng the sx expessons usng ou method, whee the oveall eo ate of each ecognton method s obtaned by aveagng all the eo ates tested on all the vews and expessons of testng mages. Fom Table 1, we can see that the lowest oveall eo ate s acheved as low as 21.65% by usng ou method, whch s much bette than the esult obtaned by Hu et al. [6] (=33.5%). Fom Fg. 4 we can see that, among the sx expessons, happy expesson and supse expesson ae ease to be ecognzed wheeas fea expesson and angy expesson ae moe dffcult to be ecognzed. Table 1. The oveall eo ates (%) of vaous methods. Ou method Fshe s LDA PCA Eo ate (%) To compae the ecognton pefomance wth espect to dffeent facal vews, we plot the aveage eo ate vesus the dffeent facal vews usng the thee methods n Fg. 5. Fom Fg. 5 we can see that, fo each facal vew, the aveage eo ate of ou method s lowe than the othe two methods. We can also see fom Fg. 5 that, among the vaous vews, the best ecognton pefomance s acheved when the facal vews ae between 30 o and 60 o. Moeove, Aveage eo ates (%) Vew n 0 o Vew n 30 o Vew n 45 o Vew n 60 o Vew n 90 o Ou method LDA PCA Methods Fgue 5. The aveage eo ates of sx expessons wth the dffeent choces of vews usng the dffeent ecognton methods. we also show the aveage eo ate (%) of dffeent emotons vesus dffeent vews usng ou method n Table 2, fom whch we can see the dffeent ecognton esults of each expesson wth the change of the vews and the best esults ae acheved when the facal vews ae between 30 o and 60 o. Table 2. Aveage eo ate (%) of dffeent emotons vesus dffeent vews usng ou method HA SA AN FE SU DI Fgue 4. The oveall confuson matx of ou method. HA SA AN FE SU DI 0 o 30 o 45 o 60 o 90 o Ave Hap Sad Ang Fea Su Dst Moeove, to compae the ecognton pefomance of the thee methods n ecognzng each expesson acoss all the vews, we plot the aveage eo ate of each expesson acoss all the vews n Fg. 6. FomFg.6 we can see agan that, fo all the thee ecognton methods, the happy expesson and the supse expesson ae the ease expessons to be ecognzed wheeas the fea expesson s the most dffcult expesson to be ecognzed. Fnally, we nvestgate the nfluences of the dmensonalty of the educed featues on the ecognton pefomance

8 Aveage eo ates (%) Happy Sad Angy Fea Supse Dsgust Ou method LDA PCA Methods Fgue 6. The aveage eo ates of each expesson acoss all the vews usng the dffeent ecognton methods. of ou method. Fo ths pupose, we show the oveall ecognton esults vesus the dffeent choces of the numbe of educed featues unde the vaous vews n Fg. 7. It can be cleay seen fom Fg. 7 that the eo ate of ecognzng the expessons n each facal vew decease wth the ncease of the numbe of pojecton featues untl the numbe of pojecton featues each 25 o so. Afte that, the eo ates become nsensble to the ncease of the pojecton featues. Aveage eo ate (%) Vew n 0 o Vew n 30 o Vew n 45 o Vew n 60 o Vew n 90 o Oveall Dmenson Fgue 7. Aveage eo ates vesus the numbe of featues selected by ou method. 5. Conclusons In ths pape we developed a novel theoy of mult-class classfcaton, based on mnmzng an estmated closedfom Bayes eo fo an mpotant, dffcult, and much less studed poblem,.e., the non-fontal facal expesson ecognton. The extensve expemental on the BU-3DFE database showed that the non-fontal vew face mages can acheve bette ecognton ates than the fontal vew face mages, especally when the facal vews fell nto the egon between 30 o and 60 o, whee the lowest eo ate (= 21.65%) s obtaned. The majo lmtaton of the poposed method s that the 83 gound tuth ponts we used as the key ponts fo SIFT featue extacton was povded by the BU-3DFE database athe than automatcally located by compute. Fo ou futue wok, we consde to use the eal-tme facal pont detectng and tackng technques to solve ths poblem. Acknowledgment Ths wok was patly suppoted by Natonal Natual Scence Foundaton of Chna unde Gants and , and patly suppoted by the US Govenment VACE pogam. Refeences [1] P. Belhumeu, J. Hespanha, and D. Kegman. Egenfaces vs. fshefaces: Recognton usng class specfc lnea pojecton. IEEE TPAMI, 19(7): , [2] J. T. Chu and J. C. Chuen. Eo pobablty n decson functons fo chaacte ecognton. Jounal of the Assocaton fo Computng Machney, 14(2): , , 3 [3] B. Fasel and J. Luettn. Automatc facal expesson analyss: asuvey.patten Recognton, 36(1): , [4] K. Fukunaga. Intoducton to Statstcal Patten Recognton (Second Edton). Academc Pess, New Yok. 2, 6 [5] G.Golub and C.Van. Matx Computatons. The Johns Hopkns Unvesty Pess, , 5 [6] Y. Hu, Z. Zeng, L. Yn, X. We, J. Tu, and T. Huang. A study of non-fontal-vew facal expessons ecognton. In Poceedngs of ICPR, pages 1 4, , 7 [7] D. G. Lowe. Dstnctve mage featues fom scale-nvaant keyponts. IJCV, 60(2):91 110, [8] Y. L. Tan, T. Kanade, and J. F. Cohn. Facal expesson analyss. In S. Z. L, A. K. Jan (Eds.), Handbook of Facal Recognton, pages , New Yok, USA, Spnge. 1 [9] J. Wang, L. Yn, X. We, and Y. Sun. 3d facal expesson ecognton based on pmtve suface featue dstbuton. In Poceedngs of CVPR, pages , [10] L. Yn, X. We, Y. Sun, J. Wang, and M. Rosato. A 3d facal expesson database fo facal behavo eseach. In Poceedngs of 7th Intenatonal Confeence on Automatc Face and Gestue Recognton, pages , , 2 [11] Z. Zeng, M. Pantc, G. Rosman, and T. Huang. A suvey of affect ecognton methods: Audo, vsual, and spontaneous expessons. IEEE TPAMI, 31(1):39 58, [12] W. Zheng. Heteoscedastc featue extacton fo textue classfcaton. IEEE Sgnal Pocessng Lettes, 16(9): ,