Let the Shape Speak - Discriminative Face Alignment using Conjugate Priors

Transcription

1 Let the Shape Speak - Dscrmnatve Face Algnment usng Conjugate Prors Pedro Martns, Ru Casero, João F. Henrques, Jorge Batsta Insttute of Systems and Robotcs Unversty of Combra Portugal Brtsh Machne Vson Conference

2 Demo Vdeo 2

3 Introducton Goal: Face algnment n unseen mages. Closely related to Constraned Local Models (CLM) and Actve Shape Models (ASM), where a set of local detectors s constraned to le n the subspace spanned by a Pont Dstrbuton Model (PDM). Two step fttng approach: (1) Local search usng the local detectors (response maps for each landmark) (2) Global optmzaton strategy that fnds the PDM parameters that jontly maxmze all the detectons. Proposed Work: New Bayesan global optmzaton strategy where the pror dstrbuton encodes the transton of the PDM parameters. The pror dstrbuton s modeled usng recursve Bayesan estmaton. The mean and covarance are assumed to be unknown and treated as random varables. 3

4 Related Work - Parametrc Image Algnment Generatve / Holstc methods Actve Appearance Models (AAM) T.F.Cootes, G.J.Edwards, C.J.Taylor - ECCV 98 3D Morphable Models (3DMM) V.Blanz, T.Vetter - SIGGRAPH 99 Real Tme Combned 2D+3D Actve Appearance Models J.Xng, S.Baker, I.Matthews, T.Kanade - CVPR 2004 Dscrmnatve / Patch-Based Actve Shape Models (ASM) T.F.Cootes, G.J.Edwards, C.J.Taylor - CVIU 95 Pont Dstrbuton Model s = S (s 0 + b; q) Constraned Local Model (CLM) D.Crstnance, T.F.Cootes - BMVC 2006 Convex Quadratc Fttng (CQF) Y.Wang, S.Lucey, J.Cohn - CVPR 2008 Bayesan Constraned Local Model (BCLM) U.Paquet - CVPR 2009 Shape Parameters Pose Parameters Subspace Constraned Mean-Shfts (SCMS) J.Saragh, S.Lucey, J.Cohn - ICCV

5 The Algnment Goal Gven a shape observaton vector (y), fnd the optmal set of shape (and pose) parameters (b) that maxmze the posteror probablty Assumng: b = arg max b p(b y) / vy =1 p(b y) / p(y b)p(b) Condtonal ndependence between landmarks Close to a soluton p(y b)! p(b b k 1) Lkelhood from the local detectors Pror on how parameters change 5

6 The Lkelhood Term Convex energy functon: p(y b) / exp (y (s 0 {z } y Observed shape (y) 1 + b)) T y 1 C (y (s 0 + b)) A Dfference between the observed and the mean shape Uncertanty covarance y... 0 The lkelhood follow a Gaussan dstrbuton p(y b) / N ( y b, y ) 0 2v 2v Block dagonal 6

7 Local Landmark Detectors Lnear SVM (+) Algned Examples (-) Msalgned Examples D lnear (I(y )) = w T I(y )+b =1,...,v landmarks 7

8 Local Detectors 8

9 Local Landmark Detectors - MOSSE Flters Correlaton n Fourer Doman G = F{I} H Cosne Wndow Gaussan mn H NX j=1 (F{I j } H G j ) F 1 {H } D MOSSE (I(y )) = F 1 {F{I(y )} H } D MOSSE (I(y )) H = MOSSE Flter P N j=1 G j P N j=1 F{I j} F{I j } F{I j } Vsual object trackng usng adaptve correlaton flters D.Bolme, J.Beverdge, B.Draper, Y.Lu, CVPR

10 Local Optmzaton Strateges p (z ) Prob. z s algned yc Pxel canddate to th landmark locaton Center Weghted Peak Response (WPR) ywpr = max (p (z )) z 2 yc WPR y ASM = dag(p (ywpr ) 1) yc Local search grd Gaussan Response (GR) ygr Patches under occluson z = (x, y ) 1 X p (z )z = d c z 2 y GR y = 1 d 1 X p (z )(z d= X p (z ) z 2 yc ygr )(z Kernel Densty Estmator (KDE) KDE( +1) y T ygr ) KDE = y SCMS 1 d KDE( ) z 2 yc P z 2 yc CQF, BCLM P 1 z p (z ) N (y KDE( ) z 2 yc X p (z ) N (y p (z )(z ykde )(z z, z, 2 h j I2 ) 2 h j I2 ) ykde )T z 2 yc 10

11 KDE Demo Vdeo 11

12 MAP Global Algnment Lkelhood p(y b) / N ( y b, y ) Bayes s Theorem for Gaussan varables Posteror p(b k y) / N (b k µ, ) Mean that s functon of b Covarance ndependent of b Pror p(b k b k 1 ) / N (b k µ b, b ) p(b k y k,...,y 0 ) / N (b k µ k, k ) =( 1 b + T 1 y ) 1 µ = ( T 1 y y + 1 b µ b) Model the Covarance of b (2 nd Order Estmate) + Bayesan Fuson of Multple Detectors k = ( bk + k 1 ) 1 + T M X µ k = k T MX m=1 1 y (m) y (m) m=1 1 y (m)! 1 +( bk + k 1 ) 1 µ bk! y (m), y(m) Multple Lkelhood Observatons 12

13 The Pror Term Mean and Covarance (µ b, b ) are assumed to be unknown and modeled as random varables. Unknown p(b k b k 1 ) / N (b k µ b, b ) Bayes Theorem: Observable vector b p(µ b, b b) / p(b µ b, b ) p(µ b, b ) Jont Posteror Normal Inverse-Wshart Jont Pror Normal Inverse-Wshart Parameters Degrees of freedom Number of measurements k = k 1 + m apple k = apple k 1 + m Conjugate Pror for a Gaussan wth unknown mean and covarance s a Normal Inverse-Wshart dstrbuton Mean Scale matrx k = apple k 1 apple k 1 + m k 1 + apple k m 1 + m b k = k 1 + apple k 1m apple k 1 + m (b k 1)(b k 1 ) T m - Number of samples to update the model b - Mean of all samples 13

14 The Pror Term Usng the expectaton of margnal posteror dstrbutons as the model parameters update. Expectaton Jont Posteror Normal Inverse-Wshart Margnalzng w.r.t Multvarate p(µ b b) / Student t b µ bk = E(µ b b) = k p(µ b, b b) Margnalzng p( µ b b) /Inv-Wshart bk = E( b b) =( k n 1) 1 k w.r.t b The Pror dstrbuton s contnuously kept up to date 14

15 The Algorthm Precompute: PDM: s 0,,, = dag( 1,..., n) MOSSE Flters: Intal estmate (b 0, 0 ), (q 0, q 0 )... H =1,...,v for k=1:1:maxiteratons Warp Image to the base mesh, usng the current pose parameters Generate current shape s = S (s 0 + b k ; q k ) for =1:1:Landmarks WPR, GR or KDE Local Strategy Evaluate detectors response Fnd the lkelhood parameters y, y... end Estmate the shape/pose parameters: Update the parameters of Normal Inv-Wshart dstrbuton Expectaton of the pror shape parameters k, apple k, k, k µ bk = k, bk =( k n 1) 1 k end Evaluate the global shape parameters and the covarance µ k, k 15

16 Herarchcal Search (BASM-KDE-H) When response maps are approxmated by KDE representatons. y KDE( +1) Mean-Shft Landmark Update Pz 2 yc z p (z ) N (y KDE( ) 2 z, hj I 2 ) Pz 2 yc p (z ) N (y KDE( ) z, 2 hj I 2 ) Bandwdth schedule 2 h = (15, 10, 5, 2) Standard Search for k=1:1:maxiteratons for =1:1:v (LandMarks) Evaluate de Detectors Response Herarchcal Search for k=1:1:maxiteratons for 2 h = (15, 10, 5, 2) for =1:1:v (LandMarks) Evaluate de Detectors Response Mean-Shft Landmark Update 2 h = (15, 10, 5, 2) Mean-Shft Landmark Update 2 h j end end end Global Optmzaton end end Global Optmzaton 16

17 Evaluaton Results 1 IMM Fttng Performance 1 XM2VTS Fttng Performance 1 BoID Fttng Performance Proporton of Images 0.6 AVG ASM CQF GMM3 SCMS BCLM KDE BASM KDE BASM KDE H BASM KDE Fuson RMS Error Proporton of Images 0.6 AVG ASM CQF GMM3 SCMS BCLM KDE BASM KDE BASM KDE H BASM KDE Fuson RMS Error Proporton of Images 0.6 AVG ASM CQF GMM3 SCMS BCLM KDE BASM KDE BASM KDE H BASM KDE Fuson RMS Error Reference 7.5 RMS IMM (240 mages) XM2VTS (2360 mages) BoID (1521 mages) ASM BASM-WPR (our method) 58.4 (+8.4) 47.4 (+16.7) 77.1 (+7.1) CQF GMM (-4.6) 10.4 (-0.5) 51.7 (+4.7) BCLM-GR 48.3 (+2.9) 15.9 (+5.0) 54.2 (+7.2) BASM-GR (our method) 51.8 (+6.4) 19.7 (+8.8) 63.5 (+16.5) SCMS-KDE BCLM-KDE 57.1 (+2.5) 43.4 (+7.7) 71.9 (+2.9) BASM-KDE (our method) 65.4 (+10.8) 57.0 (+21.3) 80.3 (+11.3) BASM-KDE-H (our method) 64.0 (+9.4) 56.6 (+20.9) 79.9 (+10.9) BASM-KDE Fuson of 2 Detectors 72.5 (+17.9) 58.7 (+23.0) 88.2 (+19.2)

18 Trackng Performance FGNET Talkng Face Vdeo Sequence RMS Error Trackng Performance ASM (10.5 / 6.4) CQF (10.6 / 3.9) GMM3 (11.1 / 4.3) SCMS KDE (8.2 / 2.6) BCLM KDE (9.5 / 3.6) BASM KDE (6.4 / 1.7) BASM KDE H (6.3 / 1.5) BASM KDE Fuson (5.9 / 1.5) Frame Number RMS Error ASM CQF GMM3 SCMS-KDE BCLM-KDE BASM-KDE BASM-KDE-H BASM-KDE Fuson Mean Standard Devaton

19 Trackng Performance 19

20 Labeled Faces n the Wld (LFW) 20

21 Conclusons Bayesan formulaton for algnng faces n unseen mages. New global optmzaton strategy nfers both shape and pose parameters, n MAP sense, by explctly modelng the pror dstrbuton. The pror dstrbuton s contnuously kept up to date. Recursve Bayesan estmaton s used to treat the mean and covarance as random varables. Extensve evaluatons were performed on several standard datasets (IMM, XM2VTS, BoID, LFW and FGNET Talkng Face) aganst state-of-the-art methods whle usng the same local detectors. Acknowledgements Work supported by the Portuguese Scence Foundaton (Fundação para a Cênca e Tecnologa - FCT) under the project grant PTDC/EIA-CCO/108791/2008. Pedro Martns, Ru Casero and João F. Henrques also acknowledge the FCT through the grants SFRH/BD/45178/2008, SFRH/BD74152/2010 and SFRH/BD/75459/2010, respectvely. 21

22 Qualtatve Evaluaton - Labeled Faces n the Wld 22