Eigenimages " Unitary transforms" Karhunen-Loève transform" Eigenimages for recognition" Sirovich and Kirby method" Example: eigenfaces" Eigenfaces vs. Fisherfaces" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 1
Unitary transforms Sort samples f [x,y] of an MxN image (or a rectangular portion of the image) into column vector of length MN" Compute transform coefficients" " " " " where A is a matrix of size MNxMN The transform A is unitary, iff" c = Af A 1 = A *T A H Hermitian conjugate" If A is real-valued, i.e., A=A*, transform is orthonormal " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 2
Energy conservation with unitary transforms c = A f For any unitary transform we obtain" c 2 = c H c = f H A H A f = f 2 Interpretation: every unitary transform is simply a rotation of the coordinate system (and, possibly, sign flips)" Vector length ( energy ) is conserved." Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 3
Energy distribution for unitary transforms Energy is conserved, but, in general, unevenly distributed among coefficients." Autocorrelation matrix" R cc = E c c H = E A f f H A H = AR ff AH Mean squared values ( average energies ) of the coefficients c i live on the diagonal of R cc " 2 E c i = R cc i,i = AR ff A H i,i Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 4
Eigenmatrix of the autocorrelation matrix Definition: eigenmatrix Φ of autocorrelation matrix R ff " l Φ is unitary" l The columns of Φ form a set of eigenvectors of R ff, i.e., " " " R ff Φ = ΦΛ Λ is a diagonal matrix of eigenvalues λ i " Λ = λ 0 0 l R ff is symmetric nonnegative definite, hence" λ i 0 for all i l R ff is normal matrix, i.e., R H H ff R ff = R ff R ff, hence unitary eigenmatrix exists ( spectral theorem )" λ 1 0 λ MN 1 Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 5
Karhunen-Loève transform Unitary transform with matrix" A = Φ H "where the columns of Φ are ordered according to decreasing eigenvalues." Transform coefficients are pairwise uncorrelated" Energy concentration property: " R cc = AR ff A H = Φ H R ff Φ = Φ H ΦΛ = Λ l Mean squared approximation error by choosing only first J coefficients is minimized. " l No other unitary transform packs as much energy into the first J coefficients, for any J" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 6
Illustration of energy concentration Strongly correlated samples, equal energies" f 2 f 1 A = cosθ sinθ sinθ cosθ c 2 c 1 After KLT: uncorrelated samples, most of the energy in first coefficient" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 7
Basis images and eigenimages For any unitary transform, the inverse transform" f = A H c "can be interpreted in terms of the superposition of basis images (columns of A H ) of size MN." If the transform is a KL transform, the basis images (aka eigenvectors of the autocorrelation matrix R ff ) are called eigenimages. " If energy concentration works well, only a limited number of eigenimages is needed to approximate a set of images with small error. These eigenimages form an optimal linear subspace of dimensionality J." Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 8
Eigenimages for recognition To recognize complex patterns (e.g., faces), large portions of an image (say of size MN) might have to be considered" High dimensionality of image space means high computational burden for many recognition techniques" Example: nearest-neigbor search requires pairwise comparison with every image in a database" Transform can reduce dimensionality from MN to J by representing the c = Wf image by J coefficients " Idea: tailor a KLT to the specific set of images of the recognition task to preserve the salient features" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 9
Eigenimages for recognition (cont.) [Ruiz-del-Solar and Navarrete, 2005]! Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 10
Computing eigenimages from a training set How to measure MNxMN covariance matrix?" l Use training set Γ 1, Γ 2,, Γ L (each column vector represents one image)" l Let be the mean image of all samples" l " Problem 1: Training set size should be L >> MN" "" If L < MN, covariance matrix R is rank-deficient Problem 2: Finding eigenvectors of an MNxMN matrix." " µ Define training set matrix S = ( Γ 1 µ, Γ 2 µ, Γ 3 µ,, Γ L µ ), L and calculate R = Γ l µ Γ l µ l=1 ( )( ) H = SS H Can we find a small set of the most important eigenimages from a small training set L << MN? " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 11
Sirovich and Kirby method " Instead of eigenvectors of SS H, consider the eigenvectors of S H S, i.e., Premultiply both sides by S S H S v i = λ i vi SS H S v i = λ i S v i By inspection, we find that S v i are eigenvectors of SS H For L << MN this gives rise to great computational savings, by " l Computing the LxL matrix S H S " l Computing L eigenvectors of S H S " l Computing eigenimages corresponding to the L 0 L largest eigenvalues according as" " S v i v i L. Sirovich and M. Kirby, "Low-dimensional procedure for the characterization of human faces," " Journal of the Optical Society of America A, 4(3), pp. 519-524, 1987." Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 12
Example: eigenfaces The first 8 eigenfaces obtained from a training set of 180 male and 180 female training images (without mean removal)" Can be used to generate faces by adjusting 8 coefficients." A. Diaco, J. DiCarlo and J. Santos, EE368 class project, Spring 2000." Can be used for face recognition by nearest neighbor search in 8-d face space. " Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 13
Gender recognition using eigenfaces " Recognition accuracy using 8 eigenimages" Female face samples" A. Diaco, J. DiCarlo and J. Santos, EE368 class project, Spring 2000." Male face samples" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 14
Fisher linear discriminant analysis " Eigenimage method maximizes scatter within the linear subspace over the entire image set regardless of classification task" W opt = argmax W ( det( WRW H )) Fisher linear discriminant analysis (1936): maximize between-class scatter, while minimizing within-class scatter W opt = argmax W ( ) ( ) det WR B W H det WR W W H R B = Samples in class i R W = c c i=1 i=1 Γ l N ( i µ ) i µ i Class(i) µ ( ) H µ i µ Mean in class i ( Γ ) l Γ l µ i ( ) H Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 15
Fisher linear discriminant analysis (cont.) " Solution: Generalized eigenvectors w i K largest eigenvalues { λ i i = 1,2,..., K}, i.e. R B w i corresponding to the = λ i R W w i, i = 1,2,..., K Problem: within-class scatter matrix R w at most of rank L-c, hence usually singular." Apply KLT first to reduce dimension of feature space to L-c (or less), proceed with Fisher LDA in low-dimensional space Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 16
Eigenimages vs. Fisherimages 2d example:" Samples for 2 classes are projected onto 1d subspace using the" KLT (aka PCA) or" Fisher LDA (FLD)." PCA preserves maximum energy, but the 2 classes are no longer distinguishable. FLD separates the classes by choosing a better 1d subspace." " +" +" +" +" +" +" +" +" +" +" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 17 [Belhumeur, Hespanha, Kriegman, 1997]!
Fisherimages and varying iillumination Differences due to varying illumination can be much larger than differences between faces!" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 18 [Belhumeur, Hespanha, Kriegman, 1997]!
Fisherimages and varying iillumination All images of same Lambertian surface with different illumination (without shadows) lie in a 3d linear subspace" Single point source at infinity" surface normal n l light source direction f ( x, y) = a( x, y) l T n( x, y) Surface albedo" Light source intensity" Superposition of arbitrary number of point sources at infinity still in same 3d linear subspace, due to linear superposition of each contribution to image " Fisherimages can eliminate within-class scatter" ( ) L Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 19
Face recognition with Eigenfaces and Fisherfaces FERET data base," 254 classes," 3 images per class" [Belhumeur, Hespanha, Kriegman, 1997]! Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 20
Fisherface trained to recognize glasses [Belhumeur, Hespanha, Kriegman, 1997]! Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 21
Fisherface trained to recognize gender Female face samples" Mean image" Female mean" Male mean" µ µ 1 µ 2 Male face samples" Fisherface" Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 22
Gender recognition using fisherface 0.14 0.12 EER = 6.5%! Male Female 0.1 Probability 0.08 0.06 0.04 0.02 0-60 -40-20 0 20 40 60 80 100 Projection Score Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 23
Gender recognition using 2 nd eigenface 0.14 0.12 EER = 24%! Male Female 0.1 Probability 0.08 0.06 0.04 0.02 0-30 -20-10 0 10 20 30 Projection Score Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 24