Principal Component Analysis (PCA) The Gaussian in D dimensions

Transcription

1 Prinipal Component Analysis (PCA) im ars, Cognitie Siene epartment he Gassian in dimensions What does a set of eqiprobable points loo lie for a Gassian? In, it s an ellipse. In dimensions, it s an ellipsoid. im ars, Cognitie Siene epartment

2 Eqiprobable ontors of a Gassian If a Gassian random etor has oariane matrix that is diagonal (all of the ariables are norrelated) hen the axes of the ellipsoid are parallel to the oordinate axes. im ars, Cognitie Siene epartment Eqiprobable ontors of a Gassian If a Gassian random etor has oariane matrix that is not diagonal (some of the ariables are orrelated) hen the axes of the ellipsoid are perpendilar to eah other, bt are not parallel to the oordinate axes. im ars, Cognitie Siene epartment

3 Priniple Component Analysis Priniple omponent analysis (PCA) finds the diretions of the axes of the ellipsoid. here are two ways to thin abot what PCA does next: Projets eery point perpendilarly onto the axes of the ellipsoid. Rotates the ellipsoid so its axes are parallel to the oordinate axes, and translates the ellipsoid so its enter is at the origin. im ars, Cognitie Siene epartment y PC x PC im ars, Cognitie Siene epartment wo iews of what PCA does ransforms x y Projets eery point perpendilarly onto the axes of the ellipsoid. Rotates spae so that the ellipsoid lines p with the oordinate axes, and translates spae so the ellipsoid s enter is at the origin. x 3

4 4 im ars, Cognitie Siene epartment ransformation matrix for PCA Let be the matrix whose olmns are the eigenetors of the oariane matrix, S. he eigenetors i are all normalied to hae length he rotation transformation is gien by L im ars, Cognitie Siene epartment ransformation matrix for PCA PCA transforms the point x (original oordinates) into the point (new oordinates). by sbtrating the mean: x m and mltiplying the reslt by Can thin of as rotation bease is an orthonormal matrix Can thin of as projetion of onto PC axes, bease is projetion of onto PC axis, is projetion onto PC axis, et.

5 5 im ars, Cognitie Siene epartment Point is a weighted sm of eigenetors o both sides of the eqation, mltiply on the left by :. Bease is orthonormal, I : I PCA expresses the mean-sbtrated point, x m, as a weighted sm of the eigenetors i : L L im ars, Cognitie Siene epartment Eigenales and ariane he eigenetors,,, n of the oariane matrix hae orresponding eigenales λ, λ,, λ n. It trns ot that λ is the ariane of the distribtion in the diretion, λ is the ariane of the distribtion in the diretion, and so on. he largest eigenale orresponds to the prinipal omponent in the diretion of greatest ariane, the next largest eigenale orresponds to the prinipal omponent in the perpendilar diretion of next greatest ariane, et. Whih eigenetor (green or red) orresponds to the smaller eigenale?

6 6 im ars, Cognitie Siene epartment x y PC PC height weight What if yo wanted to transmit someone s height and weight, bt yo old only gie a single nmber? Cold gie only height, x (nertainty when height is nown) Cold gie only weight, y (nertainty when weight is nown) Cold gie only, the ale of first PC (nertainty when first PC is nown) Giing the first PC minimies the sqared error of the reslt. o ompress -dimensional data into dimensions, order the prinipal omponents in order of largest-to-smallest eigenale, and only sae the first omponents. PCA for data ompression im ars, Cognitie Siene epartment PCA for data ompression Eqialent iew: o ompress -dimensional data into dimensions, order the eigenetors in order of largest-to-smallest eigenale, and only se the first eigenetors. PCA approximates the mean-sbtrated point, x m, as a weighted sm of the first eigenetors: L L

7 Fae spae 0 00 pixel graysale images of faes Eah image is onsidered to be a single point in fae spae (a single sample from a random etor): How many dimensions is the etor x? x Eah dimension (eah pixel) an tae real ales between 0 and 55. Yo an isalie in 000 dimensions! im ars, Cognitie Siene epartment PCA on fae images Let x, x,, x n be the n sample images. Find the mean image, m, and sbtrat it from eah image: i x i m Let A be the matrix whose olmns are the mean-sbtrated sample images. A L n 000 n matrix Estimate the oariane matrix: Σ Co( A ) A A n What are the dimensions of Σ? im ars, Cognitie Siene epartment 7

8 he transpose tri he eigenetors of Σ are the prinipal omponent diretions Σ is a matrix oo big for s to find its eigenetors nmerially he first n eigenales are sefl; the rest will all be ero. Use a tri. Σ Co( A) A A Eigenetors of Σ are eigenetors of AA ; still Instead of eigenetors of AA, we find eigenetors of A A What are the dimensions of A A? n n matrix is easy to find eigenetors of, sine n (the nmber of sample images) is relatiely small. im ars, Cognitie Siene epartment n he transpose tri We want the eigenetors of AA, bt instead we allated the eigenetors of A A. Now what? Let i be an eigenetor of A A, whose eigenale is λ i (A A) i?λ i i A (A A i ) A(λ i i ) AA (A i ) λ i (A i ) hs if i is an eigenetor of A A, A i is an eigenetor of AA, with the same eigenale. A, A,, A n are the eigenetors of Σ. A, A,, n A n are alled eigenfaes. im ars, Cognitie Siene epartment 8

9 he original images ( ot of 97) im ars, Cognitie Siene epartment he mean fae im ars, Cognitie Siene epartment 9

10 Eigenfaes Eigenfaes (the prinipal omponents of fae spae) proide a low-dimensional representation of any fae, whih an be sed for: Fae reognition Faial expression reognition Image reonstrtion im ars, Cognitie Siene epartment he first 8 eigenfaes im ars, Cognitie Siene epartment 0

11 im ars, Cognitie Siene epartment PCA for fae representation o approximate a fae sing dimensions, order the eigenfaes in order of largest-to-smallest eigenale, and only se the first eigenfaes. PCA approximates a mean-sbtrated fae, x m, as a weighted sm of the first eigenfaes: U L L U im ars, Cognitie Siene epartment Approximating a fae sing the first 0 eigenfaes

12 Approximating a fae sing the first 0 eigenfaes im ars, Cognitie Siene epartment Approximating a fae sing the first 40 eigenfaes im ars, Cognitie Siene epartment

13 Approximating a fae sing all 97 eigenfaes Sine the image was in the original training set, the reonstrtion sing all 97 eigenfaes is exat. For a new fae image, howeer, its projetion into fae spae will only approximately math the original. im ars, Cognitie Siene epartment What reglarities does eah eigenfae aptre? <atlab moie> im ars, Cognitie Siene epartment 3

14 Eigenfaes in 3 Blan and etter (SIGGRAPH 99) <mpeg ideo> he ideo an be fond at: im ars, Cognitie Siene epartment 4