Singular Value Decomposition CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Singular Value Decomposition 1 / 35
Understanding the Geometry of A R m n Critical points of the ratio: R( v) = A v 2 v 2 CS 205A: Mathematical Methods Singular Value Decomposition 2 / 35
Understanding the Geometry of A R m n Critical points of the ratio: R( v) = A v 2 v 2 R(α v) = R( v) = take v 2 = 1 R( v) 0 = study R 2 ( v) instead CS 205A: Mathematical Methods Singular Value Decomposition 2 / 35
Once Again... Critical points satisfy A A v i = λ i v i. CS 205A: Mathematical Methods Singular Value Decomposition 3 / 35
Once Again... Critical points satisfy A A v i = λ i v i. Properties: λ i 0 i Basis is full and orthonormal CS 205A: Mathematical Methods Singular Value Decomposition 3 / 35
Geometric Question What about A instead of A A? CS 205A: Mathematical Methods Singular Value Decomposition 4 / 35
Geometric Question What about A instead of A A? Object of study: u i A v i CS 205A: Mathematical Methods Singular Value Decomposition 4 / 35
Observation Lemma Either u i = 0 or u i is an eigenvector of AA with u i 2 = λ i v i 2. CS 205A: Mathematical Methods Singular Value Decomposition 5 / 35
Corresponding Eigenvalues k = number of λ i > 0 A A v i = λ i v i AA u i = λ i u i Ū R n k = matrix of normalized u i s V R m k = matrix of normalized v i s CS 205A: Mathematical Methods Singular Value Decomposition 6 / 35
Corresponding Eigenvalues k = number of λ i > 0 A A v i = λ i v i AA u i = λ i u i Ū R n k = matrix of normalized u i s V R m k = matrix of normalized v i s Lemma Ū A V e i = λ i e i CS 205A: Mathematical Methods Singular Value Decomposition 6 / 35
Lemma Ū A V e i = λ i e i Observation Σ diag( λ 1,..., λ k ) CS 205A: Mathematical Methods Singular Value Decomposition 7 / 35
Lemma Ū A V e i = λ i e i Observation Σ diag( λ 1,..., λ k ) Corollary Ū A V = Σ CS 205A: Mathematical Methods Singular Value Decomposition 7 / 35
Completing the Basis Add v i with A A v i = 0 and u i with AA u i = 0 CS 205A: Mathematical Methods Singular Value Decomposition 8 / 35
Completing the Basis Add v i with A A v i = 0 and u i with AA u i = 0 Ū R m k, V R n k U R m m, V R n n CS 205A: Mathematical Methods Singular Value Decomposition 8 / 35
Completing the Basis Add v i with A A v i = 0 and u i with AA u i = 0 Ū R m k, V R n k U R m m, V R n n Σ ij { λi i = j and i k 0 otherwise CS 205A: Mathematical Methods Singular Value Decomposition 8 / 35
Singular Value Decomposition A = UΣV CS 205A: Mathematical Methods Singular Value Decomposition 9 / 35
Geometry of Linear Transformations A = UΣV 1. Rotate (V ) 2. Scale (Σ) 3. Rotate (U) CS 205A: Mathematical Methods Singular Value Decomposition 10 / 35
SVD Vocabulary A = UΣV Left singular vectors: Columns of U; span col A Right singular vectors: Columns of V ; span row A Singular values: Diagonal σ i of Σ; sort σ 1 σ 2 0 CS 205A: Mathematical Methods Singular Value Decomposition 11 / 35
Computing SVD: Simple Strategy 1. Columns of V are eigenvectors of A A 2. AV = UΣ = columns of U corresponding to nonzero singular values are normalized columns of AV 3. Remaining columns of U satisfy AA u i = 0. CS 205A: Mathematical Methods Singular Value Decomposition 12 / 35
Computing SVD: Simple Strategy 1. Columns of V are eigenvectors of A A 2. AV = UΣ = columns of U corresponding to nonzero singular values are normalized columns of AV 3. Remaining columns of U satisfy AA u i = 0. more specialized methods! CS 205A: Mathematical Methods Singular Value Decomposition 12 / 35
Solving Linear Systems with A = UΣV A x = b = UΣV x = b = x = V Σ 1 U b CS 205A: Mathematical Methods Singular Value Decomposition 13 / 35
Solving Linear Systems with A = UΣV A x = b = UΣV x = b = x = V Σ 1 U b What is Σ 1? CS 205A: Mathematical Methods Singular Value Decomposition 13 / 35
Uniting Short/Tall Matrices minimize x 2 2 such that A A x = A b CS 205A: Mathematical Methods Singular Value Decomposition 14 / 35
Simplification A A = V Σ ΣV CS 205A: Mathematical Methods Singular Value Decomposition 15 / 35
Simplification A A = V Σ ΣV A A x = A b Σ Σ y = Σ d y V x d U b CS 205A: Mathematical Methods Singular Value Decomposition 15 / 35
Resulting Optimization minimize y 2 2 such that Σ Σ y = Σ d CS 205A: Mathematical Methods Singular Value Decomposition 16 / 35
Solution { 1 Σ + ij /σ i i = j, σ i 0, and i k 0 otherwise = y = Σ + d = x = V Σ + U b CS 205A: Mathematical Methods Singular Value Decomposition 17 / 35
Pseudoinverse A + = V Σ + U CS 205A: Mathematical Methods Singular Value Decomposition 18 / 35
Pseudoinverse Properties A square and invertible = A + = A 1 A overdetermined = A + b gives least-squares solution to A x b A underdetermined = A + b gives least-squares solution to A x b with least (Euclidean) norm CS 205A: Mathematical Methods Singular Value Decomposition 19 / 35
Alternative Form l A = UΣV = A = σ i u i v i l min{m, n} i=1 CS 205A: Mathematical Methods Singular Value Decomposition 20 / 35
Outer Product u v u v CS 205A: Mathematical Methods Singular Value Decomposition 21 / 35
Computing A x A x = i σ i ( v i x) u i CS 205A: Mathematical Methods Singular Value Decomposition 22 / 35
Computing A x A x = i σ i ( v i x) u i Trick: Ignore small σ i. CS 205A: Mathematical Methods Singular Value Decomposition 22 / 35
Computing A + x A + = σ i 0 v i u i σ i Trick: Ignore large σ i. CS 205A: Mathematical Methods Singular Value Decomposition 23 / 35
Even Better Trick Do not compute large (small) σ i at all! CS 205A: Mathematical Methods Singular Value Decomposition 24 / 35
Eckart-Young Theorem Theorem Suppose à is obtained from A = UΣV by truncating all but the k largest singular values σ i of A to zero. Then, à minimizes both A à Fro and A à 2 subject to the constraint that the column space of à has at most dimension k. CS 205A: Mathematical Methods Singular Value Decomposition 25 / 35
Matrix Norm Expressions A 2 Fro = σ 2 i A 2 = max{σ i } cond A = σ max/σ min CS 205A: Mathematical Methods Singular Value Decomposition 26 / 35
Rigid Alignment CS 205A: Mathematical Methods Singular Value Decomposition 27 / 35
Variational Formulation min R R=I 3 3 t R 3 Given x 1i x 2i R x 1i + t x 2i 2 2 i CS 205A: Mathematical Methods Singular Value Decomposition 28 / 35
Variational Formulation min R R=I 3 3 t R 3 Given x 1i x 2i R x 1i + t x 2i 2 2 i Alternate: 1. Minimize with respect to t: Least-squares 2. Minimize with respect to R: SVD CS 205A: Mathematical Methods Singular Value Decomposition 28 / 35
Procrustes via SVD min R R=I 3 3 RX 1 X t 2 2 Fro CS 205A: Mathematical Methods Singular Value Decomposition 29 / 35
Procrustes via SVD min R R=I 3 3 RX 1 X t 2 2 Fro Orthogonal Procrustes Theorem The orthogonal matrix R minimizing RX Y 2 is given by UV, where SVD is applied to factor Y X = UΣV. CS 205A: Mathematical Methods Singular Value Decomposition 29 / 35
Application: As-Rigid-As-Possible As-Rigid-As-Possible Surface Modeling Olga Sorkine and Marc Alexa Eurographics/ACM SIGGRAPH Symposium on Geometry Processing 2007. http://www.youtube.com/watch?v=ltx-qujbkdc CS 205A: Mathematical Methods Singular Value Decomposition 30 / 35
Related: Polar Decomposition F = R U Special case: F is square real-valued matrix; R is best rotation matrix approximation; U is right symmetric PSD stretch matrix. Proof by SVD. CS 205A: Mathematical Methods Singular Value Decomposition 31 / 35
Recall: Statistics Problem Given: Collection of data points x i Age Weight Blood pressure Heart rate Find: Correlations between different dimensions CS 205A: Mathematical Methods Singular Value Decomposition 32 / 35
Simplest Model One-dimensional subspace x i c i v CS 205A: Mathematical Methods Singular Value Decomposition 33 / 35
More General Statement Principal Component Analysis The matrix C R n d minimizing X CC X Fro subject to C C = I d d is given by the first d columns of U, for X = UΣV. Proved in textbook. CS 205A: Mathematical Methods Singular Value Decomposition 34 / 35
Application: Eigenfaces CS 205A: Mathematical Methods Singular Value Decomposition 35 / 35 Next