Singular Value Decomposition

Transcription

1 Singular Value Decomposition

2 Motivatation The diagonalization theorem play a part in many interesting applications. Unfortunately not all matrices can be factored as A = PDP However a factorization A = QDP is possible for any m n matrix A. A special factorization of this type, called singular value decomposition, is one of the most useful matrix factorizations in applied linear algebra. As we will see one application is finding minimum value of Ax for solving Ax=0.

3 Singular Value Decomposition Let A be a m n matrix of rank r. We cannot find eigenvalues and eigenvectors for non-square matrices. However, A can be diagonalized in such a way that Av = σ u Av = σ u Av r = σ r u r The singular vectors v v r are orthogonal and are the basis for the row space of A. The output vectors u u r are orthogonal and in column space of A. The singular values σ σ r are all positive numbers in non-increasing order. 3

4 Av i = σ i u i leads to AV = UΣ: A v v r = u u r σ Note that in above equation the dimensions are m n n r = (m r)(r r) We always can find orthogonal basis for Nul(A) and Nul(A T ) and augment those vectors in matrices U and V, respectively to make them into square orthogonal matrices. We can fill zeroes in the rest of diagonal elements in matrix Σ. ( row space is orthogonal to null space) Then the matrices dimension in the equation AV = UΣ become: m n n n = (m m)(m n) σ r 4

5 After adding null space vectors the equation AV = UΣ becomes: A v v m = u u n σ σ r 0 How to find U and V? AV = UΣ A = UΣV = UΣV T, since V is orthogonal. AA T = UΣV T (UΣV T ) T = UΣV T VΣ T U T = UΣ U T, since V T V = I and ΣΣ T = Σ A T A = (UΣV T ) T UΣV T = VΣ T U T UΣV T = VΣ V T, since U T U = I and ΣΣ T = Σ 5

6 A = UΣV T AA T = UΣ U T A T A = VΣ V T Recall that AA T and A T A are symmetric matrices where their sizes are m m and n n, respectively. Recall that symmetric matrices are diagonalizable and their eigenvector matrix are orthogonal. The eigenvalues of AA T are the same as eigenvalues of A T A since the eigenvalues of AB are the same as eigenvalues of BA. In other words, Σ are the eigenvalues and columns of V are the eigenvectors of A T A, and Uare the eigenvectors of AA T 6

7 Example: Find matrix A = 3 3 vector decomposition. to its equivalent singular Find AA T = UDU T, AA T = = λ λ = 0, λ = 0 λ 0 λ = 0 λ =, λ = 0. 7

8 Example Cont d: u = 0 u = 0 u = 0 0 u = 0 u = 0 u = U =, D =

9 Example Cont d: Find A T A = VDV T, AA T = = λ λ 4 4 λ = 0, λ λ 0 λ = 0 λ =, λ = 0, λ 3 = 0. 9

10 Example Cont d: v = v = 0 v = v = v = 0 v = v 3 = v 3 = 0 v 3 = 5 0

11 Example Cont d: V = , D = A = UΣV T 3 3 =

12 Geometric interpretation Here is a geometric interpretation of SVD for a matrix M. V T (in figure V ) rotates unit vectors. Σ scales the vectors U perform the final rotation.

13 Note that we can assume SVD of matrix A as: A = Where r is the rank of the matrix. r i= σ i u i v i T Size of each σ i u i v i T is m n. The greater σ i the greater values added to reconstruct matrix A. A = σ u v T + + σ r u r v r T 3

14 Solve Ax = 0 What minimizes Ax. And why? As defined previous we can write A as A = UΣV T = u u n σ σ r 0 Where σ r is the smallest eigenvalue. Now we have Ax = v v r r i= σ i u i v i T x If we choose x = v r. Then since all v i T v r = 0 i r we have Ax = Av r = σ r u r v r T v r = σ r u r Smallest value of Ax associate with x = v r. And if σ r = 0 then v r is an answer of Ax=0

15 Pseudo inverse Assume Ax = b, Then by singular value decomposition of A we have Ax = b UΣV T x = b ΣV T x = U T b V T x = Σ U T b x = VΣ U T b VΣ U T called the pseudo inverse of A. It is useful for finding inverse of non-square matrices.

16 Applications One of the applications of SVD is dimensionality reduction. A m n matrix can be thought of gray level of a digital image. If the image A is decomposed to its singular values and vectors, we can pick only the most significant u i s, σ i s and v i s. By doing this we can compress the information of the image. Suppose the image A is m n. A = K i= σ i u i v T i In next slide you will see the original image and its compressed up to K most significant singular values. 6

17 Image compression using SVD K=8 K=3 K=8 K=5 Original 7