The Singular Value Decomposition Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) SVD Fall 2015 1 / 13
Review of Key Concepts We review some key definitions and results about matrices that will be used in this section. The transpose of a matrix A, denoted A T is the matrix obtained from A by switching its rows and columns. In other words, if A = (a ij ) then A T = (a ji ). The conjugate transpose of a matrix A, denoted A is obtained from A by switching its rows and columns and taking the conjugate of its entries. In other words, if A = (a ij ) then A = (a ji ). A matrix A is said to be symmetric if A = A T. Symmetric matrices have the following properties: Their eigenvalues are always real. They are always diagonalizable. Their eigenvectors are orthogonal. They are orthogonally diagonalizable that is if A is such a matrix then there exists an orthogonal matrix P such that P 1 AP is diagonal. Philippe B. Laval (KSU) SVD Fall 2015 2 / 13
Review of Key Concepts A matrix A is said to be Hermitian if A = A. For matrices with real entries, being Hermitian is the same as being symmetric. An n n matrix A is said to be normal if A A = AA. Obviously, Hermitian matrices are also normal. A matrix A is said to be unitary if AA = A A = I. Unitary matrices have the following properties: They preserve the dot product that is Ax, Ay = x, y Their columns and rows are orthogonal. They are always diagonalizable. det A = 1. A 1 = A A matrix A is said to be orthogonal if AA T = A T A = I. Orthogonal matrices have the following properties: They preserve the dot product that is Ax, Ay = x, y Their columns and rows are orthogonal. det A = 1. A 1 = A T Philippe B. Laval (KSU) SVD Fall 2015 3 / 13
Review of Key Concepts A quadratic form on R n is a function Q defined on R n by Q (x) = x T Ax for some n n matrix A. Here are a few important facts about quadratic forms: In the case A is symmetric, there exists a change of variable x = Py that transforms x T Ax into y T Dy where D is a diagonal matrix. In the case A is symmetric, the maximum value of x T Ax is the absolute value of the largest eigenvalue λ 1 of A and it happens in the direction of u 1 the corresponding eigenvector. Philippe B. Laval (KSU) SVD Fall 2015 4 / 13
Introduction to the SVD of a Matrix Recall that if A is symmetric, then its eigenvalues are real. Moreover, if Ax = λx and x = 1 then Ax = λx = λ x = λ. Hence, λ measures the amount by which A stretches (or shrinks) vectors which have the same direction as the eigenvectors. If λ 1 is the eigenvalue with largest magnitude and v 1 is its corresponding eigenvectors, then v 1 gives the direction in which the stretching effect of A is the greatest. This description of v 1 and λ 1 has an analogue for rectangular matrices that will lead the the SVD. We begin with an example. Example [ ] 4 11 14 Let A =. Find a unit vector x at which the length of Ax 8 7 2 is maximized and compute this maximum length. Philippe B. Laval (KSU) SVD Fall 2015 5 / 13
The Singular Values of a Matrix Let A be an m n matrix. Then as noted in the example, A T A is an n n symmetric matrix hence orthogonally diagonalizable. Let {v 1, v 2,..., v n } be an orthonormal basis for R n consisting of the eigenvectors of A T A and let λ 1, λ 2,..., λ n be the corresponding eigenvalues of A T A. Then, for 1 i n, we have: Av i 2 = λ i we see that all the eigenvalues of A T A are nonnegative. by renumbering, we may assume that λ 1 λ 2... λ n. Definition The singular values of A are the square roots of the eigenvalues λ i of A T A, denoted σ i and they are arranged in decreasing order. In other words, σ i = λ i Since Av i 2 = λ i, we see that σ i is the length of the vectors Av i, where v i are the eigenvectors of A T A. Philippe B. Laval (KSU) SVD Fall 2015 6 / 13
The Singular Values of a Matrix Example Find the singular values of A, the matrix of the previous example. We have the following important theorem. Theorem Suppose that {v 1, v 2,..., v n } is an orthonormal basis for R n consisting of the eigenvectors of A T A arranged so that the corresponding eigenvalues of A T A satisfy λ 1 λ 2... λ n, and suppose that A has r nonzero singular values. Then, {Av 1, Av 2,..., Av r } is an orthogonal basis for ColA, and ranka = r. Philippe B. Laval (KSU) SVD Fall 2015 7 / 13
The SVD of an mxn Matrix Let A be an m n matrix with r nonzero singular values where r min (m, n). Define D to be the r r diagonal matrix consisting of these r nonzero singular values of A such that σ 1 σ 2... σ r. Let [ ] D 0 Σ = 0 0 be an m n matrix. The SVD decomposition of A will involve Σ. More specifically, we have the following theorem. Theorem Let A be an m n matrix with rank r. Then there exists an m n matrix Σ as in 1 as well as an m m orthogonal matrix U and an n n orthogonal matrix V such that A = UΣV T (1) Philippe B. Laval (KSU) SVD Fall 2015 8 / 13
The SVD of an mxn Matrix Definition Any decomposition A = UΣV T with U and V orthogonal, Σ as in 1 and positive diagonal entries for D,is called a singular value decomposition (SVD) of A. The matrix U and V are not uniquely determined, but the diagonal entries of Σ are necessarily the singular values of A. The columns of U are called the left singular vectors of A and the columns of V are called the right singular vectors of A. Example Find the SVD of the matrix A in the examples above. Philippe B. Laval (KSU) SVD Fall 2015 9 / 13
The SVD of an mxn Matrix We outline the proof. Let λ i and v i be as above. Then {Av 1, Av 2,..., Av r } is an orthogonal basis for col A. We normalize each Av i to obtain an orthonormal basis {u 1, u 2,..., u r }, where u i = 1 Av i Av i = 1 σ i Av i thus Av i = σ i u i (1 i r). Next, we extend {u 1, u 2,..., u r } to an orthonormal basis {u 1, u 2,..., u m } of R m and let U = [u 1, u 2,..., u m ] and V = [v 1, v 2,..., v n ]. By constructions, both U and V are orthogonal matrices. Also, AV = [σ 1 u 1 σ 2 u 2 σ r u r 0 0] Philippe B. Laval (KSU) SVD Fall 2015 10 / 13
The SVD of an mxn Matrix Let D and Σ be as above, then σ 1 0 0 0 0 0 σ 2...... 0 0. UΣ = [u 1, u 2,..., u m ] 0 0 σ r 0. 0 0 0 0 0.. 0 0 0 = [σ 1 u 1 σ 2 u 2 σ r u r 0 0] = AV Therefore, UΣV T = AVV T = A since V is orthogonal (VV T = I ) Philippe B. Laval (KSU) SVD Fall 2015 11 / 13
The SVD and MATLAB The MATLAB command is svd. Two useful formats for this command are: 1 X=svd(A) will returns a vector X containing the singular values of A. 2 [U,S,V] = svd(x) produces a diagonal matrix S, of the same dimension as X and with nonnegative diagonal elements in decreasing order, and unitary matrices U and V so that X = USV T. Philippe B. Laval (KSU) SVD Fall 2015 12 / 13
Exercises See the problems at the end of the notes on the basics of SVD. Philippe B. Laval (KSU) SVD Fall 2015 13 / 13