CHAPTER 6 The Singular Value Decomposition Exercise 67: SVD examples (a) For A =[, 4] T we find a matrixa T A =5,whichhastheeigenvalue =5 Thisprovidesuswiththesingularvalue =+ p =5forA Hence the matrix A has rank and a SVD of the form A = U U 5 0 V, with U, U R,, V = V R The eigenvector of A T A that corresponds to the eigenvalue =5isgivenbyv =, providing us with V = Using part of Theorem 65, one finds u = [, 5 4]T Extending u to an orthonormal basis for R gives u = [ 4, 5 ]T A SVD of A is therefore A = 4 5 5 4 0 (b) One has A = 4 5, A T = 9 9, AT A = 9 9 The eigenvalues of A T A are the zeros of det(a T A I) =(9 ) 8, yielding =8and =0,andtherefore = p 8 and = 0 Note that since there is only one nonzero singular value, the rank of A is one Following the dimensions of A, one finds p 8 0 = 4 0 05 0 0 The normalized eigenvectors v, v of A T A corresponding to the eigenvalues the columns of the matrix V =[v v ]= p, are Using part of Theorem 65 one finds u,whichcanbeextendedtoanorthonormal basis {u, u, u } using Gram-Schmidt Orthogonalization (see Theorem 49) The vectors u, u, u constitute a matrix U =[u u u ]= 4 5 4
A SVD of A is therefore given by A = 4 p 8 0 5 4 0 05 p 0 0 Exercise 68: More SVD examples (a) We have A = e and A T A = e T e = This gives the eigenpair (, v )= (, ) of A T A Hence =and = e = A As = A and V = I we must have U = I m yielding a singular value decomposition A = I m e I (b) For A = e T n,thematrix 0 0 0 A T A = 6 7 4 0 0 0 5 0 0 has eigenpairs (0, e j )forj =,,n and(, e n ) Then = e T R,n and V = e n, e n,,e R n,n Using part of Theorem 65 we get u =,yielding U = A SVD for A is therefore given by A = e T n = e T en, e n,,e (c) In this exercise A = 0 0, AT = A, A T A = 0 0 9 The eigenpairs of A T A are given by (, v )=(9, e )and(, v )=(, e ), from which we find 0 0 = 0, V = 0 Using part of Theorem 65 one finds u = e and u = matrix 0 U = 0 A SVD of A is therefore given by 0 0 0 A = 0 0 0 e, which constitute the Exercise 66: Counting dimensions of fundamental subspaces Let A have singular value decomposition U V By parts and of Theorem 65, span(a) andspan(a )arevectorspacesof the same dimension r, implyingthatrank(a) =rank(a ) This statement is known as the rank-nullity theorem, anditfollowsimmediately from combining parts and 4 in Theorem 65 As rank(a )=rank(a) by,thisfollowsbyreplacinga by A in 5
Exercise 67: Rank and nullity relations Let A = U V be a singular value decomposition of a matrix A C m n By part 5 of Theorem 64, rank(a) isthenumberofpositiveeigenvaluesof AA = U V V U = UDU, where D := is a diagonal matrix with real nonnegative elements Since UDU is an orthogonal diagonalization of AA,thenumberofpositiveeigenvaluesofAA is the number of nonzero diagonal elements in D Moreover, rank(aa )isthenumber of positive eigenvalues of AA (AA ) = AA AA = U V = UD U, which is the number of nonzero diagonal elements in D,sothatrank(A) =rank(aa ) From a similar argument for rank(a A), we conclude that rank(a) =rank(aa )=rank(a A) Let r := rank(a) =rank(a )=rank(aa )=rank(a A) Applying Theorem 64, parts and 4, to the singular value decompositions A = U V, A = V U, AA = U U, A A = V V, one finds that {v r+,,v n } is a basis for both ker(a)andker(a A), while {u r+,u m } is a basis for both ker(a )andker(aa ) In particular it follows that null(a) =null(a A), null(a )=null(aa ), which is what needed to be shown Given is the matrix A = 4 4 6 5 Exercise 68: Orthonormal bases example With B as defined its is straightforward to find these spaces Let us therefore set B = A T instead We have that A = U V T and B = V T U T,with V = 4 5 0 0, =, U = 4 0 0 5 4 From Theorem 65 we know that V forms an orthonormal basis for span(a )= span(b), V an orthonormal basis for ker(a)andu an orthonormal basis for ker(a T )= ker(b) Hence span(b) = v + v, ker(a) = v and ker(b) =0 Exercise 69: Some spanning sets The matrices A C m n and A A have the same rank r since they have the same number of singular values, so that the vector spaces span(a A)andspan(A )have the same dimension It is immediate from the definition that span(a A) span(a ), and therefore span(a A) = span(a ) From Theorem 65 we know that the columns of V form an orthonormal basis for span(a ), and the result follows 6
Exercise 60: Singular values and eigenpair of composite matrix Given is a singular value decomposition A = U V Let r = rank(a), so that r > 0and r+ = = n =0 LetU =[U, U ]andv =[V, V ] be partitioned accordingly and =diag(,, r) asinequation(67),sothat A = U V forms a singular value factorization of A By Theorem 65, 0 A ui Avi Cp i = A = 0 v i A = ip i for i =,,r u i 0 p i for i = r +,,n 0 A Cq i = A 0 ui vi Avi = A = u i iq i for i =,,r 0 q i for i = r +,,n 0 A Cr j = A 0 uj 0 0 0 = A = u j 0 =0 r j, for j = n +,,m This gives a total of n + n +(m n) =m + n eigen pairs Exercise 66: Rank example We are given the singular value decomposition 6 0 0 A = U V T = 6 7 60 6 0 7 4 4 5 40 0 05 0 0 0 Write U =[u, u, u, u 4 ]andv =[v, v, v ] Clearly r =rank(a) = (a) A direct application of Theorem 65 with r =gives {u, u } is an orthonormal basis for span(a), {u, u 4 } is an orthonormal basis for ker(a T ), {v, v } is an orthonormal basis for span(a T ), {v } is an orthonormal basis for ker(a) Since U is orthogonal, {u, u, u, u 4 } is an orthonormal basis for R 4 In particular u, u 4 are orthogonal to u, u,sothattheyspantheorthogonalcomplementspan(a)? to span(a) =span{u, u } (b) Applying Theorem 65 with r =yields q ka Bk F + = p 6 +0 =6 (c) Following Section 6, with D 0 := diag(, 0,,0) R n,n,take D A = A 0 0 := U 0 VT = 6 7 4 5 5 7
Exercise 67: Another rank example (a) The matrix B =(b ij ) ij R n,n is defined by 8 if i = j; >< if i<j; b ij = >: n if (i, j) =(n, ); 0 otherwise while the column vector x =(x j ) j R n is given by if j = n; x j = n j otherwise For the final entry in the matrix product Bx one finds that (Bx) n = b nj x j = b n x + b nn x n = n n + =0 j= For any of the remaining indices i 6= n, thei-th entry of the matrix product Bx can be expressed as Xn (Bx) i = b ij x j = b in + n j b ij j= = + n i b ii + j= Xn = + n i Xn j=i+ j=i+ n j b ij n j i = + n i n i j 0 =0 = + n i n i j 0 n i (n i) = + n i n i =0 As B has a nonzero kernel, it must be singular The matrix A, ontheotherhand, is nonsingular, as its determinant is ( ) n 6=0 ThematricesA and B di er only in their (n, )-th entry, so one has ka Bk F = p a n b n = n Inotherwords,the tiniest perturbation can make a matrix with large determinant singular (b) Let n 0bethesingularvaluesofA Applying Theorem 65 for r =rank(b) <n, we obtain n q r+ + + n = min ka Ck F ka Bk F = n CR n,n rank(c)=r We conclude that the smallest singular value n can be at most n 8