SINGULAR VALUE DECOMPOSITION

Transcription

1 DEPARMEN OF MAHEMAICS ECHNICAL REPOR SINGULAR VALUE DECOMPOSIION Andrew Lounsbury September 8 No 8- ENNESSEE ECHNOLOGICAL UNIVERSIY Cookeville, N 38

2 Singular Value Decomposition Andrew Lounsbury Department of Mathematics ennessee echnological University Cookeville, N, 38 September 8, 8 Abstract he Singular Value Decomposition (SVD) provides a cohesive summary of a handful of topics introduced in basic linear algebra SVD may be applied to digital photographs so that they may be approximated and transmitted with a concise computation Mathematics Subject Classification Primary A3, A4 Keywords singular value, matrix factorization Contents Introduction Steps for Calculation of SVD 3 3 heory 4 Examples 9 Maple 6 Conclusions 6 A Appendix 6 Introduction his paper begins with a definition of SVD and instructions on how to compute it, which includes calculating eigenvalues, singular values, eigenvectors, left and right singular vectors, or, alternatively, orthonormal bases for the four fundamental spaces of a matrix We present two theorems Completed in partial fulfillment of the requirements for MAH 4993 Mathematical Research (3 cr) in spring 8 under supervision of Dr R Ablamowicz

3 that result from SVD with corresponding proofs We provide examples of matrices and their singular value decompositions here is also a section involving Maple that includes examples of photographs It is demonstrated how the inclusion of more and more information from the SVD allows one to construct accurate approximations of a color image Definition Let A be an m n real matrix of rank r min(m, n) A Singular Value Decomposition (SVD) is a way to factor A as A UΣV, () where U and V are orthogonal matrices such that U U I m and V V I n he Σ matrix contains the singular values of A on its pseudo-diagonal, with zeros elsewhere hus, σ A UΣV [ v v u u u m }{{} σ r, () U(m m) v n }{{}}{{} V (n n) Σ(m n) with u,, u m being the orthonormal columns of U, σ,, σ r being the singular values of A satisfying σ σ σ r >, and v,, v n being the orthonormal columns of V Singular values are defined as the positive square roots of the eigenvalues of A A Note that since A A of size n n is real and symmetric of rank r, r of its eigenvalues σ i, i,, r, are positive and therefore real, while the remaining n r eigenvalues are zero In particular, A A V (Σ Σ)V, A Av i σ i v i, i,, r, A Av i, i r +,, n (3) hus, the first r vectors v i are the eigenvectors of A A with the eigenvalues σi Likewise, we have AA U(ΣΣ )U, AA u i σ i u i, i,, r, AA u i, i r +,, m (4) hus, the first r vectors u i are the eigenvectors of AA with the eigenvalues σ i Furthermore, it can be shown (see Lemmas and ) that Av i σ i u i, i,, r, and Av i, i r +,, n () If rank(a) r < min(m, n), then there are n r zero columns and rows in Σ, rendering the

4 r +,, m columns of U and r +,, n rows in V unnecessary to recover A herefore, σ v A UΣV [ u u r u r+ u m σ r vr vr+ vn σ [ v u u r σ }{{} m r vr σ r }{{}}{{} r n r r u σ v + + u r σ r v r (6) Steps for Calculation of SVD Here, we provide an algorithm to calculate a singular value decomposition of a matrix Compute A A of a real m n matrix A of rank r Compute the singular values of A A Solve the characteristic equation A A(λ) A A λi of A A for the eigenvalues λ,, λ r of A A hese eigenvalues will be positive ake their square roots to obtain σ,, σ r which are the singular values of A, that is, σ i + λ i, i,, r (7) 3 Sort the singular values, possibly renaming them, so that σ σ σ r 4 Construct the Σ matrix of size m n such that Σ ii σ i for i,, r, and Σ ij when i j Compute the eigenvectors of A A Find a basis for Null(A A λ i I) hat is, solve (A A λ i I)s i for s i, an eigenvector of A corresponding to λ i, for each eigenvalue λ i Since A A is symmetric, its eigenvectors corresponding to different eigenvalues are already orthogonal (but likely not orthonormal) See Lemma 6 Compute the (right singular) vectors v,, v r by normalizing each eigenvector s i by multiplying it by s i hat is, let v i s i s i, i,, r (8) 3

5 If n > r, the additional n r vectors v r+,, v n need to be chosen as an orthonormal basis in Null(A) Note that since Av i σ i u i for i,, vectors v,, v r provide an orthonormal basis for Row(A) while the vectors u,, u r provide an orthonormal basis for Col(A) In particular, R n Row(A) Null(A) span{v,, v r } span{v r+,, v r+(n r) } (9) 7 Construct the orthogonal matrix V [v v n 8 Verify V V I 9 Compute the (left singular) vectors u,, u r as Av i σ i u i u i Av i σ i, i r () In this method, u,, u r are orthogonal by Lemma Alternatively, (i) Note that AA U(ΣΣ )U suggests the vectors of U can be calculated as the eigenvectors of AA In using this method, the vectors need to be normalized first Namely, u i s i s i, where s i is an eigenvector of AA (ii) Since A A(λ) AA (λ) by Lemma 8, σ,, σ r are also the square roots of the eigenvalues of AA If m > r, the additional m r vectors u r+,, u m need to be chosen as an orthonormal basis in Null(A ) Note that since Av i σ i u i for i,, vectors u,, u r provide an orthonormal basis for Col(A) while the vectors u r+,, u m provide an orthonormal basis for the left null space Null(A ) In particular, R m Col(A) Null(A ) span{u,, u r } span{u r+,, u r+(m r) } () Construct U [u u m Verify U U I Verify A UΣV 3 Construct the dyadic decomposition of A, as described in hm 3: A UΣV σ u v + σ u v + + u r σ r v r () A dyad is a product of an n column vector with another n row vector, eg, u v, resulting in a square n n matrix whose rank is by Lemma 6 4

6 3 heory In this section, we provide the two theorems related to SVD along with their proofs heorem Let A UΣV be a singular value decomposition of an m n real matrix of rank r hen, AV UΣ and { Avi σ i u i, i,, r Av i, i r +,, r + (n r) { Row(A) span{v,, v r } Null(A) span{v r+,, v r+(n r) } A A V (Σ Σ)V : R n R n 3 A AV V (Σ Σ) and { A Av i σi v i, i,, r A Av i, i r +,, r + (n r) { Row(A A) span{v,, v r } Null(A A) span{v r+,, v r+(n r) } 4 U A ΣV and { u i A σ i v i, i,, r u i A, i r +,, r + (m r) { Col(A) span{u,, u r } Null(A ) span{u r+,, u r+(m r) } AA U(ΣΣ )U : R m R m 6 AA U U(ΣΣ ) and { AA u i σi u i, i,, r AA u i, i r +,, r + (m r) { Row(AA ) span{u,, u r } Null(AA ) span{u r+,, u r+(m r) } Proof of () AV (UΣV )V UΣ(V V ) UΣ So, AV [ Av Av r Av r+ Av n σ [ u u r u r+ u m σ r [ σ u σ r u r Hence,

7 Av σ u,, Av r σ r u r, and Av r+,, Av r+(n r) Proof of () A A (UΣV ) (UΣV ) (V Σ U )(UΣV ) V Σ (U U)ΣV V Σ ΣV Proof of (3) A AV (V Σ U )(UΣV )V V Σ (U U)Σ(V V ) V (Σ Σ) So, A AV [ A Av A Av r A Av r+ A Av n [ λ v λ r v r λ r+ v r+ λ n v n λ [ v v r v r+ v n λ r σ [ v v r v r+ v n σr [ σ v σ rv r Hence, A Av σ v,, A Av r σ rv r, and A Av r+,, A Av r+(n r) Proof of (4) U A U (UΣV ) (U U)ΣV ΣV 6

8 So, U A u u r u r+ u m A u A u r A u r+ A u ma σ σ r v v r v r+ v n σ v σ r v r Hence, u A σ v,, u r A σ r v r, and u r+ A,, u r+(m r) A Proof of () AA (UΣV )(UΣV ) (UΣV )(V Σ U ) UΣ (V V )ΣU UΣ ΣU Proof of (6) AA U (UΣV )(UΣV ) U (UΣV )(V Σ U )U UΣ(V V )Σ (U U) U(ΣΣ ) 7

9 So, AA U [ AA u AA u r AA u r+ AA u m [ λ u λ r u r λ r+ u r+ λ n u m λ [ u u r u r+ u m λ r σ [ u u r u r+ u m σr [ σ u σ ru r Hence, AA u σ u,, AA u r σ ru r, and AA u r+,, AA u r+(m r) heorem Let A UΣV be a singular value decomposition of an m n real matrix of rank r hen, A UΣV r u i σ i vi σ u v + σ u v + + σ r u r vr (3) i 8

10 Proof A UΣV σ u σ u σ r u r v v v r σ u σ u σ r u r v σ u (r ) σ u r σ u r σ r u rr vr vr vrr (σ u v + + σ ru r vr ) (σ u v + + σ ru r vr ) (σ u vr + + σ ru r v rr) (σ u v + + σ ru r vr ) (σ u r v + + σ ru rr v r ) (σ u r v r + + σ ru rr v rr) σ u v + (σ u v + + σ ru r vr ) (σ u v + + σ ru r vr ) (σ u vr + + σ ru r v rr) (σ u v + + σ ru r vr ) (σ u r v + + σ ru rr vr ) (σ u rr vr + + σ ru rr vrr) σ u v + σ u v + + σ r u r v r u σ v + u σ v + + u r σ r v r r u i σ i vi i 4 Examples In this section we calculate the singular value decomposition of a few matrices Example Let A, then A A 6 7, and

11 A A(λ) σ σ S λ λ 6 λ λ Σ λ 4 4λ 3 + 8λ λ λ , s 6, s s 3 3 s 4 4 S contains the transposed eigenvectors of A A , , , V , , , ( ) ( u Av σ ( ) ( u Av σ , , , , ) ) ,77,8+33, ,77,8+33, ,77,8+33, ,77,8 33, ,77,8 33, ,77,8 33, Since A is 3 4, U should be a 3 3 matrix However, there is no σ 3, so we cannot use u i Av i σ i to find u 3 Instead, we use the left null space of A Null(A ) y 4 Normalizing y to obtain u 3 : u 3 y y

12 So, U hus, A ,77,8+33, ,77,8+33, ,77,8+33, ,77,8 33, ,77,8 33, ,77,8 33, ,77,8+33, ,77,8 33, ,77,8+33, ,77,8 33, ,77,8+33, ,77,8 33, , , , , , , , , [ UΣV Example Let A Notice that when[ A is a[ symmetric [ matrix, A A AA, so U V Less work is required A A AA rank(a) rank(a A) rank(aa ) A A(λ) AA (λ) A A λi AA λi λ λ ( λ)( λ) λ 3λ + λ 3+ λ 3 σ [ λ + + σ 3 6 Σ λ v & u : [ [A A λ Ix [AA 3+ λ Is 3+ [ [ [ + s s v u s s v & u : + [ + + [ [

13 [A A λ Is [AA 3 λ Is 3 [ + [ + [ + s s v u s s V [ [, U o verify that v u and v u : [ + Av Av [ [ [ σ u σ v σ u σ v Notice that this implies the eigenvalues of A are equal to the singular values of A By Lemma 7, every symmetric matrix has this property hus, A [ Maple [ UΣV UΛV he purpose in subjecting a color photograph to the Singular Value Decomposition is to greatly reduce the amount of data required to transmit the photograph to or from a satellite, for instance A digital image is essentially a matrix comprised of three other matrices of identical size hese are the red, green, and blue layers that combine to produce the colors in the original image Obtain the three layers of the image using the Maple command GetLayers from the Imageools package It is on each of these three layers that we perform the SVD Define each one as img_r,img_g, and img_b Define the singular values of each matrix using the SingularValues command in the Linear Algebra package Maple will also calculate U and V Simply set the output of SingularValues[ U, Vt In the argument of the following procedure, the variable n denotes which approximation the procedure will compute, that is, the number of singular values that it will include posint indicates that n must be a positive integer approx:proc(img_r,img_g,img_b,n::posint)local Singr,Singg,Singb,Ur,Ug,Ub,Vtr,Vtg,Vtb,singr,

14 singg,singb,ur,ug,ub,vr,vg,vb,mr,mg,mb,i,img_rgb; In place of a Σ matrix, we create a list of the σ s for each red, green and blue layer, as well as the red, green, and blue U and V matrices It is important to note that Maple outputs the transpose of V Hence, it does not need to be transposed Singr:SingularValues(img_r,output list ): Singg:SingularValues(img_g,output list ): Singb:SingularValues(img_b,output list ): Ur,Vtr:SingularValues(img_r,output[ U, Vt ); Ug,Vtg:SingularValues(img_g,output[ U, Vt ); Ub,Vtb:SingularValues(img_b,output[ U, Vt ); Pulling out each individual σ i, u i, and v i to create the σ i u i v i dyads for i r : for i from to n do singr[i:singr[i; singg[i:singg[i; singb[i:singb[i; ur[i:linearalgebra:-column(ur,ii); vr[i:linearalgebra:-column(linearalgebra:-ranspose(vtr),ii); ug[i:linearalgebra:-column(ug,ii); vg[i:linearalgebra:-column(linearalgebra:-ranspose(vtg),ii); ub[i:linearalgebra:-column(ub,ii); vb[i:linearalgebra:-column(linearalgebra:-ranspose(vtb),ii); end do; Note that Maple stores data as floating point numbers, so values that would be otherwise are stored as a small decimal number very close to, yet still greater than his means that, when working with Maple, rank(a) r min(m, n) Adding the dyads to produce the approximations of each layer: Mr:add(singr[i*ur[iLinearAlgebra:-ranspose(vr[i),in); Mg:add(singg[i*ug[iLinearAlgebra:-ranspose(vg[i),in); Mb:add(singb[i*ub[iLinearAlgebra:-ranspose(vb[i),in); Combining the approximations of each layer: img_rgb:combinelayers(mr,mg,mb): Displaying the result in Maple: Embed(img_rgb); end proc: 3

15 Application Suppose we have a color photograph that is pixels, or entries hat s, pixels When we separate it into its red, green, and blue layers, the number of entries becomes 3, or 6, entries Apply SVD to each of these matrices and take enough of the dyad decomposition to obtain a meaningful approximation For the sake of example, suppose two products from the outer product decomposition suffice In other words, we have σ u v + σ u v σ u [ v + σ u [ v he σ s are just scalars, the u i column vectors are, and the row vectors vi are So, it follows that σ u v + σ u v has entries Multiply this by three to account for the red, green, and blue layers to obtain,86 entries, approximately 3% of the original 6, entries that we would have had to send had we not utilized the SVD Now, when the satellite sends the photograph down to Earth, it sends those σ and σ, u and u, and v and v separately All that needs to be done to recover the approximation is to multiply these together and add them up back on Earth Example 3 Here, we show a progression of approximations of a photograph of a galaxy picture has dimensions 78 96, and each approximation uses n singular values he 4

16 (a) original image (b) n his is merely σ u v, the first term in the sum of 78 dyads σi ui vi hus, it bares very little resemblance to the original image (d) n ; σ u v + + σ u v (c) n ; σ u v + + σ u v Notice that with only 78, or 3%, of the infor dyads, or 8% of all the data, we With only 78 mation contained in the original image we have can already see the shapes in the original image an approximation that is close to being indistinstarting to come together guishable from the original (e) n ; σ u v + + σ u v his is 64% of all the information in the orig (f) n 78; σ u v + + σ78 u78 v78 inal image he lack of a noticeable difference When we use all singular values, the approximabetween this picture and the previous one illustion is the same as the original image trates the fact that the σi values are getting much smaller as i increases

17 6 Conclusions In summary, the application of singular value decomposition we have detailed provides a method of calculating very accurate approximations of photographs so that they may be transmitted from satellites to Earth without requiring large amounts of data SVD provides bases for the Four Fundamental Subspaces of a matrix, and it gets its versatility from the ordering of the σ values SVD is also used in the calculation of pseudoinverses, as illustrated in [3, among other things A Appendix In this appendix, we prove a few results related to symmetric matrices Lemma Let A A hen eigenvectors corresponding to different eigenvalues are orthogonal Proof Let Av λ v and Av λ v We show that v v v v Observe that λ (v v ) (λ v ) v (Av ) v (v A )v v (Av ) v (λ v ) λ (v v ) hus, λ (v v ) λ (v v ) (λ λ )(v v ), so v v since λ λ Lemma Let A A and A be real hen, eigenvalues of A are non-negative Proof Suppose A A and A A Let Av λv, where v We show that λ λ hus, and, Av λv and (Av) v v (A v) v (Av) v (λv) λ(v v), Let v z, where z i C So, z n (λv )v λ(v v) λ(v v) (4) v v [ z,, z n z z z + + z n z n z + + z n > z n From (4) we have λ(v v) λ(v v) since v Hence, (λ λ)(v v), so λ λ since v v > Lemma 3 Let A be an m n matrix Null(A) Null(A A) Proof of ( ) Let x Null(A) hen Ax so A (Ax) (A A)x x Null(A A) 6

18 Proof of ( ) Let x Null(A A) hen (A A)x hus A (Ax), so Ax Null(A ) On the other hand, Ax Col(A) Since R m Col(A) Null(A ), we conclude that Ax since Col(A) Null(A ) {} hus, Null(A A) Null(A) Null(A) Null(A A) rank(a) rank(a A) Proof Let r rank(a) hus, r n dim(null(a)) n dim(null(a A)) rank(a A) since A A is n n 3 dim(null(a)) n r dim(null(a A)) dim(col(a)) r dim(row(a)) dim(col(a A)) rank(a A) rank(a) dim(col(a)) r 4 Null(A ) Null(AA ) Proof of ( ) Let x Null(A ), then A x so A(A x) (AA )x x Null(AA ) Proof of ( ) Let x Null(AA ) hen (AA )x hus, A(A x) so A x Null(A) On the other hand, A x Col(A ) Since R m Col(A ) Null(A), we conclude that A x since Col(A ) Null(A) {} hus, Null(AA ) Null(A ) Null(A ) Null(AA ) rank(a ) rank(aa ) rank(a), so rank(a) rank(a A) rank(a ) rank(aa ) 6 We have the following: (i) dim(null(a )) m r dim(null(aa )), (ii) dim(col(a )) rank(a ) rank(a) r dim(row(a )), (iii) dim(col(aa )) rank(aa ) rank(a) dim(col(a)) r 7 Since A Av i σ i v i, i r, and vectors {v,, v r } provide a basis for Row(A), we have Av i, i r If A Av i, then v i would belong to Null(A A) Null(A), which would give Av i contradiction So, A Av i σ i v i, which implies σ i so σ i, i r Lemma 4 σ i for i r 7

19 Lemma Since the vectors {v,, v r } are orthonormal, vectors {u,, u r } computed as are also orthonormal Proof We have the following: u i Av i σ i, i r, (Av i ) (Av j ) v i (A Av j ) v i σ j v j σ j v i v j σ j δ ij {, i j; σ j, i j hus, vectors {u i } r i are orthogonal and orthonormal because: ( ) ( ) u Avi Avj i u j ( ) { (Av i ) (Av j ) σ, i j; j δ ij σ i σ j σ i σ j σ i σ j, i j Lemma 6 Let v be a n vector hen, the dyad v v has rank Proof We compute the following: v v v v v v v n vv v [ v v vn v v v v v v n }{{} n v n }{{} v n v } v n vn {{ v n vn } n n n v v v v v n v Since the rows of vv are multiples of v, they are linearly dependent herefore, we have rank(vv ) Lemma 7 Let A be a symmetric matrix hen, the eigenvalues of A are equal to the singular values of A Proof Let A be a matrix such that A A, let λ be an eigenvalue of A, and let v be an eigenvector of A hen, Av λv, and A Av A λv λa v λav λ v Hence, λ is an eigenvalue of A A herefore, λ is a singular value of A On the other hand, let σ > be a singular value of A So, A Av σ v for some nonzero eigenvector v A A(σ ) det(a A σ I) det(a σ I) det((a σi)(a + σi)) det(a σi) det(a + σi), which implies that det(a σi), det(a + σi), or det(a σi) det(a + σi) Suppose, det(a + σi) hen, σ is an eigenvalue of A, a contradiction since σ > herefore, det(a + σi), and σ is an eigenvalue of A Lemma 8 Let A be an m n matrix hen, A A and AA share the same nonzero eigenvalues and, therefore, both provide the singular values of A In the case where m n, A A(t) AA (t) 8

20 Proof From parts and of heorem (), we have A A V Σ ΣV V Λ n V, and AA UΣΣ U UΛ m U, where Λ n is n n, and Λ m is m m Hence, A A is similar to Λ n, and AA is similar to Λ m We cannot show that Λ n is similar to Λ m However, from Definition (), we know what Σ and Σ look like, so we know that when we multiply them together to obtain Λ n and Λ m we get two square matrices with σ,, σ r, or λ,, λ r, and zeros along the diagonal, and zeros elsewhere he only difference between the two is that one is larger, depending on whether m < n or m > n, and has more zeros on its diagonal σ λ Λ n σr λ r }{{} n n σ λ σr λ r Λ m } {{ } m m Since the eigenvalues of a diagonal matrix are simply the entries on its diagonal, we know that Λ n and Λ m both have λ,, λ r as eigenvalues Because they are similar to A A and AA, respectively, we know that A A and AA both must also have λ,, λ r as eigenvalues Moreover, an n n matrix has n entries on its diagonal and hence has a characteristic polynomial of degree n hus, and A A(t) det(λ n ti) (λ t)(λ t) (λ r t) ( t) ( t) }{{}}{{} r factors n r factors ( t) n r (λ t)(λ t) (λ r t) ( t) n r h(t), AA (t) det(λ m ti) (λ t)(λ t) (λ r t) ( t) ( t) }{{}}{{} r factors m r factors ( t) m r (λ t)(λ t) (λ r t) ( t) m r h(t), 9

21 where h(t) is a monic polynomial of degree r with only λ,, λ r as roots herefore, when A is a square n n matrix, we have the special case where A A(t) AA (t) ( t) n r h(t), and Λ m Λ n References [ Maple (8) Maplesoft, A division of Waterloo Maple Inc, Waterloo, Ontario, [ APOD, 3 April 8, he Blue Horsehead Nebula in Infrared Astronomy Picture of the Day, NASA, April 8 [3 D Poole, Linear Algebra: A Modern Introduction, Stamford, Cengage Learning, [4 G Strang, Introduction to Linear Algebra, Wellesley, Wellesley-Cambridge Press,