Chapter 7 Singular Value Decomposition

Transcription

1 EE448/58 Vesion. John Stensy Chapte 7 Singla Vale Decomposition This chapte coves the singla vale decomposition (SVD) of an m n matix. This algoithm has applications in many aeas inclding nmeical analysis and signal/image pocessing. Seveal SVD applications ae discssed in Chapte 8. It is not ncommon to find in the nmeical linea algea liteate statements that poclaim the SVD algoithm as one of the most impotant diagnostic tools availale fo the analysis of linea algeaic systems. Theoem 7- (SVD) et A C m n e an m n complex-valed matix with ank. Then thee exists nitay matices U C m m and V C n n sch that U * AV R m n, (7-) whee... > ae called the singla vales of the matix A (ememe that > is the lagest and > is the smallest and these singla vales appea on the diagonal of a diagonal matix). oof: Given A C m n of ank, the matix A * A is n n Hemitian and (at least) non-negative definite (we have aged this efoe). As we know, the eigenvales of A * A ae eal-valed; of them ae positive, and n- of them ae zeo. Denote these eigenvales as... > (with sqae-oots denoted as... > ). et v, v,..., v, v +,..., v n denote the n othonomal n eigenvectos of A * A. Then, we can wite CH7.DC age 7-

2 EE448/58 Vesion. John Stensy A A vi R S T i vi, i,, + i n (7-) whee v j, vi vi v j, i j i j. (7-3) ote that v i, i, is an othonomal asis of R(A*), and v i, + i n, is an othonomal asis of K(A). ow, define the vectos i Av i/ i, i, (7-4) whee i i. ote that the vectos i, i, ae nit length m vectos since v A Av v v i Av i i ii i i i / i, (7-5) i i i and they ae othonomal. ow, se Gam-Schmidt (o any othe pocede) to fill ot the i and fom a set of m othonomal m vectos,,...,, +,..., m with i j, i j i j, (7-6) i, j m. ote that i, i, is an othonomal asis of R(A), and i, + i m, is an othonomal asis of K(A*). ow, define the two matices CH7.DC age 7-

3 EE448/58 Vesion. John Stensy U V m v v v n. (7-7) Unitay matices U and V ae m m and n n, espectively (U and V ae nitay: U * U - and V * V - ). ow, se nitay U and V to compte the m n matix Σ U AV m A v v v n Av Av Av n Av Av Av n mav mav mav n. (7-8) ow, let s ij Av i j, i m, j n, e the elements of m n matix Σ. We age that only the fist diagonal elements of Σ ae non-zeo. Fist, conside the elements s ij, i, j, in the ppe-left lock; with the aid of (7-), these elements ae compted to e vi A Av j vi j v j i, i j s ij i A v j S i i i j R T, i, j. (7-9) The second lock to e consideed ae those elements s ij, i, + j n, in the ppe-ight {n-} lock; with the aid of (7-), these elements ae compted to e vi A Av j v s i ij i A v j v j i + j n i i c h,,. (7-) The thid lock to e consideed ae those elements s ij, + i m, j, in the lowe-left {m} lock; with the aid of (7-4), these elements ae compted to e CH7.DC age 7-3

4 EE448/58 Vesion. John Stensy s ij i A v j i ( j j ), + i m, j. (7-) The foth, and last, lock to e consideed ae those elements s ij, + i m, + j n, in the lowe-ight {m-} {n-} lock. ow, fo i, the vectos Av i ae non-zeo and othogonal; this implies they ae independent. Since ank[a], we mst have Av j, + j n, dependent on Av k, k. Hence, de to the othonomal nate of the j, the elements in the lowe-ight {m-} {n-} lock ae s ij i A v j i a linea comination of Av k, k i a linea comination of k, k g g, + i m, + j n. (7-) The te nate of m n matix Σ is clea. Fom (7-9) thogh (7-), we see that Σ U AV, (7-3) a matix whose only non-zeo enties ae the fist diagonal elements. ata's SVD Fnction Given an m n matix A, ata can calclate the SVD of A. The ata syntax is [U S V] svd(a). (7-4) CH7.DC age 7-4

5 EE448/58 Vesion. John Stensy Example % Ente matix A > A [6 3;4 ; ] A % Fist we calclate the svd sing the asic theoy > B A'*A B % Calclate eignevales and eigenvectos of A T A. > [V D] eig(b) V D 7 % V is an othogonal matix sed in svd pocede % Calclate U sed in svd. Use othonomal asis of K[A T ] to %"flsh ot" U (ecall Fige 5. of the class notes) > U [A*V(:,)/sqt(D(,)) nll(a')] U % Use matices U and V to tansfom A to svd fom >U'*A*V ans % Boy, that was a lot of wok! et's do it the easy way - let % the ata svd fnction do the wok! >[U S V] svd(a) U S V % ote that S S. ata's svd fnction podced the same % decomposition (howeve, y sing diffeent U and V) CH7.DC age 7-5

6 EE448/58 Vesion. John Stensy Bases fo ange[a], ke[a*], ange[a*] and ke[a] Fom SVD In the poof of Theoem 7-, we sed the fact that,,..., ae independent and span ange[a]. ow, since k, k m, ae othonomal, the vectos +, +,..., m ae othogonal to ange[a]; that is, they span ke[a*] (ecall Fige 5.). Hence, the m m matix U can e patitioned as U [ ] m span ange[a] span ke[a ] (7-5) In the poof of Theoem 7-, we sed the fact that Av j, + j n. Fom this, it is easy to see that v +, v +,..., v n span ke[a]. This implies that v, v,..., v spans ange[a*]. Hence, the n n matix V e patitioned as V [ v ] v 44 v 443 v + v v n (7-6) span ange[a ] span ke[a] SVD Related ojection peatos The SVD povides a method of finding the othogonal pojection onto the ange and kenel of any m n matix A. Fom (7-5) and (7-6), we conclde that RA thogonal ojection on Range[A] (7-7) KA thogonal ojection on Ke[A] v + v + v v + v + v n n (7-8) CH7.DC age 7-6

7 EE448/58 Vesion. John Stensy RA* thogonal ojection on Range[A ] v v v v v v (7-9) KA* thogonal ojection on Ke[A ] m m (7-) Singla Vales of A - ; -om of A - Sppose A UΣV * is invetale. Then all singla vales k >, k n. Also, we know that A - VΣ U *, (7-) whee Σ / / / n. (7-) Clealy, / k, k n, ae the singla vales of A -. Since n is the smallest singla vale of A, / n is the lagest singla vale of A -. As a eslt, we can wite A / n. (7-3) SVD Expansion of atix A n n, we have et A e an m n matix of ank. Since A UΣV *, whee U m m, Σ is m n and V CH7.DC age 7-7

8 EE448/58 Vesion. John Stensy A m so that v v v n (7-4) A i i v i. (7-5) i ote that m n matix A can e epesented y i, v i and i, fo i. If matix A is lage, t has small ank, then matix A can e moe efficiently stoed y saving the i, v i and i, i. This techniqe has een sed in image compession and othe aeas that deal with lage dimensional, low-ank, matices. meical Rank of a atix y SVD Analysis The nmeical detemination of the ank of m n matix A is not tivial. Rank detemination can e complicated y impecise epesentation of A (i.e., fzzy data ) and onding eos that occ in all pactical calclations. Given small ε >, the ε-ank of matix A is defined as ank( A, ε) min ank( B). (7-6) AB ε That is, look at all m n matices B that ae within ε of A. Compte the ank of each εneigho of A. The smallest of these anks is ank(a,ε). Conside an application whee m n matix A {a ij } is detemined expeimentally with each a ij coect to within ±.. Then it might make sense to detemine ank(a,.). If m n matix A contains floating point nmes, then it is easonale to egad A as nmeically ank CH7.DC age 7-8

9 EE448/58 Vesion. John Stensy deficient if ank(a,ε) < min(m,n), whee ε max( m, n) A eps. The ε-ank of a matix can e detemined, in most cases, y SVD analysis. In the nmeical analysis commnity, SVD is geneally egaded as the most eliale method fo the detemination of nmeical ank. The next theoem shows how SVD is sed to detemine nmeical ank. Theoem 7- et the SVD of m n matix A e A UΣV*, whee U is m m, V is n n, and U, V ae nitay. et k e an intege, and define k Ak i i v i. (7-7) i Then we have min ank( B) k A B A A k k +. (7-8) atix A k is a ank k matix that is k+ in distance fom A, and no othe ank k matix is close to A then is A k. Eqivalently, matix A k comes as close to A as does any ank k matix. oof: Since U * A k V diag(,,..., k,,..., ), it follows that ank(a k ) k and U (A - A k ) V diag(, 4,... 4, 3, k+,..., ρ), (7-9) k zeos whee ρ min(m, n). Also, it follows that A A k k +, since the -nom is invaiant to nitay tansfomations. ow, let B any m n matix of ank k. Then (n - k) is the nllity of B, and we can find an othonomal set x, x,..., x n-k of n vectos fo which CH7.DC age 7-9

10 EE448/58 Vesion. John Stensy ke[b] span{x, x,..., x n-k}. (7-3) In an n-dimensional space, thee ae at most n linealy independent vectos so that span{x, x,..., x n-k} span{v, v,..., v k+}. (7-3) et Z span{x, x,..., x n-k} span{v, v,..., v k+} and Z. ote that F k A Z i i vi Z i vi Z i i vi Z i H G I + i K J ( ) + ( ), (7-3) i i k+ assming that k+ (othewise omit the second sm on the ight-hand side of (7-3)). ow, since Z {v k+,..., v }, the second sm (on the ight-hand side of (7-3)) is zeo and k+ A Z i ( vi Z) i. (7-33) i Finally, since BZ (ecall that Z span{x, x,..., x n-k} K(B)), we have k+ A B ( A B) Z AZ i vi Z ( ) k+. (7-34) i So a ank k matix can e no close to A than k+, and we have finished o poof. et m n matix A have fll ank; that is, ank[a] min(n,m). Theoem 7. says that the smallest singla vale of matix A is the -nom distance of A to the set of all m n, ank deficient, matices. We can calclate the ε-ank of A fom knowledge of the singla vales. If ε ank(a,ε), CH7.DC age 7-

11 EE448/58 Vesion. John Stensy then we have ε +, (7-35) ε ε ρ whee ρ min(m, n). So, to detemine ε ank, place ε in the odeing of singla vales, and ead ε as the index of the fist singla vale lage than ε. SVD and the inimm om, east Sqaes olem Conside again the geneal AX polem ( may, o may not e, in ange[a]). Hee, we want to find the minimm nom X that minimizes AX. SVD analysis can solve this polem. Theoem 7-3 Sppose U * AV Σ is the SVD of m n matix A, whee ank[a]. et (7-7) descie colmn patitions of matices U and V. Then fo any the minimm nom X that minimizes AX is given y F I X i v i, (7-36) i HG i KJ whee i, i, ae the non-zeo singla vales of A. In addition, the minimm eo satisfies m AX i i + d i. (7-37) oof: Fo any X we have CH7.DC age 7-

12 EE448/58 Vesion. John Stensy CH7.DC age 7- AX AX A X m m U U U VV U U Σα, m m + +, m m i i i i i,, d i d i i m +, (7-38) whee α [α α... α n ] T V * X. Clealy, the "optimm" X that minimizes AX eqies that α i i i /, i. If we set αi fo + i n, then the eslting X has the minimm -nom. With this α, the optimm X ecomes

13 EE448/58 Vesion. John Stensy CH7.DC age 7-3 X v v v n i i i i F HG I KJ Vα / /, (7-39) and the smallest eo satisfies (7-37). sedo Invese Fom SVD ote that (7-36) can e witten as / / / m X v v v v i i i i n ( / ) Fom this epesentation, we can conclde that A + V Σ + U *, (7-4) / / / Σ +. ote that Σ + is otained fom Σ y simply inveting the non-zeo singla vales. Eqations (7-4) (7-4)

14 EE448/58 Vesion. John Stensy (7-4) and (7-4) ae sed y ata to calclate the psedo invese A +. Comptational Sensitivity of the sedo Invese The psedo invese A + can e difficlt to compte accately. It can e vey sensitive to ond-off eos (and othe small changes in A); A + does not always depend continosly on the enties of A. This fact can e seen easily fom inspection of (7-4) and (7-4). Fo some A with vey small non-zeo singla vales, small changes in the elements of A can change the ank of A and the nme of non-zeo singla vales. This can have a damatic inflence on the compted A +. Example Conside the matix A and the petation/eo δa given y A, δa ε. It is easy to compte A +, (A + δa) + / ε. ote that a small petation matix δa (i.e., small ε) added to A can case an extemely lage change in the psedo invese (A + δa) +. In pactice, we mst compae the small singla vales of A with some small toleance that eflects the eos in the data and/o comptation. We shold set to zeo those singla vales of A that fall elow some pescied toleance. This zeoes ot the coesponding /(singla vale) tems fond in (7-4). This is the appoach taken y ata's pinv fnction. The pinv fnction accepts a theshold vale as an agment (the theshold is called tol) when it ses (7-4) and (7-4) to CH7.DC age 7-4

15 EE448/58 Vesion. John Stensy calclate A +. Fnction pinv(a,tol) sets to zeo those singla vales that ae smalle than tol. Example Conside the following ata scipt % Ente atix A and petation da A [ ; ; ] A da [ ; E-6; ] da E-6 % Calclate pinv(a+da, E-5), se tol E-5 pinv(a+da,e-5) ans %Calclate pinv(a+da, E-6), se tol E-6 pinv(a+da,e-6) ans %Calclate pinv(a+da, E-7), se tol E-7 pinv(a+da,e-7) ans % Wow! What a change! Image Compession and SVD In some applications an image can e epesented as an m n matix A of eal nmes. ow, the matix A is "highly singla", and it contains a lot of edndant infomation (think of all the "white space" occpied y "nothing" in a typical image). Using SVD, it can e epesented in the seies min( m, n) A i i v i. (7-43) i ften, only a few of the i ae significant, and the expansion can e tncated to k tems with only some loss of claity. CH7.DC age 7-5

16 EE448/58 Vesion. John Stensy Fo example, conside the it map image illstated y Fig. 7-a. This image eqies ,536 eal nmes to epesent. The SVD of this image eveals singla vales that fall off apidly. Fige 7- is a log-scale plot of the singla vales of this image. As it tns ot 86 /.5 -. By setting k, 87 k 56, we etain only those singla vales that ae within 99.5% of (we tncate the smallest.5%). Bt we edce the stoage eqiements to ,8 eal Singla Vales k Fige 7-: og plot of singla vales. nmes; this podces a edction of appoximately 33% in stoage eqiements. Howeve, not mch was lost in the compessed image, as can e seen fom Fig. 7-. By setting k, 5 k 56, we edce the stoage eqiements to ,65 eal nmes; this podces a edction of appoximately 6% in stoage eqiements. Howeve, little was lost in the compessed image, as can e seen fom Fig. 7-c. By setting k, 6 k 56, we edce the stoage eqiements to ,85 eal nmes; this podces a edction of appoximately 8% in stoage eqiements. Howeve, now the degadation in the compessed image is ovios, as can e seen fom Fig. 7-d. By setting k, 6 k 56, we edce the stoage eqiements to ,695 eal nmes; this podces a edction of appoximately 88% in stoage eqiements. Howeve, now the degadation in the compessed image is vey ovios, as can e seen fom Fig. 7-e. Finally, y setting k, 6 k 56, we edce the stoage eqiements to ,565 eal nmes; this podces a edction of appoximately 96% in stoage eqiements. Howeve, now the image is not ecognizale, as can e seen fom Fig. 7-f. CH7.DC age 7-6

17 EE448/58 Vesion. John Stensy Fige 7-a: iginal image. Fige 7-: Expansion tncated to 86 tems. Fige 7-c: Expansion tncated to 5 tems. Fige 7-d: Expansion tncated to 5 tems. Fige 7-e: Expansion tncated to 5 tems. Fige 7-f: Expansion tncated to 5 tems. CH7.DC age 7-7