On Singular Value Decomposition of Rectangular Matrices

Transcription

1 -Najah Natoal Uversty Faculty of Graduate Studes O Sgular Value Decomposto of Rectagular Matrces By Shree Najeh Issa Odeh Supervsor Dr"Mohammad Othma" Omra Ths thess s submtted Partal Fulfllmet of the Requremets for the Degree of Master Scece Mathematcs, Faculty of Graduate Studes, at -Najah Natoal Uversty, Nablus, Paleste 9 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

2

3 Dedcato To my parets "Najeh ad Nahla" PDF created wth pdffactory Pro tral verso wwwpdffactorycom

4 v ckowledgemets There are may people who have drectly or drectly made t possble for me to cotue o ths jourey I am deeply grateful to my supervsor Dr "Mohammad Othma" Omra who wthout hs support, kd supervso, helpful suggestos ad valuable remarks, my work would have bee more dffcult lso, my great thaks are due to my famly for ther support, ecouragemet ad great efforts for me I would also scerely lke to thak my teachers ad freds for helpg me durg workg o ths thess PDF created wth pdffactory Pro tral verso wwwpdffactorycom

5 ا نا الموقع ا دناه مقدم الرسالة التي تحمل العنوان v الا قرار O Sgular Value Decomposto of Rectagular Matrces حول تحليل القيمة المنفردة للمصفوفات المستطيلة اقر با ن ما اشتملت عليه هذه الرسالة ا نما هي نتاج جهدي الخاص باستثناء ما تمت الا شارة ا ليه حيثما ورد وا ن هذه الرسالة ككل ا و ا ي جزء منها لم يقدم من قبل لنيل ا ية درجة علمية ا و بحث علمي ا و بحثي لدى ا ية مو سسة تعليمية ا و بحثية ا خرى Declarato The work provded ths thess, uless otherwse refereced, s the researcher's ow work, ad has ot bee submtted elsewhere for ay other degree or qualfcatos Studet's ame: Sgature: Date: اسم الطالب : التوقیع : التاریخ : PDF created wth pdffactory Pro tral verso wwwpdffactorycom

6 v Lst of Cotets Subject Dedcato ckowledgmets Lst of Cotets bstract Page III IV VI VIII Itroducto Hstory Basc Cocepts Matrx alyss 7 Vector Spaces Over C 7 Gram-Schmdt Orthoormalzato Process 4 Some Specal Matrces 6 Utary Matrces 6 Normal Matrces 8 Hermta Matrces 9 4 Egevalues ad Egevectors 5 Norms of Vectors ad Matrces 5 6 Codto Number 8 Smlarty ad Utarly Dagoalzato 9 Dagoalzato 9 Schur's Theorem 5 Spectral Decomposto 8 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

7 v The Sgular Value Decomposto 4 Defto ad Computato of the Sgular Value Decomposto 4 The Sgular Value Decomposto Versus the Spectral Decomposto 5 Matrx Propertes va SVD 57 4 Geometrc Iterpretato 6 4 pplcatos of the Sgular Value Decomposto 6 4 Moore-Perose Pseudoverse 6 4 Computg the Moore-Perose Pseudoverse 7 4 Lear Least Squares Problem 8 5 More pplcatos of the Sgular Value Decomposto 88 5 Low Rak pproxmato of Matrces 88 5 Image Compresso Usg the Sgular Value Decomposto 95 5 Determato of the Effectve Rak 6 Refereces 9 ppedces Notatos PDF created wth pdffactory Pro tral verso wwwpdffactorycom

8 v O Sgular Value Decomposto of Rectagular Matrces Prepared by Shree Najeh Odeh Supervsor Dr"Mohammad Othma" Omra bstract The sgular value decomposto of matrces stads as oe of the most mportat cocepts mathematcs, because of ts varety of applcatos mathematcs, statstcs, bology ad may other areas of scece I ths thess, we preset the sgular value decomposto ad ts relato to the spectral decomposto We also vestgate the sgular value decomposto of a matrx together wth some of ts applcatos Some of these applcatos clude the Moore-Perose psuedoverse, the effectve rak of matrces ad mage compresso PDF created wth pdffactory Pro tral verso wwwpdffactorycom

9 Itroducto The sgular value decomposto (SVD) plays a mportat role matrx theory Whle some decompostos are restrcted to real square matrces, the (SVD) ca be appled to ay rectagular matrx Through ths decomposto, we study some of the propertes of such as: the rak, the orm ad the bass of the four fudametal subspaces related to It also has may applcatos I umercal aalyss, the SVD provdes a measure of the effectve rak of a gve matrx I statstcs, the SVD s a partcularly useful tool for dg least-squares solutos ad approxmatos It has may applcatos : sgal processg, bology, statstcal aalyss ad mathematcal modelg I ths thess, two types of the (SVD) of are defed ad computed: the full ad reduced; ths decomposto s compared wth the well kow spectral decomposto of (wheever exsts); some propertes of the matrx va ts (SVD) are studed The (SVD) s used to compute the Moore-Perose pseudo verse that ca be used solvg a system of lear equatos ad ca gve the optmal soluto of the least squares problem whe solvg a overdetermed system; we also use the (SVD) to compute the best low rak approxmato accordg to ether the Eucldea or the spectral orm I the frst chapter, some prelmary deftos are preseted, as well as basc results ad propertes of matrces; some specal matrces (utary, ormal ad Hermta) are revewd Egevalues, orms ad computatos of the codto umber are also studed PDF created wth pdffactory Pro tral verso wwwpdffactorycom

10 I the secod chapter, the dagoalzato of matrces s studed ad that cludes Schur's theorem ad the spectral decomposto I the thrd chapter, the sgular value decomposto (SVD) s defed ad ts relato to the spectral decomposto s studed Some propertes of the orgal matrx are studed va ts SVD ad the geometrc terpretato of SVD s also troduced I the fourth chapter, the SVD s used to compute the Moore-Perose pseudoverse whch s used for solvg lear systems of equatos I the ffth chapter, the SVD s used other applcatos such as, low rak approxmato wth respect to a gve orm, mage compresso ad fdg the affectve rak of a matrx PDF created wth pdffactory Pro tral verso wwwpdffactorycom

11 Hstory The sgular value decomposto has a log hstory It was orgally developed the eteeth cetury by dfferetal geometers ad algebrasts who wated to determe, for gve matrces a ] ad B b ] (R), whether the two blear forms: Φ ( x, y ) [ j, j a j x y j ad [ j Φ B M ( x, y) b x y, j, j j could be made equal for every x [ x ] & y [ y ] R, uder depedet real orthogoal trasformato of the two spaces t acts o; e, does there exst Q Q M ( ) such that Φ Q x, Q y) Φ ( x, ), R B ( y for all x, y R? Ths problem could be approached by fdg a caocal form to whch ay such blear form ca be reduced by orthogoal substtuto, or by fdg a complete set of varats for a blear form uder orthogoal substtutos The Itala dfferetal geometer Eugeo Beltram dscovered 87 that for each real matrx M (R), there are always Q Q M ( ) such that σ ( ) ( ) () σ T Q Q Σ, σ ( ), R PDF created wth pdffactory Pro tral verso wwwpdffactorycom

12 4 T where σ ) σ ( ) σ ( ) are the egevalues of as well as ( T ; he also foud that the colums of Q are egevectors of colums of Q are egevectors of T T ad the Idepedetly, 874, the Frech algebrast Camlle Jorda came to the same caocal form but from a dfferet pot of vew He foud that the egevalues of the -by- real symmetrc matrx T are pared by sg ad that ts largest egevalues are the desred coeffcets σ( ),, σ ( ) of the caocal form Φ T ( ξ, η) σ ( ) ξη σ ( ) ξ η Q Q I 889/9 uaware of Beltram ad Jorda, James Joseph Sylvester gave a thrd proof to () for real square matrces ad he called the caocal multplers of the matrx σ ' s the matrx P M I 9 L-utoe proved that every o sgular complex M ca be wrtte as UP, where U M s utary ad s postve defte I 9/5 he retured to these deas ad used the smlarty of ad to show that ay square complex matrx M ca be wrtte as UΣV where U, V M are utary ad Σ M s a oegatve dagoal matrx He also dscovered that f s osgular Hermta the ca be wrtte as T UΣU for some utary U, ad a oegatve dagoal matrx Σ I 9 Emle Pcard call the umbersσ k 's sgular values PDF created wth pdffactory Pro tral verso wwwpdffactorycom

13 5 I 99 Eckart ad Youg gave the frst complete proof of the sgular value decomposto for rectagular complex matrx ad they dd't gve ay ame to the umbersσ k 's The exstece proof of the sgular value decomposto opes may ways for the mathematca to search for equaltes, propertes ad applcatos to ths decomposto Durg 949-5, a remarkable seres of papers the Proceedg of the Natoal cademy of Scece (US) establshed all of the basc equaltes volvg sgular values ad egevalues Oe of these papers s "Iequaltes Betwee the Two Kds of Egevalues of a Lear Trasformato", establshed by Weyl I 95 Poyla gave a alteratve proof of a key lemma Weyl's 949 paper( also, establshed by US) I 954, Hor proved that Weyl's 949 equaltes were suffcet for the exstece of a matrx wth prescrbed sgular values ad egevalues, ad ths paper he used the expresso "sgular values" the cotext of matrces I 954/55 practcal methods for computg the SVD date back to Kogbetlatz Hestees 958 resemblg closely the Jacob egevalue algorthm, cosθ s θ used plae rotatos or Gves rotatos, e, s cos However, these θ θ PDF created wth pdffactory Pro tral verso wwwpdffactorycom

14 6 were replaced by the method of Gee Golub ad Wllam Kaha (the reducto to bdagoal form) publshed 965, whch uses Householder trasformatos or reflectos; they troduce the SVD to umercal aalyss It s a fact that the QR algorthm for the sgular values of bdagoal matrces was frst derved by Golub 968 wthout referece to the QR algorthm, whch has bee the workhorse for two decade Recetly 99, Demmel ad Kaha have proposed a terestg alteratve for 968's Golub algorthm I the last years, the sgular value decomposto has become a popular umercal tool statstcal data aalyss, sgal processg, system detfcato ad cotrol system aalyss ad desg [ &] PDF created wth pdffactory Pro tral verso wwwpdffactorycom

15 7 Chapter Oe Basc Cocepts Matrx alyss I ths chapter we revew some prelmary cocepts ad deftos matrx aalyss ad preset some basc propertes related to these deftos Remark: I our thess, we deote by R the set of real umbers ad by C the set of all complex umbers C { x y, x, y R} Vector Spaces Over C Defto complex vector space V s a oempty set of elemets (called vectors) together wth two operatos: vector addto ad scalar multplcato Θ satsfyg the followg propertes: For all u, v ad w V, c ad d C, the: () a - u v V (e V s closed uder vector addto ) b- u v v u c- u v w) ( u v) w ( d- there s a elemet V such that u u u ( s called the addtve detty) e- u V such that u u ( u s called the addtve verse) () a- c Θ u V (e V s closed uder scalar multplcato Θ ) b- c Θ u v) cθu cθv ( c- c d) Θ u c Θ u d Θ u ( d- c ( d Θ u) ( cd) Θ u d Θ( cθu) Θ PDF created wth pdffactory Pro tral verso wwwpdffactorycom

16 8 e- Θ u u Note: a real vector space has the same defto as a complex vector space except that the costats are real umbers Example C wth the usual addto ad scalar multplcato s a complex vector space: For ay u,, w } C { u, u,, u}, v { v, v,, v}ad w { w, w u, v ad w C,,,, ad for ay c ad d C the : () a- u v { u v, u v,, u v } C, so C s closed uder addto b- u v u v, u v,, u v } { v u, v u,, v u } v u { c- u v w) { u ( v w ), u ( v w ),, u ( v w )} ( {( u v ) w,( u v ) w,,( u v ) w } ( u v) w d- (,,,) C s the addtve detty e- The addtve verse for u s u u, u,, } C { () a- c u { cu, cu,, cu } C, so C s closed uder scalar multplcato b- c u v) c{ u v, u v,, u v } { c( u v ), c( u v ),, c( u v )} ( { cu cv, cu cv,, cu cv } { cu, cu,, cu} { cv, cu,, cv} c u c v c- (cd)u cudu d- c(du)(cd)u e- uu Defto u PDF created wth pdffactory Pro tral verso wwwpdffactorycom

17 9 subspace U of a vector space V over C s a oempty subset of V whch s by tself a vector space over C wth respect to the operatos o V Example T U {( a, b,) : a, b R} s a subspace of R whch s a real vector space Theorem oempty subset U of V s a subspace of V f U s closed uder the same operatos ad Θo V Defto set of vectors x, x,, } a vector space V s sad to be learly { xk depedet over C f there exsts coeffcets a, a,, a k C ot all zero, such that a x a x x a k k set whch s ot learly depedet s sad to be learly depedet Theorem Let x, x,, x } be learly depedet vectors C ad P a by { k osgular matrx the P x, P x,, P x are also learly depedet vectors C Defto 4 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

18 Let ( ) The rak of deoted by r(), s the umber of M m, C learly depedet colums or rows of, where M ( ) deotes all m by matrces wth etres from C m, C Note: rak rak ( where s the cojugate traspose of e, f [ a ], a C, the [ a ]), where " " deotes the cojugate Defto 5 elemet S ( e j j j subset S x, x,, x } of a vector space V s sad to spa V f every f { v V ca be represeted as a lear combato of the elemets of v V Spa S V, a, a, a C such that v ax ax a x ) We the wrte Defto 6 oempty subset S of a vector space V s sad to be a bass of V, f t's both learly depedet ad f t spas V The umber of elemets of elemets of S (a bass) s called the dmeso of V, deoted by dm V Remark Most of our work the thess wll be over fte dmesoal vector spaces, uless otherwse stated Note: If the vectors v,,, v vk form a bass for a vector space V, the they must be dstct ad ozero, so we wrte them as a set{ v v,, } Defto 7, v k PDF created wth pdffactory Pro tral verso wwwpdffactorycom

19 x y x y Let x y, C The dot product of x ad y s defed as M M x y y y x y x y [ x x x ] L x y x y x y M y The legth of x s deoted by x x x Defto 8 Two vectors x ad y C are sad to be orthogoal f xy Two subspaces U ad V of a vector space are sad to be orthogoal f { u v, u U ad v V} Defto 9 Two vectors x ad y are sad to be orthoormal f x ad y are orthogoal ad of ut legth each Defto set of vectors { x, x,, x k } C s sad to be orthogoal f they are parwse orthogoal If addto each x has a ut legth, (e x,,,, k ) the { x, x, x k} s a orthoormal set Defto PDF created wth pdffactory Pro tral verso wwwpdffactorycom

20 Let V be a subspace ofc subspace U C s sad to be the orthogoal complemet of V C f every vector u U s orthogoal to every v V ad every f vector v V s orthogoal to everyu U V s also called the orthogoal complemet of U Example 5 4 W Spa, spa{u,u } ad W spa spa {t } 4 7 are orthogoal complemets of each other R, sce u t, u t The, every elemet W s orthogoal to every elemet W Defto Let ( ) ad suppose r()rak()r The there are four M m, C fudametal subspaces related to : m - The Rage of { y C x C, x y} space of, ad dm (rage ) r - The Null space of { y C : y } the orthogoal complemet of rage m - Rage { y C : x C, x y} rowspace of 4- The Null Space of { y C : y } : It s also called the colum m ad ts dmeso m-r It s wth dmeso r It s also called orthogoal complemet of rage wth dmeso-r It s the Example 4 Let PDF created wth pdffactory Pro tral verso wwwpdffactorycom

21 , Compute the four fudametal subspaces assocated wth Soluto: By elemetary row operatos, the reduced row echelo form of (wrtte RREF()) s ) ( B RREF The - rak() -The bass of the colum space of s 7 5, 4 S - Now to fd the bass of the ull space of, fd, ga, by elemetary row operatos o, we obta 5 7 ) ( C RREF Solvg the lear system C x, we get 4 S s the bass of the ull space of PDF created wth pdffactory Pro tral verso wwwpdffactorycom

22 4 S ad S' are orthogoal complemets of each other R 4- {[ ], [ ] } T s a bass for the row space of 5- To fd the bass of the ull space of, solve the lear system Bx to get T, s a bass for the ull space of Theorem orthoormal set of vectors s learly depedet Gram-Schmdt Orthoormalzato Process I our work we requre learly depedet sets to be orthoormal We ca covert a learly depedet set to orthoormal set may ways Oe smple way to obta a orthogoal set from learly depedet set s the Gram-Schmdt process Let S u, u,, u } be a set of learly depedet vectors a { complex vector space V the the followg are the steps of the Gram- Schmdt Orthoormalzato Process - Let v u - Compute the vectors v u - k u v ( v k k ) v k,,, The vectors {v,v,,v } form a orthogoal set v - Let w v, The T { w, w,, w} s a orthoormal set of vectors PDF created wth pdffactory Pro tral verso wwwpdffactorycom

23 5 Example 5 Obta a orthoormal set from S,, Frst, these vectors are learly depedet - Let v (,,,) T - Compute v u - u vk ( ) v k, k vk u v v u - ( ) v (,,,) T 9 - (,,,) T (,,, ) T or (,4,,-) T v 6 u v v u - ( ) v v u v - ( ) v v v (,,,) T - (,,,) T T - (,4,,) 8 you ca see that v,v ad v are parwse orthogoal 4 (,,, ) T or (4,-,,-4) T v - Compute w, we obta w (,,, ) v T T w (,,, ) ad w (,,, ) respectvely Note Ths process may be appled to ay fte or coutable set of vectors (ot ecessary learly depedet) I ths case at least oe of the v 's wll equal zero, ad the set {v,v,,v } wll ot be orthoormal T, Example 6 Obta a orthogoal set from T,, Soluto: PDF created wth pdffactory Pro tral verso wwwpdffactorycom

24 Note that 6, so T s learly depedet; thus we ca't covert them to a orthoormal set, but orthogoal oly: - Let v (,, -) T - Compute v u - u vk ( ) v k, k vk 9 7 T T v (,, ) ad v (,,) These vectors are orthogoal but v s ot a ut vector Some Specal Matrces Utary Matrces Defto matrx U M (C) s sad to be utary f U UI If U s real the U s called orthogoal We have some mportat theorems: Theorem 4 Let U M (C) The followg are equvalet: - U s utary - U s o sgular ad U U (where U deotes the verse of U) - UU I 4- U s utary 5- the colums of U form a orthoormal set C 6- the rows of U form a orthoormal set C 7- U preserves legth, e, f y U x, wth x C, the y x It s easy to prove ths theorem ad we prove oly the last statemet PDF created wth pdffactory Pro tral verso wwwpdffactorycom

25 7 Suppose U s utary the y y y ( U x) ( Ux) x U Ux x x x Example 7 Cosder the utary matrx ad the 5 vector x The, x We have: Ux wth Ux Theorem 5 If U s utary the det(u) ± Theorem 6 If U ad V ( C) are utary so s the product UV M Specal cases of utary matrces are the permutato matrces Defe as follows: Defto 4 square matrx P s a permutato matrx f ts colums are a permutato of the colums of I Example 8 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

26 8 s a permutato matrx Normal Matrces Defto 5 matrx M (C) s sad to be ormal f It s obvous that utary ad dagoal matrces are ormal Example 9 Let 5 The s ormal sce Hermta Matrces Defto 6 matrx M (C) s sad to be Hermta f If s real the s sad to be symmetrc It s skew Hermta f It s obvous that Hermta ad skew Hermta matrces are ormal Theorem 7 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

27 9 Let M (C) be Hermta The: - T,, are all Hermta ad f addto, s o sgular the - s also Hermta - x x s real for all x C Theorem 8 Let ( ) the ad are Hermta matrces M m, C Remark The ma dagoal etres of a Hermta matrx are all real Defto 7 Hermta matrx M (C) s sad to be postve defte f x x > for all ozero x Cⁿ " It's postve sem defte f x x "(ote that sce s Hermta the x x s real ) Remark If ( ) the ad are postve sem defte If has M m, C learly depedet colums the s postve defte Proof x ( ) x ( x) x x, x So s postve sem defte If addto has depedet colums the x >, for all x x ( ) x ( x) x x >, x PDF created wth pdffactory Pro tral verso wwwpdffactorycom

28 Example Let, the Therefor gve wll x x Now, 5 [ ] [ ] although, 5 x But [ ] ) ( ad 5 b b a b ab a b a b a except f both a ad b 4 Egevalues ad Egevectors Defto 8 Let ) M (C The umber C λ s called a egevalue of, f there exsts a o zero vector C x such that x x λ I ths case, x s called egevector of assocated wth the egevalue λ The set of all egevalues of ) M (C s called the spectrum of ad s deoted by () Λ Theorem 9 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

29 λ the: If x ad y are egevectors of M (C) assocated wth the egevalue ) If α x β y s ot the zero vector, the α x β y s also a egevector of assocated wth the egevalue λ ) If s also Hermta the all ts egevalues are real ) If s also postve defte the ts egevalues are postve Example Let The s Hermta ad ote that so ( ) ( ) ( ) x s a egevector of assocated wth the egevalue λ also x s a egevector of assocated wth the egevalue λ Thus Λ( ) {,} Defto 9 Let M (C) The f ( ) det( ti t t a a ) det a a t a a a t a s called the characterstc polyomal of PDF created wth pdffactory Pro tral verso wwwpdffactorycom

30 Theorem The egevalues of are the roots of the characterstc polyomal of Thus to fd the egevalues of a gve matrx we must fd the roots of the characterstc polyomal The the correspodg egevectors are obtaed by substtutg the values of λ the system of equatos ( λ I ) x ad solvg the resultg system So, the egevectors of correspodg to λ spa the ull space of ( λ I ) The characterstc polyomal ca be wrtte as the product of factors each of the form λ λ ) ( polyomal so we wrte f (λ) t where λ s a root of the characterstc k k kr ( λ λ ) ( λ λ ) ( λ λr ), where λ,,,, r are the dstct egevalues of, ad k are tegers whose sum s ad whch s called the algebrac multplcty of λ,,,, r Each egevalue has also a geometrc multplcty whch s defed as the dmeso of the subspace spaed by ts egevectors egevalue s smple f ts algebrac multplcty s oe ths case the algebrac ad geometrc multplctes are equal It s easy to show that the algebrac multplcty of each egevector s greater tha or equal to the geometrc multplcty Example Let Its characterstc polyomal s PDF created wth pdffactory Pro tral verso wwwpdffactorycom

31 λ f ( λ) ( λ ) ( λ )( λ ) λ Hece the egevalues are Usg the frst egevalue λ ad, λ λ ad substtutg ( λi ) x gves x x the secod row gves x x So, (, ) T s a egevector correspodg to λ The same argumet wth λ gves the correspodg T egevector (, ) Defto The set of egevectors correspodg to a egevalue λ togother wth the zero vector form a subspace of C kow as the egespace of λ Defto Let [ a j ] M (C) a The the trace of s defed as Theorem Let M (C) The the egevalues of are the complex cojugate of the egevalues of e, f λ Λ() the λ Λ( ) It follows from ths theorem that for ay matrx ( ), ad have the same o zero egevalues M m, C Theorem PDF created wth pdffactory Pro tral verso wwwpdffactorycom

32 4 Let M (C) be Hermta The t has asset of orthogoal egevectors Example Cosder the Hermta matrx example, whch ( ) has egevectors x ad x, to show that these vectors are learly depedet, we arrage them as colums a matrx the trasform t to row echelo form: REF Sce each colum has a leadg oe, so these two vectors are learly depedet 5 Norm of Vectors ad Matrces Oe way to measure the sze of vectors ad matrces s to study the orm, so what s the orm? Defto Let V be a vector space over the feld of complex umbers, a fucto : V R s a vector orm f for all x,y V - x - x ff x - c x c x for all scalars c C 4- x y x y PDF created wth pdffactory Pro tral verso wwwpdffactorycom

33 5 Note: fucto that satsfes axoms, ad 4 of defto s called sem orm Example 4 - The Eucldea orm (or l orm) o C s x x x x ( x x ) where x ( x, x,, x x x ) - The sum orm ( or l orm) o Cⁿ s T x x x x where x ( x, x,, x ) T Defto orm s sad to be utarly varat f U x x for all x Cⁿ ad all utary matrces U M (C) Example 5 The l orm s utarly varat Defto 4 fucto : M ( ) R s a matrx orm f for all, B, ( C) m, C - - ff - c c, for all complex scalars c 4- B B tragle equalty 5- B B (f m) sub multplcatve M m Remark PDF created wth pdffactory Pro tral verso wwwpdffactorycom

34 6 By 5 of defto 4, for ay ozero matrx for whch ², we have that ; ths s because I partcular I for ay matrx orm, so f s vertble the I for ay matrx orm Example 6 F The Eucldea orm (l or Frobeus orm) o M ( ) s defed as m j a j m, C Note that the vector orms the Eucldea orm s deoted by x whle the matrx orms the l orm s deoted by F Example 7 The spectral orm s defed o M ( ) by m, C max { λ : λ s a egevalue of } s defed sce s postve sem defte ad so all ts egevalues are o egatve Note x x, where ( C) ad x C M m, Defto 5 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

35 7 The er product of ad B, ( C) s defed as B tr m B ) j, F M m ( a b j j We the wrte the Frobeus orm as Theorem M Let be a gve orm o C ad let α x α, ( C) The α max, x x α m be the matrx orm o α Defto 6 matrx orm s sad to be utarly varat f UV for all ( ) ad all utary matrces U ad V M m, C For stace both the Spectral ad Frobeus orms are utary varat e, UV ad F UV F for all utary matrces U ad V 6 Codto umber It's a measure for sgularty defed as follow: Defto 7 The codto umber of a square matrx wth respect to a gve matrx orm s de ed as: PDF created wth pdffactory Pro tral verso wwwpdffactorycom

36 cod( ) 8, s osular, s sgular The followg theorem gves us some mportat propertes of the codto umber: Theorem 4 For ay matrx, cod( ) For the detty matrx, cod (I) 45 For ay matrx ad ozero scalar α, cod( α ) cod( ) Most of the materal of ths chapter ca be foud [] ad [5] PDF created wth pdffactory Pro tral verso wwwpdffactorycom

37 9 Chapter Two Smlarty ad Utarly Dagoalzato We start ths chapter wth the defto of matrx dagoalzato Dagoalzato Defto matrx B M (C) s sad to be smlar to a matrx M (C) f there exsts a o sgular matrx P such that P BP We say B s smlar to va P We also call P the matrx of smlarty betwee ad B If B s smlar to the s smlar to B So we ca smply say ad B are smlar If P s utary the ad B are sad to be utarly smlar Example The matrx 8 B s smlar to 8 7 sce: P B P PDF created wth pdffactory Pro tral verso wwwpdffactorycom

38 Defto matrx B M (C) s sad to be dagoalzable (or ca be dagoalzed) f t s smlar to a by dagoal matrx We the wrte B P DP, where D s dagoal Example 8 The matrx B Example s dagoalzable sce t s 8 7 smlar to the dagoal matrx Theorem If a matrx M (C) s dagoalzable the t has learly depedet egevectors Proof Let M (C) be dagoalzable The there exsts a o sgular matrx P M (C) ad a dagoal matrx Λ M (C) such that PΛP or P PΛ Let P [ x x ] wth x C ad Λ dag( d ) where d C, the x [ x x ] [ x x ] Comparg the left had sde wth the rght had sde colum by colum we have x d d x Sce P s o sgular the ts colums are learly depedet ad oe of them s zero The, by defto of the d PDF created wth pdffactory Pro tral verso wwwpdffactorycom

39 egevectors, the colums of P are learly depedet egevectors of correspodg to the egevalues d Defto matrx ) M (C s utarly dagoalzable f t s dagoalzable va a utary matrx Example Let 5 5 ad let P The P s utary ad Λ P P 5 5 So, s utarly dagoalzable Defto 4 egevalue ) ( of C M λ s called defectve f ts geometrc multplcty s less tha ts algebrac multplcty matrx s defectve f t has a defectve egevalue Otherwse s o defectve Note If a egevalue s smple the t s o defectve PDF created wth pdffactory Pro tral verso wwwpdffactorycom

40 Example 4 The matrx - - s o defectve sce ts egevalues are, ad ad they are all smple Theorem Smlar matrces share the same egevalues wth the same algebrac ad geometrc multplctes Proof Suppose ad B are smlar, so there exsts a o sgular matrx P such that B P P The P t ( B)det( λ I B) det( λi P det( P P) det( P λp P )det( λ I )det( P) det( λi ) P ( ) t P) det( P ( λi ) P) Ths meas that both ad B have the same characterstc polyomal ad so they have the same egevalues (roots) wth the same algebrac multplcty If x s a egevector of assocated to λ, the x λx ad PBP x λx whch gves B( P x) P λ x λ( P x), e, P x s a egevector of B assocated to λ ; hece, by theorem, ad B have the same geometrc multplcty Note The zero matrx ( C) s o defectve sce t has oly oe M egevalue (zero) wth algebrac multplcty ad e,,, e e are ts PDF created wth pdffactory Pro tral verso wwwpdffactorycom

41 egevectors So the geometrc multplcty of the zero egevalue s ; so t's o defectve Remark y dagoal matrx s o defectve Example 5 Let The } {, ) ( Λ wth algebrac multplctes ad, respectvely To fd the geometrc multplcty for, we solve ) ( x I 5 whch gves 5 4 x x x x x The soluto of ths equatos are 4 x x ad 5 ad, x x x are free varables so we ca choose the egevectors assocated to to be e, e ad e 5 So, the geometrc multplcty for s I the same way we show that the geometrc multplcty for s So s o defectve PDF created wth pdffactory Pro tral verso wwwpdffactorycom

42 Theorem 4 Let M (C) The s o defectve f ad oly f t s dagoalzable Proof Suppose s o defectve the t has learly depedet egevectors, x,,, x x Sce x λ x the [ x, x,, x ] [ λx, λx,, λx] ad hece λ [ x, x,, x ] [ x, x,, x ] D So, we have P PD ad hece λ λ P P D, where D dag{ λ,, λ } λ Coversely, Suppose P P D, where P x, x,, x ] ad [ d D d The s smlar to D ad by Theorem have the same egevalues as D, amely d,,d, wth the same algebrac ad geometrc multplctes But D s dagoal ad thus by the remark above t s o defectve ad so s Not all matrces are dagoalzable, see the followg example: PDF created wth pdffactory Pro tral verso wwwpdffactorycom

43 5 Example 6 The matrx s ot dagoalzable To see ths, ote that s a egevalue of wth algebrac multplcty, but the egevectors assocated to are (r,) T Hece, the geometrc multplcty of s ad so t s defectve ad so s ot dagoalzable Schur's Theorem I the prevous secto we showed that ot all square matrces are dagoalzable I ths secto we prove that all square matrces are utarly smlar to a upper tragular matrx Theorem 4( Schur's Theorem): Gve M (C) wth egevalues λ, λ,, λ, there s a utary matrx U M (C) such that U U T [ ], where T s upper tragular, wth t λ,,, [], Proof t j Let x be a ut egevector assocated to λ, so x λx Sce x s ot zero we may use Gram Schmdt orthoormalzato process to exted T {x } to a orthoormal bass{ x, z, z,, z } of C The for the utary matrx U [ x z ] z PDF created wth pdffactory Pro tral verso wwwpdffactorycom

44 6 [ ] U U B z z x z z x λ The matrx ) ( C M, has egevalues λ λ λ,,, We fd a ormalzed egevector C x of correspodg to λ, ad the exted T {x } to a orthoormal bass{ x, w,, w }of C Determe a utary matrx ) ( C M U, where U U λ Let I U V where I s the by detty matrx The V ad U V are utary ad ) ( V U U V B V V λ λ Coutue ths reducto to produce utary matrces,,,, ), ( &,,, ), ( M V M U C C the the matrx V U V V U s utary ad U U t j λ λ yelds the desred form Example 7 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

45 7 Let The egevalues of are, ad ad the ormalzed egevector correspodg to s x T ),, ( Expad T {x } to a bass of C, we obta { },, e e x,, pplyg Gramschmdt orthoormalzato process to these vectors to get, 6 6 6,, the the utary matrx U s such that U U 5 ) 6 ( 6 where ) ( 5 C M ad ts egevalues are ad The ormalzed egevector assocated wth s x T ), ( Let T {x } ad repeat the same steps as above to fd U PDF created wth pdffactory Pro tral verso wwwpdffactorycom

46 8 So, U U ad I U V Let U V U The U U T s upper tragular matrx wth t λ Note Nether U or T the theorem s uque Schur s Theorem says that every square matrx s smlar to a upper tragular matrx va a utary matrx I the ext secto, we cosder smlarty of a matrx to a dagoal matrx va a utary matrx what s kow by the spectral decomposto theorem Spectral Decomposto Lemma y upper tragular ormal matrx must be dagoal Proof Let ] [ j t T ) M (C be upper tragular ad ormal The t j for >j ad TT T T Comparg the dagoal etres of both sdes, we obta k t k t k T T ) ( k k k t t TT ) (,,,, PDF created wth pdffactory Pro tral verso wwwpdffactorycom

47 9 By usg the fact that T s upper tragular we have: ( T T) tk tk tk t tk t t k k k k k ( TT ) t t t t t () k k k The equalty of () ad () gves: k k k t k t k, k > () Sce t k, k > (T s upper tragular) ad from (), t, we have: ( T T) tk t t tk t k k d by () k k k ( TT ) t t t t t t (5) k The equalty of (4) ad (5) gves: k k k t ad hece t k, k k > Cotug the same way, we obta t j for all j >,,,, () (4) So T s a dagoal matrx Lemma Let M (C) be smlar to a matrx T va a utary matrx U The s ormal f ad oly f T s ormal Proof: Let be ormal ad T U U, where U s utary The T T U UU U U U U U U UU U TT so, T s ormal Coversely, If T s ormal tha where UTU PDF created wth pdffactory Pro tral verso wwwpdffactorycom

48 Theorem 5 (Spectral Theorem for Normal Matrces) 4 Let M (C) The s ormal f ad oly f t's utarly dagoalzable [], [] Proof Let M (C) wth egevalues λ, λ,, λ The by Schur's theorem there exsts upper tragular matrx T U U,where U s a utary matrx Sce s ormal the by Lemma, T s ormal ad by Lemma, T s dagoal D So, s utarly dagoalzable Coversely, let M (C) be utarly dagoalzable the UDU, for some dagoal D ad utary U Sce D s dagoal the D s ormal ad hece by Lemma, s ormal By ths theorem, oly the ormal matrces are utarly dagoalzable Example 8 Let The ad Hece, s ormal We ow show s utarly dagoalzable The egevalues of are ad (,) T ad ( ), wth correspodg egevectors T,, respectvely [ see Example Chapter ] The egevectors (,) T utary matrx dagoalzable ad ( ) T, are orthogoal ad so we have the U ad U U So s utarly PDF created wth pdffactory Pro tral verso wwwpdffactorycom

49 4 Corollary Let UDU be the spectral decomposto of The k k UD U s ay o egatve teger (wth detty matrx) proof k ( UDU )( UDU )( UDU ) ktmes k DU UD U ktmes U DD, where k So, f s ormal the,, k, are all ormal PDF created wth pdffactory Pro tral verso wwwpdffactorycom

50 4 Chapter Three The Sgular Value Decomposto Ths chapter s the ma topc of our thess "The Sgular Value Decomposto (SVD)" We troduce ts defto, vestgate ts proof of exstece ad clarfy ts relato to the spectral decomposto We also vestgate some of ts propertes Defto ad Computato of the SVD The spectral decomposto of a matrx as show Chapter exsts oly for ormal matrces We geeralze ths decomposto to ay matrx usg the SVD I ths secto we restrct our atteto to the defto of SVD, the way of computg t ad to the proof of ts exstece Defto Let M m, ( C) o egatve real umberσ s sad to be a sgular value m for f there exsts two ut legth vectors u C ad v C such that v σu ad u σv see [, 6] Example s a sgular value of sce: 6 6 ad PDF created wth pdffactory Pro tral verso wwwpdffactorycom

51 4 Theorem ( The Sgular Value Decomposto) S Let, ( C), wth m ad rak r The ca be wrtte as M m UΣV, where M m (C) S U ad V M (C) are utary, Σ M m, ( C), dag( σ, σ,, σr, σr,, σ ), σ σ σr > σr σr σ Mm, ( C) ad σ 's are the square roots of the egevalues of [, 4] Proof Let, ( C) wth m The ( C) s Hermta so t s ormal Let M m λ,, orthoormal egevectors The, λ λ be the egevalues of, wth assocated v,, v λ v λ v,,, wth assocated egvector v Now, So, M, v v, e, v λ v,,,, So, λ s also a egevalue of v λ v λ ( v ) ( v ) v v v v λv v λ v λ, whch we deote by σ Sce rak r rak umber of o zero egevalues of, v theσ, r Defe u,,,, r The u s a ut vector sce σ v σ u I addto, u, u,, u r are orthoormal sce σ σ v v j v v j vλ jv j λ jv v j u u j ( ), for j σ σ σ σ σ σ σ σ Now, j j j j λ j j λ j j v v v v v u v j ( ) v j v v j,j,,,r σ σ σ σ σ j j sce v,,, v vr are orthoormal Wrtg all these equatos for,j,,,r matrx form gves: PDF created wth pdffactory Pro tral verso wwwpdffactorycom

52 44 u u u r [ v v ] vr σ σ σ r The orthoormal vectors ad ca expad to a bass u,,, u u r form a m dmesoal subspace of C m, u,,, u,, ur, ur, ur um of C m Sce σ for all j > r the, u (because v σ ad so, j v j v ) for > r ad j r the v σ u ad so u v u σ u, > r j j So, we have U V where u u u m σ [ v v ] v j j j j j j σ j j S Σ, σ S dag( σ, σ,, σr, σr,, σ ), σ, r,, ad Mm, ( C) Defg U [ u u u m ] M ( ) ad V [ v v ] M ( c) m C v, the both U ad V are utary ad U V Σ, hece UΣV as requred Notes o the proof of ths theorem: Ths decomposto ca be appled to all rectagular complex matrces, ad f m we compute the SVD of ad Σ are m matrces The square roots of the egevalues of are the sgular values of To see ths, sce UΣV, the we have V UΣ whch we get PDF created wth pdffactory Pro tral verso wwwpdffactorycom

53 45 v σ u,,,, Smlarly, U VΣ Hece u σ v,,,, ad u, > 4 Rak r umber of ozero sgular values of 5 The matrx Σ s uquely determed 6 The colums of U are orthoormal egevectors of ad are called left sgular vectors ad the colums of V are orthoormal egevectors of ad are called rght sgular vectors Example Let The ad The egevalues of are, ad ther assocated egevectors are T (, ) ad (,) T, respectvely Sce these egevectors are orthogoal, defev So V s a utary matrx whose colums are orthoormal egevectors of lso, we fd the egevalues of 4 ether by calculatos or 4 drectly sce they are the same as the egevalues of plus - zero egevalues So we have, ad as smple egevalues of the symmetrc matrx ad hece ther assocated egevectors (,,) T, T (,,) ad,, 5) T (, respectvely, are orthogoal Dvdg each egevector by ts legth ad by orderg these ut egevectors a matrx PDF created wth pdffactory Pro tral verso wwwpdffactorycom

54 46 decreasg order accordg to the assocated egevalues we obta the utary matrx U as 6 5 U Now, to fd Σ we take the postve roots of the ozero egevalues ad populate them o the dagoal of Σ a decreasg order σ ad σ So, we have S, Σ ad 6 5 UΣV Example Let B Sce m <, we the cosder B 9 9 Now, ad ts egevalues are 8 ad (so the sgular values of 9 9 the matrx s σ 8 ad σ ), wth assocated orthoormal egevectors, ) T v ( ad v (, ) T O the other had, the matrx PDF created wth pdffactory Pro tral verso wwwpdffactorycom

55 has egevalues 8, ad wth assocated egevectors (,,) T, T ),, ( ad T,), (, respectvely These egevectors are ot orthoormal, so by Gram Schmdt orthoormalzato process we obta T ),, ( u T ) 5,, 5 ( u ad T ) 5 4, 5 5, 5 ( u as orthoormal egevectors of Let V [v v ] ad U[u u u ] the B d hece B I the followg example we use Matlab to determe the SVD of Note that Matlab, D s used stead of Σ PDF created wth pdffactory Pro tral verso wwwpdffactorycom

56 48 Example 4 ( wth Matlab) >> [U,D,V]svd() U -5/9-4/8 9/7-57/799-85/84-75/57-587/56-55/ /7 59/97-75/8-5/88-686/667 95/9-9/84 9/548-64/7-76/99-5/58 -/97-697/6 969/ /844-57/96 -/84 D 65/477 56/ /88 94/9 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

57 49 V -4/48-98/49-468/775 -/756-5/87 7/756 45/ 9/ /87 /749-5/ /65 74/46-77/5 /9 47/577 Theorem Let, ( C), m wth sgular values σ, σ,, σ } of The, M m { ad T have the same sgular values of [, ] Proof Cosder the SVD of The VΣ U, where UΣV where Σ Σ [ ] have the same sgular values σ, σ,, σ } S ad S dag( σ, σ,, σ ) S, V ad U are utary So ad { Smlarly, (sce T U ad U are utary for ay utary matrx U) we show that ad T have the same sgular values of Theorem If s real the U ad V ca be chose to be real [4] Proof Sce s real the T T ad are symmetrc ad both have real egevalues We the choose the egevectors of T T ad to be real PDF created wth pdffactory Pro tral verso wwwpdffactorycom

58 5 The SVD decomposto that has bee foud the examples above s the full SVD Some applcatos requre a faster ad more ecoomcal SVD We ow cosder reduced versos of the SVD: th ad compact SVD - Th SVD Wth, ( C), wth m, the full SVD of s: M m S Σ We ca wrte U as [ U U m- ] where U M m,, U m M m, m UΣV, where Notg that the elemets of U m- wll multply the zero elemets of Σ, the wrte as U SV Ths verso of SVD s called the th SVD, where V s utary but U s o more utary, but t has the property that U U I (e, oly has orthogoal colums) So, the th SVD we oly eed to calculate the frst colums of U, ad obvously t s faster tha the full SVD especally whe m >> It s easy to see that f s square, the the full ad the th SVD are the same Example 5 Cosder as Example ts full SVD was gve by PDF created wth pdffactory Pro tral verso wwwpdffactorycom

59 5 Sce - so we delete the last colum of U, wth S we get the th SVD of Example 6( wth Matlab): >> [U,S,V]svd(,) U -44/6-88/59-87/977-65/47-7/656 65/465 7/58 45/49-59/79-789/ /765-6/577-69/98 6/79-7/99-66/865 5/685 49/ -69/ 44/75 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

60 5 S 49/7 57/55 9/7 4/7 V -8/4995 4/57-9/478-99/ /5-879/969 8/6 -/685-74/ 7/59 9/59 67/895-59/ /88-765/8 /5 Note SV U [ ] u u u σ σ σ v v v Where u ad v,,,, are the colums of U ad V respectvely The the outer product sum s defed as: [u,u,,u ] v v v σ σ σ r v u v u σ σ, sce r k k,,, σ [6] PDF created wth pdffactory Pro tral verso wwwpdffactorycom

61 - Compact SVD 5 If, ( C) wth m ad rak r The the compact SVD of M m s U S V, where r r r S r σ σ, U r has oly the frst r colums of U σr ad V r has oly the frst r colums of V correspodg to the o zero sgular values of Ths s the secod type of the reduced SVD, whch s the same as the th SVD f has full rak, but f r << the ths decomposto wll be faster tha both the full ad the th SVD The compact SVD s sometmes called the ecoomy verso of the SVD d t ca be calculated by matlab wth the order [U,S,V]svds(,(rak())) Example 7 Cosder Example 5, whch has full rak, the the compact SVD s the same as the th SVD whch meas that t has the same decomposto as Example 5 The Sgular Value Decomposto Versus the Spectral Decomposto I ths secto Let M (C) be ormal, ad cosder ts spectral decomposto UDU ad ts SVD VΣW wth Σ dag( σ, σ,, σ ), D dag( λ, λ,, λ ) ad U, V ad W M (C) are utary PDF created wth pdffactory Pro tral verso wwwpdffactorycom

62 54 Note We have show that f s real the V ad W ca be chose to be real But eve f s real the the spectral decomposto may ot be real But what s the relato betwee the spectral decomposto ad the SVD, ad whe are they equal? Theorem 4 If M (C) s ormal The ts SVD s gve by U D ( U E) where D dag ( λ, λ,, λ ), E dag (e ϕ, e ϕ,, e ϕ ) where λ k k k ϕk λ, ϕ ( π, π ) e, are the egevalues of [] Proof Sce s ormal the ts spectral decomposto s UDU For each ϕk egevalue λ of, λ λ, ϕ ( π, π ) k k k e So D dag( λ e ϕ, λ e ϕ,, λ e ϕ ) ad hece D dag( λ, λ,, λ ) dag (e ϕ, e ϕ,, e ϕ ) Let D dag ( λ, λ,, λ ) ad E dag (e ϕ, e ϕ,, e ϕ ) The U D EU U D ( U E) s the SVD of wth Σ D, V UE ad the colums of U are orthoormal egevectors of Remark If M (C) s ormal the we have σ λ,,,, PDF created wth pdffactory Pro tral verso wwwpdffactorycom

63 55 Example 8 Cosder the matrx of Example 8 Chapter We kow that s ormal ad ts spectral decomposto s: Sce ), ( ), ( ), ( 4 4 π π λ λ e e, the ), dag( D ad )) ( ), ( ( ), ( 4 4 e e E dag dag π π So, ) ( E U D U ) ( ) ( V UΣ s ts SVD Note If we defe sg < f f ) sg( ) ( λ λ λ λ, ad f addto of ormal, s Hermta the all ts egevalues are real so, E E ad hece e k ϕ ± Hece, E dag( sg( λ ), sg( λ ),, sg( λ ) ) where sg() ad ) ( UE D U EU D U s the sgular value decomposto of the Hermta matrx PDF created wth pdffactory Pro tral verso wwwpdffactorycom

64 56 If s also postve defte the all the egevalues are real ad postve ad EI ad ths case the spectral decomposto ad the SVD are the same Example 9 Fd the SVD of 5 Soluto: The egevalues of are 87/475, 687/9 ad /49 are all postve real umbers the the matrx s postve defte so the spectral decomposto ad the SVD are the same The spectral decomposto of s UDU where - 58/787-59/9-47/8 /49 U 47/8 58/787 59/9 ad D 687/9 59/9 47/8 58/787 87/475 Remark If UΣV s the SVD of the: UΣV VΣ U UΣΣ U VΣ U UΣV VΣ ΣV are the spectral decomposto of ad, respectvely PDF created wth pdffactory Pro tral verso wwwpdffactorycom

65 57 Matrx Propertes va SVD The power of SVD comes from all the formato that ca be gleaed from t I ths secto we show these formato For the ext theorems (see []), let, ( C) wth m, rak r M m wth sgular values σ,,, σ σ Theorem 5 Gve the SVD of as UΣV the: The sgular vectors u, u,, ur form a orthoormal bass for Rage The sgular vectors v r, vr,, v form a orthoormal bass for Null The sgular vectors v, v,, vr form a orthoormal bass for Rage 4 The sgular vectors u r, ur,, um form a orthoormal bass for Null Proof By Defto wrte v () If σ, the u σ UΣV as σu,,,, v ad so u s the rage of Sce rak r, the there exst r o zero sgular values ad assocated orthoormal egevectors u,,, u ur that spa Rage() Sce they are orthoormal the they are learly depedet, so, they form a orthoormal bass of the rage of () For r,,, we have σ,, the v, r,, ad v s the ull space of Sce the v 's are orthoormal the they are learly PDF created wth pdffactory Pro tral verso wwwpdffactorycom

66 58 depedet; sce there exsts -r zero sgular values, where r s the rak of, the v,, r, v r v form the bass of the ull space of To prove ad 4, we use u σ v ad the same dea as above Example 5 Let 4 7 wth Soluto: 4 7 6, compute the four fudametal subspaces assocated The SVD of s UΣV where - 4/558 8/94-59/784 U 66/5 75/ /99 45/97 /6 454/979 Σ 57/95 477/599 V - 44/884-5/ 79/459 9/ /48-9/58-8/89-58/549-67/97 66/57 87/7-85/96-5/574 44/85 7/8597 8/485 sce r(), we have: - The frst two colums of U form a orthoormal bass for rage so - 4/558 8/94 rage spa 66/5, 75/754 45/97 /6 - The last two colums of V form a orthoormal bass for ull so PDF created wth pdffactory Pro tral verso wwwpdffactorycom

67 59 79/459 Null spa - 8/89 87/7 7/8597, 9/ /549-85/96 8/485 - The frst two colums of V form a orthoormal bass for rage so - 44/884-5/ rage spa - 87/48-9/58, - 67/97 66/57-5/574 44/85 4- The last colum of V forms a orthoormal bass for ull so -59/784 ull spa 548/99 454/979 Theorem 6 If M (C) wth o zero sgular values ad wth SVD exsts ad V Σ U vu σ, where σ σ UΣV the σ Σ dag (,,, ) [8, ] So, the sgular values of are the recprocal of the sgular values of Theorem 7 Let UΣV the - σ - where s square ad o sgular σ - F σ σ L σ Proof - By defto of spectral orm max { λ, λs a egevalues of } So σ PDF created wth pdffactory Pro tral verso wwwpdffactorycom

68 6 - By theorem 6, wrte VΣ U, the sce the spectral orm s utary varat we have VΣ U Σ σ - F UΣV F Σ F σ σ L σ (Sce the Frobeus orm s utary varat) [] Theorem 8 The codto umber of a o zero matrx wth respect to the spectral orm s Proof σ [9] σ If s sgular the σ ad the codto umber s, So suppose s vertble the codto umber s σ σ Theorem 9 If M (C) the det Proof Let σ [ ] UΣV be ts SVD where Σ M (C), the det( ) det( UΣV ) (det( U ))(det( Σ))(det( V )) det( Σ) det( Σ) σ Sce the determat of a utary matrx s ± ad Σ s dagoal Theorem For ay M (C) ad utary matrx W, the matrces, W ad W have the same sgular values [, ] Proof PDF created wth pdffactory Pro tral verso wwwpdffactorycom

69 6 Suppose W UΣV s the SVD of W The W UΣV HΣV, where HW U s utary So, ad W have the same sgular values Smlarly, ad W have the same sgular values Theorem matrx M (C) s utary f ad oly f all ts sgular values are equal to oe Proof Suppose s utary the I The sgular values of are the postve square roots of the egevalues of I whch are all equal to Coversely, suppose that the sgular values of are all equal to, the UIV s the SVD of, where U ad V are utary Hece UV ad s utary 4 Geometrc Iterpretato The SVD provdes us wth a ce geometrc terpretato of the acto of a matrx; the mage of the ut sphere uder ay m-by- matrx s a hyperellpse (m-dmetoal geeralzato of a ellpse) Oe way to uderstad ths s to cosder the ut sphere R So, Suppose x les o ths ut sphere R The x ca be wrtte as x v x v x v, wth x x ad where v 's are orthoormal bass R Let UΣV be the SVD of The the mage of x uder s PDF created wth pdffactory Pro tral verso wwwpdffactorycom

70 6 r x σuv x v σuv v x σ xu yu, where y σx ad r s the rak of So, the mage of a ut sphere s y σ y y r u yu y r ur, where yr x σ r r σ If has full colum rak, the r ad so the equalty s actually a equalty; otherwse, some of the x are mssg o the rght, ad the sum ca be aythg from to Ths shows that maps the ut sphere of R to a k- dmesoal ellpsod wth sem-axes the drectos u ad wth the magtudesσ Example Cosder a matrx R wth rak, ths matrx wll affect the ut sphere R as fgure () llustrates Fgure () PDF created wth pdffactory Pro tral verso wwwpdffactorycom

71 6 Chapter Four pplcatos of the Sgular Value Decomposto I mathematcs, ad partcularly lear algebra, the verse of a matrx takes a bg area solvg a set of lear equatos But sce the verse s ot defed except for some square matrces, ths fact pushed Moore (9) ad Perose (955) to establsh -depedetly- a geeralzato of the verse to rectagular matrces I ths chapter we descrbe the Moore-Perose Pseudo verse, how to compute t, study some of ts propertes ad more mportat how to use t solvg a system of lear equatos or gves a least square soluto ( whether t s overdetermed or uderdetermed system ) 4 Moore-Perose Pseudoverse Ths s the frst applcato of the SVD ad t s defed as below Defto 4 Let, ( C) The Moore-Perose pseudoverse of s defed as the M m matrx M, m ( C) satsfyg the followg four crtera: ( ) ( ) 4 ( ) ( ) PDF created wth pdffactory Pro tral verso wwwpdffactorycom

72 64 Remark If [a] s a complex umber the s defed by:,, a f a f a Example Let ad B The B s the Moore-Perose pseudoverse of sce t satsfes four crtera Defto 4: B - B ) ( - B B B ) ( - ) ( B B ad 4- ) ( B B So, B Before provg the exstece of we prove a remark ad a lemma PDF created wth pdffactory Pro tral verso wwwpdffactorycom

73 Remark If [ a j ] ad B [ b j ] 65 are dagoal matrces ad f CB, the C s dagoal wth c a b j j j Proof c j ( B) j k a k b kj I the summato, fk, the a ad f k j, the b Hece, j j j jj k c a b a b If j, the c j ab If j, the a b ad j kj j hece CB s dagoal ad we have c a b j j j Lemma If D s a dagoal matrx, the ts pseudoverse D s gve by Dj j ( D ) j [7] j Proof j j j j j j j j j ( DD D) D ( D D) D D j D D D D D DD D D ( D ( DD DD ) j ( DD ) j D j ( DD ) j Dj Dj Dj Dj D DD D ( DD ) ) DD j D j ( D ) j D j D j D j D j D D j j ( DD ) j 4 Smlar to Theorem 4 If, ( C), the the Moore-Perose pseudoverse of exsts ad s M m uque [ & 4] Proof PDF created wth pdffactory Pro tral verso wwwpdffactorycom

74 66 Let, ( C) wth m Use the th SVD to wrte U SV, where V s M m utary, U U I ad S dag( σ, σ, σ ) Defe B VS The B U show below: B U SV VS U U SV U SS SV U SV BB VS U U SV VS U VS SS U VS U B 4 ( B) ( U SV VS U ) ( U SS U ) U ( SS ) U U ( SS ) U U SV VS U B ( B) ( VS U U SV ) ( VS SV ) V( S S) V V( S S) V VS U U SV B So, VS satsfes pseudoverse codtos U We ow show the uqueess Suppose C, ( C) be aother pseudoverse of the: M m B - C BBB - CCC (B) B - (C) C 4 (B) B 4- (C) C s a frst step we show BC B(B) B B ( C) B C (B) (C) BCC I the same way we ca show that BC Now, BBBBCCCC Ths theorem shows how to compute the Moore Perose psuedovers of, e, VS U, where U SV PDF created wth pdffactory Pro tral verso wwwpdffactorycom

75 67 Note The pseudoverse s a geeralzato of the verse, e, f the matrx s vertble the - [8] Lemma T T Let, ( C) The ( ) ( ),( ) ( ) ad ( ) ( ) M m [, ] Proof We wat to show that ( ) s the pseudoverse of So, we exame the four codtos: ( ) ( ) ( ) ( ) ( ) ( ) (( ) ) ( ) ( ) 4 ( ( ) ) ( ) ( ) I the same way we ca prove that: ( T T ) ( ) ad ( ) ( ) ccordg to ths lemma, we ca state ad prove the followg theorem Theorem 4 (Idetty Trasformato) ( ) ( ) ( ) 4 ( ) 5 6 PDF created wth pdffactory Pro tral verso wwwpdffactorycom

76 68 Proof ( ) ( ) ( ) ( ) ( ) ( ) 4 ( ) ( ) 5 Ths s the cojugate traspose of 6 Ths s the cojugate traspose of 4 We ow state: Theorem 4 Let, ( C) ad k C be ozero The - M m ( k) k - ) ( [4] To prove ( k), oe ca show that k codtos of k To prove pseudoverse codtos of satsfes the pseudoverse k ( ), oe also shows that satsfes the Lemma ad Theorem 4 gve us some propertes of whch are true for, but t s ot true that all propertes of the verse also hold for the psuedoverse For example ( B) B geeral Example Let [ ] ad B The ad B PDF created wth pdffactory Pro tral verso wwwpdffactorycom