Sgular Value Decomosto Lear Algera (3) m Cootes Ay m x matrx wth m ca e decomosed as follows Dagoal matrx A UWV m x x Orthogoal colums U U I w1 0 0 w W M M 0 0 x Orthoormal (Pure rotato) VV V V L 0 L 0 O M L w I Dagoal elemets are called Sgular Values Sgular Value Decomosto A UWV SVD ad Egevectors Egevector decomosto s a secal case of SVD for square, symmetrc matrces A UWV m m If A A the U V ad A Colums of U are egevectors Elemets of W are egevalues UWU Solvg Regular LEs usg SVD Cosder case whch Ax UWV x m, A ( w ) 0 ( VW U ) UWV x ( VW U ) VW ( U U) WV x VW U V( W W) V x VW U VV x VW U x VW U A 1 VW U Solvg LEs wth SVD x VW U Ca oly fal f oe of the sgular values, w, s zero or very small If so, matrx s called sgular he codto umer of a matrx s wlargest C w smallest If C>10 1 (whe usg doules), the A s llcodtoed, ad care must e take 1
Geometrc Iterretato Geometrc terretato For m case Ax UWV Pure Rotato Scale alog each axs x Pure Rotato 0 x V x 0 x UWV x (Rotato) x Ux x Wx (Scalg) (Rotato) Geometrc Iterretato Dagoal matrx scales alog axes: w1 0 x x 0 0 w 0 w 0 0 What f oe elemet s zero (or ear-zero)? w 1 Geometrc Iterretato If oe elemet s zero (or ear-zero) Volume s flatteed that drecto 0 0 0 0 x x 0 w 0 w Geometrc terretato If oe sgular value s ear zero: 0 x V x 0 x UWV x (Rotato) 0 0 x x 0 w (Scalg) x Ux Zero sgular values If oe or more sgular values s ear zero he trasformato Ax flattes sace oto a lear susace A le or ot D A lae, le or ot 3D A hyerlae -D he colums of A are ot learly deedet (Rotato)
D Case Roudg errors x Sgular values >> 0 Determat >>0 Soluto ot well defed x Sgular value ear zero Determat ear zero Soluto ot ecomes ustale test_roudg.m a1.0; d1.0; whle as(d-1.0)<1e-6 a10*a; d(a1.0)-a; ds(['a' umstr(a) '(a1)-a' umstr(d)]); ed Outut a10 (a1)-a1 a100 (a1)-a1 a1000 (a1)-a1 a10000 (a1)-a1 a100000 (a1)-a1 a1000000 (a1)-a1 a10000000 (a1)-a1 a100000000 (a1)-a1 a1000000000 (a1)-a1 a1.000000e010 (a1)-a1 a1.000000e011 (a1)-a1 a1.000000e01 (a1)-a1 a1.000000e013 (a1)-a1 a1.000000e014 (a1)-a1 a1.000000e015 (a1)-a1 a1.000000e016 (a1)-a0 test_recso.m Precso d1.0; whle (1.0d)>1.0 dd/10.0; ds(['d' umstr(d,18) ' 1d' umstr(1d,18)]); ed Outut Rak d0.10000000000000001 d0.01 d0.001 d0.0001 d1.0000000000000001e-005 d1.000000000000000e-006 d1.000000000000000e-007 d1.000000000000000e-008 d1.0000000000000003e-009 d1.0000000000000003e-010 d1.0000000000000003e-011 d1.000000000000000e-01 d1.000000000000000e-013 d1.000000000000000e-014 d1.0000000000000001e-015 d1.0000000000000001e-016 1d1.1000000000000001 1d1.01 1d1.0009999999999999 1d1.0001 1d1.0000100000000001 1d1.0000009999999999 1d1.0000001000000001 1d1.0000000099999999 1d1.0000000010000001 1d1.0000000001 1d1.00000000001 1d1.0000000000010001 1d1.0000000000000999 1d1.00000000000001 1d1.0000000000000011 1d1 he rak of matrx A s the umer of learly deedet colums of A Numer of o-zero sgular values of A x A s full rak f rak(a) 3
Uderdetermed Systems m < If there are fewer equatos tha ukows, the the system s called uderdetermed Usually a fte umer of ossle solutos Uderdetermed Systems m equatos 3 ukows a 1 Ax x a x a a 1 1 x3 3 a D vectors a 3 May ossle solutos for x Uderdetermed Systems Whe m < Ax defes a lear susace of solutos x Ay ot o le satsfes equatos We re ofte terested ot that mmses x Solvg Lear Equatos usg SVD For m solve Ax as follows Aly SVD to ota A UWV Fd largest asolute sgular value, w 1.. Set a threshold, Defe z t 10 w 1/ w f w > t { 0 otherwse w he soluto s the x VZU z1 0 Z M 0 0 L 0 z L 0 M O M 0 L z Solvg Lear Equatos usg SVD For m < fd a soluto to Ax as follows Aly SVD to A to ota A UWV so A VWU Fd largest asolute sgular value, w 1.. Set a threshold, Defe z t 10 w 1/ w f w > t { 0 otherwse w Overdetermed Case If there are more equatos tha ukows, the the system s called overdetermed m > Ideal case Real world case x x he soluto s the x UZV z1 0 Z M 0 0 L 0 z L 0 M O M 0 L z Soluto ot s aroud here somewhere 4
Overdetermed Case Ax x a L a 1 1 x a are colums of A (m-d vectors) I geeral, caot reach f <m So fd x whch gets as close as ossle m equatos 1ukow a 1 a 1 Closest ot o le to Overdetermed Case Vector etwee Ax ad s Ax- Square of dstace etwee Ax ad s d Ax ( Ax ) ( Ax ) he soluto s usually defed as the x whch mmses ths dstace 1 Fdg mmum d ( Ax ) ( Ax ) x A Ax x A Dfferetatg w.r.t x, ad equatg to zero gves A Ax A 0 ( A A) x A symmetrc (But ot ecessarly ve defte or full rak) hus we could solve ths square lear equato to fd x However, ractce t s usually solved usg SVD of A Pseudo-verse If A s a m x matrx, the there exsts a x m matrx, kow as the Moore- Perose Pseudo-verse: A A x s the least - squares soluto to Ax If the verse of A A exsts, the A AA A A A AA A ( AA ) ( A A) AA A A ( A A) A Imlemetato Issues Solvg Ax Never solve lear equatos y comutg the verse Perform decomosto, the use acksusttuto More umercally stale Successful ad A >0 A A Use Cholesky A LL m? m Symmetrc? Faled or A ear 0 m A A Use LU decom A LU Solve usg SVD Successful ad A ot 0 Solve usg Cholesky acksusttuto Solve usg SVD Solve usg LU acksusttuto 5
Iteratve mrovemet Soluto to Ax ca cota roudg errors Suose x s curret estmate of sol to Ax he a etter sol s x 1 x dx where dx s the sol to A( dx ) ( Ax ) Curret estmate of error QR Decomosto Ay matrx A ca e decomosed as m Ax QRx Rx Q A QR m orthogoal Q Q I uer tragular R tragular, so smle to solve Lear susaces Suose we have a set of k learly deedet -D vectors, wth k< { a }..k 1 hese sa a k-d lear susace Ay ot ths susace ca e wrtte x a L a Ax 1 1 x k k 1D lear susace For stace, D sace { a 1 } k 1 sas a 1-D susace of D sace a 1 Ay ot the sace ca e uquely defed y k arameters (Mafolds) Lear susaces are secal cases of mafolds A mafold s a ossly curved structure of lower trsc dmesoalty tha that of the sace whch t s emedded Pots o a crcle: 1D mafold D sace Surface of a shere: D mafold 3D sace Itrsc dmesoalty: he mmum umer of arameters requred to uquely defe ay ot the sace Lear susaces he k learly deedet -D vectors { a} 1..k Are sad to defe a ass for the susace (ca e thought of as a set of axes) If the vectors are of ut legth, ad mutually orthogoal, they are sad to defe a orthoormal ass for the susace 6
Orthoormal Bass Gve a artrary set of ass vectors we ca geerate a orthoormal ass usg SVD ( a 1 L ak ) A UWV ( u 1 L u k ) WV Colums of U gve a orthoormal ass for susace Orthoormal Bass If s a ot the susace, the x u L u Ux 1 1 x k k x U he rojecto of alog le x u s u. u u 1 u. Dstace to susace he earest ot the susace to ay -D ot s ' Ux UU Dstace to susace he dstace from ay -D ot to the earest ot the susace s d ' x where x U u 1 u. ' x u ( u )u 1 u u. d ' x u ( u )u ' d x d 7