QR Factorization and Singular Value Decomposition COS 323

Transcription

1 QR Factorzato ad Sgular Value Decomposto COS 33

2 Why Yet Aother Method? How do we solve least-squares wthout currg codto-squarg effect of ormal equatos (A T A A T b) whe A s sgular, fat, or otherwse poorly-specfed? QR Factorzato Householder method Sgular Value Decomposto Total least squares Practcal otes

3 Revew: Codto Number Cod(A) s fucto of A Cod(A) >, bgger s bad Measures how chage put propagates to output: A cod(a) A E.g., f cod(a) 45 the ca lose log(45).65 dgts of accuracy, compared to precso of A

4 Normal Equatos are Bad cod(a) A A Normal equatos volves solvg A T A A T b cod(a T A) [cod(a)] E.g., f cod(a) 45 the ca lose log(45 ) 5.3 dgts of accuracy, compared to precso of A

5 QR Decomposto

6 What f we dd t have to use A T A? Suppose we are lucky : Upper tragular matrces are ce! b R

7 How to make A upper-tragular? Gaussa elmato? Applyg elmato yelds MA Mb Wat to fd s.t. mmzes Mb-MA Problem: Mv! v (.e., M mght stretch a vector v) Aother problem: M may stretch dfferet vectors dfferetly.e., M does ot preserve Eucldea orm.e., that mmzes Mb-MA may ot be same that mmzes Ab

8 QR Factorzato Fd upper-tragular R ad orthogoal Q s.t. R R A Q, so Q Does t chage least-squares soluto Q T QI, colums of Q are orthoormal.e., Q preserves Eucldea orm: Qv v T b

9 Goal of QR???? Q R Q A m m m m R:, upper tr. (m-), all zeros

10 Reformulatg Least Squares usg QR c c R c O R b Q O R Q Q b Q O R Q b A b r T T T + O R Q A because because Q preserves legths f we call c c Q T b f we choose such that Rc because Q s orthogoal (Q T QI)

11 Householder Method for Computg QR Decomposto

12 Orthogoalzato for Factorzato Rough dea: A R Q O For each -th colum of A, zero out rows + ad lower Accomplsh ths by multplyg A wth a orthogoal matr H Equvaletly, apply a orthogoal trasformato to the -th colum (e.g., rotato, reflecto) Q becomes product H * *H, R cotas zero-ed out colums

13 Householder Trasformato Accomplshes the crtcal sub-step of factorzato: Gve ay vector (e.g., a colum of A), reflect t so that ts last p elemets become. Reflecto preserves legth (Eucldea orm) (4, 3) (-5, )

14 Outcome of Householder R H HA O T where Q H H so Q HH R so A Q O

15 Revew: Least Squares usg QR c c R c O R b Q O R Q Q b Q O R Q b A b r T T T + O R Q A because because Q preserves legths f we call c c Q T b f we choose such that Rc because Q s orthogoal (Q T QI)

16 Usg Householder Iteratvely compute H, H, H ad apply to A to get R also apply to b to get Q T b c c Solve for Rc usg back-substtuto

17 Alteratve Orthogoalzato Methods Gves: Do t reflect; rotate stead Itroduces zeroes to A oe at a tme More complcated mplemetato tha Householder Useful whe matr s sparse Gram-Schmdt Iteratvely epress each ew colum vector as a lear combato of prevous colums, plus some (ormalzed) orthogoal compoet Coceptually ce, but suffers from subtractve cacellato

18 Sgular Value Decomposto

19 Motvato Dagoal matrces are eve cer tha tragular oes:

20 Motvato What f you have fewer data pots tha parameters your fucto?.e., A s fat Itutvely, ca t do stadard least squares Recall that soluto takes the form A T A A T b Whe A has more colums tha rows, A T A s sgular: ca t take ts verse, etc.

21 Motvato 3 What f your data poorly costras the fucto? Eample: fttg to ya +b+c

22 Udercostraed Least Squares Problem: f problem very close to sgular, roudoff error ca have a huge effect Eve o well-determed values! Ca detect ths: Ucertaty proportoal to covarace C (A T A) - I other words, ustable f A T A has small values More precsely, care f T (A T A) s small for ay Idea: f part of soluto ustable, set aswer to Avod corruptg good parts of aswer

23 Sgular Value Decomposto (SVD) Hady mathematcal techque that has applcato to may problems Gve ay m matr A, algorthm to fd matrces U, V, ad W such that A U W V T U s m ad orthoormal W s ad dagoal V s ad orthoormal

24 SVD Based o Householder reducto, QR decomposto, but treat as black bo: code wdely avalable e.g., Matlab: [U,W,V]svd(A,) T V U A w w

25 SVD The w are called the sgular values of A If A s sgular, some of the w wll be I geeral rak(a) umber of ozero w SVD s mostly uque (up to permutato of sgular values, or f some w are equal)

26 SVD ad Iverses Why s SVD so useful? Applcato : verses A - (V T ) - W - U - V W - U T Usg fact that verse traspose for orthogoal matrces Sce W s dagoal, W - also dagoal wth recprocals of etres of W

27 SVD ad the Pseudoverse A - (V T ) - W - U - V W - U T Ths fals whe some w are It s supposed to fal sgular matr Happes whe rectagular A s rak defcet Pseudoverse: f w, set /w to (!) Closest matr to verse Defed for all (eve o-square, sgular, etc.) matrces Equal to (A T A) - A T f A T A vertble

28 SVD ad Codto Number Sgular values used to compute Eucldea (spectral) orm for a matr: cod(a) σ (A) ma σ (A) m

29 SVD ad Least Squares Solvg Ab by least squares: A T A A T b (A T A) - A T b Replace wth A + : A + b Compute pseudoverse usg SVD Lets you see f data s sgular (< ozero sgular values) Eve f ot sgular, codto umber tells you how stable the soluto wll be Set /w to f w s small (eve f ot eactly )

30 Total Least Squares Oe fal least squares applcato Fttg a le: vertcal vs. perpedcular error

31 Total Least Squares Dstace from pot to le: where s ormal vector to le, a s a costat Mmze: a y d a y d χ

32 Total Least Squares Frst, let s preted we kow, solve for a The y m a a y χ y a y d m y m Σ Σ

33 Total Least Squares So, let s defe ad mmze Σ Σ m y m y y ~ ~ y ~ ~

34 Total Least Squares Wrte as lear system Have A Problem: lots of are solutos, cludg Stadard least squares wll, fact, retur ~ ~ ~ ~ ~ ~ 3 3 y y y y

35 Costraed Optmzato Soluto: costra to be ut legth So, try to mmze A subject to Epad egevectors e of A T A: where the λ are egevalues of A T A ( ) ( ) A A A A A T T T ( ) T T µ µ µ λ λ µ µ µ A A e e

36 Costraed Optmzato To mmze µ λµ subject to set µ m, all other µ λ + µ + µ That s, s egevector of A T A wth the smallest correspodg egevalue

37 SVD ad Egevectors Let AUWV T, ad let be th colum of V Cosder A T A : So elemets of W are sqrt(egevalues) ad colums of V are egevectors of A T A w w T T T T T V VW VW V UWV VW U A A

38 Costraed Optmzato To mmze µ λµ subject to set µ m, all other µ λ + µ + µ That s, s egevector of A T A wth the smallest correspodg egevalue That s, s colum of V correspodg to smallest sgular value

39 Comparso of Least Squares Methods Normal equatos (A T A A T b) O(m ) (usg Cholesky) cod(a T A)[cod(A)] Cholesky fals f cod(a)~/sqrt(mache epslo) Householder Usually best orthogoalzato method O(m - 3 /3) operatos Relatve error s best possble for least squares Breaks f cod(a) ~ /(mache eps) SVD Epesve: m + 3 wth bad costat factor Ca hadle rak-defcecy, ear-sgularty Hady for may dfferet thgs