QR Factorzato ad Sgular Value Decomposto COS 33
Why Yet Aother Method? How do we solve least-squares wthout currg codto-squarg effect of ormal equatos (A T A A T b) whe A s sgular, fat, or otherwse poorly-specfed? QR Factorzato Householder method Sgular Value Decomposto Total least squares Practcal otes
Revew: Codto Number Cod(A) s fucto of A Cod(A) >, bgger s bad Measures how chage put propagates to output: A cod(a) A E.g., f cod(a) 45 the ca lose log(45).65 dgts of accuracy, compared to precso of A
Normal Equatos are Bad cod(a) A A Normal equatos volves solvg A T A A T b cod(a T A) [cod(a)] E.g., f cod(a) 45 the ca lose log(45 ) 5.3 dgts of accuracy, compared to precso of A
QR Decomposto
What f we dd t have to use A T A? Suppose we are lucky : Upper tragular matrces are ce! b R
How to make A upper-tragular? Gaussa elmato? Applyg elmato yelds MA Mb Wat to fd s.t. mmzes Mb-MA Problem: Mv! v (.e., M mght stretch a vector v) Aother problem: M may stretch dfferet vectors dfferetly.e., M does ot preserve Eucldea orm.e., that mmzes Mb-MA may ot be same that mmzes Ab
QR Factorzato Fd upper-tragular R ad orthogoal Q s.t. R R A Q, so Q Does t chage least-squares soluto Q T QI, colums of Q are orthoormal.e., Q preserves Eucldea orm: Qv v T b
Goal of QR???? Q R Q A m m m m R:, upper tr. (m-), all zeros
Reformulatg Least Squares usg QR c c R c O R b Q O R Q Q b Q O R Q b A b r T T T + O R Q A because because Q preserves legths f we call c c Q T b f we choose such that Rc because Q s orthogoal (Q T QI)
Householder Method for Computg QR Decomposto
Orthogoalzato for Factorzato Rough dea: A R Q O For each -th colum of A, zero out rows + ad lower Accomplsh ths by multplyg A wth a orthogoal matr H Equvaletly, apply a orthogoal trasformato to the -th colum (e.g., rotato, reflecto) Q becomes product H * *H, R cotas zero-ed out colums
Householder Trasformato Accomplshes the crtcal sub-step of factorzato: Gve ay vector (e.g., a colum of A), reflect t so that ts last p elemets become. Reflecto preserves legth (Eucldea orm) (4, 3) (-5, )
Outcome of Householder R H HA O T where Q H H so Q HH R so A Q O
Revew: Least Squares usg QR c c R c O R b Q O R Q Q b Q O R Q b A b r T T T + O R Q A because because Q preserves legths f we call c c Q T b f we choose such that Rc because Q s orthogoal (Q T QI)
Usg Householder Iteratvely compute H, H, H ad apply to A to get R also apply to b to get Q T b c c Solve for Rc usg back-substtuto
Alteratve Orthogoalzato Methods Gves: Do t reflect; rotate stead Itroduces zeroes to A oe at a tme More complcated mplemetato tha Householder Useful whe matr s sparse Gram-Schmdt Iteratvely epress each ew colum vector as a lear combato of prevous colums, plus some (ormalzed) orthogoal compoet Coceptually ce, but suffers from subtractve cacellato
Sgular Value Decomposto
Motvato Dagoal matrces are eve cer tha tragular oes:
Motvato What f you have fewer data pots tha parameters your fucto?.e., A s fat Itutvely, ca t do stadard least squares Recall that soluto takes the form A T A A T b Whe A has more colums tha rows, A T A s sgular: ca t take ts verse, etc.
Motvato 3 What f your data poorly costras the fucto? Eample: fttg to ya +b+c
Udercostraed Least Squares Problem: f problem very close to sgular, roudoff error ca have a huge effect Eve o well-determed values! Ca detect ths: Ucertaty proportoal to covarace C (A T A) - I other words, ustable f A T A has small values More precsely, care f T (A T A) s small for ay Idea: f part of soluto ustable, set aswer to Avod corruptg good parts of aswer
Sgular Value Decomposto (SVD) Hady mathematcal techque that has applcato to may problems Gve ay m matr A, algorthm to fd matrces U, V, ad W such that A U W V T U s m ad orthoormal W s ad dagoal V s ad orthoormal
SVD Based o Householder reducto, QR decomposto, but treat as black bo: code wdely avalable e.g., Matlab: [U,W,V]svd(A,) T V U A w w
SVD The w are called the sgular values of A If A s sgular, some of the w wll be I geeral rak(a) umber of ozero w SVD s mostly uque (up to permutato of sgular values, or f some w are equal)
SVD ad Iverses Why s SVD so useful? Applcato : verses A - (V T ) - W - U - V W - U T Usg fact that verse traspose for orthogoal matrces Sce W s dagoal, W - also dagoal wth recprocals of etres of W
SVD ad the Pseudoverse A - (V T ) - W - U - V W - U T Ths fals whe some w are It s supposed to fal sgular matr Happes whe rectagular A s rak defcet Pseudoverse: f w, set /w to (!) Closest matr to verse Defed for all (eve o-square, sgular, etc.) matrces Equal to (A T A) - A T f A T A vertble
SVD ad Codto Number Sgular values used to compute Eucldea (spectral) orm for a matr: cod(a) σ (A) ma σ (A) m
SVD ad Least Squares Solvg Ab by least squares: A T A A T b (A T A) - A T b Replace wth A + : A + b Compute pseudoverse usg SVD Lets you see f data s sgular (< ozero sgular values) Eve f ot sgular, codto umber tells you how stable the soluto wll be Set /w to f w s small (eve f ot eactly )
Total Least Squares Oe fal least squares applcato Fttg a le: vertcal vs. perpedcular error
Total Least Squares Dstace from pot to le: where s ormal vector to le, a s a costat Mmze: a y d a y d χ
Total Least Squares Frst, let s preted we kow, solve for a The y m a a y χ y a y d m y m Σ Σ
Total Least Squares So, let s defe ad mmze Σ Σ m y m y y ~ ~ y ~ ~
Total Least Squares Wrte as lear system Have A Problem: lots of are solutos, cludg Stadard least squares wll, fact, retur ~ ~ ~ ~ ~ ~ 3 3 y y y y
Costraed Optmzato Soluto: costra to be ut legth So, try to mmze A subject to Epad egevectors e of A T A: where the λ are egevalues of A T A ( ) ( ) A A A A A T T T ( ) T T µ µ µ λ λ µ µ µ + + + A A e e
Costraed Optmzato To mmze µ λµ subject to set µ m, all other µ λ + µ + µ That s, s egevector of A T A wth the smallest correspodg egevalue
SVD ad Egevectors Let AUWV T, ad let be th colum of V Cosder A T A : So elemets of W are sqrt(egevalues) ad colums of V are egevectors of A T A w w T T T T T V VW VW V UWV VW U A A
Costraed Optmzato To mmze µ λµ subject to set µ m, all other µ λ + µ + µ That s, s egevector of A T A wth the smallest correspodg egevalue That s, s colum of V correspodg to smallest sgular value
Comparso of Least Squares Methods Normal equatos (A T A A T b) O(m ) (usg Cholesky) cod(a T A)[cod(A)] Cholesky fals f cod(a)~/sqrt(mache epslo) Householder Usually best orthogoalzato method O(m - 3 /3) operatos Relatve error s best possble for least squares Breaks f cod(a) ~ /(mache eps) SVD Epesve: m + 3 wth bad costat factor Ca hadle rak-defcecy, ear-sgularty Hady for may dfferet thgs