Some Different Perspectives on Linear Least Squares

Size: px

Start display at page:

Download "Some Different Perspectives on Linear Least Squares"

Moses Little
5 years ago
Views:

1 Soe Dfferet Perspectves o Lear Least Squares A stadard proble statstcs s to easure a respose or depedet varable, y, at fed values of oe or ore depedet varables. Soetes there ests a deterstc odel y f (,, where stads collectvely for all of the depedet varables of the syste ad stads collectvely for all of the odel paraeters. I the probablstc odel t s ofte assued that each easured y value ca be epressed as y f (, + ε, here s a de that labels the fed put codto at adε represets rado error assocated wth the gve easureet. For a vald odel the ea rado error should be zero. For a gve set of data pots the proble of choosg the best values for the paraeters s typcally solved va the ethod of least squares. he paraeters are chose to ze the fucto F( y f (,. If the fucto f (, s lear the procedure s referred to as a lear least squares or lear regresso proble. he purpose of these otes s to eae the least squares soluto fro several dfferet pots of vew. he Least Squares Approato to a Vector a Desoal Eucldea Space Cosder a desoal Eucldea space, V wth a er product of vectors v ad u deoted as( uv,. Let W be a subspace of V, the V W W, where W s the orthogoal copleet of W. Let v be ay eleet of V, thev w+ u, where w W ad u W. he vector w s called the proecto of v oto W. Let g v ( ψ v ψ ( v ψ, v ψ. he followg arguet shows that ψ w s the g v ψ ca be epressed as g v ( ψ ( v w ( ψ w v w ( v w, ψ w + ψ w. If ψ W, the ψ w W ad ( v w, ψ w ( u, ψ w 0. So o W, g v ( ψ u + ψ w u, wth equalty oly at ψ w. If V R ad W has the orthooral bass C, C, C wth, the the colu vector uque u of ths fucto o W. Frst, by epadg the er product, ( { } whch s the proecto of v oto W s gve by w ( Cˆ, v Cˆ (. Defe the atr Q as Q C, C, C, that s Q, C. he atr QQ, called the outer product,, proects v oto W: ( ( ( w, C, v C C ( Q v, Q, ( Q v, ( QQ v. hus,,,, the vector QQ v zes the squared dstace fro v to W. Al Lehe 9/4/0 Madso Area echcal College

2 Soe Dfferet Perspectves o Lear Least Squares Lear Least Squares Matr Notato: he Geeralzed Iverse If a odel s lear ts paraeters desgated by the vector R, the all the estated y resposes ca be geerated as a vector R gve by A where A s a atr. We wll assue that A has ra wth. If the ra of A s less tha, t eas that the odel paraeters are ot all depedet ad eed to be reduced uber so as to ae a desg atr wth all of ts colus learly depedet. Deote the th colu of A as A,.e., A [ A, A, A ]. Sce the colus of A are learly depedet vectors R, Ab ba 0 f ad oly f each b 0. hg of A as a lear trasforato fro R to R, the age of A, W, s the spa of { A, A, A } ad has deso, whle the erel of A has deso zero ad cossts of oly the zero vector 0. hus, every vector W s orthogoal to each colu of A, or stated dfferetly, f u W, A u 0. Coversely, f A u 0, the every colu of A s perpedcular to u, so u W. hus, the erel of A traspose s the orthogoal copleet of the colu space of A. Let y R, the y w + uwhere w W ad u W. Uless u 0, the syste of equatos Ab y has o solutos. However, Ab w has a soluto,, ad sce { A, A, A } s a learly depedet set, ths soluto s uque. Furtherore, w s the closest vector the colu space of A to y ad the u value of F y ( b y Ab as b vares over R s F y ( y w y A. Now, A w, so, sce er( A A A A w A ( y u A y A u A y W. Suppose that for b R, A Ab 0, the b A Ab ( Ab Ab ( Ab, Ab Ab 0. So, f A Ab 0, Ab 0 whch ples that b 0 sce the erel of A cossts of oly 0. herefore, A As osgular ad the least squares soluto to zg F y ( b s gve by ( A A A y, the atr ( A A A s soetes called the geeralzed verse of the atr A. A Alterate Forulato of Lear Least Squares Usg Multvarable Calculus he least squares soluto s the vector R that zes the fucto of varables, F y ( b y Ab ( yab, y Ab ( y, y ( Ab, y + ( Ab, Ab as b vares over R. Rewrtg ths fucto usg sga otato gves the followg epresso. Al Lehe 9/4/0 Madso Area echcal College

3 Soe Dfferet Perspectves o Lear Least Squares 3 F y ( b ( y, y ( Ab y, + A, b b A, b ( y, y ( A by + b b( A A,, Now a ecessary codto for F y ( b to be at a u s that all of the partal dervatves F y b vash for every teger wth. Sce δ,, the followg splfcato results. b b F y ( yy, ( A b, y + b b ( A A b b, b 0 ( b b A y, + b b ( A A + b b b, ( A y, + ( δ, b b δ, ( A A +, ( A y + ( A A b + ( A A b,,, ( A y + ( A Ab + ( A A b,,, ( A y + ( A Ab,, F y he codto, 0 b uque soluto gve by ( for every teger wth, requres that A Ab A y. hs has the A A A y, whch s detcal to the result obtaed the last secto. A Forulato of Lear Least Squares Usg QR Factorzato Sce the colus of A are learly depedet a Gra Schdt procedure o the colus yelds a orthooral bass for W. Desgate ths bass as C, C, C. Specfcally, ths orthooral bass s { } defed recursvely for as C C, where C A ( A, C C C ad C A. A Al Lehe 9/4/0 Madso Area echcal College

4 Soe Dfferet Perspectves o Lear Least Squares 4 he colu vector whch s the proecto of y oto W s gve by w QQ y, where the atr Q s defed as Q (, C,.e., Q C, C, C. Now, A QQ A, sce the proecto of each, colu of atr A oto A s colu space s ust that orgal colu of A. Let the atr R be defed as R Q A. Now, ( ( R C (,, A C A. By the Gra Schdt procedure f the spa of { C, C, C } s equal to the spa of { A, A, A }. So A ( C C, A. Now, f >, ( ( ( C, A C, C C, A ( 0 ( C, A 0,.e., f >, C s orthogoal to the subspace spaed by { A, A, A }. herefore, the atr R s upper tragular. Cosder for real ubers, α,, α,, α3,, α, the su α (, ( R, α C A C, z, where z α A W. If α R, 0 for every teger,, the z ( C, z C 0. But sce the colus of A are learly depedet, ths requres that α α α3 α 0. Hece, the colus of the atr R are learly depedet ad R s therefore osgular. hs factorzato of A, A QQ A QR, s geercally called the QR factorzato. Sce the uque soluto of the lear least squares proble solves A w QQ y, we have QR QQ y. Now, Q Q I,.e., ( (,, (, Q Q Q C C C C C C C δ,.,, So R Q QR Q QQ y Q yad the uque lear least squares soluto s gve by R Q y. he followg steps show that ths soluto s the sae as that obtaed by the geeralzed verse. A ( QR R Q A A R Q QR R IR R R A A R R R R ( ( ( ( ( A A A R R R Q R Q Calculatg the Paraeter Varaces Fro the theory of the probablty dstrbutos of rado varables we have the followg fudaetal result. If y, y, y3,, y are statstcally depedet rado varables wth the varace of y beg Al Lehe 9/4/0 Madso Area echcal College

5 Soe Dfferet Perspectves o Lear Least Squares 5 σ, ad h α y, the σh α σ. Now, usg the geeralzed verse the lear least squares soluto for paraeter s gve by ( A A A y ( A A A y y A, ( A A.,,, Hece, ( σ σ A A A.,, I the specal case where all of the rado varables have a coo varace, σ, ths splfes to ( ( σ σ A A A σ A A A,.,, I the QR factorzato soluto of the lear least squares proble, ( R Q y ( R Q y y R (, Q, y R, C, so that,, σ σ R, ( C. I the specal case where all of the rado varables have a coo varace ths splfes as follows. ( ( ( R ( (, C R, C R, C R, R, C C σ R, R, ( C, C σ R, R, δ, σ ( R, A Eaple of the Methods: he Lear Model Oe Idepedet Varable If the odel s that yestate + 0, ad the paraeter vector s defed as, the the 0 desg atr gve by A. Hece, A A ad Al Lehe 9/4/0 Madso Area echcal College

6 Soe Dfferet Perspectves o Lear Least Squares 6 ( A A. Here the average value of a varable s deoted as α α ( whle the covarato of two varables αω, s defed as α ω αω + αω α ω ( αω( α( ω ( α α( ω ω αω ω α ( α( ω αω ( α( ω y Also, A y, so the least squares soluto s gve by the followg epresso, y ( y y y A A A y. 0 y y y hese are the regresso equatos whch are soetes epressed as y y y y y 0 y y y y y y y y ( y ( y Assug a coo varace for each depedet rado varable y, the varaces of the odel paraeters are coputed as follows.. Al Lehe 9/4/0 Madso Area echcal College

7 Soe Dfferet Perspectves o Lear Least Squares 7 σ σ σ ( A A A (, σ σ σ σ σ σ ( σ ( A A A ( 0, σ ( (( ( σ ( ( ( ( + ( ( σ ( ( ( ( ( ( + ( σ σ [ ] ( ( ( 0 + ( + ( ( σ + Usg the QR forulato to calculate the paraeters requres a orthooral bass for the colu space of A. A Gra Schdt procedure results the followg vectors. C, C Al Lehe 9/4/0 Madso Area echcal College

8 Soe Dfferet Perspectves o Lear Least Squares 8 Noralzg C requres the prevously ecoutered su, ( ( + (. So that ( C. ( + ( hus, Q + ad ( ( C, A C, A R Q A ( C, A ( 0 0. Proceedg, C, A ( ( + + ( ( + + ( R + ad ( y Q y. y y ( + Coputg the paraeter vector gves y y y R Q y. 0 y y Al Lehe 9/4/0 Madso Area echcal College

9 Soe Dfferet Perspectves o Lear Least Squares 9 he epresso for 0 agrees wth the earler dervato, whle the prevous epresso for s recovered as s show by the followg steps. y y y y y y y y y y y y y y y Assug a coo varace for each depedet rado varable y, the QR factorzato the varaces of the odel paraeters are coputed as follows. ( σ ( ( R, ( σ 4 σ 4 ( ( ( ( ( σ σ ( ( ( R, σ σ σ σ σ σ σ Both of these results agree wth the prevous aalyss that used the geeralzed verse. Al Lehe 9/4/0 Madso Area echcal College

7.0 Equality Contraints: Lagrange Multipliers

7.0 Equality Contraints: Lagrange Multipliers Systes Optzato 7.0 Equalty Cotrats: Lagrage Multplers Cosder the zato of a o-lear fucto subject to equalty costrats: g f() R ( ) 0 ( ) (7.) where the g ( ) are possbly also olear fuctos, ad < otherwse