Marquette Uverst MSCS6 Le Fttg ad Regresso Dael B. Rowe, Ph.D. Professor Departmet of Mathematcs, Statstcs, ad Computer Scece Coprght 8 b
Marquette Uverst Least Squares Regresso MSCS6 For LSR we have pots (, ),,(, ) ad wsh to fd the best ft le aˆ b ˆ to the data. 5 The best le defed as mmzg sum of squared resduals. 4 3 d 4 d 5 d 3 True Le Measuremet Error d d d aˆ b ˆ 3 3 4 4 5 5
Marquette Uverst Least Squares Regresso MSCS6 Defe Q ( a b ) as the score fucto to be mmzed to obta the optmal ( ab ˆ, ˆ). What we wat to do s fd the values of ( ab, ) that mmze Q. The values (a,b) that mmze Q are the optmal values are ( ab ˆ, ˆ). ( ab ˆ, ˆ) that mmze a b ( ) wrt (a,b) 3
Marquette Uverst Least Squares Regresso MSCS6 Dfferetatg Q wrt a, ad b, the set = Q ( a b ),..., Q ( a b)( ) a ab ˆ, ˆ Q ( a b )( ) b ab ˆ, ˆ 4
Marquette Uverst Least Squares Regresso MSCS6 Solvg for the estmated parameters elds bˆ aˆ ( ) ( )( ) ( ) ( ) ( )( ) ( )( ) ( ) ( ) â b ˆ 5 ˆ 4 3 d d ˆ aˆb d 5 d 4 d 3 d aˆ b ˆ 3 3 4 4 5 5 5
Marquette Uverst Least Squares Regresso MSCS6 Solvg for the estmated parameters elds bˆ S S â b ˆ 5 ˆ 4 ˆ aˆb s ( ) s ( )( ) 3 d d d 5 d 4 d 3 d aˆ b ˆ 3 3 4 4 5 5 6
Marquette Uverst Least Squares Regresso MSCS6 Because of the scetfc applcato, t ma be kow that the -tercept should trul be zero. Ths ma be kow as regresso through the org. Mmze: Q ( ) ˆ Compare to ˆ ˆ 7
Marquette Uverst Least Squares Regresso MSCS6 The least squares estmato score fucto Q ( a b ) s equvaletl represeted as Q ( X )'( X ) measured data,..., where desg matr, X, a. b regresso coeffcets 8
Marquette Uverst Least Squares Regresso MSCS6 We do t eed to take the dervatve of Q wrt β (although we could). We ca wrte wth algebra ( X )'( X ) ( X ˆ )'( X ˆ ) ( ˆ )'( X ' X )( ˆ ) add ad subtract X ˆ does ot deped o β where ˆ ( X ' X ) X '. It ca be see that vertble ˆ mamzes Q because t mmzes ( X )'( X ). 9
Marquette Uverst Least Squares Regresso MSCS6 Geerate smulated data b addg radom ose to a oseless le. 5 Let a b, 4 where ~ N(, ) 3 4 5 are depedet.,..., Measuremet Error 3 aˆ b ˆ True Le a b 3 3 4 4 5 5
Marquette Uverst Least Squares Regresso MSCS6 The smulated data ca be vewed as havg a ormal dstrbuto wth mea at the le oseless le value Let a b, where ~ N(, ) d 4 5 d 6 5 are depedet.,..., Measuremet Error d d 3 d 3 4 a b True Le
Marquette Uverst MSCS6 Least Squares Regresso Let a=, b=, ad σ= wth. a b ~ N(, ) Geerate values to go alog wth =,,3,4. = a + b + ε.5377 3.8339.74 4.86 = + 3 4 +.5377.8339 -.588.86 rg('default') =4; a=; b=; sgma=; =[,,3,4]'; e=sgma*rad(,); =a+b*+e; sumx=sum(); sumx=sum(.*); sumy=sum(); sumxy=sum(.*); bhat=(*sumxy-sumx*sumy)/(*sumx-sumx^) ahat=sumy/-bhat*sumx/
Marquette Uverst MSCS6 Least Squares Regresso a=, b=, ad σ= =,,3,4. a b ~ N(, ) fgure; le([,5],[ahat,ahat+bhat*5],'color','k') hold o scatter(,,'bo','flled') scatter(sumx/,sumy/,'ro','flled') grd o, as square lm([,5]), lm([,5]) set(gca,'tck',(:5)),set(gca,'tck',(:5)) 3
Marquette Uverst MSCS6 Least Squares Regresso As prevousl oted, the least squares regresso a b.5377 3.8339.74 4.86 = X β + (q+) (q+) = 3 4 ˆ ( X ' X ) ' (q+) X ca be wrtte as + ε.5377.8339 -.588.86 a b (q+) X (q+) 3 4 4
Marquette Uverst MSCS6 Least Squares Regresso As t turs out (ot show here), E( ˆ ) ˆ cov( ) ( X ' X ) W X 3 4.5.5 ( X ' X).5. 5
Marquette Uverst Least Squares Regresso Repeated 6 tmes ˆ ~ N, ( X ' X ) (q+) ( ab, )' W (X'X) Ea ( ˆ) aˆ.5 W.5 ˆ.55 s a MSCS6 a=, b=, ad σ= =,,3,4. um=^6; a=;b=; sgma=; =[,,3,4]'; =4; mu=a+b*';, X=[oes(,),]; =sgma*rad(um,)... +oes(um,)*mu; betahat=v(x'*x)*x'*'; fgure(), hst(betahat(,:),(-5:.:5)') fgure(), hst(betahat(,:),(-:.5:3)') betabar=mea(betahat,) covbetahat=cov(betahat') Eb ( ˆ) bˆ.9999 W. ˆ. s b aˆ bˆ W ˆ ˆ.53 cov(, ).5 s ab 6
Marquette Uverst Least Squares Regresso MSCS6 (,) pars: (,),(3,),(,3),(4,4) If we have radom error as depedet varable as depedet varable ad mmzed the sum of squared vertcal dstaces from the pot to the le. d d 3 d 4 s ˆ s ˆ ˆ d ˆ.5 ˆ.8 Toggle Forward 7
Marquette Uverst Least Squares Regresso MSCS6 (,) pars: (,),(3,),(,3),(4,4) If we have radom error as depedet varable as depedet varable ad mmzed the sum of squared horzotal dstaces from the pot to the le. d 3 d 4 ˆ s s ˆ ˆ ˆ ˆ ˆ ˆ ˆ.65 / / ˆ ˆ.5 ˆ ˆ d d Toggle Forward/Backward 8
Marquette Uverst MSCS6 Least Squares Regresso Whe there s measuremet error both ad, we model ths as ad where ad are observed ose data, ad are true uobserved values, ~ N(, ) ad ~ N(, ). If we specf that orthogoal regresso., the we have a 9
Marquette Uverst MSCS6 Least Squares Regresso I orthogoal regresso, ad are observed whle ad are true uobserved values. Dfferetatg Q wrt,, ad λ, the set =,..., obta soluto to orthogoal regresso. * * * * Q ( ) ( ) ( ) Q Q * * * * ˆ, ˆ, ˆ * * ˆ ˆ, ˆ, Q * * ˆ ˆ, ˆ, Lagrage Multpler
Marquette Uverst Least Squares Regresso MSCS6 (,) pars: (,),(3,),(,3),(4,4) as depedet varable as depedet varable ad mmzed the sum of squared orthogoal dstaces from the pot to the le. * ˆ ˆ ( s s ) ( s s ) 4s s ˆ ˆ ˆ ˆ ( ) ˆ ˆ ˆ. ˆ. Adcock. Aals of Mathematcs, 5:53-54,878. Demg. Statstcal Adjustmets of Data, 943.
Marquette Uverst Least Squares Regresso MSCS6 (,) pars: (,),(3,),(,3),(4,4) s ˆ s s ˆ s ˆ ˆ ˆ ˆ ˆ ˆ / / ˆ ˆ ˆ ˆ ˆ ( s s ) ( s s ) 4s s ˆ ˆ ˆ ˆ ( ) ˆ ˆ
Marquette Uverst MSCS6 Epermetal Desg I The Begg Eample: For m lab epermet I kow that there s a lear relatoshp betwee m depedet varable ad depedet varable. I ca select to be a value betwee m ad ma. Opto : Spread out s Select the values at ever Δ=/( ma m ). ˆ t s / S Opto : Clump the s Select / at m ad select / at ma. s ( ˆ ˆ ) S ( ) 3
Marquette Uverst MSCS6 Epermetal Desg I The Begg Eample: β =, β =, σ=.5 Opto : Spread out s = β + β + ε.688.969.876 4.43 5.594 5.346 6.783 8.73.789.3847 3 4 5.688.969 -.94.43.594 Opto : Clump the s ε.688.969 9.876.43.594 = + 6 7 8 9 + -.6538 -.68.73.789.3847 9.346 9.783.73.789.3847 = + +.688.969 -.94.43.594 -.6538 -.68.73.789.3847 same error added 4
Marquette Uverst MSCS6 Epermetal Desg I The Begg Eample: β =, β =, σ=.5 Opto : Spread out s Opto : Clump the s 5
Marquette Uverst MSCS6 Epermetal Desg I The Begg Eample: β =, β =, σ=.5 Opto : Spread out s Opto : Clump the s ˆ 9.649 ˆ.6 ˆ.887 ˆ.46 s =.73, S =8.5, t=.9 s =.84, S =.5, t=6.6 6
Marquette Uverst MSCS6 Homework:. For orthogoal regresso, let β =, β =, σ =, σ =. Geerate 6 radom les. *=[,,3,4]. For each set of 4 data pots, get estmates,, ˆ ( ) * ad Plot hstograms of estmates,, ˆ, ˆ, ˆ. Compute meas ad varaces. ˆ ( ) * Usg same data, repeat for ordar least squares. ˆ ˆ ˆ ˆ ˆ s ˆ ˆ 7
Marquette Uverst MSCS6 Homework:. For the epermetal desg regresso, repeat the two case le smulatos 6 tmes. For each smulato for each case, estmate ˆ s ( ˆ ),, ad. ˆ Plot hstograms of estmates. Compute meas & varaces of estmated values. Compute & make hstograms for the s ( X ' X ). 8