ORF 245 Fundamentals of Statistics Chapter 14 Least Squares Regression

Transcription

1 ORF 245 Fudametals of Statstcs Chapter 14 Least Squares Regresso Robert Vaderbe Fall 2014 Sldes last edted o December 12, rvdb

2 Least Squares (Recallg two sldes from Chapter 4) Suppose we kow from some uderlyg fudametal prcple (say physcs for example) that some parameter y s related learly to aother parameter x: y = α + βx but we do t kow α ad β. We d lke to do expermets to determe them. A probablstc model of the expermet has X ad Y as radom varables. Let s mage we do the expermet over ad over may tmes ad have a good sese of the jot dstrbuto of X ad Y. We wat to pck α ad β so as to mmze E(Y α βx) 2 Aga, we take dervatves ad set them to zero. Ths tme we have two dervatves: ad ( ) α E(Y α βx)2 = E (Y α βx)2 α = 2E(Y α βx) = 2(µ Y α βµ X ) = 0 ( ) β E(Y α βx)2 = E (Y α βx)2 = 2E ( (Y α βx)x ) = 2 ( E(XY ) αe(x) βe(x 2 ) ) = 0 β 1

3 Least Squares Cotued (Recallg two sldes from Chapter 4) We get two lear equatos two ukows α + βµ X = µ Y αµ X + βe(x 2 ) = E(XY ) Multplyg the frst equato by µ X ad subtractg t from the secod equato, we get Ths equato smplfes to βe(x 2 ) βµ 2 X = E(XY ) µ X µ Y ad so βσ 2 X = σ XY β = σ XY σ 2 X = ρ σ Y σ X Fally, substtutg ths expresso to the frst equato, we get α = µ Y ρ σ Y σ X µ X 2

4 Regresso (Here starts the ew stuff) As before, suppose that t s kow (or beleved) that there s a smple lear relato betwee two measurable quattes x ad y: y = α + βx Suppose further that x ca be set wth arbtrary precso but that y s measured wth some error. A statstcal model for such as stuato volves radom varables: Y = α + βx + ε Here, ε ad (therefore) Y are radom varables. Our am s to derve estmates for α ad β based o a samplg of sze. We assume that x ca be vared as we lke the samplg process. So, we have samples lke ths: Y = α + βx + ε, = 1, 2,..., We wsh to choose the α ad β that gve the best ft to the sample. I other words, we choose them so as to mmze ε 2 = (Y α βx ) 2 3

5 Mmzg the Sum of Squares To mmze, we take the dervatves wth respect to α ad β ad set them to zero: (Y α βx ) 2 = α (Y α βx ) 2 = β Dvdg both sdes by 2, these equatos smplfy to where x = 1 x, Y = 1 Y, Y α βx = 0 xy αx βx 2 = 0 2 (Y α βx ) ( 1) = 0 2 (Y α βx ) ( x ) = 0 xy = 1 x Y, x 2 = 1 Ths s two equatos two ukows. Multplyg the frst equato by x ad subtractg that from the secod equato, we ca solve for β ad, kowg β, we ca use the frst equato to solve for α: ˆβ = xy x Y x 2 x 2, ˆα = Y ˆβx x 2 4

6 Mea of ˆα ad ˆβ Recallg that ˆβ = xy x Y x 2 x 2 ad ˆα = Y ˆβx We frst compute the expected value of ˆβ: E( ˆβ) = E(xY ) x E(Y ) = E( 1 = = = β x 2 x 2 x Y ) x E( 1 x 2 x 2 1 x E(Y ) x 1 x 2 x 2 1 x (α + βx ) x 1 x 2 x 2 Y ) E(Y ) (α + βx ) Next, we compute the expected value of ˆα: E(ˆα) = E(Y ) E( ˆβ) x = E( 1 Y ) β x = 1 E(Y ) β x = 1 (α + βx ) β x = α The estmators are ubased! 5

7 Varace of ˆα ad ˆβ Aga, recall that ˆβ = xy x Y x 2 x 2 ad ˆα = Y ˆβx To compute the varace of ˆα ad ˆβ, we eed these computatos: Usg these formulas, we get xy x Y = 1 x Y x 1 Y = 1 (x x) Y var ( xy x Y ) = var 1 (x x) Y 1 = 2 (x x) 2 σ 2 ˆα = Y xy x Y x 2 x 2 x = x2 Y x xy x 2 x 2 = ) 1 (x 2 x x Y x 2 x 2 var( ˆβ) = var(xy xy ) ( x 2 x 2 ) 2 = σ2 1 (x x) 2 ( ) 2 x 2 x 2 = σ2 1 x 2 x 2 ad var( ˆα) = 1 2 ( x 2 x x ) 2 σ 2 ( x 2 x 2 ) 2 = σ2 1 (x 2 x x ) 2 ( x 2 x 2 ) 2 = = σ2 x 2 x 2 x 2 6

8 Cofdece Itervals for α ad β Subtractg the mea ad dvdg by the stadard devato, we see that the radom varables ˆα α var(ˆα) = σ ˆα α x 2 x 2 x 2 ad ˆβ β var( ˆβ) = ˆβ β σ 1 x 2 x 2 have mea zero ad varace oe. If the ε s are Normally dstrbuted (wth mea 0 ad varace σ 2 ), the the above radom varables are stadard Normal radom varables. We do t kow σ 2. It ca be show that S 2 = 1 (Y ˆα 2 ˆβx ) 2 s a ubased estmate of σ 2 ad the correspodg ormalzed rv s are approxmately ormally dstrbuted (t-dstrbuto wth 2 degrees of freedom f s small). Puttg ths all together, we get the followg 95% cofdece tervals for ˆα ad ˆβ: α ˆα ± 1.96 S x 2 x2 x 2 β ˆβ ± 1.96 S 1 x2 x 2 If s small (less, say, that 30) the replace 1.96 wth t 2 (0.025). 7

9 Back to Temp vs. Tme at McGure AFB Recall from Chapter 10 our plot of oe-year averages: Temperature (F) Year Ths s = 55 data pots where x s the year (wth, say, year 0 beg 1955) ad y s average temperature for year x. 8

10 Temp vs. Tme Regresso Ft 58 Lear Regresso Temperature (F) ˆα = 52.81±0.53 ˆβ = ± Year T = McGureAFB(:,2); [ m] = sze(t); T = T(1: 365*floor(/365)); T = reshape(t, 365, floor(/365)); % Code for cofdece terval... y = mea(t)'; [ m] = sze(y); eps = y - alpha - beta*x; x = (0:(-1))'; S = sqrt(sum(eps.^2)/(-2)); xbar = sum(x)/; alpha_wdth = 1.96 * (S/sqrt())... ybar = sum(y)/; * (sqrt(x2bar))/(sqrt(x2bar-xbar^2)); xybar = sum(x.*y)/; beta_wdth = 1.96 *... x2bar = sum(x.*x)/; (S/sqrt()) /(sqrt(x2bar-xbar^2)); beta = (xybar - xbar*ybar)/(x2bar - xbar^2) alpha = ybar - beta*xbar plot( 1955+x, y,'b-', 1955+x, y,'r+',... [ ], [alpha alpha+beta*( )], 'g-'); 9

11 Temp vs. Tme Epslos 58 Lear Regresso Temperature (F) ˆα = 52.81±0.53 ˆβ = ± Year 10

12 Temp vs. Tme Regresso Ft Goodess of Ft Here we plot the emprcal cdf for the ε s o top of the cdf of a ormal dstrbuto wth the same mea ad varace:

13 Temp vs. Tme Regresso Ft Rollg Average Temperature (F) Lear Regresso ˆα = ˆβ = Days 10 4 T = McGureAFB(:,2); wdow = oes(365,1)/365; y = cov(t,wdow,'vald'); [ m] = sze(y); x = (0:(-1))'; xbar = sum(x)/; ybar = sum(y)/; xybar = sum(x.*y)/; x2bar = sum(x.*x)/; betahat = (xybar - xbar*ybar)/(x2bar - xbar^2) alphahat = ybar - betahat*xbar plot(x,y,'b-', [x(1) x(ed)], [alphahat alphahat+betahat*365*( )], 'g'); Note: Cofdece terval ot computed because ε s are ot depedet. 12

14 Temp vs. Tme Multple Lear Regresso Let s take to accout the seasoal oscllatos: y = β 1 + β 2 x + β 3 cos(2πx /365.25) + β 4 s(2πx /365.25) + ε = 1, 2,..., Aga, we look for values of the parameters that mmze the sum of the squares of the ε s. Ths tme, however, we have to set four dervatves equal to zero. That s four equatos four ukows. Not fu. Fortuately, Matlab provdes a tool precsely for ths purpose. We put the temperatures a colum vector y, y = ad we put the umbers that multply the β j s to a matrx X wth rows ad 4 colums 1 x 1 cos(2πx 1 /365.25) s(2πx 1 /365.25) 1 x X = 2 cos(2πx 2 /365.25) s(2πx 2 /365.25) x cos(2πx /365.25) s(2πx /365.25) y 1 y 2. y, 13

15 Multple Lear Regresso Cotued It may look complcated but y ad X are just a vector ad a matrx of umbers. Here s a few etres from y ad X: y = X = Ad, our statstcal model ca be wrtte ths matrx otato: y = Xβ + ε where β = β 1 β 2 β 3 β 4 ad ε s a log vector of errors. 14

16 Multple Lear Regresso Matlab Lear least squares s so mportat that Matlab has made t especally easy to compute the vector β that mmzes the sum of the squares of the ε s: ˆβ = X\y Here s the full Matlab code used to make the plot o the followg page: load -asc data/mcgureafb.dat; y = McGureAFB(:,2); [ m] = sze(y); % least squares regresso dt = (1:)'; X = [ oes(,1) dt cos(2*p*dt/365.25) s(2*p*dt/365.25) ]; betahat = X\y betahat(2)*356.25*100 plot( 1955+dt/365.25, y, 'b+', 1955+dt/365.25, X*betahat, 'r-'); 15

17 Multple Lear Regresso Output The umercal aswers are: β 1 = , β 2 = , β 3 = , β 4 = The lear tred s ecoded β 2. The value show above s degrees Fahrehet per day. To covert to degrees per cetury, we eed to multply by We get lear tred = β 2 = F/cetury 16

18 Cofdece Itervals va Bootstrap It s possble, but tedous, to derve formulas for the varaces (ad covaraces) of the ˆβ s the clmate model. A alterate method for producg cofdece tervals s called Bootstrap. The dea s as follows. Whe s large (say, = 20,314, as the clmate model), the we ca use the computed regresso coeffcets to produce a large emprcal pool of ε s: ( ˆε = y ˆβ1 + ˆβ 2 x + ˆβ 3 cos(2πx /365.25) + ˆβ ) 4 s(2πx /365.25) I other words, ˆε = y X ˆβ By samplg (wth replacemet) from ths pool of ˆε s, we ca artfcally geerate ew data sets that are statstcally smlar to the orgal oe: ỹ = X ˆβ + ε (here, ε s oe of the ˆε j s draw at radom from the pool of such epslos). Usg ths ew data set, we ca compute ew values for the estmators: β = X\ỹ 17

19 Cofdece Itervals va Bootstrap Cotued We ca do ths over ad over aga to geerate a large sample of ew estmators. The expected value of these ew estmators s uchaged from the orgal. But, they wo t be all the same ad therefore we ca use them to compute a emprcal stadard devato ad use that to compute cofdece tervals. The Matlab code to do ths bootstrap s rather straghtforward: Ad, the aswer s... eps = y - X*beta; beta_ew = zeros(4,100); dces = rad(,); % a x matrx of radom umbers betwee 1 ad for =1:100 y_ew = X*beta + eps(dces(:,)); beta_ew(:,) = X\y_ew; ed betahat = mea(beta_ew'); betahatstd = std(beta_ew'); * betahat(2) * 1.96*betahatstd(2) β 2 = ± F/cetury 18

20 Goodess of Ft As before, we plot the emprcal cdf for the ε s o top of the cdf of a ormal dstrbuto wth the same mea ad varace:

21 A Few Fal Remarks o Clmate Data The clmate data for McGure AFB ca be dowloaded from here... Here s a publshed paper that descrbes the data ad ts aalyss some detal

22 Had Lear Algebra Bee A Prereq... The course s over. What follows s a short advertsemet for ORF

23 Had Lear Algebra Bee A Prereq... Regresso model: y = Xβ + ε. Fd β that mmzes sum of squared errors: where ˆβ = argm β y Xβ 2 = argm β (y Xβ) T (y Xβ) = argm β f(β) f(β) = y T y β T X T y y T Xβ + β T X T Xβ. Take the gradet wth respect to β ad equate to zero: The estmator s ubased: X T X ˆβ = X T y = ˆβ = (X T X) 1 X T y E( ˆβ) = (X T X) 1 X T E(y) = (X T X) 1 X T Xβ = β The covaraces amog all of the coeffcets s also easy to compute: Cov( ˆβ) = E( ˆβ ˆβ ( T ) ββ T = E (X T X) 1 X T yy T X(X T X) 1) ββ T ( = E (X T X) 1 X T (Xβ + ε)(xβ + ε) T X(X T X) 1) ββ T = σ 2 (X T X) 1 22

24 The 2-Dmesoal Case Suppose that β = [ β0 β 1 ] ad X = [ e x ] where e deotes a colum vector of all oes ad x s a colum vector cotag the x s. I ths case, X T X = [ e T x T ] [ ] [ ] e T x e x = e T x x T x Hece [ ] σ 2 (X T X) 1 = σ 2 x T x e T x 1 e T x x T x (e T x) 2 [ ] = σ 2 x 2 x 1 x 2 x 2 (x) 2 [ ] = σ2 x 2 x 1 x 1 x 2 (x) 2 The etres o the dagoal are cosstet wth the formulas we derved before. 23

25 Back To The k-dmesoal Case Recall: y = Xβ + ε, E(ε) = 0 ˆβ = (X T X) 1 X T y, E( ˆβ) = β Let ˆε = y X ˆβ = y X(X T X) 1 X T y = P y = P (Xβ + ε) where P = I X(X T X) 1 X T = UU T where U s a ( k) orthoormal matrx (U T U = I). Note that P X = 0 ad P T P = P. We compute... E ( ˆε 2) = E ( P (Xβ + ε) ) T ( P (Xβ + ε) ) = β T X T P Xβ + Eε T P T P ε = Eε T P ε = Eε T UU T ε = = ( k)σ 2 Therefore, S = 1 k ˆε 2 = 1 k y X ˆβ 2 s a ubased estmator of σ 2. 24