Unit 10: Simple Linear Regression and Correlation

Transcription

1 Unt 10: Smple Lnear Regresson and Correlaton Statstcs 571: Statstcal Methods Ramón V. León 6/28/2004 Unt 10 - Stat Ramón V. León 1 Introductory Remarks Regresson analyss s a method for studyng the relatonshp between two or more numercal varables In regresson analyss one of the varables s regarded as a response, outcome or dependent varable The other varables are regarded as predctor, explanatory or ndependent varables An emprcal model approxmately captures the man features of the relatonshp between the response varable and the key predctor varables Sometmes t s not clear whch of two varable should be the response. In ths case correlaton analyss s used. In ths unt we consder only two varables 6/28/2004 Unt 10 - Stat Ramón V. León 2

2 A Probablstc Model for Smple Lnear Regresson 2 Note that the Y are ndependent N( µ = β0 + β1x, σ ) r.v.'s Constant varablty about the regresson lne Notce that the predctor varable s regarded as nonrandom because t s assumed to be set by the nvestgator 6/28/2004 Unt 10 - Stat Ramón V. León 3 A Probablstc Model for Smple Lnear Regresson Let x1, x2,..., xn be specfc settngs of the predctor varable. Let y1, y2,..., yn be the correspondng values of the response varable. Assume that y s the observed value of a random varable (r.v.) Y, whch depends on x accordng to the followng model: Y = β0 + β1x + ε ( = 1, 2,..., n). 2 Here ε s the random error wth E( ε) = 0 and Var( ε) = σ Thus EY ( ) = µ = β0 + β1x (true regresson lne). We assume that ε are ndependent, dentcally dstrbuted (..d.) r.v.'s We also usually assume that the are normally dstrbuted. ε 6/28/2004 Unt 10 - Stat Ramón V. León 4

3 Reflectons on the Method of Tre Data Collecton: Expermental Desgn Mleage Depth Mleage n 1000 mles Depth n mls Was one tre or nne used n the experment? (Read the book!) What effect would ths answer have on the expermental error? on our ablty to generalze to all tres of ths brand? Was the data collected on one or several cars? Does t matter? Was the data collected n random order or n the order gven? Does t matter? Is there a possble confoundng problem? 6/28/2004 Unt 10 - Stat Ramón V. León 5 Example 10.1: Tread Wear vs. Mleage Usng the Ft Y by X platform n the Analyze menu 6/28/2004 Unt 10 - Stat Ramón V. León 6

4 Scatter Plot: Tread Wear vs. Mleage 6/28/2004 Unt 10 - Stat Ramón V. León 7 Least Squares Lne Ft y = ˆ β + ˆ β x 0 1 6/28/2004 Unt 10 - Stat Ramón V. León 8

5 Illustraton: Least Squares Lne Ft The least squares lne ft s the lne that mnmzes the sum of the squares of the lengths of the vertcal segments 6/28/2004 Unt 10 - Stat Ramón V. León 9 Mathematcs of Least Squares Fnd the lne,.e., values of β and β that mnmzes the sum of the squared devatons: [ ( β0 β1 ) ] = 1 How? Solve for values of β and β for whch Q β n Q= y + x Q = 0 and = 0 β 6/28/2004 Unt 10 - Stat Ramón V. León 10

6 Predct Mean Grove Depth for a Gven Mleage For x = 25 (25,000 mles) the predcted y: yˆ = ˆ β + ˆ β (25) = = mls. 0 1 Add a row to the table and leave the y value blank. 6/28/2004 Unt 10 - Stat Ramón V. León 11 Goodness of Ft of the LS Lne Ftted values of y : yˆ = ˆ β + ˆ β x, = 1,2,..., n. 0 1 ( ˆ β ˆ 0 β1 ) Resduals: e = y yˆ = y + x = segment length 6/28/2004 Unt 10 - Stat Ramón V. León 12

7 Sums of Squares Error Sum of Squares (SSE) = Total Sum of Squares (SST) = n n 2 e = y y = 1 = 1 ( y y) = 1 Regresson Sum of Squares (SSR) = ( ˆ ) 2 ( yˆ y) = 1 n 2 n 2 n 2 ( y y) = ( yˆ y) + ( y yˆ ) = 1 = 1 = 1 SST = SSR + SSE n n 2 2 These defntons apply even f the model s more complex than lnear n x, for example, f t s quadratc n x SST s nterpreted as the total varaton n y JMP calls SSR the Model Sums of Square 6/28/2004 Unt 10 - Stat Ramón V. León 13 y y Illustraton of Sums of Squares y y ˆ y yˆ y y 6/28/2004 Unt 10 - Stat Ramón V. León 14

8 Coeffcent of Determnaton SST = SSR + SSE 2 SSR SSE r = = 1 = proporton of the varaton n y SST SST that s accounted for by regresson on x SSR 50, SST = 53, /28/2004 Unt 10 - Stat Ramón V. León 15 s Estmaton of σ 2 n n n 2 e y y y 0 1x 2 = 1 = 1 = 1 2 ( ˆ ) ( ˆ β + ˆ β ) 2 = = = n 2 n 2 n 2 s 2 s Ths estmate has n-2 degrees of freedom because two unknown parameters are estmated. 6/28/2004 Unt 10 - Stat Ramón V. León 16

9 Statstcal Inference for β 0 and β 1 2. Select ˆ β0 β ˆ 0 β1 β1 N(0,1) and N(0,1) SD( ˆ β ) SD( ˆ β ) 0 1 ˆ β β ˆ β β t and t SE( ˆ β ) SE( ˆ β ) n 2 n (1- α)% CI : ˆ β ± t SE( ˆ β ) = 7.28 ± t = 1 n 2, α 2 1 7,0.025 [ ] 7.28 ± = 8.733, ˆ β ˆ 0 ± tn 2, α 2SE( β0) = ± t7, Rght clck on the Parameter Estmate area 6/28/2004 Unt 10 - Stat Ramón V. León 17 Test of Hypothess for β 0 and β 1 H : β = 0 vs. H : β ˆ β 0 ˆ β Reject H at α level f t = = > t SE( ˆ β ) SE( ˆ β ) t n 2, α 2 Tre example: ˆ β 7.28 SE( ˆ β ) = = = t7,.025 = , Strong evdence that mleage affects tread depth. (Two-sded p-value) 6/28/2004 Unt 10 - Stat Ramón V. León 18

10 The Mean Square (MS) s the Sums of Square dvded by ts degrees of freedom, e.g. MSE = SSE/df = /7 = H Analyss of Varance : β = 0 vs. H : β MSR F = MSE = F = = ( ) = t 2 2 F and t relatonshp holds only when the F numerator has one degree of freedom 6/28/2004 Unt 10 - Stat Ramón V. León 19 Predcton of a Future y or Its Mean For a fxed value of x*, are we tryng to predct -the average value of y? -the value of a future observaton of y? Example: Do I want to predct the average sellng prce of all 4,000 square feet houses n my neghborhood. Or do I want to predct the partcular future sellng prce of my 4,000 square feet house? Whch predcton s subject to the most error? 6/28/2004 Unt 10 - Stat Ramón V. León 20

11 Predcton of a Future y or Its Mean: Predcton Interval or Confdence Interval For a fxed value of x*, are we tryng to predct -the average value of y? -the value of a future observaton of y? 6/28/2004 Unt 10 - Stat Ramón V. León 21 JMP: Predcton of the Mean of y Grove Depth (n mls) Mleage (n 1000 Mles 6/28/2004 Unt 10 - Stat Ramón V. León 22

12 JMP: Predcton of a Future Value of y Grove Depth (n mls) Mleage (n 1000 Mles 6/28/2004 Unt 10 - Stat Ramón V. León 23 Formulas for Confdence and Predcton Intervals xx n ( ) 2 S = x x = 1 Predcton Interval of Chapter 7: 6/28/2004 Unt 10 - Stat Ramón V. León 24

13 Confdence and Predcton Intervals wth JMP Usng the Ft Model Platform not the Ft Y by X platform 6/28/2004 Unt 10 - Stat Ramón V. León 25 CI for Mean for µ* 6/28/2004 Unt 10 - Stat Ramón V. León 26

14 Predcton Interval for Y* 6/28/2004 Unt 10 - Stat Ramón V. León 27 Predcton for the Mean of Y or a Future Observaton of Y Pont estmate predcton s the same n both cases: But the error bands are dfferent Narrower for the mean of Y: [158.73, ] Wder for a future value of Y: [129.44, ] 6/28/2004 Unt 10 - Stat Ramón V. León 28

15 Calbraton (Inverse Regresson) 6/28/2004 Unt 10 - Stat Ramón V. León 29 Inverse Regresson n JMP 5: Confdence Interval In Ft Model Platform unchecked 6/28/2004 Unt 10 - Stat Ramón V. León 30

16 Inverse Regresson: Two Types of Confdence Intervals Calbraton s usually used n measurement. Gven that you got a certan measurement what s the true value of the measured quantty? Whch nverse predcton nterval makes more sense? 6/28/2004 Unt 10 - Stat Ramón V. León 31 Measurement Applcaton Suppose you get a measurement of 6.50 on an object. What s a calbraton nterval for ts true value? Data was supposedly obtaned by measurng standards of known true measurement wth a calbrated nstrument How the data was really generated 6/28/2004 Unt 10 - Stat Ramón V. León 32

17 Resduals The resduals are the dfferences between the observed value of y and the predcted value of y yˆ = ˆ β + ˆ β (32) = = mls 0 1 6/28/2004 Unt 10 - Stat Ramón V. León 33 The resduals e = y yˆ Regresson Dagnostcs can be vewed as the "left over" after the model s ftted. If the model s correct, then the e can be vewed as the "estmates" of the random errors ε ' s. As such, the resduals are vtal for checkng the model assumptons. Resdual plots If the model s correct the resduals should have no structure, that s, they should look random on ther plots. 6/28/2004 Unt 10 - Stat Ramón V. León 34

18 Mathematcs of Resduals If the assumed regresson model s correct, then the e ' s are normally dstrbuted wth Ee ( ) = 0 and ( x x) 2 Var( e ) = σ 1 σ f n s large. xx n Sxx The e ' s are not ndependent (even though the ε 's are ndependent) because they satsfy the followng contrants: n =1 =1 However, the dependence ntroduced by these constrants s neglble n n s modestly large, say greater than 20 or so. n e = 0, xe = 0. n ( ) 2 S = x x = 1 6/28/2004 Unt 10 - Stat Ramón V. León 35 Basc Prncple Underlyng Resdual Plots If the assumed model s correct then the resduals should be randomly scattered around 0 and should show no obvous systematc pattern. 6/28/2004 Unt 10 - Stat Ramón V. León 36

19 In Ft Y by X platform: In Ft Model platform: Red trangle menu next to Lnear Ft. Resduals n JMP 6/28/2004 Unt 10 - Stat Ramón V. León 37 Checkng for Lnearty Should E( Y) = β + β x+ β x be ftted rather than EY ( ) = β + β x? Grove Depth (n mls) Resduals Grove Depth (n mls) Mleage (n 1000 Mles Mleage (n 1000 Mles 6/28/2004 Unt 10 - Stat Ramón V. León 38

20 Predcted by Resdual Plot Resduals Grove Depth (n mls) Predcted Grove Depth (n mls) Mleage (n 1000 Mles 6/28/2004 Unt 10 - Stat Ramón V. León 39 Resduals Grove Depth (n mls) What are the advantages of ths plot? Tread Wear Quadratc Model 6/28/2004 Unt 10 - Stat Ramón V. León 40

21 Resduals for Quadratc Model Resduals have a random pattern The Ft Y by X platform was used to get these plots Left plot can also be obtaned usng the Plot Resduals opton 6/28/2004 Unt 10 - Stat Ramón V. León 41 Tread Wear Quadratc Model R 2 was for model lnear n x 6/28/2004 Unt 10 - Stat Ramón V. León 42

22 Checkng for Constant Varance: Plot of Predcted vs. Resdual If assumpton s ncorrect, often Var( Y ) s some functon of EY ( ) = µ 6/28/2004 Unt 10 - Stat Ramón V. León 43 Other Model Dagnostcs Based on Resduals Checkng for normalty of errors: Do a normal plot of the resduals Warnng: Don t plot the response y on a normal plot Ths plot has no meanng when one has regressors Don t transform the data on the bass of ths plot Many students make ths mstake n ther project. Don t be one of them. 6/28/2004 Unt 10 - Stat Ramón V. León 44

23 Other Model Dagnostcs Based on Resduals Ft Model Platform: Checkng for ndependence of errors: Plot the resdual n tme order. The order of data collecton should be always recorded. Use Durbn-Watson statstc to test for autocorrelaton. 6/28/2004 Unt 10 - Stat Ramón V. León 45 Other Model Dagnostcs Based on Resduals Checkng for outlers: See f any standardzed (Studentzed) resdual exceeds 2 (standard devatons) n absolute value. * e e e e = =, = 1,2,..., n. SE( e ) 2 1 ( x x) s s 1 n S xx Also do a box plot of resduals to check for outlers 6/28/2004 Unt 10 - Stat Ramón V. León 46

24 Standardzed (Studentzed) Resduals n JMP 4 Usng the Ft Model JMP platform wth lnear model Should we omt the frst observaton and reft the model? Possble outler The model ftted here s the one lnear n x, not the quadratc model 6/28/2004 Unt 10 - Stat Ramón V. León 47 Checkng for Influental Observatons An observaton can be nfluental because t has an extreme x-value, an extreme y-value, or both Note that the nfluental observaton above are not outlers snce ther resduals are qute small (even zero) 6/28/2004 Unt 10 - Stat Ramón V. León 48

25 Checkng for Influental Observatons n yˆ = h y where the h are some functons of the x's j j j j= 1 H = h j s called the hat matrx We can thnk of the hj as the leverage exerted by yj on the ftted value yˆ. If yˆ s largely determned by y wth very small contrbuton from the other yj's, then we say that the th observatons s nfluental (also called hgh leverage). 6/28/2004 Unt 10 - Stat Ramón V. León 49 e Checkng for Influental Observatons Snce h detemnes how much ˆ y depends on y t s used as a measure of the leverage of the -thobservaton. n Now h = k+ 1 or the average of h s ( k+ 1) / n, = 1 where k s the number of predctor varables varables. * Rule of thumb: Regard any h > 2( k+ 1) / n as hgh leverage. In smple regresson k = 1 and so h > 4 / n s regarded as hgh leverage. e e = = SE( e ) s 1 h 6/28/2004 Unt 10 - Stat Ramón V. León 50 Standardzed (Studentzed) resduals wll be large relatve to the resdual for hgh leverage observatons.

26 Graph of Anscombe Data Observaton n row 8 6/28/2004 Unt 10 - Stat Ramón V. León 51 Checkng for Influental Observatons Ft Model Platform: Anscombe Data Illustraton: > 4 / n= 4 /11 = 0.36 Thus observaton 8 s nfluental Notce: Observaton No. 8 s not an outler snce e ˆ 8 = y8 y8 = 0 6/28/2004 Unt 10 - Stat Ramón V. León 52 h

27 How to Deal wth Outlers and Influental Observatons They can gve a msleadng pcture of the relatonshp between y and x Are they erroneous observatons? If yes, they should be dscarded. If the observaton are vald then they should be ncluded n the analyss Least Squares s very senstve to these observatons. Do analyss wth and wthout outler or nfluental observaton. Or use robust method: Mnmze the sum of the absolute devatons n = 1 y ( β + β x ) 0 1 These estmates are analogous to the sample medan. LS estmates are analogous to the sample mean. 6/28/2004 Unt 10 - Stat Ramón V. León 53 Data Transformatons If the functonal relatonshp between x and y s known t may be possble to fnd a lnearzng transformaton analytcally: β For y = αx take log on both sdes: log y = logα + βlog x β x For y = αe take log on both sdes: log y = logα + βx If we are fttng an emprcal model, then a lnearzng transformaton can be found by tral and error usng the scatter plot as a gude. 6/28/2004 Unt 10 - Stat Ramón V. León 54

28 Scatter Plots and Lnearzng Transformatons 6/28/2004 Unt 10 - Stat Ramón V. León 55 β x y = αe log y = logα β x Tre Tread Wear vs. Mleage: Exponental Model 6/28/2004 Unt 10 - Stat Ramón V. León 56

29 Tre Tread Wear vs. Mleage: Exponental Model WEAR = e e = e MILES.0298MILES 6/28/2004 Unt 10 - Stat Ramón V. León 57 Tre Tread Wear vs. Mleage: Exponental Model For lnear model: R 2 =.953 Resdual plot s stll curved manly because observaton 1 s an outler 6/28/2004 Unt 10 - Stat Ramón V. León 58

30 Weghts and Gas Mleages of Model Year Cars 6/28/2004 Unt 10 - Stat Ramón V. León 59 Weghts and Gas Mleages of Model Year Cars 100 Use transformaton y. The new varable has unts y of gallons per 100 mles. Ths transformaton has the advantage of beng physcally nterpretable. Usng Ft Y by X platform A splne fts the data locally. Our pror curve fts have been global. 6/28/2004 Unt 10 - Stat Ramón V. León 60

31 Weghts and Gas Mleages of Model Year Cars 6/28/2004 Unt 10 - Stat Ramón V. León 61 Varance Stablzng Transformaton α Suppose σ µ We want to fnd a transformaton of y that yelds a constant varance. Suppose the transformaton s a power of the orgnal data, say y* = Y y λ For example: Take the square root of Posson count data snce for data so dstrbuted the mean and the varance are the same 6/28/2004 Unt 10 - Stat Ramón V. León 62

32 Box-Cox Transformaton: Emprcal Determnaton of Best Transformaton Ft Model Platform: λ λ 1 ( ) ( y 1 ) λy λ 0 λ y = yln y λ = 0 n where y = n y = geometrc mean = 1 6/28/2004 Unt 10 - Stat Ramón V. León 63 Transformaton n Tre Data A better ft than than the polynomal and wth one less parameter (3 versus 4) Transform Mleage (n 1000 Mles 6/28/2004 Unt 10 - Stat Ramón V. León 64

33 Weghts and Gas Mleages of Model Year Cars: Box-Cox Transformaton Recprocal transformaton among suggested transformatons 6/28/2004 Unt 10 - Stat Ramón V. León 65 Correlaton Analyss When t s not clear whch s the predctor varable and whch s the response varable When both varables are random Try the bvarate normal dstrbuton as a probablty model for the jont dstrbuton of two r.v. s Parameters: µ X, µ Y, σ X, σy and Cov (X, Y) ρ = Corr(X, Y) = Var(X)Var(Y) The correlaton ρ s a measure of assocaton between the two random varables X and Y. Zero correlaton corresponds to no assocaton. Correlaton of 1 or -1 represents perfect assocaton. 6/28/2004 Unt 10 - Stat Ramón V. León 66

34 Bvarate Normal Densty Functon 6/28/2004 Unt 10 - Stat Ramón V. León 67 Statstcal Inference on the Correlaton Coeffcent R = H n ( X X)( Y Y) = 1 n n 2 2 ( X X) ( Y Y) = 1 = 1 : ρ = 0 vs H : ρ R n-2 Test statstc T = t 2 n 2 1 R Equvalent to testng H : β = 0 vs. H : β 0 Approxmate test statstc avalable to Test nonzero null hypotheses Obtan confdence ntervals for ρ 6/28/2004 Unt 10 - Stat Ramón V. León See textbook

35 Exercse /28/2004 Unt 10 - Stat Ramón V. León 69 Exercse /28/2004 Unt 10 - Stat Ramón V. León 70

36 Exercse By default, a 95% bvarate normal densty ellpse s mposed on each scatterplot. If the varables are bvarate normally dstrbuted, ths ellpse encloses approxmately 95% of the ponts. The correlaton of the varables s seen by the collapsng of the ellpse along the dagonal axs. If the ellpse s farly round and s not dagonally orented, the varables are uncorrelated 6/28/2004 Unt 10 - Stat Ramón V. León 71