Chapter 11: Simple Linear Regression and Correlation

Transcription

1 Chapter 11: Smple Lnear Regresson and Correlaton 11-1 Emprcal Models 11-2 Smple Lnear Regresson 11-3 Propertes of the Least Squares Estmators 11-4 Hypothess Test n Smple Lnear Regresson Use of t-tests Analyss of varance approach to test sgnfcance of regresson 11-5 Confdence Intervals Confdence ntervals on the slope and ntercept Confdence nterval on the mean response 11-6 Predcton of New Observatons 11-7 Adequacy of the Regresson Model Resdual analyss Coeffcent of determnaton (R 2 ) 11-8 Correlaton 11-9 Regresson on Transformed Varables Logstc Regresson 1 Chapter Learnng Objectves After careful study of ths chapter you should be able to: 1. Use smple lnear regresson for buldng emprcal models to engneerng and scentfc data 2. Understand how the method of least squares s used to estmate the parameters n a lnear regresson model 3. Analyze resduals to determne f the regresson model s an adequate ft to the data or to see f any underlyng assumptons are volated 4. Test the statstcal hypotheses and construct confdence ntervals on the regresson model parameters 5. Use the regresson model to make a predcton of a future observaton and construct an approprate predcton nterval on the future observaton 6. Apply the correlaton model 7. Use smple transformatons to acheve a lnear regresson model 2

2 Emprcal Models Many problems n engneerng and scence nvolve explorng the relatonshps between two or more varables. Regresson analyss s a statstcal technque that s very useful for these types of problems. For example, n a chemcal process, suppose that the yeld of the product s related to the process-operatng temperature. Regresson analyss can be used to buld a model to predct yeld at a gven temperature level. 3 Emprcal Model - Example Data 4

3 Emprcal Model - Example Plot Fgure 11-1: Scatter dagram of oxygen purty versus hydrocarbon level from Table Smple Lnear Regresson Based on the scatter dagram, t s probably reasonable to assume that the mean of the random varable Y s related to x by the followng straght-lne relatonshp: where the slope and ntercept of the lne are called regresson coeffcents. The smple lnear regresson model s gven by where s the random error term. 6

4 Varance of Y = Varance of ε We thnk of the regresson model as an emprcal model. Suppose that the mean and varance of are 0 and 2, respectvely, then: The varance of Y gven x s: 7 Model of True Regresson Lne The true regresson model s a lne of mean values: where 1 can be nterpreted as the change n the mean of Y for a unt change n x (slope of the lne). The varablty of Y at a partcular value of x s determned by the error varance, 2. Ths mples there s a dstrbuton of Y-values at each x and that the varance of ths dstrbuton s the same at each x. 8

5 Dstrbuton of Y along Lne Fgure 11-2:The dstrbuton of Y for a gven value of x for the oxygen purty-hydrocarbon data. 9 Predctor and Response Varables The case of smple lnear regresson consders a sngle regressor or predctor x and a dependent or response varable Y. The expected value of Y at each level of x s a random varable: We assume that each observaton, Y, can be descrbed by the model: 10

6 Suppose that we have n pars of observatons (x 1, y 1 ), (x 2, y 2 ), (x n, y n ). The method of least squares s used to estmate the parameters, 0 and 1, by mnmzng the sum of the squares of the vertcal devatons. Fgure 11-3: Devatons of the data from the estmated regresson model. Method of Least Squares 11 Sum of Square Devatons Snce the n observatons n the sample can be expressed as: The sum of the squares of the devatons (errors) of the observatons from the true regresson lne s: 12

7 Least Squares Normal Equatons 13 Smple Lnear Regresson Coeffcents 14

8 Ftted Regresson Lne n x x x x S n n n xx n y x y x x x y y S n n n n xy Sums of Squares The followng notaton may also be used: Then, xx xy S S 1 ˆ x y 1 0 ˆ ˆ and (11-10) (11-11)

9 Smple Lnear Regresson - Example Example Example 11-1 (contnued) 18

10 Example 11-1 (contnued) Fgure 11-4: Scatter plot of oxygen purty y versus hydrocarbon level x and regresson model ŷ = x. 19 Computng 2 The error sum of squares s: It can be shown that the expected value of the error sum of squares s E(SS E ) = (n 2) 2. An unbased estmator of 2 s: where SS E can be easly computed usng: 20

11 21 Excel Data Analyss Tool Regresson output SUMMARY OUTPUT Regresson Statstcs Multple R R Square Adjusted R Square Standard Error Observatons ANOVA df SS MS F Sgnfcance F Regresson Resdual Total Coeffcents Standard Error t Stat P-value Intercept X Varable

12 Propertes of Least Squares Estmators Slope propertes for the mean and varance (11-15) (11-16) Intercept propertes for the mean and varance (11-17) 23 Estmated Standard Errors In smple lnear regresson the estmated standard error of the slope and the estmated standard error of the ntercept are: se 2 2 ˆ ˆ 21 x seˆ 0 ˆ 1 S xx n S xx respectvely, where the estmated varance s computed usng Equaton

13 Hypothess Test for the Slope If we wsh to test the slope s some value β 1,0 : (11-18) An approprate test statstc would be: ˆ 1 ˆ 1,0 1 T0 2 ˆ S se ˆ We would reject the null hypothess f: XX 1 1,0 (11-19) (11-20) 25 Hypothess Test for the Intercept If we wsh to test the ntercept s some value β 0,0 : (11-21) An approprate test statstc would be: (11-22) We would reject the null hypothess f: 26

14 Sgnfcance of Regresson An mportant specal case of these hypotheses s: (11-23) Falure to reject H 0 s equvalent to concludng that there s no lnear relatonshp between x and Y. In other words, f we conclude the slope could be 0 the nformaton on x tells us nothng about the varaton n the response, Y. 27 Fgure 11-5: The hypothess H 0 : 1 = 0 s not rejected. Fgure 11-6: The hypothess H 0 : 1 = 0 s rejected. 28

15 Hypothess Testng - Example Example Analyss of Varance (ANOVA) The analyss of varance dentty s: If the null hypothess, H 0 : β 1 = 0 s true, the statstc follows the F 1,n-2 dstrbuton and we would reject f f 0 > f,1,n-2. 30

16 The ANOVA Table The quanttes MS R and MS E are called mean squares of the regresson and the errors, respectvely. Analyss of varance (ANOVA) table: 31 Analyss of Varance - Example Example

17 Equvalence of t-tests and ANOVA 33 Confdence Intervals on Regresson Model Parameters The followng state the confdence ntervals for the slope and ntercept of a regresson model. 34

18 Example 11 4 (Confdence Interval on the Slope) β Confdence Interval on the Mean Response The pont estmate for the response at a gven x s: ˆ ˆ ˆ x Y x 0 The confdence nterval for the mean response s then:

19 Example 11 5 (Confdence Interval on the Mean Response) 37 Example 11 5 (contnued) 38

20 Example 11 5 (contnued) Fgure 11-7: Scatter dagram of oxygen purty data from Example 11-1 wth ftted regresson lne and 95% confdence lmts on Y x0. 39 Predcton of New Observatons The response pont estmate for a new observaton at x 0 s: Yˆ 0 ˆ ˆ x 0 The predcton nterval for the new response, Y 0, s then:

21 Example 11 6 (Predcton Interval) 41 Example 11 6 (contnued) 42

22 Example 11 6 (contnued) Fgure 11-8: Scatter dagram of oxygen purty data from Example 11-1 wth ftted regresson lne, 95% predcton lmts (outer lnes), and 95% confdence lmts on Y x0. 43 Adequacy of Regresson Models Fttng a regresson model requres several assumptons. 1. Errors are uncorrelated random varables wth mean zero; 2. Errors have constant varance; and, 3. Errors be normally dstrbuted. The analyst should always consder the valdty of these assumptons to be doubtful and conduct analyses to examne the adequacy of the model 44

23 Resdual (Error) Analyss The resduals from a regresson model are e = y - ŷ, where y s an actual observaton and ŷ s the correspondng ftted value from the regresson model. Analyss of the resduals s frequently helpful n checkng the assumpton that the errors are approxmately normally dstrbuted wth constant varance, and n determnng whether addtonal terms n the model would be useful. 45 Resdual Plots Fgure 11-9: Patterns for resdual plots. (a) satsfactory, (b) funnel, (c) double bow, (d) nonlnear. 46

24 Resdual Analyss - Example Example Example 11-7 (contnued) 48

25 Example 11-7 (contnued) Fgure 11-10: Normal probablty plot of resduals, Example Example 11-7 (contnued) Fgure 11-11: Plot of resduals versus predcted oxygen purty, ŷ, Example

26 Coeffcent of Determnaton (R 2 ) The quantty s called the coeffcent of determnaton and s often used to judge the adequacy of a regresson model. 0 R 2 1; (11-34) We often refer (loosely) to R 2 as the amount of varablty n the data explaned or accounted for by the regresson model. 51 R 2 Computatons - Example For the oxygen purty regresson model, R 2 = SS R /SS T = / = Thus, the model accounts for 87.7% of the varablty n the data. 52

27 Regresson on Transformed Varables In many cases a plot of the ndependent varable, y, aganst the dependent varable, x, may show the relatonshp s not lnear. Performng a lnear regresson would lead to a poor ft and resdual analyss would show the model s nadequate. However, we can often transform the dependent varable frst. Ths transformed varable, x, may have a lnear relatonshp wth y. 53 Therefore, we can perform a lnear regresson between the x and y. However, note that any use of the new equaton for predcton would requre a reverse transformaton to ndcate the desred value of x. Transformaton can take on many forms. Typcal ones nclude: x = logarthm (x) x = square root (x) x = nverse (x). 54

28 Example 11-9 An engneer has collected data on the DC output from a wndmll under dfferent wnd speed condtons. He wshes to develop a model descrbng output n terms of wnd speed. The table on the rght shows the data collected for output, y, as a response and wnd speed, x, as the dependent varable. The fnal column shows the transformed value, x =1/x. Obs. Output (y) Velocty (x) x'=1/x Example 11-9 (contnued) DC Output Wnd Velocty, x Orgnal Regresson Equaton (Orgnal Data): y = x R 2 =

29 Example 11-9 (contnued) DC Output Transformed Transformed Wnd Velocty, 1/x Regresson Equaton (Transformed Data): y = x R 2 = THE END OF ENGG 319 CLASS NOTES 58