Applied Regression Modeling: A Business Approach Chapter 2: Simple Linear Regression Sections

Transcription

1 Applied Regression Modeling: A Business Approach Chapter 2: Simple Linear Regression Sections by Iain Pardoe 2.1 Probability model for and 2 Simple linear regression model for and Possible relationships between and Straight-line model HOMES2 data Simple linear regression model equation Least squares criterion 7 Least squares criterion Estimating the model Estimated equation Computer output Model evaluation 11 Model evaluation Evaluating fit numerically Regression standard error, s Regression standard error interpretation Coefficient of determination, R Calculating R Interpreting R R 2 examples Correlation Correlation examples Slope parameter, b Hypothesis test for b Computer output and slope test illustration Slope confidence interval

2 2.1 Probability model for and 2 / 24 Simple linear regression model for and is a quantitative response variable (a.k.a. dependent, outcome, or output variable). is a quantitative predictor variable (a.k.a. independent or input variable, or covariate). Two variables play different roles, so important to identify which is which and define carefully, e.g.: is sale price, in $ thousands; is floor size, in thousands of square feet. How much do we expect to change by when we change the value of? What do we expect the value of to be when we set the value of at 2? Note: association (observational data) not causation (experimental data). c Iain Pardoe, / 24 Possible relationships between and Which factors might lead to the different relationships? sale price sale price floor size floor size sale price sale price floor size floor size c Iain Pardoe, / 24 2

3 Straight-line model Simple linear regression models straight-line relationships (like upper two plots on last slide). Suppose sale price is (on average) $190,300 plus 40.8 times floor size. E( i ) = i, where E( i ) means the expected value of given that is equal to i. Individual sale prices can deviate from this expected value by an amount e i (called a random error ). i i = i + e i (i = 1,...,n). i i = deterministic part + random error. Error, e i, represents variation in due to factors other than which we haven t measured, e.g., lot size, # beds/baths, age, garage, schools. c Iain Pardoe, / 24 HOMES2 data = sale price (in thousands of dollars) E( ) = e = floor size (in thousands of square feet) c Iain Pardoe, / 24 3

4 Simple linear regression model equation Population: E( ) = b 0 + b 1. = response b 0 = intercept E( ) = b 0 + b 1 b 1 = slope change in change in = predictor c Iain Pardoe, / Least squares criterion 7 / 24 Least squares criterion Which line fits the data best? = sale price = floor size c Iain Pardoe, / 24 4

5 Estimating the model Population: E( ) = b 0 + b 1. Sample: Ŷ = ˆb 0 + ˆb 1 (estimated model). Obtain ˆb 0 and ˆb 1 by finding best fit line (least squares line). Mathematically, minimize sum of squared errors: n SSE = = = i=1 ê 2 i n ( i Ŷi) 2 i=1 n ( i ˆb 0 ˆb 1 i ) 2. i=1 Can use calculus (partial derivatives), but we ll use computer software to find ˆb 0 and ˆb 1. c Iain Pardoe, / 24 Estimated equation Sample: Ŷ = ˆb 0 + ˆb 1. = sale price ^ = b^0 + b^1 (fitted value) e^ (estimated error) = b^0 + b^1 + e^ (observed value) = floor size c Iain Pardoe, / 24 5

6 Computer output Parameters a Model Estimate Std. Error t-stat Pr(> t ) 1 (Intercept) a Response variable:. Estimated equation: Ŷ = ˆb 0 + ˆb 1 = We expect =ˆb 0 when =0, but only if this makes sense and we have data close to =0 (not the case here). We expect to change by ˆb 1 when increases by one unit, i.e., we expect sale price to increase by $40,800 when floor size increases by 1000 sq. feet. For this example, more meaningful to say we expect sale price to increase by $4080 when floor size increases by 100 sq. feet. c Iain Pardoe, / Model evaluation 11 / 24 Model evaluation How well does the model fit each dataset? c Iain Pardoe, / 24 6

7 Evaluating fit numerically Three methods: How close are the actual observed -values to the model-based fitted values, Ŷ? Calculate the regression standard error, s. How much of the variability in have we been able to explain with our model? Calculate the coefficient of determination, R 2. How strong is the evidence of a straight-line relationship between and? Estimate and test the slope parameter, b 1. c Iain Pardoe, / 24 Regression standard error, s Model Summary Adjusted Regression Model Multiple R R Squared R Squared Std. Error a a Predictors: (Intercept),. Regression standard error, s, estimates the std. dev. of the simple linear regression random errors: SSE s = n 2. Unit of measurement for s is the same as unit of measurement for. Approximately 95% of the observed -values lie within plus or minus 2s of their fitted Ŷ-values. Since 2s=5.57, we can expect to predict an unobserved sale price from a particular floor size to within approx. ±$5570 (at a 95% confidence level). c Iain Pardoe, / 24 7

8 Regression standard error interpretation CLT: 95% of -values lie within ±2s of regression line. + 2s 2s c Iain Pardoe, / 24 Coefficient of determination, R 2 Measures of variation for simple linear regression. = sale price n SSE = (i ^ i) 2 i=1 n TSS = (i m ) 2 i= = floor size c Iain Pardoe, / 24 8

9 Calculating R 2 Without model, estimate with sample mean m. With model, estimate using fitted Ŷ-value. How much do we reduce our error when we do this? Total error without model: TSS = n i=1 ( i m ) 2, variation in about m. Remaining error with model: SSE = n i=1 ( i Ŷi) 2, unexplained variation. Proportional reduction in error: R 2 = TSS SSE TSS. Home prices example: R 2 = = % of the variation in sale price (about its mean) can be explained by a straight-line relationship between sale price and floor size. c Iain Pardoe, / 24 Interpreting R 2 Model Summary Adjusted Regression Model Multiple R R Squared R Squared Std. Error a a Predictors: (Intercept),. R 2 measures the proportion of variation in (about its mean) that can be explained by a straight-line relationship between and. If TSS = SSE then R 2 = 0: using to predict hasn t helped and we might as well use m to predict regardless of the value of. If SSE = 0 then R 2 = 1: using allows us to predict perfectly (with no random errors). Such extremes rarely occur and usually R 2 lies between zero and one, with higher values of R 2 corresponding to better fitting models. c Iain Pardoe, / 24 9

10 R 2 examples R 2 = 0.05 R 2 = 0.13 R 2 = 0.3 R 2 = 0.62 R 2 = 0.74 R 2 = 0.84 R 2 = 0.94 R 2 = 0.97 R 2 = 1 c Iain Pardoe, / 24 Correlation Model Summary Adjusted Regression Model Multiple R R Squared R Squared Std. Error a a Predictors: (Intercept),. Correlation coefficient, r, measures the strength and direction of linear association between and : r 1 indicates a negative linear relationship; r +1 indicates a positive linear relationship; r 0 indicates no linear relationship. Simple linear regression: R 2 = absolute value of r ( multiple R above). But, correlation is less useful than R 2 in multiple linear regression. c Iain Pardoe, / 24 10

11 Correlation examples r = 1, R 2 = 1 r = 0.97, R 2 = 0.94 r = 0.81, R 2 = 0.66 r = 0.57, R 2 = 0.32 r = 0.01, R 2 = 0 r = 0.62, R 2 = 0.38 r = 0.87, R 2 = 0.76 r = 0.97, R 2 = 0.95 r = 1, R 2 = 1 c Iain Pardoe, / 24 Slope parameter, b 1 Infer from sample slope about the population slope. E() = b 0 + b 1 (expected value) e (random error) = b 0 + b 1 + e (population value) c Iain Pardoe, / 24 11

12 Hypothesis test for b 1 Recall univariate t-statistic = m E( ) s / n Here, slope t-statistic = ˆb 1 b 1 sˆb1 t n 2. NH: b 1 =0 versus AH: b 1 0. t n 1. t-statistic = ˆb 1 b 1 sˆb1 = = Significance level = 5%. Critical value is (97.5 th percentile of t 3 ). Since t-statistic (7.18) is between and , p-value is between 0.01 and Since t-statistic (7.18) > critical value (3.182) and p-value < signif. level, reject NH in favor of AH. In other words, the sample data favor a nonzero slope (at a significance level of 5%). Exercise: do an upper tail test for this example. c Iain Pardoe, / 24 Computer output and slope test illustration Parameters a Model Estimate Std. Error t-stat Pr(> t ) 1 (Intercept) a Response variable:. c Iain Pardoe, / 24 12

13 Slope confidence interval Calculate a 95% confidence interval for b th percentile of t 3 is ˆb1 ± 97.5 th percentile (sˆb1 ) = 40.8 ± = 40.8 ± 18.1 = (22.7, 58.9). Loosely speaking: based on this dataset, we are 95% confident that the population slope, b 1, is between 22.7 and More precisely: if we were to take a large number of random samples of size 5 from our population of homes and calculate a 95% confidence interval for each, then 95% of those confidence intervals would contain the (unknown) population slope. Exercise: calculate a 90% confidence interval for b 1. c Iain Pardoe, / 24 13