STAT 4385 Topic 03: Simple Linear Regression

Transcription

1 STAT 4385 Topic 03: Simple Linear Regression Xiaogang Su, Ph.D. Department of Mathematical Science University of Texas at El Paso Spring, 2017

2 Outline The Set-Up Exploratory Data Analysis (EDA) Scatterplot Coefficient of Correlation Model Specification Model Estimation LSE of Betas Estimation of Error Variance Statistical Inference

3 Set-Up The Set-Up Data consists of {(x i, y i ) : i = 1,..., n} that consists of n IID copies of (X, Y ), where both the response Y and the predictor X are continuous. Want to study the association/relationship between Y and X. Examples: Revenue vs. advertising expenditure; Population over years Daily rainfall vs. barometric pressure in a place College GPA vs. high school GPA Data Layout ID Y X 1 y 1 x 1 2 y 2 x n y n x n

4 Set-Up A Real Example A study is conducted to investigate the relationship between cigarette smoking during pregnancy and the weights of newborn infants. A sample of 15 woman smokers kept accurate records of the number of cigarettes smoked (X ) during their pregnancies, and weights of their children (Y ) were recorded at birth. The data are given in the table. Cigarettes Birth Weight ID Per Day (X ) Weight (Y )

5 Exploratory Data Analysis (EDA) Exploratory Data Analysis (EDA) Exploratory Data Analysis (EDA) summarizes and describes data and helps see what the data can tell us before and beyond the formal modeling or hypothesis testing task. Two Approaches: Graphical or Numerical For data in SLR, EDA is aimed to explore the bivariate association between X and Y : Graphical Displays: scatterplot Numerical Measures: correlation coefficient

6 Exploratory Data Analysis (EDA) Scatterplot Scatterplot A scatterplot (also called scatter chart, scattergram, scatter diagram) displays values for typically two variables for a set of data. Can be extended to 3D; using color-coded points allows for displaying another categorical variable. Inspect a scatterplot for patterns and outliers: Linear or nonlinear pattern? Positive or negative (monotonic) association between X and Y? Any potential outlier?

7 Exploratory Data Analysis (EDA) Scatterplot Scatterplot

8 Exploratory Data Analysis (EDA) Scatterplot Birthweight Example: Scatterplot birthweight number of daily cigas

9 Exploratory Data Analysis (EDA) Coefficient of Correlation Pearson Correlation Coefficient The Pearson product moment coefficient of correlation measures the direction and strength of the linear association between two variables. cov(x,y ) The population version: ρ(x, Y ) = var(x ) var(y ) Point Estimation with Data: the sample version r(x, Y ) = = n i=1 (x i x)(y i ȳ) n i=1 (x i x) 2 n i=1 (y i ȳ) 2 n i=1 x iy i n xȳ { n i=1 x 2 i n x 2} { n i=1 y 2 i n ȳ 2}

10 Exploratory Data Analysis (EDA) Coefficient of Correlation Facts on ρ and r r is scaleless with 1 r 1. Direction of Linear Association: A positive r indicates a positive linear association (meaning Y increases as X increases) while r < 0 indicates a negative linear association. Their absolute values measure the strength of the linear association. The rule of thumb: When r = 1, perfect linear association. When r = 0.80, strong linear association When r = 0.50, moderate linear association When r = 0.20, weak linear association When r = 0, no linear association

11 Exploratory Data Analysis (EDA) Coefficient of Correlation Linear or Nonlinear Association Pearson correlation does not provide info on nonlinear association. Moreover, association does NOT imply causation.

12 Exploratory Data Analysis (EDA) Coefficient of Correlation Calculation of r Preliminary calculation of six quantities: { n, i x i, i y i, i x 2 i, i y 2 i, i x i y i }. Obtain x = i x i/n and ȳ = i y i/n; Next compute SS xx = i SS yy = i SS xy = i x 2 i n x 2 y 2 i n ȳ 2 x i y i n x ȳ; Finally compute r = SS xy SSxx SS yy.

13 Exploratory Data Analysis (EDA) Coefficient of Correlation Worksheet: Computing r Cigarettes Birth Weight ID Per Day (x i ) Weight (y i ) xi 2 yi 2 x i y i sum ,

14 Exploratory Data Analysis (EDA) Coefficient of Correlation Example: Computing r We have found that n = 15, x i = 380, y i = 115.5, x 2 i = 10842, y 2 i = , and x i y i = Hence x = 380/15 = and ȳ = 115.5/15 = 7.7 SS xx = x 2 i n x 2 = = SS yy = y 2 i nȳ 2 = = SS xy = x i y i n xȳ = = 50.7 Thus r = SS xy SSxx SS yy = = , which shows a somewhat moderate negative linear association.

15 Exploratory Data Analysis (EDA) Coefficient of Correlation Inference on ρ Case I Test for zero correlation, i.e., H 0 : ρ = 0 vs. H a : ρ 0 Preferable to use the equivalent test of zero slope in a simple linear regression model. Assuming (X, Y ) follow a bivariate normal distribution with ρ = 0, the fact r n 2/ 1 r 2 t (n 2) leads to a t test.

16 Exploratory Data Analysis (EDA) Coefficient of Correlation Inference on ρ Case II Test on a non-zero correlation H 0 : ρ = ρ 0 Assuming (X, Y ) follow a bivariate normal distribution, Fisher s (monotonic) z-transformation converts r into an almost normally distributed variable with constant variance 1/(n 3): r = arctanh(r) = 1 2 ln 1 + r 1 r N ( 1 2 ln 1 + ρ ) 1 ρ, 1, n 3 where SE(r ) = 1/ n 3. The arctanh is the inverse hyperbolic tangent function. Implemented by R function cor.test(). The above result can also be used to compare two correlations H 0 : ρ 1 = ρ 2 based on two independent data sets.

17 Exploratory Data Analysis (EDA) Coefficient of Correlation Fisher s z Transform: Simulation Study Fisher s z transform helps symmetrize the distribution of r. Each data set of size n = 30 was generated from bivariate normal with true ρ = 0.7 and 100,000 simulation runs. (a) Histogram of r (b) Histgram of Transformed r Density Density r arctanh(r)

18 Exploratory Data Analysis (EDA) Coefficient of Correlation Example: Hypothesis Testing on ρ Consider the smoking vs. infant birth weight example. Want to test H 0 : ρ = 0.5 vs. H a : ρ 0.5. This is equivalent to test H 0 : 1 2 ln 1 + ρ 1 ρ = ( 0.5) ln 2 1 ( 0.5) = The test statistic 1 2 ln 1 + r 1 r 1 2 ln 1 + ρ 0 1 ρ z obs = 0 1/ n 3 = ( ) ( ) 15 3 ln 2 1 ( ) ( ) = RR: reject H 0 if z obs z = 1.96 at significance level α = Conclusion: we cannot reject H 0 since z obs = < 1.96.

19 Exploratory Data Analysis (EDA) Coefficient of Correlation Confidence Interval for ρ Based on Fisher s Z transform, (1 α) 100% confidence interval for ρ can be constructed in two steps: First construct (1 α) 100% confidence interval for ρ = 1 2 ln 1 + ρ 1 ρ. Denote it as (L, U ), i.e., (L, U ) := r ± z 1 α/2 / n 3. Transform back to a (1 α) 100% confidence interval for ρ: ( exp(2l ) 1 (L, U) := exp(2l ) + 1, exp(2u ) ) 1 exp(2u, ) + 1 where the hyperbolic tangent function r = exp(2r ) 1 exp(2r ) + 1 = tanh(r ) is the inverse function for Fisher s z transform.

20 Exploratory Data Analysis (EDA) Coefficient of Correlation Example: CI for ρ First, a 95% CI for ρ in the infant birthweight vs. smoking example is ( ) ln 2 1 ( ) ± 1.96/ 15 3 = ( , ). Transform to a 95% CI for ρ: [ ] exp{2 ( )} 1 exp{2 ( )} + 1, exp{ } 1 exp{ } + 1 = ( 0.733, 0.194). With 95% confidence, we conclude that the true correlation between number of cigars smoker and the infant birth weight is between and R Code: cor.test(x, y, alternative = "two.sided", method = "pearson", conf.level=.95)

21 Model Specification SLR: Model Specification Mathematical modeling of relationships among variables: deterministic vs. probabilistic Simple Linear Model (first-order) y i = β 0 + β 1 x i + ε i with ε i IID N (0, σ 2 ), for i = 1,..., n, where E(y i x i ) = β 0 + β 1 x i is the deterministic component; ε is the random error component; {β0, β 1 } are the regression coefficients; σ 2 is the error variance.

22 Model Specification Model Assumptions Four assumptions are involved in the SLR model: (Linearity): The functional relationship between the (conditional) mean response is linear in the predictor, i.e., µ i E(y i x i ) = β 0 + β 1 x i ; (Independence) ε i s are independent of each other; (Homoscedasticity) ε i s have equal variance σ 2 ; (Normality) ε i s are normally distributed. In short, ε i IID N (0, σ 2 ). It follows that y i x i N ( β 0 + β 1 x i, σ 2).

23 Model Specification Illustration of Statistical Assumptions

24 Model Specification Model Interpretation β 0 is the y-intercept, which is the mean response E(Y ) at X = 0. When β0 = 0, the regression line passes through the origin (0, 0). β 1 is the slope of the regression line, which corresponds to the amount of change in the mean response E(Y X ) with every one-unit increase in X. If β1 > 0, positive association; If β 1 > 0, negative association; What is the change in mean response (or expected change in Y ) with an a-unit increase in X? (Answer: a β 1.)

25 Model Estimation LSE of Betas Model Estimation There are infinitely many choices of {β 0, β 1 }, each uniquely defining a line. Want to identify the best. One criterion is the overall distance between observed y i s and their predicted values ŷ i = β 0 + β 1 x i, as measured with squared difference: Q(β 0, β 1 ) = n {y i (β 0 + β 1 x i )} 2. i=1 The least square line is given by ( ˆβ 0, ˆβ 1 ) such that Q( ˆβ 0, ˆβ 1 ) = min β 0,β 1 Q(β 0, β 1 ).

26 Model Estimation LSE of Betas Least Square Estimator { ˆβ 0, ˆβ 1 } are called the least square estimator (LSE) of {β 0, β 1 }. LSE can be uniquely and explicitly determined by solving the first-order necessary condition Q/ β 0 = 0 and Q/ β 1 = 0: (yi ȳ)(x i x) ˆβ 1 = (xi x) 2 = SS xy SS xx ˆβ 0 = ȳ ˆβ 1 x The resultant least square line is given by y = ˆβ 0 + ˆβ 1 x. Accordingly, the fitted value can be computed ŷ i = ˆβ 0 + ˆβ 1 x i for i = 1,..., n.

27 Model Estimation LSE of Betas BWT Example: LSE For the BWT example, we need SS xy = 50.7, SS xx = , ȳ = 7.7, and x = LSE can be computed accordingly: ˆβ 1 = SS xy SS xx = = ˆβ 0 = ȳ ˆβ 1 x = 7.7 ( ) =

28 Model Estimation LSE of Betas Example: Scatterplot with LS Fitting birthweight number of daily cigas

29 Model Estimation LSE of Betas Properties of LSE Both ˆβ 0 and ˆβ 1 are linear combinations of y i s. To see this, first rewrite ˆβ 1 : (xi x)y i ˆβ 1 = (xi x) = ( ) x i x y i = w i y i, 2 SS xx with w i = (x i x)/ss xx. Using the fact ȳ = (1/n)y i, ˆβ 0 is also a linear combination of y i s. Why? It follows (why?) that [ σ ˆβ 2 ] 1 N β 1, SS xx ( 1n ˆβ 0 N [β 0, σ 2 + x 2 )] SS xx

30 Model Estimation Estimation of Error Variance Estimation of σ 2 Let sum of squared errors (SSE) denote the minimized LS criterion SSE = {y i ( ˆβ 0 + ˆβ 1 x i )} 2 = SS yy ˆβ 1 SS xy (for hand computation) It can be shown that SSE/σ 2 χ 2 (n 2). Details can be found in a course on linear model theories. It follows that E(SSE/σ 2 ) = n 2. Hence an unbiased estimator for σ 2 is given by ˆσ 2 = SSE (yi n 2 = ŷ i ) 2 MSE, n 2 where ŷ i = ˆβ 0 ˆβ 1 x i is the fitted value for x i ; MSE is the short form of Mean Square Error.

31 Model Estimation Estimation of Error Variance Example: Estimation of σ 2 First find SSE = SS yy ˆβ 1 SS xy = ( ) ( 50.7) = An estimate of the constant error variance σ 2 is given ˆσ 2 = SSE (n 2) = (15 2) =

32 Model Estimation Statistical Inference Inference on β 1 Now we know ˆβ 1 N [ β 1, σ 2 SS xx ]. However, it involves the unknown parameter σ 2, besides β 1. How can we get rid of σ 2? This can be solved by forming a t random variable by using the following facts: ˆβ σ2 1 β 1 N (0, 1) /SS xx (n 2)ˆσ2 σ 2 χ 2 (n 2). LSE { ˆβ 0, ˆβ 1 } is independent of SSE (let s assume this). Therefore (why?), t = ˆβ 1 β 1 t ˆσ (n 2). 2 /SS xx

33 Model Estimation Statistical Inference Confidence Intervals It follows (why?) that a (1 α) 100% confidence interval (CI) for β 1 : ˆβ 1 ± t (n 2) ˆσ 2 1 α/2, SS xx where SE( ˆβ 1 ) = ˆσ 2 /SS xx is the standard error of ˆβ 1. Following similar arguments, we can obtain a (1 α) 100% confidence interval (CI) for β 0 : ˆβ 0 ± t (n 2) 1 α/2 ( 1 ˆσ 2 n + x 2 ), SS xx ( ) where SE( ˆβ 0 ) = ˆσ 2 1 n + x2 SS xx is the standard error of ˆβ 0.

34 Model Estimation Statistical Inference Example on CI: The BWT Example A 95% CI for β 1 is given by ˆβ 1 ± t (n 2) ˆσ 2 SS xx = ± = ( , ) Interpretation: With 95% confidence coefficient, we estimate that the mean infant birth weight changes by somewhere between and pounds for each additional cigarette smoked by a pregnant woman. Provide a 95% CI for the change in the mean infant birth weight caused by 10 cigarette increases smoked by a pregnant woman. Hint: Ask for 95% CI for 10β 1, which can be obtained as 10 ( , ) = ( 1.078, 0.244).

35 Model Estimation Statistical Inference Example on CI: The BWT Example A 95% CI for β 0 is given by ˆβ 0 ± t (n 2) = ± = (6.9789, ) ˆσ 2 ( 1 n + x 2 SS xx ) ( ) Interpretation: With 95% confidence coefficient, we conclude that the mean infant birth weight of a non-smoking woman ranges from to pounds. A Word of Caution: Since we don t have data at X = 0, we are not certain whether a linear model is appropriate when extending the scope of the model to X = 0.

36 Model Estimation Statistical Inference Two Standard Compute Outputs Table of Parameter Estimates H 0 : β j = 0 Two-Sided Estimate SE t Test P-Value β < β Analysis of Variance Table (ANOVA) Source df SS MS F P-Value Model Error Total

37 Model Estimation Statistical Inference Residual: Worksheet Cigarettes Birth Weight fitted residual ID Per Day (x i ) Weight (y i ) ŷ i r i sum = 0

38 Model Estimation Statistical Inference Residual Plots Histogram Normal Q Q Plot Frequency Sample Quantiles residuals Theoretical Quantiles

39 Model Estimation Statistical Inference Residual Plots (a) residual vs. fitted r r (b) residual vs. x y^ x

40 Model Estimation Statistical Inference Naive Confidence/Prediction Bands Linear Fit with Naive Confidence/Prediction Bands LS fitted line confidence bands prediction bands birthweight # of cigas Note: The critical value used here is t (n 2). This approach suffers from multiplicity. 1 α/2

41 Model Estimation Statistical Inference Working-Hoteling Confidence Bands Working Hoteling Confidence Bands LS fitted line naive confidence bands Hoteling confidence bands birthweight # of cigas Note: The critical value used in the Working-Hoteling confidence band is W = 2 F (2,n 2) 1 α.

42 Model Estimation Statistical Inference Discussion Thanks! Questions?