Lecture 14 Simple Linear Regression

Size: px

Start display at page:

Download "Lecture 14 Simple Linear Regression"

Dwain Greer
6 years ago
Views:

1 Lecture 4 Simple Linear Regression Ordinary Least Squares (OLS) Consider the following simple linear regression model where, for each unit i, Y i is the dependent variable (response). X i is the independent variable (predictor). Y i α + βx i + ε i ε i is the error between the observed Y i and what the model predicts. This model describes the relation between X i and Y i using an intercept and a slope parameter. The error ε i Y i α βx i is also known as the residual because it measures the amount of variation in Y i not explained by the model. We saw last class that there exists ˆα and ˆβ that minimize the sum of ε 2 i. Specifically, we wish to find ˆα and ˆβ such that n ( Y i (ˆα + ˆβX ) 2 i ) i is the minimum. Using calculus, it turns out that ˆα Ȳ ˆβ X ˆβ (Xi X)(Y i Ȳ ). OLS and Transformation If we center the predictor, Xi X i X, then X i has mean zero. Therefore, ˆα Ȳ ˆβ Xi (Y i Ȳ ) X2 i. By horizontally shifting the value of X i, note that β β, but the intercept changed to the overall average of Y i Consider the linear transformation Z i a + bx i with Z a + b X. Consider the linear model Y i α + β Z i + ε i

2 Let ˆα and ˆβ be the estimates using X as the predictor. Then the intercept and slope using Z as the predictor are ˆβ (Zi Z)(Y i Ȳ ) (Zi Z) 2 (a + bxi a b X)(Y i Ȳ ) (a + b X a b X) 2 (bxi b X)(Y i Ȳ ) (b X b X) 2 b (X i X)(Y i Ȳ ) b 2 ( X X) 2 b ˆβ and ˆα Ȳ ˆβ Z Ȳ ˆβ b (a + b X) Ȳ ˆβ X a ˆβ b ˆα a b ˆβ We can get the same results by substituting X i Z i a b into the linear model. Y i α + βx i + ε i ( ) Zi a Y i α + β + ε i b Y i α aβ b + β b Z i + ε i Y i α + β Z i + ε i Properties of OLS Given the estimates ˆα and ˆβ, we can define () the estimated predicted value Ŷi and (2) the estimated residual ˆε i. Ŷ i ˆα + ˆβX i ˆε i Y i Ŷi Y i ˆα ˆβX i The least squared estimates have the following properties.. i ˆε i 0 n n ˆε i (Y i ˆα ˆβX n i ) Y i nˆα ˆβ i i i i nȳ nˆα n ˆβ X n(ȳ ˆα ˆβ X) n(ȳ (Ȳ ˆβ X) ˆβ X) 0 n X i 2

3 2. X and Ȳ is always on the fitted line. ˆα + ˆβ X (Ȳ ˆβ X) + ˆβ X Ȳ 3. ˆβ rxy s Y, where s Y and s X are the sample standard deviation of X and Y, and r XY is s X the correlation between X and Y. Note that the sample correlation is given by: r XY (Xi X)(Y i Ȳ ) (n )s X s Y. Therefore, ˆβ (Xi X)(Y i Ȳ ) n n s Y s Y Cov(X, Y ) s 2 X Cov(X, Y ) s X s Y r XY s Y s X s Y s Y s X s X s Y s X s X s X 4. ˆβ is a weighted sum of Yi. ˆβ (Xi X)(Y i Ȳ ) (Xi X)Y i (Xi X) 2 (Xi X)Y i (Xi X)Ȳ Ȳ (X i X) (Xi X)Y i 0 ( X i X ) Y i w i Y i, w i X i X 3

4 5. ˆα is a weighted sum of Y i. ˆα Ȳ ˆβ X n Y i X w i Y i v i Y i, v i n X(X i X) 6. X i ˆε i 0. Xi ˆε i (Y i Ŷi)X i (Y i ˆα ˆβX i )X i (Y i (Ȳ ˆβ X) ˆβX i )X i ((Y i Ȳ ) ˆβ(X i X))X i X i (Y i Ȳ ) ˆβ X i (X i X) ˆβ (X i X) 2 ˆβ X i (X i X) ˆβ [ X i (X i X) ˆβ [ X 2 i 2X i X + X2 X 2 i + X i X) ˆβ [ X2 X i X ˆβ X ( X Xi ) 0 Note that ˆβ (Xi X)(Y i Ȳ ) Xi (Y i Ȳ ) 7. Partition of total variation. (Y i Ȳ ) Y i Ȳ + Ŷi Ŷi (Ŷi Ȳ ) + (Y i Ŷi) (Y i Ȳ ): total deviation of the observed Y i from a null model (β 0). (Ŷ Ȳ ): deviation of the modeled value Ŷi (β 0) from a null model (β 0). (Ŷ Y i): deviation of the modeled value from the observed value. Total Variations (Sum of Squares) (Y i Ȳ )2 : total variation: (n ) V (Y ). (Ŷ Ȳ )2 : variation explained by the model 4

5 (Ŷ Y i) 2 : variation not explained by the model; residual sum of squares (SSE), ˆε 2 i Partition of Total Variation (Yi Ȳ )2 (Ŷi Ȳ )2 + (Y i Ŷi) 2 Proof: (Yi Ȳ )2 [ 2 (Ŷi Ȳ ) + (Y i Ŷi) (Ŷi Ȳ )2 + (Y i Ŷi) (Ŷi Ȳ )(Y i Ŷi) (Ŷi Ȳ )2 + (Y i Ŷi) 2 where, ( Ŷ i Ȳ )(Y i Ŷi) [ (ˆα + ˆβX i ) (ˆα + [ ˆβ(Xi X) ˆε i ˆβ X) ˆε i ˆβ X i ˆε i ˆβ X ˆε i Alternative formula for ˆε 2 i (Yi Ŷi) 2 (Y i Ȳ )2 (Ŷi Ȳ )2 (Y i Ȳ )2 (ˆα + ˆβX i Ȳ )2 (Y i Ȳ )2 (Ȳ + βx X + ˆβX i Ȳ )2 (Y i Ȳ )2 [ ˆβ(Xi X) (Y i Ȳ )2 ˆβ 2 (X i X) 2 or (Yi Ŷi) 2 [ (Y i Ȳ (Xi )2 X)(Y i Ȳ ) Variation Explained by the Model: 5

6 An easy way to calculate (Ŷi Ȳ )2 is ( Ŷ i Ȳ )2 (Y i Ȳ )2 (Y i Ŷi) 2 [ (Y i Ȳ (Xi )2 X)(Y i Ȳ ) 2 [ (Xi X)(Y i Ȳ ) 2 0. Coefficient of Determination r 2 : r 2 variation explained by the model total variation ( Ŷ i Ȳ )2 (Yi Ȳ )2 (a) r 2 is related the residual variance. (Yi Ŷi) 2 (Y i Ȳ )2 (Ŷi Ȳ )2 [ (Yi Ȳ )2 (Ŷi Ȳ )2 (Yi Ȳ )2 (Yi Ȳ )2 ( r 2 ) (Y i Ȳ )2 Diving the right-hand side by n 2 and the left-hand side by n, for large n s 2 ( r 2 )Var(Y ) (b) r 2 is the square of the sample correlation between X and Y. ( r 2 Ŷ i Ȳ )2 (Yi Ȳ )2 [ (Xi X)(Y i Ȳ ) 2 (Yi Ȳ )2 (X i X) 2 [ (Xi X)(Y i Ȳ ) (Yi Ȳ )2 2 r 2 XY Statistical Inference for OLS Estimates Parameters ˆα and ˆβ can be estimated for any given sample of data. Therefore, we also need to consider their sampling distributions because each sample of (X i, Y i ) pairs will result in different estimates of α and β. 6

7 Consider the following distribution assumption on the error, Y i α + βx i + ε i ε i iid N (0, σ 2 ). The above is now a statistical model that describes the distribution of Y i given X i. Specifically, we assume the observed Y i is error-prone but centered around the linear model for each value of X i. Y i iid N (α + βx i, σ 2 ) The error distribution results in the following assumptions.. Homoscedasticity: variance of Y i is the same for any X i. V (Y i ) V (α + βx i + ε i ) V (ε i ) σ 2 2. Linearity: the mean of Y i is linear with respect to X i E(Y i ) E(α + βx i + ε i ) α + βx i + E(ε i ) α + βx i 3. Independence: Y i are independent of each other. Cov(Y i, Y j ) Cov(α + βx i + ε i, α + βx j + ε j ) Cov(ε i, ε j ) 0 We can now investigate the bias and variance of OLS estimators ˆα and ˆβ. We assume that X i are known constants. ˆβ is unbiased. E[ ˆβ [ E wi Y i, w i X i X w i E[Y i w i (α + βx i ) α w i + β w i X i β 7

8 Because wi wi X i (Xi X) 0 Xi (X i X) Xi (X i X) X(Xi X) Xi (X i X) Xi (X i X) X (X i X) Xi (X i X) Xi (X i X) 0 Xi (X i X) ˆα is unbiased. E[ˆα E[Ȳ ˆβ X E[Ȳ XE[ ˆβ [ E Yi β n X E [α + βxi β n X n (nα + nβx i) β X α + β X β X α ˆβ σ 2 has variance n i (X i X ) 2 σ 2 (n )s 2, where σ 2 is the residual variance and s 2 x is the x sample variance of x,..., x n. where V [ ˆβ [ V wi Y i w 2 i w 2 i V [Y i σ 2 w 2 i, w i X i X [ (X i X) (Xi 2 2 X) 2 8

9 ( ) ˆα has variance σ 2 n + X2 n i (X i X ) 2 V [ˆα V [ vi Y i, v i n X(X i X) v 2 i V [Y i σ ( 2 n X(X i X) ) 2 σ 2 [ n 2 X 2 (X i X) 2 [ (X i X) X(X i X) n (X i X) 2 σ 2 [ n X2 [ (X i X) 2 2 ( ) σ 2 n + X2 n i (X i X ) 2 2 X n (X i X) 2 (Xi X) ˆα and ˆβ are negatively correlated. Cov(ˆα, ˆβ) ( Cov vi Y i, ) w i Y i v i w i Cov (Y i, Y i ) v i w i σ 2 σ ( 2 n [ ( σ 2, v i n X(X i X), w i X(X i X) ) ( Xi X ) X i X ) n (X i X) 2 X σ 2 [ (Xi X) n (X i X) 2 σ 2 [ 0 X (X i X) 2 [ (X i X) 2 2 σ 2 X [ (X i X) 2 2 X(X i X) X i X X i X Therefore, the sampling distributions for the regression parameters are: ˆα N [ ) n (α, σ 2 + X2 n i (X i X ) 2 ˆβ N ( β, σ 2 ) n i (X i X ) 2 9

10 However, in most cases, we don t know what σ 2 and will need to estimate it from the data. One unbiased estimator for σ 2 is obtained from the residuals. s 2 n 2 N (Y i Ŷi) 2 By replacing σ 2 with s 2, we have the following sampling distributions for ˆα and ˆβ. ( [ ) ˆα t n 2 α, s 2 n + X2 s n i (X i X ˆβ ) 2 t n 2 (β, 2 ) n i (X i X ) 2 i Note that our degrees of freedom is n 2 because we estimated two parameters (α and β) in order to calculate the residuals. Similarly recall that (Ŷi Y i ) 0. Knowing that ˆα and ˆβ follow a t-distribution with degrees of freedom n-2, we are able to construct confidence interval and carry out hypothesis test as before. For example, a 90% confidence interval for β is given by ˆβ ± t 5%,n 2 s 2 n i (X i X ) 2. Similarly, consider testing the hypotheses H 0 β β 0 versus H A β β 0. The corresponding t-test statistic is given by ˆβ β 0 s 2 n i (X i X ) 2 t n 2. Prediction Based on the sampling distribution of ˆα and ˆβ we can also investigate the uncertainty associated with Ŷ0 ˆα + ˆβX 0 on the fitted line for any X 0. Specifically, [ n Ŷ 0 N (α + X 0 β, σ 2 + (X 0 X) 2 ). Note that α + X 0 β is the mean of Y given X 0. The textbook uses the notation µ 0. Replacing the unknown with our estimates, we have ( [ Ŷ 0 t n 2 ˆα + X 0 ˆβ, s 2 n + (X 0 X) 2 ). The fitted value Ŷ0 ˆα + ˆβX 0 has mean α + X 0 β and variance σ 2 [ n + (X 0 X) 2. E(Ŷ0) E(ˆα + ˆβX 0 ) E(ˆα) + X 0 E( ˆβ) α + X 0 β 0

11 V (Ŷ0) V (ˆα + ˆβX 0 ) V (ˆα) + X 2 0V ( ˆβ) + X 0 Cov(ˆα, ˆβ) ( ) σ 2 n + X2 n i (X i X ) 2 + σ 2 X0 2 n i (X i X ) 2 X 0 X 2σ2 [ σ 2 n + (X 0 X) 2 We can also consider the uncertainty of a new observation Y0 at X 0. In other words, what values of Y0 do I expect to see at X 0? Note that this is different from Ŷ0 which is the fitted value that represents the mean of Y at X 0. Since Y 0 α + βx 0 + ε, ε N (0, σ 2 ), by replacing the unknown parameters with our estimates, we obtain ( [ Y0 t n 2 ˆα + ˆβX 0, s 2 + s 2 n + (X 0 X) 2 ) Again, the above distribution describes the actual observable data, instead of model parameters; interval estimates are also known as prediction intervals, instead of confidence intervals..

Scatter plot of data from the study. Linear Regression

1 2 Linear Regression Scatter plot of data from the study. Consider a study to relate birthweight to the estriol level of pregnant women. The data is below. i Weight (g / 100) i Weight (g / 100) 1 7 25