36-707: Regression Analysis Homework Solutions. Homework 3

Transcription

1 36-707: Regression Analysis Homework Solutions Homework 3 Fall 2012 Problem 1 Y i = βx i + ɛ i, i {1, 2,..., n}. (a) Find the LS estimator of β: RSS = Σ n i=1(y i βx i ) 2 RSS β = Σ n i=1( 2X i )(Y i βx i ) 0 = Σ n i=1x i Y i βσ n i=1x 2 i ˆβ = Σn i=1x i Y i 1

2 (b) Find an expression for the hat matrix: In the case where there is no intercept, X is an n 1 matrix. X T = (X 1, X 2,..., X n ). Then H = X(X T X) 1 X T = X(Σ n i=1x 2 i ) 1 X T = XXT (c) Describe the column space: X 1 X 2 X =. So the column space of X is spanned by one n-dimensional vector. In this case, it. X n is a line in R n passing through 0 and (X 1, X 2,..., X n ). 2

3 (d) Find the mean and variance of ˆβ assuming the model is correct: E[ ˆβ X] = E[ Σn i=1x i Y i ] = Σn i=1x i E[Y i X] = Σn i=1x i βx i = β V ar[ ˆβ X] = V ar[ Σn i=1x i Y i ] = Σn i=1xi 2 V ar(y i X) ( )2 = Σn i=1x 2 i σ 2 ( )2 = σ2 Problem 2. Weisberg 3.5, parts 1 and Because all plots in the scatterplot matrix appear to have a strong positive correlation, we would expect all numbers in the correlation matrix to be close to 1. Below is the correlation matrix for BSAAM, OPBPC,OPRC,and OPSLAKE respectively. As expected, the numbers are all close to 1. [,1] [,2] [,3] [,4] [1,] [2,] [3,] [4,] The regression summary is as follows: 3

4 BSAAM OPBPC OPRC OPSLAKE Figure 1: Scatterplot matrix of OPBPC + OPRC + OPSLAKE. Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-07 *** OPBPC OPRC ** OPSLAKE ** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: 8304 on 39 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 3 and 39 DF, p-value: < 2.2e-16 The t-value column are t-statistics that correspond to the slope parameter = 0 vs 0, given the presence of all other variables in the model. A high t-value will result in a low p-value. We can reject the null hypothesis that the coefficient for that parameter = 0 when the t-value is high. In the context of this problem, we see that the intercept, OPRC, and OPSLAKE have coefficients 0. However, it is possible that the coefficient for OPBPC = 0 because of the presence of OPRC and OPSLAKE, which is a result of the high degree of correlation between variables. 4

5 3.5.2 H 0 : BSAAM is independent of OPBPC,OPRC, and OPSLAKE H 1 : BSAAM is dependent on at least one of OPBPC,OPRC, or OPSLAKE As we see above, the F-statistics = with 3 and 39 DF for a p-value < 2.2e 16. This means that at least one of the coefficients is significant in predicting BSAAM. Therefore, we can reject independence and say that BSAAM is dependent on at least one of the variables OPBPC,OPRC, or OPSLAKE > m1 = lm(bsaam OPBPC + OPRC + OPSLAKE, data=water) > anova(m1) Analysis of Variance Table Response: BSAAM Df Sum Sq Mean Sq F value Pr(>F) OPBPC e e < 2.2e-16 *** OPRC e e e-07 *** OPSLAKE e e ** Residuals e e+07 > m2 = lm(bsaam OPBPC + OPSLAKE + OPRC, data=water) > anova(m2) Analysis of Variance Table Response: BSAAM Df Sum Sq Mean Sq F value Pr(>F) OPBPC e e < 2.2e-16 *** OPSLAKE e e e-07 *** OPRC e e ** Residuals e e+07 > m3 = lm(bsaam OPSLAKE + OPRC + OPBPC, data=water) > anova(m3) Analysis of Variance Table Response: BSAAM Df Sum Sq Mean Sq F value Pr(>F) OPSLAKE e e < 2.2e-16 *** OPRC e e ** OPBPC e e

6 Residuals e e+07 The most obvious difference is that OPBPC is unimportant when the regression is adjusted for other predictors, but important when the regression ignores other predictors. F-tests for the last term in each Anova table are equivalent to the t-tests in the regression output > m = lm(bsaam OPSLAKE, data= water) > anova(m, m3) Analysis of Variance Table Model 1: BSAAM OPSLAKE Model 2: BSAAM OPSLAKE + OPRC + OPBPC Res.Df RSS Df Sum of Sq F Pr(>F) * Since the p-value 0.02 is very small, we will reject the null hypothesis that the coefficients for OPRC and OPBPC are both zero. Problem 3 Suppose there are n samples, then F = (RSS small RSS big )/(df small df big ) RSS big /df big = { n k=1 (Y k X k ) 2 n k=1 [Y k ( ˆβ 0 + ˆβ 1 X k )] 2 } /2 n k=1 [Y k ( ˆβ 0 + ˆβ 1 X k )] 2 /(n 2) Problem 4. Weisberg See Figure [2] for scatterplot matrix. A linear regression model seems appropriate, even though the variance is clearly larger for larger flocks than that for the smaller ones. 6

7 Figure 2: The variables are all positively correlated and strongly. The error term for the simple regression model of Photo on Obs1 measures the variation in the aerial counts by Observer 1. Because photo counts are deterministic with no room for error, regressing Obs1 on Photo is not appropriate The null hypothesis is that the observers estimates are on average consistent with the actual number of geese. There are many ways to define reliability of human estimates. For example, we can 1 n Y i define it as the average amount of normalized deviation: Ŷi. In the context of this n i=1 Y i problem, we may say an observer is reliable if the observer s estimates are on average the same as the actual number of geese. > Y = photo > X = obs1 > fullmodel = lm(y X) > RSSnull = sum((y - X)**2) > DFnull = n > RSSalt = sum(fullmodel$resid**2) > DFalt = n-2 > Fstat = ( (RSSnull-RSSalt)/(DFnull-DFalt) )/(RSSalt/DFalt) 7

8 > pf(fstat,dfnull-dfalt,dfalt,lower.tail=false) [1] So we reject the null hypothesis > Y = sqrt(photo) > X = sqrt(obs1) > fullmodel = lm(y X) > RSSnull = sum((y - X)**2) > DFnull = n > RSSalt = sum(fullmodel$resid**2) > DFalt = n-2 > Fstat = ( (RSSnull-RSSalt)/(DFnull-DFalt) )/(RSSalt/DFalt) > pf(fstat,dfnull-dfalt,dfalt,lower.tail=false) [1] Again, we will reject the null hypothesis > Y = photo > X = obs1 > wgts = 1/X > fullmodel = lm(y X,weights=wgts) > RSSnull = sum( ( (Y - X)**2 )*wgts ) > DFnull = n > RSSalt = sum(fullmodel$resid**2*wgts) > DFalt = n-2 > Fstat = ( (RSSnull-RSSalt)/(DFnull-DFalt) )/(RSSalt/DFalt) > pf(fstat,dfnull-dfalt,dfalt,lower.tail=false) [1] Yet again, we will reject the null hypothesis > # Regression of Y on obs1 > Y = photo > avg = (obs1 + obs2)/2 > dif = (obs1 - obs2) 8

9 Residuals versus fitted values (Q5.5.2) resid(fullmodel) fitted(fullmodel) Figure 3: Comparison of residual plots of weighted and unweighted models Weighted residuals versus fitted values (Q5.5.4) resid(fullmodel) * wgts fitted(fullmodel) Figure 4: Comparison of residual plots of weighted and unweighted models > wgts = 1/obs1 > fullmodel = lm(y avg+dif,weights=wgts) > redmodel = lm(y obs1,weights=wgts) > RSSnull = sum( ( redmodel$resid**2 )*wgts ) > DFnull = n-1 > RSSalt = sum(fullmodel$resid**2*wgts) > DFalt = n-3 9

10 > Fstat = ( (RSSnull-RSSalt)/(DFnull-DFalt) )/(RSSalt/DFalt) > pf(fstat,dfnull-dfalt,dfalt,lower.tail=false) [1] e-05 > > # Regression of Y on obs2 > Y = photo > avg = (obs1 + obs2)/2 > dif = (obs1 - obs2) > wgts = 1/obs2 > fullmodel = lm(y avg+dif,weights=wgts) > redmodel = lm(y obs2,weights=wgts) > RSSnull = sum( ( redmodel$resid**2 )*wgts ) > DFnull = n-1 > RSSalt = sum(fullmodel$resid**2*wgts) > DFalt = n-3 > Fstat = ( (RSSnull-RSSalt)/(DFnull-DFalt) )/(RSSalt/DFalt) > pf(fstat,dfnull-dfalt,dfalt,lower.tail=false) [1] The choice of the variance of the observations is not definite; we chose to use whichever covariate we are making the comparison to. The results show that the combination of obs1 and obs2 has more information than obs1 alone, but we cannot reject the hypothesis that the actual number of geese could be predicted using obs2 only. In conclusion, human observers are not sufficient to replace photos. 10