Weighted Least Squares

Transcription

1 Weighted Least Squares ST 430/514 Recall the linear regression equation E(Y ) = β 0 + β 1 x 1 + β 2 x β k x k We have estimated the parameters β 0, β 1, β 2,..., β k by minimizing the sum of squared residuals SSE = = n (y i ŷ i ) 2 i=1 n i=1 [ y i ( ˆβ 0 + ˆβ 1 x i,1 + ˆβ 2 x i,2 + + ˆβ k x i,k )] 2. 1 / 11 Special Topics Weighted Least Squares

2 Sometimes we want to give some observations more weight than others. We achieve this by minimizing a weighted sum of squares: WSSE = = n w i (y i ŷ i ) 2 i=1 n ( )] 2 w i [y i ˆβ 0 + ˆβ 1 x i,1 + ˆβ 2 x i,2 + + ˆβ k x i,k i=1 The resulting ˆβs are called weighted least squares (WLS) estimates, and the WLS residuals are wi (y i ŷ i ). 2 / 11 Special Topics Weighted Least Squares

3 Why use weights? Suppose that the variance is not constant: var(y i ) = σ 2 i. If we use weights w i 1 σ 2 i, the WLS estimates have smaller standard errors than the ordinary least squares (OLS) estimates. That is, the OLS estimates are inefficient, relative to the WLS estimates. 3 / 11 Special Topics Weighted Least Squares

4 In fact, using weights proportional to 1/σi 2 weights give smaller standard errors. is optimal: no other When you specify weights, regression software calculates standard errors on the assumption that they are proportional to 1/σ 2 i. 4 / 11 Special Topics Weighted Least Squares

5 How to choose the weights If you have many replicates for each unique combination of xs, use s 2 i to estimate var(y x i ). Often you will not have enough replicates to give good variance estimates. The text suggests grouping observations that are nearest neighbors. Alternatively you can use the regression diagnostic plots. 5 / 11 Special Topics Weighted Least Squares

6 Example: Florida road contracts. dot11 <- read.table("text/exercises&examples/dot11.txt", header = TRUE) l1 <- lm(bidprice ~ LENGTH, dot11) summary(l1) plot(l1) 6 / 11 Special Topics Weighted Least Squares

7 The first plot uses unweighted residuals y i ŷ i, but the others use weighted residuals. Also recall that they are Standardized residuals z i = y i ŷ i s 1 h i. which are called Studentized residuals in the text. With weights, the standardized residuals are z i = ( ) yi ŷ i w i s. 1 h i 7 / 11 Special Topics Weighted Least Squares

8 Note that the Scale-Location plot shows an increasing trend. Try weights that are proportional to powers of x = LENGTH: # Try power -1: plot(lm(bidprice ~ LENGTH, dot11, weights = 1/LENGTH)) # Still slightly increasing; try power -2: plot(lm(bidprice ~ LENGTH, dot11, weights = 1/LENGTH^2)) # Now slightly decreasing. summary() shows that the fitted equations are all very similar. weights = 1/LENGTH gives the smallest standard errors. 8 / 11 Special Topics Weighted Least Squares

9 Often the weights are determined by fitted values, not by the independent variable: # Try power -1: plot(lm(bidprice ~ LENGTH, dot11, weights = 1/fitted(l1))) # About flat; but try power -2: plot(lm(bidprice ~ LENGTH, dot11, weights = 1/fitted(l1)^2)) # Now definitely decreasing. summary() shows that the fitted equations are again very similar. weights = 1/fitted(l1) gives the smallest standard errors. 9 / 11 Special Topics Weighted Least Squares

10 Note Standard errors are computed as if the weights are known constants. In the last case, we used weights based on a preliminary OLS fit. Theory shows that in large samples the standard errors are also valid with estimated weights. 10 / 11 Special Topics Weighted Least Squares

11 Note When you specify weights w i, lm() fits the model σ 2 i = σ2 w i and the Residual standard error s is an estimate of σ: s 2 = n i=1 w i (y i ŷ i ) 2 n p If you change the weights, the meaning of σ (and s) changes. You cannot compare the residual standard errors for different weighting schemes (c.f. page 488, foot). 11 / 11 Special Topics Weighted Least Squares