Chapter 11: Robust & Quantile regression

Transcription

1 Chapter 11: Robust & Adapted from Timothy Hanson Department of Statistics, University of South Carolina Stat 704: Data Analysis I 1 / 13

2 11.3: Robust regression Leverages h ii and deleted residuals t i are useful for finding outlying x i and Y i (w.r.t. the model) cases. Cook s D i and DFFIT i indicate which cases influence the fit of the model, i.e. the OLS b. What to do with influential and/or outlying cases? Are they transcription errors or somehow unrepresentative of the target population? Outliers are often interesting in their own right and can help guide the building of a better model. Robust regression dampens the effect of outlying cases on estimation to provide a better fit to the majority of cases. Useful in situations when there s no time for influence diagnostics or a more careful analysis. 2 / 13

3 Robust regression is effective when the error distribution is not normal, but heavy-tailed. M-estimation is a general class of estimation methods. Choose β to minimize Q(β) = n ρ(y i x iβ), i=1 where ρ( ) is some function. ρ(u) = u 2 gives OLS b. ρ(u) = u gives L 1 regression, or least absolute residual (LAR) regression. 3 / 13

4 Huber s method builds a ρ( ) that is a compromise between OLS and LAR. It looks like u 2 for u close to zero and u further away. { ρ(u) = u 2 if u (2 u 1.345) if u > }. 4 / 13

5 Iteratively Reweighted Least Squares Outlying values of r j i = y i x i bj are (iteratively) given less weight in the estimation process. 0 (Starting b 0 ) OLS: w 0 i = 1/e 2 i from e = Y Xb. Set j = 1. 1 (WLS for b j using W j 1 ): b j = (X W j 1 X) 1 X W j 1 Y. 2 ˆσ j = median{ Y i x i bj /Φ 1 (0.75) : i = 1,..., n}. ( 3 w j Yi x i = W i ), bj where ˆσ j Set j to j + 1. { 1 if u W (u) = u if u > }. 1 Repeat steps 1 through 3 until ˆσ j and b j stabilize. There are other weight functions (SAS default is bisquare, pp ) and other methods for updating ˆσ j see book and SAS documentation if interested. 5 / 13

6 SAS code: M-estimation w/ Huber weight data prof; input state$ mathprof parents homelib reading tvwatch absences; datalines; Alabama Arizona et cetera... Wisconsin Wyoming ; proc robustreg data=prof method=m (wf=huber); model mathprof=parents homelib reading tvwatch absences; run; proc robustreg data=prof method=m (wf=huber); model mathprof=parents homelib reading tvwatch absences; test absences reading; run; proc robustreg data=prof method=m (wf=huber); model mathprof=parents homelib tvwatch ; id state; output out=out p=p sr=sr; run; 6 / 13

7 SAS output... The ROBUSTREG Procedure Parameter Estimates Standard 95% Confidence Chi- Parameter DF Estimate Error Limits Square Pr > ChiSq Intercept <.0001 parents homelib <.0001 reading tvwatch absences Robust Linear Tests Test Test Chi- Test Statistic Lambda DF Square Pr > ChiSq Rho Rn / 13

8 Quantile Regression Another option is general quantile regression. L 1 regression (p. 438), described above, is median regression and can be carried out in SAS PROC QUANTREG. Other quantiles can similarly be regressed upon. Robust quantile regression is particularly helpful for modeling responses with non-constant error variance. QUANTREG solves minimization problem using simplex algorithm, details in documentation. 8 / 13

9 Median, or L 1 regression minimizes Q 0.5 (β) = min β R p n Y i x iβ. Let 0 < τ < 1 be a probability. Koenker and Bassett (1978) define b τ, the τth quantile regression effects vector, to minimize Q τ (β) = min τ Y i x iβ + (1 τ) Y i x iβ. β R p i:y i x i β i:y i <x i β Let Y h be a draw from the population with accompanying x h. The b τ that minimizes Q τ (β) satisfies i=1 P(Y h x h b τ ) τ. Note q τ (x) x b τ. How are the elements of b τ interpreted? 9 / 13

10 SAS code: Fall 2008 GPA data proc quantreg data=fall08; model cltotgpa=satv / quantile=0.25; output out=o1 p=p1; proc quantreg data=fall08; model cltotgpa=satv / quantile=0.50; output out=o2 p=p2; proc quantreg data=fall08; model cltotgpa=satv / quantile=0.75; output out=o3 p=p3; data o4; set o1 o2 o3; proc sort data=o4; by quantile satv; 10 / 13

11 SAS output Why is the slope for regression on q.75 not increasing? The QUANTREG Procedure Quantile % Confidence Parameter DF Estimate Limits Intercept satv Quantile % Confidence Parameter DF Estimate Limits Intercept satv Quantile % Confidence Parameter DF Estimate Limits Intercept satv / 13

12 Normal-errors regression is also quantile regression For our standard model: Y = x β + ɛ, ɛ N(0, σ 2 ), note that P(Y q τ ) = τ P(x β + ɛ q τ ) = τ ( ɛ P σ q τ x ) β = τ σ ( qτ x ) β Φ = τ σ q τ x β σ = Φ 1 (τ) q τ (x) = Φ 1 (τ)σ + x β. 12 / 13

13 When x j increases by one, every quantile increases by β j. What does this imply about the quantile functions? Could we see a plot like we saw for the GPA data a few slides ago? Final comment: L 1 regression is obtained as the MLE from a standard regression model assuming the errors are distributed double-exponential (Laplace). 13 / 13