Outline. Review regression diagnostics Remedial measures Weighted regression Ridge regression Robust regression Bootstrapping

Transcription

1 Topic 19: Remedies

2 Outline Review regression diagnostics Remedial measures Weighted regression Ridge regression Robust regression Bootstrapping

3 Regression Diagnostics Summary Check normality of the residuals with a normal quantile plot or histogram Plot the residuals versus predicted values, versus each of the X s and (when appropriate) versus time/space Examine the partial regression plots Use the graphics smoother to see if there appears to be a curvilinear pattern

4 Regression Diagnostics Summary Examine the studentized deleted residuals (RSTUDENT in the output) The hat matrix diagonals Dffits, Cook s D, and the DFBETAS Check observations that are extreme on these measures relative to the other observations

5 Regression Diagnostics Summary Examine the tolerance for each X If there are variables with low tolerance, you need to do some model building Recode variables Variable selection

6 Remedial measures Weighted least squares Ridge regression Robust regression Nonparametric regression Bootstrapping

7 i Maximum Likelihood ( β + β σ 2 ) Y~ N X, f i 0 1 i = 2 1 πσ e 1 2 L= f f f Y 1 2 n 0 1 β β X i 0 1 i 2 (likelihood function) Find β and β which maximizes σ L

8 i Maximum Likelihood What is Y have different (but known) variance? i ( β + β σ 2 ) i Y~ N X, f i 0 1 i = 1 2πσ 1 Y n i e L= f f f 0 1 β β X i 0 1 i 2 (likelihood function) Find β and β which maximizes σ i L

9 Weighted regression Maximization of L with respect to β s is equivalent to minimization of σ 1 Y X X 2 i ( ) 2 β β β p ip i 0 1 i,1 1, 1 Weight of each case is w i =1/σ i 2

10 Weighted least squares Least squares problem is to minimize the sum of w i times the squared residual for case i Computations are easy use the weight statement in proc reg b w = (X WX) -1 (X WY) where W is a diagonal matrix of the weights The problem in practice now becomes determining the weights

11 Determination of weights Find a relationship between the absolute residual and another variable and use this as a model for the standard deviation Similar approach using the squared residual to model the variance Or use grouped data or approximately grouped data to estimate the variance for all cases in the group

12 Determination of weights With a model for the standard deviation or the variance, we can approximate the optimal weights Optimal weights are proportional to the inverse of the variance

13 KNNL Example KNNL p 427 Y is diastolic blood pressure X is age n = 54 healthy adult women aged 20 to 60 years old

14 Get the data and check it data a1; infile../data/ch11ta01.txt'; input age diast; proc print data=a1; run;

15 Plot the relationship symbol1 v=circle i=sm70; proc gplot data=a1; plot diast*age / frame; run;

16 Diastolic bp vs age Strong linear relationship, no skewness but nonconstant variance

17 Run the regression proc reg data=a1; model diast=age; output out=a2 r=resid; run;

18 Regression output Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total Root MSE R-Square Dependent Mean Adj R-Sq Coeff Var

19 Regression output Parameter Estimates Variable DF Parameter Estimate Standard Error t Value Pr > t Intercept <.0001 age <.0001 Estimators still unbiased but no longer have minimum variance Prediction interval coverage often lower or higher than 95%

20

21

22 Use the output data set a2 to get the absolute and squared residuals data a2; set a2; absr=abs(resid); sqrr=resid*resid;

23 Generate plots with a smooth proc gplot data=a2; plot (resid absr sqrr)*age; run;

24 Absolute value of the absr 20 residuals vs age age

25 Squared residuals vs age

26 Model the std dev vs age (absolute value of the residual) proc reg data=a2; model absr=age; output out=a3 p=shat; Note that a3 has the predicted standard deviations (shat)

27 Compute the weights data a3; set a3; wt=1/(shat*shat);

28 Regression with weights proc reg data=a3; model diast=age / clb; weight wt; run;

29 Output Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total Root MSE R-Square Dependent Mean Adj R-Sq Coeff Var

30 Output Parameter Estimates Parameter Standard 95% Confidence Variable DF Estimate Error t Value Pr > t Limits Intercept < age < Reduction in std err of the age coeff

31

32

33 Ridge regression If (X X) is difficult to invert (near singular) then approximate by inverting (X X+kI). Estimators of coefficients are now biased but more stable. For some value of k, ridge regression estimator has a smaller mean square error than ordinary least square estimator. Can be used to reduce number of predictors Ridge = k is an option for model statement. Cross-validation / ridge plots used to determine k

34 Ridge Regression Can express ridge constraint in terms of finding b to minimize 2 ( Y Zβ) ( Y Zβ) + λ β j where Z in standardized X Note: LASSO is a variation of this approach in which you minimize ( Y Zβ) ( Y Zβ) s.t. β j t

35 Ridge Regression in SAS proc reg data=a1 outest=b; run; model fat=skinfold thigh midarm / ridge=0 to.1 by.001;

36

37 Robust regression Basic idea is to have a procedure that is not sensitive to outliers Alternatives to least squares, minimize sum of absolute values of residuals median of the squares of residuals Do weighted regression with weights based on residuals, and iterate

38 Nonparametric Several versions regression We have used i=sm70 Interesting theory All versions have some smoothing or penalty parameter similar to the 70 in i=sm70

39 Regression trees Standard approach in area of data mining replacing multiple regression Basically partition the X space into rectangles Repeatedly split data two nodes based on a single predictor Predicted value is mean of responses in rectangle

40 Bootstrap Very important theoretical development that has had a major impact on applied statistics Uses resampling to approximate the sampling distribution Sample with replacement from the data or residuals and repeatedly refit model to get the distribution of the quantity of interest

41 Background Reading We used programs topic19.sas This completes Chapter 11