Topic 18: Model Selection and Diagnostics

Transcription

1 Topic 18: Model Selection and Diagnostics

2 Variable Selection We want to choose a best model that is a subset of the available explanatory variables Two separate problems 1. How many explanatory variables should we use (i.e., subset size) 2. Given the subset size, which variables should we choose

3 KNNL Example Page 350, Section 9.2 n = 54 patients / cases Y : survival time (liver operation) X s (explanatory variables) are Blood clotting score Prognostic index Enzyme function test Liver function test

4 KNNL Example cont. We start with the usual plots and descriptive statistics Note that time-to-event / survival data are often heavily skewed and typically transformed with a log prior to model fitting

5 Ln Transform of Y Recall that regression model requires Y X to be Normally distributed, not Y Better to look at residuals With data like these, transform reduces influence of long right tail and stabilizes the variance of the residuals

6 Tab delimited Data Data a1; infile 'U:\.www\datasets512\CH09TA01.txt delimiter='09'x; input blood prog enz liver age gender alcmod alcheavy surv lsurv; run; Dummy variables for alcohol use Ln(surv)

7 Data Obs blood prog enz liver age Gender alcmod alcheavy surv lsurv

8

9 Long right tail

10

11

12 Generate scatterplots proc corr plot=matrix; var blood prog enz liver; run; proc corr plot=scatter; var blood prog enz liver; with lsurv; run;

13

14

15

16

17

18 Correlation Summary Pearson Correlation Coefficients, N = 54 Prob > r under H0: Rho=0 blood prog enz liver lsurv < <.0001

19 The Two Problems in Variable Selection 1. To determine an appropriate subset size Might use adjusted R 2, C p, MSE, PRESS, AIC, SBC (BIC) 2. To determine best model of a fixed size Might use R 2

20 Adjusted R 2 R 2 by its construction is guaranteed to increase with p SSE cannot decrease with additional X and SSTO constant Adjusted R 2 uses df to account for changes in p R n 1 SSE MSE 2 = 1 p 1 p = n p SSTO MSTO a

21 Adjusted R 2 Want to find model that maximizes Since MSTO remains constant for a given data set, equivalent to finding model that minimizes MSE Details on pages R a

22 C p Criterion The basic idea is to compare subset models with a full model A subset model is good if there is not substantial bias in the predicted values relative to the full model Looks at the ratio of total mean squared error and the true error variance See page for details

23 C p Criterion C p = SSE p MSE(Full ) ( n 2 p) SSE based on a specific choice of p-1 variables MSE(full) based on all the variables Consider full set C p =(n-p)-(n-2p)=p

24 Use of C p p is the number of regression coefficients including the intercept A model is good according to this criterion if C p p Rule: Pick the smallest model for which C p is smaller than p or pick the model that minimizes C p, provided the C p is not much larger than p

25 SBC (BIC) and AIC Criterion based on log(likelihood) plus a penalty for more complexity AIC minimize SBC minimize SSE p n log + 2p n SSE p n log + p log(n) n

26 Other approaches PRESS (prediction SS) For each case i, delete the case and predict Y using the fitted model based on the other n-1 cases Look at the SS for observed minus predicted Want to minimize the PRESS Appears this requires n regressions but not the case

27 Variable Selection in SAS Additional proc reg model statement options useful in variable selection INCLUDE=n forces the first n explanatory variables into all models BEST=n limits the output to the best n models of each subset size or total START=n limits output to models that include at least n explanatory variables

28 Variable Selection Step-type procedures Forward selection (Step up) Backward elimination (Step down) Stepwise (forward selection with a backward glance) Very popular but now have much better search techniques like BEST

29 2. Ordering models of the same subset size Use R 2 or SSE / MSE or F* This approach can lead us to consider several models that give us approximately the same predicted values May need to apply knowledge of the subject matter to make a final selection Not that important if prediction is the key goal

30 Proc Reg Code proc reg data=a1; model lsurv= blood prog enz liver/ selection=rsquare cp aic sbc b best=3; run;

31 Selection Results Number in Model R-Square C(p) AIC SBC

32 Number in Model Selection Results Parameter Estimates Intercept blood prog enz liver

33 Proc Reg Code proc reg data=a1; model lsurv= blood prog enz liver/ selection=cp aic sbc b best=3; run;

34 Selection Results Number in Model C(p) R-Square AIC SBC WARNING: selection=cp just lists the models in order based on lowest C(p), regardless of whether it is good or not

35 How to Choose with C(p) 1. Want small C(p) 2. Want C(p) near p In original paper, it was suggested to plot C(p) versus p and consider the smallest model that satisfies these criteria Can be somewhat subjective when determining near

36 Proc Reg Creates data set with estimates & criteria proc reg data=a1 outest=b1; model lsurv=blood prog enz liver/ selection=rsquare cp aic sbc b; run;quit; symbol1 v=circle i=none; symbol2 v=none i=join; proc gplot data=b1; plot _Cp_*_P P_*_P_ / overlay; run;

37 Start to approach C(p)=p line here

38 Model Validation Since data used to generate parameter estimates, you d expect model to predict fitted Y s well Should check model predictive ability for a separate data set if available Various techniques of cross validation (data split, leave one out) are possible if only one data set available

39 Additional Multiple Regression Diagnostics Partial regression plots Studentized deleted residuals Hat matrix diagonals Dffits, Cook s D, DFBETAS Variance inflation factor Tolerance

40 KNNL Example Page 386, Section 10.1 Y is amount of life insurance X 1 is average annual income X 2 is a risk aversion score n = 18 managers

41 Read in the data set data a1; infile../data/ch10ta01.txt'; input income risk insur;

42

43 Partial regression plots Also called added variable plots or adjusted variable plots One plot for each X i

44 Partial regression plots These plots show the strength of the marginal relationship between Y and X i in the full model (recall partial correlation) They can also detect Nonlinear relationships Heterogeneous variances Outliers

45 Partial regression plots Consider plot for X 1 Use the other X s to predict Y Use the other X s to predict X 1 Plot the residuals from the first regression vs the residuals from the second regression

46 The partial option with proc reg proc reg data=a1; model insur=income risk /partial; run;

47 Output Analysis of Variance Source DF Sum of Squares Mean Square F Value Pr > F Model <.0001 Error Corrected Total Root MSE R-Square Dependent Mean Adj R-Sq Coeff Var

48 Output Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > t Intercept <.0001 income <.0001 risk

49 Curvilinear relationship

50 Can also see that here

51 Other Residuals There are several versions of residuals 1. Our usual residuals ei = Yi Yˆ i 2. Studentized residuals * ei ei = MSE 1 h ( ) Studentized means dividing by its standard error Are almost distributed t (n-p) ii

52 Studentized deleted Residual Delete case i and refit the model Compute the predicted value for case i using this refitted model Compute the studentized residual Don t do this literally but this is the concept Results in t-distributed residuals

53 Studentized Deleted Residuals We use the notation (i) to indicate that case i has been deleted from the model fit computations d = Y Yˆ is the deleted residual i i i(i) Turns out d i = e i /(1-h ii ) Also Var(d i )=Var(e i )/(1-h ii ) 2 =MSE (i) /(1- h ii ) t i = ei MSE(i) 1 h ii ( )

54 Using Residuals When we examine the residuals, regardless of version, we are looking for Outliers Non-normal error distributions Influential observations

55 The r option and studentized residuals proc reg data=a1; model insur=income risk/r; run;

56 Output Output Statistics Obs income risk Dependent Variable Predicted Value Std Error Mean Predict Residual Std Error Residual Student Residual Cook's D

57

58 Cook s Distance A measure of the influence of case i on all of the Ŷ i s (all the cases) It is a standardized version of the sum of squares of the differences between the predicted values computed with and without case i Compare with F(p,n-p) Concern if distance above 50%-tile

59

60 The influence option and studentized deleted residuals proc reg data=a1; model insur=income risk /influence; run;

61 Output Output Statistics Hat Diag Cov DFBETAS Obs income risk Residual RStudent H Ratio DFFITS Intercept income risk

62 Hat matrix diagonals h ii is a measure of how much Y i is contributing to the prediction of Ŷ i Ŷ i = h i1 Y 1 + h i2 Y 2 + h i3 Y 3 + h ii is sometimes called the leverage of the i th observation It is a measure of the distance between the X values for the i th case and the means of the X values

63 Hat matrix diagonals 0 h ii 1 Σ(h ii ) = p Large value of h ii suggess that i th case is distant from the center of all X s The average value is p/n Values far from this average point to cases that should be examined carefully

64

65 Output Output Statistics Hat Diag Cov DFBETAS Obs income risk Residual RStudent H Ratio DFFITS Intercept income risk

66 DFFITS A measure of the influence of case i on Ŷ i (a single case) Thus, it is closely related to h ii It is a standardized version of the difference between Ŷ i computed with and without case i Concern if greater than 1 for small data sets or greater than 2 pn for large data sets

67 DFBETAS A measure of the influence of case i on each of the regression coefficients It is a standardized version of the difference between the regression coefficient computed with and without case i Concern if DFBETA greater than 1 in small data sets or greater than 2/ n for large data sets

68 Variance Inflation Factor The VIF is related to the variance of the estimated regression coefficients We calculate it for each explanatory variable One suggested rule is that a value of 10 or more for VIF indicates excessive multicollinearity

69 Tolerance TOL = (1-R 2 k) where R 2 k is the squared multiple correlation obtained in a regression where all other explanatory variables are used to predict X k TOL = 1/VIF Described in comment on p 410

70 Output Parameter Estimates Parameter Standard Variance Variable DF Estimate Error t Value Pr > t Tolerance Inflation Intercept < income < risk

71 Full diagnostics proc reg data=a1; model insur=income risk /r partial influence tol; id income risk; plot rstudent.*(income risk); run;

72 Plot statement inside Reg Can generate several plots within Proc Reg Need to know symbol names Available in Table 1 once you click on plot command inside REG syntax r. represents usual residuals rstudent. represents deleted resids p. represents predicted values

73

74

75 Last slide We went over KNNL Chapters 9 and 10 We used program topic18.sas to generate the output