Lecture 6. Multiple linear regression. Model validation, model choice

Transcription

1 Lecture 6. Multiple linear regression. Model validation, model choice Jesper Rydén Department of Mathematics, Uppsala University Regression and Analysis of Variance autumn 2015

2 Influential observations Influential point: A point whose removal from the data set would cause a large change in the fit. Measures of influence: Change in the coefficients ˆβ ˆβ (i) Change in the fit, X T ( β β (i) ) = ŷ ŷ (i). Some examples: DFBETA, DFFIT, PRESS, Cook s distance

3 Cook s distance and PRESS Cook s distance: D i = 1 (k + 1)s 2 (ˆβ ˆβ(i)) X X (ˆβ ˆβ(i)) = [e(s) i ] 2 h ii k + 1 (1 h ii ) 2 PRESS: e i e i, 1 = y i ŷ i (i) = 1 h ii

4 Example: Cook s distance Dataset mtcars in R: mpg wt + hp. mpg ~ wt + hp Cook s distance Index

5 Cut-off values? John Fox (1997): These criteria are no substitute for graphical examination of the residuals. Hat values: Exceeding 2(k + 1)/n. Studentized residuals: Outside the range e i 2. Influence, Cook s distance: D i > 4/(n k 1)

6 The problem of collinearity (Multicollinearity, ill conditioning) Subject which can be seen from different views: Statistics Inflated variances of regression coefficients. Tools for diagnosis: correlations, variance inflation factors. Numerical analysis Rounding errors, inaccuracy. Tools for diagnosis: singular values, condition numbers.

7 Collinearity: matrix formulation Recall that in the usual OLS model, β = (X T X) 1 X T y, Cov[ˆβ] = σ 2 (X T X) 1 Problem if X T X is singular. All matrices have generalized inverses, but in the singular case, these are not unique. In practice: columns of X are nearly linearly dependent, leads to near singularity. This is often the case e.g. with econometric data: highly correlated explanatory variables. It is recommended to scale X. Several authors recommend models both centred and scaled: controversy on the subject.

8 Stability of LS plane

9 Residual sum of squares

10 Geometrical interpretation: an unstable table Question: Dear Professor Mean, Could you describe the term collinearity for me? I understand that it has to do with variables which are not totally independent, but that is all I know! Prof. Mean (not your average professor): When you have overlap among some of the variables, it takes more data to disentangle the individual effects of these variables. Think of it as a table where you push two of the four legs away from the corners and close to the middle of the table. Such a table will be very unstable.

11 Note on partial correlation NE (in Swedish): Den korrelation mellan två variabler som kvarstår efter det att den del i de bägge variablerna som kan predikteras (linjärt) från en tredje variabel har eliminerats. Consider the random variables X 1, X 2 och X 3 with correlation coefficients ρ 12, ρ 13 och ρ 23. Related sample correlation coefficients: r 12, r 13 och r 23. Partial correlation coefficient): r 12 3 = r 12 r 13 r 23 (1 r13 2 )(1 r 23 2 ) (See e.g. G.W. Snedecor, W.G. Cochran: Statistical Methods (6th ed) R-rutin: pcor.test

12 Example Random sample, 142 older women from (IA) and Nebraska (NE). Variables: From data: A: Age, B: Blood pressure, C: blood cholesterol r AB = , r AC = , r BC = One usually assumes that B and C increase with age. Are B and C correlated merely because of their common association, or is there a real relation at every age? We compute: r BC A = ( )( ). = Statistical test can be performed, on (142-3) degrees of freedom. It turns out that correlation is not significant. It may be that within the several age groups, B and C are uncorrelated.

13 Multiple correlation Multiple correlation: Definition. The sample multiple correlation between x 1 and x 2,..., x k is defined by R 2 1 (2,...,k) = wt 12 W 1 22 w 12 w 11 where and Z = W = Z T Z x 11 x 1... x 1k x k..... x n1 x 1... x nk x k

14 Tolerance and Variance Inflational Factor Denote by Rj 2 the R 2 between the variable x j and a linear combination of all other independent variables. The tolerance TOL j is defined as TOL j = 1 R 2 j and is close to one if x j is not closely related to other regressor variables. The variance inflation factor VIF j is given by VIF j = 1 TOL j A value of VIF j close to 1 indicates no relationship, large values indicate precense of multicollinearity. Rule of thumb: VIF > 10 of concern.

15 Eigenvalues and condition numbers Denote by λ 1,..., λ k the eigenvalues of the matrix X T X. Condition number: λ 1 κ = λ k Serious concern: κ > 20, κ > 30. One may also study condition numbers like λ 1 /λ j.

16 Model selection

17 Variable selection Select the best subset of predictors... Want to explain the data in the simplest way; redundant predictors should be removed. Cf. the principle of Occam s razor, lex parsimoniae Unnecessary predictors will add noise to the estimation of other quantities that we are interested; degrees of freedom will be wasted. Collinearity is caused by having too many variables trying to do the same job. Cost: If the model is to be used for prediction, time and/or money can be saved by not measuring redundant predictors [Faraway: Practical regression and Anova using R. Chapter 10]

18 Introductory example: Hierarchical models and selection of variables Lower-order terms should not be removed from the model before higher-order terms in the same variable. Polynomial models y = β 0 + β 1 x + β 2 x 2 + ɛ Suppose after LS estimation that the regression summary shows the term in x not significant, but the term in x 2 is. Remove x term, reduced model: y = β 0 + β 2 x 2 + ɛ However, choice of scale (unit) matter (e.g. x x + a). Not good, x term reappears: y = β 0 + β 2 a 2 + 2β 2 ax + β 2 x 2 + ɛ

19 Dropping variables Correct model (fuller one). Effects on Estimates of β j Estimation of error variance Covariance matrix of estimates Predicted values Blackboard

20 Criterion based procedures Various rules of thumb: The number of observations should be 5 10 times the number of independent variables; n k 1 10; n 5k How many possible models? With k independent variables: 2 k 1 models k = 3 k = 5 k = 10 k =

21 Criterion based procedures Some common criteria: Information criteria: AIC and BIC Adjusted R 2 Predicted Residual Sum of Squares (PRESS) Mallow s C p

22 AIC and BIC Assume p potential predictors. Akaike s Information Criterion (AIC) and Bayesian Information Criterion (BIC): where SS E = (y i ŷ i ) 2. AIC = n ln(ss E /n) + 2p BIC = n ln(ss E /n) + p ln n Want to minimize AIC or BIC Larger models fit better; have smaller SS E but more parameters Try to balance fit and model size BIC penalizes larger models more heavily and tend to prefer smaller models in comparison to AIC.

23 Adjusted R 2 Recall coefficient of determination R 2 ; interpreted as percentage of variance of explained: R 2 (ŷi ȳ) 2 = (yi ȳ) 2 = 1 (ŷi y i ) 2 (yi ȳ) 2 = 1 SS E SS T Adding a variable to a model decreases SS E and thus increases R 2. Adjusted R 2, denoted as R 2 a : R 2 a = 1 SS E /(n p) SS T /(n 1) = 1 ( ) n 1 (1 R 2 ) n p

24 PRESS (Predicted residual sum of squares) Definition: PRESS = n i=1 ɛ 2 i, 1 where ɛ i, 1 are residuals calculated without case i in the fit. Thus a cross validation approach The model with lowest PRESS is chosen.

25 Mallow s C p Statistic: C p = (SS E ) p ˆσ 2 + 2p n where ˆσ 2 is an estimate of variance from the model with all predictors and (SS E ) p is the SS E from a model with p parameters. C p is easy to compute For the full model, C p = p exactly Desirable: C p p, p small

26 Model building: Exploring regression data The fundamental axiom of this data analysis is the declaration: The data analyst knows more than the computer. For example, most data analysts will know: (i) Water freezes at 0 C, so a discontinuity at that temperature might not be surprising. (ii) People tend to round their age to the nearest five-year boundary. (iii) The Arab oil embargo made 1973 an unusal year for many economic measures. (iv) The lab technician who handled Sample 3 is new on the job. H.V. Henderson, P.F. Velleman. Biometrika, 1981.

27 Backward elimination A critical α c is chosen, p to remove. 1. Start with all predictors in the model. 2. Remove the predictor with the highest p value greater than α c. 3. Refit the model, goto step Stop when all p values are less than α c.

28 Forward selection The backward method reversed. 1. Start with no variables in the model. 2. For all predictors not in the model, check the p value if they are added to the model. Choose the one with lowest p value less than α c. 3. Continue until no new predictors can be added.

29 Stepwise: Some drawbacks Due to the one-at-a-time nature of adding/dropping variables, it is possible to miss the optimal model. The p values used should not be treated too literally. There is much multiple testing occurring: validity is dubious. The procedures are not directly linked to the final objectives of prediction or exaplanation. Stepwise methods tend to pick models that are smaller than desirable for prediction purposes. (Simple example: simple linear regression; slope not significant)

30 Statistical strategy and model uncertainty Some tactics: Diagnostics Checking of assumptions: constant variance, linearity, normality, outliers, influential points, serial correlation and collinearity Transformation Transforming the response (Box Cox), transforming the predictors; tests and polynomial regression Variable selection Stepwise and criterion based methods What order? Faraway recommends: Diagnostics Transformation Variable selection Diagnostics

31 Avoid too much analysis Torture the data long enough, and sooner or later it will confess. Avoid complex data models for small datasets. Try to obtain new data to validate your proposed model. (Maybe set aside some of the existing data for this purpose) Use past experience with similar data to guide the choice of model.

32 Example: Population data from US states Data from the 50 states collected from the US Bureau of the Census, variables: y Life.Exp Life expectancy in years ( ) x 1 Population Population estimate (July 1, 1975) x 2 Income Per capita income (1974) x 3 Illiteracy Illiteracy (1970, percent of population) x 4 Murder Murder and non-negligent manslaughter rate per population (1976) x 5 HS.Grad Percent high-school graduates (1970) x 6 Frost Mean number of days with min temperature below 32 F (0 C), in capital or large city x 7 Area Land area in square miles. Example from Faraway (2002): Practical Regression and Anova using R

33

34 R output lm(formula = Life.Exp ~., data = statedata) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) 7.094e e < 2e-16 *** Population 5.180e e Income e e Illiteracy 3.382e e Murder e e e-08 *** HS.Grad 4.893e e * Frost e e Area e e Signif. codes: 0 *** ** 0.01 * Residual standard error: on 42 degrees of freedom Multiple R-squared: ,Adjusted R-squared: F-statistic: on 7 and 42 DF, p-value: 2.534e-10

35 The F test, formulated in terms of R 2 Consider a general model with RG 2 and a hypothesis model with RH 2. The quantity in the F test in the linear model can be formulated F = R2 G R2 H / 1 R2 G k l N k

36 Transformations Transformations may arise for several reasons in regression models. 1./ Some models are formulated in a multiplicative manner, e.g. µ(x) = αx β, Y = µ(1 + ɛ) or µ(x) = αe βx, Y = µ(1 + ɛ) 2./ Sometimes the response variable is transformed to cope with non-constant variance. A common technique is Box Cox transformation (Box and Cox, 1964).

37 Example, multiplicative model: A gravity model for migration Gravity model for migration between two cities (e.g. Abler et al. (1971)). Y ij : D ij : P i, P j : Gravity model: Number of migrants moving from city i to city j Distance between the cities Populations of the cities Y ij = α Pβ i P γ j D δ ij ɛ ij where α, β, γ, δ are unknown parameters to be estimated from data.

38 Identifiability If X T X is singular and cannot be inverted, there will be infinitely many solutions to the normal equations. Problem when X is not of full rank: columns are linearly dependent. Examples: Weight of a person in pounds and kg: both variables in the model. Number of years of studies at different levels as well as total number of years of studies included in the model. If p > n, more variables than cases: supersaturated model. If p = n, saturated model.

39 Identifiability There are insufficient data to estimate the parameters of interest or There are more parameters than are necessary to model the data.