Simple linear regression: estimation, diagnostics, prediction

Transcription

1 UPPSALA UNIVERSITY Department of Mathematics Mathematical statistics Regression and Analysis of Variance Autumn 2015 COMPUTER SESSION 1: Regression In the first computer exercise we will study the following subjects: Simple linear regression: estimation, diagnostics, prediction Multiple regression Examples of transformation Let s begin! 1 Simple linear regression For simplicity, we use for large parts of the session the built-in data set mtcars. Load the data by writing: data(mtcars); attach(mtcars) In simple linear regression we assume the model y i = β 0 + β 1 x i + ɛ i, i = 1,..., n where β 0 and β 1 are regression coefficients, (x i, y i ) are observed values and ɛ i NID(0, σ 2 ). Let us, to begin with, study the effect of weight on fuel consumption, that is, we take mpg as our y and wt as our x. We can plot the data by writing: plot(mpg ~ wt, xlab = "Weight", ylab="miles per gallon") As we will see below, mpg wt is interpreted as mpg as a function of wt. 1.1 Estimation of the parameters A useful R-routine for linear regression is lm (short for linear model). The model is written using a symbolic notation suggested by Wilkinson and Rogers: y = β 0 + β 1 x 1 corresponds to y x1 y = β 0 + β 1 x 1 + β 2 x 2 + β 3 x 1 x 2 corresponds to y x1+x2+x1*x2 y = β 0 + β 1 x 1 + β 2 x 2 1 corresponds to y x1+i(x1^2) The routine for fitting a linear model by the least-squares method is lm. By the following command, information is collected in the object called m1. m1 <- lm(mpg ~ wt, mtcars) 1

2 The general R command str gives an overview of the object, type str(m1) in this case. Moreover, an output summarizing the important information can be given by summary(m1). In the output from the summary command, to find where R 2 and the estimates of β i are located and which variables are significant. It possible to access the numerical values of an object. Try for instance m1$coefficients summary(m1)$coefficients[2,1] We can add the estimated curve (actually, a straight line) in red to the figure that we plotted earlier by using the following command (note that the short form coef works for coefficients): abline(coef(m1), col="red") Compare the figure with the estimated coefficients and convince yourself that the fit seems reasonable. Confidence interval for the regression coefficients The standard errors of the estimators are given in the summary-table. To calculate a confidence interval for β 1 we need a t-quantile, which we obtain using the function qt. # Some preparations beta1 <- summary(m1)$coefficients[2,1] sterror <- summary(m1)$coefficients[2,2] f <- m1$df.res # Degrees of freedom quantile <- qt(0.975,f) # the (or 0.975) t-quantile # The interval c(beta1 - quantile*sterror, beta1 + quantile*sterror) 1.2 Diagnostics To study the impact of randomness we study the residuals y i ŷ i. These are extracted from the model by residuals(m1) Let us, to begin with, plot the sequence of raw residuals. plot(residuals(m1)) Does there seem to be some evident pattern in the sequence? If so, we need to examine our data more carefully. For a (very) rough check of the normality assumption we can look at the histogram of the observations: hist(residuals(m1)) 2

3 Does the histogram resemble the normal bell-curve? Looking at a Q-Q-plot is probably a better way to investigate the normality assumption. We get one by writing qqnorm(residuals(m1)); qqline(residuals(m1)) Do the points follow the line? A popular (and, usually, powerful) formal test for normality is the Shapiro-Wilk test: shapiro.test(residuals(m1)) Remind yourself what the null hypothesis of the test is. What is the conclusions given by the p-value? Extracting the design matrix Given a model from a call of lm we can extract the so called design matrix, i.e. X in the matrix formulation of the regression model; y = Xβ. The following command extracts the matrix and calculates the useful matrix (X T X) 1. X <- model.matrix(m1); solve( t(x) %*% X ) EXERCISE. Recall from theory that the estimated covariance matrix for the regression coefficients is σ 2 (X T X) 1. For an object m1, this can be found as vcov(m1) in R. Now, verify the elements in the covariance matrix by multiplying suitable elements of (X T X) 1 found above and the estimated variance of the residuals (which is found in the summary table). Compare your answer to that found elsewhere in the summary table (squaring the standard errors forthe individual parameter estimates). Diagnostics plots Next we ll see how we can plot different residuals and leverage/influence measures. To plot some useful figures we can write # Plot are wanted in two rows and two columns: par(mfrow=c(2,2)) # Sequence of residuals: plot(residuals(m1)) # Residuals against fitted values: plot(m1$fit,m1$res) # R-Student residuals: plot(rstudent(m1)) # Cook s distance: plot(cooks.distance(m1)) Judging by the Cook s distance, it seems that some points have large influence. To find out which, use the commands below. A crosshair will appear in the figure; click on interesting points using the the left mouse button to identify them and click the right mouse button to end the identification procedure. car.models <- row.names(mtcars) identify(1:32,cooks.distance(m1),car.models) 3

4 If we only are interested in the index of the observation, we leave out the car.models part of the identify call. Given an lm object, e.g. m1 in our case, R can give a lot of figures for residual diagnostics. Simply write plot(m1) Press Enter on the keyboard to go to the next figure. 1.3 Prediction With a model from lm we can easily do prediction: predict(m1,mtcars) Compare the predicted values ŷ i to the original observations y i. Prediction intervals are obtained by adding another argument: predict(m1, mtcars, interval="prediction") Let us now assume that we want to predict arbitrary values and that we want to show the result in a figure. We plot confidence intervals (for the curve) as well as prediction intervals (for the values that we want to predict; wider and more insecure ). attach(mtcars) # Sequence of x-values that we want to do prediction for pred.frame <- data.frame(wt=seq(1.5,5.5,0.5)) # Calculate prediction and confidence intervals pp <- predict(m1, int="p", newdata = pred.frame) pc <- predict(m1, int="c", newdata = pred.frame) # Graphics (introducing the command matlines) plot(wt,mpg,ylim=range(mpg,pp,na.rm=t)) pred.mpg <- pred.frame$wt matlines(pred.mpg,pc,lty=c(1,2,2),col="blue") matlines(pred.mpg,pp,lty=c(1,3,3),col="black") For the graphical options with lty, se the help text for par. For instance, lty=1,2 or 3 correspond to solid, dashed or dotted, respectively. 2 Multiple regression Next we ll use a second explanatory variable: the power of the engine measured in horse powers (hp). An lm call storing the model in m2 is now m2 <- lm(mpg ~ wt + hp); summary(m2) Feel free to examine residuals, normality and influence as we did above, this time using m2. It might also be of interest to look at the design matrix. Compare the R 2 of m1 with that of m2. Was it what you expected? Prediction: Assume that we want to predict the fuel consumption of a new, rather heavy, car with a small engine: wt= 3.5, hp= 90. The commands are: 4

5 x0 <- data.frame(wt=3.5,hp=90) yhat <- predict(m2,x0) EXERCISE. The hat matrix H, satisfying ŷ = Hy, is given by H = X(X T X) 1 X T. Plot studentized residuals for the fitted object m2. Do you find, perhaps, three spurious observations? Find the related values of the hat matrix, either by typing in from definition (using model.matrix and diag) or the ready-to use command as hatvalues(m2). Use the rule of thumb that hat values for leverage points exceed 2(k + 1)/n where k is the number of explanatory variables and n the number of observations and draw conclusions for the observations. 2.1 Polynomial regression and model choice Suppose we want to introduce a model with a quadratic term, when modelling mpg as a function of wt. (Could seem possible from the original plot?) The commands in R are as follow: mqua <- lm(mpg ~ wt + I(wt^2), mtcars); summary(mqua) Compare the R 2 values between models m1 and mqua (also the adjusted ones). Now, consider an example also including a qualitative variable. An engineer wants to estimate the expected time E[Y ] per month (in hours) for machines out of use due to maintenance as a function of the explanatory variables machine type (1 or 2) and the age of the machine (in years). The following model is suggested: E[Y ] = β 0 + β 1 x 1 + β 2 x β 3 x 2 where x 1 is the age of the machine, x 2 the type of machine (x = 1: Type 1, x = 0, Type 2). In the file shutdown.dat data is collected. Save into a suitable directory, read into R and study the data structure: mask <- read.table("shutdown.dat") mask str(mask) attach(mask) EXERCISE. (a) Estimation of parameters: mask0 <- lm(v1 ~ V2 + I(V2^2) + V4) summary(mask0) Does the model seem reasonable? What variables are significant? 5

6 (b) A simpler model is considered, and one wants to test β 1 = β 2 = 0 (at the level α = 0.10). More precisely, H 0 : β 1 = β 2 = 0 H 1 : At least one β i 0, i = 1, 2. We may use the fact (Sundberg, page 75) that F = R2 0 R2 1 k l /1 R2 0 N k F (k l, N k) where R 2 0 and R2 1 are the R2 values of the general (full) model and hypothesis model, respectively, and we have N = 20, k = 3 and l = 1. If the observed value of the test quantity F is larger than the F quantile, we reject the hypothesis of the simpler model and reperation time depends on age. Find if this is the case for our data. (A quantile from the F distribution is found by using qf.) 3 Transformations Transformations are often useful in regression. We ll study the effect of heating on the strength of vegetables (e.g. carrots). Such experiments are of importance for packing strategies for food. The data comes from an experiment from Belgium. The temperature was fixed at 90 C, the heating times were measured in seconds and the force in N. We import the data (download the file skalla.dat from the course page of the student portal) and plot the strength as a function of the length of the heating time. We also plot a figure where we ve taken the logarithm of the strength: skalla <- read.table("skalla.dat", col.names=c("temp","time","force") ) attach(skalla) plot(force~time) plot(log(force)~time) Do you think that the use of the logarithm will give us a better linear regression and a more homogeneous variance? The regression models are fitted as usual: m3 <- lm(force ~ Time) m4 <- lm(log(force) ~ Time) Look at the result of the regression: significant variables, R 2 and so on. If you have the time, use the boxcox function in R to find out whether taking the logarithm is a good transformation. When selecting a power transformation of the response variable with the Box Cox method, a suggested exponent of 0 corresponds to taking logarithm. The routine is found in the MASS-package, activate this by typing library(mass). If not already installed in your computer environment, it can be installed by writing install.packages(), choosing Sweden and then choosing MASS. For the routine, use help(boxcox) to get instructions on its usage. 6