Biostatistics in Research Practice - Regression I

Transcription

1 Biostatistics in Research Practice - Regression I Simon Crouch 30th Januar 2007 In scientific studies, we often wish to model the relationships between observed variables over a sample of different subjects. For eample, one might measure the heights and weights of a number of subjects and investigate possible relationships between these variables. One might also wish to investigate whether other factors such as gender or particular states of health affect this relationship. Frequentl one wishes to consider a particular observation as a response variable to be eplained b the values of one or more eplanator variables. For eample, we ma wish to model someone s weight as a function of their height, gender and state of health. Outside of the field of fundamental phsics, it is unusual to have available all the variables that might provide a complete scientific eplanation of the value of some particular observation. In these cases, we frequentl use a probabilistic model in order to take account of what we cannot model deterministicall. It is also often the case that we do not have available to us a scientific model for our data. This being the case, we are often satisfied to develop a model that ma be a simplification of the processes actuall involved, provided that it is sufficient for what we actuall want to do. A reasonable first attempt would be to fit a linear model

2 How would we fit such a model? A straightforward wa, called the Method of Least Squares, is to minimize the sum of the squared residuals left over after fitting the line. 0 0 A ver useful wa of thinking about this that generalizes to a much wider class of models is the idea of Eplained Variation. Our response variable shows a degree of variabilit about its mean - for eample, our sample of twent people have differing weights. We eplain some of the variabilit of people s weights b the fact that the have different heights. We might eplain more of the variabilit b taking into account their gender. When we ve fitted our linear model, there ll still be some uneplained variation, but we can give a measure of how much variation we have eplained b our model b comparing the residual sum of squares with the total sum of squares. These are illustrated b the figures on the left and on the right respectivel. We often use a quantit called R 2 that is one minus the ratio of the residual sum of squares and the total sum of squares. Notice that so far we haven t mentioned probabilit or statistics at all! The least squares method is a geometric method of fitting a curve to some points that has no statistical content and we can make no statistical inferences from such a method. We ll now introduce the probabilistic content that will allow us to make statistical inferences about our models. (Note that because of the wa we introduce probabilit into the model, through the normal distribution, the method of least squares is still the right wa to fit a model. It turns out that minimizing the sum of squared residuals is mathematicall equivalent to maimizing the likelihood associated with the probabilistic structure of the model). Linear Regression Models are simple but powerful tools that allow us to model the structure in data in a wide variet of circumstances. Suppose that we have n observations of some response variable Y 1,..., Y n and for each i we have the values of some eplanator variables X 1,i,..., X p,i. we model the response as: Y i = β 0 + X 1,i β 1 + X 2,i β X p,i β p + ɛ i 2

3 where the {ɛ i } are independent normal random variables with mean 0 and common variance σ 2. Y is called the Response Variables and the X j are called Eplanator Variables. (You will also hear the terminolog dependent and independent variables as well as variates and covariates). If p = 1, we have simple linear regression, otherwise we have multiple linear regression. You might also hear about multivariate linear regression where the response variable is of dimension greater than one. The coefficient β 0 is often called an intercept. Unless ou have definite scientific evidence to suggest that there should be no intercept in our model, there should be one. Strictl speaking, Y has to be a continuous random variable defined on the entire real line (, + ). (Eercise: Wh?) However, in practice, it is not uncommon to model other tpes of response using this tpe of linear model, provided that the residuals from the fitted model are approimatel normal. Notice that we ve made no assumptions about the X s. The might be continuous variables, the might be categorical or ordinal. From a statistical point of view, we don t care how the are distributed (though note that their actual distribution ma have an effect on us being able to fit a model). If all the X s in a model are categorical, there s a close connection between a regression model and the Analsis of Variance; if there s a miture of categorical and continuous eplanator variables, ou ma hear the term Analsis of Covariance. Once we ve introduced a probabilistic structure to our model b declaring that the residual error process is normal, we can start to make statistical inferences using our model. For eample, we can put 95% confidence intervals around our estimates of the coefficients β i. For the data we ve been using so far, the estimate of the regression line is = % confidence intervals for the intercept and coefficient of are (.43, 8.8) and (0.24, 1.0) respectivel (and the eplained variation is 0.36). The results of this simple linear regression are eas to eplain. For ever unit increase in the value of, the epected value of increases b We can use such a model to eplain what is going on in the relationship between variables. We can also use the model to predict what some future response might be for a new observation with some value of the eplanator variable. In addition to confidence intervals for the coefficients, we can give confidence intervals for our predictions. These two intervals are sometimes called Confidence Bands and Tolerance Bands. 3

4 In the figure, the narrow error bounds are those for the fitted line, whereas the wider error bounds are for predictions from the model. We make a number of assumptions when we fit a model like this. We assume that The errors ɛ i are normall distributed with zero mean. The errors ɛ i all have the same common variance σ 2. This is sometimes called homoscedasticit. The errors ɛ i are independent of one another. The form of the model is correct, in the sense that the correct eplanator variables are in the model and have the correct functional form. When we make assumptions like this, it is important to check that the are true for the model we fit. There are a number of simple was was of checking the first two of these assumptions. We can check that the residuals are normall distributed b performing a Q-Q Plot and we can check that the residuals have the same variance b inspecting a plot of residuals versus fitted values. In most cases these visual plots are sufficient to establish whether these assumptions are true, but ou should note that ou can construct formal statistical tests for them as well. (Eercise: How?). 4

5 Normal Q Q Plot Sample Quantiles res Theoretical Quantiles fitted(mod1) Checking the independence of the observations is less easil done (but there are statistical tests such as the Durbin Watson test for serial correlation) and one would usuall justif independence on some scientific basis. You should note that if it is known that the observations are not independent, it is still possible to model the data, but that is for another course! Checking that the form of the model is correct can be much harder. We will look at strategies for deciding which eplanator variables should be in a model net week. Checking the functional form for the eplanator variables is often done b graphical methods. The regression line on the right shows a reasonable fit to the data shown on the left. But there s clearl a suggestion that something s wrong! The Q-Q plot of residuals is slightl suspicious, but the plot of residuals against fitted values is a dead giveawa. This latter plot is often ver useful for showing trends in the data (either 5

6 functional form or heteroscedasticit) and ma indicate what needs to be done to cure the problem. Normal Q Q Plot Sample Quantiles res Theoretical Quantiles fitted(mod5) How does one cure the problem? In this case, the original scatter plot of the data and the plot of residuals against fitted for the straight line fit both suggest a quadratic fit. That is, we tr fitting a model of the form Y i = β 0 + β 1 X i + β 2 X 2 i Note that despite this having a quadratic term in the eplanator variable X, this is still a linear model! When we talk about linear models we are referring to linearit in the coefficients β i. So the model above is linear but one such as Y i = β X i 0 is not. In terms of the general form of the linear model we gave earlier, some of the X s ma be combinations (products, powers or some other functions) of the others. When we fit a model with a quadratic term this is what we get. Clearl a much better fit to the data, which the residual plot on the right confirms. 6

7 res fitted(mod6) Estimate Standard Error p-value Intercept X X tin Notice two things: if ou have a model with higher order terms for some covariate in it, ou should keep the lower order terms of that covariate; despite an reported p-value, ou should also keep the intercept term in the model. We ll discuss in more depth how we decide which terms to keep in a regression model net week, but for the moment, remember these important rules. Deciding which terms should go into a regression model as eplanator variables in the first place when ou ve got more than one candidate can be trick. You should be guided b an scientific information that ou have available - but of course, it ma be that our investigation is aimed at discovering that scientific eplanation! Net week, we ll look at model building in more depth; for the moment remember that graphical plots such as scatterplots can be ver useful. 7

8 z Here, if we suppose that X and Y are eplanator variables and that Z is the response variable, there appears to be a clear linear relationship between Y and Z, but the relationship between X and Z is less clear. Still, tring a model of the form Z i = β 0 + X i β 1 + Y i β 2 would be fair. You ma also have available to ou partial regression plots that show ou the residuals from a possible model plotted against each covariate in turn after having taken into account the other covariates. 8

9 beta*+res 0 beta*+res These plots confirm that our model is reasonable. However, it would also be a good idea to include interaction terms in the model and see what happens. Z i = β 0 + X i β 1 + Y i β 2 + X i Y i β 3 The first model with no interaction term is eas to interpret. If ou hold Y fied and change X b one unit, then the epected value of Z changes b β 1. If ou hold X fied and change Y b one unit, then the epected value of Z changes b β 2. However, if there is an interaction term, the model becomes harder to interpret. If ou hold Y = Y f fied and change X b one unit, then the epected value of Z changes b β 1 + β 3 Y f. The rate of change of Z with respect to X depends on the value of Y. How this is interpreted in scientific terms depends upon the problem in hand. However, if one of the eplanator variables is discrete (binar, perhaps) then the interpretation of a model with interaction is easier. 9

10 In the left hand figure we ve plotted the value of a response variable Y against the eplanator variable X. There is a second (binar) eplanator variable T that defines two groups in the data, and these are clear on the plot. If we fit a linear regression using onl X as an eplanator variable, we get the right hand figure. Something clearl appears to be wrong! (You ma wish to find out about Simpson s Parado and relate it to these graphs) If we now do a regression using both Y and T as eplanator variables with no interaction Y i = β 0 + β 1 X i + β 2 T i then we get the fit on the left hand side. Notice that the slope of the two lines is the same. Does it look right et? 10

11 If we now fit a model with an interaction term Y i = β 0 + β 1 X i + β 2 T i + β 3 X i T i then we get the fit on the right hand side. Notice that the slope of the line of Y against X now depends on the value of T. The fit appears much more satisfactor. 11