Multiple Linear Regression

Transcription

1 Multiple Linear Regression ST 430/514 Recall: a regression model describes how a dependent variable (or response) Y is affected, on average, by one or more independent variables (or factors, or covariates). The general equation is E(Y ) = β 0 + β 1 x 1 + β 2 x β k x k. I shall sometimes write E(Y ) as E(Y x 1, x 2,..., x k ), to emphasize that E(Y ) changes with the values of the terms x 1, x 2,..., x k : E(Y x 1, x 2,..., x k ) = β 0 + β 1 x 1 + β 2 x β k x k. 1 / 21 Multiple Linear Regression General Form

2 As always, we can write ɛ = Y E(Y ), or Y = E(Y ) + ɛ, where the random error ɛ has expected value zero: E(ɛ) = E(ɛ x 1, x 2,..., x k ) = 0. So the general equation can also be written Y = β 0 + β 1 x 1 + β 2 x β k x k + ɛ. 2 / 21 Multiple Linear Regression General Form

3 Each term on the right hand side may be an independent variable, or a function of one or more independent variables. For instance, E(Y ) = β 0 + β 1 x + β 2 x 2 has two terms on the right hand side (not counting the intercept β 0 ), but only one independent variable. We write it in the general form as with x 1 = x and x 2 = x 2. E(Y ) = β 0 + β 1 x 1 + β 2 x 2, 3 / 21 Multiple Linear Regression General Form

4 Interpreting the parameters: β 0 β 0 is still called the intercept, but now its interpretation is the expected value of Y when all independent variables are zero: β 0 = E(Y x 1 = 0, x 2 = 0,..., x k = 0). In some cases, these values cannot all be achieved at the same time; in these cases, β 0 has only a hypothetical meaning. 4 / 21 Multiple Linear Regression General Form

5 Interpreting the parameters: β i, i > 0 For 1 i k, β i measures the change in E(Y ) as x i increases by 1 with all the other independent variables held fixed. Again, in some cases it is not possible to change one variable and none of the others, so β i may also have only a hypothetical meaning. You will sometimes find, for instance, some β i < 0 when you expect that Y should increase, not decrease, when x i increases. That is usually because, when x i changes, other variables also change. 5 / 21 Multiple Linear Regression General Form

6 Quantitative and Qualitative Variables Some variables are measured quantities (i.e., on an interval or ratio scale), and are called quantitative. Others are the result of classification into categories (i.e. on a nominal or ordinal scale), and are called qualitative. Some terms may be functions of independent variables: distance and distance 2, or sine and cosine of (month/12). The simplest case is when all variables are quantitative, and no mathematical functions appear: the first-order model. 6 / 21 Multiple Linear Regression General Form

7 Example: Grandfather clocks ST 430/514 Dependence of auction price of antique clocks on their age, and the number of bidders at the auction. Data for 32 clocks. Get the data and plot them: clocks = read.table("text/exercises&examples/gfclocks.txt", header = TRUE) pairs(clocks[, c("price", "AGE", "NUMBIDS")]) The first-order model is E(PRICE) = β 0 + β 1 AGE + β 2 NUMBIDS. 7 / 21 Multiple Linear Regression General Form

8 Fitting the model: least squares ST 430/514 As in the case k = 1, the most common way of fitting a multiple regression model is by least squares. That is, find ˆβ 0, ˆβ 1,..., ˆβ k so that ŷ = ˆβ 0 + ˆβ 1 x ˆβ k x k minimizes SS E = (y i ŷ i ) 2. As noted earlier, other criteria such as y i ŷ i are sometimes used instead. 8 / 21 Multiple Linear Regression Fitting the model: least squares

9 Calculus leads to k + 1 linear equations in the k + 1 estimates ˆβ 0, ˆβ 1,..., ˆβ k. These equations are always consistent; that is, they always have a solution. Usually, they are also non-singular; that is, the solution is unique. If they are singular, we can find a unique solution by either imposing constraints on the parameters or leaving out redundant variables. 9 / 21 Multiple Linear Regression Fitting the model: least squares

10 The equations are: n ˆβ 0 + x i,1 ˆβ x i,k ˆβ k = y i xi,1 ˆβ0 + x 2 i,1 ˆβ x i,1 x i,k ˆβk = x i,1 y i xi,k ˆβ0 + x i,1 x i,k ˆβ1 + + x 2 i,k ˆβ k = x i,k y i where x i,j is the value in the i th observation of the j th variable, 1 i n, 1 j k. We usually write these more compactly using matrix notation, and solve them using matrix methods.. 10 / 21 Multiple Linear Regression Fitting the model: least squares

11 Matrix formulation of least squares ST 430/514 Write X for the n (k + 1) matrix of values of the independent variables (including a column of 1 s for the intercept): 1 x 1,1 x 1,2... x 1,k 1 x 2,1 x 2,2... x 2,k X = x n,1 x n,2... x n,k Also write y for the n 1 vector of values of the dependent variable: y 1 y 2 y =.. y n 11 / 21 Multiple Linear Regression Fitting the model: least squares

12 Finally, write ˆβ for the k 1 vector of parameter estimates: ˆβ 0 ˆβ 1 ˆβ =. ˆβ k Then the equations for the parameter estimates can be written X Xˆβ = X y. 12 / 21 Multiple Linear Regression Fitting the model: least squares

13 The equations are non-singular when (X X) 1 exists, and the solution may be written ˆβ = (X X) 1 X y. However, computing first X X and then its inverse (X X) 1 can lead to large numerical errors. Using a transformation of X such as the QR decomposition or the singular value decomposition gives better numerical performance. 13 / 21 Multiple Linear Regression Fitting the model: least squares

14 Model Assumptions No assumptions are needed to find least squares estimates. To use them to make statistical inferences, we need these assumptions: The random errors ɛ 1, ɛ 2,..., ɛ n are uncorrelated and have common variance σ 2 ; For small sample validity, the random errors are normally distributed, at least approximately. 14 / 21 Multiple Linear Regression Estimating Error Variance

15 As before, we estimate σ 2 using SS E = (y i ŷ i ) 2. We can show that E [SS E ] = (n p)σ 2, where p = k + 1 is the number of βs in the model, so the unbiased estimator is s 2 = SS E = SS E df E n p. = SS E n (k + 1). 15 / 21 Multiple Linear Regression Estimating Error Variance

16 Hypothesis Tests Usually, the first test is an overall test of the model: H 0 : β 1 = β 2 = = β k = 0. H a : at least one β i 0. H 0 asserts that none of the independent variables affects Y ; if this hypothesis is not rejected, the model is worthless. For instance, its predictions perform no better than ȳ. The test statistic is usually denoted F, and P-values are found from the F -distribution with k and n p = n (k + 1) degrees of freedom. 16 / 21 Multiple Linear Regression Testing the Utility of a Model

17 Individual parameters may also be tested: H 0 : β i = 0. H a : β i 0. The test statistic is t = ˆβ i standard error of ˆβ i It is tested using the t-distribution with n p degrees of freedom. 17 / 21 Multiple Linear Regression Inferences About Individual Parameters

18 Note When you test H 0 : β i = 0, you allow the other βs to be non-zero. So with k = 2, you may find that you reject the overall null hypothesis β 1 = β 2 = 0, but do not reject either β 1 = 0 or β 2 = 0! 18 / 21 Multiple Linear Regression Inferences About Individual Parameters

19 Confidence Intervals A confidence interval for any parameter is constructed in the usual way: ˆβ i ± t α/2,n (k+1) standard error. But bear in mind that each such interval has probability α of not covering the true value β i. If you construct many such intervals, there is a greater chance that at least one of them fails to cover its true value. 19 / 21 Multiple Linear Regression Inferences About Individual Parameters

20 Example: Grandfather clocks again ST 430/514 Dependence of auction price of antique clocks on their age, and the number of bidders at the auction. Fit the first-order model and summarize it: clockslm = lm(price ~ AGE + NUMBIDS, clocks) summary(clockslm) 20 / 21 Multiple Linear Regression Example

21 Output ST 430/514 Call: lm(formula = PRICE ~ AGE + NUMBIDS, data = clocks) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-08 *** AGE e-14 *** NUMBIDS e-11 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 29 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 2 and 29 DF, p-value: 9.216e / 21 Multiple Linear Regression Example