Multiple Linear Regression

Transcription

1 Andrew Lonardelli December 20, 2013 Multiple Linear Regression 1

2 Table Of Contents Introduction: p.3 Multiple Linear Regression Model: p.3 Least Squares Estimation of the Parameters: p.4-5 The matrix approach to Linear Regression: p.5-6 Estimating σ 2 : p.7 Properties of the Least Squares Estimator: p.7-8 Test for Significance of Regression: p.8-9 R 2 and Adjusted R 2 : p.9-10 Tests On Individual Regression Coefficients and Subsets of Coefficients: p Hypothesis for General Regression Test: p.11 Confidence intervals on Individual Regression Coefficients: p.12 Confidence Interval on the Mean Response: p Prediction of New Observations: p.13 Residual Analysis: p Influential Observations: p Polynomial Regression Models: p.15 Categorical Regressors and Indicator Variables: p.15 Selection Of Variable Building: p Stepwise Regression: p.17 Forward Selection: p.17 Backward Elimination: p Multicollinearity: p.18 Data/Analysis: p

3 Introduction: In class, we went over simple linear regression where there was one predictor/ regressor variable. This regressor variable comes with the slope of a best fit line, which tries to extract most of the information in the data given. Learning how to create multiple linear regression models can give some ideas and insights into relationships of different variables with different responses. Usually engineers and scientists use multiple linear regression when working with experiments that have many different variables affecting the outcome of the experiment. Multiple Linear Regression Model There are different situations where there will be more than one regressor variable and this is called the multiple regression model. With k regressors we get (1) Y=β 0 + β 1 x 1 + β 2 x β k x k +ε This is a multiple linear regression multiple of k regressors and we consider the error term ε to be close to zero. We say linear because the equation (1) is a linear function of the unknown parameters β 0, β 1, β 2,.., β k. A multiple linear regression model/equation gives a surface where the β 0 is the intercept of the hyperplane, while the coefficients of the regressors are known as the partial regression coefficients. β 1 measures the expected change in Y per unit change in x 1 while x 2,..., and x k are all held constants. The same would be said for the other partial regression coefficients. The dependent variable will be Y, while the independent variables will be all the different x s. Multiple linear regression models are used for approximations for the Y variable or even any of the x variables. Least Squares Estimation of the Parameters 3

4 The least squares method is used to estimate the regression coefficients in the multiple regression equation. Suppose that n>k observations are available and let x ij show the ith observation of variable x j, the observations are: Data for Multiple Linear Regression y x 1 x 2 x k y 1 y 2 x 11 x 12 x 1k x 21 x 22 x 2k y n x n1 x n2 x nk (This table is depicted that same way as the NHL data table used below.) Then the model would be: The least squares function is We want to minimize the least squares function with respect to β 0, β 1,, β k. The least squares estimates of β 0, β 1,, β k must satisfy and And by simplifying the equation we get the scalar least squares equation: 4

5 Given data, solutions for all the regression coefficients can be obtained with standard linear algebra techniques. The matrix approach to Linear Regression When fitting a multiple Linear Regression model, it would be a lot simpler expressing the operations using a matrix notation. If there are k regressor variables and n observations, (x i1, x i2,, x ik, y i ), i = 1, 2,, n, the model relating to the regressor response is: This model can be expressed in matrix notation: where y i = β 0 +β 1 x i1 +β 2 x i β k x ik +ε i i = 1, 2,..., n y=xβ+ε y = X = β= β β β and ε= The X matrix is called the model matrix. o least squares estimator to solve for the vector β is: and β is the solution for β in the partial derivative equations: 5

6 Theses equations can be shown to be equivalent to the following Normal Equations This equation is the least squares equation in a matrix form which is identical to scalar least squares equation given before. The Least Squares Estimate of β This is the same equation as before, but we just isolated β. And this is the matrix form of the normal equations, as you can see, these equations hold a large resemblance to the scalar normal equations. With this, we get the fitted regression model to be: And in matrix notation, it would look like this: The residual would be the difference between the observed y i and the fitted value. Later on I will calculate the residual for my data. This is a (n x 1) vector of residuals. 6

7 Estimating σ 2 Measuring the variance of the error term, σ 2 in multiple linear regression is similar to measuring the σ 2 in just a simple linear regression model. In simple linear regression, we divide the sum of the squared residuals by n-2 because there were only 2 parameters. Instead, in a multiple linear regression model there are p parameters so we would divide the sum of the squared residuals by n-p. (SS E is the sum of the squared residuals) For my hockey data that you will see later on, there are 15 parameters in total. (14 categories + 1 intercept) While the formula for SS E is: We can substitute by making into the above equation, and obtain SS E = y y - β X y Properties of the Least Squares Estimators The properties of the least squares estimators can be found with certain assumptions on the error terms. We assume that the errors ε i are statistically independent with mean zero and variance σ 2. With these assumptions, the least squares estimators are unbiased estimators of the regression coefficients. This property is shown like this: Norice we assumed that E(ε) = 0 and used (X X) -1 X X = (the identity matrix = ). hen β is an unbiased estimator of β. he variances of the β 's are expressed in terms of the inverse of the X X 7

8 matrix, then the inverse X X multiplied by σ 2 gives the covariance matrix of the regression coefficients. The covariance matrix looks like this if there are 2 regressors: C =(X X) -1 = Then we can see that C 10 = C 01, C 20 = C 02, and C 12 = C 21 all equal each other because (X X) -1 is symmetric. Hence we have: Normally the covariance matrix is (p x p) that is symmetric and j,j th element is the variance of β and the i,j th element is the covariance between β i and β j : o obtain the estimates of the variances of these regression coefficients, we replace σ 2 with an estimate. (σ 2 ). he square root of the estimated variance of the jth regression coefficient is known as the estimated standard error of β j or se(β j) = σ2. These standard errors measure the precision of estimation for the regression coefficients. Small standard error means that there is a good precision. Test for Significance of Regression The test for a significance of regression is a test to check if there is a linear relationship between the response variable y and the regressor variables x 1, x 2,..., x k. The hypothesis used is By rejecting the null hypothesis, we can assume that at least one regressor variable contributes significantly to the model. Just like in simple linear regression there is a similar formula which is used that is applied in more general cases. First the total sum of Squares SS T is separated/partitioned into a sum of squares due to the model and the sum of squares due to the error. 8

9 Now if the null hypothesis is true, then SS R /σ 2 is a chi-squared random variable with the number of regressors equal to the degrees of freedom. We can also show that SS E /σ 2 is a chi-squared random variable with observation - parameter (n-p) degrees of freedom. The test statistic for H 0 : β 1 = β 2 = = β k = 0 is and it follows the F-distribution. We would reject H o if the computed f o is greater than f α,h,n-p. Usually the procedure is shown is summarized in an analysis of variance table like this one. Analysis of Variance for Testing Significance of Regression in Multiple Regression Source of Variation Sum of Squares Degrees of Freedom Mean Square F 0 Regression SS R k MS R MS R /MS E Error or residual SS E n - p MS E Total SS T n - 1 Since SS T is and we can write that SS E is or Therefore SS R (the regression sum of squares) will be 9

10 R 2 and Adjusted R 2 We can also use the equation from the simple linear regression model for the coefficient of determination R 2 in the general model for multiple linear regression. The R 2 statistic is used to evaluate the fit of the model. When working with multiple linear regression, many people like to use adjusted R 2 because SS E /(n-p) is the error or the residual mean squared and SS T /(n-1) is a constant. R 2 will only increase if a variable is added to a model and so we consider The adjusted R 2 statistic penalizes the analyst for adding terms to the model. This was helps guard against overfitting, which is including regressors that aren't useful. R 2 adj will be used when we look at variable selection. Now if we add a regression variable to the model, the sum of the squares will always increase while the error sum of squares will decrease. (R 2 will always increase) So adding an unimportant variable will cause R 2 to increase, so we look at R 2 adj instead because it's a better fit. Whereas R 2 adj only increase if the variable added to the model will reduce the error mean square reduces. Tests On Individual Regression Coefficients and Subsets of Coefficients We can test hypothesis on individual regression coefficients and these tests will determine the potential value of each regressor variables in the regression model. This will help make the model more effective by being able to deleting some variables and adding others. he hypothesis to test if an individual regression coefficient β j equals a value β j0 is And the test statistic for this hypothesis is 10

11 where C jj is the diagonal element of (X X) -1 which corresponds to β j. he denominator of the test statistic is the standard error of the regression coefficient β j. The null hypothesis H o : β j = β jo is rejected if This is known as the partial or marginal test because the regression coefficient β j depends on all the other regressor models x i (i j). A special case where H o : β jo = 0 is not rejected, this means that the regressor x j can be deleted. Partial F Test Where 0 means a vector of zeroes and β 1 is a subset of the regression coefficients. Now the model can written as if there are 2 regressor variables: X 1 represents the columns of X associated to β 1, and X 2 represents the columns associated to β 2. For the full model with both β 1 and β 2 we know that β = (X X) -1 X y. he regression sum of squares for all variables (with the intercept) is and The regression sum of squares of β 1 when β 2 is in the model is The sum of squares shown above has r degrees of freedom and it is called the extra sum of squares due to β 1. SS R (β 1 β 2 ) is the increase in the regression sum of squares by including the variables x 1, x 2,..., x r in the model. The null hypothesis β 1 = 0 and the test statistic is This is called the partial F-test and if f o > f α,r,n-p then we reject H 0 and conclude that at least one of the parameters in β 1 is not zero. This means that one of the variables x 1, x 2,..., x r in X 1 contributes significantly. The partial F-test can measure the contribution of each individual regressor in the model as if it was the last variable added. 11

12 This is the increase in the regression sum of squares caused by adding x j to the model that already includes x 1,..., x j-1, x j+1,..., x k. The F-test can measure the effect of sets of variables. Confidence intervals on Individual Regression Coefficients A 100(1-α)% confidence interval on the regression coefficient β j, j = 0,1,..., k in a multiple linear regression model is We can also write it this way Because is the standard error of the regression coefficient β j. We use the t-score in the confidence interval because the observations Y i are independently distributed with mean and variance σ 2. ince the least squares estimator β j is a linear combination of the observations, it follows that β j is normally distributed with mean vector β and covariance matrix. C jj is the jjth element of the (X X) -1 matrix, and is the estimate of the error variance. Confidence Interval on the Mean Response We can also get the a confidence interval on the mean response at a particular point (x 01, x 02,..., x ok ). We need to define the vector The mean response is E(Y x 0 ) = μ Y x0 = x o β, estimated by The variance is 12

13 And the 100(1-α)% confidence interval is constructed from the following variable is t-distributed: T= The 100(1-α)% confidence interval on the mean response at the point (x 01, x 02,..., x ok ) is Prediction of New Observations Say x 01, x 02,..., x ok, we can predict future observations on the response variable Y. f x 0 = [1, x 01, x 02,..., x ok ], a point estimation of the future estimation Y 0 at the point x 01, x 02,..., x ok is The 100(1-α)% prediction interval for the future observation is This is a general prediction interval and the prediction interval will always be wider than the mean interval because of the addition of 1 in the radical. There is a larger error in estimating the prediction interval than the error interval. Residual Analysis The residuals defined by help judge the model accuracy. By plotting the residuals versus other variables that are excluded but might be a factor because they are possible candidates. This model can show if variables can be improved when adding the candidate variable. 13

14 Standardized Residual can be useful when assessing the magnitude. Some people like to use the standardized residuals can be scaled so that their standard deviation is unity. Then there is studentized residual where h ii is the ith diagonal element of the matrix The H matrix is called the "hat" matrix, since Thus H transforms the observed values of y into a vector of fitted values. Since each row of the matrix X corresponds to a vector, the diagonal elements of the hat matrix is, another way to write h ii is the variance of the fitted. Under the belief that the model errors are independently distributed with mean zero and variance σ 2, we depict that the variance of the ith residual e i is This means that the h ii elements must fall in the interval 0 < h ii 1. This implies that the standardized residuals understate the true residual magnitude; thus, the studentized residuals would be better used to examine potential outliers. 14

15 Influential Observations Their maybe points or variables that is different and remote from the rest of the data. These points can be influential in determining R 2, estimating the regression coefficients and the magnitude of the error mean square. By measuring the distance we can detect if the points are influential. We measure the squared distance between the least squares estimate of β based on all n observations and the estimate obtained when the ith point is removed, say, We use Cook's distance If the ith point is influential, its removal will result in changing considerably from the value. A large value of D i means that the ith point is influential. The statistic D i is actually computed using In the cook's distance formula, D i consists of the squared studentized residual which shows how well the model fits the ith observation y i A value of D i > 1 would indicate that the point is influential. A component of D i (or both) may contribute to a large value. Polynomial Regression Models This is the second-degree polynomial in one variable. and the second-degree polynomial with two variables. They are both linear regression models. Polynomial regression models are used when the response is curvilinear. The general principles of multiple linear regression still apply. Categorical Regressors and Indicator Variables Categorical regressors are when we take into account qualitative variables instead of quantitative variables. To define the different levels of the qualitative variables, we would use numerical indicator variables. For example, 15

16 if the color red, blue and green where some kind of qualitative variables, then we can indicate 0 for red, 1 for blue and 2 for green. A qualitative variable with r-levels can also be shown with r - 1 indicator variables, which are assigned the value of either zero or one Selection Of Variables in Model Building All the models would have a intercept β 0 so we would have K+1 terms. But the problem is trying the figure out which variables is the right variable to choose for inclusion in the model. Preferably we would like a model to use only a few regressor variables but we don't want to remove any important regression variables. This can help us with predictions. One criterion that is used to evaluate and compare the regression models are the R 2 and R 2 adj. The analyst would increase the variables until the increase to the R 2 or R 2 adj is small. Often the R 2 adj will stabilize and decrease when we add variables to the model. The model that maximises the R 2 adj is the good candidate for the best regression equation. The value that maximizes the R 2 adj also minimizes the mean squared error. Another criterion is the C p statistic and this measures the total mean square for the regression model. The total standardized mean square error is We use the mean square error from the full K + 1 term model as an estimate of σ 2 ; that is,. The estimator of Γ p is C p statistic: If there is a bias in the p-term then: 16

17 The values of C p for each regression model under consideration should be evaluated to p. The regression equations that have negligible bias will have values of C p that are close to p, while those with significant bias will have values of C p that are significantly greater than p. We then choose as the best regression equation either a model with minimum C p or a model with a slightly larger C p. Prediction Error Sum of Squares( PRESS) statistic is another way to evaluate competing regression models, and it is defined as the sum of the squares of the differences between each observation y i and the corresponding predicted value based on a model fit to the remaining n - 1 points,. PRESS gives a measure of how well the model is likely to perform when predicting new data or data that was not used to fit the regression model. The formula for PRESS is Models with small values of PRESS are preferred. Stepwise Regression This procedure constructs a regression model by adding or deleting variables at each step. The criterion to add and remove variables is using the partial F-Test. Let f in be the value of the F-random variable for adding a variable to the model, and let f out be the value of the F-random variable for deleting a variable from the model. We must have f in f out, and usually f in = f out. Stepwise regression starts by making a one-variable model using the regressor variable that has the highest correlation with the variable Y. This regressor will produce the largest F-statistic. If the calculated value f 1 < f out, the variable x 1 is removed. If not, we keep the variable and we do the next test with a new variable and each variable that has been kept. At each step the set of remaining candidate regressors is examined, and the regressor with the largest partial F- statistic is entered if the observed value of f exceeds f in. Then the partial F-statistic for each regressor in the model is calculated, and the regressor with the smallest observed value of F is deleted if the observed f < f out. The procedure continues until no other regressors can be added to or removed from the model. Forward Selection 17

18 This procedure is a variation of stepwise regression and we just add a regressor to the model one at a time until there are no remaining candidate regressors that produce a significant increase in the regression sum of squares.( Variables are added one at a time as long as their partial F-value exceeds f in )Forward selection is a simplification of stepwise regression that doesn't use the partial F-test for removing variables from the model that have been added at previous steps. This is a potential weakness of forward selection because we don't check the previous variables added. Backward Elimination This begins with all K candidate regressors in the model. Then the regressor with the smallest partial F-statistic is deleted if this F-statistic is insignificant, that is, if f < f out. Next, the model with K - 1 regressors is fit, and the next regressor for potential elimination is found. The algorithm terminates when no further regressor can be deleted. (This technique will be used later on in my data.) Multicollinearity Normally, we expect to find dependencies between the response variable Y and the regressors x j. But, we can also find that there are dependencies between the regressor variables x j. In situations where these dependencies are strong, we say that multicollinearity exists. Effects of multicollinearity can be evaluated. The diagonal elements of the matrix C = (X X) -1 can be written as is the coefficient of multiple determination resulting from regressing x j on the other k - 1 regressor variables. We can think of as a measure of the correlation between x j and the other regressors. The stronger the linear dependency of x j on the remaining regressor variables, and the stronger the multicollinearity, the larger the value of will be. Recall that Therefore, we say that the variance of is inflated by the quantity. Consequently, we define the variance inflation factor for β j as If the columns of the model matrix X are perpendicular, then the regressors are completely uncorrelated, and the variance inflation factors will all be unity. VIF that exceeds indicates some level of multicollinearity. 18

19 If VIF exceeds 10, then multicollinearity is a problem. Another way to see if multicollinearity is present is when the F-test for significance of regression is significant, but the tests on the individual regression coefficients are not significant, then we may have multicollinearity. Doing more observations and maybe deleting some variables can decrease the levels of multicollinearity. Data/Analysis: we will use. Now that we finished summarizing multiple linear regression, were going to look over the data which NHL Stats of 30 Teams W Gf Ga AdV PPGF PCTG PEN BM AVG SHT PPGA PKPCT SHGF SHGA FG

20 ( STATISTICS GATHERED ON NHL.COM) Before going through many different calculations, we should first understand what each category mean. W(Y) = WINS GF(x 1 ) = Goals For GA(x 2 ) = Goals Against ADV(x 3 ) = Total Advantage. Power-play opportunities PPGF(x 4 ) = Power-play Goals For PCTG(x 5 ) = Power-play Percentage. Power-play Goals For Divided by Total Advantages PEN(x 6 ) = Total Penalty Minutes Including Bench Minors BMI(x 7 ) = Total Bench Minor Minutes AVG(x 8 ) = Average Penalty Minutes Per Game SHT(x 9 ) = Total Times Short-handed. Measures Opponent Opportunities PPGA(x 10 ) = Power-play Goals Against PKPCT(x 11 ) = Penalty Killing Percentage. Measures a Team's Ability to Prevent Goals While its Opponent is on a Power-play. Opponent Opportunities Minus Power-play Goals Divided by Opponents' Opportunities SHGF(x 12 ) = Short-handed Goals For SHGA(x 13 ) = Short-handed Goals Against FG(x 14 ) = Games Scored First With this data, I will investigate a multiple linear regression model with the response variable Y being wins and the other variables will be my regressor variables. To make a good model, I will use the Backward Elimination, by first placing all my regressor variables in Minitab and removing the variables whose individual tests for significance show p-values that are greater than The variables with highest p-values will be removed one at a time until there are no more p-values greater than The highlighted variables are the ones being removed in the next trial. (Trial 1).Regression Analysis: W versus Gf, GA,... The regression equation is W = Gf GA ADV PPGF PCTG PEN BMI AVG SHT PPGA PKPCT SHGF SHGA FG Predictor Coef SE Coef T P 20

21 Constant Gf GA ADV PPGF PCTG PEN BMI AVG SHT PPGA PKPCT SHGF SHGA FG S = R-Sq = 93.7% R-Sq(adj) = 87.8% Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV PPGF PCTG PEN BMI AVG SHT PPGA PKPCT SHGF SHGA FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI ( , ) ( , )XX XX denotes a point that is an extreme outlier in the predictors. Values of Predictors for New Observations 21

22 New Obs Gf GA ADV PPGF PCTG PEN BMI AVG SHT PPGA PKPCT SHGF New Obs SHGA FG (Trial 2) Regression Analysis: W versus Gf, GA,... The regression equation is W = Gf GA ADV PPGF PCTG PEN BMI AVG SHT PPGA PKPCT SHGA FG Predictor Coef SE Coef T P Constant Gf GA ADV PPGF PCTG PEN BMI AVG SHT PPGA PKPCT SHGA FG S = R-Sq = 93.7% R-Sq(adj) = 88.6% R 2 adj Increased, which means that the model takes into account 88.6% of the data and the error mean squared decreased. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV PPGF PCTG PEN BMI AVG SHT PPGA PKPCT SHGA FG Unusual Observations 22

23 Obs Gf W Fit SE Fit Residual St Resid R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI ( , ) ( , )XX XX denotes a point that is an extreme outlier in the predictors. Values of Predictors for New Observations New Obs Gf GA ADV PPGF PCTG PEN BMI AVG SHT PPGA PKPCT SHGA New Obs FG (Trial 3) Regression Analysis: W versus Gf, GA,... The regression equation is W = Gf GA ADV PCTG PEN BMI AVG SHT PPGA PKPCT SHGA FG Predictor Coef SE Coef T P Constant Gf GA ADV PCTG PEN BMI AVG SHT PPGA PKPCT SHGA FG S = R-Sq = 93.7% R-Sq(adj) = 89.2% R 2 adj increased, the model takes into account 89.2% of the data and the error mean square was reduced. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS 23

24 Gf GA ADV PCTG PEN BMI AVG SHT PPGA PKPCT SHGA FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI ( , ) ( , )XX XX denotes a point that is an extreme outlier in the predictors. Values of Predictors for New Observations New Obs Gf GA ADV PCTG PEN BMI AVG SHT PPGA PKPCT SHGA FG (Trial 4)Regression Analysis: W versus Gf, GA,... The regression equation is W = Gf GA ADV PCTG PEN AVG SHT PPGA PKPCT SHGA FG Predictor Coef SE Coef T P Constant Gf GA ADV PCTG PEN AVG SHT PPGA PKPCT SHGA FG S = R-Sq = 93.5% R-Sq(adj) = 89.5% R 2 adj increased, the model takes into account 89.5% of the data and the error mean square was reduced. 24

25 Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV PCTG PEN AVG SHT PPGA PKPCT SHGA FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI ( , ) ( , )XX XX denotes a point that is an extreme outlier in the predictors. Values of Predictors for New Observations New Obs Gf GA ADV PCTG PEN AVG SHT PPGA PKPCT SHGA FG (Trial 5)Regression Analysis: W versus Gf, GA,... The regression equation is W = Gf GA ADV PEN AVG SHT PPGA PKPCT SHGA FG Predictor Coef SE Coef T P Constant Gf GA ADV PEN AVG SHT PPGA

26 PKPCT SHGA FG S = R-Sq = 93.1% R-Sq(adj) = 89.5% R 2 adj stayed the same, the model still takes into account 89.5% of the data and the error mean square did not change. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV PEN AVG SHT PPGA PKPCT SHGA FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI ( , ) ( , )XX XX denotes a point that is an extreme outlier in the predictors. Values of Predictors for New Observations New Obs Gf GA ADV PEN AVG SHT PPGA PKPCT SHGA FG (Trial 6) Regression Analysis: W versus Gf, GA,... The regression equation is W = Gf GA ADV PEN SHT PPGA PKPCT SHGA FG Predictor Coef SE Coef T P Constant

27 Gf GA ADV PEN SHT PPGA PKPCT SHGA FG S = R-Sq = 92.1% R-Sq(adj) = 88.6% R 2 adj decreased by a little which is alright, the model takes into account 88.6% of the data and the error mean square has increased while there are less insignificant regressors. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV PEN SHT PPGA PKPCT SHGA FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI (73.819, ) (73.770, )XX XX denotes a point that is an extreme outlier in the predictors. Values of Predictors for New Observations New Obs Gf GA ADV PEN SHT PPGA PKPCT SHGA FG (Trial 7)Regression Analysis: W versus Gf, GA, ADV, SHT, PPGA, PKPCT, SHGA, FG 27

28 The regression equation is W = Gf GA ADV SHT PPGA PKPCT SHGA FG Predictor Coef SE Coef T P Constant Gf GA ADV SHT PPGA PKPCT SHGA FG S = R-Sq = 91.7% R-Sq(adj) = 88.5% R 2 adj decreased by a little which is alright, so did the R 2 (R 2 will always decrease if you remove a regressor) the model takes into account 88.5% of the data and the error mean squared has increased while there are less insignificant regressors. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV SHT PPGA PKPCT SHGA FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI (55.800, ) (55.746, )XX XX denotes a point that is an extreme outlier in the predictors. Values of Predictors for New Observations New Obs Gf GA ADV SHT PPGA PKPCT SHGA FG

29 (Trial 8)Regression Analysis: W versus Gf, GA, ADV, SHT, PPGA, PKPCT, FG The regression equation is W = Gf GA ADV SHT PPGA PKPCT FG Predictor Coef SE Coef T P Constant Gf GA ADV SHT PPGA PKPCT FG S = R-Sq = 91.0% R-Sq(adj) = 88.1% R 2 adj decreased by a little which is alright, the model takes into account 88.1% of the data and the error mean square has increased while there are less insignificant regressors. R 2 also decreased, but that is normal. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV SHT PPGA PKPCT FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI (33.499, ) (33.439, )XX XX denotes a point that is an extreme outlier in the predictors. 29

30 Values of Predictors for New Observations New Obs Gf GA ADV SHT PPGA PKPCT FG (Trial 9)Regression Analysis: W versus Gf, GA, ADV, SHT, PPGA, FG The regression equation is W = Gf GA ADV SHT PPGA FG Predictor Coef SE Coef T P Constant Gf GA ADV SHT PPGA FG S = R-Sq = 89.9% R-Sq(adj) = 87.2% R 2 adj decreased by a little which is alright, the model takes into account 87.2% of the data and the error mean square has increased while there are less insignificant regressors. R 2 also decreased, but that is normal. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV SHT PPGA FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI (31.564, ) (31.315, )XX XX denotes a point that is an extreme outlier in the predictors. 30

31 Values of Predictors for New Observations New Obs Gf GA ADV SHT PPGA FG (Trial 10)Regression Analysis: W versus Gf, GA, ADV, PPGA, FG The regression equation is W = Gf GA ADV PPGA FG Predictor Coef SE Coef T P Constant Gf GA ADV PPGA FG S = R-Sq = 88.5% R-Sq(adj) = 86.1% R 2 adj decreased by a little which is alright, the model takes into account 86.1% of the data and the error mean square has increased while there are less insignificant regressors. R 2 also decreased, but that is normal. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV PPGA FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI (21.457, ) (21.134, )XX XX denotes a point that is an extreme outlier in the predictors. 31

32 Values of Predictors for New Observations New Obs Gf GA ADV PPGA FG *(Trial 11)Regression Analysis: W versus Gf, GA, ADV, FG* The regression equation is W = Gf GA ADV FG Predictor Coef SE Coef T P Constant Gf GA ADV FG S = R-Sq = 87.2% R-Sq(adj) = 85.2% R 2 adj decreased by a little which is alright, the model takes into account 85.2% of the data and the error mean square has increased while there are less insignificant regressors. R 2 also decreased, but that is normal. Analysis of Variance Source DF SS MS F P Regression Residual Error Total Source DF Seq SS Gf GA ADV FG Unusual Observations Obs Gf W Fit SE Fit Residual St Resid R R denotes an observation with a large standardized residual. Predicted Values for New Observations New Obs Fit SE Fit 95% CI 95% PI (25.188, ) (22.070, ) Values of Predictors for New Observations New Obs Gf GA ADV FG

33 *The highlight represents the variables that were removed in the next trial. * Now we will look at the NHL data with the Least Squares Method with my last trial. Calculating data were n=30 and k=4 is too large to be able to do manually, instead I used the program Minitab and got a linear regression model. W = Gf GA ADV FG Some aspects of the model makes sense while others don't: Remember, the 4 regressor variables create a linear function with the response Y (wins). The β 0 is the intercept and it is In practical terms the intercept doesn't really make since because it is saying that if a team shows up to a game and does absolutely nothing, they will finish the season off with 21 wins. The regressor β 1 is the expected change in Y(wins) per unit change in x 1 (GF), if the other variables were held the same. The β 1 is and it makes sense. Goals for (GF) is when your team scores a goal which gives that team a better chance to win. The more goals you score, the better the chance of getting a win. The β 2 is which also makes sense. Goals against (GA) is the amount of goals the other team scores on you, which reduces the chances of winning a game. β 2 is the expected change in wins (Y) per unit change in (Goals against) x 2. The β 3 is and this regressor doesn't make sense. ADV is the amount of power-plays your team has, which is an advantage and can help you win hockey games. This should be a positive regressor because the more power-plays a team has, the greater chance they have of winning a game. Just like the other regressors, this regressor is the expected change in wins per unit change of x 3. Finally, β 4 is and this makes sense. FG is when your team scores first, normally when a team scores first, they take an early lead and is one step closer to winning a game. β 4 is the expected change in wins per unit change when a team scores first. This is a fitted regression which is practical to predict wins for a NHL team given the other regressor variables are. β 0 = 20.9 with p-value of β 1 =0.172 with p-value of β 2 = with p-value of β 3 = with p-value of

34 β 4 =0.365 with p-value of The p-values are acquired by doing the this test The R 2 is 87.2% while the R 2 adj is 85.2%. R 2 shouldn't be considered because with the addition of any variables, the R 2 never decreases even if the errors rise. Many people take into account R 2 adj because it holds a better fit to the model. Now this is not the largest R 2 adj seen. In trial 4 and 5 the R 2 adj was at 89.5%. This means that the model fit for 89.5% of the data and that the model was significant but the regressor variables had a p-value larger than 0.05 which made the regressor variables not significant. Which lead to the R 2 adj to be 85.2%. 85.2% is still significant and it fits for 85% of the data. R 2 adj is better because it guards for over fitting while R 2 causes it to over fit when adding a not so useful variable. We can say that R 2 adj penalizes the analyst for adding terms to the model. y es mated ariance σ 2 ) is For the prediction inter al when α = 0.05 Gf = 130, Ga =114, ADV = 155, FG = 23) is <Y o < with 95% confidence The 95% confidence interval for the mean response (Gf = 130, Ga =114, ADV = 155, FG = 23) is < μ Y x0 < with 95% confidence While the residual is , the standard residual is and the Y i =18 Work Cited 34