Multiple regression: Model building. Topics. Correlation Matrix. CQMS 202 Business Statistics II Prepared by Moez Hababou

Transcription

1 Multiple regression: Model building CQMS 202 Business Statistics II Prepared by Moez Hababou Topics Forward versus backward model building approach Using the correlation matrix Testing for multicolinearity : Variance inflationary factor Complete model versus reduced model Non-linear models Polynomial models: quadratic or cubic models Exponentional or logarithmic regression models page 2 Correlation Matrix Shows simple correlation relationship among all possible pairs of variables Useful for understanding how the explanatory variables influence the dependent variable Useful for showing the correlation among the explanatory variables page 3 1

2 Example The data for this example was derived from an air pollution study in forty cities. The variables are defined as follows TMR Total mortality rate SMIN, SMEAN, SMAX biweekly sulphate reading, smallest annual, average annual and largest annual respectively PMIN, PMEAN, PMAX biweekly suspended particulate reading, smallest annual, average annual and largest annual page 4 Example GE65 percent of population at least 65 multiplied by 10 NONPOOR percent of families above poverty level PERWH percent of whites in population LPOP logarithm of population PM2 population density page 5 Example CORRELATION MATRIX TMR SMIN SMEAN SMAX page 6 PMIN PMEAN PMAX PM2 GE65 PERWH NON POOR LPOP TMR 1 SMIN SMEAN SMAX PMIN PMEAN PMAX PM GE PERWH NON POOR LPOP

3 Variance inflationary variable Good indicator of the correlation between ONE independent variable Xj and all the other independent variables VIFj=1/(1-R 2 j) R 2 j is the coefficient of determination for the regression model X j =β 0 +β 1 x 1 +β 2 x 2 + +β j-1 x j-1 + β j+1 x j+1 + +β k x k + u page 7 Variance inflationary variable If you find that VIF j is close to 1, this means that variable Xj is not strongly correlated with the others independent variables in the model and can be kept in. If you find that VIF j is higher than 10, this implies that variable Xj is strongly correlated with the others independent variables in the model and should be dropped. page 8 Reduced Models Herein we study reduced models which contain only a subset of the set of possible explanatory variables In order to compare various models in which one is a subset of another we require a statistical test which will indicate whether there has been a loss of explanatory power by reducing the number of explanatory variables The partial F- test is outlined below for this purpose page 9 3

4 Example TMR SMIN PMAX GE65 PERWH NONPOOR page 10 Reduced Models The two models being compared are called the full model and the reduced model The full model contains all p explanatory variables and is given by: y = β 0 +β 1 x 1 + +β q x q +β q+1 x q+1 + +β k x k + u The reduced model eliminates the first q explanatory variables and is given by: y = β 0 + β q+1 x q+1 + +β k x k + u k = number of independent variables in complete model k-q = number of independent variables in reduced model page 11 Reduced Models We wish to test the null hypothesis that the reduced model is as good as the full model for explaining the variation in y Hence we have H 0 : Reduced model as good as Full model Equivalently we are testing that the coefficients for the first q explanatory variables are zero Hence H 0 : β 1 = β 2 = = β q = 0 Note that the order of the variables in the model is arbitrary so we assume the first q page 4

5 Comparing Full and Reduced Models Denote the sums of squares and coefficient of multiple determination for the full model by SST, SSR, SSE and R 2 Denote the reduced model sums of squares and coefficient of multiple determination by SSR R and Note that the total sum of squares remains fixed page 13 Test Statistic For Comparing Full and Reduced Models The test statistic is given by: F = ( SSR SSRR ) q ( ) SSE n k ( R RR ) q 2 ( 1 R ) ( n k 1) F follows a F distribution with q and (n-k-1) d.f. = page 14 Coefficients Standard Error t Stat P-value 95% LL 95% UL VIF Intercept E SMIN PMAX GE E PERWH NON POOR page 15 5

6 Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 40 ANOVA df SS MS F Significance F Regression E-14 Residual Total page 16 H 0 : Reduced model as good as Full model or the extra variables are superfluous H A : Full model superior or at least one of the 6 variables is important Full Model Reduced Model df SS df SS Regression Residual Total Regression Residual Total ( ) 6 F = = Also, page 17 Full Model Regression Statistics R Square Reduced Model Regression Statistics R Square ( ) 6 F = = ( ) 28 Mean of F distribution is usually near 1 Since F is much less than 1 obviously cannot reject H 0 Note: F 0.05,6,28 = 2.45, F 0.01,6,28 = 3.53, F 0.10,6,28 = 2.00 page 18 6

7 Variable Selection Methods The variable mix affects the significance of the explanatory variables in a regression model which leads one to ask how a subset of variables could be selected so that R squared is maximized while the number of significant explanatory variables is as small as possible Techniques that attempt to accomplish this are called variable selection methods or stepwise methods Two simple methods of variable selection are outlined below page 19 Forward Selection The simplest form of variable selection Variables are entered one at a time beginning with no variables in the model Variables added are chosen to maximize the increase in R squared at each step As the R square cannot decrease when variables are added. It is advised to focus more on the incremental in the ADJUSTED R SQUARE. Additions are made as long as change in the ADJUSTED R square is considered significant page 20 Forward Selection Because this method never removes a variable once it has been added and because some variables may never be chosen for entry it is recommended that variables be added far beyond conventional significance levels so that alternative models can be seen Recall that the significance level of some variables can change as a result of the addition of other variables page 21 7

8 A Useful Analogy Consider an employer who wishes to hire college graduates to solve various problems The first graduate hired is the one who can solve the most problems The second graduate hired is the one who can solve the most problems that remain after the first graduate has been hired The second person hired is not necessarily the one who can solve the second largest number of problems page 22 Forward Stepwise Regression Variable R Squared P(Partial F) Intercept GE65 PERWH PMEAN NONPOOR SMIN PMIN SMAX SMEAN PM2 LPOP PMAX GE E PERWH PMEAN NON POOR SMIN PMIN SMAX SMEAN PM LPOP PMAX page 23 Backward Elimination This method is the opposite of the forward method in that the process begins with all variables included in the model The variables are removed in such a way that the reduction in R squared is minimized at each step Note that R squared cannot increase as variables are removed page 24 8

9 Backward Stepwise Regression Variable R Squared P(Partial F) Intercept SMIN SMEAN SMAX PMIN PMEAN PMAX PM2 GE65 PERWH NONPOOR LPOP PMAX LPOP PM PMIN SMEAN SMAX SMIN NON POOR PMEAN PERWH GE E page 25 A Variable Selection Caution VARIABLE SELECTION METHODS ARE NOT A GOOD WAY OF STUDYING RELATIONSHIPS AMONG EXPLANATORY VARIABLES AND A DEPENDENT VARIABLE. IT IS SIMPLY A FISHING EXPEDITION. KEEP IN MIND THAT BECAUSE YOU ARE MAXIMIZING R SQUARED, THERE IS A TENDENCY TO OVER- FIT OR FIT NOISE IN THE SENSE THAT A SECOND SAMPLE MIGHT PRODUCE SOME VERY DIFFERENT RESULTS. page 26 A Variable Selection Caution IF YOU DO WANT TO OBTAIN A GOOD PREDICTIVE MODEL YOU SHOULD USE CROSS VALIDATION METHODS TO VERIFY THE QUALITY OF THE MODEL. CROSS VALIDATION INVOLVES FITTING THE MODEL FOR A PORTION OF THE DATA SET AND THEN TESTING IT ON ANOTHER PORTION OF THE DATA SET. page 27 9

10 A Variable Selection Caution IN CASES WHERE YOU HAVE OVER FITTED, TYPICALLY THE SAME VARIABLES WILL NOT BE SIGNIFICANT IN BOTH SETS OF DATA. ANOTHER APPROACH IS TO TAKE VARIABLES THAT ARE MUTUALLY HIGHLY CORRELATED AND COMBINE THEM INTO FACTORS(E.G. simple average) AND THEN USE THE FACTORS AS EXPLANATORY VARIABLES. page 28 A Variable Selection Caution A MORE SOPHISTICATED VERSION OF THIS IS CALLED FACTOR ANALYSIS WHICH IS HEAVILY USED IN THE SOCIAL SCIENCES OR PRINCIPAL COMPONENTS ANALYSIS WHICH IS HEAVILY USED IN THE PHYSICAL AND BIOLOGICAL SCIENCES. page 29 Nonlinear Models The nonlinear models we study in this section are called intrinsically linear in that they can be fitted using linear regression model methods Although the relationship between the dependent variable and the explanatory variable is nonlinear the model can be expressed as a linear combination of explanatory variables page 30 10

11 Nonlinear Models Polynomial models: quadratic and cubic Models Models With Interaction Growth Models Model with dummy variables Grafted Models page 31 QUADRATIC MODELS The quadratic linear model is given by y = β + β X + X + u 0 1 β2 It can be fitted using multiple regression It is a curvilinear model which has one change in direction Also referred to as a parabola Often in practice there is no change in direction and hence only only one branch of the parabola is used 2 page 32 Quadratic Model An example of a quadratic model is given below Y X page 33 11

12 This example data set relates bank employee salaries to age level and years of education. A variety of fitted models is illustrated below. Because of the natural skewness in the salary variable the logarithm of salary will be used as the dependent variable. page 34 LCURRENT EDUC EDAGE AGE AGE2 EDUC page 35 Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 100 ANOVA df SS MS F Significance F Regression E-19 Residual Total Coefficients Standard Error t Stat P-value 95% LL 95% UL VIF Intercept E EDUC E page 36

13 Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 100 ANOVA df SS MS F Significance F Regression E-21 Residual Total Coefficients Standard Error t Stat P-value 95% LL 95% UL VIF Intercept E EDUC EDUC page 37 Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 100 ANOVA df SS MS F Significance F Regression Residual Total Coefficients Standard Error t Stat P-value 95% LL 95% UL VIF Intercept E AGE page 38 Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 100 ANOVA df SS MS F Significance F Regression E-07 Residual Total Coefficients Standard Error t Stat P-value 95% LL 95% UL VIF Intercept E AGE AGE E page 39 13

14 Models With Interaction Given a regression model with two explanatory variables X and Z, an interaction variable can be defined which is obtained by taking the product of X and Z, given by XZ Adding this interaction variable to the regression model yields a model which defines a nonlinear surface in three dimensional space called a quadratic surface page 40 Fitting a Model With Interaction The presence of the interaction variable allows the response of Y to X to depend on the level of Z and similarly the response of Y to Z to depend on the level of X This can be seen by looking at the model below page 41 Fitting a Model With Interaction Y = β 0 + β 1 X + β 2 Z +β 3 XZ + u = β 0 + [β 1 + β 3 Z]X + β 2 Z + u or = β 0 + β 1 X + [β 2 + β 3 X] Z+ u Since this interaction model is linear in the three terms, linear regression methods can still be applied page 42 14

15 Continuing with the bank employee data used above we study the use of an interaction variable between EDUC and AGE. page 43 Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 100 ANOVA df SS MS F Significance F Regression E-22 Residual Total Coefficients Standard Error t Stat P-value 95% LL 95% UL VIF Intercept E EDUC E EDAGE E AGE E page 44 Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 100 ANOVA df SS MS F Significance F Regression E-22 Residual Total Coefficients Standard Error t Stat P-value 95% LL 95% UL VIF Intercept E EDUC EDAGE AGE AGE EDUC page 45 15

16 10.8 LNSALARY VS EDUC AND AGE LNSALARY 10 AGE= AGE= AGE= AGE= EDUC page 46 LNSALARY VS AGE AND EDUC EDUC=20 LNSALARY EDUC=16 EDUC= EDUC= page 47 AGE Growth Models In business analysis growth projections and analysis are an integral part of planning and performance evaluation It is therefore of interest to be able to fit a constant growth model to a time series such as revenue, costs or profit Fitting a linear model to a time series results in a series which grows by a constant amount each year whereas constant yearly growth requires an exponential model page 48 16

17 Fitting a Seasonal Time Series Using Dummy Variables And Slope Shifters A linear model which allows for variation over the seasons of the year can be constructed using dummy variables and slope shifters This technique will be combined with the constant growth model to study a time series As an example the monthly retail trade series for chain stores in Canada for the period will be studied page 49 Fitting a Seasonal Time Series Using Dummy Variables And Slope Shifters A linear model is fitted to the logarithm of retail trade in order to fit a constant growth model Eleven dummy variables and eleven slope shifter variables are used to represent the eleven months other than December Since December is by far the month with the largest monthly retail sales in any given year it provides a convenient base case page 50 LNSALES TIME D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 TD1 TD2 TD page TD4 TD5 TD6 TD7 TD8 TD9 TD10 TD

18 See ChartSales and ChartLogsales to see time series behavior of SALES Sheet4 has the data for these charts page 52 RETAIL SALES VS TIME SALES page 53 TIME 16.5 LNSALES VS TIME 16 LNSALES page 54 TIME 18

19 See Sheet labeled REGRESSION1 for the fitting of the constant growth model with monthly dummies and slope shifters(red table) This model permits different sales levels for each month as well as different growth rates for each month page 55 RETAIL SALES DATA REGRESSION ON DUMMIES AND SLOPE SHIFTERS Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 216 ANOVA df SS MS F Significance F Regression E-149 Residual Total page 56 Coefficients Standard Error t Stat P-value Intercept E-277 TIME E-64 D E-25 D E-26 D E-17 D E-15 D E-13 D E- D E-15 D E-13 D E- D E-11 D E-06 TD TD page 57 19

20 Coefficients Standard Error t Stat P-value D E-11 D E-06 TD TD TD TD TD TD TD TD TD TD TD page 58 See Sheet labeled REGRESSION2 for the fitting of the constant growth model with monthly dummies but no slope shifters This model permits different sales levels each month but assumes the same growth rate for all months Red table shows details of fit and green table shows fitted values and residuals page 59 RETAIL SALES DATA REGRESSION ON DUMMIES ONLY Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 216 ANOVA df SS MS F Significance F Regression E-164 Residual Total page 60 20

21 Coefficients Standard Error t Stat P-value Intercept E+00 TIME E-168 D E-57 D E-63 D E-45 D E-42 D E-35 D E-35 D E-41 D E-38 D E-37 D E-34 D E-23 page 61 Ln SALES = TIME D D D D D D D D D D D11 December: Ln SALES = TIME SALES = e x e TIME = 1,846,865 ( ) TIME page 62 January: D1 = 1 Ln SALES = TIME SALES = e ( ) TIME February: D2 = 1 Ln SALES = TIME SALES = e ( ) TIME Note how growth rate is constant for each month so annual growth rate is ( ) = or average rate of growth 9.77% per year page 63 21

22 To compare monthly Sales December: Ln SALES = TIME * where TIME * is a particular December January: Ln SALES = (TIME * +1) = TIME * Difference: January - December = = page 64 Ln = lnjan lndec = Jan Dec Jan = e = Dec So January retail sales are expected to be 56.4% of the prior December retail sales page 65 See ChartFittedSeasonality to observe the fitted constant growth model compared to the actual data page 66 22

23 Predicted Sales vs Time Sales page 67 Time 23