QUANTITATIVE STATISTICAL METHODS: REGRESSION AND FORECASTING JOHANNES LEDOLTER VIENNA UNIVERSITY OF ECONOMICS AND BUSINESS ADMINISTRATION SPRING 2013

Transcription

1 QUANTITATIVE STATISTICAL METHODS: REGRESSION AND FORECASTING JOHANNES LEDOLTER VIENNA UNIVERSITY OF ECONOMICS AND BUSINESS ADMINISTRATION SPRING 3 Introduction Objectives of course: Regression and Forecasting Computer software: SPSS and R Data Sets Grades Review of Statistics Probability Distributions Discrete random variables Continuous random variables Describing Data Summary statistics (mean, median, standard deviation, variance, correlation coefficient) Data plots (pie charts, dot diagram, histogram, box plot, scatter diagram) Sampling Issues: Population and samples Random samples Statistical Inference Sampling distributions Central limit effect for averages; proportions Confidence intervals for a population mean; population proportion Statistical tests of hypotheses Comparing means of two independent samples Inference in the blocked experiment: Comparing means of two dependent samples Copyright: Johannes Ledolter, 3

2 Illustration (Review): Results of a survey (Data file: survey) 65 students Number gender height(inch) weight(lb) age Coll-GPA ACT HS-GPA Smoking Drug Drinking * Histogram of height Copyright: Johannes Ledolter, 3

3 Histogram of College GPA 3 Frequency.. 3. CollegeGPA Boxplot of College GPA 4. Mean =.893 Std. Dev. =.5384 N =65 Descriptive Statistics Copyright: Johannes Ledolter, 3 3

4 Descriptive statistics N Minimum Maximum Mean Std. Deviation height weight age CollegeGPA ACT HSGPA Smoking Drug Drinking gender Valid N (listwise) 6 Scatter plot Correlations CollegeGPA ACT HSGPA CollegeGPA Pearson Correlation.84(**).47(**) Sig. (-tailed).. N ACT Pearson Correlation.84(**).388(**) Sig. (-tailed).. N HSGPA Pearson Correlation.47(**).388(**) Sig. (-tailed).. Copyright: Johannes Ledolter, 3 4

5 N ** Correlation is significant at the. level (-tailed). Pie chart for a single categorical variable Cross tabulation of two categorical variables and chi-square test Drug * Drinking Crosstabulation Count Drinking.. 3. Total Drug Total Chi-Square Tests Value df Asymp. Sig. (-sided) Pearson Chi-Square.346(a) 4. Likelihood Ratio Linear-by-Linear Association 3.8. N of Valid Cases 6 a 3 cells (33.3%) have expected count less than 5. The minimum expected count is.4. Copyright: Johannes Ledolter, 3 5

6 Copyright: Johannes Ledolter, 3 6

7 Comparing the mean College GPAs of Men (Group ) and Women (Group ) Group Statistics gender N Mean Std. Deviation Std. Error Mean CollegeGPA Independent Samples Test CollegeGPA Equal variances assumed Equal variances not assumed Levene's Test for Equality of Variances F Sig. t df Sig. (-tailed) t-test for Equality of Means Mean Difference 95% Confidence Interval of the Std. Error Difference Difference Lower Upper Copyright: Johannes Ledolter, 3 7

8 Illustration (Review): UFFI data set (see Chapter in AL) Descriptive Statistics N Range Minimum Maximum Mean Std. Deviation Variance Statistic Statistic Statistic Statistic Statistic Std. Error Statistic Statistic CHO Valid N (listwise) 4 One-Sample Statistics N Mean Std. Deviation Std. Error Mean CHO One-Sample Test CHO Test Value = 95% Confidence Interval of the Mean Difference t df Sig. (-tailed) Difference Lower Upper Copyright: Johannes Ledolter, 3 8

9 groups of houses (UFFI = and UFFI = ) Report CHO UFFI Mean N Std. Deviation Total Independent Samples Test CHO Equal variances assumed Equal variances not assumed Levene's Test for Equality of Variances F Sig. t df Sig. (-tailed) t-test for Equality of Means Mean Difference 95% Confidence Interval of the Std. Error Difference Difference Lower Upper Difference among the group means. Group (UFFI = ) is associated with higher levels Copyright: Johannes Ledolter, 3 9

10 Can we use the information about the house? Scatter plots (overlaid) Copyright: Johannes Ledolter, 3

11 Regression References: Ledolter, J. and Burrill, C. W.: Statistical Quality Control: Strategies and Tools for Continual Improvement, Wiley & Sons, 999 Chapter 6 Abraham, B. and Ledolter, J.: Introduction to Regression Modeling, Duxbury Press, 6 Ledolter, J. and Swersey, A.J.: Testing 3: Experimental Design with Applications in Marketing and Service Operations. Stanford University Press, 7 Many other books Applications: Fuel efficiency as function of weight of the car and characteristics of its engine. CEO Compensation as a function of training and performance Sales as a function of price and advertising Economic relationships Production yield as function of temperature, pressure, catalyst, and rate of throughput. Tensile strength as function of hardness and density of the stock Scatter diagrams and pair-wise correlation coefficients Functional relationship between a response variable (which we denote by Y) and one or more explanatory variables (say p of them, which we denote by X, X,..., X p ). Regression methods are used to model the relationship. Reasons for regression modeling (prediction, what-if,... ) Regression models: Examples Copyright: Johannes Ledolter, 3

12 Copyright: Johannes Ledolter, 3 X Y X X Y X X Y ) ( 3 X X X X Y Meaning of the regression coefficients Building a regression model: Model specification Model estimation Model diagnostic checking

13 The Simple Linear Regression Model Y X Relate response Y to a single explanatory variable X, through a straight line model. car WEIGHT GPM CAR AMC Concord Chevy Caprice Ford Country Squire Wagon. 3.3 Chevette Toyota Corona Ford Mustang Ghia.9 Mazda GLC AMC Sprint.9 3. VW Rabbit Buick Century Y = Fuel efficiency (in gallons per hundred miles, GPM) and X = weight (in, pounds) GPM Weight Meaning of the slope parameter Estimates of the Regression Coefficients Least squares estimates: Select estimates of the regression coefficients such that the sum of the squared vertical distances from the observations to the regression line is minimized. Least squares estimates b and b : Explicit expressions (page 4 of Chapter in AL; equation (.7)) The calculations for the mileage example show Copyright: Johannes Ledolter, 3 3

14 b = (8.49/5.8) =.639 b = (.639)(.9) = Fitted regression line: Yˆ b b X GPM ˆ Weight ˆ Fitted values: ˆ yi b b x i for i =,,..., n Residuals: e i ˆi i i y y y ( b b x ) for i =,,..., n i Use your computer software (SPSS, R) for the calculations. You can store the residuals and the fitted values. Assessing the Regression Model Fit Total sum of squares SST n y y) ( i SST = i When we fit the regression model, we explain the response variable through the fitted linear equation Yˆ b b X Residual sum of squares (also called the error sum of squares): Copyright: Johannes Ledolter, 3 4

15 n n SSE ( y yˆ ) ei i i SSE =.674. i i Regression sum of squares: n SSR ( yˆ y) i SSR = 3.95 i Measures variability among the fitted values. If perfect fit, then SSR = SST. It can be shown that: SST = SSR + SSE Hence it is easier to calculate SSR from SSR = = 3.95 Coefficient of determination R the regression model. measures the proportion of variability that is explained by R SSR SST SSE SST SST SSE SST Proportional reduction in the sum of squares that is due to the regression model. R must be between and. R = says that our model (the simple linear regression model) explains none of the variability in Y. R = says that percent (that is all) of the variation in Y is explained by the simple linear regression model. For the mileage data: SST = 4.589, SSE =.674, and SSR = = R = 3.95/4.589 = percent of the variability in GPM is explained by the variable Weight. Copyright: Johannes Ledolter, 3 5

16 Another (and very similar) way of assessing the importance of the regression is as follows: Sample standard deviation of the response Y, s Y SST ( n ) s. 73 Y How much variability is there in GPM if we adjust for the different weights of the cars? The unexplained variability is measured by the residuals e,..., e n. The standard deviation of the residuals is s e ( e e) /( n ) SSE ( n ) i [In Abraham and Ledolter, we denote this by s.] The quantity s e is an estimate of the variability in the y's, after having accounted for the effect of the explanatory variable X. In the mileage example, s e =.9 gallons per miles. This represents quite a big reduction from the standard deviation s. 73. In fact, it amounts to a 77. percent reduction. Y NOTE: EXCEL calls s e the standard error. The square of s e is also called the mean square error MSE ( ei e) /( n ) SSE ( n ) Standard Errors of Estimates Population and sample. Sampling variability in estimates. Copyright: Johannes Ledolter, 3 6

17 Standard error of estimates: s e.( b ) and s e.( b ) See regression output.. Confidence intervals for unknown regression parameters. Approximate 95 percent confidence interval for is given by the interval [ b () s. e.( b ), b () s. e.( b )] Testing whether regression coefficient is zero. Note that with the explanatory variable X has no influence on the response. The appropriate (standardized) test statistic is given by TS b s. e.( b ) b s. e.( b ) Computer output for our example shows that the standard error of the least squares estimate b =.639 is given by s.e.(b ) =.75. Hence an approximate 95 percent confidence interval for is given by.639 ± ()(.75), or (.384,.894). This interval does not include zero. It provides strong evidence that the regression coefficient is positive; for heavier cars, one needs more fuel. b b.639 The t-ratio for b is TS. 85 ; it is considerably larger than the s. e.( b ) s. e.( b ).75 cutoff. Therefore is definitely not zero. NOTE: Here I have argued with a normal distribution. The correct distribution is a t-distribution with parameter (degrees of freedom) given by n minus number(regression parameters) Copyright: Johannes Ledolter, 3 7

18 Hence, here df = n. Estimating the Mean Response for a given x : Section.5. (page ) Chapter of AL b b x ( t) s e n ( x n i x) ( x x) i Prediction Intervals for individual observations: Chapter.6 in Chapter of AL (page 4) b b x ( t) s e n ( x n i x) ( x x) i EXCEL, SPSS, R: Regression function Model Diagnostic Checking: Plot residuals against explanatory variable Plot residuals against fitted values Durbin-Watson test: Checks for serial correlation in the errors (residuals). DW ( r ) where r is the lag autocorrelation of the residuals. DW 4. We expect a value around. Values considerably smaller than or larger than indicate correlation of adjacent errors. Examples: Example : Hardness in Chapter of AL Example : Sales versus advertising Example 3: Traction Coefficients of 5 Lubricating Oils Example 4: Tensile Strength as a Function of Hardness (or Density) Copyright: Johannes Ledolter, 3 8

19 Copyright: Johannes Ledolter, 3 9

20 Regression applets: Data sets used: car; car; HousePrices; realestate; SalesAdvert; tensile; traction; uffi; NFL Data Copyright: Johannes Ledolter, 3

21 Example : car Scatter plot Model Summary Model R R Square Adjusted R Square Std. Error of the Estimate.977(a) a Predictors: (Constant), Weight ANOVA(b) Model Sum of Squares df Mean Square F Sig. Regression (a) Residual Total Copyright: Johannes Ledolter, 3

22 a Predictors: (Constant), Weight b Dependent Variable: GPM Coefficients(a) Unstandardized Coefficients Standardized Coefficients Model B Std. Error Beta t Sig. (Constant) Weight a Dependent Variable: GPM Copyright: Johannes Ledolter, 3

23 Example : Sales Advertising Scatter plot Model Summary(b) Model R R Square Adjusted R Square Std. Error of the Estimate Durbin-Watson.63(a) a Predictors: (Constant), advert b Dependent Variable: sales Copyright: Johannes Ledolter, 3 3

24 ANOVA(b) Model Sum of Squares df Mean Square F Sig. Regression (a) Residual Total a Predictors: (Constant), advert b Dependent Variable: sales Coefficients(a) Unstandardized Coefficients Standardized Coefficients Model B Std. Error Beta t Sig. (Constant) advert a Dependent Variable: sales Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value Residual Std. Predicted Value Std. Residual a Dependent Variable: sales Copyright: Johannes Ledolter, 3 4

25 MULTIPLE REGRESSION MODEL (CHAPTER 4 IN AL) Y X X... p X p ; p regressor variables Least Squares Estimates of regression coefficients Fitted values and residuals Analysis of variance table: sum of squares, degrees of freedom, mean squares Multiple coefficient of determination R Mean square error (MSE): n MSE ( e i i e) ( n p ) n ( ei ) i ( n p ) SSE /( n p ) s e MSE n i ( e i e) ( n p ) F test for (overall significance of regression) H... Standard errors of regression coefficients: s e.( b ). i : p Confidence intervals for regression coefficients; test statistics and probability values Interpretation of regression coefficients: partial regression coefficients; partial tests. Further issues: Multicollinearity Stepwise regression and best subset regression Model diagnostic checking: Residual plots. Leverage and influence measures Copyright: Johannes Ledolter, 3 5

26 Indicator variables: (Chapter 6 in AL) Model building strategy: Fitting all possible regressions. Plotting MSE against number of variables in the model. Examples: uffi, car, SalesAdvert (with lag sales) Copyright: Johannes Ledolter, 3 6

27 Example : UFFI data. Houses with UFFI = and UFFI = CH O Tight UFFI Model Summary(b) Model R R Square Adjusted R Square Std. Error of the Estimate.885(a) a Predictors: (Constant), UFFI, Tightness b Dependent Variable: CHO ANOVA(b) Model Sum of Squares df Mean Square F Sig. Regression (a) Residual Total a Predictors: (Constant), UFFI, Tightness b Dependent Variable: CHO Coefficients(a) Unstandardized Coefficients Standardized Coefficients Model B Std. Error Beta t Sig. (Constant) Tightness UFFI a Dependent Variable: CHO Residuals Statistics(a) Minimum Maximum Mean Std. Deviation N Predicted Value Residual Std. Predicted Value Std. Residual a Dependent Variable: CHO Next, try the model CH O Tight UFFI 3 ( Tight)( UFFI) And check the significance of the interaction Copyright: Johannes Ledolter, 3 7

28 Example : Sales vs. Advertising (regression) Data set: SalesAdver. Time series data; 36 months Month Sales Advert Copyright: Johannes Ledolter, 3 8

29 Scatter plot: Sales Adver (M) Linear model: Y X Coefficients(a) Unstandardized Coefficients Standardized Coefficients Model B Std. Error Beta t Sig. (Constant) Adver a Dependent Variable: Sales Model Summary(b) Model R R Square Adjusted R Square Std. Error of the Estimate Durbin-Watson Copyright: Johannes Ledolter, 3 9

30 .63(a) a Predictors: (Constant), Adver b Dependent Variable: Sales Note that this model is being fitted on time series data. Durbin-Watson statistic too small. Evidence of positive autocorrelation in errors (this has bad consequences) Autocorrelations of the residuals. Lag is rather high. (M) Improved model: Y t X Y t t Interpretation: Y t ( ) [ X t X t ( ) X t...] error Coefficients(a) Copyright: Johannes Ledolter, 3 3

31 Unstandardized Coefficients Standardized Coefficients Model B Std. Error Beta t Sig. (Constant) Adver LagSales a Dependent Variable: Sales Advertising effect decreases by about 5 percent each period. Model Summary(b) Model R R Square Adjusted R Square Std. Error of the Estimate Durbin-Watson.8(a) a Predictors: (Constant), LagSales, Adver b Dependent Variable: Sales Copyright: Johannes Ledolter, 3 3

32 Example 3: Fuel efficiency. Car data set MPGCity MPGTrip MPGExp Weight Displacement Cylinders HP Trans Gears Size Origin GPMTrip = /MPGTrip Regression on all variables Model Variables Entered/Removed(b) Variables Entered Origin, Gears, HP, Cylinders, Weight, Trans, Displaceme nt(a) Variables Removed a All requested variables entered. b Dependent Variable: GPMTrip. Enter Model Summary Method Model R R Square Adjusted R Square Std. Error of the Estimate.98(a) a Predictors: (Constant), Origin, Gears, HP, Cylinders, Weight, Trans, Displacement Copyright: Johannes Ledolter, 3 3

33 Coefficients(a) Unstandardized Coefficients Standardized Coefficients Model B Std. Error Beta t Sig. (Constant) Weight Displacement Cylinders HP Trans Gears Origin a Dependent Variable: GPMTrip Copyright: Johannes Ledolter, 3 33

34 Backward Elimination of Variables Unstandardized Coefficients Coefficients(a) Standardized Coefficients Model B Std. Error Beta t Sig. (Constant) Weight Displacement Cylinders HP Trans Gears Origin (Constant) Weight Displacement Cylinders Trans Gears Origin (Constant) Weight Displacement Cylinders Gears Origin (Constant) Weight Displacement Gears Origin (Constant) Weight Gears Origin (Constant) Weight Gears (Constant) Weight a Dependent Variable: GPMTrip Copyright: Johannes Ledolter, 3 34

35 Stepwise Regression Model Variables Entered/Removed(a) Variables Entered Variables Removed Weight. Method Stepwise (Criteria: Probability -of-f-toenter <=.5, Probability -of-f-toremove >=.5). a Dependent Variable: GPMTrip Model Summary Model R R Square Adjusted R Square Std. Error of the Estimate.9(a) a Predictors: (Constant), Weight ANOVA(b) Model Sum of Squares df Mean Square F Sig. Regression (a) Residual Total a Predictors: (Constant), Weight b Dependent Variable: GPMTrip Excluded Variables(b) Collinearity Statistics Model Beta In t Sig. Partial Correlation Tolerance Displacement.95(a) Cylinders.(a) HP.37(a) Trans.66(a) Gears -.8(a) Origin.53(a) a Predictors in the Model: (Constant), Weight b Dependent Variable: GPMTrip Copyright: Johannes Ledolter, 3 35

36 Indicator Variables: Size was a categorical variable with five outcomes (subcompact, compact,, van) Method : We create indicator variables. For size (5 possibilities), we create 5 indicator variables: INDS (is if size ; otherwise); INDS (is if size ; otherwise); ; INDS5 (is if size 5; otherwise). And then we add 4 of them to the model. The one that is being left out creates the standard for the comparison. Y Weight INDS 3INDS 4INDS3 5INDS4 Here van becomes the standard. Method : Or, we use Analyze > General Linear Model (with Weight as the covariate, and Size as the fixed factor); and we click parameter estimates under options. Between-Subjects Factors N Size Copyright: Johannes Ledolter, 3 36

37 Dependent Variable: GPMTrip Tests of Between-Subjects Effects Source Type III Sum of Squares df Mean Square F Sig. Corrected Model 3.93(a) Intercept Weight Size Error Total Corrected Total a R Squared =.84 (Adjusted R Squared =.8) Parameter Estimates Dependent Variable: GPMTrip 95% Confidence Interval Parameter B Std. Error t Sig. Lower Bound Upper Bound Intercept Weight [Size=.] [Size=.] [Size=3.] [Size=4.] [Size=5.] (a)..... a This parameter is set to zero because it is redundant. Copyright: Johannes Ledolter, 3 37