Introduction to Regression

Transcription

1 Introduction to Regression ιατµηµατικό Πρόγραµµα Μεταπτυχιακών Σπουδών Τεχνο-Οικονοµικά Συστήµατα ηµήτρης Φουσκάκης

2 Introduction Basic idea: Use data to identify relationships among variables and use these relationships to make predictions. Regression analysis describes the relationship between two (or more) variables. Examples: Income and educational level. Demand for electricity and the weather. Home sales and interest rates.

3 Simple Example A linear model for hours worked: Hours worked = a + b*per-capita GDP Where: Hours of work: dependent variable (Y) GDP per-capita: independent variable (X) a : intercept (or baseline), b: slope are the regression coefficients

4 Simple Example The slope of this line gives: b = Change in Hours Worked Change in GDP per - capita If b>0, hours worked increase with the level of income. If b<0, the work week gets shorter as a country develops.

5 Simple Example We want to find coefficient values that give a good fit of the data Plot of the data is called a scatter diagram It describes the relationship between Hours Worked and GDP per-capita for several countries

6 Scatter Diagram: Hours Worked and GDP per Capita Weekly Hours Worked ,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 GDP per capita

7 So Many Choices... 55,0 50,0 Weekly Hours Worked 45,0 40,0 Line C 35,0 30, GDP per capita

8 Simple Example The regression line is the line that best summarizes the data. More precisely, it s s the line that minimizes the distance between every point in the scatter diagram and the corresponding point in the line. This method of estimating the regression line is called least squares.

9 Scatter Diagram and Regression Line Weekly Hours Worked ,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 GDP per capita

10 Simple Example In our example the regression line is: Hours Worked = Per capita GDP (1.52) ( ) A $1,000 increase in GDP per-capita reduces by a quarter of an hour The standard errors (in parenthesis) are a measure of the statistical precision with which the coefficients are estimated

11 Predicting Sales from Advertising Expenditures Table 1: Advertising expenditures and first year sales of AppleGlo by region. Advertising Expenditures ($million) (x i ) First-Year Sales ($million) (y i )

12 Scatter Plot of first year sales and advertising expenditures First Year Sales ($ million) Advertising Expenditures ($ million)

13 Notation n = 14 observations Y = First Year Sales ($ million) x = Advertising Expenditures ($ million) Try to fit a simple linear regression model: Y = β 0 + β 1 x + ε noise, a Normally distributed random variable with mean 0 and standard deviation σ ( we estimate it

14 Estimation If b o and b 1 are the estimates of the intercept β 0 and the slope β 1, their values are called the regression coefficients. We want to choose b o and b 1 in such a way that the line is the best fit for the data observations (x i, y i ), i = 1,.., 14.

15 Estimation If we have trial values of b o and b 1, then the estimated or predicted values of the dependent variables are: y = b + bx ˆi 0 1 i

16 Estimation The difference then between the observed values of the dependent variable and the predicted values are called residuals and are: e = y yˆ = y b bx i i i i 0 1 i

17 Estimation Our goal is to select values of b o and b 1 in such a way that the residuals are as small as n n 2 2 possible min ( e) = min ( y yˆ ) Least Squares Method i i i i= 1 i= 1 b 1 = n i = 1 ( x x)( y y) n i i = 1 ( x x) i i 2 b0 = y bx 1

18 Estimation Advertising expenditures ($ million) Fitted values First Year Sales ($ million) Fitted values/first Year Sales ($ million)

19 Computer Output Source SS df MS Number of obs = F( 1, 12) = Model Prob > F = Residual R-squared = Adj R-squared = Total Root MSE = expenditures Coef. Std. Err. t P> t [95% Conf. Interval] adv_exp _cons S A line can be drawn Coefficient of Determination Attempt to take into account the sampling Regression Coefficients Y = x P-values Confidence Intervals for the coefficients

20 Coefficient of Determination R 2 is the proportion of the total variation of the observed values of the dependent variable Y that is accounted for by the regression equation of the independent variables. Always 0 R 2 1. Closer to 1 means that the points lie closer to the straight line. That s s why the value of R 2 is frequently used to measure the extent to which the regression model fits the data. This is WRONG!!!!!!! There are other ways to determine whether a linear regression is valid or not.

21 Coefficient of Determination Fitted values/var var5 Fitted values var6 R 2 = Fitted values/advertising expenditures ($ million) R 2 = var4 Fitted values Advertising expenditures ($ million)

22 Coefficient of Determination and Sample Correlation It can be proved that R 2 = [corr[ (x,y) ] 2 Thus R 2 is the square of the sample correlation coefficient between the independent variable y and the dependent variable x. The estimate of the slope b 1 in the simple linear regression model can be written as S x b1 = corr( x, y) S y where S X and S Y are the sample standard deviations of x, y respectively.

23 Multiple Regression Simple Linear Regression is a model to predict the value of one variable from another. Multiple Regression is a natural extension of this model: We use it to predict values of an outcome from several predictors.

24 Predicting Sales of a product based on Multiple Factors Table: Sales of Nature-Bar, advertising expenditures, promotion expenditures, and competitors sales, by region, for Region Sales ($million) Y i Advertising Expenditures ($million) X 1i Promotions Expenditures ($million) X 2i Competitors Sales ($million) X 3i Selkirk Susquehanna Kittery Acton Finger Lakes Berkshire Central Providence Nashua Dunster Endicott Five-Towns Waldeboro Jackson Stowe

25 Predicting Sales of a product based on Multiple Factors Y: dependent variable sales of nature bar k = 3 independent variables x 1 = advertising expenditures x 2 = promotional expenditures x 3 = competitors sales n = 15 number of observations

26 Predicting Sales of a product based on Multiple Factors Y i = β 0 + β 1 x 1i + β 2 x 2i + β 3 x 3i + ε i, with i = 1,,n,n and ε i are observed values of independent Normally distributed random variables with mean 0 and standard deviation σ. β 0 : baseline; the value of Y when x 1 =x 2 =x 3 =0 β 1, β 2, β 3 denotes the change in Y per unit change in each x 1, x 2, x 3 respectively

27 Predicting Sales of a product based on Multiple Factors Y = β 0 + β 1 x 1 + β 2 x 2 + β 3 x 3 + ε, where ε ~Ν(0, (0,σ 2 ) Ε(Υ x 1,x 2,x 3 ) = β 0 + β 1 x 1 + β 2 x 2 + β 3 x 3 and Standard Deviation (Υ ( x 1,x 2,x 3 ) = σ Does not depend on Does not depend on x 1,x 2,x 3

28 Predicting Sales of a product based on Multiple Factors Let b 0, b 1, b 2, b 3 be the estimates of β 0, β 1, β 2, β 3. The predicted values then are: ˆ i = i i i y b b x b x b x The residuals are: e = y yˆ = y b bx b x b x i i i i 0 1 1i 2 2i 3 3i

29 Predicting Sales of a product based on Multiple Factors The residual sum of squares is: n n n () e ˆ i = ( yi yi) = ( yi b0 bx 1 1i bx 2 2i bx 3 3i) i= 1 i= 1 i= 1 The best regression line is the one that chooses b 0, b 1, b 2, b 3 to minimize the above quantity.

30 Predicting Sales of a product based on Multiple Factors With our data it comes out that: Y = x x x 3 Based on the above regression let suppose that we want to predict sales of Nature-Bar for next year in the Nashua region given that we are planning to spend $0.7 million on advertising, $0.6 million on promotions and we estimate that competitors sales will remain flat at their current level of $31.30 million. Y = = $ million

31 Computer output Source SS df MS Number of obs = F( 3, 11) = Model Prob > F = Residual R-squared R = Adj R-squared = Total Root t MSE = sales Coef.. Std. Err. t P> t [95% Conf. Interval] Advertising promotion competitors _cons /23.625=2.53=t, β=0.95, df=15 =15-3-1, 1, c=2.201 from the tables of t For the C.I. we have ( , ) Let c be the number for which P(-c T c) c) = β/100, where T obeys a t-distribution t with (n-k-1) df.. If t >c we are confident at the β% confident level that the coefficient 0

32 Validation Linearity Normality of the ε i Heteroscedasticity Autocorrelation

33 Linearity The dependent variable Y depends linearly on the values of the independent variables x 1, x 2, x 3. When k=1 check that with a scatter plot With k>1 rely on common sense. Check value of R 2 but as discussed before with caution. You might need to add a quadratic term for example if there is a problem with linearity, or transform both the dependent and independent variables.

34 Normality The linear regression model Yi = β 0 + β 1 x 1i + β 2 x 2i + β 3 x 3i assumes that ε i ~Ν(0, (0,σ 2 ). 3i +ε i In order to check that plot a histogram of the regression residuals e = y yˆ = y b + bx + b x + b x i i i i 0 1 1i 2 2i 3 3i Frequency Residuals If there is evidence for no normality, you might need to transform your variables, usually the dependent.

35 Heteroscedasticity The linear regression model Y i = β 0 + β 1 x 1i + β 2 x 2i + β 3 x 3i +ε i assumes that ε i ~Ν(0, (0,σ 2 ),, i.e. all ε i have the same standard deviation. This property is called homoscedasticity. Plot residuals versus the independent variables or versus the fitted values yˆ i and check that there is no pattern. If there is a pattern you need to transform your dependent variable.

36 Heteroscedasticity Residuals advertising_expenditures

37 Heteroscedasticity

38 Autocorrelation The linear regression model Y i = β 0 + β 1 x 1i + β 2 x 2i + β 3 x 3i +ε i assumes that ε i ~Ν(0, (0,σ 2 ),, with ε i independent. The phenomenon of autocorrelation can occur if the assumption of independence is violated. Suppose that the regression model is specified with a time component (data for the last 14 weeks) Plot the residuals in time order of the observations and see if there is any kind of a pattern. If there is such a pattern then incorporate time as one of the independent variables.

39 Autocorrelation

40 Autocorrelation Residuals observation_number

41 Warnings and Issues 1. Overspecification by the addition of too many Independent Variables. Use only the independent variables that make sense. It is true that the more the better, since R 2 cannot be decreased by adding variables, but the simpler your model the better. n 5(k+2) Use stepwise multiple regression (start from the null model and add the best variables at each time until R 2 is quite large, or its increase is too small.

42 Warnings and Issues 2. Extrapolating beyond the Range of the Data. Y = x x x 3 Notice that all of the advertising expenditures (x 1 ) for the regions in the table with the data are between $0.4 and $1.9. The regression model is valid in this range. Thus it would be unwise to use the model to predict sales if we had spend for advertising purposes $10 million.

43 Warnings and Issues 3. Multicollinearity. Two independent variables are highly correlated. Should suspect it if R 2 is high but one or more of the variables does not pass the significant test. Check all correlations before running regression. If multicollinearity occurs, drop one of the independent variables that is highly correlated with another one.

44 Multicollinearity Table:Undergraduate grade point average (GPA), GMAT score and graduate school grade point average (GPA) for 25 MBA students Student Number Undergraduate GMAT GPA Graduate School GPA

45 Multicollinearity Graduate GPA = (Under. GPA) (GMAT) R 2 = Corr(under.. GPA, GMAT) = Not significant Graduate GPA = (Under. GPA) R 2 = significant

46 Outliers Observations that lie outside the overall pattern of the other observations. Observations with large residuals Observations falling far from the regression line while not following the pattern of the relationship apparent in the others

47 Outliers

48 Outliers Outliers can distort the regression results. Therefore many scientists remove them to have a better fitting. But be CAREFUL!!!!!!! Remove outliers only if you are sure that it is a bad data point. Transforming data is one way to soften the impact of outliers since the most commonly used expressions, square roots and logarithms, shrink larger values to a much greater extent than they shrink smaller values. Outliers should be investigated carefully. Often they contain valuable information about the process under investigation or the data gathering and recording process. Before considering the possible elimination of these points from the data, one should try to understand why they appeared and whether it is likely similar values will continue to appear. Of course, outliers are often bad data points.

49 Other Types of Regression Non linear (e.g. add a quadratic term)

50 Other Types of Regression Logistic Regression.. The independent variable Y is binary (common in medical research) Poisson Regression.. The independent variable Y is categorical.

51 Dummy Variables We would like to use linear regression to predict the effect that a particular phenomenon has on the value of the dependant variable, where the phenomenon in question either takes place or not

52 Dummy Variables Table: Annual Repair Costs for 19 vehicles at an automobile dealership Vehicle Age of Vehicle (Years) Automatic Transmission (Yes=1, No=0) Annual Repair Costs ($)

53 Dummy Variables Repair Cost = β 0 + β 1 x 1 + β 2 x 2 + ε, where ε ~Ν(0, (0,σ 2 ) R 2 = Coeff.. St.Err. Intercept Age Automatic Age Dummy Variable (x 2 =1 or 0 depending on weather or not the vehicle has an automatic transm.) Repair Cost = x x 2 Estimate of the additional annual repair cost if you have an automatic transmission