MBA Statistics COURSE #4

Transcription

1 MBA Statistics COURSE #4 Simple and multiple linear regression What should be the sales of ice cream?

2 Example: Before beginning building a movie theater, one must estimate the daily number of people entering the building. How can we estimate it? There are 2 millions individuals in the city. 2

3 Possible solutions: One could realize a local market study. However it is often imprecise, specially for new projects. One could get data from similar projects in other cities. 3

4 City Attendance (x1000) What do you think? Can we do better? 4

5 Probably, taking into account the size of the city City Attendance (x1 000) Size (millions)

6 Case study: Ice Cream Sales The file icecream.xls contains pairs of data representing ice cream sales and temperature recorded that day, for 30 days. Is there a relation between temperature and sales? Can temperature be used to predict ice cream sales? If so what s the prediction when the temperature is 25? 6

7 Introduction One of the principle objectives of statistics is to explain the variability that we observe in data. Linear regression (or linear models) is a statistical tool MUCH USED to study the presence of a linear relation between a dependent variable Y (quantitative and continuous) and one or more independent variables X 1, X 2,, X p (qualitative and/or quantitative), called independent or explanatory variables. 7

8 For example, a manager could be interested in seeing if he could explain a good part of the variability that he observes in sales in his differents branches (dependant variable Y) in the last 12 months, by the area, number of employees, number of payed overtime hours, quality of customer service, number of promotions, etc. ( independent or explanatory variables). 8

9 A regression model can be used to answer one of the following three objectives: Describe data coming from non experimental studies i.e. we observe reality as it is. Examine the hypothesis (data coming from controled experimental studies). Predict (if we like to take risks!!). 9

10 Example: We are interested in knowing what are the important factors that influence or determine the value of a property and we want to build a model that would help us evaluate this value using certain factors. To do this, we have obtained the total value for a sample of 79 properties in a given region. The following variables have also been collected for each property: 10

11 Brief glimpse of the data file:house.xls # of square feet total land first outdoor heating OBS value value # of acres floor condition type Good NatGas Good NatGas Good Electric Average Electric Average NatGas Good Electric Excellnt Electric OBS # of # of # of completed # of non completed # of rooms bedroom bathrooms bathrooms fire-places GARAGE Garage NoGarage Garage Garage NoGarage Garage Garage 11

12 Is there a link between the total value and the different factors? Total Land

13 Total Total Acre Sq.Feet Total Total Rooms Bedroom 13

14 Total Total Completed Bathrooms Bathrooms Total Total Fire-place NoGarage Garage Garage 14

15 The Pearson correlation coefficient r is used to measure the intensity of the linear relation between two quantitative variables. The correlation coefficient r will take its values between -1 and 1. If a perfect linear relation exist between X and Y, then r = ±1 (r =1 if X and Y vary in the same direction and r = -1 if X varies in the opposite direction of Y). If r = 0, there is no linear link between X and Y. The more the r value furthers from 0 to get closer to ±1, the more the linear link intensity between X and Y becomes larger. 15

16 Y 6.5 * r = Y r = 1 31 * 6.0 * * 29 * 27 * 25 * 5.5 * * 23 * 21 * 19 * 5.0 * 17 * 15 * 13 * 4.5 * * * 11 * * * X X Y r = * * * * * * * * * * * X 16

17 Descriptive statistics Variable N Mean Median Sta.Deviation Minimum Maximum Total Land Acre Sq.Feet Rooms Bedrooms C.Bathro Bathro Fire-pl Pearson Correlation Coefficients Total Land Acre Sq.Feet Rooms Bedroom C.Bathro Bathro Land Acre Sq.Feet Rooms Bedrooms C.Bathro Bathro Fire-pl

18 BE CAREFULL!! it is important to interpret the correlation coefficient with the graph. r = in all cases below * * * * * * 10.0 * 8 * * * Y1 * Y2 * * 7.5 * * 6 * * * * 5.0 * 4 * * X X 15.0 Y * 12.5 * Y * * * * * 7.5 * 7.5 * * * * * * * * * * * * X X

19 Simple linear regression To describe a linear relation between two quantitative variables or to be able to predict Y for a given value of X, we use a regression line: Y = β 0 + β 1 X + ε Since any statistical model is only an approximation (we hope the best possible!!) and because the linear link is never perfect, in the model, there is always an error, noted ε. If there was a perfect linear relation between Y and X, the error term would always be equal to 0, and all the variability of Y would be explained by the independent variable X. 19

20 So, for a given value of X, we would like to estimate Y. Thus, with the help of the data sample we will estimate the regression model parameters β 0 and β 1 in order to minimize the residuals (errors) sum of squares. The squared correlation coefficient is called the coefficient of determination and the percentage of the variability of Y explained by X: R 2 = 1 - (n-2)/(n-1){s e /S y } 2, where S e is the standard deviation of the errors and S y is the standard deviation of Y. 20

21 We can also use the adjusted coefficient of determination to indicate the percentage of the variability of Y explained by X: R 2 ajusted = 1 - {S e /S y }2. 21

22 Simple linear regression example: MODEL 1. Regression Analysis The regression equation is Total = Sq.Feet Predictor Coef StDev T P Constant Sq.Feet S = R-Sq = 58.8% R-Sq(adj) = 58.2% Analysis of Variance Source DF SS MS F P Regression E E Residual Error E Total E+11 22

23 MODEL 2. The regression equation is : Total = Rooms Predictor Coef StDev T P Constant Rooms S = R-Sq = 39.3% R-Sq(adj) = 38.5% Analysis of Variance Source DF SS MS F P Regression E E Residual Error E Total E+11 MODEL 3. The regression equation is : Total = Bedrooms Predictor Coef StDev T P Constant Bedrooms S = R-Sq = 33.9% R-Sq(adj) = 33.1% Analysis of Variance Source DF SS MS F P Regression E E Residual Error E Total E+11 23

24 Model 1: total value = *( # of squared feet ). R 2 = 58.8%. Thus 58.8% of the variability of the total value is explained by the # of squared feet. Model 2: total value = *(# of rooms ). R 2 = 39.3%. Thus 39.3% of the variability of the total value is explained by the # of rooms. Model 3: total value = *(# of bedrooms ). R 2 = 33.9%. Thus 33.9% of the variability of the total value is explained by the # of bedrooms. 24

25 Which one of the 3 previous models would you choose and why? Model 1 because it has the largest value of R 2. 25

26 1-α confidence interval for the mean of the values of Y for a specific value of X: For model 1 and a value of X=1500 sq.ft we obtain the following point estimation : est. total value = *1500 = $ 95% confidence interval for the mean of the total value for properties of 1500 sq.ft : [ , ] as calculated by CI-regression.xls 26

27 1-α confidence interval for a new value of Y (prediction) being given a specific value of X: For model 1 and a value of X=1500 sq.ft we obtain the following point estimation : est.total value = *1500 = $ 95% confidence interval for a predicted total value when the area of the first floor is 1500 sq.ft : [59 742, ] The confidence interval for a predicted value is always larger than for the mean of the value of Y for a specific X. 27

28 Inference on regression model parameters: If there is no linear link between Y and X then β 1 = 0. So, we want to examine the following hypothesis : H 0 : β 1 = 0 vs H 1 : β 1 0 We will reject H 0 when the p-value is too small This test will be valid if the relation between X and Y is linear the data are independent the variance of Y is the same for every value of X. Y has a normal distribution for every value of X or the sample size n is large. 28

29 Multiple linear regression It is more likely possible that the variability of the dependent variable Y will be explained not only by one independent variable X, but rather by a linear combination of several independent variables X 1, X 2,, X p. In this case, the multiple regression model is given by: Y = β 0 + β 1 X 1 + β 2 X β p X p + ε Also, using the sample data, we will estimate the regression model parameters β 0, β 1,, β p in order to minimize the residuals (errors) sum of squares. 29

30 The multiple correlation coefficient R 2, also called the coefficient of determination, represents the percentage of the variability of Y explained by the independent variables X 1, X 2,, X p. In the model, when we add one or more independent variables, R 2 increases. The question is to know if R 2 increases to a significant degree. Note that we cannot have more independent variables in the model that there are observations in the sample. (general rule: n 5p). 30

31 Example: MODEL 1. The regression equation is Total = Land Acre Sq.Feet Rooms Bedroom C.Bathro Bathro Fire-pl. Predictor Coef StDev T P Constant Land Acre Sq.Feet Rooms Bedroom CBathro Bathro Fire-pl S = R-Sq = 88.9% R-Sq(adj) = 87.6% Analysis of Variance Source DF SS MS F P Regression E Residual Error Total E+11 31

32 MODEL 2 Regression Analysis The regression equation is Total = Land Acre Sq.Feet Bedroom C.bathro Bathro Predictor Coef StDev T P Constant Land Acre Sq.Feet Bedroom C.bathro Bathro S = R-Sq = 88.5% R-Sq(adj) = 87.6% Analysis of Variance Source DF SS MS F P Regression E Residual Error Total E+11 32

33 MODEL 3 Regression Analysis The regression equation is Total = Land Acre Sq.Feet C.bathro Bathro Predictor Coef StDev T P Constant Land Acre Sq.Feet C.bathro Bathro S = R-Sq = 88,3% R-Sq(adj) = 87,5% Analysis of Variance Source DF SS MS F P Regression E Residual Error Total E+11 33

34 Model without the area of the land ( # of acres ) because of the multicolinearity with the land value. MODEL 4 The regression equation is Total = Land Sq.Feet C.bathro Bathro Predictor Coef StDev T P Constant Land Sq.Feet C.bathro Bathro S = R-Sq = 87.0% R-Sq(adj) = 86.3% Analysis of Variance Source DF SS MS F P Regression E E Residual Error Total E+11 34

35 Which one of the 4 previous models would you choose and why? Probably model 4 because all the independent variables are significant at the 5% level (i.e. for each β in the model, p-value < 5%) and although R 2 is smaller, it is just marginally smaller. Moreover, all the model coefficients make«sense»! In model 1, the variables # of rooms and # of fire-place are not statistically significant at the 5% level (p-value > 5%). The variable # of bedrooms is at the limit with a p-value =

36 Which one of the 4 previous models would you choose and why?(continued) In model 2 the variable # of bedroom is not statistically significant at the 5% level. In model 3 (and the previous models), the variable # of acres coefficient is negative which is contrary to «common sense» and to what we observed in the scatter plot and the positive Pearson correlation coefficient (r = 0.608). In models 1 to 3, the negative coefficient for the variable # of acres is due to the fact that there is a strong linear relation between the value of the land and the area of the land (r = 0.918): multicolinearity problem. 36

37 Multicolinearity If two or more explanatory variables are strongly correlated (> 0.85 in absolute value), one says that there is multicolinearity. It has an influence on the estimation of parameters in the model. If two explanatory variables are highly correlated, then can get rid of one of these variables. Because of the strong correlation, the contribution of the other variable is not significant. The correlation between several pairs of variables can be calculated in Excel using correlation in the Data Analysis toolbox. 37

38 How can we choose a particular linear regression model among all the possible ones? There are several techniques: Step by step selection by adding one variable at a time, starting with the most significant one (stepwise, forward). Selection starting from the model in which all the variables are included and removing one variable at a time starting with the least significant (backward). Construct all possible models and choose the best subset of variables according to certain specific criteria (ex: adjusted R 2, C p de Mallow.) 38

39 Example of selection among the best subsets: Best Subsets Regression : Response is Total B C S e b B F q R d a a i L A f o r t t r a c e o o h h e Adj. n r e m o r r p Vars R-Sq R-Sq C-p s d e t s m o o l X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X 39

40 Selection of the model without the variable # of acres Best Subsets Regression : Response is Total B C S e b B F q R d a a i L f o r t t r a e o o h h e Adj. n e m o r r p Vars R-Sq R-Sq C-p s d t s m o o l X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X 40

41 The selection of the best model is done according to the combination: The greatest value of R 2 adjusted for the number of variables in the model. The smallest value of C p. For the models with R 2 adjusted and comparable C p, we will choose the model which has the most «common sense» according to the experts in the field. For the models with R 2 adjusted and comparable C p, the model with the independent variables that are the easiest and least expensive to measure. The model validity. 41

42 1-α confidence interval for Y mean and a new value of Y (prediction) being given a specific value combination for X 1, X 2,, X p. For model 4 and property with a land= $, sq.ft = 1500, 2 completed bathrooms and 1 notcompleted, we obtain the following point estimation : est. total value = * * * *1 = $ 95% confidence interval for the mean of the total value: [ , ] 95% confidence interval for a total predicted value : [ , ] 42

43 Notes: For a 1500 sq.ft property, the multiple regression model gives a smaller 95% confidence intervals than the simple regression model. Therefore the addition of several other variables in the model helped to better explain the total value variability and to improve our estimations. If two or more independent variables are correlated we will say that there is multicolinearity. This can influence the value of the parameters in the model. Also, if two independent variables are strongly correlated then only one of the two variables would be included in the model, the other one bringing very little additional information. Certain conditions are required for the validity of the model and the corresponding inference (similar to the simple linear regression ). 43

44 Dummy variables How can one take into account qualitative information in a regression? Application: Test on two or more means 44

45 Trick If a qualitative variable takes two values, one defined one dummy variable taking values 0 or 1. Examples: Sex: 1 if male, 0 otherwise Garage: 1 if garage, 0 if not. 45

46 Trick (continued) More generally, if a qualitative variable can take m values, one defines (m-1) dummy variables all takong values 0 or 1. Example: Sex and job category (executive, white-collar, bue-collar) X 1 = 1 if male, 0 otherwise. X 2 = 1 si exe, 0 otherwise. X 3 = 1 si w-c, 0 otherwise. 46

47 Example One wants to explain the salary of an employee (Y) with the following variables: sex, job category and experience. X 1 = 1 if male, 0 otherwise. X 2 = 1 if exe, 0 otherwise. X 3 = 1 if w-c, 0 otherwise. X 4 = years of experience. 47

48 Example (continued) Regression model: Y = β 0 + β 1 X 1 + β 2 X 2 + β 3 X 3 + β 4 X 4 + ε Question: Interpret β 0, β 1, β 2, β 3, β 4. How do know if women have a smaller salary? 48

49 P-value for one-tailed tests in Excel. The evaluation of the p-value of a one-tailed test hypothesis H 1 is not given in general, only the p-value of a two-tailed test. For example, in regression, Excel calculates the p-value P corresponding to H 0 : β i = 0 vs H 1 : β i 0. How can we calculate the p-value correponding to one-tailed hypotheses H 1? 49

50 Rules : P: p-value for the two-tailed test. If H 1 is of the form β i > 0 and b i >0, then the p-value of the right-tailed is P/2. Otherwise it is 1- P/2. If H 1 is of the form β i < 0 and b i <0, then the p-value of the right-tailed is P/2. Otherwise it is 1- P/2. In other words, the one-tailed p-value is half of the two-tailed p- value when the estimated coefficient has the same sign as the coefficient in H 1. Otherwise, it is 1- p-value /2. 50

51 Question: One wants to know if having a garage increase the total value of the property. The hypotheses to be tested should be: H 0 : β garage 0 vs H 1 : β garage >0 Since b garage = > 0, the p-value corresponding to H 1 : β garage > 0 is /2 = < The anwser is yes because we accept H 1. Does the decision depend on coding? 51

52 If the dummy is defined by 0 if there is a garage and 1 otherwise, we would have got: Totale = ,83 Terrain + 47,2 Pied SbainsC Sbains Garage Predictor Coef StDev T P Constant ,08 0,000 Terrain 1,8342 0,1892 9,69 0,000 Pied2 47,175 7,013 6,73 0,000 SbainsC ,54 0,001 Sbains ,63 0,001 Garage ,01 0,058 S = R-Sq = 87,6% R-Sq(adj) = 86,8% 52

53 In that case, the right choice for hypotheses would have been: H 0 : β garage 0 vs H 1 : β garage <0 The corresponding p-value stays = 0.058/2 because b garage = < 0 has the same sign as β garage in H 1. 53

54 Comparison of several means Suppose one wants to compare the respective means of a quantitative variable Y for two groups: µ 1 = mean of group 1, µ 2 = mean of group 2. One can use regression by defining X = 1 for group 1, and X= 0 for group 2. In this case, β = µ 1 µ 2. 54

55 Hypothesis H 1 : µ 1 > µ 2 correspond to H 1 : β > 0. Hypothesis H 1 : µ 1 < µ 2 correspond to H 1 : β < 0. Hypothesis H 1 : µ 1 µ 2 correspond to H 1 : β 0. 55

56 Example A manager has some doubts on the (positive) effects of a course in order to improve the speed a given task is performed by employees. To confirm his belief, he asked a technician to choose at random 10 employees and to measure the time (hours) to complete a task. Then the same employees attend the course. After the course the employees had to realize a similar task. The results are summarized in the following table: manager.xls 56

57 Questions: a) Should the company maintain the formation program? Take α = 5%. b) The technician in charge of the measurements forget to identify employees on the measurements form. What is the conclusion using that data set? Unfortunately, case b is based on a real case. 57

58 Solution For situation a), data are paired and we have to check if the differences «Before After» are significantly positive. The p-value is < 0.05 = α. One accepts H 1 and the manager conclude that the program should be maintained. 58

59 In the second case, data are not paired. One can use regression with Y = time of execution, and X = 1 for measurements before the course and X = 0 for measurements after the course. In that case, the right choice for H 1 is: H 1 : β > 0 Results are given by: Coefficients Standard Error t Stat P-value Intercept E-19 X

60 Since H 1 : β > 0 (which is equivalent to H 1 : µ before > µ after ), and b = > 0, the p-value is 0.201/2 = > One accepts H 0, so the formation program should nt be maintained. This is a very good example of the consequence of the greater variability for two samples compared to a paired sample. 60

61 Remark: Comparing several means If one needs to compare the means of k group, for some variable Y, one can use also regression. For i=1, 2,, k-1, set: X i = 1 for group i, 0 otherwise. Then β 0 = mean of group k = µ k and β i = µ i - µ k, 1 i k-1. 61

62 Therefore, the regression test where H 0 is given by H 0 : β 1 = β 2 =... = β k-1 = 0 is equivalent to a test where H 0 is given by H 0 : µ 1 = µ 2 =... = µ k If H 0 is rejected, then we conclude that at least two means are different. 62