Predictive Analytics : QM901.1x Prof U Dinesh Kumar, IIMB. All Rights Reserved, Indian Institute of Management Bangalore

Transcription

1

2 What is Multiple Linear Regression Several independent variables may influence the change in response variable we are trying to study. When several independent variables are included in the equation, the regression is called multiple linear regression.

3 Multiple Regression Modeling Steps 1. Start with a hypothesis or belief. 2. Estimate unknown model parameters ( 0, 1, 2. ) 3. Specify probability distribution of random error term - Assumed to be a normal distribution 4. Check the assumptions of regression (normality, heteroscedasticity and multi-collinearity) 5. Evaluate Model 6. Use Model for Prediction & Estimation and other purposes.

4 Multiple Linear Regression Model Relationship between 1 dependent & 2 or more independent variables is a linear function. Population Y-intercept Population slopes Random error Y i 0 1 X 1i 2 X 2i k X ki i Dependent (response) variable Independent (explanatory) variables

5 Prediction Model The prediction equation obtained from sample data is: ki k i i X X X Y ˆ

6 Multiple Regression Model Y Y i = X 1i + 2 X 2i + i (Observed Y) Y i = Treatment Cost X 1i = Body Weight Response Plane 0 i X 2i = HR Pulse X 2 X 1 (X 1i,X 2i ) YX = X 1i + 2 X 2i

7

8 Estimating and Interpreting the -Parameters Belief Y x k x k Fitted Model ˆ ˆ ˆ yˆ 0 1x1... kxk Using least squares method Minimize ˆ 2 SSE y y

9 Estimation of Parameters in Multiple Regression The least squares function is given by The least squares estimates must satisfy n i k j ij j i n i i x y SSE j xij n i k j xij j yi j SSE n i k j xij j yi SSE and

10 Estimation of Parameters in Multiple Regression The least squares equations are The solution to the above equations are the least squares estimators of the regression coefficients.

11 DAD Example Total Hospital Cost β β1 Body weight β2 Body height β3 HR Pulse β4 0 RR

12 MODEL Treatment Cost Body weight HR Pulse RR Body height

13 Interpretation of Coefficients ( 1 ) Partial regression coefficient Total cost of treatment increases by INR for every one kg increase in body weight while other parameters like body height, HR pulse and RR are controlled (kept constant). Treatment Cost Body weight Body height HR Pulse RR

14 Interpretation of Coefficients ( 3 ) Partial regression coefficient Total cost of treatment increases by INR for every one unit increase in HR pulse while other parameters like body weight, Body height and RR are controlled. Treatment Cost Body weight Body height HR Pulse RR

15 Standardized Beta (Beta Weights) The beta weights are the regression coefficients for standardized data. Standardized Beta is the average increase in the dependent variable when the independent variable is increased by one standard deviation and other independent variables are held constant.

16 Beta Weights S Standardized Beta i S S S x y Standard deviation of X Standard deviation of Y x y

17

18 Partial Correlation Partial correlation coefficient measures the relationship between two variables (say Y and X 1 ) when the influence of all other variables (say X 2, X 3,, X k ) connected with these two variables (Y and X 1 ) are removed. Partial correlation is the correlation between residualized response and residualized predictor.

19 Semi-Partial (Part) Correlation Semi-partial (or part correlation) coefficient measures the relationship between two variables (say Y and X 1 ) when the influence of all other variables (say X 2, X 3,, X k ) connected with these two variables (Y and X 1 ) are removed from one of the variables (X 1 ). Only the predictor is residualized. The denominator of the coefficient (the total variance of the response variable, Y) remains the same no matter which predictor is being examined.

20 Y D Variance in Y = A + B + C + D A C B Variance in Y explained by the model = A + B + C Variance in Y not explained by the model = E F E G X 1 X 2 Semi partial (part) correlation for X 1 = A /(A+B+C+D) Semi partial (part) correlation for X 2 = B /(A+B+C+D) Partial correlation for X 1 = A / (A+D) Partial correlation for X 2 = B / (B + D) IMPORTANT In part correlation we do not residualize the response variable

21 r r 12,3 13,2 r 23,1 r 12 (1 r r 13 (1 r r (1 r 2 12 Partial Correlation r 13 r 23 )(1 r r 12 r 23 )(1 r r 12 r 13 )(1 r ) ) ) Correlation between y1 and x2, when the influence of x3 is removed from both y1 and x2. Correlation between y1 and x3, when the influence of x2 is removed from both y1 and x3. Correlation between x2 and x3, when the influence of y1 is removed from both x2 and x3. For Proof: Theory of Econometrics by A Koutsoyiannis

22 Semi-partial correlation (Part Correlation) Semi-partial (or part) correlation sr 12,3 is the correlation between y 1 and x 2 when influence of x 3 is partialled out of x 2 (not on y 1 ). sr 12,3 r 12 r 13 1 r r Part Correlation Partial Correlation (1 r 2 13 )

23 Semi-partial (part) correlation Square of Semi-partial (or part correlation) gives the increase in the R 2 value (coefficient of multiple determination) when an explanatory variable is added to the model.

24 Y = x Body weight + 2 x HR Pulse = (0.228) 2 = 0.173

25 Change in R 2 when a new variable is added Y 1 A C D B A X 1 X 2 Variance specific to X1 (part A) Variance common to X1 and X2 (part C) Variance specific to X2 (part B) Left over R R 2

26

27 Multiple Regression Model Diagnostics Test for overall model fitness (R-Square and Adjusted R-Square) Test for overall model statistical significance (F test test) Test for portions of the model (Partial F-test) Test for statistical significance of individual explanatory variables (t test) Test for Normality and Homoscedasticity of residuals Test for Multi-collinearity and Auto Correlation

28 Co-efficient of multiple determination R 2 SSR SST 1 SSE SST i i ( y ( y i i y) y) R Multiple coefficient of multiple determination R x 1 2, is x the percentage of 2,..., x k the variation of y explained by

29 Y = x Body Weight + 2 x HR Pulse R 2 is % of variation in Y is explained by Bodyweight and HR Pulse

30 Y = x Body Weight x HR Pulse

31 Co-efficient of determination in Multiple Regression Coefficient of determination increases as the number of explanatory variables increases. In SSR/SST, the numerator, SSR, increases as the number of explanatory variables increases, whereas the denominator, SST, remains constant. Increase in R 2 can deceptive, since more number of explanatory variables may over-fit the data.

32

33 Adjusted R 2 Inclusion of additional explanatory variable will increase the R 2 value. By introducing an additional explanatory variable, we increase the numerator of the expression for R 2 while the denominator remains the same. To correct this defect, we adjust the R 2 by taking into account the degrees of freedom.

34 Adjusted R 2 R R n k 2 A 2 A 1 SSE/(n SST/(n Adjusted R - Square number of number of k 1) 1) observations explanatory variables

35 Y = x Body Weight + 2 x HR Pulse Adjusted R 2 is 0.167

36

37 Testing For Overall Significance of Model F Test H0 : β1 β2... βk 0 H :Not allβ values are zero A Test for overall significance of multiple regression model. Checks if there is a statistically significant relationship between Y and any of the explanatory variables (X 1, X 2,, X k ).

38 F Statistic F = MSR/MSE Relationship between F and R 2 F (1 R 2 R 2 / k ) /( n k 1)

39 DAD CASE - ANOVA Table F = x /

40 Test for Individual Variables

41 Testing for Significance of Individual Parameters H H 0 A : β :β i i 0 0 t S e i ( ) i T- test: By rejecting the null hypothesis, we can claim that there is a statistically significant relationship between the response variable Y and explanatory variable X i.

42 SPSS Output for Coefficients

43

44 Testing Model Portions Partial F Test Examines the contribution of a set of explanatory variables to the regression model. Used for selecting explanatory variables in regression model building strategies such as stepwise regression.

45 Testing Model Portions Partial F Test Full Model: Y X X k X k Reduced Model (r < k): Y X X r X r Test H 0 : r+1 =.. = k = 0 Partial F (SSE reduced SSE MSE full full )/(k r) k = Number of variables in the full model r = Number of variables in the reduced model

46 Partial F-test Example (DAD) Full Model: Treatment cost RR 4 0 Bodyweight 1 Bodyheight 2 HRpulse 3 Reduced Model: Treatment cost 0 1 Bodyweight 2 HRpulse F ( SSE SSE ) /(4 reduced MSE full full 2)

47 ( SSE F reduced p value SSE MSE full full ) /(4 2) ( ) /

48

49 Categorical Variables in Regression Many regression models are likely to have qualitative (categorical) variables as explanatory variable. Qualitative variables (categorical variables) in regression are replaced with dummy variables (or indicator variables) in regression model. A categorical variable with n levels are replaced with (n-1) dummy variables. The category for which no dummy variable assigned is known as Base Category.

50 Dummy variable When there are more than one qualitative variable, it is advisable to use (n-1) dummy variables for both qualitative variables along with the intercept. multi- Use of n dummy variables along with intercept will result in collinearity, known as dummy variable trap.

51 Dummy variables in Regression The intercept, 0, is the mean value of the base category. The coefficients attached to dummy variables are called differential intercept coefficients.

52 Qualitative Variables DAD Case 1. Gender (2 levels) 2. Marital Status (2 levels) 3. Key Complaints (6 levels)

53 Model with Dummy Variable Y = β 0 + β 1 x Female Y for Male = Y for Female = =

54 Y = β 0 + β 1 x Female + β 2 x Body Weight

55 Impact of Dummy Variable Y Y 0 2 X 2 ( X 0 1) (M) 2 2 (F)

56 Qualitative Variable with Multiple Levels Key Complaints 1. ACHD 2. CAD 3.NONE 4. OS-ASD 5. OTHER 6.RHD

57

58 Regression Model Building Derived variables may help to explain the variation in the response variable. For example, consider a credit rating problem, which can be used for approving housing loan. A bank may use factors such as loan amount and value of the property etc. Customer Loan Amount 250, , ,000

59 Derived Variables Customer Loan Amount 250, , ,000 Value of Property 300, ,000 1,000,000 Loan / Value

60 Interaction Variables Consider a regression model of type: Y = β 0 + β 1 X 1 + β 2 X 2 + β 3 X 1 X 2 Interaction variable The interaction variable is usually an interaction between a quantitative variable and qualitative variable

61 One of the popular applications of regression is to predict gender discrimination. Consider a regression model with salary as response variable Y: Y = β 0 + β 1 Gender + β 2 Work Experience + β 3 Gender x Work Experience

62 Interaction Variable Let Gender = 1 implies Female: Then Y for Female is: Y = β 0 + β 1 + (β 2 + β 3 ) Work Experience Y for Male is: Y = β 0 + β 2 x Work Experience

63 Conditional relationship in Interaction Variables

64 DAD Example Interaction Variable Y = β 0 + β 1 Marital Status + β 2 Body Weight + β 3 Marital status x Bodyweight

65

66 Multicollinearity High correlation between explanatory variables is called multi-collinearity. Multi-collinearity leads to unstable coefficients. Always exists; matter of degree.

67 Multi-collinearity Y = β 0 + β 1 Marital Status + β 2 Body Weight + β 3 Marital status x Bodyweight

68 Symptoms of Multi-collinearity High R 2 but few significant t ratios. F-test rejects the null hypothesis, but none of the individual t-tests are rejected. Correlations between pairs of X variables are more than with Y variables.

69 Effects of Multi-collinearity The variances of regression coefficient estimators are inflated. The magnitudes of regression coefficient estimates may be different. Adding and removing variables produce large changes in the coefficient estimates. Regression coefficient may have opposite sign.

70 Identifying Multi-collinearity Variance Inflation factor The variance inflation factor (VIF) is a relative measure of the increase in the variance in standard error of beta coefficient because of collinearity. A VIF greater than 10 indicates that collinearity is very high. A VIF value of more than 4 is not acceptable.

71 Variance of Beta Estimates ) (1 ) ( ) (1 ) ( r SS S Var r SS S Var X X Y X e X e

72 Variance inflation factor Variance inflation factor associated with introducing a new variable X j is given by: VIF(X ) j 1 1 R 2 j 2 R j is the coefficient of determination for the regression of X j asdependent variable The standard error of the corresponding Beta is inflated by VIF

73 Impact of VIF on Se() R j Tolerance (1-R 2 ) VIF Impact on Se() VIF

74 DAD CASELET For Body Weight Actual t x Corresponding p - value 0.006

75 Remedies for Multi-collinearity Drop a variable (may result in specification bias). Principal component analysis (PCA) Partial Least Squares (PLS)

76

77 Forward Selection Method In forward selection procedure in which variables are sequentially entered into the model. The first variable considered for entry is the one with smallest p-value based on F-test.

78 Backward elimination method A variable selection procedure in which all variables are entered into the equation and then sequentially removed starting with the most nonsignificant variable. At each step, the largest probability of F is removed.

79 Stepwise Regression The first variable considered for entry into the equation is the one with the largest F value or smallest p-value based on F-test. At each step, the independent variable not in the equation that has the smallest probability of F is entered. Variables already in the regression equation are removed if their probability of F becomes sufficiently large.

80

81

82 Y = CAD RHD HR PULSE MARRIED OTHERS