Multiple Linear Regression for the Salary Data

Size: px

Start display at page:

Download "Multiple Linear Regression for the Salary Data"

Regina Blair
6 years ago
Views:

1 Multiple Linear Regression for the Salary Data Experience Salary HS BS BS Experience Salary No Yes

2 Problem & Data Overview Primary Research Questions: 1. Are company pay guidelines being followed? (Inference) 2. Use this model to establish a fair salary system. (Prediction) Issues: 1. How do we deal with the categorical variables Education and Manager?

3 Exploratory Techniques For Categorical Covariates 1. Side-by-side Boxplots Salary Salary HS BS BS+ Salary No Yes Experience Education Manager

4 Exploratory Techniques For Categorical Covariates 2. Color-coded Scatterplots Salary HS BS BS Salary No Yes Experience Experience

5 MLR with Categorical Covariates We want to use MLR: y i iid N 0 + PX p=1 px ip, 2! y i = i th person s salary x i1 = i th person s experience level But how do we put categories in a mathematical function? Use indicator functions: ( 1 if A is true I(A) = 0 otherwise.

6 MLR with Categorical Covariates x i2 = The MLR Model: y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ Or, alternatively: y i iid N 0 + ( 1 if BS 0 otherwise. x i3 = PX p=1 4 I(Manager i =Yes)+ i px ip, y i = i th person s salary 2! x i1 = i th person s experience level ( 1 if BS+ 0 otherwise. x i4 = ( 1 if Manager 0 otherwise.

7 MLR with Categorical Covariates The MLR Model: y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ 4 I(Manager i =Yes)+ i What about HS and Not a Manager? The become absorbed into the intercept term.

8 MLR with Categorical Covariates The MLR Model: y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ 4 I(Manager i =Yes)+ i How do you interpret 0? For non-managers with a HS education and zero years experience, the salary is 0, on average.

9 MLR with Categorical Covariates The MLR Model: y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ 4 I(Manager i =Yes)+ i How do you interpret 1? Holding all else constant, as the years of experience goes up by 1, the salary goes up by 1, on average.

10 MLR with Categorical Covariates The MLR Model: y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ 4 I(Manager i =Yes)+ i How do you interpret 2? For equal years of experience and managerial levels, a person with a BS has a 2 higher salary than a person with a HS degree, on average.

11 MLR with Categorical Covariates The MLR Model: y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ 4 I(Manager i =Yes)+ i How do you interpret 3? For equal years of experience and managerial levels, a person with a BS+ has a 3 higher salary than a person with a HS degree, on average. For equal experience and manager levels, how much more salary does a BS+ get than a BS, on average? 3 2

12 MLR with Categorical Covariates The MLR Model: y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ 4 I(Manager i =Yes)+ i How do you interpret 4? For equal years of experience and education levels, a person with managerial position has a 4 higher salary than a person without a managerial position, on average.

13 MLR with Categorical Covariates Fitted MLR Model: ŷ = (Experience) I(Education = BS) I(Education = BS+) I(Manager = Yes) For equal years experience and education, how much more does a manager make than a non-manager, on average? You want to say because ˆ4 = but you re wrong. Why? is our best guess but we are uncertain and we should express this uncertainty.

14 Expressing Uncertainty in MLR You can show (in, say, STAT 535) that for any p =0,...,P t = ˆp p SE( ˆp) T n P 1 1. Confidence Intervals: ˆp ± t? 1 SE( ˆp) Computer s calculate SE s for us (at least in this class) Example 95% interval for 4 : ± ! ( , ) How do you interpret this interval? For equal experience and education levels, a manager would make between 6212 and 7521 more than a nonmanager, on average.

15 Expressing Uncertainty in MLR 2. Hypothesis Testing 1. All coefficients simultaneously 2. Some coefficients simultaneously 3. One coefficient at a time

16 Expressing Uncertainty in MLR Hypothesis Test #1: Testing all coefficients simultaneously. One can show (in Stat 535) that if, H 0 : 1 = 2 = = P =0 H A : At least one is non-zero Reduced Model Full Model then, F = R 2 /p (1 R 2 )/(n P 1) F p,n P 1 F-Distribution with p and n-p-1 DF so the F-distribution can be used to compute p-values.

17 Expressing Uncertainty in MLR Consider: F = R 2 /p (1 R 2 )/(n P 1) F p,n P 1 R 2 : How much variation you explain (1 R 2 ) : How much variation you don t explain F-statistic is 1. Analyzes variances (ANOVA) 2. Ratio of explained to unexplained variance.

18 Expressing Uncertainty in MLR Testing all coefficient simultaneously for salary data H 0 : 1 = 2 = = 4 =0 H A : At least one p is non-zero F = 211.7! p value 0 Density What is the conclusion? Reject the null hypothesis and conclude that at least one covariate significantly explains salary. F F

19 Expressing Uncertainty in MLR Hypothesis Test #2: Testing some coefficients simultaneously. Let, 1,..., Q be the coefficients you think are non-zero and don t want to test. And, Q+1,..., P be the coefficients you do want to test. then, H 0 : Q+1 = = P =0 H A : At least one is non-zero R2 with all P variables R2 with just Q variables F = (R2 P RQ 2 )/(P Q) (1 RP 2 )/(n P 1) F P Q,n P 1 Reduced Model Full Model so the F-distribution can, again, be used to compute p-values. F is ratio of how much explained variation you lost, relative to total unexplained variation.

20 Expressing Uncertainty in MLR Hypothesis Test #2: Salary Data Example. Suppose we want to test if Education has an effect on salary. y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ Then the hypotheses are, H 0 : 2 = 3 =0 H A : At least one is non-zero F = ! p value 0 What is the conclusion? Education has a non-zero effect on salary. 4 I(Manager i =Yes)+ i

21 Expressing Uncertainty in MLR Hypothesis Test #3: Testing individual coefficients. Remember: t = ˆp p SE( ˆp) T n P 1 so the t-distribution can be used to calculate p-values. Example: Does managerial position have an effect on salary? H 0 : 4 =0 H A : 4 6= 0 t = p value 0 What is the conclusion? Managerial position has an effect on salary. =

22 Expressing Uncertainty in MLR Question: Why test coefficients simultaneously? We want to avoid the multiple comparison problem. Multiplicity (or multiple comparison problem): If you do lots of tests then you are more likely to commit errors (your overall Type I error rate is inflated).

23 Expressing Uncertainty in Predictions Fitted MLR Model: ŷ = (Experience) I(Education = BS) I(Education = BS+) I(Manager = Yes) What is the predicted salary for a manager with a BS education and 10 years experience? You want to say because = but you re wrong. Why? is our best guess but we are uncertain and we should express this uncertainty.

24 Expressing Uncertainty in Predictions 1. Confidence Intervals for the Mean ˆµ(x 1,...,x P ) ± t? 1 (df = n P 1) SE(ˆµ) 2. Prediction Intervals for One Value ŷ ± t? 1 (df = n P 1) SE(ŷ) Important Notes: 1. SE formulas are ugly (but not in matrix notation) so let computer calculate it for you. 2. SE(ˆµ) < SE(ŷ) so prediction intervals will be wider than confidence intervals.

25 Cross-Validation Revisited When we perform cross-validation, we are used to calculating: 1. Bias 2. RPMSE But prediction intervals should also be generated for each test observation and calculate the following: 1. Coverage = % of prediction intervals that contain the true value 2. Predictive Interval Width = average width of prediction interval

26 MLR with Categorical Covariates Fitted MLR Model: ŷ = (Experience) I(Education = BS) I(Education = BS+) I(Manager = Yes) Salary HS,No BS,No BS+,No HS,Yes BS,Yes BS+,Yes Experience

27 Interactions Just based on this picture, if you have a HS degree and become a manager, how much does your salary go up on average? About $3000 or $4000 Salary HS BS BS Salary No Yes Experience Experience

28 Interactions Just based on this picture, if you have a BS degree and become a manager, how much does your salary go up on average? About $8000 or $9000 Salary HS BS BS Salary No Yes Experience Experience

29 Interactions Key Observation: How much your salary goes up when you become a manager depends on how much education you have. Salary HS BS BS Salary No Yes Experience Experience

30 Interactions Interaction: Two (or more) variables work simultaneously to affect the response. In other words, the effect of one covariate on the response depends on the value of another covariate. Interactions enter the regression multiplicatively. Types of Interactions: 1. Quantitative-Quantitative (Q-Q) 2. Quantitative-Categorical (Q-C) 3. Categorical-Categorical (C-C)

31 Interactions Q-Q Interactions: x i1,x i2 : Quantitative Variables y i = x i1 + 2 x {z i2 + } 3x i1 x i2 + {z } i Main E ects Interaction Term Holding x i1 constant, as x i2 goes up by 1, how much does y i go up on average? x i1 + 2 (x i2 + 1) + 3 x i1 (x i2 + 1) + i ( x i1 + 2 x i2 + 3 x i1 x i2 + i ) = x i1 It depends on the value of x i1!

32 Interactions Q-Q Interactions (Example): x i1,x i2 : Quantitative Variables y i =0+0.5x i1 +( 0.5)x i2 {z } Main E ects + ( 1)x i1 x i2 + i {z } Interaction Term y x1= 1 x1=0 x1= x 2

33 Interactions Q-C Interactions: x i1 : Quantitative Variable ( 1 if Category x i2 = 0 otherwise. x i2 y i = x i1 + 2 x {z i2 + } 3x i1 x i2 + {z } i Main E ects Interaction Term x i1 Holding constant, as goes up by 1, how much does go up on average? ( 1 if x i2 = x i2 = if x i2 =1 It depends on the value of BUT is easily interpretable. x i2 y i

34 Interactions C-C Interactions: x i1,x i2 : Categorical Variables y i = x i1 + 2 x {z i2 + } 3x i1 x i2 + {z } i Main E ects Interaction Term Holding x i1 constant, as x i2 goes up by 1, how much does y i go up on average? ( 2 if x i1 = x i1 = if x i1 =1

35 Salary Example with Interactions The MLR Model for Salary with Interactions: y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ 4 I(Manager i =Yes)+ 5I(Education i =BS)I(Manager i =Yes)+ 6I(Education i =BS+)I(Manager i =Yes)+ i

36 Salary Example with Interactions The MLR Model for Salary with Interactions: ŷ = (Experience i ) I(Education i =BS) I(Education i = BS+) I(Manager i =Yes) I(Education i =BS)I(Manager i =Yes) I(Education i =BS+)I(Manager i =Yes) What is the effect of becoming a manager on salary if you have a HS education? The salary would go up by , on average.

37 Salary Example with Interactions The MLR Model for Salary with Interactions: ŷ = (Experience i ) I(Education i =BS) I(Education i = BS+) I(Manager i =Yes) I(Education i =BS)I(Manager i =Yes) I(Education i =BS+)I(Manager i =Yes) What is the effect of becoming a manager on salary if you have a BS education? The salary would go up by =9038.1, on average.

38 Salary Example with Interactions The MLR Model for Salary with Interactions: ŷ = (Experience i ) I(Education i =BS) I(Education i = BS+) I(Manager i =Yes) I(Education i =BS)I(Manager i =Yes) I(Education i =BS+)I(Manager i =Yes) What is the effect of becoming a manager on salary if you have a BS+ education? The salary would go up by = , on average.

39 Salary Example with Interactions The MLR Model for Salary with Interactions: ŷ = (Experience i ) I(Education i =BS) I(Education i = BS+) I(Manager i =Yes) I(Education i =BS)I(Manager i =Yes) I(Education i =BS+)I(Manager i =Yes) For managers, what is the effect of having a BS+ education vs. a BS education with 2 years experience? ( ) ( ) = Salary would go DOWN by

40 Salary Example with Interactions The MLR Model for Salary with Interactions: Salary HS,No BS,No BS+,No HS,Yes BS,Yes BS+,Yes Experience Are the company s salary policies being followed (R 2 =0.99)?

41 Salary Example with Interactions How would I test if the interaction between education and manager position is significant? y i = (Experience i )+ 2 I(Education i =BS)+ 3I(Education i =BS+)+ 4 I(Manager i =Yes)+ 5I(Education i =BS)I(Manager i =Yes)+ 6I(Education i =BS+)I(Manager i =Yes)+ i Perform an F-test (see earlier notes in this unit): H 0 : 5 = 6 =0 H A :At least 1 is non-zero F = p value 0 Conclusion: There is an education-manager interaction.

42 End of Salary Analysis (see webpage for R and SAS code)

Multiple Linear Regression for the Supervisor Data

Multiple Linear Regression for the Supervisor Data for the Supervisor Data Rating 40 50 60 70 80 90 40 50 60 70 50 60 70 80 90 40 60 80 40 60 80 Complaints Privileges 30 50 70 40 60 Learn Raises 50 70 50 70 90 Critical 40 50 60 70 80 30 40 50 60 70 80