Classification & Regression. Multicollinearity Intro to Nominal Data

Transcription

1 Multicollinearity Intro to Nominal

2 Let s Start With A Question y = β 0 + β 1 x 1 +β 2 x 2 y = Anxiety Level x 1 = heart rate x 2 = recorded pulse Since we can all agree heart rate and pulse are related, when anxiety increases do we increase coefficient of: x 1 x 2 No Idea 2

3 Multicollinearity y = β 0 + β 1 x 1 +β 2 x 2 y = Anxiety Level x 1 = heart rate x 2 = recorded pulse When two (or more) features are highly correlated in a regression model there is collinearity amongst them Two forms: and Structural 3

4 Multicollinearity y = Anxiety Level, x 1 = heart rate, x 2 = recorded pulse y = x x 2 Example Only (not real) x 1 :CI-[-4.2,8.2], p:.75 x 2 : CI [ 2,2], p:.62 Implications: Highly unstable estimates of the regression coefficients Estimates will be very sensitive to changes in specification of the model. Dropping a variable or excluding some observations may lead to large changes in estimated coefficients, leading to an lack of confidence in the robustness of estimates. Large standard errors of the estimated regression coefficients Result in very wide confidence intervals and low t-statistics for collinear features Not a major concern when overall goal is prediction 4

5 Visual Intuition: Multicollinearity Y x 1 x 2 5

6 Visual Intuition: Multicollinearity Y A B x 1 x 2 6

7 Visual Intuition: Multicollinearity Y A B x 1 C x 2 7

8 Visual Intuition: Multicollinearity Y A C B x 1 x 2 8

9 Visual Intuition: Multicollinearity Y A x 1 9

10 Identifying () Multicollinearity You may become aware of potential multicollinearity if High R 2 or adjusted R 2 and insignificant t-scores Significant global F-test and no significant t-test of all coefficients Computing the bi-variate correlations between each variable. However such associations can include more than 2 features Variance Inflation Factor (VIF) metric The VIF for each term in the model measures the combined effect of the dependences among the regressors on the variance of that term. There is often debate as to what is large enough 10

11 Addressing () Multicollinearity Do Nothing If your goal is prediction you are likely okay leaving the situation unaddressed Drop a variable Think carefully about this. It can introduce bias or produce an incomplete model (lack of controlling for an important feature) Create a new variable Some literature notes that if the two collinear features are related in a structural sense you can create a composite feature representing the feature(s) 11

12 Addressing () Multicollinearity Cont. Alternative forms of linear regression Thus far we have focused on ordinary least squares (OLS) Alternative forms of estimation (Lasso and Ridge regression) have been shown to be more robust with highly collinear features. Trade an increase in bias for the reduced variance estimates Additional: PCA However the features no longer interpretable Collect more data Often not feasible 12

13 Thus Far: Simple linear regression Single feature Multiple regression Multiple features Polynomial Regression In both cases the data can be transformed to support the linearity assumption However, one transformation we have not yet discussed is the addition of polynomial terms i.e. x 2 Note: this is a form of multiple regression, however it presents a unique set of challenges in execution and interpretation 13

14 Polynomial Regression Any potential problems with this linear equation? y = x x x

15 Polynomial Regression - Challenges Any potential problems with this linear equation? y = x x x 1 2 Yes We have introduced collinearity!! This is referred to as structural multicollinearity 15

16 Centering Features To reduce the multicollinearity we can center predictors just as in standardization Subtracting the mean of the feature from each value of the independent variable in question 16 x x-cent x-cent Mean(x) = 50.64

17 Centering Features The estimated intercept is now the predicted y when the independent variable equals the sample mean i.e. the expected y when x = The estimated coefficient of the centered feature has not changed. Remains the increase in Y when X increases by 1 unit. 17

18 Coefficient Relative Impact y = x x 2 When looking to identify the independent variable with the largest impact on the target variable, it is not always wise to directly compare the fitted slope coefficients. Why? 18

19 Coefficient Relative Impact y = x x 2 y = Anxiety Level x 1 = #Tests x 2 =Platelet Count When looking to identify the independent variable with the largest impact on the target variable, it is not always wise to directly compare the fitted slope coefficients. Why? 19

20 Standardized Coefficients When looking to identify the independent variable with the largest impact on the target variable, it is not always wise to directly compare the fitted slope coefficients Their size is also related to the scale on which the variables are measures Standardized coefficients are sometimes used to address this Just as in data preprocessing, standardization is achieved by subtracting the mean of each feature and dividing by it s standard deviation Resulting in a mean of 0, and a standard deviation of 1 for each feature The magnitude of coefficients can then be compared directly, however the interpretation changes drastically Comparative effects of changing the independent variables by one standard deviation Valid typically only within a single model or population sample 20

21 Nominal, Interactions 21

22 Interpret this Equation y = x x 2 y =weight x 1 = age x 2 = height grandfather 22

23 What if the is Nominal? y = x x 2 y =LOS x 1 = age x 2 = Hospital Unit If instead x 2 was Hospital Unit, β 2 would no longer makes sense You can not have 4.5 * (NICU) 23

24 What if the is Nominal? y = x x 2 y =LOS x 1 = age x 2 = Hospital Unit If instead x 2 was Hospital Unit, β 2 would no longer makes sense You can not have 4.5 * (NICU) Rather, you would use the unit as a factor to adjust your length of stay for a set age by some set amount. To do so we create indicator variables 24

25 Dichotomous (Binary) Case Dichotomous data can be addressed directly Categories can be relabeled as binary values 0 and 1 Prior to the regression, pick one category as the reference Create a new feature (X ) which indicates, for each instance, whether the feature (X) belongs to the reference (X = 0) or other category (X = 1) This is called an indicator or dummy variable Also: Intercept dummy variable x x M 1 F 0 M 1 F 0 M 1 M 1 M 1 F 0 F 0 F 0 25

26 Binary Example y = LOS, x 1 = age, x 2 = Sex y = β 0 + β 1 x 1 +β 2 x 2 26

27 Binary Example y = LOS, x 1 = age, x 2 = Sex y = β 0 + β 1 x 1 +β 2 (0) = β 0 + β 1 x 1 y = β 0 + β 1 x 1 +β 2 (1) = β 0 + β 1 x 1 +β 2 Looking at the two equations we see the (binary) categorical feature of sex allows for a set shift in average LOS for a specific age based on the individual s sex Implies a sort of flat increase or decrease for instances in a group I.e. effect of age on LOS is the same for men and women, but that the intercept (or the average difference in pay between men and women) is different 27

28 Visual Intuition y = LOS, x 1 = age, x 2 = Sex y = β 0 + β 1 x 1 +β 2 (0) = β 0 + β 1 x 1 y = β 0 + β 1 x 1 +β 2 (1) = β 0 + β 1 x 1 +β 2 Adding a dummy variable X to the regression model creates a parallel shift in the relationship by the amount β 2 β 0 + β 2 β 2 28

29 Intercept β is the mean outcome for the reference group, or the group for which x = 0. The average LOS for men Interpretation y = LOS, x 1 = Sex (reference Women) y = β 0 + β 1 x 1 Slope β is the difference in the mean outcome between the two groups (when x = 1 vs. x = 0) The average LOS difference for men compared to women For this reason, β is often called the contrast between the two categories. 29 β 0 + β 2 β 2

30 Mix of Continuous and Nominal Features y = β 0 + β 1 x 1 +β 2 x 2 y = LOS x 1 = age x 2 = Sex Perform multiple regression as per normal There is nothing special about dummy variables We interpret them as we would any additional feature. The increase / change when all other features are held constant 30

31 Mix of Continuous and Nominal Features y = β 0 + β 1 x 1 +β 2 x 2 y = LOS x 1 = age x 2 = Sex Perform multiple regression as per normal There is nothing special about dummy variables We interpret them as we would any additional feature. The increase / change when all other features are held constant β 0, measures the intercept of reference group (women) with age set to zero and β 0 + β 2 is the intercept for men β 2 represents the average difference is LOS between men and women 31

32 What About Two Nominal Features? y = β 0 + β 1 x 1 +β 2 x 2 +β 3 x 3 y = LOS x 1 = age x 2 = Sex Ref: Women x 3 = Obese Ref: No β 0 : Expected value of LOS In reference for all (Women, Non-Obese) β 1 : Expected increase of LOS when age increases by 1 β 2 : Difference in expected LOS between M and F β 3 : Difference in expected LOS between Obese and Non-Obese 32

33 Inference (Binary Variables) It is still possible, and valuable to compute t-test statistics for the coefficients of dummy variables However unlike continuous data, these values do not indicate a significant relations to the target variable β 0 + β 2 β 2 The t-test associated with a given dummy variable tests the difference between the dummy variable at level 1 and the reference category. 33

34 Inference (Binary Variables) Cont. In order to test impact on the target variable we can again utilize partial F-test: Full Model vs. Reduced Model without the nominal feature Significance indicates value of the variable in reducing (SSE) 34

35 Additional Info Standardized coefficients are often not used with dummy variables 1 standard deviation is meaningless for a 0/1 encoded value The standard interpretation of the dummy variable, showing difference in average level of Y between two categories is lost 35

36 What if Has Multiple Categories? What about nominal data, such as a risk-level? It seems reasonable we want to know if the patient came in with low, medium, or risk for some adverse condition. 36

37 What if Has Multiple Categories? What about nominal data, such as a risk-level? It seems reasonable we want to know if the patient came in with low, medium, or risk for some adverse condition. Can we simply just add more numbers? Low Medium High

38 Variables With Multiple Levels Risk Level x 1 x 2 High 1 0 Middle 0 1 Low

39 β of Multiple Levels y = β 0 + β 1 x 1 +β 2 x 2 y = LOS x 1 = High x 2 = Middle β S. Err T P-val Low Middle High