1. BINARY LOGISTIC REGRESSION

Transcription

1 1. BINARY LOGISTIC REGRESSION The Model We are modelling two-valued variable Y. Model s scheme Variable Y is the dependent variable, X, Z, W are independent variables (regressors). Typically Y values are coded by 0 or 1. Model is constructed for logarithm of the ratio P(Y = 1)/P(Y=0) (the so-called logit function): ln P(YY = 1) P(YY = 0) = CC + bb 1 XX + bb 2 ZZ + bb 3 WW. The same model can be written also as Here P(YY = 1) = ezz 1 + e zz = 1, P(YY = 0) = 1 P(YY = 1). 1 + e zz zz = CC + bb 1 XX + bb 2 ZZ + bb 3 WW. Values of coefficients C, b1, b2, b3 are unknown. Application of logit model means that we will calculate estimates CC, bb 1, bb 2, bb 3 from data. If bb 1 > 0, then larger values of X increase probability P(Y= 1). If bb 1 < 0, then larger values of X decrease probability P(Y= 0). Odds ratio By Odds we call ratio P(Y = 1) / P(Y = 0). Odds ratio for X1 shows the change in odds, when X1 increases by unity and all other variables are constant. 1

2 e bb 1 is called odds ratio P(YY = 1) P(YY = 0) nnnnnn oooooooo = e bb P(YY = 1) 1 P(YY = 0) oooooo oooooooo Forecasting Forecast is made for P(Y = 1). Firstly, we pass values of X, Z, W into Then substitute z into P (YY = 1) = zz = CC + bb 1 XX + bb 2 ZZ + bb 3 WW. ezz 1 + e zz, P (YY = 0) = 1 P (YY = 1) = e zz. Here e = 2, For practical forecast it suffices to take e 2,7183. Classification Classification means that we must decide if Y = 1 or Y = 0. Typically it is assumed that if Jeigu P (YY = 1) > 0.5, then we choose Y = 1. Otherwise, we choose Y = 0. Note that probability P (YY = 1) = 0.5 can be treated in various ways. Most logit models assume that then Y = 1 or Y = 0 is chosen arbitrary (one can flip a coin). For classification it suffices to know the sign of logit function z = ln P(YY = 1) P(YY = 0) If z > 0, we choose Y = 1. If z < 0, we choose Y = 0. If z = 0, we flip a coin. 2

3 Data 1) Dependent variable Y is dichotomous. There is enough data for both values of Y (ar least 20% of all observations correspond to Y=1 but no more than 80%). Otherwise logistic regression is inapplicable. 2) If model contains many categorical regressors, then for each combination of their values there should be at least 5 observations in the data. 3) Regressors are not strongly correlated. Though, in general, multicollinearity is not a serious problem in logisdtic models some authors recommend to drop some variables from the model if their standard errors SE > 5. For a good Model Maximum likelihood chi square test s p < 0,05. Wald test s p < 0,05 for each regressor (all regressors are statistically significant) At least 50 % of cases when Y = 1 and at least 50 % of cases when Y = 0 are correctly classified. Cook s distance 1 and all DFBeta 1 for all observations. Chosen pseudo-determinant coefficient 0,20. Logistic regression with SPSS 1. Data File ESS4EE_PT. We investigate males years of age. Variables: cntry (EE Estonia, PT Portugal), stfedu attitude toward countries educational system (0 very bad,, 10 very good), happy happiness (0 very unhappy,., 10 very happy), freehms attitude toward sexual minorities ( 1 positive,., 5 negative), imsb3 (base on imsclbn social benefits for immigrants, 0 no longer than after a year, 1 after worked and paid taxes at least a year, 2 after becoming citizen or never). Proposed model: cntry = f(stfedu, happy, freehms, imsb3) 3

4 2. SPSS options Analyze Regression Binary Logistic: Since imsb3 is nominal we choose Categorical and move imsclbn to the right In Options menu check CI for exp(b). Then in Save menu check Cook s and DfBeta(s). 4

5 3. Results Table Dependent variable Encoding provides information which country corresponds to models Y = 1 (Portugal). Dependent Variable Encoding Original Value Internal Value EE Estonia 0 PT Portugal 1 We indicated that imsb3 is nominal. Therefore it is replaced by 2 dichotomous dummy variables, taking values 0 and 1. In the table Categorical Variables Codings, we see which combination of dummy variables corresponds to initial value of each of imsb3. Categorical Variables Codings Parameter coding Frequency (1) (2) imsb3 Attitude towards social benefits for immigrants.00 positive attitude more positive than negative negative attitude Classification Table (the second classification table) shows that 76 % Estonians and 78,9% Portugals were correctly classified. The percent of correctly classified respondents is sufficient for acceptable model. Classification Table a Predicted Observed EE Estonia cntry Country PT Portugal Percentage Correct Step 1 cntry Country EE Estonia PT Portugal Overall Percentage

6 The maximum likelihood chi square test is given in Omnibus Tests of Model Coefficients. All three rows are identical. Model fits data statistically significantly, p = 0, < 0,05. Omnibus Tests of Model Coefficients Chi-square df Sig. Step 1 Step Block Model Model Summary contains two pseudodeterminant coefficients. Both are much larger than 0.20 (Cox and Snell R 2 = 0,372, Nagelkerke R 2 = 0,495.) Model Summary Step -2 Log likelihood Cox & Snell R Square Nagelkerke R Square a Table Variables in the Equation contains estimates of model coefficients, information on their statistical significance (significant are variables with p < 0.05 in Sig.) and odds ratio in Exp(B). We see that all regressors, except constant, are statistically significant. As a rule, we ignore statistical insignificance of the constant. Variables in the Equation 95 % C.I.for EXP(B) B S.E. Wald df Sig. Exp(B) Lower Upper Step 1 a freehms happy stfedu imsb imsb3(1) imsb3(2) Constant a. Variable(s) entered on step 1: freehms, happy, stfedu, imsb3. 6

7 Lgostic regression model (we recall that Y=1 corresponds to cntry= PT): ln P (cntry = PT) = 0,76 freehms + 0,30 happy 0,564 stfedu P (cntry = EE) +1, ,661 imsb3(1) + 1,137 imsb3(2) Taking into account the coding of dummy variables, instead of we can write 1, ,661 imsb3(1) + 1,137 imsb3(2) 1,661 if imsb3 = 0, 1, ,137 if imsb3 = 1, 0 if imsb3 = 0. The larger values of freehms and stfedu increase probability that respondent is from Estonia (corresponding coefficients are negative), the larger value of happy increases probability that respondent is from Portugal. Odds ratio for imbs3 allows the following interpretation: in comparison to negative attitude toward immigrants (imsb3 = 2), the positive attitude (imsb = 0) increases odds with respect to Portugal by factor 5,263. Cook s and DFB values appear in data window. One can check that all values indicate that there are no outliers in the data. Note that Descriptive statistics can be used to check for the maximum values of COO_1 and DFB. 7

8 4. Interaction of regressors Model: cntry = f(stfedu, happy, freehms, happy*freehms). Then in Logistic Regression menu mark both variables happy and freehms and choose >a*b>. Wald test shows that interaction is not statistically significant. Variables in the Equation B S.E. Wald df Sig. Exp(B) Step 1 a freehms stfedu happy freehms by happy Constant a. Variable(s) entered on step 1: freehms, stfedu, happy, freehms * happy. 8

9 2. PROBIT REGRESSION Model We are modelling two-valued variable. Probit regression can be used whenever logistic regression applies and vice versa. Model scheme Variable Y is dependent variable, X, Z, W are independent variables (regressors). Typically Y values are coded 0 or 1. Model is constructed for P(Y = 0): P(YY = 0) = Φ(CC + bb 1 XX + bb 2 ZZ + bb 3 WW). Here Φ( ) is the distribution function of the standard normal random variable. Equivalent expression Φ 1 P(YY = 0) = CC + bb 1 XX + bb 2 ZZ + bb 3 WW. Here Φ 1 ( ) is inverse function, also known as probit function. If bb 1 > 0, then as X grows, also grows P(Y= 0). If bb 1 < 0, then as X, also grows P(Y= 1). Data a) Variable Y is dichotomous. Data for Y contains at least 20% of zeros and at least 20% of 1. b) If model contains many categorical variables, for each combination of categories data should contain at least 5 observations. c) No strong correlation between regressors. 9

10 Model fit Model fits data if: Maximum likelihood p-value p < 0,05. Wald test p-value p < 0,05 for all regressors. Correct classification for many cases of Y = 1 and Y = 0. For all variables Cook s measure 1. Probit regression with SPSS 1. Data File LAMS. Variables: K2 university, K33_2 studies for achievement of my present position were (1 absolutely unimportant,..., 5 very important), K35_1 my present occupation corresponds to bachelor studies (1 agree, 2 more agree, than disagree, 3 more disagree, than agree, 4 disagree), K36_1 I use professional skills obtained during studies (1 never,..., 5 very frequently), K37_1 satisfaction with my profession (1 not at all,..., 5 very much). With recode we create a new two-valued variable Y, (0 if respondent rarely applies professional skills obtained during studies, 1 if frequently). Model scheme: Y = f( K35_1, K33_2, K37_1). Or graphically 10

11 2. SPSS options We investigate Klaipėda University: SELECT CASES : K2=3 Analyze -> Generalized Linear Models Generalized Linear Models. Choose Type of Model and check Binary probit. 11

12 Open Response and put Y into Dependent Variable. Open Predictors and move K37_1 ir K33_2 into Covariates (with some reservation we treat these variables as interval ones). Regressor K35_1 obtains only 4 values, therefore is treated as a categorical variable. Move K35_1 into Factors. 12

13 Open Model window and move all regressors to the right: Open Save and check Predicted category and Cook s distance. 3. Results Model is constructed for P(Y = 0). In Categorical Variable Information we check that there is sufficient number of respondents for each value of categorical variables (Y included). Categorical Variable Information N Percent Dependent Variable Y % % Factor K35_1 Esamo darbo atitikimas bakalauro (vientisųjų) studijų krypčiai Total % 1 Tikrai taip % 2 Greičiau taip % 3 Greičiau ne % 4 Tikrai ne % Total % In Omnibus Test table we check that p-value for the maximum likelihood test is sufficiently small p = 0,00...< 0,05. 13

14 Omnibus Test a Likelihood Ratio Chi-Square df Sig Dependent Variable: Y Model: (Intercept), K35_1, K37_1, K33_2 Parameter Estimates table contains parameter estimates and Wald tests for the significance of each regressor. We do not check the significance of Intercept. Categorical variable K35_1 was replaced by 4 dummy variables, one of which is not statistically significant. However, for one such insignificant result, it is not rational to drop K35_1 from the model. Parameter Estimates 95 % Wald Confidence Interval Hypothesis Test Wald Chi- Parameter B Std. Error Lower Upper Square df Sig. (Intercept) [K35_1=1] [K35_1=2] [K35_1=3] [K35_1=4] 0 a K37_ K33_ (Scale) Dependent Variable: Y 1 b Model: (Intercept), K35_1, K37_1, K33_2 a. Set to zero because this parameter is redundant. b. Fixed at the displayed value. We obtained four models, which differ by the constant only, They can be written in the following way: P (YY = 0) = P(rarely applies knowledge in his work) = Φ(zz), 1,57, if KK35_1 = 1, 1,02, if KK35_1 = 2, zz = 4,85 0,273 KK37 1 0,78 KK ,26, if KK35_1 = 3, 0, if KK35_1 = 4. 14

15 Signs of coefficients agrre with general logic of the models. Coefficient for K37_1 is negative. The larger value of K37_1 (more happy with his work), the less probable that knowledge is rarely used. Other signs of coefficients can be explained similarly. We treated probit regression as a partial case of the generalized linear model. Therefore, one can check the magnitude of deviance in the table Goodness of Fit. We see that deviance is close to unity (1,156), which demonstrates good fit of the model. Note that for the probit regression more important is small p-value of the maximum likelihood test (it can be find in Omnibus Test.). If all model s characteristics except deviance show good model fit, we assume that the model is acceptable. Goodness of Fit b Value Df Value/df Deviance Scaled Deviance Pearson Chi-Square Scaled Pearson Chi-Square Log Likelihood a Akaike's Information Criterion (AIC) Finite Sample Corrected AIC (AICC) Bayesian Information Criterion (BIC) Consistent AIC (CAIC) Checking for outliers we choose Analyze Descriptive Statistics Descriptives. Move variable CooksDistance intovariable(s). Choose OK. Descriptive Statistics N Minimum Maximum Mean Std. Deviation CooksDistance Cook's Distance Valid N (listwise) 322 Maximal Cook s distance value is 0,039<1. Therefore, there is no outliers in our data. To obtain classification table we choose Analyze Descriptive Statistics Crosstabs. Į Move Y into Row(s) and PredictedValue. into Column(s). Choose Cells and check Row. Then Continue and OK. 15

16 Y * PredictedValue Predicted Category Value Crosstabulation PredictedValue Predicted Category Value Total Y.00 Count % within Y 66.0% 34.0% 100.0% 1.00 Count % within Y 7.7% 92.3% 100.0% Total Count % within Y 25.8% 74.2% 100.0% From 100 respondents, who rarely use professional skills obtained during studies, 66 are correctly classified ( 66 %). From 222 respondents, who frequently use professional skills obtained during studies, 205 are correctly classified ( 92,3 %). Recalling the table Categorical Variable Information and its percents (respectively 31,1 % and 68,9 % ), we see that probit model ensures much better forecasting than random gues. Final conlusion: probit regression model fits data sufficiently well. 4. Forecasting One value can be forecasted in the following way. Let us assume that previous model is applied to respondent, for whom K33_2 = 4, K35_1 = 1, K37_1 = 4. We add additional row in data writing 4 in the column K33_2, 1 in the column K35_1, 4 in the column K37_1 and 1 in the column filter_$. Remaining columns are empty. We repeat probit analysis but check Predicted value of mean of response in window Save In data appears new column MeanPredicted containing probabilities for P( Y = 0). We got 0,175 probability for our respondent. Therefore is likely that this respondent will apply skills from studies in his professional work. 16

17 17

18 3. POISSON REGRESSION We are keeping count for some quite rare events. Variable Y is called dependent variable, variables X, Z, W are independent variables or regressors or predictors. We are modelling the behavior of the mean value of Y: μμ = e zz, zz = CC + bb 1 XX + bb 2 ZZ + bb 3 WW. Estimates for the coefficients CC, bb 1, bb 2, bb 3 are calculated from data. Data Y has Poisson distribution. We expect the mean of Y to be similar by its magnitude to the variance of Y. Possible values for Y are 0, 1,2,3,... Regressors can be interval or categorical random variables. Main steps 1) Check if Y has Poisson distribution. 2) Check if normed deviance is close to 1. 3) Check if maximum likelihood is statistically significant. If p-value 0,05, model is unacceptable. 4) Check if all regressors are significant (Wald test p < 0,05). If not drop them from the model. We do not pay attention to p-value for model constant (intercept). 18

19 Good Poisson regression model: Normed deviance is close to 1; Maximum likelihood has p < 0,05. For all regressors Wald test p < 0,05. Poisson regression with SPSS 1. Data File ESS4FR. Variables: agea respondents age, hhmmb number of household members, imptrad important to keep traditions eduyrs years of formal schooling, cldcrsv help for childcare (0 very bad,..., 10 very good). We will model number of other than respondent household members by agea and cldrsv. We will investigate respondents for whom imptrad 2 and eduyrs 10. Use Select Cases -> If condition is satisfied -> If and write imptrad <= 2 & eduyrs <= 10. Then Continue -> OK. Dependent variable (we call it numbhh) can be created with the help of Transform Compute Variable. 2. Preliminary analysis First we check if numbhh is similar to Poisson variable. Analyze Descriptive Statistics Frequences. 19

20 Further Statistics. Check Mean and Variance. We see that the mean of numbhh (1,0036) is close to its variance (1,482). Thus, numbhh satisfies one of the most important properties of the Poisson variable. Statistics numbhh N Valid 281 Missing 0 Mean Variance It is possible also to check if random variable has Poisson distribution with the help of Kolmogorov- Smirnov test: Analyze Nonparametric Tests Legacy Dialogs 1-Sample K-S. 20

21 We see that we can assume that numbhh has Poisson distribution (p = 0,169), but is not normal (p = 0,000). One-Sample Kolmogorov-Smirnov Test numbhh N 281 Normal Mean Parameters a,b Std. Deviation Most Extreme Absolute.302 Differences Positive.302 Negative Kolmogorov-Smirnov Z Asymp. Sig. (2-tailed).000 a. Test distribution is Normal. One-Sample Kolmogorov-Smirnov Test 2 numbhh N 281 Poisson Mean Parameter a,b Most Extreme Absolute.066 Differences Positive.066 Negative Kolmogorov-Smirnov Z Asymp. Sig. (2-tailed).169 a. Test distribution is Poisson. 3. SPSS Poisson regression options Choose Analyze Generalized Linear Models Generalized Linear Models. Choose Type of Model and check Poisson loglinear. 21

22 Click on Response and move numbhh into Dependent Vriable. Click on Predictors and move both regressors agea and cldcsrv into Covariates. (We do not have categorical variables, which are moved into Factors). After choosing Model both variables should be moved into Model. In Statistics check in addition Include exponential parameter estimates. Then -> OK. 22

23 4. Results In Goodness of Fit table we can find normed deviance. We see that the normed deviance is close to 1 (0,919). Thus, the Poisson regression model fits our data. It remains to decide which regressors are statistically significant. Goodness of Fit b Value df Value/df Deviance Scaled Deviance Pearson Chi-Square Scaled Pearson Chi-Square Log Likelihood a Akaike's Information Criterion (AIC) Finite Sample Corrected AIC (AICC) Bayesian Information Criterion (BIC) Consistent AIC (CAIC) In the table Omnibus Test we find p-value for maximum likelihood statistic. Since p < 0,05, we conclude that not all regressors are statistically insignificant. Omnibus Test a Likelihood Ratio Chi-Square df Sig

24 In the table Tests of Model Effects we see Wald test p-values for all regressors. We do not check p-value for intercept. Both p < 0,05. Therefore, we conclude that both regressors (agea and cldcrsv) are statistically significant and should remain in the model. Tests of Model Effects Type III Source Wald Chi-Square df Sig. (Intercept) agea cldcrsv In the table Parameter Estimates information about Wald p-values is repeated. Moreover, the tabale contains estimates of the model s coefficients (Column B). Parameter Estimates 95 % Wald Confidence Interval Hypothesis Test 95 % Wald Confidence Interval for Exp(B) Std. Wald Chi- Parameter B Error Lower Upper Square df Sig. Exp(B) Lower Upper (Intercept) agea cldcrsv (Scale) 1 a We can see that coefficient for agea is negative: -0,035 < 0. This means that when respondents age increases, the number of household members decreases. Mathematical model s expression is µ = exp {1,535 0,035 aaaaaaaa + 0,099cccccccccccccc}. Here µ is the mean value of other household members. Forecasting means that we insert given values of agea and cldcrsv into above formula. 5. Categorical regressor Categorical regressors are included into Generalized Linear Models - Predictors -> Factors 24

25 Do not forget to add ctzntr into Model window. Then, in the table Parameter Estimates 95 % Wald Confidence Std. Interval Parameter B Error Lower Upper (Intercept) agea [ctzcntr=1] [ctzcntr=2] 0 a... cldcrsv We get additional information about both ctzcntr. Model then can be written as ln µ = 1,239 0,036aaaaaaaa + 0,104cccccccccccccc + 0,352, if cccccccccccccc = 1, 0, if cccccccccccccc = 2. 25

26 4. NEGATIVE BINOMIAL REGRESSION We are keeping count for some events. Variable Y is called dependent variable, variables X, Z, W are independent variables or regressors or predictors. We are modelling the behavior of the mean value of Y: μμ = e zz, zz = CC + bb 1 XX + bb 2 ZZ + bb 3 WW. Estimates for the coefficients CC, bb 1, bb 2, bb 3 are calculated from data. NB regression is an alternative to the Poisson regression. The main difference is that the variance of Y is larger than the mean of Y. Data Y has negative binomial distribution. We expect the mean of Y to be smaller than the variance of Y. Possible values for Y are 0, 1,2,3,... Regressors can be interval or categorical random variables. Main steps 5) Check if the variance of Y is greater than the mean of Y. Otherwise, the NB regression is not applicable. 6) Check if normed deviance is close to 1. 7) Check if maximum likelihood is statistically significant. If p-value 0,05, model is unacceptable. 8) Check if all regressors are significant (Wald test p < 0,05). If not drop them from the model. We do not pay attention to p-value for model constant (intercept). 26

27 Good Negative Binomial regression model: Normed deviance is close to 1; Maximum likelihood has p < 0,05. For all regressors Wald test p < 0,05. Negative binomial regression with SPSS 1. Data File ESS4SE. Variables: emplno respondent s number of employers, emplnof father s number of employees, (1 if no empoye, 2 has 1 24 employees, 3 more than 25 employees), brmwmny borrow money for living (1 is very difficult,..., 5 very easy), eduyrs years of formal schooling. We will model the dependence of emplno from emplnof, brwmny, eduyrs. Variable emplnof has only one observation greater than 26. Therefore, with recode we create a new dichotomous variable emplnof2, (0 if no employees, 1 at least one employe). 2. SPSS options for the negative binomial regression Analyze Generalized Linear Models Generalized Linear Models. Click on Type of Model. Do not choose Negative binomial with log link. 27

28 Check Custom --> Distribution -> Negative binomial, Link function Log, Parameter Estimate value. Click on Response and put emplno into Dependent Variable. In Predictors put both variables eduyrs and brwmny into Covariates. Categorical variable emplnof2 put into Factors. 28

29 Choose Model and put all variables into Model window. 3. Results At the beginning of output we see descriptive statistics. Observe that standard deviation of emplno (moreover, its variance) is greater than mean. Categorical Variable Information N Percent Factor emplnof % % Total % Continuous Variable Information N Minimum Maximum Mean Std. Deviation Dependent Variable Covariate emplno Number of employees respondent has/had eduyrs Years of full-time education completed brwmny Borrow money to make ends meet, difficult or easy In Goodness of Fit table we see that normed deviance is 0,901, that is we see quite good overall model fit to data. Goodness of Fit b Value df Value/df Deviance Scaled Deviance

30 Omnibus Test table contains maximum likelihood statistics and its p-value. Since p < 0,05, we conclude that at least one regressor is statistically significant. Tests of Model Effects contains Wald tests for each regressor. All regressors are statistically significant (we do not check p-value for intercept). Tests of Model Effects Type III Source Wald Chi- Square df Sig. (Intercept) emplnof Eduyrs Brwmny Omnibus Test a Likelihood Ratio Chi- Square df Sig Parameter Estimates table contains parameter estimates (surprise, surprise) Parameter Estimates 95 % Wald Confidence Interval Hypothesis Test 95 % Wald Confidence Interval for Exp(B) Wald Std. Chi- Parameter B Error Lower Upper Square df Sig. Exp(B) Lower Upper (Intercept) [emplnof2=.00] [emplnof2=1.00] 0 a Eduyrs Brwmny (Scale) (Negative binomial) 1 b Estimated model: 0, if eeeeeeeeeeeeee2 = 1, ln µ = 1, ,286 eeeeeeeeeeee 0,753 bbbbbbbbbbbb + 1,629, if eeeeeeeeeeeeee2 = 0. Here µ is mean value of employees. 30