Introduction To Logistic Regression

Transcription

1 Introduction To Lecture 22 April 28, 2005 Applied Regression Analysis Lecture #22-4/28/2005 Slide 1 of 28

2 Today s Lecture Logistic regression. Today s Lecture Lecture #22-4/28/2005 Slide 2 of 28

3 Background Previous chapters have discussed cases where the dependent variable is continuous. Why Not Linear? Result of Linear Logistic regression is a technique used when the dependent variable is categorical. As with the other techniques, independent variables may be either continuous or categorical. Today s lecture will discuss logistic regression when the categorical variable takes on two levels (binary variable). The technique can be extended for use with a multi-level categorical variable. Lecture #22-4/28/2005 Slide 3 of 28

4 Why Not Just Use Linear Regression? Why Not Linear? Result of Linear In the case of a binary response variable, the assumptions of linear regression are not valid: The relationship between X and Y is nonlinear. Error terms are heteroscedastic - technically, variance of error is a function of the conditional mean of dependent variable. Error terms are not normally distributed. Lecture #22-4/28/2005 Slide 4 of 28

5 Fitting A Square Peg Into A Round Hole Why Not Linear? Result of Linear If you proceeded in light of these false assumptions, the result would be: 1. Predicted values that are not possible (greater than a value of one, smaller than a value of zero). 2. Magnitude of the effects of independent variables may be greatly underestimated. So...what is one who has a categorical dependent variable to do? Lecture #22-4/28/2005 Slide 5 of 28

6 Modeling A Categorial Recall the General Linear Model (GLM): y = Xb + e Link Functions Logistic regression is part of a family of models called the Generalized Linear Model. The main feature of these models is that instead of using y directly, it is modeled through what is called a link function (here G( )): G(y) = Xb + e Lecture #22-4/28/2005 Slide 6 of 28

7 Modeling A Categorial The ized ending to General comes from: Link Functions The model being linear in the transformed space (after G( )). The General Linear Model (linear regression, ANOVA, and ANCOVA) being a subset of the Generalized Linear Model: G(y) = y The above link function is called the identity link. Lecture #22-4/28/2005 Slide 7 of 28

8 Link Functions Link Functions Suggested models for use with dichotomous dependent variables include: The linear probability model. The logistic model. The probit model. These models all define different link functions (G( )) of the dependent variable. Logistic regression will be discussed today because it is considered to be the most versatile. After transforming the dependent variable, logistic regression parallels least-squares regression. Estimation algorithms commonly use a technique called Iteratively Reweighted Least Squares. Lecture #22-4/28/2005 Slide 8 of 28

9 Odds Ratios Probabilities Logits Parameter Interpretation If you have one dichotomous independent variable and one dichotomous dependent variable, it is convenient to display the data in a 2 2 table. X 1 0 Y 1 a b a + b 0 c d c + d a + c b + d Assume the above table comes from an experiment where subjects were randomly assigned into the two categories of X: 1 = Treatment 0 = Placebo Lecture #22-4/28/2005 Slide 9 of 28

10 In this table, take the dependent variable Y to be either: 1 = Success 0 = Failure Odds Ratios Probabilities Logits Parameter Interpretation It is of interest to compare the proportion of "successes" in the treatment group vs. the control group (or category 1 vs. category 2). This would be a comparison between a a+c and b b+d. This above comparison can be made in a meaningful way using the Odds Ratio: OR = a/c b/d = ad bc Lecture #22-4/28/2005 Slide 10 of 28

11 Odds Ratios Odds ratios of 1 indicates no relationship between the two variables. Odds Ratios Probabilities Logits Parameter Interpretation Odds ratios > 1 indicates odds for X = 1 greater than X = 0 (according to table). Odds ratios < 1 indicates odds for X = 0 greater than X = 1 (according to table). The range of values the odds ratio can take is between zero and +. The distribution of the odds ratio is not symmetric. We can create symmetry by taking a natural logarithm (ln) of the odds ratio. Lecture #22-4/28/2005 Slide 11 of 28

12 Probability v. Odds Ratio Odds Ratios Probabilities Logits Parameter Interpretation Odds Ratio Probability Lecture #22-4/28/2005 Slide 12 of 28

13 Probability v. Logit Odds Ratios Probabilities Logits Parameter Interpretation Logit Probability Lecture #22-4/28/2005 Slide 13 of 28

14 Probabilities In our model, it is more common to express the odds ratio in terms of probabilities. Odds Ratios Probabilities Logits Parameter Interpretation For Y, consider the overall probability of success: P = a + b a + b + c + d The odds of success for Y would then be: P 1 P Lecture #22-4/28/2005 Slide 14 of 28

15 Logits and Odds Ratios Probabilities Logits Parameter Interpretation In logistic regression, a logistic transformation of the odds (referred to as logit) serves as the depending variable: log(odds) = logit(p) = ln ( P 1 P If we take the above dependent variable and add a regression equation for the independent variable, we get a logistic regression: ) logit(p) = a + bx As in least-squares regression, the relationship between the logit(p) and X is assumed to be linear. Lecture #22-4/28/2005 Slide 15 of 28

16 Parameter Interpretation Odds Ratios Probabilities Logits Parameter Interpretation Interpretation of parameters in analogous to simple linear regression, the slope is the expected change in logit(p) with a unit change in X. (change in ln(odds)). You can attach meaning to the odds by transforming the preceding equation back into an odds ratio (i.e. take the antilog, take to the power of e): P 1 P = ea+bx = e a (e b ) X Algebraic manipulation will give the equation in terms of P - the probability of success: P = ea+bx 1 + e a+bx = e (a+bx) Lecture #22-4/28/2005 Slide 16 of 28

17 : Categorical X SPSS Interpretation Pretend that the following data are from a study of admissions of males and females to a mechanical engineering program: Gender 1 - M 0 - F Admit 1 - Yes No Odds of admission: Male: 0.7/0.3 = 2.33 Female: 0.3/0.7 = 0.43 Odds Ratio (Male to Female): 2.33/0.43 = 5.44 Males are 5.44 times more likely to be admitted to the engineering program than are females. Lecture #22-4/28/2005 Slide 17 of 28

18 In SPSS Analyze...Regression...Binary Logistic SPSS Interpretation Lecture #22-4/28/2005 Slide 18 of 28

19 In SPSS SPSS Interpretation ln ( P 1 P ) = X Lecture #22-4/28/2005 Slide 19 of 28

20 Interpretation SPSS Interpretation ln ( P 1 P ) = X Note that e b = 5.447, which is the odds ratio for male to females probability of acceptance. The predicted value (probability of acceptance) for males is given by: ln ( P 1 P ) = (1) 1 P = = e ( (1)) The predicted value for females is: P = 1 = e ( (0)) Lecture #22-4/28/2005 Slide 20 of 28

21 Interpretation Note that X is dummy coded: all values are either 1 or 0. SPSS Interpretation Just like in ANOVA, the group with the zero is called the reference group. The regression coefficient for the gender variable b, then gave the odds-ratio of males to females. If there were another category, the coefficient would also give the odds ratio of that category to the reference group, females. Lecture #22-4/28/2005 Slide 21 of 28

22 : Continuous X Interpretation From Agresti (1990). Categorical Data Analysis. A sample of elderly people are given a psychiatric examination to determine whether symptoms of senility are present. One explanatory variable is the score on a subtest of the Wechsler Adult Intelligence Scale. Symptoms of senility are recorded as being either present (1) or absent (0). Lecture #22-4/28/2005 Slide 22 of 28

23 : Continuous X Interpretation ln ( P 1 P ) = X Lecture #22-4/28/2005 Slide 23 of 28

24 Plot Interpretation P(Senility Symptoms) WAIS Score Lecture #22-4/28/2005 Slide 24 of 28

25 Interpretation ln ( P 1 P ) = X Interpretation This equation shows that for every 1-unit increase in the WAIS score, the logit of the probability of showing senility symptoms drops by The intercept parameter, transformed, e a /(1 + e a ) gives the probability of senility symptoms at X = 0. The plot direction of increase would switch if b was positive (shown on next page). Lecture #22-4/28/2005 Slide 25 of 28

26 Plot of positive b a= 1.776, b=0.275 Interpretation P(Senility Symptoms) WAIS Score Lecture #22-4/28/2005 Slide 26 of 28

27 Final Thought Final Thought Next Class Logistic regression is a technique for regression when the dependent variable is categorical. This lecture serves only to introduce the topic. Entire courses are devoted to this topic (take them for more information). El Camino Ad From 1977 Or, if you seek adventure, check out Agresti s book, it s very good. If VW can do it with the Beetle, Chevy can bring back the El Camino! Lecture #22-4/28/2005 Slide 27 of 28

28 Next Time Introduction to confirmatory factor analysis/structural equation modeling. Final Thought Next Class Lecture #22-4/28/2005 Slide 28 of 28