Logistic Regression in R. by Kerry Machemer 12/04/2015

Transcription

1 Logistic Regression in R by Kerry Machemer 12/04/2015

2 Linear Regression {y i, x i1,, x ip } Linear Regression y i = dependent variable & x i = independent variable(s) y i = α + β 1 x i1 + + β p x ip + ε i i = (1,,n) Assumptions: Normally distributed Weak exogeneity (low correlation between Xs) Linearity Constant variance Independence of errors (response variables are uncorrelated) Lack of multicollinearity in the predictors Simple Linear Regression Example Interpretation: β j is the expected change in y for a one-unit change in x j when the other covariates are held fixed

3 General Linearized Model (GLM) Flexible generalization of ordinary LR GLMs allow response variables arbitrary distributions Examples Values that must be positive with varying rates of change (geometric distribution) Probability (Bernoulli distribution) values are bounded between 0 & 1 Count Data (Poisson distribution) arbitrary function of the response variable (the link function) to vary linearly with the predicted values Assumptions Independence of each data points Correct distribution of the residuals Correct specification of the variance structure Linear relationship between the response and the linear predictor

4 Link Functions Provides the relationship between the linear predictor and the mean of the distribution function via Wikipedia GLMs

5 Logistic Regression Bernoulli, Binomial, Categorical & Multinomial distributions Predicted parameter is one or more probabilities, i.e. real numbers in the range [0,1] Bernoulli and binomial distributions Predicted parameter - single probability likelihood of occurrence of a single event Categorical and multinomial distributions Predicted parameter K-vector of probabilities all probabilities must add up to 1 Categorical Probability indicates likelihood of occurrence of one of the K possible values Multinomial and vector form of the categorical distribution the expected values of the elements of the vector can be related to the predicted probabilities similarly to the binomial and Bernoulli distributions.

6 Logistic Regression Example with mtcars Data ( mtcars dataset Model the probability of a manual transmission in a vehicle based on its engine horsepower and weight data Column am: type of the automobile model (0 = automatic, 1 = manual) hp : horsepower wt = weight General Formula: E y = 1/(1 + e (α+ k β k I k ) ) mtcars Example Formula: P Manual Transmission = 1/(1 + e (α+β 1 Horsepower+β 2 Weight) )

7 Estimated Logistic Regression Equation General Formula: Estimate y = 1 x 1,.. xp = 1/(1 + e (α+ k β k x k ) ) coefficients α and β k (k = 1, 2,..., p) are determined via maximum likelihood approach and allow us to estimate probability of the dependent variable y, taking on the value 1 for given values of x k (k = 1, 2,..., p). α (slope intercept) value where the regression line crosses the y-intercept β k (slope) steepness of the regression line PROBLEM: estimate the probability of a vehicle being fitted with a manual transmission if it has a 120hp engine and weighs 2800 lbs. GLM with binomial family R formula am.glm = glm(am ~ hp + wt, data=mtcars, family=binomial)

8 Estimated Logistic Regression Equation predict obtains predictions and optionally estimates standard errors of those predictions from a fitted generalized linear model object ( wrap the test parameters inside a dataframe newdata newdata = data.frame(hp=120, wt=2.8) apply the function predict to the GLM am.glm along with newdata. Select response prediction type predict(am.glm, newdata, type="response") Answer: For an automobile with 120hp engine and 2800 lbs weight, the probability of it being fitted with a manual transmission is about 64%

9 Significance Test for Logistic Regression Decide whether there is any significant relationship between the dependent variable y and the independent variables x k (k = 1, 2,..., p) in the logistic regression equation. In particular, if any of the null hypothesis that β k = 0 (k = 1, 2,..., p) is valid, then x k is statistically insignificant in the logistic regrssion model. Problem Using 0.05 significance level, decide if any of the independent variables in the logistic regression model of vehicle transmission in data set mtcars is statistically insignificant R formula summary(am.glm) logistic regression coefficients give the change in the log odds of the outcome for a one unit increase in the predictor variable Answer: As the p-values of the hp and wt variables are both less than 0.05, neither hp or wt is insignificant in the logistic regression model

10 CIs using profiled log-likelihood confint(am.glm) CIs using standard errors confint.default(am.glm) Odds Ratio exp(coef(am.glm)) Odds Ratio and CIs exp(cbind(or = coef(am.glm), confint(am.glm)))