Introduction to the Generalized Linear Model: Logistic regression and Poisson regression

Transcription

1 Introduction to the Generalized Linear Model: Logistic regression and Poisson regression Statistical modelling: Theory and practice Gilles Guillot November 4, 2013 Gilles Guillot Intro GLM November 4, / 28

2 1 Logistic regression 2 Poisson regression 3 References Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

3 US space shuttle Challenger accident data Proportion of failures Temperature ( C) Proportion of O-rings experiencing thermal distress on the US space shuttle Challenger. Each point corresponds to a ight. Proportion obtained among 6 O-rings for each ight. Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

4 US space shuttle Challenger accident data (cont') Proportion of failures Temperature ( C) Key question: what is the probability to experience a failure if the temperature is close to 0? For the previous example, we would need a model along the line of Number of failures temperature or Proportion of failures temperature Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

5 Targeted online advertising The problem A person makes a search on Google on say 'cheap ight Yucatan' Is it worthwhile to display an ad for say 'hotel del Mar Cancún? Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

6 Targeted online advertising (cont') Google knows the preferences of this person from before Google knows on which ad this person as clicked or not clicked in the past In stat. words: Google has vector of binary explanatory variables (x 1,..., x p ) that summarizes the prospect's preferences Google knows the preferences of other persons from before who have been shown the ad for 'hotel del Mar Cancún and the same ads as the prospect. Denoting y i ( {0, 1}) the past Google knows y i and (x i1,..., x ip ) for these persons Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

7 Targeted online advertising (cont') In statistics word: We have a dataset y i, x i1,..., x ip with i = 1,..., n We have a prospect known through its vector (x 1,..., x p ) We want to predict the behavior y (0 or 1) of this person regarding the ad 'hotel del Mar Cancún We need a model along the line of y x 1,..., x p The novelty (compared to the linear model) is that y is a count What follows applies to any type (quantitative or categorical) of explanatory variables to any number (p = 1, p > 1) of explanatory variables Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

8 Hints about what is coming below In a linear regression we write y i = β 0 + β 1 x i + ε i and ε i N (0, σ 2 ) which we can rephrased as y i N (β 0 + β 1 x i, σ 2 ) Idea of a logistic regression: twinkle the equation above to t data that can not be considered Normal. Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

9 The logistic function Denition of the logistic function The logistic function is a function l : R R dened as l(x) = exp(x) 1 + exp(x) = exp( x) Logistic function l(x) x This function l is a bijection (one-to-one), with inverse g(x) = l 1 (x) = ln(y/(1 y)) Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

10 Logistic regression for a 0/1 response variable We consider here the case of a single explanatory variable x and a binary response y. The data consist of y = (y 1,..., y n ) observations of a binary (0/1) or two-way categorical response variable x = (x 1,..., x n ) observations of a quantitative explanatory variable Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

11 Logistic regression for a 0/1 response variable (cont') We want a model such that P (Y i = 1) varies with x i We assume that Y i b(p i ) We assume that p i = l(β 0 + β 1 x i ) (l logistic function) This is an instance of a so-called Generalized Linear Model Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

12 Formal denition of the logistic regression for a binary response p i = l(β 0 + β 1 x i ) for a vector of unknown deterministic parameters β = (β 0, β 1 ) Y i b(p i ) = b(l(β 0 + β 1 x i )) Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

13 Note the analogy with the linear regression y i N (β 0 + β 1 x i, σ 2 ) Under this model: E[Y i ] = p i = 1/(1 + e (β 0+β 1 x 1 ) ) V [Y i ] = p i (1 p i ). The variance of Y i around its mean is entirely controlled by p i, hence by β 0 and β 1. The model enjoys the structure: "Response = signal + noise" but we never write it explicitly nor do we make any assumption or even refer to a residual ε. The structure "Response = signal + noise" is implicit. Straightforward extension for more than one explanatory variables: p i = l( p j=1 x(j) i ) Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

14 Logistic regression for nite count data We consider data y = (y 1,..., y n ) obtained as counts over several replicates denoted N = (N 1,..., N n ). [For example, each N i is equal to 6 in the 0-rings data.] p i = l(β 0 + β 1 x i ) for a vector of unknown deterministic parameters β = (β 0, β 1 ) Y i Binom(N i, p i ) Under this model: E[Y i ] = N i p i = N i /(1 + e (β 0+β 1 x 1 ) ) V [Y i ] = N i p i (1 p i ) Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

15 Maximum likelihood inference for logistic regression Data: y = (y 1,..., y n ) x = (x 1,..., x n ) and N = (N 1,..., N n ) Parameters: β = (β 0, β 1 ). NB: there is no parameter controlling the dispersion of residuals Likelihood: n ( ) Ni L(y 1,..., y n ; β 0, β 1 ) = p y i i (1 p i) N i y i i=1 n i=1 y i p y i i (1 p i) N i y i (1) ln L(y 1,..., y n ; β 0, β 1 ) = n y i ln p i + (N i p i ) ln(1 p i ) (2) i=1 Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

16 ML inference cont' Maximizing the log-likelihood In constrast with the linear model, the maximum log-likelihood of the logistic regression does not have a closed-from solution. It has to be maximised numerically. Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

17 Model tting in R Binary response With x, y two vectors of equal lengths, y being binary: glm(formula = (y~x), family=binomial(logit)) Returns parameters estimates and loads of diagnostic tools (e.g. AIC). Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

18 Example of output of glm function Call: glm(formula = (y ~ x), family = binomial(link = logit)) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) e-14 *** x < 2e-16 *** --- Signif. codes: 0 `***' `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: on 999 degrees of freedom Residual deviance: on 998 degrees of freedom AIC: Number of Fisher Scoring iterations: 4 Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

19 General binomial reponse With x, N, y three vectors of equal lengths glm(formula = cbind(y,n-y)~x, family=binomial(logit)) Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

20 Poisson regression Poisson regression what for? In many statistical studies, one tries to relate a count to some environmental or sociological variables. For example: Number of cardio-vascular accidents among people over 60 in a US state Average income in the state? Number of bicycles in a Danish household Distance to the city centre When the response is a count which does not have any natural upper bound, the logistic regression is not appropriate. Even if there is a natural upper bound, the Binomial disitrbution may give a poor t The Poisson regression is a natural alternative to the logistic regression in these cases. Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

21 Poisson regression Poisson process, Poisson equation, Poisson kernel, Poisson distribution, Poisson bracket, Poisson algebra, Poisson regression, Poisson summation formula, Poisson's spot, Poisson's ratio, Poisson zeros... Simeon Poisson Gilles Guillot Intro GLM November 4, / 28

22 Poisson regression The Poisson distribution Denition: Poisson distribution The Poisson distribution is a distribution on N with probability mass function: P (X = k) = e λ λ k for k N, and some parameter k! λ (0, + ) Notation: X P(λ) Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

23 Poisson regression The Poisson distribution (cont) If X P(λ) E[X] = λ V ar[x] = λ If X P(λ) and Y P(µ), X, Y independent X + Y P(λ + µ) Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

24 Poisson regression Set up for the Poisson regression y 1,..., y n a discrete, positive variable, e.g. number of bicycles in n households x 1,..., x n a quantitative variable, e.g. distance of household to city centre or household income Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

25 Poisson regression Poisson regression, denition A Poisson regression is a model relating an explanatory variable x to a positive count variable y with the following assumptions: Denition: Poisson regression y 1,..., y n are independant realizations of Poisson random variables Y 1,..., Y n with E[Y i ] = µ i ln µ i = αx i + β For short Y i P(exp(αx i + β)) αx i + β is called the linear predictor the function that relates the linear predictor to the mean of Y i is called the link function Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

26 Poisson regression Poisson distribution tting in R glm.res = glm(formula = (nb.cycles ~ dist), family = poisson(link="log"), data = dat) Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

27 Poisson regression Output of glm() function summary(glm.res) Call: glm(formula = (nb.cycles ~ dist), family = poisson(link = "log"), data = dat) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) e-06 *** dist < 2e-16 *** --- Signif. codes: 0 `***' `**' 0.01 `*' 0.05 `.' 0.1 ` ' 1 (Dispersion parameter for poisson family taken to be 1) Null deviance: on 99 degrees of freedom Residual deviance: on 98 degrees of freedom AIC: Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28

28 References Reading A Modern Approach to Regression with R, Chapter 8 Logistic Regression, Sheather, Springer Gilles Guillot (gigu@dtu.dk) Intro GLM November 4, / 28