MSH3 Generalized linear model

Transcription

1 Contents MSH3 Generalized linear model 5 Logit Models for Binary Data The Bernoulli and binomial distributions Mean, variance and higher order moments Normal limit Poisson limit Link functions Contingency Tables Maximum Likelihood Estimation Goodness of Fit Statistics Exact tests for logistic models Fishers Exact Test for binomial data Exact test for binary data Regression Diagnostics SydU MSH3 GLM (2015) First semester Dr. J. Chan 172

2 5 Logit Models for Binary Data 5.1 The Bernoulli and binomial distributions Mean, variance and higher order moments Define Y i = { 1 if the i-th subject has the attribute of interest, 0 otherwise. Then Y i follows a Bernoulli distribution with parameter π i and the pmf is f(y i ) = Pr(Y i = y i ) = π y i i (1 π i) y i The expected value and variance of Y i are E(Y i ) = µ i = π i ; and Var(Y i ) = σ 2 i = π i (1 π i ). Note: the variance is not constant as in a linear model but depends on the probability π i. Define 1. Y ij, the j-th unit in group i, follows a Bernoulli distribution, 2. the n i observations in each group are independent, and 3. each has the same probability π i of having the attribute of interest. n i Then Y i = Y ij, the number out of n i having the attribute, follows a j=1 binomial distribution with parameters π i and n i, that is, The pdf of Y i is given by Pr(Y i = y i ) = Y i B(n i, π i ). ( ni y i ) π y i (1 π i ) n i y i SydU MSH3 GLM (2015) First semester Dr. J. Chan 173

3 for y i = 0, 1,..., n i. The mean and variance of Y i are E(Y i ) = µ i = n i π i and Var(Y i ) = σ 2 i = n i π i (1 π i ). Dropping subscript i, the moment generating function for Y = Y Y n where Y j Ber(π) is M Y (t) = E{exp[t(Y Y n )]} = = n E[exp(tY j )] j=1 n [(1 π) exp(0t) + π exp(1t)] = [1 π + π exp(t)] n. j=1 The moments are E(Y i ) = M (i) i Y (0) = M t i Y (0). We have Hence M Y (t) = nπe t [1 π + πe t ] n 1 M Y (t) = nπe t [1 π + πe t ] n 2 [1 π + πe t + (n 1)πe t ] E(Y ) = M Y (0) = nπ E(Y 2 ) = M Y (0) = nπ[1 + (n 1)π] Var(Y ) = E(Y 2 ) E(Y ) 2 = nπ[1 + (n 1)π] n 2 π 2 = nπ(1 π) The cumulant generating function is K Y (t) = ln E[exp(tY )] = ln M Y (t) = n ln[1 π + π exp(t)] where κ 0 = K Y (0) = n ln 1 = 0, κ i = K (i) i Y (0) = K t i Y (0). We have K Y πe t (t) = n 1 π + πe t K Y (t) = n (1 π + πet )πe t π 2 e 2t (1 π + πe t ) 2 SydU MSH3 GLM (2015) First semester Dr. J. Chan 174

4 The first four cumulants are κ 1 = µ (mean), κ 2 = σ 2 (variance), κ 3 = γ (skewness), κ 4 (kurtosis) are κ 1 =K Y πe 0 (0) = E(Y ) = n 1 π + πe = nπ, 0 κ 2 =K Y (0) = E{[Y E(Y )] 2 } = n (1 π + πe0 )πe 0 π 2 e 2 0 (1 π + πe 0 ) 2 = nπ(1 π), κ 3 =K Y (0) = E{[Y E(Y )] 3 } = nπ(1 π)(1 2π), κ 4 =K Y (0) = E{[Y E(Y )] 4 } 3{E[(Y E(Y )) 2 ]} 2 =nπ(1 π)[1 6π(1 π)]. By Taylor expansion: f(x) = f(0) +f (0)x + f (0) 2! x 2 + f (3) (0) 3! x f (n) (0) n! x n +..., we have K Y (t) = i= Normal limit For Normal limit, κ i t n n! = µt + σ2t (since κ 0 = 0) Pr(Y y) 1 Φ(z ) and Pr(Y y) Φ(z + ) where Z ± = Y nπ ± 1 2. nπ(1 π) SydU MSH3 GLM (2015) First semester Dr. J. Chan 175

5 The error is asymphtotically is O(n 1 2) and is O(n 1 ) when π = 1 2. The rate of convergence is faster when π = 1 2 or κ 3 = 0. The moment and cumulant generating functions for the normal r.v. Z are M Z (t) = exp (µt + σ2 t 2 ) 2 and K Z (t) = µt + σ2 t 2 2. Since K Z (0) = µ 0+ σ2 0 2 = 0, K 2 Z(0) = µ+ σ2 2t 2 t=0 = µ, K Z(0) = σ 2, the cumulant tends to κ = (0, 0, 1, 0, 0... ) since µ = 0 and σ 2 = 1. Normal approximation is good when nπ(1 π) 2 and z, z Poisson limit When π 0 and n such that µ = nπ remains fixed, Pr(Y y) F P (y) where F P ( ) is the cdf of Poisson since the cumulant generating function of Y tends to K Y (t) = µ ln{1 + π[exp(t) 1]} π { } = µ(e t 1) ln [1 + π(e t 1 1)] π(e t 1) { } π o µ(e t 1) ln lim [1 + π(e t 1 1)] π(e t 1) π 0 }{{} e = µ[exp(t) 1] which is the cumulant generating function of a Poisson rv with mean µ. The order of error is O(n 1 ). Note that e x = lim n (1 + x n )n = e x. SydU MSH3 GLM (2015) First semester Dr. J. Chan 176

6 5.2 Link functions MSH3 Generalized linear model The logit (log of the odds) is the canoncial link for binomial data which maps the probabilities π i to a linear function of the covariates: η i = x π i iβ = logit(π i ) = ln R 1 π i where β is a vector of regression coefficients. The logits map probabilities from the range (0, 1) to the entire real line R. The following figure illustrates three transformations or link functions which are continuous and increasing. By solving for π i, the inverse transformation is π i = logit 1 (η i ) = exp(x i β) 1 + exp(x i β) From π i = β j π i (1 π i ), x ij a small change in x ij has a larger effect if π i is near 0.5 where all curves are steepest. Besides logit link, link functin can be any transformation that maps probabilities into the real line, in particular, the cdf F ( ) π i = F (η i ), η i = F 1 (π i ) for < η i <. SydU MSH3 GLM (2015) First semester Dr. J. Chan 177

7 This corresponds to a latent continuous r.v. Yi which follows these distributions such that { 1 if Y Y i = i > θ, 0 if Yi θ. Without lose of generality, we set θ = 0 for the standardized Yi. π i = Pr(Y i = 1) = Pr(Y i > 0). Popular choices of cdf s are the logistic (logit), normal (probit), and extreme value (complementary log-log) distributions. 1. Logistic disribution: Logistic distribution has the pdf and cdf f(y i µ i, σ 2 ) = F (y i µ i, σ 2 ) = exp( y i µ i σ ) σ [ 1 + exp( y i µ i σ )] 2 = 1 σ exp( y i µ i σ ) 1 + exp( y i µ i σ ) = π i. ( exp( y i µ i σ ) 1 + exp( y i µ i σ ) ) ( ) exp( y i µ i σ ) The mean and variance are µ and π2 σ 2 3. Setting µ = 0, σ = 1 and y i = η i, F (η i ) = π i or η i = F 1 (π i ) = logit(π i ). 2. Normal distribution: With cdf F (.) = Φ(.), the link function g = Φ 1 defined as π i = Φ(η i ) = Φ(x iβ) or η i = x iβ = Φ 1 (π i ) is called probit link. Both logistic and probit links are symmetric: g(π) = g(1 π) SydU MSH3 GLM (2015) First semester Dr. J. Chan 178

8 since ln π 1 π = ln 1 π π = ln 1 π 1 (1 π) = g(1 π) and Φ 1 (π) = Φ 1 (1 π) and hence they are popular links. They give similar results but have different variances. In logit model, by using a standard logistic error term, we have effectively set sd(y ) = σ = π/ 3. Thus the coefficients in a logit model should be standardized by dividing by π/ 3 before comparing them with probit coefficients. In the links plot, the logit line is the logit divided by π/ Log-Weibull (extreme value) distribution: With the cdf, the link function g = F 1 defined as π i = F (η i ) = 1 e eη i or η i = F 1 (π i ) = ln[ ln(1 π i )], is called the complementary log-log. The distribution is not symmetric. For small values of π i, the link function is close to the logit. As π i increases, the link function approaches infinity more slowly than either of the probit or logit. To compare results with probit analysis, one should standardize them, by dividing by sd(y ) = σ = π and 6 with logit analysis, by dividing by 2. The link function also has a direct interpretation in terms of hazard ratios and has practical applications in hazard models. In summary, ( ) π Logit: η = g(π) = ln, π = g 1 (η) = exp(η) 1 π 1 + exp(η) Probit: η = g(π) = Φ 1 (π), π = g 1 (η) = Φ(η) C. log-log: η = g(π) = ln[ ln(1 π)], π = g 1 (η) = 1 exp[ exp(η)]. SydU MSH3 GLM (2015) First semester Dr. J. Chan 179

9 Example: (Contraceptive use) There are 507 users among 1607 women, so the estimated probability is ˆπ = = Using logit link, the odds are The logit is ˆη = ln Using probit link, ˆπ 1 ˆπ ˆπ 1 ˆπ = = ln(0.4609) = ˆπ = Φ(η) = ˆη = Φ 1 (0.3155) = Using cloglog link, = to one. ˆπ = 1 exp( exp(η)) = ˆη = ln[ ln( )] = > y=c(rep(1,507),rep(0, )) > prop.sam=507/1607 > prop.sam [1] > glm1=glm(y~1, family=binomial(link=logit)) > summary(glm1) Call: glm(formula = y ~ 1, family = binomial(link = logit)) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 *** SydU MSH3 GLM (2015) First semester Dr. J. Chan 180

10 --- Signif. codes: 0?**?0.001?*?0.01??0.05??0.1??1 (Dispersion parameter for binomial family taken to be 1) Null deviance: on 1606 degrees of freedom Residual deviance: on 1606 degrees of freedom AIC: Number of Fisher Scoring iterations: 4 > eta1=glm1$coeff > eta1s=eta1/(pi/sqrt(3)) #standardize > pi1=exp(eta1)/(1+exp(eta1)) > glm2=glm(y~1, family=binomial(link=probit)) > summary(glm2) Call: glm(formula = y ~ 1, family = binomial(link = probit)) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 *** --- Signif. codes: 0?**?0.001?*?0.01??0.05??0.1??1 (Dispersion parameter for binomial family taken to be 1) SydU MSH3 GLM (2015) First semester Dr. J. Chan 181

11 Null deviance: on 1606 degrees of freedom Residual deviance: on 1606 degrees of freedom AIC: Number of Fisher Scoring iterations: 4 > eta2=glm2$coeff > pi2=pnorm(eta2,0,1) > glm3=glm(y~1, family=binomial(link=cloglog)) > summary(glm3) Call: glm(formula = y ~ 1, family = binomial(link = cloglog)) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: on 1606 degrees of freedom Residual deviance: on 1606 degrees of freedom AIC: SydU MSH3 GLM (2015) First semester Dr. J. Chan 182

12 Number of Fisher Scoring iterations: 5 > eta3=glm3$coeff > eta3s=eta3/(pi/sqrt(6)) #standardize > pi3=1-exp(-exp(eta3)) > c(pi1,pi2,pi3) (Intercept) (Intercept) (Intercept) > c(eta1,eta3,eta1s,eta2,eta3s) #eta1,3s comparable to eta2 (Intercept) (Intercept) (Intercept) (Intercept) (Intercept) Parameter estimates using logit and probit link are similar but are different from that using cloglog link. These may due to the similarities and differences between the link functions. SydU MSH3 GLM (2015) First semester Dr. J. Chan 183

13 Contingency Tables Binary data also arises in contingency tables. Suppose we have two binary variables U and W W = 0 W = 1 π ij = Pr(U = i, W = j), i, j = 0, 1 U = 0 π 00 π 01 p (3) Condition on W, the probabilities U = 1 π 10 π 11 p (4) p (1) p (2) π 10 p (1) = Pr(U = 1 W = 0) = π 10 + π 00 p (2) π 11 = Pr(U = 1 W = 1) =. π 11 + π 01 and The difference between these two probabilities on the logistic scale is [ ] [ ] [ ] [ ] [ ] p (1) p (2) π10 π11 π11 π 00 ln +ln = ln +ln = ln 1 p (1) 1 p (2) π 00 π 01 π 10 π 01 If we condition on U, then the probabilities =. π 01 p (3) = Pr(W = 1 U = 0) = π 01 + π 00 p (4) π 11 = Pr(W = 1 U = 1) =. π 11 + π 10 So [ ] [ ] p (3) p (4) ln +ln 1 p (3) 1 p (4) = ln [ π01 π 00 ] +ln [ π11 π 10 and ] [ ] π11 π 00 = ln =. π 10 π 01 Hence is a measure of association between U and W. If U and W are independent, π ij = Pr(U = i) Pr(W = j) implies = ln π 11π 00 π 10 π 01 = ln 1 = 0. SydU MSH3 GLM (2015) First semester Dr. J. Chan 184

14 Further if = 0 which implies π 10 π 01 = π 00 π 11 then U and W are independent since Furthermore Pr(U = 0) Pr(W = 0) = (π 00 + π 01 ) (π 00 + π 10 ) = π 00 (π 01 + π 10 + π 00 ) + π 10 π 01 = π 00 (π 01 + π 10 + π 00 ) + π 00 π 11 = π 00 (π 01 + π 10 + π 00 + π 11 ) = π 00 = Pr(U = 0, W = 0) Pr(U = 0, W = 1) = Pr(U = 0) Pr(U = 0, W = 0) = Pr(U = 0) Pr(U = 0) Pr(W = 0) = Pr(U = 0 ) Pr(W = 1) Thus = 0 π 11 π 00 = π 10 π 01 iff U and W are independent. Moreover Cov(U, W ) = E(UW ) E(U) E(W ) = π 11 (π 11 + π 10 ) (π 11 + π 01 ) = π 11 (1 π 10 π 01 π 11 ) π 10 π 01 = π 11 π 00 π 10 π 01. Thus U and W are independent iff π 11 π 00 = π 10 π 01 iff Cov(U, W ) = 0. This gives another reason for choosing logistic model. We can estimate and make inferences about the association between U and W. Conditional samples are common from retrospective studies, e.g. using hospital records to classify patients with lung cancer, say according to smoking history or prospective stuides, e.g. following a group of smokers and non-smokers over time and observing their cancer history. SydU MSH3 GLM (2015) First semester Dr. J. Chan 185

15 Example: (Smokers) The 2 2 table is No lung cancer (j = 0) Lung cancer (j = 1) Total Non smokers (i = 0) Smokers (i = 1) Total We classify patients according to without/ with lung cancer (W, j = 0, 1) and nonsmoker/smoker (U, smoking history, i = 0, 1) and compare the proportions of smokers for patients with and without lung cancer. Hence n 1 = 43 and n 2 = 63 are fixed in the design. Here the outcome is smoker/nonsmoker and predictor is without/ with lung cancer. It is possible to consider the study of cancer rate (as outcome) across the smoker and non-smoker groups (predictor). > nonsmoke=c(11,3) > smoke=c(32,60) > total=smoke+nonsmoke > y=cbind(smoke,nonsmoke) > cancer=factor(c(1,2)) > glm1=glm(y~cancer, family=binomial) # a saturated model with 2 par. > summary(glm1) Call: glm(formula = y ~ cancer, family = binomial) Deviance Residuals: [1] 0 0 Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) ** cancer ** --- Signif. codes: 0 *** ** 0.01 * SydU MSH3 GLM (2015) First semester Dr. J. Chan 186

16 (Dispersion parameter for binomial family taken to be 1) Null deviance: on 1 degrees of freedom Residual deviance: on 0 degrees of freedom AIC: 10.9 Number of Fisher Scoring iterations: 4 > glm1$fitted #prop. of smoker without and with lung cancer > smoke/total [1] > beta=glm1$coeff > phat=c(exp(beta[1])/(1+exp(beta[1])), exp(beta[1]+beta[2])/(1+exp(beta[1]+beta[2]))) > phat (Intercept) (Intercept) The odds of smoking (i = 1) to non-smoking (i = 0) among patients without (j = 0) and with (j = 1) lung cancer are ( ) π1 0 No lung cancer: ln 1 π 1 0 ( ) π1 1 Lung cancer: ln 1 π 1 1 = β 0 π 10 π 00 = e β 0 = exp(1.0678) = = β 0 + β 1 π 11 π 01 = e β 0+β 1 = exp(2.9957) = where π 1 0 = Pr(i = 1 j = 0) say. The odds of smoking given lung cancer to the odds of smoking given no lung cancer (the odds ratio) or the logistic difference is exp( ) = π 11π 00 = eβ0+β1 π 10 π 01 e β 0 = exp(β 1 ) = exp(1.9279) = = SydU MSH3 GLM (2015) First semester Dr. J. Chan 187

17 To test H 0 : β 1 = 0, we use MSH3 Generalized linear model z = β 1 SE(β 1 ) = = 2.806, z N (0, 1), or Deviance = 9.722, d χ 2 1. The p-values are and respectively. We reject H 0 and conclude that there is significant diff. in the odds of smoking among patients with and without lung cancer. The log-likelihood function and the deviance are l = i [ y i ln ( πi 1 π i ) + n i ln(1 π i ) + ln( n i y i ) ] D = i 2{y i ln(y i /µ i ) + (n i y i ) ln[(n i y i )/(n i µ i )]} > AIC=-2*sum(smoke*log(phat/(1-phat))+total*log(1-phat)+ log(choose(total,smoke)))+2*2 > AIC [1] > Dev=2*sum(smoke*log(smoke/(total*phat))+ nonsmoke*log(nonsmoke/(total-total*phat))) > Dev #so deviance is 0 [1] 0 > glm0=glm(y~1, family=binomial) # a null model with intercept > summary(glm0) Call: glm(formula = y ~ 1, family = binomial) Deviance Residuals: SydU MSH3 GLM (2015) First semester Dr. J. Chan 188

18 Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) e-11 *** --- Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: on 1 degrees of freedom Residual deviance: on 1 degrees of freedom AIC: Number of Fisher Scoring iterations: 5 > beta0=glm0$coeff > phat0=c(exp(beta0)/(1+exp(beta0)),exp(beta0)/(1+exp(beta0))) > phat0 (Intercept) (Intercept) > Dev0=2*sum(smoke*log(smoke/(total*phat0))+ nonsmoke*log(nonsmoke/(total-total*phat0))) > Dev0 [1] > Dev0i=sqrt(2*(smoke*log(smoke/(total*phat0))+ nonsmoke*log(nonsmoke/(total-total*phat0)))) > Dev0i (Intercept) (Intercept) Both refer to asymptotic results, not the exact tests. The Wald test can be used to calculate a 100(1 α)% confidence interval for β j : β j ± z 1 α/2 SE( β j ). If we change the outcome to cancer and perform the analysis again, we get the following result: > cancer=c(3,60) SydU MSH3 GLM (2015) First semester Dr. J. Chan 189

19 > nocancer=c(11,32) > total=cancer+nocancer > y=cbind(cancer,nocancer) > smoke=factor(c(1,2)) > glm2=glm(y~smoke,family=binomial) > summary(glm2) #beta1 the same Call: glm(formula = y ~ smoke, family = binomial) Deviance Residuals: [1] 0 0 Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) * smoke ** --- Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: e+00 on 1 degrees of freedom Residual deviance: e-15 on 0 degrees of freedom AIC: Number of Fisher Scoring iterations: 3 > glm2$fitted #prop. of cancer for non-smokers and smokers > cancer/total [1] Note that β 1 is the same as the previous model because exp(β 1 ) = exp( ) = π 11π 00 π 10 π 01. SydU MSH3 GLM (2015) First semester Dr. J. Chan 190

20 Both β 1 measure the association between cancer and smoker and hence it does not matter which factor is treated as outcome and which as predictor. SydU MSH3 GLM (2015) First semester Dr. J. Chan 191

21 Example: (Laundry Detergent) 1008 customers are cross classified in an experiment comparing two detergents, a new product X and a standard product M on three factors (Ries and Smith 1963). The table for the factorial design for y i, the number of customers out of n i who prefer the new product X, is Non-user of M β 1 Previous user of M β 2 Temperature Temperature Water softness Low γ 1 High γ 2 Low γ 1 High γ 2 Hard α 1 y ijk n ijk Medium α 2 y ijk n ijk Soft α 3 y ijk n ijk The full ANOVA model is ( ) pijk ln = µ + α i + β j + γ k + (αβ) ij + (αγ) ik + (βγ) jk + (αβγ) ijk, 1 p ijk i = 1, 2, 3; j, k = 1, 2 where p ijk = prob. of prefering X at softness i, user status j and temp k. To test H 0 : (αβγ) ijk = 0, the change in deviance is χ 2 2. The p-value is The data are consistent with H 0. To test for the 2 factor interaction, H 0 : (αβ) ij = (αγ) ik = (βγ) jk = 0, the change in deviance is = χ 2 5. The p-value is The data are consistent with H 0. > X=c(68,42,37,24,66,33,47,23,63,29,57,19) > T=c(110,72,89,67,116,56,102,70,116,56,106,48) > M=T-X > y=cbind(x,m) SydU MSH3 GLM (2015) First semester Dr. J. Chan 192

22 > soft=factor(c(1,1,1,1,2,2,2,2,3,3,3,3)) > user=factor(c(1,1,2,2,1,1,2,2,1,1,2,2)) > temp=factor(c(1,2,1,2,1,2,1,2,1,2,1,2)) > dat=data.frame(x,t,soft,user,temp) > glm2=glm(y~soft*user*temp,family=binomial) #a saturated model > summary(glm2) Call: glm(formula = y ~ soft * user * temp, family = binomial) Deviance Residuals: [1] Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) * soft soft user ** temp soft2:user soft3:user * soft2:temp soft3:temp user2:temp soft2:user2:temp soft3:user2:temp Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: e+01 on 11 degrees of freedom #12-1 Residual deviance: e-14 on 0 degrees of freedom AIC: Number of Fisher Scoring iterations: 3 SydU MSH3 GLM (2015) First semester Dr. J. Chan 193

23 > anova(glm2,test="chisq") Analysis of Deviance Table Model: binomial, link: logit Response: y Terms added sequentially (first to last) Df Deviance Resid. Df Resid. Dev P(> Chi ) NULL soft user e-06 temp soft:user soft:temp user:temp soft:user:temp e To evaluate the product loyality (user status) at different temp, the reduced model is ( ) pijk ln = µ + β j + γ k. 1 p ijk Given low temperature (γ 1 = 0) and non-users of M: p i11 = prev. users of M: p i21 = e µ 1 + e = e = µ 1 + e e µ+β e µ+β 2 = e e = < = p i11. The model highlights significant product loyality (p-value of β 1 =0.0000). The temperature effect which is just marginally insignificant probably worth a mention in a final report. Since γ 2 = < 0 for hot water, the preference for X is stronger amongst those using cold water. > glm3=glm(y~user+temp,family=binomial) SydU MSH3 GLM (2015) First semester Dr. J. Chan 194

24 > summary(glm3) Call: glm(formula = y ~ user + temp, family = binomial) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) *** user e-06 *** temp Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: on 11 degrees of freedom Residual deviance: on 9 degrees of freedom #12-3 AIC: Number of Fisher Scoring iterations: 3 > anova(glm3,test="chisq") Analysis of Deviance Table Model: binomial, link: logit Response: y Terms added sequentially (first to last) Df Deviance Resid. Df Resid. Dev P(> Chi ) NULL user e-06 temp SydU MSH3 GLM (2015) First semester Dr. J. Chan 195

25 > beta3=glm3$coeff > beta3 (Intercept) user2 temp > p1=exp(beta3[1])/(1+exp(beta3[1])) > p2=exp(beta3[1]+beta3[2])/(1+exp(beta3[1]+beta3[2])) > c(p1,p2) (Intercept) (Intercept) SydU MSH3 GLM (2015) First semester Dr. J. Chan 196

26 5.4 Maximum Likelihood Estimation Taking logs of the pmf and except for a constant involving the combinatorial terms, the log-likelihood function is n l(β) = ln L(β) y i ln(π i ) + (n i y i ) ln(1 π i ). i=1 To develop a Fisher scoring procedure for the ML estimates, we take the score function l (β) where l n [ yi = n ] i y i πi η i n y i n i π i π i = x ir β r π i=1 i 1 π i η i β r π i=1 i (1 π i ) η i n = (y i n i π i )x ir = [X (Y µ)] r i=1 since π i = π i (1 π i ) for linear logistic model and the information η i matrix E[l (β)] where ( 2 ) ( n ) l E = E (y i n i π i )x ir β r β s β i=1 s ( n = E W is a diagonal matrix of weights i=1 = [X W X] rs, n i π i (1 π i )x ir x is ) w ii = n i π i (1 π i ) = µ i (n i µ i )/n i = 1/Var(z i ) and the working dependent variable z i z i = η i + (y i µ i ) η i µ i = η i + (y i µ i )n i µ i (n i µ i ), SydU MSH3 GLM (2015) First semester Dr. J. Chan 197

27 since η i µ i = µ i ln Var(z i ) = µ i n i µ i = n i µ i µ i (n i µ i ) + µ i (n i µ i ) 2 = n 2 i µ 2 i (n i µ i ) 2Var(y i) = n i µ i (n i µ i ) and n 2 i µ 2 i (n i µ i ) 2µ i(n i µ i )/n i = [µ i (n i µ i )/n i ] 1. Hence Var(z) = W 1. The procedure is equivalent to iteratively reweighted least squares (IRLS). Given a current estimate β, we calculate the linear predictor ˆη = x β, the fitted values ˆµ = logit 1 (η), the working dependent variable z and weights W. Then we regress z on the covariates to obtain the weighted least squares estimate β = (X W X) 1 X W z. The large sample variance of the resulting estimate is given by Var( β) = (X W X) 1 X W Var(z)W X(X W X) 1 = (X W X) 1 where Var(z) = W 1 and W is the weight matrix evaluated in the last iteration. Example: (Smokers) > smoke=c(32,60) > nonsmoke=c(11,3) > total=smoke+nonsmoke > x=c(0,1) > y=smoke > n=2 > one=c(rep(1,n)) > X=cbind(one,x) > beta=matrix(1,2,1) > for (i in 1:10){ + eta=x%*%beta SydU MSH3 GLM (2015) First semester Dr. J. Chan 198

28 + pi=exp(eta)/(1+exp(eta)) + mu=total*pi + va=total*pi*(1-pi) + W=matrix(0,n,n) + for (j in 1:n) {W[j,j]=va[j]} + z=eta+(y-mu)/va + XWX=t(X)%*%W%*%X + XWXI=solve(XWX) + XWZ=t(X)%*%W%*%z + beta=xwxi%*%xwz + VA=diag(XWXI) + se=sqrt(va) + names(se) = NULL + result=c(i,beta[1],se[1],beta[2],se[2]) + print(result) + } [1] [1] [1] [1] [1] [1] [1] [1] [1] [1] > XWXI # covariance matrix one x one x SydU MSH3 GLM (2015) First semester Dr. J. Chan 199

29 5.5 Goodness of Fit Statistics 1. Deviance: a measure of discrepancy between observed and fitted values defined as n [ ( ) ( )] yi ni y i D = 2 y i ln + (n i y i ) ln χ 2 ˆµ i n i ˆµ n p, i i=1 as n i where n is the number of groups and p is the number of parameters including the constant. 2. The likelihood ratio test: to compare two nested models ω 1 ω 2 based on the difference between their deviances D(ω 1 ) and D(ω 2 ) including variables in X 1 and X with p 1 and p elements respectively where ( ) β1 X = (X 1 X 2 ) and β = the null hypothesis is H 0 : β 2 = 0 and the test statistic is D(ω 1 ) D(ω 2 ) χ 2 p 2. Likelihood ratio tests in GLM are based on scaled deviances, obtained by dividing the deviance by a scale factor, which is one for binomial data. 3. Pearson s chi-squared: the sum of the squared difference between observed and fitted values y i and ˆµ i, divided by the variance of y i, µ i (n i µ i )/n i defined as X 2 = i (z i η i ) 2 = n i(y i ˆµ i ) 2 ˆµ i (n i ˆµ i ) = i [ (yi ˆµ i ) 2 β 2 ˆµ i + [(n i y i ) (n i ˆµ i )] 2 n i ˆµ i ] χ 2 n p. This statistic is also the sum over both successes and failures of the observed minus expected squared over expected. It is asymptotically equivalent to the deviance or likelihood-ratio chi-squared statistic. SydU MSH3 GLM (2015) First semester Dr. J. Chan 200

30 5.6 Exact tests for logistic models If n i are not large enough to apply the asymptotic χ 2 tests then we construct small sample exact tests for the hypotheses of interest. These tests should be considered whenever the observed binomial values in any cell is less than 5. CLT for binomial dist. requires nπ 5 and n(1 π) 5, i.e. N is large or π is close to 1 2. Consider the model θ i = ln ( πi 1 π i ) = p x ij β j = η i where the x ij are known constants. The likelihood of an observed binary sample y 1,.., y n is L= n i=1 π y i i (1 π i) 1 y i = n i=1 e η iy i 1 + e η i j=1 [ = e i j x ijβ j y i p n i=1 (1 + =exp eη i ) j=1 β j ( n i=1 x ij y i )] / n i=1(1 + e η i ) Let t j = n x ij y i then {t 1,..., t p } is sufficient for β 1,..., β p. The joint i=1 distribution of t 1,..., t p is obtained by summing over all appropriate binary sequences which generate particular t 1, t 2,..., t p. ( ) p c(t 1,..., t p ) exp β j t j Pr(T 1 = t 1,..., T p = t p ) = j=1 n (1) (1 + e η i ) where c(t 1,..., t p ) is a combinatorial coefficient, that is, the number of combination of {x 1,..., x n } that gives the same set of {t 1,..., t p }. Since T 1,..., T p 1 are sufficient for β 1,..., β p 1, the conditional distribution of T p given T 1,..., T p 1 will only depend on β p. Pr(T p = t p T 1 = t 1,..., T p 1 = t p 1 ) = i=1 c(t 1,..., t p )e β pt p u c(t 1,..., t p 1, u)e β pu (2) SydU MSH3 GLM (2015) First semester Dr. J. Chan 201

31 5.6.1 Fishers Exact Test for binomial data Assume p = 2 and x ij = 0, 1. The data are given by a 2 2 table: Group 1 Group 2 Total Success Y 1 Y 2 S Failure n 1 Y 1 n 2 Y 2 n S n 1 n 2 n To test H 0 : β 1 = 0, we show that the conditional distribution reduces to a ratio of combinatorial terms. The probability of success is π 1 = eβ e (η β 1 = β 0 ) in Group 1 and π 2 = eβ0+β1 (η e β 2 = β 0 + β 1 ) 0+β 1 in Group 2. Given n 1 and n 2, the likelihood using (1) is L(y 1, y 2 n 1, n 2 ) = (n 1 Y 1 ) ( n 2 Y 2 )e β 0(Y 1 +Y 2 )+β 1 Y 2 (1 + e β 0 ) n 1 (1 + e β 0 +β 1) n 2 where t 0 = S = Y 1 + Y 2 and t 1 = Y 2 are sufficient statistics for β 0 and β 1, c(t 0, t 1 ) = ( n 1 Y 1 ) ( n 2 Y 2 ) choosing Y 1 1 from n 1 trials and Y 2 1 from n 2 trials to result in Y 2 1 from gp 2 out of Y 1 + Y 2 total 1 and 1 β j t j = β 0 (Y 1 + Y 2 ) + β 1 Y 2. Using (2), the conditional probability is j=0 n 2 or n 2 s n 1 y 2 0 Pr(Y 2 = y 2 S = s) = s y 1 + y 2 = s s s n 2 n 1 or n 1 y 1 ( n 1 s y 2 ) ( n 2 y 2 )e β 0s+β 1 y 2 min(n 2,s) ( n 1 s u) ( n 2 u=max(0,s n 1 ) = (ns y 1 ) ( n2 2 y 2 )e β 1y 2 ( n 1 s u) ( n 2 u )e β 1u u β 1 =0 = (n 1 s y 2 )( n 2 y 2 ) ( n s) u )e β 0s+β 1 u since s n 1 y 2 n 2, s SydU MSH3 GLM (2015) First semester Dr. J. Chan 202

32 which is the hypergeometric distribution with E(Y 2 S = s) = s n 2 n = s ˆπ 2 and (3) Var(Y 2 S = s) = n ( ) ( ) 1n 2 s n s n s = sˆπ n 2 2 (1 ˆπ 2 ) (4) n 1 n 1 ( ) n s where is the finite population correction factor for sampling n 1 s units from n without replacement. Hence an exact test of H 0 : β 1 = 0 (i.e. π 1 = π 2 ) can be obtained by calculating the p-value based on Y 2 : Y 2 n 2 s n 1 2 ) n1 n 2 s n 2 ( n s n 1 We compare it with a normal distribution if n 1 and n 2 are moderate. Example: (Lung Cancer) Test H 0 : β 1 = 0 vs H 1 : β 1 > 0, No lung cancer Lung cancer Total Group 1 Group 2 Smokers Non smokers Total We have n 1 = 43, n 2 = 63, n = 106, Y 1 = 32, Y 2 = 60 and S = 92. The values in blue are sufficient for the table. The exact p-value is Pr(Y 2 60 S = 92) = min(n 2,s) u=60 ( n 1 s u)( n 2 u ) ( n s) = 63 ( u)( 63 u ) ( 106 u=60 92 ) = With normal approximation to hypergeometric distribution, z 0 = Y 2 n 2 s n 1 ) 2 = 60 63(92)/ = 2.80 n1 n 2 s n 2 ( n s n 1 43(63)(92) The approximate p-value = Pr(Z 2.80) = SydU MSH3 GLM (2015) First semester Dr. J. Chan 203

33 5.6.2 Exact test for binary data Suppose we have n independent observations (x 1, y 1 ),..., (x n, y n ) and a linear logistic model η i = β 0 + β 1 x i. The sufficient statistics for β 0 and β 1 are n n T 0 = y i and T 1 = x i y i. i=1 Given T 0 = t 0 = s (exactly t 0 of y i is 1), the statistic T 1 is the sample total of a sample of size t 0 drawn from {x 1,..., x n } without replacement. If β 1 = 0, then the probability of success is constant over all n trials, regardless of x i, and so all distinct samples of size t 0 drawn from {x 1,..., x n } are equally likely. Let x = 1 n x i and v = 1 n (x i x) 2. If β 1 = 0 then n n i=1 i=1 i=1 E(T 1 T 0 = t 0 ) = t 0 x and Var(T 1 T 0 = t 0 ) = t 0 v ( ) n t0 n 1 We can use the normal approximation to the conditional distribution of T 1 given T 0 = t 0 to calculate the p-value for testing H 0 : β 1 = 0 or use the exact permutation test. In case if {x 1,..., x n } contains n 1 0 and n 2 1 as indicators of group 2, we have t 0 = s, t 1 = y 2, x = 1 n x i = n 2 n n. Then i=1 E(T 1 T 0 = t 0 ) = t 0 x = s n 2 ( n and ) n t0 Var(T 1 T 0 = t 0 ) = t 0 v = s n 1n 2 n 1 n 2 ( ) n s n 1 SydU MSH3 GLM (2015) First semester Dr. J. Chan 204

34 are the same as (3) and (4) since v = 1 n (x i x) 2 = 1 n n i=1 [ n 1 (0 n 2 n )2 + n 2 (1 n 2 n )2 ] = 1 n 3(n 1n n 2 n 2 1) = n 1n 2 n 3 (n 1 + n 2 ) = n 1n 2 n 2. Example: Suppose we have n = 7 trials and x i = i. We are given t 0 = 3, i.e. we observe 3 successes. The sample moments for x i ={1,2,3,4,5,6,7} are x = 4 and v = 1 7 (i 4) 2 = 4. 7 If β 1 = 0, there are ( 7 3) = 35 equally likely sets of size 3 which can be drawn. If we observe the sum of x i for the 3 successes is t 1 = 16, then the exact p-value is i=1 Pr(T 1 16 T 0 = 3, β 1 = 0) = 4 35 = as there are 4 subsets giving T 1 16 and they are (5,6,7), (4,6,7), (4,5,7), (3,6,7) with sum 18,17,16,16 respectively. Alternatively, if β 1 = 0, E(T 1 T 0 = 3) = t 0 x = 3 4 = 12 and ( ) n t0 3(4)(4 1) Var(T 1 T 0 = 3) = t 0 v = n The approx. p-value is Pr(T 1 16 T 0 = 3, β 1 = 0) Pr = 8. ( Z ) 2 8 = Pr(Z 1.237) = SydU MSH3 GLM (2015) First semester Dr. J. Chan 205

35 5.7 Regression Diagnostics 1. Pearson Residuals r pi = y i ˆµ i ˆµi (n i ˆµ i )/n i N(0, 1) Observations with r pi > 2 indicate lack of fit. 2. Deviance Residuals [ r di = 2 y i ln ( yi ) ( )] ni y i + (n i y i ) ln ˆµ i n i ˆµ i Observations with r d > 2 indicate lack of fit. Note that Deviance = i r2 di. 3. Leverage Pregibon (1981) developed a weighted hat matrix H = W 1/2 X(X X) 1 X W 1/2 where W is the diagonal matrix with entries w ii = ˆµ i (n i ˆµ i )/n i evaluated at the MLE. Then the variance of the residual to a first order approximation is Var(y i ˆµ i ) (1 h ii )Var(y i ) where h ii is the leverage or diagonal element of H. Note that Y µ = (I H)Y. Then the studentized residual is r si = r pi 1 hii = y i ˆµ i (1 hii )ˆµ i (n i ˆµ i )/n i 4. Cook distance SydU MSH3 GLM (2015) First semester Dr. J. Chan 206

36 Comparing β with β (i) and to avoid iterations, Pregibon (1981) proposed D i = h iir 2 si (1 h ii ) = h ii (y i ˆµ i ) 2 (1 h ii ) 2ˆµ i (n i ˆµ i )/n i as a one-step approximation to Cook s distance based on doing one iteration of the IRLS towards β (i) starting from the complete data estimate β. This is compared with for normal data. D i = h ii r 2 i (1 h ii ) 2 pˆσ 2 SydU MSH3 GLM (2015) First semester Dr. J. Chan 207