Lecture 6 STK Categorical responses

Transcription

1 Lecture 6 STK Categorical responses 22. September 2014 Plan for lecture: 1. GLM for binary and binomial data 2. Link functions 3. Parameter interpretation in logistic regression 4. Parameter interpretation with other link functions 5. Goodness-of-fit: Hosmer-Lemeshow-test 6. ROC curves 7. Over dispersion Lecture 6 STK Categorical responses p. 1

2 Binomial responses AssumeY i Bin(n i,π i ) and independent The data belongs to the exponential family with pmf ( ) ni f(y,θ i,φ i ) = π y i y (1 π i) n i y i θ i = log(π i /(1 π i )) a(θ i ) = n i log(1+exp(θ i )) =c(y)exp(yθ i a(θ i )) dispersion parameter φ i = 1 and known andc(y) = ( n i y E[Y i ] = a (θ i ) = n i exp(θ i ) 1+exp(θ i ) = n iπ i = µ i Var[Y i ] = φ i a (θ i ) = n i exp(θ i ) (1+exp(θ i )) 2 = n i π i (1 π i ). ) Lecture 6 STK Categorical responses p. 2

3 Binomial or binary responses? AssumeY i Bin(n i,π i ) and independent The data can also be represented as 1 forj = 1,...,Y i Y i,j = 0 forj = Y i +1,...,n i which gives us binary data Note: IfY i,j Bin(1,π i ), buty i,j -s are dependent within groupi, the sum are not binomial Positive dependence give overdispersion Grouping, and then taking into account over dispersion, may be a way to handle such data Lecture 6 STK Categorical responses p. 3

4 Binary responses or grouped data? Y i Bin(n i,π i ),i = 1,...,k or Y i Bin(1,π i ),i = 1,...,n = i=k i=1 n i Estimation equivalent for both representations AIC for comparing models are also equivalent The deviance goodness-of-fit test becomes different! χ 2 n q n = k for grouped data n = k i=1 n i for binary data To trust the deviance goodness-of-fit test, we require: Y i Bin(n i,π i ) where n i π i > 5 andn i (1 π i ) > 5 Lecture 6 STK Categorical responses p. 4

5 Ex: Beetles > dim(beetle) [1] 8 3 > glm(cbind(dode,ant-dode) Dose,family=binomial,data=beetle) Coefficients: (Intercept) Dose Degrees of Freedom: 7 Total (i.e. Null); Null Deviance: Residual Deviance: AIC: Residual > dim(beetle2) [1] > glm(dode Dose,family=binomial,data=beetle2) Coefficients: (Intercept) Dose Degrees of Freedom: 480 Total (i.e. Null); 479 Residual Null Deviance: Residual Deviance: AIC: Lecture 6 STK Categorical responses p. 5

6 GLM for binomial or binary (n i = 1) responses IndependentY i with probability for successπ i Linear predictorη i = β T x i Link functiong(π i ) = η i The logit link function is the most usual: which gives π i g(π i ) = log( ) = logit(π i ) 1 π i π i = exp(η i) 1+exp(η i ) = g 1 (η i ) This is the canonical link function, i.e. canonical parameter θ i = η i The logit link yield logistic regression Lecture 6 STK Categorical responses p. 6

7 Requirements for link function for binomial data g() should be smooth (can be differentiated) be strongly monotone (increasing) take values over all real numbers g([0,1]) = R or equivalentg 1 (R) = [0,1] g 1 (η) cumulative distribution function (CDF) for a continuous distribution onr Logit link satisfies these requirements. g 1 (η) is CDF in "standard" logistic distribution with density exp(η) (1+exp(η)) 2 Lecture 6 STK Categorical responses p. 7

8 CDF and pdf in "standard" logistic distribution Kumulativ logistisk fordeling Tetthet logistisk fordeling F(x) f(x) x pdf is symmetric aroundx = 0, hence expectation is 0 The variance is x 2 exp(x) (1+exp(x)) 2dx = π2 3 = Lecture 6 STK Categorical responses p. 8 x

9 Probit link: Inverse of CDF for standard normal where Φ(y) = y However, g(η) = Φ 1 (η) 1 2π exp( 1 2 x2 )dx Since the pdf in the standard normal distribution also is symmetric around 0, with probit link we often get results that are comparable with those from logistic regression However, the logistic distribution has heavier tails than the normal, and in some situations the probit link may be better Lecture 6 STK Categorical responses p. 9

10 CDF and pdf for logit and probit Kumulative fordelingsfunksjoner Tettheter F(x) logistisk probit (skalert) f(x) x x Lecture 6 STK Categorical responses p. 10

11 Comparing estimates from logit and probit E[Y i ] =g 1 (η i ) g 1 (0)+(g 1 ) (0)η i ηi l logit = 0.5+φ (0)η p i probit I.e. forη i 0, since φ (0) = 1/sqrt2π, ηi l (φ (0)/0.25)η p = (8/π)η p 1.6η p or β l j 1.6β p j Lecture 6 STK Categorical responses p. 11

12 R-output beetles: Logit vs. Probit > logfit<-glm(cbind(dode,ant-dode) Dose,binomial(link=logit),beetle) > profit<-glm(cbind(dode,ant-dode) Dose,binomial(link=probit),beetle) > logfit Coefficients: (Intercept) Dose Degrees of Freedom: 7 Total (i.e. Null); 6 Residual Null Deviance: Residual Deviance: AIC: > profit Coefficients: (Intercept) Dose Degrees of Freedom: 7 Total (i.e. Null); 6 Residual Null Deviance: Residual Deviance: AIC: > logfit$coef/profit$coef (Intercept) Dose Lecture 6 STK Categorical responses p. 12

13 Akaike information criterion (AIC) AIC = 2ˆl+2q q = number of parameters in the model ˆl is the maximum log-likelihood under the model AIC are used for model selection The model with lowest AIC model are the best according to this criterion Lecture 6 STK Categorical responses p. 13

14 R-output beetles: Logit > summary(logfit) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 *** Dose <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: on 7 degrees of freedom Residual deviance: on 6 degrees of freedom AIC: Lecture 6 STK Categorical responses p. 14

15 R-output beetles: Probit > summary(profit) Deviance Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 *** Dose <2e-16 *** --- Signif. codes: 0 *** ** 0.01 * (Dispersion parameter for binomial family taken to be 1) Null deviance: on 7 degrees of freedom Residual deviance: on 6 degrees of freedom AIC: Number of Fisher Scoring iterations: 4 Lecture 6 STK Categorical responses p. 15

16 clog-log-link based on the Gumbel distribution The linkη i = g(π i ) = log( log(1 π i )) is called the "complementary log-log-link" Its inverse is given by π i = 1 exp( exp(η i )) = F(η i ) which is CDF for (the standardized) Gumbel distribution Properties: not symmetric light tail towards + tails as the logistic distributions towards expectation = Euler s constant 0.58 variance π 2 / Lecture 6 STK Categorical responses p. 16

17 CDF and pdf in Gumbel distribution Kumulative fordelingsfunksjon Gumbel Tetthet Gumbel F(x) f(x) x x Lecture 6 STK Categorical responses p. 17

18 R-output beetles: Clog-log > clogfit<-glm(cbind(dode,ant-dode) Dose,binomial(link=cloglog),beetle) > summary(clogfit) Coefficients: Estimate Std. Error z value Pr(> z ) (Intercept) <2e-16 *** Dose <2e-16 *** Null deviance: on 7 degrees of freedom Residual deviance: on 6 degrees of freedom AIC: Number of Fisher Scoring iterations: 4 > logfit$coef/clogfit$coef (Intercept) Dose Lecture 6 STK Categorical responses p. 18

19 Comparing link functions by AIC > AIC(logfit,profit,clogfit) df AIC logfit profit clogfit clog-log-link gives the lowest AIC Since all three models has the same number of parameters, it also gives the highest log-likelihood, i.e. the best fit Lecture 6 STK Categorical responses p. 19

20 Fitted probabilities for beetle data with logit link and clog-log link: andel dode biller logistisk cloglog dose (log_10) The clog-log link fits observed proportions better than logit link, with residual deviance 3.45 for clog-log and for logit Lecture 6 STK Categorical responses p. 20

21 Including 2. order term of dose > form = cbind(dode,ant-dode) Dose+I(Doseˆ2) > logfit2<-glm(form,binomial(link=logit),beetle) > profit2<-glm(form,binomial(link=probit),beetle) > clogfit2<-glm(form,binomial(link=cloglog),beetle) > AIC(clogfit,logfit2,profit2,caufit2,clogfit2) df AIC clogfit logfit profit clogfit Lecture 6 STK Categorical responses p. 21

22 Fitted probabilities for beetle data including also models with quadratic terms of Dose andel dode biller logistisk cloglog logistisk, 2. gradsledd dose (log_10) clog-log link: Quadratic term yields residual deviance 3.19 compared to 3.44 with only linear term Lecture 6 STK Categorical responses p. 22

23 Interpretation of parameters in logistic regression The odds for an event is defined: π = Odds 1 π In logistic regression, withη = β T x, the odds are Odds = exp(η) 1+exp(η) 1 exp(η) 1+exp(η) = exp(η) 1+exp(η) 1 1+exp(η) = exp(η) i.e. η = log Odds Lecture 6 STK Categorical responses p. 23

24 Interpretation of parameters in logistic regression: Odds-ratio Let x k = x k,k j,x j = x j +1, i.e. x x = (0,...,0,1,0,...,0), The ratio between two odds with explanatory variables x andx is called the odds-ratio, (with π = e η /(1+e η ) andη = β T x ) 1 π π 1 π OR j = π = exp(β j ) = Odds Odds = exp(η η) = exp(β T (x x)) or β j = log(or j ), i.e the regression coefficients are log-odds-ratios or relative change in odds on the log scale Lecture 6 STK Categorical responses p. 24

25 Odds-ratio Relative Risk (RR) when the probabilities are small Relative risk is defined as the ratio between two probabilities: RR = π π When bothπ andπ are small, 1 π 1 and1 π 1. Therefore, OR = π π 1 π 1 π π π = RR I.e., when the probabilities are small, exp(β j ) expresses approximately the relative change in probability whenx j is increased by one unit Lecture 6 STK Categorical responses p. 25

26 The approximation OR RR Relative risk Odds-ratio π π = π = π = π = π = π = π = Lecture 6 STK Categorical responses p. 26

27 Interpretation of parameters with clog-log-link π =1 exp( exp(β T x)) or η =β T x = log( log(1 π)) If π is small, then log(1 π) π (Taylor) which gives η log(π) π exp(η) and thus RR j = π π exp(β j) Lecture 6 STK Categorical responses p. 27

28 Ex: Mortality by Wilm s tumor 444 dead, 3471 survivors > glm(d unfav+factor(stg),family=binomial(link=logit), data=nwts)$coef (Intercept) unfav factor(stg)2 factor(stg)3 factor(stg) > glm(d unfav+factor(stg),family=binomial(link=cloglog), data=nwts)$coef (Intercept) unfav factor(stg)2 factor(stg)3 factor(stg) Lecture 6 STK Categorical responses p. 28

29 Interpretation of parameters with probit link Sometimes we may have continuous responses, for instance normal distributed,y i0 N(β T x i,σ 2 ), but still prefer to study 1 ify i0 < γ = threshold value Y i = 0 if not Ex; Y i0 = birth weight Y i = 1 ify i0 < 2800 gram 0 if not Ex: Psychometric measurements, Y i0 = score on a depression scale Y i = 1 ify i0 < threshold value 0 if not Lecture 6 STK Categorical responses p. 29

30 Underlying scale Y i = 1 ify i0 < γ = threshold value 0 if not tetthet Y0 Lecture 6 STK Categorical responses p. 30

31 Probit, cont. Why binary response? Tradition to do table analysis Direct scorey i0 may have a skew distribution Direct score may not be registered, only an underlying scale we imagine exists ("latent" variable) The relation between Y i0 N(β T x i,σ 2 ) Y i = I(Y i0 γ) is given by π i = P(Y i = 1) = P(Y i0 γ) = Φ( γ σ (β σ ) x i ) Lecture 6 STK Categorical responses p. 31

32 Relationship between parameters on probit and underlying scale E[Y i0 ] = β T x i = β 0 +β 1 x i1 + +β p x ip is equivalent to the linear predictor on probit scale where α 0 = γ β 0 σ Φ 1 (π i ) = α 0 +α 1 x i1 + +α p x ip α j = β j σ forj = 1,...,p Note: The standard deviationσ on the underlying scale can not be identified by the probit analysis Lecture 6 STK Categorical responses p. 32

33 Ex: Birth weight and gestational age > summary(lm(vekt svlengde+sex)) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) svlengde e-06 *** sex * --- Residual standard error: on 21 degrees of freedom Multiple R-Squared: 0.64, Adjusted R-squared: F-statistic: on 2 and 21 DF, p-value: 2.194e-05 Here is ˆσ = Lecture 6 STK Categorical responses p. 33

34 Ex: Birth weight and gestational age cont. Defines Y i = 1 if birth weight is less than 2800 gram > lavvekt<-1*(vekt<2800) > table(lavvekt) > > glm(lavvekt svlengde+sex,family=binomial(link=probit))$coef (Intercept) svlengde sex > -lm(vekt svlengde+sex)$coef/177.1 (Intercept) svlengde sex Approximately probit-estimates from linear regression: ˆα j ˆβ j ˆσ Lecture 6 STK Categorical responses p. 34

35 Goodness of fit tests for binomial data If Y i Bin(n i,π i ) and (a)n i π i > 5 and (b)n i (1 π i ) > 5 fori = 1,...,N, we have approximately Residual deviance Pearson chi-square = 2(ľ ˆl) χ 2 N p X 2 = n i=1 (Y i n iˆπ i ) 2 n iˆπ i (1 ˆπ i ) χ2 N p ľ is log-likelihood in saturated model ˆl log-likelihood for the fitted model withpparameters and ˆπ i are estimated probabilities If D andx 2 is much larger thann p, it indicates that the model fit is bad However, they i -s are often binary, and then the conditions (a) and (b) is no fulfilled Lecture 6 STK Categorical responses p. 35

36 Two strategies for goodness of fit tests with binary data With categorical explanatory variables: Aggregate to binomial data Aggregation can not be used if there are many categorical variables with many levels, or if there are continuous variables. Can then instead use Hosmer-Lemeshow test Lecture 6 STK Categorical responses p. 36

37 Aggregation Count number of individuals within each combination of the categorical variables Count number ofy i = 1 within each combination Fit a GLM on aggregated data The model is OK ifd andx 2 are small compared to χ 2, Ñ p where Ñ is number of combinations of the categorical variables Requires that expected number of successes/failures in each group> 5 Lecture 6 STK Categorical responses p. 37

38 Ex: Aggregation on Wilm s tumor data > table(nwts$unfav) > table(nwts$stg) > nwts2 = aggregate(nwts$d,by=list(nwts$unfav,nwts$stg),fun=table) Group.1 Group.2 x.0 x > nwts2 = data.frame(unfav=nwts2$group.1,stg=nwts2$group.2, n=nwts2$x[,1]+nwts2$x[,2],d=nwts2$x[,2]) Lecture 6 STK Categorical responses p. 38

39 Ex: Aggregation on Wilm s tumor data > glmfit = glm(cbind(d,n-d) as.factor(unfav)+as.factor(stg),data=nwts2,f > glmfit (Intercept) unfavaggr factor(stgaggr)2 factor(stgaggr)3 factor(stgaggr) Degrees of Freedom: 7 Total (i.e. Null); 3 Residual Null Deviance: Residual Deviance: 3.33 AIC: > X2<-sum(residuals(glmfit,type="pearson")ˆ2) > X2 [1] Lecture 6 STK Categorical responses p. 39

40 Ex: Aggregation on Wilm s tumor data cont. The model seems to be OK, since residual deviance D = 3.33 X 2 = 3.26 = Pearson chi-square is small compared to residual degrees of freedomdf = 3 Is expected successes and failures > 5? We compute these: > round((nwts2$n*glmfit$fit,2) > round((nwts2$n*(1-glmfit$fit),2) Lecture 6 STK Categorical responses p. 40

41 Hosmer-Lemeshow test Fit the GLM model Order the individuals by fitted probabilities ˆπ (1) ˆπ (2) ˆπ (n) Divide the into G groups according to the ordering, with equally many individuals in each group ( C statistic ) Divide the interval from ˆπ (1) to ˆπ (n) into G intervals ( H statistic ) Compute the average π g = of ˆπ (i) in groupg = 1,2,...,G Compute no observationsn g and successesy g in groupg Compute Hosmer-Lemeshow X 2 hl = G g=1 Under the 0 hypothesis (model is OK) we have (Y g n g π g ) 2 n g π g (1 π g ) approximatelyx 2 hl χ2 G 2 Lecture 6 STK Categorical responses p. 41

42 Ex: Hosmer-Lemeshow test on Wilm s tumor data > glmfit<-glm(d unfav+factor(stg)+yr.regis+age, data=nwts,family=binomial) > library(mkmisc) > HLgof.test(glmfit$fit,nwts$d) $C Hosmer-Lemeshow C statistic data: glmfit$fit and nwts$d X-squared = , df = 8, p-value = $H Hosmer-Lemeshow H statistic data: glmfit$fit and nwts$d X-squared = , df = 8, p-value = Lecture 6 STK Categorical responses p. 42

43 Ex: Hosmer-Lemeshow test on Wilm s tumor data cont. > glmfit<-glm(d unfav+factor(stg)+yr.regis+age,family=binomial) > kuttoff<-sort(glmfit$fit)[c(round(length(d)*(1:10)/10))] > gr<-rep(1,length(d)) > for (i in 1:9) gr<-gr+(glmfit$fit>kuttoff[i]) > table(gr) > ngr<-as.numeric(table(gr)) > ngr [1] > dgr<-numeric(0) > for (i in 1:10) dgr[i]<-sum(d[gr==i]) > dgr [1] > for (i in 1:10) pigr[i]<-mean(glmfit$fit[gr==i]) > round(pigr,3) [1] > X2HL<-sum((dgr-ngr*pigr)ˆ2/(ngr*pigr*(1-pigr))) > X2HL [1] > 1-pchisq(X2HL,8) [1] Lecture 6 STK Categorical responses p. 43

44 Sensitivity and specificity Classification: Predict an event(y i = 1) if ˆπ i > γ, where γ is a threshold value Predict no event if ˆπ i γ Count number of correct classifications in the data set Sensitivity: Proportion of correct predictions when true Y i = 1 Specificity: Proportion of correct predictions when true Y i = 0 We want high values for both sensitivity and specificity For a given method, we can choose threshold valueγ to give a good balance in a specific classification situation Lecture 6 STK Categorical responses p. 44

45 ROC curves For evaluating and comparing models, we can vary the thresholdγ and plot a Receiver Operating Characteristics curve or ROC-curve with sensitivity on the y-axis and (1-specificity) on the x-axis Can also compute the area under curve (AUC) AUC=1 if perfect classification AUC=0.5 if random classification Lecture 6 STK Categorical responses p. 45

46 ROC for predicting bycatch of fish Shrimp fishery in Barents Sea: Predict if one can expect to catch more than 0.8 juvenile cod per kg shrimps caught If yes, the fishing area is temporarily closed Probability of correct prediction if observed> model predictor no predictability Lecture 6 STK Categorical responses p. 46

47 Over dispersion in binomial data With independent, binary data there is never over dispersion (Var(Y i ) = π i (1 π i ))) If independent, binary data Y ij with same π i are aggregated toy i = j=n i j=1 Y ij, theny i Bin(n i,π i ), Var(Y i ) = n i π i (1 π i )) and no over dispersion However, over dispersion occurs if the outcomes of the individuals trials are positively correlated. Then Var(Y i ) > n i π i (1 π i ) Possibility 1: Quasi-likelihood Possibility 2: Mixed model Possibility 3: Beta-binomial distribution Lecture 6 STK Categorical responses p. 47

48 Over dispersion in binomial data - Quasi-likelihood Specify mean structure by link function and linear predictor Specify variance structure Possibility 1: Var(Y i ) = φn i π i (1 π i )) Possibility 2: Var(Y i ) = (1+ρ(n i 1))n i π i (1 π i )) Fit the model. (Not sure if Possibility 2 is implemented in R) Lecture 6 STK Categorical responses p. 48

49 Randomπ or beta binomial response Mixed model: π i random with expectationπi If π i is random and beta distributed (continuous between 0 and 1), Y i becomes beta binomial Then Var(Y i ) = (1+ρ(n i 1))n i π(1 π)) Can be estimated i R by the betabin function from the aod library Lecture 6 STK Categorical responses p. 49