Improved likelihood inference for discrete data

Transcription

1 Improved likelihood inferene for disrete data A. C. Davison Institute of Mathematis, Eole Polytehnique Fédérale de Lausanne, Station 8, 1015 Lausanne, Switzerland. D. A. S. Fraser and N. Reid Department of Statistis, University of Toronto, Toronto M5S 3G3, Canada. Summary. Disrete data, partiularly ount and ontingeny table data, are typially analyzed using methods that are aurate to first order, suh as normal approximations for maximum likelihood estimators. By ontrast ontinuous data an quite generally be analyzed using third order proedures, with major improvements in auray and with intrinsi separation of information onerning parameter omponents. This paper extends these higher order results to disrete data, yielding a methodology that is widely appliable and aurate to seond order. The extension an be desribed in terms of an approximating exponential model expressed in terms of a sore variable. The development is outlined and the flexibility of the approah illustrated by examples. 1. Introdution In models with ontinuous response variables, reent developments in likelihood theory lead to p-value approximations for salar parameters that are aurate to third order, and to marginal likelihoods for salar or vetor parameters that are aurate to the same order. By omparison the usual normal approximations for the distributions of quantities based on the maximum likelihood estimator or the likelihood root are aurate just to first order. In this paper we show how the higher order methods an be extended to the analysis of disrete data, following Davison and Wang (2002), who examine saddlepoint methods that give seond order approximations by embedding the disrete problem in a ontinuous model, and Piere and Peters (1999), who argue that the ontinuous embedding model gives a more appropriate model for inferene than the original disrete model; broadly similar onlusions are reahed by Severini (2000b). Reent likelihood theory shows that inferene for a salar omponent parameter ψ(θ) has a well defined p-value p(ψ) for assessing ψ(θ) = ψ, and that a salar or vetor parameter ψ(θ) has a marginal log-likelihood l (ψ). The p-value is obtained from the observed loglikelihood l(θ) and a anonial parameterization ϕ(θ). In a full exponential family model ϕ(θ) is the anonial parameter; in more general models ϕ(θ) is onstruted using sample spae derivatives and approximate anillarity. Furthermore with independent observations y 1,..., y n we have n l(θ) = l i (θ) (1) i=1 where l i (θ) = log f i (y i ; θ) is the log likelihood ontribution from y i. The anonial reparameterization ϕ(θ) an similarly be expressed as a sum n ϕ(θ) = ϕ i (θ) (2) i=1

2 2 A. C. Davison, D. A. S. Fraser and N. Reid where ϕ i (θ) is a reparameterization ontribution from the ith data omponent. The anonial reparameterization is defined only up to affine transformations, whih have no effet on the inferene results. To ompute the p-value for inferene on a salar parameter of interest ψ(θ) = ψ, we use the p-value funtion p(ψ) = Φ{r + r 1 log(q/r)} (3) where Φ( ) is the standard normal distribution funtion. Expression (3) has been prominent in reent likelihood theory, aounts of whih may be found in Barndorff-Nielsen and Cox (1994), Severini (2000a), and Reid (2003). In (3) both r and q are determined by the pair {l(θ), ϕ(θ)} and the parameter of interest ψ: r is the likelihood root r(ψ) = sgn( ˆψ ψ)[2{l(ˆθ) l(ˆθ ψ )}] 1/2 (4) where ˆθ is the maximum likelihood estimator, ˆθ ψ is the onstrained maximum likelihood estimator for a given ψ(θ) = ψ; and q is a maximum likelihood departure with a nuisane parameter adjustment. An expression for q is given in Appendix 1(i). Under moderate regularity onditions and assuming that the log-likelihood has the usual asymptoti properties as n, the p-value and marginal log-likelihood are aurate to third order when the distribution of y is ontinuous. In this paper we show how to use these results for the analysis of disrete data. The auray drops from third order to seond order, and as in Davison and Wang (2002) the approximation involves a ontinuous embedding model, desribed in Appendix II. As we an only ahieve O(n 1 ) auray, we need only first order approximate anillary diretions whose onstrution we outline in Setion 2 and these are more easily obtained than the seond order diretions used for the ontinuous ase. In Setion 3 we show how these extended likelihood methods apply for inferene in a general model for disrete data, and illustrate this with 2 2 tables, binary regression with non-anonial link, and Poisson regression with omplementary log-log link. In Setion 4 we generalise to a model for overdispersion, the two-parameter negative binomial model. As an initial example, suppose we have n independent Bernoulli observations y i, with probability of suess p i related to a ovariate x i by the logisti funtion p i = exp(β 0 + β 1 x i )/{1+exp(β 0 +β 1 x i )}. Exat inferene for this binary regression model an be obtained by omputing the onditional distribution of Σy i x i given Σy i, and the onditional distribution an be very well approximated by the saddlepoint method; see for example Brazzale (2000). The arguments of Davison and Wang (2002) apply to this setting to show that the saddlepoint approximation with a ontinuity orretion approximates the exat onditional distribution funtion with relative error O(n 1 ), but more importantly that the saddlepoint approximation without ontinuity orretion approximates a ontinuous embedding of the model, whih in many ways is more useful for inferene and approximates the mid-p-value. These arguments hold for inferene about a linear funtion of the anonial parameter in full exponential families, inluding the odds ratio for 2 2 tables (Strawderman and Wells, 1998), mathed pairs, and multiple logisti regression. However, if the parameter of interest is a nonlinear parameter of the exponential family, as for example if the binary regression model is based on a non-anonial link funtion, then the nuisane parameters annot be exatly eliminated by onditioning, and the saddlepoint approah does not apply diretly. In this setting the present approah effetively eliminates the nuisane parameters, by marginalisation over a nuisane-parameter distribution.

3 Likelihood for disrete data 3 In the disrete ase exat or approximate onditioning reates a distribution with a possibly omplex lattie struture, and the maximum likelihood estimates may be on the boundary of the parameter spae. For some disussion of these points see Albert and Anderson (1984) and Frydenberg and Jensen (1989). Our development does not address this ompliation; the results require the maximum likelihood estimate to be in the interior of the parameter spae. 2. Canonial reparameterization Consider independent variables y 1,... y n where y i is a d i 1 vetor with model f i (y i ; θ) and the ommon parameter θ is 1 k. If y i is ontinuous, the reparameterization omponent from the ith observation is a 1 k vetor ϕ i (θ) = li (θ; y i ) y it V i (5) y i0 where T denotes matrix transpose and V i is a d i k weight matrix that desribes how parameter hange near the maximum likelihood value ˆθ 0 influenes the ith data omponent. The first fator, the gradient of the log-likelihood at the observed data point y 0, gives the anonial parameter if the model is a urved exponential family. The seond fator V i gives a linear adjustment to the log-likelihood gradient; these weights V i impliitly implement onditioning on an approximately anillary statisti, reduing the dimension of the problem from that of the data d d n to that of the parameter k. The numerial arrays V i provide all the information needed onerning this onditioning (Fraser and Reid, 2001). For the ontinuous ase the arrays V i are defined in Appendix I at (23) and lead to third order inferene. We now address the modifiations needed for inferene in the disrete ase. Suppose first that a omponent y i is a anonial variable in the urved exponential family model f i (y i ; θ) = exp{α i (θ)y i k i (θ)}h(y i ); (6) and let µ i (θ) = E(y i ; θ) be its mean. It is shown in Fraser and Reid (2001, Setion 7) that vetors V i tangent to an approximate anillary an be derived by desribing the effet of θ on the variable y i through its mean µ i (θ): V i = θ E(yi ; θ) = θ µi (θ) = µ i θ(ˆθ 0 ), (7) ˆθ0 ˆθ0 say, whih is a d i k matrix. Then from (5) and (6) we have ϕ i (θ) = α i (θ)v i. (8) The oordinates of y i need not be linearly independent: any alternative oordinates will lead to an equivalent ϕ(θ) due to linearity and the use of the mean of the oordinates. If the oordinates of y i are atual sore variables for the parameter θ then µ i θ (θ) is the expeted information i i θθ (θ) for that oordinate model. In (6) y i is the sore variable for α i in the full exponential family model. We extend this to more general settings by onstruting ϕ(θ) and V i with y i replaed by the loally defined sore variable s i = θ log f(yi ; θ) θ=ˆθ0

4 4 A. C. Davison, D. A. S. Fraser and N. Reid and omputing V i as V i = θ E(si ; θ) θ=ˆθ0. (9) We then have that the ontribution of the ith observation to the loal reparametrisation is ϕ i (θ) = li (θ; y i ) s i V i. (10) y i0 We then sum over i as at (1) and (2) and use the pair {l(θ), ϕ(θ)} to obtain the p-value (3) as before. Although not illustrated in the examples here, we ould also use {l(θ), ϕ(θ)} to onstrut the marginal log-likelihood l (ψ) as desribed in Fraser (2003). Fraser and Reid (2001, Setion 7) justify this onstrution through the tangent exponential model approximation to the original model. A possible reparametrisation of the model would give a new sore variable that is a linear transformation of the initial sore and would give a ompensating linear transformation of the oordinates of the V i ; this has no effet on the resulting inferene. In some models the alulations are easier if the sore s i is replaed by an affine transformation of it; we will see this in the example in Setion 4. As shown in 7 of Fraser and Reid (2001), V i is tangent to a first order anillary statisti; in urved exponential families it is tangent to the likelihood root for omparing the urved model to the full model. Inferene based on a first order anillary is suffiient to obtain a seond order approximation. In the disrete ase we an only obtain a seond order approximation in any ase, as desribed in Appendix II and in Davison and Wang (2002), and the use of expeted values to define V i and hene ϕ(θ) avoids the need to speify a pivotal funtion, whih would usually not be available for disrete models with a ontinuous parameter. 3. Categorial data model 3.1. The log-likelihood and anonial parameter Suppose we have a response variable that an fall in one of d ategories with orresponding indiator variables y 1,...,y d. Then y = (y 1,..., y d ) T has a multivariate Bernoulli distribution with probabilities p 1 (θ),..., p d (θ) proportional to say e α1(θ),...,e α d(θ). The log-likelihood ontribution from this observation is α(θ)y log A(θ), where A(θ) = j exp{α j(θ)} and α(θ) = {α 1 (θ),..., α d (θ)}. For a single observation from this multivariate Bernoulli model the derivative of the mean of the sore variable y is θ E(y; θ) = µ θ(θ) = {diag p(θ) p(θ)p T (θ)} αt θ. The array in braes is the expeted Fisher information for the anonial parameter of the full dimensional Bernoulli model and has dimension d d; the derivative of α has dimension d k. Now onsider independent multivariate Bernoulli responses y i, with dimensions d i, and with log odds parameters α i 1(θ),..., α i d (θ). For eah observation yi we have from (7) the

5 Likelihood for disrete data 5 ith weighting matrix V i = { } α diag p i (ˆθ 0 ) p i (ˆθ 0 )p it (ˆθ 0 it (ˆθ 0 ) ) θ where ˆθ 0 is the overall maximum likelihood estimate; then substituting in (5) we obtain the reparameterization ontribution ϕ i (θ). We then ompute l(θ) and ϕ(θ) by summing over i, as at (1) and (2), and use these to ompute the p-value approximation (3) from (4) and (21). This gives a p-value in this disrete ase whih is aurate to seond order, O(n 1 ), as a mid p-value. If the parameters α i (θ) do not depend on i then l(θ) is the likelihood funtion for a d-ategory multinomial distribution and ϕ is linear in the anonial parameter for that exponential family model. We illustrate this on some versions of the 2 2 table, and then on some regression models for disrete data tables Consider first a single multivariate Bernoulli variable y arrayed as a matrix ( ) y11 y y = 12, yjj = 1, y 21 y 22 with orresponding probabilities ( ) p11 (θ) p p(θ) = 12 (θ), p 21 (θ) p 22 (θ) pjj = 1, (11) so that l(θ) = y 11 log p 11 + y 12 log p 12 + y 21 log p 21 + y 22 log p 22 ; the largest possible dimension for θ is k = 3. Writing the mean as a olumn vetor µ(θ) = p(θ) = (p 11, p 12, p 21, p 22 ) T, we find that µ θ (θ) onsists of three olumn vetors orthogonal to a vetor of ones and the reparametrisation an then be taken as ( ϕ(θ) = log p 12, log p 21, log p ) 22. p 11 p 11 p 11 With n independent observations y 1,..., y n of this multivariate Bernoulli variable we have l(θ) = y i 11 log p 11 + y i 12 log p 12 + y i 21 log p 21 + y i 22 log p 22, ϕ(θ) = {n log(p 12 /p 11 ), n log(p 21 /p 11 ), n log(p 22 /p 11 )}, although the onstant multiple n in the seond expression is unneessary as the inferenes are invariant to linear transformations of ϕ. Now suppose that the row probabilities in (11) are known, thus reduing the dimension of θ to 2. We write these probabilities as ( ) q1 p 1, q 2 p 2

6 6 A. C. Davison, D. A. S. Fraser and N. Reid and let θ = (p 1, p 2 ). The log-likelihood ontribution from y i is then l i (θ) = y i 11 log q 1 + y i 12 log p 1 + y i 21 log q 2 + y i 22 log p 2. Writing the probability array in the vetor form (q 1, p 1, q 2, p 2 ) T, we obtain dµ(θ) dθ = and thus ϕ i (θ) = (log p 1 log q 1, log p 2 log q 2 ) = {log(p 1 /q 1 ), log(p 2 /q 2 )}. For n suh observations we have, l(θ) = n 11 log q 1 + n 12 log p 1 + n 21 log q 2 + n 22 log p 2, ϕ(θ) = {log(p 1 /q 1 ), log(p 2 /q 2 )}, where we have omitted unneeded fators n 1 and n 2. This is the same log-likelihood and reparametrisation as is obtained in the modelling of the 2 2 table as a omparison of two binomials; the familiar onditioning on row totals is here obtained automatially. Finally we onsider the more restrited model p(θ) = 1 4 ( 2 + θ 1 θ 1 θ θ ), 0 < θ < 1. This appears in disussions of anillarity by Fisher (1956), Basu (1964), Fraser (1979) and Fraser (2004) and has the unusual feature that the row totals and olumn totals are eah anillary statistis for θ but the ombination of them is not anillary. The onstrution below however is onditional on an approximate anillary statisti that is not needed expliitly. The log-likelihood ontribution for a single observation is l(θ) = y 11 log(2 + θ) + (y 12 + y 21 )log(1 θ) + y 22 log θ. The mean funtion µ(θ) is p(θ), and we obtain V = µ θ (ˆθ 0 ) = 1 4 ( ), yielding ϕ(θ) = 1 {log(2 + θ) 2 log(1 θ) + log θ}. 4 For n independent observations of this form we have l(θ) = n 11 log(2 + θ) + (n 12 + n 21 )log(1 θ) + n 22 log θ, ϕ(θ) = n {log(2 + θ) 2 log(1 θ) + log θ}, 4 where the fator n/4 an be ignored. Note that V written in vetor form is orthogonal to row differenes and to olumn differenes and thus in priniple agrees with onditioning on both the row totals and the olumn totals. The alulations needed for more general ontingeny tables are analogous.

7 Likelihood for disrete data 7 Perent relative error Perent relative error Poisson mean Total Fig. 1. Comparison of exat and approximate probabilities for tests on Poisson mean (left) and differene of log-odds for two binomial variables (right), for the likelihood root r (+) and the modified likelihood root r ( ). Here δ = 2, orresponding to a signifiane level of around The solid lines have slope 1 and the dashed lines have slope 1/2. The left panel also shows the dependene on ψ of the signifiane probability φ(r ) for the ontinuity-orreted observation y () Numerial examples We now give two numerial illustrations of the auray of these higher order proedures in ases where an exat answer exists for omparison. We ompute the ratio of the exat and approximate p-value as the sample size inreases, hoosing a fixed quantile for the omparison. The first illustration onerns a Poisson variable with mean ψ. We take y = ψ + δψ 1/2, for a speified value of δ and onsider ψ ; here ψ plays the role of n in independent sampling. The likelihood root r is readily omputed and the maximum likelihood departure q equals y log(y/ψ). We onsider the behaviour of approximate signifiane probabilities as the sample size, or equivalently here ψ, inreases with δ fixed. Let p ψ = pr(y y; ψ) and p ψ denote exat and approximate signifiane probabilities, and suppose that p ψ = p ψ (1 + bψ ) + o(ψ ) for some as ψ. Then a log-log graph of p ψ /p ψ 1 against ψ will be linear with slope and interept log b. The left panel of Figure 1 shows suh a graph (with δ = 2), omparing p ψ given by Φ(r) and by Φ(r ) to the mid-p-value pr(y y 1; ψ) + 1 2pr(Y = y; ψ), and omparing a ontinuity orreted version of r to the exat value pr(y y; ψ). The relative errors show the expeted dependene on sample size, while the good performane of the ontinuityorreted version of r supports the arguments of Davison and Wang (2002). For ψ = 1 the relative errors of r and r are around 3% and less than 1%, while the ontinuityorreted version of r, in whih y is replaed by y + 1 2, has relative error around 0.2% as an approximation to pr(y y; ψ). For an example with a nuisane parameter onsider the differene of log-odds for two binomial observations, with denominators m 1 = m 2 = 2ψ, r 1 = ψ, r 2 = ψ + ψ, for

8 8 A. C. Davison, D. A. S. Fraser and N. Reid ψ = 4, 9, 16,..., with respetive log-odds λ and λ + ψ. Again ψ plays the role of sample size, and alulations analogous to those outlined for the Poisson model yield the results shown in the right panel of Figure 1. Again we see the relative error of the approximation based on the likelihood root r dereasing as ψ 1/2 and the relative error of the more refined approximation (3) dereasing as ψ 1. Other approximations that may be applied with disrete data, although not developed speifially for them, have been suggested by Skovgaard (1996) and by Severini (1999). In the examples above straightforward alulations show that these yield r and r respetively. Severini s approximation involves moment estimators of expeted values and large sample sizes may be needed to estimate these aurately Binary regression Suppose y i follows a Bernoulli distribution with suess probability p i and a link funtion g(p i ) = λ+ψx i with two parameters ψ and λ. The orresponding log-likelihood ontribution is l i (λ, ψ) = y i logitp i (λ, ψ) + log{1 p i (λ, ψ)} (12) where logitu = log{u/(1 u)}. Then sine E(y i ) = p i (λ, ψ) we have V i = pi (λ, ψ) 1 = (λ, ψ) ˆθ0 g {p i (ˆλ 0, ˆψ 0 )} (1 x i) and ϕ i = logitp i (λ, ψ) V i. (13) For the logisti regression model, the link funtion is logit p i (λ, ψ) = λ + ψx i, and (13) simplifies to ϕ i 1 = (λ + ψx i ) p i (ˆλ 0, ˆψ 0 ){1 p i (ˆλ 0, ˆψ 0 )} (1 x i), and shows that ϕ(λ, ψ) is just a linear transformation of (λ, ψ), the anonial parameter of the exponential family. As the inferene is invariant to linear transformation, we an take ϕ(θ) = (λ, ψ), with l(θ) = λσx i +ψσy i x i log{1+exp(λ+ψx i )}, and so (21) simplifies to q = ( ˆψ ψ) j θθ (ˆθ) 1/2 j λλ (ˆθ ψ ) 1/2. Higher order approximations based on this have been implemented in the S language by Brazzale (2000). The omputations for vetor ovariates x i extend those above in an obvious way. If we use a non-anonial link then l(θ) and ϕ(θ) are omputed from (12) and (13), although the expliit expression for q seems unenlightening. For a numerial assessment of the effet of higher order adjustments we examine data from 53 persons with prostate aner (Brown, 1980). The binary response indiates the presene of nodal involvement and depends on five dihotomous explanatory variables. We fit the model with all ovariates, and assess how a positive response depends on one of these ovariates, the level of serum aid phosphatase. The orresponding regression parameter is denoted ψ; there are six parameters in all, inluding a onstant. Fitting a logisti regression model, for whih g(p) = logitp, gives ˆψ = with standard error 0.791; the signed likelihood ratio statisti for testing ψ = 0 is r = 2.247, with p-value The orresponding value of r is 2.083, with p-value The model with omplementary log-log

9 Likelihood for disrete data 9 Table 1. Lung aner deaths in British male physiians (Frome, 1983). The table gives manyears at risk/number of ases of lung aner, T/y, ross-lassified by years of smoking, taken to be age minus 20 years, and number of igarettes smoked per day. Years of Daily igarette onsumption x smoking t Nonsmokers / / /1 4050/ / /1 4290/ /2 2560/4 4687/6 4268/9 1580/ / /5 3529/9 1336/ /2 1386/1 1334/2 2411/ /11 924/ /2 849/2 1567/9 1409/10 556/ /3 684/4 470/2 857/7 663/5 255/ / /3 280/5 416/7 284/3 104/1 link funtion g(p) = log{ log(1 p)}, gives ˆψ = with standard error 0.618, and the values of r and r for testing ψ = 0 are and 1.843, with orresponding p-values and The higher order orretion is in the same diretion as with the logisti model, but is more substantial. Examples and of Davison (2003) also desribe the use of higher order approximations for the logisti regression model, using the software of Brazzale (2000) whih implements approximate onditional inferene for logisti regression. As noted above the proedure outlined here reovers this onditioning when the parameter of interest is linear in the anonial parameter of the exponential family Extension to Poisson ounts: smoking data The method outlined for the multivariate Bernoulli model extends diretly to the ase of Poisson ounts, whih we illustrate on the data in Table 3.5 on the relation between smoking and lung aner in British male physiians. It shows the man-years at risk T and the number of individuals dying of lung aner, y, ross-lassified by the number of igarettes smoked daily x and the number of years of smoking t, taken to be age minus twenty years. Frome (1983) suggests that the mean deaths per man-year be modelled as λ(x, t) = e θ1 t θ2 ( 1 + e θ3 x θ4), < θ 1, θ 3 <, θ 2, θ 4 > 0; (14) the death-rate in the absene of smoking is thus e θ1 t θ2. One aspet of interest is the value of θ 4, as θ 4 = 1 would orrespond to linear inrease in death rate with x, and we shall investigate this. If we assume that the number of deaths in the ith ell is Poisson with mean µ i (θ) = Tλ(x, t), then the log likelihood an be expressed as l(θ) = n { y i log µ i (θ) µ i (θ) }, i=1 where y i is the number of deaths in the ith ell of the table. The maximum likelihood estimates and their standard errors based on the observed information matrix are ˆθ 1 = 2.94 (0.57), ˆθ 2 = 4.46 (0.33), ˆθ 3 = 1.12 (1.00), and ˆθ 4 = 1.28 (0.2). The value of the signed likelihood ratio statisti r for testing θ 4 = 1 is 1.506, so the normal approximation

10 10 A. C. Davison, D. A. S. Fraser and N. Reid to the distribution of r gives a one-sided signifiane level for testing linear dependene of the death rate on x as The anonial parametrisation is from (14) we have µ i (θ) θ = T i e θ1 t θ2 i ϕ(θ) = n ( µ log µ i i (θ) (θ) θ i=1 ˆθ0 ) ; ( 1 + e θ3 x θ4 i, (1 + eθ3 x θ4 i )log t i, e θ3 x θ4 i, eθ3 x θ4 i log x i ) T, where terms involving x are understood to vanish for non-smokers. The elements of (3) are most simply obtained numerially. With θ 4 = 1 we have q = 1.47, so r = 1.491, giving a signifiane level of 0.068, only a small hange from the value based on r. 4. A more omplex model In this setion we illustrate the omputations on a model whih is not a urved exponential family model, so that the more general approah to omputing ϕ(θ) desribed at the end of Setion 2 is needed. We onsider an overdispersed Poisson distribution, where the overdispersion is generated by assuming the mean of the Poisson follows a gamma distribution with mean µ and shape parameter λ. The resulting density is the negative binomial f(y; θ) = Γ(ν + y) Γ(ν)y! The log-likelihood funtion based on a single observation is ν ν µ y, y = 0, 1,...; µ, ν > 0. (15) (ν + µ) ν+y l(θ; y) = A(y + ν) A(ν) + ν log ν + y log µ (ν + y)log(ν + µ), (16) where A(ν) = log Γ(ν) is the log-gamma funtion. We will write ζ(ν) for the digamma funtion A (ν). Differentiating (16) with respet to µ and ν at (ˆµ 0, ˆν 0 ) gives s = (s 1, s 2 ) where s 1 = l = y/ˆµ 0 (ˆν 0 + y)/(ˆν 0 + ˆµ 0 ), (17) µ ˆθ0 s 2 = l = ζ(ˆν 0 + y) ζ(ˆν 0 ) log ˆν 0 log(ˆν 0 + ˆµ 0 ) (ˆν 0 + y)/(ˆν 0 + ˆµ 0 ). ν ˆθ0 For the ith observation in a sample of size n from (15), we obtain V i from (9) as ( ) ˆν 0 /{ˆµ 0 (ˆµ 0 + ˆν 0 )} 0 0 ζ (ˆν 0 ) (1/ˆν 0 ) E{ζ (ˆν 0 + y i )} + 1/(ˆν 0 + ˆµ 0. (18) ) The expetation in the expression for V22 i in (18) is for a single observation from (15). Note that s i is affinely equivalent to the simpler form ( y i ζ(ˆν 0 + y i ) (y i + ˆν 0 )/(ˆµ 0 + ˆν 0 ) ),

11 Likelihood for disrete data 11 but by keeping the more omplex form (18) we have that Vαβ i = ( / θ β)es i α is the (α, β)th element of the expeted information matrix in a single observation from model (16). In this example the parameters ν and µ are orthogonal so the array V i is diagonal. We use V i to ompute ϕ(θ) as ϕ 1 (θ) = l(θ; y i ) s i V11 i = 1 ϕ 2 (θ) = l(θ; y i ) s i V22 i = 2 n [ζ(ν + y i0 ) + log{µ/(ν + µ)}] (19) i=1 n i=1 [ζ(ν + y i0 ) + log{µ/(ν + µ)}] ζ (ˆν 0 + y i0 ) 1/(ˆν 0 + ˆµ 0 V22 i (20) ) where l(θ)/ s i α = { li (θ)/ y i }{ s i α / yi } 1. As usual we ombine this with l(θ) to ompute r (ψ). For a numerial illustration of these alulations, we take the data from Bissell (1972) on the numbers of faults, y, in lengths of loth, x (m 10 2 ). We suppose that y i follows the two parameter negative binomial distribution (15), where now µ = µ i = βx i, and we take the ommon shape parameter, ν, to be the parameter of interest. Instead of ( s i 1/ y i ) 1 V11 i = 1 as at (19) it is now x i. The mean and variane of y i are βx i and βx i + (βx i ) 2 /ν. The overall maximum likelihood estimate of ν is ˆν = with standard error The 95% onfidene interval for ν based on the normal approximation to the distribution of the likelihood root r(ν) is (3.68, 28.41). Using the normal approximation to the distribution of r (ν) the 95% onfidene interval is (3.35, 24.13). Here the seond order orretion moves the interval towards the origin, and so inreases the fitted response varianes. This move is in the same diretion as with normally distributed responses, for whih higher order orretions orrespond to taking the appropriate denominator for a sum of squares, and hene tend to inrease maximum likelihood variane estimates. Appendix I: Details on the p-value formula (i) Computing q from ϕ We now present the formulae needed to onvert {l(θ), ϕ(θ)} to approximate p-values; for further details see Fraser et al. (1999) or Reid (2003). For inferene on a salar parameter of interest ψ(θ) = ψ, we use the p-value funtion defined at (3), with r the likelihood root. The omplementing funtion q is a nuisane parameter adjusted maximum likelihood departure, q(ψ) = sgn( ˆψ ψ) χ(ˆθ) χ(ˆθ ψ ) In (21) χ(θ) is a surrogate for ψ(θ) and is linear in ϕ(θ), { ĵ ϕϕ j (λλ) (ˆθ ψ ) } 1/2. (21) χ(θ) = ψ/ ϕ ϕ T (θ), (22) ψ/ ϕ ˆθψ with ψ/ ϕ obtained as ψ/ θ ( ϕ T / θ) 1 and both θ and ϕ taken as row vetors. This linear surrogate has a ontour or level surfae that is tangent to ψ(θ) at ˆθ ψ. Formulae for the speialized observed informations are reorded at (iii).

12 12 A. C. Davison, D. A. S. Fraser and N. Reid (ii) The maximum likelihood values For the omputation of r in (4) we need the profile log-likelihood, and if θ = (ψ, λ), where λ is expliitly available, then this is obtained simply by substituting ˆλ ψ for λ in the full loglikelihood. If an expliit nuisane parameterization is not available, then we an typially ompute the profile log-likelihood l p (ψ) = l(ˆθ ψ ) by maximising l(θ) + α {ψ(θ) ψ} over (θ, α), whih gives ˆθ ψ and the Lagrange multipler ˆα ψ. The orresponding tilted likelihood or Lagrangian l(θ) = l(θ) + ˆα ψ {ψ(θ) ψ} an be used for alulating q; see Fraser et al. (1999). (iii) The informations and estimated varianes The expression in braes in (21) is the reiproal of an estimate of the variane of χ(ˆθ) χ(ˆθ ψ ), and is a ratio of observed Fisher information matries for the full parameter and for the nuisane parameter, both realibrated in terms of ϕ. They an be omputed by resaling the usual information determinants: ĵ ϕϕ = ĵ θθ ϕ T / θ 2, j (λλ) (ˆθ ψ ) = j λλ (ˆθ ψ ) ϕ λ T(ˆθ ψ )ϕ T λ T(ˆθ ψ ) 1 where the parentheses enlosing λ are to indiate that the nuisane parameter has been alibrated loally in terms of ϕ(θ). (iv) The weighting matrix V i for the ontinuous ase In the ontinuous ase a full-dimensional pivotal quantity an desribe how the ith oordinate is influened by parameter hange near the observed maximum likelihood value. For the ith oordinate let z i (y i ; θ) be the ith pivotal quantity. The array V i is obtained from the total derivative for that oordinate pivotal: ( ) V i = dyi z i 1 z i dθ = y i0,ˆθ y i, i = 1,..., n, (23) θ 0 y i0,ˆθ 0 where ˆθ 0 = ˆθ(y 0 ) is the maximum likelihood estimator obtained from the full data, and the leftmost derivative is alulated for fixed value of the pivot. The formulae given in Fraser (2003) an be used to onvert {l(θ), ϕ(θ)} to a marginal log-likelihood for ψ, whether ψ is salar or vetor. Appendix II: A ontinuous approximation to the disrete model We use a ontinuous model that an be made arbitrarily lose to the disrete model, and apply the asymptoti methods to the ontinuous model; this gives the redution of dimension and the separation of omponent parameters. Also, as mentioned in Setion 2 we determine the influene of the parameter on a data point in terms of the mean of a loally-defined sore variable (Fraser and Reid, 2001). As the simplest disrete model we examine in detail the omponent Bernoulli model f(y; θ) = e yϕ(θ) e ϕ(θ) + e ϕ(θ), y = 1, +1. (24)

13 Likelihood for disrete data 13 and onstrut a orresponding ontinuous model with the same sore parameter and sore variable f (y; θ) = k 1 e yϕ(θ) {ϕ(θ)} e ϕ(θ) + e ϕ(θ), y S, (25) where S = ( 1 ± ) (1 ± ), and is an auxiliary parameter. The normalising onstant is k (ϕ) = 2 sinhϕ ϕ = 2{1 + (ϕ)2 3! + (ϕ)4 5! +...}. The multivariate Bernoulli and more general disrete models an be obtained by ompounding the simple Bernoulli, possibly with appropriate onditioning. The gradient of the log-likelihood of the ontinuous model with respet to y is ϕ(θ) whih is the anonial parameter of the disrete model. The likelihood funtion and the distribution funtion for the ontinuous model approximate that of the disrete model with error O( 2 ) as 0. For the asymptoti analysis we assume we have n independent omponents from models of the type (25), with possibly different ϕ funtions but a ommon parameter θ, and onsider n ; the effet of an then be made arbitrarily small. The p-value approximation is obtained from Cakmak et al. (1998), using Taylor expansions in n 1/2 neighbourhoods of the observed data point. The disrete model an be viewed as obtained by rounding to the nearest integer; the roundoff is of order O(n 1/2 ) but by interpreting the p-value as a mid p-value the effet is of order O(n 1 ). The use of the mid p-value interpretation avoids the onern for ontinuity orretions. Aknowledgement The work was supported by the Swiss National Siene Foundation, Eole Polytehnique Fédérale de Lausanne, and the Natural Sienes and Engineering Researh Counil of Canada. The authors express thanks to the referees and assoiate editor for very helpful omments on an earlier version. Referenes Albert, A. and Anderson, J. A. (1984) On the existene of maximum likelihood estimates in logisti regression models. Biometrika 71, Barndorff-Nielsen, O. E. and Cox, D. R. (1994) Inferene and Asymptotis. London: Chapman & Hall. Basu, D. (1964) Reovery of anillary information. Sankhyā 26, Bissell, A. F. (1972) A negative binomial model with varying element sizes. Biometrika 59, Brazzale, A. R. (2000) Pratial Small-Sample Parametri Inferene. Ph.D. thesis, Department of Mathematis, Swiss Federal Institute of Tehnology, Lausanne, Switzerland. Brown, B. W. (1980) Predition analysis for binary data. In Biostatistis Casebook, eds R. G. Miller, B. Efron, B. W. Brown and L. E. Moses, pp New York: Wiley.

14 14 A. C. Davison, D. A. S. Fraser and N. Reid Cakmak, S., Fraser, D. A. S., MDunnough, P., Reid, N. and Yuan, X. (1998) Likelihood entered asymptoti model exponential and loation model versions. Journal of Statistial Planning and Inferene 66, Davison, A. C. (2003) Statistial Models. Cambridge: Cambridge University Press. Davison, A. C. and Wang, S. (2002) Saddlepoint approximations as smoothers. Biometrika 89, Fisher, R. A. (1956) Statistial Methods and Sientifi Inferene. Edinburgh: Oliver and Boyd. Fraser, D. A. S. (1979) Inferene and Linear Models. New York: MGraw Hill. Fraser, D. A. S. (2003) Likelihood for omponent parameters. Biometrika 90, Fraser, D. A. S. (2004) Anillaries and onditional inferene (with Disussion). Statistial Siene 19, Fraser, D. A. S. and Reid, N. (2001) Anillary information for statistial inferene. In Empirial Bayes and Likelihood Inferene, eds S. E. Ahmed and N. Reid, pp New York: Springer. Fraser, D. A. S., Reid, N. and Wu, J. (1999) A simple general formula for tail probabilities for frequentist and Bayesian inferene. Biometrika 86, Frome, E. L. (1983) The analysis of rates using Poisson regression models. Biometris 39, Frydenberg, M. and Jensen, J. L. (1989) Is the improved likelihood ratio statisti really improved in the disrete ase? Biometrika 76, Piere, D. A. and Peters, D. (1999) Improving on exat tests by approximate onditioning. Biometrika 86, Reid, N. (2003) Asymptotis and the theory of inferene. Annals of Statistis 31, Severini, T. A. (1999) An empirial adjustment to the likelihood ratio statisti. Biometrika 86, Severini, T. A. (2000a) Likelihood Methods in Statistis. Oxford: Clarendon Press. Severini, T. A. (2000b) The likelihood ratio approximation to the onditional distribution of the maximum likelihood estimator in the disrete ase. Biometrika 87, Skovgaard, I. M. (1996) An expliit large-deviation approximation to one-parameter tests. Bernoulli 2, Strawderman, R. L. and Wells, M. T. (1998) Approximately exat inferene for the ommon odds ratio in several 2 2 tables (with disussion). Journal of the Amerian Statistial Assoiation 93,