ON THE USE OF A CORRELATED BINOMIAL MODEL FOR THE

'e ON THE USE OF A CORRELATED BINOMIAL MODEL FOR THE ~~ALYSIS OF CERTAIN TOXICOLOGICAL EXPERIMENTS by L.L. Kupper Department of Biostatistics University of North Carolina, Chapel Hill J.K. Haseman Biometry Branch National Institute of Environmental Health Sciences Research Triangle Park.. Institute of Statistics Mimeo Series No. 4 June 977

'. ON THE USE OF A CORRELATED BINOMIAL MODEL FOR THE ANALYSIS OF CERTAIN TOXICOLOGICAL EXPERIMENTS L.L. Kupper! and J.K. Haseman ABSTRACT In certain toxicological experiments with laboratory animals, the outcome of interest is the occurrence of dead or malformed fetuses in a 'litter. Previous investigations have shown that the simple one~ parameter binomial and Poisson models generally provide poor fits to this type of binary data. In this paper, a type of correlated bino~, mial model is proposed for use in this situation. First, the model is described in detail and is shown to have certain theoretical advantages over a beta-binomial model proposed by Williams [8]. These two-parameter models are then contrasted as to goodness of fit to some real~life data. Finally, numerical examples are given in which likelihood ratio tests based on these models are employed to assess the significance of treatment-control differences. i IDepartment of Biostatistics, School of Public Health, University of North Carolina, Chapel Hill, NC 754. Biometry Branch, National Institute of Environmental Health Sciences, Research Triangle Park, NC 7709, *This investigation was supported in part by NIH Research Career Development Award (No. l-k04-es00003) from the National Institute of Environmental Health Sciences.

--,. INTRODUCTION In laboratory experiments designed to investigate the teratogenic or toxicological effect of certain compounds, the response of interest is frequently binary in nature, namely, the occurrence or not of "affected" fetuses or implantations in a litter. The "effect" under consideration is generally fetal death or the occurrence of some particular malformation. The statistical treatment of such data generally requires that the variations in response be described by some underlying probability model, and hence the quality of any subsequent statistical inferences will necessarily depend on how well such a model represents the phenomenon under study. Two one-parameter models which have been employed for the analysis of fetal death data are the Poisson (see []) and binomial (see [6]) distributions. Investigators who use a Poisson model base their analysis on the numper of dead implants per female, and so are unrealistically put~ ting no theoretical restriction on the number of dead implants. Unlike the Poisson model, the binomial model takes the total number of implants per female into account, the basic variable of interest being the proportion of dead implants. Unfortunately, fetal death in mice rarely follows either a Poisson or binomial distribution. For example, Haseman and Soares [3] considered the distribution of fetal death in three large groups of control mice, and found that the Poisson and binomial models provided poor fits to these data. I el

-3- In Table below (taken from the Haseman and Soares paper), the observed distribution of fetal death in these three groups has been compared to what would be expected assuming an underlying Poisson or binomial model. There was significant (P < 0.0) lack of fit in all cases, In addition. when sample sizes were sufficiently large, separate binomial models were fit to groups of animals having the same number of total implants. In most cases, these analyses also revealed a significant lack of fit. A two-parameter alternative to the Poisson and binomial models has been suggested by Williams [8]. He assumes that responses within a litter form a set of independent Bernoulli trials whose success probability varies among litters in the same treatment group according to a two-parameter beta distribution, The parameters of the beta distribution for each treatment group are then estimated by maximum likelihood, and the significance of treatment differences is assessed via asymptotic likelihood ratio tests.

TABLE I Poisson and Binomial Fit to Control Data a Number of dead implants Observed Data Set Data Set Data Set 3 Expected Expected Expected Observed Observed Poisson binomial Poisson binomial Poisson binomial 0 4 6.3. 60.0 60 564. 548.5 86 4. 4. 54 90. 9.0 49 483.0 500. 65 00.6 99.5 83.5 4. 5 06.8 3.5 66 83.5 84.8 c c c c 3 34 43.5 43.8 49 59.0 55. 5 3. 3.3 >3 39 6.4 4.9 3 5. 0.8 5.7 5.3 63.6 75. 6.0 b 30.5 67.8 74.0 c I ---------------------------------------------~-------- ------------------------------------------------------ X 3 (test of fit) ~ I asee Haseman and Soares [3] for the complete distributions of fetal death. bp < 0.0. c P < 0.00. --.. --., e..

-5- McCaughran and Arnold [5] have suggested the use of the negativebinomial distribution to model fetal death. If, for each female, the number of deaths is assumed to be Poisson and if the rate of death (i.e., the Poisson parameter) is assumed to vary from female-to-female according to a gamma distribution, the resulting unconditional distribution of fetal deaths in the population will be negative binomial. Although this particular two-parameter distribution must necessarily fit better than the one-parameter" Poisson and binomial distributions, there is little theoretical basis for its use. As mentioned earlier, the Poisson model does not take litter size into account, and, secondly, there is nc theoretical justification for the assumption that the death rates have a gamma distribution. Because of these objections, we do not consider the negative binomial model to be appropriate and so will not mention it again. The purpose of this paper is to present an alternative to the beta-binomial model proposed by Williams, and to compare these two models on both theoretical and practical grounds. As the title of the paper suggests, we. are proposing the use of a type of llcorrelated binomial" model, since we believe that one of the basic assumptions underlying the use of the binomial distribution - namely, the assump~ tion of independent trials - is quite possibly being violated in the experimental setting we are considering. In particular, we t feel that fetuses in the same litter tend to have an inherent relationship to one another, and that an appropriate model should in some way allow for an assessment of the strength of this possible intra-litter correlation. Our approach involves "correcting" the

-6- usual binomial model via a technique suggested by Bahadur [] t~ account for such within-litter dependency. We feel that this method of handling extra-binomial variation is intuitively more appealing than Williams' approach, which requires both the assumption of mutual independence among the within-litter Bernoulli responses and the assumption of a beta prior distribution for the Bernoulli parameter. In Section of this paper, we will discuss Bahadur's work and will describe the correlated-binomial model; the very special case of two implants per litter will be briefly considered for illustrative purposes. In Section 3, the beta-binomial and correlated-binomial models will be contrasted as to goodness-of-fit to the data sets given in Table I, and illustrations using the models to compare treatment and control groups will be given. The analysis procedures to be developed in this paper are useful in a variety of experimental situations. For example, Lachenbruch and Perry [4] discuss a physical therapy experiment involving patients with severed nerves in one hand; separate measurements were made on each finger of each patient.' s hand to assess the relationship between sweat and sensation. Clearly, the observations on the fingers of a hand cannot reasonably be considered to be independent. And, Wei! [7] cites a dental study where approximately 0 teeth per child were examined for the presence of caries, and he points out that there is an inherent dependency among the observations on teeth in the same mouth.

\ -7-. THE CORRELATED-BINOMIAL MODEL.. General Considerations To establish notation, let us suppose that there are l. litters in the i-th group (i = 0 for control group and i = for treatment group), the j-th litter in the i-th group being of size n.., j =,,...,.t. ) Let. n.. ) X.. = r X k ) k=l ) \ where X..k is a Bernoulli random variable taking the value with pro- ) bability p. if the k-th implant in the j-th litter of the i-th group - possesses the attribute of interest and taking the value 0 with probability ( - p. ) if the attribute is not present.. Note that we are assuming here that the true underlying probability of possessing the attribute depends only on the group under study and does not vary from litter to litter within a group. In contrast, Williams assumes that this probability depends on j as well as i, and he accounts for this variation in a Bayesian way in terms of a beta prior distribution.

-8- Under the ordinary binomial distribution assumption that X ijl, X. J'""'X" Jn.. J the value x.. with probability J are mutually independent, it follows that X.. J takes P(I)(X ij ) x.. = O,l,...,n.. J J () However, when the assumption of mutual independence is unreasonable, then Bahadur [] has shown that the correct and most general expression for Pr (X.. = x.. ), J I) say P(x..), J takes the form, P(x.. ) = P(l) (Xl')') f(x. l'x. ' 'x.. ) I) I) I) l)n ij () where the "correction factor" f(x. 'l'x. '""'x.. ) is what one multi- I) I) l)n.. I) plies the standard binomial probability distribution by in order to "cor- rect for" the lack of mutual independence among the is standardized to X. 'k's. I) If Z"k = (X. Ok - p.)/ip. (I-p.) I) I) then Bahadur has shown that,

-9- f(x. l'x. ' 'x.. ) =+ L E(Z. 'kz, 'k')z, 'kz, 'k' ) ) l)n.. k k' ) ) ) ) ) < \ " + E(Z. 'k Z ' 'k'z' 'k")z, 'kz' 'k'z' 'k"+".+e(z. 'lz, '".Z.. )z. :lz, '".z.. k <k'<k" ) ) ) ) ) J J ) )n.. ) ) )n.. ) ) Thus, f(x. l'x. "' 'x.. ) ) J )n.. J order) correlations among the etc., up to the of p. itself. n.. -th ) is a function of the pairwise (or second- X. 'k's, the third-order correlations, ) order correlation, as well as being a function The general expression () is quite complex and motivates one to look for an approximation to an P(x..). ) An obvious procedure for effecting such approximation is to neglect correlations of order higher than are required for reasonable accuracy. For example, if all correlations are taken to be zero, then we are back to the standard mutually uncorrclated case and so we could say that P(l)(x ij ) is a "first-order" approximation to (). If we can ~easonably neglect all correlations higher than order two, then P()(x.. ) = P(I)(X'.)G + L E(Z"kZ"k')z, 'kz"k].) ) L k<k' ) ) ) ) (3) is a second-order apprdximation to that and indeed P(x..). ) P( ) (x.. ) for m ) We caution at this point < m< n. " may fail to be ) non-negative for some values of x.., even though it is always true that n.. ) L )-OP ( ) (x.. ) =. X. - m ) ) For the situation we are considering, we will write E(Z"k Z. 'k') = Corr(X. 'k'x, 'k') )' )' ) ) e. = P. = p. (l..p.), where

-0- so that (3) parameters p. and e. : Cov (X..k ' X..k,) = e., J J then becomes the following explicit function of the two P () (x ij ) = e. p.(-p.) x.. (x.. -l)(l-p.) + (n.. -x.. )(n.. -x.. -l)p. EJ J J J J J - x.. (n.. -x..)p.(-p.)l}. J ) ) <.. (4 ) Expression (4) is a two-parameter alternative to Williams' betabinomial model. If needed, one could certainly employ a third or higherorder approximation to P (x.. ), ) but it has been our experience that P()(x ij ) performs just as well as Williams' model, in addition to having somewhat more theoretical appeal. Bahadur has shown that P() (x ij ) will be a valid probabil i ty distribution if and only if where -n.. (n..-l) ) ) p. -P. ) p. (-p. ). <p < min (--,-- -. -. -Pi Pi (n.. -)p. (-p.) + -4... YO ) (5) Table II below provides values of the lower and upper bounds in (5) for various choices of n.. and p.; because of symmetry, only values for IJ Pi ~ ~ need to be tabulated.,

-- \ TABLE Permissible Ranges of Values for p. Based on (5) for Various Choices of n.. and p.. ) II 0.0 0.30 0.50 (-0.. 000) (-0,49,. 000) (..,,00.000) 3 (-0.037 0.59) (-0.43, 0.636) (-0.333,. 000) 5 (-0.0, 0.300) (-0.043, 0.40) (-0.00, 0.500) 7 (-0.005, 0.3) (-0.00, 0.96) (-0.048, 0.333) 0 (-0.00, 0.00) (-0.00, 0.00) (-0.0, 0.00) 5 (-0.00, 0.0) (-0.004, 0.35) (-0.00, 0.43) 0 (-0.00, 0.00) (-0.00, 0.00) (-0.005, 0.00) A few comments are in order regarding inequality (5) and Table II. First of all, the upper bound is clearly less restrictive than the lower bound; this is desirable since in most (but not all) situations one would expect p. to be positive. Secondly, both bounds become closer to zero as n.. increases, so that, in practice, the largest n.. in a given ) ) data set is associated. with the most restrictive and governing set of bounds. Finally, sample-based bounds using (5) should be imposed to insure that estimates of p. and 8. (e.g., those obtained by maximum likelihood) will not lead to negative estimated probabilities based on (4). Now, the likelihood under model (4) would be where.. L. =n P()(X"), j=l ) i=o,l.

-- Let "'(0) L denote the value of L when maximized subject to tl'\e constraint and let denote the value of L maximized subject, to no constraints on the parameter space (other than (5), of course). Then, an asymptotically valid test of H O : Po =PI versus H a : Po;t PI is obtained by comparing distribution with D.F. with upper percentage points of the In general, this likelihood ratio test, which we will illustrate by example in Section 3, is best carried out using standard computer function maximization routines, since explicit formulas for the maximum likelihood parameter estimates can only be obtained for the very special case \lhen n.. =. ) As al aside, it is of interest to briefly discuss the likelihood ratio test we propose for examining whether the basic assumption of independent Bernoulli trials is valid. Since the reasonableness of this assump- ~ tion generally goes unquestioned in most applications involving the binomial distribution, this test should be of some value in this regard. X For the i-th "I group, say, if ol i is the value of L. maximized subject to the constraint 8 i = 0, then the likelihood ratio test of H o : 8 i = 0 versus HI: 8 i ;t 0 " "'() would be based on the statistic - In (OL. / L. ), which would have asymp totically a central X distribution with D.F. under H O '.. The Special Case n.. = ) Since it helps to highlight some of the differences between our procedure and that of Williams, the special case brief attention. When n.. =, ) distribution of X.. is: J n.. = ) it follows from will be given some (4) that the probability

-l3~ \ x.. P() (x ij ) J 0 I +e. (l~p.) p. (l-p.)-8. p. + e. If ~. and a. denote the mean and variance of Williams' beta distri- bution for the i-th group, then it is fairly easy to show in this special case that his ~. is equal to our p. and that his a. corresponds to our e.. In fact, the two models would be completely equivalent in this very special case (although clearly not in general), except in one respect. Williams' model only allows for a positive intra-litter association since his variance parameter a. is necessarily restricted to be non~negative, while our e. can, of course, take on negative values. Thus, the beta binomial model would be inappropriate in the situation when there is a possible negative correlation between responses within a litter, and so in this sense the correlated-binomial model is slightly more general. In particular, a preliminary likelihood ratio test which favor~ a. < 0 would tend to preclude the use of Williams' model. 3. COMPARISON OF THE BETA-IHNOMIAL AND CORRELATED-BINOMIAL ~ODELS To compare the fits of the beta-binomial and correlated-binomial models to some real-life data, we will again consider the three sets of I data given in Table. are given in the Haseman &Soares paper). In Table III, we have summarized the resul ts of fitting the two models to these three data sets (the required n.. ) values As can be seen, there is little

TABLE III Beta-Binomial and Correlated-Binomial Fits to Control Data in Table I Number Data Set Data Set Data Set 3 of dead implants Expected Expected Expected Observed Observed Observed Beta- Correlated Beta- Correlated Beta- Correatedbinomial binomial binomial binomial binomial binomial 0 4.6 0.45 60 6.8 594.7 86 99.86 7.4 54 39.87 49. 49 4.47 440.43 65 34.74 53.30 83 79.6 8.85 5 94.5 00.90 66 64.03 74. 3 34 4.34 5.3 49 7. 70.33. 5 30.4 36.63 >3 39 40.37 39.36 3 8.97.6 4.46 7.8, "- Parameter l-i=.0900 '" '"p=.093 l-i=.090 '" p=.086 l-i=.0735 '" "p=.0760 estimates 0 =.0056 '"8=.0037 &=.004 8=.007 cr =.005 "8=.007 ---------------------------------------------------------------------------------------------------------------------- X (test of fit) 3.6 6.63 3.37 9.83 5.66 6.5 It.. ".. -,... -,..-I

/e " \ -5- difference between the fits of the two models, and the improvement in fit relative to the ordinary binomial distribution is quite imp~essivc (see Table I). Although we have not attempted to do so here, we can, of course, improve the fit of the correlated-binomial model by allowing for third and even higher-order correlations; such flexibility with regard to model-improvement is not available with IVilliams' approach. To illustrate the use of model (4) in comparing treated and control groups, we will consider the data of Weil [7] used by Williams in his paper. We wish to perform the likelihood ratio test described in Section, which tests H O : PI = Po versus Ha; PI ~ Po with 8 not necessarily equal to 8 0, The restricted maximized log likelihood under H O ' In L(O), has the value - 59.668 with associated restricted maximum... '" " likelihood estimates p = ~7974, 8 =.039, and 8 0 =.095. Fitting model (4) separately to each of the two groups yields the unrestricted maximized log likelihood value, In i:(l), of In L(I) = In (l) + In (l) 0 = -33.664 -. 03 = -54.867 '" '" " with associated unrestricted estimates PI =.757,8 =.030, PO=.8978,... and 8 0 =.004. Finally, I ing value - In ( (0)/ ()) = -(-59.668+54.867) = 9.60, The correspondwhich is significant when compared with Xl/.005 = 7.88. obtained using Williams' testing procedure is 5.77. It should be mentioned here that it is necessary to consider the boundary

-6- conditions (5) when finding "() L and "(0) L, so that the maximum like- lihood estimates will not lead to negative estimated probabilities. As a final example, consider the following set of unpublished laboratory data involving l = l = 0 o group; the entries below are values of pregnant female mice in each x.. /n... ) ) CONTROL GROUP: 0/5, /6, 0/7, 0/7, 0/8, 0/8, 0/8, /9, /9, /0. TREATMENT GROUP: 0/5, /5, /7, 0/8, /8, 3/8, 0/9, 4/9, /0, 6/0. A preliminary likelihood ratio test of H : O e =0 gives Xl =0.3 for the control group and Xl =4.97 versus H: e ~ 0 a for the treatment group; the presence of this significant intra-litter effect in the treatment group precludes the. use of the ordinary binomial distribution to model these data. Using the beta-binomial model, we obtain under H O : ~l = ~O the " 0'0 =.046 and restricted maxim~m likelihood estimates ~ =.599, " 0' =.000, with an associated restricted maximized log likelihood value of In L(O) = -9.8396; under H a : ~l ~ ~O' the unrestricted para- " " ",, meter estimates are ~O =.0776, 0'0 =.007 and ~l =.36, 0' =.064, giving an unrestricted maximized log likelihood value of In () "= -0.387 7.574 = -7.756. Thus, \ which is significant at the 5% level sinc~ X = 3.84., OS

j Ie Using the correlated-binomial model, the parameter estimates under H O : PI = Po are P =.653, 8 0 =.049 and 8 =.06, giving In L(O) = -9.705; the parameter estimates under H a : PI ~ Po -arc Po =.0765, 6 0 =.003 and PI =.60, 8 =.067, giving In L(l) = -0.67-7.005 = -7.9. Thus, Note that the likelihood under the correlated-binomial model is larger in every instance than the corresponding likelihood based on the beta-binomial model, suggesting that the former model is providing a better description of the data. ACKNOWLEDGEMENT The authors have benefited from some helpful discussions with Professor P.K. Sen. I

REFERENCES [] Bahadur, R.R. A Representation of the Joint Distribution of Responses to n Dichotomous Items. In H. Solomon, ed., Studies in Item Analysis and Prediction. Stanford, Calif.: Stanford University Press, 96. [] Epstein, S.S., Arnold, E., Andrea, J., Bass, W., and Bishop, Y. (97). Detection of Chemical Mutagens by the Dominant Lethal Assay in the Mouse. Toxicol. Appl. Pharmacol.~, 88-35. [3] Haseman, J.K. and Soares, E.R. (976). The Distribution of Fetal Death in Control Mice and Its Implications on Statistical Tests for Dominant Lethal Effects. MUtation Res. ~, 77-88. [4] Lachenbruch, P.A. and Perry, J. (97). Testing Equality of Proportion of Success of Several Correlated Binomial Variates. Univ. Nor. Car. Inst. Stat. Mimeo Series No. 767. P.. [5] McCaughran, D.A. and Arnold, D.W. (976). Statistical Models for Numbers of Implantation Sites and Embryonic Deaths in Mice. I Toxicol. Appl. Pharmacol. 38, 35-333. [6] Salsburg, D.S. (973). Statistical Considerations for Dominant Lethal Mutagenic Trials. Env. Hlth. Persp. ~, 5-58. [7] Weil, C.S. (970). Selection of the Valid Number of Sampling Units and a Consideration of Their Combination in Toxicological Studies Involving Reproduction, Teratogenesis or Carcinogenesis. Fd. Co~met. ToxicoZ. ~, 77-8. [8] Williams, D.A. (975). The Analysis of Binary Responses From Toxicological Experiments Involving Reproduction and Teratogenicity. Biometrics~, 949-95. \