Information Identities and Testing Hypotheses: Power Analysis for Contingency Tables

Transcription

1 Information Identities and Testing Hypotheses: Power Analysis for Contingeny Tables Philip E. Cheng 1, Mihelle Liou 1, John A. D. Aston 1, and Arthur C. Tsai 1 1 Aademia Sinia and University of Warwik Statistia Sinia 18 (8), Abstrat An information theoreti approah to the evaluation of ontingeny tables is proposed. By investigating the relationship between the Kullbak-Leibler divergene and the maximum likelihood estimator, information identities are established for testing hypotheses, in partiular, testing independene. These identities not only validate the alibration of p values, but also yield unified power analysis for the likelihood ratio test, Fisher s exat test and Pearson-ates hi-square test. It is shown that a widely disussed exat unonditional test for the equality of binomial parameters is ill-posed for testing independene, and that using this test to ritiize Fisher s exat test as being onservative is logially flawed. Key Words: Chi-square Test; Contingeny Table; Exat Test; Kullbak-Leibler Divergene; Likelihood Ratio Test; Mutual Information. Correspondene: John Aston Institute of Statistial Siene Aademia Sinia 18 Aademia Road, Se Taipei, 115 Taiwan ROC Tel: x jaston@stat.sinia.edu.tw

2 1. Introdution Evaluation of assoiation and independene between two ategorial fators is a lassi topi of interest in statistial inferene. Pearson s elebrated goodness of fit test yielded the hi-square test in the analysis of a ontingeny table (Pearson, 19; 194). ule (1911) introdued a test for assoiation through testing the equality of two independent binomial proportions of one dihotomous fator. Fisher (1934) haraterized the ombinatorial randomization of two-fator assoiation using the extended hypergeometri distribution, whih gave rise to his exat test. By the 193s the philosophy of hypothesis testing had been well established by Fisher (195, 1935), and Neyman and Pearson (198), among others. It also initiated the long debate onerning the two approahes: signifiane testing for Fisher and hypothesis testing for Neyman and Pearson. Testing independene for a table was a notable example in these arguments. While the debate was foused on the notions of indutive inferene, signifiane level, and deision theory for testing hypotheses, the importane of power evaluation was generally aepted (e.g., Fisher, 1946) with the adoption of the idea of identifying appropriate ritial regions for onstruting more sensitive tests. For example, in testing the equality of two binomial parameters by ule s test, the p values and the power at alternatives an be omputed from either the normal approximation or the exat distribution. However, unified power analysis has not been fully developed for Pearson s hi-square or Fisher s exat test for assessing independene in a table. This will be investigated here. Meanwhile, a ontroversial issue arises when using the exat test, due to its disrete nature. With the limited sample spae defined by fixed row and olumn margins, it yields a onservative test when the sample size is not large. Barnard (1945, 1949) disussed this issue using the Convexity-Symmetry-Maximum (CSM) triple-ondition test based on the sample spae of the two independent binomials model. This led to studying the so-alled unonditional test where only one margin of the table is fixed. Another lassi unonditional test proposed in the 195s is essentially a mixture of the exat onditional tests (Bennet and Hsu, 196). The test aims at finding a more powerful ritial region subjet to a nominal signifiane level. However, the advantage over Fisher s exat test an only be ahieved by onsidering biased or raised levels for the onditional tests whih are implemented in onstruting the unonditional test (Boshloo, 197). 1

3 The ritiism of onservativeness of Fisher s exat test reahed a limax when Berkson (1978) dispraised Fisher s exat test using arguments based on ule s test for the equality of two independent binomial proportions. Sine then, ule s test has been the most widely disussed exat unonditional test in the literature. ates (1984) gave supporting arguments for Fisher s exat test, noting that tests for independene in a table must be onditioned on both margins. Most disussants on ates paper agreed with his assertion. However, this remains a debated issue in the literature, primarily due to the lak of unified power analysis for both Pearson s hi-square test and Fisher s exat test. Indeed, a thorough omparison between onditional and unonditional tests has not been undertaken in the literature, and will be onsidered here. The paper proeeds as follows. Tests for independene in a ontingeny table will be defined in Setion. This is followed by a alibration of the p values between the hi-square, the exat and the likelihood ratio tests over the ommon sample spae of their null distributions. The alibration is derived together with a fundamental likelihood identity, defined using mutual information, whih yields proper representations of the p values based on the onditional distributions. In Setion 3, the likelihood identity is generalized to yield an invariane property of information deompositions, whih is used to develop the power analysis at alternative hypotheses where the odds ratios differ from one. This leads to the identifiation of the logial flaw in omparing ule s test with Fisher s test for independene in a table. Appliations of the information identity to two-way tables for testing model-data fit for general assoiation models are illustrated in Setion 4. In onlusion, we note that Fisher s most relevant set (Fisher, 1935; Bartlett, 1984) is haraterized, where a unified power analysis of Pearson s hi-square test and Fisher s exat test is validated.. Testing Independene in a Contingeny Table In the analysis of ategorial data, a fundamental problem is to deide whether an attribute A (or not A) is randomly alloated to two mutually exlusive subpopulations defined by another dihotomous fator. The statistial question is to test whether independene, or no assoiation, holds between the two dihotomous fators. In ertain designs of experiments, a random sample is often seleted from the entire population to assess the odds of having the attribute A in the two subpopulations (e.g., Lehmann, 1986, Setion 4.7). The observed data (with sample size N) are frequeny ounts, whih are expressed as a ontingeny table:

4 Group 1 Group Total A A Total x x x x x x 1 x x N 1 (.1) A general probability struture of the table of (.1) is the multinomial model, whih defines the distribution of the four mutually exlusive ategories in the population. With a fixed sample total N, the data is illustrated by the probability model: where PX { = ( X = x, X = x; X = x, X = x )} N! = p p p p x! x! x! x! x11 x1 x1 x {( p 11, p 1; p 1, p ): 1 i= 1 p j= 1 ij } Ρ= =, (.) is the parameter spae with three degrees of freedom (d.o.f.). The units of the two rows may be randomly seleted from the two mutually exlusive subpopulations separately, and the units having fator A are ounted. This defines two independent binomial samples with the row margins fixed, whih form Groups 1 and of table (.1) (e.g., ule, 1911; Barnard, 1947; Pearson, 1947). In this ase, the total ount x 1 ( = x + x ) of fator A is a random variable, and onditioned on the row margins, 11 1 formula (.) yields { =, = + = } P X x X x X X x x1 x = p q p q x x 11 1 x x x11 x1 x1 x x1 x = q1 q x11 ψ + x11 + x1 p q x11 x1 exp[ log ( )log( / )]. (.3) Here pi = pi 1/( pi 1+ pi), i = 1,, qi = 1 pi, form a parameter spae with two degrees of freedom. The funtional parameter ψ = p11 p / p1 p1 = pq 1 / pq1 is alled the odds ratio or, the ross-produt ratio. Clearly, knowing the p ij s implies knowing pi, i= 1,, and thus, knowing ψ ; but not onversely, exept that p1 = p 3

5 holds when ψ = 1. Another ommonly disussed experiment is the two omparative binomial trials (e.g., Barnard, 1947; Plakett, 1977; Kempthorne, 1978; ates, 1984; Little, 1989; Greenland, 1991). The model assumes that x 1 out of N individuals are randomly assigned to one of two treatments, yielding Group 1, and the remaining x to another, forming Group of table (.1). Under model (.3), it is often assumed that the individual status of arrying attribute A is unhanged, and the olumn margins of table (.1) are also onsidered fixed. Thus, randomization of the units, with or without the attribute A, haraterizes the extended hypergeometri distribution (Fisher, 1934; Johnson and Kotz, 1969): x = = 1, = 1 =, x11 x1 Ct ( ψ ) { } P X x X X x X X x x x ψ (.4) where C ( ψ ) t x x min( x1, x 1) 1 z = ψ z= max(, x1 x ) z x1 z. It is well known that onditional on both margins x1 and x 1, any entry, say, x 11, is suffiient for ψ ; and model (.4) defines a ase of one-parameter inferene that an be fully illustrated by the likelihood priniple, for example, Birnbaum (196)..1 Classial Tests for Independene Here three tests of independene will be onsidered. The notion of independene between the two fators is defined in the likelihood (probability) equation as ψ = 1, the odds ratio equals to 1. The null hypothesis of independene speifies a omposite hypothesis with two d.o.f. (Kendall and Stuart, 1979, p.578): { ψ } H = ( p, p ; p, p ); = p p / p p = 1. (.5) Pearson (194) developed a hi-square test for based on his goodness-of-fit test (Pearson, 19) under the multinomial model (.). The test is defined with both margins x1 and x 1 held fixed, without assuming independene between rows, and so, like (.4), it is termed a onditional test (ates, 1984). A simplified version is H 4

6 N( x x x x ) N( x x x x N /) = =, (.6) xxxx 1 1 xxxx 1 1 where the seond fration, defined as (ates, 1934), inludes the ontinuity orretion for a more aurate approximation to its distribution. The obtained and values an be ompared to the table of hi-square distribution with one d.o.f. (Fisher, 19). Conditions for or against the use of ontinuity orretion (Plakett, 1964; Grizzle, 1967), and the median probability alternative suggested by Lanaster (1949) have been muh disussed, as reviewed by Upton (198) and ates (1984), among others. While are must be exerised with multiple ommon and small values, when the table margins are small and symmetri, generally performs well as evidened by the alibration study of Setion.4. ule (1911) initiated under model (.3) a statisti for testing p disussed testing the equality : p H 1 H = p of the two binomial parameters:, whih atually = x / x x / x x x N N x x (.7) p The margin x 1 is a suffiient statisti for the ommon value p 1 = p under H, p but not anillary for ψ (f. Plakett, 1977; Little, 1989) under. The former, H has one d.o.f. under (.3), while margin H H has two d.o.f. under (.). Sine only the row p x1 is held fixed, is an unonditional test for H. It follows from (.6) and (.7) that the equality = holds. However, whether the two tests yield p equivalent effets for testing, or H, has not been rigorously examined in the H literature, but will be investigated in this study. The third lassial test is the widely disussed exat test (Fisher, 1934), whose test statisti is denoted by. The test statisti an be represented by any entry of the T E 5

7 table (.1), say, X11 ( = x11). Sine the two margins x 1 and x 1 are fixed, it is an exat onditional test. The null distribution of is the onditional distribution of (.4) with ψ = 1, namely, the hypergeometri distribution: T E x1 x x11 x 1 x11 P{ X11 = x11 X11 + X1 = x1, X11 + X1 = x 1} =. (.8) N x 1 The finite (disrete lattie) sample spae that supports the distribution (.8) onsists of all x tables having the same margins x1 and x 1, denoted by {( x, x x ; x x, x x x ): max(, x x ) x min( x, x )} X = + (.9) For observed data (.1), the p value of under H is the extremity probability, TE defined to be the sum of the probabilities, given by (.8), for the members in X, whose probabilities are not greater than those of the observed data. The number of elements in X is equal to the minimum of the four margins plus 1, whih is less than ( x + 1)( x + 1), the number of elements in the sample spae of the independent 1 binomial model (.3). When the sample size N is small, hene X is small, a p value of the exat test an be substantially less than a nominal signifiane level. While inreasing the p value by randomization is often unaeptable, the exat test has been ritiized as being rather onservative (Berkson, 1978). This is most remarkable when omparing T with among others, for testing under model (.3). E H A ommon trait of the three tests, and T E is that they all measure the deviation from independene using both margins of data (.1). While onservative in terms of p value defined by the hypergeometri distribution (.8), it does enjoy a large sample approximation to normality under model (.4). By Stirling s approximation, a standardized version of the test statisti, or ( = x ) of (.1), is asymptotially standard normal under 1969; Cox and Snell, 1989, p. 48): H TE X11 11 T E (Pearson, 1947; Feller, 1968; Lanaster, is 6

8 E = x 1x1 x11 N x 1xxx 1 N ( N 1). (.1) In general, asymptoti normality of X 11 holds under model (.4) if, and only if, x x x x / N tends to infinity as N does (Kou and ing, 1996). It is seen that E = on X, and E = if ( N 1) in (.1) is replaed with N. Moreover, using fixed x1 and x 1, the test statistis and E are invariant with respet to the sample odds ratios, and is invariant with respet to the differene between the two binomial sample rates of data (.1). Despite these ommon properties, asymptoti power evaluations under a simple alternative to have not been H established for and E, hene TE. However, has the exat independent binomial power analysis. On this issue, it is noteworthy that a lassial unonditional power analysis for testing p1 = p is based on seleting a ritial region among the mixtures of ritial regions defined by the hypergeometri distributions having the same x1 and N, but different x 1 of (.8) (e.g., Bennet and Hsu (196); Boshloo (197); Gail and Gart (1973); and Mehta and Patel (198)). The main onern in these studies is on finding a wider ritial region, for whih the onditional levels of signifiane an be raised above a nominal level in order to reah the unonditional nominal level. Sine the derived power alulation is also a sum of independent binomial probabilities, omparable to those of, similar disussions are omitted. Among the lassial unonditional tests, it is well known that Barnard disqualified his CSM test (1945, 1949). Many studies with unonditional tests have been using to define the p values and ritial regions, e.g., Berkson (1978), Suissa and Shuster (1985), and Haber (1986). These authors disussed under model (.3) the aquired level and power of the test with the aim of seleting a more powerful (atually wider) ritial region, see, for example, Santner and Duffy (1989), Agresti (199), Berger and Boos (1994), and Berger (1996). The dispraise of the exat test by Berkson (1978) that strongly advoates as a substitute for T E, has gradually reeived less onsensus, sine ates (1984, p. 433) argued that testing for must be onditioned on both margins, whether data (.1) is obtained from any one of the three experiments (.), (.3) and (.4). Readers may refer to Berkson (1978) for the H 7

9 introdution of the ritiism, and to Barnard (1979), Upton (198), and ates (1984) for the details of the debate. The logi behind the omparisons between the onditional and unonditional tests were notably disussed by Little (1989) and Greenland (1991), who also deemed the use of the unonditional inferene for testing H suspet. Nevertheless, the notion of a onservative T, as ompared to, has ontinued to be aknowledged among many statistiians, inluding, Kempthorne (1978), Upton (199) and Agresti (, p. 96). E The above literature on the lassial tests for H signals two important issues. First, if a unified power analysis holds for, E and T E altogether, then, it H p X TE would likely justify that testing independene ( ) should be onditioned on the sample spae. Naturally, the seond issue is whether the unonditional test for H using should be legitimately ompared against the exat test for testing H. These two issues will be addressed in this study using information identities developed through the likelihood ratio test.. Likelihood Ratio Test and Conditionality It seems useful to examine the relationship between the hi-square test and the exat test, based on the likelihood ratio test (LRT) statisti. Additional notations are defined for ease of exposition. Let PX ( ) = PX ( = ( x, x; x, x )) = ( x, x; x, x )/ N denote the observed sample proportion, that is, the empirial multinomial distribution. By (.), for P= ( p11, p1; p1, p) Ρ, let Pi i and P i j be likewise defined as the row and olumn margin probabilities; moreover, let ( X ; P ) denote that P is the true distribution of X, and let f ( X; P ) be the orresponding likelihood funtion. Given data X = x of (.1), the LRT statisti λ = max f ( X ; Q Q H ) / f ( X ; P ( X )) for testing P H satisfies the equation: λ x, 1 ij Nx i j ij xi x = j log = log( / ) = ( 1 ( + Op N 1/ )). (.11) Here, the first equation follows from maximum likelihood estimation and the seond is asymptotially valid for large N (Kendall et al., 1979, p.579; Wilks, 1935). The 8

10 seond term of (.11), divided by the sample size, defines the Kullbak-Leibler (1951) divergene i, j= 1 ij ij i j ij ij i j DPx ( ( ) Px ˆ( )) = ( x / N )log( Nx / xx ) = p ( x )log( p ( x ) / p ( xp ) ( x )) and haraterizes the LRT statisti for the observed data x as f( x; Q) f( x; Pˆ ( x)) ND( P( x) Pˆ ) max = = e, (.1) Q H f ( x ; P ( x )) f ( x ; P ( x )) where Px ˆ( ) = p ( xp ) ( x ) = ( xx, xx ; xx, xx )/ N i j is the unique MLE of Px ( ) under H. Clearly, (.1) is also valid for any table x in X, and xˆ = NPˆ( x) = NPˆ defines the same ˆP for all x. Although ˆx need not be a member of X, it has the same margins as x, and lies in the ontinuum extension (defined in (.15)) of the finite disrete lattie X. For the observed table X, values of (.1) over the sample spae X may be normalized to form a disrete onditional distribution. Equivalently, let CR represent a one-sided ritial region, that is, a one-sided boundary subset (see (.16) for detailed formulation) of X, then the onditional distribution of the LRT (.1) evaluates X C ( ( ); ˆ ( ) i ) ND( P( x ) ˆ j P) e P X CR P x =, (.13) SP ( ˆ) xj CR to yield the aquired p value, where ( ˆ( )) ( ˆ j SPx SP) e = = x X j ND( P( x ) Pˆ ) is the normalizing onstant. An analogue of both (.1) and (.13) an also be obtained for the hypergeometri distribution of the exat test as T E and f( x; Pˆ ( x)) e f( x; P( x)) ( N+ 1 ˆ ) D( P( x) P) ( ( ); ˆ ( ) i ) ( N+ 1 ˆ ) D( P( X j ) P) e P X CR P x =, (.14) S ( Pˆ ) X j CR E where S ( ˆ E P ) is the same normalizing onstant having the exponent N replaed by 9

11 N + 1/. Formula (.14), derived from Stirling s formula, losely approximates the exat distribution (.8). Suppose that data situated on one side of X = x has odds ratio ψ x = x11x / x1x1, and that x is xˆ = NPˆ ( x) (say, ψ x > ψ ˆ ( ) = 1) on X. An enlarged ideal P x sample spae an be defined as X = the ontinuum (lattie extension) of X, whih C onsists of all tables with non-negative entries (but not neessarily integers) and the same margins as any members in (the lattie) X. Speifially, { x( ε ) ( x ε, x ε, x ε, x ε): x ij ε, real } X = = + + ± ε. (.15) C Figure 1 exhibits (non-onvex) lattie hyperplanes of relative entropy, and be visualised as the vertial (lattie) line segment that onnets the data X C an x to the MLE xˆ = NPˆ ( x), where it is perpendiular to the null hyperplane. H Figure 1. Central pillar X is the ontinuum extension of the sample spae X ; C H is the null hyperplane with odds ratio 1; H1 has a unique odds ratio. 1

12 Now, onsider a one-sided ritial-region subset of X. Let X = x be observed with ψ > 1, and define the one-sided (boundary set) ritial region by x C CR x ( x11 + ε)( x + ε) = x( ε) X C : ψ x( ε ) = ψ. ( x1 ε)( x1 ε) (.16) x The approximation to the hi-square distribution (.6), together with equation (.13), establishes the standard weak onvergene, that for large sample size N and ND( P( z) Pˆ ) ( ( ) ˆ e dψ ND P x j P) e CRx ( ( ) ˆ ˆ ND P z P) x ( ) j CR SP e dψ x z X { ˆ z CR 1 x ( ); ( )} = ˆ { > ( ( ) ˆ) } C ND( P( z) Pˆ ) P CRx P x P ND P x P ND( P( z) P) e dψ z X e dψ C,, (.17) where the random table is realized as a member z in X. The last term of (.17) C is replaed by 1 P{ > ND( P( x) P)}/ when ψ 1, whih rarely ours as a ˆ x pratial hoie of a CR. Thus, (.17) an be used to estimate the p value of any observed table in X with nonnegative entries and an arbitrary odds ratio. C The above analysis shows that the onditional distribution of the hi-square test and the LRT are losely omparable to that of the exat test. It is of interest to examine whether the same haraterization from (.11) to (.17) holds over the entire parameter spae of testing independene..3 Likelihood Ratio Test and Mutual Information The first step is to examine whether the alibration (.13) would be valid not only for the MLE Pˆ but also for any member of the null hypotheses H. At the outset, this seems to be a redundant issue, sine the LRT (.1) is maximized over all members of. However, the logial notion is: Suppose any individual parameter H Q of H Q were a hypothetial alternative to Pˆ, would it possibly affet the validity 11

13 of (.13)? This is answered below by Lemma 1, using the definition of mutual information (Gray, 199). The observation also provides a fundamental haraterization of the MLE, but differs from the additivity of the minimum disrimination information, disussed for asymptotially optimal hypothesis testing proedures (e.g., Gokhale and Kullbak (1978)). The proof of lemma 1 is elementary and omitted. Lemma 1. (The Pythagorean Law of Relative Entropy) For given data X, P = P( X) and for any Q H, the mutual information yields the MLE Pˆ via the identity: D ( P Q) = D( P Pˆ) + D( Pˆ Q). (.18) The term Pythagorean law is oined by the fat that (.18) partitions the approximate hi-square distribution with three d.o.f. for a table (f. Kendall et al. 1979, (33.117); Rao, 1973, (6d..6)), as shown by Figures 1 and. By (.18), the term mutual information between a pair of random variables ( X, ) with joint probability density f ( xy, ) P an be equivalently defined as I( X ; ) = DP ( Pˆ ) = min D( f g). (.19) g( x, y) H As a onsequene of Lemma 1, (.13) an be generalized over. The following theorem follows by inorporating (.18) into an analogue of (.13), and anelling the H ommon fator D( Pˆ Q), thus the proof is omitted. Theorem 1. For data ( X, PX ( )) of (.1), any Q H, and for eah one-sided boundary subset CR of X, the following equation holds for testing X PX ˆ( ) = P ˆ against X Q in distribution, ND( P( x ) ˆ j P) e P( ( Xi CR); Q) =, (.) SP ( ˆ) xj CR where Pˆ is the projetion of the KL-divergene from PX ( ) onto H. 1

14 The right-hand side of (.) is the same as that of (.13), as expeted. Theorem 1 establishes that, by the onditionality priniple, testing the omposite is equivalent to testing the single null parameter H P = Pˆ, and that the unonditional MLE under model (.) redues to the same Pˆ as the onditional MLE under model (.4); moreover, the redution passes through model (.3). It also haraterizes the MLE of the LRT as the projetion root of the KL-divergene, whih is the mutual information under general hypothesis testing for independene. In the literature, the asymptoti hi-square distribution for the parametri LRT was also examined by Chernoff (1954); and, given a uniform (improper) prior supported on, the posterior mode is the projetion of the KL-divergene (Lindley, 1956). H.4 Calibration of Conditional Tests Distributions of the onditional test statistis were generated for a omparison study. Two tables with different sample sizes were evaluated using the onditioned sample sets X with fixed margins. The data table X = (5, 6;, 5) yielded 8 members in say, X, and (135, 19; 75, 1) X = (16, 8; 9, 15) yielded 4 tables. A large sample size table, would show loser approximation between the test statistis, but for brevity, it is not reported. Tables 1 and list the p values obtained by the four tests. These are one-sided p values assoiated with one-sided (upper) ritial regions, onsisting of tables in X whose odds ratios are inreasingly greater than those of the given data X. For example, Table 1 lists, for eah test, eight asending p values with orresponding odds ratio values ψ in the left-end olumn. The p values inrease toward 1 as the values of ψ derease toward, and a boxed p value orresponds to a one-sided ritial region onsisting of tables that are greater in ψ and more extreme (in probability) than the observed data X. In Tables 1 and, the two hi-square statistis of (.6) plus their p values, the p values for the exat test (.8), and those for the LRT (.13) were all alibrated together on the same sale, by mathing the same one-sided ritial region with eah individual member of the finite sample spae. X The alibration results of the tables in X, inluding the few examples presented here, an be summarized as follows. By formula (.13), the omputed p values of the exat test T E of (.8), the LRT (.1), and the ates of (.6) are very lose to eah other. As is well known, ates p values an be over-orreted when the table margins are small and symmetri, however, in other situations it losely approximates the p values of the exat test. The p values of the LRT are more leptokurti in the T E 13

15 enter and lighter in the tails, refleting the well-known most powerful (unbiased) property of the LRT. However, the p values of the Pearson are generally muh smaller, giving the most liberal results among the four tests. In general, the exat, the LRT and the ates hi-square tests yield similar p values onsistently, inluding small values near the ommonly used nominal levels suh as α =.1,.5 and Power Analysis for Testing Independene It is well known that the odds ratio plays an important role in the appliation of generalized linear models for studying biomedial, environmental, epidemiologial and pharmaeutial experiments. The onditional distribution of X given the row and olumn margins depends on a single parameter, say, the odds ratio. Lemma 1 and Theorem 1 have shown that the (lattie) hyperplane of odds ratio 1 identifies the omposite null hypothesis with two d.o.f. In ontrast, eah hyperplane of the H omposite alternative hypothesis H 1 onsists of the probability vetors having the same odds ratio unequal to 1. It is meaningful to extend the sope of Lemma 1 from the null hypothesis to general alternatives, that is, to examine whether the onditioning argument (.) would be valid if is replaed with H. H Invariane of the Entropy Identity To develop the power analysis, the notations used in Setion plus some others will be reorganized for ease of exposition. Let ( X = ( x, i, j = 1,); P( X be the ij )) * observed data. Let ( ( x ); Q'( )) be any member of H, having odds ratio = ij 1 ψ = x x / x x 1. It is straightforward to find the unique fourfold vetor * * * * ' ( X ' = ( x ); P( X ') = P') on the ontinuum XC, having the same odds ratio ij ψ (see Figure 1). An invariane property of the onditional distributions with respet to both and H H 1 will hold as an extension of the information identity of Lemma 1. The proof of Lemma is given in the Appendix. Subsequently, notations will be simplified, say, P and Q will be used, instead of PX ( ) and Q ( ). Lemma. (Extended Pythagorean Law) Let ( X ; P ) be an observed table 14

16 of (.1). Let ( ; Q') H1 and ( X '; P ') have the same odds ratio ( ψ 1), where X and X ' are members of. The following additive law of information holds: X C DP ( Q') = DP ( P') + DP ( ' Q'). (3.1) It is noted that P ' is the root of projetion from P onto the hyperplane H ( ψ ) of fourfold vetors having the ommon odds ratio ψ. In the null ase of Lemma 1, H = H ψ =, then P' = Pˆ and (3.1) redues to (.18). Lemma thus extends ( 1) Lemma 1 from the null hypothesis to the entire parameter spae of non-negative odds ratios. The main purpose is to haraterize the power analysis at any alternative in based on the test for. The next theorem, being a natural extension of Theorem 1, H fulfils this goal. The proof diretly follows by using Lemma together with a similar argument to Theorem 1. Theorem. Let ( X ; P ) be a table. Let Q H1 have odds ratio ψ. Then, for a CR subset of X, and a normalizing onstant defined by (.13), the following equation holds for testing H against Q in distribution ND( P( X j ) P') e P( ( X CR); Q) =, (3.) SP ( ') X j CR H 1 where P ' is the projetion of the KL-divergene from P onto H ( ψ ). Like Lemma, if is a member of H then (3.) redues to (.). Theorem Q thus generalizes Theorem 1 and verifies that the onditional distributions of the LRT are invariant with respet to eah ommon odds ratio hyperplane. Analogous to (.17) for testing, (3.) leads to the power evaluations disussed below. H Let ( X ; P ) be an observed table with ψ P = x11x / x1x1 > 1, and let * ( ; Q') = ( x ) H be any alternative with ψ ' > 1, where X and are situated ij 1 on the same side of ˆP ( ψ ˆ = 1 ). Given a nominal level α < 1/, let CRX, α X P be a one-sided boundary set, as defined by (.16), satisfying an analogue of (.17): Q C 15

17 1 {( ˆ, } { ˆ X α); ( ( α) } α = P CR P P > ND P X P), (3.3) where DPX ( ( ) Pˆ) = min DP ( ( ) Pˆ). Note that, for α = 1/, the obvious α CR x, α hoie is ˆ X α = P. It is straightforward to ompute the power of the test (defined by CR α ) at the alternative hypothesis ( ; Q') aording to (3.), X, 1 P ( CRX, α Q') { ( ( ) ')} P > ND P X P α, if ψ X > ψ P' 1; or, α 1 1 { ( ( ) ')} P > ND P X P α, if ψ P' ψ X α > 1. (3.4) Fisher (196) illustrated a onfidene interval for the odds ratio parameter given a table. The analysis was essentially an analogue of (3.4). Theorem has onveyed two pratial messages through (3.3) and (3.4). First, a ritial region with unbiased level via (3.3), plus a desired sensitivity via (3.4), an be designed within the ontinuum sample spae. Thus, the information identity (3.) establishes a Neyman-Pearson deision inferene within this testing frame. Seond, p values of (.17) and power omputations with (3.3) and (3.4) are validated not only for the LRT and the Pearson-ates hi-square test, but also for the Fisher exat test in lieu of (.14). The exat test obtains the power evaluation by the LRT approximation to the KL-divergene defined on X, the extension of X, and also of the support of the extended hypergeometri distribution. Altogether, the above disussion has addressed the first issue of setion.1: testing is essentially onditioned on the sample spae X. X C C H 3. Power Analyses in Pratie Data desribing an experiment of vaine inoulation, for the immunization of attle from tuberulosis (Kendall and Stuart, 1979, Table 33.4, p. 616), is used for illustration. The table is X = (nv-a = 8, nv-na = 3; v-a = 6, v-na = 13), where the row letters nv and v stand for no-vaine and vaine-inoulated, with margins 11 and 19; the olumn letters a and na stand for tuberulosis-affeted and unaffeted, respetively, with margins 14 and 16. The odds ratio of data X is ψ = Under H, the MLE is ( Xˆ; Pˆ = P( Xˆ)) with X 16

18 P ˆ = (5.13, 5.87; 8.87, 1.13)/3, the observed one-sided p value of the Pearson is.15, and similar p values of the and the exat are lose to.36. T E For power evaluations in aordane with the Neyman-Pearson theory, the nominal signifiane level α =.5 is hosen for a detailed disussion below. The disrete sample spae X, indued by the observed table X, ontains 1 members. It follows by (3.3) that X α = (7.3, 3.7; 6.7, 11.3) X C defines the boundary of a one-sided (larger-odds-ratios) ritial region at level α =.5. To give an example of a ase of power analysis using Thoerem, hoose a member ( X ' = (7, 4; 7, 1); P'= P( X ')) in X, with odds ratio ψ ' = 3. Let denote the lattie hyperplane of all tables having the same odds ratio 3 and sample size N = 3. Thus, X ' is loated on H1, indeed, X ' = X H1. Given the level-α, a one-sided ritial region with boundary X α, the power evaluation at ( X '; P ') P H 1 yields.438 as the omputed in (3.4). In addition to the lassial omparison in terms of p values, it is meaningful to ompare the onditional tests with the unonditional test based on the power evaluations arried out in the data example above. This will be examined using model (.3) as a ommon ground for omparison. Thus, suppose the null hypothesis speifies that the proportions of the vaine-inoulated units are the same aross the p olumn fator, affeted and unaffeted, denoted by H : p1 = p. Under model (.3), the sample spae is haraterized by the olumn lattie hyperplane that onsists of 55 members of tables with the same olumn margins. Using test sores of, the boundary table of a typial one-sided ritial region is found to be ( ; P( ) = (13, 11; 1, 5)/3 = Q ) having p value.498, and odds ratio α α α Meanwhile, onsider the alternative table ( ; P( ) = (9, 6; 5, 1)/3 = Q ) with odds ratio 3. It is loated on the lattie line H 1 H, whih ontains the table ( X '; P ' = (7, 4; 7, 1) / 3 ) = H 1 X C (see Figure ). For the pairs (Q α ; P ' ) and (Q ; α Q ), the test evaluates the exat binomial probability as the power at P ' and to be.48 and.436, respetively. The two power values are not equal, Q though not far from.438, the onstant previously obtained for both pairs H ( X ; P ') and ( Xα ; Q ) by (3.4), beause P ' and Q have the same odds ratio. Obviously, α one an not expet the test to be more powerful than the LRT, and the exat 17

19 T E. Figure. H, or H, is a binomial produt sample spae with fixed olumn, or row, margins, respetively; r H 1 to X ; X' = H X and X = H H. C 1 C is the horizontal hyperplane of odds ratio 3 perpendiular r By symmetry, similar omparisons of power ould also be obtained using the null hypothesis that the proportions of affeted units are equal aross the other fator, vaine-inoulated or no-vaine. It would, however, yield different ritial region and power alulations on a row hyperplane (Figure ), from those derived with H, noting that H and H are perpendiular planes as remarked after (.18). r H r 3.3 The Logi of Testing Independene It was noted in Setion.1 that the test statistis, and E are essentially equal, based on the same margins of a table. The statisti has been widely p used for testing H : p1 = p under model (.3) with exat power omputations at speified p i s. In ontrast, Theorem addresses the first question of Setion.1 by proving that both and E, hene T E, evaluate asymptoti power in terms of usual approximations to hi-square distributions as used for testing. H The seond issue of Setion.1 is thus addressed as it is not legitimate to ompare the unonditional test against the exat test TE, or E, for testing H under 18

20 model (.3). The hypothesis of independene H, ψ = 1, is universally defined and irrelevant to whihever model (.), (.3) or (.4) is assumed. From model (.) to p model (.3), the two d.o.f. is redued to the one d.o.f. ( p = p ); and H H 1 onversely, the alternative hypotheses parameter spaes are of one, and two, d.o.f., respetively, as shown by Figures 1 and. Reall the example of Setion 3., where the same power values are obtained by the onditional tests at the alternatives P ' and, having the same odds ratio on H. But, the unonditional test must Q 1 treat P' = ( p1 =.5; p =.5) and Q = ( p1 =.644; p =.375) differently, due to unequal ratios p / 1 p, whih are and 1.71, respetively. The interpretations of the two tests are different in meaning, or in purpose. Sine Fisher s exat test was defined for testing independene, this illustrates that it is logially flawed to ompare an unonditional test to a onditional test for testing independene under model (.3). 4. Appliations of the Information Identity Beyond the tables, multivariate data strutures in the form of ontingeny tables have been widely studied in the literature. To illustrate the idea, the onditioning argument of Setion 3 an be applied to testing basi assoiation models in two-way ontingeny tables. Appliations to general multi-way ontingeny tables will be presented in forthoming studies. 4.1 The Basi J Tables It is well known that testing for uniform assoiation or for independene within a J table, J 3, is equivalent to testing that the onseutive odds ratios between the two rows are all equal to 1. The LRT or the Pearson hi-square test is ommonly used with d.o.f. J 1, whih is the number of onseutive odds ratios that an be estimated or tested within the table. As an extension of Lemma, it is often useful to test an alternative hypothesis suh as a goodness-of-fit test, where the onseutive odds ratios may follow speified proportions or a trend. In Setions 4.1 and 4., the geometri viewpoint of Setion 3 is used to illustrate the division of information in testing for these general forms of assoiation. Let X and be two J tables with equal sample size. Assume that the J 1 odds ratio parameters of are known or speified, or identially equal to a positive onstant as a speial ase of uniform assoiation. Treating as an alternative hypothesis, a test of goodness-of-fit or similarity between X and an be examined by an information identity like (3.1) of Lemma. Extensions of Lemma 19

21 3 to I J tables will be disussed in Setion 4.. Lemma 3. Assume X and are J tables as defined above. Then, there exists a unique J table, having the same row and olumn margins as X, and having the same J 1 onseutive odds ratios as those of, and an analogue of Lemma holds: D( X ) = D( X ) + D( ). (4.1) A proof of Lemma 3 is given in the Appendix. The asymptoti distributions assoiated with the sample versions of (4.1) satisfy that J 1 J 1 = The I J Tables A general family of two-way assoiation models for the I J tables will be disussed. This subsetion will disuss an appliation of Lemma 3 for testing ertain uniform assoiation models for two-way tables. The tehnial extension of (4.1) is based on the assumption that the odds ratios along the rows, olumns, or rowsby-olumns are in proportion. It is found that omputing the MLE s of the odds ratio parameters by minimizing the relative entropy of (4.1) is an alternative method to standard log-linear modelling for fitting row and olumn effets of homogeneous assoiation (f. Goodman, 1984, Chapter 4, Table 4). Models of no assoiation, homogeneous assoiation, and speified odds ratios of Setion 4.1 are the basi models. For I 3, it follows from Goodman s terminology (1984, pp ) that models of interest are speified by the row effets, the olumn effets, and the row-by-olumn effets, with model parameters : ψ ψ = ψη (4.) i+ i+ = ψη (4.3) + j + j ψ = ψη η, (4.4) ij i + + j respetively, where I 1 J 1 η i= 1 i+ η j= 1 + j = 1, and = 1. (4.5)

22 Under the first onstraint of (4.5), the row effet model (4.) will estimate I 1 row effet parameters under the null model, and enjoy ( I 1)( J 1) ( I 1) = ( I 1)( J ) d.o.f. Lemma 3 an be used to yield I 1 pairs of rows having the same J 1 odds ratio in eah pair, while minimum KL divergene is ahieved by keeping the row and olumn margins unhanged with eah pair. To math the olumn margins when ombining the pairs of rows, it suffies to resale either the rows or the olumns by appropriate positive onstants to satisfy the solutions for the equations in J olumns. Thus, it estimates the fitted table for testing model (4.) suh that asymptotially DX ( ) enjoys the hi-square distribution with ( I 1)( J ) d.o.f. The argument is likewise appliable to test the olumn effet model (4.3), and then the row and olumn effet model (4.4) and (4.5). This follows from Lemma 3, to yiled an alternative approah to the method of logl-inear models as illustrated by Goodman (1984, Chapter 4). 5. Conlusion It is well known that fatorization of the likelihood defines two important notions, independene and suffiieny, and together they onstitute the likelihood approah to statistial inferene. The LRT, mostly notable in the likelihood approah, has been widely used in testing hypotheses via Neyman-Pearson theory, in partiular, testing independene with ontingeny tables. The alibration of the p values of the onditional tests, a key idea due to Fisher (1935), relies upon the LRT given the margins, whih an be derived from the mutual information identity. The invariane of information identity leads to the development of the (asymptoti) power analyses for both Pearson s hi-square test and Fisher s exat test. It also illustrates that the onditioned (and extended) sample spae offers an answer to Fisher s most relevant set (Barlett, 1984), where onditional distributions and unified power analysis of the LRT, the hi-square test and the exat test are validated. This last observation also resolves the long-term debate on the ritiism of Fisher s exat test. That is, Berkson s dispraise against the exat test, in terms of onservative p values and improved power evaluations, was logially flawed due to the different models and hypotheses under evaluation. X C Aknowledgements The authors are grateful to Shaowei Cheng for areful reading of the manusript plus suggestions; and, are also indebted to the Assoiate Editor and Referees for their many helpful omments. 1

23 Referenes Agresti, A. (199) ( nd ed., ). Categorial Data Analysis. New ork: Wiley. Barnard, G. A. (1945). A new test for tables. Nature 156, 177. Barnard, G. A. (1947). Signifiane tests for tables. Biometrika, 34, Barnard, G. A. (1949). Statistial inferene. J. Royal Statist. So., B, 11, Barnard, G. A. (1979). In ontradition to J. Berkson s dispraise: onditional tests an be more effiient. J. Statist. Planning and Inferene, 3, Bartlett, M.S. (1984). Disussion on tests of signifiane for x ontingeny tables (by F. ates). J. Royal Statist. So., A, 147, 453. Bennett, B. M. and Hsu, P. (196). On the power funtion of the exat test for the ontingeny table. Biometrika, 47, (orretion 48 [1961], 475). Berger, R. L. and Boos, D. D. (1994). P-values maximized over a onfidene set for the nuisane parameter. J. Am. Statist. Asso., 89, Berger, R. L. (1996). More powerful tests from onfidene interval p values. Amer. Statist., 5, Berkson, J. (1978). In dispraise of the exat test. J. Statist. Planning and Inferene,, 7-4. Birnbaum, A. (196). On the foundations of statistial inferene (with disussion). J. Am. Statist. Asso., 57, Boshloo, R. D. (197). Raised onditional level of signifiane for the when testing the equality of probabilities. Statistia Neerlandia, 4, table Chernoff, H. (1954). On the distribution of the likelihood ratio. Ann. Math. Statist., 5, Cox, D. R. and Snell, E. J. (1989). The Analysis of Binary Data. nd Edition, London: Chapman and Hall. Deming, W. E. and Stephan F. F. (194). On a least squares adjustment of a sampled frequeny table when the expeted marginal totals are known. Ann. Math. Statist., 11, Feller, W. (1968). An Introdution to Probability Theory and its Appliations. 3 rd Edition, New ork: Wiley. Fisher, R. A. (19). On the interpretation of alulation of P. J. Royal Statist. So., 85, from ontingeny tables, and the Fisher, R. A. (195) (5 th ed., 1934; 1 th ed., 1946). Statistial Methods for Researh Workers, Edinburgh: Oliver & Boyd. Fisher, R. A. (1935). The logi of indutive inferene. J. Royal Statist. So., A, 98,

24 Fisher, R. A. (196). Confidene limits for a ross-produt ratio. Australian Journal of Statistis, 4, 41. Gail, M. and Gart, J. J. (1973). The determination of sample sizes for use with the exat onditional test in omparative trials. Biometris, 9, Gokhale, D. V. and Kullbak, S. (1978). The Information in Contingeny Tables. New ork: Marel Dekker. Goodman, L. A. (1984). The Analysis of Cross-Classified Data Having Ordered Categories. Cambridge: Harvard University Press. Gray, R. M. (199). Entropy and information theory. New ork: Springer-Verlag. Greenland, S. (1991). On the logial justifiation of onditional tests for two-by-two ontingeny tables. Amer. Statist., 45, Grizzle, J. E. (1967). Continuity orretion in the Statist., 1, 8-3. test for tables. Amer. Haber M. (1986). An exat unonditional test for the omparative trial. Psyhol. Bull., 99, Johnson, N. L. and Kotz, S. (1969). Disrete Distributions. New ork: Wiley. Kempthorne, O. (1978). Comments on J. Berkson s paper In Dispraise of the Exat Test. J. Statist. Planning and Inferene, 3, Kendall, M. G. and Stuart, A. (1979). The Advaned Theory of Statistis. 4 th Edition, Vol., London: Charles Griffin. Kou, S. G. and ing,. (1996). Asymptotis for a table with fixed margins. Statistia Sinia, 6, Kullbak, S. and Leibler, R. A. (1951). On information and suffiieny. Ann. Math. Statist.,, Lanaster, H. O. (1949). The ombination of probabilities arising from data in disrete distributions. Biometrika, 36, 37-38, Corrig. 37, 45. Lanaster, H. O. (1969). The Chi-squared Distributions. New ork: Wiley. Lehmann, E. L. (1986). Testing Statistial Hypotheses. nd Edition, New ork: Wiley. Lindley, D. V. (1956). On a measure of the information provided by an experiment. Ann. Math. Statist., 7, Little, R. J. A. (1989). Testing the equality of two independent binomial proportions. Amer. Statist., 43, Mehta, C. R. and Patel, N. R. (198). A network algorithm for the exat treatment of the x K ontingeny table. Comm. Statist., Ser. B9(6), Neyman, J. and Pearson, E. S. (198). On the use and interpretation of ertain test riteria for purposes of statistial inferene. Biometrika, A, Pearson, E. S. (1947). The hoie of statistial tests illustrated on the interpretation of 3

25 data lassed in a table. Biometrika, 34, Pearson, K. (19). On the riterion that a given system of deviations from the probable in the ase of a orrelated system of variables is suh that it an be reasonably supposed to have arisen from random sampling. Phil. Mag., Series 5, 5, Pearson, K. (194). Mathematial ontributions to the theory of evolution XIII: On the theory of ontingeny and its relation to assoiation and normal orrelation. Draper s Co. Researh Memoirs, Biometri Series, no. 1. (Reprinted in Karl Pearson s Early Papers, ed. E. S. Pearson, Cambridge: Cambridge University Press, 1948.) Plakett, R. L. (1964). The ontinuity orretion in x tables. 51, Plakett, R. L. (1977). The marginal totals of a table. Biometrika, 64, Rao, C. R. (1973). Linear Statistial Inferene and Its Appliations. nd edition, New ork: Wiley. Santner, T. J. and Duffy, D. E. (1989). The Statistial Analysis of Disrete Data. New ork: Springer-Verlag. Suissa, S. and Shuster, J. (1985). Exat unonditional sample sizes for the binomial trial. J. Royal Statist. So., A, 148, Upton, G. J. G. (198). A omparison of alternative tests for the omparative trial. J. Royal Statist. So., A, 145, Upton, G. J. G. (199). Fisher s exat test. J. Royal Statist. So., A, 155, Wilks, S. S. (1935). The likelihood test of independene in ontingeny tables. Ann. Math. Statist., 6, ates, F. (1934). Contingeny tables involving small numbers and the test. J. Royal Statist. So., Suppl., 1, ates, F. (1984). Tests of Signifiane for ontingeny tables (with disussion). J. Royal Statist. So., A, 147, ule, G. U. (1911). An Introdution to the Theory of Statistis. London: Griffin. Appendix Proofs for Lemmas and 3 will be arried out by a naive approah using assumptions under the onditioning argument. For ease of exposition, the following notation for the table will be used: 4

26 Group 1 Group A a A b d Proof of Lemma. Suppose that the above table defines the fourfold vetor X = ( abd, ;, ) with X ' = ( a', b'; ', d') and = ( a*, b*; *, d*) analogously defined. By the basi identity alog( a/ a*) = a[log( a/ a') + log( a'/ a*)], (3.1) is valid if the next equation holds: alog( a'/ a*) + blog( b'/ b*) + log( '/ *) + dlog( d'/ d*) = a'log( a'/ a*) + b'log( b'/ b*) + 'log( '/ *) + d'log( d'/ d*). (A.1) Using the ommon odds ratio ad ' '/ b ' ' = ψ = a* d*/ b* *, it is found that (A.1) is equivalent to the following equation: a'/ a* '/ * alog + ( a+ b)log( b'/ b*) + log + ( + d)log( d'/ d *) b'/ b* d'/ d* a'/ a* '/ * = a'log + ( a' + b')log( b'/ b*) + 'log + ( ' + d')log( d'/ d *). (A.) b'/ b* d'/ d* It is seen that the four log terms with large brakets are equal due to the ommon odds ratio ψ and a+ = a' + ' sine ( X ; P ) and ( X '; P ') have the same margins. Likewise, as omplete. a+ b= a' + b' and + d = ' + d', (A.) holds and the proof is Proof of Lemma 3 (A revision of Lemma 3, Statistia Sinia, 8). It suffies to prove the ase J = 3 without loss of generality. To fix notation, let X be a 3 table with the first row ( a11, a1, a 13) and the seond row ( a1, a, a3), likewise, * * * * * * has its first row ( a, a, a ) and seond row ( a, a, a ) ; and write in short ' ' ' ' ' ' = (( a, a, a ), ( a, a, a )). By definition, it suffies to prove (4.1) as an equation between three relative divergene terms, where eah one is a sum of six log-likelihood ratios. By (A.1), it is equivalent to verifying the following equation ' between two similar terms whih differ only by the entries and : a ij a ij 5

27 ' a a ij log aij ' 3 ij 3 ' a 1 1 ij log = a i= j= * i= 1 j= 1 ij aij *. (A.3) The assumption on the odds ratios of the tables and speifies that for γ ( a a / a a ) ( a a / a a ( a a / a a ) ( a a / a a ' ' ' ' ' ' ' ' = γ = * * * * * * * * ) ). (A.4) The left-hand-side (l.h.s.) of (A.3) satisfies the equation: 3 i= 1 j= 1 a ij ' a ij log * = a ij a log a log ( a a a )log a ' γ + 1 γ * a13 ' ' 1 ( 11 1)log a a + a + a ( a * a)log * a1 a + [( a + a ) ( a + a + a )]log a ' * a3. (A.5) ' And, the r.h.s. of (A.3) also yields (A.5) with the entries replaed by, beause a ij a ij the row margins and olumn margins of are equal to those of data X. It follows that ' ' ( a a )logγ = ( a11 + a1 )logγ, but a11 + a1 and a ' 11 a ' 1 + need not be equal, and therefore, γ = 1 as desired. This ompletes the proof of Lemma 3, whih is an extension of Lemma to J tables. This proof also indiates that similar results for I tables, and for I J tables, an be useful for the disussion of Setion 4.. 6