Goodness-of-Fit Considerations and Comparisons Example: Testing Consistency of Two Histograms

Transcription

1 Goodness-of-Fit Considerations and Comparisons Example: Testing Consistency of Two Histograms Sometimes we have two histograms and are faced with the question: Are they consistent? That is, are our two histograms consistent with having been sampled from the same parent distribution? Each histogram represents a sampling from a multivariate Poisson distribution. uorv bin index 1 Frank Porter, Caltech Seminar, February 26, 2008

2 Testing Consistency of Two Histograms There are two variants of interest to this question: 1. We wish to test the hypothesis (absolute equality): H 0 : The means of the two histograms are bin-by-bin equal, against H 1 : The means of the two histograms are not bin-by-bin equal. 2. We wish to test the hypothesis (shape equality): H 0 : The densities of the two histograms are bin-by-bin equal, against H 1 : The densities of the two histograms are not bin-by-bin equal. 2 Frank Porter, Caltech Seminar, February 26, 2008

3 Testing Consistency of Two Histograms Some Context There are many tests to address whether a dataset is consistent with having been drawn from some continuous distribution. Histograms are built from such data. So why not work directly with the data? Good question! Maybe original data not available. May be able to use features of binned data, eg, normal approximation χ 2. These tests may often be adapted to address whether two datasets have been drawn from the same continuous distribution, often called two-sample tests. These tests may then be further adapted to the present problem, of determining whether two histograms have the same shape. Also discussed as comparing whether two (or more) rows of a table are consistent. 3 Frank Porter, Caltech Seminar, February 26, 2008

4 Notation, Conventions We assume that we have formed our two histograms with the same number of bins, k, with identical bin boundaries. The bin contents of the first histogram are given by realization u of random variable U, and of the second by realization v of random variable V. Thus, the sampling distributions are: P (U = u) = P (V = v) = k i=1 k i=1 μ u i i u i! e μ i, ν v i i v i! e ν i, where the vectors μ and ν are the mean bin contents of the respective histograms. 4 Frank Porter, Caltech Seminar, February 26, 2008

5 Notation, Conventions (continued) We define: N u N v k U i, i=1 k V i, i=1 Total contents of first histogram, Total contents of second histogram, μ T N u = ν T N v = t i u i + v i, k μ i i=1 k ν i i=1 i =1,...,k Will drop distinction between random variable and realization. 5 Frank Porter, Caltech Seminar, February 26, 2008

6 Power, Confidence Level, P -value Interested in the power of a test, for given confidence level. The power is the probability that the null hypothesis is rejected when it is false. Power depends on the true sampling distribution. Power 1 probability of a Type II error. The confidence level is the probability that the null hypothesis is accepted, if the null hypothesis is correct. Confidence level 1 probability of a Type I error. In physics, we don t specify the confidence level of a test in advance, at least not formally. Instead, we quote the P -value for our result. This is the probability, under the null hypothesis, of obtaining a result as bad or worse than our observed value. This would be the probability of a Type I error if our observation were used to define the critical region of the test. 6 Frank Porter, Caltech Seminar, February 26, 2008

7 Large Statistics Case If all of the bin contents of both histograms are large, we use the approximation that the bin contents are normally distributed. Under H 0, u i = v i μ i,i=1,...,k. More properly, it is U i = μ i, etc., but we are permitting u i to stand for the random variable as well as its realization, as noted above. Let the difference for the contents of bin i between the two histograms be: Δ i u i v i, and let the standard deviation for Δ i be denoted σ i. Then the sampling distribution of the difference between the two histograms is: 1 k P (Δ) = 1 (2π) k/2 exp 1 k Δ 2 i. 2 i=1 σ i i=1 σ 2 i 7 Frank Porter, Caltech Seminar, February 26, 2008

8 Large Statistics Case Test Statistic This suggests the test statistic: k T = i=1 If the σ i were known, this would simply be distributed according to the chi-square distribution with k degrees of freedom. The maximum-likelihood estimator for the mean of a Poisson is just the samplednumber. ThemeanofthePoissonisalsoitsvariance,andwe will use the sampled number also as the estimate of the variance in the normal approximation. We ll refer to this approach as a χ 2 test. Δ 2 i σi 2. 8 Frank Porter, Caltech Seminar, February 26, 2008

9 Large Statistics Algorithm absolute We suggest the following algorithm for this test: 1. For σ 2 i form the estimate ˆσ 2 i =(u i + v i ). 2. Statistic T is thus evaluated according to: T = k i=1 (u i v i ) 2 u i + v i. If u i = v i =0for bin i, the contribution to the sum from that bin is zero. 3. Estimate the P -value according to a chi-square with k degrees of freedom. Note that this is not an exact result. 9 Frank Porter, Caltech Seminar, February 26, 2008

10 Large Statistics Algorithm shapes If only comparing shapes, then scale both histogram bin contents: 1. Let N =0.5(N u + N v ). Scale u and v according to: u i u i = u i(n/n u ) v i v i = v i(n/n v ). 2. Estimate σ 2 i with: ˆσ 2 i = ( N N u ) u i + ( N N v ) v i. 3. Statistic T is thus evaluated according to: k T = i=1 ( ui N u v ) 2 i N v u i Nu 2 + v i Nv Estimate the P -valueaccordingtoachi-squarewithk 1 degrees of freedom. Note that this is not an exact result. 10 Frank Porter, Caltech Seminar, February 26, 2008

11 Application to Example With some small bin counts, might not expect this method 30 to be espe- bin index cially good for our example, but can try it anyway: Type of test T NDOF P (χ 2 >T) P -value χ 2 absolute comparison χ 2 shape comparison Likelihood Ratio shape comparison Kolmogorov-Smirnov shape comparison NA 0.22 Bhattacharyya shape comparison NA 0.32 Cramér-Von-Mises shape comparison NA 0.29 Anderson-Darling shape comparison NA 0.42 Likelihood value shape comparison NA 0.29 Column P -value attempts a more reliable estimate of the probability, under the null hypothesis, that a value for T will be as large as observed. Compare with column P (χ 2 >T) column, the probability assuming T follows a χ 2 distribution with NDOF degrees of freedom. 11 Frank Porter, Caltech Seminar, February 26, 2008 uorv

12 Application to Example (continued) uorv bin index The absolute comparison yields much poorer agreement than the shape comparison. This is readily understood: The total number of counts in one dataset is 783; in the other it is 633. Treating these as samplings from a normal distribution with variances 783 and 633, we find a difference of 3.99 standard deviations or a twotailed P -value of This low probability is diluted by the binby-bin test, but is still reflected in the poor agreement. [Auxillary lesson: The more you know about what you want to test, the better (more powerful) the test you can make. ] 12 Frank Porter, Caltech Seminar, February 26, 2008

13 An Issue: We don t know H 0! Evaluation of the probability under the null hypothesis is in fact problematic, since the null hypothesis actually isn t completely specified. The problem is the dependence of Poisson probabilities on the absolute numbers of counts. Probabilities for differences in Poisson counts are not invariant under the total number of counts. Unfortunately, we don t know the true mean numbers of counts in each bin. Thus, we must estimate these means. The procedure adopted here has been to use the maximum likelihood estimators (see later) for the mean numbers, in the null hypothesis. We ll have further discussion of this procedure below It does not always yield valid results. 13 Frank Porter, Caltech Seminar, February 26, 2008

14 Use of chi-square Probability Distribution In our example, the probabilities estimated according to our simulation and the χ 2 distribution probabilities are fairly close to each other. Suggests the possibility of using the χ 2 probabilities if we can do this, the problem that we haven t completely specified the null hypothesis is avoided. Conjecture: Let T be the χ 2 test statistic, for either the absolute or the shape comparison, as desired. Let T c be a possible value of T (perhaps the critical value to be used in a hypothesis test). Then, for large values of T c, under H 0 (or H 0 ): ( ) P (T <T c ) P T<T c χ 2 (T,ndof), where P ( T<T c χ 2 (T,ndof) ) is the probability that T < T c according to a χ 2 distribution with ndof degrees of freedom (either k or k 1, depending on which test is being performed). 14 Frank Porter, Caltech Seminar, February 26, 2008

15 Is it true? As we ll see, not really worth proving, but here is the idea: Consider the case of one bin. Suppose that ν = μ and that μ is small ( 1, say). Since we are interested in large values of the statistic, we are interested in the situation where one of u, v is large, and the other small (since μ is small). Suppose it is u that is large. Then (u v)2 T = u. u + v For given v (0, say), the probability of T is thus P (T ) μt T! e μ. 15 Frank Porter, Caltech Seminar, February 26, 2008

16 Is it true? (continued) This may be compared with the chi-square probability distribution for one degree of freedom: P (T = χ 2 )= 1 2π e T/2 T. The ratio is, dropping the constants: P (T ) P (T = χ 2 ) μt e T/2 T T! = [ ( exp T 12 +lnμ)], T Γ(T ) which approaches zero for large T, for any given μ. We conclude that the conjecture is valid in the case of one bin, and strongly suspect that the argument generalizes to multiple bins. 16 Frank Porter, Caltech Seminar, February 26, 2008

17 Is it Useful? If true, the conjecture tells us that if we use the probabilities from a χ 2 distribution in our test, the error that we make is in the conservative direction. That is, we ll reject the null hypothesis less often than we would with the correct probability. This conjecture is independent of the statistics of the sample, bins with zero counts are fine. In the limit of large statistics, the inequality approaches equality. Lest we conclude that it is acceptable use this great simplification in all situations, we hasten to point out that it isn t as nice as it sounds. The problem is that, in low statistics situations, the power of the test according to this approach can be dismal. We might not reject the null hypothesis in situations where it is obviously implausible. 17 Frank Porter, Caltech Seminar, February 26, 2008

18 Is it Useful? (continued) Probability(T<Tc) (a) (b) (c) Tc Tc Tc Comparison of actual (cumulative) probability distribution for T (solid blue curves) withchi-square distribution (dashed red curves). All plots are for 100 bin histograms, and null hypothesis: (a) Each bin has mean 100. (b)eachbinhasmean1. (c) Bin j has mean 30/j. 18 Frank Porter, Caltech Seminar, February 26, 2008

19 General (Including Small Statistics) Case If the bin contents are not large, then the normal approximation may not be good enough and the χ 2 statistic may not follow a χ 2 distribution. A simple approach is to combine bins until the normal approximation is valid. In many cases this doesn t lose too much statistical power. A common rule-of-thumb is to combine until each bin has at least 7 counts. Try this on our example, as a function of the minimum number of events per bin. The algorithm is to combine corresponding bins in both histograms until both have at least minbin counts in each bin. 19 Frank Porter, Caltech Seminar, February 26, 2008

20 Combining bins for χ 2 test uorv bin index T or k minbin Probability minbin Left: The example pair of histograms. Middle: ThesolidcurveshowsthevalueoftheteststatisticT,andthe dashed curve shows the number of histogram bins for the data in the example, as a function of the minimum number of counts per bin. Right: The P -value for consistency of the two datasets in the example. The solid curve shows the probability for a chi-square distribution, and the dashed curved shows the probability for the actual distribution, with an estimated null hypothesis. 20 Frank Porter, Caltech Seminar, February 26, 2008

21 Working with the Poissons - Normalization Test Alternative: Work with the Poisson distribution. Separate the problem of the shape from that of the overall normalization. To test normalization, compare totals over all bins between the histograms. Distribution is P (Nu,N v )= μn u T νn v T N u!n v! e (μ T +ν T ). The null hypothesis is H 0 : μ T = ν T, to be tested against alternative H 1 : μ T ν T. We are interested in the difference between the two means; the sum is a nuisance parameter. Hence, consider: P (N v N u + N v = N) = P (N N v)p (N v ) P (N) = μn N v T e μ T ν N v T e ν T /(μ T + ν T ) N e (μ T +ν T ) (N N v )! N v! N! Note! Example of use of ( )( ) Nv ( ) N Nv = NNv νt μt conditional likelihood.. μ T + ν T μ T + ν T 21 Frank Porter, Caltech Seminar, February 26, 2008

22 Normalization test (general case) This probability permits us to construct a uniformly most powerful test of our hypothesis.* It is a binomial distribution, for given N. The uniformly most powerful property holds independently of N, although the probabilities cannot be computed without N. The null hypothesis corresponds to μ T = ν T,thatis: P (N v N u + N v = N) = ( NNv )( ) 1 N. 2 For our example, with N = 1416 and N v = 633, thep -value is , for a two-tailed probability. Compare with our earlier estimate of in the normal approximation. For our binomial test we have conservatively included the endpoints (633 and 783). Mimicing more closely the normal estimate by including one-half the probability at the endpoints, we obtain , very close to the normal number. * E. L. Lehmann and Joseph P. Romano, Testing Statistical Hypotheses, 3rd ed., Springer, NY (2005), Thm 4.4.1). 22 Frank Porter, Caltech Seminar, February 26, 2008

23 Testing the Shape Catalog of Tests Many possible tests. We ll consider 7: Chi-square test (χ 2 ) Bhattacharyya distance measure (BDM) Kolmogorov-Smirnov test (KS) Cramér-von-Mises test (CVM) Anderson-Darling test (AD) Likelihood ratio test (ln λ) Likelihood value test (ln L) 23 Frank Porter, Caltech Seminar, February 26, 2008

24 Chi-square test for shape Even though we don t expect it to follow a χ 2 distribution, we may evaluate the test statistic: χ 2 = k i=1 ( ui N v ) 2 i u N v u i Nu 2 + v i Nv 2 If u i = v i = 0, the contribution to the sum from that bin is zero.. 24 Frank Porter, Caltech Seminar, February 26, 2008

25 Geometric (BDM) test for shape Geometric motivation: Let the bin contents of a histogram define a vector in a k-dimensional space. If two vectors are drawn from the same distribution (null hypothesis), they will tend to point in the same direction (not interested in the lengths of the vectors here). If we represent each histogram as a unit vector with components: {u 1 /N u,...,u k /N u }, and {v 1 /N v,...,v k /N v }, we may form the dot product test statistic: u v k T BDM = = u i v i N u N v N u N v This is known as the Bhattacharyya distance measure (BDM). This statistic is related to the χ 2 statistic the N u u v Nv dot product is close to the cross term in the χ 2 expression. i=1 1/2. 25 Frank Porter, Caltech Seminar, February 26, 2008

26 Sample application of BDM test Apply this formalism to our example. The sum over bins gives According to our estimated distribution of this statistic under the null hypothesis, this gives a P -value of 0.32, similar to the χ 2 test result (0.28). BDM per bin Frequency Bin index BDM statistic Bin-by-bin contributions to the BDM test statistic for the example. Estimated distribution of the BDM statistic for the null hypothesis in the example. 26 Frank Porter, Caltech Seminar, February 26, 2008

27 Kolmogorov-Smirnov test Another approach to a shape test may be based on the Kolmogorov- Smirnov (KS) idea: Estimate the maximum difference between observed and predicted cumulative distribution functions and compare with expectations. Modify the KS statistic to apply to comparison of histograms as follows. Assume neither histogram is empty. Form the cumulative distribution histograms according to: u ci = i u j /N u v ci = j=1 i v j /N v. j=1 Then compute the test statistic: T KS =max i u ci v ci. (We consider only the two-tail test here.) 27 Frank Porter, Caltech Seminar, February 26, 2008

28 Sample application of KS test Apply this formalism to our example. The maximum over bins is Estimating the distribution of this statistic under H 0 gives a P - value of 0.22, somewhat smaller than for the χ 2 test result, but indicating consistency of the histograms. The smaller KS P -value is presumably a reflection of the emphasis this test gives to the region near the peak of the distribution (i.e., the region where the largest fluctuations occur in Poisson statistics), where we indeed have a largish fluctuation. KS distance Frequency Bin index KS statistic Bin-by-bin distances for the KS test statistic for the example. Estimated PDF of the KS distance under H 0 in the example. 28 Frank Porter, Caltech Seminar, February 26, 2008

29 Cramér-von-Mises test The idea of the Cramér-von-Mises (CVM) test is to add up the squared differences between the cumulative distributions being compared. Used to compare an observed distribution with a presumed parent continuous probability distribution. Algorithm is adaptable to the two-sample comparison, and to the case of comparing two histograms. The test statistic for comparing the two samples x 1,x 2,...,x N and y 1,y 2,...,y M is*: NM N [ T = Ex (N + M) 2 (x i ) E y (x i ) ] 2 M [ + Ex (y j ) E y (y j ) ] 2, i=1 where E x is the empirical cumulative distribution for sampling x. That is, E x (x) =n/n if n of the sampled x i are less than or equal to x. j=1 * T. W. Anderson, On the Distribution of the Two-Sample Cramér-Von Mises Criterion, Ann. Math. Stat. 33 (1962) Frank Porter, Caltech Seminar, February 26, 2008

30 Cramér-von-Mises test Adaptation to comparing histograms Adapt this for the present application of comparing histograms with bin contents u 1,u 2,...,u k and v 1,v 2,...,v k with identical bin boundaries: Let z be a point in bin i, and define the empirical cumulative distribution function for histogram u as: E u (z) = i u i /N u. j=1 Then the test statistic is: T CVM = N u N v (N u + N v ) 2 k (u j + v j ) [ E u (z j ) E v (z j ) ] 2. j=1 30 Frank Porter, Caltech Seminar, February 26, 2008

31 Sample application of CVM test Apply this formalism to our example, finding T CVM = Theresulting estimated distribution under the null hypothesis is shown below. According to our estimated distribution of this statistic under the null hypothesis, this gives a P -value of 0.29, similar with the χ 2 test result. uorv bin index Frequency CVM statistic Example. Estimated PDF of the CVM statistic under H 0 for the example. 31 Frank Porter, Caltech Seminar, February 26, 2008

32 Anderson-Darling (AD) test for shape The Anderson-Darling (AD) test is another non-parametric comparison of cumulative distributions. It is similar to the Cramér-von-Mises statistic, but is designed to be sensitive to the tails of the CDF. The original statistic was designed to compare a dataset drawn from a continuous distribution, with CDF F 0 (x) under the null hypothesis: A 2 m = m [F m (x) F 0 (x)] 2 F 0 (x)[1 F 0 (x)] df 0(x), where F m (x) is the empirical CDF of dataset x 1,...x m. 32 Frank Porter, Caltech Seminar, February 26, 2008

33 Anderson-Darling (AD) test, adaptation to comparing histograms Scholz and Stephens* provide a form of this statistic for a k-sample test on grouped data (e.g., as might be used to compare k histograms). The expression of interest for two histograms is: T AD = 1 N u + N v k max 1 j=k min t j Σ j ( Nu + N v Σ j ) {[ (N u + N v )Σ uj N u Σ j ] 2 /Nu + [ (N u + N v )Σ vj N v Σ j ] 2 /Nv }, where k min is the first bin where either histogram has non-zero counts, k max is the number of bins counting up the the last bin where either histogram has non-zero counts, and Σ uj j u i, i=1 Σ vj j v i, and Σ j i=1 j t i =Σ uj +Σ vj. i=1 * F. W. Scholz and M. A. Stephens, k-sample Anderson-Darling Tests, J.Amer.Stat.Assoc. 82 (1987) Frank Porter, Caltech Seminar, February 26, 2008

34 Sample application of AD test for shape We apply this formalism to our example. The sum over bins gives According to our estimated distribution of this statistic under the null hypothesis, this gives a P -value of 0.42, somewhat larger than the χ 2 test result. The fact that it is larger is presumably due to the greater focus of this test on the tails, with less emphasis on the peak region (where the large fluctuation is seen in the example). uorv bin index Example. Frequency Anderson Darling statistic Estimated distribution of the AD test statistic for the null hypothesis for the example. 34 Frank Porter, Caltech Seminar, February 26, 2008

35 Likelihood ratio test for shape Base a shape test on the same conditional likelihood idea as for the normalization test. Now there is a binomial associated with each bin. Start with the null hypothesis, that the two histograms are sampled from the joint distribution: P (u, v) = k i=1 μ u i i u i! e μ i νvi i v i! e ν i, where ν i = aμ i for i =1, 2,...,k. That is, the shapes of the two histograms are the same, although the total contents may differ. With t i = u i + v i, and fixing the t i at the observed values*, we have the multi-binomial form: k ( )( ) vi ( ) ti v ti νi μi i P (v u + v = t) = v. i ν i + μ i ν i + μ i i=1 35 Frank Porter, Caltech Seminar, February 26, 2008

36 Likelihood ratio test for shape (continued) The null hypothesis is ν i = aμ i, i =1,...,k. Wewanttotestthis, but there are two complications: 1. The value of a is not specified; 2. We still have a multivariate distribution. For a, we will substitute an estimate from the data, namely the maximum likelihood estimator: â = N v N u. We use a likelihood ratio statistic to reduce the problem to a single variable. This will be the likelihood under the null hypothesis (with a given by its maximum likelihood estimator), divided by the maximum of the likelihood under the alternative hypothesis. 36 Frank Porter, Caltech Seminar, February 26, 2008

37 Likelihood ratio test for shape (continued) Thus, we form the ratio: λ = max H0 L(a v; u + v = t) k max H1 L({a i ν i /μ i } v; u + v = t) = i=1 ( â 1+â ) vi ( ) ti v 1 i 1+â ( âi 1+â i ) vi ( 1 1+â i ) ti v i. The maximum likelihood estimator, under H 1,fora i is just â i = v i /u i. Thus, we rewrite our test statistic as: λ = k ( 1+vi /u i i=1 1+N v /N u ) ti ( ) Nv u vi i. N u v i In practice, we ll work with 2lnλ = 2 k i=1 [ ( ) 1+vi /u t i ln i 1+N v /N u + v i ln If u i = v i =0, the bin contributes zero. ( ) If v i =0, contributionis 2lnλ i = 2t i ln Nu N u +N. v ( ) If u =0, the contribution is 2lnλ = 2t ln Nv. 37 Frank Porter, Caltech Seminar, February 26, 2008 ( )] Nv u i. N u v i

38 Sample application of ln λ test Apply this test to the example, obtaining 2lnλ = k i=1 2lnλ i = Asymptotically, 2lnλ should be distributed as a χ 2 with N DOF = k 1, or N DOF =35. If valid, this gives a P -value of 0.31, tobecompared with a probability of 0.35 according to the estimated actual distribution. We obtain nearly the same answer as the naive application of the chisquare calculation with no bins combined, a result of nearly bin-by-bin equality of the two statistics. uorv or χ bin index 38 Frank Porter, Caltech Seminar, February 26, 2008 Value of 2lnλ i (circles) orχ 2 i

39 Comparison of ln λ and χ 2 To investigate when this might hold more generally, we compare the values of 2lnλ i and χ 2 i as a function of u i and v i,below. Thetwo statistics agree when u i = v i with increasing difference away from that point. This agreement holds even for low statistics. However, shouldn t conclude that the chi-square approximation may be used for low statistics fluctuations away from equal numbers lead to quite different results when we get into the tails at low statistics. Our example doesn t really sample these tails the only bin with a large difference is already high statistics Value of 2lnλ i or χ 2 i as a function of u i and v i bin contents. 2 This plot assumes N u = N v. 0 T v chisq, u=0-2lnl, u=0 chisq, u=5-2lnl, u=5 chisq, u=10-2lnl, u=10 39 Frank Porter, Caltech Seminar, February 26, 2008

40 Likelihood value test An often-used but controversial goodness-of-fit statistic is the value of the likelihood at its maximum value under the null hypothesis. It can be demonstrated that this statistic carries little or no information in some situations. However, in the limit of large statistics it is essentially the chi-square statistic, so there are known situations were it is a plausible statistic to use. We look at it here. Using the results in the previous section, the test statistic is: ln L = k i=1 [ ln ( ti v i ) + t i ln If either N u =0or N v =0,thenln L =0. N u + v N u + N i ln N ] v. v N u 40 Frank Porter, Caltech Seminar, February 26, 2008

41 Sample application of the ln L test We apply this test to the example. The sum over bins gives 90. According to our estimated distribution of this statistic under the null hypothesis, this gives a P -value of 0.29, similar to the χ 2 test result. The fact that it is similar may be expected from the fact that our example is reasonably well-approximated by the large statistics limit. Frequency ln Likelihood Estimated distribution of the ln L test statistic for the null hypothesis in the opening example. 41 Frank Porter, Caltech Seminar, February 26, 2008

42 We have not been exhaustive... There are many other possible tests that could be considered, for example, schemes that partition the χ 2 to select sensitivity to different characteristics D. J. Best, Nonparametric Comparison of Two Histograms, Biometrics 50 (1994) 538.]. 42 Frank Porter, Caltech Seminar, February 26, 2008

43 Distributions Under the Null Hypothesis Our example may be too easy... When the asymptotic distribution may not be good enough, we would like to know the probability distribution of our test statistic under the null hypothesis. Difficulty: our null hypothesis is not completely specified! The problem is that the distribution depends on the values of ν i = aμ i. Our null hypothesis only says ν i = aμ i, but says nothing about what μ i might be. [It also doesn t specify a, but we have already discussed that complication, which appears manageable (although in extreme situations one might need to check for dependence on a).] 43 Frank Porter, Caltech Seminar, February 26, 2008

44 Estimating the null hypothesis We turn again to the data to make an estimate for μ i,tobeused in estimating the distribution of our test statistics. A straightforward approach is to use the M.L. parameter estimators (under H 0 ): ˆμ i = 1 1+â (u i + v i ), ˆν i = â 1+â (u i + v i ). where â = N v /N u The data is repeatedly simulated using these values for the parameters of the sampling distribution. For each simulation, a value of the test statistic is obtained. The distribution so obtained is then an estimate of the distribution of the test statistic under H 0,andP-values may be computed from this. Variations in the estimates for ˆμ i and â may be used to check robustness of the probability estimates obtained in this way. 44 Frank Porter, Caltech Seminar, February 26, 2008

45 Trouble in River City... We have just described the approach that was used to compute the estimated probabilities for the opening example. The bin contents are reasonably large, and this approach works well enough for this case. Unfortunately, this approach does very poorly in the low-statistics realm. We consider a simple test case: Suppose our data is sampled from a flat distribution with a mean of 1 count in each of 100 bins. 45 Frank Porter, Caltech Seminar, February 26, 2008

46 Algorithm to check estimated null hypothesis We test how well our estimated null hypothesis works for any given test statistic, T, as follows: 1. Generate a pair of histograms according to the distribution above. 2. Compute T for this pair of histograms. 3. Given the pair of histograms, compute the estimated null hypothesis according to the spcified prescription above. 4. Generate many pairs of histograms according to the estimated null hypothesis in order to obtain an estimated distribution for T. 5. Using the estimated distribution for T, determine the estimated P - value for the value of T found in step Repeat steps 2-6 many times and make a histogram of the estimated P -values. This histogram should be uniform if the estimated P -values are good estimates. 46 Frank Porter, Caltech Seminar, February 26, 2008

47 Checking the estimated null distribution The next two slides show tests of the estimated null distribution for each the 7 test statistics. Shown are distributions of the simulated P -values. The data are generated according to H 0, consisting of 100 bin histograms for: A mean of 100 counts/bin (left column), or A mean of 1 count/bin (other 3 columns). The estimates of H 0 are: Weighted bin-by-bin average (left two columns), Each bin mean given by the average bin contents in each histogram (third column), Estimated with a Gaussian kernel estimator (right column) based on the contents of both histograms. The χ 2 is computed without combining bins. 47 Frank Porter, Caltech Seminar, February 26, 2008

48 F requency <counts/bin> = 100 <counts/bin> = 1 H0 is weighted bin-by-bin average H0 from overall average H0 from Gaussian kernel estimator χ F requency BDM F requency F requency Estimated probability KS CVM Estimated probability Estimated probability Estimated probability 48 Frank Porter, Caltech Seminar, February 26, 2008

49 F requency <counts/bin> = 100 <counts/bin> = 1 H0 is weighted bin-by-bin average H0 from overall average H0 from Gaussian kernel estimator AD F requency ln λ F requency lnl Estimated probability Estimated probability Estimated probability Estimated probability 49 Frank Porter, Caltech Seminar, February 26, 2008

50 Summary of tests of null distribution estimates Probability that the null hypothesis will be rejected with a cut at 1% on the estimated distribution. H 0 is estimated with the bin-by-bin algorithm in the first two columns, by the uniform histogram algorithm in the third column, and by Gaussian kernel estimation in the fourth column. Test statistic Probability (%) Probability (%) Probability (%) Probability (%) Binmean= H 0 estimate bin-by-bin bin-by-bin uniform kernel χ ± ± ± ± 0.28 BDM 0.91 ± ± ± ± 0.22 KS 1.12 ± ± ± ± 0.27 CVM 1.09 ± ± ± ± 0.28 AD 1.15 ± ± ± ± 0.29 lnλ 0.97 ± ± ± ± 0.34 lnl 0.97 ± ± ± ± Frank Porter, Caltech Seminar, February 26, 2008

51 Conclusions from tests of null distribution estimates In the small statistics case, if the null hypothesis were to be rejected at the estimated 0.01 probability, the bin-by-bin algortihm for estimating a i would actually reject H 0 : 19% ofthetimefortheχ 2 statistic, 16% of the time for the BDM statistic, 24% ofthetimefor the ln λ statistic, and 29% of the time for the L statistics, all unacceptably larger than the desired 1%. TheKS,CVM,andADstatistics are all consistent with the desired 1%. In the large statistics case, where sampling is from histograms with a mean of 100 counts in each bin, all test statistics display the desired flat distribution. The χ 2,lnλ, andlnl statistics perform essentially identically at high statistics, as expected, since in the normal approximation they are equivalent. 51 Frank Porter, Caltech Seminar, February 26, 2008

52 Problem appears for low statistics The issue for the bin-by-bin approach appears at low statistics. Intuition: Consider the likely scenario at low statistics that some bins will have zero counts in both histograms. In this case our algorithm for the estimated null hypothesis yields a zero mean for these bins. The simulation used to determine the probability distribution for the test statistic will always have zero counts in these bins, that is, there will always be agreement between the two histograms. Thus, the simulation will find that low values of the test statistic are more probable than it should. The same study with a mean of 100 counts per bin we find that the probability estimates are valid, at least this far into the tails. The AD, CVM, and KS tests are more robust under our estimates of H 0 than the others, as they tend to emphasize the largest differences and are not so sensitive to bins that always agree. For these statistics, our bin-by-bin procedure for estimating H 0 does well even for low statistics, although we caution again that we are not examining the far tails of the distribution. 52 Frank Porter, Caltech Seminar, February 26, 2008

53 Obtaining better null distribution estimates A simple approach to salvaging the situation in the low statistics regime involves relying on the often valid assumption that the underlying H 0 distribution is smooth. Then only need to estimate a few parameters to describe the smooth distribution, and effectively more statistics are available. Assuming a smooth background represented by a uniform distribution yields correct results. This is cheating a bit, since we perhaps aren t supposed to know that this is really what we are sampling from. It may be remarked that the ln L and, perhaps, to a much lesser extent the BDM statistic, do not give the desired 1% result, but now err on the conservative side. It may be possible to mitigate this with a different algorithm. We may expect the power of these statistics to suffer under the approach taken here. 53 Frank Porter, Caltech Seminar, February 26, 2008

54 More honest Try a kernel estimator Since we aren t supposed to know that our null distribution is uniform, we also try a kernel estimator for the null distribution, using the sum of the observed histograms as input. We try a Gaussian kernel, with a standard deviation of 2. In general, this works pretty well, though there is room for improvement. The bandwidth was chosen here to be rather small for technical reasons; a larger bandwidth would presumably improve the results. Density bin 54 Frank Porter, Caltech Seminar, February 26, 2008 Sample Gaussian kernel density estimate of the null hypothesis (for

55 Comparison of Power of Tests The power depends on what the alternative hypothesis is. We mostly investigate adding a Gaussian component on top of a uniform background distribution. Motivated by the scenario where one distribution appears to show some peaking structure, while the other does not. We also look at a different extreme, a rapidly varying alternative. 55 Frank Porter, Caltech Seminar, February 26, 2008

56 Gaussian alternaitve The Gaussian alternative data are generated as follows: The background (null distribution) has a mean of one event per histogram bin. The Gaussian has a mean of 50 and a standard deviation of 5, in units of bin number. We vary the amplitude of the Gaussian and count how often the null hypothesis is rejected at the 1% confidence level. The amplitude is measured in percent, for example a 25% Gaussian has a total amplitude corresponding to an average of 25% of the total counts in the histogram, including the (small) tails extending beyond the histogram boundaries. The Gaussian counts are added to the counts from the null distribution. 56 Frank Porter, Caltech Seminar, February 26, 2008

57 Mean counts/bin Bin number Counts/bin Bin number Left: The mean bin contents for a 25% Gaussian on a flat background of one count/bin (note the suppressed zero). Right: Example sampling from the 25% Gaussian (filled blue dots) and from the uniform background (open red squares). The distribution of P -values) is shown for seven different test statistics. Three different magnitudes of the Gaussian amplitude are displayed. The power of the tests to reject the null hypothesis at the 99% confidence level is summarized in Table and in Fig for several different alternative hypothesis amplitudes % Frank Porter, 25% Caltech Seminar, February 50% 26, χ

58 Distribution of estimated probability, under H0, that the test statistic is worse than that observed, for seven different test statistics. The data are generated according to a uniform distribution, consisting of 100 bin histograms with a mean of 1 count, for one histogram, and for the other histogram with a uniform distribution plus a Gaussian of strength 12.5% (left column), 25% (middle column), and 50% (right column). The χ 2 is computed without combining bins. Power (%) Gaussian Amplitude (%) Summary of power of seven different test statistics, for the alternative hypothesis with a Gaussian bump. Left: linear vertical scale; Right: logarithmic vertical scale. 58 Frank Porter, Caltech Seminar, February 26, 2008

59 In Table we take a look at the performance of our seven statistics for histograms with large bin contents. It is interesting that in this largestatistics case, for the χ 2 and similar tests, the power to reject a dip is greater than the power to reject a bump of the same area. This is presumably because the error estimates for the χ 2 are based on the square root of the observed counts, and hence give smaller errors for smaller bin contents. We also observe that the comparative strength of the KS, CVM, and AD tests versus the χ 2,BDM,lnλ, andlnl tests in the small statistics situation is largely reversed in the large statistics case. Estimates of power for seven different test statistics, as a function of H 1. The comparison histogram (H 0 ) is generated with all k = 100 bins Poisson of mean 100. The selection is at the 99% confidence level. 59 Frank Porter, Caltech Seminar, February 26, 2008

60 H0 5-5 Statistic % % % χ ± ± ± 0.7 BDM 0.97 ± ± ± 0.7 KS 1.03 ± ± ± 1.0 CVM 0.91 ± ± ± 1.2 AD 0.91 ± ± ± 1.2 ln λ 0.91 ± ± ± 0.7 ln L 0.97 ± ± ± Frank Porter, Caltech Seminar, February 26, 2008

61 Power study for a sawtooth alternative To get an idea of what happens for a radically different alternative to the null distribution, we consider sensitivity to sampling from the sawtooth distribution as shown in figure. This is to be compared once again to samplings from the uniform histogram. The results are tabulated. The percentage sawtooth here refers to the fraction of the null hypothesis mean. That is, a 100% sawtooth on a 1 count/bin background oscillates between a mean of 0 counts/bin and 2 counts/bin. The period of the sawtooth is always two bins. H0 or H1 mean Bin number Counts/bin Bin number 61 Frank Porter, Caltech Seminar, February 26, 2008

62 Left: The mean bin contents for a 50% sawtooth on a flat background of one count/bin (blue), compared with the flat background means (red). Right: Example sampling from the 50% sawtooth (filled blue dots) and from the uniform background (open red squares). In this example, the χ 2 and likelihood ratio tests are the clear winners, with BDM next. The KS, CVM, and AD tests reject the null hypothesis with the same probability as for sampling from a true null distribution. This very poor performance for these tests is readily understood, as these tests are all based on the cumulative distributions, which average out local oscillations. 62 Frank Porter, Caltech Seminar, February 26, 2008

63 Power summary for sawtooth alternative Estimates of power for seven different test statistics, for a sawtooth alternative distribution Statistic % % χ ± ± 1.2 BDM 1.9 ± ± 1.2 KS 0.85 ± ± 0.2 CVM 0.91 ± ± 0.2 AD 0.91 ± ± 0.3 ln λ 4.5 ± ± 1.2 ln L 0.30 ± ± Frank Porter, Caltech Seminar, February 26, 2008

64 Conclusions These studies have demonstrated some important lessons in goodnessof-fit testing: 1. There is no single best test for all applications. Statements such as test X is better than test Y are empty without giving more context. For example, the Anderson-Darling test is often very powerful in testing normality of data against alternatives with non-normal tails (such as the Cauchy distribution)*. However, we have seen that it is not always especially powerful in other situations. The more we know about what we wish to test for, the more reliably we can choose a powerful test. Each of the tests investigated here may be reasonable to use, depending on the circumstance. Even the controversial L test works as well as the others sometimes. However, there is no known * M. A. Stephens, EDF Statistics for Goodness of Fit and Some Comparisons, Jour. Amer. Stat. Assoc. 69 (1974) Frank Porter, Caltech Seminar, February 26, 2008

65 situation where it actually performs better than all of the others, and indeed the situations where it is observed to perform as well are here limited to those where it is equivalent to another test. Thus, there seems to be no reason to advocate including it in the toolkit. 2. Computing probabilities via simulations is a very useful technique. However, it must be done with care. The issue of tests with an incompletely specified null hypothesis is particularly insidious. Simply generating a distribution according to some assumed null distribution can lead to badly wrong results. Where this could occur, it is important to verify the validity of the procedure. Note that we have only looked into the tails to the 1% level. The validity must be checked to whatever level of probability is needed for the results. Thus, we cannot blindly assume the results quoted here at the 1% level will still be true at, say, the 0.1% level. We have concentrated on the specific question of comparing two his- 65 Frank Porter, Caltech Seminar, February 26, 2008

66 tograms. However, the general considerations apply more generally, to testing whether two datasets are consistent with being drawn from the same distribution, and to testing whether a dataset is consistent with a predicted distribution. The KS, CVM, AD, ln L, andl tests may all be constructed for these other situations (as well as the χ 2 and BDM, if we bin the data). 66 Frank Porter, Caltech Seminar, February 26, 2008