cross section limit (95% CL) N = B = 10 N = B = 3 N = B = 0 number of experiments

Transcription

1 L3 note 2633 Calculating Upper Limits with Poisson Statistics Thomas Hebbeker Humboldt University Berlin February 23, 2001 Often upper limits on cross sections and similar variables are calculated assuming Poisson probabilities and using Bayesian statistics. I provide a straightforward generalization, which takes care of systematic uncertainties in the parameters `eciency' and background, and which permits to combine several limits obtained in independent measurements.

2 Introduction Frequently the following situation arises in physics: An experiment looks for possible signals arising from a hypothetical `new' process. We assume that counting events yields a small number N, statistically compatible with the average number of background events B expected from known processes. Thus, there is no hint for anything new. To quantify this, the experimental result needs to be translated into an upper limit on the `cross section' (or ux or cross section branching ratio or... ) for the hypothetical new process (neglecting interference eects). The expected average number of new events S is related to through the `eciency' (or acceptance or eciency luminosity or... ) Thus, the total number of expected events is on average S = (1) T = + B (2) There is no uniquely accepted method of how to calculate an upper limit on for a given condence level, denoted CL (e.g. 95% ), see for example the dierent procedures suggested in the `Review of Particle Properties' in the last years [1{3]. Here I refer to the Bayesian approach assuming a at prior distribution. The limit CL can be calculated from [4]: If one assumes Poisson probability CL = R CL 0 p(n; + B) d R 1 0 p(n; + B) d (3) N p(n; ) = e N! (4) one obtains in the special case of only one experiment with no systematic errors [5]: P CL CL = e N n=0( CL + B) n =n! P N n=0 Bn =n! (5) This equation can be solved for CL numerically. This method does have disadvantages: a) The limit is too conservative in the following sense: Generating events according to a Poisson distribution using CL yields for B > 0 event numbers n N with a frequency less than 1 CL [6]. b) There is no smooth transition between one-sided and two-sided limits [7]. Advantages are: The method is simple (does not need Monte Carlo studies) and widely used. Whatever one chooses to do, in any case one must state clearly which methods/assumptions have been applied! 2

3 Several experiments How can one generalize formula (3)? First, let us assume there are K independent experiments or data taking periods, all measuring the same cross section. The total probability is the product of the individual probabilities per experiment, thus we must calculate CL = R CL Q K 0 k=1 p(n k; k + B k ) d R 1 Q K k=1 p(n k; k + B k ) d 0 (6) The equation can be solved for CL numerically. Note that not only the number of measured events, but also eciencies and background expectations depend on the experiment index k. All correlations are assumed to vanish. Example: Take K = 2 experiments with the same 'eciency' = 0:1 and the same background expectation B k = 3. If one measures N 1 = 5, this leads to a limit 95% 1 = 76:6 for experiment 1 alone. Let's assume the second one measures N 2 = 1, this yields a limit of 95% 2 = 36:4. The combined limit calculated with (6) is 95% = 33:6. If one merges the two data samples (imagine two equally long data taking periods!), the combined eciency is = 0:2 and the expected total background B = 6. N = 6 events are observed, leading to the limit of 95% = 33:6 when applying the simpler formula (5). This is the same result as obtained through (6)! However, in general the combined limit can not be calculated by simply adding up eciencies, backgrounds and events and applying the one-experiment-formula (5)! Only in case of `equivalent experiments' (in particular if and B are the same) the two approaches give identical answers. The correct formula (6) takes care of properly `weighting' the experiments according to their `signicance'! Figure 1 illustrates the improvement of the limit when combining several independent experiments. For pno data events seen, the limit improves like 1=K, for large values of N the curves approach a 1= K dependence, as expected. Background uncertainty For the moment we go back to the special case K = 1. What is the eect of a `systematic' (may be of statistical origin!) error on the background prediction? We assume it to be of Gaussian nature, with a sigma of B. Now we have to replace in (3) the Poisson distribution by the following convolution: P (N; ; ; B; B) = with the Poisson probability p and the Gaussian probability density Z 1 p(n; + B + b) g(b; B) db (7) 1 g(x; ) = 1 p 2 exp( x 2 =2 2 ) (8) 3

4 1 cross section limit (95% CL) 10-1 N = B = 0 N = B = 10 N = B = number of experiments Figure 1: Limits obtained combining K experiments. The lower curve, labeled N = B = 0 shows the limit obtained if all experiments are independent and all measure N = 0 when expecting B = 0. Similar remarks apply to the other two curves. Note: The limits shown are normalized to the case K = 1. Figure 2 illustrates the eect of the background uncertainty for relative uncertainties between 0 and 100%. For N = B = 0 there is no eect at all, since in this case B 0. Note that for N = 0 the cross section limit (5) does not depend on B! For larger values of N, B the inuence remains modest. This can be explained by the fact, that the background smearing in (7) is symmetric around the central value B, and the linear dependence of the cross section limit on the background: CL 1 (N B). Eciency uncertainty Again, we consider the case K = 1 and set B = 0. What is the eect of a `systematic' error on the `eciency'? We assume it to be Gaussian, with a sigma of. Now the integrand in formula (3) needs to be generalized to P (N; ; ; B; ) = Z 1 p(n; ( + e) + B) g(e; ) de (9) 1 As gure (3) shows, an error on can change the limit a lot. The reason: there is a certain probability for being very small, thus blowing up CL N B. Apparently, this degradation of the limit depends very little on N and B. 4

5 1.7 cross section limit (95% CL) N = B = 10 N = B = 3 1 N = B = relative error background Figure 2: Limits calculated for varying the relative background uncertainty B=B between 0 and 100%. Note: The limits shown are normalized to the case B = 0. Legend: see gure cross section limit (95% CL) N = B = 3 N = B = 10 1 N = B = relative error efficiency Figure 3: Limits calculated for varying the relative uncertainty = between 0 and 30%. Note: The limits shown are normalized to the case = 0. Legend: see gure 1. The three curves nearly coincide. 5

6 General case If both B > 0 and > 0 and also K > 1 the above formulae must be `merged'. Note that the CPU time for the limit calculation increases strongly for each additional convolution integral... Code Code in FORTRAN and C++ for the limit calculation is available through Acknowledgments I thank John Conway, Glen Cowan, Aura Rosca and Serge Sushkov for interesting discussions. References [1] Particle Data Group, R.M. Barnett et al., Phys. Rev. D 54 (1996) 1. [2] Particle Data Group, C. Caso et al, Eur. Phys. J. C 3 (1998) 1. [3] Particle Data Group, D.E. Groom et al, Eur. Phys. J. C 15 (2000) 1. [4] G. Cowan, `Statistical Data Analysis', Oxford University Press, [5] O. Helene, Nucl. Instr. Meth. 212 (1983) 319. [6] T. Hebbeker, `A Remark on Poisson Statistics', L3 note 1995, [7] G.J. Feldman and R.D. Cousins, Phys. Rev. D 57 (1998)