THE SIGNAL ESTIMATOR LIMIT SETTING METHOD

Transcription

1 ' THE SIGNAL ESTIMATOR LIMIT SETTING METHOD Shan Jin, Peter McNamara Department of Physics, University of Wisconsin Madison, Madison, WI Abstract A new method of backround subtraction is presented which uses the concept of a sinal estimator to construct a confidence level which is always conservative and which is never better than. The new method yields stroner exclusions than the Bayesian method with a flat prior distribution. 1. INTRODUCTION In any search, the presence of standard model backround will derade the sensitivity of the analysis because it is impossible to unambiuously seperate events oriinatin from the sinal process from the expected backround events. Althouh it is possible, when settin a limit on a sinal hypothesis, to assume that all observed events come from the sinal, a search analyzed in this way will only be able to exclude sinals which are sinificantly larer than the backround expectation of the analysis. Backround subtraction is a method of incorporatin knowlede of the backround expectation into the interpretation of search results in order to reduce the impact of Standard Model processes on the sensitivity of the search. The end result of an unsuccessful search is an exclusion confidence for a iven sinal hypothesis based on the experimental observation. This confidence level is associated with a sinal and backround expectation and an observation, and is required to be conservative. A conservative confidence level is one in which the False Exclusion rate, or probability that an experiment with sinal will be excluded, must be less than or equal to, where is called the confidence coefficient. The classical frequentist confidence level is defined such that this probability is equal to. In the presence of a sufficiently lare downward fluctuation in the backround observation, however, the classical confidence level can exclude arbitrarily small sinals. Specifically, for sufficiently lare backround expectations, it is possible for an observation to exclude the backround hypothesis, in which case, the classical confidence level will also exclude a sinal to which the search is completely insensitive. In order to prevent this kind of exclusion, and because there is no ambiuity when zero events are observed, it is required that all methods must default to a confidence level in order to be deontoloically correct. When no events are observed, one should not perform any backround subtraction, and, the probability of observin zero sinal events should be just. Further, any observation of one or more candidate events should yield a larer value of. This correctness requirement can be easily verified for any method, and any method which is not deontoloically correct should be considered too optimistic. 2. BAYESIAN BACKGROUND SUBTRACTION METHOD A common method of backround subtraction[1], based on computin a Bayesian upper limit on the size of an observed sinal iven a flat prior distribution, calculates the confidence level in terms of the probabilities that a random repetition of the experiment with the same expectations would yield a lower number of candidates than the current observation, which observes. This method computes the backround subtracted confidence to be! #" $%!$ &" Correspondin address: CERNEP Division, 1211 Geneva 23, Switzerland. Tel: (41 22) ; fax: (41 22) ; Shan.Jin@cern.ch, Peter.McNamara@cern.ch (1) 103

2 where! #" is the probability that an experiment with sinal expectation ( and backround expectation ) yields an equal or lower number of candidates than the current observation, and! #" is the probability that an experiment with backround expectation ) yields an equal or lower number of candidates than the current observation. When is zero, this method reduces to, demonstratin that it is deontoloically correct. Further, the probability of observin events or fewer is equal to %!$ #", and the confidence coefficient for that observation is strictly larer than the probability of observin the result, so this method is conservative. The method can be extended[2] to incorporate discriminatin variables such as the reconstructed mass or neural network output values by constructin a test-statistic * for the experiment which is some function of those discriminatin variables, and constructin the confidence level as the ratio of probabilities % * + %,*- #" *.* "! where *,*0 #" is the probability that an independent experiment with sinal expectation (, backround expectation ), and some iven distributions of discriminatin variables yields a value of * less than or equal to *- seen in the current experiment, and *0,*0 #" is the probability that an independent experiment with backround expectation ) and some iven distributions of discriminatin variables yields a value of * less than *- seen in the current experiment. If the test-statistic is the number of observed events, this method reduces to the method described above, thouh the test-statistic can be constructed as a likelihood ratio or in some other appropriate way such that larer values of * are more consistent with the observation of a sinal than lower values. For an observation of zero events the probabilities 1 * %,*- #" and *-%,*- &" are simply the Poisson probabilities of observin zero events in the two cases. Because a correctly defined test-statistic has its smallest value when and only when there are no events observed, the confidence level for the eneralized version of this method then reduces to the same value as the number countin method when there are no events observed, and it is deontoloically correct. Similarly, the probability of observin a more sinal-like test-statistic value is equal to *,*0 #", and as *-%,*- &", is always reater than or equal to this value, so the method is conservative. 3. SIGNAL ESTIMATOR METHOD Thouh the Bayesian method described in Section 2 satisfies the criteria set out in Section 1, it is not the only backround subtraction method which is both conservative and deontoloically correct. The Sinal Estimator method satisfies both of these criteria usin * %,*- &" and a boundary condition to calculate the confidence level. The boundary condition imposes the correctness requirement on the confidence level, while also makin the result conservative. We wish to determine if a iven sinal hypothesis ( is excluded. If we could know the observed test-statistic based on events truly from sinal only, which we refer to as the sinal estimator * " 2, the confidence level would be riorously defined as * * " #" (3) where * * " #" is the probability that an experiment with sinal expectation ( yields a value of the sinal estimator less than or equal to * ". Unfortunately, we cannot directly know * " 2 from an experiment as it is not possible to unambiuously determine if an event comes from sinal or backround. We can only directly know a test-statistic value based on the total observation *- * + " 104 (2) (4)

3 * K R Althouh it is not possible to know * " directly, it is still possible to produce an estimate of it, with which we can calculate Eq. 3. This is most straihtforward for test-statistics of the form * + * 76 *- (5) where 6 represents a sum or product. For example, in simple event countin, 98 In this case, we can use a Monte Carlo simulation of the backround expectation to remove the backround contribution in the observed test-statistic * " 2, i.e., to estimate * ", and to calculate Eq. 3. In each Monte Carlo experiment, the estimate of * " is defined as * " * 9: * <;>= * 7: * %? * "A@7BDC * "E@7BFC ;>= *- : *-% * "A@7BDC where : represents difference or division, and * "E@7BFC is the minimum possible value of the sinal estimator, which corresponds to the physical boundary (zero sinal events). The confidence level can be computed with Monte Carlo methods in the followin way for an observed test-statistic *-. First, enerate a set a Monte Carlo experiments with test-statistic values distributed as for experiments with the expected backround but no sinal to determine a distribution of possible sinal estimator values for the observation accordin to Eq. 8. Next, usin a sample of Monte Carlo with test-statistics distributed as for experiments with sinal only, and for each possible sinal estimator value, calculate * #G * " *!HI#J%K * 2 9: * G * "A@7BDCML0" The value of * #G * " averaed over all of the sinal estimator values determined with backround Monte Carlo forms an estimate of * * " &", or 4 * * " #"ON *- G *- " (8) The Monte Carlo procedure described above is very slow, and without eneralization, it can only be used for the class of test-statistics which satisfy Eq. 5. The method can be eneralized into a much simpler mathematical format which can be used for any kind of test-statistic. The eneralization can best be illustrated with an example. In the case of simple event countin, the boundary condition for the sinal estimator can be understood intuitively. For an observation of $ events, the confidence level is computed by allowin the backround to vary freely, and accordin to Eq. 8, the sinal estimator will be " $ ;QP R ;QP?,R Usin Eq. 10, one can easily compute the confidence coefficient to be (6) (7) (9) K $ R "7S! #" 8 "7S 1!$ " $ H "7S 1!$ H " 8 TU 8 #"7S R "VL UT 8? $ #"7S R " 1 + %!2 #" 8! #"VLWS 105 (10)

4 d a b This probability reduces to YXZ \[ 3 8 " when one observes no candidates, so it is deontoloically correct, and because the confidence level is always strictly reater than 1 %!$ #", it is conservative. In order to compare the performances of this method with the Bayesian method, the confidence levels for a simple experiment are analyzed in Fi. 1. For this example, the analysis is assumed to expect three events from a possible sinal, and three events from Standard Model backround processes. For both methods, when zero events are observed, the confidence level reduces to while for observations of more events, the sinal estimator method yields a lower confidence coefficient, and thus a better exclusion confidence level. For lare numbers of events,! " approaches one, meanin that both methods approach the classical confidence level and ive very similar results. Confidence Coefficient c Bayesian Method SE Method 10-2 ] 0 ^ 2 _ 4 ` N obs Fi. 1: A comparison of Sinal Estimator method performance to the Bayesian method performance. For an experiment with three sinal and three backround events expected, the confidence levels are shown for different numbers of observed events. The Sinal Estimator method ives either an equal or better confidence level for all possible observations. This method can then be eneralized, as the method described in Section 2 was eneralized, to include discriminatin variables. The natural eneralization takes the form *,*0 #" 8 K *0,*- #"VLWS e For an observation of zero events, the eneralized method continues to ive a confidence level, and the confidence level computed with this method is always conservative, with strictly reater than *,*02 #". Generatin Monte Carlo experiments based on a simplified His analysis, one can compare the performances of the eneralized Bayesian method described in Section 2 and the Sinal Estimator method. For the comparison it is assumed that there are three events expected from backround processes, with mass distributed uniformly between 70 and 90 GeV 2f, and that the sinal process would yield three events, with mass distributed accordin to a sinle Gaussian whose width is 2.5 GeV f centered at 80 GeV 2f. Usin the test-statistic described in ref. [3], Fi. 2 shows the relative improvement in (11) 106

5 confidence level for this experiment. The Sinal Estimator method is seen to never a worse confidence level than the eneralized Bayesian method. For an observation of zero candidates, and for very sinallike observations (as *,* " approaches one) the methods convere. In the reion in between these extremes, the Sinal Estimator method ives confidence levels up to 20% better than the eneralized Bayesian method while remainin conservative. (C Bayes -C SE )C Bayes h CBayes Fi. 2: A comparison of Sinal Estimator method performance to the eneralized Bayesian method performance when discriminatin variables are used. The Monte Carlo experiments assume three sinal and three backround events are expected, and the sinle discriminatin variable has a Gaussian distribution with width 2.5 GeViUj for sinal, flat for backround over a rane of 20 GeViUj. The relative improvement in confidence level usin the Sinal Estimator method is shown for different confidence level values. 4. CONCLUSION More than one method of calculatin backround subtraction confidence levels which is conservative and deontoloically correct exist. The Sinal Estimator method proposed here yields less conservative limits than the Bayesian method, which should result in an increase in search senstitivity, ivin better limits in unsuccessful searches. References [1] O. Helene, Nucl. Instr. and Meth. 212 (1983) 319. [2] LEP His workin roup, CERNLEPC (1997). [3] J.-F. Grivaz and F. Le Diberder, NIM A333 (1993)

6 Discussion after talk of Shan Jin. Chairman: Roer Barlow. H. Prosper S. Jin I didn t quite catch your definition of better, could you just explain that aain please? Better means that under conservation of coverae, you ve ot a smaller or larer upper limit or better sensitivity of the limit. H. Prosper I ll have to think about that. 108