CONFIDENCE LIMITS AND THEIR ROBUSTNESS

Transcription

1 CONFIDENCE LIMITS AND THEIR ROBUSTNESS Rajendran Raja Fermi Naional Acceleraor laboraory Baavia, IL Absrac Confidence limis are common place in physics analysis. Grea care mus be aken in heir calculaion and use, especially in cases of limied saisics when ofen one-sided limis are quoed. In order o esimae he sabiliy of he confidence levels o addiion of more daa and/or change of cus, we argue ha he variance of heir sampling disribuions be calculaed in addiion o he limi iself. The square roo of he variance of heir sampling disribuion can be hough of as a saisical error on he limi. We hus inroduce he concep of saisical errors of confidence limis and argue ha no only should limis be calculaed bu also heir errors in order o represen he resuls of he analysis o he fulles. We show ha comparison of wo differen limis from wo differen experimens becomes easier when heir errors are also quoed. Use of errors of confidence limis will lead o abaemen of he debae on which mehod is bes suied o calculae confidence limis. 1 Inroducion Confidence limis are used o express he resuls of experimens ha are no ye sensiive o discover he objec of heir searches. In such cases, ofen a one-sided limi is used o delimi he quaniy of ineres. Limis from differen experimens are compared and aemps are made o combine hem. These limis can flucuae up or down wih he addiion of more daa or he changing of he analysis parameers. A measure of he robusness of he limis is given by he widh of he sampling disribuion of hese limis, where he sampling disribuion is obained over an ensemble of similar experimens simulaed by Mone Carlo. The sandard deviaion of he sampling disribuion of such limis can be hough of as an error on he limi. We inroduce he concep of error of confidence limis by a simple Gaussian example. Consider a sample of evens, where, characerised by he variable disribued as a uni Gaussian, wih a mean value and sandard deviaion. Then he average value of he evens will be disribued as a Gaussian of mean value zero and sandard error /. The unbiased esimae of, he variance of he disribuion is given by where, (1) Figure 1 shows he disribuion of our sample of 10 evens for a large number of samples. The expeced value is zero and is sandard deviaion is 0.32 which is consisen wih he heoreical value of! = Figure 2 shows a hisogram of deduced from a sample of 10 evens for a large number of such samples. The average value of is " 1.0, showing ha is an unbiased esimaor of. The imporan poin o noe is ha also has a variance and ha is sandard deviaion is $#&%('. This is as expeced from heory where he error on he sandard deviaion of a Gaussian sample [1] is ")! %* = Having go he value of and for our sample, one can proceed o work ou confidence limis for our observaion. The wo-sided 68+,- of and we would wrie he observaion of CL limis for our observaion of will be given by he sandard error from our sample as /.0,-, 1.23! $# 98(8. $#;:< <8 (2) 34

2 Fig. 1: The disribuion of he sample average > = over a large sample of evens. Fig. 2: Unbiased esimae? of he sandard deviaion of of he Gaussian disribuion deduced from a sample of ACB DFE evens. The average value of? is G 1.0 and is sandard deviaion is

3 where he numbers correspond o our sample of 10 evens. Noe ha he sandard error,h = derived from our sample of 10 evens is quie differen from he heoreical value of 0.32, bu his is merely due o saisical flucuaion. One can also work ou he wo-sided 90% CL limis for our observaion of which would correspond o. 3#&I*:,- and quoe he 90% CL limis as 7 $# 98(8. $#&I(I(J, which is he value observed for our sample of 10 evens. Figure 3 shows he disribuion of he 90% CL wo-sided errors on he sample average, over a large number of samples. The mean value of he disribuion is which is close o he heoreical value of 1.64,- = Noe ha he sandard deviaion of he 90% CL errors in Figure 3 is We can also calculae he sandard deviaion of he 90% CL error from our sample as 1.64,- %* and his is ploed in figure 4. The mean value of he sandard deviaion of he 90% CL error in figure 4 is 0.113, in line wih he heoreical value of When he mean value is of ineres, we quoe he mean Fig. 3: The disribuion of he calculaed wo-sided 90% CL errors of he mean value of he sample. value and he sandard error on he mean value as in equaion 2. This enables us o gauge he flucuaions in he mean value from sample o sample. When he confidence limi is of ineres, we propose ha we quoe he confidence limi along wih is sandard error. This would enable us o gauge he significance and sabiliy of he confidence level. In our example we would wrie his as 3#&I*:,-. LKNMPO QO6 1R 3#&I*:,H.0 KNMTSVU J3 +6WPX (3) where is he expecaion value of and he sandard error LKNM on he 90% CL limi would be given by LKNMY"Z,- 4 RZ 3#&I*:V $ %* (4) In our sample of 10 evens, his would lead o 7 $#&8([<\. $#;:]'*: O QO $#;:]8$. $#;:]'*: SVU J3 +^WPX (5) 36

4 Fig. 4: The disribuion of he calculaed error on he wo-sided 90% CL error of he mean value of he sample. Noe ha he error on he lower and upper 90% CL limis are correlaed by he error on which hey have in common. Half he difference beween he lower and uper 90 % CL limis is 3#&I*:,- and is error is 1.64,- %*. These wo errors added in quadraure yield he formula in equaion 4. The error in he 90% CL limi indicaes o he reader he sabiliy of he limi and he saisical significance of he resul. Very ofen, we are no ineresed in he mean value of our observaions bu are more ineresed in he confidence limis, due o he low saisics of he observaion. We may only be ineresed in an upper (one-sided) bound. So we would quoe a 95% CL upper bound on as QO $#;:]8$. $#;:]'*: S]U J([ +6W_X (6) A second sample of 10 evens from he same disribuion may yield a resul QO $#&'([*:. $#&'('([ S]U J([ +6W_X (7) bu we do no fall ino he rap of declaring he second resul a beer limi han he firs, because boh he limis are he same wihin errors. If we did no quoe he errors on he limis, we would be emped o declare he second limi superior o he firs. Similarly, as analyses proceed in discovery searches, evens can go in and ou of samples, as cus are refined and more daa is accumulaed. Appearance of a single even in a sample can change he confidence limi drasically, as was he case in he search for he op quark. These changes can be undersood as flucuaions of he confidence limi wihin errors, if we were o quoe no only he confidence limi bu also is error. 2 Reconciliaion wih he Neyman definiion of Confidence limis The consrucion of confidence levels as wrien down by Neyman [2] may be undersood wihin he conex of our curren example as follows. Using our firs sample of 10 evens drawn from a uni Gaussian, 37

5 we calculae a mean value `7 $# 98(8. Le us assume, for he sake of argumen, ha we know he variance of he mean value o be 3#a ]. In his case, we can consruc he Neyman confidence level for, he expecaion value of, as illusraed in Fig. 5. The parameer is ploed on he ordinae and is ploed on he abscissa. For each value of, he 90% CL limis of are delineaed by horizonal lines ha are delimied by he curves and, assuming is disribued abou wih variance 3#a! ]. If he rue value of is bm, hen bm Oc doc bm wih 90% probabiliy. If we now measure a value of ^7 $# 98(8, hen we can consruc he inerval AB which will conain he rue value of M if and only if M O6 eo6 M. In oher words he inerval AB has a probabiliy of 90% (also called coverage ) of conaining he rue value bm. The inerval AB is hus defined o be he 90% CL inerval of. If we were however o repea our measuremen of by creaing oher samples of 10 evens each, we would ge differen lines AB, each of which would have a 90% chance of conaining he rue value bm. Mos of he ime, one is ineresed in a cenral value of and an inerval such as AB o denoe he saisical errors (robusness) of he measuremen of. However, in experimens wih poor saisics, he cenral value is ofen no of ineres and he one-sided limi (eiher poin A or B) is ofen quoed. A his sage, he poins A or B become poin measuremens in heir own righ, and i is informaive o quoe heir saisical errors in order o evaluae heir robusness. This is illusraed furher in Fig. 6, where we now no longer assume we know he variance of. This is compued from he daa and will flucuae from sample o sample. These so-called nuisance variables are inegraed over o yield a final confidence limi in usual pracice, which would be appropriae if one were ineresed in he cenral value of. If however, one is ineresed in he one-sided limi B, i would be appropriae o use hem o esimae he robusness of he poin B due o saisical flucuaions. We use he error bands shown for and,- in he figure o compue he sampling error band on he poin B. Fig. 5: The Neyman consrucion of he confidence level for our example 38

6 O O > >3h for our example. The error band due >(h Fig. 6: The Neyman consrucion modified o illusrae flucuaions in = and band due he error >(h are shown. These are added in quadraure o produce he sampling error band of poin B. 3 An Illusraive example We can illusrae he need for confidence limis errors using he following example. In 1995, he DØ collaboraion published limis on he op quark mass and cross secion [3]. Figure 7 shows [3] he 95% CL upper limi on op quark producion as a funcion of op quark mass using 13.5 pbi of daa. The confidence limi curve is used o derive a lower limi of 128 GeV/c for he op quark mass a 95% CL. In he same paper, anoher figure, reproduced here as Figure 8 shows he op quark producion cross secion as a funcion of he op quark mass. This curve has a 1 error band around i. Bu he op quark producion cross secion may be hough of as he 50% CL upper/lower bound on he cross secion. Surely, if he 50% CL limi has an error band around i, he 95% CL limi should also have is own error band. In wha follows, we show how o calculae errors in confidence levels in general and use he mehod o calculae he error in he 95% CL curve shown in Figure 7. 4 A general algorihm o calculae errors in Confidence Limis Mos experimens have elaborae algorihms o calculae confidence limis for heir resuls. Such algorihms will include deailed calculaions and paramerizaions of efficiencies and accepances. In addiion, hey will have several oher inpu parameers such as he number of evens observed, oal inegraed luminosiy and he error on he luminosiy. Le us denoe he inpu parameers as S kjml ) j. The oupu of such a program will be he confidence limis Won j5p q jsr. Figure 9 illusraes his general case. Then, for small changes in he inpu parameers, he following equaions hold. W n WTuwv W n W n S S (8) W n WTu S S S*xyv (9) S3x 39

7 Fig. 7: The 95% confidence level [3] { z as a funcion of op quark mass. Also shown are cenral (doed line) and low (dashed line) heoreical cross secion curves [4]. Cross Secion (pb) DØ Theory Top Mass (GeV/c 2 ) Fig. 8: Measured } = } producion cross secion (solid line, shaded band = one sandard deviaion error) as a funcion of op mass [3]. Also shown are cenral (doed line), high and low (dashed lines) heoreical cross secion curves [4]. where he repeaed indices l~j are mean o be summed over and he symbols O v indicaes he average over he enclosed quaniies. The quaniy on he lef hand side of he equaion is he error marix in he confidence limis W n, denoed ƒ ƒ. The above equaion can be re-wrien in marix form as ƒ ƒ ˆ YŠsŠ where YŠ Š is he error marix of he inpu parameers S gjœl ^ j and n! Ž ƒ! Ž Šs. The error marix YŠ Š should be known o he experimener, yielding he required error marix ƒ ƒ. is he ransfer marix, such ha can be deermined numerically by varying he inpu parameers o he limis algorihm. (10) 40

8 i Fig. 9: Schemaic black box represenaion of a general confidence limi calculaing algorihm, ha has inpu parameers ( F a and oupus a confidence level in a single variable. 4.1 An Example Le us consider he calculaion of W, he 95% CL upper limi o he op quark cross secion as published in reference [3]. The oupu of he limis algorihm is W. The inpu parameers can be aken as hree, namely S, he oal number of op quark evens observed, S, he luminosiy š efficiency š branching raio of he channels under consideraion, summed over he channels and S!, he error in he luminosiy. We have used a single parameer S summed over he channels o simplify he calculaion. In principle, all channels may be varied independenly, bu since hey are uncorrelaed, and he dominan error is due o he common luminosiy facor, he above simplificaion will resul. We use his example for illusraive purposes o show how such a calculaion may proceed. The error marix of he parameers YŠ Š is a 3š 3 diagonal marix, since he parameers are uncorrelaed. The variance of S is he number of evens observed, he variance of S is calculaed using he error in luminosiy, and he variance of S is calculaed assuming ha here is a 50% uncerainy in he error in he luminosiy. The ransfer marix is calculaed by numerical differeniaion. Figure 10 shows he conribuion o ƒ, he error in he 95% CL upper limi o he cross secion, due o he hree parameers S, S and SV as a funcion of he op quark mass. The overall error ƒ, obained by adding he componen errors in quadraure, is also shown as a funcion of he op quark mass. I can be seen ha he conribuion due o uncerainies in S, is negligible. So we are no sensiive o errors in our guess of 50% uncerainy o he error in he luminosiy. The overall error is dominaed by he flucuaion in he oal number of evens. This example hus graphically illusraes why confidence limis flucuae up and down as evens fall in and ou of he seleced sample as he analysis proceeds and more daa is accumulaed. The 95% CL upper limi o he cross secion is merely flucuaing wihin is error as all saisical quaniies do. When we are ineresed in a confidence limi, i hus behooves us o compue no only ha limi bu also is error. We may superimpose hese errors on Figure ž 7 yielding Figure 11. The 95% CL lower limi o he op quark mass can hen be quoed as 128œ Ÿ GeV/c, he error bars indicaing he range of flucuaion for he mass limi. This implies ha if one were o repea he DØ experimen numerous imes wih an inegraed luminosiy of 13.5 pbi flucuaing wihin is errors, one would expec o ge a op quark lower mass limi ha flucuaes wihin he errors quoed. 5 Combining limis Combining limis from wo differen experimens is difficul a bes. We remark here ha in simple Gaussian cases, quoing he limi and is error provides us wih enough informaion o make a combined resul, as may be seen by examining equaions 3 and 4. Using he value of he limi and is error, we may deduce and,-, if he number of evens in he sample is known. Having he mean and is variance 41

9 Fig. 10: The componens he error in he 95% CL op quark cross secion upper limi, due o uncerainies in (a) error in luminosiy (b) Luminosiy efficiency branching raio (c) The overall number of evens observed as a funcion of op quark mass. (d) shows he overall in each case, we can obain he sum of squares for each measuremen. These can be added leading o a new variance for he combined daa. The combined mean of he wo disribuions can be found as usual by he weighed average of he wo means, he weighs being he inverse variances. I mus be emphasized ha he combined limi is no simply he weighed average of he wo limis as in he case of he means. One can furher ask if he wo limis are consisen wih each oher, if he errors on he limis are quoed, as shown below. 6 Comparing Limis from wo differen algorihms When wo differen algorihms are used on he same daa, wo differen limis will resul ha are correlaed. The correlaions will be due o he common inpu ino he wo algorihms. We can hink of he, he wo black box in Fig. 9 as consising of wo differen algorihms producing as oupu W and W confidence levels in quesion, using he same common inpu S Fj5l j. We can hen use equaion 10 o work ou ƒ ƒ, he error marix of he wo confidence level algorihms and use his marix o decide wheher he wo confidence levels are significanly differen from each oher as per, 7 Conclusions SV žw W ª SV «žw R S] «žw %<Œ %3 H žw j W %<T N R %(%_ % We have moivaed he concep of saisical error for a confidence limi, as he sandard deviaion of he sampling disribuion of such limis over an ensemble of similar experimens. In cases of limied saisics, our esimaes of he confidence limis can flucuae significanly. Comparing confidence limis becomes more meaningful when hese errors are quoed. Differen mehods exis (e.g Bayesian, Frequen- (11) 42

10 Fig. 11: The DØ 95% CL upper limi o he op quark cross secion [3] wih is accompanying error band, as calculaed by he mehod in he ex. is) for calculaing hese limis. The differences beween limis compued in he same experimen using differen mehods will lose heir significance if he limis are shown o be he same wihin heir sampling error. Ofen in analyses wih limied saisics, he appearance of a new even can make significan differences o he limi calculaion. An error analysis of he limi will show ha he limi is exhibiing saisical flucuaion as i is eniled o. We propose ha experimeners publish confidence limis o heir daa accompanied by he error on he limis. 8 Acknowledgemens The auhor wishes o hank Roger Barlow, Bob Cousins, Louis Lyons, Harrison Prosper, Byron Roe, and Tom Trippe for helpful commens. References [1] C.E. Weaherburn, A firs course in Mahemaical Saisics, Cambridge Universiy press, Page 137. [2] J.Neyman, Phil. Trans. Royal Soc. London, Series A, 236, 333 (1937), reprined in A Selecion of Early Saisical Papers on J. Neyman (Universiy of California Press, Berkeley, 1967). [3] Top quark search wih he DØ daa sample, Phys.Rev.D52:4877(1995) [4] E.Laenen, J.Smih, W. van Neerven, Phys. Le. 321B, 254 (1994); Nucl. Phys. B368, 543(1992). 43