Confidence Limits and Intervals 3: Various other topics. Roger Barlow SLUO Lectures on Statistics August 2006

Transcription

1 Confidence Limits and Intervals 3: Various other topics Roger Barlow SLUO Lectures on Statistics August 2006

2 Contents 1.Likelihood and lnl 2.Multidimensional confidence regions 3.Systematic errors: various techniques 4.Profile Likelihood 5.CL s 6.Banff challenge 7.Further Reading Slide 2

3 1 The Likelihood Ratio Estimate a model parameter M by maximising the likelihood In the large N limit i) This is unbiassed ii) The error is given by 1 = d 2 ln L 2 iii) ln L is a parabola dm 2 C d 2 ln L dm 2 L=L max 1 2 C M M 2 iv) We can approximate v) Read off σ from lnl=-½ M = M = d 2 ln L dm 2 Slide 3

4 Neat way to find Confidence Intervals Take lnl= -½ for 68% CL (1σ) lnl=-2 for 95.4% CL (2σ) Or whatever you choose 2-sided or 1-sided Slide 4

5 For finite N None of the above are true Never mind! We could transform from M M' where it was parabolic, find the limits, and transform back Would give lnl=-½ for 68% CL etc as before Hence asymmetric errors Everybody does this Slide 5

6 Is it valid? Try and see with toy model (lifetime measurement) where we can do the Neyman construction For various numbers of measurements, N, normalised to unit lifetime There are some quite severe differences! Slide 6

7 Conclusions on lnl=-½ Poisson count Is it valid? No We can make our curve a parabola, but we can't make the actual 2 nd derivative equal its expectation value Differences in 2 nd significant figure Neyman (solid) lnl=-½ (dashed) Will people stop using it? No But be careful when doing comparisons Further details in NIM (2005) and PHYSTAT05 Slide 7

8 2: More dimensions Suppose 2 uncorrelated parameters, a and b For fixed b, lnl=-½ will give 68% CL region for a And likewise, fixing a, for b Confidence level for square is =46% Confidence level for ellipse (contour) is 39% b L(a,b) Jointly, lnl=-½ gives 39% CL region for 68% need lnl=-1.15 a Slide 8

9 More dimensions, other limits Useful to write -2 lnl=χ 2 Careful! Given a multidimensional Gaussian, ln L =- χ 2 /2. But -2 lnl obeys a χ 2 distribution only in the large N limit... Level is given by finding χ 2 such that P(χ 2,N)=1-CL Generalisation to correlated gaussians is straightforward Generalisation to more variables is straight forward. Need the larger lnl 68% 95% 99% etc Slide 9

10 Small N non-gaussian measurements No longer ellipses/ellipsoids Use lnl to define confidence regions, mapping out contours Probably not totally accurate, but universal Slide 10

11 Have dataset What's the alternative? Toy Monte Carlo Take point M in parameter space. Is it in or out of the 68% (or...) contour? T =ln L R M ln L R M Find clearly small T is 'good' Generate many MC sets of R, using M How often is T MC >T data? If more than 68%, M is in the contour We are ordering the points by their value of T (the Likelihood Ratio) almost contours but not quite Slide 11

12 Formalism 3: Nuisance parameters Systematic Errors Model parameter M Result R Nuisance parameter(s) N Likelihood is L(M,N R) from experiment L'(N) about N These are combined Example: Poisson counting Source strength S R events seen Background b b=b 0 ±σ b Poisson(R,S+b) Gauss(b,b 0,σ b ) e s b s b R R! 1 o 2 2 b 2 e b b /2 b Slide 12

13 Approach 1 Quote joint CL contours for N and M N M This is a nonstarter. Nobody cares about N. You're losing information about M. (N may be multidimensional) Slide 13

14 Approach 2 Set N to central values to get quoted result for M. Then shift N up one sigma, repeat, and get (systematic) error on M No theoretical justification Grossly overestimates error on M Still in use in some backward areas Slide 14

15 Approach 3 Integrate out N to get L(M,R) This can be done analytically or numerically Study L(M,R) and use lnl=-½ or equivalent This is a frequentist/bayesian hybrid. Acceptable (?) if the effects are small. Slide 15

16 Approach 4 Profile Likelihood Use Find maximum L See how it falls off and use lnl=-½ or equivalent, maximising by adjusting N as you step through M L R, M =L R, M, N Intuitively sensible Studies show it has reasonable properties Slide 16

17 4: Justification (?) for using profile likelihood technique Suppose {M,N} can be replaced by {M,N'} such that L(R,M,N)=L(R,M) L'(R,N') There are cases where this obviously works There are cases where it obviously doesn't work Assuming it does, the shape of L(R,M) can be found by fixing N'. Can fix N' by taking the peak for given M, as L'(R,N') is independent of M and peak is always at the same N' Slide 17

18 Profile Likelihood Provided by Minuit Available in ROOT as TRolke Use it! Slide 18

19 5: The CL S Technique Used for Higgs searches by the combined LEP experiments. 'Frequentist-motivated' Different experiments selected events with varying degrees of Higgsishness Slide 19

20 Combining information Define test-statistic Q which increases with signal s. Use Likelihood ratio L(R M H )/L(R no H) Properties known (as function of M H ) from Monte Carlo Measure some Q obs Define CL b =P b (Q Q obs ) CL s+b =P s+b (Q Q obs ) CL s =CL s+b /CL b Treat this as a CL, even though it isn't. It therefore overcovers. Slide 20

21 Why divide? If you see a small number of events, you know that the background has a downward fluctuation. (In the limit of N=0, we know the background is zero) This is like the Bayesian formula n e s b s b r /r! = 0 0 n e b b r /r! Slide 21

22 What happens.. Yellow is 1-CL b Green is CL s+b for given m H Slide 22

23 Results (as of 2002) Rule out M H up to GeV (> %) Slide 23

24 Summary on CL s Used for several searches at LEP and elsewhere Adaptive and sensible. Frequentist but 'behaves like P(theory Data)' Well adapted to exclusion. See Alex Read's talks at CERN and Durham workshops Slide 24

25 6: The Banff Workshop outcome Particle+statistics workshop in Banff, July Task: given model n=poisson(ε s+b) y=poisson(tb) z=poisson(uε) t, u known. n,y,z known Put limit(s) on s at 90% and 99% CL. Slide 25

26 Many Models Bayesian Frequentist Hybrid... Part 1: 10,000 'experiments' have been generated Participants to run their models and report results Will be scored for coverage and shortness Part 2: same again but with 10 separate channels per experiment (same s, different t,u,y,z and n) Results to be announced in due course... Slide 26

27 7: Further reading The Particle Data Book Textbooks by Glen Cowan, Louis Lyons, R.B. Recommended Statistical Procedures for BaBar BAD 318 PHYSTAT proceedings (all Ed. Louis Lyons): CERN Durham 2002 IPPP 02/39 SLAC 2003 SLAC-R-703 Oxford 2005 Statistical problems in Particle Physics, Imperial College Press (2006) Slide 27