Method of Feldman and Cousins for the construction of classical confidence belts

Transcription

1 Method of Feldman and Cousins for the construction of classical confidence belts Ulrike Schnoor IKTP TU Dresden ATLAS Seminar U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 1 / 23

2 Feldman-Cousins method New method to solve two problems in small signal analysis at the same time: flip-flopping (choice of interval based on data) non-confidence intervals unphysical intervals (empty set) for both upper limits and central confidence intervals U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 2 / 23

3 Feldman-Cousins method 1 Reminder: confidence intervals 2 Flip-flopping and unphysical confidence intervals 3 Solution with Feldman-Cousins method U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 2 / 23

4 Reminder: Confidence intervals Bayesian confidence intervals = credible intervals For an observable x whose p.d.f. depends on the parameter θ, the Bayesian confidence interval (θ 1, θ 2 ) corresponding to C. L. α is constructed by: where with P(θ t x 0 ): posterior p.d.f. L(x 0 θ): likelihood function P(x 0 ): normalization constant P(θ t): prior p.d.f. θ2 θ 1 P(θ t x 0 ) dθ t = α P(θ t x 0 ) = L(x 0 θ) P(θt) P(x 0 ) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 3 / 23

5 Reminder: Confidence intervals Classical confidence intervals For an observable x whose p.d.f. depends on the parameter θ, a classical confidence interval (θ 1, θ 2 ) corresponding to C. L. α is a member of a set with the property with θ 1, θ 2 functions of x. (1) is true for every allowed θ, in particular θ t. P(θ [θ 1, θ 2 ]) = α (1) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 4 / 23

6 Reminder: Confidence intervals Classical confidence intervals For an observable x whose p.d.f. depends on the parameter θ, a classical confidence interval (θ 1, θ 2 ) corresponding to C. L. α is a member of a set with the property with θ 1, θ 2 functions of x. (1) is true for every allowed θ, in particular θ t. P(θ [θ 1, θ 2 ]) = α (1) Construction according to Neyman s method U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 4 / 23

7 Neyman s method of confidence belts U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 5 / 23

8 Neyman s method of confidence belts p.d.f. f (x; θ): x... outcome of experiment (estimator for θ) θ... parameter of the p.d.f. U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 5 / 23

9 Neyman s method of confidence belts p.d.f. f (x; θ): x... outcome of experiment (estimator for θ) θ... parameter of the p.d.f. Acceptance interval [x 1, x 2 ] for given C. L. α for each value of θ: P(x 1 < x < x 2 ; θ) = α = x 2 f (x; θ)dx x 1 U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 5 / 23

10 Neyman s method of confidence belts p.d.f. f (x; θ): x... outcome of experiment (estimator for θ) θ... parameter of the p.d.f. Acceptance interval [x 1, x 2 ] for given C. L. α for each value of θ: P(x 1 < x < x 2 ; θ) = α = x 2 f (x; θ)dx x 1 Union of the intervals: confidence belt D(α) x 1 (θ, α), x 2 (θ, α) monotonic inverse functions θ 1 (x) = x 1 1 (x(θ)), θ 2 (x) = x 1 2 (x(θ)) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 5 / 23

11 Neyman s method of confidence belts p.d.f. f (x; θ): x... outcome of experiment (estimator for θ) θ... parameter of the p.d.f. Acceptance interval [x 1, x 2 ] for given C. L. α for each value of θ: P(x 1 < x < x 2 ; θ) = α = x 2 f (x; θ)dx x 1 Union of the intervals: confidence belt D(α) x 1 (θ, α), x 2 (θ, α) monotonic inverse functions θ 1 (x) = x 1 1 (x(θ)), θ 2 (x) = x 1 2 (x(θ)) x > x 1 (θ) θ 1 (x) > θ x < x 2 (θ) θ 2 (x) < θ U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 5 / 23

12 Neyman s method of confidence belts p.d.f. f (x; θ): x... outcome of experiment (estimator for θ) θ... parameter of the p.d.f. Acceptance interval [x 1, x 2 ] for given C. L. α for each value of θ: P(x 1 < x < x 2 ; θ) = α = x 2 f (x; θ)dx x 1 Union of the intervals: confidence belt D(α) x 1 (θ, α), x 2 (θ, α) monotonic inverse functions θ 1 (x) = x 1 1 (x(θ)), θ 2 (x) = x 1 2 (x(θ)) x > x 1 (θ) θ 1 (x) > θ x < x 2 (θ) θ 2 (x) < θ Confidence interval (θ 2 (x 0 ), θ 1 (x 0 )) for measurement x 0 : θ between θ 1 (x) and θ 2 (x) x lies between x 1 (θ) and x 2 (θ) θ P(θ 2 (x) < θ < θ 1 (x)) = α U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 5 / 23

13 Neyman s method of confidence belts Interval choices Upper confidence limits: P(x < x 1 θ) = 1 α satisfies P(θ > θ 1 ) = 1 α Central confidence intervals: P(x < x 1 θ) = P(x > x 2 θ) = 1 α 2 Choices requiring ordering principle, e.g. Feldman-Cousins method U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 6 / 23

14 Coverage Coverage P(θ 2 (x) < θ < θ 1 (x)) = α satisfied θ U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 7 / 23

15 Coverage Coverage P(θ 2 (x) < θ < θ 1 (x)) = α satisfied θ Neyman s exact construction gives confidence interval with coverage probability α for discrete x or approximative construction methods: U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 7 / 23

16 Coverage Coverage P(θ 2 (x) < θ < θ 1 (x)) = α satisfied θ Neyman s exact construction gives confidence interval with coverage probability α for discrete x or approximative construction methods: Undercoverage θ : P(θ [θ 1, θ 2 ]) < α U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 7 / 23

17 Coverage Coverage P(θ 2 (x) < θ < θ 1 (x)) = α satisfied θ Neyman s exact construction gives confidence interval with coverage probability α for discrete x or approximative construction methods: Undercoverage θ : P(θ [θ 1, θ 2 ]) < α Overcoverage θ : P(θ [θ 1, θ 2 ]) > α U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 7 / 23

18 Coverage Coverage P(θ 2 (x) < θ < θ 1 (x)) = α satisfied θ Neyman s exact construction gives confidence interval with coverage probability α for discrete x or approximative construction methods: Undercoverage θ : P(θ [θ 1, θ 2 ]) < α Overcoverage θ : P(θ [θ 1, θ 2 ]) > α Conservative intervals Overcoverage for some values of θ (loss of power) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 7 / 23

19 Two major problems can arise from Neyman s construction of confidence intervals U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 8 / 23

20 Example I - Gaussian with boundary at origin Gaussian distribution (σ = 1) for physical values µ > 0: P(x µ) = 1 2π exp( (x µ)/2) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 9 / 23

21 Example I - Gaussian with boundary at origin Gaussian distribution (σ = 1) for physical values µ > 0: P(x µ) = 1 2π exp( (x µ)/2) Unphysical confidence level E.g. for x = 1.8: confidence interval is empty set! Reason: negative values of µ are unphysical. U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 9 / 23

22 Example I - Gaussian with boundary at origin Flip-flopping : choice of interval type based on data for x > 3σ: central interval for 0 x 3σ: upper limit for x < 0: assume x 0 =0 FLIP-FLOPPING e.g. for µ = 2: this interval only contains 85% of P(x µ) undercoverage U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 10 / 23

23 Example II - Poisson with background Poisson distribution with background b: P(n µ) = (b = 3.0 in plots) (µ + b)n e (µ+b) n! U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 11 / 23

24 Example II - Poisson with background Poisson distribution with background b: P(n µ) = (b = 3.0 in plots) (µ + b)n e (µ+b) n! Unphysical confidence level E.g. for n = 0: confidence interval is empty set! Reason: negative values of µ are unphysical. U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 11 / 23

25 Example II - Poisson with background Poisson distribution with background b: P(n µ) = (b = 3.0 in plots) (µ + b)n e (µ+b) n! Unphysical confidence level E.g. for n = 0: confidence interval is empty set! Reason: negative values of µ are unphysical. FLIP-FLOPPING Interval choice based on data undercoverage U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 11 / 23

26 Feldman-Cousins method solves both problems at the same time! U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 12 / 23

27 Feldman-Cousins ordering principle... choice of acceptance interval based on likelihood ratio λ = f (x; θ) f (x; θ best ) with f (x; θ)... likelihood of θ given data x f (x; θ best )... θ best maximizes the likelihood for given x Integration steps: λ>λ min (α) dx f (x; θ) = α see examples U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 13 / 23

28 Ex. II - Feldman-Cousins for Poisson with background Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 14 / 23

29 Ex. II - Feldman-Cousins for Poisson with background Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for values of signal mean µ = 0.5: for each n, determine P(n µ) (P(0 0.5) = 0.03), and value µ best that maximizes P(n µ) (µ best = 0 for n = 0) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 14 / 23

30 Ex. II - Feldman-Cousins for Poisson with background Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for values of signal mean µ = 0.5: for each n, determine P(n µ) (P(0 0.5) = 0.03), and value µ best that maximizes P(n µ) (µ best = 0 for n = 0) determine ratio of likelihoods R = P(n µ) ; rank points n decreasingly P(n µ best ) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 14 / 23

31 Ex. II - Feldman-Cousins for Poisson with background Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for values of signal mean µ = 0.5: for each n, determine P(n µ) (P(0 0.5) = 0.03), and value µ best that maximizes P(n µ) (µ best = 0 for n = 0) determine ratio of likelihoods R = P(n µ) ; rank points n decreasingly P(n µ best ) add points n to acceptance interval according to rank until P(n µ) α U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 14 / 23

32 Ex. II - Feldman-Cousins for Poisson with background Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for values of signal mean µ = 0.5: for each n, determine P(n µ) (P(0 0.5) = 0.03), and value µ best that maximizes P(n µ) (µ best = 0 for n = 0) determine ratio of likelihoods R = P(n µ) ; rank points n decreasingly P(n µ best ) add points n to acceptance interval according to rank until P(n µ) α repeat for all µ and different values of b; determine confidence interval (µ 1, µ 2 ) according to standard procedure (vertical line) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 14 / 23

33 Ex. II - Feldman-Cousins for Poisson with background no empty sets coherent set of intervals no flip-flopping slightly conservative due to discreteness of n U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 15 / 23

34 Ex. I - Feldman-Cousins for Gaussian with boundary at origin For Gaussian with boundary at origin P(x µ) = 1 2π exp( (x µ)/2) Determine acceptance interval for signal mean µ: for each value of x find µ best that maximizes P(x µ): µ best = max(0, x) P(x µ best ) = { 1/ 2π, x 0 exp(x 2 /2)/ 2π, x > 0 U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 16 / 23

35 Ex. I - Feldman-Cousins for Gaussian with boundary at origin For Gaussian with boundary at origin P(x µ) = 1 2π exp( (x µ)/2) Determine acceptance interval for signal mean µ: for each value of x find µ best that maximizes P(x µ): µ best = max(0, x) P(x µ best ) = { 1/ 2π, x 0 exp(x 2 /2)/ 2π, x > 0 compute R: R(x) = { P(x µ) exp( (x µ) 2 P(x µ best ) = /2), x 0 exp(xµ µ 2 /2), x > 0 U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 16 / 23

36 Ex. I - Feldman-Cousins for Gaussian with boundary at origin For Gaussian with boundary at origin P(x µ) = 1 2π exp( (x µ)/2) Determine acceptance interval for signal mean µ: for each value of x find µ best that maximizes P(x µ): µ best = max(0, x) P(x µ best ) = { 1/ 2π, x 0 exp(x 2 /2)/ 2π, x > 0 compute R: R(x) = { P(x µ) exp( (x µ) 2 P(x µ best ) = /2), x 0 exp(xµ µ 2 /2), x > 0 integrate over the R-ranked intervals: find interval [x 1, x 2 ] such that R(x 1 ) = R(x 2 ) and x 2 P(x µ)dx = α x 1 U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 16 / 23

37 Ex. I - Feldman-Cousins for Gaussian with boundary at origin For Gaussian with boundary at origin P(x µ) = 1 2π exp( (x µ)/2) Determine acceptance interval for signal mean µ: for each value of x find µ best that maximizes P(x µ): µ best = max(0, x) P(x µ best ) = { 1/ 2π, x 0 exp(x 2 /2)/ 2π, x > 0 compute R: R(x) = { P(x µ) exp( (x µ) 2 P(x µ best ) = /2), x 0 exp(xµ µ 2 /2), x > 0 integrate over the R-ranked intervals: find interval [x 1, x 2 ] such that R(x 1 ) = R(x 2 ) and x 2 P(x µ)dx = α x 1 find acceptance interval for each µ confidence interval [µ 1, µ 2 ] for each x 0 U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 16 / 23

38 Ex. I - Feldman-Cousins for Gaussian with boundary at origin Figure: Feldman-Cousins confidence belt Figure: Flip-flopping confidence belt no empty sets no flip-flopping slightly conservative at transition from upper limit to two-sided interval U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 17 / 23

39 Sources G. Cowan: Statistical data analysis (Oxford University Press 1998) G. Feldman, R. Cousins: A Unified Approach to the Classical Statistical Analysis of Small Signals F. James: Statistical Methods in Experimental Physics (World Scientific 2006) B. D. Yabsley: Neyman & Feldman-Cousins intervals for a simple problem with an unphysical region, and an analytic solution (arxiv:hep-ex/ v1 2006) PDG Review, Ch. 33 (Statistics) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 18 / 23

40 BACKUP U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 19 / 23

41 Reminder: Bayesian and Classical confidence levels Bayesian Confidence Level α : degree of belief that θ t [θ 1, θ 2 ] inference about the true value θ t experiment not repeatable prior contains all previous knowledge / believes Classical Confidence Level α : probability that interval [θ 1, θ 2 ] contains θ no prior, no degree of belief no inference about θ t, but only about (θ 1, θ 2 ) experiment repeatable: fraction α of measurements yields a confidence interval containing the true value θ t U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 20 / 23

42 Ex. II - Feldman-Cousins for Poisson with background Construction of confidence interval with ordering principle based on likelihood ratios Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for signal mean µ = 0.5 for all points n: start with n = 0: P(0 0.5) = 0.03 U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 21 / 23

43 Ex. II - Feldman-Cousins for Poisson with background Construction of confidence interval with ordering principle based on likelihood ratios Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for signal mean µ = 0.5 for all points n: start with n = 0: P(0 0.5) = 0.03 find value of µ that maximizes P(n µ): here µ = max(0, n b), µ = 0 for n = 0 U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 21 / 23

44 Ex. II - Feldman-Cousins for Poisson with background Construction of confidence interval with ordering principle based on likelihood ratios Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for signal mean µ = 0.5 for all points n: start with n = 0: P(0 0.5) = 0.03 find value of µ that maximizes P(n µ): here µ = max(0, n b), µ = 0 for n = 0 calculate the ratio of likelihoods: R = P(n µ), rank decreasingly P(n µ best ) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 21 / 23

45 Ex. II - Feldman-Cousins for Poisson with background Construction of confidence interval with ordering principle based on likelihood ratios Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for signal mean µ = 0.5 for all points n: start with n = 0: P(0 0.5) = 0.03 find value of µ that maximizes P(n µ): here µ = max(0, n b), µ = 0 for n = 0 calculate the ratio of likelihoods: R = P(n µ), rank decreasingly P(n µ best ) add points n to acceptance interval according to rank until P(n µ) α U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 21 / 23

46 Ex. II - Feldman-Cousins for Poisson with background Construction of confidence interval with ordering principle based on likelihood ratios Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for signal mean µ = 0.5 for all points n: start with n = 0: P(0 0.5) = 0.03 find value of µ that maximizes P(n µ): here µ = max(0, n b), µ = 0 for n = 0 calculate the ratio of likelihoods: R = P(n µ), rank decreasingly P(n µ best ) add points n to acceptance interval according to rank until P(n µ) α do this for all µ and different values of b U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 21 / 23

47 Ex. II - Feldman-Cousins for Poisson with background Construction of confidence interval with ordering principle based on likelihood ratios Poisson with background b = 3.0: P(n µ) = (µ + b) n exp( (µ + b))/n! Determine acceptance interval for signal mean µ = 0.5 for all points n: start with n = 0: P(0 0.5) = 0.03 find value of µ that maximizes P(n µ): here µ = max(0, n b), µ = 0 for n = 0 calculate the ratio of likelihoods: R = P(n µ), rank decreasingly P(n µ best ) add points n to acceptance interval according to rank until P(n µ) α do this for all µ and different values of b determine confidence interval (µ 1, µ 2 ) according to standard procedure (vertical line) U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 21 / 23

48 Ex. II - Feldman-Cousins for Poisson with background P(n µ) for given n, µ µ best maximizes P(n µ) for given n R = P(n µ) P(n µ best ) rank according to decreasing R comparison to Upper Limit and central intervals U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 22 / 23

49 Mild pathologies arising from Feldman-Cousins in the case of Poisson with background After determination of acceptance intervals, draw vertical line to get confidence interval. Sometimes, set of intersected lines is not connected! define θ 2 = bottom-most intersected line; θ 1 = top-most intersected line background-dependence: µ 2 (b) has to be decreasing monotonously (if not, lengthen some confidence intervals) These compensations add slightly to conservatism of the intervals, but conservatism remains dominated by discreteness of n U. Schnoor (IKTP TU DD) Felcman-Cousins ATLAS Seminar 23 / 23