Confidence Intervals Lecture 2

Transcription

1 Confidence Intervals Lecture 2 First ICFA Instrumentation School/Workshop At Morelia,, Mexico, November 18-29, 2002 Harrison B. rosper Florida State University

2 Recap of Lecture 1 To interpret a confidence level as a relative frequency requires the concept of a set of ensembles of experiments. Each ensemble has a coverage probability,, which is simply the fraction of experiments in the ensemble with intervals that contain the value of the parameter pertaining to that ensemble. The confidence level is the minimum coverage probability over the set of ensembles. November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 2

3 What s Wrong With Ensembles? Nothing, if they are obectively real, such as: The people in Morelia between the ages of 16 and 26 Daily temperature data in Morelia during the last decade But the ensembles used in data analysis typically are not There was only a single instance of Run I of DØ and CDF. But to determine confidence levels with a frequency interpretation we must embed the two experiments into ensembles, that is, we must decide what constitutes a repetition of the experiments. The problem is that reasonable people sometimes disagree about the choice of ensembles, but, because the ensembles are not real, there is generally no simple way to resolve disagreements. November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 5

4 Outline Deductive Reasoning Inductive Reasoning robability Bayes Theorem Example Summary November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

5 Deductive Reasoning Consider the propositions: A this is a baby B she cries a lot Maor premise: If A is TRUE, then B is TRUE Minor premise: A is TRUE Conclusion: Therefore, B is TRUE Maor premise: If A is TRUE, then B is TRUE Minor premise: B is FALSE Conclusion: Therefore, A is FALSE Aristotle, ~ 350 BC AB A November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

6 A this is a baby B she cries a lot Deductive Reasoning - ii Maor premise: If A is TRUE, then B is TRUE Minor premise: A is FALSE Conclusion: Therefore, B is? Maor premise: If A is TRUE, then B is TRUE Minor premise: B is TRUE Conclusion: Therefore, A is? AB A November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

7 A this is a baby B she cries a lot Inductive Reasoning Maor premise: If A is TRUE, then B is TRUE Minor premise: B is TRUE Conclusion: Therefore, A is more plausible Maor premise: If A is TRUE, then B is TRUE Minor premise: A is FALSE Conclusion: Therefore, B is less plausible Can plausible be made precise? November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

8 Yes! Bayes1763, Laplace1774, Boole1854, Jeffreys1939, Cox1946, olya1946, Jaynes1957 In 1946, the physicist Richard Cox showed that inductive reasoning follows rules that are isomorphic to those of probability theory November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

9 The Rules of Inductive Reasoning: Boolean Algebra + robability Boolean Algebra A A + 0 A + A 1 A + B B + A A + B C A + B A + C A 1 A A A 0 A B B A A B + C A B + A C November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

10 November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 12 robability robability / / A AB A B B AB B A B A AB AB B A B A + + Conditional robability A theorem

11 robability - ii roduct Rule AB B A A A B B Sum Rule A B + A B 1 Sum Rule Bayes Theorem B A A B B/ A These rules together with Boolean algebra are the foundation of Bayesian robability Theory November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

12 Bayes Theorem C D i A A C D i C D i / A if C D i C D i then we can write Bayes' Theorem as are exhaustive propositions, i.e., A i, A C D A C D i i C D i i C D i, C D i A 1, We can sum over propositions that are of no interest marginalization C i A CiD A November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

13 Bayes Theorem: Example 1 Signal/Background Discrimination S Signal B Background S Data Data S S Data S S + Data B B The probability SData Data, of an event being a signal given some event Data,, can be approximated in several ways, for example, with a feed-forward forward neural network November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 15

14 Bayes Theorem: Continuous ropositions posterior θ, λ x, I likelihood marginalization prior x θ, λ, I θ, λ I θ, λ x θ, λ, I θ, λ I θ x, I θ, λ x, I λ θ, λ x, I f θ, λ x dθ dλ I is the prior information November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

15 Bayes Theorem: Model Comparison Standard Model For each model M, we integrate over the parameters of the theory. L x M, I x θ, M, I θ M, I θ M x, I M L x M, I M I L x M, I M I Mx,I is the probability of model M given data x. SUSY Models November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 17

16 Bayesian measures of uncertainty Variance Bayesian Confidence Interval 2 2 σ θ θ θ x, I where β θ θ u x θ ˆ θ θ x, I θ θ x, I θ l x Also known as a Credible Interval November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

17 Example 1: Counting Experiment Experiment: To measure the mean rate of UHECRs above ev per unit solid angle. Assume the probability of N events to be a given by a oisson distribution θ e θ n θ, θ < n n! n >, σ θ November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 19

18 Example 1: Counting Experiment - ii Apply Bayes Theorem θ n, I n n θ, I θ I θ, I θ I What should we take for the prior probability I? θ How about I d? θ n, I e θ n! θ n November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 20

19 But why choose I d? Why not 2 I d? Or tan I d? Choosing a prior probability in the absence of useful prior information is a difficult and controversial problem. The difficulty is to determine the variable in terms of which the prior probability density is constant. In practice one considers a class of prior probabilities and uses it to assess the sensitivity of the results to the choice of prior. November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 21

20 Credible Intervals With I d 20 Bayesian Interval - oisson Distribution Bayesian 15 arameter 10 N± N Count November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 22

21 Credible Interval Widths 10 Width of Intervals 8 Bayesian Interval Width 6 4 N± N Count Bayesian RootN November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 23

22 Coverage robabilities: Credible Intervals 1 Coverage robability 0.8 Bayesian robability Even though these are 0.2 Bayesian intervals nothing prevents us from studying their frequency behavior with respect to some ensemble! oisson arameter November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 24

23 Bayes Theorem: Measuring a Cross-section section Model θ σl +b Data n rior information lˆ,, ˆ, δl b δb Likelihood n σ, b, l, I e θ n! θ n l is the efficiency times branching fraction times integrated luminosity b is the mean background count lˆ, bˆ are estimates November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

24 Measuring a Cross-section section - ii Apply Bayes Theorem: posterior likelihood n σ, b, l, I σ, b, l I σ, b, l n, I n σ, b, l, I σ, b, l I σ, b, l Then marginalize: that is, integrate over nuisance parameters σ n, I σ, b, l n, I b, l prior November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper

25 November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 27 What rior? What rior?,, so we can write, and,, But usually,,,,,, I I b I l I b l I b I b I l I b l I I b I b l I b l σ σ σ σ σ σ σ σ We can usually write down something sensible for the luminosity and background priors. But, again, what to write for the cross-section prior is controversial. In DØ and CDF, as a matter of convention, we set I d I d Rule: Always report the Rule: Always report the prior prior you have used! you have used!

26 Summary Confidence levels are probabilities; as such, their interpretation depends on the interpretation of probability adopted If one adopts a frequency interpretation the confidence level is tied to the set of ensembles one uses and is a property of that set. Typically, however, these ensembles do not obectively exist. If one adopts a degree of belief interpretation the confidence level is a property of the interval calculated. A set of ensembles is not required for interpretation. However, the results necessarily depend on the choice of prior. November 21, 2002 First ICFA Instrumentation School/Workshop Harrison B. rosper 28