Modern Methods of Data Analysis - SS 2009

Transcription

1 Modern Methods of Data Analysis Lecture X (2.6.09) Contents: Frequentist vs. Bayesian approach Confidence Level

2 Re: Bayes' Theorem (1) Conditional ( bedingte ) probability

3 Examples: Rare Disease (1) Probability of disease A: P(A) = P(not A) = Test for disease: P(+ A) = 0.98 P(- A) = 0.02 P(+ not A) = 0.03 P(- not A) = 0.97 Do you need to be worried if you get + as test result? What is the posteriori probability P(A +)?

4 Example: Rare Disease (2) The posterior probability is only 3.2%, due to the tiny prior probability and the non negligible misidentification rate.

5 Bayesian' Theorem with pdfs x1,... x10, measurements distributed accordingly to Gaussian with known σ, however mean a not known. Apriori pdf (probability) Normalization (does not depend on a): Frequentist approach: a is a fixed value, has no pdf, only work with L(a) Bayesian approach: Treat a as a random variable, work with f(a x)! This becomes very important for CL calculation.

6 Bayesian Interpretation of All available information about a is contained in f(a x). Possible interpretations are... best estimate Maxize probability error interval, in which the (true) parameter value is located with probability p center interval

7 Example: Coin Throwing N times a coin, obtaining H times head: Bayesian approach has proper description for parameter bounds.

8 Re: Unphysical Result in normal Likelihood Likelihood often sum of two or more components (signal + background) e.g. toy MC with 6 signal & 60 background events Although negative number of # signal unphysical, need to use them, when combining with other experiments otherwise bias.

9 Example Coin: Bayesian Likelihood Uniform distribution is mathematical identical to normal (Frequentist) Likelihood Seite 16

10 Example Coin: Bayesian Likelihood Seite 17

11 Example Coin: Bayesian Likelihood Use two different priors asuming almost unbiased coin: Gaussian distribution around 0.5 asuming very biased coin, don't exclude any other unlikely but possible hypothesis probability not zero in the center!

12 Example Coin: Bayesian Likelihood Seite 18

13 Example Coin: Bayesian Likelihood Seite 19 For large N prior has no impact anymore. Be careful to not exclude unlikely but possible results with prior!

14 Confidence Level So far got a set of recipes to derive an estimate for parameters of our model. A result is however useless without any statement how much it can be trusted. -> Confidence Level (CL) We warn the reader that there is no universal convention for the term confidence level - The Review of Particle Properties - When I use a word, it means just what I choose it to mean - neither more nor less. - Lewis Carroll ( Alice ) -

15 Example: Packets of Cereals Suppose cereal packets are produced according to Gaussian distribution of mean 520g and standard deviation 10g. 68% of the packets will weight more than 510 and less than 530g. Challenged by a consumer group we say, the real weight is between 510 and 530 g. This statement is true in 68% of the cases. Common values: Bild Seite % 90.0% 95.0% 95.4% 99.0% : : : : : 1σ 1.64σ 1.96σ 2σ 2.58σ Only true for 1-dim Gaussian!

16 Double Sided Confidence Level A CL is one number but defines two numbers a minimum and a maximum value => additional information needed The symmetric interval: limits are equidistant from the mean: The shortest interval: is minimal The central interval (usually the best one to use): (for symmetric distributions, the three definitions are equivalent)

17 Single Sided Confidence Level upper limit lower limit Be careful with wording: Upper limit of a 95% central confidence interval and the upper 95% limit are not the same. The former has 97.5% of the probability content below it and 2.5% above; the latter has 95% below and 5% above. Note: to give this intervals you assume that the pdf P(x) is given!

18 Confidence Intervals in Estimation Suppose we want to know parameter X. And have estimate from data. The measurement resolution is σ. Now need to turn knowledge of and σ into statement about confidence level about the true value X. Naïve answer: X lies in with 68% confidence and in with 95% confidence is dynamite. This means you interprete L(a) as a pdf of a, wich is not true! This contains hidden assumption of underlying (prior) distribution!

19 Example: Impossible Probability The weight of an empty dish is measured as 25.30±0.14g. A sample of powder is placed on the dish, and the combined weight measured as 25.50±0.14g. By subtraction and combination of errors, the weight of the powder is determined to be 0.2±0.2g. naïve approach to CL: There is a 16% chance of the weight being smaller 1σ and 16% being larger than 1σ compared to the measured value. It is nonsense to say, probability for negative mass is 16%!

20 How to determine CL range (I): L(x; A) is the probability density distribution to measure x, if A is the true value. For every value A we can compute the 90% central CL. For a given A, the probability to find a measurement result x outside this range is 10%. A A+ A- For true A smaller than A-, the chance to observe a value x is smaller 5%. For true A larger than A+, the chance to observe a value x is smaller 5%. Not say, the chance that true A is in [A-,A+] is 90%! (Frequentist: A is a parameter, it's value is fixed!)

21 How to determine CL range (II): E.g. Poission statistik: measure n=9 Naively (Gaussian approximation): 9±1.65* 9 = 9±4.9 True limits:

22 Confidence Level For Gaussian x is required to lie some number of standard deviations (in this case 1.64) above A-. This is the same as saying that A- must lie the same number of σ below the measured x, which can be written in the form (assume σ independ of x):

23 Confidence Level for Gaussian Lines are parallel (slope 45 ). This is only true for Gaussian distributions and no physical bounderies for A! A A A+ A+ A- A- Most of the time CL are given for searches for rare events... small rates not Gaussian.

24 Measurement of Constraint Quantity Measurement of the mass of a particle with mass 0.1g (clear physical boundary, m > 0) σ = 0.2: x ± 0.4 correspond to 95.4% Measurement of 0.5: Measurement of 0.3:... Measurement of -0.3: Measurement of -0.5: 95.4% CL: [0.1,0.9] 95.4% CL: [-0.1,0.7] => [0.0,0.7] 95.4% CL: [-0.7,0.1] => [0.0,0.1] 95.4% CL: [-0.9,-0.1] =>??? Extreme case is clear nonsense! At the moment physical boundary is touched, intervals don't correspond anymore to 95% CL.

25 Bayesian Approach Add restriction to physical boundaries to prior Renormalize probability in allowed range to 1. Distribution is now non-symmetric, so there is now a choice between symmetric, smallest & central interval. Prior is uniform for A in [0, ]

26 Example: measured mass: -0.5 ± 0.2, what is the 90% confidence single sided upper limit? P(A > x+2.5σ) = (see Table); P(A > x+3.24σ) = (see Table); including physical boundaries renormalization P(A>x+3.24*0.2)/P(A>0) = / < 10%; A+ = Bayesian approach only way to get meaningful conclusion from unphysical results Be aware: if you would use -A² or 1/A the resulting limits would be incompatible. The assumption of complete ignorance (flat distribution within physical allowed range) means different things when applied to different forms of the same basic variable.

27 Poisson Confidence Intervals Poisson distribution (for low rates): Assume we have measured n events: or or Note: If # observed events: 0; no lower limit can be placed. Poisson values are listed in tables

28 Exercise: Measurement with n=4 events. Give upper and lower 90% CL for # of expected events.assuming Poisson statistic, and physical boundary n 0 Compare to 90% CL obtained for assuming Gaussian prior without physical boundary. Compare to 90% CL obtained for assuming Gaussian prior with physical boundary (# expected events 0).

29 Confidence Region for Several Variables Consider for the moment being far away from any physical boundary - nσ region/surface defined as: Note: ± 1.96σ (ΔΧ² = 1.96²) does not correspond to 95% CL! To do: check Χ²-probability P(ndf=2, Χ²>5.99) < 5% (look at table) ΔΧ² = 5.99 or Δln(L) = 5.99/2 contour correspond to 95% CL

30 Student's t-distribution How to quote CL if there are add. uncertainties on σ? unbiased estimate if μ known: else: normalized Gaussian distribution t is distributed according to Student's distribution (slightly broader than Gaussian, slightly larger CL regions) Bild Student's distribution Seite 136

31 Exercise: A test of 20 students shows an average IQ of 128 with an of 15. What is the 90% central confidence limit on the IQ of all students? What is the 90% CL if same mean and spread is measured with 5 students only. The usual metal content of a certain ore is 5.6%. Samples of ore from a possible mining site are assayed and have metal contents of 6,7,9, and 8%. The average content is better than usual, but is this 95% significant? A ML fit gives you the result (a,b) with following Δln(L) ellipse. Is the indicated point (a1,b1) in the 95/98% CL region?