E. Santovetti lesson 4 Maximum likelihood Interval estimation

Transcription

1 E. Santovetti lesson 4 Maximum likelihood Interval estimation 1

2 Extended Maximum Likelihood Sometimes the number of total events measurements of the experiment n is not fixed, but, for example, is a Poisson r.v., mean. The extended likelihood function is then: If is a function of we have: Example: the expected number of events of a certain process Extended ML uses more informations, so the error on the parameters will be smaller, compared to the case in which n independent In case n doesn't depend from we have the usual likelihood 2

3 Extended ML example Consider two types of events (e.g., signal and background) each of which predict a given pdf for the variable x: fs(x) and fb(x). We observe a mixture of the two event types, signal fraction = θ, expected total number = ν, observed total number = n Let s = and b = (1- be the number of signal and background that we want to evaluate 3

4 Extended ML example (2) Consider for signal a Gaussian pdf and for background an exponential Maximize the log L, to find s and b Here errors reflect total Poisson fluctuation as well as that in proportion of signal/background 4

5 Unphysical values for estimators Here the unphysical estimator is unbiased and should nevertheless be reported, since average of a large number of unbiased estimates converges to the true value (cf. PDG). Repeat entire MC experiment many times, allow unphysical estimates. 5

6 Extended ML example II LH does not provide any information on the Goodness of the fit. This has to be checked separately. Simulate toy MC according to estimated pdf (using fit results from data as true parameter values) compare max Likelihood value in toy to the one in data Draw data in (binned) histogram, compare distribution with result of ML fit. 6

7 Extended ML example II Again you want to distinguish (and count) signal events respect to background events. The signal is the process The background is combinatorial: vertex (and then particle) reconstructed with the wrong tracks. To select signal from background we can use two main things: 1) The invariant mass of the two daughter particles has to peak to the B meson mass; 2) The time of flight of the B meson candidate has to be of the order of the B meson life time. These two variables have to behave in a complete different way for the two event categories. Let us see the distribution of this variables 7

8 Extended ML example II A first look at the distribution (mass and time) allow us to state: pdf for signal mass: double Gaussian pdf for signal time: exponential (negative) pdf for background mass: exponential (almost flat) pdf for background time: exponential + Lorentzian We build the pdf and make as: By maximizing the likelihood, we can estimate the number of signal and background as well as the B meson mass and life time 8

9 Extended ML example II signal background all data mass Life time The fit is done with the RooFit package (Root) 9

10 Weighted maximum likelihood Suppose we want to measure the polarization of the J/Ψ meson (1--). The measurement can be done by looking at the angular distribution of the decay product of the meson itself: and are respectively the polar and azimuthal angles of the positive muon, in the decay in the J/Ψ rest frame, measured choosing the J/Ψ direction in the lab frame as polar axis. θ 10

11 Weighted likelihood polarization measurement We have to measure the angular distribution and fit with the function There are two main problem to face: When we select our signal, there is an unavoidable amount of background events (evident from the mass distribution) The angular distribution of the background events is unknown and also very difficult to parametrize The likelihood function is: Where ε is the total detection efficiency, P is the above angular function and Norm is a normalization function in order to have probability normalized to 1. 11

12 Weighted likelihood polarization measurement The efficiency term at the denominator does not depend on λ parameters and then is a constant term in the maximization procedure. In order to take into account the background events the likelihood sum is extended at all events but with some proper weights: The background mass distribution is linear. The hypothesis is well satisfied otherwise we can always take into account this by readjusting the weights in a proper way right side band The combinatorial background angular distributions are the same in the signal and side bands regions (can be demonstrated shifting the three regions 300 MeV up) left side band The background events contribution cancels out if: 12

13 Weighted likelihood polarization measurement How to evaluate the Norm function (depending on detector efficiency)? We can again use the MC simulation, considering an unpolarized sample, P=1 and the sum is over the MC events. 13

14 Weighted likelihood polarization measurement Then from MC events we can compute the function: 14

15 The splot technique 15

16 Relationship between ML and Bayesian estimators In Bayesian statistics, both θ and x are random variables:. In the Bayes approach, if θ is a certain hypothesis: posterior θ pdf (conditional pdf for θ given x) prior θ probability Purist Bayesian: p(θ x) contains all the informations about θ. Pragmatist Bayesian: p(θ x) can be a complicated function: summarize by using new estimator Looking at p(θ x): what do we use for π(θ)? No golden rule (subjective!), often represent prior ignorance by π(θ) = constant, in which case 16 But... we could have used a different parameter, e.g., λ = 1/θ, and if prior π(θ) is constant, then π(λ) is not! Complete prior ignorance is not well defined.

17 Relationship between ML and Bayesian estimators The main concern expressed by frequentist statisticians regarding the use of Bayesian probability is its intrinsic dependence on a prior probability that can be chosen in an arbitrary way. This arbitrariness makes Bayesian probability to some extent subjective. Adding more measurements increases one s knowledge of the unknown parameter, hence the posterior probability will depend less, and be less sensitive to the choice of the prior probability. In those cases, where a large number of measurements occurs, in most of the cases the results of Bayesian calculations tend to be identical to those of frequentist calculations. Many interesting statistical problems arise in the cases of low statistics, i.e. a small number of measurements. In those cases, Bayesian or frequentist methods usually leads to different results. In those cases, using the Bayesian approach, the choice of the prior probabilities plays a crucial role and has great influence in the results. One main difficulty is how to chose a PDF that models one s complete ignorance on an unknown parameter. One naively could choose a uniform ( flat ) PDF in the interval of validity of the parameter. But it is clear that if we move the parametrization from x to a function of x (say log x or 1/x), the resulting 17 transformed parameter will no longer have a uniform prior PDF

18 The Jeffreys prior One possible approach has been proposed by Harold Jeffreys, adopting a choice of prior PDF that results invariant under parameter transformation. This choice is: with: Determinant of the Fischer information matrix Examples of Jeffreys prior distributions for some important parameters 18

19 Interval estimation, setting limits 19

20 Interval estimation introduction In addition to a point estimate of a parameter we should report an interval reflecting its statistical uncertainty. Desirable properties of such an interval may include: communicate objectively the result of the experiment; have a given probability of containing the true parameter; provide informations needed to draw conclusions about the parameter possibly incorporating stated prior beliefs. Often use +/- the estimated standard deviation of the estimator. In some cases, however, this is not adequate: estimate near a physical boundary, e.g., an observed event rate consistent with zero. We will look briefly at Frequentist and Bayesian intervals. 20

21 Neyman confidence intervals Rigorous procedure to get confidence intervals in frequentist approach Consider an estimator for a parameter (measurable) We also need the pdf Specify upper and lower tail probabilities, e.g., α = 0.05, β = 0.05, then find functions uα(θ) and vβ(θ) such that Integral over all the possible estimator values We obtain a confidence interval (CL = 1-α-β) for the estimator function of the true parameter value θ. This is the interval for the estimator No unique way to define this interval with the same CL 21

22 Confidence interval from the confidence belt Confident belt region is a function of the parameter Find points where observed estimate intersects the confidence belt This gives the confidence interval for The true parameter Confidence level = 1 - α - β = probability for the interval to cover true value of the parameter (holds for any possible true θ) with 22

23 Confidence intervals by inverting a test Confidence intervals for a parameter θ can be found by defining a test of the hypothesized value θ (do this for all θ): Define values of the data that are disfavored by θ (critical region) such that P(data in critical region) γ for a specified γ, e.g., 0.05 or 0.1. If data observed in the critical region, reject the value θ. Now invert the test to define a confidence interval as: set of θ values that would not be rejected in a test of size γ (confidence level is 1 - γ ). We have to collect many data... The interval will cover the true value of θ with probability 1 - γ. Equivalent to confidence belt construction; confidence belt is acceptance region of a test. 23

24 Relation between confidence interval and p-value Equivalently we can consider a significance test for each hypothesized value of θ, resulting in a p-value, pθ. Equivalently we can consider a significance test for each hypothesized value of θ, resulting in a p-value, pθ. The confidence interval at CL = 1 γ consists of those values of θ that are not rejected. E.g. an upper limit on θ is the greatest value for which pθ γ. In practice find by setting pθ = γ and solve for θ 24

25 Confidence intervals in practice In practice, in order to find the interval [a, b] we have to solve: we replace uα(θ) with and get a we replace uβ(θ) with and get b a is hypothetical value of θ such that b is hypothetical value of θ such that 25

26 Meaning of a confidence interval Important to keep in mind: The interval is random The true θ is an unknown constant Often we report this interval as: This does not mean but: repeat the measurements many times, build the interval according to the same prescription each time, in 1 experiments the interval will contain θ 26

27 Central vs. one-sided confidence intervals Fixed the CL, the choice of and is not unique, in literature this is the so called ordering rule Sometimes, only specified or : one-side interval (limit) Often = = / 2: coverage probability 1- : central confidence interval N.B.: central confidence level does not mean symmetric interval around θ. In HEP the convention to quote the error is: = = / 2 with 1 - = 68.3% = 1σ 27

28 Intervals from the likelihood function In the large sample limit it can be shown for ML estimators: N-dimensional Gaussian, variance V defines a hyper ellipsoidal confidence region If the θ follows a multi-dimentional Gaussian 28

29 Approximate confidence regions from L(θ) So the recipe to find the confidence region with CL = 1 - γ is: For finite samples, these are approximate confidence regions. Coverage probability not guaranteed to be exactly equal to 1 - γ ; no simple theorem to say by how far off it will be (use MC). Remember here the interval is random, not the parameter. 29

30 Example of interval from ln L(θ) For n=1, CL = 1 - γ = Q = 1 30

31 Setting limits on Poisson parameter Consider again the case in which we have a sample of events that contains signal and background (means s and b), and both of them are Poisson variables. Suppose that we can say how many background we expect. Unfortunately we observe: There is clearly no evidence of signal. This means that we cannot exclude s = 0, we can anyway put un upper limit to the number of signal 31

32 Upper limit for Poisson parameter We have to find the hypothetical s such that there is a given small probability, say, γ = 0.05, to find as few events as we did or less. Solving numerically for s, it gives an upper limit at a confidence level of 1-γ (usually 0.95). Suppose b = 0 and we find n = 0 32

33 Calculating Poisson parameter limits To find the lower and upper limits we can use the relation to the 2 distribution with z/2. It can be found: For low fluctuation of n this can give negative result for sup; i.e. confidence interval is empty. 33

34 Limits near a physical boundary Suppose e.g. b = 2.5 and we observe n = 0. If we choose CL = 0.9, we find from the formula for sup negative!? Physicist: We already knew s 0 before we started; can t use negative upper limit to report result of expensive experiment! Statistician: The interval is designed to cover the true value only 90% of the time this was clearly not one of those times. Not uncommon dilemma when limit of parameter is close to a physical boundary 34

35 Expected limit for s = 0 Physicist: I should have used CL = 0.95, then sup = even better: for CL = we get sup = 10-4! We are not taking into account the background fluctuation Reality check: with b = 2.5, typical Poisson fluctuation in n is at least 2.5 = 1.6. How can the limit be so low? Look at the mean limit for the no-signal hypothesis (s = 0) (sensitivity). Distribution of 95% CL limits with b = 2.5, s = 0. Mean upper limit = 4.44 With N MC experiments (poisson) with μ=2.5, I extract n and then evaluate sup with 95% CL 35

36 The flip-flopping problem In order to determine confidence intervals, a consistent choice of ordering rule has to be adopted. Feldman and Cousins demonstrated that the ordering rule choice must not depend on the outcome of the measurements, otherwise the quoted confidence intervals or upper limits could be incorrect. In some cases, experiment searching for a rare signal make the chose, while quoting their result, to switch from a central interval to an upper limit depending on the outcome of the measurement. A typical choice is to quote an upper limit if the significance of the observed signal is smaller than 3σ, and a central value otherwise We have than to quote the error fixing the CL, say 90%. If x 3σ we choose a symmetric interval (5% each), while if x < 3σ an upper limit implies a completely asymmetric interval. 36

37 The flip-flopping problem From a single measurement of x we can decide to quote an interval with a e certain CL if x>3σ: or we can decide to quote only an upper limit if our measurement is x<3σ. 37

38 The flip-flopping problem The choice to switch from a central interval to a fully asymmetric interval (upper limit) based on the observation of x clearly spoils the statistical coverage. Looking at the figure, depending on the value of μ, the interval [x1, x2] obtained by crossing the confidence belt in by an horizontal line, one may have cases where the coverage decreases from 90% to 85%, which is lower than the desired CL. To avoid flip-flopping, decide before the measurement if you quote limit or 2sided interval - and stick to it. Or use Feldman-Cousins 38

39 The Feldman Cousins method The ordering rule proposed by Feldman and Cousins provides a Neyman confidence belt that smoothly changes from a central or quasi-central interval to an upper limit in the case of low observed signal yield. The ordering rule is based on the likelihood ratio given a value θ0 of the unknown parameter under a Neyman construction, the chosen interval on the variable x is defined from the ratio of two PDFs of x, one under the hypothesis that θis equal to the considered fixed value θ0, the other under the hypothesis that θ is equal to the maximum-likelihood estimate value θbest(x) corresponding to the given measurement x. 39

40 Feldman Cousins: Gaussian case Let apply the Feldman-Cousins method to a Gaussian distribution When we divide by f(x μbest) we obtain: That is a an asymmetric function with a longer tail to the negative x values Using Feldman-Cousin approach, for alrge x we have the usual symmetric confidence interval. Going at small x (close to the boundary) the interval becomes more and more asymmetric and at certain point it become a completely asymmetric interval (upper limit) 40

41 The Bayesian approach In Bayesian statistics need to start with prior pdf π(θ), this reflects degree of belief about θ before doing the experiment. Bayes theorem tells how our beliefs should be updated in light of the data x: Then we have to integrate posterior probability to the desired probability confidence level. For the Poisson case, suppose 95% CL, we have: 41

42 Bayesian prior for Poisson parameter Include knowledge that s 0 by setting prior π(s) = 0 for s<0. Often try to reflect prior ignorance with e.g. Not normalized but this is OK as long as L(s) dies off for large s. Not invariant under change of parameter if we had used instead a flat prior for, say, the mass of the Higgs boson, this would imply a non-flat prior for the expected number of Higgs events. Does not really reflect a reasonable degree of belief, but often used as a point of reference; or viewed as a recipe for producing an interval whose frequentist properties can be studied (coverage will depend on true s). 42

43 Bayesian interval with flat prior for s Solve numerically to find limit sup. For special case b = 0, Bayesian upper limit with flat prior numerically same as classical case ( coincidence ). Otherwise Bayesian limit is everywhere greater than classical ( conservative ). Never goes negative. Doesn t depend on b if n = 0 43