FS 2 - Chapter 11 Searhes and limits Paolo Bagnaia last mod. 19-May-17
11 Searhes and limits 1. Probability 2. Searhes and limits 3. Limits 4. Maximum likelihood 5. Interpretation of results methods ommonly used for all reent searhes (e.g. LEP, LHC, gravitational waves); not really new (e.g. "Laboratorio di Meania"); but probably forgotten; not a well-organized disussion, beyond the sope of present letures ( referenes + next year). Paolo Bagnaia FS 2 11 2
1/2 probability: a new entry in the game modern partile physis makes a large use of (relatively) new sienes : probability and her sister statistis; [already in lassial physis, when measuring an observable, "probability to get x, if true = x* and resolution x"]; [we are sientists, not gamblers, and do OT disuss poker and die here]; [q.m. is intrinsially probabilisti, sine the preditions are distributions, while the experiments produe single/disrete values]; but the sentene "the probability that a theory be true" [e.g. "the p. that the Higgs boson exist and its mass be "] is really new; however, a simple investigation shows that life is intrinsially probabilisti (risk, hane, luk many words whih essentially mean "probability", while experiene, past, use, mean "statistis"). Paolo Bagnaia FS 2 11 3
2/2 probability: Kolmogorov axioms Andrei ikolayevih Kolmogorov [Андрей Николаевич Колмогоров] (1903 1987), a Russian (sovieti) mathematiian, in 1933 wrote a fundamental axiomatization of probability; he introdued the spae S of the events (A, B, ) and the event probability as a measure (A) in S. K. axioms are : 1. 0 (A) 1 A S; 2. (S) = 1; 3. A B = Ø (A B) = (A) + (B). Some theorems (easily demonstrated): (Ā) = 1 (A); (A Ā) = 1; (Ø) = 0; A B (A) (B); (A B) = (A) + (B) (A B). A B S Paolo Bagnaia FS 2 11 4
1/4 searhes and limits Sometimes, the result of the study is OT the measurement of an observable x : "x = x exp ± x", but, instead, a qualitative "searh" : "the phenomenon Y (does not) exists", or, alternatively : "the phenomenon Y does OT exist in the kinematial range Φ". [statements with "not" apply if effet is not found, and an "exlusion" (a "limit", when Φ is not omplete) is established] In modern experiments, the searhes oupy more than 50% of the published papers, and almost all are negative [but the Higgs searh at LHC, of ourse]. Obviously, a negative result is OT a failure : if any, it is a failure of the theory under test. [but a positive result is muh more pleasant (and rewarding)] A rigorous method, well understood and "easy" to apply, is imperative. This method is a major suess of the LEP era : it uses math, statistis, physis, ommon sense and ommuniation skill. It MUST be in the panoply of eah partile physiist, both theoretiians and experimentalists. use the Higgs searhes at LEP and LHC as a test ase [these letures must remain inside the SM]; after the Higgs disovery, the fous has shifted toward "bsm" searhes, but the method has not hanged. Paolo Bagnaia FS 2 11 5
2/4 searhes and limits: definitions In the next slides : L int : integrated luminosity; σ s : ross setion of signal (searhed for); σ b : ross setion of bakground (known); ε s : effiieny for signal (0 1, larger is better); ε b : effiieny for bakground (0 1, smaller is better); s : # expeted events of signal [s = L int ε s σ s ]; b : # expeted events of bakground [b = L int ε b σ b ]; n : # expeted events [n = s + b, or n = b]; : # found events ( flutuates around n with Poisson ( Gauss) statistis; : probability, aording to a given pdf ("probability density funtion"); CL : "onfidene level", a limit (< 1) in the umulative probability; Λ : likelihood funtion for signal+bkgd (Λ s ) or bkgd-only (Λ b ) hypotheses; μ : parameter defining the signal level [n = b + µ s], used for limit definition; p : "p-value", probability to get the same result or another less probable, in the hypothesis of bkgd-only; E[x] : expeted value of the quantity "x". Paolo Bagnaia FS 2 11 6
3/4 searhes and limits: verify/falsify A theory (SM, SUSY, ) predits a phenomenon (a partile, a dynami effet), e.g. e +, p, Ω, W ± /Z, H; [in some ases the phenomenon depends on some unknown parameter, e.g. the Higgs boson mass] a new devie (e.g. an aelerator) is potentially able to observe the phenomenon [fully or in a range of the parameters spae still unexplored]; therefore, two possibilities: A. observation: the theory is "verified" (à la Popper) [and the free parameter(s) are measured]; B. non-observation: the theory is "falsified" (à la Popper) [or some subspae in the parameter spae, e.g. an interval in one dimension, is exluded a "limit" is established]; different, less ommon, approah ("model independent") : look for unknown effets, without theoretial guidane, e.g. CP violation, J/ψ. Paolo Bagnaia FS 2 11 7
4/4 searhes and limits: blind analysis the key point : usually b s, but ƒ b (b) and ƒ s (s) are very different uts in the event variables (masses, angles), suh that to enhane s wrt b; when is large ( ), statistial flutuations (s.f.) do OT modify the result; usually (not only for impatiene) is small and its s.f. are important; small variations in the seletion ( small hange in the expeted number for s, b) orrespond to large differenes in the results [e.g., if b=0.001, when hanges 0 1, as in the artoon]; a "neutral" analysis is impossible; a posteriori, it is always easy to justify a hange in the uts, whih strongly affets the results; therefore, the only honest proedure onsists in defining the seletion a priori (e.g. by optimizing the expeted visibility on a m event sample); then, the seletion is "blindly" applied to the atual event sample ( "blind analysis"). bakground region signal region whih is the "right" ut? Paolo Bagnaia FS 2 11 8
1/9 [in the "good ole times", life was simpler : if the bakground is negligible, the first observation led to the disovery, as for e +, p, Ω, W ± and Z] in most ases, the bakground (reduible or irreduible) is alulable; a disovery is defined as an observation that is inompatible with a +ve statistial flutuation respet to the expeted bakground alone; a limit is established if the observation is inompatible with a ve flutuation respet to the expeted (signal + bakground); both statements are based on a "redutio ad absurdum"; sine all values of in [0, ] are possible, it is ompulsory to predefine a CL to "ut" the pdf; the CL for disovery and exlusion an be different : usually for the disovery striter riteria are required; limits a priori the expeted signal s an be ompared with the flutuation of the bakground (in approximation of large number of events, s b) : n σ = s / b is a figure of merit of the experiment; a posteriori the observed number () is ompared with the expeted bakground (b) or with the sum (s + b). Example. We expet 100 bakground events and 44 signal; we use the "large number" approximation ( n = n) : b = 100, b = b = 10; n = s + b = 144, n = 12. The pre-hosen onfidene level is "3 σ". The disovery orresponds to an observation of > (100+3 10) = 130 events. A limit is established if < (144 3 12) = 108 events. There is no deision if 108 < < 130. The values < 70 and > 180 are "impossible". Paolo Bagnaia FS 2 11 9
2/9 limits: problem Problem (based on previous example) : ompute the fator, wrt to previous luminosity, whih allows to avoid the "nodeision" region. Answer : 9/4 = 2.25 Solution : all ƒ the fator; then 100 ƒ + 3 100 ƒ = 144 ƒ - 3 144 ƒ ƒ=9/4; b=225; (s+b)=324, s=99. [ritiism : = 270 is still "no-deision"; find a better solution maybe : 100 ƒ + 3 100 ƒ + 1 = 144 ƒ - 3 144 ƒ but must be integer!] Paolo Bagnaia FS 2 11 10
3/9 limits: Poisson statistis In general, the searhes looks for proesses with VERY limited statistis (want to disover asap); therefore the limit ("n large", more preisely n >> n) annot be used (neither its onsequenes, like the Gauss pdf); searhes are learly in the "Poisson regime" : large sample and small probability, suh that the expeted number of events ("suesses") be finite; use the Poisson distribution : m e m ( m) = ; = m; σ = m;! therefore, in a searh, two ases : a. the signal does exist : (b+ s) e (b + s) = b + s; ( b + s) = ;! σ = b+ s; [two possibilities : s known/unknown] b. the signal does OT exist : b e b ( b) = ; = b; σ = b;! the strategy is : use (observed) to distinguish between ase (a) and (b); sine is positive for in [0, ] in both ases, the proedure is to define a CL a priori, and aept the hypothesis (a or b) only if it falls in the predefined interval; modern (LHC) evolution : define a parameter, usually alled "µ" : (b+µ s) e (b + µ s) = b + µ s; ( b + µ s) = ;! σ = b+ µ s; learly, µ = 0 is bkgd only, while µ = 1 means disovery; sometimes results are presented as limits on "µ" [e.g. exlude µ = 0 means "disovery"]. Paolo Bagnaia FS 2 11 11
4/9 limits : disovery, exlusion the "rule" on the CL usually aepted by experiments is: experiment, with exatly the expeted number of events "]. DISCOVERY : (b only) 2.86 10-7, [alled also "5σ" (1) ]; EXCLUSIO : (s+b) 5 10-2 ; [alled also "95% CL"]; a priori, the integrated luminosity L int for disovery / exlusion an be omputed : L dis : L int min, suh that 50% of the experiments (2) (i.e. an experiment in 50% of the times) had (b only) "5σ"; (1) this probability orresponds to 5σ for a gaussian pdf only; but the experimentalists use (always) the ut in probability and (sometimes) all it "5σ"; (2) for ombined studies an "experiment" at LEP [LHC] results from the data of all 4 [2] ollaborations; in this ase L int 4(2) L int. L exl : L int min, suh that 50% of the experiments (2) (i.e. an experiment in 50% of the times) had (s+b) "2σ"; B : this rule orresponds to the median ["an experiment, in 50% of the times "], and it is different from the average ["an Paolo Bagnaia FS 2 11 12
5/9 limits: flowhart m signal (theory for many values of the parameters θ k, σ, dσ/dosθ, final state partiles, ) m bkgd (σ, dσ/dosθ, final state partiles, ) idential!!! detetor m (response, resolution, failures, ) detetor m ( ) analysis : optimization of uts/seletion to maximize signal visibility (e.g. s/ b) [sometimes the seletion is funtion of free parameters θ k (e.g. m H ) or L int ]. optimal seletion one-way only real data sensitivity disovery θ k??? limit on θ k Paolo Bagnaia FS 2 11 13
6/9 limits : Luminosity of disovery, exlusion b b+s ut 0.5 2.86 10-7 i assume inreasing luminosity (L int = L dis [L exl ]) and onstant ε s, ε b, σ s, σ b ; assume to start with small L int : the two distributions overlap a lot, no satisfies the system (i.e. the tail above the median is large); when L int inreases, the two distributions are more and more distint (overlap 1/ L int ); The values of L dis and L exl ome from the previous equations; ompute L dis (L exl is similar) : " " = e ( b) i b 7 (5 σ) = 2. 86 10 ; i! i= ( b s) ( b + s) + " " = e i= i! b = L εσ ; s = L dis B B i ε 0.5; σ dis S S. for a given value of L int, it exists a number of events, suh that the uts at 2.86 10-7 (0.5) in the first (seond) umulative oinide; this value of L int orrespond to L dis ; this is the luminosity when, if the signal exists, 50% of the experiments have (at least) 5σ inompatibility with the hypothesis of bkgd only. Paolo Bagnaia FS 2 11 14
7/9 limits : ex. m H (b=0, n=0) n s(m H ) = σ s (m H ) L int ε s [theory + m detetor + analysis] just an example, not an atual plot nothing observed: =n=0 limit @ 95% CL (=0 n) 1-CL n - ln (1-CL) n 2.996 3 3 95% exlusion m limit m H Paolo Bagnaia FS 2 11 15
8/9 limits : ex. m H (a priori, b>0) n just an example, not an atual plot s+b = L int [σ s (m H ) ε s + σ b ε b ] B : ε s and ε b may be funtions of m H, or not ( mass independent seletion ). n b = L int σ b ε b limit @ 95% CL Σ n j=0 (j n) 1-CL 0 3.00 1 4.74 2 6.30 3 7.75 n+ >1 1.96 n expeted exlusion @ 95% CL m limit m H Paolo Bagnaia FS 2 11 16
9/9 limits : ex. m H (a posteriori, b>0) n just an example, not an atual plot andidate events (resolution inluded) B the result may be larger or smaller than expetation (m limit is a statistial variable). limit @ 95% l m limit m H Paolo Bagnaia FS 2 11 17
1/5 maximum likelihood: definition A random variable x follows a "probability density funtion" (pdf) ƒ(x θ k ); the parameters θ k (k = 1,,M), sometimes unknown, define ƒ; assume a repeated measurement ( times) of x : x j ( j = 1,,); define the likelihood Λ and its logarithm ln(λ) (see box). Example : observe deays with (unknown) lifetime τ, measuring the lives t j, j = 1,,. 1 t/ τ 1 j t/ j τ Λ= ƒ(t τ ) = e = e ; τ τ j j= 1 j= 1 1 t/ τ 1 j n( Λ= ) n[ e ] = n( τ ) t j. τ τ Λ= ƒ(x θ ); j= 1 n( Λ= ) n[ƒ(x θ )]. j= 1 j= 1 j= 1 j j k k then look for the value τ*, whih maximizes Λ (or lnλ). τ* is the max.lik. estimate of τ. n( Λ ) 1 = 0 = + 2 tj τ τ* τ* 1 τ * = tj =< t >. j= 1 j= 1 Paolo Bagnaia FS 2 11 18
2/5 maximum likelihood: parameter estimate the m.l. method has the following important properties [no dem., see the referenes] : asymptotially onsistent; asymptotially no-bias; the results θ* is asymptotially distributed around θ t, with a variane oinident with the Cramér-Frehet- Rao limit; "invariant" for a hange of parameters, i.e. the m.l. estimate of a quantity, funtion of the parameters, is given by the funtion of the estimates [e.g. (θ 2 )* = (θ*) 2 ]; suh a value is also asymptotially nobias; popular wisdom : "the m.l. method is like a Rolls-Roye: expensive, but the best". B. "asymptotially" means : the onsidered property is valid in the limit meas ; if is finite, the property is OT valid anymore; sometimes the physiists show some "lak of rigor" (say). Paolo Bagnaia FS 2 11 19
3/5 maximum likelihood: hypothesis test The likelihood funtion [PDG] is the produt of the pdf for eah event, alulated for the observed values; for searhes, it is the Poisson probability for observing events times the pdf of eah single event (see box); sine there are two hypotheses (b only and b+s), there are two pdf s and therefore two likelihoods; both are funtions of the parameter(s) of the phenomenon under study (m H ); the likelihood ratio is a powerful (atually the most powerful) test between two hypotheses, mutually exlusive; the term -2 ln is there only for onveniene [both for omputing and beause -2ln(Λ) χ 2 for n large]; in the box : Λ =1, C refers to different hannels to be ombined, if any [e.g. H ZZ, γγ]; ƒ s,b are the pdf (usually from m) of the kinematial variables x for event j : s b s ƒ (x ) + b ƒ ( x) given ƒ (x), ƒ (x), ƒ (x) : s b b+ s = s + b ( b + s ) C poisson Λ =Λ = s s(m H) + ; b s = 1 ƒ ( x j ) j = 1 ( b ) C poisson =Λ (m ) = ; = b 1 ƒ( x j ) j = 1 b b H Λ S 2 nq = 2 n = 2( n Λ B n Λ S). ΛB Paolo Bagnaia FS 2 11 20
4/5 maximum likelihood: m H at LEP (1) A "simple" appliation of the likelihood ratio for the Higgs mass at LEP2 : "hannel ", =1,,C : one experiment one s one final state [e.g. (L3) ( s = 204 GeV) (e + e HZ bb µ + µ )] (atually C > 100); "m = m H, test mass" : the mass under study ("the hypothesis"), whih must be aepted/rejeted (a sansion in mass, with interval ~ mass resolution); for eah hannel and eah test mass, (in priniple) a different analysis, i.e. a different value of {σ S, σ B, ε S, ε B, L, n},m [L ε S σ S s,m, L ε B σ B b,m, both ƒ(m H )]; for eah hannel and eah test mass, a different set of andidates (= events surviving the uts) (j=1,, ); event j has a set of kinematial variables (4- momenta of partiles, seondary vertees) x j [observables of event j of hannel ]; the m samples (both signal and bkgd) allows to define ƒ S,m(x) and ƒ B,m(x), the pdf for signal and bkgd of all the variables, after uts and fits; define all the appropriate variables (e.g. reonstruted mass m j, seondary vertex probability, define also the total number of andidates for a test mass M m,m ; notie that, generally speaking, all variables are orrelated [e.g. m j = m jm = m j (m H ), beause effiieny, uts and fits do depend on m H ]. Paolo Bagnaia FS 2 11 21
5/5 maximum likelihood: m H at LEP (2) and therefore [one again, remember that everything is an impliit funtion of the test mass m H ]. ( + ) + e s b n S B (s b ) n j j Λ S = = = 1 n! j= 1 s + b = e n! (s + b ) = 1 j= 1 n B b ƒ (x j) j= 1 s ƒ (x ) + b ƒ (x ) n S B s ƒ (x j) + b ƒ (x j) ; b n n e b B Λ B = ƒ (x j) = = 1 n! j= 1 b e = = 1 n! ; e (s + b ) n S B s ƒ (x j) + b ƒ (x j) S = 1 n! Λ j= 1 2 nq = 2 n = 2n = b n ΛB e B b ƒ (x j) 1 n! = j= 1 n S s ƒ (x j) = 2 s 2 n 1+ B. = 1 j= 1 bƒ( x j) = 1 Paolo Bagnaia FS 2 11 22
1/3 interpretation of results : disovery plot The likelihood is expeted to be larger when the orret pdf is used; this is easily onfirmed by the large n limit [ -2ln(Λ) χ 2 ] : ( Λ Λ ) 2 2 2nQ = 2n χ χ = s b s b -2lnQ b "small" - << 0 "large" = "large" - >> 0 "small" if (s+b) ok if (b) ok ; real data m H the plot is a little artoon of an ideal situation (e.g. Higgs searh at LEP2), that never happened : s+b good separation bad " the ross-setion dereases when m H inreases (TRUE); disovery!!! (FALSE). Paolo Bagnaia FS 2 11 23
2/3 interpretation of results: parameter µ put : σ = σ b + µ σ SMs ; plot : horizontal : m H. vertial : µ (= σ obs s / σ SMs ) ; the lines show, with a given L int and analysis, the expeted limit (--), and the atual observed limit ( ), i.e. the µ value exluded at 95% CL; the band ( ) shows the flutuations at ±1σ (±2σ) of the "bkgd only" hypothesis. B : the ase µ 1 has no well-defined physial meaning (= a theory idential to the SM, but with a saled ross setion); if the lines are at µ > 1, the "distane" respet to µ=1 reflets the L int neessary to get the limit in the SM. 130 170 140 500 in this hypothetial ase, the region 140 < m H < 170 GeV is exluded at 95% CL, while the expeted limit was 130 500 GeV (either bad luk or hint of disovery). Paolo Bagnaia FS 2 11 24
3/3 interpretation of results: p-value ƒ Q exp (H 0 ) ƒ B (Q H 0 ) Q obs p Q the p-value is the probability to get the same result or another less probable, in the hypothesis of bkgd only. x = statistis (e.g. likelihood ratio); H 0 = "null hypothesis (i.e. bkgd only); i.e. p ƒ(x H 0)dx obs x p small H 0 OT probable disovery!!! vertial : p-value; horizontal : m H. the band ( ) shows the flutuations at 1σ (2σ). B the disovery orresponds to the red line below 5σ (or 2.86 10-7 ), not shown in this fake plot. Paolo Bagnaia FS 2 11 25
Referenes 1. lassi textbook : Eadie et al., Statistial methods in experimental physis; 2. modern textbook : Cowan, Statistial data analysis; 3. simple, for experimentalists : Cranmer, Pratial Statistis for the LHC, [arxiv:1503.07622v1]; (also CER Aademi Training, Feb 2-5, 2009); 4. bayesian : G.D'Agostini, YR CER-99-03. 5. statistial proedure for Higgs : A.L.Read, J. Phys. G: ul. Part. Phys. 28 (2002) 2693; 6. mass limits : R.Cousins, Am.J.Phys., 63 (5), 398 (1995). 7. [PDG has everything, but very onise] bells are related to dramati events even outside partile physis Paolo Bagnaia FS 2 11 26
End End of hapter 11 Paolo Bagnaia FS 2 11 27