Statistics for Particle Physics. Kyle Cranmer. New York University. Kyle Cranmer (NYU) CERN Academic Training, Feb 2-5, 2009
|
|
- Randell Carson
- 5 years ago
- Views:
Transcription
1 Statistics for Particle Physics Kyle Cranmer New York University 91
2 Remaining Lectures Lecture 3:! Compound hypotheses, nuisance parameters, & similar tests! The Neyman-Construction (illustrated)! Inverted hypothesis tests: A dictionary for limits (intervals)! Coverage as a calibration for our statistical device! The likelihood principle, and the logic for likelihood-based methods Lecture 4:! Systematics, Systematics, Systematics! Generalizing our procedures to include systematics! Eliminating nuisance parameters: profiling and marginalization! Introduction to ancillary statistics & conditioning! High dimensional models, Markov Chain Monte Carlo, and Hierarchical Bayes! The look elsewhere e"ect and false discovery rate 92
3 Lecture 3 93
4 Addition to the References Flip-flopping Talking with Fred James over lunch, he mentioned Gary Feldman s lectures: Journeys of an Accidental Statistician How might a typical physicist use these plots? (1) If the result x < 3!, I will quote an upper limit. (2) If the result x > 3!, I will quote central confidence interval. (3) If the result x < 0, I will pretend I measured zero. This results in the following: 10% 5% In the range 1.36 " µ " 4.28, there is only 85% coverage! Due to flip-flopping (deciding whether to use an upper limit or a central confidence region based on the data) these are not valid confidence intervals. Gary Feldman 11 Journeys 94
5 Simple vs. Compound Hypotheses The Neyman-Pearson lemma is the answer for simple hypothesis testing! a hypothesis is simple if it has no free parameters and is totally fixed f(x H 0 ) vs. f(x H 1 ) What about cases when there are free parameters?! eg. the mass of the Higgs boson f(x H 0 ) vs. f(x H 1,m H ) A test is called similar if it has size parameters for all values of the A test is called Uniformly Most Powerful if it maximizes the power for all values of the parameter Uniformly Most Powerful tests don t exist in general α 95
6 Similar Test Examples In some cases Uniformly Most Powerful tests do exist:! some examples just to clarify the concept:! H0 is simple: a Gaussian with a fixed µ = µ 0, σ = σ 0! H1 is composite: a Gaussian with µ<µ 0, σ = σ 0 # consider H - and H-- # same size, di"erent power, but both max power H- H0 H-- H0
7 Similar Test Examples In some cases Uniformly Most Powerful tests exists:! some examples just to clarify the concept:! H0 is simple: a Gaussian with a fixed µ = µ 0, σ = σ 0! H1 is composite: a Gaussian with µ>µ 0, σ = σ 0 # consider H + and H++ # same size, di"erent power, but both max power H0 H+ H0 H++ 97
8 Similar Test Examples Slight variation, a Uniformly Most Powerful test doesn t exit:! some examples just to clarify the concept:! H0 is simple: a Gaussian with a fixed µ = µ 0, σ = σ 0! H1 is composite: a Gaussian with # Either H + has good power and H- has bad power # or vice versa µ = µ 0, σ σ 0 H- H0 H+ H- H0 H+ 98
9 Similar Test Examples Another example that is Uniformly Most Powerful:! H0 is simple: a Gaussian with a fixed µ = µ 0, σ = σ 0! H1 is composite: a Gaussian with µ = µ 0, σ > σ 0 # consider H + and H++ # same size, di"erent power, but both max power H0 H+ 99
10 Similar Test Examples Another example that is Uniformly Most Powerful:! H0 is simple: a Gaussian with a fixed µ = µ 0, σ = σ 0! H1 is composite: a Gaussian with µ = µ 0, σ > σ 0 # consider H + and H++ # same size, di"erent power, but both max power H0 H++ 99
11 Composite Hypothese & the Likelihood Function When a hypothesis is composite typically there is a pdf that can be parametrized f( x θ)! for a fixed θ it defines a pdf for the random variable x! for a given measurement of x one can consider f( x θ) as a function of θ called the Likelihood function! Note, this is not Bayesian, because it still only uses P(data theory) and # the Likelihood function is not a pdf! Sometimes θ has many components, generally divided into:! parameters of interest: eg. masses, cross-sections, etc.! nuisance parameters: eg. parameters that a"ect the shape but are not of direct interest (eg. energy scale) 100
12 A simple example: A Poisson distribution describes a discrete event count n for a real-valued mean $. P ois(n µ) =µ n e µ n! The likelihood of $ given n is the same equation evaluated as a function of $! Now it s a continuous function! But it is not a pdf! Common to plot the -2 log L! why? more later L(µ) =P ois(n µ)!"#$%&'(%)*'+,'-)$."/.0''''''''''''' 1*,'2,'345.,'67'789':;88<= 101
13 Confidence Interval What is a Confidence Interval?! you see them all the time: 80.5 LEP1 and SLD LEP2 and Tevatron (prel.) 68% CL m W [GeV] α m H [GeV] m t [GeV] 102
14 Confidence Interval What is a Confidence Interval?! you see them all the time: 80.5 LEP1 and SLD LEP2 and Tevatron (prel.) 68% CL Want to say there is a 68% chance that the true value of (mw, mt) is in this interval m W [GeV] α m H [GeV] m t [GeV] 102
15 Confidence Interval What is a Confidence Interval?! you see them all the time: 80.5 LEP1 and SLD LEP2 and Tevatron (prel.) 68% CL Want to say there is a 68% chance that the true value of (mw, mt) is in this interval m W [GeV] α m H [GeV] m t [GeV] 102
16 Confidence Interval What is a Confidence Interval?! you see them all the time: 80.5 LEP1 and SLD LEP2 and Tevatron (prel.) 68% CL Want to say there is a 68% chance that the true value of (mw, mt) is in this interval! but that s P(theory data)! m W [GeV] α m H [GeV] m t [GeV] 102
17 Confidence Interval What is a Confidence Interval?! you see them all the time: 80.5 LEP1 and SLD LEP2 and Tevatron (prel.) 68% CL Want to say there is a 68% chance that the true value of (mw, mt) is in this interval! but that s P(theory data)! Correct frequentist statement is that the interval covers the true value 68% of the time m W [GeV] α m H [GeV] m t [GeV] 102
18 Confidence Interval What is a Confidence Interval?! you see them all the time: 80.5 LEP1 and SLD LEP2 and Tevatron (prel.) 68% CL Want to say there is a 68% chance that the true value of (mw, mt) is in this interval! but that s P(theory data)! Correct frequentist statement is that the interval covers the true value 68% of the time! remember, the contour is a function of the data, which is random. So it moves around from experiment to experiment m W [GeV] α m H [GeV] m t [GeV] 102
19 Confidence Interval What is a Confidence Interval?! you see them all the time: 80.5 LEP1 and SLD LEP2 and Tevatron (prel.) 68% CL Want to say there is a 68% chance that the true value of (mw, mt) is in this interval m W [GeV] 80.4! but that s P(theory data)! Correct frequentist statement is that the interval covers the true value 68% of the time! remember, the contour is a function of the data, which is random. So it moves around from experiment to experiment 80.3 α m H [GeV] m t [GeV]! Bayesian credible interval does mean probability parameter is in interval. The procedure is very intuitive: P (θ V )= V π(θ x) = V dθ f(x θ)π(θ) dθf(x θ)π(θ) 102
20 Inverting Hypothesis Tests There is a precise dictionary that explains how to move from from hypothesis testing to parameter estimation.! Type I error: probability interval does not cover true value of the parameters (eg. it is now a function of the parameters)! Power is probability interval does not cover a false value of the parameters (eg. it is now a function of the parameters) # We don t know the true value, consider each point θ 0 as if it were true What about null and alternate hypotheses?! when testing a point it is considered the null! all other points considered alternate So what about the Neyman-Pearson lemma & Likelihood ratio?! as mentioned earlier, there are no guarantees like before! a common generalization that has good power is: f(x H 0 ) f(x H 1 ) θ 0 f(x θ 0 ) f(x θ best (x)) 103
21 The Dictionary From Kendall: 104
22 Extending a model We can extend our simple number counting example P (n H 0 )=P ois(n b) Probability ±10 50 Events Events Observed P (n H 1 )=P ois(n s + b) 105
23 Extending a model We can extend our simple number counting example P (n H 0 )=P ois(n b) Probability ±10 50 Events P (n H 1 )=P ois(n s + b) Events Observed into a parameter estimation in a more general problem P (n µ) =P ois(n µs + b) H 0 : µ =0 H 1 : µ 0 or H 1 : µ =1 105
24 Extending a model We can extend our simple number counting example P (n H 0 )=P ois(n b) Probability ±10 50 Events P (n H 1 )=P ois(n s + b) Events Observed into a parameter estimation in a more general problem P (n µ) =P ois(n µs + b) H 0 : µ =0 H 1 : µ 0 or H 1 : µ =1 Discovery corresponds to the 5σ confidence interval for µ 105
25 Extending a model We can extend our simple number counting example P (n H 0 )=P ois(n b) Probability into a parameter estimation in a more general problem P (n µ) =P ois(n µs + b) H 0 : µ =0 H 1 : µ 0 or H 1 : µ =1 Discovery corresponds to the 5σ confidence interval for µ not including µ =0 ±10 50 Events P (n H 1 )=P ois(n s + b) Events Observed 105
26 Now let s take on Feldman-Cousins µ x 106
27 Neyman Construction example For each value of θ consider f(x θ) f(x θ) θ θ 2 θ 1 θ 0 x 107
28 Neyman Construction example Let s focus on a particular point f(x θ o ) f(x θ 0 ) x 108
29 Neyman Construction example Let s focus on a particular point f(x θ o )! we want a test of size α! equivalent to a 100(1 α)% confidence interval on θ! so we find an acceptance region with 1 α probability f(x θ 0 ) 1 α x 109
30 Neyman Construction example Let s focus on a particular point f(x θ o )! No unique choice of an acceptance region! here s an example of a lower limit f(x θ 0 ) 1 1 α α α x 110
31 Neyman Construction example Let s focus on a particular point f(x θ o )! No unique choice of an acceptance region! and an example of a central limit f(x θ 0 ) 1 α α/2 x 111
32 Neyman Construction example Let s focus on a particular point f(x θ o )! choice of this region is called an ordering rule! In Feldman-Cousins approach, ordering rule is the likelihood ratio. Find contour of L.R. that gives size α f(x θ 0 ) 1 α f(x θ 0 ) f(x θ best (x)) = k α x 112
33 Neyman Construction example Now make acceptance region for every value of f(x θ) θ θ θ 2 θ 1 θ 0 x 113
34 Neyman Construction example This makes a confidence belt for θ f(x θ) θ θ 2 θ 1 θ 0 x 114
35 Neyman Construction example This makes a confidence belt for the regions of data in the confidence belt can be considered as consistent with that value of θ θ θ θ 0 x 115
36 Neyman Construction example Now we make a measurement the points θ where the belt intersects x 0 a part of the confidence interval in θ for this measurement eg. [θ, θ + ] θ x 0 θ + θ x 0 x 116
37 Neyman Construction example Because the value of is random, so is the confidence interval [θ, θ + ]. However, the interval has probability 1 α x 0 to cover the true value of θ. θ θ + θ x 0 x 117
38 A Point about the Neyman Construction This is not Bayesian... it doesn t mean the probability that the true value of is in the interval is 1 α! θ θ θ true θ + θ x 0 x 118
39 A Joke Maybe it s funnier this time? Bayesians address the question everyone is interested in, by using assumptions no-one believes Frequentists use impeccable logic to deal with an issue of no interest to anyone - P. G. Hamer 119
40 From the archives 120
41 The Dictionary (again) Showing this again to reinforce the point that there is a formal 1-to-1 mapping between hypothesis tests and confidence intervals:! some refer to the Neyman Construction as an inverted hypothesis test 121
42 Feldman-Cousins (again) µ x 122
43 Flip-flopping One of the features of Feldman- Cousins is that it provides a unified method for upperlimits and ~central confidence intervals. Flip-flopping How might a typical physicist use these plots? (1) If the result x < 3!, I will quote an upper limit. (2) If the result x > 3!, I will quote central confidence interval. (3) If the result x < 0, I will pretend I measured zero. This results in the following: There is a some of debate about how important this flip-flopping problem is and how satisfactory the unified limits are, but flipflopping is definitely important and the Feldman-Cousins approach avoids the problem.! see phystat, the F-C paper, or Feldman s lectures for more 10% 5% In the range 1.36 " µ " 4.28, there is only 85% coverage! Due to flip-flopping (deciding whether to use an upper limit or a central confidence region based on the data) these are not valid confidence intervals. Gary Feldman 11 Journeys 123
44 Coverage Coverage is the probability that the interval covers the true value. Methods based on the Neyman-Construction always cover... by construction.! sometimes they over-cover (eg. conservative ) Bayesian methods, do not necessarily cover! but that s not their goal.! but that also means you shouldn t interpret a 95% Bayesian Credible Interval in the same way Coverage can be thought of as a calibration of our statistical apparatus. [explain under-/over-coverage] 124
45 Discrete Problems In discrete problems (eg. number counting analysis with counts described by a Poisson) one sees:! discontinuities in the coverage (as a function of parameter)! over-coverage (in some regions)! Important for experiments with few events. There is a lot of discussion about this, not focusing on it here 125
46 Coverage Coverage can be di"erent at each point in the parameter space Coverage coverage Example: G. Punzi - PHYSTAT 05 - Oxford, UK Max coverage ε µ Max 126
47 Another point about the construction Note, that the confidence belt is constructed before we have any data. In some sense, the inference is influenced by data that we didn t get. θ θ + θ x 0 x 127
48 The Likelihood Principle Likelihood Principle As noted above, in both Bayesian methods and likelihood-ratio based methods, the probability (density) for obtaining the data at hand is used (via the likelihood function), but probabilities for obtaining other data are not used! In contrast, in typical frequentist calculations (e.g., a p-value which is the probability of obtaining a value as extreme or more extreme than that observed), one uses probabilities of data not seen. This difference is captured by the Likelihood Principle*: If two experiments yield likelihood functions which are proportional, then Your inferences from the two experiments should be identical. L.P. is built in to Bayesian inference (except e.g., when Jeffreys prior leads to violation). L.P. is violated by p-values and confidence intervals. Although practical experience indicates that the L.P. may be too restrictive, it is useful to keep in mind. When frequentist results make no sense or are unphysical, in my experience the underlying reason can be traced to a bad violation of the L.P. *There are various versions of the L.P., strong and weak forms, etc. Bob Cousins, CMS,
49 Probability density Building the distribution of the test statistic In the case of LEP Higgs: (a) Q = L(x H 1) L(x H 0 ) = Nchan LEP Observed m H = 115 GeV/c 2 Expected for background Expected for signal plus background i P ois(n i s i + b i ) n i Nchan N chan q = ln Q = s tot -2 ln(q) -2 ln(q) i i j s i f s (x ij )+b i f b (x ij ) s i +b i P ois(n i b i ) n i j f b (x ij ) ( ln 1+ s ) if s (x ij ) b i f b (x ij ) n i j Observed Expected for background Expected for signal plus background LEP m (GeV/c 2 ) 129
50 Building the distribution of the test statistic!!!!!!!!!!!! LEP Higgs Working group developed formalism to combine channels and take advantage of discriminating variables in the likelihood ratio.! b b! s+b s+b Q = L(x H 1) L(x H 0 ) = sf s (x) sf bf s (x) (x) b bf (x) b f b(x) q(x)=log(1+ ) f b(x) q(x)=log(1+ ) f (x) s f (x) s f (x) 1,b (q(x))=! 1,s (q) b!! (q) 1,b (q(x))= f (x) 1,s b CL CL b b s! b b! (q)! 1,b (q) 1,b! s+b s+b Nchan i P ois(n i s i + b i ) n i Nchan N chan q = ln Q = s tot L!s L FFT FFT s ~(s+b)l!s ~(s+b)l FFT 1 FFT!1! (q) 1,b! (q) 1,b! (q) 1,s! (q) 1,s = exp[b(! 1)]! b = exp[b( 1,b!!1)] b 1,b = exp[b(! 1) + s(! 1)]! s+b = exp[b( 1,b!!1) + s( 1,s!!1)] s+b! b b! s+b s+b 1,b + i i 1,s j s i f s (x ij )+b i f b (x ij ) s i +b i P ois(n i b i ) n i j f b (x ij ) ( ln 1+ s ) if s (x ij ) b i f b (x ij ) n i j Hu and Nielsen s CLFFT used Fourier Transform and exponentiation trick to transform the log-likelihood ratio distribution for one event to the distribution for an experiment Cousins-Highland was used for systematic error on background rate. Getting this to work at the LHC is tricky numerically because we have channels with n i from events (physics/ ) 130
51 Building the distribution of the test statistic!!!!!!!!!!!! LEP Higgs Working group developed formalism to combine channels and take advantage of discriminating variables in the likelihood ratio. f b(x) q(x)=log(1+ ) f b(x) q(x)=log(1+ ) f (x) s f (x)! b b! s+b s+b CL CL b b Q = L(x H 1) L(x H 0 ) = s! b b! s+b s+b Nchan i P ois(n i s i + b i ) n i Nchan N chan q = ln Q = s tot L s ~(s+b)l!s ~(s+b)l FFT 1 FFT!1! s+b s+b N=0 i i j s i f s (x ij )+b i f b (x ij ) s i +b i P ois(n i b i ) n i j f b (x ij ) ( ln 1+ s ) if s (x ij ) b i f b (x ij ) n i = exp[b(! 1)]! b = exp[b( 1,b!!1)] b 1,b = exp[b(! 1) + s(! 1)]! s+b = exp[b( 1,b N times!!1) + s( 1,s!!1)] s+b! b b j sf! s (x)! (q) 1,b (q) Hu and Nielsen s CLFFT used Fourier Transform and exponentiation trick to transform sf 1,b bf (x) FFT! s (x)! (q) 1,b (q) b 1,b bf (x) FFT b For N events, use! f (x) (q) 1,s 1,b (q(x))=! 1,s (q) Fourier transform to perform the log-likelihood N convolutions ratio distribution for one b!! s! (q) f (x) (q) 1,b (q(x))= 1,s b 1,s event to the distribution for an experiment!s L Z n o ρ N,i (q) = ρ N,i (q) ρ N,i (q) = F 1 [F (ρ 1,i )] N {z } Cousins-Highland was used for systematic error on background rate. 1,b 1,s To include Poisson fluctuations on N for a given luminosity, one can exponentiate ρ i (q) = X + Getting this to work at the LHC is tricky numerically because we have channels with n i from events (physics/ ) P (N; Lσ i ) ρ N,i (q) = F 1 n e Lσ i[f(ρ 1,i (q)) 1] o 130
52 Examples of Likelihood Analysis In these examples, a model that relates precision electroweak observables to parameters of the Standard Model was used! the inference is based only on the likelihood function for data at hand # there is no prior, so it s not Bayesian. And no Neyman Construction! # what is the meaning of this contour if it s not the Neyman Construction? χ Theory uncertainty α had = α (5) ± ± incl. low Q 2 data m Limit = 144 GeV m W [GeV] LEP1 and SLD LEP2 and Tevatron (prel.) 68% CL 1 Excluded Preliminary m H [GeV] 80.3 α m H [GeV] m t [GeV] 131
53 Logic of Likelihood-based Methods Likelihood-based methods settle between two conflicting desires:! We want to obey the likelihood principle because it implies a lot of nice things and sounds pretty attractive! We want nice frequentist properties (and the only way we know to incorporate those properties by construction will violate the likelihood principle) If we had a way to approximately get the distribution of our test statistic for every value of θ based only f(x θ) on the likelihood function (and no prior) then we would have a θ workable solution! θ 0 θ 1 θ 2 x There is a way to get approximate frequentist results. It s the basis of MINUIT/MINOS. Next Time! 132
54 Systematics, Systematics, Systematics 133
Statistics for Particle Physics. Kyle Cranmer. New York University. Kyle Cranmer (NYU) CERN Academic Training, Feb 2-5, 2009
Statistics for Particle Physics Kyle Cranmer New York University 1 Hypothesis Testing 55 Hypothesis testing One of the most common uses of statistics in particle physics is Hypothesis Testing! assume one
More informationJourneys of an Accidental Statistician
Journeys of an Accidental Statistician A partially anecdotal account of A Unified Approach to the Classical Statistical Analysis of Small Signals, GJF and Robert D. Cousins, Phys. Rev. D 57, 3873 (1998)
More informationStatistics of Small Signals
Statistics of Small Signals Gary Feldman Harvard University NEPPSR August 17, 2005 Statistics of Small Signals In 1998, Bob Cousins and I were working on the NOMAD neutrino oscillation experiment and we
More informationPhysics 403. Segev BenZvi. Credible Intervals, Confidence Intervals, and Limits. Department of Physics and Astronomy University of Rochester
Physics 403 Credible Intervals, Confidence Intervals, and Limits Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Summarizing Parameters with a Range Bayesian
More informationStatistical Methods in Particle Physics Lecture 2: Limits and Discovery
Statistical Methods in Particle Physics Lecture 2: Limits and Discovery SUSSP65 St Andrews 16 29 August 2009 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationPrimer on statistics:
Primer on statistics: MLE, Confidence Intervals, and Hypothesis Testing ryan.reece@gmail.com http://rreece.github.io/ Insight Data Science - AI Fellows Workshop Feb 16, 018 Outline 1. Maximum likelihood
More informationE. Santovetti lesson 4 Maximum likelihood Interval estimation
E. Santovetti lesson 4 Maximum likelihood Interval estimation 1 Extended Maximum Likelihood Sometimes the number of total events measurements of the experiment n is not fixed, but, for example, is a Poisson
More informationStatistics for the LHC Lecture 2: Discovery
Statistics for the LHC Lecture 2: Discovery Academic Training Lectures CERN, 14 17 June, 2010 indico.cern.ch/conferencedisplay.py?confid=77830 Glen Cowan Physics Department Royal Holloway, University of
More informationPractical Statistics for Particle Physics
Practical Statistics for Kyle Cranmer, New York University 1 Lecture 3 98 Outline Lecture 1: Building a probability model preliminaries, the marked Poisson process incorporating systematics via nuisance
More informationLecture 5. G. Cowan Lectures on Statistical Data Analysis Lecture 5 page 1
Lecture 5 1 Probability (90 min.) Definition, Bayes theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests (90 min.) general concepts, test statistics,
More informationStatistical Methods for Particle Physics Lecture 4: discovery, exclusion limits
Statistical Methods for Particle Physics Lecture 4: discovery, exclusion limits www.pp.rhul.ac.uk/~cowan/stat_aachen.html Graduierten-Kolleg RWTH Aachen 10-14 February 2014 Glen Cowan Physics Department
More informationStatistical Methods in Particle Physics
Statistical Methods in Particle Physics Lecture 11 January 7, 2013 Silvia Masciocchi, GSI Darmstadt s.masciocchi@gsi.de Winter Semester 2012 / 13 Outline How to communicate the statistical uncertainty
More informationHypothesis testing (cont d)
Hypothesis testing (cont d) Ulrich Heintz Brown University 4/12/2016 Ulrich Heintz - PHYS 1560 Lecture 11 1 Hypothesis testing Is our hypothesis about the fundamental physics correct? We will not be able
More informationMODIFIED FREQUENTIST ANALYSIS OF SEARCH RESULTS (THE CL s METHOD)
MODIFIED FREQUENTIST ANALYSIS OF SEARCH RESULTS (THE CL s METHOD) A. L. Read University of Oslo, Department of Physics, P.O. Box 148, Blindern, 316 Oslo 3, Norway Abstract The statistical analysis of direct
More informationStatistical Challenges of the LHC. Kyle Cranmer
Statistical Challenges of the LHC Outline: - The state of the art of frequentist hypothesis testing with no systematics - Including systematics: translating confidence intervals to hypothesis testing -
More informationUse of the likelihood principle in physics. Statistics II
Use of the likelihood principle in physics Statistics II 1 2 3 + Bayesians vs Frequentists 4 Why ML does work? hypothesis observation 5 6 7 8 9 10 11 ) 12 13 14 15 16 Fit of Histograms corresponds This
More informationRecent developments in statistical methods for particle physics
Recent developments in statistical methods for particle physics Particle Physics Seminar Warwick, 17 February 2011 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk
More informationHypothesis Testing - Frequentist
Frequentist Hypothesis Testing - Frequentist Compare two hypotheses to see which one better explains the data. Or, alternatively, what is the best way to separate events into two classes, those originating
More informationStatistical Methods in Particle Physics Lecture 1: Bayesian methods
Statistical Methods in Particle Physics Lecture 1: Bayesian methods SUSSP65 St Andrews 16 29 August 2009 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationConfidence Limits and Intervals 3: Various other topics. Roger Barlow SLUO Lectures on Statistics August 2006
Confidence Limits and Intervals 3: Various other topics Roger Barlow SLUO Lectures on Statistics August 2006 Contents 1.Likelihood and lnl 2.Multidimensional confidence regions 3.Systematic errors: various
More informationStatistical Data Analysis Stat 3: p-values, parameter estimation
Statistical Data Analysis Stat 3: p-values, parameter estimation London Postgraduate Lectures on Particle Physics; University of London MSci course PH4515 Glen Cowan Physics Department Royal Holloway,
More informationStatistical Methods for Discovery and Limits in HEP Experiments Day 3: Exclusion Limits
Statistical Methods for Discovery and Limits in HEP Experiments Day 3: Exclusion Limits www.pp.rhul.ac.uk/~cowan/stat_freiburg.html Vorlesungen des GK Physik an Hadron-Beschleunigern, Freiburg, 27-29 June,
More informationSome Topics in Statistical Data Analysis
Some Topics in Statistical Data Analysis Invisibles School IPPP Durham July 15, 2013 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan G. Cowan
More informationTopics in Statistical Data Analysis for HEP Lecture 1: Bayesian Methods CERN-JINR European School of High Energy Physics Bautzen, June 2009
Topics in Statistical Data Analysis for HEP Lecture 1: Bayesian Methods CERN-JINR European School of High Energy Physics Bautzen, 14 27 June 2009 Glen Cowan Physics Department Royal Holloway, University
More informationSome Statistical Tools for Particle Physics
Some Statistical Tools for Particle Physics Particle Physics Colloquium MPI für Physik u. Astrophysik Munich, 10 May, 2016 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk
More informationStatistical Methods for Particle Physics Lecture 3: systematic uncertainties / further topics
Statistical Methods for Particle Physics Lecture 3: systematic uncertainties / further topics istep 2014 IHEP, Beijing August 20-29, 2014 Glen Cowan ( Physics Department Royal Holloway, University of London
More informationDiscovery significance with statistical uncertainty in the background estimate
Glen Cowan, Eilam Gross ATLAS Statistics Forum 8 May, 2008 Discovery significance with statistical uncertainty in the background estimate Introduction In a search for a new type of event, data samples
More informationConstructing Ensembles of Pseudo-Experiments
Constructing Ensembles of Pseudo-Experiments Luc Demortier The Rockefeller University, New York, NY 10021, USA The frequentist interpretation of measurement results requires the specification of an ensemble
More informationStatistical Methods for Particle Physics Lecture 3: Systematics, nuisance parameters
Statistical Methods for Particle Physics Lecture 3: Systematics, nuisance parameters http://benasque.org/2018tae/cgi-bin/talks/allprint.pl TAE 2018 Centro de ciencias Pedro Pascual Benasque, Spain 3-15
More informationConfidence intervals fundamental issues
Confidence intervals fundamental issues Null Hypothesis testing P-values Classical or frequentist confidence intervals Issues that arise in interpretation of fit result Bayesian statistics and intervals
More informationChapter 3: Interval Estimation
Chapter 3: Interval Estimation 1. Basic Concepts: Probability, random variables, distributions, convergence, law of large numbers, Central Limit Theorem, etc. 2. Point Estimation 3. Interval Estimation
More informationAdvanced Statistics Course Part I
Advanced Statistics Course Part I W. Verkerke (NIKHEF) Wouter Verkerke, NIKHEF Outline of this course Advances statistical methods Theory and practice Focus on limit setting and discovery for the LHC Part
More informationP Values and Nuisance Parameters
P Values and Nuisance Parameters Luc Demortier The Rockefeller University PHYSTAT-LHC Workshop on Statistical Issues for LHC Physics CERN, Geneva, June 27 29, 2007 Definition and interpretation of p values;
More informationIntroductory Statistics Course Part II
Introductory Statistics Course Part II https://indico.cern.ch/event/735431/ PHYSTAT ν CERN 22-25 January 2019 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationConfidence Intervals. First ICFA Instrumentation School/Workshop. Harrison B. Prosper Florida State University
Confidence Intervals First ICFA Instrumentation School/Workshop At Morelia,, Mexico, November 18-29, 2002 Harrison B. Prosper Florida State University Outline Lecture 1 Introduction Confidence Intervals
More informationStatistics for the LHC Lecture 1: Introduction
Statistics for the LHC Lecture 1: Introduction Academic Training Lectures CERN, 14 17 June, 2010 indico.cern.ch/conferencedisplay.py?confid=77830 Glen Cowan Physics Department Royal Holloway, University
More informationParameter estimation and forecasting. Cristiano Porciani AIfA, Uni-Bonn
Parameter estimation and forecasting Cristiano Porciani AIfA, Uni-Bonn Questions? C. Porciani Estimation & forecasting 2 Temperature fluctuations Variance at multipole l (angle ~180o/l) C. Porciani Estimation
More informationMethod of Feldman and Cousins for the construction of classical confidence belts
Method of Feldman and Cousins for the construction of classical confidence belts Ulrike Schnoor IKTP TU Dresden 17.02.2011 - ATLAS Seminar U. Schnoor (IKTP TU DD) Felcman-Cousins 17.02.2011 - ATLAS Seminar
More informationStatistical Methods for Particle Physics (I)
Statistical Methods for Particle Physics (I) https://agenda.infn.it/conferencedisplay.py?confid=14407 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationSearch and Discovery Statistics in HEP
Search and Discovery Statistics in HEP Eilam Gross, Weizmann Institute of Science This presentation would have not been possible without the tremendous help of the following people throughout many years
More informationarxiv: v3 [physics.data-an] 24 Jun 2013
arxiv:07.727v3 [physics.data-an] 24 Jun 203 Asymptotic formulae for likelihood-based tests of new physics Glen Cowan, Kyle Cranmer 2, Eilam Gross 3, Ofer Vitells 3 Physics Department, Royal Holloway, University
More informationAsymptotic formulae for likelihood-based tests of new physics
Eur. Phys. J. C (2011) 71: 1554 DOI 10.1140/epjc/s10052-011-1554-0 Special Article - Tools for Experiment and Theory Asymptotic formulae for likelihood-based tests of new physics Glen Cowan 1, Kyle Cranmer
More informationStatistical Methods for Particle Physics
Statistical Methods for Particle Physics Invisibles School 8-13 July 2014 Château de Button Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationSolution: chapter 2, problem 5, part a:
Learning Chap. 4/8/ 5:38 page Solution: chapter, problem 5, part a: Let y be the observed value of a sampling from a normal distribution with mean µ and standard deviation. We ll reserve µ for the estimator
More informationPractical Statistics part II Composite hypothesis, Nuisance Parameters
Practical Statistics part II Composite hypothesis, Nuisance Parameters W. Verkerke (NIKHEF) Summary of yesterday, plan for today Start with basics, gradually build up to complexity of Statistical tests
More informationSTAT 499/962 Topics in Statistics Bayesian Inference and Decision Theory Jan 2018, Handout 01
STAT 499/962 Topics in Statistics Bayesian Inference and Decision Theory Jan 2018, Handout 01 Nasser Sadeghkhani a.sadeghkhani@queensu.ca There are two main schools to statistical inference: 1-frequentist
More informationStatistics Challenges in High Energy Physics Search Experiments
Statistics Challenges in High Energy Physics Search Experiments The Weizmann Institute of Science, Rehovot, Israel E-mail: eilam.gross@weizmann.ac.il Ofer Vitells The Weizmann Institute of Science, Rehovot,
More informationPhysics 403. Segev BenZvi. Classical Hypothesis Testing: The Likelihood Ratio Test. Department of Physics and Astronomy University of Rochester
Physics 403 Classical Hypothesis Testing: The Likelihood Ratio Test Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Bayesian Hypothesis Testing Posterior Odds
More informationRooStatsCms: a tool for analyses modelling, combination and statistical studies
RooStatsCms: a tool for analyses modelling, combination and statistical studies D. Piparo, G. Schott, G. Quast Institut für f Experimentelle Kernphysik Universität Karlsruhe Outline The need for a tool
More informationUnified approach to the classical statistical analysis of small signals
PHYSICAL REVIEW D VOLUME 57, NUMBER 7 1 APRIL 1998 Unified approach to the classical statistical analysis of small signals Gary J. Feldman * Department of Physics, Harvard University, Cambridge, Massachusetts
More information32. STATISTICS. 32. Statistics 1
32. STATISTICS 32. Statistics 1 Revised September 2007 by G. Cowan (RHUL). This chapter gives an overview of statistical methods used in High Energy Physics. In statistics, we are interested in using a
More informationSystematic uncertainties in statistical data analysis for particle physics. DESY Seminar Hamburg, 31 March, 2009
Systematic uncertainties in statistical data analysis for particle physics DESY Seminar Hamburg, 31 March, 2009 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationarxiv: v1 [hep-ex] 9 Jul 2013
Statistics for Searches at the LHC Glen Cowan Physics Department, Royal Holloway, University of London, Egham, Surrey, TW2 EX, UK arxiv:137.2487v1 [hep-ex] 9 Jul 213 Abstract These lectures 1 describe
More informationTwo examples of the use of fuzzy set theory in statistics. Glen Meeden University of Minnesota.
Two examples of the use of fuzzy set theory in statistics Glen Meeden University of Minnesota http://www.stat.umn.edu/~glen/talks 1 Fuzzy set theory Fuzzy set theory was introduced by Zadeh in (1965) as
More informationStatistical Methods in Particle Physics Day 4: Discovery and limits
Statistical Methods in Particle Physics Day 4: Discovery and limits 清华大学高能物理研究中心 2010 年 4 月 12 16 日 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More information32. STATISTICS. 32. Statistics 1
32. STATISTICS 32. Statistics 1 Revised April 1998 by F. James (CERN); February 2000 by R. Cousins (UCLA); October 2001, October 2003, and August 2005 by G. Cowan (RHUL). This chapter gives an overview
More informationStatistical Methods for Particle Physics Lecture 1: parameter estimation, statistical tests
Statistical Methods for Particle Physics Lecture 1: parameter estimation, statistical tests http://benasque.org/2018tae/cgi-bin/talks/allprint.pl TAE 2018 Benasque, Spain 3-15 Sept 2018 Glen Cowan Physics
More informationSecond Workshop, Third Summary
Statistical Issues Relevant to Significance of Discovery Claims Second Workshop, Third Summary Luc Demortier The Rockefeller University Banff, July 16, 2010 1 / 23 1 In the Beginning There Were Questions...
More informationPhysics 509: Bootstrap and Robust Parameter Estimation
Physics 509: Bootstrap and Robust Parameter Estimation Scott Oser Lecture #20 Physics 509 1 Nonparametric parameter estimation Question: what error estimate should you assign to the slope and intercept
More informationHYPOTHESIS TESTING: FREQUENTIST APPROACH.
HYPOTHESIS TESTING: FREQUENTIST APPROACH. These notes summarize the lectures on (the frequentist approach to) hypothesis testing. You should be familiar with the standard hypothesis testing from previous
More informationMODIFIED FREQUENTIST ANALYSIS OF SEARCH RESULTS (THE METHOD)
MODIFIED FREQUENTIST ANALYSIS OF SEARCH RESULTS (THE METHOD) A. L. Read University of Oslo, Department of Physics, P.O. Box 148, Blindern, 316 Oslo 3, Norway Abstract The statistical analysis of direct
More informationThe Jeffreys-Lindley Paradox and Discovery Criteria in High Energy Physics
The Jeffreys-Lindley Paradox and Discovery Criteria in High Energy Physics Bob Cousins Univ. of California, Los Angeles Workshop on Evidence, Discovery, Proof: Measuring the Higgs Particle Univ. of South
More informationStatistical Tools in Collider Experiments. Multivariate analysis in high energy physics
Statistical Tools in Collider Experiments Multivariate analysis in high energy physics Lecture 5 Pauli Lectures - 10/02/2012 Nicolas Chanon - ETH Zürich 1 Outline 1.Introduction 2.Multivariate methods
More informationConditional probabilities and graphical models
Conditional probabilities and graphical models Thomas Mailund Bioinformatics Research Centre (BiRC), Aarhus University Probability theory allows us to describe uncertainty in the processes we model within
More informationCREDIBILITY OF CONFIDENCE INTERVALS
CREDIBILITY OF CONFIDENCE INTERVALS D. Karlen Λ Ottawa-Carleton Institute for Physics, Carleton University, Ottawa, Canada Abstract Classical confidence intervals are often misunderstood by particle physicists
More informationMathematical Statistics
Mathematical Statistics MAS 713 Chapter 8 Previous lecture: 1 Bayesian Inference 2 Decision theory 3 Bayesian Vs. Frequentist 4 Loss functions 5 Conjugate priors Any questions? Mathematical Statistics
More informationIntroduction to Likelihoods
Introduction to Likelihoods http://indico.cern.ch/conferencedisplay.py?confid=218693 Likelihood Workshop CERN, 21-23, 2013 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk
More informationFrequentist Confidence Limits and Intervals. Roger Barlow SLUO Lectures on Statistics August 2006
Frequentist Confidence Limits and Intervals Roger Barlow SLUO Lectures on Statistics August 2006 Confidence Intervals A common part of the physicist's toolkit Especially relevant for results which are
More informationLecture 2. G. Cowan Lectures on Statistical Data Analysis Lecture 2 page 1
Lecture 2 1 Probability (90 min.) Definition, Bayes theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests (90 min.) general concepts, test statistics,
More informationLecture 5: Bayes pt. 1
Lecture 5: Bayes pt. 1 D. Jason Koskinen koskinen@nbi.ku.dk Photo by Howard Jackman University of Copenhagen Advanced Methods in Applied Statistics Feb - Apr 2016 Niels Bohr Institute 2 Bayes Probabilities
More informationETH Zurich HS Mauro Donegà: Higgs physics meeting name date 1
Higgs physics - lecture 4 ETH Zurich HS 2015 Mauro Donegà Mauro Donegà: Higgs physics meeting name date 1 Outline 1 2 3 4 5 6 Introduction Accelerators Detectors EW constraints Search at LEP1 / LEP 2 Statistics:
More informationHarrison B. Prosper. CMS Statistics Committee
Harrison B. Prosper Florida State University CMS Statistics Committee 08-08-08 Bayesian Methods: Theory & Practice. Harrison B. Prosper 1 h Lecture 3 Applications h Hypothesis Testing Recap h A Single
More informationBayesian inference. Fredrik Ronquist and Peter Beerli. October 3, 2007
Bayesian inference Fredrik Ronquist and Peter Beerli October 3, 2007 1 Introduction The last few decades has seen a growing interest in Bayesian inference, an alternative approach to statistical inference.
More informationParameter Estimation. William H. Jefferys University of Texas at Austin Parameter Estimation 7/26/05 1
Parameter Estimation William H. Jefferys University of Texas at Austin bill@bayesrules.net Parameter Estimation 7/26/05 1 Elements of Inference Inference problems contain two indispensable elements: Data
More informationA Calculator for Confidence Intervals
A Calculator for Confidence Intervals Roger Barlow Department of Physics Manchester University England Abstract A calculator program has been written to give confidence intervals on branching ratios for
More informationDealing with Data: Signals, Backgrounds and Statistics
Dealing with Data: Signals, Backgrounds and Statistics Luc Demortier The Rockefeller University TASI 2008 University of Colorado, Boulder, June 5 & 6, 2008 [Version 2, with corrections on slides 12, 13,
More informationarxiv:physics/ v2 [physics.data-an] 16 Dec 1999
HUTP-97/A096 A Unified Approach to the Classical Statistical Analysis of Small Signals arxiv:physics/9711021v2 [physics.data-an] 16 Dec 1999 Gary J. Feldman Department of Physics, Harvard University, Cambridge,
More information32. STATISTICS. 32. Statistics 1
32. STATISTICS 32. Statistics 1 Revised September 2009 by G. Cowan (RHUL). This chapter gives an overview of statistical methods used in high-energy physics. In statistics, we are interested in using a
More informationA Very Brief Summary of Statistical Inference, and Examples
A Very Brief Summary of Statistical Inference, and Examples Trinity Term 2008 Prof. Gesine Reinert 1 Data x = x 1, x 2,..., x n, realisations of random variables X 1, X 2,..., X n with distribution (model)
More informationStatistical Data Analysis Stat 5: More on nuisance parameters, Bayesian methods
Statistical Data Analysis Stat 5: More on nuisance parameters, Bayesian methods London Postgraduate Lectures on Particle Physics; University of London MSci course PH4515 Glen Cowan Physics Department Royal
More informationStatistical Inference: Estimation and Confidence Intervals Hypothesis Testing
Statistical Inference: Estimation and Confidence Intervals Hypothesis Testing 1 In most statistics problems, we assume that the data have been generated from some unknown probability distribution. We desire
More informationLecture 3. G. Cowan. Lecture 3 page 1. Lectures on Statistical Data Analysis
Lecture 3 1 Probability (90 min.) Definition, Bayes theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests (90 min.) general concepts, test statistics,
More informationDiscovery and Significance. M. Witherell 5/10/12
Discovery and Significance M. Witherell 5/10/12 Discovering particles Much of what we know about physics at the most fundamental scale comes from discovering particles. We discovered these particles by
More informationFYST17 Lecture 8 Statistics and hypothesis testing. Thanks to T. Petersen, S. Maschiocci, G. Cowan, L. Lyons
FYST17 Lecture 8 Statistics and hypothesis testing Thanks to T. Petersen, S. Maschiocci, G. Cowan, L. Lyons 1 Plan for today: Introduction to concepts The Gaussian distribution Likelihood functions Hypothesis
More informationStatistical Inference
Statistical Inference Robert L. Wolpert Institute of Statistics and Decision Sciences Duke University, Durham, NC, USA Spring, 2006 1. DeGroot 1973 In (DeGroot 1973), Morrie DeGroot considers testing the
More informationPoS(ACAT2010)057. The RooStats Project. L. Moneta CERN, Geneva, Switzerland K. Belasco. K. S. Cranmer. S.
The RooStats Project CERN, Geneva, Switzerland E-mail: Lorenzo.Moneta@cern.ch K. Belasco Princeton University, USA K. S. Cranmer New York University, USA S. Kreiss New York University, USA A. Lazzaro CERN,
More informationStatistics and Data Analysis
Statistics and Data Analysis The Crash Course Physics 226, Fall 2013 "There are three kinds of lies: lies, damned lies, and statistics. Mark Twain, allegedly after Benjamin Disraeli Statistics and Data
More informationComputing Likelihood Functions for High-Energy Physics Experiments when Distributions are Defined by Simulators with Nuisance Parameters
Computing Likelihood Functions for High-Energy Physics Experiments when Distributions are Defined by Simulators with Nuisance Parameters Radford M. Neal Dept. of Statistics, University of Toronto Abstract
More informationIrr. Statistical Methods in Experimental Physics. 2nd Edition. Frederick James. World Scientific. CERN, Switzerland
Frederick James CERN, Switzerland Statistical Methods in Experimental Physics 2nd Edition r i Irr 1- r ri Ibn World Scientific NEW JERSEY LONDON SINGAPORE BEIJING SHANGHAI HONG KONG TAIPEI CHENNAI CONTENTS
More informationFirst two sided limit on BR(B s μ + μ - ) Matthew Herndon, University of Wisconsin Madison SUSY M. Herndon, SUSY
First two sided limit on BR(B s μ + μ - ) Matthew Herndon, University of Wisconsin Madison SUSY 2011 M. Herndon, SUSY 2011 1 B s(d) μ + μ - Beyond the SM Indirect searches for new physics Look at processes
More informationIs there evidence for a peak in this data?
Is there evidence for a peak in this data? 1 Is there evidence for a peak in this data? Observation of an Exotic S=+1 Baryon in Exclusive Photoproduction from the Deuteron S. Stepanyan et al, CLAS Collab,
More informationStatistics notes. A clear statistical framework formulates the logic of what we are doing and why. It allows us to make precise statements.
Statistics notes Introductory comments These notes provide a summary or cheat sheet covering some basic statistical recipes and methods. These will be discussed in more detail in the lectures! What is
More informationCMS Internal Note. The content of this note is intended for CMS internal use and distribution only
Available on CMS information server CMS IN 2003/xxxx CMS Internal Note The content of this note is intended for CMS internal use and distribution only August 26, 2003 Expected signal observability at future
More informationIntroduction to Statistical Methods for High Energy Physics
Introduction to Statistical Methods for High Energy Physics 2011 CERN Summer Student Lectures Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationStatistical Methods for Astronomy
Statistical Methods for Astronomy If your experiment needs statistics, you ought to have done a better experiment. -Ernest Rutherford Lecture 1 Lecture 2 Why do we need statistics? Definitions Statistical
More informationOVERVIEW OF PRINCIPLES OF STATISTICS
OVERVIEW OF PRINCIPLES OF STATISTICS F. James CERN, CH-1211 eneva 23, Switzerland Abstract A summary of the basic principles of statistics. Both the Bayesian and Frequentist points of view are exposed.
More informationIntroduction to Bayesian Methods
Introduction to Bayesian Methods Jessi Cisewski Department of Statistics Yale University Sagan Summer Workshop 2016 Our goal: introduction to Bayesian methods Likelihoods Priors: conjugate priors, non-informative
More informationConfidence intervals and the Feldman-Cousins construction. Edoardo Milotti Advanced Statistics for Data Analysis A.Y
Confidence intervals and the Feldman-Cousins construction Edoardo Milotti Advanced Statistics for Data Analysis A.Y. 2015-16 Review of the Neyman construction of the confidence intervals X-Outline of a
More informationRooFit/RooStats Analysis Report
RooFit/RooStats Analysis Report D. Hayden Department of Physics Royal Holloway University of London Supervisor: Dr Tracey Berry December 9, 2009 Abstract I have started to use a new and highly flexible
More informationLECTURE 10: NEYMAN-PEARSON LEMMA AND ASYMPTOTIC TESTING. The last equality is provided so this can look like a more familiar parametric test.
Economics 52 Econometrics Professor N.M. Kiefer LECTURE 1: NEYMAN-PEARSON LEMMA AND ASYMPTOTIC TESTING NEYMAN-PEARSON LEMMA: Lesson: Good tests are based on the likelihood ratio. The proof is easy in the
More information