Journeys of an Accidental Statistician
|
|
- Lawrence McCormick
- 5 years ago
- Views:
Transcription
1 Journeys of an Accidental Statistician A partially anecdotal account of A Unified Approach to the Classical Statistical Analysis of Small Signals, GJF and Robert D. Cousins, Phys. Rev. D 57, 3873 (1998) and further unpublished work. Gary Feldman 1 Journeys
2 A Simple Problem Suppose you are searching for a rare process and have a well-known expected background of 3 events, and you observe 0 events. What 90% confidence limit can you set on the unknown rate for this rare process? A classical (or frequentist) statistician makes a statement about the probability of data given theory. That is, given a hypothesis for the value of an unknown true value µ, he or she will give you the probability of obtaining a set of data x, P(x µ). A classical confidence interval (Jerzy Neyman, 1937) is a statement of the form: The unknown true value of µ lies in the region [µ 1,µ 2 ]. If this statement is made at the 90% confidence level, then it will be true 90% of the time, and false 10% of the time. Poisson statistics P(x = 0 µ = 2.3) = 0.1. Therefore, in the standard classical approach, µ < 2.3 at 90% C.L. Since µ = s + b, and b = 3.0, s < -0.7 at 90% C.L. Thus, we are led to a statement that we know is a priori false. Gary Feldman 2 Journeys
3 Bayesian Statistics A Bayesian takes the opposite position from a classical statistician. He or she calculates the probability of theory given data. That is, given a set of data x, he or she will calculate the probability that the unknown true value is µ, P(µ x). This appears attractive because it is what you really want to know. However, it comes at a price: P(x µ) and P(µ x) are related by Bayes Theorem, which in set theory is the statement that an element is in both A and B is P(A B) P(B) = P(B A) P(A) which for probabilities becomes P(µ x) = P(x µ) P(µ)/P(x). P(x) is just a normalization term, but Bayes Theorem transforms P(µ), the prior distribution of degree of belief in µ, to P(µ x), the posterior distribution. A credible interval or Bayesian confidence interval is µ 2 formed by P(µ x) dµ = 90%. µ 1 Gary Feldman 3 Journeys
4 An Example Suppose you have a large number of marbles, which are either white or black, and you wish information on the fraction that are white, µ. You draw a single marble, and it is white. What can you say at 90% confidence? Classical: µ 0.1 Prior Bayesian: flat µ /µ µ 0.1 1/(1-µ) unnormalizable µ µ (1-µ) µ Notice that most of the Bayesian priors do not cover, i.e., they are not true statements the stated fraction of the time (90% in this case). There is no requirement that credible intervals cover. However, Bob Cousins warns [Am. J. Phys. 63, 398 (1995)], if a Bayesian method is known to yield intervals with frequentist coverage appreciably less than the stated C.L. for some possible value of the unknown parameters, then it seems to have no chance of gaining consensus acceptance in particle physics. Gary Feldman 4 Journeys
5 The Role of Bayesian Statistics Prosper [Phys. Rev. D 37, 1153 (1988)] argues for a 1/µ prior based on a scaling argument. I found it unsatisfactory for two reasons: (1) It fails for x = 0. (Unnormalizable) (2) In general, it undercovers. I decided to continue my search for a solution to the simple problem in classical statistics. To quote Bob s prose from our paper: In our view, the attempt to find a non-informative prior within Bayesian inference is misguided. The real power of Bayesian inference lies in its ability to incorporate informative prior information, not ignorance. Prosper wrote that he was using a Bayesian approach because we are merely acknowledging the fact that a coherent solution to the small-signal problem is more easily achieved within a Bayesian framework than one which uses the methods of classical statistics. By the end of this talk, I hope to convince you that this is no longer true. Gary Feldman 5 Journeys
6 Construction of Confidence Intervals Neyman s prescription: Before doing an experiment, for each possible value of theory parameters determine a region of data that occurs C.L. of the time, say 90%. After doing the experiment, find all of values of the theory parameters for which your data is in their 90% region. This is the confidence interval. Notice that there is complete freedom of choice of which 90% to choose. This will be the key to our solution. Gary Feldman 6 Journeys
7 Examples of Poisson Confidence Belts 90% C.L. upper limits for Poisson µ with background = 3 90% C.L. central limits for Poisson µ with background = 3 Gary Feldman 7 Journeys
8 The Solution For both the upper limit and central limit, x = 0 excludes the whole plane. But consider the problem from the point of view of the data. If one measures no events, then clearly the most likely value of µ is zero. Why should one rule out the most likely scenario? Therefore, we propose a new ordering principle based on the ratio of a given µ to the most likely µ, µ ˆ : P(x µ) R = P(x µ ˆ ) Example for µ = 0.5 and b = 3: x P(x µ) ˆ µ P(x ˆ µ ) R rank U.L. C.L Gary Feldman 8 Journeys
9 The Poisson Limits 90% C.L. unified limits for Poisson µ with background = 3 Excerpt from Table IV of the paper (90% C.L.): x/b Gary Feldman 9 Journeys
10 Examples of Gaussian Confidence Belts 90% C.L. upper limits for Gaussian µ 0 vs. x (total background) in σ 90 % C.L. central limits for Gaussian µ 0 Gary Feldman 10 Journeys
11 Flip-flopping How might a typical physicist use these plots? (1) If the result x < 3σ, I will quote an upper limit. (2) If the result x > 3σ, I will quote central confidence interval. (3) If the result x < 0, I will pretend I measured zero. This results in the following: 10% 5% In the range 1.36 µ 4.28, there is only 85% coverage! Due to flip-flopping (deciding whether to use an upper limit or a central confidence region based on the data) these are not valid confidence intervals. Gary Feldman 11 Journeys
12 Unified Solution to the Gaussian Case Notes: (1) This approaches the central limits for x >>1. (2) The upper limit for x = 0 is 1.64, the two-sided rather than the one-sided limit. (3) From the defining 1937 paper of Neyman, this is the only valid confidence belt, since there are 4 requirements for a valid belt: (a) It must cover. (b) For every x, there must be at least one µ. (c) No holes (only valid for single µ). (d) Every limit must include its end points. Gary Feldman 12 Journeys
13 Including Errors on the Background Our paper assumed perfect knowledge of the backgrounds. However, often backgrounds are estimated from an ancillary experiment (sidebands, target-out run, etc.) How does the error on the ancillary experiment affect the limits? A Bayesian can solve this problem trivially by integrating over the background distribution. However, this is not permitted for frequentist. He or she must provide coverage for all possible values of the unknown true background rate. [The unknown true background rate is technically known as a nuisance parameter, because one must provide coverage for it, but you really do not care what its value is.] The exact efficient solution to this problem is computationally taxing, and has never been published. This is a second reason why physicists have been driven to Bayesian statistics. However, there is a simple solution to this problem within the unified approach, and Bob and I have agreed that we will publish it this summer. Gary Feldman 13 Journeys
14 Trick for Including Errors on Backgrounds If one provides coverage for the worst case of the nuisance parameter, then perhaps one will have provided coverage for all possible values of the nuisance parameter. Let µ = unknown true value of the signal parameter β = unknown true value of the background parameter x = measurement of µ + β b = measurement of β in the ancillary experiment The rank R becomes R ( µ, x,b) = P(x µ + ˆ β )P(b ˆ β ) P(x µ ˆ + β ˆ )P(b ˆ β ), where µ ˆ and β ˆ maximize the denominator (as usual), and ˆ β maximizes the numerator. Since R is only a function of µ and the data, one proceeds to construct the confidence belt as before. Gary Feldman 14 Journeys
15 Examples and Test of the Trick Let x be a Poisson measurement of µ + β and b be a Poisson measurement of β/r in an ancillary experiment (i.e., r = signal region/control region), r / n, rb 0, 3 3, 3 6, 3 9, A test of coverage for r = 1.0, 0 µ 20, 0 β 15 at 10,000 randomly picked values of µ and β: Median coverage: 90.25% Fraction that undercovered: 4.10% Median coverage for those that 89.96% undercovered: Worst undercoverage: 89.46% Comment from Harvard statisticians: We have never seen a statistical approximation work this well. Gary Feldman 15 Journeys
16 Back to Neutrinos As I had hoped, once we had the solution to the simple problem, the solution to the neutrino problem was obvious, unique, and natural. (My postdoc, Fred Weber, was doing almost the correct thing instinctively.) Probability of oscillation: P = sin 2 (2θ) sin 2 (1.27 m 2 L/E) Since χ 2 = 2ln(P), the ordering principle becomes a χ 2 = [χ 2 (data point in plane) - χ 2 (data best fit point in plane)]. The prescription: To calculate the acceptance region for each point in the plane, do a Monte Carlo simulation and determine what critical value of χ 2 will contain 90% of the data. Gary Feldman 16 Journeys
17 Combining Experiments: The Higgs Mass An example that may be of interest to LEP experimenters is how to use these techniques to combine different experimental measurements of the Higgs Mass. Let µ be the unknown true value of the Higgs mass, which is presumably a known function of the counting rate, ρ i, where i is an index denoting the experiment. In each experiment i, let β i be the unknown true rate of background. Let b i be a measure of the background in an ancillary experiment, say the sidebands. Finally, let n i be the measured rate of ρ i + β i. The ordering principle becomes: ˆ ˆ P(n i µ, β i )P(b i β i ) i R(µ,n i,b i ) = P(n i µ ˆ, β ˆ i )P(b i β ˆ, i ) i where µ ˆ and β ˆ i maximize the denominator and ˆ β i maximize the numerator. Gary Feldman 17 Journeys
18 The Higgs Mass Example (continued) The maximization of the numerator can be done analytically; in general, the maximization of the denominator requires a simple numerical solution. For each value of µ, the critical value of R c (µ) n i, b i R(µ,n i, b i )<R c ( µ) P(n i µ, ˆ β i )P(b i ˆ β i ) = 0.9 is determined. If the explicit sum is computationally impractical, a Monte Carlo calculation can be substituted. If the measured values of n i and b i are N i and B i, then the 90% confidence interval for µ is given by R(µ,N i, B i ) R c (µ) Gary Feldman 18 Journeys
19 The Higgs Mass Example (continued) Using the following information from CERN-EP/98-046, Lower bound for the Standard Model Higgs boson mass from combining the results of the four LEP experiments Experiment Expected Background Seen Aleph Delphi Opal Total (Not enough information given to include L3.) and the figure giving expected signal vs. Higgs mass, assuming backgrounds are perfectly known, I obtained m H < 75 GeV at 95% C.L., but a sensitivity to the Higgs Mass of only 71 GeV. (5% probability of observing 4 events when you expect 9.15.) (CERN-EP/ numbers: 77.5 GeV and 75.5 GeV) Gary Feldman 19 Journeys
20 Brand X Comparisons: Raster Scan We built a toy model to compare this technique with three other classical techniques that have, or could have, been used in previous experiments. Raster scan: For each m 2, scan along sin 2 (2θ) to find the minimum χ 2 (including the unphysical region), and take the 90% confidence interval to have χ 2 2 χ min (the two-sided limit). This technique (and the other two to be discussed) will have all the problems of the possibility of excluding all physically allowed values, which we originally set out to solve. However, it does cover exactly, but it is not powerful (i.e., it does not do an efficient job of rejecting false hypotheses). Gary Feldman 20 Journeys
21 Brand X Comparisons: Flip-flop Raster Scan Flip-flop raster scan: Same as raster scan, but use a onesided limit (χ 2 2 χ min ), unless the signal is > 3σ. This has all the features of the raster scan, but it significantly undercovers (85%) in a substantial region: Gary Feldman 21 Journeys
22 Brand X Comparisons: Global Scan Global Scan: Find the global minimum and take (χ 2 2 χ min , the two-sided 90% C.L. for χ 2 with 2 degrees of freedom). This technique is powerful, but has significant regions of both undercoverage (76%) and overcoverage (94%). Possible reasons: (1) Sinusoidal nature of the oscillation function. (2) One-dimensional regions. Gary Feldman 22 Journeys
23 Brand X Comparisons: Scorecard Technique Always gives useful information Gives proper coverage Is powerful Raster scan X X Flip-flop R.S. X X X Global scan X X This technique Gary Feldman 23 Journeys
24 Sensitivity The main objection to this work has been that an experiment that observes fewer events than the expected background may report a lower upper limit than a (better designed?) experiment that has no background. To address this problem and to provide additional information for the reader s assessment of the significance of the results, we suggest that experiments that have fewer counts than expected background also report their sensitivity, which we define as the average upper limit that would be obtained by an ensemble of experiments with the expected background and no true signal. For example, in the initial NOMAD publication of 1995 data, we reported an upper limit on high-mass ν µ ν τ oscillations of sin 2 (2θ) < and a sensitivity of A few days ago, at the Neutrino 98 conference, we reported preliminary results from a partial analysis of data yielding an upper limit of and a sensitivity of Gary Feldman 24 Journeys
25 Visit to Harvard Statistics Department Towards the end of this work, I decided to try it out on some professional statisticians whom I know at Harvard. They told me that this was the standard method of constructing a confidence interval! I asked them if they could point to a single reference of anyone using this method before, and they could not. They explained that in statistical theory there is a one-toone correspondence between a hypothesis test and a confidence interval. (The confidence interval is a hypothesis test for each value in the interval.) The Neyman-Pearson Theorem states that the likelihood ratio gives the most powerful hypothesis test. Therefore, it must be the standard method of constructing a confidence interval. I decided to start reading about hypothesis testing Gary Feldman 25 Journeys
26 Kendall and Stuart (1961) At the start of chapter 24 of Kendall and Stuart s The Advanced Theory of Statistics (chapter 23 of Stuart and Ord), I found 1 1/4 cryptic pages that contain all of the information, published and unpublished, in this lecture. (See text) Gary Feldman 26 Journeys
27 1998 Particle Data Group Review The 1998 PDG Review of Particle Physics [European Physics Journal C3 (1998) 1] has supported this approach strongly: The Bayesian methodology, while well adapted to decision-making situations, is not in general appropriate for the objective presentation of experimental data. In order to correct this problem [undercoverage due to flip-flopping] it is necessary to adopt a unified procedure. The appropriate unified procedure and ordering principle are given in Ref. 11 [Feldman- Cousins]. If it is desired to report an upper limit in an objective way such that it has a well-defined statistical meaning in terms of limiting frequency, then report the Frequentist confidence bound(s) as given by the unified Feldman- Cousins approach. we suggest that a measure of sensitivity should be reported whenever expected background is larger or comparable to the number of observed counts. The best such measure we know of is that proposed and tabulated in Ref. 11. Gary Feldman 27 Journeys
28 Conclusion I have yet to find a problem in the construction of classical confidence intervals and regions that is not solvable by the ordering principle suggested here. For example, we have been using it with success in the NOMAD experiment to combine the results for different τ decay modes, including errors on the background determination. Post script for LEP audiences: The LEP Higgs paper (CERN-EP/98-046) states: There is no unique, exact, method to deal with this problem. The method presented here is exact by construction. I believe that this method, due to its simplicity and reliance on classical methods, the classical construction of the confidence interval and the use of the likelihood ratio, deserves to be called unique, if any method does. Gary Feldman 28 Journeys
Statistics of Small Signals
Statistics of Small Signals Gary Feldman Harvard University NEPPSR August 17, 2005 Statistics of Small Signals In 1998, Bob Cousins and I were working on the NOMAD neutrino oscillation experiment and we
More informationPhysics 403. Segev BenZvi. Credible Intervals, Confidence Intervals, and Limits. Department of Physics and Astronomy University of Rochester
Physics 403 Credible Intervals, Confidence Intervals, and Limits Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Summarizing Parameters with a Range Bayesian
More informationStatistics for Particle Physics. Kyle Cranmer. New York University. Kyle Cranmer (NYU) CERN Academic Training, Feb 2-5, 2009
Statistics for Particle Physics Kyle Cranmer New York University 91 Remaining Lectures Lecture 3:! Compound hypotheses, nuisance parameters, & similar tests! The Neyman-Construction (illustrated)! Inverted
More informationUnified approach to the classical statistical analysis of small signals
PHYSICAL REVIEW D VOLUME 57, NUMBER 7 1 APRIL 1998 Unified approach to the classical statistical analysis of small signals Gary J. Feldman * Department of Physics, Harvard University, Cambridge, Massachusetts
More informationarxiv:physics/ v2 [physics.data-an] 16 Dec 1999
HUTP-97/A096 A Unified Approach to the Classical Statistical Analysis of Small Signals arxiv:physics/9711021v2 [physics.data-an] 16 Dec 1999 Gary J. Feldman Department of Physics, Harvard University, Cambridge,
More informationChapter 3: Interval Estimation
Chapter 3: Interval Estimation 1. Basic Concepts: Probability, random variables, distributions, convergence, law of large numbers, Central Limit Theorem, etc. 2. Point Estimation 3. Interval Estimation
More informationE. Santovetti lesson 4 Maximum likelihood Interval estimation
E. Santovetti lesson 4 Maximum likelihood Interval estimation 1 Extended Maximum Likelihood Sometimes the number of total events measurements of the experiment n is not fixed, but, for example, is a Poisson
More informationConfidence intervals and the Feldman-Cousins construction. Edoardo Milotti Advanced Statistics for Data Analysis A.Y
Confidence intervals and the Feldman-Cousins construction Edoardo Milotti Advanced Statistics for Data Analysis A.Y. 2015-16 Review of the Neyman construction of the confidence intervals X-Outline of a
More informationSolution: chapter 2, problem 5, part a:
Learning Chap. 4/8/ 5:38 page Solution: chapter, problem 5, part a: Let y be the observed value of a sampling from a normal distribution with mean µ and standard deviation. We ll reserve µ for the estimator
More informationMethod of Feldman and Cousins for the construction of classical confidence belts
Method of Feldman and Cousins for the construction of classical confidence belts Ulrike Schnoor IKTP TU Dresden 17.02.2011 - ATLAS Seminar U. Schnoor (IKTP TU DD) Felcman-Cousins 17.02.2011 - ATLAS Seminar
More informationConfidence Intervals. First ICFA Instrumentation School/Workshop. Harrison B. Prosper Florida State University
Confidence Intervals First ICFA Instrumentation School/Workshop At Morelia,, Mexico, November 18-29, 2002 Harrison B. Prosper Florida State University Outline Lecture 1 Introduction Confidence Intervals
More informationConstructing Ensembles of Pseudo-Experiments
Constructing Ensembles of Pseudo-Experiments Luc Demortier The Rockefeller University, New York, NY 10021, USA The frequentist interpretation of measurement results requires the specification of an ensemble
More informationStatistical Methods for Particle Physics Lecture 4: discovery, exclusion limits
Statistical Methods for Particle Physics Lecture 4: discovery, exclusion limits www.pp.rhul.ac.uk/~cowan/stat_aachen.html Graduierten-Kolleg RWTH Aachen 10-14 February 2014 Glen Cowan Physics Department
More informationConfidence intervals fundamental issues
Confidence intervals fundamental issues Null Hypothesis testing P-values Classical or frequentist confidence intervals Issues that arise in interpretation of fit result Bayesian statistics and intervals
More informationStatistical Methods in Particle Physics
Statistical Methods in Particle Physics Lecture 11 January 7, 2013 Silvia Masciocchi, GSI Darmstadt s.masciocchi@gsi.de Winter Semester 2012 / 13 Outline How to communicate the statistical uncertainty
More informationUse of the likelihood principle in physics. Statistics II
Use of the likelihood principle in physics Statistics II 1 2 3 + Bayesians vs Frequentists 4 Why ML does work? hypothesis observation 5 6 7 8 9 10 11 ) 12 13 14 15 16 Fit of Histograms corresponds This
More informationStatistical Methods in Particle Physics Lecture 2: Limits and Discovery
Statistical Methods in Particle Physics Lecture 2: Limits and Discovery SUSSP65 St Andrews 16 29 August 2009 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationStatistics Challenges in High Energy Physics Search Experiments
Statistics Challenges in High Energy Physics Search Experiments The Weizmann Institute of Science, Rehovot, Israel E-mail: eilam.gross@weizmann.ac.il Ofer Vitells The Weizmann Institute of Science, Rehovot,
More informationAdvanced Statistics Course Part I
Advanced Statistics Course Part I W. Verkerke (NIKHEF) Wouter Verkerke, NIKHEF Outline of this course Advances statistical methods Theory and practice Focus on limit setting and discovery for the LHC Part
More informationSecond Workshop, Third Summary
Statistical Issues Relevant to Significance of Discovery Claims Second Workshop, Third Summary Luc Demortier The Rockefeller University Banff, July 16, 2010 1 / 23 1 In the Beginning There Were Questions...
More informationHypothesis testing (cont d)
Hypothesis testing (cont d) Ulrich Heintz Brown University 4/12/2016 Ulrich Heintz - PHYS 1560 Lecture 11 1 Hypothesis testing Is our hypothesis about the fundamental physics correct? We will not be able
More informationStatistical Methods for Discovery and Limits in HEP Experiments Day 3: Exclusion Limits
Statistical Methods for Discovery and Limits in HEP Experiments Day 3: Exclusion Limits www.pp.rhul.ac.uk/~cowan/stat_freiburg.html Vorlesungen des GK Physik an Hadron-Beschleunigern, Freiburg, 27-29 June,
More informationSome Topics in Statistical Data Analysis
Some Topics in Statistical Data Analysis Invisibles School IPPP Durham July 15, 2013 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan G. Cowan
More informationPhysics 403. Segev BenZvi. Classical Hypothesis Testing: The Likelihood Ratio Test. Department of Physics and Astronomy University of Rochester
Physics 403 Classical Hypothesis Testing: The Likelihood Ratio Test Segev BenZvi Department of Physics and Astronomy University of Rochester Table of Contents 1 Bayesian Hypothesis Testing Posterior Odds
More informationOVERVIEW OF PRINCIPLES OF STATISTICS
OVERVIEW OF PRINCIPLES OF STATISTICS F. James CERN, CH-1211 eneva 23, Switzerland Abstract A summary of the basic principles of statistics. Both the Bayesian and Frequentist points of view are exposed.
More informationStatistics for the LHC Lecture 2: Discovery
Statistics for the LHC Lecture 2: Discovery Academic Training Lectures CERN, 14 17 June, 2010 indico.cern.ch/conferencedisplay.py?confid=77830 Glen Cowan Physics Department Royal Holloway, University of
More informationStatistical Methods in Particle Physics Lecture 1: Bayesian methods
Statistical Methods in Particle Physics Lecture 1: Bayesian methods SUSSP65 St Andrews 16 29 August 2009 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationLecture 24. Introduction to Error Analysis. Experimental Nuclear Physics PHYS 741
Lecture 24 Introduction to Error Analysis Experimental Nuclear Physics PHYS 741 heeger@wisc.edu References and Figures from: - Bevington Data Reduction and Error Analysis 1 Seminar Announcement Friday,
More informationCREDIBILITY OF CONFIDENCE INTERVALS
CREDIBILITY OF CONFIDENCE INTERVALS D. Karlen Λ Ottawa-Carleton Institute for Physics, Carleton University, Ottawa, Canada Abstract Classical confidence intervals are often misunderstood by particle physicists
More informationStatistics for Particle Physics. Kyle Cranmer. New York University. Kyle Cranmer (NYU) CERN Academic Training, Feb 2-5, 2009
Statistics for Particle Physics Kyle Cranmer New York University 1 Hypothesis Testing 55 Hypothesis testing One of the most common uses of statistics in particle physics is Hypothesis Testing! assume one
More informationFrequentist Confidence Limits and Intervals. Roger Barlow SLUO Lectures on Statistics August 2006
Frequentist Confidence Limits and Intervals Roger Barlow SLUO Lectures on Statistics August 2006 Confidence Intervals A common part of the physicist's toolkit Especially relevant for results which are
More informationLecture 5. G. Cowan Lectures on Statistical Data Analysis Lecture 5 page 1
Lecture 5 1 Probability (90 min.) Definition, Bayes theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests (90 min.) general concepts, test statistics,
More informationP Values and Nuisance Parameters
P Values and Nuisance Parameters Luc Demortier The Rockefeller University PHYSTAT-LHC Workshop on Statistical Issues for LHC Physics CERN, Geneva, June 27 29, 2007 Definition and interpretation of p values;
More informationTopics in Statistical Data Analysis for HEP Lecture 1: Bayesian Methods CERN-JINR European School of High Energy Physics Bautzen, June 2009
Topics in Statistical Data Analysis for HEP Lecture 1: Bayesian Methods CERN-JINR European School of High Energy Physics Bautzen, 14 27 June 2009 Glen Cowan Physics Department Royal Holloway, University
More informationComputing Likelihood Functions for High-Energy Physics Experiments when Distributions are Defined by Simulators with Nuisance Parameters
Computing Likelihood Functions for High-Energy Physics Experiments when Distributions are Defined by Simulators with Nuisance Parameters Radford M. Neal Dept. of Statistics, University of Toronto Abstract
More informationStatistical Methods for Particle Physics
Statistical Methods for Particle Physics Invisibles School 8-13 July 2014 Château de Button Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationStatistical Methods for Particle Physics Lecture 3: Systematics, nuisance parameters
Statistical Methods for Particle Physics Lecture 3: Systematics, nuisance parameters http://benasque.org/2018tae/cgi-bin/talks/allprint.pl TAE 2018 Centro de ciencias Pedro Pascual Benasque, Spain 3-15
More informationarxiv: v1 [hep-ex] 9 Jul 2013
Statistics for Searches at the LHC Glen Cowan Physics Department, Royal Holloway, University of London, Egham, Surrey, TW2 EX, UK arxiv:137.2487v1 [hep-ex] 9 Jul 213 Abstract These lectures 1 describe
More informationReview: Statistical Model
Review: Statistical Model { f θ :θ Ω} A statistical model for some data is a set of distributions, one of which corresponds to the true unknown distribution that produced the data. The statistical model
More informationRecent developments in statistical methods for particle physics
Recent developments in statistical methods for particle physics Particle Physics Seminar Warwick, 17 February 2011 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk
More informationWhy Try Bayesian Methods? (Lecture 5)
Why Try Bayesian Methods? (Lecture 5) Tom Loredo Dept. of Astronomy, Cornell University http://www.astro.cornell.edu/staff/loredo/bayes/ p.1/28 Today s Lecture Problems you avoid Ambiguity in what is random
More information32. STATISTICS. 32. Statistics 1
32. STATISTICS 32. Statistics 1 Revised April 1998 by F. James (CERN); February 2000 by R. Cousins (UCLA); October 2001, October 2003, and August 2005 by G. Cowan (RHUL). This chapter gives an overview
More informationStatistical Methods for Particle Physics (I)
Statistical Methods for Particle Physics (I) https://agenda.infn.it/conferencedisplay.py?confid=14407 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationStatistics for the LHC Lecture 1: Introduction
Statistics for the LHC Lecture 1: Introduction Academic Training Lectures CERN, 14 17 June, 2010 indico.cern.ch/conferencedisplay.py?confid=77830 Glen Cowan Physics Department Royal Holloway, University
More informationIntroductory Statistics Course Part II
Introductory Statistics Course Part II https://indico.cern.ch/event/735431/ PHYSTAT ν CERN 22-25 January 2019 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk www.pp.rhul.ac.uk/~cowan
More informationModern Methods of Data Analysis - SS 2009
Modern Methods of Data Analysis Lecture X (2.6.09) Contents: Frequentist vs. Bayesian approach Confidence Level Re: Bayes' Theorem (1) Conditional ( bedingte ) probability Examples: Rare Disease (1) Probability
More informationSTATS 200: Introduction to Statistical Inference. Lecture 29: Course review
STATS 200: Introduction to Statistical Inference Lecture 29: Course review Course review We started in Lecture 1 with a fundamental assumption: Data is a realization of a random process. The goal throughout
More informationLeïla Haegel University of Geneva
Picture found at: http://www.ps.uci.edu/~tomba/sk/tscan/pictures.html Leïla Haegel University of Geneva Seminar at the University of Sheffield April 27th, 2017 1 Neutrinos are: the only particle known
More informationSearch and Discovery Statistics in HEP
Search and Discovery Statistics in HEP Eilam Gross, Weizmann Institute of Science This presentation would have not been possible without the tremendous help of the following people throughout many years
More informationPhysics 509: Error Propagation, and the Meaning of Error Bars. Scott Oser Lecture #10
Physics 509: Error Propagation, and the Meaning of Error Bars Scott Oser Lecture #10 1 What is an error bar? Someone hands you a plot like this. What do the error bars indicate? Answer: you can never be
More informationChapter Three. Hypothesis Testing
3.1 Introduction The final phase of analyzing data is to make a decision concerning a set of choices or options. Should I invest in stocks or bonds? Should a new product be marketed? Are my products being
More informationLecture 2. G. Cowan Lectures on Statistical Data Analysis Lecture 2 page 1
Lecture 2 1 Probability (90 min.) Definition, Bayes theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests (90 min.) general concepts, test statistics,
More informationSensitivity of oscillation experiments to the neutrino mass ordering
Sensitivity of oscillation experiments to the neutrino mass ordering emb@kth.se September 10, 2014, NOW2014 1 Our usual approach 1 Our usual approach 2 How much of an approximation? 1 Our usual approach
More informationPractical Statistics part II Composite hypothesis, Nuisance Parameters
Practical Statistics part II Composite hypothesis, Nuisance Parameters W. Verkerke (NIKHEF) Summary of yesterday, plan for today Start with basics, gradually build up to complexity of Statistical tests
More informationAST 418/518 Instrumentation and Statistics
AST 418/518 Instrumentation and Statistics Class Website: http://ircamera.as.arizona.edu/astr_518 Class Texts: Practical Statistics for Astronomers, J.V. Wall, and C.R. Jenkins Measuring the Universe,
More informationStatistical Methods for Astronomy
Statistical Methods for Astronomy Probability (Lecture 1) Statistics (Lecture 2) Why do we need statistics? Useful Statistics Definitions Error Analysis Probability distributions Error Propagation Binomial
More informationBayesian Inference: Concept and Practice
Inference: Concept and Practice fundamentals Johan A. Elkink School of Politics & International Relations University College Dublin 5 June 2017 1 2 3 Bayes theorem In order to estimate the parameters of
More informationMathematical Statistics
Mathematical Statistics MAS 713 Chapter 8 Previous lecture: 1 Bayesian Inference 2 Decision theory 3 Bayesian Vs. Frequentist 4 Loss functions 5 Conjugate priors Any questions? Mathematical Statistics
More informationBayesian inference. Fredrik Ronquist and Peter Beerli. October 3, 2007
Bayesian inference Fredrik Ronquist and Peter Beerli October 3, 2007 1 Introduction The last few decades has seen a growing interest in Bayesian inference, an alternative approach to statistical inference.
More informationThe Jeffreys-Lindley Paradox and Discovery Criteria in High Energy Physics
The Jeffreys-Lindley Paradox and Discovery Criteria in High Energy Physics Bob Cousins Univ. of California, Los Angeles Workshop on Evidence, Discovery, Proof: Measuring the Higgs Particle Univ. of South
More information32. STATISTICS. 32. Statistics 1
32. STATISTICS 32. Statistics 1 Revised September 2007 by G. Cowan (RHUL). This chapter gives an overview of statistical methods used in High Energy Physics. In statistics, we are interested in using a
More informationHypothesis Testing - Frequentist
Frequentist Hypothesis Testing - Frequentist Compare two hypotheses to see which one better explains the data. Or, alternatively, what is the best way to separate events into two classes, those originating
More informationBayesian Inference for Normal Mean
Al Nosedal. University of Toronto. November 18, 2015 Likelihood of Single Observation The conditional observation distribution of y µ is Normal with mean µ and variance σ 2, which is known. Its density
More informationBayes Theorem (Recap) Adrian Bevan.
Bayes Theorem (Recap) Adrian Bevan email: a.j.bevan@qmul.ac.uk Overview Bayes theorem is given by The probability you want to compute: The probability of the hypothesis B given the data A. This is sometimes
More informationtheta a framework for template-based modeling and inference
theta a framework for template-based modeling and inference Thomas Müller Jochen Ott Jeannine Wagner-Kuhr Institut für Experimentelle Kernphysik, Karlsruhe Institute of Technology (KIT), Germany June 17,
More informationStatistical Methods for Particle Physics Lecture 3: systematic uncertainties / further topics
Statistical Methods for Particle Physics Lecture 3: systematic uncertainties / further topics istep 2014 IHEP, Beijing August 20-29, 2014 Glen Cowan ( Physics Department Royal Holloway, University of London
More informationParameter estimation and forecasting. Cristiano Porciani AIfA, Uni-Bonn
Parameter estimation and forecasting Cristiano Porciani AIfA, Uni-Bonn Questions? C. Porciani Estimation & forecasting 2 Temperature fluctuations Variance at multipole l (angle ~180o/l) C. Porciani Estimation
More informationThe Signal Estimator Limit Setting Method
EUROPEAN LABORATORY FOR PARTICLE PHYSICS (CERN) arxiv:physics/9812030v1 [physics.data-an] 17 Dec 1998 December 15, 1998 The Signal Estimator Limit Setting Method Shan Jin a,, Peter McNamara a, a Department
More informationSequential Monitoring of Clinical Trials Session 4 - Bayesian Evaluation of Group Sequential Designs
Sequential Monitoring of Clinical Trials Session 4 - Bayesian Evaluation of Group Sequential Designs Presented August 8-10, 2012 Daniel L. Gillen Department of Statistics University of California, Irvine
More informationSearch Procedures in High Energy Physics
Version 2.20 July 1, 2009 Search Procedures in High Energy Physics Luc Demortier 1 Laboratory of Experimental High-Energy Physics The Rockefeller University Abstract The usual procedure for searching for
More informationthe unification of statistics its uses in practice and its role in Objective Bayesian Analysis:
Objective Bayesian Analysis: its uses in practice and its role in the unification of statistics James O. Berger Duke University and the Statistical and Applied Mathematical Sciences Institute Allen T.
More informationHarrison B. Prosper. CMS Statistics Committee
Harrison B. Prosper Florida State University CMS Statistics Committee 7 August, 2008 Bayesian Methods: Theory & Practice. Harrison B. Prosper 1 h Lecture 2 Foundations & Applications h The Bayesian Approach
More informationStatistical Data Analysis Stat 3: p-values, parameter estimation
Statistical Data Analysis Stat 3: p-values, parameter estimation London Postgraduate Lectures on Particle Physics; University of London MSci course PH4515 Glen Cowan Physics Department Royal Holloway,
More informationDiscovery significance with statistical uncertainty in the background estimate
Glen Cowan, Eilam Gross ATLAS Statistics Forum 8 May, 2008 Discovery significance with statistical uncertainty in the background estimate Introduction In a search for a new type of event, data samples
More informationarxiv: v1 [physics.data-an] 7 Jan 2013
January 8, 2013 BAYES AND FREQUENTISM: A PARTICLE PHYSICIST S PERSPECTIVE 1 arxiv:1301.1273v1 [physics.data-an] 7 Jan 2013 Louis Lyons Blackett Lab., Imperial College, London SW7 2BW, UK and Particle Physics,
More informationStatistical Methods for Particle Physics Lecture 2: statistical tests, multivariate methods
Statistical Methods for Particle Physics Lecture 2: statistical tests, multivariate methods www.pp.rhul.ac.uk/~cowan/stat_aachen.html Graduierten-Kolleg RWTH Aachen 10-14 February 2014 Glen Cowan Physics
More informationMiscellany : Long Run Behavior of Bayesian Methods; Bayesian Experimental Design (Lecture 4)
Miscellany : Long Run Behavior of Bayesian Methods; Bayesian Experimental Design (Lecture 4) Tom Loredo Dept. of Astronomy, Cornell University http://www.astro.cornell.edu/staff/loredo/bayes/ Bayesian
More informationDiscovery and Significance. M. Witherell 5/10/12
Discovery and Significance M. Witherell 5/10/12 Discovering particles Much of what we know about physics at the most fundamental scale comes from discovering particles. We discovered these particles by
More informationPrimer on statistics:
Primer on statistics: MLE, Confidence Intervals, and Hypothesis Testing ryan.reece@gmail.com http://rreece.github.io/ Insight Data Science - AI Fellows Workshop Feb 16, 018 Outline 1. Maximum likelihood
More informationShould all Machine Learning be Bayesian? Should all Bayesian models be non-parametric?
Should all Machine Learning be Bayesian? Should all Bayesian models be non-parametric? Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/
More informationLine Broadening. φ(ν) = Γ/4π 2 (ν ν 0 ) 2 + (Γ/4π) 2, (3) where now Γ = γ +2ν col includes contributions from both natural broadening and collisions.
Line Broadening Spectral lines are not arbitrarily sharp. There are a variety of mechanisms that give them finite width, and some of those mechanisms contain significant information. We ll consider a few
More informationStatistical Methods for Astronomy
Statistical Methods for Astronomy If your experiment needs statistics, you ought to have done a better experiment. -Ernest Rutherford Lecture 1 Lecture 2 Why do we need statistics? Definitions Statistical
More informationWhaddya know? Bayesian and Frequentist approaches to inverse problems
Whaddya know? Bayesian and Frequentist approaches to inverse problems Philip B. Stark Department of Statistics University of California, Berkeley Inverse Problems: Practical Applications and Advanced Analysis
More informationMachine Learning 4771
Machine Learning 4771 Instructor: Tony Jebara Topic 11 Maximum Likelihood as Bayesian Inference Maximum A Posteriori Bayesian Gaussian Estimation Why Maximum Likelihood? So far, assumed max (log) likelihood
More informationA Calculator for Confidence Intervals
A Calculator for Confidence Intervals Roger Barlow Department of Physics Manchester University England Abstract A calculator program has been written to give confidence intervals on branching ratios for
More informationMarkov Chain Monte Carlo
Department of Statistics The University of Auckland https://www.stat.auckland.ac.nz/~brewer/ Emphasis I will try to emphasise the underlying ideas of the methods. I will not be teaching specific software
More informationParameter Estimation. William H. Jefferys University of Texas at Austin Parameter Estimation 7/26/05 1
Parameter Estimation William H. Jefferys University of Texas at Austin bill@bayesrules.net Parameter Estimation 7/26/05 1 Elements of Inference Inference problems contain two indispensable elements: Data
More informationSome Statistical Tools for Particle Physics
Some Statistical Tools for Particle Physics Particle Physics Colloquium MPI für Physik u. Astrophysik Munich, 10 May, 2016 Glen Cowan Physics Department Royal Holloway, University of London g.cowan@rhul.ac.uk
More informationBayesian vs frequentist techniques for the analysis of binary outcome data
1 Bayesian vs frequentist techniques for the analysis of binary outcome data By M. Stapleton Abstract We compare Bayesian and frequentist techniques for analysing binary outcome data. Such data are commonly
More informationBios 6649: Clinical Trials - Statistical Design and Monitoring
Bios 6649: Clinical Trials - Statistical Design and Monitoring Spring Semester 2015 John M. Kittelson Department of Biostatistics & nformatics Colorado School of Public Health University of Colorado Denver
More informationStat 535 C - Statistical Computing & Monte Carlo Methods. Arnaud Doucet.
Stat 535 C - Statistical Computing & Monte Carlo Methods Arnaud Doucet Email: arnaud@cs.ubc.ca 1 CS students: don t forget to re-register in CS-535D. Even if you just audit this course, please do register.
More informationFrequentist-Bayesian Model Comparisons: A Simple Example
Frequentist-Bayesian Model Comparisons: A Simple Example Consider data that consist of a signal y with additive noise: Data vector (N elements): D = y + n The additive noise n has zero mean and diagonal
More informationHarrison B. Prosper. CMS Statistics Committee
Harrison B. Prosper Florida State University CMS Statistics Committee 08-08-08 Bayesian Methods: Theory & Practice. Harrison B. Prosper 1 h Lecture 3 Applications h Hypothesis Testing Recap h A Single
More informationDiscovery Potential for the Standard Model Higgs at ATLAS
IL NUOVO CIMENTO Vol.?, N.?? Discovery Potential for the Standard Model Higgs at Glen Cowan (on behalf of the Collaboration) Physics Department, Royal Holloway, University of London, Egham, Surrey TW EX,
More informationHypothesis testing I. - In particular, we are talking about statistical hypotheses. [get everyone s finger length!] n =
Hypothesis testing I I. What is hypothesis testing? [Note we re temporarily bouncing around in the book a lot! Things will settle down again in a week or so] - Exactly what it says. We develop a hypothesis,
More informationFrequentist Statistics and Hypothesis Testing Spring
Frequentist Statistics and Hypothesis Testing 18.05 Spring 2018 http://xkcd.com/539/ Agenda Introduction to the frequentist way of life. What is a statistic? NHST ingredients; rejection regions Simple
More informationDealing with Data: Signals, Backgrounds and Statistics
Dealing with Data: Signals, Backgrounds and Statistics Luc Demortier The Rockefeller University TASI 2008 University of Colorado, Boulder, June 5 & 6, 2008 [Version 2, with corrections on slides 12, 13,
More informationStatistics notes. A clear statistical framework formulates the logic of what we are doing and why. It allows us to make precise statements.
Statistics notes Introductory comments These notes provide a summary or cheat sheet covering some basic statistical recipes and methods. These will be discussed in more detail in the lectures! What is
More informationConfidence Limits and Intervals 3: Various other topics. Roger Barlow SLUO Lectures on Statistics August 2006
Confidence Limits and Intervals 3: Various other topics Roger Barlow SLUO Lectures on Statistics August 2006 Contents 1.Likelihood and lnl 2.Multidimensional confidence regions 3.Systematic errors: various
More informationLecture Testing Hypotheses: The Neyman-Pearson Paradigm
Math 408 - Mathematical Statistics Lecture 29-30. Testing Hypotheses: The Neyman-Pearson Paradigm April 12-15, 2013 Konstantin Zuev (USC) Math 408, Lecture 29-30 April 12-15, 2013 1 / 12 Agenda Example:
More information