Statistical Tests: Discriminants

Transcription

1 PHY310: Lecture 13 Statistical Tests: Discriminants Road Map Introduction to Statistical Tests (Take 2) Use in a Real Experiment Likelihood Ratio 1

2 Some Terminology for Statistical Tests Hypothesis This is what you are testing In physics, our models are usually functions that we will compare to data A model with specific parameters (e.g. the muon decay lifetime is 2.2 μs) A model with any parameter (e.g. particle decay follow an exponential p.d.f.) A model and the best fit parameters. Null Hypothesis This is the model that you are testing that is Usually denoted H0 Alternate Hypotheses Any other models you are comparing to the Null Hypothesis Usually denoted H1, H2,... Test Statistic A function of the measured variables, t(x1, x2,...) There will be a different pdf for each hypothesis (e.g. g(t H0), f(t H1), &c) 2

3 Defining Test Statistics A test statistic can be any function of the measured variables But some are better than others Examples: Two hypothesis statistics (also called discriminants) The Likelihood Ratio = t x The optimal way to compare two alternate hypotheses Fisher's Discriminant g x ; H0 g x ; H1 =at x = ai x i t x A linear approximation to the likelihood ratio Single hypothesis statistics (also call goodness-of-fit tests) The Likelihood Distribution Works for unbinned data Doesn't require an alternate hypothesis The χ² Distribution (chi-squared) Only works for binned data The absolute most common test 2 observed expected 2 = expected variance 3

4 The Problem: Classifying Data Many physics experiments are measuring things that come from several different classes of data. Particle Decays It decayed in the time window It didn't decay in the time window Super-Conductivity The sample is super conducting The sample isn't super conducting Super Novae Searchs A galaxy has a super nova A galaxy doesn't have a super nova Notice: The problem is (usually) phrased as a binary question Yes/No, True/False, Left/Right 4

5 Digression: Discriminants in Real Experiments Neutrino Oscillations: Cosmic Rays vs the Standard Model The Atmospheric Neutrino Problem Started as a background study for proton decay searches Based on measuring the electron and muon neutrino fluxes from cosmic rays striking the upper atmosphere Required classifying events as electron-like and muon-like One of the first solid body of data to significantly contradict standard model of particle physics That means the standard of proof had to be very high 5

6 The Standard Model of Particle Physics Standard Model of Particle Physics has been INCREDIBLY successful Predicted that neutrinos: are massless, can't oscillate, not their own anti particle 6

7 The Atmospheric Neutrino Problem Too Few Nu Mu Five experiments with slightly contradictory results First two were very small external contamination problems crude estimate of expectation Next three were much larger Still not many events IMB-3 biggest with ~970 events less (no!) external contamination better estimate of expectation but, coarser detector 7

8 SuperTankers of Physics: Ring Imaging Water Cherenkov Detectors Massive Active Volume for Atmospheric Interactions Solar Interactions Relic Supernova n Nucleon Decay Signals Measure light Tank of Water (all Active) light direction 8

9 Start with a Big Hole in the Ground IMB Hole Super-Kamiokande Hole 9

10 Install Light Sensors 10

11 Fill it with Water 11

12 The Discriminant Problem Discriminate between Events with an electron 12

13 The Discriminant Problem Discriminate between Events with an muon 13

14 Electrons vs Muons 14

15 Discriminant Analysis Construct a test statistic that has different PDFs for alternate hypotheses Remember a test statistic is any function of the measured variables t cut accept= dt g t H0 0 reject= dt g t H 0 t cut t cut confusion= dt g t H1 0 The acceptance should be large, the rejection small, and the confusion very small! This is also called multi-variate analysis since we almost always are trying to construct a statistical test using several different variables 15

16 Constructing the Test Statistic Suppose you've got measurements x = x 1, x 2,..., x n And likelihood functions for the alternate hypotheses ; H0 =g L x x H 0 ; H1 =g L x x H 1 The best test statistic will be the likelihood ratio = t x L x ; H0 ; H0 log L or log t =log L x x ; H 1 L x ; H1 The problem is that we usually can't construct the full likelihood Mostly generate likelihood distributions with an MC For multi-dimensional likelihoods, not enough memory (or time)! A five dimensional likelihood approximated by a pdf with 40 bins per dimension has ~100M bins Optimal selection will be t > constant 16

17 Simplifying the Likelihood Ratio A typical multi-dimensional likelihood has lots of internal correlations Approximate as a product of independent functions ; H0 = L x 1 ; H 0 L x 2 ; H0... L x n ; H 0 L x Independent likelihoods are much easier to construct Resulting statistic may not be optimal, but remember any test statistic that works is great Usually working in several dimensions (e.g. 5+) so can't make a nice scatter plot to see internal structure of p.d.f. 17

18 Building a Linearized Likelihood L x ; H0 = L x 1 ; H0 L x 2 ; H0... L x n ; H0 Best you can usually do is to make projections of each variable Can't see the internal correlations. But, it's quick and easy! Each projection can be studied independently. WARNING: generate p.d.f. and test efficiency with a different data sets The p.d.f.s are generated by histograming the projection onto each axis. This is used as a look-up to generate the full likelihood. 18

19 Applying to the Data The p.d.f. histograms for each variable must Have the same number of bins be normalized to have the same integrals. TH1F* type1x = new TH1F( type1x, X PDF, 100, -5, 5); TH1F* type2x = new TH1F( type2x, X PDF, 100, -5, 5); // Fill the histograms, then normalize type1x->scale(1.0/type1x->getentries()); type2x->scale(1.0/type2x->getentries()); The likelihood for a particular point x, y is calculated by looking up the number of entries in the bin of the pdf histograms // Find the bin numbers for each variable int binx = type1x->getxaxis()->findbin(x); int biny = type1y->getxaxis()->findbin(y); // Look up the bin contents double t1x = std::max(type1x->getbincontent(binx),1e-3); double t1y = std::max(type1y->getbincontent(biny),1e-3); double t2x = std::max(type2x->getbincontent(binx),1e-3); double t2y = std::max(type2y->getbincontent(biny),1e-3); // Calculate the log likelihood difference. double value = TMath::Log10(t1X); value += TMath::Log10(t1Y); value -= TMath::Log10(t2X); value -= TMath::Log10(t2Y); The usual likelihood cut is taken at zero. Might use a different value, it depends on the desired efficiency and purity. 19

20 Evaluating Efficiency and Contamination Data is classified by calculating the log(l) for each point The cut is chosen based on the desired efficiency and purity Plot the distribution of log(l) Expected distribution should be compared with measured. Efficiency is number correctly classified. Contamination is number incorrectly classified 20

21 Finally Statistical tests that compare alternate hypotheses are used to classify data into two or more categories Called Discriminant Analysis, or Multi-Variate Analysis If you know the full multi-dimensional p.d.f. then the likelihood is an optimal statistical test It's as good as it can get. Usually you don't know the full p.d.f. but you can approximate it as a product of independent p.d.f.s for each variable Looses information about the internal correlations, but makes the problem computationally tractable. For a wide class of data, the log likelihood difference is a simple and nearly optimal statistic. Choose this unless there are indications that it is significantly sub-optimal There are more complex statistics to use when the log likelihood difference is extremely sub-optimal The End 21