Physics 6720 Introduction to Statistics April 4, 2017

Transcription

1 Physics 6720 Introduction to Statistics April 4, Statistics of Counting Often an experiment yields a result that can be classified according to a set of discrete events, giving rise to an integer count or set of integer counts as a result. For example the measurement of a radioactive decay may yield the number of counts in a detector over a period of time. The measurement of the scattering of a beam of particles from a target may yield so may counts over a particular range of deflection angles over a period of time. Any series of measurements, the result of which can be classified into histogram bins, produces counts. In this section we will discuss briefly the statistics of counting, with particular emphasis on Poisson statistics. 1.1 Binomial Distribution We begin with the binomial distribution. Here we consider an experiment that is repeated many times. There are two possible outcomes: A and B. The probability for outcome A is p and the probability for outcome B is 1 p. We assume that each experiment has the same probability for each outcome and that there is no correlation between the outcome of one experiment and that of another. We may then ask the question, out of N repetitions of the experiment, what is the probability that we get A exactly k times? For example, suppose N = 4 and k = 3. The answer is found by first asking for the probability for a particular sequence of outcomes AABA, for example. The probability is just the product of the probabilities for each event: pp(1 p)p. This statement makes use of the fact that there is no correlation between one experiment and another. Since our question doesn t ask for a particular order of outcomes, but just any order that yields 3 A s out of 4 trials, we then ask how many different ways there are of getting 3 A s. We can enumerate them: BAAA, ABAA, AABA, and AAAB. Since the probability for each is the same, the probability for any of them is four times the probability for just one of them. So the probability is 4p 3 (1 p) to get 3 A s out of 4 trials. The general 1

2 expression is called the binomial distribution. The probability for getting k A s (and N k B s) out of N trials is P (k, N) = N! k!(n k)! pk (1 p) N k. (1) Notice that the binomial probabilities generate the binomial series, which adds up to 1, as it should: 1.2 Poisson Distribution N P (k, N) = [p + (1 p)] N = 1. (2) k=0 The Poisson distribution applies in cases in which the probability for getting A is very small compared with other possible outcomes. In that case we would use the binomial formula with a value of k much smaller than N. For example, suppose we were counting radioactive decays as a function of time and we observe the decays over a time interval dt that is much smaller than the decay lifetime, so the amount of radioactive material available for decay does not change noticeably during the time of observation. If the decay rate is λ and we consider just one single atom, the probability that it decays in a time interval dt is p = λdt (true as long as this is very small). Call this event A. If it doesn t decay (probability 1 p) we call it event B. If we now consider N atoms we can use the binomial distribution to give us the probability that that k atoms out of N atoms decay in the time interval dt. We expect that on average there will be k = pn decays. Let s find the probability for getting k events in the limit of large N, if the expected (average) number k is constant as we take the limit. Notice that to keep k constant we have to decrease p as we increase N. This means we are decreasing the time interval dt as we increase N. To get the probability, we start with the binomial distribution, substitute p = k/n and take the limit P (k, k) N! = lim N k!(n k)! ( k/n) k (1 k/n) N k (3) After some algebra, using the Stirling approximation for the factorial and the Taylor expansion for the exponential function, we get the Poisson distribution: P (k, k) = k k e k. (4) k! 2

3 This distribution is normalized to 1 as well. The sum generates the Taylor series for the exponential function: P (k, k) = e ke k = 1. (5) k=0 We will return to a discussion of properties of the Poisson distribution after discussing the Gaussian normal distribution. 2 Normal Distribution 2.1 Populations and Their Means and Standard Deviations In order to develop confidence in the result of a measurement of a single quantity, such as the length of a table top, we often repeat the measurement process a number of times. The results of the measurement vary because of difficulties in reading the meter stick scale to the last tenth of a millimeter, and for other reasons. Suppose we repeated the measurement N times, getting a list of values x i. Our best guess for the true value is usually the average of these values: N x = x i /N, (6) i=1 which is also called the mean value of this sample set of observations. In our notation, x indicates our best, imperfect estimate of the true value x. If we repeat the measurement an infinite number of times, ideally, the mean value should approach the true value of the measurement. The statistical way to describe what is happening is that our set of N measurements is a sample of N values taken from an infinite population. The true population mean is given by x = lim N N x i /N. (7) i=1 We might ask of this infinite population, what is the probability of getting a value of x in the range (x, x+dx) when we make a measurement? This probability is expressed in terms of a probability function P (x) as P (x)dx. The factor dx is necessary because as the interval width dx gets smaller, the probability of getting a value in that tiny range must get smaller in proportion to dx. 3

4 If we make enough measurements, we can begin to construct this probability function, but usually we don t make enough measurements to know it very well. So we often assume for want of any better reason that the probability is given by the Gaussian distribution function (normal distribution) P (x) = exp[ (x x)2 /2σ 2 ] 2πσ (8) In this expression the true mean of the population is x and σ is the true standard deviation. This probability is normalized so that P (x)dx = 1. (9) i.e. the probability of measuring any value of x is 1. The Gaussian distribution is peaked at x = x and falls off on either side of x over a distance in x that is controlled by the value of σ. If σ is large, the fall off is slow and the most probable values of x are in a broad range around x; if σ is small, the fall off is rapid, and the most probable values of x are narrowly clustered around x. A property of the Gaussian distribution is that the probability of making a measurement and getting a value in the range x σ and x + σ is about 68%. (This value is found by calculating the integral under the probability distribution from x σ to x + σ.) Thus in common usage, we say that for a single measured value of x, the result is x ± σ. The standard deviation of a quantity is sometimes called the error in that quantity, so we say the error in a single measurement is σ. The statement that x lies in the range x ± σ is a statement we can make with 68% confidence. That means the result of a measurement is likely to be outside this range 32% of the times we repeat the experiment. A measure of the width of this peak is given by Var(x) = σ 2 = (x x) 2 P (x)dx (10) This is just the average of (x x) 2 over the population. If we made an infinite number of measurements, we would be able to determine the two parameters x and σ of the distribution exactly. With a finite set of measurements, however, we can estimate them. To estimate the mean value, we simply compute the average of the measurements x i : N x = x = x i /N. (11) i=1 4

5 Notice that we have put a star on x to distinguish the estimate from the true value x. The sample also permits an estimate of this population standard deviation σ. It is just N Var(x 1, x 2,... ) = σ 2 = (x i x ) 2 /(N 1). (12) i=1 The quantity σ is the estimated standard deviation, and its square is called the estimated variance of x from the mean value x, or just the estimated variance of x. 1 Another useful formula is obtained by expanding the square on the right side to give (N 1)σ 2 = x 2 i 2 x x i + N x 2 = N( x 2 x 2 ). (13) The x 2 means the average of x 2 i. In other words the estimated variance is just the difference between the average of the squares and the square of the average times N/(N 1). As an exercise, try writing a program that reads a list of values x i and calculates x and σ. 2.2 The Error in the Estimated Mean So we see that if we have a finite data sample, we can get an estimate of the true values of x and σ. But how far is our estimate x from the true value x? This is the central question of every measurement, because it tells us how much confidence we may put in our result. Measurements without error ranges are meaningless! For example, there is really no meaning to the statement that the length of the table top is 3 meters, because the associated error might be a kilometer. There is meaning only if we can associate an error with this figure and say, for example, that the length is 3.00 meters with an error of plus or minus 0.01 meter. Now suppose we make N measurements to make up one data sample on one day and make another N measurements to make a second sample on the next, and so collect a large number of samples. We determine the estimated mean value x for each sample. What is the probability distribution 1 This expression for σ 2 is based on the sample mean x, and so is biased. To compensate for the bias, we divide by N 1 instead of N. 5

6 for this estimated mean value? Note that it is not the same as the probability distribution of the population. One way to see this is to realize that if we take larger and larger samples almost all of our values would be expected to be closer and closer to the true mean x. In fact, a famous theorem of statistics, called the central limit theorem states that the probability distribution of the mean value approaches a Gaussian normal distribution as the sample size increases, regardless of whether the underlying population distribution P (x) is itself Gaussian. The standard deviation of the mean value is estimated by σ mean = σ / N, (14) where σ is given by Eq (13). As the sample size grows, σ stabilizes, and the standard deviation of the mean shrinks as 1/ N, so that the distribution of sample means x gets sharper around the true value x. So as a result of measuring one sample, we estimate the true mean value to be x ± σ mean = x ± σ / N. This is a practical formula. With it we need only make N measurements, then estimate the population mean from Eq. (11) and the population standard deviation from Eq. (12). Then we compute the error in the mean from Eq. (14). Please bear in mind the difference between σ, which is the estimate of the error in a single measurement, i.e. the population standard deviation, and σmean, which is the estimate of the error in our estimated mean value x. 2.3 Systematic Error The error we have been discussing so far is a statistical error. It is an error that can be made smaller by simply making more and more measurements of the same type. Another error that occurs all-too-frequently is a systematic error. For example, in measuring the table top, it may happen that our meter stick was slightly miscalibrated, so it gave consistently large results. We would not be able to correct for such an error by repeating the measurement. We would instead have to recalibrate the meter stick. Sometimes we aren t able to do the recalibration, but are assured by the manufacturer or by some other means that the meter stick agrees with a precise standard to within an error of, say m. We might then quote the result of a measurement of the table top by saying it is ± 0.002(stat) ± 0.005(syst) meters, thereby identifying separately the two sources of error. 6

7 3 Properties of the Poisson Distribution 3.1 Mean and Variance Figure 1: Poisson distribution with mean value 5 The Poisson distribution for k = 5 is shown in Fig. 1. Notice that it peaks at k = 5. Let us determine the mean and variance for the Poisson distribution. The mean is just k = kp (k, k). (15) A little algebra gives k=0 k = k. (16) This result is naturally what we would expect, of course. The variance is given 7

8 by Var(k) = k 2 k 2 = k 2 P (k, k) k 2 (17) k=0 A little algebra shows that the first term is just k( k + 1) so Var(k) = k. (18) This result says that the standard deviation is approximately k. Actually we have to be careful about using the term standard deviation for the Poisson distribution, unless k is large. For small k the shape is not very much like a Gaussian, but for large k the shape approximates a Gaussian reasonably well. 3.2 Bayes Theorem and Maximum Likelihood So far we have been thinking of the probability for getting a result k if we know that the mean value should be k. Now suppose we make a measurement and get k counts, but we don t know anything about k, except that it must be nonnegative, of course. We may turn the question around and ask what is the most likely value for k, given the result of our measurement. To make this turned-around idea more concrete, we use the concept of conditional probability. We say that the Poisson distribution P (k, k) tells us the probability that we get k, on the condition that the mean value is k. The notation P (A C) denotes the probability for getting A, given that C occurs or C is true. Thus we could write P (k k) = P (k, k) = k k e k. (19) k! Now the reverse question is, What is the probability that the mean value is k, given that we just made a measurement and got k?. This probability would be denoted P ( k k). Now a trivial but important theorem due to Bayes states that P (A C)P (C) = P (C A)P (A) (20) where P (C) is the a priori probability for C to occur, regardless of whether the event A occurs, and P (A) is the a priori probability for A to occur, regardless of whether the event C occurs. From this theorem we conclude that P ( k k) = P (k k)p ( k)/p (k) (21) 8

9 So we need to know P ( k) and P (k) to make progress. The first is the a priori probability for getting a particular value for k. If we don t know anything about k, except that it is nonnegative, then we must say that any nonnegative value whatsoever is equally probable. Thus without benefit of knowing the outcome of the measurement, we say P ( k) is constant, independent of k for nonnegative k, and it is zero for negative k. So the rhs of this equation reduces simply to P ( k k) = N k k e k (22) k! where the normalization factor N = P ( k)/p (k) can be determined by requiring that the total probability for having any k is 1. In fact it turns out that N = 1, so P ( k k) = k k e k. (23) k! This distribution is called the likelihood function for the parameter k. Notice that we are now thinking of the rhs as a continuous function of k with fixed k. This result is very remarkable, since a single measurement is giving us the whole probability distribution! Recall that if we were to measure the length of a table top, even if we started by assuming we were going to get a Gaussian distribution, a single measurement would allow us only to guess x and would tell us nothing about σ. To get σ takes at least two measurements, and even then we would be putting ourselves at the mercy of the gods of statistics for taking a chance with only two measurements. If we weren t so rash as to assume a Gaussian, we would have to make many measurements of the length of the table top to get the probability distribution in the measured length. We now ask, what is the most probable value of k, given that we just found k? This is the value with maximum likelihood. If we examine the probability distribution, we see that it peaks at k, just as we might have expected. We may then ask, what is the error in the determination of this value. This is a tricky question, because the Poisson distribution is not shaped like a Gaussian distribution. However, for large k it looks more and more like a Gaussian. Expanding the log of the Poisson distribution for large k and fixed k k gives P ( k k) exp[ (k k) 2 /(2 k)]/ 2π k (24) so for large k the error is σ k = k. (25) 9

10 To summarize, a single measurement yields the entire probability distribution. For large enough k we can say that k = k ± k. (26) To see how Bayesian statistics works, suppose we repeated the experiment and got a new value k. What is the probability distribution for k in light of the new result? Now things have changed, since the a priori probability for k is no longer constant because we already made one measurement and got k. Instead we have P ( k) = k k e k (27) k! so P ( k k ) = N k k e k k k e k. (28) k! k! Notice that the likelihood function is now the product of the individual likelihood functions. A more systematic notation would write this function as P ( k k, k ), i.e. the probability for k having a particular value, given that we made two measurements and found k and k. The normalization factor N is obtained by requiring the total probability to be 1. The most likely value of k is easily shown to be just the average k = (k + k )/2, (29) as we should have expected. The Bayesian approach insists that we fold together all of our knowledge about a parameter in constructing its likelihood function. Thus a generalization of these results would state that the likelihood function for the parameter set C, given the independently measured results A 1, A 2, A 3, etc. is just P (C A 1, A 2, A 3,... ) = NP (A 1 C)P (A 2 C)P (A 3 C)..., (30) where N is a normalization factor. Again, this is just the product of the separate likelihood functions. The result is completely general and applies to any probability distribution, not just a Poisson distribution. We will use this result in discussing χ 2 fits to data as a maximum likelihood search. 10