N! (N h)!h! ph 1(1 p 1 ) N h. (2) Suppose we make a change of variables from h to x through h = p 1 N + p 1 1 2

Transcription

1 Physics 48-0 Lab : An example of statistical error analysis in coin ip experiment Intro This worksheet steps you through the reasoning behind how statistical errors in simple experimental measurements are often dealt with. The simple (gedanken) experiment chosen to illustrate the reasoning is a coin ip experiment. After ipping a coin 50 times and counting the number of heads that are obtained, one should be able to state based on the experimental result that the probability of obtaining heads in a single coin ip is x ± y with 95% condence. Error Analysis Theory. Obtaining the probability of a single coin ip being heads Suppose we ip the coin times. We record the number of heads and the number of tails: heads=h and tails= h. Although we would naively conclude that the probability of obtaining heads in a single coin ip is P = h. () However, because is nite, this is not rigorously true. We need to assess with what condence we would expect the outcome of the data being h for a given. Let this probability be φ(, h). If we assume that every coin ip is independent and has a probability p (note the lower case), we nd φ(p,, h) =! ( h)!h! ph ( p ) h. () Suppose we make a change of variables from h to x through h = p + p + x where we are now treating h as a continuous variable. In the limit that, the well known Stirling's formula can be used in Eq. () to obtain where! π e (3) φ(p,, p + p + x) x e σ (4) πσ σ (p, ) = p ( p ) (5) characterizes the width. Hence, the most probable outcome of the experiment is h p as expected but the deviations of order p from this most probable value is not very improbable. ote that in terms of the original variable h, we can write φ(p,, h) (h p πσ e p +/) σ. (6) This functional form is called a Gaussian distribution, and this form of the functional form being a good approximation is expected on general grounds based on a theorem called the central limit theorem. Also, the small terms in the parentheses of the exponent such as p + / are not numerically important and the peak of the distribution should really be regarded as being at h = p. The fact that the sum of probabilities add up to is expressed as φ(p,, h) =. (7) h=0

2 In the approximate form of φ(, h) (given by Eq. (4)), we have a continuous variable for φ(, h). Hence, the conservation becomes dhφ(p,, h) = (8) which means that we had to extend the h variable to unphysical negative values. However, this is not problematic since φ(, 0) in the limit is negligible. Experimentally, we would like to measure p. Since the data sample is nite, we can only obtain an estimate of the true p from the experiment. Hence, we will refer to the data derived p as p (estimate). Suppose we assume that our experimental result corresponds to the most likely outcome. This is called the maximum likelihood method. To this end, we need a probability of p being the true probability given the data h (data). We turn to constructing this next.. Probability of p being true and the statistical uncertainty in the estimate of the probability The probability function φ computed in the previous subsection is the probability of obtaining h heads given that the probability of obtaining heads is p. In this subsection, we would like to estimate the probability of p being the true probability, given that the data is h = h (data). The conditional probability given XXX is denoted as P (something happening XXX): i.e. φ(p,, h)dh = P (obtain h p is the right answer). (9) Suppose p () and p () are two dierent probability value possibilities. According to what is known as Bayes' theorem, we have P (p () is the right answer obtain h) P (p () is the right answer obtain h) = P (p () P (p () Suppose we assume Then we nd or in other words We can obtain using is the right answer) is the right answer) P (obtain h p () is the right answer) P (obtain h p () is the right answer). (0) P (p () is the right answer). () P (p () is the right answer) P (p () is the right answer obtain h) P (p () is the right answer obtain h) φ(p,, h) φ(p,, h) () P (p is the right answer obtain h) φ(p,, h)dp. (3) ˆ 0 dp φ(p,, h) =. (4) Because φ(p,, h) should drop o rapidly as p is close to p (estimate) obtained from data (see below), we should be able to approximate this as a Gaussian. In particular, we can Taylor expand the exponent of Eq. (6) to quadratic order in p (about p which satises f h (p ) = 0) as φ(p,, h) πσ (p, ) e f h (p * ) f h (p* )(p p ) (5) where f h (p ) = (h p p + /) σ. (6) (p, )

3 Figure : Comparison of Eq. (7) and Eq. () for { = 0, h =, 5} and { = 0, h = 4, 0}. Can you guess which curve corresponds to which parameters? ote that this is not a normalized distribution and we must compute the normalization as discussed in the text. It is clear that the numerical approximation is excellent Since the dominant p dependence is in the exponent f(p ), for the purposes of this exercise we can neglect the p dependence in the prefactor for simplicity and write φ(p,, h) πσ (p, )e f h (p * ) f h (p* )(p p ). (7) More explicitly, we nd p = + h ( + ) h (8) f(p ) = 0 (9) f h (p * 4( + ) 4 ) = ( + h)( h + ) 3 h( h) Before moving on, let's check that the approximate formula Eq. (7) is a good approximation to the exact formula Eq. (). This is done by plotting the two functions for { = 0, h =, 5}, { = 0, h = 4, 0} in Fig.. Finally, solving Eq. (4) for, we arrive at P (p is the right answer obtain h) (0) e f (p* h )(p p ) dp () π/f h (p ) First, we can use the maximum likelihood reasoning to estimate p based on the data. The most probable p given the data h = h (data) is p (estimate) = p h(data) h=h (data) which is what we naively expected. ext, we can easily compute the answer to the question What is the probability that p (estimate) p (true) > p? It is approximately I P (the right answer is farther away from p (estimate) than p obtain h (data) ) = P (p (estimate) + p + ɛ is the right answer obtain h (data) ) +P (p (estimate) + p + ɛ is the right answer obtain h (data) ) +... (3) () 3

4 This can be written using Eq. () as f π h (data) (p ) p (estimate) + p P (the right answer is farther away from p (estimate) than p obtain h (data) ) (p * )(p p ) dp e f (p * ˆ p(estimate) h (data) )(p p ) p dp + e f h (data) where we use h (data) when evaluating p in this expression. Using this, one can also answer the question What is the p > 0 value for which I can make the statement that the true value of p is p (estimate) ± p with 95% condence? Explicitly, this is accomplished by solving p (estimate) + p e f (p * ˆ p(estimate) h (data) )(p p ) p dp + e f h (data) The p solving this equation is referred to as the 95% condence error bar. f π h (data) (p * )(p p ) dp (p ) (5) ote that quite generically, the width of the function e f (p * h (data) )(p p ) is approximately / f (p * h (data) ) h (data) ( h (data) ) (6) 3 and that data typically would yield (based on Eq. (6)) h (data) p (true) ( + O p (true) ( p (true ) ) ). (7) This means that in the limit, the error p scales as p (true) ( p (true) ) p (8) 3 p (true) ( p (true) ) =. (9) Hence, the error in determination of p (true) using experimental data decreases like /. This is a typical property of experimental systems with Gaussian-like uctuations. In the limit that, the error in p vanishes, recovering the well known statement that an innite set of measurements will yield the true probability..3 More typical experimental situations The coin ip situation is somewhat special because each of the data points (heads or tail) had only two outcomes with no really preferred value. Its main advantage was the fact that the probability functions for the observables were easy to compute. Most observables in physics experiments have a sharper preferred value within a continuous range of possible values. The probability function for each observation data point is typically approximately Gaussian centered about an ideal value that you want to extract from the experiment. Hence, the likelihood function (analog of φ(p,, h)) will be a product of Gaussians. In fact, we can treat the extracted p (estimate) for a single experiment as a single data point and repeat the experiment 4 (4)

5 to obtain a set of data points where each data point is of the form that is more typical of physics experimental observations. In fact, you have been working with this implicitly in error propagation of physics 47 labs. To avoid overwhelming you with material, we will discuss the mathematics of this later in future labs and now turn to exercises regarding the coin ip experiment. (Some related discussion is given in worked Exercise 4 below.) On the other hand, note that the general philosophy regarding error bars of the situation involving products of Gaussians is the same as the coin ip experiment just described. 3 Exercises. Suppose after 50 coin ips, one obtains 3 heads. What is the estimate of the probability of obtaining heads in a single coin ip from this experiment?. Find the 95% condence error bar associated with the result of the experiment above. [Hint: ote that and ] πy x m+y e y (x xm) dx Suppose at the same time as the experiment just described (which we will refer to as the rst experiment) takes place, another coin ip experiment with a dierent coin (but same procedure of ipping 50 times) takes place next to the rst one. Suppose this second experiment numbers its 50 coin ip results sequentially, just like the rst experiment. The second experiment luckily results in 3 heads as well. When the nth ip of the rst experiment has heads that matches the heads of the nth ip of the second experiment, we say there is a heads coincidence. What is the predicted number of coin ip heads coincidences assuming that every coin ip is independent based on the experimentally measured probabilities. 4. Worked exercise explained by the TA: What is the 95% condence error bar on this prediction? ans Our prediction in the last problem is based on estimating the true probability to be p (estimate) p (estimate) We need to estimate an error on this probability. The probability distribution for having p and p is π/f h (data ). e f (data (p * h ) )(p p ) dp f (data (p * (p ) π/f h (data (p h ) )(p p ) dp. ) )e When integrated over a particular (p, p ) region centered about (p, p ), the probability is h (data ) = h (data ) (which simplies the mathematics), we can write ˆ π/f h (data ) (p ) region e f h (data ) (p * )[(p p ) +(p p ) ] dp dp = 0.95 Since ote that when (p p ) + (p p ) = constant, the integrand has the same value. This is an equation for a circle centered about (p, p ). Hence, we can dene the radial coordinate centered about (p, p ) to be r = (p p ) + (p p ) and write Furthermore, let ˆ π/f h (data ) (p ) region e f h (data ) (p * )r drrdθ = f h (data ) (p* )r = R. 5

6 Choosing a circular region of radius R, we nd ˆ π circular region of radius R e R drrdθ = e ( R) = Hence, we nd R =.45 which is equivalent to an error radius in terms r = (p p ) + (p p ) of.45 = r f h (data (p * ) ). Since p = /f (p (estimate) h (data) ) in exercise, we nd r =. p. This means that we want the maximum and the minimum of (p p ) p p + p p (where we have linearized the dierence) lying on the circle Hence, we want to extremize ( r) = ( p ) + ( p ). (p p ) p p p ( r) ( p ) by taking a derivative with respect to p. One nds (p p ) ± r p + p. Dividing by p p, we nd (p p ) p p = ± r = ± r p p + p = ±. p p (30) Thus, one concludes that the fractional error does not increase by a factor of (which would have given (p p ) p = p p p ) even though one is multiplying two uncertain quantities p (estimate) p (estimate). It increases by a factor close to. The answer to the present question of 95% condence error bar in the prediction is ±5. Instead of going through this long, tedious reasoning, what people typically assume is that the random variable p p has a probability distribution which is Gaussian. In that case, one can calculate the width of the distribution by computing the expectation value of [ (p p )] after linearizing (p p ) = p p +p p : P (p and p data)(p p + p p ) = p,p p,p P (p and p data)(p ( p ) + p ( p ) + p p p p ). The last term which is a cross term evaluates to zero when summed over p and p. The ( p ) and ( p ) terms evaluate to the σ of the Gaussian function (see e.g. Eq. (4)) of the purported probability distribution of p p. Hence, one obtains the eective σ being σ = p σ + p σ. 6

7 This implies that the fractional σ error (which is not a 95% condence error bar) in the quantity of interest p p is σ σ p = p p + σ p. (3) If σ = σ and p = p as for our problem, then one concludes σ p p = σ p suggestive of Eq. (30). The addition of quadratic fractional errors as in Eq. (3) are referred to as adding errors in quadratures. The advantage of adding errors in quadratures is that the computation is utterly simple and gets you in the ball park of the correct answer. The disadvantage is the loss of understanding the assumptions in the approximations (for example, it is not true that p p will be rigorously Gaussian distributed, although it will typically have a similar shape). 5. What are some possible systematic errors associated with the single coin ip probability measurement and the coincident experiment? 7