Lecture 7: Statistics and the Central Limit Theorem Philip Moriarty, philip.moriarty@nottingham.ac.uk NB Notes based heavily on lecture slides prepared by DE Rourke for the F32SMS module, 2006
7.1 Recap and Overview In Lectures 1 6 we have seen how Fourier analysis is at the core of a variety of fields in physics (and science in general) with a particular recent focus on signal and image analysis/processing. In the final two lectures of the F32SMS module we will examine how Fourier transforms (FTs) can be used (perhaps surprisingly) in two areas rather different to those studied in this module to date: statistics and the solution of differential equations. Today we will see how Fourier analysis can solve a central problem in statistics the derivation of the central limit theorem. In doing so you will see why the Gaussian distribution is so prevalent in Nature and find a remarkable link between convolution and the statistics of random processes. 7.2 Probability Density Functions You will be familiar with the concept of a probability density function from the 1 st year Thermal & Kinetic module and your 1 st and 2 nd year quantum mechanics-related modules. Let s start with considering the distributions produced by perhaps the simplest random number generator: dice.? Sketch the probability density function you would expect for the numbers generated by the throw of a single die.? Sketch the probability density function you would expect for the numbers generated by the throw of two dice.
Let s try another example. We buy a random apple from a supermarket and we weigh it. The weight is an example of a random variable. The apple is unlikely to weigh exactly 1 kg or 1g there will be a most probable mass. The probabilities for the mass are reflected in the probability density function (PDF), φ(x), shown in Fig. 1 below. Fig. 1 Probability density function for the mass of an apple. The area under the φ(x) curve between, for example, a and b gives the probability, p, that the mass takes a value between a and b. That is, This of course means that the total area under the φ(x) curve, if the function is appropriately normalised, must be 1, since the probability of x lying between - and + must be 1.? Given a normalised probability density function φ(x), how would you find the mean value of x, i.e. <x>? Similarly, how would you find the variance, σ 2, where σ 2 = <(x-<x>)> 2?
The property that is taken as the defining property of a PDF is : Now, for simplicity, let s assume that the PDF for the weights of the apples is the uniform distribution (i.e. like that for the single die example) shown in Fig. 2 below. Fig. 2 A uniform probability distribution function. This means that no apples weigh less than 90g or more than 110 g and that any weight between 90 and 110 is equally likely. The average mass is clearly 100 g and, perhaps less clearly, the standard deviation, σ, is a/ 3 where a is the half-width of the distribution.? Why is the height of the function 0.05 g -1? Again, see the 1 st year Thermal & Kinetic notes (available at www.nottingham.ac.uk/~ppzpjm/f31st1)
Now, suppose we buy two apples and calculate the sum of the two masses, S = x 1 + x 2. What is the PDF for the sum, S? As each individual apple weighs between 90g and 110 g, there s no way the sum can be less than 180g or more than 220g, so the PDF must be zero for S<180g or S>220g. Similarly, what is the PDF for the sum of the masses if we buy three apples, four apples, or, in general, n apples? What happens in the limit n? These are the questions we ll address in this lecture. A very important result called the central limit theorem tells us what happens in the n limit. Importantly, while we ll deal with apples, you ll not be surprised to know that the central limit theorem is not restricted to the analysis of fruit! It is a general theorem which is applicable, in particular, to the measurement of quantities in an experiment. If we try to measure a quantity, Q, in an experiment, the measured value will on the whole equal Q plus random noise. If the measurement is repeated n times (assuming n is large) and the values summed together, one would expect to average out the noise, and so the sum calculated will be near to nq. This means that the PDF should be strongly peaked at nq.
7.3 Numerical experiments PDFs with Matlab Fig.3 below shows the results of a Matlab program to generate random numbers between 90 and 110. Note that the random number generator produces a flat, i.e. uniform distribution. Fig. 3 A uniform probability distribution function generated by Matlab? From what we ve done before, perhaps you can guess what the PDF for the sum, S, of two random numbers (x 1 and x 2 ) should look like? and as we increase the number of random values we use in the sum, the distribution function approaches a limiting (and hopefully very familiar) form:
7.4 Analytically calculating PDFs Although numerical experiments provide a lot of insight, it is of course much more elegant (and efficient) if we can analytical derive an equation for the PDF. Let s look at the two apple case first of all, as shown in Fig. 4. Fig. 4 Probability distribution function for, S, the sum of the masses of two apples. We need to bear in mind the definition of φ s (S): That is, φ s (S) ds is the probability of the total weight of two apples falling between S and S + ds. We now need to work out what this probability is. (It s best to refer to the Powerpoint slides for Lecture 7 at this point if you re not already doing so.) Imagine whenever we pick an apple of weight x 1 followed by an apple of weight x 2, we draw a dot on a 2D graph at co-ordinates x 1, x 2 (see Fig. 5(a) below). The probability of a dot landing in a square defined by the ranges x 1 x 1 + dx 1 and x 2 x 2 + dx 2 is then given by p (see Fig. 5). Fig. 5(a) Probability of choosing apple weights within given ranges.
Suppose we re interested not in x 1 and x 2 individually but in the sum, S, of x 1 and x 2. If S has some fixed value then x 1 and x 2 must lie on the line shown in Fig. 5(b) below. Fig. 5(b)
On the other hand, if the sum equals S + ds then x 1 and x 2 must lie on the line shown in Fig. 5(c) below. Fig. 5(c)
The probability of our dot landing between the two lines is the probability written above (i.e. φ(x 1 )φ(x 2 )dx 1 dx 2 ) but integrated over the total area between the two lines. For a fixed x 1, x 2 can range from S-x 1 to S+dS-x 1. Thus, we can write down the probability as follows: Although this looks a little formidable, the integral isn t too scary! The first thing we can do is move φ(x 1 ) to outside the inner integration. Although it at first appears surprising that we can do this, remember the following formula: That means we can write the integral involving x 1 and x 2 as shown on the following page.
That is, And now perhaps you start to see the link with Fourier transforms. The convolution of two functions f and g can be written as: In words, the PDF for the sum of x 1 and x 2 is the self-convolution of the PDF for a single apple s weight, φ.
The numerical experiments (and our dice games) suggested that for two apples or two dice the PDF should be triangular. This is exactly what we d expect from the convolution of two top hat functions! (See Problems Class 3 and Fig. 6 below). Fig. 6 Convolution of the single apple probability distribution functions yields the PDF for, S, the sum of the weights of two apples. It s then easy to generalise this result to the PDF for the total weight of 3 or more apples. As shown in Fig. 7, we convolve φ with itself n times to get the PDF for the total weight of n apples. Fig. 7 Use of convolution to derive PDF for weight of n apples.
7.5 PDFs and Fourier Transforms Let s now consider using Fourier transforms to carry out the convolution (i.e. let s switch to k-space (and back again)). Fig. 8 shows the Fourier transform of the PDF which, as the PDF is a top-hat function, you should know is a sinc function. (? Why is the exp(- ik<x>) term required?) If the sinc function is raised to the tenth power (i.e. if we are considering 10 apples), then the function that results is shown in the bottom right corner of Fig.8. To get the resulting PDF function we have to move from k-space and thus need to use an inverse Fourier transform. This yields the function shown in the bottom left corner of Fig. 8. Fig. 8 Convolution in Fourier space. The PDF for the total weight of ten apples is shown in the bottom left corner.
7.6 The central limit theorem. While the analytical expression for the final PDF of the total weight of ten apples is too horrible to write down, there is an approximate formula which gets more and more accurate the more apples we choose to include, i.e. the larger n is. This formula is shown in Fig. 9 and while it too perhaps looks horrible (!), it is nothing more than the Gaussian or normal distribution. This is called the central limit theorem: φ s (S) tends to a Gaussian distribution for large n. The central limit theorem is just a statement about the result of convolving a function with itself n times. It doesn t matter what the shape of the PDF is the central limit theorem tells use that if we convolve it with itself n times (where n is a large number), we ll get a Gaussian. This is a remarkable result: even if we don t know how noise is affecting our measurements, if we make enough measurements, we know that their sum must have a Gaussian distribution! Fig. 9 Upper: Equation for, and sketch of, a Gaussian distribution; Lower: A top-hat function convolved with itself 10 times produces a Gaussian distribution.