Introduction to Probability and Statistics Winter 2015

Transcription

1 CALIFORNIA INSTITUTE OF TECHNOLOGY Ma 3/13 KC Border Introduction to Probability and Statistics Winter 215 Lecture 11: The Poisson Process Relevant textbook passages: Pitman [15]: Sections 2.4,3.8, 4.2 Larsen Marx [14]: Sections 3.8, 4.2, What makes a density? Any function f : R + R + such that f(x) dx < can be turned into a probability density function by normalizing it. That is, if the real number satisfies c f(x) dx, then f(x)/c is a probability density. Likewise for functions f : R R + such that f(x) dx <. The constants c are sometimes called normalizing constants, and they account for the odd look of many densities. For instance, /2 e z2 dz 2π is the normalizing constant for the Normal family. Much space is devoted in introductory statistics and probability textbooks to computing various integrals. In this course, I shall not spend time in lecture on the details of evaluating integrals. My view is that the evaluation of integrals, while a necessary part of the subject, frequently offers little insight. You all have had serious calculus classes recently, so you are probably better at integration than I am these days. On the occasions where it does provide some insight, we may spend some time on it. I do recommend the exposition in Pitman [15, Section 4.4, pp ] and Larsen Marx [14, Section 3.8, pp ]. We shall now consider some other important densities and pmfs The Gamma Function Definition The Gamma function is defined by Γ(s) t s 1 e t dt. Larsen Marx [14]: Section 4.6, pp Pitman [15]: p

2 Ma 3/13 Winter 215 KC Border The Poisson Process 11 2 The Gamma function is a continuous version of the factorial, and has the property that Γ(s + 1) sγ(s) for every s >, and Γ(m) (m 1)! for every natural number m. In particular, Γ(2) Γ(1) 1, Γ(1/2) π. There is no closed form formula for the Gamma function except at integer multiples of 1/2. See for instance, Pitman [15, pp ] or Apostol [2, pp ] for the cited properties, which follow from integration by parts Pitman [15]: Exercise , p. 294; p The Gamma family of distributions The density of the general Gamma(r, λ) distribution is given by f(t) λr Γ(r) tr 1 e λt (t > ). (1) v ::11.22 KC Border

3 Ma 3/13 Winter 215 KC Border The Poisson Process 11 3 The parameter r is referred to as the shape parameter or index and λ is a scale parameter. Why is it called the scale parameter? T Gamma(r, λ) λt Gamma(r, 1) It has mean and variance given by E X r λ, Var X r λ 2. According to Pitman [15, p. 291], In applications the distribution of a random variable may be unknown, but reasonably well approximated by some gamma distribution. ( λ) 2. λ.5 λ1 λ KC Border v ::11.22

4 Ma 3/13 Winter 215 KC Border The Poisson Process 11 4 ( λ) λ.5 λ1 λ ( λ).5 λ.5 λ1 λ v ::11.22 KC Border

5 Ma 3/13 Winter 215 KC Border The Poisson Process 11 5 ( λ) λ.5 λ1 λ ( ).5 r.5 r 1 r KC Border v ::11.22

6 Ma 3/13 Winter 215 KC Border The Poisson Process 11 6 ( ) r.5 r 1 r v ::11.22 KC Border

7 Ma 3/13 Winter 215 KC Border The Poisson Process 11 7 Hey: Read Me There are (at least) three incompatible, but easy to translate, naming conventions for the Gamma distribution. Pitman [p. 286][15] and Larsen and Marx [14, Defn , p. 272] refer to their parameters as r and λ, and call the function in equation (1) the Gamma(r, λ) density. Note that the shape parameter is the first parameter and the scale parameter is the second parameter for Pitman and Larsen and Marx. This is the convention that I used above in equation (1). Feller [p. 47][12] calls the scale parameter α instead of λ, and he calls the shape parameter ν instead of r. Cramér [p. 126][8] also calls the scale parameter α instead of λ, but the shape parameter he calls λ instead of r. Other than that they agree that equation (1) is the Gamma density, but they list the parameters in reverse order. That is, they list the scale parameter first, and the shape parameter second. Casella and Berger [4, eq , p. 99] call the scale parameter β and the shape parameter α, and list the shape parameter first and the scale parameter second. But here is the confusing part, their scale parameter β is our 1/λ. a Mathematica [2] and R [16] also invert the scale parameter. To get my Gamma(r, λ) density in Mathematica 8, you have to call PDF[GammaDistribution[r, 1/λ], t], to get it in R, you would call dgamma(t, r, rate 1/λ). I m sorry. It s not my fault. But you do have to be careful to know what convention is being used. a That is, C B write the gamma(α, β) density as 1 Γ(α)β α tα 1 e t/β Random Lifetime For a lifetime or duration T chosen at random according to a density f(t) on [, ), and cdf F (t), the survival function is Pitman [15]: 4.2 G(t) P (T > t) 1 F (t) t f(s) ds. When T is the (random) time to failure, the survival function G(t) at epoch t gives the probability of surviving (not failing) until t. Note the convention that the present is time t, and durations are measured as times after that. KC Border v ::11.22

8 Ma 3/13 Winter 215 KC Border The Poisson Process 11 8 Pitman [15]: 4.3 Aside: If you ve ever done any programming involving a calendar, you know the difference between a point in time, called an epoch by probabilists, and a duration, which is the difference between two epochs. The hazard rate λ(t) is defined by ( ) P T (t, t + h) T > t λ(t) lim. h h Or λ(t) f(t) G(t). Proof : By definition, P ( T (t, t + h) T > t ) P (T (t, t + h)) P (T > t) P (T (t, t + h)). G(t) Moreover P (T (t, t + h)) F (t + h) F (t), so the limit is just F (t)/g(t) f(t)/g(t). The hazard rate f(t)/g(t) is often thought of as the instantaneous probability of death or failure The Exponential Distribution The Exponential(λ) is widely used to model random durations or times. It is another name for the Gamma(1, λ) distribution. That is, the random time T has an Exponential(λ) distribution if it has density and cdf which gives survival function and hazard rate f(t) λe λt (t ), F (t) 1 e λt, G(t) e λt, λ(t) λ. That is, it has a constant hazard rate. The only distribution with a constant hazard rate λ > is the Exponential(λ) distirbution. v ::11.22 KC Border

9 Ma 3/13 Winter 215 KC Border The Poisson Process 11 9 The mean an Exponential(λ) random variable is given by λte λt dt Proof : Use the integration by parts formula: h g hg g h, with h (t) λe λt and g(t) t (so that h(t) e λt and g (t) 1) to get E T λte λt dt te λt + te λt + 1 e λt λ 1 λ. e λt dt λ e λt The variance of an Exponential(λ) is 1 λ 2. Proof : Var T E(T 2 ) (E T ) 2 t 2 λe λt dt 1 λ 2. Setting h (t) λe λt and g(t) t 2 and integrating by parts, we get t 2 e λt + 2 te λt dt 1 }{{} λ 2 E T/λ + 2 λ 2 1 λ 2 1 λ 2. KC Border v ::11.22

10 Ma 3/13 Winter 215 KC Border The Poisson Process The Exponential is Memoryless A property that is closely related to having a constant hazard rate is that the exponential distribution is memoryless in that for an Exponential random variable T, Pitman [15]: p. 279 P ( T > t + s T > t ) P (T > s), (s > ). To see this, recall that by definition, P ( T > t + s T > t ) P ((T > t + s) (T > t)) P (T > t) P (T > t + s) as (T > t + s) (T > t) P (T > t) G(t + s) G(t) e λ(t+s) e λt e λs G(s) P (T > s). In fact, the only memoryless distributions are Exponential. Proof : Rewrite memorylessness as or G(t + s) G(t) G(s), G(t + s) G(t)G(s) (t, s > ). It is well known that this last property (plus the assumption of continuity at least one point) is enough to prove that G must be an exponential (or identically zero) on the interval (, ). See J. Aczél [1, Theorem 1, p. 3] Joint distribution of Independent Exponentials Pitman [15]: p. 352 Let X Exponential(λ) and Y Exponential(µ) be independent. Then f(x, y) λe λx µe µy λµe λx µy, 1 Aczél [1] points out that there is another kind of solution to the functional equation when we extend the domain to [, ), namely G() 1 and G(t) for t > 1. v ::11.22 KC Border

11 Ma 3/13 Winter 215 KC Border The Poisson Process so P (X < Y ) y λe λx µe µy dx dy ) λe λx dx dy ( y µe µy ( µe µy e λx y) dy µe µy ( 1 e λy) dy µe µy dy }{{} 1 1 µ λ + µ 1 µ λ + µ λ λ + µ. µe (λ+µ)y dy (λ + µ)e (λ+µ)y dy } {{ } The sum of independent Exponentials Let X and Y be independent and identically distributed Exponential(λ) random variables. The density of the sum for t > is given by the convolution: f X+Y (t) t t f X (t y)f Y (y) dy λe λ(t y) λe λy dy λ 2 e λt dy tλ 2 e λt. since f Y (t y) if y > t Pitman [15]: pp This is a Gamma(2, λ) distribution. More generally, the sum of n independent and identically distributed Exponential(λ) random variables has a Gamma(n, λ) distribution, given by t n 1 f(t) λ n e λt (n 1)! Survival functions and moments For a nonnegative random variable with a continuous density f, integration by parts allows us to prove the following. KC Border v ::11.22

12 Ma 3/13 Winter 215 KC Border The Poisson Process Proposition Let F be a cdf with continuous density f on [, ). Then the p th moment can be calculated as x p f(x) dx px p 1( 1 F (x) ) dx px p 1 G(x) dx. Proof : Use the integration by parts formula: h g hg g h, with h (x) f(x) and g(x) x p (so that h(x) F (x) and g (x) px p 1 ) to get and let b. b x p f(x) dx x p F (x) b b px p 1 F (x) dx b b p F (b) px p 1 F (x) dx b b F (b) px p 1 dx px p 1 F (x) dx b px p 1( F (b) F (x) ) dx, In particular, the first moment, the mean, is given by the area under the survival function: E ( ) 1 F (x) dx G(x) dx. Larsen Marx [14]: Section 4.3 Pitman [15]: p The Poisson(λ) distribution The Poisson(λ) distribution is a discrete distribution that is concentrated on the nonnegative integers, and is sometimes used as a counter of rare events. For a random variable X with the Poisson(λ) distribution, where λ >, the probability mass function is P (X k) p λ (k) e λ λk, (k, 1, 2, 3,... ). k! To see why k p λ (k) 1 consult the supplementary notes on series. The Poisson distribution is named for the French mathematician Siméon Denis Poisson. But there is a model of fishing that predicts that the number of fish caught per hour by an angler follows a Poisson distribution. v ::11.22 KC Border

13 Ma 3/13 Winter 215 KC Border The Poisson Process μ5 μ1 μ2 μ λ5 λ1 λ2 λ KC Border v ::11.22

14 Ma 3/13 Winter 215 KC Border The Poisson Process The mean and variance of the Poisson If X has a Poisson(λ) distribution, then A similar argument shows that E X λ k e λ k1 j λ. j λ λk ke k! λ λλj e j! λ k (k 1)! λ λj e j! Var X λ The Poisson approximates the Binomial Another useful property of the Poisson is that approximates the Binomial distribution is a peculiar way. Fix λ and let X n have the Binomial distribution Binomial(n, λ/n). A n gets large, for each k we have ( ) ( ) k ( n λ P (X n k) 1 λ ) n k k n n ( ) k ( n(n 1)(n 2) (n k + 1) λ 1 λ k! n n n (n 1) (n k+1) ( ) k ( n n n λ n k 1 λ ) n k k! n n n (n 1) n n (n k+1) ( n λ n k k! n (1 (1) n )(1 2 n n λ λk e k!, 1 k! λk 1 e λ which is the Poisson(λ) probability of k. ) k ( ) n k ) n k 1 λ n (k 1) ) (1 ) ( n λ k 1 λ ) k ( 1 λ ) n k! n n v ::11.22 KC Border

15 Ma 3/13 Winter 215 KC Border The Poisson Process ( / ) ( ) Binomial Poisson ( / ) ( ) Binomial Poisson How might the Poisson distribution arise? The following story is a variation on one told by Feller [11, Section VI.6, pp ] and Pitman [15, 3.5, pp ], where the describe the Poisson scatter. This is not quite that phenomenon. Consider a number m of BBs (small metal balls) that are to be scattered KC Border v ::11.22

16 Ma 3/13 Winter 215 KC Border The Poisson Process among n bins. Assume that for each BB i, Let E i,b be the event that BB i hits Bin b. Prob(E i,b ) 1 n, (each bin is equally likely to be hit by BB i), and that for distinct BBs i 1,..., i k and any collection b 1,..., b k of bins, the events E is,b s, s 1,..., k are independent. Set λ m n, and let p λ (k) be the Poisson(λ) probability mass function. Then, fixing λ and k, if n is large enough, the number of bins with k hits np λ (k). (2) This result was called the Law of Small Numbers by von Bortkiewicz [18]. Here is a mildly bogus argument to convince you that it is plausible: Pick a bin, say Bin b and pick some number k of hits. The probability that BB i hits Bin b is 1/n λ/m. So the number of hits on Bin b, has a Binomial(m, λ/m) distribution, which for fixed λ and large m is approximated by the Poisson(λ) distribution, so Prob (Bin b contains k BBs) p λ (k). But because the total number of BBs, m, is fixed, the number of balls in Bin b and Bin c are not independent. (In fact, the joint distribution is one giant multinomial.) So I can t simply multiply this by n bins to get the number of bins with k hits. So imagine independently replicating this experiment r times. Say the experiment itself is a success if Bin b has k hits. The probability of a successful experiment is thus p λ (k). By the Law of Large Numbers, the number of successes in a large number r of experiments is close to rp λ (k). Now note that there is nothing special about Bin b, there are n bins, so summing over all bins and all replications one would expect that the number of bins with k hits would be n times the number of experiments in which Bin b has k hits (namely, rp λ (k)). Thus all together, in the r replications there about nrp λ (k) bins with k hits. Since all the replications are the same experiment, there should be about nrp λ (k) np λ (k) r bins with k hits per experiment. In this argument, I did a little handwaving (using the terms close and about). Note though that r has to be chosen after k, so we don t expect (2) to hold for all values of k, just the smallish ones. v ::11.22 KC Border

17 Ma 3/13 Winter 215 KC Border The Poisson Process Applications of the Poisson distribution Does the argument I just gave you work? There are many stories of data that fit this model, and many are told without any attribution. Many of these examples can ultimately be traced back the very carefully written book by William Feller [1] in 195. (I have the third edition, so will cite it.) During the Second World War, Nazi Germany use unmanned aircraft, the V1 Buzz Bombs, to attack London. (They weren t quite drones, since they were never designed to return or to be remote controlled. Once launched, where they came down was reasonably random.) Feller [11, pp ] cites R. D. Clarke [7] (an insurance adjuster for The Prudential), who reports that 144 square kilometres of South London was divided into 576 sectors of about 1/4 square kilometre, and the number of hits in each sector was recorded. There were 537 Buzz Bombs that hit south London, so λ 537/ Our model then predicts that the number of districts with k hits should be approximately 576p.9323 (k). Here is the actual data compared to the model s prediction: No. of Hits k: No. of Sectors with k hits: Poisson model prediction: That is amazingly close. The bins don t have to be geographical, they can be temporal. So distributing a fixed average number of events per time period over many independent time periods, should also give a Poisson distribution. Indeed Chung [6, p. 196] cites John Maynard Keynes [13, p. 42], who reports that Ladislaus von Bortkiewicz [18] reports that the distribution of the number of cavalrymen killed from being kicked by horses is described by a Poisson distribution! Here is Keynes s table. It covers the years and fourteen different Prussian Cavalry Corps. Annual Number of Casualties CorpsYears with N casualties. N Actual Theoretical KC Border v ::11.22

18 Ma 3/13 Winter 215 KC Border The Poisson Process Keynes [13, p. 42] also reports that von Bortkiewicz [18] reports that the distribution of the annual number of the number of child suicides follows a Poisson distribution. Chung [6, p. 196] also lists the following as examples of Poisson distributions. The number of color blind people in a large group. The number of raisins in cookies. (Who did this research?) The number of misprints on a page. (Again who did the counting? 2 ) Feller [ VI.7, pp ][11] lists these additional phenomena, and supplies citations to back up his claims. An experiment by Rutherford, Chadwick, and Ellis on the emission of α- particles for N 268 time intervals of 7.5 seconds each. Feller cites Harald Cramér [8, p. 436], who reports remarkable agreement with the Poisson. See the first three columns in Figure The number of chromosome interchanges in cells subjected to X-ray radiation. [5] Telephone connections to a wrong number. (Frances Thorndike [17]) Bacterial and blood counts. The Poisson distribution describes the number of occurrences of a rare phenomenon in independent large samples Sums of independent Poissons Let X be Poisson(µ) and Y be Poisson(λ) and independent. Poisson(µ + λ). Convolution: n P (X + Y n) P (X j, Y n j) j n j n j P (X j) P (Y n j) µ µj λ n j e j! e λ (n j)! (µ+λ) (µ + λ)n e, n! Then X + Y is 2 According to my late coauthor, Roko Aliprantis, Apostol s Law states there are an infinite number of misprints in any book. The proof is that every time you open a book, you find another misprint. v ::11.22 KC Border

19 Ma 3/13 Winter 215 KC Border The Poisson Process Figure Cramér [8, p. 436] KC Border v ::11.22

20 Ma 3/13 Winter 215 KC Border The Poisson Process 11 2 where the last step comes from the binomial theorem: n (µ + λ) n n! j!(n j)! µj λ n j. j Pitman [15]: 4.2; and pp The Poisson Arrival Process The Poisson arrival process is a mathematical model that is useful in modeling the number of events (called arrivals) over a continuous time period. For instance the number of telephone calls per minute, the number of Google queries in a second, the number of radioactive decays in a minute, the number of earthquakes per year, etc. In these phenomena, the events are rare enough to be counted, and to have measurable delays between them. (Interestingly, the Poisson model is not a good description of LAN traffic, see [3, 19].) The Poisson arrival process with parameter λ works like this: Let W 1, W 2,... be a sequence of independent and identically distributed Exponential(λ) random variables, representing waiting times for an arrival, on the sample space (Ω, E, P ). At each ω Ω, the first arrival happens at time W 1 (ω), the second arrival happens a duration W 2 (ω) later, at W 1 (ω) + W 2 (ω). The third arrival happens at W 1 (ω) + W 2 (ω) + W 3 (ω). Define T n W 1 + W W n. This is the epoch when the n th event occurs. variables is a nondecreasing sequence. The sequence T n of random An alternative description is this: Arrivals are scattered along the interval [, ) so that the number of arrival in disjoint intervals are independent, and the expected number of arrivals in an interval of length t is λt. For each ω we can associate a step function of time, N(t) defined by N(t) the number of arrivals that have occurred at a time t the number of indices n such that T n t Remark Since the function N depends on ω, I should probably write N(t, ω) the number of indices n such that T n (ω) t. But that is not traditional. Something a little better than no mention of ω that you can find, say in Doob s book [9] is a notation like N t (ω). But most of the time we want to think of N as a random function of time, and putting t in the subscript disguises this. v ::11.22 KC Border

21 Ma 3/13 Winter 215 KC Border The Poisson Process Definition The random function N is called the Poisson process with parameter λ. So why is this called a Poisson Process? Because N(t) has a Poisson(λt) distribution. There is nothing special about starting at time t. The Poisson process looks the same over every time interval. The Poisson process has the property that for any interval of length t, the distribution of the number of arrivals is Poisson(λt) Stochastic Processes A stochastic process is a set {X t : t T } of random variables on (Ω, E, P ) indexed by time. The time set T might be the natural numbers or integers, a discrete time process; or an interval of the real line, a continuous time process. Each random variable X t, t T is a function on Ω. The value X t (ω) depends on both ω and t. Thus another way to view a stochastic process is as a random function on T. In fact, it is not uncommon to write X(t) instead of X t. The Poisson process is a continuous time process with discrete jumps at exponentially distributed intervals. Other important examples of stochastic processes include the Random Walk and its continuous time version, Brownian motion. Bibliography [1] J. D. Aczél. 26. Lectures on functional equations and their applications. Mineola, NY: Dover. Reprint of the 1966 edition originally published by Academic Press. An Errtat and Corrigenda list has been added. It was originally published une the title Vorlesungen über Funktionalgleichungen and ihre Anwendungen, published by Birkhäuser Verlag, Base, [2] T. M. Apostol Calculus, 2d. ed., volume 1. Waltham, Massachusetts: Blaisdell. [3] J. Beran, R. P. Sherman, M. S. Taqqu, and W. Willinger Variable-bitrate video traffic and long-range dependence. IEEE Transactions on Communications 43(2/3/4): DOI: 1.119/ [4] G. Casella and R. L. Berger. 22. Statistical inference, 2d. ed. Pacific Grove, California: Wadsworth. KC Border v ::11.22

22 Ma 3/13 Winter 215 KC Border The Poisson Process [5] D. G. Catchside, D. E. Lea, and J. M. Thoday Types of chromosomal structural change induced by the irradiation of Tradescantia microspores. Journal of Genetics 47: [6] K. L. Chung Elementary probability theory with stochastic processes. Undergraduate Texts in Mathematics. New York, Heidelberg, and Berlin: Springer Verlag. [7] R. D. Clarke An application of the Poisson distribution. Journal of the Institute of Actuaries 72:481. [8] H. Cramér Mathematical methods of statistics. Number 34 in Princeton Mathematical Series. Princeton, New Jersey: Princeton University Press. Reprinted [9] J. L. Doob Stochastic processes. New York: Wiley. [1] W. Feller An introduction to probability theory and its applications, 1st. ed., volume 1. New York: Wiley. [11] An introduction to probability theory and its applications, 3d. ed., volume 1. New York: Wiley. [12] An introduction to probability theory and its applications, 2d. ed., volume 2. New York: Wiley. [13] J. M. Keynes A treatise on probability. London: Macmillan and Co. [14] R. J. Larsen and M. L. Marx An introduction to mathematical statistics and its applications, fifth ed. Boston: Prentice Hall. [15] J. Pitman Probability. Springer Texts in Statistics. New York, Berlin, and Heidelberg: Springer. [16] R Core Team R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. [17] F. Thorndike Applications of Poisson s probability summation. Bell System Technical Journal 5(4): [18] L. von Bortkiewicz Das Gesetz der kleinen Zahlen [The law of small numbers]. Leipzig: B.G. Teubner. [19] W. Willinger, M. S. Taqqu, R. P. Sherman, and D. V. Wilson Selfsimilarity through high variability: Statistical analysis of ethernet LAN traffic at the source level (extended version). IEEE/ACM Transactions on Networking 5(1): DOI: 1.119/ v ::11.22 KC Border

23 Ma 3/13 Winter 215 KC Border The Poisson Process [2] Wolfram Research, Inc. 21. Mathematica 8.. Champaign, Illinois: Wolfram Research, Inc. KC Border v ::11.22

24