CS 70 Discrete Mathematics ad Probability Theory Summer 2014 James Cook Note 15 Some Importat Distributios I this ote we will itroduce three importat probability distributios that are widely used to model realworld pheomea. The first of these which we already leared about i the last ote is the biomial distributio Bi(, p). This is the distributio of the umber of Heads, S, i tosses of a biased coi with probability p to be Heads. Its expected value is E[S ] = p, ad as we saw, P[S = k] = ( k) p k (1 p) k. Geometric Distributio Questio: A biased coi with Heads probability p is tossed repeatedly util the first Head appears. What is the distributio ad the expected umber of tosses? As always, our first step i aswerig the questio must be to defie the sample space Ω. A momet s thought tells us that Ω = {H,TH,TTH,TTTH,...}, i.e, Ω cosists of all sequeces over the alphabet {H,T} that ed with H ad cotai o other Hs. This is our first example of a ifiite sample space (though it is still discrete). What is the probability of a sample poit, say ω = TTH? Sice successive coi tosses are idepedet (this is implicit i the statemet of the problem), we have Pr[TTH] = (1 p) (1 p) p = (1 p) 2 p. Ad geerally, for ay sequece ω Ω of legth i, we have Pr[ω] = (1 p) i 1 p. To be sure everythig is cosistet, we should check that the probabilities of all the sample poits add up to 1. Sice there is exactly oe sequece of each legth i 1 i Ω, we have ω Ω Pr[ω] = (1 p) i 1 p = p i=0 (1 p) i = p 1 1 (1 p) = 1, as expected. (I the secod-last step here, we used the formula for summig a geometric series.) Now let the radom variable X deote the umber of tosses i our sequece (so X(ω) is the legth of ω). Its distributio has a special ame: it is called the geometric distributio with parameter p (where p is the probability that the coi comes up Heads o each toss). Defiitio 15.1 (geometric distributio): A radom variable X for which Pr[X = i] = (1 p) i 1 p for i = 1,2,3,... is said to have the geometric distributio with parameter p. This is abbreviated as X Geom(p). If we plot the distributio of X (the values Pr[X = i] o oe axis ad i o the other) we get a curve that decreases mootoically by a factor of 1 p at each step. See Figure 1. Our ext goal is to compute E(X). Despite the fact that X couts somethig, there s o obvious way to write it as a sum of simple r.vs as we did i may examples i a earlier lecture ote. (Try it!) I a later lecture, CS 70, Summer 2014, Note 15 1
Figure 1: The Geometric distributio. we will give a slick way to do this calculatio. For ow, let s just dive i ad try a direct computatio of E(X). Note that the distributio of X is quite simple: So from the defiitio of expectatio we have Pr[X = i] = (1 p) i 1 p for i = 1,2,3,... E(X) = (1 p) + (2 (1 p)p) + (3 (1 p) 2 p) + = p i(1 p) i 1. This series is a bled of a arithmetic series (the i part) ad a geometric series (the (1 p) i 1 part). There are several ways to sum it. Here is oe way, usig a auxiliary trick (give i the followig Theorem) that is ofte very useful. Theorem 15.1: Let X be a radom variable that takes o oly o-egative iteger values. The E(X) = Pr[X i]. Proof: For otatioal coveiece, let s write p i = Pr[X = i], for i = 0,1,2,... From the defiitio of expectatio, we have E(X) = (0 p 0 ) + (1 p 1 ) + (2 p 2 ) + (3 p 3 ) + (4 p 4 ) + = p 1 + (p 2 + p 2 ) + (p 3 + p 3 + p 3 ) + (p 4 + p 4 + p 4 + p 4 ) + = (p 1 + p 2 + p 3 + p 4 + ) + (p 2 + p 3 + p 4 + ) + (p 3 + p 4 + ) + (p 4 + ) + = Pr[X 1] + Pr[X 2] + Pr[X 3] + Pr[X 4] +. I the third lie, we have regrouped the terms ito coveiet ifiite sums. You should check that you uderstad how the fourth lie follows from the third. Let us repeat the proof, this time usig more compact mathematical otatio: Pr[X i] = j i Pr[X = j] = j=1 i j Pr[X = j] = j Pr[X = j] = E[X] j=1 CS 70, Summer 2014, Note 15 2
Usig Theorem 15.1, it is easy to compute E(X). The key observatio is that, for our coi-tossig r.v. X, Pr[X i] = (1 p) i 1. (1) Why is this? Well, the evet X i meas that at least i tosses are required. This is exactly equivalet to sayig that the first i 1 tosses are all Tails. Ad the probability of this evet is precisely (1 p) i 1. Now, pluggig equatio (1) ito Theorem 15.1, we get E(X) = Pr[X i] = (1 p) i 1 = 1 1 (1 p) = 1 p. So, the expected umber of tosses of a biased coi util the first Head appears is 1 p. For a fair coi, the expected umber of tosses is 2. For posterity, let s record two importat facts we ve leared about the geometric distributio: Theorem 15.2: For a radom variable X havig the geometric distributio with parameter p, 1. E(X) = 1 p ; ad 2. Pr[X i] = (1 p) i 1 for i = 1,2,... The ext sectio discusses a rather more ivolved applicatio, which is importat i its ow right. The Coupo Collector s Problem Questio: We are tryig to collect a set of differet baseball cards. We get the cards by buyig boxes of cereal: each box cotais exactly oe card, ad it is equally likely to be ay of the cards. How may boxes do we eed to buy util we have collected at least oe copy of every card? The sample space here is similar i flavor to that for our previous coi-tossig example, though rather more complicated. It cosists of all sequeces ω over the alphabet {1,2,...,}, such that 1. ω cotais each symbol 1,2,..., at least oce; ad 2. the fial symbol i ω occurs oly oce. (Check that you uderstad this!) For ay such ω, the probability is just Pr[ω] = 1 i, where i is the legth of ω (why?). However, it is very hard to figure out how may sample poits ω are of legth i (try it for the case = 3). So we will have a hard time figurig out the distributio of the radom variable X, which is the legth of the sequece (i.e, the umber of boxes bought). Fortuately, we ca compute the expectatio E(X) very easily, usig (guess what?) liearity of expectatio, plus the fact we have just leared about the expectatio of the geometric distributio. As usual, we would like to write X = X 1 + X 2 +... + X (2) for suitable simple radom variables X i. But what should the X i be? A atural thig to try is to make X i equal to the umber of boxes we buy while tryig to get the ith ew card (startig immediately after we ve got the (i 1)st ew card). With this defiitio, make sure you believe equatio (2) before proceedig. What does the distributio of X i look like? Well, X 1 is trivial: o matter what happes, we always get a ew card i the first box (sice we have oe to start with). So Pr[X 1 = 1] = 1, ad thus E(X 1 ) = 1. CS 70, Summer 2014, Note 15 3
How about X 2? Each time we buy a box, we ll get the same old card with probability 1, ad a ew card with probability 1 1. So we ca thik of buyig boxes as flippig a biased coi with Heads probability p = ; the X 1 is just the umber of tosses util the first Head appears. So X 2 has the geometric distributio with parameter p = 1, ad E(X 2 ) = 1. How about X 3? This is very similar to X 2 except that ow we oly get a ew card with probability 2 (sice there are ow two old oes). So X 3 has the geometric distributio with parameter p = 2, ad E(X 3 ) = 2. Arguig i the same way, we see that, for i = 1,2,...,, X i has the geometric distributio with parameter p = i+1, ad hece that E(X i ) = i + 1. Fially, applyig liearity of expectatio to equatio (2), we get E(X) = E(X i ) = + 1 + + 2 + 1 = 1 i. (3) This is a exact expressio for E(X). We ca obtai a tidier form by otig that the sum i it actually has a very good approximatio 1, amely: 1 l + γ, i where γ = 0.5772... is Euler s costat. Thus the expected umber of cereal boxes eeded to collect cards is about (l +γ). This is a excellet approximatio to the exact formula (3) eve for quite small values of. So for example, for = 100, we expect to buy about 518 boxes. The Poisso distributio Cosider the umber of clicks of a Geiger couter, which measures radioactive emissios. The average umber of such clicks per secod,, is a measure of radioactivity, but the actual umber of clicks fluctuates accordig to a certai distributio called the Poisso distributio. What is remarkable is that the average value,, completely determies the probability distributio of the umber of clicks X. Defiitio 15.2 (Poisso distributio): A radom variable X for which Pr[X = i] = i i! e for i = 0,1,2,... (4) is said to have the Poisso distributio with parameter. This is abbreviated as X Poiss(). The Poisso distributio is also a very widely accepted model for so-called rare evets, such as miscoected phoe calls, radioactive emissios, crossovers i chromosomes, the umber of cases of disease, the umber of births per hour, etc. This model is appropriate wheever the occurreces ca be assumed to happe radomly with some costat desity i a cotiuous regio (of time or space), such that occurreces i 1 This is aother of the little tricks you might like to carry aroud i your toolbox. CS 70, Summer 2014, Note 15 4
disjoit subregios are idepedet. Oe ca the show that the umber of occurreces i a regio of uit size should obey the Poisso distributio with parameter. To make sure this defiitio is valid, we had better check that (4) is i fact a distributio, i.e, that the probabilities sum to 1. We have i i=0 i! e = e i i=0 i! = e e = 1. (I the secod-last step here, we used the Taylor series expasio e x = 1 + x + x2 2! + x3 3! +.) What is the expectatio of a Poisso radom variable X? This is a simple hads-o calculatio, startig from the defiitio of expectatio: E(X) = i=0 i Pr[X = i] = i=0 i i i! e = e i (i 1)! = e i 1 (i 1)! = e e =. So the expectatio of a Poisso r.v. X with parameter is E(X) =. A plot of the Poisso distributio reveals a curve that rises mootoically to a sigle peak ad the decreases mootoically. The peak is as close as possible to the expected value without exceedig it, i.e, at i =. Poisso ad Coi Flips To see this i a cocrete settig, suppose we wat to model the umber of cell phoe users iitiatig calls i a etwork durig a time period, of duratio say 1 miute. There are may payig customers i the etwork, ad all of them ca potetially make a call durig this time period. However, oly a very small fractio of them actually will. Uder this sceario, it seems reasoable to make two assumptios: The probability of havig more tha 1 customer iitiatig a call i ay small time iterval is egligible. The iitiatio of calls i disjoit time itervals are idepedet evets. The if we divide the oe-miute time period ito disjoit itervals, tha the umber of calls X i that time period ca be modeled as biomially distributed with parameter ad probability of success p, the probability of havig a call iitiated i a time iterval of legth 1/. But what should p be i terms of the relevat parameters of the problem? If calls are iitiated at a average rate of calls per miute, the E(X) = ad so p =, i.e. p = /. So X Bi(,/). As we shall see below, as we let ted to ifiity, this distributio teds to the Poisso distributio with parameter. We ca also see why the Poisso distributio is a model for rare evets. We are thikig of it as a sequece of a large umber,, of coi flips, where we expect oly a fiite umber of Heads. Now we will prove that the Poisso distributio is the limit of the biomial distributio as. Let s look i more detail at the distributio of X as defied above. Recall that X is defied to be a special case of the biomial distributio, with p = ad. For coveiece, we ll write p i = Pr[X = i] for i = 0,1,2,... We will first determie what the distributio is (i.e. the values of p i ) ad the show that the expectatio is. CS 70, Summer 2014, Note 15 5
Begiig with p 0, we have ( p 0 = 1 ) e as. What about the other p i? Well, we kow from the biomial distributio that p i = ( ) i ( )i (1 ) i. Sice we kow how p 0 behaves, let s look at the ratio p 1 p 0 : p 1 p 0 = (1 ) 1 (1 ) = 1 = as. (Recall that we are assumig is a costat.) So, sice p 0 e, we see that p 1 e as. Now let s look at the ratio p 2 p 1 : ( ) p 2 2 ( = )2 (1 ) 2 p 1 ( ) (1 = 1 ) 1 2 1 (1 ) = 1 2 2 as. So p 2 2 2 e as. For each value of i, somethig very similar happes to the ratio p i p i 1 : p i p i 1 = ( ) i ( ) ( ( i 1 )i (1 ) i )i 1 (1 = i + 1 ) i+1 i Puttig this together, we see that, for each fixed value i, = i + 1 i i as. p i i i! e as. This is exactly accordig to our defiitio of the Poisso distributio, where Pr[X = i] = i i! e. We have see that the Poisso distributio arises as the limit of the umber of balls i bi 1 whe balls are throw ito bis. I other words, it is the limit of the biomial distributio with parameters ad p = as, beig a fixed costat. CS 70, Summer 2014, Note 15 6