CS 70 Discrete Mathematics for CS Sprig 2007 Luca Trevisa Lecture 22 Aother Importat Distributio The Geometric Distributio Questio: A biased coi with Heads probability p is tossed repeatedly util the first Head appears. What is the expected umber of tosses? As always, our first step i aswerig the questio must be to defie the sample space Ω. A momet s thought tells us that Ω = {H,T H,TTH,T T T H,...}, i.e., Ω cosists of all sequeces over the alphabet {H,T } that ed with H ad cotai o other H s. This is our first example of a ifiite sample space (though it is still discrete). What is the probability of a sample poit, say ω = T T H? Sice successive coi tosses are idepedet (this is implicit i the statemet of the problem), we have Pr[T T H] = ( p) ( p) p = ( p) 2 p. Ad geerally, for ay sequece ω Ω of legth i, we have Pr[ω] = ( p) i p. To be sure everythig is cosistet, we should check that the probabilities of all the sample poits add up to. Sice there is exactly oe sequece of each legth i i Ω, we have as expected. ω Ω Pr[ω] = ( p) i p = p i=0 ( p) i = p [I the secod-last step here, we used the formula =0 xk = valid for all 0 < x <.] x ( p) =, for summig a geometric series, which is Now let the radom variable X deote the umber of tosses i our sequece (i.e., X(ω) is the legth of ω). Our goal is to compute E(X). Despite the fact that X couts somethig, there s o obvious way to write it as a sum of simple r.v. s as we did i may examples i the last lecture. (Try it!) Istead, let s just dive i ad try a direct computatio. Note that the distributio of X is quite simple: So from the defiitio of expectatio we have Pr[X = i] = ( p) i p for i =,2,3,... ( p)+(2 ( p)p)+(3 ( p) 2 p)+ = p i( p) i. CS 70, Sprig 2007, Lecture 22
This series is a bled of a arithmetic series (the i part) ad a geometric series (the ( p) i part). There are several ways to sum it. Here is oe way, usig a auxiliary trick (give i the followig Theorem) that is ofte very useful. Theorem 22.: Let X be a radom variable that takes o oly o-egative iteger values. The Pr[X i]. Proof: For otatioal coveiece, let s write p i = Pr[X = i], for i = 0,,2,... From the defiitio of expectatio, we have (0 p 0 )+( p )+(2 p 2 )+(3 p 3 )+(4 p 4 )+ = p +(p 2 + p 2 )+(p 3 + p 3 + p 3 )+(p 4 + p 4 + p 4 + p 4 )+ = (p + p 2 + p 3 + p 4 + )+(p 2 + p 3 + p 4 + )+(p 3 + p 4 + )+(p 4 + )+ = Pr[X ]+Pr[X 2]+Pr[X 3]+Pr[X 4]+. I the third lie, we have regrouped the terms ito coveiet ifiite sums. You should check that you uderstad how the fourth lie follows from the third. [Note that our... otatio here is a little iformal, but the meaig should be clear. We could give a more rigorous, but less clear proof usig iductio.] Usig Theorem 2., it is easy to compute E(X). The key observatio is that, for our coi-tossig r.v. X, Pr[X i] = ( p) i. () Why is this? Well, the evet X i meas that at least i tosses are required. This is exactly equivalet to sayig that the first i tosses are all Tails. Ad the probability of this evet is precisely ( p) i. Now, pluggig equatio () ito Theorem 2., we get Pr[X i] = ( p) i = ( p) = p. So, the expected umber of tosses of a biased coi util the first Head appears is p. For a fair coi, the expected umber of tosses is 2. What about the variace? If we wat to compute the variace uig the equatio Var(X) = E(X 2 ) (E(X)) 2, ow that we kow E(X) it remais to fid E(X 2 ). X 2 is a radom variable that takes the value k 2 with probability ( p) k p, ad so we have E(X 2 ) = p k 2 ( p) k To estimate this sum we will use the followig trick. For real umbers 0 < x <, defie the fuctio f(x) = k=0 x k (2) CS 70, Sprig 2007, Lecture 22 2
We kow that we also have f(x) = x Let us kow take the first derivative of f(x). By equatio (2) we have f (x) = kx k, ad by equatio (3) we have f (x) = ( x) 2. Thus, we have foud out that, for 0 < x <, we have kx k = ( x) 2. Note that, after the substitutio x p ad after multiplyig both sides by p, the above equatio gives us a alterative way of provig /p. There is, of course, o reaso to stop at the first derivative. We ca compute the secod derivative of f(x) accordig to the two defiitios, ad fid f (x) = k=2 k (k ) x k 2 ad f (x) = 2 ( x) 3, thus k(k )x k 2 = 2 ( x) 3. [Note that it makes o differece if we start summig from or from 2 o the left-had side.] Now we are ready to compute E(X 2 ): (3) Fially, we have E(X 2 ) = p = p = p( p) k 2 ( p) k (k(k )+k) ( p) k = p( p) 2 p 3 + p = 2 p p 2 + p = 2 p 2 p. k(k )( p) k 2 + p k( p) k Var(X) = E(X 2 ) (E(X)) 2 = 2 p 2 p p 2 = p 2 p. For example, if p = %, the 00, ad Var(X) = 9, 900, so that the stadard deviatio of X is about 99.5. CS 70, Sprig 2007, Lecture 22 3
The geometric distributio The distributio of the radom variable X that couts the umber of coi tosses util the first Head appears has a special ame: it is called the geometric distributio with parameter p (where p is the probability that the coi comes up Heads o each toss). Defiitio 22. (geometric distributio): A radom variable X for which Pr[X = i] = ( p) i p for i =,2,3,... is said to have the geometric distributio with parameter p. If we plot the distributio of X (i.e., the values Pr[X = i] agaist i) we get a curve that decreases mootoically by a factor of p at each step. For posterity, let s record two importat facts we ve leared about the geometric distributio: Theorem 22.2: For a radom variable X havig the geometric distributio with parameter p,. Pr[X i] = ( p) i for i =,2,...; 2. p ; ad 3. Var(X) = p p 2. The geometric distributio occurs very ofte i applicatios because frequetly we are iterested i how log we have to wait before a certai evet happes: how may rus before the system fails, how may shots before oe is o target, how may poll samples before we fid a Democrat, etc. The ext sectio discusses a rather more ivolved applicatio, which is importat i its ow right. The Coupo Collector s Problem Questio: We are tryig to collect a set of differet baseball cards. We get the cards by buyig boxes of cereal: each box cotais exactly oe card, ad it is equally likely to be ay of the cards. How may boxes do we eed to buy util we have collected at least oe copy of every card? The sample space here is similar i flavor to that for our previous coi-tossig example, though rather more complicated. It cosists of all sequeces ω over the alphabet {, 2,..., }, such that. ω cotais each symbol,2,..., at least oce; ad 2. the fial symbol i ω occurs oly oce. [Check that you uderstad this!] For ay such ω, the probability is just Pr[ω] = i, where i is the legth of ω (why?). However, it is very hard to figure out how may sample poits ω are of legth i (try it for the case = 3). So we will have a hard time figurig out the distributio of the radom variable X, which is the legth of the sequece (i.e., the umber of boxes bought). Fortuately, we ca compute the expectatio E(X) very easily, usig (guess what?) liearity of expectatio, plus the fact we have just leared about the expectatio of the geometric distributio. As usual, we would like to write X = X + X 2 +...+ X (4) CS 70, Sprig 2007, Lecture 22 4
for suitable simple radom variables X i. But what should the X i be? A atural thig to try is to make X i equal to the umber of boxes we buy while tryig to get the ith ew card (startig immediately after we ve got the (i )st ew card). With this defiitio, make sure you believe equatio (4) before proceedig. What does the distributio of X i look like? Well, X is trivial: o matter what happes, we always get a ew card i the first box (sice we have oe to start with). So Pr[X = ] =, ad thus E(X ) =. How about X 2? Each time we buy a box, we ll get the same old card with probability, ad a ew card with probability. So we ca thik of buyig boxes as flippig a biased coi with Heads probability p = ; the X is just the umber of tosses util the first Head appears. So X has the geometric distributio with parameter p =, ad E(X 2 ) =. How about X 3? This is very similar to X 2 except that ow we oly get a ew card with probability 2 (sice there are ow two old oes). So X 3 has the geometric distributio with parameter p = 2, ad E(X 3 ) = 2. Arguig i the same way, we see that, for i =,2,...,, X i has the geometric distributio with parameter p = i+, ad hece that E(X i ) = i+. Fially, applyig liearity of expectatio to equatio (4), we get E(X i ) = + + + 2 + = i. (5) This is a exact expressio for E(X). We ca obtai a tidier form by otig that the sum i it actually has a very good approximatio, amely: i l+γ, where γ = 0.5772... is Euler s costat. Thus the expected umber of cereal boxes eeded to collect cards is about (l + γ). This is a excellet approximatio to the exact formula (5) eve for quite small values of. So for example, for = 00, we expect to buy about 58 boxes. This is aother of the little tricks you might like to carry aroud i your toolbox. CS 70, Sprig 2007, Lecture 22 5