Expectation, inequalities and laws of large numbers

Transcription

1 Chapter 3 Expectation, inequalities and laws of large numbers 3. Expectation and Variance Indicator random variable Let us suppose that the event A partitions the sample space S, i.e. A A S. The indicator of an event A is the (indicator) random variable I A defined by { if s A I A (s) 0 if s A The event A occurs if and only if I A. The probability distribution is p IA (0) P (A) P (A) p IA () P (A). The corresponding distribution function reads 0 for x < 0 F IA (x) P (A) for 0 x < for x. Exercise 3. (Indicator random variable) Consider a probabilistic space over a set S. Show that for every event A S and its indicator I A it holds E (I A ) P(A). (An indicator is defined as I A (w) for all w A and I A (w) 0 for all w / A.) Solution of Exercise 3. : E (I A ) w S P(w)I A (w) w A P(w)+ w (S A) ( ) P(w)0 P {w} +0 P(A). w A

2 CHAPTER 3. EXPECTATION, INEQUALITIES AND LAWS OF LARGE NUMBERS Exercise 3. Consider two discrete random variables X, Y such that w S : X(w) Y (w). Prove that E (X) E (Y ). Solution of Exercise 3. : E (Y ) E (X) w S P(w)Y (w) w S w S P(w)0 0. P(w)X(w) w S P(w)[Y (w) X(w)] Exercise 3.3 Suppose that after a long night n drunken sailors return to the ship and they sequentially, independently at random enter one of r cabins and fall asleep. Assuming they select each cabin with uniform probability distribution (and ignore any other sailors already present inside), what is the expected number of empty cabins? Solution of Exercise 3.3 : Let X i if cabin i is empty and let X i 0 otherwise. The number of empty cabins is X r X i and we want to calculate E(X). Using the fact that the expectation of a sum is the sum of expectations (even for not independent random variables), we have ( r ) E(X) E X i r E(X i ). We have to compute E (X i ). Probability that i-th cabin is empty is ( r )n (this follows from the negative binomial probability distribution where one of the parameters is equal to 0). We have E (X i ) ( r ) n and ( E(X) r ) n. r Exercise 3.4 Let X be uniformly distributed on {0,,..., n}. Find the mean and variance of X. Solution of Exercise 3.4 :. Mean: E (X) i n + n + i0 i n(n + ) n n +

3 3.. EXPECTATION AND VARIANCE 3. Variance: ( ) V ar(x) E(X ) (E(X)) i ( n ) n + i0 n(n + )(n + ) ( n ) n + 6 n(n + ) n 6 4 4n + n 3n n + n Exercise 3.5 Having two dice, let the random variable X be the outcome of the first die and Y be the maximum of their outcomes. Compute E(X), E(Y ), V ar(x), Cov(X, Y ) and the joint distribution of X and Y. Solution of Exercise 3.5 : E(X) E(Y ) E(X ) 6 6 i 6 6 i 7 6 (i + (i )) i 6 ( ) 6 6 i 6 6 i 9 6 V ar(x) E(X ) E(X) 9 6 ( 6 ) We can use the equation: P (X i Y j) P (Y j X i)p (X i) For probability P (X i), we have for each i,..., 6 that P (X i) 6. Conditional probability is given by Thus we get the joint distribution: 0 j < i i P (Y j X i) 6 j i 6 j < i

4 4CHAPTER 3. EXPECTATION, INEQUALITIES AND LAWS OF LARGE NUMBERS Y \X E(XY ) i0 j0 ijp r(x i, Y j) jp r(x, Y j) + + j0 6jP r(x 6, Y j) j Cov(X, Y ) E(XY ) E(X)E(Y ) 66 6 Exercise Suppose a box contains 3 balls labeled,,3. Two balls are selected without replacement from the box. Let X be the number on the first ball and let Y be the number on the second ball. Compute Cov(X, Y ) and ϱ(x, Y ). Solution of Exercise 3.6 : We have Cov(X, Y ) E ((X E (X) (Y E (Y )) i,j p xi,y j ( xi E (X) )( y j E (Y ) ). Note that X and Y have identical marginal distributions. Since E (X) E (Y ), and p xi,y j for all i,j, we obtain 6 ( p xi,y j xi E (X) )( y j E (Y ) ) 6 i,j 6( ( )( ) + ( )(3 ) + ( )( ) + ( )(3 )+ + (3 )( ) + (3 )( ) ) ( ) + ( ) 6 3. Also, we have V ar(y ) V ar(x) E(X ) (E(X)) 3 (+4+9) ()

5 3.. EXPECTATION AND VARIANCE 5 3 and thus ϱ(x, Y ) Exercise 3.7 Suppose X and Y are two independent random variables such that E ( X 4), E ( Y ), E ( X ) and E (Y ) 0. Compute Var(X Y ). Solution of Exercise 3.7 : (X Var(X Y ) E( Y E(X Y ) ) ) ( (X E Y E(X )E(Y ) ) ) E ( (X Y 0) ) E(X 4 Y ) E(X 4 )E(Y ) Exercise 3.8 A p-random graph on v vertices is an unoriented graph where between every distinct vertices i < j there is an edge with probability p. Compute the expected value and the variance of the number of all edges in the graph. Solution of Exercise 3.8 : Denote X the number of all edges and X i,j the indicator of the event that there is an edge between i and j. Observe that X i<j X i,j. Then E (X) i<j E (X i,j ) ( ) v p. Since all X i,j are mutually independent, we know, that Var(X) ( ) v Var(X i,j ) (p p ). i<j In fact, we have just calculated the expectation and variance of the binomial distribution, if you substitute n ( v ). Exercise 3.9 Let X have the binomial distribution with parameters n and p. Find E (X). Solution of Exercise 3.9 : Here we provide a direct calculation of the expectation, in contrast to the Exercise 3.8, where we used the linearity of expectation. We have p k (X k) p X (k) ( n k) p k ( p) n k and E (X) i x if(x i ). The the binomial distribution assigns positive probabilities to 0,,... n Thus, E (X) ( ) n i p k ( p) n k. k i0

6 6CHAPTER 3. EXPECTATION, INEQUALITIES AND LAWS OF LARGE NUMBERS To calculate this quantity we observe that ( ) n in! i i i!(n i)! n(n )! (i )! ( (n ) (i ) )! ( ) n n i Thus n ( ) n E (X) n p i ( p) n i. i Making the substitution i l + we see that By the binomial theorem l0 n ( ) n E (X) np p l ( p) n l l l0 n ( ) n p i ( p) n l ( p + ( p) ) n l so we see that E (X) np. Exercise 3.0 Find V ar(x) for X from the previous example. Solution of Exercise 3.0 : Here we provide a direct calculation of the expectation, in contrast to the Exercise 3.8, where we used the linearity of expectation. We have V ar(x) E ( X ) ( E (X) ) We use again the identity k ( ) ( n k n n k ) and get E ( X ) s0 ( ) n k p k ( p) n k n k k0 ( ) n k p k ( p) n k. k k0 We put m n and s k and obtain E ( X ) m ( ) ( m m ( ) m m ( ) m np (s+) p s ( p) m s np s p s ( p) m s + )p s ( p) m s s s s s0 s0

7 3.. EXPECTATION AND VARIANCE 7 The first sum is equal to mp (see Exercise 3.9), the second is equal to from the binomial theorem. We obtain E ( X ) np(mp + ) np ( (n )p + ) np(np p + ) Var(X) E ( X ) (E (X)) np(np p + ) (np) np( p) Exercise 3. Consider a group of n people. A special day is a day such that exactly k people in the group have a birthday. What is the expected number of special days in a year? (Assume all years are non-leap.) Solution of Exercise 3. : We will define a family of random variables X, X,..., X 5 as follows. {, if exactly k people have their birthday on the i-th day of a year, X i 0, otherwise In other words, X i if and only if i is a special day. probability of X i. Let s compute the P (X i ) ( ) ( ) k ( ) n k n 4 k 5 5 Note, that although all 5 random variables have the same probability distribution, they still denote different random variables. In addition, these random variables are not independent! Let us now define another variable X 5 X i. It can be seen that X m if there are m special days in a year. Since we are interested in the expected number of special days, we will calculate E (X). E (X) E 5 ( 5 ) 5 5 X i E (X i ) P (X i ) ( n k ) ( 5 ) k ( 4 5 ) n k Exercise 3. Consider the same group of n people. What is the expected number of days such that at least two people have a birthday? How large should be n to make this expectation exceed? Solution of Exercise 3. : Note that this exercise is almost the same as the previous one, only the definition of a special day has changed. The solution can

8 8CHAPTER 3. EXPECTATION, INEQUALITIES AND LAWS OF LARGE NUMBERS therefore be found in the same manner as above. We will once again define a family of random variables X, X,..., X 5 as follows. {, if two people have their birthday on the i-th day of a year, X i 0, otherwise Now to calculate the probability distribution of X i and the expectation of 5 X i. ( ) n ( ) n 4 4 P (X i 0) + n E P (X i ) P (X i 0) ( 5 ) 5 5 X i E (X i ) P (X i ) ( ( ) n 4 5 n 5 ( ( ) n 4 5 n ( ) ) n 4 5 ( ) ) n 4 5 Exercise 3.3 Let X have a geometric distribution with parameter p. Find E (X). Solution of Exercise 3.3 : The expectation of the geometric distribution is E (X) p jp( p) j j0 p j( p) j j0 j0 d ( p)j dp Since a power series can be differentiated term by term, it follows that E (X) p d dp ( p) j Using the formula for the sum of a geometric progression, we can see that Exercise 3.4 j0 E (X) p d dp p p p p

9 3.. MARKOV AND CHEBYSHEV INEQUALITY; CHERNOFF BOUNDS9 Let the random variable X be representable as a sum of random variables X n X i. Show that, if E[X i X j ] E[X i ]E[X j ] for every pair of i and j with i < j n, then var[x] n var[x i] Solution of Exercise 3.4 : First we have var(x) E(X ) (E(X)) E(( X i ) ) (E( X i )) E( j X i X j ) (E( X i )) By the linearity of expectation we have var( X i ) Using the assumption we get var( X i ) E(X i X j ) ( E(X i )) (3.) j j E(X i X j ) j E(Xi ) (E(X i )) E(X j )E(X i ) (3.) var(x i ) (3.3) 3. Markov and Chebyshev inequality; Chernoff bounds Exercise 3.5 Suppose we flip a fair coin n times to obtain n random bits. Consider all m ( ) n pairs of these bits in some order. Let Y i be the exclusive or of the ith pair of bits, and let Y m Y i be the number of Y i that equal.. Show that each Y i is 0 with probability / and with probability /.. Show that Y i are not mutually independent. 3. Show that Y i satisfy the property E[Y i Y j ] E[Y i ]E[Y j ]. 4. Using previous exercise find V ar[y ] 5. Using Chebyshev s inequality prove a bound on P r[ Y E(Y ) n]

10 0CHAPTER 3. EXPECTATION, INEQUALITIES AND LAWS OF LARGE NUMBERS Solution of Exercise 3.5 :. Straightforward.. Consider n 3 and probability P r(y, Y, Y 3 ). If Y i were mutually independent, P r(y, Y, Y 3 ) would be equal to 8. In fact it is equal to zero (consider all 3-bit strings to see this). 3. First we will show, that Y i are pairwise independent. Consider possibilities..) Y i and Y j do not share a bit position. They are obviously independent..) Y i and Y j share a bit position. Consider all 3-bit strings to confirm the independence. 4. V ar[y ] pairwise independence m m V ar(y i ) E(Y i ) E(Y i ) m (0 + ) (0 + ) m 4 m 4 5. Chebyshev has the form P r[ Y E(Y ) n] V ar(y ) n n(n ) 8 n 8. Exercise 3.6 Suppose that Y has the geometric distribution with parameter p 3/4. Compute the exact value and the Chebyshev bound for the probability that Y is at least standard deviations away from the mean. Note: If Y has geometric distribution, E(Y ) p p and var(y ) p. Solution of Exercise 3.6 : E(Y ) 4/3 and var(y ) 4/9 and σ(y ) 3 We need to compute P r[y ] + P r[y ] 0 + P r[y 3] F Y () ( (/4) ) /6. By Chebyshev inequality P r[ Y E(Y ) 4 3 ] 4/9 6/9 4

11 3.. MARKOV AND CHEBYSHEV INEQUALITY; CHERNOFF BOUNDS Exercise 3.7 Alice and Bob play checkers often. Alice is a better player, so the probability that she wins any given game is.6, independent of all other games. They decide to play a tournament of n games. Bound the probability that Alice loses the tournament using a Chernoff bound. Solution of Exercise 3.7 : Alice looses the game is she wins less than half of the games, i.e. we want the probability P (X n ). We will use the Chernoff bound P r(x ( δ)µ) e µδ /. To bound our probability from above, we have to solve (for µ 3n/5) the inequality ( δ)µ n ( δ)(3n/5) n δ 5 n 6 n what gives e.g. δ /3 for n 5. In fact, we can use δ ɛ for any ɛ > /6 and sufficiently large n. We get P r(x (n )/) P r(x ( δ)µ) e µδ / e (3n/5)(/9)/ e n/30. Exercise 3.8 We have a standard six-sided die. Let X be the number of times that a 6 occurs over n throws of the die. Let p be the probability of the event X n/4. Compare the best upper bounds on p that you can obtain using Markov s inequality, Chebyshev s inequality and the Chernoff bounds. Solution of Exercise 3.8 : Chernoff states that p P r(x n/4) P r(x ( + δ)µ) e µδ /3. Observing that E(X) µ n 6, we solve ( + δ)µ n/4 to get δ /. Consequently, we can bound p as p e (n/6)(/4)/3 e n/7. Markov states that ( ) E(X) P (X t) t where E(X) E ( X i ) n/6. We can deduce that p P (X n/4) ( ) n/6 n/4 /3.

12 CHAPTER 3. EXPECTATION, INEQUALITIES AND LAWS OF LARGE NUMBERS Chebyshev states that P (X t) P ( X E(X) t) var(x) t. Noting that var(x) nq( q), we have P (X n/4) P ( X E(X) n/4) n (n/4) 0 9n. Exercise 3.9 We plan to conduct an opinion poll to find out the percentage of people in a community who want its president impeached. Assume that every person answers either yes or no. If the actual fraction of people who want the president impeached is p, we want to find and estimate X of p such that P r( X p εp) > δ for a given ε and δ, with ε > 0 and δ <. Solution of Exercise 3.9 : Find X such that P r( X p εp) > δ. This is equivalent to P r( nx np εpn) > δ. This holds if (but is not necessarily equivalent) P r( nx np < εpn) > δ. Finally, we can replace this by P r( nx np εpn) δ. It remains to solve apply a suitable Chernoff bound P r( Y µ ɛµ) e µɛ /3. For µ np we solve e µɛ /3 δ to get 3 ln δ pɛ n. Exercise 3.0 Consider a collection X,..., X n of n independent integers chosen uniformly from the set {0,, }. Let X n X i and 0 < δ <. Derive a Chernoff bound for P r(x ( + δ)n) and P r(x ( + δ)n). Solution of Exercise 3.0 : M Xi E(e txi ) 3 (et + e t + ) e et +e t.

13 3.. MARKOV AND CHEBYSHEV INEQUALITY; CHERNOFF BOUNDS3 Then M X (t) Setting t ln( + δ) we get n M Xi (t) e n(et +e t). P r(x ( + δ)n) <...