Introduction to discrete probability. The rules Sample space (finite except for one example)

Transcription

1 Algorithms lecture notes 1 Introduction to discrete probability The rules Sample space (finite except for one example) say Ω. P (Ω) = 1, P ( ) = 0. If the items in the sample space are {x 1,..., x n } each x i has some probability p i and n i=1 p i = 1. If s S has probability P (s) it means that among all possible outcomes, we get s with probability P (s). What does it Really mean? Make the experiment a million times independently and count the number of times s appears. Without a doubt, it will be very close to P (s) 10 6.

2 Algorithms lecture notes 2 Events a subset of Ω. Events An event A Ω happens if one of the points of A is selected. Axiom: A B = implies P (A B) = P (A) + P (B). Note that A = {x 1, x 2,..., x q } = {x 1 } {x 2 }... {x q } implies P (A) = q i=1 P (x i). The probability of a subset is the sum of probabilities of its points.

3 Algorithms lecture notes 3 We throw a fair coin twice By symmetry each one of {HH, T T, HT, T H} has probability 1/4. P ({HT, HH}) = 1/2. Note that if A B, then P (A B) = P (A) + P (B) P (A B). Example for the above sample space. A is the event at least one H B the event T H. P (A) = 3/4, P (B) = 1/4. P (A B) = P (T H) = 1/4. Thus P (A B) = 3/4. The points are {T H, HHHT }.

4 Algorithms lecture notes 4 We throw a fair die. Example P ({1, 6}) = 1/6 + 1/6 = 1/3. P (result at most 4) = 4/6 = 2/3. You throw 2 dice. What is the probability of sum 5. S = 1/36. Points {1, 4}, {4, 1}, {2, 3}, {3, 2}. Probability 4/36 = 1/9. Sum 7 probability: {1, 6}, {6, 1}, {2, 5}, {5, 2}, {3, 4}, {4, 3}. 6/36 = 1/6. Finally sum 8: {2, 6}, {6, 2}, {3, 5}, {5, 3}, {4, 4}. 5/36.

5 Algorithms lecture notes 5 Balls and bins. Important for later (Hash function) Throw n balls to n bins independently and uniformly at random. This just means that a ball has probability 1/n to reach a given entry. Also what happened to one ball does not affects what happens to others. Ω = n n. What is the probability that each bin has one ball? n!/n n. This is roughly 1/e n What is the probability that ball 7 and ball 3 fall in the same bin. Smart answer. Throw 7 first. Never mind where 7 falls the second ball has to fall in the bin of 7 and this is 1/n. By the book: Thinks of 3, 7 as one ball. n n 1 possibilities for them in the same bin. n n 1 /n n = 1/n.

6 Algorithms lecture notes 6 More balls versus bins What is the probability that bin 1 is empty? A ball has probability (n 1)/n not to fall in bin 1. Thus (1 1/n) n which is very close to 1/e. What is the probability all balls are in the same bin? Throw the first. Never mind the result its now 1/n n 1. Question: If we know that 3, 7 are in the same bin, does it change the probability that 3, 5 are in the same bin? No. 5 still has to fall in the bin of 3. But if we know that 7, 5 are in the same bin and 5, 9 are in the same bin then P (7, 9 are in the same bin ) = 1.

7 Algorithms lecture notes 7 Independence Given a collection X 1,..., X n of events, for every subset of indices (this are 2 n constrains) P r(x i1 X i2 X ik ) = P r(x i1 ) P r(x i2 ) P r(x ik ). The events i and j reached the same bin are pairwise independent but not independent.

8 Algorithms lecture notes 8 Random variables A random variable is a function from Ω to R. Example: we throw a dice 10 times. How many had result 4? Such X is called a random variable. Ball and bins: You throw n balls to n bind independently at random. Let X i be 1 if ball i gets to bin i and zero otherwise. Let Z be a random variable which is the number of empty bins.

9 Algorithms lecture notes 9 Indicator variables Let Z be the random variable that counts the number of empty bins. Let X i = 1 if bin i is empty. Say that we have 6 bins. Bin A[1] has three balls. Bin 3 has 1 ball. Bin 5 has 2 balls. Thus X 1 = 0, X 2 = 1, X 3 = 0, X 4 = 1, X 5 = 1, X 6 = 0. Note that the empty bins are bin A[2], A[4] and A[6]. The bins for which the X variable is 1. Since we get 1 fair every empty bin it always holds that: Z = X 1 + X X X 4 + X 5 + X 6, since non empty bins give 0 and empty bins give 1.

10 Algorithms lecture notes 10 Expectation of the sum of random variable Definition: E(X) = i i P (X = i). Example. Say that we throw a dice. Let X the the random variable of the result on the dice. E(X) = 1/ / / / / /6 6 = ( )/6 = 3.5 If we throw two dice? The expectation can be seen to be 7. You can compute going over all 36 results. Each has 1/36 probability. No coincidence.

11 Algorithms lecture notes 11 Expectation Sum of random variables E( n i=1 X i) = n i=1 E(X i) Example: A secretary has letters 1, 2,..., n and envelopes 1, 2,... n. But he/she puts every letter in a random envelope. Namely Ω has size n! with all points having probability 1/n!. What is the expected number of letter that arrive to their place? Set X i = 1 if i got to i and 0 otherwise.

12 Algorithms lecture notes 12 Continued Let Z be the random variable that counts the number of letters that got to their proper place. For example, say that 3, 4, 6 reach their proper place. So X 1 = 0, X 2 = 0, X 3 = 1, X 4 = 1, X 5 = 0, X 6 = 1. Each letter that got to its place gives +1 and each letter not in its place gives 0 and so: Z = n i=1 X i. Note that by definition E(X i ) = 1 1/n + 0 (1 1/n) = 1/n. E(X) = E( n i=1 X i) = n i=1 E(X i) = 1. He/She will be fired.

13 Algorithms lecture notes 13 The expected number of empty bins Let X i = 1 if bin i is empty and 0 otherwise. Let n = 6. Say Bin 1 has 2 balls. Bin 2, one ball, bin 3 is empty, bin 4 has 3 balls, and bins 5, 6 are empty. X 1 = 0, X 2 = 0, X 3 = 1, X 4 = 0, X 5 = 1, X 6 = 1. Clearly X = n i=1 X i.

14 Algorithms lecture notes 14 continued P (X i = 0) = (1 1/n) n. E(X) = n i=1 E(X i) = n i=1 (1 1/n)n = n (1 1/n) n. (1 1/n) n is very close to 1/e for large n. So E(X) is about n/3. More than a third empty! Similarly the expected value of a die is 3.5 Thus if you throw two dice, the expected sum is 7 Let X, Y be the random variables, so that X is the value of the first die and Y of the second. E(X + Y ) = E(X) + E(Y ) = = 7.

15 Algorithms lecture notes 15 The expected number of times the maximum is changed Say that we apply the algorithm to find the maximum. Say that the number of elements is a million. Please guess how many times will the minimum be replaced. This means that A[i] > M

16 Algorithms lecture notes 16 Analysis Let in general the number of elements be n. Let Z be the random variable that counts the number of times the maximum has changed. Let X i be 1 if M has been change in iteration i, and 0 otherwise. Thus Z = n i=1 X i. The probability that A[i] > M is exactly the probability that among the first i elements, the maximum is number i. By symmetry, this probability is 1/i. E(Z) = n i=1 1/i = ln n + O(1). For a million, less than 13.9 The time for random input is almost negligible compared to the time for a non random input, in the worst case.

17 Algorithms lecture notes 17 Variance How far are we expected to be from the average. Example: let n = Let X i = 1 with probability 1/2 and 1 with probability 1/2. Is n i=1 X i close to its expectation 0? Yes. Never much larger than say n. Could be 3 standard deviations: 3000 << If X i = 10 6 with probability 1/2 and 10 6 with probability 1/2 the expectation is 0. But the random variable is never even close to 0. So can we do µ = E(x) E(X µ)? No. This is always 0. Absolute value? Hard to work with. Therefore V ar(x) = E(x µ) 2 = E(X 2 ) 2µ E(x) + E(x) 2 = E(X 2 ) E(X) 2, Standard deviation measures how far from expectation σ(x) = V ar(x).

18 Algorithms lecture notes 18 The variance of a sum is not always the sum of variances You need that every two random variables are independent. Always: P (X 1 = 3 and X 2 = 9) = P (X 1 = 3) P r(x 2 = 9). X = n i=1 X i so that X i = 1 w.p 1/2 and 1 w.p 1/2 X 2 = 1. E(X) 2 = 0. Variance V ar( n i=1 X i) = n i=1 V ar(x i) = n. σ(x) = n. Very small. Say X = 10 6 w.p. 1/2 and 10 6 w.p 1/2. X 2 = (E(X)) 2 = 0. Variance is σ(x) = Huge variance Equals the maximum number.

19 Algorithms lecture notes 19 Geometrical variable Let a test have probability p for success and q = 1 p for failure. Let X be a random variable that counts the number of times until the first success. This is called Geometrical distribution. Thus P r(x = 1) = p. P r(x = 2) = p q P r(x = i) = p q i 1. If p = 1/4 is it intuitive that the expectation is 4? In any case the expectation of a geometrical distribution is p (the sum that shows that is rather complex). Will be used in Skip Lists.