1 INFO Sep 05

Transcription

1 Events A 1,...A n are said to be mutually independent if for all subsets S {1,..., n}, p( i S A i ) = p(a i ). (For example, flip a coin N times, then the events {A i = i th flip is heads} are mutually independent.) Example: suppose events A, B, and C are pairwise independent, i.e., A and B are independent, B and C are independent, and A and C are independent. Note that this pairwise independence does not necessarily imply mutual independence of A, B, and C. To check that p( i S A i ) = i p(a i) for all subsets S {A, B, C} in this case means checking the non-trivial subsets with 2 or more elements: {A, B}, {A, C}, {B, C}, {A, B,C}. By assumption it follows for the first three, so the only one we need to check is p(a, B, C)? = p(a)p(b)p(c). But that this is not always the case can be seen by an explicit counterexample: consider tossing a fair coin three times, and consider the three events: A = the number of heads is even, B = the first two flips are the same, C = the second two flips are heads. It follows that p(a) = p(b) = 1/2, p(c) = 1/4, p(a, B) = 1/4 = p(a)p(b), p(a, C) = p(b, C) = 1/8 = p(a)p(c) = p(b)p(c); but p(a, B, C) = 0 p(a)p(b)p(c) = 1/16.) The complement of a set A S in S is denoted A = S A, i.e. the set of elements in S not contained in A. We can prove that an event A is independent of another event B if and only if A is independent of B. To show this, first recall that if S can be written as the union of a set of non-interecting subsets S i : S = i S i, S i S j = φ, then p(a) = i p(a S i) = i p(a, S i). The two sets S 1 = B, S 2 = B clearly satisfy these conditions, so we can write p(a) = p(a, B) + p(a, B). Note also that p(b)+p(b) = 1. If A and B are independent, then by definition p(a, B) = p(a)p(b) and substituting in the above results in p(a, B) = p(a)(1 p(b)) = p(a)p(b), so A and B are independent. In the opposite direction, if p(a, B) = p(a)p(b) then substitution in the above gives p(a, B) = p(a)(1 p(b)) = p(a)p(b), and A and B are independent. Finally, note that the notions of disjoint and independent events are very different. Two events A, B are disjoint if their intersection is empty, whereas they are independent if P(A, B) = p(a)p(b). Two events that are disjoint necessarily have p(a, B) = p(a B) = 0, so if their independent probabilites are non-zero they are necessarily negatively correlated (P(A, B) < p(a)p(b)). For example, if we flip 2 coins, and event A = exactly 1 H, and event B = exactly 2 H, these are disjoint but not independent events: they re negatively correlated since p(a, B) = 0 is less than p(a)p(b) = (1/2)(1/4). Non-disjoint events can be positively or negatively correlated, or they can be independent. If we take event C = exactly 1T, then A and C are not disjoint (they re equal): and they re positively correlated since p(a, C) = 1/2 is greater than p(a)p(c) = 1/4. If we now flip 3 coins and let C = at least 1 H and at least one T, and D = at most 1 H. We see that C D=1H, and independence of events C, D follows from p(c)p(d) = (6/8)(1/2) = 3/8 = p(c, D). 1 INFO Sep 05

2 Random variables, mean and variance: Suppose in a collection of number of people there are some number with height 6, and equal numbers with heights 5 11 and 6 1. The mean or average of this distribution is 6, as can be determined by summing the heights of all the people and dividing by the number of people, or equivalently by summing over distinct heights weighted by the fractional number of people with that height. Suppose for example, that the numbers in the above height categories are 5,30,5, then the latter calculation corresponds to (1/8) (3/4) 6 + (1/8) 6 1 = 6. But the average gives only limited information about a distribution. Suppose there were instead only people with heights 5 and 7, and an equal number of each, then the average would still be 6 though these are very different distributions. It is useful to characterize at least the variation within the distribution from the mean. The average deviation from the mean gives zero due to equal positive and negative variations (as proven below), so the quantity known as the variance (or mean square deviation) is defined as the average of the squares of the differences between the values in the distribution and their mean. For the first distribution above, this gives the variance V = 1 8 ( 1 ) (0 ) (1 )2 = 1 4 (inch)2, and for the second distribution the much larger result V = 1 2 ( 1 ) (1 )2 = 1(foot) 2. The standard or r.m.s ( root mean square ) deviation σ is defined as the square root of the variance, σ = V. The above two distributions have σ = (1/2 inch) and σ = (1 foot) respectively. More generally, a random variable is a function X : S IR, assigning some real number to each element of the probability space S. The average of this variable is determined by summing the values it can take weighted by the corresponding probability, <X> = s S p(s)x(s). (An alternate notation for this is E[X] = <X>, for the expectation value of X.) Example 1: roll two dice and let X be the sum of two numbers rolled. X({1, 1}) = 2, X({1, 2}) = X({2, 1}) = 3,..., X({6, 6}) = 12. The average of X is Thus <X> = = 7. Example 2: flip a coin 3 times, and let X be the number of tails. The average is <X> = = 3 2. The expectation of the sum of two random variables X, Y satisfies <X + Y > = <X>+<Y >. In general, they satisfy a linearity of expectation <ax +by > = a<x>+ b<y > proven as follows: <ax + by > = s p(s)(ax(s) + by (s)) = a s p(s)x(s) + b s p(s)y (s) = a<x> + b<y >. Thus an alternate way to calculate the mean of X = 2 INFO Sep 05

3 X 1 + X 2 for the two dice rolls in example 1 above is to calculate the mean for a single die, X 1 = ( )/6 = 21/6 = 7/2, and so for two rolls <X> = <X 1 >+<X 2 > = 7/2 + 7/2 = 7. By definition, independent random variables X, Y satisfy p(x=a Y =b) = p(x = a)p(y = b) (i.e., the joint probability is the product of their independent probabilities, just as for independent events). For such variables, it follows that the expectation value of their product satisfies <XY > = <X><Y > (X, Y independent) since r,s p(r, s)x(r)y (s) = r,s p(r)p(s)x(r)y (s) = ( r p(r)x(r))( s p(s)y (s)). As indicated above, the average of the differences of a random variable from the mean vanishes: s S p(s)( X(s) <X> ) = <X> <X> s p(s) = <X> <X> = 0. The variance of a probability distribution for a random variable is defined as the average of the squared differences from the mean, V [X] = s S p(s) ( X(s) <X> ) 2. (V 1) The variance satisfies the important relation V [X] = <X 2 > <X> 2, (V 2) following directly from the definition above: V [X] = p(s) ( X(s) <X> ) 2 s S = s X 2 (s)p(s) 2<X> s p(s)x(s) + <X> 2 s p(s) = <X 2 > 2<X> 2 + <X> 2 = <X 2 > <X> 2. In the case of independent random variables X, Y, as defined above, the variance is additive: V [X + Y ] = V [X] + V [Y ]. To see this, use (V 2) together with <XY > = <X><Y >: V [X + Y ] = <(X + Y ) 2 > (<X> + <Y >) 2 = <X 2 > + 2<XY > + <Y > 2 <X> 2 2<X><Y > <Y > 2 = <X 2 > <X> 2 + <Y 2 > <Y > 2 = V [X] + V [Y ]. Example: again flip a coin 3 times, and let X be the number of tails. <X 2 > = = 3 so V [X] = 3 (3/2)2 = 3/4. If we let X = X 1 + X 2 + X 3, where X i 3 INFO Sep 05

4 is the number of tails (0 or 1) for the i th roll, then the X i are independent variables with <X i > = 1/2 and <Xi 2> = (1/2) 1 + (1/2) 0 = 1/2, so V [X i] = 1/2 1/4 = 1/4 (or equivalently V [X i ] = 1/2(1/2) 2 + 1/2( 1/2) 2 = 1/8 + 1/8 = 1/4). For the three rolls, V [X] = V [X 1 ] + V [X 2 ] + V [X 3 ] = 1/4 + 1/4 + 1/4 = 3/4, confirming the result above. A Bernoulli trial is a trial with two possible outcomes: success with probability p, and failure with probability 1 p. The probability of r successes in N trials is ( N r ) p r (1 p) N r. Note the correct overall normalization automatically follows from N ( N ) r=0 r p r (1 p) N r = [ ] N ( p + (1 p) = 1 N = 1. The overall probability for r successes is a competition between N ) r, which is maximum at r N/2, and p r (1 p) N r with is largest for small r when p < 1/2 (or large r for p > 1/2). In class, we considered the case of rolling a standard six-sided die, with a roll of 6 considered a success, so p = 1/6. For a larger number N of trials, the distribution of expected number of successes became more narrowly peaked and more symmetrical about a fractional distance r = N/6. To analyze this in the framework outlined above, let the random variable X i = 1 if the i th trial is success. Then <X i > = p. Let X = X 1 + X X N count the total number of successes. Then it follows that the average satisfies <X> = i <X i > = Np. (B1) From V [X i ] = <X 2 i > <X i> 2 = p p 2 = p(1 p), it follows that the variance satisfies V [X] = i V [X i ] = Np(1 p), (B2) and the standard deviation is σ = V [X] = Np(1 p). This explains the observation that the probability gets more sharply peaked as the number of trials increases, since the width of the distribution (σ) divided by the average <X> behaves as σ/<x> N/N 1/ N, a decreasing function of N. By the central limit theorem (not proven in class), many such distributions under fairly relaxed assumptions always tend for sufficiently large number of trials to a gaussian or normal distribution, of the form P(x) 1 (x µ) 2 σ 2π e 2σ 2. (G) This is properly normalized, with dx P(x) = 1, and also has dx xp(x) = µ, dx x2 P(x) = σ 2 + µ 2, so the above distribution has mean µ and variance σ 2. Setting µ = Np and σ = Np(1 p) for p = 1/6 in (G) thus gives a good approximation to the distribution of successful rolls of 6 for large number of trials in the example above. 4 INFO Sep 05

5 Telegraphic review of exponentials and logarithms: x m represents the result of multiplying m factors of x, and satisfies the useful relations x m x n = x m+n, (x n ) m = x nm, x 0 = 1. (E1) The exponential function f(x) = y x is frequently employed with y = e, where e = is a transcendental number, which can be introduced as follows. Consider investing $1 at a 100%/year interest rate for a year. If interest is compounded a single time at the end of the year, then the result is $1 (1 +1) = $2. If instead interest is compounded twice, at 50% after a half year and the remaining 50% at the end of the year, then the result is the slightly larger, $1 (1+1/2)(1+1/2) = $2.25. Now consider breaking up the time interval into N pieces, so that interest is compounded N times, then the total is (1 + 1/N) N. In the limit that N becomes arbitrarily large (compounded continuously), the result converges to lim (1 + N 1/N)N = e. (D1) If the interest rate is instead only 5% per year compounded continuously, then the result would be lim N (1+.05/N) N = lim M (1+1/M).05M = e For general interest rate x, it follows that lim N (1 + x/n) N = lim M (1 + 1/M) Mx = e x. This function can also be written as an infinite sum by expanding the product, and keeping only the terms that survive in the large N limit: ( e x = lim 1 + x ) N ( = lim 1 + N x N N N N = 1 + x + x2 2! + x3 3! + + xm m! + = N(N 1) ( x ) ( 2 N ! N m n=0 x n n! ) ( x N ) m + ). (E2) The logarithm function is the inverse of the exponential function: if y x = X then x = log y X. For example, 10 3 = 1000 so log = 3, and 2 10 = 1024 so log = 10. Useful properties of the logarithm function follow directly from the corresponding properties (E1) of the exponential function log XZ = log X + log Z, log X s = s log X, log 1 = 0. (L1) (For example if X = y x, Z = y z, then log y XZ = log y y x y z = x+z = log y X +log y Z, and log X s = log y y xs = xs = s log y X. Logarithms to different bases y, z are simply related by a numerical factor log z y: log y x = log z x log z y Note that the rough relation implies that 10 3/10 2 and 2 10/3 10, which permits estimating log /10 and log /3, good approximations to the actual values of and , respectively. 5 INFO Sep 05

6 (following from log z x = log z y log y x = log y x log z y). The special notation ln = log e is used for logarithms to the base e. For example, ln 2 = log e The series expansion for the function ln(1+t) = a 1 t+a 2 t 2 +a 3 t 3 + can be determined by setting e x = 1 + t (defined so that t is small when x is small) and substituting in (E2): 1+t = 1+(a 1 t+a 2 t 2 +a 3 t 3 + )+ 1 2 (a 1t+a 2 t 2 +a 3 t 3 + ) ! (a 1t+a 2 t 2 +a 3 t 3 + ) 3 +. Comparing the coefficients of powers of t gives a 1 = 1, a 2 +a 2 1/2 = 0, a 3 +2a 1 a 2 /2+a 3 1/6 = 0,..., and we infer that a 2 = 1/2, a 3 = 1/2 1/6 = 1/3, i.e., ln(1+t) = t t 2 /2+t 3 /3. Letting t t, the full result can be written in the form ln(1 t) = t t2 2 t3 3 = t n n. n=1 (L2) Example: Suppose a software program has a bug that occurs with probability p = 1/1000. a) How many times do we need to run the program in order to have a 50% chance of seeing the bug occur? b) What is the probability of seeing the bug if we run the program 1000 times? a) The probability of not seeing the problem occur in a single run is P 1 = 1 p so, assuming that successive runs are independent, the probability of seeing no problem after N runs is P N = (1 p) N. For (1 p) N = 1/2, we find N ln(1 p) = ln(1/2), or using (L2) to lowest order, Np ln 2, so N (1/p) ln2 = 1000 ln2 693, so after N = 693 trials the probability of no problem falls to roughly 50%. b) P 1000 = ( ) is large so by (D1) this is approximated by P 1000 e (Equivalently, using (L2) we can estimate lnp 1000 = 1000 ln( ) 1000( ) = 1, so again P 1000 e 1 ). The probability of seeing the bug is therefore 1 P , i.e., has climbed to roughly 63.2% after 1000 trials. This problem is directly analogous to the problem of nuclear decay, where p is instead the probability per unit time of decay of some nuclear species. (In the original problem, p was a probability per trial, so a trial becomes an infinitesimal time step in the nuclear problem.) The half-life, i.e., the time for half of some sample to be likely to decay, is given by the equivalent formula τ = (1/p) ln2. 6 INFO Sep 05

7 Figure: probability of seeing a problem, given by 1 (999/1000) N 1 e N/1000. The points plotted are at the values (693,.5), (1000,.632), and (5000,.993), corresponding to 50% at N = 693, 63.2% at N = 1000, and 99.3% at N = INFO Sep 05