SUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416)

SUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416) D. ARAPURA This is a summary of the essential material covered so far. The final will be cumulative. I ve also included some review problems at about the level I would expect you to do. 1. Axioms for probability A sample space consists of set S consisting of outcomes that we are interested in. A subset E S is called an event. A probability measure on a sample space is an assignment of a number P (E) [0, 1], called the probability of E, subject to the following conditions: (1) P (S) = 1 (2) If E 1, E 2,... are mutually exclusive events, i.e. E i E j = when i j, then P ( E i ) = i P (E i). Note that E 1 E 2 = E 1 E 2. One can deduce that P (E c ) = 1 P (E). In particular, P ( ) = 1 1 = 0. Also, we have P (E F ) = P (E) + P (F ) P (EF ) The simplest useful example of such a set up is when S has two elements S = {Success, F ailure} with P (Success) = p [0, 1] and P (F ailure) = 1 p. In general, when S = {s 1,..., s n } is finite, the probably measure is an assignment of probabilities P (s i ) such that P (s i ) = 1. The simplest case is when P (s i ) = 1/n, in which case, we say that the outcomes are equally likely. Then P (E) = E S So calculation of probability in this case becomes a counting problem. Exercise 1. Two cards are dealt from a standard deck of 52. What s the probability of getting a pair (which means that both cards have the same value)? The conditional probability of E given F is P (E F ) = P (EF ) P (F ) provided P (F ) 0. The idea is that we know F occurs, so we replace the original sample space by F and adjust everything else. Exercise 2. Two cards are dealt as above. What s the probability of getting a pair given that both cards are black (clubs or spades)? It comes up in Bayes formula P (E) = P (E F )P (F ) + P (E F c )P (F c ) The following problem was from the first test. 1

2 D. ARAPURA Exercise 3. A farm produces two varieties of apples, 80% red and the rest yellow. 20% of red apples are not sweet, and 90% of the yellow are sweet. What s the probability that an apple coming from the farm is sweet? Solution: We can find P (Sweet Red) =.8 and P (Sweet Red c ) =.9, so Bayes formula says P (Sweet) = (.8)(.8) + (.9)(.2) Two events E 1, E 2 are independent if P (E 1 E 2 ) = P (E 1 )P (E 2 ). This is equivalent to the conditional probability P (E 1 E 2 ) = P (E 1E 2 ) P (E 2 ) = P (E 1 ) This means that knowing E 2 has no effect on the value of P (E 1 ). Exercise 4. 3 coins are flipped. Let E be the event that we exactly one tail among the first two flips, and F is the event that we have exactly one tail among last two flips. Show E and F are independent. 2. Random variables A random variable X on a sample space is a numerical function on it. It should be thought of a quantity that we want to measure, e.g., the number of Heads, the sum of values of dice, the grade of a student... Typically we would be interested in the probabilities of the events {s X(s) = a} or {s a X(s) b} which we would simply write as P (X = a) or P (a X b). We distinguish between two types of random variables. X is discrete if it takes a finite or countable set of values x 0, x 1,.... In most cases, the values are simply nonnegative integers, and we will assume this below. The opposite case is where X takes all values in some interval in R; then X is called continuous. In the discrete case, much of the information we care about is given by the probability masses p X (i) = P (X = i) We usually write this as p(i) when X is understood. The axioms of probability imply that P (s) = p(i) = 1 s S Note that in practice, this might be finite sum. The expected value or expectation is the weighted average of the values of X: i=0 (2.1) E(X) = s S X(s)P (s) This is also called the mean. By regrouping terms, the expectation can also be written as E(X) = ip(i) The variance (2.2) V ar(x) = E[(X E(X)) 2 ] = E(X 2 ) E(X) 2 measures the spread, in the sense that it is small if the values of X are concentrated near the mean. Chebyshev s inequality discussed later, will give a more i=0

SUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416) 3 quantitative justification for this statement. Sometimes, we use the standard deviation σ which is V ar(x). Exercise 5. Two dice are rolled. Let X be the sum of the two numbers on top. What s the expectation and the variance? In the continuous case, instead of the mass, we look at the cumulative distribution or the probability density F X (a) = F (a) = P (X a) = P ({s X(s) a}) By the fundamental theorem of calculus Note these functions satisfy f X (a) = f(a) = F (a) F (a) = a f(x)dx F is increasing lim F (a) = f(x)dx = 1 a The expectation is defined by E(X) = and the variance by the formula (2.2) above. Exercise 6. Let f(x) = xf(x)dx { cx 2 if 1 x 1 0 otherwise Assuming this a probability density, find c. What s the expectation and variance? 3. Binomial and other random variables Here are some important examples. (1) Suppose n independent experiments (or coin flips...) are performed, where the probability of success is p and failure is 1 p. Let X be the number of successes. Then ( ) n p(i) = p i (1 p) n i i X is called a binomial random variable with parameters (n, p). The binomial theorem gives (x + y) n = n i=0 Setting x = p, y = 1 p gives p(i) = 1 ( ) n x i y n i i as required. Differentiating (x + y) n with respect to x, multiplying by x, and then setting x = p, y = 1 p gives the expected value E(X) = ip(i) = np

4 D. ARAPURA A similar trick allows us to compute the variance V ar(x) = np(1 p) We will give a slicker derivation later. (2) A Poisson random variable with parameter µ has probability mass µ µi p(i) = e i! This is discrete, but takes infinitely many values. We have p(i) = e µ 0 0 µ i i! = 1 by using the Taylor series for the exponential. We have and E(X) = µ V ar(x) = µ Exercise 7. Check the first formula for E(X) by differentiating e λt = (λt) i /i! and setting t = 1. The Poisson distribution can be considered as a limiting case of the binomial distribution X n with parameters (n, p) in the following sense. Setting µ = np to the mean, p Xn (i) is ( n i ) p i (1 p) n i = (n)(n 1)... (n i + 1) µ i } n {{ i } i! 1 (1 µ/n)n i }{{} e µ µi i! e µ as n but µ = np stays constant. (3) A random variable X is uniformly distributed over an interval [a, b] if f(x) = { 1 b a if a x b 0 otherwise This is the continuous version of outcomes are equally likely. E(X) = xf(x)dx = 1 b a b (b a)2 V ar(x) = 2 Exercise 8. Check the last formula. a xdx = a + b 2 (4) A random variable X is normally distributed with parameters µ and σ 2 if f(x) = 1 e (x µ)2 /2σ 2 σ Some people call this the Gaussian distribution. Although this is more complicated than the others, it is one of the most important. The graph is the familiar bell curve with maximum at µ. The fact that f(x)dx = 1

SUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416) 5 can be reduced to checking 1 e x2 /2 dx = 1 by using the substitution u = (x µ)/σ. However, the last integral requires a tricky calculation (rewrite its square as a double integral, and evaluate in polar coordinates); it cannot be done using just freshman calculus. We also have E(X) = µ V ar(x) = σ 2 so that σ is the standard deviation. The first formula is easy to see. After doing the above substitution x u, we can reduce to checking But this is clear, because and 1 1 0 xe x2 /2 dx = 0 xe x2 /2 dx 1 xe x2 /2 dx 0 cancel. The normal distribution can also be viewed as limiting case of a binomial distribution, but in a somewhat more complicated way than before. The precise statement is the DeMoivre-Laplace theorem: if X n is binomial with parameters (n, p), set then Y n = X n np np(1 p) P (a Y n b) 1 b e x2 /2 dx, a n It says roughly that if you translate the binomial distribution so that the mean become zero, and then rescale it, so the standard deviation is 1, it will converge to the normal distribution with µ = 0, σ = 1 as n. This is generalized by central limit theorem given at the end of these notes. We encountered some other examples, in passing, such as hypergeometric and exponential distributions. Although I won t expect you to know these for the exam, you might want to review them anyway. 4. Joint distributions of 2 random variables Given two random variables X and Y, we might ask whether there is some relationship between them. This leads us to look at joint distributions. In the discrete case, we consider the joint probability mass p XY (i, j) = P (X = i, Y = j) = P ({s X(s) = i, Y (s) = j})

6 D. ARAPURA Usually, we just write p(i, j). This determines the individual masses. For example, p X (i) = p(i, j) j=0 X and Y are called independent random variables if X = i and Y = j are independent events, or equivalently p(i, j) = p X (i)p Y (j) Roughly, this means that X gives no information about Y. By contrast, Y = 2X are clearly dependent. Defining condition probability mass as p X Y (i j) = P (X = i Y = j) = We see that independence means that p X Y (i j) = p X (i) p(i, j) p Y (j) Exercise 9. Roll two dice. Let X be the number on the first die, and let Y be the sum of numbers on both dice. Compute the p(i, j) and p X Y. Are X and Y independent? In the continuous case, we use the joint probability density f(x, y) which satisfies P (a X b, c Y d) = The individual densities are determined by f X (x) = b d a c f(x, y)dy etc. Independence means f(x, y) = f X (x)f Y (y) or equivalently that f X Y (x, y) = f X (x) where the conditional probability density f X Y (x, y) = f(x, y) f Y (y) f(x, y)dydx Exercise 10. Let X, Y have a joint density { ce x 5y if < x <, y > 0 f(x, y) = 0 otherwise for suitable c. Find c. What s f X Y? Are X and Y independent? The conditional expectation is in discrete case, and in the continuous case. E(X Y = j) = i E(X Y = y) = ip X Y (i j) xf X Y (x, y)dx Exercise 11. Find E(X Y = 1), where X and Y are defined in exercise 10.

SUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416) 7 5. Functions of a random variable Given discrete random variable X and a function g from the set of nonnegative to itself, E(g(X)) = g(x(s))p (s) = g(i)p(i) s S i In the continuous case, we have similarly E(g(X)) = g(x)f(x)dx Exercise 12. Let R be a uniformly distributed random variable in [0, 1]. What s the expected value of the area of a circle of radius R? There is a version of this for two variables: E(g(X, Y )) = g(i, j)p(i, j) i j E(g(X, Y )) = g(x, y)f(x, y)dxdy Of special importance is when g(x, Y ) = ax + by, for constants a, b, then these formulas say E(aX + by ) = ae(x) + be(y ) Note that this also follows immediately from (2.1) in the discrete case. For example, we used this implicitly to go between the two formulas for variance. If we write µ = E(X) for the mean, then V ar(x) = E((X µ) 2 ) = E(X 2 2µX + µ 2 ) = E(X 2 ) 2µE(X) + µ 2 = E(X 2 ) E(X) 2 For another example, consider a binomial random variable X which counts number of success with n independent trials with probability p of success. Let { 1 if ith trial is a succes I i = 0 otherwise One can see that X = I 1 +... + I n We have E(I i ) = 0 + (1)P (I i = 1) + 2P (I 1 = 2) +... = P (I i = 1) = p. So we recover the fact that E(X) = p +... p = np Given a continuous random variable X, a decreasing or increasing differentiable function g(x), the density of Y = g(x) is given by { f X (g 1 (y) d dy f Y (y) = g 1 (y) if g 1 (y) is defined 0 otherwise There are couple of additional related things that we covered, and that you should be aware of. However, I won t expect you to know this for the exam, (1) We found the density of X + Y (it s the convolution of densities of X and Y ). (2) If Y 1 = g 1 (X 1, X 2 ) and Y 2 = g 2 (X 1, X 2 ), we have transformation formula for the new joint density f Y1,Y 2 in terms of the old f X1,X 2

8 D. ARAPURA 6. Covariance The covariance of two random variables X and Y is Cov(X, Y ) = E[(X E(X))(Y E(Y )] = E(XY ) E(X)E(Y ) E(X)E(Y ) + E(X)E(Y ) = E(XY ) E(X)E(Y ) To understand what this tells us, observe that given independent random variables X and Y, E(XY ) = f(x, y)dxdy = f X (x)f Y (y)dxdy = E(X)E(Y ) Therefore Cov(X, Y ) = 0 when X and Y are independent; equivalently X and Y are dependent it Cov(X, Y ) 0. If Cov(X, Y ) = 0, then we say that X and Y are uncorrelated, but be aware that this does not imply that X and Y independent. The correlation where σ X, σ Y ρ(x, Y ) = Cov(X, Y ) σ X σ Y are the standard deviations. We have 1 ρ(x, Y ) 1 Covariance can help with computing variance. We have that n V ar(x 1 + X 2 +... X n ) = Cov(X i, X j ) Therefore V ar(x 1 + X 2 +... X n ) = i=1 V ar(x i ) + 2 i<j n V ar(x i ) if X 1,..., X n are pairwise independent. As an example, we can calculate the variance of a binomial random variable X with parameters n and p. As before X = I 1 +... I n, where I i is 1 or 0 depending on whether ith trial is success or failure. i=1 V ar(i i ) = E(I 2 i ) E(I i ) 2 = p p 2 = p(1 p) We can see that I i and I j are independent when i j. Therefore V ar(x) = V ar(i i ) = np(1 p) Exercise 13. A jar contains 3 red and 2 blue marbles. Draw 2 marbles with replacement (which means the first marble is returned to the jar before drawing the second). Let I i = 1 if ith marble is red and 0 otherwise, and let X = I 1 + I 2 = number of reds drawn. Use independence of I 1 and I 2 to calculate V ar(x) with formula above. Now assume no replacement. Are I 1 and I 2 still independent? Calculate V ar(x) in this case.

SUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416) 9 7. Moment generating functions Given a random variable X, the moment generating function When X is continous Differentiating gives and again Setting t = 0, gives M(t) = M X (t) = E(e tx ) M(t) = M (t) = M (t) = e tx f(t)dx xe tx f(t)dx x 2 e tx f(t)dx E(X) = M (0) E(X 2 ) = M (0) The last expression is called the second moment. Both formulas are valid from discrete random variables also. We can use these formulas to (re)calculate the expectation and variation for various distributions. (1) Let X be a binomial with parameters (n, p). Then M(t) = ( ) n e it p i (1 p) n i = (pe t + 1 p) n i Then so So M (t) = n(pe t + 1 p) n 1 pe t M (t) = n(n 1)(pe t + 1 p) n 2 (pe t ) 2 + n(pe t + 1 p) n 1 pe t E(X) = np E(X 2 ) = n(n 1)p 2 + np V ar(x) = E(X 2 ) E(X) = np(1 p) (2) Let X be Poisson with parameter λ. Then M(t) = e it λ λi e i! = exp[λ(et 1)] So i=0 M (t) = λe t exp[λ(e t 1)] M (t) = (λe t ) 2 exp[λ(e t 1)] + λe t exp[λ(e t 1)] Setting t = 0 gives Therefore E(X) = λ E(X 2 ) = λ 2 λ V ar(x) = λ

10 D. ARAPURA (3) Let X be normal with mean µ = 0 and variance σ 2 = 1. Then M(t) = 1 e tx e x/2 dx = 1 By completing the square and simplifying, we get So Therefore M(t) = e t2 /2 M (t) = te t2 /2 M (t) = t 2 e t2 /2 + e t2 /2 E(X) = 0 E(X 2 ) = 1 e tx x/2 dx V ar(x) = 1 Besides the above the examples, moment generating functions are used in the proof of the central limit theorem give later. 8. Law of large numbers This is sometimes call the law of averages in ordinary speech. Given a sequence of independent random variables X 1, X 2,... with the same distribution, and therefore the same mean µ, set X n = X 1 + X 2 +... X n n to the average of the first n of them. This is also called the sample average. The law of large numbers says roughly that X n µ as n. There are couple of ways to make this mathematically precise. THEOREM 8.1 (Weak law of large numbers). For any ɛ > 0, the probability as n. P ( X n µ ɛ) 0 THEOREM 8.2 (Strong law of large numbers). P ( lim n X n µ ) = 1 The weak law says the value of X n is likely to be close to µ as n gets larger. The strong law actual says that the limit lim X n is the constant function µ almost everywhere, i.e. the probability that it fails to equal µ is zero. To understand what the strong law tells us, suppose that the same experiment is repeated n times, and these repetitions are independent. Let E be some event whose probability we are interested in finding. Set { 1 if E occurs in the ith experiment X i = 0 otherwise We used this sort of random variable before, and we have seen that E(X i ) = P (E). The strong laws says that for large n, X n = 1 (# of times E occurs after n trials) n

SUMMARY OF PROBABILITY CONCEPTS SO FAR (SUPPLEMENT FOR MA416) 11 gives a good approximation for P (E). This justifies our intuitive belief that if we flip a coin many times, we should get heads half the time. Although the strong law is the more useful result, the weak law is a lot easier to prove. We do this below. The proof is based on: THEOREM 8.3 (Markov s inequality). If X is a nonnegative random variable, and a > 0, then P (X a) E(X) a THEOREM 8.4 (Chebyshev s inequality). If X has finite mean µ and variance σ 2, then for any k > 0 P ( X µ k) σ2 k 2 To prove the weak law, assuming further that each X i has finite variance σ 2, we note that an E(X n ) = 1 n E(X 1 +... + X n ) = 1 n E(Xi ) = µ V ar(x n ) = 1 n V ar(x 1 +... + X n ) = 1 n 2 V ar(xi ) = σ2 n Applying Chebyshev to X n with k = ɛ, gives (8.1) P ( X n µ ɛ) σ2 nɛ 2 The right hand side goes to 0 as n. The proof gives useful information. Namely, it says how large we need to talk n to get a desired bound. Let s say we have a coin that s biased, which means that the probability of heads p is not 1/2. To estimate p, we can flip it n times, and use p (# heads)/n Suppose we wanted to be sure that our estimate is accurate to one decimal place with 90% confidence. Applying (8.1) with ɛ = 0.05 tells us we need σ 2 1 0.9 = 0.1 nɛ2 Of course σ 2 = p(1 p) is not known, but we can see from calculus this is maximized at p = 1/2, so that σ 2 1/4. So we need n 0.25 ɛ 2 (0.1) 9. Central limit theorem Suppose we a sequence of independent random variables X 1, X 2,... with the same distribution as before. Assume that the mean µ and variance σ 2 are finite. The law of large numbers says that X 1 +... + X n µ n for large n. To get a sense how good this is subtract these to get X 1 +... + X n nµ n

12 D. ARAPURA We can see that the variance of this is σ/n, which goes to 0. Let us rescale this to Y n = X 1 +... + X n nµ σ n so that Y n has variance 1 and mean 0 for all n. THEOREM 9.1 (Central limit theorem). Y n converges to a normal random variable with mean 0 and variance 1, i.e. lim P (a Y n b) = 1 a e x2 /2 dx n This, along with the law of large numbers, is the most important theorem in this class. When X n are binomial this is just the DeMoivre-Laplace theorem, but the point is that the conclusion holds without knowing anything about the actual distribution. It explains why normal distributions occur so often in practice. Exercise 14. Suppose that 20 dice are rolled, and X is the total (sum of numbers on each die). Estimate P (90 X 110) using the above theorem. Solution: Let X i number on ith die. We have 6 E(X i ) = i/6 = 7/2 V ar(x i ) = By the central limit theorem where a = 90 70 35 12 20 1 6 i 2 /6 (7/2) 2 = 35/12 1 P (90 X 110) = P (a X 70 b) 20 and b = 110 70 b 35 12 1 a e x2 /2 dx 35 12 20. software, or using a calculator and tables. b Note this integral is easy to compute using