Chapter 2 Probability Probability theory is a branch of mathematics dealing with chance phenomena. The origins of the subject date back to the Italian mathematician Cardano about 1550, and French mathematicians Pascal and Fermat in 1654. The modern mathematical foundation for probability theory was laid down by Russian mathematician A. N. Kolmogorov (1903-1989). Cardano (1501-1576) was a passionate gambler. Around 1550, Cardano wrote a little book on Dice Game, in which he develops a theory of odds and probabilities. The two major results of the book are: The probability that one of two exclusive events occurs equals the sum of their probabilities The probability that two independent events occurs simultaneously equals the product of their probabilities The book appears more as a diary than as a book. The book was never published, actually Cardano kept it secret. since he was the sole person in the world having a notion of odds and the capacity of computing them, it gave him a significant advantage over his opponents, and a good reason to keep his findings secret. The book was first discovered more than hundred years after his death. But at that time the theory of probabilities had been rediscovered by Fermat (1601-1665) and Pascal (1623-1662). In this chapter, we will concentrate on ideas from probability that we need to go more deeply into statistics. Examples: How can gambling which depends on the unpredictable fall of dice and cards be a profitable business for a casino? If you buy a lottery ticket every day for many years, how much will each ticket win on the average? Give a test for the AIDS virus to the employees of a small company. What is the chance of at least one positive test even if all the employees are free of the virus? What is the probability of drawing an ace from an ordinary deck of 52 playing cards? What is the probability of getting a heads when flipping a coin 2.1 Sample Space and Events A statistic takes various values in each individual sample but there is nevertheless a regular distribution in a large number of repetitions. This is called a random phenomenon. Toss a coin 100 times, what is the proportion of heads? Figure 4.1, p.291 A random phenomenon has outcomes that we cannot predict but that have a regular distribution in repetitions 1-1
Basic concepts: Experiment refers to any process of observation or measurement that (i) can be repeated, theoretically, an infinite number of times; (ii) has a well-defined set of possible outcomes. Sample space denote by Ω or S: The set of all possible outcomes of an experiment Event: a subset of the sample space Example 1.1 Roll a pair of coins, one red and one blue, what is the sample space? S = {(H, H), (H, T), (T, H), (T, T)} Example 1.2 Describe the event B that the total number of points rolled with the pair of dice is 7 B = {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)} Let A and B be any two events defined over the sample space Ω (a) The intersection of A and B, denoted by A B, (A and B), is the event whose outcomes belong to both A and B (both A and B occur) (b) The union of A and B, denoted by A B, (A or B), is the event whose outcomes belong to either A or B (either A or B occurs) 1-2
Example 1.3 Ω = {1, 2,..., 6}. (a) A = {2, 3, 4}, B = {2, 4, 6}. Then (b) A = {1, 3, 5}, B = {2, 4, 6}. Then A B = {2, 4}, A B = {2, 3, 4, 6} A B = {1, 2, 3, 4, 5, 6}, A B = A and B are said to be mutually exclusive (disjoint) if they have no outcomes in common, that is, A B =, null set, emptyset The complement of A, denoted by A c or A, is the event consisting of all the outcomes in Ω other than those contained in A ( A does not occur) Remark: A and A c are disjoint, A A c = Ω De Morgan Law: (A B) c = A c B c (A B) c = A c B c 1-3
2.2 Counting Sample Points 2.2.1 The Basic Principle of Counting If an operation consist of a sequence of k separate steps of which the first can be performed in n 1 ways, followed by the second n 2 ways, and so on until the kth can be performed in n k ways, then the operation consisting of k steps can be performed by n 1 n 2 n k ways. Example 1.4 Select 3 digits from 0, 1, 2, 3, 4, 5, and 6. (a) How many three-digit numbers can be formed? (b) How many of these are odd numbers? (c) How many are greater than 330? (a) The digit in the hundreds position can t be 0. The answer is 6 6 5 = 180 (b) The digit in the units position is odd and the digit in the hundreds position is not zero. (c) 5 5 3 = 75 Case I: The digit in the hundreds position is greater than 3 3 6 5 = 90 Case II: The digit in the hundreds position is equal to 3, and the digit in the tens position is bigger than 3 3 5 = 15 1-4
Example 1.5 In how many different ways can one answer all the questions of a true-false test consisting of 20 questions? 2 20 = 1, 048, 576 2.2.2 Permutations A permutation is an arrangement of all or part of a set of objects. The number of permutations of n distinct objects is n! Example 1.6 4 boys and 4 girls are to be seated in a row of chairs numbered 1 through 8. (a) How many total arrangements are possible? (b) If one boy and one girl want to sit together, how many arrangements are possible? (a) 8! = 40320 (b) 2! 7! = 10080 The number of permutations of the four letters a, b, c, d is 24, but what is the number of permutations if we take only two of the four letters or, as it is usually put, if we take the four letters two at a time? The number of permutations of n distinct objects taken r at a time is: np r = n! (n r)! Example 1.7 How many different letter arrangements can be formed using the letters PROB- ABILITY? 1-5
11!/(2!2!)=.. The number of distinct permutations of n objects, of which n 1 are of one kind, n 2 are of a second kind, n k are of a kth kind, and n 1 + n 2 + + n k = n is n! n 1! n 2! n k! The number of permutations of n distinct objects arranged in a circle is (n 1)! 1-6
2.2.3 Combinations In many problems we are interested in the number of ways of selecting r objects from n without regard to order. These selections are called combinations. The number of combinations of n distinct objects taken r at a time is ( ) n n! = r r! (n r)! 0! = 1, ( ( n 1) = n, n 2) = n(n 1)/2 ( ) n The values are often called binomial coefficients r Example 1.8 From a group of 5 women and 7 men, how many different committees consisting of 2 women and 3 men can be formed? What if 2 of the men are feuding and refuse to serve on the committee together? The number of ways of selecting 2 women from 5 is ( ) 5 = 5! 2 2! 3! = 10. The number of ways selecting 3 men from 7 is ( ) 7 = 7! 3 3! 4! = 35. It follows from the basic principle that there are 10 35 = 350 possible committees consisting of 2 women and 3 men. If 2 of the men refuse to serve on the committee together, then there are ( )( ) 2 5 0 3 possible groups of 3 men not containing either of the 2 feuding men, and ( )( ) 2 5 1 2 1-7
groups of 3 men containing exactly 1 of the feuding men. Hence, there are ( )( ) ( )( ) 2 5 2 5 + = 30 0 3 1 2 ( ) 5 groups of 3 men not containing both of the feuding men. Since there are = 10 ways to 2 choose the 2 women, it follows that, in this case, there are 30 10 = 300 possible committees. The number of ways of partitioning a set of n distinct objects into r distinct groups with n 1 elements in the first group, n 2 elements in the second, and so forth, is ( ) n n! = n 1, n 2,..., n r n 1! n 2! n r! where n 1 + n 2 + + n r = n. 1-8
2.3 Probability Measure A Probability Measure on Ω is a function P from subsets of Ω to [0, 1] that satisfies the following axioms: (1) P (Ω) = 1 (2) For any event A, P (A) 0 (3) If A and B disjoint, then P (A B) = P (A) + P (B) More general, if A 1, A 2,..., A k are disjoint, then P (A 1 A 2 A k ) = P (A 1 ) + P (A 2 ) + + P (A k ) Probability (chance) of an event is the proportion of times the event occurs in a very long series of repetitions. Properties of Probability Measure Complement rule: P (A c ) = 1 P (A) P ( ) = 0 If A B, then P (A) P (B) For any two events A and B, P (A B) = P (A) + P (B) P (A B) 1-9
2.4 Assigning Probabilities Empirical Probability/ Relative Frequency If an experiment is repeated n times and an event A is observed f times, then its empirical probability of event A to happen is P (A) = f/n Subjective Probability - the probability assigned to an event based on subjective judgment, experience, information or belief Theoretical probability 2.4.1 Assigning Probabilities: Finite number of outcomes Suppose that a sample space consists of a finite number of outcomes. Assign a probability to each individual outcome. These probabilities must be numbers between 0 and 1 and must have sum 1. Probability of an event is the sum of the probabilities of outcomes making up the event Example 1.1 All human blood can be typed as one of O, A, B, or AB. Here are the probabilities that the person you choose will have blood type O, A, or B Blood type O A B AB Probability.47.27.2? What is the probability that the person chosen has either type O or type AB? P (T ypeab) = 1 (.47 +.27 +.2) = 0.06, P (T ypeo or T ypeab) = 0.47 +.06 = 0.53 1-10
Example 1.9 An experiment has four possible outcomes, A, B, C, D, which are mutually exclusive. Explain why the following assignments of probabilities are not permissible. (a) P (A) =.12, P (B) =.63, P (C) =.45, P (D) =.2 (b) P (A) =.12, P (B) =.63, P (C) =.1, P (D) =.1 (c) P (A) =.12, P (B) =.63, P (C) =.1, P (D) =.35 2.4.2 Assigning probabilities: equally likely outcomes If an experiment has k possible outcomes, all equally likely, then each individual outcome has probability 1 k P (A) = count of outcomes in A k Example 1.2 Toss a fair die. Let A be the event that an even number appears and B the event that number 3 or 5 occurs. Find P (A) and P (B). P (A) = 3/6 = 1/2, P (B) = 2/6 Example 1.3 Flip a fair coin 4 times. What is the probability that at least one heads occurs? P (at least one heads) = 1 (1/2) 4 = 15/16 Example 1.4 Powerball: select 5 numbers from 1 49, select 1 number from 1-42. What is the chance you win the prize? 1-11
P (win) = ( 49 5 1 )( 42 ) = 1 1 79, 179, 128 Some statisticians say you have a better chance of dying in a plane crash, being struck by a killer lightning bolt or suffering a fatal fall from your bed. Example 1.5 (Birthday problem). If n people are present in a room, what is the probability that at least two of them celebrate their birthday on the same day of the year? How large need n be so that this probability is greater than 1/2? As each person can celebrate his or her birthday on any one of 365 days, there is total of (365) n possible outcomes. Let A be the event that at least two of them celebrate their birthday on the same day. Then P (A) = 1 P (A ), where A is the event that no two of them celebrate their birthday on the same day. Noting that the number of points in A is 365 364 363 (365 n + 1), we have P (A ) = = 365 364 363 (365 n + 1) (365) n 365! (365) n (365 n)! and It is rather surprising that n P (A) 15.253 22.475 23.507 40.891 50.97 70.999 n 1 365! P (A) = 1 (365) n (365 n)! = 1 (1 i 365 ) P (A) i=1 > 1/2 for n = 23.97 for n = 50 > 3 106 3 10 6 + 1 for n = 100 1-12
2.5 Conditional Probability In one of the earliest studies (1936) establishing a link between smoking and lung cancer, two British doctors reported that of 135 men afflicted with lung cancer, 122 or 90% were heavy smokers. In nontechnical language, the inference is clear. If you have lung cancer, it is much more likely the case than not that you are a heavy smoker. Is the converse true? If you are a heavy smoker, does it necessarily follow that your chances of developing lung cancer are much higher than for a non-smoker? Example 1.6 Smoking and Lung Cancer Smoking Lung cancer Yes No Total Yes 122 13 135 No 2000 5000 7000 Total 2122 5013 7135 (a) If a person is smoking, what is the chance the person has a lung cancer? (b) If a person is not smoking, what is the chance the person has a lung cancer? P (Cancer Smoking) = 122/2122 = 0.05749, P (cancer nosmoking) = 13/5013 = 0.002593 Example 1.10 Toss a die, it is known an even number occurs, what is the probability it is a 2? Definition 1.1 If A and B are any two events in a sample space Ω and P (A) > 0, the conditional probability of B given A is P (B A) = P (A B) P (A) Solve Examples?? and??. Example 1.11 Mr. Smith has two children. It is known that he has at least one boy, what is the probability that he has two boys? 1-13
Let A be the event that he has at least one boy, B be the event that he has 2 boys. P (B A) = P (A B) P (A) = P (A) P (B) = 1/4 3/4 = 1 3 Properties of the conditional probability P (B A) 0 P (Ω A) = 1 If B 1, B 2,... are disjoint, then P (B A) = 1 P (B c A) P ( B i A) = P (B i A) Think A as a new sample space, the conditional probability is a probability measure defined on the new reduced sample space. Example 1.12 Three cards are randomly selected, without replacement, from an ordinary deck of 52 playing cards. Compute the conditional probability that the third card selected is a spade, given that the first and second cards are spades. Solution 1: Let A 3 be the event that the first card selected is a spade, and A 1,2 the event that the second and third cards are spades. Given A 1,2, imaging that we put aside the two spades, there are only 50 outcomes in the reduced sample space, and there are 11 spades left. Therefore, the probability of A 3, given A 1,2, equals 11/50. 1 Solution 2: P (A 3 A 1,2 ) = P (A 1,2A 3 P (A 1,2 ) 13 12 11 = 52 51 50 = 11 50 1-14 / 13 12 50 52 51 50
Example 1.7 The probability that a regularly scheduled flight departs on time is P (D) = 0.83; the probability that it arrives on time is P (A) = 0.82; and the probability that it departs and arrives on time is P (D A) = 0.78. Find the probability that a plane (a) arrives on time given that it departed on time, (b) departed on time given that it has arrived on time. (a) P (A D) = 0.78/0.83 = 0.94, (b) P (D A) = 0.78/0.82 = 0.95 The Multiplication Rule: P (AB) = P (A)P (B A) = P (B)P (A B) P (A 1 A 2 A n ) provided P (A 1 A n 1 ) > 0. = P (A 1 )P (A 2 A 1 )P (A 3 A 1 A 2 ) P (A n A 1 A n 1 ) Example 1.8 Select 5 cards randomly from a poker deck of 52 cards. What is the probability of (a) a full house (i.e., three cards of one denomination and two of another)? (b) pair? (c) a straight (i.e., having consecutive denominations but not all in the same suit)? (a) (b) ( 13 )( 4 )( 12 )( 4 1 3 1 2 ( 52 )( 5 12 ( 13 1 )( 4 2 ) ) = 0.00144 ) )( 4 4 4 3 1)( 1)( 1 ( 52 ) = 0.42 5 (c) 10 ( 4 5 1) 40 ) = 0.00392 ( 52 5 1-15
Example 1.13 An urn initially contains 5 white and 7 black balls. Each time a ball is selected, its color is noted and it is replaced in the urn along with 2 other balls of the same color. Compute the probability that (a) the first 2 balls selected are black and the next 2 white; (b) of the first 4 balls selected, exactly 2 are black. Let W i be the event that the i th ball selected is white, and B i the event that the i th ball selected is black. (a) By the multiplication rule, P (B 1 B 2 W 3 W 4 ) = P (B 1 )P (B 2 B 1 )P (W 3 B 1 B 2 )P (W 4 B 1 B 2 W 3 ) = 7 12 (b) The desired probability is 9 14 5 16 = 35 768 0.0456 7 18 P (B 1 B 2 W 3 W 4 ) + P (B 1 W 2 B 3 W 4 ) + P (B 1 W 2 W 3 B 4 ) + P (W 1 B 2 B 3 W 4 ) + P (W 1 B 2 W 3 B 4 ) + P (W 1 W 2 B 3 B 4 ) = 6 35 768 0.2734 Example 1.14 A man has n keys on a key ring, only one of which can open the door to his apartment. He chooses a key at random and tries it. If it fails to open the door, he will discard it and choose at random one of the remaining n 1 keys, and so on. What is the probability that he opens the door with the k th key he tries? 1 n 1-16
Law of total probability Let B 1, B 2,..., B k be a partition of the sample space Ω, i.e., B 1, B 2,..., B k are mutually exclusive events such that k i=1b i = Ω. Then for any event A P (A) = k P (B i )P (A B i ) i=1 Bayes formula Let B 1, B 2,..., B k be a partition of the sample space Ω. Then for any event A P (B r A) = P (B r)p (A B r ) P (A) = P (B r)p (A B r ) k P (B i )P (A B i ) i=1 Remark: For any two events A and B, P (A) = P (B)P (A B) + P (B c )P (A B c ) and P (B A) = P (B)P (A B) P (B)P (A B) + P (B c )P (A B c ) Remarks. 1. The law of the total probability is useful in computing the probability of a composite event, that is, an event A which depends on a series of causes B 1, B 2,. The formula tells us how to compute P (A) when the probabilities of the causes B 1, B 2, and the conditional probabilities of the event A given the cause B i are known. 2. Bayes rule has been interpreted as a formula for inverse probabilities : If B 1, B 2, is a series of causes and A is a possible effect, then P (A B i ) is the probability of A when it is known that B i is the cause, whereas P (B r A) is the probability that B r is the cause when it is known that A is the effect. 1-17
Example 1.15 A cancer diagnostic test is 99% accurate if the person has the disease, it is 98% accurate if the person does not have the disease. If 0.2% of the population have the disease. (a) What is the probability that the test result is positive? (b) What is the probability that a tested person has cancer, given that his test result is positive? Let A the test result is positive, B the person has the disease (a) P (A) = P (B)P (A B) + P (B )P (A B ) =.002 0.99 + 0.998 0.02 = 0.02194 (b) P (B A) = P (B)P (A B) P (A) = 0.002 0.99 0.02194 = 0.0902461 Example 1.16 During a power blackout, 100 persons are arrested on suspicion of looting. Each is given a polygraph test. From past experience it is known that the polygraph is 90% reliable when administrated to a guilty suspect and 98% reliable when given to someone who is innocent. Suppose that of the 100 persons taken into custody, only 12 were actually involved in any wrongdoing. What is the probability that a given suspect is innocent given that the polygraph says he is guilty? Let A be the event that the polygraph says the person is guilty, B the person is innocent. P (A) = P (B)P (A B) + P (B )P (A B ) =.88 0.02 + 0.12.9 =.1256 P (B)P (A B) P (B A) = =.88 0.02 = 0.0176 =.140127 P (A).1256.1256 Example 1.9 Have you cheated on a test in the past year? 38 students are randomly selected for the survey. Each student is given a fair coin and asked to flip the coin and answer yes to the question if the coin was heads and answer truthfully to the question if the flip of the coin was tails, but the student doesn t show the outcome of the coin flip. Let P (yes) be the probability of yes response and P (cheat) the probability that a student has cheated on a test. Find the relation of P (yes) and P (cheat). Suppose that 24 students among the 38 gave a yes response, what is the estimation of P (cheat)? 1-18
P (yes) = P (Heads)P (Y es Heads) + P (tails)p (yes tails) = 1/2 + 1/2P (cheat), P (cheat) = 2P (yes) 1 (cheat) = 2(24/38) 1 = 10/38 = 0.26 1-19
2.6 Independent Events Example 1.17 Select 2 balls from an urn containing 6 white and 5 black balls. What is the conditional probability that the second ball is black given that the first ball is black? Without replacement, P (B 2 B 1 ) = 4/10, P (B 2 ) = 5/11 With replacement, P (B 2 B 1 ) = 5/11, P (B 2 ) = 5/11 Generally speaking, P (B A) is not equal to P (B). In other words, knowing that A has occurred generally changes the chances of B s occurrence. However, the above example shows that there are some special cases where P (B A) does in fact equal P (B). Definition 1.2 A and B are said to be independent if P (A B) = P (A)P (B) Two events A and B that are not independent are said to be dependent. Remarks: If A and B are independent, then P (B A) = P (B), P (A B) = P (A) Suppose P (A) > 0, P (B) > 0. If A and B are independent, then A and B can not be mutually exclusive. If A and B are mutually exclusive, then A and B can not be independent. The sample space Ω as well as the empty set is independent of any event. If A and B are independent, then so are A and B c, A c and B c. Definition 1.3 The events A 1, A 2,..., A n are called mutually independent (or simply independent) if for all combinations 1 i < j < k < n the multiplication rules P (A i A j ) = P (A i )P (A j ) (1) P (A i A j A k ) = P (A i )P (A j )P (A k ). P (A 1 A 2 A n ) = P (A 1 )P (A 2 ) P (A n ) hold. 1-20
There are total 2 n n 1 equations! A very useful property: Suppose A 1, A 2,..., A n are independent, and T is a subset of {1, 2,..., n}. If B is an event formed from {A i, i T }, and C is an event formed from {A i, i T }, then B and C are independent. Definition 1.4 Suppose that an experiment consists of performing a sequence of subexperiments. The subexperiments are said to be independent if the outcomes of any group of the subexperiments have no effect on the probabilities of the outcomes of the other subexperiments. If each subexperiment has the same subsample space and the same probability function on its events, then the subexperiments are called trials. If the subexperiments are independent, then outcomes of different subexperiments are independent. Example 1.18 Toss a coin 4 times. Assume P (Heads) = 2/3. Find the probabilities of getting (a) No heads, (b) one heads, (c) 2 heads, (d) 3 heads, (e) 4 heads, (a) (1/3) 4 (b) 4(2/3)(1/3) 3 (c) ( ) 4 2 (2/3) 2 (1/3) 2 (d) ( ) 4 3 (2/3) 3 (1/3) 1 (e) (2/3) 4 1-21
Example 1.19 Perform an infinite sequence of independent trials. In each trial, P (success) = p, P (failure) = 1 p, 0 < p < 1 What is the probability that exactly k successes occur in the first n trials? Let X = # of successes in the first n trials, A i = {success on the i th trial}, i 1 P (X = k) =? Note that P (a sequence contains k successes and n k failures) = p k (1 p) n k ( ) n and that there are such sequences (containing k successes and n k failures). We have k ( ) n P (X = k) = p k (1 p) n k k Example 1.20 The first recorded probabilistic problem (Luca Pacioli, 1494) Two men A and B are playing balla. Both have set 10 gold coins at stake, and the one who first wins 6 games gets the 20 gold coins at stake. However, at a point where A has won 5 games and B has won 3 games, the play is interrupted. Suppose that the two men are equally skillful, how should they divide the stake? Answers: Luca Pacioli (1494) 5 : 3 Niccolo Tartaglia (1556) 2 : 1 G.F. Peverone (1558) 6 : 1 Blaise Pascal (1654) 7 : 1 Which of these answers is correct? P (B wins) = (1/2) 3 = 1/8. P (A wins) = 7/8 1-22