2 Chapter 2: Conditional Probability

Transcription

1 STAT 421 Lecture Notes 18 2 Chapter 2: Conditional Probability Consider a sample space S and two events A and B. For example, suppose that the equally likely sample space is S = {0, 1, 2,..., 99} and A is the event that the outcome is at least 50 and that B is the event of observing a prime. Hence, B = {2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97}. Note that B = 25 and AB = 10. Further, 1 2 = Pr(A) = Pr(AB) + Pr(ABc ). Suppose that it is known that B has occurred. The question of interest is, given this knowledge of B having happened, what is the probability of A? Definition of Conditional Probability Terminology: Pr(A B) is the conditional probability of A given, or conditional on, B. The conditioning event is B, and Pr(A B) is the probability of A given that B has happened. Recognizing that B has happened implies that for every y / B, Pr({y}) = 0 and Pr(B c ) = 0. Therefore, knowing that B has happened implies Pr(A B) = Pr(AB B) + Pr(AB c B). = Pr(AB B). (1) Usually, Pr(A B) Pr(A). Consider two cases: 1. Every outcome in B is equally likely. Then, the probability Pr(AB B) can be deduced by noting that the outcome of the experiment (x) is in B and that any of the outcomes in B are equally likely to belong to B. (We have no other information besides x B, hence, we must assume that every outcome in B is equally likely to be the outcome of the experiment. Hence, Pr(AB B) is the proportion of outcomes in B that are also in A, i.e., Pr(AB B) = =.4. In summary, Pr(A B) = Pr(x A x B) = Pr(x AB x B) = Pr(AB B). 1 The more general form of this question (given that B has happened, what is the probability of A?) is the subject of the chapter.

2 2 CHAPTER 2: CONDITIONAL PROBABILITY Outcomes in B may have different probabilities. We have no other information besides the outcome of the experiment is in B, and so while outcome probabilities in the conditional sample space may be different, the relative differences should remain the same. In other words, if x, y B, then Pr({x} B) Pr({y} B) = Pr({x}) Pr({y}). This second case actually covers the first case, so, in conclusion, the conditioning event generating a new sample space consisting of the conditioning event itself. Probabilities of outcomes in this new sample space are proportional their probabilities in the original space. Returning to the example (an example of Case 1), for every y B, 1 25 = Pr({y} B) = c Pr({y}) = c If the constant of proportionality c is known, then Pr(A B) = c Pr(AB), and determining conditional probabilities is trivial. The constant c can be determined using the fact that Pr(A B) = c Pr(AB) as follows: 1 = Pr(A B) + Pr(A c B) = c Pr(AB) + c Pr(A c B) = c Pr(B) c = 1 Pr(B). In either Case 1 or 2, we have : for any events A and B such that Pr(B) 0, Pr(A B) = Pr(AB) Pr(B). (2) If P (B) = 0, then conditioning on B makes no sense, and the conditional probability does not exist. Question 1 : Determine P (B A). Example A neighbor has two children. You learn that one child is a boy. What is the probability that the sibling of the boy is also a boy? The experiment of interest is that of selecting a random family having two children and recording the gender of the children. The sample space is S = {BB, BG, GB, GG}. Assuming boys and girls are equally likely to be born, S is an equally likely sample space. The

3 2 CHAPTER 2: CONDITIONAL PROBABILITY 20 two relevant events are that the neighbor has a son is the set E = {BB, BG, GB}, and the event that the neighbor has two boys is F = {BB}. Applying formula (2) yields Pr(F E) = Pr(F E) Pr(E) = Pr({BB}) Pr({BB, BG, GB}) = 1/4 3/4 = 1 3. If it was not known that the one child was a boy, then the probability that both are boys is 1/4. But knowing that one is a boy has eliminated the possibility of GG being the outcome of the experiment, and changed the probability of F. Example If three cards are selected at random (without replacement) from a standard deck of 52 cards, what is the probability there will be no pairs? (two cards of the same denomination). Let E i be the event that the first i cards have no pair among them. To proceed, observe that Pr(E 2 E 1 ) = Pr(E 1E 2 ) Pr(E 1 ) Pr(E 1 E 2 ) = Pr(E 2 E 1 ) Pr(E 1 ). Pr(E 1 E 2 ) is the probability that there are no pairs among in the first two cards selected. Further, Pr(E 1 E 2 E 3 ) is easily computed: Pr(E 3 E 1 E 2 ) = Pr(E 3E 1 E 2 ) Pr(E 1 E 2 ) Pr(E 1 E 2 E 3 ) = Pr(E 3 E 1 E 2 ) Pr(E 1 E 2 ) = Pr(E 3 E 1 E 2 ) Pr(E 2 E 1 ) Pr(E 1 ). Pr(E 1 E 2 E 3 ) = Pr(E 1 ) Pr(E 2 E 1 ) Pr(E 3 E 1 E 2 ) = Example: A study published in the journal Circulation examined the relationship between anger and coronary heart disease. Researchers examined 8,474 people, all of whom had normal blood pressure at the onset of the study. All participants took the Spielberger Trait Anger Scale test to measure proneness to anger and were followed for 4 years. The contingency table below tabulates the number of participants who fell into each coronary heart disease (CHD)/anger category. Totals for each level of CHD and anger level are given in the last column and row respectively. For example, of the 8,474 people in the study, 110 had CHD and were classified as having moderate anger.

4 2 CHAPTER 2: CONDITIONAL PROBABILITY 21 Anger Level Coronary Heart Disease Low Moderate High Total Presence Absence Total What is the conditional probability of coronary heart disease given that an individual has a high level of anger? Let H denote the event that a individual selected at random from among the subjects classified as having high anger levels and C denote the event that a individual selected at random from among the subjects classified as having coronary heart disease. The question asks for Pr(C H). Since sampling is random, all individuals satisfying the stated conditions are equally likely to be sampled, and Pr(H) = , Pr(CH) = , Pr(C H) = Pr(CH) Pr(H) = Pr(CH) Pr(H) = = = This probability is simply the proportion of individuals with high anger levels that have coronary heart disease. For comparison, the conditional probability that an individual with low anger levels has coronary heart disease is 53/3110 = At least among the individuals in this study, a person with a high anger level is almost three times more likely to have coronary heart disease as an individual with a low level. 2 The multiplication rule for conditional probabilities Suppose that Pr(B) = 0. Then, Pr(AB) = Pr(A B) Pr(B). Likewise, Pr(A) 0 Pr(AB) = Pr(B A) Pr(A). 2 If the statement is extended to a larger population, then statistical inferences are being drawn. The numerical values are surely different (since no two individuals are alike), and the next question to be asked is how accurate are these estimates?

5 2 CHAPTER 2: CONDITIONAL PROBABILITY 22 Example Suppose that 2 cards are drawn at random from a standard deck of 52 cards. The probability of obtaining 2 kings can be computed by multiplying the probability that the first drawn card is a king with the conditional probability that the second drawn card is a king conditional on the first being a king. Let A and B denote the events that the first and second cards are kings, respectively. Then, Pr(AB) = Pr(B A) Pr(A) = = Suppose that A 1,..., A n are events. The multiplication rule for conditional probabilities is extended as follows: P (A 1 A n ) = Pr(A n A 1 A n 1 ) Pr(A 1 A n 1 ) = Pr(A n A 1 A n 1 ) Pr(A n 1 A 1 A n 2 ) Pr(A 1 A n 2 ) =. = Pr(A n A 1 A n 1 ) Pr(A n 1 A 1 A n 2 ) Pr(A 2 A 1 ) Pr(A 1 ). Example Suppose that an urn contains n balls and that r are red and b are black, and n = r + b. If balls are drawn randomly and without replacement, then the probability of obtaining 3 reds in 3 draws is r r 1 r 2 r + b r 1 + b r 2 + b. Question 2 : What is the probability of obtaining 2 red balls in the experiment above (that is, supposing that 3 balls are drawn randomly and without replacement)? Conditional probabilities possess the same properties as conventional probabilities. For example, given that Pr(C) > 0, Pr(A B C) = Pr(A C) + Pr(B C) Pr(AB C) and Pr(A c C) = 1 Pr(A C). The arguments are straightforward; for example: Pr(A c B) = Pr(Ac B) Pr(B) Pr(B) Pr(AB) = Pr(B) = 1 Pr(AB) Pr(B) = 1 Pr(A B).

6 2 CHAPTER 2: CONDITIONAL PROBABILITY 23 Question 3 : Prove Pr(A B C) = Pr(A C) + Pr(B C) Pr(AB C). A related theorem shows that the conditional probability of a finite intersection of events can be expressed as a product o conditional probabilities. Theorem Suppose that Pr(B) > 0 and Pr(A 1 A n 1 B) > 0. Then, Pr(A 1 A 2 A n B) = Pr(A 1 B) Pr(A 2 A 1 B) Pr(A n A 1 A n 1 B). (3) The proof of formula (3) is based on the expansion of the righthand side as Pr(A 1 B) Pr(A 2 A 1 B) Pr(B) Pr(A 1 B) Pr(A n A n 1 A 1 B). Pr(A n 1 A 1 B) Conditional probabilities and partitions A partition of a set S is a set {B 1,..., B k } of disjoint sets such that S = k i=1 B i. Every A S can be partitioned as well: A = k (A B j ). (4) j=1 For example, an optical scanner classifies characters as letters based on the attributes of the character. A mathematical algorithm for classification defines k = 26 subpopulations corresponding to each letter (e.g., the event B 2 denotes the event that the character is a b). The event A corresponds to the character possessing a particular attribute. If an unclassified character possesses the attribute, then the character is classified as the letter with the maximum conditional probability among Pr(B 1 A),..., Pr(B 26 A). In practice, the conditional probabilities must be estimated, and the classifier uses the estimates in place of the true probabilities. From the mathematical perspective, the classification problem amounts to finding a good estimation method. Theorem The law of total probability states that if {B 1,..., B k } is a partition of the sample space S such that Pr(B j ) > 0 for every j, then for every A S, Pr(A) = k Pr(B j ) Pr(A B j ). (5) j=1 The proof is a straightforward and based on the partitioning of A by {B 1,..., B k } (formula 4). Example Returning to the anger level/coronary heart disease example, recall that there

7 2 CHAPTER 2: CONDITIONAL PROBABILITY 24 are 3 classes of anger level: high, medium, and low. The events that a randomly selected individual belongs to the subpopulations are H, M, and L, and the event that the individual has belongs to the coronary heart disease group is C. Suppose that an individual is not in the coronary heart disease group. It is most likely that the individual has a high anger level since Pr(H C) = =.073 Pr(M C) = =.558 Pr(L C) = =.369. Given that the individual has coronary heart disease, it is predicted that the individual s anger level is moderate. Example A screening program for a certain type of cancer can be analyzed by letting D denote the event that the disease is present and A is the event the test gives a positive result. It is assumed that Pr(A D) =.98 and Pr(A D c ) =.05, and that 1 in 10,000 of the population has the disease. Suppose that an individual has a positive test result. With no other information besides the individual belonging to the population, what is the probability that that person has the disease? Answer: We require Pr(D A). First, find Pr(A): Then, Pr(A) = Pr(A D) Pr(D) + Pr(A D c ) Pr(D c ) = = Pr(D A) = Pr(AD) Pr(A) = =.002. The person is still very unlikely to have the disease even though the test is positive. This simple example is unrealistic because of the assumption that without information from the test, all individuals are equally likely to have the cancer. In reality, other information (symptoms) almost always indicate that the individual may have the disease, implying that Pr(D) = 1/10, 000 =.0001 is not appropriate. without cause). (People are not tested for rare diseases

8 2 CHAPTER 2: CONDITIONAL PROBABILITY Independent Events Suppose that A and B are events. If knowing that A has occurred provides no information regarding the likelihood that B occurs, then A and B are independent. Mathematically, we have Pr(B A) = Pr(B); furthermore, knowing that B has occurred provides no information regarding the likelihood that A occurs, since Formal definitions of independence Pr(A B) = Pr(B A) Pr(A) Pr(B) = Pr(B) Pr(A) Pr(B) = Pr(A). Events E 1 and E 2 are said to be independent if Pr(E 1 E 2 ) = Pr(E 1 ) Pr(E 2 ). Consequently, it s straightforward to show that Pr(E 1 E 2 ) = Pr(E 1 ) and Pr(E 2 E 1 ) = Pr(E 2 ). Thus, knowledge of the occurrence of one of the events does not affect the likelihood of occurrence of the other. Events E 1,..., E k are pairwise independent if Pr(E i E j ) = Pr(E i ) Pr(E j ) for all i j. They are mutually independent if for all subsets A of {E 1,..., E k }, Pr( A E j ) = A Pr(E j). Mutual independence implies pairwise independence, but pairwise independence does not imply mutual independence. Example Redundancy is often used for critical systems. A redundant system is an independent, second system that runs in parallel with the first and replaces the first upon failure of the first. Suppose that E 1 and E 2 denote the event that systems 1 and 2 fail (system 2 is redundant). The probability that both fail given that Pr(E 1 ) = Pr(E 2 ) =.01 is Pr(E i E 2 ) = Pr(E 1 ) Pr(E 2 E 1 ) = =.001. A third redundant system reduces the probability of complete failure to Binomial example Suppose that there are n independent systems running in parallel, and that every system has a common probability p of failure during the operation of the systems. Let 0 k n denote the number of systems that fail. Then the probability that k fail can be determined by counting the number of combinations, or sequences, obtained from choosing k from n (order of failure is not important), and computing the probability of obtaining any

9 2 CHAPTER 2: CONDITIONAL PROBABILITY 26 particular combination. The probability of obtaining, say (f,..., f, s,..., s) ( ) is p k (1 p) n k and the number of sequences obtained from choosing k from n is n k ; hence, the probability that k fail is ( ) n k p k (1 p) n k. ( ) Looking ahead, n k p k (1 p) n k, is known as a binomial probability. Furthermore, the probability that at least one system fails is 1 (1 p) n since the probability that none fail is (1 p) n. Geometric example Suppose that a fair coin is tossed until a head is observed. The probability that 1 k tosses take place until the head is observed is p(1 p) (1 p) = (1 p) k 1, where p = 1/2. This is known as a geometric probability. Since there is no upper bound on k, the sum over all possible values for k is 1 = (1 p) k 1 p = k=1 Shifting the index yields the familiar geometric series identity 1 = (1 p) k p. k=0 Suppose that A and B are independent. Then Pr(A B c ) = Pr(A) Pr(A B) = Pr(A) Pr(A) Pr(B) = Pr(A)[1 Pr(B)] = Pr(A) Pr(B C ) and it can be concluded that A and B c are independent; further, A c and B are independent as are A c and B c. Mutually exclusive events A and B are events that cannot happen simultaneously such as observing an even number of heads in n tosses and observing a prime number of tosses in n tosses. Question 4 : Give either an example of mutually exclusive events that are independent, or an argument that mutually exclusive events cannot be independent.

10 2 CHAPTER 2: CONDITIONAL PROBABILITY Bayes Theorem Bayes theorem is a widely known and perhaps intimidating result that is nearly trivial. Suppose that B i, i = 1,..., k is a partition of a sample space (hence, i Pr(B i) = 1) such that Pr(B i ) 0 for each i. Since B i, i = 1,..., k is a partition, Pr(A) = Pr( k i=1a B i ) k = Pr(A B i ) Bayes theorem states that the conditional probability Pr(B j A) is = i 1 k Pr(B i ) Pr(A B i ). (6) i 1 Pr(B j A) = Pr(B j) Pr(A B j ) k i=1 Pr(B i) Pr(A B i ). The denominator is Pr(A) written as a sum as expressed in equation (6). Example Consider drug testing carried out at random among a population (e.g., employees or athletes). Suppose that 3% of the population are drug users and that a positive test result is obtained 95% of the time when the tested individual is a user, and returns a negative result 90% of the time when the tested individual is not a user. To determine the probability that the individual uses a banned substance (event U) given the positive test result (event P ), we apply Bayes Theorem: 1. Pr(P U) =.95 (the sensitivity of the test), 2. Pr(P c U c ) =.9 (the specificity of the test, or the probability that the test correctly identifies non-users), 3. Pr(U) =.03 (the prior probability, or the probability that a randomly selected individual is a user). The probability that an individual uses a banned substance given a positive test result is Pr(U P ) = Pr(U) Pr(P U) Pr(U) Pr(P U) + Pr(U c ) Pr(P U c ) = =.227. Example The sinking of the Titanic is a famous event, and the source of some information on western societal position at the beginning of the twentieth century. The data below (Table 1) were originally collected by the British Board of Trade in their investigation of the sinking

11 2 CHAPTER 2: CONDITIONAL PROBABILITY 28 though there is not complete agreement among primary sources as to the exact numbers on board, rescued, or lost. However, it is of interest to use these data to estimate the probability of survival given ticket class (a surrogate for societal status). Table 1 is a contingency table showing the cross-classification of passengers by survival status and ticket. Table 1: Titanic data. Class First Second Third Crew Total Survived Died Total The conditional distributions of survivorship (conditioning on ticket class) gives tells us the relative frequency of survivorship given ticket class. For example, given that the ticket class was first-class, the relative frequency of survival is 203/325 =.625, and the relative frequency of death is = 122/325 =.375. Table 2 shows the conditional probabilities of survival and death by ticket class. The probability of survivorship without knowledge of ticket class was Pr(S) = 711/2201 =.323. Table 2: Conditional probabilities of survival and death for Titanic passengers. Class First Second Third Crew Survived Died The ticket class distribution is shown below: Table 3: Probability that a randomly selected individual belongs to each ticket class. First Second Third Crew Probability Using Bayes theorem, probability that a survivor was a first-class ticket holder can be computed according to Pr(B 1 A) = Pr(B 1 ) Pr(A B 1 ) k i=1 Pr(B i) Pr(A B i ) where A is the event that a randomly selected passenger was a survivor, B 1 is the event that a randomly selected passenger was a first-class ticket holder, B 2 is the event that a randomly

12 2 CHAPTER 2: CONDITIONAL PROBABILITY 29 selected passenger was a second-class ticket holder, and so on. Thus Pr(B 1 A) = = =.348. The conditional probabilities for each of the classes, given survivorship are given in Table 4. Table 4: Conditional probability of ticket class given survivorship. First Second Third Crew Probability The conditional distribution of ticket class given survivorship suggests that society at the beginning of the twentieth century was relatively egalitarian. Is this the correct interpretation? 3 More insight can be gained by comparing Tables 3 and 4. Bayesian statistics is an approach to statistics that presumes that some quantitative information is available regarding a question of interest, yet additional data is collected and integrated with the prior information. In the the context of the Titanic questions, before examining the survivorship data, the prior probability of of a survivor belonging to a particular ticket class is the proportion of individuals belonging to each. After examining the survivorship data, the posterior probability of a survivor belonging to a particular ticket class given in Table 4. Problem 7, p. 85 To illustrate, problem 7 supposes that a box contains 5 coins and that the probabilities of observing heads given the coin that was tossed are Pr(H C 1 ) = 0 Pr(H C 2 ) = 1 4 Pr(H C 3 ) = 2 4 Pr(H C 4 ) = 3 4 Pr(H C 5 ) = 1 where selecting the ith coin is the event C i. A coin is randomly selected from the box, tossed, and a head is observed. The objective is to compute the probability that the ith coin was selected given the data 3 No, the probabilities in the table depend strongly on the number of ticket holders in each class, and those numbers reflect the allocation of ticket classes determined by the owners of the Titanic. More insight is gained from the conditional probabilities of survival given ticket class. Of course, given the chaos of the situation, we ought to be conservative in drawing inferences from the data.

13 2 CHAPTER 2: CONDITIONAL PROBABILITY 30 (namely: a head was observed), i.e., Pr(C i H), i = 1,..., 5. Note that Pr(C 1 H) = 0 since it is impossible to get a head if the first coin is tossed. However, it is not obvious what the updated probabilities ought to be for the remaining coins. We will use Bayes Theorem to compute these probabilities. Without knowing that the outcome was heads, we assume that each coin is equally likely to be selected (the coin was randomly sampled from the box). Hence, Pr(C i ) = 1/5, i = 1,..., 5. These values are the prior probabilities since the values are established before the event is observed. The general objective is to modify these prior probabilities by observing an event (or equivalently, data) that is informative. The modified probabilities are posterior probabilities since the values are computed after observing the event. Notationally, the posterior probabilities are Pr(C i H), where H is the event of observing a head when the randomly selected coin is tossed. The posterior probabilities are computed by applying Bayes theorem: Pr(C i H) = Pr(C i H) Pr(H) Pr(H C i ) Pr(C i ) = 5 j=1 Pr(H C j) Pr(C j ) The denominator is Pr(H) = 5 Pr(H C j ) Pr(C j ) j=1 = = 1 2. Then, the posterior probability of drawing the second coin is Pr(C 2 H) = Pr(C 2 H) Pr(H) = = 1 10.

14 2 CHAPTER 2: CONDITIONAL PROBABILITY 31 Part b asks for Pr(H), the probability of obtaining another head (given a second toss of the same coin). The prior probabilities are carried over from part a: Pr(C 1 ) = 0 Pr(C 2 ) =.1 Pr(C 3 ) =.2 Pr(C 4 ) =.3 Pr(C 5 ) =.4. Then, Pr(H) = 5 Pr(H C j ) Pr(C j ) = = 3 4 =.75. j=1