Probability COMP 245 STATISTICS. Dr N A Heard. 1 Sample Spaces and Events Sample Spaces Events Combinations of Events...

Transcription

1 Probability COMP 245 STATISTICS Dr N A Heard Contents Sample Spaces and Events. Sample Spaces Events Combinations of Events Probability 3 2. Axioms of Probability Simple Probability Results Independence Interpretations of Probability Examples 3. De Méré s problem Joint events Conditional Probability 8 4. Definition Examples Conditional Independence Bayes Theorem and the Partition Rule More Examples Sample Spaces and Events. Sample Spaces We consider a random experiment whose range of possible outcomes can be described by a set S, called the sample space. We use S as our universal set (Ω). Ex. Coin tossing: S = {H, T}. Die rolling: S = {,,,,, }. 2 coins: S = {(H, H), (H, T), (T, H), (T, T)}.

2 .2 Events An event E is any subset of the sample space, E S; it is a collection of some of the possible outcomes. Ex. Coin tossing: E = {H}, E = {T}. Die rolling: E = { }, E = {Even numbered face} = {,, }. 2 coins: E = {Head on the first toss} = {(H, H), (H, T)}. Extreme possible events are (the null event) or S. The singleton subsets of S (those subsets which contain exactly one element from S) are known as the elementary events of S. Subsets of S containing more than one outcome are known as compound events. Suppose we now perform this random experiment; the outcome will be a single element s S. Then for any event E S, we will say E has occurred if and only if s E. If E has not occurred, it must be that s / E s E, so E has occurred; so E can be read as the event not E. First notice that the smallest event which will have occurred will be the singleton {s }. For any other event E, E will occur if and only if E is a compound event and {s } E. Thus we can immediately draw two conclusions before the experiment has even been performed. Remark.. For any sample space S, the following statements will always be true:. the null event will never occur; 2. the universal event S will always occur. Hence it is only for events E in between these extreme events, E S for which we have uncertainty about whether E will occur. It is precisely for quantifying this uncertainty over these events that we require the notion of probability..3 Combinations of Events Set operators on events Consider a set of events {E, E 2,...}. The event i E i = {s S i s.t. s E i } will occur if and only if at least one of the events {E i } occurs. So E E 2 can be read as event E or E 2. The event i E i = {s S i, s E i } will occur if and only if all of the events {E i } occur. So E E 2 can be read as events E and E 2. The events are said to be mutually exclusive if i, j, E i E j = (i.e. they are disjoint). At most one of the events can occur. 2

3 2 Probability 2. Axioms of Probability σ-algebras of events So for our random experiment with sample space S of possible outcomes, for which events/subsets E S would we like to define the probability of E occurring? Every subset? If S is finite or even countable, then this is fine. But for uncountably infinite sample spaces it can be shown that we can very easily start off defining sensible, proper probabilities for an initial collection of subsets of S in a way that leaves it impossible to then carry on and consistently define probability for all the remaining subsets of S. For this reason, when defining a probability measure on S we (usually implicitly) simultaneously agree on a collection of subsets of S that we wish to measure with probability. Generically, we will refer to this set of subsets as S. There are three properties we will require of S, the reasons for which will become immediately apparent when we meet the axioms of probability. We need S to be. nonempty, S S; 2. closed under complements: E S = E S; 3. closed under countable union E, E 2,... S = i E i S. Such a collection of sets is known as a σ-algebra. Axioms of Probability A probability measure on the pair (S, S) is a mapping P : S [0, ] satisfying the following three axioms for all subsets of S on which it is defined (S, the measurable subsets of S). E S, 0 P(E) ; 2. P(S) = ; 3. Countably additive: For disjoint subsets E, E 2,... S ( ) P E i = P(E i ). i i 2.2 Simple Probability Results Exercises: From -3 it is easy to derive the following:. P(E) = P(E); 2. P( ) = 0; 3. For any events E and F, P(E F) = P(E) + P(F) P(E F). 3

4 2.3 Independence Two events E and F are said to be independent if and only if P(E F) = P(E)P(F). This is sometimes written E F. More generally, a set of events {E, E 2,...} are said to be independent if for any finite subset {E, E 2,..., E n }, ( ) n n P E i = P(E i ). i= i= If events E and F are independent, then E and F are also independent. Proof: Since F = (E F) (E F) is a disjoint union, P(F) = P(E F) + P(E F) by Axiom 3. So P(E F) = P(F) P(E F) = P(F) P(E)P(F) = ( P(E))P(F) = P(E)P(F). 2.4 Interpretations of Probability Classical If S is finite and the elementary events are considered equally likely, then the probability of an event E is the proportion of all outcomes in S in which lie inside E, Ex. P(E) = E S. Rolling a die: Elementary events are { }, { },..., { }. P({ }) = P({ }) =... = P({ }) =. P(Odd number) = P({,, }) = 3 = 2. Randomly drawn playing card: 52 elementary events { 2}, { 3},..., { A}, { 2}, { 3},...,..., { K}, { A}. P( ) = P( ) = P( ) = P( ) = 4. The joint event {Suit is red and value is 3} contains two of 52 elementary events, so P({red 3}) = 2 52 = 2. Since suit and face value should be independent, check that P({red 3}) = P({, }) P({any 3}). The equally likely (uniform) idea can be extended to infinite spaces, by apportioning probability to sets not by their cardinality but by other standard measures, like volume or mass. Ex. If a meteorite were to strike Earth, the probability that it will strike land rather than sea would be given by Total area of land Total area of Earth. 4

5 Frequentist Observation shows that if one takes repeated observations in identical random situations, in which event E may or may not occur, then the proportion of times in which E occurs tends to some limiting value - called the probability of E Ex. Proportion of heads in tosses of a coin: H, H, T, H, T, T, H, T, T, Subjective Probability is a degree of belief held by an individual. For example, De Finetti (937/94) suggested the following: Suppose a random experiment is to be performed, where an event E S may or may not happen. Now suppose an individual is entered into a game regarding this experiment where he has two choices, each leading to monetary (or utility) consequences:. Gamble: If E occurs, he wins ; if E occurs, he wins 0; 2. Stick: Regardless of the outcome of the experiment, he receives P(E) for some real number P(E). The critical value of P(E) for which the individual is indifferent between options and 2 is defined to be the individual s probability for the event E occurring. This procedure can be repeated for all possible events E in S. Suppose after this process of elicitation of the individual s preferences under the different events, we can simultaneously arrange an arbitrary number of monetary bets with the individual based on the outcome of the experiment. If it is possible to choose these bets in such a way that the individual is certain to lose money (this is called a Dutch Book ), then the individuals degrees of belief are said to be incoherent. To be coherent, it is easily seen, for example, that we must have 0 P(E) for all events E, E F = P(E) P(F), etc. 5

6 3 Examples 3. De Méré s problem Antoine Gombaud, chevalier de Méré (07-84) posed to Pascal the following gambling problem: which of these two events is more likely?. E = {4 rolls of a die yield at least one }. 2. F = {24 rolls of two dice yield at least one pair of }. De Méré observed that E seemed to lead to a profitable even money bet whereas F did not. We calculate P(E) and P(F).. Each roll of the die is independent from the previous rolls, and so there are 4 equally likely outcomes. Of these, 5 4 show no s. So the probability of no showing is So P(E), the probability of at least one showing, is = There are 3 24 equally likely outcomes here. Of these, don t show a. So the probability of no is So P(F), the probability of at least one, is = Hence P(E) > 2 > P(F). 3.2 Joint events Coin and Die Consider tossing a coin and rolling a die. We would consider each of the 2 possible combinations of Head/Tail and die value as equally likely. So we can construct a probability table: H 2 T From this table we can calculate the probability of any event we might be interested in, simply by adding up the probabilities of all the elementary events it contains. For example, the event of getting a head on the coin {H} = {(H, ), (H, ),..., (H, )}

7 has probability P({H}) = P({(H, )}) + P({(H, )}) P({(H, )}) = = 2. Notice the two experiments satisfy our probability definition of independence, since for example P({(H, )}) = 2 = 2 = P({H}) P({ }). Coin and Two Dice A crooked die called a top has the same faces on opposite sides. Suppose we have two dice, one normal and one which is a top with opposite faces numbered,, or. Now suppose we first flip the coin. If it comes up heads, we roll the normal die; tails, and we roll the top. To calculate the probability table easily, we notice that this is equivalent to the previous game using one normal die except with the change after tails that a roll of a is relabelled as a,,. So we can just merge those probabilities in the tails row. H T The probabilities of the different outcomes of the dice change according to the outcome of the coin toss. And note, for example, P({(H, )}) = 2 = 24 = 2 = P({H}) P({ }). 2 So the two experiments are now dependent. 7

8 4 Conditional Probability 4. Definition For two events E and F in S where P(F) = 0, we define the conditional probability of E occurring given that we know F has occurred as P(E F) = P(E F). P(F) Note that if E and F are independent, then P(E F) = P(E F) P(F) = P(E)P(F) P(F) = P(E). 4.2 Examples Example - Rolling a We roll a normal die once.. What is the probability of E = {the die shows a }? 2. What is the probability of E = {the die shows a } given we know F = {the die shows an odd number}? Solution:. P(E) = Number of ways a can come up Total number of possible outcomes =. 2. Now the set of possible outcomes is just F = {,, }. Number of ways a can come up So P(E F) = Total number of possible outcomes = 3. Note P(F) = 2 P(E F) and E F = E, and hence we have P(E F) =. P(F) Example 2 - Rolling two dice Suppose we roll two normal dice, one from each hand. Then the sample space comprises all of the ordered pairs of dice values S = {(, ), (, ),..., (, )}. Let E be the event that the die thrown from the left hand will show a larger value than the die thrown from the right hand. P(E) = # outcomes with left value > right total # outcomes 8 = 5 3.

9 Suppose we are now informed that an event F has occurred, where F = {the value of the left hand die is } How does this change the probability of E occurring? Well since F has occurred, the only sample space elements which could have possibly occurred are exactly those elements in F = {(, ), (, ), (, ), (, ), (, )(, )}. Similarly the only sample space elements in E that could have occurred now must be in E F = {(, ), (, ), (, ), (, )}. So our revised probability is # outcomes with left value > right total # outcomes (, ) = 4 = P(E F) P(F) P(E F). Discussion of Examples In both examples, we considered the probability of an event E, and then reconsidered what this probability would be if we were given the knowledge that F had occurred. What happened? Answer: The sample space S was replaced by F, and the event E was replaced by E F. So originally, we had P(E) = P(E S) = P(E S) P(S) (since E S = E, and P(S) = by Axiom 2). So we can think of probability conditioning as a shrinking of the sample space, with events replaced by their intersections with the reduced space and a consequent rescaling of probabilities. 4.3 Conditional Independence Earlier we met the concept of independence of events according to a probability measure P. We can now extend that idea to conditional probabilities since P( F) is itself a perfectly good probability measure obeying the axioms of probability. For three events E, E 2 and F, the event pair E and E 2 are said to be conditionally independent given F if and only if P(E E 2 F) = P(E F)P(E 2 F). This is sometimes written E E 2 F. 4.4 Bayes Theorem and the Partition Rule Bayes Theorem For two events E and F in S, we have but also, since E F F E P(E F) = P(F)P(E F); () P(E F) = P(E)P(F E). (2) Equating the RHS of () and (2), provided P(F) = 0 we can rearrange to obtain P(E F) = P(E)P(F E). P(F) 9

10 Partition Rule Consider a set of events {F, F 2,...} which form a partition of S. Then for any event E S P(E) = P(E F i )P(F i ). i Proof: E = E S = E i F i = i (E F i ). So ( ) P(E) = P (E F i ), i which, by countable additivity (Axiom 3) and noting that since the {F, F 2,...} are disjoint so are {E F, E F 2,...}, implies P(E) = P(E F i ). (3) i (3) is known as the law of total probability; and it can be rewritten P(E) = P(E F i )P(F i ). i For any events E and F in S, note that {F, F}, say, form a partition of S. So by the law of total probability we have P(E) = P(E F) + P(E F) = P(E F)P(F) + P(E F)P(F). Terminology When considering multiple events, say E and F, we often refer to probabilities of the form P(E F) as conditional probabilities; probabilities of the form P(E F) as joint probabilities; probabilities of the form P(E) as marginal probabilities. 0

11 4.5 More Examples Example - Defective Chips Ex. A box contains 5000 VLSI chips, 000 from company X and 4000 from Y. 0% of the chips made by X are defective and 5% of those made by Y are defective. If a randomly chosen chip is found to be defective, find the probability that it came from X. Let E = chip was made by X ; let F = chip is defective. First of all, which probabilities have we been given? A box contains 5000 VLSI chips, 000 from company X and 4000 from Y. = P(E) = = 0.2, P(E) = = % of the chips made by X are defective and 5% of those made by Y are defective. = P(F E) = 0% = 0., P(F E) = 5% = We have enough information to construct the probability table E E F F The law of total probability has enabled us to extract the marginal probabilities P(F) and P(F) as 0.0 and 0.94 respectively. So by Bayes Theorem we can calculate the conditional probabilities. In particular, we want P(E F) = P(E F) P(F) = = 3.

12 Example 2 - Kidney stones Kidney stones are small (< 2cm diam) or large (> 2 cm diam). Treatment can succeed or fail. The following data were collected from a sample of 700 patients with kidney stones. Success (S) Failure (S) Large (L) Small (L) Total For a patient randomly drawn from this sample, what is the probability that the outcome of treatment was successful, given the kidney stones were large? Clearly we can get the answer directly from the table by ignoring the small stone patients or we can go the long way round: P(L) = , P(S L) = , P(S L) = P(S L) = P(S L) P(L) = = Example 3 - Multiple Choice Question A multiple choice question has c available choices. Let p be the probability that the student knows the right answer, and p that he does not. When he doesn t know, he chooses an answer at random. Given that the answer the student chooses is correct, what is the probability that the student knew the correct answer? Let A be the event that the question is answered correctly; let K be the event that the student knew the correct answer. Then we require P(K A). By Bayes Theorem P(K A) = P(A K)P(K) P(A) and we know P(A K) = and P(K) = p, so it remains to find P(A). By the partition rule, P(A) = P(A K)P(K) + P(A K)P(K) and since P(A K) =, this gives c Hence P(A) = p + ( p). c P(K A) = p p + p c = cp cp + p. Note: the larger c is, the greater the probability that the student knew the answer, given that they answered correctly. 2

13 Example 4 - Super Computer Jobs Measurements at the North Carolina Super Computing Center (NCSC) on a certain day showed that 5% of the jobs came from Duke, 35% from UNC, and 50% from NC State University. Suppose that the probabilities that each of these jobs is a multitasking job is 0.0, 0.05, and 0.02 respectively.. Find the probabilities that a job chosen at random is a multitasking job. 2. Find the probability that a randomly chosen job comes from UNC, given that it is a multitasking job. Solution: Let U i = job is from university i, i =, 2, 3 for Duke, UNC, NC State respectively; let M = job uses multitasking.. 2. P(M) = P(M U )P(U ) + P(M U 2 )P(U 2 ) + P(M U 3 )P(U 3 ) = = P(U 2 M) = P(M U 2)P(U 2 ) P(M) = = Example 5 - HIV Test A new HIV test is claimed to correctly identify 95% of people who are really HIV positive and 98% of people who are really HIV negative. Is this acceptable? If only in a 000 of the population are HIV positive, what is the probability that someone who tests positive actually has HIV? Solution: Let H = has the HIV virus ; let T = test is positive. We have been given P(T H) = 0.95, P(T H) = 0.98 and P(H) = We wish to find P(H T). P(T H)P(H) P(H T) = P(T H)P(H) + P(T H)P(H) = = That is, less than 5% of those who test positive really have HIV. 3

14 Example 5 - continued If the HIV test shows a positive result, the individual might wish to retake the test. Suppose that the results of a person retaking the HIV test are conditionally independent given HIV status (clearly two results of the test would certainly not be unconditionally independent). If the test again gives a positive result, what is the probability that the person actually has HIV? Solution: Let T i = i th test is positive. P(H T T 2 ) = P(T T 2 H)P(H) P(T T 2 ) P(T = T 2 H)P(H) P(T T 2 H)P(H) + P(T T 2 H)P(H) P(T = H)P(T 2 H)P(H) P(T H)P(T 2 H)P(H) + P(T H)P(T 2 H)P(H) by conditional independence. Since P(T i H) = 0.95 and P(T i H) = 0.02, P(H T T 2 ) = So almost a 70% chance after taking the test twice and both times showing as positive. For three times, this goes up to 99%. 4