Topic 5: Jue 1, 2011 1 Itroductio Mathematical structures lie Euclidea geometry or algebraic fields are defied by a set of axioms. Mathematical reality is the developed through the itroductio of cocepts ad the proofs of theorems. These axioms are ispired, i the istaces itroduced above, by our ituitive uderstadig, for example, of the ature of parallel lies or the real umbers. Probability is a brach of mathematics based o three axioms ispired origially by calculatig chaces from card ad dice games. Statistics, i its role as a facilitator of sciece, begis with the collectio of data. From this collectio, we are ased to mae iferece o the state of ature, that is to determie the coditios that are liely to produce these data. Probability, i udertaig the tas of ivestigatig differig states of ature, taes the complemetary perspective. It begis by examiig radom pheomea, i.e., those whose exact outcomes are ucertai. Cosequetly, i order to determie the scietific reality behid the data, we must sped some time worig with the cocepts i the theory of probability to ivestigate properties of the possible states of ature to assess which are most useful i maig iferece from data. We will motivate the axioms of probability through the case of equally liely outcomes for some simple games of chace ad loo at some of the direct cosequeces of the axioms. I order to exted our ability to use the axioms, we will lear coutig techiques, e.g, permutatios ad combiatios, based o the multiplicatio priciple. A probability model has two essetial pieces of its descriptio. Ω, the sample space, the set of possible outcomes. A evet is a collectio of outcomes. We ca give explicitly defie ad evet via its outcomes, A = {ω 1, ω 2,, ω } or with a descriptio A = {ω : ω has property P}. I either case, A is subset of the sample space, A Ω. P, the probability assigs a umber to each evet. Thus, a probability is a fuctio. We are familiar with fuctios i which both the domai ad rage are subsets of the real umbers. The domai of a probability fuctio is the collectio of all possible outcomes. The rage is still a umber. We will see soo which umbers we will accept as probabilities of evets. c 2011 Joseph C. Watis 1
Data Aalysis ad Probability 2 Set Theory - Probability Theory Dictioary Evet Laguage Set Laguage Set Notatio Ve Diagram sample space uiversal set Ω evet subset A, B, C, outcome elemet ω impossible evet empty set ot A A complemet A c A or B A uio B A B A ad B A itersect B A B A ad B are A ad B are A B = mutually exclusive disjoit if A the B A is a subset of B A B 2
Data Aalysis ad Probability Equally Liely Outcomes The essetial relatioship betwee evets ad the probability are described through the three axioms of probability. These axioms ca be motivated through the first uses of probability, amely the case of equal liely outcomes. If Ω is a fiite sample space, the if each outcome is equally liely, we defie the probability of A as the fractio of outcomes that are i A. P (A) = #(A) #(Ω). Thus, computig P (A) meas coutig the umber of outcomes i the evet A ad the umber of outcomes i the sample space Ω ad dividig. Exercise 1. 1. Toss a coi. The Ω = {H, T } A = {H} P {heads} = #(A) #(Ω) =. 2. Toss a coi three times. Ω = { HHH, HHT, HT H, HT T, T HH, T HT, T T H, T T T } P {toss at least two heads i a row} = #(A) #(Ω) =. Roll two dice. Ω = { (1, 1), (1, 2), (1, ), (1, 4), (1, 5), (1, 6) (2, 1), (2, 2), (2, ), (2, 4), (2, 5), (2, 6) (, 1), (, 2), (, ), (, 4), (, 5), (, 6) (4, 1), (4, 2), (4, ), (4, 4), (4, 5), (4, 6) (5, 1), (5, 2), (5, ), (5, 4), (5, 5), (5, 6) (6, 1), (6, 2), (6, ), (6, 4), (6, 5), (6, 6)} P {sum is 7} = #(A) #(Ω) = Because we always have 0 #(A) #(Ω), we always have ad P (A) 0 (1) P (Ω) = 1 (2) This gives us 2 of the three axioms, The third will require more developmet. Toss a coi 4 times. #(Ω) = 16 A = {exactly heads} = {HHHT, HHTH, HTHH, THHH} #(A) = 4
Data Aalysis ad Probability P (A) = 4 16 = 1 4 B = {exactly 4 heads} = {HHHH} #(B) = 1 P (B) = 1 16 Now let s defie the set C = {at least three heads}. If you are ased the supply the probability of C, your ituitio is liely to give you a immediate aswer. P (C) =. Let s have a loo at this ituitio. The evets A ad B have o outcomes i commo,. We say that the two evets are disjoit or mutually exclusive ad write A B =. I this situatio, #(A B) = #(A) + #(B). If we tae this additio priciple ad divide by #(Ω), the we obtai the followig idetity: If A B =, the P (A B) = P (A) + P (B). () Usig this property, we see that P {at least heads} = P {exactly heads} + P {exactly 4 heads} = 4 16 + 1 16 = 5 16. We are sayig that ay fuctio P that accepts evets as its domai ad returs umbers as its rage ad satisfies (1), (2), ad () ca be called a probability. If we iterate the procedure i Axiom, we ca also state that if the evets, A 1, A 2,, A, are mutually exclusive, the P (A 1 A 2 A ) = P (A 1 ) + P (A 2 ) + + P (A ). ( ) This is a sufficiet defiitio for a probability if the sample space Ω is fiite. Thus, statemets (1), (2), ad ( ) give us the complete axioms of probability for a fiite sample space. 4 Cosequeces of the Axioms Other properties that we associate with a probability ca be derived from the axioms. 1. The Complemet Rule. Because A ad its complemet A c = {ω; ω / A} are mutually exclusive P (A) + P (A c ) = P (A A c ) = P (Ω) = 1 or P (A c ) = 1 P (A). For example, if we toss a biased coi. We may wat to say that P {heads} = p where p is ot ecessarily equal to 1/2. By ecessity, P {tails} = 1 p. Toss a coi 4 times. P {fewer tha heads} = 1 P {at least heads} = 1 5 16 = 11 16. 4
Data Aalysis ad Probability A A B B Figure 1: (left) Mootoicity Rule. If A B, the P (B \ A) = P (B) P (A). (right) The Iclusio-Exclusio Rule. P (A B) = P (A) + P (B) P (A B). Usig area as a aalogy for probability, P (B \ A) is the area betwee the circles ad the area P (A) + P (B) double couts the les area P (A B). 2. The Differece Rule Write B \ A to deote the outcomes that are i B but ot i A. If A B, the P (B \ A) = P (B) P (A). Because P (B \ A) 0, we have the followig:. Mootoicity Rule If A B, the P (A) P (B) We already ow that for ay evet A, P (A) 0. The mootoicity rule adds to this the fact that P (A) P (Ω) = 1. This, the rage of a probability are the umbers betwee 0 ad 1. 4. The Iclusio-Exclusio Rule. For ay two evets A ad B, P (A B) = P (A) + P (B) P (A B) (4). (P (A) + P (B) accouts for the outcomes i A B twice, so remove P (A B).) Deal two cards. A = {ace o the secod card}, B = {ace o the first card} P (A B) = P (A) + P (B) P (A B) P {at least oe ace} = 1 1 + 1 1? To complete this computatio, we will eed to compute P (A B) = P {both cards are aces}. 5. The Boferroi Iequality. For ay two evets A ad B, P (A B) P (A) + P (B). 5
Data Aalysis ad Probability 6. Cotiuity Property. Use the symbol to deote cotais i. If evets satisfy B 1 B 2 ad B = The, by the mootoicity rule, P (B i ) is a icreasig sequece satisfyig Similarly, use the symbol to deote cotais. If evets satisfy i=1 B i P (B) = lim i P (B i ). (5) C 1 C 2 ad C = Agai, by the mootoicity rule, P (C i ) is a decreasig sequece satisfyig Exercise 2 (odds). The statemet of a : b odds for a evet A idicates that Show that i=1 C i P (C) = lim i P (C i ). (6) P (A) P (A c ) = a b P (A) = a a + b. So, for example, 1 : 2 odds meas P (A) = 1/ ad 5 : odds meas P (A) = 5/8. 5 Coutig I the case of equally liely outcomes, fidig the probability of a evet A is the result of two coutig problems - amely fidig #(A), the umber of outcomes i A ad fidig #(Ω), the umber of outcomes i the probability space. These coutig problems ca become quite challegig ad advaced mathematical techiques have bee developed to address these problems.. However, havig some facility i coutig is ecessary to have a sufficietly rich umber of examples to give meaig to the axioms of probability. We begi this sectio o coutig with the multiplicatio priciple. Suppose that two experimets are to be performed. Experimet 1 ca have 1 possible outcomes ad for each outcome of experimet 1, experimet 2 has 2 possible outcomes. The together there are 1 2 possible outcomes. Example. For a group of idividuals, oe is chose to become the presidet ad a secod is chose to become the treasurer. By the multiplicatio priciple, if these positio are held by differet idividuals, the this tas ca be accomplished i ( 1) possible ways Exercise 4. Geeralize the multiplicatio priciple of coutig to experimets. 6
Data Aalysis ad Probability Assume that we have a collectio of objects ad we wish to mae a ordered arragemet of of these objects. Usig the geeralized priciple of coutig, the umber of possible outcomes is We will write this as () ad say fallig. Defiitio 5. The ordered arragemet of all objects is factorial. We tae 0! = 1. ( 1) ( + 1). () = ( 1) 1 =!, Exercise 6. () =! ( )!. 5.1 Combiatios Write ( ) for the umber of umber of differet groups of objects that ca be chose from a collectio of. We will ext fid a formula for this umber by coutig the umber of possible outcomes i two differet ways. To itroduce this with a cocrete example, suppose cities will be chose out of 8 uder cosideratio for a vacatio. If we thi of the vacatio as visitig three cities i a particular order, for example, The we are looig at permutatios. This results i New Yor the Bosto the Motreal. 8 7 6 choices. If we are just cosiderig the cities we visit, irrespective of order, the these uordered choices are combiatios. The umber of ways of doig this is writte ( ) 8, a umber that we do ot yet ow how to determie. After we have chose the three cities, we will also have to also pic a order to see the cities ad so usig the multiplicatio priciple, we have ( ) 8 2 1 possible vacatios if the order of the cities is icluded i the choice. These two strategies are coutig the same possible outcomes ad so must be equal. ( ) ( ) 8 8 8 7 6 = 2 1 or = 8 7 6 2 1. Thus, we have a formula for ( 8 ). Let s do this more geerally. Theorem 7. ( ) = ()! The secod equality follows from the previous exercise. =!!( )!. 7
Data Aalysis ad Probability We will form a ordered arragemet of objects from a collectio of by: 1. First choosig a group of objects. The umber of possible outcomes for this experimet is ( ). 2. The, arragig this objects i order. The umber of possible outcomes for this experimet is!. So, by the multiplicatio priciple, () = ( )!. Now complete the argumetst by dividig both sides by!. Exercise 8 (biomial theorem). (x + y) = =0 ( ) x y. Exercise 9. ( ) ( 1 = ( 1) =. ) ( = ( ). Thus, we set ( ) = 0) = 1 The umber of combiatios is computed i R usig choose. For example, ( ) 8 > choose(8,) [1] 56 Theorem 10 (Pascal s triagle). ( ) = ( ) 1 + 1 ( 1 ). To establish this idetity, distiguish oe of the objects i the collectio. Say that we are looig at a collectio of marbles, 1 are blue ad 1 is red. 1. For outcomes i which the red marble is chose, we must choose 1 marbles from the 1 blue marbles. Thus, ( 1 1) differet outcomes have the red marble. 2. If the red marble is ot chose, the we must choose blue marbles. Thus, ( ) 1 outcomes do ot have the red marbles.. These choices of groups of marbles have o overlap. Ad so ( ) is the sum of the values i 1 ad 2. To see this usig the example above, ( ) 8 = ( ) 7 + 2 ( ) 7. Assume that oe of the 8 cities o the list icludes New Yor. The of the ( ( 8 ) vacatios, 7 2) iclude New Yor ad ( 7 ) do ot. This gives us a iterative way to compute the values of ( ). Build a table of values for (vertically) ad (horizotally). The, by the Pascal s triagle formula, a give table etry is the sum of the umber directly above it ad the umber above ad oe colum to the left. We ca get started by otig that ( ( 0) = ) = 1. 8
Data Aalysis ad Probability 1 1 ( ) ( 1 1 ) 1 ( ) Pascal s triagle the sum of these two umbers equals this umber 0 1 2 4 5 6 7 8 0 1 1 1 1 2 1 2 1 1 1 4 1 4 6 4 1 5 1 5 10 10 5 1 6 1 6 15 20 15 6 1 7 1 7 21 5 5 21 7 1 8 1 8 28 56 70 56 28 8 1 Example 11. For the experimet o hoey bee quee - if we rear 60 of the 90 quee eggs, the we have > choose(90,60) [1] 6.71e+2 more tha 10 2 differet possible simple radom samples. Example 12. Deal out three cards. There are ( ) 52 possible outcomes. Let x be the umber of hearts. The we have chose x hearts out of 1 ad x cards that are ot hearts out of the remaiig 9. Thus, by the multiplicatio priciple there are ( ) ( ) 1 9 x x possible outcomes. To fid the probability of x hearts > x<-c(0:) > prob<-choose(1,x)*choose(9,-x)/choose(52,) > data.frame(x,prob) x prob 1 0 0.4152941 2 1 0.458825 2 0.1764706 4 0.01294118 Notice that > sum(prob) [1] 1 Exercise 1. Deal out 5 cards. Let x be the umber of fours. What values ca x tae? Fid the probability of x fours for each possible value. 6 Aswers to Selected Exercises 1. 1. 1/2, 2. /8,. 6/6 = 1/6 9
Data Aalysis ad Probability 7. If The, a b = P (A) P (A c ) = P (A) 1 P (A). a ap (A) = bp (A), a = (a + b)p (A), P (A) = a a + b. 9. Suppose that experimets are to be performed ad experimet i ca have i possible outcomes irrespective of the outcomes o the other 1 experimets. The together there are 1 2 possible outcomes. 12. () = ( 1) ( + 1) ( )! ( 1) ( + 1)( )!! = = ( )! ( )! ( )!. 14. Expasio of (x + y) = (x + y) (x + y) (x + y) will result i 2 terms. Each of the terms is achieved by oe choice of x or y from each of the factors i the product (x + y). Each oe of these terms will thus be a result i factors - some of them x ad the rest of them y. For a give from 0, 1,...,, we will see choices that will result i factors of x ad factors of y, i. e., x y. The umber of such choices is the combiatio ( ) Add these terms together to obtai ( ) x y. Next addig these values over the possible choices for results i (x + y) = =0 ( ) x y. 15. The formulas are easy to wor out. Oe way to cosider ( ) ( 1 = ( 1) is to ote that 1) is the umber of ways to choose 1 out of a possible. This is the same as ( 1), the umber of ways to exclude 1 out of a possible. A similar reasoig gives ( ) ( = ). 19. The possible values for x are 0, 1, 2,, ad 4. Whe we have chose x fours out of 4, we also have 5 x cards that are ot fours out of the remaiig 48. Thus, by the multiplicatio priciple, the probability of x fours is ( 4 ( x) 52 ) 5 x ( 52 5 ). 10