PROILITY THEORY. asics Probability theory deals with the study of random phenomena, which under repeated experiments yield different outcomes that have certain underlying patterns about them. The notion of an experiment assumes a set of repeatable conditions that allow any number of identical repetitions. When an experiment is performed under these conditions, certain elementary events occur in different but completely uncertain ways. We can assign nonnegative number as the probability of the event in various ways: P ξ ), ( i ξ i ξ i
Laplace s Classical Definition: The Probability of an event is defined a-priori without actual experimentation as P ( ) Number of outcomes favorable to Total number of possible outcomes, (-) provided all these outcomes are equally likely. Consider a box with n white and m red balls. In this case, there are two elementary outcomes: white ball or red ball. n Probability of selecting a white ball. n + m We can use above classical definition to determine the probability that a given number is divisible by a prime p. 2
If p is a prime number, then every p th number (starting with p) is divisible by p. Thus among p consecutive integers there is one favorable outcome, and hence { } P a given number is divisible by a prime p Relative Frequency Definition: The probability of an event is defined as n ) lim (-3) n n where n is the number of occurrences of and n is the total number of trials. We can use the relative frequency definition to derive (-2) as well. To do this we argue that among the integers, 2, 3,, n, the numbers p, 2 p, are divisible by p. p (-2) 3
Thus there are n/p such numbers between and n. Hence P a given number N is divisiblebyaprimep In a similar manner, it follows that and P { } n/ p n. p lim (-4) n { p 2 divides any given number N} p 2 (-5) P { pq divides any given number N }. pq (-6) The axiomatic approach to probability, due to Kolmogorov, developed through a set of axioms (below) is generally recognized as superior to the above definitions, (-) and (-3), as it provides a solid foundation for complicated applications. 4
ξ i, The totality of all known a priori, constitutes a set Ω, the set of all experimental outcomes. Ω { ξ ξ,, ξ, }, 2 k (-7) Ω has subsets,, C,. Recall that if is a subset of Ω, then ξ implies ξ Ω. From and, we can generate other related subsets,,,, etc. { ξ ξ or ξ } { ξ ξ and ξ } and { ξ } ξ (-8) 5
If φ, the empty set, then and are said to be mutually exclusive (M.E). partition of Ω is a collection of mutually exclusive subsets of Ω such that their union is Ω. i j φ, Fig.. and i i Ω. (-9) 2 i φ Fig..2 j n 6
De-Morgan s Laws: ; (-0) Fig..3 Often it is meaningful to talk about at least some of the subsets of Ω as events, for which we must have mechanism to compute their probabilities. Example.: Consider the experiment where two coins are simultaneously tossed. The various elementary events are 7
ξ ( H, H), ξ2 ( H, T), ξ3 ( T, H), ξ4 ( T, T) and Ω { ξ ξ, ξ, }., 2 3 ξ 4 { } The subset ξ, ξ 2, ξ 3 is the same as Head has occurred at least once and qualifies as an event. Suppose two subsets and are both events, then consider Does an outcome belong to or Does an outcome belong to and Does an outcome fall outside? 8
Thus the sets,,,, etc., also qualify as events. We shall formalize this using the notion of a Field. Field: collection of subsets of a nonempty set Ω forms a field F if (i) (ii) (iii) Ω F If If F, F then and F F, then F. (-) Using (i) - (iii), it is easy to show that etc., also belong to F. For example, from (ii) we have F, F, and using (iii) this gives applying (ii) again we get where we have used De Morgan s theorem in (-0).,, F ; F, 9
Thus if F, F, then F { Ω,,,,,,,, }. (-2) From here on wards, we shall reserve the term event only for members of F. ssuming that the probability p i ξ i ) of elementary outcomes ξ i of Ω are apriori defined, how does one assign probabilities to more complicated events such as,,, etc., above? The three axioms of probability defined below can be used to achieve that goal. 0
xioms of Probability For any event, we assign a number ), called the probability of the event. This number satisfies the following three conditions that act as axioms of probability. (i) (ii) (iii) ) 0 (Probability is a nonnegative number) Ω) (Probabili ty of the whole set is unity) If φ, then ) +. (-3) (Note that (iii) states that if and are mutually exclusive (M.E.) events, the probability of their union is the sum of their probabilities.)
The following conclusions follow from these axioms: a. Since Ω, we have using (ii) ) P ( Ω ) ut φ, and using (iii),. ) ) + ) or ) ). (-4) b. Similarly, for any, { φ } { φ }. Hence it follows that ( { φ }) P ( ) P ( φ ). P + { φ }, { φ } 0. ut and thus P (-5) c. Suppose and are not mutually exclusive (M.E.) How does one compute P ( )? 2
To compute the above probability, we should re-express in terms of M.E. sets so that we can make use of the probability axioms. From Fig..4 we have, (-6) where and are clearly M.E. events. Thus using axiom (-3-iii) Fig..4 P ( ) P ( ) P ( ) + P ( To compute we can express as Thus, Ω ( ) ( ) ( ) P ( + ), since and are M.E. events. ). (-7) (-8) (-9) 3
From (-9), and using (-20) in (-7) ) +. Question: Suppose every member of a denumerably infinite collection i of pair wise disjoint sets is an event, then what can we say about their union i? (-22) i.e., suppose all i F, what about? Does it belong to F? Further, if also belongs to F, what about )? i (-20) (-2) (-23) (-24) 4
The above questions involving infinite sets can only be settled using our intuitive experience from plausible experiments. For example, in a coin tossing experiment, where the same coin is tossed indefinitely, define head eventually appears. (-25) Is an event? Our intuitive experience surely tells us that is an event. Let n { head appears for the st time on the nth toss} { t, t, t,, t, h} n Clearly i φ. Moreover the above is j 2 3 i. (-26) (-27) 5
We cannot use probability axiom (-3-iii) to compute ), since the axiom only deals with two (or a finite number) of M.E. events. To settle both questions above (-23)-(-24), extension of these notions must be done, based on our intuition, as new axioms. σ-field (Definition): field F is a σ-field if in addition to the three conditions in (-), we have the following: For every sequence i, i, of pair wise disjoint events belonging to F, their union also belongs to F, i.e., i F. (-28) i 6
In view of (-28), we can add yet another axiom to the set of probability axioms in (-3). (iv) If i are pair wise mutually exclusive, then P n P ( n ). (-29) n n Returning back to the coin tossing experiment, from experience we know that if we keep tossing a coin, eventually, a head must show up, i.e., P ( ). (-30) ut n, and using the fourth probability axiom n in (-29), ( ) P P n P ( n ). (-3) n n 7
From (-26), for a fair coin since only one in 2 n outcomes is in favor of n, we have n ) and ) n n 2 n n which agrees with (-30), thus justifying the, reasonableness of the fourth axiom in (-29). In summary, the triplet (Ω, F, P) composed of a nonempty set Ω of elementary events, a σ-field F of subsets of Ω, and a probability measure P on the sets in F subject the four axioms ((-3) and (-29)) form a probability model. The probability of more complicated events must follow from this framework by deduction. 2 n (-32) 8
Conditional Probability and Independence In N independent trials, suppose N, N, N denote the number of times events, and occur respectively. ccording to the frequency interpretation of probability, for large N N N N ),,. (-33) N N N mong the N occurrences of, only N of them are also found among the N occurrences of. Thus the ratio N N N / N (-34) N / N 9
is a measure of the event given that has already occurred. We denote this conditional probability by We define Probability of the event given that has occurred. P (, (-35) provided 0. s we show below, the above definition satisfies all probability axioms discussed earlier. 20
We have (i) ) 0 > 0 0, (-36) Ω (ii) Ω, since Ω. (-37) (iii) Suppose C 0. Then C ( C ) ut C φ, hence C ) satisfying all probability axioms in (-3). Thus (-35) defines a legitimate probability measure. C P ( C + + C, P ( C ) ) + C ).. (-38) (-39) 2
Properties of Conditional Probability: a. If,, and ) since if, then occurrence of implies automatic occurrence of the event. s an example: in a dice tossing experiment. Then, and b. If,, and (-40) {outcome is even}, {outcome is 2},. ) ) P ( > ). (-4) 22
(In a dice experiment, {outcome is 2}, {outcome is even}, so that. The statement that has occurred (outcome is even) makes the odds for outcome is 2 greater than without that information). c. We can use the conditional probability to express the probability of a complicated event in terms of simpler related events. Let,, 2, are pair wise disjoint and their union is Ω. Thus φ, and Thus i j n n i i Ω. ( 2 n ) 2 n (-42). (-43) 23
ut φ φ, so that from (-43) i j n i i With the notion of conditional probability, next we introduce the notion of independence of events. j n P ( ) P ( ) P ( ) P ( ). (-44) i i i i Independence: and are said to be independent events, if P ( ) P ( ) P ( ). (-45) Notice that the above definition is a probabilistic statement, not a set theoretic notion such as mutually exclusiveness. 24
Suppose and are independent, then P ( ) P ( ) P ( ) P ( ) P ( ). P ( ) P ( ) Thus if and are independent, the event that has occurred does not shed any more light into the event. It makes no difference to whether has occurred or not. n example will clarify the situation: Example.2: box contains 6 white and 4 black balls. Remove two balls at random without replacement. What is the probability that the first one is white and the second one is black? Let W first ball removed is white (-46) 2 second ball removed is black 25
We need W ) 2? We have Using the conditional probability rule, W. 2 W2 2W P ( W2 ) P ( 2W ) P ( 2 W) P ( W). (-47) ut and and hence 6 6 P ( W ) 6 + 4 0 P 4 2 W ) 5 + 4 ( 3 4 2 PW ( 2) 0.27. 5 9 45 4 9 3 5,, 26
re the events W and 2 independent? Our common sense says No. To verify this we need to compute 2 ). Of course the fate of the second ball very much depends on that of the first ball. The first ball has two options: W first ball is white or first ball is black. Note that W φ, and W Ω. Hence W together with form a partition. Thus (see (-42)-(-44)) and P ( ) P ( W) PW ( ) + P ( P ( ) 2 2 2 4 3 3 4 4 3 2 4+ 2 2 + +, 5+ 4 5 6+ 3 0 9 5 3 5 5 5 2 3 2 P ( 2) PW ( ) PW ( 2 ). 5 5 45 s expected, the events W and 2 are dependent. 27
From (-35), Similarly, from (-35) or P ( ) P ( ) P ( ). P ( ) P ( ) P ( ) From (-48)-(-49), we get P ( ) P ( ) P ( ) P ( ) P ( )., (-48) (-49) or P ( ) P ( ) P ( ) P ( ). P ( ) P ( ) P ( ) P ( ) Equation (-50) is known as ayes theorem. (-50) 28
lthough simple enough, ayes theorem has an interesting interpretation: ) represents the a-priori probability of the event. Suppose has occurred, and assume that and are not independent. How can this new information be used to update our knowledge about? ayes rule in (-50) take into account the new information ( has occurred ) and gives out the a-posteriori probability of given. We can also view the event as new knowledge obtained from a fresh experiment. We know something about as ). The new information is available in terms of. The new information should be used to improve our knowledge/understanding of. ayes theorem gives the exact mechanism for incorporating such new information. 29
more general version of ayes theorem involves partition of Ω. From (-50) P ( i ) P ( i ) P ( i ) P ( i ) P ( i ) n P ( ) P ( ) P ( i where we have made use of (-44). In (-5), i i represent a set of mutually exclusive events with associated a-priori probabilities i ), i n. With the new information has occurred, the information about i can be updated by the n conditional probabilities i ), i n, using ( - 47). i i ), (-5), n, 30
Example.3: Two boxes and 2 contain 00 and 200 light bulbs respectively. The first box ( ) has 5 defective bulbs and the second 5. Suppose a box is selected at random and one bulb is picked out. (a) What is the probability that it is defective? Solution: Note that box has 85 good and 5 defective bulbs. Similarly box 2 has 95 good and 5 defective bulbs. Let D Defective bulb is picked out. Then 5 5 P ( D ) 0.5, P ( D 2 ) 00 200 0.025. 3
Since a box is selected at random, they are equally likely. Thus and 2 form a partition as in (-43), and using (-44) we obtain P ( ) P ( 2 ) P ( D ) P ( D ) P ( ) + P ( D 2 ) P ( 2 0.5 + 0.025 0.0875. 2 2 Thus, there is about 9% probability that a bulb picked at random is defective. 2. ) 32
(b) Suppose we test the bulb and it is found to be defective. What is the probability that it came from box? P ( D ) P ( ) P ( D ) P ( D ) 0.5 / 2 0.0875 P ( D )? 0.857. (-52) Notice that initially then we picked out a box ) 0.5; at random and tested a bulb that turned out to be defective. Can this information shed some light about the fact that we might have picked up box? From (-52), and indeed it is more D) 0.857 > 0.5, likely at this point that we must have chosen box in favor of box 2. (Recall box has six times more defective bulbs compared to box2). 33