ADVANCE TOPICS IN PROBABILITY THEORY

Transcription

1 ADVANCE TOPICS IN PROBABILITY THEORY NOTES COMPILED BY KATO LA Professor: Dr. Henry Landau Textbook: A First Course in Probability by Ross. Reference: An Introduction to Probability Theory and its Applications by Feller. Grade: Homework - 50%, Midterm - 5%, Final - 35%, and Paper - 0%. Topics: Probability Structure/Space Outcome/Weight Events Pascal, Fermant, Bernoulli, Laplace,... Counting [Combinatorics] Conditional Probability Independence Bayes Formula/Applications Random Variables Probability Distributions - Binomial - Poisson/Gaussian Expectation and Variance Joint Distributions Limiting Behavior Chebyshev s Inequality Probability as a Frequency - Law of Large Numbers Scaling for a Gaussian - Central Limit Theorem Empirical Estimation Random Walks Entropy Other Applications Statistics - Sampling Queuing - Operations Biology, Hereditary, Epidemic Information - Coding Decision Theory Date: Spring 203.

2 2 NOTES COMPILED BY KATO LA 203 January 23 Two Main Points: () Mathematical thinking ñ Ask yourself, others, and in-class. (2) The nature of mathematical modeling. outside world observations experiments Ñ our thinking axioms assumptions Ó Ð conclusions Probability Structure Definition: A sample space is the set of all possible outcomes of an experiment. Examples: (i) Flip two coins. Ñ What happens/is the outcome? Ñ (T, T), (H, T), (T, H), (H, H). p, q, p, 2q,, p, 6q (ii) Roll two dice:.. p6, q, p6, 2q,, p6, 6q (iii) Flip coins until we get heads: T H T, T T, H ñ Infinite sample space Definition: An event is some [or a] set of outcomes. Example: Flip a coin three times and the first two are heads: Sample space Event HHH, HHT, HT H, HTT T HH, T HT, TT H, TTT Note: Event Sample Space. Subsets of a set includes the set itself and the empty set.

3 Definition: Weight is Ppsq - weight Ppsq 0 [nonnegative] Pp0q 0 PpS q [sample space] ADVANCE TOPICS IN PROBABILITY THEORY 3 If E, F S with E X F Ø ñ PpE Y Fq PpEq PpFq. If S has a finite [n] of elements ñ If we want each n to be equal in weight, are assignment is Ppsq n # of elements in s P S, then for an event E, PpEq. This is known as # of elements in S Laplace s rule. This leads to counting [combinatorics]. Multiplication Principle: If experiment has any one of m outcomes, and for each of these outcomes from experiment, then experiment 2 can have any one of n outcomes, then experiments and 2 together have any one of mn outcomes. Examples: (i) How many ways can we arrange a, b, and c? - There are three choices to chose from, then two, the one. Thus, ! What about n things? ñ npn qpn 2q n! n n? An approximation of n! is Stirling s formula: n! 2πn. e (ii) License plates - k digits = 0 k. There are approximately 3.4 million people in the United States. If using only numerical digits, there are only nine choices. Thus, including j letters and l numbers, the possible number of license plates involving letters and numbers is 26 j 0 l. (iii) How many pairs are there among a, b, c, and d? Take all pairs and represent as the following: 4 3. However, some pairs are counted more than once. This is compensated by dividing by two like so: 4 3. The reason for using two is it takes two to form a pair. 2 npn q How many pairs are there among n? ñ 2 More generally, how many k element sets can we find in n elements? Consider the first element:

4 4 NOTES COMPILED BY KATO LA npn qpn 2q pn k npn qpn 2q pn k q q k! k things n choose k is the number of subsets of length k out of n. pn kq! pn kq! n! pn kq!k! 6 choose 3 is choosing subsets of size three for six things Ñ number of subsets, counting repetitions Ñ number of rearrangements 3! 3 2 6! 3!3!

5 ADVANCE TOPICS IN PROBABILITY THEORY January 30 world observations measurements Ñ frequencies of occurrence our ideas axioms calculations assumptions Ó Ð conclusion Probability Space: Outcomes of some experiment. Events: Collections of outcomes. P-Weight: Or probability. P is defined on events, denoted PpEq. Some facts: () PpEq 0; PpS q (2) If E X F Ø ñ PpE Y Fq PpEq PpFq What about the probability of the complement of an event E? Given events E and E : It is denoted : P E E Ó E PpEq P E ñ P E PpEq Examples: () Flipping a coin where the results are either heads (H) or tails (T) and the probabilities are as follows: PpHq 2, PpTq 2. (2) Let E and E 2 be two events. We already know if, PpE X E 2 q Ø, then PpE Y E 2 q PpE q PpE 2 q, but what if PpE X E 2 q Ø? Ó E E 2

6 6 NOTES COMPILED BY KATO LA Then we have PpE Y E 2 q PpE q PpE 2 q PpE E 2 E XE 2 q (3) What about three events? E 0 E E 2 PpE Y E 2 Y E 3 q PpE q PpE 2 q PpE 3 q PpE E 2 q PpE E 3 q PpE 2 E 3 q [subtracting pairwise sets] PpE E 2 E 3 q [adding back the triple intersect] In general, for E, E 2,..., E n : PpE Y E 2 Y Y E n q ņ i PpE i q i i 2 PpE i E i2 q PpE E 2 E 3 q i i 2 i 3 p q n PpE E 2 E n q Suppose S set of outcomes is finite. Examples: (i) Roll a die ñ Six [6] outcomes (ii) Flip two coins ñ Four [4] outcomes If so, then [Laplace -Rule of Succession]: PpU i q N if we have N outcomes. Pp Event E U i,u 2,,U k q k N Number of outcomes in E N of outcomes

7 ADVANCE TOPICS IN PROBABILITY THEORY 7 npn q Suppose we have n objects and we are concerned with two people: for, 2,, n. 2 What are the number of pairs? What are the number of subsets of size k out of n? npn q pn pk qq Number of combinations of k npn qpn 2q pn pk qq k! Rationalizing the right-hand side by pn kqpn k q 2 npn qpn 2q pn pk qq k! pn kqpn k q 2 pn kqpn k q 2 n! pn kq! k! n choose k, also denoted as n k Note : By definition, n 0 Examples: 52 (i) Suppose we had a deck of cards where is the total number of poker hands. 5 What is the outcome for a Full House? Number of three-fold-value (ii) Suppose we have the integers from to 0,000. How many numbers have at least one 7 within the numbers? Consider the complement: How many numbers have no 7? ñ 9 k 0, 000

8 8 NOTES COMPILED BY KATO LA (iii) How many people are necessary for the chance that at least two people have the same birthday is greater than a half? Consider the complementary question/event: All birthdays are different. The point? It pays to look at the complement! P : k ñ k 23 Matching Suppose N people put hats in a box, and each retrieve a hat at random. What is the chance that no one gets their own hat?, Rearrangement of,, N. Ù - In this, how many are fixed? 2 5 N Complement Event: At least one person gets their own hat. Let E i be the event - i th person has their own hat. PpE Y E 2 Y Y E n q PpE q PpE i E i2 q At least one person has their own hat PpEi E i2 E i3 q (by the Inclusion-Exclusion Principle)! 2! 3! series

9 Continuing the previous problem... ADVANCE TOPICS IN PROBABILITY THEORY February 6 Matchings: n people leave their hats and return and pick a hat at random. What is the chance that no one gets their own hat? Suppose there were only four people: A = a grabs their own hat. ñ PpAq 4 B = b grabs their own hat. ñ PpBq 4 C = c grabs their own hat. ñ PpCq 4 D = d grabs their own hat. ñ PpDq 4 Hats a b c d PpNo one grabs their own hatq PpAt least one person gets their own hatq Once everyone picks up a hat, one permutation of, 2, 3, and 4 is Total number of outcomes is 4! a b c d Ppat least one gets their own hatq PpAq PpBq PpCq PpDq Two people get their own hats Three people get Everyone gets their own hats their own hats PpNo one has their own hatq. e x x 2 2! 2! 3! 4! x 3 3!

10 0 NOTES COMPILED BY KATO LA Condition Probability S E E X T T Where PpEq Number of elements in E PpE X Tq. Suppose we are in T. PpE Tq def Number of elements in S PpTq Definition: Two events E and F are independent if PpE X Fq PpEqPpFq ñ PpE Fq PpE X Fq PpFq PpEqPpFq PpFq PpEq

11 ADVANCE TOPICS IN PROBABILITY THEORY 203 February 3 Probability space S with the following possible outcomes: If S is finite, then px, x 2,, x n q. If D some weight, P px k q n [Laplace] What if S has an infinite number [not finite] number of outcomes? Example: Toss a coin until it turns up heads. i.e. H T H TT H T T k-times With the above example, what is the weight of PpHq? PpT Hq? PpT T Hq? As it turns out, H H PpHq 2 p T H PpT Hq 4 qp... TT T H PpTT T Hq 2 k q k p k-times ñ If H-weight is p, then T-weight is p. To add a a 2, we add N times: a a 2 a n S N ñ lim NÑ8 S N Recall: Geometric Progression r r 2 r 3 r k r r 2 r 3 r N S N

12 2 NOTES COMPILED BY KATO LA Notice, S N p rq p r r 2 r 3 r N qp rq r r 2 r 3 r N r r 2 r 3 r N r N S N p rq r N ñ S N rn r S N Ñ r Back to the above example and applying the geometric progression: pp q q 2 q 3 q k q p q p p p Example: Suppose two students arrive late to a final. They are both using the same lie - They were carpooling and had a flat tired, but forgot a minor detail - which tire the agreed on to designate as the flat tire. What is the probability that chosen at random, they both state the same tire was flat? The probability space is 6 because the first student has four wheels to choose from, and the second student has the same four wheels to choose from, yet they are independent events. Thus,

13 ADVANCE TOPICS IN PROBABILITY THEORY 3 A graphical representation of the probability of the students agreeing on the flat tire: student 2 student Where each point has an equal probability of of being chosen by each student respectively and 4 the meeting of each students choice is the product of the probabilities. ñ By equal weight, the probability two students agree on the same flat tire by random is 4 6. i.e., The diagonal. 4 Suppose we use actual statical weight for tires failing: % 0% 2 58% 60% 3% 0% 3 4 8% 20% Then following the same diagonal pictured above and multiply the new rounded probabilities together: ñ Given the weighted probability outcomes for the tire, the chances of the two students agreeing by random is 42%. Independent Events Definition: Two events E and E 2 are independent if PpE XE 2 qrordenotedppe E 2 qs is PpE qppe 2 q. Question: If E X E 2 Ø, can they be independent? Note: Events that are independent are never disjoint.

14 4 NOTES COMPILED BY KATO LA Conditional Probability Originally, PpAq weight is assigned to A as a fraction of S. We are now in B, reconsider B as the new probability space. Divide by the total weight of B to normalize. Rescale, and the probability space is no longer in S, given the weight of B is one. PpA Bq def Which can be rearranged as: PpA X Bq PpBq PpA X Bq PpA BqPpBq We know that A X B B X A, by the above definition PpA X Bq PpA BqPpBq and, PpB X Aq PpB AqPpAq ñ PpA BqPpBq PpB AqPpAq For two events A and B, the denominator can be decomposed as follows when necessary: PpBq PpB AqPpAq P B A P A PpB Aq PpAq PpB X A q Suppose now B B B 2 B k where all B i are disjoint from one another. Then for k events, the denominator can be decomposed in the same manner: ñ PpBq PpB q PpB 2 q PpB k q PpB X B q PpB X B 2 q PpB X B k q PpB B qppb q PpB B 2 qppb 2 q PpB B k qppb k q

15 ADVANCE TOPICS IN PROBABILITY THEORY 5 Thus, PpA Bq PpA X Bq PpBq PpB AqPpAq PpBq PpB AqPpAq PpB AqPpAq P B A P A PpB AqPpAq PpB AqPpAq P B A PpAq Example: Suppose a student either knows the answer rps, or guesses the answer to a question. If the student guesses, they have a probability of getting the question correct. Given the correct m answer, what is the probability the student knew the answer? Let K denote the student knowing the answer and CA denote the correct answer. Then, PpK CAq PpK and CAq PpCAq Decomposing the denominator, PpCA KqPpKq PpCAq PpCA KqPpKq PpCA KqPpKq P CA K P K p p mp pq p p p mp pq What about the probability of the student getting the correct answer given they guessed [G]? PpAC Gq p p m m

16 6 NOTES COMPILED BY KATO LA 203 February 20 Review: Bayes Theorem S A B Suppose we are in B. Then PpA Bq PpA X Bq PpBq ô PpA X Bq PpA BqPpBq. B replaces S as the probability space. If S, S 2,, S n is a subdivision of space into disjoint pieces. A S S 2 S n A A X S A X S 2 A X S n PpAq PpA X S q PpA X S 2 q PpA X S n q PpA S qpps q PpA S 2 qpps 2 q PpA S n qpps n q If A and B are independent, then PpA X Bq PpAqPpBq.

17 ADVANCE TOPICS IN PROBABILITY THEORY 7 Examples: () Suppose a family has two children. What is the probability that both children are girls? The probability is as well as each other possible outcome: Girl-Girl; Girl-Boy; Boy-Girl; and 4 Boy-Boy. Now, Suppose the first child is a girl. What is the probability of both children being girls given the first child is a girl? Let C denote the first child. PpGG C Gq 2 What about if one child is a girl? i.e., PpGG C Gq? Let s consider a graphical representation of all possible outcomes for two children: GG GB BG BB Analyzing the current probability space, we want to find the probability of both children being girls given that a child [order not specified] is a girl. Notice there are only three scenarios [i.e., New probability space] where a single child can be a girl: Girl-Girl; Girl-Boy; and Boy-Girl. Among those three outcomes, only one is the desired outcome: Girl-Girl. Thus, the probability of both children being girls given that a child is a girl is 3.

18 8 NOTES COMPILED BY KATO LA This time, we derive the same answer by using Bayes theorem. PpGG C Gq PpGG and C Gq PpC Gq Following Bayes theorem and decomposing the denominator, PpC G GGqPpGGq PpC G GGqPpGGq PpC G GBqPpGBq PpC G BGqPpBGq PpC G BBqPpBBq (2) Suppose there are two children in a house hold, what is the probability that a girl opens the door? Let OD denote opens door. Using Bayes theorem, PpGG OD Gq GG and OD G PpOD Gq PpOD G GGqPpGGq PpOD Gq Decomposing the denominator, ñ PpOD Gq PpOD G GGqPpGGq PpOD G GBqPpGBq PpOD G BGqPpBGq PpOD G BBqPpBBq 4?????? 0 ñ Not enough information! Bayes theorem reveals there is not enough information when decomposing the denominator. i.e, D circumstances we do not know such as the eldest child opening the door.

19 ADVANCE TOPICS IN PROBABILITY THEORY 9 (3) Suppose we have three cards. One card has two red faces, another has one red and one black face, and the last has two black faces. A card is picked at random and we see a red face. What is the probability the bottom face of the randomly picked card is also red? # #2 #3 R R B R B B PpBottom R See Rq PpSee R Bottom RqPpBottom Rq PpSee Rq Decomposing the denominator, notice the following pattern: The first term of the decomposed denominator [before the first addition sign] is the same as the numerator. This can be useful if the decomposed denominator is easier to determine than the numerator initially. PpSee R Bottom RqPpBottom Rq PpSee R C qppc q PpSee R C 2 qppc 2 q PpSee R C 3 qppc 3 q ñ PpSee R C qppc q PpSee R Bottom RqPpBottom Rq PpSee R C qppc q PpSee R C qppc q PpSee R C 2 qppc 2 q PpSee R C 3 qppc 3 q

20 20 NOTES COMPILED BY KATO LA 203 February 27