It applies to discrete and continuous random variables, and a mix of the two.

Size: px

Start display at page:

Download "It applies to discrete and continuous random variables, and a mix of the two."

Randolph Harper
5 years ago
Views:

1 3. Bayes Theorem 3.1 Bayes Theorem A straightforward application of conditioning: using p(x, y) = p(x y) p(y) = p(y x) p(x), we obtain Bayes theorem (also called Bayes rule) p(x y) = p(y x) p(x). p(y) It applies to discrete and continuous random variables, and a mix of the two. Deceptively simple, and the predictions are counterintuitive. combination. This is a very powerful 3.2 Examples of Bayes Theorem Let s do some simple examples, adapted from the book The Drunkard s Walk How Randomness Rules Our Lives by Leonard Mlodinow. (The solutions to these problems are given at the end of the chapter (Sec. 3.); they will be made available after the class. One of the purposes of this lecture is to demonstrate how counterintuitive probability can be.) Girls named Lulu Case 1: A family has two children. What is the probability that both are girls? Case 2: A family has two children. Given that one of them is a girl, what is the probability that both are girls? Case 3: A family has two children. Given that one of them is a girl named Lulu, what is the probability that both are girls? Disease diagnosis Doctors were asked to estimate the probability that a woman with no symptoms, between and 5 years old, who has a positive mammogram (i.e. the test reports cancer), actually has breast cancer. The doctors were given the following facts: The facts 7% of mammograms give false positives. 1% of mammograms give false negatives. Actual incidence of breast cancer in her age group is.8%. Date compiled: March 15, 218 1

2 Results Germany: A third of doctors said 9%, the median estimate was 7%. USA: 95% said the probability was approximately 75%. What do you think? Aren t you tempted to say 93%? Prosecutor s fallacy The facts Couple s first child died of SIDS (Sudden Infant Death Syndrome) at 11 weeks. They had another child, who also died of SIDS at 8 weeks. They were arrested and accused of smothering their children, even though there was no physical evidence. Results The prosecutor brought in an expert, who made a simple argument: the odds of SIDS are 1 in 8,53. Therefore, for two children: 1 in 73 million. The couple was convicted. Do you agree with the conviction? 3.3 Combining Priors with Observations Bayes theorem: x: unknown quantity of interest (in this class, usually the system state). p(x z) = p(z x) p(x) p(z) p(x): prior belief of state. z: observation related to state (usually a sensor measurement). p(z x): observation model: for a given state, what is the probability of observing z? p(x z): posterior belief of state: for a given observation, what is the probability that the state is x? p(z) = x p(z x) p(x), by the total probability theorem (here, for discrete random variables): probability of observation, essentially a normalization constant (does not depend on x). This is a systematic way of combining prior beliefs with observations. Question: If we observe the state x through z, doesn t that tell us the state directly? No! 1. Dimension: dim(z) usually smaller than dim(x). 2. Noise: p(z x) is not necessarily sharp. 2

3 Generalization to Multiple Observations We have N observations, each of which can be vector valued: z 1,..., z N. Often, we can assume conditional independence: p(z 1,..., z N x) = p(z 1 x) p(z N x) One interpretation: each measurement is a function of the state x corrupted by noise, and the noise is independent. Example: z i = g i (x, w i ); p(w 1,..., w N ) = p(w 1 ) p(w N ). Then: p(x z 1,..., z N ) = }} posterior Normalization: prior }} p(x) observation likelihood }} p(z i x) i p(z 1,..., z N ) }} normalization p(z 1,..., z N ) = x X p(x) i p(z i x) Example by the total probability theorem (here, for discrete random variables). Let the random variable z denote the outcome of N consecutive coin tosses. More specifically, we set z to be the number of times we observed heads. The coin might potentially be biased, but we have no prior information about the bias. We model the bias using the continuous random variable x, which we assume to be uniformly distributed in [, 1]. Given the bias x [, 1], heads occurs with probability x, whereas tails occurs with probability 1 x for a single coin toss. Calculate the posterior distribution of the coin bias, provided that we observed z heads in the N consecutive coin tosses. Given the bias x [, 1], the number of heads observed in N consecutive coin tosses is distributed according to a Binomial distribution. Hence, we have that ( ) N z x B(N, x), p z x ( z x) = x z (1 x) N z, z where ( ) N := z N! z!(n z)!. We can now use Bayes rule to infer the posterior distribution of the bias x. This leads to p x z ( x z) = p z x( z x) p x ( x) p z ( z) = ( N z ) x z (1 x) N z. p z ( z) 3

4 The normalization constant p z ( z) is independent of x and can be calculated by applying the total probability theorem, 1 1 ( ) ( ) N N z!(n z)! p z ( z) = p z x ( z λ) p x (λ) dλ = λ z (1 λ) N z dλ = z z (N + 1)!. The above integral can be evaluated using repeated integration by parts. This yields p x z ( x z) = (N + 1)! z!(n z)! x z (1 x) N z. We consider the special case when z = N, meaning that we observed N heads during the N consecutive coin tosses. The posterior probability density function of the bias reduces in that case to p x z ( x N) = (N + 1) x N. The probability density function p x z ( x N) is shown below. As N increases, the likelihood of having a coin with a bias towards heads increases. 1 p x z ( x N) 5 N = N = N = x We consider the case when z = N/2 and N is even. This represents the case that we observed N/2 heads and N/2 tails during the N coin tosses. The posterior probability density function of the bias reduces to p x z ( x N/2) = (N + 1)! ((N/2)!) 2 xn/2 (1 x) N/2 and is shown below. Note the symmetry around x = 1/2. 3 N = 1 p x z ( x N/2) 2 1 N = N = x

5 3. Solutions to Bayes Theorem Examples of Section Girls named Lulu Case 1: A family has two children. What is the probability that both are girls? Hopefully you said 1. (Assuming that the probability of a child being a boy or a girl is 1 2.) Case 2: A family has two children. Given that one of them is a girl, what is the probability that both are girls? The answer is 1 3, which can be seen as follows: Define: x = 1 : no boys in the family : boy in the family y = 1 : no girls in the family : girl in the family Then p x (1) = 1, p x() = 3, p y(1) = 1, p y() = 3 Therefore p x y (1 ) = p y x( 1)p x(1) p y() = = 1 3 Alternative approach: Generally, there are four possible gender combinations for the two children, which are all equally likely: (B,B), (B,G), (G,B), (G,G). Knowing that one of the children is a girl, only three options remain: (B,G), (G,B), (G,G). Therefore, the probability of (G,G) is 1 out of 3. Case 3: A family has two children. Given that one of them is a girl named Lulu, what is the probability that both are girls? Here, the key is that Lulu is an unusual name for a girl. Define: x = 1 : no boys in the family : boy in the family We want to know: p x y (1 ) = p y x( 1) p x (1) p y () y = 1 : no girl named Lulu in the family : one girl named Lulu in the family What is p y x ( 1)? Given that they are both girls, what is probability that there is a girl named Lulu in the family? Let r be the probability that someone names a girl Lulu, where we assume that r 1 (do you know anyone named Lulu?). Then consider the random variables c 1 and c 2, where c 1 refers to the first child, c 2 to the second child (note that we exclude the case that two girls in the same family are both named Lulu): c i = : not named Lulu 1 : named Lulu Therefore p y x ( 1) = 2r r 2 2r since r 1. c 1 c 2 p c1,c 2 x( c 1, c 2 1) (1 r)(1 r) 1 (1 r)r 1 r 1 1 How about p y (), the probability that there is a girl named Lulu in the family? Proceed similarly to above: 5

6 : boy c i = 1 : girl, not named Lulu 2 : girl, named Lulu c 1 c 2 p(c 1, c 2 ) (1 r) 2.25r 1.25(1 r) (1 r) r(1 r) 2.25r r 2 2 (Sanity check (sum):.25(1 + 1 r + r + 1 r + 1 2r + r 2 + r r 2 + 2r) =.25() = 1.) Therefore, p y () =.75r +.25r.25r 2 r. From the above, we get p x y (1 ) = p y x( 1) p x (1) p y () = (2r r2 ) 1 r 1 r2 2r 1 r 1 r = 1 2. Even though you can follow all the steps, the end result probably still seems wrong. An alternate, heuristic way to calculate this: assume 1 in 1 girls is named Lulu. Of 1, families with two children, 75, will have at least one girl: 5, will have a girl and a boy, and 25, will have two girls. Of the 5, girl/boy families, we expect 5 to have a girl named Lulu. Of the 25, girl/girl families, we expect 5 to have a girl named Lulu: 25 where the first-born is Lulu, 25 where the second born is Lulu. The probability is thus Disease diagnosis Define the following random variables: 1 : patient does not have cancer x = : patient has cancer y = 1 : test provides a negative result : test provides a positive result We want to know: We calculate p x y ( ) = p y x( ) p x (). p y () p y x ( ) =.9 (false negative rate = 1%), p x () =.8, and, using the total probability theorem p(y) = x p(y x) p(x), Therefore, we get p y () = p y x ( ) p x () + p y x ( 1) p x (1) p x y ( ) = = The probability is 9.%! Reason: most positive results are due to false positives. 6

7 3..3 Prosecutor s fallacy Problems with the conviction: The events are not independent. A more detailed study showed, chances of two cases of SIDS are 1 in 2.75 million however, these are still small odds. Inversion: the probability that two children die of SIDS is actually of no interest. What we want to know: (1) Given that two children have died, what is the probability that they died of SIDS? (2) Given that two children have died, what is the probability that they were murdered? These were calculated, and (1) is nine times more likely than (2). The conviction was eventually overturned. 7

Lecture Stat 302 Introduction to Probability - Slides 5

Lecture Stat 302 Introduction to Probability - Slides 5 AD Jan. 2010 AD () Jan. 2010 1 / 20 Conditional Probabilities Conditional Probability. Consider an experiment with sample space S. Let E and F be