2 The Bayesian Perspective of Distributions Viewed as Information

Transcription

1 A PRIMER ON BAYESIAN INFERENCE For the next few assignments, we are going to fous on the Bayesian way of thinking and learn how a Bayesian approahes the problem of statistial modeling and inferene. The Bayesian Interpretation of Probability Dislaimer: I am going to use sports as an analogy so if you are not a sports person then I apologize up front but I think you ll still get the idea. If the Utah Jazz play the Golden State Warriors, what is the probability that the Jazz will win? This question has two answers: the frequentist answer and the Bayesian answer. A frequentist would define the probability of the Jazz winning as the long run frequeny of the Jazz winning if the Jazz and the Warriors played lots and lots of times. A Bayesian would laugh at this answer beause, honestly, you an t have the Jazz and the Warriors play over and over again. Even if they did play over and over, the Jazz and Warriors would adapt their game play so that omparing game to game wouldn t make a lot of sense beause the situation is different. Bayesian s view probability as information. Speifially, a Bayesian would define the probability of the Jazz winning as the information that I had about the event of the Jazz winning. Namely, Bayesians view the Jazz playing the Warriors as an event that hasn t happened yet and I use probability to quantify what I know about the event before it happens. For example, if I think the probability is 0.7 then I have strong beliefs (or information) that the Jazz will beat the Warriors. On the other hand, if that probability is 0. then I have strong beliefs (information) that the Warriors will beat the Jazz. In this light, Bayesians view events as unknown and use probability to quantify what they know (or think) about the event. They strongly argue against the frequentist view of the long-run frequeny beause any single event really an t be repliated over and over again (e.g. injuries an happen, different players get different playing time, et. to make the game fundamentally different). 2 The Bayesian Perspetive of Distributions Viewed as Information Lets look at a bit more ompliated of a situation and ompare Bayesian s to frequentists. Consider the N (0, ) distribution shown in Figure for a random variable x. The value x will take when I sample it is the event that I don t know the outome of (the same as winning/losing is the event that I don t know about before the Jazz play the Warriors). The shaded area orresponds to [, ] and, beause this is a normal distribution, we know Pr(x [, ]) = 0.68 by the rule. What you were taught in Stat 2 (and likely all other lasses) is that we interpret the probability Pr(x [, ]) = 0.68 as the relative frequeny of x [, ] relative to ALL values of x. That is, if you kept drawing random values of x from this distribution nearly forever, the perentage of all x s between [, ] would be This long-run frequeny interpretation of probability is what gives rise to the name frequentist whih is the perspetives that most statistis lasses use. The Bayesian interpretation of probability is that probabilities quantify information about an unknown quantity x. That is, Bayesian view x as an unknown event and beause Pr(x [, ]) = 0.68 we have information that the value of x is likely between [, ] beause this interval represents about 68% of the possible value that x ould be. Similarly, we know that x is not likely to be greater than 4 beause the Pr(x > 4) 0. Again in this setting, probabilities quantify information about what the event x ould be. From this perspetive, distribution urves (probability density funtions) suh as those in Figure give the information that I know about an, as of yet, unknown event. The magi of adopting the Bayesian perspetive of probability as information is it allows us to say things like µ N (0, ) where µ is the population mean (a quantity or event that we don t know). That

2 y is, as a Bayesian, if we say µ N (0, ) then we believe µ to most likely be between [ 2, 2] sine this interval is 95% of the whole (by the rule). Note that Bayesian do NOT think of µ as random in the sense that µ hanges (it obviously doesn t hange beause µ desribes an entire population). Rather, Bayesians believe µ is a single fixed number (the population mean) but we are simply using the N (0, ) distribution to quantify how muh we know about µ. In ontrast, frequentist s freak out when you say things like µ N (0, ) beause µ is a fixed number and there an t be a relative frequeny interpretation for µ beause µ never hanges. 3 The Bayesian Paradigm for Statistial Inferene From the Bayesian view (of distributions/probabilities as information rather than frequenies), Bayesian inferene (the proess of using observed data to infer the value of a population parameter) is a 3-step proess: (Step ) find a prior distribution that represents your knowledge of a parameter before you gather any data, (Step 2) relate the data you ollet to the parameter and (Step 3) update your prior distribution to reflet the knowledge you gained about the parameter from your observed data into a new distribution alled the posterior distribution. For Step #, we simply find a distribution that mathes our prior beliefs about the parameter we are interested in learning about. For example, if we want to learn about the population mean µ, a ommon distribution distribution is the N (m, s 2 ) distribution where we pik a value of m and s 2 to represent our prior beliefs. To illustrate, suppose we are interested in learning the mean number of hours of sleep per night that all BYU students get during finals week. Let µ represent the mean number of hours of sleep that BYU students get during finals week. Well, we know that, espeially during finals week, that students tend to go sleep deprived so we ould state our prior distribution as N (6, ). Aording to this prior distribution, we think (prior to olleting any data) that µ = 6 is most likely (meaning we think most students get an average of 6 hours of sleep per night) but we think that any value of µ (6 3, ) = (3, 9) is feasible (see how we are using a distribution to quantify what we know about µ?). For Step #2, we go and ollet data then relate that data to our parameter. Continuing with our sleep example, say we sample 00 students and, for eah student, we write down the number of hours of sleep per night they got during finals week. Beause µ is the population mean of number of hours that all students got during finals week, we know that Y i N (µ, σ 2 ) where Y i is the number of hours that person i slept during finals week, µ is the population mean and σ is the population standard deviation x Figure : A distribution for a random variable x. The shaded area represents the area x [, ] and, beause this is a normal urve, we know Pr(x [, ]) = 0.68 by the rule. 2

3 Finally Step #3 is the most mathematial where we update our prior distribution to reflet the knowledge gained from our data. This updated distribution is alled the posterior distribution (beause it s alulated post (i.e. after) data olletion). This step is where the Reverend Thomas Bayes (702-76) omes into play (and where Bayesians get their namesake). Reverend Bayes showed a really simple theorem (now alled Bayes theorem ) that an be used to update a prior distribution to a posterior distribution via the following formula: post(parameter(s)) = Pr(Data parameter(s)) Prior(parameter(s)). () where is some number that ensures that the area under the posterior distribution is equal to. Intuitively, all Bayes theorem says is to multiply the probability of all of our data times our prior and figure out what the posterior distribution is. Lets return to our sleep example where we now have data Y,..., Y 00 whih tell us how many hours eah of the 00 students in our sample slept. Beause we assume Y i N (µ, σ 2 ) we an write, Pr(Y i µ, σ 2 ) = exp } 2πσ 2 2σ 2 (Y i µ) 2 (2) where the funtion above is just the height of the normal urve. Now to get the probability of ALL of our data Y,..., Y 00 we assume independene and alulate Pr(Y,..., Y 00 µ, σ 2 ) = 00 i= exp } 2πσ 2 2σ 2 (Y i µ) 2. (3) The last thing we need to do is to then multiply the above by our prior distribution for µ so that we get post(µ) exp (µ m)2 2πs 2 2s2 } 00 i= exp } 2πσ 2 2σ 2 (Y i µ) 2 where, remember, we hose m = 6 and s = to orrespond to our prior knowledge of µ. Through a longbit of algebra (whih you get to do below :) we an show that post(µ) is atually a normal distribution but with a different mean and standard deviation than we started with in our prior. Beause I am going to ask you go through Steps -3 for various problems, let me first walk you through two examples so you an see how this Bayesian inferene thing works. 3. A Simple First Example (You may reognize a similar problem from the primer on maximum likelihood). Suppose we have two six sided die in a blak box that are idential in all ways exept their probabilities. The parameter we are interested in is whih die do I have in my hand so we want to make Bayesian inferene for this quantity. Step : Find Your Prior. We know that if we reah into the box we are just as likely to grab die # as die #2 so lets set our prior probabilities as Pr(Die#) = Pr(Die#2) = 0.5 whih represents our knowledge that either die is equally likely. Step 2: Relate Data to Parameter and Collet Data. Suppose the probabilities of eah roll of the die is assoiated with eah die as follows: Outome Die # /6 /6 /6 /6 /6 /6 Die #2 /2 2/2 2/2 4/2 /2 2/2 3 (4)

4 Now, we roll the 5 times and get outomes Y = 3, Y 2 = 2, Y 3 = 4, Y 4 = 4 and Y 5 = 3 where Y i is the outome from the i th roll of the die. Step 3: Update your prior to the posterior. We do this last step via Bayesian theorem. Via Bayes theorem we have, post(die #) Pr(Data Die #) prior(die#) (5) = ( ) ( ) ( ) ( ) ( ) ( ) (6) = (7) Aording to Bayes theorem, the posterior probability that we have Die # is So, what is?. Well, is just a number that makes the posterior probabilities sum up to. That is, we need to have post(die#) + post(die#2) =. So, lets figure out what post(die#2) is proportional to using Bayes theorem: post(die #2) = Pr(Data Die #2) prior(die#) (8) = ( ) ( ) ( ) ( ) ( ) ( ) (9) = (0) So, to make our posterior distribution sum to we solve for by doing = whih (solving for ) gives = and we get post(die#) = = 0.2 () post(die#2) = = 0.8. (2) The posterior distribution in Equations () and (2) are our Bayesian inferene answer. That is, we are pretty sure (about 80% sure to be preise) that we are holding Die #2 and only 20% sure we are holding Die #. If we had to pik, we d pik that we are holding Die #2 beause we are more sure of that die (in terms of the posterior distribution). 3.2 A Real Example Let s look at a real example to get a better feel for how Bayesian inferene works. Suppose we are interested in figuring out what perentage of BYU students think that BYU should allow men to have beards. Step : Choose our prior distribution. Let p represent the true perentage of BYU students who think that BYU should allow men to have beards. What do we know about p? Well, first we know that p (sine it is a proportion) has to be between 0 and. So, for our prior, we are going to have to find a distribution where the values are between 0 and. One suh distribution is the beta-distribution whih has the following probability distribution funtion: Γ(a+b) Γ(a)Γ(b) dbeta(x a, b) = xa ( x) b if x (0, ) (3) 0 otherwise. 4

5 The beta distribution looks ugly but we really just need to pik 2 numbers (the a and the b) to hoose our prior distribution. But, how do we pik these? Well, from Wikipedia (or Stat 340) we know that the mean of a beta-distributed random variable is m = a/(a + b) whih means that a = (mb)/( m) where m is the mean. We an hoose a mean m by hoosing what we think p is and then just selet a b that gives us a distribution that roughly mathes our prior beliefs (i.e. guess and hek). I think more than /2 of BYU students would favor allowing beards so I am going to set m = 0.65 to reflet this belief (you may pik a different number but that is what I am going with). Figure 2 shows urves for 5 different values of b with fixing m = 0.65 (to draw these I just piked a b value, alulated a = mb/( m) then plotted using the R ommand dbeta(x,shape=a,shape2=b)). From, these urves I like the b = 5 urve (whih gives me a = ( 0.65) = 9.29) beause, while I would guess that p = 0.65 I am really not sure so I want enough variane to allow for me to be wrong. So, my prior is Beta(a = 9.29, b = 5). 6 4 fator(b) 2 dbeta p Figure 2: Five possible hoies for my prior distribution for p in the beard example. Step 2: Link data to our parameters and ollet the data. The dataset Beards.txt gives the results of asking 87 BYU students if they think beards should be allowed. The data is oded as Y i = 0 if they DON T think beards should be allowed and a Y i = if they do. As a distribution for the data, we need a distribution that models a 0/ outome so we are going to use the Bernoulli distribution. That is, we will assume that, Pr(Y i ) = p Y i ( p) Y i (4) where Y i an either be a zero or a. This distribution links our parameter p to our data Y i. Step 3: Update your prior to the posterior. Following Bayes rule we have, post(p) (Pr(Data parameter(s))) Prior(parameter(s)) (5) ( = n ) p Y i ( p) Y Γ(a + b) i Γ(a)Γ(b) pa ( p) b (6) i= where we have a = 9.29 and b = 5 from our prior distribution. Beause Γ(a + b)/γ(a)γ(b) is just a onstant with respet to p we an absorb this into the onstant to make it easier. Now, lets do some 5

6 algebra, ( post(p) = n ) p Y i ( p) Y i p a ( p) b (7) i= = p n i= Y i ( p) n i= ( Y i) p a ( p) b (8) = pnyes ( p) n nyes p a ( p) b (9) = pnyes+a ( p) n nyes+b (20) = pa ( p) b (2) where I just labeled n yes = n i= Y i, a = n yes + a and b = n n yes + b. At this point notie that Equation (2) looks a lot like the beta distribution in Equation (3) exept we have a and b and we are missing the Γ( ) funtions out in front. Beause post(p) is a distribution, we need to find so that 0 pa ( p) b =. The hard way to find here is to notie from the above Equation that = 0 pa ( p) b whih we ould try to integrate (but I don t want to). The easier way to find is to ompare Equation (2) with Equation (3) and notie that we need to multiply Equation (2) by Γ(a + b )/Γ(a )Γ(b ) so that it beomes a probability distribution and integrates to. In other words, we an just let = Γ(a + b )/Γ(a )Γ(b ). Notie in this example that my prior was a beta distribution and the posterior is a beta distribution. This is a property alled onjugate prior or onjugay - namely, when the prior and posterior are the same named distribution (even if they have different numbers they have the same name). Lets reap. Aording to our math in Equations (7)-(2), we know that our posterior distribution is Beta(a, b ). So, Figure 3 shows this urve along with the prior. While the posterior distribution shown in Figure 3 is our Bayesian inferene, it is ommon to summarize the distribution using numerial quantities. For example, people often report the posterior mean (whih in this ase is a /(a + b )), the posterior median or a 95% redible interval defined as the and quantiles of the beta distribution (in R speak, the 95% redible interval is qbeta((0.025,0.975),shape=a,shape2=b ). 4 Problems For the four following problems, perform the three steps of Bayesian inferene. One you have found the posterior distribution, plot it along with your prior distribution and report the posterior mean, median, mode and a 95% redible interval.. Barry Bonds was one of the most ontroversial figures in major league baseball. On the one hand, he hit the most home runs by any baseball player. On the other, he used performane enhaning drugs. Here, we seek to estimate the probability that Barry Bonds hits a home run on any given at-bat. Let p (0, ) represent the probability of a home run. The dataset BarryBonds.txt ontains data from Barry s 200 season by game. That is, in game i Barry had n i at bats (the seond olumn) and hit y i home runs. Assume that the data (n i, y i )} follow a binomial distribution with suess probability p. 6

7 20 dbeta 0 Dist prior post p Figure 3: The prior vs. posterior distribution for the beard example. 2. Muh of soiety today runs on high performane omputing. So, when these super omputers go down it an ause a lot of problems. Let λ > 0 represent the mean number of failures per month of the Los Alamos National Laboratory superomputer and λ is the parameter of interest. The dataset ComputeFails.rtf ontains the number of failures of the superomputer and, beause it is ount data, an be modeled using a Poisson distribution with unknown mean λ. As a prior distribution, beause λ > 0 we need a distribution that keeps this onstraint so we will use the Gamma distribution as our prior. 3. Let µ (, ) be the mean final exam sore of all 2 students and is the parameter we are interested in. For this problem we will use the normal distribution as a prior for µ. The dataset ExamSores.txt ontains a random sample of student final exam sores whih will be assume to be well approximated by a N (µ, 9 2 ) distribution. 4. Still onsidering the exam sore problem from above, assume we know µ = 87 but we don t know the variane σ 2 (hint: use the inverse-gamma distribution here as the prior for σ 2 with a N (87, σ 2 ) distribution for the data). 7