One Parameter Models

Transcription

1 One Parameter Models p. 1/2 One Parameter Models September 22, 2010 Reading: Hoff Chapter 3

2 One Parameter Models p. 2/2 Highest Posterior Density Regions Find Θ 1 α = {θ : p(θ Y ) h α } such that P (θ Θ 1 α Y ) = 1 α All points in Θ 1 α have a higher density than any point outside the regions. Often requires iterative solution: Find points such that p(θ Y ) > h Find probability of that set Adjust h until reach desired coverage may not have symmetric tail areas multimodal posterior may not have an interval

3 One Parameter Models p. 3/2 solve.hpd.beta Key ideas: use relative posterior by dividing posterior density by the density at the mode (range is 0 to 1) for a height h in (0,1) find points where p(θ Y ) = h on either side of the mode, using the uniroot function use pbeta function to caclulate area move h up or down so that area is closer to coverage 1 α

4 One Parameter Models p. 4/2 Exponential Example: Rate β The time between accidents modeled with an exponential distribution with a rate of β accidents per day. n = 10 observations: f(y β) = β exp( yβ) y > 0 L(β; y 1,..., y n ) = i β exp ( y i β) (Likelihood) = β n exp ( i y i β) Look at plot of L(β) for y i = 336.

5 One Parameter Models p. 5/2 Likelihood function l.exp = function(theta, y) { n = length(y) sumy = sum(y) l = thetaˆn*exp(-sumy*theta) return(l) } Vectorized: can be used to evaluate L(θ) at multiple values ot θ rather than using a loop beta = seq(.00001,.25, length=1000) l.exp(beta, y)

6 Plot of Likelihood Exponential Likelihood 0.0e e e e e e β One Parameter Models p. 6/2 L(β)

7 One Parameter Models p. 7/2 Conjugate Prior Distributions Recall that a family of distributions is conjugate for a sampling model, if for any prior in the class the posterior is also in the class If the uniform distribution is in the class, then that means that the posterior must be proportional to the Likelihood. Use the form of the likelihood to help identify the conjugate prior: L(β) β n exp( β y i )

8 One Parameter Models p. 8/2 Gamma Distribution Z Gamma(a, b) with mean a/b and variance a/b 2 with density f(z) = ba Γ(a) za 1 exp( zb) z > 0 In this parameterization b is a rate parameter. rgamma(n, shape=a, scale=s) # mean = shape*scale rgamma(n, shape=a, rate=r) # mean = shape/rate Mode is at (a 1)/b if a > 1 and at 0 if a 1

9 One Parameter Models p. 9/2 Examples of Gamma Distributions Density a = 1, r = 1/5 a = 2, r = 1/5 a = 2, r = 1/ x

10 One Parameter Models p. 10/2 Distribution of β Identify normalizing constant to make this a density for β or recognize the form of the distribution where p(β y 1,..., y n ) = cβ n exp( β y i ) c = 1/ 0 β n exp( β y i )dβ Looks like a Gamma(a = n + 1, b = y i ) normalized likelihood has same shape has likelihood same form that would be obtained using Bayes Theorem with a uniform prior

11 One Parameter Models p. 11/2 Uniform Prior Uniform prior p(β) = 1 in exponential example is not a proper distribution; although the posterior distribution is a proper distribution. Formal Bayes posterior distribution obtained as a limit of a proper Bayes procedure. Be very careful with improper prior distributions, they may not lead to proper posterior distributions!

12 One Parameter Models p. 12/2 Gamma Prior/Posterior Distributions Prior and Posterior distributions are in the same family p(β Y ) β G(a, b) (1) ba Γ(a) βa 1 e βb β n e β P yi (2) β a+n 1 e β(b+p y i ) (3) β Y G(a + n, b + y i ) (4) Uniform is a limiting case with a = 1 and b 0 default prior is p(β) 1/β a Gamma(0,0)

13 One Parameter Models p. 13/2 Posterior Quantities posterior distribution G(10, 336) posterior mode ˆβ = posterior mean E(β Y ) = 10/336 = Interval Estimates Probability Intervals or Credible Regions Highest Posterior Density region [0.012, 0.049] Equal Tail area [0.014, 0.051] After observing the data, we believe there is a 95% chance of [0.01, 0.05] accidents per day (or 1 to 5 accidents per every 100 days)

14 One Parameter Models p. 14/2 Highest Posterior Density Region 95% Highest Posterior Density [0.012, 0.05] Posterior Density P (.012 < β <.049 Y ) = θ

15 One Parameter Models p. 15/2 Equal Tail Area Intervals Easier alternative is to find points such that P (θ < θ l Y ) = α/2 P (θ > θ u Y ) = α/2 (θ l, θ u ) is a 1 α 100% Credible Interval (or Posterior Probability Interval) for θ. > qgamma(.025, 10, 336) [1] > qgamma(.975, 10, 336) [1]

16 One Parameter Models p. 16/2 Examples of Conjugate Families Data Distributions Exponential Poisson Binomial Normal (known variance) Normal (unknown mean/variance) Prior/Posterior Gamma Gamma Beta Normal Normal-Gamma Always available in exponential families

17 One Parameter Models p. 17/2 Exponential Family One parameter exponential family is expressed as p(y θ) = h(y)c(θ) exp(θt(y)) where θ is the natural parameter of the exponential family and t(y) is the sufficient statistic Likelihood: L(θ) c(θ) n exp(θ n i t(y i )/n) prior: p(θ) c(θ) n 0 exp(θ n 0 t 0 ) where n 0 may be interpreted as the number of prior observations and t 0 is the prior expected value of t(y )

18 One Parameter Models p. 18/2 Conjugate Prior & Posterior Likelihood: L(θ) c(θ) n exp(θ n i t(y i)/n) Prior: p(θ) c(θ) n 0 exp(θ n 0 t 0 ) Posterior: p(θ Y ) c(θ) n 0+n exp(θ(n 0 t 0 + n i t(y i)/n) subject to finite normalizing constant Updating of hyperparameters: n 0 n 0 + n t 0 t(y) P n i=1 t(y i) n n 0 t 0 + nt(y) n 0 + n

19 One Parameter Models p. 19/2 Exponential p(y β) = β exp( βy) = θ exp(θ( y)) natural parameter θ = β sufficient statistic ȳ conjugate prior p(θ) θ n 0 exp(θn 0 t 0 ) or Gamma(n 0, n 0 t 0 ) or Gamma(n 0, n 0 ȳ 0 ) Note: Exponential distribution is NOT the same as an exponential family

20 One Parameter Models p. 20/2 Binomial Rearrange Bernoulli (a Binomial with n = 1) to get in to exponential family form: p(y π) = π y (1 π) 1 y = = π y (1 π) 1 π [ { ( ) π exp log 1 π = c(θ) exp(θy) where θ = log(π/(1 π)) c(θ) = (1 + exp(θ)) 1 }] y (1 π) Natural parameter θ is the log-odds

21 One Parameter Models p. 21/2 Conjugate Prior p(θ) ( ) n0 1 exp(θ n 0 t 0 ) 1 + exp(θ) where t 0 is prior expected value for the probability that Y is 1 Normalizing constant? Implied Prior distribution for π Review change of variables