Lecture 1 Bayesian inference

Size: px

Start display at page:

Download "Lecture 1 Bayesian inference"

Doreen Sanders
5 years ago
Views:

1 Lecture 1 Bayesian inference olivier.francois@imag.fr April 2011

2 Outline of Lecture 1 Principles of Bayesian inference Classical inference problems (frequency, mean, variance) Basic simulation algorithms

3 What is Bayesian data analysis? Model building. Build a joint distribution for both observable quantities (data) and non-observable quantities (parameters). Parameter inference. Compute the conditional distributions of the non-observable quantities given the data. Model criticism and improvement. Evaluate the t of the model to the data and check their predictions.

4 Model denition. Parameter θ = (θ 1,..., θ J ), J 1. Data y = (y 1,..., y n ), n 1. A model is a joint distribution p(y, θ) = p(y θ)p(θ) p(θ) is the prior distribution. p(y θ) is the likelihood or sampling distribution.

5 Inference. Use the Bayes formula to compute the posterior distribution p(θ y) = p(y θ)p(θ) p(y) where p(y) = p(y θ)p(θ)dθ is the marginal distribution. The marginal distribution is usually a highly dimensional impossible to compute integral, and we write p(θ y) p(y θ)p(θ).

6 Prediction. The posterior predictive distribution is p(y rep y) = p(y rep θ)p(θ y)dθ. Models are wrong, and the posterior predictive distribution can be used to evaluate aspects of the model that do not t to the data (model checking).

7 Examples of application (in this course) Bayesian clustering: How many groups in the data? What are the within-group means and variances? For a given individual, what is the assignment probability? Population genetics: For an individual genome, what fraction of DNA can be assigned to putative source (ancestral) populations?

8 Mixture models Histogram of y Density y

9 Example 1: Inferring allele frequencies Natural populations are of nite size, N. New genetic variants can arise from mutation or migration Genes frequencies at a bi-allelic locus (ancestral/derived allele) can uctuate #{carriers of the derived allele} beta(α, β) N where the beta distribution is beta(x, α, β) = Γ (α + β) Γ (α)γ (β) x α 1 (1 x) β 1, x (0, 1) and α, β > 0 depend on mutation or migration rates.

10 Beta distribution α = β = 1/2 α = β = 1 α = 2 β = x x x Expectation and mode of the beta distribution E[X ] = α α + β Mode(X ) = α 1 α + β 2

11 Model Prior distribution on the allele frequency: θ beta(1,1) (uniform). Data: We observe the derived allele y = 9 times in a sample of size n = 20 genes (frequency =.45) Likelihood p(y θ) = binom(n, θ)(y) θ y (1 θ) n y Posterior distribution (Exercise) p(θ y) = beta(y + 1, n + 1 y)(θ)

12 Remarks Point estimate (conditional mean) dierent from the maximum likelihood estimate E[θ y] = y + 1 n + 2 y n, as n Credible interval I so that Pr(θ I y) =.95 (R command quantile) I = (0.25, 0.65) Not a condence interval!

13 Joint distribution Y.sim theta

14 Computing the posterior distribution from simulations Rejection algorithm Repeat theta <- unif(0,1) y.s <- binom(n,theta) Until (y.s == y) return(theta) It generates samples from the posterior distribution p(θ y) (Exercise).

15 R scripts Rejection in R (sample of random size) y = 9 ; n = 20 theta <- runif(10000) y.s <- rbinom(10000, n, theta) theta.post <- theta[ y.s == y ] Exercise: Compute a 95% credible interval for θ and a histogram of the posterior predictive distribution given y.

16 Is the rejection algorithm ecient? Posterior distribution posterior predictive distribution Frequency Density u y.rep The acceptance rate is only 4.5%. It leaves room for improvement (Lecture 2).

17 Gaussian model: θ = m (σ 2 known) Case 1: One-dimensional data: y R Prior distribution p(θ) exp Sampling distribution p(y θ) exp Posterior distribution (Exercise) ( 1 ) (θ 2σ0 2 m 0 ) 2, β 0 = 1 σ0 2 ( 1 ) (y θ)2, β = 1 2σ2 σ 2 θ y N(m 1, σ 2 1) with 1/σ 2 1 = β 1 = β 0 + β, and m 1 = (β 0 m 0 + βy)/β 1.

18 Gaussian model: θ = m (σ 2 known) Non-informative prior distribution p(θ) 1, β 0 0 (σ 2 0 = ). Posterior distribution θ y N(y, σ 2 ) Exercise: Posterior predictive distribution ỹ y N(m 1, σ 2 + σ1) 2 = N(m 1, 2σ 2 )

19 Gaussian model: θ = m (σ 2 known) Case 2: n data, y = (y 1,..., y n ) Sampling distribution p(y 1,..., y n θ) ȳ = n i=1 y i/n is sucient n ( exp 1 ) 2σ 2 (y i θ) 2, i=1 p(θ y) = p(θ ȳ) Posterior distribution (Uninformative prior) θ y N(ȳ, σ 2 /n).

20 Gaussian model: θ = σ 2 (m known) χ 2 n distribution p(x) x n/2 1 e x/2, x > 0 Invχ 2 (ν, s 2 ) distribution: X = νs2 χ 2 ν (Exercise) p(x) 1 x ν/2+1 e ν 2 s 2x, x > 0.

21 Gaussian model: θ = σ 2 (m known) Prior distribution (not a density) p(θ) 1 θ, p(log(θ)) 1. Sampling distribution where p(y 1,..., y n θ) 1 ( θ n/2 exp n ) 2θ s2 n s 2 n = 1 n n (y i m) 2 i=1 Posterior distribution (Exercise) σ 2 y Invχ 2 (n, s 2 n)

22 Joint inference θ = (m, σ 2 ) Prior distribution (not a density) p(m, σ 2 ) 1 σ 2. Posterior distribution ( p(m, σ 2 1 y) exp 1 ) (σ 2 ) n/2+1 2σ 2 ((n 1)s2 n 1 + n(ȳ m) 2 ) where the unbiased empirical variance is s 2 n 1 = 1 n 1 n (y i ȳ) 2 i=1 The (marginal) posterior distribution of σ 2 is (exercise) σ 2 y Invχ 2 (n 1, s 2 n 1)

23 Multi-dimensional parameters: A basic algorithm θ = (θ 1, θ 2 ) 1. Simulate θ 1 from the marginal distribution p(θ 1 ) 2. Given θ 1, simulate θ 2 from the conditional distribution p(θ 2 θ 1 ).

24 Simulation θ = (m, σ 2 ) 1. σ 2 y (n 1)var(y)/χ 2 n 1 2. m σ 2, y N(mean(y), σ 2 /n) # simulated data n = 20; y = rnorm(n) # Posterior distribution sampling sigma.2 = (n-1)*var(y)/rchisq(10000, n-1) m = rnorm(10000, mean(y), sd = sqrt(sigma.2/n))

25 Posterior distribution (Gaussian model) Histogram of m Histogram of sigma.2 Density Density m sigma.2

26 Model checking Our data are perhaps not from a Gaussian model # Example y = rcauchy(n) sigma.2 = (n-1)*var(y)/rchisq(10000, n-1) m = rnorm(10000, mean(y), sd = sqrt(sigma.2/n)) Use a test statistic (skewness) post.pred = NULL for (i in 1:1000) { ind = sample(10000, 1) post.pred[i] = skewness(rnorm(20, m[ind], sqrt(sigma.2[ind]))) } hist(post.pred) skewness(y)

27 Take-home messages Bayesian inference is about computing the conditional distribution of a parameter given the data. This can be achieved by using computational Monte Carlo methods More to come in lecture 2.

28 Exercises Ex1. Find the posterior distribution in the beta-binomial model (answer: beta(y + α, n + β y)). Ex2. Prove the rejection algorithm. Ex3. Compute the 95% credible interval for θ and the posterior predictive distribution given y from the rejection algorithm Ex4. Simulate from the posterior distribution in the Gaussian model (two parameters). Use your own statistic for model checking. Ex5. Run inference for the sepal length in data(iris)

29 Bibliography and resources Gelman A, Carlin JB, Stern HS, Rubin DB (2004) Bayesian Data Analysis 2nd ed. Chapman & Hall, New-York. E. Paradis (2005) R pour les débutants. Univ. Montpellier II. R website:

Principles of Bayesian Inference

Principles of Bayesian Inference Sudipto Banerjee University of Minnesota July 20th, 2008 1 Bayesian Principles Classical statistics: model parameters are fixed and unknown. A Bayesian thinks of parameters