Bayesian Statistics. Debdeep Pati Florida State University. April 3, 2017

Transcription

1 Bayesian Statistics Debdeep Pati Florida State University April 3, 2017

2 Finite mixture model The finite mixture of normals can be equivalently expressed as y i N(µ Si ; τ 1 S i ), S i k π h δ h h=1 δ h = probability measure concentrated at the integer h, S i {1, 2,..., k} indexes the mixture component for subject i, i = 1,..., n. A prior on π = (π 1, π 2,..., π k ) and (µ h, τ h ), h = 1,..., k is given by π Dir(α 1, α 2,..., α k ) (µ h, τ h ) N(µ h ; µ 0, κτ 1 h )Ga(τ h; a τ ; b τ ), h = 1,..., k

3 Posterior Computation in finite mixture models Update S i from its multinomial conditional posterior with Pr(S i = h ) = π h N(y i ; µ h, τ 1 h ) k l=1 π hn(y i ; µ h, τ 1 h = 1,..., k ), h Let n h = #{S i = h, i = 1,..., n} and ȳ h = 1 n h i:s i =h y i and update (µ h, τ 1 h ) from its conditional posterior (µ h, τ 1 h ) = N(µ h, ˆµ h, ˆκ h τ 1 h )Ga(τ h, â τh, ˆb τh ) where ˆκ h = (κ 1 + n h ) 1, ˆµ h = ˆκ(κ 1 µ 0 + n h ȳ h ), â τh = a τ + n h ˆb τh = b τ + 1 { (y i ȳ h ) 2 + n } h (ȳ h µ 0 ) κn h i:s i =h Update π as (π ) = Dir(a 1 + n 1,..., a k + n k ). 2,

4 Some comments Gibbs sampler is trivial to implement Discarding a burn-in,monitor f (y) = k h=1 π hn(y; µ h, τ 1 for a large number of iterations & a dense grid of y values h ) Bayes estimate of f (y) under squared error loss averages the samples Can also obtain 95% pointwise intervals for unknown density

5 Choosing the Dirichlet hyperparameters The choice of hyperparameters in the mixture model can have an important impact Focus initially on the choice of (a 1,..., a k ) in the Dirichlet prior A common choice is a 1 = = a k = 1,which seems non-informative. However, this is actually a poor choice in many cases, as it favors assigning roughly equal weights to the different components Ideally, we could choose k as an upper bound and choose hyperparameters, which favor a small number of components with relatively large weights

6 Finite Approximation to Dirichlet Process Ishwaran and Zarepour (2002), Dirichlet prior sieves in finite normal mixtures, Statistica Sinica, 12, ) propose a finite approximation to the DP In particular, they propose letting π Dir(α/k,..., α/k) iid Assuming also that θ h = (µ h, τ h ) P 0, they show that lim k h=1 k π h δ θh DP(αP 0 ) In addition, the posterior for the density is L 1 consistent if log k/n 0.

7 Implications We can implement a finite mixture model analysis with a carefully chosen prior & sufficiently large k to obtain an accurate approximation to the DP mixture (DPM) model: f (y) = K(y; θ)dp(θ), P DP(αP 0 ) Here, K(y; θ) is a kernel parameterized by θ - e.g., K(y; θ) = N(y; µ, τ 1 ) with θ = (µ, τ 1 ) for normal mixtures P is now an unknown mixing measure Hence, we no longer use the DP as a prior directly for the distribution of the data but instead use it for the mixture distribution

8 Dirichlet Process Mixtures - comments The discreteness of the Dirichlet process is not a problem when it is used for a mixture distribution instead of directly for the data distribution In fact, in this setting the discreteness is appealing in leading to a simple representation of the mixture distribution that leads to clustering of the observations as a side effect Focusing on the finite approximation, P DP k (α, P 0 ), let f (y) = N(y; µ, τ)dp(µ, τ) = k h=1 π h N(y; µ h, τ 1 h ) This induces a prior on f.

9 Dirichlet Process Mixtures For density estimation,consider the DP mixture (DPM)model y i µ i, τ i N(µ i, τ 1 i ), θ i = (µ i, τ i ) P, P DP(αP 0 )( ) Not immediate clear how to conduct posterior computation One strategy relies on marginalizing out P to obtain ( ) α i 1 1 (θ i θ 1,..., θ i 1 ) P 0 + α i + 1 α + 1 δ θ j j=1 DP prediction rule or Polya urn scheme (Blackwell & MacQueen, 73)

10 Avoiding Marginalization By marginalizing out the RPM P, we give up the ability to conduct inferences on P By having approaches that avoid marginalization, we open the door to generalizations of DPMs Stick-breaking representation (Sethuraman, 94), θ i P = i.i.d i.i.d V h (1 V l )δ Θh, V h beta(1, α), θ h P 0 h=1 l<h

11 Samples from Dirichlet process with precision α

12 Implications of Stick-Breaking For small α, most of the probability is allocated to the first few components, favoring few latent classes Expected number of occupied components α log n Weights π h decrease stochastically towards near zero rapidly in the index h Suggests truncation approximation (Muliere & Tardella, 98), P = N V h (1 V l )δ Θh h=1 l<h with V N = 1 so that weights sum to one

13 Blocked Gibbs Sampler (Ishwaran & James, 01) 1. Update S i {1,..., N} by multinomial sampling with P(S i = h ) = π hn(y i ; Θ h ) N l=1 π ln(y i ; Θ l ), h = 1,..., N 2. Update stick-breaking weight V h, h = 1,..., N 1, from ( Beta 1 + n h, α + N l=h+1 n l ). 3. Update Θ h, h = 1,..., N, exactly as in the finite mixture model.

14 Comments on Blocked Gibbs N acts as an upper bound on the number of mixture components in the sample By choosing a large value, the approximation error should be small Possible to monitor this error during the MCMC Approximate inferences on functionals of P are possible Slice (Walker, 07) & retrospective sampling (Papaspiliopoulos & Roberts, 08) approaches avoid truncation - exact block Gibbs (Papaspiliopoulos, 08) combine these approaches

15 Choosing the DP precision parameter The DP precision parameter α plays a key role in controlling the prior on the number of clusters A number of strategies have been proposed in the literature - 1. Fix α at a small number to favor allocation to few clusters relative to the sample size - a commonly used default value is α = Assign a hyperprior (typically gamma) to α - refer to technical report by West (92) & recent article by Dorazio (09, JSPI, 139, ) 3. Estimate α via empirical Bayes (Liu 96; McAulliffe et al. 06)