POSTERIOR PROPRIETY IN SOME HIERARCHICAL EXPONENTIAL FAMILY MODELS

Size: px

Start display at page:

Download "POSTERIOR PROPRIETY IN SOME HIERARCHICAL EXPONENTIAL FAMILY MODELS"

Gwendolyn Ellis
5 years ago
Views:

1 POSTERIOR PROPRIETY IN SOME HIERARCHICAL EXPONENTIAL FAMILY MODELS EDWARD I. GEORGE and ZUOSHUN ZHANG The University of Texas at Austin and Quintiles Inc. June 2 SUMMARY For Bayesian analysis of hierarchical models, it may be of interest to try and use improper priors to represent ignorance. In this article, we consider hierarchical exponential family models and obtain conditions on prior tail behavior that guarantee posterior propriety. In particular, we study three special cases, the Poisson-Gamma, the Binomial-Beta, and the Multinomial-Dirichlet models. Interestingly, in all three cases, flat priors produce improper posteriors. Although some data sets can yield proper posteriors under improper priors with the Poisson-Gamma model, we show that this cannot happen with the Binomial-Beta and Multinomial-Dirichlet models. Keywords: Hierarchical Bayes Models, Improper Priors, Poisson-Gamma, Binomial-Beta, Multinomial- Dirichlet. INTRODUCTION Hierarchical models provide a useful approach to combining data from several similar populations. A typical setup for such models consists of pairs (x,θ ),...,(x p,θ p )wherex i is the observed data from population i, andθ i is a parameter which identifies F (x i θ i ), the distribution of x i given Edward I. George holds the Ed and Molly Smith Chair and is Professor of Statistics, Department of MSIS, University of Texas, Austin, TX , egeorge@mail.utexas.edu. Zuoshun Zhang is a Biostatistician at Quintiles, Inc., 8 Rockville Pike, Suite 3, Rockville, MD , zzhang@qrrg.quintiles.com.

2 θ i. A hierarchical model is then obtained by using another model to describe the variation across the parameters θ,...,θ p. Such hierarchical models enable a statistical analysis that simultaneously uses all the data x,...,x p to make inference about all the parameters θ,...,θ p. In this paper, we focus on the following conditionally independent, three stage hierarchical model (Lindley and Smith 972, Kass and Steffey 989, George, Markov and Smith 993, 994):. Conditionally on θ,...,θ p,eachx i has distribution F (x i θ i ), and the data x,...,x p are independent of each other and are independent of λ. 2. Conditionally on λ, the parameters θ,...,θ p are iid with density π(θ λ). 3. The hyperparameter λ has a density π(λ), which may or may not be proper. In particular, we consider setups for which F (x θ) belongs to an exponential family (Brown 986) df (x θ) =exp[x θ ψ(θ) dν(x), for θ Θ, where dν(x) is a fixed σ-finite measure on the Borel sets of R k, Θ is a subset of the natural parameter space N = {θ R k : exp(x θ) dν(x) < }, and ψ(θ) =ln exp(x θ) dν(x) isthe cumulant generating function. For such F (x θ), we consider conjugate priors on θ (Diaconis and Ylvisaker 979), namely π(θ x,n )=exp[x θ n ψ(θ) ϕ(x,n )I Θ (θ), where I Θ (θ) is the indicator function and ϕ(x,n )=ln Θ exp[x θ n ψ(θ) dθ, with the range of (x,n ) properly chosen to make π(θ x,n ) a proper density. Finally, the hyperparameters (x,n ) are given some prior π(x,n ), which may be proper or improper. For any of these hierarchical exponential family setups, the only remaining specification issue is the choice of the hyperparameter prior π(x,n ). In many problems, there will be little available prior information about the hyperparameters, and it will be desirable to use a prior which is noninformative in some sense. Because improper priors are often considered for this purpose, it is of interest to know whether the resulting posterior π(θ,...,θ p,x,n x,...,x p ) is proper. This issue has increased in importance with the advent of Markov chain Monte Carlo posterior computation, which relies on such propriety to guarantee overall convergence of the simulated Markov chain, see George and Casella (992), Hobert and Casella (996, 997) and Casella (996). Berger and Strawderman (996) and Hobert and Casella (996) contain results for proper posteriors related to normal models. Natarajan and McCulloch (995) contains posterior propriety results for a class of mixed models for Binomial data. In this article, we obtain posterior propriety results for some special cases of the above hierarchical exponential family model, namely the Poisson-Gamma, the Binomial-Beta and Multinomial- Dirichlet hierarchical models. These results are conditions on the hyperparameter prior tail behavior 2

3 that guarantee posterior propriety. For the Poisson-Gamma hierarchical model, these conditions allow for some improper priors to guarantee posterior propriety. However, for the Binomial-Beta and Multinomial-Dirichlet hierarchical models, it turns out that no improper prior can guarantee a proper posterior. In particular, under the traditional parameterizations, flat priors can never guarantee posterior propriety in any of our three hierarchical setups. 2 THE POISSON-GAMMA MODEL We first consider the Poisson-Gamma model, where the data have Poisson distributions, and the conjugate prior on the parameters is the Gamma distribution. The three stages for the conditionally independent hierarchical model are then the following. For i =,...,p, θ f(x i θ i )= e θi xi i, x i =,,..., x i! π(θ i α, ) = α Γ(α) θα i e θi, θ i >, (α, ) π(α, ), α >, > where π(α, ) is a (possibly improper) prior on the hyperparameters α and. The joint posterior for all the parameters is Integrating out θ,...,θ p yields π(θ,...,θ p,α, x,...,x p ) [ ( ) α p p ( θ xi+α i exp ( +) Γ(α) i= ) p θ i π(α, ). i= π(α, x,...,x p ) [ ( ) α p p Γ(x i + α) Γ(α) ( +) xi+α π(α, ) i= [ αp p = ( +) αp+s (α + x i ) (α +)α π(α, ) i= [ = ( +) s c(i)α i e pα log(+ ) π(α, ), where s = p i= x i, k is the number of nonzero entries in the data vector (x,...,x p ), and s c(i)αi := p i= (α+x i ) (α+)α. We use the convention that the above polynomial is identically when k = s =. Hence we have k p, k s, and c(i) >,i= k,...,s. Our aim is to find conditions on the hyperparameter prior π(α, ) such that the posterior is proper, which is equivalent to m(x,...,x p )= π(θ,...,θ p,α, x,...,x p ) π(α, x,...,x p ) dα d <, 3

4 for any nonnegative integers x,,...,x p. When π(α, ) is proper, the posterior is always proper. Hence we concentrate on improper priors. We impose the following boundary conditions: (C) π(α, ) =O(α a b ) as α, (C2) π(α, ) =O(α a b ) as α, (C3) π(α, ) =O(α a b ) as α, (C4) π(α, ) =O(α a b ) as α, Our problem is now to determine the values of a,b,a,b such that the posterior is proper. Let us state our results and then carry out the proofs. Theorem 2. Let the prior π(α, ) satisfy the conditions (C) (C4) and let the data x,...,x p be fixed. Then the posterior π(θ,...,θ p,α, x,...,x p ) is proper if and only if one of the following two groups of conditions hold: (I) a <k+,b <,b + s>,a <k+,a + b > 2; (II) a <k+,b <,b + s>,a k +. Corollary 2.2 The posterior π(θ,...,θ p,α, x,...,x p ) is proper for all values of x,...,x p if and only if a <,b <,b >,a + b > 2. Corollary 2.3 Let π (α) =π 2 () =be the flat priors. Then the posterior π(θ,...,θ p,α, x,...,x p ) is improper for all values of x,...,x p. Proof: First we partition the integral over (, ) (, ) four parts, [ π(α, x,...,x p ) dα d = A + A 2 + A 3 + A 4. We proceed to derive necessary and sufficient conditions for each of A,A 2,A 3 and A 4 to be finite. A b ( +) s p log(+ ) c(i)α i a e pα log(+ ) dα d so that A < if and only if b < anda <k+. A 2 b ( +) s p log(+ ) b ( +) s [p log( + )i a+ c(i)ui a e u du d c(i)α i a e pα log(+ ) dα d b ( +) s [p log( + )i a + c(i)ui a e u du d 4

5 so that A 2 < if and only if b <. A 3 b ( +) s b +s c(i)α i a e pα log(+ ) dα d c(i)α i a dα d so that A 3 < if and only if b + s>, and a <k+. A 4 b ( +) s p log(+ ) p c(i)α i a e pα log(+ ) dα d c(i)u i a e u s+b [p log( + du d )i a + c(i)u i a e u du d. s+b +a i To establish conditions for A 4 to be finite, we consider three subcases. Subcase : When a <k+, if and only if b + a > 2. Subcase 2: When a = k +, A 4 Γ(i a +) d <, s+b +a i if and only if s + b >. A 4 p c(k)log() d <, s+b c(i) ui k e u du d s+b +k i Subcase 3: When a >k+, if and only if s + b >. A 4 p p u i a e u c(i) du d s+b +a + i c(k)u k a e u d du s+b +a k d <, s+b Combining the conditions of all the cases above, shows that m(x,...,x p )=A +A 2 +A 3 +A 4 < if and only if either of the following two groups of conditions hold: (I) a <k+,b <,b + s>,a <k+,a + b > 2; 5

6 (II) a <k+,b <,b + s>,a k +. which are exactly the conditions of the theorem. Combining them and considering all possible values of k and s from the data yields the two corollaries. 3 THE BETA-BINOMIAL MODEL Next, we turn to the Binomial-Beta model, where the data have Binomial distributions, and the conjugate prior on the parameters is the Beta distribution. The three stages for the conditionally independent hierarchical model are then the following. For i =,...,p, f(x i θ i )= n i θ xi i ( θ i) ni xi, x i =,,...,n i, x i π(θ i α, ) = B(α, ) θα i ( θ i ), (α, ) π(α, ), α >, >, where π(α, ) is a (possibly improper) prior on the hyperparameters α and. The joint posterior for all the parameters is Integrating out θ,...,θ p yields π(θ,...,θ p,α, x,...,x p ) p n i B(α, ) p θ α+xi i ( θ i ) +ni xi π(α, ). i= x i π(α, x,...,x p ) [ p B(α + x i,+ n i x i ) π(α, ) B(α, ) i= [ p Γ(α + x i )Γ( + n i x i )Γ(α + ) = π(α, ) Γ(α)Γ()Γ(n i= i + α + ) [ p (α + x i ) (α +)α( + n i x i ) ( +) = π(α, ) (n i= i + α + ) (α + +)(α + ) := g(α, )π(α, ). Our aim is to find conditions on the hyperparameter prior π(α, ) such that the posterior π(θ,...,θ p,α, x,...,x p ) is a proper distribution. For any number c such that <c<, let S c = {(α, ) :<c<α</c} be a cone-like region in the first quadrant of the (α, ) plane. Let us state our results and then carry out the proofs. Theorem 3. (A) If the posterior π(θ,...,θ p,α, x,...,x p ) is proper then S c π(α, ) dα d < 6

7 for any S c with <c<. (B) For any improper prior π(α, ), there exist values of x,...,x p for which the posterior π(θ,...,θ p,α, x,...,x p ) is improper. Corollary 3.2 Let π(α, ) =be the flat prior. Then the posterior π(θ,...,θ p,α, x,...,x p ) is improper for all values of x,...,x p. Proof: For any set S c,itcanbeeasilyshownthatthereexistconstants<a<bsuch that a g(α, ) b, where(α, ) S c. Hence a proper posterior implies S c π(α, ) dα d <, the conclusion of (A). Next we prove (B). First we pick x = = x p =. LetR = {(α, ) :<α } be the upper half of the first quadrant of the (α, ) plane. The function g(α, ) can be expressed as g(α, ) = p i= ( + n i ) ( +) (α + + n i ) (α + +)(α + ), from which we can deduce that /2 s g(α, ), for (α, ) R,where s = p i= n i. It follows that π(α, x,...,x p ) dα d < R if and only if R π(α, ) dα d <. Similarly, we may pick x = n,,x p = n p and R 2 = {(α, ) :α >}, inwhichcase, π(α, x,...,x p ) dα d < R 2 if and only if R 2 π(α, ) dα d <. Because an improper prior cannot yield finite integrals on both R and R 2, it follows that we can always find values of x,...,x p such that the posterior is improper. 4 THE MULTINOMIAL-DIRICHLET MODEL Similar results as the last section hold for its natural generalization, the Multinomial-Dirichlet model (Good 983, Leonard 977). Here, the data have Multinomial distributions, and the conjugate prior on the parameters is a Dirichlet distribution. The three stages for the conditionally independent hierarchical model are then the following. For i =,...,p, f(x (i) θ (i) )= n i! t j= x(i) j! t (θ (i) j j= )x(i) j, π(θ (i) ) = Γ( t j= j) t t j= Γ( (θ (i) j j) )j, j= π() =π(,..., t ), j >,j =,...,t 7

8 where x (i) j are non-negative integers, t j= x(i) j = n i,<θ (i) j <, t j= θ(i) j =,x (i) =(x (i),...,x(i) t ), θ (i) =(θ (i),...,θ(i) t )and =(,..., t ). Calculating the posterior as p π(θ (),...,θ (p), x (),...,x (p) ) [ f(x (i) θ (i) )π(θ (i) )π(), i= and defining S c = {(,..., t ):<c j < k < j /c, j, k =,...,t} for any c such that <c<, the proofs of the following results are very similar to those in the previous section. Theorem 4. (A) If the posterior π(θ (),...,θ (p), x (),...,x (p) ) is proper, then S c π(,..., t ) d d t <, for any possible S c with <c<. (B) For any improper prior π(,..., t ), there exist values of x,...,x p for which the posterior π(,..., t ) is improper. Corollary 4.2 Let π() = be the flat prior. Then the posterior π(θ (),...,θ (p), x (),...,x p ) is improper for all values of x,...,x p. ACKNOWLEDGEMENT The authors are grateful to George Casella and James Hobert for helpful suggestions. This work was supported by NSF grants DMS and Texas ARP grants , REFERENCES Berger, J.O. and Strawderman W.E. (996). Choice of hierarchical priors: admissibility in estimation of normal means. Ann. Statist. 24, Brown, L.D. (986). Fundamentals of statistical exponential families. IMS Lecture Notes Series (ed. S. Gupta). Hayward, California. Casella, G. (996). Statistical Inference and Monte Carlo Algorithms (with discussion) Test. 5, Casella, G. and George, E.I. (992). Explaining the Gibbs sampler. The American Statistician. 46,

9 Diaconis, P. and Ylvisaker D. (979). Conjugate priors for exponential families. Ann. Statist. 7, George, E.I., Makov, U.E., and Smith, A.F.M. (993). Conjugate likelihood distributions. Scand. J. Statist. 2, George, E.I., Makov, U.E., and Smith, A.F.M. (994). Fully Bayesian hierarchical analysis for exponential families via Monte Carlo computation. Aspects of Uncertainty A Tribute to D.V.Lindley (P.R.Freeman, A.F.M.Smith, eds). pp Chichester: John Wiley and Sons. Good, I.J. (983). The robustness of a hierarchical model for Multinomials and contingency tables. Statistical Inference, Data Analysis, and Robustness (G.Box,T.Leonard,C.-F.Wu,eds). pp 9-2. New York: Academic Press. Hobert, J.P. and Casella, G. (996). The effect of improper priors on Gibbs sampling in hierarchical linear mixed models. J. Amer. Statist. Assoc. 9, Hobert, J.P. and Casella, G. (998). Functional compatibility, Markov chains and Gibbs sampling with improper posteriors. J. Comp. Graph. Statist. 7, Leonard, T. (977). A Bayesian approach to some Multinomial estimation and pretesting problems. J. Amer. Statist. Assoc. 72, Lindley, D.V. and Smith, A.F.M. (972). Bayes estimates for the linear model (with discussion). J. R. Statist. Soc. B. 34, -4. Kass, R.E. and Steffey, D. (989). Approximate Bayesian inference in conditionally independent hierarchical models (parametric empirical Bayes). J. Amer. Statist. Assoc. 84, Natarajan, R. and McCulloch, C.E. (995). A note on the existence of the posterior distribution for a class of mixed models for Binomial responses. Biometrika. 82,

10. Exchangeability and hierarchical models Objective. Recommended reading

10. Exchangeability and hierarchical models Objective Introduce exchangeability and its relation to Bayesian hierarchical models. Show how to fit such models using fully and empirical Bayesian methods.