Hierarchical priors and mixture models, with applications in regression and density estimation

Transcription

1 Hierarchical prirs and mixture mdels, with applicatins in regressin and density estimatin By Mike West, Peter Müller & Michael D Escbar In: Aspects f Uncertainty: A Tribute t D.V. Lindley (eds: P Freeman et al), 1994, p

2 Hierarchical prirs and mixture mdels, with applicatin in regressin and density estimatin MIKE WEST y, PETER M ULLER y and MICHAEL D ESCOBAR z 1Intrductin In his 1972 review f Bayesian statistics, Dennis Lindley identied as a success stry fr Bayesian ideas the advances made in prblems f many parameters and the grwth f what is nw referred t as Bayesian hierarchical mdelling (Lindley 1972 sectin 8). In that same mngraph, Lindley identied nn-parametrics as an area ntable fr lack f Bayesian prgress, bemaning the fact that nn-parametric statistics was a \subject abut which the Bayesian methd is embarrassingly silent" (Lindley 1972 sectin 12.2). Our purpse here is t develp and review recent wrk with mixture prirs that has substantially cntributed t bth hierarchical mdelling and nn-parametrics, the latter fcussed n prblems usually referred t as nn-parametric density estimatin. It is certainly the case that the wide applicatin f hierarchical mdels is ne f the majr success stries f mdern Bayesian statistics. In the early 1990s, we are experiencing tremendus grwth in serius applied hierarchical mdelling wrk, substantially driven by the invigrating breakthrughs in Bayesian cmputatin via Markv chain simulatin (Gelfand and Smith 1990 Gelfand, Hills, Racine-Pn and Smith 1990). Simultaneusly, these cmputatinal methds allw develpment and applicatin f data and prir mdels that signicantly extend the scpe f mre traditinal, mathematically tractable frms, prviding scpe fr clser representatin f real-wrld prblems, rutine sensitivity and rbustness analysis. Fr example, lng-suered cnstraints t cnjugate prir distributins in hierarchical mdels (Smith, in discussin f DuMuchel and Harris 1983 West 1984, 1985) can be relieved by the use f mixtures f cnjugate prirs with ease (Escbar and West 1992). Mixture prirs, especially Dirichlet mixtures (Antniak 1974 Escbar and West 1991), have als pened the way t serius Bayesian develpments in (s-called) nn-parametric mdelling and density estimatin, and smth regressin estimatin (Erkanli, Muller and West 1992). It is ur purpse in this paper t exhibit a general framewrk fr hierarchical linear mdelling and density estimatin, t shw hw psterir cmputatins via Markv chain simulatins can be rutinely applied, and t prvide illustratins in each cntext. Sectin 2 prvides a rather general theretical setting and summarises key features f multivariate data mdels with hierarchical mixture prirs. Sectin 3 discusses Markv chain simulatin methds in these mdels, with special emphasis n mdels centred arund traditinal nrmal structures. Sectin 4 cncerns an applicatin in hierarchical regressin, highlighting the use f mixture prirs fr rbustness and sensitivity analysis, and Sectin 5 develps an applicatin t multivariate density estimatin. y z Institute f Statistics and Decisin Sciences, Duke University, Durham, NC , USA Department f Statistics, Carnegie Melln University, Pittsburgh, PA 15213,USA

3 Hierarchical Prirs and Mixture Mdels February 9, Hierarchical Mdels with Mixture Prirs Cnsider a sample f n vectr bservatins y i in r dimensins, mdelled as arising frm a cllectin f distributins y i F i (:j i ) where i is a vectr f parameters specic t case i is a vectr f parameters cmmn t each case,f i is f specied frm and cmpletely knwn when i and are knwn, and the y i are cnditinally independent given F i the i and : Each F i is assumed t have a density functin f i (:j i ): A regressin example has f i (:j:) multivariate nrmal with mean X i i and variance matrix 2 i hereeach X i is a knwn regressin matrix, i = f i i g and is a cmmn scale parameter. Nn-linear/nrmal regressins are bvius alternative examples. Generally the interpretatin and dimensins f the i may dier, thugh ften they represent case specic realisatins f parameters with cmmn interpretatin acrss i wemake that assumptin here, s that the i enter int the f i (:j:) thrugh a cmmn functinal frm, and they have cmmn dimensin p say. Atypical hierarchical mdel assumes the case specic parameters i are cnditinally independently distributed accrding t sme prir distributin G(:j) here represents a cllectin f hyperparameters and a three stage hierarchical mdel is cmpleted by specifying an apprpriate hyperprir distributin fr : Dirichlet prcess mixture mdels are based n Dirichlet prcess prirs fr the primary parameters i : Such a mdel assumes that the prir distributin functin G itself is uncertain, drawn frm a Dirichlet prcess G D(G 0 ) (in standard ntatin, such asinantniak 1974). Here E(G) =G 0 is the base prir, and >0 the scalar precisin parameter. This is cnditinal n hyperparameters which will include the precisin and determine the prir expectatin G 0 : In this sectin we suppress dependence n bth and in the ntatin, fr clarity it shuld be brne in mind that the fllwing discussin is all cnditinal n bth and : The fllwing prperties and reinterpretatin f this mdel clearly elucidate the resulting structure (see Antniak 1974, and West 1990). (i) Any realisatin f n case specic parameters i generated frm G lie in a set f k n distinct values, dented by = f 1 ::: k g: (ii) The i are a randm sample frm G 0 (:): (iii) k n is drawn frm a prir distributin that is implicitly determined and depends nly n n and the precisin >0 it is Pissn like, thugh fatter tailed, with asympttic mean lg(1 + n=) frn!1(antniak 1974 West 1992). (iv) Given k the n values i are selected frm the set accrding t a unifrm multinmial distributin. Cnditinal n k intrduce indicatrs S i = j if i = j s that, given S i = j and y i F i (:j j ): The cnguratin S = fs 1 ::: S n g (West 1990 MacEachern 1992) determines a ne-way classicatin f the data Y = fy 1 ::: y n g int k distinct grups r clusters the n j =#fs i = jg bservatins in grup j share the cmmn parameter value j : Write I j fr the set f indices f bservatins in grup j I j = fi : S i = jg and let Y j = fy i : S i = jg be the crrespnding grup f bservatins. Then, as the j are a randm sample frm the base prir G 0 psterir analysis devlves t a cllectin f k independent analyses specically, the j are cnditinally independent with psterir densities p( j jy S k) p( j jy j S k) / Y i2ij 1 f i (y i j j ) A dg0 ( j ) (1) fr j =1 ::: k: 2

4 Hierarchical Prirs and Mixture Mdels February 9, 1994 A further feature f the Dirichlet structure f imprtance in psterir inference is the mechanism fr allcating bservatins t grups, summarised thrugh cnditinal distributins fr individual i given existing cnguratins f ther parameters. Fr i = 1 ::: n write (i) = f 1 ::: i;1 i+1 ::: n g and dente by S (i) the cnguratin f (i) int sme k (i) distinct values, with n (i) j taking cmmn value (i) j : Then ( i j (i) S (i) k (i) ) ( + n ; 1) ;1 G 0 +( + n ; 1) ;1 X j6=i ( + n ; 1) ;1 G 0 +( + n ; 1) ;1 k X (i) j=1 ( j ) n (i) j ((i) j ) (2) where (x) dentes the distributin degenerate at i = x: This shws that i is distinct frm the ther parameters and drawn frm G 0 with chance =( + n ; 1) therwise it is chsen frm existing realised values accrding t a multinmial allcatin with prbabilities prprtinal t existing grups sizes n (i) : j Extensin f n t n + 1 in (2) is directly relevant in predicting a further case i = n +1 this requires the distributin f ( n+1 j S k) which, incidentally, is just the psterir mean E(Gj S k) i.e. ( n+1 j S k) E(Gj S k) ( + n) ;1 G 0 +( + n) ;1 kx i=1 n i ( i ): (3) where ( i ) dentes the distributin degenerate at n+1 = i : As a result, predictive inference fr future data y n+1 fcuses n cnditinal predictive densities (y n+1 j S k) ( + n) ;1 F n+1 (:j k+1 )+( + n) ;1 kx i=1 n i F n+1 (:j i ) (4) where k+1 is a further independent draw frmg 0 : In sme cntexts where alternatives t the cmmn \shrinkage" eects induced thrugh the use f a mre traditinal prir G 0 are sught, this mixture prir cnstructin has appeal it reects autmatic adaptive shrinkage induced by the implicit gruping f subsets f parameter, and prvides mechanisms fr assessing dierences amng the parameters. The prblems f extreme r utlying parameters ptentially crrupting inference (West 1984,1985 O'Hagan 1988) are avided. 3 Psterir Cmputatins Escbar (1988) intrduced Mnte Carl Markv chain methds in simplied versins f the abve mdel, later extended t univariate nrmal mixtures fr density estimatin in Escbar and West (1991). Further extensins and applicatins in Escbar and West (1992), West and Ca (1992), Turner and West (1993), and elsewhere, are based n nrmal data distributins F i and cnditinally cnjugate base prirs G 0 (:): Recent wrk f Erkanli, Muller and West (1992) uses a rened Mnte Carl Markvchain algrithm based n riginal wrk f MacEachern (1992), and als adpts nn-cnjugate base prirs. The general Gibbs sampling strategy ppsed by MacEachern is utlined here, fllwed by cmments specic t nrmal linear mdels and assumed frms f G 0 and hyperparameter estimatin. 3

5 Hierarchical Prirs and Mixture Mdels February 9, Basic Gibbs Sampling As usual in Gibbs sampling, we identify cllectins f cmplete cnditinal psterir distributins that determine the full psterir fr all parameters = f 1 ::: n g: (Nte that we are cntinuing under the assumptin that and are knwn, which is relaxed belw). Nte that knwledge f is equivalent t knwledge f k, S and = f 1 ::: k g: Original simulatin methds in Escbar (1988) and Escbar and West (1991) simulate values in via psterirs based n the cnditinal prirs (2). MacEachern (1992) suggested a mre ecient algrithm that wrks in terms f the theretically equivalent parameters k, S and = f 1 ::: k g: MacEachern's algrithm is generally preferable and recmmended, fr reasns given in his paper it applies quite generally, aswe illustrate here. Frm (2), we can immediately deduce sets f cnditinal psterirs where the chances q i j ( i jy (i) S (i) k (i) ) q i 0 G i 0 + are given by k X (i) j=1 q i j ( (i) ) (5) j c hi (y i ) if j =0 q i j = c n (i) f j j (y i j j ) if j>0 where: G i 0 dentes the psterir btained by updating the prir G 0 via the likelihd functin f i (y i j i ) namely dg i 0 ( i ) / f i (y i j i )dg 0 ( i ) and subject t nrmalisatin h i (y i )isaweight btained via h i (y i )= Z f i (y i j i )dg 0 ( i ) (6) i.e. the marginal density fy i evaluated at the realised datum, under the base prir fr i c is a cnstant f nrmalisatin. Equatin (5) immediately implies cnditinal psterirs fr the cnguratin indicatrs, namely P (S i = jjy (i) S (i) k (i) )=q i j : (7) We maynw successively sample sets f values k, S and = f 1 ::: k g by iterating thrugh the fllwing sequence: (a) Given ld values and S generate a new cnguratin by sequentially sampling indicatrs frm the psterirs (7), successively simulating and substituting S 1 S 2 ::: fr any index i such that S i =0 draw a new i frm G i 0 in (5) (b) Given k and S generate a new set f parameters by sampling each new j frm the relevant cmpnent psterir in (1), namely Q r2ij f r(y r j j ) dg 0 ( j ): 4

6 Hierarchical Prirs and Mixture Mdels February 9, 1994 Successive simulatins eventually lead t sampled values drawn apprximately frm the jint psterir p(k S jy ): General issues f cnvergence are cvered in MacEachern and Muller (1994). Inference may be based n histgrams f sampled values f individual paremeters, r the mre usual renements based n Mnte Carl averages f analytically available cnditinal distributins and mments the abve reference gives further details. Fr example, averaging (4) with respect t simulated values f 1 ::: k+1 prvides an apprpriate Mnte Carl estimate f the actual predictive density functin p(y n+1 jy ): Each such sampled mixture is based n apprpriate values f 1 ::: k generated at the crrespnding Gibbs iteratin, and n an apprpriate values f k+1 drawn directly frm G 0 : The abve discussin is all cnditinal n the cmmn parameter : In sme mdels, there will be n such parameter and nthing mre need be said. If cmmn parameters are included in the data mdel, then the simulatin analysis extends as fllws t include steps t generate values frm apprpriate cnditinal psterirs. Given a prir p() we have p(jy S k) / p() ny i=1 f i (y i j i )= ky Y j=1 i2ij f i (y i j j ) (8) The successive substitutin sampling scheme abve then extends t include a simulated value f frm this distributin, cnditining n the mst recently sampled values f S and k at each step. 3.2 Extensin t Hyperparameters Usually the hyperparameters will be uncertain, having a specied prir. In Escbar and West (1991) and sme f the ther references, fr example, extensins t include in analyses are detailed fr nrmal mdels. Mre generally, the structure and cnsequent extensins f simulatin analyses are detailed here. Discussin in Sectin 3.1 is all cnditinal n which includes the Dirichlet precisin and ther parameters in G 0 call the latter parameters s that = f g and nw make explicit the dependence n hyperparameters in the ntatin G 0 (:j): Distributins (1) and (7) simulated in steps (a) and (b) are cnditinal n : Gibbs sampling is directly extensible t incrprate simply by adding a further step t draw frm an apprpriate cnditinal psterir. Generally, the relevant distributin is (jy ) (jy S k): This may bedevelped, as in the special cases in West (1992) and Escbar and West (1991), as fllws. First, we assume that and are apriri independent with specied density p( ) = p()p(): It fllws, as in the abve references, that and are als cnditinally independent given Y S and k hence the hyperparameters and can be cnsidered separately. Fr the Dirichlet prcess structure is such that nly k is relevant in inference abut s that p(jy S k) p(jk): Develpments in West (1992), applied in Escbar and West (1991) and ther papers cited abve, shw hw this psterir p(jk) may be represented as a mixture f gamma distributins if the prir p() is a mixture f gammas. Applicatins t date have assumed a single gamma prir, in which case the psterir is easily simulated. Examples f applicatins appear in the abve references. Fr nte that this parameter enters nly thrugh G 0 then, using (1), p(jy S k) p(j k) / p() ky j=1 dg 0 ( j j): (9) Base prirs G 0 that enable direct sampling f psterirs (9) maybesughtinanygiven applicatin. Assume this is the case then knwledge f Y S and k reduced t the sucient infrmatin set and k leads immediately int simulatin f a value fr f g as fllws: 5

7 Hierarchical Prirs and Mixture Mdels February 9, 1994 (c) Given k draw a new value f frm p(jk) (West 1992 Escbar and West 1991) given k and draw anewvalue f frm (9). The resulting value fr = f g is then used in cnditining at (a) and (b) in the next iteratin. 3.3 Nrmal Linear Mdels: Cnjugate Base Prirs The abve develpment is quite general, and certain applicatins with nn-nrmal mdels have been implemented. Mst f the experience t date is, hwever, with nrmal linear mdels in regressin and density estimatin, and we nw specialise t that cntext, assuming f i (:j i )t be nrmal with mean X i i and variance matrix i herex i is a knwn r p design matrix, i an uncertain p;vectr, and i an uncertain p p variance matrix. Als, i = f i i g: Nte particularly that there is n cmmn, uncertain parameter in this case (we nte an extensin t include a cmmn scale parameter at the end f this sub-sectin.) In the special case f randm sampling, where multivariate density estimatin is the fcus, r = p and X i is the p p identity matrix. Immediately, equatin (1) suggests a cnjugate nrmal/inverse Wishart frm fr the base prir G 0 : With such a frm, psterir analysis fr invlves a set f k independent nrmal regressin analyses with traditinal cnjugate prirs. Resulting independent psterirs (1) fr the individual parameters i are then nrmal/inverse Wishart t. Full details appear in Escbar and West (1992). Thus (1) may be easily sampled. Furthermre, in the steps t simulate S and (as a result) k frm (7), the required weights q i j are easily cmputed. Fr each j>0 this invlves the evaluatin f a multivariate nrmal density functin fr j = 0 the required expressin (6) reduces t a multivariate T density functin, als easily evaluated. In this cntext the hyperparameters determine the chsen nrmal/inverse Wishart base prir G 0 fr each i and i : Explicitly, such a prir has the frm i N(m b i ) and ;1 i W (s (ss) ;1 ) (10) with = fm b s Sg herem is the p;vectr prir mean fr i band s are psitive scalars, S is a p p variance matrix. s p is the Wishart degrees f freedm index, and the ntatin crrespnds t density functin p( ;1 js S) =cjssj s=2 j i i j ;(s;p;1)=2 exp(;trace(ss ;1 )=2) with i E( ;1 )=S ;1 thus S is a prir estimate fr i i (the prir harmnic mean). Early implementatins by the authrs use this structure (Escbar and West 1992), which is illustrated in Sectin 4 belw, ften xing at specied prir values. Realistically, uncertainty abut these quantities is almst surely evident, s we turn t the issue f suitable prirs fr : In all cases, we assume s t be specied. Fr fm b Sg the structure f equatin (9) is suggestive we simply nte that {a cnjugate nrmal/inverse gamma prir fr m and b implies a cnjugate cnditinal psterir p(m bj s S) and {a Wishart prir fr S implies a Wishart cnditinal psterir p(sj m b s): Under such prirs, with s xed and fm bg a priri independent fs the cnditinal psterirs may be sampled{rst generating m b given and the previusly sampled value f S, then sampling S based n and the new values f m and b. This is easily implemented. Typically, a rather diuse prir may be adpted fr m and b{indeed, West and Ca (1993) use the traditinal reference prir p(m b) / b ;1 {as lng as the prir is relatively infrmative frs. All applicatins t date assume s specied in applicatins f West and Ca (1993), and Turner and West (1993), substantial prir infrmatin is available in the assessment fs and S: In ther applicatins, anticipated ranges f 6

8 Hierarchical Prirs and Mixture Mdels February 9, 1994 data values may be used t assess the prir fr S while the Wishart degrees f freedm parameter s will typically be taken in the lw integers t reect high uncertainty. Sme further discussin f prir assessment fr appears in Sectins 4 and 5 belw. In sme applicatins, a cmmn scale parameter (r variance matrix) may be desirable in the individual regressin equatins, s that extensin f the abve develpment is needed. Assuming a cmmn scale factr >0implies f i (:j i ) is nrmal with mean X i i and variance matrix 2 i : Nw the abve develpment f the simulatin sequences applies all cnditinal n and the analysis simply needs t be supplemented with simulatin f values frm the apprpriate cnditinal psterirs in (8). We simply nte that, if p( 2 )isinverse gamma, then (8) is immediately an inverse gamma, and s easily simulated t, neatly cmpleting the Mnte Carl iteratin. 3.4 Nrmal Linear Mdels: Nn-cnjugate Base Prirs Mst recent applicatins have adpted a nn-cnjugate base prir specicatin, described in this subsectin, and used in illustratins in Sectins 4 and 5 belw. The cnjugate frm has implicatins that are smetimes undesirable. Under the abve prir specicatin, suppse the hyperprir fr m and b is diuse relative t the likelihd functin in (9) when is knwn then, fr example, the cnditinal psterir mean fr m is essentially a matrix weighted average f the i with matrix weights prprtinal t ;1 : Cnsider a cntext in which ne f the i i is truly quite separate frm the rest, and that the number n i f assciated bservatins in that grup is very small cmpared t n. The psterir fr i will supprt small variatin, hence high precisin ;1 as a result, the i term ;1 i i in the cnditinal mean f m will be heavily inuential. Indeed, the entire psterir fr m and b given will be verly inuenced by such an extreme and small grup. This is a feature induced by the cnjugate frm f G 0 and is alleviated by assuming an alternative, nn-cnjugate and independent frm i N(m B) independently f ;1 i W (s (ss) ;1 ) (11) where nw B is a variance matrix, and s = fm B s Sg: The develpment f the previus sectins can be fllwed thrugh with certain bvius mdicatins t the iterative sampling f cnditinal psterirs, described belw. (i) The cmpnent psterirs fr the i in (1) are n lnger cnjugate frms, and s each i = f i g is updated by rst sampling i cnditinal n a value f and then i i cnditinal n. The cnditinal psterir fr i ( i j Y i i S k )ismultivariate nrmal, and i the distributin ( i j Y i i S k )isinverse Wishart. Nte that iterating these tw draws suciently ften wuld simulate an apprximate draw frm( i jy i i S k ). While nt generally necessary, this might be desirable if the previus draw fr was based n a very i dierent cnguratin. (ii) In sampling cnguratins using (7) at step (b) f the iteratins, the new cmpnent prbability q i 0 requires evaluatin f the integral in (6). This can be apprximated numerically using quadrature methds r Mnte Carl integratin. The latter may be implemented by replacing the integral in (6) with an average ver draws frm the base prir G 0 : Alternatively, the integratin in (6) can be partially perfrmed analytically. Fr the cmpnents m B and S f the hyperparameter the abve develpment is mdied usefully in nn-cnjugate frm as fllws: m has a nrmal prir, hence a nrmal psterir when B and S are xed B has an inverse Wishart prir, hence an inverse Wishart psterir when m and S are xed the prir and cnditinal psterirs fr S are Wishart, as in the previus sectin. 7

9 Hierarchical Prirs and Mixture Mdels February 9, 1994 As nted at the end f Sectin 3.3, sme applicatins invlve a cmmn scale parameter multiplying the individual variance matrices i : This is the case in an example in the fllwing sectin. The immediate extensin f the simulatin scheme t incrprate a cmmn scale factr and extend the simulatin analysis applies here as discussed in Sectin 3.3: if the mdel is rephrased in terms f (y i j i ) N(X i i 2 i ) the abve develpment applies cnditinal n and values f are simulated frm the cnditinal psterirs in (8) if p( 2 )isinverse gamma, then (8) is als inverse gamma, and s trivially simulated. Other prirs are pssible, f curse, as (8) is a univariate density. 4 A Regressin Example In the develpment f mixture prirs in hierarchical mdels f Escbar and West (1992), ne particular theme was the use f Dirichlet prcess prirs as a way f mdelling uncertainty abut the functinal frm f the usually assumed prir G: In particular, traditinal nrmal hierarchical mdels assume G t be either cnjugate nrmal/inverse Wishart r independent nrmal (fr regressin vectrs) and inverse Wishart (fr variance matrices) (e.g. Gelfand et al 1990). The Dirichlet mixture structure elabrates this traditinal assumptin by using ne such frm fr the baseline prir G 0 s that G is initially expected t be in sme apprpriate neighburhd f the usual frm. Psterir inference abut G then permits sensitivity analysis t assess the traditinal assumptin, and resulting psterir and predictive inferences will nt be s dependent n the assumed frms if they are a psterir revealed t be inapprpriate. We illustrate these cncepts here thrugh a direct applicatin f the hierarchical regressin analysis in Sectin 3.4. In additin t prviding an interesting example and sensitivity analysis, we elabrate the structure f the cnditinal psterirs used in Gibbs sampling in sme detail. Gelfand et al (1990) cnsidered analysis f a grwth curve prblem where individual grwth curves f several subjects are mdelled using a standard linear regressin with nrmal errr. The assumed prirs fr the regressin cecients are there taken t be cnditinally nrmal{the traditinal baseline analysis. Our analysis makes this assumptin fr the actual baseline distributin G 0 suppsing that the prir G is \near" a bivariate nrmal, and hence admitting uncertainty abut the between-grup ppulatin distributin in these mdels. The data set in Gelfand et al cntains the weights f rats (n =30taken frm the cntrl grup in that paper), measured at the same ve time pints. Let y i j dente the jth measurement ntheith rat, cllected in the vectr y i =(y i 1 ::: y i 5 ) 0 fr i =1 ::: n=30: Individual grwth curves are mdelled as (y i j i ) N(X i i 2 I) where X i is 5 2 design matrix with jth rw (1 x i j )andx i j is the age f rat i at the jth time pint, and I is the 5 5 identity matrix. Nte that the generality f Sectin 3 is nt needed here since the individual variance matrices i are degenerate at i = I and s the crrespnding psterir simulatin steps are vacuus. Als, i i fr each i and we nte that the rats have a cmmn within-grup variance 2. Gelfand et al (1990) use the standard cnjugate assumptins t mdel the i adpting the prir ( i jm B) N(m B) independently. Their prir fr the parameters f m Bgassumes mutual independence and cnditinal cnjugacy, with u 2 IG 2 uu 2 m N(a A) B ;1 W (c (cc) ;1 ) 8

10 Hierarchical Prirs and Mixture Mdels February 9, 1994 where IG dentes the inverse gamma distributin, and the parameters u U a A c and C determining these prirs will be specied in any analysis. T sample the psterir distributin, Gelfand et al perfrmed a Markv chain Mnte Carl dened by the fllwing cnditinal distributins: u +5n ( 2 jy ) IG 1 h nx i uu + (y i ; X i i ) 0 (y i ; X i i ) 2 2 i=1 ( i jy m B ) N(D i (B ;1 m + ;2 X 0 iy i ) D i ) (mj B) N(V (A ;1 a + B ;1 n h (B ;1 jm ) W c + n cc + nx i=1 nx i=1 i ) V) ( i ; m)( i ; m) 0 i ;1 where D i =(B ;1 + ;2 X 0X i i) ;1 and V =(A ;1 + nb ;1 ) ;1. Nte that the cnditining f all distributins here explictly recgnises nly thse quantities that are relevant fr example, the full, required distributin fr ( 2 jy m B) des nt depend n m and B s they are nt written in the cnditining. This implicit simplicatin is used thrughut the fllwing discussin t. Cnsider uncertainty in the distributinal frm f the prir n i mdelled by assuming the abve baseline prir within a Dirichlet prcess framewrk. Then i = i = fm Bg, and f 1 ::: k g represents the k distinct values amng the full set f i 's. The parameter 2 is cmmn acrss all bservatins and we willcntinue t use an inverse gamma prir fr this parameter. >Frm Sectin 3 and using equatins (5){(7), we deduce the specic frms f relevant cnditinal psterir distributins detailed belw. On a pint f ntatin, we are replacing by thrughut (e.g. in equatins (5) and fllwing) s that (i) = f 1 ::: i;1 i+1 ::: n g: We then have cnguratins generated accrding t chances q i j in equatin (7). In this specic cntext, these are as fllws: q i 0 is prprtinal t times the prbability density functin f a multivariate nrmal distributin, evaluated at y i with mean X i m and cvariance matrix X i BX 0 + i 2 I fr j >0 q i j is prprtinal t the prbability density functin f a multivariate nrmal, evaluated at y i with mean X i j and cvariance P matrix 2 I the chances are nrmalised s that q i 0 + q j6=i i j =1. Given a cnguratin and the crrespnding k distinct means the remaining cnditinal psterirs are as fllws. 2 is sampled frm just the inverse gamma detailed abve{thugh the i cluster int k distinct values, the sum f squares term P in p( 2 jy ) des nt explicitly recgnise the fact (mj B) N(V (A ;1 a + B ;1 k j=1 j) V ) with V =(A ;1 + kb ;1 ) ;1 and n h P i ;1 (B ;1 k j m) W c + k cc + ( j=1 j ; m)( j ; m) : 0 In extending frm the traditinal baseline analysis f Gelfand et al t the Dirichlet prcess analysis, nte that there is n change t the cnditinal distributin f 2 and that the cnditinal distributins f m and B just have then assumedly distinct i values replaced by k actually distinct j values. In cnnectin with the data analysed in Gelfand et al, we assume the same, rather diuse prirs fr, m, andb as in that paper. Specically, these are given by u =0 A ;1 =0 c =2 and C = :1 :

11 Hierarchical Prirs and Mixture Mdels February 9, 1994 The prir fr is taken as gamma with shape and scale bth unity, very diuse. In cmparing the Dirichlet prcess prir methd with the mre traditinal baseline analysis, ne f the mst surprising ndings with analysis f this data set is the strng indicatin f multimdality in the ppulatin f i evidenced in Figure 1. Here we display Mnte Carl y estimates f the univariate margins f the psterir expectatin E(G()jY ) this is cmputed as the average f sampled densities E(Gj S k) frm equatin (3), each a mixture f a small number f bivariate nrmals. The displayed densities therefre represent the marginal predictive distributins fr the elements f a further vectr, s indicate predictins fr the grwth curve parameters fr a further rat frm the same ppulatin. These tw densities appear in the gure as full lines the dashed lines are crrespnding densities frm a repeat f the baseline analysis f Gelfand et al. In the latter, these margins are cnstrained t be smth and unimdal, but embedding within the \semi-parametric" Dirichlet family \arund" the baseline prir allws fr and apparently supprts rather strng hetergenity in the rat ppulatin. Figure 1 abut here 5 A Multivariate Density Estimatin Example Dirichlet mixture mdels have been cnsidered as framewrks fr nn- (r semi-) parametric density estimatin fr ver twenty years, thugh nly recently has rutine applicatin becme cmputatinally feasible. Restricting the mdelling framewrk f Sectins 2 and 3 t the multivariate randm sampling cntext leads directly t density estimatin. We illustrate this here with a 5- dimensinal data set frm Lubischew (1962), prviding ve measurements f physical characteristics f male insects f the species chactcnema cncina, chactcnema heikertinger, and chactcnema heptaptamica. Fr any beetle, we dentethevectr y f ve measurements by y =(x 1 :::x 5 ), s that the ith beetle has measurements y i =(x i 1 :::x i 5 ) 0 (i =1 :::n) the data set has n =74 cases. The ve variables are width f the rst jint (x 1 ), width f the secnd jint (x 2 ), head width (x 3 ), aedeagus width (x 4 ), and aedeagus side width (x 5 ). Figure 2 shws scatterplts f the rst tw pairs f variables. One simulatin analysis is partly summarised in the remaining gures. The randm sampling cntext has y i N( i i ) with i = f i i g there is n cmmn parameter in this mdel. We assume a baseline prir G 0 f the nn-cnjugate frm in Sectin 3.4, equatin (11). The analysis illustrated assumed the mutually independent hyperprir n the parameters = f m B Sg in which: p() is gamma with shape parameter a 0 and scale parameter b 0 m N(a A) B ;1 W (c (cc) ;1 ) and S W (q q ;1 Q): The specic prirs chsen fr this illustratin have c = q =9 s=15 and the matrix parameters A C and Q are diagnal matrices with variances simply reecting the scale f the prblem. The gamma prir fr has shape a 0 = 5 and scale b 0 =0:5 therefre supprting a diuse range f reasnably large values cnsistent with pssibly large values f k: In this case f n = 74 bservatins, y Gibbs sampling here is based n 20,000 cnsecutive iteratins perfrmed, with the cnditinal densities averaged t give Figure 1 cmputed n every 20th iteratin, beginning at the 20th frm the start. The simulatin was initialised by setting the i t least squares values fr each subject, and the hyperparameters m and B t their least squares values assuming the initialized i : 10

12 Hierarchical Prirs and Mixture Mdels February 9, 1994 R the implied prir p(kjn) = p(kj n)p()d can be (numerically) evaluated, and supprts a wide range f k values between abut k =8andk =35(p(kjn) > 0 fr all k, f curse, thugh is very small utside this range f k values) with mean/mde near k =20: Such a prir is cnsistent with the idea that we are cncerned with lcal smthing, nt verduly cnstraining the mdel t favur a small number f nrmal cmpnents. Ntice that we d nt suggest that the mdel, and this ver-smthing prir, are in any way designed t address issues f clustering r discriminatin the fcus is simply ne f density estimatin. Examining prir predictive densities and ther features f predictive distributins allws fr assessment fimplicatinsfchsen prirs. Fr example, taking m B and i at their prir means implies that the cnditinal prir predictive distributin fr any single y i is just N(m B + i ) cnturs f ne bivariate margin f this density are pltted, at fractins f 0.8, 0.6, 0.4 and 0.2 f the mdal value, in Figure 5(a). This is quite diuse ver the data range fr these tw variables, as it is fr the ther variables, and clearly des nt anticipate specic departures frm nrmality. Analysis reprted here is based n 5000 iteratins f the Gibbs sampling scheme with summarised predictive densities cmputed by averaging the cnditinal mixtures f nrmals (4) acrss these all draws. Figure 3 shws tw f the bivariate prjectins f the full ve dimensinal predictive density estimate. A cncentratin int three clearly distinguished cmpnents is apparent, and, in fact, these crrespnd precisely with the three species f beetle, thugh, f curse, n infrmatin abut species f individuals is used in the analysis. Indeed, the analysis is remarkably accurate in reprducing the species discriminatin the Mnte Carl estimate f the psterir fr k gives apprximate prbabilities 0.01, 0.48, 0.30, 0.15, 0.02 and 0.01 fr values f k =2 ::: 8, respectively, with the mde at k = 3, \truth" in this case. The data substantially verrides the prir n k which favured much larger values f k: It may be thught that the appreciable psterir mass n k = 4 and 5 is a result f this ver-smthing prir, thugh the usual \ver-tting" aspects f mixture mdelling likely cntribute t. That is, whatever the prir fr k, psterirs in these (and ther) mixture mdels will typically give residual mass t larger values f k crrespnding t pssible cnguratins in which sme grups cntain just a very few bservatins. T illustrate in this example, simply ignring simulated grups with fewer that 7 bservatins (ie. cnditining the psterir n appreciable sized grups) leads t apprximate psterir chances f 0.01, 0.92 and 0.06 n k =2 3and 4 respectively, perfectly highlighting the three beetle ppulatins. With a much less severe, and perhaps mre apprpriate and realistic, restrictin t grups sizes f at least 3 bservatins, the cnditinal psterir chances are apprximately 0.01, 0.71 and 0.25 n k =2 3 and 4, respectively. Psterirs fr the parameters i culd be estimated and summarised in similar fashins. We simply nte here that the psterir reects very little uncertainty abut the lcatins i f the suggested three clusters Figure 4 prvides sme insight by displaying the rst tw cmpnents f just a few f the sampled i : Sme applicatins f these mdels are mre cncerned with smth regressin estimatin (Erkanli, Muller and West 1992) rather than density estimatin per se. Regressin estimatin simply invlves summarising inferences abut cnditinal predictive distributins, and s is trivially btained within the existing mixture analysis. Fr example, the cnditinal predictive expectatins E(x 2 jy x 1 ) deduced directly frm the Mnte Carl estimate f the bivariate distributin, appears in Figure 6(a) with the crrespnding data pints superimpsed. This frame als presents apprximate 66% credible bands cmputed pintwise. Cnditinal mdes, rather than means, appear in Figure 6(b), superimpsed n the predictive densitycnturs nte the bifurcatin induced acrss the range where the estimated cnditinal density p(x 2 jy x 1 ) is bimdal. 11

13 Hierarchical Prirs and Mixture Mdels February 9, 1994 Sme technical cmments abut implementatin f the Gibbs sampling scheme are in rder. It is well knwn that Markv chain Mnte Carl methds generally may becme \trapped" in lcal mdes f psterir distributins. An incidence f this can be bserved in the current example. While the psterir gives mst weight tk = 3 r 4 clusters, there is sme remaining psterir prbability (estimated at 0:01) fr k =2 but a transitin frm k =3tk = 2 is mst unlikely in the Gibbs sampling scheme. Since nly ne bservatin at a time is being changed in the sampling f cnguratins in ur simulatin rutine, such a transitin wuld require the Gibbs sampler t pass thrugh many rather unlikely cnguratins (when k is still 3, but sme bservatins are \misclassied") befre settling n k = 2. Hwever, bth states, k = 2 as well as k = 3, are easily reached frm the cnguratin k = n, which was used t initialize the Gibbs scheme. Fr this reasn we implemented the Gibbs sampling iteratins by \reinitialising" the cnguratins at k = n (thugh nt the hyperparameter values) after each 1000, rather than simply running ne lnger chain. All f the abve discussin is based n the full ve dimensinal density estimate and its bivariate prjectins. The picture changes slightly when deriving the bivariate density estimate frm nly the marginal data set f x 1 and x 2 : Reanalysis in just the tw dimensinal space is briey summarised in Figure 7 this prvides the bivariate density estimate and draws i frm the psterir distributin as in the riginal ve dimensinal analysis. While the density estimate has changed nly a little, the psterir n the mdel parameters exhibits greater uncertainty. Indeed, the estimated psterir prbabilities fr the number f clusters k in this case are cnsiderably dierent with mass diusely spread ver the range k =9 ::: 27 cnsistent with the prir the data is rather uninfrmative abut k and cnguratins. This greater diuseness f psterir inference fr k (which is als reected in increased uncertainty abut the cluster means i shwn in Figure 7(b)) indicates that, crucial infrmatin n the discriminatin f the three species is lst when cnsidering nly this tw dimensinal slice f the full ve dimensinal data set. Acknwledgements Figures 2{7 (inclusive) abut here Discussins with Cli Littn and Steve MacEachern cntributed materially t the nal revisin f this paper. Mike West was partially supprted by the Natinal Science Fundatin under grant DMS Michael D Escbar was partially nanced by Natinal Cancer Institute #RO1- CA and a Natinal Research Service Award frm NIMH Grant #MH References Antniak, C.E. (1974) Mixtures f Dirichlet prcesses with applicatins t nnparametric prblems. Ann. Statist., 2, DuMuchel, W.M. and Harris, J.E. (1983) Bayes methds fr cmbining the results f cancer studies in humans and ther species (with discussin). J. Amer. Statist. Assc. 78, Erkanli, A., Muller, P., and West, M. (1992) Curve tting using mixtures mdels. Invited revisin fr Bimetrika. Escbar, M.D. (1988) Estimating the means f several nrmal ppulatins by nnparametric estimatin f the distributin f the means. Unpublished PhD dissertatin, Yale University. Escbar, M.D., and West, M. (1991) Bayesian density estimatin and inference using mixtures. Invited revisin fr J. Amer. Statist. Assc. 12

14 Hierarchical Prirs and Mixture Mdels February 9, 1994 Escbar, M.D. and West, M. (1992) Cmputing Bayesian nnparametric hierarchical mdels. ISDS Discussin Paper #92-A20, Duke University. Gelfand, A.E., and Smith, A.F.M. (1990) Sampling based appraches t calculating marginal densities. J. Amer. Statist. Assc. 85, pp Gelfand, A.E., Hills, S.E., Racine-Pn, A., and Smith, A.F.M. (1990), Illustratin f Bayesian inference in nrmal data mdels using Gibbs sampling. J. Amer. Statist. Assc. 85, Ku, L., and Smith, A.F.M. (1992) Bayesian cmputatins in survival mdels via the Gibbs sampler. In Survival Analysis: State f the Art, J.P. Klein and P.K. Gel (eds.) Kluwer. Lindley, D.V. (1972) Bayesian Statistics, A Review, SIAM, Philadelphia. Lindley, D.V. and Smith, A.F.M. (1972) Bayes' estimates fr the linear mdel (with discussin). J. Ry. Statist. Sc. (Ser B), 34, Lubischew, A. (1962), On the use f discriminant functins in taxnmy, Bimetrics, 18, MacEachern, S.M. (1992) Estimating nrmal means with a cnjugate style Dirichlet prcess prir. Technical reprt N 487, Department f Statistics, The Ohi State University. MacEachern, S.M. and Muller, P. (1994) Estimating mixture f Dirichlet prcess mdels. ISDS Discussin Paper, Duke University. O'Hagan, A. (1988) Mdelling with heavy tails. In Bayesian Statistics 3, J.M. Bernard, M.H. De Grt, D.V. Lindley and A.F.M. Smith (eds.) Oxfrd. Smith, A.F.M. (1983) Cmment n article by DuMuchel and Harris, J. Amer. Statist. Assc. 78, Turner, D.A.T. and West, M. (1993) Statistical analysis f mixtures applied t pstsynptential uctuatins. Jurnal f Neurscience Methds (t appear). West, M. (1984) Outlier mdels and prir distributins in Bayesian linear regressin. J. Ry. Statist. Sc. (Ser. B), 46, West, M. (1985) Generalised linear mdels: scale parameters, utlier accmmdatin and prir distributins. In Bayesian Statistics 2, J.M. Bernard, M.H. De Grt, D.V. Lindley and A.F.M. Smith (eds.) Nrth Hlland, Amsterdam. West, M. (1990) Bayesian kernel density estimatin. ISDS Discussin Paper #90-A02, Duke University. West, M. (1992) Hyperparameter estimatin in Dirichlet prcess mixture mdels. ISDS Discussin Paper #92-A03, Duke University. West, M. and Ca, G. (1993) Assessing mechanisms f neural synaptic activity. InBayesian Statistics in Science and Technlgy: Case Studies, C. Gatsnis, J. Hdges, R. Kass and N. Singpurwalla (eds.) (t appear). 13

15 p(n+1 1jY ) p(n+1 2jY ) Figure 1: Marginal predictive densities fr the tw elements f the regressin vectr n+1 =(n+1 1 n+1 2) 0 f the grwth curve fr a future rat. The full lines represent densities frm the mixture analysis, t be cmpared with the dashed curves frm the traditinal r baseline analysis f Gelfand et al (1990). 1

16 SECOND JOINT AEDEAGUS WIDTH FIRST JOINT HEAD WIDTH Figure 2: Scatterplts f x 1 x 2 and x 3 x 4 frm the beetle data. Fr infrmatin, the three dierent species are marked by diamnds, squares and triangles (thugh the classicatin infrmatin is nt used in analysis). 2

17 SECOND JOINT SECOND JOINT FIRST JOINT FIRST JOINT p(x 1 x 2 jy ) p(x 3 x 4 jy ) Figure 3: Predictive densities p(x 1 x 2 jy ) and p(x 3 x 4 jy ). Bth densities are tw dimensinal marginals f the estimated ve dimensinal distributin p(yn+1jy ). Tw f the three clusters are verlapping in the (x 3 x 4 )-prjectin. The estimated clustering crrespnds exactly t the three species. 3

18 SECOND JOINT FIRST JOINT Figure 4: Sme elements f a few simulated draws i p(ijy ). The gure shws the rst tw elements f the sampled values f i, i = 1 :::k,after t = and 5000 iteratins f the Gibbs sampler. The center f each circle crrespnds t the value f sme i. The areas f the circles shw the crrespnding sizes ni f the clusters. 4

19 SECOND JOINT FIRST JOINT ALPHA (a) Prir/psterir predictive distributins (b) Prir/psterir fr. Figure 5: Panel (a) cmpares psterir predictive p(yn+1jy ) (grey shades) and prir predictive p(yn+1) (cnturs). Panel (b) shws the gamma prir p() (line) and the estimated psterir p(jy ) (histgram). The implicatin is that inference in this example is mstly driven by the data, which partially justies the use f sme default chices fr hyperprir parameters. 5

20 SECOND JOINT SECOND JOINT FIRST JOINT (a) Cnditinal means FIRST JOINT (b) Cnditinal mdes Figure 6: Taking predictive cnditinal means E(x 2 jx 1 Y) leads t the nnlinear regressin line shwn in (a). Alternatively, mdes f the cnditinal predictive distributin p(x 2 jx 1 Y) can be used t btain mdal regressin traces shwn in (b). The nnlinear regressin line is shwn tgether with 66% credible bands cmputed pintwise{ fr each x 1 -value, the margins give a predictin interval fr x 2 and the intervals are fund by taking the shrtest interval f cnditinal psterir prbability 0.66 which cntains the cnditinal mde (nte that highest density regins becme discnnected when the cnditinal density becmes bimdal.) 6

21 FIRST JOINT SECOND JOINT FIRST JOINT SECOND JOINT (a) p(xn+1 1 xn+1 2jY ) (b) Sme sampled i Figure 7: Predictive density p(x 1 x 2 jy )andaavur f the psterir distributin fr elements f the i parameters in the bivariate mdel. While the density estimate shws nly little change frm Figure 5(a), the psterir n i displays much mre uncertainty abut the clusters. 7