Bayesian Nonparametrics: some contributions to construction and properties of prior distributions

Transcription

1 Bayesian Nonparametrics: some contributions to construction and properties of prior distributions Annalisa Cerquetti Collegio Nuovo, University of Pavia, Italy Interview Day, CETL Lectureship in Statistics, PSC & Dept of Mathematics and Statistics Lancaster University, April 29, 2009

2 Outline Some background on Bayesian Nonparametrics Exchangeability and de Finetti s theorem What we need for BNP How to construct a.s. discrete priors : resorting to Pitman & Gnedin work A note on priors obtained from exponentially tilted PK models On a Gibbs characterization of N-GG priors Generalized Chinese Restaurant rules for GIBBS partitions On a class of priors based on subordination of Stable processes.

3 Exchangeability and de Finetti s theorem What we need for BNP Three constructions for a.s discrete RPM: Dirichlet prior Some background on BNP In BNP no assumption is made on the law P on (S, S ) governing (X n ) n 1 w.r.t. belonging to a parametric family, we just assume exchangeability i.e. P (X 1 B 1,..., X n B n ) = P (X π(1) B 1,..., X π(n) B n ) the order of the observed values is assumed to be irrelevant to the inference P is invariant to the family of finite permutations

4 Exchangeability and de Finetti s theorem What we need for BNP Three constructions for a.s discrete RPM: Dirichlet prior Some background: de Finetti s representation theorem exchangeable p.m.s are mixtures of product p.m.s P (B) = P P(B i )q(dp) there exists a random probability measure P on (S, S) such that X i P i.i.d P q (the prior) on (P, P) is the law of the a.s. limit of the empirical distributions: 1 n i n δ Xi ( ) P( ) q i=1

5 Exchangeability and de Finetti s theorem What we need for BNP Three constructions for a.s discrete RPM: Dirichlet prior Some background: What we need for BNP inference Propose Priors i.e. random probability measures (a.s. discrete) by specifying finite dimensional distributions by means of Infinite sum representations by Normalization of Completely Random Measures Calculate Posteriors Without conjugacy: derive Predictive distributions for generalized Pólya urn Gibbs sampler. An example: Dirichlet Process Prior (first BNP prior proposed)

6 Exchangeability and de Finetti s theorem What we need for BNP Three constructions for a.s discrete RPM: Dirichlet prior Some background: Dirichlet process prior Finite dimensional distributions (Ferguson, 1973), P(B 1 ),..., P(B k ) Dir(θH(B 1 ),..., θh(b k )) for each measurable partition (B 1,..., B k ) of S, H a non-atomic p.m. on S θh( ) a positive measure on S, with θh(s) <.

7 Exchangeability and de Finetti s theorem What we need for BNP Three constructions for a.s discrete RPM: Dirichlet prior Some background: Dirichlet process prior Infinite sum representations (Kingman, 1975) P( ) := P i δ Xi ( ) i=1 P( ) has the Ferguson-Dirichlet (θh) distribution if (P i ) = (J i / i J i) have the law of the normalized ranked lengths of the points of a Poisson prox with mean intensity ρ(x) = θx 1 e x (Gamma) (X i ) iid H( ), independent of (P i ),

8 Exchangeability and de Finetti s theorem What we need for BNP Three constructions for a.s discrete RPM: Dirichlet prior Some background: Dirichlet process prior Normalization of a Completely Random Measure (Ferguson, 1973; Kingman, 1975) P( ) := µ( ) T has the Ferguson-Dirichlet (θh) distribution if µ( ) Ga(θH( ), 1), (Gamma CRM) T = i J i = µ(s) Ga(θ, 1) a.s. positive Lévy density ρ θ (x) = θx 1 e x (infinite total mass),

9 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions Motivation BNP has dramatically re-emerged during the last ten years. Many contributions, very active community, a regular series of Workshops... BNP theory actually lies at the intersection among: Bayesian statistical inference, Random partitions theory, Lévy processes theory, Combinatorics... It happens that some new contributions turn out to be reformulations of results known in collateral topics. Clarifying these intersections/connections is important for future developments

10 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions Pitman & Gnedin work on Poisson-Kingman (ρ α, γ) models A large class of models for random ranked atoms (P i ) is introduced in Pitman (2003), first generalizing Kingman (1975) (P i ) = (J i / i J i) Poisson-Kingman (ρ α ) on P 1 for ρ α the α-stable Lévy density, α (0, 1) then deriving the class of mixed PK(ρ α, γ) models PK(ρ α, γ) := 0 PK(ρ α t)γ(dt) PK(ρ α, γ) are models for exchangeable random partitions

11 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions Pitman & Gnedin work on Poisson-Kingman (ρ α, γ) models (X 1,..., X n,... ) from a.s. discrete P induces a random partition Π = {A 1,..., A k,... } of N by i l X i = X l for each restriction to [n], for n j = A j, Pr(Π n = {A 1,..., A k }) = p(n 1,..., n k ) p (symmetric) is the exchangeable partition probability function (EPPF) which completely describes Π.

12 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions Pitman & Gnedin work on Poisson-Kingman (ρ α, γ) models Gnedin & Pitman (2005) show that EPPFs induced by PK(ρ α, γ) models, for α (0, 1), are characterized by having Gibbs form i.e. p(n 1,..., n k ) = V n,k k (1 α) nj 1. Consistency conditions imply the non-negative weights V = (V n,k ) must satisfy the recursion j=1 V n,k = (n αk)v n+1,k + V n+1,k+1

13 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions A note on RPM derived from exponentially tilted PK models [Stat & Prob Lett 07] In Lijoi et al. (JASA 2005), by mimicking the finite dimensional distributions construction of the Dirichlet process priors with the Inverse Gaussian density playing the role of the Gamma density, Normalized Inverse Gaussian priors are proposed. the Inverse Gaussian is an Infinite Divisible (ID) r.v. arising from the exponential family generated by 1/2-stable r.v. (exponential tilting)

14 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions A note on RPM derived from exponentially tilted PK models [Stat & Prob Lett 07] I have shown that: since the same operation on α-stable r.v. for α (0, 1) produces Generalized Gamma distributions (Brix, 1999) we can derive Normalized Generalized Gamma priors by Infinite sum representations for (P i ) PK(ρ GG = ρ exp α ) and all the relevant quantities for BNP can be easily derived from Pitman s results: EPPF induced by N-GG priors sequential rules to describe predictive distributions distribution of the number of blocks of the N-GG partition.

15 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions On a Gibbs characterization of N-GG [Stat & Prob Lett, 08] In Lijoi et al. (Stat Sinica 2008), an analytic procedure is proposed to prove the following characterization: N-GG priors for α (0, 1) are the unique class of priors inducing EPPFs in Gibbs form and also admitting a construction by normalization of Completely random measures. I have shown that it suffices to merge two already known characterization results: Pitman (2003), for exponentially tilted PK models Gnedin & Pitman (2005), for Gibbs partitions of type α to prove a slightly more general characterization, circumventing the need for an analytical derivation.

16 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions Generalized Chinese Restaurant rules for GIBBS partitions [arxiv: ] In Lijoi et al. (AoAP 2008) a BNP analysis under Gibbs priors with applications in species sampling problems is proposed. Given a sample from P with (P i ) PK(ρ α, γ) they obtain distributional posterior results for an additional sample of size m w.r.t. number of new distinct species number of new observations belonging to new species random partition induced by the additional sample

17 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions Generalized Chinese Restaurant rules for GIBBS partitions [arxiv: ] I have derived a generalized (group sequential) version of the Pitman s sequential construction of EPPFs, (known as Chinese Restaurant Process) as the theoretical framework in which such analysis should and can be effectively embedded. According to Prof. Pitman this generalization: provides some clarification of Lijoi et al. s work, connects it better to the Gnedin & Pitman development, provides a better platform for future work.

18 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions Generalized Chinese Restaurant rules for GIBBS partitions [arxiv: ] Given an infinite EPPF, p(n) = V n,k k j=1 (1 α) n j 1, assume that an unlimited number of group of customers arrives sequentially in a restaurant for n 1, given (n 1,..., n k ), the placement of the first group of n customers in a (n 1,..., n k ) configuration in k tables, the new group of m customers is: all seated at the old k tables in configuration (m 1,..., m k ), with probability p(m n) = V k n+m,k (n j α) mj, (1) V n,k j=1

19 Some fundamental results of Pitman & Gnedin A note on RPM derived from exponentially tilted PK models On a Gibbs characterization of N-GG Generalized Chinese Restaurant rules for GIBBS partitions Generalized Chinese Restaurant rules for GIBBS partitions [arxiv: ] all seated at k new tables in configuration (s 1,..., s k ) with probability p(s n) = V k n+m,k+k (1 α) sj 1, (2) V n,k s < m are seated at k new tables in configuration (s 1,..., s k ) and the remaining m s customers at the k old tables in configuration (m 1,..., m k ) with probability j=1 p(m, s n) = V n+m,k+k V n,k k k (n j α) mj (1 α) sj 1. (3) j=1 j=1 These rules, by means of some combinatorial calculus and properties of generalized Stirling numbers, are enough to embed Lijoi et al. BNP analysis of Gibbs partitions.

20 On priors derived by subordination of Stable processes Ongoing work: On priors derived by convolution mixtures of Stable r.v. Each BNP prior correspond to an ID r.v. (with specific properties) Until now BNP researchers focus on Exponential Tilting of r.v.s (N-IG, N-GG) Finite Convolutions of ID r.v.s (Generalized Dirichlet Priors) A natural conjecture is to investigate the possibility to exploit the third method to obtain ID from ID, i.e. Convolution mixtures

21 On priors derived by subordination of Stable processes Ongoing work: On priors derived by convolution mixtures of Stable r.v. In Lévy process theory this corresponds to Bochner s subordination: the random time change of a subordinator by a subordinator independent of the original one. (e.g. Sato, 2000) I am investigating convolution mixtures of Stable r.v. with Generalized Gamma Convolutions (Bondesson, 1992). By exploiting recent results on BNP for normalized CRM, (James et al. 2009) it should be possible to derive a complete posterior and predictive BNP analysis.

22 Selected references Cerquetti, A. (2007) A note on Bayesian nonparametric priors derived from exponentially tilted Poisson-Kingman models. Stat. Prob. Lett.,77, Cerquetti, A. (2008) On a Gibbs characterization of normalized generalized Gamma processes. Stat. Prob. Letters, 78, Cerquetti, A. (2009) Generalized Chinese restaurant construction of exchangeable Gibbs partitions and related results. arxiv: v1 [math.pr] (Submitted) Ferguson, T. S. (1973) A Bayesian analysis of some nonparametric problems. Ann. Stat., 1, 2, Gnedin, S. and Pitman, J. (2005) Exchangeable Gibbs partitions and Stirling triangles arxiv:math.pr/ Kingman, J.F.C. (1975) Random discrete distributions. JRSS B, 37, Lijoi, A., Mena, R. and Prünster, I. (2005) Hierarchical mixture modeling with normalized Inverse-Gaussian priors. JASA, 100, Lijoi, A., Pruenster, I. and Walker, S.G. (2008) Investigating nonparametric priors with Gibbs structure. Stat. Sinica, 18, Lijoi, A. Pruenster, I. and Walker, S.G. (2008) Bayesian nonparametric estimators derived from conditional Gibbs structures. Ann. Appl. Probab. 18, Pitman, J. (2003) Poisson-Kingman partitions. IMS, LN 40, Institute of Mathematical Statistics, Hayward, California. (arxiv:math.pr/ ).

23 For those of you interested... 7th Workshop on Bayesian Nonparametrics Collegio Carlo Alberto, Moncalieri, Torino (Italy) June 21 to 25, 2009 Registration deadline: 30th April