An Infinite Product of Sparse Chinese Restaurant Processes

Transcription

1 An Infinite Product of Sparse Chinese Restaurant Processes Yarin Gal Tomoharu Iwata Zoubin Ghahramani

2 CRP quick recap The Chinese restaurant process (CRP) Distribution over partitions of N natural numbers P N. Marginal distribution of the Dirichlet process. Following a Chinese restaurant metaphor Restaurant = partition R PN, Table = partition block T R, Customer sitting at a table = element in a block i T. Restaurant configuration = sequence of block sizes n = (n 1, n 2,..., n K ), in order of appearance 2 of 18

3 CRP quick recap Follows a recursive construction Given discount parameter d [0, 1) and concentration parameter c > d, Given restaurant R N with configuration n = (n 1,..., n K ) with K tables and K i=1 n i = N customers, Probability of customer N + 1 to sit at an existing table T i : Or at a new table T K +1 : p(n + 1 T i R N ) = n i d N + c p(n + 1 T K +1 R N ) = c + K d N + c. 3 of 18

4 An interesting question We assumed discount d [0, 1) and concentration c > d with conditional probabilities What happens when... c is fixed and d 1? d is fixed and c? c is fixed and d 0? d is fixed and c d? n i d N + c, c + K d N + c. 4 of 18

5 An interesting question We assumed discount d [0, 1) and concentration c > d with conditional probabilities What happens when... c is fixed and d 1? d is fixed and c? n i d N + c, c + K d N + c. c is fixed and d 0? one-parameter CRP d is fixed and c d? very interesting indeed 5 of 18

6 A CRP parametrisation Discount d is fixed and concentration c d. Let γ > 0 and M N. Given restaurant R N M with configuration (n 1,..., n K ) (sum to N), Let R N+1 M follow a CRP distribution with concentration parameter d + γ/m and discount parameter d [0, 1). Probability of customer N + 1 to sit at an existing table T i R N M : p(n + 1 T i R N M ) = Or at a new table T K +1 : n i d N d + γ/m M n i d N d, p(n + 1 T K +1 RM N d + γ/m + K d M (K 1)d ) = N d + γ/m N d. 6 of 18

7 A CRP parametrisation Discount d is fixed and concentration c = d + γ/m. We have M R M R. First customer sits at table T 1, Probability of second customer to sit at existing table T 1 R 1 : p(2 T 1 R 1 ) = 1 d 1 d = 1, Or at a new table T 2 : p(2 T 2 R 1 ) = (1 1)d 1 d = 0, For all i N: p(i T 1 R i 1 ) = i 1 d i 1 d = 1. A.s. resulting configuration with N customers all sitting at T 1 : (N) referred to as a degenerate configuration. Not that interesting... 7 of 18

8 A sparse CRP (scrp) parametrisation But if we had M restaurants with concentration c = d + γ/m... Theorem (Sparse parametrisation of the CRP) Let γ > 0 be a sparsity parameter and m M N. Let RM,m N follow a CRP distribution with concentration d + γ/m and discount d [0, 1) with N customers. Denote by (n m ) M m=1 the random sequence of configurations of the restaurants (RM,m N )M m=1. The following holds true as M : the expected count of degenerate configurations (N) is unbounded, the expected count of non-degenerate configurations is given by γh d (N 1) with H d (N) = N i=1 1 i d. 8 of 18

9 A sparse CRP (scrp) parametrisation So, in the infinite sequence of sparse CRPs (scrps)... almost all restaurants have all customers sitting next to a single table, but finitely many restaurants have more than a single table. In fact, These have exchangeable partition probability function (EPPF): K ( ni 1 i=1 j=1 p(n 1, n 2,..., n K ) = (j d)) K 2 i=1 (id) N 1 i=1 (i d). We can analytically write the probability of a sequence of non-degenerate configurations 9 of 18

10 Infinite product of sparse CRPs For an infinite product of sparse CRPs the probability of configurations {n m } K + m=1 follows, Theorem (Infinite product of sparse CRPs) The probability of K + non-degenerate configurations {(n m 1,..., nm k m ) m [K + ]} is given by γ K + K +! e γh d(n 1) K + m=1 ( k m ( n m i 1 i=1 j=1 (j d)) k m 2 i=1 (id) ) N 1 i=1 (i d) Obtained by setting c = d + γ M CRPs with M. in a product of M sparse 10 of 18

11 Infinite product of sparse CRPs We can re-write this equation as a product of Poisson densities and the EPPFs above... γ K + K +! e γh d(n 1) } {{ } Poissons K + ( k m ( n m i 1 i=1 And obtain a recursive construction. j=1 (j d)) k m 2 i=1 (id) ) N 1 i=1 m=1 } (i d) {{ } EPPFs Instead of extending the Chinese restaurant metaphor, we use an urn scheme: 11 of 18

12 Infinite product of scrps urn scheme Time permits... Definition (Urn scheme) Create Poi( γ ) urns; for each urn 1 d add two balls with distinct colours. At the l th step (l 3) For each existing urn with balls of k colours, ni balls of colour i select colour i with probability n i d and add another ball of the same l 1 d colour, or add a ball of a new colour with probability (k 1)d. l 1 d Create Poi( γ ) new urns; for each urn l 1 d add a ball of a new colour, add l 1 balls of a distinct colour. This suggests an MCMC scheme. 12 of 18

13 Infinite product of scrps properties y=sparsity*h(disc) Draws from the prior 50 Average number of urns Sparsity parameter Expected number of urns as a function of the sparsity parameter γ with d = 0.1 and N = 100. y = γh d (N 1) 13 of 18

14 Infinite product of scrps properties 45 y=sparsity*h(disc) Draws from the prior 40 Average number of urns Discount Expected number of urns as a function of the discount parameter d with γ = 5 and N = 100. y = γh d (N 1) 14 of 18

15 Infinite product of scrps properties y=sparsity*h(disc) Draws from the prior 45 Average number of urns Number of draws Expected number of urns as a function of the number of draws N with γ = 5 and d = 0.4. y = γh d (N 1) 15 of 18

16 Infinite product of scrps properties Draws from the prior, p=1 Draws from the prior, p=0.5 y N pd, p=1 y N pd, p=0.5 Number of colours in the urn N (1 p) Number of draws Expected number of colours in the first urn (p = 1, blue) and in the N 0.5 th urn (p = 0.5, green) as a function of the number of draws N y N pd. 16 of 18

17 Relation to existing research The process induces a categorical feature allocation Restaurants = features Tables = feature values i th customer in each restaurant = data point i i th customer sitting next to table T in restaurant R = data point i takes value T for feature R Relates to existing literature in the field: IBP (Griffiths and Ghahramani, 2011) binary feature allocation Continuum of urns (Roy, 2014) a product of urns with balls of two colours characterising the IBP 17 of 18

18 Finally - future research What can we do with this process? Multi-view clustering; principled extension to Cross-Cat (Mansinghka, Jonas, Petschulat, Cronin, Shafto, Tenenbaum, 2009), Cross-Cat: Infinite latent attribute model (Palla, Knowles, Ghahramani, 2012), etc. Bayesian representation learning Features as data representation Currently running experiments! Thank you 18 of 18