Image segmentation combining Markov Random Fields and Dirichlet Processes

Transcription

1 Image segmentation combining Markov Random Fields and Dirichlet Processes Jessica SODJO IMS, Groupe Signal Image, Talence Encadrants : A. Giremus, J.-F. Giovannelli, F. Caron, N. Dobigeon Jessica SODJO ANR meeting 1 / 28

2 Plan 1 Introduction 2 Segmentation using DP models Mixed MRF / DP model Inference : Swendsen-Wang algorithm 3 Hierarchical segmentation with shared classes Principle HDP theory 4 Conclusion and perspective Jessica SODJO ANR meeting 2 / 28

3 Introduction Segmentation partition of an image in K homogeneous regions called classes label the pixels : pixel i z i {1,..., K } Bayesian approach prior on the distribution of the pixels all the pixels in a class have the same distribution characterized by a parameter vector U k Markov Random Fields (MRF) : exploit the similarity of pixels in the same neighbourhood Constraint : K must be fixed a priori Idea : use the BNP models to directly estimate K Jessica SODJO ANR meeting 3 / 28

4 Segmentation using DP models Plan 1 Introduction 2 Segmentation using DP models Mixed MRF / DP model Inference : Swendsen-Wang algorithm 3 Hierarchical segmentation with shared classes Principle HDP theory 4 Conclusion and perspective Jessica SODJO ANR meeting 4 / 28

5 FIGURE: Example of partition Jessica SODJO ANR meeting 5 / 28 ANR meeting Segmentation using DP models Notations N is the number of pixels Y is the observed image Z = {z 1,..., z N } Π = {A 1,..., A K } is a partition and m = {m 1,..., m K } with m k = A k A 1 A 3 A 2 AK m 1 = 1 m 2 = 5 m 3 = 6 m K = 4

6 Segmentation using DP models Mixed MRF / DP model Markov Random Fields (MRF) Description of the image by a neighbouring system Considered pixel Neighbours 4-neighbours 8-neighbours FIGURE: Examples of neighbouring system A clique c is either a singleton either a set of pixels in the same neighbourhood Jessica SODJO ANR meeting 6 / 28

7 Segmentation using DP models Mixed MRF / DP model Markov Random Fields Let θ i {U 1,..., U K } be the parameter vector associated to the i-th pixel MRF p(θ i θ i ) = p(θ i θ V(i) ) where V(i) is the set of neighbours of pixel i Hammersley-Clifford theorem Gibbs field ( p(θ) = 1 exp ( Φ(θ)) = 1 exp Z Φ Z Φ c Φ c (θ c ) ) (1) with Φ c (θ c ) the local potential and Φ(θ) the global one Limitation : K is assumed to be known Jessica SODJO ANR meeting 7 / 28

8 Segmentation using DP models Mixed MRF / DP model Potts model The Potts model is a special MRF defined by : M(Π) exp β ij 1 zi =z j (2) i j where i j means that the pixels i and j are neighbours β ij > 0 if i and j are neighbours and β ij = 0 otherwise Jessica SODJO ANR meeting 8 / 28

9 Segmentation using DP models Mixed MRF / DP model The DP model τ k γ, H Beta(1, γ) τ where Beta(.) is the Beta distribution k 1 k = τ k (1 τ l ) (3) Let us write τ Stick(γ), τ = {τ 1, τ 2,...} and k=1 τ k = 1 with G γ, H DP(γ, H) G = l=1 τ k δ Uk (4) k=1 U k H iid H (5) The distribution of the observations is f, defined as : y i θ i f (. θ i ) and θ i G G (6) Jessica SODJO ANR meeting 9 / 28

10 Segmentation using DP models Mixed MRF / DP model The DP model The Chinese Restaurant Process says, θ i θ i K i k=1 m i k N 1 + γ δ U k + γ N 1 + γ H m i k is the size of cluster k if we remove pixel i from the partition K i is the number of clusters in the image with the i-th pixel removed U k is the parameter vector associated to the k-th cluster Limitation : the spatial interactions are not taken into account Jessica SODJO ANR meeting 10 / 28

11 Segmentation using DP models Mixed MRF / DP model Principle of the segmentation using DP models Define a distribution on the partitions using : a model that allows that pixels in the same neighbourhood are likely to be in the same cluster (MRF) DP model to deduce automatically the number of clusters (and if needed their parameters) Jessica SODJO ANR meeting 11 / 28

12 Segmentation using DP models Mixed MRF / DP model Prior distribution mixing DP and MRF p(θ) 1 exp( Φ i (θ i )) Z G i }{{} Ψ(θ) DP model where C 2 means c 2 and. is the size. Φ i (.) is defined as : Φ i (θ i ) = log G(θ i ) and Z G = Ψ(θ) = N G(θ i ) i=1 1 Z M exp( c C 2 Φ c (θ c )) }{{} M(θ) MRF model N i=1 exp( log G(θ i ))dθ 1... dθ N P. Orbanz & J. M. Buhmann Nonparametric Bayesian image segmentation, International Journal of Computer Vision, 2007 Jessica SODJO ANR meeting 12 / 28

13 Segmentation using DP models Mixed MRF / DP model Prior distribution mixing DP and MRF We can deduce : P(θ i θ i ) K k=1 M(θ i θ i )m i k δ U k + γ Z Φ H (7) Probability of assignment to a new cluster : q i0 f (y i θ)h(θ)dθ (8) Ω θ Probability of assignment to an existing cluster : Parameter update : q ik m i k exp( Φ(U k θ i ))f (y i U k ) (9) U k G 0 (U k ) i i A k f (y i U k ) (10) Jessica SODJO ANR meeting 13 / 28

14 Segmentation using DP models Inference : Swendsen-Wang algorithm Swendsen-Wang algorithm : principle * Estimation based on the joint posterior p(θ, Z Y ) * Intractable Markov Chain Monte Carlo (MCMC) Problem : very slow convergence Goal : Sample faster the partition of the image Introduction of a new set of latent variables r such that : p(π, r) = p(π)p(r Π) p(r Π) = p(r ij Π) 1<i<j<N p(r ij = 1 Π) = 1 exp(β ij δ ij 1 zi =z j ) The marginal posterior p(θ, Z Y ) is unchanged The links define the "so-called" spin-clusters Jessica SODJO ANR meeting 14 / 28

15 Segmentation using DP models Inference : Swendsen-Wang algorithm Swendsen-Wang algorithm : principle Update the labels of the spin-clusters This operation update simultaneously the labels of all the pixels in a spin-cluster FIGURE: Example of label update for spin-clusters Jessica SODJO ANR meeting 15 / 28

16 Segmentation using DP models Inference : Swendsen-Wang algorithm Swendsen-Wang algorithm : principle r ij Ber(1 exp(β ij δ ij 1 zi =z j )) with Ber(.) is the Bernouilli distribution Let S = {S 1,..., S p } be the set of spin-clusters. While removing the spin-cluster S l, Π l = {A l 1,..., A l K l } is the partition obtained while removing all pixels in spin-cluster S l m l k = A l k Jessica SODJO ANR meeting 16 / 28

17 Segmentation using DP models Inference : Swendsen-Wang algorithm Swendsen-Wang algorithm : principle For l = 1 : p * The probability to assign pixels in spin-cluster S l to cluster k is : q lk Ψ(m l 1,..., m l k {(i,j) i S l,r ij =0} + S l,..., m l K l )p(y Sl exp(β ij (1 δ ij )1 zi =z j ) y A l ) k * The probability to assign pixels in spin-cluster S l to a new cluster is : with p(y Ak ) = q l0 = Ψ(m l 1,..., m l K l, S l )p(y Sl ) i A k f (y i U k )H(U k )du k Jessica SODJO ANR meeting 17 / 28

18 Hierarchical segmentation with shared classes Plan 1 Introduction 2 Segmentation using DP models Mixed MRF / DP model Inference : Swendsen-Wang algorithm 3 Hierarchical segmentation with shared classes Principle HDP theory 4 Conclusion and perspective Jessica SODJO ANR meeting 18 / 28

19 Hierarchical segmentation with shared classes Principle Proposed idea Different levels of classification can be considered Coarse categories : urban, sub-urban, forest, etc. Sub-classes shared between the categories : trees, roads, buildings Taking into account the fact that the classes are shared between different categories can help estimating their parameters and thereby improve the segmentation Jessica SODJO ANR meeting 19 / 28

20 Hierarchical segmentation with shared classes HDP theory Solution : Hierarchical DP Let J be the number of categories G 0 γ, H DP(γ, H) G j α 0, G 0 DP(α 0, G 0 ) for j = 1,..., J α 0 R + G 0 is a discrete distribution Discreteness of G 0 clusters shared among categories Jessica SODJO ANR meeting 20 / 28

21 Hierarchical segmentation with shared classes HDP theory G 0 = τ k δ Uk (11) k=1 where τ γ Stick(γ), τ = {τ 1, τ 2,...} and U k H H G j = π jk δ Uk (12) k=1 with π j α 0, τ DP(α 0, τ ) and π j = {π j1, π j2,...} ϕ ji G j G j (13) So, samples of the processes G 0 and G j can be seen as infinite countable mixtures of Dirac measures with respective coefficients τ and π j. Jessica SODJO ANR meeting 21 / 28

22 Hierarchical segmentation with shared classes HDP theory Principle - Chinese Restaurant Franchise NOTATIONS J restaurants Same menu for all restaurants - U 1, U 2,... T j is the number of tables in restaurant j θ jt is the t-th table of restaurant j ϕ ji is the i-th client in restaurant j n jt is the number of clients at a table t η jk is the number of tables in restaurant j which have chosen dish U k and η k = k η jk Jessica SODJO ANR meeting 22 / 28

23 Hierarchical segmentation with shared classes HDP theory Principle - Chinese Restaurant Franchise Menu U 1 U 2 U 3... ϕ 11 ϕ 13 ϕ 12 ϕ 14 θ 11 = U 1 ϕ 21 ϕ 23 ϕ 22 ϕ 25 θ 21 = U 2 ϕ 31 ϕ 32 ϕ 33 θ 31 = U 1 θ 12 = U 2 θ 22 = U 1 θ 32 = U 2 Restaurant 1 θ 13 = U 2... Restaurant 2... Restaurant 3 θ 33 = U 3... Jessica SODJO ANR meeting 23 / 28

24 Hierarchical segmentation with shared classes HDP theory Principle - Chinese Restaurant Franchise ϕ Exemple : Restaurant θ 11 = U 1 n11 = 2 Menu U 1 U 2 η1 = 3 η 2 = 4 ϕ 12 ϕ 14 θ 12 = U 2 θ 13 = U 2 n 12 = 1 n 13 = 1 α 0 ϕ 15 U 3 η 3 = 1 U 4 γ θ 14 Jessica SODJO ANR meeting 24 / 28

25 Hierarchical segmentation with shared classes HDP theory Principle - Chinese Restaurant Franchise ϕ Exemple : Restaurant θ 11 = U 1 n11 = 2 Menu U 1 U 2 η1 = 3 η 2 = 4 ϕ 12 ϕ 14 θ 12 = U 2 θ 13 = U 2 n 12 = 1 n 13 = 1 α 0 ϕ 15 U 3 η 3 = 1 U 4 γ θ 14 Jessica SODJO ANR meeting 24 / 28

26 Hierarchical segmentation with shared classes HDP theory Principle - Chinese Restaurant Franchise ϕ Exemple : Restaurant θ 11 = U 1 n11 = 2 Menu U 1 U 2 η1 = 3 η 2 = 4 ϕ 12 ϕ 14 θ 12 = U 2 θ 13 = U 2 n 12 = 1 n 13 = 1 α 0 ϕ 15 U 3 η 3 = 1 U 4 γ θ 14 Jessica SODJO ANR meeting 24 / 28

27 Hierarchical segmentation with shared classes HDP theory Principle - Chinese Restaurant Franchise ϕ Exemple : Restaurant θ 11 = U 1 n11 = 2 Menu U 1 U 2 η1 = 3 η 2 = 4 ϕ 12 ϕ 14 θ 12 = U 2 θ 13 = U 2 n 12 = 1 n 13 = 1 α 0 ϕ 15 U 3 η 3 = 1 U 4 γ θ 14 Jessica SODJO ANR meeting 24 / 28

28 Hierarchical segmentation with shared classes HDP theory Chinese Restaurant Franchise ϕ ji ϕ j1,..., ϕ ji 1, α 0, G 0 θ jt θ j1,..., θ 21,..., θ jt 1, γ, H T j t=1 K k=1 n jt i 1 + α 0 δ θjt + α 0 i 1 + α 0 G 0 (14) η k k η k + γ δ γ U k + k η k + γ H (15) Y. W. Teh, M. I. Jordan, M. J. Beal & D. M. Blei Hierarchical Dirichlet Processes, JASA, 2006 Jessica SODJO ANR meeting 25 / 28

29 Conclusion and perspective Plan 1 Introduction 2 Segmentation using DP models Mixed MRF / DP model Inference : Swendsen-Wang algorithm 3 Hierarchical segmentation with shared classes Principle HDP theory 4 Conclusion and perspective Jessica SODJO ANR meeting 26 / 28

30 Conclusion and perspective Conclusion Spatial constraints : Potts model Flexibility : DP model Rapidity : Swendsen-Wang algorithm Sharing : HDP Perspective Efficient sampling algorithm Jessica SODJO ANR meeting 27 / 28

31 Thank Thank you for your attention Jessica SODJO ANR meeting 28 / 28