Statistical approach for dictionary learning Tieyong ZENG Joint work with Alain Trouvé Page 1
Introduction Redundant dictionary Coding, denoising, compression. Existing algorithms to generate dictionary K-SVD MOTIF Our object: find a dictionary more pertinent to a class of image translation-invariant Page 2
M. Elad and M. Aharon 2006 K-SVD results Page 3
MOTIF P. Jost, S. Lesage, P. Vandergheynst and R. Gribonval, 2006 Results of MOTIF: 19 generating functions learnt on natural images Page 4
Plan Generative model Bernoulli-Exponential Model (BEM) Bernoulli-Gaussian Model (BGM) Identifiably issues MCMC-EM for BEM Mean field method for BGM Experiments Conclusion Page 5
Generative model We denote: the discrete tore, support of images the spaces of images a family of local atoms generating the dictionary translation of the atoms over the plan We consider the following generative model : where Bernoulli variable, i.i.d, flag: active or disactive random coefficient, i.i.d, intensity white noise Goal: given, to learn Page 6
Two models: BEM and BGM The Bernoulli-Exponential Model (BEM) is of exponential distribution Parameter space The complete likelihood The Bernoulli-Gaussian Model (BGM) is of Gaussian distribution Parameter space Page 7
Identifiably problem We are interested in a statistical question: Can we distinguish two distributions on Y given by two different parameters? For both models: under weak conditions, YES! BEM Equivalent relation: if and modulo a permutation on the indices Prop. If are different, then the BEM is identifiable in BGM Similarly result holds. Page 8
Expectation-Maximization method Recall that our model is: B h,s,x h,s where are hidden variables. Usually we can use the EM method: E-step: specify a value for parameter, get the hidden variables by the conditional expectation M-step: given hidden variables, get a value for parameter by maximization the likelihood Page 9
From likelihood to MCMC The approach of maximum likelihood calculates The method of EM iterates where is the a posteriori distribution of (X, B) known Y with Z is a normalization factor and is the dominated measure. (E)-step, (M)-step We can approximate the a posteriori distribution by MCMC: we generate a Markov chain associated to Page 10
Pseudo-code of MCMC-EM Algorithm of MCMC-EM Page 11
Accept-rejection algorithm Suppose that we have a target distribution and an auxiliary distribution such that on the support of. Then we can simulate by the flowing algorithm: 1. GenerateX πandu U([0,1]) 2. AcceptY =X if 3. ReturnStep1ifrejected Page 12
The update of θ Now, let us look at the update of θ. Since, Calculating in a straightforward manner, we have, Page 13
How to update Φ In order to update the dictionary, we need solve: We first solve this problem without constraints, then project the elements on sphere by normalization. Denoting this is equivalent to: Page 14
Plan Generative model Bernoulli-Exponential Model (BEM) Bernoulli-Gaussian Model (BGM) Identifiably issues MCMC-EM for BEM Mean field method for BGM Experiments Conclusion Page 15
Mean field method for BGM is very difficult to calculate Mean field: we approximate by a product of distribution on and Considering the collection of distribution product on is of Bernoulli distribution is of distribution Then we take and such that where K is the Kullback-Leibler divergence: Page 16
The fixed point equation and update of the parameter For the Kullback-Leibler divergence, the optimal distribution satisfies a fixed point equation Then, the step of updating the parameter is given by In consequence, we have: Page 17
Numerical aspects Grids for fixed point equation Thresholding to get sparse elements Support compact where Initialization of parameters Matching Pursuit Page 18
Experiments for MCMC and mean field Training set Ideal dictionary MCMC Mean field Page 19
Experiment of mean field Example of Training set Real dictionary Mean field Page 20
Experiment of mean field Real dictionary Mean field Page 21
Mean field for natural image Page 22
Conclusion Generative model Bernoulli-Exponential Model (BEM) Bernoulli-Gaussian Model (BGM) Identifiably issues MCMC-EM for BEM Mean field method for BGM Experiments Future works Theoretical aspects, consistence Rigid atoms ==== Deformation atoms Natural images Page 23
Thank you! Page 24