Variational Bayesian EM algorithm for modeling mixtures of non-stationary signals in the time-frequency domain (HR-NMF)

Transcription

1 Variational Bayesian EM algorithm for modeling mixtures of non-stationary signals in the time-frequency domain HR-NMF Roland Badeau, Angélique Dremeau To cite this version: Roland Badeau, Angélique Dremeau. Variational Bayesian EM algorithm for modeling mixtures of non-stationary signals in the time-frequency domain HR-NMF. ICASSP, 2013, Vancouver, Canada. IEEE, pp , <hal > HAL Id: hal Submitted on 24 Mar 2014 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

2 VARIATIONAL BAYESIAN EM ALGORITHM FOR MODELING MIXTURES OF NON-STATIONARY SIGNALS IN THE TIME-FREQUENCY DOMAIN HR-NMF Roland Badeau, Angélique Drémeau Institut Mines-Telecom, Telecom ParisTech, CNRS LTCI ABSTRACT We recently introduced the high-resolution nonnegative matrix factorization HR-NMF model for analyzing mixtures of nonstationary signals in the time-frequency domain, and highlighted its capability to both reach high spectral resolution and reconstruct high quality audio signals. In order to estimate the model parameters and the latent components, we proposed to resort to an expectation-maximization EM algorithm based on a Kalman filter/smoother. The approach proved to be appropriate for modeling audio signals in applications such as source separation and audio inpainting. However, its computational cost is high, dominated by the Kalman filter/smoother, and may be prohibitive when dealing with high-dimensional signals. In this paper, we consider two different alternatives, using the variational Bayesian EM algorithm and two mean-field approximations. We show that, while significantly reducing the complexity of the estimation, these novel approaches do not alter its quality. Index Terms Nonnegative Matrix Factorization, High Resolution methods, Expectation-Maximization algorithm, Variational inference. 1. INTRODUCTION Nonnegative matrix factorization NMF [1] is a powerful tool for decomposing mixtures of non-stationary signals in the timefrequency TF domain. However, unlike the high resolution HR methods [2] dedicated to mixtures of complex exponentials, its spectral resolution is limited by that of the underlying TF representation. Following previous works which aimed at providing a probabilistic framework for NMF [3 6], we introduced in [7, 8] a unified probabilistic model called HR-NMF, that permits to overcome this limit by taking both phases and local correlations in each frequency band into account. It can be used with both complex-valued and realvalued TF representations like the short-time Fourier transform or the modified discrete cosine transform. Moreover, we showed that HR-NMF generalizes some very popular models: the Itakura-Saito NMF model IS-NMF [6], autoregressive AR processes, and the exponential sinusoidal model ESM, commonly used in HR spectral analysis of time series [2]. In [7, 8], HR-NMF was estimated with the expectation-maximization EM algorithm, which involves time-demanding Kalman filtering and smoothing. In this paper, we introduce two faster algorithms based on variational inference, and compare the performance of the three algorithms. This paper is organized as follows. In Section 2, we present the HR-NMF model, as introduced in [7]. We recall the basics of the This work is supported by the French National Research Agency ANR as a part of the DReaM project ANR-09-CORD and partly supported by the Quaero Program, funded by OSEO. variational Bayesian EM algorithm in Section 3, before particularizing it to the HR-NMF model in Section 4. Section 5 is devoted to experimental results, and conclusions are drawn in Section 6. The following notation will be used throughout the paper: M :conjugateofmatrixorvector M, M : transpose of matrix or vector M, M H :conjugatetransposeofmatrixorvectorm, [M; N]: vertical concatenation of M and N, c =: equality up to an additive constant, h m: discrete convolution of times series h and m, N F µ, R: real if F = R or circular complex if F = C multivariate normal distribution of mean µ and covariance matrix R. 2. HR-NMF TIME-FREQUENCY MIXTURE MODEL The HR-NMF mixture model of TF data xf,t F where F = R or C is defined for all discrete frequencies 1 f F and times 1 t T as the sum of K latent components c k f,t F plus a white noise nf,t N F 0,σ 2 : where c k f,t = xf,t =nf,t+ P k,f p=1 K c k f,t 1 ap, k, f c k f,t p +b k f,t is obtained by autoregressive filtering of a non-stationary signal b k f,t F where ap, k, f F and P k, f N is such that ap k, f,k,f 0, b k f,t N F 0,v k f,t where v k f,t is defined as v k f,t =wk, f hk, t, 2 with wk, f 0 and hk, t 0, processes n and b 1...b K are mutually independent. Moreover, k, f {1...K} {1...F}, the random vectors c k f,0 = [c k f,0;...; c k f, P k, f +1]are assumed to be independent and distributed according to the prior distribution c k f,0 N F µ k f, Q k f 1, where the mean µ k f and the precision matrix Q k f are fixed parameters 1. Lastly, we assume that f {1...F}, t 0, xf,t is unobserved. Let c denote the set {c k f,t} k,f,t, x denote the set {xf,t} f,t and θ the set of model parameters σ 2, {ap, k, f} p,k,f, {wk, f} k,f 1 In practice we choose µ k f = [0;...;0] and Q k f 1 = ξi, where I is the identity matrix and ξ is small relative to 1, in order to both enforce the causality of the latent components and avoid singular matrices /13/$ IEEE 6171 ICASSP 2013

3 and {hk, t} k,t. Considering model 1, we focus on the maximum a posteriori MAP estimation of the latent components c =argmaxpc x; θ, 3 c where the model parameters are estimated according to a maximum likelihood ML criterion θ =argmaxpx; θ. 4 θ The solution of 3-4 can be found by means of an EM algorithm. We proposed in [7, 8] an efficient implementation, using a Kalman filter/smoother in the E-step. However, the computational cost remains high, dominated by the complexity of the Kalman filter/smoother, and may be prohibitive when dealing with large dimensions. We propose here an alternative, based on the variational Bayesian EM VB-EM algorithm, which uses a mean-field approximation of the posterior pc x; θ to reach a good compromise between quality and complexity of the MAP estimation VARIATIONAL BAYESIAN EM ALGORITHM Variational inference [9, 10] is now a classical approach for estimating a probabilistic model involving both observed variables x and latent variables c, parametrized by θ. Let F be a set of probability density functions PDF over the latent variables c. For any PDF q Fand any function fc, we note f q = fcqcdc. Then for any PDF q Fand any parameter θ, the log-likelihood Lθ =lnpx; θ can be decomposed as Lθ =D KLq pc x; θ + Lq; θ 5 qc where D KLq pc x; θ = ln 6 pc x; θ is the Kullback-Leibler divergence between q and pc x; θ, and pc, x; θ Lq; θ = ln 7 qc is called the variational free energy. Moreover, Lq; θ can be further decomposed as Lq; θ =Eq; θ+hq, where Eq; θ = ln pc, x; θ q, 8 and Hq = ln qc q is the entropy of distribution q. Since D KLq pc x; θ 0, Lq; θ is a lower bound of Lθ. The variational Bayesian EM algorithm is a recursive algorithm for estimating θ. It consists of the two following steps at each iteration i: E-step update q: q =argmin q F M-step update θ: θ i =argmax θ D KLq pc x; θ i 1 = argmax Lq; θ i 1 q F 9 Lq ; θ =argmaxeq ; θ. 10 θ F defines a set of constraints leading to a particular approximation of the posterior distribution pc x; θ i 1. We note that: In the standard EM algorithm, q is not constrained, thus q = pc x; θ i 1 and D KLq pc x; θ i 1 = 0. Therefore Lθ i Lq ; θ i Lq ; θ i 1 =Lθ i 1, which proves that the log-likelihood is non-decreasing. In the general case, Lθ is no longer guaranteed to be non-decreasing, but its lower bound Lq; θ is still nondecreasing. q q 4. VARIATIONAL BAYESIAN EM FOR HR-NMF Considering the HR-NMF model defined in 1, k, f {1...K} {1...F}, let c kf denote the set {c k f,t} t { P k,f+1...t }. Moreover, let α =1if F = C, and α =2if F = R. Then α lnpc, x = α K F lnpc kf +α F T δf,tlnpxf,t c 1f,t...c Kf,t f=1 = KFT + K F P k, f lnαπ K + K K F where f=1 F c k f,0 µ k f H Q k fc k f,0 µ k f F lndetq k f + T lnρ k f,t F T P k,f ρ k f,t c kf,t ap, k, fc k f,t p p=1 T δf,t lnαπσ xf,t K 2 c σ 2 k f,t 2 11 δf,t =1if xf,t is observed, and δf,t =0else in particular δf,t =0 t <1 and t >T, ρ k f,t = 1 v k f,t if t {1...T}, and ρ kf,t =0else. In the following subsections, we will first recall the EM-based algorithm presented in [7, 8] as a particular case of the variational procedure 9-10 Sections 4.1 and 4.2 and then propose two different alternatives to this costly approach, based on two mean-field approximations, i.e. two different definitions of F Sections 4.3 and 4.4. These three algorithms only differ in the E-step, but they share the same implementation of the M-step M-step The M-step defined in equation 10 consists in maximizing Eq ; θ w.r.t. the model parameters θ. First, equations 8 and 11 yield αeq ; θ c = F f=1 T δf,tlnαπσ 2 +ef,t/σ 2 K F T lnwk, fhk, t + ak, fh Sk, f, tak, f, wk, fhk, t 12 where ef,t =δf,t xf,t K 2 c k f,t Sk, f, t = c k f,t c k f,t q, c k f,t =[c k f,t;...; c k f,t P k, f], q, ak, f =[1; a1,k,f;...; ap k, f,k,f]. We note that ef,t and Sk, f, t can be computed as ef,t =δf,t xf,t K 2 m k f,t + K Γ k f,t, Sk, f, t =R k f,t + m k f,t m k f,t, where we have defined: 6172

4 m k f,t = c k f,t q, Γ k f,t = c k f,t m k f,t 2 q, m k f,t = c k f,t q, R k f,t = c k f,t m k f,t c k f,t m k f,t H q. The maximization of Eq ; θ in equation 12, w.r.t. σ 2, ap, k, f, wk, f, and hk, t, can then be performed as in the M-step presented in [8], using the current estimations of m k f,t and R k f,t derived from the E-steps as presented in the next sections E-step in the exact EM algorithm As mentioned in Section 3, in the exact EM algorithm q is not constrained, thus the solution of 9 is given by q = pc x; θ, and the variational free energy Lq,θ i is equal to the log-likelihood Lθ i. In [7, 8], we showed that the posterior distribution pc x; θ is Gaussian, and that its first and second order moments, as well as the value of Lθ i, can be computed by means of Kalman filtering and smoothing. The resulting E-step can symbolically be written as: for 1 f F do {m k f,t, R k f,t} 1 k K = Kalman {xf,t} 1 t T 1 t T end for Its computational complexity was shown to be OFTK P 3, where P =maxp k, f. k,f 4.3. E-step with structured mean field approximation If K>1, we assume that F, introduced in Section 3, is the set of PDFs which can be factorized in the form qc = K F q kf c kf. 13 Using this particular factorization for qc, the solution of9satisfies see [9]: k, f {1...K} {1...F}, lnq kf c kf c = lnpc, x l,g k,f q lg. 14 Then, reformulating equation 11 and using equation 14, we get α lnpc, x = c T δf,t α lnpc kf c σ 2 k f,t ĉ k f,t 2, 15 with ĉ k f,t =xf,t m l f,t. We observe that q kf is the l k posterior distribution of a HR-NMF model of order K =1, where the posterior means of all components other than k have been subtracted to the observed data xf,t. Hence q kf is Gaussian, and its first and second moments can be computed by applying the Kalman filter/smoother presented in [7, 8] to ĉ k f,t instead of xf,t. The resulting E-step can symbolically be written as: for 1 f F do for 1 k K do 1 t T, ĉ k f,t =xf,t l k m l f,t {m k f,t, R k f,t} 1 t T = Kalman{ĉ k f,t} 1 t T end for end for The complexity of this procedure is OFTK1 + P 3 instead of OFTK P 3 for the "classical" E-step. In order to evaluate this algorithm, we are also interested in computing the variational free energy L. After some straightforward calculations, we note that the entropy Hq kf satisfies αhq kf =T + P k, f lnαπ+1 + T lndetr k f,t T 1 lndetr k f,t, 16 where R k f,t is the P k, f P k, f top-left submatrix of R k f,t. Thus equations 7, 11 and 16 yield αlq; θ =KFT + K K + K K F P k, f traceq k fr k f,0 F m k f,0 µ k f H Q k fm k f,0 µ k f F lndetq k f + T lnρ k f,t F F f=1 + K T ρ k f,t ak, f H Sk, f, tak, f T δf,tlnαπσ 2 +ef,t/σ 2 F T lndetr k f,t T 1 lndetr k f,t 17 where m k f,0 is the P k, f 1 top subvector of m k f, E-step with mean field approximation If P>0, we further assume that F is the set of PDFs which can be factorized in the form qc = K F T t= P k,f 1 q kft c k f,t. 18 With this particular factorization of qc,thesolutionof9satisfies see [9]: k, f, t {1...K} {1...F} { P k, f+1...t}, lnq kft c k f,t c = lnpc, x l,g,u k,f,t q lgu. 19 Let us define the filter of impulse response h kf, such that h kf 0= 1, h kf p = ap, k, f p {1...Pk, f}, and h kf p = 0 everywhere else, and the filter h kf p = h kf p. After some straightforward calculations, equations 11 and 19 yield k, f, t {1...K} {1...F} { P k, f+1...t}, q kft c k f,t N F m k f,t, Γ k f,t, where 2 δf,t Γ k f,t = σ q k f,t+ h kf 2 ρ k f,t with q k f,t =Q k f 1 t,1 t if P k, f +1 t 0 and q k f,t =0else, and m k f,t =m k f,t+γ k f,t q k f,t H m k f,0 µ k f + δf,t xf,t K m σ 2 l f,t h kf ρ k f,th kf m k f,t l=1 2 h kf 2 denotes the filter whose coefficients are the square magnitude of the corresponding coefficients of h kf. 6173

5 Log likelihood with EM 120 Log likelihood with structured mean field Free energy with structured mean field 140 Log likelihood with mean field Free energy with mean field Iterations Fig. 1. Maximization of the log-likelihood and the variational free energy over the iterations. where q k f,t is the 1 t th column of Q k f if P k, f + 1 t 0 and q k f,t =[0;...;0] else 3. The computational complexity of the E-step is thus further reduced from OKFT1 + P 3 to OKFT1 + P, which is linear w.r.t. all dimensions. Finally, note that the variational free energy Lq; θ can be calculated as in equation 17, where R k f,t becomes a diagonal matrix of diagonal coefficients Γ k f,t...γ k f,t P k, f. 5. SIMULATION RESULTS The VB-EM algorithm aims to maximize the free energy. As we mentioned in Section 3, the log-likelihood is thus no longer guaranteed to increase, while remaining an indicator of the estimation quality. It can then be interesting to evaluate the influence of the approximations 13 and 18 on the maximization of the log-likelihood. To this end, we consider a fully observed TF data xf,t generated according to model 1 with T =20, F =3, P k, f=3 k, f and K =2 and random parameters θ, and compare the performance of the three algorithms described respectively in Subsections 4.2, 4.3 and 4.4 with regard to the maximization of the log-likelihood. Figure 1 presents the value of the log-likelihood at each iteration of the three algorithms. Interestingly, we can observe that although focusing on the maximization of the free energy, the VB-EM algorithm permits here to increase the log-likelihood, whatever the considered approximation mean-field or structured mean-field. In addition, as intuitively expected, the most constrained factorization 18 leads to a lesser increase of the log-likelihood. In practice however, this expected quality loss is not tangible. As an example of the good behavior of the VB-EM approach, we focus here on a simple case of source separation, where the observation is the whole STFT xf,t of dimensions F =400 and T =44 of a 1.05 s-long piano sound sampled at Hz, containing three notes, C3, C4 and C5, starting respectively at 0 ms, 260 ms and 525 ms, and lasting until the end of the sound. Within this scenario, we aim at separating K =3 components c k f,t of order P k, f=2 in the frequency band f which corresponds to the first harmonic of C5, the second harmonic of C4 and the fourth harmonic of C3 around 540 Hz. These three sinusoidal components whose real parts are represented as red solid lines in Figure 2 have very close frequencies, making them hardly separable. We compare then three different approaches, namely, the HR-NMF model estimated by means of the EM algorithm, the HR-NMF model estimated by means of the VB-EM algorithm using the mean-field approximation 18 and the IS-NMF model [6]. 3 In this equation, although the term m k t appears several times in the right-hand side, it can be easily verified that its contributions add up to zero. 4nd harmonic 540 Hz 2th harmonic 540 Hz 1rs harmonic 540 Hz a First component C Time s b Second component C Time s Real signal IS NMF EM based HR NMF VBEM based HR NMF c Third component C Time s Fig. 2. Separation of three sinusoidal components. Two important observations can be made about Figure 2. As previously noticed in [7, 8], IS-NMF in black dash-dotted lines, which involves Wiener filtering, is not able to properly separate the components when they overlap. As a comparison, the components estimated by HR-NMF blue dashed lines and magenda dotted lines better fit the ground truth. We see on this example that the EMbased and VB-EM-based approaches lead to very similar results: the separated components are often merged. More precisely, we measured an averaged mean squared error of for IS-NMF, for the VBEM-based HR-NMF and for the EM-based HR- NMF on the whole set of frequencies and components reconstructed within this experiment. The slight quality loss due to the mean-field approximation is largely compensated by a significant computation time saving: with a 2.20GHz CPU processor and 8Go RAM, the CPU time required to run the E-step in the exact EM approach with a Matlab implementation is 19.5s, while 1.9s is enough for the E-step with mean-field approximation. 6. CONCLUSIONS This paper introduces two novel methods as alternatives to estimate the HR-NMF model introduced in [7,8]. These methods are based on the variational Bayesian EM algorithm and two different mean-field approximations. Their low complexities allow using the HR-NMF model in high-dimensional problems without altering the good quality of the estimation. We illustrated these good properties with a simple example of source separation. In future work, we will investigate other kinds of structured and unstructured mean field approximations, as well as a fully Bayesian approach involving uninformative or informative priors for the various model parameters. We will also apply variational inference to the future extensions of the HR-NMF model e.g. involving convolutive and multichannel mixtures. 6174

6 7. REFERENCES [1] Daniel D. Lee and H. Sebastian Seung, Learning the parts of objects by non-negative matrix factorization, Nature, vol. 401, pp , Oct [2] Roland Badeau, Bertrand David, and Gaël Richard, High resolution spectral analysis of mixtures of complex exponentials modulated by polynomials, IEEE Trans. Signal Process., vol. 54, no. 4, pp , Apr [3] Mikkel N. Schmidt and Hans Laurberg, Non-negative matrix factorization with Gaussian process priors, Computational Intelligence and Neuroscience, vol. 2008, pp. 1 10, 2008, Article ID [4] Paris Smaragdis, Blind Speech Separation, chapter Probabilistic decompositions of spectra for sound separation, pp , Springer, [5] T. Virtanen, A.T. Cemgil, and S. Godsill, Bayesian extensions to nonnegative matrix factorisation for audio signal modelling, in Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP, LasVegas,Nevada,USA,Apr.2008,pp [6] Cédric Févotte, Nancy Bertin, and Jean-Louis Durrieu, Nonnegative matrix factorization with the Itakura-Saito divergence. With application to music analysis, Neural Computation, vol. 21, no. 3, pp , Mar [7] Roland Badeau, Gaussian modeling of mixtures of non-stationary signals in the time-frequency domain HR-NMF, in Proc. of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics WASPAA, New Paltz, New York, USA, Oct. 2011, pp [8] Roland Badeau, High resolution NMF for modeling mixtures of non-stationary signals in the time-frequency domain, Tech. Rep. 2012D004, Télécom ParisTech, Paris, France, July [9] M. Beal, Variational Algorithms for Approximate Bayesian Inference, Ph.D. thesis, Univ. College of London, London, U.K., May [10] Martin J. Wainwright and Michael I. Jordan, Graphical Models, Exponential Families, and Variational Inference, vol. 1 of Foundations and Trends R in Machine Learning, Now Publishers,