Time and Frequency Domain Optimization with Shift, Convolution and Smoothness in Factor Analysis Type Decompositions

Transcription

1 Downloae from orbit.tu.k on: May 9, 218 Time an Frequency Domain Optimization with Shift, Convolution an Smoothness in Factor Analysis Type Decompositions Masen, Kristoffer Hougaar; Hansen, Lars Kai; Mørup, Morten Publication ate: 29 Document Version Early version, also known as pre-print Link back to DTU Orbit Citation (APA): Masen, K. H., Hansen, L. K., & Mørup, M. (29). Time an Frequency Domain Optimization with Shift, Convolution an Smoothness in Factor Analysis Type Decompositions. General rights Copyright an moral rights for the publications mae accessible in the public portal are retaine by the authors an/or other copyright owners an it is a conition of accessing publications that users recognise an abie by the legal requirements associate with these rights. Users may ownloa an print one copy of any publication from the public portal for the purpose of private stuy or research. You may not further istribute the material or use it for any profit-making activity or commercial gain You may freely istribute the URL ientifying the publication in the public portal If you believe that this ocument breaches copyright please contact us proviing etails, an we will remove access to the work immeiately an investigate your claim.

2 TIME AND FREQUENCY DOMAIN OPTIMIZATION WITH SHIFT, CONVOLUTION AND SMOOTHNESS IN FACTOR ANALYSIS TYPE DECOMPOSITIONS. Kristoffer H. Masen an Lars K. Hansen an Morten Mørup Technical University of Denmark Informatics an Mathematical Moelling Richar Petersens Plas, Builing 321 DK-28 Kgs. Lyngby, Denmark ABSTRACT We propose the Time Frequency Graient Metho (TFGM) which forms a framework for optimization of moels that are constraine in the time omain while having efficient representations in the frequency omain. Since the constraints in the time omain in general are not transparent in a frequency representation we emonstrate how the class of objective functions that are separable in either time or frequency instances allow the graient in the time or frequency omain to be converte to the opposing omain. We further emonstrate the usefulness of this framework for three ifferent moels; Shifte Non-negative Matrix Factorization, Convolutive Sparse Coing as well as Smooth an Sparse Matrix Factorization. Matlab implementation of the propose algorithms are available for ownloa at 1. INTRODUCTION Factor analysis [1] is wiely use to reconstruct multiple latent effects from mixtures base on the moel x m,n a m, s,n. (1) However, the ecomposition is not unique since  = AQ an Ŝ = Q 1 S yiel the same approximation as A, S. Consequently, constraints have been impose such as Varimax rotation for Principal Component Analysis (PCA) [2], statistical inepenence of the sources S as in Inepenent Component Analysis (ICA) [3, 4]. A relate strategy is sparse coing where the objective of minimizing the error is combine with a term enforcing sparsity of S []. Factor analysis in the setting of ICA is often illustrate by the so-calle cocktail party problem. Here mixtures of several speakers are recore in a number of microphones forming the measure signal X. The task is to recover the original speech signals S. Harshman an Hong [6] propose a generalization of the factor moel in which the source signals may be elaye on arrival at the the sensors. The moel is calle shifte factor analysis (SFA), an reas x m,n a m, s,n τm,. (2) equivalent moels have been propose in [4, 7, 8]. In real acoustic environments we expect echoes ue to multiple signal propagation paths, that arise from reflections off surfaces. To account for general elaye mixing, factor analysis moels have been further generalize to convolutive mixtures, see e.g., [9, 1, 11, 12, 13] x m,n a τ m, s,n τ. (3) τ, Here A τ is a filter that accounts for the presence of each source in the sensors at time elay τ. The shifte factor moel, thus is a special case of the convolutive moel where the filter coefficients a τ m, = a m, if τ m, = τ otherwise a τ m, =. A very comprehensive survey of methos for convolutive ICA is foun in [14]. It is well known that the frequency representation of time series ata provies an efficient framework to account for moels with shifts, convolutions an smoothness ue to the following basic observations Shift an convolution in the time omain correspons to multiplication in the frequency omain.

3 Smoothness of signals in the time omain correspon to low frequency representations in the frequency omain. The presence of efficient FFT algorithms for transformation between the time an frequency omain has a limite computational cost (O(n log n)). Unfortunately, moels are often constraine in the time omain in a way that is not transparent in a frequency representation an this may hamper optimization in the frequency omain. To improve component ientifiability we impose constraints such sparseness, nonnegativity, or smoothness. While shifts an convolution are efficiently implemente in the frequency omain constraints in the form of non-negativity, sparseness an smoothness are typically efine in the time omain without an explicit representation in the frequency omain. We will here propose a graient base technique that facilitates taking avantage of a frequency representation while accounting for constraints in the timeomain. We will emonstrate the usefulness of this approach on the following moels Shifte Non-negative Matrix Factorization, the ata an moel parameters are constraine nonnegative in the time omain while temporal shifts are represente efficiently through the frequency omain representation. Convolutive Sparse Coing, here convolution is efficiently implemente through multiplication in the frequency omain while the filter length in the time omain constrains regions of A τ to zero. Furthermore, the sparseness impose on S resies in the time omain. Sparse an Smooth Matrix Factorization, while smoothness constraints can efficiently be implemente in the frequency omain, sparseness constraint resies in the time omain. The paper is structure as follows. We will first escribe the propose Time Frequency Graient Metho (TFGM) that combines time an frequency omain optimization. Base on this approach we erive efficient algorithms for the three moels above an emonstrate their use on simulate as well as real ata. We have previously use the TFGM for Shifte Non-negative Matrix in [17], here the aim is to formalize the approach as well as generalize to broaer classes of moels. 2. NOTATION In the following u an U will enote a vector an matrix in the time omain, while ũ an Ũ enotes the corresponing vector an matrix in the frequency omain. Furthermore, Ũ H enotes the conjugate transpose of Ũ while U V enotes the irect prouct, i.e. element-wise multiplication. Ũ ( f ) f 1 i2π = U e N τ f 1 i2π where e N τ enotes element wise raising the el- f 1 i2π ements, i.e.(e N τ f 1 i2π ) n, = e N τ n,. Also, let u enote the th column an u n, a given element of U. Finally, let U F = m,n u m,n 2 be the Frobeniusnorm an F an F 1 enote the iscrete Fourier an inverse Fourier transform, i.e. F 1 ( s) : F(s) : s k = s n = 1 N N 1 i 2πk s n e N n n= N 1 s k e i 2πk N n. k= 3. METHODS AND RESULTS We will presently consier objective functions, C, of the form C = t f t (x t ) + 1 N f g f ( x f ), (4) where f t an g f are real value functions of the real an complex variables x t an x f such that x = F(x). Thus, we require the objective functions to be separable in either the time or frequency omain. The graient with respect to x t an x f of objective functions satisfying (4) can be written as C x t = f t (x t ) + 1 N f g f ( x f )e i2π f t = f t (x t ) + F 1 (g ) t C = x f f (x t )e i2π f t + g f ( x f ) t = F(f ) f + g f ( x f ) Thus, the graient of the objectives can be converte arbitrarily between the time an frequency omain enabling what we will enote the Time-Frequency Graient Metho (TFGM). The crux of this property follows from the separability into sums over time or frequency instances (t or f ). We note, that from Parseval s ientity we can arbitrarily state the least square

4 objective in a form satisfying (4) both in the time an frequency omain, i.e., x n 2 F = 1 n N f x f 2 F. () We note that a variable whichx is upate in the frequency omain has to remain real when applying the inverse DFT. For F 1 (g) to be real value the following has to hol g N f +1 = g f, where enotes the complex conjugate. This constraint is enforce by only consiering the first N/2 + 1 frequencies, i.e. up to the Nyquist frequency, while setting the functions of the remaining frequencies accoring to (??) Shifte Non-negative Matrix Factorization A popular approach for enforcing non-negativity is the use of multiplicative upates as introuce in [1, 16]. Given a cost function C(S) over the non-negative variables S, efine C(S)+ an C(S) as the positive an negative part of the erivative with respect to S,m. Then the multiplicative upate has the following form s,n s,n C(S) C(S) + A small constant ε = 1 9 is ae to the numerator an enominator to avoi ivision by zero or forcing s,n to zero. If the graient is positive C(S)+ α. > C(S), hence, s,n will ecrease an vice versa if the graient is negative. Thus, there is a one-to-one relation between fixe points an points where the graient is zero. The attractive property of multiplicative upates is that they automatically ensure non-negativity as the upates are base on multiplication, ivision, an raising to the power of purely non-negative variables. α is a "step size" parameter that potentially can be tune. Notice, when α only very small steps in the negative graient irection are taken. In [16] it was proven that for the least squares error α = 1 is guarantee to monotonically ecrease the cost function. The shifte non-negative matrix factorization moel propose in [17] is given by x m,n a m, s,n τm,, where X,A an S are non-negative. While shifts correspon to simple multiplication of a complex phase, the non-negativity constraint is not transparent in the frequency omain. Thus, a metho combining the apparent representation of non-negativity in the time omain with the efficient implementation of shifts in the frequency omain is esire. The least squares objective can be written as C LS (A, S) = 1 2N f x f à ( f ) s f 2 F. Thus, in the frequency omain the objective becomes separable over frequencies, however the non-negativity constraint resies in the time omain. The moel can be estimate alternatingly solving for A, S an τ as escribe in [17]. Here, the TFGM is applie for upate of the variable S. The graient of the least squares cost function in the frequency omain is [18] G f = C LS H f = 1 N Ã( f )H ( x f à ( f ) s f ). By applying the inverse DFT of the graient in the frequency omain the corresponing graient in the time omain is obtaine. Splitting the graient in the frequency omain into what constitutes the positive an negative part of the corresponing graient in the time-omain gives G + f = 1 N Ã( f )H à ( f ) s f, G f = 1 N Ã( f )H x f. Consequently, by applying the inverse DFT to G + f an G f the corresponing positive an negative parts of the graient in the time-omain are foun. As a result, S can be upate using multiplicative upates in the time omain, hence, enforcing non-negativity through the upate ( ) g α,n s,n = s,n g +.,n In Figure 1 we emonstrate the usefulness of the ShifNMF over regular instantaneous NMF when shifts are present in the ata Convolutive Sparse Coing The Convolutive Sparse Coing moel is given by x m,n a τ m, s,n τ.,τ

5 (a) True components (b) Estimate components NMF (c) Estimate components ShiftNMF Fig. 1. Left panel: The true factors forming the synthetic ata (X R 9 14 ). To the left, the strength of the mixing A of each source is inicate in gray color scale. In the mile, the three sources are shown an to the right is given the time elays of each source to each channel. Mile panel: Results obtaine by conventional instantaneous NMF for the synthetic ata given in figure 3. Clearly, the moel can not account for the shifts in the ata hence the sources are incorrectly estimate. Notice, only 68 % of the variance of the ata can be accounte for. Right panel: The estimate factors obtaine by a ShiftNMF analysis. Clearly, the moel with shifts has correctly recovere the components of the synthetic ata hence accounts for all the variance in the ata. Where S is sparse. The moel is separable in the frequency omain an can be optimize using the following objective of the form given in (4) C = 1 2N f x f à f s f 2 F λ log(sp(s n )) n Where the first term is the reconstruction error an secon term the sparsity penalty impose with strength λ given by the sparse prior istribution sp. We will presently consier the Laplace prior given by sp(s n ) = e s n forming a l 1 -norm regularization penalty. As for non-negativity constraints, the sparsity in the time omain as well as regions where the filter A τ is zero is not transparent in a frequency omain representation. However, the convolutive moel is efficiently estimate in a frequency omain representation. Thus, again the TFGM amits the benefits of the representations in the two omains. The graient of the least squares error in the frequency omain is given by LS ã m,, f = 1 N ( x m, f ã m,, f s, f ) s, f LS s, f = 1 N m x m,, f ( x m, f ã m,, f s, f ) Thus, the graient in the time omain is given by A τ = F 1 ( LS à ) τ S = F 1 ( S LS) Hence, by computing the graient in the time omain it becomes transparent how A τ can be estimate such that only active regions of the filter A τ are upate. Furthermore, the upate in the time omain of S enables the combination of sparseness constraint in the time omain with efficient representation in the frequency omain. A τ an S are upate using linesearch, i.e., by A τ A τ µ A A τ an S S µ S S. In Figure 2 we emonstrate the propose algorithm on an EEG-ata set escribe in [19] base on a visual paraigm. We remove the three frontal electroes EOG1, EOG2 an FPz heavily confoune by eye artifacts prior to the analysis Sparse an Smooth Matrix Factorization Smoothness constraints in the time omain correspons to reuce high frequency content. Hence smoothness can be impose by penalizing high frequency regions of the components, i.e., by consiering the following objective in the form given in (4) C = n 1 2 x n As n 2 F + λ 2 2N f w f s f 2 F. From the objective above it can be seen that smoothness oes not improve the ientifiability of the moel since multiplying the sources S by the orthogonal matrix Q result in a representation that is equally smooth, i.e. s f F = Q s f F. Thus, aitional constraints

6 Aτ S Fig. 2. Convolutive Sparse Coing analysis of EEG ata obtaine from a visual paraigm. The size of the ata is X <29 34 while Aτ <29 4 an τ [1, 2,..., 128]. Top panel: Analysis for λ =, clearly S given to the right is not sparse thus the EEG activity is moele both in the convolutive filter Aτ an in the sources S. The scalp maps to the left gives the power of the filter coefficients for the electroes of each component. The explaine variation is 91% Mile panel: Analysis base on λ = 2, clearly S has become sparse while the temporal structure of the EEG ata mainly is coe in the filter Aτ. The explaine variation is 66%. Bottom panel: When increasing the sparsity strength (λ = 7) S becomes even more sparse. The explaine variation is 3%. The activity capture by the moels are mainly the powerful alpha activity resiing in a frequency ban aroun 8-12 Hz.

7 are necessary in orer to obtain an unambiguous representation. We will here improve ientifiability of the moel by imposing sparseness on S. Again, sparsity is not transparent in a frequency representation. However, the sparsity an smoothness constraints can again be combine using the propose TFGM. Consier the following sparse an smooth matrix factorization C = 1 2 n ( x n As n 2 F + λ 1 s n 1 ) + λ 2 2N f w f s f 2 F Where w f weights frequencies accoring to the smoothness esire. Clearly, the objective has the form given in (4). Thus, the graient of the above objective is given by A = (AS X)S T S = A T (AS X) + λ 1 sign(s) + λ 2 F 1 ( S ). where, s f = w f s f. Again A an S are upate using line-search, i.e. by A A µ A A an A S µ S S. In Figure 3 we emonstrate the algorithms ability to impose smoothness on simulate ata while at the same time obtain better ientifiability through sparseness. 4. DISCUSSION AND CONCLUSION We have propose the Time Frequency Graient Metho that takes avantage of mixing representations which allows us to hanle convolutions an shifts efficiently while imposing time omain constraints. The framework can be use for objective functions that are separable in time or frequency parts such that variables can be arbitrarily upate in the time or frequency omain. We emonstrate the viability of this simple approach for moels that involve shifts, convolutions, or smoothness, but we expect the framework to be useful for a wie range of moels where the frequency representation facilitates efficient computation.. REFERENCES [1] Charles Spearman, "general intelligence," objectively etermine an measure, American Journal of Psychology, vol. 1, pp , 194. [2] H. F. Kaiser, The varimax criterion for analytic rotation in factor analysis, Psychometrica, vol. 23, pp , 198. [3] Pierre Comon, Inepenent component analysis, a new concept?, Signal Processing, vol. 36, pp , [4] Anthony J. Bell an Terrence J. Sejnowski, An information maximization approach to blin source separation an blin econvolution, Neural Computation, vol. 7, pp , 199. [] B. A. Olshausen an D. J. Fiel, Emergence of simple-cell receptive fiel properties by learning a sparse coe for natural images, Nature, vol. 381, pp , [6] R.A. Harshman, S. Hong, an M.E. Luny, Shifte factor analysis part i: Moels an properties, Journal of Chemometrics, vol. 17, pp , 23. [7] K Torkkola, Blin separation of elaye sources base on informationmaximization, Acoustics, Speech, an Signal Processing. ICASSP-96, vol. 6, pp , [8] Arie Yereor, Time-elay estimation in mixtures, ICASSP, vol., pp , 23. [9] H. Attias an C.E. Schreiner, Blin source separation an econvolution: the ynamic component analysis algorithm, Neural Computation, vol. 1, no. 6, pp , [1] L. Parra, C. Spence, an B.D. Vries, Convolutive blin source separation base on multiple ecorrelation, IEEE Workshop on Neural Networks an Signal Processing, pp , [11] J. Anemuller, Terrence J. Sejnowski, an S. Makeig, Complex inepenent component analysis of frequency-omain electroencephalographic ata, Neural Networks, vol. 16, no. 9, pp , 23. [12] T. Blumensath an M. Davies, On shiftinvariant sparse coing, International Conference on Inepenent Component Analysis an Blin Source Separation, vol. 26, pp , 24. [13] M. S. Lewicki an T. J. Sejnowski, Coing timevarying signals using sparse shift-invariant representations, Av. Neural Inform. Process. Systems (NIPS 99), vol. 11, pp , [14] Michael Peersen Syskin, Jan Larsen, Ulrik Kjems, an Lucas C. Parra, A survey of convolutive blin source separation methos, Springer Hanbook on Speech Processing an Speech Communication, 27.

8 Fig. 3. Left Panel: The nine realizations of the simulate input ata (X R 9 1 ) base on a mixture of the three sources given in the top right panel with aitive Normal istribute noise. Notice that the signal is heavily contaminate with a signal to noise ratio of b. Right Panel: The top figure show the three true unerlying sources use to simulate ata, all three components are both sparse an smooth. The mile figure shows the sources ientifie by the propose smooth an sparse matrix factorization algorithm imposing (λ 1 =.2, λ 2 =, i.e., without any smoothness regularization). The bottom figure shows how introucing smoothness regularization suppresses noise in the solution (λ 1 =.2, λ 2 = 2 for frequencies larger than.14 samples 1 ). While smoothness suppresses noise in the sources it will also introuce a bias in this particular case reucing the explaine variation from 64% to 3%. [1] D.D. Lee an H.S. Seung, Learning the parts of objects by non-negative matrix factorization, Nature, vol. 41, no. 67, pp , [16] Daniel D Lee an H. Sebastian Seung, Algorithms for non-negative matrix factorization, in NIPS, 2, pp [17] M. Mørup, K. H. Masen, an L. K. Hansen, Shifte non-negative matrix factorization, 27, pp [18] Kaare B. Petersen an Michael S. Peersen, The matrix cookbook, [19] S. Makeig, M. Westerfiel, T.P. Jung, J. Covington, J. Townsen, T.J. Sejnowski, an E. Courchesne, Functionally inepenent components of the late positive event-relate potential uring visual spatial attention, J. Neurosci., vol. 19, pp , Apr 1999.