NONNEGATIVE matrix factorization finds its application

Transcription

1 Multilicative Udates for Convolutional NMF Under -Divergence Pedro J. Villasana T., Stanislaw Gorlow, Meber, IEEE and Arvind T. Hariraan arxiv: v2 [cs.lg 5 May 208 Abstract In this letter, we generalize the convolutional NMF by taking the -divergence as the contrast function and resent the correct ultilicative udates for its factors in closed for. The new udates unify the -NMF and the convolutional NMF. We state why alost all of the existing udates are inexact and aroxiative w.r.t. the convolutional data odel. We show that our udates are stable and that their convergence erforance is consistent across the ost coon values of. Index Ters Convolution, nonnegative atrix factorization, ultilicative udate rules, -divergence. I. INTRODUCTION NONNEGATIVE atrix factorization finds its alication in the fields of achine learning and in connection with inverse robles, ostly. It becae iensely oular after Lee and Seung derived ultilicative udate rules that ade the u until then additive stes in the direction of the negative gradient obsolete [. In [2, they gave eirical evidence of their convergence to a stationary oint, using a the squared Euclidean distance, and b the generalized Kullback Leibler divergence as the contrast function. The factorization s origins can be traced back to [3, [4. A convolutional variant of the factorization is introduced in [5 based on the Kullback Leibler divergence. The ain idea is to exloit teoral deendencies in the neighborhood of a oint in the tie-frequency lane. In their original for, the udates result in a biased factorization. To rovide a reedy, ultile coefficient atrices are udated in [6, one for each translation, and the final udate is by taking the average over all coefficient atrices. A nonnegative atrix factor deconvolution in 2D based on the Kullback Leibler divergence is found in [7. Not only do the authors give a derivation of the udate rules, they show a sile way of aking the udate rules ultilicative. It ay be ointed out that the udate rule for the coefficient atrix is different fro the one in [6. The sae guiding rinciles are alied to derive the convolutional factorization based on the squared Euclidean distance in [8. But in the attet to give a foral roof for their udate rules, the authors largely reforulate a biased factorization coarable to [5. In [9, nonnegative atrix factorization is generalized to a faily of α-divergences under the constraints of sarsity and soothness, while the unconstrained -divergence is brought into focus in [0. For both cases ultilicative udate rules were given. The roerties of the -divergence are discussed in detail in [0, [. The cobined α--divergence together with the corresonding ultilicative udates can be found in [2. In this letter, we rovide ultilicative udate rules for the factorial deconvolution under the -divergence. Furtherore, we argue that the udates in [5, [6 and in [7 are eirical and/or inexact w.r.t. the convolutional data odel. According to our siulation, the udates in [8 do not signify any extra iroveent over [6 desite the additional load. Finally, we show that the exact udates are stable and that their behavior is consistent for {0,, 2}. II. NONNEGATIVE MATRIX FACTORIZATION The nonnegative atrix factorization NMF is an ubrella ter for a low-rank atrix aroxiation of the for V W H with V R K N, W R K I, and H RI N, where I is the redeterined rank of the factorization. The letters above hel distinguish between visible v and hidden variables h that are ut in relation through weights w. The factorization is usually forulated as a convex iniization roble with a dedicated cost function C according to with iniize W, H CW, H subject to w ki, h in 0 2 CW, H = LV, W H, 3 where L is a loss function that assesses the error between V and the factorization W H. A. -Divergence The loss in 3 can be exressed by eans of a contrast or distance function between the eleents of V and W H. Due to its robustness with resect to outliers for certain values of the inut araeter R, we resort to the -divergence [3 as a subclass of the Bregan divergence [, [4, which for the oints and q is given by [ + q q, 0,, d, q = log + q, =, 4 q q log, = 0. q Accordingly, the -divergence for atrices V and W H can be defined entrywise, as D V W H = def K N k= n= d v, i w ki h in, 5 which can further be viewed as the -divergence between two unnoralized arginal robability ass functions with k as the arginal variable. Note that the -divergence has a single global iniu for n v = i,n w ki h in, k, although it is strictly convex only for [, 2 [0, [.

2 2 B. Discrete Convolution As can be seen exlicitly in 5, the weight w ki for the ith hidden variable h in at oint k, n is alied using the scalar roduct. Given that h i evolves with n, we can assue that h i is correlated with its ast and future states. We can take this into account by extending the dot roduct to a convolution in our odel. Postulating causality and letting the weights have finite suort of cardinality M, the convolution writes M un = w ki h i n = def w ki h i,n. 6 =0 The oeration can be converted to a atrix ultilication by lining u the states h in in a truncated Toelitz atrix: h i, h i,2 h i,3 h i,n h i,n 0 h i, h i,2 h i,n 2 h i,n h i,n M h i,n M+ In accordance with, the convolutional NMF CNMF can be forulated as follows to accoodate the structure given in 7, see also [5, [6: V U = M =0 W H, 8 where is a colunwise right-shift oeration siilar to a logical shift in rograing languages that shifts all the coluns of H by ositions to the right, and fills the vacant ositions with zeros. The oeration is size-reserving. It can be seen that the CNMF has M ties as any weights as the regular NMF, whereas the nuber of hidden states is equal. III. -CNMF Ensuing fro the reliinary considerations in Section II, we adot the forulation of the CNMF fro 8 and derive ultilicative udate rules for gradient descent, while taking the entrywise -divergence fro 5 as the loss function. The result is referred to as the -CNMF. A. Proble Stateent Under the reise that V is factorizable into {W } and H, = 0,,..., M, and given the cost function C{W }, H = D V U, 9 we seek to find the ultilicative equivalents of the iterative udate rules for gradient descent: W t+ = W t κ H t+ = H t µ W t C { W t }, H t, 0a H t C{ W t }, H t, 0b where 0a and 0b alternate at each iteration t 0. The ste sizes κ and µ are allowed to change at every iteration. B. Multilicative Udates Rules Couting the artial derivatives of C w.r.t. W and H, and by choosing aroriate values for κ and µ, the iterative udate rules fro 0 becoe ultilicative in the for W t+ = W t [U t H tt [ V U t 2 H tt, H t+ = H t [ WtT Ũt [ WtT V for = 0,,..., M, with Ũt 2 a, b Ũ t = Wt+ H t, 2 where denotes the Hadaard, i.e. entrywise, roduct, is equivalent to entrywise exonentiation and, resectively, stands for the entrywise inverse. The oerator is the leftshift counterart of the right-shift oerator. The details of the derivation of can be found in the Aendix. C. Relation to Previous Works Several ultilicative udates for the CNMF can be found in the existing literature using different loss functions. In [5, the loss function is stated as LV, U = K N v log v 2 v + u u 3 k= n= and the corresonding udate rules for W and H are [ W t+ = W t [ H tt V U t H tt, 4a [ [ H t+ = H t W tt W t T V Ũt, 4b [ where the -atrix is of size K-by-N with =. At t, for every, H is udated first using W t and subsequently W t+ is couted fro the udated H, or vice versa. In [5 it is entioned and ore exlicitly stated in [6 that to avoid bias in H towards W M it is best to first udate all {W } using H t and to udate H according to H t+ = M M =0 W t+t H t [W t+t [ V Ũt. 5 Coaring 4 and 5 with, it can be noted that both udate rules have the sae factors for =. The resective loss function in this case is the generalized Kullback Leibler divergence D V U and not 3. Moreover, the -atrix in 4b is not aligned with the U-atrix. Given that =, 4 and are identical for M =, i.e. when the NMF is nonconvolutional. For M >, 4 is equal to the udates in [2 for D V W H for different W -atrices where the H-atrix is tie-aligned via. Eqs. 5 and b are also different for M > because unlike b 4b is the udate derived fro a nonconvolutional odel. H in 5 brings out

3 VILLASANA et al.: -CNMF 3 the central tendency of the eleents in H but does not ake the factorization in the original loss convolutional. For all the reasons given above, the udates in 4b and 5 are at best an aroxiation of the udate in b for =. In [7, ultilicative udates are given for a CNMF in 2D tie and frequency with the generalized Kullback Leibler divergence and the squared Euclidean distance as the loss or cost function. In the diension of tie, the udates are very uch the sae as the udates in for = 2. For =, there is the inor difference that the -atrix is not aligned with the U-atrix, just like in in 4b. Other ultilicative udates for a CNMF with the squared Euclidean distance can be found in [8. In essence, they are derived in the exact sae anner as the udates in 4, but for a different loss function, which is D 2 V U. Thus, the udates are equal to the ones in [2 with W = W, U as in 8 and H tie-aligned. For the sae reasons that the udates in 4b and 5 are aroxiative of the udate in b for =, the udates in [8 are aroxiative of the udate in b for = 2. Beyond, 2 is udated ore efficiently as Ũ t = Ũt + W t+ W t H t 6 in between the udates of W and W +. D. Interretation The two udate rules given in are of significant value because, aart fro being exact: They are ultilicative, and thus, they converge fast and are easy to ileent. Eq. a extends the udate rule of the -NMF for W to a set of M weight atrices that are linked through a convolution oeration. It also extends the corresonding udate rule of the existing convolutional NMFs with the squared Euclidean distance or the generalized Kullback Leibler divergence to the faily of -divergences. Eq. b is even ore iortant, as it yields a colete udate of the hidden states at every iteration taking all M weight atrices into account at once. The udate rule in b can be viewed as an equivalent to a descent in the direction of the Reynolds gradient, which is to take the average over the artial derivatives under the grou of tie translations in. The average oerator reduces the gradient sread as a function of W at each iteration t such that the loss function converges to a single oint that is ost likely. The -NMF being referred to here is the heuristic -NMF derived in [9. It was shown in [0 to converge faster than a coutationally equivalent axiization-iniization MM algorith for [, 2 and equally fast in the oosite case. The heuristic udates were roven to converge for [0, 2, which is the interval of ractical value. IV. SIMULATION In this section, we coare our roosed udates with the existing ones in ters of convergence behavior and run tie TABLE I AVERAGE RUN TIME OF THE EXISTING CONVOLUTIONAL UPDATES RELATIVE TO THE PROPOSED UPDATES FOR DIFFERENT BETAS Biased Saragdis Schidt et al. Wang et al. Average = = = for 000 iterations. To that end, we generate 00 distinct V- atrices fro M = 6 χ 2 -distributed W -atrices, 2 w ki = wki 2 χ 2 2 w ki N 0,, 7 = and a uniforly distributed H-atrix, h in U0,. 8 The factorizations are reeated with 0 rando initializations of {W} t0 and H t0 with non-zero entries. The results shown in Fig. thus are couted over ensebles of 000 losses at each iteration. The nuber of visible and hidden variables is K = 000 and I = 0, and the nuber of realizations tie sales is N = 00. The run tie was easured on an Intel Xeon E v3 CPU at 3.5 GHz with 6 GB of RAM. In Table I, the figures reresent the averages over 000 runs ut into relation to the average run tie of the roosed udates for 000 iterations. In Fig. it can be seen that the biased udates are clearly least stable under the divergence for which they were eant in the first lace =. Already in [7, convergence issues with these udates were reorted. For {0, 2} at less than 00 iterations they can converge faster because H and U are udated M ties er iteration, which exlains the significant increase in run tie. Between 00 and 000 iterations, other udates show better erforance. Wang s udates are siilar in erforance to Saragdis average udates in site of the additional interediate udates of U for =, and slightly worse otherwise. As stated above, Schidt s udates are the sae as ours for, and so is their behavior. For =, our udates show the sallest variance overall and yield the the lowest cost below 00 iterations, which is a tyical uer bound in ractice. The longer run tie is due to the shifting of the -atrix. For {0, 2}, the loss distributions of our, Wang s, and Saragdis average udates look like they have the sae ean but a slightly different standard deviation. To test the hyothesis that the costs are statistically equivalent in resect of the ean, we eloy Welch s t-test. The -values suggest that the null hyothesis can be rejected alost surely for = 0, whereas for = 2 in general it cannot. V. CONCLUSION To the best of our owledge, our letter is the only one to rovide a colete and exact derivation of the ultilicative udates for the convolutional NMF. Above, the cost function is generalized to the faily of -divergences. It is shown by

4 4 Cost Cost Welch s t-test = 0 = Villasana et al. Saragdis biased Saragdis average Schidt et al. Wang et al. = Fig.. Siulation results including the ean and the standard deviation of the loss distribution and the -value fro Welch s t-test for the hyothesis that the roosed and the existing udate rules on average have the sae cost and eventually converge to the sae iniu as a function of iteration. siulation that the udates are stable and that their behavior is consistent for {0,, 2}. APPENDIX Let u = i, w ki h i,n and U = [u R K N. Then, for any {, 2,..., K}, q {, 2,..., I}, and r {0,,..., M }: w q r C{W }, H = w q r = u n n Choosing κ fro 0a as leads to the first udate rule w t+ ki = wki t n u n v n u 2 n v n u n hq,n r. 9 w q r κ =, 20 n u n h q,n r n v u t 2 n ut ht i,n ht i,n. 2 Further, for any {, 2,..., I} and q {, 2,..., N}: h q C{W }, H = h q k,n It is straightforward to show that u v u. 22 h q u = w k n q 23 by setting n = q = n q. As a result, lugging in q + for n in 22 and using 23, we finally obtain h q C{W }, H = k, w k Choosing µ fro 0b as µ = u k,q+ v k,q+ u 2 k,q+. 24 h q k, w k u, 25 k,q+ leads to the second udate rule k, wt ki v k,n+ u t 2 h t+ in = ht in k, wt ki ut k,n+ k,n+. 26

5 VILLASANA et al.: -CNMF 5 REFERENCES [ D. D. Lee and H. S. Seung, Learning the arts of objects by nonnegative atrix factorization, Nature, vol. 40, , 999. [2, Algoriths for non-negative atrix factorization, in Advances in Neural Inforation Processing Systes, 200, [3 P. Paatero and U. Taer, Positive atrix factorization: A non-negative factor odel with otial utilization of error estiates of data values, Environetrics, vol. 5, no. 2,. 26, 994. [4 P. Paatero, Least squares forulation of robust non-negative factor analysis, Cheoetrics and Intelligent Laboratory Systes, vol. 37, no., , 997. [5 P. Saragdis, Non-negative atrix factor deconvolution; extraction of ultile sound sources fro onohonic inuts, in Indeendent Coonent Analysis and Blind Signal Searation, 2004, [6, Convolutive seech bases and their alication to suervised seech searation, IEEE Transactions on Audio, Seech, and Language Processing, vol. 5, no.,. 2, [7 M. N. Schidt and M. Møru, Nonnegative atrix factor 2-D deconvolution for blind single channel source searation, in Indeendent Coonent Analysis and Blind Signal Searation, 2006, [8 W. Wang, A. Cichocki, and J. A. Chabers, A ultilicative algorith for convolutive non-negative atrix factorization based on squared Euclidean distance, IEEE Transactions on Signal Processing, vol. 57, no. 7, , [9 A. Cichocki, R. Zdunek, and S. Aari, Csiszár s divergences for nonnegative atrix factorization: Faily of new algoriths, in Indeendent Coonent Analysis and Blind Signal Searation, 2006, [0 C. Févotte and J. Idier, Algoriths for nonnegative atrix factorization with the -divergence, Neural Coutation, vol. 23, no. 9, , 20. [ A. Cichocki and S.-i. Aari, Failies of alha- beta- and gaadivergences: Flexible and robust easures of siilarities, Entroy, vol. 2, no. 6, , 200. [2 A. Cichocki, S. Cruces, and S.-i. Aari, Generalized alha-beta divergences and their alication to robust nonnegative atrix factorization, Entroy, vol. 3, no., , 20. [3 A. Basu, I. R. Harris, N. L. Hjort, and M. C. Jones, Robust and efficient estiation by iniising a density ower divergence, Bioetrika, vol. 85, no. 3, , 998. [4 L. M. Bregan, The relaxation ethod of finding the coon oint of convex sets and its alication to the solution of robles in convex rograing, USSR Coutational Matheatics and Matheatical Physics, vol. 7, no. 3, , 967.