L P -NORM NON-NEGATIVE MATRIX FACTORIZATION AND ITS APPLICATION TO SINGING VOICE ENHANCEMENT. Tomohiko Nakamura and Hirokazu Kameoka,

Size: px

Start display at page:

Download "L P -NORM NON-NEGATIVE MATRIX FACTORIZATION AND ITS APPLICATION TO SINGING VOICE ENHANCEMENT. Tomohiko Nakamura and Hirokazu Kameoka,"

Vernon Perkins
5 years ago
Views:

1 L P -NORM NON-NEGATIVE MATRIX FACTORIZATION AND ITS APPLICATION TO SINGING VOICE ENHANCEMENT Tomohio Naamura and Hiroazu Kameoa, Graduate School of Information Science and Technology, The University of Toyo. 7-3-, Hongo, Bunyo-u, Toyo, , Jaan. NTT Communication Science Laboratories, Nion Telegrah and Telehone Cororation. 3-, Morinosato Waamiya, Atsugi, Kanagawa, 43-98, Jaan Tomohio_Naamura@ic.i.u-toyo.ac.j,ameoa.hiroazu@lab.ntt.co.j ABSTRACT Measures of sarsity are useful in many asects of audio signal rocessing including seech enhancement, audio coding and singing voice enhancement, and the well-nown method for these alications is non-negative matrix factorization (NMF), which decomoses a non-negative data matrix into two non-negative matrices. Although revious studies on NMF have focused on the sarsity of the two matrices, the sarsity of reconstruction errors between a data matrix and the two matrices is also imortant, since designing the sarsity is equivalent to assuming the nature of the errors. We roose a new NMF technique, which we called L -norm NMF, that minimizes the L norm of the reconstruction errors, and derive a comutationally efficient algorithm for L -norm NMF according to an auxiliary function rincile. This algorithm can be generalized for the factorization of a real-valued matrix into the roduct of two real-valued matrices. We aly the algorithm to singing voice enhancement and show that adequately selecting imroves the enhancement. Index Terms Non-negative matrix factorization, L norm, auxiliary function rincile. INTRODUCTION Non-negative matrix factorization (NMF) [] is a owerful technique that aroximates a data matrix Y by using the roduct of two nonnegative matrices W and H. NMF has been actively studied in many scientific and engineering fields in recent years (see []). In articular, in the field of music signal rocessing, successful results were obtained by regarding a magnitude sectrogram as a non-negative matrix [3]. NMF is formulated as the roblem of minimizing a measure between a data matrix and a model. One standard measure is the Frobenius norm, and it corresonds to assuming that the reconstruction errors are additive Gaussian noise. While NMF with the Frobenius norm wors well for small errors, NMF erformance deteriorates for large errors such as sie noise, even if there are a few entries with errors. We thus need to exlore aroriate measures for large errors. Measures of sarsity (e.g. L norm) have been widely used in many audio signal rocessing techniques such as NMF and robust rincile comonent analysis [4, 5]. Many revious studies of NMF have used sarseness measures [6, 7], regularizers [8] and riors [9] This wor was artly suorted by JSPS Grant-in-Aid for Young Scientists B Grant Number 673. on W and H, which induce sarse solutions. However, it is difficult to use the measures between a data matrix and a model directly since the measures are often non-linear and not differentiable, and it is intractable to solve the NMF roblem. In this aer, we roose a new NMF technique, which we called L -norm NMF, that minimizes the L norm of reconstruction errors between a data matrix and a model. We formulate L -norm NMF with < and derive a convergence-guaranteed algorithm that consists of multilicative udate equations for W and H based on the auxiliary function rincile [, ]. The constant controls the sarsity of the reconstruction errors. To examine the effect of varying, we aly the roosed algorithm to singing voice enhancement for monaural audio signals. We henceforth denote sets of real values and non-negative real values as R and R, resectively.. L P -NORM NON-NEGATIVE MATRIX FACTORIZATION.. Problem setting Let us define frequency, time, and basis indexes, resectively, as ω [, Ω ], t [, T ] and [, K ] such that K < Ω and K < T. Given a non-negative data matrix Y := {y } ω Ω, t T, L -norm NMF is the roblem of minimizing the L norm L(W, H) := Y WH = y w,t w ω, h,t () subject to, W ω, =. () ω Here W = {w ω, } ω, is a Ω K non-negative matrix and H = {h,t },t is a K T non-negative matrix. Eq. () is introduced to avoid an indeterminacy in the scaling. When Y is a magnitude sectrogram, the columns of W reresent sectral temlates and the rows of H reresent the temoral activities of the sectral temlates. The constant controls the sarsity of the reconstruction error. A smaller induces the sarsity, and most of the entries of the estimated data matrix WH are the same as those of Y, while the other entries differ greatly from those of Y. On the other hand, a larger does not induce the sarsity, and all the entries of the estimated data matrix may be non-zero. We thus assume that the L norm satisfies < <, which romotes sarsity. Note that when =, the objective function equals the NMF with Frobenius norm /5/$3. 5 IEEE 5 ICASSP 5

2 .. Efficient algorithm based on auxiliary function rincile Since L(W, H) involves a summation over in the L norm, it is intractable to solve the current minimization roblem analytically. However, we can develo a comutationally efficient algorithm to find a locally otimal solution based on the auxiliary function rincile [, ]. When alying an auxiliary function rincile to the minimization roblem, the first ste is to define an uer bound function for the objective function. Introducing auxiliary variables ξ = {ξ R }, we derive y w,t w ω, h,t ξ y w,t w ω, h,t + ( )ξ. (3) The roof of the inequality is described in Lemma of []. The equality of Eq. (3) holds if and only if ξ = y w h,t. (4) The term in the square function of Eq. (3) includes a summation over, and we further derive the uer bound of the right-hand side of Eq. (3). Since the square function is a convex function, we can invoe Jensen s inequality: ( ) w ω, h,t (w ω, h,t ) (5) λ, where λ = {λ, R }, are auxiliary variables that sum to unity: λ, =. The equality of Eq. (5) holds if and only if λ, = w ω, h,t w ω, h. (6),t The uer bound of L(W, H) can thus be described as L + (W, H, λ, ξ) { = y w,t y ξ w ω, h,t + λ, (w ω, h,t ) } + ( )ξ. (7) We can derive udate equations for the arameters, using the above uer bound. By setting the artial derivative of L + (W, H, ξ, λ) with resect to W and H at zero and using Eqs. (4) and (6), we obtain W W [{(Y C)H } {(WH C)H }] (8) H H [{W (Y C)} {W (WH C)}] (9) where, denote element-wise roduct and division, and C is an Ω T matrix, whose (ω, t)-th element is c = y w ω, h,t. () It is worth noting that each udate equation consists of multilication by a non-negative factor. Hence, it is guaranteed that the entries of W and H are always non-negative when their initial values are set at non-negative values. 3. L P -NORM MATRIX FACTORIZATION 3.. Problem setting Although the algorithm in the revious section is for non-negative matrices, it can be generalized for real-valued matrices, according to the auxiliary function rincile. We call the roblem L -norm matrix factorization (L -norm MF). L -norm MF is the roblem of finding real-valued matrices W, H for a given real-valued data matrix Y such that min J(W, H) = Y WH, W R Ω K,H R K T subject to, W ω, = where < < and J(W, H) is the objective function. 3.. Iterative algorithm based on auxiliary function rincile As in Sec., J(W, H) involves summation over in the L norm, and we consider the uer bound of J(W, H) to aly the auxiliary function rincile to the intractable minimization roblem. The inequality used in Eq. (3) is alicable to J(W, H), and the uer bound of J(W, H) is the same as the right-hand side of Eq. (3). To derive the uer bound of the square function of Eq. (3), we can use the generalization of Jensen s inequality for y, w ω,, h,t R, which was emloyed in []: α, w ω, h,t y w ω, h,t () β, where α = {α, R},, β = {β, [, ]}, are auxiliary variables subject to α, = y, β, =. The equality of Eq. () holds if and only if ( ) α, = w ω, h,t β, w ω, h,t y. () The uer bound of J + (W, H) can thus be described as J + α, w ω, h,t (W, H, ξ, α, β) = + ( )ξ. ξ ω β, (3) By differentiating J + (W, H, ξ, α, β) artially with resect to W and H and setting them at zero, we can obtain the udate equations: ( t c h,t β, w ω, w ω,h,t + y w ω, h ),t (4) h,t ω c w ω, ( β where c is defined as Eq. () Relation to L -norm NMF t c β, h,t, w ω,h,t + y w ω, h,t ω c β, w ω, ) (5) The arameters β can be chosen arbitrarily subject to β, R and β ω,,t =. Choosing β as in [], we obtain the multilicative udate equations as in Eqs. (8) and (9). Therefore, we can say that the algorithm of L NMF is a secial case of that of L MF. 4. APPLICATION TO SINGING VOICE ENHANCEMENT 4.. Singing voice enhancement In this section, we aly L -norm NMF to singing voice enhancement whose aim is to extract a singing voice from a monaural mixed audio signal consisting of vocal and accomaniment arts. Singing voice enhancement is often used in music information retrieval (MIR) alications such as automatic lyrics recognition [3, 4], automatic singer identification [5], and automatic araoe generators [6]. 6

When inut audio signals are multichannel, we can use satial cues, based on the fact that the vocal arts of music audio signals are frequently mixed in the center of the sound field.

3 When inut audio signals are multichannel, we can use satial cues, based on the fact that the vocal arts of music audio signals are frequently mixed in the center of the sound field. However, for monaural inuts, we need other cues instead of the satial cues. 4.. Enhancement algorithm We utilize the fact that there is a difference in sectrogram between accomaniments and singing voices. Accomaniment sounds are generated by musical instruments, which reroduce aroximately the same sounds every time they are layed. We can see the sectrograms of accomaniment signals as a low-ran matrix. In constrast, singers fluctuate their singing voices to obtain musical effects such as vibrato. The sectrograms of singing voices are relatively high ran and sarse, and corresonds to reconstructions errors of L -norm NMF. As mentioned in Sec.., the model sectrogram WH of L -norm NMF is a non-negative matrix with the ran of K, and small induces sarsity of the reconstruction errors. Setting K and at sufficiently small values, we exect the low-ran matrix WH to contain accomaniment signals and the reconstruction errors Y WH to contain singing voice signals. Let the estimated magnitude sectrogram of a singing voice be Ŝ = {ŝ } ω Ω, t T. First, an inut signal containing a singing voice and accomaniment is converted into a comlex sectrogram with the short-time Fourier transform (STFT). We regard the magnitude sectrogram of the inut signal as a non-negative matrix, and aly L -norm NMF to the magnitude sectrogram. The obtained model sectrogram WH should corresond to the accomaniment, and the reconstruction errors between the observed magnitude sectrogram and the model sectrogram corresonds to the singing voice. The time-frequency comonents of the model sectrogram may be larger than those of the observed sectrogram, and we derive Ŝ as y w ω, h,t (y w ω, h,t ) ŝ = (y < w ω, h,t ). (6) The estimated magnitude sectrogram with the hase of the original comlex sectrogram is converted into an audio signal by the inverse STFT. 5. EXPERIMENTAL EVALUATION 5.. Exerimental conditions To evaluate the erformance of the roosed method, we conducted two exeriments on singing voice enhancement: an evaluation of the effect of, which controls sarsity, and frame length F, and a comarison of our results with the state-of-the-art [5, 7, 8]. The criteria for evaluating the singing voice enhancement were the normalized signal-to-distortion ratio (NSDR) and the global NSDR (GNSDR), given as NSDR({ fˆ t } t ; { f t } t, {x t } t ) = SDR({ fˆ t } t, { f t } t ) SDR({x t } t, { f t } t ), (7) i w i NSDR({ fˆ i,t } t ; { f i,t } t, {x i,t } t ) GNSDR =, (8) i w i SDR({ ˆ f t } t, { f t } t ) = log ( t t ˆ f t f t )( fˆ ) t t f t t, (9) fˆ t f t where fˆ i,t, f i,t and x i,t denote the estimated signal, the target signal and the inut signal of the i-th iece. NSDR reresents the imrovement in SDR, and GNSDR denotes the weighted averages of Frequency [Hz] Frequency [Hz] Time [s] (a) Sectrogram of inut audio signal Time [s] (b) Sectrogram of audio signal after enhancement. Fig.. Sectrogram examles of (a) a mixed audio signal and (b) the singing-voice-enhanced audio signal obtained with the roosed method. the NSDR of all the music ieces by the length of the i-th iece, {w i } i. These criteria have also been emloyed in many revious studies [5, 7 ]. To calculate the SDR, we used the BSS Eval Toolbox [, 3]. As an evaluation dataset, we used the MIR-K dataset [4], following the evaluation framewor in [5, 7, 8]. The dataset consists of Chinese song clis erformed by amateur singers. The durations of the clis range from 4 to 3 s, and the audio signals are monaural with a samling rate of 6 Hz. The accomaniment and vocal arts were recorded searately, and we could mix them with any signal-to-noise ratio (SNR), where the SNR corresonds to the voice to accomaniment ratio. The accomaniment and vocal arts for each cli were mixed at 5 db (accomaniment is louder), db (same level) and 5 db (vocal is louder) SNRs. 5.. Effect of sarsity and frame lengths We first comared the roosed method in and F. We used =.,.,,. and F = 5, 4, 48, 496 samle oints. For STFT, the window function was the sine window, and the frame shifts were half the length of the frames. The number of bases was set at K =. The entries of W and H were initialized randomly. There were iterations, which is suosed to be sufficient emirically. Fig. shows one of the enhanced results obtained with the roosed method. The figures show the sectrograms of the inut signal and the enhanced result. We can see that most of the accomanying sounds (vertically and horizontally smooth comonents) are suressed, and the singing voice comonent of the sectrogram is clearer than that of the inut sectrogram. 7

4 Global normalized frame: 5 frame: 4 frame: 48 frame: Global normalized frame: 5 frame: 4 frame: 48 frame: Global normalized frame: 5 frame: 4 frame: 48 frame: (a) GNSDR at 5 db SNR. (b) GNSDR at db SNR. (c) GNSDR at 5 db SNR. Fig.. Global normalized signal-to-distortion ratio (GNSDR) of the roosed method with resect to of the L norm and frame length F. Red, blue, green and urle oints corresond to F = 5, 4, 48, 496 samle oints. The results are for (a) 5 db, (b) db, and (c) 5 db SNRs. Table. Global normalized signal-to-distortion ratio (GNSDR) comarison with revious studies. F denotes the length of a frame in samle oint. Hsu, Rafii and Huang reresent the corresonding methods [5, 7, 8]. Inut Proosed method Proosed method SNR [db] F = 4 F = 48 Hsu [7] Rafii [8] Huang [5] 5.84 ( =.6) 3.7 ( =.7) ( =.).95 ( =.) ( =.8).4 ( =.8) Normalized signal to distortion ration [db] Inut signal to noise ration [db] Fig. 3. Box lot of the normalized signal-to-distortion ratio (NSDR) of the roosed method for the MIR-K dataset. The results for SNRs of 5, and 5 db are for (, F) = (.7, 48), (., 48) and (.8, 4), resectively. As illustrated in Fig. for SNRs of 5,, 5 db, the results show that the GNSDRs deended strongly on for all frame lengths. The highest GNSDRs for all SNRs were 3.7 at (, F) = (.7, 48) for 5 db SNR,.95 at (, F) = (., 48) for db SNR, and.43 at (, F) = (.8, 4) for 5 db SNR. We can see that at which GNSDR was the highest for each inut SNR decreased as the inut SNR became higher. With a high inut SNR, the non-zero timefrequency comonents of the singing voice sectrogram are large, and increasing the sarsity is referred. On the other hand, with a low inut SNR, the non-zero time-frequency comonents are small, and decreasing the sarsity is referred. The obtained results are consistent with this idea. Fig. 3 shows the distributions of NSDRs for each SNR. The results were for (, F) = (.7, 48), (., 48), (.8, 4) for SNRs of 5,, 5 db. Since most of the NSDRs exceeded db, and we can confirm that the roosed method wored well for most of the inut signals Comarison with revious studies Finally, we comared the roosed method with the state-of-theart [5, 7, 8]. The results are summarized in Tab.. The roosed method with F = 4 outerformed two revious methods for all inut SNRs. While the GNSDRs of the roosed method were lower than those of [5] at SNRs of and 5 db, the GNSDR with the roosed method was.4 to.4 db larger than those of three revious methods at a SNR of 5 db. This result indicates that the roosed method wors well articularly in a low SNR environment. 6. CONCLUSION We roosed a new NMF that minimizes the L norm of the reconstruction errors between a data matrix and the model. A comutationally efficient algorithm was derived according to the auxiliary function rincile, and it has multilicative udate equations, which guarantee the non-negativity of W and H. The algorithm of L -norm NMF can be generalized for that of L -norm MF for real-valued matrices. We alied L -norm NMF to singing voice enhancement and showed by exeriments that adequately selecting imroves the enhancement quality, and the roosed method outerformed three revious wors under a low SNR situation. There are several ways to extend L -norm NMF to other alications. One romising alication is seech enhancement, since the sectrogram of bacground noise is sometimes aroximated as low ran and the seech sectrogram is relatively sarse. 8

5 7. REFERENCES [] D. D. Lee and H. S. Seung, Learning the arts of objects by non-negative matrix factorization, Nature, vol. 4, no. 6755, , 999. [] Y.-X. Wang and Y.-J. Zhang, Nonnegative matrix factorization: A comrehensive review, IEEE Trans. Knowl. Data Eng., vol. 5, no. 6, , 3. [3] P. Smaragdis and J. C. Brown, Non-negative matrix factorization for olyhonic music transcrition, in Proc. IEEE Worsho Alications Signal Process. Audio Acoust. IEEE, 3, [4] J. Wright, A. Ganesh, S. Rao, Y. Peng, and Y. Ma, Robust rincial comonent analysis: Exact recovery of corruted low-ran matrices via convex otimization, in Proc. Adv. Neural Inf. Process. Syst., 9, [5] P.-S. Huang, S. D. Chen, P. Smaragdis, and M. Hasegawa- Johnson, Singing-voice searation from monaural recordings using robust rincial comonent analysis, in Proc. Int. Conf. Acoust. Seech Signal Process.,, [6] P. O. Hoyer, Non-negative sarse coding, in IEEE Worsho Neural Networs for Signal Process.,, [7] P. O. Hoyer, Non-negative matrix factorization with sarseness constraints, J. Mach. Learn. Res., vol. 5, , 4. [8] B. Shen, L. Si, R. Ji, and B.-D. Liu, Robust nonnegative matrix factorization via l norm regularization, CoRR, vol. abs/4.3,. [9] T. Virtanen, A. T. Cemgil, and S. Godsill, Bayesian extensions to non-negative matrix factorisation for audio signal modelling, in Proc. Int. Conf. Acoust. Seech Signal Process., Mar. 8, [] J. M. Ortega and W. C. Rheinboldt, Iterative solution of nonlinear equations in several variables, Number 3.. [] H. Kameoa, M. Goto, and S. Sagayama, Selective amlifier of eriodic and non-eriodic comonents in concurrent audio signals with sectral control envelos, in Proc. SIG Tech. Reorts on Music and Comuter of IPSJ, Aug. 6, vol. 6- MUS-66, , in Jaanese. [] H. Kameoa, N. Ono, K. Kashino, and S. Sagayama, Comlex NMF: A new sarse reresentation for acoustic signals, in Proc. Int. Conf. Acoust. Seech Signal Process., 9, [3] M. Suzui, T. Hosoya, A. Ito, and S. Maino, Music information retrieval from a singing voice using lyrics and melody information, EURASIP J. Alied Signal Process., vol. 7, no.,. 5 5, 7. [4] A. Mesaros and T. Virtanen, Automatic recognition of lyrics in singing, EURASIP J. Audio, Seech, and Music Process., vol., no. 4,. 7,. [5] H. Fujihara, T. Kitahara, M. Goto, K. Komatani, T. Ogata, and H. G. Ouno, Singer identification based on accomaniment sound reduction and reliable frame selection, in Proc. Int. Symosium Music Info. Retrieval, 5, [6] M. Ryynänen, M. Virtanen, J. Paulus, and A. Klauri, Accomaniment searation and araoe alication based on automatic melody transcrition, in Proc. IEEE Int. Conf. Multimedia Exo, 8, [7] C.-L. Hsu and J.-S. R. Jang, On the imrovement of singing voice searation for monaural recordings using the mir- dataset, IEEE Trans. Acoust., Seech, and Language Process., vol. 8, no.,. 3 39,. [8] Z. Rafii and B. Pardo, A simle music/voice searation method based on the extraction of the reeating musical structure, in Proc. Int. Conf. Acoust. Seech Signal Process.,,. 4. [9] A. Ozerov, P. Philie, R. Gribonval, and F. Bimbot, One microhone singing voice searation using source-adated models, in Proc. IEEE Worsho Alications Signal Process. Audio Acoust., 5, [] H. Tachibana, N. Ono, and S. Sagayama, Singing voice enhancement in monaural music signals based on two-stage harmonic/ercussive sound searation on multile resolution sectrograms, IEEE/ACM Trans. Audio, Seech and Language Process., vol., no.,. 8 37, 4. [] Z. Rafii, Z. Duan, and B. Pardo, Combining rhythm-based and itch-based methods for bacground and melody searation, IEEE/ACM Trans. Audio, Seech and Language Process., vol., no., , 4. [] E. Vincent, R. Gribonval, and C. Févotte, Performance measurement in blind audio source searation, IEEE Trans. Acoust., Seech, and Language Process., vol. 4, no. 4, , 6. [3] Bss eval, htt://bass-db.gforge.inria.fr/bss_ eval/. [4] Mir- dataset, htts://sites.google.com/site/ unvoicedsoundsearation/mir-. 9

MULTI-RESOLUTION SIGNAL DECOMPOSITION WITH TIME-DOMAIN SPECTROGRAM FACTORIZATION. Hirokazu Kameoka

MULTI-RESOLUTION SIGNAL DECOMPOSITION WITH TIME-DOMAIN SPECTROGRAM FACTORIZATION Hiroazu Kameoa The University of Toyo / Nippon Telegraph and Telephone Corporation ABSTRACT This paper proposes a novel