Phase Reconstruction with Learned Time-Frequency Representations for Single-Channel Speech Separation

Transcription

1 MITSUBISHI ELECTRIC RESEARCH LABORATORIES Phase Reonstrution with Learned Time-Frequeny Representations for Single-Channel Speeh Separation Wihern, G.; Le Roux, J. TR September 26, 2018 Abstrat Progress in solving the oktail party problem, i.e., separating the speeh from multiple overlapping speakers, has reently aelerated with the invention of tehniques suh as deep lustering and permutation free mask inferene. These approahes typially fous on estimating target STFT magnitudes and ignore problems of phase inonsisteny. In this paper, we expliitly integrate phase reonstrution into our separation algorithm using a loss funtion defined on timedomain signals. A deep neural network struture is defined by unfolding a phase reonstrution algorithm and treating eah iteration as a layer in our network. Furthermore, instead of using fixed STFT/iSTFT time-frequeny representations, we allow our network to learn a modified version of these representations from data. We ompare several variants of these unfolded phase reonstrution networks ahieving state of the art results on the publily available wsj0-2mix dataset, and show improved performane when the STFT/iSTFT-like representations are allowed to adapt. International Workshop on Aousti Signal Enhanement (IWAENC) This work may not be opied or reprodued in whole or in part for any ommerial purpose. Permission to opy in whole or in part without payment of fee is granted for nonprofit eduational and researh purposes provided that all suh whole or partial opies inlude the following: a notie that suh opying is by permission of Mitsubishi Eletri Researh Laboratories, In.; an aknowledgment of the authors and individual ontributions to the work; and all appliable portions of the opyright notie. Copying, reprodution, or republishing for any other purpose shall require a liense with payment of fee to Mitsubishi Eletri Researh Laboratories, In. All rights reserved. Copyright Mitsubishi Eletri Researh Laboratories, In., Broadway, Cambridge, Massahusetts 02139

2

3 PHASE RECONSTRUCTION WITH LEARNED TIME-FREQUENCY REPRESENTATIONS FOR SINGLE-CHANNEL SPEECH SEPARATION Gordon Wihern Jonathan Le Roux Mitsubishi Eletri Researh Laboratories (MERL), Cambridge, MA, USA ABSTRACT Progress in solving the oktail party problem, i.e., separating the speeh from multiple overlapping speakers, has reently aelerated with the invention of tehniques suh as deep lustering and permutation free mask inferene. These approahes typially fous on estimating target STFT magnitudes and ignore problems of phase inonsisteny. In this paper, we expliitly integrate phase reonstrution into our separation algorithm using a loss funtion defined on timedomain signals. A deep neural network struture is defined by unfolding a phase reonstrution algorithm and treating eah iteration as a layer in our network. Furthermore, instead of using fixed STFT/iSTFT time-frequeny representations, we allow our network to learn a modified version of these representations from data. We ompare several variants of these unfolded phase reonstrution networks ahieving state of the art results on the publily available wsj0-2mix dataset, and show improved performane when the STFT/iSTFT-like representations are allowed to adapt. Index Terms Soure separation, iterative phase reonstrution, learnable representations 1. INTRODUCTION Progress in separating multiple overlapping speakers using a single mirophone, often referred to as the oktail party problem, has aelerated greatly with the advent of deep neural network based tehniques [1]. In partiular, disriminative time-frequeny masking tehniques suh as deep lustering [2], permutation-free mask inferene [2, 3, 4], and their ombination under the himera framework [5, 6] have signifiantly advaned the state of the art. These methods only modify magnitude spetrograms, and use the mixture phase to synthesize the separated speeh via a simple inverse shorttime Fourier transform (istft). As magnitude proessing improves and begins to approah orale performane, i.e., the upper bound of soure separation performane using the noisy phase, interest in proessing phase for soure separation has inreased. Reent proposed approahes inlude avoiding phase proessing by using time domain waveforms as the input/output of learned separation networks [7, 8, 9], or estimating omplex (real + imaginary) time-frequeny masks [10]. However, at the present time, tehniques operating on magnitude spetrograms are the state of the art in speaker separation and other audio soure separation tasks, e.g., speeh enhanement [11] and musi separation [12]. Thus, an approah that inorporates phase proessing into these magnitude estimation networks would be highly valuable. When ombined with the noisy phase, separated soure magnitudes may be inonsistent, i.e., no orresponding timedomain signal may exist [13, 14]. Iterative reonstrution tehniques suh as Griffin-Lim [15] and multiple input spetrogram inversion (MISI) [16] attempt to reover eah soure s lean phase by fixing its magnitude estimate and running alternating STFT and istft iterations starting from the noisy phase. Applying iterative phase reonstrution as a post proessing to a magnitude enhanement network often results in modest improvements in soure separation performane [17, 6]. This has inspired reent tehniques where phase reonstrution is omputed by a generative network [18] or by a phase subnetwork trained in tandem with a magnitude subnetwork for audiovisual speeh enhanement [19]. This paper presents an extension to reent work by Wang et al. [9], whih learns a time-frequeny mask estimation network by training through multiple unfolded MISI phase reonstrution iterations. Following the deep unfolding framework [20], eah phase reonstrution iteration is treated as a neural network layer. In [9], the phase reonstrution layers are fixed, i.e., they have no learnable parameters implementing the STFT, istft, absolute value, and angle operations neessary to perform phase reonstrution. Similarly to [7, 8] in whih STFT-like enoders are implemented as onvolutional layers (and istft-like deoders are implemented as transposed onvolution layers), we here allow the DFT/iDFT basis matries of the STFT/iSTFT layers to be adapted based on data. The learning of these phase reonstrution layers happens in tandem with a mask inferene network for estimating the separated soure magnitudes, and the entire system an be trained end-to-end. This allows the magnitude estimation network to produe outputs suitable for subsequent iterative phase reonstrution. We evaluate multiple variations of this underlying approah and show improvements over the fixed unfolded phase reonstrution approah of [9] on the publi wsj0-2mix orpus, leading to a new state of the art.

4 2. MASK INFERENCE NETWORK Unfolded MISI network An overall blok diagram of our system is shown in Figure 1. In this setion, following [6, 9], we desribe how to train a mask inferene network to estimate a magnitude representation of high enough quality that a subsequent iterative phase reonstrution network an lead to improved performane. Let X C F T be the omplex spetrogram ontaining the mixture of C soures S C F T for = 1,..., C. Our goal is to estimate a real valued mask for eah soure ˆM R F T by learning a nonlinear funtion Φ( ) from an input feature spae (typially log-magnitude, independently whitened in eah dimension), i.e., ˆM 1,..., ˆM C = Φ(log X ). (1) Here Φ( ) is learned by the mask inferene network in Figure 1 to minimize the trunated phase sensitive approximation (tpsa) objetive [11] in a permutation-free manner [2, 3, 4]: L tpsa = min π P ˆM π() X T γ X 0 ( S os( S X)), (2) 1 where P is the set of all possible permutations over the set of soures {1,..., C}, denotes element-wise produt, S is the true phase of soure spetrogram, and X is the mixture phase. The l 1 norm is used in (2) as opposed to the mean square error beause it was found to empirially perform better in [6]. The trunation funtion in (2) is defined as T b a(x) = min(max(x, a), b), where a = 0 and b = γ X. For the sigmoid and softmax ativation funtions often used for mask inferene [4, 6], γ is typially set to γ = 1, however, setting γ > 1 an aount for phase anellation, but requires a modified ativation funtion for the output of the mask layer. The study in [9] noted improved performane by setting γ = 2, and defining a onvex softmax nonlinearity where the final fully-onneted layer uses a softmax to output a probability distribution [p 0, p 1, p 2 ] over a set of potential mask values {0, 1, 2} for eah time frequeny bin of eah soure, whih is then used to ompute the expeted mask value y = [p 0, p 1, p 2 ][0, 1, 2] T. Intuitively, the potential mask values {0, 1, 2} orrespond to three outomes: (1) the soure is not present (y = 0), (2) the soure is dominant (y = 1), or (3) the soure is involved in phase anellation (y = 2). As proposed in [6], separation performane an be further improved by adding a deep lustering branh to the mask network and using the multi-task training objetive L hi ++ α = αl DC + (1 α)l tpsa (3) where L DC is the whitened k-means objetive defined in [6] and the weight α is typially set to a high value, e.g., Although the deep lustering branh is not used during inferene time, it ats as an effetive regularizer during training. Time-frequeny objetives suh as (2) and (3) do not aount for phase inonsistenies, so [9] proposed the waveform Mask inferene network MISI Layer Fig. 1. Proposed speeh separation system. C is equal to 3C for onvex softmax and to C for other nonlinearities. approximation objetive L WA = min π P 1 ŝ π() s, (4) that operates diretly on the reonstruted time domain signals ŝ 1,..., ŝ C. The time domain signals an be obtained via a single istft or by further using an unfolded MISI network as desribed next. 3. ITERATIVE PHASE RECONSTRUCTION NETWORK The MISI algorithm modifies the lassi Griffin-Lim approah to phase reonstrution speifially for soure separation by enforing the onstraint that the sum of the separated and reonstruted soures should equal the mixture. Algorithm 1 desribes an extension of the MISI algorithm onsidered here, where the STFT and istft operations are generalized to STFT-like and istft-like operations that inorporate parameters. These parameters are part of a parameter set STFT/iSTFT Convolution Layers For real-valued sequenes suh as audio signals, an N point DFT has N/2 + 1 unique omplex oeffiients. The DFT an be implemented using only real valued operations by staking the real and imaginary omponents and defining the elements of the basis matrix W R (N+2) N as { w(n) os(2πki/n), i [1, N W i,n = 2 +1]] w(n) sin(2πki/n), i [ N (5) 2 +2, 2N +2] where we have inorporated the analysis window w(n) into the basis matrix. By treating W as the weight matrix in a

5 one-dimensional onvolution layer, and setting the stride parameter of this layer equal to the hop size, we an effiiently reate STFT-like layers with learnable basis matries. The inverse DFT matrix an be defined similarly to (5) by using the synthesis window and aounting for the appropriate normalization terms. We an again implement a trainable istft-like layer using transposed onvolutions Unfolded MISI Depending on the number of unique STFT (k) and istft(k) operators in Algorithm 1, we an implement the unfolded phase reonstrution network from Figure 1 in the following variations: Post [6]: A mask inferene network is trained using the himera objetive (3), and MISI is only used as a post proessing step, i.e., no bak propagation through MISI; Fixed [9]: The mask inferene network of Figure 1 is trained using the objetive (4) while keeping the STFT and istft layers fixed; Tied (Proposed): Together with the mask inferene network, the DFT/iDFT matries of the phase reonstrution network are also adapted after being initialized as in (5). Only one DFT-like weight matrix and one idftlike weight matrix are learned for the entire network and shared aross the unfolded phase reonstrution layers; Untied (Proposed): The forward and inverse transform parameters are untied and independently learned for eah STFT-like and istft-like layer. Even when the number of MISI iterations is zero, we an still learn to adapt the first forward transform representation applied to the mixture and the last inverse transform representation used to reonstrut the estimated waveforms. It is also interesting to point out that the phase reonstrution network struture in Figure 1 is omposed of alternating onvolutional layers and residual/skip onnetions, a general struture broadly shared by several reent audio soure separation arhitetures suh as those based on WaveNets [21] and, ResNets [12]. This is an exiting side effet of expliitly using the deep unfolding framework [20] to reate network arhitetures. For example, even as we begin to replae portions of the network arhiteture suh as the magnitude and phase funtions with more traditional neural network nonlinearities (suh as sigmoid and relu as was done in [8]), we retain some intuition as to what the network is learning. 4. RESULTS We use the publi wsj0-2mix dataset [2], whih ontains 30 h of two-speaker mixtures for training, 10 h for validation, and 5 h for testing with a sampling rate of 8 khz and signal to noise ratios (SNRs) between 0 and 5 db. We use the same spetral analysis parameters for both the log- Input: Mixture signal x in the time domain, estimated masks ˆM for = 1,..., C, and number of iterations K X = STFT (0) (x); S (0) = ˆM X, for = 1,..., C; for k = 1,..., K do ŝ (k 1) = istft (k 1) ( δ (k 1) = x C end S (k) = S (0) =1 ŝ(k 1) ; e j STFT(k) (ŝ (k 1) S (k 1) ), for = 1,..., C; + δ(k 1) C return ŝ (K) = istft (K) (K) ( S ), for = 1,..., C; Algorithm 1: Unfolded MISI. STFT (k) spetrogram of a signal, and istft (k) reonstruts a timedomain signal from a omplex spetrogram. ), for = 1,..., C; extrats a omplex magnitude features input to the mask network and for the initial STFT/iSTFT layers in the phase reonstrution network: window size of 32 ms (256 samples) with a stride of 64 samples and square root Hann analysis/synthesis windows designed for perfet reonstrution after overlap-add. The mask inferene network uses the arhiteture from [6, 9] with four 600 unit BLSTM layers and dropout of 0.3 applied to the first three layers. During training, input sequenes are limited to 400 frames (approximately 3.2 s), and optimization is performed using the ADAM algorithm [22]. We train the network for 100 epohs, and if the loss funtion on the validation set does not derease for five onseutive epohs the learning rate is deayed by 0.5. Furthermore, we ahieved a signifiant performane boost by following the urriulum learning strategy proposed in [9] for eah of the unfolded MISI approahes desribed in Setion 3.2: 1. We first train only the mask network using the himera objetive (3) and an initial ADAM step size of α = Next, we disard the deep lustering portion of the himera network and train using the waveform approximation objetive (4) by adding forward and inverse time-frequeny representation layers (but no MISI layers yet) with an initial ADAM step size of α = Starting from the network with k 1 MISI layers, we train a network with k MISI layers and an initial ADAM step size of α = Repeat up to K = 5, where performane begins to saturate. We report sale-invariant SDR (SI-SDR) on the wsj0-2mix test set, as opposed to the version omputed using the bss eval software pakage for reasons disussed in [23]. Figure 2 ompares network arhitetures for different numbers of MISI layers for both the sigmoid and onvex softmax ativations. As first reported in [9], training the mask inferene network through fixed MISI layers provides a signifiant boost in performane ompared to using phase reonstrution only for post proessing. Furthermore, learning the time frequeny representations in the phase network provides a boost in performane for all numbers of iterations/layers with untied performing better than tied. Finally, we note that for the

6 Fig. 2. Performane of different MISI reonstrution onfigurations. Solid lines orrespond to sigmoid ativation and dashed lines to onvex softmax at the mask network output. Table 1. SI-SDR (db) omparison with other reent systems on the wsj0-2mix test set. Approah MISI Iterations SI-SDR [db] TasNet-BLSTM [8] Chimera++ [6] Fixed (onvex softmax) [9] Tied-MISI (sigmoid) Untied-MISI (sigmoid) Orale Algorithms Magnitude Ratio Mask Ideal Binary Mask Phase Sensitive Mask Ideal Amplitude Mask fixed ase, as first shown in [9], using the onvex softmax ativation provides a performane boost ompared to sigmoid, as allowing the mask network to predit values greater than 1 allows for time-frequeny magnitude values whih are loser to those obtained from atual time-domain signals. However, for the onditions with learned time-frequeny representations (tied/untied), the differene in performane between ativation funtions almost disappears. We hypothesize this is beause the network learns time-frequeny representations that are appropriately saled for the mask network output, something not possible with fixed STFT/iSTFT representations. The biggest SI-SDR gain for the tied/untied ases also happens when no phase reonstrution is performed (i.e., 0 MISI iterations), ahieving over 12.2 db SI-SDR by learning just one forward transform and one inverse transform. Two MISI iterations are required to math suh performane when using the fixed STFT/iSTFT representations. Table 1 ompares the performane of the iterative phase reonstrution algorithms proposed in this paper, with some reent ompetitive approahes, and orale approahes both Fig. 3. Adapted forward and inverse filters for the Tied-MISI- 5 ondition. Labeled by original enter frequeny (F ). Fig. 4. Example filters for MISI layers from the Untied-MISI- 5 ondition. Labeled by original enter frequeny (F ). with and without MISI. Our results improve upon the best known previous results from [9] by 0.4 db for 0 MISI iterations and 0.2 db for 5 MISI iterations, approahing the performane of some orale masks. Figure 3 ompares some learned time frequeny representations to their orresponding original STFT versions. Some hanges in the shape of the analysis/synthesis windows an be observed. It also illustrates that some original high frequeny representations (e.g., the far right olumn in Figure 3) learn to fous on low frequeny signal omponents. This is onsistent with the results of [7, 8] where many learned time-frequeny filters fous on the low frequeny range. Figure 4 displays learned filters for two MISI layers in an untied network onfiguration. While the filters are signifiantly different from those in the STFT/iSTFT, they are more diffiult to interpret. 5. CONCLUSION We proposed an approah to speeh separation based on deep unfolding of the MISI iterative phase reonstrution algorithm. By learning time-frequeny representations from data, we obtain quantitative improvements in separation performane. Future work inludes improving the initial phase estimates over the noisy phase and exploring the appliation of the proposed tehniques to other domains suh as speeh enhanement, musi, or environmental sound separation.

7 6. REFERENCES [1] D. Wang and J. Chen, Supervised speeh separation based on deep learning: An overview, in arxiv preprint arxiv: , [2] J. R. Hershey, Z. Chen, and J. Le Roux, Deep lustering: Disriminative embeddings for segmentation and separation, in Pro. IEEE International Conferene on Aoustis, Speeh and Signal Proessing (ICASSP), [3] Y. Isik, J. Le Roux, Z. Chen, S. Watanabe, and J. R. Hershey, Single-hannel multi-speaker separation using deep lustering, in Pro. Interspeeh, [4] M. Kolbæk, D. Yu, Z.-H. Tan, and J. Jensen, Multitalker speeh separation with utterane-level permutation invariant training of deep reurrent neural networks, IEEE/ACM Transations on Audio, Speeh and Language Proessing, [5] Y. Luo, Z. Chen, J. R. Hershey, J. Le Roux, and N. Mesgarani, Deep lustering and onventional networks for musi separation: Stronger together, in Pro. IEEE International Conferene on Aoustis, Speeh and Signal Proessing (ICASSP), [6] Z.-Q. Wang, J. Le Roux, and J. R. Hershey, Alternative objetive funtions for deep lustering, in Pro. IEEE International Conferene on Aoustis, Speeh and Signal Proessing (ICASSP), [7] S. Venkataramani and P. Smaragdis, End-to-end soure separation with adaptive front-ends, in arxiv preprint arxiv: , [8] Y. Luo and N. Mesgarani, TasNet: Time-domain audio separation network for real-time, single-hannel speeh separation, in Pro. IEEE International Conferene on Aoustis, Speeh and Signal Proessing (ICASSP), [9] Z.-Q. Wang, J. Le Roux, D. Wang, and J. R. Hershey, End-to-end speeh separation with unfolded iterative phase reonstrution, in arxiv preprint arxiv: , [10] D. S. Williamson, Y. Wang, and D. Wang, Complex ratio masking for monaural speeh separation, IEEE/ACM Transations on Audio, Speeh, and Language Proessing, [11] H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux, Phase-sensitive and reognition-boosted speeh separation using deep reurrent neural networks, in Pro. IEEE International Conferene on Aoustis, Speeh and Signal Proessing (ICASSP), [12] N. Takahashi, N. Goswami, and Y. Mitsufuji, MM- DenseLSTM: An effiient ombination of onvolutional and reurrent neural networks for audio soure separation, in arxiv preprint arxiv: , [13] J. Le Roux, N. Ono, and S. Sagayama, Expliit onsisteny onstraints for STFT spetrograms and their appliation to phase reonstrution, in Pro. ISCA Workshop on Statistial and Pereptual Audition (SAPA), Sep [14] T. Gerkmann, M. Krawzyk-Beker, and J. Le Roux, Phase proessing for single-hannel speeh enhanement: History and reent advanes, IEEE Signal Proessing Magazine, vol. 32, no. 2, [15] D. W. Griffin and J. S. Lim, Signal estimation from modified short-time Fourier transform, IEEE Transations on Aoustis, Speeh, and Signal Proessing, vol. 32, no. 2, [16] D. Gunawan and D. Sen, Iterative phase estimation for the synthesis of separated soures from single-hannel mixtures, in IEEE Signal Proessing Letters, [17] Y. Zhao, Z.-Q. Wang, and D. Wang, A two-stage algorithm for noisy and reverberant speeh enhanement, in Pro. IEEE International Conferene on Aoustis, Speeh and Signal Proessing (ICASSP), [18] K. Oyamada, H. Kameoka, T. Kaneko, K. Tanaka, N. Hojo, and H. Ando, Generative adversarial networkbased approah to signal reonstrution from magnitude spetrograms, in arxiv preprint arxiv: , [19] T. Afouras, J. S. Chung, and A. Zisserman, The onversation: Deep audio-visual speeh enhanement, in arxiv preprint arxiv: , [20] J. R. Hershey, J. Le Roux, and F. Weninger, Deep unfolding: Model-based inspiration of novel deep arhitetures, in arxiv preprint arxiv: , [21] D. Rethage, J. Pons, and X. Serra, A wavenet for speeh denoising, in Pro. IEEE International Conferene on Aoustis, Speeh and Signal Proessing (ICASSP), [22] D. Kingma and J. Ba, Adam: A method for stohasti optimization, in arxiv preprint arxiv: , [23] J. Le Roux, J. R. Hershey, A. Liutkus, F. Stöter, S. T. Wisdom, and H. Erdogan, SDR half-baked or well done? Mitsubishi Eletri Researh Laboratories (MERL), Cambridge, MA, USA, Teh. Rep., 2018.