DIRECTLY MODELING VOICED AND UNVOICED COMPONENTS IN SPEECH WAVEFORMS BY NEURAL NETWORKS. Keiichi Tokuda Heiga Zen

Transcription

1 DIRECTLY MODELING VOICED AND UNVOICED COMPONENTS IN SPEECH WAVEFORMS BY NEURAL NETWORKS Keiichi Tokda Heiga Zen Google Nagoya Institte of Technology, Japan ABSTRACT This paper proposes a noel acostic model based on neral networks for statistical parametric speech synthesis The neral network otpts parameters of a non-zero mean Gassian process, which defines a probability density fnction of a speech waeform gien lingistic featres The mean and coariance fnctions of the Gassian process represent deterministic (oiced) and stochastic (noiced) components of a speech waeform, as the preios approach considered the noiced component only Experimental reslts show that the proposed approach can generate speech waeforms approximating natral speech waeforms Index Terms Statistical parametric speech synthesis; neral network; waefom 1 INTRODUCTION Typical statistical parametric speech synthesis systems first extract a set of parametric representation of speech (eg, cepstra 1], line spectrm pairs 2], fndamental freqency, and aperiodicity 3]) then model relationships between the extracted acostic parameters and lingistic featres associated with the speech waeform sing an acostic model 6] (eg, hidden Marko models 4], neral networks 5]) There hae been a cople of attempts to integrate acostic featre extraction into acostic modeling, sch as the log spectral distortion-ersion of minimm generation error training 7], statistical ocoder 8], waeform-leel statistical model 9], and melcepstral analysis-integrated hidden Marko models 1] Tokda and Zen recently proposed a neral network-based approach to integrate acostic featre extraction into acostic modeling 11] Here, a neral network otpts parameters of a zero mean Gassian process, which defines a probability density fnction of a speech waeform gien lingistic featres The coariance fnction of the Gassian process is parameterized by minimm-phase cepstrm The network weights are optimized so as to maximize the log likelihood of the Gassian process gien corresponding pairs of speech waeforms (target) and lingistic featre seqences (inpt) This approach can oercome the limitations of the preios approaches, sch as two-step optimization (acostic featre extraction acostic modeling), se of spectra rather than waeforms, and se of oerlapping and shifting frames as nit Howeer, the speech signal model sed in this approach has only a stochastic (noiced) component, as hman speech has both stochastic and deterministic (oiced) components This paper extends the preios approach 11] to hae both the oiced and noiced components in its speech signal model A neral network otpts parameters of a non-zero mean Gassian process, which defines a probability density fnction of a speech waeform gien lingistic featres The deterministic (oiced) component of a speech waeform is modeled by the mean fnction of the Gassian process, which is parameterized by mixed-phase complex cepstrm Its training algorithm which can rn seqentially in a sample-by-sample or segment-by-segment manner is also deried The rest of the paper is organized as follows Section 2 defines the signal model and the waeform-leel probability density fnction Section 3 deries the training algorithm Preliminary experimental reslts are presented in Section 4 Conclding remarks are gien in the final section 2 WAVEFORM-LEVEL DEFINITION OF PROBABILITY DENSITY FUNCTION OF SPEECH 21 Signal model This paper adopts the signal model shown in Fig 1 Ths, the probability density fnction of a discrete-time speech signal x = x(), x(1),, x(t 1)] corresponding to an tterance or whole speech database is assmed to be a non-zero mean stationary Gassian process, which can be written as p(x λ) = N (x; H c p, Σ c ), (1) λ is the model parameter set, p = p(), p(1),, p(t 1)] is a plse seqence haing ale 1 at pitch mark positions otherwise : p =,,, 1,,,, 1,,,, 1,,, ], (2) and H c is a deterministic component matrix 1 gien as H c = h() h( 1) h( T +1) h(1) h() h( 1) h(t 1) h(1) h(), (3) whose elements are gien by the implse response of the mixed phase system fnction H (z): h(n) = 1 2π π π H (e jω ) e jωn dω (4) In this paper, we assme that the system fnction H (z) generating oiced component (t) is parameterized by complex cepstrm c as H (e jω ) = exp c (m) e jωm, (5) c = c ( M),, c (),, c (M)] The system H (z) shold not model delay since it cases an nder-determined problem 1 Althogh we assme that x and p are infinite seqences, they are described as finite seqences for notation simplicity When they are finite seqences, H shold be a circlant matrix rather than a Toeplitz matrix

2 Plse train p(t) e(t) N(, 1) White noise Mixed phase H (z) H (z) Minimm phase Voiced component (t) (t) Unoiced component Fig 1 Speech signal model Speech x(t) when we estimate H (z) and plse positions of p(t) simltaneosly The complex cepstral representation can aoid the problem becase it intrinsically does not represent delay of the system The coariance matrix Σ c is gien as Σ c = r() r(1) r(t 1) r(1) r() r(1) r(t 1) r(1) r() π (6) r(k) = 1 π H (e jω ) 2 e jωk dω, (7) 2π and H (e jω ) 2 is the power spectrm of the noiced component (t) This paper assmes that the corresponding minimm-phase system fnction H (z) is parameterized by minimm cepstrm c as H (e jω ) = exp c (m) e jωm, (8) m= c = c (), c (1), c (2),, c (M)] The inerse of the coariance matrix Σ c can be written as the same form in 11] as A c = Σ 1 c = A c A c, (9) a() a(1) a() a(t 1) a(1) a() (1) and a(n) is the implse response of the inerse system gien as a(n) = 1 π H 1 (e jω ) e jωn dω (11) 2π π From the aboe definition, the logarithm of the probability density fnction can be written as log p(x λ) = T 2 log 2π log A c A c 1 2 (x H c p) A c A c (x H c p) (12) the model parameter set is gien as λ = {c, c, p} We assme that plse positions in p are extracted by sing an external pitch marker and therefore p is fixed in the following discssion 22 Non-stationarity modeling Eqation (12) can be rewritten as log p(x λ) = T 2 log 2π log A c A c G = A c H c is gien as G = 1 2 (A c x Gp) (A c x Gp), (13) g() g( 1) g( T +1) g(1) g() g( 1) g(t 1) g(1) g() (14) and g(n) is the implse response of the system fnction G(z) = H (z)h 1 (z): G(e jω ) = exp {c (m) c (m)} e jωm, (c (m) =, m < ), (15) that is, g(n) = 1 π G(e jω ) e jωn dω (16) 2π π To model the non-stationary natre of the speech signal, x is assmed to be segment-by-segment piecewise-stationary: A c in Eq (1) and G in Eq (14) are redefined as a (i 1) () a (i) (1) a (i) () a (i) (1) a (i) () A c = L a (i) (1) a (i) () a (i+1) (1) a (i+1) () (17) and g (i 1) () g (i 1) ( 1) g (i) (1) g (i) () g (i) ( 1) g (i) (1) g (i) () g (i) ( 1) G = g (i) (1) g (i) () g (i) ( 1) g (i+1) (1) g (i+1) () (18) i is the segment index, L is the size of each segment, a (i) (n) is the implse response of the inerse system of H (i) (z) represented by cepstrm = (), (1),, (M)], (19) as in Eq (8) for the i-th segment, and g (i) (n) is the implse response of the system G (i) (z) represented by cepstrm and = ( M),, (),, (M)] (2) L,

3 as in Eq (15) for the i-th segment Here the model parameter set of the probability density fnction p(x λ) can be written { } as { λ = c = {c, c }, } c = c (), c (1),, c (I 1), c = c (), c (1),, c (I 1), and I is the nmber of segments in x corresponding to an tterance or whole speech database, and ths T = L I Note that p is omitted from λ since it is assmed to be fixed in this paper 3 TRAINING ALGORITHM 31 Deriatie of the log likelihood With some elaboration, 2 the partial deriatie of Eq (13) wrt can be deried as, (),, (M)] = log p(x c)/ = ( M),, Plse train p(t) x(t) Speech Cepstrm Forward propagation f(t) G(z) H 1 (z) s(t) e(t) z z 1 d (i) Deriaties c (i) Back propagation L 1 (m) = e (i) (Li + k) f (i) (Li + k m), (21) k= and f (i) (t) is the otpt of G (i) (z), whose inpt is p(t), ie f (i) (t) = The signal e (i) (t) is gien as g (i) (n) p(t n), (22) n= e (i) (t) = s (i) (t) f (i) (t), (23) s (i) (t) is the otpt of the inerse of H (i) (z), whose inpt is x(t), ie s (i) (t) = a (i) (n) x(t n) (24) n= The partial deriatie of Eq (13) wrt can also be deried as, = log p(x c)/ = (), d (i) (1),, (M)] L 1 (m) = e (i) (Li + k) e (i) (Li + k m) δ(m)l, (25) k= and δ(m) is the nit implse fnction 32 Seqential algorithm For calclating the implse response a (i) (n) or g (i) (n) sing a recrsie formla 13], O(MN) operations are reqired at each segment i, een if it is trncated with a sfficiently large nmber of N Frthermore, for calclating s (i) (t) in Eq (24) or f (i) (t) in Eq (22), O(N(M + L)) operations are reqired for each segment i First, to redce the comptational brden in calclating s (i) (t) in Eq (24), the following two approximations are applied; 1 By assming s (i) (t) e (i 1) (t), t = Li M,, Li 1 (26) 2 Similar deriation can be fond in Eqs (14) and (16) in 12] Lingistic featres l (i) Lingistic featre extraction Text analysis TEXT Fig 2 Block diagram of the proposed waeform-based framework (M = 2, L = 1, ie i = t) The element z can be realized in the training phase becase it is in an offline processing mode For notation simplicity, here acostic model is illstrated as a feed-forward neral network rather than long short-term memory recrrent neral network (LSTM-RNN) s (i) (t) can be calclated as the otpt of the inerse system whose parameters change segment by segment as follows: s (i) (t) = s(t) = a t (n) x(t n), (27) n= a t (n) = a (i) (n), t = Li,, Li + L 1 (28) 2 As an approximation, inerse filtering in Eq (27) can be efficiently calclated by the log magnitde approximation (LMA) filter 3 12] whose coefficients are gien by c t =, t = Li,, Li + L 1 (29) The same approximation can be applied to calclation of s (i) (t) in Eq (24), except that the system fnction G(z) corresponding to Eq (22) is decomposed into minimm- and maximm-phase components as G(z) = G + (z)g (z), G + (e jω ) = exp G (e jω ) = exp {c (m) c (m)} e jωm, (3) m= 1 c (m) e jωm (31) 3 The LMA filter is a special type of digital filter which can approximate the system fnction of Eq (8)

4 Each of them are implemented in the LMA filter strctre Howeer, G (z) is an anticasal system while G + (z) is a casal system, and ths G (z) shold rn in a time-reersal manner With the aboe approximations, a simple strctre for training the neral network acostic model, which represents a mapping from lingistic featres to speech signals, can be deried It can rn in a seqential manner as shown in Fig 2 This neral network otpts cepstrm c gien } lingistic featre ector seqence 4 l = {l (), l (1),, l (I 1), which in trn gies a probability density fnction of speech signals x corresponding to an tterance or whole speech database conditioned on l as p(x l, M) = N ( x; H c(l) p, Σ c(l) ), (32) M denotes a set of network weights, c(l) is gien by actiations at the otpt layer of the network gien inpt lingistic featres, and the RHS is gien by Eq (12) By back-propagating the deriatie of the log likelihood fnction throgh the network, the network weights can be pdated to maximize the log likelihood It shold be noted that the proposed approach optimizes the acostic featre extraction part and acostic modeling part simltaneosly As a reslt, better modeling accracy can be expected 33 Synthesis strctre The speech waeform can be generated by sampling x from the probability density fnction p(x l, M) It can be done by sing the signal model strctre shown in Fig 1 By decomposing H (z) into minimm- and maximm-phase components as H (z) = H + (z)h (z), the system fnctions H + (z), H (z) and H (z) can be implemented by sing the LMA filter strctre, H + (z) rns in a time-reersal manner It shold be noted that we need an external F predictor for generating the plse train p(t) 4 EXPERIMENTS Speech data in US English from a female professional speaker was sed for the experiments The training and deelopment data sets consisted of 35,497 and 1 tterances, respectiely A speakerdependent nidirectional LSTM-RNN 14] was trained The lingistic featres deried from speech data, transcriptions, and alignments, inclded 56 lingistic contexts, 1 nmerical featres for coarse-coded position of the crrent frame within the crrent phoneme, and one nmerical featre for dration of the crrent phoneme The speech data was downsampled from 48 khz to 16 khz Then 39 mel-cepstrm, 5 band aperiodicity, 1 log F, and 1 oiced/noiced binary flag were extracted at each frame Glottal closre instants (GCI) locations were also extracted sing REAPER 15] Both the inpt and otpt featres were normalized to hae zero-mean nit-ariance The architectre of the LSTM- RNN had 1 feed-forward hidden layer with 256 nits and rectified linear (ReLU) actiation 16] followed by 3 forward-directed LSTMP 17] hidden layers with 512 memory blocks and 256 projection nits, 1 feed-forward hidden layer with 256 nits and ReLU actiation, and an otpt layer with 47 nits 5 and linear actiation 4 The definition of the lingistic featre ector sed in this paper can be fond in 5] and 14] 5 It inclded 39 mel-cepstrm, 5 band aperiodicity, 1 log F, and 1 oiced/noiced flag 2591 Natral Predicted GCI Fig 3 A segment of the generated speech waeform for a sentence Two elect only two not inclded in the training data To redce the training time and the impact of haing many silences, 8% of silence regions were remoed After setting the network weights randomly, they were first pdated to minimize the mean sqared error between the extracted and predicted acostic featres Then the last layer was replaced by a randomly initialized otpt layer with 119 nits 6 and linear actiation After pdating the weights associated with the otpt layer, all weights in the network were pdated by the proposed seqential algorithm so as to maximize the log likelihood of Eq (12) They were first pdated by non-distribted Adam 18] then distribted AdaGrad 19] The mini-batch back propagation throgh time (BPTT) 2] algorithm was sed 17] in both cases Dropot 21] stochastic reglarization (5%) was sed throghot to preent oerfitting Fig 3 shows a synthesized speech waeform generated from the trained neral network It can be seen from the figre that a speech waeform approximating the natral speech waeform is generated 5 CONCLUSIONS An acostic modeling approach based on neral networks to statistical parametric speech synthesis was proposed The network otpts parameters of a non-zero mean Gassian process, which defines a probability density fnction of a speech waeform gien lingistic featres The stochastic (noiced) component of a speech waeform is modeled by the coariance fnction of the Gassian process, parameterized by minimm-phase cepstrm, as the deterministic (oiced) component is modeled by the mean fnction of the Gassian process, parameterized by mixed-phase complex cepstrm Its training algorithm which can rn seqentially on speech waeform in a sample-by-sample or segment-by-segment manner was deried Ftre work incldes simltaneosly estimating pitch marks inclding fractional pitch search in the model training One of the limitations of this approach is that both acostic modeling (l c) and waeform modeling (c x) error goes to the noiced component Introdction of a coariance strctre for the oiced component can alleiate this problem Performance ealation in practical conditions as a text-to-speech synthesis application is also inclded in ftre work 6 It inclded 39 minimm-phase noiced cepstrm, 39 minimmphase oiced cepstrm, and 1 39 maximm-phase oiced cepstrm

5 6 REFERENCES 1] S Imai and C Frichi, Unbiased estimation of log spectrm, in Proc EURASIP, 1988, pp pp ] F Itakra, Line spectrm representation of linear predictor coefficients of speech signals, The Jornal of the Acost Society of America, ol 57, no S1, pp S35 S35, ] H Kawahara, J Estill, and O Fjimra, Aperiodicity extraction and control sing mixed mode excitation and grop delay maniplation for a high qality speech analysis, modification and synthesis system straight, in Proc MAVEBA, 21, pp ] T Yoshimra, K Tokda, T Masko, T Kobayashi, and T Kitamra, Simltaneos modeling of spectrm, pitch and dration in HMM-based speech synthesis, in Proc Erospeech, 1999, pp ] H Zen, A Senior, and M Schster, Statistical parametric speech synthesis sing deep neral networks, in Proc ICASSP, 213, pp ] H Zen, K Tokda, and A Black, Statistical parametric speech synthesis, Speech Commn, ol 51, no 11, pp , 29 7] Y-J W and K Tokda, Minimm generation error training with direct log spectral distortion on LSPs for HMM-based speech synthesis, in Proc Interspeech, 28, pp ] T Toda and K Tokda, Statistical approach to ocal tract transfer fnction estimation based on factor analyzed trajectory hmm, in Proc ICASSP, 28, pp ] R Maia, H Zen, and M Gales, Statistical parametric speech synthesis with joint estimation of acostic and excitation model parameters, in Proc ISCA SSW7, 21, pp ] K Nakamra, K Hashimoto, Y Nankak, and K Tokda, Integration of spectral featre extraction and modeling for HMM-based speech synthesis, IEICE Trans Inf Syst, ol 97, no 6, pp , ] K Tokda and H Zen, Directly modeling speech waeforms by neral networks for statistical parametric speech synthesis, in Proc ICASSP, 215, pp ] K Tokda, T Kobayashi, and S Imai, Adaptie cepstral analysis of speech, IEEE Trans Speech Adio Process, ol 3, no 6, pp , ] AV Oppenhem and RW Schafer, Descrete-Time Signal Processing, Prentice Hall, ] H Zen and H Sak, Unidirectional long short-term memory recrrent neral network with recrrent otpt layer for lowlatency speech synthesis, in Proc ICASSP, 215, pp ] REAPER: Robst Epoch And Pitch EstimatoR, githbcom/google/reaper, ] M Zeiler, M Ranzato, R Monga, M Mao, K Yang, Q-V Le, P Ngyen, A Senior, V Vanhocke, J Dean, and G Hinton, On rectified linear nits for speech processing, in Proc ICASSP, 213, pp ] H Sak, A Senior, and F Beafays, Long short-term memory recrrent neral network architectres for large scale acostic modeling, in Proc Interspeech, ] D Kingma and J Ba, Adam: A method for stochastic optimization, arxi preprint arxi: , ] J Dchi, E Hazan, and Y Singer, Adaptie sbgradient methods for online learning and stochastic optimization, The Jornal of Machine Learning Research, pp , 211 2] R Williams and J Peng, An efficient gradient-based algorithm for on-line training of recrrent network trajectories, Neral Compt, ol 2, no 4, pp 49 51, ] G Hinton, N Sriastaa, A Krizhesky, I Stskeer, and R Salakhtdino, Improing neral networks by preenting co-adaptation of featre detectors, arxi preprent arxi:12758, 212