Joint Spectral Distribution Modeling Using Restricted Boltzmann Machines for Voice Conversion

Transcription

1 INTERSPEECH 2013 Join Specral Disribuion Modeling Using Resriced Bolzann Machines for Voice Conversion Ling-Hui Chen, Zhen-Hua Ling, Yan Song, Li-Rong Dai Naional Engineering Laboraory of Speech and Language Inforaion Processing, Universiy of Science and Technology of China, Hefei, PR China {zhling, songy, Absrac This paper presens a new specral odeling and conversion ehod for voice conversion In conras o he convenional Gaussian ixure odel (GMM) based ehods, we use resriced Bolzann achines (RBMs) as probabiliy densiy odels o odel he join disribuions of source and arge specral feaures The Gaussian disribuion in each ixure of GMM is replaced by an RBM, which can beer capure he iner-diensional and iner-speaker correlaions wihin he join specral feaures Specral conversion is perfored by he axiu condiional oupu probabiliy crierion Our experienal resuls show ha he siilariy and nauralness of he proposed ehod are significanly iproved coparing wih he convenional GMM based ehod Index Ters: Voice Conversion, Resriced Bolzann Machine, Specral envelope 1 Inroducion Voice conversion echnique ais o odify he inpu speech of one speaker (source) o ake i sounds like uering by anoher cerain speaker (arge) Many ehods have been developed for voice conversion during he pas years Aong hese ehods, GMM [1] and join densiy GMM (JDGMM) [2] based ehods have becoe a ainsrea ehod because of is saisical approach In JDGMM based ehod, join specral feaure space of source speaker and arge speaker is odeled by a GMM, and specral conversion is perfored by iniizing ean square error (MMSE) crierion This ehod is furher iproved by involving dynaic feaures and using axiizing oupu probabiliy paraeer generaion (MOPPG) algorih [3] Alhough he perforance of JDGMM based voice conversion is quie sable, here sill exis soe probles The os serious one is ha he qualiy of he generaed speech is severely degraded This is ainly due o he over-soohing effec in he convered speech feaures caused by he inadequae odeling of Gaussian based fraework In his approach, he convered specral feaures are ainly deerined by he arge speaker par of he ean vecors of he JDGMM [4] The odel is rained by axiizing likelihood (ML) crierion, which eans he ean of each ixure is a weighed average on all raining saples fro he raining se This averaging process reoves os of he deailed characerisics in he raining saples and leads o a uffled voice qualiy in convered speech Many approaches have been proposed o cope wih he his proble, eg inegraing global variance (GV) in conversion This work was parially funded by he Naional Naure Science Foundaion of China (Gran No and ) and he Naional 973 progra of China (Gran No 2012CB326405) wih MOPPG algorih [3], using rajecory odel [5] o copensae he inadequae odeling in GMM wih a whole senence rajecory odeling, or frequency warping ehod o direcly odify he specral envelopes o reain heir deailed characerisics [6], ec In his paper, we propose a new odeling ehod o iprove he join specral feaure odeling We use he resriced Bolzann achines (RBMs) o odel he feaure space insead of Gaussians In his paper, he feaure can be no only high-level specral feaures (eg el-cepsra or line specral pairs), bu also he raw specral envelopes exraced by STRAIGHT analysis [7] An RBM is a graphic odel, i has been successfully applied o hidden Markov odel (HMM) based saisical paraeric speech synhesis [8] as a densiy odel o describe he disribuion of specral feaures a each HMM sae RBMs have beer abiliy o capure he correlaions beween wo speakers and can also describe he disribuion of high order specral envelopes The reaining of his paper is organized as follows In secion 2, we will describe he deails of our proposed ehod Secion 3 shows our experienal resuls Conclusions will be given in secion 4 2 Mehod In he convenional join space based voice conversion ehods which incorporae dynaic feaures [3], he join feaure vecor is v = [X, Y ], where X = [x, x ] and Y = [y, y ], x and y denoe he saic feaures of he source and arge speakers, x and y denoe he corresponding dynaic feaures A saisical odel λ (v) is rained o odel he join feaure space When convering an inpu feaure sequence X = [X 1,, X T ], where T is he lengh of he sequence, wo seps are aken Firs, a sequence of condiional disribuions P (Y X, λ (v) ) is deerined frae-byfrae Then, he rajecory of he whole sequence is generaed by MOPPG algorih fro he condiional disribuions: ỹ = arg ax ỹ P (Ỹ X, λ (v) ) = arg ax ỹ T P (Y X, λ (v) ) =1 (1) where he sequence of convered feaures (including dynaic feaures) Ỹ is a linear ransforaion of he saic feaure sequence ỹ 21 Convenional GMM based ehod In convenional JDGMM based ehods [3], he densiy odel is a GMM λ (v) = {α, µ (v), Σ (v) ; = 1,, M}, in which Copyrigh 2013 ISCA Augus 2013, Lyon, France

2 M is he oal nuber of ixures, α, µ (v) and Σ (v) are he weigh, ean vecor and covariance arix of he -h ixure, and µ (v) = [ µ (x) µ (y) ], Σ (v) = [ Σ (xx) Σ (yx) Σ (xy) Σ (yy) ] (2) In order o reduce he nuber of odel paraeers and copuaional cos, diagonal cross covariance arices are ofen used in conversion phase, which eans diagonal arices are adoped for Σ (xx), Σ (yy), Σ (xy) and Σ (yx) In conversion phase, he oupu disribuion given an inpu X is sill an M ixure GMM λ (y x), whose paraeers of he -h ixure are µ y x Σ y x = µ (y) = Σ (yy) + Σ (yx) Σ (yx) Σ (xx) 1 Σ (xx) 1 (X µ (x) ), (3) Σ (xy), (4) α y x = P ( X, λ (v) ) (5) For sipliciy, only a single Gaussian wih he axiu poserior probabiliy is ofen adoped for each frae Then, a closefor soluion o (1) can be achieved The deailed forula can be found in [3] However, he voice qualiy of convered speech is degraded due o an over-soohing effec In JDGMM he iner-speaker correlaions are odeled by Σ (xy), bu in he diagonal case, only he correlaions beween wo speakers in he sae diension are considered We observed in our experiens ha he sae diension in wo speaker s feaures is ofen no he os correlaed As a resul, he linear ransforaion er Σ (yx) Σ (xx) 1 in (3) ofen end o be very sall Therefore he ain coponen in convered feaure is he arge eans µ (y) of he Gaussian disribuions The esiaed eans of GMM are weighed averages of all raining saples This is a soohing process which wipes os of he deailed characerisics in original specral feaures and leads o he over-soohing effec in convered specral feaures 22 Proposed ehod 221 RBM as a densiy odel An RBM 1 is an undireced graphical odel [9] This odel has a wo-layer archiecure: a visible layer and a hidden layer The visible sochasic unis v = [v 1,, v V ] corresponding o he acousic feaures are conneced o he hidden sochasic unis h = [h 1, h H ], where V and H denoe he nubers of he visible and hidden unis In specral feaure odeling, feaures are real-valued In his case, Gaussian-Bernoulli RBMs (GBRBMs) are adoped The GBRBM is coposed by Gaussian sochasic visible unis and binary sochasic hidden unis, and is energy funcion is defined as: E(v, h) = V i=1 (a i v i) 2 2σ 2 i V b jh j i=1 w ijh j v i σ i, (6) where a = [a i,, a V ], b = [b 1,, b H ] and W = {w ij } V H are he paraeers of his odel, he paraeer {σ i} is coonly fixed o he sandard deviaion of he raining daa, and is usually oied for noaional sipliciy 1 In he res of his paper, GBRBM is wrien as RBM for shor An RBM can also be reaed as a probabiliy densiy odel, whose probabiliy of generaing a saple v is P (v) = 1 exp( E(v, h)), (7) Z h where Z = h exp( E(v, h))dv is he pariion funcion Since he calculaion of he pariion funcion is inracable, he learning of odel paraeers {W, a, b} is norally carried ou by using he conrasive divergence (CD) algorih [10] under ML crierion and he pariion funcion can be approxiaed by using he annealed iporance sapling ehod [11] 222 Syse consrucion In his paper, we divide he acousic space ino several subspaces, and odel each of he wih an RBM Therefore, our RBM-based voice conversion syse is buil based on he convenional GMM-based syse A GMM is rained on acousic feaure space (eg el-cepsra) in advance o divide he acousic space Training saples are assigned o each of sub-spaces according o he axiu poserior probabiliies of generaing he saples fro corresponding Gaussian ixures Then, he RBM for each sub-space λ (v) is rained In his way, he proposed odel can be viewed as a ixure of RBMs In case of specral conversion, he visible layer is coposed by wo pars, each par corresponding o one speaker As a resul, he odel paraeers for -h subspace can be wrien as a [W (x) = [a (x), a (y) ] and W =, W (y) ], where ( ) (x) and ( ) (y) denoe o he odel paraeers for he source par and arge par In our early experiens on saic feaure conversion, he qualiy of convered speech is degraded due o a disconinuous proble in he feaures generaed direcly by RBMs In order o address his proble, dynaic feaures are used o sooh he convered feaure sequence A conversion ie, he condiional disribuion P (Y X, λ (v) ) is approxiaed wih a single Gaussians, whose ean vecors are generaed fro he RBMs by = arg ax Y M =1 P ( X, λ (v) )P (Y X, λ (v) ), (8) = arg ax P (Y X, λ (v) ), (9) Y where is he ixure wih axiu poserior probabiliy deerined by he JDGMM and he covariance arix is approxiaed wih he variances of all raining saples in he corresponding sub-spaces Siilarly, only he ixure wih axiu poserior probabiliy is used in conversion Once he oupu disribuion sequence is deerined, he convered saic specral feaure sequence is calculaed in he sae anner wih GMM-based voice conversion by MOPPG algorih (1) 223 Specral odeling using RBMs As enioned in secion 21, diagonal cross covariance arix is ofen adoped for each ixure of JDGMM This is because full precision arices for high-order feaures are difficul o ge due o he over-fiing caused by large nuber of paraeers and he cos calculaion of invering high order arix Bu his proble doesn exis in he RBM odeling The iner-diensional correlaions can be odeled by he weigh paraeer W and can be beer learned by CD algorih wih Gibbs sapling Thus, 3053

3 RBMs can be rained on no only high-level specral feaures bu also direcly on he raw specral envelopes 224 Specral conversion In conversion phase, given an inpu X, he ean of he approxiae Gaussian disribuion P (Y X, λ (v) ) is esiaed by y f(x) g(x) = arg ax Y P (Y X, λ (v) ) = arg ax P (X, Y ) (10) where P (X, Y ) is he join disribuion of he visible unis given by he RBM corresponding o he -h sub-space The index is ignored o siplify he noaion The las er is ge by applying Bayesian rules because Y is irrelevan o P (X ) There is no closed-for soluion for (10), he gradien descen algorih can be adoped: Y (i+1) = Y (i) + γ log P (X, Y ) Y Y Y =Y (i), (11) where i denoes he ieraion nuber, γ is he sep size and log P (X, Y ) Y = (Y a (y) ) + exp(b j + v w j) (1 + exp(b j + v w j )) w(y) j (12) Where w j is he j-h colun vecor of W In conras o he GMM-based conversion, Y isn he average of raining saples any ore However, unlike ha in HMM-based speech synhesis syses [8], he gradien based searching process us be perfored frae-by-frae in real-ie conversion This is very ie consuing and is nearly unbearable aribue o he nonlinear er in calculaion of log probabiliy derived fro (7): V log P (v) = (a i v i) 2 + f(b j + v w j) log Z, i=1 (13) where f(x) = log(1 + exp(x)) Bu we can approxiae f(x) wih a siple funcion: g(x) = { x x 0, 0 x < 0 (14) Figure 1 indicaes ha g(x) is a good approxiaion for f(x) when x > 4, and we observed in our experiens ha os of he acivaion er of hidden unis s j = b j + v w j is eiher very big or very sall (eg as we couned in he raining se, p( s > 4) = 094), so his approxiaion should be reasonable Therefore, we go an approxiaed soluion for ean generaion: = a (y) + j {s j >0} w (y) j (15) An iniial value of Y should be se o deerinae which unis are acivaed for he inpu frae In his paper, he iniial value is se o he arge par of he ode of he RBM [8] Figure 1: Approxiaion of funcion f(x) = log(1 + exp(x)) 31 Experienal condiions 3 Experiens We used a Chinese speech corpus daabase wih wo speakers in our experiens The source speaker is feale and he arge is ale 100 parallel senences were used We randoly seleced 80 senences for odel raining and used he reaining 20 senences for es Two kinds of specral feaures were used in our experiens: el-cepsru and original specral envelope 513- order specral envelopes were calculaed by STRAIGHT analysis a 5s frae shif wih 1024 FFT lengh, hen 40-order el-cepsra (no including he 0-h frae power coefficiens) were exraced fro he specral envelopes In specral envelope odeling, log specral envelopes were adoped Dynaic ie warping (DTW) algorih was adoped o align he raining feaure sequences of wo speakers Specral envelopes were aligned using he ie warp inforaion of corresponding elcepsra An 128-ixure GMM was rained on el-cepsra, and he feaure space was divided ino 128 sub-spaces by his GMM The RBMs were rained by CD learning wih 1-sep Gibbs sapling (CD1) and he learning rae is Bach size for bach gradien descen learning was se o 10 and 200 epochs were execued in odel paraeer esiaion The raining saples in each sub-space were noralized o zero ean and uni variance before RBM raining For copuaional efficiency, he nuber of hidden unis was se o 20 for el-cepsra and 100 for specral envelopes 32 Subjecive evaluaion In order o evaluae he perforance of he proposed ehod Five syses were buil for coparison 2 : MCEP-GMM The baseline GMM-based ehod in secion 21, el-cepsra were used as specral feaures MCEP-RBM The proposed ehod for el-cepsru odeling SPE-RBM The proposed ehod for specral envelope odeling SPE-GMM Buil in he siilar anner wih SPE-RBM, he difference is ha a single Gaussian disribuion was adoped for specral envelope odeling in each subspace SPE-RBM-AP The only difference beween his syse and SPE-RBM is ha he approxiaed conversion funcion (15) is used 2 Soe speech exaples convered by hese syses can be found a hp://hoeusceducn/ chenlh/deo/is2013_ SMCRBMhl x 3054

4 Nor specral envelop SOURCE TARGET MCEP-GMM SPE-GMM SPE-RBM Frequency (Hz) Figure 2: Exaples of source, arge and convered specral envelopes, specral envelopes were noralized by heir axiu value Figure 3: Mean opinion score (MOS) of siilariy and nauralness Error bars shows he 95% confidence inerval The laer 3 syses were rained on sub-spaces divided by MCEP-GMM Siilarly, in conversion sage of specral envelope syses, el-cepsra were also exraced o deerine which sub-space each frae belongs o Tweny senences in es se were convered by each syse for he evaluaions Tie cos in SPE-RBM-AP was abou less han 1 inue for each senence I was grealy decreased coparing wih SPE-RBM, in which each senence ook abou 1 hour wih 20 ieraions Figure 2 presens soe represenaive exaples of source, arge and convered specral envelopes In MCEP-GMM syse, convered specral envelopes were recovered fro convered el-cepsra The forans of SPE-RBM specral envelope is uch closer o ha of arge Moreover, coparing wih he oher 2 convered specral envelopes, he SPE-RBM envelope has sharper forans and reained ore deailed characerisics, which indicaes ha he over-soohing effec in he proposed ehod was decreased Table 2: Subjecive preference scores (%) beween SPE-RBM and SPE-RBM-AP syses SPE-RBM SPE-RBM-AP N/P p Siilariy Nauralness We see ha here isn significan difference beween he wo syses in siilariy Bu in nauralness, SPE-RBM-AP is significanly beer We can see fro Table 3 ha SPE-RBM-AP generaed specral envelopes wih higher probabiliy, and he percenage of acivaions ha have absolue value larger han 4 (p( s > 4)) in SPE-RBM-AP is higher han ha in SPE-RBM, which eans he hidden unis of SPE-RBM have higher deerinacy And he conversion by he approxiaed soluion is ore sable han he gradien based searching Table 1: Subjecive preference scores (%) beween MCEP- GMM and MCEP-RBM syses, where N/P denoes No Preference, p eans he p-value of -es beween wo syses MCEP-GMM MCEP-RBM N/P p Siilariy Nauralness Table 3: Average log oupu probabiliy and p( s > 4) in he es se The join feaures are join of inpu and convered feaures Avg log oupu prob p( s > 4) SPE-RBM % SPE-RBM-AP % We ook several subjecive lisening ess o evaluae hese syses Seven liseners ook apar in hese ess Figure 3 shows he ean opinion scores (MOS) (5 scale: 1-wors, 5-bes) for MCEP-GMM, SPE-GMM and SPE-RBM We see ha he proposed SPE-RBM syse is significanly beer han he convenional GMM syses SPE-GMM is also beer han MCEP- GMM, especially in nauralness A coparison beween MCEP-RBM and MCEP-GMM was also aken Resuls shown in Table 1 indicae ha MCEP- RBM is superior o MCEP-GMM The feedback fro liseners was ha hey can ell he difference bu he iproveen is no significan as ha beween SPE-RBM and SPE-GMM This can be aribue o ha specral envelopes have sronger inerdiensional correlaions which can be well odeled by RBMs Alhough el-cepsra have weaker iner-diensional correlaions, he iner-speaker correlaions are sill beer odeled by RBMs Tha s why MCEP-RBM ouperfored MCEP-GMM Then, a preference es beween SPE-RBM and SPE-RBM- AP was aken The preference scores were shown in Table 2 Our proposed conversion ehod for specral envelopes was superior o he convenional over-soohing ehods Soe inforal lisening ess show ha is perforance can be furher iproved by cobing GV based paraeer generaion, especially in MCEP-RBM 4 Conclusions We have proposed an RBM-base specral odeling and conversion ehod in his paper The join acousic space are divided ino several sub-spaces, each of which is odeled by an RBM The RBM can be rained direcly on STRAIGHT specral envelope A conversion sage, acousic feaures are convered by axiizing he condiional oupu probabiliy crierion Experiens show ha he proposed ehod significanly ouperfored he convenional GMM based ehods Even so, he proposed ehod sill needs o be buil based on GMM syses, herefore, using ixure of RBM o direcly odel he enire acousic space will be a fuure work 3055

5 5 References [1] Y Sylianou, O Cappe, and E Moulines, Coninuous probabilisic ransfor for voice conversion, Audio, Speech, and Language Processing, IEEE Transacions on, vol 6, no 2, pp , 1998 [2] A Kain and M Macon, Specral voice conversion for ex-ospeech synhesis, in Proc ICASSP, 1998, pp [3] T Toda, A Black, and K Tokuda, Voice conversion based on axiu-likelihood esiaion of specral paraeer rajecory, Audio, Speech, and Language Processing, IEEE Transacions on, vol 15, no 8, pp , nov 2007 [4] Y Chen, M Chu, E Chang, J Liu, and R Liu, Voice conversion wih soohed g and ap adapaion, in Eurospeech, 2003, pp [5] H Zen, Y Nankaku, and K Tokuda, Coninuous sochasic feaure apping based on rajecory hs, Audio, Speech, and Language Processing, IEEE Transacions on, vol 19, no 2, pp , 2011 [6] D Erro and A Moreno, Weighed frequency warping for voice conversion, in Proc InerSpeech, 2007, pp [7] H Kawahara, I Masuda-Kasuse, and A de Cheveigné, Resrucuring speech represenaions using a pich-adapive iefrequency soohing and an insananeous-frequency-based F0 exracion: Possible role of a repeiive srucure in sounds, Speech Counicaion, vol 27, no 3, pp , 1999 [8] Z Ling, L Deng, and D Yu, Modeling specral envelopes using resriced bolzann achines for saisical paraeric speech synhesis, in Proc ICASSP, 2013(Acceped) [9] P Solensky, Inforaion processing in dynaical syses: foundaions of harony heory, in Parallel disribued processing: exploraions in he icrosrucure of cogniion, D E Ruelhar and J L McClelland, Eds Cabridge, MA, USA: MIT Press, 1986, vol 1, ch 6, pp [10] G Hinon, Training producs of expers by iniizing conrasive divergence, Neural Copuaion, vol 12, no 14, pp , 2002 [11] R Salakhudinov, Learning deep generaive odels, PhD disseraion, Universiy of Torono,