Speaker Adaptation Based on Sparse and Low-rank Eigenphone Matrix Estimation

Transcription

1 INTERSPEECH 2014 Speaker Aaptation Base on Sparse an Low-rank Eigenphone Matrix Estimation Wen-Lin Zhang 1, Dan Qu 1, Wei-Qiang Zhang 2, Bi-Cheng Li 1 1 Zhengzhou Information Science an Technology Institute, Zhengzhou, China 2 Department of Electronic Engineering, Tsinghua University, Beijing, China zwlin 2004@163com, quanquan@sinacom, wqzhang@tsinghuaeucn, lbclm@163com Abstract The eigenphone base speaker aaptation outperforms the conventional MLLR an eigenvoice methos when the aaptation ata is sufficient, but it suffers from severe over-fitting when the aaptation ata is limite In this paper, l 1 an nuclear norm regularization are applie simultaneously to obtain a more robust eigenphone estimation, resulting in a sparse an low-rank eigenphone matrix The sparse constraint can reuce the number of free parameters while the low rank constraint can limit the imension of phone variation subspace, which are both benefit to the generalization ability Experimental results show that the propose metho can improve the aaptation performance substantially, especially when the amount of aaptation ata is limite Inex Terms: eigenphones, speaker aaptation, l 1 regularization, nuclear norm regularization 1 Introuction Moel space speaker aaptation is an important technique in moern speech recognition system Given some aaptation ata, the parameters of a speaker inepenent (SI) system are transforme to match the speaking pattern of an unknown speaker, resulting in a speaker aapte (SA) system To eal with the sparsity of the aaptation ata, parameter sharing schemes are usually aopte For example, in eigenvoice base metho [1], the speaker epenent (SD) moels are assume to lie in a low imensional subspace, namely the speaker subspace The subspace bases, ie eigenvoices, are share among all speakers For each new speaker, a speaker-specific coorinate vector, namely speaker factor, is estimate to obtain the SA moel The maximum likelihoo linear regression (MLLR) metho [2] estimates a set of linear transformations to transform an SI moel into a new SD moel Using regression class trees, the HMM state components can be groupe into regression classes with each class sharing the same transformation matrix Recently, a novel phone subspace base metho, ie eigenphone base metho, was propose [3] Differently from the speaker subspace base metho, the phone variation patterns of a speaker are assume to be in a low imensional subspace, calle phone variation subspace The coorinates of the whole phone set are share among ifferent speakers During speaker aaptation, a speaker epenent eigenphone matrix which represents the main phone variation patterns for a specific speaker is estimate Due to its more elaborate moeling, the eigenphone metho performs better than both the eigenvoice an MLLR methos when sufficient amounts of aaptation ata are available However, with limite amounts of aaptation ata, the maximum likelihoo estimation shows severe over-fitting, resulting in very ba aaptation performance [3] Even with a fine tune Gaussian prior, the eigenphone matrix estimate by the maximum a posterior (MAP) criterion still oes not match the performance of the eigenvoice metho In machine learning, regularization techniques are wiely employe to aress the problem of ata sparsity an moel complexity Recently, regularization has been wiely aopte in speech processing an recognition applications For instance, l 1 an l 2 regularization are propose for spectral e-noising in speech recognition [4,5] In [6], similar regularization methos are aopte to improve the estimation of state-specific parameters in the subspace Gaussian mixture moel (SGMM) In [7], l 1 regularization is use to reucing the nonzero connections of eep neural networks without sacrificing speech recognition performance In this paper, we investigate the regularize estimation of the eigenphone matrix for speaker aaptation l 1 norm regularization is use to control the sparsity of the matrix an the nuclear norm regularization forces the eigenphone matrix to be low-rank The basic consierations are that being sparse can alleviate over-fitting an being low-rank can automatically control the imension of the phone variation subspace In the next section, a brief overview of the eigenphone base speaker aaptation metho is given The use of the l 1 norm an nuclear norm regularization are escribe in Section III, an the optimization of the sparse an low-rank eigenphone matrix is presente in Section IV Finally, in Section V, we present experiments on supervise speaker aaptation of a Manarin tonal syllable recognition system 2 Review of the eigenphone base speaker aaptation metho Given a set of speaker inepenent HMMs containing a total of M mixture components across all states an moels an a D-imensional speech feature vector, let µ m, µ m (s) an u m (s) = µ m (s) µ m enote the SI mean vector, the SD mean vector an the phone variation vector for speaker s an mixture component m respectively In eigenphone base speaker aaptation metho, the phone variation vectors {u m(s)} M m=1 are assume to be locate in a speaker epenent N(N << M) imensional phone variation subspace Let v 0(s) an {v i(s)} N i=1 enote the origin an the basis vectors of speaker s s phone variation subspace respectively, then the phone variation vector {u m (s)} M m=1 can be written as u m (s) = v 0 (s) + N l mn v n (s) (1) n=1 Copyright 2014 ISCA September 2014, Singapore

2 where l mn is the coefficient of component m corresponing to basis vector v n(s) We call {v i(s)} N i=0 the eigenphones of speaker s an [ ] T l m1 l m2 l mn the phone coorinate vector of component m The eigenphone ecomposition of speaker s s phone variation matrix can be expresse by the following equation U(s) = [ u 1(s) u 2(s) u M (s) ] = V (s) L (2) where V (s) = [ v 0(s) v 1(s) v 2(s) v N (s) ] an l 11 l 21 l 31 l M1 L = l 12 l 22 l 32 l M2 l 1N l 2N l 3N l MN Equation (2) can be viewe as the ecomposition of the phone variation matrix U(s) to the multiplication of two lowrank matrices V (s) an L The eigenphone matrix V (s) is speaker epenent, which summarizes the main phone variation patterns of speaker s The phone coorinate matrix L is speaker inepenent, which implicitly reflects the correlation information between ifferent Gaussian components Given a set of training speaker SD moels, L can be obtaine using principal component analysis (PCA) [3] During speaker aaptation, given some aaptation ata, the eigenphone matrix V (s) is estimate using the maximum likelihoo criterion Let O = {o(1), o(2),, o(t )} enotes the sequence of feature vectors of the aaptation ata Using the expectation maximization (EM) algorithm, the auxiliary function to be optimize is given as follows Q(V (s)) = 1 γ m (t) 2 t m [o(t) µ m (s)] T Σ 1 m [o(t) µ m (s)], (3) where µ m (s) = µ m + u m, γ m(t) is the posterior probability of being in mixture m at time t given the observation sequence O an current estimation of SD moel Suppose the covariance matrix Σ m is iagonal, let σ m, enotes its th iagonal element an o (t), µ m, an v n, (s) represent the th component of o(t), µ m an v n (s) respectively Then Equation (3) can be simplifie to Q(V (s)) = 1 2 [ γ m(t)σ 1 m, o m,(t) ˆl T 2 mν (s)], t m (4) where o m,(t) = o (t) µ m,, ˆl m = [1, l m1, l m2,, l mn ] T an ν (s) = [v 0, (s), v 1, (s), v 2, (s),, v N, (s)] T, which is the th row of the eigenphone matrix V (s) Define A = t m b = t m Equation (4) can be further simplifie to Q(V (s)) = 1 2 γ m(t)σ 1 m,ˆl mˆlt m γ m (t)σ 1 m,o m,(t)ˆl m, [ ] ν (s) T A ν (s) b T ν (s) + Const (5) Setting the erivative of (5) with respect to ν (s) to zero yiels ˆν (s) = A 1 b Because of the inepenence of ifferent feature imensions, {ˆν (s)} D =1 can be calculate in parallel very efficiently The size of the eigenphone matrix V (s) is (N + 1) D, which has more free parameters than the eigenvoice metho For the MLLR metho with a global transformation matrix an a bias vector, the parameter size is of (D+1) D So the eigenphone metho is more flexible an elaborate When sufficient amounts of aaptation ata are available, better aaptation performance can be obtaine But when the aaptation ata is limite, performance egraes quickly The recognition rate can be even worse than the unaapte SI system when very limite amounts of aaptation are available In orer to alleviate the overfitting problem, a Gaussian prior is assume an a MAP aaptation metho is erive in [3] In this paper, we aress the problem using an explicit matrix regularization function 3 Sparse an low-rank eigenphone matrix estimation In fact, the center of the eigenphone aaptation metho is the robust estimation of the eigenphone matrix V (s) This type of problem, ie the estimation of an unknown matrix from some observation ata, has appeare frequently in the literature of iverse fiels Regularization has been prove to be a vali metho to overcome the ata scarcity One wiely use regularizer is the l 1 norm For the eigenphone matrix V (s), the matrix l 1 norm can be written as V (s) 1 = ν (s) 1 = n v n,(s) The l 1 norm regularization is sometimes referre to as the lasso, which can rive an element-wise shrinkage of V (s) towars zero, thus leaing to a sparse matrix solution Recently, in many matrix estimation problems, such as matrix completion [8] an robust PCA [9], a nuclear norm regularizer was use to obtain a low-rank solution In fact, this approach is closely relate to the iea of using the l 1 norm as a surrogate for sparsity, because low-rank correspons to sparsity of the vector of singular values an the nuclear norm is the l 1 norm of the vector of singular values For the eigenphone matrix V (s), the nuclear norm can be written as V (s) = N i=1 κi, where κi are the singular values of V (s) In eigenphone base speaker aaptation, sparsity an lowrank constraints can be applie simultaneously to obtain more robust estimation of the eigenphone matrix The reasons are as follows: firstly, sparsity constraint can reuce the free parameters, thus alleviates over-fitting; seconly, when the aaptation ata is insufficient, many speaker specific phone variation pattern will not be observe an a low imensional phone variation subspace shoul be assume, ie the rank of the eigenphone matrix shoul be limite The solutions of low-rank estimation problems are in general not sparse at all In this paper, a linear combination of the l 1 an nuclear norm was use to obtain a simultaneously sparse an low-rank matrix [10] The resulting regularize objective function is as following Q (V (s)) = Q(V (s)) + λ 1 V (s) 1 + λ 2 V (s), (6) where λ 1, λ 2 > 0 4 Optimization There is no close form solution to the regularize objective function (6) Numerous approaches have been propose in literature to solve the l 1 norm an nuclear norm penalty problems 2973

3 separately For the mixe norm penalty problem, we aopte the incremental proximal escent algorithm [10, 11] For a convex regularizer R(X), X R m n, the proximal operator is efine as 1 prox R (X) = arg min Y 2 Y X 2 F + R(Y ) (7) where F enotes the Frobenius norm of a matrix The proximal operator for the l 1 norm is the soft thresholing operator prox γ 1 (X) = sgn(x) ( X γ) + (8) where enotes the Haamar prouct of two matrices, (x) + = max{x, 0} The sign function (sgn), prouct an maximum are all taken component-wise For the nuclear norm, the proximal operator is given by the shrinkage operation as follows [11] If X = P iag(ν 1, ν 2,, ν n)q T is the singular value ecomposition of X, then prox γ (X) = P iag((ν i γ) +)Q T (9) The proximity operator of a convex function is a natural extension of the notion of a projection operator onto a convex set The incremental proximal escent algorithm [11] coul be viewe as a natural extension of the iterate projection algorithm, which activates each convex set moeling a constraint iniviually by means of its projection operator In this paper, an accelerate version of the incremental proximal escent algorithm is introuce for estimation of the eigenphone matrix V, which can be summarize as following Algorithm 1 Accelerate Incremental Proximal Descent Algorithm for Sparse an Low-rank Eigenphone Matrix Estimation 1: θ θ 0 Initialize the escent step size 2: V ˆV ˆV is the solution of (5) 3: Q new Q(V ) + λ 1 V 1 + λ 2 V Equation (6) 4: repeat 5: Q ol Q new, θ ηθ 6: repeat Search for the step size 7: V V θ V Q(V ) 8: V prox θλ1 1 (V ) 9: V prox θλ2 (V ) 10: Q new Q(V ) + λ 1 V 1 + λ 2 V 11: if Q new > Q ol then 12: θ η 1 θ 13: en if 14: until Q new < Q ol 15: until Q ol Q new / Q ol < ϵ In Algorithm 1, V Q(V ) is the graient of (5), which can be easily calculate from ν (s)q(v ) = A ν (s) + b Step 7 is the normal graient escent step of the original objective function Q(V (s)) In Step 8 an Step 9, the proximal operators of the l 1 norm an nuclear norm are applie in sequential The initial escent step size θ 0 can be set to inverse of the Lipschitz constant [12] of Q(V (s)) In this paper, to accelerate the convergence spee, the escent step size is increase by a preefine factor η(> 1) for each iteration (Step 5) From Step 6 to 14, we check for the value of the regularize objective function (6) an reuce the step size by a factor of η 1 until it is ecrease The whole proceure is iterate until the relative change of (6) is small than a preefine threshol ϵ (Step 15) 5 Experiments Experiments were performe on a Manarin Chinese continuous speech recognition task using the Microsoft speech corpus [13] The training set contains 19,688 sentences from 100 speakers with a total of 454,315 syllables (about 33 hours total) The testing set consists of 25 speakers an each speaker contributes 20 sentences (the average length of a sentence is about 5 secons) All experiments were base on the stanar HTK (v 341) tool set ( [14]) The frame length an frame step size were set as 25ms an 10ms, respectively Acoustic features were constructe from 13 imensional Mel-frequency cepstral coefficients an their first an secon erivatives The basic units for acoustic moeling are 27 initial an 157 tonal final units of Manarin Chinese as escribe in [13] Monophone moels were first create using all 19,688 sentences Then all possible cross-syllable triphone expansions base on the full syllable ictionary were generate, resulting in 295,180 triphones Out of these triphones, 95,534 triphones actually occur in the training corpus Each triphone was moele by a 3-state left-to-right HMM without skips After ecision tree base state clustering, the number of unique tie states was reuce to 2,392 We then use the HTKs Gaussian splitting capability to incrementally increase the number of Gaussian components per state to 8, resulting in 19,136 ifferent Gaussian components in the SI moel Stanar regression class tree base MLLR metho was use to obtain the 100 training speakers SD moels HVite was use as the ecoer with a full connecte syllable recognition network All 1,679 tonal syllables are liste in the network an any syllable can be followe by any other syllable, or they may be separate by short pause or silence This recognition task puts the highest eman on the quality of the acoustic moels We rew 1, 2, 4, 6, 8 an 10 sentences ranomly from each testing speaker for aaptation in supervise moe an tonal syllable recognition rate was measure among the remaining 10 sentences To ensure statistical robustness of the results, each experiment was repeate 8 times using cross-valiation an the recognition rates were average The recognition accuracy of the SI moel is 5304% (the baseline reference result reporte in [13] is 5121%) For the purpose of comparison, we carrie out three experiments using conventional MLLR + MAP, eigenvoice an eigenphone base aaptation methos without regularization For MLLR + MAP aaptation, we experimente with ifferent parameter settings an the best result was obtaine at a prior weighting factor of 10 (for MAP) an 32 regression classes with a 3-block-iagonal transformation matrix (for MLLR) For eigenvoice aaptation, the imension K of the speaker subspace was varie from 10 to 100 For the eigenphone base metho, both the ML an MAP estimation schemes were teste Aaptation experiment results of the above methos are summarize in Table I For MAP eigenphone metho, λ enotes the prior weighting factor From Table 1, it can be observe that when the aaptation ata is sufficient, the eigenphone base metho outperforms the MAP+MLLR metho But when the aaptation is limite to 1 or 2 sentences (about 5 10 secons), performance egraation emerges ue to overfitting The situation is worse when higher imensional eigenphone subspace is use MAP estimation using a Gaussain prior (equivalent to an l 2 regularization term) can alleviate overfitting to some extent To prevent the performance from egraation, a large prior weight is require, which egraes the performance when the aaptation ata is sufficient 2974

4 Table 1: Average tonal syllable recognition rate (%) after speaker aaptation using conventional methos Methos Number of aaptation sentences MAP+MLLR Eigenvoice K = K = K = K = K = ML Eigenphone N = N = MAP Eigenphone, N = 50 λ = λ = λ = λ = MAP Eigenphone, N = 100 λ = λ = λ = λ = We teste the propose metho with ifferent regularization parameters, where λ 1 is varie between 0 an 100, λ 2 is varie between 0 an 200 Table 2 presents the typical results It can be observe that the nuclear norm regularization (λ 1 = 0, λ 2 0) improves the performance for both N = 50 an N = 100, especially when the aaptation ata is limite to less than 2 sentences A large weighting factor (λ 2 > 100) is neee to obtain the best recognition rates We calculate the average rank of the eigenphone matrix (V (s), which imension is (N + 1) D) over all the testing speakers in each test It is observe that for 1 an 2 sentences, the average rank is small than the feature imension (D=39) When more aaptation ata is provie, the average rank keeps equal to 39 So it can be conclue that the nuclear norm regularization effectively prevents the imension of the phone variation subspace from large than it is necessary Compare with the nuclear norm regularization, l 1 regularization (λ 1 0, λ 2 = 0) can improve the performance further with a small weighting factor (λ 1 < 50) This can be attribute to the sparse constraint introuce, which can reuce the free parameters, thus prevents the estimation of the eigenphone matrix from over-fitting The larger the number of eigenphones (N), the larger the weighting factor (λ 1 ) to achieve the best performance In all testing conition, many elements of (V )(s) become zero, resulting a sparse eigenphone matrix When less aaptation is provie or a large weighting factor λ 1 is use, the eigenphone matrix become more sparse, which means that less free parameters are estimate Combining the l 1 norm an the nuclear norm regularization, performance can be further improve In this situation, compare with using the nuclear norm regularization alone, a relatively small weighting factor of λ 2 < 30 is neee For 1 sentence (about 5s) aaptation, the best result is 5524% (when λ 1 = 20, λ 2 = 10 an N = 50), which is comparable to the best result obtaine by the eigenvoice metho (5572% when K = 60) There is about 1% relative improvement compare Table 2: Average tonal syllable recognition rate (%) after speaker aaptation base on sparse an low-rank eigenphone matrix estimation (λ 1, λ 2 ) Number of aaptation sentences N = 50 (0, 120) (0, 140) (0, 160) (20, 0) (20, 10) (20, 20) (30, 0) (30, 10) (30, 20) N = 100 (0, 120) (0, 140) (0, 160) (20, 0) (20, 10) (20, 20) (30, 0) (30, 10) (30, 20) with the l 1 regularization (5472% when λ 1 = 20, λ 2 = 0 an N = 50) an about 24% relative improvement compare with the MAP Eigenphone metho (5392% when σ ( 2) = 2000 an N = 100) For 2-sentence (about 10s) aaptation, the best result is 5724%, which is slightly better than the best result of eigenvoice (5711% when K = 80) For 4 sentences an more aaptation ata, the performance is also improve compare with the ML eigenphone metho Even with 10 sentences (about 50s) aaptation ata, the best result (6144%) is better than that of the MAP (6070%) an ML eigenphone metho (6062%) Again, the average rank of the eigenphone matrix (V (s)) is small than 39 when there is 1 or 2 aaptation sentences It seems that sparse constraint plays a key role in the performance improvement an the low-rank constraint is a goo complement 6 Conclusion In this paper, we investigate applying l 1 an nuclear norm regularization simultaneously to improve the robustness of the estimation of the eigenphone matrix in eigenphone base speaker aaptation The l 1 regularization introuces sparseness an reuces the number of free parameters, thus alleviates over-fitting The nuclear norm regularization forces the eigenphone matrix to be low-rank, thus prevents the imension of the phone variation subspace from being too high than necessary Their linear combination results in a simultaneous sparse an low-rank eigenphone matrix From our experiments on a Manarin Chinese syllable recognition task, we observe substantial performance improvement uner all testing conitions compare with conventional methos 7 Acknowlegements This work was supporte in part by the National Natural Science Founation of China (No an No ) 2975

5 8 References [1] R Kuhn, J-C Junqua, P Nguyen, an N Niezielski, Rapi speaker aaptation in eigenvoice space, IEEE Trans Speech Auio Process, vol 8, no 6, pp , Nov 2000 [2] M J F Gales, Maximum likelihoo linear transformations for HMM-base speech recognition, Comput Speech Lang, vol 12, no 2, pp 75 98, Apr 1998 [3] W-L Zhang, W-Q Zhang, an B-C Li, Speaker aaptation base on speaker-epenent eigenphone estimation, in Proc of ASRU, Dec 2011, pp [4] Q F Tan, P G Georgiou, an S S Narayanan, Enhance sparse imputation techniques for a robust speech recognition front-en, IEEE Trans Acoust, Speech, Signal Process, vol 19, no 8, pp , Nov 2011 [5] Q F Tan an S S Narayanan, Novel variations of group sparse regularization techniques with applications to noise robust automatic speech recognition, IEEE Trans Acoust, Speech, Signal Process, vol 20, no 4, pp , May 2012 [6] L Lu, A Ghoshal, an S Renals, Regularize subspace gaussian mixture moels for speech recognition, IEEE Signal Process Lett, vol 18, no 7, pp , July 2011 [7] D Yu, F Seie, G Li, an L Deng, Exploiting sparseness in eep neural networks for large vocabulary speech recognition, in Proc of ICASSP, Mar 2012, pp [8] J-F Cai, E J Canès, an Z Shen, A singular value thresholing algorithm for matrix completion, SIAM J Optimization, vol 20, no 4, pp , Jan 2010 [9] E J Canès, X Li, Y Ma, an J Wright, Robust principal component analysis? J ACM, vol 58, no 3, pp 11:1 11:37, May 2011 [10] E Richar an P-A Savalle, Estimation of simultaneously sparse an low rank matrices, in Proc of ICML, July 2012, pp [11] D P Bertsekas, Incremental proximal methos for large scale convex optimization, Math Program, vol 129, no 2, pp , Oct 2011 [12] K-C Toh an S Yun, An accelerate proximal graient algorithm for nuclear norm regularize linear least squares prolems, Pacific J Optim, vol 6, no 3, pp , 2010 [13] E Chang, Y Shi, J Zhou et al, Speech lab in a box : a Manarin speech toolbox to jumpstart speech relate research, in Proc of Eurospeech, 2001, pp [14] S Young, G Evermann, M Gales et al, The HTK Book (for HTK Version 34),