Eigenvoice Modeling With Sparse Training Data

Transcription

1 IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 13, NO. 3, MAY Eigenvoice Modeling With Sparse Training Data Patrick Kenny, Member, IEEE, Gilles Boulianne, Member, IEEE, and Pierre Dumouchel, Member, IEEE Abstract We derive an exact solution to the problem of maximum likelihood estimation of the supervector covariance matrix used in extended MAP (or EMAP) speaker adaptation and show how it can be regarded as a new method of eigenvoice estimation. Unlike other approaches to the problem of estimating eigenvoices in situations where speaker-dependent training is not feasible, our method enables us to estimate as many eigenvoices from a given training set as there are training speakers. In the limit as the amount of training data for each speaker tends to infinity, it is equivalent to cluster adaptive training. Index Terms Cluster adaptive training, eigenvoices, extended MAP (EMAP), speech recognition, speaker adaptation. I. INTRODUCTION EIGENVOICE modeling has proved to be effective in small vocabulary speech recognition tasks where enough data can be collected to carry out speaker-dependent training for large numbers of speakers [1] [4]. Our objective in this article is to show how this type of modeling can be extended to situations where the training data is sparse in the sense that speaker-dependent training is not feasible. Suppose, to begin with, we ask an oracle to supply speakerdependent models for any speaker. If denotes the total number of mixture components in a speaker model and the dimension of the acoustic feature vectors then, for each speaker, we can concatenate the mean vectors associated with the mixture components to form a supervector of dimension. The idea behind eigenvoice modeling is that principal components analysis can be used to constrain these supervectors to lie in a low dimensional space with little loss of accuracy. This ensures that only a small number of parameters need to be estimated in order to enroll a new speaker. Thus, speaker adaptation saturates quickly in the sense that the recognition accuracy for a test speaker reaches a plateau after a small amount of adaptation data has been collected. To flesh this out a bit, let and denote the mean and covariance matrix of the supervectors for the speaker population. The basic assumption in eigenvoice modeling is that most of the eigenvalues of are zero. This guarantees that the speaker supervectors are all contained in a linear manifold of low dimension, namely, the set of supervectors of the form where lies in the range of. This set of supervectors is known as the eigenspace; The eigenvoices of the population are the eigenvectors of corresponding to nonzero eigenvalues. (It will be helpful to bear in mind that the dimension of the eigenspace is Manuscript received March 26, 2003; revised October 30, The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Ramesh A. Gopinath. The authors are with the Centre de Recherche Informatique de Montréal, Montréal, QC H3A 1B9, Canada ( pkenny@crim.ca). Digital Object Identifier /TSA equal to the number of eigenvoices and this is equal to the rank of.) In order to construct a speaker-adapted HMM for a given speaker (absent the oracle) we need a mechanism to impose the constraint that the speaker s supervector lies in the eigenspace. In this article, we will use maximum a posteriori (MAP) estimation for this purpose rather than use eigenvoices explicitly. The idea behind MAP estimation is that we have a prior probability distribution for the speaker s supervector, namely, the Gaussian distribution with mean and covariance matrix. Since this prior distribution is concentrated on the eigenspace, the same is true of the posterior distribution derived from it using the speaker s adaptation data. In particular, the mode of the posterior distribution lies in the eigenspace. Thus, MAP estimation of the speaker s supervector ensures that the constraint is satisfied. The term EMAP adaptation E for extended [5] is generally used to refer to MAP adaptation using a supervector covariance matrix but, since the traditional usage assumes that is invertible whereas, our concern in this article is with the case where is of less than full rank, we will use the term eigenvoice MAP instead. The idea of integrating eigenvoice modeling with MAP speaker adaptation has been suggested by other authors [6] [8]. Of course other mechanisms can be used to constrain the supervectors to lie in the eigenspace [2], [9], [10] but a good case can be made that MAP estimation is the most natural way. First, MAP adaptation implicitly takes account of the eigenvalues of, as well as the eigenvectors. In fact, if we use the MAP approach, there is no reason in principle why we should make a hard decision to suppress the eigenvectors of which correspond to minor eigenvalues (as required by other approaches). Secondly, MAP adaptation is asymptotically equivalent to speaker-dependent training as the amount of adaptation data increases. Thirdly, it is natural from a mathematical point of view since, as we will show in this article, it plays a central role in estimating eigenvoices by probabilistic principal components analysis. (In particular, no extra software needs to be developed for speaker adaptation if the eigenvoices are estimated in this way.) Finally, the computational burden of MAP adaptation is quite modest provided that the rank of is reasonably small since it essentially boils down to inverting an matrix where is the rank of [6] [8]. The only difficulty with MAP adaptation arises in the case where is large (more than a few thousand) in which case it is computationally intractable unless is constrained in some way. (For example, if is assumed to be of full rank and sparsity constraints are imposed on, MAP speaker adaptation can be implemented by viewing the posterior distribution as a Gaussian Markov random field [11].) The main issue that needs to be addressed in order to implement an eigenvoice model is to estimate the population co /$ IEEE

2 346 IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 13, NO. 3, MAY 2005 variance matrix. If large numbers of training speakers can be enlisted then we can take the sample covariance matrix of the training set supervectors as an estimate of. But note that the rank of the sample covariance matrix is just the number of speakers in the training set. In most situations of interest this is far less than the dimension of the supervector space so the sample covariance matrix may not provide a reasonable estimate of the population covariance matrix. In such cases it may be necessary to impose some constraints on the population covariance matrix in order to produce an eigenspace of sufficiently high dimension that MAP adaptation has some hope of being asymptotically equivalent to speaker-dependent training. For instance, we could impose a block diagonal structure on by ignoring the correlations between mixture components associated with different phonemes as in the seminal paper [5]. (Block diagonal constraints have the effect of boosting the rank of by a factor of where is the number of blocks.) Alternatively, as we will explain, we can boost the rank of by streaming the acoustic features. Or we could guarantee correct asymptotic behavior, at least in theory, simply by forcing the supervector covariance matrix to be of full rank. (Note that a singular covariance matrix can be made nonsingular by an arbitrarily small perturbation. For example, we could set the zero eigenvalues to a small positive value as in [6]. We say in theory because very large amounts of data hundreds of sentences may be needed for speaker adaptation to saturate with a covariance matrix of full rank [11].) Generally speaking, increasing the dimension of the eigenspace can be expected to slow down the rate at which speaker adaptation saturates so, in practice, it may not be possible to ensure that speaker adaptation saturates quickly and that it behaves like speaker-dependent training as the amount of adaptation data increases. Assuming that a satisfactory tradeoff between these two considerations can be found, there remains the question of how to estimate the population covariance matrix if speaker-dependent training is not feasible. (Recall that we began our discussion by appealing to an oracle to supply the training speakers supervectors.) In this article we will show how to estimate the principal eigenvectors of the population covariance matrix directly from the training data without having recourse to speaker-dependent models. Our solution to this problem is a variant of the probabilistic principal components approach introduced in [12]. This methodology was developed in order to extend principal components analysis to handle priors specified by a mixture of Gaussians (rather than by a single Gaussian as in traditional principal components analysis). Priors of this type give rise to an interesting class of MAP estimators which, loosely speaking, are locally linear but globally non linear. These MAP estimators can be applied to a variety of data compression and pattern recognition tasks but in the context of eigenvoice modeling large numbers of training speakers seem to be needed to take advantage of them. So we have to limit ourselves to the case of a single Gaussian as in conventional principal components analysis. (This is unfortunate since clear evidence of at least two modes one male, one female is presented in [9], [13]). The probabilistic approach still has an advantage over other approaches even in the unimodal case because it enables us to estimate as many eigenvoices as there are speakers in the training set. Even if it turns out that most of the variability in the training data can be captured by a smaller number of eigenvoices, MAP speaker adaptation can use the remaining eigenvoices to good advantage since it implicitly takes account of the corresponding eigenvalues. A clear example of this is given in [14, Table I] which reports the results of some closed-set speaker-identification experiments on a population of 319 speakers. Decreasing the number of eigenvoices from 300 to 100 was found to increase the error rate from 14.8% to 16.5% in the case of mean adaptation and from 14.7% to 17.7% when the variances were adapted as well. The need to use as many eigenvoices as possible is particularly evident in the case of large vocabulary speech recognition given the very high dimensionality of the supervector space. The experiments we report in this article were conducted on a large vocabulary task using as many eigenvoices in each stream as there are speakers in the training set. We begin by explaining why cluster adaptive training [9] and the maximum likelihood eigenspace method [13] would break down if we attempted to use them to estimate a full complement of eigenvoices in situations where speaker-dependent training is not feasible and how this problem can be avoided by adopting the probabilistic approach. II. MAXIMUM LIKELIHOOD FORMULATIONS OF THE ESTIMATION PROBLEM Let denote the supervector for a speaker. Our assumption is that for a randomly chosen speaker, is Gaussian distributed with mean and covariance matrix. The speaker-independent supervector can be estimated by Baum-Welch training in the usual way so the problem confronting us is how to estimate if speaker-dependent training is not feasible (so that the supervectors for the training speakers are unobservable). This problem is complicated by the fact that the dimensions of are enormous but it is amenable to maximum likelihood estimation because the maximum likelihood estimate of is of low rank in practice (the rank is bounded by the number of training speakers). This fact will allow us to work out an exact solution to the estimation problem (without having to make any approximations as in [11], [15]). Previous approaches to the problem of estimating eigenvoices have adopted a different perspective: instead of estimating, they estimate a basis for the eigenspace. This point of view is equivalent to ours in the sense that given an estimate of we can write down a basis for the eigenspace (namely, the eigenvectors of corresponding to nonzero eigenvalues) and, conversely, given a basis for the eigenspace we can estimate (by fitting a multivariate Gaussian distribution to the coordinate representations of the supervectors). However the two points of view give rise to different maximum likelihood formulations of the estimation problem and hence to different estimation procedures. To describe the likelihood function used to estimate the eigenspace basis in [9], [13] we need to introduce some notation. For each mixture component, let be the subvector of which corresponds to it. It is assumed that there is a covariance matrix such that, for any speaker, acoustic

3 KENNY et al.: EIGENVOICE MODELING WITH SPARSE TRAINING DATA 347 observation vectors (frames) associated with the mixture component are normally distributed with mean and covariance matrix. Note that, although is independent of, it is not a speaker-independent covariance matrix in the usual sense since it measures deviations from the speaker-dependent mean vectors rather than deviations from a speaker-independent mean vector. Let denote the block diagonal matrix whose diagonal blocks are. For each speaker, let denote the speaker s training data and for each supervector, let denote the likelihood of calculated with the HMM specified by the supervector and the supercovariance matrix. (We assume throughout that the HMM transition probabilities and mixture weights are fixed so there is no need to include these in the notation.) Although the bias term can be eliminated by augmenting the dimension of the eigenspace [9], it will be convenient for us to retain it. Given a basis for the eigenspace, let be the matrix whose columns are the basis supervectors so that every speaker supervector can be written in the from. (The vector is of dimension where is the dimension of the eigenspace.) The likelihood function which serves as the estimation criterion in [9], [13] is where ranges over the speakers in the training set. (Strictly speaking this is not a likelihood function since it does not integrate to 1. In order to obtain a proper likelihood function the max operation would have to be replaced by a suitable integral with respect to.) The optimization proceeds by iterating the following two steps: 1) For each training speaker, use the current estimates of and to find the supervector which maximizes the HMM likelihood of the speaker s training data. Set 2) Update and by maximizing where the product extends over all speakers in the training set. These two steps are referred to as maximum likelihood eigendecomposition and the maximum likelihood eigenspace method in [13] and as estimating the cluster weights and estimating the cluster means in [9]. The methods in [9], [13] differ as to how the maximization in the second step is performed. In particular, it is assumed in [13] that, for each mixture component, is equal to the speaker-independent covariance matrix for the mixture component and no attempt is made to re-estimate it. It will be helpful to briefly review Gales s method [9] for estimating since our approach will require a similar calculation. For each speaker, let where is given by the first step and suppose that each frame in the training data has been aligned with a mixture component (as in a Viterbi alignment although a forward-backward alignment can also be used). (1) For each mixture component, let denote the number of frames aligned with. Then the quantity to be maximized in the second step is where ranges over all speakers in the training set, ranges over all mixture components and for each pair the sum over extends over all frames aligned with. If we ignore the issue of how to estimate, the problem is just to minimize regarded as a function of. It is characteristic of this type of optimization problem that the covariance matrices drop out so the problem reduces to an exercise in least squares, namely, to minimize For our purposes, the important thing to note is that linear regression (with the s as the explanatory variables and the columns of as the regression coefficients) provides the solution. Our objection to using the likelihood function (1) as the estimation criterion is that, in order to obtain a reasonable estimate of the eigenspace basis in situations where speaker-dependent training is not feasible, the dimension of the eigenspace has to be strictly less than the number of training speakers. (If not, the maximum value of (1) is obviously attained by the basis which consists of the estimates of the training speakers supervectors given by speaker-dependent training.) So in order to avoid this type of degeneracy, only a relatively small number of eigenvoices can be estimated or the basis vectors have to be constrained in some other way. (For example, the basis vectors can be constrained to lie in the low dimensional space consisting of all MLLR transforms of [9].) These constraints are unreasonable because they are artifacts of the estimation procedure. In order to outline the alternative estimation procedure that we will develop, note first that, if is the rank of, we can write where is a matrix of rank. This implies that for each speaker, there is a unique vector such that To say that is normally distributed with mean and covariance matrix, is equivalent to saying has a standard normal distribution. So the problem of estimating the supervector covariance matrix can be formulated in terms of estimating using a different generative model for the training data. Fix a speaker and suppose that each frame in the training data has been aligned with a mixture component. If the

4 348 IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 13, NO. 3, MAY 2005 vector were observable then, using the notation introduced above, the log likelihood of would given by Since we are treating as a random vector with a standard normal distribution, we define the likelihood of by integrating this conditional likelihood function with respect to the standard Gaussian kernel. We denote the value of this integral by (a closed form expression is given in Proposition 2 below). Thus our approach will be to estimate and by maximizing the log likelihood function where the product extends over all speakers in the training set. We will develop an EM algorithm for this purpose in which the role of the hidden variables is played by the vectors. We will show that successive estimates of and are convergent, at least to the extent that the likelihood of the training data increases on successive iterations. However, as in [12], the EM algorithm generally has more than one fixed point so there is no guarantee that if is an EM fixed point then the eigenvectors of are the principal eigenvectors of the supervector covariance matrix. Our procedure consists in iterating the following three steps. We will spell out the details in Section III. 1) For each training speaker, use the current alignment of the speaker s training data and the current estimates of and to carry out MAP speaker adaptation. Use the speaker-adapted model to realign the speaker s training data. 2) The E-step: For each speaker, calculate the posterior distribution of using the current alignment of the speaker s training data, the current estimates of and and the prior. 3) The M-step: Update and by a linear regression in which the s play the role of the explanatory variables. Calculating the posterior distribution of in the E-step rather than the maximum likelihood estimate is the key to avoiding the degeneracy problem that arises when (1) is used as the criterion for estimating the eigenvoices in situations where speaker-dependent training is not feasible and the number of eigenvoices is large compared with the number of training speakers. This calculation is also essentially all that is required for the first step. The principal mathematical difficulty that we will encounter is in carrying out the regression in the M-step, given that for each speaker, is only observable up to the posterior distribution calculated in the E-step. III. ADAPTATION AND ESTIMATION PROCEDURES The main computation that needs to be done both for estimating and and for MAP speaker adaptation for a speaker is to calculate the posterior distribution of given the speaker s training data. To explain how this is done we need to introduce some notation. Assuming that the data for the speaker has been aligned with either the speaker-independent or a speaker-adapted HMM, so that each frame is labeled by a mixture component, we denote by the entire collection of labeled frames for the speaker. We extract the following statistics from. For each mixture component, let be the number of frames in the training data for speaker which are accounted for by the given mixture component and set where the sums extend over all frames for speaker that are aligned with the mixture component and is the speakerindependent mean vector. Let be the block diagonal matrix whose diagonal blocks are where denotes the identity matrix. Let be the vector obtained by concatenating. Let be the matrix defined by Proposition 1: For each training or test speaker, the posterior distribution of given and a parameter set is Gaussian with mean and covariance matrix. The proof of this proposition can be found in the Appendix. Note that is strictly positive definite (and hence, invertible). Furthermore, if is reasonably small (at most a few thousand) there is no difficulty in calculating the inverse of. Hence calculating the posterior distribution is straightforward even in the large vocabulary case. Corollary 1: If, for each training or test speaker, we denote the posterior mean and covariance of by and, then This corollary is the key to MAP speaker adaptation. Invoking the Bayesian predictive classification principle [16], we can adapt both the mean vectors and the variances in the speaker independent HMM to a given speaker as follows. With each mixture component, we associate the mean vector and a covariance matrix given by where is the th entry of when is considered as a block matrix (each block being of dimension ). This type of covariance estimation seems to have been (2)

5 KENNY et al.: EIGENVOICE MODELING WITH SPARSE TRAINING DATA 349 first suggested in [11]. The rationale behind it is that if is an observation for speaker and mixture component, we have where and denotes the th diagonal block of the matrix where the residual is distributed with mean 0 and covariance matrix and is distributed with mean and covariance matrix. Since and are assumed to be statistically independent Cov Cov Cov which gives (2). We will refer to this as covariance adaptation although this is not strictly correct since it is really a mechanism for incorporating the uncertainty about the MAP estimate of into the HMM. (Note that as the amount of adaptation data increases, the uncertainty tends to zero and the covariance estimate reduces to for all speakers.) Corollary 2: If, for each training speaker, denotes the posterior expectation of given and a parameter set and likewise for then Then where the sums extend over all speakers in the training set. Equation (4) is the system of normal equations for our linear regression problem. To solve it, note that since the matrices are diagonal, equating the th row of the left hand side with the th row of the right hand side for gives where is the th row of and similarly for and. Note also that if we write in the form where and then so we obtain This also follows immediately from Proposition 1 and it is the key to implementing the E-step of the EM algorithm described in Proposition 3 below. Next we explain how to calculate the likelihood function that we used in our formulation of the estimation problem. (This calculation is not strictly necessary; its only role is to provide a diagnostic for verifying the implementation of the EM algorithm.) Proposition 2: For each speaker, let denote the Gaussian log likelihood function given by the expression Then Finally, the EM algorithm: Proposition 3: Suppose we are given initial parameter estimates. For each training speaker, let and be the first and second moments of calculated with these estimates according to Corollary 2 to Proposition 1. Let be new estimates of the model parameters defined as follows: is the solution of and for each (3) (4) (5) This is just an system of equations so there is no difficulty in solving for and in the limit as the amount of training data tends to infinity (so that, for each speaker, the posterior distribution of becomes concentrated at a single point, namely the maximum likelihood estimate of ) it is equivalent to [9, eq. (49)]. Equation (5) stands in the same relation to [9, eq. (51)] so Gales s method and ours are asymptotically equivalent. Note that the EM algorithm does not impose any restrictions on which seems to suggest that it is possible to estimate a larger number of eigenvoices than there are speakers in the training set. This paradox can be resolved by proving that if is a fixed point of the EM algorithm then (that is, the rank of ) is necessarily less than or equal, the number of speakers in the training set. It follows that there is nothing to be gained by taking so we took in our experiments. IV. IMPLEMENTATION ISSUES In order to apply the EM estimation procedure given in Proposition 3 we need the first and second order statistics for each training speaker and mixture component. These statistics can be extracted by aligning the training data with the speaker-independent HMM using either a Viterbi alignment or the Baum Welch algorithm. (We used the Baum Welch procedure in our experiments.) But note that after estimates of and have been obtained we can use MAP adaptation (i.e., Corollary 1 to Proposition 1) to construct speaker-adapted models for each of the training speakers and use these to align the training data instead of the speaker-independent model. Accordingly, in the training phase of our experiments, we alternate between alignment and EM iterations until the estimates of converge. (A similar issue arises when applying MAP adaptation to a test speaker. We deal with it in the same way by performing several alignment iterations on the speaker s adaptation data.)

6 350 IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 13, NO. 3, MAY 2005 As we mentioned in the introduction, some constraints have to be imposed on the supervector covariance matrix in order to produce an eigenspace of sufficiently high dimension that MAP adaptation has some hope of being asymptotically equivalent to speaker-dependent training. Our experience has been that the type of block diagonal constraint used in [5] which we referred to in the introduction is not very effective so we experimented with another type of constraint which we implemented by splitting the acoustic features into streams and applying the EM algorithm in each stream separately. This has the effect of boosting the dimension of the eigenspace by a factor of where is the number of streams (so that the greatest effect is achieved by treating each acoustic feature as a separate stream). As we mentioned in Section II, Nguyen and colleagues [13] construct speaker HMMs by applying eigenvoice adaptation to the HMM mean vectors and copying the HMM covariance matrices from a speaker-independent HMM trained in the usual way. Since this is easier to implement than our Bayesian version of Gales s treatment of the HMM covariances, we compared both approaches in all of our experiments. Contrary to our expectations we found that Nguyen s approach (which is really a type of variance inflation) almost always gives better results. TABLE I RECOGNITION ACCURACIES (%) AVERAGED OVER 20 SPEAKERS, 26 STREAMS. IS THE NUMBER OF ADAPTATION SENTENCES, INDICATES MEAN ADAPTATION, INDICATES MEAN AND VARIANCE ADAPTATIONS TABLE II RECOGNITION ACCURACIES (%) AVERAGED OVER 20 SPEAKERS. TWO STREAMS. IS THE NUMBER OF ADAPTATION SENTENCES, INDICATES MEANADAPTATION, INDICATES MEAN AND VARIANCE ADAPTATION V. EXPERIMENTS We carried out some experiments on the French language AUPELF task [17] using the BREF-80 training set which consists of 10.6 h of data collected from 80 speakers. The speaker-independent HMM was a tied-state triphone model with 782 output distributions each having four mixture components with diagonal covariance matrices. For signal processing we used a 10 ms frame rate and a 26-dimensional acoustic feature vector (13 liftered mel-frequency cepstral coefficients together with their first derivatives). For recognition we used a dictionary containing words and a language model consisting of bigrams and trigrams. (Admittedly a much larger language model would be needed to deal with the problem of homophone confusions in French.) We modeled liaisons in both training and recognition [18]. For the test set we chose 20 speakers (not represented in the training set) and five sentences per speaker for a total of words of which 3.0% were out of vocabulary. A. Speaker Adaptation We experimented with 26 streams (each of dimension 1) and two streams (each of dimension 13, one for cepstra, and the other for their first derivatives). We used the training set to estimate 80 eigenvoices in each stream and performed supervised adaptation with various adaptation sets comprising 1, 2, 5, 10, 15, 20, and 100 sentences per speaker; the average length of a sentence was 6 s. 1) 26 Streams: Recognition results averaged over the 20 test speakers are reported in Table I. The second column of Table I gives the results of adapting the HMM mean vectors and copying the HMM covariance matrices from the speaker-independent HMM. Adapting only the mean vectors with small amounts of adaptation data gave small improvements in accuracy and performance saturated slowly, only reaching a plateau after 20 sentences (2 min of speech). The third column shows that adapting the variances as well as the mean vectors was unhelpful across the entire range of adaptation sets. The only way we were able to achieve any improvement with variance adaptation was by imposing block diagonal constraints on (compare [5]). However these constraints diminished the effectiveness of mean adaptation so that no net gain in performance was obtained. Thus, partitioning the mixture components into four blocks gave 73.4% accuracy for mean adaptation versus 74.0% for mean and variance adaptation (using 100 adaptation sentences for each speaker). Similarly, partitioning the mixture components into ten blocks gave 74.7% accuracy for mean adaptation versus 74.8% for mean and variance adaptation. 2) 2 Streams: Table II reports recognition results obtained under the same conditions as in Table I using two streams rather than 26. Aside from the fact that variance adaptation is still ineffective, the main thing to note here is that the performance for mean adaptation saturates much more quickly but reaches a lower plateau than in Table I. This type of behavior is to be expected since the eigenspace is of much lower dimension (160 versus 2080). The substantial improvement obtained with a single sentence of adaptation data and a relatively small number of eigenvoices confirms Botterweck s result [16]. B. Multispeaker Modeling Since variance adaptation with low dimensional eigenspaces was consistently unhelpful, we tried another way of producing speaker-adapted models for the test speakers, namely adding the adaptation data to the training data and estimating the model parameters using the extended training set. With sufficient adaptation data for the test speakers, this increases the

7 KENNY et al.: EIGENVOICE MODELING WITH SPARSE TRAINING DATA 351 number of eigenvoices that can be estimated (from 80 to 100 per stream in the case at hand) and so ensures that the eigenspace is big enough to contain the test speakers as well as the training speakers. We refer to this type of training as multispeaker modeling since it only produces speaker-adapted models for the speakers in the extended training set. We performed a multispeaker recognition experiment using 100 adaptation sentences for each of the test speakers. Performing Baum Welch estimation of the speaker independent HMM with the extended training set and running recognition on the test speakers gave a new benchmark recognition accuracy of 72.1%. Performing mean adaptation on all of the speakers in the extended training set in the course of estimating and running recognition on the test speakers gave a recognition accuracy of 76.4%. Adapting both the means and the variances also led to a recognition accuracy of 76.4% so there is no degradation in performance in this case but no improvement either. VI. DISCUSSION Eigenvoice methods of acoustic phonetic modeling were developed initially to tackle small vocabulary tasks where speaker-dependent training can be carried out for large numbers of speakers. In this article we have shown how to estimate the principal eigenvoices of a speaker population in situations where the training data is too sparse to permit speaker-dependent training by formulating the problem in terms of maximum likelihood estimation of the supervector covariance matrix used in EMAP speaker adaptation. Unlike other methods of eigenvoice estimation, this approach enables us to estimate as many eigenvoices from a given training set as there are training speakers. Our results, like Botterweck s, show that eigenvoice MAP can yield substantial improvements in accuracy on a large vocabulary task with very small amounts of adaptation data (one or two sentences). However, it is doubtful that eigenvoice modeling can provide a complete solution to the problem of acoustic phonetic adaptation in the large vocabulary case because, although it seems reasonable to assume that the true supervector covariance matrix is of relatively low rank and our approach enables us to extract the largest possible number of eigenvoices from a given training set, it seems unlikely that sufficiently many training speakers could ever be enlisted to estimate the supervector covariance matrix reliably. (An exception is the case of multispeaker modeling where the training and test speaker populations coincide. This type of modeling is more likely to be of use in speaker recognition than in speech recognition [14]. It is also interesting to note that the eigenchannel MAP estimator introduced in [14] which uses the methods developed here to tackle the problem of blind channel compensation for speaker identification does not seem to suffer from this rank deficiency problem.) If the supervector covariance matrix is not reliably estimated then speaker adaptation may saturate quickly but there is no guarantee that it will be asymptotically equivalent to speaker-dependent training. Ad hoc constraints designed to increase the dimension of the eigenspace (such as block diagonal constraints or statistical independence conditions on the acoustic features) result in better asymptotic behavior but the improvement is slight and the principal effect of such constraints seems to be to slow down the rate at which speaker adaptation saturates. If the goal of speaker adaptation is to attain speaker-dependent performance with the smallest possible amount of adaptation data (rather than to attain the best possible performance with one or two sentences of adaptation data), then it may be necessary to adopt a different approach to estimating the supervector covariance matrix in the large vocabulary case. Instead of using a principal components analysis to estimate the principal eigenvectors of the supervector covariance matrix, we could carry out a factor analysis. That is, we could assume a decomposition of the form where is a nonsingular diagonal covariance matrix (this guarantees that is of full rank). Ideally this type of model will exhibit both the correct asymptotic behavior of classical MAP (the case ) and the rapid saturation of eigenvoice MAP (the case ). A factor analysis of this type has been implemented in a connected digit recognition task (where speaker-dependent training is feasible) [8]. However, a factor analysis of the training data in the absence of speaker-dependent models will require a good deal of algorithmic development. (Note that even if speaker-dependent models are given, the natural procedure for estimating a factor loading matrix is an EM algorithm [19].) We will take up this question elsewhere. Our results to the effect that variance adaptation is generally less effective than simply copying the speaker-independent HMM variances were contrary to our expectations. They seem to indicate that the best way to model variances is to inflate them and raise the question of whether heavy-tailed distributions ought to be used instead of Gaussians in HMM mixture modeling. Since replacing individual Gaussians by discrete scale Richter mixtures has been found to be a moderately effective strategy [20], continuous scale Richter mixtures might be worth exploring. (These have proved to be effective in dealing with the nongaussian behavior of images containing edges as well as textures [21].) More ambitiously, one could replace each Gaussian in a HMM output distribution by an independent components analyzer, tying the mixing matrices in the spirit of [22]. (See [23] for an outstanding tutorial on ICA. Integrating ICA with HMMs is straightforward in principle [24].) This suggests that non-gaussian modeling of speaker supervectors could also be considered but we do not have any evidence to indicate whether this is worth pursuing. APPENDIX PROOFS OF THE PROPOSITIONS For each speaker, let denote the conditional likelihood of given and the parameter set. Lemma 1: For each speaker

8 352 IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 13, NO. 3, MAY 2005 where is defined by (3) and b) Proof of Proposition 2: If denotes the Gaussian kernel with mean and covariance matrix, then Proof: To simplify the notation, let us drop the reference to. Let. For each mixture component, let denote the th block of (where each block is an vector) and let By Lemma 1, we can write this as where the sum extends over all the frames for the given speaker which are aligned with the mixture component. The log likelihood of conditioned on is (6) where the sum extends over all mixture components. This can be simplified by writing so that If denotes the Gaussian kernel with mean and covariance matrix then the integral can be written in the form By the formula for the Fourier-Laplace transform of the Gaussian kernel [25] this simplifies to which, by corollary 1 to proposition 1, can be written as for. This implies that so and the result follows by substituting this expression in (6). a) Proof of Proposition 1: In order to show that the posterior distribution of is of the stated form it is enough to show that as required. It will be helpful to make a few remarks on differentiating functions of a matrix variable before turning to the proof of proposition 3. If is a function of a matrix variable, then for any matrix of the same dimensions as we set Dropping the reference to, we have and we refer to as the derivative of in the direction of. These directional derivatives are easily calculated for linear and quadratic functions of. For the determinant function they are given by as required. (This can be derived from the identity where is the adjoint of.) If for all matrices then is a critical point of. Define an inner product by setting

9 KENNY et al.: EIGENVOICE MODELING WITH SPARSE TRAINING DATA 353 Since, for each, the functional is linear, there is a unique matrix (the gradient of ) such that and the second term on the right hand side can be simplified as follows: for all. So to find the critical points of we first find the matrix-valued function which enables us to express the directional derivatives of in the form and then set. If the domain of is restricted to symmetric matrices (e.g., covariance matrices), the critical points of are defined by the condition that for all symmetric matrices. In this case, the critical points can be found by setting c) Proof of Proposition 3: We begin by constructing an EM auxiliary function with the s as hidden variables. By Jensen s inequality, We derive the re-estimation formula (4) by differentiating this expression with respect to and setting the gradient to. If is a matrix of the same dimensions as then the derivative with respect to in the direction of is given by In order for this to be equal to for all we must have Since the right hand side of this inequality simplifies to the total log likelihood of the training data can be increased by choosing the new estimates of so as to maximize the left hand side. Since for each speaker from which (4) follows. It remains to derive the re-estimation formula (5) for. Let be any block diagonal matrix having the same dimensions as, so that it has the form..., this is equivalent to opti- and similarly for mizing the quantity Note that, for each, the derivative of with respect to in the direction of is. Hence the directional derivative of with respect to in the direction of is which we refer to as the auxiliary function and denote by. Observe that where is the conditional expectation operator introduced in the statement of the proposition. Furthermore, by Lemma 1 which, by (4), simplifies to

10 354 IEEE TRANSACTIONS ON SPEECH AND AUDIO PROCESSING, VOL. 13, NO. 3, MAY 2005 This can be written as where, for each mixture component and speaker, is the th diagonal block of the matrix. The latter expression evaluates to 0 for all symmetric matrices iff for. Since and are symmetric for all and, (5) follows immediately. (The estimates for the covariance matrices can be shown to be positive definite with a little bit of extra work.) REFERENCES [1] R. Kuhn et al., Eigenfaces and eigenvoices: Dimensionality reduction for specialized pattern recognition, in Proc. IEEE Workshop Multimedia Signal Processing, Dec [2], Eigenvoices for speaker adaptation, in Proc. ICSLP, Sydney, Australia, Dec [3], Fast speaker adaptation using a priori knowledge, in Proc. ICASSP, Phoenix, AZ, Mar [4] R. Kuhn, J.-C. Junqua, P. Nguyen, and N. Niedzielski, Rapid speaker adaptation in eigenvoice space, IEEE Trans. Speech Audio Process., vol. 8, no. 6, pp , Nov [5] G. Zavaliagkos, R. Schwartz, and J. Makhoul, Batch, incremental and instantaneous adaptation techniques for speech recognition, in Proc. ICASSP, Detroit, MI, May 1995, pp [6] H. Botterweck, Anisotropic MAP defined by eigenvoices for large vocabulary continuous speech recognition, in Proc. ICASSP, Salt Lake City, UT, May [7] E. Jon, D. Kim, and N. Kim, EMAP-based speaker adaptation with robust correlation estimation, in Proc. ICASSP, Salt Lake City, UT, May [8] D. Kim and N. Kim, Online adaptation of continuous density hidden Markov models based on speaker space model evolution, in Proc. ICSLP, Denver, CO, Sep [9] M. Gales, Cluster adaptive training of hidden Markov models, IEEE Trans. Speech Audio Process., vol. 8, no. 4, pp , Jul [10] R. Westwood, Speaker Adaptation Using Eigenvoices, M.Phil., Cambridge University, Cambridge, UK, [11] B. Shahshahani, A Markov random field approach to Bayesian speaker adaptation, IEEE Trans. Speech Audio Process., vol. 5, no. 2, pp , Mar [12] M. Tipping and C. Bishop, Mixtures of probabilistic principal component analyzers, Neural Computation, vol. 11, pp , [13] P. Nguyen, C. Wellekens, and J.-C. Junqua, Maximum likelihood eigenspace and MLLR for speech recognition in noisy environments, in Proc. Eurospeech, Budapest, Hungary, Sep [14] P. Kenny, M. Mihoubi, and P. Dumouchel, New MAP estimators for speaker recognition, in Proc. Eurospeech, Geneva, Switzerland, Sep [15] Q. Huo and C.-H. Lee, Online adaptive learning of the correlated continuous-density hidden Markov models for speech recognition, IEEE Trans. Speech Audio Processing, vol. 6, no. 4, pp , Jul [16] C.-H. Lee and Q. Huo, On adaptive decision rules and decision parameter adaptation for automatic speech recognition, Proc. IEEE, vol. 88, pp , Aug [17] J. Dolmazon et al., ARC B1 Organization de la première campagne AUPELF pour l évaluation des systèmes de dictée vocale, in JST97 FRANCIL, Avignon, France, Apr [18] G. Boulianne, J. Brousseau, P. Ouellet, and P. Dumouchel, French large vocabulary recognition with cross-word phonology transducers, in Proc. ICASSP, Istanbul, Turkey, Jun [19] D. Rubin and D. Thayer, EM algorithms for ML factor analysis, Psychometrika, vol. 47, pp , [20] M. Gales and P. Olsen, Tail distribution modeling using the richter and power exponential distributions, in Proc. Eurospeech, Budapest, Hungary, Sep [21] M. Wainwright, E. P. Simoncelli, and A. Willsky, Random cascades on wavelet trees and their use in modeling and analyzing natural imagery, Applied Computational Harmonic Analysis, vol. 11, pp , Jul [22] M. Gales, Semi-tied covariance matrices for hidden Markov models, IEEE Trans. Speech Audio Processing, vol. 7, pp , [23] A. Hyvärinen and E. Oja. (1999) Independent Component Analysis: A Tutorial. [Online]. Available: [24] W. Penny, R. Everson, and S. Roberts, Hidden Markov independent components analysis, in Advances in Independent Components Analysis, M. Girolami, Ed. New York: Springer-Verlag, [25] H. Stark and J. Woods, Probability, Random Processes and Estimation Theory for Engineers. Englewood Cliffs, NJ: Prentice-Hall, Patrick Kenny (M 04) received the B.A. degree in mathematics from Trinity College, Dublin, Ireland, and the M.Sc. and Ph.D. degrees, also in mathematics, from McGill University. He was Professor of electrical engineering at INRS-Télécommunication, Montréal, QC, Canada, from 1990 to 1995, where he started up Spoken Word Technologies, RIP to spin off INRS s speech recognition technology. He joined the Centre de Recherche Informatique de Montréal in 1998, where he now holds the position of principal research scientist. His current research interests are concentrated on Bayesian speaker- and channel-adaptation for speech and speaker recognition. Gilles Boulianne (M 99) received the B.Sc. degree in unified engineering from the Université du Québec, Chicoutimi, QC, Canada, and the M.Sc. degree in telecommunications from INRS-Télécommunications, Montréal, QC. He worked on speech analysis and articulatory speech modeling at the Linguistics Department in the Université du Québec, Montréal, until 1990 and then on large vocabulary speech recognition at INRS and Spoken Word Technologies until 1998, when he joined the Centre de Recherche Informatique de Montréal. His research interests include finite state transducer approaches and practical applications of large vocabulary speech recognition. Pierre Dumouchel (M 97) received the Ph.D. degree in telecommunications from INRS-Télécommunications, Montréal, QC, Canada, in He is currently Vice-President R&D of the Centre de Recherche Informatique de Montréal, a Professor at the Ecole de Technologie Supérieure, and a board member of the Canadian Language Industry Association (AILIA). His research interests include broadcast news speech recognition, speaker recognition and audio-visual content extraction.