Linear Hidden Transformations for Adaptation of Hybrid ANN/HMM Models

Transcription

1 Linear Hidden Transformaions for Adapaion of Hybrid ANN/HMM Models Robero Gemello, Franco Mana, Sefano Scanzio, Piero Laface, Renao De Mori To cie his version: Robero Gemello, Franco Mana, Sefano Scanzio, Piero Laface, Renao De Mori. Linear Hidden Transformaions for Adapaion of Hybrid ANN/HMM Models. Speech Communicaion, Elsevier : Norh-Holland, 2007, 49 (10-11), pp.827. < /j.specom >. <hal > HAL Id: hal hps://hal.archives-ouveres.fr/hal Submied on 9 Jul 2010 HAL is a muli-disciplinary open access archive for he deposi and disseminaion of scienific research documens, wheher hey are published or no. The documens may come from eaching and research insiuions in France or abroad, or from public or privae research ceners. L archive ouvere pluridisciplinaire HAL, es desinée au dépô e à la diffusion de documens scienifiques de niveau recherche, publiés ou non, émanan des éablissemens d enseignemen e de recherche français ou érangers, des laboraoires publics ou privés.

2 Acceped Manuscrip Linear Hidden Transformaions for Adapaion of Hybrid ANN/HMM Models Robero Gemello, Franco Mana, Sefano Scanzio, Piero Laface, Renao De Mori PII: S (06) DOI: /j.specom Reference: SPECOM 1593 To appear in: Speech Communicaion Received Dae: 31 March 2006 Revised Dae: 16 Ocober 2006 Acceped Dae: 29 November 2006 Please cie his aricle as: Gemello, R., Mana, F., Scanzio, S., Laface, P., De Mori, R., Linear Hidden Transformaions for Adapaion of Hybrid ANN/HMM Models, Speech Communicaion (2006), doi: /j.specom This is a PDF file of an unedied manuscrip ha has been acceped for publicaion. As a service o our cusomers we are providing his early version of he manuscrip. The manuscrip will undergo copyediing, ypeseing, and review of he resuling proof before i is published in is final form. Please noe ha during he producion process errors may be discovered which could affec he conen, and all legal disclaimers ha apply o he journal perain.

3 Linear Hidden Transformaions for Adapaion of Hybrid ANN/HMM Models 1 Robero Gemello 1, Franco Mana 1, Sefano Scanzio 2, Piero Laface a2 and Renao De Mori 3 a Corresponding Auhor Piero Laface 2 Poliecnico di Torino Corso Duca degli Abruzzi, Torino Ialy {Piero.Laface, Sefano.Scanzio}@polio.i Phone: , Fax: Coauhors 1 LOQUENDO Via Val della Torre, 4 A Torino Ialy {Robero.Gemello, Franco Mana}@loquendo.com Phone: , Fax: Absrac 3 LIA - Universiy of Avignon 339, Chemin des Meinajaries Agroparc BP AVIGNON Cedex 9 France Renao.Demori@lia.univ-avignon.fr, Phone: , Fax: This paper focuses on he adapaion of Auomaic Speech Recogniion sysems using Hybrid models combining Arificial Neural Neworks (ANN) wih Hidden Markov Models (HMM). Mos adapaion echniques for ANNs repored in he lieraure consis in adding a linear ransformaion nework conneced o he inpu of he ANN. This paper describes he applicaion of linear ransformaions no only o he inpu feaures, bu also o he oupus of he inernal layers. The moivaion is ha he oupus of an inernal layer represen discriminaive feaures of he inpu paern suiable for he classificaion performed a he oupu of he ANN. 1 This work was suppored by he EU FP-6 IST Projecs DIVINES and HIWIRE - 1 -

4 In order o reduce he effec due o he lack of adapaion samples for some phoneic unis we propose a new soluion, called Conservaive Training. Supervised adapaion experimens wih differen corpora and for differen ypes of adapaion are described. The resuls show ha he proposed approach always ouperforms he use of ransformaions in he feaure space and yields even beer resuls when combined wih linear inpu ransformaions. Keywords:Auomaic Speech Recogniion; Speaker Adapaion; Neural Nework Adapaion; Caasrophic Forgeing 1 Inroducion The lieraure on adapaion of speaker, environmen, and applicaion is rich of echniques for refining Auomaic Speech Recogniion (ASR) sysems by adaping he acousic feaures and he parameers of sochasic models [1-5]. More recenly, paricular aenion has been paid o discriminaive raining echniques and heir applicaion o he acousic feaure ransformaion [6,7]. Since discriminaive mehods are also used o rain he acousic-phoneic Arificial Neural Neworks (ANN) models, i is worh exploring mehods for adaping heir feaures and model parameers. Several soluions o his problem have been proposed. Some of hese echniques for adaping neural neworks are compared in [8,9]. A classical approach consiss in adding a linear ransformaion nework (LIN) ha acs as a pre-processor o he main nework. Alernaively, i could be possible o simply adap all he weighs of he original nework. A ied-poserior approach is proposed in [10] o combine Hidden Markov Models (HMM) wih ANN adapaion sraegies. The weighs of a hybrid ANN/HMM sysem are adaped by opimizing he raining se cross enropy. A sub-se of he hidden unis is seleced for his purpose. The adapaion daa are propagaed hrough he original ANN. The nodes ha exhibi he highes variances are seleced, since hidden nodes wih a high variance ransfer a larger amoun of informaion o he oupu layer. Then, only he weighs of he links coming ou of he seleced nodes are adaped. Recen adapaion echniques have been proposed wih he useful properies of no requiring o sore he previously used raining daa, and o be effecive even wih a small amoun of adapaion daa. Mehods based on speaker space adapaion [2] and eigenvoices [3] are of his ype and can be applied boh o Gaussian Mixure HMMs as well as o he ANN inpus as proposed in [11]. The parameers of he ransformaions are considered he componens of a vecor in a parameer adapaion space. The principal componens of his space define a speaker space. Rapid adapaion consiss in finding he values of he coordinaes of a specific speaker poin in he speaker space. Anoher approach is he regularized adapaion proposed in [12], where he original weighs of he neworks, rained wih unadaped daa, are he a priori knowledge used o conrol he degree of adapaion, o avoid overfiing on adapaion daa. This paper explores a new possibiliy consising in adaping ANN models wih ransformaions of an enire se of inernal model feaures. Values for hese feaures are colleced a he oupu of a hidden layer for which he number of oupus is usually of he - 2 -

5 order of a few hundreds. These feaures are supposed o represen an inernal srucure of he inpu paern. As for inpu feaure ransformaion, a linear nework can be used for hidden layer feaure ransformaion. In boh cases, he esimaion of he parameers of he adapaion neworks can be done wih error Back-Propagaion by keeping unchanged he values of he parameers of he ANN. A problem, however, occurs in disribued connecionis learning when a nework, rained wih a large se of paerns, has o be adaped o classify inpu paerns ha differ in some aspecs from he ones used originally o rain he nework. A problem called caasrophic forgeing [13] arises when a nework is adaped wih new daa ha do no adequaely represen he knowledge included in he original raining daa. This causes a paricularly severe performance degradaion. This happens when adapaion daa do no conain examples for a subse of he oupu classes. A review of several approaches ha has been proposed o solve his problem is presened in [13]. One of hem uses a se of pseudo-paerns, i.e. random paerns, associaed o he oupu values produced by he connecionis nework before adapaion. These pseudo-paerns are added o he se of he new paerns o be learned [14]. The aemp is o keep sable he classificaion boundaries relaed o classes ha have few or no samples in he new se of paerns. This effecively decreases he caasrophic forgeing of he knowledge provided by originally learned paerns. Tess of his soluion have been repored wih small neworks and low dimensional arificial inpu paerns. Unforunaely, hey do no scale well because i is difficul o generae effecive pseudo-paerns when he dimensionaliy of he inpu feaures is high. For his reason, i has been proposed [15] o include examples of he missing classes, aken from he raining se, in he adapaion se. However, he addiion of a small subse of raining examples relaed o he missing classes could redefine he class boundaries according o he disribuion of hese small subses. This disribuion would be differen from he one of he complee raining se. Moreover, his approach has a main pracical problem: i is mandaory o sore raining se samples for he adapaion sep. The number of samples should be large enough o provide a good preservaion of he class boundaries. Finally, since he ask independen nework could be adaped o several applicaions, differen ses of raining paerns would be necessary o compensae classes missing in differen adapaion ses. This paper proposes a soluion o his problem by inroducing Conservaive Training, a variaion o he sandard mehod of assigning he arge values, which compensaes for he lack of adapaion samples in some classes. The key idea of Conservaive Training is ha he probabiliy of classes wih no adapaion samples available should be replaced by he bes available esimaions of heir real values. The only way o obain hese esimaions is wih he model provided by he original nework. Experimenal resuls on he adapaion es for he Wall Sree Journal ask [16], using he proposed approaches, compare favorably wih published resuls on he same ask [10,16]. The paper is organized as follows: Secion 2 gives a shor overview of he acousic-phoneic models of he ANN used by he ASR sysem, and presens he Linear Hidden Neworks, which ransform he feaures a he oupu of hidden layers. Secion 3 is devoed o he - 3 -

6 illusraion of he problem of caasrophic forgeing in connecionis learning, and proposes our Conservaive Training approach as a possible soluion. Secion 4 illusraes he benefis of Conservaive Training using an arificial classificaion ask of 16 classes. Secion 5 repors he experimens performed on several daabases wih he aim of clarifying he behavior of he new adapaion echniques wih respec o he classical LIN approach. Finally, he conclusions and fuure developmens are presened in he las Secion. 2 Feaure ransformaions The LOQUENDO-ASR decoder uses a hybrid combinaion of Hidden Markov Models (HMM) and a 4-layer Muli Layer Percepron (MLP), where each phoneic uni is described in erms of a single or double sae lef-o-righ auomaon wih self-loops. The HMM ransiion probabiliies are uniform and fixed, and he emission probabiliies are compued by a MLP [17]. The MLP has an inpu layer of 273 unis (39 parameers of a 7 frame conex), a firs hidden layer of 315 unis, a second hidden layer of 300 unis and an oupu layer including a variable number of unis, which is language dependen (600 o 1000). The advanage of using wo hidden layers, raher han a larger single hidden layer, is ha he oal number of connecions is reduced. Moreover, his archiecure allows o consider he acivaion values of each hidden layer as a progressively refined projecion of he inpu paern in a space of feaures more suiable for classificaion. The acousic models are based on a se of vocabulary and gender independen unis, including saionary conex-independen phones and diphone-ransiion coariculaion models [17]. The models have been rained using large 8 khz elephone speech daabases, e.g. he SpeechDa corpora. The raining se includes mainly phoneically balanced senences, and some samples of applicaion liss (e.g. digis, currency, yes-no). The weighs of he ANN/HMM sysem are adaped by opimizing he raining se cross enropy. These models are he acousic models of he 15 languages available wih he LOQUENDO- ASR recognizer. They are used as seed models for he adapaion experimens of Secion 5, unless differenly specified. 2.1 Inpu feaure ransformaions The simples and more popular approach o speaker adapaion wih ANNs is Linear Inpu Transformaion (LIN) [8,9]. A linear ransformaion roaes he inpu space o reduce he discrepancy beween arge and raining condiions. A LIN, as shown in Figure 1, performs his ransformaion. The LIN weighs are iniialized wih an ideniy marix, and hey are rained by minimizing he error a he oupu of he ANN sysem keeping fixed he weighs of he original ANN. Using few raining daa, he performance of he combined archiecure LIN/ANN is usually beer han he one obained by adaping he weighs of he whole nework, because he adapaion involves a lower number of parameers

7 Oupu layer Second Hidden Layer Firs Hidden Layer Linear Inpu Nework Inpu Layer Fig. 1. Arificial Neural Nework including a linear inpu layer Oupu Layer Linear Hidden Nework Second Hidden Layer Firs Hidden Layer Inpu Layer Fig. 2. Arificial Neural Nework including a linear hidden layer 2.2 Hidden feaure ransformaions - 5 -

8 In a layered neural nework, he acivaion values of each hidden layer are a progressively refined projecion of he inpu paern in a space of feaures more suiable for classificaion. The weighs beween he las hidden layer and he oupu layer perform a linear discriminaion of he oupu classes. For his purpose, he weighs of he lower layers of he nework are rained o produce acivaions of he las hidden unis ha are linearly separable. These acivaions can be considered as new feaures obained by a non-linear discriminan analysis. Because of heir properies, hey are used - properly decorrelaed - as observaion feaures for Coninuous Densiies Gaussian Mixure models in he TANDEM approach [18]. Since he acivaion values of a hidden layer represen an inernal srucure of he inpu paern in a space more suiable for classificaion, i is worh considering he adapaion of hese feaures. A Linear Hidden Nework (LHN) performs such an adapaion. Exacly as in he LIN case, he values of an ideniy marix iniialize he weighs of he LHN. The weighs are esimaed using a sandard Back-Propagaion algorihm keeping frozen he weighs of he original nework. I is worh noing ha, since he LHN performs a linear ransformaion, when he adapaion process is compleed, he LHN can be removed. This can be done combining he LHN weighs wih he ones leading o he nodes of he nex layer using he following simple marix operaions: W B a a = W = B LHN SI + B W SI LHN W SI (1) where W a and B a are he weighs and he biases of he adaped layer, W SI and B SI are he weighs and biases of he layer above he LHN in he original Speaker Independen nework, and W LHN and B LHN are he adaped weighs and he biases of he linear hidden nework. In our experimens he LHN has been applied o he las hidden layer, bu since he oupus of an inernal layer can be considered as feaures more discriminaive han he original ones, he LHN can be applied o whaever inernal layer. 3 Caasrophic Forgeing I is well known ha in connecionis learning, acquiring new informaion in he adapaion process may cause a parial or oal oblivion of he previously learned informaion [13,14]. This effec mus be aken ino accoun when adaping an ANN wih a limied amoun of daa, i.e. when he probabiliy of he absence of samples for some acousic-phoneic unis is high. The problem is more severe in he ANN modeling framework han in he classical Gaussian Mixure HMMs. The reason is ha an ANN uses discriminaive raining o esimae he poserior probabiliy of each acousic-phoneic uni. The minimizaion of he oupu error is performed by means of he Back-Propagaion algorihm ha penalizes he unis wih no observaions in he adapaion se by seing o zero he arge value of he heir oupu unis for every adapaion frame. This arge assignmen policy reduces he ANN capabiliy of correcly classifying he corresponding acousic-phoneic unis. On he conrary, he Gaussian Mixure models wih lile or no observaions remain un-adaped, or share some adapaion ransformaions of heir parameers wih oher similar acousic models, mainaining he knowledge acquired before adapaion

9 To miigae he jus inroduced oblivion problem, i has been proposed [15] o include in he adapaion se examples of he missing classes aken from he raining se. The disadvanage of his approach is ha a subsanial amoun of he raining se mus be sored in order o have enough examples of he missing classes for each adapaion ask. In [14], i has been proposed o approximae he real paerns wih pseudo-paerns raher han using he raining se. A pseudo-paern is a pair of a random inpu acivaion and is corresponding oupu. These pseudo-paerns are included in he se of he new paerns o be learned o preven caasrophic forgeing of he original paerns. The proposed soluions have problems when applied o he adapaion of large ANNs. In fac, here are no crieria for selecing adapaion samples from he raining daa which are ofen no available when adapaion is performed. Moreover, he seleced daa should share some characerisics ha make he adapaion environmen differen from he raining one, bu he elemens of such a difference are ofen unknown. Furhermore, i is unclear how effecive pseudo-paerns can be generaed when he dimensionaliy of he inpu feaures is high. A soluion, called Conservaive Training (CT), is now proposed o miigae he forgeing problem. Since he Back-Propagaion echnique used for MLP raining is discriminaive, he unis for which no observaions are available in he adapaion se will have zero as a arge value for all he adapaion samples. Thus, during adapaion, he weighs of he MLP will be biased o favor he oupu acivaions of he unis wih samples in he adapaion se and o weaken he oher unis, which will always have a poserior probabiliy geing closer o zero. Conservaive Training does no se o zero he value of he arges of he missing unis; i uses insead he oupus compued by he original nework as arge values. Regularizaion as proposed in [12] is anoher soluion o he forgeing problem. Regularizaion has heoreical jusificaions and affecs all he ANN oupus by consraining he nework weigh variaions. Unforunaely, regularizaion does no direcly address he problem of classes ha do no appear in he adapaion se. We esed he regularizaion approach in a preliminary se of experimens, obaining minor improvemens. Furhermore, we found difficul o une a single regularizaion parameer ha could perform he adapaion avoiding caasrophic forgeing. Conservaive Training, on he conrary, akes explicily all he oupu unis ino accoun, by providing arge values ha are esimaed by he original ANN model using samples of unis available in he adapaion se. Le F p be he se of phoneic unis included in he adapaion se (p indicaes presence), and le F m be he se of he missing unis. In Conservaive Training he arge values are assigned as follows: T ( f i T ( f i T ( f i F F (1.0 F m O ) = OUTPUT _ ORIGINAL _ NN ( f O p j F p O m & OUTPUT _ ORIGINAL _ NN ( f & correc( f! correc( f i i O )) = O )) = 0.0 i O ) j O ) (2) - 7 -

10 where T ( f F O ) is he arge value associaed o he inpu paern O for a uni i p ha is presen in he adapaion se. T f F O ) is a arge value associaed o he inpu ( i m paern O for a uni no presen in he adapaion se,. OUTPUT _ ORIGINAL_ NN( f i O ) is he oupu of he original nework (before adapaion) for he phoneic uni i given he inpu paern O, and correc ( f ) i O is a predicae which is rue if he phoneic uni f i is he correc class for he inpu paern O. Thus, a phoneic uni ha is missing in he adapaion se will keep he value ha i would have had wih he original un-adaped nework, raher han obaining a zero arge value for each inpu paern. This policy, like many oher arge assignmen policies, is no opimal. Neverheless, i has he advanage of being applicable in pracice o large and very large vocabulary ASR sysems using informaion from he adapaion environmen, and avoiding he desrucion of he class boundaries of missing classes. I is worh noing ha in badly mismached raining and adapaion condiions, for example in some environmenal adapaion asks, acousically mismached adapaion samples may produce unpredicable acivaions in he arge nework. This is a real problem for all adapaion approaches: if he adapaion daa are scarce and hey have largely differen characerisics SNR, channel, speaker age, ec. oher normalizaion echniques have o be used for ransforming he inpu paerns o a domain similar o he original acousic space. Alhough differen sraegies of arge assignmen can be devised, he experimens repored in he nex secions have been performed using only his approach. Possible variaions, wihin he same framework, include he fuzzy definiion of missing classes and he inerpolaion of he original nework oupu wih he sandard 0/1 arges. 4 Experimenal resuls on arificial daa An arificial wo-dimensional classificaion ask has been used o invesigae he effeciveness of he Conservaive Training echnique. The examples have been designed o illusrae he basic dynamics of he class boundaries. They reproduce he problems due o missing classes in he adapaion se, emphasizing hem. An MLP has been used o classify poins belonging o 16 classes having he recangular shapes shown by he green borders in Figure 3. The MLP has 2 inpu unis, wo 20 node hidden layers, and 16 oupu nodes. I has been rained using 2500 uniformly disribued paerns for each class. Figure 3 shows he classificaion behavior of he MLP afer raining based on Back- Propagaion. In paricular, a do has been ploed only if he score of he corresponding class f i - 8 -

11 Fig. 3. Training 16 classes on a 4-layer nework wih 760 weighs Fig. 4. Adapaion of all he nework weighs. The adapaion se includes examples of class 6 and class 7 only. Adapaion mehod Forgeing miigaion echnique Average classificaion rae (%) Class 6 classificaion rae (%) Class 7 classificaion rae (%) 1. None None Whole nework None Whole nework CT LIN None LIN CT LHN None LHN CT Table 1 Correc classificaion raes on he arificial daa ask

12 Fig. 5. Conservaive Training adapaion of all he nework weighs. was greaer han 0.5. MLP oupus have also been ploed for es poins belonging o regions ha have no been rained, and ouside he green recangles: hey are a he lef and righ sides of Figure 3. The average classificaion rae for all classes, and he classificaion rae for classes 6 and 7, is repored in he firs row of Table 1. Aferward, an adapaion se was defined o simulae an adapaion condiion where only wo of he 16 classes appear. The 5000 poins in his se define a border beween classes 6 and 7 shifed oward he lef, as shown in Figure 4. In he firs adapaion experimen, all he 760 MLP weighs and 56 biases of he nework were adaped. The caasrophic forgeing behavior of he adaped nework is eviden in Figure 4, where a blue grid has been superimposed o indicae he original class boundaries learned by full raining. Classes 6 and 7 do acually show a relevan increase of heir correc classificaion rae, bu hey have a endency o invade he neighbor classes. Moreover, a marked shif oward he lef affecs he classificaion regions of all classes, even he ones ha are disan from he adaped classes. This undesired shif of he boundary surfaces induced by he adapaion process damages he overall average classificaion rae, as shown in he second row of Table 1. To miigae he caasrophic forgeing problem, he adapaion of he nework has been performed using Conservaive Training. Figure 5 shows how he rend of classes 6 and 7 o invade neighbor classes is largely reduced, Class 6 and 7 fi well heir rue classificaion regions, and alhough he lef shif syndrome is sill presen, he adaped nework performs beer as shown by he average classificaion rae in he hird row of Table 1. Our arificial es-bed is no well suied o LIN adapaion because he classes cover recangular regions: hus, a linear ransformaion marix ha is able o perform a single global roaion of he inpu feaures is ineffecive. Moreover, he degree of freedom of his LIN is really poor: he LIN includes 4 weighs and 2 biases only. These consideraions are confirmed by he resuls repored in line 4 of Table 1. Classes 6 and 7 are well classified, bu he average classificaion is very bad because he adapaion of he LIN weighs o fi he

13 Fig. 6. Conservaive Training LIN adapaion Fig. 7. LHN Adapaion Fig. 8 Conservaive Training LHN adapaion

14 boundary beween class 6 and 7, has he caasrophic forgeing effec of enlarging he regions of all classes. The miigaion of hese effecs inroduced by Conservaive Training is shown in Figure 6, and in line 5 of Table 1. The shif oward lef syndrome is sill visible, bu he horizonal boundary surfaces are correc. If we add, insead, a LHN beween las hidden layer and he oupu layer, and we adap is 420 weighs plus biases only, we obain beer resuls han LIN adapaion (see line 6 of Table 1). However, as Figure 7 shows, he class separaion surfaces are ugly. Class 6, and especially class 7 are spread ou, class 3 is spli, and hus he average classificaion rae is unaccepable. Conservaive Training does again a very good job, as shown in Figure 8 and in las line of Table 1, even if class 12 does no presen high scores. 5 Experimenal resuls on speech recogniion asks Adapaion o a specific applicaion may involve he speakers, he channel, he environmenal noise and he vocabulary, especially if he applicaion uses specific liss of erms. The proposed echniques have been esed on a variey of cases requiring differen ypes of adapaion. The adapaion asks ha have been considered are lised in sub-secion 5.1 below. The LOQUENDO defaul speaker and ask independen Ialian models, described in Secion 2, were he seed models for he adapaion. The resuls of our experimens show ha he problem of forgeing is dramaic especially when he adapaion se is no characerized by a good coverage of he phonemes of he language. The use of Conservaive Training miigaes he forgeing problem, allowing adapaion wih a limied performance decrease of he model on oher asks (some performance reducions are ineviable because he ANN is adaped o a specific condiion and hus i is less general). 5.1 Tess on various adapaion asks Applicaion adapaion: Direcory Assisance We esed he performance of models adaped o a Direcory Assisance applicaion. The corpus includes sponaneous uerances of he 9325 Ialian ciy names. The adapaion se has uerances; he es se includes 3917 uerances. Vocabulary adapaion: Command words The liss A1-2-3 of he SpeechDa-2 Ialian corpus, conaining 30 command words, have been used. The adapaion and he es ses include 6189 and 3094 uerances respecively. Channel-Environmen adapaion: Aurora-3 The benchmark is he sandard Aurora3 Ialian corpus. The Well-Mached rain se has been used for adapaion (2951 uerances), while he resuls on he Well-Mached es se (he noisy channel, ch1) are repored (654 uerances)

15 Adapaion ask Adapaion mehod Applicaion Direcory Assisance Vocabulary Command Words Channel-Environmen Aurora3 Ch 1 No adapaion Whole nework LIN LIN + CT LHN LHN + CT Table 2 Adapaion resuls (WER %) on differen asks using various adapaion mehods. The seed adapaion models are he sandard LOQUENDO elephone models. The model are adaped on a given ask and esed on senences of he same domain. Adapaion ask Adapaion mehod Direcory Assisance Adaped Models Command Words Adaped Models Aurora3 Ch1 Adaped Models Whole nework LIN LIN + CT LHN LHN + CT No adapaion 29.3 Table 3 Evaluaion of he forgeing problem: recogniion resuls (WER%) on Ialian coninuous speech wih models adaped on differen asks. The resuls on hese ess, repored in Table 2, show ha a linear ransform on hidden unis (LHN) always ouperforms a linear ransform on he inpu space (LIN). This indicaes ha he hidden unis represen a projecion of he inpu paern in a space where i is easier o learn or adap he classificaion expeced a he oupu of he MLP. The whole nework row in he able corresponds o he adapaion of all he ANN weighs by incremenal raining of he original nework. This adapaion is feasible only if many adapaion daa are available, and i is less effecive han LHN. Conservaive raining imposes some consrains o he adapaion process because i ries o preven forgeing missing classes. Thus, he performance of he LIN/LHN models, adaped on a given ask and esed on senences of he same domain, are slighly beer han he performance of he corresponding models adaped by means of CT in addiion o LIN/LHN. This happens because, using a large adapaion se, here are enough samples for mos of he

16 oupus. Thus, here is lile or no risk of caasrophic forgeing, and here is no need for he esimaes obained when inpu samples of differen unis are applied a he nework inpu. Conservaive raining, however, preserves he generaliy of he adaped model. This claim has been assessed by using a model adaped on a ask and evaluaing is recogniion performance on a differen generic common ask. The generic ask is large vocabulary (9.4k words) coninuous speech recogniion of 4296 phoneically balanced senences uered by 966 speakers. The average senence lengh is 6 words. The resuls are obained wihou language modeling. The adapaion asks and he reference word error rae (29.3 %) achieved using unadaped acousic models on he generic ask are given in he firs and las row of Table 3 respecively. Since he adaped models have been specialized o a specific condiion, hey are no expeced o perform as well as he original model ask independen - on a coninuous speech ask. I was ineresing, however, o have a measure of he generalizaion loss afer adaping he sandard model wih differen approaches. Table 3 highlighs he effecs of caasrophic forgeing, which akes place when he vocabulary of he adapaion se is small and has a poor phoneic coverage. This is paricularly eviden for he models adaped on he Command words and Aurora 3 asks whose resuls on he generic coninuous speech recogniion ask are emphasized in ialics. If he adapaion se has enough daa for each class appearing in he es se of he new ask, hen LIN or LHN approach perform well wihou CT. The resuls repored in he firs column of Table 3 confirm ha he use of CT is no relevan when he model has been obained using he phoneically rich adapaion se provided by he uerances colleced for a Direcory Assisance ask. On he oher hand, if he es se includes phoneic classes ha are missing in he adapaion se, he use of CT in addiion o LIN/LHN produces beer resuls avoiding he caasrophic forgeing. This is shown in he second and hird columns of Table 3, where he es se is sill large vocabulary coninuous speech, bu he models have been adaped using command words and digis respecively. Conservaive Training, hus, miigaes he forgeing problem, preserving an accepable performance of he adaped model on he ask for which he original nework was rained (open vocabulary, ask independen speech recogniion). The phoneic classes ha were rarely represened, or were missing, in he adapaion se can be sill reasonably well recognized by he adaped model. 5.2 Speaker Adapaion Furher experimens have been performed on he WSJ0 speaker adapaion es in several condiions. Three baseline models have been used: he defaul LOQUENDO 8 khz elephone speech model (rained wih LDC MACROPHONE [19] referred as MCRP in he Tables); a model rained wih he WSJ0 rain se (SI-84), 16 khz. a model rained wih he WSJ0 rain se (SI-84), down-sampled o 8 khz. Furhermore, we esed wo archiecures for each ype of models: he sandard one (STD), described in sub-secion 2.1 and an improved (IMP) archiecure, characerized by a wider inpu window modeling a ime conex of 250 ms [20], and by he presence a hird 300 unis hidden layer

17 Train Se Ne ype Adapaion mehod Bigram LM Trigram LM NO adapaion Sandard LIN MCRP STD LIN+CT LHN+CT LIN+LHN+CT NO adapaion Sandard LIN WSJ0 STD LIN+CT LHN+CT LIN+LHN+CT NO adapaion Sandard LIN WSJ0 IMP LIN + CT LHN + CT LIN+LHN+CT Table 4 Speaker Adapaion word error rae on he WSJ0 8 khz ess using differen adapaion approaches. Train Se Ne ype Adapaion mehod Bigram LM Trigram LM NO adapaion Sandard LIN STD LIN+CT LHN+CT WSJ0 LIN+LHN+CT NO adapaion Sandard LIN IMP LIN+CT LHN+CT LIN+LHN+CT Table 5 Speaker Adapaion word error rae on he WSJ0 16 khz ess using differen adapaion approaches. The adapaion se is he sandard adapaion se of WSJ0 (si_e_ad, 8 speakers, 40 uerances per speaker), down-sampled o 8 khz when necessary. The es se is he sandard SI 5K read NVP Senneheiser microphone (si_e_05, 8 speakers x ~40 uerances). The bigram or rigram sandard Language Models provided by Lincoln Labs have been used in hese experimens

18 The resuls, repored in Tables 4 and 5, show ha also in hese ess LHN always achieves beer performance ha LIN. The combinaion of LIN and LHN (rained simulaneously) is usually beer han he use of LHN alone LHN LHN+CT WER % Adapaion senences Figure 9 Word error rae on he WSJ0 8 khz ess as a funcion of he amoun of adapaion daa (1, 5, 10, 20, 30, and 40 senences). Conservaive raining effecs are of minor imporance in hese ess because he adapaion se has a good phoneic coverage and he problem of unseen phoneic classes is no dramaic. Neverheless, is use improves he performance (compare Sandard LIN and LIN+CT), because i avoids he adapaion of prior probabiliies of he phoneic classes on he (poor) prior saisics of he adapaion se. Finally, Figure 9 shows how CT influences he performance of models adaped wih a differen amoun of adapaion daa. The resuls, obained wih a bigram language model, refer o LHN adapaion of he sandard models (STD) using up o 40 senences of each speaker. The senences are down-sampled o 8 khz. The firs poin in he graph shows he 13.4% WER achieved using he sandard model wihou adapaion. Conservaive raining is paricularly useful when he adapaion se is very small (a few senences), because in ha case he problem of he missing phoneic classes is more relevan. In paricular, using a single senence he error rae for LHN adapaion alone acually increases. The experimens show an overall benefi of CT, considering ha he minimal performance degradaion repored for he improved nework models adaped wih LIN has been obained using he rigram LM, which could mask he acousic qualiy of he models. 6 Conclusions A mehod has been proposed for adaping all he oupus of he hidden layer of ANN acousic models and for reducing he effecs of caasrophic forgeing when he adapaion se does no conain examples for some classes. Experimens have been performed for he adapaion of an exising ANN o a new applicaion, a new vocabulary, a new noisy environmen and

19 new speakers. They show he benefis of CT, and ha LHN ouperforms LIN. Furhermore, experimens on speaker adapaion show ha furher improvemens are obained by he simulaneous use of LHN and LIN showing ha linear ransformaions a differen levels produce differen posiive effecs ha can be effecively combined. An overall WER of 5% afer adapaion on WSJ0 using he sandard rigram LM and wihou across word specific acousic models compares favorably wih published resuls. References [1] Gauvain, J. L., Lee, C. H., Maximum a poseriori esimaion for mulivariae gaussian mixure observaions of Markov chain. IEEE Transacions on Speech and Audio Processing, Vol. 2, n. 2, pp [2] Gales, M.J.F., Maximum likelihood linear ransformaions for HMM-based speech recogniion. Compuer Speech and Language, Vol. 12, pp [3] Kuhn, R., Junqua, J.-C, Nguyen, P., Niedzielski N., Rapid speaker adapaion in eigenvoice space, IEEE Transacions on Speech and Audio Processing, Vol. 8, no. 4, pp [4] Sagayama, S., Shinoda, K., Nakai, M., Shimodaira, H., Analyic mehods for acousic model adapaion: A review. Proc. Adapaion Mehods for Speech Recogniion, ISCA ITR-Workshop, pp [5] Lee, C.-H., Huo, Q., On adapive decision rules and decision parameer adapaion for auomaic speech recogniion. Proc. IEEE, vol. 88, no. 8, pp [6] Hsiao, R., Mak, B., Discriminaive feaure ransformaion by guided discriminaive raining. Proc. ICASSP-04, Monreal, pp [7] Liu, X., Gales, M.J.F., Model complexiy conrol and compression using discriminaive growh funcions. Proc. ICASSP-04, pp [8] Abrash, V., Franco, H., Sankar, A., Cohen, M., Connecionis speaker normalizaion and adapaion. Proc. EUROSPEECH 1995, pp [9] Neo, J., Almeida, L., Hochberg, M., Marins, C., Nunes, L., Renals, S., Robinson, T., Speaker-adapaion for hybrid HMM-ANN coninuous speech recogniion sysem. Proc. EUROSPEECH 1995, pp , [10] Sadermann, J., Rigoll, G., Two-sage speaker adapaion of hybrid ied-poserior acousic models. Proc. ICASSP-05, Philadelphia, pp. I [11] Dupon, S., Cheboub. L., Fas speaker adapaion of arificial neural neworks for auomaic speech recogniion. Proc. ICASSP-00, pp [12] Li, X., Bilmes, J., Regularized adapaion of discriminaive classifiers. Proc. ICASSP-06, pp [13] French, M., Caasrophic forgeing in connecionis neworks: causes, consequences and soluions. Trends in Cogniive Sciences, 3(4), pp [14] Robins, A., Caasrophic forgeing, rehearsal, and pseudo-rehearsal. Connecion Science, 7, [15] BenZeghiba, M.F., Bourlard, H Hybrid HMM/ANN and GMM combinaion for usercusomized password speaker verificaion. Proc. ICASSP-03, pp [16] Palle, D. S., Fiscus, J. G., Fisher, W. M., Garofolo, J. S., Lund, B. A., Przybocki, M. A., Benchmark ess for he ARPA spoken language program. Proc. of he Human Language Technology Workshop, pp [17] Albesano, D., Gemello, R., Mana, F., Hybrid HMM-NN modeling of saionary-ransiional unis for coninuous speech recogniion. Proc. Neural Informaion Processing, pp [18] Hermansky, H., Ellis, D., Sharma, S., Tandem connecionis feaure exracion for convenional HMM sysems. Proc. ICASSP-00, pp [19] Available a hp://

20 [20] Dupon, S., Ris, C., Couvreur L., Boie, J. M., A sudy of implici and explici modeling of coariculaion and pronunciaion variaion. Proc. Inerspeech-05, pp