HIDDEN MARKOV MODELS FOR AUTOMATIC SPEECH RECOGNITION: THEORY AND APPLICATION. S J Cox

Transcription

1 HIDDEN MARKOV MODELS FOR AUTOMATIC SPEECH RECOGNITION: THEORY AND APPLICATION S J Cox ABSTRACT Hdden Markov modellng s currenly he mos wdely used and successful mehod for auomac recognon of spoken uerances. Ths paper revews he mahemacal heory of hdden Markov models and descrbes how hey are appled o auomac speech recognon.. Inroducon Machne ranscrpon of fluen speech - he so-called 'phonec ypewrer' - remans a challengng research goal. A machne capable of performng hs ask would requre an enormous amoun of knowledge abou he real world workng n conuncon wh a mechansm for decodng acousc sgnals. However, auomac speech recognon (ASR) has reached he pon where, gven some resrcons on speakers, vocabulary and envronmenal nose, relable recognon of solaed words or shor phrases s possble. A dsncon s generally made n ASR beween recognon of uerances from a speaker who has prevously 'enrolled' hs voce (speaker dependen recognon) and a speaker whose voce he recognser has never 'heard' prevously (speaker ndependen recognon). To some exen, s possble o rade vocabulary sze for speaker dependence, so ha s currenly feasble o buld eher a speaker ndependen ASR devce ha recognses a small vocabulary (5-20 words or phrases), or a speaker dependen devce ha recognses a large vocabulary ( words). For elephony applcaons, enrolmen free ASR s clearly crucal and urns ou ha many useful applcaons are possble wh a small vocabulary. Broadly speakng, aemps a ASR fall no wo caegores - a knowledge-based approach, n whch knowledge abou speech from he domans of lnguscs and phonecs s used o consruc a se of rules whch s n urn used o nerpre he acousc npu sgnal, and a 'paern -machng' approach n whch a pror knowledge abou speech s largely gnored and echnques of paern classfcaon are appled o he npu sgnal. The knowledge-based approach s able o explo a body of knowledge abou speech and, n parcular, he relaonshp beween feaures exraced from he speech sgnal and hgher level lngusc represenaons (e.g. phonemes, syllables). However, hs relaonshp, whch s very complex, s sll far from beng undersood, largely

2 because of he enormous varably of sgnals nerpreed by he bran as represenng he same lngusc uns. Of he knowledge ha s avalable, s no ye clear how bes o represen or use n a compuaonal framework. The paern-machng approach makes no aemp o use hs knd of knowledge, bu gahers s 'knowledge' of he speech sgnal n a sascal form by beng shown examples of speech paerns. As such, would work equally well on any acousc paerns - brdsong', machnery nose, sesmc sgnals ec. However, powerful mahemacal echnques are avalable whch are guaraneed o opmse he echnque and hese ensure ha he approach s surprsngly successful, even hough almos all knowledge of speech producon, percepon and he speech sgnal s gnored. Furhermore, hese echnques are applcable o paerns a any level and hence can be used o opmally decode oher represenaons of speech sgnals (e.g. phonec segmens, words), hus provdng a coheren framework for speech recognon and undersandng. Ths paper focuses exclusvely on he paern machng approach whch s used by all commercal ASR devces. In parcular, wo echnques have been found o be especally approprae o ASR - dynamc me warpng (DTW) and hdden Markov modellng. DTW s a specal case of hdden Markov modellng and s dscussed n Appendx A. A hrd connecons approach s now evolvng, based on adapve parallel dsrbued processng neworks [] 2. Paern classfcaon Fgure shows a block dagram of a paern classfer n ranng and recognon modes. Fg : A paern classfer n ranng and recognon modes The wo approaches o ASR have been phly summarsed by one researcher as 'dong he rgh hng wrong or he wrong hng rgh'. 2

3 In ranng mode, many examples of each class are used o buld a model for he class, and hese models are subsequenly sored. In recognon mode, a paern of unknown class s compared wh each model and classfed accordng o he model o whch s 'closes'. When performng recognon of solaed words, each dfferen word s regarded as a class. The feaure exracon shown n Fg s necessary for wo reasons. Frsly, enables focusng on nformaon whn he sgnal whch s mporan for dscrmnang beween paerns of dfferen classes; a good se of feaures '...enhances whn-class smlary and beween-class dssmlary' [2]. Secondly, enables daa reducon so ha manpulaon of paerns becomes compuaonally feasble. (A elephone speech sgnal has a ypcal daa-rae of bs/s, oo hgh for praccal compuaon.) Feaures for speech recognon are generally eher relaed o he nsananeous specrum of he speech sgnal or o he nsananeous shape of he vocal rac; clearly, here s a large overlap of nformaon n hese represenaons. Feaure selecon s of grea mporance n speech recognon, as accuracy s hghly dependen on he ype and number of feaures used [3]. Because of he sluggshness of he speech arculaors, an adequae represenaon of he speech paern can be made by measurng feaures a regular nervals of approxmaely /00 s. Each sample s a d dmensonal vecor of feaures, so ha an uerance of lengh seconds s reduced o a sequence of T 00 * d-dmensonal analyss vecors. In hs paper, uerances are consdered o be solaed words and so he approach o recognon s somemes called whole word paern machng [4]. 3. Speech paern classfcaon usng hdden Markov models Speech dffers from mos sgnals deal wh by convenonal paern classfers n ha nformaon s conveyed by he emporal order of speech sounds. A sochasc process provdes a way of dealng wh boh hs emporal srucure and he varably whn speech paerns represenng he same perceved sounds. Sochasc processes '...develop n me or space n accordance wh probablsc laws' [5]; a sochasc process ha has been found o be parcularly useful n ASR s a hdden Markov model (HMM). 3. Descrpon of a hdden Markov model Fgure 2 shows an example HMM. 3

4 Fg 2 A 5-sae lef-rgh hdden Markov model wh 4 oupu symbols. The fve crcles represen he saes of he model and, a a dscree me nsan, he model occupes one of he saes and ems an observaon. A nsan +, he model eher moves o a new sae or says n he same sae and ems anoher observaon, and so on. Ths connues unl a fnal ermnang sae s reached a me T. An mporan characersc of hs process s ha he sae occuped a me nsan + s deermned probablscally and depends only on he sae occuped me - hs s he Markov propery. The probables of movng from sae o sae are abulaed n a N N sae ranson marx, A [ a ] - an example A s shown n Fg 2. The, h enry of A s he probably of occupyng sae s a me + gven sae s a me. Snce he oal probably of makng a ranson from a sae mus be.0, each row of A sums o.0. Noce also ha A s upper rangular, because hs parcular HMM s a lef-o-rgh model n whch no 'backwards' umps are allowed and he model progresses hrough he saes n a lef-rgh manner. In a generalsed HMM, a ranson s possble from any sae o any oher sae, bu here s clearly no pon n usng anyhng bu a lef-o-rgh HMM for ASR as only lef-o-rgh models wll effecvely model he emporal orderng of speech sounds. The opology of he model shown n Fg 2 s raher smple, and rcher opologes n whch saes may be skpped are used o advanage n ASR. An observaon emed a a me nsan by he HMM shown n Fg 2 can be one of only 4 symbols, A, B, C or Z. In general, a model can em any of a fne alphabe of M symbols ( v, v2,..., vm ) from each sae, and he probably of emng symbol v k from sae s s gven by he, k h enry of a N M marx B [ b k ] (an example B s 4

5 shown n Fg 2). A fnal parameer needed o se he model n moon s a vecor π whose h componen π () s he probably of occupyng sae a - agan, an example s gven. An observaon sequence s produced by he model of Fg 2 as follows. Use he vecor π and a random number generaor (RNG) o deermne whch sae he model sars n - assume hs s sae s. Se. Use he probables n row of B and he RNG o selec a symbol v k o oupu. Use he probables n row of A and he RNG o deermne whch sae s o occupy nex. Se +. Repea he second and hrd seps unl a ermnang sae s reached. (Sae 5 n he model of Fg 2 s such a sae, because has a self ranson probably of.0 and only oupus he symbol Z. Hence he observaon sequence would be an nfne successon of Z s once sae 5 s reached.) A ypcal observaon sequence produced by hs process acng on he model of Fg 2 mgh be: CCBCCBMACAAACACBBBZ. Noce ha he saes ha produced each symbol are hdden n he sense ha, gven an observaon sequence, s n general mpossble o say wha he sae sequence ha produced hese observaons was. An algorhm whch fnds he mos lkely sae sequence gven an observaon sequence and a model s nroduced laer. If a one-o-one correspondence beween saes and symbols exss, he process s known as a Markov chan. 5

6 3.2 Relaonshp of HMMs o speech producon and recognon. The above descrpon of an HMM was delberaely kep absrac, bu he reader may already have an nklng of how he process s relaed o speech producon. Le us make he followng premse abou he producon of an uerance of a parcular word: an uerance s produced by he arculaors passng hrough an ordered sequence of 'saonary saes' of dfferen duraon; he 'oupus from each sae (.e. he observaons) can be regarded as probablsc funcons of he sae. Ths s a crude and drascally smplfed model of he complexes of speech; speech s a smooh and connuous process and does no ump from one arculaory poson o anoher. However, he success of HMMs n speech recognon demonsraes ha f he model s correcly opmsed on speech daa, capures enough of he underlyng mechansm o be powerful. The observaon sequence of he HMM s hus he sequence of analyss vecors descrbed n secon 2. The model descrbed n secon 3. can oupu only a fne alphabe of symbols bu he analyss vecors are connuously valued - hence each analyss vecor mus be mapped o a symbol by he process of vecor quansaon [6]. Ths process causes an nevable loss of accuracy and wll laer be seen how HMMs can be exended o handle connuous dsrbuons raher han dscree symbols. The correspondence of HMM saes o arculaory saes s by no means clear cu because s dffcul o defne wha one means by a 'sae' n a smoohly produced uerance. The closes physcal correspondence o a sae mgh be he arculaory poson n a long vowel sound. Alhough hs correspondence s of consderable heorecal neres for fuure work n modellng speech sgnals, s no necessary o aemp o make for he purposes of performng speech recognon, and s no of presen concern. There are wo problems n he applcaon of HMMs o solaed word speech recognon. Gven a se of uerances of a vocabulary of words (he ranng se), consruc one or more HMMs for each word (ranng). Gven a se of HMMs, one or more for each word n he vocabulary, classfy an unknown uerance (recognon). The recognon problem s solved by compung he lkelhood of each model emng he observaon sequence correspondng o he unknown uerance and assgnng he unknown uerance o he class of he model whch produced he greaes lkelhood. The ranng problem s more dffcul, bu a powerful algorhm exss (he Baum-Welch algorhm) whch guaranees o fnd a locally opmal model. These wo problems are consdered n deal n secons 4. and

7 4. The recognon problem The recognon problem s ackled frs because s he more sraghforward of he wo and he ranng problem bulds on some of he defnons nroduces. Frsly, a more formal defnon of some of he noaon already used s gven below: 2 S h sae of he HMM,2, N k v k h oupu symbol n he alphabe k,2, M a Pr( sae + sae ),2, N,2, N b k Pr(oupung symbol v k from sae s ),2, N k,2, M O h observaon (analyss vecor),2, T b O ) Pr(oupung observaon O from sae s ) ( b k when O vk w he h word n he regocnon vocabulary,2, W The hdden Markov model M s fully defned by he parameer se [ π, A, B]. We are gven W such models, M, M 2, K, MW, one for each word n he vocabulary, and an unknown uerance O, whch consss of a sequence of T observaons O, O2, K, OT ; each O s one of he symbols v, v2,..., vm. Recognon s acheved by compung he lkelhood of each model havng produced O,.e. by compung Pr ( O M ),,2, K W and assgnng O o class k where max Pk Pr ( O M,2, KW 4. Baum- Welch recognon ) The mos obvous way of compung Pr( O M ) s o consder every possble sequence of saes ha could have generaed he observaon sequence and fnd he one whch produces he hghes Pr ( O M ). However, s easy o see ha hs s unrealsc, as n T general here are N possble sequences. Ths number s consderably reduced f A s T 20 sparse, bu s sll mpossbly large for praccal purposes ( N 9 0 for N 5, T 30 ). Forunaely, a recursve algorhm exss o calculae Pr ( O M ). The algorhm depends upon calculang he so-called forward probables, whch are he probables of he on even of emng he paral observaon sequence O, O2, K, O and 2 The noaon used here generally follows ha of [7] and [8]. 7

8 occupyng sae s a me. These probables are laer used n he Baum-Welch ranng algorhm and so he assocaed Pr ( O M ) s denoed by P BW. Le he forward probables be denoed by α ( ).e. α (, ) Pr( O, O2 O, sae me M ).(),2, KN I should hen be clear ha he requred probably s: P BW N α ( ) T.(2) s he probably of emng O and endng n sae s. The snce αt ( ) compued recursvely, as follows. Suppose some nsan, hen ( O, O2, KO, sae me + sae me, M ) α Pr ( ) a α s can be α ( ),,2, KN, has been compued a Hence he probably of occupyng sae s a me + s: N, O2, KO, sae me + M ) ( ) Pr( O α Fnally, accounng for observaon O + from sae s gves: a N α ( ) ( ) ( ),2, + α a b O + KT.(3) Fgure 3 llusraes hese seps by showng he compuaon of α +(4) for a sx sae HMM. Snce α ) Pr( O, sae ), he recurson n equaon (3) s nalsed by ( M ( ) π b ( O seng α ). 4.2 Verb recognon BW In he above calculaon of P eachα +, s calculaed by summng conrbuons of α BW from all saes and hence P s he lkelhood of emngo, summed over all sae sequences. The Verb algorhm compues he lkelhood P v of he mos lkely sae sequence emngo ; by backrackng, hs sequence can also be recovered. 8

9 Ths lkelhood P v, and he assocaed mos lkely sae sequence S T, are found by compung φ ( ), where: φ ( ) max φ ( ) a b ( O+ ),2, KN,2, K + T.(4) Fg 3 Compuaon of α +(4) for a sx sae HMM. Equaon (4) s dencal o equaon (3) excep he summaon has been replaced by he max operaor. The probably of oupungo s hen gven by: P v max φ ( ),2, KN T Compare equaon (5) wh equaon (2). If s desred o recover he mos lkely sae sequence, ψ ( ) s also recorded, where ψ ( ) s he mos lkely sae a me gven sae s a me. Hence, ψ maxmses he RHS of equaon (4). ( ) Havng calculaed all heφs andψs for where s he number of he sae whch,2, KN and,2, T he backrackng proceeds as follows: he mos lkely sae a met s sae s k, where k maxmses he RHS 9

10 of equaon (5); hence ψ (k) gves he mos lkely sae a met, from whch he mos lkely sae a met 2s found and so on unl he mos lkely sae a s recovered. Fgure 4 gves an example of he mos lkely sae sequence for a 20 frame uerance and a sx sae HMM. Fg 4 A Verb sae sequence for an observaon sequence of lengh 20 and a sx sae hdden Markov model. v BW Noce ha P P, wh he equaly sasfed only when he sae-sequence producng he observaons s unque. 5. The ranng problem 0

11 The prncpal reason for usng HMMs n solaed word recognon s o capure sascal nformaon abou he varably n paerns represenng he same word, and so a ranng echnque ha opmses he model for a large number of uerances (observaon sequences) s requred. However, assume for he momen a sngle observaon sequenceo for each word; he exenson o mulple observaon sequences s sraghforward (secon 5.2). The essenal seps n he ranng algorhm for a gven word are as follows: an nal esmae of M s made; he re-esmaon algorhm ando are used o generae a new model M', wh he propery ha Pr( O M' ) Pr( O M ) ; M' hen plays he rôle of M and a new esmae s deermned. Ths process eraes unl he nequaly n he above expresson s arbarly small. Two reesmaon algorhms, he Baum-Welch and Verb algorhms, are dscussed here. 5. The Baum- Welch (forward- backward) algorhm The underlyng dea behnd hs algorhm s ha, gven some esmae M of he model and an observaon sequenceo, he bes esmaes of he parameers of a new model M' are: Pr(ranson from s o sae ) ' s M a.(6) Pr(ranson from s o any sae M) Pr( emng symbol v k from sae s M ' b k.(7) Pr(emng any symbol from sae s M ' π Pr(observaon sequence begns n sae s M.(8) The remarkable propery of he Baum-Welch algorhm s ha he above re-esmaes of A, B and π are guaraneed o ncrease Pr( O M ) unl a crcal pon s reached, a whch pon he parameers do no change. To compue hese quanes, he forward probables α ( ) are complemened, by defnng backward probables β () as follows: β ( ) Pr(O, O 2, OT In oher words, β (), sae me + + M ) s he probably of sarng n sae s a me and hen compleng O ). The reader he observaon-sequence (N.B. begnnng wh observaono +, no should have no dffculy n convncng hmself ha, usng smlar reasonng o he calculaon of equaon (3), β () can be calculaed by he followng backward recurson: N + + β ( ) ab ( O ) β ( ) T, T 2, K.(9)

12 Wh β T ( ),,2, N. The numeraor of equaon (6) s hen gven by: Pr(ranson from sae s o sae s M ) T α ( ) a b ( O + ) β + ( ) Ths s bul up as follows: α () gves he probably of beng n sae S a me ; ab ( O+ ) accouns for movng o anoher sae s and oupung symbolo+ from hs sae; β +( ) accouns for occupyng sae s a me + and hen compleng he sequence. Ths probably s summed over all mes a whch s possble o make a ranson.e. from o T (N.B. not ). To calculae he denomnaor of equaon (6) recall ha: α ( ) Pr( O, O2, O, sae me M ) β ) Pr(O, O, O, sae me ( T M Hence α ( ) β ( ) s he probably of occupyng sae s a me gven M. So he denomnaor of equaon (6) s: Pr(ranson from sae s o any sae ) M α ( ) β ( ) and he complee equaon s expressed n erms of he forward and backward probables as: T ) a ' T α ( ) ab ( O+ ) β + ( ) T.(0) α ( ) β ( ) A smlar reasonng leads o expressng equaon (7) as: b ' k O v α ( ) β ( ) k T.() α ( ) β ( ) 2

13 The summaon of he numeraor of he above equaon s read: 'Sum over alls a whcho s symbolv k '. Fnally, he re-esmaon ofπ s shown o be: π α ( ) β ( ).(2) P BW Alhough equaons (6), (7) and (8) seem a plausble way of updang he model parameers, no proof has been gven ha hey are guaraneed o ncrease Pr ( O M ). The reader neresed n rgorous proofs wll fnd hem n Lporace [9]. 5.2 Tranng on mulple observaon sequences Equaons (0), () and (2) are exended o perm re-esmaon on many observaon sequences by smply consderng he defnons n equaons (6), (7) and (8) o ac over all he observaon sequences. However, gven a ceran model, each observaon BW sequence wll have a dfferen P and he probables from sequences havng low P BW wll gve a dsproporonaely low conrbuon n equaons (6), (7) and (8) - he resul s ha he model s opmsed only for uerances havng hgh P BW. The soluon s BW ' o wegh all probables by / P. Denong equaon (0) as a N / D, he change s: a ' U l U l P P BW l BW l N D l l.(3) N l and D l, are he numeraors and denomnaors formed when processng observaon BW sequence (uerance) O l, Pl s Pr( O l M ) andu s he oal number of uerances n he ranng se. Smlar aleraons-apply o equaons () and (2). When appled o mulple observaon sequences, he Baum-Welch algorhm s guaraneed o ncrease U P BW l l 5.3 Verb ranng The Verb algorhm can be used for model re-esmaon as well as for recognon. In hs case, he backward probables ( β s ) are no requred and here s a sgnfcan compuaonal savng. Concepually, he Verb ranng procedure s smple: he Verb algorhm s used o segmen each uerance accordng o he curren model, 3

14 as n Fg 4. The new values of [π, A, B ] are hen derved drecly by examnng he numbers of ransons o and from each sae and he symbols oupu by each sae. The procedure s as follows. 0 Make an nal esmae of he model, M M. Usng model, M execue he Verb algorhm on each of he 2 U observaonso, O, KO. Sore he se of mos lkely sae sequences produced, 2 S, S, S L U l P U and se O ). V l ( M Use he Verb re-esmaon equaons ((4) (5) and (6) below) o generae a new M. Ierae he second and hrd seps unl he ncrease n L upon each eraon s arbarly small. 2 The re-esmaes are gven by consderng all he sequences S, S, S (No of ransons from sae s o sae s M ) a.(4) (Toal number of ransons ou of sae s M ) (No of emssons of symbol vk from sae s M ) b k.(5) (Toal number of symbols emed from sae s M ) U and seng: ( No observaon sequences begnng n sae s M ) π U.(6) Noe ha weghng by/ P s no requred here snce numbers of ransons are consdered raher han probables of ransons. As wh he Baum-Welch algorhm, U hs procedure s guaraneed o ncrease P v l l a each eraon. 6. Exenson o connuous probably densy funcons So far, has been assumed ha he HMM mus oupu one of a fne alphabe of M symbols. Ths means ha he observaon vecors from he speech sgnal mus be quanzed, wh an nheren loss of accuracy. If s aemped o mnmse hs quansaon dsoron by usng a large number of quansaon symbols, a large number of probables n he marx B mus hen be re-esmaed (for nsance, wh 28 symbols and 0 saes n he HMM, s necessary o re-esmae 280 probables on each 4

15 eraon of he ranng algorhm). There s rarely enough ranng daa avalable o esmae such a large number of parameers. If can be assumed ha he observaon vecors from a parcular sae are drawn from some underlyng connuous probably dsrbuon, he nformaon n he correspondng row of B can be replaced by he parameers of hs dsrbuon. Ths s equvalen o makng a Normal approxmaon o some hsogram daa and replacng he ndvdual probables n he hsogram by he mean and varance of he Normal dsrbuon. The queson s hen mmedaely rased of how well he observed daa fs he paramerc dsrbuon. Expermenally, he analyss vecors n a parcular sae end o cluser raher han produce a smooh dsrbuon [8]. However, has been shown ha an arbrary mulvarae connuous probably densy funcon (PDF) can be approxmaed, o any desred accuracy, by mxures (weghed sums) of mulvarae Gaussan PDFs. I s herefore approprae o model he probably dsrbuon of each sae as a Gaussan mxure (alhough he re-esmaon formulae have been proved for any log-concave dsrbuon [9]). The probably b ( O ) of emng an analyss vecoro from sae s s hen: X b ( O ) c m N[ O, µ m, U m ],2, N m N[ O, µ m, U m ] where X s he number of mxures, c m s he wegh of mxure m n sae s and N[ O, µ m, U m ] s he probably of drawng he vecoro from a mulvarae Normal dsrbuon wh mean-vecor µ m and covarance marxu 3 m. X Clearly, O c m andc m. m 6. Re-esmaon wh connuous mxure dsrbuons How does he change from dscree o connuous dsrbuons affec he equaons bul up n he prevous secons? The only change n he equaons n secons s ha b ( O ) s now defned as equaon (7), so ha once b ( O ) has been calculaed for eacho, he recognon algorhms (Baum-Welch or Verb) are unaffeced. 3 For readers unfamlar wh mulvarae Normal dsrbuon, a bref revew s gven n Appendx A. 5

16 ' The re-esmaon formulae (boh Baum-Welch and Verb) for a and π ' are smlarly only affeced by he change n he defnon of b ( O ). However, he ranng procedure mus now re-esmae c m, µ m, U m, m,2, X for each sae s. To cope wh a number of mxures n each sae, a hrd funcon augmens he forward and backward probables: ρ (, m) Pr( O, O2, O, sae s, mxure m@ me M ).(8) Comparng equaons () and (8), s clear ha: X ρ (, m) α ( ) m The mxure-weghs for each sae are hen calculaed as: c ' m Pr ( mxure m, sae s Pr (sae s M ) M ) T T ρ (, m ) β ( ) α ( ) β ( ) The re-esmae of he mean vecor of mxure m n sae s s: T ρ (, m) β ( ) O ' µ m T.(9) ρ (, m) β O Noce ha f X (.e. a sngle Gaussan dsrbuon per sae), equaon (9) becomes: ' µ T T T α ( ) β ( ) O α ( ) β ( ) Pr(sae s Pr(sae M ). O.(20) M ).(2) 6

17 The re-esmae of he mean vecor of a sae s hus an average of each of het analyss vecors, weghed by he probably of occupyng he sae a me. The elemens of he covarance marx are found n he usual way by consderng he mean vecor µ m and het analyss vecors. In pracce, a dagonal covarance marx s usually used because of he dffculy n esmang d( d +) / 2 componens for each mxure and sae wh lmed ranng daa. Proofs of all he re-esmaon equaons for he case of a sngle mxure densy per sae are gven n Lporace [9]. The Verb algorhm can also be exended o re-esmae he parameers of connuous mxure dsrbuons. 7. Some consderaons of mplemenaon Secons 2-6 have presened he mahemacal heory of HMMs. Ths secon dscusses some of he praccal ssues rased n usng HMMs for auomac speech recognon. 7. Prevenng underflow I s apparen ha many of he equaons n secons 4 and 5 wll underflow que rapdly on real compuers as hey nvolve recursve producs of small probables. One soluon s o nroduce scalng erms a ceran pons n he recurson [7]. A seemngly more drec mehod s smply o use logarhms hroughou. However, s necessary o add erms n many of he equaons, so a funcon reurnng log( a + b) gven log a and logb s requred. Ths funcon requres an exponenaon and a logarhm so ha compuaonally, here s lle o choose beween hese soluons. 7.2 Zero symbol probables When usng dscree HMMs wh fne ranng daa, somemes happens ha he probably of observng a ceran symbol vk from sae s s 0. If hs occurs, anyα, β orφ of an equaon n secon 5 wll be 0, whch wll be faal o boh recognon and ranng algorhms. The obvous, alhough somewha nelegan soluon, s o se he offendng probably o some small bu non-zero value,ε. Ths suaon has been analysed heorecally by Levnson e al [ 7 ] who also ran some expermens on he effec on recognon accuracy of he value ofε [0]. Ther conclusons were ha he 0 3 performance was almos dencal usng values n he range 0 0, for M Inal model esmae The Baum-Welch re-esmaon algorhm s a hll-clmbng algorhm whch converges o a locally opmal model; hence he fnal model wll depend on he nal model. The queson hen arses of how o make a good nal esmae of A and he oupu parameers, B (n he dscree case) or c m, µ m, U m (n he connuous case). The nal 7

18 esmae of A s chefly deermned by he opology chosen for he model, bu he nal esmae of he oupu parameers can be based upon he ranng daa. Rabner, Levnson e al have shown ha connuous HMMs are very sensve o poor nal esmaes of means []; herefore seems wse o base he nal esmae of he oupu parameers on he avalable ranng daa. Two sensble mehods of obanng nal esmaes of he oupu parameers are as follows. Unform segmenaon: each uerance n he ranng se s paroned no N segmens of equal lengh and he analyss vecors n each segmen are pooled. In he case of dscree dsrbuons, hey are hen quanzed and an esmae Of b k s made: b k No of vecors of symbol vk n segmen Toal no of vecors n segmen and covarance marces are made by applyng a cluserng algorhm o he vecors of segmen (sae s ) o produce X clusers. µ m andu m are hen respecvely he mean vecor and covarance marx of cluser m n he sae. Opmal segmenaon: each uerance n he ranng se s paroned no N segmens accordng o he algorhm of Brdle and Sedgwck [2]. Ths algorhm fnds he opmal segmenaon of he uerance no N segmens n he sense ha, f he mean vecor of each segmen s compued, he sum of he nrasegmen varances s mnmsed. The vecors n each segmen are pooled and he esmaes are made as descrbed above. All elemens of A are se nally equal o/ K wherek s he number of allowed ransons (non-zero elemens) n row, and smlarly, he frsl elemens ofπ may be se o / l. 8. Summary Hdden Markov models provde a coheren framework for dealng wh varably n speech paerns. Alhough he assumpons abou speech made by HMMs are crude, hey are offse o a grea exen by he avalably of powerful opmsaon echnques. An aracve feaure of HMMs s ha s possble o exend and mprove hem whls reanng her mahemacal rgour. Ths can be done n such a way as o ulse some knowledge abou he speech sgnal (see, for nsance, Russell and Cook [3] ) and hs offers promse for fuure speech recognon algorhms. HMMs need large amouns of ranng daa o work effecvely on large speaker populaons, leadng o a consderable compuaonal requremen for buldng he models. However, hs s done 'off- lne' and recognon of a small vocabulary usng 8

19 HMMs can be mplemened o work n real me. Usng some of he echnques descrbed n hs paper, a speaker ndependen recognon algorhm has acheved an accuracy of 98. % on a vocabulary of he dgs. 9

20 Acknowledgemens My hanks are due o Dr M J Russell of he Speech Research Un, RSRE, for hs helpful commens and dscussons durng he preparaon of hs paper, and o Dr R K Moore (also of he Speech Research Un) for permsson o reproduce Fg 5. Appendx A Dynamc me warpng and hdden Markov models An earler and sll much used paern machng echnque for ASR s dynamc me warpng (DTW). I was he desre o exend and generalse DTW ha lead o he use of HMMs for ASR and was quckly realsed ha DTW s a specal case of hdden Markov modellng [4,5]. The scence fcon sound of he name of hs echnque refers o he way n whch an uerance s non-lnearly 'sreched' or 'compressed' n me o algn wh anoher uerance. Fgure 5 shows an example of hs process. Fg 5 Dynamc me warpng algnmen of wo uerances. The analyss vecors (frames) of he wo uerancesu, andu 2, of lenghst andt 2 respecvely, are posoned wh her frs frames n he op lef corner of he fgure and subsequen vecors followng n a rghwards and downwards drecon respecvely. In hs case, each frame s a specrum of he nsananeous speech sgnal, formed by samplng he oupus of 9 bandpass flers along he audo specrum. The parally 20

21 enclosed recangle can be hough of as a T T2 grd whose, h elemen, M,, s he dsance beween frame of U, and frame of U 2. Commonly used dsance mercs are he Eucldean and he Cy-Block (Manhaan) merc. The zg-zag pah hrough he grd s known as he opmal me regsraon pah and maps every frame ofu 2 o a frame of U. If a consran s mposed on he mappng ha he begnnngs and he ends of he uerances mus concde, he opmal mappng s a pah hrough he grd whch sars n he op lef square, ends n he boom rgh square, and mnmses he oal cumulave 'dsance' en roue. Ths mnmsaon s performed by usng he echnque of Dynamc Programmng. Imagne a second grd whose, h elemen N, holds he lowes cumulave dsance o ha poson. Ths elemen may be calculaed as follows: N(, ) M (, ) + mn[( N(, ), N(, ), N(, 2)].(22) In oher words, he cumulave dsance o elemen, s found by mnmsng over hree possble 'roues' o he elemen, as shown n Fg 6. Fg 6 Calculaon of he cumulave dsance o elemen (, ). The cumulave dsance n he boom rgh square, N ( T, T 2 ), s regarded as a measure of he dssmlary of he wo uerances. The smlary o he calculaon of he Verb coeffcens (secon 4.2) should be obvous. If each frame of U, s regarded as an HMM 'sae', hen he DTW 'pah' s equvalen o he 'mos lkely sae sequence' of Fg 4. Furhermore, f he Eucldean merc s used o measure he dsance beween wo frames, hs s equvalen n HMM erms o he assumpons of a mulvarae Normal dsrbuon wh deny covarance marx for each 'sae' (see Appendx B). Noce ha n he DTW algorhm, he 'sae ranson probables' are equal and n he algorhm of equaon (22), a skp of only one sae s allowed. Hsorcally, aemps were made o refne he DTW algorhm by he addon of 'penales' (whch can be nerpreed n erms of alerng HMM sae ranson probables) and ncorporang more complex 'roues' o an elemen (skppng saes n HMM erms). 2

22 The chef drawback of DTW s ha has no mechansm for opmsng models on large amouns of daa, alhough he echnque of 'cluserng' [6] goes some way n he drecon of HMMs. However, DTW s especally aracve n applcaons where only lmed ranng daa s avalable, because a sngle uerance can funcon as ranng daa for a class. Appendx B The mulvarae normal dsrbuon The equaon of he PDF of he mulvarae Normal dsrbuon s: f ( x) (2π ) d / 2 U / 2 exp T ( x µ ) U 2 ( x µ ).(23) Here, d s he dmensonaly of he vecors x and µ, and he marxu (he covarance marx) s a d d symmercal marx. The superscrp T denoes vecor ransposon. Noce ha when d, he equaon collapses o he famlar unvarae Normal probably densy funcon: f ( x) exp 2πσ 2 ( x µ ) 2 2σ Noce also ha he erm ( x µ ) T U ( x µ ) 2 s a[ d] vecor a ( d d ) marx ( ) n equaon (23) s a scalar, snce d a (d x ) vecor. Compuaon of hs erm s parcularly smplfed f he covarance marx U s dagonal, when becomes: 2 d ( x µ ) σ 2 2.(24 2 whereσ s he varance n dmenson. Furhermore, fσ, equaon (24) reduces o squared Eucldean dsance beween x and µ. In Appendx A, was saed ha f he Eucldean dsance was used as a merc beween wo vecors A and B, was equvalen o calculang he probably of observng A from an HMM sae wh a mulvarae Normal dsrbuon, of mean vecor B and deny covarance marx. Because he erm n equaon (24) s exponenaed n he calculaon of he probably, here s, n fac a non-lnear relaonshp beween he Eucldean dsance and he probably. However, as he exponenal s a monoonc funcon, hs relaonshp s monoonc and hence classfcaon s no affeced. 2 22

23 References Lpprnann R P and Gold B: 'Neural ne classfers useful for speech recognon', In Proc s In Conf on Neural Neworks, San Dego (June 987). 2 Devver P and Kler J: 'Paern recognon - a sascal approach', Prence-Hall Inernaonal Inc (982). 3 Daurch B A, Rabner L R and Marn T B: 'On he effecs of varyng fler bank parameers on solaed word recognon', leee Transacons on Acouscs, Speech and Sgnal Processng, 3, pp (Augus 983). 4 Russell M J, Moore R K and Tomlnson M J: 'Dynamc programmng and sascal modellng n auomac speech recognon', J Opl Res Soc 37(), pp 2-30 (986). 4 Cox D R and Mller H D: 'The heory of sochasc processes', Meuhuen and Co Ld (965). 6 Makhoul J, Roucos S and Gsh H: 'Vecor quanzaon n speech codng', Proc of he leee, 73 pp (985). 7 Levnson S E, Rabner L R and Sondh M M: 'An nroducon o he applcaon of he heory of probablsc funcons of a Markov process o auomac speech recognon', The Bell Sysem Techncal Journal, 62, pp (983). 8 Rabner L R and Juang B H: 'An nroducon o hdden Markov models', leee ASSP Magazne (January 986). 9 Lporace L A: 'Maxmum lkelhood esmaon for mulvarae observaons of Markov sources', IEEE Transacons on Informaon Theory, 28, pp (982). 0 Levnson S E, Rabner L R and Sondh M M: 'On he applcaon of vecor quanzaon and hdden Markov models o speaker-ndependen, solaed word recognon', The Bell Sysem Techncal Journal, 62, pp (983). Rabner L R, Juang B H, Levnson S E and Sondh M M: 'Some properes of connuous hdden Markov model represenaons', AT and T Techncal Journal, 64, pp (985). 2 Brdle J S and Sedgwck N S: 'A mehod for segmenng acousc paerns, wh applcaons o auomac speech recognon', In Proc leee Conf on Acouscs, Speech and Sgnal-processng, pp (977). 3 Russell M J and Cook A E: 'Expermenal evaluaon of duraonal modellng echnques for auomac speech recognon', In Proc leee Conf on Acouscs, Speech and Sgnal-processng, pp (987) 23

24 4 Brdle J S: 'Sochasc models and emplae machng: some mporan relaonshps beween wo apparenly dfferen echnques for auomac speech recognon', In Proc The Insue of Acouscs (984). 5 Juang B H: 'On he hdden Markov model and dynamc me warpng for speech recognon - a unfed vew', AT&T Bell Laboraores Techncal Journal, 63, pp (984). 6 Wlpon J G and Rabner L R: 'A modfed k-means cluserng algorhm for use n solaed word recognon', leee Transacons on Acouscs, Speech and Sgnal Processng, 33, pp (June 985). 24