Isolated-word speech recognition using hidden Markov models

Transcription

1 Isolaed-word speech recogniion using hidden Markov models Håkon Sandsmark December 18, 21 1 Inroducion Speech recogniion is a challenging problem on which much work has been done he las decades. Some of he mos successful resuls have been obained by using hidden Markov models as explained by Rabiner in 1989 [1]. A well working generic speech recognizer would enable more efficien communicaion for everybody, bu especially for children, analphabes and people wih disabiliies. A speech recognizer could also be a subsysem in a speech-o-speech ranslaor. The speech recogniion sysem implemened during his projec rains one hidden Markov model for each word ha i should be able o recognize. The models are rained wih labeled raining daa, and he classificaion is performed by passing he feaures o each model and hen selecing he bes mach. apple banana kiwi speech feaure exracion lime classificaion orange peach pineapple Figure 1: Flow char of he sysem. The feaures exraced from he speech signal are passed o each word model and he bes mach is seleced. 1

2 2 Background heory 2.1 Hidden Markov models Basic knowledge of hidden Markov models is assumed, bu he wo mos imporan algorihms used in his projec will be described. The observable oupu from a hidden sae is assumed o be generaed by a mulivariae Gaussian disribuion, so here is one mean vecor and covariance marix for each sae. We will also assume ha he sae ransiion probabiliies are independen of ime, such ha he hidden Markov chain is homogenous. We will now define he noaion for describing a hidden Markov model as used in his projec. There is a oal number of N saes. An elemen a ss in he ransiion probabiliy marix A denoes he ransiion probabiliy from sae s o sae s, and he probabiliy for he chain o sar in sae s is π s. The mean vecor and covariance marix for he mulivariae Gaussian disribuion modeling he observable oupu from sae s are µ s and Σ s, respecively. For an observaion o, b s (o) denoes he probabiliy densiy of he mulivariae Gaussian disribuion of sae s a he values of o. We will someimes denoe he collecion of parameers describing he hidden Markov model as λ = {A, π, µ, Σ}. 2.2 The forward algorihm We wan o calculae he probabiliy densiy of an observaion o 1,..., o T for a specific model. This will be used o selec he model (i.e. word) ha mos likely generaed he speech signal. f(o 1,..., o T ; λ) = s T f(o 1,..., o T, s T ; λ) (1) = s T f(o T o 1,..., o T 1, s T ; λ)f(o 1,..., o T 1, s T ; λ) (2) = s T b st (o T ) s T 1 f(o 1,..., o T 1, s T 1, s T ; λ) (3) = s T b st (o T ) s T 1 f(s T o 1,..., o T 1, s T 1 ; λ)f(o 1,..., o T 1, s T 1 ; λ) = s T b st (o T ) s T 1 a st 1 s T f(o 1,..., o T 1, s T 1 ; λ) (5) (4) The recursive srucure is revealed as we reduced he problem from needing f(o 1,..., o T, s T ; λ) for all s T o needing f(o 1,..., o T 1, s T 1 ; λ) for all s T 1. Le us inroduce he forward variable o ease he noaion. 2

3 α 1 (s) f(o 1, S 1 = s; λ) (6) = b s (o 1 )π s (7) α (s) f(o 1,..., o, S = s; λ) (8) = b s (o ) a s sα 1 (s ) s (9) Then our soluion can be expressed nicely as f(o 1,..., o T ; λ) = s α T (s). (1) Implemened naïvely op-down (backwards in ime) his would no bring us any luck because of he exponenially recursive srucure. The naïve algorihm is however easily converible o an efficien varian using dynamic programming where we calculae he forward variables boom-up (forwards in ime). We simply calculae α (s) for all saes s, firs for = 1 and hen all he way up o T. This way all he forward variables from he previous ime sep are readily available when needed. 2.3 The Baum-Welch algorihm We wan o find he parameers λ ha maximize he likelihood of he observaions. This will be used o rain he hidden Markov model wih speech signals. The Baum-Welch algorihm is an ieraive expecaion-maximizaion (EM) algorihm ha converges o a locally opimal soluion from he iniializaion values. The M-sep consiss of updaing he parameers in he following inuiive way: expeced number of imes in sae s a = 1 π s := π s = expeced number of imes a = 1 (11) expeced number of ransiions from s o s a ss := a ss = expeced number of ransiions from s (12) µ s := µ s = expeced observaion when in sae s (13) Σ s := Σ s = observaion covariance when in sae s (14) The E-sep hus consiss of calculaing hese expecaions for a fixed λ. Le V s () denoe he even of ransiion from sae s a ime sep, and V () s,s he even of ransiion from s o s a. Then we calculae hese expecaions by using indicaor funcions and lineariy of expecaion. 3

4 π s = E{1[V s (1) ]} = P (V s (1) ) (15) a ss = E{ () 1[V s,s ]} E{ () 1[V s ]} = µ s = E{ () 1[V s ]o } E{ () 1[V s ]} = P (V () s,s ) P (V () s ) () P (V s )o () P (V s ) Σ s = E{ () 1[V s ](o o T µ s µ T s )} E{ () 1[V s ]} = () P (V s )o o T () P (V s ) µ s µ s T Noe ha he non-ialic T denoes ranspose and has nohing o do wih ime. To be able o calculae hese probabiliies we firs inroduce he backward variable which is very similar o he forward variable previously defined. (16) (17) (18) β T (s) 1 (19) β (s) f(o +1,..., o T S = s; λ) (2) = s a ss b s (o +1 )β +1 (s ) (21) The backward variable has is name because i is firs calculaed for he las ime sep and hen backwards in ime when implemened wih dynamic programming (essenially he reverse procedure of he one described in deail for he forward variable). Then we rename he probabiliies o he same symbols as used by Rabiner and express hem by forward and backward variables: γ (s) P (V s () ) = P (S = s o 1,..., o T ; λ) (22) = f(o 1,..., o T S = s)p (S = s) f(o 1,..., o T ) (23) = f(o 1,..., o, S = s)f(o +1,..., o T S = s) f(o 1,..., o T ) (24) = α (s)β (s) f(o 1,..., o T ) (25) And similarly (deails omied): ξ (s, s ) P (V () s,s ) = P (S = s, S +1 = s o 1,..., o T ; λ) (26) And finally we ge he following: = α (s)b s (o +1 )a ss β +1 (s ) f(o 1,..., o T ) (27) 4

5 π s = γ 1 (s) (28) a ss = ξ (s, s ) γ (29) (s) µ s = γ (s)o γ (3) (s) Σ s = γ (s)o o T T γ µ s µ s (31) (s) To summarize he E-sep boils down o compuing γ (s) and ξ (s, s ) for all s, s and while he parameers λ are fixed, and hen he M-sep will updae λ by using he calculaions done in he E-sep. This is ieraed unil saisfacion. 3 Sysem design 3.1 Feaure exracion The source speech is sampled a 8 Hz and quanized wih 16 bis. The signal is spli up in shor frames of 8 samples corresponding o 1 ms of speech. The frames overlap wih 2 samples on each side. The idea is ha he speech is close o saionary during his shor period of ime because of he relaively limied flexibiliy of he hroa. We will pick ou our feaures from he frequency domain, bu before we ge here by aking he fas Fourier ransform, we muliply by a Hamming window o reduce specral leakage caused by he framing of he signal Speech signal Hamming window 3.5 x X(F) ime (a) Speech signal and Hamming window in ime domain F [Hz] (b) Single-sided magniude specrum of he same speech signal muliplied by he Hamming window. Figure 2: An 8 sample frame of an unvoiced par of a speech signal. Unvoiced speech, like sh, is more noisy and conains higher frequencies han voiced speech. The D larges local maxima from he single-sided magniude specrum are are picked as feaures for each frame, and D is indeed an imporan parameer of he sysem ha will be discussed laer. 5

6 Speech signal Hamming window X(F) ime (a) Speech signal and Hamming window in ime domain F [Hz] (b) Single-sided magniude specrum of he same speech signal muliplied by he Hamming window. Figure 3: An 8 sample frame of a voiced par of a speech signal. 3.2 Training The raining is a combinaion of boh supervised and unsupervised echniques. We rain one hidden Markov model per word wih already classified speech signals. One imporan choice is he number of differen saes in each model. The goal is ha each sae should represen a phoneme in he word. The clusering of he Gaussians is however unsupervised and will depend on he iniial values used for he Baum-Welch algorihm. For his projec, oally random guesses (ha obey he saisical properies) for A and π were used as iniial values. For Σ s, he diagonal covariance marix for he raining daa was used for all saes. For each sae a random raining daa poin was chosen as µ s. The raining examples for each word are concaenaed ogeher, and Baum-Welch is run for 15 ieraions. 3.3 Classificaion Le λ i denoe he parameer se for word i. When presened wih an observaion o 1,..., o T, he selecion is done as follows. prediced word = arg max f(o 1,..., o T ; λ i ) (32) i And we recognize ha f(o 1,..., o T ; λ i ) is exacly wha he forward algorihm compues. 4 Experimenal seup and resuls For each of he seven words, 15 uerances by his auhor were recorded. The performance of he sysem was measured by five-fold cross-validaion on he recorded daa se of 15 uerances. Experimenaion indicaed ha he wo mos imporan parameers 6

7 4 Training apple 4 Training lime F2 [Hz] 2 F2 [Hz] F1 [Hz] F1 [Hz] (a) Apple. (b) Lime. 4 Training orange 4 Training peach F2 [Hz] 2 F2 [Hz] F1 [Hz] F1 [Hz] (c) Orange. (d) Peach. Figure 4: Fied Gaussians afer en ieraions of he Baum-Welch algorihm. We have six saes wih one Gaussian each. The wo mos dominan frequencies (feaures) are shown. Each green plus is represens a frame from a raining speech signal. The sars are he means of each Gaussian, and he ellipses indicae heir 75% confidence inerval. Noice he higher frequencies presen in he words conaining unvoiced phonemes ( peach and orange ) compared o he words ha do no ( apple and lime ). were he number of hidden saes, N, and he number of frequencies exraced from each frame, D. The cross-validaion was herefore run wih differen values for hese parameers, and he resuls are shown in able 1. 5 Discussion The resuls are quie good compared o he simple approach aken, especially in he feaure exracion phase. More advanced feaures like Mel-frequency cepsral coefficiens were considered, bu we decided on simple frequencies due o he low misclassificaion 7

8 N\D % 8.6% 3 21.% 15.2% 9.5% 12.4% 1.9% 14.3% 5.7% % 11.4% 8.6% 5.7% 3.8% 6.7% 4.8% % 8.6% 9.5% 4.8% 2.9% 5.7% 4.8% % 1.5% 3.8% 5.7% 7.6% 6.7% 1.5% % 12.4% 6.7% 1.5% 7.6% 2.9% 8.6% % 5.7% Table 1: Misclassificaion raes for five-fold cross-validaion wih differen values for he number of hidden saes, N, and he number of frequencies exraced from each frame, D. Each five-fold cross-validaion procedure akes abou 7 minues wih he 15 uerances on a 2 GHz Inel Core 2 Duo (serial execuion). raes achieved. I should be noed ha his sysem would no perform well if rained and esed wih differen speakers. This is because of he differen frequency characerisics of differen voices, especially for speakers of differen gender. We also experimened wih increasing he number of raining ieraions for he Baum- Welch algorihm, including seing a hreshold on he likelihood difference beween seps. Tha, however, proved o have lile benefi in pracice; neiher he execuion ime nor he misclassificaion rae showed any menionable improvemens over jus fixing he number of ieraions o 15. The reason why he execuion ime did no show any significan improvemens is because mos of he execuion ime is spen during feaure exracion, and no in raining. I is also ineresing o noe ha when N is oo small, here are many apple s misclassified as pineapple s, and vice versa, due o he loss of emporal informaion. Anoher imporan parameer is he number of samples in each frame. If he frame is oo small, i becomes hard o pick ou meaningful feaures, and if i is oo large, emporal informaion is los. However, due o ime consrains, we did no es anyhing else han 8 samples for his projec. The concaenaion of he raining examples rains a probabiliy of ransiioning from he las sae o he iniial sae ha is no needed for classificaion. Rabiner gives a modified Baum-Welch algorihm for muliple raining examples such ha concaenaion is no necessary, bu ha was no implemened during his projec as he concaenaion seemed o work well. 6 Conclusion and fuure work During his projec a sysem for isolaed-word speech recogniion was implemened and esed. The cross-validaion resuls are good for a single speaker. Two obvious exensions are beer suppor for several speakers, and suppor for coninuos speech. The firs sep owards he former would be more, and more robus, feaures. For he laer he simples approach is probably o deec word boundaries and hen proceed wih an isolaed-word 8

9 recognizer. The Malab implemenaion along wih he daa se is published as open source and can be found a hp://code.google.com/p/hmm-speech-recogniion/. 7 References [1] L. R. Rabiner, A uorial on hidden Markov models and seleced applicaions in speech recogniion, Proceedings of he IEEE, vol. 77, pp , Feb [2] C. M. Bishop, Paern Recogniion and Machine Learning (Informaion Science and Saisics). Springer, 1s ed. 26. corr. 2nd prining ed., Ocober 27. [3] X. Huang, A. Acero, and H.-W. Hon, Spoken Language Processing: A Guide o Theory, Algorihm and Sysem Developmen. Prenice Hall PTR, May 21. 9