0.1 MAXIMUM LIKELIHOOD ESTIMATION EXPLAINED

Transcription

1 0.1 MAXIMUM LIKELIHOOD ESTIMATIO EXPLAIED Maximum likelihood esimaion is a bes-fi saisical mehod for he esimaion of he values of he parameers of a sysem, based on a se of observaions of a random variable ha is relaed o he parameers being esimaed. oe ha he parameers being esimaed are no hemselves random variables. Raher, hey are assumed o be unknon consans. Given a se of observaions X of a random variable X and possibly a se of observaions Y of a condiioning random variable Y, he maximum likelihood esimae of a se of parameers φ is given by φˆ argmax φ { P( X Y; (1.1 here P( X Y; represens he probabiliy ha samples of he random variable X ake he value X, given ha he samples of he random variable Y have aken he value Y. The erms φ and λ afer he semicolon represen he parameers of P( X Y, he disribuion of X condiioned on Y. The erms o he lef of he semicolon are all random variables and he ones o he righ are no random (and have no disribuions associaed ih hem. oe ha in Equaion 1.1 Y and λ are opional. In he simples esimaion here may be no condiioning variables or addiional parameers, and he maximum likelihood esimaor ould simply be given by φ argmax φ { P( X; φ In Equaions 1.1 and 1.2 he probabiliy of X ih respec o φ is maximized. The soluion obained ould be idenical if any monoonically increasing funcion of P( X Y; ere maximized insead of P( X Y;. One paricularly useful monoonic funcion is he logarihm. The logarihm has he simplifying propery ha i ransforms he muliplicaion operaion o an addiion. Thus, if he elemens of he se X ere independen, e ould ge (1.2 log( P( X Y; log( P( X 1, X 2,, X Y; log( P( X 1 Y; P( X 2 Y; P( X Y; log( P( X i Y; (1.3 here X 1, X 2, ec. are elemens of he se X. The maximum likelihood esimaor no becomes φˆ argmax φ log( P( X i Y; (1.4 Maximizaion of Equaion 1.4 is frequenly simpler han maximizaion of Equaion 1.1. The erm log ( P( X Y; is referred o as he log-likelihood of X, or simply as he likelihood of X and is represened as L( X Y;. 1

2 The maximizaion given by Equaion 1.4 resuls in an esimae for he parameer φ such ha he resuling disribuion P( X i Y; is he closes fi o he daa se X. We illusrae his by he folloing example. Example: Consider a given se of seven samples from a random variable X : X { X 1, X 2,, X The random variable is furher assumed o have a Gaussian disribuion ih uni variance. The mean of his disribuion µ is o be esimaed from he sample X. The log-likelihood of any sample X i from a Gaussian disribuion is given by L( X ; µ ( X i µ log e C 2π ( X i µ here C is a consan resuling from he erms no involving µ. To maximize L( X ; µ, e differeniae i ih respec o µ and ge (1.5 leading o 0 d L( X ; µ Xi µ dµ ( X i µ (1.6 µˆ X i hich leads us o he conclusion ha he maximum likelihood esimae of he mean of a Gaussian random variable is merely he average of he observed samples. The maximum likelihood esimaion of he mean of a Gaussian random variable is illusraed picorially by Figure 1.1. In general he maximum likelihood esimae of any parameer of he disribuion of any Random variable resuls in a disribuion ha fis he hisogram of he observed samples mos closely. Thus, he ML esimae is dependen only he observed samples. This can lead o errors in esimaion since he observed samples may no be a fair represenaion of he disribuion of he random variable. This usually occurs hen he number of observed samples is very small. 1.2 HOW HMM-BASED AUTOMATIC SPEECH RECOGITIO (ASR SYS- TEMS FUCTIO ASR sysems are essenially paern classifiers. In any such sysem all uerances of speech are modeled as sequences of sounds. These sounds may eiher be he phonemes in a language, ords in ha language, or larger unis, depending on various facors. The complee se of sounds ha he ASR sysem has o recognize consiues he classes modeled by i. In his discussion, e assume ihou loss of generaliy, ha he (1.7

3 Figure 1.1 Picorial illusraion of ML esimaion of he mean of a Gaussian RV. The hisogram represens he hisogram of he observed samples. The curve in he firs figure represens a Gaussian ih mean µ 1. The curve in he second figure represens a Gaussian hose mean is he average of he observed samples. The hird figure shos a Gaussian ih mean µ 2. Clearly, he Gaussian in he figure o he cener fis he hisogram of he daa mos closely. The mean of his Gaussian is herefore he maximum likelihood esimae of he mean of he RV. sound classes modeled by he sysem are ords. The ASR sysem hen classifies segmens of speech as belonging o one of hese ords, hus idenifying hem. In ASR sysems, classificaion is no performed using he speech signal direcly. Insead, he speech signal is ansformed ino a sequence of feaure vecors, or parameer vecors, and classificaion is performed using hese feaure vecors. The feaure vecors mos idely used used are cepsral coefficiens, or varians of cepsra (such as PLP cepsra derived from poer specra of shor indoed segmens, or frames of speech. Each feaure vecor hus corresponds o a frame of speech. Le S represen he sequence of feaure vecors derived from he uerance being recognized. ASR sysems idenify he sequence of ords in ha uerance using he opimal classifier equaion Ŵ argmax W { P( S WP( W (1.8 here Ŵ is he recognized sequence of ords in ha uerance. P( W is he a priori probabiliy ha he ord sequence W as uered and is usually specified by a language model. For in-deph informaion on language models nd language modeling e refer he reader o [JELIEKS BOOK REF]. P( S W is he likelihood of S given ha he W as he sequence of ords uered. I is ermed as he acousic likelihood of he daa and is obained from he probabiliy disribuion of all sequences feaure vecors ha could represen he sequence of ords W. In HMM-based speech recogniion sysems his probabiliy disribuion of sequences is modeled by an HMM. The folloing secion describes he hidden Markov model in greaer deail. HMM-based modeling of he disribuions of sequences of vecors In HMM-based recogniion sysems he mechanism ha generaes he sequence of feaure vecors represening any ord is modeled by an HMM. When generaing he sequence, he generaor is assumed o be in one of a finie se of saes a any insan of ime. Each sae has a probabiliy disribuion funcion, referred o as he sae disribuion of ha sae, associaed ih i. The hidden Markov modeling paradigm assumes ha o generae he feaure vecor a any insan, he generaor dras a vecor from he sae disribuion of he sae i is in a ha insan. The vecors ha he generaor dras from a sae disribuion are

4 Figure 1.1 Example of a 5 sae HMM ih one non-emiing iniial sae, and a non-emiing erminaing sae. Each of he circles represens a sae. The arros represen valid ransiions from he sae, and he numbers belo he arros represen he probabiliy of ha ransiion. For example, he arros from sae 1 indicae ha if he generaor is in sae 1 a ime, a ime +1 i can be in sae 1 ih probabiliy 0.5, sae 2 ih probabiliy 0.3 and sae 3 ih probabiliy 0.2. The doed arros poin o he sae disribuions associaed ih ha sae. An observaion is dran from his disribuion every ime he generaor visis he sae. The iniial sae (sae 0 and he erminaing sae (sae 4 have no sae disribuions associaed ih hem, and no daa are generaed hen he generaor is in hese saes. oe ha in his figure all ransiions poin lef o righ. In a more generic HMM, ransiions may occur in any direcion, from any sae o any oher sae. said o belong o ha sae. The HMM also has a se of ransiion probabiliies associaed ih each sae. A generaor ha is in sae i a ime and moves o sae j a ime insan + 1 is said o ransi from sae i o sae j a ime insan. The ransiion probabiliies of a sae refer o he probabiliy disribuion of he saes ha he generaor can be in a he nex insan, given ha i is in ha sae a he curren insan. The generaor dras from his disribuion in order o deermine hich sae i ill be in a he nex insan of ime. The ransiion probabiliies and he sae disribuions are all specific o he ord being modeled by he HMM. Figure 1.1 shos an example of an HMM ih 5 saes. The HMM in his figure only permis ransiions in one direcion. Transiions ih probabiliy 0 are no shon. This HMM has a non-emiing iniial sae and a non-emiing erminaing sae. on-emiing saes are saes ih hich here are no probabiliy disribuions associaed - he generaor does no generae observaions hen i is in hese saes. The non-emiing iniial sae in figure 1.1 implies ha a 0, i.e. jus before he generaor begins generaing vecors, i is in he iniial sae here i does no generae any observaions. Similarly, if he generaor eners he erminaing sae i can no longer ransi o any of he oher saes in he HMM, nor can i generae any more observaions. Thus, o generae a sequence of vecors for he ord, he generaor is assumed o ransi hrough a sequence of + 2 saes in he HMM, beginning ih he non-emiing iniial sae and erminaing in he non-emiing final, or absorbing sae. A each ime insan i dras observaions from he sae disribuion of he sae i is in a ha ime insan. The sequence of vecors so generaed is said o be generaed by he HMM. The model for he generaing mechanism for a sequence of ords is also an HMM and easily consruced by concaenaing HMMs for individual ords. Figure 1.2 shos an example here he HMMs for hree ords have been concaenaed o obain an HMM modeling a sequence of hree ords. The saisical parameers of he HMM represening a sequence of ords W are he se of ransiion probabiliies, represened as a marix A W, and he se of sae probabiliy disribuion funcions. The marix A W consiss of elemens a ( i, j, each of hich represens he probabiliy ha he generaor ill be in sae j in he nex ime insan, given ha i is currenly in sae i. Thus, for an HMM ih K saes, e have

5 K a ( i, j 1.0 j 1 (1.9 The sae disribuion of he k h sae is represened by P W, k ( X, here X represens any feaure vecor ha belongs o ha sae. In speech recogniion sysems he various sae disribuions are usually modeled as Gaussians or mixures of Gaussians. Typically, for compuaional efficiency, hese Gaussians are assumed o have diagonal covariance marices, i.e. covariance marices here he off-diagonal elemens are all 0. For simpliciy e represen he sae disribuion of he k h sae as P W, k ( X M( X; φ k (1.10 here M( X; φ k denoes a Gaussian mixure disribuion corresponding o he kh sae of he HMM represening he ord sequence W and represens he se of parameers associaed ih i. We denoe he se of φ k for all he saes in he HMM for W as λ W. A W and λ W represen he complee se of parameers needed o uniquely idenify he HMM modeling W. φ k The probabiliy of any vecor sequence S ha is generaed by he HMM for W is no given by P( S W P( S, s W s Ξ s Ξ P( s WP( S s (1.11 here s represens any sae sequence ha he generaor can follo hen generaing S, and Ξ represens he se of all possible sae sequences. The sae sequence s is, quie lierally, a sequence of saes, one for every feaure vecor in S. Tha is, s [ s 1, s 2, s 3,, s ] (1.12 here is he sae associaed ih S(, he h vecor in S, and is he oal number of vecors in he s sequence S. The probabiliy erms in he righ hand side of Equaion (1.11 can no be rien as WORD 1 WORD 2 WORD 3 Figure 1.2 Example of consrucing he HMM for a sequence of ords from he HMMs of individual ords. The non-emiing erminaing sae of any ord is merged ih he non-emiing iniial sae of he nex ord. The merged sae is no longer an iniial sae or a erminaing sae. Hoever, i remains non-emiing, and no sae disribuion is associaed ih i. The resuling HMM has a non-emiing iniial sae, a non-emiing erminaing sae and several inermediae non-emiing saes as ell.

6 P( S s M( S( ; φ s P( s W a( 0, s 1 a( s, s + 1 (1.13 here a( 0, s represens he probabiliy of ransiing from he h 1 0 sae (i.e he iniial non-emiing sae of he HMM for W o he firs sae in he sae sequence s, and a( s, s + 1 represens he probabiliy of ransiing from sae s o sae s + 1. Equaion (1.11 can no be rerien as P( S W a( 0, s 1 a( s, s + 1 s Ξ, p Ideally, recogniion ould be performed as M( S( ; φ s (1.14 Ŵ argmax W P( W P( s WP( S s s Ξ (1.15 Hoever, for easy implemenaion, HMM based speech recogniion sysems usually esimae no jus he bes ord sequence, bu also he bes sae sequence associaed ih he ord sequence. i.e. recogniion is performed as: Ŵ argmax W, s { P( WP( S, s W Using Equaion (1.13, his can be furher expanded ino arg max W, s { P( WP( s WP( S s (1.16 Ŵ argmax W, s P( W a( 0, s 1 a( s, s + 1 M( S( ; φ s (1.17 In order o evaluae Equaion (1.17 fully, he erm ihin he braces ould have o be compued for every possible ord sequence in he language. This ould be impracical. In pracice, dynamic programming mehods are used o obain locally opimal esimaes for Ŵ. There, is hoever, furher refinemen possiblein HMM-based sysems. The CMU Sphinx-III HMM based recogniion sysem, for example, is a phone-based recogniion sysem in hich ords are furher decomposed ino sequences of phoneic unis and he HMMs for ords are buil by concaenaing he HMMs modeling hese phoenic unis. In order o reduce he oal number of parameers needed o consruc HMMs for all he phoneic unis modeled by he sysem, he sae disribuions of he HMMs of he various phoneic unis are shared, i.e. he same disribuion is used by he saes of he HMMs of several phoneic unis. In he nex secion e ill explain he precise manner in hich he Sphinx (or any oher HMM-based sysem models any given lanugage and learns he parameers of he models used o represen he basic sound unis of he language.