. Projec Iroduco Fudameals of Speech Recogo Suggesed Projec The Hdde Markov Model For hs projec, s proposed ha you desg ad mpleme a hdde Markov model (HMM) ha opmally maches he behavor of a se of rag sequeces ha wll be provded o you as par of hs projec. The goal wll be o use he sadard se of forward-backward esmao algorhms o opmally deerme he bes (maxmum lkelhood) HMM ha maches a gve se of rag daa. You are also asked o use he Verb algorhm for esmag he model parameers ad comparg ad corasg he resuls for he wo mehods. You may also wa o vesgae he effecs of usg subses of he rag daa o he esmaed models. Fally, f me perms, you are asked o desg your ow sequece geeraor ad deerme he effecs of chagg he rag sequece characerscs o he esmaed models. 2. HMM Geerave Models A oal of four models were creaed wh he followg characerscs: Model : umber of saes, Q=5 ype of desy of observaos: dscree umber of observao possbles, K=5 ype of model: ergodc sae raso marx desy: radom sae observao marx desy: radom sae pror desy: radom Model 2: umber of saes, Q=5 ype of desy of observaos: dscree umber of observao possbles, K=5 ype of model: ergodc sae raso marx desy: skewed sae observao marx desy: skewed sae pror desy: skewed Model 3: umber of saes, Q=5 ype of desy of observaos: dscree umber of observao possbles, K=5 ype of model: lef-rgh sae raso marx desy: cosraed 2/9/2005 :55 AM HMM Specal Projec
sae observao marx desy: radom sae pror desy: cosraed Model 4: umber of saes, Q=5 ype of desy of observaos: dscree umber of observao possbles, K=5 ype of model: lef-rgh sae raso marx desy: cosraed sae observao marx desy: skewed sae pror desy: cosraed A plo showg he geerc srucure of boh he ergodc model (Fgure ) ad he lefrgh model (Fgure 2) s gve below. (Noe ha oly a subse of he sae rasos are show he fgure sce go very messy showg all possble sae-sae raso pahs.) Fgure 5 sae ergodc model (oly some of he acual sae rasos show fgure) Fgure 2 5 sae lef-rgh model 3. HMM Trag Sequeces For hs projec here are four rag ses (avalable from he course webse), labeled:. hmm_observaos_ergodc_radom.ma, 2. hmm_observaos_ergodc_skewed.ma, 2/9/2005 :55 AM 2 HMM Specal Projec
3. hmm_observaos_lef-r_radom.ma 4. hmm_observaos_lef-r_skewed.ma The characerscs of hese four daases are as follows:. each ma fle coas he followg formao: a. a observao array, called daa(ex,t), where ex s 50 ad T s 00. The elemes of daa are he observao values (bewee ad K) of each of he observao sequeces as draw from he dscree sae observao marx (see below) b. a hdde saes array, called saes(ex,t). The elemes of saes are he sae dces (bewee ad Q) of each of he observao sequeces as draw from he dscree sae pror desy ad he sae raso marx. The hdde saes array cao be used ay calculao, s provded for say checks o he sysem. c. a durao array, called durao(ex), whch specfes he acual legh of each of he ex rag sequeces. I should be oed ha for he ergodc sequeces, he durao of each sequece s exacly T=00 observaos (wh he acual observaos gve daa), bu for he lef-rgh models, he durao of each sequece s dffere ad s specfed he durao array d. a sae pror array, called pror0(q), whch specfes he probably of each rag sequece sarg each of he Q saes e. a sae raso marx, called rasma0(q,q), whch specfes he probably, a j, of a sae raso from sae o sae j, where <=,j<=q f. a sae observao marx, called obsma0(q,k), whch specfes he probably of symbol k appearg sae, where <=k<=k, ad <=<=Q 2. he frs wo ma fles coa rag sequeces for he ergodc model wh eher radom eres for he sae pror desy, he sae raso marx, ad he sae observao marx (he frs ma fle), or skewed eres for he sae pror desy, he sae raso marx, ad he sae observao marx (he secod ma fle). The secod wo ma fles coa rag sequeces for he lef-rgh model wh eher radom eres for he sae observao marx (he hrd ma fle), or skewed eres for he sae observao marx (he fourh ma fle). The model sae pror desy ad sae raso marx are boh hghly skewed for boh he hrd ad fourh ma fles. Aga s oed ha for lef-rgh models, he durao of he rag sequeces are varable ad are specfed he durao array. For he ergodc models, he durao of each rag sequece s fxed a T=00 observaos. 4. Revew of he Forward-Backward Algorhm The goal of he forward-backward algorhm s o deerme he values of he sae pror desy, he sae raso marx, ad he sae observao desy ha maxmzes he 2/9/2005 :55 AM 3 HMM Specal Projec
probably of he rag se. Ths maxmzao s acheved a erave maer, amely by frs posulag a (radom or well educaed) guess as o he values of he sae pror desy, he sae raso marx, ad he sae observao desy, ad he calculag he forward probables, α (, ), for me, sae, ad rag sequece, he backward probables, β (, ), for me, sae, ad rag sequece, he scalg sequece c (, ), for me, ad rag sequece, ad he scaled values of α ad β, amely ˆα ad, ˆ β whch are all compued as follows:. Oba al esmaes of he sae pror desy, he sae raso marx, ad he sae observao marx from eher radom guesses, or from skewed al codos. Ial esmaes of π, a, ad b of he form: π = { π, π,..., π }; π = probably of beg sae a me 2 Q a a2 a Q a2 a22 a 2Q a= aq aq2 a QQ aj = probably of makg a raso from sae o sae j b() b(2) b( K) b2() b2(2) b2( k) b = bq() bq(2) bq( K) h b( k) = probably of observg he k symbol sae 2. Use as he rag se oe of he se of observaos read earler. We deoe he complee rag se of ex sequeces of observaos, wh T beg he durao of he -h observao sequece, as: O = value of he observao a me ad for sequece, O K, =,2,..., T, =,2,..., ex 2/9/2005 :55 AM 4 HMM Specal Projec
3. Compue he forward, backward, scale facor, ad scaled esmaes of α ad β as: Forward Loop--For each observao sequece =, 2,..., ex he rag se (wh sequece durao T ), compue he followg: Ialzao Sep ( = ) : α (, ) = π b( O), =, 2,..., Q(uscaled α) Q c(, ) = α (, ) = = ˆ α (, ) = α (, )/ c(, ) (scaled α) Ierao Sep: for = 2,3,..., T Q α (, ) ˆ = α ( ja, ) j b( O ), =, 2,..., Q j= Q c (, ) = α (, ) (scale facors) ˆ α (, ) = α (, ) / c (, ), =, 2,..., Q Backward Loop--For each observao sequece =, 2,..., ex he rag se (wh sequece durao T ), compue he followg: Ialzao Sep: β (, ) = ; =, 2,..., Q T ˆ β (, ) = ; =,2,..., Q T Ierao Sep: for = T, T 2,..., Q ˆ j β+ j + j= (uscaled β) (scaled β ) β (, ) = a ( jb, ) ( O ), =, 2,..., Q ˆ β (, ) = β (, ) / c ( +, ), =, 2,..., Q 2/9/2005 :55 AM 5 HMM Specal Projec
4. Compue he sae desy, γ (, ), ad he sae raso desy, ξ (, j, ), as: γ (, ) = probably of beg sae a me for sequece γ (, ) ˆ (, ) ˆ = α β(, ), =, 2,..., Q, =, 2,..., T, =, 2,..., ex ξ (, j, ) = probably of beg sae a me ad makg a raso o sae j a me + for sequece ξ (,, ) ˆ (, ) ( ˆ j = α aj bj O+ ) β+ ( j, )/ c( +, ), =,2,..., Q, j =, 2,..., Q, =, 2,..., T, =,2,..., ex 5. Re-esmae he sae pror desy, he sae raso marx ad he sae observao marx, usg he relaos:. re-esmao of sae pror desy: ex γ(, ) = π =, =,2,..., Q ex 2. re-esmao of sae raso marx: ex T ξ (, j, ) = = aj =, =, 2,..., Q, j =, 2,..., Q ex T γ (, ) = = 3. re-esmao of sae observao desy marx: ex T = = O = k = = γ ( j, ) bj ( k) =, j =, 2,..., Q, k =,2,..., K ex T γ ( j, ) 6. Trag sequece lkelhood calculao. I s esseal whe dog re-esmao of HMM parameers o moor he egave log probably of he oal se of observao sequeces o esure ha a each erao he oal egave log probably s decreasg, 2/9/2005 :55 AM 6 HMM Specal Projec
ad as a check as o how closely he lkelhoods from he bes esmae mach or exceed he lkelhood from he acual geeraor deses. Ths calculao of oal egave log probably ca be accomplshed by keepg rack, a each erao, of he sum of he egave logs of all he scale facors,.e., L rag ex T = = = log( c (, )) A eresg feaure ha ca be used whe he acual geeraor sequece s kow s o do a sgle forward-backward pass usg he acual geeraor sae pror desy, he acual sae raso marx, ad he acual sae observao marx, hereby gvg he rag sequece lkelhood for he acual geeraor. Ieresgly, he bes esmaed model ca have a rag sequece lkelhood ha s acually hgher ha ha of he geeraor sequece especally whe he acual sequeces are geeraed from radomly geeraed sae prors, sae raso marces, ad sae observao marces. However, mos of he me, he rag sequece lkelhoods are worse ha ha of he geeraor sequece a leas ul he re-esmao procedure has reached a sable opmum ad o goe suck a a local mmum. 5. Revew of he Verb Algorhm A alerave approach o he forward-backward mehod, for re-esmao of HMM model parameers, s he Verb algorhm whch herely fds he bes machg pah ha bes algs each rag sequece wh he curre model. By keepg rack oly of he bes sequece, he compuao ca be reduced cosderably. However, cases where he bes pah s o much beer ha alerave pahs, he Verb algorhm ca lead o hghly sub-opmal soluos. Ths occurs more ofe wh radomly creaed HMM model parameers ha wh skewed model parameers, or wh lef-rgh models whch have srog cosras o he sae pror desy ad he sae raso desy. The deals of how he Verb algorhm operaes are as follows: 2/9/2005 :55 AM 7 HMM Specal Projec
. Coverso o log probables--oe me operao π = log( π ), =, 2,..., Q a = log( a ), =,2,..., Q, j =,2,..., Q j j b ( k) = log( b( k)), =,2,..., Q, k =,2,..., K 2. For each sequece =,2,..., ex he rag se (wh sequece durao T ), compue he followg: Ialzao ( = ): δ(, ) = π + b ( O ), ψ(, ) = 0; =, 2,..., Q =,2,..., Q Recurso: for = 2,3,..., T δ (, ) max (, ) ( O ), j =, 2,..., Q j = δ + aj + bj Q ψ ( j, ) = arg max δ (, ) a +, j=,2,..., Q Termao: j Q P ( ) = max δt (, ) Q qt ( ) = argmax δt (, ) Q Backrackg he bes sae sequece: for = T, T 2,..., q =ψ + + ( ) ( q ( )) Oce he Verb algme of he rag sequeces wh he curre model has bee obaed, we have esseally ucovered he hdde pars of he model, so we have a uque assgme of each observao o a dsc model sae. Hece he re-esmao of he sae pror desy, he sae raso marx, ad he sae observao marx s cosderably smpler ha for he forward-backward mehod, ad ca be saed smply as follows: 2/9/2005 :55 AM 8 HMM Specal Projec
. re-esmao of sae pror desy: ex cq [ ( ) = ] = π = ex, =, 2,..., Q where cq [ ( ) = ] = whe q ( ) = ad s zero oherwse 2. re-esmao of sae raso marx: a j = ex T = = cq [ ( ) = q, + ( ) = j], =, 2,..., Q, j =, 2,..., Q ex T cq [ ( ) = ] = = where c[ q ( ) =, q ( ) = j] = whe q ( ) = ad q ( ) = j + + ad s zero oherwse 3. re-esmao of sae observao desy marx: ex T cq [ ( ) = jo, = k] = = j ( ) =, =,2,...,, =,2,..., ex T cq [ ( ) = j] = = b k j Q k K where [ ( ), ] whe ( ) ad cq = jo = k = q = j O = k ad s zero oherwse 6. HMM Esmao Projec The purpose of hs projec s o each you how o realze boh a forward-backward ad a Verb procedure for esmag parameers of Hdde Markov Models. To ha ed you are asked o read each of he four rag ses, ad o program HMM re-esmao algorhms usg boh a forward-backward approach ad a Verb approach, ad o compare ad coras he wo procedures erms of speed, accuracy, sesvy o al esmaes, compuaoal ssues (akg logs of zero-valued quaes), ad ay oher ssue ha arses durg he course of hs projec. 2/9/2005 :55 AM 9 HMM Specal Projec
I s suggesed ha you frs wre some smple code o geerae a radom sae pror desy, a radom sae raso desy, ad a radom sae observao desy for he parameers of hs exercse, amely Q=5 sae models, wh K=5 possble observaos each sae. The acual geeraor deses are labeled pror0, rasma0 ad obsma0 so you mgh wa o label your radom selecos as pror, rasma, ad obsma o dsgush hem from he acual sequece geeraors. Oce you read each of he rag ses, famlarze yourself wh he daa sequece whch s of he form daa(ex,t) where ex s 50 ad T s 00. Ths daa sequece s he observed se of oupus of he model, bu, of course, here are o observed saes as he saes are hdde. Acually he array saes(ex,t) has he geeraor sae daa bu hs cao be used ay maer your smulaos. You should also remd yourself ha he durao of each rag sequece (=,2,,ex) s specfed he array durao(:ex). (For he ergodc models ha we wll be usg, he durao of each sequece s T=00. For he lef-rgh models he durao s varable ad s specfed durao(:ex).) Oce he rag daa has bee read, he frs sep you should do s score he rag se o he geeraor model by rug oe full erao of he forward-backward roue (whch you have o wre), ad by og he sum of he egave logs of he scalg sequeces. Ths lkelhood score represes a arge value for your acual re-esmao roue a arge ha you wll somemes surpass slghly (why s hs he case) ad mosly wll mss hg due o local opma or bad al sars. The ex sep s o creae full forward-backward esmao ad HMM model reesmao roues, as well as a Verb esmao ad HMM model re-esmao roue. Oce hese roues have bee debugged, you ca beg playg wh he four rag ses ad deermg how well you ca esmae he model parameers. I s a eresg exercse o use he acual HMM deses (raher ha radom esmaes) o deerme how he oal log lkelhood score vares as you re-esmae he deses sarg from he deal codos ; however hs exercse s jus o show how good a soluo ca be obaed f you ca ge pas local mma of he re-esmao roues. I summary, you are asked o buld wo ses of HMM re-esmao roues, oe based o he forward-backward algorhm, oe based o he Verb sae sequece, ad o deerme how well your algorhms work o wo ypes of models, amely a ergodc model wh 5 saes ad 5 possble observaos each sae, ad a lef-rgh model wh 5 saes ad 5 possble observaos each sae. You should cosder usg several radom sars o deerme he bes soluo for each model ad each rag se. You are also gve wo versos of each model, oe wh radomly chose values for he sae pror desy, he sae raso marx ad he sae observao marx, ad oe wh skewed values for he sae pror desy ad he sae raso marx. Hece you should experme wh each of hese rag ses o udersad whch rag mehods work bes, ad why. Fally, you should vesgae he effecs of usg hghly cosraed models (such as he lef-rgh model) o he sae pror desy esmao ad he sae raso marx esmao. 2/9/2005 :55 AM 0 HMM Specal Projec
Oher quesos you should address as par of hs projec clude he followg:. for cases whe he coverged lkelhood s comparable o he geeraor lkelhood, how do he model deses mach hose of he geeraor? 2. for cases whe he coverged lkelhood s comparable o he geeraor lkelhood, how do he model saes mach hose of he geeraor? 3. for cases whe he coverged lkelhood s smaller ha he geeraor lkelhood (.e., covergece a a local mmum bu o he global mmum), wha s happeg wh he saes ad how do he resulg models compare o he geeraor model? 4. how does he speed of covergece compare for he forward-backward re-esmao ad he Verb re-esmao mehods. 5. how do he compuaoal requremes for he forward-backward ad Verb mehods compare? 6. how do he lkelhood scores of he geeraor models compare whe usg forwardbackward scorg wh hose whe usg Verb scorg? Wha accous for hese dffereces ad how do hey compare for radom ad skewed rag sequeces. 7. wha do you hk would be he effec of fewer rag sequeces (you ca ry hs ou by usg less ha he ex=50 sequeces suppled)? Wha do you hk would be he effec of more rag sequeces? If you are successful makg he re-esmao mehods work properly ad effcely, you mgh wa o buld your ow HMM sequece geeraor ad experme wh creag a rage of model observaos ad see how well your re-esmao roues work o hs ew daa. I s especally eresg o vesgae he effec of creased umbers of observao sequeces o deerme how may are eeded o make he covergece more rapd ad less sesve o he al esmaes of model parameers. 2/9/2005 :55 AM HMM Specal Projec