The Performance of Expectation Maximization (EM) Algorithm in Gaussian Mixed Models (GMM)

Transcription

1 Peranka J. Sc. & Technol. 7 (): 3 43 (9) ISSN: Unvers Pura Malaysa Press The Performance of Expecaon Maxmzaon (EM) Algorhm n Gaussan Mxed Models (GMM) Mohd. Izhan Mohd. Yusoff *, Mohd. Rzam Abu Bakar and Abu Hassan Shaar Mohd. Nor 3 Operaon Suppor Sysems Program, Appled Research Dvson, Telekom Research & Developmen Sdn Bhd, TMR&D Innovaon Cenre, Lngkaran Teknokra Tmur, 63 Cyberjaya, Selangor, Malaysa Deparmen of Mahemacs, Faculy of Scence, Unvers Pura Malaysa, 434 UPM Serdang, Selangor, Malaysa 3 Faculy of Economcs and Busness, Unvers Kebangsaan Malaysa, 436 UKM, Bang, Selangor, Malaysa * E-mal: zhan@mrnd.com.my ABSTRACT Expecaon Maxmzaon (EM) algorhm has experenced a sgnfcan ncrease n erms of usage n many felds of sudy. In hs paper, he performance of he sad algorhm n fndng he Maxmum Lkelhood for he Gaussan Mxed Models (GMM), a probablsc model normally used n fraud deecon and recognzng a person s voce n speech recognon feld, s shown and dscussed. A he end of he paper, some suggesons for fuure research works wll also be gven. Keywords: Expecaon Maxmzaon (EM), Gaussan Mxed Models (GMM), Box and Muller Transformaon INTRODUCTION Every year, elecommuncaon companes regser heavy loses due o fraud acves amounng o mllon of dollars. Vendors, seeng he above as an opporuny no o be mssed, compee o provde daa mnng applcaons whch could deec he sad acvy effecvely usng mehods such as OLAP, devaon based ouler deecon, Hdden Markov Model, and he model whch became he focal area of hs paper, he Gaussan Mxed Models (GMM). GMM s bes known n provdng a robus speaker represenaon for he dffcul ask of speaker denfcaon on shor-me speech specra, whch s a cosne, ransformed of log energy fler oupus from processed magnude specrum from a ms shor me segmen of speech, by smulaed me-scale fler-bank (Reynolds e al., 995). Is funcon s furher exended o deec fraud acves on daly number of calls and lengh of calls occurrng durng he offce hours, he evenng hours and he ngh hours for boh naonal and nernaonal calls (Mohd Yusoff e al., 6; Tangush e al., 998). Maxmum lkelhood esmaon for GMM s dffcul o fnd and he soluon s Expecaon Maxmzaon (EM) algorhm. The EM algorhm was frs nroduced by Dempser e al. (977) and snce hen, here has been a sgnfcan ncrease n erms of s usage, parcularly n fndng he Receved: 5 Sepember 7 Acceped: 7 February 9 * Correspondng Auhor

2 Mohd. Izhan Mohd. Yusoff, Mohd. Rzam Abu Bakar and Abu Hassan Shaar Mohd Nor Maxmum Lkelhood for probablsc models (such as mssng daa, groupng, censorng, runcaon, and fne mxures). The man ssue wh respec o he EM algorhm s fndng he rgh choce of nal parameers and he number of componens. Ths parcular ssue or problem s llusraed n several examples n hs paper. The subsequen secons provde a bref nroducon of he EM algorhm and GMM, generae he smulaon unvarae and mulvarae daa wh clear and hdden componens, presen he resuls gahered from he GMM and EM algorhm where he emphass gven on he choce of he nal parameers and he number of componens, and some suggesons for fuure research works. The Gaussan Mxed Models (GMM) and Expecaon Maxmzaon (EM) Algorhm Le x R d and K be he number of componens where each componen havng s own pror probably (wegh) a and probably densy funcon wh he mean μ and covarance Σ. All of hem are mxed resulng n he followng formula, whch s also known as he Gaussan Mxed Models (GMM): K a ( x K, ) a ( ) exp x n R x n z n R = d -^ - h ^ - h / / n = = d r R () K where pror probably (wegh) of componen ha s a sasfy he consran / a =. For he case of voce recognon, assumng here are n number of speakers. The m number of samples were colleced from each speaker. Equaon () s derved for each sample, where s parameers were kep n he daabase for comparson purposes. Fraud deecon would follow smlar seps. n From equaon (), he lkelhood funcon and log lkelhood funcon by L( X ) = % f( xj ) n K j = and, l( X ) = log L( X ) = / log e/ az( x j n, R) owere defned, respecvely. The maxmum j= = lkelhood esmaon (m.l.e) amed a fndng θˆ whch maxmzed l(x θ), wh respec o θ (Marda K e al., 979). The expresson log e/ az( x j n, R) o n he log lkelhood funcon s dffcul o = solve, and n order o overcome hs problem, he Expecaon Maxmzaon (EM) algorhm was used. In he EM algorhm, he dsrbuon of X needs o be esmaed n he sample space χ, bu X can only be observed ndrecly hrough Y n he sample space Y. In many cases, here s a mappng x y(x) from χ o Y, and x s only known o le n a subse of χ, denoed by χ (y), whch s deermned by equaon y = y(x). The dsrbuon of X s parameerzed by a famly of dsrbuons f(x θ), wh parameers θ Ω or x. The dsrbuon of Y, g(y θ) s herefore: gy ( ) = # fx ( ) dx () () y The EM algorhm ams a fndng θ whch maxmzes g(y θ) gven an observed y. Le he funcon Q(θ θ) = E(log f(x θ ) y,θ) (3) - = 3 Peranka J. Sc. & Technol. Vol. 7 () 9

3 The Performance of Expecaon Maxmzaon (EM) Algorhm n Gaussan Mxed Models (GMM) be he expeced value of log f(x θ ) gven y and θ. The expecaon was assumed o exs for all he pars (θ,θ). In parcular, was assumed ha f(x θ) > for θ Ω. EM Ieraon E-Sep: Compue Q(θ θ (p) ) M-sep: Choose θ (p+) o be a value of θ Ω ha maxmzes Q(θ θ (p) ) (Dempser e al., 977). In he n ' ' ' case of GMM, was defned ha Q(θ θ) = E< log % ayz_ x ny, Ry X, F, where y {,,, K}, = y =k f he h sample was generaed by he k h mxure componen. I was smplfed usng (among oher) he Bayes formula whch s f(θ x) f(x θ)p(θ), where f(θ x) = poseror probably, f(x θ) = lkelhood funcon, and P(θ) = pror probably (Tsay, 5; Blmes, 997) o he followng equaons: n K ' ' ' Q( ' ) = // pk, logak + / / pk, log z_ x nk, Rk (4) = k= = k= n K where p k = a / a l z_ x n, R _ x n, R k k k z l l l (5) and - (, ) ( ) ( ) x x nk Rk x nk z nk Rk = expd n (6) d ( r) R k The EM Ieraon (for GMM) E-Sep: Equaon (5) s calculaed. M-Sep: The followng formulas (derved from he Lagrange mulplers, respecvely) are calculaed. Furher deals are gven n Appendx A.. Q = nj and Q R = -, j a j = n/ pj (7) / pjx n j = / p j (8) Peranka J. Sc. & Technol. Vol. 7 () 9 33

4 Mohd. Izhan Mohd. Yusoff, Mohd. Rzam Abu Bakar and Abu Hassan Shaar Mohd Nor R = j / pj^x-njh^x-njh / p j (9) The above seps (.e. E-sep and M-sep) were repeaed unl a convergence was acheved. SIMULATION DATA A program called Smulae was developed (usng C++ language) o generae smulaon daa for equaon () wh he parameers as per gven n Table (aken from Ever e al., 98, wh modfcaons) usng Box and Muller Transformaon (Box e al., 958) and Equaon (). The smulaon daa were hen labelled as Sample, Sample, Sample3 and Sample4, and her hsograms are shown n Fgs. and. zj = n+ (- v loguj) cos ruj+ zj+ = n+ (- v loguj) sn ruj+, uj, uj+ ~ U(, ) () The Smulae program would generae one random number, denoed by U, from he unform dsrbuon U(,), and check wheher was less han say a (=,). If he answer s yes, he wo random numbers, denoed by U and U 3, are generaed from he unform dsrbuon U(,) and used n he compung equaon (), along wh he correspondng µ and σ, aken from Table. In hs sudy, hese seps were repeaed unl observaons were obaned. For Sample 4, apar from equaon (), he formulas gven n Appendx A. (n he marx forma) were also used. In Fg.., wo humps are observed and hese represen wo componens: (µ,σ ) = (.,.) and (µ,σ ) = (.,.5). Boh of hem are well-separaed, n whch he observaons for he laer componen are grouped around he mean. One would never expec o fnd he wo componens n Fg... The hsogram s domnaed by he componen (µ,σ ) = (.,.) due o he fac ha a =.85. In Fg..3, wo humps are vvdly dsplayed and hey represen wo componens: (µ,σ ) = (-.,.5) and (µ 3,σ 33 ) = (4.,4.). The hrd componen, (µ,σ ) = (.,.), s hdden from he vew by he wo componens ndcaed earler. The observaons are grouped around he mean for he componen (µ,σ ) = (-.,.5). The hsograms n Fgs.. and. appear o spl no wo represenng componens (μ, μ ) = (5., 5.9) and (μ, μ 3 )=(.78,.95), respecvely; whereas Fgs..3 and.4 no hree represenng componens (μ 3, μ 3, μ 33 ) = (.46, 4., 5.48) and (μ 4, μ 4, μ 34 ) = (.5,.3,.98), respecvely. 34 Peranka J. Sc. & Technol. Vol. 7 () 9

5 The Performance of Expecaon Maxmzaon (EM) Algorhm n Gaussan Mxed Models (GMM) Table a s, µ s, n s, v s and R s for each sample used n he Smulae program. The number of observaons generaed by he program s gven n he bracke Sample (N=) a =.4 a =.6 µ =. µ =. σ =. σ =.5 Sample (N=) a =.85 a =.5 µ =. µ =. σ =. σ =.5 Sample3 (N=) a =.33 a =.33 a 3 =.34 µ =. µ =-. µ 3 =4. σ =. σ =.5 σ 33 =4. Sample4 (N=) a =.33 a =.3 μ = μ = Σ = Σ = a 3 =.37 μ 3 = Σ 3 = Peranka J. Sc. & Technol. Vol. 7 () 9 35

6 Mohd. Izhan Mohd. Yusoff, Mohd. Rzam Abu Bakar and Abu Hassan Shaar Mohd Nor (.) (.) (.3) Fg. : The hsograms of Sample (wh overall mean and sandard devaon equal o.8 and.9, respecvely); Sample (wh overall mean and sandard devaon equal o.34 and.9, respecvely); and Sample3 (wh overall mean and sandard devaon equal o.8 and.6, respecvely) 36 Peranka J. Sc. & Technol. Vol. 7 () 9

7 The Performance of Expecaon Maxmzaon (EM) Algorhm n Gaussan Mxed Models (GMM) (.) (.) (.3) (.4) Fg. : The hsograms of Sample4 x (wh overall mean and sandard devaon equal o 5.88 and.8, respecvely); x (wh overall mean and sandard devaon equal o 3.5 and.44, respecvely); x 3 (wh overall mean and sandard devaon equal o 3.79 and.75, respecvely); and x 4 (wh overall mean and sandard devaon equal o. and.8, respecvely) RESULTS A program known as he GMM was developed usng he Java language o fnd he parameers of equaon () by employng he EM algorhm, where eraon s sopped when θ p+ -θ p <.. Oher mehods nvolved n he calculaon of EM algorhm nclude he Cholesky mehod (Marda e al., 979). In hs secon, wo scenaros are herefore presened. Scenaro : In Table, wh he excepon of Sample4 (where nal parameers were aken from Ever e al., 98), he nal parameers for Sample, Sample and Sample3 were deermned usng vsual nspecon of he hsograms gven n Fg.. Ths was done by concenrang on he observaon(s) ha gave he hghes frequency, as shown by he componens whch were clearly dsplayed. Peranka J. Sc. & Technol. Vol. 7 () 9 37

8 Mohd. Izhan Mohd. Yusoff, Mohd. Rzam Abu Bakar and Abu Hassan Shaar Mohd Nor Table a s, µ s, n s, σ s and R s for each sample used n he GMM program, where hey were reaed as he nal parameers Sample a =.5 a =.5 µ =. µ =. σ =. σ =. Sample a =.5 a =.5 µ =. µ =.5 σ =. σ =. Sample3 a =.33 a =.33 a 3 =.34 µ =. µ =-. µ 3 =4. σ =. σ =. σ 33 =. Sample4 a =.33 4 R S μ = 4 S R = S S T R a =.3 7 S μ S = R 3 = S S T R a 3 =.37 8 S S μ 3 = 4 R3 = S 5 3 S T V W W W W X V W W W W X V W W W W X The values gven n Table were used by he GMM program as he nal parameers o fnd he fnal ones for he smulaon daa, as shown n Fgs. and. The resuls are as abulaed below. 38 Peranka J. Sc. & Technol. Vol. 7 () 9

9 The Performance of Expecaon Maxmzaon (EM) Algorhm n Gaussan Mxed Models (GMM) Table 3 â s, û s, û s, ˆσ s and ˆΣ s for each sample produced by he GMM program, usng (Table ) as he nal parameers. The GMM program converged s gven n he bracke I s crucal o noe ha for he unvarae samples, he convergence was acheved wh more han eraons, whle for he mulvarae samples, less han eraons were requred. The choce of he nal parameers mgh play an mporan role n makng he convergence process faser, as llusraed by he laer. The GMM program managed o fnd (fnal) parameers even n cases where he componens were hdden from he vew, bu hs s provded ha he number of componens and he observaons whch gve he hghes frequency for he denfable componens are known. Scenaro : Grea care should be aken when choosng he nal parameers (o sar he EM algorhm) as well as he number of componens, where wrong choce wll lead o he suaon exemplfed n Table 4. Oher examples can be found n Ever e al. (98) and Reynolds e al. (995). Peranka J. Sc. & Technol. Vol. 7 () 9 39

10 Mohd. Izhan Mohd. Yusoff, Mohd. Rzam Abu Bakar and Abu Hassan Shaar Mohd Nor Table 4 The nal ( s row) and fnal ( nd row) parameers for Sample 3 (chosen for havng hdden componens), where: wo componens were used for (4.), four componens were used for (4.), and sx componens were used for (4.3) and (4.4). The acual number of he componens s hree (4.) (4.) (4.3) (4.3) 4 Peranka J. Sc. & Technol. Vol. 7 () 9

11 The Performance of Expecaon Maxmzaon (EM) Algorhm n Gaussan Mxed Models (GMM) (4.4) Noce ha Table 4. s n 3 = n 4 = -. and v = v =.89 and f a a 4 were compued,.8 would herefore be obaned, and hs s no far dfferen from he ones gven n Table 3. Table 4.3 also shows smlar resuls, where û 3 = = ˆμ 6 = -., = v 33 = = v 66 =.89 and a 3 = = a 6 =.7 where a a 6 =.8. Despe convergng a eraon no. (he lowes so far), he fnal parameers shown n Table 4.4 are compleely dfferen from hose n Table 3, and hs s a drec consequence from gnorng he characerscs shown by he observaons n he hsograms. CONCLUSIONS In he prevous secons, Sample, Sample, Sample3, and Sample4 (usng a program called Smulae ) were generaed wh known number of boh componens and parameers. Usng he same nformaon, parcularly on he number of componens and deermnng he nal parameers o sar he EM algorhm by nspecng he hsograms, he fnal parameers produced from he EM algorhm (usng he program known as he GMM ) are smlar o he real ones. Jus o show how mporan he process of choosng he nal parameers s (o sar he EM algorhm) and he number of componens, Sample3 was seleced for havng hdden componens, whle he process of deermnng he nal parameers o sar EM algorhm (.e. by nspecng he hsograms) and reducng he number of componens was repeaed; he fnal parameers produced were ncorrec. The same resuls were also obaned when he number of componens was ncreased; for he nal parameers o sar he EM algorhm, le he mean equals o and he sandard devaon equals o (a common msake done by mos of he praconers). In conrary o he above, when he number of componens was ncreased and he nal parameers o sar he EM algorhm was deermned by nspecng he hsograms and for he res (especally he hdden componens) by leng he mean equals o and sandard devaon equals o, he fnal parameers produced (wh mnor adjusmens) were smlar o he real ones (a characersc where some mgh consder as unmporan and herefore choose o gnore). The deermnaon of he nal parameers o sar he EM algorhm could be made easer and z faser usng he graphcal echnques such as plong log + agans x z where each approxmaely sragh lne, wh negave slope represens an area where one componen domnaes and he kernel Peranka J. Sc. & Technol. Vol. 7 () 9 4

12 Mohd. Izhan Mohd. Yusoff, Mohd. Rzam Abu Bakar and Abu Hassan Shaar Mohd Nor m mehod defned by f ( ) exp m kl exp h l exp b a m m- r m kl r r k = / c- m e- d n oe / pk c mo, m = r l=- m k= (Ever e al., 98; Bhaacharya, 967; Slverman, 986). Neverheless, he man dsadvanage of boh mehods s ha hey can no be used o deec hdden componens. - Appendx A A. Smulae program uses he followng formulas o produce Sample4 (where he subscrp represens he dmenson of he marx). X nx = C nxn Z nx + m nx, Σ nxn = C nxn C nxn where c j Z ] ] v ] = [ ] fv ] ] \ j- j - k= / cc k jk j- jj - / cc jk jk k= p, j #, j and z, h componen of Z, s as per defned n equaon (), where μ and σ are se/fxed a and, respecvely. A. Dervaon of Equaons (7), (8) and (9) A.. Usng Lagrange mulplers defned by max/mn F(x,y,z) subjec o Φ(x,y,z)=, G(x,y,z)=F(x,y,z)+ λφ(x,y,z), G x, G y, G = = = z (Spegel, 974) on max // pjlog( aj) j subjec o / aj = (or / aj - =, Equaon (7) would be obaned. j c j m A.. From p x x - j -nj R j - n j = n d // ^ h ^ hn, equaon (8) would be obaned usng he j j followng marx properes, xay x Ay, ax = = x a A..3 The frs and second expressons of p x x p log d // j^ -njhr j ^ - njhn+ - d // j Rj n= use he followng Rj j Rj j r( xy) marx properes, = y+ y - Dag() y, and / x Ax r( A x x ) x = / o ge equaon (9). (Marda e al., 979). 4 Peranka J. Sc. & Technol. Vol. 7 () 9

13 The Performance of Expecaon Maxmzaon (EM) Algorhm n Gaussan Mxed Models (GMM) REFERENCES Bhaacharya, C.G. (967). A smple mehod of resoluon of a dsrbuon no Gaussan componens. Bomercs, 3, Blmes, J.A. (997). A Genle uoral of he EM algorhm and s applcaon o parameer esmaon for Gaussan Mxure and Hdden Markov Models. Techncal Repor, Unversy of Berkeley, ICSI-TR-97-. Box, G.E.P. and Muller, M.E. (958). A noe on he generang of random normal devaes. Annals of Mahemacal Sascs, 9, 6-6. Dempser, A.P., Lard, N.M. and Rubn, D.B. (977). Maxmum Lkelhood from ncomplee daa va he EM algorhm. Journal of Royal Sascs Socey, 39(), -. Ever, B.S. and Hand, D.J. (98). Fne Mxure Dsrbuons. London: Chapman and Hall Ld. Marda, K.V., Ken, J.T. and Bbby, J.M. (979). Mulvarae Analyss. London: Academc Press Inc. Ld. Mohd Yusoff, M.I., Abu Bakar, M.R. and Mohd Nor, A.H.S. (7). Fraud deecon n elecommuncaon usng Daa Mnng applcaon. In Proceedngs of 9 h Islamc Counres Conference on Sascal Scences 7 (ICCS-IX). Reynolds, D.A. and Rose, R.C. (995). Robus ex-ndependen speaker denfcaon usng Gaussan Mxure Speaker Models. IEEE Transacons on Speech and Audo Processng, 3(), January 995. Slverman, B.W. (986). Densy Esmaon for Sascs and Daa Analyss. London: Chapman and Hall Ld. Spegel, M.R. (974). Shaum s Oulne Seres: Theory and Problems of Advanced Calculus: SI (Merc) edon. Mc Graw-Hlls, Inc. Tangush, M., Haf, M., Hollmen, J. and Tresp, V. (998). Fraud deecon n communcaons neworks usng neural and probablsc mehods. In Proceedng of he 998 IEEE Inernaonal Conference n Acouscs, Speech and Sgnal Processng (ICASSP 98), II, Tsay, R.S. (5). Analyss of Fnancal Tme Seres: Fnancal Economercs. John Wley and Sons. Peranka J. Sc. & Technol. Vol. 7 () 9 43