Maximum Likelihood Estimation of the Multivariate Normal Mixture Model

Transcription

1 MPRA Munch Personal RePEc Archve Maxmum Lkelhood Esmaon of he Mulvarae Normal Mxure Model Ola Boldea and Jan R. Magnus Unversy of Tlburg 2009 Onlne a hp://mpra.ub.un-muenchen.de/23149/ MPRA Paper No , posed 8. June :47 UTC

2 Maxmum lkelhood esmaon of he mulvarae normal mxure model Ola Boldea Jan R. Magnus May Revson acceped May 15, 2009 Forhcomng n: Journal of he Amercan Sascal Assocaon, Theory and Mehods Secon Proposed runnng head: ML Esmaon of he Mulvarae Normal Mxure Model Absrac: The Hessan of he mulvarae normal mxure model s derved, and esmaors of he nformaon marx are obaned, hus enablng conssen esmaon of all parameers and her precsons. The usefulness of he new heory s llusraed wh wo examples and some smulaon expermens. The newly proposed esmaors appear o be superor o he exsng ones. Key words: Mxure model; Maxmum lkelhood; Informaon marx Ola Boldea (E-mal: o.boldea@uv.nl) and Jan R. Magnus (E-mal: magnus@uv.nl) are boh a he Deparmen of Economercs & OR, Tlburg Unversy, PO Box 90153, 5000 LE Tlburg, The Neherlands. The auhors are graeful o Hamparsum Bozdogan for askng a queson whch led o hs paper, o John Enmahl and Evangelos Evangelou for useful commens, o Geoffrey McLachlan for provdng hs EMMIX FORTRAN code free of charge and for hs promp response o our quesons, and o he edor, assocae edor, and he referees for helpful commens. The frs verson of hs paper was wren durng a vs of one of us o he Wang Yanan Insue for Sudes n Economcs (WISE), Xamen Unversy, Chna. 1

3 1 Inroducon In fne mxure models s assumed ha daa are obaned from a fne collecon of populaons and ha he daa whn each populaon follow a sandard dsrbuon, ypcally normal, Posson, or bnomal. Such models are parcularly useful when he daa come from mulple sources, and hey fnd applcaon n such vared felds as crmnology, engneerng, demography, economcs, psychology, markeng, socology, plan pahology, and epdemology. The normal (Gaussan) model has receved mos aenon. Here we consder an m-dmensonal random vecor x whose dsrbuon s a mxure (weghed average) of g normal denses, so ha g f(x) = π f (x), (1) where =1 f (x) = (2π) m/2 V 1/2 exp{ 1 2 (x µ ) V 1 (x µ )} (2) and he π are weghs sasfyng π > 0 and π = 1. Ths s he socalled mulvarae normal mxure model. The parameers of he model are (π, µ, V ) for = 1,..., g subjec o wo consrans, namely ha he π sum o one and ha he V are symmerc (n fac, posve defne). The orgn of mxure models s usually arbued o Newcomb (1886) and Pearson (1894), alhough some ffy years earler Posson already used mxures o analyze convcon raes; see Sgler (1986). Bu was only afer he nroducon of he EM algorhm by Dempser e al. (1977) ha mxure models have ganed wde populary n appled sascs. Snce hen an exensve leraure has developed. Imporan revews are gven n Terngon e al. (1985), McLachlan and Basford (1988), and McLachlan and Peel (2000). There are wo heorecal problems wh mxures. Frs, as noed by Day (1969) and Hahaway (1985), he lkelhood may be unbounded n whch case he maxmum lkelhood (ML) esmaor does no exs. However, we can sll deermne a sequence of roos of he lkelhood equaon ha s conssen and asympocally effcen; see McLachlan and Basford (1988, Sec. 1.8). Hence, hs s no necessarly a problem n pracce. Second, he parameers are no denfed unless we mpose an addonal resrcon, such as π 1 π 2 π g, 2

4 see Terngon e al. (1985, Sec. 3.1). Ths s no a problem n pracce eher, and we follow Aken and Rubn (1985) by mposng he resrcon bu carryng ou he ML esmaon whou. The ask of esmang he parameers and her precsons, and formulang confdence nervals and es sascs, s dffcul and edous. Ths s smply because n sandard suaons wh ndependen and dencally dsrbued observaons, he lkelhood conans producs and herefore he loglkelhood conans sums. Bu here he lkelhood self s a sum, and herefore he dervaves of he loglkelhood wll conan raos. Takng expecaons s herefore ypcally no feasble. Even he ask of obanng he dervaves of he loglkelhood (score and Hessan marx) s no rval. Currenly here are several mehods o esmae he varance marx of he ML esmaor n (mulvarae) mxure models n erms of he nverse of he observed nformaon marx, and hey dffer by he way hs nverse s approxmaed. One mehod nvolves usng he complee-daa loglkelhood, ha s, he loglkelhood of an augmened daa problem, where he assgnmen of each observaon o a mxure componen s an unobserved varable comng from a prespecfed mulnomal dsrbuon. The advanage of usng he complee-daa loglkelhood nsead of he ncomplee-daa (he orgnal daa) loglkelhood les n s form as a sum of logarhms raher han a logarhm of a sum. The nformaon marx for he ncomplee daa can be shown o depend only on he condonal momens of he graden and curvaure of he complee-daa loglkelhood funcon and so can be readly compued; see Lous (1982). Anoher mehod, n he conex of he orgnal loglkelhood, was proposed by Dez and Böhnng (1996), explong he fac ha n large samples from regular models, wce he change n loglkelhood on omng ha varable s equal o he square of he -sasc of ha varable; see McLachlan and Peel (2000, p. 68). Ths mehod was exended by Lu (1998) o mulvarae models. There s also a condonal boosrap approach descrbed n McLachlan and Peel (2000, p. 67). In addon, he sandard errors of he ML esmaor can be compued by a leas hree boosrap mehods: he paramerc boosrap (Basford e al. 1997; McLachlan and Peel 2000), he non-paramerc boosrap (McLachlan and Peel 2000) whch s an exenson of Efron (1979), and he weghed boosrap (Newon and Rafery 1994) whch s a verson of he nonparamerc boosrap based on scalng he daa wh weghs ha are proporonal o he number of mes an orgnal pon occurs n he boosrap sample. Basford e al. (1997) compare he paramerc boosrap wh a mehod based on he ouer produc of he scores as a proxy for he observed nformaon marx, and fnd smulaon evdence ha he boosrap-based sandard errors are more relable n small samples. 3

5 In hs paper we explcly derve he score and Hessan marx for he mulvarae normal mxure model, and use he resuls o esmae he nformaon marx. Ths provdes a wofold exenson of Behboodan (1972) and Al and Nadarajah (2007), who sudy he nformaon marx for he case of a mxure of wo (raher hen g) unvarae (raher han mulvarae) normal dsrbuons. Snce we work wh he orgnal ( ncomplee ) loglkelhood, we compare our nformaon-based sandard errors o he boosrap-based sandard errors whch are he naural small-sample counerpar. We fnd ha n correcly specfed models he mehod based on he observed Hessan-based nformaon marx s he bes n erms of roo mean squared error. In msspecfed models he mehod based on he observed sandwch marx s he bes. Ths paper s organzed as follows. In Secon 2 we dscuss how o ake accoun of he wo consrans: symmery of he varance marces and he fac ha he weghs sum o one. Our general resul (Theorem 1) s formulaed n Secon 3, where we also dscuss he esmaon of he varance of he ML esmaor and nroduce he msspecfcaon-robus sandwch marx. These resuls allow us o formally es for msspecfcaon usng he Informaon Marx es (Theorem 2), dscussed n Secon 4. In Secon 5 we presen he mporan specal case (Theorem 3) where all varance marces are equal. In Secon 6 we sudy wo well-known examples based on he hemophla daa se and he Irs daa se. These examples demonsrae ha our formulae can be mplemened whou any problems and ha he resuls are credble. Bu hese examples do no ye prove ha he nformaon-based esmaes of he sandard errors are more accurae han he ones currenly n use. Therefore we provde Mone Carlo evdence n Secon 7. Secon 8 concludes. An Appendx conans proofs of he hree heorems. 2 Symmery and wegh consrans Before we derve he score vecor and he Hessan marx, we need o dscuss wo consrans ha play a role n mxure models: symmery of he varance marces and he fac ha he weghs sum o one. To deal wh he symmery consran we nroduce he half-vec operaor vech( ) and he duplcaon marx D; see Magnus and Neudecker (1988) and Magnus (1988). Le V be a symmerc m m marx, and le vech V denoe he 1 m(m+1) 1 vecor ha 2 s obaned from vec V by elmnang all supradagonal elemens of V. Then he elemens of vec V are hose of vech V wh some repeons. Hence, here exss a unque m 2 1 m(m+1) marx D, such ha D vech V = vec V. Snce 2 he elemens of V are consraned by he symmery, we mus dfferenae wh respec o vech V and no wh respec o vec V. 4

6 The weghs π mus all be posve and hey mus sum o one. We maxmze wh respec o π = (π 1, π 2,...,π g 1 ) and se π g = 1 π 1 π g 1. We have where d log π = a dπ, d 2 log π = (dπ) a a (dπ), (3) a = (1/π )e ( = 1,...,p 1), a g = (1/π g )ı, (4) e denoes he -h column of he deny marx I g 1, and ı s he (g 1)- dmensonal vecor of ones. The model parameers are hen π and, for = 1,...,g, µ and vech V. Wrng ( ) µ θ =, vech V he complee parameer vecor can be expressed as θ = (π, θ 1,...,θ g). 3 Score vecor, Hessan and varance marx Gven a sample x 1,...,x n of ndependen and dencally dsrbued random varables from he dsrbuon (1), we wre he loglkelhood as L(θ) = n log f(x ). =1 The score vecor s defned by q(θ) = q (θ), where q (θ) = log f(x ) θ = vec(q π, q1,..., qg ), and he Hessan marx by Q(θ) = Q (θ), where Q ππ Q π1... Q πg Q (θ) = 2 log f(x ) Q 1π Q Q 1g = θ θ.... Q gπ Q g1... Q gg Before we can sae our man resul we need some more noaon. We defne φ = π f (x ), α = φ j φ, (5) j b = V 1 (x µ ), B = V 1 b b, (6) 5

7 and ( C = V 1 ( ) b c = 1, (7) 2 D vec B D (b V 1 1 ) 2 D ((V 1 (b V ) 1 )D 2B ) V 1. (8) )D We also recall ha a s defned n (4) and we le ā = α a. We can now sae Theorem 1, whch allows drec calculaon of he score and Hessan marx. Theorem 1: The conrbuon of he -h observaon o he score vecor wh respec o he parameers π and θ ( = 1,...,g) s gven by q π = ā, q = α c, and he conrbuon of he -h observaon o he Hessan marx s and Q ππ = ā ā, Q π = α (a ā )c, Q = (α C α (1 α )c c ), Qj = α α j c c j ( j). We noe ha he expressons for he score n Theorem 1 are he same as n Basford e al. (1997). The expressons for he Hessan are new. We nex dscuss he esmaon of he varance of ˆθ. In maxmum lkelhood heory he varance s usually obaned from he nformaon marx. If he model s correcly specfed, hen he nformaon marx s defned by I = E(Q) = E(qq ), where he equaly holds because of second-order regulary. In our case we can no oban hese expecaons analycally. Moreover, we can no be ceran ha he model s correcly specfed. We esmae he nformaon marx by n I 1 = q (ˆθ) q (ˆθ), =1 based on frs-order dervaves, or by I 2 = Q(ˆθ) = n Q (ˆθ), =1 6

8 based on second-order dervaves. The nverses I 1 1 and I 1 2 are conssen esmaors of he asympoc varance of ˆθ f he model s correcly specfed. In general, he sandwch (or robus ) varance marx I 1 3 = var(ˆθ) = I 1 2 I 1 I 1 2 (9) provdes a conssen esmaor of he varance marx, wheher or no he model s no correcly specfed. Ths was noed by Huber (1967), Whe (1982), and ohers, and s based on he realzaon ha he asympoc normaly of ˆθ ress on he facs ha he expeced value of (1/n)q(θ)q(θ) has a fne posve semdefne (possbly sngular) lm, say I 1, and ha (1/n)Q(θ) converges n probably o a posve defne marx, say I 2, and ha hese wo lmng marces need no be equal; see also Davdson and MacKnnon (2004, pp ). We noe n passng an mporan and somewha counernuve propery of he sandwch esmaor, whch s seldom menoned. If I 1 = I 2, hen I 1 = I 2 = I 3. If I 1 I 2, hen one would perhaps expec ha I 1 3 les n-beween I 1 1 and I 1 2, bu hs s ypcally no he case, as s easly demonsraed. Le Ψ = I 1 1 I 1 2. Then, I 1 3 = I 1 2 I 1 I 1 2 = I 1 2 (I Ψ) 1 I 1 2 = (I 2 + I 2 ΨI 2 ) 1. If Ψ s posve defne (I 1 2 < I 1 1 ) hen I 1 3 < I 1 2 < I 1 1 ; f Ψ s negave defne (I 1 2 > I 1 1 ) hen I 1 3 > I 1 2 > I 1 1. In pracce here s no reason why Ψ should be eher posve defne or negave defne. Neverheless, we should expec an ndvdual varance based on he Hessan o le n-beween he varance based on he score and he varance based on he robus esmaor, and hs expecaon s confrmed by he smulaon resuls n Secon 7. 4 Informaon marx es The nformaon marx (IM) es, nroduced by Whe (1982), s well known as a general es for msspecfcaon of a paramerc lkelhood funcon. Despe he fac ha he asympoc dsrbuon s a poor approxmaon o he fne-sample dsrbuon of he es sasc, he IM es has esablshed self n he economercs professon. Below we oban he IM es for mxure models. Le us defne W (θ) = Q (θ) + q (θ)q (θ). 7

9 From Theorem 1 we see ha 0 a 1 (q 1 ) a 2 (q 2 )... a g (q g ) q 1a 1 W W (θ) = q 2a 2 0 W , q g a g W g where a and q have been defned before, and ( W = α (C c c B Γ ) = α D ) D Γ D D wh represenng skewness, and = (1/2)(V 1 Γ = b V 1 + (1/2)(vecB )b V 1 ) B V 1 (1/4)(vec B )(vec B ) represenng kuross. The purpose of he nformaon marx procedure s o es for he jon sgnfcance of he non-redundan elemens of he marx W(ˆθ) = W (ˆθ). Now, snce q(ˆθ) = q (ˆθ) = 0, he IM procedure n our case ess for he jon sgnfcance of he non-redundan elemens of W (ˆθ) for = 1,..., g. Followng Chesher (1983) and Lancaser (1984) we formulae he Whe s (1982) IM es as follows. Theorem 2 (Informaon Marx es): Defne he varance marx Σ(θ) = 1 n n =1 w w ( 1 n ) ( n w q 1 n =1 ) 1 ( n q q 1 n where q denoes he -h ncremen o he score, and w = vec ( vech W 1, vech W ) 2 g,..., vech W. =1 ) n q w Then, evaluaed a ˆθ and under he null hypohess of correc specfcaon, ( ) ( ) 1 n IM = n w Σ 1 1 n w n n =1 asympocally follows a χ 2 -dsrbuon wh gm(m+3)/2 degrees of freedom. 8 =1 =1

10 The above form of he IM es s a varan of he ouer-produc-of-hegraden (OPG) regresson, ofen used o calculae Lagrange mulpler ess. Such ess are known o rejec rue null hypoheses far oo ofen n fne samples, and hs s also rue for he OPG form of he IM es. We llusrae hs fac hrough some smulaons a he end of Secon 7. To use he asympoc crcal values s no a good dea. Insead, hese values can be boosrapped; see Horowz (1994) and Davdson and MacKnnon (2004, Sec. 16.9) for deals and references. 5 Specal case: equal varance marces There are many mporan specal cases of Theorem 1. We may encouner cases where he weghs π are known or where he means µ are equal across dfferen mxures. The mos mporan specal case, however, s he one where he varances V are equal: V = V. Ths s he case presened n Theorem 3. Furher specalzaon s of course possble: V could be dagonal or even proporonal o he deny marx, bu we do no explo hese cases here. When V = V, we wre he parameer vecor as θ = (π, µ 1,..., µ g, v ), where v = vech V. The score s q(θ) = q (θ) wh q (θ) = vec(q π, q1,...,qg, q v ), and he Hessan marx s Q(θ) = Q (θ) wh Q ππ Q π1... Q πg Q 1π Q Q 1g Q (θ) =. Q gπ Q vπ.. Q g1... Q gg Q v1... Q vg Q πv Q 1v. Q gv Q vv. Theorem 3 (V = V ): The conrbuon of he -h observaon o he score vecor wh respec o he parameers π, µ ( = 1,..., g), and v s gven by where q π = ā, q = α b, q v = 1 2 D vec B, B = V 1 g α b b, and he conrbuon of he -h observaon o he Hessan marx s =1 Q ππ = ā ā, Qπ = α (a ā )b, 9

11 Q πv = 1 2 g α (a ā )(vec B ) D, =1 Q = α V 1 + α (1 α )b b, Qj = α α j b b j Q v = α ( b V b (vec(b B )) ) D, ( j), and Q vv = D (( 1 4 g α b b ) V V 1 V 1 =1 g α (vec B )(vec B ) (vec B )(vec B ) )D. =1 As n Theorem 1 we can use hese resuls o compue I 1 1, I 1 2, and I Two examples To llusrae our heorecal resuls we presen wo examples. The maxmum lkelhood esmaes hemselves are usually compued va he EM algorhm, whch s a dervave-free mehod, bu hey can also be compued drecly from he lkelhood or by seng he score equal o zero or n some oher manner. In many cases knowledge of he score (and Hessan) allows an opon whch wll speed up he compuaons; see Xu and Jordan (1996) for a dscusson of graden-based approaches. The resulng esmaes, however, are he same for each mehod. The purpose of he wo examples s o look a he behavor of he nformaon-based sandard error esmaes n pracce and o compare hem o oher avalable mehods. Snce no explc formula for he nformaon marx has been avalable, researchers ypcally compue sandard errors n mulvarae mxure models by means of he boosrap. The well-known EMMIX sofware package developed by McLachlan e al. (1999) repors sandard errors of he esmaes based on four dfferen mehods. Mehods (A1) and (A2) are paramerc and nonparamerc boosrap mehods, respecvely, alored o he nal sample. They perform repeaed draws from eher a mulvarae normal mxure wh parameers fxed a her esmaed values or from he nonparamerc esmae of he samplng dsrbuon of he daa, hen esmae he model for each sample and compue he n-sample boosrap sandard errors of he correspondng parameer esmaes. Mehod (A3) follows Newon and Rafery (1994) and performs he boosrap on a weghed verson of he daa. 10

12 The fourh mehod compues sandard errors from he ouer produc of he score, and s based on Basford e al. (1997, Sec. 3). Ths should be he same as our formula for I 1 1, bu verfcaon of hs fac s no possble because EMMIX does no always provde credble resuls n hs case. Ths leaves us wh hree boosrap mehods o consder. Noe however ha, snce we have coded I 1, we can provde comparsons of he Hessan and sandwch esmaes of sandard errors wh boh boosrap-based and ouer produc-based sandard error esmaes. Furher deals abou he four mehods can be found n McLachlan and Peel (2000, Sec. 2.16). We compare hese hree EM boosrap sandard errors wh he hree sandard errors compued from our formulae. Mehod (B1) employs I 1 1 based on he ouer produc of he score, (B2) uses I 1 2 based on he Hessan marx, whle (B3) uses he robus sandwch marx var ˆθ as gven n (9). We consder wo popular and much-suded daa ses: he hemophla daa se and he Irs daa se. The hemophla daa se Human genes are carred on chromosomes and wo of hese, labeled X and Y, deermne our sex. Females have wo X chromosomes, males have an X and a Y chromosome. Hemophla s a heredary recessve X-lnked blood clong dsorder where an essenal clong facor s eher parly or compleely mssng. Whle only males have hemophla, females can carry he affeced gene and pass on o her chldren. If he moher carres he hemophla gene and he faher does no have hemophla, hen a male chld wll have a 50:50 chance of havng hemophla (because he wll nher one of hs moher s wo X chromosomes, one of whch s fauly) and a female chld wll have a 50:50 chance of carryng he gene (for he same reason). If he moher s no a carrer, bu he faher has hemophla, hen a male chld wll no be affeced (because he nhers hs faher s normal Y chromosome) bu a female chld wll always be a carrer (because she nhers her faher s fauly X chromosome). The hemophla daa were colleced by Habbema e al. (1974), and were exensvely analyzed n a number of papers; see ner ala McLachlan and Peel (2000, pp ). The queson s how o dscrmnae beween normal women and hemophla A carrers on he bass of measuremens on wo varables: anhemophlc facor (AHF) acvy and AHF-lke angen. We have 30 observaons on women who do no carry he hemophla gene and 45 observaons on women who do carry he gene. We hus have n = 75 observaons on m = 2 feaures from g = 2 groups of women. Our fndngs are recorded n Table 1, where all esmaes and sandard 11

13 Table 1: Esmaon resuls Hemophla daa Varable Esmae Sandard Error EM Boosrap Our mehod (A1) (A2) (A3) (B1) (B2) (B3) Wegh π Woman does no carry hemophla µ µ v v v Woman carres hemophla µ µ v v v errors (excep for π 1 ) have been mulpled by 100 o faclae presenaon. The EM boosrap resuls are obaned from 100 samples for each mehod and he sandard errors correspond closely o hose repored n he leraure. The hree EM boosraps sandard errors are roughly of he same order of magnude. We shall compare our nformaon-based sandard errors wh he paramerc boosrap (A1), whch s he mos relevan here gven our focus on mulvarae normal mxures. The sandard errors obaned by he explc score and Hessan formulae are somewha smaller han he boosrap sandard errors, whch confrms he fndng n Basford e al. (1997) concernng I 1 1 (ouer score). In egh of he eleven cases, he sandard errors compued from I 1 2 (Hessan) le nbeween he sandard error based on he score and he sandard error based on he robus esmaor, as predced n Secon 3. When hs happens, he msspecfcaon-robus sandard error (B3) s he smalles of he hree. For boh groups of women he robus sandard error s abou 63% of he sandard error based on paramerc boosrap (A1). The Irs daa se The Irs flower daa were colleced by Anderson (1935) wh he purpose o quanfy he geographc varaon of Irs flowers n he Gaspé Pennsula, 12

14 locaed on he easern p of he provnce of Québec n Canada. The daa se consss of ffy samples from each of hree speces of Irs flowers: Irs seosa (Arcc rs), Irs verscolor (Souhern blue flag), and Irs vrgnca (Norhern blue flag). Four feaures were measured from each flower: sepal lengh, sepal wdh, peal lengh, and peal wdh. Based on he combnaon of he four feaures, Sr Ronald Fsher (1936) developed a lnear dscrmnan model o deermne whch speces hey are. The daa se hus consss of n = 150 measuremens on m = 4 feaures from g = 3 Irs speces. Table 2 conans parameer esmaes and sandard errors of he means µ and varances v (he covarance esmaes v j for j have been omed), where all esmaes and sandard errors (excep π 1 and π 2 ) have agan been mulpled by 100. As before, he EM boosrap resuls are obaned from 100 samples for each mehod and he sandard errors correspond closely o hose repored n he leraure. In conras o he frs example, he sandard errors obaned by I 1 1 (ouer score) are somewha larger han he paramerc boosrap sandard errors, agan n accordance o he fndng n Basford e al. (1997). In 18 of he 26 cases, he sandard errors compued from I 1 2 (Hessan) le n-beween he sandard error based on he score and he sandard error based on he robus esmaor, as predced n Secon 3. And agan, remarkably, when hs happens he msspecfcaon-robus sandard error (B3) s he smalles of he hree. In hs example, conrary o he prevous example, he robus sandard error s only slghly smaller on average han he sandard error based on paramerc boosrap. Our wo examples demonsrae ha he mplemenaon of second-order dervave formulae s a praccal alernave o he currenly used boosrap. Our program for compung he sandard errors of I 1 1 (ouer produc), I 1 2 (Hessan), and I 1 3 (sandwch) s exremely fas. The resulng sandard errors are comparable n sze o he boosrap sandard errors, bu hey are suffcenly dfferen o jusfy he queson whch sandard errors are he mos accurae. Ths queson can no be answered n esmaon exercses. We need a small Mone Carlo expermen where he precson of he esmaes s known. 7 Smulaons We wsh o assess he small sample behavor of he nformaon-based esmaes and compare o he behavor of he radonal boosrap-based mehods. We shall assume ha he daa are generaed by an m-varae normal mxure model, deermned by he parameers (π, µ, V ) for = 1,..., g, so ha we have g 1 +gm(m +3)/2 parameers n oal. I s convenen o 13

15 Table 2: Esmaon resuls Irs daa Varable Esmae Sandard Error EM Boosrap Our mehod (A1) (A2) (A3) (B1) (B2) (B3) Weghs π π Irs seosa µ µ µ µ v v v v Irs verscolor µ µ µ µ v v v v Irs vrgnca µ µ µ µ v v v v

16 consruc marces A such ha A A = V. We hen oban R samples, each of sze n, from hs dsrbuon where each sample s generaed as follows. Draw a sample of sze n from he caegorcal dsrbuon defned by Pr(z = ) = π. Ths gves n neger numbers, say z 1,...,z n, such ha 1 z j g for all j. Defne n as he number of mes ha z j =. Noce ha n = n. For = 1,...,g draw mn sandard-normal random numbers and assemble hese n m 1 vecors ǫ,1,...,ǫ,n. Now defne x,ν = µ + A ǫ,ν N(µ, V ) (ν = 1,...,n ). The se {x,ν } hen consss of n m-dmensonal vecors from he requred mxure. Gven hs sample of sze n we esmae he parameers and sandard errors, assumng ha we know he dsrbuon s a mxure of g normals. We perform R replcaons of hs procedure. For each r = 1,..., R we oban an esmae of each of he parameers. The R esmaes ogeher defne a dsrbuon for each parameer esmae, and f R s suffcenly large he varance of hs dsrbuon s he rue varance of he esmaor. Our queson now s how well he nformaon-based sandard error approxmae hs rue sandard error. We perform four expermens. In each case we ake m = g = 2, π 1 = π 2 = 0.5, and we le n = 100 and n = 500 respecvely. (a) Correc specfcaon. The mxure dsrbuons are boh normal. There s no msspecfcaon, so he model s he same as he daagenerang process. We le µ 1 = ( 0 0), µ 2 = ( 5 5), V 1 = (b) Overspecfcaon. Same as (a), excep ha ( ) 1 0 V 1 = V 2 =. 0 1 ( ) 1 0, V = ( ) However, we do no know ha he varance marces are he same and hence we esmae hem separaely. (c) Consraned esmaon. Same as (b), excep ha we now know ha he varance marces are equal and herefore ake hs consran no accoun, usng Theorem 3 raher han Theorem 1. 15

17 (d) Msspecfcaon n dsrbuon. The wo mxure dsrbuons are no normal. The rue underlyng dsrbuons are F(k 1, k 2 ), bu we are gnoran abou hs and ake hem o be normal. Insead of samplng from a mulvarae F-dsrbuon we draw a sample {ηh } from he unvarae F(k 1, k 2 )-dsrbuon. We hen defne η h = k 1 (k 2 4) 2(k 1 + k 2 2) ( ) k2 2 ηh k 1, 2 so ha he {η h } are ndependen and dencally dsrbued wh mean zero and varance one, bu of course here wll be skewness and kuross. For = 1,...,g draw mn random numbers η h n hs way, assemble hese n m 1 vecors ǫ,1,...,ǫ,n, and oban x,ν as before. We le k 1 = 5 and k 2 = 10, so ha he frs four momens exs bu he ffh and hgher momens do no. Each esmaon mehod provdes an algorhm for obanng esmaes and sandard errors of he parameers θ j, whch we denoe as ˆθ j and s j = var 1/2 (ˆθ j ) respecvely. Based on R replcaons we approxmae he dsrbuons of ˆθ j and s j from whch we can compue momens of neres. Leng ˆθ (r) j and s (r) j error (SE) of ˆθ j as denoe he esmaes n he r-h replcaon, we fnd he sandard SE(ˆθ j ) = 1 R R r=1 (ˆθ (r) j θ j ) 2, θj = 1 R R r=1 ˆθ (r) j. We wsh o know wheher he repored sandard errors are close o he acual sandard errors of he esmaors, and we evaluae hs closeness n erms of he roo mean squared error (RMSE) of he sandard errors of he parameer esmaes. We frs compue from whch we oban S 1j = 1 R R r=1 s (r) j, SE(s j ) = S 2j = 1 R S 2j S 2 1j. R r=1 (s (r) j ) 2, In order o fnd he bas and mean squared error of s j we need o know he rue value of s j. For suffcenly large R, hs value s gven by SE(ˆθ j ). We fnd BIAS(s j ) = S 1j SE(ˆθ j ), RMSE(s j ) = SE 2 (s j ) + BIAS 2 (s j ), 16

18 and hus we oban he RMSE, BIAS, and SE of s j for each j. In our expermens we use R = 50, 000 replcaons for compung he rue sandard errors (10,000 n case (d)) and R = 10, 000 replcaons for compung he esmaed sandard errors (1000 n case (d)). The reason we use less replcaons n case (d) s ha we wan o avod draws wh badly separaed means ha could nduce label swchng. To compue boosrap-based sandard errors, we rely on 100 boosrap samples (Efron and Tbshran 1993). We use he EMMIX Forran code convered o run n R o generae mxure samples, and oban parameer esmaes and boosrap-based sandard errors. We hen mpor he parameer esmaes no MATLAB and use hem o oban he nformaon-based sandard error esmaes. Noce ha n all four cases he means are well separaed. Ths s useful for hree reasons: frs, label swchng problems across smulaons are less lkely o occur; second, he ML esmaes for well-separaed means are accurae enough o allow us o focus on sandard error analyss raher han naccuraces n parameer esmaes; and hrd, we expec he boosrap-based sandard errors o work parcularly well when accurae parameer esmaes are used for boosrap samples. Thus, o brng ou possble advanages of he nformaon-based mehod, we consder cases where he boosrap-based mehods should work parcularly well. Table 3: Smulaon resuls, case (a), n = 500 Varable Value Roo mean square error of SE EM Boosrap Our mehod (A1) (A2) (A3) (B1) (B2) (B3) Wegh π Group 1 µ µ v v v Group 2 µ µ v v v

19 Le us now dscuss he smulaon resuls, where we confne our dscusson o he sandard errors of he ML esmaes, because he ML esmaes hemselves are he same for each mehod. In Table 3 we repor he RMSE of he esmaed sandard errors for n = 500 n he correcly specfed case (a). We see ha mehod (B2) based on I 1 2 (he Hessan) ouperforms he EM paramerc boosrap mehod (A1), whch n urn s slghly beer han mehods (B3) (sandwch) and (B1) (ouer score). The observed nformaon marx I 1 1 based on he ouer produc of he scores ypcally performs wors of he hree nformaon-based esmaes and s herefore no recommended. The poor performance of he ouer score marx confrms resuls n prevous sudes, see for example Basford e al. (1997). In correcly specfed cases we would expec ha he paramerc boosrap and he Hessan-based observed nformaon marx perform well relave o oher mehods, and hs s ndeed he case. Our general concluson for correcly specfed cases s ha mehod (B2) based on I 1 2 performs bes, followed by he paramerc boosrap mehod (A1). In conras o he clam of Day (1969) and McLachlan and Peel (2000, p. 68) ha one needs very large sample szes before he observed nformaon marx gves accurae resuls, we fnd ha very good accuracy can be obaned for n = 500 and even for n = 100. The mean squared error of he sandard error s he sum of he varance and he square of he bas. The conrbuon of he bas s small. In he case repored n Table 3, he rao of he absolue bas o he RMSE s 9% for mehod (B2) when we average over all 11 parameers. The bas s ypcally negave for all mehods. As McLachlan and Peel (2000, p. 67) pon ou, dela mehods such as he supplemened EM mehod or he condonal boosrap ofen underesmae he sandard errors, and he same occurs here. Snce he bas s small n all correcly specfed models, hs s no a serous problem. We noce ha he RMSE of he sandard error of he mxng proporon ˆπ 1 s relavely hgh for mehods (B2) and (B3), boh of whch employ he Hessan marx. The suaon s somewha dfferen here han for he oher parameers, because he sandard error of ˆπ 1 s esmaed very precsely bu wh a relavely large negave bas. Of course, he bas decreases when n ncreases, bu n small samples he sandard error of ˆπ 1 s sysemacally underesmaed. Ths seems o be a general phenomenon when esmang mxng proporons wh nformaon-based mehods, and can possbly be repared hrough a bas-correcon facor. We do no, however, pursue hs problem here. Even wh he relavely large RMSE of he mxng proporon, mehod (B2) performs bes, and hs underlnes he fac ha hs mehod esmaes he sandard errors of he means µ and he varance componens v j very precsely. 18

20 Table 4: Overvew of he four smulaon expermens Expermen Roo mean square error of SE EM Boosrap Our mehod (A1) (A2) (A3) (B1) (B2) (B3) Correcly specfed Overspecfed Consraned Msspecfed, F(5, 10) In Table 4 we provde a general overvew of he RMSE resuls of all four cases consdered, for n = 100 and n = 500. In cases (b) and (c) we llusrae he specal case where V 1 = V 2. In case (b) we are gnoran of hs fac and hence he model s overspecfed bu no msspecfed. In case (c) we ake he consran no accoun and hs leads o more precson of he sandard errors. The RMSE s reduced by abou 50% when n = 100 and by abou 35% when n = 500. Agan, he Hessan-based esmae I 1 2 s he mos accurae of he sx varance marx esmaes consdered. In case (d) we consder msspecfed models where boh skewness and kuross are presen n he underlyng dsrbuons, bu gnored n he esmaon. One would expec ha he nonparamerc boosrap esmaes (A2) and (A3) and our proposed sandwch esmae (B3) would perform well n msspecfed models, and hs s usually, bu no always, he case. Our sandwch esmae I 1 3 has he lowes RMSE n all cases. The ouer score esmae (B1) fals o produce credble oucomes when n = 100. If we repea he expermen based on oher F-dsrbuons we oban smlar resuls. Fnally we consder he nformaon marx es presened n Secon 4. The IM es has lmaons n pracce because he asympoc χ 2 -dsrbuon s ypcally a poor approxmaon o he fne sample dsrbuon of he es sasc. We brefly nvesgae he fne sample properes of our verson of he IM es va smulaons o gve some dea of jus how useful can be. Le us consder he correcly specfed model (a) wh m = g = 2 so ha he IM es of Theorem 2 should be asympocally χ 2 -dsrbued wh 19

21 gm(m + 3)/2 = 10 degrees of freedom. In Table 5 we compue he szes for Table 5: Sze of IM es, smulaon resuls Crcal values n n = 100, 500, and 1000, based on 10,000 replcaons and usng he crcal values ha are vald n he asympoc dsrbuon. As expeced, he resuls are no encouragng, hus confrmng fndngs by many auhors; see Davdson and MacKnnon (2004, Sec. 16.9). There s, however, a vable alernave based on he same IM sasc, proposed by Horowz (1994) (see also Davdson and MacKnnon 2004, pp ), namely o boosrap he crcal values of he IM es for each parcular applcaon. Ths s wha we recommend. 8 Conclusons Despe McLachlan and Krshnan s (1997, p. 111) clam ha analycal dervaon of he Hessan marx of he loglkelhood for mulvarae mxures seems o be dffcul or a leas edous, we show ha pays o have hese formulae avalable for normal mxures. In correcly specfed models he mehod based on he observed Hessan-based nformaon marx I 1 2 s he bes n erms of RMSE. In msspecfed models he mehod based on he sandwch marx I 1 3 s he bes, even f he sandard errors of he observed nformaon marx based on he ouer produc of he scores are large, as s somemes he case. In general, he bas of he wo mehods s eher he smalles n her caegory (correcly specfed or msspecfed) or f no, becomes he smalles as he sample sze ncreases o n = 500. Our MATLAB code for compung he sandard errors runs n vrually no me unless boh m and g are very large, and s even faser han he boosrap. There are a leas wo addonal advanages n usng nformaon-based mehods. Frs, he Hessan we compued can be useful o deec nsances where he EM algorhm has no converged o he ML soluon. Second, f he sample sze s no oo large relave o he number of parameers o esmae, he mehods based on I 1 2 and I 1 3 can be readly used o compue asympocally vald confdence nervals, whle nonparamerc boosrap confdence 20

22 nervals are ofen dffcul o compue. Appendx: Proofs Proof of Theorem 1. Le φ and α be defned as n (5). Then, snce f(x ) = φ, we oban d log f(x ) = df(x ) g f(x ) = dφ j φ j =1 = g α d log φ (10) =1 and ( ( ) ) d 2 d 2 2 ( ) f(x ) df(x ) log f(x ) = = d2 φ f(x ) f(x ) j φ dφ 2 j j φ j ( g ) g = α (d 2 log φ + (d logφ ) 2 ) ( α d log φ ) 2. (11) =1 To evaluae hese expressons, we need he frs- and second-order dervaves of log φ. Snce, usng (2), we fnd log f (x) = m 2 log(2π) 1 2 log V 1 2 (x µ ) V 1 (x µ ), d logf (x) = 1 2 d log V + (x µ ) V 1 dµ 1 2 (x µ ) d(v 1 )(x µ ) and = 1 2 r(v 1 dv ) + (x µ ) V 1 =1 dµ (x µ ) V 1 d 2 log f (x) = 1 2 r ( ) (dv 1 )dv (dµ ) V 1 (dµ ) + (x µ ) (dv 1 (x µ ) V 1 = 1 2 rv 1 2(x µ ) V 1 (x µ ) V 1 )dµ (x µ ) V 1 (dv )V 1 (dv )V 1 (x µ ) (dv )V 1 dµ (dv )V 1 (x µ ) (dv )V 1 dv (dµ ) V 1 (dµ ) (dv )V 1 dµ (dv )V 1 (dv )V 1 (x µ ), 21

23 and hence, usng (3) and he defnons (6) (8), d log φ = d log π + (x µ ) V 1 dµ (x µ ) V 1 (dv )V 1 (x µ ) = a dπ + b dµ 1 2 r (B dv ) 1 rv dv and d 2 log φ = d 2 log π (dµ ) V 1 (x µ ) V 1 = a dπ + b dµ 1 2 (vec B ) D d vech V = a dπ + c dθ (12) (dµ ) 2(x µ ) V 1 (dv )V 1 (dv )V 1 (x µ ) (dv )V 1 (dµ ) r V (dv )V 1 (dv ) 2 = (dπ) a a (dπ) (dµ ) V 1 (dµ ) 2b (dv )V 1 (dµ ) 1 1 r(v 2B )(dv )V 1 (dv ) 2 = (dπ) a a (dπ) (dµ ) V 1 (dµ ) 2(d vec V ) (b V 1 )(dµ ) 1 2 (d vec V ) ((V 1 2B ) V 1 )(d vec V ) = (dπ) a a (dπ) (dµ ) V 1 (dµ ) 2(d vech V ) D (b V 1 )(dµ ) = 1 2 (d vech V ) D ((V 1 2B ) V 1 )D(d vech V ) ( ) ( ) ( ) dπ a a 0 dπ. (13) dθ 0 C dθ Inserng (12) n (10), and (12) and (13) n (11) complees he proof. Proof of Theorem 2. Ths follows from he expresson of W (θ) and he developmen n Lancaser (1984). Proof of Theorem 3. From (12) we see ha d log φ = a dπ + c dθ = a dπ + b dµ 1 2 (vec B ) D dv, 22

24 and from (13) ha d 2 log φ = (dπ) a a (dπ) (dθ ) C (dθ ) = (dπ) a a (dπ) (dµ ) V 1 (dµ ) 2(dµ ) (b V 1 )D(dv) 1 2 (dv) D ((2b b V 1 ) V 1 )D(dv). The resuls hen follow afer some edous bu sraghforward algebra from (10) and (11). References Aken, M., and Rubn, D. B. (1985), Esmaon and Hypohess Tesng n Fne Mxure Models, Journal of he Royal Sascal Socey, Ser. B, 47, Al, M. M., and Nadarajah, S. (2007), Informaon Marces for Normal and Laplace Mxures, Informaon Scences, 177, Anderson, E. (1935), The Irses of he Gaspé Pennsula, Bullen of he Amercan Irs Socey, 59, 2-5. Basford, K. E., Greenway, D. R., McLachlan, G. J., and Peel, D. (1997), Sandard Errors of Fed Means Under Normal Mxure Models, Compuaonal Sascs, 12, Behboodan, J. (1972), Informaon Marx for a Mxure of Two Normal Dsrbuons, Journal of Sascal Compuaon and Smulaon, 1, Chesher, A. D. (1983), The Informaon Marx Tes: Smplfed Calculaon va a Score Tes Inerpreaon, Economcs Leers, 13, Davdson, R., and MacKnnon, J. G. (2004), Economerc Theory and Mehods, New York: Oxford Unversy Press. Day, N. E. (1969), Esmang he Componens of a Mxure of Normal Dsrbuons, Bomerka, 56, Dempser, A. P., Lard, N. M., and Rubn, D. B. (1977), Maxmum Lkelhood from Incomplee Daa va he EM Algorhm (wh dscusson), Journal of he Royal Sascal Socey, Ser. B, 39,

25 Dez, E., and Böhnng, D. (1996), Sascal Inference Based on a General Model of Unobserved Heerogeney, n Advances n GLIM and Sascal Modelng, eds. L. Fahrmer, F. Francs, R. Glchrs, and G. Tuz, Lecure Noes n Sascs, Berln: Sprnger, pp Efron, B. (1979), Boosrap Mehods: Anoher Look a he Jackknfe, The Annals of Sascs, 7, Efron, B., and Tbshran, R. (1993), An Inroducon o he Boosrap, London: Chapman & Hall. Fsher, R. A. (1936), The Use of Mulple Measuremens n Taxonomc Problems, Annals of Eugencs, 7, Habbema, J. D. F., Hermans, J., and van den Broek, K. (1974), A Sep- Wse Dscrmnan Analyss Program Usng Densy Esmaon, n Proceedngs n Compuaonal Sascs, Compsa 1974, Wen: Physca Verlag, pp Hahaway, R. J. (1985), A Consraned Formulaon of Maxmum-Lkelhood Esmaon for Normal Mxure Dsrbuons, The Annals of Sascs, 13, Horowz, J. L. (1994), Boosrap-Based Crcal Values for he Informaon Marx Tes, Journal of Economercs, 61, Huber, P. J. (1967), The Behavor of Maxmum Lkelhood Esmaes under Non-Sandard Condons, n Proceedngs of he Ffh Berkeley Symposum on Mahemacal Sascs and Probably, Vol. 1, eds. L. M. LeCam and J. Neyman, Berkeley: Unversy of Calforna Press, pp Lancaser, A. (1984), The Covarance Marx of he Informaon Marx Tes, Economerca, 52, Lu, C. (1998), Informaon Marx Compuaon from Condonal Informaon va Normal Approxmaon, Bomerka, 85, Lous, T. A. (1982), Fndng he Observed Informaon Marx When Usng he EM Algorhm, Journal of he Royal Sascal Socey, Ser. B, 44, Magnus, J. R. (1988), Lnear Srucures, Grffn s Sascal Monographs and Courses, No. 42, London: Edward Arnold and New York: Oxford Unversy Press. 24

26 Magnus, J. R., and Neudecker, H. (1988), Marx Dfferenal Calculus wh Applcaons n Sascs and Economercs, Chcheser/New York: John Wley, Second edon, McLachlan, G. J., and Basford, K.E. (1988), Mxure Models: Inference and Applcaons o Cluserng, New York: Marcel Dekker. McLachlan, G. J., and Krshnan, T. (1997), The EM Algorhm and Exensons, New York: John Wley. McLachlan, G. J., and Peel, D. (2000), Fne Mxure Models, New York: John Wley. McLachlan, G. J., Peel, D., Basford, K. E., and Adams, P. (1999), Fng of Mxures of Normal and -Componens, Journal of Sascal Sofware, 4, Issue 2, gjm/emmx/emmx.hml. Newon, M. A., and Rafery, A. E. (1994), Approxmae Bayesan Inference wh he Weghed Lkelhood Boosrap (wh dscusson), Journal of he Royal Sascal Socey, Ser. B, 56, Newcomb, S. (1886), A Generalzed Theory of he Combnaon of Observaons so as o Oban he Bes Resul, Amercan Journal of Mahemacs, 8, Pearson, K. (1894), Conrbuon o he Mahemacal Theory of Evoluon, Phlosophcal Transacons of he Royal Socey, Ser. A, 185, Sgler, S. M. (1986), The Hsory of Sascs: The Measuremen of Uncerany Before 1900, Cambrdge, MA: Belknap. Terngon, D. M., Smh, A. F. M., and Makov, U. E. (1985), Sascal Analyss of Fne Mxure Dsrbuons, New York: John Wley. Whe, H. (1982), Maxmum Lkelhood Esmaon of Msspecfed Models, Economerca, 50, Xu, L., and Jordan, M. I. (1996), On Convergence Properes of he EM Algorhm for Gaussan Mxures, Neural Compuaon, 8,