Singularity structures and impacts on parameter estimation in finite mixtures of distributions

Transcription

1 Snguarty structures and mpacts on parameter estmaton n fnte mxtures of dstrbutons Nhat Ho and XuanLong Nguyen Department of Statstcs Unversty of Mchgan Abstract Snguartes of a statstca mode are the eements of the mode s parameter space whch make the correspondng Fsher nformaton matrx degenerate. These are the ponts for whch estmaton technques such as the maxmum kehood estmator and standard Bayesan procedures do not admt the root-n parametrc rate of convergence. We propose a genera framework for the dentfcaton of snguarty structures of the parameter space of fnte mxtures, and study the mpacts of the snguarty eves on mnmax ower bounds and rates of convergence for the maxmum kehood estmator over a compact parameter space. Our study makes expct the deep nks between mode snguartes, parameter estmaton rates and mnmax bounds, and the agebrac geometry of the parameter space for mxtures of contnuous dstrbutons. The theory s apped to estabsh concrete convergence rates of parameter estmaton for fnte mxture of skewnorma dstrbutons. Ths very rch and ncreasngy popuar mxture mode s shown to exhbt a remarkaby compex range of asymptotc behavors that have not been htherto reported n terature. Introducton In the standard asymptotc theory of parametrc estmaton, a customary reguarty assumpton s the non-snguarty of the Fsher nformaton matrx defned by the statstca mode (see, for exampe, Lehmann and Casea [998] (pg. 24; or van der Vaart [998], Sec Ths condton eads to the chershed root-n consstency, and n many cases asymptotc normaty of the parameter estmates. When the non-snguarty condton fas to hod, that s, when the true parameters represent a snguar pont n the statstca mode, very tte s known about the asymptotc behavor of the parameter estmates. In the past, the snguarty stuaton mght have been brushed asde as dosyncratc by the practtoners of mutvarate parametrc statstcs. But as compex and hgh-dmensona modes are ncreasngy embraced by statstcans and practtoners ake, snguartes are no onger a rarty. For exampe, the many zeros present n a hgh-dmensona near regresson probem represent a type of snguartes of the underyng mode, ponts whch resut n rank-defcent Fsher nformaton matrces [Haste et a., 25]. In another exampe, the zero skewness n the famy of skewed dstrbutons represents a snguar pont [Chogna, 25]. In both exampes, snguarty ponts are qute easy to spot out t s the mpacts of ther presence on mproved parameter estmaton procedures and the asymptotc propertes such procedures enta that are nontrva matters occupyng the best efforts of many researchers n the past decade. The textbooks by Bühmann and van de Geer [2], Haste et a. [25], for nstance, address such ssues for hgh-dmensona regresson probems, whe the recent Ths research s supported n part by grants NSF CAREER DMS and NSF CNS-4933 to XN. The authors woud ke to acknowedge Larry Wasserman and Ya acov Rtov for vauabe dscussons reated to ths work. AMS 2 subect cassfcaton: Prmary 62F5, 62G5; secondary 62G2. Keywords and phrases: Fsher snguartes, system of poynoma equatons, semagebrac set, mxture mode, nonnear parta dfferenta equaton, mnmax ower bound, maxmum kehood estmaton, convergence rates, Wassersten dstances.

2 papers by Ley and Pandavene [2], Han and Ley [22, 24] nvestgate statstca nference n the skewed fames. By contrast, wth fnte mxture modes a popuar and very rch cass of modeng toos for densty estmaton and heterogenety nference [Lndsay, 995] and a subect of ths work, the snguarty phenomenon s not qute understood, to the best our knowedge, except for some specfc nstances. One of the smpest nstances s the snguarty of Fsher nformaton matrx n an (overftted fnte mxture that ncudes a homogeneous dstrbuton. Lee and Chesher [986] anayzed a test of heterogenety based on fnte mxtures, addressng the chaenges arsng from the aforementoned snguarty. Recent works on the same topc ncude Chen and Chen [23], Kasahara and Shmotsu [24]. Rotntzky et a. [2] nvestgated the kehood-based parameter estmaton n a somewhat genera parametrc modeng framework, subect to the constrant that the Fsher nformaton matrx s one rank defcent. For overftted fnte mxtures, Chen [995] showed that under a condton of strong dentfabty, there are estmators whch acheve the generc convergence rate n /4 for parameter estmaton. Recent papers estabshed a smar knd of generc behavor of estmaton under somewhat broader settngs of overftted fnte mxture modes wth both maxmum kehood estmaton and Bayesan estmaton [Rousseau and Mengersen, 2, Nguyen, 23, Ho and Nguyen, 26b]. Strong dentfabty, however, remans far too strong a condton to adequatey capture the spectrum of behavor for mxture modes. For exampe, t was shown recenty that popuar modes such as the ocaton-scae Gaussan mxtures, and the shape-rate Gamma mxtures, do not admt such a generc rate of convergence for an estmaton method such as MLE [Ho and Nguyen, 26a]. For nstance, snguartes arse n fnte mxtures of Gamma dstrbutons, even when the number of mxng components s known ths phenomenon resuts n an extremey sow convergence behavor for the mode parameters yng n the vcnty of snguar ponts, eventhough such parameters are (perfecty dentfabe. Fnte mxtures of Gaussan dstrbutons, though dentfabe, exhbt both mnmax owerbounds and maxmum kehood estmaton rates that are drecty nked to the sovabty of a system of rea poynoma equatons, rates whch deterorate qucky wth the ncreasng number of extra mxng components. The resuts obtaned for such specfc nstances contan consderabe nsghts about parameter estmaton n fnte mxture modes, but they ony touch upon the surface of a more genera phenomenon. Indeed, as we sha see there s a much rcher spectrum of asymptotc behavor n whch reguar (non-snguar mxtures, strongy dentfabe mxtures, and weaky dentfabe mxture modes (such as the one studed by Ho and Nguyen [26a] occupy but a sma spot. Obectves and man resuts In ths paper we propose a theoretca framework for anayzng parameter estmaton behavor n fnte mxture modes, addressng drecty the stuatons where the nonsnguarty condton of Fsher nformaton matrx may not hod. Our approach s to take on a drect and systematc nvestgaton of the snguarty structure of a compact and mut-dmensona parameter space of mxture modes, and then study the mpacts of the presence of snguartes on parameter estmaton. It s no onger suffcent to speak of the standard noton of Fsher nformaton snguartes. A more fundamenta noton that we ntroduce s caed snguarty eve, a natura or nfnte vaue gven to every eement n the parameter space. Fsher nformaton snguartes smpy correspond to ponts n the parameter space whose snguarty eve s non-zero. Wthn the set of Fsher nformaton snguartes the parameter space can be parttoned nto dsont subsets determned by dfferent snguarty eves. The snguarty eve descrbes n a precse manner the varaton of the mxture kehood wth respect to changes n the parameters. Ths concept enabes us to rgorousy quantfy the varyng degrees of dentfabty and snguarty, some of whch were mpcty expoted n prevous works mentoned above. The statstca mpcaton of the snguarty eve s easy to descrbe: gven an..d. n-sampe 2

3 from a (true mxture mode, a parameter vaue of snguarty eve r admts n /2(r+ mnmax ower bound for any estmator tendng to the true parameter(s, as we as the same maxmum kehood estmator s convergence rate (up to a ogarthmc factor and under some condtons. Thus, snguarty eve resuts n root-n convergence rate for parameter estmaton. Fsher snguarty corresponds to snguarty eve or greater than, resutng n convergence rates n /4, n /6, n /8 or so on. The detaed pcture of the dstrbuton of snguarty eves, however, can be extremey compex to capture. Remarkaby, there are fnte mxtures for whch the compact parameter space can be parttoned nto dsont subsets whose snguarty eve ranges from to to 2,..., up to nfnty. As a resut, f we were to vary the true parameter vaues, we woud encounter a phenomenon akn to that of phase transton on the statstca effcency of parameter estmaton wthn the same mode cass. Man technques The maor component of our framework s a procedure for characterzng subsets of ponts carryng the same snguarty eve. It w be shown that these ponts are n fact a subset of a rea affne varety. A rea affne varety s a set of soutons to a system of rea poynoma equatons. The poynoma equatons can be derved expcty by the kerne densty functons that defne a gven mxture mode. The study of the soutons of poynoma equatons s a centra subect of agebrac geometry [Stumfe, 22, Cox et a., 27]. The contacts made between statstca modes and agebrac geometry have been studed for dscrete Markov random feds [Drton et a., 29], as we as fnte mxtures of categorca data [Aman et a., 29]. On the other hand, for fnte mxtures of contnuous dstrbutons, the nk to agebrac geometry s dsted from a deeper source of agebrac structure: t s traced to the parta dfferenta equatons satsfed by the mxture mode s kerne denstes. For Gaussan mxtures, t s the reaton captured by Eq. (3 for the Gaussan kerne. The parta dfferenta equatons can be nonnear, wth coeffcents gven by ratona functons defned n terms of mode parameters. It s ths reaton that s prmary responsbe for the compexty of mxture mode s snguarty structure. A quntessenta exampe of such a reaton s gven by Eq. (2 for the skewnorma kerne denstes. Athough our method for the anayss of snguarty structure and the asymptotc theory for parameter estmaton can be used to re-derve od and recent resuts such as those of Chen [995], Ho and Nguyen [26a], the more sgnfcant outcome s to estabsh new resuts on mxture modes for whch no asymptotc theory have htherto been acheved. Ths eads us to a story of fnte mxtures of skewnorma dstrbutons. Skewnorma dstrbuton was orgnay proposed n Azzan [986], Azzan and Vae [996], Azzan and Captano [999]. Skewnorma dstrbuton generazes the norma dstrbuton, whch s enhanced by the capabty of handng asymmetrc data dstrbutons. Due to ts more reastc modeng capabty for mut-modaty and asymmetrc components, skewnorma mxtures are ncreasngy embraced n recent years for mode based nference of heterogenety by many researchers [Ln et a., 27, Areano-Vae et a., 28, 29, Ln, 29, Schnatter and Pyne, 29, Ghosa and Roy, 2, Lee and McLachan, 23, Prates et a., 23, Canae and Scarpa, 25, Zeer et a., 25]. Due to ts popuarty, a thorough understandng of the asymptotc behavor of parameter estmaton for skewnorma mxtures s aso of nterest n ts own rght. The snguarty structure of the skewnorma mxtures s perhaps one of the most compex among a parametrc modes that we have encountered n asymptotc statstcs. By comparson, strongy dentfabe modes admt the same snguarty eve (, to be precse for a parameter vaues resdng n a compact space, resutng n n /4 convergence rate for the MLE. Locaton-scae Gaussan mxtures are a step up n the compexty, n that a the parameter vaues carry the same snguarty eve, whch depends ony on the number of extra mxng components. But ths s not the pcture of skewnorma mxtures. We w be abe to dentfy subsets wth snguarty eve,, 2,... a the way up to nfnty. Even n the settng of mxtures wth known number of mxng components, the snguarty structure s 3

4 remarkaby compex. Thus, the resuts for skewnorma mxtures present an usefu ustraton for the fu power of the genera theory for fnte mxtures of contnuous dstrbutons. The source of compexty of skewnorma mxtures s the structure of the skewnorma kerne densty. The evdence for the atter was made cear by Chogna [25], Ley and Pandavene [2], Han and Ley [22, 24], whose works provded a very detaed pcture of the snguartes for the cass of skewnorma denstes, and ther mpacts on the non-standard rates of convergence of MLE. Not ony can we recover the resuts of Han and Ley [22, 24], whch correspond to a trva mxture that has exacty one skewnorma component, fundamentay new resuts are estabshed for mxtures of two or more components. It s n ths settng that new types of snguartes arse out of the nteractons between dstnct skewnorma components. These nteractons defne subset of snguar ponts of a gven eve that can be characterzed by a system of rea poynoma equatons. Ths characteraton aows us to estabsh ether the precse snguarty eve or an upper bound for the mxture mode s entre parameter space. The pan for the remander of our paper s as foows. Secton 2 ays out the notaton and reevant concepts such as parameter spaces and the underyng geometres. Secton 3 presents the genera framework of anayss of snguarty structure, and the mpact on convergence rates of parameter estmaton for snguar ponts of a gven snguarty eve. Secton 4, Secton 5, and Secton 6 ustrate the theory on the fnte mxture of skewnorma dstrbutons, by gvng concrete mnmax bounds and MLE convergence rates for ths cass of modes for the frst tme. We concude wth a dscusson n Secton 7. Further detas of the proofs and some addtona resuts are gven n the Appendx. 2 Background A fnte mxture of contnuous dstrbutons admts densty of the form p G (x = f(x ηdg(η wth respect to Lebesgue measure on an Eucdean space for x, where f(x η denotes a probabty densty kerne, η s a mut-dmensona parameter takng vaues n a subset of an Eucdean space Θ, G denotes a dscrete mxng dstrbuton on Θ. The number of support ponts of G represents the number of mxng components n mxture mode. Suppose that G = k p δ η, then p G (x = k p f(x η. 2. Parameter spaces and geometres There are dfferent knds of parameter space and geometres that they carry whch are reevant to our work. We proceed to descrbe them n the foowng. Natura parameter space The customary defned parameter space of the k-mxture of dstrbutons s that of the mxng component parameters η, and mxng probabtes p. Throughout ths paper, t s assumed that η Θ, whch s a compact subset of R d for some d, for =,..., k. The mxng probabty vector p = (p,..., p k k, the (k -probabty smpex. To smpfy the theory we w further assume (n Secton 4 that a p c for some sma postve constant c. For the remander of the paper, we aso use Ω to denote the compact subset of the Eucdean space for parameters (p, η. Exampe 2.. The skewnorma densty kerne on the rea ne has three parameters η = (θ, σ, m R R + R, namey, the ocaton, scae and skewness (shape parameters. It s gven by, for x R, f(x θ, σ, m := 2 ( x θ σ f Φ(m(x θ/σ, σ 4

5 where f(x s the standard norma densty and Φ(x = f(t(t x dt. Ths generazes the Gaussan densty kerne, whch corresponds to fxng m =. The parameter space for the k-mxture of skewnormas s therefore a subset of an Eucdean space for the mxng probabtes p and mxng component parameters η = (θ, v = σ 2, m R 3. For each =,..., k, θ, σ, m are restrcted to resde n compact subsets Θ R, Θ 2 R +, Θ 3 R respectvey,.e., Θ = Θ Θ 2 Θ 3. Semagebrac sets The snguarty structure of the parameter space carres a dfferent geometry. It w be descrbed n terms of the zero sets (sets of soutons of systems of rea poynoma equatons. The zero set of a system of rea poynoma equatons s caed a (rea affne varety [Cox et a., 27]. In fact, the sets whch descrbe the snguarty structure of fnte mxtures are not affne varetes per se. We w see that they are the ntersecton between rea affne varetes the rea-vaued soutons of a fnte coecton of equatons of the form P (p, η =, and the set of parameter vaues satsfyng Q(p, η >, for some rea poynomas P and Q. The ntersecton of these sets s aso referred to as semagebrac sets. Exampe 2.2. Contnung on the exampe of skewnorma mxtures, we w see that frst two types of snguartes that arse from the mxture of skewnormas are soutons of the foowng poynoma equatons ( Type A: P (η = k m. ( Type B: P 2 (η = k { ] 2 } (θ θ 2 + [σ 2 ( + m2 σ2 ( + m2. These are ust two among many more poynomas and types of snguartes that we w be abe to enumerate n the seque. We qucky note that Type A refers to the one (and ony knd of snguarty ntrnsc to the skewnorma kerne: P = f ether one of the m = one of the skewnorma components s actuay norma (symmetrc. Ths type of snguarty has receved n-depth treatments by a number of authors [Chogna, 25, Ley and Pandavene, 2, Han and Ley, 22, 24]. One the other hand, Type B refers to somethng ntrnsc to a mxture mode, as t descrbes the reaton of parameters of dstnct mxng components and. Space of mxng measures and transportaton dstance As descrbed n the Introducton, a goa of ths work s to turn the knowedge about the nature of snguartes of parameter space Ω nto the statstca effcency of parameter estmaton procedures. For ths purpose, the convergence of parameters n a mxture mode s most naturay anayzed n terms of the convergence n the space of mxng measures endowed by transportaton dstance (Wassersten dstance metrcs [Nguyen, 23]. Ths s because the roe payed by parameters p, η n the mxture mode s va mxng measure G. It s mxng measure G that determnes the mxture densty p G accordng to whch the data are drawn from. Snce the map (p, η G(p, η = G = p δ η s many-to-one, we sha treat a par of parameter vectors (p, η = (p,..., p k ; η,..., η k and (p, η = (p,..., p k ; η,..., η k to be equvaent f the correspondng mxng measures are equa, G(p, η = G(p, η. For r, the Wassersten dstance of order r between G(p, η and G(p, η takes the form (cf. Van [23], ( W r (G(p, η, G(p, η = nf, q η η r r /r, 5

6 where r s the r norm endowed by the natura parameter space, the nfmum s taken over a coupngs q between p and p,.e., q = (q [, ] k k such that k q = p and k q = p for any =,..., k and =,..., k. (For the exampe of skewnorma mxtures, f η = (θ, v, m and η = (θ, v, m, then η η r r := θ θ r + v v r + m m r. Suppose that a sequence of probabty measures G n = pn δ η n tendng to G under W r metrc at a rate ω n = o(. If a G n have the same number of atoms k n = k as that of G, then the set of atoms of G n converge to the k atoms of G, up to a permutaton of the atoms, at the same rate ω n under. If G n have the varyng k n [k, k] number of atoms, where k s a fxed upper bound, then a subsequence of G n can be constructed so that each atom of G s a mt pont of a certan subset of atoms of G n the convergence to each such mt aso happens at rate ω n. Some atoms of G n may have mt ponts that are not among G s atoms the tota mass assocated wth those redundant atoms of G n must vansh at the generay faster rate ωn. r 2.2 Estmaton settngs The mpact of snguartes on parameter estmaton behavor s dependent on whether the mxture mode s ftted wth a known number of mxng components, or f ony an upper bound on the number of mxng components s known. The former mode fttng settng s caed e-mxtures for short, whe the atter o-mxtures ( e for exact-ftted and o for over-ftted. Specfcay, gven an..d. n-sampe X, X 2,..., X n accordng to the mxture densty p G (x = f(x ηg (dη, where G = G(p, η = k p δ η s unknown mxng measure wth exacty k dstnct support ponts. We are nterested n fttng a mxture of k mxng components, where k k, usng the n-sampe X,..., X n. In the e-mxture settng, k = k s known, so an estmate G n (such as the maxmum kehood estmate s drawn from ambent space E k, the set of probabty measures G = G(p, η wth exacty k support ponts, where (p, η Ω. In the o-mxture settng, Ĝn s drawn from ambent space O k, the set of probabty measures G = G(p, η wth at most k support ponts, where (p, η Ω. Assumpton Throughout ths paper, severa condtons on the kerne densty f(x η are assumed to hod. Frsty, the coecton of kerne denstes f as η vares s neary ndependent. It foows that the mxture mode s dentfabe,.e., p G (x = p G (x for amost a x entas G = G. Secondy, we say f(x η satsfes a unform Lpschtz condton of order r, for some r, f f as a functon of η s dfferentabe up to order r, and that the parta dervatves wth respect to η, namey κ f/ η κ, for any κ = (κ,..., κ d N d such that κ := κ κ d = r satsfy the foowng: for any γ R d, ( κ f η κ (x η κ f η κ (x η 2 γ κ C η η 2 δ r γ r r κ =r for some postve constants δ and C ndependent of x and η, η 2 Θ. It s smpe to verfy that most kerne denstes used n mxture modeng, ncudng the skewnorma kerne, satsfy the unform Lpschtz property over compact doman Θ, for any fnte r. Notaton We utze severa famar notons of dstance for mxture denstes, wth respect to Lebesgue measure µ. They are tota varaton dstance V (p G, p G = p G (x p G (x dµ(x and Henger 2 dstance h 2 (p G, p G = ( pg (x 2 2dµ(x. p (x G 6

7 3 Snguarty structure n fnte mxture modes 3. Beyond Fsher nformaton Gven a mxture mode { k p G (x G = G(p, η = } p δ η, (p, η Ω from some gven fnte k and f a gven kerne densty (e.g., skewnorma, et G denote the score vector, that s, the coumn vector made of the parta dervatves of the og-kehood functon og p G (x wth respect to each of the mode parameters represented by (p, η. The Fsher nformaton matrx s then gven by I(G = E( G G, where the expectaton s taken wth respect to p G. We say that the parameter vector (p, η (and the correspondng mxng measure G = G(p, η s a snguar pont n the parameter space (resp., ambent space of mxng measures, f I(G s degenerate. Otherwse, (p, η (resp., G s a non-snguar pont. Accordng to the standard asymptotc theory, f the true mxng measure G s non-snguar, and the number of mxng components k = k (that s, we are n the e-mxture settng, then basc estmators such as the MLE or Bayesan estmator yed the optma root-n rate of convergence. Athough the standard theory remans sent when I(G s degenerate, t s cear that the root-n rate may no onger hod. Moreover, there may be a rcher range of behavors for parameter estmaton, requrng us to ook nto the deep structure of the zeros of I(G. Ths w be our story for both settngs of e-mxtures and o-mxtures. In fact, the Fsher nformaton matrx I(G s no onger suffcent n assessng parameter estmaton behavors. Exampe 3.. To ustrate n the context of skewnorma mxtures, where parameter η = (θ, v, m, observe that the mxture densty structure aows the foowng characterzaton: I(G s degenerate f and ony f the coecton of parta dervatves { pg (x, p } { G(x pg (x :=, p G(x, p G(x, p } G(x p η p θ v m =,..., k as functons of x are not neary ndependent. Ths s equvaent to havng that for some coeffcents (α, =,..., 4 and =,..., k, not a of whch are zeros, there hods k α f(x η + α 2 f θ (x η + α 3 f v (x η + α 4 f m (x η =, ( for amost a x R. Lemma 4. ater shows that the (Fsher nformaton matrx s snguar ponts are the zeros of some poynoma equatons. We have seen that for the e-mxtures G s non-snguar f the coecton of densty kerne functons f(x η and ther frst parta dervatves wth respect to each mode parameter are neary ndependent. Ths condton s aso known as the frst-order dentfabty. For o-mxtures, the reevant noton s the second-order dentfabty [Chen, 995, Nguyen, 23, Ho and Nguyen, 26b]: the condton that the coecton of kerne densty functons f(x η, ther frst and second parta dervatves, are neary ndependent. Ths condton fas to hod for skewnorma kerne denstes, whose frst and second parta dervatves are nked by the foowng nonnear parta dfferenta equatons: 2 f θ 2 2 f v + m3 + m f v m =. 2m f (2 m + (m2 + 2 f m 2 + 2vm 2 f v m =. 7

8 The proof of these denttes can be found n Lemma 8. n Appendx B. Note that f m =, the skewnorma kerne becomes norma kerne, whch admts a (smper near reatonshp: 2 f θ 2 = 2 f v. (3 Ths reaton pays a fundamenta roe n the anayss of fnte mxtures of ocaton-scae norma dstrbutons [Ho and Nguyen, 26a]. Compared to Gaussan densty kerne, the nonnear reatonshp exhbted by skewnorma densty kerne resuts n a much rcher behavor. Anayzng ths requres the deveopment of a more genera theory that we now embark on. 3.2 Behavor of kehood n a Wassersten neghborhood Instead of dweng on the Fsher nformaton matrx, we sha empoy a drect approach whch studes the behavor of the kehood functon p G (x as G vares n a Wassersten neghborhood of a mxng measure G = k p δ η. Fx r, and consder a sequence of G n O k, such that W r (G, G n. Let k n k be the number of dstnct support ponts of G n. Then each supportng atom η as {,..., k } of G w have at east one atom of G n that converges to. By reabeng the support ponts of G n, we can express t as G n = k s n p n δ η n, (4 where η n η for a =,..., k, =,..., s n. Addtonay, k s n = k n. There exsts a subsequence of G n accordng to whch k n and a s n are constant n n. (Note that for the settng of e-mxtures, the sequence of eements G n s restrcted to E k, so k n = k for a n. It foows that s n = for a =,..., k. For o-mxtures, to smpfy the presentaton, we have omtted the cases where some G n may have atoms that do not converge to the atoms of G. Thus, from here on we repace the sequence of G n by ths subsequence. To smpfy the notaton, n w be dropped from the superscrpt when the context s cear. In addton, we use the notaton η := η η for =,..., k, =,..., s. Aso, p. := s p, and p. := p. p, for =,..., k. (For e-mxtures, snce s = for a, the notaton s smpfed further: et η := η = η η, p = p. = p p for a =,..., k. The foowng emma reates Wassersten dstance metrc to a sempoynoma of degree r (a sempoynoma s a poynoma of the absoute vaue of some varabes. Lemma 3.. Fx r. For any eement G represented by Eq. (4, defne k s k D r (G, G := p η r r + p.. We have that W r r (G, G D r (G, G, as W r (G, G. To nvestgate the behavor of kehood functon p G (x as G tends to G n Wassersten dstance W r, by representaton (4, we can express k s k p G (x p G (x = p (f(x η f(x η + p. f(x η. (5 8

9 By Tayor expanson up to order r, we obtan k s p G (x p G (x = p r κ = ( η κ κ! κ f η κ (x η + k p. f(x η + R r (x, (6 where R r (x s the Tayor remander. Moreover, t can be verfed that sup x R r (x/wr r (G, G snce f s unform Lpschtz up to order r. We arrve at the foowng formuae, whch measures the speed of change of the kehood functon as G vares n the Wassersten neghborhood of G : p G (x p G (x W r r (G, G = r k s κ = ( p ( η κ /κ! W r r (G, G κ f η κ (x η + k p. W r r (G, G f(x η + o(. (7 The rght hand sde of Eq. (7 s a near combnaton of the parta dervatves of f evauated at G. In addton, by Lemma 3., the coeffcents of ths near representaton s asymptotcay equvaent to the rato of two sempoynomas. Equaton (7 hghghts the dstnct roes of mode parameters and the kerne densty functon n the formaton of a mxture mode s kehood. The former appear ony n the coeffcents, whe the atter provdes the parta dervatves whch appear as f bass functons for the near combnaton. We wrote as f, because the parta dervatves of kerne f may not be neary ndependent functons (reca the exampes n Secton 3.. When a parta dervatve of f can be represented as a near combnaton of other parta dervatves, t can be emnated from the expresson n the rght hand sde. Ths reducton process may be repeatedy apped unt a parta dervatves that reman are neary ndependent functons. Ths motvates the foowng. Defnton 3.. The foowng representaton s caed r-mnma form of the mxture kehood for a sequence of mxng measures G tendng to G n W r metrc: p G (x p G (x W r r (G, G T r ( (r ξ (G = Wr r (G, G whch hods for a x, wth the ndex rangng from to a fnte T r, f ( H (r (x for a are neary ndependent functons of x, and = H (r (x + o(, (8 (2 coeffcents ξ (r (G are poynomas of the components of η, and p., p. It s suffcent, but not necessary, to seect functons H (r from the coecton of parta dervatves κ f/ η κ evauated at partcuar atoms η of G, where κ r, by adoptng the emnaton technque. The precse formuaton of ξ (r (G and H (r (x w be determned expcty by the specfc G. The r-mnma form for each G s not unque, but they pay a fundamenta roe n our noton of the snguarty eve of G reatve to a cass of mxng dstrbutons G. Defnton 3.2. Fx r and et G be a cass of dscrete probabty measures whch has a bounded number of support ponts n Θ. We say G s r-snguar reatve to G, f G admts a r-mnma form gven by Eq. (8, accordng to whch there exsts a sequence of G G tendng to G under W r such that ξ (r (G/W r r (G, G for a =,..., T r. 9

10 We now verfy that the r-snguarty noton s we-defned, n that t does not depend on any specfc choce of the r-mnma form. Ths nvarant property s confrmed by part (a of the foowng emma. Part (b estabshes a cruca monotonc property. Lemma 3.2. (a (Invarance The exstence of the sequence of G n the statement of Defnton 3.2 hods for a r-mnma forms once t hods for at east one r-mnma form. (b (Monotoncty If G s r-snguar for some r >, then G s (r -snguar. Proof. (a The exstence of the sequence of G descrbed n the defnton of a r-mnma form mpes for that sequence, (p G (x p G (x/wr r (G, G hods for any x. Now take any r-mnma form (8 gven by the same sequence. Let C(G = max Tr = not, we have m nf C(G >. It foows that T r = ( ξ (r (G Wr r(g,g ξ (r (G C(GWr r H (r (G, G (x.. If m nf C(G =, we are done. If Moreover, a the coeffcents n the above dspay are bounded from above by, one of whch s n fact. There exsts a subsequence of G by whch these coeffcents have mts, one of whch s. Ths s aso a contradcton due to the near ndependence of functons H (r (x. (b Let G be an eement n the sequence that admts a r-mnma form such that ξ (r (G/Wr r (G, G for a =,..., T r. It suffces to assume that the bass functons H (r are seected from the coecton of parta dervatves of f. We w show that the same sequence of G and the emnaton procedure for the r-mnma form can be used to construct a r -mnma form by whch ξ (r (G/Wr r (G, G for a =,..., T r. There are two possbtes to consder. Frst, suppose that each of the r-th parta dervatves of densty kerne f (.e., κ f/ η κ, where κ = r s not n the near span of the coecton of parta dervatves of f at order r or ess. Then, for each =,..., T r, ξ (r (G = ξ (r (G for some [, T r ]. Snce Wr r (G, G Wr r (G, G, due to the fact that the support ponts of G and G are n a bounded set, we have that ξ (r (G/Wr r (G, G ξ (r (G/W r r (G, G whch vanshes by the hypothess. Second, suppose that some of the r-th parta dervatves, say, κ f/ η κ where κ = r, can be emnated because they can be represented by a near combnaton of a subset of other parta dervatves H (r (n addton to possby a subset of other parta dervatves H (r wth correspondng fnte coeffcents α κ,,. It foows that for each =,..., T r, the coeffcent ξ (r (G that defnes the r -mnma form s transformed nto a coeffcent n the r-mnma form by Snce ξ (r ξ (r (G := ξ (r (G + (G/W r r (G, G tends to, so does ξ (r that κ = r, k ξ (r k s α κ,, κ; κ =r p ( η κ /κ!. (G/Wr r (G, G. By Lemma 3. for each κ such r (G, G. It foows that s p ( η κ /κ! = o(d r (G, G = o(w r (G/W r r (G, G tends to, for each =,..., T r. Ths competes the proof.

11 The monotoncty of r-snguarty naturay eads to the noton of snguarty eve of a mxng measure G (and the correspondng parameters reatve to an ambent space G. Defnton 3.3. The snguarty eve of G reatve to a gven cass G, denoted by (G G, s, f G s not r-snguar for any r ;, f G s r-snguar for a r ; otherwse, the argest natura number r for whch G s r-snguar. The roe of the ambent space G s crtca n determnng the snguarty eve of G G. Ceary, f G G are both subsets of probabty measures that contan G, r-snguarty reatve to G entas r-snguarty reatve to G. Ths means (G G (G G. Let us ook at the foowng exampes. Take G = E k,.e., the settng of e-mxtures. It s easy to verfy that f the coecton of { κ f/ η κ (x η =,..., k ; κ } evauated at G s neary ndependent, then G s not -snguar reatve to E k. It foows that (G G =. On the other hand, f G = O k for any k > k,.e., the settng of o-mxtures. Then t can be shown that G s aways -snguar reatve to O k for any k > k. Thus, (G G. Now, f the coecton of { κ f/ η κ (x η =,..., k ; κ 2} evauated at G s neary ndependent, then t can be observed that G s not 2-snguar reatve to O k. Thus, (G G =. In fact, the condtons descrbed n the two exampes above are referred to as strong dentfabty condtons studed by Chen [995], Nguyen [23], Ho and Nguyen [26b]. Our concept of snguarty eve generazes such strong dentfabty condtons, by aowng us to consder stuatons where such condtons fa to hod. Ths s when (G G = 2, 3,...,. The sgnfcance of ths concept can be apprecated by the foowng theorem. Theorem 3.. Let G be a cass of probabty measures on Θ that have a bounded number of support ponts, and fx G G. Suppose that (G G = r, for some r. p G p G ( If r <, then nf G G Ws s (G, G > for any s r +. V (p G, p G ( If r <, then nf G G Ws s > for any s r +. (G, G ( If r < and n addton, (a f s (r + -order dfferentabe wth respect to η and for some constant c >, ( r+ 2 f sup η α (x η /f(x η dx < (9 for any α = r +. η η c x X (b There s a sequence G G tendng to G n Wassersten metrc W r and the coeffcents of the r-mnma form ξ (r (G = for a =,..., T r. Then, for any s < r +, h(p G, p G m nf G G:W (G,G W s(g, G =.

12 (v If r = and the condtons (a, (b n part ( hod for any N (here, the parameter r n these condtons s repaced by, then the concuson of part ( hods for any s. We make a few remarks. Part ( and part ( show how the snguarty eve of G reatve to an ambent space G may be used to transate the convergence of mxture denstes (under the sup-norm and the tota varaton dstance nto the convergence of mxng measures under a Wassersten metrc. Part ( shows a suffcent condton under whch the power r + n the bounds from part ( and ( s n fact tght. In part ( the condton regardng the ntegrand of the parta dervatve of f (cf. Eq. (9 can be easy checked to be satsfed by many kernes, such as Gaussan kerne, Gamma kerne, Student t s kerne, and skewnorma kerne. Condton (b regardng the sequence of G appears somewhat opaque n genera, but t w be seen n specfc exampes for skewnorma mxtures n the seque. It s suffcent, but not necessary, for verfyng the r-snguarty of G to construct the sequence of G so that ξ (r (G = for a T r, provded such a sequence exsts. Ths requres fndng an approprate parameterzaton of a sequence of G tendng toward G that satsfy a number of poynoma equatons defned n terms of the parameter perturbatons. Proof. Here, we provde the proof for part ( and ( of the theorem. The proof for part ( and (v s deferred to the Appendx. ( It suffces to prove the frst nequaty for s = r +. Frsty, we w demonstrate that m nf p G p G /Ws s (G, G >. G G:W s(g,g If ths s not true, then there exsts a sequence of G such that W s (G, G, and for any x, (p G (x p G (x/w s s (G, G. Take any s-mnma form for ths rato, we have p G (x p G (x W s s (G, G T s ( (s ξ (G = Ws s (G, G = H (s (x + o(. ξ (s (G For each G n the sequence, et C(G = max Ws s (G, G. If m nf C(G =, then ths means G s s-snguar, so (G G s. Ths voates the gven assumpton. So we have m nf C(G >. It foows that T s = ( ξ (s (G C(GWs s H (s (G, G (x. Moreover, a coeffcents n the above dspay are bounded from above by, one of whch s n fact. There exsts a subsequence of G by whch these coeffcents have a mt, one of whch s. Ths s aso a contradcton due to the near ndependence of functons H (s. Therefore, we can fnd a postve number ɛ such that p G p G Ws s (G, G for any W s (G, G ɛ. Now, to obtan the concuson of part (, t suffces to demonstrate that nf p G p G /Ws s (G, G >. G G:W s(g,g >ɛ 2

13 If ths s not the case, there s a sequence G such that W s (G, G > ɛ and p G p G /W s s (G, G. Snce Θ s compact and G contans ony probabty measures wth bounded number of support ponts n Θ, we can fnd G G such that W s (G, G and W s (G, G ɛ. As W s (G, G W s (G, G >, we have p G p G. Now, due to the frst order unform Lpschtz condton of f, we obtan p G (x p G (x for a x X. Thus, p G (x = p G (x for amost a x X, whch entas that G = G, a contradcton. Ths competes the proof. ( Turnng to the second nequaty, we aso frsty demonstrate that m nf V (p G, p G /Ws s (G, G >. G G:W s(g,g If t s not true, then we have a sequence of G such that W s (G, G and V (p G, p G /W s s (G, G. By Fatou s emma = m nf V (p G, p G C(GWs s (G, G m nf G ξ (s (G C(GWs s (G, G H(s (x dx. The ntegrand must be zero for amost a x, eadng to a contradcton as before. Hence, to obtan the concuson of part (, we ony need to show that nf G G:W s(g,g >ɛ V (p G, p G /W s s (G, G >. where ɛ > such that V (p G, p G Ws s (G, G for any W s (G, G ɛ. If t s not true, by usng the same argument as that of part (, there s a sequence of G such that W s (G, G, V (p G, p G, whe W s (G, G ɛ and p G (x p G (x for a x X. By Fatou s emma, = m nf V (p G, p G m nf p G (x p G (x dx = V (p G, p G, whch eads to G = G, a contradcton. We obtan the concuson of ths part. We are ready to state the mpact of the snguarty eve of mxng measure G reatve to an ambent space G on the rate of convergence for an estmate of G, where G = E k n e-mxtures, and G = O k n o-mxtures. Let G be structured nto a seve of subsets defned by the maxmum snguarty eve reatve to G. G = G r, where G r := r= { G G }, (G G r r =,,...,. The frst part of the foowng theorem gves a mnmax ower bound for the estmaton of the mxng measure G, gven that the snguarty eve of G s known up to a constant r. The second part gves a quck resut on the convergence rate of a pont estmate such as the MLE. Theorem 3.2. (a Fx r. Assume that for any G G r, the concuson of part ( of Theorem 3. hods for G r (.e., G s repaced by G r n that theorem. Then, for any s [, r + there hods nf Ĝ n G r sup E pg W s (Ĝn, G n /2s. G G r Here, the nfmum s taken over a sequences of estmates Ĝn G r and E pg denotes the expectaton taken wth respect to product measure wth mxture densty p n G. 3

14 (b Let G G r for some fxed r. Let Ĝn G r be a pont estmate for G, whch s obtaned from an n-sampe of..d. observatons drawn from p G. As ong as h(pĝn, p G = O P (n /2, we have W r+ (Ĝn, G = O P (n /2(r+. Proof. Part (a of ths theorem s a consequence of the concuson of Theorem 3., part (. The proof of ths fact s qute standard, and smar to that of Theorem.. of [Ho and Nguyen, 26a] and s omtted. Part (b foows mmedatey from part ( of Theorem 3., as we have h(pĝn, p G V (pĝn, p G W r+ r+ (Ĝn, G. We concude ths secton wth some comments. It s we-known that many densty estmaton methods, such as MLE and Bayesan estmaton apped to a compact parameter space for parametrc mxture modes, guarantee a root-n rate (up to a ogarthmc term of convergence under Henger dstance metrc on the densty functons (cf. [???]. It foows that, as far as we are concerned, the remanng work n estabshng the convergence behavor of parameter estmaton (as opposed to densty estmaton es n the cacuaton of the snguarty eves,.e., the dentfcaton of sets G r. For skewnorma mxtures, such cacuatons w be carred out n Secton 4 and Secton 5. For the settngs of G where we are abe to obtan the exact snguarty eves, we can aso construct the sequence of G requred by part ( of Theorem 3.. Whenever the exact snguarty eve s obtaned, we automatcay obtan a mnmax ower bound and a matchng upper bound for MLE convergence rate under a Wassersten dstance metrc, thanks to the above theorem. In some cases, however, the snguarty eve of G may be not determned exacty, but ony an upper bound s gven. In such cases, ony an upper bound to the convergence rate of the MLE can be obtaned, whe mnmax ower bounds may be unknown. 3.3 Constructon of r-mnma forms As we mentoned above, a smpe way of constructng an r-mnma form s to seect a subset of parta dervatves of f taken up to order r such that a these functons are neary ndependent. A smpe procedure s to start from the smaest order r = and then move up to r = 2, 3,... and so on. For each r, assume that we have obtaned a neary ndependent subset of parta dervatves up to order r. Now, gong over the ordered st of r-th parta dervatves: { κ f/ η κ κ N d, κ = r}. For each κ such that κ = r, f the parta dervatve of f of order κ can be expressed as a near combnaton of other parta dervatves aready seected, then ths one s emnated. The process goes on unt we exhaust the st of the parta dervatves. Exampe 3.2. Contnung from Exampe 3., suppose that G satsfes Eq. (. Accordng to the proof of Lemma 4., we can choose α 4k, so the parta dervatve may be emnated va the reducton: f(x η k m = k α f(x η + α 2 f(x η + α 3 f(x η α 4k α 4k θ α 4k v k α 4 f(x η α 4k m Note that ths emnaton step s vad ony for a subset of G = G(p, η for whch Eq. ( hods. That s, ony f P (η = or P 2 (η =. Exampe 3.3. If f(x η = f(x θ, v, m where m =, the skewnorma kerne becomes the Gaussan kerne. Due to (3, a parta dervatves wth respect to both θ and v can be emnated va the foowng reducton: for any κ, κ 2 N, for any =,..., k, κ +κ 2 f(x η θ κ v κ 2 = 2 κ 2 4 κ +2κ 2 f(x η θ κ +2κ 2.

15 Thus, ths emnaton s vad for a parameter vaues (p, η, and r-mnma forms for a orders. Exampe 3.4. For the skewnorma kerne densty f(x η = f(x θ, v, m, Eq. (2 yeds the foowng reductons: for any =,..., k, any η = (θ, v, m = η = (θ, v, m such that m 2 f θ 2 = 2 f v m3 + m f v m, ( 2 f v m = f v m m2 + 2 f 2vm m 2. ( Ths resuts n a rppe effect on subsequent emnatons at hgher orders. For exampes, parta dervatves up to the thrd order of f evauated at η = η = (θ, v, m for any =,..., k where m can be expressed as foows: 3 f θ 3 = 2 2 f θ v m3 + m 2 f v θ m, 3 f θ 2 = 2 2 f v v 2 + m3 + m f v 2 m m3 + m 2 f v v m, 3 f θ 2 m = 2 2 f v m 3m2 + f v m m3 + m 2 f v m 2, 3 f v m 2 = m2 + 3 f 2vm m 3 3m2 2 f 2vm 2 m 2, 3 f v 2 m = 2 2 f v v m m2 + 3 f 2vm v m 2 = (m f 4v 2 m 2 m 3 + (m2 + (7m 2 2 f 4m 3 v 2 m f v 2 m, 3 f θ v m = + 3 f m2 2vm θ m 2 2 f v θ m. (2 A three exampes above demonstrate how the dependence among parta dervatves of kerne densty f, among dfferent orders κ, and among those evauated at dfferent component, has a deep mpact on the representaton of r-mnma forms. In genera, the r-mnma form (8 may be expressed somewhat more expcty as foows p G (x p G (x W r r (G, G = (,κ I,K ξ (r,κ (G Wr r (G, G H(r,κ (x G + k ζ (r (G Wr r (G, G f(x η + o(. where I {,..., k } and K N d of eements κ such that κ r. It s emphaszed that the sets I and K are specfc to a partcuar r-mnma form under consderaton. H (r,κ are a coecton of neary ndependent parta dervatves of f that are aso ndependent of a functons f(x η. H(r,κ are taken from the coecton of parta dervatves wth order at most r. We aso observe that ξ (r,κ and ζ(r take the foowng poynoma forms: s ξ (r,κ (G = ζ (r s p ( η κ + β,κ, κ!,κ (G p ( η κ κ, (3!,κ s (G = p. + γ,,κ (G p ( η κ κ. (4!,κ 5

16 In the rght hand sde of each of the ast two equatons, s taken from a subset of {,..., k } and κ s from a subset of N d such that κ κ r. The actua deta of these subsets depend on the specfc emnaton scheme eadng to the r-mnma form. Lkewse, the non-zero coeffcents β,κ,,κ (G and γ,κ,,κ (G arse from the specfc emnaton scheme. We ncude argument G n these coeffcents to hghght the fact that they may be dependent on the atoms of G (cf. Exampe 3.2 and 3.4. By the defnton of r-snguarty for any r, G s r-snguar reatve to G f there exsts a sequence of G tendng to G n the ambent space G such that the sequences of sempoynoma fractons, namey, ξ (r,κ (G/W r r (G, G and ζ (r (G/Wr r (G, G (whose numerators are gven by Eq. (3 and Eq. (4, must vansh. As a consequence, the queston of r-snguarty for a gven eement G s determned by the mtng behavor of a fnte coecton of nfnte sequences of sempoynoma fractons. 3.4 Poynoma mts of r-mnma form coeffcents It s worth notng that the mtng behavor of sempoynoma fractons descrbed above s ndependent of a partcuar choce of the r-mnma form, n a sense that we now expan. In part (a of Lemma 3.2, we estabshed an nvarance property of the r-snguarty, whch does not depend on a specfc form of the r-mnma form. Let us restrct the bass functons to be members of the coecton of a parta dervatves of f up to order r. In the proof of part (b of Lemma 3.2 t was shown that the coeffcents (G have to be eements of a set of poynomas of η, p., and p, whch are cosed under near combnatons of ts eements. Let us denote ths set by P(G, G, whch s nvarant wth respect to any specfc choce of the bass functons (from the coecton of parta dervatves for the r-mnma form. Moreover, G s r-snguar f and ony f a sequence of G tendng to G n W r can be constructed ξ (r such that for any eement ξ (r (G P(G, G, we have ξ (r (G/Wr r (G, G. Equvaenty, ξ (r (G/D r (G, G for a ξ r (G P(G, G. (5 Extractng the mts of a snge mutvarate sempoynoma fracton (a.k.a. ratona sempoynoma functons s qute chaengng n genera, due to the nteracton among mutpe varabes nvoved [Xao and Zeng, 24]. Anayzng the mts of not one but a coecton of mutvarate ratona sempoynomas s consderaby more dffcut. To obtan meanngfu and concrete resuts, we need to consder specfc systems of mutvarate ratona sempoynomas that arse from the r-mnma form. In the remander of ths paper we w proceed to do ust that. By workng wth a specfc choce of kerne densty f (the skewnorma, t w be shown that under the compactness of the parameter spaces, one can extract a subset of mts from the system of ratona sempoynomas ξ (r (G/D r (G, G. These mts are expressed as a system of poynomas admttng non-trva soutons. For a gven r, f the extracted system of poynoma mts does not contan admssbe soutons, then t means that there does not exst any sequence of mxng measures G for whch a vad r-mnma form can be constructed, because (5 s not fufed. Ths woud enta the upper bound (G G < r. On the other hand, f the extracted system of poynoma mts does contan at east one admssbe souton, ths s a hnt that the r-snguarty eve of G reatve to the ambent space G mght hod. Whether ths s actuay the case or not requres an expct constructon of a sequence of G G (often budng upon the admssbe soutons of the poynoma mts and then the verfcaton that condton (5 ndeed hods. For the verfcaton purpose, t s suffcent (and smper to work wth a specfc choce of r-mnma form, as Defnton 3.2 aows. The foregong descrpton, aong wth the presentaton n the prevous subsecton on the constructon of r-mnma forms, provdes the outne of a genera procedure whch nks the determnaton 6

17 of the snguarty eve to the sovabty of a system of poynoma mts. Ths procedure w be ustrated n the next sectons for the remarkabe word of mxtures of skewnorma dstrbutons. 4 O-mxtures of skewnorma dstrbutons In ths secton, we focus on the o-mxture settng of skewnorma dstrbutons. To avod heavy techncaty, we study the snguarty eve of G E k reatve to ambent space O k,c for some k > k and sma constant c >, where O k,c O k contans ony (dscrete probabty measures whose pont masses are bounded beow by c. Moreover, we w focus on the snguarty eve of G S, a subset to be defned shorty by (6. Ths case s nterestng n that t ustrates the fu power of the genera method of anayss that was descrbed n Secton 3 n a concrete fashon. Due to the compex nature, we w report a resuts on the case where G s n the compement of S n Secton 6. Instead, n Secton 5 we study the snguarty eve of G reatve to the smaer ambent space E k (that s, e-mxture settng, for whch a more compete pcture of the snguarty structure s acheved. Lemma 4.. For skewnorma densty kerne f(x η, the coecton of { κ f/ η κ (x η =,..., k ; κ } s not neary ndependent f and ony f η = (η,..., η k are the zeros of ether poynoma P or P 2, whch are defned as foows: Type A: P (η = k Type B: P 2 (η = m. k { ] 2 } (θ θ 2 + [σ 2 ( + m2 σ2 ( + m2. Ths emma eads us to consder { } S = G = G(p, η (p, η Ω, P (η, P 2 (η. (6 In the o-mxture settng, we w see that (G O k,c may grow wth k k, the number of extra mxng components. The man excercse s to arrve at a sutabe r-mnma form, for whch the vanshng behavor of ts coeffcents can be anayzed. Secton 3.3 descrbes a genera strategy for the constructon of r-mnma form based on the parta dervatves of the densty kerne f wth respect to the parameters η = (θ, v, m up to order r. For skewnorma kerne densty f, the foowng emma provdes an expct form for reducng a parta dervatve of f to other parta dervatves of ower orders. Lemma 4.2. For any α = (α, α 2, α 3 N 3, denote A r = {(α, α 2, α 3 : α, α 3 =, and α r}. A r 2 = {(α, α 2, α 3 : α, α 2 =, α 3, and α r}. F r = A r A r 2. Let f(x η = f(x θ, v, m denote the skewnorma kerne. Then, as m, there hods α f θ α v α 2 m α 3 = κ F α κ P,κ 2,κ 3 α,α 2,α 3 (m H κ,κ 2,κ 3 α,α 2,α 3 (mq κ,κ 2,κ 3 α,α 2,α 3 (v κ f θ κ v κ 2 m κ 3, where, P κ,κ 2,κ 3 α,α 2,α 3 (m, H κ,κ 2,κ 3 α,α 2,α 3 (m, and Q κ,κ 2,κ 3 α,α 2,α 3 (v are poynomas n terms of m, v respectvey. 7

18 f θ f v f m 2 f θ 2 2 f v 2 2 f θm 2 2 f θ v 2 f θ m 2 f vθm 3 f θ 3 3 f v 3 3 f θm 3 3 f θ 2 v 3 f θ 2 m 3 f v 2 m 3 f θ v 2 3 f θ m 2 3 f v m 2 3 f θ v m Fgure : The ustraton of the reducton step from a compete coecton of dervatves of f up to the order 3 to a reduced system of neary ndependent parta dervatves, cf. Lemma 4.3. The crced dervatves are emnated from the parta dervatves present n the 3-mnma form. A B means that B s nvoved n the representaton of A under the reducton. Next, we show that the parta dervatves on the RHS of the above dentty are n fact neary ndependent, under addtona assumptons on G. Lemma 4.3. Reca the notaton from Lemma 4.2. If G S, then for any r, the coecton of parta dervatves of the skewnorma densty kerne f(x η, namey { κ } f(x η θ κ v κ 2m κ 3 κ = (κ, κ 2, κ 3 F r, η = η,..., ηk are neary ndependent. Fgure gves an ustraton of Lemma 4.3 when r = 3. Armed wth the foregong emmas we can easy obtan a sutabe r-mnma form for the mxture denstes of skewnormas. 4. Speca cases To ustrate our technques and resuts, consder a speca case n whch G has exacty one atom, and k = k + = 2. The genera resuts w be present n Secton 4.2. G s -snguar G S mpes that a frst order dervatves of f are neary ndependent. Hence, from Eq. (8, the -mnma form takes the form: p G (x p G (x W (G, G + 2 ( p. f(x η W (G, G + p v f v (x η p θ f θ (x η f p m m (x η + o(. (7

19 Snce k = 2 and k =, we have p. =. A sequence of G can be chosen so that 2 p v =, reatve to O 2,c. G s 2-snguar 2-mnma form: 2 p θ =, 2 p m =. Ceary, a of the coeffcents n (8 are. Hence G s -snguar where ξ (2 κ,κ 2,κ 3 are gven by ξ (2 Usng the method of reducton descrbed n Exampe 3.4 we obtan the foowng ( W2 2(G, G ξ (2 κ F 2 ξ (2 ξ (2,, = (m 3 + m 2v,2, = 2 κ,κ 2,κ 3 α f θ κ v κ 2 m κ 3 (x η + o(, (8 2 2,, = p θ, ξ (2,, = p v + 2 p ( θ 2, 2 p ( θ v p v m + p m, 2 2 p v m + p (m 2, p ( v 2, ξ (2,,2 = (m 2 + 2v m ξ (2 2 2,, = p θ v, ξ (2,, = p θ m. Note n partcuar the formuas for ξ (2,,, ξ(2,, and ξ(2,,2 are the resuts of reducton equaton (. It remans to construct a sequence of G tendng to G so that ξ κ (2 /W2 2(G, G vansh for a κ = (κ, κ 2, κ 3 F 2. Defne M = max { θ, θ 2, v /2, v 2 /2, m /2, m 2 /2}. Then, t can be observed that W 2 2 (G, G M 2 and ξ (2 κ,κ 2,κ 3 = O(M κ +2κ 2 +2κ 3. So, for any κ F 2 such that κ + 2κ 2 + 2κ 3 3, as ξ κ (2,κ 2,κ 3 = O(M s where s 3, t mpes that ξ κ (2,κ 2,κ 3 /W2 2(G, G. So we ony need to consder the coeffcents where κ +2κ 2 +2κ 3 2 and κ. They are ξ (2,, /W 2 2 (G, G, ξ (2,, /W 2 2 (G, G, and ξ (2,, /W 2 2 (G, G. Now, by dvdng both the numerator and numerator of each of these coeffcents by M, M 2, and M 2, respectvey, we extract the foowng system of poynoma mts: d 2 a + d 2 2a 2 =, d 2 a 2 + d 2 2a d 2 b + d 2 2b 2 =, (m 3 + m 2v (d 2 a 2 + d 2 2a d 2 c + d 2 2c 2 =, (9 where θ /M a, v /M 2 b, m /M 2 c, p d 2 for a 2. One souton to the above system of poynoma equatons s d = d 2, a = a 2, b = b 2 = a 2 /2, c = c 2 = ( (m 3 + m /2v. It foows that f we choose the sequence of G so that p = p 2 = /2, θ = θ 2, v = v 2 = ( θ 2 /2, and m = m 2 = ( θ 2 ( (m 3 + m /2v, then a coeffcents of the 2-mnma form vansh. Hence, G s 2-snguar reatve to O 2,c. 9