Finite Mixtures of Multivariate Skew Laplace Distributions

Transcription

1 Fiite Mixtures of Multivariate Skew Laplace Distributios Fatma Zehra Doğru 1 *, Y. Murat Bulut 2 ad Olcay Arsla 3 1 Giresu Uiversity, Faculty of Arts ad Scieces, Departmet of Statistics, Giresu/Turkey. ( fatma.dogru@giresu.edu.tr) 2 Eskisehir Osmagazi Uiversity, Faculty of Sciece ad Letters, Departmet of Statistics, Eskisehir/Turkey.( ymbulut@ogu.edu.tr) 3 Akara Uiversity, Faculty of Sciece, Departmet of Statistics, Akara/Turkey. ( oarsla@akara.edu.tr) Abstract I this paper, we propose fiite mixtures of multivariate skew Laplace distributios to model both skewess ad heavy-tailedess i the heterogeeous data sets. The maximum likelihood estimators for the parameters of iterest are obtaied by usig the EM algorithm. We give a small simulatio study ad a real data example to illustrate the performace of the proposed mixture model. Key words: EM algorithm, ML estimatio, multivariate mixture model, MSL. 1. Itroductio Fiite mixture models are used for modelig heterogeeous data sets as a result of their flexibility. These models are commoly applied i fields such as classificatio, cluster ad latet class aalysis, desity estimatio, data miig, image aalysis, geetics, medicie, patter recogitio etc. (see for more detail, Titterigto et al. (1985), McLachla ad Basford (1988), McLachla ad Peel (2000), Bishop (2006), Frühwirth-Schatter (2006)). Geerally, due to its tractability ad wide applicability, the distributio of mixture model compoets is assumed to be ormal. However, i practice, the data sets may be asymmetric ad/or heavy-tailed. For this, there are some studies i literature for multivariate mixture modelig usig the asymmetric ad/or heavy-tailed distributios. Some of these studies ca be summarized as follows. Peel ad McLachla (2000) proposed fiite mixtures of multivariate t distributios as a robust extesio of the multivariate ormal mixture model (McLachla ad Basford (1988)), Li (2009) itroduced multivariate skew ormal mixture models, Pye et al. (2009) ad Li (2010) proposed fiite mixtures of the restricted ad urestricted variats of multivariate skew t distributios of Sahu et al. (2003), Cabral et al. (2012) proposed multivariate mixture modelig based o the skew-ormal idepedet distributios ad Li et al. (2014) itroduced a flexible mixture modelig based o the skew-t-ormal distributio. I the multivariate aalysis, the multivariate skew ormal (MSN) (Azzalii ad Dalla Valle (1996), Gupta et al. (2004) ad Arellao-Valle ad Geto (2005)) distributio has bee proposed as a alterative to the multivariate ormal (MN) distributio to deal with skewess i the data. However, sice MSN distributio is ot heavy-tailed some alterative heavy-tailed skew distributios are eeded to model skewess ad heavy-tailedess. Oe of the examples of heavy-tailed skew distributio is the multivariate skew t (MST) distributio defied by Azzalii ad Capitaio (2003) ad Gupta (2003). The other heavy-tailed skew distributio is the multivariate skew Laplace (MSL) distributio proposed by Arsla (2010). The advatage of this distributio is that the MSL distributio has the less umber of parameters tha MST distributio ad it has the same umber of parameters with the MSN distributio. Cocerig the fiite mixtures of distributios, sice the mixtures of MN distributios are ot able to 1

2 model skewess, fiite mixtures of MSN distributios have bee proposed by Li (2009) to model such data sets. I this study, we explore the fiite mixtures of MSL distributios as a alterative to the fiite mixtures of MSN distributios to deal with both skewess ad heavy-tailedess i the heterogeeous data sets. The rest of this paper is orgaized as follows. I Sectio 2, we briefly summarize some properties of MSL distributio. I Sectio 3, we preset the mixtures of MSL distributios ad give the Expectatio- Maximizatio (EM) algorithm to obtai the maximum likelihood (ML) estimators of proposed mixture model. I Sectio 4, we give the empirical iformatio matrix of MSL distributio to compute the stadard errors of proposed estimators. I the Applicatio Sectio, we give a small simulatio study ad a real data example to illustrate the performace of proposed mixture model. Some coclusios are give i Sectio Multivariate skew Laplace distributio A p-dimesioal radom vector Y R p is said to have MSL distributio (Y MSL p (μ, Σ, γ)) which is give by Arsla (2010) if it has the followig probability desity fuctio (pdf) f MSL (y; μ, Σ, γ) = Σ p π p 1 2 αγ ( p ) exp { α (y μ)t Σ 1 (y μ) + (y μ)t Σ 1 γ}, (1) where α = 1 + γ T Σ 1 γ, μ R p is a locatio parameter, γ R p is a skewess parameter ad Σ is a positive defiite scatter matrix. Propositio 1. The characteristic fuctio of MSL (μ, Σ, γ) is Φ Y (t) = e ittμ [1 + t T Σt 2it T γ] (p+1)/2, tεr p. Proof. Sice we kow coditioal distributio of Y give V, we get characteristic fuctio as follows Φ Y (t) = E (E Y V (e itty )) = e ittμ E V (e V 1 ( tt Σt 2 itt γ) ) = e ittμ e v 1 ( tt Σt 2 itt γ) g(v)dv 0 = e ittμ [1 + t T Σt 2it T γ] (+1)/2. If Y MSL p (μ, Σ, γ) the the expectatio ad variace of Y is E(Y) = μ + (p + 1)γ, Var(Y) = (p + 1)(Σ + 2γγ T ). The MSL distributio ca be obtaied as a variace-mea mixture of MN distributio ad iverse gamma (IG) distributio. The variace-mea mixture represetatio is give as follows Y = μ + V 1 γ + V 1 Σ 1/2 X (2) 2

3 where X N p (0, I p ) ad V IG( p+1, 1 ). Note that if γ = 0, the desity fuctio of Y reduces to the 2 2 desity fuctio of symmetric multivariate Laplace distributio give by Naik ad Plugpogpu (2006). Also, the coditioal distributio of Y give V = v will be Y v N(μ + v 1 γ, v 1 Σ). The joit desity fuctio of Y ad V is f(y, v) = 1 Σ 2e (y μ)t Σ 1 γ 2 p π p 1 2 αγ ( p + 1 {v 3 2e 1 2 {(y μ)t Σ 1 (y μ)v+(1+γ T Σ 1 γ)v 1} }. 2 ) The, we have the followig coditioal desity fuctio of V give Y f(v y) = α 2π eα (y μ)t Σ 1 (y μ) v 3 2 e 1 2 {(y μ)t Σ 1 (y μ)v+α 2 v 1}, v > 0. (3) Propositio 2. Usig the coditioal desity fuctio give i (3), the coditioal expectatios ca be obtaied as follows 1 + γ T Σ 1 γ E(V y) = (y μ) T Σ 1 (y μ), (4) E(V 1 y) = 1 + (1 + γt Σ 1 γ)(y μ) T Σ 1 (y μ) 1 + γ T Σ 1. γ (5) Note that these coditioal expectatios will be used i the EM algorithm give i Subsectio Fiite mixtures of the MSL distributios Let y 1,, y be a p-dimesioal radom sample which come from a g-compoet mixtures of MSL distributios. The pdf of a g-compoet fiite mixtures of MSL distributios is give by g f(y Θ) = π i f(y; μ i, Σ i, γ i ), i=1 (6) g where π i deotes the mixig probability with i=1 π i = 1, 0 π i 1, f(y; μ i, Σ i, λ i ) represets the pdf of the ith compoet (pdf of the MSL distributio) give i (1) ad Θ = (π 1,, π g, μ 1,, μ g, Σ 1,, Σ g, γ 1,, γ g ) T is the ukow parameter vector. 3.1 ML estimatio The ML estimator of Θ ca be foud by maximizig the followig log-likelihood fuctio g l(θ) = log ( π i f(y j ; μ i, Σ i, γ i )). i=1 (7) 3

4 However, there is ot a explicit maximizer of (7). Therefore, i geeral, the EM algorithm (Dempster et al. (1977)) is used to obtai the ML estimator of Θ. Here, we will use the followig EM algorithm. Let z j = (z 1j,, z gj ) T be the latet variables with z ij = { 1, if jth observatio belogs to i th compoet 0, otherwise (8) where j = 1,, ad i = 1,, g. To implemet the steps of the EM algorithm, we will use the stochastic represetatio of the MSL distributio give i (2). The, the hierarchical represetatio for the mixtures of MSL distributios will be Y j v j, z ij = 1 N(μ + v j 1 γ, v j 1 Σ), V j z ij = 1 IG ( p + 1 2, 1 2 ). (9) Let (y, v, z) be the complete data, where y = (y 1 T,, y T ) T, v = (v 1,, v ) ad z = (z 1,, z ) T. Usig the hierarchical represetatio give above ad igorig the costats, the complete data loglikelihood fuctio ca be writte by l c (Θ; y, v, z) = z ij {log π i 1 2 log Σ i + (y j μ i ) T Σ 1 i γ i i=1 1 2 v j(y j μ i ) T Σ i 1 (y j μ i ) 1 2 γ i T Σ i 1 γ i v j (3 log v j + v j 1 )}. (10) To overcome the latecy of the latet variables give i (10), we have to take the coditioal expectatio of the complete data log-likelihood fuctio give the observed data y j g E(l c (Θ; y, v, z) y j ) = E(z ij y j ) {log π i 1 2 log Σ i (y j μ i ) T Σ 1 i γ i i=1 1 2 E(V j y j )(y j μ i ) T Σ i 1 (y j μ i ) 1 2 γ i T Σ i 1 γ i E(V j 1 y j )}. (11) Sice the last part of the complete data log-likelihood fuctio does ot iclude the parameters, the above coditioal expectatio of the complete data log-likelihood fuctio oly cosists of the ecessary coditioal expectatios. Note that the coditioal expectatios E(V j y j ) ad E(V j 1 y j ) ca be calculated usig the coditioal expectatios give i (4) ad (5) ad the coditioal expectatio E(z ij y j ) ca be computed usig the classical theory of mixture modelig. EM algorithm: 1. Set iitial parameter estimate Θ ad a stoppig rule Δ. 2. E-Step: Compute the followig coditioal expectatios for k = 0,1,2, iteratio = E(zij y j, Θ ) = π i f (yj ; μ i, Σ i, γ i ) f(y j ; Θ ), (12) 4

5 v 1ij = E(Vj y j, Θ ) = 1 + T 1 γ i Σ i γ i, T (y j 1 μ i ) Σ i (y j μ i ) (13) v 2ij = 1 E(Vj y j, Θ ) = 1 + (1 + T 1 T γ i Σ i γ i ) (yj 1 μ i ) Σ i (y j μ i ) 1 + T 1 γ i Σ i γ i. (14) The, we form the followig objective fuctio Q(Θ; Θ ) = g i=1 {log π i 1 2 log Σ i (y j μ i ) T Σ i 1 γ i 1 2 v 1ij (yj μ i ) T Σ 1 i (y j μ i ) 1 2 v 2ij T γi Σ 1 i γ i }. (15) 3. M-step: Maximize the Q(Θ; Θ ) with respect to Θ to get the (k + 1)th parameter estimates for the parameters. This maximizatio yields the followig updatig equatios (k+1) π i = (k+1) μ i = v 1ij, v 1ij yj v 1ij γ i, ( (k+1) ) ( y j ) ( ) ( ) γ i = ( v 1ij ) ( v 2ij ) ( ) 2, (18) (k+1) Σ i = T v 1ij (yj μ i ) (yj μ i ) γ i γ i T v 1ij yj v 2ij 4. Repeat E ad M steps util the covergece rule Θ (k+1) Θ < Δ is obtaied. Alteratively, the absolute differece of the actual log-likelihood l(θ (k+1) ) l(θ ) < Δ or l(θ (k+1) ) l(θ ) 1 < Δ ca be used as a stoppig rule (see Dias ad Wedel (2004)).. (16) (17) (19) 3.2 Iitial values To determie the iitial values for the EM algorithm, we will use the same procedure give i Li (2009). The steps of the selectig iitial values are give as follows. i) Perform the K-meas clusterig algorithm (Hartiga ad Wog (1979)). g ii) Iitialize the compoet labels z j = {zij } accordig to the K-meas clusterig results. i=1 iii) The iitial values of mixig probabilities, compoet locatios ad compoet scale variaces ca be set as 5

6 π i = Σ i = y j, μ i = T (y j μ i ) (yj μ i ) iv) For the skewess parameters, use the skewess coefficiet vector of each clusters.,. 4. The empirical iformatio matrix We will compute the stadard errors of ML estimators usig the iformatio based method give by Basford et al. (1997). Here, we will use the iverse of the empirical iformatio matrix to have a approximatio to the asymptotic covariace matrix of estimators. The iformatio matrix is I e = s js jt, (20) where s j = E Θ ( l cj(θ;y j,v j,z j ) y Θ j ), j = 1,, are the idividual scores ad l cj (Θ; y j, v j, z j ) is the complete data log-likelihood fuctio for the jth observatio. The, the elemets of the score vector s j is (s j,π1,, s j,πg 1, s j,μ1,, s j,μg, s j,σ1,, s j,σg, s j,γ1,, s j,γg ) T. After some straightforward algebra, we obtai the followig equatios s j,πr = z rj z gj π r π g, r = 1,, g 1, (21) s j,μi = Σ i 1 (v 1ij (y j μ i) γ i), (22) s j,σi = vech ( { (Σ i 1 v 1ij Σ i 1 (y j μ i)(y j μ i) Σ i 1 v 2ij Σ i 1 γ iγ it Σ i 1 ) diag (Σ i 1 v 1ij Σ i 1 (y j μ i)(y j μ i) Σ i 1 (23) v 2ij Σ i 1 γ iγ it )} ), s j,γi = Σ i 1 ((y j μ i) v 2ij γ i). (24) Thus, the stadard errors of Θ will be foud by usig the square root of the diagoal elemets of the iverse of (20). 5. Applicatios I this sectio, we will illustrate the performace of proposed mixture model with a small simulatio study ad a real data example. All computatios for simulatio study ad real data example were coducted usig MATLAB R2013a. For all computatios, the stoppig rule Δ is take as Simulatio study I the simulatio study, the data are geerated from the followig two-compoet mixtures of MSL distributios 6

7 f(y j Θ) = π 1 f p (y j ; μ 1, Σ 1, γ 1 ) + (1 π 1 )f p (y j ; μ 2, Σ 2, γ 2 ), where σ i,12 μ i = (μ i1, μ i2 ) T, Σ i = [ σ i,11 σ i,21 σ ], γ i = (γ i1, γ i2 ) T, i = 1,2 i,22 with the parameter values μ 1 = (2,2) T, μ 2 = ( 2, 2) T, Σ 1 = Σ 2 = [ ], γ 1 = (1,1) T, γ 2 = ( 1, 1) T, π 1 = 0.6. We set sample sizes as 500, 1000 ad We take the umber of replicates (N) as 500. The tables cotai the mea ad mea Euclidea distace values of estimates. For istace, the formula for mea Euclidea distace of μ i is give below μ i μ i = 1 N N ((μ ij μ ij ) 2 ) 1 2. Similarly, the other mea Euclidea distaces for the other estimates are obtaied. Note that for the π 1 the distace will be mea squared error (MSE). The formula of MSE is give MSE (π ) = 1 N N (π j π) 2 where π is the true parameter value, π j is the estimate of π for the jth simulated data ad π = 1 N N. Table 1 shows the simulatio results for the sample sizes 500, 1000 ad I the tables, we give mea ad mea Euclidea distace values of estimates ad true parameter values. We ca observe from this table that the proposed model is workig accurately to obtai the estimates for all parameters. This ca be observed from mea Euclidia distaces that are gettig smaller whe the sample sizes icrease. π j 7

8 Table 1. Mea ad mea Euclidea distace values of estimates for = 500, 1000 ad Compoets 1 2 Parameter True Mea Distace True Mea Distace π μ i1 μ i σ i, σ i, σ i, γ i γ i π μ i μ i σ i, σ i, σ i, γ i γ i π μ i μ i σ i, σ i, σ i, γ i γ i Real data example I this real data example, we will ivestigate the bak data set which was give i Tables 1.1 ad 1.2 by Flury ad Riedwyl (1988) ad examied by Ma ad Geto (2004) to model with a skew-symmetric distributio. There are six measuremets made o 100 geuie ad 100 couterfeit old Swiss 1000 frac bills for this data set. This data set also aalyzed by Li (2009) to model mixtures of MSN distributios. They used the sample of X 1 : the width of the right edge ad X 2 : the legth of the image diagoal which reveals a bimodal distributio with asymmetric compoets. I this study, we will use the Swiss bak data to illustrate the applicability of the fiite mixtures of multivariate skew Laplace distributios (FM- MSL) ad also we compare the results with the fiite mixtures of multivariate skew ormal distributios (FM-MSN) which was give by Li (2009). We give the estimatio results i Table 2 for FM-MSN ad FM-MSL. The table cosists of the ML estimates, stadard errors of estimates for all compoets, the log-likelihood, the values of the Akaike iformatio criterio (AIC) (Akaike (1973)) ad the Bayesia iformatio criterio (BIC) (Schwarz (1978)) for the FM-MSL. Also, we give the estimatio results ad criterio values for FM-MSN which was computed by Li (2009) i Table 2. Accordig to iformatio criterio values that the FM-MSL has better fit tha the FM-MSN. I Figure 1, we display the scatter plot of the data alog with the cotour plots of the fitted two-compoet FM-MSL model. We ca observe from figure that the FM-MSL captures the asymmetry ad accurately fit the data. 8

9 Table 2. ML estimatio results of the Swiss bak data set for FM-MSN ad FM-MSL. FM-MSN FM-MSL Estimate SE Estimate SE Estimate SE Estimate SE w μ i μ i σ i, σ i, σ i, γ i γ i l(θ ) AIC BIC Figure 1. Scatter plot of the Swiss bak data alog with the cotour plots of the fitted twocompoet FM-MSL model 6. Coclusios I this paper, we have proposed the mixtures of MSL distributios. We have give the EM algorithm to obtai the estimates. We have provided a small simulatio study to show the performace of proposed mixture model. We have observed from simulatio study that the proposed mixture model has accurately estimated the parameters. We have also give a real data example to compare the mixtures of MSL distributios with the mixtures of MSN distributios. We have observed from this example that the proposed model has the best fit accordig to the iformatio criterio values. Thus, the proposed model ca be used as a alterative mixture model to the mixtures of MSN distributios. Refereces Akaike, H Iformatio theory ad a extesio of the maximum likelihood priciple. Proceedig of the Secod Iteratioal Symposium o Iformatio Theory, B.N. Petrov ad F. Caski, eds., , Akademiai Kiado, Budapest. 9

10 Arellao-Valle, R.B. ad Geto, M.G O fudametal skew distributios. Joural of Multivariate Aalysis, 96(1), Arsla, O A alterative multivariate skew Laplace distributio: properties ad estimatio. Statistical Papers, 51(4), Azzalii, A. ad Dalla Valle, A The multivariate skew-ormal distributio. Biometrika, 83(4), Azzalii, A. ad Capitaio, A Distributios geerated by perturbatio of symmetry with emphasis o a multivariate skew t distributio. Joural of the Royal Statistical Society: Series B (Statistical Methodology), 65(2), Basford, K.E., Greeway, D.R., McLachla, G.J. ad Peel, D Stadard errors of fitted meas uder ormal mixture. Computatioal Statistics, 12, Bishop, C.M Patter Recogitio ad Machie Learig. Spriger, Sigapore. Cabral, C.R.B., Lachos, V.H. ad Prates, M.O Multivariate mixture modelig usig skew-ormal idepedet distributios. Computatioal Statistics & Data Aalysis, 56(1), Dempster, A.P., Laird, N.M. ad Rubi, D.B Maximum likelihood from icomplete data via the EM algorithm. Joural of the Royal Statistical Society, Series B, 39, Dias, J.G. ad Wedel, M A empirical compariso of EM, SEM ad MCMC performace for problematic gaussia mixture likelihoods. Statistics ad Computig, 14, Flury, B. ad H. Riedwyl Multivariate Statistics, a Practical Approach, Cambridge Uiversity Press, Cambridge. Frühwirth-Schatter, S Fiite Mixture ad Markov Switchig Models. Spriger, New York. Gupta, A. K Multivariate skew t-distributio. Statistics: A Joural of Theoretical ad Applied Statistics, 37(4), Gupta, A.K., Gozález-Farıás, G. ad Domıńguez-Molia, J.A A multivariate skew ormal distributio. Joural of multivariate aalysis, 89(1), Hartiga, J. A. ad Wog, M. A Algorithm AS 136: A k-meas clusterig algorithm. Joural of the Royal Statistical Society. Series C (Applied Statistics), 28(1), Li, T.I Maximum likelihood estimatio for multivariate skew ormal mixture models. Joural of Multivariate Aalysis, 100, Li, T.I Robust mixture modelig usig multivariate skew t distributios. Statistics ad Computig, 20(3), Li, T.I., Ho, H. J. ad Lee, C.R Flexible mixture modellig usig the multivariate skew-t-ormal distributio. Statistics ad Computig, 24(4), Ma, Y. ad Geto, M.G Flexible class of skew-symmetric distribtios, Scadiavia Joural of Statististics, 31, McLachla, G.J. ad Basford, K.E Mixture Models: Iferece ad Applicatio to Clusterig. Marcel Dekker, New York. McLachla, G.J. ad Peel, D Fiite Mixture Models. Wiley, New York. Naik, D.N. ad Plugpogpu, K A Kotz-type distributio for multivariate statistical iferece. I Advaces i distributio theory, order statistics, ad iferece (pp ). Birkhäuser Bosto. Peel, D. ad McLachla, G. J Robust mixture modellig usig the t distributio. Statistics ad computig, 10(4), Pye, S., Hu, X., Wag, K., Rossi, E., Li, T.I., Maier, L., Baecher-Alla, C., McLachla, G.J., Tamayo, P., Hafler, D.A., De Jager, P.L. ad Mesirov, J.P Automated high-dimesioal flow cytometric data aalysis. Proc. Natl. Acad. Sci. USA 106, Sahu, S. K., Dey, D. K. ad Braco, M. D A ew class of multivariate skew distributios with applicatios to Bayesia regressio models. Caadia Joural of Statistics, 31(2), Schwarz, G Estimatig the dimesio of a model. The Aals of Statistics, 6(2), Titterigto, D.M., Smith, A.F.M. ad Markov, U.E Statistical Aalysis of Fiite Mixture Distributios. Wiley, New York. 10