Optimal Penalty Functions Based on MCMC for Testing Homogeneity of Mixture Models

Size: px

Start display at page:

Download "Optimal Penalty Functions Based on MCMC for Testing Homogeneity of Mixture Models"

Leo Perkins
5 years ago
Views:

1 Resear Joural of Applied Siees, Egieerig ad Teology 4(4: 04-09, 0 ISSN: Maxwell Sietifi Orgaizatio, 0 Submitted: Otober 5, 0 Aepted: November 5, 0 Publised: July 5, 0 Optimal Pealty Futios Based o MCMC for Testig Homogeeity of Mixture Models Rama Faroos, Morteza Ebraimi ad 3 Arezoo Hajirajabi Departmet of Matematis, Sout Tera Bra, Islami Azad Uiversity, Tera, Ira Departmet of Matematis, Karaj Bra, Islami Azad Uiversity, Karaj, Ira 3 Sool of Matematis, Ira Uiversity of Siee ad Teology, Narma, Tera 6846, Ira Abstrat: Tis study is iteded to provide a estimatio of pealty futio for testig omogeeity of mixture models based o Marov ai Mote Carlo simulatio. Te pealty futio is osidered as a parametri futio ad parameter of determiative sape of te pealty futio i ojutio wit parameters of mixture models are estimated by a Bayesia approa. Differet mixture of uiform distributio are used as prior. Some simulatio examples are perform to ofirm te effiiey of te preset wor i ompariso wit te previous approaes. Key words: Bayesia aalysis, expetatio-maximizatiotest, marov ai mote arlo simulatio, mixture distributios, modified lieliood ratio test INTRODUCTION Tere ave bee umerous appliatios of fiite mixture models i various braes of siee ad egieerig su as mediie, biology ad astroomy. Tey ave bee used we te statistial populatio is eterogeeous ad otais several subpopulatio. Te mai problem i appliatio of fiite mixture models is estimatio of parameters. To date various metods ave bee proposed for estimatig te parameters of fiite mixture models. Some of te most importatoes are Bayesia metod, maximum lieliood metod, miimum-distae metod ad metod of momets (Lidsay ad Basa, 993; Diebolt ad Robert, 994; Lavie ad West, 99. After estimatig te parameters of fiite mixture models a partiular statistial problem of iterest is weter te data are from a mixture of two distributio or a sigle distributio. Te problem is alled testig of omogeeity i tese models. At te begiig for testig omogeeity may be te Lieliood Ratio Test (LRT tat is te most extesively used metod for ypotesis testig problem, is employed. Te LRT as a i-squared ull limitig distributio uder te stadard regularity oditio. Sie te mixture models do t ave regularity oditios terefore limitig distributio of LRT isvery omplex (Liu ad Sao, 003. For overome te above metioed weaess of LRT metod te Modified Lieliood Ratio Test (MLRT as bee itrodued by Ce (998, Ce ad Ce (00 ad Ce et al. (004. I MLRT a pealty is added to te Log-lieliood futio ad as a result a simple ad smoot maer is establised to te problem. But depedee o establismet of several regularity oditios ad ose pealty futio are te mailimitatio of MLRT. Li et al. (008 as bee proposed Expetatio-Maximizatio (EM test based o aoter form of pealty futio tat suggested previously by Ce et al. (004. Tis test is idepedet of some eessary oditios for MLRT. But bot of te MLRT ad EM tests are based o pealized lieliood futio. Terefore te effiiey of tese tests is iflueed by te sape of te ose pealty futio. Hee oe of te tese tests is geerally optimal. To remove te above metioed disadvatage, i te preset study we osider te pealty futio as a parametri futio ad employ Metropolis-Hastigs samplig as a MCMC metod for estimatio parameters of mixture models ad parameter of determiative sape of te pealty futio. To our best owledge te parametri pealty futio as ot bee studied before. Furtermore, aordig to latest iformatio from te resear wors it is believed tat te MCMC estimatio of pealty futio based o differet priors to testig omogeeity of mixture model as bee ivestigated for te first time i te preset study. METHODOLOGY Te fiite mixture models: A mixture distributio is a ovex ombiatio of stadard distributios f j : j= p j f j( y, j= p j = ( Correspodig Autor: Rama Faroos, Departmet of Matematis, Sout Tera Bra, Islami Azad Uiversity, Tera, Ira 04

2 Res. J. Appl. Si. Eg. Teol., 4(4: 04-09, 0 I most ases, te f j s are from a parametri family, wit uow parameter j leadig to te parametri mixture model: j= p j f ( yθ j, j= p j = ( were, j, ad is ompat subset of real lie. We a sample y = (y,...,y of mixture distributio is observed, sie it is ot ow tatea observatio is belogto te wat subpopulatio, so obtaied observatios are alled iomplete data. I tis ase lieliood futio ad loglieliood futio of iomplete data is as follows, respetively: ( θ L, P y = j p f y i = j j( iθj = l ( θ, Py = i = log ( j = pjfj( yiθj θ = ( θ,..., θ were, ad p = (p,...,p. Iomplete data a be overted to omplete data by osiderig a set of idiator variables z = (z,, z ad zi = (z i,, z i Terefor it is determiedtat wat subpopulatio iludewi observatios: y i* z ij = :f(y i * j, i =,...,, j =,... (3 Te lieliood futio ad Log-lieliood futio of omplete data wi is usually more useful for doig iferee is as follows, respetively: ( θ,, = i j j j( i j = = θ L P y z p f y ( θ, Pyz, = z logp + log = = f ( y θ l i j ij j j i j Te LRT ad te MLRT: Suppose Y,, Y is a radom sample of mixture distributio: ( θ ( Y( θ pf y P f y Y Zij (4 were j,, j =, ad is ompat subset of real lie. We wis to test: H 0 : p(-p(! = 0 (5 H : p(-p(! 0 were, ull ypotesis meas omogeeity of populatio ad alterative ypotesis meas osistig populatio from two eterogeeous subpopulatio as (. Loglieliood futio a be writte as follow: l = { } ( p, θ, θ = i log pf( yθ + ( p f( yθ If $ ad p $, $ θ, $ θ are maximizatio of te lieliood futio uder ull ad alterative ypotesis, respetively, te te statisti tat obtaied from LRT is as follow: θ 0 ( { log l ( $, $ θ, $ θ log l( $, $ θ, $ θ } R = p p 0 0 Large value of tis statisti leads to rejetig of ull ypotesis. Ce ad Ce (00 sowed tat two soures of o-regularity are exist wi ompliate te asymptoti ull distributio of te LRT. Oe of tese soures is tat te ypotesis lies o te boudary of parameter spae p = 0 or p = ad oter is te mixture model tat is ot idetifiable uder ull model terefore p = 0 or p = ad = are equivalet.for overomig te boudary problem ad o-idetifiability, Ce ad Ce (00 proposed a pealty futio i terms of p as T(p add to LRT statisti su tat: lim T( p =,arg max T( p = 05. p 0 or p [ 0, ] (6 I tis ase pealized Log-lieliood futio is as follows: ( θ θ l ( θ θ Tl p,, = p,, + T( p Use of tis pealty futio lead to te fitted value of p uder te modified lieliood tat is bouded away from 0 or. Ce ad Ce (00 sowed if some regularity oditios old o desity erel te asymptoti ull distributio of MLRT statisti is te mixture of te χ adχ wit te same weigts, i.e., χ χ 0 were, χ 0 is a degeerate distributio wit all its mass at 0. Parameters estimatio based o EM algoritm: EM algoritmis te most popular metod for fidig te estimatios of lieliood maximum. Tis algoritm is osist of two steps E ad M. I E-step oditioal expetatio of omplete data provide observatioal data pj f j( yθ j ad uow parameters, i.e., substitute j= pj f j( yθ j i z ij ad i M-step te value of parameters wi maximize log-lieliood futio of omplete data is ompute. Te E ad M steps repeat util a stop oditio ad reaig to overgee. 05

3 Res. J. Appl. Si. Eg. Teol., 4(4: 04-09, 0 SupposeY,,Y is a radom sample of mixture distributio (4, te log-lieliood futio of omplete data is as follows: l { } { } ( θ, p = i = [ zi log( p + log( f( yiθ ( zi log( p log( f( yiθ + + (7 I E-step, matematial expetatio is omputed as follow: ( zi t ( E zi yi t ( p t = ; θ ( t p f( yiθ ( t ( θ + ( ( θ = ( t p f yi ( t ( t p f yi ( t, i =,,..., (8 I M-step, by puttig t value of (8 i log-lieliood futio of omplete data ad maximizig tis futio wit respet to model parameters, we ave: ad t z p i i ( ( t = = ( t i ( θ = arg max = θ Θ θ { zi t log f( yiθ } {( zi log( ( θ } t f yi ( t ( = arg max i = θ Θ Now MLE is obtaied by iteratio of steps util overgee of algoritm. Estimatio of p depeds o oosig pealty futio. Ce ad Ce (00proposed a pealty futio as follows: T( p = Clog( 4p( p (9 were, C is a positive ostat. Li et al. (008 also used a pealty futio as te followig form: ( T( p = C*log p (0 Tese test futio, (9 ad (0, old i oditio (6. It is simply sow tat by usig pealty futios (9 ad (0, te value of p i M-step from (t+-t iteratio of EM algoritm is obtaied as follow respetively: ad p ( t ( t i=zi + = C+ i ( zi t C i ( = + * = zi t mi{, 05. } < 05. ( t + C* p = i ( zi t i ( zi t i ( = = zi t.., max{ + C*,. = > 05. Te oie of pealty futio: However te limitig distributio of te MLRT does ot deped o te speifi form of T(p, but te preisio of te approximatio ad its power do. A modified test a obtaied from te pealty futio (9 ad solve te problem of LRT. Sie te pealty futio (9 puts too mu pealty o te mixig proportio we it is lose to 0 or tei spite of lear observatio of mixture distributio, MLRT statisti a't rejet ull ypotesis. Hee a more reasoable pealty futio eed su tat power of MLRT ireases eve mixig proportio is lose to 0 or. Terefore te pealty futio (0 as bee proposed tat is olds i te followig relatio: ( p ( p = ( 4p p log log log ( For p = 0.5 te above metioed iequality overt to equality ( p p.. So pealty futios (9 ad (0 are almost equivalet for values of p tat are lose to 0.5, but for values of p tat are lose to 0 or, a osiderable differee exist betwee tese two pealty futio. Te pealty futio tat suggested by Li et al. (008 as advatages ad irease te effiiey of LRT, but tere is o reaso tat it a remove te problem of optimality of tese pealty futios. Optimal proposed pealty futio: I te preset study te followig pealty futio tat is able to overome te disadvatages of te pealty futios (9 ad (0 is proposed: ( gp (, = Clog p, 0 < ( It is lear tat te pealty futios (9 ad (0 are speial ase of pealty futio ( ad tey are obtaied from ( by substitutig = ad =, respetively. How to oose te value of effets o te sape of pealty futio ad osequetly o te obtaied iferees. I lassial approaes te oie of aoter value for leads to omplexity of pealty futio (, tat it also would lead to diffiulty of maximizatio i M- step of EM algoritm. Due to diffiulty of lassial approa i determiig parameters of model ad pealty futio, tese parameters will be estimated i paradigm of Bayesia approa. Terefore at te bigiig suitable prior distributios are osidered for give parameters ad te te estimatio of tese parameters will be omputed as posterior. Rage of parameter i pealty futio ( is iterval (0, ] but beause te values of 06

4 Res. J. Appl. Si. Eg. Teol., 4(4: 04-09, 0 Fig. : Compariso of four pealty futios Table : Mixture models tat a data set is simulated from tem Model p µ µ Table : Mea square error of bayesia ad lassi estimators by usig U(0, as prior distributio for Parameters of model Model Estimator $µ $µ $p $ $ θ ML $ θ ML $ θ ML $ θ ML i iterval (0, may furter improve te power of MLRT i tis study te uiform distributio as bee used as prior distributio for p, i order to oldig impartiality betwee ull ad alterative ypotesis. Te Uiform ad mixture of two uiform distributios ave bee osidered as prior distributio for. I te ext setio we are goig to demostrate MCMC algoritm tat is used i te preset study for estimatio of pealty futio. MCMC algoritm: Suppose (Y, Y is a radom sample from a mixture pn (:,+(-p (:, wit a uow mixig proportio P = (p = p, p = -p, p terefore for te pealized lieliood futio we a write: z ( ( ij µ, Ρ,, = j i µ j i= j= 05.( yi µ j p ( p e ( p j= i, zij= ( y µ ( y µ p ( p e e ( p L z y p f y = were, : = (:, : ad j (for j = ad are te umber of observatios assoiated to te j ompoets. Now, by osiderig te ormal prior distributio N (a, /b, a, Radb > 0, for bot : ad :, prior distributio (a, a for p ad fially by osiderig prior distributio U (0, ad au (0, 0.5+( au(0.5, for,we obtai: b ( µ a ( µ a a a π µ e e p p p π µ, Ρ, yz, L µ, Ρ, z, yπ ( µ, Ρ, (, Ρ, ( = ( b ( ( b e y e ( µ ( µ a e ( y µ b + α + α e ( µ a p ( p ( p Bayesia aalysis of mixture models as may diffiulties beause of its atural omplexity (Diebolt ad Robert, 994. I tis ase posterior distributio dos't as a losed form ee for doig iferee, MCMC algoritm failitates samplig of posterior distributio. 07

5 Res. J. Appl. Si. Eg. Teol., 4(4: 04-09, 0 Table 3: Values ad Mea square error for bayesia ad lassi estimators by usig 0.5U (0, U (0.5, as prior distributio for Parameters of model Model Estimator $µ $µ $p $ $ θ ML -.6( ( ( ( ( ( (0.06 $ θ ML 0.549( ( ( ( ( ( ( $ θ ML ( ( ( ( ( ( ( $ θ ML 0.30( ( ( ( ( ( (0.0 Table 4: Values ad Mea square error for bayesia ad lassi estimators by usig 0.75U(0, U(0.5, as prior distributio for Parameters of model Model Estimator $µ $µ $p $ $ θ ML -.604( ( ( ( ( ( ( $ θ ML -.975( ( ( ( ( ( ( $ θ ML ( ( ( ( ( ( ( $ θ ML ( ( ( ( ( ( (0.068 distributio.typial use of MCMC samplig a oly approximate te target distributio. Two geeratio meaism for produtio su Marov ais are Gibbs ad Metropolis-Hastigs. Sie te Gibbs sampler may fail to esape te attratio of te loal mode (Mari et al., 005 a stadard alterative i.e., Metropolis- Hastigs is used for samplig from posterior distributio. Tis algoritm uses a proposal desity wi depeds o te urret state. I te followig te geeral Metropolis- Hastigs samplig is itrodued: Iitializatio: Coose P (0 ad (0, (0 Stept: For t =,,... ~ ~ ~ (,, ( θ Ρ θ, Ρ, θ ( t, Ρ ( t, ( t C Geerate from q C Compute C Geerateu: u-u [0, ] ~ (, ~ ~, ~ ( t ( t ( t (, ~ ~,,, πθ pyqθ P θ p r = ( t ( t ( P t y q p ( t P ( t ( t ~ π θ,, θ, ~ ~, θ,, ~ if r < u te ( (t, p (t, (t = (, ~ ~ θ p, Else ( (t, p (t, (t = ( (t!, p (t!, (t! were q is proposal distributio tat ofte is osidered as te radom wal Metropolis-Hastigs (Mari et al., 005. SIMULATION STUDY AND DISCUSSION Figure exibits te grap of two pealty futios (9 ad (0 for * = ad pealty futio wit Bayesia estimatio for = ad = It is see tat pealty futio (9 i te poit p = 0.5 is almost smoot ad puts o pealty wile pealty futio (0 ad pealty futio wit = ad = put more less pealty for values tat are lose to 0 or. I fat te purpose of tis setio is to ompare estimatio of parameters usig te pealty futio of (0 ad pealty futio tat obtaied from Bayesia estimatio. Te simulatio experimet was oduted uder ormal erel wit a ow variae. Te mea value for te ull distributio ad alterative distributios is 0 ad te variae for te alterative models is set to be.5 times greater ta te variae uder ull model. Terefore we a osider: E Ho (y = 0, Var H0 (y = E H (Y = p: +(!p: 08

6 Res. J. Appl. Si. Eg. Teol., 4(4: 04-09, 0 ad Var H (y = E H (y!e H(y = p(!p(:!: + Table preset four alterative models uder tese oditios. Te sample size = 00 is simulated from tese models. Table, 3 ad 4 preset ompariso betwee lassial ad bayesia estimatio of mixture model parameters. Te distributios U(0,, 0.75U(0, U(0.5, ad 0.5U(0, U(0.5, are used as te prior distributio for i Table, 3 ad 4, respetively. Te simulatio result sow tat we p is lose to 0 or te Mea Square Error (MSE of bayesia estimator is less ta oe i lassial estimator tat obtaied via EM algoritm. CONCLUSION I Tis study a optimal pealty futio i ompariso to well ow pealty futios for testig omogeeity of mixture models is proposed ad suessfully Bayesia approa is employed to estimatio parameter of pealty futio ad parameters of mixture models. Simulatio experimets sow tat Bayesia approa is better ta te lassial approa bot i estimatio of model parameters ad i determiig of optimal pealty futio i positio tat mixture models goes to o-idetifiability. REFERENCES Ce, J., 998. Pealized lieliood ratio test for fiite mixture models wit multiomial observatios. Ca. J. Stat., 6: Ce, H. ad J. Ce, 00. Te lieliood ratio test for omogeeity i te fiite mixture models. Ca. J. Stat., 9: 0-5. Ce, H., J. Ce ad J.D. Kalbfleis, 004. Testig for a fiite mixture model wit two ompoets. J. R. Stat. So. B., 66: Diebolt, J. ad C.P. Robert, 994. Estimatio of fiite mixture distributios troug Bayesia samplig. J. R. Stat. So. B., 56: Lavie, M. ad M. West, 99. A bayesia metod for lassifiatio ad disrimiatio. Ca. J. Stat., 0: Li, P., J. Ce ad P. Marriott, 008. No-fiite Fiser iformatio ad omo-geeity: Te EM approa. Biometr., 96: Lidsay, B.G. ad E. Basa, 993. Multivariate ormal mixtures: A fast osistet metod of momets. J. Ame. Stat. Ass., 88: Liu, H.B. ad X.M. Sao, 003. Reostrutio of jauary to april mea temperature i te qilig moutai from 789 to 99 usig tree rig roologies. J. Appl. Meteorol. Si., 4: Mari, J.M., K. Megerso ad C. Robert, 005. Bayesia modelig ad iferee o mixture of distributios. Hadboo Stat., 5:

Nonparametric Goodness-of-Fit Tests for Discrete, Grouped or Censored Data 1

Nonparametric Goodness-of-Fit Tests for Discrete, Grouped or Censored Data 1 Noparametri Goodess-of-Fit Tests for Disrete, Grouped or Cesored Data Boris Yu. Lemeshko, Ekateria V. Chimitova ad Stepa S. Kolesikov Novosibirsk State Tehial Uiversity Departmet of Applied Mathematis