Spike-and-Slab Dirichlet Process Mixture Models

Transcription

1 Ope oural of Sascs 5-58 hp://dxdoorg/436/os566 Publshed Ole December (hp://wwwscrporg/oural/os) Spke-ad-Slab Drchle Process Mxure Models Ka Cu Wesha Cu Deparme of Sascal Scece Duke Uversy Durham USA School of Scece ad Iformao Qgdao Agrculural Uversy Qgdao Cha Emal: Receved Ocober 6 ; revsed November 8 ; acceped November 3 ABSTRACT I hs paper Spke-ad-Slab Drchle Process (SS-DP) prors are roduced ad dscussed for o-paramerc Bayesa modelg ad ferece especally he mxure models coex Specfyg a spke-ad-slab base measure for DP prors combes he mers of Drchle process ad spke-ad-slab prors ad serves as a flexble approach Bayesa model seleco ad averagg Compuaoally Bayesa Expecao Maxmzao (BEM) s ulzed o oba MAP esmaes Two smulaed examples mxure modelg ad me seres aalyss coexs demosrae he models ad compuaoal mehodology Keywords: Spke ad Slab; Drchle Process; Bayesa Expecao-Maxmzao (BEM); Mxure Iroduco Drchle Process (DP) prors are used across a wde varey of applcaos of Bayesa aalyss cludg Bayesa model valdao desy esmao ad mxure modelg Specfcally herarchcal mxure models he oparamerc aure of he Drchle process raslaes o mxure models wh a couably fe umber of compoes ad s used o specfy lae paers of heerogeey I allows for uceray abou he umber of mxure compoes ad compoe specfc parameers O he oher had f he ypcal Drchle process pror wh a couous base measure G s used as he pror for he dsrbuo of parameer hs requres a kow paramerc model ad fxed parameer space However hs s ofe o he case For example mxure models we ofe deal wh cases wh model uceray where he parameer space of oe mxure compoe mgh be smaller ha ha of aoher I hese cases a DP prors wh couous base measure are boh heorecally ad phlosophcally o rgh o use Isead we wa o furher egrae model seleco echques o he model o allow for more flexbly Bayesa spke-ad-slab approaches o parameer seleco have bee proposed [] ad used as pror dsrbuos he Bayesa model seleco ad averagg leraure [3] Spke-ad-slab dsrbuos are mxures of wo dsrbuos: he spke refers o a po mass dsrbuo (say a zero) ad he oher dsrbuo s a couous dsrbuo for he parameer f s o zero Recely he use of spke-ad-slab dsrbuo combed wh Drchle process pror has bee proposed mul- ple hypohess esg [45] Ths approach s show o readly accommodae sharp hypohess whch cao be esed usg regular Drchle process wh couous base measure (due o he fac ha sharp hypoheses wll have zero poseror probably) [6] I hs sudy we propose he use of Spke-ad-Slab Drchle process (SS-DP) prors especally mxure models The uceray abou model ad parameer space s corporaed he model by modelg he ukow dsrbuos of parameers wh SS-DP prors whch allow degeerao of cera parameers We show ha SS-DP mxure models are flexble models allowg boh ukow umber of compoes ad dffere compoe-specfc parameer spaces Models ad Mehods Spke-ad-Slab Drchle Process Mxure Models Cosder a Drchle Process Mxure (DPM) model wh fe umber of compoes of he form y π f y Hece y s dsrbued as a mxure of dsrbuos wh he same paramerc form f ad parameer space bu dffere parameer values Assumg ha all he parameers are from he coues dsrbuo G he mxure model ca be expressed herarchcally as a DPM model of he form: y f y G G G DP G () Copyrgh ScRes

2 K CUI W S CUI 53 where DP G s he Drchle Process wh couous base measure G ad spread [78] ad G s a radom dsrbuo draw from he Drchle process The SS-DP mxure model we proposed here s a exeso of he DPM model whch allows dffere parameer spaces of f especally wh some of he parameers degeeraed Ths flexbly s acheved by leg he base measure G be spke-ad-slab wh a mxure of po mass (ypcally a ) ad he oher (couous) dsrbuo sead Hece he SS-DP mxure Model s: y f y G G G DPG () G w H w Bea r s where s a Drac dela fuco H s a couous measure ad w s he mxg wegh wh a Bea pror dsrbuo Clearly by allowg he values of parameers o be exacly SS-DP mxure models smulaeously corporae he uceray of compoe-specfc parameer space I s worh og ha may applcaos hs degeerao of specfc parameer(s) (or a exac value of a parameer) defes mpora regme swchg or phase chage ha cao be modeled by radoal DPM models wh couous base measure Examples of such cases are show he smulao sudy seco Trucaed SS-DP Mxure Models I pracce for problems ha requre ukow umber of mxure compoes ad uceray of compoe-specfc parameer space we ca use he sck-breakg represeao of Drchle process [9] o rucae he SS-DP mxure models a a maxmum umber of compoes To llusrae he followg example of rucaed SS-DP mxure models s show whch wll also be he basc model for our smulao sudes Assume ha y s dsrbued as a mxure of K ormal dsrbuos ad he compoe meas are lear combaos of covaraes x ad x where he regresso coeffces of x may be zero some mxure compoes Wh he umber of compoes K ukow we model y as comg from a ormal mxure wh maxmum umber of compoes K : π x x y N Furhermore o model he possble degeerao of SS-DP prors wh a mxure of po mass a ad a ormal dsrbuo are placed for whle ypcal DP prors wh ormal base measures are used for as descrbed () ad () The model ad prors ca be wre uder sckbreakg represeao [9] as follows: y π N x x N m k w wn m π V π V V SS where V Bea K Gamma w Bea r s Gamma e f Gve he model ad prors Bayesa poseror aalyss of parameers provdes ferece of boh model parameers ad model uceray 3 Compuaoal Mehods For ferece ad poseror aalyss we propose o use Bayesa Expecao Maxmzao (BEM) o oba Maxmum A Poseror (MAP) esmaes of all he parameers To help BEM vs he (local) maxmum of he regos wh hgh mass Markov Cha Moe Carlo (MCMC) wll be coduced before BEM ad mulple sarg pos of BEM wll be chose from MCMC samples I comparso label swchg s expeced o be a cocer o erpre he MCMC resuls f sead Gbbs sampler s used whch s o a recommeded compuaoal mehod The deals of BEM are show as follows: Le z z z deoe he lae compoe dcaors where z f observao s from he h compoe Ad le π P z y deoe he assgme probably The he BEM s o maxmze he poseror log lkelhood ad eraed bewee he wo seps: E-sep: Calculae Q E log f z y z y f z y f z p E log log log z y f y z p π log log π log where deoes all he parameers s he para- (3) Copyrgh ScRes

3 54 K CUI W S CUI meer learg a he π h erao P z y ad p s he o pror ds- rbuo of parameers M-sep: Fd he whch maxmzes =arg max Q Q : The dervaos of he M-seps gve our SS-DP mxure models ad prors show (3) are lsed deal Appedx 4 Smulao Sudy for SS-DP Mxure Model Daa are geeraed from a mxure of 3 ormal dsr- π 3 buos y N x as follows where he compoe mea s depede of x oe of he compoes f f y y N x4 f y N x 3 3 N4 ad f y= f y 3 We assume ha he umber of ormal compoes are ukow a pror ad se he maxmum umber of compoes o be Models ad prors are used as show (3) ad MAP esmaes of all ukow quaes are obaed usg BEM Several aspecs of poseror fe- rece are checked o gauge he performace of he model ad algorhms: The MAP esmaes of he model gve 4 mxure compoes he parameers of whch are show Table Compoes -3 recover he hree compoes of he smulaed daa very well Zero of he hrd compoe s also ferred exacly The model defes oe addoal rval (%) fourh compoe The ferece of oe more umber of compoes s due o he fac ha we obaed he maxmum lkelhood po esmae whch shows small devao from he rue umber of compoes Furher f we do model-based cluserg based o he poseror probably of daa belogg o a specfc compoe he classfcao resuls are show Fgure whch show grea smlary wh he smulaed compoes 5 Coegraed Tme Seres Aalyss va SS-DP Mxure Models Oe of he movag applcaos for hs paper arses me seres aalyss where here s lkely o be regme swchg bewee a mxure of mulple saoary ad o-saoary saes whle he umber of he saes s ukow I pracce sascal arbragers ca ake advaage of he separao of regme-specfc formao oe of he examples of whch s par radg [] a Table MAP esmaes of parameers va SS-DP mxure model The esmaes compared o he rue values (show pareheses) show good ferece Compoes w (Mxure Wegh) 94 () () 54 (5) 336% (333%) 98 ( ) 86 () 967 () 36% (333%) (4) 9 () 36% (333%) (NA) 546 (NA) (NA) % () Fgure Model-based poseror cluserg of he daa compared wh rue cluserg The lef fgure shows he rue 3 compoes of he smulaed daa he rgh oe shows he model-based clusered daa o he 4 compoes defed usg SS-DP mxure models represeed by dffere colors Copyr gh ScRes

4 K CUI W S CUI 55 popular ye smple shor-erm speculao sraegy The dea s of par radg ha: fd wo secures whose prces have bee hsorcally movg ogeher So whe he spread bewee hem wdes we shor he wer ad buy he loser Ad f we beleve ha he hsory would repea self prces wll coverge aga ad he arbrager wll prof Ths movg-ogeher relaoshp bewee wo me seres s called co-egrao Mahemacally f wo me seres U ad V are co-egraed he here exss a umber k such ha Y U V s a saoary me seres However alhough here have bee may sascal sudes o fd co-egraed me seres s somemes very hard o fd such co-egrag relaoshp oe ma reaso of whch s he exsece of srucural breaks or regme swchg I a aemp o solve he problem we propose o use he SS-DP mxure models for modelg possble regme-swchg co-egraed me seres aalyss whch allows he co-egrao relaoshp o be swched o ad off bewee mulple regmes 5 Error Correco Model ad Coegrao Suppose we have wo I() saoary me seres U ad V ad Y U V ( s kow ypcally people propose a ad he es he saoary propery of Y ) If Y s I() saoary he we say me seres U ad V are co-egraed The way o es f Y s I() saoary s usg he full Error Correco Model (ECM) whch s o es he lear model: m Y Y Y N The he ECM model ells ha he me seres Y s o-saoary f ad oly f Gve our movao ha he me seres s lkely o swch bewee a mxure of mulple saoary ad osaoary regmes whle he possble umber of regmes are a pror ukow SS-DP mxure model combed wh he ECM model (SS-DP ECM mxure model) s a perfec ool here o explore he regme swchg ad regme-specfc parameers Also poseror ferece of a for a specfc regme has he sraghforward erpreao ha he me seres s o-saoary wh he regme 5 Smulaed Coegrao Aalyss To esfy he performace of SS-DP ECM mxure model for coegraed me seres aalyss a mxure of saoary ad o-saoary me seres s cosruced whch swches bewee wo saoary AR() processes ad oe o-saoary AR() process We assume ha he umber of compoes are a pror ukow Ad hs me seres mmcs he mes seres geeraed by a co-egraed par The wo saoary AR() model ad oe o-saoary AR() model ad her correspodg Error Correco Model (ECM) are show (he (o-)saoary propery ca be easly esed by he u roo es []): Sae : y 36y 8 y ECM : y 3 68y 8 y ; y * * # # y Sae : y 5y 5 y ECM : y y 5 y ; Sae : y 4 7y 3 y ECM : y 4 3 The smulaed me seres of legh 3 s show Fgure wh he probably of smulag from he hree saes beg 4% 5% ad % respecvely We appled SS-DP mxure model combed wh he ECM model o model he regme-swchg me seres as a mxure of saoary ad o-saoary me seres The model ad sck-breakg represeao of he SS-DP prors are show as follows: m Y π Y Y π V π V V K V Bea = Gamma e f N m wn m w p w r s Gamma v SS We se maxmum umber of compoes o be rucaed a ad used BEM o oba MAP esmaes of all ukow quaes The resuls show correc defcao of 3 mxure compoes (regmes) Sce mos cases we are especally eresed defyg ad excludg me pos he o-saoary regme Fgure shows ha he poseror ferred o-saoary me pos (gree dos) recover he rue o-saoary me pos (red dos) well Good MAP esmaes of he regme-specfc parameers are also obaed as show Table 53 Model-Based Decso Makg SS-DP mxure models are especally mpora hs case because he characerzao of saoary regme s Copyrgh ScRes

5 56 K CUI W S CUI Table MAP esmaes of SS-DP ECM mxure models for he regme-swchg me seres aalyss MAP esmaes are compared o he rue values (show pareheses) ad show good ferece Compoes Classfcao Wegh π 399 (3) 88 (8) 96 () Saoary (Saoary) 44% (4%) 43 () 54 (5) 3 ( ) Saoary (Saoary) 5% (5%) (4) 33 ( 3) () No-Saoary (No-Saoary) 76% (%) abou he compoe-specfc parameer space Gve he wde use of mxure models a varey of felds cludg bologcal scece mache learg ecoomercs ad face more flexble whle compuaoally effce models are defely worh explorg Fgure Characerzao of o-saoary saes va SS-DP ECM mxure models The smulaed me seres Y s show The red dos mark he rue me pos whe he me seres s he o-saoary regme ad he gree dos are ferred me pos a whch Y s he osaoary regme from SS-DP ECM mxure model he key for decso makg To llusrae hs we use he par radg coex whch oly f he me seres Y geeraed by he par of U ad V s saoary regme ca people do par radg based o he radoal rule ha you ope a log-shor poso whe he par prces have dverged by more ha wo hsorcal sadard devaos Ad you uwd he poso whe reurs o hsorcal mea Oherwse f Y s wh he o-saoary regme a a specfc me po o aco should be ake Compared o he radoal co-egrao es SS-DP ECM mxure model helps make more reasoable decso sce s oly based o seleced saoary regmes I comparso he radoal co-egrao ess ca oly es he whole me seres whch mos cases are bldly pooled daa cossg of boh saoary ad o-saoary saes 6 Cocludg Remarks I hs paper we roduced ad suded spke-ad-slab Drchle process prors especally he mxure models framework whch add more flexbly o he radoal oparamerc mxure models by allowg uceray REFERENCES [] F Geweke Varable Seleco ad Model Comparso Regresso Workg Papers 539 Federal Reserve Bak of Meapols Meapols 994 [] T Mchell ad Beauchamp Bayesa Varable Seleco Lear Regresso oural of he Amerca Sascal Assocao Vol 83 No pp 3-3 [3] E I George ad R E McCulloch Varable Seleco va Gbbs Samplg oural of he Amerca Sascal Assocao Vol 88 No pp [4] D B Dahl ad M A Newo Mulple Hypohess Tesg by Cluserg Treame Effecs oural of he Amerca Sascal Assocao Vol No pp [5] D B Dahl Q Mo ad M Vaucc Smulaeous Iferece for Mulple Tesg ad Cluserg va a Drchle Process Mxure Model Sascal Modellg Vol 8 No 8 pp 3-9 [6] S Km D B Dahl ad M Vaucc Spked Drchle Process Pror for Bayesa Mulple Hypohess Tesg Radom Effecs Models Bayesa Aalyss Vol 4 No 4 9 pp [7] T S Ferguso A Bayesa Aalyss of Some No- Paramerc Problems The Aals of Sascs Vol No 973 pp 9-3 [8] T S Ferguso Pror Dsrbuos o Spaces of Probably Measures Aals of Sascs Vol No pp [9] Sehurama A Cosrucve Defo of Drchle Prors Sasca Sca Vol pp [] E P Cha Quaave Tradg oh Wley ad Sos Hoboke 8 [] D A Dckey ad W A Fuller Dsrbuo of he Esmaors for Auoregressve Tme Seres wh a U Roo oural of he Amerca Sascal Assocao Vol 74 No pp Copyrgh ScRes

6 K CUI W S CUI 57 Appedx Bayesa Expecao Maxmzao (BEM) The dervaos for BEM are based o he model for smulao Sudy : y Nπ x ad prors specfed Equao (3) M-sep for he SS-DP mxure model for smulao Sudy Q N y x p π log ; log π log π log log log log k y x V Vk p The soluos o he frs-order dffereal equaos are: π k k V ; π V π ; V V π f e log V However whe we look a he frs order dffereal equaos for w ad hey are complcaed due o he exsece of dela fuco (o show here) I order o fd he opmal values w ad The followg wo-sep rck ca be appled ) Spl o ad wo cases For each case fd he values of parameers ha are (local) maxmum Noe: For case here mgh be o (local) maxmum ha ca be foud (or very hard o drecly cofrm ha a maxmum s foud) bu we ca deed use he referece soluos provded below ( w ad ) o coue o fd he global maxmum ) Compare he soluos of he wo cases by log poseror desy he oes ha gve he bgger log poseror desy wll be se o be w ad Dervaos for w If : The soluos o w ad all have close form whch are: m x y x m y x π π π π π x π π x m y x m x y π r w r s π π π π π x π π x π y x m m SS Copyrgh ScRes

7 58 K CUI W S CUI If : We ca mmedaely ge he soluo o whch s: π y m π For w ad apparely he soluos o he wo Q Q equaos (whch are ad ) do o w have close form Numercal Mehods wll be appled o ge accurae eough approxmae soluos π y m π T S π S πtr s S π S πt Sm S πt π y m SS where w T S m ; exp w Ad he las wo equaos are used o eravely fd he approxmae soluo o ad w whch are: S π S πt Sm π y m SS = S πt w s S π S πt r Compare soluos w wh w he oe ha gves larger log poseror desy s se o w BEM for smulao Sudy wh SS-DP ECM mxure model s obaed smlarly Copyrgh ScRes