Maximum a posteriori estimation for Markov chains based on Gaussian Markov random fields

Transcription

1 Proceda Computer Scence Proceda Computer Scence (1) 1 1 Maxmum a posteror estmaton for Markov chans based on Gaussan Markov random felds H. Wu, F. Noé Free Unversty of Berln, Arnmallee 6, Berln, Germany Abstract In ths paper, we present a Gaussan Markov random feld (GMRF) model for the transton matrces (TMs) of Markov chans (MCs) by assumng the exstence of a neghborhood relatonshp between states, and develop the maxmum a posteror (MAP) estmators under dfferent observaton condtons. Unlke earler work on TM estmaton, our method can make full use of the smlarty between dfferent states to mprove the estmated accuracy, and the estmator can be performed very effcently by solvng a convex programmng problem. In addton, we dscuss the parameter choce of the proposed model, and ntroduce a Monte Carlo cross valdaton (MCCV) method. The numercal smulatons of a dffuson process are employed to show the effectveness of the proposed models and algorthms. Keywords: Markov chan, Gaussan Markov random feld, maxmum a posteror, cross valdaton 1. Introducton Markov chan (MC) models provde a general modelng framework for descrbng state evolutons of stochastc and memoryless systems, and are now mportant and powerful tools for an enormous range of mathematcal applcatons, ncludng scence, economcs, and engneerng. Here we only focus on the fnte dscrete-tme homogeneous MC model, whch s one of the most common MC models, and whose dynamcs can be smply characterzed by a transton matrx (TM) T = [ T R n n wth T the transton probablty from the -th state to the -th state. In most applcatons, the man problem s to estmate the transton probabltes from observed data. In the past few decades, a lot of dfferent technques have been proposed to estmated the TMs. Many early researches devoted to the least-square (LS) approaches [1 3, for MC models Correspondng author Emal addresses: hwu@zedat.fu-berln.de (H. Wu), frank.noe@fu-berln.de (F. Noé)

2 H. Wu, F. Noé / Proceda Computer Scence (1) can be transformed to lnear stochastc systems wth zero-mean nose. However, the conventonal LS estmators may volate the nonnegatve constrants on TMs. Thus, some restrcted LS methods [4 6 based on constraned quadratc programmng algorthms were developed to avod ths problem. Some researchers [2, 5 suggested utlzng the weghted LS and weghted restrcted methods to solve the problem of heteroscedastcty. By now, the best known and most popular estmaton method of MC models s maxmum lkelhood (ML) estmator whch was proposed n [7, for t s consstent and asymptotcally normally dstrbuted as the sample sze ncreases [8, and can be effcently calculated by countng transton pars. Some experments show ML estmator s superor to the LS estmators [9. Moreover, the ML method can be appled to revsersble TM estmaton for some physcal and chemcal processes [1. Recently, the Bayesan approach [11, 12 to TM estmaton has receved a good deal of attenton. In ths approach, an unknown TM s assumed to be a realzaton of some pror model, and the posteror dstrbuton gven observed data can be obtaned by Bayes rule. Comparng to the non-bayesan methods, the Bayesan estmator can provde much more nformaton than a sngle pont estmate, and s more relable for small sze data set f the pror model s approprately desgned. The most commonly used pror dstrbuton s the matrx beta dstrbuton wth densty p (T Θ), T θ 1. It s a conugate pror and can be easly analyzed and effcently sampled snce each row of T follows the Drchlet dstrbuton. In some applcatons, Θ = 1 and Θ = are recommended, because p (T Θ) s equvalent to the unform dstrbuton when Θ = 1 [13, and Θ = makes the posteror mean of the TM dentcal to the ML estmate [14. The matrx Θ can also be optmzed by usng the emprcal Bayes approach [15. The matrx beta pror dstrbuton based Bayesan estmaton of revsersble TM was nvestgated n [13. The shortcomng of the matrx beta pror s that t does not take nto account possble correlatons between dfferent rows of the transton matrx. Assoudou and Essebbar [16, 17 proposed the Jeffreys pror (a non-nformatve pror) model for TMs to overcome ths problem, and no extra parameter s requred n ths model. However, the Jeffreys pror dstrbuton s too complcated for dervng the Bayesan estmator, and can only be appled to MC models wth very few states n practce. The maor obectve of ths paper s to propose a new pror model for MCs based on the Gaussan Markov random fled (GMRF). The GMRF [18 21 model s a specfc Gaussan feld model, and frequently used n spatal statstcs and mage processng, whch constructs a global dstrbuton of a spatal functon by consderng the local correlatons between ponts or regons. In ths paper, we assume that the state space of the MC has neghborhood structure and the adacent states have smlar transton behavors. Ths assumpton generally holds for the grd based approxmate models of contnuous space MCs, and the case that the state space has a dstance metrc. A GMRF pror model of TMs s then desgned accordng to the assumpton, and the correspondng maxmum a posteror (MAP) estmator s developed. In comparson wth the exstng models, the new pror model s able to utlze the smlarty relatonshp of states better. And there s only one extra parameter s requred, whch can be selected by the cross valdaton (CV) method. Moreover the estmaton problem wth nosy data s consdered, and the expectaton maxmzaton (EM) algorthm s used to get the MAP estmate. 2. Background 2.1. Gaussan Markov random felds Let G = (V, E) be an undrected graph wthout loop edges, where V s the set of vertces and E V V s the edge set. And vertces u, v V are sad to be adacent ff (u, v) E, whch s

3 H. Wu, F. Noé / Proceda Computer Scence (1) denoted by u v. It s clear that u, v V, v v and u v v u. A Gaussan Markov random feld (GMRF) Y on G s a Gaussan stochastc functon that assgns to each vertex v a real number Y (v). Here we only ntroduce the wdely used ntrnsc GMRF model [19, 22, whch s often specfed through the followng dstrbuton p GMRF (y σ) exp 1 ( ) 2 Y (u) Y (v) 2σ 2 d 2 (u, v) (1) where y = {Y (v) v V}, σ s a parameter, and d (, ) denotes a dstance measure between vertces. It s clear that the neghborng data ponts are desred to have the smlar values Markov chans We consder a tme-homogeneous Marv chan (MC) {x t t } on the fnte state space S = {s 1,..., s n }. Its probablty model can be descrbed by a transton matrx (TM) T = [ T R n n whose entres are gven by T = p ( ) x t+1 = s x t = s (2) where u v T = 1, T > (3) Here we defne Ω n = {T T R n n s a stochasc matrx}, whch s a convex set. And the probablty dstrbuton of the fnte-length state sequence {x, x 1,..., x m } gven T can be expressed as p (x :m T) = T C (4) where entres of count matrx C = [ C are numbers of observed transton pars wth, C = { (xt, x t+1 ) x t = s, x t+1 = s, t m 1 } (5) 3. GMRF Based MC Model Estmaton 3.1. GMRF pror Gven an MC state space S = {s 1,..., s n }, the purpose of ths subsecton s to provde a GMRF model based pror dstrbuton for the TM T = [ T. Assumng a neghborhood structure on the state space, we construct a neghborhood relaton between the transton pars as ( s, s ) (sk, s l ) ( s, s ) ( sk {s k }) ( s l {s l }) \ {(s k, s l )} (6) Then the unknown matrx T can be modeled by GMRF wth dstrbuton p GMRF (T σ) exp ( u (T, σ)) (7) where u (T, σ) = 1 2σ 2 (s,s ) (s k,s l ) T T kl d 2 kl 2 (8)

4 H. Wu, F. Noé / Proceda Computer Scence (1) d kl s the dstance between ( s, s ) and (sk, s l ), and here defned as d sk = d 2 (s, s k ) + d 2 ( s, s l ) (9) However, the realzaton of dstrbuton (7) does not satsfy (3) n the general case. Therefore we modfy the pror dstrbuton as 1 p GMC (T σ) = p GMRF (T σ, T Ω n ) z(σ) exp ( u (T, σ)), = T Ω n (1), T Ω n where z (σ) = exp ( u (T, σ)) dt Ω n (11) 3.2. MAP estmaton The maxmum a posteror (MAP) estmate of the TM T of an MC from observed data {x,..., x t } wth count matrx C = [ C s gven by { } ˆT = arg max log p (C T) + log p (T) T Usng the proposed GMRF pror model and assumng the parameter σ s known, (12) s equvalent to the followng optmzaton problem ˆT (σ) = arg mn T Ω n C log T + u (T, σ) (13), It s a convex problem and can be solved wthout any spurous local mnma. In ths paper, we perform the optmzaton by the dagonalzed Newton (DN) method (see [23 for detals) Choce of σ We now consder the case that σ s unknown. Motvated by the above analyss, t seems reasonable to ontly estmate T and σ by MAP method. But t s ntractable to compute the ont pror dstrbuton p(t, σ) = p(σ)p GMC (T σ) for z (σ) has no closed form. So here we use cross-valdaton (CV) approach to select the value of σ, and adopt the Monte Carlo cross-valdaton (MCCV) method proposed n [24. The MCCV of a σ s conducted by the followng steps: Step 1. Partton the set of observed state transton pars randomly nto tran and test subsets, where the tran subset s a fracton β (typcally.5) of the overall set, and the correspondng count matrces are denoted by C tran k and C test k. Step 2. Calculate and the predctve log-lkelhood (12) ˆT k (σ) = arg max T Ω n { log p ( C tran k T ) u (T, σ) } (14) CV k (σ) = log p ( C test k ˆT k (σ) ) (15)

5 H. Wu, F. Noé / Proceda Computer Scence (1) Step 3. Repeat the above steps for k = 1,..., K and select wth CV (σ) = k CV k (σ) /K. σ = arg max CV (σ) (16) σ It can be seen from (15) that CV k (σ) f the (, )-th entry of C test k s postve and that of ˆT k (σ) convergences to. In order to avod the possble sngularty, we approxmate the logarthmc functon as log ( ) ( ) 1 ( T PLη T = T η η 1) (17) when calculatng CV k (σ), where η (, 1) s a small number (η = n ths paper). It s easy to prove that lm η PL η (x) = log(x) for x >. 4. Estmaton wth Stochastc Observatons In ths secton, we wll take nto account that the actual state transtons are unknown, and only stochastc observatons o t x t p (o t x t ) (18) for t =,..., m are avalable. In ths case, the MAP estmator of the TM wth pror parameter σ can be expressed by { } ˆT (σ) = arg max log p (O T) u (T, σ) (19) T Ω n where O = {o,..., o m }, and computed wth the expectaton maxmzaton (EM) algorthm [25 consstng of the followng steps: Step 1. Choose an ntal T () Ω n and let k =. Step 2. Compute the functonal Q ( T T (k)) = E [ log (C (X) T) u (T, σ) T (k), O = C log T u (T, σ) (), where X = {x,..., x m }, C (X) = [ C (X) denotes the count matrx of X, and C = [ C = E [ C (X) T (k), O (21) Step 3. Fnd T (k+1) whch maxmzes the functon Q ( T T (k)) as T (k+1) = arg mn T Ω n C log T + u (T, σ), (22) Step 4. Termnate f ( ( ) log p O T (k+1) u ( T (k+1), σ )) ( log p ( O T (k)) u ( T (k), σ )) s small enough.

6 Step 5. Let k = k + 1 and go to Step 2. H. Wu, F. Noé / Proceda Computer Scence (1) Note that (22) has the same form as (13) wth C for any,, so (22) s a convex optmzaton problem and can be solved by the DN algorthm too. Further, n a smlar manner to Secton 3.3, the value of σ can be desgned through the MCCV algorthm. Due to space lmtatons, we omt detals here. 5. Smulatons 5.1. Brownan dynamcs model In ths secton, the estmaton method proposed n ths paper wll be appled to a Brownan dynamcs (BD) model, whch s descrbed as dr = f (r) dt + ρdw (23) where ρ = 1.4, W s a standard Brownan moton, f (r) = dv (r) /dr and V (r) s the potental functon (see Fg. 1) gven by 111.1r r r , r < r r r ,.75 r < 1 V (r) = r r 2 (24) 595.6r r < r r r , 1.25 < r Dscretzng the moton equaton (23) wth tme step t = 1 3 and decomposng the state space {r r 2} nto n = cells S = {s 1,..., s n } wth s = 2 1 n, we can get the grd based approxmate MC model p ( ( ) s x k+1 = s x k = s exp s + t f (s ) ) 2 2ρ 2 t (25) The correspondng TM T = [ T s shown n Fg. 2. Furthermore, the neghborhood structure on S s here defned by s = {s 1, s +1 } S wth dstance measure d ( s, s ) = TM estmaton Here, we wll use the MAP method presented n Secton 3 to estmate the TM T based on a realzaton {r (t) t 3} of (23) (see Fg. 3), and compare t wth the ML method [11. Fg. 4 plots the MCCV results of σ and the optmal σ = The comparsons of the dfferent estmators are based on the Kullback-Lebler (KL) dvergence rate metrc [26 defned as KLR ( ˆT T ) = ˆπ ˆT log ˆT T (26) where ˆπ = [ˆπ denotes the statonary dstrbuton of TM ˆT = [ ˆT. It can measure the dstances between Markov chans on the same state space.

7 H. Wu, F. Noé / Proceda Computer Scence (1) V r Fgure 1: Potental Functon Fgure 2: T r t Fgure 3: r (t) CV σ 1 1 Fgure 4: CV (σ) Fg. 5 shows the estmaton results of the proposed MAP method wth dfferent σ and ML method. Clearly, the ML method fals to estmate the values T wth [1, 7 [34, 44 [91, for there are few x k are sampled wthn the ranges. The GMRF pror based MAP estmator overcome ths problem by nterpolatng from the other T accordng to the GMRF model. Moreover, as observed from the fgures, the parameter σ determnes the overall smoothness of the estmated TM, and the MCCV approach can provde an approprate value of σ TM estmaton wth nosy data In ths subsecton, we study the performance of our proposed algorthms for estmatng T from nosy observatons o (t) r (t) N ( r (t), v 2) (27) wth v =. The MAP estmator wth GMRF pror n Secton 4 wll now be compared to the ML estmator mplemented usng Baum-Welch algorthm [27. The MCCV results are shown n Fg. 6 and the optmal σ = 947.

8 H. Wu, F. Noé / Proceda Computer Scence (1) (a) MAP (σ = σ ), KLR ( ˆT T ) =.872 (b) MAP (σ = 3σ ), KLR ( ˆT T ) = (c) MAP (σ = σ /3), KLR ( ˆT T ) =.729 (d) ML, KLR ( ˆT T ) =.2316 Fgure 5: ˆT 1 CV σ Fgure 6: CV (σ)

9 H. Wu, F. Noé / Proceda Computer Scence (1) (a) MAP (σ = σ ), KLR ( ˆT T ) = 7 (b) MAP (σ = 3σ ), KLR ( ˆT T ) = (c) MAP (σ =.6159), KLR ( ˆT T ) =.492 (d) ML, KLR ( ˆT T ) = Fgure 7: ˆT In Fg. 7, we show plots for ˆT obtaned usng our MAP estmator wth σ = σ, 3σ and.6159 (σ n Subsecton 5.2) and ML estmator. As can be seen from Fg. 7d, the ML estmator exhbts strong overfttng. Wth comparson to ML method, the proposed MAP estmator avods overfttng by the regularzaton term u (T, σ), whch penalzes excessvely large value of T. Note that here σ = 947 s bgger than the σ =.6159 n the prevous subsecton, whch may be related to nosy observaton and nsuffcent sample sze. From Fgs. 5a and 7c, we can see that the observaton nose makes ˆT obtaned from {o,..., o m } smoother than that drectly estmated by states {x,..., x m }. Therefore the MCCV approach wll select a bgger σ to get a sutably smooth ˆT and maxmze the predctve lkelhood. 6. Conclusons The GMRF model of TMs dscussed n ths paper provdes a general and flexble framework for analyzng and estmatng MCs wth smooth TMs by extendng the neghborhood relatonshp between states to that between transton pars. Ths model s helpful to mprove the robustness and accuracy of estmators n many practcal cases, especally when the sample sze

10 H. Wu, F. Noé / Proceda Computer Scence (1) s small wth respect to the sze of state space. And the convex form of GMRF model benefts the numercal calculaton. The parameter choce s a dffcult problem for our model, but t can be solved by CV methods snce there s only one undetermned parameter. References [1 G. Mller, Fnte Markov processes n psychology, Psychometrka 17 (2) (1952) [2 A. Madansky, Least squares estmaton n fnte Markov processes, Psychometrka 24 (2) (1959) [3 L. Telser, Least-squares estmates of transton probabltes, Measurement n Economcs (1963) [4 T. Lee, G. Judge, T. Takayama, On estmatng the transton probabltes of a Markov process, Journal of Farm Economcs 47 (3) (1965) [5 H. Thel, G. Rey, A quadratc programmng approach to the estmaton of transton probabltes, Management Scence 12 (9) (1966) [6 G. Judge, T. Takayama, Inequalty restrctons n regresson analyss, Journal of the Amercan Statstcal Assocaton 61 (313) (1966) [7 T. Anderson, L. Goodman, Statstcal nference about Markov chans, The Annals of Mathematcal Statstcs 28 (1) (1957) [8 M. Kendall, A. Stuart, The advanced theory of statstcs, Vol. 2, Charles Grffn, London, [9 T. Lee, G. Judge, A. Zellner, Estmatng the parameters of the Markov probablty model from aggregate tme seres data, North-Holland, 197. [1 G. Bowman, K. Beauchamp, G. Boxer, V. Pande, Progress and challenges n the automated constructon of Markov state models for full proten systems, The Journal of Chemcal Physcs 131 (9) [11 T. Lee, G. Judge, A. Zellner, Maxmum lkelhood and Bayesan estmaton of transton probabltes, Journal of the Amercan Statstcal Assocaton 63 (324) (1968) [12 J. Martn, Bayesan decson problems and Markov chans, Wley New York, [13 F. Noé, Probablty dstrbutons of molecular observables computed from Markov models, The Journal of Chemcal Physcs 128 (8) [14 C. Fuh, T. Fan, A Bayesan bootstrap for fnte state Markov chans, Statstca Snca 7 (1997) 5 1. [15 M. Meshkan, L. Bllard, Emprcal Bayes estmators for a fnte Markov chan, Bometrka 79 (1) (1992) [16 S. Assoudou, B. Essebbar, A Bayesan Model for Markov Chans va Jeffrey s Pror, Communcatons n Statstcs 32 (11). [17 S. Assoudou, B. Essebbar, A Bayesan model for bnary Markov chans, Internatonal Journal of Mathematcs and Mathematcal Scences 4 (8) (4) [18 J. Besag, Spatal nteracton and the statstcal analyss of lattce systems, Journal of the Royal Statstcal Socety. Seres B (Methodologcal) 36 (2) (1974) [19 J. Besag, C. Kooperberg, On condtonal and ntrnsc autoregressons, Bometrka 82 (4) (1995) [ H. Rue, L. Held, Gaussan Markov random felds: theory and applcatons, Chapman & Hall, 5. [21 S. L, Markov random feld modelng n mage analyss, Sprnger, 9. [22 S. Saqub, C. Bouman, K. Sauer, ML parameter estmaton for Markov random felds wth applcatons to Bayesan tomography, IEEE Transactons on Image Processng 7 (7) (1998) [23 T. Larsson, M. Patrksson, C. Rydergren, An effcent soluton method for the stochastc transportaton problem, n: Optmzaton Methods for Analyss of Transportaton Networks, Lnköpng Studes n Scence and Technology, no. 72, Department of Mathematcs, Lnköpng Unversty, [24 J. Shao, Lnear model selecton by cross-valdaton, Journal of the Amercan Statstcal Assocaton (1993) [25 A. Dempster, N. Lard, D. Rubn, et al., Maxmum lkelhood from ncomplete data va the EM algorthm, Journal of the Royal Statstcal Socety. Seres B (Methodologcal) 39 (1) (1977) [26 Z. Rached, F. Alaa, L. Campbell, The Kullback-Lebler dvergence rate between Markov sources, IEEE Transactons on Informaton Theory 5 (5) (4) [27 L. Welch, Hdden Markov models and the Baum-Welch algorthm, IEEE Informaton Theory Socety Newsletter 53 (4) (3) 1 1.