Variational Markov chain Monte Carlo for Bayesian smoothing of non-linear diffusions

Transcription

1 Compu Sa DOI.7/s ORIGINAL PAPER Variaional Markov chain Mone Carlo for Bayesian smoohing of non-linear diffusions Yuan Shen Dan Cornford Manfred Opper Cedric Archambeau Received: 4 December 9 / Acceped: 6 February Springer-Verlag Absrac In his paper we develop se of novel Markov chain Mone Carlo algorihms for Bayesian smoohing of parially observed non-linear diffusion processes. The sampling algorihms developed herein use a deerminisic approimaion o he poserior disribuion over pahs as he proposal disribuion for a miure of an independence and a random walk sampler. The approimaing disribuion is sampled by simulaing an opimized ime-dependen linear diffusion process derived from he recenly developed variaional Gaussian process approimaion mehod. The novel diffusion bridge proposal derived from he variaional approimaion allows he use of a fleible blocking sraegy ha furher improves miing, and hus he efficiency, of he sampling algorihms. The algorihms are esed on wo diffusion processes: one wih double-well poenial drif and anoher wih SINE drif. The new algorihm s accuracy and efficiency is compared wih sae-of-he-ar hybrid Mone Carlo based pah sampling. I is shown ha in pracical, finie sample applicaions he algorihm is accurae ecep in he presence of large observaion errors and low observaion densiies, which lead o a muli-modal srucure in he poserior disribuion over pahs. More imporanly, he variaional approimaion assised sampling algorihm ouperforms Y. Shen (B) D. Cornford Non-lineariy and Compleiy Research Group, Ason Universiy, Birmingham, UK sheny@ason.ac.uk D. Cornford d.cornford@ason.ac.uk M. Opper Arificial Inelligence Group, Technical Universiy Berlin, Berlin, Germany opperm@cs.u-berlin.de C. Archambeau Deparmen of Compuer Science, Universiy College London, London, UK c.archambeau@cs.ucl.ac.uk

2 Y. Shen e al. hybrid Mone Carlo in erms of compuaional efficiency, ecep when he diffusion process is densely observed wih small errors in which case boh algorihms are equally efficien. Keywords Sochasic dynamic sysems Daa assimilaion Bridge sampling Inroducion Sochasic dynamic sysems, also ofen referred o as diffusion processes or sochasic differenial equaions (SDEs), have been used for modelling real sysems in various areas ranging from physics o sysem biology o environmenal science (Honerkamp 994; Wilkinson 6; Miller e al. 999). The curren work was moivaed by he problem of daa assimilaion (Kalnay 3). In he daa assimilaion cone dynamic sysems represening he evoluion of he amosphere sysem are parially observed by an array of differen insrumens. The primary aim of daa assimilaion is he esimaion of he curren sae of he sysem o provide iniial condiions for forecasing. Such coninuous ime sysems are ypically only parially observed, which makes likelihood based saisical inference difficul. In his paper, he inference problem we focus on is smoohing, ha is esimaion of he poserior disribuion over pahs in sae space. From a mehodological poin of view, he smoohing problem for sochasic dynamic sysems has been pursued in hree main direcions. The firs direcion is based on solving he Kushner Sraonovich Pardou (KSP) equaions (Kushner 967; Pardou 98) which are he mos general opimal soluions o he smoohing problem. However, soluion of he KSP equaions is numerically inracable for even quie low-dimensional non-linear sysems (Miller e al. 999), so various approimaion sraegies have been developed. In he paricle approimaion framework (Kiagawa 987), he soluion of he KSP equaions is approimaed by a discree disribuion wih random suppor. As proposed by Kiagawa (996), he smoohed densiy, namely he poserior densiy, is obained from he combinaion of a forward filer and a backward filer. For linear, Gaussian sysems, he forward KSP equaion reduces o he well-known Kalman Bucy filer (Kalman and Bucy 96). The smoohed esimaes of mean and covariance can be calculaed from he filering resuls by recursion, a mehod ofen referred o as he Kalman smooher (Jazwinski 97). To rea non-linear sysems a number of approimaion sraegies have eended he Kalman smooher, for eample, he ensemble Kalman smooher which employs a moderaely sized, randomly sampled, ensemble o esimae he prediced sae covariance mari and hen applies he linear Kalman updaes (Evensen 994, ) and he unscened Kalman smooher which is similar in spiri bu uses an opimal deerminisic sampling sraegy o selec he ensemble members (Julier e al. ; Wan and van der Merwe ). The second direcion involves a variaional approimaion o he poserior process. In Archambeau e al. (7), a linear diffusion approimaion is proposed and is ime varying linear drif is opimised globally. This is eplained in more deail in Sec... In he work by Eyink e al. (4), a mean field approimaion is applied o he KSP equaions and he mean field represenaion of possible rajecories is opimised

3 Variaional Markov chain Mone Carlo for Bayesian smoohing globally, o deermine he mos probable rajecory in a similar vein o 4DVAR daa assimilaion mehods (Derber 989). 4DVAR mehods are widely used in operaional daa assimilaion (Rabier e al. ) and essenially seek he mode of he approimae poserior smoohing disribuion bu do no provide any esimae of he uncerainy abou his mos probably rajecory. The hird direcion employs Markov Chain Mone Carlo (MCMC) mehods (Andrieu e al. 3) o sample he poserior process, which is he focus of his paper. A each sep of an MCMC simulaion, a new sae is proposed and will be acceped or rejeced in a probabilisic manner. For applicaions o coninuous-ime sochasic dynamic sysems, i is also ofen referred o as pah sampling. A single-sie updae approach o pah sampling is adoped by Eraker () in he cone of parameer esimaion of diffusion processes. The auhor repored arbirarily poor miing of his basic algorihm. To achieve beer miing, wo closely relaed MCMC algorihms for pah sampling, namely he Meropolis-adjused Langevin (Suar e al. 4) and he Hybrid Mone Carlo (HMC) algorihm (Aleander e al. 5) have recenly been proposed. Boh mehods updae he enire sample pah a each sampling ieraion while keeping he accepance of new pahs high. This is achieved by combining he basic MCMC algorihm wih a ficiious dynamics so ha he MCMC sampler proposes moves owards he regions of higher probabiliy in he sae space while mainaining deailed balance. Anoher sraegy o achieve beer miing in pah sampling is o updae one of he sub-pahs beween wo neighbouring observaions a each Meropolis-Hasings sep leading o blocking sraegies. In Golighly and Wilkinson (8), he so-called modified diffusion bridge approach is used o propose candidaes for such subpahs. This mehod is a furher developmen of he Brownian bridge sampler proposed by Robers and Sramer () and Durham and Gallan (). Also, sequenial Mone Carlo (SMC) mehods are used o implemen he above blocking scheme in Golighly and Wilkinson (6) [for he use of SMC o build efficien proposal disribuions, we refer o Andrieu e al. ()]. Furher, a random blocking scheme is proposed by Elerian e al. (). To he same end, Beskos e al. (6) have developed a so-called rerospecive sampling mehod which can simulae a wide class of diffusion bridge processes eacly, under cerain condiions on he driving noise process. The bridge sampling, variaional assised mehod proposed herein is applicable o cerain diffusion processes ha canno be reaed by he scheme in Beskos e al. (6), for eample he double well sysem. Our paper also provides a unique comparison of compuaional performance beween he variaional assised bridge sampling mehod developed herein and a hybrid Mone Carlo sampler based on Aleander e al. (5). For an overview of sraegies o develop efficien MCMC algorihms, we refer o he book by Liu (). Recenly, a new sraegy combining sampling mehods wih variaional mehods was inroduced by de Freias e al. (). The saring poin of his sraegy is o use an opimized approimaion o he poserior disribuion as a proposal disribuion in a Meropolis-Hasings (MH) sep. The sampling scheme is implemened in an independence sampler seing. The resuling algorihm is called variaional MCMC (VMC). In he presence of approimaion error, as saed by de Freias e al. (), he accepance rae of hose proposals for a high-dimensional sae space is likely o be very low. To conrol he accepance rae, a block sampling

4 Y. Shen e al. algorihm is proposed by de Freias e al. () o updae only a subse of he componens of sae variables a individual MH seps. They argued ha a variaional approimaion -assised approach o sampling is helpful in eploring he regions of high probabiliy efficienly, however he sampler could ge suck in he neighbouring regions of lower probabiliies as he approimae poserior is ofen more peaked han he rue one. To avoid his, he algorihm is furher developed by combining a Meropolis-ype sampler wih he independence sampler in a probabilisic manner (de Freias e al. ). This mehodology is illusraed in de Freias e al. ()using an eample of Bayesian parameer esimaion for logisic belief neworks. In his paper, we employ he variaional approimaion mehod developed by Archambeau e al. (7) o produce a compuaionally efficien sampling mehod. From he variaional mehod, we obain a ime-varying linear SDE represening he opimized Gaussian process approimaion o he rue poserior process, where we emphasise his is over he full smoohing pah densiy. From he approimae linear SDE, we can derive any bridging process wihin he smoohing window eacly, which eends he blocking sraegies proposed by Golighly and Wilkinson (8), allowing blocks of arbirary size. To implemen he miure sraegy, we spli he proposal procedure ino wo seps: he firs sep generaing he driving whie noise and he second sep simulaing he process forward in ime wih generaed noise. The whie noise can be generaed eiher by direc sampling from a sandard mulivariae Gaussian disribuion or by a random walk MH sampler. The implemenaion can be seen as an adapaion of he reparameerisaion sraegy used in Golighly and Wilkinson (8) for parameer esimaion, however here our aim is o produce efficien proposals for he driving noise process. The paper is organised as follows; Sec. presens a Bayesian reamen of nonlinear smoohing which is followed by a summary of he Markov Chain Mone Carlo smooher (Aleander e al. 5) in Sec.. and he variaional Gaussian process smooher (Archambeau e al. 7) in Sec... The novel sampling algorihms are described in Sec. 3 and he performance of hese algorihms is demonsraed in Sec. 4 by numerical eperimens considering wo sochasic dynamic sysems. The paper also includes a comparison wih he recenly proposed modified diffusion bridge developed in Golighly and Wilkinson (8). The paper concludes wih a discussion on he applicabiliy of our new mehod, and draws conclusions from he comparison wih oher algorihms. Compuaional mehods for Bayesian smoohing of non-linear diffusions Consider a sochasic dynamical sysem represened by d() = f(, )d + D / ()dw(), () where () R d is he sae vecor, D R d d is he so-called diffusion erm, and f represens a deerminisic dynamical process, generally called he drif. The driving noise process is represened by a Wiener process W(). Noe ha SDE ()isalso

5 Variaional Markov chain Mone Carlo for Bayesian smoohing referred o as a sub-class of diffusion processes whose diffusion erm D is independen of sae (Klöden and Plaen 99). The sae is observed via some measuremen funcion h( ) a discree imes, say { k } k=,...,m. The observaions are assumed conaminaed by i.i.d Gaussian noise: y k = h(( k )) + R η () where y k R d is he k-h observaion, R R d d is he covariance mari of measuremen errors, and η represens sandard mulivariae Gaussian whie noise. A Bayesian approach o smoohing is ypically adoped in which he poserior disribuion p(([, T ]) {y,...,y M, < < < M < T }), is formulaed and esimaed, using for eample he mehods described in Sec.. In his work he coninuous-ime SDE is discreised using an eplici Euler Maruyama scheme (Klöden and Plaen 99). This discreisaion induces an approimae nonlinear discree ime model which describes a Markov chain. The discreised version of ()isgivenby k+ = k + f( k, k )δ + D / ( k ) δ ξ k, (3) wih k = k δ, k =,,...,N, and a smoohing window from =ot = N δ. Noe ha ξ k are whie noise random variables. An iniial sae,, needs o be se. There are M observaions wihin he smoohing window chosen a a subse of discreisaion imes ( k j, y j ) j=,...,m wih { k,..., km } {,..., N }. In he following he poserior disribuion is formulaed sep by sep. As a resul of Euler Maruyama discreisaion, he prior of a diffusion process can be wrien as p(,..., N ) = p( ) p( )... p( N N ), where p( ) is he prior on he iniial sae and p( k+ k ) wih k =,...,N are he ransiion densiies of he diffusion process. Noe ha in he limi of small enough δ, hose ransiion densiies can be well approimaed by a Gaussian densiy and hus p( k+ k ) = N ( k + f ( k )δ, Dδ). Therefore, he prior over he pah, defined by he SDE is given by where H dynamics = p(,..., N ) p( ) ep( H dynamics ), N k= δ [ k+ k δ ] [ ] f( k, k ) D k+ k f( k, k ). δ

6 Y. Shen e al. Assuming he measuremen noise is i.i.d. Gaussian, he likelihood is simply given by p(y,...,y M ( ),..., ( N )) ep( H obs ), where H obs = M [ ] h((k j )) y j R [ ] h(( k j )) y j. (4) j= In summary, he poserior disribuion of all saes, i.e. p({,..., N } {y,...,y M }) is given by p( ) ep ( log(p( )) H dynamics H obs ). (5) In Fig., Bayesian smoohing is illusraed using an eample of a one-dimensional sochasic double-well sysem. This sysem is a diffusion process wih wo sable saes; for deails refer o Sec. 4. The reference smoohing resul is obained using a Hybrid Mone Carlo algorihm based on he work by Aleander e al. (5) Fig. Upper panel: a ypical realisaion of a sochasic double-well sysem wih small diffusion noise. The circles denoe a se of observaions obained from his paricular realisaion; Lower panel: mean pah (solid line) and is sandard deviaion envelope esimaed by eensive HMC sampling. For sae esimaion, he variance of diffusion noise is assumed known

7 Variaional Markov chain Mone Carlo for Bayesian smoohing. Hybrid Mone Carlo mehods In HMC approaches (Duane e al. 987; Aleander e al. 5), a molecular dynamics simulaion algorihm is applied o make proposals in a MH algorihm, for eample, X k = { } { } k,...,k N X k+ = k+,..., k+ N, a sep k. To make a proposal of X k+ a ficiious deerminisic sysem is simulaed as follows dx dτ = P dp dτ = X Ĥ(X, P) where P = (p,...,p N ) represens momenum and Ĥ is a ficiious Hamilonian which is he sum of poenial energy H po and kineic energy H kin = Nk= p k.for he poserior disribuion of he non-linear smoohing problem given in Sec., he poenial energy is given by H po = log[p( )]+H dynamics + H obs. The above sysem is iniialised by seing X (τ = ) = X k and sampling a random number from N (, ) for each componen of P(τ = ). Afer ha, one inegraes he sysem equaions forward in ime wih ime incremen δτ by using a leapfrog scheme as follows: X = X + δτp + δτ ( ) X Ĥ P = P + δτ ( ) X Ĥ X Ĥ Afer J ieraions, he sae X (τ = Jδτ) is proposed as X k+ which will be acceped wih probabiliy { ( min, ep Ĥ k+ + Ĥ k)}. The sequence of saes generaed from his mechanism is hen a sample for he poserior smoohing disribuion, (5).

8 Y. Shen e al.. Variaional Gaussian process approimaion smooher The saring poin of he Variaional Gaussian Process Approimaion (VGPA) mehod (Archambeau e al. 7) is o approimae () by a linear SDE: d() = f L (, )d + D / ()dw(), (6) where he ime varying linear drif approimaion is given by f L (, ) = A()() + b(). (7) The mari A() R d d and he vecor b() R d are he variaional parameers o be opimised in he procedure. The approimaion in (7) implies ha he rue poserior process, i.e. p( y,..., y M ), is approimaed by a Gaussian Markov process, q( ) where we have dropped he eplici dependency on he observaions y in he approimaing process, q for noaional convenience, alhough i should be made clear ha he approimae poserior includes he effec of he observaions. Discreising he linear SDE in he same way as he rue SDE, he approimae poserior can be wrien as q(,..., N ) = q( ) N k= N ( k+ k + f L ( k )δ, Dδ ), where q( ) = N ( m(), S()). The opimal A() and b(), ogeher wih he opimal marginal means and covariances m() and S(), are obained by minimising he Kullback Leibler (KL) divergence of q( ) and p( ) (Archambeau e al. 7). The variaional approimaion can also be derived in coninuous ime using Girsanov s change of measure heorem, however i is hen sill necessary o discreise he sysem for compuaional implemenaion (Archambeau e al. 8). In Figs. and 3, he VGPA smoohing mehod is illusraed using he same double-well eample as in Fig.. I can be observed from Fig. ha he smooh variaion of he parameers A and b is inerruped by jumps a observaion imes. This no only draws he mean pah owards he observaions bu also reduces marginal variance around hose observaion imes, which can be seen in he upper panel of Fig. 3. The lower panel shows ha a ypical realisaion generaed by he opimized approimae diffusion is visually similar o he original sample pah shown in Fig.. 3 Variaional MCMC mehods In a Meropolis-Hasings algorihm (Hasings 97) for sampling a poserior densiy π(), defined on a general sae space X, a new sae X is proposed according o some densiy q(, ). The proposed sae will be acceped wih probabiliy α(, ),

9 Variaional Markov chain Mone Carlo for Bayesian smoohing 5 A b Fig. The emporal evoluion of he rend A (upper panel) and offse b (lower panel) of he approimae diffusion process wih is linear drif f L () = A + b which is opimised by he VGPA algorihm for he daa se shown in Fig Fig. 3 Upper panel: he mean pah (solid line) and is sandard deviaion envelope obained from he VGPA for he daa se shown in Fig. ; Lower panel: a ypical realisaion from he VGPA poserior

10 Y. Shen e al. given by { } α = min, π( ) π() q(, ) q(,. ) When he Meropolis-Hasings (MH) algorihm is applied o a paricular Bayesian inference problem, inelligen proposal mechanisms are ofen required o make he algorihm efficien. 3. The variaional independence sampler In he variaional MCMC algorihms oulined below, proposals are made using he variaional parameers A and b, as well as he marginal momens m and S, fromhe VGPA mehod described in Sec... In his seing, an independence sampler o updae he whole sample pah a once is implemened as follows. Firs, propose he iniial sae by sampling from a normal disribuion specified by he marginal momens a =, i.e. N ( m, S ); hen, inegrae he SDE wih is ime-varying linear drif, f L (, ) = A + b, forward in ime using an Euler scheme. The implemenaion is hus: and = m + S w k = k + f L (, )δ + D / δ w k for k =,...,N, where w = (w,w,...,w N ) is a se of realisaions of whie noise, which can be seen as a realisaion of he driving process of he approimae SDE. The resuling sample pah is aken as a new proposal. The accepance rae of he independence sampler is given by, { α = min, π( {y,...,y M }) π( {y,...,y M }) q() } q(. ) where π( ) and q( ) are he poserior and proposal densiy of sample pah, respecively. For simpliciy {y,...,y M } is omied in π( ) in he remainder of his secion. Le q(w) denoe he proposal densiy of whie noise w. For boh original and approimae SDE, here is one-o-one relaionship beween and w and he corresponding Jacobian in he MH-raio is cancelled due o a consan diffusion coefficien. Accordingly, q() q( ) = q(w) q(w ) wih q(w) ( ) N ep w k. k=

11 Variaional Markov chain Mone Carlo for Bayesian smoohing We see his as an independence sampler ha proposes pahs based on he variaional approimaion, leading o higher accepance raes from he proposals. I can also be helpful o hink abou he independence sampler as being for whie noise incremens w raher han he laen pahs direcly, and his helps us o link his approach o he random walk sampler furher eplored in Sec The mehod above differs from he one proposed in Robers and Sramer () which uses an Ozaki approimaion (Ozaki 99) o consruc a linear bridge beween wo (noise-free) observaions. In conras, he approimaion scheme herein is no based on local Taylor epansion of he non-linear drif bu opimized globally, condiioning on all observaions. Moreover, he variaional approimaion derived drif is ime-varying where he bridge proposed by Robers and Sramer () has consan drif beween wo observaions. The efficiency of he independence sampler depends on how well he proposal densiy approimaes he arge measure. In his work, he proposal densiy is a Gaussian process approimaion o he arge measure. Therefore, he efficiency of he above algorihms is deermined by how far he arge measure deviaes from a Gaussian one. In cases wih a highly non-linear drif erm in he diffusion and relaively few observaions, he above proposal mechanisms need o be furher refined. 3. Condiional block sampling: he variaional diffusion bridge The idea of blocking helps o improve he performance of independence samplers. Simply speaking, only a sub-pah of he full sample pah is proposed, X sub ={ k,..., k+l } X ={,..., T } say, a each MH sep while he remaining pars are fied. The inde k is chosen randomly while he sub-pah lengh L is a uning parameer of he algorihm whose value is deermined on he basis of pilo runs, and hen fied during eecuion of he main sampler. In selecing L here is a rade-off: oo shor a sub-pah and accepance raes are high, bu miing is poor; oo long and miing is good bu accepance raes are low. Due o he Markov propery of he process X, he sub-pah needs only be condiioned on k and k+l, insead of he enire remaining pah. To implemen his blocking sraegy, he simulaion of a ime-varying, linear bridging process is required o make proposals. For he bridging simulaion, he effecive drif and diffusion erm of a imevarying linear SDE are derived. For clariy, a one-dimensional SDE is considered and he problem of simulaing a bridging process is furher reduced o he following quesion: how o sample a ime wih = + δ and < T condiioning a ime and T a ime T? Noe ha = T δ. To sample, firs compue he condiional probabiliy p(, T ) p( ) p( T ).

12 Y. Shen e al. As he ime incremen δ is he one used o discreise boh he original and approimae SDE by a Euler Maruyama scheme On he oher hand, p( ) ep Dδ ( f L ( )δ). (8) }{{} I p( T ) p( T ) p( T ). In fac, he righ side can be undersood as he marginal densiy of a ime for a linear SDE represened by (6) which runs from = T o =. The momen equaions of his backward linear SDE are and d d c = Ac + b d d d = Ad D, where c and d are he marginal mean and variance, respecively. For a bridge, he sae a ime T is fied o T which implies c T = T and d T =. Thus p( T ) ep ( c ). (9) d }{{} Re-formulaing I and I in (8) and (9), respecively, he effecive drif and diffusion erms for he bridging process are obained as follows: I f ef f L = da + D } d + {{ Dδ } A ef f + cd + bd } d + {{ Dδ } b ef f () and D ef f d = D d + Dδ. () As he forward effecive SDE is equivalen o he process specified by he above backward momen equaions, i is clear ha is realizaion mus hi T a ime T.

13 Variaional Markov chain Mone Carlo for Bayesian smoohing In he coninuous-ime limi, he above effecive drif- and diffusion erms are in accordance wih hose derived from he corresponding Fokker Planck equaions, which are f ef f L = f L + σ ln p(x T ) and D ef f = D, respecively. In Fig. 4, block sampling is illusraed wih he same sochasic double-well eampleasinfig.. Noice ha boh effecive A and b rise sharply a he righ end of he block (middle and lower panel, respecively). In his paricular eample, he rise of b forces he effecive mean pah o evolve owards he fied end poin. The proposed sub-pah also approaches he fied end poin because he effecive marginal variance is reducing o zero due o he increase of A. The general principle of consrucing a bridging process from he approimae linear SDE has been oulined. To implemen i efficienly and correcly, he following wo echnical deails also need o be addressed:. Wih blocking, he MH-raio is given by { α = min, π( sub ) π( sub ) q(w } sub) q(w sub ) A b Fig. 4 An illusraion of he blocking sraegy for pah sampling. From op o boom: choosing a sub-pah for updaing, whose ends are indicaed by black circles. A proposal of his sub-pah is highlighed by he hick black line; compuing he effecive rend A and offse b (solid lines) condiioned on he sae,, a boh ends of he block, compared o he original A and offse b from he VGPA (dashed lines)

14 Y. Shen e al. As saed above, w sub is used o propose sub by inegraing he effecive linear SDE forward in ime. However, w sub mus be re-consruced from sub wih he same SDE ha generaes sub. Noe ha w sub are differen from hose ha are acually used o propose sub. This is because he condiioning has been changed;. To increase he accepance rae, he proposal of mus be condiioned on. This is obained by sampling from a condiional Gaussian disribuion as follows where N ( m S + m S S + S, S S ) S + S m = b d A d and S = D d ( A d) Numerical eperimens show ha he above algorihm for pah sampling could have poor miing if he sampler ges suck in he regions of lower probabiliies. Therefore, a VGPA-assised random walk sampler is developed for pah sampling in he following secion. 3.3 The variaional random walk sampler To augmen he independence sampler, a random walk scheme for pah sampling is also developed. A rivial implemenaion of such scheme would updae he sample pah direcly, which leads o a vanishingly low accepance rae. Insead, he driving process w is updaed using a random walk sampler, i.e. w = w + σ η. where η is whie noise and σ represens he sep size of he random walk. Given w, a new sample pah is obained by simulaing he approimae SDE forward in ime. Denoe his deerminisic mapping funcion by f T, i.e. = f T (w ). Up o a Jacobian, he join poserior is hus given by π(, w y) p() q(w) p(y f T (w)) where p() and q(w) represens he prior densiy of he diffusion process and he driving process, respecively. As he Jacobian is cancelled due o a consan diffusion coefficien, he resuling MH raio is given by { } α = min, π( ) π() q(w ). q(w) Recall ha π() = p() p(y = f T (w)).

15 Variaional Markov chain Mone Carlo for Bayesian smoohing The idea of a random walk in w-space is very similar o he innovaion scheme proposed by Golighly and Wilkinson (8). In Golighly and Wilkinson (8) he innovaion scheme is uilised o break high dependence beween unknown diffusion parameers and missing sample pahs whereas in his work he random walk sampler helps an independence sampler o avoid geing suck. Noe ha he random walk sampler could be very slow in eploring he poserior densiy. Thus, a miure sraegy needs o be adoped. 3.4 The variaional diffusion sampler Finally, he independence sampler and he random walk sampler proposed here are combined ino a miure ransiion kernel in he variaional assised MCMC algorihm devised for pah sampling. This resuls in a ransiion kernel given by T = p T ind + ( p) T rand, wih probabiliyp, where T ind and T rand represen he ransiion kernel of an independence sampler and a random walk sampler, respecively. The complee algorihm is deailed in Algorihm. 4 Numerical eperimens In his secion, he variaional MCMC (VMC) algorihms described in Sec. 3 are compared wih he sae-of-he-ar HMC mehod based on he implemenaion developed by Aleander e al. (5). HMC mehods are used in he comparison since hese are know o be amongs he mos compuaionally efficien sampling sraegies for high dimensional problems and o our knowledge here is no eising comparison of he compuaional efficiency of HMC and bridge sampling approaches. In addiion, a comparison of he variaional diffusion bridge (Sec. 4.3) and modified diffusion bridge (Golighly and Wilkinson 8) is also presened. 4. Eperimenal seup The algorihms are esed on wo one-dimensional sysems: a sochasic double-well poenial sysem (DW) and a SINE-drif diffusion process (SINE). Noe ha while he dynamic sysems have a single sae variable, he discree ime sae vecor (pah) being sampled is very high dimensional, ypically around 8 5. The sysems are described by d = 4( )d + Ddw, Thevalueofp is chosen on he basis of a se of pilo runs in he curren implemenaion, alhough as noed by a referee and discussed laer in he paper more sophisicaed approaches are possible.

16 Y. Shen e al. Algorihm The VMC pah sampler. : fi block lengh, L, apply VGPA o esimae A and b. : generae a se of whie noise {w,,...,t } 3: iniialise N ( m, S ) using w 4: iniialise,...,t by using w,...,t o inegrae d = ( A + b)d + Ddw 5: repea 6: randomly choose block k from {,,...,T } 7: if k T L (block fully wihin ime window) hen 8: compue effecive Â, ˆb and ˆD for he block condiioned on k and k+l 9: compue he effecive whie noise ŵk k+l : choose an independence sampler or a random walk sampler wih probabiliy p : if an independence sampler is chosen hen : generae new whie noise wk k+l 3: propose k k+l by inegraing d = ( Â + ˆb)d + ˆDdw 4: make a Meropolis-Hasings updae for k k+l 5: else 6: generae whie noise ξk k+l 7: propose new wk k+l by w = ŵ + ξ sep size 8: propose k k+l by inegraing d = ( Â + ˆb)d + ˆDdw 9: make a Meropolis-Hasings updae for wk k+l and k k+l joinly : end if : else : updae and k,...,t applying mehods similar o he above, using he original A and b from VGPA 3: end if 4: unil a sufficien number of sample pahs are obained and d = π D sin(π )d + Ddw, respecively, where D is he diffusion variance and dw represens whie-noise. As can be seen from Figs. 5, 7, 8, boh sysems have mea-sable saes. They are =±for DW and =±k, wih k N for SINE. Noe ha SINE sysems have an infinie number of mea-sable saes. Clearly, he parameer D deermines how frequen he ransiion beween hose mea-sable saes are. Two varians of DW are invesigaed, one wih D =.5 ypical of rare ransiions and one wih D =. ypical of frequen ransiions. For SINE, D is se o.656 so ha is ransiion frequency is comparable wih he nd varian of DW (abou 6 ransiions in a ime inerval of 5 unis, see Figs. 7, 8). Noe ha he average ei ime for DW wih D =.5 is abou 3 ime unis, hus we have seleced a rare ransiion o illusrae he wors case behaviour of he new algorihms on his sysem. In his eample, he ime inerval is se o 8 unis o enable a large-scale Mone Carlo eperimen o be carried ou in order o assess he variabiliy of he numerical resuls. I is clear ha he miing of he MCMC algorihms will depend on he qualiy of observaions in erms of observaion densiy ρ (number of observaions per ime uni) as well as observaion error variance R. Therefore, boh HMC and VMC are esed wih 9 combinaions of 3 differen ρ-values and 3 differen R-values, i.e. ρ =,ρ =,ρ 3 = 4, R =.4, R =.9, and R 3 =.36.

17 Variaional Markov chain Mone Carlo for Bayesian smoohing ACF seps Fig. 5 Comparison of marginal esimaes (lef) and miing properies (righ) beween VMC (black) and HMC (grey) for a double-well sysem wih diffusion coefficien D =.5 and ime window T = 8. Lef panel: he esimae of mean pah and is ± sandard deviaion envelopes for HMC and VMC. The dos denoe he observaions, a a densiy of one per ime uni, and he observaion error variance, R =.4. Righ panel: decay of he auocorrelaion funcion of ime series of he summary saisic, VMC(black, riangles) andhmc(grey, circles) (seee) The efficiency of he differen MCMC algorihms is compared using heir miing properies while also rigorously assessing he accuracy of he new algorihm. Given a fied amoun of compuing ime, he miing propery is a criical measure for he compuaional efficiency of a paricular MCMC algorihm. The auo-correlaion beween subsequen samples is ofen used o quanify he miing. Here, each sample is a realisaion, or sample pah, of he diffusion process condiioned on he daa. The auo-correlaion of a simple summary saisic of sample pahs is compued as a diagnosic, namely T = ()d. To summarise he auo-correlaion funcion, a so-called auo-correlaion ime τ is defined by τ = + ACF(k). k=

18 Y. Shen e al. wih ACF(k) = ( ˆ < ˆ >)( ˆ+k < ˆ >) Var( ) where < > represens he epecaion and ˆ denoes algorihmic ime. The summaion is runcaed a lag k = 4 o minimise he impac of finie sample size errors on he esimaes. When he oal compuing ime is fied, he lengh of he Markov chains generaed by differen algorihms varies. Thus he chains are sub-sampled so ha he same number of sample pahs are obained from all algorihms. The accuracy of he new algorihm is characerised by he inegraed marginal KL-divergence beween is samples and hose from HMC: KL = T { ˆπ HMC ( ) log ˆπ HMC ( ) ( ) d ˆπ VMC } d, where ˆπ HMC and ˆπ VMC are he esimaes of he marginal poserior a ime compued from samples obained by HMC and VMC algorihms respecively. I is clear ha he smaller his KL-divergence esimae is, he more accurae is he algorihm. However a non-vanishing residual value of KL-divergence is epeced even for wo eac algorihms due o finie sample size. Therefore, he KL-divergence esimaes are also compued for wo independen ses of sample pahs from HMC which are addiionally used for convergence assessmen of HMC. As seen in previous secions, boh HMC and VMC have several uning parameers: he number of molecular dynamics seps J and he (ficiious) ime incremen δτ for HMC; sub-pah lengh L used in block sampling, sep size σ used in random walk sampling, and probabiliy p used in he miure ransiion kernel for VMC. As he compuing ime of a HMC algorihm increases wih J, he opimal choice of J and τ is obained by minimising he auocorrelaion measure per compuing ime uni based on a se of pilo eperimens. The corresponding accepance raes vary beween 6 and 7%. As repored in he lieraure, he opimal sep size σ is chosen so ha he algorihm considered has an accepance rae beween and 4%. The same principle applies o he choice of block size L. For he deerminaion of he opimal p, he accuracy facor needs o be aken ino accoun. In boh ereme cases of a purely independence sampler based scheme (p = ) and a purely random walk sampler based scheme (p = ), he resuling esimaed disribuion of sample pahs could be inaccurae given a sufficienly small sample. Once he opimal parameers are deermined on he basis of pilo runs hese are fied in all eperimens. Technical deails of he numerical eperimens are as follows. Boh DW and SINE are discreised wih ime incremen δ =.. From pilo runs, he opimal seings for he HMC uning parameers depend srongly on he choice of δ bu varies lile over differen sysems and differen observaion ses. This also applies o oher uning parameers. In he repored eperimens J =,δτ =., L =,σ =.5, and p =. are he fied values used hroughou. All algorihms, including HMC, are iniialised by a realisaion of he approimae ime-varying linear SDE from VGPA.

19 Variaional Markov chain Mone Carlo for Bayesian smoohing For every daa se, he amoun of compuing ime is fied o ha of a HMC chain of 5, MH updaes. All chains of boh HMC and VMC are hen sub-sampled o obain 5, sample pahs from each. The burn-in period is generally small as a resul of good iniialisaion and he firs sample pahs are discarded as burn-in. An ecepional case is he daa se generaed by SINE wih ρ = and R =.36. For his case, he miing of he HMC algorihm is very poor and is burn-in is eremely large so i is necessary o run a chain of 5,, MH-updaes. The VMC chains are also adaped accordingly. All resuls are summarised in Tables,, 3. For illusraion, Figs. 5, 7, 8 show hree eamples of boh HMC and VMC resuls summarised by he mean pahs, he sandard deviaions envelopes, and he decay of he auo-correlaion funcion. 4. HMC comparison The resuls for DW wih D =.5 are iniially discussed. From he hird and fourh columns of Table, i can be seen ha he auocorrelaion imes of boh MCMC algorihms, τ HMC and τ VMC, decrease wih increasing observaion densiies, as migh be epeced. Miing improves wih reducing observaion errors in line wih epecaions. For HMC, boh condiions have he effec of increasing he informaion provided o he sampler from he gradiens of log poserior which helps o more efficienly eplore he poserior. However, he overall rend is much more marked for HMC han VMC. VMC can improve he miing compared o HMC by an order of magniude for low observaion densiy and large observaion error, i.e. ρ = and R =.36. In conras, boh algorihms show a comparable efficiency in cases where he process is densely observed, i.e. ρ = 4, wih lower observaion errors i.e. R =.4 and R =.9. A reducion of auo-correlaion imes for VMC is observed wih raes ranging beween Table Comparison of he inegraed auocorrelaion imes beween wo ses of sample pahs from VMC and HMC, and comparisons of he inegraed KL-divergence beween wo independen ses of sample pahs from HMC (KL ), beween VMC and HMC (KL ), and beween he se of sample pahs from HMC and he esimaed marginal disribuions of he sae,, from VGPA (KL 3 ) ρ obs R τ HMC τ VMC KL KL KL ±.37.3 ±.. ±..4 ± ± ±.4.37 ±.. ±..6 ± ± ± ±.46.9 ±..46 ± ± ±.4.37 ±.34.9 ±..7 ±..49 ± ±.6.4 ±.3. ±..48 ± ± ± ±.7.4 ±..98 ± ± ±..3 ±.7.7 ±.. ±..4 ± ±.7. ±..8 ±.. ±..9 ± ±..43 ±.9. ±..9 ±.85.3 ± 8.5 The resuls are obained from a double-well sysem wih diffusion coefficien wih D =.5, T = 8and are shown as a funcion of observaion densiy ρ obs and error variance R. The variabiliy of all above esimaes are indicaed by heir sandard deviaions which were obained by Mone Carlo repeiions of he eperimens

20 Y. Shen e al. Table As Table bu for a double-well sysem wih diffusion coefficien wih D =., T = 5 ρ obs R τ HMC τ VMC KL KL KL Table 3 As Table bu for a SINE-drif sysem wih diffusion coefficien wih D =.656, T = 5 ρ obs R τ HMC τ VMC KL KL KL The figures wih are obained from he chains which are sub-sampled wih a fied beween-sample inerval imes longer han oher figures, due o he eremely poor miing of HMC for his paricular eample. These wo τ-esimaes are scaled accordingly, which makes he values more noisy. We believe he τ VMC -value is closer o 3 and 8 for he observaion ses beween he above wo ereme cases (for eample, see Fig. 5). The VMC resuls are more sable across he differen Mone Carlo eperimens as shown by he esimaed sandard deviaion of he auo-correlaion imes. The accuracy of VMC is indicaed by he inegraed marginal KL-divergence per ime uni, KL, shown in he sih column. A similar overall rend o ha found in he auocorrelaion imes is observed. The KL-divergence esimaes decrease wih improving qualiy of observaions. From he fifh column, KL, i seems ha he residual (finie sample derived) KL-divergence value is no consan, flucuaing around.7.9 per ime uni for differen observaion densiies and ses. By comparing KL and KL, we conclude ha VMC does sample from he poserior eacly for he daa ses wih (ρ =, R =.4), (ρ = 4, R =.4), and (ρ = 4, R =.9). For oher daa ses, heir non-zero KL-divergence values are saisically, bu probably no pracically, significan. This indicaes ha for finie sample sizes he corresponding esimaed poserior disribuions from VMC are slighly less accurae han hose obained from HMC. This is a resul of he approimaion employed in he VGPA. The

21 Variaional Markov chain Mone Carlo for Bayesian smoohing marginal poserior..8 marginal poserior ( = 3.5) ( = 3.5) Fig. 6 Comparison of HMC (Grey) andvgpa(black) esimae of he marginal poserior of he sae a ime = 3.5 for he eamples shown in Fig. 5 and an ereme eample wih high observaion noise ha shows a muli-modal marginal poserior (lef and righ panel, respecively). Noe ha he rue value for a ime = 3.5 is.8 VGPA canno capure non-gaussian behaviour, and hus he approimaion qualiy of he VGPA poserior varies, as shown by KL 3 (final column). I has been repored ha he approimae poserior obained from variaional mehods is ofen much more narrowly peaked compared o he rue poserior. However such bias remains small ecep for he ereme case wih infrequen and low qualiy observaions (ρ =, R =.36), see Fig. 6. A range of visual diagnosics were eplored o assess he reasons for he relaively poor accuracy in he VGPA and hus VMC. The main issue is in he ails of he disribuion lending furher suppor o he idea ha he variaional poserior is raher narrow, paricularly in he regions where he rue poserior is muli-modal. The presence of large observaion errors leads o muli-modaliy of he poserior. The righ panel in Fig. 6 shows ha he marginal poserior seems o have hree modes a = and = ± in he ransiion phase ( = 3.5) whereas he approimae poserior from VGPA can only possess a single mode, whose variance is underesimaed. For wo eamples of frequen ransiions, i.e. DW wih D =. and SINE wih D =.656, similar resuls are observed, shown in Tables and 3. However, he KL-values have increased for boh cases when compared wih hose of he previous eample. The increase in residual KL-divergence values, KL, indicaes ha his is parially he resul of he larger smoohing window used in he inference. Bearing his in mind, he resuls for DW wih D =. are beer han DW wih D =.5 (compare Figs. 5, 7). One facor ha can eplain his improvemen is he increased diffusion coefficien (D =.) which also drives he variaionally opimised SDE and hus spreads he realisaions from he variaional diffusion bridge more widely

22 Y. Shen e al. A B C D E F ACF seps Fig. 7 As Fig. 5 bu for a double-well sysem wih diffusion coefficien D =. and ime window T = 5. Observaions are made wih densiy wo per ime uni and he observaion error variance R =.9. For clariy, he marginal esimaes and observaions are displayed in separae panels (a e) for 5 consecuive ime inervals of lengh unis in sae space. I should also be noed ha he VGPA resuls, KL 3, are also relaively improved in all cases, also helping he VMC sampler. Compared o wo DW eamples, he accuracy of VMC has declined for SINE (Table 3). Recall ha he SINE sysem has a series of mea-sable saes which could make muli-modaliy problems more severe, and his is a problem for boh HMC (wih very slow miing) and VMC (wih relaively poor accuracy). However, as he observaion densiy and accuracy increases he VGPA aains good accuracy and he VMC works very well (see Fig. 8). For all hree sysems, good accuracy and a significan improvemen in miing are achieved in he cases where observaion densiies ρ and observaion errors R are eiher boh low or boh high. For higher ρ and lower R, a sligh improvemen in miing is observed accompanied by a high accuracy. For lower ρ and higher R, an increase in miing by an order of magniude is seen bu he accuracy is relaively poor. When compared wih he approimaion error arising from VGPA, however, he relaive accuracy is sill of grea value, ha is KL is always a subsanial reducion from KL 3.This means ha he approimaion error of VGPA is significanly reduced when employing

23 Variaional Markov chain Mone Carlo for Bayesian smoohing A B C D E F 4 ACF seps Fig. 8 As Fig. 5 bu for a SINE-drif sysem wih diffusion coefficien D =.656 and ime window T = 5. Observaions are made wih densiy four per ime uni and he observaion error variance R =.4. For clariy, he marginal esimaes and observaions are displayed in separae panels (a e) for 5 consecuive ime inervals of lengh unis a VMC run using only a fracion of he compuing ime needed for a HMC run. In pracical applicaions his could be of grea value in obaining improved approimaions in reasonable compuaional ime. Thus he VMC mehod has applicabiliy across a wide range of sysems, and improves on he VGPA significanly in all cases. 4.3 Diffusion bridge comparison In his work, we also implemened a pah sampling algorihm similar o VMC bu use he modified diffusion bridge (Golighly and Wilkinson 8) for he sub-pah proposal. As sub-pahs are randomly chosen, i is possible here could be no observaion in a such sub-pah. In his case, modified diffusion bridge is reduced o he modified Brownian bridge proposed by Durham and Gallan () because is proposals are no condiioned on observaions. We compare he modified diffusion bridge algorihm wih he VMC in erms of he accepance raes for a double well sysem wih D =.5 and T = 8. The resuls are shown in Table 4, wih differen observaion

24 Y. Shen e al. Table 4 Comparison of accepance raes beween wo diffusion bridge schemes for sub-pah sampling: modified diffusion bridge (MDB) and variaional diffusion bridge (VDB) ρ obs R L =.5 L = L = MDB (%) VDB (%) MDB (%) VDB (%) MDB (%) VDB (%) The resuls are obained from a double-well sysem wih diffusion coefficien wih D =.5, T = 8and are shown as a funcion of observaion densiy ρ obs, error variance R, and sub-pah lengh L (in ime uni) densiies, noise level, and sub-pah lengh. I can be seen ha he accepance raes of VMC are generally wice as large as hose of he modified diffusion bridge. For ρ = and ρ = 4, VMC wih L = shows a comparable or even beer accepance rae han he modified diffusion bridge wih L =.5. This indicaes ha VMC has beer miing han a pah sampling algorihm using he modified diffusion bridge. I should however be noed ha he VMC algorihm is slighly more compuaionally epensive. Boh algorihms also show some common feaures. The accepance raes do no depend on noise level bu do increase wih observaion densiies for longer pahs, L = and L =. 5 Discussion and conclusions This paper develops a novel MCMC algorihm which combines wo paricular MH algorihms, namely an independence sampler and a random walk sampler, wih he recenly developed VGPA for Bayesian inference in non-linear diffusions. This demonsraes ha variaional approimaions can be combined wih MCMC mehods o good effec in pah sampling. We sress ha he variaional approimaion inroduced in his paper is no he radiional fully facorising approimaion, raher he approimaion is over he join poserior disribuion of he sae pah, which makes he variaional approimaion an aracive proposal disribuion for pah sampling. The basic implemenaion of he VMC sampling scheme is enhanced by inroducing a fleible blocking sraegy o improve he performance of he variaional samplers. For pah sampling, he implemenaion of blocking needs he simulaion of a bridging process. The idea has already been applied o likelihood inference in diffusion processes (Durham and Gallan ; Golighly and Wilkinson 6). However boh previous papers made a relaively crude Gaussian approimaion o he ransiion densiy of he process, based on a modified Brownian bridge scheme. The mehod is

25 Variaional Markov chain Mone Carlo for Bayesian smoohing furher improved in Golighly and Wilkinson (8) by condiioning bridge proposals on observaions available in he block, producing a scheme which has similariies o our approach. To make proposals wih reasonable accepance probabiliies he lengh of he blocking ime inerval is limied in all eising schemes. In conras, he novel bridging process developed herein is derived eacly from he approimae linear diffusion which has been opimised by he VGPA mehod. This sophisicaed mehod of proposing sub-pahs renders VMC an accurae and efficien sampling scheme. We have compared our mehod wih he one proposed in Golighly and Wilkinson (8). The variaional diffusion bridge mehod inroduced in his paper is able o propose longer sub-pahs (a facor of beween and 4 imes longer) while achieving a comparable accepance rae when compared o he modified diffusion bridge of Golighly and Wilkinson (8). As a resul, VMC has beer miing properies compared o a pah sampling algorihm using he modified diffusion bridge. As in he original VGPA framework, he VMC sampling algorihms apply only o a sub-class of diffusion processes where eiher he diffusion coefficien mus be saeindependen or he diffusion model can be ransformed ino he one of uni diffusion coefficien. However, such a ransformaion is generally no available for mulivariae diffusion models (Papaspiliopolous e al. 3). The sophisicaed framework of eac sampling, developed by Beskos e al. (6), is also resriced o he above sub-class, alhough here are ineresing models of his kind o which eac sampling does no apply. For insance, he sochasic double-well sysems sudied here do no fulfil he condiion on he drif erm required by eac sampling mehods. This resricion is removed for he refined eac sampling algorihm proposed by Beskos e al. (8). For mulivariae sysems, however, his new eac sampling algorihm sill requires ha he drif erm needs o have a poenial which is no rue for all processes wihin he class of diffusions o which our algorihm applies. As seen in Sec. 4, he VMC algorihm ouperforms HMC by miing while obaining good accuracy when boh observaion densiies and observaion errors are moderae. If he observaion densiy is low and he observaion error is large, he marginal poserior clearly shows some muli-modal srucure. This leads o a large approimaion error in he VGPA, which in urn causes lower accuracy of VMC, alhough HMC mehods also encouner problems wih slow miing in his case. A possible soluion, which should be eplored in fuure work, is o adop a empering sraegy. Simply speaking, he rue poserior needs o be empered such ha he VGPA can approimae he empered poserior wih desired accuracy. The essence of he idea is o use he diffusion variance D as he emperaure in a empering scheme. The algorihm would be compleed by consrucing a ladder connecing he eac and approimae poseriors. We believe such a empering scheme could mainain he efficiency of he VMC, while also improving he accuracy. In summary our resuls indicae ha he VMC mehod is more efficien han HMC (Aleander e al. 5) and he modified diffusion bridge based sampler (Golighly and Wilkinson 8) in he regime beween he ereme cases of very high observaion densiy (when all mehods perform similarly) and low observaion densiy (when no mehod works very well) as shown in he resuls secion. Here high and low observaion densiy has o be inerpreed for each dynamical sysem and as a funcion of observaion noise.