arxiv: v5 [stat.me] 23 Oct 2016

Transcription

1 Bayesia Aalysis (216) TBA, Number TBA, pp Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Oksaa A. Chkrebii, David A. Campbell, Be Calderhead, ad Mark A. Girolami arxiv: v5 [sa.me] 23 Oc 216 Absrac. We explore probabiliy modellig of discreizaio uceraiy for sysem saes defied implicily by ordiary or parial differeial equaios. Accouig for his uceraiy ca avoid poserior uder-coverage whe likelihoods are cosruced from a coarsely discreized approximaio o sysem equaios. A formalism is proposed for iferrig a fixed bu a priori ukow model rajecory hrough Bayesia updaig of a prior process codiioal o model iformaio. A oe-sep-ahead samplig scheme for ierrogaig he model is described, is cosisecy ad firs order covergece properies are proved, ad is compuaioal complexiy is show o be proporioal o ha of umerical explici oe-sep solvers. Examples illusrae he flexibiliy of his framework o deal wih a wide variey of complex ad large-scale sysems. Wihi he calibraio problem, discreizaio uceraiy defies a layer i he Bayesia hierarchy, ad a Markov chai Moe Carlo algorihm ha arges his poserior disribuio is preseed. This formalism is used for iferece o he JAK-STAT delay differeial equaio model of proei dyamics from idirecly observed measuremes. The discussio oulies implicaios for he ew field of probabilisic umerics. Keywords: Bayesia umerical aalysis, uceraiy quaificaio, Gaussia processes, differeial equaio models, uceraiy i compuer models. 1 Quaifyig uceraiy for differeial equaio models May scieific, ecoomic, ad egieerig disciplies represe he spaio-emporal evoluio of complex sysems implicily as differeial equaios, usig few bu readily ierpreable parameers. Differeial equaio models describe he aural depedece bewee sysem saes ad heir raes of chage o a ope spaio-emporal domai, D R d. Their relaioship, represeed by he fucio F, is fully specified up o some physical cosas θ belogig o a parameer space Θ. Mahemaically, sysem saes, u : D Θ R p saisfy he parial differeial equaio (PDE), ( F x,, u, u, u 2 ) u,,..., u, θ =, (x, ) D D, x 1 x 2 x 1 x 2 Relaed aricles: DOI: /16-BAXXX, DOI: /16-BAXXX, DOI: /16-BAXXX; rejoider a DOI: /16-BAXXX. Deparme of Saisics, The Ohio Sae Uiversiy, Columbus, OH, USA, oksaa@sa.osu.edu Deparme of Saisics & Acuarial Sciece, Simo Fraser Uiversiy, Caada, dac5@sa.sfu.ca Deparme of Mahemaics, Imperial College Lodo, Lodo, UK, b.calderhead@ucl.ac.uk Deparme of Saisics, The Uiversiy of Warwick, Covery, UK, M.Girolami@warwick.ac.uk c 216 Ieraioal Sociey for Bayesia Aalysis DOI: /16-BA117 imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

2 2 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios ad cosrais a he boudary D of D. Whe saes evolve wih respec o a sigle variable, such as ime, he model simplifies o a ordiary differeial equaio (ODE). The explici soluio, deoed u : D Θ R p, of a differeial equaio problem over spaio-emporal locaios, (x, ) D, is used o explore sysem dyamics, desig experimes, or exrapolae o expesive or dagerous circumsaces. Therefore, icreasig aeio is beig paid o he challeges associaed wih quaifyig uceraiy for sysems defied by such models, ad i paricular hose based o mahemaical descripios of complex pheomea such as he weaher, ocea curres, ice shee flow, ad cellular proei raspor (Ghaem ad Spaos, 23; Kaipio e al., 24; Huue ad Kaipio, 27; Marzouk e al., 27; Marzouk ad Najm, 29; Suar, 21). The mai challege of workig wih differeial equaio models, from boh mahemaical ad saisical perspecives, is ha soluios are geerally o available i closed form. Whe his occurs, prior exploraio of he mahemaical model, u (x,, θ), cao be performed direcly. This issue is deal wih hroughou he lieraure by replacig he exac soluio wih a N dimesioal approximae soluio, û N (x,, θ), obaied usig umerical echiques over a size N grid pariioig of he domai D. Iferece ad predicio based o û N (x,, θ) he iform he reasoig process whe, for example, assessig fiacial risk i decidig o oil field bore cofiguraios, or formig goverme policy i respose o exreme weaher eves. Limied compuaio ad coarse mesh size are coribuors o umerical error which, if assumed egligible ca lead o serious misspecificaio for hese highly oliear models. The sudy of how uceraiy propagaes hrough a mahemaical model is kow as he forward problem. Numerical error aalysis provides local ad global discreizaio error bouds ha are characerized poi-wise ad relae o he asympoic behaviour of he deermiisic approximaio of he model. However, accouig for his ype of verificaio error for he purpose of model iferece has prove o be a difficul ope problem (Oberkampf ad Roy, 21), causig discreizaio uceraiy o be ofe igored i pracice. For complex, large-scale models, maiaiig a mesh size ha is fie eough o igore discreizaio uceraiy demads prohibiive compuaioal cos (Arridge e al., 26), so ha a coarse mesh is he oly feasible choice despie associaed model uceraiy (Kim, J.-h. e al., 214). Furhermore, some ODE ad PDE models exhibi highly srucured bu seemigly upredicable behaviour. Alhough he log erm behaviour of such sysems is ereched i he iiial saes, he presece of umerical discreizaio error iroduces small perurbaios which eveually lead o expoeial divergece from he exac soluio. Cosequely, iformaio abou iiial saes rapidly decays as sysem soluio evolves i ime (Berlier, 1991). The calibraio or saisical iverse problem cocers he uceraiy i ukow model ipus, θ, give measuremes of he model saes (Bock, 1983; Ioides e al., 26; Dowd, 27; Ramsay e al., 27; Xue e al., 21; Bruel, 28; Liag ad Wu, 28; Calderhead ad Girolami, 211; Campbell ad Seele, 211; Gugushvili ad Klaasse, 212; Campbell ad Chkrebii, 213; Xu e al., 213). However, iferece for differeial equaio models lackig a closed form soluio has maily proceeded uder he uverified assumpio ha umerical error is egligible over he eire parameer space. From he perspecive of Keedy ad O Haga (21), he discrepacy bewee he umerical approximaio, û N, ad he uobservable mahemaical imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

3 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 3 model, u, a he observaio locaios (x, ) D T would be modelled by he fucio, δ(x, ) = u (x,, θ) û N (x,, θ). Esimaio of δ(x, ) requires observaios y(x, ) from u (x,, θ). However, magiude ad srucure of discreizaio error chage wih θ, while δ(x, ) is agosic o he specific form of he uderlyig mahemaical model, reaig i as a black box. Cosequely explorig he model uceraiy for give values of θ requires observaios for each parameer seig or he assumpio ha δ(x, ) is idepede of θ. Aleraively, i he discussio of Keedy ad O Haga (21), H. Wy argues ha he sesiiviy equaios ad oher uderlyig mahemaical srucures ha gover he umerical approximaio could be icluded i he overall uceraiy aalysis. 1.1 Coribuios ad orgaizaio of he paper This paper develops a Bayesia formalism for modellig discreizaio uceraiy wihi he forward problem, ad explores is coribuio o poserior uceraiy i he model parameers. We subsaially expad ad formalize ideas firs described by Skillig (1991) ad develop a geeral class of probabiliy models for soluio uceraiy for differeial equaios i he so-called explici form, A = f(x,, B, θ), (x, ) D D, (1) where A ad B are subses of he deermiisic soluio u = u(x,, θ) ad is parial derivaives, ad he fucio f is Lipschiz coiuous i B. Alhough (1) is very geeral, Secios 2 ad 3 cosider ordiary differeial equaios for exposiioal simpliciy, where A = du(, θ)/d := u ad B = u(, θ) := u for [, L], L >, i.e. u = f(, u, θ), wih a give iiial codiio. Secios 4 ad 5 exed his o he geeraliy of (1), showcasig a wide class of ODEs ad PDEs, such as mixed boudary value problems wih muliple soluios, delay iiial fucio problems, ad chaoic differeial equaios. Secio 2 cosiders fixed model parameers θ, hyperparameers Ψ, ad discreizaio grid of size N. Prior uceraiy abou he soluio u ad is derivaive u is ecoded i a probabiliy model [u, u θ, Ψ] o a space of fucios saisfyig he iiial codiio. Secio 3 describes a sequeial model ierrogaio mehod ha cosiss of (i) drawig a sample u 1 from he margial [u f 1,..., f 1, θ, Ψ], ad (ii) usig his o compue model ierrogaios f f(, u 1, θ) a grid locaios [, L], L >. Codiioig o hese icremeally obaied ierrogaios defies he sequece of models [u, u f 1, θ, Ψ],..., [u, u f 1,..., f N, θ, Ψ]. I Secio 3.3 we margialize over f = (f 1,..., f N ) o obai he probabiliy model, or probabilisic soluio, [u θ, Ψ, N] = [u, u, f θ, Ψ, N] du df, (2) which coracs o he rue deermiisic soluio u as N. Compuaioal complexiy is show i Secio 3.2 o be of he same order as comparably sampled explici umerical solvers, ad scalabiliy is illusraed i Secio 5 for he chaoic Navier Sokes sysem ivolvig he soluio of over 16, coupled siff ordiary differeial equaios. The probabilisic soluio defies a rade-off bewee discreizaio ucerimsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

4 4 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Figure 1: Exac soluio (gree lie) of ODE (6); 1 draws (grey lies) from p{u() Ψ, N} over a dese grid ; firs order explici Euler approximaio (red lie), obaied usig equally-spaced domai pariios of sizes N = 5, 1, 2 (lef o righ). aiy ad compuaioal cos. To illusrae his poi, Figure 1 compares he exac, bu a priori ukow soluio wih sample rajecories from he proposed probabiliy model (2) ad a firs order umerical scheme obaied over progressively refied discreizaio grids. The exac soluio lies wihi a high poserior desiy regio of (2), ad as he grid is refied yieldig more iformaio abou he model, kowledge of he soluio icreases, coceraig o he exac soluio. Srog poserior o-saioariy of he error srucure (2) is illusraed i Figure 2 for he hree saes of a sysem whose rajecory is resriced aroud a chaoic aracor (deails are provided i Secio 5.1). I Secio 4 we embed discreizaio uceraiy i he ifereial hierarchy, [θ, u, Ψ, Σ y, N] [y u, θ, Σ] }{{} likelihood [u θ, Ψ, N] }{{} soluio uceraiy [θ, Ψ, Σ], (3) }{{} prior where Σ icludes parameers of he observaio process, such as he covariace srucure of he error erms. We adop a forward simulaio approach for he middle layer wihi a Markov chai Moe Carlo (MCMC) algorihm o sample from his poserior disribuio. The imporace of explicily modellig discreizaio uceraiy wihi he iverse problem is illusraed i Secio 4.1 by he oliear poserior depedece bewee solver hyperparameers Ψ ad differeial equaio model parameers θ. This approach provides a useful exploraory ool o diagose he presece of discreizaio effecs. The problem of poserior uder-coverage resulig from igorig discreizaio iduced soluio uceraiy i (3) is illusraed i Secio 5.4. Discussio of various exesios ad implicaios for he emergig field of probabilisic umerics is provided i Secio 6. Algorihmic ad heoreical deails are described i he Suppleme (Chkrebii e al., 216). MATLAB implemeaios of all examples are provided a gihub.com/ochkrebii/uqdes. 1.2 Noaio We use of he mahemaical framework for defiig umerical problems o fucio spaces developed hrough he work of Suar (21), Kaipio ad Somersalo (27), O Haga (1992), ad Skillig (1991). We deoe parial derivaives usig subscrips o idicae he variable wih respec o which a parial derivaive is ake. For example imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

5 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 5 Figure 2: 1 draws from (2) for he Lorez63 model wih fixed iiial saes ad model parameers i he chaoic regime. Red verical lies i u(1) () correspod o four ime pois used i he diagrams i he las wo rows. 2 uxx := x 2 u refers o he secod parial derivaive of u wih respec o x. Superscrips i pareheses deoe elemes of a vecor, for example, u(k) is he kh eleme of he P dimesioal vecor u. The mehodology proposed i his paper is based o a sequeially updaed probabilisic soluio, cosequely we use superscrips wihou pareheses, e.g., mk, o refer o k-imes updaed fucios. Superscrips o radom variables, e.g., uk, represe a sample from is k-imes updaed disribuio. For oaioal simpliciy, we omi he depedece of he differeial equaio soluio u, ad derivaive u o θ, Ψ, N, ad x ad/or whe he coex makes his depedece clear. imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

6 6 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios 2 Prior model for a fixed bu ukow soluio This secio is cocered wih probabiliy modellig of iiial beliefs abou he ukow fucio, u : D Θ R p, saisfyig a ODE iiial value problem of he form, u = f (, u, θ), [, L], L >, (4) wih fixed iiial value u() = u (). We describe how o icorporae boudary ad regulariy assumpios io a flexible Gaussia process (GP) prior model (Rasmusse ad Williams, 26; Suar, 21) for he ukow soluio. Prior beliefs are updaed i Secio 3 by codiioig o iformaio abou he differeial equaio model obaied over a size N pariio of he domai D accordig o a chose samplig scheme. 2.1 GP prior o he soluio of a ODE iiial value problem GPs are a flexible class of sochasic processes ha ca be defied joily wih heir derivaives i a cosise way hrough he specificaio of a appropriae covariace operaor (Solak e al., 23). We model uceraiy joily for he ime derivaive i (4) ad is soluio, (u, u), usig a GP prior wih covariace srucure, (C, C ), ad mea fucio, (m, m ). The margial meas mus saisfy he cosrai m (z)dz = m (). The covariace srucure is defied as a flexible covoluio of kerels (see, e.g., Higdo, 1998, 22). A covariace operaor for he derivaive u evaluaed bewee locaios j ad k is obaied by covolvig a square iegrable kerel R λ : R R R wih iself, C ( j, k ) = α 1 R λ ( j, z)r λ ( k, z)dz, R parameerized by he legh-scale λ > ad prior precisio α >, o be discussed i Secio 3.4. A example is he ifiiely differeiable kerel, R λ ( j, k ) = exp{ ( j k ) 2 /2λ 2 }, which decreases expoeially wih squared disace bewee j ad k weighed by he legh-scale λ. Is covoluio yields he squared expoeial covariace C ( j, k ) = πα 1 λ exp{ ( j k ) 2 /4λ 2 }. A example of a kerel wih bouded suppor is R λ ( j, k ) = 1 (j λ, j+λ)( k ), which yields he piecewise liear uiform covariace C ( j, k ) = {mi( j, k ) max( j, k ) + 2λ} 1 (, ) {mi( j, k ) max( j, k ) + 2λ}. Deails are provided i Suppleme D.4.A margial covariace operaor, C, o he sae is obaied by iegraig C wih respec o evaluaio pois j ad k or, equivalely, by covolvig he iegraed kerel Q λ ( j, k ) = j R λ(z, k )dz wih iself: C ( j, k ) = α 1 Q λ ( j, z)q λ ( k, z)dz. R Defiig he covariace over he sae by iegraio esures ha C (, ) =, which eforces he boudary codiio, u() = m () (see Suppleme D.1; a aleraive way o eforce he boudary cosrai is o obai he derivaive process u by differeiaio of u ad codiio he joi prior for (u, u) o he exac iiial sae, u ()). This iroduces aisoropy over he sae ha is cosise wih exac kowledge of he soluio a he iiial boudary ad icreasig uceraiy hereafer. The cross-covariace imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

7 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 7 erms are defied aalogously as j C (z, k )dz = α 1 R Q λ ( j, z)r λ ( k, z)dz ad k C ( j, z)dz = α 1 R R λ ( j, z)q λ ( k, z)dz, where each is he adjoi of he oher. The iiial joi GP prior probabiliy disribuio for he soluio a a vecor of evaluaio imes k ad is ime derivaive a a possibly differe vecor of evaluaio imes j, give fixed hyperparameers Ψ = (m, m, α, λ) is, [ ] u ( j ) GP u( k ) ( [m ] ( j ) m, ( k ) [ ]) C k ( j, j ) C ( j, z)dz C (z, j )dz C. (5) ( k, k ) k This prior modellig sraegy is sraighforwardly geeralized o explici iiial value problems of he form (1) by defiig he prior joily o he sae ad he required higher order derivaives. As a simple example, cosider he secod order iiial value ODE problem, { u () = si(2) u, [, 1], u () =, u() = 1. Is exac soluio, u () = { 4 cos() + 2 si() si(2) + cos()}/(4 1), is assumed ukow a priori. The firs colum of Figure 3 shows five draws from he margial desiies of he prior [u (), u()] defied over a fie grid [, 1] T usig a squared expoeial covariace wih (λ, α) = (.8, 5). Secio 3 describes a sequeial updaig procedure of he prior model o obai he probabilisic soluio for his example. (6) 3 Updaig prior beliefs for fixed bu ukow ODE soluio The prior model (5) o he ukow soluio ad is derivaive will be updaed codiioally o model iformaio colleced over he ordered pariio s = (s 1,..., s N ) of he domai [, L]. Defie a model ierrogaio a ime s as he oupu f f{s, u 1 (s ), θ} from he righ had side of equaio (4) evaluaed a a sample u 1 (s ) from he margial prior [u(s ) f 1 ]. Model ierrogaios are geeraed sequeially based o sae samples draw from progressively refied prior models i order o accommodae fas chagig dyamics. Because he ODE model (4) defies he soluio implicily, we are uable o measure he exac sae or derivaive, ad herefore boh he model ierrogaios ad heir error srucure will ake io accou icreasig uceraiy wih ime. This secio begis by expadig o hese fucio esimaio seps wih full deails of he Bayesia updaig algorihm. We illusrae he forward soluio algorihm o he simple example (6) wih kow closed form soluios before cosiderig more realisic sysems. Secio 3.2 examies modellig choices for which his procedure ca aai compuaioal complexiy of O(N), while Secio 3.3 esablishes he cosisecy ad firs order covergece properies of his approach. Choice of hyperparameers is addressed i Secio 3.4. Specific exesios o he full geeraliy of (1) are preseed i Secios 4 ad 5. imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

8 8 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Figure 3: Five draws (solid lies) from he margial desiies of (2) over he sae (firs row) ad is derivaive (secod row) for ODE (6) afer =, 12, 24, 36, 48 ieraios (from lef o righ) of Algorihm 1 wih discreizaio grid of size N = 5; sep-ahead samples u ad f are show as circles; exac soluio is show as a doed red lie. 3.1 Model updaig based o ierrogaios from sampled saes This secio describes a sequeial samplig scheme o ierrogae he model afer each updae. Model ierrogaios occur a pre-deermied discreizaio grid locaios s = (s 1,..., s N ), while he resulig sequece of poserior soluio models ca be compued a a possibly differe vecor of ime pois,. There are may ways o geeralize his basic oe-sep-ahead samplig scheme, as discussed i Secio 6. Firs updae The derivaive of he exac soluio o he boudary, s 1 =, ca be obaied exacly by evaluaig he righ had side of (4) a he kow iiial codiio u (s 1 ) as, f 1 f {s 1, u (s 1 ), θ} = u (s 1 ). The firs updae of he prior probabiliy model (5) is performed by codiioig o he exac derivaive f 1. This yields he joi codiioal predicive probabiliy disribuio a evaluaio imes j ad k, [ u ( j ) u( k ) ] f 1 GP ( [m ] [ 1 ( j ) m 1, ( k ) ]) C 1 k ( j, j ) C 1 ( j, z)dz C 1 (z, j )dz C 1, (7) ( k, k ) k where margial meas ad covariaces a he vecors of ime j ad k are give by: m 1 ( j ) = m ( j ) + C ( j, s 1 )C (s 1, s 1 ) 1 { f 1 m (s 1 ) }, m 1 ( k ) = m ( k ) + C (s 1, s 1 ) 1 k C o (z, s 1 )dz { f 1 m (s 1 ) }, imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

9 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 9 C 1 ( j, j ) = C ( j, j ) C ( j, s 1 )C (s 1, s 1 ) 1 C (s 1, j ), { C 1 ( k, k ) = C k } { ( k, k ) C (z, s 1 )dz C (s 1, s 1 ) 1 k C (z, s 1 )dz}. Closed forms for meas ad covariaces are obaied by iegraio (alog wih crosscovariaces, Suppleme D.1). We refer o his predicive probabiliy measure as he firs updae of he algorihm ad use i as he prior for he ex ieraio of he algorihm. Secod updae The derivaive of he exac soluio is ukow beyod s 1 =. We herefore iroduce he radom variable f 2, represeig a model ierrogaio a he subseque ime sep, s 2 >. A realizaio of f 2 is obaied by drawig a sample u 1 (s 2 ) a ime s 2 from he margial predicive disribuio of (7), u(s 2 ) f 1 N ( m 1 (s 2 ), C 1 (s 2, s 2 ) ), ad he applyig he rasformaio from he righ had side of equaio (4), f 2 f { s 2, u 1 (s 2 ), θ }. The uceraiy i u(s 2 ) f 1 meas ha f 2 is o guaraeed o equal o he exac derivaive a ime s 2 (he fucio f is a differeial operaor oly for he ipu u = u ). Ideed, wihou he exac soluio, he probabiliy disribuio of f 2 is uavailable. As wih ay model for a daa geeraig process, he likelihood (error model) should be elicied o a case-by-case basis. Here we provide geeral guidace o cosrucig his likelihood i he absece of addiioal exper iformaio abou he error model for f 2, ad all subseque ierrogaios f. I geeral, we expec ha (i) he discrepacy bewee u () ad f{, u(), θ}, [, L] is zero whe u is he exac soluio; ad (ii) he fucio f is smooh i he secod argume, herefore discrepacy decreases smoohly wih decreasig uceraiy i he derivaive of he sampled sae realizaio u 1 (s 2 ). These facs sugges a spherically symmeric error model for f 2 wih a sigle mode a f 2 = u (s 2 ) ad dispersed as a icreasig fucio of disace from s 1 =. For example, we may cosider he Gaussia error model, f 2 u, f 1 N (u (s 2 ), r 1 (s 2 )), (8) wih mea u (s 2 ) ad variace r 1 (s 2 ) = C 1 (s 2, s 2 ). A degeerae special case of his error model is discussed a he ed of his secio. We may ow updae (7) by codiioig o he model ierrogaio f 2 uder he error model (8) o obai he predicive probabiliy disribuio a he vecors of evaluaio imes j ad k, [ u ( j ) u( k ) ] f 2, f 1 GP ( [m ] 2 ( j ) m 2, ( k ) [ ]) C 2 k ( j, k ) C 2 ( j, z)dz C2 (z, k )dz C 2. (9) ( k, k ) j Defiig g 2 = C 1 (s 2, s 2 ) + r 1 (s 2 ), he margial meas ad covariaces a he vecors of evaluaio imes j ad k are give by, m 2 ( j ) = m 1 ( j ) + C 1 ( j, s 2 ) g2 1 { f2 m 1 (s 2 ) }, imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

10 1 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios m 2 ( k ) = m 1 ( k ) + k C 1 (z, s )dz g 1 { 2 f2 m 1 (s 2 ) }, C 2 ( j, j ) = C 1 ( j, j ) C 1 ( j, s 2 ) g2 1 C1 (s 2, j ), { C 2 ( k, k ) = C 1 k } { ( k, k ) C 1 (z, s 2 )dz g2 1 k C 1 (z, s 2 )dz}, where sae mea, covariace, ad derivaive cross-covariaces ca be obaied aalyically via iegraio (see Suppleme D.1). Subseque updaes Subseque updaes, 2 < N, begi by drawig a sample u 1 (s ) a ime s (illusraed by circles i he op row of Figure 3) from he margial predicive poserior [u(s ) f 1,..., f 1 ], whose mea ad covariace are give i Algorihm 1. A realizaio of he model ierrogaio f is cosruced by applyig he model rasformaio (4) o u 1 (s ). The updae is performed relaive o he prior [u, u f 1,..., f 1 ], by assimilaig f uder he Gaussia error model, f u (s ), f 1,..., f 1 N {u (s ), r 1 (s )}, 1 N, (1) where, e.g., r 1 (s ) = C 1 (s, s ), o obai he predicive probabiliy disribuio a he vecors of evaluaio imes j ad k, [ u ( j ) u( k ) ] f,..., f 1 GP ( [m ] ( j ) m, ( k ) [ ]) C k ( j, j ) C ( j, z)dz C (z, k )dz C. (11) ( k, k ) j Defiig g = C 1 (s, s ) + r 1 (s ), he resulig joi predicive probabiliy disribuio is Gaussia wih margial meas ad covariaces: m ( j ) = m 1 m ( k ) = m 1 ( k ) + g 1 C ( j, k ) = C 1 ( j ) + C 1 ( j, s ) g 1 k ( j, k ) C 1 ( j, s ) g 1 C ( j, k ) = C 1 ( j, k ) g 1 { f m 1 (s ) }, C 1 (z, s )dz { f m 1 (s ) }, j C 1 C 1 (z, s )dz (s, k ), { k C 1 (z, s )dz}, ad wih sae mea, covariaces, ad derivaive cross-covariaces agai obaied via iegraio (see Algorihm 1 ad Suppleme D.1). Here he sae ad derivaive, u ad u respecively, may be of arbirary dimesio. Uder a bouded suppor covariace, C ( j, k ) ad C ( j, k ) become sparse, bad diagoal marices. Afer N updaes have bee performed, we obai a joi poserior probabiliy disribuio [u, u f] for he ukow soluio ad is derivaive, codiioal o realizaios of oe rajecory of he model ierrogaio vecor f. Draws from he margial model (2) ca be obaied via Moe Carlo, as described i Algorihm 1, yieldig realized imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

11 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 11 rajecories from he forward model (2) a desired ime locaios. Five draws from he joi disribuio of (2) ad he ierrogaios f a ieraios = 12, 24, 36, 48 are illusraed i he secod hrough fifh colums of Figure 3 for he secod-order ODE iiial value problem (6). For oaioal simpliciy, we will hereafer assume ha he emporal locaios of ieres,, form a subse of he discreizaio grid locaios, s. Poi mass likelihood o model ierrogaios A special case of his procedure models each ierrogaio, f, as he exac ukow derivaive, u (s j ), regardless of grid size, N. Thus, model ierrogaios are ierpolaed i he derivaive space by replacig (8) ad (1) wih he poi mass desiy: P (u (s ) = f ) = 1, 1 N. Because sampled process realizaios ca sill be icosise wih he soluio, compuaioal implemeaio of his aleraive scheme ofe requires usig a arbirary covariace ugge icreasig wih ime, or he use of a rougher covariace fucio ha he smoohess implied by he differeial equaio model. For hese reasos we will resric our aeio o explicily modellig he ierrogaio error, e.g. (8) ad (1). 3.2 Compuaioal complexiy A reasoable modellig assumpio for he sae is ha emporal correlaio decays o zero wih disace. Ideed, mos deermiisic umerical mehods make he aalogous assumpio by defiig a lag of M 1 ime seps, beyod which he soluio has o direc effec. This Markovia assumpio boh adds flexibiliy for modellig very fas-chagig dyamics ad drasically reduces compuaioal cos. Uder a compacly suppored covariace srucure, Algorihm 1 requires oly O(N) operaios by allowig rucaio of he weigh marices employed i he mea ad covariace updaes. I coras, whe employig a covariace srucure wih ubouded suppor, Algorihm 1 aais compuaioal complexiy proporioal o he cube of he umber of ime seps, sice he umber of operaios ca be wrie as a fiie sum over 1 N wih each ieraio requirig a order O( 2 ) operaios. A compuaioal compariso of Algorihm 1 wih a similarly sampled explici firs order solver is provided i he Suppleme C. Alhough probabiliy models for he soluio aai he same compuaioal scalig, he probabilisic mehod is more expesive. However, he model of soluio uceraiy makes up for is added compuaioal cos i pracical applicaios where discreizaio grids cao be refied eough o esure egligible discreizaio effecs. Wihi he iverse problem, he adapive esimaio of legh-scale ca subsaially icrease solver accuracy (see Figure 1 i Suppleme C). 3.3 Covergece We ow cosider he rae a which he sochasic process wih probabiliy disribuio (2) geeraed via Algorihm 1 coceraes aroud he exac soluio of (4). The rae imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

12 12 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Algorihm 1 For he iiial value problem (4), draw oe sample from he forward model (2) over = ( 1,, T ), give Ψ ad discreizaio grid s = (s 1,, s N ). A ime s 1 = iiialize he derivaive f 1 = f{s 1, u (), θ}, ad defie m, m, C, C as above; for = 1 : N do If = 1, se g 1 = C (s 1, s 1 ), oherwise se g = C 1 (s, s ) + r 1 (s ); Compue for each modelled sysem compoe, m (s) = m 1 (s) + g 1 m (s) = m 1 (s) + g 1 C (s, s) = C 1 (s, s) g 1 C (s, s) = C 1 s C (z, s)dz = s C 1 s (s, s) g 1 C 1 (z, s )dz { f m 1 (s ) }, C 1 (s, s ) { f m 1 (s ) }, s C 1 (z, s )dz { s C 1 s (z, s)dz g 1 (s, s ) C 1 (s, s), C 1 C 1 (z, s )dz}, (z, s )dz C 1 (s, s); if < N he Sample oe-sep-ahead realizaio u (s +1 ) from he predicive disribuio o he sae for each modelled sysem compoe, u (s +1 ) p {u (s +1 ) f,..., f 1 } = N {u (s +1 ) m (s +1 ), C (s +1, s +1 )}, ad ierrogae he model by compuig f +1 = f{s +1, u (s +1 ), θ}; ed if ed for Sample ad reur u = (u N ( 1 ),, u N ( T )) GP{m N (), C N (, )}, where s. of covergece is as fas as he reducio of local rucaio error for he aalogously sampled oe-sep explici Euler mehod. Proof is provided i he Suppleme D.2. Theorem 1. Cosider a fucio f(, u()) : [, L] R R ha is coiuous i he firs argume ad Lipschiz coiuous i he secod argume. The sochasic process wih probabiliy disribuio (2) obaied usig Algorihm 1 wih r (s) =, s [, L] ad a covariace kerel saisfyig codiios (D.6) ad (D.7) i Suppleme D.3, coverges i L 1 o he uique soluio saisfyig (4) as he maximum discreizaio grid sep size, h, he legh-scale, λ, ad he prior variace, α 1, ed o zero. The rae of covergece is O(h), proporioal o he decrease i he maximum sep size, h. 3.4 The choice of hyperparameers Aalogously wih choosig a umerical solver, he choice of probabilisic solver hyperparameers mus be based o prior kowledge abou he soluio, such as is smoohess. imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

13 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 13 Moreover, as wih may oparameric fucio esimaio problems, asympoic resuls o covergece may be used o guide our choice of covariace hyperparameers wihi he forward problem. For example, for he explici firs order samplig scheme, he resul of Theorem 1 suggess seig he prior variace, 1/α, ad he legh-scale, λ, proporioally o he maximum disace, h, bewee subseque discreizaio grid locaios. Whe experimeal daa is available, i also becomes possible o obai margial poserior desiies for he hyperparameers as par of he calibraio problem (see Secio 4), allowig daa drive adapaio over he space of model parameers. I coras, deermiisic umerical differeial equaio solvers effecively fix he aalogues of hese hyperparameers, such as he quadraure rule or he sep umber, boh wihi he forward ad iverse problems. 4 Exac Bayesia poserior model iferece The discreizaio uceraiy model i Secio 3 may ow be icorporaed io he calibraio problem. We wish o ifer model parameers θ defiig saes u = (u( 1 ),..., u( T )) from measureme daa y = (y( 1 ),..., y( T )), via he hierarchical model (3), [y u, θ, Σ] ρ {y H (u, θ)}, [u θ, Ψ, N] = p (u θ, Ψ, N), [θ, Ψ, Σ] = π(θ, Ψ, Σ), where he rasformaio H maps he differeial equaio soluio o he observaio space, ρ is a probabiliy desiy, π is a prior desiy, Σ defies he observaio process, such as he error srucure, ad θ may iclude ukow iiial codiios or oher model compoes. We describe a Markov chai Moe Carlo procedure argeig he joi poserior of he sae ad ukow parameers codiioal o a vecor of oisy observaios from [y u, θ, Σ]. Secio 4.1 describes a case where saes are idirecly observed hrough a oliear rasformaio H. Algorihm 2 arges he poserior disribuio (3) by geeraig forward model proposals via codiioal simulaio from Algorihm 1 (Suppleme B provides a parallel emperig implemeaio). Wihi Algorihm 2, proposed sample pahs are codiioally simulaed from p(u θ, Ψ) = p(u, f θ, Ψ)df, removig he depedece of he accepace probabiliy o his iracable margial desiy. Such parially likelihoodfree Markov chai Moe Carlo mehods (Marjoram e al., 23) are widely used for iferece o sochasic differeial equaios (e.g., Golighly ad Wilkiso, 211). As wih umerical based approaches, compuaio of he forward model is he rae limiig sep i Algorihm 2. Compuaioal savigs ca be obaied by exploiig he srucure of he probabilisic solver, such as by argeig progressively refied poserior models i a Sequeial Moe Carlo framework, or usig esemble Markov chai Moe Carlo mehods (Neal, 211) whe mixure compoes of (2) ca be quickly compued. We ow have he compoes for accouig for discreizaio uceraiy i differeial equaio models of aural ad physical sysems. The remaider preses a calibraio problem illusraig poserior sesiiviy o discreizaio uceraiy. imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

14 14 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Algorihm 2 Draw K samples from poserior p(θ, u y, α, λ, Σ) give observaios of he rasformed soluio of a differeial equaio wih ukow parameers θ. Iiialize θ, α, λ π( ) where π is he prior desiy; Usig a probabilisic solver (e.g., Algorihm 1) simulae a vecor of realizaios, u codiioal o θ, α, λ, over he size N discreizaio grid; for k = 1 : K do Propose θ, α, λ q( θ, α, λ) where q is a proposal desiy; Usig a probabilisic solver (e.g., Algorihm 1) simulae a vecor of realizaios, u codiioal o θ, α, λ, over he size N discreizaio grid; Compue he rejecio raio, ρ = p ( y u, θ, Σ ) p ( y u, θ, Σ ) π( θ, α, λ ) π( θ, α, λ ) q( θ, α, λ θ, α, λ ) q( θ, α, λ θ, α, λ ), where he raio of margial desiies, p(u f, θ, α, λ) o p(u f, θ, α, λ ), is omied because he realizaios u ad u are simulaed direcly from hese desiies; Sample v U[, 1] if v < mi(ρ, 1) he Updae (θ, α, λ) (θ, α, λ ); Updae u u ; ed if Reur (θ, α, λ, u). ed for 4.1 Iferece o a delay iiial fucio model The JAK-STAT mechaism describes a series of reversible biochemical reacios of STAT-5 rascripio facors iiiaed by bidig of he Eryhropoiei (Epo) hormoe o cell surface recepors (Pellegrii ad Dusaer-Four, 1997). Afer gee acivaio occurs wihi he ucleus, he rascripio facors rever o heir iiial sae, reurig o he cyoplasm o be used i he ex acivaio cycle. This las sage is o well udersood ad is replaced by he ukow ime delay, τ >. Oe model describes chages i 4 reacio saes of STAT-5 via he oliear delay iiial fucio problem, u (1) () = k 1 u (1) () EpoR A () + 2 k 4 u (4) ( τ), [, 6], u (2) () = k 1 u (1) () EpoR A () k 2 u (2)2 (), [, 6], u (3) () = k 3 u (3) () k 2 u (2)2 (), [, 6], u (4) () = k 3 u (3) () k 4 u (4) ( τ), [, 6], u() = φ(), [ τ, ]. (12) The iiial fucio compoes φ (2) () = φ (3) () = φ (4) () are ideically zero, ad he cosa iiial fucio φ (1) () is ukow. The saes for his sysem cao be measured direcly, bu are observed hrough a oliear rasformaio, H : R 3 Θ imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

15 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 15 Model compoe Prior τ χ 2 6 k i, i = 1,..., 6 Exp(1) λ i, i = 1,..., 4 χ 2 1 α i + 1, i = 1,..., 4 Log-N (1, 1) u (1) () N(y (3) (), 4 2 ) u (i), i = 1,..., 4 GP(, C ) u (i), i = 1,..., 4 1{i = 1}u (1) () + u (i) (s) ds Table 1: Priors o he saes ad parameers, assumed o be a priori idepede. R 4, defied as, { k 5 u (2) () + 2u (3) () } { H {u(), θ} = k 6 u (1) () + u (2) () + 2u (3) () } u (1) () u (3) () { u (2) () + u (3) () } 1, (13) ad parameerized by he ukow scalig facors k 5 ad k 6. The idirec measuremes of he saes, y (j) ( j,i ) = H (j) {u( j,i ), θ} + ε (j) ( j,i ), 1 i T j, 1 j 4, are assumed o be coamiaed wih addiive zero mea Gaussia oise, {ε (j) ( j,i )} 1 i Tj, 1 j 4, wih experimeally deermied sadard deviaios. Aalysis is based o experimeal daa (Swameye e al., 23; Raue e al., 29). As per Raue e al. (29), he forcig fucio EpoR A shares he same smoohess as he soluio sae (piecewise liear firs derivaive) as modelled by a GP ierpolaio. This daa has bee used for calibraig he JAK-STAT pahway mechaism by a umber of auhors (e.g., Schmidl e al., 213; Swameye e al., 23; Raue e al., 29; Campbell ad Chkrebii, 213) uder a variey of modellig assumpios ad ifereial approaches. The iaccuracy ad compuaioal cosrais of umerical echiques required o solve he sysem equaios have led some auhors o resor o coarse ODE approximaios, or o employ semiparameric Geeralized Smoohig mehods. This moivaes aalysis of he srucure ad propagaio of discreizaio error hrough he iverse problem. A furher issue is ha he model (12) ad is varias may be misspecified, which mus be firs ucoupled from discreizaio effecs for furher sudy. Ideed, he above cied aalyses coclude ha he model is o flexible eough o fi he available daa, however i is o clear how much of his misfi is due o model discrepacy, ad how much may be aribued o discreizaio error. The probabilisic approach for forward simulaio of delay iiial fucio problems is described i Suppleme B, exedig he updaig sraegy o accommodae curre ad lagged sae process realizaio iformaio. The probabilisic soluio is geeraed usig a equally spaced discreizaio grid of size N = 5, ad a covariace srucure based o he uiform covariace kerel. Prior disribuios are lised i Table 1. imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

16 16 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Figure 4: Sample pahs (grey lies) of he observaio processes (firs row) obaied by rasformig a sample from he margial poserior disribuio of he saes (secod row) by he observaio fucio (13); experimeal daa (red pois) ad error bars showig wo sadard deviaios of he measureme error. Markov chai Moe Carlo is used o obai samples from he poserior disribuio (3). A parallel emperig sampler (Geyer, 1991) described i Suppleme B efficiely raverses he parameer space of his mulimodal poserior disribuio. A sample from he margial poserior of sae rajecories ad he correspodig observaio process are show i Figure 4. As expeced, he poserior rajecories, which icorporae discreizaio uceraiy i he forward problem, appear more diffuse ha hose obaied i e.g., Schmidl e al. (213), while more flexibly describig he observed daa, wih he excepio of a small umber of ouliers. Correlaio plos ad kerel desiy esimaes wih priors for he margial poseriors of rae parameers k 1, k 2, k 3, k 4, observaio scalig parameers k 5, k 6, ime lag τ, iiial fucio value u (1) (), ad hyperparameers for he soluio uceraiy Ψ = (α, λ) are show i Figure 5. All parameers wih he excepio of he prior precisio α are ideified by he daa, while he rae parameer k 2 appears o be oly weakly ideified. We observe srog correlaios bewee parameers, cosise wih previous sudies o his sysem. For example, here is srog correlaio amog he scalig parameers k 5, k 6 ad he iiial firs sae u (1) (). Imporaly, here is a relaioship bewee he legh-scale λ of he probabilisic solver ad he firs, hird ad fourh reacio raes. Furhermore, his relaioship has a oliear srucure, where he highes parameer desiy regio seems o chage wih legh scale implyig srog sesiiviy o he solver specificaios. The leghscale is he probabilisic aalogue, uder a bouded covariace, of he sep umber i a umerical mehod. However, i aalyses based o umerical iegraio, he choice of a umerical echique effecively fixes his parameer a a arbirary value ha is chose a priori. The resul here suggess ha for his problem, he iferred parameer values are highly ad oliearly depede o he choice of he umerical mehod used. Because we have explicily quaified discreizaio uceraiy, he observed lack of fi may sugges addiioal model refieme may be required. imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

17 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 17 Figure 5: 5, poserior samples of he model parameers usig a probabilisic solver wih a grid of size 5, geeraed usig parallel emperig Algorihm 7 wih e chais. Prior probabiliy desiies are show i black. Noe he complex poserior correlaio srucure bewee model parameers ad solver hyperparameers across he lower rows. 5 Applicaios We exed probabiliy modellig of soluio uceraiy o a variey of forward problems: chaoic dyamical sysems, a ill-codiioed mixed boudary value problem, a siff, high-dimesioal iiial value problem, ad exesios o PDEs. 5.1 Discreizaio uceraiy uder chaoic dyamics The classical Lorez63 iiial value problem (Lorez, 1963) is a hree-sae ODE model of covecive fluid moio iduced by a emperaure differece bewee a upper ad lower surface. Saes u (1), u (2) ad u (3) are proporioal o he iesiy of he fluid s covecive moio, he emperaure differece bewee (ho) risig ad (cold) descedig curres, ad he deviaio of he verical emperaure profile from lieariy respecively. The model describig he ime-evoluio of heir dyamics, u (1) = σ(u (1) + u (2) ), [, 2], u (2) = ru (1) u (2) u (1) u (3), [, 2], u (3) = u (1) u (2) bu (3), [, 2], u = u, =, depeds o dimesioless parameers σ, r ad b. The sadard choice of θ = (σ, r, b) = (1, 8/3, 28) wih iiial sae u () = ( 12, 5, 38) falls wihi he chaoic regime. This sysem has a uique soluio whose rajecory lies o a bouded regio of he phase space (e.g., Robiso, 21, pp ), ad is ukow i closed form. (14) imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

18 18 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Figure 6: 1 poserior realizaios from (17). Lef ad cere: saes, u ad v, evaluaed over a fie grid,, over he domai, D = [.5, 1]. Righ: margial u-ormalized log poserior desiy over he ukow iiial codiio u( =.5). A sample of 1 rajecories geeraed from he probabilisic soluio for his sysem is show i Figure 2 give hyperparameers N = 51, α = N, λ = 2h, h = 2/N, cosa prior mea fucio ad a squared expoeial covariace srucure. There is a shor ime widow wihi which here is egligible uceraiy i he soluio, bu he accumulaio of discreizaio uceraiy resuls i rapidly divergig, ye highly srucured errors. Figure 2 shows poserior samples i he 3 sae dimesios (op row) ad desiy esimaes of he samples for u (1) () (boom row) a 4 disic evaluaio imes. The srucured uceraiy i he forward model (2) is o simply defied by wideig of he high poserior desiy regio, bu isead exhibis a behaviour ha is cosise wih he dyamics of he uderlyig sable aracor. 5.2 Soluio mulipliciy i a mixed boudary value problem Cosider a special case he Lae Emde ODE mixed boudary problem, which is used o model he desiy, u, of gaseous spherical objecs, such as sars, as a fucio of radius,, (Shampie 23). The secod order sysem ca be rewrie as a sysem of a firs order equaios o he ierval [.5, 1] wih mixed boudary values, u = v, v = 2v 1 u 5, u( = 1) = 3/2, v( =.5) = 288/2197, where he oaio u( = 1) ad v( =.5) is used o emphasize ha boudary codiios are differe for each sae. The wo lefmos paels of Figure 6 illusrae mulimodaliy i he poserior soluio, whe muliple fucios saisfy he mixed boudary value problem. These high poserior desiy regios cocerae as he umber of grid pois ad he prior precisio grow, suggesig he poeial for defiig iermediae arge desiies for sequeial ad paricle Markov chai Moe Carlo schemes o sample realizaios from he poserior disribuio over ukow model parameers wihi he calibraio problem. This example highlighs he eed for accuraely modellig fucioal raher ha poi-wise soluio uceraiy. (15) imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

19 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 19 Mixed boudary value problems iroduce challeges for may umerical solvers, which employ opimizaio ad he heory of ODE iiial value problems o esimae he ukow iiial sae, here u( =.5), subjec o he sae cosrais a eiher boudary of he domai of iegraio, here [.5, 1]. The opimizaio over u( =.5) is performed uil he correspodig iiial value problem soluio saisfies he kow boudary codiio, u ( = 1), o wihi a user specified olerace. Cosiderig he mixed boudary value problem (15), he probabilisic soluio cosiss of a iferece problem over he uspecified iiial value, u( =.5), give is kow fial value u ( = 1). The likelihood aurally defies he mismach bewee he boudary value u( = 1), obaied from he realized probabilisic soluio give u ( = 1), as follows, { } u ( = 1) u( =.5), v ( =.5) N m N(u) ( = 1), C N(u) ( = 1, = 1), (16) where m N(u) ( = 1) ad C N(u) ( = 1, = 1) are he poserior mea ad covariace for sae u a ime = 1, obaied via Algorihm 1. The poserior over he saes is, [u, v, u( =.5) u ( = 1), v ( =.5)] (17) [u, v u( =.5), u ( = 1), v ( = 1)] [u( =.5) u ( = 1), v ( =.5)] [u( =.5)]. The possibiliy of mulimodaliy suggess he use of a appropriae Markov chai Moe Carlo scheme, such as parallel emperig (Geyer, 1991), implemeed i Suppleme B, which ca quickly ideify ad explore disjoi regios of high poserior probabiliy. 5.3 Discreizaio uceraiy for a model of fluid dyamics The followig example illusraes ha he probabilisic framework developed may be reliably ad sraighforwardly applied o a high-dimesioal dyamical sysem. The Navier Sokes sysem is a fudameal model of fluid dyamics, icorporaig laws of coservaio of mass, eergy ad liear momeum for a icompressible fluid over a domai give cosrais imposed alog he boudaries. I is a impora compoe of complex models i oceaography, weaher, amospheric polluio, ad glacier moveme. Despie is exesive use, he dyamics of Navier Sokes models are poorly udersood eve a small ime scales, where hey ca exhibi urbulece. The Navier Sokes PDE model for he ime evoluio of 2 compoes of he velociy, u : D R 2, of a icompressible fluid o a orus, D = [, 2π) [, 2π], ca be expressed i spherical coordiaes as, u θ u + (u ) u = g p, (x, ) D [a, b], u =, (x, ) D [a, b], u (j) dx =, (x, ) D [a, b], j = 1, 2, u = u, (x, ) D {}, where := [ ] is he Laplacia operaor such ha u = (u (1) x 2 1 x 2 x 1x 1 +u (1) x 2x 2, u (2) x 1x u (2) x 2x 2 ), ad := [ x 1 + x 2 ] is he gradie operaor such ha u = (u (1) x 1 + u (1) x 2, u (2) x 1 + u (2) x 2 ). The model is paramerized by he viscosiy of he fluid, θ R +, he (18) imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

20 2 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Figure 7: Time evoluio of four forward simulaed realizaios (alog rows) of fluid voriciy, govered by he forced Navier Sokes model (18), over wo spaial dimesios: he agle of he ier rig (horizoal axis) ad ouer rig (verical axis) of a wo dimesioal orus. Agles are expressed i radias. Voriciies are evaluaed a imes = (.2,.4,.6,.8) uis (alog colums). pressure fucio p : D [a, b] R, ad he exeral ime-homogeeous forcig fucio g := cos[( 1 2, 1 2 ) x]. We furher assume periodic boudary codiios, ad viscosiy θ = 1 4 i he urbule regime. The exac soluio of he Navier Sokes boudary value problem (18) is o kow i closed form. We will visualize he probabilisic soluio of he Navier Sokes sysem by reducig he wo compoes of velociy o a oe dimesioal fucio: he local spiig moio of he icompressible fluid, called voriciy, which we defie as, ϖ = u, where u represes he roaioal curl (he cross produc of ad u), wih posiive voriciy correspodig o clockwise roaio. We discreize he Navier Sokes model (18) over a grid of size 128 i each spaial dimesio. A pseudo specral projecio i Fourier space yields 16,384 coupled, siff ODEs wih associaed cosrais (deails provided o he accompayig websie). Figure 7 shows four forward simulaed voriciy rajecories (alog rows), obaied from wo compoes of velociy a four disic ime pois (alog colums). Differeces i he sae dyamics arise by he fial ime show, where he four rajecories visibly diverge from oe aoher. These differeces describe uceraiy resulig from discreizig he exac bu ukow ifiie dimesioal soluio. 5.4 Discreizaio uceraiy for parial differeial equaios Similarly o he ODE case, we model he prior belief abou he fixed bu ukow soluio ad parial derivaives of a PDE boudary value problem by defiig a probaimsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

21 O. A. Chkrebii, D. A. Campbell, B. Calderhead, ad M. A. Girolami 21 biliy measure over mulivariae rajecories, as well as firs or higher order derivaives wih respec o spaial ipus. The prior srucure will deped o he form of he PDE model uder cosideraio. We illusrae discreizaio uceraiy modellig for PDEs by cosiderig he parabolic hea equaio describig hea diffusio over ime alog a sigle spaial dimesio implicily via, u (x, ) = κu xx (x, ), [,.25], x [, 1], u(x, ) = si (xπ), =, x [, 1], u(x, ) =, (,.25], x {, 1}, wih coduciviy parameer of κ = 1. Noe ha his PDE has he explici form (1) wih A = {u, u } ad B = u xx. Oe modellig choice for exedig he prior covariace o a spaio-emporal domai is o assume separabiliy by defiig covariaces over ime ad space idepedely. Accordigly we choose emporal kerel R λ as i Secio 2.1 ad le R ν : R R R be a possibly differe spaial kerel fucio wih legh-scale ν. We iegrae he spaial kerel oce o obai Q ν (x j, x k ) = x j R ν (z, x k )dz ad wice o obai S ν (x j, x k ) = x j Q ν (z, x k )dz. The covariace srucures bewee emporal ad spaial evaluaio pois [x j, j ] ad [x k, k ] are obaied from he process covoluio formulaio wih emporal ad spaial prior precisio parameers α ad β respecively, as follows, Cxx([x j, j ], [x k, k ]) = α 1 β 1 R R ν (x j, z)r ν (x k, z)dz R Q λ ( j, w)q λ ( k, w)dw, C ([x j, j ], [x k, k ]) = α 1 β 1 R S ν (x j, z)s ν (x k, z)dz R R λ ( j, w)r λ ( k, w)dw, C ([x j, j ], [x k, k ]) = α 1 β 1 R S ν (x j, z)s ν (x k, z)dz R Q λ ( j, w)q λ ( k, w)dw, wih cross-covariaces defied aalogously. The mea fucio ad covariace specificaio uiquely deermie he joi GP prior o he sae ad is parial derivaives give hyperparameers Ψ, which iclude α, β, λ, m xx, m, m, a vecors of emporal ad spaio-emporal locaios (x j, j ), (x k, k ), (x l, l ) as: u xx (x j, j ) m xx(x j, j ) C 1,1 C 1,2 C 1,3 u (x k, k ) GP m (x k, k ), C 2,1 C 2,2 C 2,3, (2) u(x l, l ) m (x l, l ) C 3,1 C 3,2 C 3,3 where, C 1,1 = Cxx([x j, j ], [x j, j ]), C 1,2 = x k z k Cxx([x j, j ], [w, k ])dwdz, C 1,3 = x l z Cxx([x j, j ], [w, l ])dwdz, C 2,2 = C ([x k, k ], [x k, k ]), C 2,3 = l C ([x k, k ], [x l, w])dw, C 3,3 = C ([x l, l ], [x l, l ]). Whe iegraed covariaces are available i closed form (e.g., Suppleme D.4, he above marices ca be compued aalyically. A geeralizaio of he sequeial updaig scheme preseed i Algorihm 1 is oulied i Suppleme B. The forward i ime, coiuous i space (FTCS) samplig scheme akes seps forward i ime, while smoohig over spaial model ierrogaios (19) imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216

22 22 Bayesia Soluio Uceraiy Quaificaio for Differeial Equaios Figure 8: Uceraiy i he soluio of he hea equaio discreized usig wo spaioemporal grids i blue ad red respecively: 15 5 ad Spaial poserior predicios are show a = (.2,.12,.22). The exac soluio is show i gree; error bars show he empirical mea ad 2 sadard deviaios compued from 5 simulaios. a each sep. I Figure 8 he exac soluio for he PDE iiial value problem (19) wih κ = 1 o he domai [, 1] [,.25] is compared wih he probabiliy model of uceraiy give wo differe grid sizes. The simulaios illusrae ha as he mesh size becomes fier, he uceraiy i he soluio decreases. The characerizaio of spaial uceraiy is criical for more complex models i cases where here are compuaioal cosrais limiig he umber of sysem evaluaios ha may be performed. We illusrae is effec by performig poserior iferece for he parameer κ from daa simulaed from a exac soluio wih κ = 1 ad zero mea Gaussia error wih sadard deviaio of.5. Figure 9 shows he poserior disribuio over κ obaied by usig boh a deermiisic forward i ime, cered i space (FTCS) fiie differece solver ad a probabilisic solver uder a variey of discreizaio grids. For he probabilisic soluio, a squared expoeial covariace srucure was chose over he spaial domai, wih legh-scale 1.5 imes he spaial sep size. For compuaioal efficiecy a uiform covariace srucure was chose over he emporal domai, wih legh-scale 2 imes he emporal sep size. The prior precisio was se o 1,. Noe he chage i he scale as he mesh is refied ad he poserior variace decreases. The use of a deermiisic solver illusraes he problem of poserior uder-coverage ha may occur if discreizaio uceraiy is o ake io accou ad oo coarse a grid is employed relaive o he complexiy of model dyamics. If he discreizaio is o fie eough, he approximae poserior assigs egligible probabiliy mass o he rue value of κ, i coras o he ubiased poserior obaied whe employig he exac soluio for iferece. I his illusraive seig, he use of a probabilisic solver propagaes discreizaio uceraiy i he soluio hrough o he poserior disribuio over he parameers. By usig a probabilisic solver wih he same coarsely discreized grid i his example, he poserior desiy is more diffuse bu approximaely cered o he rue value of κ. imsar-ba ver. 214/1/16 file: BA117.ex dae: Ocober 25, 216