A Note on Effi cient Conditional Simulation of Gaussian Distributions. April 2010

Transcription

1 A Note o Effi ciet Coditioal Simulatio of Gaussia Distributios A D D C S S, U B C, V, BC, C April 2010 A Cosider a multivariate Gaussia radom vector which ca be partitioed ito observed ad uobserved compoetswe review a techique proposed almost twety years ago i the astrophysics literature to sample from the posterior Gaussia distributio of the uobserved compoets give the observed compoets [6] This techique ca be computatioally cheaper tha the stadard approach which requires computig the Cholesky decompositio of the posterior covariace matrix This useful method does ot appear to be widely kow ad has bee rediscovered idepedetly i various publicatios Keywords: forward filterig backward samplig, Gaussia processes, Kalma filter ad smoother, multivariate ormal distributio, state-space models Prelimiary Remark This ote cotais o origial material ad will ever be submitted aywhere for publicatio However it might be of iterest to people workig with Gaussia radom fields/processes so I am makig it publicly available 1 P S Let Z be a R valued Gaussia radom vector such that X Z = Y where X takes values i R x ad Y i R y We assume that Z follows a multivariate ormal distributio of mea m ad covariace Σ with m = mx m y Z N m, Σ ad Σ = Σxx Σ xy Σ T xy Σ yy where m x = E X, m y = E Y, Σ xx = cov X, Σ yy = cov Y ad Σ xy = cov X, Y 1

2 A N E C S G D 2 where It is easy to establish that give Y = y, we have X Y = y N m x y, Σ x y m x y = m x + Σ xy Σ 1 yy y m y, Σ x y = Σ xx Σ xy Σ 1 yy Σ T xy Assume we are iterested here i samplig from N m x y, Σ x y The stadard approach cosists of computig the Cholesky decompositio of Σ x y deoted here Σx y ad usig X = m x y + Σ x y U where U N 0, I is a x -dimesioal vector of idepedet stadard ormal radom variables It is ideed easy to check that X N m x y, Σ x y However, it might too expesive to compute this Cholesky decompositio if x 1 2 M 21 Algorithm The algorithm proposed i [6] to sample N m x y, Σ x y ca be summarized as follows X Sample Z = N m, Σ Y Retur X = X + Σ xy Σ 1 yy y Y Compared to the stadard method, this algorithm bypasses the computatio of the posterior covariace Σ x y ad of its Cholesky decompositio Cotrary to the stadard method, it requires beig able to simulate a radom vector from the prior ad to use a stadard regressio update I may applicatios, it is computatioally much cheaper ad easier to implemet this algorithm tha the stadard method 22 Validity of the algorithm To establish that X N m x y, Σ x y, we ote that X satisfies X = m x y + X E X Y 1 It follows that E X Y = mx y + E X Y E X Y = m x y Hece, we have E X = E E X Y = mx y

3 A N E C S G D 3 We also have cov X Y = cov X E X Y Y = Σ x y as the posterior covariace is idepedet of the specific realizatio of the observatios Hece, we obtai cov X = Σ x y which establishes the validity of the samplig method 3 A To the best of our kowledge, this algorithm first appeared i astrophysics where it was applied to Gaussia radom fields [6]; see [7] for a recet review I this cotext, x is so large that it is virtually impossible to compute Σ x y ad its Cholesky decompositio This algorithm might also prove useful for Gaussia processes applicatios arisig i spatial statistics [2] ad machie learig [8] We preset here two differet applicatios of this algorithm which have bee derived idepedetly from [6] 31 Esemble Kalma filter Cosider a liear Gaussia state-space model satisfyig for 1 X = AX 1 + V, Y = CX + W, where X 0 N 0, Σ 0, V N 0, Σ v ad W N 0, Σ w For ay geeric sequece {z k } k 0, let us deote z i:j = z i, z i+1,, z j We are iterested i the posterior desities {p x y 1: } 1 These posterior desities are Gaussia ad their statistics m x, = E X y 1: ad Σ xx, = cov X y 1: ca be computed usig the Kalma filter However if the dimesio x of the state X is very high, the it is ot possible to implemet the Kalma filter equatios This has motivated the developmet of approximatio techiques i geoscieces A very popular approach i this field is kow as the Esemble Kalma filter [4] I the esemble Kalma filter, the posterior distributios are approximated by radom samples Assume you have at time 1, N samples X i 1 N m x, 1 1, Σ xx, 1 1 i = 1,, N where m x, 1 1, Σ xx, 1 1 are estimates of m x,, Σ xx, The at time, the algorithm proceeds as follows Sample X i N Compute m x, 1 = 1 N m x, 1 AX i 1, Σ v ad Y i N i=1 Xi m T y, 1, Σ yy, 1 = 1 N, m y, 1 = 1 N N i=1 Y i Y i CX i, Σ w i=1 Y i, Σ xy, 1 = 1 N T my, 1 m T y, 1 i=1 Xi Y i T

4 A N E C S G D 4 Compute X i = X i Compute m x, = 1 N m T x, + Σ xy, 1 Σ 1 yy, 1 y Y i i=1 Xi ad Σ xx, = 1 N N i=1 Xi X i T mx, As N goes to ifiity, it follows directly from the previous developmets that X i N m x,, Σ xx, 32 Posterior simulatio i Gaussia state-space models Cosider agai a liear Gaussia state-space model X = AX 1 + V, 2 Y = CX + W, 3 where X 0 N 0, Σ 0, V N 0, Σ v ad W N 0, Σ w Let us deote y 1: = y 1, y 2,, y Whe implemetig a Markov chai Mote Carlo MCMC algorithm to estimate the hyperparameters of this model, it is usually ecessary to sample from p x 0: y 1: This is typically achieved usig the Forward Filterig Backward Samplig FFBS techique [1], [5] A alterative to this well-kow techique is give by the followig algorithm [3] Sample X 0:, Y 1: usig Eq 2-3 Use the Kalma smoother to compute both E X 0: Y 1: ad E X 0: y 1: Retur X 0: = E X 0: y 1: + X 0: E X 0: Y 1: The fact that X 0: p x 0: y 1: follows directly from Eq 1 A mior advatage of this method over the FFBS approach is that it oly relies o stadard Kalma smoothig code Actually, the algorithm discussed i [3] is slightly differet I this paper, the authors propose to sample from p x 0, v 1: y 1: istead of p x 0: y 1: usig the disturbace smoother E V 1: Y 1: The ratioale for samplig from p x 0, v 1: y 1: is that Σ v is typically a low-rak matrix R [1] CK Carter ad R Koh, O Gibbs samplig for state space models, Biometrika, vol 81, pp , 1994 [2] N Cressie, Statistics for Spatial Data, New York: Wiley, 1993 [3] J Durbi ad SJ Koopma, A simple ad effi ciet simulatio smoother for state space time series aalysis, Biometrika, vol 89, pp , 2002

5 A N E C S G D 5 [4] G Everse, Data assimilatio: The Esemble Kalma Filter, 2d ed, Spriger, 2009 [5] S Frühwirth-Schatter, Data augmetatio ad dyamic liear models, J Time Series Aalysis, vol 15, pp , 1994 [6] Y Hoffma ad E Ribak, Costraied realizatios of Gaussia fields - a simple algorithm, The Astrophysical Joural, vol 380, pp L5 L8, Oct 1991 [7] Y Hoffma, Gaussia fields ad costraied simulatios of the large-scale structure, i Data Aalysis i Cosmology, Lecture Notes i Physics, Berli: Spriger- Verlag, pp , 2009 [8] CE Rasmusse ad CKI Williams, Gaussia Processes for Machie Learig, MIT Press, 2006