Fast algorithms that utilise the sparsity of Q exist (c-package ( Q 1. GMRFlib) Seattle February 3, 2009

Transcription

1 GMRF:s Results References Some Theory for and Applications of Gaussian Markov Random Fields Johan Lindström 1 David Bolin 1 Finn Lindgren 1 Håvard Rue 2 1 Centre for Mathematical Sciences Lund University & 2 Department of Mathematical Sciences NTNU, Trondheim Seattle February 3, 2009 GMRF:s Results References Overview Gaussian Markov random fields Basics Approximating Matérn covariances INLA Fast estimation Results The Sahel Vegetation Precipitation Depth Data Gaussian Markov Random Fields (GMRF:s) A Gaussian Markov random field (GMRF) is a Gaussian random field with a Markov property. The neighbours Ni to a point si are the points that in some sense are close to si. The Gaussian random field x N ( Ñ,Q 1) has a joint distribution that satisfies p(xi {xj : j i}) = p(xi {xj : j Ni}). j / Ni xi xj { xk : k / {i,j} } Qi,j = 0. The density is p(x) = Q 1/2 exp (2Ô) n/2 ( 1 2 (x Ñ) Q(x Ñ) ) Fast algorithms that utilise the sparsity of Q exist (c-package GMRFlib) See Rue and Held (2005) for extensive details on GMRF:s. GMRF:s Simplified expressions Formulating the density through the precision matrix instead of through the covariance matrix simplifies several common expression for Gaussian models. Conditional expectation {xi xj,j I} N ( Ñ I Q 1 I,I Q I,J(xJ Ñ J ),Q 1 I,I ). Hirearchical modelling x θ N ( Bθ,Q 1), θ N ( Ñ,Q 1 ) θ, [ ] ( [BÑ ] [ x Q QB N, θ B Q B QB + Q θ ] 1 ) Ñ

2 GMRF:s Simplified expressions Formulating the density through the precision matrix instead of through the covariance matrix simplifies several common expression for Gaussian models. Conditional expectation {xi xj,j I} N ( Ñ I Q 1 I,I Q I,J(xJ Ñ J ),Q 1 I,I ). Hirearchical modelling x θ N ( Bθ,Q 1), θ N ( Ñ,Q 1 ) θ, [ ] ( [BÑ ] [ x Q QB N, θ B Q B QB + Q θ ] 1 ) GMRF:s Previous limitations How to choose or construct Q? A GMRF might be computationally effective but it is difficult in construct precision matrices that result in reasonable covariance functions for the underlying Gaussian fields. Various ad-hoc methods exist. A common solution is to use a small neighbourhood and let the precision between two points depend on the distance between the points. Rue and Tjelmeland (2002) created GMRF:s on rectangular grids in that approximate Gaussian fields with a wide class of covariance functions. Having the field defined only on a regular grid leads to issues with mapping the observations to the grid points. GMRF:s Previous limitations How to choose or construct Q? A GMRF might be computationally effective but it is difficult in construct precision matrices that result in reasonable covariance functions for the underlying Gaussian fields. Various ad-hoc methods exist. A common solution is to use a small neighbourhood and let the precision between two points depend on the distance between the points. Rue and Tjelmeland (2002) created GMRF:s on rectangular grids in that approximate Gaussian fields with a wide class of covariance functions. Having the field defined only on a regular grid leads to issues with mapping the observations to the grid points. GMRF:s Previous limitations How to choose or construct Q? A GMRF might be computationally effective but it is difficult in construct precision matrices that result in reasonable covariance functions for the underlying Gaussian fields. Various ad-hoc methods exist. A common solution is to use a small neighbourhood and let the precision between two points depend on the distance between the points. Rue and Tjelmeland (2002) created GMRF:s on rectangular grids in that approximate Gaussian fields with a wide class of covariance functions. Having the field defined only on a regular grid leads to issues with mapping the observations to the grid points. Ñ

3 Matérn covariances (Bertil Matérn, ) The Matérn covariance family on u R d : r(u,v) = C(x(u),x(v)) = 2 21 Ò (Ò) ( v u )Ò K Ò ( v u ) with scale (inverse range) > 0 and shape/smoothness Ò > 0, and K Ò a modified Bessel function. Fields with Matérn covariances are solutions to an SPDE Whittle (1954) based on the Laplacian, =, ( 2 ) /2 x(u) = Ø 2 (u), where (u) is spatial white noise, and = Ò + d/2. Parameter link: 2 = Ø 2 (Ò) SPDE issues Non-uniqueness: If x(u) is a solution to the SPDE for = 2, so is x(u) + c exp( e u), for any unit length vector e and any constant c. Non-stationarity: On a bounded domain, the SPDE solutions are non-stationary, unless conditioned on suitable boundary distributions. Practical solution to the non-uniqueness and non-stationarity: Zero-normal-derivative (Neumann) boundaries reduce the impact of the null-space solutions. Resulting covariance, for Ï = [0, L] R: C(x(u),x(v)) rm(u,v) + rm(u, v) + rm(u,2l v) = rm(0,v u) + rm(0,v + u) + rm(0,2l (v + u)). SPDE issues Non-uniqueness: If x(u) is a solution to the SPDE for = 2, so is x(u) + c exp( e u), for any unit length vector e and any constant c. Non-stationarity: On a bounded domain, the SPDE solutions are non-stationary, unless conditioned on suitable boundary distributions. Practical solution to the non-uniqueness and non-stationarity: Zero-normal-derivative (Neumann) boundaries reduce the impact of the null-space solutions. Resulting covariance, for Ï = [0, L] R: C(x(u),x(v)) rm(u,v) + rm(u, v) + rm(u,2l v) = rm(0,v u) + rm(0,v + u) + rm(0,2l (v + u)). The finite element method A stochastic weak formulation of the SPDE states that [ k, ( 2 ) /2 x ] k=1,...,n D = [ k, W ] k=1,...,n for each set of test functions { k}. We use N simple piecewise linear test functions, equal to the piecewise linear basis functions, x(u) = j k(u)wj, and compute the resulting distributions by explicitly calculating the expectation vector and precision matrix for w (= x). Ý For = 2, the weak formulation can be written Kw = [ ] i, 2 )Ýj w = D [ k, W ] i,j=1,...,n ( k=1,...,n ( ) 2Ò (4Ô) d/2

4 Construction of Q With the help of Green s first identity, Ci,j = Ýi, Ýj, Ki,j = Ýi, ( 2 )Ýj = 2 C i,j + Ýi, Ýj, for Neumann boundaries. Markovified Least Squares and Galerkin solutions: C = diag( Ýi, 1 ), ( optimal approximation) Q 1, = K, (Least Squares) Q 2, = KC 1 K, (Galerkin) Q, = KC 1 Q 2, C 1 K, = 3,4,... (Galerkin recursion) Neighbourhood radius equals. Simple closed-form expressions for the matrix elements. Approximating Matérn covariances (Lindgren and Rue, 2007) That fields with Matérn covariances are solutions to an SPDE has been used to construct GMRF:s that approximate fields with Matérn covariance for Z +. Well-defined SPDE on curved manifolds (e.g. a globe). Possible to introduce both anisotropy and non-stationarity. Spatially oscillating fields can be introduced via a complex version of the SPDE: (h1 + ih2 (H1 + ih2) )(x1(u) + ix2(u)) = 1(u) + i 2(u) The precision matrix of the approximating GMRF is found using the finite element method on a triangulation of irregularly spaced points. The resulting GMRF is defined on the points of the triangulation, making it suitable for modelling fields that are observed at irregular locations. Approximating Matérn covariances (Lindgren and Rue, 2007) That fields with Matérn covariances are solutions to an SPDE has been used to construct GMRF:s that approximate fields with Matérn covariance for Z +. Well-defined SPDE on curved manifolds (e.g. a globe). Possible to introduce both anisotropy and non-stationarity. Spatially oscillating fields can be introduced via a complex version of the SPDE: (h1 + ih2 (H1 + ih2) )(x1(u) + ix2(u)) = 1(u) + i 2(u) The precision matrix of the approximating GMRF is found using the finite element method on a triangulation of irregularly spaced points. The resulting GMRF is defined on the points of the triangulation, making it suitable for modelling fields that are observed at irregular locations. Approximating Matérn covariances (Lindgren and Rue, 2007) That fields with Matérn covariances are solutions to an SPDE has been used to construct GMRF:s that approximate fields with Matérn covariance for Z +. Well-defined SPDE on curved manifolds (e.g. a globe). Possible to introduce both anisotropy and non-stationarity. Spatially oscillating fields can be introduced via a complex version of the SPDE: (h1 + ih2 (H1 + ih2) )(x1(u) + ix2(u)) = 1(u) + i 2(u) The precision matrix of the approximating GMRF is found using the finite element method on a triangulation of irregularly spaced points. The resulting GMRF is defined on the points of the triangulation, making it suitable for modelling fields that are observed at irregular locations.

5 INLA Fast estimation (Rue and Martino, 2007) Assume an underlying GMRF with (possibly non-gaussian) point observations, i.e. x N ( Ñ( ),Q( ) 1) p(yi x, ) = p(yi xi, ). If the observations are non-gaussian the log-likelihood can be approximated with a Gaussian by a Taylor expansion of the observation density, p(yi xi, ). Do numerical optimisation of the log-likelihood, using the above approximation. Once the MAP-estimator is found, use the Hessian of the log-likelihood to determine the likely variations of the likelihood and do numerical integration. INLA Fast estimation (Rue and Martino, 2007) Assume an underlying GMRF with (possibly non-gaussian) point observations, i.e. x N ( Ñ( ),Q( ) 1) p(yi x, ) = p(yi xi, ). If the observations are non-gaussian the log-likelihood can be approximated with a Gaussian by a Taylor expansion of the observation density, p(yi xi, ). Do numerical optimisation of the log-likelihood, using the above approximation. Once the MAP-estimator is found, use the Hessian of the log-likelihood to determine the likely variations of the likelihood and do numerical integration. INLA Fast estimation (Rue and Martino, 2007) Assume an underlying GMRF with (possibly non-gaussian) point observations, i.e. x N ( Ñ( ),Q( ) 1) p(yi x, ) = p(yi xi, ). If the observations are non-gaussian the log-likelihood can be approximated with a Gaussian by a Taylor expansion of the observation density, p(yi xi, ). Do numerical optimisation of the log-likelihood, using the above approximation. Once the MAP-estimator is found, use the Hessian of the log-likelihood to determine the likely variations of the likelihood and do numerical integration. INLA Fast estimation (Rue and Martino, 2007) Assume an underlying GMRF with (possibly non-gaussian) point observations, i.e. x N ( Ñ( ),Q( ) 1) p(yi x, ) = p(yi xi, ). If the observations are non-gaussian the log-likelihood can be approximated with a Gaussian by a Taylor expansion of the observation density, p(yi xi, ). Do numerical optimisation of the log-likelihood, using the above approximation. Once the MAP-estimator is found, use the Hessian of the log-likelihood to determine the likely variations of the likelihood and do numerical integration.

6 INLA Fast estimation (cont.) The Good: Very fast compared to MCMC. Gives GOOD approximations of the posterior densities, p(xi Y, map), p(xi Y), p( i Y). The Bad: Limited to a few (5-10) hyperparameters,. Gives limited information about interaction between different parameters, p(xi,xj Y), p( i, j Y). The Ugly: Large memory requirments compared to MCMC of a GMRF. This is mainly due to the numerical integration. INLA Fast estimation (cont.) The Good: Very fast compared to MCMC. Gives GOOD approximations of the posterior densities, p(xi Y, map), p(xi Y), p( i Y). The Bad: Limited to a few (5-10) hyperparameters,. Gives limited information about interaction between different parameters, p(xi,xj Y), p( i, j Y). The Ugly: Large memory requirments compared to MCMC of a GMRF. This is mainly due to the numerical integration. INLA Fast estimation (cont.) The Good: Very fast compared to MCMC. Gives GOOD approximations of the posterior densities, p(xi Y, map), p(xi Y), p( i Y). The Bad: Limited to a few (5-10) hyperparameters,. Gives limited information about interaction between different parameters, p(xi,xj Y), p( i, j Y). The Ugly: Large memory requirments compared to MCMC of a GMRF. This is mainly due to the numerical integration. African Sahel The inspiration for the following applications was an article by Eklundh and Olsson (2003). A semi-arid region directly south of the Sahara desert. Starting in the late 1960 s, the area suffered droughts for over twenty years. Recent studies indicate a vegetation recovery.

7 Normalised Difference Vegetation Index Data taken from the NOAA/NASA Pathfinder AVHRR Land (PAL) data set. NDVI is calculated using satellite measurements of the reflectance from the Earth s surface. The PAL data set contains 36 measurements per year. We use aggregated yearly data. The question is Has the amount of vegetation increased? Figure: NDVI data for the Sahel 1983 Vegetation model (Bolin et al., 2008) Spatial model for measured NDVI: Y(si,t) = xt(si) + it, Yt = AtXt + Et. where xt are latent fields constrained to a sum of temporal trends, xt(si) = m j=1 fj(t) Kj(si) where fj are temporal trend functions with spatially varying regression coefficients Kj The observation matrices At determine which points are observed at each time point Regression model Without the trend constraint, a simple prior model for xt would be a Whittle field with = 0, with precision Q x. The trend restriction provides a natural prior for K = [K 1,...,K m] as K Ø N(0,(ØQ) 1 ) where Q = (F F) Q x, and F = [f1,...,fm] The residual variance is allowed to vary across space, Ë = diag( 2 i ) Collecting the data and observation matrices in a similar manner, Y = AK + E = diag(at)(f In)K + E Posterior distribution The posterior distribution for K given the data Y = [Y 0,...,Y T 1] and the parameters is Ë (K Y,, Ø) N(Ñ K,Q 1 K ), with K A Ë 1 Ñ K = Q 1 Q K = ØQ + A 1 Ë Y, The unknown parameters (Ø, 2 1,..., 2 n) can be estimated using an EM algorithm, exploiting the GMRF structure in the updating equations. A.

8 Linear trend estimation Slope (upper) and Intercept (lower) estimates. K 2 Estimate K 1 Estimate Trend significance Significance estimates for the slope of the linear trends OLS GMRF Significant Negative Non Significant Negative Non Significant Positive Significant Positive Precipitation (Lindström and Lindgren, 2008) Yearly precipitation, in metres, for the 443 stations that have reported measurements for Data from the Global Historical Climatology Network. Irregular triangulation We do not need a dense regular grid for modelling the precipitation. The GMRF construction allows for irregular triangulations, which reduces the computational burden

9 Spatio-temporal model Given a latent precipitation field x, transformed observations are assumed Gaussian: Y(si,t) x N(x(si,t), 2 ). We use an AR(1)-process in time, with spatially correlated noise and spatially varying mean: Xt+1 Ñ = a (Xt Ñ ) + t, S ), X1 N(Ñ, Q 1 S /(1 a2 )), = Bθ. Ñ ( t 2, Ø) N(0, Q 1 and we can write X N(1 Bθ, ( Q t Q s ) 1) Basis functions for the mean transformed precipitation latitude Left Latitude is modelled using a broken linear trend. Right In addition to the broken trend 15 B-spline surface basis functions are used. Hierarchical model graph Ø Ø Ñ θ Q θ Ø 2 a θ X 2 Y Directed acyclic graph describing the resulting hierarchical model. The jointly Gaussian fields X and θ can be integrated out. Markov chain Monte Carlo First we transform the parameters to obtain parameters valid on R: 2 = log( 2 ), ã = log(1 + a) log(1 a), 2 = 2 ), Õ = log(õ), log( Construct a proposal distribution by using a random walk proposal on the transformed variable space. However initial runs on a reduced datasets with coarser triangulation showed strong dependencies in the posterior distribution, leading us to a correlated proposal distribution: Ýnew N ( Ýold, Ë proposal).

10 Markov chain Monte Carlo First we transform the parameters to obtain parameters valid on R: 2 = log( 2 ), ã = log(1 + a) log(1 a), 2 = 2 ), Õ = log(õ), log( Construct a proposal distribution by using a random walk proposal on the transformed variable space. However initial runs on a reduced datasets with coarser triangulation showed strong dependencies in the posterior distribution, leading us to a correlated proposal distribution: Ýnew N ( Ýold, Ë proposal). Markov chain Monte Carlo (cont.) 2 a 2 a 2 2 ã Õ 2 ã 2 Two-dimensional histograms illustrating the dependence between the components of (Ý Y) before (left pane) and after (right pane) the transformation. Computational burden The dominating cost for each MCMC iteration is calculation of the Cholesky factorisation of Q. Inverting a full covariance matrix is O ( n 3), Given a spatial GMRF on a lattice with n points the Cholesky factor is O ( n 3/2), Given a spatio-temporal GMRF on a lattice with n points the Cholesky factor is O ( n 2), How bad is the additional burden for the temporal dependence? n Spatial Spatio-temporal s = s Computational burden The dominating cost for each MCMC iteration is calculation of the Cholesky factorisation of Q. Inverting a full covariance matrix is O ( n 3), Given a spatial GMRF on a lattice with n points the Cholesky factor is O ( n 3/2), Given a spatio-temporal GMRF on a lattice with n points the Cholesky factor is O ( n 2), How bad is the additional burden for the temporal dependence? n Spatial Spatio-temporal s = s Õ

11 Computational burden The dominating cost for each MCMC iteration is calculation of the Cholesky factorisation of Q. Inverting a full covariance matrix is O ( n 3), Given a spatial GMRF on a lattice with n points the Cholesky factor is O ( n 3/2), Given a spatio-temporal GMRF on a lattice with n points the Cholesky factor is O ( n 2), How bad is the additional burden for the temporal dependence? n Spatial Spatio-temporal s = s 46.8 s Kriging results from MCMC simulation of (Ø, 2, a, 2 ) Interpolated yearly precipitation, in metres, for Depth Data We have measurements of seabead depth for an area outside the Swedish coast. The data has been modelled using an underlying GMRF field with Matérn covariance and unknown expectation consisting of an arbitrary plane (e.g. Kriging). The roughly measurement locations have been trinagulated resulting in a grid containing points. Posterior densities for the three hyperparameters of the model and for the underlying field where estimated using INLA. Depth Data We have measurements of seabead depth for an area outside the Swedish coast. The data has been modelled using an underlying GMRF field with Matérn covariance and unknown expectation consisting of an arbitrary plane (e.g. Kriging). The roughly measurement locations have been trinagulated resulting in a grid containing points. Posterior densities for the three hyperparameters of the model and for the underlying field where estimated using INLA. Total estimation time on a Core2Duo desktop: 8 minutes.

12 Depth Data Results GMRF:s Results References Bibliography Bolin, D., Lindström, J., Eklundh, L., and Lindgren, F. (2008), Fast Estimation of Spatially Dependent Temporal Vegetation Trends using Gaussian Markov Random Fields, Submitted to Comput. Statist. and Data Anal. Eklundh, L. and Olsson, L. (2003), Vegetation index trends for the African Sahel , Geophys. Res. Lett., 30, Lindgren, F. and Rue, H. (2007), Explicit construction of GMRF approximations to generalised Matérn Fields on irregular grids, Tech. Rep. 12, Centre for Mathematical Sciences, Lund University, Lund, Sweden. Lindström, J. and Lindgren, F. (2008), Modeling Yearly cumulative Precipitation over the African Sahel using a Gaussian Markov Random Field, In preparation. Rue, H. and Held, L. (2005), Gaussian Markov Random Fields; Theory and Applications, vol. 104 of Monographs on Statistics and Applied Probability, Chapman & Hall/CRC. Rue, H. and Martino, S. (2007), Approximate Bayesian inference for hierarchical Gaussian Markov random field models, J. Statist. Plann. and Inference, 137, Rue, H. and Tjelmeland, H. (2002), Fitting Gaussian Markov Random Fields to Gaussian Fields, Scand. J. Statist., 29, Whittle, P. (1954), On Stationary Processes in the Plane, Biometrika, 41,