Low-rank methods and predictive processes for spatial models

Transcription

1 Low-rank methods and predictive processes for spatial models Sam Bussman, Linchao Chen, John Lewis, Mark Risser with Sebastian Kurtek, Vince Vu, Ying Sun February 27, 2014

2 Outline Introduction and general notation Bayesian estimation via MCMC Classical estimation via fixed-rank kriging (FRK) Choice of expansion matrix H Predictive processes Evaluation criteria Improvements

3 Introduction: Full-Rank Geostatistical Setup Following Wikle (2010): {η(s)} - mean zero Gaussian Spatial process on domain D c η (s, r) = Cov(η(s), η(r)) Observations at n locations Data model Y = η + ɛ, ɛ Gau(0, Σ ɛ ) Y = (Y (s 1 ),..., Y (s n )) and η = (η(s 1 ),..., η(s n )) η Gau(0, Σ η ) Goal: predict the true process at locations {r 1,..., r m } denoted by η 0 = (η(r 1 ),..., η(r m ))

4 Introduction: Full-Rank Geostatistical Setup We want the predictive distribution [η 0 Y] = [η 0 η][η Y]dη Given the Gaussian assumptions, [η 0 Y] is conveniently Gaussian Mean and var-cov are functions of (Σ η + Σ ɛ ) 1 As we have established, inverting an n n matrix is hard for large n

5 Introduction: Reduced-Rank Hierarchical Parameterization Consider the alternative representation of the spatial model Y = η + ɛ, ɛ Gau(0, Σ ɛ ) η = Hα + ξ, ξ Gau(0, Σ ξ ) α Gau(0, Σ α ) α (α 1,..., α p ) with p << n H is the n p expansion matrix: maps the low dimensional α to the true spatial process η ξ accounts for the difference between η and Hα Model for latent process at unobserved locations is η 0 = H 0 α + ξ 0

6 Introduction: Reduced-Rank Hierarchical Parameterization Again: Gaussian in, Gaussian out [η 0 Y] has mean and var-cov depending on (HΣ α H + V ) 1 Making the usual assumption Σ ɛ = σ 2 I and setting V = Σ ξ + σ 2 I But, we still must invert an n n matrix Two work arounds: MCMC and fixed-ranked Kriging

7 MCMC Approach Consider the posterior [η, α y] induced by the hierarchical model η is Gaussian with mean and var-cov depending on I + Σ 1 ξ ) 1 ( 1 σ 2 ɛ To make this feasible we add the assumption that Σ ξ is diagonal α is also Gaussian. Only need to invert p p matrices Now [η 0 Y] = [η 0 α][α Y]dα where we have [η 0 α] = Gau(H 0 α, Σ ξ0 )

8 Fixed-rank kriging approach Fixed-rank kriging (FRK), from Cressie and Johannesson (2008), outlines how kriging (i.e., spatial best linear unbiased predictor or BLUP) can be applied in this reduced-rank parameterization of a spatial process.

9 Fixed-rank kriging approach Main idea: Primary computational cost in (general) kriging is computing (Σ η + σ 2 ɛ I n ) 1, a dense, n n matrix. For FRK, all we need do is calculate the inverse (HΣ α H + V) 1 = V 1 V 1 H(H V 1 H + Σ 1 α ) 1 H V 1, (using the Sherman-Morrison-Woodbury Formula) which only requires the simple inverse calculation of a diagonal matrix V = σ 2 ɛ I n + Σ ξ (assuming Σ ξ is diagonal) and the inverse of (dense) p p matrices (O(p 3 ) computations). See Cressie and Johannesson (2008) for the closed-form kriging equations.

10 Fixed-rank kriging approach The FRK spatial predictor and corresponding standard errors require the following calculations: 1 H (HΣ α H + V) 1 H, 2 H (HΣ α H + V) 1 a, and 3 (HΣ α H + V) 1 a, where a is a vector of length n. The computational cost for each of these calculations is 1 O(p 3 ), 2 O(np 2 ), and 3 O(np 2 ), so that the overall computational cost is O(np 2 ).

11 Fixed-rank kriging approach Two important components in this model: 1 Σ α, to be estimated from the data Cressie and Johannesson (2008) use a plug-in estimate ˆΣ α, calculated by minimizing a weighted Frobenius norm Classical geostatistical methods: variogram estimation (Zimmerman and Stein (2010)) Choose a parametric form for Σ α Σ α (θ); for low-dimensional θ use maximum likelihood or REML (Zimmerman (2010)) 2 H, determined by the analyst (fixed!) Orthogonal basis functions (so that H H = I), Non orthogonal basis functions (smoothing splines, wavelet, radial, multiresolutional)

12 Choice of expansion matrix H Reminder: η α Gau(Hα, Σ ξ ) α Gau(0, Σ α ) Want dimension of α to be small compared to n. Would love for Σ ξ to have simple form. If H is orthogonal then essentially we have no redundancy or duplicate info. Also, if Σ ξ = σ 2 ξ I then V = (σ2 ξ + σ2 ɛ )I = σ 2 v I so H V 1 H = σ 2 vh H = σ 2 vi which simplifies FRK and MCMC. Typically as p increases, the residual structure will have smaller and smaller scale spatial dependence so that Σ ξ = σ 2 ξ I is reasonable.

13 Choice of expansion matrix H Examples of Orthogonal Basis Functions: Fourier, orthogonal Polynomials, certain wavelets, eigenvectors of a specified covariance matrix How to choose? Not a lot of guidance on this. Driven perhaps by problem specific knowledge.

14 Choice of expansion matrix H Karhunen-Loeve Expansion: The continuous K-L expansion of a covariance function of a mean zero spatial process η(s) over a spatial domain D is c η (s, r) = E[η(s), η(r)] = λ k φ k (s)φ k (r) k=1 where φ k are the eigenfunctions and λ k are the eigenvalues of the Fredholm integral equation: c η (s, r)φ k (s)ds = λ k φ k (r) D Can expand the process using these basis functions: η(s) = k=1 α kφ k (s) (where α k is the projection of η). Truncating this to p dimensions minimizes the variance of the truncation error and thus is optimal. The continuous expansion is typically not used in practice due to discrete nature of data and issues with solving the above integral.

15 Choice of expansion matrix H When you have repeat observations over time (or you are willing to grid your data) then you can calculate an empirical covariance matrix for the spatial process. Then you can perform a PCA on your estimate of Σ η, i.e. ˆΣ η Φ = ΦΛ. The eigenvectors, Φ are known as the empirical orthogonal functions (EOFs). K-L PCA when each observation has equal area of influence. Now let H = Φ p and α = Φ pη where Φ p is now n p containing the first p eigenvectors.

16 Choice of expansion matrix H Nice: No assumption of stationarity required. Σ α is diagonal due to orthogonality If p is large enough to account for larger scale spatial variation then Σ ξ = σξ 2 I is likely reasonable. EOFs give maps of principal spatial structures. Not so nice: Not obvious how to handle prediction since ˆΣ η isn t available at unobserved locations. Have to interpolate eigenvectors somehow or play with ˆΣ η to make it into a valid covariance model with known covariance function. Directly estimating covariance of η is perhaps difficult since we can t observe it directly. If we have replication, can get an estimate of Σ Y and σ 2 ɛ and base our EOFs on ˆΣ Y ˆσ 2 ɛ

17 Choice of expansion matrix H Example: Modeling Migratory Patterns

18 Choice of expansion matrix H The model for the process was as follows: η t = Φα t + ξ t, ξ t Gau(0, Σ ξ ) α t = Mα t 1 + ζ t, ζ t Gau(0, Σ ζ ) M is the state transition matrix.

19 Choice of expansion matrix H Example: Modeling Migratory Patterns: The EOF maps Spatial Map of loc. 22 perc. of variation in first Evector= 0.38 Spatial Map of loc. 22 perc. of variation in second Evector= 0.07 Spatial Map of loc. 22 perc. of variation in third Evector= 0.04 yg xg Time Series of loc. 22 perc. of variation in first Evector= 0.38 A[1, ] yg xg Time Series of loc. 22 perc. of variation in second Evector= 0.07 A[2, ] yg xg Time Series of loc. 22 perc. of variation in third Evector= 0.04 A[3, ] Index Index Index

20 Choice of expansion matrix H Example: Simulation using first 5 EOFs Original values for 2002 Simulated Values yg yg xg xg

21 Predictive Process Model (Banerjee et al., 2008) Univariate Predictive Process Modeling Spatial regression model (parent model) Predictive process Y (s) = x T (s)β + ω(s) + ɛ(s) (1) ω(s) GP(0, C(s, s )), ɛ(s) IID N(0, τ 2 ) Given w = [ω(s i )]m i=1, spatial interpolant (kriging) over S 1 ω(s) = c T (s)c 1 w ω(s) GP(0, C(.)), C(s, s ) = c T (s)c 1 c(s ) Approximate ω(s) in (1) by ω(s). Inference requires inversion of an m m matrix, m << n (Sherman-Morrison-Woodbury Formula). 1 S - n observed locations, S - m selected knots; w MVN(0, C )

22 Predictive process modeling Low-rank approximation Previous notation: the spatial process is approximated by η Hα Under the PP notation: the spatial process is approximated by ω(s) c T (s)c 1 w So, α = w and H = c T (s)c 1.

23 Properties of Predictive Process Models ω(s) is an orthogonal projection of ω(s) onto the linear subspace generated by w. min f E[ ω(s) f (w ) w ] Exact interpolant at the knots Predictive process model has less variability than parent model Minimize reverse KL divergence of posterior p(w a Y ) 2 and its approximations min q KL{q(w a Y), p(w a Y)}, q(y w a ) = q(y w) p(w a Y) p(w a )p(y w) p(w a )q(y w ) = q(w a Y) 2 w a = (w, w ). w is the value over S. w is the value over S

24 Selection of Knots Spatial design Space filling knot selection based on geometric criteria (Nychka and Saltzman, 1998) Spatial balance of design locations is more efficient than simple random sampling. (Stevens Jr and Olsen, 2004) Diggle and Lophaven (2006) suggest augmenting the lattice with close pairs or infills (See Simulation I later) Evaluation of knots performance Comparison of covariance function from parent model with that of predictive process model Details: 200 locations are uniformly generated over a [0, 10] [0, 10] rectangle. Knots consist of a equally spaced grid. Matern covariance with σ 2 = 1, change range and smooth parameters

25 Selection of Knots Comparison of Covariance Function Fix range parameter φ = 2, set ν = 0.5, 1, 1.5, 5 Fix smooth parameter ν = 0.5, set φ = 2, 4, 6, 12 What matters is the size of the range relative to the spacing of the grid for the knots

26 Simulation I Goal: Evaluate the approximation; Choice of m, spatial design Details 3000 irregularly scattered (observed) locations over a domain; stationary anisotropic Matern covariance with smoothness (ν = 0.5), rotation angle (ψ = 45 ), rate of spatial decay (λ 1 = 300, λ 2 = 50) Run predictive process model with m = 144, 256, 529, spatial designs (regular grid; lattice plus close pair configuration; lattice plus infill configuration). For comparison, run parent model. Comparing run time and estimation/prediction results Results 3 parallel chains: 1000 iterations for burnin, 4000 iterations for inference h (m = 144), 1.5 h (m = 256), 4.25 h (m = 529), 18 h (parent model)

27 Simulation I 3 Fig 3(b) Posterior mean surface, lattice plus close pair, 256 knots Fig 3(c) Posterior mean surface, lattice plus infill, 256 knots 3 Table 2: lattice plus close pair design; Prediction - empirical coverage of 95% prediction intervals for a set of 100 hold-out locations

28 Predictive Processes Conclusion from Simulation I Improvement in estimation with increasing m, irrespectively of spatial design Although 144 knots are adequate for capturing β, higher knot densities are required for capturing anisotropic covariance parameters and nugget variance τ 2 Estimation is more sensitive to the number of knots than to the underlying design; Predictions are much more robust Simulation II(Section 5.1.2) locations over the same domain; non-stationary anisotropic Matern covariance (Paciorek and Schervish, 2006); parent model is infeasible computationally Comparison among number of knots, spatial designs gives similar results as Simulation I

29 Extensions Non-Gaussian first-stage models Y (s) Poisson or Bernoulli with E[Y (s)] = µ(s) η(s) = g(µ(s)) = X T (s)β + ω(s) + ɛ(s) Replace ω(s) by predictive process ω(s) Spatiotemporal versions Y (s, t) = x T (s, t)β + ω(s, t) + ɛ(s, t) Replace ω(s, t) by ω(s, t) = c(s, t) T C 1 w 4 Multivariate predictive process modelling (See section 3 (Banerjee et al., 2008) for the model and section 5.2 for real data example. ) 4 w - random variables at m knots over D [0, T ] with covariance matrix C ; c(s, t) - covariances of ω(s, t) with w

30 Performance of low-rank methods Low-rank methods give computational efficiency Obvious question: What do we lose by using the low-rank representation? (How much statistical efficiency is lost?)

31 Performance of low-rank methods Limitations on low rank approximations: Stein (2013) Evaluation criteria: Kullback-Leibler (KL) divergence For two probability measures P and Q, the KL divergence of Q from P, denoted KL(P, Q), equals ( )] dp KL(P, Q) = E P [log = E P [log(dp)] E P [log(dq)]. dq The KL divergence is simply the expected difference in loglikelihood when using model Q instead of P, when in fact P is the correct measure.

32 Performance of low-rank methods Under a mean-zero Gaussian process assumption on the spatial process η; i.e., then P = N n (0, Σ 0 ), Q = N n (0, Σ 1 ), 2KL(P, Q) = tr(σ 1 1 Σ 0) log Σ 1 + log Σ 0 n. The assumption is that P is the true (unknown) model for η and Q is the model we choose.

33 Performance of low-rank methods A low-rank form for Σ 1 is assumed; i.e., Σ 1 = σ 2 ɛ I n + R, where R is n n positive semidefinite with rank at most p << n (computational cost O(np 2 )). Results: Even for optimal R, the KL divergence will be large if the fine scale variations are not well-described by white noise (this would be the ξ term from before). When the nugget (σ 2 ɛ ) is small: the smoother the process, the worse the low-rank approximation does.

34 Performance of low-rank methods Alternative: assume Σ 1 is block diagonal (i.e., assume observations in different blocks are independent; not appropriate for most spatial data); with n/p blocks of size p this requires O ( 1 3 np2) computations. (Surprising) Result: When the error/nugget variance (σ 2 ɛ ) is sufficiently small and observations sufficiently dense, the low-rank approximation performs disastrously and the independent block model much better.

35 Performance of low-rank methods Numerical results Comparison of the low rank and independent block approximations, when the true correlation function is Matérn (stationary/isotropic), for n = 1200: 1 Fixed range, smoothness; vary rank/block size (p) and nugget effect. 2 Fixed smoothness, nugget effect; vary spatial range and rank/block size.

36 Performance of low-rank methods 1. Fixed range, smoothness; vary rank/block size and nugget effect: M.L. Stein / Spatial Statistics ( ) Fig. 1. KL divergences of low rank (curves) and independent block (+symbols) approximations to 1200 observation with spacing 1 and autocovariance function K(x) = e 12 x plus a nugget. Black (curves/+symbols) corresponds t 1200 of 0, medium gray to 0.01 and light gray to 0.1. Low rank approximations are best possible. L divergences of low rank (curves) and independent block (+symbols) approximations to 1200 observations on a line ing 1 and autocovariance function K(x) = e 12 x Fig. 2. Same as Fig. 1 except that the exponential autocovariance function is replaced by the Whittle autocovarianc plus a nugget. Black (curves/+symbols) corresponds to a nugget x K1(12 x ). ium gray to 0.01 and light gray to 0.1. Low rank approximations are best possible. Left: smoothness ν = 0.5 (Exponential); right: smoothness ν = 1 (Whittle) more favorable to the low rank approximation by considering positive nuggets and values smoothness parameter greater than 1. However, in my experience, values of much large are not common for environmental processes. Next consider varying the range of an exponential covariance function. One way to look at th KL divergences of low rank (solid lines) and independent block (+ symbols); of the range parameter is to fix the nugget and consider K(x) = e x for different values o black corresponds to a nuggetspectral of 0, density medium corresponding gray to tok(x) 0.01, = e x and is f (!) light = ( gray, so increasing range (dec 2 +! 2 ) to 0.1. ) corresponds to increasing variation at low frequencies but decreasing variation at suf high frequencies. For evenly spaced observations, the relationship between the eigenvalue covariance matrix and the spectrum is similar to the periodic case considered in Section 2, w Main idea: low-rank is better decay of thewith eigenvalues large being closely nugget, tied to the large decay ofsmoothness. the spectrum. Thus, when there is a effect, we should expect the low rank approximation to perform increasingly well for suf large ranges, especially for larger values of the rank, which is exactly what we see in the le of Fig. 3. Specifically, with the nugget fixed at 0.1 (the largest value in Fig. 1), the independe approximation dominates for short ranges for a wide range of block sizes, but as the range in the KL divergence monotonically increases for the independent block approximation but ev

37 Performance of low-rank methods 2. Fixed smoothness, nugget effect; vary spatial range and rank/block size: M.L. Stein / Spatial Statistics ( ) 11 KL divergences of low rank (black) and independent block (gray) approximations; nugget effect of 0.1. Line types indicate the block size/rank; the solid, dashed, dotted and dashdot curves correspond to B = 3, 12, 50 and 200 respectively. Fig. 3. KL divergences of low rank (black) and independent block (gray) approximations to 1200 observations on a line with 1 spacing and autocovariance function K(x) = e x plus a nugget of 0.1 (left panel) and K(x) = 12 1 e x plus a 1200 nugget of 0.1 (right panel). Black curves correspond to low rank approximations (best possible) and gray to independent block approximations. Line types indicate B, the block size, with the rank of the corresponding low rank approximation equal to B 1; the solid, dashed, dotted and dash dot curves correspond to B = 3, 12, 50 and 200 respectively. Main idea: in the presence of a nugget effect, low-rank performs very well as the range increases (i.e., low-rank approximations can capture long-range dependence well), especially for large rank. observations converges to a diagonal matrix plus a rank 1 matrix (the matrix with all elements equal to 1), so the KL divergence of the low rank approximation will tend to 0 as!1for any B 2. In contrast, the KL divergence for the independent block approximation will not tend to 0 even for

38 Improvements Low rank + taper: Sang and Huang (2012) Low rank performs poorly for short range dependence, tapering performs poorly for long range dependence: use both! Recall the decomposition η = Hα + ξ Use low rank methods to work with Σ α = Cov(α). Use a tapered covariance structure for Σ ξ = Cov(ξ). Then, Cov 1 (η) = (HΣ α H + Σ ξ ) 1 can be computed with less computational burden than full-rank model.

39 Improvements Low rank + GMRF: Nychka et al. (2013) The spatial process is represented as a sum of p independent processes η(s) = p k=1 η k(s), where each η k (s) is defined through a basis function expansion M k η k (s) = c kj φ kj (s). j=1 The {c kj } are stochastic (Gaussian) coefficients and the {φ kj ( )} are fixed radial (compactly supported) basis functions with differing ranges of dependence (multiresolutional). The basis functions are centered on gridded locations; thus the coefficients {c kj } can be estimated using GMRF techniques.

40 Bibliography I Banerjee, S., Gelfand, A. E., Finley, A. O., and Sang, H. (2008). Gaussian predictive process models for large spatial data sets. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(4): Cressie, N. and Johannesson, G. (2008). Fixed rank kriging for very large spatial data sets. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(1): Diggle, P. and Lophaven, S. (2006). Bayesian geostatistical design. Scandinavian Journal of Statistics, 33(1): Nychka, D., Bandyopadhyay, S., Hammerling, D., Lindgren, F., and Sain, S. (2013). A multi-resolution gaussian process model for the analysis of large spatial data sets. Unpublished draft, -(-):. Nychka, D. and Saltzman, N. (1998). Design of air-quality monitoring networks. In Case studies in environmental statistics, pages Springer. Paciorek, C. J. and Schervish, M. J. (2006). Spatial modelling using a new class of nonstationary covariance functions. Environmetrics, 17(5): Sang, H. and Huang, J. Z. (2012). A full scale approximation of covariance functions for large spatial data sets. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74(1): Stein, M. L. (2013). Limitations on low rank approximations for covariance matrices of spatial data. Spatial Statistics, (0):. Stevens Jr, D. L. and Olsen, A. R. (2004). Spatially balanced sampling of natural resources. Journal of the American Statistical Association, 99(465): Wikle, C. K. (2010). Innovation and intellectual property rights. In Gelfand, A., Fuentes, M., Guttorp, P., and Diggle, P., editors, Handbook of Spatial Statistics, Chapman & Hall/CRC Handbooks of Modern Statistical Methods. Taylor & Francis. Zimmerman, D. (2010). Innovation and intellectual property rights. In Gelfand, A., Fuentes, M., Guttorp, P., and Diggle, P., editors, Handbook of Spatial Statistics, Chapman & Hall/CRC Handbooks of Modern Statistical Methods. Taylor & Francis. Zimmerman, D. and Stein, M. (2010). Innovation and intellectual property rights. In Gelfand, A., Fuentes, M., Guttorp, P., and Diggle, P., editors, Handbook of Spatial Statistics, Chapman & Hall/CRC Handbooks of Modern Statistical Methods. Taylor & Francis.