Bayesian Modeling and Inference for High-Dimensional Spatiotemporal Datasets

Size: px

Start display at page:

Download "Bayesian Modeling and Inference for High-Dimensional Spatiotemporal Datasets"

Bertram Page
5 years ago
Views:

1 Bayesian Modeling and Inference for High-Dimensional Spatiotemporal Datasets Sudipto Banerjee University of California, Los Angeles, USA

2 Based upon projects involving: Abhirup Datta (Johns Hopkins University) Andrew O. Finley (Michigan State University) Nicholas A.S. Hamm (University of Twente) Martjin Schaap (TNO Built Environment and Geosciences)

3 Example 1: U.S. forest biomass data Figure: Observed biomass (left) and NDVI (right) Forest biomass data collected over 114,371 plots Normalized Difference Vegetation Index (NDVI) is a measure of greenness Forest Biomass Regression Model: Biomass(l) = β 0 (l) + β 1 (l)ndvi(l) + error

4 Example 2: European Particulate Matter (PM 10 ) data Northing (km) Northing (km) Easting (km) Easting (km) (a) PM 10 levels in March, 2009 (b) PM 10 levels in June, 2009 Significant variation across space and time Daily observations at 308 stations for 2 years i.e., n = = 224, 840

5 Example 2: European PM 10 data Computer models like Chemistry Transport Model (CTM) consistently underestimate PM 10 levels CTM outputs used as covariates to improve fits log(pm 10 )(l) = β 0 (l) + β 1 (l)ctm(l) + ɛ(l)

6 Example 3: Tanana Valley (Alaska) forest canopy height analysis (a) (b) Figure: Tanana vally, Alaska, study region. (a) G-LiHT flight lines where canopy height was measured at locations over the percent forest canopy covariate. (b) Occurrence of forest fire in the past 20 years and areas of interest for prediction illustration.

7 Spatiotemporal regression models Y(l) = β 0 (l) + X(l)β 1 (l) + e(l) Produce maps for intercept and slope: {β 0 (l) : l L} and {β 1 (l) : l L} L is spatial domain (e.g., D R d ) or spatiotemporal domain (e.g., D R d R + ) Potentially very rich: understand spatially- and/or temporally-varying impact of predictors on outcome. Model-based predictions: Y(l 0 ) {y(l 1 ), y(l 2 ),..., y(l n )}.

8 Gaussian spatiotemporal process {w(l) : l L} GP(0, K θ (, )) implies w = (w(l 1 ), w(l 2 ),..., w(l n )) N(0, K θ ) for every finite set of points l 1, l 2,..., l n. K θ = {K θ (l i, l j )} is a spatial variance-covariance matrix Stationary: K θ (l, l ) = K θ (l l ). Isotropy: K θ (l, l ) = K θ ( l l ). With nugget (esp. when modeling data): K θ = C (σ,φ) + D τ, where θ = {σ, φ, τ} No nugget (esp. when modeling random effects): K θ = C (σ,φ), where θ = {σ, φ}

9 Likelihood from (full rank) GP models L = {l 1, l 2,..., l n } are locations where data is observed y(l i ) is outcome at the i th location, y = (y(l 1 ), y(l 2 ),..., y(l n )) Model: y N(Xβ, K θ ) Estimating process parameters from the likelihood: 1 2 log det(k θ) 1 2 (y Xβ) K 1 (y Xβ) Bayesian inference: Priors on {β, θ} Challenges: Storage and chol(k θ ) = LDL. θ

10 Burgeoning literature on spatial big data Low-rank models (Wahba, 1990; Higdon, 2002; Kamman & Wand, 2003; Paciorek, 2007; Rasmussen & Williams, 2006; Stein 2007, 2008; Cressie & Johannesson, 2008; Banerjee et al., 2008; 2010; Gramacy & Lee 2008; Sang et al., 2011, 2012; Lemos et al., 2011; Guhaniyogi et al., 2011, 2013; Salazar et al., 2013; Katzfuss, 2016) Spectral approximations and composite likelihoods: (Fuentes 2007; Paciorek, 2007; Eidsvik et al. 2016) Multi-resolution approaches (Nychka, 2002; Johannesson et al., 2007; Matsuo et al., 2010; Tzeng & Huang, 2015; Katzfuss, 2016) Sparsity: (Solve Ax = b by (i) sparse A, or (ii) sparse A 1 ) 1. Covariance tapering (Furrer et al. 2006; Du et al. 2009; Kaufman et al., 2009; Shaby and Ruppert, 2013) 2. GMRFs to GPs: INLA (Rue et al. 2009; Lindgren et al., 2011) 3. LAGP (Gramacy et al. 2014; Gramacy and Apley, 2015) 4. Nearest-neighbor models (Vecchia 1988; Stein et al. 2004; Stroud et al 2014; Datta et al., 2016)

11 Reduced (Low) rank models K θ B θ K θ B θ + D θ B θ is n r matrix of spatial basis functions, r << n K θ is r r spatial covariance matrix D θ is either diagonal or sparse Examples: Kernel projections, Splines, Predictive process, FRK, spectral basis... Computations exploit above structure: roughly O(nr 2 ) << O(n 3 ) flops

12 Oversmoothing due to reduced-rank models (a) True w (b) Full GP (c) PPGP 64 knots Figure: Comparing full GP vs low-rank GP with 2500 locations. Figure (4(c)) exhibits oversmoothing by a low-rank process (predictive process with 64 knots)

13 Simple method of introducing sparsity (e.g. graphical models) Full dependency graph p(y 1 )p(y 2 y 1 )p(y 3 y 1, y 2 )p(y 4 y 1, y 2, y 3 ) p(y 5 y 1, y 2, y 3, y 4 )p(y 6 y 1, y 2,..., y 5 )p(y 7 y 1, y 2,..., y 6 ).

14 Simple method of introducing sparsity (e.g. graphical models) 3 Nearest neighbor dependency graph p(y 1 )p(y 2 y 1 )p(y 3 y 1, y 2 )p(y 4 y 1, y 2, y 3 ) p(y 5 y 1, y 2, y 3, y 4 )p(y 6 y 1, y 2, y 3, y 4, y 5 )p(y 7 y 1, y 2, y 3, y 4, y 5, y 6 )

15 Gaussian graphical models: linearity Write a joint density p(w) = p(w 1, w 2,..., w n ) as: p(w 1 )p(w 2 w 1 )p(w 3 w 1, w 2 ) p(w n w 1, w 2,..., w n 1 ) Example: For Gaussian distribution N(w 0, K θ ), we have a linear model w 1 = 0 + η 1 ; w 2 = a 21 w 1 + η 2 ; w 3 = a 31 w 1 + a 32 w 2 + η 3 ; w i = a i1 w 1 + a i2 w a i,i 1 w i 1 + η i ; i = 4,..., n. More compactly: w = Aw + η ; η N(0, D).

16 Simple method of introducing sparsity (e.g. graphical models) For Gaussian distribution N(w 0, K θ ), K θ = (I A) 1 D(I A) D = diag(var{w i w {j<i} }) If L is from chol(k θ ) = LDL, then L 1 = I A. a ij s obtained from n 1 linear systems implied by j<i:j i a ij w j = E[w i w {j<i} ] i = 2,..., n Example: for(i in 1:n) { a[i+1,] = solve(k[1:i,1:i], K[i, 1:i]) }

17 Let a ij = 0 for all but m nearest neighbors of node i implies solving j N[i] a ij w j = E[w i w {j N[i]} ] i = 2,..., n, where N[i] = {j < i : j i} are indices for neighbors of i from its past. Example: for(i in 1:n) { } a[i+1,] = solve(k[n[i],n[i]], K[i, N[i]]) We need to solve n 1 linear systems of size at most m m We effectively model a (sparse) Cholesky factor instead of computing it

18 Sparse precision matrices N(w R 0, K θ ) N(w R 0, K θ ) ; K 1 θ = (I A) D 1 (I A)

19 Sparse precision matrices N(w R 0, K θ ) N(w R 0, K θ ) ; K 1 θ = (I A) D 1 (I A) (a) I A (b) D 1 (c) K 1 θ

20 Sparse precision matrices N(w R 0, K θ ) N(w R 0, K θ ) ; K 1 θ = (I A) D 1 (I A) (a) I A (b) D 1 (c) K 1 θ det( K 1 θ ) = n i=1 D 1 ii, K θ 1 is sparse with O(nm 2 ) entries

21 Sparse precision matrices N(w R 0, K θ ) N(w R 0, K θ ) ; K 1 θ = (I A) D 1 (I A) (a) I A (b) D 1 (c) K 1 θ det( K 1 θ ) = n i=1 D 1 ii, K θ 1 is sparse with O(nm 2 ) entries

22 Sparse likelihood approximations (Vecchia, 1988; Stein et al., 2004) Let R = {l 1, l 2,..., l r } With w(l) GP(0, K θ ( )), write the joint density p(w R ) as: N(w R 0, K θ ) = where N(l i ) H(l i ). r p(w(l i ) w H(li )) i=1 r i=1 p(w(l i ) w N(li )) = N(w R 0, K θ ). Shrinkage: Choose N(l) as the set of m nearest-neighbors among H(l i ). Theory: Screening effect of kriging. K θ 1 depends on K θ, but is sparser with at most nm 2 non-zero entries

23 Extension to a GP (Datta et al., JASA, 2016) Fix reference set R = {l 1, l 2,..., l r } (e.g. observed points) N(l i ) is the set of at most m nearest neighbors of l i among {l 1, l 2,..., l i 1 }. First piece: Model w R N(0, K θ ) ( Vecchia prior ) Second piece: If l / R, then N(l) is the set of m-nearest neighbors of l in R Third piece: w(l) = r i=1 a i (l)w(l i ) + η(l) with a i (l) = 0 if l i / N(l). Nonzero a i (l) s obtained by solving m m system: E[w(l) w N(l) ] = a i (l)w(l i ) i:l i N(l)

24 Neighbors in Space and Time No universal definition of distance in a space-time domain Use K θ (, ) as a proxy for distance Datta et al. (2016, AoAS): Efficient algorithm O(4nm) flops to do this

25 Example 1: Hierarchical NNGP model Start with a desired full GP specification: GP(0, K θ ( )) Derive the NNGP: NNGP(0, K θ ( )) Y(l) ind P θ exponential family ; g(e[y(l)]) = β 0 (l) + X(l)β 1 (l) (β 0 (l), β 1 (l)) NNGP( β 0 + X(l) β 1, K θ ( )) ( β 0, β 1 ) N(0, V β ) ; θ p(θ) Posterior predictive inference for β 0 (l 0 ), β 1 (l 0 ) and Y(l 0 )

26 Example 2: Hierarchical NNGP model Start with a desired full GP specification for Y(l): Y(l) GP(x (l)β, K θ ( )) Derive the NNGP: Y(l) NNGP(x (l)β, K θ ( )) Y N(Xβ, K θ ) ; β N(0, V β ) ; θ p(θ) No need for Cholesky: it is modeled. Easy posterior predictive inference for Y(l 0 ) at new l 0. But no latent spatial-temporal process

27 (a) True w (b) Full GP (c) PPGP 64 knots (d) NNGP, m = 10 (e) NNGP, m = 20

28 RMSPE NNGP RMSPE NNGP Mean 95% CI width Full GP RMSPE Full GP Mean 95% CI width Mean 95% CI width m Figure: Choice of m in NNGP models: Out-of-sample Root Mean Squared Prediction Error (RMSPE) and mean width between the upper and lower 95% posterior predictive credible intervals for a range of m for the univariate synthetic data analysis

29 Back to European PM 10 data Northing (km) Northing (km) Easting (km) Easting (km) (a) PM 10 levels in March, 2009 (b) PM 10 levels in June, 2009 Interest in estimating short and long term temporal (and spatial) decay (to improve the CTMs) log(pm 10 )(s, t) = β 0 + β 1 CTM(s, t) + w(s, t) + ɛ(s, t) w(s, t) DNNGP(0, K θ ( ))

30 European PM 10 Dataset Significantly improved fit OLS DNNGP RMSPE Total time 24 hrs

31 European PM 10 Dataset Northing (km) Missing [0,20) [20,40) [40,60) [60,80) [80,100) [100,120] Northing (km) [0,0.1) [0.1,0.2) [0.2,0.3) [0.3,0.4) [0.4,0.5) [0.5,0.6) [0.6,0.7) [0.7,0.8) [0.8,0.9) [0.9,1] (a) PM 10 for (b) Pr( PM 10 > 50µgm 3 ) Easting (km) Easting (km)

32 European PM 10 Dataset Northing (km) Missing [0,10) [10,20) [20,30) [30,40) [40,50) [50,60) [60,70) [70,80) [80,90] Northing (km) [0,0.1) [0.1,0.2) [0.2,0.3) [0.3,0.4) [0.4,0.5) [0.5,0.6) [0.6,0.7) [0.7,0.8) [0.8,0.9) [0.9,1] (a) PM 10 for (b) Pr( PM 10 > 50µgm 3 ) Easting (km) Easting (km)

33 Concluding remarks: Storage and computation Algorithms: Gibbs, RWM, HMC, VB, INLA; NNGP/HMC especially promising Model-based solution for spatial BIG DATA Never needs to store n n distance matrix. Stores n small m m matrices Total flop count per iteration is O(nm 3 ) i.e linear in n Scalable to massive datasets because m is small you never need more than a few neighbors. Compare with reduced-rank models: O(nm 3 ) << O(nr 2 ). New R package spnngp (on CRAN soon)

34 Concluding remarks: Comparisons Are low-rank spatial models well and truly beaten? Certainly do not seem to scale as nicely as NNGP Have somewhat greater theoretical tractability (e.g. Bayesian asymptotics) Can be used to flexibly model smoothness Can be constructed for other processes e.g., Spatial Dirichlet Predictive Process Compare with scalable multi-resolution frameworks (Katzfuss, 2016) Highly scalable meta-kriging frameworks (Guhaniyogi, 2016) Future work: High-dimensional multivariate spatial-temporal variable selection

35 Thank You!

On Gaussian Process Models for High-Dimensional Geostatistical Datasets

On Gaussian Process Models for High-Dimensional Geostatistical Datasets Sudipto Banerjee Joint work with Abhirup Datta, Andrew O. Finley and Alan E. Gelfand University of California, Los Angeles, USA May