WEB application for the analysis of spatio-temporal data

Size: px

Start display at page:

Download "WEB application for the analysis of spatio-temporal data"

Marcia Fleming
5 years ago
Views:

1 EXTERNAL SCIENTIFIC REPORT APPROVED: 17 October 2016 WEB application for the analysis of spatio-temporal data Abstract Machteld Varewyck (Open Analytics NV), Tobias Verbeke (Open Analytics NV) In specific contract No 5 issued under the framework agreement OC/EFSA/AMU/2015/02, EFSA requested Open Analytics to implement software for analysing and visualizing spatio-temporal data. In the exploratory analysis, local statistics (Moran's I and Geary's C) help to recognize potential clusters and hotspots. Smoothed predictions over space and time can be calculated and visualized using ordinary kriging. Logistic regression models allow to include spatial and temporal effects, spatiotemporal interactions and potential covariates. These models are either Bayesian hierarchical models or generalized additive models and several model structures can be compared using model diagnostics. Summary measures for the estimated response values can be visualized over space and time. An interactive component guides the user in uploading data, performing an exploratory analysis and fitting spatio-temporal models. European Food Safety Authority, 2016 Key words: GAM, INLA, kriging, local spatial statistics, spatio-temporal data, R Question number: EFSA-Q Correspondence: amu@efsa.europa.eu EFSA Supporting publication 2016:EN-1102

2 Disclaimer: The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors. Acknowledgements: The authors would like to thank the EFSA staff members: José Cortiñas Abrahantes for the support provided to this scientific output Suggested citation: Varewyck M and Verbeke T, Software for the analysis of spatio-temporal data. EFSA supporting publication 2016:EN pp. European Food Safety Authority, 2016 Reproduction is authorised provided the source is acknowledged. 2 EFSA Supporting publication 2016:EN-1102

3 Table of contents Abstract Introduction Background as provided by the requestor Terms of Reference as provided by the requestor Subject Matter Overview of work performed Literature Study on R-packages for spatio-temporal models Implementation of spatio-temporal analysis software Implementation of the user interface for spatialanalysis Authoring of user manual Deployment References EFSA Supporting publication 2016:EN-1102

4 1. Introduction 1.1. Background as provided by the requestor This contract/grant was awarded by EFSA to: Contractor/Beneficiary: Open Analytics NV Contract: OC/EFSA/AMU/2015/02 Contract number: specific contract 5. The WEB application should be preferably developed in R and should be able to provide different visualization of the data at hand, as well as exploratory analysis to identify potential clustering or hotspots using local statistics. When data is collected over space and time, analysis should take into account the spatial and/or temporal dependence of the observations Terms of Reference as provided by the requestor The WEB application should provide different visualization of the data, exploratory analysis as well as statistical models dealing with the spatio-temporal nature of the data. When data is collected across space (e.g. different countries, NUTS regions, point locations) and possibly over time (i.e. different years, months, weeks, days), analysis should take into account the spatial and/or temporal dependence of the observations. The linear component of the spatio-temporal model for the binary data for a specific response (sample, time, location ) can be written as: ( ) ( ), where is the temporal effect, is the location/spatial effect and is the spatio-temporal interaction term. For the temporal effect, different choices can be made, depending on the data. The idea is to investigate at least a saturated time effect (time is treated as a factor), a linear time effect, a firstorder random walk (RW1), a first-order autoregressive (AR1), and a second-order random walk (RW2). RW1, AR1 and RW2 are flexible smooth functions of time which assumes that the present observation is a function of the immediate past. Specifically, RW1:, AR1: RW2:, where is a correlation parameter and ( ). On the other hand, the Besag, York and Mollie's (BYM) model was fitted to the spatial effect. The BYM model takes into account not only the spatial auto-correlation present in the data (structured spatial effect) but also assumes that the estimates obtained between areas are independent of each other (IID or unstructured effect). The spatial effect of the BYM model was set to have an intrinsic conditional auto-regressive distribution (ICAR), which assumes that the expected value of each area depends on the values of the neighboring areas (in this case, areas sharing boundaries). Thus, areas close together are more similar than areas that are far apart. The spatio-temporal interaction models the relationship between the temporal and spatial trend. In the model different types of interaction were investigated: unstructured, structured over space but unstructured over time, structured over time but unstructured over space, and structured over time and space. 4 EFSA Supporting publication 2016:EN-1102

5 Weighting and incorporation of potential covariates in the model should also be possible to investigate their effects at different spatial-resolution levels. The weighing could be defined as the proportion of planned versus actual sample. Other spatio-temporal models should be included in the tool (see in order to investigate different options when modeling the observed patterns and comparison of model fits should be possible. Selection of models to be included in the tool will be discussed with EFSA and their incorporation as potential Spatiotemporal models will be accompanied by a detailed justification on the inclusion of such models. The inclusion will be based on capacity of the model proposed to incorporate flexible spatial and temporal patterns in combination with assessment of other factors effect on the observed pattern, model outcomes, as well as model performance, in particular of computational requirements. 2. Subject Matter In specific contract No 5 issued under the framework agreement OC/EFSA/AMU/2015/02, EFSA requested Open Analytics to implement a WEB application visualize and analyze spatio-temporal data. The following sections detail the work performed by Open Analytics up until the full delivery of the project. 3. Overview of work performed 3.1. Literature In a first phase Open Analytics has reviewed the literature as a methodological basis for the desired software solution. This included: Lecture notes on exploratory analysis of spatial data by Corey Sparks: Oliver, Margaret A., and Richard Webster. Basic steps in geostatistics: the variogram and kriging. Springer International, Marta Blangiardo and Michela Cameletti. Spatial and Spatio-temporal Bayesian Models with R- INLA. Wiley Simon Wood. Generalized Additive Models: An Introduction with R. Chapman & Hall, Study on R-packages for spatio-temporal models The currently available R-packages for analysing spatio-temporal data were explored, in order to find an alternative to the Bayesian hierarchical models fitted with the R-package INLA. Thereby focus was given to models that can help explaining previous outbreaks and prevent future outbreaks of an infectious disease at specific geographic areas. This is based on spatio-temporal models for a binary response including a set of measured covariates. Thereby space is restricted to polygon regions (at most LAU2 level) and the time level of interest can be day, week, month or year. Covariates data are collected from different sources: some are varying in space and time e.g. meteorological data, some are varying in space only e.g. geographical data (forest/agriculture area ratio, geographical barriers). These high-dimensional data often imply a large computational burden for regression model estimation. However, fitting exploratory models with relatively fast estimation procedures is of interest and at the same time offers a large flexibility in modelling space, time and covariate effects. The R-packages SpatioTemporal, spate, gstat, GWR, GWmodel, spgwr, surveillance and mgcv were considered. It was decided to include generalized additive models (GAMs) from the R-package mgcv as an alternative to Bayesian hierarchical models (BHMs) from the R-package INLA and the main reasons for not retaining the other packages were discussed and are listed below: 5 EFSA Supporting publication 2016:EN-1102

6 Implemented models are for continuous or quantal response type data. It is reported that computation might take long time. Main focus is on equidistant rectangular grid data. When not, alternatives are available but are known to slow down calculations severely. Including covariates will need further pre-processing steps. No specific time effects are implemented, thus only linear or saturated time effect can be included in the model. Not user-friendly in the sense that the numerical estimation of the variogram parameters might be tricky and needs a large degree of attention/experience. Extra biological information necessary to fit the models which might not always be available. Methods depend on the specific disease and population of interest which limits generalizability. Both implemented approaches have the advantage of allowing incorporation of a number of flexible time effects whose performance can be easily compared to each other. Moreover, they handle discrete spatial regions with a neighbouring structure and allow to include effects for measured covariates (linear, saturated and for GAM smooth). Comparing computational performance, it was found that in general the GAMs were fitted much faster than the BHMs. Especially for the more complex models such as for spatial unit LAU1 and time unit week the differences were remarkable. Considering the number of models that converged, the GAMs were also superior to the BHMs. With respect to mean squared prediction error, only indicative conclusions can be drawn as cross-validation was not used. However, in general smaller prediction error for the GAMs compared to the BHMs was found Implementation of spatio-temporal analysis software In a third phase, the back-end code for the spatial statistical engine was developed in line with best practices of R package development. The precise requirements for this application were refined iteratively during work meetings with EFSA and laid down in the relevant meeting minutes. These implementations have been delivered under the form of a formal R package spatialanalysis including technical documentation and integrated automated testing. Some of the main functions included in the package are: readshapedata: Read shape data from zipped file makeunionpolygons: Create all union polygons of the shapedata at the given spatial levels selectpolygons: Select subset of polygons for the given spatial level collapsedata: Collapse data according to the user-defined variable mergealldata: Merge the temporal, spatial and covariates data at lowest space and time level makespacetimedata: Collapse and extend data frame to have one row for each time unit at the spatial level of interest calculatelocal: Calculate local Moran's I or local Geary's C createclasses: Create classes for response according to breaksstyle performkriging: Perform ordinary kriging performinla: Fit Bayesian hierarchical spatio-temporal models with the R-package INLA performgam: Fit Generalized Additive spatio-temporal models with the R-package mgcv 6 EFSA Supporting publication 2016:EN-1102

7 The formal testing of the functions implemented in the package is included in the tests directory of the R package in line with best practices in R package development. This directly includes example computations for all functions Implementation of the user interface for spatialanalysis In a third phase, a user interface was prototyped and developed in line with the EFSA requirements. User interface needs were discussed during EFSA work meetings and conclusions laid down in meeting minutes. The user interface prototypes were shared with EFSA collaborators and uploaded to the EFSA document management system for review. The screenshots in Figures 1 until 7 illustrate the modalities of the spatialanalysis user interface. Figure 1: Screenshot of the spatialanalysis user interface for loading spatial data. Figure 2: Screenshot of the spatialanalysis user interface for defining neighbours. 7 EFSA Supporting publication 2016:EN-1102

Figure 4: Screenshot of the spatialanalysis user interface

8 Figure 3: Screenshot of the spatialanalysis user interface for exploratory analysis. Figure 4: Screenshot of the spatialanalysis user interface for ordinary kriging. 8 EFSA Supporting publication 2016:EN-1102

9 Figure 5: Screenshot of the spatialanalysis user interface for fitting a set of Bayesian hierarchical spatio-temporal models summary of fitted models. 9 EFSA Supporting publication 2016:EN-1102

Figure 7: Screenshot of the spatialanalysis user interface for fitting a Generalized additive

10 Figure 6: Screenshot of the spatialanalysis user interface for fitting a set of Bayesian hierarchical spatio-temporal models visualizing estimated probabilities. Figure 7: Screenshot of the spatialanalysis user interface for fitting a Generalized additive spatio-temporal model interactive visualization of estimated probabilities EFSA Supporting publication 2016:EN-1102

11 3.5. Authoring of user manual For the spatialanalysis application an extensive user manual has been authored by Open Analytics. This manual was submitted for review to EFSA and discussed during EFSA work meetings. The document has been uploaded to the EFSA document management system Deployment The WEB applications have been made available on a dedicated EFSA model hosting. It has already been used in a workshop organized by EFSA on the analysis of spatio-temporal data in the context of African Swine Fever disease for four member states at the end of June Further developments have been incorporated and are now included in the EFSA hosting platform. References Oliver, Margaret A., and Richard Webster. Basic steps in geostatistics: the variogram and kriging. Springer International, Marta Blangiardo and Michela Cameletti. Spatial and Spatio-temporal Bayesian Models with R-INLA. Wiley Simon Wood. Generalized Additive Models: An Introduction with R. Chapman & Hall, EFSA Supporting publication 2016:EN-1102

Bayesian SAE using Complex Survey Data Lecture 4A: Hierarchical Spatial Bayes Modeling

Bayesian SAE using Complex Survey Data Lecture 4A: Hierarchical Spatial Bayes Modeling Jon Wakefield Departments of Statistics and Biostatistics University of Washington 1 / 37 Lecture Content Motivation