How to make R, PostGIS and QGis cooperate for statistical modelling duties

Transcription

1 How to make R, PostGIS and QGis cooperate for statistical modelling duties a case study on hedonic regressions Olivier Bonin UPE IFSTTAR LVMT OGRS 2012

2 Modelling requirements Hedonic models In an hedonic model (Rosen, 1974), the price of a product depends on a vector of its characteristics. When applied to housing, three kinds of characteristics must be taken into account (Kain and Quigley, 1970). p i = α j x ij + β j y ij + γ j z ij + ε i with x ij the structural characteristics, y ij the neighbourhood characteristics, z ij the market location characteristics, and ε i a Gaussian error term.

3 Modelling requirements Data Statistical data (tabular) as well as geographical data: several hundred thousands of records of residential property transactions coordinates of housing locations several GIS layers to compute the spatial characteristics of the dwellings: road networks and public transit networks, location of employment centers, of amenities etc. Construction of x ij from the tabular data (property transactions database). Construction of y ij and z ij from spatial analysis

4 Modelling requirements Spatial dimension of the problem Main difficulty of the spatial analysis: the size of the property transaction database. Necessity to visualize the error term of models to check for spatial auto correlation. Necessity to produce maps on zones rather than on points representing the locations of dwellings.

5 Software setup Statistical software R is the obvious choice. (R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN , ) Extensive set of libraries allowing advanced modelling techniques such as spatial regressions or multi-level modelling (used in Bonin, 2009). Existing connectors with PostGIS and QGis.

6 Software setup Spatial analysis software GIS or spatially-aware RDBMS? GIS and spatially-aware RDBMS? PostGIS proved to be necessary, because of the large amount of data to handle, and of the need for a spatial index. A GIS software seemed to be useful for data visualization; QGis was selected because of its native ability to connect both to PostGIS and to QGis.

7 Software setup How to connect R, QGis and PostGIS? It is theoretically possible to connect the three pieces of software, with bi-directional connexions. Is it simple? Efficient? Useful? Required? R RODBC RDbi PostGIS manager spqr native "add PostGIS layer..." QGis

8 Software setup R PostGIS connexion PostGIS on a GNU/Linux server. R on GNU/Linux, Mac OS X and Windows clients. RODBC: straightforward solution (directly available at CRAN), but depends on ODBC (open source solution on Mac OS X if you like to use the Terminal and to compile libraries), and very slooooow. Rdbi + RdbiPgSQL: hosted on BioConductor, outdated website, but very good performance.

9 Example Mapping data in QGis Rapid transit network and housing locations in the Ile-de-France region (source: notaries and STIF).

10 Example Modelling in R Import from PostGIS into R: a few tens of seconds to load 125,000 records with 87 columns. Linear mixed-effects model fit by REML Data: bien Subset: condap AIC BIC loglik Random effects: Formula: ~1 dep (Intercept) StdDev: Formula: ~1 code_commn %in% dep (Intercept) Residual StdDev:

11 Example Fixed effects: Value Std.Error DF t-value p-value (Intercept) surfh nbppr nbppr anc bi_epoqub bi_epoquc bi_epoqud bi_epoque bi_epoquf bi_epoqug bi_epoquh saldb saldb bi_asceno etage etage etage etage etage garag garag access_fer_n access_fer_n access_metrotrue distc surfh:nbppr surfh:nbppr

12 Example Cartographic representation of the error term The model seems correct. Is there any spatial structure in the error term? As the model is estimated on 250,000 property transactions (possibly with several transactions at the same location), it is necessary to aggregate the error terms on zones to visualize it. We choose here the finest available census level: the IRIS. We transfer the point dataset into PostGIS, and then use sql queries to compute average error terms on the IRIS areas. It is easy, but a little longish: more than 4 minutes.

13 How to make R, PostGIS and QGis cooperate for statistical modelling duties Example Residuals of the hedonic model on housings aggregated at the IRIS level in Ile-de-France

14 Example R mapping capabilities Actually, R can make acceptable maps, with the help of RColorBrewer (for the color palettes), and of the sp and maptools libraries. To tell the truth: much more complicated, but a lot quicker.

15 Example Residuals of the hedonic model on housings aggregated at the IRIS level in Ile-de-France

16 Conclusion Convergence R, PostGIS and QGis can be connected, but they also have many common capabilities (e.g. spatial queries with the help of the GEOS library Geometry engine open source). RDBMS have moved early towards spatial data (first version of PostGIS in 2001), as well as R. R is now able to perform most of the GIS duties (except data acquisition). GIS are moving slowly towards software or libraries that could enhance their data processing and modelling capabilities (or turning into specialized platforms).

17 Conclusion Conclusion Many researchers in social science with quantitative approaches (in my field: geography, regional science, transportation science) heavily rely on software with modelling and analysis capabilities: R for statistical modelling, Netlogo, Repast Simphony or GAMA for agent-based simulation, etc. All these platforms move towards GIS: many libraries in R, NetLogo GIS extension, Repast GIS support, build-in GIS capabilities in GAMA. Classical GIS platforms have to react if they want to remain attractive for these researchers. The USM OrbisGIS plugin (Rousseaux et al., 2012) is a very good step in this direction!