Introduction to spatial data analysis

Transcription

1 Introduction to spatial data analysis 3 Scuola di Dottorato in Economia Sapienza 2018 Instructors: Filippo Celata (and Luca Salvati) Spatial statistics: - f(location, distance..) - to identify invisible geographical properties of data (eg. distribution patterns) - spatial association: to verify the degree of similarity of spatial events which are a function of their distance John Snow s map of Cholera London, 1854 Types of spatial association: 1. That are due to spatial dependence between geographical features (eg. Similar plants require similar soils) 2. That are due to spatial autocorrelation: the presence of a certain event increases the probability of finding similar events nearby, due to a reciprocal influence or «real contagion» (eg. Similar plants cluster because they are generated by other similar plants) Methods: A. To analyze the spatial distribution of a pre-selected set of similar event (point patterns or point processes) (eg. Firms owned by foreign born) B. Autocorrelation analysis: the degree to which nearby features are more similar than distant ones (to identify relations between proximity and intensity)

2 A. POINT PROCESSES: spatial distribution of events in a point pattern (or scheme) -> Cluster analysis -Spatial cluster:the spatial distribution of (similar) events (points) is (more) clustered (than a complete spatial random distribution, and/or than the general/global distribution of the process. Eg. Diseases due to local causes). -Clustering: a general tendency of (similar) events to colocate -Hot-spot:areas with an anomalous concentration of similar events A. Point processes analysis and clustering: to verify if the spatial distribution of (similar) events is clustered, dispersed (uniform or inhibitory) vs. the complete spatial randomness hypothesis random (concentrated*) Identification/mapping of local clusters (eg. density maps) vs. global indexes of the overall degree of clustering uniform / inhibitory clustered B. Spatial autocorrelation First law of geography (Tobler) = "Everything is related to everything else, but near things are more related than distant things." Complete spatial randomness (Diggle, 1983) = the event has the same probability to locate anywhere = - The number of events in any subregion is distributed as a Poisson -The location of events is not depending upon the location of similar events (indipendence) - The number of events in two nonoverlapping regions are independent 3) The average number of events per unit area (intensity) is homogeneous throughout the area (spatial statitionery) SPATIAL AUTO CORRELATION: the degree to which nearby geographical features are similar (vs. the complete spatial randomness hypothesis: CSR) Random distributions implies a certain degree of concentration and/or clustering. This distribution is clustered whenever the degree of concentration is higher than what we would expect in case of complete spatial randomness. Different techniques imply different CSR hypothesis

3 Some spatial statistics tools (ArcGIS and Crimestat) Events (spatial distribution) Intensity (spatial dependence) Global indexes Average nearest neighbour (Multi scale) K Ripley Global indexes of autocorrelation: Moran s I Spatial clustering Geary s C Kernel density maps Local indicators of spatial association (LISA): Local indexes Nearest neighbour hierarchical clustering Local Anselin of Moran s I (Cluster and outlier analys.) Risk Adjusted Nearest Neighbor Hierarchical Clustering Getis Ord Gi (Hot spot analysis) Example: The clustering of migrant entrepreneurs: segregation /assimilation vs the ethnic enclaves hypothesis (Portes and Wilson. "Immigrant Enclaves, American Journal of Sociology, 1980) Clustering: global indexes (to measure the global degree of clustering for the whole set of events) -> methods based on quadrats (joint count) vs. on distances AVERAGE NEAREST NEIGHBOUR: the distance between events is less (clustering) or more (pattern inibitorio) of the expected distance in case of complete spatial randomness? (Clark-Evans, 50s) Nearest neighbour ratio = observed mean distance / expected mean distance (CSR) -> Input: Points: unweighted (= 1) / Projected coordinate system! (Polygons and lines: convert into points with x, y = centroids)

4 Output: -Observed Mean Distance -Expected Mean Distance - Nearest Neighbor Index -Graphic report - Test variables: -> Toolbox / Spatial statistics / Analyzing patterns p-value: probabilty of the spatial distribution to be random z-score: standard deviation of the real values from expected values Bivariate point patterns : co-agglomeration, co-location, competition/cooperation, related variety: Bivariate/Cross K function, Pairwise interaction point process.. Crimestat, R.. Multi-variate point patterns ( ). -> Bivariate point patterns analysis for each couple of patterns Risk-Adjusted Nearest Neighbor Hierarchical Spatial Clustering (Rnnh) (Crimestat) Clustering index in which the probability of identifying clusters for certain categories of events is assessed in relation to the spatial distribution of all events, by using an interpolation between the (kernel) density surfaces of the primary file (e.g. crimes) and the secondary files (eg. population) Clustering processes at different scales In the figure: 10 clusters of first order, 8 clusters of second order, 3 of third order, and so on..

5 NEAREST NEIGHBOR HIERARCHICAL CLUSTER: constantdistance clustering routine for non-weighted events, hierarchical: first order clusters are considered points which may cluster at the second order and so on, until criteria are satisfied (for each order). RIPLEY'S K-FUNCTION: A multi-scale global spatial clustering analisys to identify clustered/inhibitory/random point patterns at different scales/distances between points (Ripley 1976, 1981 Spatial statistics ) -> to confirm/reject the null/random hypothesis at various scales/distances + to dientify the scale/distance where the clustering/inhibition is more intense/weak K = expected number of events / real number of events In case of complete spatial randomness: K(d) = πd2 : Output (dbf, shp): n. cluster, mean center, deviational ellipse and convex hull (spezzata) of points beloning to each cluster, area and cluster density. Results are heavily influenced by the identified first order clusters Linearization of the K function: L function (Besag 1977) In case of complete spatial randomness: L(d) = d (ArcGIS): Or L(d) = 0 -> (Crimestat) K value (clustering) Confidence interval Expected value Confidence interval

6 Ripley s K Lower and upper confidence envelops: beyond which results may be considered significant Confidence envelops are estimated thanks to the reiteration of a Montecarlo simulation (Crimestat: 100 simulations; ArcGIS: 0 / 9%, 99% o 99,9% of the confidence interval). Corollary: simulations work better if the number of points is not small (> 100) Spatial statistics / Analyzing patterns / Multi-Distance Spatial Cluster Analysis Maximum distance Crimestat: SQRT(A)/3 ArcGIS:? Distance ranges Crimestat: 100 ArcGIS: from 1 to 100 (or: beginning distance + distance increment ) (K function) other parameters: Weight field: default: 1, fixed: weight (number of events at each point). The weighted estimation gives different results (clustering is likely to be higher)!: points cannot have distance=0* Problems with the analysis of spatial data #1: -Study area extension (if too small, the analysis may not include elements which are important to provide an exhaustive explanation. If too big, the spatial distribution pattern may be due of a diversity of processes which have nothing to do with what we want to explain. Example: suburban, scattered and low density urban areas). Is an area sensitive tool: results are influenced by the area extension Study area methods: Default: minimum enclosing rectangle User provided: via polygonial layer -> «Study Area» -> reduce the size of the area Creat a mask of the area within the GRA (ring road) by selecting (manually) the zone urbanistiche within the GRA and exporting the selection as mask_area.shp

7 Problems in the analysis of spatial data #2: Boundary problems: given the probability of non observed events beyond the study area s boundaries (with a similar or dissimilar spatial distribution), con distribuzione spaziale simile o dissimile), clustering near the boundaries is under-estimated. Boundary correction methods: NONE: because events are only to be found within the boundaries. Or because the point layer is wider than the study area: points beyond the boundaries of the study area are used for estimating the K function (!!!) SIMULATE_OUTER_BOUNDARY_VALUES: simulate a «mirrored» distribution of points beyond the bounadries REDUCE_ANALYSIS_AREA: reduces the study area. RIPLEY'S_EDGE_CORRECTION_FORMULA: those points whose distance from the boundary is smaller than to other points, are weighted more (good only for non irregular study areas) Output: table(+ Display result graphically): - ExpK (K expected value in case of CSR), - Envelopes (confidence intervals), - ObservedK (value of K) - DiffK (ObservedK-ExpK) Cautions: - Works better for clustered than for inhibitory processes - It s mainly a tool for identifying second-order clusters, i.e. localized clusters, intra-regional scales or medium distances. - Not reliable for small numbers of events (>30, >100) - Not reliable for strongly irregular areas (if it s not possible to solve adequately the boundary problem) Space-time Ripley s K Problems with the analysis of spatial data #3 Apparent contagion (or attraction): clustering is due to exogenous conditions vs. real contagion (or attraction): concentration is due to attraction or interdependency among the clustered features (Anselin)

8 Global indexes of spatial autocorrelation Why to estimate the degree of spatial autocorrelation: - To understand the process (or the variety of processes..) which explain the geographical distribution of intensities - To estimate the degree to which nearby features potentially influence each other (=interaction, interdependence, attraction, contagion, clustering, segregation, etc ) - To verify the degree to which the observed variables are (not) statistically indipendent (eg. autocorrelation reduces the dataset s information content or obscures what is specific about each area, because intensities in one area are partially influenced by what is happening nearby) - (Eg. to test the spatial autocorrelation of models residuals) - (Eg. to assist in the identification of the spatial sample size) Exploratory Spatial Data Analysis (and mapping) vs. Modelling (formal verification and testing of hypothesis) Spatial auto-correlation: global indexes Moran s I Spatial autocorrelation (MORAN S I): Global co-variance index adapted from the analysis of the memory effect in time series (Moran 40s, Whittle 1954). Measures the gobal degree of similarity between the (upper and lower) intensities (-/+) of nearby features Xi X = intensity in point Xi average intensity (Xi-X)(Xj-X): Cross-product, high if values are similar Wij: spatial weights (/influences) matrix * Clustered/high autocorrelation if I is high (I>0), dispersed/low autocorrelation if I is low (I<0), vs. the CSR hypothesis Iexp=-[1/(n-1)]

9 Spatial statistics / Analyzing patterns / Spatial autocorrelation (Moran s I) Conceptualization of spatial relationships: Inverse distance (squared): spatial relationships between features are inversely proportional to their (squared) distance. Computational problems with small distances (crimestat: adjust for small distances ) and no threshold (n to n) Fixed distance band: within the threshold (band) any feature weights 1. Appropriate in the case of non-uniform polygons, and for large point datasets. Zone of indifference: neighbors (or features within the distance threshold) weight 1. Other features weight is inversely proportional to their distance. Appropriate as above, when the influence of distant features is relevant. Computational problems. Polygon contiguity (adjacency!): considers only bordering features (1 if bordering, 0 all the others). Appropriate only for regular polygons (original Moran s I. Generalized by Cliff and Ord Widely used in spatial econometrics) Conceptualization of spatial relationships (2): Spatial statistics / Modeling spatial relationships / Generate spatial weight matrix Distance Band or Threshold Distance (mostly for large datasets): threshold beyond which influence is null (with inverse distance = i) 0: all features are considered; ii) Empty: applies a default threshold distance (min distance at which any feature has a neighbour); iii) defined by the user Weights Matrix from file: uses a spatial weight matrix file (.swm) created/adapted by the user Spatial weight matrix Table in.swm format in which any cell includes an expression of the distance, time, cost, influence, spatial relationship between any couple of features (presence/absence or intensity)

10 Conceptualization of spatial relationships (3): INVERSE_DISTANCE: ( ) + Exponent (!), eg. 2 FIXED_DISTANCE: ( ) K_NEAREST_NEIGHBORS: considers only a K number of the most proximate features CONTIGUITY_EDGES_ONLY: considers only features which share a boundary ( rooke ) CONTIGUITY_EDGES_CORNERS: considers only features which share a boundary and/or vertex ( queen ) ROW STANDARDIZATION: values in the spatial weight matrix are standardized in order for their sum to be = 1. To avoid the indexes to be influenced by the different number of nearby features: appropriate in the case of sample data and compulsory in the case of polygon contiguity, because (irregular) polygons have a different and arbitrary number of bordering features. Test variables: Z-score = standard deviation / p-value DELAUNAY_TRIANGULATION: create overlapping triangles connecting polygons centroids, and considers only features which share a triangle s vertex.. CONVERT_TABLE: to specify spatial relationships in a table [Convert spatial weight matrix to table (utilities)] Normality: the Z-score displays a normal distribution? Output: -Moran s index - Expected index - Variance - Z-score e p-value Cautions: -Significant only above a certain number of features (> 30) Local indexes of spatial association/autocorrelation Vs. Geary s autocorrelation index (Moran is more robust) HIGH/LOW CLUSTERING (Getis & Ord). The probability for high or low values (+) to be clustered or dispersed (similar to Average Nearest Neighbour)

11 4. LOCAL INDEX OF SPATIAL AUTOCORRELATION To measure the degree of autocorrelation for each geographical feature (where and which features?) Input: polygons (crimestat) and points(arcgis) Output: Anselin local of Moran s I (Anselin L. 1995, Local indicators of spatial association LISA. Geographical Analysis 27, ) To attribute to each feature a degree of high/low autocorrelation based on its (high/low) intensity being similar/dissimilar to nearby features Z: intensity, S: variance, W: spatial weight matrix Cluster type (COType) identifies (and renders): - Features which are part of high (HH) or low (LL) values clusters, because nearby features have similar values, and are statistical significant (positive and high z-score). - outlier features, with high or low values, surrounded by features with low (HL) or high (LH) values, and are statistical significant (low and negative z-score) Grado di segregazione tra aree a prevalenza di imprenditori cinesi e aree a prevalenza di imprenditori italiani Contributo Anselin locale alla local segregazione of Moran s tra aree a I prevalente of the presenza distance of unità condotte entrepreneurs da imprenditori from cinesi o their italiani country of origin Spatial statistics / Mapping clusters / Cluster and outlier analysis

12 Local indexes of spatial autocorrelation (2): Getis-Ord Gi, high/low clustering (Hot Spot Analysis) Identifies features which are part of hot spots : areas with unusual clustering of high or low values (Cliff & Ord, Spatial autocorrelation, 1973), based on the value of the GiZScore (categorized according to the standard deviation: the higher the GiZscore, the more nearby features have high values, and viceversa. (You may do a density map of using the Z-Score as weight) Cautions: - reliable only with large dataset (>30 features) - test problems (the significativity test is based on global indexes of spatial autocorrelation) Spatial statistics/mapping clusters/cluster and outlier analysis LAB: identify those Airbnb listings which are part of clusters where the degree of overall satisfaction is particularly high or low. Method: inverse distance. Threshold distance=1000 mt + MAPPING

13 LAB: a spatial analysis of public schools quality in Rome = to what extent school quality depends on the context? = what is the degree of spatial autocorrelation of school quality? Input: spatial17/addxy/schools_roma_xy_dprv.dbf = a table with XY coordinates of all primary and secondary schools in Rome, including a (normalized) «deprivation index» f(dropouts, repetitions, students to teachers ratio, students per classroom, foreign students ratio). 1. Georefer the Schools_Roma dbf using Add XY data + export the data output setting its coordinate system «as the data frame» 2. Estimate the global autocorrelation of the normalized deprivation index, using arctoolbox/spatial statistics/analyzing patterns/moran s I, and setting all the parameters.. LAB: what is the degree of spatial autocorrelation of school quality? 2. Estimate the global autocorrelation of the normalized deprivation index, using the Moran s I. Parameters: Input feature class: schools Input field = «DPRV_NORM» Conceptualization of spatial relationships:? Row standardization:? Threshold distance: meters Generate report -> Verify the graphic report and test variables: what is the result? Is this statistically significant? Do high or low quality schools cluster in certain zones, and where? -> Local indicators of spatial autocorrelation LAB: a spatial analysis of public schools quality in Rome = To what extent school quality depends on the context? Do high or low quality schools cluster in certain zones, and where? 1. Identify and render those schools which are part of clusters of nearby low or high quality schools using arctoolbox/spatial statistics/mapping clusters/cluster and outlier analysis Input: Schools with data shapefile, input field: DPRV_NORM Spatial relationships: Inverse distance -> Modify the symbology of the ouput layer in order to visualize only the schools in clusters of high or low and significant spatial autocorrelation values -> Open and verify the ouput layer attribute table -> In a copied layer, represent the value of the index (L_Milndex) disregarding of the degree of significance - > Check (and try to make sense) of outliers

14 Proximity = concentration, agglomeration, interaction, attraction, influence, contagion, interdependency, similarity, clustering, spatial autocorrelation, segregation, etc. -Buffering Proximity analysis and distance decay function Distance and spatial interaction: interaction opportunities decrease more than proportionally as the distance between interacting features increases = distance decay function (tiranny of distance) Inverse distance decay: b > 1 = eg. 2 (interaction opportunities are inversely proportional to the square distance)

15 Exponential distance decay: Problems with standard (discrete, regional, a spatial) concentration measures (eg. GINI index) 1) MAUP (modifiable area unit problem): the degree of concentration is influenced by the spatial partition and spatial resolution of data (Geographical concentration and spatial autocorrelation) 2) The degree of concentration is not function of the degree of polarization of the most dense regions (Arbia 2001)

16 Concentration vs. polarization Degree of concentration (..more) problems with the analysis of spatial data Degree of spatial auto correlation Example of the modifiable area unit problem (MAUP): Gerrymandering (distortions due to the shape of electoral partitions) The modifiable area unit problem (MAUP): any geographical discontinuity is artificial, (more or the less) arbitrary, modifiable, and influences the results and explanation - Scale problem, f(spatial resolution). E.g. statistical relations are stronger the lower is the degree of spatial resolution, because variance is lower = the more we aggregate data, the stronger they correlate. The more we disaggregate date (and increase spatial resolution), the more the variance and the risk this is due to chance or mistakes - Zoning problem, f(geodata geometry), for any given number of zones, results are influenced by their shape The urban (and mostly liberal) concentration of Columbus, Ohio, located at the center of the map, is split into thirds, each segment then attached to - and outnumbered by - largely conservative suburbs. -Non uniformity: a uniform geographical partition, will be non uniform in terms of statistical attributes, and viceversa (e.g. population). Data in less dense areas are less reliable. -Irregularity, vs. compactness (e.g. administrative divisions)

17 And.. - Ecological fallacy: the results of aggregate analysis cannot be attributed to each individuals, or to higher scales (the rate of suicides is higher where more catholics live = catholics more keen to suicide?) - Outliers: very frequent in spatial data. The higher the spatial resolution of data, the more the probability of outliears. - Geodata quality (accuracy, completeness, consistence, resolution..) Specific problems: measurement mistakes are not indipendent (e.g. population subtracted from an area is attributed to the neighbour). The more dense the areas, the lower the data quality (but the lower the distortion due to measurement mistakes) - Categorial data: spatial analysis tools for categorial data are still largely missing - Coincident locations (distance = 0) -> collect events (to turn coincident points of unique events into weighted points) Spatial interpolation: to obtain surface data from point sample observations ArcGIS desktop/online Help.. ArcGIS desktop/online Help (2)

18 Help!!! arcgis.com esri.com/en/ esri.com esri/arcgis/ Textbooks: - Mitchell A. (2005), The ESRI Guide to GIS Analysis: Volume 1. Volume 2: Spatial Measurement and Statistics. Esri Press - Rogerson P.A. (2010), Statistical Methods for Geography. Sage - Pielou E.C. (1977) Mathematical Ecology. Wiley. - Cliff A.-Ord J. (1981) Spatial Processes, Models and Applications. Pion. - Upton G., Fingleton B. (1985) Spatial data analysis by examples, Wiley. - Arbia G., Espa G. (1996) Statistica economica territoriale. Cedam. - Haining P. (2001) Spatial Data Analysis: Theory and Practice. Cambridge University Press. - Fotheringham A.S., Rogerson P.A. (2009) The Sage handbook of spatial analysis. Sage. - Chun Y., Griffith D.A. (2013), Spatial Statistics and Geostatistics. Sage R applications: Bivand R.S., Pebesma E.J., Gómez Rubio V. (2008) Applied spatial data analysis with R. Springer. Going more in-depth..: - Cressie N. (1993) Statistics for spatial data. Wiley. - Diggle P.J. (2003) Statistical Analysis of Spatial Point Patterns (2nd edition). Hodder Arnold - Diggle P.J.-Ribeiro P.J. (2007) Model-based Geostatistics. Springer. - LeSage J., Pace R.K. (2009), Introduction to Spatial Econometrics. Chapman and Hall. - Fischer M.-Getis A. (2010) Handbook of applied spatial analysis. Springer