Performing Geospatial Analysis Using Latitude and Longitude Data Ginger Ni - Software Engineer
Hewlett Packard Enterprise confidential information This is a rolling (up to three year) roadmap and is subject to change without notice. This Roadmap contains Hewlett Packard Enterprise Confidential Information. If you have a valid Confidential Disclosure Agreement with Hewlett Packard Enterprise, disclosure of the Roadmap is subject to that CDA. If not, it is subject to the following terms: for a period of three years after the date of disclosure, you may use the Roadmap solely for the purpose of evaluating purchase decisions from HPE and use a reasonable standard of care to prevent disclosures. You will not disclose the contents of the Roadmap to any third party unless it becomes publically known, rightfully received by you from a third party without duty of confidentiality, or disclosed with Hewlett Packard Enterprise s prior written approval.
Please give me your feedback Session ID: B9697 Speaker: Ginger Ni Use the mobile app to complete a session survey 1. Access My schedule 2. Click on the session detail page 3. Scroll down to Rate & review If the session is not on your schedule, just find it via the Discover app s Session Schedule menu, click on this session, and scroll down to Rate & Review If you have not downloaded our event app, please go to your phone s app store and search on Discover 2016 Las Vegas Thank you for providing your feedback, which helps us enhance content for future events.
Overview Vertica geospatial support GEOMETRY and GEOGRAPHY data types Built-in spatial functions WGS84 longitude and latitude data Hurricane risk analysis example Tools and datasets Vertica SQL commands Hurricane damage heat map Image Credit: D. Kahle and H. Wickham. ggmap: Spatial Visualization with ggplot2. The R Journal, 5(1), 144-161. URL http://journal.r-project.org/archive/2013-1/kahle-wickham.pdf 5
Geospatial analysis in daily life People are using technology of Geographic Information Systems (GIS) in their everyday lives. View images on Google Earth Navigate cars using Global Positioning System (GPS) devices Show your locations when you upload pictures to social media Search nearby restaurants on your smart phone Government, companies, and individuals are using, generating and collecting geospatial data. From industry-leading businesses, to individual daily activities, vast amounts of geospatial data are being collected each day. Vertica can help you analyze geospatial data. 6
Spatial analysis with Vertica People who are using or developing GIS for data management and analysis would benefit from the geospatial package in Vetica. Vertica supports two spatial data types, GEOMETRY and GEOGRAPHY, allowing spatial data to be stored and queried. A lot of GIS-enabled applications can work with Vertica. Vertica provides 65+ built-in functions for spatial analysis. Vertica geospatial analysis functions follow the Open Geospatial Consortium (OGC) standards, which are international open standards for geospatial interface and content. Vertica also has additional spatial functions beyond OGC standard functions. 7
Vertica spatial data types Vertica supports two spatial data types that can be used to store geographical objects such as points, lines, and polygons. GEOMETRY : Used to store planar data. It is generally used to store XY coordinates in a two-dimensional space. GEOGRAPHY: Used to store spherical (round-earth) data. It is used to store longitude and latitude coordinates that represent points, lines, and polygons on the earth's surface.
Import/export spatial data Parse and load data from a Shapefile with a COPY command COPY table_name(columns) WITH SOURCE STV_ShpSource(file = filename ) PARSER STV_ShpParser() Exports data from Vertica to a Shapefile STV_Export2Shapefile (columns) USING PARAMETERS shapefile = shapefile_name ) OVER() Import spatial data from WKT/WKB formats (OGC standard geographic data formats) ST_GeomFromText(wkt), ST_GeomFromWKB(wkb), ST_GeographyFromText(wkt), ST_GeographyFromWKB(wkb) Export spatial data as WKT/WKB formats ST_AsText(g), ST_AsBinary(g) Export spatial data as GeoJSON format STV_AsGeoJSON(g)
Vertica spatial analysis functions Vertica provides SQL functions that allow for the creation, comparison, analysis and retrieval of spatial data. ST_<function_name> functions are compliant with OGC standards. (40+ functions) ST_Area ST_Distance ST_Boundary ST_Intersects ST_IsValid STV_<function_name> functions are unique to Vertica and not compliant with OGC standards.(25+ functions) STV_Create_Index STV_Refresh_Index STV_Intersect STV_Export2Shapefile STV_AsGeoJSON 10
Vertica spatial function examples ST_GeomFromText Converts a well-known text (WKT) string into its corresponding GEOMETRY object. Can bulk loading spatial data from a data file with the COPY statement 11
Vertica spatial function examples ST_AsText Creates the well-known text (WKT) representation of a spatial object. The OGC defines the format of a WKT string. ST_SRID Identifies the spatial reference system identifier (SRID) stored with a spatial object. 12
Vertica spatial function examples ST_Transform Returns a new GEOMETRY with its coordinates converted to the target SRID. 13
Vertica spatial function examples ST_Area Calculates the area of a spatial object. ST_IsValid Determines if a spatial object is well formed or valid. If the object is not valid, STV_IsValidReason returns a string that explains where the invalidity occurs. 14
Vertica spatial function examples ST_Intersects Determines if two spatial objects intersect or touch at a single point 15
Spatial Joins Spatial Joins allow you to identify spatial relationships between two sets of spatial data. Extensive applications are based on spatial join analysis. For example: Calculate the density of mobile calls in various regions to determine the location of a new cell phone tower Identify homes that fall within the impact zone of a hurricane Calculate the number of users who live within a certain ZIP code Calculate the number of customers in a retail store at any given time Spatial join functions in Vertia are using spatial indexes which provide good performance. 16
Longitude and latitude data World Geodetic System(WGS84) is an industry standard when mapping the Earth s oblate spheroidal surface, and it is also the reference coordinate system used by GPS enabled devices. WGS84 is common nowadays and is the standard for GPS data, so when people refer to lon lat, they mean WGS84 lon lat. 17
GEOGRAPHY data type Vertica supports GEOGRAPHY spatial data type. A GEOGRAPHY object is a spatial object defined as on the surface of a perfect sphere, or a spatial object in the WGS84 coordinate system. 18
GEOGRAPHY data type Compute ST_Distance on perfect sphere and WGS84. (Results in unit of meters) 19
Using GEOGRAPHY data type Pros Cons If your data covers the globe or a large area, this is probably the best. WGS84 is not projected, so distance measurement and spatial joins are more accurate than Universal Transverse Mercator (UTM). Most web mapping expect data to be fed to them in WGS84, so GEOGRAPHY will work fine in your spatial analysis work flow. Processing functions for GEOGRAPHY are limited. Computationally, functions for GEOGRAPHY are more expensive thus slower. For regional data, we usually use GEOMETRY data type. 20
Summary Vertica provides 65+ functions that allow you to manipulate complex spatial objects. Functions follow the OGC standards -- Easy for migration. Fast computations based on spatial index. Geosaptial analysis package automatically deploy on Vertica cluster. Support spatial analysis on longitude and latitude data in WGS84 coordinate system.
Hurricane risk analysis example In his example, I will show you perform spatial analysis on longitude latitude data with Vertica. Make a heat map with your spatial data using Vertica and third-party softwares. The ability to perform spatial queries, and display data spatially, is the essence and power of GIS. Vertica is perfect fit for your GIS development. 22
Hurricane risk analysis with Vertica Hurricanes leave behind extensive property damage. Using spatial analysis to estimate the impact of disasters in high-risk areas will provide important information to insurance companies on potential loss of property value. In this example, we do a point-in-polygon analysis to identify properties that fall within the impact zone of a hurricane. Determine areas most affected by a hurricane and visualize them on a heat map. The example shows how you can gain insight into real-world challenges using your spatial data with Vertica. 23
Hurricane heat map The National Hurricane Center(NHC) provides various GIS datasets. In this example, we use the hurricane BONNIE dataset. The United States Census Bureau provides various census data. We downloaded the landmark dataset from the TIGER database. We do a spatial join analysis of a set of points and a set of polygons. Polygon dataset Hurricane BONNIE from http://www.nhc.noaa.gov/gis/ Point dataset Landmarks from ftp://ftp2.census.gov/geo/tiger/tiger2015/pointlm/ Tools Vertica + third-parth softwares 24
25
Convert spatial reference system Both data set are geographic longitude latitude data set. The BONNIE data is in SR-ORG7219 and the landmark data is in NAD83. The first step is converting them to WGS84. Convert hurricane BONNIE polygon data to WGS84 ogr2ogr f ESRI Shapefile t_srs EPSG:4326 al022016-001_5day_pgn_4326.shp al022016-001_5day_pgn.shp Convert landmarks point data to WGS84 ogr2ogr f ESRI Shapefile t_srs EPSG:4326 tl_2015_09_pointlm_4326.shp tl_2015_09_pointlm.shp s_srs EPSG:4269 26
Load shapefile into a Vertica table -- Load the hurricane BONNIE shapefile SELECT stv_shpcreatetable(using parameters file=bonnie.shp) OVER(); CREATE TABLE tmp( gid IDENTITY(64) PRIMARY KEY, ADVDATE VARCHAR(28), geom GEOMETRY(7581) ); COPY tmp SOURCE STV_ShpSource(file=bonnie.shp) PARSER STV_ShpParser(); -- Convert GEOMETRY to GEOGRAPHY CREATE TABLE bonnie AS SELECT gid, ADVDATE, stv_geography(geom) geom FROM tmp; 27
Perform a spatial join analysis CREATE TABLE countlm AS SELECT ADVDATE, STATEFP, count(*) FROM bonnie a, landmarks b WHERE st_intersects(a.geom, b.geom) GROUP BY ADVDATE, STATEFP; Using this count in our visualization helps us see properties in which state are impacted the most by hurricane BONNIE. 28
More than 236 thousands landmarks on US east coast line are computed in each frame. Spatial join computation of all 236+ thousands instances takes ~1 second run time. You can export the intersect results to a shapefile and then load to a visualization tool. Or you can connect GIS tools with Vertica though JDBC/ODBC to fetch those spatial analysis results.
Q&A 30