Harvard Center for Geographic Analysis Geospatial on the MOC

Transcription

1 2017 Massachusetts Open Cloud Workshop Boston University Harvard Center for Geographic Analysis Geospatial on the MOC Ben Lewis Harvard Center for Geographic Analysis Aaron Williams MapD

2 Small Team Supporting Many Projects

3 Platforms to Scale Ourselves WorldMap (Various funders) Dataverse <-> WorldMap (BARI) HHypermap (NEH) BOP (Sloan and Cloud Dataverse) Spark with GeoMesa (Sloan and MOC) MapD (MapD)

4 Interoperability: enabling an ecosystem greater than the sum of parts Containers (Kontena, Docker) APIs (open specifications) Protocols (open specifications)

5 WorldMap Allows One to Discoverdata and visualize it in new ways Organizeone s own (large) mapping datasets online Mashup one s data with that of others Collaborateby letting several people edit the same map Publishdata to the world or to just a few collaborators

6 Upload, Create, Share Scholars outside the academy Memory institutions Universities, schools Businesses Government agencies

7 Mapping Contemporary China Faculty: Peter Bol, Chinese History

8 Boston Research Map Faculty: Chris Winship (Sociology) and Kennedy School, Radcliffe, Boston Area Research Initiative. To bridge the town/gown divide and facilitate data sharing between the city of Boston which creates data and Harvard scholars who make sense of it. Three major phases of software and data development over 4 years.

9 HHypermap Solves search problem for WorldMap Plus millions of geo-services exist a click away, but no good way to discover them. Geo search still in early days like web before search engines.

10

11 Big geo viz via spatial faceting Spatial density summary grid faceting, also useful for point-plotting search results Lucene & Solr APIs Scalable & fast usually Usually rendered with a gradient radius ->

12 Built on: Celery RabbitMQ Django Lucene Solr Elasticsearch MapProxy Memcached OWSLib PostgreSQL PostGIS pycsw HHypermap architecture

13 Billion Object Platform (BOP) Platform to lower barriers for scholars accessing large geospatial datasets.

14 Latest billion + long term archive Latest ~billion geo-tweets Sharded Solr/Lucene Streaming update Interactive exploration Appox 3 years of data Archive for persistent storage all tweets Currently on Kafka May move to Swift Eventual access from Cloud Dataverse Back to 2012

15 BOP Search Stack Apache Lucene: text search library Apache Solr: a search platform built on Lucene Apache Kafka: a message broker written in Scala for streaming data Apache ZooKeeper: coordinates Lucene shards Swagger: a framework for building APIs scikit-learn library: Machine Learning in Python OpenLayers: mapping client AngularJS: a javascript framework for client development

16 BOP High-level Architecture BOP Harvesting Enrichment Kafka (archive) Solr Web Service Browser UI Data flows via Apache Kafka HTTP Docker, Kontena, OpenStack Hosting: Mass OpenCloud

17

18 Streaming Data (geo-tweets) Draft Architecture for Geospatial Analytics with GPUs and Spark on the MOC Big geodata exploration and analytics -Spark - Harvesting and enrichment -GPU MapD - Cloud Dataverse - Swift Object Storage (Long-term archive) - Cloud Dataverse - -Spark GeoMesa -WorldMap - Mashup analytic maps (WMS) Display map-based results, store data results.

19 Next-gen BOP: persistent storage on Swift Persistent streaming archive in OpenStack Swift accessible by Cloud Dataverse Analytic tools available such as MapD (GPUs), GeoMesa (Spark), and others.

20 Next-gen BOP uses GPUs + CPUs: MapD s GPU database MapD with origins at CGA in 2012, now a successful startup. MapD has open-sourced its GPU database and is working with CGA to explore innovative research applications. Funded by MapD though the NSF Spatiotemporal Innovation Center Focus is initially on evaluation of hydrologic models

21 MapD Architecture

22 MapD Memory Management Hot Data Speedup = 1500x to 5000x Over Cold Data Warm Data Speedup = 35x to 120x Over Cold Data COMPUTE LAYER GPU RAM (L1) 24GB to 256GB GB/sec CPU RAM (L2) 32GB to 3TB GB/sec Cold Data STORAGE LAYER SSD or NVRAM STORAGE (L3) 250GB to 20TB 1-2 GB/sec Data Lake/Data Warehouse/System Of Record

23 Try MapD Live demos: Download Community Edition: Join our forums:

24 Thank you Benjamin Lewis Aaron Williams

25 Streaming Data (geo-tweets) Overview of CGA (and related) platforms on the MOC - Python/Solr - Harvesting and enrichment -BOP Solr WorldMap/HHypermap General purpose mapper -Kafka - Swift Object Storage (Long-term archive) - Cloud Dataverse - Display map-based results, store data results.

26 SDI + Modern Search Developed 2D faceting for Solr Built temporal faceting, temporal histogram, time miner Instant previewing Results ranking with usage statistics Open API

27 Requirements Develop back end and client to support interactive visualization of a billion point objects Support sub-second queries including heatmaps and temporal histograms Expose a general purpose RESTful API so other clients could access the data System should run on low cost commodity hardware or VMs

28 Initial focus on Geo-tweets (could be any streaming dataset) 1-2% of tweets have GPS coordinates from the user s device, currently about 1 million per day available via the Twitter API The CGA has been harvesting geo-tweets since 2012 and has an informal archive of about 8 billion objects Northeastern Professor Ryan Qi Wang also harvested during this period and we plan to eventually merge the two datasets to create a more complete version.

29