SpaceNet Round 2: Automated Mapping Using Satellite Imagery Accelerated by Deep Learning

Transcription

1 SpaceNet Round 2: Automated Mapping Using Satellite Imagery Accelerated by Deep Learning David Lindenbaum, SpaceNet Lead, CosmiQ Works, an IQT Lab Todd M. Bacastow, SpaceNet Lead, DigitalGlobe Radiant 2017 In- Q- Tel, Inc In- Q- Tel, Inc. 1

2 2017 In- Q- Tel, Inc. 2

3 2017 In- Q- Tel, Inc. 3

4 SpaceNet is an open repository of over 5,700+ km2 of satellite imagery, 520,000+ vectors, and a series of challenges to accelerate geospatial machine learning. Automated Mapping Challenge: Building Extraction Rounds 1 & 2 Nov Jun Automated Mapping Challenge: Road Network Extraction Launching Fall 2017 Rio, Las Vegas, Paris, Khartoum, Shanghai Las Vegas, Paris, Khartoum, Shanghai 475,000 building footprints 6,500 km of roads $50,500 in total prizes Road extraction and routing optimization 2017 In- Q- Tel, Inc. High Revisit Challenge: Off- Nadir Object Detection Launching Winter 2018 Off- nadir object identification to simulate increased persistence of satellite imagery 4

5 SpaceNet Completed Datasets and Challenges Rio de Janeiro Building Footprints Rio de Janeiro Points of Interest (POIs) Las Vegas, Paris, Khartoum, and Shanghai Buildings Dataset: Imagery: 50 cm WV- 2 mosaic and 8- band MSI over 1,900 km2 Building Footprints: 220,594 Dataset: Imagery: 50 cm WV- 2 mosaic POIs: 120,155 individual POIs from 460 feature classes Released in collaboration with NGA Dataset: Imagery: 30 cm WV- 3 image strips and 8- band MSI covering 3,800 km² Building Footprints: 181,619 Released in August 2016 SpaceNet Challenge Round 1: Focus on automated mapping 42 competitors, 242 submissions F- Score of.26 (scored from 0 to 1) $35,000 in prizes Released in January In- Q- Tel, Inc. Released in February 2017 SpaceNet Challenge Round 2: Continued with automated mapping 14 competitors, 136 submissions Top avg. AOI F1- score: 0.69 (0 to 1) $15,500 in prizes 5

6 Providing open access to geospatial labeled training data Source Data Large Scale 16- bit GeoTIFF s Competition Data 200m x 200m 16- bit GeoTIFF chips and associated GeoJSON labels 2017 In- Q- Tel, Inc. 6

7 SpaceNet Building Footprint Evaluation Metric * Images from pyimagesearch.com We assign a correct detection (true positive) as a prediction with an IOU >= 0.5 Predictions are evaluated with the F1 score 2017 In- Q- Tel, Inc. 7

8 Round 2 Competition Details What was different from Round 1? 1. Higher resolution data 30 cm Imagery 2. Larger training set 3500 more training images 3. Four geographically distinct locations Las Vegas, Paris, Shanghai, Khartoum 4. Open Source datasets 5. Dockerized solutions with training and inference (test) open sourced Competition Timeline 2/24/2017 3/21/2017 3/24/2017 5/23/2017 6/13/2017 6/?/2017 Data Release Competition Start Early Incentive Awarded to Wleite Competition Ends Validation Completed/ Winners announced Top 3 winners code released at In- Q- Tel, Inc. 8

9 Round 2 Competition Results Competitors 10 Competitors with F1- Score above 0.1 User Country of Origin Total Vegas Paris Shanghai Khartoum 1 XD_XD Japan Submissions 136 solutions submissions 2 wleite Brazil platero China nofto Slovakia Winning Result F1 Score of from Japan International Top 5 submissions were international 5 yghlc China A. Most Algorithms performed significantly better in Las Vegas than other AOIs B. Performance in Las Vegas is potentially reaching utility for applications C. Winner integrated OpenStreetMaps data into solution 2017 In- Q- Tel, Inc. 9

10 Winning Solution: Examples AOI 2 Vegas: Image 1014 AOI 2 Vegas: Image 104 AOI 3 Paris: Image 1729 AOI 5 Khartoum: Image In- Q- Tel, Inc. 10

11 Histogram of IoU Scores for each Area of Interest 2017 In- Q- Tel, Inc. 11

12 SpaceNet Benchmarks and Possible Future Tests Labeling Techniques Time Avg. F1 Score Area InferenceSpeed Ground Truth: Dedicated team of labelers (302,701 building footprints) ~24 days n/a km 2 ~29 km 2 /day Automated: Winning SpaceNet Challenge Round 2 Solution 1 Crowdsourced 2 : OSM like contributors Automated + Crowdsourced Improvements: Winning SpaceNet algorithm + OSM like contributors ~6 days training ~4 hours to run km 2 ~1800 km 2 /day Future tests Future tests On The Humanitarian OpenStreetMap Team s Tasking Manager there are currently 173 open mapping tasks for buildings and 130 open tasks for road networks 3 Future tests will benchmark results against crowdsourcing and algorithms + crowdsourced improvements. 1 Metrics based on 1 Titan X Maxwell GPU. The SpaceNet Challenge building footprint algorithms are optimized for IOU scoring and the F1 metric. An incorrect building footprint proposal causes a false positive and a false negative, while missed building footprints (not proposed) is only a false negative. 2 Crowdsourcing communities could include OSM 3 On 11/02/ In- Q- Tel, Inc. 12

13 SpaceNet : What s Next New Challenges: 1. Fall 2017: Expansion of foundational mapping: road network detection & routing 2. Winter 2018: Explore the effect of off- nadir imagery on mapping problems New Labeled Datasets & Algorithms Benchmarking: o Benchmarking building extraction against crowdsourcing and winning algorithms + crowdsourced improvements o Foundational mapping: Road network extraction o High revisit collection: Explore the effect of off- nadir imagery o New modalities/ sensors o Long- term hosting of related competition data 2017 In- Q- Tel, Inc. 13

14 Road Detection & Routing Competition 2017 In- Q- Tel, Inc. 14

15 Automated Road Extraction for Routing Determining optimal routing paths is at the heart of many humanitarian, civil, and military challenges Current methods for building road networks rely heavily upon mobile phone location services and manual labeling We aim to automatically determine road networks directly from satellite imagery The SpaceNet Roads Challenge will use the Average Path Length Similarity Metric (APLS) to evaluate road network proposals 2017 In- Q- Tel, Inc. 15

16 SpaceNetV3 Public Data Set Built upon the imagery of SpaceNetV2 Release: Imagery: 30cm WV- 3 single strip images + 8- band MSI Total Road centerlines: Over 8,000 km covering a 3,880 km2 AOI across for 4 additional cities: Las Vegas, Paris, Shanghai, and Khartoum Las Vegas Paris Shanghai Khartoum 270 km2 1,560 km2 1,170 km2 800 km2 69GB Raster Data 402GB Raster Data 302GB Raster Data 373GB Raster Data 109,807 Footprints 16,663 Footprints 69,433 Footprints 25,463 Footprints 3686 km Roads 425 km Roads 3000 km Roads 1032 km Roads In- Q- Tel, Inc. 16

17 Thank You In- Q- Tel, Inc. nc. 17