NOAA s Big Data Partnership: Leveraging the Value of Data with Industry

Transcription

1 NOAA s Big Data Partnership: Leveraging the Value of Data with Industry Edward J. Kearns, Amy Gaskins, Brian Eiler, Maia Hansen, Alan Steremberg, Chelsea Mastrapa National Oceanic and Atmospheric Administration (NOAA) BiDS March 2016 US Department of Commerce NOAA Satellite and Information Service NOAA s National Centers for Environmental Information

2 Acknowledgements Many thanks to: NOAA: Steve Ansari, Steve Del Greco, Jeff de la Beaujardiere, Brian Nelson, Tony LaVoi, Jay Morris, Carlos Rivero, Ken Casey, Ken Knapp NC State University / CICS-NC: Otis Brown, Jonathon Brannock, Lou Vazquez, Scott Stevens NOAA s Big Data Collaborators and their partners involved in the NEXRAD project Amazon: Arial Gold, Jeff Layton Climate Corporation: Adam Pasch, Valliappa Lakshmanan, Unidata: Jeff Weber Microsoft: Sam Khoury, Sid Krishna Google: Eli Bixby, Tino Tereshko, Amy Unruh, Tanya Shastri, Ossama Alami Open Commons Consortium: Maria Patterson, Walt Wells 15 March 2016 BiDS'16 Big Data from Space Tenerife 2

3 Outline Brief Description of the origins of the Big Data Partnership at the US National Oceanic and Atmospheric Administration (NOAA) Progress and lessons learned from the recent Big Data Partnership activities Upcoming opportunities and challenges 15 March 2016 BiDS'16 Big Data from Space Tenerife 3

4 From BiDS 14 (Roma), Kearns et al. NOAA was reaching out to partner with industry: will launch a public private initiative to unlock the economic potential of all NOAA data, with the goal of maximizing their economic and social potential, incubating new industries around climate and oceans information US Dept of Commerce, Open for Business Agenda 2013 NOAA formally sought information from industry through activities and negotiations in February and October 2014 A description of a Data Platform and the role of scientific data stewardship in building/maintaining that platform 15 March 2016 BiDS'16 Big Data from Space Tenerife 4

5 Accelerating User Demand for NOAA data NOAA s National Centers for Environmental Information (NCEI) alone is now serving >8 PB of data annually Servicing over 20,000 personal contacts across many sectors 2.6 billion web hits in FY14 with 19 million users Significant load on NOAA infrastructure 15 March 2016 BiDS'16 Big Data from Space Tenerife 5

6 NOAA Big Data Partnership 1. NOAA has a lot of data, which is expensive to store and disseminate. 2. Due to a variety of accessibility issues, much of NOAA s environmental data are also under-utilized especially beyond the expert community. 3. There is untapped economic value in all that data. 4. That value can be leveraged to improve accessibility and pay for staging of that data on the public cloud, where people and organizations of all kinds can innovate as part of a market ecosystem. 15 March 2016 BiDS'16 Big Data from Space Tenerife 6

7 April 2015: Partnership Announcement Unusual, no-net-cost proposition Could the value inherent in NOAA s datasets support the cost of their distribution? Enthusiastic, cross-industry response Interest from 200 companies 70+ responses to NOAA s request for information NOAA is using Cooperative Research and Development Agreements (CRADAs), not typical acquisitions Selection/announcement made on 21 Apr March 2016 BiDS'16 Big Data from Space Tenerife 7

8 BDP CRADA Collaborators IaaS companies to serve as project anchors These CRADA Collaborators are nuclei for data alliances and markets Members of industry, research, and academia may join these alliances 15 March 2016 BiDS'16 Big Data from Space Tenerife 8

9 Big Data Partnership Rules Level Playing Field, for all interested companies via Collaborators All NOAA Data, available equally, with no privileged access NOAA-sourced data are free and open - only cost recovery allowed 5 individual CRADAs with NOAA, 3 year terms + 2 one-year options CRADAs can be ended early with appropriate notice No funds may move from NOAA to Collaborators 9

10 Conceptual Relationship for BDP 10

11 Methodology Collaborators & their partners identify datasets of interest Demonstrate business use case(s), with no restrictions on open distribution of data Develop a strategy for data sharing/delivery from NOAA to BDP Collaborator(s) Engage NOAA subject matter experts, BDP Collaborators, and their Associates/Partners for technical interchanges Collaborators and their Partners create applications NOAA continues all of its existing data services No interruption of services to customers BDP activities are an augmentation of existing services 11

12 Requested Initial Data Included Multi-Radar Multi-Sensor GOES / GOES-R Numerical Models NEXRAD 15 March 2016 BiDS'16 Big Data from Space Tenerife 12

13 Why was NEXRAD Weather Radar selected first? Archived NEXRAD data are optimized for preservation, not access All NEXRAD data are publicly available, but difficult to use unwieldy size (270 TB for compressed Level II alone) specialized format (radial volume scans) resides on NOAA s NCEI tape archive, relatively slow to access Highly popular dataset for use in industry Many industry users of realtime and archived NEXRAD data Multiple derivative uses possible - hail, rain, snow, tornados, etc. The utilization of entire NEXRAD archive never before realized NOAA had recently reprocessed NEXRAD Level II half of the dataset already resided on disk Cooperative Institute for Climate and Satellites in NC (CICS-NC) easy to jump-start initial delivery 13

14 NEXRAD #2 in US National Observation Value National Plan for Civil Earth Observations (2014) 14

15 NEXRAD Data Transfer Archived Level II data from NESDIS National Centers for Environmental Information (NCEI) Over 270 TB for compressed Level II volume scan file (>1 PB uncompressed) Move from the NCEI tape archive to disks at CICS-NC CICS-NC acting as middleman allowed minimal impact to NOAA s data operations Updates from NCEI archive operations as new data are pushed to the archive Realtime Level II data from NOAA s National Weather Service Established through Unidata, part of the Open Commons Consortium (OCC) Data moved to 4 of the 5 Big Data Partnership Collaborators Amazon Web Services (entire archive plus realtime) Microsoft (entire archive) Google (entire archive) Open Commons Consortium (2015 plus realtime) 15

16 AWS Access to NEXRAD Level II Data Oct 27, 2015 rollout AWS now serving all NEXRAD Level II from min ago Single point of access for archived and realtime data on S3 Free data download available to users Many in industry & research already utilize the AWS Cloud User-contributed tools and services are available that utilize the AWS platform, e.g. THREDDS from Unidata Tutorials available from Unidata, Climate Corporation, CartoDB Free AWS credits and grants may be available for educators, researchers, and students Google, Microsoft, and OCC have not announced their access. 16

17 Steering Users to New Services New services are being integrated from within NOAA s NEXRAD sites. Archive data ordering system 60% decrease in anticipated data orders from NOAA/NCEI (Feb 2016, S. Ansari) Integration into the Weather and Climate Toolkit 15 March 2016 BiDS'16 Big Data from Space Tenerife 17

18 Steering Users to New Services Other BDP Collaborators sites will be added as they roll out their data services 15 March 2016 BiDS'16 Big Data from Space Tenerife 18

19 Improvements to Data Stores NCEI & CICS-NC worked with AWS, Unidata & Climate Corporation to verify inventories/checksums Identified and fixed ~20K corrupted archive files (0.01% of 180M total) An improved NOAA archive is a direct and positive result of the AWS Big Data Partnership effort for NEXRAD Level II Climate Corp discovered the corruption, NOAA fixed Recovered data were reintroduced to the NOAA archive and sent to all the other BDP Collaborators. 15 March 2016 BiDS'16 Big Data from Space Tenerife 19

20 The Elements of a Successful Market Ecosystem 3 rd Party Value-Added Service Providers Environmental Data User Community 15 March 2016 BiDS'16 Big Data from Space Tenerife 20

21 Role of NOAA in the BDP Market Ecosystem NOAA s data are widely available, with free and open access NOAA s role will be to provide objective scientific expertise, ensure long-term preservation and sound data management (scientific data stewardship) Data + Expertise + Need = Opportunity 15 March 2016 BiDS'16 Big Data from Space Tenerife 21

22 NCEI s General User Profile tie to BDP Market? Fraction (%) Typical User Data or Info Need Preferred Format Access Volume Access Frequency 70 General business, media, public Qualitative Point-andclick, graphics, assessments Low High 15 Researchers, business consultants Quantitative Digital downloads High Low 15 Value-added Providers (database scrapers) Quantitative Digital downloads machine to machine Low High 15 March 2016 BiDS'16 Big Data from Space Tenerife 22

23 Impact on the Information Lifecycle NOAA makes foundational investments in environmental information production and preservation BDP promotes increased data access and encourages application development across private and public sectors. Earth Environment Data Observations Environmental Records Reports Decisions Data Access Decision- Generating Record to maker Systems Creation and - Observing Preservation Applications Needs - Models 15 March 2016 BiDS'16 Big Data from Space Tenerife 23

24 So What? One valuable, large, unwieldy dataset has been liberated for wider utilization, and improvement, by industry and the public -- at no net US taxpayer cost. Seamless access across time - users find both historical and realtime NEXRAD Level II data in the same place, in the same way. Historical data provides context for new realtime observations important for decision-making New business opportunities are being created custom severe weather products, bird migration studies, etc. are easier to develop linked data approaches with socio-economic, health, agricultural data? New applications can be developed FASTER - data are co-located with processing Quicken the pace of app development and time-to-market Faster pace of innovation and scientific discovery NOAA s NEXRAD recent reprocessing (of 11 years data) took years Same volume of processing today could now take weeks GOES data, Weather Forecast Models, Fisheries bycatch information and other NOAA data are up next 24

25 NOAA CLASS Access Volume by Dataset GOES 25

26 Can BDP enable wider GOES use? Can BDP make NOAA/NCEI s 2nd most downloaded satellite dataset and the USA s 4th most impactful observing system* more useful? Today, numerous factors hinder access to GOES data, making a simple request a very complex task 1. Determine satellites operating at time of request 2. Order Data from CLASS website, download them to your own disk 3. Figure out how to handle duplicate files 4. Learn how to read GVAR and/or create AREA files 5. Investigate how to calibrate data, then apply calibration 6. Remap and subsample data for desired Courtesy region of Dr. Ken Knapp 7. Process data to level/product desired 26

27 A path to improved GOES data access? NOAA provides improved dataset? Extract all GOES data from CLASS While sending to BDP Collaborators: Remap and format (to GOES-R HDF5 format) Remove duplicates and apply calibration BDP Collaborators provide seamless access to GOES data? Provide single API or website for entire time series Collaborators, Partners, Expert Community develop apps? Create visualization and analysis tools Develop value-added products Provide interpretation of results User Community utilizes the data and the apps? Easier access to GOES basic data products, if desired Enhanced access to GOES-derived information products BDP is just getting started with GOES and the Collaborators. 27

28 Summary NOAA has engaged 5 Collaborators in research activities to host NOAA s data Big Data success requires not just access to the data, but the expertise (algorithms, workflows, interpretive skill) and a solid use case as well. NOAA has moved the NEXRAD dataset first, based on market need and data opportunity. AWS experience starting in Oct 2015 appears successful so far New applications created, increased use of AWS infrastructure, reduced load on NOAA systems NOAA is investigating distributing GOES data next to the Collaborators, to improve utilization of archived data and future realtime flows. NOAA and the Collaborators have not really gotten into Big Data territory yet, mostly simply enabling wider use of unwieldy and high volume datasets. Moving slowly and deliberately, and easier to estimate markets for traditional products. NOAA model outputs, fisheries data, and Collaborators/Partners business and socio-economic datasets may provide real Big Data opportunities More results to come later 15 March 2016 BiDS'16 Big Data from Space Tenerife 28

29 Thank You Questions? 29

30 15 March 2016 BiDS'16 Big Data from Space Tenerife 30

31 31

32 32