ACCESS to Terra Fusion Products

Transcription

1 ACCESS to Terra Fusion Products Guang Yu Zhao and Larry Di Girolamo Department of Atmospheric Sciences University of Illinois at Urbana-Champaign

2 The Terra Data Fusion Project Department of Atmospheric Sciences, University of Illinois Larry Di Girolamo Guangyu Zhao Yizhe Zhan Landon Clipp Shashank Bansal Yat Long Lo Dongwei Fu Brandon Chen Department of Geography and GIS, University of Illinois Shaowen Wang Yan Liu Yizhao Gao The HDF Group MuQun (Kent) Yang H Joe Lee National Center for Supercomputing Applications, University of Illinois John Towns Kandace Turner Michelle Butler Sean Stevens David Ralia Jonathan Kim Donna Cox Stuart Levy Robert Patterson Andrew Christiensen Department of Atmospheric Sciences, Texas A&M Ping Yang Hioki Souichiro Yi Wang NASA Langely/SSAI Lusheng Liang NASA Goddard Space Flight Center Ralph Kahn Jim Limbacher Acknowledgements NASA Advancing Collaborative Connections for Earth System Science (ACCESS) program NSF Blue Waters Project NSF ROGER MRI NASA Langley Research Center Atmospheric Sciences Data Center NASA Atmosphere Archive & Distribution System Distributed Active Archive Center NASA Land Processes Distributed Active Archive Center

3

4 Terra, in 2015 alone More than 360 million files Totaling more than 3.4 PB data Delivered to more than 100,000 users around the world. More than 1,800 peer-reviewed publications (over 15,000 to date) Results from Terra cited more than 49,000 times (over 250K to date) The high publication rate includes an increasing number of papers capitalizing on fusion of data among Terra sensors NASA Senior Review 2017: Terra

5 Geology Earth s Radiation Budget and Climate Glaciology Agriculture and Land Use Ocean Biology Hydrology Atmospheric Dynamics, Transport, and Air Pollution Volcanology Meteorology

6 Current Problems: The Terra Data Fusion Project The challenges for individual investigators for mission-scale Terra analyses are great just from a computing stand point. Instrument data reside at different locations, in different file formats, with different granularity, and using different projections. > 1 PB and counting (~2 TB/day) There exists inadequate cyberinfrastructure to tackle wholemission data fusion and mining problems. Distribution of fusion datasets. A range of expertise, not normally had at an investigator s institution, is required to get the job done These problems greatly limits/discourages scientists to take full advantage of the Terra data.

7 ACCESS to Terra Data Fusion Products tackles the following questions: 1) How do we efficiently generate and deliver Terra data fusion products? 2) How do we facilitate the use of Terra data fusion products by the community in generating new products and knowledge through national computing facilities, and disseminate these new products and knowledge through national data sharing services? We aim to provide a significant community-element addition to NASA s system of systems infrastructure for scientists requiring mission-scale access, processing, analytics, and distribution.

8 Step 1: Transferring All L1B Data from DAACs to NPCF Instrument Size (TB) DAAC Protocol Rate (MB/s) Days NCSA s Help Completion MISR 240 ASDC GridFTP MODIS 331 LAADS FTP stable unstable 14 Yes 100% 210 No 100% ASTER 400 LPDAAC HTTP CERES 1.5 ASDC GridFTP 0-20 unstable >300 Yes 100% stable 0.2 Yes 100% MOPITT 0.8 ASDC GridFTP stable 0.1 Yes 100% Worked with NASA networking (CNOC) and ASDC teams to diagnose and remedy performance obstacles for data transfer out of the ASDC. Result: a 2.5x increase in data transfer rates out of the Langley ASDC.

9 Lessons Learned Data Transfer Mission scale data transfers require active support from the DAACs, which is not a scalable (A special shout out to the ASDC amazing!) GridFTP provides much more efficient and reliable protocol than ftp and http for bulk data transfer of the kinds we are dealing with. Issues that need to be addressed by the DAACs Complete lists of the original data files along with their sizes staged on the DAACs servers are unavailable, making it difficult to verify the integrity of data transfers. Corrupted data files exist on the DAACs servers. Mission-wide L1B ASTER data could not be filled by the LPDAAC, so we had to settle with L1T data, which doesn t contain radiance data for ASTER s oblique camera (3B). Presentation Title 9

10 Step 2: Develop the Basic Fusion Product Goal: Merge the radiance fields from the L1B granules for all the Terra Instruments into one common granule, design it to be easy to use, and optimize it for large scale processing. Granularity: One Terra Orbit File Format: HDF5 (conformed to NetCDF4 CF conventions) File Content : 1. Unscaled Radiance (floating-point format) 2. Radiance Data Quality 3. Geolocation for each FOV at its native resolution 4. Sun-view Geometry 5. Observational Time 6. Other Attributes/Metadata Keep instrument contents as is

11 Step 2: Develop the Basic Fusion Product File Naming Convention: : TERRA_BF_L1B_OXXXX_YYYYMMDDHHMMSS_F000_V000.h5 (Opening movie from TERRA_BF_L1B_O3571_ _F000_V001.h5) File Size: One granule: GB without compression; GB with compression, depending on the number of available ASTER granules for one orbit Processing Time (node hours): One granule: ~30 min without compression ~45 min with compression and chunking

12 Development of the Resampling and Reprojection Tool (Next Level of Fusion: Step 3) Functional Requirements: A community, open source tool to resample/reproject the radiance fields for one Terra instrument onto the grids used by another Terra instrument, a regularly spaced grid, or any other user specified grid. Accomplishments: nearest neighbor resampling for source and destination grids having similar pixel resolutions (e.g., MISR -> MODIS) statistical resampling for source and destination grids having distinct pixel resolutions (e.g., ASTER -> MISR) Regularly spaced grid defined by any map projection User-friendly user interface HDF5 and GeoTIFF output Performance: Developed in C and C++ and optimized for HPC It takes 40 seconds to resample MISR 1.1km radiance data onto MODIS 1.0km grid for a whole orbit and 4 minutes to resample MISR 275m radiance data onto MODIS 250m grid using 32 threads (1 node) on Blue Waters.

13 Development of the Resampling and Reprojection Tool MODIS Blue Band MISR blue band radiance resampled on the MODIS grid (Next Level of Fusion: Step 3) Presentation Title TERRA_BF_L1B_O69626_ _F000_V000.h5 13

14 Development of the Resampling and Reprojection Tool ASTER Band 3N ASTER resampled on the MODIS grid MODIS Band 8 (Next Level of Fusion: Step 3) Presentation Title 14

15 Metadata Generation Metadata files: 1 collection-level XML file granule-level XML files (We do provide metadata for no-data BF files caused by instrument anomalies, PGE failure and et al.) Granule-level Metadata include: CDL metadata generated from BF granules Metadata from original input granules used to generate BF Metadata can only be obtained through the NASA CMR queries We did 20,000 granule queries per hour via CMR APIs and then: Ø It was bringing CMR to it's knees from the CMR Operations Lead Ø Earthdata Search experienced transient search timeouts for a whole weekend. Ø Our IPs were blocked for a few days. Presentation Title 15

16 Example Science Use Cases to Aid Development Fu, D., L. Di Girolamo, L. Liang, and G. Zhao, 2017: The observed behavior of the bias in MODIS-retrieved cloud drop effective radius through MISR-MODIS data fusion. American Geophysical Union 2017 Fall Meeting, December 10-15, New Orleans, LA. Wang, Y., S. Hioki, P. Yang, L. Di Girolamo, and D. Fu, 2017: Seasonal bias of retrieved ice cloud optical properties based on MISR and MODIS measurements. American Geophysical Union 2017 Fall Meeting, December 10-15, New Orleans, LA. Di Girolamo, L., et al., 2017: The Terra data fusion project: an update. American Geophysical Union 2017 Fall Meeting, December 10-15, New Orleans, LA. Zhao, G., L. Di Girolamo, D.J. Diner, C.J. Bruegge, K. Mueller, and D.L. Wu, 2016: Regional changes in Earth s color and texture as observed from space over a 15-year period. IEEE Trans. Geosci. Remote Sens., 54(7), , doi: /tgrs First true color image of the Earth s climate at ~ 10:30 AM. Aka: Climate Marble TM Presentation Title 16

17 Climate Marble TM MISR RGB 15-year average, daily time step, 20-day averageing window University of Illinois

18 Presentation Title 18