National Climatic Data Center Data Management Issues Tom Karl Director, NOAA s National Climatic Data Center Opening Meeting NOAA Science Advisory Board s Data Archiving and Access Requirements Working Group (DAARWG)
Briefing Outline NCDC Overview Mission Data Users Current Access Current Archive NCDC Archive Issues Projected Large Data Volumes Requirements on what to Archive NCDC Access Issues Data Integration Emerging Portal Technology 2
National Climatic Data Center (NCDC) Asheville, North Carolina! NCDC is the steward of the Nation s in-situ and satellite data and information. Collocated with the U.S. Air Force and Navy Climatology offices. The 3 agencies fulfill much of the Nation s climatological requirements 03/2006 3
Mission Statement To provide access and stewardship to the Nation s resource of global climate and weather related data and information, and assess and monitor climate variation and change. 03/2006 4 4
NCDC DATA ACCESS BY USER TYPE VERSUS PLATFORM Earth (In-Situ) Platform Education 4% Government 6% Primary User (~ 3 million / year) Commercial Commercial 90% Satellite Platform Government 70% Commercial 25% Education 5% Government Radar Platform Government 32% Commercial 20% Education 5 Education 48%
Web Access to Data at NCDC Data Delivered to Users each Month (National Climatic Data Center) Volume (Terabytes) 28 24 20 16 12 8 4 0 Jan- 03 Apr- 03 Jul- 03 Oct- 03 Jan- 04 Apr- 04 Jul- 04 Oct- 04 Jan- 05 Apr- 05 Jul- 05 Oct- 05 Jan- 06 Apr- 06 Jul- 06 Oct- 06 NEXRAD CLASS NOMADS CDO TOTAL NEXRAD=Radar CLASS =Currently Satellite NOMADS=Models CDO=Climate In-Situ 6
Climate Data Users/Applications Examples 7
Data Archive Managed by NCDC 1990-Current NCDC Data Archive 3,000 2,500 Terabytes 2,000 1,500 1,000 500 0 1990 1992 1994 1996 1998 2000 2002 2004 2006 Fiscal Year Ending Total GOES POES & Climate In-Situ NEXRAD NOMADS Includes primary and back-up copies 8
NEXRAD Doppler Weather Radar Data Example of a large NOAA Data System Receipt from NWS, FAA, DOD Data statistics Archive: 1,600 terabytes adding 200 terabytes/year Access: ~7 terabytes delivered each month to users Future radar technologies will increase the digital archive Dual polarization, Phased array: by a factor >100 (after 2015) Visualization Tools developed based upon user needs Java application; multiple map overlays, image export, animation, GIS queries NOAA Application - QC surface in-situ weather data Station Network Access Tools Research Gage-Radar Mosaic 9
NOAA Operational Model Archive and Distribution System Example of a large NOAA Data System Provides access to high volume weather models and associated data Promotes model evaluation and product development Fosters research within the geo-science communities (ocean, weather, and climate) Browse, subset, obtain weather model data over the Internet using desktop applications GrADS, Ferret, MatLAB, IDV, IDL, OPeNDAP-G Data statistics: Archive: 60 terabytes adding 24 terabytes/year Access: 6 terabytes delivered each month to users 10
National Climate Data Center Data Archive Issues 11
Major Challenges Great increase in Data Volume Projected PETABYTES 160 140 120 100 80 60 Current Estimate of NOAA NESDIS DATA ARCHIVE VOLUME PROJECTIONS (under CLASS Environment - 2 site concept) September 2006 Model Data NEXRAD NPOESS NPP GOES NASA EOS (MODIS) METOP Ocean Related Data DMSP 12 40 20 0 2004 2005 2006 2007 2008 2009 2010 2011 2012 YEAR 2013 2014 2015 2016 2017 2018 2019 2020 IN-SITU (Weather & Climate) CORS POES Sorted by year 2020 volumes Misc.
Which Climate Data to Archive Current Archive Considerations Legal Requirements Data required by law to be archived Practical Requirements Data which would require long processing time to reproduce Policy Requirements Data used in Scientific Assessments and for Regulatory purposes As required by the Data Quality Act 13
Which Climate Data to Archive? Legal Requirements U.S. National Archives & Records Administration Policy preserving records that contain adequate and proper documentation of the organization, functions, policies, decisions, procedures, and essential transactions of the agency Climate Data Archive Legal Requirement Examples In-situ Original weather observations (e.g., surface, upper air) Products: weather forecasts (e.g., text, model outputs) Satellite Original observations that have been geo-located (e.g., level 1b) Radar Original data quantities recorded: Reflectivity, Mean radial velocity, and Spectrum width 14
Which Climate Data to Archive? Practical Requirements Certain data products require large processing times to reproduce Climate Data Archive Practical Requirement Examples In-situ: Model output products produced from in-situ observations Examples: Weather and climate models, Climate Model based Reanalyses Satellite: Products produced from geo-located observations Examples: Vegetation Index, blended sea winds, blended sea surface temperatures, etc. Radar: Level-III NEXRAD products Examples: Base reflectivity, Echo tops, One-hour precipitation, etc. 15
Which Climate Data to Archive? Policy Requirements Data to Reproduce Scientific Assessments & for Regulatory Purposes Examples of Scientific Assessment U.S. Climate Change Science Program Assessments State of the Climate Reports Drought Monitor Reports International International Panel on Climate Change (IPCC) Examples of Policy Made by Government Agencies using Weather/Climate Data US Army Corp Engineers use of revised hurricane statistics for rebuilding New Orleans levees Climate Normals used by Utilities commissions in setting power rates CCSP 1.1 Assessment Released March 31, 2006 16
Which Climate Data to Archive? Other issues: How many data set versions to keep? Example Data Set Evolution: NOAA s Microwave Sounder Unit History of data set adjustments based upon scientific discoveries Data set improvements occur when data are regularly used & more thoroughly scrutinized 17
National Climate Data Center Data Access Issues 18
Need for Data Integration: GEO IDE Important societal issues require data from many observing systems Discipline Specific View Whole System View Atmospheric Observations Land Surface Observation Ocean Observations Space Observations Data Systems 19 Current systems are program specific, focused, individually efficient. But incompatible, not integrated, isolated from one another and from wider environmental community Coordinated, efficient, integrated, interoperable
Challenges to Integration Incompatible syntax (formats) and semantics (terminology) among science disciplines within NOAA. Thousands exist. Several examples: Naming standards Surface Air Temperature Meteorology (WMO) named Temperature/dry bulb temperature Meteorology (air pollution) named Boundary layer temperature Oceanography named Air Temperature Location standards (latitude, longitude, elevation) Lat/Lon can be degrees/minutes/seconds or degrees to tenths and hundredths Latitude E/W, 0-180 positive and negative, or 0-360 running east or west Z used to designate elevation in both atmosphere and ocean but positive is up in the atmosphere and down in the ocean Time standards (Greenwich, local standard, local daylight) Formats (>50 formats used within NOAA; translators and standards needed) GRIB, NetCDF, HDF and others used for gridded data BUFR, NetCDF, and many others used for observations Observing standards Direction Wind: the direction the wind is coming from Ocean Current: the direction the current is going to Potential for no answer or the wrong answer to important societal questions due to separate NOAA data management systems 20
Climate Data Access Issues Emerging Web Portals Technology A web portal is a site or system that creates a single point of access to information collected from different sources Consistent interface among disparate data systems Personalized views and navigation based upon user interests Users are able to easily locate (and relocate) information and services Community-building tools such as chat rooms, bulletin boards, list servers, others Portal Example: National Integrated Drought Information System (NIDIS) 21
Questions 22