Processing EO data and its analysis with respect to models Professor John Remedios NCEO/U.Leicester Inputs: H.Boesch, D. Ghent, D. Moore, R. Parker
Outline Characteris/cs of EO data Some examples from EO data processing Future scaling EO data and models Summary
(Some) Characteris/cs of EO data There are different levels of EO data Level 1, geophysically located and calibrated signals Level 2, geophysical products at instrument resoluoon Level 3, gridded products Level 4, assimilated and synergeocproducts The largest EO data sets are level 1 from which products are derived. However, even level 2 data sets are increasing in size. The largest call on CPU Ome is in level 1 - > level 2 products. MathemaOcally complex data inversions (retrievals), parocularly from spectral data. Physical retrievals (image processing) with large i/o as lots of measurements to process CompilaOon of auxiliary informaoon needed for the retrieval process.
Greenhouse gases SCIAMACHY to Carbonsat
Spectrometers (Carbonsat Instrument requirements) Spectral resolution, spectral coverage and SNR requirements where optimised with the goal to maintain high precision and accuracy, while allowing a less challenging instrument design (During Phase A preparation activities) Data Products Spectral range Spectral resolution Spectral Sampling Ratio Band NIR SWIR-1 SWIR-2 aerosol, cloud, CO 2, H 2 O p surf, CO 2,, CH 4, scattering fluorescence correction, cirrus spectral requirements 747 773 1590-1675 1925 2095 0.1 0.3 0.55 3 6 3-6 3 6 parameters for the SNR requirement CarbonSat EE8 candidate L ref [phot/s/ 3.0 x 10 12 1.0 x 10 12 3.0 x 10 11 nm/cm 2 / sr] SNR ref (T) 150 160 130 SNR ref (G) 300 320 260
Mature Observation Concept Heritage from missions like SCIAMACHY, GOSAT and OCO with a wealth of published results GOSAT CO2 (Cogan et al., JGR, 2012) TCCON GOSAT GOSAT CH4 (Parker et al., GRL, 2011) SCIAMACHY CO2 SCIAMACHY CH4 CarbonSat EE8 candidate (Schneising et al., ACP, 2011; Heymann et al., AMT, 2012, Reuter et al., JGR, 20011, Buchwitz. et al., ACP, 2007)
GHG group - Processing OpOmal esomaoon (complex mathemaocal inversions) From spectra Large processing acovioes. Currently process using combinaoon of Auxiliary data preparaoon has large memory requirements (20 Gb+ RAM) Ø Takes L1B data and various other inputs (model and satellite) to create inputs needed to run retrievals Level 1 to level 2 processing Ø 1 month of global satellite data contains 30,000+ spectra to be retrieved Ø 3 year mission - > 1 million + retrievals Ø may be run several Omes due to development/reprocessing/ updated input data Ø Each retrieval can take anyway from 10 minutes (methane) to 2 hours (carbon dioxide); pair of quad- core; 2.67 GHz Intel Xeon X5550 ; 12 Gbytes of RAM Ø Run as large array jobs on HPC Data storage: L1 is 15 Tbytes (GOSAT) for mission. L2 is only 2 Tbytes.
Greenhouse gases: future Future greenhouse gas missions will lead to a step change in data volumes SCIAMACHY and GOSAT are current; OCO- 2 will fly; Carbonsat is proposed. CARBONSAT GOSAT SCIAMACHY OCO Figure: Max Reuter / IUP Ø There will also be mulople greenhouse gas missions, parocularly CH 4. Ø Improved products Ø SynergeOc products
Carbon Monoxide and organic species (tropospheric and UTLS chemistry) (MIPAS to) IASI to IRS on MTG- Sounder
ULIRS - University of Leicester IASI Retrieval Scheme Pre- process and Cloud Clear IASI Data Get ECMWF T, AlOtude and H 2 O Profiles Get Surface Emissivity from CIMSS Calculate a priori staosocs from TOMCAT (for CO) or mix INTEX- B/ ACE/MIPAS for organics. Convert to Total Column Using DEM Data Perform OEM Retrieval Using Oxford RFM Select Climatology Full details of ULIRS in Illingworth et al., AMT, 2011
IASI data processing Metop series producing large datasets IASI global brightness temperatures Carbon Monoxide opomal esomaoon retrieval
IASI Formic acid HCOOH Volume mixing raoo at 5 km HCOOH flight B622 High HCOOH originates from west of Quebec (Hysplit trajectories) IASI HCOOH vmrs at 5 km compare well with BORTAS measurements IASI most sensiove to HCOOH in mid- troposphere (~4km) Lek: A Formic acid averaging kernel (enhanced case) 5 km
ORGANIC COMPOUNDS Hilton et al, Hyperspectral Earth Observa/on from IASI: four years of accomplishments, BAMS, 2012 Moore, Remedios, Waterfall, MIPAS acetone, ACP, 2012 IASI and MIPAS satellite instrument observaoons of tropospheric polluoon (Moore) CO C 2 H 2 HCOOH ObservaOon from Black Saturday - A series of bushfires which started in South- Eastern Australia on February 7 th 2009 ObservaOons from the NERC- funded BORTAS campaign in July 2011. Satellite data in conjuncoon with aircrak data Aerosol as well as trace gases MIPA S
Data Processing Rapidly expanding data storage requirements MIPAS (limb sounder; sparse coverage) = 13 Tbytes for whole mission IASI (nadir sounder; dense coverage) = 11 Tbytes per year; 15 year lifeome of Metop - > 165 Tbytes for mission More than one IASI (Metop- A and Metop- B) Pre- processing (MIPAS and IASI) Rapidly expanding no of trace gases. MIPAS originally had 7 operaoonal products. Offline retrievals of over 40 species plus aerosols, clouds! The future: IRS on MTG- Sounder (GeostaOonary). 19 Gbytes/ 15 mins or 700 Tbytes/year from 2018 onwards.
Land Surface Temperature 1 km spa/al resn. temperature ATSR, MODIS, SEVIRI
ATSR data processing AATSR Level-3 product at user-defined spatial resolution Global daytime July 2007 at 0.05 AATSR urban heat mapping (Leicester) Europe daytime Feb 2011 at 0.25
ATSR /me series AATSR LST dayome anomalies during relaove heatwaves
LST Data Processing Rapidly expanding data storage requirements ATSR = 37 Tbytes for whole mission (3 instruments; 1991-2012) Future: SenOnel- 3 SLSTR L1 = 230 Tbytes/yr; Synergy L1 = 280 Tbytes/yr SLSTR L2 (marine) = 30 Tbytes/yr; SLSTR L2 (land) = 14 Tbytes/yr Need to re- process to climate quality. Hence several re- processings of data Trade- off between speed, numbers of data files, file sizes and transfer Ome from storage to processing units. Growth of synergy in EO data processing
Synergy: SLSTR and OLCI Complete overlap with OLCI (SLSTR nadir view offset from orbit track) One common VIS channel (865 nm) used for co- locaoon. Synergy level 1c product. Synergy aerosols over land.
Greenhouse gas data sets and models Cogan, Boesch, Parker et al, JGR (in review); Parker et al, GRL (methane); Reuter et al, ACPD
Driving mesoscale Example: assimilaoon of MSG LST into JULES in Sahel (Chris Taylor, Phil Harris CEH) JULES forced by precipitation, radiation etc Assimilating LST, provides mesoscale structure critical for storm initiation
Summary Increasing data volumes mean that transfer and storage of data sets is a problem again (local storage and processing having been very possible over the last five years for many data sets). Processing needs to be carried out close to the data to avoid file transfer (file outputs are generally much smaller than file inputs). Increasing need for re- processing for climate and generaoon of mulople products from same dataset Need mulople data sets co- hosted. CEMS concept meets these objecoves Increasing requirements for: Model- data intercomparison Data assimilaoon Hence Jasmin- CEMS