Introducing NOAA s Microwave Integrated Retrieval System (MIRS)

Transcription

1 Introducing NOAA s Microwave Integrated Retrieval System (MIRS) S-A. Boukabara, F. Weng, R. Ferraro, L. Zhao, Q. Liu, B. Yan, A. Li, W. Chen, N. Sun, H. Meng, T. Kleespies, C. Kongoli, Y. Han, P. Van Delst, J. Zhao and C. Dean NOAA/NESDIS Camp Springs, Maryland, USA 5 th International TOVS Study Conference (ITSC-5), Maratea, Italy, October 9 th 2006

2 Contents Overview 2 Algorithm Scientific Basis 3 Performance Evaluation 4 Summary & Online Access 2

3 Overview Stated Goals of MIRS Algorithm for sounding, imaging, or combination thereof Applicable to all Microwave Sensors Extend over non-oceanic surfaces & in all-weather conditions Operate independently from NWP model forecasts Benefits Reduction of Time/Cost to Adapt to New Sensors Reduction of Time/Cost to Transition to Operations Improvements in Severe Weather Forecasts Better Climate Data Records 3

4 MIRS Concept Variational Retrieval (DVAR) CRTM as forward operator, validity-> clear, cloudy and precip conditions Emissivity spectrum is part of the retrieved state vector Algorithm valid in all-weather conditions, over all-surface types Cloud & Precip profiles retrieval (no cloud top, thickness, etc) Sensor-independent Flexibility and Robustness EOF decomposition Highly Modular Design Selection of Channels to use, parameters to retrieve Modeling & Instrumental Errors are input to algorithm 4

5 Contents Overview 2 Algorithm Scientific Basis 3 Performance Evaluation 4 Summary & Online Access 5

6 6 ( ) ( ) ( ) ( ) + = Y(X) Y E Y(X) Y 2 X X B X X 2 J(X) m T m 0 T 0 Cost Function to Minimize: To find the optimal solution, solve for: Assuming Linearity This leads to iterative solution: Cost Function Minimization 0 J'(X) X J(X) = = + + = + n K n ΔX Y(X n ) Y m K n T E Kn K n T E B n ΔX + + = + n K n ΔX Y(X n ) Y m E K n BK T n BK T n n ΔX + = 0 x K x ) 0 y(x y(x) More efficient ( inversion) Preferred when nchan << nparams (MW) Jacobians & Radiance Simulation from Forward Operator: CRTM

7 Quality Control of MIRS Outputs Convergence Metric: χ 2 Uncertainty matrix S: K T K T S BB K B + E K B = Contribution Functions D: indicate amount of noise amplification happening for each parameter. D = B K T K B K T + E Y(X) K X 0 Average kernel A: A = D K If close to zero, retrieval coming essentially from background If close to unity, retrieval coming from radiances: No artifacts from background 7

8 System Design & Architecture Raw Measurements Level B Tbs Radiometric Bias NEDT Matrx E Heritage Algorithms NOAA-8/METOP st Guess MSPPS + Locally Developed Algorithms for: Temperature Profile Water Vapor Profile Emissivity Spectrum Ready-To-Invert Radiances Heritage Algorithms Radiance Processing Advanced DMSP SSMI/S Retrieval (DVAR) FNMOC Operational Algs (Grody 99, Ferraro et al. 994, Weng and Grody 994, selection Alishouse et al. 990) + Inversion Process MIRS Products Locally Developed Algorithms for: Temperature Profile Water Vapor Profile Emissivity Spectrum Vertical Integration & Post-processing External Data & Tools Other MW sensors Inputs To Inversion WINDSAT Combination of : Monitored Used In Geoph. Monit. - Existing Ready-To-Invert Published Algorithms Radiances Complemented by RTM Uncert. Matrx - Locally Developed F Statistical Algorithms NWP Ext. Data Legend EDRs 8

9 System Design & Architecture Raw Measurements Level B Tbs Advanced Algorithm Outputs Radiometric Bias NEDT Matrx E Ready-To-Invert Radiances Radiance Processing Heritage Vertical st Guess Integration Advanced and Post-Processing Retrieval Algorithms Advanced Algorithm (DVAR) Measured Radiances Temp. Profile TPW RWP Humidity Profile Vertical IWP Comparison: Fit Yes Integration CLW Initial State Vector Simulated Liq. Amount Radiances Prof Ice. Amount Prof Rain Amount Prof Emissivity Spectrum Skin Temperature CRTM Core Products selection No Inversion Process MIRS Update ProductsPost EDRs Within Noise Level? State Vector Processing (Algorithms) New State Vector Vertical Integration & Post-processing External Data & Tools Solution Reached Inputs To Inversion Monitored Used In Geoph. Monit. Ready-To-Invert Radiances -Snow Pack Properties -Land Moisture/Wetness -Rain Rate RTM Uncert. Matrx -Snow Fall Rate F -Wind Speed/Vector -Cloud Top -Cloud Thickness NWP Ext. Data -Cloud phase -Etc. Legend 9

10 Contents Overview 2 Algorithm Scientific Basis 3 Performance Evaluation 4 Summary & Online Access 0

11 Performance Evaluation 4 Hurricane Conditions 3 Validation 2 Over Land (TPW) Imaging (TPW) Profiling (T,Q)

12 QC of the Validation Set Error increases with increasing time separation 2

13 Improvement Assessment (wrt Heritage) MSPPS: NOAA s operational system responsible for deriving microwave products MSPPS MIRS MSPPS MIRS (bias) (Bias) (Std) (Std) N Improvement 6% (%) N % N % Average TPW Standard Deviation Improvement is 6% over ocean Better scan angle handling Independence from NWP forecast outputs Capability extended over land 3

14 Performance Evaluation Temperature 4 & Humidity profiles using N5,6,7,8 Hurricane Conditions Comparison with radiosondes (statistical) 3 Validation 2 Over Land (TPW) Imaging (TPW) Profiling (T,Q) 4

15 Temperature & Humidity Profiles Raob Profiles with at least 30 levels used. Ocean cases only. Retrievals up to 0.05 mbars. Assessment only up to 300 mbars. These are real data performances (stratified by sensor) Results shown here are cloudy (up to 0.5 mm from MIRS retrieval) Independent from NWP forecast information, including surface pressure Improvements in progress (scan-dependent covariance Matrix, air-mass preclassification, etc) 300 mb Pressure (mb) NPOESS IORD-II Requirements 300 mb Pressure (mb) NOAA-5 NOAA-6 NOAA-7 Temperature Std Deviation Error Humidity Std Deviation Error 5

16 Importance of the Evaluation Set Polar Profiles Tropical Profiles Temperature Std Deviation Error Focus of current efforts: - Air-mass preclassification of the background - Improvement expected from first guess 300 All Profiles & All sensors together Temperature Std Deviation Error Temperature Std Deviation Error Caution must be exercised when comparing performances of algorithms on different sets. Types of sets are critical (tropical, polar, clear, cloudy, etc and their relative percentage in the set). 6

17 Performance Evaluation TPW retrieval extended over land Comparison with GDAS analyses Validation Hurricane Conditions Comparison with Radiosondes over land Sanity Check of MIRS Emissivity Over Land 2 4 Over Land (TPW) 3 Imaging (TPW) Profiling (T,Q) 7

18 Microwave TPW Extended over Land snow-covered surfaces need better handling GDAS Analysis Retrieval over sea-ice and most land areas capturing same features as GDAS MIRS Retrieval 8

19 Validation of TPW Retrieval over Land MIRS-based TPW Performances over Land Bias: -.3 mm Std Dev: 4.09 mm Corr. Factor: 0.86 #Points: 4293 ~4000 NCDC IGRA points collocated with NOAA-8 radiances Only convergent points over land used Only points within 0.5 degrees and within hour Cloudy points included, up to 0.5 mm 9

20 Extension of MIRS Validity Over Land Same MIRS code used for retrieval over land and ocean. Approach based on a switch between surface emissivity constraints matrices River/Coast signature(based on surface classifier/topography) 20

21 Global Temperature Profiling No Scan-Dependence in retrieval Smooth Transition Land/Ocean QC-failure is based on convergence: Focus of on-going work Similar Features Captured 2

22 Performance Evaluation 4 Hurricane Conditions Temperature profiling in active regions (using NOAA-8 sounders) Validation 2 Over Land (TPW) 3 Imaging (TPW) Profiling (T,Q) 22

23 Challenges of Profiling in Active Areas 700 mb Case of July 8 th mb DeltaT=3K MHS footprint size at nadir is 5 Kms. But at this angles range (around 28 o ), the MHS All these 4 Dropsondes were footprint is around 30 Kms dropped within 45 minutes and are located within 0 kms from each other Zoom in space (over the Hurricane Eye) and Time (within 2 hours) Temperature [K] DeltaQ=4g/Kg Water Vapor [g/kg] 23

24 N-8 Profiling In Active Areas 700 mb 700 mb 700 mb 0.2 Hrs 2.6 Kms 0.30 Hrs. Kms 0.7 Hrs 4.2 Kms Deg C Retrieval GDAS DropSonde [Deg. C] [Kms] Profile of DS Distance Departure 24

25 Contents Overview 2 Algorithm Scientific Basis 3 Performance Evaluation 4 Summary & Online Access 25

26 MIRS Applications DMSP SSMIS POES N8 MIRS is applied to a number of microwave sensors, Each time gaining robustness and improving validation for Future New Sensors CORIOLIS WINDSAT NPP/NPOESS ATMS, MIS Cumulative Validation & Consolidation of MIRS : Applied Daily : Applied occasionally : Tested in Simulation 26

27 Online Access Online Scrolling Menus 27

28 Products Performance Monitoring Functionalities (cont d) 28

29 Questions?

30 BACKUP SLIDES 30

31 Core Retrieval Mathematical Basis In Bayes plain words: Theorem (of Joint probabilities) Main Goal P(X, Y) in = ANY P(Y Retrieval X) P(X) = System P(X Y) is P(Y) to find a vector X with a maximum probability of being the source responsible for the measurements vector Y m Mathematically: P(X Y m ) = P(Y m X) P(Y m P(X) ) P(X Y m ) is Max Main Goal in ANY Retrieval System is to find a vector X: = 3

32 32 Mathematically: Core Retrieval Mathematical Basis Main Goal in ANY Retrieval System is to find a vector X with a maximum probability of being the source responsible for the measurements vector Y m Main Goal in ANY Retrieval System is to find a vector X: P(X Y m ) is Max In plain words: Problem reduces to how to maximize: Probability PDF Assumed Gaussian around Background X 0 with a Covariance B Mathematically: Probability PDF Assumed Gaussian around Background Y(X) with a Covariance E Y(X) Y m E T Y(X) Y m 2 exp 0 X X B T 0 X X 2 exp Maximizing Y(X) Ym E T Y(X) Ym 2 exp 0 X X B T 0 X X 2 exp Is Equivalent to Minimizing ) m Y P(X ln Which amounts to Minimizing J(X) also called COST FUNCTION Same cost Function used in DVAR Data Assimilation System ( ) ( ) ( ) ( ) + = Y(X) Y E Y(X) Y 2 X X B X X 2 J(X) m T m 0 T 0 = ) m Y P(X

33 System Design & Architecture Raw Measurements Level B Tbs Heritage Algorithms st Guess Ready-To-Invert Radiances Radiance Processing Advanced Retrieval (DVAR) Vertical Integration & Post-processing External Data & Tools Inputs To Inversion Monitored Legend Used In Geoph. Monit. Radiometric Bias NEDT Matrx E selection Inversion Process MIRS Products Ready-To-Invert Radiances RTM Uncert. Matrx F NWP Ext. Data EDRs 33

34 System Design & Architecture Raw Measurements Level B Tbs Radiance Processing & Radiometric Bias Monitoring External Data & Tools Radiometric Bias NEDT Matrx E Geophysical Bias Inversion Process EDRs Ready-To-Invert Radiances RTM Uncert. Matrx F NWP Ext. Data Comparison In-Situ Data 34

35 Nominal approach: Simultaneous Retrieval Necessary Condition (but not sufficient) F(X) Does not Fit Y m within Noise F(X) Fits Y m within Noise levels X is not the solution X is a solution X is the solution All parameters are retrieved simultaneously to fit all radiances together 35

36 Assumptions Made in Solution Derivation The PDF of X is assumed Gaussian Operator Y able to simulate measurements-like radiances Errors of the model and the instrumental noise combined are assumed () non-biased and (2) Normally distributed. Forward model assumed locally linear at each iteration. 36

37 Retrieval in Reduced Space (EOF Decomposition) All retrieval is done in EOF space, which allows: Retrieval of profiles (T,Q, RR, etc): using a limited number of EOFs More stable inversion: smaller matrix but also quasi-diagonal Time saving: smaller matrix to invert Mathematical Basis: EOF decomposition (or Eigenvalue Decomposition) By projecting back and forth Cov Matrx, Jacobians and X Θ = LT B L Diagonal Matrix (used in reduced space retrieval) Transf. Matrx (computed offline) Covariance matrix (geophysical space) 37

38 Retrieval in Logarithm Space P=46 mb P=606 mb P=506 mb P=78 mb P=46 mb P=606 mb P=506 mb P=78 mb P=80 mb P=80 mb P=95 mb P=95 mb J = R = R x = J l Log(x) x Log(x) x 38 Advantages: () Distributions made more Gaussian & (2) No risk of having unphysical negative values Applied to WV, Cloud and precip

39 Validation Approach Use of Multiple Microwave Sensors: AMSU A/B (or MHS) onboard NOAA WINDSAT onboard CORIOLIS SSMI/S onboard DMSP F-6 Two Types of Validation, depending on parameter Quantitative Validation NWP Data (GDAS) Heritage Algorithms (MSPPS) Conventional Radiosondes (from NCEP and from NCDC) GPS-DropSondes Qualitative Validation Science Constraints in Retrieval System Capture of known meteorological phenomena Metrics: Standard statistical metrics Bias/RMS/StdV/Correlation Case By Case Evaluation (especially for active areas) 39

40 TPW Comparison MIRS vs MSPPS MSPPS TPW used as reference MSPPS TPW [mm] MIRS retrieves the humidity profile. The TPW is integrated in postprocessing stage. MSPPS relies on NWP forecast for both SST and Wind (emissivity). MIRS TPW [mm] MIRS is independent of NWP data (even from surface pressure). 40

41 Cloud Retrieval Using SSMI/S FNMOC Operational Algorithm MIRS Algorithm MIRS is more sensitive to small values (due to use of higher frequency channels) Retrieval using DMSP F6 SSMI/S 4

42 In-Situ Global Distribution Source Period Coverage # of Points Ref. POES NOAA5 NCEP Ocean 255 Liu & Weng 2004 POES NOAA6 POES NOAA7 NCEP NCEP Ocean Ocean Liu & Weng 2004 Liu & Weng 2004 POES NOAA8 NCDC-IGRA Land ~8,000 Durre et al Dataset used for both TPW and Profiling Validation 42

43 Retrieval Bias vs. Viewing Angles Match-up TPW from radiosondes and AMSU retrieval in Bias variation to viewing angles. Bias = radiosonde AMSU 4 NOAA-6, bias=radiosonde-retrieval 3 TPW bias (mm) 6 NOAA-5, bias = radiosonde - retrieval viewing angle (degree) dvar MSPPS TPW bias (mm) dvar MSPPS TPW bias (mm) NOAA-7, bias-radiosonde-retrieval dvar MSPPS viewing angle viewing angle (degree) 43

44 Emissivity Qualitative Validation Emissivity 3 GHz Em3GHz MIRS Retrieval EM3GHz Analytical Computation Using MIRS and AMSU/MHS: Diagonal emissivity variances boosted (over land) Off-diagonal zeroed out (channels independent) Use of GDAS (T,Q, Tskin) as st guess & Backg but parameters free to move ε = = Analytic Extraction: T T B + Γ. T T s + Γ.T ( ) Two conditions: T s T Γ 0 44

45 Global Humidity Profiling No Scan-dependence noticed: Angle dependence properly accounted for 45

46 Performance Evaluation 5 Hurricane Conditions Effect of using scattering RTM on convergence Advanced vs Heritage (using NOAA-8 2 sounders) Advanced Products Validation Skin temperature retrieval using WINDSAT 3 in eye of hurricane Temperature profiling in active regions Profiling Over Land 4 46

47 WINDSAT Retrieval (Chi Square) Rain Model OFF Rain Rain Model Model OFF ON Retrieval using Windsat data (sdr68) Spatial resolution of 6.8 GHz (50 kms) But with a lot of oversampling Non-convergence along coast lines and at cloud/rainy cell boundaries Beam filling (cloud and coastal) Not handled properly 47

48 SST Retrieval In Hurricane Conditions During Hurricane Dennis on July 6 th 2005, WINDSAT Data captured Skin Temperature Cooling inside Eye of Hurricane Dennis Hurricane Track, Courtesy of National Hurricane Center MIRS SST Retrieval Using WINDSAT (Frq 6 to 37 GHz) GDAS SST Analysis 48