Kernel-Based Retrieval of Atmospheric Profiles from IASI Data

Transcription

1 Kernel-Based Retrieval of Atmospheric Profiles from IASI Data Gustavo Camps-Valls, Valero Laparra, Jordi Muñoz-Marí, Luis Gómez-Chova, Xavier Calbet Image Processing Laboratory (IPL), Universitat de València. Spain EUMETSAT, Darmstadt, Germany IGARSS 2011, 25-29th July, Vancouver, Canada 1 / 23

2 Motivation Retrieval of atmospheric profiles Temperature and humidity are basic meteorological parameters for weather forecasting and atmospheric chemistry studies High spectral resolution infrared sounding instruments provide high accuracy and vertical resolution of temperature and water vapour profiles Fast, non-linear, multi-output regression methods are needed 2 / 23

3 MetOp-IASI MetOp satellite series managed by EUMETSAT Band 1 Band 2 Band 3 Complex signal processing problem: High input (radiances) dimensionality High output (state vectors) dimensionality High levels of noise in particular channels High temporal and spatial redundancy: high data volume Nonlinear input-output relations 3 / 23

4 Objectives Main objective Nonlinear retrieval of atmospheric states from IASI radiance spectra Specific objectives Develop advanced nonlinear multi-output regression for IASI data The retrieval method must be scalable, fast and accurate Robust to noisy scenarios, clouds, both over ocean and land Provide confidence intervals for estimations Nonlinear anomaly detection (quality flags) are developed T Td O3 X Y n x 8461 n x / 23

5 Approaches Current approach in EUMETSAT L2 PPF: PCA + Linear Regression (LR) retrievals are fed to Optimal Estimation (OE) procedure LR is fast, but too simple and inaccurate OE is accurate, but extremely slow 5 / 23

6 Approaches Current approach in EUMETSAT L2 PPF: PCA + Linear Regression (LR) retrievals are fed to Optimal Estimation (OE) procedure LR is fast, but too simple and inaccurate OE is accurate, but extremely slow Neural networks nonlinear retrieval (Aires, 02; Blackwell 05; Camps-Valls, 10) Neural nets have been successfully used for IASI and AIRS Fast (on test) and accurate Slow and difficult to train (many parameters to adjust) 6 / 23

7 Approaches Current approach in EUMETSAT L2 PPF: PCA + Linear Regression (LR) retrievals are fed to Optimal Estimation (OE) procedure LR is fast, but too simple and inaccurate OE is accurate, but extremely slow Neural networks nonlinear retrieval (Aires, 02; Blackwell 05; Camps-Valls, 10) Neural nets have been successfully used for IASI and AIRS Fast (on test) and accurate Slow and difficult to train (many parameters to adjust) Kernel Ridge Regression (KRR) retrieval Can tackle efficiently with multioutput problems Training is easier and faster (only two intuitive parameters must be tuned) It provides a ranked list of most important IFOV used in training Confidence intervals for the predictions can be obtained 7 / 23

8 Learning scheme Developed NLR processor Feature Selection (Calbet, 2008) Feature Extraction (PCA) Nonlinear Regression (KRR) Feature selection (Calbet, 2008): Avoid channels with negative radiances Noise bias-variance criteria: bias > 4K and std dev. > 3K for IASI Feature extraction: PCA feature extraction Multioutput nonlinear regression: Kernel ridge regression 8 / 23

9 Nonlinear regression Kernel ridge regression (KRR), aka Least Squares SVM Regression model: Y = ΦW + E Assume a squared loss function in H: min n Y ΦW 2 + λ W 2o W Representer s theorem: W = Φ α Solve: α = (λi + ΦΦ {z } ) 1 Y K The prediction function: Ŷ = f (x ) = Φ(x )W = Φ(x )Φ α = K(X, x )α We use the RBF kernel function: K(x i,x j ) = exp(- x i x j 2 /(2σ 2 )) Confidence on the prediction: V[f (x )] = K(x, x ) K(x, X)(K + λi) 1 K(X, x ) 9 / 23

10 Nonlinear regression Key features KRR generalizes LR Tune two parameters: σ and λ Fast for training (few hours) and test (25 ms/fov) 10 / 23

11 Nonlinear regression Key features KRR generalizes LR Tune two parameters: σ and λ Fast for training (few hours) and test (25 ms/fov) Fast implementation Standard code: >> alpha = inv(lambda*eye(n) + K) * Y; Cholesky decomposition is 4 times faster: >> R = chol(gamma*eye(n) + K); >> alpha = R\(R \Y); 11 / 23

12 Experiments Datasets Training done with ECMWF a Chevallier s database: IASI cloud free, emissivity sea, noise-free FOVs: Methodology for training: Feature selection (Calbet, 2008): X X = [X, surface pressure, scan angle, latitude] Feature extraction: PCA with X, and retain a number of features p LR: use all training data to estimate model weights KRR: cross-validation ( 2 3, 1 3 ) to estimate model parameters use all data to estimate model weights a European Centre for Medium-Range Weather Forecasts Results Real datasets, IASI orbits with FOVs Predicted error profiles of temperature and water vapour Confidence maps and detection of anomalies 12 / 23

13 KRR testing at EUMETSAT over ocean... KRR clearly outperforms LR Very good results in water vapour 13 / 23

14 KRR testing at EUMETSAT over land... KRR outperforms LR, which dramatically fails Errors are similar to estimations over the ocean Temperature errors are reasonable, while water vapour is really good 14 / 23

15 KRR results Results with and without cloud masking T LR (all) KRR (all) LR (masked) KRR (masked) 10 2 Td p [hpa] p [hpa] RMSE [K] RMSE [K] Clouds and anomalies are an important error source Cloud screening is mandatory An anomaly detector can be developed 15 / 23

16 Predictions, discrepancies and confidence maps: Madagascar AVHRR KRR confidence map IASI cloud flag T : ˆT ECMWF ˆT KRR Cloud flag T 16 / 23

17 Predictions, discrepancies and confidence maps: Mexico coast AVHRR KRR confidence map IASI cloud flag T : ˆT ECMWF ˆT KRR Cloud flag T 17 / 23

18 Anomaly detection scheme Radiances T predictions PCA SVM T errors > 5K Inputs: radiances and/or predictions Output: nonlinear prediction of KRR big discrepancies with ECMWF 18 / 23

19 Anomaly detection results Overall accuracy LDA QDA MAHAL TREE SVM Kappa statistic, κ Test OA Test κ Training samples Training samples Linear and nonlinear classifiers developed Anomalies can be detected accurately (OA 92%, κ = 0.81) SVM outperforms all other classifiers 19 / 23

20 Conclusions Developed and implemented KRR nonlinear regression KRR outperforms LR KRR provides confidence maps 20 / 23

21 Conclusions Developed and implemented KRR nonlinear regression KRR outperforms LR KRR provides confidence maps Developed nonlinear anomaly detection methods 1 Thresholding the discrepancies to ECMWF, = ˆT ECMWF ˆT KRR 2 KRR confidence on predictions, ˆσ T [0, 1] 3 SVM prediction of anomalies: 90% 21 / 23

22 Conclusions Developed and implemented KRR nonlinear regression KRR outperforms LR KRR provides confidence maps Developed nonlinear anomaly detection methods 1 Thresholding the discrepancies to ECMWF, = ˆT ECMWF ˆT KRR 2 KRR confidence on predictions, ˆσ T [0, 1] 3 SVM prediction of anomalies: 90% Future work Include better spatial information Channel emissivity prediction 22 / 23

23 Conclusions Kernel-Based Retrieval of Atmospheric Profiles from IASI Data Gustavo Camps-Valls, Valero Laparra, Jordi Muñoz-Marí, Luis Gómez-Chova, Xavier Calbet Image Processing Laboratory (IPL), Universitat de València. Spain EUMETSAT, Darmstadt, Germany IGARSS 2011, 25-29th July, Vancouver, Canada 23 / 23