Accounting for Model Uncertainty in the Navy s Global Ensemble Forecasting System C. Reynolds, M. Flatau, D. Hodyss, J. McLay, J. Moskaitis, J. Ridout, C. Sampson, J. Cummings Naval Research Lab, Monterey, CA P. Flatau, Scripps Institute of Oceanography, La Jolla, CA Efren Serra, Devine Consulting Incorporated, Freemont, CA Focus on parameter variation results Mention results using SKEB, Stochastic Convection, and Diurnal SST model
Quantifying Forecast Uncertainty Using Ensembles Uncertainty in initial state: Forecasts with different initial conditions Methods to perturb initial conditions: Parallel data assimilation cycles, rapidly growing linear perturbations, Kalman filter methods Ensemble Transform (ET, McLay et al. 2008; banded ET, McLay et al. 2010): Transform 6-h ensemble perturbations to be consistent with analysis error estimates. Because it is a cycling scheme, model perturbations impact initial perturbations. Uncertainty in model formulation: Forecasts with varying models Methods to include model uncertainty: Different forecast models, different subgrid-scale parameterizations, stochastic forcing, boundary forcing (SST, land) Parameter variations Stochastic convection, stochastic kinetic energy backscatter, diurnal SST model 2
Parameter Variation Experiments Atmospheric Forecast Model NOGAPS (Navy Operational Global Atmospheric Prediction System): Spectral model with full suite of physical parameterizations. T119L30 (approximately 110-km horizontal resolution) Ensemble Design: All use ET initial perturbations (McLay et al. 2008); 33 members; 240-h forecasts, May-Sept. 2007 CTL: Control Ensemble ET initial perturbations only, no model uncertainty. PAR1: First Parameter Variation Set 4 parameters varied in deep convection parameterization only PAR2: Second Parameter Variation Set Similar to PAR1, but 3 parameters varied within deep convection and 1 parameter varied in boundary layer parameterization 3
Parameter Variation Experiments Model developer chose parameters to vary, and range of values: Chose parameters to which forecasts were sensitive Set ranges that gave reasonable results Parameters values: Differ for each ensemble member Are held fixed throughout domain and through forecast integration Result in different biases for different members Average summer forecast skill of individual ensemble members very similar (conservative parameter range) Focus on Tropics: Very little impact on extra-tropics 4
Parameter Variations: Ensemble Spread CTL Average Ensemble Spread, 850-hPa Wind Speed, 5-d Forecasts PAR1: Changes in convection only PAR2: Changes in convection and PBL PAR1 CTL % DIFF Small ensemble spread in tropics for CTL. Too small when compared to forecast errors. PAR2 CTL % DIFF PAR1 and PAR2 have significantly larger ensemble spread than CTL in tropics (greater in PAR2 than PAR1) 5
Parameter Variations: RMSE and Ensemble Spread 850-hPa Wind Speed Ensemble Mean RMS Error: Tropics Ensemble Mean RMSE Ensemble Spread Control ensemble significantly under-dispersive 6
Parameter Variations: RMSE and Ensemble Spread 850-hPa Wind Speed Ensemble Mean RMS Error: Tropics Ensemble Mean RMSE Ensemble Mean RMSE with Bias Removed Ensemble Spread Removal of Bias decreases RMSE, but ensemble still under-dispersive. 7
Parameter Variations: RMSE and Ensemble Spread 850-hPa Wind Speed Ensemble Mean RMS Error: Tropics Ensemble Mean RMSE Ensemble Mean RMSE with Bias Removed Ensemble Spread PAR1: Changes in convection only PAR2: Changes in convection and PBL PAR1 and PAR2 increase spread by 10-20%. Very small impact on RMSE. 8
Parameter Variations: Fraction of Outliers 850-hPa Wind Speed Fraction of Outliers: Tropics Bias-removed Ensembles (dashed) Ideal Raw Ensembles (solid) Verification lies outside of control ensemble much more frequently then expected. Bias correction reduces extraneous outliers. Ensembles still under-dispersive, less so as forecast time increases. 9
Parameter Variations: Fraction of Outliers 850-hPa Wind Speed Fraction of Outliers: Tropics Bias-removed Ensembles (dashed) Raw Ensembles (solid) Erroneous outliers decrease with parameter variations (PAR2 better than PAR1), with or without bias removed. All ensembles still under-dispersive. Ideal 10
Parameter Variations: Brier Scores Brier Scores for 10-m Wind Speed in Tropics: 5 m/s Threshold Parameter variations significantly improve probabilistic wind speed forecasts (lower Brier Score is better). PAR2 better than PAR1. Improvement in Brier score expected if spread of underdispersive ensemble is increased (better reliability). 11
Parameter Variations: Brier Scores Brier Score Decomposition: Resolution (sharpness) Reliability (calibration) c d Both resolution and reliability are improved: Parameter variations are improving ability to capture flow-dependent variations in predictability, not just improving match to climatological variance
error (nm) Parameter Variations: Tropical Cyclone Tracks 400 2007 Northern Hemisphere Homogeneous TC Forecast Error (nm) 350 300 250 200 150 CTL PAR1 PAR2 Small improvements to TC track forecasts with inclusion of parameter variations. 100 50 0 12 24 36 48 72 96 120 PAR2 improvements significant (95% level) at 24, 72, and 120 h. forecast time # Cases 103 93 77 69 52 30 17 13
Brier Score Stochastic Kinetic Energy Backscatter: Brier Scores Error (Brier) Score for probability that 10-m Wind Speed 10-m Wind Speed Brier Score: 5 m/s Threshold will exceed 5 m/s threshold, in the tropics. 0.16 0.15 0.14 0.13 0.12 0.11 0.10 0.09 0.08 Global ET: Highest Error New Banded ET: Reduces Error 24 48 72 96 120 144 168 Forecast Hour Banded ET with Stochastic Forcing: Best Performance Global ET Banded ET Stochastic Forcing Banded Ensemble Transform (ET) enhances ensemble performance under many measures (operational Feb. 2010). SKEB (following Berner et al. 2009) has even greater impact. (McLay et al. 2010 WAF, Reynolds et al. 2011 MWR)
SKEB: Rank Histogram Outliers Fraction of Outliers: Tropical 10-m Wind Speed Global ET: New Banded ET: Banded ET with Stochastic Forcing Banded ET decreases the number of outliers over the Global ET, but difference between the two disappears by 168 h. SKEB decreases the number of outliers throughout the forecast. All ensembles are still under-dispersive.
Stochastic Convection: Typhoon Jangmi (2008) No Model Uncertainty Observed track outside ensemble, no TD forecasts Stochastic Forcing Reasonable tracks, more spread, TDs. 21 Sept. 66h before Tropical Depression 26 Sept. 54h after TD Small spread, few recurve More spread, recurvers Stochastic forcing increases ensemble spread and improves prediction of TC genesis (Snyder et al. 2011 MWR).
Prognostic Diurnal SST Prognostic diurnal SST model, along with method to perturb initial SST, modified from Takaya et al. (2010) SST ensemble variance (deg. C) 2 at T+240h. Ensemble maintains SST variance out to 10 days. Prognostic diurnal SST model (along with method to perturb initial SST) improves probabilistic forecasts across a broad range of metrics in the tropics (McLay et al. 2012 GRL) Improved 120-h probabilistic skill scores (CRPS) in the tropics across a broad range of metrics and variables when compared to static SST. 17
Summary and Future Work Summary: Inclusion of model uncertainty: Increase ensemble spread and improves probabilistic forecast skill (e.g., Brier score) in the tropics. Has less impact in the extratropics Has little impact on ensemble mean performance Future Work Explore more comprehensive methods of sampling parameter space (e.g., Latin hypercube) Combine different methods of model uncertainty (stochastic forcing, parameter variations, diurnal SST variations) Apply to the Navy Global Environmental Model 18
Impact on MJO: 5-day Forecast Projection onto WH2004 MJO Index EOFs Ensemble members with high (low) values of the Von Karman constant shown in red (blue). Verifying analysis in black. EOF1 EOF2 Day Day No impact on ensemble mean. Spread increased substantially, but still under-dispersive. Certain high-low parameter values show systematic differences in projection onto Wheeler and Hendon (2004) MJO Index EOFs. Certain parameter values appear to give more skill, but sample too small to be conclusive.
Pressure Pressure 100 Ensemble spread 200 Ensemble mean error 12 -h Total Energy as a Function of Pressure. 350 750 1000 Control error Total Energy in the Tropics: 72 hours CTL SPRD: 12 STOC SPRD: 12 CTL EN MN ERR: 12 STOC EN MN ERR: 12 CTL ERROR: 12 1.00E-02 1.00E-01 1.00E+00 100 72 -h Ensemble Spread (red), Ens. Mean Error (blue), Control Error (black) CTL (dashed) STOC (solid) 200 CTL SPRD: 72 STOC SPRD: 72 CTL EN MN ERR: 72 STOC EN MN ERR: 72 STOC increases spread in tropics throughout depth of troposphere. 350 CTL ERROR: 72 750 1000 1.00E-02 1.00E-01 1.00E+00
Decomposition of the Brier Score The most common verification method for probabilistic forecasts, the Brier score BS is similar to the RMSE, measuring the difference between a forecast probability of an event (p) and its occurrence (o), expressed as 0 or 1 depending on if the event has occurred or not. As with RMSE, the lower the Brier score the "better" The reliability measures the ability of the system to forecast accurate probabilities. Out of a large number of, for example 20% probability forecasts, the predicted event should verify for 20% of the forecasts, not more, not less. The reliability can be displayed in a reliability diagram where the x-axis is the forecast probability and the y-axis the frequency it occurs on those occasions. The resolution indicates the ability of the forecast system to correctly separate the different categories, whatever the forecast probability. For a given reliability, the resolution thus indicates the "sharpness" of the forecast. The maximum resolution corresponds to a deterministic forecast (only 0% and 100% are forecast), the minimum resolution corresponds to a climatological forecast (the same probability is always forecast).
SKEB:T319 20-member Ensemble TC Track Forecasts SKEB does increase ensemble spread (left), but has little impact on the ensemble mean track error (right).
Parameter Variation Details NOGAPS T119 32-member Ensembles for 10 May through 12 Sept 2007 Initial perturbations using global ET CTL ensemble, initial perturbations only PAR1 ensemble, parameter variations in Emanuel only cu: coefficient that scales the computed convective momentum transport, [0, 0.25, 0.5] dtmax: which represents magnitude of small-scale temperature perturbations associated with rising updraft source-layer parcels [0.5, 0.8 1.1] Alpha [0.375, 0.5, 0.625] and damp [0.08, 0.1 0.1333]: which are parameters that control the rate of approach to quasi-equilibrium. PAR2 ensemble, parameter variations in Emanuel (cu, dtmx, sigs) and PBL (von Karman Constant) Sigs: fraction of precipitation falling outside the cloud [.1,.12,.14] Vkrm: constant of the logarithmic wind profile in the surface layer [0.38, 0.4, 0.42]