NCEP Ensemble Forecast System Yuejian Zhu Ensemble team leader Environmental Modeling Center NOAA/NWS/NCEP Acknowledgments: Geoff DieMigo, John Ward, Bill Lapenta and Stephen Lord 1
Index SREF implementation and plan GEFS implementation Bias correction and statistical downscaling Bias correction globally (NAEFS 50 variables) and regional (SREF) Downscaling for CONUS (4 variables) and Alaska (8) The values of NAEFS products Calibration of precipitation forecast Working in progress for improved version GEFS Future plan Increase the resolutions Improve the initializations Improve the stochastic physics Improve post-processing Software for post processing and probabilistic verification Public access through subversion 2
SREF Implementation (October 27 th 2009) -Geoff Dimego, Jun Du and etc Upgrade model versions WRF-NMM from v2.0+ to v2.2+ WRF-ARW from v2.0+ to v2.2+ RSM from v2007 to v2009 Increase horizontal resolution WRF-NMM from 40km to 32km WRF-ARW from 45km to 35km RSM from 45km to 32km Adjust membership Replace 2 Eta (BMJ-sat) members with 2 WRF-NMM members Replace 2 Eta (KF-det) members with 2 WRF-ARW members Enhancement physics diversity of RSM: replace Zhao cloud scheme with Ferrier cloud scheme for 3 SAS members Enhance initial perturbation diversity: Replace regional bred perturbations with global ET perturbations for 10 WRF members 3
SREF Domains model domain (dash) output domain (solid) 4
SREF Evolution 2010 2011 2012 2013 2014 2015 SREF SREF11 SREF13 SREF15* (32km, 4-model) (22km, add 7 NEMS mem, 3-model) (12km, 2-model, NEMS-only, 5days?) 4km? (White Paper) (NAEFS_LAM) *NEMS = NOAA Environmental Modeling System (a unified modeling framework) *SREF (32 22 12 4km) 5
GEFS Implementation (February 23 rd 2010) Use operational GFS (version 8.0) Upgrade horizontal resolution from T126 to T190 4 cycles per day, 20+1 members per cycle Up to 384 hours (16 days) Use 8 th order horizontal diffusion for all resolutions Improved forecast skills and ensemble spread Introduce ESMF (Earth System Modeling Framework) for GEFS Version 3.1.0rp2 Allows concurrent generation of all ensemble members Needed for efficiency of stochastic perturbation scheme Add stochastic perturbation scheme to account for random model errors Increased ensemble spread and forecast skill (reliability) Add new variables (28 more) to pgrba files Based on user request Supports NAEFS ensemble data exchange From current 52 (variables) to future 80 (variables) 6
Anomaly Correlation NH Anomaly Correlation for 500hPa Height GEFSg is better than GFS at 48 hours Period: August 1 st September 30 th 2007 GFS GEFS GEFSg 1 0.9 0.8 0.7 GEFSg could extend skillful forecast (60%) for 9+ days 24 hours better than current GEFS 48 hours better than current GFS 0.6 0.5 0.4 0.3 0.2 0.1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Forecast Lead Time (days) 7
Errors (miles) Summary of the important cases of Bill, Jimena, Rick and Ida TS track errors (2009) opr par 300 250 For 00UTC only 200 150 100 50 0 0 12 24 36 48 72 96 120 Cases 28 26 24 22 20 15 9 6 Forecast hours 8
9
Bias correction and downscaling Bias correction at 1*1 degree resolution (weight=0.02 for Kalman filter algorithm) Bias corrected NCEP/GEFS, GFS (out to 180 hours) and CMC/GEFS forecasts Consider the same bias for NCEP all ensemble members Consider the different bias for each model (member) Combine bias corrected high resolution GFS and low resolution ensembles Dual resolution ensemble approach for short lead time GFS has higher weights at short lead time NAEFS products based on all bias corrected forecasts Produce Ensemble mean, spread, mode, 10% 50%(median) and 90% probability forecast Climate anomaly (percentile) forecasts also generated for ensemble mean Statistical downscaling to NDGD grids (weight=0.2 for Kalman filter algorithm) Proxy for truth - RTMA at 5km/6km resolutions Variables (surface pressure, 2-m temp (and Max/min), and 10-m wind (and speed/direction) Downscaling vector Interpolate GDAS analysis to 5km/6km resolutions Compare difference between interpolated GDAS and RTMA Apply decaying weight to accumulate this difference downscaling vector Downscaled forecast Interpolate bias corrected 1*1 degree NAEFS to 5km/6km resolutions Add the downscaling vector to interpolated NAEFS forecast NAEFS products from downscaling CONUS NDGD grid/resolution (5km) 4 variables for Ensemble spread, mean, mode, 10%, 50%(median) and 90% forecasts Alaska NDGD grid/resolution (6km) 8 variables for Ensemble spread, mean, mode, 10%, 50%(median) and 90% forecasts 10
Bias Correction Method & Application Bias Correction Techniques array of methods Estimate/correct bias moment by moment Simple approach, implemented partially May be less applicable for extreme cases Moment-based method at NCEP: apply adaptive (Kalman Filter type) algorithm decaying averaging mean error = (1-w) * prior a.m.e + w * (f a) For separated cycles, each lead time and individual grid point, a.m.e = averaging mean error 6.6% Test different decaying weights. 0.25%, 0.5%, 1%, 2%, 5% and 10%, respectively Toth, Z., and Y. Zhu, 2001 3.3% 1.6% Decide to use 2% (~ 50 days) decaying accumulation bias estimation 11
Hybrid GFS and GEFS weights Statistically demonstrate that GFS has more skill than ensemble control (lower resolution) for short lead time Combined GFS and GEFS bias Corrected forecast for first 180 hours improves the skill (red) than GEFS only (black) 12
NH500hPa NH1000hPa NHT2M All these stats show the positive impact for probabilistic forecast by apply bias correction and multimodel ensemble (NAEFS) for upper atmosphere (500hPa) and near surface (2-m temperature). Green line is for NAEFS. 13
Probabilities (percent) 2-m temp 10/90 probability forecast verification Northern Hem, period of Dec. 2007 Feb. 2008 P10-expect P10-ncepraw P10-naefs P90-expect P90-ncepraw P90-naefs 100 90 90% 80 70 60 50 40 3-month verifications 30 20 10 0 10% 24 48 72 96 120 144 168 192 216 240 264 288 312 336 360 384 Lead time (hours) Top: 2-m temperature probabilistic forecast (10% and 90%) verification red: perfect, blue: raw, green: NAEFS Left: example of probabilistic forecasts (meteogram) for Washington DC, every 6-hr out to 16 days from 2008042300 14
Downscaling Method with Decaying Averaging Algorithm True = high resolution analysis Operational North American Real-Time Mesoscale Analysis (RTMA) 5x5 km National Digital Forecast Database (NDFD) grid (e.g. G. DiMego et al.) 4 variables available: surface pressure, T2m, 10m U and V Other data can also be used Downscaling method: apply decaying averaging algorithm Downscaling Vector 5km (t 0 ) = (1-w) * prior DV 5km (t -1 ) + w * (GDAS 5km (t 0 ) RTMA 5km (t 0 )) GDAS 5km : GDAS 1x1 analysis interpolated to RTMA 5km grids by bilinear interpolation 4 cycles, individual grid point, DV 5km = Downscaling Vector on 5km grids choose different weight: 0.005, 0.01, 0.02, 0.05, 0.1, 0.2 and 0.5 weight = 0.2 is best and used for weight to calculate downscaling vector Downscaling forecast: Downscaled Forecast 5km (t) = Bias-corrected Forecast 5km (t) DV 5km (t 0 ) Bias-corrected Forecast 5km: interpolated to RTMA 5km grids by bilinear interpolation subtract DV 5km from bias-corrected forecast 5km valid at analysis time 15
One month average of mean absolute error for 2-m temp (against RTMA) GEFS raw forecast GEFS bias-corr. & down scaling fcst. 12hr 2m temp forecast Mean Absolute Error w.r.t RTMA for CONUS average for Sept 2007 MAE (raw) =1.999 MAE (bc_gefs) = 1.161 MAE (NAEFS) = 1.002 NAEFS forecast 16
NCEP/GEFS raw forecast 4+ days gain from NAEFS NAEFS final products From Bias correction (NCEP, CMC) Dual-resolution (NCEP only) Down-scaling (NCEP, CMC) Combination of NCEP and CMC 17
NCEP/GEFS raw forecast 8+ days gain NAEFS final products From Bias correction (NCEP, CMC) Dual-resolution (NCEP only) Down-scaling (NCEP, CMC) Combination of NCEP and CMC 18
Track forecast error for 2009 season (AL+EP+WP) 400 NCEPraw NCEPbc NAEFS 350 300 250 200 150 100 50 0 0 12 24 36 48 72 96 120 Cases 240 223 196 169 144 110 75 42 NAEFS is combined NCEP (NCEPbc) and CMC s (CMCbc) bias corrected ensemble and bias corrected GFS 19 Contributed by Dr. Jiayi Peng (EMC/NCEP)
The evaluations from WFO (State College) forecaster for NAEFS mean minimum temperature Minimum temperature forecast: Average over past 30 days: (20080929-20081028) MAE Bias >10 err <3 err off. rank Best G. 2nd G. Worst G. 1 12-hr 3.17-1.2 1.0% 53.4% 3 out of 7 NAEFS 59.7% SREF 57.1% NGM80 21.8% 2 24-hr 3.03-0.9 0.6% 55.5% 2 out of 7 SREF 57.2% NAEFS 54.2% NGM80 24.9% 3 36-hr 3.25-0.8 0.9% 51.6% 3 out of 7 NAEFS 54.2% SREF 53.9% NGM80 23.2% 4 48-hr 3.94-1.1 2.9% 43.2% 3 out of 7 NAEFS 51.9% SREF 45.8% NGM80 6.2% 5 60-hr 4.30-0.4 4.4% 39.1% 4 out of 6 NAEFS 49.2% SREF 43.0% NAM40 8.9% 6 72-hr 4.76 0.1 6.4% 33.7% 5 out of 5 NAEFS 42.9% SREF 40.1% NAM12 35.2% 7 84-hr 4.85 0.3 7.5% 34.7% 2 out of 6 NAEFS 40.0% MOSGd 33.4% NAM12 8.9% 8 96-hr 5.24 0.4 13.0% 33.1% 1 out of 3 NAEFS 32.7% MOSGd 29.9% MOSGd 29.9% 9 108-hr 5.11 0.8 12.8% 35.4% 1 out of 4 HPCGd 34.5% NAEFS 32.1% MOSGd 30.5% 10 120-hr 5.31 0.7 12.0% 31.9% 1 out of 3 MOSGd 31.6% NAEFS 24.8% NAEFS 24.8% 11 132-hr 4.97 0.7 9.9% 35.1% 2 out of 4 HPCGd 38.0% MOSGd 30.9% NAEFS 27.2% 12 144-hr 5.42 0.6 15.0% 35.0% 1 out of 3 MOSGd 31.3% NAEFS 29.0% NAEFS 29.0% 13 156-hr 5.40 0.5 14.9% 35.7% 1 out of 4 HPCGd 32.9% MOSGd 32.7% NAEFS 23.4% 14 168-hr 5.46 1.1 17.7% 38.1% 1 out of 3 MOSGd 35.6% NAEFS 28.4% NAEFS 28.4% Official Guidance: NGM80, NAM40, SREF, NAM12, MOSGd, HPCGd, NAEFS NAEFS downscaled 5km (NDGD) minimum temperature (mean) is the best guidance for first 96hr forecasts from 7 different guidance Contributed by Richard Grumm (WFO) 20
Application for Alaska region and HPC Alaska desk 10-m U Max Temp Solid RMS error Dash - spread Bias (absolute value) 10-m wind speed Max Temp CRPS small is better Bias (absolute value) 21
Calibration of precipitation forecast Implemented May 2004 (HPC and CPC endorsed) Latest experiments for every 6hr instead of 24hr METHOD 1) Construct cumulative frequency distributions for forecast QPF & corresponding observed values 2) For each forecast value, find the observed value that has the same frequency as forecast value 3) Re-label forecast value with corresponding observed value DETAILS Observations used: CCPA climatelogcal calibrated precipitation analysis Adaptive method, training data accumulated over: Most recent ~30-day period Decaying averaging More weight on most recent data Continental US Linear inter/extrapolation Corrections applied CONUS (and globally) on model grid Correction is function of forecast value 1*1 degree (and 5km downscaled) spatial resolution Every 6-hour forecast interval (3-hour later) 22
Precipitation calibration for 2009-2010 winter season (CONUS only) Comparison for GFS and ensemble control (raw and bias corrected) ETS for all lead-time ETS for 0-6hr fcst BIAS for all lead-time BIAS for 0-6hr fcst Perfect bias = 1.0 The probabilistic scores (CRPS -not show here) is much improved as well. We are still working on the different weights, different RFC regions, downscaled 23 to 5km as well. More results will come in soon. Plan for implementation: Q2FY11
NCEP next GEFS Configurations (Q4 FY11) - Yuejian Zhu (07/14/2010) According to total resource distribution for each model (Jigsaw puzzle) GEFS has 40% of total CPUs (52 of 130) during +4:35 and +6:00 for main integration and main post-process Current: GEFS and GEFS/NAEFS post processing T190L28 for all 384 hours lead-time 20+1 members per cycle, 4 cycles per day Computation usage: average 20 nodes (22 high mark) for 50 minutes Next GEFS and GEFS/NAEFS post processing (Q3/Q4FY2011): T254L42 (0-192hr) increasing both horizontal and vertical resolutions Factor of 3.6 by comparing T190L28 T190L42 (192-384hr) increasing vertical resolution Factor of 1.5 by comparing T190L28 20+1 members per cycle, 4 cycles per day Total cost for integration and post processing Factor of 3.6 for 0-192hrs, factor of 1.5 for 192-384 Average factor for processing (0-384hrs) is 2.55 51 nodes for 50 minutes (start: +4:35 end: +5:25) Why do we make this configurations? Considering the limited resources Increasing horizontal resolution is more important than vertical resolution 24
Different ensemble size from The same model resolution (T126L28) 80 members, 60 members 40 members, 20 members 10 members, 5 members Trade off for resolution And ensemble size 20h is for T190L28 Others are T126L28 20 (T190) = 70 (T126) 25
Plans for improvement NCEP/GEFS Looking for: Higher resolutions New GFS model Hind-cast at real time and reforecast (ESRL) Coupling ocean-land-atmosphere model for GEFS Considering ocean model in ESMF MOM4 Extended to 45 days Hydro-meteorological ensemble (river ensemble) Pending on operational LDAS/GLDAS Science: Optimum ETR initialization Combine EnKF initial analysis with ETR Stochastic physics to improve forecast uncertainties 26
Development Plan of Statistical Post-Processing for NAEFS Cui/Yuan s THORPEX proposal (2010-2013) Opportunities for improving the post-processor Utilization of additional input information More ensembles - high resolution control forecasts, SREF, GEFS Using reforecast information to improve week-2 and precipitation Improving analysis fields (such as RTMA and etc..) Improving calibration technique Calibration of higher moments (especially spread) Use of objective weighting in input fields combination Processing of additional variables with non-gaussian distribution Improve downscaling methods Future plan (overlook beyond 2-3 years) 27 Bias correct all model output variables (>200 which include precipitation)
Flow Chart for Hybrid Variation and Ensemble Data Assimilation System (HVEDAS) Ensemble fcst t=j-1 j ETR/EnKF t=j Ensemble fcst t=j, j+1 ETR/EnKF t=j+1 Lower resolution Two-way hybrid Estimated Background Error Covariance from Ensemble Forecast (6 hours) Replace Ensemble Mean? Estimated Background Error Covariance from Ensemble Forecast (6 hours) 3DVAR/4DVAR t=j Best Analysis? 3DVAR/4DVAR t=j+1 Higher resolution 28
NAEFS inclusion of FNMOC ensembles Yuejian Zhu & Bo Cui December 2010 29
Value-added by including FNMOC ensemble into NAEFS T2m: Against analysis (NCEP s evaluation, 4 of 4) Stat. corr. NAEFS + FNMOC 0.5 CRPS skill Raw NCEP NAEFS Raw NCEP ensemble has modest skill (3.4d) Statistically corrected NCEP ensemble has improved skill (4.8d) Combined NCEP CMC (NAEFS) show further increase in skill (6.2d) Addition of FNMOC to NAEFS leads to modest improvement (6.7d) 30
Example of score cards for ensembles evaluation Comparison for NAEFS with/without FNMOC ensembles Blue means better to have FNMOC ensemble in NAEFS, red is not 31
Software public access through subversion Available software already in NWS operation Bias correction for all near Gaussian distribution variables Downscaling for surface variables Include maximum/minimum temperature (derived variables) Include wind speed/direction (derived variables) Precipitation calibration (2004 version) New version will be implemented soon Ensemble verification packages Verification for ensemble mean (such as RMS error, bias, et al.) Verification for probabilistic forecast Advantage shared the same algorithm MSC (Meteorological Service of Canada) uses it in operation FNMOC already receive it, will use it later this year ESRL/GSD (Toth) in testing ESRL/PSD (Whitaker) use EMC s verification package OHD (DJ Seo) shared verification package Many institutions use EMC s verification package Public access through subversion The same as GFS, GSI, HWRF and et al. 32
Background!!! 33
Preliminary Conclusions and Plans Individual ensemble systems (individual Centers forecasts) NCEP and CMC have similar performance FNMOC performance similar to NCEP & FNMOC for near surface variables, including precipitation FNMOC is less skillful than NCEP and CMC for upper atmosphere variable (500hPa) Combined ensemble system (without bias correction) Multi-model ensembles have higher skill than single system Adding FNMOC ensemble to current NAEFS (NCEP+CMC) adds value for most forecast variables Noticeable improvement for surface variables Minimal improvement for upper atmosphere Combined ensemble system (with operational NAEFS bias correction) Improved near surface variables with FNMOC ensemble NCEPbc + CMCbc + FNMOCbc Less improvement for upper atmosphere (e.g. 500hPa height)) Some degradation for short lead times (related to large spread in FNMOC ensemble) Plan to NAEFS upgrade (NUOPC IOC Q1FY11) Based on score card for past season Include the variables/parameters to current NAEFS if it adds values 34
A proposed configuration (based on 16 July 2010 telecon discussion) Assume T254L42 GFS model used for 0-8 days. From days 7.5-16, T190L42. Estimated 92.19 CPUh for 16-day forecast on DOE Franklin. Say 30 years, 365 days/year, 10 members, 2x daily ~= 20.2M CPU h, which is greater than 14.5 M total allocation. Also, will need perhaps 10% of allocation for generating initial conditions, testing. Proposed configuration based on discussion: At 00Z, full 10- member forecast every day for 30 years out to 16 days. At 12Z, a T254L42 ensemble to day 8, 5 members, reforecast every other day. Total = 10.1 + 1.75M CPUh = 11.85 M CPUh. Doable. Extra CPU time may be devoted to running some 45-day reforecasts at T126 for the sake of comparison against CFSR reforecasts, with coupled ocean model (it d be nice to have a few spare cycles to do this). 35 35
NCEP SREF System (21 members) Model Membership Resolution Forecast Hours IC/IC perturbation LBC/LBC perturbation Output Frequency for pgrb files Output Frequency for bufr soundings Eta_BMJ 3 (ctl1, n1, p1) 32km 87hr (4 times, 3, 9, 15, 21z) ndas/regional BV GFS/GEFS 1hrly to 39hr, 3hrly afterward 1hrly and breakdown to sites Eta_KF 3 (ctl2, n2, p2) 32km 87hr (4 times, 3, 9, 15, 21z) ndas/regional BV GFS/GEFS 1hrly to 39hr, 3hrly afterward 1hrly and breakdown to sites RSM_SAS_ Ferrier 3 (ctl1, n1, p1) 32km 87hr (4 times, 3, 9, 15, 21z) GFS 3hr fcst/regiona l BV GFS/GEFS 1hrly to 39hr, 3hrly afterward 1hrly and breakdown to sites RSM_RAS_ Zhao 2 (n2, p2) 32km 87hr (4 times, 3, 9, 15, 21z) GFS 3hr fcst/regiona l BV GFS/GEFS 1hrly to 39hr, 3hrly afterward 1hrly and breakdown to sites NMM 5 (ctl, n1, p1, n2, p2) 32km 87hr (4 times, 3, 9, 15, 21z) GFS 3hr fcst/global ET GFS/GEFS 1hrly to 39hr, 3hrly afterward 1hrly and breakdown to sites ARW 5 (ctl, n1, p1, n2, p2) 35km 87hr (4 times, 3, 9, 15, 21z) GFS 3hr fcst/global ET GFS/GEFS 1hrly to 39hr, 3hrly afterward 1hrly and breakdown to sites 36
T2m bias New SREF Old SREF 37
38
STATS for Atlantic basin 00UTC only 2 months (August and September 2008) 39
References: December 14 2007 implementation: http://www.emc.ncep.noaa.gov/gmb/yzhu/html/imp/200711_imp.html February 23 2010 implementation: http://www.emc.ncep.noaa.gov/gmb/yzhu/html/imp/201002_imp.html Q4 FY10 implementation: http://www.emc.ncep.noaa.gov/gmb/yzhu/html/imp/201004_imp.html Zoltan and et al. 2005: http://www.emc.ncep.noaa.gov/gmb/ens/papers/toth_naefs_thorpex_montreal.pdf Cui and et al. 2006: http://www.emc.ncep.noaa.gov/gmb/ens/papers/manuscript_thorpex_bocui.pdf Zhu and Toth, 2008: http://www.emc.ncep.noaa.gov/gmb/yzhu/gif/pub/ams_zhu_2008.pdf Son and et al. 2008: http://www.emc.ncep.noaa.gov/gmb/ens/papers/npg-15-1013- 2008.pdf Cui and et al. 2010: http://www.emc.ncep.noaa.gov/gmb/yzhu/gif/pub/manscript_bocui_bias_correction_ 20100709.pdf Cui and et al. 2010 (draft for downscaling): 40