Mul$- model ensemble challenge ini$al/model uncertain$es

Transcription

1 Mul$- model ensemble challenge ini$al/model uncertain$es Yuejian Zhu Ensemble team leader Environmental Modeling Center NCEP/NWS/NOAA Acknowledgments: EMC ensemble team staffs Presenta$on for WMO/WWRP PDEP WG May

2 Highlights Introduc$on Ensemble systems review Single model ensemble Mul$- model ensemble The value of mul$- model ensemble Challenges for mul$- model ensemble applica$on Ini$al uncertainty / model uncertainty Summary and recommenda$on

3 Description of the ECMWF, MSC and NCEP systems Each ensemble member evolu$on is given by integra$ng the following equa$on e j ( T) Ini$al uncertainty Model uncertain$es where e j (0) is the ini$al condi$on, P j (e j,t) represents the model tendency component due to parameterized physical processes (model uncertainty), dp j (e j,t) represents random model errors (e.g. due to parameterized physical processes or sub- grid scale processes stochas$c perturba$on) and A j (e j,t) is the remaining tendency component (different physical parameteriza$on or mul$- model). Reference: = e0 (0) + de j(0) + [ Pj ( ej, t) + dpj ( ej, t) + Aj( ej, t)] dt Buizza, R., P. L. Houtekamer, Z. Toth, G. Pellerin, M. Wei, Y. Zhu, 2005: T t= 0 Opera&on: ECMWF- 1992; NCEP- 1992; MSC "A Comparison of the ECMWF, MSC, and NCEP Global Ensemble PredicBon Systems Monthly Weather Review, Vol. 133, MSC, NCEP and ECMWF plan to have 2 nd ensemble review in coming year

4 Mul$- model ensemble? Category one running at one opera$onal center Same ini$al control analysis Different models (dynamics, physics and parameteriza$ons) For example: NCEP Short- range ensemble forecast system (three different models with different parameteriza$ons) NCEP GEFS and ESRL FIME (research) (the same ini$al analysis, but different dynamics) CMC GEFS (different parameteriza$ons) Category two running at mul$ple opera$onal centers Different ini$al control analysis The same model (?) Different models and/or parameteriza$ons For example: NAEFS opera$onal TIGGE research

5 Mul$- model ensemble? Category one running at one opera$onal center Same ini$al control analysis Different models (dynamics, physics and parameteriza$ons) Rela$vely expensive compared to stochas$c physics Quality is ensured Category two running at mul$ple opera$onal centers Different ini$al control analysis Different models and/or parameteriza$ons Rela$vely cheaper, but it could be late (or delayed) Quality is not ensured from various upgrades

6 Successful opera&onal applica&on of mul&- model ensemble North American Ensemble Forecast System (NAEFS) Interna$onal project to produce mul$- center opera$onal ensemble products Combines global ensemble forecasts from Canada & USA with bias correc$on and downscaling Generates products for: Weather forecasters Specialized users End users Opera$onal outlet for THORPEX research using TIGGE archive

7 Con&nuous Ranked Probabilis&c Skill Scores NH 500- hpa Height 30- day running mean GEFS NAEFS 5- day forecast 10- day forecast NAEFS has been in daily opera&on since 2006

8 NCEP/GEFS raw forecast 4+ day gain from NAEFS From NAEFS final products Bias correc$on (NCEP, CMC) Dual- resolu$on (NCEP only) Combina$on of NCEP and CMC Down- scaling (NCEP, CMC) 8 8

9 NCEP/GEFS raw forecast 8+ day gain NAEFS final products From Bias correc$on (NCEP, CMC) Dual- resolu$on (NCEP only) Combina$on of NCEP and CMC Down- scaling (NCEP, CMC) 9 9

10 Challenge One: Sta&s&cally adding value from mul&- model ensemble applica&ons? Answer: yes, but depends on

11 Example one Early study, but it is s&ll valid for today Model A Model B Model C Model D What do you see? Model A comparable system Model B lower skill system Model C comparable system Model D beuer skill system

12 Example one Early study, but it is s&ll valid for today Model A A+B A+B+C A+B+C+D What do you expect? 1. Model A only 2. Model A and B combined 3. Models A, B and C combined 4. Models A, B, C and D combined Equal weight

13 Example one Early study, but it is s&ll valid for today Model D A+B+C+D Are you surprised? All models combined Model D (best) only

14 Example two Latest individual model performance of winter Model A Model B Model C Model A = Model A+C = Model A+B+C = % improvement Adding third model (B) may not add value by using equal weight!

15 Example three A+B+C (BC) - - Combine three bias corrected ensembles (equal weight) A+B+C (BC) A+B+C (BC_BMA) A+B+C (BC_RBMP) A+B+C (BC_BMA) Combine three bias corrected ensembles (BMA) A+B+C (BC_RBMP) Combine three bias corrected ensemble (RBMP - Recursive Bayesian Model Process 2 nd moment adjustment) Solid line RMS error Dash line - Spread Over- dispersion Mul$- model ensemble applica$on : 1. Reference combine three bias corrected ensemble with equal weight 2. BMA could improve 3 ensemble s mean, but spread could be over if original spread is larger 3. RBMP could keep similar BMA future, but 2 nd moment will be adjusted internally Adding third model (B) may add value by using un- equal weight! 15

16 Challenge Two: Do mul&- model ensemble applica&ons help decision making? Answer: Yes in most cases, but some&mes difficult.

17 1. NAEFS products Meteogram (many successes) 17

18 2. Surface temperature forecasts from two models (ini: ) Model B Model B Loca$on one Loca$on two 5 K Model A Make initial adjustment? Or bias correction? Model A How can we adjust this? Model B Loca$on three Loca$on four Model B Model A Has similar initial analysis Model A Difficult to understand 18

19 3. Regional MME - QPF Plumes for LaGuardia Apt, NY 3 Inches 2 Same ini&als 1 Model A average Model B average Model C average Regional ensemble system (3 models) 1. Three different solu$ons are from three models. 2. Less spread from each model? 3. How to weight each ensemble for heavy precipita$on?

20 4. Three Global Ensemble Forecast Systems Different ini$als Model A+B+C Model A Large differences among model A, B and C Model C Model B

21 4. Three Global Ensemble Forecast Systems Different ini$als Model A+B+C Model A Observa$on Model C Model B

22 Challenge Three: Do we really know the analysis (or observa&on) uncertainty and model (forecast) uncertainty? Answer: not really!

23 Time(series(of(analyses,((( Amazon(basin,(South(America( At(this(point(in(the(Amazon(basin,( the(analyses(are(quite(different( from(each(other,(even(in(yearly& means,(which(range(from(296.7k(( to(301.5k.(((ncep(is(rather(( consistently(the(coldest,(ukmo( the(warmest.((this(suggests(that( a(primary(difference(between( analyses,(especially(in(rela/vely( data5sparse(regions,(may(be(a( systema/c(bias(in(each(analysis( system.( Courtesy of Dr. Tom Hamill Approximately 10 K difference from these five analysis in summer 6(

24 Time(series(of(analyses,((( Amazon(basin,(South(America,(smoothed( America, Smoothed Both(the(warmer(UKMO( the and and(cooler(ncep(analyses( analyses stand out stand(out(here.( Courtesy of Dr. Tom Hamill Aper smoothing (or running mean), it is about 6-8 K difference from these five analysis 7( in summer

25 T2m analysis difference accumulation (out to ~ 10 days) ABSAVG=1.359 ABSAVG=0.764 ABSAVG=1.546 ABSAVG=0.818 Courtesy of Dr. Bo Cui

26 Changes in short- term forecast bias due to changes in data assimila$on system Domain: 103 W - 90 W 30 N 37 N c/o CPC. In 2011, the reforecasts changed from CFSR ini$aliza$on to GSI ini$aliza$on, which used a slightly different version of the forecast model. Courtesy of Mr. Mike Charles (CPC) 26

27 How do they perform? Ensembles must be reliable Ensemble reliability is sensi$ve to verifica$on field (analysis or observa$on) Large uncertainty from observa$ons as well fcst v analysis fcst v radiosondes fcst v AMSUA t+24h t+24h t+24h t+120h w/o stoch. phys. with stoch. phys. t+120h w/o stoch. phys. with stoch. phys. t+120h w/o stoch. phys. with stoch. phys. NH 500- hpa T w/o stoch. phys. with stoch. phys. w/o stoch. phys. with stoch. phys. w/o stoch. phys. with stoch. phys. This verifica&on indicates that the sta&s&cs are sensi&ve to verifica&on analysis/observa&on for short lead- &me, but not for 120 hours (From M Yamaguchi)

28 Ver&cal distribu&on of perturba&on amplitude NCEP hybrid DA system One case for Black EnKF first analysis Red EnKF final analysis Green EnKF 6- hr forecast 1 st genera$on of EnKF You have seen how much differences from these two assimilated analysis uncertain&es at NCEP 2 nd genera$on of EnKF

29 Latest Performance of Global Ensemble Forecast Systems NH 500hPa height RMS error (solid).vs. Spread (dash) One year sta$s$cs of three ensembles: NCEP, CMC and ECMWF

30 80% <- > 65% 10% <- > 25% Summer- Fall 2013 Six months Typical example of over- confident for precipita$on forecast e.g. when we predict 10% chance of 1+ mm, it happens 25% of the $me Precipita$on reliability for 12-36hr and greater than 1mm/day

31 Improvement of ensemble spread from introducing stochas&c physics % diff from spread:error ra&o Without stochas&c With stochas&c

32 Summary There are various MME systems in our daily opera$onal applica$on. Skill of mul$- model ensemble There is an added value from combining models of similar skill. Combined two models could be best, addi$onal value should be reduced from third model. Model performance will be degraded from adding less skillful models. Applica$ons of mul$- model ensemble Provide reliable forecasts and skillful probabilis$c forecasts. Many challenges for sensible elements, such as surface temperature forecast, and precipita$on forecast. Very hard to deal with imperfect systems large analysis uncertainty and model uncertainty

33 Recommenda$ons Encourage scien$sts to con$nue the inves$ga$ons of mul$- model ensemble applica$ons through THORPEX legacy projects and the TIGGE archive Link to HIW, PPP and S2S projects Es$mate analysis uncertainty and model uncertainty quan$ta$vely for mul$- system ensembles Link to DAOS WG Es$mate analysis bias and model bias quan$ta$vely for mul$- system ensemble Link to DAOS WG and WGNE WG Develop and improve ensemble post- processing to enhance forecast reliability and forecast skills Link to NAEFS and other projects Develop and improve mul$- model ensemble based probabilis$c forecast products (or guidance) Link to NAEFS/NUOPC and other projects Train/educate users to understand analysis uncertainty and forecast uncertainty

34 References: Buizza, R., P. L. Houtekamer, Z. Toth, G. Pellerin, M. Wei, Y. Zhu, 2005:"A Comparison of the ECMWF, MSC, and NCEP Global Ensemble PredicBon Systems Monthly Weather Review, Vol. 133, Swinbank, R., et al, 2015: The TIGGE Project and its Achievements BAMS TIGGE reference: htp://$gge.ecmwf.int/references.html NAEFS reference: htp:// ens/naefs.html NUOPC reference: htp:// yzhu/nuopc/nuopc.html

35 Background!!!

36 Real example single model ensemble Propaga$on of small ini$al uncertain$es

37 00UTC Bimodality? Thick blue: ensemble mean Opr: T254L42 (55km) (8 days) Para: T574L64 (33km) Red arrow means good forecast 06UTC