Analog Ensemble for Probabilis1c Weather Predic1ons

Transcription

1 Analog Ensemble for Probabilis1c Weather Predic1ons Luca Delle Monache, F. Anthony Eckel * Daran Rife, Badrinath Nagarajan, and Keith Searight Research ApplicaCons Laboratory, NCAR - - Boulder, CO * Office of Science and Technology, NWS/NOAA - - Silver Spring, Maryland, USA Acknowledgments DATA PROVIDER: Mar1n Charron and Ronald FreneCe of Environment Canada SPONSORS: Na1onal Weather Service Office of Science and Technology (NWS/OST), Defense Threat Reduc1on Agency (DTRA), U.S Army Test and Evalua1on Command (ATEC)

2 Analog Ensemble (AnEn) Mon Tue Wed Thu Fri Sat Sun t = 0 Time Analog search as in Delle Monache et al. (MWR 2011) 2

3 Analog Ensemble (AnEn) PRED Mon Tue Wed Thu Fri Sat Sun t = 0 Time Analog search as in Delle Monache et al. (MWR 2011) 3

4 Analog Ensemble (AnEn) PRED OBS Mon Tue Wed Thu Fri Sat Sun t = 0 Time Analog search as in Delle Monache et al. (MWR 2011) 4

5 Analog Ensemble (AnEn) PRED OBS Mon Tue Wed Thu Fri Sat Sun t = 0 Time Analog search as in Delle Monache et al. (MWR 2011) 5

6 Analog Ensemble (AnEn) PRED OBS Mon Tue Wed Thu Fri Sat Sun t = 0 Time Analog search as in Delle Monache et al. (MWR 2011) 6

7 Analog Ensemble (AnEn) PRED OBS Mon Tue Wed Thu Fri Sat Sun t = 0 Time Wed farthest analog Fri Sat Tue Sun Mon Thu closest analog Analog Space Analog search as in Delle Monache et al. (MWR 2011) 7

8 Analog Ensemble (AnEn) PRED OBS Mon Tue Wed Thu Fri Sat Sun t = 0 Time Wed farthest analog Fri Sat Tue Sun Mon Thu closest analog Analog Space Analog search as in Delle Monache et al. (MWR 2011) 8

9 Analog Ensemble (AnEn) PRED OBS Mon Tue Wed Thu Fri Sat Sun t = 0 Time 2- member AnEn Wed farthest analog Fri Sat Tue Sun Mon Thu closest analog Analog Space Analog search as in Delle Monache et al. (MWR 2011) 9

10 How skillful is AnEn? AnEn generated with Environment Canada GEM (15 km), 0-48 hours Comparison with Environment Canada Regional Ensemble Prediction System (REPS, next slide) Period of 15 months (verification over the last 3 months) 10-m wind speed, 2-m temperature 550 surface stations over CONUS (in two slides) Probabilistic prediction attributes: reliability & sharpness, statistical consistency, utility/value 10

11 Regional Ensemble PredicCon System (REPS) Model: GEM (vertical staggering) 20 members + 1 control run 72 hours forecast lead time Resolution: ~33 km with 28 levels Initial conditions (i.e., cold start) and 3-hourly boundary condition updates from GEPS (EnKF + multi-physics) Physics: o o o Kain et Fritsch (1993) for deep convection Li et Barker (2005) for the radiation ISBA scheme (Noilhan et Planton, 1989) for surface Stochastic Physics: Markov Chains on physical tendencies 11

12 Ground truth dataset 550 hourly METAR Surface Observations 1 May July 2011, for a total of 457 days 10-m wind speed 12

13 Analysis of reliability and sharpness Reliability diagram: 10-m wind speed > 5 m s -1, 9-h fcst (a) REPS raw (b) REPS cal * Observed Relative Frequency, Forecast Frequency (d) (c) AnEn PsEn * Eckel et al. (Weather and Forecasting 2012) (d) AnEn Forecast Probability

14 Analysis of StaCsCcal Consistency Rank Histogram, 9- h forecast REPS raw REPS cal AnEn 14

15 Analysis of spread- error consistency (1) Dispersion diagram for 10-m wind speed REPS raw REPS cal AnEn 15

16 Analysis of ResoluCon RelaCve OperaCng CharacterisCcs skill score, 10- m wind speed 5, 10 m s - 1 WSPD > 5 m s - 1 WSPD > 10 m s AnEn REPScal REPSraw PsEn (a) 0.9 (b) ROCSS Forecast Lead Time (hours) Forecast Lead Time (hours) PsEn: Persistence analog Ensemble ; the 21 members are past observations from the most recent week and within ± 1 week centered on the date of the current forecast from the previous year 16

17 Analysis of ResoluCon, sensicvity RelaCve OperaCng CharacterisCcs skill score, 10- m wind speed 5 m s AnEn REPScal AnEn33 AnEnshort AnEn33: generated from REPS member n. 20 at 33-km, rather than from GEM deterministic at 15-km as in AnEn ROCSS AnEnshort: generated with a training period 9-month long, rather than 15 months as in AnEn Forecast Lead Time (hours) 17

18 AnEn vs. NWP ensemble (point prediccons of 10- and 80- m speed, 2- m T, wind power) AnEn a_empts to sample directly from the true forecast PDF avoiding the challenges of simulacng model uncertainty, and provides a naturally calibrated ensemble Resources required to run any n- member NWP ensemble could be put toward producing a single NWP at a much higher resolucon o AnEn (out of the GEM 15- km) costs computaconally about half of REPS 21 members at 33- km AnEn algorithm is intrinsically parallel, as the analogs can be searched at the same Cme for every locacon (or grid point) and forecast lead Cme 18

19 Summary and future work Analog- ensemble similar skill to the calibrated REPS however, is much cheaper/simpler!!!! Could this be a new paradigm to generate probabiliscc prediccons for certain applicacons? Current/Future work: o Explore new predictors to search for the best analogs (e.g., for 10- m wind speed u *, surface fluxes, etc.) o Test other variables at the surface and with upper- air data o Tests with both obs and analysis data as ground truth o Port the analog code on GPU o Tests with mulc- year training data set o OpCmizaCon of current algorithm o Hybrid An/NWP ensemble? Tony Eckel s Poster: op1miza1on of AnEn, and Hybrid An/NWP ensemble 19

20 Thanks! 20

21 Analog strength for a parccular forecast lead Cme t is measured by the distance ~ ~ between current and past forecast, over a short window, to d t = f t g t = 1 σ f The Analog Ensemble (AnEn) ~ t + k = ~ t ( f g ) t+ k t+ k 2 t t t + t σ f : Forecasts standard deviacon over encre analog training period Expanded to mulcple predictor variables, but scll focused on predictand f: (for wind speed, predictors are speed, direccon, sfc. temp., and PBL depth) d t = f t g t = N v ( ) v v v ft+ k gt+ k v= 1 w σ f v ~ + t k= ~ t 2 N v : Number of predictor variables w v : Weight given to each predictor Wind Speed Past Forecast, g t t-1 t+1 Current Forecast, f t-1 t t h h

22 The Analog Ensemble (AnEn) Ajer finding the n strongest analogs, each of the n AnEn members is taken as the verifying observacon from each analog. Wind Speed Past Forecast, g t t-1 t+1 observation h Current Forecast, f t h t t+1 AnEn member #7 AnEn Specifics: Model: Regional 15-km GEM, (~15km), 40? levels Fcst Length : 48 h Fcst Cycle: 12Z only Fcst Cases: 23 Apr July 2011 (last 100 days in the dataset) Ensemble Members: unlimited -- used 21 to match REPS

23 Cost- benefit of the analog technique (1) Design, implementacon, and maintenance of the analog and NWP ensemble techniques - Shared requirements NWP- model- based data assimilacon and forecast. CalibraCon: both approaches use a calibracon technique, and each requires about the same effort to develop and implement - Unique requirements for REPS MulCple physics packages (for mulcmodel ensembles), and StochasCc physics roucnes 23

24 Cost- benefit of the analog technique (2) ComputaConal expense - SCENARIO I: You must run your own NWP model REPS requires about 3 Cmes more calculacons than the analog technique - SCENARIO II: Use an available NWP product (e.g., from NCEP) REPS requires orders of magnitude more calculacons than the analog technique 24

25 ProbabilisCc forecast a_ributes: Reliability Example: An event (e.g., wind speed > 5 m/s) is predicted to happen with a 30% probability We collect the observations that verified every time we made the prediction in 1 If the frequency of the event in the observation collected is 30%, then the forecast is perfectly RELIABLE A reliable system generates probabilistic predictions with low bias, i.e., in the predicted probabilities systematic errors are low

26 Probabilistic forecast attributes: Economic value (value score) Potential value of a forecast in a decision making framework; it can be estimated using a static cost-loss decision model for a dichotomous event (Wilks, 2006). A decision maker can chose to pay a cost C (e.g., cost of evacuation efforts) to protect against a possible loss L (with L > C): if protective action is not taken, than the decision maker incurs a loss L if the adverse event incurs (e.g., lost lives).

27 ProbabilisCc forecast a_ributes: Spread- error consistency The ensemble spread tell us how uncertain a forecast is. Ideally, large spread should be associate with larger uncertainties, low spread should indicate higher accuracy If an ensemble is perfect, than the observations are indistinguishable from the ensemble members If the point above is true, then the RMSE of the ensemble mean, should be equal the ensemble spread

28 Analysis of StaCsCcal Consistency Rank Histogram, 9- h forecast REPS raw REPS cal PsEn MRE = Missing Rate Error VOP = Verification Outlier Percentage AnEn 28 28

29 Analysis of spread- error consistency (2) Binned spread- skill diagram, 10- m wind speed, 42- h fcst REPS raw REPS cal Standard Deviation of Ensemble Mean Error (m s -1 ) Ensemble Spread (m s - 1 ) AnEn (GEM 15-km) AnEn (REPSmem km) Ensemble Spread (m s - 1 ) Ensemble Spread (m s - 1 ) 29

30 Analysis of Value Economic value diagram, 10- m wind speed 5 m/s WSPD > 5 m s - 1, 42-h forecast (a) AnEn REPScal m T < 15 C, 42-h forecast (b) Value Score REPSraw PsEn User C/L Ratio User C/L Ratio 30

31 Standard Deviation of Ensemble Mean Error Over Verification Points in Bin (m/s) Analysis of spread- error consistency (2) Binned spread- skill diagram, 10- m wind speed, 42- h fcst REPS raw AnEn REPS cal 31

32 Analysis of ResoluCon, Value ROCSS, economic value diagram, 10- m wind speed 5 m/s ROC Skill Score Economic Value Diagram 9-h Forecast * * Economic Value Diagram 42-h Forecast * * Eckel et al. (Weather and Forecasting 2011) 32