m Detective Miss Marple in Murder at the Gallop
Slide m Now, this is how we verify the forecasters at DWD: we watch them intensily and when they get it wrong, they'll get a bang over their head. OK, I'm slightly exaggerating and I will explain later what Detective Miss Marple has to do with warning verification. mgoeber, 3//27
Comparing peaches and apples On the accuracy of gust warnings issued by forecasters and the accuracy of the model guidance Martin Göber Department Weather Forecasting Deutscher Wetterdienst Acknowledgements: R. Kirchner, T. Kratzsch, D. Richardson, G. Schweigert, S. Tremmel 2
3
Heidke skill score forecaster Local model p,9,8,7,6,5,4,3,2, near gale gale storm violent storm hurrican force 4
hit rate forecaster Local model p,9,8,7,6,5,4,3,2, near gale gale storm violent storm hurrican force 5
false alarm ratio forecaster Local model p,9,8,7,6,5,4,3,2, near gale gale storm violent storm hurrican force 6
Frequency bias forecaster Local model 9 8 7 6 p 5 4 3 2 near gale gale storm violent storm hurrican force 7
relative value for C/L=, forecaster Local model,8,6,4 p,2 -,2 near gale gale storm violent storm hurrican -,4 8
relative value for C/L=, forecaster Local model,8,6,4 p,2 -,2 near gale gale storm violent storm hurrican -,4 9
Signal Detection Theory
35 3 25 all all cases frequency 2 5 5 2 3 4 5 6 7 8 9 indicator (=CAPE, wind speed, finger prints,...) all cases
35 3 event no event all cases 25 frequency 2 5 5 event all cases 2 3 4 5 6 7 8 9 indicator 2
frequency 35 3 25 2 5 5 event no event all cases False alarms POD=7% FAR=5% ETS=5% Bias=8% 2 3 4 5 6 7 8 9 misses indicator threshold event all cases 3
frequency 35 3 25 2 5 5 event no event all cases False alarms POD=9% FAR=4% ETS=42% Bias=5% 2 3 4 5 6 7 8 9 misses indicator threshold event all cases 4
m2 Detective Miss Marple in Murder at the Gallop 5
Slide 5 m2 Now, this is how we verify the forecasters at DWD: we watch them intensily and when they get it wrong, they'll get a bang over their head. OK, I'm slightly exaggerating and I will explain later what Detective Miss Marple has to do with warning verification. mgoeber, 3//27
Violent storm warning: gusts>= 29 m/s n(f O<29) n(f O>=29) frequency 3 5 7 9 3 5 7 9 2 23 25 27 29 3 33 35 37 39 m/s 6
Violent storm warning: gusts>= 29 m/s correct NO missed false alarm hit frequency 3 5 7 9 3 5 7 9 2 23 25 27 29 3 33 35 37 39 m/s model at face value 7
Violent storm warning: gusts>= 29 m/s correct NO missed false alarm hit frequency 3 5 7 9 3 5 7 9 2 23 25 27 29 3 33 35 37 39 m/s 8
Violent storm warning: gusts>= 29 m/s 3 hit rate 25 false alarm ratio Bias 2 Heidke skill score % 5 hit rate>9% Bias free 5 3 5 7 9 2 23 25 27 29 3 33 35 37 threshold in m/s 9
Re-labeling model bias = forecaster bias model in m/s ----> model gust interpretation for warnings 3 ----> 4 (near gale) 6 ----> 8 (gale) 22 ----> 25 (storm) 25 ----> 29 (violent storm) 3 ----> 33 (hurricane force) Verification of heavily biased model? Quite similar to forecaster! 2
Relative Operating Characteristics (ROC),9,8,7 forecaster smaller model values= overforecasting H,6,5,4,3,2, larger model values= underforecasting Model at face value model: near gale (>4m/s) forecaster: near gale (>4m/s) no skill,,2,4,6,8, F 2
forecaster,9,8,7 overforecasting H,6,5,4,3,2, underforecasting Face value,,2,4,6,8, F model: violent storm forecaster: violent storm no skill model: near gale (>4m/s) forecaster: near gale 22
H.9.8.7.6.5.4.3.2...2.4.6.8. F near gale gale storm violent storm hurrican force forecaster: gale forecaster: storm forecaster: violent storm forecaster: hurrican force no skill 23
My conclusions End user forecast verification: face value (incl. space-time point) Guidance verification: measure potential of the guidance using Fuzzy, Object,..., Signal detection Theory (ROC) 24