Verification of the operational NWP models at DWD - with special focus at COSMO-EU. Ulrich Damrath

Transcription

1 Verification of the operational NWP models at DWD - with special focus at COSMO-EU Ulrich Damrath Ulrich.Damrath@dwd.de

2 Ein Mensch erkennt (und das ist wichtig): Nichts ist ganz falsch und nichts ganz richtig Eugen Roth

3 A man discerns - perhaps you too: Nothing is entirely wrong and nothing entirely true Eugen Roth

4 Basic tasks of verification 4Comparison of forecast results with the truth (observation and analysis) 4Calculation of forecast quality in terms of so called scores 4Feedback to endusers and modellers

5 Outline: A mixture of 4General remarks 4The verification system at DWD 4Examples of verification results 4How did verification help (in some cases)?

6 Purposes of verification 4Watching the operational system 4Assessment of new model versions 4Evaluation of predictability 4Interpretation of model errors

7 Course of verification on principal Forecast Modeller Analysis Verification Enduser Data control Observation Data control

8 Methods of verification 4Subjective assessment 4Objective assessment 4Special investigations

9 Forecast of precipitation

10 Observation ( UTC UTC) and forecast of precipitation (T+18-T+42 from UTC

11 Observation ( UTC UTC) and forecast of precipitation (T+06-T+30 from UTC

12 Objective assessment calculation of scores which characterize the truth of the forecast Advantage: 8objective criteria for all forecasts Disadvantage: 8Perhaps only some aspects of the forecasts are catched. 8Impossibility to express all degrees of freedom of a forecast in some simple numbers

13 Methods of objective verification 4calculation of scores for continuous weather elements 4calculation of scores for categorical weather elements 4calculation of scores for probabilistic weather elements (a short information only) 4calculation of scores that are representative for very high resolution models

14 Scores for continuous weather elements 8Mean error 8Mean absolute error 8Root mean square error 8Standard deviation 8Correlation coefficient 8Skill scores

15 A basic rule Try to interpret each score dependent on its characteristics as - a physical and - a statistical term!

16 Scores for continuous weather elements - mean error or BIAS BIAS = F i ME = 1 N forecast N ( F O i i= 1 Properties: ideal value: 0 no general information about forecast quality If errors have a Gaussian distribution: Estimation of the expected value of errors i ) O i observation

17 Time series of BIAS of 2m-temperatures over Germany (January 1994 till May 2007)

18 Time series of BIAS of 2m-temperatures over Germany (January 1994 till May 2007)

19 Time series of BIAS of wind speed 10m over Germany (May May 2007)

20 Time series of BIAS of wind speed 10m over Germany stations below 100 m (May May 2007)

21 Monthly mean values of surface weather elements for station January 2008

22 Monthly mean values of surface weather elements for station January 2008

23 Monthly mean values of surface weather elements for station (cases with cloud cover observed and forecasted >4/8)

24 Example for conditional verification Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 (RMSE and STDV) Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 observed and forecasted values lower than 1020 hpa (RMSE and STDV) Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 observed and forecasted values higher than 1020 hpa (RMSE and STDV)

25 Distribution of surface level pressure in COSMO-EU and in GME

26 Distribution of surface level pressure in COSMO-EU and in GME

27 Distribution of surface level pressure in COSMO-EU and in GME

28 Distribution of surface level pressure in COSMO-EU and in GME

29 Distribution of surface level pressure in COSMO-EU and in GME

30 Verification of surface weather elements: The effect of the SSO-scheme in COSMO-EU Consideration of frictional and gravity effects caused by processes that are not resolved by model orography

31 Distribution of surface level pressure in COSMO-EU and in GME

32 Distribution of surface level pressure in COSMO-EU and in GME

33 Distribution of surface level pressure in COSMO-EU and in GME

34 Distribution of surface level pressure in COSMO-EU and in GME

35 Scores for continuous weather elements - mean absolute error MAE = 1 N N i= 1 F i O i Properties: ideal value: 0 errors are simply added

36 Scores for continuous weather elements - root mean square error RMSE = N 1 ( F i N i= 1 O i ) 2 Properties: ideal value: 0 sensitive concerning phase and amplitude errors

37 Verification of surface weather elements: The effect of the SSO-scheme in COSMO-EU (Spring)

38 Verification of surface weather elements: The effect of the SSO-scheme in COSMO-EU(Autumn)

39 Verification of surface weather elements: Modification of T2m-diagnosis in GME, COSMO-EU and COSMO-DE z P + z 0 z A + z 0 0 z 0 l = l κ logarithmic freeturbulent-layer profile φ A expon. roughness-layer profile 0 φ S φ 2 φ δ φ m h 2m h z P + z d z A + z d z d 0 l d l = d κ unstable stable φ 2m φ S φ 20m 10m z P z A h 2m 0 z φ upper boundary of the lowest model layer lower boundary of the lowest model layer lowest model main level

40 Verification of surface weather elements: Modification of T2m-diagnosis in GME, COSMO-EU and COSMO-DE Z 0 taken from model surface Z 0 taken as 2 cm

41 Verification of surface weather elements: Modification of T2m-diagnosis in GME, COSMO-EU and COSMO-DE Z 0 taken from model surface Z 0 taken as 2 cm

42 Verification of surface weather elements: Modification of T2m-diagnosis in COSMO-EU (Winter)

43 Verification of surface weather elements: Modification of T2m-diagnosis in COSMO-EU (Winter)

44 Verification of surface weather elements: Modification of T2m-diagnosis in COSMO-EU (Summer)

45 Verification of surface weather elements: Modification of T2m-diagnosis in COSMO-EU (Sommer)

46 Diurnal cycle of forecasted and observed T2m- COSMO-EU, Summer 2007, Start 00 UTC

47 Diurnal cycle of forecasted and observed T2m - COSMO-EU, Summer 2008, Start 00 UTC

48 Diurnal cycle of forecasted and observed precipitation - COSMO-EU, Summer 2007(top) and 2008(bottom), Start 00 UTC!!!!!

49 Bias T2m- GME/LM/COSMO-EU, since 1999, Stations below 100 m, Start 00 UTC

50 Bias T2m- GME/LM/COSMO-EU, since 1999, Stations below 100 m, Start 00 UTC

51 Scores for continuous weather elements - standard deviation STDV = N 1 ( F i Oi ( F N i= 1 O)) Properties: ideal value: 0 If errors have a Gaussian distribution: around 68% of errors can be expected in the interval: < BIAS-STDV, BIAS+STDV > 2

52 Scores for continuous weather elements - correlation coefficient R : reference forecast short range : persistence medium range: climate CC = Properties: range: -1 till +1 (ideal value) mainly sensitive concerning phase errors critical value: 0.6 ( F ( F R R ( F ( F R)) R)) 2 ( A ( A R R ( A ( A R)) R)) 2

53

54

55 Scores for continuous weater elements Examples of scores for four artificial situations:

56 Scores for continuous weater elements Examples of scores for four artificial situations:

57 Scores for continuous weater elements Examples of scores for four artificial situations:

58 Scores for continuous weater elements Examples of scores for four artificial situations:

59 Scores for continuous weather elements - Skill Score S1 Properties: range: 0 (ideal value) till 200 (worst value)

60 Scores for continuous weather elements - skill score - relates a score of interest to a score of a reference forecast SKSC = SCORE( forecast) SCORE( ideal) SCORE( reference) SCORE( reference) Properties: range: - till +1 (ideal value) example: reduction of variance: sometimes given in percent RV = 1 RMSE RMSE 2 forecast 2 reference

61 Scores for categorical weather elements aim: to express the correctness of forecasts for yes/no events for example: fog, thunderstorm, cloudiness, precipitation (over a limit of interest), temperature: warming or cooling,... Method: stratification of forecasts build up contingency tables

62 Scores for categorical weather elements General structure of contingency tables: Observation yes Observation no Forecast yes A B Forecast no C D

63 Scores for categorical weather elements Probability of detection: POD OY ON POD = A A + C FY A B FN C D range : 0-1 perfect : 1 maximise : overforecast event disadvantage: only yes-forecasts are considered

64 Scores for categorical weather elements False alarm rate: FAR FAR = B A+ B OY ON FY A B FN C D range : 0-1 perfect : 0 maximise : underforecast event disadvantage: only yes-forecasts are considered

65 Scores for categorical weather elements Percentage correct: PEC A+ D = A+ B+ C+ D A D N PEC = + range : 0-1 perfect : 1 maximise : do everything correct disadvantage: for rare events high correctness is suggested OY ON FY A B FN C D

66 Scores for categorical weather elements Frequency bias: FBI FBI = A+ B A+ C OY ON FY A B FN C D range : 0 - perfect : 1 maximise : do everything correct disadvantage: no account to accuracy

67 Scores for categorical weather elements Threat score (critical success index): CSI CSI = A A+ B+ C range : 0-1 perfect : 1 advantage : false alarms and missed events are included disadvantage: Depends on climatological frequency of events (poorer scores for rare events) since some hits can occur purely due to random chance. OY ON FY A B FN C D

68 Scores for categorical weather elements Equitable threat score : ETS FY A B relates CSI to a reference forecast FN C D = A E ( A+ B) ( A+ B+ C E E( chance) = A+ B+ C range : -1/3-1 perfect : 1 advantage : false alarms and missed events are included, effect of a reference forecasts is included ETS OY ON A+ C) + D

69 Scores for categorical weather elements Heidke skill score : HSS OY relates PEC to a reference forecast = A+ D R A+ B+ C+ D R HSS R( chance) range : perfect : 1 advantage : false alarms and missed events are included, effect of a reference forecasts is included disadvantage: very sensitive to dry events = ( A+ B) ( A+ C) + ( C+ D) ( B+ D) A+ B+ C+ D ON FY A B FN C D

70 Scores for categorical weather elements True skill statistics (Hansen-Kuiper discriminant) : TSS TSS = A D B C ( A+ C) ( B+ D) range : perfect : 1 advantage : equal emphasis to yes/noevents disadvantage: for rare events similar to POD OY ON FY A B FN C D

71 Gütemaße für kategorische Wetterelemente Beispiele für Gütemaße für zwei fiktive Situationen: B J B N B J B N V J 1 7 V N V J V N PEC: 90% POD: 25% FAR: 88% HSS: 12% PEC: 80% POD: 71% FAR: 29% HSS: 56%

72 Scores for categorical weather elements: GME-grid 2007

73 Scores for categorical weather elements COSMO-EU-grid 2007

74 A global score for COSMO models: COSI - a mixture of continuous and categorical scores rmse = 1 n ( ) t f t o n 2 Continuous elements rmsvwe = 1 n Vf V n o 2 Wind SS = 1 r r 2 f 2 p Skill score for wind and continuous elements related to persistence ETS = R "chance" T "chance" Skill score for categorical elements related to chance

75 Final definition of the score: 1 Continuous elements, sum over 8 time steps: (start: vv=3, end: vv=24, step: 3) Equal weights for all forecast times! SK SK j j 8 1 = sk 8 : i= 1 Skill i, j Score for element j sk i, j : Skill Score for element j at time step i

76 SK sk Final definition of the score: 2 Categorical elements, sum over N time steps: (start: vv=vvb, end: vv=vve, step: vvs) Cloud cover: vvb=3, vve=24 vvs=3 Precipitation: vvb=6, vve=24 vvs=6 SK Equal weights for all forecast times and categories! j j i, j, m M N Skill Skill M l= 1 N = : : 1 * i= 1 sk Score Score in category m i, j, m for element for element j j at time step i

77 Final definition of the score: 3 Sum over all elements: Equal weights for all elements! SK = 1 N E N E j= 1 SK j

78 The Score COSI : All days COSMO-EU V3.19 V 3.22 T 2m SSO

79 The Score COSI : All days

80 Long term trends of verification results for surface weather elements: Wind 00 UTC

81 Long term trends of verification results for surface weather elements: Wind 12 UTC

82 Long term trends of verification results for surface weather elements: Temperature 2m 12 UTC

83 Long term trends of verification results for surface weather elements: Minimumtemperature 2m 06 UTC

84 Long term trends of verification results for surface weather elements: Maximumtemperature 2m 18 UTC

85 Long term trends of verification results for surface weather elements: Temperature 2m 18 UTC

86 Single components of the COSI: Day 1 Trend: + Trend: + Trend: + Trend:? Trend:?

87 Verification of precipitation area mean

88 Scores for probabilistic weather elements: The Brier score BRSC = N 1 ( F i N i= 1 O i F i : forecasted probability O i : observed probability 0: if event was not observed 1: if event was observed Properties: ideal value: 0 maximum: 1 ) 2

89 Scores for probabilistic weather elements: The Brier skill score The reliability diagram The ROC curve The Talagrand diagram

90 Verification of very high resolution models - basic problem: The double penalty hhigh resolution models often give right signals hbut 8 forecasts may be a little bit shifted compared to observations htherefore 8 forecasts are penalized at those points where the event was predicted 8 forecasts are penalized at those points where the event was not predicted 8 The right signal is not recompensed!

91 Verification of very high resolution models - basic problem: Solution look into windows with different sizes calculate a set of scores that represent the content of the windows

92 Fraction of pixels with rain accumulation 4 mm in 1x1 km windows (left) and in 35x35 km windows (right) Fuzzy Verification of High Resolution Gridded Forecasts: A Review and Proposed Framework Elizabeth E. Ebert Bureau of Meteorology Research Centre Submitted to Meteorological Applications FSS 1 N N ( P P fcst obs i = 1 = 1 N N P + fcst N i = 1 N i = 1 ) P 2 2 obs

93 GME COSMO-EU COSMO-DE Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time hours

94 The verification system at DWD verification against analysis verification against surface observations verification of vertical structures using different observation systems

95 Verification against analysis verification of geopotential, temperature, wind and humidity at all pressure levels calculation of ME, RMSE, correlation coefficients, Skill-Score S1,... calculation of scores for 26 regions calculation of horizontal distribution of errors

96 Verification against surface weather elements 8Verification worldwide (~3300 stations) 8 cloud cover(s) and 6h-precipitation (contingency tables), gusts 8 wind, temperature, dew point(depression), extreme temperatures 8Verification over Germany (~205 stations) 8 cloud cover(s) and 1h-precipitation (contingency tables), gusts 8 wind, temperature, dew point(depression), extreme temperatures

97 Verification of vertical structures 4Geopotential, wind, temperature, humidity 4TEMPs, AMDARs, wind profiler 4GME, COSMO-EU, COSMO-DE 4single TEMPs, single wind profiler,... 4Wait perhaps a moment for the next slide

98 BIAS 00 H 24 H 48 H Geopotential RMSE

99 BIAS 00 H 24 H 48 H Geopotential GME GMEPseudoTemps PseudoTemps RMSE

100 BIAS 00 H 24 H 48 H Geopotential GME GME40km 40km RMSE

101 BIAS 00 H 24 H 48 H Geopotential COSMO-EU COSMO-EUEinführung Einführung RMSE

102 BIAS 00 H 24 H 48 H Geopotential Einschalten Einschaltendes dessso-schemas SSO-Schemas RMSE

103 BIAS 00 H 24 H 48 H Geopotential RMSE Runge-Kutta-Kern/Änderung Runge-Kutta-Kern/Änderungder derreferenzatmosphäre Referenzatmosphäre

104 BIAS Windspeed RMSE 00 H 24 H 48 H

105 BIAS Windspeed GME GME PseudoTemps PseudoTemps RMSE 00 H 24 H 48 H

106 BIAS Windspeed GME GME 40km 40km RMSE 00 H 24 H 48 H

107 BIAS Windspeed COSMO-EU COSMO-EU Einführung Einführung RMSE 00 H 24 H 48 H

108 BIAS Windspeed Einschalten Einschalten des des SSO-Schemas SSO-Schemas RMSE 00 H 24 H 48 H

109 BIAS Windspeed RMSE Runge-Kutta-Kern/Änderung der der Referenzatmosphäre Referenzatmosphäre 00 H 24 H 48 H

110 BIAS Temperature RMSE 00 H 24 H 48 H

111 BIAS Temperature GME GME PseudoTemps PseudoTemps RMSE 00 H 24 H 48 H

112 BIAS Temperature GME GME 40km 40km RMSE 00 H 24 H 48 H

113 BIAS Temperature COSMO-EU COSMO-EU Einführung Einführung RMSE 00 H 24 H 48 H

114 BIAS Temperature Einschalten Einschalten des des SSO-Schemas SSO-Schemas RMSE 00 H 24 H 48 H

115 BIAS Temperature RMSE Runge-Kutta-Kern/Änderung der der Referenzatmosphäre Referenzatmosphäre 00 H 24 H 48 H

116 BIAS Temperature

117 BIAS Relative Humidity RMSE 00 H 24 H 48 H

118 BIAS Relative Humidity Prognostisches Prognostisches Wolkeneis Wolkeneis RMSE 00 H 24 H 48 H

119 BIAS Relative Humidity COSMO-EU COSMO-EU Einführung Einführung RMSE 00 H 24 H 48 H

120 BIAS Relative Humidity Verbesserte Verbesserte Qualitätskontrolle Qualitätskontrolle der der Radiosondenfeuchten Radiosondenfeuchten RMSE 00 H 24 H 48 H

121 BIAS Relative Humidity RMSE Änderung Änderung seitliche seitliche RB RB für für Regen/Schneewassergehalt 00 H 24 H 48 H

122 Vertical profiles of forecast errors according to AMDAR measurements bias of wind speed Forecasts starting 00 UTC since 2000 positive bias negative bias Fc-time:00 h Fc-time:24 h Fc-time:48 h

123 Vertical profiles of forecast errors according to TEMP measurements bias of wind speed Forecasts starting 00 UTC since 2000 positive bias negative bias Fc-time:00 h Fc-time:12 h Fc-time:24 h Fc-time:36 h Fc-time:48 h

124 Number of scores for each model run (crude approximation) 8Verifcation against surface observations: ~ scores 8Verification against analysis: ~ scores 8Verification against upper air observations: ~ scores 8Calculate the scores AND watch the system! 8Verification is a full time job for at least 2 persons!

125 A typical situation

126 First possible solution of the problem: t-test,... Hypothesistests: Are average values of two different time series significantly different? Significancetests for the parameters of these time series - t Test weak t test t = t test N X µ S N: Number of cases X: sample mean value S: sample standard deviation µ: hypothetial mean value t S = 2 D = x1 x S D ( N ) S1 + ( N2 1) S N + N 2 1 N1N2 N + N

127 Second possible solution of the problem: bootstrapping Sample with 10 elements Realisation 1: mean value using elements: Realisation 2: mean value using elements: Realisation 3: mean value using elements: Realisation 4: mean value using elements: Realisation 5: mean value using elements: Realisation 6: mean value using elements: Realisation 7: mean value using elements: Realisation 8: mean value using elements: Realisation 9: mean value using elements: Realisation 10: mean value using elements:

128 A typical situation

129 Examination of statistical significance of fuzzy -verification results using bootstrapping Basic idea of bootstrapping: Repeat a resampling all elements of a given in a sample of forecasts and observations as often as necessary (N times) and calculate the relevant score(s) Calculate from N scores statistical properties of the sample such as mean value standard deviation, confidence intervals and quantiles Application to fuzzy -verification Resampling is done using blocks. Blocks are defined as single days. Number of resampling cases: N=Days*100 Calculation scores from N samples for NT thesholds and NW windows Calculation of quantiles for each window and threshold

130 Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-DE, period June -August 2009 Germany

131 Differences between GME and COSMO-DE ETS(COSMO-DE) - ETS(GME) Significance test Germany COSMO-DE better than GME COSMO-DE worse than GME

132 Differences between COSMO-DE and COSMO-EU ETS(COSMO-DE) - ETS(COSMO-EU) Significance test Germany COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU

133 Differences between COSMO-DE and COSMO-EU ETS(COSMO-DE) - ETS(COSMO-EU) Significance test COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU

134 One of open questions: Surprisingly bad forecasts for temperature on some days during winter

135 One of open questions: Surprisingly bad forecasts for temperature on some days during winter

136 Weather situation from UTC

137 Verificationmeteogram from UTC

138 Verificationmeteogram from UTC

139 Verificationmeteogram vom UTC

140 Verificationmeteogram vom UTC

141