Verification of the operational NWP models at DWD - with special focus at COSMO-EU

Transcription

1 Verification of the operational NWP models at DWD - with special focus at COSMO-EU Ulrich Damrath Ulrich.Damrath@dwd.de

2 Ein Mensch erkennt (und das ist wichtig): Nichts ist ganz falsch und nichts ganz richtig Eugen Roth

3 A man discerns - perhaps you too: Nothing is entirely wrong and nothing entirely true Eugen Roth

4 Basic tasks of verification 4Comparison of forecast results with the truth (observation and analysis) 4Calculation of forecast quality in terms of so called scores 4Feedback to endusers and modellers

5 Outline: A mixture of 4General remarks 4The verification system at DWD 4Examples of verification results 4How did verification help (in some cases)?

6 Purposes of verification 4Watching the operational system 4Assessment of new model versions 4Evaluation of predictability 4Interpretation of model errors

7 Methods of verification 4Subjective assessment 4Objective assessment 4Special investigations

8 Course of verification on principal Forecast Modeller Analysis Verification Enduser Data control Observation Data control

9 Forecast of precipitation

10 Climate simulation for precipitation from: U. Böhm et.al.: Climate reconstruction over Europe...

11 Climate simulation for precipitation

12 Observation ( UTC UTC) and forecast of precipitation (T+18-T+42 from UTC

13 Observation ( UTC UTC) and forecast of precipitation (T+06-T+30 from UTC

14 Objective assessment calculation of scores which characterize the truth of the forecast Advantage: 8objective criteria for all forecasts Disadvantage: 8Perhaps only some aspects of the forecasts are catched. 8Impossibility to express all degrees of freedom of a forecast in some simple numbers

15 Methods of objective verification 4calculation of scores for continuous weather elements 4calculation of scores for categorical weather elements 4calculation of scores for probabilistic weather elements 4calculation of scores that are representative for very high resolution models

16 Scores for continuous weather elements 8Mean error 8Mean absolute error 8Root mean square error 8Standard deviation 8Correlation coefficient 8Skill scores

17 A basic rule Try to interpret each score dependent on its characteristics as - a physical and - a statistical term!

18 Scores for continuous weather elements - mean error or BIAS BIAS = F i ME = 1 N forecast N ( F O i i= 1 Properties: ideal value: 0 no general information about forecast quality If errors have a Gaussian distribution: Estimation of the expected value of errors i ) O i observation

19 Time series of BIAS of 2m-temperatures over Germany (January 1994 till May 2007)

20 Time series of BIAS of 2m-temperatures over Germany (January 1994 till May 2007)

21 Time series of BIAS of wind speed 10m over Germany (May May 2007)

22 Time series of BIAS of wind speed 10m over Germany stations below 100 m (May May 2007)

23 Monthly mean values of surface weather elements for station January 2008

24 Monthly mean values of surface weather elements for station January 2008

25 Monthly mean values of surface weather elements for station (cases with cloud cover observed and forecasted >4/8)

26 Example for conditional verification Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 (RMSE and STDV) Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 observed and forecasted values lower than 1020 hpa (RMSE and STDV) Forecasted and observed values of surface level pressure over the region of Germany during DJF 2005/2006 observed and forecasted values higher than 1020 hpa (RMSE and STDV)

27 Distribution of surface level pressure in COSMO-EU and in GME

28 Distribution of surface level pressure in COSMO-EU and in GME

29 Distribution of surface level pressure in COSMO-EU and in GME

30 Verification of surface weather elements: The effect of the SSO-scheme in COSMO-EU Consideration of frictional and gravity effects caused by processes that are not resolved by model orography

31 Distribution of surface level pressure in COSMO-EU and in GME

32 Distribution of surface level pressure in COSMO-EU and in GME

33 Distribution of surface level pressure in COSMO-EU and in GME

34 Distribution of surface level pressure in COSMO-EU and in GME

35 Scores for continuous weather elements - mean absolute error MAE = 1 N N i= 1 F i O i Properties: ideal value: 0 errors are simply added

36 Scores for continuous weather elements - root mean square error RMSE = N 1 ( F i N i= 1 O i ) 2 Properties: ideal value: 0 sensitive concerning phase and amplitude errors

37 Verification of surface weather elements: The effect of the SSO-scheme in COSMO-EU (Spring)

38 Verification of surface weather elements: The effect of the SSO-scheme in COSMO-EU(Autumn)

39 Verification of surface weather elements: Modification of T2m-diagnosis in GME, COSMO-EU and COSMO-DE z P + z 0 z A + z 0 0 z 0 l = l κ logarithmic freeturbulent-layer profile φ A expon. roughness-layer profile 0 φ S φ 2 φ δ φ m h 2m h z P + z d z A + z d z d 0 l d l = d κ unstable stable φ 2m φ S φ 20m 10m z P z A h 2m 0 z φ upper boundary of the lowest model layer lower boundary of the lowest model layer lowest model main level

40 Verification of surface weather elements: Modification of T2m-diagnosis in GME, COSMO-EU and COSMO-DE Z 0 taken from model surface Z 0 taken as 2 cm

41 Verification of surface weather elements: Modification of T2m-diagnosis in COSMO-EU (Winter)

42 Verification of surface weather elements: Modification of T2m-diagnosis in COSMO-EU (Winter)

43 Verification of surface weather elements: Modification of T2m-diagnosis in COSMO-EU (Summer)

44 Verification of surface weather elements: Modification of T2m-diagnosis in COSMO-EU (Sommer)

45 Diurnal cycle of forecasted and observed T2m- COSMO-EU, Summer 2007, Start 00 UTC

46 Diurnal cycle of forecasted and observed T2m - COSMO-EU, Summer 2008, Start 00 UTC

47 Diurnal cycle of forecasted and observed precipitation - COSMO-EU, Summer 2007(top) and 2008(bottom), Start 00 UTC!!!!!

48 Bias T2m- GME/LM/COSMO-EU, since 1999, Stations below 100 m, Start 00 UTC

49 Bias T2m- GME/LM/COSMO-EU, since 1999, Stations below 100 m, Start 00 UTC

50 Scores for continuous weather elements - standard deviation STDV = N 1 ( F i Oi ( F N i= 1 O)) Properties: ideal value: 0 If errors have a Gaussian distribution: around 68% of errors can be expected in the interval: < BIAS-STDV, BIAS+STDV > 2

51 Scores for continuous weather elements - correlation coefficient R : reference forecast short range : persistence medium range: climate CC = Properties: range: -1 till +1 (ideal value) mainly sensitive concerning phase errors critical value: 0.6 ( F ( F R R ( F ( F R)) R)) 2 ( A ( A R R ( A ( A R)) R)) 2

52

53

54 Scores for continuous weater elements Examples of scores for four artificial situations:

55 Scores for continuous weater elements Examples of scores for four artificial situations:

56 Scores for continuous weater elements Examples of scores for four artificial situations:

57 Scores for continuous weater elements Examples of scores for four artificial situations:

58 Scores for continuous weather elements - Skill Score S1 Properties: range: 0 (ideal value) till 200 (worst value)

59 Scores for continuous weather elements - skill score - relates a score of interest to a score of a reference forecast SKSC = SCORE( forecast) SCORE( ideal) SCORE( reference) SCORE( reference) Properties: range: - till +1 (ideal value) example: reduction of variance: sometimes given in percent RV = 1 RMSE RMSE 2 forecast 2 reference

60 A global score for COSMO models: COSI rmse = 1 n n ( ) t f t o 2 Continuous elements rmsvwe = 1 n n V f V o 2 Wind SS = 1 r r 2 f 2 p Skill score for wind and continuous elements related to persistence ETS = R "chance" T "chance" Skill score for categorical elements related to chance

61 Final definition of the score: 1 Continuous elements, sum over 8 time steps: (start: vv=3, end: vv=24, step: 3) Equal weights for all forecast times! SK SK j j 8 1 = 8 : i= 1 Skill sk i, j Score for element j sk i, j : Skill Score for element j at time stepi

62 SK sk Final definition of the score: 2 Categorical elements, sum over N time steps: (start: vv=vvb, end: vv=vve, step: vvs) Cloud cover: vvb=3, vve=24 vvs=3 Precipitation: vvb=6, vve=24 vvs=6 SK Equal weights for all forecast times and categories! j j i, j, m M N Skill Skill M l= 1 N = : : 1 * i= 1 sk Score Score in category m i, j, m for element for element j j at time stepi

63 Final definition of the score: 3 Sum over all elements: Equal weights for all elements! SK = 1 N E N E j= 1 SK j

64 The Score COSI : All days COSMO-EU V3.19 V 3.22 T 2m SSO

65 Long term trends of verification results for surface weather elements: Wind 00 UTC

66 Long term trends of verification results for surface weather elements: Wind 12 UTC

67 Long term trends of verification results for surface weather elements: Temperature 2m 12 UTC

68 Long term trends of verification results for surface weather elements: Minimumtemperature 2m 06 UTC

69 Long term trends of verification results for surface weather elements: Maximumtemperature 2m 18 UTC

70 Long term trends of verification results for surface weather elements: Temperature 2m 18 UTC

71 Single components of the COSI: Day 1 Trend: + Trend: + Trend: + Trend:? Trend:?

72 Verification of precipitation area mean

73 Scores for categorical weather elements aim: to express the correctness of forecasts for yes/no events for example: fog, thunderstorm, cloudiness, precipitation (over a limit of interest), temperature: warming or cooling,... Method: stratification of forecasts build up contingency tables

74 Scores for categorical weather elements General structure of contingency tables: Observation yes Observation no Forecast yes A B Forecast no C D

75 Scores for categorical weather elements Probability of detection: POD OY ON POD = A A + C FY A B FN C D range : 0-1 perfect : 1 maximise : overforecast event disadvantage: only yes-forecasts are considered

76 Scores for categorical weather elements False alarm rate: FAR OY ON FAR = A B + B FY A B FN C D range : 0-1 perfect : 0 maximise : underforecast event disadvantage: only yes-forecasts are considered

77 Scores for categorical weather elements Percentage correct: PEC A D A+ B+ + = C+ D A N D PEC = + range : 0-1 perfect : 1 maximise : do everything correct disadvantage: for rare events high correctness is suggested OY ON FY A B FN C D

78 Scores for categorical weather elements Frequency bias: FBI FBI = A A + + C B OY ON FY A B FN C D range : 0 - perfect : 1 maximise : do everything correct disadvantage: no account to accuracy

79 Scores for categorical weather elements Threat score (critical success index): CSI CSI = A+ B A + C range : 0-1 perfect : 1 advantage : false alarms and missed events are included disadvantage: Depends on climatological frequency of events (poorer scores for rare events) since some hits can occur purely due to random chance. OY ON FY A B FN C D

80 Scores for categorical weather elements Equitable threat score : ETS FY A B relates CSI to a reference forecast FN C D = A E A+ B+ ( A+ B) ( C E E( chance) = A+ B+ C range : -1/3-1 perfect : 1 advantage : false alarms and missed events are included, effect of a reference forecasts is included ETS OY ON A+ C) + D

81 Scores for categorical weather elements Heidke skill score : HSS OY relates PEC to a reference forecast = A+ D R A+ B+ C+ D R HSS R( chance) range : - -1 perfect : 1 advantage : false alarms and missed events are included, effect of a reference forecasts is included disadvantage: very sensitive to dry events = ( A+ B) ( A+ C) + ( C+ D) ( B+ D) A+ B+ C+ D ON FY A B FN C D

82 Scores for categorical weather elements True skill statistics (Hansen-Kuiper discriminant) : TSS TSS = A D B C ( A+ C) ( B+ D) range : perfect : 1 advantage : equal emphasis to yes/noevents disadvantage: for rare events similar to POD OY ON FY A B FN C D

83 Scores for categorical weather elements: GME-grid 2007

84 Scores for categorical weather elements COSMO-EU-grid 2007

85 Scores for probabilistic weather elements: The Brier score BRSC = N 1 ( F i N i= 1 O i F i : forecasted probability O i : observed probability 0: if event was not observed 1: if event was observed Properties: ideal value: 0 maximum: 1 ) 2

86 Verification of very high resolution models - basic problem: The double penalty hhigh resolution models often give right signals hbut 8 forecasts may be a little bit shifted compared to observations htherefore 8 forecasts are penalized at those points where the event was predicted 8 forecasts are penalized at those points where the event was not predicted 8 The right signal is not recompensed!

87 !!! A question for a quiz program: Which forecast is the best?!!!

88 !?!

89 Verification of very high resolution models - basic problem: Solution look into windows with different sizes calculate a set of scores that represent the content of the windows

90 GME COSMO-EU COSMO-DE Fractions skill score for forecasts of GME, COSMO-EU and COSMO-DE for December 2008, forecast time hours

91 Examination of statistical significance of fuzzy -verification results using bootstrapping Basic idea of bootstrapping: Repeat a resampling all elements of a given in a sample of forecasts and observations as often as necessary (N times) and calculate the relevant score(s) Calculate from N scores statistical properties of the sample such as mean value standard deviation, confidence intervals and quantiles Application to fuzzy -verification Resampling is done using blocks. Blocks are defined as single days. Number of resampling cases: N=Days*100 Calculation scores from N samples for NT thesholds and NW windows Calculation of quantiles for each window and threshold

92 Values and quantiles 0.1 and 0.9 for Upscaling ETS COSMO-DE, period June -August 2009 Germany

93 Differences between GME and COSMO-DE ETS(COSMO-DE) - ETS(GME) Significance test Germany COSMO-DE better than GME COSMO-DE worse than GME

94 Differences between COSMO-DE and COSMO-EU ETS(COSMO-DE) - ETS(COSMO-EU) Significance test Germany COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU

95 Differences between COSMO-DE and COSMO-EU ETS(COSMO-DE) - ETS(COSMO-EU) Significance test COSMO-DE better than COSMO-EU COSMO-DE worse than COSMO-EU

96 The problem of precipitation seems to be solved!!! BUT!

97 One of open questions: Surprisingly bad forecasts for temperature on some days during winter

98 One of open questions: Forecasted temperature over snow

99 One of open questions: Forecasted temperature over snow

100 The verification system at DWD verification against analysis verification against surface observations verification of vertical structures using different observation systems

101 Verification against analysis verification of geopotential, temperature, wind and humidity at all pressure levels calculation of ME, RMSE, correlation coefficients, Skill-Score S1,... calculation of scores for 26 regions calculation of horizontal distribution of errors

102 Verification against surface weather elements 8Verification worldwide (~3300 stations) 8 cloud cover(s) and 6h-precipitation (contingency tables), gusts 8 wind, temperature, dew point(depression), extreme temperatures 8Verification over Germany (~205 stations) 8 cloud cover(s) and 1h-precipitation (contingency tables), gusts 8 wind, temperature, dew point(depression), extreme temperatures

103 Verification of vertical structures 4Geopotential, wind, temperature, humidity 4TEMPs, AMDARs, wind profiler 4GME, COSMO-EU, COSMO-DE 4single TEMPs, single wind profiler,...

104 Vertical profiles of forecast errors according to AMDAR measurements bias of wind speed Forecasts starting 00 UTC since 2000 positive bias negative bias Fc-time:00 h Fc-time:24 h Fc-time:48 h

105 Vertical profiles of forecast errors according to TEMP measurements bias of wind speed Forecasts starting 00 UTC since 2000 positive bias negative bias Fc-time:00 h Fc-time:12 h Fc-time:24 h Fc-time:36 h Fc-time:48 h

106 Number of scores for each model run (crude approximation) 8Verifcation against surface observations: ~ scores 8Verification against analysis: ~ scores 8Verification against upper air observations: ~ scores 8Calculate the scores AND watch the system! 8Verification is a full time job for at least 2 persons!

107 A typical situation

108 First possible solution of the problem: t-test,... Hypothesistests: Are average values of two different time series significantly different? Significancetests for the parameters of these time series -t Test weak t test t = t test N X μ S N: Number of cases X: sample mean value S: sample standard deviation μ: hypothetial mean value t S = 2 D = x1 x S D ( N ) S1 + ( N2 1) S N + N 2 1 N1N2 N + N

109 Second possible solution of the problem: bootstrapping Sample with 10 elements Realisation 1: mean value using elements: Realisation 2: mean value using elements: Realisation 3: mean value using elements: Realisation 4: mean value using elements: Realisation 5: mean value using elements: Realisation 6: mean value using elements: Realisation 7: mean value using elements: Realisation 8: mean value using elements: Realisation 9: mean value using elements: Realisation 10: mean value using elements:

110 A typical situation

111