Statistical interpretation of Numerical Weather Prediction (NWP) output 1
2
3
Four types of errors: Systematic errors Model errors Representativeness Synoptic errors Non-systematic errors Small scale noise 4
Even when we get rid of systematic errors, make the synoptic forecast perfect and only verify against representative observations the meso-scale 5 noise will still yield non-perfect forecasts
The two neighbouring stations Potsdam and Lindenberg outside Berlin are just 75 kilometres apart and are situated in almost the same environment. How well would a forecast based on the other one s observation verify? Other nearby stations were also used (Magdeburg, Dresden, Poznan and Stettin). They provided, together with the previous two data to calculate an average temperature as forecast. 6
Stettin 3 m Magdeburg 84 m Potsdam 99 m Lindenberg 115 m Poznan 92 m Dresden 226 m 7
Four tests were conducted all with the objective to estimate ( forecast ) the temperature at Lindenberg: 1.Using the observation from Potsdam as forecast 2.Using an average of all five surrounding stations 3.The same but with weights proportional to the square of the distance from Lindenberg 4.The same, but without using the observation from nearby Potsdam 8
Other weightings RMSE Potsdam s day & night observations applied on Lindenberg All five surrounding weighted observations 9
Other weightings SDE Potsdam s day & night observations applied on Lindenberg All five surrounding weighted observations 10
Other weightings MAE Potsdam s day & night observations applied on Lindenberg All five surrounding weighted observations 11
all kind of Potsdam s observation applied on Lindenberg All five surrounding weighted observations MAE ECMWF 12UTC + 12h and +24 h 12
all kind of Rather 0.6 Error=0 at t=0? 13
Conclusions from this observation investigation: 1. During favourable conditions the lowest RMSE and SDE would be around 0.8ºK, for MAE 0.6ºK 2. During seasons when the temperature depends quite a lot on the clouds the values increase to around 2ºK resp. 1½ºK. 3. Verified against a specific site, the weighted area average (3) provided the best forecast, whereas the neighboring station observation method (1) provided the worst. 14
Conclusions for all kinds of forecasts beyond a few hours: 1. Due to micro-scale variability the 2 metre temperature is at present not possible to forecast with higher accuracy than 0.8ºK (RMSE,SDE) or 0.6ºK (MAE). 2. Provided homogenous environment an area average forecast, applied to a specific site, might be superior to a site specific. 3. Site specificness only has meaning if the site is not representative to the area, if its climate is different to the area as a whole. 15
True and false error curves 16
Four types of errors: Systematic errors Model errors Representativeness Synoptic errors Non-systematic errors Small scale noise 17
18
There is much more to say about this - at some other time 19
Four types of errors: Systematic errors Model errors Representativeness Synoptic errors Non-systematic errors Small scale noise 20
21
RMSE errors of raw T399 grid point +24h forecasts 2007 for Tromsö airport [T] 2.5 3.5 2.8 T 6.2 6.0 3.7 3.0 4.2 The main contributor to the large RMSE for inland grid points are mean errors between up to -5º 22
RMSE after Kalman-2 filtering makes the quality almost the same for all grid points 1.7 1.7 1.6 T 1.7 1.6 1.7 1.8 1.7 23
Statistical correction, calibration or interpretation: A heavily biased temperature forecast Tromsø (northern Norway) 24
The EPS plume after statistical correction Tromsø (northern Norway) 25
The forecast (- - - - ) varies more than reality. The adaptive statistical filtering corrects for both mean error and overvariability Tromsø (northern Norway) 26
No simple, straight bias. The mean error depends on the forecast Tromsø (northern Norway) 27
Four types of errors: Systematic errors Model errors Representativeness Synoptic errors Non-systematic errors Small scale noise 28
29
The systematic errors we want to correct for are not only 1-dimensional flat biases... 30
Obs-T fc = correction Bias? T fc Corr = A(t) 31
Obs-T fc = correction Bias? T fc Corr = A(t) It appears as if the old bias has abruptly changed into a new one 32
Obs-T fc = correction Systematic error T fc Corr = A(t) + B(t) T fc In reality the systematic error has stayed more or less the same, but defined by two coefficients, A and B 33
Err Flat true bias Err 2-D systematic errors T fc T fc Apparent non-systematic errors.. Err Err but when projected into an additional dimension they appear to be systematic T fc T fc T850 fc 34
A very, very brief introduction to the adaptive procedure...before the break 35
T fc - obs 2-dim error equation Expected error T fc Last NWP forecast 36
T fc - obs 2-dim error equation Obs error Last verified NWP forecast T fc 37
T fc - obs 2-dim error equation T fc 38
T fc - obs 2-dim error equation Slightly modified values of A and B Next forecast Expected error T fc 39
The historical background to and its classical application 2nd lecture RSHU 40
The origin of the Kalman filter 1960 launching intercontinental ballistic missiles Track uncertainty Corrected track Observation error Airborne radar Ground radar Estimated position 2nd lecture RSHU 41
Latest observed position Remote observer Intended position True position 2nd lecture RSHU Estimated position 42
2nd lecture RSHU 43
2nd lecture RSHU 44
1-D corrects for mean errors ( biases ) but can also illustrate the basic philosophy, here in three ways 1. Pictorial description 2. Mathematical derivation 3. Graphical illustration 2nd lecture RSHU 45
How is it done? The pictorial version 2nd lecture RSHU 46
Obs-T fc = correction The filter makes a cold start i.e. no correction is applied The assumed covariance of a cold start T fc 2nd lecture RSHU 47
Obs-T fc = correction The latest verified numerical forecast The ideal correction (=the inverse of the error) and its error T fc The forecast 2nd lecture RSHU 48
Obs-T fc = correction The latest verified numerical forecast T fc makes the filter change its value and the initial uncertainty is shrunk 2nd lecture RSHU 49
Obs-T fc = correction T fc The correction is defined by the relative size of the observation uncertainty and the filter uncertainty 2nd lecture RSHU 50
Obs-T fc = correction The ideal correction (=the inverse of the error) and its error T fc A new forecast makes the filter change its value and the initial uncertainty is shrunk 2nd lecture RSHU 51
Obs-T fc = correction T fc The correction is again defined from the relative size of the observation uncertainty and the filter uncertainty 2nd lecture RSHU 52
Obs-T fc = correction Finally we end up with the filter oscillating around a mean state with a certain variance (uncertainty) correction Bias T fc This uncertainty have a lower threshold and can never be = 0, which would lock the Kalman filter 2nd lecture RSHU 53
How is it done? The mathematical derivation 2nd lecture RSHU 54
Y τ =the observed forecast error at verification time τ Y Tfc Tobs Y τ is the sum of the ideal correction χ τ and the noise ν τ Y v 2nd lecture RSHU 55
We introduce the first guess values X τ/τ-1 = A τ X τ/τ-1 where A τ = 1-F 1, where F 1 << 1 and Q τ/τ-1 to be discussed later The difference between the first guess value X τ/τ-1 and the observed value Y τ must obviously affect how much we shall modify X τ/τ-1 2nd lecture RSHU 56
We now introduce δ τ (0 <δ τ < 1) which indicates how much of the difference between Y τ and X τ/τ-1 that shall modify X τ/τ-1 X X ( Y X 1) / / 1 / Assume that the error in our estimation of χ τ is ε τ X / which yields X ( Y X 1) / 1 / 2nd lecture RSHU 57
And with the noise term Y v we get X ( v X 1) / 1 / and after rearrangement of the terms X ) ( 1 )( / 1 v 2nd lecture RSHU 58
The (co) variance term which indicates the (un) certainty of our estimation The uncertainty of Y v depends on sub-grid turbulence, non-systematic synoptic errors or measurement errors, what we choose to call the observation error D τ cov( v ) D 2nd lecture RSHU 59
cov( ) cov( X ) / Q / and cov( X ) / 1 Q / 1 and cov( v ) D...yields: Q 1 ) 2 / ( Q / 1 2 D 2nd lecture RSHU 60
We differentiate Q 1 ) 2 / ( Q / 1 2 D with respect to δτ dq d / 2(1 ) Q / 1 2 D 2nd lecture RSHU 61
from dq d / 2(1 ) Q / 1 2 D we get min D ( 1 ) Q / 1 0 min D Q / 1 Q / 1 Which is the final result 2nd lecture RSHU 62
1 / 1 / min Q D Q 2 2 / / 1 ) 1 ( D Q Q 2 2 1 / / ) 1 ( D Q Q with and the updated forward (co)variances become 2nd lecture RSHU 63
How is it done? The graphical illustration 2nd lecture RSHU 64
Assume an unknown process χ which can be 1-dim (the mean error or bias of NWP) or N- dim (the N coefficients in an error correction equation) τ-1 τ τ+1 χ 2nd lecture RSHU 65
We have at τ-1 an estimated value X τ-1/τ-1 of the unknown process χ with variance Q τ-1/τ-1 Q τ-1/τ-1 X τ-1/τ-1 τ-1 τ τ+1 χ 2nd lecture RSHU 66
We carry X forward in time by a linear model A, assuming that the variance increases slightly X τ-1/τ-1 Q τ-1/τ-1 +C A X τ-1/τ-1 τ-1 τ τ+1 χ 2nd lecture RSHU 67
We have a predicted estimate, X τ/τ-1 and Q τ/τ-1 similar to the first guess in numerical weather prediction Q τ/τ-1 X τ/τ-1 τ-1 τ τ+1 χ 2nd lecture RSHU 68
The observation Y τ, with variance D, of the unknown process χ will modify the first guess value X τ/τ-1 Q τ/τ-1 X τ/τ-1 Y τ D χ τ-1 τ τ+1 2nd lecture RSHU 69
The weighting together of D and Q τ/τ-1 yields a variance of the new estimation X τ/τ and Q τ/τ Q τ/τ-1 X τ/τ Q τ/τ D χ τ-1 τ τ+1 2nd lecture RSHU 70
The new estimation of X τ/τ and Q τ/τ Q τ/τ X τ/τ τ-1 τ τ+1 χ 2nd lecture RSHU 71
The new estimation starts with predicting X τ+1/τ and Q τ+1/τ χ X τ/τ A τ Q τ/τ +c X τ+1/τ Q τ+1/τ τ-1 τ τ+1 2nd lecture RSHU 72
...and a new observation arrives χ A τ Q τ/τ +c X τ+1/τ Q τ+1/τ New Y τ τ-1 τ τ+1 2nd lecture RSHU 73
But there are fundamental differences between 1- dimensional filtering and multi-dimensional 74
24 hour 2 m temperature forecast for Kiruna in Lapland winter 2001-2002 The verification yielded RMSE=5.0 2nd lecture RSHU 75
A 1-dimensional Kalman filter reduces an overall bias Correction out of step The corrections yielded a reduction of the mean error from 2.6 to 0.3 and RMSE from 5.0 to 4.2 2nd lecture RSHU 76
A 2-dimensional Kalman filter system also improves the forecasts of the extremes Correction not out of step Two good achievements: The Kalman filtering has reduced two systematic errors: a positive mean error and an underestimation of the variability 2nd lecture RSHU 77
Why does the improvement not show up in the verification? The corrections still yielded a reduction of the mean error from 2.6 to 0.3 but the RMSE from 5.0 only to 4.6 and not to 4.2 as with the 1-D 2nd lecture RSHU Is the 2-D worse than the 1-D?? 78
END of part I 79