Evaluating weather and climate forecasts

Transcription

1 Evaluating weather and climate forecasts Chris Ferro Department of Mathematics University of Exeter, UK RSS Highlands Local Group and University of St Andrews (St Andrews, 1 September 2016)

2 Monitoring forecast performance (8 August 2016)

3 Informing forecast development (8 August 2016)

4 Guiding responses to forecasts Figure 9.7 Relative error measures of CMIP5 model performance, based on the global seasonal-cycle climatology ( ) computed from the historical experiments. Rows and columns represent individual variables and models, respectively. The error measure is a space time root-mean-square error (RMSE), which, treating each variable separately, is portrayed as a relative error by normalizing the result by the median error of all model results (Gleckler et al., 2008). For example, a value of 0.20 indicates that a model s RMSE is 20% larger than the median CMIP5 error for that variable, whereas a value of 0.20 means the error is 20% smaller than the median error. No colour (white) indicates that model results are currently unavailable. A diagonal split of a grid square shows the relative error with respect to both the default reference data set (upper left triangle) and the alternate (lower right triangle). The relative errors are calculated independently for the default and alternate data sets. All reference data used in the diagram are summarized in Table 9.3. IPCC AR5 Chapter 9

5 Introduction: forecasts and their evaluation Probability forecasts: proper scoring rules Evaluation when truth is uncertain: unbiased scoring rules Ensemble forecasts: fair scoring rules Conclusion

6 Types of forecast (8 August 2016)

7 Probabilities and ensembles Definition: A probability forecast is a probability distribution for a predictand. Definition: An ensemble forecast is a set of ( equally likely ) possible values for a predictand. Weather/climate models produce ensembles, not probabilities. Ensembles are produced by running a model several times with perturbed initial conditions and/or perturbed model parameters. Post-processing turns ensembles into probability forecasts.

8 An ensemble forecast (11 August 2016)

9 Use scoring rules to measure forecast performance Definition: A scoring rule, s, is a real-valued function of a forecast, f, and an outcome, x. Calculate the score, s(f, x), for each forecast and measure performance by the mean score, e.g. mean squared error. We use negatively-oriented scoring rules: lower scores indicate better forecasts.

10 Use scoring rules to measure forecast performance Definition: A scoring rule, s, is a real-valued function of a forecast, f, and an outcome, x. Calculate the score, s(f, x), for each forecast and measure performance by the mean score, e.g. mean squared error. We use negatively-oriented scoring rules: lower scores indicate better forecasts. Other measures are prone to inflation due to trends. Example: correlation =.87

11 Use scoring rules to measure forecast performance Definition: A scoring rule, s, is a real-valued function of a forecast, f, and an outcome, x. Calculate the score, s(f, x), for each forecast and measure performance by the mean score, e.g. mean squared error. We use negatively-oriented scoring rules: lower scores indicate better forecasts. Post-processing models, x w f (x; w, θ) for ensemble w and parameter θ, can be fitted by minimum score estimation: ˆθ := argmin θ n s(f j, x j ) j=1 where f j (x) := f (x; w j, θ).

12 Introduction: forecasts and their evaluation Probability forecasts: proper scoring rules Evaluation when truth is uncertain: unbiased scoring rules Ensemble forecasts: fair scoring rules Conclusion

13 Use proper scoring rules for probability forecasts Definition: A scoring rule is proper if the expected score, E x {s(f, x)}, over any distribution, p, for x, is optimized at f = p.

14 Use proper scoring rules for probability forecasts Definition: A scoring rule is proper if the expected score, E x {s(f, x)}, over any distribution, p, for x, is optimized at f = p. Example: Let x {0, 1}, p := Pr(x = 1) and f [0, 1]. s 1 (f, x) := (f x) 2 is proper since is minimized at f = p. E x {s 1 (f, x)} = (f p) 2 + p(1 p)

15 Use proper scoring rules for probability forecasts Definition: A scoring rule is proper if the expected score, E x {s(f, x)}, over any distribution, p, for x, is optimized at f = p. Example: Let x {0, 1}, p := Pr(x = 1) and f [0, 1]. s 1 (f, x) := (f x) 2 is proper since is minimized at f = p. E x {s 1 (f, x)} = (f p) 2 + p(1 p) s 2 (f, x) := f x is improper since E x {s 2 (f, x)} = p + (1 2p)f is minimized at f = 0 if p < 1/2 and at f = 1 if p > 1/2.

16 Good forecasts are calibrated and sharp p f := conditional distribution of outcomes given forecast f Definition: Forecasts are calibrated if p f = f for all f.

17 Good forecasts are calibrated and sharp p f := conditional distribution of outcomes given forecast f Definition: Forecasts are calibrated if p f = f for all f. Definition: Forecasts are sharp if p f is concentrated for all f.

18 Good forecasts are calibrated and sharp p f := conditional distribution of outcomes given forecast f Definition: Forecasts are calibrated if p f = f for all f. Definition: Forecasts are sharp if p f is concentrated for all f. Let x {0, 1}, f [0, 1] and p f := Pr(x = 1 f ). Plot the graph of p f and the distribution of f.

19 Proper scoring rules favour calibrated, sharp forecasts Let S(f, p) := E x {s(f, x)} when x p so that E f,x {s(f, x)} = E f [E x f {s(f, x) f }] = E f {S(f, p f )}, where p f denotes the conditional distribution of x given f, and S(f, p f ) = S(f, p f ) S(p f, p f ) + S(p f, p f ) =: C(f, p f ) + R(p f ).

20 Proper scoring rules favour calibrated, sharp forecasts Let S(f, p) := E x {s(f, x)} when x p so that E f,x {s(f, x)} = E f [E x f {s(f, x) f }] = E f {S(f, p f )}, where p f denotes the conditional distribution of x given f, and S(f, p f ) = S(f, p f ) S(p f, p f ) + S(p f, p f ) =: C(f, p f ) + R(p f ). Let s be proper so that S(f, p) S(p, p) for all f, p. C measures calibration: C(f, p f ) 0 with equality when p f = f. R measures sharpness: R is concave. Example: Let x {0, 1}, f [0, 1] and s(f, x) := (f x) 2. Then C(f, p f ) = (f p f ) 2 and R(p f ) = p f (1 p f ).

21 Windspeed forecasts 36h-ahead forecasts of 80m windspeed at Lerwick for n = 580 days from Let x = 1 if windspeed exceeds 15 ms 1. Score: n 1 n j=1 (f j x j ) 2 = 0.14 (0.01)

22 Windspeed forecasts 36h-ahead forecasts of 80m windspeed at Lerwick for n = 580 days from Let x = 1 if windspeed exceeds 15 ms 1. Score: n 1 n j=1 (f j x j ) 2 = 0.14 (0.01) Forecasts take values p k = k/12 for k = 0, 1,..., 12. Let n k be the number of times the forecast is p k and let x k be the mean of the corresponding outcomes. Calibration: n 1 k n k(p k x k ) 2 = 0.02 Sharpness: n 1 k n k x k(1 x k ) = 0.12

23 Windspeed forecasts: calibration diagram

24 Introduction: forecasts and their evaluation Probability forecasts: proper scoring rules Evaluation when truth is uncertain: unbiased scoring rules Ensemble forecasts: fair scoring rules Conclusion

25 Observation error x = true value of predicted quantity y = observed value of predicted quantity f = forecast probability distribution for x s = scoring rule: assigns score s(f, y) to f

26 Observation error x = true value of predicted quantity y = observed value of predicted quantity f = forecast probability distribution for x s = scoring rule: assigns score s(f, y) to f Observation error causes two problems. 1. Proper scoring rules favour good forecasts of y, not of x. Proper scores can favour forecasters who issue worse forecasts of the truth. 2. Scores s(f, y) are biased estimates of s(f, x). Changes in proper scores over time can be due to changes in observation quality.

27 Observation error Example: Let x, y {0, 1}, f [0, 1], s(f, y) = (f y) 2. Denote the observation error probabilities by r x := Pr(y x x). Then E y {s(f, y)} = (f q) 2 + q(1 q) where q = (1 r 1 )p + r 0 (1 p) when Pr(x = 1) = p. Graphs of E y {s(f, y)} against f when p = 0.1 and the observation error probabilities are r 0 = r 1 = 0 (solid line) and r 0 = r 1 = 0.1,..., 0.5 (dashed). Dots ( ) mark the minima.

28 Proper, unbiased scoring rules x = true value of predicted quantity y = observed value of predicted quantity r = conditional distribution of y given x s = scoring rule: assigns score s(f, y) to f

29 Proper, unbiased scoring rules x = true value of predicted quantity y = observed value of predicted quantity r = conditional distribution of y given x s = scoring rule: assigns score s(f, y) to f Definition: The scoring rule is proper under r if E y {s(f, y)}, calculated for any distribution, p, for x, is optimized at f = p.

30 Proper, unbiased scoring rules x = true value of predicted quantity y = observed value of predicted quantity r = conditional distribution of y given x s = scoring rule: assigns score s(f, y) to f Definition: The scoring rule is proper under r if E y {s(f, y)}, calculated for any distribution, p, for x, is optimized at f = p. Definition: The scoring rule is unbiased for s 0 under r if E y x {s(f, y) x} = s 0 (f, x) for all f and x.

31 Proper, unbiased scoring rules x = true value of predicted quantity y = observed value of predicted quantity r = conditional distribution of y given x s = scoring rule: assigns score s(f, y) to f Definition: The scoring rule is proper under r if E y {s(f, y)}, calculated for any distribution, p, for x, is optimized at f = p. Definition: The scoring rule is unbiased for s 0 under r if E y x {s(f, y) x} = s 0 (f, x) for all f and x. A scoring rule, s, has both properties if it is unbiased under r for a proper scoring rule, s 0.

32 Binary predictands Let x, y {0, 1}, f [0, 1], r x := Pr(y x x). Proposition: Let s 0 be a (non-trivial, proper) scoring rule. There exists a scoring rule, s, that is unbiased for s 0 under r if and only if r 0 + r 1 1, in which case s(f, y) = s 0 (f, y) + r y{s 0 (f, y) s 0 (f, 1 y)} 1 r 0 r 1.

33 Binary predictands Let x, y {0, 1}, f [0, 1], r x := Pr(y x x). Proposition: Let s 0 be a (non-trivial, proper) scoring rule. There exists a scoring rule, s, that is unbiased for s 0 under r if and only if r 0 + r 1 1, in which case s(f, y) = s 0 (f, y) + r y{s 0 (f, y) s 0 (f, 1 y)} 1 r 0 r 1. Example: Artificial aircraft icing data (after Briggs et al. 2005, Monthly Weather Review): 1000 forecasts f = Pr(icing), error probabilities r 0 = r 1 = 0.2, scoring rule s 0 (f, x) = (f x) 2. s 0 (f, x) = (0.006) s 0 (f, y) = (0.009) s(f, y) = (0.014)

34 Continuous predictands Given proper s 0, we need s to satisfy the integral equation s(f, y)r(y x)dy = s 0 (f, x) for all f and x.

35 Continuous predictands Given proper s 0, we need s to satisfy the integral equation s(f, y)r(y x)dy = s 0 (f, x) for all f and x. Example: The (proper) Dawid-Sebastiani scoring rule is s 0 (f, x) = log σ + (x µ)2 2σ 2 where µ and σ are the mean and standard deviation of f. Let E(y x) = x and var(y x) = c 2. Then s(f, y) = log σ + (y µ)2 c 2 2σ 2.

36 Introduction: forecasts and their evaluation Probability forecasts: proper scoring rules Evaluation when truth is uncertain: unbiased scoring rules Ensemble forecasts: fair scoring rules Conclusion

37 Evaluating ensemble forecasts Weather/climate models produce ensembles, not probabilities. Performance depends on the meaning of the ensemble empirical distribution is a probability forecast? 2. set of functionals (e.g. quantiles) of a probability forecast? 3. simple random sample from a probability forecast?

38 Evaluating ensemble forecasts Weather/climate models produce ensembles, not probabilities. Performance depends on the meaning of the ensemble empirical distribution is a probability forecast? 2. set of functionals (e.g. quantiles) of a probability forecast? 3. simple random sample from a probability forecast? Let members w 1,..., w m be i.i.d. with unknown distribution f. Try evaluating a proper scoring rule, s(f m, x), for the e.d.f., f m. When x p, E w,x {s(f m, x)} typically is not optimized at f = p. Example: Let x Ber(p), w i Ber(f ) and s(f m, x) = ( w x) 2. If m = 10 then E w,x {s(f m, x)} is minimized at f = 0 for p 5%.

39 Use fair scoring rules for ensemble forecasts Definition: A scoring rule is fair if the expected score, E w,x {s(w, x)}, over i.i.d. ensemble members sampled from a distribution f and any distribution, p, for x, is optimized at f = p. Fair scoring rules effectively evaluate f given only a sample from f, e.g. E w f,x {s(w, x) f, x} is a proper scoring rule for f.

40 Use fair scoring rules for ensemble forecasts Definition: A scoring rule is fair if the expected score, E w,x {s(w, x)}, over i.i.d. ensemble members sampled from a distribution f and any distribution, p, for x, is optimized at f = p. Fair scoring rules effectively evaluate f given only a sample from f, e.g. E w f,x {s(w, x) f, x} is a proper scoring rule for f. Theorem: Let x, w 1,..., w m {0, 1} and let s(w, x) = s m w,x. The scoring rule is fair if (m k)(s k+1,0 s k,0 ) = k(s k 1,1 s k,1 ) for k = 0, 1,..., m and s k+1,0 s k,0 for k = 0, 1,..., m 1. Example: s(w, x) = ( w x) 2 w(1 w)/(m 1) is fair.

41 Introduction: forecasts and their evaluation Probability forecasts: proper scoring rules Evaluation when truth is uncertain: unbiased scoring rules Ensemble forecasts: fair scoring rules Conclusion

42 Summary Evaluation makes forecasts more useful. Performance depends on the meaning of the forecast. Rank probability forecasts using proper scoring rules. Rank i.i.d. ensemble forecasts using fair scoring rules. Account for observation error using unbiased scoring rules. Other methods may be needed to understand performance.

43 Some other research topics Evaluating forecasts of high-dimensional space-time fields Evaluating forecasts of phenomena such as hurricanes Evaluating forecasts of rare, extreme events Predicting the performance of climate forecasts

44 References Ferro CAT (2016) Proper scoring rules in the presence of observation error. In preparation. Ferro CAT (2014) Fair scores for ensemble forecasts. Quarterly Journal of the Royal Meteorological Society, 140, Fricker TE, Ferro CAT, Stephenson DB (2013) Three recommendations for evaluating climate predictions. Meteorological Applications, 20, Mitchell K, Ferro CAT (2016) Proper scoring rules for interval probabilistic forecasts. Submitted. Otto FEL, Ferro CAT, Fricker TE, Suckling EB (2015) On judging the credibility of climate predictions. Climatic Change, 132,

45

46 Concavity of R(p) := S(p, p) for binary outcomes For x {0, 1}, S(f, p) := E x {s(f, x)} = ps(f, 1) + (1 p)s(f, 0). Fix f [0, 1]. The graph of S(f, p) against p is a straight line intersecting the graph of R(p) at p = f. If s is proper then S(f, p) R(p) for all p. This can happen only if the straight line is tangent to R(p) at f and if R(p) is concave.

47 Decision problems define proper scoring rules Let L(ˆx, x) := loss incurred by ˆx when outcome is x. Let ˆx f := argmin E x {L(ˆx, x)} when x f be the Bayes action. x The incurred loss, s(f, x) := L(ˆx f, x), is a proper scoring rule.

48 Evaluating climate forecasts 1. Long lead times and unknowable greenhouse gas concentrations mean few or no observations are available. Evaluate hindcasts instead of forecasts. Hindcast scores inform belief about forecast performance. Train to become good judges of relative performance. 2. Good models or good forecasts? Initial-condition ensembles are not designed to be well calibrated, so proper scoring rules may be inappropriate.

49 Proper scoring rules for interval probability forecasts Forecaster must issue one of several fixed probability intervals, e.g. [0,.05], (.05,.15],..., (.95, 1]. Scoring rules are functions of an interval, I, and an outcome, x. Definition: A scoring rule is interval-proper if its expectation, E x {s(i, x)}, over any distribution, p, for x, is optimized at I p. Example: The scoring rule s(i, x) := (m x) 2, where m is the midpoint of I, is not interval-proper. For example, E x {s(i, x)} is optimized by I = [0,.05] when p =.06. Let I k := [a k 1, a k ). For any increasing sequence g j the scoring rule s(i k, x) := k 1 j=1 (g j g j 1 )(a j x) is interval-proper. Example: s(i k, x) := (x a k 1 )(x a k ) is interval-proper.

50 Windspeed forecasts Weibull forecast distributions with density f and cdf F. Two proper scoring rules are the logarithmic score log f (x) and the continuous ranked probability score 0 {F(t) H(t x)} 2 dt.

51 Windspeed forecasts: PIT histogram Probability Integral Transform: if x F then F (x) Uni(0, 1). Plot the distribution of PIT values, u j := F j (x j ).