WG 3: Verification and applications of ensemble forecasts

Size: px

Start display at page:

Download "WG 3: Verification and applications of ensemble forecasts"

Erik Moody
5 years ago
Views:

1 WG 3: Verificatin and applicatins f ensemble frecasts Participants: Ken Mylne (Chair), Martin Leutbecher (secretary), Magdalena A. Balmaseda, Pac Dblas-Reyes, Lizzie Frude, Jse A. Garcia-Mya, Anna Ghelli, Renate Hagedrn, Edit Hagel, Flrian Pappenberger, Fernand Prates, Anders Perssn, Cristina Prim, Kamal Puri, Thmas Schumann, Olivier Talagrand, Jutta Thielen, Helen Titley First, the discussins f the wrking grup n the Verificatin f Ensemble Frecasts and the Applicatin f Ensemble frecasts are summarised in Sectins 1 and 2, respectively. This is fllwed by specific recmmendatins fr ECMWF in Sectin 3. Mst f these recmmendatins are als cnsidered t be relevant fr ther prducers f ensemble frecasts. 1. Verificatin f Ensemble Frecasts 1.1. Purpse f Verificatin The wrking grup identified three different purpses f the verificatin: Objective measure fr imprving the EPS and guide future develpments f system and funding Guide frecasters, service prviders and users n which prduct(s) t trust and use. Hwever, sme participants thught that, at present, nt many frecasters lk at verificatin scres fr prbabilistic frecasts. Demnstrate usefulness f ensemble predictin systems fr particular applicatins r varius user grups (frm general public t decisin makers) Statistical Significance f Results It was recmmended that cnfidence intervals shuld be prvided in all verificatin statistics. Further research will be required in rder t study methds f cmputing cnfidence intervals and significance tests and t verify the reliability f cnfidence intervals. Examples: Btstrap methds, analytic methds. Latter require assumptins abut the distributin. Btstrap requires fewer assumptins but may be cstly t calculate Accunting fr the uncertainty f the verificatin data There was cnsensus that it is necessary t accunt fr bservatin errrs/analysis errrs in the prbabilistic verificatin. One accepted way f ding this (fr rank-histgram and prbabilistic verificatin in general) is t add nise with the same statistics as that f the verifying data t the individual ensemble frecasts (Saetra et al). A secnd alternative is t use a decnvlutin methd (Bwler). A third methd is t cnsider the bservatin as a prbability distributin (Talagrand & Candille). The first methd will tell us nly hw well we can predict an bservatin nt hw well we can predict the true state. It was nted that it can be difficult t estimate the errr characteristics f sme verificatin data: e.g. rain gauge data Chice f verificatin measures There are many different verificatin measures in use, and the grup did nt attempt t review them all. Discussin centred arund hw t cmmunicate perfrmance effectively t decisin-makers and users wh may nt be specialist in the details f the science. Wrkshp n Ensemble Predictin, 7-9 Nvember

2 The chice f the verificatin will reflect its purpse (see abve). There was general cnsensus that classic upper air verificatin has its value in prviding guidance cncerning the general accuracy and reliability f a (prbabilistic) predictin system. It shuld be cmplemented by verificatin f surface weather variables which are particularly relevant t users and fr shrt-range ensemble predictin. The wrking grup agreed that several measures are required t examine different aspects f the Prbability Density Functin (PDF). Fr instance, the rank histgram, n the ne hand, and the reliability cmpnent f the Brier scre with respect t a threshld, n the ther hand, measure different aspects f reliability. A single summary measure (althugh desirable frm a management perspective) was cnsidered inadequate. It was nted that reliability is a statistical prperty, and glbal reliability, as estimated by a particular measure ver a given set f realizatins f an EPS, may always result frm mutual cmpensatin between individually unreliable subsets (Example: flatness f a rank-histgram is a necessary but nt a sufficient cnditin fr the statistical reliability f an EPS). The decmpsitin f scres t better understand results was encuraged where apprpriate fr the audience (e.g. reliability and reslutin cmpnent f Brier-type scres) Cncerning recmmendatins fr system upgrades, selective use f particular scres was discuraged as this may encurage the selectin f the favurable subsets. In ther wrds, the impact f system upgrades (e-suite versus -suite) shuld always be dcumented by the same standard set f scres (further discussin is required which nes these shuld be). Use f Reliability Diagrams (including sharpness f the a psteriri calibrated prbabilities) was cnsidered t be the easiest way t cmmunicate prbabilistic verificatin t nn-specialist audiences. There is need fr care in the use f the Relative Operating Characteristic, and the area under the ROC in particular, as a verificatin measure. It is particularly difficult t explain t users and nnspecialists. Value f ROC area is very sensitive t hw it is calculated and t small changes in perfrmance. Use f cnfidence measures wuld help. Prbabilistic scres need t be cmplemented by mre general mdel validatin: (ability t simulate climatlgical mean and variance, ) 1.5. Avidance f False Skill (added by Chair) It was nted, but nt discussed in detail due t time cnstraints, that withut due care verificatin can indicate mre skill in frecast systems than is warranted due t climatlgical differences between lcatins included in the verificatin set - the base rate effect. This was illustrated by Tm Hamill s presentatin in the wrkshp. It was nted that ECMWF has already taken steps t avid this in sme f the verificatin presented at the wrkshp. Tw methds are prpsed t minimise the effect: Define events fr prbabilistic verificatin as percentiles f the climatlgical distributin rather than abslute values [eg. p(t>90 th percentile) r p(t> 1 s.d. abve nrmal) but nt p(t>5 Celsius) ]. (This methd is prpsed by Hamill and Juras, and was used by ECMWF in sme verificatin presented at the wrkshp.) Prpsed by Hamill in his presentatin: grup sites accrding t their climatlgical frequency f the event (eg grup all sites fr which 5mm/24h ccurs with climatlgical frequencies f 0-5%, 6-15%, 16-25%, etc). Estimate the gegraphical area fr which each climatlgical categry is 2 Wrkshp n Ensemble Predictin, 7-9 Nvember 2007

3 representative. Calculate verificatin statistics fr each grup f sites, and then average results weighting the values accrding t the prprtin f gegraphical area represented by each grup Verificatin f rare/extreme events Sme discussin fcussed n whether a different methdlgy needs t be adpted fr the verificatin f rare/extreme events (t be distinguished frm high-impact events which need nt be rare in the climatlgical sense) There is a prblem f sample size fr rare events; n statistical significance may be reached. The same is true fr events which ccur almst always because f the symmetry f scres (event ccurring / nt ccurring). The predictin f events within a finite space-time dmain selected arund a particular weather event r atmspheric feature (as ppsed t pint values) will lead t higher prbabilities fr sme extreme events than lcal prbabilities (example cyclne strike prbabilities). Sme extraplatin may be pssible frm the verificatin fr the mre mderate threshlds which will have statistically mre reliable results. Use f errr bars wuld help guide hw far it is reasnable t extraplate cnclusins. Case studies were cnsidered t be essential in assessing perfrmance fr extreme events. It was prpsed that case studies shuld include cases where ensembles predict nn-zer prbabilities f extreme events, but which d nt verify, t ensure a reliable prbabilistic predictin. This shuld be used t cmplement bjective statistical verificatin f less extreme events. It was nted that where extreme events d ccur, the bservatin is very ften clse t the extreme f the ensemble distributin - hence users shuld be strngly encuraged t pay attentin t lw prbability alerts. This has als implicatins fr decisin making (cst/lss analysis). Supprt frm high-reslutin mdels may strengthen the signal. Fr sme cases f severe events, experience suggests that the actual predictability may be higher than implied by the EPS. Since predictability is a prperty f a frecast system it is f interest t quantify hw the ensemble dispersin relates t the frecast accuracy f state-f-the-art deterministic mdels Aspects relevant fr the verificatin/applicatins f seasnal/mnthly frecasts The questin was raised whether an effrt shuld be made t unify verificatin tls used fr different applicatins (e.g. medium-range, mnthly, seasnal predictins). The limitatins f btaining statistically significant verificatin results fr seasnal and lnger predictins was discussed A member f the wrking grup nted that the repetitin f similar anmalies in subsequent years in the seasnal frecast may decrease trust f users in the useful signal in this prduct. It was suggested that the repetitin f similar anmalies might be due t the chice f climate and its lack f accunting fr the climate change trend. It was suggested that the climatlgy shuld include a trend; this aspect is relevant bth fr the verificatin and the cmmunicatin f seasnal frecasts.

4 Case studies: It was recmmended t identify a priri events where the predicted PDF deviates significantly frm the climatlgical PDF t avid a selective verificatin f nly the events that did verify (see discussin abve n extreme events). Discussin was held arund whether seasnal frecasts shuld be issued in areas r seasns with little r n skill. It is always imprtant t cmmunicate infrmatin n the level f skill. In cases f n skill it is preferable that n frecast shuld be issued, but the user culd be prvided with the climatlgy. Where there is sme skill and frecasts are issued, it was recmmended that frecasts shuld be issued cnsistently, including thse ccasins when there is n strng signal. In this case the frecast shuld revert t climatlgy tgether with the infrmatin that there is n strng r useful signal in the seasnal frecast. In summary, issue frecasts where there is skill but n signal, but NOT where there is signal but n skill. It was mentined that the cnsistency f subsequent frecasts can increase trust f users in the prduct (as an example the mnthly frecast was mentined). It was briefly discussed whether the cmmunicatin f seasnal predictins in the frm f anmalies with respect t the recent N years (with N being sme number up t 10) may be mre useful and/r easier t interpret fr sme users than anmalies with respect t a lnger term climate which includes perids beynd the persnal memry f the users. This was cnsidered particularly useful where the climate has undergne significant change in recent decades Benchmark(s) Sme fair cmparisn f EPS with prbability distributins built frm the High-Reslutin deterministic frecast shuld be examined. A simple example f such prbability distributins are Gaussian distributins centred n the deterministic frecast with a standard deviatin depending n frecast range Educatinal Aspects f Verificatin It was felt that mre educatin/explanatin f the meaning f scres and changes f the scres was required. A shrt guide t the meaning/usefulness f different verificatin measures and their apprpriateness fr different applicatins wuld be useful. When results f EPS verificatin are presented, a range f scres is required. The relative meaning f different scres must be explained as the apparent meaning f sme results may be cunterintuitive (a well knwn example fr deterministic frecasts is the reductin f RMS errr when the activity f the mdel is reduced and vice versa). 2. Applicatins f Ensemble Frecasts 2.1. Cmmunicatin f prbabilistic frecasts A WMO guide n the cmmunicatin f prbabilistic frecasts will be published sn. (It is nt published n web yet please cntact ken.mylne@metffice.gv.uk fr update) Use and limitatin f Ensemble Mean. There was sme disagreement n the value f the ensemble mean. Sme experience suggested that it was useful in intrducing use f ensembles int predminantly deterministic envirnments. Hwever, there was cnsiderable cncern abut the limitatins f the mean, in particular its inability ever t predict extreme events and the view was 4 Wrkshp n Ensemble Predictin, 7-9 Nvember 2007

5 als expressed that the ensemble mean shuld never be used. Where used it shuld always be accmpanied by prbabilities f extreme r high-impact events Hydrlgical applicatins The need fr re-frecasts was stressed nt nly fr calibratin but als fr verificatin (the frmer shuld be cvered by WG2) A limited sample size is an issue fr fld frecasting. Fr instance, the definitin f a climatlgical PDF fr streamflw frm a catchment pses a prblem. Thus, it is difficult t evaluate skill scres r bjectively cmpare different ensemble cnfiguratins There may be particular verificatin difficulties arising frm the effects f river cntrl measures and changing river prfiles which impact the cnsistency f the bservatins Recmmendatins cncerning design and testing f frecast system Based n predictability thery, medium-range and later range frecasts shuld be issued in prbabilistic frm. Therefre, the prime aim f ECMWF shuld be t prvide the tls t predict a reliable and sharp PDF f the atmspheric state rather than just a single deterministic high-reslutin frecast. The PDF shuld be based n all available infrmatin, i.e. the EPS and the high-reslutin deterministic frecast. Cnsequently, the EPS shuld be given the same level f attentin as the deterministic frecast system. Research n hw t best cmbine high-reslutin deterministic frecast and EPS shuld be cntinued at ECMWF. It was, hwever, als stated that a user/applicatin-specific cmbinatin may be superir t a generic cmbinatin. In such cases, the prductin f an ptimally cmbined PDF wuld fall int the respnsibility f member states r individual end-users. The EPS shuld therefre be affrded: Sufficient cmputer resurce t allw ptimal mdel perfrmance tuning f the frecast mdel/physical parameterisatins duratin f experimental suites It was felt that it shuld be further investigated whether there is a benefit f ging frm 62 vertical levels t 91 in the EPS (eg benefit f imprved stratspheric reslutin n medium-range frecast). 3. Summary f Recmmendatins fr ECMWF and ther EPS Prducers 3.1. Recmmendatins n Verificatin Cnfidence intervals shuld be prvided in all verificatin statistics. Sme research is required in the best techniques fr estimating cnfidence intervals, but there is already sme useful wrk in the literature. Methds fr accunting fr bservatin r analysis errr shuld be applied in calculatin f verificatin results. Several methds are prpsed in the literature. Methds shuld be used t minimise the impact f apparent false skill in verificatin results caused by differences in climatlgical frequency f events (base rate) at different lcatins. Several measures are required t examine different aspects f EPS perfrmance. A single summary measure, althugh desirable frm a management perspective, is inadequate.

6 These several measures shuld be used in a cnsistent fashin. The impact f prpsed system changes shuld be dcumented using the full set t give a balanced picture f the strengths and weaknesses f the change; selective use f particular scres is strngly discuraged. Reliability diagrams, tgether with sharpness diagrams f the crrespnding a psteriri calibrated prbabilities, prvide an effective way t cmmunicate prbabilistic perfrmance t nn-specialists. The Relative Operating Characteristic, while useful in research, shuld be used with care as it is difficult t explain and the ROC area summary measure can be very sensitive t hw it is calculated and is frequently mis-interpreted. Statistically significant verificatin f rare extreme events is impssible. Judicius use f cnfidence intervals may allw sme extraplatin f results frm less extreme events. Use f case studies fr extreme events shuld be balanced with cases where ensembles indicated a prbability f a severe event but nne verified. Fair cmparisn shuld be made between EPS frecasts and the high-reslutin deterministic frecast dressed with errr statistics. Sme investigatin is encuraged f whether the ensemble spread reflects the true predictability f extreme events frm the high-reslutin deterministic mdel. A simple summary guide f cmmnly used prbabilistic verificatin statistics and their interpretatin is required Recmmendatins n Applicatins The prime aim f ECMWF (r ther NWP systems) shuld be t prvide tls t predict a reliable and sharp prbability distributin f the future state f the atmsphere based n all available infrmatin the needs f EPS shuld therefre be affrded equal attentin as high-reslutin deterministic frecasts, and apprpriate levels f resurces. Seasnal frecast shuld be issued cnsistently, and where there is n signal this shuld be clearly cmmunicated. One shuld cnsider the pssibility f describing the trend in the reference climatlgy fr seasnal frecasts, t avid issuing frecasts which are dminated by the climate change trend. 6 Wrkshp n Ensemble Predictin, 7-9 Nvember 2007

Distributions, spatial statistics and a Bayesian perspective

Distributions, spatial statistics and a Bayesian perspective Distributins, spatial statistics and a Bayesian perspective Dug Nychka Natinal Center fr Atmspheric Research Distributins and densities Cnditinal distributins and Bayes Thm Bivariate nrmal Spatial statistics