barbara.casati@ec.gc.ca Verificatio of cotiuous predictads Barbara Casati 9 Ja 007
Exploratory ethods: joit distributio Scatter-plot: plot of observatio versus forecast values Perfect forecast obs, poits should be o the 45 o diagoal Provides iforatio o: bias, outliers, error agitude, liear associatio, peculiar behaviors i extrees, isses ad false alars (lik to cotigecy table) Page
Exploratory ethods: argial distributio Quatile-quatile plots: OBS quatile versus the correspodig FRCS quatile Perfect: FCSTOBS, poits should be o the 45 o diagoal q 0.75 Page 3
Scatter-plot ad qq-plot: exaple Q: is there ay bias? Positive (over-forecast) or egative (uder-forecast)? Page 4
Scatter-plot ad qq-plot: exaple Describe the peculiar behavior of the low teperatures Page 5
Scatter-plot: exaple 3 Describe how the error varies as the teperatures grow outlier Page 6
Scatter-plot: exaple 4 Quatify the error Q: how ay forecasts exhibit a error larger tha 0 degrees? Q: How ay forecasts exhibit a error larger tha 5 degrees? Q: Is the forecast error due aily to a uderforecast or a overforecast? Page 7
Scatter-plot ad Cotigecy Table Does the forecast detect correctly teperatures above 8 degrees? Does the forecast detect correctly teperatures below 0 degrees? Page 8
Scatter-plot ad cotigecy table: exaple 5 Aalysis of the extree behavior Q: How does the forecast hadle the teperatures above 0 degrees? How ay isses? How ay False Alars? Is there a uder- or over-forecast of teperatures larger tha 0 degrees? Q: How does the forecast hadle the teperatures below -0 degrees? How ay isses? Are there ore issed cold evets or false alars cold evets? How does the forecast iiu teperature copare with the observed iiu teperature? Page 9
Exploratory ethods: argial distributios Visual copariso: Histogras, box-plots, Suary statistics: Locatio: ea edia X q 0.5 i x i Spread: st dev i ( x X ) i MEAN MEDIAN STDEV IQR Iter Quartile Rage IQR q 0.75 q 0.5 OBS FRCS 0.7 8.6 0.5 7.00 5.8 5.99 8.5 9.75 Page 0
Exploratory ethods: coditioal distributios Coditioal histogra ad coditioal box-plot Page
Exploratory ethods: coditioal qq-plot Page
Exploratory ethods: class activity Cosider the data set of teperatures provided by Marti Beko (beko.csv). Select a locatio ad for the correspodig observatio ad forecasts:. Produce the scatter-plot ad quatile-quatile plot: aalyze visually if there is ay bias, outliers, peculiar behaviors at the extrees,. Produce the coditioal quatile plot: are there sufficiet data to produce it? is it coheret with the scatter-plot? 3. Produce side to side the box-plots of forecast ad observatio: how do the locatio ad spread of the argial distributios copare? 4. Evaluate ea, edia, stadard deviatio ad Iter-Quartile- Rage: do the statistics cofir what you deduced fro lookig at the box-plot, scatter-plot ad quatile-quatile plot? Page 3
Cotiuous scores: liear bias liear bias ME i ( y x ) Y X i i Attribute: easures the bias Mea Error average of the errors differece betwee the eas It idicates the average directio of error: positive bias idicates over-forecast, egative bias idicates uder-forecast (yforecast, xobservatio) Does ot idicate the agitude of the error (positive ad egative error ca cacel outs) Bias correctio: isses (false alars) iprove at the expeses of false alars (isses). Q: If I correct the bias i a over-forecast, do false alars grow or decrease? Ad the isses? Good practice rules: saple used for evaluatig bias correctio should be cosistet with saple corrected (e.g. witer separated by suer); for fair validatio, cross validatio should be adopted for bias corrected forecasts Page 4
Cotiuous scores: MAE MAE i y i x i Attribute: easures accuracy Average of the agitude of the errors Liear score each error has sae weight It does ot idicates the directio of the error, just the agitude Q: If the ME is siilar to the MAE, perforig the bias correctio is safe, if MAE >> ME perforig the bias correctio is dagerous: why? A: if MAE >>ME it eas that positive ad egative errors cacel out i the bias evaluatio Page 5
Cotiuous scores: MSE MSE ( y ) i x i i Attribute: easures accuracy Average of the squares of the errors: it easures the agitude of the error, weighted o the squares of the errors it does ot idicate the directio of the error Quadratic rule, therefore large weight o large errors: good if you wish to pealize large error sesitive to large values (e.g. precipitatio) ad outliers; sesitive to large variace (high resolutio odels); ecourage coservative forecasts (e.g. cliatology) Page 6
Cotiuous scores: RMSE RMSE MSE ( y i x i ) i Attribute: easures accuracy RMSE is the squared root of the MSE: easures the agitude of the error retaiig the variable uit (e.g. O C) Siilar properties of MSE: it does ot idicate the directio the error; it is defied with a quadratic rule sesitive to large values, etc. NOTE: RMSE is always larger or equal tha the MAE Q: if I verify two sets of data ad i oe I fid RMSE MAE, i the other I fid RMSE MAE, which set is ore likely to have large outliers? Which set has larger variace? Page 7
Cotiuous scores: liear correlatio r XY ( y y)( x x) i i i Y X ( y y) ( x x) i i i i cov( Y, X ) s s Attribute: easures associatio Measures liear associatio betwee forecast ad observatio Y ad X rescaled (o-diesioal) covariace: rages i [-,] It is ot sesitive to the bias The correlatio coefficiet aloe does ot provide iforatio o the icliatio of the regressio lie (it says oly is it is positively or egatively tilted); observatio ad forecast variaces are eeded; the slope coefficiet of the regressio lie is give by b (s X /s Y )r XY Not Robust better if data are orally distributed Not resistat sesitive to large values ad outliers Page 8
MSE ad bias correctio MSE ( Y X ) + s Y + s X s Y s X r XY MSE ME + var( Y X ) Q: if I correct the forecast fro the bias, I will obtai a saller MSE. If I correct the forecast by usig a cliatology (differet fro the saple cliatology), will I obtai a MSE saller or larger tha the oe I obtaied for the forecast with the bias corrected? MSE MSE bias cli ( Y ( Y X ) X ) ( Y c X ) MSE var( Y cme X ) + c MSE ME ( ME c) 0 MSE cli MSE bias Page 9
Cotiuous scores: class activity 5. Evaluate ME, MAE, MSE, RMSE ad correlatio coefficiets: Copare MAE ad ME, is it safe to perfor a bias correctio? Copare MAE ad RMSE: are there large values i the data? Is the data variability very high? 6. Substitute soe values of your data with large (outliers) values. Reevaluate the suary statistics ad cotiuous scores. Which scores are the ost affected oes? 7. Add to your forecast values soe fixed quatities to itroduce differet biases: does the correlatio chage? Ad the regressio lie slope? Multiply your observatios for a costat factor: does the correlatio chage? How does the observatio stadard deviatio ad the regressio lie slope chage? Multiply ow the forecast values for a costat factor: how does this affect correlatio, forecast stadard deviatio ad regressio lie slope? 8. Perfor a bias correctio o your data. How does this affect ME, MSE ad correlatio? The, chage the variace of forecast ad observatio by ultiplyig their values with soe costat factors. How does this affect the ME, MSE ad correlatio? Page 0
Other suggested activities (advaced) Separate your data to siulate a cliatology ad a saple data set. Evaluate the MSE for the forecast corrected with the saple bias ad the cliatology: verify that MSE cli MSE bias Deduce algebraically the relatio betwee MSE ad correlatio if bias is corrected ad forecast rescaled by s X /s Y: Does the MSE deped o the observatio variace? What happe if I rescale both forecast ad observatios with their correspodig stadard deviatios? Sesitivity of scores to spatial forecast resolutio: evaluate MSE for your spatial forecast, observatio ad forecast variace, ME ad correlatio. The sooth the forecast ad observatio (e.g. averagig earby x pixels) ad re-copute the statistics. Which scores are ostly affected? Page
Cotiuous skill scores: MAE skill score SS MAE MAE MAE ref MAE perf MAE ref MAE MAE ref Attribute: easures skill Skill score: easure the forecast accuracy with respect to the accuracy of a referece forecast: positive values skill; egative values o skill Differece betwee the score ad a referece forecast score, oralized by the score obtaied for a perfect forecast ius the referece forecast score (for perfect forecasts MAE0) Referece forecasts: persistece: appropriate whe tie-correlatio > 0.5 saple cliatology: iforatio oly a posteriori actual cliatology: iforatio a priori Page
Cotiuous skill scores: MSE skill score SS MSE MSE MSE ref MSE perf MSE ref MSE MSE ref Attribute: easures skill Sae defiitio ad properties as the MAE skill score: easure accuracy with respect to referece forecast, positive values skill; egative values o skill Sesitive to saple size (for stability) ad saple cliatology (e.g. extrees): eeds large saples Reductio of Variace: MSE skill score with respect to cliatology. If saple cliatology is cosidered: Y X liear correlatio ; MSE s Y Y X MSEcli sx ad RV rxy rxy sx sx sx bias reliability: regressio lie slope coeff b(s X /s Y )r XY Page 3
Suggested activities: Reductio of Variace Show atheatically that the Reductio of Variace evaluated with respect to the saple cliatology forecast is always saller tha the oe evaluated by usig the actual cliatology as referece forecasts? Copute the Reductio of Variace for you forecast with respect to the saple cliatology, ad copute each of its copoets (liear associatio, reliability ad bias) as i the give equatio. Modify your forecast ad observatio values i order to chage, oce at a tie, each ter: aalyze their effect o the RV. The, odify the forecast ad observatio i order to chage two (or all) ters at the sae tie, but aitaiig RV costat: aalyze of how the ters balace each other Page 4
Cotiuous skill scores: good practice rules Use sae cliatology for the copariso of differet odels Whe evaluatig the Reductio of Variace, saple cliatology gives always worse skill score tha log-ter cliatology: ask always which cliatology is used to evaluate the skill If the cliatology is calculated pullig together data fro ay differet statios ad ties of the year, the skill score will be better tha if a differet cliatology for each statio ad oth of the year are used. I the forer case the odel gets credit fro forecastig correctly seasoal treds ad specific locatios cliatologies; i the latter case the specific topographic effects ad log-ter treds are reoved ad the forecast discriiatig capability is better evaluated. Choose the appropriate cliatology for fulfillig your verificatio purposes Persistece forecast: use sae tie of the day to avoid diural cycle effects Page 5
Cotiuous scores: aoaly correlatio y' x' y x c c Forecast ad observatio aoalies to evaluate forecast quality ot accoutig for correct forecast of cliatology (e.g. drive by topography) AC AC cet uc ap ap ( y' y' )( x' x' ) ( y' y' ) ( x' x' ) ap ap ap ( y c )( x c ) ( y c ) ( x c ) ap ap ap Cetered ad u-cetered AC for weather variables defied over a spatial doai: c is the cliatology at the grid-poit, over-bar deotes averagig over the field ( y' )( x' ) ( y' ) ( x' ) ap Page 6
Cotiuous Scores of Raks Cotiuous scores sesitive to large values or o robust (e.g. MSE or correlatio coefficiet) are soe-ties evaluated by usig the raks of the variable, rather tha its actual values Tep o C 7.4.7 4. 3. 9.8 5.5 4.6.3 rak 8 5 4 7 6 3 The value-to-rak trasforatio: diiish effects due to large values trasfor argial distributio to a Uifor distributio reove bias Rak correlatio is the ost used of these statistics Page 7
Liear Error i Probability Space LEPS i F X ( y ) F ( x ) i X i The LEPS is a MAE evaluated by usig the cuulative frequecies of the observatio LEPS pealizes less error i the tail of the distributio ad ore errors i the cetre of the distributio q 0.75 MAE ad LEPS are iiized by the edia correctio Page 8
Suggested Activities: raks ad LEPS Evaluate the correlatio coefficiet ad rak correlatio coefficiet for your data. Substitute soe values with large (outliers) values ad recalculate the scores. Which oe is ostly affected? Cosider a precipitatio data set: is it orally distributed? Produce the observatio-forecast scatter-plot ad copute the MAE, MSE ad correlatio coefficiet for. the actual precipitatio values. the raks of the values 3. the logarith of the values, after addig to all values 4. the th root of the values (,3,4, ) 5. the forecast ad obs cuulative probabilities of the values Copare the effects of the differet trasforatios If you recalibrate the forecast, so that F X F Y, ad evaluate the MAE after perforig the 5 th trasforatio above, which score do you calculate? Page 9
barbara.casati@ec.gc.ca Refereces: Jolliffe ad Stepheso (003): Forecast Verificatio: a practitioer s guide, Wiley & Sos, 40 pp. Wilks (005): Statistical Methods i Atospheric Sciece, Acadeic press, 467 pp. Staski, Burrows, Wilso (989) Survey of Coo Verificatio Methods i Meteorology http://www.euetcal.org.uk/euetcal/verificatio/www/egl ish/courses/sgcrs/idex.ht http://www.bo.gov.au/brc/wefor/staff/eee/verif/verif_web _page.htl Page 30