Spatial verification of NWP model fields. Beth Ebert BMRC, Australia

Transcription

1 Spatial verification of NWP model fields Beth Ebert BMRC, Australia WRF Verification Toolkit Workshop, Boulder, February 2007

2 New approaches are needed to quantitatively evaluate high resolution model output WRF Verification Toolkit Workshop, Boulder, February

3 What modelers want Diagnostic information What scales are well represented by the model? How realistic are forecast features / structures? How realistic are distributions of intensities / values? What are the sources of error? How can I improve the model? How can I score? It's not so easy! WRF Verification Toolkit Workshop, Boulder, February

4 Spatial forecasts Weather variables defined over spatial domains have coherent spatial structure and features (intrinsic spatial correlation) Spatial verification techniques aim to: account for field spatial structure provide information on error in physical terms account for uncertainties in timing and location WRF Verification Toolkit Workshop, Boulder, February

5 Recent research in spatial verification Scale decomposition methods measure scale-dependent error Fuzzy (neighborhood) verification methods give credit to "close" forecasts Object-oriented methods evaluate attributes of identifiable features Field verification evaluate phase errors WRF Verification Toolkit Workshop, Boulder, February

6 Scale decomposition methods scale-dependent error WRF Verification Toolkit Workshop, Boulder, February

7 Wavelet scale components Briggs and Levine (1997) ECMWF Analysis 36-h Forecast (CCM-2) 500 mb GZ, 9 Dec 1992, 12:00 UTC, N. America WRF Verification Toolkit Workshop, Boulder, February

8 Intensity-scale verification technique Casati et al. (2004) Measures the skill as function of intensity and spatial scale of the error Intensity: threshold Categorical approach Scale: 2D Wavelets decomposition of binary images For each threshold and scale: skill score associated to the MSE of binary images = Heidke Skill Score Intense storm displaced threshold = 1mm/h scale (km) /16 ¼ ½ threshold (mm/h) Skill

9 Multiscale statistical properties Harris et al. (2001) Does a model produce the observed precipitation scaledependent variability, i.e. does it look like real rain? Compare multi-scale statistics for model and radar data Power spectrum Structure function Moment scaling WRF Verification Toolkit Workshop, Boulder, February

10 Fuzzy (multi-scale) verification methods give credit to "close" forecasts WRF Verification Toolkit Workshop, Boulder, February

11 "Fuzzy" verification methods Don't require an exact match between forecasts and observations Unpredictable scales Uncertainty in observations Why Look is it in called a space "fuzzy"? / time neighborhood around the point of interest Squint your eyes! Evaluate using categorical, observation continuous, probabilistic forecast scores / methods t - 1 t t + 1 Frequency Forecast value WRF Verification Toolkit Workshop, Boulder, February

12 "Fuzzy" verification methods Treatment of forecast data within a window: Mean value (upscaling) Occurrence of event* somewhere in window Frequency of event in window probability Distribution of values within window May apply to observations as well as forecasts (neighborhood observation-neighborhood forecast approach) * Event defined here as a value exceeding a given threshold, for example, rain exceeding 1 mm/hr WRF Verification Toolkit Workshop, Boulder, February

13 Spatial multi-event contingency table Atger (2001) Forecasters mentally "calibrate" the deterministic forecast according to how close the forecast is to the place / time / magnitude of interest. Vary decision thresholds: magnitude (ex: 1 mm h -1 to 20 mm h -1 ) distance from point of interest (ex: within 10 km,..., within 100 km) timing (ex: within 1 h,..., within 12 h) Very close high probability Not very close low probability anything else that may be important in interpreting the forecast ROC Sydney "high probability single threshold of some heavy rain near Sydney", not "62 mm of rain will fall in Sydney" EPS WRF Verification Toolkit Workshop, Boulder, February

14 Fractions skill score Roberts (2005) We want to know How forecast skill varies with neighbourhood size. The smallest neighbourhood size that can be can be used to give sufficiently accurate forecasts. Does higher resolution provide more accurate forecasts on scales of interest (e.g. river catchments) Compare forecast fractions with observed fractions (radar) in a probabilistic way over different sized neighbourhoods FSS N! i = 1 = 1" N 1 N 1 N ( P fcst " P 1 N obs N 2! Pfcst +! i = 1 i = 1 ) P 2 2 obs observed forecast WRF Verification Toolkit Workshop, Boulder, February

15 Fractions skill score Roberts (2005) WRF Verification Toolkit Workshop, Boulder, February

16 Decision models Fuzzy method Matching strategy* Decision model for useful forecast Upscaling (Zepeda -Arce et al. 2000; Weygandt et al. 2004) NO-NF Resembles obs when averaged to coarser scales Minimum coverage (Damrath 2004) NO-NF Predicts event over minimum fraction o f region Fuzzy logic (Damrath 2004), joint probability (Ebert 2002) NO-NF More correct than incorrect Fractions skill score (Roberts 2005) NO-NF Similar frequency of forecast and observed events Area -related RMSE (Rezacova et al. 2006) NO-NF Similar int ensity distribution as observed Pragmatic (Theis et al. 2005) SO -NF Can distinguish events and non -events CSRR (Germann and Zawadzki 2004) SO -NF High probability of matching observed value Multi -event contingency table (Atger 2001) SO -NF Predicts at lea st one event close to observed event Practically perfect hindcast (Brooks et al. 1998) SO -NF Resembles forecast based on perfect knowledge of observations *NO-NF = neighborhood observation-neighborhood forecast, SO-NF = single observation-neighborhood forecast WRF Verification Toolkit Workshop, Boulder, February

17 Fuzzy verification framework good performance poor performance WRF Verification Toolkit Workshop, Boulder, February

18 Object-oriented methods evaluate attributes of features WRF Verification Toolkit Workshop, Boulder, February

19 Entity-based approach (CRA) Ebert and McBride (2000) Define entities using threshold (Contiguous Rain Areas) Horizontally translate the forecast until a pattern matching criterion is met: minimum total squared error between forecast and observations maximum correlation maximum overlap The displacement is the vector difference between the original and final locations of the forecast. Observed Forecast WRF Verification Toolkit Workshop, Boulder, February

20 CRA information Gives information on: Location error RMSE and correlation before and after shift Attributes of forecast and observed entities Error components displacement volume pattern WRF Verification Toolkit Workshop, Boulder, February

21 MODE* *Method for Object-based Diagnostic Evaluation Davis et al. (2006) Two parameters: 1. Convolution radius 2. Threshold WRF Verification Toolkit Workshop, Boulder, February

22 MODE object matching/merging Compare attributes: - centroid location - intensity distribution - area - orientation - etc. 24h forecast of 1h rainfall on 1 June 2005 When objects not matched: - false alarms - missed events - rain volume - etc. WRF Verification Toolkit Workshop, Boulder, February

23 MODE methodology Identification Measure Attributes Merging Matching Comparison Summarize Convolution threshold process Fuzzy Logic Approach Compare forecast and observed attributes Merge single objects into composite objects Compute interest values Identify matched pairs Accumulate and examine comparisons across many cases WRF Verification Toolkit Workshop, Boulder, February

24 Cluster analysis approach Marzban and Sandgathe (2006) Goal: Assess the agreement between fields using clusters identified using agglomerative hierarchical cluster analysis (CA) Optimize clusters (and numbers of clusters) based on Binary images (x-y optimization) Magnitude images (x-y-p optimization) Compute Euclidean distance between clusters in forecast and observed fields (in x-y and x-y-p space) MM5 precipitation forecasts 8 clusters identified in x-y-p space WRF Verification Toolkit Workshop, Boulder, February

25 Cluster analysis example Stage IV Error = average distance between matched clusters in x-y-p space COAMPS log e error WRF Verification Toolkit Workshop, Boulder, February

26 Composite approach Nachamkin (2004) Goal: Characterize distributions of errors from both a forecast and observation perspective Procedure: Identify events of interest in the forecasts Define a kernel and collect coordinated samples Compare forecast PDF to observed PDF Repeat process for observed events Forecast Observation x Event center WRF Verification Toolkit Workshop, Boulder, February

27 Composite example Compare kernel grid-averaged values Average rain (mm) given an event was predicted Average rain (mm) given an event was observed FCST-shade OBS-contour WRF Verification Toolkit Workshop, Boulder, February

28 Field verification evaluate phase errors WRF Verification Toolkit Workshop, Boulder, February

29 Feature calibration and alignment (Hoffman et al., 1995; Nehrkorn et al., 2003) Original forecast X f (r) Forecast adjustment 500 mb analysis X v (r) Error decomposition e = X f (r) - X v (r) where X f (r) is the forecast, X v (r) is the verifying analysis, and r is the position. Adjusted forecast X a (r) Residual error e r e = e p + e b + e r where e p = X f (r) - X d (r) phase error e b = X d (r) - X a (r) local bias error e r = X a (r) - X v (r) residual error WRF Verification Toolkit Workshop, Boulder, February

30 Forecast quality measure (FQM) Keil and Craig (2007) Combines distance measure and intensity difference measure Pyramidal image matching (optical flow) to get vector displacement field edistance Unmatched features are penalized for their intensity errors eintensity Forecast quality measure FQM = 1 max(edistance, eintensity )! A A satellite WRF Verification Toolkit Workshop, Boulder, February 2007 orig.model morphed model 30

31 Conclusions What method should you use for model verification? Depends what question(s) you would like to address Many spatial verification approaches Scale decomposition scale-dependent error Fuzzy (neighborhood) credit for "close" forecasts Object-oriented attributes of features Field verification phase error WRF Verification Toolkit Workshop, Boulder, February