THE MODEL EVALUATION TOOLS (MET): COMMUNITY TOOLS FOR FORECAST EVALUATION

Transcription

1 THE MODEL EVALUATION TOOLS (MET): COMMUNITY TOOLS FOR FORECAST EVALUATION Tressa L. Fowler John Halley Gotway, Barbara Brown, Randy Bullock, Paul Oldenburg, Anne Holmes, Eric Gilleland, and Tara Jensen July 2011

2 What is MET? Modular forecast evaluation tools Freely available Highly configurable Fully documented Support via the web, and user tutorials DTC development and support

3 What does MET do? Compares gridded NWP forecasts to point observations. Compares gridded NWP forecasts to gridded analyses.

4 What does MET do? Traditional statistics continuous and categorical Multi-category contingency tables Confidence intervals Statistics for probability forecasts Ensemble preprocessor and statistics Cloud verification capability Neighborhood methods MODE object based verification Wavelet decomposition Analysis tools to aggregate through time and space

5 How is it structured? Input Reformat Statistics Analysis Gridded GRIB Input: Observation Analyses Model Forecasts ASCII Point Obs Gen Poly Mask PCP Combine ASCII2NC NetCDF Mask Gridded NetCDF NetCDF Point Obs MODE Wavelet Stat Grid Stat Ensemble Stat ASCII NetCDF PS STAT ASCII NetCDF PS STAT ASCII NetCDF STAT ASCII NetCDF MODE Analysis Stat Analysis ASCII ASCII PrepBufr Point Obs PB2NC NetCDF Point Obs Point Stat STAT = optional

6 This is Hard! Large amount of data configurable options. GRIB, NetCDF, PREPBUFR formats. Map projections. Matching models to observations. Great Circle distances. Interpolation. Loads of statistics. Confidence Intervals. Complex methods.

7 Confidence Intervals (CI) MET provides two CI approaches Normal Bootstrap CIs are critical for appropriate and meaningful interpretation of verification results Ex: Regional comparisons

8 Spatial Verification Methods Meaningful evaluations of spatially-coherent fields (e.g., precipitation) Examples What is wrong with the forecast? At what scales does the forecast perform well? How does the forecast perform on attributes of interest to users? Methods included in MET Object-based: Method for Object-based Diagnostic Evaluation (MODE) Neighborhood: Example: Fractional Skill Score (FSS) Scale-separation: Casati s Intensity-Scale measure

9 HMT Ensemble: MODE Output 4 More Plots

10 Wavelet-Stat Tool Implements Intensity-Scale verification technique, Casati et al. (2004) Evaluate skill as a function of intensity and spatial scale of the error. Method: Threshold raw forecast and observation to create binary images. Decompose binary thresholded fields using wavelets (Haar as default). For each scale, compute the Mean Squared Error (MSE) and Intensity Skill Score (ISS). At what spatial scale is this forecast skillful? Difference (F-O) for precip > 0 mm Wavelet decomposition difference

11 Neighborhood Method: FSS observed forecast Intensity threshold exceeded where squares are blue Slide from Mittermaier

12 Neighborhood Method: Smoothing Smoothing Filters in MET Minimum, Maximum, Median, Mean, Nearest Neighbor, Least Squares, Distance Weighted Mean original median mean max min

13

14

15 What s new for METv3.0? Ensemble-Stat Tool Computes summary ensemble fields Verifies using ranked histograms Multi-dimensional contingency tables Additions to Point-Stat, Grid-Stat, and STAT-Analysis NetCDF output of the WRF-ARW pinterp tool Tools subdirectory for optional utilities World Wide Merged Cloud Analysis (WWMCA) Plotting tool: wwmca_plot Re-gridding tool: wwmca_regrid Plotting utility for point observations: plot_point_obs

16 Ensemble-Stat Tool Derives ensemble fields: Mean Standard deviation Ensemble mean +/- 1 standard deviation Minimum, maximum, and range Relative frequency Verification output: Computes rank histogram, CRPS, and IGN scores Outputs rank matched pairs

17 Ensemble-Stat: Derived Fields

18 HWT Ensemble: Traditional Scores

19 Where is MET heading? METViewer Database and display tool for the output of MET Includes web application for interactive plotting Development version used internally for DTC projects Upcoming release to friendly users Ongoing Future direction based on community input Regular software releases User support via met_help and tutorials DTC Verification Workshops For more information: