Verification as Diagnosis: Performance Diagrams for Numerical Forecasts of Severe Precipitation in California during the HMT Winter Exercises

Transcription

1 Verification as Diagnosis: Performance Diagrams for Numerical Forecasts of Severe Precipitation in California during the HMT Winter Exercises Edward Tollerud 1,2, Barbara Brown 1,3, Tara Jensen 1,3, John Halley Gotway 1,3,, Paul Oldenburg 1,3, Stanislav Stoytchev 1,2, Brian Etherton 1,3, and Linda Wharton 1,2 1 Developmental Testbed Center (DTC) 2 Earth Systems Research Laboratory, NOAA 3 RAL/NCAR (National Center for Atmospheric Research)

2 Verifier s Creed Not for us to question why; Just for us to test and pry (But remember that the next two lines are these:) Into the valley of death Rode the six hundred (Apologies to Alfred Lord Tennyson and the Light Brigade) 2

3 High-Level HMT/DTC Project Goals Evaluation and Diagnoses for HMT-West Ensemble Forecasts of Extreme Precipitation Events (e.g., real-time web product for HMT) Motivate, Develop, and Evaluate new verification strategies (MET and MODE in particular; e.g., roc, auroc, rank histogram, performance diagram, ) Inter-compare Forecasting Systems in high-precipitation scenarios, including research and EMC operational models Assess Impacts of Verification dataset selection (analyses, point obs, etc.) Assess Model and Verification Configuration Options (Resolution, Initialization, Domain, Event Selection, etc.) 3

4 ESRL and HMT Ensemble Modeling System WRF model 9-member ensemble Outer domain 9km; Nested domain 3 km Hybrid members: Multi physics packages, two model cores, and different GFS initial conditions Model runs to 5 day lead time DTC built demonstration real-time web display Heavy focus on QPF 4

5 Performance Diagrams Take advantage of relationships among scores to show multiple scores at one time Best Only need to plot POD and 1- FAR (Success Ratio) Success ratio = 1 - FAR From Roberts et al. 2011; after Roebber 2009 and C. Wilson 2008 Equal lines of CSI Equal lines of Bias 5

6 Ex: Relationships among scores CSI is a nonlinear function of POD and FAR CSI depends on base rate (event frequency) and Bias 1 CSI = POD 1- FAR CSI Bias POD = 1 - FAR Very different combinations of FAR and POD lead to the same CSI value 6

7 The GSD Ensemble Mean Performance Application to the GSD QPF verification from Color clusters of lead times at several thresholds Limits of useable leads Typical patterning on a performance diagram 72h 72h 72h 6h 6h 6h 6h 72h 7

8 NMM vs. ARW GSD Ensemble Membership Dynamic Core Impact? Blue: ARW Core Members Red: NMM Core members 6h lead, 0.5 in threshold 8

9 NMM vs. ARW GSD Ensemble Membership Impacts of Cores? Impact of Initialization? Gold 0.1 in Red 0.5 in Blue 1.0 in Solid ARW Dashed - NMM 9

10 Inter-Model Comparisons: Operational and HMT 0.5 inch/6h Black NNMB Red HMT Ensemble Blue GFS Brown NAM Green HRRR 10

11 Inter-Model Comparisons: Operational and HMT Non-severe events in the HMT 0.5 inch/6h Select Episodes with Care!! 11

12 Domain and Resolution Impacts on Verification Scores Comparison of HMT nest and full domain 1 2 1: Black circle High-res Nest 2: Square Low-res Nest 3: Triangle Low-res large domain Gold 0.1 in Red 0.5 in Blue 1.0 in 2 12

13 Verification Dataset Impacts: Choosing Your Analysis StageIV vs CCPA 13

14 Verification Dataset Impacts: StageIV analyses vs. Gage Observations 14

15 Impact of Choice of Microphysics Parameterization Red: Ferrier Blue: Schultz Green: Thompson 15

16 Valuable Insights and Lessons Learned: Some Gained, Some in Need of Confirmation Resolution improves performance Ensemble means are generally superior to an individual member Event Selection: Pick your scenario with care Model Core Impacts?? Microphysical Impacts?? Initialization impacts?? Choices of verification datasets matters: StageIV vs CCPA, gages vs grids, 6h vs 24h 16