Level of Accuracy Goals and Model Performance Evaluation NW-Airquest Modeling Level of Accuracy Workgroup
Mike s Original Goals (2006) Level of Accuracy Required To develop an understanding of the degree of error that is acceptable for products that support policy or operational needs Ability to clarify consequence if accuracy not met Extent that the core U of W and WSU infrastructure can fulfill need What products are already fulfilling the needs? Who has unique accuracy needs?
PSCAA Preliminary LOA Needs (2006)
Wa DOE Preliminary LOA Needs (2006)
EPA Guidance on PM/O3/RH Attainment Modeling (2007) All O3/PM2.5/RH modeled attainment tests use model estimates in a relative sense Premise: models are better at predicting relative changes in concentrations than absolute concentrations Relative Response Factors (RRF) are calculated by taking the ratio of the model s future to current predictions of PM2.5 or ozone RRFs are calculated for ozone and for each PM2.5 component Note, forecasts are best used in a relative sense (as trends) also!
Proposed Goals Level of Accuracy Required To develop an understanding of the degree of error that is acceptable for products that support policy or operational needs Ability to clarify consequence if accuracy not met Extent that the core U of W and WSU infrastructure can fulfill need What products are already fulfilling the needs? Who has unique accuracy needs? To develop an understanding of model limitations and manage expectations in light of benchmarks set and attained in other areas. To set up a system to systematically track model performance, to understand when it degrades and identify problem areas.
Diagnostic Met Benchmarks Emery et al. (2001) proposed benchmarks Not Pass/Fail, but Targets Wind Speed Wind Direction Temperature Humidity RMSE 2 m/s Mean Bias ± 0.5 m/s ± 10º ± 0.5 K ±1 g/kg Gross Error 30º 2 K 2 g/kg Index of Agreement 0.6 0.8 0.6 Source: MM5 Modeling for BART in Alaska, Bart Brashers (PNWIS 2007) Geomatrix Consultants, 3500 188th Street SW Suite 600 Lynnwood, WA 98037 bbrashers@geomatrix geomatrix.com
PM Model Performance Goals and Criteria, Boylan, J. National RPO Modeling Meeting, Denver, CO 2005. Goals (close to best achievable) MFE <= 50% MFB <= +/- 30% Criteria (acceptable) MFE <= 75% MFB <= +/- 60% Minor species < 0.5 to 1 ug/m3 MFE <= 200% MFB <= +/- 200%
Regional Grid Models for PM2.5
Regional Grid Models for PM2.5 Conclusions: AIRPACT-3 performance is comparable to benchmarks proposed and performance achieved by regional diagnostic models. However we shouldn t expect 2ug/m3 or less than 10% accuracy any time soon. Nevertheless, use of model trends and forecasts a relative sense makes them useful.
Proposed Path Forward Complete Level of Accuracy table on Wiki forum, including: User needs/goals Current performance statistics State-of-the-art performance levels by others Consider routinely plotting model performance statistics on web page Facilitate day to day diagnosis of model performance and performance trends (e.g. 7 day rolling average?) Track performance improvement toward goals
Proposed Model Forecast Parameters to Track (at key locations where observations are available) Meteorological Temp Wind speed Wind Direction BL Height Sfc-inversion intensity BV Ventilation Categ. Vent Index Pollutant Species 8 hr peak Ozone 24hr PM2.5 24hr PSO4 24hr PNO3 24hr POC 24hr PEC Diesel & Benzene (toxics) Weekly N,S, Hg deposition
Benefits to Daily Performance Tracking Improvement in day-to-day understanding of model performance Forecast Analysis Improvement Longer term identification of performance problem areas and gradual improvement in all products