CALPUFF Performance Evaluations

Transcription

1 CALPUFF Performance Evaluations Bret Anderson, USEPA R7/OAQPS Roger W. Brode, USEPA/OAQPS/AQMG Jaime Julian, USEPA R5 Adina Wiley, USEPA R6 Herman Wong, USEPA R10

2 Original Scope of Project Update CALMET/CALPUFF performance evaluations originally conducted from for Version 5.8. CAPTEX (1983) Great Plains (1980) Savannah River (1975) Use evaluations to update CALPUFF implementation guidance to include IWAQM Phase II recommendations.

3 Issues Regarding Previous Evaluations Previous evaluations of CALPUFF limited in scope. LRT evaluations largely focused upon performance metrics similar to near-field dispersion models such as ISC or AERMOD. Evaluations were more subjective without a focus upon performance evaluation methods used for LRT models. Near-field evaluation(s) only evaluated CALPUFF peroformance using CTDM profile data, which is not a method we use to capture complex winds, was necessary to develop full 3-D winds for this purpose. Necessary to develop a new methodology for evaluating LRT models.

4 Near-Field Evaluation Procedure Various methods, including graphical and statistical measures, are used for evaluating model performance for near-field regulatory applications, such as AERMOD For regulatory design concentrations, evaluation procedures are focused on predicting the peak of the concentration distribution, unpaired in time and space Graphical methods include quantile-quantile (Q-Q) plots in which quantiles of one distribution (e.g., observed concentrations) are plotted against quantiles of another distribution (e.g., estimated concentrations). The observed and predicted concentration distributions are ranked irrespective of time and space, and plotted as ranked pairs. Statistical measures for regulatory model evaluations have historically been based on the Cox-Tikvart methodology, which uses a Robust Highest Concentration as the basis for comparing observed and predicted concentrations The Cox-Tikvert methods uses the Fractional Bias (FB) as the statistical measure of agreement between the observed and predicted concentration; The FB is bounded between -2 and +2, with a value of 0 (zero) indicating perfect agreement. An absolute value of less than for FB indicates agreement within a factor of 2

5 Robust Highest Concentration The test statistic that is used in the comparisons of model performances, is a robust estimate of the highest concentration (RHC) using the largest concentrations within a given data category. The robust estimate is based on a tail exponential fit to the upper end of the distribution and is computed as follows: RHC= X(N)+ X - X(N) ln 3N -1 2, where: M = number of values exceeding a threshold value; Mo = number of values used to characterize the "upper end" of the distribution; N = minimum of M, Mo; X = average of the N-1 largest values; and X(N) = Nth largest value.

6 Composite Performance Measure A composite performance measure (CPM) is a weighted linear combination of the individual fractional bias components. A CPM is computed for each model and data base. Within the operational evaluation component, each of the results for the various averaging periods receives equal weight. For the scientific component, each of the results for the various diagnostic conditions receives equal weight. Because the operational evaluation component is deemed to be the more important of the two, it receives twice the weight of the scientific component. The algebraic expression for the composite performance measure is: 1 2 (AFB ) 3 +(AFB ) 24 CPM = AFB r,s , where: (AFB)r,s (AFB)3 (AFB)24 = Absolute Fractional Bias for diagnostic condition r at station s; = Absolute Fractional Bias for 3-hour averages; and = Absolute Fractional Bias for 24-hour averages.

7 Near-Field Complex Terrain Lovett, NY SO 2 Study (1988) EPA OAQPS Westvaco, MD SO 2 Study (1980) EPA Region 5 Clifty Creek SO 2 Study (1983) EPA Region 6

8 Lovett Near-Field Complex Terrain CALMET 3-D Windfields Developed 3 Level Tower (10m, 50m, 100m) simulated as 3 surface stations with corresponding anemometer heights Upper air data from Albany, NY 125 meter grid spacing using SRTM-1 (terrain) and NLCD92 (landuse). Various CALPUFF configurations tested P-G AERMOD turbulence Plume Half-Height Adjustment CALPUFF Strain Based Adjustment AERMOD surface and profile data

9 Models Evaluated CALPUFF1: P-G Dispersion/Plume Half Height Adjustment CALPUFF2: AERMOD Turbulence/Plume Half Height Adjustment CALPUFF3: P-G Dispersion/CALPUFF Strain Based Adjustment CALPUFF4: AERMOD Turbulence/CALPUFF Strain Based Adjustment CALPUFF5: AERMOD Profile Data/Plume Half Height Adjustment AERMOD

10 Robust Highest Concentration 3-Hour MODEL RHC FB 24-Hour MODEL RHC FB OBSERVED OBSERVED CALPUFF CALPUFF CALPUFF CALPUFF E-02 CALPUFF CALPUFF CALPUFF CALPUFF CALPUFF E-02 CALPUFF AERMOD E-02 AERMOD E-03

11 Composite Performance Measure (CPM) Model CPM +/- C.I. +/- C.I. 90% 95% CALPUFF CALPUFF CALPUFF CALPUFF CALPUFF AERMOD E

12 Model Comparison Measure (MCM) Model - Model MCM +/- C.I. 90% CALPUFF1-CALPUFF E CALPUFF1-CALPUFF CALPUFF1-CALPUFF CALPUFF1-CALPUFF CALPUFF1-AERMOD CALPUFF2-CALPUFF CALPUFF2-CALPUFF CALPUFF2-CALPUFF CALPUFF2-AERMOD CALPUFF3-CALPUFF CALPUFF3-CALPUFF E CALPUFF3-AERMOD E CALPUFF4-CALPUFF CALPUFF4-AERMOD CALPUFF5-AERMOD E

13 3-Hour Q-Q Plot - Lovett

14 24-Hour Q-Q Plot - Lovett

15

16

17

18

19 Evaluation Paradigm for Long Range Transport Models LRT models play a unique role in air quality modeling. This class of models plays several roles. Emergency response modeling Class I increments and visibility Requires additional level of skill to reflect time and space considerations of LRT model use Statistical measures should examine spatiotemporal pairing ability of LRT models. Original CALPUFF evaluations largely qualitative with limited (if any) focus on spatiotemporal pairing

20 Evaluation Goal Develop meteorological and tracer databases for evaluation of long range transport models. Develop a consistent and scientifically objective method for evaluating long range transport (LRT) models used by the EPA. Promote the best scientific application of models based upon lessons learned from evaluations and reflect this in EPA modeling guidance.

21 Statistical Measures Pearson s Correlation Coefficient PCC i M i i M i M M 2 P i i P P i P 2 Normalized Mean Square Error NMSE 1 N i P i M PM i 2 Fractional Bias FB 2B P M

22 Statistical Measures Figure of Merit in Space Kolmogorov-Smirnov Parameter Model Rank RANK FMS KS N 100 N p p 0 0 N m N m 0 0 MAXC Mk C P k R 2 1 FB FMS KS 100

23 Other Measures Percentage of observations within factor of two of observed (FA2) Percentage of observations within factor of five of observed (FA5) Factor of Exceedance measure of overprediction.

24 Mesoscale Tracer Experiments Great Plains Tracer Experiment (OKC80) 1980 (EPA OAQPS and EPA Region 6) Savannah River Laboratory Tracer Experiment (SRL) 1975 (EPA OAQPS and EPA Region 10) Cross-Appalachian Tracer Experiment (CAPTEX) 1983 (EPA OAQPS) European Tracer Experiment (ETEX) 1994 (EPA OAQPS)

25 Great Plains Mesoscale Tracer Experiment Regional-scale experiment designed to test new atmospheric tracer system using perfluorocarbon tracers (PFT). Perfluorodimethylcyclohexane (PDCH) Perfluoromonomethylcyclohexane (PMCH) Measurements on arcs at distance of 100 km and 600 km. 2 PFT Releases, July 8 and July

26 Synoptic Meteorology As noted in Ferber et al (1981), the sampling began on the 600 kilometer arc around 0200 LST on July 9, Forecast trajectories based upon the 12Z July 8, 1980 rawindsonde release time indicated that the tracer release starting at 1300 LST July 8, 1980 would arrive at the 600 kilometer arc at approximately 0700 LST on July 9, Sampling was to commence five hours prior the forecasted arrival of the tracer. When sampling commenced at 0200 LST July 9, 1980, peak concentrations had already arrived when sampling had commenced. The sampling along the 600 kilometer arc likely missed some of the tracer material since the peak concentrations were observed during the first 3-hour averaging period when sampling commenced. A low-level nocturnal jet had formed during the evening and early morning hours of July 8-9, 1980, transporting the tracer material faster to the north and west than was anticipated by results of the 12Z forecast trajectories. The monitored plume extended from approximately Firth, Nebraska to Hamilton, MO. By approximately 1700 LST July 9, 1980, monitored values of the tracer material dropped to background concentrations.strong nocturnal low-level jet developed overnight on July 8-9, 1980, transporting

27 Synoptic Meteorology July 8-9, 1980

28 Model Experiment Design 500 Model Receptor Arcs Two Dispersion Models 1. CALPUFF: CALMET Meteorology Observation only Hybrid (MM5 + Obs) NOOBS=1 (Obs surface, MM5 aloft) NOOBS=2 (No Observations, only MM5) MM5CALPUFF Meteorology All Dispersion Options P-G Turbulence AERMOD Turbulence CALPUFF Turbulence Puff-Splitting on 600 km simulation, none for 100 km 100 km and 600 km arcs of receptors, 0.25 increments, 361 total receptors for each. 2 Domains: 100km, 600 km arcs 20 km CALMET for 600 km 4 km CALMET for 100 km 2. FLEXPART Lagrangian Particle Model

29 MM5 Configuration MM5 Version nested domains 108 km 36 km 12 km 34 vertical layers ICBC: NCEP/NCAR Reanalysis Data available every 6 hours at 2.5 x 2.5 Physics: Kain-Fritsch II Cumulus ETA PBL NOAH LSM RRTM Radiation Simple Ice Microphysics

30 CALPUFF km Arc

31 Observed Maximum Omax (ppt) actual maximum CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Great Plains, July 8, 100 km, CALMET Winds Great Plains, July 8, 100 km, MM5 Winds Great Plains, July 8, 600 km CALMET Obs Only Winds Great Plains, July 8, 600 km CALMET Hybrid Winds Great Plains, July 8, 600 km CALMET NOOBS=1 Winds Great Plains, July 8, 600 km CALMET NOOBS=2 Winds Great Plains, July 8, 600 km, MM5 Winds Turner Stability Golder Stability (0.1431)

32 Plume Sigma-y σ y (km) CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Great Plains, July 8, 100 km, CALMET Winds Great Plains, July 8, 100 km, MM5 Winds Great Plains, July 8, 600 km, CALMET Obs Only Winds Great Plains, July 8, 600 km CALMET Hybrid Winds Great Plains, July 8, 600 km CALMET NOOBS=1 Winds Great Plains, July 8, 600 km CALMET NOOBS=2 Winds Great Plains, July 8, 600 km, MM5 Winds Turner Stability Golder Stability (43.5)

33 Plume Arrival Time on Arc Arrival Time at Arc (Julian day: hour LST) CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Great Plains, July 8, 100 km, CALMET Winds Great Plains, July 8, 100 km, MM5 Winds Great Plains, July 8, 600 km CALMET Obs Only Winds Great Plains, July 8, 600 km CALMET Hybrid Winds Great Plains, July 8, 600 km CALMET NOOBS=1 Winds Great Plains, July 8, 600 km CALMET NOOBS=2 Winds Great Plains, July 8, 600 km, MM5 Winds SRDT Stability Golder Stability 190: : : : : : : : : : : : : : : : : : : : : : : : : : : : :0300

34 Arc Transit Time Length of Plume Passage (hours) CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Great Plains, July 8, 100 km, CALMET Winds Great Plains, July 8, 100 km, MM5 Winds Great Plains, July 8, 600 km CALMET Obs Only Winds Great Plains, July 8, 600 km CALMET Hybrid Winds Great Plains, July 8, 600 km CALMET NOOBS=1 Winds Great Plains, July 8, 600 km CALMET NOOBS=2 Winds Great Plains, July 8, 600 km, MM5 Winds SRDT Stability Golder Stability (15) 15 6

35 Plume Centerline Location Plume Centerline (degrees from north) CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Great Plains, July 8, 100 km CALMET Hybrid Winds Great Plains, July 8, 100 km, MM5 Winds Great Plains, July 8, 600 km CALMET Obs Only Winds Great Plains, July 8, 600 km CALMET Hybrid Winds Great Plains, July 8, 600 km CALMET NOOBS=1 Winds Great Plains, July 8, 600 km CALMET NOOBS=2 Winds Great Plains, July 8, 600 km, MM5 Winds (17.53)

36 Global Statistics - Great Plains 600km MM5CALPUFF-AERMOD Turb Correlation (C): 0.45 NMSE: Avg Bias: Fractional Bias (FB): Fig of Merit in Space (FMS): K-S Parameter (KSP): Overall Rank (C FB,FMS,KSP): 1.74 CALMET OBS-AERMOD Turb Correlation (C): NMSE: Avg Bias: Fractional Bias (FB): 1.43 Fig of Merit in Space (FMS): 0.0 K-S Parameter (KSP): Overall Rank (C FB,FMS,KSP): 0.42

37 3-Hour Running Averages CALMET Hybrid MM5CALPUFF

38 Effect of CALMET Operation Choice on Plume Centerline

39 Savannah River Tracer release from Savannah River Laboratory on December 10, SF6 tracer 3 hour, elevated release 100 km arc of downwind monitors

40 Arrival Time and Duration - SRL Arrival Time at Arc (Julian day: hour LST) CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Savannah River, July 8, 100 km, CALMET Winds 344: : :1300 Length of Plume Passage (hours) CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Savannah River, July 8, 100 km, CALMET Winds

41 CWIC and Sigma Y - SRL CWIC (ppt-m x 10 5 ) CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Savannah River, December 10, 100 km, CALMET Winds σ y (km) CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Savannah River, December 10, 100 km, CALMET Winds

42 Fitted and Observed Max - SRL Cmax (ppt) fitted maximum CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Savannah River, December 10, 100 km, CALMET Winds Omax (ppt) actual maximum CALPUFF Dispersion Option Observed P-G CALPUFF AERMOD Savannah River, December 10, 100 km, CALMET Winds

43 SRL-75: 100km Arc Results Observed plume centerline: Predicted plume centerline: P-G: CALPUFF Turb: AERMOD Turb: Shorter than observed transit times of 100 km arc result in significant model underprediction (2-6 x underprediction)

44 CAPTEX-83 Tracer Experiment Ground-level 3-hour releases of the PMCH tracer during September October times from Dayton, Ohio 2 times from Sudbury, Ontario 84 ground level monitoring sites monitoring 3 and 6 hour averages of PMCH.

45 CALPUFF Experiment Design Initial Phase: CTEX R3 (Dayton, OH) and R5 (Sudbury, Ont) CALMET Hybrid 12KM MM5CALPUFF 12 KM All CALPUFF dispersion options Puff Splitting

46 MM5 Configuration MM5 Version nested domains 108 km 36 km 12 km 34 vertical layers ICBC: NCEP/NCAR Reanalysis Data available every 6 hours at 2.5 x 2.5 Physics: Kain-Fritsch II Cumulus Pleim-Xu PBL Pleim-Chang LSM RRTM Radiation Simple Ice Microphysics

47 Global Scatter Plot CAPTEX Release CAPTEX Release 7 - CALPUFF (AERMOD Turbulence) Predicted Observed

48 Global Statistics CAPTEX Release 7 MM5CALPUFF-AERMOD Turb Correlation (C): 0.17 NMSE: Avg Bias: Fractional Bias (FB): Fig of Merit in Space (FMS): K-S Parameter (KSP): Overall Rank (C, FB,FMS,KSP): 0.93 CALMET HYBRID-AERMOD Turb Correlation (C): 0.49 NMSE: Avg Bias: Fractional Bias (FB): Fig of Merit in Space (FMS): K-S Parameter (KSP): Overall Rank (C, FB,FMS,KSP): 1.37

49 European Tracer Experiment Two 3-hour atmospheric releases of PFT in October and November of Sampling for PFT occurred for 3 days after initial release across wide monitoring network in Europe. 168 ground level monitoring sites in Eastern and Western Europe. 30 modeling research groups all over the world were informed of the time, location and characteristics of the release, and predicted in real-time the dispersion patterns of the tracer over the subsequent 60 hours. The real-time transmission of model results was useful for evaluating their capability to respond in an emergency by making such information available to decision makers. When the chemical analyses of the samples were completed, the statistical evaluation of model predictions against measured tracer concentration values took place. Measurements are now available to the scientific community to reproduce the ETEX dispersion experiments with present and future updated models.

50 Synoptic Meteorology ETEX R1 Deep extratropical cyclone located in North Sea west of Norway. Secondary extratropical cyclone developing near Balkan Peninsula. Resultant winds over Europe cause strong north-south deformation of the windfield.

51 Model Experiment Design MM5 Version km domain only Same physics as GP-80 Dispersion Models: CALPUFF MM5CALPUFF Meteorology P-G Turbulence AERMOD Turbulence CALPUFF Turbulence Puff-Splitting FLEXPART Lagrangian Particle Model 168 monitoring sites throughout Europoe simulated as discrete model receptors

52 Global Scatter Plot ETEX 1

53 ETEX Release 1 - CALPUFF 24 HR 36 HR 48 HR 60 HR

54 ETEX Release 1 - FLEXPART 24 HR 36 HR 48 HR 60 HR

55 FLEXPART Simulation ETEX Release 1

56 ETEX Release 2 - CALPUFF 12 HR 24 HR 36 HR 48 HR

57 ETEX Release 2 - FLEXPART 12 HR 24 HR 36 HR 48 HR

58 FLEXPART Simulation ETEX Release 2

59 Observations Great Plains Tracer Experiment and Savannah River Experiment 1. Unable to replicate results from original GP-80 and SRL studies conducted by PES in 1997 despite using same raw meteorological data, horizontal, and vertical grid configurations. Only major difference is use of lambert conformal projection for GP-80 and SRL. 2. CALPUFF performance varied significantly due to variations in CALMET options selected. CALPUFF results are highly sensitive to manner in which meteorology is supplied to the model (e.g. How is CALMET run ). 3. Underlying P-G dependencies in CALPUFF discovered when varying method for computing P-G class. These dependencies exist even when using turbulence dispersion. Greater understanding of P-G dependencies of model is necessary before P-G v. turbulence question can effectively be answered. 4. Arrival time, location, and duration of tracer plume on arc of receptors is highly sensitive to dispersion option selected, especially when employed with puff-splitting. Puff-splitting appears to be an essential element of more realistic LRT simulations, especially for the 600 km GP-80 study. There is limited understanding of how best to employ puff-splitting or more effectively tune the parameters to maximize performance.

60 Observations (cont d) CAPTEX CALMET hybrid winds outperformed MM5CALPUFF for Release 7 from Sudbury, Ontario. Results were marginally better. Necessary to complete MM5 simulations for remaining 5 releases and complete tracer simulations to form a more complete understanding of how CALPUFF does on this European Tracer Experiment (ETEX) CALPUFF with MM5CALPUFF winds performs badly for both ETEX1 and ETEX2. In fairness, most models examined during the ATMES-II study performed poorly on ETEX2, so no surprise for CALPUFF. CALPUFF performance ETEX1 performance was surprising. To better simulate deformation effect on puff dispersion, puff splitting was turned on for each time step and all restrictions on when puffs split were eliminated. It did little to change performance. Advection wind computation method wind of CALPUFF cannot represent a rapidly changing windfield with deformation since the advection wind is averaged across the vertical column from puff bottom to puff top. Lagrangian Particle Model FLEXPART using same MM5 meteorology developed for ETEX1 does significantly better.

61 Next Steps OAQPS and Regions 5, 6, and 10 to complete SRL, Westvaco, and Clifty Creek OAQPS to complete CALPUFF simulations for remainder of CAPTEX and ETEX releases OAQPS to complete FLEXPART simulations on remainder of tracer releases Commence complex terrain tracer simulations for Brush Creek (ASCOT 1982), Project MOHAVE (1993), and VTMX (Salt Lake City 2000) Contact European agencies to obtain tracer data for Project Riviera, ALPNAP, etc.

62 Questions?