Final Report. Project Factors Influencing Ozone-Precursor Response in Texas Attainment Modeling

Transcription

1 Final Report Project Factors Influencing Ozone-Precursor Response in Texas Attainment Modeling Principal Investigator: Daniel Cohan, Rice University Co-Principal Investigators: Greg Yarwood, ENVIRON International Bonyoung Koo, ENVIRON International Project Scientists: Xue Xiao, Rice University Antara Digar, Rice University Submitted to: Texas Air Quality Research Program August 31, 2011

2 Executive Summary This report presents the findings of Texas Air Quality Program Project , which investigated the influence of input uncertainties on model predictions of pollutant responsiveness to emission controls. This project characterized how various model formulations (structural uncertainty) and input parameters (parametric uncertainty) influence predictions of ozoneprecursor response in Texas State Implementation Plan (SIP) modeling episodes. Both Bayesian and non-bayesian approaches were applied to compute probabilistic representations of the sensitivity of ozone to changes in precursor emissions. Base case modeling was taken from TCEQ s CAMx simulations of ozone during two monthlong episodes in Structural scenarios were then developed by applying alternate options for the biogenic emissions model, the deposition scheme, the chemical mechanism, the global model for deriving boundary conditions, and satellite-based photolysis rates. Screening analysis of the impacts of these options on ozone concentrations and sensitivities led to a focus on scenarios involving alternate choices for biogenic emissions model and chemical mechanism. The base model achieved very low bias during the June 2006 episode (NMB = -1.0% relative to ozone monitors in the 12-km domain), so the structural scenarios provide plausible alternatives but could not dramatically improve model performance. For parametric uncertainties, screening analysis identified the specific emission rates, reaction rate constants, and boundary conditions that most influence ozone concentrations and their sensitivities to nitrogen oxide (NO x ) and volatile organic compound (VOC) emissions. Some parameters such as ozone boundary conditions were found to impact concentrations far more strongly than sensitivities, whereas the converse was true for some other parameters such as anthropogenic VOC emissions. Bayesian Monte Carlo analysis was then applied to weight the relative likelihood of alternate structural and parametric scenarios, based on model performance in simulating observed concentrations within the Dallas-Fort Worth (DFW) region during the June 2006 episode. Metric 1 evaluated model performance on high-ozone days at three DFW monitors, while Metric 2 considered average 8-hour ozone concentrations across all DFW monitors on each episode day. A non-bayesian metric for assigning weights based on standard model performance statistics

3 (Metric 3) was also developed and was applied to produce alternative weightings of the Monte Carlo scenarios. The Bayesian and non-bayesian analyses generated probabilistic representations of ozone responses to changes in precursor emissions and of model input parameters. All of the results confirmed the findings of the base model that 8-hour ozone in the DFW region during the June 2006 episode was predominately NO x -limited. However, the three metrics yielded conflicting shifts in the probability distributions of ozone sensitivities. For example, results from Metric 1 tended to increase the predicted sensitivity of ozone to NO x, whereas Metric 2 indicated slightly greater sensitivity to VOC than originally modeled (Figure ES-1). Non-Bayesian Metric 3 yielded a slight shift toward greater sensitivity to VOCs, but retained the primarily NO x -limited conditions of the base model. Further work is needed to refine the metrics and incorporate consideration of other measurements beyond ozone for evaluating model performance. Nevertheless, the project has demonstrated how probabilistic analyses via an ensemble approach can supplement deterministic estimates of ozone response and characterize the uncertainty of those results.

4 O 3 sensitivity to ANO X Ozone sensitivity to AVOC METRIC 3 METRIC 2 METRIC 1 Figure ES-1. Cumulative probability distribution functions of the sensitivity of ozone at the Denton monitor in June 2006 to DFW anthropogenic NO x (left) and VOC (right) emissions for Bayesian metrics 1 and 2, and non-bayesian Metric 3. Green line shows deterministic (base case) results.

5 TABLE OF CONTENTS 1. Introduction and Motivation 1 2. Description of Model and Inputs 3 3. Screening Analysis for Structural Scenarios Screening Analysis for Parametric Scenarios Bayesian and non-bayesian Monte Carlo Analysis: Methods and Pseudo Case Testing Bayesian and non-bayesian Monte Carlo analysis: Results and Discussion Conclusions References Appendix 1: CB-6 Mechanism 80

6 1. Introduction and Motivation Developing control strategies to provide for attainment of ozone standards relies upon photochemical modeling to predict the responses of pollutants to emission changes. Model estimates of pollutant-emission responses or sensitivities help inform control strategy selection and indicate the amount of emission reduction needed to attain air quality standards. Thus, models for attainment planning must reliably predict not only pollutant concentrations, but also their responsiveness to emission changes. However, despite the abundance of methods to gauge model performance for pollutant concentrations, the accuracy of sensitivity predictions cannot be directly gauged. Pollutant concentrations are routinely observed at numerous monitors, but concentration-emission sensitivity relationships cannot be directly measured in the ambient atmosphere. Dynamic evaluation of how pollutant concentrations respond to emission changes over weekly (i.e., weekday vs weekend) or interannual (e.g., before and after nitrogen oxides (NO x ) SIP Call) time scales (Gilliland et al., 2008; Dennis et al., 2010; Pierce et al., 2010) can provide a proxy for ground-truthing sensitivity estimates, but the accuracy and uncertainty of those estimates remain poorly characterized. Uncertainties in pollutant-emission response can arise from choices of model formulations such as vertical mixing scheme and chemical mechanism (structural uncertainty), and of input parameters such as reaction rate constants and deposition velocities (parametric uncertainty) (Fine et al., 2003; Deguillaume et al., 2008; Pinder et al., 2009). Uncertainty can be especially pronounced for pollutants such as ozone which form from nonlinear interactions of multiple precursor compounds (Lin et al., 1988; Cohan et al., 2005). Recent work has introduced efficient methods for characterizing structural and/or parametric uncertainties in ozone response (Pinder et al., 2009; Digar and Cohan, 2010; Tian et al., 2010). Of these, only Pinder et al. jointly considered structural and parametric uncertainties. Other studies have shown that Bayesian Monte Carlo analysis can be applied to weight the relative likelihood of each model formulation based on its performance in simulating observed concentrations (Bergin and Milford, 2000; Deguillaume et al., 2007a). These weighted results 1

7 can yield probability distributions for predicting the actual values of pollutant-emission sensitivities as well as model inputs such as emission rates. This project merges some of the best practices of the recent uncertainty studies and applies them for a reanalysis of photochemical modeling episodes in Texas. Specifically, the reduced form model methods introduced by Tian et al. (2010) and Digar and Cohan (2010) are extended beyond parametric uncertainties to also consider structural uncertainties in model inputs and formulations. Whereas these previous applications of reduced form models assumed each scenario to be equally likely, here we apply the Bayesian Monte Carlo method of Bergin and Milford to weight each scenario based on its performance in simulating observed pollutant concentrations in Texas. This work yields probabilistic representations of ozone responses to emission reductions in the Dallas-Fort Worth region, and highlights model inputs that most strongly influence those estimates. 2

8 2. Description of Model and Inputs All analyses are conducted with the Comprehensive Air quality Model with extensions (CAMx) version 5.32 ( for Texas Commission on Environmental Quality (TCEQ)- developed episodes in order to optimize the relevance and transferability of the results to Texas attainment planners. Specifically, an August 13-September 15, 2006 episode is considered for Houston-Galveston-Brazoria (HGB) ( and a May 31-July 2, 2006 episode for Dallas-Fort Worth (DFW) and other regions ( These two episodes were identified by TCEQ based on their prevalence of observed eight-hour (8-hr) daily maximum ozone concentrations exceeding the hr ozone National Ambient Air Quality Standard (NAAQS) (Chapter 3 of (TCEQ, 2009)). For example, there were ten days on which 8-hr ozone exceeded 84 ppb at one or more HGB monitors during the Aug/Sept episode (Chapter 3, TCEQ, 2009). The two episodes encompassed a sufficient number of days to reflect a variety of meteorological conditions that favor ozone production. The HGB episode occurred during the TexAQS II field intensive period, which enhances opportunity for comparing model results with observations. Figure 2.1. DFW (left) and HGB (right) CAMx modeling domain. 3

9 The horizontal modeling domain structure consists of a coarse-grid (36 km resolution) eastern US domain and nested fine-grid subdomains: an East Texas subdomain (12 km resolution), and a DFW subdomain (4 km resolution) or HGB subdomain (4 km resolution) (Figure 2.1). The 2km HG subdomain was not considered. CAMx applies two-way nesting across the domains. Vertically there are 28 layers for the 36/12/4 km domains for the DFW episode, and 17 layers for the 36/12 km domains and 28 layers for the 4 km domain for the HGB episode. The input meteorology, emissions, initial/boundary concentrations and overall model configuration were obtained from the TCEQ base case for the two episodes. Development details of the base case inputs can be found elsewhere (TCEQ, 2009, Chapter 3). MM5 Version was used to generate meteorological inputs to CAMx including wind speed, wind direction, temperature, humidity, etc. The four model parameters have been evaluated against meteorological observations by a statistical package developed by TCEQ and shown good performance. Day-specific, gridded, speciated and temporally (hourly) allocated emission inventories were created by the emissions modeling processors, version 3 of the Emissions Processing System (EPS3) (for point, area, and mobile sources) and the Global Biosphere Emissions and Interactions System (GloBEIS) biogenics emissions model (for biogenic sources). The global air quality model MOZART was used to derive the boundary conditions. The base case model uses the Carbon Bond Mechanism (CB-05) and the Regional Acid Deposition Model (RADM) dry deposition scheme. A primary aim of this work is to assess how structural changes to the photochemical model and its inputs may influence predictions of ozone sensitivity to precursor emissions. Structural uncertainty analysis thus requires obtaining a variety of alternate CAMx inputs that differ from those used for the original TCEQ SIP modeling, focusing on the factors of emission inventories, chemical mechanism, photolysis rates, boundary conditions, and dry deposition scheme Plausible scenarios of model formulations and input parameter settings were then developed based on combinations of these inputs. 4

10 2.1. Alternate Boundary Conditions The original modeling provided by TCEQ used boundary conditions (BCs) generated by the MOZART model. ENVIRON obtained the 2006 annual GEOS-CHEM global model simulation outputs from NASA. BCs for both episodes on the national RPO 36-km modeling domain were extracted from the GEOS-CHEM outputs using the GEOS2CMAQ processor (version 3.0). BCs for the TCEQ 36-km modeling grid were generated using CAMx simulations with 2-way nesting as described in reports to TCEQ 1. Table 2.1 shows how the GEOS-CHEM model species are mapped to CAMx CB05 species. The 3-hourly GEOS-CHEM output data were linearly interpolated to hourly CAMx BCs. Table 2.1. Mapping of GEOS-CHEM to CAMx CB05 species. CAMx NO2 O3 CO NXOY HNO3 PNA H2O2 NTR FORM ALD2 ALDX PAR OLE ETHA MEPX PAN PANX ISOP ISPD SO2 NH3 ISP PSO4 GEOS-CHEM NOx Ox - NOx CO 2 N2O5 HNO3 HNO4 H2O2 R4N2 CH2O 0.5 ALD2 RCHO PRPE + ALK C3H8 + ACET + MEK + RCHO PRPE 0.5 C2H6 MP PAN PPN + PMN 0.2 ISOP MACR + MVK SO2 + DMS NH3 0.2 ISOP SO4 + MSA 1 Reports are available at 5

11 Figure 2.2 compares vertical profiles of average ozone boundary conditions from the MOZART (base case) and GEOS-CHEM (alternate case) global model simulations. GEOS-CHEM BCs exhibit higher ozone concentrations than MOZART BCs at all model layers and the differences range from 0.7 ppb (west boundary) to 8 ppb (north boundary). 6

12 (a) West Boundary (b) East Boundary (c) South Boundary (d) North Boundary Figure 2.2. Average ozone boundary conditions (ppb) by model layer estimated from the MOZART and GEOS-CHEM global model simulations (Aug 13 Sep 15 episode). 7

13 2.2. Alternate Biogenic Emissions Alternate biogenic emissions inputs were prepared for each modeling grid using the Model of Emissions of Gases and Aerosols from Nature (MEGAN) biogenic emission model (version 2.03a) which employs updated land cover data with 1-km of spatial resolution based on satellite and ground observations (Guenther et al., 2006). Figure 2.3 shows spatial distributions of average daily total biogenic NOx, non-methane volatile organic compound (NMVOC) and carbon monoxide (CO) emissions from the GloBEIS (base case) and MEGAN (alternate case) biogenic emission models. Although both models produced similar spatial patterns of biogenic emissions, MEGAN estimated lower NOx emissions and higher NMVOC emissions than GloBEIS. CO emissions produced by MEGAN are higher than those from GloBEIS. Table 2.2 presents domain total biogenic emissions from the two models for the 36- and 12-km grids. Strong differences between biogenic emission estimates from BEIS and MEGAN have been documented in previous studies (Carlton and Baker, 2011), though it is unclear the extent to which they arise from different model formulations or different inputs such as land cover data. 8

14 (a) NOx (GloBEIS) (b) NOx (MEGAN) (c) NMVOC (GloBEIS) (d) NMVOC (MEGAN) (e) CO (GloBEIS) (f) CO (MEGAN) Figure 2.3. Average daily total biogenic NOx, NMVOC and CO emissions over the 12-km modeling domain estimated from the GloBEIS and MEGAN biogenic emission models (Aug 13 Sep 15 episode) 9

15 Table 2.2. Domain total daily NOx, NMVOC and CO emissions (TPD) estimated from the GloBEIS and MEGAN biogenic emission models (Aug 13 Sep 15 episode) Species 36-km Domain 12-km Domain GloBEIS MEGAN GloBEIS MEGAN NOx 6,932 2,168 1, NMVOC 159, ,059 48,176 59,527 CO 16,622 19,684 4,457 5, Land Use Inputs for the Zhang Dry Deposition Scheme CAMx version 5.3 (ENVIRON, 2010) offers two dry deposition options: the original approach based on the work of Wesely (1989) and Slinn and Slinn (1980); and an updated approach based on the work of (Zhang et al., 2001; 2003). The new Zhang scheme incorporates vegetation density effects via leaf area index (LAI), possesses an updated representation of non-stomatal deposition pathways including a better snow cover treatment, and has been tested extensively through its use in daily air quality forecasting. The original Wesely/Slinn model is formulated for 11 land use categories, while the Zhang model uses 26 land use categories (Table 2.3). A new land use input file format is introduced that supports both land use categorizations as well as an optional LAI data field. ENVIRON prepared the new land use inputs for the Zhang scheme for each modeling grid. 10

16 Table 2.3. CAMx land use categories for the Wesely/Slinn model and the Zhang model. Wesely/Slinn Model Zhang Model Category Category Land Cover Category Number Number Land Cover Category 1 Urban 1 Water 2 Agricultural 2 Ice 3 Rangeland 3 Inland lake 4 Deciduous forest 4 Evergreen needleleaf trees 5 Coniferous forest, wetland 5 Evergreen broadleaf trees 6 Mixed forest 6 Deciduous needleleaf trees 7 Water 7 Deciduous broadleaf trees 8 Barren land 8 Tropical broadleaf trees 9 Non-forested wetlands 9 Drought deciduous trees 10 Mixed agricultural/range 10 Evergreen broadleaf shrubs 11 Rocky (with low shrubs) 11 Deciduous shrubs 12 Thorn shrubs 13 Short grass and forbs 14 Long grass 15 Crops 16 Rice 17 Sugar 18 Maize 19 Cotton 20 Irrigated crops 21 Urban 22 Tundra 23 Swamp 24 Desert 25 Mixed wood forest 26 Transitional forest 2.4. Chemistry Mechanism CB6 Chemistry Mechanism CB6 is the 6 th version of the Carbon Bond mechanism (Yarwood et al., 2010). Several organic compounds that are long-lived and relatively abundant, namely propane, acetone, benzene and ethyne (acetylene), are added explicitly in CB6 so as to improve oxidant formation from these compounds as they are oxidized slowly at the regional scale. CB6 also includes several updates for organic and inorganic aerosol chemistry. The core inorganic chemistry mechanism for CB6 is based on evaluated data from the International Union of Pure and Applied Chemistry (IUPAC) 11

17 tropospheric chemistry panel as of January, 2010 (Atkinson et al., 2010). IUPAC also is the primary source for photolysis data in CB6 with some data from the 2006 NASA/JPL data evaluation ( or other sources for photolysis of some organic compounds. There are changes to the organic chemistry for alkanes, alkenes, aromatics and oxygenates. The most extensive changes are for aromatics and isoprene. Chemistry updates for aromatics were based on the updated toluene mechanism (CB05-TU) developed by Whitten et al. (2010) extended to benzene and xylenes. The isoprene mechanism was revised based on several recently published studies (Paulot et al., 2009a; Paulot et al., 2009b; Peeters et al., 2009). Compared to the CB05 mechanism, CB6 increases the number of model species from 51 to 76 and the number of reactions from 156 to 218. A listing of reactions in the CB6 mechanism is provided in Appendix 1. Note that for CB6 (as well its modified form presented in Section 2.2), the rate constant for Reaction 45 (OH+NO 2 ) was adjusted as shown in Appendix 1 to reflect the findings of Mollner et al. (2010). However, there remains considerable uncertainty in this rate constant, as a subsequent chemical kinetics report from NASA JPL in 2011 chose not to update this value ( Alternate Chemistry Mechanism ENVIRON developed an alternate CB6 isoprene mechanism that produces more OH radicals at low NOx conditions without breaking the mechanism evaluation against chamber experiments. The motivation for this alternate mechanism is to explore a potential approach for addressing reported underpredictions of OH by most chemical mechanisms in isoprene-rich, low-no x conditions (Lelieveld et al., 2008). Potential approaches to adjusting isoprene oxidation mechanisms to influence HO x levels were discussed by (Archibald et al., 2010). Table 2.4 shows the adjustments to the base CB6 mechanism that were adopted for the alternate CB6 mechanism, which is termed CB6MOD4 in the sensitivity analyses in subsequent sections of this report. 12

18 Table 2.4. Changes between the base and alternate CB6 mechanisms Rxn # Base Mechanism Alternate Mechanism 116 GLY + OH = 1.7 CO XO RO2 + HO2 118 GLY + NO3 = HNO3 + CO + HO2 + XO2 + RO2 151 ISO2 + HO2 = 0.88 ISPX OH HO FORM ISPD 154 ISO2 = 0.8 HO OH FORM ISPD k = 1.0 s ISPX + OH = EPOX OH ISO RO IOLE ALDX GLY + OH = 1.7 CO HO OH GLY + NO3 = HNO CO HO OH ISO2 + HO2 = 0.88 EPOX OH HO FORM ISPD ISO2 = 0.95 HO OH FORM ISPX k = 0.1 s -1 ISPX = OH + CXO3 j(ispx) = j(aldx) 60.0 ENVIRON evaluated the alternate CB6 mechanism by 6 environmental chamber experiments using isoprene performed by University of California (UC) Riverside: Xenon arc Teflon Chamber (XTC) experiment 093 Evacuable Chamber (EC) experiment 520 Outdoor Teflon Chamber (OTC) experiments 309A, 309B, 316A & 316B The box model performance was evaluated using model-experiment errors in the following quantities: Maximum ozone concentration (Max(O 3 )) Maximum D(O 3 NO) D(O 3 NO) = ([O 3 ] [NO]) t=t ([O 3 ] [NO]) t=0 (quantifies the amount of O 3 formed and NO oxidized during the experiment) NO x crossover time The time when [NO 2 ] = [NO] (provides information on the rate of NO oxidation into NO 2 ) 13

19 Table 2.5 shows averages and standard deviations of model errors with the base and alternate mechanisms. Performance differences between the base and alternate mechanisms are within experimental uncertainty range. Figure 2.4 compares time-series plots of the box model simulations and chamber experiments for ozone, NO and isoprene. Table 2.5. Box model performance statistics for the base and alternate CB6 mechanisms Model error (%) in Max(O 3 ) Model error (%) in Max(D(O 3 NO)) Model error (min) in NOx crossover time Base Alternate Base Alternate Base Alternate Average Std. dev

20 XTC093 EC520 OTC309A OTC309B OTC316A OTC316B Figure 2.4. Time-series plots of the box model simulations and chamber experiments for ozone, NO and isoprene. 15

21 3. Screening Analysis for Structural Scenarios Initial screening with Decoupled Direct Method in CAMx (CAMx-DDM) was applied to identify the structural input choices that most significantly influence ozone-precursor sensitivity results. Structural scenarios were constructed from combinations of discrete choices of four structural factors: dry deposition (RADM/Zhang), chemical mechanism (CB05/CB-6), boundary conditions (MOZART/GEOS-CHEM), and biogenic emissions models (GloBEIS/MEGAN), plus one case using the alternate CB-6 mechanism (CB6MOD4) and one case using GOES-based photolysis rates (SATTR) provided by the University of Alabama-Huntsville. CAMx-DDM simulations were conducted on the 36/12 km domains for all 16 possible combinations of the four main factors, plus two additional cases for CB6MOD4 and SATTR. Outputs were generated to assess ozone (and other species) concentrations and their sensitivities to anthropogenic NO x (ANO x ) and VOC (AVOC) emissions from each of five regions (DFW, Austin, San Antonio, HGB, and the rest of 12 km domain) for the June 2006 DFW episode. Analysis are mainly focused on the DFW region (assumed control scenario) while the methodology can be applied to other regions. Note that the base case uses the original RADM dry deposition scheme, the CB-05 chemical mechanism, boundary conditions from the MOZART global model, and GloBEISgenerated biogenic emissions. The perturbation cases switch one or more of the structural factors to its alternate setting Comparison between perturbation cases and the base case We compare each perturbation case to the base case, focusing on results averaged across the episode for an 8-hr window (10:00-18:00) each day. A fixed time window rather than daily 8- hour maximum is chosen so that the same hours are compared in each case, making conditions such as plume locations more comparable. The statistical measures that serve as the basis for the comparisons are presented in Table

22 Table 3.1. Statistical measures used to compare the structural cases. Statistical measures Formula* Correlation Coefficient (R 2 ) R 2 = N 1 N 1 (case j j case (case case ) j j 2 )(base base) N 1 (base base) 2 Root-Mean-Squares (RMS) (ppb) RMS N = 1 (case j N base) 2 N 1 Mean Bias (BIAS) (ppb) BIAS = (case j base) N 1 (case j base) 1 Normalized Mean Bias (NMB) (percent) NMB = 100% N base N 1 case j base 1 Normalized Mean Error (NME) (percent) NME = 100% N base 1 (case base) Mean Normalized Bias (MNB) (percent) MNB = N j 100% N base N 1 1 Mean Normalized Gross Error (MNGE) (percent) 1 case base MNGE = N j 100% N base 1 * base denotes concentrations or sensitivities from base case CAMx simulations; case j denotes each perturbation case; case j represents the average value of the N data points in that case. 17

23 Results are compared for ozone concentrations at US Environmental Protection Agency (EPA) monitors within DFW (Table 3.2) and their sensitivities to anthropogenic NO x (ANO x ) (Table 3.4) and VOC (AVOC) (Table 3.5) emissions from the DFW non-attainment region. The 8-hr ozone concentrations from each structural case are also compared to ambient ozone observations from EPA Air Quality System (AQS) database ( (Table 3.3). Among the individual choices in structural inputs, ozone concentrations are most influenced by the chemical mechanism (CB-6), satellite-based photolysis rates (SATTR), and deposition scheme (Zhang), based on the RMS results in Table 3.2. The boundary conditions, biogenic emissions inventory, and modified form of CB-6 all exert much smaller influences on concentrations. As expected, perturbation cases that involve combinations of structural changes typically yield larger changes in ozone concentrations. Statistical evaluation of the scenarios against ambient ozone observations shows that none of the perturbation scenarios dramatically improve model performance in terms of bias or error (Table 3.3). This largely reflects the fact that the base model achieved very low bias during the June 2006 episode, so perturbations can impair that performance. For informing the prioritization of control strategies, the sensitivities of ozone to changes in emission rates are of paramount importance. Ozone sensitivities both to DFW ANO x (Table 3.4) and to DFW AVOC (Table 3.5) are most influenced by the choices of photolysis rates (SATTR), biogenic emissions (MEGAN), and chemical mechanism (CB-6). The other two structural input choices (boundary conditions and deposition scheme) yield far smaller influences on ozone sensitivities. The small impact of boundary conditions was to be expected, given the large size of the 36-km domain (Figure 2.1) and the small impact of boundary conditions on ozone concentrations (Table 3.2). However, the lack of influence of deposition scheme on sensitivities is surprising, given its strong influence on concentrations. Switching between CB6 and CB6MOD4 significantly changes ozone sensitivities to ANO x emissions, but has little impact on sensitivity to AVOC emissions. 18

24 Table 3.2. Comparison of each perturbation case to base case simulations of 8-hour (10:00-18:00) ozone concentrations at all regulatory monitors within the 12 km domain for the June 2006 episode. Cases R 2 RMS ppb BIAS ppb NMB % NME % MNB % MNGE % Base Zhang Dep. (Z) CB-6 (C) GEOSCHEM_BC (G) MEGAN EI (M) Z + C Z + G Z + M C + G C + M G + M Z + C + G Z + C + M Z + G + M C + G + M Z + C + G + M CB6MOD CB6MOD4 vs CB SATTR* * Results for SATTR are available only for 5/31-6/15. 19

25 Table 3.3. Comparison for daily eight-hour averaged ozone concentrations (8-hr [O 3 ]) from the structural cases, evaluated against observations from regulatory ozone monitors within the 12-km domain for the June 2006 episode. Cases R 2 RMS ppb BIAS ppb NMB % NME % MNB % MNGE % Base Zhang Dep. (Z) CB-6 (C) GEOSCHEM_BC (G) MEGAN EI (M) Z + C Z + G Z + M C + G C + M G + M Z + C + G Z + C + M Z + G + M C + G + M Z + C + G + M CB6MOD SATTR* * Results for SATTR are available only for 5/31-6/15. 20

26 Table 3.4. Comparison of each perturbation case to base case for 8-hr sensitivities of ozone at DFW monitors to ANO x emissions from the DFW region for the June 2006 episode. Cases R 2 RMS ppb BIAS ppb NMB % NME % MNB % MNGE % Base Zhang Dep. (Z) CB-6 (C) GEOSCHEM_BC (G) MEGAN EI (M) Z + C Z + G Z + M C + G C + M G + M Z + C + G Z + C + M Z + G + M C + G + M Z + C + G + M CB6MOD CB6MOD4 vs CB SATTR* * Results for SATTR are available only for 5/31-6/15. 21

27 Table 3.5. Comparison of each perturbation case to base case for 8-hr ozone sensitivities at DFW monitors to AVOC emissions from the DFW region for the June 2006 episode. Cases R 2 RMS ppb BIAS ppb NMB % NME % MNB % MNGE % Base Zhang Dep. (Z) CB-6 (C) GEOSCHEM_BC (G) MEGAN EI (M) Z + C Z + G Z + M C + G C + M G + M Z + C + G Z + C + M Z + G + M C + G + M Z + C + G + M CB6MOD CB6MOD4 vs CB SATTR* * Results for SATTR are available only for 5/31-6/15. Figure 3.1 shows the diurnal profile of ozone sensitivities to DFW ANO x and AVOC emissions for the structural cases, averaged over the June episode and the DFW monitors. Afternoon ozone in DFW is primarily NO x -limited in all of the structural cases, with ozone about an order of magnitude more sensitive to ANO x than AVOC. In general, the MEGAN case increases (relative to the base case) ozone-ano x sensitivities and decreases ozone-avoc sensitivities during daytime because of its stronger biogenic VOC emissions. The CB-6 case also affected daytime ozone sensitivities but in the opposite direction, yielding strong sensitivities to AVOC. The new deposition scheme (Zhang) mainly affected nighttime ozone sensitivities and exerted little influence the 8-hr ozone sensitivities very much. Shallow planetary boundary layer conditions at night magnify the influence of deposition on conditions near the ground. 22

28 [O 3 ] / (E DFW ANOx ) (ppb) base Zhang(Z) CB6(C) GEOS(G) MEGAN(M) Sens of Region DFW to E DFW ANOx Time (hr) [O 3 ] / (E DFW AVOC ) (ppb) base Zhang(Z) CB6(C) GEOS(G) MEGAN(M) Sens of Region DFW to E DFW AVOC Time (hr) Figure 3.1. Diurnal profile of ozone sensitivities to DFW ANO x and AVOC emissions for the indicated structural cases, averaged over the June episode and the DFW region. 23

29 The results of the initial screening shown in the figure and tables above led us to retain biogenic emissions and chemical mechanism as the primary structural factors for further full analysis, including joint consideration with parametric uncertainties on the finer 4 km domain. Thus, four structural cases base case, CB-6, MEGAN, and CB-6+MEGAN are targeted. Two other structural factors, deposition scheme and boundary conditions, are excluded due to their much smaller impacts on sensitivities. The satellite-based photolysis rates did show a strong influence on concentrations and sensitivities, but were not used in this phase of the analysis because inputs for the full episode are not yet available. Figures 3.2 and 3.3 show the diurnal profile of ozone sensitivities to HGB and DFW ANO x and AVOC emissions for each of the four targeted structural cases. Consistent with the findings from the June DFW episode, the MEGAN case increased (relative to the base case) ozone-ano x sensitivities and decreased ozone-avoc sensitivities during daytime because of the stronger magnitude of biogenic VOC emissions. The CB-6 case also affected daytime ozone sensitivities but in the opposite direction, again with strong increases in sensitivity to AVOC. The CB- 6+MEGAN case showed the combined effects of the two structural factors. 24

30 base CB6(C) MEGAN(M) C+M Sens of Region HGB to E HGB ANOx [O 3 ] / (E HGB ANOx ) (ppb) Time (hr) base CB6(C) MEGAN(M) C+M Sens of Region HGB to E HGB AVOC [O 3 ] / (E HGB AVOC ) (ppb) Time (hr) Figure 3.2. Episode (Aug/Sept) and HGB region averaged diurnal profile of ozone sensitivities to HGB ANO x and AVOC emissions under the four targeted structural cases. 25

31 6 4 base CB6(C) MEGAN(M) C+M Sens of Region DFW to E DFW ANOx [O 3 ] / (E DFW ANOx ) (ppb) Time (hr) base CB6(C) MEGAN(M) C+M Sens of Region DFW to E DFW AVOC [O 3 ] / (E DFW AVOC ) (ppb) Time (hr) Figure 3.3. Episode (June) and DFW region averaged diurnal profile of ozone sensitivities to DFW ANOx and AVOC emissions under the four targeted structural cases. 26

32 3.2. Comparison between CB-6 and CB6MOD4 in CAMx ENVIRON developed the alternate form of CB-6, referred to here as CB6MOD4, specifically to test the importance of altered reaction rates that would boost the production of OH radicals under low-no x conditions without undermining performance relative to chamber experiments. Considerable scientific attention has been devoted in recent years to the inability of most chemical mechanisms to predict sufficient levels of OH in isoprene-rich, low-no x conditions (Lelieveld et al., 2008). Since the modified chemical mechanism allows the impact of isoprene oxidation reactions to be isolated, we devote attention in the following subsections to comparing the CB6 and CB6MOD4 results Comparison for OH concentrations between CB6 and CB6MOD4 Figure 3.4 shows the simulated OH fields using the CB6 and CB6MOD4 mechanisms at local noontime on a sample high ozone day, June 30. Figure 3.4. OH concentrations predicted by CAMx with the CB6 (left) and CB6MOD4 (middle) chemical mechanisms on June 30 at 12:00 CST, and the difference (right). OH concentrations are increased by the CB6MOD4 mechanism, not only in low NO x regions, but also in the urban area. However, on a percentage basis, the greatest increases in OH concentrations are simulated in rural low NO x areas. Thus, the modified chemical mechanism did in fact enhance rural OH levels, though it is beyond the scope of this study to explore whether 27

33 the increase in OH levels would be sufficient to address the underpredictions noted in the literature Comparison for ozone concentrations between CB6 and CB6MOD4 Figure 3.5 shows the simulated fields of ozone concentrations using the CB6 and CB6MOD4 mechanisms and their differences on a June 30 at 15:00 CST. The major high ozone plumes are captured by both mechanisms, and the differences in concentrations are relatively small. Figure 3.5. Ozone concentrations predicted by CAMx with the CB6 (left) and CB6MOD4 (middle) chemical mechanisms on June 30 at 12:00 CST, and the difference (right). Averaged over the EPA sites in the 12-km domain over all episode days, 8-hr ozone concentrations are only 0.12 ppb (0.19%) higher in CB6MOD4 than in CB-6 (Table 3.2). The slight increase in 8-hr ozone concentrations by CB6MOD4 likely resulted from the increase in OH concentrations noted in the previous subsection Comparison for ozone sensitivities between CB6 and CB6MOD4 Despite the small average change in ozone concentrations, were the sensitivities of ozone to emissions affected by the modifications to the chemical mechanism? Figure 3.6 shows the simulated ozone sensitivities to DFW ANO x emissions using the CB6 and CB6MOD4 mechanisms and their differences on several days during the June episode. 28

34 On June 10, 2006 On June 14, 2006 On June 30, 2006 Figure 3.6. Sensitivities of ozone to DFW ANOx emissions under the CB6 (left) and CB6MOD4 (middle) chemical mechanisms on three afternoons, and the difference (right). Sensitivities of ozone in the DFW region to DFW ANO x emissions are usually negative during early morning rush hours, and then turn positive during daytime hours (NO x -limited regime), 29

35 with smaller positive sensitivities simulated downwind. The plots for selected days show larger simulated ozone sensitivities in the DFW region with CB6MOD4 than CB6 during the afternoon. Statistically, CB6MOD4 also indicated larger sensitivities of 8-hr ozone concentrations to DFW ANO x emissions than CB6 when considering the whole episode (0.36 ppb and 12.51% higher, Table 3.3). With OH increased by CB6MOD4, daytime photochemistry became faster. More OH removed NO 2 (via OH+NO 2 HNO 3 ) and increased the ozone production efficiency of each NO x molecule, making ozone formation more sensitive to NO x emissions. Although the differences caused by the modified mechanism were small, it is interesting to note the much larger percentage impacts on ozone sensitivities than on ozone concentrations. This is consistent with an earlier study which showed that changes in uncertain reaction rate constants can exert much larger percentage influences on sensitivities than on concentrations (Cohan et al., 2010). 30

36 4. Screening Analysis for Parametric Scenarios A key feature of this project is the joint consideration of structural and parametric uncertainties influencing model results. Following upon the analysis of the structural scenarios in the previous section, parametric uncertainties are considered here. Screening was conducted on a list of input parameters that were hypothesized to potentially impact ozone concentrations and sensitivities. The list includes most of the parameters that have been identified by previous studies that ranked the relative importance of various input parameters in generating uncertainties in model outputs for ozone (Gao et al., 1995; Bergin et al., 1999; Hanna et al., 2001b; Cohan et al., 2010; Digar and Cohan, 2010). Specifically, we focused on the emission rates, reaction rate constants, and boundary conditions listed in Table 4.1. Each parameter was assumed to have a lognormal probability distribution, characterized by the sigma value reported in Table 4.1. Note that the reported uncertainty column of Table 4.1 reflects how uncertainty for that parameter was reported in the literature; we then computed sigma from the associated factors of uncertainty by the equation: sigma=ln(factor). 31

37 Table 4.1. Uncertain CAMx input parameters considered in the initial screening. Emission Rates: Parameter Reported Uncertainty* Factor of 1σ# Reference Domain-wide NO X ± 40% (1σ) Domain-wide Anthropogenic VOC ± 40% (1σ) Domain-wide Biogenic VOC ± 50% (1σ) Reaction Rate Constants: All Photolysis Frequencies Factor of 2 (2σ) R(All VOCs+OH) ± 10% (1σ) R(OH+NO 2 ) ± 30% (2σ) R(NO+O 3 ) ± 10% (1σ) Boundary Conditions: (Deguillaume et al., 2007b) (Deguillaume et al., 2007b) (Deguillaume et al., 2007b) (Hanna et al., 2001a) (Hanna et al., 2001a), (Deguillaume et al., 2007b) (Sander S P, 2006) (Hanna et al., 2001a) Boundary Cond. O 3 ± 50% (2σ) (Deguillaume et al., 2007b) Boundary Cond. NO X Factor of 3 (2σ) (Deguillaume et al., 2007b) Boundary Cond. HNO 3 Factor of 3 (2σ) (Deguillaume et al., 2007b) Boundary Cond. PAN Factor of 3 (2σ) (Deguillaume et al., 2007b) Boundary Cond. HONO Factor of 3 (2σ) (Deguillaume et al., 2007b) Boundary Cond. N 2 O 5 Factor of 3 (2σ) (Deguillaume et al., 2007b) *Uncertainty as reported in literature, all of which assumed lognormal Factor by which base value is multiplied or divided for a ±1σ range lognormal distribution; # sigma=ln(factor) To screen parameters that affect O 3 concentrations and responses to emission, relevant impact factors were evaluated by computing first-order sensitivity of O 3 to source controls (DFW anthropogenic NO X and VOCs) and its cross-sensitivity with each uncertain parameter, for a 2 week sub-episode spanning from June 6-20, 2006 for the base-case simulation. The impact factors take into account both the uncertainty in the input parameter itself, and the rate at which a 32

38 change in that parameter leads to a change in model output. Specifically, impact factors for the influence of a parameter on concentrations are calculated as σ j S (1) i,j /C i and impact factors for the influence of a parameter on sensitivities are calculated as σ j S (2) (1) i,j,k /S i,k In the equations above, σ j denotes the 1σ uncertainty in parameter j (taken from column 4 of Table 4.1); S (1) i,j denotes the first-order sensitivity coefficient of concentrations C i to parameter j; and S (2) i,j,k denotes the cross-sensitivity of C i to parameters j and k. An impact factor of 0.1 would mean that a one sigma increase in the input parameter would cause an approximately 10% change in that concentration or first-order sensitivity. CAMx-HDDM modeling was conducted on the 36/12/4 km domains, and the results within the 12-km domain were used to select key parameters needed for our final Bayesian analysis. Tables show the results for these screening tests. The parameters with impact factors greater than (next to last column) were selected in each case (final column). 33

39 Table 4.2: Results of the screening test for the selection of uncertain input parameters influencing O 3 concentrations. Parameter Uncertainty in parameter (1σ) 1 st order sensitivity # (ppb) Impact Factor* Selection (Y/N) Emission Rates Domain-wide NO X Y Domain-wide biogenic VOC Y Domain-wide anthropogenic VOC N Reaction Rates All photolysis rates Y R(NO 2 +OH) Y R(NO+O 3 ) Y R(all VOCs+OH) N Boundary Conditions BC(O 3 ) Y BC(NO X ) N BC(HNO 3 ) N BC(PAN) N BC(HONO) N BC(N 2 O 5 ) N # First-order sensitivity of O 3 to each uncertain parameter at time of the maximum daily 8-h average O 3, averaged over all the 12-km grid-cells corresponding to the regulatory monitors within DFW and over a 2 week period in summer spanning from June 6-20, *Impact factor: The fractional change in first-order sensitivity of ozone to emissions, due to a 1σ change in an input parameter. Computed as Impact Factor = σs (1) (1) j /C where S j is the first-order sensitivity of O 3 to an uncertain parameter and C is the concentration of O 3 (63.33 ppb). 34

40 Table 4.3: Results of the screening test for the selection of uncertain input parameters influencing O 3 sensitivity to ANOx. Parameter Uncertainty in parameter (1σ) Cross-sensitivity # (ppb) Impact Factor* Selection (Y/N) Emission Rates Domain-wide NO X Y Domain-wide biogenic VOC Y Domain-wide anthropogenic VOC Y Reaction Rates All photolysis rates Y R(NO 2 +OH) Y R(NO+O 3 ) Y R(all VOCs+OH) Y Boundary Conditions BC(O 3 ) N BC(NO X ) N BC(HNO 3 ) N BC(PAN) N BC(HONO) N BC(N 2 O 5 ) N # Cross-sensitivity of O 3 to DFW anthropogenic NO X (ANO X ) emissions and each uncertain parameter at time of the maximum daily 8-h average O 3, averaged over all the 12-km grid-cells corresponding to the regulatory monitors within DFW and over a 2 week period in summer spanning from June 6-20, *Impact factor: The fractional change in first-order sensitivity of ozone to emissions, due to a 1σ change in an input parameter. Computed as Impact Factor = σs (2) j,k /S (1) j where S (1) j is the first-order sensitivity of O 3 to DFW ANO X (5.40 ppb) and S (2) j,k is the cross sensitivity of S (1) j with an uncertain parameter. 35

41 Table 4.4: Results of the screening test for the selection of uncertain input parameters influencing O 3 sensitivity to AVOC. Parameter Uncertainty in parameter (1σ) Cross-sensitivity # (ppb) Impact Factor* Selection (Y/N) Emission Rates Domain-wide NO X Y Domain-wide biogenic VOC Y Domain-wide anthropogenic VOC Y Reaction Rates All photolysis rates Y R(NO 2 +OH) Y R(NO+O 3 ) Y R(all VOCs+OH) Y Boundary Conditions BC(O 3 ) Y BC(NO X ) N BC(HNO 3 ) N BC(PAN) N BC(HONO) N BC(N 2 O 5 ) N # Cross-sensitivity of O 3 to DFW anthropogenic VOC (AVOC) emissions and each uncertain parameter at time of the maximum daily 8-h average O 3, averaged over all the 12-km grid-cells corresponding to the regulatory monitors within DFW and over a 2 week period in summer spanning from June 6-20, *Impact factor: The fractional change in first-order sensitivity of ozone to emissions, due to a 1σ change in an input parameter. Computed as Impact Factor = σs (2) j,k /S (1) j where S (1) j is the first-order sensitivity of O 3 to DFW AVOC (0.626 ppb) and S (2) j,k is the cross sensitivity of S (1) j with an uncertain parameter. Although there was considerable overlap in the selected parameters, there were also some differences in those found to most influence concentrations and the two sensitivities. Domainwide NO x and biogenic VOC emissions, photolysis rates, and the reaction rate constants R(NO 2 +OH) and R(NO+O 3 ) significantly impacted all three categories. Meanwhile, boundary conditions of all of the NO y compounds were not major influences on any of the results. However, the BC(O 3 ) parameter significantly impacted concentrations but not sensitivity to NO x, whereas anthropogenic VOC emissions impacted sensitivities but not concentrations. 36

42 Targeting the parameters separately for the concentrations and the two sensitivity cases allows the scenarios to be modeled more efficiently by CAMx-HDDM. As will be explained in Chapter 5.2.2, reduced form model (RFM) calculations of ozone concentrations require first- and secondorder sensitivity results for each selected input parameter, along with cross-sensitivities between each parameter; RFM calculations of ozone sensitivities require first-order and cross-sensitivities between each targeted parameter and the control scenario (i.e., DFW ANO x or AVOC). The Monte Carlo sampling in Chapters 5 and 6 selects perturbation factors for all of the parameters that significantly impact concentrations and/or sensitivities. However, due to the requirements of the RFM, for computational efficiency Bayesian comparisons of modeled and observed concentrations adjust model results only by the parameters that influence concentrations, whereas sensitivities are adjusted only by the parameters that influence that sensitivity. 37

43 5. Bayesian and non-bayesian Monte Carlo Analysis: Methods and Pseudo Case Testing A Bayesian inference approach (Bergin and Milford, 2000; Deguillaume et al., 2007) is applied to construct probabilistic representations of ozone-precursor response based on the relative performance of the model under various structural and parametric settings in simulating observed ozone and precursor concentrations. Figure 5.1 shows the concept of the Bayesian Monte Carlo analysis as an extension to the standard Monte Carlo method. Both analyses involve generating hundreds or thousands of Monte Carlo simulations with different model formulations and input parameter settings randomly selected from predefined probability density functions. The standard Monte Carlo then develops a priori estimates of the probabilistic distributions of model outputs (ozone concentrations and response to emissions), assuming the simulations are equally likely. The Bayesian Monte Carlo analyses adjust the probability of each simulation by taking into account its model performance relative to the observations, leading to the a posteriori estimates of the probabilistic distributions. 38

44 Figure 5.1. Conceptual diagram of Bayesian Monte Carlo analysis, adapted from Deguillaume et al., The Monte Carlo method of randomly sampling inputs has often been used to explore how various input settings influence model outputs. Most previous applications of Monte Carlo to characterize photochemical model uncertainty have assumed that each of the input scenarios is equally likely to reflect true conditions (Bergin et al., 1999; Digar and Cohan, 2010; Tian et al., 2010). However, some combinations of input settings may yield model results that perform poorly relative to observations. Bayesian inference methods allow the relative likelihood of each model scenario to be weighted based on model performance. Initial applications of Bayesian Monte Carlo to parametric uncertainty analysis of photochemical models have been demonstrated by a few studies (Bergin and Milford, 2000; Beekman and Derognat, 2003; Deguillaume et al., 2007a) but remains an area of emerging interest. 39

45 5.1. Pseudo case tests Although the ultimate goal of Bayesian analysis is to use observations to assess the relative likelihood of each model scenario, there are many choices to be made regarding the metric(s) for comparing modeling results with observations and the computation of the likelihood function. For example, should model results be compared against observations for each monitor-day, or should results be aggregated spatially and/or temporally? Should the model be evaluated on high ozone days that drive nonattainment, or on all days? To address these and other questions in a controlled manner, pseudo case experiments were conducted. We first conducted pseudo case tests to evaluate how different metrics would influence the relative likelihoods assigned to different scenarios, following the Bayesian Monte Carlo method of Bergin and Milford (2000). The pseudo case modeling was designed to provide insights into how each metric would perform in assigning relative likelihoods to various model results of specified performance. To design the pseudo cases, we use model base case ozone concentrations for each data point) as pseudo observations O k, i.e., Y b, k ( b for base; k O k = Y b, k with Gaussian errors σ k = max( 20,25% O k ) Bergin and Milford (2000) used a 30% standard deviation for the observations. Due to lack of information a Gaussian error with the standard deviation of σ = max( 20,25% O ) (the maximum between 20 ppb and 25 % O ) is used in pseudo case tests for the hourly pseudo k ozone observations. As discussed later in this report, subsequent analysis led us to choose a lower value of σ k for Bayesian analysis of the actual model scenarios. k k 40

46 Then we generate pseudo model cases ( Y j, k ) by applying different systematic errors ( Bias j in ppb) and Gaussian random errors ( ζ ) to the base case ozone concentration fields (thus we know how bad each model cases is). For each case j : j Y = Y + Bias j + Yb, j, k b, k k j * ζ * random(0,1) (5.1) where random(0,1) represents a set of Gaussian random values with zero mean and unity standard deviation. For evaluating the model performance against observations under each scenario, a Gaussian likelihood function L( Y j O) for simulation j given the observations O is used (as defined by Bergin and Milford (2000), assuming that the errors in the interpolated concentrations at all monitor/days are independent and normally distributed with mean of zero) and extended as ( O Y ) LY ( j O) = exp = ( 2 π) σ N k jk, N 2 N 2 k 1 σ k k k = 1 (5.2) Where Y j, k and k O represent each ozone concentration from model simulation j and observation, respectively. An observation standard error of σ k is considered for any measurement O k (k = 1, 2,., N where N = total number of data points for computing likelihood). Note that Equation 5.2 in effect multiplies together the likelihoods that would be computed for each of the N measurements; this tends to accentuate the differences between the likelihoods assigned to the model cases as N grows larger. Bayes theorem is then applied to compute the a posteriori probability (or Bayesian weight) of the relative likelihood of model output for the j th simulation ( j = 1, 2,. M, where M = total number of Monte Carlo simulations) as follows, p'( Y O) = j LY ( O) py ( ) M j= 1 j LY ( O) py ( ) j j j (5.3) 41

47 where p'( Yj O) and p Y ) represent a posteriori and a priori probability of the model output, respectively. ( j For representation of Y, and O k, metrics have been designed mainly regarding the high j k concentrations of ozone at the targeted regulatory monitors ( For the June episode three ozone monitors are chosen within the DFW region: Denton, Eagle Mountain Lake, and Keller. The hr ozone design values at these three sites are 95, 96, and 94 ppb, respectively, which are among the highest of the monitors within the DFW region (dfw8h_o3_dv_ xlsx, ftp://amdaftp.tceq.texas.gov/pub/dfw8h2/data/). Note that for the pseudo-data testing we have run each structural scenario for May 31 July 2, When applying the metrics for the pseudo-data testing, the first five days were discarded as spin up days. On each episode day, hourly ozone concentrations were averaged within a fixed window of hr local time at each target monitor. We chose the fixed 8-hr window instead of the running 8-hr for comparison between different structural cases. The 8-hr averaged concentrations (from both pseudo model cases and pseudo observations) are calculated at these three target monitors, and the standard deviations for the 8-hr averaged pseudo observations are derived from those for the hourly concentrations. Then the 8-hr ozone results for both the model and the observations are further aggregated or selected in the following metrics in Table 5.1 (N = number of data points selected; Mo = number of monitors (3); D = number of days (28); and R = number of regions (DFW and HGB)). 42

48 Table 5.1. Metrics to select/aggregate model and observation data points in model performance evaluation for the pseudo-data tests. Metric A Description Daily 8-hr ozone at each target monitor on each day, considering only monitor-days when obs > 70 ppb (N <= Mo*D) B Average of unpaired 3 highest 8-hr ozone days at each target monitor (N = Mo) C D Rank-order the 8-hr results from all episode days at each target monitor, and then compare only 4 cut-points (95th percentile, 75th percentile, median, and 25th percentile) (N = Mo*4). The cut-points can be unpaired in time. Unpaired 8-hr peaks within the DFW region on each day, considering only days when obs > 70 ppb (N <= R*D). The 8-hr peaks are picked among the monitors in DFW for the observations, and the grid cells containing the monitors for the model simulation. We use the following statistics calculated using the simulated and observed metrics Y k (j omitted) and O k to evaluate the model-observation performance. Normalized Mean Bias (percent): B% = N k = 1 ( Y N k k = 1 O O k k ) 100% Weighted root-mean-squared error (WRMSE): N 2 N ( Y E: = k O k ) 1 WRMSE 2 2 k= 1 σ k k= 1 σ k Sum of the weighted squared errors (Sum): Sum = N k = 1 ( Y k O k ) 2 σ k 2 43

49 Product of the standard deviations: N N P = ( 2π ) σ Relative likelihood: k= 1 k L * N 1 ( Y ) = exp k O k 2 2 k = 1 σ k 2 = exp( Sum / 2) The following two tables show how the four metrics compare in evaluating the performance of various pseudo model cases in predicting the pseudo observations. Recall that the pseudo model cases were developed by applying Equation 5.1 with pre-assigned levels of model bias and error Bias, ζ ) to the base ozone field, and the pseudo observations were developed by applying ( j j random Gaussian errors of σ = max( 20,25% O ) to that same field. Ideally, a metric would k assign greater likelihoods (L * ) to pseudo models with lower bias and error, though it is unclear what level of spread in likelihoods is optimal. It would also be hoped that a metric would assign equal weights to two models assigned to have the same error and biases of equal magnitude but opposite sign. As can be seen, the larger k Bias j and/or ζ for a case, the larger percentage bias, weighted root- j mean-squared error, and sum of the weighted squared errors between the pseudo case concentrations and pseudo observations, and the smaller relative likelihood of the case. Comparing the different metrics, we see that the fewer data points (N) used by a metric, the narrower the spread of likelihoods assigned to the model cases (see Column L* in Tables 5.2 and 5.3). For example, metrics A and D indicate that the relative likelihoods of the model cases range by more than 7 orders of magnitude, whereas metric B shows a range of only 1 order of magnitude. This occurs because the more data points that are considered, the greater confidence that Bayesian analysis will place on a better performing model being the correct one. 44

50 Table 5.2. Statistical measures and likelihoods for pseudo model cases using metrics A and B. Metric A (N=31) Metric B (N=3) Bias j, ζ j B% E Sum L* B% E Sum L* -20%, 0% E %, 0% E %, 0% %, 0% %, 0% E %, 0% E %, 5% E %, 10% E %, 20% E %, 30% E E %, 40% E %, 50% E %, 30% E %, 40% E %, 50% E %, 30% E %, 40% E %, 50% E %, 20% E %, 20% E %, 20% E %, 20% %, 20% E %, 20% E

51 Table 5.3. Statistical measures and likelihoods for pseudo model cases using metrics C and D. Metric C (N=12) Metric D (N=34) Bias j, ζ j B% E Sum L* B% E Sum L* -20, 0% E E-7-10, 0% E-2-5, 0% , 0% , 0% E-2 20, 0% E E-7 0, 5% E , 10% E , 20% , 30% E-3 0, 40% E-5 0, 50% E-8 10, 30% E-7 10, 40% E E-11 10, 50% E E-15 20, 30% E E-15 20, 40% E E-20 20, 50% E E-26-20, 20% E E-4-10, 20% , 20% , 20% E-3 10, 20% E-5 20, 20% E E-11 The results for Metric A are especially important to consider, since it most closely approximates the manner in which SIP models are often evaluated. EPA-recommended SIP modeling methodology focuses on results on days in which ozone at a monitor was observed to be above a particular concentration threshold. The use of a threshold is motivated by the fact that the most polluted days drive attainment status for ozone, which is regulated based on the annual fourthhighest concentration. However, Table 5.2 shows that, because threshold-based metrics like Metric A target the days that were observed, but not necessarily modeled, to have the highest ozone concentrations, they will tend to favor models with a positive bias. 46

52 How can a threshold-based metric be maintained in order to maximize policy relevance, without leading to skewed preferences toward positively biased models? Extensive discussions were undertaken with statisticians to explore methods for overcoming the inherent positive bias in some threshold-based metrics. As discussed in the following section, the truncated likelihood function emerged as an effective approach for use in the Bayesian analysis Bayesian and non-bayesian methods for full ensemble Metrics for BMC Analysis The Bayesian Monte Carlo analysis of the actual structural and parametric scenarios followed much of the methodology described above for the pseudo case testing, but with important adjustments due to the insights gained from those tests. Upon further consideration, it was recognized that Pseudo-case Metrics B, C, and D from the pseudo case testing (Table 5.1) are problematic, because their use of unpaired data means that model results and observations are not being compared at the same places and/or times. However, the pseudo case testing also demonstrated that for metrics such as Pseudo-case Metric 1 (Table 5.1) that apply thresholds to screen observational data, normal likelihood functions will skew the weightings in favor of biased models (Table 5.2). Thus, alternate metrics and/or alternate likelihood functions were needed. To retain the use of Metric 1 (daily 8-hr O 3 at targeted monitors when observed O 3 >70ppb), which well captures model performance for polluted monitor-days, a truncated normal distribution function was adopted to avoid the bias noted above. The truncated likelihood function computes the likelihood of ozone prediction ( Y j, k ) given that observation ( O k ) exceeds a threshold concentration (a) by the following equation: L Y j O > a = P(O > a Y) = 1 2π N N k=1 σ k exp 1 2 N O k Y j,k 2 k=1 σ k (5.4) N 1 1 erf 2 a Y j,k k=1 2σ 2 k 47

53 In addition, a new Metric 2 was adopted in order to consider average conditions across the DFW region. This metric does not include a threshold, and thus the original normal likelihood function remains applicable. Thus, in sum, the following two metrics are considered for the BMC analysis of the full ensemble comprising structural and parametric scenarios: Metric 1: Daily 8-hr ozone at each target monitor on each day, considering only monitor-days when obs > 70 ppb (N = 48) (using truncated normal function). Metric 2: Daily 8-hr ozone averaged over all sites within DFW (N = 30) (normal likelihood function). For each metric, it is necessary to identify a value of σ that characterizes the amount of uncertainty in each observation used to evaluate model results. Measurements of ozone are conducted by well established techniques, and thus instrumental error is relatively small. Additional uncertainty is caused by the use of a point observation to represent a grid-cell average concentration. One indicator of this uncertainty can be obtained by examining the variability between ozone concentrations measured by multiple monitors within the same grid cell. The 12- km modeling domain has 5 grid cells that contain 2 or more ozone monitors. Analysis by Dr. Kristen Foley at US EPA showed that the standard deviation between observed 8hr ozone values at these same-grid-cell sites ranges from 3.0 to 10.5ppb (Figure 5.2). Thus, we chose σ = 8 ppb for Metric 1 to be near the midpoint of this range. Metric 2 should have less uncertainty due to its averaging across sites, and thus σ = 5 ppb was chosen for this metric. Note that both values of σ are much smaller than those used in the pseudo data testing, which had been based on the earlier literature rather than on the domain-specific conditions considered here. 48

54 Figure 5.2. Scatter plot showing the relationship between 8-hr ozone concentrations observed at pairs of monitors within the same grid cell of the 12-km domain. The different colors for the dots represent different grid cells with more than one monitor. Figure courtesy of Dr. Kristen Foley, US EPA. Most other aspects of the BMC methodology remained unchanged, except that 3 initialization days were used rather than 5 in order to retain more days for consideration. Analysis focused on the sensitivity of DFW ozone concentrations to DFW emissions during the June 2006 episode Metric for non-bayesian Analysis The Bayesian metrics are constrained to follow Bayes Theorem and the associated likelihood functions (Eqs ). Under those metrics, the more presumably independent observations that are used, the greater the number of multiplications involved in the likelihood functions, 49

55 leading to very large spreads in relative likelihoods. However, the more general goal of weighting ensemble cases based on performance against observations can be accomplished through alternate metrics that do not invoke Bayes Theorem. By non-bayesian approaches, we refer to any other observation-based effort to assign relative weights to ensemble cases without directly invoking Bayesian approaches. Numerous metrics could be postulated for assigning weights depending on the performance evaluation statistics of interest. For example, Mallet and Sportisse (2006) used model evaluation statistics as the basis for assigning weights to their original ensemble to better predict the observed data. For the sake of analysis, a single new metric (Metric 3) was created based upon three model evaluation statistics recommended by EPA for screening the adequacy of ozone SIP models (US-EPA, 2007): (1) Mean Normalized Bias (MNB) 1 y o n i MNB = n i= 1 o i (2) Mean Normalized Gross Error (MNGE) n 1 MNGE = n i= 1 y i i i o (3) Unpaired Peak Accuracy (UPA) UPA = y max o o max o max i To ensure meaningful results, MNB and MNGE were computed for model results (y i ) when O 3 observations (o i ) were greater than the recommended threshold of 60 ppb (US-EPA, 2007). This resulted in 356 data points being considered in DFW during the June 2006 episode. Although weights could be assigned based on these statistics in any number of ways, for the sake of analysis weights for Metric 3 were computed as the inverse of the sum of these 3 measures (neglecting the signs for bias and accuracy), as shown in the equation below: Non-Bayesian Weight = 1 MNB +MNGE+ UPA (Metric 3) 50

56 Weights were then normalized to sum to 100% as in the Bayesian analyses Reduced Form Models for parametric uncertainties The approach to selecting input parameters for BMC analysis was described in Chapter 4, and resulted in the targeting of the parameters highlighted in Tables Adjusted O 3 concentrations based on the uncertainties in selected input parameters can be determined using the relationship given by Cohan et al. 2005: C j+k = C 0 + φ j S (1) j j + φ k S (1) k k + φ 2 j j 2 S j,j 1 (2) (2) + (2) φ 2 k k 2 S k,k j,k φ j φ k S j,k (5.5) where, C 0 is the modeled concentration, ϕ j and ϕ k are the perturbations in parameters j and k respectively, and S (1) and S (2) denotes first- and high-order sensitivities to parameters given in suffix. Note that in the calculation of adjusted concentrations, S j,k (2) denotes cross-sensitivity between two input parameters. We conduct Bayesian Monte Carlo simulations (sample size 1000) for selected O 3 metrics within the structural ensemble composed of the 4 selected members (Base, CB-6, MEGAN, and CB- 6+MEGAN) selected in Chapter 3. This enabled characterization of C* for various perturbations ϕ in the input parameters. Thus, the total number of Monte Carlo cases was 4000 (M = 1000 x 4). Initially, we consider that each simulation within any given structural case to be equally likely (prior probability, p(c*) = 1/M). Then we use eqs 1-3 to compute the posterior probabilities, p (C*) for each of the two ozone metrics. Testing with a parametric sample size of 10,000 showed that the larger number of samples did not substantially influence the posterior distributions. Finally, we assume that the posterior probabilities developed from the model output for ozone concentration can also be applied to obtain Bayesian estimates of ozone responses to emission changes (sensitivity results) and of input parameter values as well. To characterize adjusted O 3 sensitivity due to uncertainties in input parameter j, we use the Reduced Form Model (RFM) given by Digar and Cohan (2008),

57 S (1) j = 1 + φ j S (1) j + φ j S (2) (2) j + k φ k S j,k (5.6) where, ϕ j denotes the perturbation (factor of uncertainty) in parameter j. Note that in the calculation of adjusted sensitivities, S (2) j,k denotes cross-sensitivity between an input parameter and the control scenario (DFW ANO x or DFW AVOC). In the RFMs for both concentrations and sensitivities, the value of ϕ j is restricted to within a 2σ range, to avoid extreme values of input parameters which would extend beyond the reliability of the RFM equations. 52

58 6. Bayesian and non-bayesian Monte Carlo analysis: Results and Discussion This chapter presents results for the Bayesian and non-bayesian Monte Carlo analyses of the structural ensemble and of the final full ensemble. The full ensemble allows the parameters targeted in Tables to vary within the four selected structural scenarios Base Case (B), CB-6 (C), MEGAN (M) and CB-6 with MEGAN (C+M). Monte Carlo randomly samples 1000 sets of input parameter values from their a priori lognormal probability distributions, with σ taken from Table 4.1 and truncation applied at ±2σ to avoid extreme values. Each set is paired with each of the four structural scenarios, resulting in 4000 cases that are originally assumed to be equally likely. The Bayesian analysis then weights each of the 4000 cases based on its performance in simulating observed O 3 concentrations, yielding a posteriori probability distributions not only for the targeted input parameters but also for the simulated O 3 -emission sensitivities. Differences between the a posteriori and a priori probability distributions for input parameters may highlight potential changes to inputs that could be investigated in further research. Meanwhile, the a posteriori sensitivity results will provide observation-adjusted expectations for the amount of air quality improvement that would result from emission controls. Due to shortcomings observed in the Bayesian results, results for non-bayesian Metric 3 are also presented Bayesian and non-bayesian probability distributions of input scenarios and parameters Section discussed the selection of two Bayesian metrics and one non-bayesian metric for weighting each model case based on its performance in simulating observed O 3 concentrations. To recap, Metric 1 compares daily 8-hr O 3 at each of three targeted DFW monitors when observed O 3 >70ppb (N = 48), whereas Metric 2 considers average 8-hr O 3 concentrations across all sites on each day (N = 30). Non-Bayesian Metric 3 considers performance statistics for 8-hr O 3 concentrations across all DFW monitors and days (N=356) based on aggregate statistics. Figure 6.1 shows the performance of each of the structural scenarios, with unperturbed input parameters, in simulating each of the Bayesian observational metrics. Three of the four scenarios 53

59 tend to underpredict Metric 1 on most monitor-days, whereas the C+M scenario yields unbiased predictions but with considerable scatter. Underpredictions of Metric 1 may in part reflect the 70 ppb threshold that is applied to observations in this metric. For Metric 2, the two cases with CB- 6 chemistry (C and C+M) yield the least biased predictions, whereas the other two cases tend to underpredict O 3 in the DFW region. (A) (B) Figure 6.1: Boxplots showing ozone differences for the 4 structural members based on (A) Metric 1 (B) Metric 2, with input parameters set at default values. Bayesian analysis is first conducted for the structural-only ensemble (i.e., 4 structural cases with unperturbed input parameters) under the two metrics. As discussed in Chapter 5, the truncated likelihood function (Eq. 5.4) is applied to assign weights to Metric 1, in order avert the positive bias that can arise due to the metric having a threshold (a=70 ppb). The normal likelihood function (Eq. 5.2) is applied to Metric 2. Since the CB-6 cases simulate higher O 3 levels in DFW that better matched the metrics, these cases dominate the weightings (Table 6.1, w/o parametric results). Metric 1 shows that pairing MEGAN biogenics with CB-6 yields the best results, whereas Metric 2 favors the original biogenic inputs. The strong differences between probabilities arise from the fact that Eqs. 5.2 and 5.4 essentially multiply together likelihoods evaluated against each of the N observations. Incorporating parametric uncertainties dramatically shifts the a posteriori weightings among the structural scenarios (Table 6.1, w/ parametric ). For Metric 1, the rankings are flipped, with the 54

60 CB-05 cases (M and B) now preferred over CB-6. For Metric 2, CB-6 chemistry remains preferred, but with MEGAN rather than GloBEIS biogenic emissions. Again, since Bayesian Eqs. 5.2 and 5.4 essentially multiply together N likelihoods, there are enormous spreads in the weightings of the 4000 cases under Metrics 1 and 2, with most of the weight placed on a handful of cases (Figure 6.2). Non-Bayesian Metric 3 yields more even spread among the cases (Figure 6.2) and thus a flatter spread among the structural scenarios (Table 6.1). Table 6.1: Posterior probability of the structural ensemble. The structural only results consider only the four cases with their default inputs; the with parametric results consider the full ensemble of 4000 cases (4 structural * 1000 Monte Carlo samplings of parameters). a posteriori probabilities BASE CB6 (C) MEGAN (M) C+M Metric 1 (N = 48) structural only 0.00% 16.34% 0.00% 83.66% w/ parametric 14.91% 5.26% 65.01% 14.82% Metric 2 (N = 30) Metric 3 (non-bayesian, N=356) structural only 0.19% 80.06% 0.16% 19.58% w/ parametric 0.00% 25.32% 0.00% 74.68% w/ parametric 21.63% 29.57% 21.42% 27.39% 55

61 Figure 6.2. Weights assigned to the 4000 members of the full ensemble under Bayesian Metrics 1 and 2 and non-bayesian Metric 3. Note for Metric 1 and 2 a handful of cases receive most of the weight, and most cases receive near zero weight; weightings are more dispersed in Metric 3. (Kristen Foley, US EPA, assisted with this image). How could the rankings of structural scenarios in Table 6.1 differ so radically with the inclusion of parametric variability? This could only occur if there are perturbation values of the input parameters which, when paired with a structural scenario, help it far better match the observed data. We closely examine the prior and posterior input distributions within each of the structural scenarios to investigate the cause of the flip in Bayesian weights (Figures 6.3 and 6.4). The blue lines in Figures 6.3 and 6.4 depict the probability densities for the 1000 Monte Carlo cases randomly sampled from the truncated lognormal a priori probability distributions of each input parameter. The dashed lines show the a posteriori probability distributions from Bayesian weightings within each of the structural scenarios, and the solid red lines show the final a posteriori distributions resulting from joint consideration of the full 4000 case ensemble. For Metric 1, Bayesian weightings under all the scenarios tended to prefer higher levels of BC(O 3 ) and photolysis rates (i.e., 1+φ>1.0), and lower levels of R(OH+NO 2 ) (i.e., 1+φ<1.0) (Table 6.2 and Figure 6.3). All of these changes tend to favor higher O 3 concentrations. Results were 56

62 ambiguous across the structural scenarios for scaling biogenic VOC emissions and R(NO+O 3 ). However, the Bayesian analysis favored scaling down the NO x emission inventory for the CB-6 cases (C and C+M), but scaling it up for the CB-05 cases. The boost in E(NO x ) for the CB-05 cases allowed them to overcome their negative bias that had been documented in Figure 6.1, and to actually outperform the CB-6 cases. 57

63 ENO X EBVOC R(photo) R(NO 2 +OH) R(NO+O 3 ) BC(O 3 ) Figure 6.3: Prior and posterior distributions of input parameters under Metric 1. 58

64 ENO X EBVOC R(photo) R(NO 2 +OH) R(NO+O 3 ) BC(O 3 ) Figure 6.4: Prior and posterior distributions of input parameters under Metric 2. 59

65 For Metric 2, scaling up the ozone boundary conditions and photolysis rates was again preferred (Figure 6.4 and Table 6.2), which tends to raise O 3 concentrations. However, scaling up R(NO 2 +OH) became preferred under Metric 2, yielding an opposite tendency. Again, the CB-05 cases tended to prefer scaling up the NO x emissions inventory to overcome their initial low bias in O 3 predictions, whereas the CB-6 cases performed better with NO x scaled down (for CB-6 alone) or held near its base levels (for CB-6+MEGAN). The MEGAN cases (M and C+M) tended to prefer scaling down the BVOC emissions inventory under Metric 2, counteracting the larger BVOC inventory of MEGAN compared to the default GloBEIS (see Figure 2.3 and Table 2.2). Even small changes in the input parameter distributions can cause major changes in the likelihood weightings assigned to the cases, due to the nature of the likelihood functions applied here. That is because, as noted earlier, the functions essentially multiply the likelihoods for each of the observation points, accentuating differences. This is illustrated in Figure 6.5, which depicts the performance of each structural case against observations for Metric 2. In the top plot, note that under default parameter settings, the CB-6 and C+M cases do perform better than the other cases, especially during the first two weeks of the episode. However, it is questionable whether the outperformance in Figure 6.5(top) merits over 99% of the Bayesian weighting being placed on the two cases as shown in Table 6.1, or that the CB-6 case is 4 times more likely than the very similar C+M results. Metric 3 yielded much flatter weightings. The bottom plot of Figure 6.5 shows an initial visualization of how the parametric adjustments affected performance of the CB-6 and C+M cases under Metric 2. It applies the posterior mean of the input parameters that were derived in each of those cases, to provide a rough approximation of the performance of these two scenarios within the full ensemble. Close inspection comparing the bottom and top plots shows that the parametric adjustments do indeed slightly improve how well the CB-6 and C+M cases match the observations. Again, however, it appears unlikely that the difference between the cases really merits the three times greater likelihood placed on C+M in the full ensemble (Table 6.1). Alternate approaches to the metrics and likelihood functions will need to be considered in further research. 60

66 8-h Ozone Concentrations, ppb OBS BASE CB-6 MEGAN C+M 30 6/2/2006 6/7/2006 6/12/2006 6/17/2006 6/22/2006 6/27/2006 7/2/ h Ozone Concentrations, ppb CB-6_w_posterior_mean C+M_w_posterior_mean OBS /2/2006 6/7/2006 6/12/2006 6/17/2006 6/22/2006 6/27/2006 7/2/2006 Figure 6.5: Daily 8-hr ozone concentration observations averaged over all 20 monitors in DFW (i.e., Metric 2; dash-dotted line), compared to each of the structural cases under default parameters (top plot) and to the C and C+M cases under their mean settings of input parameters resulting from the Metric 2 Bayesian analysis (bottom plot). Table 6.2 summarizes the a posteriori scaling factors for the input parameters derived from the full ensemble under the three metrics. Under both Bayesian metrics, the weightings tended to prefer higher photolysis rates and ozone boundary conditions in order to avert the ozone underpredictions that occurred under default parameters. However, the results for other reaction 61

67 rates and for NO x emissions are ambiguous. The parametric scaling factors for BVOC in Table 6.2 cannot be reliably interpreted, since they aggregate across scenarios that used different biogenic emissions models; while the scenario-specific results (dashed lines in Figures 6.3 and 6.4) avoid that problem, they yielded conflicting signals between the two metrics. Non-Bayesian Metric 3 placed less varied weights on the cases (Figure 6.2), and thus did not show such wide perturbations in the input parameters, with all of the mean values within 1σ of the original values (Table 6.2). Metric 3 did tend to scale up NO x emissions, to compensate for the slight underprediction of ozone by the base model. Table 6.2: Comparison of weighted distributions of input parameter scaling factors for the full ensemble of cases based on the 3 metrics. a posteriori Input Parameters mean ± 1σ Metric 1 Metric 2 Metric 3 ENO X 1.05 ± ± ± 0.25 EBVOC 1.07 ± ± ± 0.25 R(photolysis) 1.06 ± ± ± 0.08 R(NO 2 +OH) 0.89 ± ± ± 0.27 R(NO+O 3 ) 0.97 ± ± ± 0.08 BC(O 3 ) 1.23 ± ± ± Ensemble Evaluation Do the weighted ensembles outperform simple equal weighting of cases in representing DFW ozone observations? Ensemble accuracy is tested by evaluating the root mean squared error (RMS), normalized mean bias (NMB) and the correlation of the ensemble mean with the observations. These statistical measures are represented in the equations below: RMSE = N k= 1 ( y o ) k N k 2 62

68 NMB = N k= 1 ( y o ) N k k= 1 correlation = o k k N k= 1 N ( yk y) ( o o k ) 2 N ( yk y) ( o o k ) k= 1 k= 1 where N is the number of observation data (site/days); y k represents the ensemble mean prediction for k th observation o (k = 1, 2, 3,, N); y denotes the average value for all the N k mean ensemble predictions; and o is the mean observation over all site/days. The accuracy test statistics, evaluated based on 8-hour ozone concentrations for all monitors within the DFW regions and all days of the June 2006 episode (excluding initialization), are provided in Table 6.3 and Figure 6.6. Even before application of weights, the equal-weighted ensemble already outperforms the deterministic base case, in part because it includes runs with the CB-6 mechanism that help correct for the slight low bias of the base model within the DFW region. Note in Table 6.3 that Metric 1 slightly overcorrects the original underprediction of ozone, and thus does not improve overall accuracy. This may reflect the fact that Metric 1 considered only monitor-days above the 70 ppb threshold, and thus placed large weights on cases that overpredict low ozone days. Although its median result most closely matched observation, the bias arises from the high cases (Figure 6.6). Metric 2, which considered average 8-hour ozone across DFW monitors on all episode days, led to better ensemble accuracy, reducing RMS by 10% and improving the bias and correlation. Metric 3 placed greater weights on CB-6 and high ENO x cases to correct for the negative bias of ozone predictions in the DFW region, but did not improve performance in terms of RMS or correlation (Table 6.3). 63

69 Table 6.3. Statistical evaluation of the original and weighted ensembles (i.e, the 4000 cases), evaluated against 8-hour ozone at all sites/days within DFW during the June 2006 episode. Statistics Base Case (deterministic) Equal weighted full ensemble Bayesian (Metric-1) Bayesian (Metric-2) Non- Bayesian (Metric-3) RMS (ppb) NMB (%) Correlation Figure 6.6. Boxplot evaluating model performance against 8-hr ozone at all site-days within DFW for the June 2006 episode. The Talagrand diagram, popularly known as the rank histogram (Talagrand et al., 1997), is a statistical tool to assess the measure of differences in the ensemble predictions (spread). The ensemble is distributed into (B + 1) bins, where B = number of ensemble predictions (in our case B = 4000). For each of the N observations (site/days), the ensemble predictions are ranked along with the observed value to find out the bin in which the observation is falling. A rank histogram 64

70 is then constructed by tallying over these N site/days and plotting the frequency of the rank of the observation. A rank histogram therefore evaluates whether the model-ensemble is able to predict the actual observations such that the occurrence of the observation within each bin is equally likely, and a flat rank histogram would indicate that the ensemble has the correct spread (rank uniformity). For the prior full ensemble with equal weights (Figure 6.7, top plot), the rank histogram shows an underforecasting bias, reflected in the preponderance of observations that fall on the right of the histogram, above the majority of the model cases. The rank histogram also shows the prior ensemble spread to be too narrow (underdispersive), as reflected in the U shape. Note both the large first bin, which shows that many observations fall below most or all of the model cases (see Figure 6.5, which showed episode days around June 21 and July 2 to have observed ozone lower than model results), and the large bins toward the right. For the a posteriori ensembles, the U shape of the rank histograms (over-confidence) becomes even more pronounced (Figure 6.7, middle and bottom plots). That is because, as had been noted in Figure 6.2, the Bayesian analysis for each metric placed most of the weight on a handful of cases. Thus, the bulk of each weighted ensembles lie above some observations (leading to the large left-most column), and above other observations (leading to the large right-most column). Metric 1 (M1) resulted in a slight positive bias (reflected by the larger leftmost bin than rightmost bin), despite the use of the truncated normal function (Eq. 5.4) to counteract its threshold. Metric 2 is essentially unbiased, but is even more over-confident, since it places more than half of its weight on 2 of the 4000 cases (see the two outlier points in Figure 6.2). Metric 3 applied a relatively narrow spread of weights to the ensemble cases (Figure 6.2). Thus, its rank histogram (Figure 6.7, bottom) retains much of the structure of the prior (equalweighted) ensemble. Its distribution is essentially unbiased but is still somewhat over-confident (under-dispersive), though not as strongly so as Metrics 1 and 2. 65

71 Figure 6.7. Rank histogram for the full ensemble with equal weights (top) and weighted by Metrics 1, 2, and 3, evaluated based on 8-hr ozone observations from DFW monitors during the June 2006 episode. (Method for images courtesy of K. Foley, US EPA). 66

72 Figure 6.7 (cont.). Rank histogram for the full ensemble with equal weights (top) and weighted by Metrics 1, 2, and 3, evaluated based on 8-hr ozone observations from DFW monitors during the June 2006 episode. (Method for images courtesy of K. Foley, US EPA) Bayesian and non-bayesian estimation of ozone sensitivities We now turn to considering how the Bayesian and non-bayesian weights affect predictions of ozone sensitivity to emissions within the DFW region during the June 2006 episode. As discussed in Chapter 3, afternoon ozone in DFW is primarily NO x -limited under base conditions, with ozone about an order of magnitude more sensitive to DFW ANO x than to DFW AVOC (Figure 3.1). As shown in Figure 6.8, NO x -limited conditions persist regardless which structural scenario is assumed. As noted earlier, MEGAN tends to enhance sensitivities to NO x because it predicts more biogenic VOC and less biogenic NO x. CB-6 favors sensitivity to VOC, though conditions remain predominantly NO x -limited under either chemical mechanism. 67

73 S (1) ANO X (ppm) S (1) AVOC (ppm) CB-6 MEGAN BASE Figure 6.8: Ozone sensitivity to NO x and VOC emission from DFW for different structural model scenarios under default settings of input parameters. Episode average results are shown for a 4-km grid resolution for the region near DFW. How do the relative sensitivities of ozone to NO x and VOC change as Bayesian and non- Bayesian analyses are applied to the full ensemble? The Bayesian and non-bayesian weights from the full parametric-structural ensemble are used to characterize the final a posteriori distribution for O 3 sensitivity to NO X and VOC emissions from DFW. Results are probed for three grid-cells corresponding to three EPA sites in DFW - two having the highest O 3 historical DVs (Denton and Eagle Mountain Lake) and one with the lowest O 3 concentrations (Kaufman) 68

74 among all the 20 DFW sites (Figure 6.9). Daily 8-hr O 3 sensitivities were averaged over the episode to have a representative value for each site. Denton Eagle Mt. Lake Kaufman Figure 6.9: Map showing locations of location of EPA Ozone Monitors (results are probed for the sites marked as Yellow). The mean (μ ) and standard deviation (σ ) of the resulting posterior distribution of values Y j can be computed by M μ = j=1 (Y j p j ) (6) σ = M Y j μ 2 j=1 p j (7) where p j denotes the posterior probability for the j th iteration and M = Table 6.4 summarizes the statistics for the a priori and a posteriori distributions of O 3 sensitivity to ANO X and AVOC emission from DFW. We find that at all three sites, for Metric 1, the posterior mean for O 3 sensitivity to ANO x always exceeds the prior mean; whereas, for Metric 2, it is just the opposite. In other words, under Metric 1, Bayesian weighting enhances the relative importance of anthropogenic NO x emission controls compared to VOC controls. On the contrary, Metric 2 makes O 3 slightly more sensitive to VOC and less sensitive to NO x. Under each metric, 69