Calibration of numerical model output using nonparametric spatial density functions

Transcription

1 Calibration of numerical model output using nonparametric spatial density functions Jingwen Zhou 1, Montserrat Fuentes 1, and Jerry Davis 2 1 North Carolina State University, Department of Statistics, NC, U.S. Environmental Protection Agency May 24, 2011 Abstract The evaluation of physically based computer models for air quality applications is crucial to assist in control strategy selection. Selecting the wrong control strategy has costly economic and social consequences. The objective comparison of mean and variances of modeled air pollution concentrations with the ones obtained from observed field data is the common approach for assessment of model performance. One drawback of this strategy is that it fails to calibrate properly the tails of the modeled air pollution distribution, and improving the ability of these numerical models to characterize high pollution events is of critical interest for air quality management. In this work we introduce an innovative framework to assess model performance, not only based on the two first moments of models and field data, but on their entire distribution. Our approach also compares the spatial dependence and variability in both models and data. More specifically, we estimate the spatial quantile functions for both models and data, and we apply a nonlinear monotonic regression approach on the quantile functions taking into account the spatial dependence to compare the density functions of numerical models and field data. We use a Bayesian approach for estimation and fitting to characterize uncertainties in data and statistical models. We apply our methodology to assess the performance of the US Environmental Protection Agency (EPA) Community Multiscale Air Quality model (CMAQ) to characterize ozone ambient concentrations. Our approach shows a 75% reduction in the root of mean square error (RMSE) compared to the default approach based on the 2 moments of models and data. Key Words: Bayesian spatial quantile regression, CMAQ calibration, non-crossing quantile Corresponding author. address: jzhou3@ncsu.edu 1

2 1 Introduction Environmental research increasingly uses deterministic model outputs to understand and predict the behavior of complex physical processes, particularly in the area of air quality. As opposed to statistical models, deterministic models are simulations based on differential equations which attempt to represent the underlying chemical processes. Using a large number of grid cells, they generate average concentrations which have full spatial coverage and high temporal resolution without missing value. Ideally, such outputs would help fill the space-time gaps between traditional observations. For instance, inference combining information from simulations with field data are deemed to provide a complete map or real physical system. However, the reality is that the outputs are only estimated, and residual uncertainty about them should be recognized (Kennedy, et al., 2001; Paciorek, et al., 2009)[1][2]. The various sources of uncertainty are classified as low quality of emissions data, model inadequacy and residual variability (Kennedy, et al., 2001; Paciorek, et al., 2010; Fuentes, et al., 2005; Lim et al., 2009)[1][2][3][4]. As a result, to obtain subsequent predictions from the model it may be necessary first to calibrate the model, given sparse observations and complicated spatio-temporal dependences. Besides scientific studies, model-based predictions are also used to assess current and future air quality regulations designed to protect human health and welfare (Eder, et al., 2007)[5]. Indeed, the evaluation of computer models is crucial to providing assist in control strategy selection. Selecting the wrong control strategy has costly economic and social consequences. The objective comparison of the means and variances of modeled air pollution concentrations with the ones obtained from the observed field data is the common approach of model performance. However, the model outputs and the observations are on different spatial scales; this is referred to as change of support problem. The measurements are made at specific locations in the spatial domain, while modeled concentrations are recorded as averages over grid cells (Eder et al., 2007)[5]. Thus the two data sources are not directly comparable. To resolve such incommensurability, downscaling methods have been widely used to assess and calibrate numerical models. For example, Berrocal et al. (2010) propose a univariate down- 2

3 scaler using a linear regression model with spatially-varying coefficients, thus developing a spatial-temporal model that will allow ozone level to be predicted at unmonitored sites[6]. Although downscaling techniques provide computational feasibility and flexibility, this approach may be questionable for two main reasons. First, ozone data are always right-skewed, which implies that the assumed Gaussian models may underestimate the tail probability. In fact, the US Environmental Protection Agency (EPA) ozone standards are based on the fourth highest day of the year (97.5 th quantile), thus improving the ability of downscaling models to characterize high pollution events is thereby of critical inportances for air quality management. Second, since the context-specific outputs are treated as if they were known, the subsequent plug in calibrations take no account of the model s spatially-correlated uncertainty (Paciorek, et al., 2009)[2]. For characterizing the tail probability, quantile regression is an important tool and has been widely used in recent literature(koenker, R. 2005)[7]. From a Bayesian point of view, Kozumi et al., (2011) develop a Gibbs sampling algorithm based on a location-scale mixture presentation of the asymmetric Laplace distribution[8]. Despite its efficiency in practice, this method only generates individually estimated functions, but is lack of adjustments through various quantile levels between two data sources. In addition, as discussed in Wu et al., (2009), Bondell et al., (2010) and Tokdar et al., (2010), the quantile curves can cross, leading to an invalid distribution for the responses; thus, a simultaneous analysis is essential to attain the true potential of the quantile framework[9][10][11]. To achieve this purpose, the stepwise approach, linear programming and interpolation of monotone curves have been used to simplify the computationally challenging due to the associated monotonicity constraints. Particularly, Reich et al. (2010) applied a nonlinear monotonic regression model to the sample quantile functions, followed by the transformation of the outputs based on the obtained regression functions to calibrate the model distributions with observations[12]. In their studies, the regression functions are expressed as a weighted sum of a set of basis functions with constraints, thus making transformations between modeled and observed quantiles to be monotonic. Nevertheless, this approach does not consider temporal effects on the distribution s upper tail probability. Therefore, it becomes necessary 3

4 to not only flexibly model the individual regression functions subject to the non-decreasing constraints but also to characterize spatio-temporal dependency. When there is uncertainty about the distribution, the Bayesian nonparametric methods are useful; however, the non-fully specified likelihood making a posterior density hard to calculate. To solve this problem, Lavine M. (1995) introduced a substitution likelihood approach which split quantile values into separate bins, and the number of corresponding data counted within the bins obey a multinomial distribution[13]. In 2005, Dunson et al. apply this approximation in a Bayesian framework, and the posterior densities are characterized by a vector of quantiles and truncated priors[14]. These approximating methods have only focused on discrete quantile levels. Further development of these proposed evaluation procedures is needed. In this paper, we are concerned with the discrepancy due to the shape of the distributions, especially the tails. In order to compare the density functions of numerical models and field data, we estimate the spatial quantile functions for both models and data, and we apply a nonlinear spatial monotonic regression approach to the quantile functions. We use a Bayesian approach for estimating and fitting in order to characterize the uncertainties in the data and statistical models. The paper is organized as follows. In section 2, we present the monitoring data and the numerical model output. In section 3, we provide the calibration procedure. We discuss the Bayesian framework in section 4, by first modeling CMAQ quantile processes, and then adjusting spatio-temporal misalignment in the distributions. In section 5 we conduct a simulation study for comparing our method with the classic quantile regression spline. Section 6 presents analysis of a spatiotemporal ozone data set over eastern US. We end with some conclusions and final remarks, presented in Section 7. 2 Data description We use maximum daily 8-hour average ozone concentrations in parts per billion (ppb) from n = 68 sites covering the eastern U.S. from May, 1 st, 2002 to September, 30 th 2002, which were obtained from the EPA Air Quality System (AQS) and can be acquired from the following 4

5 website: Another source of data is the 2002 base-run simulations from the Community Multiscale Air Quality (CMAQ) model. CMAQ is a multi-pollutant, multi-scale air quality model that uses state-of-the science techniques for simulating all atmospheric and land processes that affect the transport, transformation, and deposition of atmospheric pollutants and their precursors on both regional and urban scales. It is designed as a modeling tool for handling all the major pollutant issues based on a whole atmosphere approach. In this study, four annual (2002 to 2005) CMAQ model runs were completed over the eastern U.S. using a 12 km by 12 km horizontal grid. We use the ozone monitoring stations as the spatial unit and extract climate data from the grid cell containing the ozone monitoring station. Additional information and a complete technical description of the CMAQ model are given by Byun and Schere (2006)[15]. The range of the CMAQ forecast data is quite similar to the range of the ground level ozone monitoring data. To compare the CMAQ forecasts with the observed monitoring data, we plot the sample quantile levels for the 90 th percentile for our data set over US in Figure 1. Specifically, we extract data from a randomly selected site (the 59 th site is marked on the map as ), and investigate the histogram, sample quantile and density function of both observed and CMAQ data on this site. The observed ozone data have a heavier tail than CMAQ data. Also, modeled ozone data agree quite well with the observations at its 50 th percentile, but present an overall lower 90 th percentile level over our study region. This implies that there is unknown discrepancies in the CMAQ forecasts and appropriate calibration is needed. 3 Spatial-quantile calibration model This section serves to introduce the notation used throughout this paper. Let s = (s 1, s 2 ) be a point measured by EPA monitors using the latitude/longitude coordinates and let B s be the associated 12 km CMAQ simulated grid cell in which s lies. At each overlapping location s and grid cell B s, we assume that the observed Y (t, s) and CMAQ ozone Z(t, B s ) are available and re-scaled according to CMAQ s minimum and range value. At location s let u t = (u t1, u t2,..., u tj ), where u t1 1 and u tj is the B-spline of t with df=j-1, j=2,..., 5

6 J. Subsequently at each s, we model the spatial τ th quantile process of the observations given u t as q Y (τ u t, s) and the CMAQ simulations as q Z (τ u t, B s ). Let α τ be a vector of calibration parameters which are assumed to variate across the quantile level τ. In general, the calibration model can then be summarized explicitly: q Y (τ u t, s) = G(q Z (τ u t, B s ), α τ ) (1) Here, G is an unknown function assumed smooth in s and monotonic in τ. Now suppose there are n s points and n B grid cells. Let Q Y (τ x, s) and Q Z (τ x, B s ) be the column vector formed by vectorizing these n s EPA observations. In addition, let A τ,s denote the vector of all calibration parameters. By combining the information for all points and grid cells, the ozone calibration model can be expressed as: Q Y (τ u t, s) = G(Q Z (τ u t, B s ), A τ,s ) (2) where A τ,s is assumed to be a Gaussian process, and its spatial covariance is characterized by: Cov(A τ,s, A τ,s ) = σ 2 τexp( s s ρ τ ) (3) The interpretation of this non-parametric model is that the quantile process of Y is monotonic after an approximate change in the τ system. Hence, if we take Q Z as a mapping from a R 2 τ t system to R 3 τ t Q Z quantile process system, then G projects τ t Q Z to the observed τ t Q Y quantile process system. In other words, instead of using the regression methods based on the 2 moments of models and data, we are aimed at calibrating CMAQ and observations through their underlying spatial quantile processes (see Figure 2). 4 Methodology We first present an overall Bayesian framework for our calibration model, then describe a monotonic regression to characterize the quantiles, using both observed and CMAQ data with spatially-varying coefficients. This model can be used for annual τ th quantile of grid cells or monitoring sites if of interest. Finally, we extend it by adjusting the smoothed temporal trend to handle the spatio-temporal calibration refer to the entire distribution. 6

7 4.1 Bayesian framework for spatial-quantile calibration We regard the quantile processes Q Y ( ), Q Z ( ) and A as random variables. The calibration system (2) now expresses the relationship between the two data sources, especially in terms of their quantile level τ. Using a Bayesian framework, the posterior probability of A and Q Z given the measured quantile process Q Y can be expressed as: f(q Z, A Y ) = f Y (y Q Z, A)π(Q Z, A) f(y ) (4) Thus, we express the τ th quantile process of Y as a function of Q Z and unknown parameters A. Note that the calibration system G is implicitly incorporated in the likelihood function f Y (y Q Z, A). Our purpose is to estimate, based on the data Y t,s, the transformation G, the spatial variance σ and the parameters of the spatial structure ρ. We first express the CMAQ quantile Q Z as: Q Z (τ u t, B s ) = u t β 0,B s + β 1,Bs (τ) u β t Bs (τ) (5) At the grid cell B s, β 0,Bs adjusts the overall temporal effect and β 1,Bs (τ) represents the quantile process, respectively. The full conditional of β Bs (τ) is: π( β Bs (τ) Z) f Z (z β Bs (τ))π( β Bs (τ)) (6) Then, the τ th CMAQ quantile of the predictive posterior distribution (Yu et al., 2001)[16] is given by: f(q Z Z) f(q Z (τ u t, B s ); β Bs (τ))π( β Bs (τ) Z) (7) By integrating (7) in (4) and combining with the A prior assumption (3), the posterior of calibration parameters to be maximized is: f(a Y, Z) π(a) f Y (y Q Z, A)f(Q Z Z)dQ Z (8) The algorithm structures from (4) to (8) are summarized in Figure 3. Based on this Bayesian framework, we discuss the detailed conditions to obtain a valid quantile process and a proper posterior distribution in the following sections. 7

8 4.2 System calibration and spatial quantile processes Our model is motivated by a desire to improve the calibration strategy, especially correcting outputs at extreme monitoring events. In this section, we briefly consider how the calibration problem can be posed in the above Bayesian framework, particularly, how to determine likelihood of both CMAQ and observed data via Q Z (τ u t, B s ) and G(Q Z (τ u t, B s ), A τ,s ) Spatial-quantile process for CMAQ In general, all the points s falling in the same 12 km square region are assigned the same CMAQ output value. However, the model outputs and the observations are incomparable due to such different spatial scales. Therefore, we link the spatial process in the model to a point level process before using it for calibration. We model the quantile function from the CMAQ models as follows: Q Z (τ B s ) = β(τ, B s ) (9) where the parameter function β(τ, B s ) are the spatially-varying coefficients for the τ th quantile level. Because Q Z (τ) is nondecreasing in τ given a grid cell B s, the process β(τ, B s ) must be constructed as a monotonic function as: β(τ, B s ) = I(τ) β(bs ) = β 0 (B s ) + M I m (τ)β m (B s ) (10) To achieve the monotonic properties, truncate power functions and polynomial basis functions are widely used in the recent literature ( Cai ( et al., ) 2007; Reich et al., 2010)[17][12]. M For instance, Berstein basis polynomials I m (τ) = τ m (1 τ) M m reduces the complicated monotonicity constraints to a sequence of simple constraints β m β m 1 0, for m m = 2,..., M (Reich et al.(2010))[12]. However, polynomials do have a limitation: changing the behavior of β(τ, B s ) near one value τ 1 has radical implications for its behavior for any other value τ 2. Thus, when M is small, the polynomial transformation which is satisfactory for the central portion of the distribution, might exhibit unpleasing features in the tails (Ramsay, 1988)[18]. Choosing a large M helps but the computing burden becomes heavy. This poses m=1 8

9 the problem of how to retain flexibility, while leaving the function elsewhere constrained as desired. In this paper, we model the function I using monotone spline regression by piecewise polynomials. In particular, we focus on the integrated splines I m, or I-splines for the sake of brevity (Ramsay J. O., 1988; John Lu et al.)[18][19]. For a simple knot sequence {γ 1,..., γ M+h }, M is the number of free parameters that specify the spline function having the specified continuity characteristics, and h is the degree of piecewise polynomial I m. For all τ, there exists m such that γ m τ < γ m+1. For application to the important case where k=3, let: I1 (τ γ m ) (γ m+2 γ m+1 ) ; I 2 (τ γ m+1) 2 (γ m+3 τ) 2 (γ m+3 γ m+1 )(γ m+2 γ m+1 ) ; I3 (γ m+3 τ) 3 (γ m+3 γ m+1 )(γ m+3 γ m )(γ m+2 γ m+1 ) (τ γ m ) 3 (γ m+3 γ m )(γ m+2 γ m )(γ m+2 γ m+1 ). The I-spline I m will be piecewise cubic, zero for τ < γ m and unity for τ γ m+3, with the direct expressions: 0, if τ < γ m (τ γ m ) 3 (γ m+1 γ m )(γ m+2 γ m )(γ m+3 γ m ), if γ m τ < γ m+1 I m (τ γ) = I1 + I2 + I3, if γ m+1 τ < γ m+2 (γ m+3 τ) 3 1 (γ m+3 γ m+2 )(γ m+3 γ m+1 )(γ m+3 γ m ), if γ m+2 τ < γ m+3 1, if τ γ m+3 As the I-spline is an integral of nonnegative splines, this provides a set of which, when M combined with nonnegative values of the coefficients β m (B s ), yields monotone splines I m (τ)β m (B s ). To ensure the quantile constraint, we introduce latent unconstrained variable β m (B s ) and take: β m (B s ) = { βm (B s ) if β m (B s ) 0 0 otherwise Therefore a model using β(b s ) induces via (10) a quantile process of Q Z (τ B s ). Without loss of generality, we choose the knots series within γ 1 = 0 and γ M+h = 1. The quantile process thus satisfies the boundary conditions: Q Z (0 B s ) = β 0 (B s ) = L z (B s ), Q Z (1 B s ) = β 0 (B s ) + 9 (11) m=1 (12) M β m (B s ) = U z (B s ) (13) m=1

10 where [L z (B s ), U z (B s )] gives the range of Z over the grid cell B s in formula (9). Here, we rescale CMAQ data on themselves at each grid cell, thus L z (B s ) 0 and U z (B s ) 1. In addition, assuming β m (B s ) have prior β m (B s ) N( β m, Σ m ), with Σ m (B s,b s ) = σ 2 m B exp( s s /ρ mb ). The full conditional distribution of π(β m (B s ) Z) are then given by f(z β m (B s ), β m (B s ) )π(β m (B s ) β m (B s ) )π(β m (B s ) ). Subsequently, the predictive posterior distribution f(q Z (τ, B s ) Z) of the the τ th CMAQ quantile is obtained by (7) Spatial-quantile calibration : from CMAQ to monitoring processes For the purpose of calibrating spatial-quantile process, we make use of monotonically increasing map η s drawing from the CMAQ predictive posterior distribution: η s (τ) d = f(q Z Z) f(q Z (τ B s ); β Bs (τ))π( β Bs (τ) Z) (14) Thus we have the observed quantiles of Y as follows: Q Y (τ Z, s) = I(η s (τ)) α(s) = α 0 (s) + M I m (η s (τ))α m (s) (15) m=1 α(s) are spatially-varying coefficients. Similar as equation (12), we introduce a latent unconstrained variable α m (s) to ensure the quantile constraints: { αm (s) if α α m (s) = m (s) 0 0 otherwise α m (s) are modeled as multivariate mean-zero Gaussian spatial process with boundary conditions: Q Y (0 Z, s) = α 0 (s) = L y (s), Q Y (1 Z, s) = α 0 (s) + (16) M α m (s) = U y (s) (17) where (L y (s), U y (s)) are the range of Y given location s. However, strict bounds on Y may not be known a priori. To satisfy that the posterior has a proper distribution (see appendix), we take a truncate likelihood: m=1 f Y (y Q Y ) = {e ω L(α 0 y) } 1(y < α 0 ) {e ω U(y (α 0 + α m ))) } 1(y > α 0 + α m ) {f Y (y Z, s)} 1(α 0 y α 0 + α m ) (18) 10

11 where ω L, ω U are known positive rate parameters and f Y (y Z, s) is the density function derived from both the CMAQ and observed quantile functions, and its computing algorithm is provided in Section The resulting likelihood has an exponential decay once the estimated quantile boundaries do not include certain observed values. Also, we assume that there exist (M+1) mean-zero unit-variance independent Gaussian processes α 0 (s), α 1 (s),..., α M (s) such that, cov(α m (s), α m (s )) = σ 2 msexp( s s /ρ ms ) and ρ ms is the spatial decay parameter for Gaussian process α m (s), m=0,1,...,m Model fitting : likelihood approximations using calibrated quantiles In this section, we focus on discussing how to obtain Y s likelihood only based on its quantile process Q Y (τ Z, s) = I(η s (τ)) α(s) and CMAQ predictive quantile η s (τ). Suppose the constraints (12) and (16) are satisfied, then τ Q Y (τ Z, s) is monotonically increasing. Hence, the process (15) uniquely determines a unconditional sampling density for Y in the form (Tokdar et al. 2010)[11]: 1 f Y (y Z, s) = Q τ Y (τ Z, s) τ=τ Z,s (y) (19) where τ Z,s (y) is the solution y = Q Y (τ Z, s) in τ, and we apply the truncated likelihood (18) to approximate the density function: when α 0 f Y (y Q Y (Z, s), η s (τ Z,s )) = {e ω L(α 0 y) } 1(y < α 0 ) {e ω U(y (α 0 + α m ))) } 1(y > α 0 + α m ) 1 { τ Q Y (τ Z, s) 1(α 0 y α 0 + α m ) τ=τ Z,s(y)} (20) y α 0 + α m, the partial log-likelihood function of f Y (y Z, s), over the monotonicity restrictions of (η s, α(s)) is defined as: log f Y (y i s) = i i = i log τ Q Y (τ s) τ=τz,s (y i ) log Q Y (τ s) η s (τ) η s(τ) τ τ=τz,s (y i ) (21) 11

12 where τ Z,s (y i ) solves y i = Q Y (τ Z, s), i = 1,2,...,n. A solution τ Z,s (y) to Q Y (τ Z, s) y = 0 can be efficiently obtained using Newton s Recursion: τ (k+1) Z,s (y) = τ (k) Z,s (y) Q Y (τ Z, s) y Q (22) τ Y (τ (k) Z,s (y) Z, s), where τ (0) Z,s is an initial value in [0, 1], and we choose the lower bound of an estimated quantile interval where y lies in our practice. The evaluations of Q Y (τ Z, s) and τ Q Y (τ Z, s) at various values of τ [0, 1] can be done by: τ Q Y (τ Z, s) = Q Y (τ Z, s) η s τ η s M = { I m (η s (τ Z,s (y)))α m (s)} { η s m=1 M m=1 τ I m(τ Z,s (y))β m (s)} (23) To simplify the notation, let D1 3 = (γ m+2 γ m+1 ) ; 3(γ m+3 η) 2 D 2 = (γ m+3 γ m+1 )(γ m+3 γ m )(γ m+2 γ m+1 ) 3(η γ m ) 2 + (γ m+3 γ m )(γ m+2 γ m )(γ m+2 γ m+1 ). Then the derivative of I-spline, η I m(η( )) consists of straightline segments as follows 0, if η < γ m 3(η γ m ) 2 (γ m+1 γ m )(γ m+2 γ m )(γ m+3 γ m ), if γ m η < γ m+1 η I m(η γ) = D1 + D2, if γ m+1 η < γ m+2 (24) 3(γ m+3 η) 2 (γ m+3 γ m+2 )(γ m+3 γ m+1 )(γ m+3 γ m ), if γ m+2 η < γ m+3 0, if η γ m+3 The steps given in equations (21) and (24) provide a fast algorithm to compute the likelihood at any given value of the parameter η (Tokdar et al., 2010)[11]. Using Markov Chain Monte Carlo (MCMC), the posterior distributions are summarized subsequently by evaluating the likelihood (20) and CMAQ distribution (14). 4.3 Spatial-temporal quantile calibration The calibration model in section 4.2 can be extended to accommodate data collected over space and time. If we denote time with t, t=1,2,...,t, u t =(u t1, u t2,..., u tj ). u t1 1 and u tj 12

13 is the B-spline of t with df=j-1, j=2,...,j. Then Q Y (τ u t, s) denotes the τ th quantiles process of observed daily 8-hour maximum ozone concentration at s and time t, while Q Z (τ u t, B s ) is the τ th CMAQ quantile levels for grid cell B s given time t. Again, we relate the 12 km CMAQ grid cell B s to each monitoring site s. We start by using quantile functions to vary with B s, u t and τ for CMAQ output, thus they give a density regression model where the temporal trend is allowed to affect the shape of CMAQ distribution. This means that: Q Z (τ u t, B s ) = u t β 0,B s + β Bs (τ) = J M u tj β 0j (B s ) + I m (τ)β m (B s ), (25) j=1 m=1 To specify monotonic constraints for Q Z (τ u t, B s ) with the temporal component u t, the nonnegativity of β Bs (τ) is required. More specifically, we introduce latent unconstrained variables βm(b s ) and take constraints as (12) in section In order to construct quantile functions of Y based on CMAQ process, we first consider the predictive CMAQ spatialquantile processes η ut,s(τ) as monotonically increasing maps from [0,1] onto itself given any location s: η ut,s(τ) f(q Z (τ u t, B s ) Z) f(q Z (τ u t, B s ); β 0,Bs, β Bs (τ))π(β 0,Bs Z)π(β Bs (τ) Z), (26) Then we have the quantiles of observed data Y as follows: Q Y (τ Z, u t, s) = u t α 0,s + α s (η ut,s(τ)) J M = u tj α 0j (s) + I m (η ut,s(τ))α m (s) (27) j m similarly as (16), we subject the monotonic spatially-variant α m (s), m = 1,..., M to the following latent variables: α m (s) = { α m (s) if α m(s) 0 0 otherwise m = 1,..., M (28) also as section 4.2.2, we assume that there exist Gaussian processes α m (s) such that, E(α m (s)) = ᾱ m (Θ α ) and Cov(α m (s), α m (s )) = σ ms 2 exp( s s /ρ ms ) and ρ ms is the 13

14 spatial decay parameter for Gaussian process α m (s). The different temporal trends between CMAQ and observed quantile process are then adjusted through the calibration parameters α 0 (s), α 1 (s),..., α m (s). 5 Simulation study For nonparametric quantile regression, the proposed Bayesian spatial quantile method (BSQ) is compared with classic quantile regression splines(cqrs). The data is given by: z (ti,s i ) = f(t i, s i ) + g(t i, s i )ɛ i (29) for the mean function f and variance function g. The time t i has a U(0,1) distribution, and ɛ i N(0, 1) with n =. Examples are given as: Example 1: Simple quantile: f(t i, s i ) = 2, and g(t i, s i )= 2, while s i s. Example 2: Temporal quantile: f(t i, s i ) = t i + sin(2πt i 0.5), and g(t i, s i )=1, while s i s. Without loss of generality, we rescale the simulated data onto [0, 1]. The results are presented in Table 1 and Table 2. In example 1, the root mean squared error RMSE = K [K 1 ( ˆβ(τ k ) q z (τ k ))] 1/2 is calculated for both the CQRS method and our Bayesian ap- k=1 proach. In order to evaluate the effects of the I splines and the truncated likelihood, we selected different interior knots and the weight parameters ω L (ω U ). The I splines having knots at (0.3, 0.7) have a better performance with less stability compared to frequentist method (see Figure 4). Example 2 is evaluated in terms of the empirical root mean intergrated squared error n RMISE = [n 1 (ˆq τ (t i ) q τ (t i ))] 1/2 for τ= 0.01, 0.1, 0.5, 0.9, and ˆq τ (t i ) is the i=1 estimated function and q τ (t i ) is the real function. The interior knots at (0.2, 0.8) provide a smaller RIMSE, especially at the upper tails (i.e., τ=0.9, 0.99). In Figure 5, we plot a time series of the simulated data and its underlying τ th curve as the real process. The quantile spline regression captures most of the variations in the data but demonstrate a 14

15 crossing problem. Our BSQ approach yields an overall trend with a wide band, indicating that a small weight parameter is preferred. Also, the more points that are included between a pair of knots, i.e.,(0.2, 0.8), the better the definition of the curve. Table 1: (Example 1) Root mean squared error ( ), with its standard error in parentheses Method Interior Knots ω L (ω U ) RMSE SE BSQ (0.3, 0.7) (0.3, 0.7) BSQ (0.4,0.6) CQRS Table 2: (Example 2) Empirical root mean intergrated squared error ( ), with its standard error in parentheses Method Knots ω L (ω U ) τ = 0.01 τ = 0.1 τ = 0.5 τ = 0.9 τ = 0.99 BSQ BSQ (0.2, 0.8) (0.038) 6.9(0.041) 7.7(0.035) 8.1(0.039) 9.6(0.047) (0.2, 0.8) (0.037) 6.7(0.032) 7.9(0.035) 8.0(0.038) 9.3(0.045) (0.3, 0.7) 0 8.6(0.035) 6.9(0.020) 9.4(0.046) 11.1(0.062) 12.1(0.070) (0.3, 0.7) (0.034) 7.0(0.020) 9.4(0.051) 10.6(0.061) 11.7(0.064) CQRS (0.038) 8.0(0.024) 6.1(0.017) 8.5(0.030) 15.0(0.042) 6 Application: calibration of eastern US ozone data To compare spatial surfaces and distributions between the observed data and the CMAQ output, we choose two data sources in the eastern US. The prior distributions of the CMAQ quantile parameters β and calibration parameters α are determined using restricted least squares with large variances. We use the Metropolis-Hastings approach for updating β, α, σm 2 B, σm 2 s, ρ mb, and ρ ms individually. The likelihood is calculated by the likelihood approximation approach of Q Y (τ Z, s) on a grid of equally-spaced τ k [0, 1]. The I-splines have interior knots at (0.2, 0.8). The weight parameters ω U, ω L are supposed to have a dense uniform distribution, and we choose a known value of 0 for the purpose of computing efficiency. 15

16 The estimated CMAQ quantile and its calibration for monitoring data are plotted in Figure 6. Both of the two spatial-quantile processes are obtained by our Bayesian algorithm. At τ= 0.05, 0.5, and 0.95, the empirical root mean integraded squared error RMISE = n [n 1 (ˆq zτ (s i ) ˆq yτ (s i ))] 1/2 is calculated. The RMISE at the 50 th quantile is equal to 7.13, i=1 while the value is for the 5 th percentile and for the 95 th percentile, respectively. The results show agreement between the distributions of CMAQ output and the monitoring data at their median level, but show large differences for the tails. Also, from the contour plot, we conclude that the CMAQ data are smoother than the observed spatial structure, indicating that the physically based numerical models can not capture both the extreme values and spatial correlations that are in the monitoring data. Due to these differences, it is critical to calibrate the CMAQ data considering its spatialquantile structure. Based on the estimated CMAQ-monitoring calibration model, a nonlinear M transformation is made to the CMAQ data using G(Z t,s, Â(τ, s)) = ˆα 0 + I m (Z t,s )ˆα m, where ˆα are the posterior estimations. range. Then we rescale G(Z t,s, Â(τ, s)) to its original Because G is a monotonic function, the quantiles of G(Z t,s, A(τ, s)) are equal to G(Q Z (τ s), A(τ, s)) = Q Y (τ s). We calculate ˆq M (τ k, s) (the sample quantiles of the monitoring data), q C (τ k, s) (the quantiles of the Bayesian calibrated data) and q L (τ k, s) (the quantiles from the linear regression model), at τ k [0.01, 0.97] and location s. The root mean squared error RMSE(ˆq M, q s) = [K 1 K k=1 (ˆq M (τ k, s) q(τ k, s))] 1/2 is calculated for both linear regression method and our Bayesian approach at each location s. Figure 7 shows maps of the above quantiles when τ = 0.95, and the difference root mean squared error DRMSE = (RMSE(ˆq M, q C s) RMSE( qˆ M, q L s)) /RMSE( qˆ M, q L s) between the linear regression method and the quantile calibration method. The differences range from -77% to 66%, and is -30% on average. The results show that 57 out of 68 (83.8%) sites have a reduced RMSE using the Bayesian calibration method. As we expected, the performance of the calibrated CMAQ model data is consistent with the performance of the monitoring data in terms of the quantile level τ. m 16

17 7 Discussion In this paper, we propose a Bayesian spatial quantile calibration model for adjusting the behavior between CMAQ model output and monitoring data. Particularly, we focus on calibrating the extreme values. Thus, instead of using the default approach based on the first two moments of the models and data, we calibrated the two data sources through their underlying quantile processes. We investigated two quantile processes: (1) estimated spatialquantiles for CMAQ; (2) the predicted monitoring quantiles based on CMAQ calibrations. We conclude that the CMAQ and monitoring data are similar around their median values, but present large differences at the upper and lower tails over eastern US. The investigated transformation between CMAQ and the observed quantile process is then applied to model output data, resulting in a calibrated series whose spatial and quantile structure is consistent with the monitoring data. Due to the different spatial scales of the CMAQ output and the observations, we assume that both the CMAQ and observed quantile processes have a spatial structure with exponential decay parameters. This assumption is made to obtain computing efficiency. More complicated spatial processes, i.e., conditional autoregressive (CAR) model for gridded CMAQ data, and spatial linear coregionalization models for calibrating spatial quantiles, will be considered in future work. Also, temporal components, known to be an important factor for ozone trend, play less of a role when taking both quantile and spatial structure into account (see Figure 8). Another approach is to consider the smoothing spline as a covariate, then evaluate its effect on the conditional distributions (see Figure 9 for the individual quantile surfaces for both the CMAQ data and monitoring data at a specific site); however, the quantile calibrations, as a tranformations of one quantile process to another simultaneously, require a valid quantile process with the non-crossing and monotonic constraints. An efficient way to calibrate this type of spatial-temporal-quantile surface simultaneously is another avenue for future work. 17

18 8 Appendix If the likelihood is given by fomula (18) and p(α) 1, then the posterior distribution of α, π(α Y ), will have a proper distribution. In other words: 0 < π(α Y )dα < (30) Proof. Suppose y (1) y (2)... y (n), and both ω L and ω U We first consider two extreme situations: (1) y i < α 0, for all y i, i=1, 2,..., n. Hence, we have y (n) < α 0 and: π(α Y )dα = n i=1 {α 0 y (n) } f Y (y i (α)π(α)dα exp{ {α 0 y (n) } i exp{ nω L (α 0 ȳ)}dα are two finite positive numbers. ω L (α 0 y i )}dα 1 nω L exp{ nω L (y (n) ȳ)} (0, ) (31) (2) Another situation is: y i > α 0 + α m, for all y i, i=1, 2,..., n. As a result, we have y (1) > α 0 + α m and: π(α Y )dα = n i=1 f Y (y i (α)π(α)dα exp{ ω {α 0 + U (y i (α 0 + α m ))}dα m αm y (1)} i m exp{ nω {α 0 + U (ȳ (α 0 + α m )}dα m αm y (1)} m 1 exp{ nω U (ȳ y (1) )} nω U (0, ) (32) In general, suppose y (1)..., y (u) < α 0 y (u+1)... y (l) α 0 + m α m <y (l+1)..., y (n) (see 18

19 Figure 10), then we have: π(α Y )dα 1 uω U exp{ ω U (uy (u) u y (i) )} i=1 1 n exp{ ω L ( y (i) (n l)y (l+1) )} (n l)ω L i=l+1 l 1 { i=u+1 τ Q Y (τ) τ=τ }dα (y (i) ) (0, ) (33) The statement is proved. References [1] M. Kennedy and A. O Hagan, Bayesian calibration of computer models, Journal of the Royal Statistical Society: Series B (Statistical Methodology), vol. 63, no. 3, pp , [2] C. Paciorek, Combining spatial information sources while accounting for systematic errors in proxies, Journal of the Royal Statistical Society, [3] M. Fuentes and A. E. Raftery, Model evaluation and spatial interpolation by bayesian combination of observations with outputs from numerical models, Biometrics, vol. 61, no. 1, pp , [4] C. Y. Lim, M. Stein, J. K. Ching, and R. Tang, Statistical properties of differences between low and high resolution cmaq runs with matched initial and boundary conditions, Environmental Modelling and Software, no. 25(1), pp , [5] B. K. Eder and S. Yu, A performance evaluation of the 2004 release of models-3 cmaq, Air Pollution Modeling and Its Application XVII, no. 6, pp , [6] V. J. Berrocal, A. E. Gelfand, and D. M. Holland, A spatio-temporal downscaler for output from numerical models, Journal of Agricultural, Biological, and Environmental Statistics, vol. 15, pp , [7] R.Koenker, Quantile Regression. Econometric Society Monograph Series, Cambridge U. Press, [8] H. Kozumi and G. Kobayashi, Gibbs sampling methods for bayesian quantile regression, Journal of Statistical Computation and Simulation,

20 [9] Y. Wu and Y. Liu, Stepwise multiple quantile regression estimation, Statistics and Its Interface, vol. 2, [10] H. D. Bondell, B. J. Reich, and H. Wang, Non-crossing quantil regression curve estimation, Biometrika, vol. 97, [11] S. Tokdar and J. Kadane, Simultaneous linear quantile regression: A semiparametric bayesian approach., In press, [12] B. J. Reich, M. Fuentes, and D. Dunson, Bayesian spatial quantile regression, Journal of the American Statistical Association, vol. In press, [13] M. Lavine, On an approximate likelihood for quantiles, Biometrika, vol. 82, [14] D. B. Dunson and J. A. Taylor, Approximate bayesian inference for quantiles, Journal of Nonparametric Statistics, vol. 17, [15] D. Byun and K. L. Schere, Review of the governing equations, computational algorithms, and other components of the models-3 community multscale air quality (cmaq) modeling system, Appl. Mech. Rev., [16] K. Yu and R. A. Moyeed, Bayesian quantile regression, Statistics & Probability Letters, vol. 54, no. 4, pp , [17] B. Cai and D. B. Dunson, Bayesian multivariate isotonic regression splines:applications to carcinogenicity studies, Journal of the American Statistical Association, vol. 102, pp , [18] J. O. Ramsay, Regression splines in action, Statistical Science, vol. 3, pp , [19] Z. Q. J. Lu and D. B. Clarkson, Monotone spline and multidimensional scaling, 20

21 Latitude CMAQ 90th quantile frequentist approach * Latitude Monitoring 90th quantile frequentist approach * Longitude Longitude Histogram of CMAQ ozone Histogram of monitoring ozone Density Density Density comparison Sample quantile Density CMAQ Monitorming data ozone 40 CMAQ Monitorming data Tau1 Figure 1: Maps of the sample 90 th quantile levels of the ozone concentration; the represents a randomly selected (i.e., 59 th ) monitoring site. We draw the maps for both observed and CMAQ data to identify their differences. 21

22 MODEL DATA Z ( t 1, B s ) 1 Z ( t 2, B s ) 2 Quantile Process for CMAQ Q, B ), Q, B ) Q u, B ) Z ( 1 u t s Z ( 2 u t s Z ( K t s Z ( t n, B s ) n System Calibration: 1. Model CMAQ Quantile System Calibration: 3. Calibrating CMAQ to Monitoring data MONITORING DATA Y t 1, s ) ( 1 Y t 2, s ) ( 2 Y ( t, s n n) Quantile Process for Observations Q ( 1, s), Q ( 2, s) Q ( u, s) Y u t Y u t Y K t Estimated Parameters Α( τ,s ) 1 Α( τ,s ) 2... Α( τ,s n ) System Calibration: 2. Link with Observed Quantile Figure 2: A process chart for spatial quantile calibration for going from CMAQ to the observations. We calibrate the original CMAQ data with the corresponding observations through their underlying spatial-quantile processes. 22

23 Spatial quantile process for CMAQ Q ( s) ( s) I ( ) ( s) I m : Monotonic I spline; Z 0 m m m 1 M Spatially variant coefficients β(s) for CMAQ Q ( s) ; Likelihood approximation by Q ( s) ; Z Z ( s) : Predictive Z posterior quantile for CMAQ A(, s) : Monotonic mapping from ( s) to Q ( s) Y Z Spatial quantile process for monitoring data M Y 0 m Z m 1 Q ( s) ( s) I ( ( s)) ( s) Spatially variant calibration parameters α(s) ; Likelihood approximation by predictive CMAQ Z ( s) and monitoring quantile QY ( s). m Figure 3: The Bayesian framework for the spatial-quantile calibration approach. The left and middle panels present CMAQ quantile and monitoring quantile estimates at the 59 th site. The right panel provides the 90 th ozone quantile over the eastern U.S. using our Bayesian spatial quantile calibration method. 23

24 Density Density Figure 4: Simulation results for the simple quantile functions in Example 1. Interior knots are placed at 0.15, 0.8 with a weight parameter equal to. CQRS Real process time time BSQ Simulated data y time time Figure 5: Bayesian nonparametric quantile (BSQ) regression from Example 2. Interior knots are placed at 0.2, 0.8 with weight parameter equal to We add a sin function to mimic the temporal trend in reality. The classic quantile regression spline (CQRS) has crossed quantile curves, which violate the concept of a valid quantile process. 24

25 Latitude th CMAQ quantile Bayesian approach Latitude th monitoring quantile Bayesian approach Longitude Longitude Latitude th CMAQ quantile Bayesian approach Latitude th monitoring quantile Bayesian approach Longitude Longitude Latitude th CMAQ quantile Bayesian approach Latitude th monitoring quantile Bayesian approach Longitude Longitude Figure 6: Quantile comparison plots. The 5 th, 50 th and 95 th quantile for the Bayesian estimated CMAQ and calibrated monitoring data. 25

26 95 th monitoring quantile 95 th monitoring quantile Frequentist approach Bayesian approach Latitude Latitude Longitude Longitude 95 th monitoring quantile DRMSE between Bayesian and linear regression Linear regression approach Latitude Latitude Longitude Longitude Figure 7: The 95 th quantile for the monitoring data, using both the quantile calibration and linear regression method. We compare the differences between the linear regression and the Bayesian quantile calibration methods in terms of the RMSE. 26

27 CMAQ temporal quantile CMAQ temporal quantile ozone frequentist approach ozone Bayesian approach time time monitoring temporal quantile monitoring temporal quantile ozone frequentist approach ozone Bayesian approach time time Figure 8: The CMAQ and monitoring temporal quantiles at site 4. Under the non-crossing constraints, ozone quantile curves show little trend for both the CMAQ models and the monitoring data. 27

28 OBS.Quantile surface CMAQ.Quantile surface Q(y) Error using packet 1 NAs are not allowed in subscripted assignments Q(y) τ τ t 0 t 20 Figure 9: Temporal quantile surfaces at the 19 th location for both the CMAQ data and Observed data. 28

29 p(y) y u y l+1 α 0 α 0 + α m y Figure 10: The likelihood approximation using estimated quantile functions. 29