PUBLICATIONS. Water Resources Research

Transcription

1 PUBLICATIONS Water Resources Research RESEARCH ARTICLE 1.1/14WR15483 Key Points: We test the efficiency and accuracy of the PCGA with implementation guidance The PCGA requires making calls to a forward model without the adjointstate method The randomized technique enables the low-rank approximation of large covariance Correspondence to: J. Lee, jonghyun@stanford.edu Citation: Lee, J., and P. K. Kitanidis (14), Largescale hydraulic tomography and joint inversion of head and tracer data using the Principal Component Geostatistical Approach (PCGA), Water Resour. Res., 5, , doi:1.1/14wr Received 1 FEB 14 Accepted 9 JUN 14 Accepted article online 13 JUN 14 Published online 3 JUL 14 Large-scale hydraulic tomography and joint inversion of head and tracer data using the Principal Component Geostatistical Approach (PCGA) J. Lee 1 and P. K. Kitanidis 1 1 Department of Civil and Environmental Engineering, Stanford University, Stanford, California, USA Abstract The stochastic geostatistical inversion approach is widely used in subsurface inverse problems to estimate unknown parameter fields and corresponding uncertainty from noisy observations. However, the approach requires a large number of forward model runs to determine the Jacobian or sensitivity matrix, thus the computational and storage costs become prohibitive when the number of unknowns, m, and the number of observations, n increase. To overcome this challenge in large-scale geostatistical inversion, the Principal Component Geostatistical Approach (PCGA) has recently been developed as a matrixfree geostatistical inversion strategy that avoids the direct evaluation of the Jacobian matrix through the principal components (low-rank approximation) of the prior covariance and the drift matrix with a finite difference approximation. As a result, the proposed method requires about K runs of the forward problem in each iteration independently of m and n, where K is the number of principal components and can be much less than m and n for large-scale inverse problems. Furthermore, the PCGA is easily adaptable to different forward simulation models and various data types for which the adjoint-state method may not be implemented suitably. In this paper, we apply the PCGA to representative subsurface inverse problems to illustrate its efficiency and scalability. The low-rank approximation of the large-dimensional dense prior covariance matrix is computed through a randomized eigen decomposition. A hydraulic tomography problem in which the number of observations is typically large is investigated first to validate the accuracy of the PCGA compared with the conventional geostatistical approach. Then the method is applied to a largescale hydraulic tomography with 3 million unknowns and it is shown that underlying subsurface structures are characterized successfully through an inversion that involves an affordable number of forward simulation runs. Lastly, we present a joint inversion of head and tracer test data using MODFLOW and MT3DMS as coupled black-box forward simulation solvers. These applications demonstrate the advantages of the PCGA, i.e., the scalability to high-dimensional inverse problems and the ability to utilize multiple forward models as black boxes. 1. Introduction In the subsurface inverse problem, spatially distributed parameters of interest such as hydraulic conductivity are characterized from a limited number of hydrogeophysical measurements like hydraulic heads, contaminant concentration history, or electrical resistivity data. Such problems are ill posed, i.e., the solution is nonunique or too sensitive to data, and subject to uncertainty, thus they are commonly evaluated within a statistical framework [McLaughlin and Townley, 1996; Carrera et al., 5; Oliver et al., 8; Kaipio and Somersalo, 7]. Among many inversion approaches, the Bayesian quasi-linear geostatistical approach [Kitanidis, 1995, 1] is a versatile method to estimate unknown subsurface parameters and quantify the corresponding uncertainty rigorously. The geostatistical approach has been applied to many engineering applications such as contaminant source identification [Snodgrass and Kitanidis, 1997], historical groundwater contaminant distribution estimation [Michalak and Kitanidis, 4], tracer data inversion [Nowak and Cirpka, 6; Fienen et al., 9], hydraulic tomography [Li et al., 7, 8; Cardiff et al., 9; Cardiff and Barrash, 11], upstream flow estimation in rivers [D Oria and Tanda, 1], atmospheric modeling [Yadav et al., 1; Miller et al., 13], and others. With rapid advances in the computing power and sensor technology, large-scale inverse modeling using large volumes of various types of data, requiring multiphysics modeling, has been an active area of research [Flath et al., 11; Bui-Thanh et al., 1; Yeh et al., 8; Liu and Kitanidis, 11]. When the geostatistical LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 541

2 1.1/14WR15483 approach is applied to high-dimensional problems, two main challenges arise: (1) as the number of unknowns m increases, the computation and storage costs regarding the dense covariance matrix increase nonlinearly and become prohibitive, () when the number of measurements n increases, the Jacobian (sensitivity) matrix needs n forward model evaluations using the adjoint-state method, which would be often the most expensive part during the inversion. Building adjoint-state models when multiple processes are involved, such as flow, reactive transport, electrical resistance, etc., can be quite challenging. Previous efforts have addressed the former, i.e., the fast computation of dense covariance matrix products, and fast linear algebra methods such as the fast Fourier transform (FFT)-based approach [Nowak et al., 3] and the hierarchical matrix approach [Saibaba and Kitanidis, 1; Ambikasaran et al., 13] reduce the computational and storage costs involving the covariance matrix products. However, these approaches still need to access the Jacobian matrix whose computational cost is prohibitive when one has to employ expensive forward simulation models and process many measurements during the inversion. In an effort to reduce the computation time associated with forward model runs, Liu et al. [13] presented the geostatistical reduced-order inversion (GROM) approach in which the m-dimensional unknown parameter space is reparameterized by the solution space of the geostatistical approach [Kitanidis, 1998], i.e., the n 1 p dimensional space spanned by the n-column space of the cross covariance between measurements and unknowns and the p-column space of the drift. In this way, only n 1 p full numerical model simulations are executed at the beginning to construct a reduced-order model and inexpensive low-dimensional forward models are used during the inversion. While n 1 p full numerical simulations need to be simulated repeatedly at each iteration when the data misfit does not decrease, it is observed that the full-model simulations at the beginning are sufficient to obtain acceptable estimates for their laboratory-scale hydraulic tomography experiments where the measurement error is greater than the approximation error. P. K. Kitanidis and J. Lee (Principal component geostatistical approach for large-dimensional inverse problems, submitted to Water Resources Research, 14) presented the theoretical framework of the Principal Component Geostatistical Approach (PCGA) that avoids the direct evaluation of the Jacobian matrix through the principal components associated with the prior covariance and the drift matrix and the finite difference derivative computation. The method is best suited for large-scale inverse problems since only a relatively small number of forward model runs is needed. In addition, the method can be easily adapted to any numerical simulation models, which is especially beneficial for joint inverse problems that require coupled numerical simulations. In this paper, we test the efficiency and scalability of thepcgatochallengingsubsurfaceinverseproblemssuch as large-scale hydraulic tomography and joint data inversion of head and tracer test data and provide implementation guidance. In addition, while our previous work showed the use of a prior ensemble as a driver to decompose the prior covariance, we here use a different randomized eigen decomposition method [Halko et al., 11] to directly decompose the prior covariance matrix with controlled accuracy. The randomized technique implements a linear combination of a randomly selected column space of a matrix to find a near-optimal lowrank approximation of the matrix in a very efficient manner and has become a popular tool due to its accuracy and easy implementation over the last decade. With the efficient matrix decomposition tool supported and maintained by the applied mathematics research community, the PCGA can greatly reduce the computational costs without losing the accuracy compared to the conventional geostatistical approach as shown later. The remainder of this paper is organized as follows. In section, we review the quasi-linear geostatistical approach and the PCGA. In section 3, we present a steady state hydraulic tomography example to examine the accuracy and efficiency of our approach. The method is then applied to a large-scale hydraulic tomography with m unknowns to test the scalability of the method. Lastly, the PCGA is applied to a joint inversion application using hydraulic head and tracer data to investigate the applicability of our method to general joint inverse problems. Coupled flow and transport simulations are carried out using the widely used software tools, MODFLOW [Harbaugh et al., ] and MT3DMS [Zheng and Wang, 1999].. Overview of the Principal Component Geostatistical Approach In this section, we review the quasi-linear geostatistical approach [Kitanidis, 1995] and the PCGA (Kitanidis and Lee, submitted manuscript, 14). The observation equation, which relates the m 3 1 vector of unknowns s to the n 3 1 vector of the data y is LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 5411

3 1.1/14WR15483 y5hðsþ1v (1) where h is the forward model mapping the parameter space R m to the measurement space R n ; v is Gaussian with zero mean and covariance R that accounts for errors in the data y and the forward model h. The prior probability of s is Gaussian with mean Xb, where X is the m 3 p known (polynomial) matrix and b is the p 3 1 unknown vector (typically p 5 1), and Q is the generalized covariance matrix. The posterior pdf of s and b are obtained through Bayes theorem and its negative loglikelihood, ln p ðs; bþ, is ln p ðs; bþ5 1 ðyhðsþþ> R 1 ðyhðsþþ1 1 ðsxbþ> Q 1 ðsxbþ () By minimizing () with respect to s and b, we can obtain the maximum a posterior (MAP) or most likely value ^s, commonly computed through an iterative Gaussian-Newton method described as follows. Starting with the latest best estimate s, we update to a new solution until s converges to ^s. First, the n 3 m Jacobian or senstivity matrix H of h at s is evaluated as: Then, assuming that the actual best estimate ^s is close to s, linearize hð^sþ (3) s5s hð^sþ5hðsþ1hð^ssþ (4) and based on the linearization, the updated solution for the next iteration is computed as s5xb1qh > n (5) where b and n are computed by solving a single linear system of n 1 p equations: " #" # " HQH > 1R HX n 5 yhðsþ1hs # ðhxþ > b (6) The equations (3 6) are repeated until s converges. In most cases, the system (6) can be solved using direct solvers such as UMFPACK [Davis, 4] up to the dimension of n Oð1; Þ efficiently on current hardware (with Oð1Þ GB storage). Larger systems can be computed in a matrix-free fashion using iterative Krylov space solvers as described in Saibaba and Kitanidis [1]. Note also that the objective function to be minimized can be written as J5 1 ðyhðsþþ> R 1 ðyhðsþþ1 1 ðsxbþ> Q 1 ðsxbþ 5 1 ðyhðxb1qh> nþþ > R 1 ðyhðxb1qh > nþþ1 1 n> HQH > n (7) The equation (7) can be used to gauge the progress of minimization and to make sure that the new solution is not worse than the previous one. For strongly nonlinear problems, the Levenberg-Marquardt method [Nowak and Cirpka, 4] or a line search [Zanini and Kitanidis, 9; Liu and Kitanidis, 11] can be adopted for the better convergence. Once the best estimate is determined, uncertainty of the estimate is characterized by the posterior covariance matrix V computed as: V5QXMQH > K > (8) LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 541

4 1.1/14WR15483 where X and M are computed from " #" # " HQH > 1R HX K > 5 HQ # ðhxþ > M X > (9) Alternatively, the estimation uncertainty can be quantified by the posterior ensemble, i.e., conditional realizations sampled from the posterior pdf using either the direct decomposition of the posterior covariance or the parametric bootstrapping sampling method [Kitanidis, 1995; Kitanidis and Lee, submitted manuscript, 14]. The geostatistical approach described above can be applied to small-to-moderate-scale inverse problems (up to m Oð1 4 Þ). However, when the method is implemented on finely resolved grids with a large number of measurements, computational challenges arise mainly from the construction of H and the matrix products of H, i.e., Hs; HX; HQ, and HQH >. The Jacobian matrix H must be calculated several times and each computation using the adjoint-state method, which is recommended when n m, corresponds to n solutions of the forward problem. Usually, this is the most computationally expensive part of the geostatistical approach, particularly when many measurements are available and several Gaussian-Newton iterations are needed. Additionally, the storage of H and its multiplication with Q become prohibitive for large m and n. Furthermore, the computation and storage costs of the posterior covariance matrix may not be tractable in many cases. In order to tackle these difficulties, Kitanidis and Lee (submitted manuscript, 14) developed PCGA that implements the low-rank approximation of Q and a finite difference approximation to avoid the direct evaluation of H. Assume we approximate Q as Q Q K 5ZZ T 5 XK i51 f i f > i (1) where Q K is a rank-k approximation of Q; Z is a m 3 K matrix and f i is the ith column vector of Z. Furthermore, a generic Jacobian product Hu needed in (4) (e.g., u5s or f i ) can be computed approximately at the cost of two forward model evaluations from the Taylor series expansion: hðs1duþ5hðsþ1dhu1oðd Þ (11) where u is a m 3 1 vector and d is the finite difference interval. For example, Hs can be computed from Hs5 1 d ½hðs1dsÞhðsÞŠ1OðdÞ 1 d ½hðs1dsÞhðsÞŠ (1) As a result, the computational cost associated with a Jacobian-vector product reduces from n to two forward simulations and the storage cost becomes OðmKÞ from OðmnÞ. Similarly, HX can be computed by HX i 1 ½ d h ð s1dx iþhðsþš (13) where X i is the ith column of X. Next, the matrix-matrix products HQ and HQH > are computed by HQ HQ K 5H XK i51 f i f > i 5 XK i51 ðhf i Þf > i XK g i f > i (14) i51 where HQH > HQ K H > 5 XK i51 ðhf i ÞðHf i Þ > XK g k g > k (15) i51 LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 5413

5 1.1/14WR15483 g i 5Hf i 1 ½ d hðs1df iþhðsþš (16) respectively. Replacing the explicit construction and multiplication of H in (3 6) by (1 15) requires a total of K 1 p 1 forward model runs in each iteration. One can also take advantage of the PCGA to quantify the uncertainty of the estimation. The diagonal entries of the posterior covariance matrix V in (8) are often presented as the estimation variance and can be computed without constructing V explicitly as " V ii 5Q ii HQ # > " # i HQH > 1 " # 1R HX HQ i ðhxþ > X > i X > i (17) where V ii is the ith diagonal element of V, Q ii is the ith diagonal entry or the prior variance of ith parameter, HQ i is the ith column of HQ, and X > i is the ith column of X T. The cost of computing the entire variance map of the estimate after obtaining (1 15) is Oðn mþ. One may store and reuse the inverse of the cokriging matrix for a small value n Oð1Þ to reduce the computation time further. Before investigating the computational savings in detail, we first examine the accuracy of the approximation. Since the matrix-vector products are replaced by the finite difference approximation, the choice of d is crucial to the success of the PCGA. Assuming h is smooth and second-order differentiable, the optimal choice of the finite difference interval d for the forward operator h at s that minimizes both first-order approximation and finite-precision arithmetic errors [Gill et al., 1983] can be computed as rffiffiffiffiffiffi ^d hðsþ 5 juj (18) where is a machine epsilon (bound on the relative error due to rounding) and U is a representative value of the second derivative of h around s. If we do not have any information on the second derivative of the forward model, the rule of thumb for choosing d is ^d p hðsþ ffiffi, i.e., 1 4 for the single precision floating format and 1 8 for the double precision floating format. In addition, the generic vector u evaluated in the forward model h should have the same unit as s, thus the optimal choice of d can be computed as ^d5^d hðsþ jjsjj jjujj (19) where is the norm operator. While one can use a single value of d when the forward model outputs vary within a couple of orders of magnitude, several d s can be implemented for the evaluation of different rows in the matrix-vector products if needed. For joint inverse problems using coupled forward models that predict different types of data, a different d for each type of data can be chosen. One may want to experiment with a high-order finite difference method in order to choose an appropriate d for a specific application. With the optimal choice of d, the accuracy of the approximation also depends on the approximation of Q. While any suitable approximation method can be implemented in (1) as shown in Kitanidis and Lee (submitted manuscript, 14), in this study, we choose the truncated eigen decomposition of Q for the optimal approximation or compression in terms of the spectral norm: Q Q K 5V K K K V > K 5XK f i f > i () i51 p f i 5 ffiffiffiffi k ivi (1) where K K is the K 3 K diagonal matrix whose ith diagonal element is the ith eigenvalue k i of Q sorted in descending order and V K is the m 3 K matrix whose ith column V i is the eigenvector corresponding k i. The LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 5414

6 1.1/14WR15483 error of K-rank approximation is the K 1 1th eigenvalue of Q, i.e., jjqq K jj5k K11. Then K may be chosen based on a user-defined accuracy such as jjqq K jj 1 or up to the point that the computation resources allow. In fact, the accuracy of the low-rank covariance matrix approximation, i.e., the number of eigen pairs one should keep, depends on the smoothness of covariance kernel [Frauenfelder et al., 5; Schwab and Todor, 6]. For covariance matrices whose eigenvalues decay fast, our method performs best with a small value for K n. For nondifferentiable covariance kernels such as the exponential function with a short correlation length, one may have to select a large value of K to ensure an accurate approximation of Q. However, we are primarily interested in the approximation of QH > and HQH >, not Q because the solution in the geostatistical approach (5) is constrained in the space spanned by the n columns of the cross-covariance QH > and the p columns of the drift HX [Kitanidis, 1998]. Because the n 3 m Jacobian (sensitivity) matrix H is typically low-rank ( n) reflecting the low information content in the measurements, the approximation errors in Q H > and HQH > may not be significant with a small value of K. It is also noted that the error matrix R, which represents uncertainty in the forward model and measurements, reduces the effects of approximation error further as we observe smooth estimates for large measurement error. Thus, the effect of the error introduced by the low-rank covariance and finite difference approximations on the solution would become negligible with a moderate number of K in most practical cases. In this paper, we use the relative error of the low-rank approximation as jjqq K jj 5 k K11 () jjqjj k 1 to determine the number of eigen modes K used in the PCGA. However, the cost of the conventional eigen decomposition in () is Oðm KÞ, which can be extremely expensive for large-scale applications. For the efficient decomposition scalable to large-scale problems, we employ the randomized eigen decomposition [Halko et al., 11], a method to approximate the exact truncated eigen decomposition with smaller decomposition costs, summarized in Appendix A. The idea of the randomized method is to use randomly sampled columns of Q to capture the dominant eigen spaces of Q. Combined with fast matrix linear algebra methods such FFT [Nowak et al., 3], the fast-multipole method [Greengard and Rokhlin, 1987; Fong and Darve, 9] or hierachical matrices [Ambikasaran et al., 13; Saibaba and Kitanidis, 1], one can achieve a computational cost of OðmKlog mþ. With a certain choice of parameters (see Appendix A), the estimated error of the randomized technique with a high probability (11 1 ) is given by [Halko et al., 11; Rokhlin et al., 9]: jq ~Q K jcm 1=6 k K11 (3) where ~Q K is the K-rank approximation of Q using the randomized eigen decomposition and C is a constant. Note that the variance of the error in (3) is practically small. In Figure 1, the performance of the randomized eigen decomposition is examined for two covariance matrices constructed from exponential qðx; x Þ5 exp jxx j :3 and Gaussian (double exponential) qðx; x Þ5exp jxx j :3 kernels on a 1 by 1 grid in x ½; 1Š. 5 randomized eigen decompositions are tested and the results shows clearly that the approximation error is almost negligible and the accuracy of the randomized method is comparable to the conventional decomposition method. For most problems, the decomposition is very fast and fits in the RAM memory of contemporary standard PCs. For example, for a covariance matrix with m 5 1 6, it took about 3 min and required 8 MB memory to finish a K 5 1-rank approximation in a laptop equipped with quadcore.1 GHz CPU and 8 GB RAM. Thus, one may start the randomized eigen decomposition with large K, e.g., K Oð1Þ, then choose a value for K that satisfies a reasonable approximation accuracy. Once we have the decomposition of the prior matrix Q in (1), only K 1 p 1 forward runs are needed in each iteration, which can be a significant reduction of computational costs compared to n 1 1 forward model runs with the adjoint-state method. Additionally, the computational costs for matrix-multiplication reduce to OðmKlog mþ from Oðm nþ in the conventional geostatistical approach or Oðmnlog mþ in the approach with fast-linear algebra methods. A comparison of the computational costs is presented in Table 1. One could achieve further computation savings by starting with a small value K during the first few LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 5415

7 1.1/14WR q(x,x ) = exp( x x /.3), x [ 1] λ i q(x,x ) = exp( x x /.3 ), x [ 1] λ i+1 Q Q k approx 1 Q Q k approx K K Figure 1. Estimation errors (jjqq K jj) from 5 randomized eigen decompositions for the (left) exponential covariance function qðx; x Þ5 exp jxx j :3 and (right) Gaussian covariance function qðx; x Þ5exp jxx j :3. The covariance matrices are constructed on a 1 by 1 grid in x ½1Š. Theoretical minimum errors of the K-rank approximation (k11th largest eigenvalue k i11 ) are also plotted. iterations or until the minimization of the objective value in (7) does not progress then increasing K later because inaccurate approximation may be sufficient to compute Gauss-Newton steps when the initial guess is far from the minimum. Besides the improvement in the computation and storage costs, additional advantages can be expected from the PCGA due to its simplicity and easy of implementation. In general, the development and implementation of the adjoint-state method for coupled nonlinear and/or time-dependent forward models is challenging and time consuming because a formulation of adjoint equations and a differentiation of each nonlinear forward operator are required and a massive amount of transient solutions may have to be stored for a time-dependent adjoint calculation. Rewriting the adjoint-state codes from existing complex forward model code would also take a large amount of time and effort. In some cases, one might prefer efficient, stable and highly optimized forward solvers that do not allow to access the code. The PCGA can deal with such difficulties effectively. Table 1. Performance Comparison of the Conventional Geostatistical Approach and the PCGA a Method Textbook PCGA Number of forward model runs n 1 1 K 1 p 1 Matrix multiplication Oðmnlog nþ OðmKlog mþ Storage OðmnÞ OðmKÞ a We refer the conventional geostatistical approach with the adjointstate method and the fast-linear algebra method as the textbook approach. 3. Numerical Experiments We demonstrate the performance of the PCGA on two applications a steady state hydraulic tomography using sequential pumping tests and a joint inversion using hydraulic head and tracer test data. In the first application, we start with a moderate number of unknowns and compare the results with those obtained from the LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 5416

8 1.1/14WR m Constant head h = 1 m No flow 5 1 m No flow Figure. Domain of Application 1 and the location of 3 well locations. Constant head h = 11 m conventional geostatistical approach. Then a large-scale hydraulic tomography problem with three million unknowns is investigated. In the second application, a joint inversion problem with hydraulic head and tracer test data is examined. The PCGA and a -D steady state groundwater flow model used in the first two applications were implemented in MATLAB. The coupled flow and transport for the joint inversion were simulated using MODFLOW [Harbaugh et al., ] and MT3DMS [Zheng and Wang, 1999]. The coupling was easily done using mflab, an efficient MATLAB interface ( code.google.com/p/mflab/). All test cases were run on a modest workstation equipped with 1-core 3 GHz processors and 18 GB RAM Application 1: Hydraulic Tomography In this section, we consider a steady state hydraulic tomography problem to examine the accuracy of our method compared with the textbook approach. By the textbook approach, we here mean the conventional geostatistical approach with the adjoint-state method and the fast-linear algebra implementation. A synthetic two-dimensional test case adapted from Zanini and Kitanidis [9] is used as shown in Figure and all parameters used in this application are listed in Table. The governing equation for the steady state depth-integrated flow in a confined aquifer is with the following boundary conditions: rðtrhþ5n1q i dðxx i Þ (4) h5h D ; on C D (5) n Trh5; on C N (6) Table. Parameters Used in Application 1 Description Application 1 Geometric Parameters L x, L y Domain length and width (m) 1, 75 Q i Pumping rates (m =d) 5 N Recharge rates (m/d).1 Dx, Dy Grid size (m) 1 Geostatistical Parameters ln T Mean log transmissivity (ln m =d ).5 (T m =d) qðx; x Þ Covariance kernel for Case 1 qðx; x Þ5ð:jxx jþ 3 qðx; x Þ Covariance kernel for Case qðx; x Þ5exp ðjxx j=15þ Measurement Error n obs Number of measurements 87 r h Standard deviation of the measurement error (m).5 where T is the transmissivity, h is the hydraulic head, N is the recharge rate, q i is the pumping rate for a pumping test at a well location x i marked with a circle in Figure, d(x) is the Dirac delta function, and n is the unit vector normal to the boundaries. A constant-head boundary condition on the boundaries of x 5 and 1 m (C D ) and no-flux condition on y 5 and 75 m (C N ) are considered. Two True log transmissivity (ln T) fields shown in Figure 3 were generated on the grid using isotropic cubic generalized covariance and exponential covariance models representing a highcontrast smooth field (Case 1) and a highly heterogeneous field (Case ), respectively. Based on the true ln T fields, sequential LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 5417

9 1.1/14WR15483 y [m] Case x [m] ln T (m /d) Case 5 1 x [m] ln T (m /d) Figure 3. True log-transmissivity fields for (left) Case 1 and (right) Case : the ln T fields were generated from the cubic covariance model qðx; x Þ5ð:jxx jþ 3 and the exponential model qðx; x Þ5exp ðjxx j=15þ. pumping tests were carried out numerically at each well location and hydraulic heads were measured at the well locations except the pumping well resulting a total of 87 head measurements from 3 pumping tests with 9 observations per test. Therefore, we estimate 75 ln T values from 87 head observations. A Gaussian error with zero mean and the standard deviation of.5 m was added to the simulated measurements and this error corresponds to 5% of the maximum drawdown in the domain. The finite difference interval d was chosen from (19) as ^d51 6 based on the double precision machine epsilon 1 16 and h N=T. We employ isotropic exponential covariance models to estimate ln T, and the structural parameters for variogram and measurement error were determined using the cr/q criteria [Kitanidis, 1997]. The initial guess was set to a homogeneous field of ln T5 for all the considered cases. The solutions in all tests converged within 1 iterations. The best estimates of the log transmissivity distribution for K 5 5, 5, and 75 are shown in Figure 4. For the comparison purpose, the textbook approach was applied to both cases in order to use the estimate as the reference (Figures 4d and 4h). In the first case, the reference field in Figure 4d captures most features of the true ln T field since the measurements are enough to estimate the smooth field generated from the differentiable cubic covariance model. As expected, the PCGA performs well, even the estimate with K 5 5 in Figure 4a is very similar to the reference estimate. The spectrum of the prior matrix Q shown in Figure 5 (black) indicates that the approximation error becomes negligible with a moderate number of K > 5. In the second case, on the other hand, the best estimate in Figure 4h identifies only large-scale features of the true ln T field in Figure 3 because the measurements cannot resolve the small-scale details of the highly (a) lnt est with K = 5 (b) lnt est with K = 5 (c) lnt est with K = 75 (d) lnt ref (e) lnt est with K = 5 (f) lnt est with K = 5 (g) lnt est with K = 75 (h) lnt ref Figure 4. The best estimates from the PCGA with K 5 5, 5, and 75 for (a c) Case 1 and for (e g) Case. The best estimates from textbook approach for (d) Case 1 and (h) Case is also plotted as the reference estimates. LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 5418

10 1.1/14WR15483 Figure 5. The eigenvalue spectrum of the prior covariance for Case 1 (black) and Case (red); square indicates the 31th (Case 1) and the 49th (Case ) eigenvalues whose relative error (k i11 =k 1 ; equation ()).1. heterogenous true field. For the PCGA, K 5 5 is required to obtain a close estimate (Figure 4f) to the reference. The eigenvalues of the prior covariance Q decay slowly as in Figure 5 (red) so that we need to keep more eigen pairs to reduce the approximation error. Still, the number of forward runs in the PCGA is dramatically reduced compared to the textbook approach that requires n forward runs in each iteration to evaluate the full Jacobian H. These results indicate that even for strongly heterogeneous ln T field, we can achieve significant computational savings. Figure 6 shows the corresponding variance of the ln T estimation. The uncertainty nearby the observation well is found to be low, while high uncertainty is observed in the region where large drawdowns take place. Similar to the best estimates, the variance estimates using K 5 5 or larger are close to the reference estimates for both cases. In Figures 7a and 7b, we investigate the effect of K to the accuracy of estimates in terms of element-wise root-mean-square-error (RMSE) defined as sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 X m RMSE5 ln Ti ref ln Ti est (7) m i51 where ln Ti ref is the log transmissivity of the reference estimate in the grid cell i, and ln Ti est is the ith log transmissivity estimated by the PCGA in the grid i. The accuracy of the estimation variance is also investigated in Figures 7c and 7d. Each test was executed five times and we confirmed that the variability from randomized eigen decomposition does not affect the accuracy of the final estimates. For both cases, K 1 gives negligible errors (RMSE :1) to the reference estimate. Since the approximation error comes (a) σ est with K = 5 (b) σ est with K = 5 (c) σ est with K = 75 (d) σ ref.3..1 (e) σ est with K = 5 (f) σ est with K = 5 (g) σ est with K = 75 (h) σ ref.6.4. Figure 6. The estimation variance map r ðln T est Þ from the PCGA with K 5 5, 5, and 75 for (a c) Case 1 and (e g) Case and the estimation variance map from textbook approach for (d) Case 1 and (h) Case. LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 5419

11 1.1/14WR15483 (a) RMSE(lnT est ) for Case (b) RMSE(lnT est ) for Case x (c) RMSE(σ ) for Case est (d) RMSE(σ ) for Case est K Figure 7. The root-mean-square error of the estimates for (a) Case 1 and (b) Case and the estimation variance for (c) Case 1 and (d) Case. mostly from the covariance approximation, the spectrum analysis shown in Figure 5 can be implemented as a priori performance indicator to find an appropriate K. For this application, we observe that a low-rank approximation with about 1% relative error in Q is sufficient to obtain acceptable results (RMSE <.1), e.g., K 5 31 for Case 1 and K 5 49 for Case. 3.. Application : Large-Scale Hydraulic Tomography With m To investigate the scalability of the PCGA, the number of ln T values is set to by finer discretization of the previous domain, while other parameters are kept the same. The true transmissivity fields used in the previous section were downscaled by a spline interpolation correspondingly creating the 3 15 grids as shown in Figures 8a and 8b. The 87 noisy measurements were generated from the true ln T fields. The exponential covariance model with the same structural parameters found in the previous section was used in the inversion. The leading 5 eigenvalues for the two cases shown in Figure 9 were computed to find the number of eigen modes K used in this study. Based on the spectrum of eigenvalues, K 5 36 for the first case and K 5 48 for the second case were chosen in order to take advantage of parallelization with 1 processors. The computation time for the eigen decomposition of the by posterior covariance was 3 min and 1.5 GB RAM of the storage for elements was required. The best estimates for the two cases and their corresponding variance maps are shown in Figures 8c 8f. The estimates for both cases are practically similar to the true ln T fields as observed in the previous case. In Figure 1, the fitting between simulated and measured hydraulic heads (blue) is plotted for both cases. For the comparison purpose, hydraulic heads (black) simulated from the mean log transmissivity field ln ðt Þ5 :5 is also plotted to show the quality of the inversion results. Even with a relative small number of forward model runs, our approach is still able to estimate ln T and fit measurement data very well. Note that because of the downscaling by the interpolation, the details in the true field was slightly changed compared to the previous case in Figure 3, especially for the second case, resulting in a larger variance of log transmissivity. The estimation results would be improved further if one uses different structural parameters based on the cr/q criteria. To reduce the computation time, we start with a small value of K (K init 5 1), and K is increased to the chosen value after the first two iterations. By doing so, we could save a significant amount of computational LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 54

12 1.1/14WR15483 Figure 8. True log transmissivity fields for (a) Case 1 and (b) Case ; the best estimates for (c) Case 1 and (d) Case ; corresponding estimation variance for (e) Case 1 and (f) Case. costs while the resulting images were similar to those obtained using constant K throughout the iterations. The best estimates were obtained in seven iterations, spending about 43 min with a total of 56 forward runs for the first case and about 71 min with 38 forward runs for the second case. The variance of the best estimate for each case was computed in about 5 min. The results demonstrate that the PCGA scales well with large-scale problems using an affordable number of forward model simulations independent of the problem size and the number of measurements Application 3: Joint Inversion With Head and Tracer Travel Time Data Using Black-Box Forward Models In this section, we examine the applicability of the PCGA to general inverse problems that incorporate different types of measurements and require complicated coupled simulation models. A typical example of this type is the joint inversion of hydraulic head and tracer test data [Harvey and Gorelick, 1995; Cirpka and Kitanidis, ; Nowak and Cirpka, 6], which uses coupled flow and transport simulations to incorporate hydraulic head and tracer concentration measurements to infer subsurface properties. The goal of this study is to test our method as a generic tool for solving joint inverse problems by treating coupled simulation models as a black-box model. The widely used numerical simulation tools MODFLOW- [Harbaugh et al., LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 541

13 1.1/14WR15483 λ i Case 1 Case K Figure 9. The 5 leading eigenvalues of the by prior covariance matrix used in the Application for (a) Case 1 and (b) Case. ] and MT3DMS [Zheng and Wang, 1999] are linked to the PCGA inversion code to obtain head and concentration simulation data from alnt distribution. We consider an artificial twodimensional test case adapted from Cirpka and Kitanidis []. All parameters used in this study are listed in Table 3. In an aquifer with the dimension of L x 5 4 m and L y 5 m, an injection well with the flow rate Q w is located at x w 5 1 m, y w 5 1 m. No-flow boundaries are considered at y 5 m and y 5 m while fixed heads h x5 5.4 m and h x54 5 m are used at the boundaries at x 5 m and x 5 4 m. A true log transmissivity (ln T) field was generated using an anisotropic exponential covariance function and is shown in Figure 11a. Using the true ln T field, we simulated steady state flow using MODFLOW- and transient transport of a tracer introduced into the injection well at (x w, y w ) using MT3DMS. The steady state head data was collected at x 5 5, 1, 15,, 5, 3, 35 m and y 5 4, 8, 1, 1, 16 m. The tracer of concentration C 5 1 was injected over a period of an hour at the injection well, and the transient concentrations were measured every hour for 1 days at x 5 1, 15,, 5, 3, 35 m and y 5 8, 1, 1 m except at the injection well. No-mass flux boundary condition is assumed at y 5 m and y 5 m and advective boundary condition is imposed at x 5 and x 5 4. Instead of the entire concentration history, the mean travel time of the tracer data was used in the inversion because temporal moments of the transient tracer inversion data is better suited for the tracer inversion [Harvey and Gorelick, 1995; Ezzedine and Rubin, 1996; Cirpka and Kitanidis, ]. The mean travel time t x at the monitoring location x is given by simulated head (m) (a) Case 1 RMSE = best estimate ln(t) = (b) Case 135 RMSE = best estimate ln(t) = measured head (m) measured head (m) Figure 1. Measurement data fitting: measured heads versus hydraulic heads simulated from the best estimate (blue circle) and heads simulated from the mean log transmissivity field ln ðt Þ5:5 (black asterisk) for (a) Case 1 and (b) Case. LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 54

14 1.1/14WR15483 Table 3. Parameters Used in Application 3 Parameter Description Value L x, L y Domain length and width (m) 4, Dx,Dy Grid spacing (m). h x5,h x54 Boundary condition (m).4, x w, y w Injection well location (m) 1, 1 Q w Injection rate (m =s) 1:531 4 h Porosity.5 a l Longitudinal dispersivity (m).5 a t Transverse dispersivity (m).1 D m Diffusion coefficient (m =s) 1 9 Geostatistical Parameters ln T Mean of ln T (m/s) 7 qðx; x Þ Covariance kernel qðx; x Þ5exp ðjxx j=lþ l x, l y Correlation length in x and y (m) 4, Measurement Error r h Standard deviation of measurement error for head (m).1 rt x Standard deviation of measurement error for travel time (s).1 t x Ð 1 t x 5 Ð 1 tcðx ; tþ dt Cðx ; tþ dt (8) where C(x, t) is the concentration at time t. Equation (8) was computed numerically from discrete concentration values at each monitoring location. The random error with standard deviation of.5 m of the head value and 1% of the mean travel time was added to the head and tracer travel time data, respectively. The domain is discretized into 1 by 11 grid cells, thus we estimate,31 (1 3 11) ln T values using n h 5 35 head and n t 5 17 travel time data while other parameters are assumed to be known. y [m] y [m] y [m] (a) True ln(t) 1 (b) Only Head Data 1 (c) Both Head and Tracer Data x [m] ln(t) (m /s) Figure 11. (a) True log transmissivity field generated from the exponential model q ðx; x Þ5exp ðjxx j=lþ where l x 5 4 m and l y 5 m; (b) the best estimate for head data inversion; (c) the best estimate for head and mean travel time data inversion; the monitoring locations indicated by circle for head and asterisk for tracer concentration data Since MODFLOW- stores simulation results in the single precision floating point format, the finite difference interval for the head data, d h, is chosen as using (19) while the interval for tracer data, d C, is set to the square root of the machine epsilon, i.e., Setting a different interval for each simulation program requires K MODFLOW executions to compute Jacobian products with respect to head data and another K MODFLOW and MT3DMS runs for the travel time. Based on the eigenspecturm analysis, K 5 96 is chosen that gives the relative approximation error of %. The structural parameters of the covariance matrix and the measurement errors were identical to those used to create the true solution. The initial guess ln T57 was used and the estimates converged in five iterations for the head data inversion and seven iterations for the joint inversion. The best estimate of the log transmissivity field using the head data alone is shown in Figure 11b. Because of sparse and noisy head observations, the best estimate characterizes the large-scale features of the true field while details at a LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 543

15 1.1/14WR15483 y [m] y [m] (a) Only Head Data 1 (b) Both Head and Tracer Data x [m] Figure 1. The estimation variance map r ðln T est Þ for (a) head data and (b) joint inversion; the monitoring locations indicated by circle for head and asterisk for tracer concentration data. smaller scale cannot be recovered. The corresponding variance of estimation in Figure 1 indicates low estimation variance around the measurement locations while high uncertainty near the boundaries is observed. In Figure 11, the best estimate using both head and mean travel time data shows a clear improvement compared to the head data inversion result. Within the plume, the smallscale features are detected relatively well while the large-scale features outside the plume are still captured in the estimate. This result is expected because the tracer data is only informative within the plume due to the advection-dominant process. The variance of estimation also shows the smaller error bounds in the interior of the plume compared to the variance map for head-data inversion. In Figure 13, we display the fitting between the simulated and measured data for the two cases. It can be concluded that the use of travel time data improves the estimation of ln T. These results match what is reported in Cirpka and Kitanidis [] and confirm that the PCGA can be applied to joint inversion problems without difficulty. Note that the geostatistical approach using the temporal moments [Cirpka and Kitanidis, ] solves steady state equations to obtain the Jacobian matrix. The inversion would require n h flow simulations for head data and n h flow simulations plus n t steady state transport simulations for firstmoment measurements in each iteration, which needs much smaller computational efforts than K flow simulations and K 5 96 transient transport simulations required in our method for this application. However, the efficiency of the temporal-moment-based inversion method can be degraded in general cases; for instance, if the injection is not uniform along the entire inflow boundary, an additional transport simulation is required for the zeroth temporal moment computation. The PCGA is flexible in that it can deal with any practical and complex conditions, e.g., time-varying boundary conditions, multiple tracer injection or various tracer injection schemes such as continuous or time-varying tracer injection. When one has more travel time measurements from additional monitoring wells or chooses to add high-order temporal moments, the PCGA still requires the same number of forward model runs because the number of forward runs is determined independent of the number of measurements. Furthermore, our proposed method can include additional sources of information without difficulty, for example, we only need to add a reactivetransport module in the forward simulation for multiple reactive tracer injection tests Concluding Remark In this work, we have demonstrated the efficiency and accuracy of the PCGA (Kitanidis and Lee, submitted manuscript, 14) on the large-scale and complex joint geostatistical inversion problems. The randomized eigen decomposition makes the direct decomposition of the dense prior covariance matrix feasible and, combined with the finite difference approximation, we avoid the computation of the complete Jacobian matrix. The number of forward model simulations can be determined by users based on the eigenspecturm of the prior covariance or their computational resources and budgets [cf. Leube et al., 13]. The numerical experiments presented in this paper show that the PCGA can be suitable for the large-scale problem and be used as a general inversion tool with other black-block simulation models. LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 544

16 1.1/14WR15483 simulated head (m) (a) Only Head Data RMSE =.1 (b) Head and Tracer Data RMSE = measured head (m) measured head (m) (c) Only Head Data 8 (d) Head and Tracer Data 8 simulated mean travel time (d) 6 4 RMSE = RMSE = measured mean travel time (d) measured mean travel time (d) Figure 13. Measurement data fitting: simulated versus measured hydraulic head for (a) head data and (b) joint inversion; simulated versus measured mean travel time for (c) head data and (d) joint inversion. Appendix A: Randomized Decomposition Approach We summarize the randomized technique to compute the eigen decomposition of the covariance matrix Q [Halko et al., 11]. First, we would like to find a m by K orthonormal matrix W that makes WW > Q close to Q Q WW > Q Assuming we have a good W, we can compute the singular value decomposition of the K 3 m matrix W > Q (A1) W > Q5URV > (A) where U is a K 3 K left-singular matrix, R is a K 3 K diagonal matrix whose diagonal elements are singular value of W > Q, and V is a m 3 K right-singular matrix. Now we have a low-rank approximation of Q: Q WW > Q5WURV > VRV > Note that we assume the numerical error is negligible here so that we obtain WU5V. Please refer to Halko et al. [11] to construct exact symmetric matrices. In the randomized approach, we use random (Gaussian) linear combinations of columns of Q to determine W: (A3) LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 545

17 1.1/14WR15483 W RðQXÞ (A4) where R is the range of the space and X is a m 3 K random i.i.d Gaussian matrix, i.e., X ij Nð; 1Þ. One can use the QR factorization of QX to obtain an orthornormal matrix W. However, when the eigenvalues of the prior matrix Q decay slowly, the accuracy of the presented algorithm above is degraded significantly. This can be fixed by increasing the weights on the dominant eigenvalues: W RðQ q XÞ (A5) We also increase the accuracy of the decomposition by finding K 1 p eigen modes, i.e., use the m by K 1 p orthonormal matrix W where p is the over-sampling parameter (p 5 15 is used in this paper). The expected error bound in (3) is computed for q 5 3. Acknowledgments The research was funded by the National Science Foundation through its ReNUWIt Engineering Research Center ( NSF EEC ) and the Charles H. Leavell Graduate Student Fellowship. References Ambikasaran, S., J. Y. Li, P. K. Kitanidis, and E. Darve (13), Large-scale stochastic linear inversion using hierarchical matrices, Comput. Geosci., 17(6), Bui-Thanh, T., C. Burstedde, O. Ghattas, J. Martin, G. Stadler, and L. C. Wilcox (1), Extreme-scale UQ for Bayesian inverse problems governed by PDEs, in Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, pp. 1 11, IEEE Computer Society Press. Cardiff, M., and W. Barrash (11), 3-D transient hydraulic tomography in unconfined aquifers with fast drainage response, Water Resour. Res., 47, W1518, doi:1.19/1wr1367. Cardiff, M., W. Barrash, P. K. Kitanidis, B. Malama, A. Revil, S. Straface, and E. Rizzo (9), A potential-based inversion of unconfined steadystate hydraulic tomography, Ground Water, 47(), Carrera, J., A. Alcolea, A. Medina, J. Hidalgo, and L. J. Slooten (5), Inverse problem in hydrogeology, Hydrogeol. J., 13(1), 6, 5. Cirpka, O. A., and P. K. Kitanidis (), Sensitivity of temporal moments calculated by the adjoint-state method and joint inversing of head and tracer data, Adv. Water Resour., 4(1), Davis, T. A. (4), Algorithm 83: Umfpack v4.3 An unsymmetric-pattern multifrontal method, ACM Trans. Math. Software, 3(), D Oria, M., and M. G. Tanda (1), Reverse flow routing in open channels: A Bayesian geostatistical approach, J. Hydrol., 46, Ezzedine, S., and Y. Rubin (1996), A geostatistical approach to the conditional estimation of spatially distributed solute concentration and notes on the use of tracer data in the inverse problem, Water Resour. Res., 3(4), Fienen, M., R. Hunt, D. Krabbenhoft, and T. Clemo (9), Obtaining parsimonious hydraulic conductivity fields using head and transport observations: A bayesian geostatistical parameter estimation approach, Water Resour. Res., 45, W845, doi:1.19/8wr7431. Flath, H. P., L. C. Wilcox, V. Akcelik, J. Hill, B. V. Waanders, and O. Ghattas (11), Fast algorithms for bayesian uncertainty quantification in large-scale linear inverse problems based on low-rank partial hessian approximations, SIAM J. Sci. Comput., 33(1), Fong, W., and E. Darve (9), The black-box fast multipole method, J. Comput. Phys., 9(3), Frauenfelder, P., C. Schwab, and R. A. Todor (5), Finite elements for elliptic problems with stochastic coefficients, Comput. Methods Appl. Mech. Eng., 194(-5), 5 8. Gill, P. E., W. Murray, M. A. Saunders, and M. H. Wright (1983), Computing forward-difference intervals for numerical optimization, SIAM J. Sci. Stat. Comput., 4(), Greengard, L., and V. Rokhlin (1987), A fast algorithm for particle simulations, J. Comput. Phys., 73(), Halko, N., P. G. Martinsson, and J. A. Tropp (11), Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions, SIAM Rev., 53(), Harbaugh, A. W., E. R. Banta, M. C. Hill, and M. G. McDonald (), Modflow-, the US geological survey modular ground-water model: User guide to modularization concepts and the ground-water flow process, Tech. Rep. -9, U.S. Geol. Surv. Harvey, C. F., and S. M. Gorelick (1995), Mapping hydraulic conductivity Sequential conditioning with measurements of solute arrival time, hydraulic-head, and local conductivity, Water Resour. Res., 31(7), Kaipio, J., and E. Somersalo (7), Statistical inverse problems: Discretization, model reduction and inverse crimes, J. Comput. Appl. Math., 198(), Kitanidis, P. K. (1995), Quasi-linear geostatistical theory for inversing, Water Resour. Res., 31(1), Kitanidis, P. K. (1997), Introduction to Geostatistics: Applications to Hydrogeology, Cambridge Univ. Press, N. Y. Kitanidis, P. K. (1998), How observations and structure affect the geostatistical solution to the steady-state inverse problem, Ground Water, 36(5), Kitanidis, P. K. (1), Bayesian and geostatistical approaches to inverse problems, in Large-Scale Inverse Problems and Quantification of Uncertainty, edited by L. Biegler, G. Biros, O. Ghattas, M. Heinkenschloss, D. Keyes, B. Mallick, Y. Marzouk, L. Tenorio, B. van Bloemen Waanders, and K. Willcox, pp , Wiley, Chichester, West Sussex, U. K. Leube, P. C., F. P. J. de Barros, W. Nowak, and R. Rajagopal (13), Towards optimal allocation of computer resources: Trade-offs between uncertainty quantification, discretization and model reduction, Environ. Modell. Software, 5, Li, W., A. Englert, O. A. Cirpka, J. Vanderborght, and H. Vereecken (7), Two-dimensional characterization of hydraulic heterogeneity by multiple pumping tests, Water Resour. Res., 43, W4433, doi:1.19/6wr5333. Li, W., A. Englert, O. A. Cirpka, and H. Vereecken (8), Three-dimensional geostatistical inversion of flowmeter and pumping test data, Ground Water, 46(), Liu, X., and P. K. Kitanidis (11), Large-scale inverse modeling with an application in hydraulic tomography, Water Resour. Res., 47, W51, doi:1.19/1wr9144. Liu, X., Q. Zhou, J. Birkholzer, and W. A. Illman (13), Geostatistical reduced-order models in underdetermined inverse problems, Water Resour. Res., 49, McLaughlin, D., and L. R. Townley (1996), A reassessment of the groundwater inverse problem, Water Resour. Res., 3(5), LEE AND KITANIDIS VC 14. American Geophysical Union. All Rights Reserved. 546