arxiv: v1 [stat.ap] 11 Nov 2017

Transcription

1 arxiv: v1 [stat.ap] 11 Nov 017 On constraining projections of future climate using observations and simulations from multiple climate models Philip G. Sansom, David B. Stephenson University of Exeter and Thomas J. Bracegirdle British Antarctic Survey April 16, 018 Abstract Appropriate statistical frameworks are required to make credible inferences about the future state of the climate from multiple climate models. The spread of projections simulated by different models is often a substantial source of uncertainty. This uncertainty can be reduced by identifying emergent relationships between future projections and historical simulations. Estimation of emergent relationships is hampered by unbalanced experimental designs and varying sensitivity of models to input parameters and boundary conditions. The relationship between the climate models and the Earth system is uncertain and requires careful modeling. Observation uncertainty also plays an important role when emergent relationships are exploited to constrain projections of future climate in the Earth system A new Bayesian framework is presented that can constrain projections of future climate using historical observations by exploiting robust estimates of emergent relationships while accounting for observation uncertainty. A detailed theoretical comparison with previous multi-model frameworks is provided. The proposed framework is applied to projecting surface temperature in the Arctic at the end of the 1st century. Projected temperatures in some regions are more than C lower when constrained by historical observations. The uncertainty about the climate response is reduced by up to 30 % where strong constraints exist. Keywords: Emergent constraints; Bayesian modeling; Hierarchical models; Measurement error; CMIP5. 1

2 1 Introduction Scientific inquiry into complex systems such as the climate, the economy, or epidemics naturally give rise to multiple models of a given system. The motivation behind the use of multi-model ensembles in climate science is to quantify the uncertainty in projections of future climate introduced by choices in model design Tebaldi & Knutti 007). Rather than treating them as incompatible or competing, each model is treated as a plausible representation of the climate system Parker 006). The multi-model paradigm is becoming established in other areas including ecology Spence et al. 016), epidemiology Webb et al. 017) and hydrology Le Vine 016). As the multi-model paradigm becomes more widespread, so does the importance of statistical methods for making credible probabilistic inferences from ensembles of models. Figure 1 shows an example of projections of future near-surface temperature change in the Canadian Arctic from an ensemble of 13 climate models. There is a large spread in both the historical temperature and future temperature change simulated by the models. There is also a negative correlation between the future temperature change and historical temperature simulated by the models. The spread of projections simulated by different models is often the dominant source of uncertainty in projections of future climate Yip et al. 011). Systematic relationships between the historical and future states of the models have the potential to constrain projections of future climatebracegirdle & Stephenson 01). Such relationships are often called emergent constraints because they emerge from the analysis of a collection of models rather than by direct calculation Allen & Ingram 00). Climate models are coarse and incomplete representations of the Earth system, and we so need to account for discrepancies between the models and the system Craig et al. 001, Kennedy & O Hagan 001). In this study, we propose a framework for inference in multimodel experiments that joins up model uncertainty, emergent constraints and model inadequacy in a clear and consistent manner. The proposed framework is robust to unbalanced designs and differences in internal variability between models. We also explicitly address observation uncertainty and natural variability in the climate system. Two main interpretations of multi-model experiments have emerged Stephenson et al. 01). The truth plus error approach treats the output of each model as the true state of the Earth system plus some error that is unique to each model Cubasch et al. 001, Tebaldi et al. 005, Smith et al. 009). The exchangeable approach treats the Earth system as though it were another computer model Räisänen & Palmer 001, Annan & Hargreaves 010). Crucially, neither interpretation adequately addresses the issue of model inadequacy. The idea of model inadequacy was extended to multi-model experiments by Chandler 013) and Rougier et al. 013) who independently introduced the idea of a common discrepancy between the consensus of an ensemble of models and the Earth system. The most common heuristic approach to combining projections from multiple climate models, and that used by the Intergovernmental Panel on Climate

3 14 1 Temperature response C) Historical temperature C) 9 7 Figure 1: Near-surface temperature change in the Canadian Arctic 76 N,111 W). Thirty-year mean temperature change between and as simulated by an ensemble of 13 climate models under the RCP4.5 mid-range mitigation scenario. Crosses indicate the spread of runs from a single model. The dashed line indicates the mean climate and climate response of the ensemble. The solid line is a simple linear regression estimate of the emergent relationship between the climate response and the historical climate. The dotted line indicates the observed historical climate and projected climate response given the estimated emergent relationship. Change, is to weight each model equally, leading to the so-called multi-model mean estimate Collins et al. 013). Various approaches to weighting projections from multiple climate models have been proposed, including multiple regression Greene et al. 006, Bishop & Abramowitz 013), pattern scaling Shiogama et al. 011, Watterson & Whetton 011), and Bayesian Model Averaging Min & Hense 006). Sansom et al. 013) showed that certain choices of weights correspond to standard analysis of variance models. Recent heuristic methods have attempted to account for interdependence between models in addition to model performance Knutti et al. 017, Sanderson et al. 015). Given observations of the Earth system, an almost unlimited number of performance metrics may be formulated and used to weight projections. Climate scientists have long been recognized that no single model will perform best for all metrics or all variables Lambert & Boer 001, Jun et al. 008). If a model is unable to adequately reproduce observed data, we should certainly question its reliability in the future. However, projections of future climate are often extrapolations to states that have never been observed. Therefore, the ability to reproduce observed data, does not guarantee any skill in projecting future events Oreskes et al. 1994). It has been shown that weights that do no reflect the skill of the models are likely to lead to less reliable projections than weighting all models 3

4 equally Weigel et al. 010). Emergent constraints provide a principled way of exploiting data from multiple models to constrain future projections without having to explicitly assign performance based weights to individual models. Examples of emergent constraints have been found in the cryosphere Boé et al. 009, Bracegirdle & Stephenson 01), atmospheric chemistry Eyring et al. 007, Karpechko et al. 013), the carbon cycle Cox et al. 013, Wenzel et al. 014) and various other areas of the Earth system. However, just because a relationship holds across many models does not mean that it will hold in the system under study. Therefore, we prefer the term emergent relationship. We reserve the term emergent constraint for when physical insight indicates that the relationship should also hold in the real system. Multi-model ensembles are sometimes called ensembles of opportunity because they tend not to be designed, but simply collected Tebaldi & Knutti 007). The resulting ensemble cannot be treated as a random sample from some hypothetical population of models, nor is it designed to systematically span a population of models Stephenson et al. 01). In climate science the situation is complicated by the fact that some modeling centers maintain more than one model, and models from different research centers often share common components Knutti et al. 013). Care is required in order not to bias future projections by inadvertently giving too much weight to particular model families. The lack of design in multi-model ensembles also presents other difficulties. The number of initial condition runs of each from each model will usually not be equal, i.e., an unbalanced design. Table 1 lists the number of runs of each model for the example shown in Figure 1. In addition, the range of conditions simulated by each model in response to varied initial conditions may differ. The range of conditions simulated by each model in the Canadian Arctic example is visualized by the size of the crosses in Figure 1. Differences in the precision with which we know the state of each model can lead to regression dilution and biased estimates of emergent relationships Frost & Thompson 000). The framework proposed in this study is robust to unbalanced designs and differences in internal variability between models. The remainder of this study proceeds as follows. In Section, we develop a hierarchical Bayesian framework for analyzing multi-model ensemble experiments. Section 3 compares our proposed framework to existing multi-model ensemble approaches. Section 4 applies our new framework to the projection of future climate change in the Arctic. We end with concluding remarks in Section 5. 4

5 An inferential framework for multi-model experiments Our objective is to make credible probabilistic inferences about the future climate given simulations of both present and future climate from multiple models, and observations of present climate. We build the framework for inference in three parts. First we describe the relationship between the outputs of the models. Then we consider the relationship between the models and the Earth system. Finally, we build a model for observation uncertainty when no a priori estimate is available..1 The multi-model ensemble Let X smr be the output from run r, of scenario s, by model m. The random quantity X smr will typically represent the long-term e.g., 30-year) mean of some climate variable of interest e.g., surface temperature). Each model performs one or more simulation runs of each scenario, with each run starting from different initial conditions. We assume there are M models, i.e., m = 1,..., M, and two scenarios, the historical present) scenario denoted H, and the future scenario denoted F. The number of runs from each model for each scenario is denoted R sm, i.e., r = 1,..., R sm. We do not require that the number of runs from each model be equal, or that the number of runs of each scenario by any given model be equal. Further, we assume that the historical and future scenarios are sufficiently separated in time that runs from each scenario can be considered independent. The individual runs X smr are modeled as X Hmr X Hm N X Hm, σm ) X F mr X F m N X F m, ϕ m σ m ) ) 1) where X sm is the expected climate of model m in scenario s, and the model specific variances σm and ϕ m quantify the spread of the runs from each model in each scenario, i.e., initial condition uncertainty or internal variability. Since each model is trying to simulate the same target, the expected climates X sm of each model should be similar. We model the expected climates as X Hm N µ H, σh ) X F m X Hm N µ F + β X Hm µ H ), σ F H ). ) The common components µ H and µ F represent the ensemble consensus in the historical and future scenarios respectively. The variances σ H and σ F H quantify the spread of the models about the ensemble consensus, i.e., uncertainty due to model differences. The expected future climate X F m of model m is modeled as dependent on its expected departure from the historical consensus X Hm µ H. The coefficient β determines the dependence and is referred to as the emergent constraint. A value of β = 1 implies conditional independence of the expected response X F m X Hm of model m from its expected historical climate X Hm. The variances σ m in Equation 1 quantify the spread of the runs of each model in the historical scenario. The coefficients ϕ m allow the internal variability of 5

6 each model to change in the future scenario. The internal variability in each scenario is expected to be similar across different models, so we model σm and ϕ m as arising from common distributions σm νh Inv-gamma, ν Hψ ) ϕ νf m Inv-gamma, ν F θ ) 3) ], θ = 1/ E [ ] ϕ and θ ψ = 1/ E [ ϕ m σ m ) ]. The so that ψ = 1/ E [ σm m degrees-of-freedom ν H and ν F control the precision of σm and ϕ m respectively. We will refer to ψ and θ as the ensemble consensus internal variability and coefficient of internal variability.. The climate system When we talk about climate, what we really mean is the distribution of possible weather Stephenson et al. 01, Rougier & Goldstein 014). However, climate is usually expressed as the 30-year average of weather. Let Y sa represent the actual climate average of weather) in scenario s, modeled as Y Ha Y H N Y H, σa ) Y F a Y F N Y F, ϕ a σ a ) ) 4) where Y s is the expected climate of the Earth system in scenario s. The variances σa and ϕ a quantify sampling uncertainty owing to natural variability in the Earth system, i.e., weather. Since the models are all attempting to simulate the climate of the Earth system, we expect the ensemble consensus climate µ s to be similar to that of the Earth system Y s, but not the same. We model the expected climate of the Earth system as Y H N µ H, σ ) H Y F Y H N µ F + β Y H µ H ), σ ) F 5) The expected future climate Y F is modeled as dependent on the expected discrepancy between the historical climate and the ensemble consensus Y H µ H. We assume that the same emergent constraint applies to the Earth system and constrains the future climate conditional on present conditions. The variances σ H and σ F H quantify our uncertainty about the climate of the Earth system given the ensemble consensus, i.e., they quantify our beliefs about model inadequacy..3 Observation uncertainty The observations will differ from the actualized climate due to measurement error. Let Z H be the observed state of the system in the historical period. We model the observed state of the system as Z H N Y Ha, σz ) 6) where the variance σz quantifies the observation uncertainty due to measurement error. 6

7 Internal variability Natural variability σ m ϕ m ψ θ σ a ϕ a Model runs Model climates Consensus climate Expected climate Actual climate Observed climate X Hmr X F mr X Hm X F m µ H µ F Y H Y F Y Ha Y F a Z H σ H σ F H β σ H σ F H σ Z Model uncertainty Emergent constraint Model inadequacy Observation uncertainty Figure : Graphical representation. The framework proposed in Section, including the optional component in Equation 7 represented as a directed acyclic graph. Diamonds represent data, circles represent latent quantities, and squares represent parameters. The degrees-of-freedom ν H, ν F, ν Ha, ν F a are neglected for brevity..4 Making inferences about future climate The full hierarchical framework is summarized in graphical form in Figure The multi-model ensemble is described by nine parameters µ H, µ F, β, σ H, σ F H, ψ, θ, ν H, and ν F. Given multiple runs of each scenario from one or more models, all of the parameters associated with the ensemble can be inferred from data..4.1 Natural uncertainty Natural variability or sampling uncertainty in the Earth system is described by an additional two parameters σa and ϕ a. The historical natural variability σa in Equation 4 could be estimated from a time series of observations. However, the change in variance ϕ a cannot be estimated directly since we have no observations of future climate. The same assumptions that would be required to estimate the natural uncertainty σa from observations could be applied to model data to provide alternative estimates of the internal variability σm. Therefore, we expect the natural variability σa to be similar to the ensemble consensus natural variability ψ. We propose to estimate the natural variability in the Earth system as σ a Inv-gamma νha, ν Haψ ) ϕ νf a a Inv-gamma, ν F aθ ) 7) 7

8 The degrees-of-freedom ν Ha and ν F a quantify model inadequacy in simulating natural variability in the Earth system..4. Model inadequacy In principle, the historical model inadequacy quantified by σ H and ν Ha could also be estimated from a time series of observations and corresponding simulations. This would require careful modeling to account for time-varying trends and to separate model inadequacy from natural variability. In addition, an extremely long time series would be required since the the discrepancy between the Earth system and the ensemble is expected to change only slowly over time. Instead, we estimate the model inadequacy relative to the ensemble spread as σ H = κ σ H σ F H = κ σ F H ν Ha = ν H /κ ν F a = ν F /κ 8) where κ 1. This parametrization reflects the judgment that due to processes not represented by any model, the actual climate has less in common with the ensemble consensus than do the individual models Rougier et al. 013). The coefficient κ is fixed a priori..4.3 Observation uncertainty Estimates of the observation uncertainty σz are not generally readily available. Several modeling centers produce reanalysis products that combine multiple observation sources using complex data assimilation and numerical weather models. Given multiple reanalysis data sets we can quantify our uncertainty about the observed state of the climate. Let W i be the output of reanalysis i, which we model as W i N µ W, σw ) 9) where µ W represents the consensus of the reanalyzes and the variance σ W quantifies the spread of the reanalysis outputs. We expect the actual climate Y Ha to be similar to the reanalysis consensus µ W and so model the actual climate as Y Ha N µ W, σ W ) 10) The variance σ W quantifies our uncertainty about the discrepancy between the actual climate and the reanalysis consensus. Similar to the models, we judge that the actual climate is less like the reanalysis consensus than the individual reanalyses are, and so we set σ W = κ W σ W κ W 1. 11) 8

9 3 Comparison to previous frameworks 3.1 Ensemble regression The standard method for projection using emergent constraints is known as ensemble regression Bracegirdle & Stephenson 01). Ensemble regression is equivalent to simple linear regression of the model mean responses on the model mean historical climates, and can be written in our notation as X F m X Hm N XF X H + β X Hm X H ), σf H) where X sm = r X smr/r sm and X s = m X sm /M. This is equivalent to our Equation where β = β 1, since E [ Xsm ] = Xsm and E [ Xs ] = µs. However, ensemble regression ignores the uncertainty in the historical departures X Hm X H due to internal variability. It is well known that if the independent variable in a linear regression is measured with error, then the maximum likelihood estimate of the regression coefficient is biased towards zero Frost & Thompson 000). Consider a balanced ensemble R Hm = R F m = R m) in which all models simulate the same internal variability in each scenario, i.e., σ Hm = σ F m = σ m. The expected value of the linear regression estimate of the emergent constraint is [ ] E ˆβ = β σh σ /R σh + σ /R where β is the true value of the emergent constraint. The bias is largest when the internal variability σ is large compared to the model uncertainty σh, or when the number of runs R from each model is small. The Bayesian framework presented here avoids this bias by integrating over the internal variability from each model, thereby properly accounting for the additional uncertainty. The ensemble regression estimate of the response of the Earth system is Y F Y H N XF X H + β Z H X H ), σf H). This is equivalent to assuming the Earth system is exchangeable with the models, ignoring the possibility of discrepancies between the models and the climate system, as well as the effects of observation and sampling uncertainty. 3. The coexchangeable model Rougier et al. 013) propose a model of the joint distribution of the historical and future climate in multi-model experiments known as the coexchangeable model. In our notation X m N µ, Σ) Y N µ, Σ ) m = 1,..., M Z H N Y H, σz ) 9

10 where X m = X Hm, X F m ), Y = Y H, Y F ), µ = µ H, µ F ). This is equivalent to our Equations, 5 and 6 when σ Σ = H βσh ) βσh β σh +. σ F H However, the coexchangeable model does not distinguish between uncertainty due to differences between the models and internal variability in the models, so it is vulnerable to biases due to regression dilution. In addition, observation uncertainty is accounted for, but sampling uncertainty due to natural variability in the Earth system is ignored. Rougier et al. 013) suggest the following estimate of the model inadequacy Σ = κ Σ + D where D is a diagonal matrix with diagd) = DH, D F ). The variances DH and DF guard against overly precise projections when models are in close agreement However, standard results for the multivariate normal distribution show that κ σh E [Y F Y H ] = µ F + κ σh + βy H µ H ). D H The emergent constraint shrinks towards 0 by an amount that depends on our assessment of DH. Similar terms D H and D F could be added to our Equation 8, but without effecting the emergent constraint. 3.3 Truth-plus-error Chandler 013) proposed an alternative joint framework for multi-model projection X mr N X m, Σ m ) r = 1,..., N m X m N µ, Λ m ) m = 1,..., M µ N Y, Σ ) Y Ha N Y H, σa ) where X mr = X Hmr, X F mr ), X m = X Hm, X F m ), µ = µ H, µ F ), Y = Y H, Y F ). The variances Λ sm represent the propensity of each simulator to deviate from the ensemble consensus. This provides flexibility to incorporate prior knowledge that certain simulators are more or less similar to each other. Internal variability and model inadequacy are both accounted for. The main difference between this framework and those proposed here and by Rougier et al. 013) is the direction of conditioning between the system Y and the ensemble consensus µ. This is a modeling choice, but we find it is perhaps more natural to consider the actual climate as the sum of our knowledge the ensemble consensus) plus what we don t understand model inadequacy), than vice-versa. In contrast to Rougier et al. 013), sampling uncertainty is accounted for, but observation uncertainty is ignored. 10

11 Chandler 013) suggests estimating the historical model inadequacy from data as σ H = Y Ha µ H ) then setting ) σ Σ = H σ H σ H 1 + κ)σ H for κ > 0. This parametrization ignores any emergent constraints in the projection of the future climate. In addition, estimating σ H from a single observation Y Ha provides very limited information and makes the analysis vulnerable to outlying or spurious measurements 3.4 Reliability ensemble averaging Tebaldi et al. 005) proposed a probabilistic interpretation of the heuristic reliability ensemble averaging model of Giorgi & Mearns 003). Multivariate extensions were proposed by Smith et al. 009) and Tebaldi & Sansó 009), and a similar spatial model was proposed by Furrer et al. 007). The basic model in our notation is given by X Hm N Y H, λ ) m X F m X Hm N Y F + βx Hm Y H ), θλ m ) ) Y Ha N Y H, σ a). Similar to Chandler 013), the simulator states X sm are conditioned on the state of the system Y s, and the variances λ s are interpreted as the propensity of each simulator to deviate from the system state. The coefficient θ allows the propensity of the simulators to differ from the system to change in the future scenario. Somewhat confusingly, sampling uncertainty in the Earth system is accounted for, but internal variability in the models is ignored. Observation uncertainty and model inadequacy are also both neglected. Tebaldi & Sansó 009) later proposed the inclusion of a common model bias term. However, the common bias was treated as a fixed quantity to be estimated, and does not contribute to our uncertainty about the Earth system in the same way as the model inadequacy terms proposed here and by Rougier et al. 013) and Chandler 013). The framework proposed by Tebaldi et al. 005) includes something similar to an emergent constraint. It is instructive to consider this alternative formulation in detail. The expectation of the full conditional posterior distribution of future climate Tebaldi et al. 005, Eqn. A9) is E [Y F...] = m λ m X F m + β m λ m Y H m λ m X Hm m λ m This is equivalent to our Equation 5, if λ m = σh m, i.e., if all the models are exchangeable. Let λ m = σh and θλ m = σf H for all m, then the posterior expectation of Y H Tebaldi et al. 005, Eqn. A8) is σa Y Ha + M σ X H H + σ F H βy F X F + β X ) H ) E [Y H...] = ) σa + M σ H + Mσ F H β ). 11

12 In comparison, the posterior expectation of Y H in our framework is E [Y H...] = σ a Y Ha + σ H µ H + σ σ a + σ H F H β Y F µ F + βµ H ) + σ F H β assuming κ = 1, i.e., the models are exchangeable with the Earth system. Both posterior expectations are weighted averages of the model outputs and the actualized climate Y Ha. The two estimates effectively differ only in the weight given to the models. Under the framework proposed by Tebaldi et al. 005), the models receive M times more weight than under our framework. As a result, the posterior estimate of the expected climate Y H, and consequently the projected climate Y F, will lie much closer to the ensemble consensus, and approach the consensus as the number of models increases. 4 Application to Arctic temperature change 4.1 Data To illustrate the framework proposed in the Section, it has been applied to the projection of winter December-January-February) near-surface m) temperature change in the Arctic at the end of the 1st century. We use model outputs from the World Climate Research Programme s Coupled Model Intercomparison Project phase 5 CMIP5, Taylor et al. 01). The historical period is taken to be the 30 year average winter temperature between December 1975 and January 005, as simulated under the CMIP5 historical scenario. The future period of interest is the 30 year average winter temperature between December 069 and January 099, as simulated under the CMIP5 RCP4.5 mid-range mitigation scenario Moss et al. 010). The domain of interest is 45 N 90 N, including not only the Arctic Ocean but also the Bering Sea and the Sea of Okhotsk, both of which also currently experience significant seasonal sea ice coverage. Prior to analysis, data from all models were bicubically interpolated to a common grid of size.5.5. The CMIP5 ensemble includes output from more than 40 models submitted by over 0 centers around the world. In order to satisfy the assumptions of Section.1 we consider a subset of the models that we judge to be approximately exchangeable. The models included in the thinned ensemble were chosen to have similar horizontal and vertical resolutions, but to minimize common component models according to the detailed information in Table 9.A.1 of Flato et al. 013). In particular, only one model was retained from any one modeling center, generally the most recent and feature complete version submitted by each center. The model genealogy of Knutti et al. 013) provides a useful check for relationships not evident from the meta-data submitted by modeling centers. The 13 chosen models and the number of runs available from the historical and future scenarios are listed in Table 1. Our approach to ensemble thinning differs from that of Rougier et al. 013) who chose models judged to be most similar to a familiar model. By minimizing rather than maximizing the commonalities 1

13 Table 1: Multi-model ensemble. Number of runs available from each model for the historical and future scenarios. Runs Modeling center Model Historical Future N Hm N F m BCC BCC-CSM1.1m) 3 1 CCCMA CanESM 5 5 NSF-DOE-NCAR CESM1CAM5) 3 3 ICHEC EC-EARTH 8 9 LASG-CESS FGOALS-g 5 1 NOAA GFDL GFDL-ESMG 1 1 NASA GISS GISS-E-R 6 6 MOHC HadGEM-ES 4 4 INM INM-CM4 1 1 IPSL IPSL-CM5A-MR 3 1 MIROC MIROC5 5 3 MPI-M MPI-ESM-LR 3 3 MRI MRI-CGCM3 3 1 Total among the models, we aim capture the broadest range of uncertainty due to model differences. Observational data in the Arctic are very sparse and no spatially complete data sets exist that include estimates of observational uncertainty. Therefore, we combine four contemporary reanalysis data sets ERA-Interim Dee et al. 011), NCEP CFSR Saha et al. 010), JRA-5 Onogi et al. 007), NASA MERRA Rienecker et al. 011)) using the methodology proposed in Section.4. Reanalysis data was interpolated to the same grid as the models. 4. Prior modeling and posterior computation Before computing posterior projections of late 1st century warming in the Arctic, we need to specify prior distributions for all unknown parameters. For the purposes of this example, we assume that the models are exchangeable with the Earth system, and the reanalyzes are exchangeable with the actualized climate. Therefore, we set κ = κ W = 1. Vague normal priors N 0, 10 6) ) were specified for the consensus parameters µ H, µ F, and µ w. A vague normal prior was also specified for the emergent constraint β N 1, 10 6) ). Vague inverse-gamma priors Inv-gamma 10 3, 10 3) ) were specified for the model variances σh, σ F and σ W, the consensus internal variability ψ and θ, and the degrees-of-freedom ν H and ν F. The resulting full conditional posterior distributions all have standard forms with the exception of the degrees-of-freedom ν H and ν F. Therefore, posterior inference can be efficiently accomplished by Gibbs sampling with Metropolis- Hastings steps for ν H and ν F. Full details of the posterior sampling procedure 13

14 a) b) p value of Kolmogorov Smirnov test p value of Kolmogorov Smirnov test Figure 3: Cross validation. p-values of Kolmogorov-Smirnov tests for uniformity of leave-one-out cross-validated climate response predictions leaving out a) all data; and b) only future data from each model in turn. used here are given in the Supplementary Material. Posterior analysis was performed for each grid box separately. Identical priors were specified at all grid boxes. Gibbs samplers were run for samples for each grid box, the first 1000 samples were discarded as burn-in, after which every 50th sample was retained to give a final sample size of 1000 per grid box. Inspection of a random selection of grid boxes indicated that the samplers converged reliably and mixed efficiently. 4.3 Model checking We check the assumption of exchangeability between models using a leave-oneout cross-validation approach similar to Smith et al. 009) and Rougier et al. 013). Each model in turn is left out of the analysis, and the response of a new model is predicted. The predictions are compared to the model output using a probability integral transform, i.e., by computing the probability that the response under the leave-one-out predicted distribution is less than the mean response of the excluded model. If the models are exchangeable, then the distribution over the models of the transformed projections should be uniform. Kolmogorov-Smirnov tests were used to assess uniformity at each grid box see Figure 3). In Figure 3a, we withhold all data from each model turn. Above 60 N there is little evidence against the null hypothesis that the models are exchangeable, although there may be some cause for concern at lower latitudes, particularly over northern Europe. In Figure 3b we withhold only the future simulations to test conditional exchangeability given any emergent relationships. Northern Europe remains mildly concerning, but otherwise there is no significant evidence that the model responses are not exchangeable. Overall, the cross-validation procedure suggests that the chosen models can be considered exchangeable. 14

15 a) b) c) Historical climate Y H C) Future climate Y F C) Climate response Y F Y H C) Figure 4: Expected climate. The posterior mean of a) the historical climate Y H ; b) the future climate Y F ; and c) the climate response Y F Y H. 4.4 Results The posterior mean estimates of the expected historical climate Y H, future climate Y F, and climate response Y F Y H are shown in Figure 4. The 0 C contour that approximates the ice edge has receded noticeably in the projected future climate Y F in Figure 4b compared to the historical climate Y H in Figure 4a. The projected warming tends to increase with latitude in Figure 4c. The greatest warming occurs near the islands of Svalbard and Franz Josef Land in the north of the Barents sea. Figure 5a investigates the strong warming near Svalbard in detail. The consensus climate response of the models µ F µ H in Figure 5a is already high at 10.5 C 90 % equal-tailed credible interval 7.9 C 13. C). The consensus response may be influenced by 3 models with unusually large responses. There is a positive emergent relationship β = ,1.9) at this grid box, and a historical discrepancy of Y H µ H =.8 C.0 C,+7.3 C). The emergent relationship predicts an additional 0.9 C 1.6 C,+4.1 C) of warming. This is relatively insignificant compared to the uncertainty about the response, even when conditioned on the historical climate. The emergent relationship here does little to constrain our uncertainty about the climate response. A negative emergent relationship is estimated in the North West Passage near Devon Island in northern Canada in Figure 5b. The consensus climate response µ F µ H in Figure 5b is more moderate at 6.6 C 5.0 C,8.1 C). There is a negative emergent relationship β = 0.4, 0.,0.7) and a historical discrepancy of Y H µ H = 3.6 C 0.9 C,+7.8 C). The emergent relationship combines with the historical discrepancy to project.1 C 5. C,+0.5 C) less warming than the ensemble consensus. At this grid box, our uncertainty is usefully constrained by the emergent relationship. The modification to both the mean and standard deviation of the posterior projected response is shown in the right hand margin of Figure 5b. The posterior standard deviation of the projected 15

16 a) b) Climate response C) Climate response C) Historical climate C) Historical climate C) Figure 5: Gridbox details. Data and projections from grid boxes a) north of Svalbard 81 N,39 E), and b) east of Devon Island 76 N,94 W). The solid red line indicates the estimated emergent relationship and the dotted red lines indicate a 95 % credible interval. The black dashed line indicated the consensus climate µ H and climate response µ F µ H. The red dashed line indicates the actual climate Y H and climate response Y F Y H. The blue density represents the distribution of the observations. The blue dashed line indicates the observed climate Z H and the climate response based directly on the observations. Auxiliary plots in the right hand margins show the posterior distribution of the climate response Y F Y H with red) and without black) an emergent constraint. response Y F Y H is reduced by 13 %, falling from 3.3 C to.9 C. The examples of Svalbard and Devon Island in Figure 5 both demonstrate the important role of observation and sampling uncertainty when combining models and observations. Due to the sparsity of observations in these remote regions, the observation uncertainty is quite large relative to the model uncertainty. In both cases, there is noticeable shrinkage of the posterior mean estimate of the historical climate Y H away from the observations Z H and towards the ensemble consensus µ H. As a result, the projected response Y F Y H lies closer to the ensemble consensus response µ F µ H than it would if observation uncertainty were ignored. Figure 6 shows the effects of emergent relationships in near-surface temperature in the Arctic. The negative emergent visible around the ice edge in Figure 6b is well known, and is due to albedo feedbacks caused by variations in sea-ice thickness simulated by the models Boé et al. 009). The posterior mean estimate of the historical discrepancy between the actual climate and the ensemble consensus Y H µ H is C 3 C across most of the Arctic Figure 6a). The historical discrepancy is largest in the Greenland and Barents differences in ocean heat transport simulated by the models Mahlstein & Knutti 011). the ensemble µ F µ H Equation 5). In the absence of an emergent relationship 16

17 a) b) c) Historical discrepancy YH µh C) Difference in mean of response YF YH C) d) Emergent constraint β Ratio of SD of response YF YH Figure 6: Effect of emergent constraints. The posterior mean of a) the historical discrepancy Y H µ H, b) the emergent constraint β, and c) the difference in the projected climate response Y H due to the emergent constraint β 1)Y H µ H ). d) Ratio of posterior standard deviation of the response Y F Y H with and without an emergent constraint. i.e., β = 1), the expected climate response Y F Y H is equal to the ensemble consensus response µ F µ H. The difference in the projected response due to the emergent relationship is given by β 1)Y H µ H ) and is plotted in Figure 6c. The expected warming is reduced by up to 3 C in the far north of Canada, and by around 1 C along most of the ice edge. Figure 6d compares the poster uncertainty about the climate response Y F Y H with and without an emergent constraint. Around the ice edge, the emergent constraint reduces the posterior standard deviation of the climate response Y F Y H by 0 % 30 %. Our estimate of the emergent relationship in the Beaufort sea, north of Alaska in Figure 6b, is much stronger than that of Bracegirdle & Stephenson 013). Internal variability is small compared to model uncertainty in the Arctic not shown), so the difference is not due to regression dilution in the ensemble regression estimates. Bracegirdle & Stephenson 013) analyzed an ensemble of 17

18 CMIP5 models, some of which were excluded from the ensemble analyzed here. Further investigation revealed that two of the models excluded from our analysis are strongly warm biased in this region, and two are strongly cold biased, but all four simulate similar climate responses. This acts to neutralize the emergent relationship evident in the remaining models not shown) in the analysis of Bracegirdle & Stephenson 013). A positive emergent relationship is estimated over Svalbard, Franz Josef Land and parts of Siberia in Figure 6b. This agrees with the analysis of Bracegirdle & Stephenson 013), and actually leads to increased warming in this region. Emergent constraints in air temperatures over polar land regions are particularly relevant for constraining estimates of changes in permafrost, which by melting in future could lead to accelerated emissions in greenhouse gases such as methane Burke et al. 013). There are significant differences in model temperatures over polar land regions related to model representation of processes such as snow physics and soil hydrology Koven et al. 013, Slater & Lawrence 013). It remains an interesting open question as to why models are showing a positive emergent relationship in the vicinity of the Barents Sea and Western Siberia. 5 Conclusion In this study, we have presented a new Bayesian hierarchical framework for the analysis of multi-model ensemble experiments. The main contribution of this study is to incorporate the concepts of model inadequacy in an ensemble of models and emergent relationships. Our framework allows the inclusion of multiple runs from each simulator for the first time in a practical application. This allows us to separate uncertainty due to differences between models and internal variability within models. We have shown that if internal variability is not accounted for, then projections based on emergent constraints may be biased. Another unique aspect of the framework presented here is the separation of sampling uncertainty and observation uncertainty in the climate system. Emergent relationship have become an important topic in climate science for their potential to constrain our uncertainty about future climate change. It is differences in the representation of key processes that lead to emergent relationships. In this study, we have argued that such relationships can be used to constrain discrepancies due to model inadequacy, if there a physical mechanism for the relationship can be identified. Parameter perturbations in an individual model might also produce emergent behavior. However, initial conditions should be forgotten over sufficiently long time scales and therefore should not lead to emergent behavior. If the internal variability is known to be small compared to the model uncertainty, then the distinction can safely be neglected. Future multi-model studies exploiting emergent constraints should at least check the relative magnitudes of the internal variability and model uncertainty, and preferably include multiple runs from each simulator to separate the two sources of uncertainty and avoid biased projections. 18

19 The negative emergent constraint on near surface temperature in the Arctic is well understood, and the application of our framework broadly confirms the findings of previous studies. The projected warming in the Arctic is reduced by up to 3 C by the emergent constraint. Internal variability in the Arctic is large compared to lower latitudes, but is dwarfed by structural uncertainty due to the difficulty of representing the many complex processes involved in simulating sea ice, snow cover and the polar vortex. Therefore, regression dilution is unlikely to have significantly biased previous studies. However, the sparsity of observations in the Arctic means there is significant observation uncertainty, and this is the first time that observation uncertainty has been accounted for when exploiting emergent constraints. Shrinkage of the expected climate towards the ensemble consensus results in differences of up to 1 C in the projected response compared estimates based on the observations directly. The framework proposed in this study allows robust estimation and projection using emergent constraints, but there are still open problems to be addressed both in general multi-model experiments and emergent relationships. In climate science, emergent constraints have generally been found to be a local phenomenon and are analyzed one grid box at a time. Spatial statistical methods could exploit the correlation structure between grid boxes to allow smoother projections exploiting emergent relationships. The CMIP5 multi-model ensemble included four future climate change scenarios, but we have analyzed only one. As with the climate models, the scenarios are difficult to interpret when taken together as an ensemble. Innovative methods are required to extract meaningful probabilistic projections that span the range of likely future climate change. In order to satisfy the assumption of exchangeability between the simulators we only analyze a subset of the available simulators. In principle, additional levels could be added to the hierarchy to represent collections of simulators from the same modeling group, removing the need to thin the ensemble. The complex relationships between models induced by components shared between modeling groups may make such a highly structured approach difficult or impossible. Alternative methods are required that can dynamically group simulators that exhibit similar characteristics. However, the greatest statistical challenge in climate projection is meaningful quantification of model inadequacy. The results here and in Rougier et al. 013) demonstrate how far we can go with simple judgments. However, additional co-operation between statisticians and climate scientists is required to make further progress. Acknowledgments This work was supported by the Natural Environment Research Council grant NE/I0050X/1. The authors thank Stephan Siegert for helpful comments. 19

20 References Allen, M. R. & Ingram, W. J. 00), Constraints on future changes in climate and the hydrologic cycle, Nature ), 4 3. Annan, J. D. & Hargreaves, J. C. 010), Reliability of the CMIP3 ensemble, Geophysical Research Letters 37, L0703. Bishop, C. H. & Abramowitz, G. 013), Climate model dependence and the replicate Earth paradigm, Climate Dynamics 413-4), Boé, J., Hall, A. & Qu, X. 009), Current GCMs Unrealistic Negative Feedback in the Arctic, Journal of Climate 17), Bracegirdle, T. J. & Stephenson, D. B. 01), Higher precision estimates of regional polar warming by ensemble regression of climate model projections, Climate Dynamics 391), Bracegirdle, T. J. & Stephenson, D. B. 013), On the robustness of emergent constraints used in multimodel climate change projections of arctic warming, Journal of Climate 6), Burke, E. J., Jones, C. D. & Koven, C. D. 013), Estimating the permafrostcarbon climate response in the CMIP5 climate models using a simplified approach, Journal of Climate 614), Chandler, R. E. 013), Exploiting strength, discounting weakness: combining information from multiple climate simulators, Philosophical transactions. Series A, Mathematical, physical, and engineering sciences 371, Collins, M., Knutti, R., Arblaster, J., Dufresne, J.-L., Fichet, T., Friedlingstein, P., Gao, X., Gutowski, W. J., Johns, T., Krinner, G., Shongwe, M., Tebaldi, C., Weaver, A. J. & Wehner, M. 013), Long-term Climate Change: Projections, Commitments and Irreversibility, in T. F. Stocker, D. Qin, G.-K. Plattner, M. M. B. Tignor, S. K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex & P. M. Midgley, eds, Climate Change 013: The Physical Science Basis, Cambridge University Press. Cox, P. M., Pearson, D., Booth, B. B. B., Friedlingstein, P., Huntingford, C., Jones, C. D. & Luke, C. M. 013), Sensitivity of tropical carbon to climate change constrained by carbon dioxide variability., Nature ), Craig, P. S., Goldstein, M., Rougier, J. C. & Seheult, A. H. 001), Bayesian forecasting for complex systems using computer simulations, Journal of the American Statistical Association 96454), Cubasch, U., Meehl, G. A., Boer, G. J., Stouffer, R. J., Dix, M., Noda, A., Senior, C. A., Raper, S. & Yap, K. S. 001), Projections of Future Climate Change, in J. Houghton, Y. Ding, D. J. Griggs, M. Noguer, P. J. van der Linden, K. Dai, X. and Maskell & C. A. Johnson, eds, Climate Change 001: The Scientific Basis, Cambridge University Press. 0

21 Dee, D. P., Uppala, S. M., Simmons, A. J., Berrisford, P., Poli, P., Kobayashi, S., Andrae, U., Balmaseda, M. A., Balsamo, G., Bauer, P., Bechtold, P., Beljaars, A. C. M., van de Berg, L., Bidlot, J., Bormann, N., Delsol, C., Dragani, R., Fuentes, M., Geer, A. J., Haimberger, L., Healy, S. B., Hersbach, H., Hólm, E. V., Isaksen, L., Kållberg, P., Köhler, M., Matricardi, M., McNally, A. P., Monge-Sanz, B. M., Morcrette, J. J., Park, B. K., Peubey, C., de Rosnay, P., Tavolato, C., Thépaut, J. N. & Vitart, F. 011), The ERA-Interim reanalysis: Configuration and performance of the data assimilation system, Quarterly Journal of the Royal Meteorological Society ), Eyring, V., Waugh, D. W., Bodeker, G. E., Cordero, E., Akiyoshi, H., Austin, J., Beagley, S. R., Boville, B. A., Braesicke, P., Brühl, C., Butchart, N., Chipperfield, M. P., Dameris, M., Deckert, R., Deushi, M., Frith, S. M., Garcia, R. R., Gettelman, A., Giorgetta, M. A., Kinnison, D. E., Mancini, E., Manzini, E., Marsh, D. R., Matthes, S., Nagashima, T., Newman, P. A., Nielsen, J. E., Pawson, S., Pitari, G., Plummer, D. A., Rozanov, E., Schraner, M., Scinocca, J. F., Semeniuk, K., Shepherd, T. G., Shibata, K., Steil, B., Stolarski, R. S., Tian, W. & Yoshiki, M. 007), Multimodel projections of stratospheric ozone in the 1st century, Journal of Geophysical Research Atmospheres 11D16). Flato, G., Marotzke, J., Abiodun, B., Pascale, B., Chou, S. C., Collins, W., Cox, P. M., Driouech, F., Emori, S., Eyring, V., Forest, C., Gleckler, P. J., Guilyardi, E., Jakob, C., Kattsov, V., Reason, C. & Rummukainen, M. 013), Evaluation of Climate Models, in T. F. Stocker, D. Qin, G.-K. Plattner, M. M. B. Tignor, S. K. Allen, J. Boschung, A. Nauels, V. Xia, Yu AND Bex & P. M. Midgley, eds, Climate Change 013: The Physical Science Basis, Cambridge University Press, chapter 9, pp Frost, C. & Thompson, S. G. 000), Correcting for regression dilution bias: comparison of methods for a single predictor variable, Journal of the Royal Statistical Society: Series A Statistics in Society) 163), Furrer, R., Sain, S. R., Nychka, D. W. & Meehl, G. A. 007), Multivariate Bayesian analysis of atmosphere-ocean general circulation models, Environmental and Ecological Statistics 143), Giorgi, F. & Mearns, L. O. 003), Probability of regional climate change based on the Reliability Ensemble Averaging REA) method, Geophysical Research Letters 301), 169. Greene, A. M., Goddard, L. & Lall, U. 006), Probabilistic multimodel regional temperature change projections, Journal of Climate 1917), Jun, M., Knutti, R. & Nychka, D. W. 008), Spatial Analysis to Quantify Numerical Model Bias and Dependence, Journal of the American Statistical Association ),