Climate model errors, feedbacks and forcings: a comparison of perturbed physics and multi-model ensembles

Transcription

1 Clim Dyn (2011) 36: DOI /s Climate model errors, feedbacks and forcings: a comparison of perturbed physics and multi-model ensembles Matthew Collins Ben B. B. Booth B. Bhaskaran Glen R. Harris James M. Murphy David M. H. Sexton Mark J. Webb Received: 23 September 2009 / Accepted: 27 March 2010 / Published online: 7 May 2010 Ó Crown Copyright 2010 Abstract Ensembles of climate model simulations are required for input into probabilistic assessments of the risk of future climate change in which uncertainties are quantified. Here we document and compare aspects of climate model ensembles from the multi-model archive and from perturbed physics ensembles generated using the third version of the Hadley Centre climate model (HadCM3). Model-error characteristics derived from time-averaged two-dimensional fields of observed climate variables indicate that the perturbed physics approach is capable of sampling a relatively wide range of different mean climate states, consistent with simple estimates of observational uncertainty and comparable to the range of mean states sampled by the multi-model ensemble. The perturbed physics approach is also capable of sampling a relatively wide range of climate forcings and climate feedbacks under enhanced levels of greenhouse gases, again comparable with the multi-model ensemble. By examining correlations between global time-averaged measures of model error and global measures of climate change feedback strengths, we conclude that there are no simple emergent relationships between climate model errors and the magnitude of future global temperature change. Algorithms for quantifying uncertainty require the use of complex multivariate metrics for constraining projections. Keywords Ensembles Uncertainty Model errors Climate feedbacks Observational constraints 1 Introduction Quantitative predictions of future climate change on time scales of decades to centuries are required inform society in its endeavours to both adapt to the consequences of climate change and to put in place mitigation efforts to control it. The complexity of interacting processes in the climate system means that we must use three-dimensional numerical models that represent all those processes and feedbacks in order to make predictions that directly feed into decision making. Complex models are required to provide regional detail, details of changes in extremes and for the assessment of non-linear, rapid or abrupt climate change. Uncertainties or errors 1 in numerical models limit the utility of projections from any individual model. Ensemble approaches have been applied in other prediction problems to increase utility by producing estimates of uncertainties in short-term predictions (e.g. Molteni et al. 2006). By first measuring the prediction uncertainties, and then tracing those uncertainties to model biases and errors, we should be better able to target research to improve models and ultimately produce better, less uncertain, climate projections. In parallel, there is a need to use information from the current generation of models to inform policy and planning now, hence there is a need to develop techniques to extract robust information from models and make credible projections. A component of any projection system should be an ensemble of models which sample natural variability, forcing uncertainty and the uncertainties in the underlying M. Collins (&) B. B. B. Booth B. Bhaskaran G. R. Harris J. M. Murphy D. M. H. Sexton M. J. Webb Met Office Hadley Centre, FitzRoy Road, Exeter EX1 3PU, UK matthew.collins@metoffice.gov.uk 1 Here we use the term error and model error to mean differences between models and the real world, as is common in numerical weather and climate modelling, rather than, e.g. coding errors or bugs that might be easily corrected.

2 1738 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics physical (and increasingly chemical and biological) processes which drive regional and global climate change. Two approaches have been adopted in recent years. The first we term the multi-model ensemble, sometimes called the ensemble-of-opportunity, meaning the collection of the output from the world s climate models. Recent efforts to collect such information (Meehl et al. 2007b) have produced an unprecedented array of studies that fed directly into the most recent IPCC assessment. The second ensemble technique we term the perturbed-physics ensemble (e.g. Murphy et al. 2004) whereby a single model structure is used and perturbations are made to uncertain physical parameters within that structure, including the potential to switch in and out existing sections of code in some cases. One strength of the multi-model approach is in the ability to sample a wide range of structural choices which may impact model errors, climate change feedbacks and climate forcings; widely different dynamical cores and widely different techniques for parameterising physical processes. There is a potentially large gene pool of possible models. Extensive coordination is required to ensure that modelling groups produce compatible experiments (the list of which is growing: e.g., Hibbard et al. 2007) and increasingly, as models become more complex including earth-systems processes and data assimilation schemes for example, modelling groups share components, potentially limiting the gene-pool. Despite great efforts world-wide, the number of ensemble members produced is, at most, of the order of tens of members. Knutti et al. (2010) discuss a wide range of issues relating to multimodel ensembles. The key strength of the perturbed physics approach is the ability to produce a large numbers of ensemble members in a relatively easy way. It is possible to control the experimentation and systematically explore uncertainties in processes and feedbacks. For example, it is possible to produce a set of ensemble experiments where the input forcing data (e.g. in a twentieth century simulation) is the same in each experiment, but the parameters which control, say, the climate sensitivity of the model are varied. Thus, the different sources of uncertainty can be isolated. It is also possible to explore a wide range of feedback processes in the model by de-tuning it, potentially revealing the impact of previous compensating errors. Such de-tuning also ameliorates the potential for double-counting when constraining models with observations (e.g. Allen et al. 2002); that is the assigning of a relative likelihood to different model versions based on observed data that has already been used in their development. The main motivation for this paper is to document the design and characteristics of a number of perturbed physics ensembles that have been produced as part of an extensive programme of research at the Met Office Hadley Centre to produce regional climate projections (e.g. Murphy et al. 2007, 2009) and to contrast aspects of those perturbed physics ensembles with corresponding multi-model ensembles. Such basic comparisons are important when we consider the number of approaches which use either or both types of ensembles to produce societally relevant information about climate change (see Murphy et al and the special edition of the Philosophical Transactions of the Royal Society A Collins 2007). In documenting studies which produce projections in terms of probability distribution functions (PDFs), it is not always possible to devote space to basic model diagnostics. This paper is intended to address this issue. While we clearly cannot investigate all possible aspects of the many stored Tbytes of model output we have access to in one paper, a number of questions and issues have driven the analysis herein: 1. What are relative model-error characteristics of the two approaches? We might naively assume that the multi-model ensemble contains members with a wide range of different error characteristics, whereas the perturbed-physics approach produces members with very similar baseline climates and thus very similar errors. Is it possible to identify systematic and random components of model error? What is the relative partitioning of systematic and random errors between the two types of ensembles? Why, in the multi-model case, is the ensemble mean so often the best model? 2. We know that the perturbed physics approach is capable of producing model variants with a wide range of different feedbacks strengths under climate change (e.g. Webb et al. 2006; Sanderson et al. 2008). Are the ranges comparable with those found in the CMIP3 models for both equilibrium and transient climate change? What are the main drivers of uncertainties in global climate change feedbacks in the two types of ensemble? 3. The total uncertainty in global mean change under, e.g. historical forcing and future SRES scenarios is a combination of uncertainties in feedbacks and uncertainties in radiative forcings. To the extent that the latter can be estimated, what are the differences between radiative forcings in the two ensemble approaches? 4. Finally, are there clear relationships between measures of model error and the magnitudes of climate change feedbacks? Question 4 is highly relevant when we use ensembles of climate model projections to generate predictions of climate change expressed in terms of PDFs which provide a measure the uncertainty (or credibility) in that prediction.

3 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics 1739 We cannot simply form histograms from, or fit statistical distributions to, the output from model simulations of future change. A key stage in forming PDFs is to assign a relative likelihood to each member of the ensemble by comparing simulations of past climate and climate change with observations (e.g. Rougier 2007). If we can clearly deduce that, for example, a model with a very high climate sensitivity performs less well than a model with a lower climate sensitivity when examining a wide range of observational tests, then we have less belief in that higher sensitivity model. Formally that model should receive a lower weight when forming a PDF from the ensemble and for this to be the case, i.e. to be able to distinguish between different models, there should be some relationship between the predict and say, climate sensitivity and the particular metric. This we call an observational constraint. A particular metric, or more generally a particular collection of metrics, is useful in assessing model fidelity if, and only if, there is some relationship (perhaps indirect) between that set of metrics and the prediction variable of interest. Furthermore, we may seek predictions of joint PDFs of variables, e.g. future temperature and precipitation change in a particular region. A metric optimised to constrain the PDF of future regional temperature change may not be optimal in constraining the PDF of future precipitation change in that region. Likewise, an observational constraint on climate variables in one region may not provide a constraint on the variable in another remote region. Murphy et al. (2007, 2009) outline a particular method to produce joint PDFs of future climate change using perturbed physics ensembles and observational constraints. The perturbed physics ensembles described here, together with others documented elsewhere, are combined with a statistical emulator of the model parameter space (see e.g., Rougier et al for an example) and a time-scaling technique (Harris et al. 2006) which maps equilibrium to transient responses taking into account any errors that may arise because of a mismatch between the patterns of transient and equilibrium. Using these tools it is possible to mimic the behaviour of HadCM3 at any choice of parameter values and allow the effective sampling of many more ensemble members than those described here. The prior predictive distributions obtained from the emulated ensemble are then constrained with observations of the time-averaged fields projected onto a truncated multivariate EOF space, and constrained with trends in various simple surface air temperature indices to produce likelihood-weighted posterior predictive distributions. Murphy et al. (2007, 2009) go further and estimate the impact of structural uncertainty in a term called the discrepancy which is estimated from the multi-model ensemble to produce joint PDFs of future changes. To now, the principal driver of for such work has been the quantification of uncertainty and the production of probabilistic projections. We might also use the concept of observational constraints and relative likelihoods of different models to improve models in a more targeted way (see e.g., Jackson et al. 2008). At present we test models during their development phase using a wide variety of different metrics and diagnostics, using different observations and different experiments. If we find a model to be deficient in a particular way (e.g., if surface temperatures are too warm in summer) we devote resources to improving that particular aspect of the model. We rely on our previous experience or belief of which variables are the most important and secondly how well those variables need to be simulated in order to produce the most accurate predictions. There is a danger in this approach that we might devote significant resources to improving a model in an area which is largely irrelevant for our particular prediction problem of interest. Alternatively we may neglect a variable which is highly influential in the prediction problem. By systematically relating the errors in the model simulation of present day and historical climate to uncertainties (errors) in our prediction variable of interest, it should be possible to produce a better priority list for which variables are most important. The above issues are touched upon in Sect. 5 of the manuscript, but a more complete analysis, including the use of observations to produce PDFs will be presented in future publications and is also part of an ongoing programme of research. In the recently released UK Climate Projections (Murphy et al. 2009) the rather complex statistical technique alluded to above is employed to relate model errors to future predictions. As we shall see in Sect. 5, there is no simple metric or diagnostic that provides a clear constraint on predictions of global-mean climate change. That is to say, there is no single field that a model needs to simulate perfectly in order for us to have complete confidence in a prediction from that model: a fact that has been known intuitively by modellers for some time. The list of metrics for testing models is multivariate. It is likely to be incomplete as there are, in general, more climate variables in a model than are observed. The list is also likely to contain redundant information in the sense that there are covariances between errors in different fields that means not all metrics are independent from each other. The extraction of useful information about climate change from imperfect climate models is likely to be a complex endeavour, on a par with the complexity of climate models themselves or the data-assimilation schemes used in initialvalue prediction. Section 2 of the paper describes the ensemble experiments examined, with a particular focus on the perturbed physics ensemble experiments. Section 3 presents an

4 1740 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics analysis and comparison of model errors. In Sect. 4 feedbacks and radiative forcings are contrasted. Section 5 presents a simple analysis of the relationships between model errors and feedback strengths. Finally Sect. 6 summarises the results of the analysis. 2 Climate model ensembles and variables 2.1 Perturbed physics ensembles The perturbed physics approach was developed in response to the call for better quantification of uncertainties in climate projections (see e.g., Chapter 14 of the IPCC Third Assessment Report Moore et al. 2001). The approach involves perturbing the values of uncertain parameters within a single model structure, with the choice and range for the perturbed parameters determined in discussion with colleagues involved in parameterisation development, or by surveys of the modelling literature. In some cases, different variants of physical schemes may be also be switched in and out as well as parameters in those alternative schemes being varied. Any number of experiments that are routinely performed with single models can then be produced in ensemble mode subject to constraints on computer time. A significant amount of perturbed physics experimentation been done with HadCM3 and variants, starting with the work of Murphy et al. (2004) and Stainforth et al. (2005) and continuing with Piani et al. (2005), Barnett et al. (2006), Webb et al. (2006), Knutti et al. (2006), Collins et al. (2006), Harris et al. (2006), Collins et al. (2007), Sanderson et al. (2007, 2008) and Rougier et al. (2009). Other modelling centres are also investigating the approach using GCMs (e.g. Annan et al. 2005, Niehörster et al. 2006) and more simplified models (e.g. Schneider von Deimling et al. 2006) with a view to both understanding the behaviour of their models and to quantifying uncertainties in predictions. Sokolov et al. (2009) use a version of the perturbed physics approach to make a comprehensive assessment of future global-scale change sampling uncertainties in physical, biogeochemical and economic factors. Here we make use of perturbed physics ensembles produced using in-house supercomputer resources at the Met Office Hadley Centre. Analysis of a much larger set of perturbed experiments performed as part of the climateprediction.net project are presented in other publications (e.g. Piani et al. 2005; Knutti et al. 2006; Sanderson and Piani 2007; Sanderson et al. 2008, Frame 2009). A comparison between the smaller in-house and larger publicresource ensembles performed with the mixed-layer version of HadCM3 is presented in Rougier et al. (2009) in the context of model emulation (see below) Considerations in the design of perturbed physics ensembles Given that one of the key strengths of the perturbedphysics approach is the ability to control the design of the ensemble, a design must be produced. However, there are a number of competing factors that might influence that ensemble design: 1. To aid understanding of the results, it may be useful to perturb one model parameter at a time. However, this limits the potential for interactions between uncertainties in different processes, such as clouds and radiation for example, which we might expect to be important. 2. To reduce the risk of over-confidence in predictions, it is necessary to produce model versions with a widerange of baseline climates and climate change feedbacks. This may mean relaxing a small number of the usual strict criteria for producing models, such as the near-balance of the top-of-atmosphere energy fluxes and may reveal errors in model variables that have been previously compensated for by the adjustment of a number of different parameters and/or the introduction of different representations of processes. 3. In contrast, given limited and expensive computer resources, it may be best to attempt to produce model versions which are somehow good, perhaps by trying to predict and minimise a collection of simple model metrics such the root mean squared error characteristics for time mean climate fields. At least we would not want to produce a large number of model versions that we would consider, by normal standards, to be a complete waste of computer resource. The potential issue in producing such tuned ensembles is the possibility of double counting model errors when the ensemble is weighted to produce PDFs of climate change. Double counting may lead to over-constrained predictions and potential for underestimating uncertainty. 4. To facilitate the building of the best emulator (e.g. Rougier et al. 2009), a statistical model which relates model parameters to outputs, it may be necessary to explore a wide range of model parameters and interactions between parameters in ways which aid the building of that emulator. Techniques such as Latin-Hypercubes (e.g. McKay 1979) may be employed for example. While this may result in model versions which may be considered unacceptable when compared to observational data, they would get downweighted in any posterior PDF calculation. Their job is to minimise the amount of extrapolation by the emulator outside sampled parameter space.

5 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics For more complex versions of the model (e.g. using a dynamical ocean component rather than a mixed-layer, q-flux or slab component) fewer ensemble members are possible because of the extra resources required to spin-up model versions and run scenario experiments. No one experimental design is capable of fulfilling all the above design criteria, yet they have all, at some time, guided our work on quantifying uncertainty in the presence of limited computer resources. For this reason we choose to separate our archive of perturbed physics versions of HadCM3 into the different sub-ensembles described below. We call the model HadSM3 when referring to the version of the model with a simplified mixed-layer, q-flux or slab ocean and use the letter S to prefix the ensemble name. In the case of the version coupled to a dynamical ocean, HadCM3, we use the prefix AO Description of perturbed physics ensembles S-PPE-S The ensemble described in Murphy et al. (2004) in which 31 parameters and switches in the atmosphere component of atmosphere-slab version, HadSM3, are perturbed. Perturbations are made to a single parameter at a time (as denoted by the suffix S in S-PPE-S), either to the minimum or to the maximum of the range specified in consultation with modelling experts, or on/off in the case of a switch. This results in 53 different model versions, including the standard parameter setting as defined in the standard published version of the model (Gordon et al. 2000; Pope et al. 2000), rather than the median or best-guess parameter values. In this design, if a perturbation in one physical scheme has an impact on a process or model variable that is also related to another scheme; there can be no compensation achieved by perturbing a related parameter, as might be done in the model development process. In that sense, the single-perturbation approach might be thought of as the simplest form of model de-tuning (Stocker 2004) in that no attempt is made to a priori maximise the model performance when compared to observations (it should be stressed that no systematic tuning of model performance was done to produce the standard parameter settings). The initial purpose of this ensemble was to provide a simple, understandable assessment of the parameter uncertainty in HadSM3. Details of all the parameters perturbed are presented in the appendix to Murphy et al. (2004) and also in Barnett et al. (2006) and Rougier et al. (2009) S-PPE-M This ensemble also utilises the mixedlayer ocean version, HadSM3, but in this case simultaneous multiple (suffix M) perturbations are made to the parameters, i.e. all 31 parameters and switches for the S-PPE-S case are perturbed simultaneously. Here, there can be compensation between perturbations to physical processes. In the design of the ensemble, an attempt was made to minimise the average of the root mean squared error of a number of time-averaged model fields while sampling a wide range of surface and atmospheric feedbacks under climate change. This tuned design of the ensemble was guided by deriving a linear predictor (based on the S-PPE-S ensemble), relating the 31 parameters of HadSM3 to the climate sensitivity and the Murphy et al. (2004) Climate Prediction Index or CPI. Further details of experimental design are given in Webb et al. (2006) who also examine cloud-feedback processes under climate change in some detail and compare with a multi-model ensemble. In contrast to the S-PPE-S ensemble, the interactive sulphur cycle (Jones et al. 2001) is activated in all ensemble members although no changes to sulphate emissions are employed. The ensemble contains 129 members, which includes a version with the standard parameter settings but with interactive sulphur cycle activated. A particular feature of models with mixed-layer oceans is a cooling instability that can appear during the 19CO 2 and/or 29CO 2 phase (a description of the mechanism for the instability is presented in the supplementary information in Stainforth et al. (2005)). This happens in one of the 129 members, leaving 128 members analysed here S-PPE-E An additional 103 HadSM3 experiments are grouped into this ensemble using the same parameters perturbed in S-PPE-S and S-PPE-M. A small number of experiments were performed to make initial estimates of the non-linearity of parameter combinations in Murphy et al. (2004) (see the appendix of that paper) but the majority of the members were produced to explore parts of parameter space not covered by the other HadSM3 ensembles for use in the building of an emulator of the parameter space of the atmosphere component of the model (further details can be found in Rougier et al. (2009), Murphy et al. (2009)). The generic function of an emulator is to map the parameters of the model onto variables of interest and, as a consequence, there is a requirement to explore parameter space without recourse to potential model validity. Thus, in contrast to the tuned S-PPE-M ensemble, no attempt is made to minimise root-meansquared (RMS) errors for example; the exploration of parameter space being the main motivation for the large majority of the members of this ensemble. The 103 are a subset of a larger ensemble in which 13 parameter combinations suffer the cooling instability described above, so are not analysed. For each member of the mixed-layer model version ensembles, a calibration phase (from 10 to 25 years

6 1742 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics depending on decadal drift and variability) is performed and the heat convergence from within the mixed-layer component is averaged into monthly values and kept fixed in both the 19 and 29CO 2 experiments. Unfortunately, a coding error was subsequently discovered in the S-PPE-M experiments, and also in some members of the S-PPE-E ensemble, such that the heat convergence field was specified from only 1 year of the calibration phase, rather than being averaged over many years. This has the potential impact of introducing noise into the heat convergence field, which may drive the SSTs in the 19CO 2 phase away from the seasonally varying climatology. As we shall see in later analysis, the impact is on average rather small, in particular when one contrasts errors in the SST fields in the perturbed mixed-layer experiments with those in non-fluxed adjusted coupled model runs. Repeat experiments in which average year heat convergence fields are applied to members with the largest SST noise show no significant differences in global-scale features such as RMS errors for non-sst related variables nor in the components of the atmospheric and surface feedbacks at 29CO 2. The model versions are therefore suitable for quantifying uncertainty and examining feedbacks, etc AO-PPE-A This ensemble uses the fully coupled version of HadCM3 but with perturbations only to parameters in the atmosphere component (an updated version of the ensemble described in Collins et al. 2006). The standard settings and 16 combinations of parameter settings selected from the S-PPE-M ensemble are used in order to sample a range of surface and atmosphere feedbacks under transient climate change. Members are selected based on an approximately uniform sampling of the climate sensitivity of the larger S-PPE-M ensemble while ensuring that a wide range of different parameter settings are sampled. The choice was made by examining the table of sensitivities and parameters in S-PPE-M, rather than using any numerical algorithms. In addition, the interactive sulphur cycle is activated as it is in the S-PPE-M ensemble but in contrast, sulphate emissions are varied in some simulations (see Sect. 2.3). Murphy et al. (2009) also describe an ensemble with perturbations to parameters within the HadCM3 sulphur-cycle. The results from this ensemble will be described elsewhere. Flux adjustments are employed in these coupled model simulations to: (1) prevent model drift that would result from perturbations to the parameters that lead to top-of-atmosphere net flux imbalances, and (2) to improve the credibility of the simulations in simulating regional climate change and feedbacks. The limitations of coupled modelling the presence of flux adjustments has been discussed widely, e.g. Dijkstra and Neelin (1999). Here the similarity of baseline surface-climate states facilitates the combination of the HadSM3 and HadCM3 ensembles to produce time-scaled response for a larger number of combinations of model parameters (Harris et al. 2006). The spin-up technique is similar to that described in Collins et al. (2006) except that a less vigorous salinity relaxation is employed during the Haney-forced phase (relaxation coefficients are those used by Tziperman et al. (1994); 30 and 120 days for temperature and salinity, respectively) which significantly alleviates the problem of SST and sea-ice biases found in the Collins et al. (2006) ensemble (Fig. 1). The 16 perturbed sets of parametercombinations are selected from the 128-member S-PPE- M, although the combinations are not the same as those shown in Table 1 of Collins et al. (2006). For historical reasons, the sea-ice scheme in HadCM3 is contained in the atmosphere component of the model and parameters in the scheme are perturbed in line with the equivalent S-PPE-M ensemble AO-PPE-O The fully coupled HadCM3 is used with the standard atmosphere settings (with interactive sulphur cycle) but with perturbations to parameters and schemes in the ocean component. The ensemble extends the work of Collins et al. (2007) and Brierley et al. (2009, 2010) who provide details of the physical schemes in HadCM3 that were surveyed for parameters and switches to perturb. Briefly, parameters in the schemes which control horizontal mixing of heat and momentum, the vertical diffusivity of heat, isopycnal mixing, mixed layer processes and water type are varied. A Latin Hypercube design is employed which is efficient in permitting interactions between perturbations to parameters. The same spin-up technique used in the AO-PPE-A ensemble is employed to generate flux-adjustment terms. This is in contrast to the experiments described in Collins et al. (2007) where no flux adjustments were employed. In that study it was found that model drift can introduce biases in surface climate which lead to differences in atmosphere/ surface feedbacks under climate change. Such biases were considered undesirable here as we wish to isolate the impact of ocean parameter perturbations. The use of flux adjustments also facilitates comparison with the ensembles which employ a slab-ocean and with the AO-PPE-A ensemble. 2.2 Multi-model ensembles Much has been written about the CMIP3 archive of model output and the reader is referred to Meehl et al. (2007b) for a history and to the PCMDI web site and for a constantly evolving list of papers based on the archive. Here we also augment the analysis by using archived output from the CFMIP project (e.g. Webb et al. 2006) in the case of model

7 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics 1743 Table 1 Models used in the multi-model ensembles in this study Model name Atmos-slab Atmos-ocean BCC-CM1 9 BCCR-BCM2.0 9 CCSM3 9 9 CGCM3.1(T47) 9 9 CGCM3.1(T63) 9 9 CNRM-CM3 9 CSIRO-Mk ECHAM5/MPI-OM 9 9 ECHO-G 9 FGOALS-g1.0 9 GFDL-CM GFDL-CM2.1 9 GISS-EH 9 GISS-ER 9 9 INGV-SXG 9 INM-CM IPSL-CM4 9 9 MIROC3.2 (hires) 9 9 MIROC3.2 (medres) 9 9 MIROC3.2 (high sensitivity) 9 MRI-CGCM PCM 9 UKMO-HadCM3 9 UKMO-HadGEM1 9 9 UIUC 9 HadCM4 9 The slab-ocean version of UKMO-HadCM3 is not selected as a member of the multi-model ensemble as that is included as a member of the perturbed-physics ensembles. However, the coupled version is included as this version of HadCM3 is run without flux adjustments and hence may be considered to be different from the flux-adjusted perturbed physics couple model standard member S-MME Fig. 1 Annual mean SST biases in fixed pre-industrial CO 2 simulations with HadCM3 with standard parameter settings. a The non fluxadjusted version of the model submitted to CMIP3. b The version of the model with interactive sulphur cycle and flux adjustments reported in Collins et al. (2006). c The standard version of the model with interactive sulphur cycle and adjusted Haney relaxation coefficients used in this paper in generating ensembles AO-PPE-A and AO- PPE-O. Adjusting the Haney coefficients leads to a reduction in SST biases in all coupled-model simulations versions which use mixed-later ocean formulations. We denote the multi-model ensembles used as follows, to be consistent with the notation adopted above. Different atmosphere models coupled to simple mixedlayer oceans. Model output is extracted from the CFMIP (see e.g., Webb et al. 2006) and WCRP CMIP3 database at PCMDI (Meehl et al. 2007b). 20-year averages from 19CO 2 and 29CO 2 experiments are used. The models used are show in Table 1 and the ensemble consists of 16 members AO-MME We use coupled model output from the 23 models in the WCRP CMIP3 database. Again, the models used are shown in Table 1. There is a significant overlap in model versions between the S-MME and AO-MME ensembles.

8 1744 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics There are a number data limitations in this archive and analysis is only performed on the subset of multi-models for which the data exists and is suitable. 2.3 Experiments and variables Sheer volume of data prevents us from examining all variables from all experiments run using the model ensembles described above. Hence we focus on the following set of experiments and variables because (1) there exists a common set of core experiments that can be easily and fairly compared and (2) they allow us to examine the main feedbacks and forcings under commonly used scenarios for climate change. The experiments examined are: 1. The 19 and 29CO 2 equilibrium runs in the case of all models with mixed-layer/q-flux/slab oceans. For some model experiments 19CO 2 is taken to mean preindustrial levels, while in others it is taken to mean present day, or some other level (year 1900 in the case of the MIROC models for example). We make no practical distinction here as the differences between feedbacks dominate the response at 29CO 2 and because the applied forcing due to doubling does not depend significantly on the chosen 19CO 2 baseline value. 2. Pre-industrial (and in the case of some AO-MME members, present day) control experiments with no external forcing and experiments with 1% per year compounded increase in CO 2. We use 80 years of output from control experiments for both MME and PPE members and 80 years of 1% per year experiments which, for most MME members, are taken from the experiment in which CO 2 continues to increase after year 70 (the 1%to49 experiments). For a handful of MME members, this experiment was not available and the run in which CO 2 is stabilised past the 70 year mark are employed ( 1%to29 experiments). In practice, this makes little difference to the calculation of the transient climate response, effective climate feedback parameter, and other quantities of interest. 3. Experiments forced with historical changes in radiatively important factors. For the PPE ensemble experiments, historical changes in CO 2, methane and some minor greenhouse gases are used, together with changes in sulphate-aerosol emissions and variations in solar irradiance and volcanic optical depth. The origin of the anthropogenic and natural forcing is the same as that in experiments using a subsequent version of the Met Office Hadley Centre climate model (HadGEM1), and are described in Stott et al. (2006). For some of the multi-model members, both anthropogenic and natural factors are included but for others only anthropogenic factors are used for the 20cm3 simulation (see e.g. Forster and Taylor 2006 and Sect. 4.4 later). 4. Experiments forced with future changes in anthropogenic greenhouse gases and aerosols under the SRES A1B scenario. For the AO-PPE members, the solar variability is prescribed by repeating the solar cycle in the period for the years of the scenario. The future volcanic forcing is set constant by holding the volcanic optical depth to the year 2000 values (close to that in the AO-PPE-A control simulations). A range of options appear to be used in the AO-MME. See Forster and Taylor (2006) for more information on both historical and A1B forcings in the WCRP CMIP3 ensemble. We also make use of a very long multi-century simulation of the standard un-flux adjusted coupled version of HadCM3 with fixed concentrations of greenhouse gases. This is in order to estimate the natural variability of model error, climate change feedback parameters and radiative forcing. While multi-century fixed-forcing experiments with other models may yield slightly different estimates of such variability, as we see below, it is common for intermodel or inter-model-version differences to dominate, so the use of output from just one multi-century model experiment is valid. The list of variables examined is; surface air temperature (SAT), sea surface temperature (SST), average precipitation rate, net top-of-atmosphere (TOA) energy fluxes and the shortwave (SW) and longwave (LW) components, TOA cloud radiative forcing (CRF) SW and LW components, mean sea level pressure (MSLP), cloud amount, surface sensible and latent heat fluxes, surface SW and LW fluxes and zonal mean relative humidity. The use of TOA cloud radiative forcing rather than simply examining the clear-sky fluxes is preferable as in regions of sea-ice and land ice/snow small differences between the position of the edge of the ice can dominate the calculation of fields such as root-mean-squared-errors. By differencing the all- and clear-sky fluxes the relative difference in the model performance in terms of the radiative effects of clouds is better captured. We use only time-averaged seasonal and annual fields so that atmosphere-slab and fully coupled simulations may be compared. This list thus represents a combination of impact-relevant variables and variables that have been shown to be linked to climate change feedback processes. They are also the list of variables used by Murphy et al. (2009) in constraining PDFs of future change using the ensemble output described here. Observational data is taken from a number of sources indicated in Table 2. Only one data set is used to calculate

9 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics 1745 Table 2 Observational data employed in this study to assess model errors The principal fields used are indicated in bold. Other observed fields are used to estimate observational errors Variable Observational field Reference Land surface air HadCRUT3 Brohan et al. (2006) Temperature Legates and Willmot (1990) Sea surface temperature HadISST (used to calibrate flux adjustment) Rayner et al. (2003) NCDC SST Smith and Reynolds (2004) GISS SST Hansen et al. (1996) Precipitation CMAP Xie and Arkin (1997) GPCP Adler et al. (2003) Top-of-atmosphere radiative fluxes ERBE Harrison et al. (1990) CERES Wielicki et al. (1996) ISCCP FD Rossow and Lacis (1990) Mean sea level pressure HadSLP2 Allan and Ansell (2006) ERA40 Uppala et al. (2005) Cloud amount ISCCP D2 Rossow et al. (1996) HIRS Wylie et al. (1994) Surface fluxes SOC Grist and Josey (2003) DaSilva Da Silva et al. (1994) Zonal mean relative humidity ERA40 Uppala et al. (2005) AIRS version 5 Aumann et al. (2003) model error terms, but the other fields are used to produce an order-of-magnitude estimate of observational error in the calculation as described below. In some cases gridded observational data are derived from the same raw point information and simply use different statistical techniques to produce the gridded product. The treatment of uncertainties in observations remains a limitation of this study as comprehensive estimates of uncertainty simply do not exist for most variables. Nevertheless, this does represent an advance on previous studies (e.g. Gleckler et al. 2008). 3 Model Errors The purpose of this section is to make comparisons between the modelled and observed mean climate of the members of the different ensembles in order to contrast the perturbed physics and multi model approaches. There are a number of simple and widely used metrics which may be used to quantitatively compare models with observed climate variables (e.g. Taylor 2001). It is not possible here to make a completely comprehensive comparison of all variables, with all possible observational sources, using all possible metrics. We seek rather to perform an analysis and inter-comparison of some of the main features of observed climate between the two different approaches. The analysis uses the climate variables outlined above, which are chosen (based on previous experience) on the basis of their userrelevance and because of their key role in physical feedbacks under climate change. For each of the observed climate variables considered, we interpolate both the observations and the multi-model output onto the spatial grid of the perturbed physics ensemble. This results in the minimum number of interpolation steps because of the large number of perturbed physics members. The global mean bias in a climate variable is defined as the area-weighted globally averaged sum of the grid-box difference between the 20-year and 80-year time-averaged 19CO 2 or pre-industrial or present-day control climates (for slab and coupled models, respectively) and the observed climate variable. The sum is calculated only on grid points at which the observed timeaveraged field exists. The root mean squared (RMS) error, e, is calculated similarly but with the global mean bias removed before the calculation (sometimes called the centred RMS error Taylor 2001). The same step is performed when calculating the correlation between the observed and modelled field. These types of metrics are in the spirit of the Taylor Diagram cited above. The use of either pre-industrial or present-day control run is related to that chosen by the different modelling groups for the initial state of the 1%/year CO 2 increase experiment. As stated above, while there are detectable differences in metrics computed from the two differently specified control runs for a single model (e.g. Reichler and Kim 2008), the generic model error tends to dominate so there is little sensitivity in the final model comparison. We calculate the bias, RMS error and correlation for both seasonal and annual-mean fields but present only the annual-mean values for reasons of space and because they are

10 1746 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics representative of generic errors in different models. We note though that in terms of constraining model predictions using observations, information from the annual cycle may be of some use (e.g. Knutti et al. 2006). In order to get an order-of-magnitude estimate of the observational error, we compute biases, RMS differences and correlations between all pairs of the observational fields listed in Table 2. The maximum bias and RMSE and minimum correlation is then used as a crude estimate of the likely magnitude of the error in the observations. In the absence of numerical estimates of both systematic and random errors in the majority of the observational fields, this is the most simple approach. A conclusion of this study is that more comprehensive estimates of errors in observational data sets are required in order to quantify uncertainty in model projections of future climate change. In the case of the AO-PPE-O ensemble, all bias, RMS error and correlation fields are indistinguishable from the standard version of the model, presumably because of the use of identical atmosphere parameters and flux adjustments, so that ensemble is not discussed extensively in what follows. The values of the error metrics for the AO-PPE-O ensemble are included in the figures for completeness. 3.1 Errors in two-dimensional time-averaged fields Examining land surface air temperature errors in the perturbed-physics model versions with slab-ocean components first, we see biases and RMS errors of the order of a few degrees globally (Fig. 2). In the case of the de-tuned S-PPE-S ensemble with only single parameters perturbed, land surface temperature biases are exclusively negative when compared to the HadCRUT3 observational data set, with the standard model versions placed towards the end of the distribution which is closest to observations. In the case of the tuned S-PPE-M ensemble, there is a wider spread of biases than in the model versions with only one single parameter perturbed, in which positive values are evident. A similar range of RMS errors is evident in the two ensembles, reflecting the optimisation of RMS errors in ensemble design (see above and Webb et al. 2006). Bigger RMS errors are seen in the S-PPE-E ensemble which explores more regions of parameter space. In the slab-ocean multi model ensemble, S-MME, we see a similar range of land SAT biases as in the case of the perturbed physics ensembles, but a somewhat wider range of RMS errors. It is possible that the specification of different surface boundary conditions, which may impact surface air temperature in the multi-model ensemble, promotes a wider range of spatial patterns of surface air temperature. Fields such as orographic height, vegetation and soil properties are identical in each of the members of Fig. 2 Bias, centred root mean squared errors (RMSE) and correlations between two-dimensional time-mean modelled and observed c fields. From top to bottom; land surface air temperatures (SAT), sea surface temperatures (SST), precipitation, net top-of-atmosphere (TOA) fluxes (positive incoming), outgoing SW fluxes, outgoing LW fluxes, outgoing SW cloud forcing, outgoing LW cloud forcing, mean seal level pressure (MSLP), cloud amount, surface sensible heat flux, surface latent heat flux, surface SW fluxes, surface LW fluxes and zonal mean relative humidity. Different ensembles (S-PPE-E, etc., see Sect. 2.1) are indicated and one dot is plotted for each ensemble member. The light blue dots show the bias, RMSE and correlation for the ensemble mean of all the models in the ensemble. The red dot indicates the experiment with standard HadCM3 parameter settings, flux adjustments and interactive sulphur cycle. The light grey shading represents an estimate of the uncertainty in observational fields (see text). The dark grey shading indicates the mean and ±2SD of 20 or 80-year means of the value calculated from a multi-century integration of the non-flux-adjusted version of HadCM3 and hence gives an order-of-magnitude estimate of natural variability in the calculated errors the perturbed physics ensembles, although some surfacerelated processes such as the roughness length are perturbed see the appendix to Murphy et al. (2004). We also note at this point that correlation scores are of little use when comparing land surface air temperatures in models as they are is close to unity for all model versions, being dominated by the pole to equator temperature gradient. SSTs in models with mixed-layer or slab oceans are tied more closely to observations because of the calibration phase in which the implied ocean heat transports are calculated. The exception is some members of the S-PPE-M and S-PPE-E ensembles where, while part of the spread in biases and RMS errors is due to the multiple-parameter perturbations, part may also be attributed to the aforementioned error that was inadvertently introduced into the calculation of the implied heat transports. Despite this, both SST bias and RMS errors are of a similar magnitude in slab-ocean perturbed physics and multi model ensembles and are in many cases smaller than those errors seen in the non-flux-adjusted CMIP3 coupled models (AO-MME). As mentioned above, we have re-run a number of experiments where noise in the calculation of the slab-model heat flux convergence fields was present and found that this has a relatively small impact on global error characteristics and feedbacks. Turning to the coupled model ensemble experiments, the range of biases in SST is generally smaller in both atmosphere-parameter-perturbed (AO-PPE-A) and oceanparameter-perturbed (AO-PPE-O) ensembles in comparison with the coupled multi-model ensemble (AO-MME). Similarly, RMS errors are smaller. This is because of the exclusive use of flux adjustments in the former which tend to limit (but not eliminate) the formation of SST errors. Perhaps surprisingly however, the range of land SAT biases is also smaller in the flux-adjusted coupled PPE simulations than in the multi-model case and, correspondingly,

11 M. Collins et al.: Climate model errors, feedbacks and forcings: a comparison of perturbed physics 1747 the land surface air temperature RMS errors are generally smaller than those seen in many of the non-flux-adjusted coupled multi-model members. There are reasonably large top-of-atmosphere net flux imbalances in some of the AO- PPE-A members which might be expected to lead to large land SAT errors, but it seems that having better ocean SSTs