Assessment of the CMIP5 global climate model simulations of the western tropical Pacific climate system and comparison to CMIP3

Transcription

1 INTERNATIONAL JOURNAL OF CLIMATOLOGY Int. J. Climatol. 34: (2014) Published online 7 February 2014 in Wiley Online Library (wileyonlinelibrary.com) DOI: /joc.3916 Assessment of the CMIP5 global climate model simulations of the western tropical Pacific climate system and comparison to CMIP3 Michael R. Grose, a * Jaclyn N. Brown, b Sugata Narsey, a Josephine R. Brown, c Bradley F. Murphy, c Clothilde Langlais, b Alex Sen Gupta, d Aurel F. Moise c and Damien B. Irving a a Centre for Australian Weather and Climate Research, CSIRO Marine and Atmospheric Research, Aspendale, Australia b Centre for Australian Weather and Climate Research, CSIRO Marine and Atmospheric Research, Hobart, Australia c Centre for Australian Weather and Climate Research, Bureau of Meteorology, Melbourne, Australia d Climate Change Research Centre, University of New South Wales, Sydney, Australia ABSTRACT: A set of 27 global climate models from the Coupled Model Inter-comparison Project Phase 5 (CMIP5) ensemble are assessed for their performance for the purpose of making future climate projection studies in the western tropical Pacific and differences to Coupled Model Inter-comparison Project Phase 3 (CMIP3) are assessed. The CMIP5 models show some improvements upon CMIP3 in the simulation of the climate in the western tropical Pacific in the late 20th century. There are fewer CMIP5 models with very poor skill scores than in CMIP3 for some measures and a small group of the well-performing models in CMIP5 have lower biases than in an equivalent group from CMIP3. These best-performing models could be particularly informative for studying certain climate sensitivities and feedbacks in the region. There is evidence to reject one model as unsuitable for making regional climate projections in the region, and another two models unsuitable for analysis of the South Pacific Convergence Zone (SPCZ). However, while there have been improvements, many of the systematic model biases in the mean climate in CMIP3 are also present in the CMIP5 models. They are primarily related to the shape of the transition between the Indo-Pacific warm pool and equatorial cold tongue, and the associated biases in the position and orientation of the SPCZ and Inter-Tropical Convergence Zone, as well as in the spatial pattern, variability and teleconnections of the West Pacific monsoon, and the simulation of El Niño Southern Oscillation. Overall, the results show that careful interpretation and consideration of biases is required when using CMIP5 outputs for generating regional climate projections for the western tropical Pacific, particularly at the country scale, just as there was with CMIP3. KEY WORDS CMIP5; climate model; model skill; model bias; Pacific Ocean; regional climate projections Received 25 November 2012; Revised 28 October 2013; Accepted 7 December Introduction The western tropical Pacific Ocean is a challenging region to simulate in coupled global climate models (CGCMs), yet there is a strong need to understand the climate of the region and to produce useful climate change projections for adaptation planning of small island nations (Australian Bureau of Meteorology and CSIRO, 2011a, 2011b). There are several factors that affect the confidence in model projections of climate change, including the reproduction of recent trends and the simulation of physically plausible changes to climate dynamics. An important influence on confidence in model projections is the realism with which the models simulate the current climate mean and variability. While models * Correspondence to: M. Grose, Centre for Australian Weather and Climate Research, CSIRO Marine and Atmospheric Research, Aspendale, Victoria 3195, Australia. Michael.Grose@csiro.au are now sufficiently reliable to provide useful insights into many aspects of the climate system in the region, systematic biases in the simulation of some important features in the Pacific region persist in models, including those in the World Climate Research Project Coupled Model Inter-comparison Project Phase 3 (CMIP3) CGCM database (Meehl et al., 2007). As such, there is strong interest in the Coupled Model Inter-comparison Project Phase 5 (CMIP5) CGCM database (Taylor et al., 2011) compared to CMIP3, particularly to determine if any increased confidence can be placed on climate projections in the region from these latest models. In the western tropical Pacific region there are several cases where the climate dynamics are understood well enough and the CMIP3 models perform with sufficient fidelity to give high confidence in projections of future change. For example, an increase in temperature, sea level and ocean acidification are consistently simulated with enhanced greenhouse conditions, along with an increase in precipitation in the Inter-Tropical 2014 Royal Meteorological Society

2 ASSESSMENT OF CMIP5 CLIMATE MODELS FOR THE WESTERN TROPICAL PACIFIC 3383 Convergence Zone (ITCZ) due to increased moisture transport in a warmer world (Christensen et al., 2007; Australian Bureau of Meteorology and CSIRO, 2011a, 2011b). However, some other key processes in the Pacific are particularly sensitive to model biases, so there is low confidence in the projection, or else projections can only be made after carefully accounting for biases (Brown et al., 2013a). For example, biases in the CMIP3 simulation of tropical atmospheric circulation, convection and mean precipitation may be at the heart of the lack of consensus of models as to the sign of precipitation change over parts of the tropics (Randall et al., 2007). Also, the projection of whether warming will strengthen or weaken El Niño Southern Oscillation (ENSO) is sensitive to the simulation of ENSO in the present climate by CMIP3 models (Collins et al., 2010; Vecchi and Wittenberg, 2010). The systematic model biases in the CMIP3 CGCM database (Meehl et al., 2007) are seen in sea surface temperature (SST) features such as an erroneous location of the eastern edge of the Indo-Pacific Warm Pool and an overly westward extension of the cold tongue, known collectively as the cold-tongue bias (Brown et al., 2013a). Such mean-state errors are pervasive through the ocean-atmosphere coupled system, including biases in the position and orientation of the South Pacific Convergence Zone (SPCZ; Brown et al., 2011), an overly strong or even double ITCZ, an incorrect strength and seasonality of the West Pacific monsoon (WPM) and an incomplete spatial separation of the SPCZ and ITCZ (Irving et al., 2011; Smith et al., 2012). Climatological biases also affect the fidelity of the simulation of climate variability such as ENSO (AchutaRao and Sperber, 2006) and the related latitudinal movements of the SPCZ (e.g. Folland et al., 2002). The CMIP3 models generally exhibit some degree of spurious long-term changes that are unrelated to changes in external forcing, known as model drift (Sen Gupta et al., 2012). The evaluation of mean state, climate features, variability and drift in CMIP3 models was used to reject a set of models as unsuitable for inclusion in regional climate projections in the western Pacific (Irving et al., 2011), and these were not considered in reporting for the region (Australian Bureau of Meteorology and CSIRO, 2011a, 2011b). For the models that are considered, the presence of biases and drift in model simulations means that careful analysis and interpretation is required when generating climate projections at the scale of tropical Pacific nations using CMIP3 model outputs (Christensen et al., 2007; Brown et al., 2013a, 2013in press). So, as with CMIP3 models, the evaluation of CMIP5 models is crucial to inform the confidence ratings in climate projections for the region made using GCM outputs. The new CMIP5 database represents the latest generation of CGCMs, and it updates the CMIP3 database. CMIP models are used in various simulations, including control, past climate and sensitivity experiments, and also global climate projections under future emission scenarios. The CMIP5 models include numerous improvements and additions compared to CMIP3. The largest development is that several models in CMIP5 have coupled biogeochemical components, and some have interactive prognostic aerosol, chemistry and dynamical vegetation components (Tables 1 and 2; Taylor et al., 2011). Models with these additions are often termed Earth System Models (ESMs) rather than as CGCMs. None of the CMIP5 models reviewed here employs any flux-correction. Five of the CMIP3 models featured some form of flux-correction (CGCM3.1(T47), CGCM3.1(T63), ECHO-G, MRI-CGCM1.3.2 and INM- CM3.0). Flux-correction resulted in smaller amounts of model drift in these CMIP3 model simulations (Irving et al., 2011), but its use raises concerns because it interferes with the model simulation in a nonphysical manner. There are also general increases in atmospheric and ocean horizontal resolution and an increase in the number of vertical layers in the atmosphere (Tables 1 and 2) and ocean. Simulations are made using forcings for the 20th century that are similar but not identical to those for CMIP3, and for a new set of future scenarios. There have been ongoing refinements in the fundamental equations and parameterisations in the models, such as cloud processes and feedbacks (e.g. Bony et al., 2011). This has led to some improvements in many model parameters such as column-integrated cloud amount, but little change in others such as water vapour simulation (Jiang et al., 2012). Differences in the model simulation of cloud feedbacks continue to be a large source of uncertainty, and this is one contribution to the large range of modelled climate sensitivity to a doubling of carbon dioxide in CMIP5 models (ranging from 2.1 to 4.7 K; Andrews et al., 2012). Preliminary examination of fourteen CMIP5 models showed that there is no dramatic improvement in ENSO performance in the multi-model mean state compared to CMIP3, but the CMIP5 models from several modelling centres showed improvements over their CMIP3 contribution (Guilyardi et al., 2012). Specifically, many of the CMIP5 models still show biases in the intensity of El Niño events, however, there are improvements in the spatial patterns of ENSO and a greater consistency of model performance (Kim and Yu, 2012). The CMIP3 models are a valuable resource for making climate projections in the tropical Pacific, giving high confidence in the projection of numerous aspects of the climate system in the region, and providing useful input into climate change impact analysis, but lower confidence in some other aspects (Irving et al., 2011). Here we evaluate a group of CMIP5 models in terms of mean climate, key climate features, ENSO and drift in the western Pacific Ocean, as a basis for making climate projections for the nations in the region. We compare the multi-model mean ensemble of CMIP3 with that of CMIP5, as well as discussing the change in the spread between the models. We also assess whether the best models in CMIP5 improve upon the best models in CMIP3. Where possible we also highlight the regions

3 3384 M. R. GROSE et al. Table 1. CMIP5 model details and results for the Pacific Ocean region in the late 20th century (see methods for exact time period and geographic range for each analysis). Resolution is shown by the average latitude resolution of atmosphere/ocean cells. SST Longitude is the position of the SST 28.5 isotherm at the equator. RMSE are calculated for the annual average compared to ERA Interim Reanalysis (Temp and MSLP) or GPCP gridded climate dataset (precipitation). ITCZ spatial correlation is the summary of two regions and each season, ITCZ displacement is the mean difference in the mean ITCZ latitude in El Niño and La Niña events, SPCZ spatial correlation is of DJF rainfall to CMAP, SPCZ correlation to NINO3.4 is of SPCZ latitude in DJF compared to CMAP, WPM correlation is of DJF rainfall to GPCP, WPM SD indicates the inter-annual variability, WPM correlation to NINO3.4 index is of DJF rainfall, SD of the NINO3.4 index and number of La Niña and El Niño events is for , correlation and RMSE of NINO3.4 are calculated for 18 months of El Niño periods compared to HadISST, S* statistics (after Taylor, 2001) are for December February SST and November April precipitation. Values for some scores are colour-coded according to the relative size of the bias across all models in CMIP3 and CMIP5 for visual comparison (green = lowest bias, red = highest bias), colour coding does not represent any meaningful physical threshold. The four example models chosen for generally low biases (green) and the three models considered for rejection (red) are marked in the far right column. A, aerosol; Atm., atmosphere; B, biogeochemistry; cor., correlation; disp., displacement; MMM, multi-model mean; N3.4, NINO3.4 index; No. events, number of El Niño and La Niña events; precip., precipitation; res., resolution; S*, S-statistic; SD, standard deviation; Temp, surface air temperature.

4 ASSESSMENT OF CMIP5 CLIMATE MODELS FOR THE WESTERN TROPICAL PACIFIC 3385 Table 2. CMIP3 model details and results for the Pacific Ocean region in the late 20th century (see methods for exact time period and geographic range for each analysis). Resolution is shown by the average latitude resolution of atmosphere/ocean cells. Root mean squared errors (RMSE) are calculated for the annual average compared to ERA Interim Reanalysis (Temp and MSLP) or GPCP gridded climate dataset (precipitation). SD of the NINO3.4 index and number of La Niña and El Niño events is for , correlation and RMSE of NINO3.4 are calculated for 18 months of El Niño periods compared to HadISST, S* statistics (after Taylor, 2001) are for December February SST and Nov-Apr precipitation. Values for scores are colour-coded according to the relative size of the bias across all models in CMIP3 and CMIP5 for visual comparison (green = lowest bias, red = highest bias), colour coding does not represent any meaningful physical threshold. Ranking from Irving et al. (2011) is shown in the far right column. Atm., atmosphere; cor., correlation; N3.4, NINO3.4 index; No. events, number of El Niño and La Niña events; precip., precipitation; res., resolution; S*, S-statistic; SD, standard deviation; Temp, surface air temperature.

5 3386 M. R. GROSE et al. over which model performance is weak and where confidence in regional climate projections is affected. 2. Data and methods Model outputs from the CMIP3 and CMIP5 archives were accessed through the Earth System Grid (ESG) gateway on the Australian National Computational Infrastructure (NCI, Outputs were from the historical simulations from each model archive; 20c3m from CMIP3, historical from CMIP5 (20th century forcings in these are similar but not identical). A subset of CMIP5 models currently available with sufficient outputs were examined (27 models; Table 1) and all CMIP3 models with relevant data (24 models; Table 2). There will eventually be over 50 models in the CMIP5 archive, so these results are indicative but not a comprehensive evaluation of the full model archive. Run 1 from each model is used for this assessment. Model outputs were examined at native grid resolution, and interpolated to a regular grid for examining multimodel means ( grid for atmospheric parameters and a 1 1 grid for ocean parameters). While analysis of some metrics was performed over slightly different time periods, they are all indicative of late 20th century (periods as noted). Some assessments are performed over longer time periods as this is more appropriate and there are observations available (e.g. SST), and some assessments of particular climate features are performed for the months in which they are most active (e.g. SPCZ considers precipitation in December, January and February). Many evaluation methods used here were adapted from Irving et al. (2011); see this article for more information. Simulated SST is compared to the HadISST gridded dataset (Rayner et al., 2003; Met Office, Hadley Centre, 2006). Modelled surface air temperature, mean sea level pressure (MSLP) and winds at the 1000 hpa level are compared to the ECMWF Interim Reanalysis (ERA Interim; Dee et al., 2011). Reanalysis is used for surface air temperatures rather than station records or gridded datasets as these contain substantial temporal and spatial gaps and inhomogeneities for the Pacific region (Murphy et al., 2011). Precipitation is compared to the Global Precipitation Climatology Project (GPCP) gridded dataset (Adler et al., 2003) or the Climate Prediction Centre Merged Analysis of Precipitation (CMAP; Xie and Arkin, 1997). It should be noted that over the tropical oceans there are some systematic differences between GPCP and CMAP datasets (Yin et al., 2004), and also differences between reanalyses (e.g. Chelliah et al., 2011) so that ERA interim will not be precisely the same as other reanalyses. The mean SST was examined over the entire Pacific Ocean for Surface air temperature, MSLP, 1000 hpa winds and precipitation were assessed over the domain 30 N to 30 S and 110 E to 140 W for annually, for December February (DJF) and June August (JJA). Spatial correlation, standard deviation and root mean squared error (RMSE) of the climatology in the models were compared to reanalysis and gridded climate datasets over this period, using the method of Taylor (2001). As well as mean precipitation in key seasons, the modelled seasonal precipitation cycle was assessed by correlating the simulated seasonal cycle (based on the period) with the GPCP seasonal cycle, on a grid point by grid point basis. Climate features were assessed over the period for CMIP3 and for CMIP5 models except where noted. The pattern of ITCZ precipitation was assessed through spatial correlations calculated over west (0 15 N, 165 E 165 W) and central (0 15 N, W) tropical Pacific regions in each season and then these regional and seasonal values were averaged to give a single score (after Irving et al., 2011). In addition, the displacement of the ITCZ between phases of ENSO in DJF was calculated as the difference in the latitude of the peak precipitation in the ITCZ during La Niña and El Niño events in the period (events defined as when DJF NINO3.4 SST exceeds ±0.6 standard deviations). The spatial pattern of the SPCZ was assessed by the spatial correlation of mean DJF precipitation with CMAP precipitation in the area 0 30 S, 155 E 140 W. The latitude and slope of the SPCZ is assessed in mean DJF precipitation using the line-fitting technique of Brown et al. (2011). Also, the temporal correlation of the detrended DJF SPCZ latitude series with the DJF NINO3.4 index in ( for CMIP3, and for CanCM4) was assessed in comparison to the correlation between CMAP precipitation and observed NINO3.4. The WPM is defined as the southern extension of the larger Asian-Australian monsoon that moves into the tropical southern hemisphere in the austral summer months (primarily DJF). We assessed the WPM over the region of E and 20 S 20 N. The spatial correlation of modelled DJF precipitation to GPCP precipitation, the inter-annual variability in DJF precipitation, and the ENSO teleconnection through the correlation of DJF precipitation to the NINO3.4 index were assessed. ENSO variability was assessed for the period in all available CMIP3 and CMIP5 models. The standard deviation of the NINO3.4 anomaly in all months was calculated and El Niño and La Niña events were defined as when the December to February mean of the NINO3.4 index exceeded ±0.66 standard deviation of the monthly index. The total number of El Niño and La Niña events was summed as a measure of frequency of events. The temporal evolution of El Niño events was assessed in each model from a composite time series of the 18 months starting in January of each El Niño year and ending in June of the following year. The temporal correlation and RMSE between the modelled NINO3.4 index composite and that from HadISST was calculated. The spatial patterns of SST and precipitation during El Niño and La Niña events were assessed through the S-statistic (S*), which combines the pattern

6 ASSESSMENT OF CMIP5 CLIMATE MODELS FOR THE WESTERN TROPICAL PACIFIC 3387 correlation and standard deviations of patterns (Taylor, 2001). S-statistics were calculated for mean patterns of December to February SST anomalies and November to April precipitation anomalies during El Niño and La Niña events in the area 25 S 20 N and 120 E 150 W but excluding Australia (south of 10 S and west of 155 E), and compared to observations. The SST along the equator during each phase of ENSO (El Niño, neutral and La Niña) in the period was assessed in 24 CMIP3 models and 24 CMIP5 models by comparing the modelled profile to the equivalent in HadISST observations. For this analysis, El Niño and La Niña months were defined as when the NINO3.4 SST (with long-term trend removed) exceeds ±0.66 of a standard deviation (all other months are considered neutral). The relative importance of drift to historical trends in SST and precipitation was assessed in 23 CMIP5 models for the region 25 S 20 N and 120 E 150 W. The magnitude of drift was calculated as the trend in the control simulation (picontrol) from 1950 to 2050, and this was compared to the trend in the historical simulation from 1950 to Drift may be corrected for, or excessive drift may be a basis for rejecting a model from model projections. We have not ranked the models in order of overall quality from first to last, as the set of results presented here is not comprehensive, some measures are not independent and it is not clear how to weight each result to create an overall ranking. Instead, we make general comments about the multi-model mean and discuss particular examples. We examine averages for a group of four models with low biases across most measures for the purpose of comparison only. The four selected from CMIP3 are taken from the overall ranking given in Irving et al. (2011): CSIRO-Mk3.5, ECHO-G, GFDL-CM2.0 and MRI-CGCM The four selected for CMIP5 are ACCESS1.0, CCSM4, CNRM-CM5 and NorESM1-M based on Table 1. The selection of these four models is not a rating of best models, but a selection to compare models within CMIP3 and CMIP5that perform well for these features only (models may not have low biases for other features). This selection should not be taken as any guide for weighting or sub-setting models. 3. Results 3.1. Mean state The CMIP5 models examined here exhibit biases in the shape of the Indo-Pacific warm pool and intensity and position of the equatorial cold tongue, known collectively as the cold-tongue bias. A useful gauge of the coldtongue bias is the shape and longitude of the eastern edge of the Indo-Pacific warm pool. In observations this marks the boundary between warm, fresh water and cooler, saltier water and is commonly defined by the 28.5 C isotherm (Brown et al., 2013b). A bias in the shape of the edge of the warm pool is present in all (a) (b) (c) Figure 1. Annual mean sea surface temperature from (top) HadISST observations, (middle) the multi-model mean of CMIP3 GCMs and (bottom) the multi-model mean of CMIP5 GCMs for the period Marked on the temperature plots is the 28.5 Cisotherm, model results show each model member (black lines) and multi-model means (white lines), the CMIP3 mean is shown as a dashed white line and the subset of four selected CMIP5 models are indicated by aqua lines in the CMIP5 panel. of the CMIP5 models examined, as there was in the CMIP3 models, including the four selected models (as outlined in the methods section), seen in Figure 1. The equatorial cold tongue penetrates too far westward in most of the CMIP5 models, as it did in CMIP3. At the equator, the mean longitude of the 28.5 C isotherm in CMIP5 models (155 ) is slightly closer to observations (181 ) than the mean of CMIP3 (151 ). Similarly, the mean of the group of four selected CMIP5 models (163 ) is slightly closer to observed compared to the four selected CMIP3 models (158 ). Alternative definitions of the warm pool edge such as salinity gradients (Brown et al., 2013c) also reflect this cold-tongue bias in CMIP5 models. An outlier in the CMIP5 group is the GISS- E2-R model, which has upwelling that is too weak, so places the 28.5 C isotherm at the equator to the east of observations (Figure 1; Table 1). Some CMIP5 models tend to overestimate SST in the ITCZ region as CMIP3 models tended to do, however, no CMIP5 model has a 28.5 C isotherm that extends across to the eastern Pacific. The ocean and atmosphere are directly coupled, so that the transition between the cold tongue and warm pool directly corresponds with the edge of deep atmospheric convection. Therefore, biases in SST directly correspond with many of the biases in air temperature, pressure and precipitation (Figures 2 4). The most prominent bias in air temperature in the multi-model mean of the CMIP5 models is at the equator associated with the cold-tongue bias, as it was in CMIP3

7 3388 M. R. GROSE et al. Figure 2. Mean surface air temperature in reanalysis and models for the period in ERA Interim reanalysis, the multi-model mean of CMIP3 models and the multi-model mean of CMIP5 models. Rows show the mean for annual, June August (JJA) and December February (DJF). (Figure 2). In all seasons, cooler air temperatures extend too far west along the equator than observed. There is little difference between the multi-model means of CMIP3 and CMIP5 models in most seasons, except there is a distinctly reduced cold-tongue bias in JJA in the CMIP5 models (Figure 2). The cold-tongue bias has an impact on the simulation of mean temperature and the seasonal cycle of temperature in the broad Pacific region, and has flow-on effects on the simulation of moisture and also precipitation. The bias has particular relevance for climate projections directly within the cold-tongue region, including many islands within the nations of Kiribati and Nauru. The bias in the current climate has an inevitable impact on the dynamics of climate change of this region, affecting confidence in projections. The mean and range of RMSE in mean annual surface air temperature is slightly lower in the CMIP5 models (mean of models and inter-model standard deviations 1.12 ± 0.31 C) than in the CMIP3 models (1.24 ± 0.32 C), shown in Table 1. The main difference between models with a low RMSE and a high RMSE over this domain is in the size of the cold-tongue bias (see Supporting Information). There are fewer models with very high RMSEs for surface air temperature in CMIP5 compared to CMIP3 (Tables 1 and 2), and a lower RMSE for the mean of the groups of four selected models (0.86 C for CMIP5 and 1.22 C for CMIP3). Over the Southeast Asian region including Papua New Guinea there are generally smaller biases in the land-ocean contrast of surface air temperature in this group of CMIP5 models, at least partly due to an increased resolution of topography. It should be noted that there is a lack of observations to constrain reanalyses in this region, contributing to some inherent biases in the reanalyses the models are compared to (e.g. Tebaldi and Knutti, 2007). The simulation of mean atmospheric circulation over the tropics is strongly coupled to the SST patterns such as the east-west gradient in SST between the coldtongue and the Indo-Pacific warm pool. For example, in CMIP3 the bias in the longitude of the edge of the Indo-Pacific warm pool led to a systematic bias in the location of the rising branch of the Walker circulation, and this in turn may affect the simulation of ENSO (e.g. Turner et al., 2005). The multi-model mean of CMIP5 models also shows a systematic bias in the location of the rising branch of the Walker circulation, seen as a negative pressure bias over parts of southeast Asia and as overly strong mean wind in parts of the E region (Figure 3). The RMSE of mean annual MSLP is slightly lower in the CMIP5 models (mean of models and inter-model standard deviations 1.27 ± 0.82 hpa) than in the CMIP3 models (1.51 ± 1.15 hpa), shown in Table 1. There is also a slight improvement between the four selected models for each archive (0.80 hpa for CMIP5, 0.91 hpa for CMIP3). Some models show a distinct negative pressure bias over southeast Asia and a positive pressure bias in the eastern Pacific (e.g. CSIRO-Mk3.6, see Supporting Information).

8 ASSESSMENT OF CMIP5 CLIMATE MODELS FOR THE WESTERN TROPICAL PACIFIC 3389 Figure 3. MSLP and mean 1000 hpa wind in ERA Interim Reanalysis and the multi-model mean of CMIP3 archive and the CMIP5 archive for the period Rows show the mean for annual, June August (JJA) and December February (DJF). The HadCM3 CMIP5 model has a strong negative bias over southeast Asia, giving it the highest RMSE of 3.91 hpa (Table 1), but the bias in this model is smaller than the CMIP3 model with the highest RMSE; GISS-EH (6.36 hpa). Model biases in the simulation of circulation and convection and their interaction are key to the simulation of mean tropical precipitation (Randall et al., 2007). The features of the broad circulation described above, together with the simulation of convection at all scales are reflected in the regional pattern of precipitation (Figures 4 and 5). The spatial pattern of bias in the multi-model mean of precipitation in CMIP5 models appears very similar to the equivalent from CMIP3, with only minor differences in each season (Figures 4 and 5). Notably, these CMIP5 models show some regions of larger positive bias in precipitation compared to CMIP3 (e.g. the SPCZ region in JJA), which is consistent with anomalous fresh water seen in CMIP5 (Figure 1). The RMSE in mean annual precipitation is higher over this domain in the CMIP5 models (mean of models and inter-model standard deviations 2.11 ± 0.44 mm day 1 ) compared to the CMIP3 archive (2.06 ± 0.46 mm day 1 ). However, the bias for the four selected models is slightly lower in CMIP5 (1.76 mm day 1 ) than for CMIP3 (1.79 mm day 1 ). A composite of the bias in these four models shows a much reduced dry bias over the equator consistent with a reduced cold-tongue bias and a reduced wet bias over the maritime continent in both seasons (Figure 5). However, in these four models the wet bias is slightly greater in the vicinity of the convergence zone in the colder season; at the northern edge of the SPCZ in JJA and within the ITCZ in DJF. A Taylor diagram of the mean annual precipitation (Figure 6) shows a difference in spread of models in terms of spatial correlation. The CMIP5 archive contains fewer models with low correlation scores (<0.5) and more models in the higher range of correlation scores (>0.8) compared to CMIP3. The location of the CMAP climatology on the Taylor diagram illustrates the difference between observed datasets for the region. It shows that none of the models lie within this range of observed uncertainty for spatial correlation, but many models are for spatial variance. The range of spatial standard deviation scores is similar between the two model groups and the standard deviation of both ensemble means are within the observed uncertainty. Two models that had the poorest simulation of mean rainfall in CMIP3 were GISS-EH and INMCM3 (Irving et al., 2011), but the latest versions of these models (GISS-E2H and INMCM4) perform significantly better for mean annual rainfall over this domain. However, another version of GISS (GISS- E2R) has a high RMSE Climate features and seasonal cycle As mentioned above, the mean rainfall climatologies of CMIP5 models generally show biases in climate features

9 3390 M. R. GROSE et al. Figure 4. Mean precipitation in GPCP gridded climate dataset, the multi-model mean of the CMIP3 archive and the CMIP5 archive for the period Rows show the mean for annual, December February (DJF) and June August (JJA). that reflect the cold-tongue bias and associated circulation biases, similar to many CMIP3 models (Figures 4 and 5). Models typically simulate too little precipitation along the equator and too much precipitation to the north and south of the cold tongue in the ITCZ and SPCZ regions. The CMIP5 models also have an overly zonal SPCZ that can be too far north in austral winter in some models, similar to CMIP3. Values of the slope and latitude of the SPCZ in DJF in these CMIP5 models are generally similar to CMIP3 models (Figure 7), however, there are fewer outliers (only one CMIP5 model has an SPCZ north of 9 S, compared to eight CMIP3 models). Biases in wind fields within the SPCZ region and between the SPCZ and the equator in DJF (Figure 3) indicate a similar bias in atmospheric convergence in the multi-model mean of both CMIP3 and CMIP5 (bias in each model shown in Supporting Information). The spatial correlation of DJF rainfall over the SPCZ region is slightly improved (closer to one) in CMIP5 model compared to CMIP3 models (mean of models and intermodel standard deviations 0.74 ± 0.17 for CMIP5 and 0.68 ± 0.14 for CMIP3), and the four selected CMIP5 models are also slightly improved (0.89) compared to CMIP3 (0.85). Two models had a particularly poor spatial correlation of SPCZ rainfall in DJF (MIROC-ESM and MIROC-ESM-CHEM). The SPCZ is the dominant precipitation feature for many Pacific Island nations, and biases in the simulation of the SPCZ are crucial to making precipitation projections there. The nations of the Solomon Islands and Vanuatu are directly within the region of positive precipitation bias, and the nations from Tuvalu across to French Polynesia are in the region where the SPCZ can extend too far east in DJF and can be too strong in JJA. The ability of CMIP5 models to simulate the SPCZ is investigated further in Brown et al. (2012), and further analysis into the projection of the SPCZ accounting for model biases are included in Widlansky et al. (2012), and extreme swings of the SPCZ in Cai et al. (2012). A visual inspection of the mean DJF rainfall climatology for the entire tropical Pacific (not shown) shows that some models have an obvious double ITCZ bias, as defined by de Szoeke and Xie (2008) as overly zonal SPCZ that extends as a continuous band to South America (e.g. GISS-E2-H). A larger number of CMIP5 models have a double ITCZ bias where there is an additional rain band in the eastern Pacific just south of the equator (e.g. INM-CM4 and GFDL-CM3) compared to CMIP3. The double ITCZ error is thought to be associated with a range of model biases including SST, wind and cloud biases in the eastern equatorial Pacific (de Szoeke and Xie, 2008), biases in ocean-atmosphere feedbacks (Lin, 2007; Zhang et al., 2007), thresholds for deep convection (Bellucci et al., 2010) and an overly strong seasonal cycle of winds in the eastern Pacific (Wittenberg et al., 2006). Spatial pattern correlation summaries for precipitation in the ITCZ region (Table 1) are closer to one in these CMIP5 models (multi-model mean score and intermodel standard deviation is 0.70 ± 0.22) than the CMIP3

10 ASSESSMENT OF CMIP5 CLIMATE MODELS FOR THE WESTERN TROPICAL PACIFIC 3391 Figure 5. Difference in mean precipitation between GPCP gridded observations and the multi-model mean of the CMIP3 models (left column) and the multi-model mean of CMIP5 models (middle column) and in the subset of four CMIP5 models (right column) in the period Rows show the mean for annual, June August (JJA) and December February (DJF). archive (0.64 ± 0.22; Irving et al., 2011). Also, several CMIP5 models have a high correlation of over 0.9 (Had- GEM2-ES, HadGEM2-CC, CNRM-CM5) and the mean correlation in the four selected models is higher in CMIP5 (0.82) than in CMIP3 (0.75). The mean position of the ITCZ shifts 2.9 in latitude in response to ENSO cycles, and this is underestimated in the CMIP5 models (mean of models and inter-model standard deviations 2.2 ± 1.5 ) by the same amount as the CMIP3 models (2.2 ± 1.5 ; Tables 1 and 2). The ITCZ north of the equator is the key precipitation feature for parts of the Southern Marshall Islands and the eastern islands of the Federated States of Micronesia (e.g. Chuuk, Pohnpei and Kosrae). The positive bias in the ITCZ here has a direct impact on confidence in climate projections for these islands. There is a large range of results in the simulation of the WPM, with a few models out-performing others in each metric (Tables 1 and 2). The spatial pattern of the WPM precipitation was simulated with lower correlation compared to GPCP in CMIP5 (mean of models and intermodel standard deviations of correlation 0.61 ± 0.10) compared to the CMIP3 models (0.65 ± 0.14; Irving et al., 2011). Three models that also rated highly for mean rainfall and SST have the highest correlation to GPCP (CNRM-CM5, NorESM1-M, NorESM1-ME). A slightly different set of models performed closest to GPCP in terms of inter-annual variability, and most models simulated an inter-annual variability that is far lower than observed (Table 1). There is a strong teleconnection of the WPM to NINO3.4 present in observations (correlation of approximately 0.8). In some cases, the strength of the cold-tongue bias affects the ability of a model to reproduce this teleconnection. Some models with a low cold-tongue bias show a correlation close to 0.7 or even better. Other models with a stronger cold-tongue bias show a correlation that is not negative enough, or even positive (e.g. IPSL-CM5A-LR; Table 1). The biases in the SPCZ, ITCZ and WPM have consequences for the seasonal cycle of rainfall. The models that have an SPCZ that is too zonal and too far north generally have too little rainfall in DJF and too much rainfall in JJA at the northern edge of SPCZ region (0 10 S). This creates a seasonal cycle of rainfall with a negative correlation to observations in these models. A negative correlation is found in this region in the multimodel means, and this bias is stronger in the mean of CMIP5 models than in CMIP3 (Figure 8). This pattern is fairly consistent across most models. Since the CMIP5 models have a larger bias in the SPCZ intensity and latitude than CMIP3, especially in JJA (Figure 4), the correlation of the seasonal cycle to observations is also poorer at the northern edge of the SPCZ (Figure 8). This bias is over some islands of Tuvalu, so is of particular relevance to making climate projections for this nation. There is also an area with negative correlation between the modelled annual cycle and observations at approximately 5 N and 135 E (Figure 8). This is where the models produce too much rainfall in DJF and too

11 3392 M. R. GROSE et al. Figure 6. Taylor diagram (after Taylor, 2001) for annual mean rainfall over the period over the area 30 Nto30 S, 110 E to 140 W in reference to GPCP, showing CMAP, CMIP3 models, CMIP5 models and multi-model means. Slope ( N/ E) X L H Q J J X NK O D MB WV A S H V IC U E R Z I E U G A C N O F G D B M P a T W T O # Latitude ( S) Figure 7. Slope and latitude of a line fitted through the DJF rainfall maxima of the SPCZ: CMAP (black #) GPCP (black O), CMIP3 models (blue letters), CMIP5 models (red letters), key as for Figure 6. The SPCZ slope and line are calculated for DJF precipitation using the method outlined in Brown et al. (2011). The MIROC-ESM, MIROC- ESM-CHEM MIROC3.2(medres) and MIROC3.2(hires) models did not show a distinct SPCZ precipitation band in DJF that extended east of the dateline, so these are not fitted. little in JJA, linked in part to problems simulating the monsoon. This region contains some islands of southern Palau and the western islands of the Federated States of Micronesia (e.g. Yap State). Gridded datasets such as GPCP have biases in rainfall in many of these regions, especially over the ocean, since there are few observation station inputs (e.g. Yin et al., 2004), which may also contribute to this difference between models and observations. Plots of mean surface air temperature bias, S P K MSLP and winds and precipitation bias in each individual model are included in Supporting Information El Niño Southern Oscillation All of the CMIP5 models simulate ENSO-like variability, unlike the CMIP3 archive that included two GISS models that had no ENSO-like variability. Various measures of ENSO show incremental improvement in the CMIP5 models as a group compared to CMIP3 (Figure 9, Tables 1 and 2; see also Guilyardi et al., 2012). The magnitude of the variability, measured as the standard deviation of NINO3.4, has improved relative to HadISST (0.84) on average in CMIP5 (mean of models and intermodel standard deviations of correlation 0.86 ± 0.24) compared to CMIP3 (0.92 ± 0.44). The group of four models selected for generally low biases for CMIP3 had a standard deviation that was higher than the mean of all models (1.05), and this is the case for the CMIP5 models as well (0.93). The frequency of El Niño and La Niña events in the period in observations calculated here (35 events) is underestimated by the CMIP5 models (31.3 ± 3.5 events) more than the CMIP3 models (32.4 ± 4.1 events). However, with the small number of years the difference in performance between the CMIP5 and CMIP3 model is not significant. Temporal correlations and RMSE of SST evolution over the course of an El Niño event are closer to one in CMIP5 (mean of models and inter-model standard deviations of correlation 0.93 ± 0.05 SD, 1.49 ± 0.86 RMSE) compared to CMIP3 (0.92 ± 0.05 SD, 1.57 ± 0.87). The four selected CMIP5 models had very similar mean scores (0.95 SD and 1.14 RMSE) to CMIP3 (0.95 SD and 1.16 RMSE). The pattern correlation and standard deviation of SST and precipitation during El Niño and La Niña events

12 ASSESSMENT OF CMIP5 CLIMATE MODELS FOR THE WESTERN TROPICAL PACIFIC 3393 Figure 8. Multi-model mean temporal correlation between the climatological mean seasonal (monthly) rainfall cycle from GPCP gridded observations and each of the models in CMIP3 (left panel) and CMIP5 (right panel). are assessed using the S-statistic (S*) in each model compared to observed (Figure 9, Tables 1 and 2). The mean S* of the December to February SST anomalies in the region during El Niño and La Niña events in CMIP5 is 0.68 ± 0.15, markedly more skilful than in CMIP3 models (0.52 ± 0.23). The mean of the four selected models shows a marked improvement between CMIP3 (0.67) and CMIP5 (0.82). Similarly, the mean S* scores for November to April precipitation anomalies over the region during El Niño and La Niña events in the CMIP5 models (0.37 ± 0.22) is more skilful than in the CMIP3 models (0.26 ± 0.21), with an improvement between the means of the four selected models (0.65 in CMIP5 and 0.53 in CMIP3). The difference between the mean of all models and these four selected for good performance indicates the broad spectrum of quality in the simulation of ENSO, and the potential value in examining a few well-performing models for process studies of sensitivities and feedbacks. An important difference in the spatial patterns of mean rainfall anomaly through ENSO cycles in CMIP5 models compared to observations is in the boundary between a positive correlation with rainfall to regions of negative correlation with rainfall. Regions where the models are opposite in sign to observations include regions of Papua New Guinea (not shown), and this is a very relevant feature to account for when making regional climate projections for this nation. Mean SSTs along the equator during the phases of El Niño, neutral and La Niña (Figure 10) show that the oceanic states through ENSO events are broadly captured by the models, though the amount of distinction is subject to the extent of each model s cold-tongue bias. Each CMIP5 model shows a differentiation between a La Niña-like condition where the SST cools and the edge of the warm pool and the cold tongue is advected westward, and an El Niño-like condition where the SST warms and the edge is advected eastward and a neutral state between these two (Figure 10), as was found by Taschetto et al., 2014 and Brown et al. (2013c). No CMIP5 model shows a flat SST profile across the Pacific and a lack of differentiation between phases, as two CMIP3 models do (GISS-ER and GISS-AOM). There is less model spread in the CMIP5 models than in CMIP3, with a greater convergence towards observations indicating some improvements of both cold-tongue bias and also east-west SST profile. However, only a few models show the SST plateau associated with the warm pool. The extent of the warming or cooling and zonal advection of the warm pool edge is different in each model for a variety of reasons, including the strength of the cold-tongue bias, the depth of the thermocline cycle, the depth and strength of the thermocline and the strength of westerly wind Drift Model drift relates to spurious long-term changes in the modelled climate that are unrelated to changes in external forcing. In the CMIP3 models, the magnitudes of modelled trends over the 20th century were often influenced by model drift (Sen Gupta et al., 2012). One CMIP3 model (INGV-SXG) was rejected because of excessive drift in Irving et al. (2011). For these CMIP5 models, drift can distort the externally forced signal, particularly at smaller scales, however, in general the importance of drift has diminished. Figure 11 shows a metric that expresses the typical local importance of drift in a control simulation in relation to historical trends over the latter half of the 20th century. For most of these CMIP5 models, drift in SST typically makes up less than 20% of the historical trend. For precipitation, the drift makes up in excess of 30% of the historical trend and in the case of the GFDL-ESM-2M model makes up almost 70%, although large internal variability in rainfall means that the size of the drift may not be distinguishable from sampling error. Indeed, Sen Gupta

13 3394 M. R. GROSE et al. sd_n34 Nevents CMIP5 CMIP3 CMIP5 CMIP3 Cor_AC RMSE_AC CMIP5 CMIP3 CMIP5 CMIP3 S*_SST S*_Rain CMIP5 CMIP3 CMIP5 CMIP3 Figure 9. Boxplots showing the distribution of model scores for CMIP3 and CMIP5 models of the ENSO metrics from 1950 to 1999: (a) NINO3.4 standard deviation, (b) number of El Niño and La Niña events, (c) correlation between observed and modelled annual cycle of NINO3.4 in El Niño events (Jan of year 0 to June of year +1), (d) RMS error of modelled El Niño NINO3.4 annual cycle, (e) S* statistic for El Niño and La Niña SST anomaly patterns and (f) S* statistic for El Niño and La Niña rainfall anomaly patterns. The box shows the inter-quartile range and outliers are given by circles. et al. (2013) show that for surface properties like SST and precipitation, in models with multiple historical ensemble members, the trend differences (associated with aliasing by internal variability) is generally smaller than the drift signal. While these metrics indicate drift is of typical importance over the whole region, in some sub-regions drift may become more important and should be assessed on a case-by-case basis. We find no CMIP5 model is suitable for rejection due to excessive drift (comparable to the CMIP3 model INGV-SXG). Also, unlike for the CMIP3, all of the CMIP5 models will have concurrent control simulations that run parallel to the historical and RCP simulations. As a result, when necessary a drift correction can be made.

14 ASSESSMENT OF CMIP5 CLIMATE MODELS FOR THE WESTERN TROPICAL PACIFIC 3395 (a) 31 LaNina phase 31 Neutral phase 31 ElNino phase [ C] bccr bcm2 0 cccma cgcm3 1 t47 cccma cgcm3 1 t63 cnrm cm3 csiro mk3 0 csiro mk3 5 gfdl cm2 0 gfdl cm2 1 giss aom giss model e h giss model e r iap fgoals1 0 g ingv echam4 inmcm3 0 ipsl cm4 miroc3 2 hires miroc3 2 medres miub echo g mpi echam5 mri cgcm2 3 2a ncar ccsm3 0 ncar pcm1 ukmo hadcm3 ukmo hadgem1 HadISST CMIP3 MMM Longitude [ E] Longitude [ E] Longitude [ E] (b) [ C] LaNina phase Neutral phase ElNino phase ACCESS1 0 ACCESS1 3 BCC CSM1 1 CanCM4 CanESM2 CCSM4 CNRM CM5 CSIRO Mk3 6 0 GFDL ESM2M GFDL ESM2G GFDL CM3 GISS E2 H GISS E2 R HadCM3 HadGEM2 CC HadGEM2 ES INMCM4 IPSL CM5A LR IPSL CM5A MR MIROC5 MPI ESM LR MRI CGCM3 NorESM1 M NorESM1 ME HadISST CMIP5 MMM Longitude [ E] Longitude [ E] Longitude [ E] Figure 10. Modelled and observed SST along the equator in the Pacific. Top row shows CMIP3 models, bottom row shows CMIP5 models, during left: La Niña phases, centre: ENSO neutral and right: El Niño phases. 4. Discussion This article assesses the late 20th century climate simulation of the tropical western Pacific Ocean as context for making regional climate change projections for the tropical western Pacific nations. Here we examine the simulation of the mean climate and variability as an important influence on confidence in climate projections. A number of improvements were found in the simulation of the mean and variability of the western tropical Pacific climate by the considered group of CMIP5 models over the previous CMIP3 model archive. Improvements in mean-state statistics include a slight reduction in the cold-tongue bias SSTs, reductions in the RMSE of surface air temperature and MSLP including improvements in land-sea contrasts, fewer models with poor spatial correlation of mean annual precipitation. Improvements in the climate features include fewer models with large biases in the latitude and slope of the SPCZ, and an improvement in the spatial correlation of ITCZ precipitation. All CMIP5 models have ENSO-like variability, with slight improvements in the measures examined compared to CMIP3. Also, no CMIP5 model showed unacceptable drift. These improvements are consistent with previous analysis of CMIP5 models (e.g. ENSO analysis in Guilyardi et al., 2012), and demonstrate that ongoing model development has delivered benefits. However, these improvements are generally incremental and not radical. Also, there are some aspects of the mean climate that are not improved in CMIP5 compared to CMIP3. The RMSE of mean annual precipitation over the western Pacific region and DJF precipitation over the WPM region are slightly higher in CMIP5 models than in CMIP3. Also, while fewer models show the classic double ITCZ problem, more models show the double ITCZ problem of an additional rainband in the eastern Pacific south of the equator. The CMIP5 models we examine here retain a number of the same fundamental physical characteristics as the CMIP3 archive described by Irving et al. (2011). Most models continue to have an overly strong equatorial coldtongue and an incorrect shape of the eastern edge of the Indo-Pacific warm pool and a biased strength and slope of the SPCZ. These biases, among others, lead to nontrivial biases in the spatial distribution and seasonality of rainfall. The simulation of ENSO is incrementally improved in CMIP5 compared to CMIP3, but retains much of the same character to the biases (e.g. biases in the spatial patterns of SST and precipitation). It is expected that a model should show some consistency across biases since the cold tongue, warm pool, SPCZ, ITCZ, WPM and ENSO are part of the same system, and each feature interacts with all others. For example, the dynamics of ENSO involve the oscillation of oceanic and atmospheric features such as SST and precipitation around the mean position of the warm pool edge (Clarke, 2008). The consistency of biases in models is shown by the good performance of the four selected CMIP5 models in most measures examined here. These four selected CMIP5 models had lower biases than the four selected CMIP3 models in almost all the scores analysed here (Tables 1 and 2). However, it can be difficult to clearly determine the contributing factors to biases in the mean state and ENSO variability, and their interactions. Indeed, it is now established that the amplitude of