Evaluation of Sub-monthly Precipitation Forecast Skill. from Global Ensemble Prediction Systems

1 2 Evaluation of Sub-monthly Precipitation Forecast Skill from Global Ensemble Prediction Systems 3 Shuhua Li and Andrew W. Robertson International Research Institute for Climate and Society, The Earth Institute at Columbia University, Palisades, NY 4 April 25, 2014 5 To be Submitted to Monthly Weather Review. Correspondence address: IRI - Monell 230, 61 Route 9W, Palisades, NY 10964. Phone: +1 845 680 4491, Fax: +1 845 680 4865, E-mail: shuhua@iri.columbia.edu. 1

6 Abstract 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 The prediction skill of precipitation at sub-monthly time scales during the boreal summer season is investigated based on hindcasts from three global ensemble prediction systems (EPS). The results valid for up to 4 weeks indicate good skill or predictability over some regions particularly over southeastern Asia and the Maritime Continent. The hindcasts from all the three models correspond to high predictability over the first week compared to the following three weeks. The ECMWF forecast system tends to yield higher prediction skill than the other two systems, in terms of both anomaly correlation and mean squared skill score. The sources of sub-monthly predictability are examined over the Maritime Continent with focus on the intra-seasonal MJO and interannual ENSO phenomena in the EPS hindcasts. Rainfall variations for neutral-enso years are found to correspond well with the dominant MJO phase, whereas for moderate/strong ENSO events, the relationship of rainfall anomaly with MJO appears to become weaker, while the contribution of ENSO to the sub-monthly skill is substantial. However, if a moderate/strong MJO crosses the Indian Ocean (phases 2 3) during typical El Niño years, or MJO phases 5 6 during La Niña years, the MJO impact can become dominant, regardless of whether the ENSO event is strong or moderate. These results support the concept that windows of opportunity of high forecast skill exist as a function of ENSO and the MJO in certain locations and seasons, that may lead to subseasonal to seasonal forecasts of substantial societal value in the future. 2

27 1 Introduction 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 There has been a recent surge of interest in prediction on sub-seasonal timescales between the realm of medium-range weather forecasts (up to 7 10 days) and seasonal climate forecasts (3 6 months) (WMO 2013). The sub-seasonal realm lies beyond the predictable impact of atmospheric initial conditions in numerical weather forecasts(lorenz 1975), but falls short of the seasonal averages where anomalies in the atmosphere s surface boundary conditions are felt, principally from sea surface temperatures (SSTs) (Charney and Shukla 1981). This interest has been driven by recent advances in modeling (Vitart 2004) and understanding of climate phenomena on sub-seasonal timescales, and in particular the Madden Julian Oscillation (MJO, see recent review by Zhang (2013)). The concept of seamless prediction, in which there is potential predictive power on all time scales associated with phenomena occurring on that time scale (Hoskins 2012), is particularly attractive in the sub-seasonal time range, where there is evidence that in addition to the inertia of SST anomalies, the MJO (Waliser et al. 2013) and processes involving the stratosphere (Baldwin and Dunkerton 2001, Scaife and Knight 2008), soil moisture (Koster et al. 2010), snow cover (Lin and Wu 2011) surface and sea ice (Holland et al. 2011) can play important roles. Besides the growing scientific basis for sub-seasonal prediction, the weekly-to-seasonal time range is one in which early warning systems for high-impact weather events could be of substantial societal benefit (WMO 2013). This is particularly urgent in the context of increasing societal exposure to extreme weather threats, be they caused by growing populations or due to decadal climate variability or anthropogenic climate change. 3

48 49 50 51 52 53 54 55 56 57 58 59 60 The aim of this paper is to examine sub-monthly precipitation prediction skill for the June September boreal summer season using three ensemble prediction systems (EPS) that are currently used for opertational sub-seasonal prediction. While weather forecasts are usually evaluated on a daily basis, seasonal forecasts are typically issued as 3-month averages, so as to enhance the predictable climate signal by averaging out the daily weather noise (Palmer and Anderson 1994, Mason et al. 1999). In keeping with the intermediate time scale, we focus here on weekly averages at forecast leads of 1 4 weeks. The sub-seasonal range is beyond deterministic weather prediction so that some time aggregation is required, and the weekly scale makes a natural choice. Weekly time averages are chosen for simplicity and convenience of comparison with observations, although predictability is likely to be enhanced (perhaps considerably) using daily rainfall frequency, especially in the tropics (Moron et al. 2006). Due to the limited ensemble size (4 5) for all those models, the skill evaluation will be confined to deterministic skill metrics only. 61 62 63 In Section 2 a description of models and precipitation data, along with the analysis methods, is provided. The results are presented in Section 3, with a summary and discussion given in Section 4. 4

64 2 Models and data 65 2.1 Description of models 66 67 68 69 70 71 72 73 The model forecast data analyzed in this study are hindcasts from the National Centers for Environmental Prediction (NCEP) Coupled Forecast System, version 2 (CFSv2) (Saha et al. 2010), the Japanese Meteorological Agency (JMA) One-month EPS (JMA 2013), and the European Center for Medium-range Weather Forecasts (ECMWF) VarEPS-monthly system used for operational monthly forecasts at ECWMF (Vitart et al. 2008). All the three sets of hindcasts are produced based on global EPS forecast systems (see Table 1 for detail). Since these forecast data have 4 5 ensemble members only, our analysis uses the ensemble mean for each of the three model hindcast sets. 74 75 76 77 78 79 80 The NCEP CFSv2 (Saha et al. 2014) is a fully coupled dynamical prediction system. The atmospheric component of CFSv2 is a 2007 version of the NCEP global forecast system (GFS) (Saha et al. 2010), with a spectral truncation of 126 waves (T126) in the horizontal and 64 levels in the vertical. The oceanic component is the Geophysical Fluid Dynamics Laboratory (GFDL) Modular Ocean Model version 4 (MOM4) (Griffies et al. 2004). The CFSv2 system increases the length of skillful MJO forecasts from 6 to 17 days (dramatically improving subseasonal forecasts) over its predecessor (Saha et al. 2014). 81 82 83 The ECMWF forecasting system is referred to as VarEPS-monthly, which is a merged 32-day ensemble system updated once a week that has been run since 2002 (Vitart 2004). The EPS integration uses atmospheric model at T399 resolution forced by persisted SST anomalies 5

84 85 from day 0 to day 10, and then coupled ocean-atmosphere integrations at T255 resolution from day 9 to day 32 (Vitart et al. 2008, Hirons et al. 2013). The ocean component used 86 87 for the coupled system is the Hamburg Ocean Primitive Equation (HOPE) model. oceanic initial conditions include one control and four perturbed ocean analysis. The Thus, 88 89 90 91 the VarEPS-monthly reforecasts (or hindcasts) consist of a five-member ensemble of 32-day merged integrations for the 1992 2009 period. The model s skill at predicting the MJO has improved significantly since 2002, with an average gain of about 1 day of prediction skill per year (Vitart 2004). 92 93 94 95 96 97 98 99 JMA s One-month EPS consists of an atmospheric GCM (AGCM) run here at T159 resolution with atmospheric initial conditions obtained from the JMA Global Analysis and initial land surface conditions obtained from the JMA Land Surface Analysis System. Persisted SST anomalies are used as the lower boundary condition, together with a climatological distribution of sea ice (http://ds.data.jma.go.jp/tcc/tcc/products/model/outline/index.html). The MJO forecast in hindcasts is skillful up to a lead time of 13 days, but the model fails to reproduce the realistic eastward and northward propagation of active convection (Matsueda and Takaya 2012). 100 101 102 103 104 105 As seen in Table 1, the three forecast systems have different model resolution, different frequency of production (ranging from 5-day to 10-day or so), and different physical configuration. However, they all provide daily (6-hourly for CFSv2) forecasts for up to about one month or beyond. Since our analysis is focused on targets over 4 weeks, all the hindcast data of precipitation were truncated at day-28, but spanning over all the available start dates during late May through mid-september. The common period for the three data sets 6

106 107 is 1992 2008, and the spatial resolution, varying greatly as shown in Table 1, is re-gridded to the resolution of the observed precipitation data set, i.e., 144 x 72 grid points at 2.5 108 longitude/latitude. Note that the original horizontal resolution for ECMWF hindcast is 109 110 111 T255, equivalent to approximately 80 km, while the JMA hindcast originally has T159 reso- lution, or about 125 km grid. The data sets for these two hindcast sets provided to us were re-gridded onto 1 and 2.5, respectively, as shown in Table 1. 112 113 114 The hindcasts used in this study were obtained in early 2102, and correspond to the versions of the oprational systems used at that time. Both the ECMWF and JMA systems have since been updated, while the CFSv2 is fixed. Table 1: Three global EPS dynamical models. The JMA and ECMWF hindcasts start from May 20, and then updated every 10-day or so (i.e., May 20, 31; June 10, 20, 30; and so on), and every 7 days, respectively. The CFSv2 hindcast starts from May 21, and then updated every 5 days. Model Grid Resolution Ensemble Frequency # of Starts Period JMA 144 x 73 5 3/month 13 1979 2008 CFSv2 384 x 190 4 5-day 25 1982 2010 ECMWF 360 x 181 5 weekly 18 1992 2009 7

115 2.2 Precipitation data 116 117 118 119 120 121 The NCEP Climate Prediction Center (CPC) Merged Analysis of Precipitation (CMAP) provides global precipitation data from 1979 onward at 2.5 resolution, using gauge observations, satellite estimates, and numerical model predictions (Xie and Arkin 1996, 1997). We use the CMAP pentad (5-day average) precipitation data during 1992 2008. This dataset is used to obtain daily estimates first using interpolation, from which the weekly averages are then calculated to match the model hindcast data. 122 2.3 Skill metrics 123 124 125 126 127 128 129 130 131 132 The most basic and commonly-used skill metric for verification of seasonal climate forecasts is the anomaly correlation coefficient (ACC), calculated between ensemble mean of model products against observed quantities (e.g., Barnston et al. 2010). In this study, we use the ensemble means of the model precipitation forecasts, averaged over all available start dates (ranging from 13 to 25 during the boreal summer season, varying among the three models) and valid for weeks 1 to 4. The corresponding 4 weekly observations are computed from CMAP pentad mean rainfall data. Thus, the ACC is computed over all the available start dates during late May through mid-september over the 1992 2008 period. Note that in order to reduce the impact of annual cycle, the seasonality over the entire period is removed for the model hindcast and observed rainfall, respectively. 133 A second skill score is the mean square skill score (MSSS), which is based on mean squared 8

134 135 136 137 138 error and often used in verification of deterministic forecasts (e.g., Goddard et al. 2013). The MSSS is a function of both the prediction to be evaluated, a reference prediction (here climatology), and the corresponding observations (Murphy 1988). Thus the overall bulk MSSS value provides a comparison of forecast performance relative to the reference climatological prediction, together with an evaluation of the contribution from the conditional biases in- 139 herent in the forecasts (see Goddard et al. 2013). Due to the very limited ensemble size 140 (4 5) of the three EPS products, no probabilistic skill metrics are examined in this study. 141 3 Results 142 3.1 Anomaly correlation coefficient skill (ACC) 143 3.1.1 Spatial distribution over the globe 144 145 146 147 148 149 150 151 The ACC maps between JMA precipitation hindcast and CMAP observation over the globe (50 S 60 N) are shown in Figure 1. The correlation skill is generally very high during the first week and drops rapidly in most regions in the subsequent 3 weeks. Nevertheless, there is consistently high ACC for all the leads, or weeks 1 4, over the equatorial Pacific, located to the south of the Intertropical Convergence Zone (ITCZ). Weeks 2 4 also exhibit relatively higher skills (0.2 0.3) over the tropical Atlantic, and the Maritime Continent, along with some land areas and the Indian Ocean at week 1 only. The 13 hindcasts per year over 17 years yields a very large sample size of 221 hindcasts. Using a two-sided Student t-test with 9

152 153 154 220 degrees of freedom, an ACC value of 0.2 is highly statistically significant, with a p-value of 0.003. Poor ACC values are evident at all leads in the climatologically dry regions of the southeastern tropical Pacific and much of southern tropical Atlantic. 155 156 157 158 159 160 161 162 163 The ACC distribution for the CFSv2 and ECMWF precipitation forecasts exhibit most of the above characteristics (Figures 2 and 3). However, the ACC skill of ECMWF is notably higher than the other two models, especially over the tropics for weeks 2 4. Indeed, relatively high ACC over the tropical Pacific resembles slightly broader ITCZ-like characteristic than that from JMA or CFSv2. The high ACC feature from the ECMWF precipitation hindcast is also seen over the equatorial Atlantic and tropical Indian ocean, as well as the Maritime Continent (Figure 3), where higher ACC exists compared to the JMA and CFSv2 hindcasts. These dominant characteristics will be further discussed in next section based on aggregate ACC skill over the tropical land area only and several specific regions of interest. 164 165 Skill levels over the continents are generally disappointing especially in weeks 3 4, although the ECMWF model does exhibit substantial skill at week 2 over South America, Eurasia, 166 and Australia. Skill levels over Africa are poor even in week 1 which suggests that the 167 168 observational data quality may be poor on the weekly timescale. This may be particularly an issue when constructing the weekly averages from the CMAP pentad data. 169 3.1.2 Aggregate ACC over specific regions 170 171 Figure 4 shows the spatially aggregated ACC values for all the three models over the tropical land only within 20 latitudes, represented by percentage of grid points where the ACC 10

172 173 174 exceeds a threshold value between 0.1 and 0.9 (Li et al. 2008). Over weeks 1 4, the aggregate skill from ECMWF hindcast is obviously above the other two models, while JMA and CFSv2 illustrate skill close to each other, especially for the first week. 175 176 Over southeast Asia (Figure 5, ranging from 90 135 E and 10 S 15 N), relatively higher aggregate skill prevails for all the 4 weeks, even though it gradually drops with lead time 177 as well. Again, the summarized skill from ECMWF hindcast outperforms the other two 178 179 180 models, particularly for weeks 2 through 4. A small area within this region over Borneo Island will be further investigated in next subsection, along with exploring possible sources of sub-monthly predictability with focus on this area as a case study. 181 3.2 Sources of sub-monthly predictability 182 183 184 185 186 While sub-seasonal predictability may arise from various phenomena as discussed in the introduction, our attention here is devoted to the impact of MJO and ENSO. We first examine the boreal summer teleconnections of each in our CMAP dataset globally, and then focus on a small region of southeast Asia where the skill is particularly high, i.e., Borneo Island. 187 3.2.1 ENSO and MJO teleconnections 188 189 Figure 6 shows the anomaly correlation between precipitation from CMAP and ENSO index (defined as SST anomaly over the NINO3.4 region [5 S 5 N, 120 170 W]) for June-July- 11

190 191 192 193 194 195 196 197 198 August during the 1992 2008 period. The ACC map for the three-month season, with a 95% significance mask, indicates a pattern of high positive correlations over the equatorial Pacific, in general correspondence to the relatively high ACC of the model hindcasts in Figures 1 3. Also evident is the negative, but strong correlations over the Maritime Continent, as well as the eastern Indian Ocean and the tropical Atlantic. These statistically significant features imply possible important relationships between ENSO and sub-monthly predictability of precipitation. Within-season changes in ENSO teleconnections are quite pronounced in some regions such as the tropical north Atlantic which suggest that there may be within season changes in model skill as well. This issue is left to future study. 199 200 201 202 203 204 205 206 207 To further examine the impact of ENSO on the skill of sub-monthly precipitation forecasts, we compute the ACC skill separately for 5 ENSO years vs. 5 neutral years based on the JJA Nino 3.4 index. Figure 7 shows ACC maps for the lead-3 ECMWF hindcasts for the two sets of years. The p-value for an ACC of 0.2 is 0.059 for a sample size of 90 (5 years x 18 hindcasts per year), so the pink shaded areas in Figure 7 are still highly significant. The ACC skill for the ENSO years is indeed generally much higher than for the neutral years, except over the equatorial Atlantic and to some extent the Indian Ocean as well. Modes of tropical Atlantic variability as well as the Indian Ocean dipole may be important contributors to the sub-monthly skill in these regions. 208 209 210 211 Figure 8 shows the correlation between CMAP pentad precipitation and the MJO indices defined by (Wheeler and Hendon 2004), namely the first two Empirical Orthogonal Functions (EOFs) of Real-time Multivariate MJO (RMM) series, RMM1 and RMM2. These represent the two leading principal components of intraseasonal near-equatorially-averaged outgoing 12

212 213 214 215 216 217 218 219 220 longwave radiation (OLR) and zonal winds. A correlation value of 0.1 has a p-value of 0.081 for a sample size of 18 samples per JJA season x 17 years. Strong negative correlations exist for both RMM1 and RMM2 over South and Southeast Asia, consistent with the pattern of the summertime MJO (Yoo et al. 2010), also called the boreal summer intraseasonal oscillation (BSISO, Lee et al. 2013). RMM1 also exhibits significant correlations over Central America, the South Atlantic Convergence Zone, and the Gulf of Guinea. Of these regions of significant MJO teleconnections in CMAP precipitation, only SE Asia is a region of significant ACC skill of the three EPS systems beyond week 2, although this MJO influence may partly account for the skill of the ECMWF system in week 2 over Central and South America. 221 3.2.2 The Maritime Continent 222 223 224 225 226 227 Turning to the Maritime Continent, Figure 9 shows a scatter plot of the CMAP precipitation anomalies, over a portion of Borneo Island (four grid-box centered ranging 1.25 S 1.25 N, 111.25 113.75 E), versus the ECMWF lead-3 hindcasts, valid for 18 weeks from 6/4 6/10 through 10/1 10/7 during the period 1992 2008. There is a clear near-linear relationship with an anomaly correlation of 0.52. The lead-2 hindcasts (valid for weeks 5/28 6/3 through 9/24 9/30) exhibit reduced scatter (not shown), with an ACC of 0.62. 228 229 230 231 Figure 10 shows the MJO RMM1-RMM2 phase portrait for June-September 2002 as an example of a typical El Niño year. The MJO was very active with amplitude greater than 1.0 standard deviation during most of June August period. A strong MJO event propagated across the Indian Ocean and Maritime Continent (mostly phases 2 3 4) for mid-june. Strong 13

232 233 234 235 236 MJO events tend to influence the rainfall pattern over the Maritime Continent and Borneo Island in particular. Figure 8 indicates that the negative polarity of both RMM1 and RMM2, in particular of RMM2, are most strongly associated with enhanced CMAP rainfall over Borneo during the June August season, and vice versa. This corresponds approximately to phases 2 3 and 6 7 respectively, which corresponds to the regional labeling in Figure 10. 237 238 239 By compositing the pentad CMAP rainfall anomaly over Borneo (averaged for the 4 grid boxes), in relation to the correponding dominant MJO phase (1 to 8) during boreal summer, 1992-2008, we make the MJO phase composites of Borneo-averaged rainfall anomalies, as 240 shown in Figure 11. It provides overall relationship of MJO phase with Borneo CMAP 241 242 243 244 rainfall anomalies. That is, phases 2 3 of MJO correpond to enhanced Borneo rainfall, while phases 6 7 are associated with greatly-reduced precipitation over Borneo. This appears to be in agreement with the correlation between Borneo rainfall and the leading EOFs of MJO (Figure 8), in particular with the second EOF component, RMM2. 245 246 247 248 249 250 251 252 253 We next select three typical years during 1992 2008 to represent El Niño (2002), La Niña (1999), and neutral ENSO (2001), respectively, and examine the evolution of CMAP vs the ECMWF hindcasts averaged over Borneo. Figure 12 shows time-series of weekly precipitation from lead-2 (top panel) and lead-3 (bottom panel) of the ECMWF hindcasts and observed rainfall from CMAP, averaged for the 4 grid boxes over Borneo Island in 2002. Note that no bias correction of these anomalies was done beyond subtraction of the respective climatology. The verification period is 5/28 6/3 through 9/24 9/30, i.e. 18 weeks in total, for lead-2, and 6/4 6/10 through 10/1 10/7 for lead-3. The upper-case letters, along with the MJO phase number, at the bottom of each panel denote the dominant MJO phase (amplitude 14

254 1.0) for the corresponding 7-day periods. That is, A for Western Hemisphere and 255 256 257 258 259 260 261 262 Africa (phases 1 and 8); I for Indian Ocean (phases 2 and 3); M for Maritime Continent (phases 4 and 5); P for Western Pacific (phases 6 and 7). As shown in Figure 12, there is good correspondence of rainfall anomalies between CMAP and the ECMWF hindcasts. For this El Niño year, the CMAP rainfall anomalies are mostly negative as expected, and this is captured by the model at both leads. However, the observed rainfall becomes far-above average for the third week (6/11 6/17), when the MJO is dominantly over the Indian Ocean, or phases 2 to 3, with strength of around 2.0 (Figure 10). This excursion was captured by the model at both leads, although it was too weak, especially at lead-3. 263 264 265 266 267 For the La Niña year 1999, the weekly rainfall anomalies are mostly positive (Figure 13), in response to the La Niña impact. However, during the week 7/23 7/29, CMAP rainfall was nearly 5.5 mm/day below the average, while the MJO was in phase 5, with a strength of about 1.0. This negative excursion was captured in the model hindcasts, although again with weaker amplitude than observed. 268 269 270 271 272 273 274 For the case of neutral ENSO year 2001 (Figure 14), the CMAP rainfall exhibits large intraseasonal fluctuations, quite symmetrically about zero where less precipitation corresponds to MJO phase over Western Pacific (phase 6 or 7) and above-average rainfall tends to be with MJO phases over the Indian Ocean (e.g., phase 2). These rainfall anomalies are remarkably well captured by the ECMWF model at both lead-2 and lead-3, and the attribution of the model s skill to the MJO is clear. Again, the Borneo rainfall anomalies coresponds well with the evolution of MJO (dominant) phases as manifested in Figure 11. 15

275 3.3 Mean squared skill score 276 277 278 279 280 281 282 283 284 285 286 The anomaly correlation coefficient (ACC) is a scale-invariant measure of the linear association between the predicted mean and the observations (e.g., Murphy 1988). Therefore, it can only provide a measure of relative association or potential skill. The mean squared skill score (MSSS) is another important skill metric used to assess deterministic forecasts, based on the mean squared error, which takes into account both the potential skill and biases inherent in the forecasts (Murphy 1988). Figure 15 shows global maps of MSSS from the JMA hindcasts over the same period and starts as for ACC. In general, substantial skill exists for the first week over most of tropical ocean/land and even the northern mid-latitudes, but the skill drops considerably from week 2 to 4. Similar patterns of MSSS skill can also be seen for the CFSv2 precipitation hindcast (Figure 16), even though it looks slightly lower overall compared to the JMA hindcast. 287 Like the intercomparison of ACC among these three models, the MSSS from ECMWF hind- 288 cast (Figure 17) still indicates better skill than the other two models. In particular, it 289 290 demonstrates better skill for the first week, persisting over the tropics for week 2 and over the equatorial Pacific and maritime continent through weeks 3 and 4. 291 292 293 294 295 As mentioned above, the MSSS skill differs from ACC, mainly in the incorporation of forecast biases relative to the ensemble mean. As a summary metric, MSSS combines the square of the correlation coefficient and the square of the conditional prediction bias (Goddard et al. 2013). In other words, the primary difference between ACC and MSSS lies largely in the representation of forecast biases. If the forecast contained no conditional biases, the MSSS 16

296 297 298 299 300 301 302 skill would be determined by the correlation coefficient alone, i.e., their spatial patterns would be identical. Figure 18 shows the difference between the two skill metrics, ACC and MSSS, valid for week 3 from all the three prediction systems. Overall, the CFSv2 hindcast corresponds to largest ACC-MSSS difference, JMA the second, and ECMWF the least out of the three, suggesting that the ECMWF hindcast contains less prediction bias, in general, than the other two models. This comparison also applies to the other leads, i.e., weeks 1, 2 and 4 (not shown). 303 4 Summary and discussion 304 305 306 307 308 309 310 This study has compared the performance of sub-monthly precipitation hindcasts based on three ensemble prediction systems against objectively analyzed precipitation data. The model predictions valid for four consecutive weeks (weeks 1 4 of the forecasts), composited from 13, 18, and 25 start dates, respectively (Table 1), are evaluated using processed weekly precipitation datasets during late May through mid-september over the 17-year period 1992 2008. The prediction skill is quantified using ACC and MSSS deterministic skill metrics over the globe, as well as the tropical lands and several specific regions. 311 312 313 314 315 In general, all the three model hindcast sets indicate very good skill for the first week, and relatively good skill for the 2nd week over the tropics, but dramatically decreased skill for weeks 3 and 4 except for the equatorial Pacific and the maritime continent. Among the three EPS models, the ECMWF forecast system demonstrates noticeably better skill than the other two, especially for weeks 3 and 4. This comparison applies to the prediction skill as 17

316 317 318 manifested by both ACC and MSSS, suggesting that the ECMWF hindcast corresponds to not only the best potential skill, but also the combined measure of relative (linear) association and the biases inherent in the forecasts. 319 The predictability of sub-monthly precipitation has been connected with the low-frequency 320 ENSO variability and intra-seasonal MJO phase/strength. Such associations were exam- 321 322 323 324 325 326 327 328 329 330 331 332 333 ined qualitatively by selecting three typical years of neutral, El Niño and La Niña events in correspondence with weekly time-series of precipitation anomalies over a portion of the maritime continent (Borneo Island), along with the corresponding dominant phase of MJO. Our results suggest that the rainfall variation for neutral-enso years corresponds well with the dominant MJO phase (e.g., above-average rainfall in response to the phase over Indian Ocean); whereas for the situation with moderate/strong ENSO events, the relationship of rainfall anomaly with MJO appears to become weaker, while the contribution of ENSO to the sub-monthly skill is substantial. However, if a moderate/strong MJO event crosses over the Indian Ocean near the maritime continent (e.g., phases 2 3 with stength of greater than 2.0), the rainfall tends to be far above-average for typical El Niño years, such as 2002. For the La Niña year 1999, below-average rainfall is observed when a moderate MJO is over phases 5 6 during late July. In such cases, the MJO impact or modulation becomes significant, regardless of whether the ENSO event is strong or moderate. 334 335 336 337 The advantage of multi-model ensemble (MME) combination forecast has been documented by numerous studies, e.g., Robertson et al. (2004), Li et al. (2008). It has been acknowledged that the overall prediction skill from MME products outperforms the best individual model. Our study here is constrained to model-to-model analysis, simply because the start dates are 18

338 339 non-uniform among the three EPS systems, which makes it dufficult to verify the performance of MME forecast. 340 341 342 343 344 345 346 347 The association of rainfall predictability with ENSO and MJO is only analyzed here for a portion of the maritime continent and for a few typical ENSO years, and our findings may not extend to other continents and years. Further, it is still challenging to quantify the relative contribution of ENSO and MJO to the predictability of precipitation, since ENSO is confined to a fixed location, while the MJO is a propagating wave with variable amplitude and phase. Nonetheless, these results support the concept that windows of opportunity of high forecast skill exist as a function of ENSO and the MJO in certain locations and seasons, that may lead to subseasonal to seasonal forecasts of substantial societal value in the future. 348 349 350 351 Acknowledgments: The authors thank Simon Mason and Anthony Barnston for their insights on skill metrics and ENSO index data usage. This work was funded by grants to the IRI from the US Agency for International Development (AID-OAA-A-11-00011) and the National Oceanic and Atmospheric Administration (NA050AR4311004). 19

352 References 353 354 Baldwin and Dunkerton, 2001: Science, 244, 581 584. Stratospheric harbingers of anomalous weather regimes, 355 356 357 Barnston, A. G., S. Li, S. J. Mason, D. G. DeWitt, L. Goddard, and X. Gong, 2010: Veri- fication of the first 11 years of IRI s seasonal climate forecasts. J. Appl. Meteor. Climat., 49, 493 520. 358 359 Charney, J. G., and J. Shukla, 1981: Predictability of monsoons. Pp.99-109 in Monsoon Dynamics, J. Lighthill abd R. P. Pearce, eds. New York: Cambridge University Press. 360 361 362 Goddard, L., A. Kumar, A. Solomon, D. Smith, G. Boer, P. Gonzalez, V. Kharin, W. Merry- field, C. Deser and S. J. Mason, et al., 2013: A verification framework for interannnual-to- decadal prediction experiments. Clim. Dyn., 40, 245 272, doi: 10.1007/s00382-012-1481-2 363 Griffies, S. M., M. J. Harrison, R. C. Pacanowski, and A. R. Rosat, 2004: A Technical 364 365 Guide to MOM4. GFDL Ocean Group Technical Report NO. 5, 342 pp. [Available online at www.gfdl.noaa.gov/bibliography/related files/smg0301.pdf]. 366 367 368 Hirons L. C. P. Inness, F. Vitart, and P. Bechtold, 2013: Understanding advances in the simulation of intraseasonal variability in the ECMWF model. Part I: The representation of the MJO. Quart. J. Roy. Meteor. Soc., 139, 1417 1426, doi: 10.1002/qj.2060 369 370 371 Holland, M. M., D. A. Bailey, and S. Vavrus, 2011: Inherent sea ice predictability in the rapidly changinf Arctic environment of the Community Climate System Model, version 3, Clim. Dyn., 36, 1239 1253, doi: 10.1007/s00382-010-0792-4 20

372 373 374 Hoskins, B., 2012: The potential for skill across the range of the seamless weather-climate prediction problem: a stimulus for our science, Quart. J. Roy. Meteor., 138, 573 584, doi: 10.1002/qj.1991 375 376 377 378 379 Japan Meteorological Agency, 2013: Outline of the Operational Numerical Weather Prediction at the Japan Meteorological Agency, Appendix to WMO Technical Progress Report on the Global Data-Processing and Forecasting System and Numerical Weather Prediction, 188 pp (available online at http://www.jma.go.jp/jma/jma-eng/jmacenter/nwp/outline2013-nwp/index.htm). 380 381 382 Koster, R. D., and Coauthors, 2010: Contribution of land surface initialization to subseasonal forecast skill: First results from a multi-model experiment. Gephys. Res. Let., 37, L02402, doi: 10.1029/2009GL041677 383 384 385 Lee, J. Y., B. Wang, M. C. Wheeler, X. Fu, D. E. Waliser, and I.-S. Kang, 2013: Real-time multivariate indices for the boreal summer intraseasonal oscillation over the Asian summer monsoon region, Clim. Dyn., 40, 493 509, doi: 10.1007/s00382-012-1544-4 386 387 Li, S., L. Goddard, and D. G. DeWitt, 2008: Predictive skill of AGCM seasonal climate forecasts subject to different SST prediction methodologies. J. Climate, 21, 2169 2186. 388 389 Lin H., and Z. Wu, 2011: Contribution of the autumn Tibetan Plateau snow cover to seasonal prediction of North American winter temperature. J. Climate, 24, 2801 2813. 390 391 392 Lorenz, E. N., 1975: Climate Predictability. Pp.133-136 in The Physical Basis of Climate abd Climate Modelling, GAPR Publiaction Series No. 16. Geneva, World Meteorological Organization. 21

393 394 395 Mason, S. J., L. Goddard, N. E. Graham, E. Yulaeva, L. Sun, abd P. A. Arkin, 1999: The IRI seasonal climate prediction system and the 1997/98 El Niño event. Bull. Amer. Meteor. Soc., 80, 1853 1873. 396 Matsueda, S., and Y. Takaya, 2012: Forecast skill of MJO with the JMAs One-month 397 398 Ensemble Prediction System. CAS/JSC WGNE Res. Activ. Atmos. Oceanic Model, 42, 6.11-12. 399 400 401 Moron, V., A. W. Robertson, and M. N. Ward, 2006: Seasonal predictability and spatial coherence of rainfall characteristics in the tropical setting of Senegal. Mon. Wea. Rev., 134, 3248 3262, doi: http://dx.doi.org/10.1175/mwr3252.1 402 403 Murphy, A. H., 1988: Skill scores based on the mean square error and their relationships to the correlation coefficient. Mon. Wea. Rev., 116, 2417 2424. 404 405 406 National Academy of Sciences 2010 Assessment of Intraseasonal to Interannual Climate Prediction and Prdictability, National Research Council, Washington DC, ISBN-10: 0-309-15183-X, 193 pp. 407 408 Palmer, T. N. and D. L. T. Anderson, 1994: The prospects for seasonal forecasting. Quart. J. Roy. Meteor. Soc., 120, 755 793. 409 410 411 Robertson, A. W., U. Lall, S. E. Zebiak, and L. Goddard, 2004: Improved combination of multiple atmospheric GCM ensembles for seasonal prediction. Mon. Wea. Rev., 132, 2732 2744. 412 413 Saha, S., and Coauthors, 2014: The NCEP Climate Forecast System Version 2. J. Climate, 27, 2185 2208, doi: http://dx.doi.org/10.1175/jcli-d-12-00823.1 22

414 415 Saha, S., S. Moorthi, H. Pan, and coauthors, 2010: The ncep climate forecast system reanal- ysis. Bull. Amer. Meteor. Soc., 91, doi:10.1175/2010bams3001.1 416 417 Scaife, A. A, and J. R. Knight, 2008: Ensemble simulations of the cold European winter of 2005/6. Quart. J. Roy. Meteor. Soc., 134, 1647 1659. 418 Vitart, F., 2004: Monthly forecasting at ECMWF. Mon. Wea. Rev., 32, 2761 2779. 419 420 Vitart, F., 2014: Evolution of ECMWF sub-seasonal forecast skill scores. Quart. J. Roy. Meteor. Soc., doi: 10.1002/qj.2256 421 422 423 424 Vitart, F., Buizza, M. A. Balmaseda, G. Balsamo, J-R Bidlot, A. Bonet, M. Fuentes, A. Hofstadler, F. Moleni, and T. N. Palmer, 2008: The new VarEPS-monthly forecasting system: A first step towards seamless prediction. Quart. J. Roy. Meteor. Soc., 134, 1789 1799. 425 426 Waliser, D. E., K. M. Lau, W. Stern, and C. Jones, 2003: Potential predictability of the Madden-Julian oscillation. Bull. Amer. Meteor. Soc., 84, 33 50. 427 428 Wheeler, M. C., and H. Hendon, 2004: An all-season real-time multivariate MJO index: Development of an index for monitoring and prediction. Mon. Wea. Rev., 132, 1917 1932. 429 430 431 World Meteorological Organization, 2013: Sub-seasonal to Seasonal Prediction Research Implementation Plan, December 2013, 71 pp (available online at http://www.wmo.int/pages/prog/arep/wwrp/new/s2s project main page.html). 432 433 Xie, P. and P. A. Arkin, 1996: Analyses of global monthly precipitation using gauge obser- vations, satellite estimates, and numerical model predictions. J. Climate, 9, 840 858. 23

434 435 1997: A 17-year monthly analysis based on gauge observations, satellite estimates and numerical model outputs. Bull. Amer. Meteor. Soc., 78, 2539 2558. 436 437 438 Yoo, J. H, A. W. Robertson, and I. S. Kang, 2010: Analysis of intraseasonal and interannual variability of the Asian summer monsoon using a hidden Markov model. J. Climate, 23, 5498 5516, doi: http://dx.doi.org/10.1175/2010jcli3473.1 439 440 Zhang, C., 2013: MaddenJulian Oscillation: Bridging weather and climate. Bull. Amer. Meteor. Soc., 94, 1849 1870, doi: http://dx.doi.org/10.1175/bams-d-12-00026.1 24

441 List of Figures 442 443 1 Anomaly correlation coefficient between JMA model precipitation hindcast and CMAP rainfall data for weeks 1 4 during the period 1992 2008..... 28 444 2 Same as Figure 1 but for the CFSv2 hindcast................. 29 445 3 Same as Figure 1 but for the ECMWF hindcast................ 30 446 447 448 449 4 Aggregate anomaly correlation coefficient between model precipitation hindcast and CMAP rainfall data for weeks 1 4, represented by percentage of ACC over the tropical land (20 S 20 N) exceeding discrete thresholds. The ACC values were calculated locally and then averaged spatially........... 31 450 5 Same as Figure 4 but for the Southeast Asia (90 130 E, 10 S 15 N)..... 32 451 452 6 Anomaly correlation between CMAP precipitation and Nino3.4 index for June-July-August, 1992 2008, with a 95% significance mask......... 33 453 454 455 7 Anomaly correlation between ECMWF lead-3 precipitation hindcasts (over 18 start dates) and CAMP observation during (a) 5 ENSO years: 1997 2000, 2002; and (b) 5 Neutral years: 1992, 2001, 2003, 2005, 2008.......... 34 456 457 458 8 Anomaly correlation between CMAP pentad precipitation and 5-day mean Real-time Multi-variate MJO (RMM) components, RMM1 and RMM2, dur- ing June August, 1992 2008, with a 95% significance mask.......... 35 25

459 460 461 9 Scatter plot of CMAP weekly precipitation anomaly versus ECMWF hindcast (lead-3) valid for 18 weeks (6/4 6/10 through 10/1 10/7), 1992 2008, over the Borneo Island. The dotted line is a 45 line................... 36 462 463 10 Real-time Multi-variate MJO (RMM) phase space (Wheeler and Hendon, 2004) for June 01 through September 30, 2002, a typical year of El Niño... 37 464 465 11 Composite Borneo-averaged pentad CMAP precipitation anomaly in relation to pentad dominant MJO phase during June-August, 1992-2008....... 38 466 467 468 469 470 471 472 12 Time series of precipitation anomalies (mm/day) over a portion of Borneo Island, from the CMAP data and ECMWF hindcast, valid for weeks 2 and 3 during June September 2002 (typical El Niño). The upper-case letters along with single digit at the bottom of each panel denote the dominant MJO phase sector and phase numer (A, I, M, P corresponding to MJO phases 8 1, 2 3, 4 5, and 6 7, respectively) with amplitude greater than 1.0 (Wheeler and Hendon 2004)................................... 39 473 13 Same as Figure 12 but for 1999 (La Niña)................... 40 474 14 Same as Figure 12 but for 2001 (neutral ENSO)................ 41 475 476 15 Mean square skill score (MSSS) between the JMA model precipitation hind- cast and CMAP rainfall data over weeks 1 4.................. 42 26

477 478 16 Mean square skill score (MSSS) between CFSv2 precipitation hindcast and CMAP rainfall data over weeks 1-4........................ 43 479 480 17 Mean square skill score (MSSS) between ECMWF precipitation hindcast and CMAP rainfall data over weeks 1-4........................ 44 481 482 18 Difference between ACC and MSSS of precipitation hindcast versus CMAP rainfall data valid for Week-3: (a) JMA, (b) CFSv2, and (c) ECMWF.... 45 27

Figure 1: Anomaly correlation coefficient between JMA model precipitation hindcast and CMAP rainfall data for weeks 1 4 during the period 1992 2008. 28

Figure 2: Same as Figure 1 but for the CFSv2 hindcast. 29

Figure 3: Same as Figure 1 but for the ECMWF hindcast. 30

% Grid pts Exceeding "R" % Grid pts Exceeding "R" 100 90 80 70 60 50 40 30 20 Tropics: Week-1 JMA CFSv2 ECMWF 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 70 60 50 40 30 20 10 Tropics: Week-3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Anomaly Correlation % Grid pts Exceeding "R" % Grid pts Exceeding "R" 90 80 70 60 50 40 30 20 10 Tropics: Week-2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 70 60 50 40 30 20 10 Tropics: Week-4 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Anomaly Correlation Figure 4: Aggregate anomaly correlation coefficient between model precipitation hindcast and CMAP rainfall data for weeks 1 4, represented by percentage of ACC over the tropical land (20 S 20 N) exceeding discrete thresholds. The ACC values were calculated locally and then averaged spatially. 31

% Grid pts Exceeding "R" % Grid pts Exceeding "R" 100 90 80 70 60 50 40 30 20 SE Asia: Week-1 JMA CFSv2 ECMWF 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100 90 80 70 60 50 40 30 20 SE Asia: Week-3 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Anomaly Correlation % Grid pts Exceeding "R" % Grid pts Exceeding "R" 100 90 80 70 60 50 40 30 20 SE Asia: Week-2 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100 90 80 70 60 50 40 30 20 SE Asia: Week-4 10 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Anomaly Correlation Figure 5: Same as Figure 4 but for the Southeast Asia (90 130 E, 10 S 15 N). 32

Figure 6: Anomaly correlation between CMAP precipitation and Nino3.4 index for June- July-August, 1992 2008, with a 95% significance mask. 33

Figure 7: Anomaly correlation between ECMWF lead-3 precipitation hindcasts (over 18 start dates) and CAMP observation during (a) 5 ENSO years: 1997 2000, 2002; and (b) 5 Neutral years: 1992, 2001, 2003, 2005, 2008. 34

Figure 8: Anomaly correlation between CMAP pentad precipitation and 5-day mean Realtime Multi-variate MJO (RMM) components, RMM1 and RMM2, during June August, 1992 2008, with a 95% significance mask. 35

PCP Anomaly: ECMWF vs CMAP (Lead-3) 10 ECMWF precip anomaly (mm/day) 5 0-5 -5 0 5 10 CMAP precip anomaly (mm/day) Figure 9: Scatter plot of CMAP weekly precipitation anomaly versus ECMWF hindcast (lead-3) valid for 18 weeks (6/4 6/10 through 10/1 10/7), 1992 2008, over the Borneo Island. The dotted line is a 45 line. 36

4 (RMM1, RMM2) phase space for 1 Jun to 30 Sep 2002 7 Western Pacific 6 RMM2 2 0-2 8 West Hem. and Africa 1 13 12 9 10 11 8 9 12 11 10 6 5 7 4 28 27 13 3 2 29 301 25 15 14 24 15 17 16 14 16 18 20 19 22 23 17 2425 13 232728 21 18 17 22 18 26 19 29 30 19 21 20 23 15 16 31 14 21221234 2425 5 13 20 20 26 1 10 9 1112 19 2 27 6 12 11 8 21 28 10 3 22 7 29 78 9 17 18 4 30311 28 29 30 27 2524 23 3 45 6 26 2 16 5 6 7 8 14 15 26 5 Maritime Continent 4 2 Indian Ocean 3-4 -4-2 0 2 4 Labeled dots for each day. RMM1 Blue for Jun, orange for Jul, green for Aug, and red for Sep. Figure 10: Real-time Multi-variate MJO (RMM) phase space (Wheeler and Hendon, 2004) for June 01 through September 30, 2002, a typical year of El Niño. 37

3 CMAP precipitation anomaly vs MJO phase 2 Precipitation anomaly (mm/day) 1 0-1 -2-3 1 2 3 4 5 6 7 8 MJO phase Figure 11: Composite Borneo-averaged pentad CMAP precipitation anomaly in relation to pentad dominant MJO phase during June-August, 1992-2008. 38

10 2002 ECMWF vs CMAP (Lead-2) CMAP ECMWF 5 PCP anomaly 0-5 A8 I2 I3 M5 P6 P7 P7 A8 M4 P6 A8 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Weeks since May 28 10 2002 ECMWF vs CMAP (Lead-3) 5 PCP anomaly 0-5 I2 I3 M5 P6 P7 P7 A8 M4 P6 A8 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Weeks since May 28 Figure 12: Time series of precipitation anomalies (mm/day) over a portion of Borneo Island, from the CMAP data and ECMWF hindcast, valid for weeks 2 and 3 during June September 2002 (typical El Niño). The upper-case letters along with single digit at the bottom of each panel denote the dominant MJO phase sector and phase numer (A, I, M, P corresponding to MJO phases 8 1, 2 3, 4 5, and 6 7, respectively) with amplitude greater than 1.0 (Wheeler and Hendon 2004). 39

10 1999 ECMWF vs CMAP (Lead-2) CMAP ECMWF 5 PCP anomaly 0-5 P7 A8 A1 A1 M5 I2 A1 A1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Weeks since May 28 10 1999 ECMWF vs CMAP (Lead-3) 5 PCP anomaly 0-5 A8 A1 A1 M5 I2 A1 A1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Weeks since May 28 Figure 13: Same as Figure 12 but for 1999 (La Niña). 40

10 2001 ECMWF vs CMAP (Lead-2) CMAP ECMWF 5 PCP anomaly 0-5 A1 I2 P6 P7 A8 A1 I2 M4 P6 P7 I2 I2 I2 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Weeks since May 28 10 2001 ECMWF vs CMAP (Lead-3) 5 PCP anomaly 0-5 I2 P6 P7 A8 A1 I2 M4 P6 P7 I2 I2 I2 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Weeks since May 28 Figure 14: Same as Figure 12 but for 2001 (neutral ENSO). 41

Figure 15: Mean square skill score (MSSS) between the JMA model precipitation hindcast and CMAP rainfall data over weeks 1 4. 42

Figure 16: Mean square skill score (MSSS) between CFSv2 precipitation hindcast and CMAP rainfall data over weeks 1-4. 43

Figure 17: Mean square skill score (MSSS) between ECMWF precipitation hindcast and CMAP rainfall data over weeks 1-4. 44

Figure 18: Difference between ACC and MSSS of precipitation hindcast versus CMAP rainfall data valid for Week-3: (a) JMA, (b) CFSv2, and (c) ECMWF. 45