Evaluation of ocean forecast performance for Royal Australian Navy exercise areas in the Tasman Sea

Transcription

1 Journal of Operational Oceanography ISSN: X (Print) (Online) Journal homepage: Evaluation of ocean forecast performance for Royal Australian Navy exercise areas in the Tasman Sea Robert H. Woodham, Oscar Alves, Gary B. Brassington, Robin Robertson & Andrew Kiss To cite this article: Robert H. Woodham, Oscar Alves, Gary B. Brassington, Robin Robertson & Andrew Kiss (2015) Evaluation of ocean forecast performance for Royal Australian Navy exercise areas in the Tasman Sea, Journal of Operational Oceanography, 8:2, , DOI: / X To link to this article: Published online: 10 Nov Submit your article to this journal Article views: 216 View Crossmark data Citing articles: 1 View citing articles Full Terms & Conditions of access and use can be found at

2 Journal of Operational Oceanography, 2015 Vol. 8, No. 2, , Evaluation of ocean forecast performance for Royal Australian Navy exercise areas in the Tasman Sea Robert H. Woodham a *, Oscar Alves b, Gary B. Brassington b, Robin Robertson c and Andrew Kiss c a Royal Australian Navy, Strategic Command, Canberra, Australia; b Centre for Australian Weather and Climate Research, Melbourne, Australia; c School of Physical, Environmental and Mathematical Sciences, University of New South Wales Canberra, Australia The variability of sea surface height (SSH) and sea surface temperature (SST) in eddy-resolving re-analyses and forecasts of the East Australian Current (EAC) separation and eddy-shedding region is investigated. The baseline skill attainable by a naïve forecasting method is established using root mean square errors and correlation coefficients. The forecasting system is assessed against this baseline and found to be skilful throughout the forecast period for SSH, and for four days (of six) for SST. A skilful forecast is much more likely than an unskilful one. However, a failure mode of the model, associated with eddy merging, is identified. Introduction The equatorward limb of the South Pacific subtropical gyre propagates to the west in low latitudes as the South Equatorial Current (Marchesiello & Middleton 2000). On reaching the Australian coast between 15 deg S and 22 deg S, it splits in two, with the southern arm feeding warm equatorial water into the Coral Sea (Hill et al. 2010). It is this inflow which provides the East Australian Current (EAC) with its warm source water. The EAC may be understood in terms of four stages (shown schematically, along with the study domain and geographic features, in Figure 1; Ridgway & Dunn 2003): formation in the Coral Sea, between 15 deg S and the southern extremity of the Great Barrier Reef at 24 deg S; intensification between the southern end of the formation stage and the separation point; separation, which occurs over a 150 km range in the vicinity of 32 deg S and involves most of the EAC transport heading offshore; and a declining stage containing eddies which continues southwards along the coast as far as Tasmania. To the south of the EAC separation, warm- and cold-core eddies dominate the mesoscale dynamics, the former having been shed from the main current (Ridgway & Dunn 2003). The Royal Australian Navy (RAN) has designated maritime exercise areas known as the East Australian Exercise Areas (EAXA), which extend from the vicinity of the EAC separation point to the eddy-rich declining stage to its south (Figure 1). These exercise areas are routinely used for antisubmarine warfare exercises in which the principal sensors employed are acoustic ones, as well as other naval activities which benefit from ocean forecasts (Jacobs et al. 2009; Woodham 2011). Because acoustic energy is refracted by sound speed gradients, and because sound speed in the ocean is a function of temperature, salinity and pressure, the performance of acoustic sensors can be qualitatively inferred, and quantitatively modelled, from the temperature and salinity fields. This paper evaluates the performance of an ocean forecasting system, namely the BLUElink system, over the EAXA. Sea surface height (SSH) is the principal variable investigated in this paper. It is taken to be the primary indicator of the mesoscale dynamics because mesoscale features are readily discernible from the SSH field (Fu et al. 2010). Since the majority of the ocean s kinetic energy is associated with the mesoscale dynamics (Fu et al. 2010), a forecasting system which shows skill in predicting the SSH field is likely to be generally skilful. SSH is an important variable, because the SSH field contains information about the geostrophic flow near the surface, resulting from the horizontal sea surface gradient, and also about the vertically-integrated density anomalies, which is the main determinant of the SSH at any particular point. If temperature and salinity (and hence density) data is available for the water column, from in situ observations for example, this information can be combined with the SSH field to deduce the pressure field at any level. This, in turn, enables the velocity shear and absolute geostrophic velocities at all depths to be deduced. Because the SSH field contains information about the interior of the ocean as well as its surface, and because the steric height of the ocean s surface is directly and frequently observed, SSH is a variable which provides significant insights into the *Corresponding author. robert.woodham@defence.gov.au 2015 Institute of Marine Engineering, Science & Technology

3 148 R.H. Woodham et al. Figure 1. Schematic diagram of the East Australian Current (EAC), the location of the study domain (black square) and surrounding geographic features. Note: Bathymetry is the General Bathymetric Chart of the Oceans (GEBCO) dataset. Roman numerals indicate the (I) Formation, (II) Intensification, (III) Separation and (IV) Declining stages, which are fully described in the text (see the Introduction). physical state of the ocean. It is therefore a particularly useful variable to investigate when assessing the performance of forecasting systems. To a lesser extent, sea surface temperature (SST) is included in this investigation because of its association with the ocean s near-surface acoustic properties, which are of great interest to the navy due to the effect on ships hull-mounted sonars (Woodham 2011). While the SSH contains information about the ocean surface and the vertically-integrated water column beneath, the SST contains information about the upper layers of the ocean, through its covariance with salinity and currents. Like the SSH, the SST is directly and frequently observed. Where particular insights can be obtained from the subsurface temperature structure, this is also investigated. The BLUElink ocean forecasting system (Oke et al. 2005, 2008; Brassington et al. 2007; Schiller et al. 2008) provides eddy-resolving re-analyses and forecasts of the ocean around Australia as part of the Global Ocean Data Assimilation Experiment (GODAE; Bell et al. 2009). The variability in the EAC is particularly difficult for this system to reproduce (Oke et al. 2008), and this problematic area is the focus of the present study. The paper investigates the variability of the SSH and SST in the EAXA, and assesses the skill of the BLUElink ocean forecasting system in predicting these variables. Two measures of skill are employed, namely root mean square (RMS) error, which can be regarded as a direct measure of accuracy, and correlation coefficient (CC), a complementary measure of skill that rewards a forecast when features are in the right place, even if their amplitudes are erroneous or if there is bias. Care is taken in the BLUElink system to ensure that observed SSTs can be compared to the surface temperature values in the BLUElink model, at the topmost grid point, which represents the temperature of the top 10 m of the ocean (Andreu-Burillo et al. 2010). The term SST, as used in the current investigation, refers to the temperature in this top 10 m layer, and does not include shallower, transient structures, such as might be associated with diurnal variations. This paper has three objectives: firstly, to estimate the natural variability of the SSH and SST in the study area by comparing the skill of forecasts based on the persistence of BLUElink re-analyses with those based on the

4 Journal of Operational Oceanography 149 climatology of these re-analyses, and hence establish a baseline forecast skill which these naïve methods are able to achieve; secondly, to determine the skill of the BLUElink ocean forecasting system, relative to this baseline, in predicting the SSH and SST in the study area; and thirdly, to determine the variation with space and time of model skill, particularly with a view to identifying a-priori indicators of poor skill. In the following sections, the study domain and its physical oceanography are first described, followed by a description of the data used and the methods employed in its analysis. Results are then presented from an investigation of the errors associated with naïve methods of forecasting the SSH and SST, namely forecasts based on monthly climatologies, and on persistence. The errors associated with forecasts from the BLUElink ocean forecasting system are then presented, along with their spatial and temporal variation. significant SSH changes over the course of an eddy-shedding event, with an observed RMS sea level variability of around m in the eddy-shedding region (Mata et al. 2006). In addition, the EAC poleward transport displays seasonal variability, being at its strongest in summer and weakest in winter (Ridgway & Godfrey 1997). Complex interactions between eddies have been observed to occur in the region. One of the first documented examples of this occurred in 1983, when two warmcore eddies coalesced over a period of around 20 days, resulting in the merging of their vertical thermal signatures, such that the vertical stratification of the original eddies could be identified in the coalesced entity (Cresswell 1982). Stratified vortex mergers have also been described more recently by Brassington et al. (2011). Eddy clustering and merging has been demonstrated to provide a mechanism for the inverse cascade of energy, from relatively small-scale baroclinic modes to larger scale barotropic modes (Vallis 2003). Domain, data and methods Domain and physical oceanography The domain chosen for the investigation is along the east coast of Australia, deg E and from 32 deg S to 37 deg S (Figure 1). This area has been chosen to encompass the EAXA, where the performance of a forecasting system is of practical relevance to the RAN. The EAC separation region is at the extreme north of the domain, and the domain itself is dominated by the eddy-shedding region. Although the peak speed of the EAC can exceed 2 m/s (Roughan & Middleton 2002), the EAC is the weakest of the subtropical western boundary currents in terms of mean transport. The EAC transport is also highly variable, with intra-annual variability of comparable amplitude to the mean; for example, Mata et al. (2000) measured a mean transport of 22.1 Sv between the coast and deg E at 30 deg S but with an RMS variability (associated with excursions of the separation point either side of this latitude) of 30 Sv. The variability of the EAC is particularly marked in the separation region, where there is a repeated cycle of southward extension of the EAC to about 35 deg S, followed by pinch-off of large anticyclonic eddies and a northward retreat of the current roughly every 100 days (Mata et al. 2006). This intense variability makes forecasting particularly difficult in the EAC separation region (Oke et al. 2008). Mata et al. (2006) present a model of the EAC separation in which a train of anomalies, growing through barotropic and baroclinic instabilities, propagate polewards. The leading warm-core eddy forms the shed eddy, and a cyclonic anomaly develops between it and the next anticyclonic anomaly, eventually causing the shed eddy to pinch off from the main core of the EAC. The pattern of SSH reflects the changing structure of the EAC. This leads to Ocean data SSH and SST data are obtained from the BLUElink project (Oke et al. 2005, 2008; Brassington et al. 2007; Schiller et al. 2008). BLUElink has delivered eddy-resolving sixday forecasts of the ocean state around Australia since August The system has several components, including an ocean general circulation model known as the Ocean Forecasting Australia Model (OFAM) which is nudged towards analysed observations generated by the BLUElink Ocean Data Assimilation System (BODAS; Brassington et al. 2007). The current study uses OFAM 1.0., which is based on the Geophysical Fluid Dynamics Laboratory s Modular Ocean Model v4.0d (MOM4; Griffies et al. 2004), with some local enhancements to the mixed layer parameterization and insolation. The model domain is global, with 47 vertical levels and eddy-resolving horizontal resolution (0.1 deg) in the Australian region (90 deg E to 180 deg E and 16 deg N to 75 deg S). BODAS (Oke et al. 2005, 2008) combines OFAM output with ocean observations (including sea level from tide gauges and altimetry, SST from satellite radiometry, and vertical temperature and salinity structure from in-situ sources such as moorings and Argo floats), generating analyses on a grid with half the horizontal resolution of OFAM (i.e., 0.2 deg near Australia). BODAS creates analyses via an ensemble optimal interpolation method based on a fixed set of multivariate covariances obtained from a run of OFAM without data assimilation. OFAM model runs are periodically nudged towards analysis (interpolated onto the finer OFAM grid) over 24 hours, which typically drives OFAM about 70 to 90% of the way to the analysis (Oke et al. 2008). Reanalysed data has been routinely generated since 2007 as part of the operational forecasting system, and has also been generated from older observations in a

5 150 R.H. Woodham et al. series of BLUElink Re-Analysis (BRAN) experiments. Oke et al. (2005, 2008) cover the technical details and performance of BRAN 1.0 and 1.5, respectively. The current study uses data from the BRAN 2.1 reanalysis (Schiller et al. 2008). This is an OFAM run, forced by atmospheric momentum, heat and freshwater fluxes, using six-hourly European Centre for Medium-Range Weather Forecasts (ECMWF) ERA-40 re-analyses until 2002, and operational ECMWF forecasts thereafter (Schiller et al. 2008). It is nudged towards BODAS analysis for up to 24 hours every seven days, with BODAS using a symmetrical data assimilation window extending five days either side of the analysis day. The first part of this study addresses variability in the EAC using data from a ten-year subset of the BRAN 2.1 reanalysis running from 1 January 1993 to 31 December 2002 (the more recent BRAN 2.2 data was not used because this time series is too short). The second part of the study focuses on the real-time operational forecast system. In contrast to BRAN, the real-time system initializes OFAM in a two-step process, starting with nudging to a symmetric BODAS analysis at eight or nine days behind real time, followed by nudging to an asymmetric analysis (using observations from five days before to one day after the analysis time) closer to the forecast start time (T+0). The symmetric analysis alternates between eight and nine days before T+0. Although the second nudging step has no new altimetry data available to it (but does use new SST data), the altimetry observations are weighted differently in the first and second nudging steps, according to their ages with respect to the analysis time. The ocean forecast evolving from the initial state is forced by atmospheric fluxes obtained from an operational numerical weather prediction (NWP) system, namely the Bureau of Meteorology s Australian Community Climate and Earth System Simulator Global (ACCESS-G), Australian Parallel Suite 0 (APS0) implementation (Bureau of Meteorology 2010). Although the BODAS-nudged reanalysis fields are not available at T+0 (due to the use of a symmetric window), they are available a posteriori for validation purposes. They are used in the current paper to evaluate the forecasts. This approach is justified, because only a small proportion of the observational data used to generate a particular reanalysis is available to the ocean forecasting system for the same validity time, since the reanalysis system runs eight to nine days behind real time. As implemented for the data sets used in this paper, forecast runs took place twice per week and generated forecasts of daily means of temperature, salinity, SSH and currents every 24 hours out to six days (T+144 hours). For the current study, forecasts from the period 18 August 2008 to 20 August 2009 were used, giving a total of 105 forecasts. This period differs from that used in the BRAN study; it was chosen because it is the only time for which reanalysed ocean data and forecasts are both available. Since the time period spans only one calendar year, seasonal variability is not expected to be adequately captured in the investigation. It is further noted that, since the system generates daily means, shorter timescales such as diurnal variations are not captured in this study. The reanalysis and forecast systems use identical model grids, and have 1900 surface wet grid points within the domain of the current study. For both the BRAN and real-time studies, the SST is taken as the temperature at a depth of 5 m, in the middle of the 10 m top layer of the model. The reanalysed ocean data from BRAN 2.1 is used as a reference for validation, as this is the most comprehensive and dynamically consistent estimate available of the past state of the ocean. Oke et al. (2008) give a detailed characterization of the errors in the BRAN 1.5 reanalysis relative to observations, which is also applicable to the similar (but longer) BRAN 2.1 dataset used here. Errors arise in the observations (coverage and instrumental errors), their analysis by BODAS, and in the initialization of OFAM by nudging; Oke et al. (2008) showed that the last of these is the main source of error in the re-analyses. The real-time system has an additional source of error due to the unavoidable eight- or nine-day delay between the reanalysis based on symmetric BODAS assimilation and the start of the forecast (mitigated somewhat by the second nudging step), so the initialization errors of the forecast run will exceed the errors of the BODAS-nudged reanalysis on the same day. Oke et al. (2008) did not assess the real-time forecasting system; the current study is complementary to theirs in that it investigates errors between analyses and forecasts, while their study investigates errors between observations and analyses. Evaluation methods A persistence forecast is one in which it is assumed there will be no change in the state of the ocean with time. A climatological forecast is one in which the state of the ocean at a future time is estimated from historical data, such as a monthly mean over an earlier decade, as used in the current study. On average, a persistence forecast is known to perform better than a climatological forecast during the early part of the forecast period (in a mean square sense), but a climatological forecast performs better than a persistence forecast later in the forecast period (Kalnay 2003). A skilful, but imperfect, deterministic forecast tracks dynamical developments and the response of the real ocean to forcing, which means that its error should grow more slowly than that of a forecast based on persistence. Its error does not grow without bound, however, since the range of values that the model can adopt is limited, if it has no gross unbounded drift. If the model climatology and variance are realistic, the expected forecast mean squared error (MSE, defined below) asymptotes with time

6 Journal of Operational Oceanography 151 to twice the climatological MSE (Leith 1974; Kalnay 2003), because the forecast suffers from the double error of not having anomalies where they should be and also having anomalies where they should not be. In the atmospheric sciences, persistence and climatological forecasts are often regarded as naïve methods, which establish a minimum baseline of skill. A more sophisticated forecasting system must exceed this baseline if it is to be considered skilful (Jolliffe & Stephenson 2003). In this paper, errors are measured using RMS differences and CCs between two sets of gridded, time-dependent variables, f (forecast data) and x (verifying data): (Murphy & Epstein 1989) RMSE( f, x) = MSE = k( f i x i ) 2 l, by 90 deg using the Hilbert transform. Each of the real and imaginary parts are standardized by removing the temporal mean and dividing the amplitude at each spatial location by its standard deviation. Factorization of this matrix into complex eigenvectors enables moving patterns of variability to be identified. Each CEOF mode can be illustrated by projecting the original data series onto that mode, to produce a time-varying representation of the variability which it represents. CEOFs can also be illustrated in terms of their spatial phase angle, spatial amplitude, temporal phase angle and temporal amplitude. Emery and Thomson (1998) note that CEOF and EOF modes represent statistical variability, and caution against relating them directly to physical processes without good reason. CC = kfxl kf lkxl s f s x, where the subscript i indicates the ith grid point, s denotes the spatial standard deviation, and angle brackets denote the spatial mean over all grid points. The RMS measure can be misleading in that it favours forecasts based on climatology. It also contains a bias component, where bias is defined as the difference between the mean forecast and the mean observation (Potts 2003). RMS error (RMSE) measures therefore penalize a deterministic forecast that predicts anomalies correctly, but has bias in the background field. The CC is higher for a forecast in which the anomaly is spatially correct, but of the wrong amplitude, and is not affected by bias. Conversely, this ignoring of bias errors means that CCs may overestimate the performance of forecasts. Martin (2010) characterizes RMSE and CC as measuring accuracy and pattern respectively. Filtering SSH data was filtered to isolate the 100-day period associated with eddy shedding, using a fifth-order Butterworth bandpass filter, with cutoffs ( 3 db) corresponding to periods of 66 and 133 days. Complex empirical orthogonal function (CEOF) analysis Complex empirical orthogonal functions (CEOFs) were computed following the Hilbert transform method of Venegas (2001). In this method, the time-varying, gridded data is first filtered to isolate the frequencies of interest. It is then arranged as a complex matrix, where the real part is the original data and the imaginary part is a version of the same data which has been phase shifted Results and discussion Establishing the baseline Reanalysis data from BRAN 2.1 was used to investigate the natural persistence of the SSH and SST in the study domain (Figure 2). The RMSE of persistence forecasts of the SSH grows to a maximum value, and the CC falls to a minimum value, in a timescale of around days [Figures 2(a), (c)]. There is a regular modulation in the CC for the SSH on a 100-day timescale over the first nine months [Figure 2(c)]; this matches the periodicity of anticyclonic eddy shedding (Mata et al. 2006) and will be further investigated in due course. The minimum CC for the SSH is between 0.2 and 0.4 [Figure 2(c)], with a mean over the one- to two-year period of This is because the mean SSH, calculated over the entire ten-year period of the reanalysis dataset, is higher in the north of the domain, and lower in the south [Figure 3(a)]. This persistent pattern gives rise to a nonzero CC when two days are picked at random from the dataset and compared. The expected RMSE of a climatological forecast is simply the mean anomaly, calculated over the averaging period (monthly in this case), and is independent of the forecast lead time. The mean monthly anomaly for the SSH is 0.14 m; this has been marked as a horizontal line on Figure 2(a). It shows that the crossover point between the expected errors of a persistence and climatological forecast is at day 19. Usui et al. (2006) report similar SSH RMSE characteristics for the Kuroshio south of Japan: the expected RMSE of a persistence forecast of SSH asymptotes to around 20 cm after 80 days, while the expected RMSE of a climatological forecast is 13.5 cm and the crossover point between the two is at day 25. This similarity is interesting, given the differing characteristics of the Kuroshio and EAC in terms of volume transport and variability. The expected CC of a climatological forecast is the mean CC between each day s verifying data and the

7 152 R.H. Woodham et al. Figure 2. Expected (a), (b) RMSE and (c), (d) CC for persistence forecasts (blue lines) and climatological forecasts (black lines) of (a), (c) SSH and (b), (d) SST for a period of up to two years. Note: Data is taken from the BRAN reanalysis and compares the SSH and SST on a given day to the subsequent 730 days. The red lines in (b) and (d) are as the blue lines, but with the mean monthly SST removed. The red line in (c) is as the blue line, but with mean SSH removed (i.e., equivalent to the anomaly CC).

8 Journal of Operational Oceanography 153 Figure 3. (a), (c) Mean and (b), (d) Standard Deviation of (a), (b) SSH and (c), (d) SST for BRAN data, monthly mean. For the SSH it is 0.59 [marked as a horizontal line on Figure 2(c)]. The crossover between persistence and climatological SSH forecasts is at 20 days, similar to the RMSE crossover. The SSH variability resembles offshore eddy shedding [Figure 3(b)], and a spectral analysis of the grid point at the Figure 4. Spectrum taken from the grid point at the centre of the northern SSH standard deviation maximum [marked with a dot in Figure 3(b)]. centre of the northern SSH standard deviation maximum shows a clear peak at 100 days (Figure 4), which is a period associated with eddy shedding (Mata et al. 2006). The relationship between SSH variability and eddy shedding will be discussed shortly. Unlike the SSH data, the natural persistence of the SST is dominated by the annual heating-cooling cycle [Figure 2(b)]. This mode of variability has a range of around 2.4 C. When the annual cycle is removed, by basing the persistence forecast on the daily SST anomaly from the monthly mean, the error growth curve is similar to the curve for the SSH, albeit with a slightly earlier crossover point at 12 days. The scalloping on the plot of the RMSE of the SST anomaly [Figure 2(b)] is caused by the monthly averaging process; days at each end of the month are (on average) further from the mean than those in the middle of the month. The CC for the SST reaches a minimum value between 0.5 and 0.6 (the mean over the one- to two-year period is 0.54). This is greater than for the SSH, and reflects the greater persistent structure in the SST, associated with the warm waters of the EAC flowing south, with cooler water on the continental shelf. The expected CC of a persistence forecast of the SST anomaly falls to zero [Figure 2(d), red line] in a timescale of

9 154 R.H. Woodham et al. around days. Again, there is a 100-day modulation in the CC over the first few months. This is a particularly interesting result, since its periodicity, and its similarity to the equivalent plot for SSH, indicate that the controlling dynamical process is associated with eddies rather than surface fluxes. The crossover between the expected CCs of climatological and persistence forecasts is at eight days, somewhat earlier than the RMSE crossover at 12 days. The expected CC of a climatological forecast of the SST is higher than for the SSH, at The relationship between persistence and climatological forecast errors, both for SSH and SST anomaly, is as predicted by theory: the MSE of persistence forecasts rises to twice the climatological MSE (RMSE rises to 2 times) over time (the RMSE of SSH persistence at long lead times is 0.20 m, the RMSE of climatology is 0.14 m; the RMSE of SST anomaly persistence is 1.35 C, the RMSE of climatology is 0.96 C). Persistence error growth was also computed for temperatures at a depth of 545 m (model level 30). This depth was chosen because it is below the seasonal thermocline, as judged by inspecting a range of vertical temperature profiles throughout the domain and at different times of year. At this depth, there is no discernible annual cycle, but the eddy structure is evident. When the persistence error growth at this depth is compared to the SST, SST anomaly and the SSH, with each variable scaled to its long-term value (i.e., the value to which the RMSE asymptotes at long forecast lead times), their rates of growth are very similar. This indicates that the SSH, sub-surface temperature and part of the SST variation is due to eddy dynamics, with the remaining portion of SST variability attributed to the annual cycle. It also indicates that there is no discernible contribution from the surface forcing, since the scaled error growth in the SST and temperature at depth is so similar. Wilkin and Zhang (2007) report similar results for a somewhat larger domain over the EAC region; using satellite observations and a five-year limited area ocean model study, they note that the variability, as represented by CEOFs, in the temperature at a depth of 250 m is almost identical to the SSH. Furthermore, they investigated correlations between the SST, the time rate of change of the SST and net air-sea heat fluxes, concluding that mesoscale variability is driven by the circulation, with negligible contribution from air-sea heat fluxes (Wilkin & Zhang 2007). Anticyclonic eddy shedding has been found to occur on a 100-day timescale (Mata et al. 2006). It has been noted above that the modulation of CC for the SSH seems to match this timescale, the spatial pattern of the standard deviation of the SSH resembles offshore eddy shedding, and the SSH spectrum at the standard deviation maximum has a strong peak at 100 days. A 100-day modulation is also evident in the RMSE of persistence forecasts of temperature at a depth of 545 m. A CEOF analysis was carried out on the SSH data in order to further investigate the significance of its variability on a 100-day timescale. This analysis showed that the first eight CEOF modes account for 90% of the variance in the filtered data, and that all of these modes are associated with the generation, movement and weakening of eddies. Wilkin and Zhang (2007) report a CEOF analysis of streamfunction and SST data over a larger domain in the EAC region (from the Australian coast to 162 deg E and from 26 deg S to 46 deg S). They estimated the streamfunction directly from the SSH, scaled by the local Coriolis parameter, thus assuming geostrophy. Their streamfunction CEOFs are therefore comparable to the SSH CEOFs presented here. The ten-year BRAN 2.1 data was filtered to isolate the 100-day frequency, using a Butterworth bandpass filter. The first CEOF, which accounts for 34.9% of the variance, describes eddies entering the domain in the north-east corner, from an easterly direction, and then moving down the coast towards the south-west (Figure 5). The eddies weaken considerably when they reach the south-west corner of the domain. An interesting feature of this mode is the leading peak in SSH along the continental shelf break, evident in the spatial phase diagram [Figure 5(a)], which travels south in advance of the eddy. This feature, which is not reported by Wilkin and Zhang (2007), suggests that the topography of the continental slope has a role in the eddy dynamics. The second CEOF (17% of the variance) contains SSH features moving perpendicularly to the coast, from an offshore to an onshore direction, and splitting into two centres as they do so (Figure 6). Figure 6 has been plotted by first identifying the largest amplitude from the temporal amplitude plot, then identifying the days which correspond to 0, 45, 90 and 135 deg from the temporal phase angle plot, then projecting the data onto the CEOFs on these four days to show the development of the SSH associated with the mode throughout the phase cycle. The days are: 3 November (0 deg), 15 November (45 deg), 27 November (90 deg) and 10 December 2000 (135 deg). These first two CEOF modes account for over half of the variance in the filtered data. The next six modes (not shown), which account for a further 40% of the variance, also display structures which are clearly associated with eddies. Wilkin and Zhang (2007) describe their first two streamfunction CEOF modes in detail. They have similar properties to the first two CEOFs reported here. They characterize the first mode (35% of the variance) as an eddy mode, with energy propagation south-westwards along the coast, and the second (23% of the variance) as a wave mode, with onshore, Rossby wave-like propagation. The baseline naïve forecasting method is therefore established, for SSH, as persistence to day 19, then climatology for longer lead times, for both RMSE and CC measures. SSH variability in the study domain is dominated

10 Journal of Operational Oceanography 155 Figure 5. (a) Spatial phase angle and (b) spatial amplitude of the first CEOF of SSH, filtered to extract the 100-day frequency associated with eddy shedding. Note: There is no particular significance to the thick contour in (a), which merely marks the transition from 2π radians back to zero. by mesoscale eddy dynamics, with an associated timescale of 100 days. For SST, the baseline is persistence of the anomaly from the monthly mean to day 12 (for RMSE) or persistence of the SST to day 8 (for CC), and then climatology. SST variability in the study domain is dominated by the annual heating-cooling cycle but, when this is removed, the remaining variability is dominated by eddy dynamics, with no discernible contribution from surface heat exchanges. The temperature at a depth of 545 m is also dominated by eddy dynamics. Figure 6. Second CEOF of SSH, plotted for the largest amplitude event on (phase 0) 3 November, (phase 45) 15 November, (phase 90) 27 November and (phase 135) 10 December 2000.

11 156 R.H. Woodham et al. Forecast model skill Having established that persistence forecasts provide the baseline of skill for the early part of the forecast period (19 days for SSH, 8 days for SST CC and 12 days for SST RMSE), we move on to assess the skill of BLUElink forecasts relative to this baseline. Because the BLUElink initialization process is spread over a number of days, however, the forecast start time becomes a matter of definition. The BLUElink model does not ingest a complete set of oceanic observations for eight or nine days prior to its nominal start time (depending on the forecast run; for ease of reading, from here on, a reference to nine days should be taken to mean eight or nine). In a sense, the BLUElink system produces fifteen-day forecasts, the first nine days of which have already happened by T+0. A partial set of oceanic observations has been used to update the model five days before T+0. Upper and lower bounds on the expected errors of persistence forecasts are therefore taken to be defined by the expected error of persistence forecasts starting nine days (upper bound) and five days (lower bound) prior to T+0 (Figure 7). Relative to these limits, the BLUElink SSH forecasts show skill for the entire forecast period (Figure 7). In the latter part of the forecast period, a better forecast is expected, using the CC measure, from climatology than from persistence. On the final two days of the forecast period, the BLUElink SST forecasts are less skilful than the benchmark. Spatial and temporal variation in model skill The RMSE of SSH forecasts grows offshore in a pattern which resembles eddy dynamics [Figure 8, compare to Figures 5(b) and 6]. Errors on the continental shelf are low (Figure 8); this is related to the propagation of coastally-trapped waves along the coastal waveguide, and is investigated in more depth by Woodham et al. (2013). The temporal variation in SSH forecast skill is characterized by a majority of very good forecasts, but with a small number of forecast failures, associated with very poor skill (Figure 9). These very poor forecasts are seen to occur in early to mid-february 2009, and mid-march Figure 7. (a), (c) RMSE and (b), (d) CC for (a), (b) SSH and (c), (d) SST benchmarks and forecasts.note: The blue lines show expected errors of persistence forecasts starting at the symmetrical analysis (zero at day 0), and the asymmetric update (zero at day 4); the black squares and lines show the BLUElink forecasts; the red vertical lines show the forecast start time (T+0); the black horizontal lines show the expected RMSE and CC of climatological forecasts [the expected climatological RMSE is off-scale in (a)].

12 Journal of Operational Oceanography 157 Figure 8. RMSE of (a) T+0 and (b) T+72 forecasts of SSH [Figure 9(a)]. The T+72 forecast (19 February 2009, marked on Figure 9) from the forecast run starting on 16 February 2009, for example, has a CC of This is much lower even than a forecast based on climatology, which has an expected CC of The probability of a skilful forecast is much higher than an unskilful one [Figure 9(b)], and the vast majority of forecasts exceed the skill achieved by the naïve methods of persistence and climatology (Figure 9). The forecast failure event during the 16 February 2009 forecast run is due to the squeezing of a warm-core eddy into two, between two cold-core eddies. The forecast model evolves the squeezed eddy to the east, and merges the two cold-core eddies, but the verifying model run evolves the squeezed eddy to the west, and keeps the cold-core eddies separate (Figure 10). By the end of the forecast run, the two SSH maps are quite different, with a warm-core eddy much further south in the forecast than in the verifying data. This event will be reported more fully in a forthcoming paper. Conclusions The BLUElink ocean forecasting system shows skill in an area of the Tasman Sea which is of relevance to naval operations. SSH and SST variabilities in the study domain are dominated by mesoscale eddy dynamics, overlaid in the case of SST by a strong annual cycle. There is a negligible contribution to SSH and SST variability from surface heat fluxes, noting that the SST as defined in this study refers to the temperature in the top 10 m layer. Variability associated with eddies is consistent with a previous study, using a larger domain (Wilkin & Zhang 2007), in which over half Figure 9. (a) CC of 72-hour forecasts of SSH for the period August 2008 August 2009 and (b) histogram of CCs of 72-hour forecasts of SSH. Note: The red line in each plot shows the upper bound on the expected CC achieved by persistence forecasts (the naïve benchmark).

13 158 R.H. Woodham et al. Figure 10. SSH analyses and BLUElink forecasts for the forecast run starting on 16 February Note: Contours and colour stretches are the same for each sub-plot.

14 Journal of Operational Oceanography 159 the variability on timescales associated with eddies is attributed to eddy and Rossby wave-like modes. The growth of RMS forecast errors in two naïve forecasting methods, namely persistence and climatology, agrees with theory, in that the expected MSE of a persistence forecast at long lead times asymptotes to twice the expected MSE of a climatological forecast. For SSH and the RMSE measure, the crossover time between the two methods occurs on day 19, with persistence forecasts giving lower RMSE prior to this time, and climatological forecasts after this time. For the CC measure, the crossover point is very similar, at day 20. The persistently higher SSH in the north of the domain gives rise to a long-term expected CC of around 0.3 for persistence forecasts. Because the BLUElink system is subject to periodic upgrades, the results presented in this paper should be interpreted with due regard to their dependence on the overall model system. Observational data streams vary with time, as new observing systems come online and old ones are decommissioned. The BLUElink data assimilation scheme has recently been improved and vertical resolution has increased, although the underlying forecasting model remains the same. Forecast runs are now more frequent than in the system which produced the data for the current study, taking place daily rather than twice weekly. Nonetheless, some of the fundamental characteristics of the BLUElink system, and the Tasman Sea variability which the model captures, are enduring. For example, the occasional large forecast error events, associated with eddy merging, which this paper reports, continue to be seen in the latest version of the operational BLUElink system. The most recent system offers a four-cycle forecast where the ensemble variance has been shown to reliably track large error growth (Brassington 2013). The characteristics of RMSE for persistence and climatological forecasts of SSH in the EAC are remarkably similar to those reported for the Kuroshio (Usui et al. 2006). These two western boundary currents are located at similar latitudes, but differ markedly in their volume transport, typical geometries and variability. Given that the current study uses different analysis methods, this may be coincidental, but warrants further investigation. SSH variability in the study domain is strongly associated with eddy shedding, taking place on a timescale around 100 days. This has been demonstrated using a CEOF analysis of SSH variability. In the absence of long-range forecasts, navy forecasters who require an estimate of the future SSH field should use monthly climatology at lead times in excess of 20 days, but should bear in mind that this product averages out the detail associated with eddies. In order to obtain a more realistic assessment of acoustic conditions, it may be preferable to pick a random day from the reanalysis dataset. Using the reanalysis data as a scenario database in this way, the ocean data on which acoustic assessments are based will at least contain realistic eddies, even if they are in the wrong place and of the wrong amplitude. The characteristics of SST variability differ significantly from that of SSH, due to the strong annual heating-cooling cycle and the persistent pattern of SST associated with the EAC flowing south along the coast. This latter effect means that the expected CCs of persistence and climatological forecasts are much higher than for SST (0.54 for persistence and 0.75 for climatology). The crossover point is located earlier than for SSH, at 12 days for RMSE and 8 days for CC. Future SSTs should therefore be estimated using the monthly climatology at lead times beyond 12 days in order to minimize RMSE, and beyond 8 days in order to maximize CC although this result may prove to be system-dependent. The SST is dominated by the annual heating-cooling cycle, and anomalies from the monthly mean quickly de-correlate with time. BLUElink forecasts of the SSH are expected to be skilful throughout the forecast period (to T+144), and should be used in preference to the naïve methods described above for forecasts within this lead time. For SST, the BLUElink forecasts show skill to 4 days (T+96), but do not exceed the climatological benchmark after this time. Forecasters should be aware that the crossover times on which the above advice is based are means of a relatively large number of forecasts, and should therefore take into account the performance of the forecast in the days leading up to current day. They should look for early indications that the forecast is going off track (which would indicate an earlier crossover), or that the situation is developing as forecast (indicating a later crossover). The investigation of the OFAM 1.0 system indicates that BLUElink forecasts of SSH are susceptible to occasional large forecast error events. These failures are associated with complex eddy shedding or splitting. In synoptic situations of this character, forecasters are advised to exercise caution, and to make maximum use of all available observations in order to confirm that the forecast is developing realistically. Where vertical temperature profiles are available, such as from expendable bathy-thermographs (XBTs) launched from navy ships, submarines or aircraft, the temperature structure at a depth of around 545 m which they reveal will give a good indication of the location of eddies. Acknowledgements The BLUElink ocean data products were provided by the Commonwealth Scientific and Industrial Research Organisation (CSIRO, BRAN) and the Bureau of Meteorology (OFAM, BODAS). BLUElink is a collaboration involving CSIRO, the

15 160 R.H. Woodham et al. Commonwealth Bureau of Meteorology and the Royal Australian Navy. Robert Woodham wishes to gratefully acknowledge the support for this work provided by the Royal Australian Navy, through the Postgraduate Study at the Australian Defence Force Academy scheme. The support to Robert Woodham was not financial in nature. Disclosure statement No potential conflict of interest was reported by the authors. References Andreu-Burillo I, Brassington G, Oke P, Beggs H Including a new data stream in the BLUElink Ocean Data Assimilation System. Australian Meteorological and Oceanographic Journal. 59: Bell MJ, Lefebvre M, Le Traon P-Y, Smith N, Wilmer-Becker K GODAE the Global Ocean Data Assimilation Experiment. Oceanography. 22(3): Brassington GB Multicycle ensemble forecasting of sea surface temperature, Geophys Res Lett. 40: doi: / 2013GL Brassington GB, Pugh T, Spillman C, Shulz E, Beggs H, Schiller A, Oke PR BLUElink> development of operational oceanography and servicing in Australia. Journal of Research and Practice in Information Technology. 39 (2): Brassington GB, Summons N, Lumpkin R Observed and simulated Lagrangian and eddy characteristics of the East Australian Current and the Tasman Sea. Deep Sea Research Part II: Topical Studies in Oceanography. 58: Bureau of Meteorology Operational implementation of the ACCESS Numerical Weather Prediction systems, NMOC Operations Bulletin number. 83, 21 September Cresswell G The coalescence of two East Australian Current warm-core eddies. Science. 215: Emery WJ, Thomson RE Data analysis methods in physical oceanography. Pergamon; 400pp. Fu L-L, Chelton DB, le Traon P-Y, Morrow R Eddy dynamics from satellite altimetry. Oceanography. 23(4): Griffies S, Harrison MJ, Pacanowski RC, Rosati A A technical guide to MOM4. GFDL Ocean Group Technical Report 5, NOAA/Geophysical Fluid Dynamics Laboratory, 342pp. Hill KL, Rintoul SR, Oke PR, Ridgway K Rapid response of the East Australian Current to remote wind forcing: the role of barotropic-baroclinic interactions. Journal of Marine Research. 68: Jacobs GA, Woodham RH, Jourdan D, Braithwaite J GODAE applications useful to navies throughout the world. Oceanography. 22(3): Jolliffe IT, Stephenson DB Introduction. In: Forecast verification: a practictioner s guide in atmospheric science. Chichester, UK: John Wiley and Sons Ltd; p. 6, section Kalnay E Atmospheric modeling, data assimilation and predictability. Cambridge, UK: Cambridge University Press, Section pp. Leith CE Theoretical skill of Monte Carlo forecasts. Monthly Weather Review. 102(6): Marchesiello P, Middleton JH Modeling the East Australian Current in the western Tasman Sea. Journal of Physical Oceanography. 30: Martin M Ocean forecasting systems: product evaluation and skill. In: Schiller A, Brassington GB, editors. Operational oceanography in the 21st century. Dordrecht: Springer; p Mata MM, Tomczak M, Wijffels S, Church JA East Australian Current volume transports at 30S: Estimates from the World Ocean Circulation Experiment hydrographic sections PR11/P6 and the PCM3 current meter array. J. Geophys. Res. 105(C12): Mata MM, Wijffels SE, Church JA, Tomczak M Eddy shedding and energy conversions in the East Australian Current. Journal of Geophysical Research-Oceans. 111:C Murphy AH, Epstein ES Skill scores and correlation coefficients in model verification. Monthly Weather Review. 117: Oke PR, Brassington GB, Griffin DA, Schiller A The BLUElink ocean data assimilation system (BODAS). Ocean Modelling. 21: Oke PR, Schiller A, Griffin DA, Brassington GB Ensemble data assimilation for an eddy-resolving ocean model of the Australian region. Q J R Met Soc. 131(613): Potts J Basic concepts. Chapter 2. In: Jolliffe IT, Stephenson DB, editors. Forecast verification: a practictioner s guide in atmospheric science. Chichester, UK: John Wiley and Sons Ltd; p Ridgway KR, Dunn JR Mesoscale structure of the mean East Australian Current system and its relationship with topography. Progress in Oceanography. 56: Ridgway KR, Godfrey JS Seasonal cycle of the East Australian Current. Journal of Geophysical Research: Oceans. 102(C10): Roughan M, Middleton JH A comparison of observed upwelling mechanisms off the east coast of Australia. Continental Shelf Research. 22(17): Schiller A, Oke PR, Brassington GB, Entel M, Fiedler R, Griffin DA, Mansbridge JV Eddy-resolving ocean circulation in the Asian Australian region inferred from an ocean reanalysis effort. Progress in Oceanography. 76(3): Usui N, Tsujino H, Fujii Y, and Kamachi M Short-range prediction experiments of the Kuroshio path variabilities south of Japan. Ocean Dynamics. 56(5): Vallis GK Mean and eddy dynamics of the main thermocline. In: Velasco Fuentes OU, Sheinbaum J, Ochoa J, editors. Nonlinear processes in geophysical fluid dynamics. Kluwer Academic Publishers, p Venegas SA Statistical methods for signal detection in climate, DCESS Report number 2, Danish Center for Earth System Science (DCESS), Niels Bohr Institute for Astronomy, Physics and Geophysics, University of Copenhagen, Denmark. Wilkin JL, Zhang WG Modes of mesoscale sea surface height and temperature variability in the East Australian Current. Journal of Geophysical Research: Oceans. 112:C Woodham R Defence applications of operational oceanography: an Australian perspective. In: Schiller A, Brassington GB, editors. Operational oceanography in the 21st century. Dordrecht: Springer, p Woodham R, Brassington GB, Robertson R, Alves O Propagation Characteristics of Coastally Trapped Waves on the Australian Continental Shelf. Journal of Geophysical Research: Oceans. 118: Nomenclature < > Spatial mean over all grid points of a gridded dataset CC Correlation coefficient