The Regional Ice Prediction System (RIPS): verification of forecast sea ice concentration

Transcription

1 Quarterly Journal of the Royal Meteorological Society Q. J. R. Meteorol. Soc. 142: , January 2016 B DOI: /qj.2526 The Regional Ice Prediction System (RIPS): verification of forecast sea ice concentration Jean-François Lemieux, a * Christiane Beaudoin, a Frédéric Dupont, b François Roy, a Gregory C. Smith, a Anna Shlyaeva, c Mark Buehner, c Alain Caya, c Jack Chen, d Tom Carrieres, d Lynn Pogson, d Patricia DeRepentigny, e André Plante, b Paul Pestieau, d Pierre Pellerin, a Hal Ritchie, a Gilles Garric f and Nicolas Ferry f a Recherche en Prévision Numérique Environnementale, Environnement Canada, Dorval, Québec, Canada b Centre Météorologique Canadien, Environnement Canada, Dorval, Québec, Canada c Data Assimilation and Satellite Meteorology Research Section, Environment Canada, Dorval, Québec, Canada d Canadian Ice Service, Environment Canada, Ottawa, Ontario, Canada e Department of Atmospheric and Oceanic Sciences, McGill University, Montréal, Québec, Canada f Mercator Océan, Ramonville St Agne, France *Correspondence to: J.-F. Lemieux, Recherche en Prévision Numérique Environnementale, Environnement Canada, 2121 Route Transcanadienne, Dorval, Québec H9P 1J3, Canada. Jean-Francois.Lemieux@ec.gc.ca Reproduced with the permission of the Minister of Environment Canada In recent years, the demand for improved environmental forecasts in the Arctic has intensified as maritime transport and offshore exploration increase. As a result, Canada has accepted responsibility for the preparation and issuing services for the new Arctic MET/NAV Areas XVII and XVIII. Environmental forecasts are being developed based on a new integrated Arctic marine prediction system. Here, we present the first phase of this initiative, a short-term pan-arctic 1/12 resolution Regional Ice Prediction System (RIPS). RIPS is currently set to perform four 48 h forecasts per day. The RIPS forecast model (CICE 4.0) is forced by atmospheric forecasts from the Environment Canada regional deterministic prediction system. It is initialized with a 3D-Var analysis of sea ice concentration and the ice velocity field and thickness distribution from the previous forecast. The other forcing (surface current) and initialization fields (mixed-layer depth, sea surface temperature and salinity) come from the 1/4 resolution Global Ice Ocean Prediction System. Three verification methods for sea ice concentration are presented. Overall, verifications over a complete seasonal cycle (2011) against the Ice Mapping System ice extent product show that RIPS 48 h forecasts are better than persistence during the growth season while they have a lower skill than persistence during the melt period. A better representation of landfast ice, oceanic processes (wave ice interactions, upwelling events, etc.) in the marginal ice zone and better initializing fields should lead to improved forecasts. Key Words: sea ice; sea ice forecasting; verification method; Arctic Ocean Received 22 August 2014; Revised 12 January 2015; Accepted 28 January 2015; Published online in Wiley Online Library 25 March Introduction Since 1979, consistent satellite observations of the Arctic reveal that the sea ice areal extent is decreasing for all months of the year with a more pronounced trend at the end of summer. Hence, from 1979 to 2010, it is estimated that the September downward trend is 12.4% decade 1 (Stroeve et al., 2012). In September 2012, the minimum extent clearly reached a level below all other minima since 1979 when it shrank to km 2 ( accessed 13 February 2015). The reduction of sea ice in the Arctic region has led to increased economic activities such as maritime transport and oil exploration. There is therefore a growing need to obtain reliable sea ice forecasts over many strategic regions and on time-scales ranging from short-term to seasonal. In response to this need, five additional MET/NAV Areas (referred to as METAREAs) over the Arctic have been created as part of the Global Marine Distress and Safety System. Following published by John Wiley & Sons Ltd on behalf of the Royal Meteorological Society. This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distributioninany medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.

2 The Regional Ice Prediction System 633 the successful operational implementation of a fully coupled atmosphere ice ocean system for the Gulf of St Lawrence (Pellerin et al., 2004; Smith et al., 2012), Canada has accepted responsibility for the preparation and issuing services for two of the Arctic METAREAs (XVII and XVIII). These METAREAs cover the pie-shaped region of the Arctic bounded roughly from Bering Strait to Greenland up to the North Pole. As a result, a new initiative is under way at Environment Canada to develop an integrated Arctic marine prediction system. Here, we present the first phase of this initiative, a short-term pan-arctic 1/12 resolution Regional Ice Prediction System (RIPS). This system will be subsequently updated to include a coupled ice ocean model. In the final stage, a fully coupled atmosphere ice ocean snow wave model will be implemented. This initiative is part of a broader interdepartmental activity called the Canadian Operational Network of Coupled Environmental Prediction Systems (CONCEPTS). Compared to the field of numerical weather prediction, operational sea ice forecasting is still in its infancy, with only a few institutions around the world producing operational sea ice predictions. The Naval Research Laboratory in the United States has pioneered the development of such operational sea ice forecasting systems. In the 1990s, they implemented the Polar Ice Prediction System (Preller and Posey, 1996), a 0.28 resolution ice ocean coupled system. Recently, they have developed a more advanced and higher-resolution (1/12 ) ice ocean coupled forecasting system called the Arctic Cap Nowcast/Forecast System (ACNFS; Posey et al., 2010). ACNFS is based on the Los Alamos CICE model and the HYCOM ocean model. Through the MyOcean Project initiative, the European Union has also recently implemented an ice ocean prediction system: Towards an Operational Prediction system for the North Atlantic European coastal Zones (TOPAZ; Bertino and Lisaeter, 2008). The main objectives of this article are to introduce RIPS (focusing on the forecasting component), to present the verification methods that we have used/developed and to provide objective assessments of RIPS forecasting skill over a complete seasonal cycle. Emphasis is put on the verification of the sea ice concentration field. An earlier version of RIPS was running experimentally in operations at the Canadian Meteorological Center from September 2013 to September The latest version described in this manuscript has been running experimentally in operations since August This article is structured as follows. Section 2 presents RIPS with a focus on the forecasting component. We then discuss in section 3 the verification methods used to evaluate RIPS sea ice concentration forecasts. The forecasting skill of RIPS over a complete seasonal cycle is assessed in section 4. A discussion about the sensitivity of our forecasting scores is given in section 5. Concluding remarks and presentation of future work are found in section The Regional Ice Prediction System RIPS is a short-term sea ice forecasting system. Its domain, which is shown in Figure 1, includes the Arctic Ocean, the North Atlantic and the ice-infested waters around Canada. The RIPS 1/12 grid is a subset of the ORCA12 grid (Drakkar Group, 2007). The mesh has a spatial resolution of 4 5 km in the central Arctic and as high as 2 km in some channels of the Canadian Arctic Archipelago (CAA). All the lateral boundaries are closed (including the Bering Strait). Four 48 h forecasts are produced per day, initialized at 0000, 0600, 1200 and 1800 UTC. The sea ice model is the Los Alamos CICE model version 4.0 (Hunke and Lipscomb, 2010). CICE also includes a simple mixed-layer ocean model which is used to simulate the evolution of the sea surface temperature (SST) during the 48 h forecasts. CICE is a dynamic/thermodynamic sea ice model. It is a continuum-based model; it does not track individual ice floes but rather calculates the evolution of a thickness distribution. The Figure 1. RIPS domain and horizontal resolution (km). The black hatched line defines the limit of the atmospheric regional deterministic prediction system. thickness distribution evolves because of both thermodynamic processes (vertical growth/melt and lateral melt) and dynamic processes (advection and redistribution). The thermodynamic component calculates the temperature profile and growth/melt of snow and ice. The dynamic component calculates the velocity field by solving the 2D sea ice momentum equation. We give below a very brief description of the equations describing the dynamics and the thermodynamics of CICE. More details can be found in Hunke (2001), Lipscomb et al. (2007) and Hunke and Lipscomb (2010). The evolution of the thickness distribution g(h)isgivenby g t = (gu) + ψ (fg) h + L, (1) where g dh is the fractional area covered by ice with a thickness between h and h + dh, u = ui + vj is the horizontal sea ice velocity vector (i and j are unit vectors aligned with the x, y axes), ψ is the redistribution function, f defines the rate of thermodynamic growth/melt and L is the lateral melt. The terms on the right-hand side of Eq. (1) respectively represent the horizontal advection, redistribution of thicknesses due to mechanical processes, transport in thickness space due to thermodynamic growth or melt, and creation of open water because of lateral melting. The integral of g dh for h > 0 gives the total fractional area covered by ice and is therefore equal to the ice concentration C. The velocity field in Eq. (1) is obtained by solving the 2D sea ice momentum equation which is given by m Du Dt = mf ck u + τ a + τ w + σ mg e H o, (2) where m is the combined mass of ice and snow per unit area, D/Dt is the total derivative, f c the Coriolis parameter, k isaunit

3 634 J.-F. Lemieux et al. vector aligned with the vertical z axis, τ a is the air ice stress, τ w the water ice stress, σ the internal ice stress tensor, g e the gravity of the Earth and H o the sea surface height. As the advection of momentum is neglected, the term mdu/dt becomes m u/ t. The water ice stress τ w is expressed using a simple quadratic law with a constant turning angle θ w (McPhee, 1975). As surface currents are used to force RIPS, the turning angle is set to 0. However, the air ice stress τ a involves a more sophisticated formulation which takes into account the stability of the atmospheric boundary layer. More details can be found in the CICE documentation (Hunke and Lipscomb, 2010). A viscous-plastic (VP) constitutive law relates the internal ice stresses to the strain rates and the ice strength P. TheVP constitutive law used by CICE is based on an elliptical yield curve and a normal flow rule (Hibler, 1979). The ice strength can be calculated either by using the simple formulation of Hibler (1979) or the more physically realistic formula of Rothrock (1975) for a multi-thickness category model (we use the approach of Rothrock, 1975). In CICE, the momentum equation is solved with the Elastic-VP (EVP) scheme (Hunke and Dukowicz, 1997; Hunke, 2001). The thermodynamic component calculates the snow/ice temperature profile in the vertical by solving a heat diffusion equation. In RIPS, the default number of layers (four ice layers and one for the snow) per ice category is used (with a customized number of categories, see below). The upper boundary condition is the net heat flux from the atmosphere. The various heat fluxes are the latent, sensible, incoming and outgoing long-wave and short-wave radiations. We use the default Community Climate System Model version 3 scheme (CCSM3; Hunke and Lipscomb, 2010) to calculate the albedo and the attenuation of the absorbed short-wave radiation. The ice is assumed to have a constant salinity of 0 psu and therefore the effect of brine pockets is neglected. The lower boundary condition is the heat flux from the ocean to the ice. Based on the temperature profile and the boundary conditions, growth (at the base) or melt (at the base and/or at the top) are calculated. Lateral melting is parametrized based on the approach of Steele (1992) and depends on a specified value of the average diameter of the ice floes. Recently, a 1/4 resolution Global Ice Ocean Prediction System (GIOPS) was implemented operationally in experimental mode at Environment Canada. GIOPS includes assimilation of ocean and sea ice data. It produces daily analyses of ice and ocean conditions and performs 10 day forecasts each day. SST data are assimilated every day. Wednesday analyses are more complete as sea level anomaly and in situ data such as Argo profiles and moorings observations are also assimilated. Details of GIOPS can be found in Smith et al. (2014). GIOPS forms the backbone of operational ice ocean systems being developed at Environment Canada. RIPS relies on GIOPS for forcing by surface currents and for some fields at initialization. More precisely, RIPS is forced by 3 h mean surface (0.5 m) currents from GIOPS forecasts. Note that surface currents are used to force CICE dynamics and to calculate the ocean heat flux at the base of the ice. However, the simple mixed-layer ocean model in CICE does not use the surface currents to calculate lateral advective heat fluxes. The mixed-layer ocean model in RIPS is initialized with SST, sea surface salinity (SSS) and mixed-layer depth (MLD) fields from GIOPS. The GIOPS MLD is calculated as the point where the water density is equal to the surface density plus 0.01 kg m 3. During a RIPS forecast, the spatially varying SSS and MLD fields are held constant whereas the SST evolves based on forcing and ice conditions. New ice (frazil) can grow when the temperature of the mixed layer reaches the freezing point temperature. The change in sea ice concentration associated with this new ice depends on the energy available for growth and the specified thickness of the frazil ice. As GIOPS provides a 0000 UTC SST analysis, the RIPS 0000 UTC forecast is initialized with this field. For consistency between the SST and SSS fields, the 0000 UTC SSS field from GIOPS is used. For the MLD field, we observed a small improvement in RIPS forecast skill (not shown) when using the GIOPS MLD field in the middle of RIPS forecast (i.e. at time 24 h) instead of using the 0000 UTC GIOPS MLD. Ideally, GIOPS would also provide analyses at 0600, 1200 and 1800 UTC for initializing the RIPS SST field. As only one SST analysis is available each day, all four RIPS forecasts are initialized with the same GIOPS 0000 UTC SST analysis. For consistency, the GIOPS 0000 UTC SSS field is used as well for RIPS 0600, 1200 and 1800 UTC forecasts. It was also decided to use the same MLD field for all four forecasts. Note that the recycling of RIPS SST field was tested for 0600, 1200 and 1800 UTC forecasts but this sometimes led to less skilful forecasts. RIPS atmospheric forcing fields come from the 10 km resolution forecasts from the Environment Canada Regional Deterministic Prediction System (RDPS). The RDPS produces four 48 h forecasts per day with the same starting times as RIPS (0000, 0600, 1200 and 1800 UTC). However, as the RIPS domain is slightly larger than that of the RDPS (Figure 1), RDPS forcing fields are blended with the Global Deterministic Prediction System (GDPS) 25 km resolution forecasts at the boundaries of the RDPS grid. Both the RDPS and GDPS are based on the Environment Canada Global Environmental Multiscale (GEM) model (Côté et al., 1998). The forcing fields provided by the RDPS and GDPS are the winds at approximately 40 m above the surface, surface air temperature, humidity, precipitation, and downward long-wave and short-wave fluxes. These fields are spatially interpolated on the RIPS grid at intervals of 3 h and then linearly interpolated in time to force the CICE model. For each forecast, the analysis component of RIPS, based on three-dimensional variational (3D-Var) data assimilation (Buehner et al., 2013, 2014) provides the initial ice concentration field at the same 1/12 resolution. These ice concentration analyses (sometimes simply referred to as analyses in this article) are produced every 6 h (at 0000, 0600, 1200 and 1800 UTC) using persistence as the background field. Retrievals of sea ice concentration from passive microwave (Special Sensor Microwave Imager, SSM/I; Special Sensor Microwave Imager/Sounder, SSMIS) and advanced scatterometer data, and manually produced sea ice charts from the Canadian Ice Service (CIS) are assimilated by the 3D-Var system. A spreading procedure is used to propagate information from data-rich regions to regions with low data coverage (e.g. narrow channels of the CAA). Every Wednesday at 0000 UTC, the snow field, the velocity and the ice thickness distribution from GIOPS are used to initialize RIPS. Constraining RIPS with these GIOPS fields each week is justified by the fact that GIOPS should maintain a more (longterm) realistic ice thickness distribution and snow cover as it is based on a sophisticated ice ocean system with assimilation. Hence, forecasts starting on Wednesdays at 0000 UTC do not depend at all on RIPS previous forecasts. At any other times, the initial ice thickness distribution, snow and velocity fields are cycled from the previous RIPS forecast. The partial concentrations of the thickness distribution are rescaled such that the total ice concentration is equal to the one given by the 3D-Var system. Given a total concentration C u for the uncorrected distribution and C for the 3D-Var concentration, the updated partial concentrations are obtained by multiplying the uncorrected ones by C/C u (referred to as the Rescale Existing Distribution method in Smith et al., 2014). RIPS and GIOPS use the same ten sea ice categories with intervals (in cm) of [0 10, 10 15, 15 30, 30 50, 50 70, , , , , 600]. When the analysis indicates that ice is present in a grid cell while the unscaled distribution (either from RIPS or GIOPS) does not have ice, 25 cm thick ice is put at this location. The ice temperature at the beginning of the forecast is set to a linear profile between the freezing point temperature at the base and the atmospheric temperature at the surface.

4 The Regional Ice Prediction System 635 atmospheric forecasts surface currents CICE Sea ice model errors for the persistence are defined as the difference between the ice concentration of the initial analysis and the ice concentration of the analysis 48 h later. To quantify the errors, the Root Mean Square Error (RMSE) and the bias are calculated for every forecast. For a lead time of 48 h, the RMSE and the bias for the forecast are defined as SST SSS MLD Mixed-layer ocean Figure 2. Schematic of RIPS showing the fields used to initialize and force the ocean and ice components of the CICE model. Atmospheric forecasts come from the 10 km RDPS blended with the 25 km GDPS fields. The initial ice concentration field is obtained using a 3D-Var approach. The mixed-layer ocean model is initialized with the 0000 UTC GIOPS SST analysis, the 0000 UTC GIOPS SSS field and the GIOPS MLD field. CICE dynamics are forced by GIOPS surface currents. The ice thickness distribution (ITD), snow and velocity fields are initialized from the previous forecast fields (except on Wednesdays). Fields from GIOPS are interpolated to the RIPS grid using a bilinear approach. As the GIOPS grid has a lower resolution than the 1/12 resolution RIPS grid, a spreading procedure is used to ensure that narrow channels and fjords (not resolved by GIOPS) arealso covered. GIOPS simulated ice velocities were previously compared to data from the International Arctic Buoy Program (IABP; Rigor and Ortmeryer, 2004). The surface ice roughness parameter and the ice ocean drag coefficient were adjusted in order to reduce the errors between the simulated and observed drifts. The GIOPS parameters are m for the roughness and for the drag coefficient. Note that the C f parameter (which accounts for energy loss due to friction) was kept to its standard value of 17. Despite the RIPS higher spatial resolution, the same parameter values were also found to be optimal (results not shown). The other physical parameters are the default values of CICE 4.0 (Hunke and Lipscomb, 2010) except the thickness of the frazil ice which is set to 8 cm instead of 5 cm (this is discussed in section 5). Figure 2 describes schematically the RIPS initialization fields, forcing and output fields. At this stage of development, the most useful output fields from RIPS are the ice concentration (C), the ice velocity u, and the internal ice pressure p i. Once the forecast is completed, the simulated concentration field is verified (section 3 gives details). RIPS is a relatively simple system as its modelling part is only composed of a sea ice model with a mixed-layer ocean model. This simplicity makes it computationally efficient despite its high spatial resolution. 3. Sea ice concentration verification methods We present here the three verification methods that were developed to evaluate RIPS forecast sea ice concentration. We focus on the forecasts starting at 0000 UTC and mostly on lead times of 48 h. Similar scores are obtained for forecasts starting at 0600, 1200 and 1800 UTC (not shown). The impact of the lead time is not presented in this article Method 1: verification against the 3D-Var analyses Method 1 compares the sea ice concentration forecasts at lead times of 48 h to the RIPS 3D-Var analyses valid at the same time. To obtain a reference, the analyses at the initial time (persistence) are compared to the analyses at the same lead times. Consequently, the initial analysis (A 0h ) and the 48 h forecast (F 48h ) are both compared to the analysis (A 48h ) at the valid time of the forecast. The forecast errors are defined as the difference between the forecast ice concentration and the analysis 48 h later. Similarly, N ( i=1 F i RMSE f = 48h A i 2 48h), (3) N N ( i=1 F i bias f = 48h A i 48h), (4) N where N is the size of the sample, i.e. the number of grid cells used for these calculations. As explained below, N is not constant as it changes from one forecast to the next. Similarly, for persistence, these are calculated as N ( i=1 A i RMSE p = 0h A i 2 48h), (5) N N ( i=1 A i bias p = 0h A i 48h), (6) N where A 0h is defined as the persistence P. Calculating these statistics over the whole domain leads to very small signals. Indeed, over 48 h, the conditions change very little over most of the domain. A large part of the ice cover has a concentration close to 1 and at 48 h later still has a concentration close to 1 (this is especially true in winter). Similarly, the majority of grid cells without ice at time 0 of the forecast are also ice-free 48 h later. To amplify the signal, we use a verification method similar to the one of Van Woert et al. (2004). This verification method was also used by Smith et al. (2014). The statistics are only calculated where the concentration in the analysis has changed by more than 0.15 (15%) during the 48 h forecast period. The use of 0.15 was chosen to restrict the verification to areas of significant changes, and to maintain a large number of grid cells verified. This allows us to eliminate small changes in the analyses due to observational errors, precision of the data (e.g. RADARSAT image analyses are encoded with a precision of 0.1) and observations assimilated far from the point of interest. Applying this criterion allows one to define a verification mask where the RMSEs and biases are calculated. As opposed to Van Woert et al. (2004), the verification mask only depends on changes in the analysis and not on changes in the forecast concentration. For example, with a change of less than 0.15 in the analyzed concentration at a grid cell, this grid cell would not be included in the statistics even if the forecast ice concentration has changed by more than This is done for two reasons. First, it was observed that, during the growth season, the analysis tends to underestimate the ice concentration in narrow channels such as in the CAA and near the coasts. This is a consequence of the low resolution of the SSM/I and SSMIS sensors which leads to higher uncertainties in these areas (however the spreading procedure partly mitigates this problem). As the model grows ice in these areas (as expected), the inclusion of these zones in the calculation of error statistics would falsely deteriorate the forecasting skill compared to persistence. Second, with a 5 km spatial resolution, CICE produces some clearly defined linear kinematic features (LKFs) such as leads which are not seen in the analyses due to the large footprint of the SSM/I and SSMIS sensors. Kwok et al. (2008) show examples of LKFs simulated by sea ice models. These features would also illegitimately degrade the forecast skill compared to the reference skill of the persistence. Another filter is applied to define the verification mask by eliminating locations where there was no assimilation of recent observations in the analysis system. This is justified by the fact

5 636 J.-F. Lemieux et al. Figure 3. Sea ice concentration 48 h forecast valid on 10 October Error (forecast minus analysis) where the validation mask is 1 with method 1 (verification against the 3D-Var analyses). Errors are capped to ±0.5. that we compare the forecast to the persistence; as the analysis system uses the 6 h persistence as the background field, the skill of the persistence would have an unfair advantage over the model forecasts. This filter is easily implemented as the analysis also includes an index which specifies at each grid cell the number of days since the last observation (DSLO). An observation is considered recent if it is less than half a day old. Combining these two filters, the grid cells used to calculate the statistics are the ones for which A48h A0h > 0.15 and DSLO < 0.5. Figure 3 shows an example of a 48 h RIPS forecast sea ice concentration field. The forecast started on 8 October 2011 at 0000 UTC. Figure 3 shows the 48 h error field where the validation mask is 1. Comparing these two panels, it is obvious that most of the grid cells used for the validation are close to the ice edge. In fact, verification method 1 clearly focuses on the marginal ice zone (MIZ) as this is where most of the changes in 48 h occur (less true in summer when significant concentration changes can happen in the middle of the pack). This is an advantage as predicting sea ice changes in the MIZ is a top priority for navigation purposes. Method 1 is very useful to investigate the sensitivity of sea ice forecasts in the MIZ to initial conditions and parameter values. However, a drawback is that the method only evaluates the model s ability to simulate the ice evolution and does not take into account the analysis error. In addition, the method does not take into account the misses and false alarms (i.e. changes in the model simulated ice concentration where the analysis is reliable and indicates no change) Method 2: verification against the CIS RADARSAT image analyses The CIS image analyses of ice concentration are subjective interpretations of RADARSAT data ( accessed 13 February 2015). These analyses, provided on a 5 km resolution grid, are reliable estimates of the ice concentration field at specific times and locations. They are localized in ice-infested waters around Canada and grouped in subregions (eastern Arctic, western Arctic, etc.). These analyses cover a small area compared to the RIPS domain. With method 2, the forecast and persistence ice concentrations are linearly interpolated to the data points of the CIS image analyses. The RMSE and bias for both the forecast and persistence are calculated over all the points of the image analysis (Eqs (3) (6)). As the image analyses could be valid at any time during the 48 h forecast, the closest (in time) output from RIPS is used for the comparison. RIPS outputs for the year 2011 were produced every hour. Hence, given an image analysis valid on 20 April 2011 at 1645 UTC, the 17 h forecast from RIPS that started on 20 April UTC would be used for the verification. For comparison, persistence is also compared to the image analysis. To facilitate the presentation and interpretation of the results, the verifications are grouped by lead times and subregions. Hence, verifications of h forecasts are grouped with the time label 0600 h, h forecasts are grouped with the time label 1800 h and so on for the other times. However, as the satellite follows a sun-synchronous orbit, it crosses a given area at discrete times. Hence the verification times for a specific region and label do not span 12 h but usually 2 3 h. Method 2 is useful to assess the performance of the system over specific regions. However, a weakness of this method is that the image analyses provide only a limited coverage as they are produced to meet the needs of CIS operations (often guided by assistance to navigation) Method 3: verification against the IMS extent analyses As mentioned before, method 1 does not take into account the analysis error and false alarms and method 2 only considers small areas of the RIPS domain. In order to assess the quality of the full prediction system (analysis and model) over the whole domain and over a complete seasonal cycle, an independent ice analysis was used: the 4 km resolution ice extent product from the Ice Mapping System (IMS; Helfrich et al., 2007). The IMS daily ice extent is a subjective analysis produced manually by the US National Ice Center by using satellite imagery, mapped products and surface observations. It is a binary field: when the observed concentration is below 40%, the IMS analysis considers it is open water, while above this threshold the grid point is taken to be ice covered. Persistence and 48 h forecasts (starting at 0000 UTC) were verified against the IMS product for the whole year To compare the forecast and persisted concentrations to the IMS product, these fields were first interpolated on the IMS 4 km grid and then converted into a binary field using the same 40% threshold. Our IMS verification package (described in Buehner et al., 2013; Smith et al., 2014) then calculates the proportion of points over the whole domain that are correct. This metric is referred to as the proportion correct total (PCT). An example of c 2015 The Authors and Environment Canada. Quarterly Journal of the Royal Meteorological Society

6 The Regional Ice Prediction System 637 a correct point is a point for which the IMS indicates that ice is present and the forecast (or persisted) concentration is higher than 40% at the same location (with the same idea for open water points). The optimal PCT value is 1.0 which means that all the points are in agreement with the IMS analysis. With method 3, the bias is defined as the number of grid points covered with ice in the interpolated field being verified divided by the number of ice-covered points in the IMS analysis. The optimal value for this frequency bias is 1. Our verification package also assesses performances over subregions but these results are not presented in this article. As method 3 is based on an independent and mature ice analysis available every day and covering the whole RIPS domain, it is the method prioritized to optimize and assess the forecasting skill of RIPS. However, the fact that method 3 is a binary product is a drawback: the exact value of the ice concentration is not considered and the metrics are sensitive to the value of the ice concentration threshold (Smith et al., 2014). 4. Verification results RIPS sea ice concentration forecasts were evaluated for the year 2011 (the first 11 days of 2011 were not evaluated because GIOPS fields were not available). Care must be taken when comparing the domain-averaged persistence and forecast biases, as large errors can compensate and lead to an overall low bias. When assessing whether RIPS forecasts are better or worse than persistence with methods 1 and 2, the emphasis is given to the RMSE values: RIPS forecasts are considered better than persistence if RMSE f <RMSE p. For the IMS verification approach (method 3), the PCT metric is used to assess the forecasting skill. RIPS is considered better than persistence if the PCT of the forecast (PCT f ) is higher than the one for persistence (PCT p ). Note that we want the RMSE to be as small as possible and the PCT to be as close to 1.0 as possible. For methods 1 and 3 we focus on a lead time of 48 h. For method 2, lead times between 24 and 36 h are considered for the example that is given. The x-axis for the time series results always corresponds to the time of verification (not the starting time of the forecast). Figure 4 shows the RMSE, the bias and the number of points used for verification with method 1 as a function of time. With this verification approach, RIPS forecasts are better than persistence for most of the year; the forecasts are slightly worse than persistence in January and at the end of December and they are equivalent to persistence in February. These more difficult periods for the forecasts are associated with larger positive biases. Overall, the forecasts exhibit a positive bias throughout the year. As we will see later, one must be careful not to draw too strong conclusions with this verification method as it does not take into account the analysis errors nor areas of incorrect model changes. However, method 1 remains a very useful tool for sensitivity studies (section 5). An example of a verification done with method 2 for lead times between 24 and 36 h is shown in Figure 5. Compared to verification method 1, this method usually leads to smaller RMSEs and biases. This is due to the fact that verification method 2, as opposed to method 1, considers all the data points and does not discriminate between the ones in the middle of the pack or in the open ocean (with small errors) and the ones located in the MIZ (with larger errors). This method is useful to identify bad forecasts over a specific region and to guide case-studies. Figure 5 indicates that the forecast that started on 18 December 0000 UTC (with verification time 19 December at 1100 UTC) clearly has larger RMSE and bias than the persistence. For this specific verification time, spatial errors for persistence and the forecast are respectively shown in Figure 6(a, b). As this is the growth season, errors tend to be negative for persistence (Figure 6). On the other hand, (c) Figure 4. RMSEs, biases and (c) the number of points used to calculate the statistics for 2011 with verification method 1 (verification against the 3D-Var analyses). The lead time is 48 h. The persistence is in dashed grey and the forecasts in solid blue. An averaging window of 10 days was used to smooth the curves.

7 638 J.-F. Lemieux et al. (c) Figure 5. RMSEs, biases and (c) number of points used to calculate the statistics for December 2011 with verification method 2 (verification against the RADARSAT image analyses). Lead times for the verifications are between 24 and 36 h. The region of verification is the Eastern Arctic (Baffin Bay, the northern part of the Labrador Sea and the area north of Que bec). The persistence is in dashed grey and the simulation in solid blue. Figure 6. Difference (error) between RIPS persistence and the forecast and the RADARSAT image analysis valid on 19 December 2011 at 1100 UTC (method 2). This RADARSAT image analysis is located north of Que bec. The lead time is 35 h. Errors are capped to ±0.5. Figure 6 reveals that the model overall captures the ice growth. This RIPS forecast has very small errors in the northern part of the RADARSAT image but it has too much ice in Hudson Strait, Ungava Bay and Frobisher Bay. This specific forecast could be a candidate for a future detailed case-study. Before we evaluate the forecasting skill of RIPS with method 3, it is useful to consider only the 3D-Var ice concentration analyses. Indeed, as they are used for initialization, the quality of RIPS forecasts strongly depends on the quality of these analyses. Figure 7 compares the PCT and frequency bias, for the whole domain, for the 48 h persistence and for the 3D-Var ice concentration field valid at the same time as the IMS analysis. For example, on 16 June 0000 UTC (verification time), the ice concentration analysis on 14 June 0000 UTC (persistence) and the one valid on 16 June 0000 UTC (labelled analysis) are both compared to the 16 June 0000 UTC IMS analysis. The 3D-Var analyses valid at the same time as IMS are expected to have a higher PCT and a bias closer to 1 than the persistence analyses. We define PCTa and biasa respectively as the PCT and bias of the 3D-Var analyses valid at the same time as IMS. As all the grid cells are considered for the calculation of the PCT, the values are close to 1 and the difference between PCTa and PCTp is small (on the order of 0.001). PCTa is higher than PCTp for the whole year 2011 except for August and the beginning of September. The fact that PCTa < PCTp c 2015 The Authors and Environment Canada. Quarterly Journal of the Royal Meteorological Society

8 The Regional Ice Prediction System PCT pers analysis Bias Figure 7. Verification against the IMS analysis (method 3). shows the proportion of correct points and shows the frequency bias as a function of time (month) for the year The 48 h persistence is in dashed grey while the 3D-Var analysis valid at the same time as IMS is in solid black. A 10-day running mean was used to smooth the curves. month for this period is a consequence of a frequency bias significantly lower than 1 at the end of the melt period. This may be due to a misinterpretation of passive microwave data when a lot of melt ponds are present: ice-covered grid cells are interpreted as open water (Buehner et al., 2013, 2014). For the rest of the year, the 3D-Var analysis tends to have a frequency bias slightly smaller than 1. As RIPS is initialized with the 3D-Var ice concentration analysis, the difference in PCT between the 48 h persistence and the 3D-Var analysis valid at the same time as IMS can be loosely interpreted as a potential predictibility. PCT f is not expected to be higher than PCT a unless the model corrects deficiencies in the ice concentration analyses. In Figure 7, three periods of potential predictibility can be identified. These are periods for which PCT a is clearly higher than PCT p : the first four months of the year (JFMA), June and the beginning of July, and mid-october until the end of the year. We now verify to what extent RIPS is able to provide this level of forecasting skill. Figure 8 shows the PCT and bias of the persistence and of the forecasts valid at the same time as the IMS analysis. The time series for persistence in Figure 8 are the same as those in Figure 7. With respect to the three periods of potential predictibility, the forecasts lead to a higher PCT than PCT p for JFMA and for the last months of the year (except at the end of December). However, PCT f < PCT p during the melt period. For the end of the melt season, the fact that the frequency bias is significantly lower than 1 in the 3D-Var analyses is an important factor to explain these poor performances: RIPS does not have enough ice at initialization and further degrades the bias as it melts more ice. However, looking earlier in the melt season (June and beginning of July), it does seem that there is too much ice melting in RIPS forecasts. Indeed, the forecasts consistently have a frequency bias below 1 while the 3D-Var analyses do not. Understanding the cause of this excessive melt requires more investigation. During the growth season, the model tends to correct the slight negative bias in the analysis as bias f is usually closer to 1 than bias a (seen by comparing Figures 7 and 8). To investigate the statistical significance of these results, a bootstrap method was applied to the PCT time series (before the use of the averaging window). With this bootstrap method, n days are chosen randomly, with replacement, among the n days in a certain month. This means that some days can be selected more than once and others might not be chosen. The monthly mean PCT for the forecast and the persistence are then calculated with these n chosen days. This process is repeated times for each month. The forecasts are significantly better than persistence if PCT f is greater than PCT p more than times out of the bootstrap iterations (confidence level of 90%). The results indicate that the 48 h forecasts are statistically significantly better than persistence in JFMA, October and November. The forecasts are better than persistence in December, but not statistically significant. During the melt period, persistence is better than the forecasts (however with very small signals in May and September). These results are statistically significant. It is also useful to illustrate where, in the domain, the forecasts perform better or worse than persistence. Figure 9 shows the difference between PCT f and PCT p for JFMA (a period for which the forecasts are overall better than persistence) and Figure 9 displays PCT f -PCT p for MJJA (a period for which the forecasts are overall worse than persistence). A positive value (in red) means that, overall, for the period, the forecasts are more skilful than persistence at this location (vice versa for a negative value in blue). Overall, in JFMA the forecasts have more forecasting skill than persistence in regions such as the Barents Sea, Baffin Bay, Hudson Strait and at the ice edge of the Labrador Sea. However, the forecasts are worse than persistence in regions close to the coasts of Labrador, the Kara and the Laptev Seas. We think that this could partly be explained by the model s inability to simulate landfast ice. In MJJA (which corresponds to almost all the melt period), blue patches can be seen over many regions of the domain. Nevertheless, there are some regions where RIPS forecasts are more skilful than persistence (e.g the Chukchi Sea and close to the island of Novaya Zemlya). While method 1 indicates that RIPS forecasts are better than persistence in summer (Figure 4), method 3, which takes into account the false alarms, the misses and the analysis error, points out that RIPS does not have forecasting skill over the same period. As mentioned before, this is a consequence of excessive melt in the model and larger errors at the end of summer in the 3D-Var analyses due to the presence of melt ponds on the ice cover. Using the IMS product to assess RIPS forecasting skill does not exclude the fact that IMS has errors as well. As the IMS products are subjective binary analyses, there are certainly errors when interpreting ice concentration values close to the threshold of 40%. Here we used the same threshold to convert our forecast and persistence concentration fields to binary fields (in order to compare

9 640 J.-F. Lemieux et al PCT pers forecast Bias month Figure 8. Verification against the IMS analysis (method 3). shows the proportion of correct points and shows the frequency bias as a function of time (month) for the year Persistence is in dashed grey while the forecast is in solid blue. A 10-day running mean was used to smooth the curves. The lead time is 48 h. to IMS). Smith et al. (2014) have shown that for GIOPS, the PCT and bias, in summer, are very sensitive to this threshold value used to convert the forecast and persistence concentration fields. The fact that RMSEf >RMSEp in January and that RMSEf RMSEp in February (Figure 4) while PCTf > PCTp over the same period is due to disagreement between the 3D-Var analyses and the IMS product in some small regions such as Hudson Strait and in the St Lawrence Estuary (not shown). In fact, the model partly corrects these deficiencies in the initial ice concentration fields as it leads to a PCTf > PCTa in January and for some days of February (Figures 7 and 8). 5. Sensitivity of the verification scores As most of the changes in sea ice conditions in 48 h occur in the MIZ and close to the ice edge, verification scores are strongly dependent on initial conditions and forcing in this region. During the growth season for example, the MLD and the SST at time 0 are of primary importance for the quality of the forecasts. These fields (with SSS) determine the amount of heat that needs to be extracted from the mixed layer before ice can grow. As vertical ocean profiles are sparse and far from ice-infested waters, the vertical ocean structure (hence the MLD) in GIOPS is largely determined by the model and weakly constrained by the data. To illustrate the importance of the MLD, we performed a sensitivity experiment for the beginning of the growth season (from midseptember to the end of November 2011). Figure 10 illustrates the impact of the MLD on the RMSE and bias (verification method 1). The MLD field has a strong impact on the timing of the ice growth and therefore on the bias. Overall, the deeper the mixed layer, the lower is the ice production; this explains why the bias curve for the 2 MLD experiment is systematically below the one for 1 MLD which in turn is below the 0.5 MLD one. The same behaviour can be observed for method 3 (IMS) scores (not shown). A parameter that strongly affects the scores during the growth season is the specified thickness of the frazil ice (called hfrazilmin in CICE). Indeed, once the mixed-layer temperature reaches the freezing point, the change in sea ice concentration for a certain volume of new ice calculated by CICE depends on thickness of the new ice. In CICE, this parameter is (by default) fixed to 5 cm in both space and time. The influence of the thickness of the new Figure 9. Difference between the PCT of the forecast minus the PCT of the persistence (method 3). Red shows regions where the forecasts are better than persistence for JFMA 2011, and MJJA The differences in PCT are binned in 1 1 cells. The lead time is 48 h. c 2015 The Authors and Environment Canada. Quarterly Journal of the Royal Meteorological Society

10 The Regional Ice Prediction System 641 Figure 10. RMSEs and biases as a function of time for forecasts with the initial MLD field multiplied by 0.5 (in red), by 1.0 (standard, in blue) and by 2.0 (in black). These are verifications against the 3D-Var analyses (method 1). The persistence is in dashed grey. Forecasts were started every Wednesday at 0000 UTC between 14 September 2011 and 30 November The lead time is 48 h. ice on the verification scores is depicted in Figure 11 for mid- September to the end of November Using a thickness of 8 cm for the new ice improves both the bias and the RMSE during the growth season. Both PCT and bias with hfrazilmin = 8cm are better than with hfrazilmin = 5 cm (not shown). This is why hfrazilmin is set to 8 cm in RIPS. More work is required to determine the sensitivity of these scores during the melt season and to other fields and parameters. A final point about sensitivity of these scores is exclusively related to the verification methods themselves. In general, it is observed that the model produces sharper ice edges than the ones seen in the 3D-Var analyses. The smoother features in the analyses are a consequence of the covariance length-scale used in the assimilation method (Buehner et al., 2013, 2014) and the ingestion of low-resolution passive microwave data. Statistical metrics such as RMSE and contingency tables as used in method 3 are very sensitive to small-scale errors associated with fine structures (Gilleland et al., 2009). In other words, the forecast scores might be penalized compared to the persistence scores by the fact that the model produces sharp features. In Figure 12, the differences between the PCT for spatially averaged concentration fields and the PCT for the standard fields for the forecasts and the persistence are shown. Over the complete seasonal cycle, the smoothed forecasts lead to higher PCT than the standard ones. Spatially averaging the persistence field has a positive impact but it is usually smaller than for the forecast fields. Figure 12 shows another way to look at this: it presents the impact of spatial averaging on the forecasting skill compared to persistence. Overall, the spatial averaging increases the forecasting skill compared to persistence (especially true in MAMJ). 6. Conclusion This article introduces the first phase in the development of an integrated Arctic marine prediction system in support of the new Arctic MET/NAV areas XVII and XVIII. The Regional Figure 11. RMSEs and biases as a function of time for forecasts with thickness of the frazil ice of 8 cm (standard, blue) and 5 cm (black). These are verifications against the 3D-Var analyses (method 1). The persistence is in dashed grey. Forecasts were started every Wednesday at 0000 UTC between 14 September 2011 and30 November The lead time is 48 h.

11 642 J.-F. Lemieux et al. Figure 12. Impact of spatial averaging on the IMS PCT scores (method 3). Panel shows the PCT with spatial averaging (PCT avg ) minus the standard PCT for persistence (dashed grey) and the forecast (blue). Panel displays PCT f PCT p for the standard experiment (dashed black) and for spatially averaged fields (blue). The spatial averaging for cell i, j was obtained by averaging the concentration value of cell i, j and of the surrounding 80 cells (less than 80 cells are used when some of the surrounding cells are land points). A complete seasonal cycle is shown with forecasts done every Wednesday. The lead time is 48 h. Note the differentscalesonthe y-axis in and. Ice Prediction System (RIPS) is a short-term 1/12 resolution sea ice forecasting system. The grid of RIPS covers the Arctic Ocean, the North Atlantic and the ice-infested waters around Canada. The RIPS modelling component is the Los Alamos model CICE version 4.0. The mixed-layer ocean model included in CICE is also used as a simple ocean component. RIPS produces four 48 h forecasts per day, initialized at 0000, 0600, 1200 and 1800 UTC. RIPS relies on some forcing and initialization fields from the recently developed 1/4 resolution Global Ice Ocean Prediction System (GIOPS; Smith et al., 2014). GIOPS produces daily analyses and 10 day forecasts of ice ocean conditions. RIPS forecasts are forced by 3 h mean GIOPS forecast surface currents and initialized with some fields from GIOPS (such as the SST). RIPS atmospheric forcing fields come from the 10 km resolution forecasts from the Environment Canada RDPS which is based on the Environment Canada GEM model (Côté et al., 1998). The initial ice concentration field for the forecast is obtained using a 3D-Var data assimilation system (Buehner et al., 2013, 2014). These ice concentration analyses are produced every 6 h with persistence as the background field. Every Wednesday at 0000 UTC, the snow field, the velocity and the ten-category ice thickness distribution from GIOPS are used to initialize RIPS. Reinitializing these fields every week from GIOPS is justified by the fact that GIOPS should maintain a more (long-term) realistic ice thickness distribution (and snow cover) as it is based on a sophisticated ice ocean forecasting and assimilation system. At any other times, the initial ice thickness distribution, snow and velocity fields are those from the previous RIPS forecast. This manuscript presents three verification methods for forecast sea ice concentration. Method 1 verifies forecasts against the 3D-Var ice concentration analyses. To amplify the signal we are interested in, the forecast concentration is only verified where the 3D-Var concentration has changed by more than 0.15 (15%) over 48 h (similar to the work of Van Woert et al., 2004; Smith et al., 2014). This method focuses on a very important region for short-term sea ice forecasting: the MIZ. It is also very useful for performing sensitivity studies. However, it does not include the analysis error and the misses and the false alarms. Method 2 uses the CIS RADARSAT image analyses for verification. These images, manually produced, are subjective analyses of RADARSAT data. They provide reliable estimates of sea ice concentration at specific times over small regions of the RIPS domain. This method is useful to identify bad forecasts (over a certain region) which could then be investigated more thoroughly in a case-study. Finally, method 3 is based on the binary (ice no ice) extent product from the Ice Mapping System (Helfrich et al., 2007). As method 3 is based on a mature product available daily and covering the whole domain of RIPS, we use this method to assess RIPS forecasting skill. RIPS forecasts were validated for a complete seasonal cycle (2011). Proportion Correct Total (PCT) scores (method 3) show that RIPS is overall more skilful than persistence during the growth season. Furthermore, over the same period, the forecast bias is improved compared to that of persistence. During the melt season, there are some rare events for which the forecast is superior to persistence. However, during the period, the forecasts clearly do not have enough ice and are less skilful than persistence. This is a consequence of the insufficient amount of ice at initialization at the end of summer as well as excessive melt by the model (Figures 7 and 8 show details). The exact cause of this excessive melt will require further investigation. Conditions in the MIZ and close to the ice edge at initialization strongly determine short-term sea ice forecasts. A few sensitivity studies presented in this article point out that, during the growth season, the MLD field has a strong impact on the timing of ice formation and therefore on the amount of ice produced during the forecast (it therefore strongly affects the bias for all our verification scores). Once the SST reaches the freezing point, the change in area associated with the formation of ice in the model depends on the specified thickness of this new ice. This specified thickness considerably affects the bias and RMSE (and PCT) of our verification scores. This points out that, for better forecasts, the specified thickness should not be fixed in space and in time but should rather be parametrized based on conditions in the MIZ (such as the SSS and the wave ice interactions). These results highlight the importance of reliable and extended observations close to the ice edge for better constraining GIOPS conditions (SST, MLD, thickness distribution, etc.) in the MIZ. Metrics such as RMSE and the PCT score with method 3 are very sensitive to small-scale errors associated with fine structures (Gilleland et al., 2009). The forecasts tend to be penalized by these scores as they usually exhibit finer structures than the persistence analyses. This suggests that work is also required to use/develop verification methods more suited for correctly assessing the forecasting skill versus persistence. This is part of future work.