Imager-assisted cloud detection for assimilation of Infrared Atmospheric Sounding Interferometer radiances

Transcription

1 Quarterly Journal of the Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: , October 214 A DOI:1.12/qj.234 Imager-assisted cloud detection for assimilation of Infrared Atmospheric Sounding Interferometer radiances Reima Eresmaa* European Centre for Medium-range Weather Forecasts, Reading, UK *Correspondence to: R. Eresmaa, European Centre for Medium-range Weather Forecasts, Shinfield Park, Reading, RG2 9AX, UK. r.eresmaa@ecmwf.int The operational assimilation of Infrared Atmospheric Sounding Interferometer (IASI) radiances at the European Centre for Medium-range Weather Forecasts (ECMWF) relies primarily on the use of clear data, either in completely cloud-free locations or restricting the assimilation to channels that are insensitive to underlying cloud. Prior to the data assimilation, cloud-contaminated channels are identified and rejected in cloud detection, i.e. in a screening process based on observation minus background departure data. Background errors have the potential to confuse the cloud detection. On the one hand, a false alarm occurs when a background error is incorrectly interpreted as a cloud. On the other hand, cloud is missed if the background error compensates for the cloud radiative effect. This article outlines a method to improve the cloud detection by making additional use of collocated imager data from the Advanced Very High Resolution Radiometer (AVHRR). An independent cloud-detection scheme, based only on the AVHRR data, is formulated and compared with the departure-based scheme currently in operational use at ECMWF. The intercomparison reveals a considerable number of discrepancies, with only one of the two schemes suggesting the presence of cloud. Combining the two schemes results in an imager-assisted scheme, where the AVHRR data are used to set an additional requirement before allowing an IASI field of view to be diagnosed completely clear of clouds. In data assimilation experiments, using the imager-assisted scheme results in systematic lower tropospheric warming in the winter hemispheres, particularly over the Arctic sea ice. The modified cloud detection is shown to have a modestly positive impact on independent observation departure statistics and forecast scores. Key Words: AVHRR clusters; IASI; numerical weather prediction; observation quality control; radiance assimilation Received 5 July 213; Revised 15 November 213; Accepted 2 November 213; Published online in Wiley Online Library 7 February Introduction Hyper-spectral infrared sounders, like the Atmospheric Infrared Sounder (AIRS) and Infrared Atmospheric Sounding Interferometer (IASI), are highly beneficial observing systems in current global Numerical Weather Prediction (NWP) systems (McNally et al., 26; Collard and McNally, 29; Hilton et al., 29). The operational assimilation of hyper-spectral radiances still relies primarily on cloud-free data, even though substantial progress in the widely studied subject of assimilation of cloud-affected radiances has been made (Pavelin et al., 28; McNally, 29; Bauer et al., 211). Assimilation of cloud-free data is relatively straightforward, thanks primarily to advanced forward modelling techniques that are applicable in variational data assimilation systems. One challenge remaining in the assimilation of cloud-free infrared sounder radiances is to identify clear and cloudcontaminated data correctly. In the past, different approaches have been proposed for the cloud detection. The scheme of McNally and Watts (23) is a departure-based one: it is based on assumptions about the radiative effect of cloud on observation minus background departures (hereafter background departures). In contrast, the scheme proposed by Joiner et al. (24) assumes that cloud introduces horizontal variability in observed radiances, so that cloud can be identified by comparing between adjacent fields of view (FOVs). More traditionally, the differential cloud effect between microwave and infrared regimes is sometimes exploited, in which case the fundamental assumption is that the FOV of a collocated microwave instrument is sufficiently representative of that of the infrared sounder (English et al., 1999). In the context of the hyper-spectral infrared sounders, most currently used cloud-detection methods aim at detecting clear channels, rather than only identifying FOVs that are completely free of clouds. There are two ways in which the cloud detection can go wrong. Firstly, a false alarm occurs when the cloud-detection scheme c 213 Royal Meteorological Society

2 Imager-Assisted Cloud Detection for Radiance Assimilation 2343 suggests the presence of cloud in a cloud-free situation. Secondly, a miss occurs when a cloud occupying the FOV goes unnoticed. The extreme situations are usually the easiest to deal with in cloud detection, as the radiative effect of a thick and opaque cloud can be two or even three orders of magnitude larger than typical cloud-free background departures on some infrared channels. Failures are more likely to take place when the cloud either is transparent or covers only a small fraction of the FOV, so that its radiative effect is comparable with a typical background error. In the departure-based scheme of McNally and Watts (23), failures of cloud detection are usually associated with a background error: in cloud-free conditions, the background error can be misinterpreted as cloud (false alarm), whereas in cloudy conditions the cloud can be missed if the background error sufficiently compensates for the cloud radiative effect. In the case of the IASI instrument, information available at each sounding location consists of not only observed radiances on the sounder channels but also statistical radiance and reflectance properties within clusters of the Advanced Very High Resolution Radiometer (AVHRR) pixels occupying the IASI FOV (Cayla, 21). Intuitively, the collocated AVHRR data relate to surface properties and the presence of cloud within the IASI FOV and therefore can be considered for use in cloud-detection schemes. The statistical properties of IASI-collocated AVHRR have already proved useful for screening purposes in the context of assimilating cloud-affected radiances with explicit treatment of microphysical variables (Martinet et al., 213). In this article, we describe a method for using the collocated AVHRR information to assist the cloud detection. We expect the use of additional information to improve the cloud detection in the presence of background error. We will start by outlining the departure-based cloud-detection scheme that we are dealing with in this work, i.e. the operational scheme used at the European Centre for Medium-range Weather Forecasts (ECMWF), and discussing some of its characteristic weaknesses in section 2. We will then describe an independent cloud-detection scheme that is only based on the collocated AVHRR information in section 3. The independent AVHRR-based scheme is then combined with the departure-based scheme to provide an imager-assisted scheme, which is described and evaluated in section 4. The article is concluded in section Cloud detection using the background departures on IASI channels The departure-based cloud-detection scheme is based on McNally and Watts (23). At the time of writing, this scheme is applied as part of the operational assimilation of AIRS (McNally et al., 26) and IASI (Collard and McNally, 29) radiances at ECMWF. The scheme is designed to look for a signature of cloud in a spectrum of background departures, based on the assumption of a completely clear FOV. The implementation for IASI makes use of a selection of 184 long-wave channels, which are sensitive to different atmospheric layers and are relatively unaffected by gases other than carbon dioxide. Each channel is assigned with a height indicating the highest possible altitude at which an opaque cloud top would not measurably alter the radiance observation from its clear-sky value. Using these height assignments, background departures are ranked in the vertical and the resulting curve of background departures is processed with a moving average filter to reduce noise and make it easier to identify the radiative effect of cloud. Three examples of the vertically ranked and smoothed curve are shown in Figure 1 (note that the higher the channel rank index, the lower the channel is ranked in the vertical). Depending on the properties of the smoothed curve, one of three possible scenarios is chosen. In the case in which the smoothed curve does not go outside pre-defined thresholds (these are denoted by dashed lines in Figure 1), the quick-exit scenario is chosen. This will flag all channels clear. In the case in which the smoothed curve exceeds the positive threshold near the lowest-ranked channel only and otherwise stays between the thresholds, the processing will be continued in the warm-start scenario, where cloud is assumed to be warmer than the underlying surface. Most often, however, the properties of the smoothed curve are such that neither quick exit nor warm start can be chosen: in these cases the processing will be continued in the cold-start scenario, i.e. the cloud is assumed to be colder than the underlying surface. In the cold- and warm-start scenarios, the scheme determines the vertical extent of the cloud by searching for the lowestranked clear channel. This channel is found by stepping upwards, one channel at a time, from a sufficiently low-ranked starting position, until a cloud radiative effect is no longer identified. Both the smoothed curve value and its gradient are required to be within pre-defined threshold values at this point. Those channels ranked higher than the lowest clear channel are retained for the assimilation, while those ranked lower are flagged cloudy. In the warm-start example of Figure 1, the search is started from the lowest-ranked channel at position 184 and finished at the channel index 125; this is the position where the gradient of the smoothed curve gets sufficiently small for the first time. In the cold-start example, the search is started at channel index 95 (i.e. from the highest position where the curve is outside the departure thresholds shown by the dashed lines) and finished at channel index 7. Despite specifying alternative scenarios corresponding to cold and warm clouds, the scheme is not fully symmetrical with regard to positive and negative background departures. This is intentional, because positive background departures are far less likely to be associated with clouds than negative departures, especially when upper tropospheric channels are considered. The asymmetry is introduced primarily in the way the starting position for the search of the lowest clear channel is determined. In the cold-start scenario, the starting position is chosen as the highest point where the smoothed curve goes below the negative departure threshold, or, if the curve is everywhere above the line, as the global minimum of the curve. In the warm-start scenario, the starting position is always at the lowest-ranked end (a) Quick exit (b) Warm start (c) Channel rank index Channel rank index Cold start Channel rank index Figure 1. Examples of the smoothed background departure curve in cases entering (a) quick-exit, (b) warm-start and (c) cold-start scenarios of the departure-based cloud-detection scheme. Dashed lines indicate the threshold values separating the scenarios and dotted lines indicate the positions of the lowest clear channel in the warm- and cold-start scenarios. c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)

3 2344 R. Eresmaa of the smoothed curve. Therefore, clouds extending up to upper tropospheric, or even stratospheric channels, are rarely diagnosed in the warm-start scenario, but commonly in the cold start Failures of cloud detection in a simulated framework As the cloud-detection scheme described above bases its decisions on background departure information only, difficulties may be expected in situations where background errors have a large contribution to the departures. In particular, false alarms are likely to take place in clear situations that are associated with errors in background skin temperature and/or humidity fields. On the other hand, the presence of cloud in the FOV can also be missed if the background error compensates sufficiently for the cloud radiative effect. To assess how much these hypothetical possibilities have practical significance, we have set up a simulated framework where all sources of background departure are fully known. The set-up of the simulated framework is as follows. Standalone versions of the radiative transfer model RTTOV 9.3 (Saunders et al., 28) and the departure-based cloud-detection scheme (NWP SAF Aerosol and Cloud Detection Package: see are used. Background atmospheric profiles are obtained by adding simulated background errors that are consistent with the background-error covariance specification used in the ECMWF 4D-Var data assimilation system on top of reference model profiles, which are treated as truth. The truth dataset is based on a sample of 5 model profiles provided by the NWP SAF (Chevallier et al., 26). For each profile, four brightness temperature spectra are created. Three spectra are obtained through calls to the radiative transfer model: these correspond to the true profile assuming clear conditions, the true profile in cloudy conditions and the background profile assuming clear conditions. As the reference model profiles include a physically consistent description of cloud liquid and ice water contents, the detailed cloud model of the RTTOV 9.3 is used to simulate cloudy radiances. The fourth spectrum corresponds to the observed brightness temperature and is obtained by adding simulated observation errors on top of the true cloudy spectrum. The simulated observation errors are Gaussian with zero mean and a standard deviation of.4 K and no interchannel error correlations are assumed. The true clear and true cloudy spectra are used as such for determining the true cloud flags, whereas the simulated cloud flags are obtained by applying the cloud-detection scheme to the simulated departure between observed and background spectra. Table 1 summarizes simulated percentages of correct and incorrect cloud flags and the corresponding probabilities of detection (PoD) and false-alarm rates (FAR) on three IASI channels. The statistics are given for an upper tropospheric channel (with central wavenumber 7.75 cm 1 ), a midtropospheric channel (711 cm 1 ) and a lower tropospheric channel ( cm 1 ). The table shows high counts of false alarms and low counts of misses, highlighting the conservative tuning of the departure-based scheme. On the upper tropospheric channel, false alarms are by far the most common type of failure. FAR exceeding.25 means that more than every fourth cloudy diagnosis is a false alarm, whereas PoD exceeding.98 implies that fewer than 2% of all cloudy situations are missed. False alarms are even more common on the two lower-peaking channels but, because of the increased percentage of all cloudy situations, FAR is slightly lower there. On the other hand, the poorer PoD means that the probability of missing a cloud reaches around 5 6% on the mid- and lower tropospheric channels. In all cases, but especially at cm 1, false alarms as compared with correct clear flags are common. Figure 2 aims at illustrating the role of background errors in the origin of the cloud-detection failures. The top panels ((a) and (b)) show spectra of mean background errors (in IASI channel Table 1. Simulated percentages of correct clears, correct cloudies, misses and false alarms (FA) on upper, mid- and lower tropospheric IASI channels in the departure-based scheme. The two columns on the right show the probability of detection (PoD) and false-alarm rate (FAR). Correct Incorrect Wavenumber (cm 1 ) Clear Cloudy Miss FA PoD FAR space) in populations where a false alarm is made on either (a) the upper tropospheric channel or (b) the mid-tropospheric channel. Channels used in the cloud detection are indexed along the abscissae: the upper, mid-, and lower tropospheric channels of Table 1 are ranked at positions 76, 95 and 149, respectively. The strong gradient in the mean background errors among the lowest channels suggests that the false alarms are typically associated with warm surface temperature errors. There is no similar systematic nature in tropospheric background errors causing false alarms. Figure 2(c) and (d) shows scatter plots of the background error and cloud radiative effect in populations consisting of missed clouds on either (c) mid- or (d) lower tropospheric channels. The idea that the misses take place because of the background error compensating for the cloud radiative effect is strongly supported in the case of the lower tropospheric channel. This scatter plot is dominated by dots falling near the diagonal, so that majority of the cloud radiative effect is indeed cancelled out by the background error. In the case of the mid-tropospheric channel, this interpretation is weaker, though. In the simulations, the misses on the mid-tropospheric channel are often related to ineffective smoothing of vertically ranked background departures. 3. Cloud detection using the AVHRR cluster information This section describes a cloud-detection scheme based solely on the AVHRR cluster information. This scheme is also compared with the departure-based scheme and discrepancies found in cloud flags provided by the two schemes are discussed An imager-based cloud-detection scheme AVHRR is a widely exploited imager being flown on many meteorological satellites and providing information with high horizontal resolution for a variety of applications. The sensor measures outgoing radiation on six broad-band channels. These include a visible channel, two near-infrared channels, a shortwave infrared channel and two long-wave infrared channels. The visible and near-infrared channels provide useful reflectance data only in the daytime, whereas the radiance data from the shortwave channel are most useful at night. FOV (pixel) size of the sensor is around 1.1 km in nadir. The observational data stream for IASI includes statistical properties of radiance and reflectance within AVHRR pixels occupying the IASI FOV (Cayla, 21). Within each IASI FOV, a cluster analysis is applied to the collocated AVHRR pixels and up to seven clusters are formed. The mean and standard deviation of radiance, or reflectance, for the six AVHRR channels are determined within each cluster. Additionally, fractional coverage and location information within the IASI FOV are given for the clusters. It should be noted that there are also operational AVHRRbased cloud-top height products available. Compared with the cluster statistics, using such products in the context of radiance assimilation is complicated, because one would have to combine two observational data streams together. It is much easier to make use of the cluster statistics, because these are readily available in the IASI radiance data files. c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)

4 Imager-Assisted Cloud Detection for Radiance Assimilation 2345 (a) Background error [K].2.1. (b) Background error [K] (c) Background error [K] Channel index (d) Background error [K] Channel index Cloud effect [K] Cloud effect [K] Figure 2. Mean background error in channels used in the departure-based scheme in populations where a false alarm is made in (a) upper and (b) mid-tropospheric IASI channels. The bottom panels show the background error as a function of cloud radiative effect in populations where cloud is missed in (c) mid-tropospheric and (d) lower tropospheric channels. For the purpose of validating and improving the operational cloud detection scheme at ECMWF, we have developed an independent scheme (hereafter imager-based scheme) that makes use of the brightness temperature statistics (derived from radiance statistics) of the two long-wave channels of the AVHRR instrument. It is assumed that relying on the long-wave channels only makes the scheme equally usable during both day and night, even though some diurnal variation may be introduced by the different availability of the shorter-wave channels used in the clustering algorithm. Furthermore, the scheme is based on the fundamental assumption that each cluster consists of pixels that are either all clear or all cloudy. The spectral characteristics of the AVHRR instrument provide little information for obtaining a reliable diagnosis of cloud-top height in the case of some cloud in the IASI FOV. Therefore, we are only aiming to diagnose whether the IASI FOV is completely clear or not. The imager-based scheme consists of three separate checks, called the homogeneity check, the intercluster consistency check and the background departure check. In order to declare the input IASI FOV clear of clouds, none of the three checks is allowed to suggest the presence of cloud. The number of checks suggesting the presence of cloud could in principle be used as a very crude uncertainty estimate to accompany the diagnosis, but this option is not made use of in the first implementation. The homogeneity check makes use of the standard deviation of infrared brightness temperature, as computed over all pixels occupying the IASI FOV. The presence of cloud is suggested if the two standard deviations (one for each channel) both exceed their pre-determined threshold values (t H1 and t H2, respectively). In the intercluster consistency check, the presence of cloud is determined by intercomparing the properties of different clusters occupying the same IASI FOV. Each cluster is assigned with a distance to the background in the two-dimensional AVHRR observation space and that distance is compared with the distances to other clusters. It is assumed that the background does not vary horizontally within the IASI FOV. In the case in which there is a background error that is large enough to confuse the cloud detection, all clusters occupying the IASI FOV will be assigned with a large distance to the background. More importantly, a situation in which some clusters are assigned with a small distance while others are not can only be understood in terms of a cloud providing a partial cover within the IASI FOV. The presence of cloud is therefore diagnosed only if the clusters occupying the IASI FOV are inconsistent with each other. The intercluster consistency check is designed to look for a pair of clusters separated by a greater distance from each other than from the background. The minimum fractional coverage, t C, that both of the considered clusters need to span is specified in order to prevent very small (i.e. radiatively insignificant) clusters from affecting the decision. The distance between clusters j and k is computed as the squared-summed intercluster departure: D jk = 5 i=4 ( R j i Rk i ) 2, (1) where R j i is the mean brightness temperature of cluster j on channel i. Similarly, the distance to the background is computed for each cluster j as 5 ( ) D j = R j 2 i RBG i, (2) i=4 where R BG i is the background brightness temperature for channel i. The presence of cloud is diagnosed only if there are clusters j and k that both span at least the minimum coverage t C of the IASI FOV and the inequality D jk > min(d j, D k ) (3) is true. For the background departure check, we compute a test quantity as a fraction-weighted mean of the squared-summed background departures, i.e. D mean = N f j D j, (4) where N is the number of clusters in the IASI FOV and f j is the fractional coverage of cluster j. The presence of cloud is diagnosed if D mean exceeds a threshold value t R. As in most cloud-detection schemes, success in finding clear and cloudy data depends crucially on applied tuning parameter j=1 c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)

5 2346 R. Eresmaa Table 2. Tuning parameters of the imager-based cloud detection scheme. Parameter Symbol Threshold Unit 11 μm brightness temperature t H1.75 K standard deviation 12 μm brightness temperature t H2.8 K standard deviation Minimum coverage t C.3 Squared-summed background departure t R 1. K 2 Table 3. Percentages of cloud flags falling in each category in the departure-based and imager-based schemes. Departure-based Clear Cloudy Total Imager-based Clear Cloudy Total values. Those values used in this work are listed in Table 2. For the homogeneity check, the threshold values are based on quantile analysis, applied to homogeneous IASI FOVs consisting of one cluster only. Assuming that such FOVs represent predominantly clear situations, the threshold values are chosen near the 95th percentiles of the histograms of brightness temperature standard deviations. Threshold values for the intercluster consistency and background departure checks are specified such that the radiative effect from undetected clouds is likely to be small Intercomparison with the departure-based scheme Unlike the departure-based scheme, the imager-based scheme does not attempt to identify clear channels on top of cloudcontaminated ones. Therefore, intercomparison of the two schemes is only meaningful for the case of window channels. We have applied the imager-based scheme to a sample consisting of IASI FOVs. The data are collected during two subsequent 12 h assimilation time windows (corresponding to UTC and 12 UTC analyses on 21 November 211). The sample is based on a research experiment run at resolution T511L91 using version 37r3 of the Integrated Forecasting System (IFS) at ECMWF. IASI data from over land, as well as data on extreme scan positions, are excluded from the comparison. Processing the sample through the imager-based scheme and having access to cloud flags provided by the departure-based scheme as part of the experiment run allows us to construct a contingency table (Table 3) that consists of percentages of IASI FOVs falling in different categories according to their departurebased and imager-based cloud flags. The departure-based cloud flags are taken from the IASI channel with central wavenumber 875. cm 1. Comparing counts of clear flags in the two schemes indicates a slight frequency bias between the two schemes: the count of clear flags is higher in the departure-based scheme (1.7%) than in the imager-based scheme (8.6%), suggesting that, with respect to the imager-based scheme, the departure-based scheme is biased towards flagging too many FOVs clear. Just relying on these statistics, there is an equally valid but contrasting interpretation that, with respect to the departure-based scheme, the imagerbased scheme is biased towards flagging too few FOVs clear. Judging which interpretation is closer to the truth would require a more detailed view behind the statistics. Controversial populations, i.e. those cases that are flagged clear in one scheme and cloudy in the other, are relatively large compared with counts of clear flags in the two schemes. More than 5% of cases flagged clear in the departure-based scheme are diagnosed cloudy in the imager-based scheme, suggesting a considerable likelihood for the departure-based scheme missing clouds (and/or the imager-based scheme making false alarms). Similarly, approximately 4% of cases flagged clear in the imagerbased scheme are diagnosed cloudy in the departure-based scheme, suggesting that either false alarms are common in the departure-based scheme or the imager-based scheme misses lots of clouds. It is worth having a closer look in order to gain an understanding of what kind of failures dominate within the two controversial populations. We will first focus on the population that is flagged cloudy in the departure-based scheme and clear in the imager-based scheme. Data points falling in this population are characterized by non-zero background departure on some IASI channels and near-zero departures on the two AVHRR channels. It remains unclear whether the non-zero background departure is due to background error or cloud contamination. Two scenarios, one clear and one cloudy, can be considered to explain cases falling in this population. In the cloudy scenario, the cloud top is required to be homogeneous and relatively transparent: otherwise it would be detected in the imager-based scheme. In this scenario, the cloud would be successfully detected in the departure-based scheme, thanks to the high spectral resolution and sounding capability of IASI. In the clear scenario, there is no cloud but rather a background error that contributes to the background departure on some particular IASI channels while having no significant effect on the broad-band channels of AVHRR. This leads to a false alarm only in the departure-based scheme. Figure 3 provides some more information to assist interpretation of this population. Histograms of background departures are shown for nine IASI channels in populations that are flagged clear either in both schemes (solid lines) or only in the imager-based scheme (dashed lines). Histograms shown for the former population are symmetrical (and Gaussian) around zero on all channels, which is what we would expect from data that are correctly diagnosed as clear. Histograms shown for the latter population differ from pure Gaussian distributions in two important aspects. Firstly, the histogram of the window channel (875 cm 1 ) is bimodal. Secondly, histograms of lower tropospheric sensitive channels ( cm 1 ) stretch towards negative departures. We believe that the bimodality of the window channel can be attributed to background humidity errors, which do not affect the broad-band radiances of AVHRR strongly but impact some of the high spectral resolution channels of IASI. In other words, we tend to interpret the bimodality as false alarms in the departure-based scheme. Furthermore, we believe that the negative tails on sounding channels are fingerprints from cloud contamination and as such should be interpreted as misses in the imager scheme. Therefore, we conclude that the controversial population is contributed substantially by both false alarms in the departure-based scheme and misses in the imager-based scheme. Exactly how large the contribution from each type of classification error is remains unclear and is beyond the scope of this work. The other controversial population is the one where a clear flag in the departure-based scheme is associated with a cloudy flag in the imager-based scheme. The clear departure-based flag implies that the background departure is near zero throughout the long-wave band of IASI, but the presence of a background error that happens to compensate for the cloud radiative effect cannot be excluded. Again, two alternative scenarios can be formulated to explain cases falling in this population. In the cloudy scenario, the imager-based scheme is correct in finding the cloud, while the departure-based scheme misses it, primarily due to a compensating background error. In contrast, in the clear scenario the imager-based scheme is making a false alarm while the departure-based scheme correctly finds no clouds. Taking the spectral characteristics of AVHRR and IASI into account, the latter scenario is intuitively unlikely, because any background error that makes a noticeable departure in the broad-band channels of AVHRR would probably make a similar background departure c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)

6 Imager-Assisted Cloud Detection for Radiance Assimilation 2347 (a) (b) (c) cm cm cm (d) (e) (f) cm cm cm (g) (h) (i) cm cm cm Figure 3. Background departure histograms for nine IASI channels in populations where a clear imager-based cloud flag is associated with a clear (solid lines) and a cloudy (dashed lines) departure-based cloud flag. in at least some of the IASI channels as well. Therefore, it is considered more likely that the population consists mostly of cloudy scenes. A more detailed assessment of this population reveals that approximately 8% of those cases involved are diagnosed cloudy only because of the intercluster consistency check, i.e. because of two or more sufficiently different clusters occupying the IASI FOV. Interpreting this particular controversial population is therefore reduced to assessing whether the intercluster consistency check is correctly tuned. To make such an assessment, we have shown in Figure 4 the ratio of brightness temperature standard deviations as a function of cluster-mean brightness temperature difference for those pairs of clusters that trigger the cloudy diagnosis in the intercluster consistency check (only plotting those cases where the departure-based flag is clear). Interpretation of this figure follows the hypothesis that (on average) the effect of cloud on cluster statistics is not only to decrease mean brightness temperature but also to increase brightness temperature standard deviation within the cluster. Therefore, if the controversial population consisted predominantly of FOVs occupied by clear clusters only, no correlation would be expected between the two statistical measures. This is contrary to what is seen in Figure 4: there is a tendency that the colder cluster is often also the less homogeneous one. This is understandable in the context of the hypothesis described above. Therefore, we conclude that the controversial population contains a large contribution from cases of missed clouds in the departure-based scheme. It is interesting to consider circumstances in which both the departure-based scheme and two out of three checks of the imager-based scheme fail to detect the cloud, which is suggested only by the intercluster consistency check. Some idea of such circumstances can be gained through examining individual cases within the population. A typical example of such a case is illustrated in Figure 5. Panel (a) illustrates the performance of the departure-based scheme in this example. The smoothed background departure (solid line) stays within the pre-defined thresholds (dashed lines) and therefore provides no evidence of cloud. Statistical characteristics of the collocated clusters of AVHRR pixels are illustrated by Gaussian distributions (although it is known that the actual distributions behind the statistics are not necessarily Gaussian), together with the vertical Ratio of BT standard deviations Mean BT difference [K] Figure 4. Ratio of 11 μm brightness temperature standard deviation as a function of mean 11 μm brightness temperature difference for pairs of clusters triggering the cloudy diagnosis in the intercluster consistency check of the imager-based scheme. Only cases associated with a clear flag in the departure-based scheme are included. line corresponding to background brightness temperature, in panel (b). Mean brightness temperatures of the two clusters differ from each other by almost 1 K, but from the background brightness temperature by less than.5 K each. The two clusters are therefore inconsistent with each other and it is difficult to explain the radiative properties of the two clusters without the assumption of at least one cluster being cloudy. The most likely interpretation here is that the larger and warmer cluster represents clear pixels, whereas the colder and smaller cluster is cloudy. In terms of associated background error, this would mean the presence of either a cold error in lower tropospheric or surface temperature or a moist error in tropospheric humidity (or some combination of these). As a net effect of the cloud radiative effect and the background error, near-zero background departures are found on most IASI channels and the cloud contamination is consequently missed in the departure-based scheme as well as c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)

7 2348 R. Eresmaa (a) Background departure [K] (b) Normalized pixel count Background Clusters Channel rank index AVHRR BT [K] Figure 5. (a) The smoothed background departure (solid) for vertically ranked IASI channels in a single case that is flagged clear in the departure-based scheme and cloudy in the imager-based scheme. Dashed lines indicate the threshold values separating the quick-exit scenario from the other scenarios in the scheme. (b) Gaussian distributions (dashed lines) corresponding to the mean and standard deviation of 11 μm brightness temperature in two clusters of AVHRR pixels within the IASI FOV. The solid vertical line shows the background brightness temperature. in the background departure check of the imager-based scheme. Apparently, the radiative effect of cloud in this case is too small to trigger the cloudy diagnosis through the homogeneity check of the imager-based scheme either. 4. An imager-assisted cloud-detection scheme Both the intercomparison exercise presented in the previous section and the simulated cloud flags discussed in section 2.1 suggest that a considerable fraction of cloudy diagnoses in the departure-based scheme are false alarms in cloud-free situations. Unfortunately, individual cases where a false alarm is taking place remain difficult to point out with sufficient certainty. We will therefore refrain from trying to use the collocated AVHRR data to reduce the count of false alarms until a better understanding (possibly through a sophisticated use of visible and/or nearinfrared channels) is gained. However, the imager-based scheme also shows skill in identifying cloud contamination in cases that are diagnosed as clear in the departure-based scheme. Given that the cloudy interpretation made by the imager-based scheme is correct in these cases, use could be made of such information to reduce the count of misses in the operational scheme. In this section, we develop and evaluate an imager-assisted cloud-detection scheme, where this particular skill of the imager-based scheme is incorporated into the departure-based scheme Implementation As outlined in section 2, the departure-based cloud detection scheme of ECMWF includes the capability to recognize a completely clear FOV in the quick-exit scenario. This only happens when the smoothed curve of background departures stays within pre-defined limits throughout the band of IASI channels. In the imager-assisted scheme, we set an additional requirement that the imager-based cloud flag must be clear for the quick-exit scenario to be allowed. By this additional requirement we aim to improve the detection of clouds that are potentially associated with a compensating background error. In the vast majority of cases, the operation of the imagerassisted scheme does not differ from that of the departure-based one. In a global sample, approximately 9% of cases go through either the cold- or warm-start scenario of the departure-based scheme, in which case the imager-based cloud flag will not be used. The quick-exit scenario is chosen in the remaining 1% of all cases. Broadly speaking, the imager-based flag is cloudy in half of those cases that enter the quick-exit scenario, implying that using the imager data in this implementation alters the output of the cloud detection in 5% of all cases. When a quick exit is prevented only because of the cloudy imager-based flag, the imager-assisted scheme will revert to the cold-start scenario. As explained in section 2, the search for the lowest clear channel will be started from the global minimum of the smoothed curve of vertically ranked background departures. In the extremes, this might mean flagging either all channels or no channels as cloudy. It is, however, expected that the modification is the most effective on lower tropospheric and window channels, as these are the most likely channels to be ranked below the global minimum of the smoothed curve Performance assessment A set of experiments is run to investigate the performance of the modified cloud detection. The experiments are based on the global NWP system of ECMWF (IFS) run at resolution T511L91. The first experiment (hereafter winter experiment) is based on IFS version 37r3, which became operational on 17 November 211; this experiment is run over the two-month period 21 November January 212. The second experiment (hereafter summer experiment) is based on IFS version 38r1, which became operational on 19 June 212; this experiment covers the time period 1 July September 212. Both experiments are based on the full operational set-up in terms of conventional and non-conventional data usage. Each experiment run is verified against a control run, which differs from the experiment run only in terms of cloud detection for IASI data. The introduction of the imager-based scheme as part of the cloud detection is not found to increase the computing time needed to execute the full analysis in the ECMWF 4D-Var system significantly Impact on observation departure statistics As a first diagnostic, the global counts of actively assimilated data in the control and experiment runs of the winter experiment are shown in Figure 6(a) as a function of the peak pressure of channel temperature Jacobians (which are computed in a tropical reference profile). As expected, the modified run (circles) contains fewer active data than the control run (crosses). The effect is most notable on the low-peaking window channels, where the active data count is decreased by up to 3%. The higher the peak pressure, the smaller the effect and it is unidentifiable in channels peaking higher than 3 hpa. An exception to this behaviour is found in channels that are strongly sensitive to ozone absorption. Despite having temperature Jacobians that peak in the stratosphere, these channels are also strongly sensitive to surface emission and therefore affected by the modification to the cloud-detection scheme. Figure 6(b) shows the mean observation minus analysis departure for IASI channels in the winter experiment. The statistics are computed over actively assimilated data only. The large increase in the mean departure for window channels is c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)

8 Imager-Assisted Cloud Detection for Radiance Assimilation 2349 (a) Peak pressure [hpa] Control Modified [millions] (b) Peak pressure [hpa] Mean analysis departure [K] Figure 6. (a) of active data and (b) mean observation minus analysis departure for IASI as a function of channel peak pressure in the (northern hemispheric) winter experiment. Crosses and circles refer to the control and imager-assisted runs, respectively. Values in the ordinate are offset by 1 hpa in the control run in order to improve readability. mainly a sampling issue. Because of the nature of the clouddetection scheme, the rejected data constitute an asymmetric distribution, such that observations on the colder side of the background are more likely to be affected than those on the warmer side. Therefore, the actively assimilated population is shifted towards warmer data, leading to increased mean departures. Independent observation departure statistics of collocated sounders (those instruments sharing the satellite platform with IASI) suggest an improved analysis fit when the imager-assisted cloud detection is applied. The mean analysis departure in the control and experiment runs (again for the winter experiment) is shown in Figure 7 for the Advanced Microwave Sounding Unit A (AMSU-A), Microwave Humidity Sounder (MHS) and High-resolution Infrared Radiation Sounder (HIRS), all on board the Metop-A satellite. The mean departure is shown separately for the northern and southern extratropics ((a) (c), (g) (i)) and the Tropics ((d) (f)). No difference between the runs is found in the mean analysis departure of AMSU-A ((a), (d), (g)). For HIRS ((c), (f), (i)), the modified run (dashed line) shows decreased mean departure compared with the control run (solid line) in channels 7 and 9 in the northern extratropics, suggesting warming of the lower troposphere, potentially together with a reduction in stratospheric ozone concentration. Outside the northern extratropics, the impact on the mean departure is small and most easily attributable to changes in the model s humidity distribution. The decreased departure in HIRS channels 11 and 12 suggests drying of the tropical troposphere, which is supported by the decreased mean departure in channel 4 of MHS ((b), (e), (h)) as well. The MHS channel 3 suggests slight upper tropospheric moistening in the southern extratropics but this is not confirmed by the humidity-sensitive channels of HIRS. Figure 8 shows the mean analysis departure for the collocated sounders in the summer experiment. Again, there is no identifiable impact on the statistics of AMSU-A. HIRS suggests lower tropospheric warming (channel 7) in the Tropics and southern extratropics and drying (channel 11) in the Tropics only. Additionally, a reduced concentration of stratospheric ozone (channel 9) is found in the southern extratropics. The drying of the tropical lower troposphere is supported by MHS (channels 4 and 5). Drying is also suggested by MHS in the upper troposphere of the southern extratropics. We can summarize the outcomes from Figures 7 and 8 by concluding that the analysis departure data of collocated sounders consistently suggest drying of the tropical troposphere, together with lower tropospheric warming and reduction of stratospheric ozone in the winter hemisphere Impact on NWP analyses and forecasts The analysis departure data discussed above are qualitatively consistent with mean analysis differences between the experiment and control runs. In the winter experiment, the warming of the lower troposphere is restricted to the Arctic Ocean and it is (a) (b) (c) (d) (e) (f) (g) (h) (i) Figure 7. Mean observation minus analysis departure for instruments on board the Metop-A satellite in the control (solid) and imager-assisted (dashed) runs of the (northern hemispheric) winter experiment. Panels on the left ((a), (d), (g)), middle ((b), (e), (h)) and right ((c), (f), (i)) refer to AMSU-A, MHS and HIRS, respectively. Top ((a) (c)), middle ((d) (f)) and bottom ((g) (i)) panels refer to northern extratropics, Tropics and southern extratropics, respectively. c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)

9 235 R. Eresmaa (a) (b) (c) (d) (e) (f) (g) (h) (i) Figure 8. As Figure 7, but for the summer experiment. Figure hpa mean relative topography difference between imager-assisted and control runs in the winter experiment. The contour interval is.4 m and negative contours are dashed. strongest in the boundary layer below the 85 hpa pressure level. The geographical distribution of the warming is illustrated in Figure 9 in terms of the impact on geopotential height difference (i.e. relative topography) between 7 and 1 hpa. The strong impact over the Arctic Ocean does not spread over land and coincides well with typical sea ice coverage of the season. Outside the Arctic Ocean, the impact is fragmented. Some isolated warmings are found over oceans in the northern midlatitudes and southern high latitudes, as well as over land in the Tropics. We believe that the imager-assisted scheme improves the cloud detection in the presence of large surface temperature errors that happen to compensate for the radiative effect of cloud. The distribution of the warming supports this hypothesis, as surface temperature errors are generally larger over sea ice than over open sea. On the other hand, the hypothesis described above is inconsistent with indications of slight cooling found over the tropical Indian and Pacific Oceans. The origin of the tropical cooling is not clear. As the imager-assisted scheme can only reject data that would otherwise be assimilated (and not vice versa), an explanation for the cooling can be formulated by assuming a systematic cold background error. Radiance data assimilated as clear in the control run have considerable potential to introduce warm analysis increments to correct the cold errors, but switching to the imager-assisted cloud detection will reduce this potential. The tropical cooling would therefore follow from the lack of sufficient warm analysis increments in the region. In the summer experiment (not shown), the lower tropospheric temperature impact is less pronounced but includes a similar warming in sea-ice covered regions off the Antarctic coast. The impact on the mean analysis of water vapour and ozone concentrations in the winter experiment is shown in Figure 1 for zonally averaged total column values. The mean total column value for each absorbing gas is shown by a dashed line for the control run, whereas solid lines show the difference between c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)

10 Imager-Assisted Cloud Detection for Radiance Assimilation 2351 (a) Column water vapour [kg m -2 ] Change [kg m -2 ] (b) Column ozone [kg m -2 ] Change [g m -2 ] Latitude [deg] Latitude [deg] Figure 1. (a) Zonal mean total column water vapour and (b) ozone (dashed lines) and its change (solid lines), when the departure-based scheme is replaced by the imager-assisted scheme in the winter experiment. (a) (b) (c) (d) Figure 11. Control-normalized forecast anomaly correlation in the winter experiment. Positive values imply better scores in the imager-assisted run. Vertical bars indicate 95% confidence intervals. 5 hpa geopotential height in (a) northern and (b) southern extratropics and (c) 5 and (d) 85 hpa geopotential height in northern high latitudes are shown. the experiment and control runs. In the case of water vapour (panel (a)), the drying effect is confirmed between approximately latitudes 4 Nand5 S, with the exception that slight moistening is found around 15 S. With respect to actual total column water vapour values, the drying effect is very small and accounts approximately for.1% only. For ozone (panel (b)), the reduction is globally more uniform (compared with water vapour), but the relative impact is again of the order of.1% only. The strongest reduction occurs near the North Pole. The impact in the summer experiment (not shown) is nearly perfectly symmetrical to that in the winter experiment. However, the winter-hemispheric reduction of ozone in high latitudes is stronger in the summer experiment (i.e. in southern high latitudes) than in the winter experiment. Finally, making use of the imager data in cloud detection results in a slight improvement in statistical forecast scores of geopotential and lower tropospheric temperature. This is demonstrated in Figure 11 for the winter experiment. Solid lines show the relative improvement in anomaly correlation and vertical bars are added to indicate 95% confidence intervals for the scores. In the southern extratropics (panel (b)), the forecast impact on headline scores (5 hpa geopotential height anomaly correlation) is neutral throughout the ten-day forecast range. A positive (but statistically non-significant) headline forecast impact is found beyond day six in the northern extratopics (panel (a)). Scores computed over northern high latitudes only are consistently (but still statistically non-significantly) positive for forecast ranges from 2 1 days. In the northern high latitudes, the impact is slightly larger at 85 hpa (panel (d)) than at 5 hpa (panel (c)). In the summer experiment (not shown) the forecast impact is less pronounced, but a similar positive impact is also found in this case beyond day six in the northern extratropics. 5. Conclusions We have developed a method to make use of collocated AVHRR cluster information to assist the cloud detection of IASI radiances c 213 Royal Meteorological Society Q. J. R. Meteorol. Soc. 14: (214)