A principal component noise filter for high spectral resolution infrared measurements

Transcription

1 JOURNAL OF GEOPHYSICAL RESEARCH, VOL. 109,, doi: /2004jd004862, 2004 A principal component noise filter for high spectral resolution infrared measurements P. Antonelli, H. E. Revercomb, and L. A. Sromovsky Space Science Engineering Center, University of Wisconsin, Madison, Wisconsin, USA W. L. Smith NASA Langley Research Center, Hampton, Virginia, USA R. O. Knuteson, D. C. Tobin, R. K. Garcia, H. B. Howell, H.-L. Huang, and F. A. Best Space Science Engineering Center, University of Wisconsin, Madison, Wisconsin, USA Received 2 April 2004; revised 29 July 2004; accepted 3 August 2004; published 4 December [1] This paper describes the application of principal component analysis to reduce the random noise present in the hyperspectral infrared observations. Within a set of spectral observations the number of components needed to characterize the atmosphere is far less than the number of wavelengths observed, typically by a factor between 50 and 70. The higher-order components, which mainly serve to characterize noise, can be eliminated along with the noise that they characterize. The results obtained depend on the variability of the selected sets of observations and on specific instrument characteristics such as spectral resolution and noise statistics. For a set of 10,000 Fourier transform spectrometer (FTS) simulated spectra, whose standard deviation is about 10% of the mean, we were able to obtain noise reduction factors between 5 and 8. Results obtained from real FTS, with standard deviation of about 10% of the mean, indicated practical noise reduction between 5 and 6. To avoid loss of information in the presence of highly deviant observations, it is necessary to use a conservative number of principal components higher than the optimum to maximum noise reduction. However, even then, noise reduction factors of 4 are still achievable. INDEX TERMS: 6924 Radio Science: Interferometry; 6974 Radio Science: Signal processing; 6969 Radio Science: Remote sensing; 6999 Radio Science: General or miscellaneous; 6994 Radio Science: Instruments and techniques; KEYWORDS: noise, filter, hyperspectral Citation: Antonelli, P., H. E. Revercomb, L. A. Sromovsky, W. L. Smith, R. O. Knuteson, D. C. Tobin, R. K. Garcia, H. B. Howell, H.-L. Huang, and F. A. Best (2004), A principal component noise filter for high spectral resolution infrared measurements, J. Geophys. Res., 109,, doi: /2004jd Introduction [2] The next generation of infrared remote sensors will achieve very high spectral (.1.5 cm 1 ) and spatial resolutions (4 15 km). Fourier transform spectrometers like Infrared Atmospheric Sounding Interferometer (IASI) [Diebel et al., 1996], Cross-Track Infrared Sounder (CrIS) [Glumb et al., 2003], Geostationary Imaging Fourier Transform Spectrometer (GIFTS) [Smith et al., 2001], and Hyperspectral Environmental Sounder (HES) [Li et al., 2003], and grating spectrometers like the Atmospheric Infrared Sounder (AIRS) [Aumann and Miller, 1995; Rosenkranz and Staelin, 1994] represent the present and the future of the atmospheric remote sensing in the infrared part of the spectrum. The observing capabilities of these systems, in terms of accuracy and spatial resolution of the retrieved geophysical parameters, can be significantly improved by increasing the spectral resolutions and/or enhancing the Copyright 2004 by the American Geophysical Union /04/2004JD signal to noise ratios [Smith et al., 1979; Huang et al., 1992; Zhou et al., 2002]. [3] To improve retrieval accuracy, one can reduce noise by averaging the measurements over a number of adjacent instantaneous fields of view (I-FOV). This procedure, however, improves the signal to noise ratio in the spectral domain at the expense of the instrument spatial resolution, with significant degradation of the signal for non uniform scenes (for example in proximity to cloud edges or coastal regions). [4] Antonelli [2001] and Huang and Antonelli [2001], while investigating the use of principal component analysis (PCA) to compress high spectral resolution IR data, showed that a substantial improvement of the signal to noise ratio can be achieved by taking advantage of the redundancy present in the spectral domain. This paper extends their analysis and describes the implementation of a PCA-based noise filter (PNF) and its application to simulated and real data. The application to simulated data is intended to assess the accuracy of PNF and to compare it to the theoretical limit defined by the best linear estimation theory [Papoulis, 1of22

2 1965]. The application to real aircraft data (Scanning High- Resolution Interferometer Sounder (S-HIS) [Revercomb et al., 1988] and National Polar-orbiting Operational Environmental Satellite System (NPOESS) Aircraft Sounder Test bed-interferometer (NAST-I) [Smith et al., 1999; Cousins and Gazarick, 1999]) is intended to show that PNF is an efficient tool to estimate and reduce the random component of the instrument noise while preserving the information embedded in raw data, even for rarely occurring and/or highly deviant observations (outliers). [5] Because the noise reduction is achieved through a lossy compression procedure, we address the issue of information loss due to the application of PCA. Traditionally the lack of knowledge about the nature and magnitude of this information loss has been the main source of skepticism when using PCA. We address this issue by investigating in detail the impact of PNF on simulated and real data from both a statistical point of view and on a case-by-case basis. However, these studies are intended to address the issues related to the noise filtering of local (both in time and space) data. The investigation of noise filtering on global scale is beyond the scope of this paper (see section 6). [6] The problem of noise filtering and the application of PCA to solve it are introduced in section 2. Sections 3 and 4 describe the results obtained by applying PNF respectively to simulated and real observations. Section 5 summarizes the results and describes a probable scenario for the future work. Appendix A is dedicated to the description of the instruments used in the study, with particular emphasis on the aircraft instrument noise characteristics. 2. Noise Filter Problem [7] Let us assume that for a specific instrument, we have a set L obs ={~L obs,1, ~L obs,2,.., ~L obs,m } of observed radiance spectra ~L obs,i ={L obs,i (n 1 ), L obs,i (n 2 ),.., L obs,i (n N )} such that: L obs;i n j ¼ Latm;i n j þ hi n j ; ð1þ where ~L atm,i is the noise-free (or atmospheric) component of the signal and ~h i is the instrument noise of the ith observation. Hereafter the subscript i will be dropped when dealing with multiple observations, while the subscript j will be replaced by the ~:: symbol. Given equation (1), the filtering problem for an individual observation ~L obs,i (which may or may not belong to L obs ) consists in providing an estimate ~L est,i of the atmospheric signal ~L atm,i, that minimizes the estimation error: ~ est;i ¼ ~L est;i ~L atm;i : ð2þ The estimated noise-free signal is obtained by applying an operator F, derived from the observations in L obs which satisfy equation (1), such that: ~L est;i ¼ F ~L obs;i : ð3þ If no restrictions are imposed on F, the optimal solution to this problem is, in general, very difficult analytically. However, if the permissible operations on the data are linear, the distributions of both L atm and h are Gaussian, and the optimum is interpreted in the mean square sense, the problem is considerably simplified and a theoretical optimal solution can be determined by linear mean square estimation theory [Papoulis, 1965]. According to this theory the root of the expected mean square estimation error for an observation ~L obs,i, rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi e est;i ¼ E ~ est;i 2 ¼ E ~L atm;i ~L est;i 2 rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ E ~L atm;i F~L obs;i 2 ; ð4þ is minimum for F ¼ RRþ ð SÞ 1 ; ð5þ where E() is the expected value, R is the covariance matrix of the set of noise-free atmospheric signals L atm, and S is the covariance matrix of the instrument noise spectra h. Unfortunately equation (5) cannot be directly applied to real data as it requires the knowledge of the covariance matrices of the true (and unobservable) atmospheric and noise signals separately. Therefore, in this paper, we propose the use of PCA to solve the noise filtering problem for high spectral resolution observations and we limit the use the minimum mean square error theory to simulated data only, to derive a theoretical upper bound for the accuracy of linear filters. [8] When dealing with real data, ~ est,i is generally not available, therefore to asses the accuracy and efficiency of PNF we will use the noisy signal estimation error or reconstruction residual ~ rr,i defined as the difference between the estimated signal and the original noisy signal: ~ rr;i ¼ ~L est;i ~L obs;i : ð6þ This quantity accounts simultaneously for the atmospheric information loss and for the amount of noise that is filtered out. In fact it can be written as where ~ rr;i ¼ F~L atm;i ~L atm;i þ F~h i ~h i ¼ ~k atm;i þ ~k noise;i ; ð7þ ~k atm;i ¼ F~L atm;i ~L atm;i ; ð8þ is the atmospheric information loss, and ~k noise;i ¼ F~h i ~h i ¼ ~h rec;i ~h i ; ð9þ is the desired noise information loss (noise that is filtered out by F) where ~h rec;i ¼ F~h i ; ð10þ is the amount of noise that leaks through the filter, which we also refer to as reconstructed noise. Because the estimation error is a combination of the atmospheric information loss and of the reconstructed noise: ~ est;i ¼ ~L est;i ~L atm;i ¼ ~ rr;i þ ~h i ¼ ~k atm;i þ ~k noise;i þ ~h i ¼ ~k atm;i þ ~h rec;i : ð11þ 2of22

3 Hereafter we will focus more of our attention on ~h rec,i rather than ~k noise,i. Equations (8) and (10) provide two useful quantities that allow for a complete characterization of the impact of PNF on simulated hyperspectral data. Once their relative contributions to the reconstruction residuals ~ rr,i have been assessed with simulated data, the analysis of the real data results, based only on ~ rr,i, will be more effectively described Theoretical Basis of PCA [9] PCA is a multivariate analysis technique [Jolliffe, 1986] that was first introduced by Pearson [1901] and developed independently by Hotelling [1933] at the beginning of the 20th century. It is commonly used to reduce the dimensionality of observations characterized by a large number, N (spectral channels), of interdependent variables. The reduction is achieved by projecting the deviation of the observations L obs,i (n j ) from the mean hl obs (n j )i = 1 M MP k¼1 L obs,k(n j ) onto the set of N t (where the subscript t stands for truncated) orthogonal vectors, with N t < N, which accounts for most of the input data variance. Hence the problem of dimensionality reduction is reduced to finding a linear transformation U Nt N that maps the observations ~L obs,i into an N t -dimensional array of compressed (or reduced) observations ~w i : ~w i;j ¼ XN k¼1 T U k;j ~L obs;i ðn k Þ h~l obs ðn k Þi ; ð12þ with minimal loss of atmospheric information. Such transformation is obtained when the rows of U are the eigenvectors, also referred to as principal components (PCs), of the covariance matrix C: X M C k;j ¼ 1 L obs;i ðn k Þ hl obs ðn k Þi Lobs;i n j hlobs n j i ; M i¼1 k ¼ 1; ::; N; j ¼ 1; ::; N: ð13þ The first PC is the direction along which the variance of the input data L obs has its maximum. The second PC is the vector in the orthogonal (N t 1)-dimensional subspace complementary to the first PC, which explains most of the remaining variance and so on until the last PC is the direction of minimum variance. More precisely the kth PC derived from C is the normalized eigenvector corresponding to the eigenvalue l k of C, where the eigenvalues are ordered l 1 > l 2 >.. > l Nc [Deco and Obradovic, 1996]. The eigenvalue l k represents the variance of the output component ~w i explained by the corresponding eigenvector [Deco and Obradovic, 1996]. It can be easily shown that for the case N t = N, PCA linearly decorrelates the output variables ~w i, i.e., diagonalizes their covariance matrix. Hence PCA essentially performs a Singular Value Decomposition (SVD) of the covariance matrix C. Once U has been determined and ~L obs,i has been compressed into ~w i, the noise reduction is achieved by re-expanding the compressed data: X Nt L est;i n j ¼ ~w k Uk;j T þhl n j i ði ¼ 1; ::; NÞ: ð14þ k¼1 If the generic spectrum ~L obs,i in equations (12) and (14) belongs to the set ~L obs (used in equation (13)) the derived PCs are referred to as dependent PCs, otherwise they are considered independent. Hereafter, if not otherwise stated, we will assume that the observation ~L obs,i always belongs to the set L obs used to derive the dependent PCs. The subscript est (for estimated) will be used to indicate specifically the PCA estimated quantities, and, for the sake of simplicity, equations (12) and (14) will be represented in the more compact form: L est ¼ PL obs : ð15þ This equation obtained by dropping the subscript i, represents the matrix form of equation (3) where the filter is applied to the entire set of observations L obs rather than to a single one ~L i, and the linear operator F is replaced by the PCA operator P PCA-Based Noise Filter [10] The value of PCA processing of high spectral resolution infrared data is its ability to take advantage of spectral redundancy to discriminate between the atmospheric signal and the random uncorrelated Gaussian distributed component of the noise. To explain how PCA reduces the noise, let us assume that we have access to an ideal instrument characterized only by a random, spectrally uncorrelated noise of constant variance across the spectral region of interest. Under these assumptions the noise h is homogeneously distributed in a N-dimensional sphere and its variance along the basis vector is independent of the basis selection. This implies that, after projecting the input data L obs onto the PC space, if only N t N PCs are retained to accurately represent the matrix of atmospheric (or noisefree) signals L atm then the noise variance of the reconstructed spectra is significantly reduced by ratio of qffiffiffi N N t (section 2.3.1). [11] Therefore for a set of observations L obs, for which L atm is known, which is the case for simulated data, the application of PCA to the noise filtering problem reduces to finding the value of N t that minimizes e est, the root mean square of the atmospheric signal estimation error est (where e est is calculated from equation (4)). For real data, when L atm is unknown, the problem is more complicated. A widely accepted method to determine the optimal value N t of PCs [Jolliffe, 1986] requires the ratio L ¼ P Nt i¼1 l i P Nc i¼1 l i ; ð16þ between the variance explained by the first N t PCs and the total data variance, to be close to 1 (for example L 0.999). This heuristic method ensures that the retained PCs explain most of the raw signal variance. However, the value of L is set arbitrarily and more important the method does not take in consideration the trade-off between reconstructed noise and atmospheric information loss, and thus leads to excessive noise in the estimated signal. For this reason, in sections and 4.2 we will suggest two alternative methods based on the optimization of this tradeoff, which do not require the knowledge of L atm PNF Implementation Strategy [12] The implementation of the PNF algorithm, following the theory developed in sections 2.1 and 2.2, can be summarized by the following steps: (1) normalize each 3of22

4 observed spectrum by dividing it by the estimated noise equivalent radiance (this procedure is described in section 2.3.1); (2) generate the covariance matrix of the observations; (3) derive the principal components (eigenfunctions of the covariance matrix); (4) determine an optimal number, N t of PCs to be used; (5) project each spectrum onto the PCs; (6) estimate the noise free signal by retaining only the first N t PCs; (7) multiply by the estimate of the noise to return to radiance space. Using simulated data we found that two elements need special attention for an efficient implementation of the filter: data normalization (section 2.3.1) and the size of the observation set L obs, or training set, used to derive the PCs (section 2.3.2). It is worth noting that in our simulation studies the PCs are derived in a dependent mode and the optimal value of N t is determined simply by minimizing e est in equation (4) Noise Normalization [13] For high spectral resolution instruments the uncorrelated component of the instrument noise is not uniform in the wave number space. Spectral regions of especially high noise can cause PCA to selectively fit the noise in these regions. To avoid fitting the noise instead of signal variations, the spectra are noise normalized to make the noise approximately white, before applying PCA. This not only improves the ability of PCA to enhance the signal-to-noise ratio consistently throughout the spectrum, but it also facilitates a quantitative estimation of the noise reduction. Given the RMS of the instrument noise: rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 e h n j ¼ E hn j ; ð17þ where E() indicates the mean value over the number of samples (hereafter E() will indicate the mean over a finite number of samples). Noise normalization causes the noise to become uniform in the spectral domain, with the jth channel root mean square (RMS) equal to one: e h n j ¼ 1 j ¼ 1; ::; N: ð18þ Under these circumstances, by mapping an N dimensional space onto an N t dimensional space, PCA causes a reduction in the sum of the noise variances, e h 2 (n j ), from a precompression value: S 2 ¼ XN e 2 hj j¼1 to a postcompression value: es 2 ¼ XNt j¼1 e 2 hj ¼ XNt j¼1 ¼ N j ¼ 1; ::; N; ð19þ e 2 hj ¼ N t i ¼ 1; ::; N t ; ð20þ where ~e j 2 is the noise variance in the PCs space. This allows for a noise reduction factor (in the sense of total noise variance reduction) gðn t ; NÞ ¼ sffiffiffiffiffi rffiffiffiffiffi S 2 N ¼ : ð21þ es Number of Samples [14] For PNF to be efficient, the size M of the training set L has to be as large as possible, and in any case M > N. For N t small training sets (M N), the noise distribution is not isotropic and the presence of noise affects the orientation of the principal components, causing the higher-order PCs (large N t values) to explain more noise variance than the corresponding PCs derived from large data sets with (M N). Even if the mathematical links between the training sample size M, the reconstructed noise h rec, and the atmospheric information loss k atm is not provided in rigorous way, the evidence obtained with simulated data (section 3.4) suggests that the larger the training set, providing that the atmospheric variability does not change considerably with M, the smaller the impact of noise on the orientation of the PCs, and consequently the more efficient PNF in removing noise. 3. PNF Applied to Simulated Data [15] Applying PNF to simulated data facilitates four main objectives: (1) to quantify the atmospheric information loss k atm, and the reconstructed instrument noise h rec, as functions of N t (section 3.1); (2) to compare the accuracy of PNF to the theoretical limit provided by linear estimation theory (section 3.2); (3) to verify the effects of noise normalization (section 3.4); (4) to verify the importance of using large training sets to derive the PCs (section 3.4). [16] The simulated data set, used for these studies, has been generated by applying a S-HIS fast forward model [Strow et al., 1998] to a set of 10000, clear sky, radiosonde profiles collected from 1990 to 2000 in South Africa for the months July and October. Each radiosonde record is characterized by vertical profiles of temperature and water vapor. The ozone profiles, the surface temperatures and the surface emissivities, required by the forward model to generate synthetic radiances, have been generated from a climatological data set and/or using a regression algorithm [Zhou et al., 2002]. Figure 1 shows the mean (top) and the standard deviation (bottom) of the simulated observations. In addition to the radiances also the random component h of the instrument noise has been generated such that it is Gaussian distributed with standard deviation derived from real blackbody measurements (observed during the SAFARI field experiment on 7 September 2000), uncorrelated in the wave number space and scene independent (section A2) Noise Reduction and Information Loss [17] To fully understand the noise filtering properties of PCA we introduce two quantities: the rms value of est rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 e est n j ¼ E est n j ; ð22þ and the rms of h rec (asterisks in Figure 2) rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 e hrec n j ¼ E h rec n j ; ð23þ where the mean E() is taken over the total number of observations. The functional dependence of these quantities on N t, available only for simulated data, provide the explanation of how PNF works. [18] Figure 2 shows the spectral means of e est (n) and e h rec (n). The figure shows that the spectral mean he est i has a 4of22

5 Figure 1. set L obs. S-HIS simulated LW: (top) mean spectrum and (bottom) the standard deviation of the training sharp minimum for N t = 15, indicating that 15 is the optimal number of PCs to be used as it guarantees the minimal loss of atmospheric information. For fewer than 15 PCs the atmospheric signal estimation error he est i is larger than the reconstructed noise he h rec i, while for more than 15 PCs the two curves converge indicating that the signal information that is lost due to compression (i.e., signal reconstruction residuals), largely describes the instrument noise rather than the atmospheric state. However, the figure does not provide any insight on the spectral characteristics and the statistical properties of the reconstruction residuals. To investigate the spectral distribution of the estimation error, of the atmospheric information loss, and of the reconstructed noise we introduce the rms of the atmospheric information loss, k atm : e k atm n j ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 E k atm n j : ð24þ The upper plot in Figure 3 shows that e katm (n j ) calculated for N t = 15, is much smaller (about one order of magnitude) than the RMS of the instrument noise h obs : e h obs n j ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 E h obs n j : ð25þ Figure 2. S-HIS simulated LW: spectral mean of the rms of the reconstructed noise, he h rec i (asterisks) and the estimation error, he est i (diamonds). For fewer than 15 PCs, he est i is larger than he h rec i, indicating that the estimation error is characterized by a relevant loss of atmospheric information. For more than 15 PCs, the two curves converge indicating that the atmospheric information loss tends to become smaller and smaller as N t increases. 5of22

6 Figure 3. S-HIS simulated LW: (top) the spectral representation of e k atm (n) and (bottom) the region between 775 and 825 cm 1. The upper plot indicates that the RMS of atmospheric information loss is significantly smaller than the instrument noise (dotted curve). The lower plot shows that the RMS of the atmospheric information loss is correlated in the wave number space: the peaks are, in fact, aligned with the absorption lines of the normalized spectrum representing the mean of the training set. [19] In both equations E() is taken over the total number of observations. The presence of spectral lines magnified in the lower plot (by focusing on a smaller spectral region) indicates that the e k atm (n) is correlated in the wave number space. Figure 4 is similar to Figure 3, it shows the spectral dependence of e h rec (n) forn t = 15. As expected, the RMS of the noise that leaks through the filter appears to be correlated; if this were not the case the filter could have been indefinitely re-applied to further lower the noise. Figure 5 shows the correlation matrices for the atmospheric information loss (on the left) and for the reconstructed noise (on the right) for N t = 15. The large off-diagonal elements indicate strong correlation of the reconstructed noise in the CO 2 absorption band and in the window. The problem can be mitigated by increasing N t. Figure 6 shows that for N t = 30, when more uncorrelated noise leaks through the filter, the correlation decreases. This suggests that the optimal value of N t in terms of the minimization of the estimation error might not be optimal in terms of the statistical properties of the reconstructed noise. In this case, by increasing N t from 15 to 30 the expected noise reduction factor would change from about 7.5 to 5.5, but both the correlation of the atmospheric information loss and of the reconstructed noise would decrease PCA Versus MMSE [20] To provide an objective evaluation of the PNF performances, the e k atm(n) and e h rec (n) in Figures 3 and 4 are compared their theoretical optimal values, in the framework of linear filters, provided by the Linear Estimation Theory through equation (5) (section 2). The upper and middle plots in Figure 7 respectively compare e h rec (n) and e k atm (n) for the minimum mean square error (MMSE, equation (5)) algorithm (black) and for PNF (red). The plots have been obtained using N t = 15 PCs. The bottom plot compares e h rec (n) (red circles) and e k atm (n) (black diamonds) to the MMSE estimates (red dashed and black solid lines). This figure shows that for the optimal number N t of PCs, the accuracy of PNF approaches the theoretical limit achievable by linear filters. Most importantly it shows that the noise that leaks through linear filters is in both cases correlated in the spectral domain and that this correlation is not simply an artifact introduced PCA. The correlation of this noise is clearly associated with the spectral distribution of the atmospheric information: absorption lines tend to carry more information than window channels. The higher the information content for a single channel, the smaller the angle(s) between the channel axis and the one or more of the retained PCs, and consequently the higher the noise that leaks through the filter for that channel. The MMSE filter, in equation (5), has provided slightly better results then PCA, on simulated data. However, we did not succeed in obtaining acceptable results with MMSE on real data, as they could not characterize correctly the covariance matrices of the noise free signal (atmospheric signal) and of the instrument noise Noise Normalization [21] To avoid wasting principal components fitting the noise instead of signal variations, the spectra are divided by e h (n) (equation (17)). Noise normalization changes the structure of the PCs in the spectral regions where the noise 6of22

7 Figure 4. S-HIS simulated LW: (top) the spectral representation of e h rec (n) and (bottom) the region between 775 and 825 cm 1. The upper plot indicates that, in the rms sense, the reconstructed noise is significantly smaller than the original noise e h (n). The lower plot shows that e h rec (n), is correlated in the spectral domain: the peaks are, in fact, aligned with the absorption lines of the normalized spectrum representing the mean of the training set. is high. Figure 8 shows the first five PCs, obtained from S-HIS simulated, clear-sky data, for nonnormalized (on the left) and normalized (on the right) data. Large differences are common to all the PCs at the extremes of the band (n < 630 and n > 1000 cm 1 ) and in the middle (830 < n < 870 cm 1 ). These spectral regions are characterized by relatively high noise due respectively to the detector responsivity and to the interference of the detector cooler. As for higher-order PCs (associated with larger values of N t )the noise variance becomes comparable to the signal (deviation of the observations from the mean spectrum) variance, the differences between normalized and nonnormalized PCs are expected to become larger across the entire band. [22] To verify the benefit obtained through the normalization procedure, we compare the noise reduction factor, g(n t ), and the spectral mean of the atmospheric information loss, he k atm i, for normalized and nonnormalized data. The upper plot of Figure 9 compares the theoretical noise reduction factor g(n t ) derived from equation (21) (solid line) to the g obtained from the nonnormalized (triangles) and normalized (asterisks) data. The values of g for normalized and nonnormalized data approach the theoretical Figure 5. S-HIS simulated LW: the plots show the correlation (left) of the atmospheric information loss and (right) of the reconstructed noise for N t = 15. The high compression ratio introduces large correlation values of the reconstructed noise in the CO 2 band and in the window. 7of22

8 Figure 6. S-HIS simulated LW: the plots show the correlation (left) of the atmospheric information loss and (right) of the reconstructed noise for N t = 30. For larger values of N t the correlation values of the reconstructed noise in the CO 2 band and in the window tend to decrease. limit only for N t < 15, but the higher values of g for normalized data suggest that normalization increases the efficiency of PNF in filtering out the noise. The theoretical limit is reached only if the PCs are derived from noise free data (circles). When most of the atmospheric signal variance is explained (N t > 15) the noise variance becomes significant and the orientation of the subsequent PCs (15 < N t < 100) is affected by the nonhomogeneity of the noise distribution due to the finite size of the training set so that they are less efficient in removing the noise (section 3.4). The lower plot in Figure 9 shows that the he k atm i for normalized data (asterisks) is consistently smaller than the Figure 7. S-HIS simulated LW: (top) e h rec for the MMSE (black) and for PNF (red) and (middle) the comparison between e k atm for MMSE (black) and PNF (red). The plots have been obtained using N t = 15 PCs. (bottom) The PNF spectral means he h rec i (red circles) and he k atm i (black diamonds) to the MMSE estimates (red dashed and black solid lines). For an optimal number of PCs (15 for this case), PNF approaches the theoretical limit achievable by linear filters. 8of22

9 Figure 8. S-HIS simulated LW. First five PCs derived from the S-HIS simulated, clear sky, noisy data for the two cases: (left) nonnormalized and (right) normalized. Large differences are common to all the PCs for the spectral regions characterized by relatively high noise. As for higher-order PCs (associated with larger values of N t ) the noise variance becomes comparable to the signal variance, the differences between normalized and nonnormalized PCs are expected to become larger across the entire band. one for nonnormalized data. This suggests that normalization not only makes PNF more efficient in removing the noise but it also enhances the accuracy of the filter. Noise normalization has been preferred to signal normalization to ensure that the amount of noise (rather than signal) explained by every PC is as uniform as possible Training Set Size [23] To determine the impact of the training set size M on the PNF performances we investigated the relationship between the eigenvectors and eigenvalues of the covariance matrix of the observations for two different sizes of the training set (M N and M N) such that the smaller data set is a subset of the larger data set, and compared the noise reduction factor g and the spectral mean he k atm i for the two different cases. [24] Figure 10 displays the eigenvalues associated with the PCs derived from simulated S-HIS spectra for the two different values of M. In both cases the first few PCs (N t < 10) explain similar amounts of variance. For small sizes of the training set (M = 1000 spectra) the higher-order PCs (large N t values) explain an amount of variance significantly smaller than 1, while according to the theory (section 2.3.1), for noise-normalized data, each PCs is expected to explain at least the noise variance (equal to 1). Figure 10 shows also that, for large training sets, each eigenvector tends to explain the same amount of noise variance. The reason is that in the limit M! 1, the normalized noise tends to become homogeneously distributed in the N-dimensional hypersphere, and the presence of noise does not affect the orientation of the principal components. For small training sets (M = 1000), the noise distribution is less isotropic and the presence of noise does affect the orientation of the principal components. [25] The top plot in Figure 11 shows the theoretical noise reduction factor g(n t ) derived from equation (21) (solid line) and the ones obtained from small (triangles) and large (asterisks) training set size. The values of g in both cases approach the theoretical limit only for small numbers of N t, but the higher values of g obtained for the large data set (M = 10,000) indicate that when larger, in the sense of higher degree of redundancy, training set are used PNF is more efficient in filtering out the noise. The theoretical limit is reached only if the PCs are derived from noise free data (circles). This confirms that, for noisy data, when most of the atmospheric signal variance is 9of22

10 Figure 9. S-HIS simulated LW. (top) The comparison between the theoretical noise reduction factor g(n t ) (solid line), equation (21), obtained from nonnormalized (triangles) and normalized (asterisks) data. g approaches the theoretical limit only for small numbers of N t. Higher values of g for normalized data suggest that normalization increases the efficiency of PNF in filtering out the noise. The theoretical limit is reached only if the PCs are derived from noise free data (circles). (bottom) he k atm i for normalized data (asterisks) is consistently smaller than the one for nonnormalized data and suggests that normalization makes PNF more efficient. explained (N t > 15) the noise variance becomes significant. The orientation of the higher-order PCs (15 < N t < 100) is affected by the nonhomogeneity of the noise distribution due to the finite size of the training set. The filter for high values of N t is less efficient in removing the noise. The lower plot in Figure 11 shows that the spectral mean he k atm i for large values of M (asterisks) is consistently smaller than the one for smaller values of M (triangles). The PCs derived from larger training sets allow for a more accurate reconstruction of the atmospheric signal than the PCs derived from smaller training sets Conclusion on PNF Impact on Simulated Data [26] In summary, the experiments performed on simulated data show that, in the rms sense, both the atmospheric information loss, k atm, and the undesired reconstructed Figure 10. S-HIS simulated LW: eigenvalues of the covariance matrix C. For a training set of 10,000 spectra the higher-order PCs (large N t ) explain an amount of noise variance close to 1 while for a training set of only 1000 samples the higher-order PCs explain a negligible variance. 10 of 22

11 Figure 11. S-HIS simulated LW. (top) The theoretical noise reduction factor g(n t ) derived from equation (21) (solid line) and the ones obtained from small (triangles) and large (asterisks) training set size. g in both cases approaches the theoretical limit only for small numbers of N t. The higher values of g obtained for the large data set (M = 10,000) indicate that for highly redundant training set PNF is more efficient. The theoretical limit is reached only if the PCs are derived from noise free data (circles). (bottom) he k atm i for large values of M (asterisks) is consistently smaller than the one for smaller values of M (triangles). The PCs derived from larger training sets allow for more accurate reconstruction of the atmospheric signal than the PCs derived from smaller training sets. noise, h rec, approach the optimal values defined by the linear estimation theory. For the simulated data used in the study, both e k atm(n) and e h rec (n) are about 7 or 8 times smaller than the instrument noise, e h (n), but they both are correlated in the wave number space. As the correlation for the reconstructed noise decreases with the increasing of N t, it is recommended to use a conservative number of PCs, in general larger than the optimal value defined by the minimization of the estimation error. In addition it is worth emphasizing that these numbers depend on the instrument characteristics and on the variability of the data set under consideration. Preliminary investigation on the correct number of PCs to be used, and on the magnitude of the reconstruction residual, are recommended when using PNF. While the conclusions according to which the noise normalization and large sizes of the training set enhance the accuracy and efficiency of PNF, hold true independently of the particular data set under consideration. 4. PNF Applied to Real Data [27] This section describes the application of PNF to real observations collected by the prototypes of the future high spectral resolution infrared sounders (S-HIS, and NAST-I) in different field experiments and it is intended to: (1) show the impact of PNF on real observations; (2) investigate how PNF affects outliers, i.e., spectra far from the mean of the training set, and/or poorly represented in the training set (section 4.1.3); (3) show why dependent compression is important when dealing with outliers (section 4.1.4); (4) validate the results obtained in section 3.3 by showing the benefits of noise normalization (section 4.1.5); (5) investigate the possibility of using PNF to estimate the instrument noise, allowing for data normalization even when an a priori estimate of the instrument noise was not available (section 4.2) PNF on S-HIS Observations [28] The data set used for this case consists of about 15,000 spectra observed by S-HIS on 7 September 2000 during the SAFARI mission. SAFARI 2000 was an international regional science initiative that was conducted in southern Africa in August/September For the selected flight the instrument operated in nadir mode only and the observations were collected over land and over ocean, in both clear and cloudy (low clouds only) conditions. In addition some observations were collected over ground fires. Figure 12 shows the mean (top) and the standard deviation (bottom) of selected S-HIS observations. S-HIS data have been used to show the impact of PNF on real data and to address the problem of how outliers, i.e., spectra poorly represented in the data set, are affected by the noise filter How to Select the Optimal Number of PCs [29] When dealing with real data, the first problem to address is the determination of the optimal number of PCs to be used. As L atm is unknown the criterion based on the minimization of he est i is not applicable. Therefore we 11 of 22

12 Figure 12. S-HIS observed LW: (top) mean spectrum and (bottom) the standard deviation of the training set L obs. introduced an approach based on the minimization of the function, d(n t ), defined as: dðn t Þ ¼ 1 X N N j¼1 2 err 2 e h n j n j ; ð26þ between the root mean square instrument noise e h and the root mean square of the reconstruction error, rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 e rr n j ¼ E rr n j ; ð27þ where the expected value E() inside the square root is taken over the total number of observations. Experience with simulated data shows that the minimum he est i occurs when the reconstruction residuals ~ rr mostly represent the instrument noise ~h, i.e., when the spectral RMS reconstruction residuals approaches the spectral RMS of the instrument noise (e rr (n j ) e h (n j ) 8 n j ) with the constrain e rr (n j )< e h (n j ) 8 n j. Values of N t too large allow PNF to remove only a small portion of the noise (e rr (n j ) e h (n j ) 8 n j ), while values too small cause significant loss of atmospheric information (e rr (n j )>e h (n j ) 8 n j ) This approach is preferred every time a good estimate of the RMS of the instrument noise is available. It is worth emphasizing that, as it will be shown, this criterion provides an estimate of the optimal value of N t only in a statistical sense. In the following sections the optimal number N t will be selected by minimizing d in equation (26), however, as outliers may require different number of PCs, for these special cases the values of N t will be assigned on a case by case basis Filtering of Well Represented Spectra [30] To render visually the effect of PNF on real observations we show (Figure 13) 60 spectra collected over the Indian Ocean (from 1003:00 to 1005:00 UTC). The top plot shows the observations before filtering while the bottom plot shows the same observations after the application of PNF (with N t = 30). The selected FOVs are very homogeneous, under clear sky condition, and with minimal surface and atmospheric variations. The bottom plot shows how PNF reduces the variability of the observations due to the random component of the noise. [31] To verify that the effect of PCA on real data we applied PNF to two observations collected over the ocean for very similar adjacent FOVs in clear sky conditions. The observations were compared before and after filtering. This test was designed to verify that, for well represented cases, the information loss is mainly characterized by instrument noise and that the reconstruction residuals are not very sensitive to the number of PCs used by PNF. The results are shown in Figures 14 and 15. In Figure 14 the top plot shows an observation in the longwave (LW) region over the ocean, while the bottom plot shows the differences between the LW spectra observed for the two consecutive and largely overlapping FOVs. The differences are centered around zero and their magnitude is 6 times larger (in the spectral RMS sense) for unfiltered data (D U ) than for filtered data (D F ). For the two FOVs the correlated component of the instrument noise, due to the interferometer mirror jittering, has been removed from the observed unfiltered and filtered spectra before the comparison. The magnitude of the correlated component of the noise has been evaluated from the low spectral resolution differences of the observed spectrum and the average of 50 nearby homogeneous spectra. The small magnitude of D U compared to D F indicates that PNF reduces the uncorrelated component of the noise. In general, the correlated component of the noise is treated by PNF as atmospheric signal, therefore different techniques have to be used to deal with this component of the noise. Figure 15 shows the observed data, in the 12 of 22

13 Figure 13. S-HIS observed SW: clear sky scene over the ocean. (top) The unfiltered radiances (60 spectra observed between 1003:00 and 1005:00 UTC on 7 September 2000) in the SW band observed over the ocean in clear sky condition. (bottom) The same spectra after filtering (N t = 30). The uncorrelated component of the noise is reduced by PNF. Figure 14. S-HIS observed LW. (top) One filtered (N t = 15) spectrum observed over the ocean (7 September 2000, 1024:28 UTC). (bottom) The differences between two spectra observed for the two consecutive, almost completely overlapping, homogeneous FOVs (7 September 2000, 1024:27 and 1024:28 UTC). The differences before filtering (light) are about 6 times larger, in the spectral RMS sense, than after filtering (dark) spectra. 13 of 22

14 Figure 15. S-HIS observed SW: clear sky over the ocean. (top) The observed filtered (lower curve) and unfiltered (upper curve) spectrum observed over the ocean (7 September 2000, 1024:28 UTC), in the shortwave band. The comparison between (middle) ~ rr (N t = 10) and (bottom) ~ rr (N t = 45) is also shown. The plots indicate that the reconstruction residuals ~ rr for homogeneous, well represented, scenes are not very sensitive to the number of PCs used by the PNF. shortwave (SW) portion of the spectrum, for a single FOV over the ocean (left), and the corresponding SW reconstruction residuals ~ rr for 10 and 45 PCs (right). The residual are not very sensitive to the number of PCs because the spectra under consideration are very well represented in the data set used to derive the PCs. The SW have been used in the second part of the test (Figure 15) simply to verify that the conclusions hold true for different regions of the S-HIS spectrum Filtering of Poorly Represented Spectra [32] The observations highly deviant from the mean and/ or rarely occurring, commonly classified as outliers, often represent the most interesting cases. We verified that these observations, obtained from different instruments, and for a variety of special situations, are not irreversibly compromised by the lossy compression used by PNF. [33] For this study PNF was applied to observations collected in proximity to and over a fire. The distinctive feature under consideration is the Blue Spike [Heland and Schafer, 1997] in the SW region at about 2397 cm 1 (left image in Figure 16); the observations selected are not well represented in data set used to derive the principal components: among the 15,000 training spectra only about 80 show the blue spike feature, and these 80 spectra are relatively far from the mean training spectrum. The blue spike is characteristic of FOVs partially obstructed by hot (350 K) atmospheric gases heated by the fires, above colder surface temperature. The results shown in Figure 16 indicate that the reconstruction residuals for the FOV over the fire (0843:20 UTC) are more sensitive to the number N t of PCs. As expected, spectra labeled as outliers require more PCs than well represented spectra to be properly reconstructed. However, providing that the distinctive spectral features are represented in the PCs, for an adequate number N t 30 (section 3.1) they are properly filtered. The plots in Figure 17 show unfiltered (left) and filtered (right) radiances observed over the fire between 0843:12 and 0843:22 UTC. The filter, not only preserves the information about spectral lines due to CH 4 and other gases, but it also makes it easier to identify them. This effect is even more evident in Figure 18, which shows 50 spectra observed immediately after the fire overpass (between 0843:26 and 0844:03 UTC). These observations are characterized by blue spikes of smaller magnitude not easily detectable in the noisy observations (left plot) but clearly identifiable in the filtered spectra (right plot) Relevance of Dependent Filtering [34] The third test on S-HIS data was performed to verify the importance of training set representativeness when dealing with outliers. We filtered the same data using two 14 of 22

15 Figure 16. S-HIS observed SW. (top) The SW unfiltered and filtered radiances observed at 0840:20 UTC. (middle and bottom) The reconstruction residual ~ rr for the same observation for N t = 10 and N t = 45. The results indicate that, in presence of outliers, the reconstruction residuals ~ rr are very sensitive to the number of PCs and the optimal number of PCs, statistically determined, might introduce large correlated errors. different sets of PCs. The first set was derived from all the observations collected during the flight, including the ones collected in the proximity and over the fires and characterized by the presence of the blue spike (dependent PCs), and the second set was derived from all the observations except the ones (about 80) observed in the proximity of the fires (Independent PCs). Figure 19 compares the dependent PCs (left column) to the independent ones (right column): as Figure 17. S-HIS observed SW. A series of 9 (left) unfiltered and (right) filtered spectra collected consecutively over fires on 7 September 2000 at about 0843:20 UTC. The blue spike is not affected by the noise filter and, more important, some of the absorption lines (CH 4, NH 3, OH, etc..) between 2400 and 2410 cm 1 are actually more evident in the filtered data than in the unfiltered ones. 15 of 22

16 Figure 18. S-HIS observed SW. A series of 50 (left) unfiltered and (right) filtered spectra collected consecutively immediately after the fires (after 0843:25 UTC). The blue spike signal is about 20 times smaller than the peak value, and is not clearly identifiable in the unfiltered data, but it becomes visible in the filtered data along with some of the absorption lines (CH 4, NH 3, OH, etc.) between 2400 and 2410 cm 1. expected the independent PCs do not show any feature related to the blue spike while all the first five dependent PCs do show the spike. Figure 20 shows how the two different sets of PCs perform in filtering one of the spectra observed over the fires (at 0843:20 UTC) and characterized by the presence of the blue spike. The first 30 independent PCs (top plot) do not allow for an accurate reconstruction of the spectrum: the information about the blue spike is completely lost (it would take a number of PCs, N t > 300, to partially recover it) and the estimated signal is noisier, while using the first 30 dependent PCs, not only most of the noise is filtered out, but the information about the blue spike Figure 19. S-HIS observed SW. (left)dependent PCs (the training set includes the spectra observed over the fires): all the 5 highest-order PCs show the blue spike feature. (right)independent PCs (the training set that does not include any spectrum observed over the fires): the PCs do not show any feature similar to the blue spike. 16 of 22