Median Filter Effects on Radar Wind Profiler Spectral Noise Statistics

Transcription

1 288 J O U R N A L O F A T M O S P H E R I C A N D O C E A N I C T E C H N O L O G Y VOLUME 31 Median Filter Effects on Radar Wind Profiler Spectral Noise Statistics TIMOTHY L. WILFONG, ELIAS M. LAU, BOB L. WEBER, DAVID A. MERRITT, AND SCOTT A. MCLAUGHLIN DeTect Inc., Longmont, Colorado (Manuscript received 2 April 214, in final form 19 May 214) ABSTRACT Radar wind profiler (RWP) systems observe radar returns from refractive index fluctuations due to clear-air turbulence. The Doppler spectra used to compute the moments of the returned signal always include noise from various sources and may contain multiple signals. A critical first step in detecting signals is the objective determination of the noise level in each spectrum. Several spectra may be averaged to improve signal detection. In addition to or instead of a mean, a median may be applied to successive spectra in order to reject transient interference. Monte Carlo simulations were used to examine the effects of the median versus the mean on the objective noise determination. When a median is used, it was found the noise statistics calculations must be slightly modified. 1. Introduction Radar wind profilers (RWP) sense the mean and turbulent motion of the clear air through Doppler shifts induced along several upward-looking beams. RWP signals are often contaminated by non-atmospheric signals. In early implementations, part of the contamination was due to an aliasing of higher-frequency signals into the clear-air portion of the spectrum due to the application of long boxcar-type averaging in the time domain. Today s computers, however, are able to process larger amounts of data at greatly increased speeds. Minimal time domain averaging and long fast Fourier transforms (8192 points or greater) can be done, allowing rejection of out-of-band interference. An effective technique for removing transient in-band interference is the application of a median instead of, or in addition to, a mean to successive spectra (Schumann et al. 1999; Wilfong et al. 1993). 2. Doppler spectrum characteristics As discussed in appendix A, the statistical spectral power distributions for both system noise and atmospheric return are theoretically exponential; however, Corresponding author address: Timothy L. Wilfong, DeTect Inc., 117 S. Sunset St., Ste. L, Longmont, CO tim.wilfong@detect-inc.com their mean values and their spectral distributions are generally different. The atmospheric signals are frequency band limited, while the noise is usually white or uniformly distributed in frequency. The detection of atmospheric signals depends upon the ability to quantify noise in the spectra. Individual spectra consist of large numbers of data points that can be used in statistical tests to objectively determine the noise level. These tests attempt to separate the signal(s) from the noise for the purpose of estimating the moments of the noise. The technique introduced by Hildebrand and Sekhon (1974, hereafter HS) is widely used to determine the noise level in Doppler spectra. Essentially, the HS technique compares statistical moments estimated from data with their expected values. Complex time series signals from both radar system noise and the atmospheric return are Gaussian random variables (Petitdidier et al. 1997), for which the power spectral densities (S n ) have an exponential probability density distribution. The exponential distribution has the property that the mean and the standard deviation are equal. Thus, Var(S n ) 5 hs n i 2, (1) where the angle brackets denote ensemble average and the variance on the left-hand side of (1) is obtained from an ensemble of realizations of the process. For white DOI: /JTECH-D Ó 214 American Meteorological Society

2 OCTOBER 214 W I L F O N G E T A L. 289 FIG. 1. Sample simulated noise-only power spectrum with NFFT noise, hs n i 2 can be estimated from one realization by averaging S n over the number of spectral points N. To reduce the variance, several spectra may be averaged and/or a running average performed across a spectrum. Then (1) becomes Var(S n ) 5 hs n i 2 /r, (2) where r is the product of the number of spectra averaged and the length of the running average applied to the mean spectrum. The recommended test from HS is to successively construct a new spectrum from the original, rejecting signal densities stronger than a decreasing value and computing the ratio R 5 P 2 /Qr, (3) where P is the mean of the new spectrum given by P 5 å n S n /N, (4) and Q is the variance of the new spectrum given by Q 5 å S 2n /N 2 P 2. (5) n For white noise, it is expected R will be unity; and, for spectra containing weather signals, R is expected to be,1. In practice, the averaged spectrum is sorted small to large. Then beginning with the smallest sorted values, N is initially set to some value, nominally 25% of the number of points in the fast Fourier transform (NFFT); and, R is repeatedly computed and tested, incrementing N each time until the threshold R 5 1is crossed. The noise threshold the largest noise value in the spectrum is then the previous point. It is important to note the r assumes the spectra have been averaged together. We now examine how the r needs to be modified if the spectra have been median filtered. a. Median versus mean properties As discussed previously, the HS noise calculation depends on the relationship between the mean and standard deviation about the mean in an exponential distribution: s mean 5 mean. (6) As discussed in appendix A, the theoretical value for a median and the standard deviation about the median is related to the mean by median 5 :693(mean), and (7) s median 5 1:46(mean). (8) To examine how the theoretical values are related to actual values when relatively small numbers are involved for the mean and median, several Monte Carlo

3 29 J O U R N A L O F A T M O S P H E R I C A N D O C E A N I C T E C H N O L O G Y VOLUME 31 FIG. 2. Comparison of the 1-point ensemble mean and median noise statistics for spectral-noise-only simulations using 8192-point spectra. (top) The spectral power density relationships; (bottom) the median/mean ratio statistics.

4 OCTOBER 214 W I L F O N G E T A L. 291 simulations were accomplished where the power spectra containing only white noise were generated using (e.g., see Zrnic 1975) S n 5 (2noise) ln(rand), (9) where noise is the average noise per spectral n,(1# n # NFFT), and Rand generates a uniform random variable between and 1. An ensemble of 1 realizations was generated for NFFT lengths of 8192, 124, and 256 points. Figure 1 shows a sample unaveraged 256-point spectrum. Each of the 1 realizations consisted of generating independent sets of spectra, where each set contained from 1 through 1 unaveraged spectra (NSPEC). The median and the mean were applied independently to each of these sets of spectra in order to examine the statistical relationships. The average noise power for all the simulations was set to 27.5 db ( linear). Figure 2 presents the results for the 8192-point spectra. The results for the 124- and 256- point spectra are completely analogous and are not shown. Note that for an even number of spectra, the median is calculated by taking the mean of the center two points after sorting. Thus, the median and mean of two spectra are exactly the same; and of course, the mean and median of a single spectrum are the same. There are a few important conclusions. First, while the median of many spectra approach the theoretical value of.693 times the mean, this relationship does not hold for just a few spectra, such as normally used in processing wind profiler spectra. Second, the theoretical relationship that the standard deviation of the median is 1.46 times the standard deviation of the mean is approximately true only for a median of 3, and then it rapidly approaches unity for odd values of the median. For even valued medians, the median standard deviations begin (for a median of 4) at approximately.97 times the mean standard deviation and then rapidly approach unity. Finally, and most importantly, the HS parameters must be modified if a median is used instead of or in conjunction with a mean to produce smoothed spectra. b. Computation of the noise threshold Regarding the computation of the noise threshold, (3) really compares the mean squared of the spectrum to the variance, which is adjusted for the number of spectra averaged to produce the mean (r). Thus, if a median is used, then R must be adjusted to account for the smaller value of the median (the P ) as well as the much smaller effect of the difference in the median variance (the Q ): P 2 R 5, (1) Qr(Median_Factor) NMed where the subscript NMed refers to the number of spectra used to form the median and the Median_Factor is given by (Median_Factor) NMed 5 (median/mean)2 NMed (s median /s mean ) 2. (11) NMed For a small number of spectra used to construct median filtered spectra, such as normally used in wind profiler signal processing (NMed, 3), the Median_ Factor must be determined empirically using the results of the Monte Carlo data shown graphically in Fig. 2. These data are also presented in tabular form in Tables B1 and B2. Consider, for example, a strategy where four spectra are averaged to form a mean and then a median is applied to three of these mean spectra. The Median_Factor from appendix B is.641 and 13 becomes 3. Conclusions P 2 R 5 Q (:641). (12) The median may be used in processing Doppler radar wind profiler spectra to reject intermittent interference and to improve detectability by reducing the variance in the spectral noise. However, when calculating the noise statistics using the HS method, we have shown the ratio used to determine the noise threshold must be adjusted to account for the unique properties of the median. Acknowledgments. The authors express their gratitude to Dr. R. Strauch for the numerous discussions and suggestions during the preparation of this manuscript. APPENDIX A Doppler Spectra Distribution Characteristics Signals from both radar system noise and the atmospheric return are Gaussian random variables (Petitdidier et al. 1997), for which the power spectral densities have an exponential probability density distribution,

5 292 J O U R N A L O F A T M O S P H E R I C A N D O C E A N I C T E C H N O L O G Y VOLUME 31 p(x) 5 1 m exp(2x/m), (A1) where x is any possible spectral value in the range # x, and where m is the mean power spectral density that is, m 5 ð xp(x) dx. The cumulative probability distribution P(x # x) 5 C(x) 5 ð x p(x ) dx exp(2x/m) (A2) (A3) gives the probability that the spectral power x is less than or equal to some value x. The cumulative probability approaches the value 1 in the limit as x/. The probability of x exceeding some value x is given by the complementary function P(x. x) C(x) 5 exp(2x/m). (A4) The probability that spectral power will be less than the median value M is P(x # M) 5 C(M) 5 ð M p(x ) dx 512 exp(2m/m) 5 1/2. (A5) The probability that the spectral power will be greater than the median value M is also one-half: P(x. M) C(M) 5 exp(2m/m) 5 1/2. Therefore, the median and mean are related by M 52m ln(1/2) 5 ln(2)m 5 :693m. (A6) (A7) NSPEC Using (A7) Median/ mean s 2 M 5 2m2 2 2Mm 1 M 2. (A11) s 2 M 5 m2 1 m 2 2 2[m ln(2)]m 1 [m ln(2)] 2, or (A12) s 2 M 5 1:94 15m2. TABLE B1. Odd median s. P Median std dev/ mean std dev Q Median (A13) The variance is s 2 5 ð (x 2 m) 2 p(x) dx 5 m 2. (A8) Therefore, the standard deviation about the mean is s 5 m. (A9) The variance about the median is ð s 2 M 5 (x 2 M) 2 p(x) dx, or (A1) Finally, Table B1 Table B2 s M 5 1:46m 5 1:46s. APPENDIX B The Median Factor Tables (A14)

6 OCTOBER 214 W I L F O N G E T A L. 293 NSPEC Median/ mean TABLE B2. Even median s. P Median std dev/ mean std dev Q Median REFERENCES Hildebrand, P. H., and R. S. Sekhon, 1974: Objective determination of the noise level in Doppler spectra. J. Appl. Meteor., 13, , doi:1.1175/152-45(1974)13,88: ODOTNL.2..CO;2. Petitdidier, M., A. Sy, A. Garrouste, and J. Delcourt, 1997: Statistical characteristics of the noise power spectral density in UHF and VHF wind profilers. Radio Sci., 32, , doi:1.129/97rs25. Schumann, R. S., G. E. Taylor, F. J. Merceret, and T. L. Wilfong, 1999: Performance characteristics of the Kennedy Space Center 5-MHz Doppler radar wind profiler using the median filter/first-guess data reduction algorithm. J. Atmos. Oceanic Technol., 16, , doi:1.1175/ (1999)16,532: PCOTKS.2..CO;2. Wilfong, T. L., S. A. Smith, and R. L. Creasey, 1993: High temporal resolution velocity estimates from a wind profiler. J. Spacecr. Rockets, 3, , doi:1.2514/ Zrnic, D. S., 1975: Simulation of weatherlike Doppler spectra and signals. J. Appl. Meteor., 14, , doi:1.1175/ (1975)14,619:SOWDSA.2..CO;2.