Ultra-Faint Radio Frequency Interference in the Murchison Widefield Array

Transcription

1 Ultra-Faint Radio Frequency Interference in the Murchison Widefield Array Michael Wilensky March 2018 Abstract The mitigation of radio frequency interference (RFI) is important in precision radio astronomy endeavors. A method of differencing visibilities to reveal the underlying thermal noise and RFI is introduced. A means of exposing extremely faint RFI is constructed by averaging these visibility differences together. These methods are tested on both simulated data as well as measured, uncalibrated data. The results are compared to theoretical statistical predictions. 1 Introduction The Murchison Widefield Array (MWA) is an interferometric radio telescope located in the Western Australian outback. Interferometric telescopes work by correlating measurements of the electric field among pairs of receiving elements. In our case, each receiving element is a tile consisting of a 4x4 array of crossed dipoles. We refer to a pair of tiles as a baseline. The correlations from each baseline, known as visibilities, are reported on a per-frequency basis, for each polarization pairing, and are averaged over a length of time known as the integration time. The visibility data should essentially consist of three components: the sky signal, noise-like variations due to factors such as the temperature of the receiving elements, and radio frequency interference (RFI) from terrestrial sources. Mitigation of RFI is important in precision radio astronomy experiments such as those searching for the Hydrogen Epoch of Reionization (EoR). The extent of the EoR will be found by searching for 21-cm emission from neutral Hydrogen. The 21-cm emission line is due to a highly forbidden spin-flip transition in neutral Hydrogen. A strong presence of it indicates an abundance of neutral Hydrogen. Most Hydrogen today is ionized in stars, and does not emit this type of radiation. This desired signal is expected to be very weak. Long integrations of many observations will be required to average down the thermal noise. However, RFI will not necessarily average down in the way thermal noise will, and so the data must be cleaned of RFI beforehand in order to give the most precise limit possible. 1

2 Current MWA RFI flagging procedures implement AOFlagger on a baselineby-baseline basis. AOFlagger is effective, but there is some faint RFI which can escape its notice. This document focuses on techniques which combine information from different baselines and times in order to more readily expose ultra-faint RFI that may have been missed by the primary flagging procedures. 2 Visibility Differences Let us denote a visibility for a tile pairing, ij, a frequency, ν, and an integration time center, t, as V ij (ν, t), ignoring separate polarizations. Let us denote the integration time as t. We may form a temporally differenced visibility in the following way, V ij (ν, t) = V ij (ν, t + t) V ij (ν, t) (1) For an Earth-based observing station, if the integration time is significantly less than the time of significant baseline foreshortening relative to a source, then this visibility subtraction should cancel the sky signal and leave only noise variations and RFI which vary significantly over t. The thermal noise in each tile has real and imaginary components that are independently distributed according to zero-mean Gaussians of equal width. The visibilities are given by a cross-multiplication in the frequency domain of the filtered tile signals. Therefore the visibility noise at the finest time resolution is given by a distribution which corresponds to the product of two zero-mean Gaussian random variables. This distribution is analytically calculable according to the rules for forming a product of two independent random variables, and the result is a zeroth order modified Bessel function of the second kind. Consult the author for a detailed calculation. The visibilities are not reported at the finest correlator resolution since the data rate would be too vast. They are averaged in time up to the length of the integration time. All MWA data is averaged up to at least 0.5s. The integration time for data in this memo is 2s. Given a sample rate of Mhz and the parameters of the cascaded polyphase filter applied to each tile, it can be seen that an integration time of 2s implies samples are averaged together for each frequency in the reported visibilities. In light of the Central Limit Theorem, which is stated and discussed in section 3.2, this implies that the visibilities themselves have thermal noise with real and imaginary components independently distributed according to Gaussians of equal width, albeit at a different width than either of the corresponding tiles due to the aforementioned filtration, cross-multiplication, and averaging. Let us denote the standard deviation of these Gaussians which describe each component of the complex visibilities as σ ν. The subscript is written to indicate that the width of the distribution varies as a function of frequency in our physical considerations. It can be shown with relative ease that such a process would have amplitudes that are Rayleigh distributed. A Rayleigh random variable, X, 2

3 has a probability density function of the form f X (x; σ ν) = x 2 σ ν 2 e x /2σ 2 ν, x > 0 (2) The mean and variance of this distribution, which we will denote by µ ν and σν, 2 are readily calculated and are given by π µ ν = 2 σ ν (3) σν 2 = 4 π σ ν 2 (4) 2 These may of course be combined to give σ 2 ν = 4 π π µ2 ν (5) Given a set of N measurements at frequency ν, {X i,ν }, the maximum likelihood estimator for the Rayleigh distribution is given by σ ν 2 = 1 2N N Xi,ν 2 (6) The quantity described by equation 1 was formed for each sequential time pair at every frequency, baseline, and polarization for almost three thousand uncalibrated observations of length 112s. Two seconds at the beginning and 6s at the end of each observation were removed due to a pestilent effect which can occur quite frequently at the boundary between observations. This gives 51 visibility differences for each baseline, frequency, and polarization. This memo contains several notable cases. The amplitudes of the visibility differences for two different observations are binned in figures 1 and 2 below, with an accompanying distribution fit determined from the maximum likelihood estimator, binned in the same manner as the data. i=1 3

4 Figure 1: Here is an example of some visibility difference amplitude histograms. There are three histograms here. One corresponds to all of the data, regardless of flagging results. Another contains only data that was considered uncontaminated by the RFI (dubbed Unflagged since it was left unflagged after an examination by the flagger). For the third, a distribution fit for each frequency was generated using the supposedly uncontaminated data according to equation 6. The calculated histograms for each frequency were then summed together to generate the total calculated histogram. All three histograms nearly coincide, and there were no data outside the expected tails of the distribution. This is an extremely well-behaved data set. Figure 2a: Here is an example of some visibility difference amplitude histograms with significant RFI occupation. The blue histogram, representing all of the data, extends far beyond the expected tail. The orange histogram represents the data which passed through the flagger without incident. The difference between the blue and orange histograms indicate that AOFlagger managed to catch some of the bright RFI, but it appears that there were data left unflagged by the AOFlagger which greatly depart from the attempted Rayleigh fit. The size of departure indicates that these remaining unflagged data may be RFIlike, rather than an extension of the noise. Unflagged data whose amplitudes fall between the two black lines are shown in figure 2b. 4

5 Figure 2b: Here is a reverse indexing of the unflagged data in figure 2a which fell between the black lines. The colorbar indicates how many baselines were affected at a particular time and frequency for this polarization pair. Note that the coarse band edges, demarcated by the ticks on the horizontal axis, are always manually flagged due to anomalous behavior which is suspected to arise from quantization effects. Only the XX reverse index is shown because all of the data in question were XX polarized. This anomalous data occupied a bandwidth of about 6.5 Mhz for the entire observation. This is very RFI-like behavior, and in fact these frequencies correspond quite well to a known digital television station in Western Australia (DTV7). Occupations of these frequencies feature prominently in a significant fraction of our observations. 3 Probability Theory Background Here we discuss linear transformations of random variables as well as the central limit theorem (CLT). First, we will seek the appropriate transformation of the probability density function for a linear transformation of a random variable. Then, we will review the CLT as it applies to Rayleigh random variables. 3.1 Linear Transformation of Random Variables 1 Consider a random variable, X, whose mean and variance are finite. Let us denote the probability density function of X as f X (x) and its cumulative distribution function as F X (x). Consider the random variable, Y, defined by Y = ax + b where a and b are real constants. Note that F Y (y) = P r{y y} = P r{ax + b y} = { P r{x y b a }, a > 0 P r{x y b a }, a < 0 1 A version of the following derivation for more arbitrary transformations can be found at jwatkins/f-transform.pdf 5

6 This may also be written F Y (y) = { F X ( y b a ), a > 0 1 F X ( y b a ), a < 0 From this we may differentiate, employing the chain rule, to obtain the appropriate transformed probability density function { 1 f Y (y) = a f X( y b a ), a > 0 1 a f X( y b a ), a < 0 Or more succinctly, f Y (y) = 1 ( ) y b a f X a (7) 3.2 The Central Limit Theorem for Rayleigh Random Variables Let us define the sample mean for a sequence of independent and identically distributed random variables, {X 1, X 2,..., X N } S N = 1 N N i=1 X i The CLT may be stated in the following way Theorem 1 Suppose {X 1, X 2,..., X N } is a sequence of independent and identically distributed random variables with mean µ and finite variance σ 2. Then if N is sufficiently large, S N has approximately a normal distribution 2 N (µ, σ2 N ) In other words, the probability density function for the sample means should approach that of a Gaussian density f SN (s; µ, σ2 N N ) 2 2πσ 2 e N(s µ) /2σ 2 (8) In light of the discussion above, let us consider the random variable defined by Y = S N µ (9) µ where µ is an arbitrary constant. Combining equations 7 and 8, the density should approach N µ µ f Y (y) 2 [y ( µ µ e N 1)]2 /2σ 2 (10) 2π σ 2 See, for instance, Devore, Jay L. (2004). Probability and Statistics for Engineering and the Sciences (6th ed.). Thomson, Brooks/Cole. p

7 If the sample means belong to a Rayleigh random variable, then we can use equation 5 to write this as { N µ f Y (y) 2(4 π) µ exp Nπ µ 2 [ ( ) ] } 2 2(4 π)µ 2 y µ µ 1 (11) In the event that µ = µ, { } N f Y (y) 2(4 π) exp Nπ 2(4 π) y2 (12) We see that the density is delightfully expressed only in terms of mathematical constants and the number of elements that went into the sample mean! This means that Rayleigh random variables of different widths (i.e. frequencies in the physical case) may be considered in very nearly the same manner so long as a way of very closely calculating the mean for each frequency exists. One manner of approximating the mean would be to simply take a deeper sample mean (constructed from more samples) than the one in consideration. In this case, equations would not apply exactly, since this deeper sample mean would itself be distributed about the actual mean in the same manner as in equation 8, but with a smaller width due to the increased number of samples. To achieve the exact result, one would need to form the product distribution between the random variables S N and 1 µ and then offset that distribution to account for the subtraction. The degree to which this is necessary is essentially determined by the ratio of the widths of the two Gaussian distributions for S N and µ, since this describes how constant µ is, relative to S N. The exact result is grotesque. If curious, please consult the author for an unilluminating calculation. Figure 3 and the accompanying text should assuage any feelings of doubt. 4 The Incoherent Noise Spectrum Let us assume for the moment that the receiving tiles are operating all at the same temperature, and that this temperature does not significantly vary over the course of a single observation. Given the set of baselines, B, we consider the following quantity V ij (ν, t) B = ( 1 N 1 N ) 2 i=1 N V ij (ν, t) (13) which we refer to as the incoherent noise spectrum (INS), since, in the absence of RFI, it is an incoherent average of the thermal noise in the visibilities. See figure 4 for an example of a clean spectrum. The quantity above can also be considered when data has already been flagged, in which case the average will be taken only over those baselines which were not considered contaminated by the flagger. j>i 7

8 Assuming that the system temperature is relatively constant over the time of observation, a deeper sample mean can be created by averaging equation 13 over time. We denote the result as V ij (ν, t) B t. A mean-subtracted incoherent noise spectrum, as in equation 9, can then be constructed. We might denote this as ζ(ν, t) = V ij(ν, t) B V ij (ν, t) B t V ij (ν, t) B t (14) This mean subtracted quantity is very useful because it makes ultra-faint features in an otherwise featureless landscape much more obvious. It is also readily analyzed by the statistical results of section Simulated Results Rayleigh distributed noise was generated to form a data cube for 8128 baselines, 55 time pairs, and 384 fine frequency channels. This corresponds to one polarization pair s worth of visibility difference amplitudes at 80 khz resolution in a typical 112s 128-tile MWA observation with no times or frequencies removed. The data of each simulated frequency channel were drawn from Rayleigh distributions of differing widths. A set of sample means was formed by averaging the data across the set of baselines as in equation 13. Deeper sample means were constructed by averaging these means across the time axis. The meansubtracted spectrum was then formed. The expectation is that the first set of sample means ought to be distributed about their respective means according to equation 8, with N = 8128, while the deeper sample means ought to be distributed about their respective means according to the same density but with N = 8128(55). The ratio of the widths of these two densities is 55, and so it is reasonable to expect that equation 12 is useful. In order to test this, the mean-subtracted spectrum was binned and fit. See figure 3. To test the frequency independence, all these processed data were binned together. Two curves overlay the histogram. One (red) is a Gaussian fit constructed from the calculated mean and variance of the data, while the other (blue) is the theoretical curve from equation 12, having set N = One can see that the simulated data conforms to the theory quite nicely. The curves almost coincide, with the blue being slightly narrower than the red. Figure 3: Simulated mean-subtracted INS histogram 8

9 4.2 Results from Measurements All data in this section are uncalibrated. The measured data in this section here differ from the simulated data by the number of time pairs, since as mentioned in earlier, some commonly problematic times have been trimmed. An observation is thought to be clean when its mean-subtracted INS appears as uncorrelated noise distributed approximately according to the results of section 3.2. See figure 4. All of the raw INS in this document exhibt periodicity in frequency as a result of a cascaded polyphase filter. This produces 24 coarse channels of width 1.28 Mhz. The correlator outputs at 10 khz resolution, but these spectra were constructed from frequency-averaged data at 80 khz resolution. The hot centerlines in each coarse channel result from uneven frequency-averaging. Figure 4a: An example of an incoherent noise spectrum formed from uncalibrated measurements. Lacking structures other than those that result from the filter, this spectrum looks RFI-free. Figure 4b: Here is a mean-subtracted incoherent noise spectrum from the same data as in 4a. The fractional deviation from the deeper (temporal) mean reveals noisy deviations with no obvious structure whatsoever in frequency or time. There is no discernible evidence of RFI in this observation even in the meansubtracted spectrum. 9

10 Figure 4c: A histogram with a Gaussian fit of the processed data in figure 4b. The results of section 3 appear to hold up quite well with measured data. An observation is thought to be contaminated by RFI when its meansubtracted INS has bright outliers and/or clusters in time and frequency. There are many cases where structure, though often quite faint, can be seen in the INS. These faint structures are often missed by preliminary flagging. The remaining figures illustrate different types of interference which vary in shape and brightness. Figure 5a: INS for a well-studied observation which is hypothesized to have digital TV interference (particularly DTV7). The entire observation was used for this spectrum, regardless of flagging. One type of RFI that shows up in our data is digital TV interference. These events usually show up in the spectra as bright, 6.5 Mhz wide, and lasting anywhere from a few seconds to the entire observation. Since this type of interference is so bright, a rudimentary flagging procedure involving just flagging the outlying data points in the mean-subtracted INS can successfully identify this type of interference. Curiously, sometimes this type of interference shows as highly polarized, usually in XX, but at other times can appear relatively evenly in all four polarizations. See figure 5. 10

11 Figure 5b: A mean-subtracted INS for the same observation as in figure 5a, constructed only from data which passed through the flagger unscathed. We can see that not all of the interference had been flagged, and that the residual interference in each of the four polarizations is about equal. The periodic frequency structure in this figure and others using flagged data is solely due to the manual flagging of the coarse band edges, which are ill-behaved. Figure 5c: Here is a histogram of the data in all four polarizations of figure 5b. There is what appears to be a visible Gaussian peak in the data despite a fair number of outliers and a poor fit. 11

12 Figure 5d: Reverse indexing the outliers and masking them seems to block out the interference quite well, although some residual excesses are visible by eye. Since the temporal mean is changed when an outlier is flagged, the spectrum is recalculated after each flag is applied. Thus, the final set of outliers is determined in an iterative way. Figure 5e: Removing the outliers indeed yields a set of data which is approximately Gaussian. There is a slight pileup on the right-hand-side, possibly matching the residual excesses in figure 5d. 12

13 Another commonly seen interference in these spectra takes the form of faint, broadband streaks. They often only last a few seconds, but may repeat throughout the observation in a non-periodic way. One may also see this persist throughout a range of temporally neighbouring observations. These events are so faint that they often generate very few outliers, and so are not adequately flagged by simply looking for brighter samples. A method for measuring temporal correlations in patches of data will need to be introduced to catch these events. Figure 6a: An INS with very faint broadband streaks. It is difficult to see these by eye. In almost all instances of this type of interference, the preliminary flagging misses these faint structures. Streaks are just barely visible in YY. Figure 6b: A mean-subtracted version of the figure 6a. The streaks are made much more obvious. While by themselves the YY streaks may have been questionable, the temporal correlation of YY events with the XX and crosspolarization streaks makes a much more positive indication of interference. 13

14 Figure 6c: The histograms for the mean-subtracted INS in figure 6b. We can see that even these faint streaks can cause the data to vary wildly from Gaussianity. Notice that the faint YY streaks do not strongly register as outliers in this histogram. The only way to flag this streak from these data will be to intentionally search for its shape. Some observations appear to have a clean INS by eye, but the mean-subtracted INS tells a different story. The figures below truly demonstrate the sensitivity of the mean-subtracted INS. Events such as those below are common enough that the widespread contamination of observations with ultra-faint RFI ought to be investigated. Relative occupancy levels at this sensitivity will determine future directions. Figure 7a: An INS which appears totally clean by eye. 14

15 Figure 7b: A mean-subtracted INS corresponding to figure 7a. While it does appear mostly noisy, we can see a very short broadband event during the observation in both XX and YY. XY and YX are not shown. They appeared clean. 5 Conclusion and Future Directions Whether a detailed flagger constructed from these techniques that precisely and accurately removes RFI is worth implementing remains to be seen. Some efforts have been made at this, and have revealed that where there is RFI, there is oftentimes fainter RFI. Depending on faint RFI occupancy, it may be worth flagging observations as a whole when enough faint RFI is present, rather than attempting to surgically excise widespread contamination. One hypothesized flagger, planned to be developed and tested, is a match shape filter. Patches of data in the shape of common RFI signals in the meansubtracted INS will be averaged together, and the results of the averages will be analyzed essentially according to section 3.2. The aim is to hopefully isolate pileups in the histograms by narrowing their distributions according to the CLT. For instance, the histograms in figure 6c may transform from a Gaussian peak with a pileup into two distinct peaks. One peak would correspond to the thermal noise, while the other peak would correspond to the streaks. Of course, multiple shapes will be tested for each observation, and a logical combination of the flagging results will determine the final flags. Generally speaking, this is not the end of the story. Note that a single point in the INS corresponds to an average of visibility difference amplitudes over many baselines. Contaminated data points in the spectra inform us of problem times and frequencies in the observations, but give no baseline information. Ultimately, the original contaminated visibilities themselves must be flagged in order to enhance the EoR analysis pipeline. While in a few special cases it may be true that all of the baselines at a given time and frequency were contaminated by RFI, this is not generally true. The most ideal flagger would follow the match-shape filter technique with a more thorough investigation of individual baselines. 15