Statistics and Visualization of Radio Frequency Interference

Transcription

1 Statistics and Visualization of Radio Frequency Interference Ellie White October 2016 Abstract This paper gives a summary of the results of a radio frequency interference mitigation project. Radio frequency interference (RFI) is a problem radio astronomers are facing increasingly in recent years. The goals of this project were: 1) investigate the cost vs. benefits of real-time RFI flagging, 2) determine ways of statistically flagging RFI by doing noise simulations, and 3) to develop a visualization tool in order to allow astronomers to easily see the RFI in their data set. The outcomes of these goals are described here. 1 Introduction In order to carry out their research, astronomers depend on sensitive, accurate, and unobstructed measurements of the light from faint, distant celestial objects. Unfortunately, astronomers up and down the spectrum are facing increasing challenges from electromagnetic interference. In the optical wavelengths, this takes the form of light pollution; errant ambient light from cities and suburbs. The analogue to this in radio astronomy is radio frequency interference (RFI). RFI is any unwanted, obstructive signal in radio astronomers data, caused by narrow- and broad-band transmitters that emit radiation at frequencies which overlap with those used for radio astronomy [1]. It has become harder and harder to avoid these harmful emissions due to the increase in electronic technologies which emit at RF wavelengths both intentionally (e.g. cell phones), and unintentionally (power lines) 1. The growing prevalence of RFI is proving to be a challenge for many radio astronomy observatories, causing a need for coping strategies. Some such solutions are location-based; observatories seek to locate themselves in remote areas away from cities and heavily populated places where RFI is more common [1]. In special cases, the observatory may set up, or locate itself within, a radio quiet zone. A current example of this is the U.S. National Radio Quiet Zone (NRQZ) which encompasses a 13,000 square mile area spanning parts of Virginia and West Virginia. It was put in place to protect the radio sky for the Green Bank Observatory (formerly the Green Bank National Radio Astronomy Observatory), and also for the (now defunct) U.S. Naval Facility in Sugar Grove, WV. The NRQZ imposes regulations on RF transmitters within its perimeter to ensure that they interfere minimally with the observing that is being done by the GBO and by the formerly Navy-operated NSA facility in Sugar Grove 2. However, even in remote places or radio quiet zones, RFI cannot be totally eliminated - emissions from sources such as satellites, lightning, and the occasional stray digital camera continue to cause interference. Therefore, radio astronomers must have ways to eliminate RFI once it has entered their receivers. Several methods of removing RFI from data have been implemented, both offline and real-time mitigation. In section 2, the reasons for attempting real-time RFI mitigation will be outlined in order to explain the purpose of this project. A description of the statistical analysis that was done in order to characterize the RFI in the data stream is given in section 3. Section 4 provides a description of the development of the visualization tool (RFIND) used to display the RFI data. Section 5 gives an overview of the results and features of the RFIND tool. In conclusion, the results of this project are discussed in section 6. 1 Radio Frequency Interference National Radio Quiet Zone

2 Figure 1: The spike on the right hand side (from MHz) is an example of RFI, possibly caused by the Military Satellite Communications system, MUOS. 2 Real-Time Flagging The purpose of this project was to experiment with ways to flag radio frequency interference in data streaming from GUPPI and (eventually) VEGAS in real time, and to visually display the flagged data. There were a few motivations for investigating real-time RFI mitigation; one is that post-correlation processing is quite a bit time consuming than real-time flagging [2]. Another issue is the fact that during offline processing, data is averaged over second-long time intervals. Thus, relatively large chunks of data may be ruined if they contain any RFI, even if the RFI only lasts for a fraction of the total time interval. Figure 1 shows an example of a 1 second piece of data containing RFI (possibly from the Military Satellite Communications system, MUOS). Although this particular data can be salvaged by offline processing, it clearly shows how the intensity of the RFI poorly affects the average value, bringing it artificially higher. Flagging RFI in real time provides a solution to this problem. By implementing algorithms which delete short time intervals containing data displaying the properties of RFI, the rest of the data in the sample can be salvaged. Keith Omogrosso of Oregon Tech developed the RFI Characterization on GUPPI (RCOG) program, which will eventually be capable of flagging and removing very short time slices ( 10 ms) of RFI in the data stream in real time [3]. Data is flagged as RFI based on statistical parameters, with the thresholds determined partially by simulation. 2

3 3 Calculation of Flagging Statistics Most astronomical and natural signals display the properties of Additive White Gaussian Noise (AWGN). AWGN is, as the name suggests, a type of random, uniform-across-frequency noise, with values that are normally distributed around a mean of zero 3. RFI displays statistical characteristics which are usually clearly distinguishable from those of AWGN. In order to determine what typical (good) data should look like in the absence of RFI, Python simulations of AWGN were conducted. To accomplish this, statsdemo.py was written. In this program, two NumPy random normal (Gaussian) arrays - xi and xq - were generated to represent the in-phase and quadrature components of the X-polarization voltage signal. After taking the power of these arrays and adding them, statistical analysis was done on trials. The probability distribution of the power values fit a chi-squared distribution with two degrees of freedom. With this in mind, it was decided to use a few different methods of calculating the flag statistics; the two methods were labelled Flag 1 and Flag 2. Figure 2: This plot shows the distribution of the results of the Flag 1 calculations. Flag 1 was calculated by taking the mean and the root mean squared (rms) of the power values for each of the trials, then calculating (rms/mean) 1 (1) for each trial, to get the resulting flag values. Since for a chi-squared distribution the mean is usually equal to the rms, Equation 1 should normally equal to zero or near zero for the Gaussian values of AWGN 4. 3 Additive white Gaussian noise white Gaussian noise. 4 Chi-squared distribution distribution 3

4 To test if this was true in practice, the flag values were plotted in a histogram. As is clear from Figure 2, the values were indeed centered around a mean of zero. The Flag 1 values appeared to be approximately normally distributed around this mean, so it was assumed that for any value greater than ±3σ from the mean (in this case, above +0.1 or below 0.1), the data would be flagged as containing RFI. Figure 3: This plot shows the distribution of the results of the Flag 2 calculations. The second flag value (Flag 2) was calculated by a slightly different method. This time kurtosis (a measure of the tailedness of a distribution) was used. For a chi-squared distribution with 2 degrees of freedom, the expected excess kurtosis is 6, so (excess kurtosis/6) 1 (2) was calculated, and was expected to equal around zero in most cases. Again, the resulting values of calculated trials were plotted on a histogram. This time, for whatever reason the resulting distribution, shown in Figure 3, was clearly not Gaussian and seemed to be somewhat positively skewed. However, it did appear that percent (3σ) of the values fell between and 3.3, so any values outside that range may be considered RFI. To obtain the statistical values of the input data, as mentioned in section 2 Keith Omogrosso developed the RCOG tool. This program reads in stored data from the GUPPI backend and uses GPU code to create the spectrogram (by performing Fast Fourier Transforms) and calculate the mean, rms, skew, kurtosis, min, max, and variance of this data. RCOG demonstrated with stored GUPPI data that GPU RFI characterization is a fast and accurate, and thus a practical solution. See [3] for more information. The spectrogram and statistics from RCOG were streamed to the Python visualization program RFIND, which is described in the following section. 4

5 4 Visualization Tool In addition to calculating the statistics necessary to flag the RFI, part of this project s scope was to develop a tool which would visually represent the flagged data in a readable and user-friendly manner. The desired outcome was an updating spectrogram plot which would emphasize the RFI present in the data map with color (like a weather map) and would have a few other helpful interactive features as well. A few different options were considered for developing this tool. One possibility was GNURadio, an open-source toolkit which provides pre-made radio signal processing blocks in an intuitive and user-friendly environment. It seemed like the most helpful choice due to the fact that it included a spectrogram waterfall display sink and a time raster display sink, both of which seemed like they would possibly do what was needed with some slight tweaking. Unfortunately, as it turned out, the time raster plot didn t work at all, and the waterfall plot was missing features and did not quite provide what was needed. The GNURadio mailing list provided little assistance with these problems, so with all of those issues in mind it was decided that GNURadio wasn t suitable for the purpose of this project. The next tools considered were Matplotlib and PyQtGraph, which are open-source Python GUI and graphics libraries. Although using a Python library would not give ready-made display tools, this would allow for more flexibility and seemed to be the next-simplest option available. It was decided early on that Matplotlib would not be the main package used; although it has more features and documentation than PyQtGraph, it is less streamlined and considerably slower to update plots. Since the spectrogram plot needed to update as near to real time as possible, PyQtGraph ended up as the main package that was used due to the speed of its updating plots. Matplotlib was used as a supplement for filling in gaps in PyQtGraph s functionality. Once this was determined, Python code was implemented to develop the tool: the Radio Frequency Interference Display (RFIND). 5 RFIND: Features and Results 5.1 Features The RFIND GUI, as seen in Figure 4, consists of four simultaneously updating plots, each of which assist in the data analysis process. The central feature is the spectrogram, streaming from RCOG, which shows the time (y-axis) and frequency (x-axis) of the data, and updates as the data is read in. Overlaid on this plot is a semi-translucent colormap, which labels high RFI values in red and low values in blue. Another feature of this plot is a moving crosshair, the x- and y- coordinates of which are given in the lower right corner of the GUI. In addition to this spectrogram plot, there are three other slightly smaller updating plots in RFIND. On the left of the spectrogram is the time series plot, and on the bottom is the bandpass. To the right is a plot showing the value of the flagging statistics which update alongside the spectrogram. This plot reads in the mean and rms values of the data from RCOG and performs the Flag 1 calculation. An additional nice built-in feature of all of these plots is that they may be zoomed in and panned, or autoscaled, depending on the observer s needs. 5.2 Results The end result was a fully functioning display GUI with some helpful features for data analysis and RFI flagging. As was the goal, it is a weather map type plot; while previous tools in use only displayed a very short time increment of perhaps only a second at a time, RFIND displays longer intervals. This helps give a more general feel for the current RFI climate, which could be helpful in terms of dynamic scheduling. For example, if project A depended upon having a relatively pristine RFI environment, but project B was less dependent upon that, then if the RFI is bad today as indicated by RFIND, project B would go ahead and take its observations and project A would wait for better conditions, etc. Of course there is room for improvement in some areas. One slight issue is that the spectrogram colormap seems to autoscale such that unremarkable levels of RFI/background noise may be labelled too brightly relative to their strength; with some minor tweaking this could likely be resolved. It would be useful 5

6 Figure 4: RFIND: Radio Frequency Interference Display eventually to be able to flip between Flag 1 and Flag 2 statistics based on user input, and also at some point a good addition would be to enable the display of multiple frequency channels at once in a tab-like format. 6 Conclusion In general, the main goals of the project were accomplished. RCOG efficiently and accurately produces the spectrogram and statistics from the stored GUPPI data faster than real-time. Since GUPPI is no longer the backend in use at Green Bank, RCOG will eventually be adapted to become RCOV, which will read in data from VEGAS, the new backend [3]. The goal of investigating flagging statistics through simulation was met as well. To test the accuracy of the estimates made based on the Python AWGN simulations, a sample set of GUPPI data containing known RFI was processed by RCOG and streamed to RFIND, where the flag calculations were performed. As it turned out, when the spectrogram displays RFI the corresponding flag plot data displays spikes exceeding the 3σ values determined in statsdemo.py, while normal (non-rfi) data falls within 3σ. So, in short, for the data used in this test the Python simulation results held. However, for greater accuracy it may be helpful to do more in-depth studies in the future. Finally, the goal of creating a real-time RFI display was achieved through the development of RFIND. As was desired, RFIND interactively displays the spectrogram and statistics from RCOG in a manner which allows the user to get a broader look at the present RFI conditions in a weather map type format. Although 6

7 there are certainly improvements which could be made (as outlined in section 5), the main objectives of RFIND were satisfied and it was demonstrated that PyQtGraph was a reasonably good tool for our purposes. References [1] RD Ekers and JF Bell. Radio frequency interference. In: arxiv preprint astro-ph/ (2000). [2] RA Series. Techniques for mitigation of radio frequency interference in radio astronomy. In: (2013). [3] Keith Omogrosso. Enabling GPUs to Mitigate RFI