PASSIVE array signal processing has gained considerable

IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002 2617 General Asymptotic Analysis of the Generalized Likelihood Ratio Test for a Gaussian Point Source Under Statistical or Spatial Mismodeling Jonathan Friedmann, Eran Fishler, Member, IEEE, and Hagit Messer, Fellow, IEEE Abstract This paper investigates the robustness of the generalized likelihood ratio test (GLRT) for a far-field Gaussian point source. Given measurements from an array of sensors, the performance of the GLRT under two types of common modeling errors is investigated. The first type is spatial mismodeling, which relates to errors due to multipath effects or errors in the assumed number of sources, i.e, deviation from the single point source assumption. The second type is statistical mismodeling, which relates to errors due to non-gaussianity in either the noise or the signal, i.e., deviation from the Gaussian assumption. It is shown that for some types of modeling errors, the detector s performance improves, and general conditions for such an improvement are found. Moreover, for both types of errors, the change in performance is analyzed and quantified. This analysis shows that for a distributed source with small spatial spreading, the degradation in performance is significant, whereas for a constant modulus point source, the performance improves. Simulations of various cases are shown to verify the analytical results. I. INTRODUCTION A. Motivation PASSIVE array signal processing has gained considerable interest during the last three decades. The problems under study concern the extraction of information from measurements using an array of sensors. The measurements are assumed to be a superposition of plane waves corrupted by noise. Given the observations of the sensor outputs, the objective is to estimate the unknown parameters associated with the wavefronts. aditionally, the unknown parameters are divided into three groups, and consequently, the literature is concerned with three types of problems: 1) Detection and determination of the number of sources: In this problem, we want to decide whether the measurements consist of noise only (the null hypothesis) or of a number of sources corrupted by noise (the alternate hypothesis). Once the null hypothesis is rejected, it is of interest to determine the exact number of sources impinging on the array (see, among many others, [1] [3]). Manuscript received November 27, 2001; revised June 24, 2002. The associate editor coordinating the review of this paper and approving it for publication was Prof. S. M. Jesus. J. Friedmann was with the Department of Electrical Engineering Systems, Tel Aviv University, Tel Aviv, Israel. He is now with Metalink, Yakum, Israel (e-mail: jonathanf@metalink.co.il). E. Fishler was with the Department of Electrical Engineering Systems, Tel Aviv University, Tel Aviv, Israel. He is now with the Department of Electrical Engineering, Princeton University, Princeton NJ, 08544 USA (e-mail: efishler@ee.princeton.edu). H. Messer is with the Department of Electrical Engineering Systems, Tel Aviv University, Tel Aviv, Israel (e-mail: messer@eng.tau.ac.il). Digital Object Identifier 10.1109/TSP.2002.804098. 2) Estimation of the sources spectral parameters: For example, there are the sources powers [4]. This problem is usually of interest for the communication community. 3) Estimation of the sources spatial parameters: For example, we have the sources bearing or range (see, among many others, [4] and [5]). In general, problems 2) and 3) are approached by assuming that the solution of problem 1) is known, i.e., the number of sources is known. However, in some cases, all three problems are solved simultaneously, e.g., [6]. The most common approach for estimating the number of sources (problem 1) is to use an information theoretic criteria-based estimator like the minimum description length (MDL) or the Akaike information criterion (AIC). In [6], it is suggested that we use the MDL estimator and combine the three problems 1) 3) into a single estimation problem, resolving for all unknown parameters simultaneously. In spectrum monitoring systems, which employ broadband scanning schemes, a vast number of narrowband channels (up to a few million) are scanned every second. In these systems, a fast and simple detection procedure is employed. If the null hypothesis noise only is rejected, then a more powerful and complicated algorithm for estimating the number of sources and their bearings is employed on the relevant channel. Other systems that process huge amounts of data in a short time make use of a similar solution: First, a simple and computationally efficient detection algorithm is used. Then, once a signal is detected, computationally demanding algorithms are used to estimate the number of sources and their unknown parameters, like range, bearing, and so on. One of the most popular algorithms used for the detection at the first step is the generalized likelihood ratio test (GLRT). This commonly used detector assumes that both the source (if present) and the additive noise are Gaussian. Although planned for a single Gaussian point source embedded in additive white Gaussian noise, this detector usually operates in a different environment, i.e., when there exists more than one source or when the additive noise is non-gaussian. Such a mismatch between the assumed model and the actual environment in which the detector operates may cause degradation in performance and gives rise to the question of the detector s robustness. In this paper, we provide a systematic performance and robustness analysis of the GLRT. Specifically, we explore the performance of the GLRT under statistical and spatial modeling errors. This analysis is essential for understanding the advantages and disadvantages of the detector in scenarios where the assumed model is most likely to be incorrect. 1053-587X/02$17.00 2002 IEEE

2618 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002 B. Problem Formulation Assume an array of sensors, and denote by the received -dimensional signal vector at instant. A common model for the received signal vector is where is an -dimensional vector, and lies on the array manifold { }. The vector is usually called the array response vector or the steering vector. It represents the array complex response to a narrowband point source situated at an angle. The source signal is denoted by, and is an 1 additive noise vector. Finally, in addition, assume that and are ergodic, independent random vector processes. Consider the following detection problem: Under the null hypothesis, only noise exists, whereas under the alternate hypothesis, in addition to the noise, there exists a point source at bearing. Based on independent array snapshots, we want to decide whether to accept the null hypothesis (noise only), or to accept the alternate hypothesis (signal plus noise). C. GLRT The simplest variant of the detection problem in (2) is the case where under both hypotheses, the probability density function (pdf) of the measurements is known. In this case, the problem is a simple hypotheses testing problem, and the optimal (in the Neyman-Pearson sense) likelihood ratio test (LRT) can be implemented. This detector turns out to be the beamformer steered at direction, followed by a simple linear filter (Echardt) [7]. For cases in which some of the parameters are unknown, the detection problem in (2) becomes a composite hypotheses testing problem. Although not necessarily optimal, the GLRT is widely used in such problems. In many problems, including the problem at hand, the GLRT is a constant false alarm rate (CFAR) detector. This desirable property is one of the main reasons why this detector was investigated extensively over the last three decades. Denote by the pdf of the received vectors under hypothesis, where stands for the unknown parameters under hypothesis, and denotes the parameter space of. The GLRT for deciding whether to accept or to accept is given by (1) (2) (3) Assume the following. 1) The additive noise is a circular, zero mean, white (both temporally and spatially), complex Gaussian random vector process with unknown variance, i.e.,, where 0 and are, respectively, a zero vector and the identity matrix of appropriate dimensions. 2) The signal is a circular, zero mean, temporally white, complex Gaussian random process with unknown variance, i.e.,. 3) The signal and the noise are statistically independent, i.e.,. 4) The source bearing (if the source exists) is unknown. Given assumptions 1) 4), each snapshot is a circular, zero mean, white, complex Gaussian random vector with covariance matrix under the null hypothesis, and under the alternate hypothesis. Thus, the detection problem (2) is now given by Note that under the null hypothesis, the unknown parameter vector contains only the unknown noise level, whereas under the alternate hypothesis, the unknown parameter vector contains the unknown source bearing, the unknown source power, and the unknown noise level, i.e., Under the Gaussian assumptions, the GLRT of (3) takes the form of (4) (5) (6) where is the empirical correlation matrix, i.e.,, and is the correlation matrix of the received measurements, assuming that the unknown parameter vector is equal. Once again, is the ML estimate of the unknown parameter vector given the th hypothesis. Plugging the exact distributions in (4) into (6) results in where stands for the ML estimates of the unknown parameters under hypothesis, i.e.,, and is the detector threshold. (7)

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST 2619 where, and. In the sequel, we refer to the detector given by (7) as the Gaussian generalized likelihood ratio test (GGLRT). Note that the GGLRT requires only a simple one-dimensional (1-D) search. In addition, note that for the problem of interest, the threshold is negative, or else, the false alarm probability is strictly one. This is since contains, which means that. D. Summary of Paper s Main Results It is well known that in bearing estimation problems, the ML estimate of the bearing of a Gaussian source is robust. That is, under mild regularity conditions, the performance of the ML estimator for the bearing is insensitive to the source s distribution [8]. This property is usually termed as asymptotic robustness. However, a similar study of the robustness of the GGLRT has yet to be conducted. This paper carries out such an analysis, investigating the robustness properties of the GGLRT (7). The first robustness study of the GLRT statistic of (3) appeared in [9]. In this study, the asymptotic distribution of the GLRT statistic, given the null hypothesis for nested hypotheses testing problem, was computed. The result was a generalization of Wilks theorem for the case of mismatch. In later studies (see, among many others, [10] [12]), other theoretical aspects of the robustness of the GLRT, as well as other detectors, were studied. These studies are generally not concerned with some specific model, and thus, the results obtained do not provide insight into specific problems like the one at hand. In [1] and [13], the problem of estimating the number of sources is studied using statistical tools similar to the one used in this study. The study in [1] and [13] provides an insight into the performance of the MDL estimator when various sets of probabilistic assumptions are used. The improvement in performance due to the utilization of a priori information is studied as well. This paper investigates the behavior of the GGLRT under two types of modeling errors: statistical and spatial. Spatial modeling errors occur when the point source assumption is violated, i.e., when the frequency band contains several sources from different directions instead of a single one or when the received signal is a distributed source instead of a point source. Statistical modeling errors correspond to deviations from the Gaussian assumptions, i.e., when the additive noise and/or the signal are non-gaussian. The paper provides a comprehensive analysis of the performance of the GGLRT in various scenarios. The results of this analysis are closed-form expressions for the asymptotic variation in the probability of detection for various types of modeling errors. Specifically, we start with analyzing the performance of the GGLRT with no modeling errors. The results obtained for this case are used as our baseline for comparison with the performance of the GGLRT under various modeling errors. Next, the performance of the GGLRT is examined when the source is a distributed source instead of the assumed point source, i.e., spatial mismatch. A closed-form expression for the degradation in performance as a function of the source SNR and the angular spreading is obtained. The cases of non-gaussian noise or non-gaussian source are also examined, and closed-form expressions for the performance change due to the various deviations from the Gaussian assumptions are provided as well. It is shown both theoretically and empirically that the existence of statistical mismatch might result in performance improvement compared with the case when no mismatch exists. This property is not only new but is rather surprising as well. Although, usually, mismodeling errors cause some degradation in performance, this is not the case in this detection problem. In fact, this paper gives exact conditions for cases in which the detection probability increases. The rest of the paper is organized as follows. Section II gives a general asymptotic analysis of the GGLRT. The case of a general type of model mismatch is examined in this section as well. Section III examines spatial mismodeling errors, whereas Section IV examines statistical mismodeling errors. To validate analytical results, simulations are presented at the end of each section. Finally, Section V summarizes the paper. II. GENERAL ANALYSIS OF THE GGLRT A. Analysis Based on the theory of misspecified models [14], this subsection presents an asymptotic analysis of the GGLRT evaluated under very general conditions. The result is then used to analyze the performance of the GGLRT, given normally distributed measurements with possible mismodeling errors. Denote by the distribution of a single snapshot. Introducing enables one to consider cases where the actual distribution of is different from its assumed distribution under either or. In addition, assume that there exist two unique, interior points and (hereafter referred to as stationary points) in and, respectively, which minimize (8) where is the true correlation matrix of the measurements,, and. In addition, assume that for all matrices, the following two conditions are met: (9) (10) where is the expectation of with respect to the pdf. Theorem 1: Assume that the alternate hypothesis is true. Given the conditions stated above, the GGLRT statistic of (7) is asymptotically normally distributed, with mean and variance, i.e.,, where (11)

2620 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002 (12) Proof: See Appendix A. Note that Theorem 1 does not contradict the Wilks theorem [15]. The two parts of the Wilks theorem state that under some regularity conditions, whenever is true, is chi-squared distributed, whereas whenever is true, is noncentral chi-squared distributed. However, these regularity conditions require that either or, which is not true under Theorem 1. In some sense, Theorem 1 extends the Wilks theorem. A detector s performance is measured by the probability of detection given a predetermined false alarm probability. Theorem 1 provides a way to evaluate the asymptotic probability of detection of the GGLRT as a function of the threshold (and, therefore, as a function of the false alarm probability) set by the detector Since is a consistent estimator of [14], is easily computed by replacing with its mean value, i.e., Using similar arguments (15) (16) Note that in order to compute the ML estimates and,itis assumed that under both the null and the alternate hypothesis, the received vector is normally distributed. It is now straightforward to compute the mean and the variance of the GGLRT statistic (full evaluation in Appendix B) (13) where is the complementary error function,, and, are given by (11) and (12), respectively. Thus, in order to quantify the performance of the GGLRT under various modeling errors for a given threshold, it suffices to examine,. In the following, we present an asymptotic analysis of the GGLRT (7) under some specific modeling errors. This analysis results in simple, closed-form expressions for the mean and the variance of the GGLRT statistic ( of (7)) given the alternate hypothesis. These simple expressions can be used with (13) to obtain approximations for the probability of detection in various cases. B. Case of Normally Distributed Measurements Assume that the measurements are independent and identically distributed (iid), zero mean complex normal random vectors with correlation matrix, which can be written as (14) where and { } are the eigenvalues and eigenvectors of the received vector correlation matrix, respectively, and is the normalized steering vector. Note that under the alternate hypothesis (with no modeling errors), the GGLRT assumes that the correlation matrix of the received vector is given by (14) with. Whenever for some, there exists a modeling error. Analyzing the performance of the GGLRT requires the evaluation of the stationary points and. The ML estimate of is [16] (17) (18) C. Case of No Modeling Errors The general results in (17) and (18) allow the evaluation of the performance of the GGLRT in various scenarios. The first is the case of no modeling errors, i.e., in (14). Assume that the measurements are an iid, zero mean, complex normal random vector with correlation matrix equal to. In this case, the eigenvalues of the correlation matrix are given by (19) It is now straightforward to show that the asymptotic distribution of the GGLRT statistic of (7) is given by (20) where is the per-element SNR. Clearly, this means that when there are no modeling errors, the asymptotic performance of the GGLRT is strictly a function of the SNR and the number of sensors. It is well known that in the corresponding bearing estimation problem, the mean square error (MSE) of the ML estimate for the bearing decreases as the SNR increases [5]. It seems natural

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST 2621 Most of the existing literature model the statistics of the observed data in the case of a distributed source by a zero mean, white, complex Gaussian random vector (for full derivation, see, e.g., [23]) (21) A number of parametric models have been used to describe the channel covariance matrix. Most involve the use of some parameterized form of the angular power density function, e.g., [17], [23] (22) where is the average power intensity of the received signal from direction. Common models of include Fig. 1. Performance of the GGLRT for a point source as a function of the SNR for various probabilities of false alarm. that in the described detection problem, for any given false alarm probability, an increase in the SNR will imply an increase in the detection probability. The following proposition proves that this is indeed the case when using the GGLRT. Proposition 1: For any given false alarm probability, the probability of detection is an asymptotically monotonic increasing function of the SNR. Proof: See Appendix C. D. Simulations To validate proposition 1, simulations of a uniform linear array (ULA) with six sensors at half wavelength spacing were carried out. One hundred independent snapshots of a white, Gaussian point source at angle were taken. Fig. 1 depicts the probability of detection as a function of the SNR for false alarm probabilities equal to 10 and 10. The SNR was defined at the element level, i.e., SNR. The theoretical curves were evaluated using (13) and (20), whereas the empirical curves were calculated based on 1000 Monte Carlo runs. The empirical probability of detection was calculated as the number of runs in which the source was detected divided by 1000. The figure shows a very good fit between the empirical and the theoretical results. III. ANALYSIS OF THE GLRT WITH SPATIAL MISMODELING A. Analysis Perhaps the most restrictive assumption of the model in (2) is the point source assumption. Frequently, there exist local scatters at the vicinity of the emitter that create multipath effects. Such may be the case in radar, sonar, astronomy, or wireless communications applications (e.g., [17]). As a result, bearing estimation for a distributed source has begun to attract interest in the literature (see, among others, [18] [21]). The spatial extent of a distributed source is typically characterized by some type of a parametric model. The models in the litrature have formed the basis of a variety of recently reported bearing estimation techniques and performance studies, e.g., [22]. otherwise Gaussian Uniform (23) The mean angle parameter is assumed to be the source bearing, and the spatial spreading parameter is denoted by. Since the point source assumption is sometimes incorrect, it is of interest to investigate the performance of the GGLRT under such a modeling error. In what follows, we investigate the performance of the GGLRT when the source is a scattered source with small spreading. It is proved that the performance of the GGLRT deteriorates as the spatial spreading of the source increases. In [24], it is shown that small spreading implies a shift of energy from the first eigenvalue of the channel covariance matrix to the second. As the spreading increases, more energy is shifted with some being drifted to the next eigenvalue. In the sequel, we assume that energy is shifted from the first eigenvalue to the second only, i.e., the eigenvalues of are given by (24) where, and is the decrease (increase) in the first (second) eigenvalue with respect to a point source with the same SNR. Note that this assumption corresponds to small spreading. Analyzing the probability of detection in such a scenario can be done by plugging (24) into (17) and (18). Denote by, the mean and standard deviation of the GGLRT statistic as a function of the SNR,, and the scaled energy shift from the first eigenvalue to the second. Once again, the probability of detection is equal to, where is defined by (25)

2622 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002 Proposition 2: For a fixed threshold level, the probability of detection of the GGLRT decreases as the spreading increases. Proof: First, note that the case of corresponds to a point source with SNR equal to. Expanding about using a Taylor s expansion results in (26) The first term on the right-hand side of (26) is the expression for for the point source case when there is no modeling error. Therefore, in order to show the degradation in performance of the GGLRT, it suffices to investigate the sign of the second term. In Appendix D, it is shown that the second term in (26) is strictly negative. Therefore, for small spreading, decreases linearly with the energy shift from the first eigenvalue to the second. This decrease implies a decrease in the probability of detection since is a monotonic decreasing function. B. Simulations To validate theoretical results, simulations of a ULA with six elements at half wavelength spacing were carried out. The source in the simulations was assumed to be at an angle with a Gaussian scattering function (23). One hundred snapshots were taken with SNR per element equal to 8dB. Fig. 2 depicts the probability of detection as a function of the source spreading for various levels of false alarm probability. The source spreading was varied between 0 (point source) and 16. The theoretical curves were evaluated using (13), (17), and (18), whereas the empirical curves were calculated based on 1000 Monte Carlo runs. The empirical probability of detection was calculated as the number of runs in which the source was detected divided by 1000. The figure shows good fit between the empirical and theoretical curves. It is seen that the prediction of the empirical results by theoretical curves degrades as the spreading increases. This is since the theoretical analysis assumes that the energy shifts from the first eigenvalue to the second. This assumption becomes invalid as the spreading increases. As mentioned earlier, bearing estimation of scattered sources is an extensively investigated subject. It is known that as long as the source spreading is less than the array beamwidth [5], the performance of the beamformer, which is the ML estimator for the bearing of a point source, is almost invariant to the source spreading [25]. In our example, the array beamwidth is equal to 20, and as such, the performance of the bearing estimator is hardly affected by the existence of the scattered source. On the other hand, the GGLRT suffers from a significant performance degradation, even in the presence of small spreading. IV. ANALYSIS OF THE GLRT IN STATISTICAL MISMODELING A. Analysis It is well known that in bearing estimation problems, the asymptotic performance of the stochastic Gaussian ML estimator is invariant to the source distribution [8] and to the noise Fig. 2. Performance of the GGLRT as a function of the source spreading for various levels of probability of false alarms. distribution [26], i.e., even when the signal (or the noise) distribution is non-gaussian, the asymptotic covariance of the ML estimate of the bearing remains the same. This section examines the performance of the GGLRT under similar mismodeling errors. The analysis carried out in this section studies how a non- Gaussian environment affects the performance of the GGLRT, which assumes that both the source and the noise are Gaussian. Note that it is assumed that while the Gaussian assumption is violated, the point source assumption is not. In the context of non-gaussian distributions, the focus is on the mixed Gaussian distribution. The model of a finite mixed Gaussian distribution is of special interest since it provides a general framework for modeling many non-gaussian distributions. For example, Middleton s class A model [27] can be approximated by a finite Gaussian mixture. In addition, it is known that any monotonic spherically symmetric pdf can be approximated by a continuous Gaussian mixture distribution (see [28] and references therein). The pdf of a finite Gaussian mixture is given by (27) where complex Gaussian pdf; number of elements in the mixture; mean of the th element in the mixture; covariance matrix of the th element in the mixture. The probability of choosing the th element in the mixture is denoted by. In what follows, we examine two types of mixed Gaussian models. The first model is associated with a finite alphabet source (digital signal) in Gaussian additive noise. In this model, the signal at each instant is a zero mean, discrete complex random variable taking values from some discrete signal constellation denoted by of size. The discrete time baseband equivalent model, after down conversion, matched filtering, and sampling at the symbol rate

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST 2623 is. The pdf of is a finite, mixed, complex Gaussian distribution. This mixture has components, each corresponding to a different value of the signal. The covariance matrix is [29], i.e., (28) where is the probability of the source transmitting (usually equal to ). Note that the received vector correlation matrix is equal to, where is the average signal power. The second model is associated with a Gaussian point source in additive mixed Gaussian noise. In this model, the noise pdf is given by (27) with for every. As a result, the measurements are distributed according to (29) Note that the received vector correlation matrix is equal to, where is the average noise power. As proven in Theorem 1, the GGLRT statistic is asymptotically normally distributed. In Appendix B, it is shown that the mean of the GGLRT statistic is independent of the underlying distribution and depends only on the correlation matrix of the measurements. Thus, the relation between the performance of the GGLRT when there are no modeling errors and the performance of the GGLRT when there exist statistical modeling errors depends only on the relation between the variance of the GGLRT statistic in the two cases. Denote by the variance of the GGLRT statistic when the measurements are distributed according to (28), i.e., digital sources. Lemma 1: For fixed SNR, the variance of the GGLRT statistic when no modeling errors exist is smaller than its variance when the measurements are distributed according to (28) if and only if (iff). That is, iff, where denotes the variance of the GGLRT statistic when no modeling errors exist. Proof: See Appendix E. Note that Appendix E contains explicit expressions for. Lemma 1 provides a characterization of the relation between the performance of the GGLRT with and without statistical modeling errors. In order to examine the performance of the GGLRT, it suffices to compare and. Whenever, the performance of the GGLRT improves. On the other hand, whenever, the performance of the GGLRT deteriorates. If, then the performance of the GGLRT is not affected by the existence of modeling errors. Since the mean of the GGLRT statistic is independent of the exact source distribution, it suffices to examine the standard deviation of the GGLRT statistic. Conditions for an Increase in the Detection Probability due to Statistical Mismodeling: 1) (which implies that the probability of detection is larger than 50%), and the source is radiating a digital signal. By using Lemma 1, if, then, which implies that the probability of detection of the GGLRT increases. 2) (which implies that the probability of detection is smaller than 50%), and the source is radiating a digital signal. By using Lemma 1, if, then, which implies that the probability of detection of the GGLRT increases. Conditions for a Decrease in the Detection Probability due to Statistical Mismodeling : 1), AND THE SOURCE IS RADIATING A DIGITAL SIGNAL: By using Lemma 1, if, then, which implies that the probability of detection of the GGLRT decreases. 2), AND THE SOURCE IS RADIATING A DIGITAL SIGNAL: By using Lemma 1, if, then, which implies that the probability of detection of the GGLRT decreases. Conditions for the Detection Probability not to Be Effected by Statistical Mismodeling: 1), I.E., THE MEAN OF THE GGLRT STATISTIC IS EQUAL TO THE THRESHOLD: In this case, the probability of detection is equal 50% and is independent of modeling errors. Due to their frequent use, special attention is given to the case of digital signals. It is easily shown that for constant modulus sources, such as binary phase shifting keys (BPSKs) or M-phase shifting key (MPSK) signals,. Thus, for constant modulus signals, whenever the probability of detection is larger than 50%, the probability of detection of the GGLRT increases. It is also easily seen that the condition is not met by sources that are impulsive in nature, e.g., the source signals emitted by an impulse radio transmissions [30]. Fig. 3 depicts a qualitive graph of the detection probability of the GGLRT as a function of the threshold for three scenarios: 1) There are no modeling errors. 2) The source radiates a digital constant modulus signal. 3) The source radiates an impulsive signal. As shown by the figure, all the curves intersect when the probability of detection is equal 50%, i.e., when the probability of detection is equal to 50%, the modeling errors have no effect on the detector s performance. When the probability of detection is less than 50%, the case of a source emitting an impulsive signal exhibits the best performance, whereas the constant modulus digital source case exhibits the worst performance. On the other hand, when the probability of detection is larger than 50%, it is the other way around. We now turn our attention to the case of non-gaussian noise. Denote by the variance of the GGLRT statistic when the measurements are distributed according to (29), i.e., non- Gaussian noise. The following lemma proves that the variance of the GLRT statistic is increased due to the existence of the non-gaussian noise.

2624 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002 Fig. 3. Performance of the GGLRT as a function of the threshold for various scenarios. Fig. 4. signal. Performance of the GGLRT as a function of the threshold for BPSK Lemma 2: For fixed SNR, is minimized when, i.e., when the measurements are complex Gaussian random vectors (no mismatch). Proof: See Appendix F. Lemma 2 characterizes the behavior of the GGLRT in the non-gaussian noise as a function of the threshold used by the detector. Similarly to the case of digital sources, we note that for a fixed threshold, if is smaller (greater) than, the probability of detection will decrease (increase) due to the existence of the non-gaussian noise. It should be noted that the conclusion drawn from Lemma 2 is under the assumption of a fixed threshold. However, for a fixed threshold, both the false alarm probability and the probability of detection are a function of the exact noise distribution (in contrast with the previous cases of a distributed source or a digital source). B. Simulations Once again, to validate the analytical results, simulations of a ULA with six elements at half a wavelength spacing were carried out. Two hundred snapshots of a source at bearing with SNR equal 0 db were taken. Fig. 4 depicts the probability of detection of the GGLRT as a function of the threshold. Since both Lemma 1 and 2 analyze the performance as a function of the threshold, depicting the probability of detection as a function of the threshold is a natural choice for this figure. We tested three sources: The first was taken to be a BPSK signal, i.e., and Gaussian additive noise; the second was taken to be an impulsive signal, i.e.,, and, ; the third was taken to be a Gaussian source. The empirical curves were calculated based on 1000 Monte Carlo runs. As expected, when the probability of detection is larger than 50%, the performance of GGLRT with BPSK source is the best, whereas when the probability of detection is smaller than 50%, the impulsive source performs the best. In addition, the accuracy of the analysis can be seen by the exact fit between the analytical and empirical curves. Note that both curves match the qualitative Fig. 3. Fig. 5. Performance of the GGLRT as a function of the threshold for a mixed Gaussian noise. Fig. 5 depicts the results of a similar scenario to the previous one. In this case, the source was taken to be a white Gaussian signal, whereas the additive noise was mixed-gaussian with,,, and. Once again, the figure shows good fit between the theoretical and the empirical results. V. SUMMARY This paper provides an analysis of the performance of the GLRT in various scenarios. The analysis contains closed-form expressions for the probability of detection under various modeling errors. Focus is set on the GLRT for the problem of detecting a far-field, Gaussian point source (GGLRT) using an array of sensors. Using the general analysis, the performance of the detector is evaluated for a wide variety of scenarios. Specifically, spatial and statistical modeling errors are examined for the problem of interest, and the variation in performance is quantified. It is shown that the performance of the detector increases

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST 2625 as the SNR increases and degrades as the source becomes more scattered. It is also shown that the performance of the GGLRT may improve in cases of statistical errors (when the source is digital or when the noise is non-gaussian). APPENDIX A PROOF OF THEOREM 1 This Appendix proves a slightly more general theorem than the one presented in Theorem 1. It is proved that under general conditions, the GLRT statistic of (3) is asymptotically normally distributed. The asymptotic distribution of the GGLRT statistic, and hence Theorem 1, can be viewed as a special case of this more general proof. The following proof is based on the derivations done in [9, proof of Th. 1], [14], and [31]. Denote by the Kullback Leibler distance (divergence) between two pdfs and, i.e.,. Assume that and exist for every and, respectively, where and are the parameter spaces of,, respectively. In addition, assume that there exist two unique, interior points and (hereafter, referred to as stationary points) in and, respectively, that minimize and, respectively. We note that for Theorem 1, the existence of these two points is ensured by the conditions given in (8). The GLRT statistic denoted by in (3) is given by (30) The ML estimators and are consistent estimators for and, respectively [14]. The Taylor expansion for the functions and, around and, respectively, is [9] where, 1, and it is assumed that both and exist at least in the local vicinity of. Plugging (31) back into (30) results in the following approximation for the GLRT: (35) The exact asymptotic distribution of the GLRT statistic is very complicated. Both and are Gaussian quadratic forms and, as such, can be shown to be distributed as a weighted sum of chi-squared random variables. Noting that the mean and variance of these quadratic forms are bounded and that as approaches infinity the mean and variance of are unbounded, the approximation of the GLRT statistic can be further simplified by neglecting the quadratic forms. As a result, asymptotically, is given by (36) Invoking the central limit theorem, the asymptotic distribution of is given by (31) (32) where is equal to ;, 1. The asymptotic distribution of, is given by [14] (33) (34) (37) (38) (39) We note that Theorem 1 is a special case of (37) (39), where and are the normal distributions detailed in (4). APPENDIX B MOMENTS OF THE GLRT STATISTIC IN THE CASE OF MISMODELING Denote by, 1 the correlation matrix under hypothesis at the stationary point and by the underlying correlation matrix of the data. In order to compute the mean

2626 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002 and the variance of the GLRT statistic in (11) and (12), we will make use of the following identities: is the complex Gaussian distribution, and (40) (41) In addition, since (47) (42) Straightforward calculations result in (43) then (44) Therefore, since under both hypotheses is a Gaussian random vector Similarly (48) (45) Note that the asymptotic mean is independent of the underlying distribution. Similarly (46) where the last equality holds since the underlying distribution (49)

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST 2627 Define APPENDIX C PROOF OF PROPOSITION 1 (50) where is the threshold of the GGLRT. The probability of detection is simply. Since is a monotonic decreasing function of, in order to show that the detection probability of the GGLRT increases with the SNR, it suffices to show that is a monotonic increasing function of. Differentiating with respect to and neglecting positive constants yields (53) Next, evaluate The first inequality holds since for any.to establish the second inequality, set. It can now be easily verified that for any, (note that the threshold is always negative, and thus, is positive). Therefore, is a monotonic increasing function of, which implies that the detection probability of the GGLRT increases with. APPENDIX D SPATIAL MODELING ERROR It is of interest to investigate the sign of the second term in (26). Note that the second term s sign is determined completely by its nominator multiplied by Therefore which means that (54) (55) (56) In addition, we have (57), shown at the bottom of the next page, which means Hence (58) First, evaluate (51) (59) Clearly, the sign of is invariant to multiplication with positive constants. Hence, the focus is on (52) (60) since is negative for (recall that is negative). In order to show that is negative, it suffices to show that

2628 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002 is a monotonic decreasing function of. Differentiating with respect to yields (61), shown at the bottom of the page, which is clearly negative, implying that is strictly negative, which concludes the proof. APPENDIX E DIGITAL SIGNAL IN GAUSSIAN NOISE In this Appendix, we evaluate the asymptotic variance of the GGLRT statistic for the case of a digital signal and Gaussian noise (28) and find the condition in which it is greater than the variance of the GGLRT statistic when there are no modeling errors, i.e.,. The distribution of the measurements is given by where is defined in (47), is the second moment of the quadratic form, and is a complex Gaussian random vector with mean and covariance matrix. Therefore, we have (64), shown at the bottom of the next page, where we used the fact that for a complex Gaussian random vector with mean and covariance matrix Recall that (62) where. Once again, the correlation matrix of the measurements is, where. Using Theorem 1 and following (75), is given by (63) Thus (65) (66) (57) (61)

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST 2629 Averaging (66) with respect to yields and thus Therefore (67) (70) Finally, combining (68) and (70), results in (71), shown at the bottom of the page. Subtracting from yields (68) In addition, since, then (69) Thus iff. APPENDIX F GAUSSIAN SIGNAL IN MIXED-GAUSSIAN NOISE (72) In this Appendix, we evaluate the asymptotic variance of the GGLRT statistic for the case where the signal is Gaussian and (64) (71)

2630 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 50, NO. 11, NOVEMBER 2002 the noise is mixed-gaussian (29) and show that it is greater than the variance of the GGLRT statistic when there are no modeling errors, i.e.,. The distribution of the measurements is given by (73) The correlation matrix of the measurements is given by, where. Using Theorem 1, is given by (74) (75) where is defined in (47), and is the second moment of the quadratic form, where is a complex Gaussian random vector with zero mean and covariance matrix. Therefore It is easy to verify that Thus (76) (77) where the second inequality holds since for any random variable,. REFERENCES [1] E. Fishler, M. Grosmann, and H. Messer, Detection of signals by information theoretic criteria: General asymptotic performance analysis, IEEE ans. Signal Processing, vol. 50, pp. 1027 1036, May 2002. [2] M. Wax and T. Kailath, Detection of signals by information theoretic criteria, IEEE ans. Acoust., Speech Signal Processing, vol. ASSP-33, pp. 387 392, Apr. 1985. [3] H. T. Wu, J. F. Yang, and F. K. Chen, Source number estimators using transformed Gerschgorin radii, IEEE ans. Signal Processing, vol. 43, pp. 1325 1333, June 1995. [4] P. Stoica and A. Nehorai, MUSIC, maximum likelihood, and Cramér-Rao bound, IEEE ans. Signal Processing, vol. 37, pp. 720 741, May 1989. [5] P. M. Schultheiss and H. Messer, Optimal and suboptimal broad-band source location estimation, IEEE ans. Signal Processing, vol. 41, pp. 2752 2763, Sept. 1993. [6] M. Wax, Detection and localization of multiple sources via the stochastic signals model, IEEE ans. Signal Processing, vol. 39, pp. 2450 2456, Nov. 1991. [7] J. B. Lewis and P. M. Schultheiss, Optimum and conventional detection using a linear array, J. Acoust. Soc. Amer., vol. 49, pp. 1083 1091, Apr. 1971. [8] B. Ottersten, M. Viberg, and T. Kailath, Analysis of subspace fitting and ML techniques for parameter estimation from sensor array data, IEEE ans. Signal Processing, vol. 40, pp. 590 599, Mar. 1992. [9] J. T. Kent, Robust properties of likelihood ratio test, Biometrika, vol. 69, pp. 19 27, 82. [10], The underlying structure of nonnested hypothesis tests, Biometrika, vol. 73, no. 2, pp. 333 343, August 1986. [11] A. W. van der Vaart, Asymptotic statistics, in Cambridge Series in Statistical and Probabilistic Mathematics, 1st ed. Cambridge, U.K.: Cambridge Univ. Press, 1998. [12] K. Viraswami and N. Reid, Higher-order asymptotic under model misspecification, Can. J. Statist., vol. 24, no. 2, pp. 263 278, June 1996. [13] E. Fishler and H. Messer, On the effect of a-priori information on performance of the MDL estimator, Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2002. [14] H. White, Maximum likelihood estimation of misspecified models, Econometrica, vol. 50, no. 1, pp. 1 25, Jan. 1982. [15] S. M. Kay, Fundamentals of Statistical Signal Processing Detection Theory. Englewood Cliffs, NJ: Prentice-Hall, 1998. [16] M. A. Doron, A. J. Weiss, and H. Messer, Maximum-likelihood direction finding of wide-band sources, IEEE ans. Signal Processing, vol. 41, pp. 411 414, Jan. 1993. [17] Y. Meng, P. Stoica, and K. M. Wong, Estimation of the directions of arrival of spatially dispersed signals in array processing, Proc. Inst. Elect. Eng., Radar, Sonar, Navigat., vol. 143, pp. 1 9, Feb. 1996. [18] Y. U. Lee, J. Choi, I. song, and S. R. Lee, Distributed source modeling and direction-of-arrival estimation techniques, IEEE ans. Signal Processing, vol. 45, pp. 960 969, Apr. 1997. [19] S. Valaee, B. Champagne, and P. Kabal, Parametric localization of distributed sources, IEEE ans. Signal Processing, vol. 43, pp. 2144 2153, Aug. 1995. [20] J. Friedmann, R. Raich, J. Goldberg, and H. Messer, Bearing estimation for a distributed source of non constant modulus, IEEE ans. Signal Processing, Apr. 2001, submitted for publication. [21] O. Besson, F. Vincent, P. Stoica, and A. B. Gershman, Approximate maximum likelihood estimators for array processing in multiplicative noise environments, IEEE ans. Signal Processing, vol. 48, pp. 2506 2518, Sept. 2000. [22] O. Besson and P. Stoica, Decoupled estimation of DOA and angular spread for a spatially distributed source, IEEE ans. Signal Processing, vol. 48, pp. 1872 1882, July 2000. [23] T. ump and B. Ottersten, Estimation of nominal direction of arrival and angular spread using an array of sensors, Signal Process., vol. 50, pp. 57 69, Apr. 1996. [24] H. B. Lee, Eigenvalues and eigenvectors of covariance matrices for signals closely spaced in frequency, IEEE ans. Signal Processing, vol. 40, pp. 2518 2535, Oct. 1992. [25] R. Raich, J. Goldberg, and H. Messer, Bearing estimation for a distributed source via the conventional beamformer, in Proc. Ninth IEEE SP Workshop Statist. Array Process., Portland, OR, 1998. [26] A. Satora and P. Bentler, Model conditions for asymptotic robustness in the analysis of linear relations, Comput. Statist. Data Anal., vol. 10, pp. 235 249, 1990. [27] A. Spaulding and D. Middleton, Optimum reception in an impulsive interference environment-part I: Coherent detection, IEEE ans. Commun., vol. 25, pp. 910 923, Sept. 1977. [28] R. J. Kozick and B. Sadler, Maximum-likelihood array processing in non-gaussian noise with Gaussian mixtures, IEEE ans. Signal Processing, vol. 48, pp. 3520 3536, Dec. 2000.

FRIEDMANN et al.: GENERAL ASYMPTOTIC ANALYSIS OF THE GENERALIZED LIKELIHOOD RATIO TEST 2631 [29] E. Moulines, J.-F. Cardoso, and E. Gassiat, Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models, Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., vol. 5, pp. 3617 3620, 1997. [30] M. Z. Win and R. A. Scholtz, Impulse radio: How it works, IEEE Commun. Lett., vol. 2, no. 2, pp. 36 38, Feb. 1998. [31] H. White, Correction to: Maximum likelihood estimation of misspecified models, Econometrica, vol. 51, p. 513, 1983. [32] S. Bose and A. O. Steinhardt, A maximal invariance framework for adaptive detection with structures and unstructured covariance matrices, IEEE ans. Signal Processing, vol. 43, pp. 2164 2175, Sept. 1995. [33] S. A. Kassam and H. V. Poor, Robust techniques for signal processing: A survey, Proc. IEEE, vol. 73, pp. 433 481, Mar. 1985. [34] H. Messer, The use of spectral information in optimal detection of a source in the present of a directional interference, IEEE J. Ocean. Eng., vol. 19, pp. 422 430, July 1994. [35] D. Parsons, The Mobile Radio Propagation Channel. New York: Pentech, 1992. [36] S. Pasupathy and P. M. Schultheiss, Passive detection of Gaussian signals with narrow-band and broad band components, J. Acoust. Soc. Amer., vol. 56, no. 3, pp. 917 921, Sept. 1974. [37] K. I. Pedersen, P. E. Mogensen, B. H. Fleury, F. Frederikse, K. Olesen, and S. L. Larsen, Analysis for time, Azimuth and Doppler dispersion in outdoor radio channels, in Proc. ACTS Summit, Aalborg, Denmark, 1997. [38] A. M. Richardson and A. H. Welsh, Covriate screening in mixed linear models, J. Multivariate Anal., vol. 58, no. 1, pp. 27 54, 1996. [39] P. Stoica and A. Nehorai, MUSIC, maximum likelihood, and Cramér Rao bound Further results and comparisons, IEEE ans. Signal Processing, vol. 38, pp. 2140 2150, Dec. 1990. Jonathan Friedmann was born in Israel in 1971. He received the B.Sc. degree in industrial engineering in 1996, the M.Sc. degree in electrical engineering in 1998, and the Ph.D. degree in electrical engineering in 2002 from Tel Aviv University, Tel Aviv, Israel. His research interests include statistical signal processing and digital communications. Eran Fishler (M 01) was born in Tel Aviv, Israel, in 1972. He received the B.Sc. degree in mathematics and the M.Sc. and Ph.D. degrees in electrical engineering in 1995, 1997, and 2001, respectively, from Tel Aviv University, Tel Aviv, Israel. From 1993 to 1995, he was with the Israeli Navy as a research engineer. From 1996 to 2000, he was with Tadiran Electronic Systems, Holon, Israel. He is currently a post-doctoral fellow at Princeton University, Princeton, NJ. His research interests include statistical signal processing and communication theory. Hagit Messer (F 01) received the B.Sc, M.Sc., and Ph.D. degrees in electrical engineering from Tel Aviv University, Tel Aviv, Israel, in 1977, 1979, and 1984, respectively. Since 1977, she has been with the Department of Electrical Engineering Systems at the Faculty of Engineering, Tel Aviv University, first as a Research and Teaching Assistant and then, after a one-year post-doctoral appointment at Yale University (from 1985 to 1986), as a Faculty member. Her research interests are in the field of signal processing and statistical signal analysis. Her main research projects are related to tempo-spatial and non-gaussian signal detection and parameter estimation. Dr. Messer was an associate editor of the IEEE TRANSACTIONS ON SIGNAL PROCESSING from 1994 to 1996, and she is now an associate editor for the IEEE SIGNAL PROCESSING LETTERS. She is a member of the Technical Committee of the IEEE Signal Processing Society for Signal Processing, Theory, and Methods and was a co-organizer of the IEEE Workshop on Higher Oreder Statistics in 1999. Since 2000, she has been the chief scientist of the Israeli Ministry of Science, Culture, and Sport.