Detection of Signals by Information Theoretic Criteria

Transcription

1 IEEE TRANSACTIONS ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-33, NO. 2, APRIL Detection of Signals by Information Theoretic Criteria MAT1 WAX AND THOMAS KAILATH, FELLOW, IEEE Abstract-A new approach is presented to the problem of detecting the number of signals in a multichannel time-series, based on the application of the information theoretic criteria for model selection introduced by Akaike (AIC) and by Schwartz and Rissanen (MDL). Unlike the conventional hypothesis testing based approach, the new approach does not require any subjective threshold settings; the number of signals is obtained merely by minimizing the AIC or the MDL criteria. Siulation results that illustrate the performance of the new method for the detection of the number of signals received by a sensor array are presented. I. INTRODUCTION N many problems in signal processing, the vector of observa- I tions can be modeled as a superposition of a finite number of signals embedded in an additive noise. This is the case, for example, in sensor array processing, in harmonic retrieval, in retrieving the poles of a system from the natural response, and in retrieving overlapping echoes from radar backscatter. A key issue in these problems is the detection of the number of signals. One approach to this problem is based on the observation that the number of signals can be determined from the eigen- values of the covariance matrix of the observation vector. Bartlett [4] and Lawley [14] developed a procedure, based on a nfsted sequence of hypothesis tests, to implement this approach. For each hypothesis, the likelihood ratio statistic is computed and compared to a threshold; the hypothesis accepted is the first one for which the threshold is crossed. The problem with this method is the subjective judgment required for deciding on the threshold levels. In this paper we present a new approach to the problem that is based on the application of the information theoretic cri- teria for model selection introduced by Akaike (AIC) and by Schwartz and Rissanen (MDL). The advantage of this approach is that no subjective judgment is required in the decision process; the number of signals is determined as the value for which the AIC or the MDL criteria is minimized. The paper is organized as follows. After the statement and formulation of the problem in Section 11, the information theoretic criteria for model selection are introduced in Section 111. The application of these criteria to the problem of detecting the number of signals and the consistency of these criteria are discussed in Sections IV and V, respectively. Simulation results that illustrate the performance of the new method for sensor array processing are described in Section VI. Frequency domain extensions and some concluding remarks are presented in Sections VI1 and VIII, respectively. 11. FORMULATION OF THE PROBLEM The observation vector in certain important problems in signal processing such as sensor array processing [15], [20],[5], [6], [lo], [ 121, [22], [25], harmonic retrieval [15], [12], pole retrieval from the natural response [ll], [24], and retrieval of overlapping echos from radar backscatter [7], denoted by the p X l vector x(t), is successfully described by the following model 4 x(t) = A(q) Sj(t) i- n(t) i= 1 where si( e ) = scalar complex waveform referred to as the ith signal A(@i) =a p X 1 complex vector, parameterized by an unknown parameter vector ai associated with the ith signal n( -)= a p X 1 complex vector referred to as the additive noise. We assume that the q (q <p) signals sl ( e), *, sq(.) are complex (analytic), stationary, and ergodic Gaussian random processes, with zero mean and positive definite covariance matrix. The noise vector n( -) is assumed to be complex, stationary, and ergodic Gaussian vector process, independent of the signals, with zero mean and covariance matrix given by a21, where a2 is an unknown scalar constant and Z is the identity matrix. A crucial problem associated with the, model described in (1) is that of determining the number of signals 4 from a finite set of observations x(tl),, x(tn). A promising approach to this problem is based on the struc- ture of the covariance matrix of the observation vector x(.). To introduce this approach, we first rewrite (1) as x(t) = As(t) + n(t) where A is the p X 4 matrix (2a) Manuscript received May 19, 1983; revised June 1, This work was supported in part by the Air Force Office of Scientific Research, Air Force Systems Command, under Contract AF C-0058, the U.S. ArmyResearch Office, undercontract DAAG29-79-C-0215, andby the Joint Services Program at Stanford University under Contract DAAG29-81-K The authors are with the Information Systems Laboratory, Stanford University, Stanford, CA A = [A(%) * * N@q>I (2b) and s(t) is the 4 X 1 vector ST@) = [s, (t) * * sq(t)]. Because the noise is zero mean and independent of the signals, it follows that the covariance matrix of x( *) is given by /85/ $01.OO IEEE

2 388 IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-33, NO. 2, APRIL 1985 R = \k t azi (34 where \k = ASAi (3b) with t denoting the conjugate transpose, and S denoting the covariance matrix of the signals, i.e., S = E[s(. ) s( s ) ~ ]. Assuming that the matrix A is of full column rank, i.e., the vectors A(Qi) (i = 1, *, q) are linearly independent, and that the covariance matrix of the signals S is nonsingular, it follows that the rank of Ik is q, or equivalently, the p - 4 smallest eigenvalues of \E are equal to zero. Denoting the eigenvalues of R by AI > ha..> A, it follows, therefore, that the smallest p - q eigenvalues of R are all equal to d, i.e., -... = Xq+1 =Aq+2 h, = 02. (4) 111. INFORMATION THEORETIC CRITERIA The information theoretic criteria for model selection, introduced by Akaike [l], [2], Schwartz [21], and Rissanen [17] address the following general problem. Given a set of N observations X = {x(l), * *, x(n)} and a family of models, that is, a parameterized family of probability densities f(xlo), select the model that best fits the data. Akaike s proposal was to select the model which gives the minimum AIC, defined by AIC = -2 log f(xl6) t 2k (6) where 6 is the maximum likelihood estimate of the parameter vector 0, and k. is tlie number of free adjusted parameters in 0. The first term is the well-known log-likelihood of tlie maximum likelihood estimator of the parameters of the model. The second term is a bias correction term, inserted so as to make the AIC an unbiased estimate of the mean Kulback- Liebler distance betwey the modeled density f(xl0) and the estimated densityf(xl0). Inspired by Akaike s pioneering work, Schwartz and Rissanen approached the. problem from quite different points of view. Schwartz s approach is based on Bayesian arguments. He assumed that each competing model can be assigned a prior probability, and proposed to select the model that yields the maximum posterior probability. Rissanen s approach is based on information theoretic arguments. Since each model can be used to encode the observed data, Rissanen proposed to select the model that yields the minimum code length. It turns out that in the large-sample limit, both Schwartz s and Rissanen s approaches yield the same criterion, given by MDL= -logf(xis) t i k log N. (7) Note that apart from a factor of 2, the first term is identical to the corresponding one in the AIC, while the second term has an extra factor of 3 log N. IV. ESTIMATING THE NUMBER OF SIGNALS To apply the information theoretic criteria to detect the number of signals, or equivalently, to determine the rank of the matrix 9, we must first describe the family of competing models, or density functions that we are considering. Regarding the observations x(tl ), * * *, x(tn) as identical and statisticaily independent complex Gaussian random vectors of zero mean, the family of models is necessarily described by the co- The number of signals q can hence be determined from the muztiplicity of the smallest eigenvalue of R. The problem is that the covariance matrix R is unknown in practice. When variance matrix of x( e). Since our model s covariance matrix estimated from a finite sample size, the resulting eigenvalues is given by (3), it seems natural to consider the following famare all different with probability one, thus making it difficult ily of covariance matrices to determine the number of signals merely by observing the R(k) eigenvalues. A more sophisticated approach to the problem, = \kc (8) developed by Bartlett [4] and Lawley [14], is based on a se- where denotes a semipositive matrix of rank k, and u dequence of hypothesis tests. The problems associated with this notes an unknown scalar. Note that k E (0, l, - *., p - l} approach is the subjective judgment needed in the selection of ranges over the set of all possiblenumber of signals. the threshold levels for the different tests. Using the well-known spectral representation theorem from In this paper we take a different approach. We pose the delinear algebra, we can express R(k) as tection problem as a model selection problem and then apply the information theoretic criteria for model selection introduced by Akaike (AIC) and by Schwartz and Rissanen (MDL). where XI,. *, Xk and v,, *, Vk are the eigenvalues and eigenvectors, respectively, of R@). Denoting by dk) the parameter vector of the model, it follows that With this parameterization we now proceed to the derivation of the information theoretic criteria for the detection problem. Since the observations are regarded as statistically independent complex Gaussian random vectors with zero mean, their joint probability density is given by Taking the logarithm and omitting terms that do not depend on the parameter vector dk), we find that the log-likelihood function L(@) is given by I,(@@)) = -N log det R@) - tr [R@)]-l R (1 2a) where R is the sample-covariance matrix defined by

3 WAX AND KAILATH: DETECTION OF SIGNALS BY INFORMATION THEORETIC CRITERIA 389 The maximum-likelihood estimate is the value of dk) that maximizes (12). Following Anderson [3], these estimates are given by A hi=li i=l,.-.,k (134 where 1, > la * * * > lp and CI, *, Cp are the eigenvalues a,nd eigenvectors, respectively, of the sample covariance matrix R. Substituting the maximum likelihood estimates (13) in the log-likelihood (12), with some straightforward manipulations, we obtain The number of signals d^ is determined as the value of kg (0, 1, *. -, p - I} for which either the AIC or the MDLis minimized. V. CONSISTENCY OF THE CRITERIA We have described two different criteria for estimating number of signals. The natural question is which one should be preferred. What can be said about the goodness of the estimates obtained by these criteria. One possible benchmark test is the behavior as the sample size increases. One would prefer an estimator that yields the true number of signals with probability one as the sample size increases to infinity. An estimator with this property is said to be consistent. By generalizing a method of proof given in Rissanen [18] and Hannan and Quinn [9], we shall show that the MDL yields a consistent estimate, while the AIC yields an inconsistent estimate that tends, asymptotically, to overestimate the number of signals. The consistency of the MDL is proved by showing that in the large-sample limit, MDL(k) is minimized for k = q. Taking first k < 4, it follows from (17) that Note that the term in the bracket is the ratio of the geometric mean to the arithmetic mean of the smallest p -.k eigenvalues. The number of free parameters in dk) is obtained by counting the number of degrees of freedom of the space spanned by dk). Recalling that the eigenvalues of a complex covariance matrix are real but that the eigenvectors are complex, it follows that dk) has kt 1 t'2pk parameters. However, not all of the parameters are independently adjusted; the eigenvectors are constrained to have unit norm and to be mutually orthogonal. This amounts to reduction of 2 k degrees of freedom due to the normalization and 2&k(k - 1) degrees of freedom due to the mutua1.orthogonaiization. Thus, we obtain (number 'of free adjusted parameters) = k t 1 t 2pk - [ik(k - l)] = k(2p - k) t 1. (15) The form of AIC for this problem is therefore given by 1 - [MDL(q) - MDL(k)] N = log t log i=k+l i=k+l q-k k) N t 2k(2p - k) Since the eigenvalues of the sample-covariance matrix lj (i = 1, * *,4) are consistent estimates of the eigenvalues of the true + covariance matrix hi, it follows that in the large-sample limit the eigenvalues li (i = k t 1, -., q) are not ali equal with probability one. Therefore, by the arithmetic-mean geometricmean inequality it follows that in the large-sample limit while the MDL criterion is given by This implies that the first term in (18) is negative with probability one in' the large-sample limit'. Similarly, by the generalized arithmetic-mean geometric-mean inequality k(2p - k) log N. 2 w,a1 + w 4, >AY'A,W2 w 1 t wa = 1 (20) it follows that in the large sample limit

4 ) = 390 IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-33, NO. 2, APRIL 1985 This implies that the second term in (1 8) is also negative with probability one in the large-sample limit. Now, since the last term in (18) goes to zero as the sample size increases, it follows that the difference [MDL(q) - MDYk)] is negative with probability one in the large-sample limit for k < q. Taking now k > 4, it follows from (17) that 2 [MDL(k) - MDL(q)] = (k - 4)(2p - k - 4) log N zero. Hence, the AIC tends to overestimate the number of signals q in the large-sample limit. We should note that from the analysis above it follows that any criteria of the form -log f(xl4) t Q(N) k (23) where or(n) + 00 and or(n)/n+ 0 yields a consistent estimate of the number of signals. VI. SIMULATION RESULTS In this section we present simulation results that illustrate the performance of our method when applied to sensor array processing. By the well-known duality between spatial frequency and temporal frequency, these examples can also be interpreted in the context of harmonic retrieval. The examples refer to a uniform linear array of p sensors with 4 incoherent sinusoidal plane waves impinging from directions {G1,..., Gq}. Assuming that the spacing between the sensors is equal to half the wavelength of the impinging wavefronts, the vector of the received signal at the array is then given by Note that the terms in the curly bracket are twice the loglikelihoods of the maximum likelihood estimator under the hypotheses that the rank of \k is q and k, respectively. Thus, their difference is the likelihood-ratio for deciding between these two hypotheses. From the general theory of likelihood ratios (see, e.g., Cox and Hinkley [8]) it follows that the asymptotic distribution of this statistic is x with number of degrees of freedom equal to the difference of the dimensions of the parameter spaces under the two hypothesis, i.e., [WP - k) M2P - 4) + 11 = (k - q)(2p - k - 4). Thus, as the sample size increase, the probability that the term in the curly bracket in (20) exceeds the first term in (20) is given by the area in the tail from (k - 4)(2p - k - q) log N of the mentioned x2 distribution with (k - q)(2p - k - 4) degrees of freedom. Since the area in this tail approaches zero as the sample size increases, it follows that in the large-sample limit the difference [MDL(k) - MDL(q)] is positive with probability one for k > q. Combining this with the previous result for k < q, it follows that MDL(k) has a minimum at k = q. Repeating the above arguments for the AIC, it follows that in the large-sample limit and for k <q, the difference [AIC(q) - AIC(k)] is negative with probability one. However, for k > 4, the difference [AIC(k) - AIC(q)] has nonzero probability to be negative even in the large-sample limit, since the tail from (k - 4)(2p - k - q t 1) of the x2 distribution with (k - q)(2p - k - 4 t 1) degrees of freedom is definitely not 9 x(t) = 2 A (G~) t n(t) (24a) k=1 where A(&) is the p X 1 direction vector wavefront. with and A($~)T = [le-jtk... e-j(q-l)tk 1 7k = 77 sin q5k of the kth (24b) (24c) Q( e ) = random phase uniformly distributed on (0, 277) n( 0 vector of white noise with mean zero and covariance u21. Note that this model is a special case of the general model presented in (1). In the first example, we considered an array with seven sensors ( p = 7) and two sources (4 = 2) with directions-of-arrivals 20 and 25. The signal-to-noise ratio, defined as 10 log 1 /2u2, was 10 db. Using N= 100 samples, the resulted eigenvalues of the sample-covariance matrix were , , , , 1,0544, , and Observing the gradual decrease of the eigenvalues it is clear that the separation of the 3 smallest eigenvalues from the 2 large ones is a difficult task in which a naive approach is likely to fail. However, applying the new approach we have presented above yielded the following values for the AIC and MDL AIC MDL The minimum of both the AIC and the MDL is obtained, as expected, for the 4 = 2. In the second example, we added another source at 10 to the scenario described in the first example. The eigenvalues of the sample-covariance in this cpe were , , , , , , and The resulting values of the AIC and the MDL were as follows.

5 & AX AND KAILATH: DETECTION OF SIGNALS BY INFORMATION THEORETIC CRITERIA AIC MDL In this case, the minimum value of both the AIC and the MDL is obtained, incorrectly, for 4 = 2. The failure of the AIC and MDL in this case is because of the inherent limitations of the problem. Indeed, repeating the example with signal-to-noise ratio of 10 db yielded the following eigenvalues , ,0.3786,0.1372,0.1109,0.0946, and , and consequentially the following values of the AIC and the MDL AIC MDL Now the minimum of both the AIC and the MDL is obtained, correctly, for q = 3. VII. EXTENSION TO THE FREQUENCY DOMAIN The starting point of our approach was the time domain relation (1). However, in certain cases, especially in array processing, the frequency domain is more natural. As we shall show, our approach is easily extended to handle frequency-domain observations. Consider the frequency domain analog of the time domain model (1); the observed vector is a Fourier coefficients vector expressed by 4 dwn) = A(mn 3 ai) si(wn) + n(wn> (25) i= 1 where si(wn)= the Fourier coefficient of the ith signal at frequency on, A(w,, ai) = a p X 1 complex vector, determined by the parameter vector Qii associated with the ith signal n(wn) = a p X 1 complex vector of the Fourier coefficients of the additive noise. The problem, as in the time domain, is to estimate the number of signals 4 from L samples of the Fourier-coefficients vectorxl(on),,xl(an)- By the well-known analogy between multivariate analysis and time-series analysis (see, e.g., Wahba [23]), the time- domain approach carries over to the frequency domain with the role of the sample-covariance matrix played by the periodogram estimate of the spectral density matrix, given by Thus, the frequency-domain version of the AIC is given by f V where Zl (on) * > lp(wn) are the eigenvalu? of the periodogram estimate of the spectral-density matrix K(wn). When the signals are wide band, namely, occupy several frequency bins, say, 01,.. -, OZ+M, the information on the number of signals is contained in all the M frequency bins. Assuming that the observation time is much larger than the correlation times of the signals, it follows that the Fourier coefficients corresponding to different frequencies are statistically independent (see, e.g., Whalen [26, p. 811). The AIC for detecting the number of wide-band signals that occupy the frequency band wz, e *., WI+M, is given by the sum of (27) over the frequency range of interest i- 2M[k(2p - k)]. (2 8) The corresponding expression similarly derived. for the MDL criterion can be VIII. CONCLUDING REMARKS A new approach to the detection of the number of signals in a multichannel time series has been presented. The approach is based on the application of the AIC and MDL information theoretic criteria for model selection. Unlike the conventional hypothesis test based approach, the new approach does not require any subjective threshold settings; the number of signals is determined merely by minimizing either the AIC or the MDL criterion. It has been shown that the MDL criterion yields a consistent estimate of the number of signals, while the AIC yields an inconsistent estimate that tends, asymptotically, to overestimate the number of signals. The detection problem addressed in this paper is usually a part of a more complex combined detection-estimation problem, where one wants to estimate the number as well as the parameters Q1, * *., Qi, of the signals in (1). Because of the computational complexity involved, the problem is usually solved in two steps: first the number of signals is detected, and then, with an estimate of the number of signals 6 at hand, the parameters of the signals are estimated. It should be pointed out, however, that in principle the AIC and MDL criteria can be applied to the combine detection-estimation problem [26]. The evaluation of the AIC and MDL in this case in- ;elves th% computation of the maximum likelihood estimates 4 1,. * *,@k, for every possible number of signals k E (0, 1, * *., p - 1). Solving this highly nonlinear problem for all Valuesof k is computationally very expensive. Nevertheless, if one is ready to pay the price, the gain in term of performance, especially in difficult situations such as low signal-to-noise ratio, small sample size, or closely spaced signals, may be significant. REFERENCES [ 11 H. Akaike, Information theory and an extension of the maximum likelihood principle, in hoc. 2nd Int. Symp. Inform.

6 392 IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOL. ASSP-33, NO. 2, APRIL 1985 Theory, suppl. Problems of Control and Inform. Theory, 1973, pp [ 21 -, A new look at the statistical model identification, IEEE Trans. Automat. Contr., vol. AC-19, pp , [3] T. W. Anderson, Asymptotic theory for principal component analysis, Ann. J. Math. Stat., vol. 34,, pp , [4] M:S. Bartlett, A note on the multiplying factors for various x2 approximations, J. Roy. Stat. SOC., ser. E, vol. 16, pp , [5] G. Bienvenu and L. Kopp, Adaptivity to background noise spatial coherence for high resolution passive methods, in Proc. ICASSP 80, Denver, CO, 1980, pp [6] T.P. Bronez and J. A. Cadzow, An algebraic approach to super resolution array processing, IEEE Trans. Aerosp. Electron. Syst., VO~. AES-19, pp , [7] A. Bruckstein, T. J. Shan, and T. Kailath, The resolution of overlapping echoes, IEEE Trans. Acoust., Speech, Signal Processing, submitted for publication. [8] D. R. Cox and D. V. Hinkley, Theoretical Statistics. London, England: Chapman and Hall, [9] E. J. Hannan and B. G. Quinn, The determination of the order of an autoregression, J. Roy. Stat. Soc., ser. E, vol. 41, pp ,1979. [lo] D. H. Johnson and S. R. Degraff, Improving the resolution of bearing in passive sonar arrays by eigenvalue analysis, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-30, pp ,1982. [ 111 R. Kumaresan and D. W. Tufts, Estimating the parameters of exponentially damped sinusoids and pole-zero modeling in noise, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-30, pp ,1983. [ 121 -, Estimation of arrival of multiple plane waves, IEEE Trans. Aerosp. Electron. Syst., vol. AES-19, pp , [13] S. Y. Kung and Y. H. Hu, Improved Pisarenko s sinusoidal spectrum estimate via the SVD subspace approximation method, inproc. 21 CDC, Orlando, FL, 1982,pp [14] D. N. Lawley, Tests of significance of the latent roots of the covariance and correlation matrices, Biometrica, vol. 43, pp ,1956. [ 151 N. L. Owsley, Spectral signal set extraction, Aspects of Signal Processing: Part II, G. Tacconi, Ed., Dordecht, Holland: Reidel, pp ,1977. [16] V. F. F isarenko, The retrieval of harmonics from a covariance function, Geophys. J. Roy. Astron. Soc., vol. 33, pp , [ 171 J. Rissanen, Modeling by shortest data description, Automatica, VO~. 14, pp ,1978. [18] -, Consistent order estimation of autoregressive processes by shortest description of data, Analysis and Optimization of Stochastic Systems, Jacobs et al. Eds. New York: Academic, [19] -, A universal prior for the integers and estimation by minimum descriptionlength, Ann. Stat.,vol. ll,pp , I201 R. 0. Schmidt, Multiple emitter location and signal parameter estimation, in Pvoc. RADC Spectral Estimation Workshop, Rome, NY, 1979, pp [21] G. Schwartz, Estimating the dimension of a model, Ann. Stat., VO~. 6, pp ,1978. [22] G. Su and M. Morf, The signal subspace approach for multiple emitter location, in Proc. 16th Asilomar Con$ Circuits Syst. Comput., Pacific Grove, CA, 1982, pp [23] G. Wahba, On the distribution of some statistics useful in the analysis of jointly stationary time-series, Ann. Math. Stat., vol. 39, pp ,1968. [24] M. Wax, R. 0. Schmidt, and T. Kailath, Eigenstructure method for retrieving the poles from the natural response, in Proc. 22nd CDC, San Antonio, TX, 1983, pp [25] M. Wax, T. J. Shan, and T. Kailath, Spatio-temporal spectral analysis by eigenstructure methods, IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-32, pp ,1984. [26] M. Wax, Detection and estimation of superimposed signals, Ph.D. dissertation, Stanford Univ., Stanford, CA, Mar [27] A.D. Whalen, Detection of Signals in Noise. New York: Academic, 1971.