Notes on Linear Minimum Mean Square Error Estimators

Transcription

1 Notes on Linear Minimum Mean Square Error Estimators Ça gatay Candan January, 0 Abstract Some connections between linear minimum mean square error estimators, maximum output SNR filters and the least square solutions are presented. The notes hae been prepared to be distributed with EE 503 (METU, Electrical Engin.) lecture notes. Linear Minimum Mean Square Error Estimators The following signal model is assumed: r = Hs + () Here r is a N column ector denoting the obserations. In this model, s is the desired signal ector, is the interference ector on the obserations. We assume that s and are uncorrelated as in all stochastic filtering applications. It can be noted that the obserations are linear combinations of desired quantities and interference. Here the matrix H is an N M matrix (N can be less than or greater than M, hence the system can be under or oer determined). H can be considered as the channel or the obseration system or a linear combiner for the modes where the modes are the columns of H. The column ector s contains M entries each of which is to be estimated. In class, we hae deried the optimal linear minimum mean square error estimator as the solution of the following system of equations: R r W = R rs () Here R r = E{rr H } is the auto-correlation matrix of the obserations and R rs = E{rs H } is the cross-correlation matrix of obserations and the desired ariables. The minimum mean square error estimate for s is gien as follows: ŝ = W H r (3) The error coariance matrix for the minimum error estimator is as shown below: R e = E{(s ŝ) (s ŝ) H )} = E{e(s H r H W)} e e H (a) = E{es H } E{er H } W 0 = E{(s W H r)s H } = R s W H R rs (4) The zero matrix on the right hand side of equation shown with (a) is due to the orthogonality condition of the optimal estimator. Next, we explicitly calculate the estimator in terms of H, R s and R matrices. The following can be easily erified: R r = HR s H H + R R rs = HR s (5)

2 Then the optimal filter is gien as follows: The estimator and its error coariance matrix is then W = (HR s H H + R ) HR s (6) Linear Min. MMSE Estimator (Form ) ŝ = W H r = R s H H (HR s H H + R ) r (7) R e = R s R s H H (HR s H H + R ) HR s This concludes the deriation of the estimator. In the following section, we present an alternatie form which is mathematically identical but has some implementation adantages if H matrix is tall and long (N > M or the number of obserations > the number of unknowns) which is the case of interest for most of the time.. An Alternatie Form for Linear MMSE Estimator We follow the deriation gien by Luenberger in [, p.90]. The matrix identity RW H (WRW H + Q) = (W H Q W + R ) W H Q (8) can be proen easily by pre and post multiplying both sides of the relation by W H Q W + R and WRW H + Q respectiely. Using this identity we can write the following: R s H H (HR s H H + R ) = (H H R Replacing the related terms in (7), we get the second form of the estimator: ŝ = W H r = (H H R R e = R s (H H R = (H H R = (H H R s ) H H R (9) Linear Min. MMSE Estimator (Form ) s ) H H R r s ) H H R HR s s ) ( (H H R s )R s H H R ) HR s ) s In many applications the number of obserations is significantly larger than the number of ariables to estimate. As an example, if 0 obserations are collected to estimate ariables; the first form requires 0 0 matrix inersion, the second form requires matrix inersion which makes a lot of difference in the implementation. Maximum SNR Filters In many applications, SNR determines the performance of a system. Therefore its improement is a major goal in signal processing. We assume that N obserations of a WSS random process s[n] are made under additie noise process [n] which is also WSS and uncorrelated with s[n]. The SNR of the obserations is defined as the input SNR. (0) r[n] = s[n] + [n] () (SNR) in = E{s [n]} E{ [n]} = σ s σ () (It should be noted that both s[n] and [n] are assumed to be zero-mean processes as we hae assumed during the lectures.)

3 As we know, the output of a LTI filter with the input r[n] is known to be WSS stationary and its ariance can be calculated as follows. y[n] = N k=0 h k r[n k] = hh r[n] (3) Here we assume that the LTI filter is FIR and with the impulse response of h n. The ectors h and r[n] appearing on the right most side of (3) are column ectors which are defined as h T = [h 0... h N ] T and r[n] T = [r[n] r[n ]... r[n (N )] ] T. The signal and noise term at the filter output can then be expressed as y[n] = h H s[n] + h H [n] signal noise (4) The output SNR becomes as follows: (SNR) out = E{ hh s[n] } E{ h H [n] } = hh R s h h H R h (5) The filter maximizing the output SNR can be found as the solution of the following optimization problem: h = arg max h h H R s h h H R h The filter maximizing this ratio is called as the maximum SNR filter h and it can be shown that it is the eigenector of R R s with the maximum eigenalue. h eigenector of (R R s ) with the maximum eigenalue (7) It should be noted that h is uniquely determined apart from a constant scaling. It is clear that by scaling h with an arbitrary complex constant, we can achiee the same output SNR. More information on the discussed optimization problem can be found in the linear algebra books under the topic of Rayleigh quotient. It should also be noted that h is also called the generalized eigenector of R and R s with the maximum eigenalue. (Matlab s eig.m function has an option of calculating the generalized eigenectors.) This concludes the discussion of maximum-snr filters. Below we examine a special case which naturally emerges in many applications. In many applications, the auto-correlation matrix of s[n] is rank-, that is (6) R s = σ spp H (8) For this special case, the maximum SNR filter and the maximum SNR alue can be written as follows: h R p max(snr) out = p H R p (9) It should be noted that the maximum-snr filter is well known whitened matched filter for this case. It is the matched filter for the white noise case. 3 Connection Between Max-SNR and Min-MMSE Filters for Processes with Rank- Spectral Expansion It is intuitiely clear that filtering the obserations to remoe the noise is helpful to improe the SNR. In this section, we show that the min-mmse filters and Max-SNR filters are (apart from a constant complex scaling) identical for rank- R s matrices. Hence min-mmse filter is equialent to Max-SNR filter in the sense of optimizing the SNR for this type of processes. 3

4 Let s[n] be a rank- process obsered under additie white noise, r = ps + (0) Here r, p and are length N column ectors; s is a zero-mean scalar with ariance σ s and is a complex alued process with zero-mean and coariance matrix R. By substituting H p, R s σ s in (0), we get ŝ = w H MMSE r = p H R p + σ s p H R r. () wmax SNR H This equation shows that the min-mmse filter is the scaled ersion of the max-snr filter. Hence they yield the same output SNR. The minimum error of MMSE filter can also be written from (0): σ e = = (p H R p + ) Max(SNR) out + () Here we hae identified p H R pσ s as the maximum SNR for rank- stochastic signals, see (9). The last equation can be written in the following simple form Max(SNR) out = norm-min-mmse (3) where norm-min-mmse is σ e which is the normalized minimum mean square estimation error. The generalized ersion of similar results are aailable in [, around eq.5]. 4 Connection Between MMSE and Weighted Least Square Filters The minimum mean square error filter always produces an output een when the number of obserations is less than the number of unknowns (undertermined case). This is due to a-priori moment information about the obserations and desired process. As the a-priori information gets looser, that is less informing, the minimum mean square error filter approaches weighted least square (LS) filter. The weighted LS filter is known to be optimal for the estimation of non-random parameters which are obsered with linear models under correlated noise, [3]. The conergence of (MMSE) (weighted LS) can be noted from (0) by substituting R s I: ŝ = WMMSEr H = (H H R H) H H R r (4) weighted LS The last equation is the weighted least square solution of r = Hs equation system with the weight matrix R. The mean square error coariance matrix becomes R e = (H H R H) (5) When R is taken as unit ariance white noise, the classical least square solution emerges. In communications, the LS solution is also known as the zero-forcing solution. References [] D. G. Luenberger, Optimization by ector space methods. Wiley,

5 [] H. Sampath, P. Stoica, and A. Paulraj, Generalized linear precoder and decoder design for MIMO channels using the weighted MMSE criterion, IEEE Transactions on Communications, ol. 49, no., pp , 00. [3] S. M. Kay, Fundamentals of Statistical Signal Processing, Volume : Estimation Theory. Prentice Hall,