Lecture 19: Bayesian Linear Estimators

Transcription

1 ECE 830 Fall 2010 Statistical Signal Processing instructor: R Nowa, scribe: I Rosado-Mendez Lecture 19: Bayesian Linear Estimators 1 Linear Minimum Mean-Square Estimator Suppose our data is set X R n, which is considered to be a random vector governed by a distribution p(x θ), which depends on the parameter θ Moreover, the parameter θ R is treated as a random variable with E[θ] = 0 and E[θθ T ] = Σ θθ Also, assume that E[x] = 0 and let Σ xx := E[xx T ] and Σ θx := E[θx T ] Then, as we saw in the previous lecture, the best linear estimator of θ is given by:  = arg min E [ θ A T x 2 ] 2 A R n  = Σ 1 xx Σ xθ As a consequence, our Linear Minimum Mean Square Error Estimator or LMMSE estimator becomes: 2 Orthogonality Principle ˆθ = ÂT x = Σ θx Σ 1 xx x Let ˆθ = Σ θx Σ 1 xx x be the LMMSE estimator, defined above Then E [ (θ ˆθ) T x ] = E [ tr ( )] T θ ˆθ)x E [ (θ ˆθ) T x ] = tr ( Σ θx Σ θx Σ 1 ) xx Σ xx E [ (θ ˆθ) T x ] = 0 In other words, the error (θ ˆθ) is orghogonal to the data x This is shown graphically in Fig 1 It is important to note that if ˆθ = A T x, then, by the orthogonality principle which is the previously derived LMMSE estimator 0 = E [ (θ ˆθ)x T ] = Σ θx A T Σ xx A T = Σ θx Σ 1 xx 1

2 Lecture 19: Bayesian Linear Estimators 2 21 Linear signal model Figure 1: Orthogonality between the estimator ˆθ and its error θ ˆθ Suppose we model our detected signal as X = Hθ + W, where X and W R n, θ R, H n is a nown linear transformation, and W is a noise process Furhtermore we now that E[w] = 0, E[ww T ] = σ 2 wi n n E[θ] = 0, E[θθ T ] = σ 2 θi In addition, we now that the parameter and the noise process are uncorrelated, ie, E[θw T ] = E[wθ T ]0 As demonstrated before, the LMMSE estimator is ˆθ = Σ 1 xx Σ xθ x where Σ 1 xx and Σ xθ can be obtained as follows: Therefore, the LMMSE estimator is given by Σ xθ = E[θx T ] = E[θ(Hθ + w) T ] = σ 2 θh T Σ xx = E[xx T ] = E[(Hθ + w)(hθ + w) T ] = σ 2 θhh T + σ 2 w ˆθ = σ 2 θh T (σ 2 θhh T + σ 2 wi n n ) 1 x ˆθ = H T (HH T + σ2 w σθ 2 I n n ) 1 x When does the LMMSE estimator minimize the Bayes MSE amog all possible estimators? When is the linear estimar optimal? The LMMSE esteimator is optimal, ie, it is the minimum Bayesian MSE estimator when the Maximum A Posteriori estimator is linear

3 Lecture 19: Bayesian Linear Estimators 3 3 Gauss-Marov Theorem Let X and Y be jointly Gaussian random vectors, whose joint distribution can be expressed as [ X Y ] N then the conditional distribution of Y given X is ([ µx ] [ ]) Σxx Σ, xy µ y Σ yx Σ yy Y X N ( µ y + Σ yx Σ 1 xx (x µ x ), Σ yy Σ yx Σ 1 xx Σ xy ) 31 Application to the Linear Signal Model We model the detected signal as X = Hθ + W where W N (0, σ 2 wi n n ) and θ N (0, σ 2 θ I ) Then, the vector [Xθ] T is a multivarian Gaussian random vector As we saw in previous lectures, the Bayesian MSE is minimized by the posterior mean E[θ X] which, in this case, using the Gauss-Marov theorem, is E[θ x] = µ θ + Σ θx Σ 1 xx (x µ x ) E[θ x] = 0 + σ 2 θh T (σ 2 θhh T + σ 2 wi n n ) 1 (x 0) E[θ x] = σ 2 θh T (σ 2 θhh T + σ 2 wi n n ) 1 x, which is the previously derived LMMSE estimator Therefore, Gaussian case linear estimators are optimal in the 32 Proof of the Gauss-Marov theorem Without loss of generality assume that X and Y are zero-mean random vectors Therefore P (Y X) = P (X, Y ) P (X) [ ] = (2π) n/2 (2π) n/2 Σ 1 exp{ [ 1 2 x y Σ 1 x y ] T } (2π) n/2 Σ xx 1 exp{ 1 2 xt Σ 1 xx x} where Σ = [ ] Σxx Σ xy Σ yx Σ yy To simplify the formula we need to determine Σ 1 The inverse can be written as: where [ ] 1 [ ] Σxx Σ xy Σ 1 xx 0 = 0 0 Σ yx Σ yy [ ] Σ 1 xx Σ + xy Q 1 [ Σ I yx Σ 1 xx I ] Q := Σ yy Σ yx Σ 1 xx Σ xy

4 Lecture 19: Bayesian Linear Estimators 4 This formula for the inverse is easily verified by multiplying it by Σ to get the identity matrix Substituting the inverse into P (Y X) yields P (X Y ) = (2π) n/2 Q 1 exp{ 1 2 (y Σ yxσ 1 xx x) T Q 1 (y Σ yx Σ 1 xx x)} which shows that Y X N (Σ yx Σ 1 xx x, Q) For the general case when E[X] = µ x and E[Y ] = µ y then 4 The Wiener Filter (Y µ y ) (X µ x ) N (Σ yx Σ 1 xx (x µ x ), Q) Y X N (µ y + Σ yx Σ 1 xx (x µ x ), Q) When the expected values of the parameter θ R and the data x R n are zero, then the Wiener filter A opt is obtained by minimizing the mean square error between the parameter and estimator: A opt = arg min E [ θ Ax 2 ] 2 A:ˆθ=Ax which results in A opt = Σ θx Σ 1 xx, involving second order moments and which becomes the optimal estimator when both the data and the parameter are jointly Gaussian distributed 41 Signal + Noise Model We model our detected signal as X = S +W where the noiseless signal S (our parameter) follows a Gaussian distribution N (0, Σ ss ) and W N (0, Σ ww ) In addition, S and W are uncorrelated Therefore, the data vector X N (0, Σ ss +Σ ww ) and E[sx T ] = E[s(s+w) T ] = Σ ss From here, the LMMSE estimator ŝ becomes: 42 Linear Signal + Noise Model ŝ = Σ ss (Σ ss + Σ ww ) 1 x Now we assume that the detected signal can be model as X = Hθ + W where now θ N (0, Σ θθ ) and W N (0, Σ ww ) where θ R and W R n (which are uncorrelated) and H is an n linear transformation matrix Therefore X N (0, HΣ θθ H T + Σ ww ) In addition, E[θx T ] = E[θ(Hθ + w) T ] = Σ θθ H T and the estimator becomes ˆθ = Σ θθ H T (HΣ θθ H T + Σ ww ) 1 x Now suppose that Σ θθ = σ 2 θ I and Σ ww = σ 2 wi n n, then ˆθ = σ 2 θh T (σ 2 θhh T + σ 2 wi n n ) 1 x and the LMMSE estimator of the noisless signal becomes

5 Lecture 19: Bayesian Linear Estimators 5 ŝ = H ˆθ = σ 2 θhh T (σ 2 θhh T + σ 2 wi n n ) 1 x In some cases HH T can be diagonalized under an orthonormal transformation U (its columns are orthonormal to each other) in such a way that only the first elements of the diagonal are nonzero As a consequence λ HH T = U 0 λ U T = UDU T ŝ = σ 2 θudu T (σ 2 θudu T + σ 2 wi n n ) 1 x ŝ = σ 2 θudu T (σ 2 θudu T + σ 2 wuu T ) 1 x ŝ = σ 2 θudu T (U [ σ 2 θd + σ 2 wi n n ] U T ) 1 x ŝ = σ 2 θud [ σ 2 θd + σ 2 wi n n ] 1U T x ŝ = U(σ 2 θd [ σ 2 θd + σ 2 wi n n ] 1)U T x Note that the term in parenthesis reduces to a diagonal matrix of the form σ 2 θ λ σθ 2 λ1+σ2 w σ 0 2 θ λ σθ 2 λ 0 0 +σw which, as σ 2 w/σ 2 θ tends to zero, converges to and ŝ P H x

6 Lecture 19: Bayesian Linear Estimators 6 43 Frequency Domain Wiener Filter In tihs part we review the signal + noise example (41) but approaching the problem from Fourier domain Again, our model is X = S + W and now we tae the DFT of both sides: U T X = U T S + U T W X = S + W where S N (0, Λ s ) and W N (0, Λ w ) Equivalently X = UU T S + UU T W = S + W so S N (0, UΛ s U T ) and W N (0, UΛ w U T ) Therefore, the Wiener-Fielter estimator becomes ŝ = U ŝ = Σ ss (Σ ss + Σ ww ) 1 x ŝ = UΛ s U T (U[Λ s + Λ w ]U T ) 1 x ŝ = UΛ s U T U[Λ s + Λ w ] 1 U T x σ 2 1 σ1 2+γ ŝ = UΛ s [Λ s + Λ w ] 1 U T x σ 2 i 0 0 σi 2+γ2 i σ 2 n σ 2 n +γ2 n U T x where σ 2 j and γ2 j are the jt h elements of the diagonal matrices Λ s and Λ w, respectively Therefore the filtering process can be synthesized by the following algorithm: 1 Tae the DFT of the measured signal 2 Attenuate each frequency component according to 1 1+SNR 1 j 3 Tae the inverse DFT of the attenuated spectrum 44 Classical derivation of the Wiener Filter at frequency ω j, where SNR j = σ 2 j /γ2 j Again, we start with the model X = S+W where X, S, W are wide-sense stationary processes We re-express them as time series x[n] = s[n] + w[n] We aim at defining a filter h[n] that will be convolved with x[n] to etstimate s[n]

7 Lecture 19: Bayesian Linear Estimators 7 ŝ[n] = Σ h[]x[n ] Our filter should minimize the MSE: MSE(ŝ[n]) = E [ (s[n] ŝ[n]) 2] MSE(ŝ[n]) = E [ (s[n] 2 2s[n]Σ h[]x[n ] + (Σ h[]x[n ]) 2] Differentiating with respect to h[m] and maing the derivative equal to zero MSE(ŝ[n]) h[m] MSE(ŝ[n]) h[m] MSE(ŝ[n]) h[m] = E [ 2s[n]h[m] + 2(Σ h[]x[n ])x[n m] ] = 2R sx [m] + 2(Σ h[]r xx [m ]) ( ) = 2R ss [m] + 2 Σh[](R ss [m ] + R ww [m ]) = 0 Therefore the optimal filter satisfies R ss [m] = Σ h[](r ss [m ] + R ww [m ]), which is just the familiar Wiener-Hopf equation Taing the DFT of both sides, we get S ss (ω) = H(ω) ( S ss (ω) + S ww (ω) ) where S ss (ω) and S ww (ω) are the power spectra of the signal and the noise process, respectively Therefore, the filter becomes: 5 Deconvolution H(ω) = S ss (ω) S ss (ω) + S ww (ω) The final topic of this lecture is deconvolution We model the detected signal as X = GS + W where G is a circular convolution operator (a blurring transformation, shown in Fig 2) As in the previous sections S N (0, UΛ s U T ) and W N (0, UΛ w U T ) Furthermore, since G is circulant, G = UDU T, where D is a diagonal matrix, which is basically the frequency response of G In this case, the Wiener filter solution is computed as follows: ŝ = Σ ss G T (GΣ ss G T + Σ ww ) 1 x ŝ = UΛ s U T G T (GUΛ s U T G T + UΛ w U T ) 1 x ŝ = UΛ s U T UD T U T (UDU T UΛ s U T UD T U T + UΛ w U T ) 1 x ŝ = UΛ s D T (DΛ s D T + Λ w ) 1 U T x

8 Lecture 19: Bayesian Linear Estimators 8 Figure 2: Blurring process (a) Original impulse signal (b) Blurring function (c) Blurred signal ŝ = U DU T x, where D = D T D 2 + P 1 and P = Λ s(, ) Λ w (, ) Do not forget that the transpose operator wors as the conjugate transpose operator when the matrix has complex elements 51 Classical Wiener Filter Following a derivation simmilar to that of Section 44, in the case of a blurred, noise time series modeled as x[n] = g[n] s[n] + w[n] we aim at obtaining a filter h[n] such that the estimator of the deblurred, noisless signal is computed from ŝ[n] = Σ h[]x[n ] The resulting filter in Fourier domain is: H(ω) = G (ω)s ss (ω) G(ω) 2 S ss (ω) + S ww (ω) where G(ω) is the transfer function of the blurring filter g[n] and G is its complex conjugate