26. Filtering. ECE 830, Spring 2014

Transcription

1 26. Filtering ECE 830, Spring / 26

2 Wiener Filtering Wiener filtering is the application of LMMSE estimation to recovery of a signal in additive noise under wide sense sationarity assumptions. Problem statement: x[n] = s[n] + w[n]. We observe x[n], x[n 1],..., x[n p + 1], and would like to estimate θ = s[n + D] where D is an integer, using a linear estimator p 1 θ = ŝ[n + D] = h p [k]x[n k]. k=0 2 / 26

3 There are three cases of interest: D = 0 (filtering) D > 0 (signal prediction) D < 0 (smoothing) Assumptions: 1. We will assume all 1st and 2nd order moments are known. 2. s[n] and w[n] are zero-mean. 3. x[n] is wide-sense stationary (WSS) with autocorrelation r xx [k] = E[x[n]x[n + k]] 4. x[n] and s[n] are jointly WSS with cross-correlation r xs [k] = E[x[n]s[n + k]] Example: These conditions hold when s[n] and w[n] are zero-mean, WSS, and uncorrelated. 3 / 26

4 Gauss-Markov Theorem and Wiener Filtering Let x and θ be jointly Gaussian distributed: [ ] ([ ] [ x µx Rxx R N, xθ θ µ θ R θx R θθ ]) Then the conditional distribution of θ given x is θ x N ( µ θ + R θx R 1 xx (x µ x ), Q ), where Q = R θθ R θx R 1 xx R θx We know that the conditional mean θ = µ θ + R θx R 1 xx (x µ x ) is the best estimate of θ given x in a (mean) squared error sense. So, in theory, we can compute the Wiener filter. In practice, however, we want a fast, online algorithm for computing and updating θ as data streams in. 4 / 26

5 Direct Optimization The Wiener filter can also be derived from a direct optimization perspective. If x = θ + w as above, and we restrict our attention to estimators of the form θ = Gx for some linear operator G, we can try to find the G which minimizes the MSE: G = arg min G MSE = E [ (θ Gx) T (θ Gx) ] E [ (θ Gx) T (θ Gx) ] = E [ tr ( (θ Gx)(θ Gx) T )] = tr E [ (θ Gx)(θ Gx) T ] Minimizing MSE is equivalent to minimizing (if µ θ = 0 and µ x = 0) ɛ 2 = tr E [ (θ Gx)(θ Gx) T ] = Taking the gradient with respect to G dɛ 2 dg = 2R θx + 2GR xx = 0 G = R θx Rxx 1 = R θθ (R θθ + R ww ) 1 The problem is that it s not always easy to compute R θx R 1 xx, especially in streaming data applications. 5 / 26

6 Adaptive Filtering Applications Channel/System Identification Noise Cancellation - suppression of maternal ECG component in fetal ECG. ŷ is an estimate of the maternal ECG signal present in abdominal signal. 6 / 26

7 7 / 26

8 Channel Equalization: Adaptive Controller: 8 / 26

9 Iterative Minimization Most adaptive filtering algorithms (LMS included) are modifications of standard iterative procedures for solving minimization problems in a real-time or on-line fashion. Therefore, before deriving the LMS algorithm we will look at iterative methods of minimizing error criteria such as MSE. Consider the following set-up: at k th time step, we have x k : observation A linear estimator has the form θ k : signal to be estimated θ k = h 1 x k + h 2 x k h p x k p+1 This can be thought of as an FIR filter applied to x, where the impulse response of filter is..., 0, 0, h 1, h 2,..., h p, 0, 0,... 9 / 26

10 Vector Notation: θ k = x T k h x k = [ x k x k 1 x k p+1 ] T h = [ h 1 h 2 h p ] T Error Signal: Assumptions: e k = y k θ k = y k x T k h (x k, θ k ) are jointly stationary with zero-mean MSE: E [ e 2 ] x = E [ (θ k x T k h) 2] = E [ θ 2 k] 2h T E [x k θ k ] + h T E [ x k x T k ] hk = R θθ 2h T R xθ + h T R xx h R θθ = variance of θ k R xx = covariance matrix of x k R xθ = E [x k θ k ] = cross-covariance vector 10 / 26

11 Note: The MSE is quadratic in h MSE surface is bowl shaped with a unique minimum point Optimal Filter Minimize MSE: de[e2 k] dh = 2R xθ + 2R xx h = 0 h opt = R 1 xx R xθ This weight vector has similar to form of Wiener filter. Notice that we can re-write the optimization as E [ x k x T k h] = E [x k θ k ] or E[x k (θ k x T k }{{ h )] = 0 } e k which shows that the error signal is orthogonal to the input x k. 11 / 26

12 Steepest Descent Although we can easily determine h opt by solving the system of equations R xx h = R xθ let s look at an iterative procedure for solving this problem. This will set the stage for our adaptive filtering algorithm. We want to minimize the MSE. The idea is simple. Starting at some initial weight vector h 0, iteratively adjust the values to decrease the MSE. We want to move h 0 towards the optimal vector h opt. In order to move in the correct direction, we must move downhill or in the direction opposite to the gradient of the MSE surface at the point h / 26

13 Thus a natural and simple adjustment takes the form h k = h k 1 µ de[e 2 k ] 2 dh h=hk 1 where µ is the step size - tells us how far to move in negative gradient direction. We can repeatedly update h Hopefully each subsequent h k is closer to h opt. We have several questions we need to address: Does this procedure converge? Will it always get to the bottom? Can we adapt it to an on-line, real-time, dynamic situation in which the signals may not be stationary? 13 / 26

14 Example: Scalar Case Estimate θ k based on observation x k using a 1-tap filter. That is θ k = hx k (cost function) = C(h) = E [ e 2 ] [ k = E (θk hx k ) 2] = E [ θk] 2 2hE [θk x k ] + h 2 E [ x 2 ] k Let s assume: then = R θθ 2hR xθ + h 2 R xx R θθ = 1 R xθ = 1 R xx = 2 C(h) = 1 2h + 2h 2 h opt = arg min h C(h) = / 26

15 Example: (cont.) Rather than solving for h opt directly, let s try the method of steepest descent. First note that Initial Guess: h 0 = 2 Step Size: µ = 1 4 dc(h) dh = 2 + 4h 15 / 26

16 Example: (cont.) h 1 = h µ( 2 + h 0 4) = 2 1 ( 2 + 8) = h 2 = ( ) = h 3 = h 4 = h 5 = h 6 = h 7 = h = Converges to minimizer h opt = 1 2! 16 / 26

17 We can summarize our observations 1. Too large of a step size can lead to divergence, rather than convergence. 2. Too small a step size leads to extremely slow convergence. How can we choose a reasonable step size? 17 / 26

18 Example: Vector Case Observations : x k Signal : θ k Estimator : θ k = h 1 x k + h 2 x k h p x k p+1 h = [ h 1 h 2 h p ] T x k = [ x k x k 1 x k p+1 ] T Cost: C(h) = MSE = E [ e 2 ] [ k = E (θk x T k h)2] = R θθ 2h T R xθ + h T R xx h quadratic in h (strictly) convex Gradient: dc(h) dh = dc(h) dh 1. dc(h) dh p = 2R xθ + 2R xx h 18 / 26

19 Example: (cont.) SD Algorithm: h k+1 = h k 1 2 µ ( 2R xθ + 2R xx h k ) [ ] [ Suppose p = 2, R xθ =, R 1 θθ = 1, and R xx = 0 3 that [ ] h opt = Rxx 1 1/2 R xθ = 1/3 SD (Steepest Descent) Algorithm: h k+1 = h k 1 [ 1 ( 2 2 µ 1 = [ ] µ + µ ] [ [ 1 2µ µ ] h k ] h k ) ], so 19 / 26

20 20 / 26

21 Step Size and Convergence Let s focus on the convergence (or divergence) of steepest descent in the MSE minimization problem we have been looking at. h k = h k µ ( 2R xθ + 2R xx h k 1 ) = h k 1 µ (R xx h k 1 R xθ ) h k h k h k 1 = µ (R xθ R xx h k 1 ) Recall that h opt = R 1 xx R xθ and define v k h k h opt, so we can write h k =µr xx (h opt h k 1 ) = µr xx v k 1 v k =v k v k 1 =h k h opt h k 1 + h opt = h k = µr xx v k 1 Note that if the SD algorithm converges, then h k h opt (v k 0) as k 21 / 26

22 v k = v k 1 µr xx v k 1 = (I µr xx ) v k 1 This is the so-called weight error difference equation and its stability/convergence can be analyzed as follows. Let s first diagonalize the system by computing the eigendecomposition of R xx. R xx = UDU T, D = Now we have v k = ( I µudu T ) v k 1 λ λ p U = [ ] u 1 u p = (UU T µudu T )v k 1 = U(I µd)u T v k 1 22 / 26

23 Now define z k = U T v k z k = U T v k = U T U(I µd)u T v k 1 = (I µd)z k 1 z (1) ḳ 1 µλ 1 0 z (1) = =... k 1. z (p) k z (l) k = (1 µλ l )z (l) k µλ p l = 1, 2,..., p In order for v k 0 we must have z k 0, which happens if zk l < z(l) k 1 1 µλ l z (l) k 1 < zl k 1, l = 1,, p 1 µλ l < 1, l = 1,, p 0 < µλ min < µλ max < 2. z (p) k 1 Thus, we have two requirements: 1. λ min > 0 R xx is full rank (invertible) 2. µ < 2 λ max 23 / 26

24 Summary: The SD algorithm h k+1 = h k 1 2 µ ( 2R xθ + 2R xx h k ) converges to the unique, global minimizer h opt of C(h) = MSE(h) = E[e 2 k (h)] = E [ (y k h T x k ) 2] for every initial point h 0 if and only if λ min > 0 (R xx is full rank) AND µ < 2 λ max 24 / 26

25 The Least-Mean-Square (LMS) Algorithm In LMS we essentially adopt a Steepest Descent type algorithm, but we replace MSE E[e 2 k ] with the instantaneous error squared: e 2 k (h) = ( y k x T k h) 2. From our previous discussion of SD algorithms we are led to an adaptive algorithm as follows: h k = h k µ de2 k (h) dh de 2 k (h) dh = 2 ( y k x T k h) x k h=hk 1 = h k = h k 1 µx k e k µ gain (step size) x k gradient e k instantaneous error = y k x T k h k 1 25 / 26

26 Convergence of LMS Convergence in Mean if: (i) λ min (R xx ) > 0 (ii) µ < 2 λ max(r xx) Convergence in MSE: (i) λ min (R xx ) > 0 (ii) µ < 2/3 λ max(r xx) Adaptation to Nonstationarity lim k E[h k] = h opt lim E[(h opt h k ) 2 ] = 0 k Smaller µ slower adaptation 26 / 26