Optimal and Adaptive Filtering

Transcription

1 Optimal and Adaptive Filtering Murat Üney Institute for Digital Communications (IDCOM) 26/06/2017 Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

2 Table of Contents 1 Optimal Filtering Optimal filter design Application examples Optimal solution: Wiener-Hopf equations Example: Wiener equaliser 2 Adaptive filtering Introduction Recursive Least Squares Adaptation Least Mean Square Algorithm Applications 3 Optimal signal detection Application examples and optimal hypothesis testing Additive white and coloured noise 4 Summary Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

3 Optimal Filtering Optimal filter design Optimal filter design Observation sequence Linear time invariant system Estimation Figure 1: Optimal filtering scenario. y(n): Observation related to a stationary signal of interest x(n). h(n): The impulse response of an LTI estimator. ˆx(n): Estimate of x(n) given by ˆx(n) = h(n) y(n) = i= h(i)y(n i). Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

4 Optimal Filtering Optimal filter design Optimal filter design Observation sequence Linear time invariant system Estimation Figure 1: Optimal filtering scenario. Find h(n) with the best error performance: e(n) = x(n) ˆx(n) = x(n) h(n) y(n) The error performance is measured by the mean squared error (MSE) [ ( ) ] 2 ξ = E e(n). Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

5 Optimal Filtering Optimal filter design Optimal filter design Observation sequence Linear time invariant system Estimation Figure 1: Optimal filtering scenario. The MSE is a function of h(n), i.e., h = [, h( 2), h( 1), h(0), h(1), h(2), ] [ ( ) ] [ 2 ( ) ] 2 ξ(h) = E e(n) = E x(n) h(n) y(n). Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

6 Optimal Filtering Optimal filter design Optimal filter design Observation sequence Linear time invariant system Estimation Figure 1: Optimal filtering scenario. The MSE is a function of h(n), i.e., h = [, h( 2), h( 1), h(0), h(1), h(2), ] [ ( ) ] [ 2 ( ) ] 2 ξ(h) = E e(n) = E x(n) h(n) y(n). Thus, optimal filtering problem is h opt = arg min ξ(h) h Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

7 Optimal Filtering Application examples Application examples 1) Prediction, interpolation and smoothing of signals Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

8 Optimal Filtering Application examples Application examples 1) Prediction, interpolation and smoothing of signals d = 1 Prediction for anti-aircraft fire control. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

9 Optimal Filtering Application examples Application examples 1) Prediction, interpolation and smoothing of signals d = 1 (smoothing) d = 1/2 (interpolation) Signal denoising applications, estimation of missing data points. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

10 Optimal Filtering Application examples Application examples 2) System identification Figure 2: System identification using a training sequence t(n) from an ergodic and stationary ensemble. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

11 Optimal Filtering Application examples Application examples 2) System identification Figure 2: System identification using a training sequence t(n) from an ergodic and stationary ensemble. y(n) transmitter microphone Echo cancellation in full duplex data transmission. Modem filter synthesized echo x^ ( n) hybrid transformer two wire line earpiece receiver e (n) x (n) received signal + echo Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

12 Optimal Filtering Application examples Application examples 3) Inverse System identification Figure 3: Inverse system identification using x(n) as a training sequence. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

13 Optimal Filtering Application examples Application examples 3) Inverse System identification Figure 3: Inverse system identification using x(n) as a training sequence. Channel equalisation in digital communication systems. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

14 Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Normal equations Consider the MSE ξ(h) = E [(e(n)) 2] The optimal filter satisfies ξ(h) hopt = 0. Equivalently, for all j =..., 2, 1, 0, 1, 2,... [ ξ h(j) = E 2e(n) e(n) ] h(j) [ = E 2e(n) ( x(n) ] i= h(i)y(n i)) h(j) [ ] ( h(j)y(n j)) = E 2e(n) h(j) = 2E [e(n)y(n j)] Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

15 Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Normal equations Consider the MSE ξ(h) = E [(e(n)) 2] The optimal filter satisfies ξ(h) hopt = 0. Equivalently, for all j =..., 2, 1, 0, 1, 2,... [ ξ h(j) = E 2e(n) e(n) ] h(j) [ = E 2e(n) ( x(n) ] i= h(i)y(n i)) h(j) [ ] ( h(j)y(n j)) = E 2e(n) h(j) = 2E [e(n)y(n j)] Hence, the optimal filter solves the normal equations E [e(n)y(n j)] = 0, j =..., 2, 1, 0, 1, 2,... Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

16 Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Wiener-Hopf equations The error of h opt is orthogonal to its observations, i.e., for all j Z E [e opt (n)y(n j)] = 0 which is known as the principle of orthogonality. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

17 Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Wiener-Hopf equations The error of h opt is orthogonal to its observations, i.e., for all j Z E [e opt (n)y(n j)] = 0 which is known as the principle of orthogonality. Furthermore, [( ) ] E [e opt(n)y(n j)] = E x(n) h opt(i)y(n i) y(n j) i= = E [x(n)y(n j)] h opt(i)e [y(n i)y(n j)] = 0 i= Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

18 Optimal Filtering Optimal solution: Wiener-Hopf equations Optimal solution: Wiener-Hopf equations The error of h opt is orthogonal to its observations, i.e., for all j Z E [e opt (n)y(n j)] = 0 which is known as the principle of orthogonality. Furthermore, [( ) ] E [e opt(n)y(n j)] = E x(n) h opt(i)y(n i) y(n j) i= = E [x(n)y(n j)] h opt(i)e [y(n i)y(n j)] = 0 i= Result (Wiener-Hopf equations) h opt (i)r yy (i j) = r xy (j) i= Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

19 Optimal Filtering Optimal solution: Wiener-Hopf equations The Wiener filter Wiener-Hopf equations can be solved indirectly, in the complex spectral domain: h opt (n) r yy (n) = r xy (n) H opt (z)p yy (z) = P xy (z) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

20 Optimal Filtering Optimal solution: Wiener-Hopf equations The Wiener filter Wiener-Hopf equations can be solved indirectly, in the complex spectral domain: h opt (n) r yy (n) = r xy (n) H opt (z)p yy (z) = P xy (z) Result (The Wiener filter) H opt (z) = P xy(z) P yy (z) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

21 Optimal Filtering Optimal solution: Wiener-Hopf equations The Wiener filter Wiener-Hopf equations can be solved indirectly, in the complex spectral domain: h opt (n) r yy (n) = r xy (n) H opt (z)p yy (z) = P xy (z) Result (The Wiener filter) H opt (z) = P xy(z) P yy (z) The optimal filter has an infinite impulse response (IIR), and, is non-causal, in general. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

22 Optimal Filtering Optimal solution: Wiener-Hopf equations Causal Wiener filter We project the unconstrained solution H opt (z) onto the set of causal and stable IIR filters by a two step procedure: First, factorise P yy (z) into causal (right sided) Q yy (z), and anti-causal (left sided) parts Q yy(1/z ), i.e., P yy (z) = σ 2 yq yy (z)q yy(1/z ). Select the causal (right sided) part of P xy (z)/q yy(1/z ). Result (Causal Wiener filter) H + opt (z) = 1 σ 2 yq yy (z) [ Pxy (z) ] Qyy(1/z ) + Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

23 Optimal Filtering Optimal solution: Wiener-Hopf equations FIR Wiener-Hopf equations received sequence { y (n)} z -1 z -1 z -1 h 0 h h N output { x(n) } Figure 4: A finite impulse response (FIR) estimator. Wiener-Hopf equations for the FIR optimal filter of N taps: Result (FIR Wiener-Hopf equations) N 1 i=0 h opt(i)r yy (i j) = r xy (j), for j = 0, 1,..., N 1. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

24 Optimal Filtering Optimal solution: Wiener-Hopf equations FIR Wiener Filter FIR Wiener-Hopf equations in vector-matrix form. r yy (0) r yy (1)... r yy (N 1) h(0) r xy (0) r yy (1) r yy (0)... r yy (N 2) h(1) r xy (1).... =.. r yy (N 1) r yy (N 2)... r yy (0) h(n 1) r xy (N 1) }{{}}{{}}{{} R yy : Autocorrelation matrix of y(n) which is Toeplitz. h opt r xy Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

25 Optimal Filtering Optimal solution: Wiener-Hopf equations FIR Wiener Filter FIR Wiener-Hopf equations in vector-matrix form. r yy (0) r yy (1)... r yy (N 1) h(0) r xy (0) r yy (1) r yy (0)... r yy (N 2) h(1) r xy (1).... =.. r yy (N 1) r yy (N 2)... r yy (0) h(n 1) r xy (N 1) }{{}}{{}}{{} R yy : Autocorrelation matrix of y(n) which is Toeplitz. h opt r xy Result (FIR Wiener filter) h opt = R 1 yy r xy. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

26 Optimal Filtering Optimal solution: Wiener-Hopf equations MSE surface MSE is a quadratic function of h ξ(h) = h T R yy h 2h T r xy + E [(x(n)) 2] ξ(h) = 2R yy h 2r xy MSE surface 30 MSE contours with gradient vectors MSE (db) tap (a) 20 0 tap 0 20 tap tap 0 Figure 5: For a 2-tap Wiener filtering example: (a) the MSE surface, (b) gradient vectors. (b) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

27 Optimal Filtering Example: Wiener equaliser Example: Wiener equaliser noise η( n) (a) data {x( n)} channel { y(n) } equaliser C( z) H( z) ^ {x(n-d )} noise η(n) (b) data { x(n)} channel C( z) { y(n) } equaliser H(z) ^ { x(n-d )} { e( n-d )} z -d { x(n-d )} Figure 6: (a) The Wiener equaliser. (b) Alternative formulation. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

28 Optimal Filtering Example: Wiener equaliser Wiener equaliser η(n) data {x(n)} channel C ( z) y (n) { y(n) } equaliser H( z) ^ {x(n d)} { e( n d)} e (n) d z { x(n d)} x (n) Figure 7: Channel equalisation scenario. For notational convenience define: x (n) = x(n d) e (n) = x(n d) ˆx(n d) (1) Label the output of the channel filter as y (n) where y(n) = y (n) + η(n) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

29 Optimal Filtering Example: Wiener equaliser Wiener equaliser η(n) data {x(n)} channel C ( z) y (n) { y(n) } equaliser H( z) ^ {x(n d)} { e( n d)} e (n) d z { x(n d)} x (n) Figure 7: Channel equalisation scenario. Wiener filter h opt = R yy 1 r x y (2) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

30 Optimal Filtering Example: Wiener equaliser Wiener equaliser η(n) data {x(n)} channel C ( z) y (n) { y(n) } equaliser H( z) ^ {x(n d)} { e( n d)} e (n) d z { x(n d)} x (n) Figure 7: Channel equalisation scenario. Wiener filter h opt = R yy 1 r x y (2) The (i, j)th entry in R yy is r yy (j i) = E [y(j)y(i)] = E [ (y (j) + η(j))(y (i) + η(i)) ] = r y y (j i) + σ2 ηδ(j i) P yy (z) = P y y (z) + σ2 η Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

31 Optimal Filtering Example: Wiener equaliser Wiener equaliser η(n) data {x(n)} channel C ( z) y (n) { y(n) } equaliser H( z) ^ {x(n d)} { e( n d)} e (n) d z { x(n d)} x (n) Figure 7: Channel equalisation scenario. Remember y (n) = c(n) x(n) r y y = c(n) c( n) r xx(n) P y y (z) = C(z)C(z 1 )P xx (z) Consider a white data sequence x(n), i.e., r xx (n) = σ 2 xδ(n) P xx (z) = σ 2 x. Then, the complex spectra of the autocorrelation sequence of interest is P yy (z) = P y y (z) + σ2 x = C(z)C(z 1 )σ 2 x + σ 2 η Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

32 Optimal Filtering Example: Wiener equaliser Wiener equaliser η(n) data {x(n)} channel C ( z) y (n) { y(n) } equaliser H( z) ^ {x(n d)} { e( n d)} e (n) d z { x(n d)} x (n) Figure 7: Channel equalisation scenario. Wiener filter h opt = R yy 1 r x y (3) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

33 Optimal Filtering Example: Wiener equaliser Wiener equaliser η(n) data {x(n)} channel C ( z) y (n) { y(n) } equaliser H( z) ^ {x(n d)} { e( n d)} e (n) d z { x(n d)} x (n) Figure 7: Channel equalisation scenario. Wiener filter h opt = R yy 1 r x y (3) The (j)th entry in r x y is r x y(j) = E [ x (n)y(n j) ] = E [ x(n d)(y (n j) + η(n j)) ] = r xy (j d) P x y(z) = P xy (z)z d (4) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

34 Optimal Filtering Example: Wiener equaliser Wiener equaliser η(n) data {x(n)} channel C ( z) y (n) { y(n) } equaliser H( z) ^ {x(n d)} { e( n d)} e (n) d z { x(n d)} x (n) Figure 7: Channel equalisation scenario. Remember y (n) = c(n) x(n) r xy = c( n) r xx (n) P xy (z) = C(z 1 )P xx (z) Then, the complex spectra of the cross correlation sequence of interest is P x y(z) = P xy (z)z d = σ 2 xc(z 1 )z d Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

35 Optimal Filtering Example: Wiener equaliser Wiener equaliser Suppose that c = [c(0) = 0.5, c(1) = 1] T C(z) = (0.5 + z 1 ) Then, P yy (z) = C(z)C(z 1 )σ 2 x + σ 2 η = (0.5 + z 1 )(0.5 + z)σ 2 x + σ 2 η P x y(z) = σ 2 xc(z 1 )z d = (0.5z d + z d+1 )σ 2 x Suppose that d = 1, σ 2 x = 1, and, σ 2 η = 0.1 r yy (0) = 1.35, r yy (1) = 0.5, and r yy (2) = 0 r x y(0) = 1, r x y(1) = 0.5, and r x y(2) = 0 The Wiener filter is obtained as h opt = = The MSE is found as ξ(h opt ) = σ 2 x h T opt r x y = Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

36 Adaptive filtering Introduction Adaptive filtering - Introduction observed sequence z-1 z-1 z-1... estimated signal training sequence impulse response error signal adaptation algorithm Figure 8: FIR adaptive filtering configuration. For notational convenience, define y(n) [y(n), y(n 1),..., y(n N + 1)] T The output of the adaptive filter is Optimum solution ˆx(n) = h T (n)y(n) h opt = R 1 yy r xy, h(n) [h 0, h 1,..., h N 1 ] T Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

37 Adaptive filtering Recursive Least Squares Adaptation Recursive least squares Minimise cost function n ξ(n) = (x(k) ˆx(k)) 2 (5) k=0 Solution R yy (n)h(n) = r xy (n) LS autocorrelation matrix n R yy (n) = y(k)y T (k) k=0 LS cross-correlation vector n r xy (n) = y(k)x(k) k=0 Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

38 Adaptive filtering Recursive Least Squares Adaptation Recursive least squares Recursive relationships Substitute for r xy R yy (n) = R yy (n 1) + y(n)y T (n) r xy (n) = r xy (n 1) + y(n)x(n) R yy (n)h(n) = R yy (n 1)h(n 1) + y(n)x(n) Replace R yy (n 1) ( ) R yy (n)h(n) = R yy (n) y(n)y T (n) h(n 1) + y(n)x(n) Multiple both sides by R 1 yy (n) h(n) = h(n 1) + R 1 yy (n)y(n)e(n) e(n) = x(n) h T (n 1)y(n) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

39 Adaptive filtering Recursive Least Squares Adaptation Recursive least squares Recursive relationships Apply Sherman-Morrison identity R yy (n) = R yy (n 1) + y(n)y T (n) R 1 yy (n) = R 1 yy (n 1) R 1 yy (n 1)y(n)y T (n)r 1 yy (n 1) 1 + y T (n)r 1 yy (n 1)y(n) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

40 Adaptive filtering Recursive Least Squares Adaptation Summary Recursive least squares (RLS) algorithm: 1: R yy (0) = 1 δ I N with small positive δ Initialisation 1 2: h(0) = 0 Initialisation 2 3: for n = 1, 2, 3,... do Iterations 4: ˆx(n) = h T (n 1)y(n) Estimate x(n) 5: e(n) = x(n) ˆx(n) Find the error 6: R 1 1 yy (n) = α ( R 1 yy (n 1) R 1 yy (n 1)y(n)y T (n)r 1 yy (n 1) α+y T (n)r 1 yy (n 1)y(n) Update the inverse of the autocorrelation matrix 7: h(n) = h(n 1) + R 1 yy (n)y(n)e(n) Update the filter coefficients 8: end for ) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

41 Adaptive filtering Least Mean Square Algorithm Stochastic gradient algorithms MSE contour - 2-tap example: h wiener optimum (MMSE) contour of constant MSE Figure 9: Method of steepest descent. h 0 Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

42 Adaptive filtering Least Mean Square Algorithm Steepest descent MSE contour - 2-tap example: 10 h gradient vector initial guess h(n) h (n) h 0 The gradient vector [ ] ξ h (n) = h(0), ξ h(1),..., ξ T = 2R h(n 1) yy h(n) 2r xy h=h(n) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

43 Adaptive filtering Least Mean Square Algorithm Steepest descent MSE contour - 2-tap example: h µ (n) h h 0 Update initial guess in the direction of steepest descent: h(n + 1) = h(n) µ h (n) Step-size µ. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

44 Adaptive filtering Least Mean Square Algorithm Steepest descent MSE contour - 2-tap example: h h 0 Gradient at new guess. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

45 Adaptive filtering Least Mean Square Algorithm Convergence of steepest descent MSE contour - 2-tap example: 10 h Wiener optimum (MMSE) h(n + 1) = h(n) µ h (n) [ ξ h (n) = h(0), 0 < µ < 1 λ max ξ h(1),..., contour of constant MSE initial guess h(n) gradient vector (n) h h 0 ] ξ T = 2R yy h(n) 2r xy h(n 1) h=h(n) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69 (6)

46 Adaptive filtering Least Mean Square Algorithm Stochastic gradient algorithms A time recursion: The exact gradient: h (n) = 2E h(n + 1) = h(n) µ ˆ h (n) [ ] y(n)(x(n) y(n) T h(n)) = 2E [y(n)e(n)] A simple estimate of the gradient ˆ h (n) = 2y(n + 1)e(n + 1) The error e(n + 1) = x(n + 1) h(n) T y(n + 1) (7) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

47 Adaptive filtering Least Mean Square Algorithm The Least-mean-squares (LMS) algorithm: 1: h(0) = 0 Initialisation 2: for n = 1, 2, 3,... do Iterations 3: ˆx(n) = h T (n 1)y(n) Estimate x(n) 4: e(n) = x(n) ˆx(n) Find the error 5: h(n) = h(n 1) + 2µy(n)e(n) Update the filter coefficients 6: end for Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

48 Adaptive filtering Least Mean Square Algorithm LMS block diagram y ( n) z -1 z -1 z -1 z -1 h 0 h 1 h 2 h N-1 x^ (n) z -1 e ( n) x (n) 2µ scaled error Figure 10: Least mean-square adaptive filtering. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

49 Adaptive filtering Least Mean Square Algorithm Convergence of the LMS MSE contour - 2-tap example: h Wiener optimum (MMSE) contour of constant MSE initial guess h(n) gradient vector (n) h Eigenvalues of R yy (in this example, λ 0 and λ 1 ). The largest time constant τ max > λmax 2λ min Eigenvalue ratio (EVR) is λmax λ min Practical range for step-size 0 < µ < 1 3Nσ 2 y h 0 Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

50 Adaptive filtering Least Mean Square Algorithm Eigenvalue ratio (EVR) v max contour of constant MSE v min Wiener optimum 1/λmin (a) 1/λ max 6 6 tap tap (b) tap 0 (c) tap 0 Figure 11: Eigenvectors, eigenvalues and convergence: (a) the relationship between eigenvectors, eigenvalues and the contours of constant MSE; (b) steepest descent for EVR of 2; (c) EVR of 4. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

51 Adaptive filtering Least Mean Square Algorithm Comparison of RLS and LMS white noise white noise noise shaping filter y(n) unknown system h opt x(n) adaptive filter h(n) x(n) ^ Figure 12: Adaptive system identification configuration. Error vector norm ρ(n) = E [ ] (h(n) h opt ) T (h(n) h opt ) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

52 Adaptive filtering Least Mean Square Algorithm Comparison: Performance norm (db) RLS LMS iterations Figure 13: Covergence plots for N = 16 taps adaptive filtering in the system identification configuration: EVR = 1 (i.e., the impulse response of the noise shaping filter is δ(n)). Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

53 Adaptive filtering Least Mean Square Algorithm Comparison: Performance norm (db) RLS LMS iterations Figure 14: Covergence plots for N = 16 taps adaptive filtering in the system identification configuration: EVR = 11. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

54 Adaptive filtering Least Mean Square Algorithm Comparison: Performance LMS norm (db) RLS iterations Figure 15: Covergence plots for N = 16 taps adaptive filtering in the system identification configuration: EVR (and, correspondingly the spectral coloration of the input signal) progressively increases to 68. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

55 Adaptive filtering Least Mean Square Algorithm Comparison: Complexity Table: Complexity comparison of N-point FIR filter algorithms. Algorithm Implementation Computational load class multiplications adds/subtractions divisions RLS fast Kalman 10N+1 9N+1 2 SG LMS 2N 2N BLMS (via FFT) 10log(N)+8 15log(N)+30 Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

56 Adaptive filtering Applications Applications Adaptive filtering algorithms can be used in all application areas of optimal filtering. Some examples: Adaptive line enhancement Adaptive tone suppression Echo cancellation Channel equalisation Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

57 Adaptive filtering Applications y(n) unknown x(n) system adaptive lter x^ ( n) (a) e(n) x(n) unknown system y(n) adaptive lter x(n) ^ (b) e(n) x(n) delay y(n) adaptive lter ^(n) x (c) e(n) Figure 16: Adaptive filtering configurations: (a) direct system modelling; (b) inverse system modelling; (c) linear prediction. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

58 Figure 17: Adaptive line enhacement: (a) signal spectrum; (b) system configuration. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69 Adaptive filtering Applications Adaptive line enhancement spectrum narrow-band signal broad-band noise (a) 0 ω 0 frequency x ( n) z -1 z -1 z -1 a0 a 1 a2 prediction ^x (n) signal (b) prediction error e (n) noise

59 Adaptive filtering Applications Adaptive predictor x (n) x(n 1) x(n 2) z 1 z 1 z 1 x(n 3) a 0 a 1 a 2 prediction filter prediction signal ^ x(n) prediction error filter prediction error e (n) noise Prediction filter: a 0 + a 1 z 1 + a 2 z 2 Prediction error filter: 1 a 0 z 1 a 1 z z 2 a 2 z 3 Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

60 Figure 18: Adaptive tone suppression: (a) signal spectrum; (b) system configuration. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69 Adaptive filtering Applications Adaptive tone suppression spectrum interference signal spectrum (a) 0 ω 0 frequency x ( n) z -1 z -1 z -1 a0 a 1 a2 prediction ^x (n) interference (b) prediction error e (n) signal

61 Figure 19: Adaptive noise whitening: (a) input spectrum; (b) system configuration. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69 Adaptive filtering Applications Adaptive noise whitening spectrum coloured noise (a) 0 ω 0 frequency x ( n) z -1 z -1 z -1 a0 a 1 a2 prediction ^x (n) (b) prediction error e (n) whitened noise

62 Adaptive filtering Applications Echo cancellation A typical telephone connection transmitter transmitter microphone hybrid transformer two wire line hybrid transformer receiver receiver earpiece Telephone A Telephone B Hybrid transformers to route signal paths. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

63 Adaptive filtering Applications Echo cancellation (contd) Echo paths in a telephone system transmitter transmitter microphone hybrid transformer two wire line hybrid transformer near end echo receiver receiver earpiece Telephone A far end echo Telephone B Near and far echo paths. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

64 Adaptive filtering Applications Echo cancellation (contd) transmitter y(n) microphone filter synthesized echo x^ ( n) hybrid transformer two wire line earpiece receiver e (n) x (n) received signal + echo Fixed filter? Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

65 Adaptive filtering Applications Echo cancellation (contd) transmitter y(n) microphone e( n) adaptive filter ^x (n) hybrid transformer two-wire line receiver x (n) earpiece Figure 20: Application of adaptive echo cancellation in a telephone handset. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

66 Channel equalisation Adaptive filtering noise Applications x (n) channel filter y( n) adaptive filter delay x( n-d) error ^x(n-d) Figure 21: Adaptive equaliser system configuration. Simple channel y(n) = ±h 0 + noise Decision circuit if y(n) 0 then x(n) = +1 elsex(n) = 1 Channel with intersymbol interference (ISI) 2 y(n) = h i x(n i) + noise i=0 Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

67 Adaptive filtering Applications Channel equalisation y(n ) transversal ^x (n) filter decision circuit e(n) x( n) delay Figure 22: Decision directed equaliser. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

68 Optimal signal detection Application examples and optimal hypothesis testing Optimal signal detection - Introduction Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

69 Optimal signal detection Application examples and optimal hypothesis testing Optimal signal detection - Introduction Example: Detection of gravitational waves Figure 23: Gravitational waves from a binary black hole merger (left, Uni. of Birmingham, Gravitational Wave Group), LIGO block diagram (middle), expected signal (right) Abbot et. al., Observation of gravitational waves from a binary black hole merger, Phys. Rev. Let., Feb Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

70 Optimal signal detection Application examples and optimal hypothesis testing Optimal signal detection Signal detection as 2-ary (binary) hypothesis testing: H 0 : y(n) = η(n) H 1 : y(n) = x(n) + η(n) (8) In a sense, decide which of the two possible ensembles y(n) is generated from. Finite length signals, i.e., Vector notation n = 0, 1, 2,..., N 1 H 0 : y = η H 1 : y = x + η Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

71 Optimal signal detection Application examples and optimal hypothesis testing Optimal signal detection Example (radar): In active sensing, x(t) is the probing waveform subject to design. y(n) = a 0 x(n n 0 ) + a 1 x(n n 1 ) + + η(n) (9) Figure 24: Probing waveform x(n) and returns from the surveillance region constituting y(n). Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

72 Optimal signal detection Application examples and optimal hypothesis testing Optimal signal detection Example (radar): In active sensing, x(t) is the probing waveform subject to design. y(n) = a 0 x(n n 0 ) + a 1 x(n n 1 ) + + η(n) (9) Figure 24: Probing waveform x(n) and returns from the surveillance region constituting y(n). A similar problem also arises in digital communications. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

73 Optimal signal detection Application examples and optimal hypothesis testing Bayesian hypothesis testing Consider a random variable H with H = H(ζ) and H {H 0, H 1 }. The measurement ȳ = y(ζ) is a realisation of y. The measurement model specifies the likelihood p Y H (ȳ H) Find the probabilities of H = H 1 and H = H 0 based on the measurement vector ȳ. Decide on the hypothesis with the maximum a posteriori probability: Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

74 Optimal signal detection Application examples and optimal hypothesis testing Bayesian hypothesis testing Consider a random variable H with H = H(ζ) and H {H 0, H 1 }. The measurement ȳ = y(ζ) is a realisation of y. The measurement model specifies the likelihood p Y H (ȳ H) Find the probabilities of H = H 1 and H = H 0 based on the measurement vector ȳ. Decide on the hypothesis with the maximum a posteriori probability: Find the maximum a-posteriori (MAP) estimate of H. Ĥ = arg max P H y(h ȳ) (10) H {H 0,H 1 } where the a posterior probability of H is given by for i = 0, 1. P H y (H i ȳ) = p Y H (ȳ H i )P H (H i ) p Y H (ȳ H 0 )P H (H 0 ) + p Y H (ȳ H 1 )P H (H 1 ) (11) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

75 Optimal signal detection Application examples and optimal hypothesis testing Bayesian hypothesis testing: The likelihood ratio test MAP decision rule: Ĥ = arg max P(H ȳ) H {H 0,H 1 } MAP decision as a likelihood ratio test: p(h 1 ȳ) Ĥ=H 1 > < Ĥ=H 0 p(h 0 ȳ) p(ȳ H 1 )P(H 1 ) p(ȳ H 1 ) p(ȳ H 0 ) H 1 > < H 0 p(ȳ H 0 )P(H 0 ) H 1 > < H 0 P(H 0 ) P(H 1 ) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

76 Optimal signal detection Additive white and coloured noise Bayesian hypothesis testing - AWGN Example Example: Detection of deterministic signals in additive white Gaussian noise: H 0 : y = η H 1 : y = x + η where x is a known vector, η N (.; 0, σ 2 I). The likelihoods are specified by the noise distribution: p(ȳ H 1 ) p(ȳ H 0 ) N (ȳ; x, σ 2 I) N (ȳ; 0, σ 2 I) H 1 > P(H 0 ) < H 0 P(H 1 ) H 1 > P(H 0 ) < H 0 P(H 1 ) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

77 Optimal signal detection Additive white and coloured noise Detection of deterministic signals - AWGN (contd) The numerator and denominator of the likelihood ratio are Similarly Therefore p(ȳ H 1 ) = N (ȳ x; 0, σ 2 I) N 1 1 } (ȳ(n) x(n))2 = exp { (2πσ 2 ) N/2 2σ 2 n=0 { ( 1 = exp 1 N 1 )} (ȳ(n) x(n)) 2 (2πσ 2 ) N/2 2σ 2 p(ȳ H 0 ) = N (ȳ; 0, σ 2 I) = p(ȳ H 1 ) p(ȳ H 0 ) = exp 1 exp (2πσ 2 ) N/2 { 1 σ 2 { ( N 1 1 2σ 2 n=0 ( N 1 )} (ȳ(n)) 2 n=0 )} (ȳ(n)x(n) 1 2 x(n)2 ) n=0 (12) (13) (14) Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

78 Optimal signal detection Additive white and coloured noise Detection of deterministic signals - AWGN (contd) Take the logarithm of both sides of the likelihood ratio test { ( N 1 )} 1 log exp (ȳ(n)x(n) 1 H1 σ 2 2 x(n)2 > ) < log P(H 0) H 0 P(H 1 ) n=0 Now, we have a linear statistical test N 1 n=0 H 1 > < ȳ(n)x(n) σ 2 log P(H 0) H 0 P(H 1 ) N 1 n=0 x(n) 2 } {{ } τ:decision threshold Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

79 Optimal signal detection Additive white and coloured noise Detection of deterministic signals - AWGN (contd) Matched filtering for optimal detection: Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

80 Optimal signal detection Additive white and coloured noise Detection of deterministic signals under coloured noise For the case, the noise sequence ρ(n) has an auto-correlation function r ν [l] different than r ν [l] = ση 2 δ[l], and, H 0 : y = ν H 1 : y = x + ν ν N.; 0, C ν = r ν (0) r ν ( 1)... r ν ( N + 1) r ν (1) r ν (0)... r ν ( N + 2).... r ν (N 1) r ν (N 2)... r ν (0) Whitening lter Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

81 Optimal signal detection Additive white and coloured noise Detection of deterministic signals - coloured noise ex (t) < (t) > H GPS time relative to (s) (upper left) Noise (amplitude) spectral density. (upper right) Abstract, Abbot et. al., Phys. Rev. Let., Feb (lower left) Matched filter outputs: Best MF (blue) and the expected MF (purple). (lower right) Measurement, reconstructed and noise signals around the detection. Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

82 Summary Summary Optimal filtering: Problem statement General solution via Wiener-Hopf equations FIR Wiener filter Adaptive filtering as an online optimal filtering strategy Recursive least-squares (RLS) algorithm Least mean-square (LMS) algorithm Application examples Optimal signal detection via matched filtering Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69

83 Summary Further reading C. Therrien, Discrete Random Signals and Statistical Signal Processing, Prentice-Hall, S. Haykin, Adaptive Filter Theory, 5th ed., Prentice-Hall, B. Mulgrew, P. Grant, J. Thompson, Digital Signal Processing: Concepts and Applications, 2nd ed., Palgrave Macmillan, D. Manolakis, V. Ingle, S. Kogon, Statistical and Adaptive Signal Processing, McGraw Hill, Murat Üney (IDCOM) Optimal and Adaptive Filtering 26/06/ / 69