Parametric Signal Modeling and Linear Prediction Theory 4. The Levinson-Durbin Recursion

Transcription

1 Parametric Signal Modeling and Linear Prediction Theory 4. The Levinson-Durbin Recursion Electrical & Computer Engineering North Carolina State University Acknowledgment: ECE slides were adapted from ENEE630 slides developed by Profs. K.J. Ray Liu and Min Wu at the University of Maryland, College Park. Contact: Updated: February 14, ECE Lecture Part I 1 / 20

2 Complexity in Solving Linear Prediction (Refs: Hayes 5.2; Haykin 4th Ed. 3.3) Recall Augmented Normal Equation for linear prediction: [ ] [ PM 0 FLP R M+1 a M = BLP R 0 M+1 a B M = P M ] As R M+1 is usually non-singular, a M may be obtained by inverting R M+1, or Gaussian elimination for solving equation array: Computational complexity O(M 3 ). ECE Lecture Part I 2 / 20

3 Exploiting Structures in Matrix and LP Problem Complexity in solving a general linear equation array: Method-1: invert the matrix, e.g. compute determinant of R M+1 matrix and the adjacency matrices matrix inversion has O(M 3 ) complexity Method-2: use Gaussian elimination approximately M 3 /3 multiplication and division By exploring the Toeplitz structure of the matrix, Levinson-Durbin recursion can reduce complexity to O(M 2 ) M steps of order recursion, each step has a linear complexity w.r.t. intermediate order Memory use: Gaussian elimination O(M 2 ) for the matrix, vs. Levinson-Durbin O(M) for the autocorrelation vector and model parameter vector. ECE Lecture Part I 3 / 20

4 Levinson-Durbin Recursion The Levinson-Durbin recursion is an order-recursion to efficiently solve linear systems with Toeplitz matrices, e.g., Augmented N.E. M steps of order recursion, each step has a linear complexity w.r.t. intermediate order The recursion can be stated in two ways for the linear prediction problem: 1 Forward prediction point of view 2 Backward prediction point of view ECE Lecture Part I 4 / 20

5 Two Points of View of LD Recursion Denote a m C (m+1) 1 as the tap weight vector of a forward-prediction-error filter of order m = 0,..., M. a m 1,0 = 1, a m 1,m 0, a m,m = Γ m (a constant reflection coefficient ) Forward prediction point of view a m,k = a m 1,k + Γ m am 1,m k, k = 0, 1,..., m [ ] [ ] am 1 0 In vector form: a m = + Γ 0 m a B m 1 ( ) Backward prediction point of view am,m k = a m 1,m k + Γ m a m 1,k, k = 0, 1,..., m [ ] [ ] 0 In vector form: a B m = + Γ a am 1 B m (can be obtained by m 1 0 reordering and conjugating ( )) ECE Lecture Part I 5 / 20

6 Recall: Forward and Backward Prediction Errors f m [n] = u[n] û[n] = a H m u[n] }{{} (m+1) 1 b m [n] = u[n m] û[n m] = a B,T m u[n] ECE Lecture Part I 6 / 20

7 (3) Verify the Update Equations of the LD Recursion Left multiply both sides of ( ) by R m+1 : [ ] Pm LHS: R m+1 a m = (by augmented N.E.) 0 m [ ] [ ] [ ] am 1 Rm r RHS (1): R m+1 = B m am 1 0 r BT m r(0) 0 [ ] P Rm a m 1 = m 1 r BT m a = 0 m 1 where m 1 r BT m a m 1 m 1 RHS (2): R m+1 [ = 0 a B m 1 [ r H a B m 1 R m a B m 1 m 1 ] [ ] [ r(0) r H = r R m ] m 1 = 0 m 1 P m 1 0 a B m 1 ] ECE Lecture Part I 7 / 20

8 Computing Γ m Put together LHS and RHS: for the order update recursion ( ) to hold, we should have [ ] P m 1 Pm m 1 = 0 0 m 1 + Γ m 0 m 1 m m 1 P m 1 { P m = P m 1 + Γ m m 1 0 = m 1 + Γ m P m 1 a m,m = Γ m = m 1 P m 1 P m = P m 1 ( 1 Γm 2) Caution: Do not confuse the power term P m and the ratio term Γ m. ECE Lecture Part I 8 / 20

9 (4) Reflection Coefficients Γ m To ensure the prediction MSE P m 0 and P m non-increasing as we increase the order of the predictor (i.e., 0 P m P m 1 ), we require Γ m 2 1, m > 0. Let P 0 = r(0) as the initial estimation error has power equal to the signal power (i.e., no regression is applied), we have P M = P 0 M m=1 (1 Γ m 2 ) Question: Under what situation Γ m = 0? i.e., increasing order won t reduce error. Consider a process with Markovian-like property in 2nd order statistic sense (e.g. AR process) s.t. info of further past is contained in k recent samples ECE Lecture Part I 9 / 20

10 (5) About m One can show that the cross-correlation of BLP error and FLP error E [ b m 1 [n 1]f m 1 [n]] is equal to m 1. (Derive from the definition m 1 r BT m a m 1, and use definitions of b m 1[n 1], fm 1[n] and orthogonality principle.) Thus the reflection coefficient can be written as Γ m = m 1 = E [ bm 1 [n 1]fm 1 [n]] P m 1 E [ f m 1 [n] 2 ] which is also the negative partial correlation coefficient. Note: for the 0th order predictor, use the mean value, i.e., zero, as the estimate, s.t. f 0 [n] = u[n] = b 0 [n], 0 = E [b 0 [n 1]f 0 [n]] = E [u[n 1]u [n]] = r( 1) = r (1) ECE Lecture Part I 10 / 20

11 Preview: Relations of w.s.s and LP Parameters For any w.s.s. process {u[n]}: ECE Lecture Part I 11 / 20

12 (6) Computing a M and P M by Forward Recursion Case-1 : If we know the autocorrelation function r( ): # of iterations = M m=1 m = M(M+1) 2, comp. complexity is O(M 2 ) r(k) may be estimated from time average of one realization of {u[n]}: ˆr(k) = 1 N k N n=k+1 u[n]u [n k], k = 0, 1,..., M (recall correlation ergodicity) ECE Lecture Part I 12 / 20

13 (6) Computing a M and P M by Forward Recursion Case-2 : If we know Γ 1, Γ 2,..., Γ M and P 0 = r(0), we can carry out the recursion for m = 1, 2,..., M: { a m,k = a m 1,k + Γ m a m 1,m k, k = 1,..., m P m = P m 1 ( 1 Γm 2) Note: a m,m = a m 1,m + Γ m a m 1,0 = 0 + Γ m 1 = Γ m ECE Lecture Part I 13 / 20

14 (7) Inverse Form of Levinson-Durbin Recursion Given the tap-weights a M, find the reflection coefficients Γ 1, Γ 2,..., Γ M : { (FP) a m,k = a m 1,k + Γ m am 1,m k Recall:, k = 0,..., m (BP) am,m k = a m 1,m k + Γ m a m 1,k, a m,m = Γ m Multiply (BP) by Γ m and subtract from (FP): a m 1,k = a m,k Γ ma m,m k 1 Γ m 2 Γ m = a m,m, Γ m 1 = a m 1,m 1,..., iterate with m = M 1, M 2,... = a m,k a m,ma m,m k 1 a m,m 2, k = 0,..., m 1 i.e., From a M a m Γ m to lower order see 5 Lattice structure: ECE Lecture Part I 14 / 20

15 (8) Autocorrelation Function & Reflection Coefficients Recall: The 2nd-order statistics of a stationary time series can be represented in terms of autocorrelation function r(k), or equivalently the power spectral density by taking DTFT. Another way is to use {r(0), Γ 1, Γ 2,..., Γ M }. To find the relation between them, recall: m 1 r BT m a m 1 = M 1 k=0 a m 1,kr( m + k) and Γ m = m 1 P m 1 Γ m P m 1 = m 1 k=0 a m 1,kr(k m), where a m 1,0 = 1. ECE Lecture Part I 15 / 20

16 (8) Autocorrelation Function & Reflection Coefficients 1 r(m) = r ( m) = Γ mp m 1 m 1 k=1 a m 1,kr(m k) Given r(0), Γ 1, Γ 2,..., Γ M, can get a m using Levinson-Durbin recursion s.t. r(1),..., r(m) can be generated recursively. 2 Recall if r(0),..., r(m) are given, we can get a m. So Γ 1,..., Γ M can be obtained recursively: Γ m = a m,m 3 These facts imply that the reflection coefficients {Γ k } can uniquely represent the 2nd-order statistics of a w.s.s. process. ECE Lecture Part I 16 / 20

17 Summary Statistical representation of w.s.s. process ECE Lecture Part I 17 / 20

18 Detailed Derivations/Examples ECE Lecture Part I 18 / 20

19 Example of Forward Recursion Case-2 ECE Lecture Part I 19 / 20

20 Proof for m 1 Property ECE Lecture Part I 20 / 20