Lecture 7: Linear Prediction

Transcription

1 1 Lecture 7: Linear Prediction Overview Dealing with three notions: PREDICTION, PREDICTOR, PREDICTION ERROR; FORWARD versus BACKWARD: Predicting the future versus (improper terminology) predicting the past; Fast computation of AR parameters: Levinson Durbin algorithm; New AR parametrization: Reflection coefficients; Lattice filters References : Chapter 3 from S Haykin- Adaptive Filtering Theory - Prentice Hall, 22

2 Lecture 7 2 Notations, Definitions and Terminology Time series: u(1), u(2), u(3),, u(n 1), u(n), u(n + 1), Linear prediction of order M FORWARD PREDICTION Regressor vector û(n) = w 1 u(n 1) + w 2 u(n 2) + + w M u(n M) = M k=1 w k u(n k) = w T u(n 1) u(n 1) = [ u(n 1) u(n 2) u(n M) ] T Predictor vector of order M FORWARD PREDICTOR w = [ w 1 w 2 w M ] T a M = [ 1 w 1 w 2 w M ] T = [ a M, a M,1 a M,2 a M,M ] T and thus a M, = 1, a M,1 = w 1, a M,2 = w 2,, a M,M = w M, Prediction error of order M FORWARD PREDICTION ERROR f M (n) = u(n) û(n) = u(n) w T u(n 1) = a T M u(n) u(n 1) = a T Mu(n)

3 Lecture 7 3 Notations: r(k) = E[u(n)u(n + k)] R = E[u(n 1)u T (n 1)] = E = R = r B u(n 1) u(n 2) u(n 3) u(n M) autocorrelation function [ u(n 1) u(n 2) u(n 3) u(n M) ] Eu(n 1)u(n 1) Eu(n 1)u(n 2) Eu(n 1)u(n M) Eu(n 2)u(n 1) Eu(n 2)u(n 2) Eu(n 2)u(n M) Eu(n M)u(n 1) Eu(n M)u(n 2) Eu(n M)u(n M) r() r(1) r(m 1) r(1) r() r(m 2) r(m 1) r(m 2) r() r = E[u(n 1)u(n)] = [ r(1) r(2) r(3) r(m) ] T = E[u(n 1)u(n M 1)] = [ r(m) r(m 1) r(m 2) r(1) ] T = autocorrelation matrix autocorrelation vector Superscript B is the vector reversing (Backward) operator ie for any vector x, we have x B = [ x(1) x(2) x(3) x(m) ] B = [ x(m) x(m 1) x(m 2) x(1) ]

4 Lecture 7 4 Optimal forward linear prediction Optimality criterion Optimal solution: J(w) = E[f M (n)] 2 = E[u(n) w T u(n 1)] 2 J(a M ) = E[f M (n)] 2 = E[a T M u(n) u(n 1) ] 2 Optimal Forward Predictor w o = R 1 r Forward Prediction Error Power P M = r() r T w o Two derivations of optimal solution 1 Transforming the criterion into a quadratic form J(w) = E[u(n) w T u(n 1)] 2 = E[u(n) w T u(n 1)][u(n) u(n 1) T w] = E[u(n)] 2 2E[u(n)u(n 1) T ]w + w T E[u(n 1)u(n 1) T ]w = r() 2r T w + w T Rw = r() r T R 1 r + (w R 1 r) T R(w R 1 r) (1)

5 Lecture 7 5 The matrix R is positive semi-definite because x T Rx = x T E[u(n)u(n)] T x = E[u(n) T x] 2 x and therefore the quadratic form in the right hand side of (1) (w R 1 r) T R(w R 1 r) attains its minimum when (w o R 1 r) =, ie w o = R 1 r For the predictor w o, the optimal criterion in (1) equals P M = r() r T R 1 r = r() r T w o 2 Derivation based on optimal Wiener filter design The optimal predictor evaluation can be rephrased as the following Wiener filter design problem: find the FIR filtering process y(n) = w T u(n) as close as possible to desired signal d(n) = u(n + 1), ie minimizing the criterion E[d(n) y(n)] 2 = E[u(n + 1) w T u(n)] 2 Then the optimal solution is given by w o = R 1 p where R = E[u(n)u(n) T ] and p = E[d(n)u(n)] = E[u(n + 1)u(n)] = E[u(n)u(n 1)] = r, ie w o = R 1 r

6 Lecture 7 6 Augmented Wiener Hopf equations The optimal predictor filter solution w o and the optimal prediction error power satisfy r() r T w o = P M Rw o r = which can be written in a block matrix equation form r() rt r R 1 w o = P M But 1 = a w M o r() rt r R = R M+1 Autocorrelation matrix of dimensions (M + 1) (M + 1) Finally, the augmented Wiener Hopf equations for optimal forward prediction error filter are R M+1 a M = P M or r() r(1) r(m) r(1) r() r(m 1) r(m) r(m 1) r() a M, a M,1 a M,2 a M,M Whenever R M is nonsingular, and a M, is set to 1, there are unique solutions a M and P M = P M

7 Lecture 7 7 Optimal backward linear prediction Linear backward prediction of order M BACKWARD PREDICTION û b (n M) = g 1 u(n) + g 2 u(n 1) + + g M u(n M + 1) = M k=1 g k u(n k + 1) = g T u(n) where the BACKWARD PREDICTOR is g = [ g 1 g 2 ] T g M Backward prediction error of order M BACKWARD PREDICTION ERROR b M (n) = u(n M) û b (n M) = u(n M) g T u(n) Optimality criterion J b (g) = E[b M (n)] 2 = E[u(n M) g T u(n)] 2

8 Lecture 7 8 Optimal solution: Optimal Backward Predictor g o = R 1 r B = w B o Forward Prediction Error Power P M = r() (r B ) T g o = r() r T w o Derivation based on optimal Wiener filter design The optimal backward predictor evaluation can be rephrazed as the following Wiener filter design problem: find the FIR filtering process y(n) = g T u(n) as close as possible to desired signal d(n) = u(n M), ie minimizing the criterion E[d(n) y(n)] 2 = E[u(n M) g T u(n)] 2 Then the optimal solution is given by g o = R 1 p where R = E[u(n)u(n) T ] and p = E[d(n)u(n)] = E[u(n M)u(n)] = E[u(n M 1)u(n 1)] = r B, ie and the optimal criterion value is g o = R 1 r B J b (g o ) = E[b M (n)] 2 = E[d(n)] 2 g T o Rg o = E[d(n)]2 g T o rb = r() g T o rb

9 Lecture 7 9 Relations between Backward and Forward predictors g o = w B o Useful mathematical result: If the matrix R is Toeplitz, then for all vectors x (Rx) B = Rx B (Rx) B i = (Rx B ) i (Rx) M i+1 = (Rx B ) i Proof: (Rx B ) i = M R i,j x M j+1 = M j=1 = M k=1 j=1 r(i j)x M j+1 j=m k+1 = R M i+1,k x k = (Rx) M i+1 = (Rx) B i The Forward and Backward optimal predictors are solutions of the systems Rw o = r Rg o = r B M k=1 r(i M + k 1)x k Rg o = r B = (Rw o ) B = Rw B o

10 Lecture 7 1 and since R is supposed nonsingular, we have g o = w B o Augmented Wiener-Hopf equations for Backward prediction error filter The optimal Backward predictor filter solution g o and the optimal Backward prediction error power satisfy Rg o r B = r() (r B ) T g o = P M which can be written in a block matrix equation form R r B (r B ) T r() g o 1 = P M But g o 1 = c M R r B (r B ) T = R r() M+1 Autocorrelation matrix of dimensions (M + 1) (M + 1) Finally, the augmented Wiener Hopf equations for optimal backward predictioon error filter are R M+1 c M = P M

11 Lecture 7 11 or r() r(1) r(m) r(1) r() r(m 1) r(m) r(m 1) r() c M, c M,1 c M,2 c M,M = r() r(1) r(m) r(1) r() r(m 1) r(m) r(m 1) r() a M,M a M,M 1 a M,M 2 a M, = P M Levinson Durbin algorithm Stressing the order of the filter All variables will receive a subscript expressing the order of the predictor: R m, r m, a m, w o m Some order recursive equations can be written: r m+1 = [ r(1) r(2) r(m) r(m + 1) ] T = R m+1 = R m r B m (r B m) T r() = r() r m rt m R m r m r(m + 1)

12 Lecture 7 12 Main recursions where ψ = a m 1 m 1 a B m 1 m 1 = r T ma B m 1 = a T m 1r B m = r(m) + m 1 Multipling the right hand side of Equation (2) by R m+1 = R m r B m (r B m) T r() R m+1 ψ = R m+1 = = R ma m 1 (r B m) T a m 1 R ma m 1 m 1 a m 1 m 1 m 1 m rt ma B m 1 R m a B m 1 m 1 R m a B m 1 a B m 1 = = m 1 m 1 R m r B m (r B m) T r() m 1 k=1 m 1 m 1 (2) a m 1,k r(m k) a m 1 = = r() r m m 1 rt m R m r() r m 2 m 1 m 1 m 1 m 1 But since ψ(1) = a m 1, = 1, and we suppose R m nonsingular, the unique solution of R m+1 ψ = 2 m 1 we obtain rt m R m a B m 1 = 2 m 1 provides the optimal predictor a m = ψ with the recursion (2) and the optimal prediction error power m m P m = 2 m 1 = (1 2 m 1 ) = (1 Γ 2 P m) m 1 P 2 m 1

13 Lecture 7 13 with the notation Γ m = m 1 Levinson Durbin recursions a m = m 1 + Γ m a B m 1 Vector form of L D recursions a m,k = a m 1,k + Γ m a m 1,m k, k =, 1,, m Scalar form of L D recursions m 1 = a T m 1r B m = r(m) + m 1 Γ m = m 1 P m = (1 Γ 2 m) k=1 a m 1,k r(m k)

14 Lecture 7 14 Interpretation of m and Γ m 1 m 1 = E[f m 1 (n)b m 1 (n 1)] Proof (Solution of Problem 9 page 238 in [Haykin91]) E[f m 1 (n)b m 1 (n 1)] = E[a T m 1u(n)][u(n 1) T a B m 1] = a T m 1 = [ 1 w T m 1 2 = E[f (n)b (n 1)] = E[u(n)u(n 1)] = r(1) 3 Iterating P m = (1 Γ 2 m) we obtain P m = P ] rt ma B m 1 m 1 m k=1 = r T ma B m 1 = m 1 (1 Γ 2 k) r T m R m 1 r B m 1 wb m Since the power of prediction error must be positive for all orders, the reflection coefficients are less than unit in absolute value: Γ m 1 m =,, M 5 Reflection coefficients equal last autoregressive coefficient, for each order m: Γ m = a m,m, m = M, M 1,, 1

15 Lecture 7 15 Algorithm (L D) Given r(), r(1), r(2),, r(m) for example, estimated from data u(1), u(2), u(3),, u(t ) using r(k) = 1 T T n=k+1 u(n)u(n k) 1 Initialize = r(1), P = r() 2 For m = 1,, M 21 Γ m = m 1 22 a m, = 1 23 a m,k = a m 1,k + Γ m a m 1,m k, k = 1,, m 24 m = r(m + 1) + m a m,k r(m + 1 k) k=1 25 P m = (1 Γ 2 m) Computational complexity: For the m th iteration of Step 2: 2m + 2 multiplications, 2m + 2 additions, 1 division The overall computational complexity: O(M 2 ) operations

16 Lecture 7 16 Algorithm (L D) Second form Given r() and Γ 1, Γ 2,, Γ M 1 Initialize P = r() 2 For m = 1,, M 21 a m, = 1 22 a m,k = a m 1,k + Γ m a m 1,m k, k = 1,, m 23 P m = (1 Γ 2 m) Inverse Levinson Durbin algorithm a m,k = a m 1,k + Γ m a m 1,m k a m,m k = a m 1,m k + Γ m a m 1,k and using the identity Γ m = a m,m a m,k a m,m k = 1 Γ m Γ m 1 a m 1,m k a m 1,k a m 1,k = a m,k a m,m a m,m k 1 (a m,m ) 2 k = 1,, m

17 Lecture 7 17 The second order properties of the AR process are perfectly described by the set of reflection coefficients This immediately follows from the following property: The sets {P, Γ 1, Γ 2,, Γ M } and {r(), r(1),, r(m)} are in one-to-one correspondence Proof (a) {r(), r(1),, r(m)} (b) From we can obtain immediately (Algorithm L D) = {P, Γ 1, Γ 2,, Γ M, } Γ m+1 = m P m = r(m + 1) + m k=1 a m,k r(m + 1 k) P m r m+1 = Γ m+1 P m m a m,k r(m + 1 k) which can be iterated together with Algorithm L D form 2, to obtain all r(1),, r(m) Whitening property of prediction error filters k=1 In theory, a prediction error filter is capable of whitening a stationary discrete-time stochastic process applied to its input, if the order of the filter is high enough Then all information in the original stochastic process u(n) is represented by the parameters {P M, a M,1, a M,2,, a M,M } (or, equivalently, by {P, Γ 1, Γ 2,, Γ M }) A signal equivalent (as second order properties) can be generated starting from {P M, a M,1, a M,2,, a M,M } using the autoregressive difference equation model

18 Lecture 7 18 These analyze and generate paradigms combine to provide the basic principle of vocoders Gram-Schmidt orthogonalization algorithm Using the notations b (n) = u(n) b 1 (n) = a 1,1 u(n) + a 1, u(n 1) b 2 (n) = a 2,2 u(n) + a 2,1 u(n 1) + a 2, u(n 2) b M (n) = a M,M u(n) + a M,M 1 u(n 1) + + a M, u(n M) u(n) = [u(n), u(n 1),, u(n M)] T b(n) = [b (n), b 1 (n),, b M (n)] T L = we may write G S orthogonalization procedure as 1 a 1,1 1 a M,M a M,M 1 1 b(n) = Lu(n)

19 Lecture 7 19 Orthogonality of Backward prediction errors The following orthogonality property holds: Proof Suppose j i E[b i (n)b j (n)] = E[b j (n) ] b i (n) = i a i,i k u(n k) k= P m, i = j, i j Eb j (n)b i (n) = Eb j (n) i a i,i k u(n k) = = i k= k= a i,i k Eb j (n)u(n k) = Eb 2 i (n), i = j, i j = P m, i = j, i j due to orthogonality of optimal Wiener filter error to the inputs involved in the computation of that error: Eb j (n)u(n k) = for k j In matrix form Substituting b(n) = Lu(n) E[b(n)b(n) T ] = P P 1 P M = D E[b(n)b(n) T ] = LE[u(n)u(n) T ]L T = LRL T = D

20 Lecture 7 2 which can be use to factorize the matrices R and R 1 as R = L 1 DL T R 1 = (L 1 DL T ) 1 = L T D 1 L = L T D 1/2 D 1/2 L = (D 1/2 L) T D 1/2 L (3) Equation (3) provides the Cholesky factorization of R 1