Adap>ve Filters Part 2 (LMS variants and analysis) ECE 5/639 Sta>s>cal Signal Processing II: Linear Es>ma>on

Transcription

1 Adap>ve Filters Part 2 (LMS variants and analysis) Sta>s>cal Signal Processing II: Linear Es>ma>on Eric Wan, Ph.D. Fall

2 LMS Variants and Analysis LMS variants Normalized LMS Leaky LMS Filtered-X LMS and Adjoint LMS Transform and Frequency domain LMS Block LMS LMS Analysis Misadjustment Tracking Performance 2

3 Normalized LMS c(n +1) = c +!µ x 2 e* x 0 <!µ < 2 Time varying step size Can be derived as a constrained op>miza>on problem that minimizes the change in the coefficients from >me step to >me step Helps mi>gate effects of gradient noise when the number of coefficients is large or when the error or input values are large. Analysis of rate of convergence and misadjustment is much more difficult than LMS Rate of convergence is poten>ally significantly faster than standard LMS Only slight increase in computa>onal complexity Minor change to avoid issues when the input goes to zero!µ!µ x 2 x 2 +δ ( ) 3

4 Leaky LMS c(n +1) = (1 µα)c + 2µe * x Adds a constraint on the coefficients P = e 2 +α c 2 lim E[c] = R +αi n 1 d Like adding a small amount of white noise to the input Keeps the coefficients small, which can be useful if the input is near zero or poorly condi>oned. May improve eigenvalue spread. Adds a bias to the solu>on 4

5 Filtered-X and Adjoint LMS Consider the following adap>ve filter setup y x c ŷ P(z) ŷ P - e 5

6 Filtered-X and Adjoint LMS Consider the following adap>ve filter setup H(z) = z D y x c ŷ P(z) ŷ P - e open-loop Inverse Control Pre-equaliza>on 6

7 Filtered-X and Adjoint LMS Consider the following adap>ve filter setup M (z) y x c ŷ P(z) ŷ P - e Controller Model Reference Adap>ve Control 7

8 Filtered-X and Adjoint LMS Concert hall room impulse response y stereo x c ŷ P(z) ŷ P - e Acous>c room equaliza>on Your living room impulse response 8

9 Filtered-X and Adjoint LMS y x c ŷ P(z) ŷ P - e Ac>ve noise (or vibra>on) cancella>on 9

10 Filtered-X LMS How to train? Re-derive stochas>c gradient descent to account for Block diagram manipula>on P(z) y x c ŷ P(z) ŷ P - e y x P(z) x P c ŷ P - e When are these equal? LMS 10

11 Filtered-X LMS x P = ˆp x = ˆp T x ŷ = c H x P e = y ŷ P c(n +1) = c + 2µex P y x c ŷ P(z) ŷ P - e ˆP(z) x P LMS ŷ P 11

12 Filtered-X LMS Mul>ple Error X-LMS Generaliza>on to MIMO systems c kn p nl x 1 x 2 x 3 K N L Same input x k x k x k x k ˆp 1 ˆp 2 ˆp 3 ˆp 4 Same filter c c c c For each filter do L filtered X-LMS implementa=ons e 1 e 2 e 3 e 4 e 1 e 2 e 3 e 4 x k c Consider 1 filter c(n +1) = c µ et e c = c + 2µ e 2 1 c + e 2 2 c + e 2 3 c Requires a total of K N L filtered-x implementa=ons p 1 p 2 p 3 p 4 e 1 e 2 e 3 e 4 + e 2 4 c 12

13 Adjoint LMS Alterna>ve to Filtered-X LMS y x c M u P(z) N ŷ P - e LMS δ? Assume Calculate the gradient: e 2 c modeled by an FIR filter P(z) ŷ P = p 0 u + p 1 u(n 1) + p 2 u(n 2) = 2e e c = 2e n 2 k=n e u(k) u(k) c u z 1 z 1 N = 3 p 0 p 1 p 2 ŷ P e u(n i) = p i u(k) c = x(k) 13

14 Adjoint LMS ˆ n = 2e p 0 x + p 1 x(n 1) + p 2 x(n 2) ˆ n+1 = 2e(n +1) p 0 x(n +1) + p 1 x + p 2 x(n 1) ˆ n+2 = 2e(n + 2) p 0 x(n + 2) + p 1 x(n +1) + p 2 x 2 e p 0 + e(n +1) p 1 + e(n + 2) p 2 x N M mul=plica=ons N + M mul=plica=ons δ δ z z c(n +1) = c + 2µδx p 0 p 1 p 2 e y x c M u P(z) N ŷ P - e LMS δ P(z 1 ) 14

15 Adjoint LMS c(n +1) = c + 2µδx y x c M u P(z) N ŷ P - e LMS δ ˆP(z 1 ) δ not instantaneous gradient es>mate averaged gradient Delayed implementa>on c(n +1) = c + 2µδ(n N )x(n N ) Works for MIMO systems Significant computa>onal savings z 1 P(z) P(z 1 ) z δ 1 δ 2 P(z 1 ) e 1 e 2 e 3 e 4 15

16 Transform domain LMS Consider transforming the input before the linear es>mator u 1 T r a n s f o r m u 2 u M What would be the ideal transform? 16

17 Transform domain LMS For an orthogonal input the eigenvalue spread = 1 and we have fast convergence for LMS (circular bowl) R = 2 σ x 2 σ x 0! 0 σ x 2 T χ ( R) =1 Consider first the transform Q T where R = QΛQ T u = Q T x E uu T = QT xx T Q = Q T RQ = Λ c 1 Q T c 2 c 2 c 1 In image compression Q T is known as the Karhunen Loeve Transform 17

18 Transform domain LMS We want a circular bowl ( χ ( R) =1 ). So just power normalize u = Λ 1/2 Q T x = R 1/2 x R = R 1/2 R T /2 E uu T = I R 1 = R 1/2 T /2 R Cholesky Factoriza1on Possible Fixed Transforms DFT u i (n +1) = u i exp( j2πi / M ) + x(n +1) x(n +1 M ) c(n +1) = c + 2µe * u Order M update Complex coefficients DCT m i,m = R i 2 M cos(i(m +.5)π M R i =1/ 2 k = 0,M 2 =1 else u = M T x M T M = I 18

19 Transform domain LMS Last step is to power normalize each element separately u i = u i / ˆσ ui c(n +1) = c + 2µe * u u 1 u 1 1/ ˆσ ui T r a n s f o r m u 2 1/ ˆσ ui u 2 u M 1/ ˆσ ui u M Transform domain approaches ooen result in faster convergence Final MSE may be different 19

20 Block LMS Basic idea is to average the instantaneous gradient over a block of data Consider a single LMS update ŷ = c H (n 1)x c = c(n 1) + 2µex One step later c(n +1) = c + 2µe(n +1)x(n +1) Iterate for L steps c(n + L) = c(n 1) + 2µ e(n + l)x(n + l) L l=0 Weights are only updated every L steps. The sequence of weights will not be the same. Why? 20

21 Block LMS c(n + L) = c(n 1) + 2µ e(n + l)x(n + l) Convergence, stability, and >me constants all the same Is the misadjustment the same? L l=0 P ex P o µ L Trace(R) Depends on how you look at it. So what are the advantages? Fast implementa>on using FFT to perform convolu>ons L l=0 e(n + l)x(n + l) 21

22 Fast Block LMS Assuming a block length of M equals the FIR order M: LMS: 2M 2 mul>plica>ons Fast Block LMS: 10M log(2m ) +16M Can also improve convergence by power normalizing in the frequency domain See Haykin text for details 22

23 LMS Analysis - Misadjustment Start by calcula>ng the covariance of the LMS coefficients (at the bopom of the bowl) cov[!c] = E[!c!c T ]!c = c c o LMS: c(n +1) = c µ ˆ n write: ˆ n = n +V Where n = = 2R!c +V V = true gradient sta=onary noise Subs=tute into LMS rotate!c(n +1) = ( I 2µR) c! µv ( )!ʹ c!ʹ(n +1) = I 2µΛ E c!ʹ(n +1)!ʹ c (n +1) T = E I 2µΛ c µ V ʹ ( ) 2!ʹ c!ʹ = Q T!c c!ʹ c T + µ 2 V ʹ V ʹ T + 2(cross terms)(cross terms) 23

24 LMS Analysis - Misadjustment E c!ʹ(n +1)!ʹ c (n +1) T = E I 2µΛ ( ) 2!ʹ c!ʹ c T + µ 2 V ʹ V ʹ T + 2(cross terms)(cross terms) Assume sta=onarity of c!ʹ at the bohom of the bowl, zero mean noise V uncorrelated with c!ʹ. Thus the cross terms go to zero solving ( ) 2 cov!ʹ cov c!ʹ = I 2µΛ ( ) 1 cov ʹ cov c!ʹ = µ 4 Λ 2µΛ2 At the bohom of the bowl n 0 c + µ 2 cov V ʹ V V = ˆ n = 2ex cov V = 4E e2 xx T Near op=mal, e and x are uncorrelated by the principle of orthogonality cov V = 4P R cov V ʹ o = cov QT V = 4P Λ o 24

25 LMS Analysis - Misadjustment ( ) 1 cov ʹ cov c!ʹ = µ 4 Λ 2µΛ2 V cov V ʹ = cov QT V = 4P Λ o ( ) 1 4P o Λ cov c!ʹ = µ 4 Λ 2µΛ2 µp o Λ 1 Λ = µp o I cov!c = Qcov c!ʹ QT µp o I assume µλ << I P( n) P( n) = P o + P tr + P ex Misadjustment = P ex P o P o n 25

26 LMS Analysis - Misadjustment MSE P = P o + P ex Assume!c is sta=onary aner convergence Excess MSE P ex = E!c T R!c = E c!ʹ T Λ!ʹ c = P ex µp o M 1 λ k k=0 = µp o Trace R M 1 λ E[ cʹ ] 2 k k k=0 Diagonal elements of cov c!ʹ Misadjustment = P ex P o µ Trace R 26

27 LMS Analysis - Misadjustment Misadjustment Misadjustment = P ex P o µ Trace R Shows trade-off between learning rate and misadjustment In terms of >me constants M 1 M 1 1 Misadj. = µ λ k = µ = 1 M 1 1 = M k=0 k=0 2µτ k 4 k=0 (τ k ) mse 4 1 τ av Shows trade-off between misadjustment and rate of convergence 27