Special Topics: Data Science

Transcription

1 Special Topics: Data Science L6b: Wiener Filter Victor Solo School of Electrical Engineering University of New South Wales Sydney, AUSTRALIA

2 Topics Norbert Wiener MIT Professor (Frequency Domain) Wiener Filters 1 Signal Extraction: Wiener optimum Filter MSE Example Deconvolution 3 Spectral Identification- Estimation Signal Extraction occurs in many disciplines e.g. estimation of: long term temperature trends; employment; signals from seismic arrays. Deconvolution - where the signal of interest has been filtered before being recorded - also occurs widely eg medical imaging; oceanography; geological prospecting; remote sensing. It is closely associated with ill-conditioned inverse problems. It is possible to obtain the (frequency domain) Wiener filter by taking limits in the finite data Wiener filter. But it is simpler to derive the frequency domain Wiener filter from scratch. Prof. V. Solo (UNSW) / 10

3 Signal Extraction I Model y t = s t + n t, s t n u for all t, u s t is stationary with spectrum F s (ω) n t is stationary with spectrum F n (ω). So y t is stationary with spectrum (ω) = F s (ω) + F n (ω). Proof I ŝ t = Σ W t u y u = (W y) t s t ŝ t = s t (W (s + n)) t = ((δ W ) s) t (W n) t E(s t ŝ t ) = E((δ W ) s) t + E(W n) t Result III: The MMSE linear infinite data estimator is: ŝ t = W (z 1 )y t W (e jω ) = Fs (ω) (ω) = 1 1+F n(ω)/f s (ω) Note that W (z 1 ) is a two-sided filter. W (z 1 ) = Σ w r z r All signals have zero mean. W = W (ω) = W (e jω ) F (ω) = γ(e jω ) = Σ γr e jωtr By Parseval s Theorem E((δ W ) s) t = π π F ((δ W ) s(ω) dω π By the spectral-filtering results F ((δ W ) s (ω) = 1 W F s (ω) Similarly E(W n) t = π π W F n (ω) dω π mse = π π ( 1 W F s + W F n ) dω π This is quadratic in W and can be optimized by completing the square. Prof. V. Solo (UNSW) 3 / 10

4 Signal Extraction II Proof II Set W o = which we add and subtract inside the integrand. mse(ω) = 1 W o ( W W o ) F s + W o + ( W W o ) F n = 1 W o F s + W W o F s M s Re( W W o )(1 W o )F s + W o F n + W W o F n M n +Re( W W o ) W o F n = M s + cp + M n = mse o + cp cp = Re( W W o )[ W o F n (1 W o )F s ] = Re( W W o )[ W o (F n + F s ) F s ] = 0 Fs (ω) (ω) = Fs So mse(ω) = mse o (ω) + W W o mse o (ω) with equality iff W = W o. Optimum MSE mse o (ω) = 1 W o F s + W o F n = 1 Fs F s +F n F s + Fs F n = F n F F y s + F s F F y n = F n F s (F n + F s )/Fy = F n F s / = W o F n = F n F s /(F n + F s ) = Fs 1+λ ω λ ω = Fs F n is a frequency domain vsnr. So mse o = π F s dω π 1+λ ω π Optimum relative mse is then rmse = mse o var(s = Fs dω 1+λω t) dωπ Fs π < 1 Prof. V. Solo (UNSW) 4 / 10

5 Signal Extraction Example: AR(1)+WN AR(1) in white noise σ F s = s = σ 1 φe jω s A & F n = σn = σs / A + σn Fs = σ s +σ n A A = σ s A A N = σ s N = N A Now look for a spectral factorization N = B = V 1 αe jω. V (1 + α αcos(ω)) = σ s + σ n(1 + φ φcos(ω)) V (1 + α ) = σ s + σ n(1 + φ ) αv = σ nφ Divide nd equation by 1st to get α 1+α = σ n φ σ s +σ n (1+φ ) = φ λ+1+φ = 1 M where λ = vsnr = σ s σ n So solve: α Mα + 1 = 0 α = 1 (M ± M 4) So need M (λ φ )/φ λ φ φ λ + (1 φ) 0 which holds. Stable solution α = 1 (M M 4) (why?) Next sgn(φ) = sgn(m) = sgn(α) V = σn φ α = σ φ n α Prof. V. Solo (UNSW) 5 / 10

6 AR(1)+WN II W o W o = Fs = = σ s N σ s V 1 αe jω = λ α φ = σ s α σ n φ 1 1 αe jω 1 σ = 1 αe jω 1 αe jω Filter Properties quadratic equation α 1+α = φ λ+1+φ < φ α φ 1+α = λ+1+φ 1+φ Check that, for x > 0 x 1+x is an increasing function of x But LHS<RHS α < φ V > σn quadratic as λ 0 then α φ quadratic as λ then α 0. Optimum mse 1st var(s t ) = σ s 1 φ then mse o = dω W o F n π = σ σ 1 αe jω n dω π = σ σ n 1 α So relative mse is σ σ n 1 α 1 φ σs 1 φ λ 1 α = α 1 α / φ 1 φ = λ α 1 φ Check that for 0 < x < 1, x 1 x is an increasing function of x. It follows that, as expected, rmse < 1. as λ 0 then rmse 1 as λ then rmse 0 Prof. V. Solo (UNSW) 6 / 10

7 AR(1)+WN III W o = Forwards-Backwards Filter W o = Two-sided Filter The last term is an AR(1) spectrum so = γ o Σ α r e jωr σ 1 αe jω where γ o = σ 1 α = λ α 1 φ 1 α. So the weights are w r = γ o α r So the Wiener signal estimator in the time domain is: ŝ t = γ o Σ α r y t r = γ o (w y) t In practice the sum is truncated once the weights are very small. σ We rewrite W = 1 αe jω in z-transform notation as σ 1 αz σ 1 αz 1. W (z 1 ) = Then ŝ t = W (z 1 )y t = σ σ 1 αz 1 αz y 1 t. This suggests ŝ t can be computed in two stages: Forward or Causal filtering ξ t = σ 1 αz 1 y t Followed by Backwards or one-sided anti-causal filtering ŝ t = σ 1 αz ξ t Prof. V. Solo (UNSW) 7 / 10

8 General Approach to Computation of Wiener Filter Numerous approaches possible. We use direct MA spectral factorisation via Wilson s algorithm. It is compact, fast and reliable. Suppose both F s, F n are described by ARMA models Bs F s = A s and F n = Bn A n Then W = Fs Bs Bs = A s /( A s + Bn A n ) = B s A n B s A n + B na s Use Wilson s algorithm to get: B s A n + B n A s = θ where θ(z 1 ) is one-sided. Then W o = W o (z 1 ) = Bs An θ Wilson s Newton Algorithm Given MA covariances cr, 0 r m form the vector c = [c0, c 1,, c m] Introduce the MA parameter vector θ = [θ 0, θ,, θ m ] and corresponding covariances c r (θ) = Σ l=m r l=0 θ l θ l+r and vector c(θ) Wilson s iteration is: θ (k+1) = T 1 (θ k )(c(θ (k) ) + c ) where T (θ) = T L (θ) + T R (θ) T L (θ) = Bs (z)an(z) B s (z 1 )A n(z 1 ) θ(z) θ(z 1 ).. = backwards filter forwards θ 0 filter Prof. V. Solo (UNSW) 8 / 10 T R (θ) = θ 0 θ 1 θ m θ 1 θ θ m 0.. θ m 0 0 θ 0 θ 1 θ m 0 θ 0 θ 1 θ m 1 θ m

9 Topics Deconvolution 1 Wiener Filter Optimum MSE 3 Computation Prof. V. Solo (UNSW) /5

10 Wiener Deconvolution Filter Deconvolution generalizes the signal extraction problem. The model for the recorded data is y t = s t + n t But the signal of interest has been filtered to generate s t s t = k x t = k(z 1 )x t where k(z 1 ) could be two-sided. The problem is to estimate: x t from infinite data y, We consider a linear estimator ˆx t = (W y) t and show W o = k Fx = k Fx k F x +F n The mse is E(x t ˆx t ) = E(et ) = dω F e π Then F e = F x + Fˆx ReF x ˆx = F x + W Re(F xy W ) Set W o = Fxy and complete the square to show this is the optimal filter. Continuing W o = F x k k F x +F n Fx k = k Fx k F x +F n F s F s +F n W o = 1 k = 1 k Thus the WF functions by first estimating s and then inverse filtering that to get ˆx t. But k may have zeros so this implementation is not reliable. The 1st formula does not have that problem. Prof. V. Solo (UNSW) 3 / 5

11 Optimum MSE MSE We have mse(ω) = F e and F e = F x + W o Re(F xy W o ) = F x + W o Re(F x k W o ) = F x + k F x F y = F x k F x = F x (1 k F x F = F n x = F x F n = k F x +F n F x 1+ k λ ω F x F x k ) natural extension of signal extraction where λ ω = Fx F n which is also a natural extension of the signal extraction formula. Relative MSE The relative mse is then rmse = Fx Fn Fy dω π Fx dω π < 1 as λ ω 0 then rmse 1 as λ ω then rmse 0 Prof. V. Solo (UNSW) 4 / 5

12 Computation via Wilson s Algorithm Suppose F x, F n are ARMA spectra and k is a proper rational transfer function F x = Nx D x, F n = Nn D n, k = N k D k W o = k = N k D k = N k D k Fx k F x +F n N x / D x Nx N k Dx D k + Nn Dn N x D n D k N x N k D n + D k N n D k Here we do the one-sided case. If k is two-sided we write it as a sum of a causal filter + anti-causal filter. Then the aproach here is easily modified. Now apply Wilson s spectral factorization algorithm to factor the denominator as θ where θ(z 1 ) is one-sided. Then W o = N k N Dk θ where N = N x D n D k is one-sided. So W o (z 1 ) = N k (z) D k (z) = backwards filter backwards filter N(z) N(z 1 ) θ(z) θ(z 1 ) forwwards filter The two-sided weights can also be computed similarly to the signal extraction case but will now not be symmetric in lag. Prof. V. Solo (UNSW) 5 / 5

13 Appendix A: Backwards or Anti-causal Filtering Let H(z) = Σ 0 h r z r be an anti-causal filter. We show s t = H(z)u t can be implemented by a causal filtering of the reversed input sequence. Given a fixed time interval [0, T ] introduce the reversed sequences: ū t = u T t and s t = s T t We illustrate with a single pole filter H(z) = 1 1 φz. Then s t = 1 1 φz u t (1 φz)s t = u t s t = φs t+1 + u t. Consider the pairs of equations. s T 1 = φs T + u T 1 s 1 = φ s 0 + ū 1 s T = φs T 1 + u T s = φ s 1 + ū.. s T t = φs T t+1 + u T t s t = φ s t 1 + ū t The LHS is the anti-causal filtering; the RHS is the causal equivalent on the reversed input sequence. It generates the reverse output sequence. Reversing that gives the output sequence. Prof. V. Solo (UNSW) 9 / 10

14 Appendix B: Wilson s Newton Spectral Factorisation Newton s Algorithm Solve φ(x (m+1) 1 ) = f (m+1) 1 (x) a. Let x o be a start value. Do a Taylor series around x o φ(x) φ(x o ) + dφ (x x dxo T o ) Us this to find x 1 so that φ(x 1 ) 0. 0 = φ(x o ) + dφ (x dxo T 1 x o ) x 1 x o = [ dφ ] 1 (f (x dxo T o ) a) Iterate this update. There is no general proof of convergence. Results can be found under various conditions. In this case Wilson supplied a proof under the minimal condition that the MA covariance generating function is positive definite. Wilson s Algorithm Here we have f r = c r cr = Σ l=m r l=0 θ l θ l+r cr, r = 0, 1,, m fr θ s =< θ s+r > + < θ s r > where { θi if i = 0,, m < θ i >= 0 if otherwise Next check that T L (θ)θ = c(θ) = T R (θ)θ [ fr θ s ] = T L (θ) + T R (θ) = T (θ) Applying the Newton update gives the iteration θ (k+1) = T 1 (θ (k) )(c(θ (k) ) + c ) Prof. V. Solo (UNSW) 10 / 10