12.6 Sequential LMMSE Estimation

Transcription

1 12.6 Sequetial LMMSE Estimatio Same kid if settig as fo Sequetial LS Fied umbe of paametes (but hee they ae modeled as adom) Iceasig umbe of data samples Data Model: [ H[ θ + w[ (+1) 1 p 1 [ [[0] [] ukow PDF kow mea & cov (+1) p kow H[ 1] H[ h [ (+1) 1 w[ [w[0] w[] ukow PDF kow mea & cov C w must be diagoal with elemets σ 2 θ & w ae ucoelated Goal: Give a estimate θˆ [ 1] based o [ 1], whe ew data sample [ aives, update the estimate to θ ˆ[ ] 1

2 Developmet of Sequetial LMMSE Estimate Ou Appoach Hee: Use vecto space ideas to deive solutio fo DC Level i White Noise the wite dow geeal solutio. [ A + w[ Fo coveiece Aume both A ad w[ have zeo mea Give [0] we ca fid the LMMSE estimate 2 ˆ E A[0]} E A( A + w[ )} σ A 0 [0] [0] A [0] E [0]} E( A w[ ) } + A σ + σ Now we seek to sequetially update this estimate with the ifo fom [1] 2

3 Fom Vecto Space View: A [0] Fist poject [1] oto [0] to get ˆ[1 0] Â 0 Â 1 [1] Estimate ew data give old data Pedictio! Use Othogoality Piciple ~ [1] [1] ˆ[1 0] is to [0] Notatio: the estimate at 1 based o 0 his is the ew, o-edudat ifo povided by data [1] It is called the iovatio ~ [1] [0] Â 0 [1] 3

4 Fid Estimatio Update by Pojectig A oto Iovatio ~ ~ ~ ~ ˆ [1] ~ [1]} [1] ~ ~ [1] ~ [1] E A A 1 A, A, [1] [1] [1] ~ 2 ~ 2 [1] E [1]} Gai: k 1 Recall Popety: wo Estimates fom data just add: Aˆ 1 Aˆ 0 + Aˆ 1 ~ [1] [0] Aˆ Aˆ Aˆ 0 + k 1 ~ [1] [ [1] ˆ[1 0] ] k1 Â 1 ~ [1] Â 0 [1] Old Estimate Gai New Data Pedicted New Data [0] Â 1 Iovatio is Old Data 4

5 he Iovatios Sequece he Iovatios Sequece is Key to the deivatio & implemetatio of Seq. LMMSE A sequece of othogoal (i.e., ucoelated) RVs Boadly sigificat i Sigal Poceig ad Cotols ~ [0], ~ [1], ~ [2], } [0] [ 1] ˆ[1 0] [ 2] ˆ[2 1] Meas: Based o ALL data up to 1 (iclusive) 5

6 Geeal Sequetial LMMSE Estimatio Iitializatio No Data Yet! Use Pio Ifomatio ˆ 1 θ E θ} Estimate M 1 C θθ MMSE Mati Update Loop Fo 0, 1, 2, k M h Gai Vecto Calculatio σ + h M h θˆ ˆ [ ˆ ] [ h θ θ 1 + k 1 Estimate Update M [ ] I k h M 1 MMSE Mati Update 6

7 Sequetial LMMSE Block Diagam Obsevatio Iovatio 2 M 1, h, σ Compute Gai Data Model [ H[ θ + w[ H[ 1] H[ h [ [ + Σ ~ [ ] k + Σ + θˆ 1 ( [ h θˆ ) + k 1 θˆ ˆ [ 1] h Pedicted Obsevatio θˆ 1 h h Pevious Estimate ˆ 1 θ z -1 Delay Updated Estimate Eact Same Stuctue as fo Sequetial Liea LS!! 7

8 Commets o Sequetial LMMSE Estimatio 1. Same stuctue as fo sequetial liea LS. BU they solve the estimatio poblem ude vey diffeet aumptios. 2. No mati ivesio equied So computatioally Efficiet 3. Gai vecto k weighs cofidece i ew data (σ 2 ) agaist all pevious data (M -1 ) whe pevious data is bette, gai is small do t use ew data much whe ew data is bette, gai is lage ew data is heavily used 4. If you kow oise statistics σ 2 ad obsevatio ows h ove the desied age of : Ca u MMSE Mati Recusio without data measuemets!!! his povides a Pedictive Pefomace Aalysis 8

9 12.7 Eamples Wiee Filteig Duig WWII, Nobet Wiee developed the mathematical ideas that led to the Wiee filte whe he was wokig o ways to impove ati-aicaft gus. He posed the poblem i C- fom ad sought the best liea filte that would educe the effect of oise i the obseved A/C tajectoy. He modeled the aicaft motio as a wide-sese statioay adom poce ad used the MMSE as the citeio fo optimality. he solutios wee ot simple ad thee wee may diffeet ways of itepetig ad castig the esults. he esults wee difficult fo egiees of the time to udestad. Othes (Kolmogoov, Hopf, Leviso, etc.) developed these ideas fo the D- case ad vaious special cases. 9

10 Weie Filte: Model ad Poblem Statemet Sigal Model: [ s[ + w[ Obseved: Noisy Sigal Model as WSS, Zeo-Mea C R Desied Sigal Model as WSS, Zeo-Mea C R Noise Model as WSS, Zeo-Mea C ww R ww C coelatio mati R E covaiace mati E( E })( E }) } } Same if zeo-mea Poblem Statemet: Poce [ usig a liea filte to povide a de-oised vesio of the sigal that has miimum MSE elative to the desied sigal LMMSE Poblem! 10

11 Filteig, Smoothig, Pedictio emiology fo thee diffeet ways to cast the Wiee filte poblem Filteig Smoothig Pedictio Give: [0], [1],, [ Give: [0], [1],, [N-1] Give: [0], [1],, [N-1] Fid: sˆ [ Fid: s ˆ[0], sˆ[1],, sˆ[ N 1] Fid: ˆ [ N + l], l > 0 [ [ [ Note!! ŝ[0] ŝ[1] ŝ[2] ŝ[3] ŝ [0] ŝ [1] ŝ [2] ŝ[3] ˆ[5] All thee solved usig Geeal LMMSE Est. ˆ CθC 1 θ 11

12 ˆ CθC 1 θ Filteig θ s[ (scala) Cθ C sˆ[ E E } s[ } s[ s [ [ " [0]] ~ E (vecto!) } ( s + w)( s + w E ) ~ R } + ww ( R + R ww + R ww 1 [ 1 ( + 1) ][( + 1) ( + 1) ][( + 1) 1] ) C Smoothig θ s (vecto) C θ E E E E R } s } s( s + w) } + sw (Mati!) } ( s + w)( s + w E ) sˆ R R } + ww ( R + R ww + R ww [ N N][ N N][ N 1] ) 1 Pedictio θ [ N 1+ l] (scala) C θ ˆ[ N E ot s! } [ N 1+ l] [ [ N 1+ l] " [ l] ] ~ C (vecto!) 1+ l] E R ~ } R 1 [ 1 N][ N N][ N 1] 12

13 1 s ˆ[ ~ ( R + R ww ) a &# ## %### $ h a ( ) ( ) ( ) [ h [0] h [1]... h [ ] [ a a... a ] 1 Commets o Filteig: FIR Wiee 0 sˆ[ k 0 h ( ) Wiee-Hopf Filteig Equatios [ k] [ k] Wiee Filte as ime-vayig FIR Filte Causal! Legth Gows! ( R + R ww ) h &#%#$ # R [ [0] [1]... [ ] [0] [1] " [ h [1] [0] " [ 1] h ' ' ( ' [ [ 1] " [0] h &###### %###### $ Symmetic & oeplitz ( ( ( ) [0] ) [1] ' ) [ [0] [1] ' [ I Piciple: Solve WHF Eqs fo filte h at each I Pactice: Use Leviso Recusio to Recusively Solve 13

14 Commets o Filteig: IIR Wiee Ca Show: as Wiee filte becomes ime-ivaiat hus: h () [k] h[k] he the Wiee-Hopf Equatios become: k 0 h[ k] [ l k] [ l] l 0,1, ad these ae solved usig so-called Spectal Factoizatio Ad the Wiee Filte becomes IIR ime-ivaiat: sˆ[ k 0 h[ k] [ k] 14

15 Revisit the FIR Wiee: Fied Legth L he way the Wiee filte was fomulated above, the legth of filte gew so that the cuet estimate was based o all the past data Refomulate so that cuet estimate is based o oly L most ecet data: [3] [4] [5] [6] [7] [8] [9] ŝ[6] ŝ[7] ŝ[8] sˆ[ L 1 k 0 h[ k] [ k] Wiee-Hopf Filteig Equatios fo WSS Poce w/ Fied FIR ( R + R ww ) h &#%#$ # R [ [0] [1]... [ ] [0] [1] " [ h[0] [1] [0] " [ 1] h[1] ' ' ( ' ' [ [ 1] " [0] h[ &####### %####### $ Symmetic & oeplitz Solve W-H Filteig Eqs ONCE fo filte h [0] [1] ' [ 15

16 Commets o Smoothig: FIR Smoothe sˆ R R + R &# ( ## ) %### ww $ W 1 W Each ow of W like a FIR Filte ime-vayig No-Causal! Block-Based o itepet this Coside N1 Case: sˆ[0] [0] [0] + ww [0] [0] SNR SNR 1 &#%#$ + 1, High SNR 0, Low SNR [0] 16

17 Commets o Smoothig: IIR Smoothe Estimate s[ based o, [ 1], [0], [1], } k s ˆ[ h[ k] [ k] ime-ivaiat & No-Causal IIR Filte k he Wiee-Hopf Equatios become: h[ k] [ l k] [ l < l < h[ [ [ ] Diffes Fom Filte Case Sum ove all k H( f ) 1 whe P ( f ) >> P ww ( f ) H( f ) 0 whe P ( f ) << P ww ( f ) Diffes Fom Filte Case Solve fo all l H ( f ) P P P ( f ) ( f ) ( f P ( f ) ) + P ww ( f ) 17

18 Relatioship of Pedictio to AR Est. & Yule-Walke Wiee-Hopf Pedictio Equatios R h [ [ l] [ l + 1]... [ l + N 1] ] [0] [1] " [ N 1] h[0] [1] [0] " [ N 2] h[1] ' ' ( ' ' [ N 1] [ N 2] " [0] h[ &####### %######## $ Symmetic & oeplitz [ l [ l + [ l] ' + 1] Fo l1 we get EXACLY the Yule-Walke Eqs used i E to solve fo the ML estimates of the AR paametes!!! FIR Pedictio Coefficiets ae estimated AR pams Recall: we fist estimated the ACF lags [k] usig the data he used the estimates to fid estimates of the AR paametes N 1] Rˆ h ˆ 18

19 Relatioship of Pedictio to Ivese/Whiteig Filte Sigal Obseved AR Model u[k] 1 1 a( z) White Noise [k] Ivese Filte: 1 a(z) FIR Ped. a(z) + Σ ˆ [ k] u[k] White Noise Imagiatio & Modelig Physical Reality 1-Step Pedictio 19

20 Results fo 1-Step Pedictio: Fo AR(3) At each k we pedict [k] usig past 3 samples 4 2 Eo Sigal Pedictio Sigal Value Sample Ide, k Applicatio to Data Compeio Smalle Dyamic Rage of Eo gives Moe Efficiet Biay Codig (e.g., DPCM Diffeetial Pulse Code Modulatio) 20