Predictive Coding. U n " S n

Transcription

1 Intrductin Predictive Cding The better the future f a randm prcess is predicted frm the past and the mre redundancy the signal cntains, the less new infrmatin is cntributed by each successive bservatin f the prcess Predictive cding idea: 1 Predict the current sample/vectr using an estimate which is a functin f past samples/vectrs f the input signal 2 Quantize residual between input signal and its predictin 3 Add quantizer residual and predictin t btain decded sample U n U n " S n +! Q! +! " -! S n S ˆ ˆ n S n Hw t btain the predictr Ŝn? Hw t cmbine predictr and quantizer? January 14, / 52

2 Outline Outline Predictin Linear Predictin Differential Pulse Cde Mdulatin (DPCM) Adaptive Differential Pulse Cde Mdulatin (ADPCM) Frward Adaptive DPCM Backward Adaptive DPCM Gradient Descent and LMS Algrithm Transmissin Errrs in DPCM January 14, / 52

3 Predictin Predictin Statistical estimatin prcedure: value f randm variable S n f randm prcess {S n } is estimated using values f ther randm variables f the randm prcess Set f bserved randm variables: B n Typical example: N randm variables that precede S n S n B n = {S n 1, S n 2,, S n N } (1) Predictr! ˆ S n +! -! Predictr fr S n : deterministic functin f bservatin set B n U n Ŝ n = A n (B n ) (2) Predictin errr U n = S n Ŝn = S n A n (B n ) (3) January 14, / 52

4 Predictin Predictin Perfrmance Defining MSE distrtin using u i = s i ŝ i and s i = u i + ŝ i d N (s, s ) = 1 N 1 (s i s N i) 2 = 1 N 1 (u i + ŝ i u i ŝ i ) 2 = d N (u, u ) (4) N i=0 i=0 Operatinal distrtin rate functin f a predictive cding systems is equal t the peratinal distrtin rate functin fr scalar quantizatin f the predictin residuals Operatinal distrtin rate functin fr scalar quant.: D(R) = σ 2 U g(r) σ 2 U : the variance f the residuals g(r): depends nly n the type f the distributin f the residuals Neglect the dependency n the distributin type Define: predictr A n (B n ) given an bservatin set B n is ptimal if it minimizes variance σ 2 U Assume statinary prcesses: A n ( ) becmes A( ) January 14, / 52

5 Predictin Optimal Predictin Optimizatin criterin used in literature [Makhul, 1975, Vaidyanathan, 2008, Gray, 2010] ɛ 2 U = E { { (Sn Un} 2 ) } { 2 (Sn = E Ŝn = E A n (B n ) ) } 2 (5) Minimizatin f secnd mment ɛ 2 U = E { (U n µ U + µ U ) 2} = E { (U n µ U ) 2} + 2E {(U n µ U )µ U } + E { µ 2 } U = σ 2 U + µ 2 U + 2µ U (E {U n } µ U ) = σ 2 U + µ 2 U (6) implies minimizatin f variance σ 2 U and mean µ U Slutin: cnditinal mean Ŝ n = A (B n ) = E {S n B n } (7) Prf: see [Wiegand and Schwarz, 2011, p. 150] January 14, / 52

6 Predictin Optimal Predictin fr Autregressive Prcesses Autregressive prcess f rder m (AR(m) prcess) S n = Z n + µ S + m a i (S n i µ S ) i=1 = Z n + µ S (1 a T me m ) + a T ms (m) n 1 (8) where {Z n} is a zer-mean iid prcess µ S is the mean f the AR(m) prcess a m = (a 1,, a m) T is a cnstant parameter vectr e m = (1,, 1) T is an m-dimensinal unit vectr Predictin f S n given the vectr S n 1 = (S n 1,, S n N ) with N m E {S n S n 1 } = E { Z n + µ S (1 a T N e N ) + a T N S n 1 S n 1 } where a N = (a 1,, a m, 0,, 0) T = µ S (1 a T N e N ) + a T N S n 1 (9) January 14, / 52

7 Linear Predictin Affine Predictin Affine predictr Ŝ n = A(S n k ) = h 0 + h T NS n k (10) where h N = (h 1,, h N ) T is a cnstant vectr and h 0 a cnstant ffset Variance σu 2 f predictin residual nly depends n h N { (Un σu 2 (h 0, h N ) = E E {U n } ) } 2 { (Sn } )2 } = E h 0 h T NS n k E {S n h 0 h T N S n k { (Sn = E E {S n } h T ( N Sn k E {S n k } )) } 2 (11) Mean squared predictin errr } 2 ɛ 2 U (h 0, h N ) = σu 2 (h N ) + µ 2 U (h 0, h N ) = σu 2 (h N ) + E {S n h 0 hn T S n k = σu 2 (h N ) + ( µ S (1 hn T ) 2 e N ) h 0 (12) Minimize mean squared predictin errr by setting h 0 = µ S (1 hn T e N ) (13) January 14, / 52

8 Linear Predictin Linear Predictin fr Zer-Mean Prcesses S n z "1 "1 z h 1 h 2 "1 z h N + -! U n ˆ S n +! +! The functin used fr predictin is linear, f the frm Ŝ n = h 1 S n 1 + h 2 S n h N S n N = h T NS n 1 (14) Mean squared predictin errr { σu 2 (h N ) = E (S n Ŝn) 2} { } = E (S n h T NS n 1 )(S n S T n 1h N ) = E { Sn 2 } } } 2E {h T NS n 1 S n + E {h T NS n 1 S T n 1h N = E { { } Sn} 2 2h T N E {S n S n 1 } + h T NE S n 1 S T n 1 h N (15) since h N is nt a randm variable January 14, / 52

9 Linear Predictin Aut-Cvariance Matrix and Aut-Cvariance Vectr Variance σ 2 S = E { S 2 n}, Aut-cvariance vectr (fr zer mean: aut-crrelatin vectr) c k = E {S n S n k } = σs 2 ρ k. ρ i. ρ N + k 1 with ρ i = E {S n S n i } /σ 2 S Aut-cvariance matrix (fr zer-mean: aut-crrelatin matrix) 1 ρ 1 ρ 2 ρ N 1 { } ρ 1 1 ρ 1 ρ N 2 C N = E S n 1 S T n 1 = σs 2 ρ 2 ρ 1 1 ρ N ρ N 1 ρ N 2 ρ N 3 1 (16) (17) January 14, / 52

10 Linear Predictin Optimal Linear Predictin Predictin errr variance σ 2 U (h N ) = σ 2 S 2h T N c k + h T NC N h N (18) Minimizatin f σ 2 U (h N) yields a system f linear equatins When C N is nn-singular C N h N = c k (19) h N = C 1 N c k (20) Minimum f σ 2 U (h N) is given as (with (C 1 N c k) T = c T k C 1 N ) σu 2 (hn) = σs 2 2 (hn ) T c k + (hn) T C N hn = σs 2 2 ( c T k C 1 ) N ck + ( c T k C 1 N )C N (C 1 N c ) k = σs 2 2 c T k C 1 N c k + c T k C 1 N c k = σs 2 c T k C 1 N c k. (21) In ptimal predictin, signal variance σ 2 S is reduced by ct k C 1 N c k January 14, / 52

11 Linear Predictin Verificatin f Optimality The ptimality f the slutin can be verified by inserting h N = h N + δ N int yieding σ 2 U (h N ) = σ 2 S 2 h T N c k + h T N C N h N (22) σ 2 U (h N ) = σ 2 S 2(h N + δ N ) T c k + (h N + δ N ) T C N (h N + δ N ) = σ 2 S 2 (h N ) T c k 2 δ T N c k + (h N ) T C N h N + (h N) T C N δ N + δ T N C N h N + δ T N C N δ N = σ 2 U (h N) 2δ T N c k + 2δ T N C N h N + δ T N C N δ N = σ 2 U (h N) + δ T N C N δ N (23) The additinal term is always nn-negative being equal t 0 nly if h N = h N δ T N C N δ N 0 (24) January 14, / 52

12 Linear Predictin The Orthgnality Principle Imprtant prperty fr ptimal affine predictrs { (Sn E {U n S n k } = E h 0 hn T ) } S n k Sn k = E {S n S n k } h 0 E {S n k } E { } S n k S T n k h N = c k + µ 2 S e N h 0 µ S e N (C N + µ 2 S e N e T N ) h N = c k C N h N + µ S e N ( µs (1 h T N e N ) h 0 ). (25) Inserting yields h N = C 1 N c k and h 0 = µ S (1 h T N e N ) (26) E {U n S n k } = 0 (27) January 14, / 52

13 Linear Predictin Orthgnality Principle and Gemetric Interpretatin Fr ptimal affine predictin, the predictin residual U n is uncrrelated with the bservatin vectr S n k E {U n S n k } = 0 (28) Therefre fr ptimum affine filter design, predictin errr shuld be rthgnal t input signal S 2 S ˆ * 0 S 0 U 0 * S 1 Apprximate a vectr S 0 by a linear cmbinatin f S 1 and S 2 Best apprximatin Ŝ 0 is given by prjectin f S 0 nt the plane spanned by S 1 and S 2 Errr vectr U 0 has minimum length and is rthgnal t the prjectin January 14, / 52

14 Linear Predictin One-Step Predictin I Randm variable S n is predicted using the N directly preceding randm variables S n 1 = (S n 1,, S n N ) T Using φ k = E {( S n E {S n } )( S n+k E {S n+k } )}, the nrmal equatins are given as φ 0 φ 1 φ N 1 φ 1 φ 0 φ N φ N 1 φ N 2 φ 0 h N 1 h N 2... h N N = φ 1 φ 2. φ N (29) where h N k represent elements f h N = (h N 1,, h N N )T Changing the equatin t 1 φ 1 φ 0 φ 1 φ N 1 h N 1 φ 2 φ 1 φ 0 φ N h N 2... φ N φ N 1 φ N 2 φ 0 h N N = (30) January 14, / 52

15 Linear Predictin One-Step Predictin II Including the predictin errr variance fr ptimal linear predictin using the N preceding samples σ 2 N = σ 2 S c T 1 C 1 N c 1 = σ 2 S c T 1 h N = φ 0 h N 1 φ 1 h N 2 φ 2 h N Nφ N (31) yields and additinal rw in the matrix φ 0 φ 1 φ 2 φ N φ 1 φ 0 φ 1 φ N 1 φ 2 φ 1 φ 0 φ N } φ N φ N 1 φ N 2 {{ φ 0 } C N+1 1 h N 1 h N 2. h N N }{{} a N = σn (32) Augmented nrmal equatin January 14, / 52

16 Linear Predictin One-Step Predictin III Multiplying bth sides f the augmented nrmal equatin with a T N σ 2 N = a T N C N+1 a N (33) Cmbing the equatins fr 0 t N preceding samples int ne matrix equatin yields σ 2 N X X X. h N σ C N+1. h N 2 h N N 1 X X = X X σ 2 1 X h N N h N 1 N 1 h σ0 2 Taking the determinant f bth sides f the equatin gives C N+1 = σ 2 N σ 2 N 1... σ 2 0 (34) Predictin errr variance σn 2 fr ptimal linear predictin using the N preceding samples σn 2 = C N+1 C N (35) January 14, / 52

17 Linear Predictin One-Step Predictin fr Autregressive Prcesses Recall: AR(m) prcess with mean µ S and a m = (a 1,, a m ) T S n = Z n + µ S (1 a T me m ) + a T ms (m) n 1 (36) Predictin using N preceding samples in h N with N m: define a N = (a 1,, a m, 0,, 0) T Predictin errr U n = S n h T NS n 1 = Z n + µ S (1 a T N e N ) + (a N h N ) T S n 1 (37) Subtracting the mean E {U n } = µ S (1 a T N e N) + (a N h N ) T E{S n 1 } U n E {U n } = Z n + (a N h N ) T ( S n 1 E {S n 1 } ) (38) Optimal predictin: cvariances between U n and S n 1 must be equal t 0 0 = E {( U n E {U n } )( S n k E {S n k } )} = E { Z n ( Sn k E {S n k } )} + C N (a N h N ) (39) yields h N = a N (40) January 14, / 52

18 Linear Predictin One-Step Predictin in Gauss-Markv Prcesses I Gauss-Markv prcess is a particular AR(1) prcess S n = Z n + µ S (1 ρ) + ρ S n 1, (41) fr which the iid prcess {Z n } has a Gaussian distributin Cmpletely characterized by its mean µ S, its variance σs 2, and the crrelatin cefficient ρ with 1 < ρ < 1 Aut-cvariance matrix and its inverse ( ) ( ) C 2 = σs 2 1 ρ C ρ ρ 1 2 = σs 2 (1 (42) ρ2 ) ρ 1 Aut-cvariance vectr c 1 = σ 2 S ( ρ ρ 2 ) (43) Optimum predictr h 2 = C 1 2 c 1 h 2 = 1 ( ) ( ) 1 ρ ρ 1 ρ 2 ρ 1 ρ 2 = 1 ( ) ( ρ ρ 3 ρ 1 ρ 2 ρ 2 + ρ 2 = 0 First element f h N is equal t ρ, all ther elements are equal t 0 ( N 2) ) January 14, / 52

19 Linear Predictin One-Step Predictin in Gauss-Markv Prcesses II Minimum predictin residual σu 2 = C 2 C 1 = σ4 S σ4 S ρ2 σs 2 = σs 2 (1 ρ 2 ) (44) Predictin residual fr filter h 1 U n = S n h 1 S n 1 Average squared errr σu 2 (h 1 ) = E { Un 2 } = σs(1 2 + h 2 1 2ρh 1 ) Nte: btain minimum MSE by σ 2 U (h 1) h 1 = σ 2 S(2h 1 2ρ)! = 0 als yields the result h 1 = ρ ρ January 14, / 52

20 Linear Predictin Predictin Gain Predictin gain using Φ N = C N /σs 2 and φ 1 = c 1 /σs 2 G P = E { } Sn 2 E {Un} 2 = σ2 S σs 2 1 σu 2 = σs 2 ct 1 C 1 N c = 1 1 φ T 1 Φ 1 N φ, (45) 1 Predictin gain fr ptimal predictin in first-rder Gauss-Markv prcess G P (h ) = 1 1 ρ 2 (46) Predictin gain fr filter h 1 20 G P (h 1 ) = = σ 2 S σs 2 (1 + h2 1 2ρh 1) h 2 1 2ρh lg 10 G P (h ) At high bit rates, 10 lg 10 G P : SNR imprvement achieved by predictive cding 0 10 lg 10 G P (h 1 ), h 1 = ρ January 14, / 52

21 Linear Predictin Optimum Linear Predictin fr K = 2 The nrmalized aut-crrelatin matrix and its inverse fllw as ( ) 1 ρ1 Φ 2 = Φ 1 ρ = 1 ( ) 1 ρ1 1 ρ 2 ρ With nrmalized crrelatin vectr φ 1 = ( ρ1 ρ 2 ) we btain the ptimum predictr h 2 = Φ 1 2 φ 1 = 1 ( ) ( ) 1 ρ1 ρ1 1 ρ 2 ρ ρ 2 ( ) 1 ρ1 (1 ρ = 2 ) 1 ρ 2 ρ 1 2 ρ 2 1 (47) (48) = 1 ( ρ1 ρ 1 ρ 2 1 ρ 2 ρ ρ 2 Result is identical t h fr the first-rder Gauss-Markv surce when setting ρ 1 = ρ and ρ 2 = ρ 2 Fr a surce with ρ 2 = ρ 2 1: secnd cefficient desn t imprve predictin gain can be generalized t N th-rder Gauss-Markv surces ) (49) January 14, / 52

22 Linear Predictin Predictin fr Speech Example Example fr speech predictin: ρ 1 = 0.825, ρ 2 = G P (1) = 5.0 db, G P (2) = 5.5 db Anther speech predictin example s[n] u[n], G P (1) = 4.2 db u[n], G P (3) = 7.7 db u[n], G P (12) = 11.7 db January 14, / 52

23 Linear Predictin Predictin in Images: Intra Frame Predictin Past and present bservable randm variables are prir scanned samples within that image Derivatins n linear predictin fr zer-mean randm variables (subtract µ S r rughly 127 frm 8-bit picture) Pictures are typically scanned line-by-line frm upper left crner t lwer right crner 1-D hrizntal predictin: Ŝ 0 = h 1 S 1 1-D vertical predictin: 2-D predictin: Ŝ 0 = h 2 S 2 Ŝ 0 = 3 h i S i i=1 h 3 S 3 + S 1 S 0 S 2 h 2 h U 0 ˆ S 0 January 14, / 52

24 Linear Predictin Predictin Example: Test Pattern σ 2 S = (s 127) Vertical Predictr h 1 = 0 h 2 = h 3 = 0 σu 2 (h) = G P = 8.82 db Hrizntal Predictr h 1 = h 2 = 0 h 3 = 0 σu 2 (h) = G P = db 2-d Predictr h 1 = h 2 = h 3 = σu 2 (h) = G P = db January 14, / 52

25 Linear Predictin Predictin Example: Lena center crpped picture σ 2 S = (s 127) Vertical Predictr h 1 = 0 h 2 = h 3 = 0 σu 2 (h) = G P = db Hrizntal Predictr h 1 = h 2 = 0 h 3 = 0 σu 2 (h) = G P = db 2-d Predictr h 1 = h 2 = h 3 = 0.48 σu 2 (h) = G P = db January 14, / 52

26 Linear Predictin Predictin Example: PMFs fr Picture Lena p(s) p(u) s u Pmfs p(s) and p(u) change significantly due t predictin peratin Entrpy changes significantly (runding predictin signal twards integer: E { U n } = 80.47) H(S) = 7.44 bit/sample H(U) = 4.97 bit/sample (50) Linear predictin can be used fr lssless cding: JPEG-LS January 14, / 52

27 Linear Predictin Asympttic Predictin Gain Upper bund fr predictin gain as N One-step predictin f a randm variable S n given the cuntably infinite set f preceding randm variables {S n 1, S n 2, } and {h 0, h 1, } U n = S n h 0 h i S n i, (51) Orthgnality criterin: U n is uncrrelated with all S n i fr i > 0 But U n k fr k > 0 is fully determined by a linear cmbinatin f past input values S n k i fr i 0 i=1 Hence, U n is uncrrelated with U n k fr k > 0 φ UU (k) = σ 2 U, δ(k) Φ UU (ω) = σ 2 U, (52) where σu, 2 is the asympttic ne-step predictin errr variance fr N January 14, / 52

28 Linear Predictin Asympttic One-Step Predictin Errr Variance I Fr ne-step predictin we shwed which yields C N = σ 2 N 1 σ 2 N 2 σ 2 N 3 σ 2 0 (53) N 1 1 N ln C N = ln C N 1 1 N = ln σi 2 (54) N If a sequence f numbers α 0, α 1, α 2, appraches a limit α, the average value appraches the same limit, Hence, we can write yielding lim N lim N ln C N 1 N N 1 1 N i=0 i=0 α i = α (55) N 1 1 = lim ln σi 2 = ln σ 2 (56) N N i=0 ( ) σ 2 = exp lim ln C N 1 N N = lim N C N 1 N (57) January 14, / 52

29 Linear Predictin Asympttic One-Step Predictin Errr Variance II Asympttic One-Step Predictin Errr Variance σ 2 U, = lim N C N 1 N Determinant f N N matrix: prduct f its eigenvalues ξ (N) i lim N C N 1 N = lim N ( N 1 i=0 ξ (N) i ) 1 N = 2 ( lim N N 1 i=0 1 N lg 2 ξ(n) i ) (58) Apply Grenander and Szegö s therem lim N N 1 1 N i=0 ( G ξ (N) i Expressin using pwer spectral density ) = 1 π G (Φ(ω)) dω (59) 2π π σu, 2 = lim C N 1 1 π N = 2 2π π lg 2 Φ SS(ω) dω N (60) January 14, / 52

30 Linear Predictin Asympttic Predictin Gain Predictin gain G P G P = σ2 S σu, 2 = 1 π 2π π 2 1 2π Φ(ω) dω Arithmetic mean π (61) π lg 2 Φ(ω) dω Gemetric mean Result fr first-rder Gauss-Markv surce (can als be cmputed differently) 20 Φ(ω) lg10 G P (ρ) db db 10 ρ = ρ = 0.5 ρ = ω/π db ρ January 14, / 52

31 Differential Pulse Cde Mdulatin (DPCM) Differential Pulse Cde Mdulatin (DPCM) Cmbining predictin with quantizatin requires simultaneus recnstructin f predictr at cder and decder use f quantized samples fr predictin U n U n S n + Q + S n - S ˆ S ˆ n n Re-drawing yields blck-diagram with typical DPCM structure S n P U n U n + Q - S ˆ + n S n P January 14, / 52

32 Differential Pulse Cde Mdulatin (DPCM) DPCM Cdec Redrawing with encder α, mapping frm index t bit stream γ, and decder β yields DPCM encder S n + - U n α I n B n B n γ Channel γ -1 I n ˆ S n β U ʹ n + ˆ S n β U ʹ n + P S ʹ n P S ʹ n DPCM Encder DPCM Decder DPCM encder cntains DPCM decder except fr γ 1 January 14, / 52

33 Differential Pulse Cde Mdulatin (DPCM) DPCM and Quantizatin Predictin Ŝn fr a sample Sn is generated by linear filtering f recnstructed samples S n frm the past K K Ŝ n = h i S n i = h i (S n i + Q n i) = h T (S n 1 + Q n 1 ) (62) i=1 i=1 with Q n being the quantizatin errr between the recnstructed samples S n and riginal samples S n Predictin errr variance (fr zer-mean input) is given by σu 2 = E { Un 2 } { = E (S n Ŝn)2} = E {(S n h T S n 1 h T Q n 1 ) 2} = E { { } { } Sn} 2 + h T E S n 1S T n 1 h + h T E Q n 1 Q T n 1 h (63) 2h T E {S ns n 1} 2h T E { } { } S nq n 1 + 2h T E S n 1Q T n 1 h Defining Φ = E { S n 1Sn 1} T /σ 2 S and φ = E {S ns n 1} /σs 2 we get ( ) σu 2 = σs h T Φ h 2h T φ (64) { } +h T E Q n 1 Q T n 1 h 2h T E { } { } S nq n 1 + 2h T E S n 1Q T n 1 h January 14, / 52

34 Differential Pulse Cde Mdulatin (DPCM) DPCM fr a First-Order Gauss-Markv Surce Calculate R(D) fr zer-mean Gauss-Markv prcess with 1 < ρ < 1 and variance σ 2 S S n = Z n + ρ S n 1 (65) Cnsider a ne-tap linear predictin filter h = (h) Nrmalized aut-crrelatin matrix Φ = (1) and crss-crrelatin φ = (ρ) Predictin errr variance σu 2 = σs 2 ( 1 + h 2 2 h ρ ) + h 2 E { Q 2 n 1} 2hE {S n Q n 1 } + 2h 2 E {S n 1 Q n 1 } (66) Using S n = Z n + ρ S n 1, the secnd rw in abve equatin becmes 2hE {S n Q n 1 } + 2h 2 E {S n 1 Q n 1 } = 2hE {Z n Q n 1 } 2hρE {S n 1 Q n 1 } + 2h 2 E {S n 1 Q n 1 } = 2hE {Z n Q n 1 } + 2h(h ρ)e {S n 1 Q n 1 } (67) With setting h = ρ, we have E {Z n Q n 1 } = 0 2h(h ρ)e {S n 1 Q n 1 } = 0 (68) January 14, / 52

35 Differential Pulse Cde Mdulatin (DPCM) Cmbinatin f DPCM with ECSQ fr Gauss-Markv Prcesses Expressin fr predictin errr variance simplifies t σu 2 = σs 2 ( 1 ρ 2 ) + ρ 2 E { Q 2 n 1} (69) Mdel expressin fr quantizatin errr D = E { Q 2 n 1} by an peratinal distrtin rate functin D(R) = σ 2 U g(r) (70) Example: Assume ECSQ and with that g(r) as g(r) = ε2 ln 2 a with a = and ε 2 = πe/6 lg 2 (a 2 2R + 1) (71) Expressin fr predictin errr variance becmes dependent n rate σ 2 U = σ 2 S 1 ρ 2 1 g(r) ρ 2 (72) January 14, / 52

36 Differential Pulse Cde Mdulatin (DPCM) Cmputatin f Operatinal Distrtin Rate Functin fr DPCM Operatinal distrtin rate functin fr DPCM and ECSQ fr a first-rder Gauss-Markv surce D(R) = σ 2 U g(r) = σ 2 S 1 ρ 2 g(r) (73) 1 g(r) ρ2 Algrithm fr ECSQ in DPCM cding 1 Initializatin with a small value f λ, set s n = s n, n and h = ρ 2 Create signal u n using s n and DCPM 3 Design ECSQ (α, β, γ) using signal u n and the current value f λ by minimizing D + λr 4 Cnduct DPCM encding/decding using ECSQ (α, β, γ) 5 Measure σ 2 U (R) as well as rate R and distrtin D 6 Increase λ and start again with step 2 Algrithm shws prblems at lw bit rates: instabilities January 14, / 52

37 Differential Pulse Cde Mdulatin (DPCM) Cmparisn f Theretical and Experimental Results I SNR [db] Space-Filling Gain: 1.53 db Distrtin-Rate Functin D(R) D(R) = σu 2 (R)g(R) EC-Llyd and DPCM G P = 7.21 db 10 EC-Llyd (n predictin) 5 D(R) = σs 2 g(r) Bit Rate [bit/sample] Fr high rates and Gauss-Markv surces, shape and memry gain achievable Space-filling gain can nly be achieved using vectr quantizatin Theretical mdel prvides a useful descriptin January 14, / 52

38 Differential Pulse Cde Mdulatin (DPCM) Cmparisn f Theretical and Experimental Results (I Predictin errr variance σu 2 depends n bit rate Theretical mdel prvides a useful descriptin 1 σ 2 U (R) σ 2 U (R) = σ2 U 1 ρ 2 1 g(r) ρ measurement σ 0.2 U 2 ( ) = σs 2 (1 ρ2 ) R[bit/symbl] January 14, / 52

39 Adaptive Differential Pulse Cde Mdulatin (ADPCM) Adaptive Differential Pulse Cde Mdulatin (ADPCM) Fr quasi-statinary surces like speech, fixed predictr is nt well suited ADPCM: Adapt the predictr based n the recent signal characteristics Frward adaptatin: send new predictr values - additinal bit rate S n + - U n α I n B n B n γ Channel γ -1 I n ˆ S n P β U n ʹ + ʹ S n ˆ S n P β U ʹ n + S ʹ n Buffer / APF APF DPCM Encder APF Decder APF DPCM Decder January 14, / 52

40 Adaptive Differential Pulse Cde Mdulatin (ADPCM) Frward-Adaptive Predictin: Mtin Cmpensatin in Vide Cding Since predictr values are sent, extend predictin t vectrs/blcks Use statistical dependencies between tw pictures Predictin signal btained by searching a regin in a previusly decded picture that best matches the blck t be cded Let s[x, y] represent intensity at lcatin (x, y) Let s [x, y] represent intensity in a previusly decded picture als at lcatin (x, y) J = min (dx,dy) (s[x, y] s [x dx, y dy]) 2 + λr(dx, dy) (74) x,y Predicted signal is specified thrugh mtin vectr (dx, dy) and R(dx, dy) is its number f bits Predictin errr u[x, y] is quantized (ften using transfrm cding) Bit rate in vide cding is sum f mtin vectr and predictin residual bit rate January 14, / 52

41 Adaptive Differential Pulse Cde Mdulatin (ADPCM) Backward Adaptive DPCM Backward adaptatin: use predictr cmputed frm recently decded signal N additinal bit rate Errr resilience issues Accuracy f predictr S n + - U n α I n B n B n γ Channel γ -1 I n ˆ S n P β U n ʹ + ʹ S n ˆ S n P β U ʹ n + S ʹ n APB APB APB DPCM Encder APB DPCM Decder January 14, / 52

42 Adaptive Differential Pulse Cde Mdulatin (ADPCM) Adaptive Linear Predictin Cmputatinal prblems when inverting Φ fr cmputing h = Φ 1 φ Gradient f bjective functin dσ2 U (h) dh = σu 2 (Φh φ) Instead f setting dσ2 U (h)! dh = 0 which leads t matrix inversin, apprach minimum by iteratively adapting predictin filter Steepest Descent Algrithm: Update filter cefficients in the directin f negative gradient f bjective functin h[n + 1] = h[n] + h[n] = h[n] + κ (φ Φh[n]) (75) σ 2 S σ 2 U (h 1) min h1 σ 2 U (h 1) 0 h 1 h 1 January 14, / 52

43 Adaptive Differential Pulse Cde Mdulatin (ADPCM) Least Mean Squared (LMS) Algrithm LMS is a stchastic gradient algrithm [Widrw, Hff, 1960] apprximating steepest descent LMS prpses simple current-value apprximatin { } Φ σs 2 = E S n 1 S T n 1 s n 1 s T n 1 (76) Update equatin becmes φ σ 2 S = E {S n 1 S n } s n 1 s n (77) h[n + 1] = h[n] + κ (s n 1 s n s n 1 s T n 1h[n]) = h[n] + κ s n 1 (s n s T n 1h[n]) (78) Realizing that predictin errr is given as u n = s n s T n 1h[n] h[n + 1] = h[n] + κ s n 1 u n (79) LMS is ne f many adaptive algrithms t determine h, including [Itakura and Sait, 1968, Atal and Hanauer, 1971, Makhul and Wlf, 1972] Autcrrelatin slutin Cvariance slutin Lattice slutin January 14, / 52

44 Adaptive Differential Pulse Cde Mdulatin (ADPCM) Linear Predictive Cding f Speech Speech cding is dne using surce mdeling All-ple signal prcessing mdel fr speech prductin is assumed Speech spectrum S(z) is prduced by passing an excitatin spectrum, V (z), thrugh an all-ple transfer functin H(z) = S(z) = H(z) V (z) = where, A(z) = 1 P k=1 a k z k Crrespnding difference equatin s(n) = G A(z) G V (z) (80) A(z) P a k s[n k] + G v[n] (81) k=1 When input v[n] is train f impulses, it prduces viced speech When v[n] is nise-like, it prduces unviced speech (e.g. sunds like f, s, etc) January 14, / 52

45 Adaptive Differential Pulse Cde Mdulatin (ADPCM) Predictin in Speech Predictin based n LPC is called Shrt Term Predictin (STP) as it generally perates n recent speech samples (e.g. arund 10 samples) ŝ[n] = After STP, resulting predictin errr N a i s[n i] (82) i=1 u[n] = s[n] ŝ[n] (83) still has distant sample crrelatin (knwn as Pitch) Pitch is predicted by Lng Term Predictin (LTP) by blck matching using crss-crrelatin R(l) = N 1 n=0 u[n] u[n l] N 1 n=0 u[n l] u[n l] (84) Lcatin l pt that maximizes crss-crrelatin is called lag Signal blck at lag is subtracted frm u[n] - resulting signal is called excitatin sequence January 14, / 52

46 Adaptive Differential Pulse Cde Mdulatin (ADPCM) Predictin in Speech: Cde Excited Linear Predictin (CELP) Instead f quantizing and transmitting the excitatin signal, CELP attempts t transmit an index frm a cdebk t apprximate the excitatin signal One methd wuld be t Vectr Quantize the excitatin signal t best match a cdebk entry But since this signal passes thrugh LPC synthesis filter, the behavir after filtering might nt necessarily be ptimal Analysis-by-Synthesis (AbS) apprach: Encding (analysis) is perfrmed by ptimising the decded (synthesis) signal in a clsed lp At encder, excitatin sequences in cdebk are passed thrugh synthesis filter and the index f best excitatin is transmitted January 14, / 52

47 Adaptive Differential Pulse Cde Mdulatin (ADPCM) CELP and AbS January 14, / 52

48 Transmissin Errrs in DPCM Transmissin Errrs in DPCM Fr a linear DPCM decder, the transmissin errr respnse is superimpsed t the recnstructed signal s Fr a stable DPCM decder, transmissin errr respnse decays Finite wrd-length effects at decder can lead t residual errrs that d nt decay (e.g. limit cycles) Belw: (a) errr sequence (BER f 0.5%) (b) errr-free transmissin (c) errr prpagatin January 14, / 52

49 Transmissin Errrs in DPCM Transmissin Errrs in DPCM fr Pictures Example: Lena, 3 b/p(fixed cde wrd length) 1D pred. hr. a H = 0.95, 1D pred. ver. a V = 0.95, 2D pred. a H = a V = 0.5 January 14, / 52

50 Transmissin Errrs in DPCM Transmissin Errrs in DPCM fr Mtin Cmpensatin in Vide Cding When transmissin errr ccurs, mtin cmpensatin causes spati-tempral errr prpagatin Try t cnceal image parts that are in errr Cde lst image parts withut referencing cncealed image parts helps but reduces cding efficiency intra blck Intra blck Cncealed image part Use clean reference picture fr mtin cmpensatin Cncealed image part January 14, / 52

51 Transmissin Errrs in DPCM Summary n Predictive Cding Predictin: Estimatin f randm variable frm past r present bservable randm variables Optimal predictin nly in special cases Optimal linear predictin simple and efficient Wiener-Hpf equatin fr ptimal linear predictin Gauss-Markv prcess f rder N requires predictr with N cefficients that are equal t crrelatin cefficients Nn-matched predictr can increase signal variance Optimal predictin errr is rthgnal t input signal Optimal predictin errr filter perates as a whitening filter January 14, / 52

52 Transmissin Errrs in DPCM Summary n Predictive Cding (cnt d) Differential pulse cde mdulatin (DPCM) is structure fr cmbinatin f predictin with quantizatin In DPCM: predictin is based n quantized samples Simple and efficient: cmbine DPCM and ECSQ Extensin f Entrpy-Cnstrained Llyd algrithm twards DPCM Fr Gauss-Markv surces, EC-Llyd fr DPCM achieves shape and memry gain Adaptive DPCM: frward and backward adaptatin Frward adaptatin requires transmissin f predictr values Backward adaptatin pses prblem f errr resilience and accuracy questins Adaptive linear predictin using steepest descent algrithm: LMS, autcvariance, cvariance, and lattice slutins Transmissin errrs cause errr prpagatin in DPCM Errr prpagatin can be mitigated by interrupting errneus predictin chain January 14, / 52