SOURCE CODING WITH SIDE INFORMATION AT THE DECODER (WYNER-ZIV CODING) FEB 13, 2003

Transcription

1 SOURCE CODING WITH SIDE INFORMATION AT THE DECODER (WYNER-ZIV CODING) FEB 13, 2003

2 SLEPIAN-WOLF RESULT { X i} RATE R x ENCODER 1 DECODER X i V i {, } { V i} ENCODER 0 RATE R v Problem: Determine R, the set of all achievable rate pairs. Result: H(V) R v H(V X) H(X V) H(X) R x R= {(R x,r v ) R x H(X V ), R v H(V X), R x + R v H(X, V )}

3 SOURCE CODING WITH SIDE INFORMATION { X i } ENCODER 1 RATE R 1 DECODER 1 { X i } { V i } ENCODER 0 RATE R 0 Goal: Design encoders E 0, E 1 to encode X with the optimal rate, and decoder D 1 to decode ˆX with arbitrarily low probability of error. {V k } is statistically dependent on {X k }. Rate of E i is R i bits/source-symbol. A rate pair (R 0,R 1 ) is achievable if encoders E 0, E 1 (with parameters R 0, R 1 ) and a decoder D 1 that can reproduce X with arbitrarily high reliability i.e., if mappings Encoder 0: f 0 : V n {0, 1,...,2 nr 0 1}, Encoder 1: f 1 : X n {0, 1,...,2 nr 1 1}, Decoder 1: g 1 : {0, 1,...,2 nr 0 1} {0, 1,...,2 nr 1 1} X n such that the reconstruction error = 1 n E[d H(X n, ˆX n )] ɛ,

4 Problem: Determine R, the set of all achievable rate pairs. Result: If Q(x, v) =Pr{X = x, V = v}, then (R 0,R 1 ) is achievable if and only if: R 1 H(X W ) and R 0 I(V ; W ) for some auxiliary random variable W satisfying (a) p(x, v, w) =Q(x, v), (b) p(x, v, w) =Q(x, v)p t (w v) w W [A. D. Wyner, On source coding with side information at the decoder, IEEE Transactions on Information Theory, May, 1975.]

5 ZERO-ERROR SOURCE CODING WITH SIDE INFORMATION { X i} RATE R 1 ENCODER 1 DECODER 1 { X i} { V i} ENCODER 0 RATE R 0 { X i } ENCODER 1 RATE R 1 DECODER 1 { X i } { Y i} ENCODER 2 RATE R 2 DECODER 2 { Y i} Y 0 { V i } ENCODER 0 RATE R 0 R = Result: If Q(x, y, v) =Pr{X = x, Y = y, V = v}, then (R 0,R 1,R 2 ) is achievable if and only if: R 1 H(X W ),R 2 H(Y W ), and R 0 I(V ; W ) for some auxiliary random variable W satisfying (a) w W p(x, y, v, w) =Q(x, y, v), (b) p(x, y, v, w) =Q(x, y, v)p t (w v) Let P be the family of probability mass functions p(x, y, v, w) satisfying (a) and (b). The set of achievable rate triplets is: { (R 0,R 1,R 2 ) : R 0 I(V ; W ),R 1 H(X W ),R 2 H(Y W ),p P } [c] [c] denotes closure.

6 LOSSY SOURCE CODING WITH SIDE INFORMATION { X i } ENCODER RATE R DECODER { X i } A { Y i } Goal: Design an encoder E c that encodes X at the optimal rate and a decoder D c that reconstructs ˆX within an average distortion d, given side information Y. Y is statistically dependent on X. Rate of E c is R bits/source-symbol. A pair (R, d) is achievable if an encoder E c of rate R and a decoder D c that can reconstruct X within an average distortion d i.e., mappings Encoder: f E : X n {0, 1,...,2 n(r) 1}, such that: Decoder: f D : Y n {0, 1,...,2 n(r) 1} ˆX n, lim sup E[D(X n,f D (Y n,f E (X n )))] d n

7 Let R X Y (d) be the smallest allowable rate if Y is available to both E c and D c. Let RY (d) be the smallest allowable rate if Y is available only to D c. Problem: Determine the set of achievable pairs (R, d). Theorem 1 If (X, Y ) are drawn i.i.d. according to Q(x, y)=pr{x = x, Y = y}, and D : X ˆX R +, (D(x n, ˆx n )= 1 n i D(x i, ˆx i )), then R Y (d) = min min p(w x) f where W is an auxiliary random variable satisfying (a) w W (I(X; W ) I(Y ; W )) (1) p(x, y, w) =Q(x, y), (b) p(x, y, w) =Q(x, y)p t (w x) The minimization in (1) is over all f : Y W ˆX and p(w x), W X +1,such that E[D(x, ˆx)] = x w y Q(x, y)p(w x)d(x, f(y, w)) d. RY (d) R X Y (d) (For d =0, R Y (0) = R X Y (0) = H(X Y ) (Slepian-Wolf)) If X and Y are jointly Gaussian, then R Y (d) =R X Y (d) [A. D. Wyner and J. Ziv, The rate-distortion function for source coding with side information at the decoder, IEEE Transactions on Information Theory, Jan, 1976.]

8 Without Side Information EXAMPLE: SOURCE CODING WITH SIDE INFORMATION X, Y continuous valued RVs. Quantize x using 8-level quantizer. w = Q 8 (x). Encoder: Send index i of the region containing x { X i } RATE R s ENCODER DECODER { X i } R s =3bits/sample. ] Decoder: ˆx =argmin a R E [D(x, a) X Γ i r 0 r 1 r 2 r 3 r 4 r 5 r 6 r 7 With Side Information 0 Case 1: Encoder is the same. [ ] Decoder: ˆx =argmin a R E D(x, a) X Γ i,y = y Case 2: W c = {r 0,...,r 1 } C = {r 0,r 2,r 4,r 6 },rateofc is R c =2bits/sample. Encoder: Send index j of the coset containing w = Q 8 (x). RateisR = R s R c =1bit/sample. Decoder: Look for the most likely w i in coset j i.e., Y = y is the output of channel P (Y W ). Estimate: ˆx =argmin a R E [ ] D(x, a) X Γ i,y = y { X i } { Y i } ENCODER RATE R DECODER { X i }

9 Case 3: C = {r 0,r 4 },rateofc is R c =1bits/sample. Encoder: Send index j of the coset containing w = Q 8 (x). RateisR = R s R c =2 bit/sample. Rate of the channel code reduced, distance of the channel code C increased. ENCODER DECODER X Find index of the Active Codeword Compute index of the coset havingactive Codeword U Find codeword closest to Y in coset U Extimate X X Canwedobetter? Block encoding, vector quantizers or scalar quantizers with memory. design a channel code C for the fictitious channel P (Y W ) such that C partitions the set of codewords W c into disjoint cosets. R c I(Y ; W ). (From channel coding theorem) R s I(X; W ). (From rate distortion theorem) R = R s R c (I(X; W ) I(Y ; W )) [S. S. Pradhan, Distributed Source Coding Using Syndromes (DISCUS). PhD thesis, University of California, Berkeley, Y 2001.]

10 THEOREM 1: PROOF OF CONVERSE [T. M. Cover and J. A. Thomas, Elements of Information Theory. Wiley Series, 1991.] Lemma 2 (Convexity): R Y (d) is a non-increasing convex function in d. Proof: R Y (d) monotonic since the domain in (1) increases with d. For distortions d 1,d 2,letf 1,W 1 and f 2,W 2 be the parameters that achieve the minima in (1). Let Q = { 1 with prob λ 2 with prob 1 λ Let W =(W Q,Q) d=e[d(x, ˆX)] = λe[d(x, f 1 (Y,W 1 ))] + (1 λ)e[d(x, f 2 (Y,W 2 ))] = λd 1 +(1 λ)d 2 I(X; W ) I(Y ; W )=H(X) H(X W Q,Q) H(Y )+H(Y W Q,Q) = H(X) λh(x W 1 ) (1 λ)h(x W 2 )+ ( ) ( ) = λ I(X; W 1 ) I(Y ; W 1 ) +(1 λ) I(X; W 2 ) I(Y ; W 2 ) R Y (d) =min(i(x; U) I(Y ; U)) I(X; W ) I(Y ; W ) = λr Y (d 1 )+(1 λ)r Y (d 2 )

11 Converse Theorem: Suppose f n : X n {1,...,2 nr } and g n : Y n {1,...,2 nr } ˆX n such that E[d(X n,g n (Y n,f n (X n )))] d, then R R Y (d). Proof: Let T = f n (X n ) and let W i =(T,Y i 1,Y n i+1 ). nr H(T ) H(T Y n ) I(X n ; T Y n )= n i=1 I(X i ; T Y n,x i 1 ) = i (H(X i Y n,x i 1 ) H(X i T,Y n,x i 1 )) = i (H(X i Y i ) H(X i T,Y n,x i 1 )) i (H(X i Y i ) H(X i T,Y n )) = i (H(X i Y i ) H(X i W i,y i )) = i I(X i ; W i Y i )= i (H(W i Y i ) H(W i X i,y i )) = i (H(W i Y i ) H(W i X i )) = i (H(W i ) H(W i X i ) (H(W i ) H(W i Y i ))) = i (I(X i ; W i ) I(Y i ; W i )) i R Y (E[d(X i,g ni (Y i,w i ))]) = n 1 n R Y (E[d(X i,g ni (Y i,w i ))]) nr Y (E[ 1 n d(x i,g ni (Y i,w i ))]) = nr Y (d) i i (where ˆX n = g n (Y n,f n (X n )) = g n (Y i,w i ), ˆX i = g ni (Y i,w i )) R R Y (d)

12 DUALITY BETWEEN CHANNEL CAPACITY AND RATE DISTORTION Channel Coding Theorem: All rates below capacityc are achievable. Specifically, ɛ > 0 and rate R<C, there exists a sequence of (2 nr,n) codes with P e (n) 0. Conversely, any sequence of (2 nr,n) codes with P e (n) 0 must have R C. Rate Distortion Theorem: All rates above rate distortion function R(D) are achievable. Specifically, ɛ >0 and rate R>R(D), there exists a sequence of (2 nr,n) rate distortion codes with average distortion D. Conversely, any sequence of (2 nr,n) rate distortion codes with average distortion D must have R R(D).

13 Proof of Achievability of CCT Fix p(x). Let R < C = max p(x) I(X; Y ) Generate a (2 nr,n) code at random p(x). Encoder: If message is w {1,...,2 nr }, send codeword X n (w). Decoder: Receive Y n and decode ŵ if a unique ŵ such that (X n (ŵ),y n ) A (n) ɛ. Prob of error calculation E1: there is no X n (ŵ that is jointly typical with Y n. This prob. is very small, say P E1 <ɛ. E2: there is more than codeword that is jointly typical with Y n. This prob is P E2 2 n(i(x;y ) 3ɛ) 2 nr. P E2 0 if R<I(X; Y ) 3ɛ. Proof of Achievability of RDT Fix p(ˆx x). Let R > R(d) = min p(ˆx x):ed(x, ˆX) d I(X; ˆX) Generate a (2 nr,n) RD code at random p(ˆx). Encoder: If X n is the message, look for a w {1,...,2 nr } such that (X n, ˆX n (w)) A (n) d,ɛ. If there is no such w send w =1, else send the smallest w. Decoder: Estimate is ˆX n (w). Average distortion calculation E1: If there is at least one w such that ( ˆX n (w),x n ) A (n) d,ɛ, then the average distortion is D1 d + ɛ. E2: If there is no such w, then D2 P e d max P e 0 if R>I(X; ˆX)+3ɛ.

14 STRONG TYPICALITY AND MARKOV LEMMA A sequence x n X n is ɛ-strongly typical if: (1) a X, 1 n N(a xn ) p(a) < ɛ X, (2)N(a xn )=0if p(a) =0(where N(a x n ) is the number of occurrences of a in x n ). A (n) ɛ is the set of ɛ-strongly typical sequences x n. A pair (x n,y n ) is ɛ-strongly typical if: (1) a X,b Y, 1 N(a, n b xn,y n ) p(a, b) < ɛ X Y, (2) N(a, b x n,y n )=0if p(a, b) =0(where N(a, b x n,y n ) is the number of occurrences of the pair (a, b) in (x n,y n )). A (n) ɛ (X, Y ) is the set of ɛ-strongly typical sequence pairs (x n,y n ). Lemma 3: Pr(A (n) ɛ ) 1 as n. Lemma 4: Let Y 1,...,Y n be drawn p(y). Forx n A (n) ɛ,wehave 2 n(i(x;y )+ɛ 1) Pr((x n,y n ) A (n) ɛ ) 2 n(i(x;y ) ɛ 1) A (n) ɛ (X, Y, Z) is the set of ɛ-strongly typical sequence triplets (x n,y n,z n ). If (x n,y n,z n ) A (n) ɛ (x n,y n ) A (n) ɛ and (y n,z n ) A (n) ɛ

15 Converse is not necessarily true. Markov Lemma 5: If X Y Z, i.e., p(x, y, z) =p(x, y)p(z y). If for a given (y n,z n ) A (n) ɛ, X n is drawn i p(x i y i ), then Pr{(X n,y n,z n ) A (n) ɛ (X, Y, Z)} > 1 ɛ for n sufficiently large.

16 PROOF OF ACHIEVABILITY Fix p(w x) and f(w, y) that achieves equality in (1) for Ed(X, ˆX) d. Calculate p(w) = x p(w x)p(x). Suppose R 2 >R Y (d). Codebook: Let R 1 = I(W ; X)+ɛ. Generate 2 nr1 codewords W n (s) n i=1 p(w i), s {1,...,2 nr 1 }. Let R 2 = I(X; W ) I(Y ; W )+5ɛ. Randomly assign s {1,...,2 nr 1 } to one of 2 nr2 bins. Encoder: If message is X n,lookforassuch that (X n,w n (s)) A (n) ɛ. If there is no such s, sets =1, else choose the smallest s. Send index of bin i containing s. Decoder: Look for a s in bin i such that (W n (s),y n ) A (n) ɛ. If there is a unique such s, then ˆX n is found by ˆX i = f(w i,y i ),else ˆX n =ˆx n arbitrary. Probability of error and average distortion analysis E 1 : (X n,y n ) / A (n) ɛ. P E1 <ɛ, D1 <ɛd max E 2 : X n is typical, but there is no s such that (X n,w n (s)) A (n) ɛ. P E2 0 if R 1 >I(X; W ) (Proof of the rate distortion theorem)

17 E 3 : (X n,w n (s)) A (n) ɛ but (W n (s),y n ) / A (n) ɛ. P E3 is small (Markov Lemma). E 4 : There is another s in bin i such that (W n (s ),Y n ) A (n) ɛ. P E4 2 n(r1 R2) 2 n(i(y ;W ) 3ɛ). (Proof of the channel capacity theorem) P E4 0 since R1 R2 <I(Y ; W ) 3ɛ. If s is decoded correctly, then (X n,w n (s)) A (n) ɛ. Since P E1 <ɛ, we can assume (X n,y n ) A (n) ɛ. Then, Markov Lemma (X n,w n (s),y n ) A (n) ɛ. empirical joint distribution is close to p(x, y)p(w x), (X n, ˆX n ) will have a joint distribution that is close to the distribution achieving distortion d. h(x, ˆx) d(x, ˆx) = p(x, y)p(w x) d(x, f(w, y)) d x,ˆx x y,w:f(w,y)=ˆx

18 PROOF OF ACHIEVABILITY Wyner-Ziv argument Lemma A Let X, Y, W, f be as before. For ɛ 0 > 0 and large n 0, there exists a code (n 0,M 0, 0 ) defined by (F 0 E,F 0 D) such that: 0 d + ɛ 0, 1 n H(T Y n 0 ) R 0 + ɛ 0 where T = F 0 E(X n 0 ), R 0 = I(X; W ) I(Y ; W ). Lemma B (Slepian-Wolf) Let (F 0 E,F 0 D) be a (n 0,M 0, 0 ) code. Then if R 0 = 1 H(T Y n 0 ) (where T = F 0 n E(X n 0 )), then for a sufficiently large n 1 and δ>0 there exists a code (n, M, ) such that n = n 0 n 1,M 2 n 1(n 0 R 0 +δ) 2 n(r0+δ), and 0 + δ Lemma A and Lemma B Prove achievability of RDT with Side Information.