Lossy Distributed Source Coding

Size: px

Start display at page:

Download "Lossy Distributed Source Coding"

Cory Curtis
5 years ago
Views:

1 Lossy Distributed Source Coding John MacLaren Walsh, Ph.D. Multiterminal Information Theory, Spring Quarter, 202 Lossy Distributed Source Coding Problem X X 2 S {,...,2 R } S 2 {,...,2 R2 } Ẑ Ẑ 2 E d(z,n, Ẑ,n) D E d(z 2,n, Ẑ2,n) D 2 X M S M {,...,2 RM } Ẑ L E d(z L,n, ẐL,n) D L Y (X,n,...,X M,n,Y n,z,n,...,z L,n ) iid n p X,...,X M,Y,Z,...,Z L Figure : The general multiterminal lossy source coding problem. The general multiterminal lossy source coding problem depicted in Fig. is not solved, however we have inner and outer bounds. As we will discuss in section.3, these inner and outer bounds do not match on even very simple variants of this structure. Important special cases of this problem include when L = M and Z m = X m (the lossy variant of Slepian-Wolf), as well as the case L = which is known as the CEO problem.. Berger Tung Inner Bound The Berger Tung Inner bound uses the familiar random binning technique. A series of conditional distributions for a collection of message random variables U,..., U M and reconstructions Ẑ,..., ẐL are created according to the Markov chain relationships (X \m, U \m, Y, Z :L ) X m U m and (X :M, Z :L ) (Y, U :M ) Ẑ :L that also obey the distortion constraints E[d m (Z l, Ẑl)] D l, l {,..., L} () Encoder m generates a 2 ˆR m dimensional matrix with elements IID according to p Um, where the ith row is denoted by U m(i). Each codeword is randomly assigned to one of 2 Rm bins B m (j), j {,..., 2 Rm }. The bins and codebooks are shared with the decoder. Encoder m operates by selecting a codeword that is jointly typical with its observations: (U m(i), X m) Tɛ (U m, X m ), then transmitting the bin index j m of the bin to which this codeword was assigned. If there was no such codeword, it simply transmits the first bin index. The decoder operates by looking in the bins for (i,..., i M ) B (j ) B M (j M ) giving a collection of codewords jointly typical with each other and the side information: (U (i ),..., U M (i M ), Y ) Tɛ (U,..., U M, Y ). If it can find no such collection of codewords, it selects i m = for all m {,..., M}. The following error events are useful in analyzing the rates necessary to achieve the desired distortions: E m: There does not exists i m {,..., 2 ˆR m } such that (U m(i m ), X m) T ɛ (U m, X m )

2 E 2 : For every m {,..., M} there exists a i m {,..., 2 ˆR m } such that (U m(i m ), X m) Tɛ (U m, X m ) however (U :M (i), X :M, Y, Z :L ) / T ɛ (U :M, X :M, Y, Z :L ). EA 3 : For every m {,..., M} there exists a i m {,..., 2 ˆR m } such that (U m(i m ), X m) Tɛ (U m, X m ) and (U :M (i), X :M, Y, Z :L ) T ɛ (U :M, X :M, Y, Z :L ). However, for the set A {,..., M}, (U A(i A ), U A c(i A c), Y ) Tɛ (U A, U A c, Y ) for some i m i m, m A with i m B m (j m ) for each m A. The first error event involves the same style of analysis as in the rate distortion proof: P[E m] = 2 ˆRm i= ( ) 2 ˆRm P[(U m(i), X m) / Tɛ (U m, X m )] = P[(U m(i), X m) Tɛ (U m, X m )] ( ( ( ɛ)2 (I(Um;Xm)+ɛc) 2 ˆRm ) ( ɛ) + exp 2 ˆR m 2 (I(Xm;Um)+ɛc)) (3) (2) From this we observe that the probability of error will be arbitrarily small as provided that ˆR i > I(X i ; U i ) (4) Error E 2 has a probability that is arbitrarily small as via an extension of the Markov Lemma. Finally, turning our attention to the third collection of error events, we have [ ] P[EA] 3 = P (U A(i A), U A c(i A c), Y ) Tɛ (U A, U A c, Y ), i m B m (j m ) m A (U :M (i), X :M, Y ) Tɛ i A,i m im m A [ ] = P (U A(i A), U A c(i A c), Y ) T ɛ (U A, U A c, Y ) (U :M (i), X :M, Y ) Tɛ P [i m B m (j m ) m A] i A,i m im m A 2 Rm (2 ˆR m ) 2 i A,i m im m A ( m A 2 (I(U A;(Y,U Ac )) ɛc) ˆR m R m I(U A ; (Y, U A c)) + ɛc This can be made arbitrarily small as if i A 2 Rm 2 (I(U A;(Y,U Ac )) ɛc) m A m A ) ˆR i R i I(Y ; U A U A c) < 0 (6) Putting the two sets of inequalities together, we find that the requirements on the rates are R i > I(X A ; U A Y, U A c) (7) i A When its union over all possible choices of auxiliary variables U :M obeying the constraints is taken, this region may not be convex, but any two achievable strategies can be time shared to get a convex combination in (R, D) space, hence can take a convex hull. One way to show this convex hull in the math is to introduce a time sharing variable Q, as shown in the text..2 Berger Tung Outer Bound The Berger Tung outer bound involves the same rate and distortion expressions as the Berger Tung inner bound, but uses changes the message Markov Chain conditions: (X \m, Y, Z :L ) X m U m. (ote that the other messages U \m are absent in the condition.).3 Tightness & Other Bounds As noted in the text, both the Berger Tung Inner and the Berger Tung Outer bounds have cases where they are not tight. There is a gap between the Berger Tung inner bound and the Berger Tung Outer Bound for the scalar Gaussian squared error CEO problem, and the Berger Tung inner bound was shown to give the rate distortion region in this case (Oohama 05 and Prabhakaran, Tse, Ramachandran 04). As discussed in 2.3, the Berger Tung inner bound is also tight for two Gaussian source and squared error distortions. Wagner and Anatharam have given a tighter outer bound than the Berger Tung Outer bound (2008). These problems are an area of active work in the community. (5) 2

3 2 Multiple Descriptions Problem S {,...,2 R } ˆX E[d(X n, ˆX,n )] D S X ˆX 0 S 2 E[d(X n, ˆX 0,n )] D 0 S 2 {,...,2 R2 } ˆX 2 E[d(X n, ˆX 2,n )] D 2 Figure 2: The still generally unsolved multiple descriptions problem. Surprisingly, this problem is not solved. We presented a pair of achievable regions and discussed when one of them was tight. 2. El Gamal Cover Region The El Gamal cover region splits each of the two descriptions S, S 2 up into two parts S = (S 0, S ) and S 2 = (S 20, S 22 ). The parts S 0 and S 20 are discarded by the decoders receiving only one description, and are concatenated into S 0 = (S 0, S 20 ) by the decoder receiving both descriptions. Codebook Generation: Generate a 2 R matrix with elements IID according to p X, denote the ith row of this by X (i). Similarly, generate a 2 R2 matrix with elements IID according to p X2, denote the jth row of this by X 2 (j). For each (i, j) {,..., 2 R } {,..., 2 R22 } generate a 2 R0 matrix, with kth row X 0 (i, j, k), and with elements distributed according to the conditional distribution p X0 X,X 2 (x 0,n (i, j, k) x,n (i), x 2,n (j)) (8) Share these matrices with the encoder and decoder. Encoder: Select an i, j, k such that (X (i), X 2 (j), X 0 (i, j, k), X ) Tɛ (X, X 2, X 0, X). Split the bits in the index k up into two parts k = (k, k 2 ), and transmit the messages S = (i, k ) and S 2 = (j, k 2 ). Decoder: The decoder receiving only S = (i, k ) discards k and reproduces X (i). The decoder receiving only S 2 = (j, k 2 ) discards k 2 and reproduces X 2 (j). The decoder receiving both S, S 2 reproduces X 0 (i, j, k). Error events E x : X / Tɛ (X) E : for every i (X (i), X ) / T n ɛ (X, X) E 2 : for every j (X 2 (j), X ) / T n ɛ (X 2, X) E 2 : for every i, j (X (i), X 2 (j), X ) / T ɛ (X, X 2, X) E 0 : there exists i, j such that (X (i), X 2 (j), X ) Tɛ (X, X 2, X), but for every k {,..., 2 R0 }, (X (i), X 2 (j), X 0 (i, j, k), X ) / Tɛ (X, X 2, X 0, X) Properties of the typical set imply P[E x ] 0 as. A now standard proof bounding the probability that two independent sequences with matching marginals to a joint distribution wind up in the typical set shows that if R > I(X ; X) and if R 22 > I(X 2 ; X), P[E ] and P[E 2 ] go to 0 as. 3

4 The novel idea in the proof is the bounding of P[E 2 ] via a clever application of Chebyshev s inequality. In this vein, for each x X, define the random set G x = { (i, j) {,..., 2 R } {,..., 2 R22 } (X (i), X 2 (j), x) T ɛ (X, X 2, X) } (9) and denote, naturally, the cardinality of this set by G x oting that the event E 2 is equivalent to the event G X = 0, and applying the Chebyshev s inequality which implies by selecting ɛ = [ P X E[X] ɛ ] var(x) ɛ 2, (0) E[X] var(x) that P [ X E[X] E[X]] var(x) (E[X]) 2, we have P [E 2 E c x ] = P [ G X = 0 E c x ] P [ G X E[ G X ] E[ G X ]] var( G X ) (E[ G X ]) 2 () The mean and the variance can be bounded by viewing the cardinality as a sum of indicator functions. For x Tɛ E[ G x ] = i,j P[(X (i), X 2 (j), x) Tɛ (X, X 2, X)] = i,j (X), x,x 2 (x,x 2,x) T ɛ (X,X2,X) p X(i)(x )p X2(j)(x 2 ) (2) ( ɛ)2 (R+R22 H(X) H(X2)+H(X,X2 X) ɛc) (3) while the variance can be bounded via var( G x ) = [ (X (i), X 2 (j), x) Tɛ (X, X 2, X), (X (i ), X 2 (j ), x) Tɛ (X, X 2, X) ] = i,j,i,j P i,j,i,j P,j j + + = i,j,i,j P + [ (X (i), X 2 (j), x) Tɛ (X, X 2, X) ] P [ (X (i ), X 2 (j ), x) Tɛ (X, X 2, X) ] (4) P [ (X (i), X 2 (j), x) Tɛ (X, X 2, X) ] P [ (X (i ), X 2 (j ), x) Tɛ (X, X 2, X) ] P [ (X (i), X 2 (j), x) Tɛ (X, X 2, X), (X (i ), X 2 (j), x) Tɛ (X, X 2, X) ] P [ (X (i), X 2 (j), x) Tɛ (X, X 2, X), (X (i), X 2 (j ), x) Tɛ (X, X 2, X) ] [ (X (i), X 2 (j), x) Tɛ (X, X 2, X) ] P [ (X (i ), X 2 (j ), x) Tɛ (X, X 2, X) ] (5) P [ (X (i), X 2 (j), x) T ɛ P [ (X (i), X 2 (j), x) T ɛ (X, X 2, X) ] P [ (X (i ), X 2 (j), x) Tɛ (X, X 2, X) (X 2 (j), x) Tɛ n (X 2, X) ] (X, X 2, X) ] P [ (X (i), X 2 (j ), x) Tɛ (X, X 2, X) (X (i), x) Tɛ (X, X) ] P [ (X (i), X 2 (j), x) Tɛ (X, X 2, X) ] P [ (X (i), X 2 (j ), x) Tɛ (X, X 2, X) ] P [ (X (i), X 2 (j), x) Tɛ (X, X 2, X) ] P [ (X (i ), X 2 (j), x) Tɛ (X, X 2, X) ] ( [ P (X (i), X 2 (j), x) Tɛ (X, X 2, X) ]) 2 i,j 2 ( H(X) H(X2)+H(X,X2 X)+ɛc) 2 ( H(X) H(X2 X)+H(X,X2 X)+ɛc2) + 2 ( H(X) H(X2)+H(X,X2 X)+ɛc) 2 ( H(X X) H(X2)+H(X,X2 X) 2 (R+R22) ( (2 R ) + (2 R22 ) + ) ( ɛ) 2 2 2( H(X) H(X2)+H(X,X2 X) ɛc) (7) (6) (8) 4

5 2 2( H(X) H(X2)+H(X,X2 X)+ɛc) 2 I(X2;X) + ) ( ɛ) 2 2 4ɛc (2 R + 2 R22 2 (R+R22) ) 2 I(X;X) ( 2 2(R+R22 H(X) H(X2)+H(X,X2 X)+ɛc) 2 R22 ( 2 R )2 I(X2;X) + 2 R ( 2 R22 )2 I(X;X) ) ( ɛ) 2 2 4ɛc (2 R + 2 R22 2 (R+R22) ) (9) (20) Substituting these bounds (3) and (9) into our bound (), we have P [ G x E[ G x ] E[ G x ]] ( ɛ) 2 2 4cɛ ( 2 R22 ( 2 R )2 I(X2;X) + 2 R ( 2 R22 )2 I(X;X)) 2 2ɛc (2 R + 2 R22 2 (R+R22) ) (2) Which approaches 0 as for a fixed, but small enough ɛ, since R > I(X ; X) and R 22 > I(X 2 ; X). Then selecting R + R 22 > H(X ) + H(X 2 ) H(X, X 2 X) (22) in (3) will ensure that the associated E[ G x ] will grow large and not shrink to zero as. Finally considering the last error event P[E 0 ] P [ k {,..., 2 R0 }(X (i), X 2 (j), X 0 (i, j, k), X ) / Tɛ (X, X 2, X 0, X) (X (i), X 2 (j), X ) Tɛ (X, X 2, X) ] ( [ P (X (i), X 2 (j), X 0 (i, j, k), X ) Tɛ (X, X 2, X 0, X) (X (i), X 2 (j), X ) Tɛ (X, X 2, X) ]) (23) k ( ( ɛ)2 (I(X0;X X,X2)+ɛc)) 2 R 0 (24) ( ( ɛ) + exp 2 R0 2 (I(X0;X X,X2)+ɛc)) (25) which can be made arbitrarily small if R 0 > I(X 0 ; X X, X 2 ) (26) Putting the requirements (2), and (25) together we have R > I(X ; X) (27) R 22 > I(X 2 ; X) (28) R + R 22 > H(X ) + H(X 2 ) H(X, X 2 X) (29) R 0 > I(X 0 ; X X, X 2 ) (30) which implies for the rates R = R 0 + R and R 2 = R 02 + R 22 the region R I(X ; X) (3) R 2 I(X 2 ; X) (32) R + R 2 I(X ; X 2 ) + I((X 0, X, X 2 ); X) (33) where the last inequality was derived through H(X ) + H(X 2 ) H(X, X 2 X) + I(X 0 ; X X, X 2 ) = I(X ; X 2 ) + I((X 0, X, X 2 ); X) (34) Adding a time sharing variable Q to convexify the region, we have R I(X ; X Q) (35) R 2 I(X 2 ; X Q) (36) R + R 2 I(X ; X 2 Q) + I((X 0, X, X 2 ); X Q) (37) 5

6 2.2 Cases where EGC is Tight o Excess Rate: Regarding the decoder obtaining both descriptions, it is clear that is can not require less rate than a decoder receiving a single message requiring this distortion. When the rates are selected so that this bound is tight, that is for those rates such that R + R 2 = R(D 0 ), the (Ahlswede) Gaussian Squared Error. For X n scalar Gaussian distributed and d the squared error, the EGC region is tight. 2.3 Zhang Berger Region The Zhang Berger region grows the El Gamal cover region by allowing a certain part of each of the two descriptions to be identical, with the remaining descriptions generated conditionally on this part. The achievability construction is as follows Codebook Generation: Generate a 2 Rc matrix with elements IID according to p U, and denote the ith row of this matrix by U (i). For each i {,..., 2 Rc } generate a 2 R matrix whose rows X (i, j) are IID and distributed according to p X U (x,n (i, j) u n (i)) (38) and a 2 R22 matrix whose rows X 2 (i, k) are IID and distributed according to p X2 U (x 2,n (i, k) u n (i)) (39) For each (i, j, k) generate a 2 R0 matrix whose rows X 0 (i, j, k, l) are IID according to p X0 X,X 2,U (x 0,n (i, j, k, l) x,n (i, j), x 2,n (i, k), u n (i)) (40) Share the codebook matrices with the encoders and decoders. Encoder: Find (i, j, k, l) {,..., 2 Rc } {,..., 2 R } {,..., 2 R22 } {,..., 2 R0 } such that (U (i), X (i, j), X 2 (i, k), X 0 (i, j, k, l), X ) Tɛ (U, X, X 2, X 0, X). The bits of the index l is broken up into two parts l = (l, l 2 ), and the two descriptions that are sent are S = (i, j, l ) and S 2 = (i, k, l 2 ). Decoder: The decoder receiving only S discards l and produces X (i, j). The decoder receiving only S 2 discards l 2 and produces X 2 (i, k). The decoder receiving both descriptions produces X 0 (i, j, k, l). While we did not go into the details of the proof, we noted that this rate region obtains some extra points that are not in the El Gamal Cover region for the binary source Hamming distortion case. 6

7 3 Successive Refinements S {,...,2 R } X S S 2 ˆX ˆX 0 E[d(X n, ˆX,n )] D E[d(X n, ˆX 0,n )] D 0 S 2 {,...,2 R2 } Figure 3: The problem of successive refinement of information. In successive refinements, we neglect one of the two decoders receiving an individual description in the multiple descriptions problem, and we obtain the problem shown in Fig. 3. The El Gamal Cover inner bound for the multiple descriptions problem can be specialized to this case by selecting X 2 = 0. The region so obtained is 3. EGC Tightness: Converse Proof R I(X ; X) (4) R + R 2 I((X, X 0 ); X) (42) Successive refinements represent a special case of the more general multiple descriptions problem for which the El Gamal Cover region is tight. We presented the proof in class. Suppose that R and R 2 are the rates of the successive descriptions achieving distortions D and D 0, respectively. Following the normal rate distortion converse R H( ˆX ) H( ˆX ) H( ˆX X ) = I( ˆX ; X ) = H(X ) H(X ˆX ) = H(X n ) H(X n ˆX, X n ) H(X n ) H(X n ˆX,n ) = I(X n ; ˆX,n ) (43) (R + R 2 ) H( ˆX, ˆX 0 ) H( ˆX, ˆX 0 ) H( ˆX, ˆX 0 X ) = I(X ; ( ˆX 0, ˆX )) = H(X ) H(X ˆX 0, ˆX ) H(X n ) H(X n ˆX 0, ˆX, X n ) H(X n ) H(X n ˆX 0,n, ˆX,n ) = I(X n ; ( ˆX 0,n, ˆX,n )) (44) Rearranging, we obtain the following inequalities R R + R 2 D D 0 I(X n ; ˆX,n ) (45) I(X n ; ( ˆX 0,n, ˆX,n )) (46) E[d(X n, ˆX,n )] (47) E[d(X n, ˆX 0 )] (48) showing that the vector (R, R 2, D, D 2 ) is elementwise superior to a point in the convex hull of the rate distortion region given by (40) and (4). 3.2 Successive Refinability of a Source and Distortion Measure Regarding each of the two decoders individually in Figure 3, it is quite clear that they can not require less rate than a decoder which must only reproduce the source at their associated distortion. Hence, if R(D) is the rate distortion function for the 7

8 source X and the distortion metric d(, ), the rates in the successive refinements problem must obey the bound R R(D ) and R + R 2 R(D 0 ). A source is successively refineable if these bounds are achievable. That is, for a successively refineable source, the minimum average amount of extra information per symbol necessary to refine a description of the source at distortion D to a description of the source at D 0 is R(D 0 ) R(D ). A source is successively refineable if and only if for every D 0 D there is a conditional distribution p ˆX0, ˆX X = p ˆX0 X p ˆX ˆX 0 (i.e. obeying the Markov chain X ˆX 0 ˆX with p X Xand p X0 X attaining the rate distortion bounds: I(X; ˆX ) = R(D ) and I(X; ˆX 0 ) = R(D 0 ) 8

Network Coding on Directed Acyclic Graphs

Network Coding on Directed Acyclic Graphs John MacLaren Walsh, Ph.D. Multiterminal Information Theory, Spring Quarter, 0 Reference These notes are directly derived from Chapter of R. W. Yeung s Information