Network coding for multicast relation to compression and generalization of Slepian-Wolf

Network coding for multicast relation to compression and generalization of Slepian-Wolf 1

Overview Review of Slepian-Wolf Distributed network compression Error exponents Source-channel separation issues Code construction for finite field multiple access networks 2

Distributed data compression Consider two correlated sources (X, Y ) p(x, y) that must be separately encoded for a user who wants to reconstruct both What information transmission rates from each source allow decoding with arbitrarily small probability of error? E.g. H(X 1 ) X 1. X 2. H(X2 X1 ) 3

Distributed source code A ((2 nr 1, 2 nr 2),n) distributed source code for joint source (X, Y ) consists of encoder maps and a decoder map f 1 : X n {1, 2,...,2 nr 1} n f 2 : Y {1, 2,...,2 nr 2} n g : {1, 2,...,2 nr 1} {1, 2,...,2 nr 2} X n Y - X n is mapped to f 1 (X n ) - Y n is mapped to f 2 (Y n ) -(R 1,R 2 ) is the rate pair of the code 4

Probability of error P e (n) =Pr{g(f1 (X n ),f 2 (Y n )) = ( X n,y n )}

Slepian-Wolf Definitions: A rate pair (R 1,R 2 ) is achievable if there exists a sequence of ((2 nr 1, 2 nr 2),n) distributed source codes with probability of error (n) 0as n P e achievable rate region - closure of the set of achievable rates Slepian-Wolf Theorem: 5

For the distributed source coding problem for source (X, Y ) drawn i.i.d. p(x, y), the achievable rate region is R 1 H(X Y ) R 2 H(Y X) R 1 + R 2 H(X, Y )

Proof of achievability Main idea: show that if the rate pair is in the Slepian-Wolf region, we can use a random binning encoding scheme with typical set decoding to obtain a probability of error that tends to zero Coding scheme: Source X assigns every sourceword x X n randomly among 2 nr 1 bins, and source Y independently assigns every y Y n randomly among 2 nr 2 bins Each sends the bin index corresponding to the message 6

the receiver decodes correctly if there is exactly one jointly typical sourceword pair corresponding to the received bin indexes, otherwise it declares an error

Random binning for single source compression An encoder that knows the typical set can compress a source X to H(X)+ɛ without loss, by employing separate codes for typical and atypical sequences Random binning is a way to compress a source X to H(X) + ɛ with asymptotically small probability of error without the encoder knowing the typical set, as well as the decoder knows the typical set the encoder maps each source sequence X n uniformly at random into one of 2 nr bins 7

the bin index, which is R bits long, forms the code the receiver decodes correctly if there is exactly one typical sequence corresponding to the received bin index

Error analysis An error occurs if: a) the transmitted sourceword is not typical, i.e. event (n) E 0 = {X / A ɛ } b) there exists another typical sourceword in the same bin, i.e.event E 1 = { x )= f(x), x (n) = X : f(x A ɛ } Use union of events bound: (n) P e = Pr(E 0 E 1 ) Pr(E 0 )+Pr(E 1 ) 8

Error analysis continued Pr(E 0 ) 0 by the Asymptotic Equipartition Property (AEP) Pr(E 1 ) = Pr{ x = x : f(x )= f(x), x x x x x (n) A ɛ (n) } (n) = A ɛ 2 nr x 2 nr 2 n(h(x)+ɛ) 0if R>H(X) x A ɛ Pr(f(x )= f(x)) 9

For sufficiently large n, Pr(E 0 ), Pr(E 1 ) < ɛ (n) P ɛ < 2ɛ

Jointly typical sequences (n) The set A ɛ of jointly typical sequences is the set of sequences (x, y) X n Y n with probability: 2 n(h(x)+ɛ) p X (x) 2 n(h(x) ɛ) 2 n(h(y )+ɛ) p Y (y) 2 n(h(y ) ɛ) 2 n(h(x,y )+ɛ) p X,Y (x, y) 2 n(h(x,y ) ɛ) for (X, Y) sequences of length n IID according to p X,Y (x, y) = i=1 p X,Y (x i,y i ) n Size of typical set: (n) 2 n(h(x,y )+ɛ) A ɛ 10

Proof: 1 = p(x, y) p(x, y) (n) A ɛ A ɛ (n) 2 n(h(x,y )+ɛ)

Conditionally typical sequences (n) The conditionally typical set A ɛ (X y) for a given typical y sequence is the set of x sequences that are jointly typical with the given y sequence. Size of conditionally typical set: (n) A ɛ (X y) 2 n(h(x Y )+ɛ) Proof: 11

For (x, y) A ( ɛ n) (X, Y ),. p(y) = 2 n(h(y )±ɛ). p(x, y) = 2 n(h(x,y )±ɛ) p(x y) = p(x, y) p(y). = 2 n(h(x Y )±2ɛ) Hence 1 p(x y) (n) x A ɛ (X y) (n) A ɛ 2 n(h(x Y )+2ɛ)

Proof of achievability error analysis Errors occur if: a) the transmitted sourcewords are not jointly typical, i.e. event (n) E 0 = {(X, Y ) / A ɛ } b) there exists another pair of jointly typical sourcewords in the same pair of bins, i.e. one or more of the following events E 1 = { x = X : f 1 (x )= f 1 (X), (x (n), Y) A ɛ } E 2 = { y )= f 2 (Y), (X, y (n) = Y : f2 (y ) A ɛ } E 12 = { (x, y ): x X, y Y,f 1 (x )= f 1 (X), f 2 (y )= f 2 (Y), (x, y ) A ( ɛ n) } 12

Use union of events bound: (n) P e = Pr(E 0 E 1 E 2 E 12 ) Pr(E 0 )+Pr(E 1 )+Pr(E 2 )+Pr(E 12 )

Error analysis continued Pr(E 0 ) 0 bythe AEP Pr(E 1 ) = Pr{ x x : f 1 (x )=f 1 (x), (x,y) (x, y) A (n) = (x,y) (x,y) 2 nr 1 2 x = x (x, y) A (n) ɛ ɛ } A (n) ɛ (X y) 2 nr 1 n(h(x Y )+2ɛ) 0ifR 1 >H(X Y ) Pr(f 1 (x )=f 1 (x)) 13

Similarly, Pr(E 2 ) 2 nr 2 2 n(h(y X)+2ɛ) 0if R 2 >H(Y X) Pr(E 12 ) 2 n(r 1+R 2 ) 2 n(h(x,y )+ɛ) 0if R 1 + R 2 >H(X, Y )

Error analysis continued Thus, if we are in the Slepian-Wolf rate region, for sufficiently large n, Pr(E 0 ), Pr(E 1 ), Pr(E 2 ), Pr(E 12 ) < ɛ (n) P ɛ < 4ɛ Since the average probability of error is less than 4ɛ, there exist at least one code (f 1,f 2,g ) with probability of error < 4ɛ. (n) Thus, there exists a sequence of codes with P ɛ 0. 14

Model for distributed network compression arbitrary directed graph with integer capacity links discrete memoryless source processes with integer bit rates randomized linear network coding over vectors of bits in F 2 coefficients of overall combination transmitted to receivers receivers perform minimum entropy or maximum a posteriori probability decoding 15

Distributed compression problem Consider two sources of bit rates r 1,r 2, whose output values in each unit time period are drawn i.i.d. from the same joint distribution Q linear network coding in F 2 over vectors of nr 1 and nr 2 bits from each source respectively Define 16

m 1 and m 2 the minimum cut capacities between the receiver and each source respectively m 3 the minimum cut capacity between the receiver and both sources L the maximum source-receiver path length

Theorem 1 The error probability at each receiver using minimum entropy or maximum a posteriori probability decoding is at most i 3 =1 p e i,where { ( p 1 e exp n min D(P X1 X 2 Q) X 1,X 2 ) } 1 + + m 1 (1 log L) H(X 1 X 2 ) +2 2r 1+r 2 log(n +1) n { ( p 2 e exp n min D(P X1 X 2 Q) X 1,X 2 ) } 1 + + m 2 (1 log L) H(X 2 X 1 ) +2 r 1+2r 2 log(n +1) n { ( p 3 e exp n min D(P X1 X 2 Q) X 1,X 2 ) } + m 3 (1 1 + log L) H(X 1 X 2 ) +2 2r 1+2r 2 log(n +1) n 17

Distributed compression Redundancy is removed or added in different parts of the network depending on available capacity Achieved without knowledge of source entropy rates at interior network nodes For the special case of a Slepian-Wolf source network consisting of a link from each source to the receiver, the network coding error exponents reduce to known error exponents for linear Slepian-Wolf coding [Csi82] 18

Proof outline Error probability 3 i =1 pi e,where p1 e is the probability of correctly decoding X 2 but not X 1, p2 e is the probability of correctly decoding X 1 but not X 2 p3 e is the probability of wrongly decoding X 1,X 2 Proof approach using method of types similar to that in [Csi82] Types P xi, joint types P xy are the empirical distributions of elements in vectors x i 19

Proof outline (cont d) Bound error probabilities by summing over sets of joint types {P X 1 = X 1, X 2 = X 2 } i = 1 X 1 X 1X 2 X 2 P i n = {P X X 1 = X 1, X 2 X 2 } i = 2 1 X 1X 2 X 2 {P X X 2 = X 2 } 1 X 1X X 2 X 2 1 = X 1, i = 3 nr i where Xi, X i F 2 20

sequences of each type T X1 X 2 = [ x y { } n(r ] F 1 +r 2 ) 2 Pxy = P X1 X 2 { n(r 1 +r 2 ) T X 1X 2 X 1 X 2 (xy) = [ x ỹ ] F 2 P x yxy = P X 1X 2X 1 X 2 }

Proof outline (cont d) Define P i,i =1, 2, the probability that distinct (x, y), ( x, y), where x = x, at the receiver P 3, the probability that (x, y), ( x, y), where x = x,y = y, are mapped to the same output at the receiver These probabilities can be calculated for a given network, or bounded in terms of block length n and network parameters 21

Proof outline (cont d) A link with 1 nonzero incoming signal carries the zero signal with probability 2 1 nc, where c is the link capacity this is equal to the probability that a pair of distinct input values are mapped to the same output on the link We can show by induction on the minimum cut capacities m i that ( P i 1 (1 1 ) mi ) L ( ) L mi 2 n 2 n 22

Proof outline (cont d) We substitute in cardinality bounds P 1 < (n +1) 22r 1 +r 2 n P 2 < (n +1) 2r 1 +2r 2 n P 3 < (n +1) 22r 1 +2r 2 n T X1 X 2 exp{nh(x 1 X 2 )} T X 1X 2 X 1 X (xy) exp{nh(x 1X 2 X 2 1 X 2 )} probability of source vector of type (x, y) T X1 X 2 Q n (xy) = exp{ n(d(p X1 X 2 Q)+ H(X 1 X 2 ))} 23

Proof outline (cont d) and the decoding conditions minimum entropy decoder: H(X 1X 2) H(X 1 X 2 ) maximum a posteriori probability decoder: D(P Q)+ H(X 1X 2) D(P X1 X 2 Q)+ H(X 1 X 2 ) X 1 X 2 to obtain the result 24

Conclusions Distributed randomized network coding can achieve distributed compression of correlated sources Error exponents generalize results for linear Slepian Wolf coding Further work: investigation of non-uniform code distributions, other types of codes, and other decoding schemes 25

MIT OpenCourseWare http://ocw.mit.edu 6.441 Information Theory Spring 2010 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.