Graph Coloring and Conditional Graph Entropy

Transcription

1 Graph Coloring and Conditional Graph Entropy Vishal Doshi, Devavrat Shah, Muriel Médard, Sidharth Jaggi Laboratory for Information and Decision Systems Massachusetts Institute of Technology Cambridge, MA 0239 {vdoshi, devavrat, medard, mit.edu Abstract We consider the remote computation of a function of two sources where one is receiver side information. Specifically, given side information, we wish to compute f(, ) based on information transmitted by over a noise-less channel. The goal is to characterize the minimal rate at which must transmit information to enable the computation of the function f. Recently, Orlitsky and Roche (200) established that the conditional graph entropy of the characteristic graph of the function is a solution to this problem. Their achievability scheme does not separate functional coding from the well understood distributed source coding problem. In this paper, we seek a separation between the functional coding and the coding for correlation. In other words, we want to preprocess (with respect to f), and then transmit the preprocessed data using a standard Slepian-Wolf coding scheme at the Orlitsky-Roche rate. We establish that a minimum (conditional) entropy coloring of the product of characteristic graphs is asymptotically equal to the conditional graph entropy. This can be seen as a generalization of a result of Körner (973) which shows that the minimum entropy coloring of product graphs is asymptotically equal to the graph entropy. Thus, graph coloring provides a modular technique to achieve coding gains when the receiver is interested in decoding some function of the source. I. INTRODUCTION Consider a network of sensors relaying correlated measurements to a central receiver with measurements of its own. Usually, the receiver is interested in computing a function of the received data. Hence, to the receiver the bulk of the data received from the various sensors is irrelevant. Contemporary data compression techniques (see [], [2]) compress the data utilizing only the correlation of the measurements and do not take the receiver s function of interest into account. To increase the efficiency for such a system, we seek to implement better compression rates by exploiting the end-user s intentions (the computation of the function f of the data received). Consider the following example: suppose there is a source (uniformly) producing an n-bit integer. The receiver also produces an n-bit integer and is only interested in the parity of the sum. Assuming independence of the integer data, contemporary techniques would have the source transmit all n bits, when it is clear that only the final bit is necessary. The use of such compression techniques enables possibly large gains over contemporary methods depending upon the complexity of the function. As a first step towards the implementation of methods like this, we consider the case (as in the example) where there is a single source, denoted by the random variable with support. The information available at the receiver is denoted by ENCODER DECODER f(,) Fig.. is encoded such that f(, ) is recovered based on the coded information and side information. the random variable with support. Both and are discrete memoryless sources. The receiver wishes to compute a function f : Z, where Z is some finite set. The goal here is to minimize the rate at which transmits information to the receiver so as to enable computation of f at the receiver. Figure illustrates the question of interest. A. Setup and Related Work This question has been very well studied in different contexts in the literature [3], [4], [5]. Among them, Orlitsky and Roche [5] recently characterized the minimum rate requirement as the conditional graph entropy of the characteristic graph. We introduce the necessary definitions and notation to explain these results as well as formally state our question of interest. Assume that the random variables and have some joint distribution p(x, y). The n-sequence of random variables, denoted by and are such that (, ) ={( i, i )} n i= are i.i.d. with each ( i, i ) drawn according to p(x, y). The receiver is interested in computing a function f : Z or f : n n Z n, its obvious extension. The primary goal is to determine the minimal rate for to transmit data to the receiver so that the receiver can compute the function f with vanishing probability of error. Wyner and Ziv [6] considered the side information rate distortion problem. amamoto [4] extended their rate distortion function to the case of functional compression. Their functions relied on auxilliary random variables. In a sense, this paper makes those variables explicit for the zero distortion case. Further, we describe them in such a way that our scheme builds upon a Slepian-Wolf coding scheme [7] layered with a preprocessing of the source data. Define the characteristic graph of with respect to, f, and p(x, y) is G =(V,E) with V = and E defined as follows: an edge (x,x 2 ) 2 is in E if there exists a y /06/$

2 such that p(x,y),p(x 2,y) > 0 and f(x,y) f(x 2,y). Effectively, G is the confusability graph from the perspective of the receiver with side information. This was first defined by Witsenhausen [8]. The graph entropy of a graph G with some distribution on its vertices was introduced by Korner [3] and is defined as: H G () = min I(W ; ), W Γ(G) where Γ(G) is the set of all independent sets of G. Here, W means that the joint distribution p(w, x) on Γ(G) is such that p(w, x) > 0 implies x w. To motivate the importance of graph entropy, consider the following: given a function g : Z, define deterministic random variable on = {} and set f(x, ) = g(x) for all x. Then, the graph entropy H G () is the minimal rate required to distinguish functional values of g [8]. Orlitsky and Roche [5] established that the conditional graph entropy, an extension of graph entropy, is the minimum required transmission rate from to receiver to compute function f(, ) with small probability of error. The conditional graph entropy is defined as H G ( )= min W Γ(G) W I(W ; ), A natural extension of this problem is the computation of rate-distortion region. amamoto characterized the ratedistortion function for the question posed here by extending the Wyner-Ziv side information result [6]. This was recently built upon by Feng, Effros and Savari [9]. Our work in this paper is closer (in spirit) to the result by Körner [3]. In essence, Körner showed that achieving the graph entropy is the same as coloring a large graph and sending the colors. Consider the OR-product graph of G, denoted as G n = (V n,e n ) where V n = V n = n and two vertices (x, x 2 ) are in E n if any component (x i,x 2i ) E. By coloring, we mean standard graph vertex coloring, i.e. a function c : V N of a graph G =(V,E) such that (x,x 2 ) E implies c(x ) c(x 2 ). The entropy of a coloring is given by the induced distribution on colors p(c(x)) = p(c (c(x))) where c (x) ={ x : c( x) =c(x)}. Next, we define ε-colorings of any graph G (for any ε>0). Let A be any subset such that p(a) ε. Define ˆp(x, y) =p(x, y) (x,y) A (where Q is the indicator function for condition Q). While this is not a true probability, our definition of characteristic graph of with respect to, f, and ˆp is valid. Denote this characteristic graph as Ĝ. Note that the set of edges of Ĝ is a subset of the set of edges of G. Finally, we say c is an ε-coloring of G if it is a valid colorings of Ĝ defined with respect to some high probability set A. The chromatic entropy of a graph is () =inf{h(c()) : c is an ε-coloring of G}. ε>0 A subset of vertices of a graph is an independent set if no two nodes in the subset are adjacent to each other in G. COLOR Distributed Source Coding ENCODE DECODE LOOKUP f(,) Fig. 2. Coloring based coding allows separation between functional coding and distributed source coding as shown above. We define it s natural extension conditional chromatic entropy as follows: ( ) = inf {H(c() ):c is an ε-coloring of G}. ε>0 Now, we state the Körner s result [3] as follows: n n Hχ G n() =H G(). The result requires ε-colorings and not the simple colorings because, as seen in [0], the ratio of the above quantities can be arbitrarily large. The implications of this result are that for large enough n, one can color a high probability subgraph of the product graph G n to achieve near optimum functional source coding when the function is only of one source. This effectively layers the solution to a problem of graph coloring followed by using standard coding techniques to transmit the color sequence. The Orlitsky and Roche s solution is an end-to-end solution. That is, it includes functional and distributed source coding. Their achievability scheme involves coding over the space of independent sets. It is a larger class than the space of colors because each coloring gives rise to disjoint independent sets. Like the implication of Körner s result, the coloring will allow for separation between functional coding and distributed source coding. Hence we want a result similar to that of Körner s result so that the problem can be decoupled without losing any coding gains asymptotically. B. Our Contribution As the main result of this paper, we show that the minimum conditional entropy coloring of the OR-product graph is asymptotically the same as the conditional chromatic number. Theorem : n n Hχ G n( ) =H G( ) This validates the idea of using graph coloring to achieve better compression ratios. The variable W and the distribution on it that achieves H G ( ) is the auxilliary random variable from the amamoto result. In the entropy sense, the induced variable, c(), converges to W. This method thus leads to a natural layering of the problem just as in Körner s case. Figure 2 illustrates the layering. Nevertheless, we need large enough n to achieve rates close to the conditional graph entropy. Consider the following 238

3 example from [5]. We have = = {, 2, 3}, p(x, y) = 6 for all x y, and f(x, y) =if x>yand 0 otherwise. Then, our graph G has only one edge: (, 3). Because of uniformity, we know ( )= 2 3 using the coloring c() = c(2) = and c(3) = 2. But H G ( ) 0.54 bits as calculated in [5]. This is worrisome because minimum entropy graph coloring has been shown to be NP complete []. Nevertheless, the problem is layered into two. The second part is that of distributed source coding which is well understood. There exist methods to achieve corner points of the Slepian-Wolf rate region (e.g. []), and there exist methods to achieve all other points on the rate region (e.g. [2]). Further, the first part, that of graph coloring, has very rich literature (e.g. [0], [2]) and heuristics exist (e.g. [3]) for this hard problem. II. PROOF OF MAIN RESULT We borrow some techniques that are present in Körner [3], Orlitsky and Roche [5]. First, we recall some useful known results. Then, we establish the claimed equality in Theorem. A. Preinaries We begin by listing the results that we will use in our proof. Definition (Typicality): We use the notion of ε-strong typicality defined in [4, p. 358]. For any n, and any sequence x of length n, define the empirical frequency of the symbol x in x as ν x (x) = {i : x i = x}. n Then a sequence x is ε-strongly typical for ε>0 if for all x with p(x) > 0, ν x (x) p(x) ε, and for all x with p(x) =0, ν x (x) =0. The set of all such ε-strongly typical sequences, called the ε-typical set will be denoted as Tε n (), ortε n when the variables are clear from context. A similar definition naturally extends for the case of joint variables. Lemma : Suppose (, ) is a sequence of n random variables drawn independently and according to the joint distribution p(x, y), which is the marginal of p(w, x, y). Let an n-sequence W be drawn independently according to its marginal, p(w). Suppose the joint distribution p(w, x, y) forms a Markov Chain, W (i.e. p(w, x, y) =p(w x)p(x, y)). Then, for all ε>0, there is a ε = k ε, where k depends only on the distribution p(w, x, y), such that for sufficiently large n, ) P [ / Tε n ] <ε, P [ / Tε n ] <ε, and P [(, ) / Tε n ] <ε, 2) For all x Tε n, P [(x, W) Tε n ] 2 n(i(w ;)+ε). 3) For all y Tε n, P [(y, W) Tε n ] 2 n(i(w ;) ε). 4) For all (w, x) Tε n, P [(w, ) T n ε (x, ) T n ε ] ε. Part follows from Lemma 3.6., parts 2 and 3 follow from Lemma 3.6.2, and part 4 follows from Lemma 4.8., all from [4]. COLOR Distributed Source Coding ENCODE DECODE LOOKUP ENCODE f(,) Fig. 3. We use Slepian-Wolf codes with sending at full rate to achieve the corner point R x = H(c() ). B. Lower bound To prove the lower bound, consider any n and the corresponding OR-product graph G n. Let the coloring c of G n be such that n( ) =H(c()). Such a coloring exists because the number of possible colorings for a graph with n vertices is a finite set. Using this coloring, we will show that there is a scheme that requires rate n Hχ Gn( ) and provides a way to compute the function at receiver with small probability of error. This will establish the achievability of rate n Hχ Gn( ). By Orlitsky and Roche [5], no scheme can achieve rate smaller than H G ( ). Lemma 2: inf n n Hχ G n( ) H G( ). Proof: Given n>0, let the coloring c, as above, denote one the coloring that achieves n( ) for the OR-product graph G n. For every color, σ and y, define g(σ, y) =f(x, y) where x is any element of c (σ) ={x : c(x) =σ} such that p(x, y) > 0. If no such x exists, g is undefined. Suppose x is the information at the source, and y is the information at the receiver. Without loss of generality, presume p(x, y) > 0. Suppose c(x) and y are available at the decoder. Then, we have an error if g(σ, y) f(x, y). Suppose x, c(x )=c(x), and p(x, y) > 0. Because x and x have the same color, we know (x, x ) / E n, and therefore for all y where p(x, y),p(x, y) > 0, including ours, we must have f(x, y) =f(x, y). Because f has the same value for all x with p(x, y) > 0, f(x,y)=g(σ, y). Thus, we have shown that the color of and are sufficient to determine the function f(, ). The Slepian-Wolf Theorem [7] states that for two sources, and, there exists a sequence of codes with vanishing probability of error as n grows without bound for the rates: R H( ), R 2 H( ), and R + R 2 H(, ). We apply that theorem to the source (c(), ). In our case, is side information. In other words, R 2 H( ). See Figure 3 for an illustration. Thus, we get the following per symbol rates are achievable for an encoding of where c is any ε-coloring of G n (with 239

4 ε>0). R n H(c() ). Thus, the minimal value of R is by definition n( ). Putting the above discussion together, we obtain the desired result: inf n n Hχ G n( ) H G( ) C. Upper Bound Next we show that, indeed, the information needed to recover c n () given is at most H G ( ). Lemma 3: inf n n Hχ G n( ) H G( ). Proof: Suppose ε, δ > 0. Suppose n is large enough so that the following holds: () Lemma applies with some ε <, (2) 2 nδ <ε, and (3) n>2+ 3 2ε. Let p(w, x, y) be the distribution that achieves H G ( ) with the Markov chain, W. Let marginals be denoted simply as p(w), p(x), p(y). For any integer M, we define an M-system (W,...,W M ) where each W i is drawn independently according to p(w) = n i= p(w i). Declare an error if (x, y) / Tε n. (Thus, we are in fact looking at colorings over Tε n, or ε -colorings.) This, by construction, happens with probability less than ε. Henceforth assume that (x, y) Tε n. Declare an error if there is no i such that (W i, x) Tε n. The probability for this event is P [(W i, x) / T n ε i] (a) i= (b) M P [(W i, x) / Tε n ] =( P [(W, x) Tε n ]) M ( 2 n(i(w ;)+ε)) M (c) (d) 2 M 2 n(i(w ;)+ε ) where (a) and (b) follow because the W i are i.i.d., (c) follows from Lemma, part (2), and (d) follows because for α [0, ], ( α) n 2 nα. Assuming M>2 n(i(w ;)+ε+δ), P [(W i, x) / Tε n i] 2 δn <ε, because n is large enough. Henceforth, fix an M-system (W,...,W M ) for some M>2 n(i(w ;)+ε+δ). Further, assume there is some i such that (W i, x) Tε n. For every x, let the smallest (or any) such i be denoted as ĉ(x). Note that ĉ is a valid coloring of the graph G n. Next, for each y, define the following: S(y) ={ĉ(x) :(x, y) Tε n }, Z(y) ={W i :(W i, y) Tε n }, s(y) = S(y), z(y) = Z(y). Using these definitions, s(y) = M i= i S(y), where we have used the fact that our coloring scheme ĉ is simply an assignment of the indices of the M-system. Thus, we know E[s()] = P [i S()]. i= Similarly, we get z(y) = M i= W i Z(y). Thus, E[z()] = = P [W i Z()] i= P [W i Z() and i S()] i= P [i S()]P [W i Z() i S()] i= We know that if i S(), there is some x such that ĉ(x) =i and (x, ) T n ε. For each such x, we must have (by definition of our coloring), (W i, x) T n ε. For each such x where (W i, x) T n ε, P [W i Z() i S()] = P [(W i, ) Tε n (x, ) Tε n ] ε by Lemma, part (4). Thus, we have E[z()] ( ε )E[s()]. This and Jensen s inequality imply that E[log s()] loge[s()] [ ] z() log E. ε Finally, using a Taylor series expansion, we know log ε ε 2 =2ε + 2 when 0 <ε <. Thus, Next, we compute E[log s()] log E[z()] + ε 2, () E[z()] = P [(W i, ) Tε n ] i= = M P [(W i, ) T n ε ] because the W i are i.i.d. Therefore, E[z()] M 2 n(i(w ; ) ε) (2) by Lemma, part (3). By the definition of S(y), we know that determining ĉ given = y requires at most log s(y) bits. Therefore, we have H(ĉ() ) E[log s()]. (3) 240

5 Putting it all together, we have (a) n( ) H(ĉ() ) (b) E[log s()] (c) log E[z()] + ε 2 (d) ( log M 2 n(i(w ; ) ε)) + ε 2 ( ) (e) =log 2 n(i(w ;) I(W ; )+2ε+δ) + + ε 2 (f) n(i(w ; ) I(W ; )+2ε + δ)++ε 2 where (a) follows by definition of the conditional chromatic entropy, (b) follows from inequality (3), (c) follows from inequality (), (d) follows from inequality (2), (e) follows by setting M = 2 n(i(w ;)+ε+δ), and (f) follows because log(α +) log(α)+ for α. We know for Markov chains W, I(W ; ) I(W ; )=I(W ; ). Thus, for our optimal distribution p(w, x, y), we have n( ) n(h G( )+2ε + δ)++ε 2 +ε 2 Because n > ε, n < ε. Thus, n Hχ G n( ) H G ( )+3ε + δ. Therefore, sup n n Hχ G n( ) H G( ). Combining the lower bound, Lemma 2, and the upper bound, Lemma 3, we have our result: n n Hχ G n( ) =H G( ). (4) III. CONCLUSION We have validated the idea of graph coloring to achieve all possible rates. As a consequence, it is possible to decouple the side information functional source coding problem into a problem of graph coloring followed by a coding scheme that removed the redundancy between the colors (like Slepian- Wolf). While the graph coloring problem is shown to be NP complete, this decoupling allows for the decomposition of a hard problem into one for which many heuristics and copious literature exist (graph coloring) and one that is solved (distributed source coding). We plan to show that this decoupling is possible in the more general case when is a separate source (and not side information). In this case, we will wish to code both and. This will generalize our current results where we assume the rate for is H( ). Further, we plan to consider a relaxation of the fidelity criterion. We will wish to see if amamoto s rate distortion function can be achieved with graph coloring, as well as considering the case where is not side information. Thus, we hope to have a unified solution that will allow for the distributed compression of functions of correlated sources. REFERENCES [] S. S. Pradhan and K. Ramchandran, Distributed source coding using syndromes (DISCUS): design and construction, IEEE Trans. Inform. Theory, vol. 49, no. 3, pp , Mar [2] T. P. Coleman, A. H. Lee, M. Médard, and M. Effros, Low-complexity approaches to Slepian-Wolf near-lossless distributed data compression, IEEE Trans. Inform. Theory, vol. 52, no. 8, pp , Aug [3] J. Körner, Coding of an information source having ambiguous alphabet and the entropy of graphs, 6th Prague Conference on Information Theory, 973, pp [4] H. amamoto, Wyner-ziv theory for a general function of the correlated sources, IEEE Trans. Inform. Theory, vol. 28, no. 5, pp , Sept [5] A. Orlitsky and J. R. Roche, Coding for computing, IEEE Trans. Inform. Theory, vol. 47, no. 3, pp , Mar [6] A. Wyner and J. Ziv, The rate-distortion function for source coding with side information at the decoder, IEEE Trans. Inform. Theory, vol. 22, no., pp. 0, Jan [7] D. Slepian and J. K. Wolf, Noiseless coding of correlated information sources, IEEE Trans. Inform. Theory, vol. 9, no. 4, pp , July 973. [8] H. S. Witsenhausen, The zero-error side information problem and chromatic numbers, IEEE Trans. Inform. Theory, vol. 22, no. 5, pp , Sept [9] H. Feng, M. Effros, and S. Savari, Functional source coding for networks with receiver side information, Allerton Conference on Communication, Control, and Computing, Sept [0] J. Cardinal, S. Fiorini, and G. Joret, Minimum entropy coloring, Lecture Notes on Computer Science, ser. International Symposium on Algorithms and Computation,. Deng and D. Du, Eds., vol Springer-Verlag, 2005, pp [] J. Cardinal, S. Fiorini, and G. V. Assche, On minimum entropy graph colorings, ISIT 2004, June July 2004, p. 43. [2] C. McDiarmid, Colourings of random graphs, in Graph Colourings, ser. Pitman Research Notes in Mathematics Series, R. Nelson and R. J. Wilson, Eds. Longman Scientific & Technical, 990, pp [3] G. Campers, O. Henkes, and J. P. Leclerq, Graph coloring heuristics: A survey, some new propositions and computational experiences on random and Leighton s graphs, in Operations research 87, G. K. Rand, Ed., 988, pp [4] T. M. Cover and J. A. Thomas, Elements of Information Theory. New ork: Wiley,