Inequalities for Entropies of Sets of Subsets of Random Variables

Transcription

1 Iequalities for Etropies of Sets of Subsets of Radom Variables Chao Tia AT&T Labs-Research Florham Par, NJ 0792, USA. Abstract Ha s iequality o the etropy rates of subsets of radom variables is a classic result i iformatio theory, which ofte fids its applicatio i multiuser iformatio theoretic problems. I this ote, we geeralize Ha s iequality to allow commo compoets amog the radom variables, or, i a equivalet maer, to replace the simple radom variables i Ha s iequality by subsets of radom variables. This additioal igrediet sigificatly complicates the matter ad the form of the resultat iequalities are rather differet from the origial Ha s iequality. Our proof oly relies o the sub-modularity property of the etropy fuctio ad the super-modularity property of the coditioal etropy fuctio. This ew set of iequalities also provides a ew li betwee Ha s iequality ad the -way sub-modularity iequality. I. INTRODUCTION Ha s iequality [1 o the etropy rates of subsets of radom variables is a classic result i iformatio theory. It essetially states that the average etropy rates of subsets decrease as the size of subset icreases [2 (p. 490). This iequality has foud applicatios i multi-user iformatio theoretic problems; e.g. [ [5. Ha s iequality, however, does ot aturally deal with commo compoets amog radom variables efficietly. For example, cosider two radom variables 1 V 1 = {W 1, W 12 } ad V 2 = {W 2, W 12 }, W 12 is the commo compoet betwee them. The a simple applicatio of Ha s iequality to the radom variables V 1 ad V 2, which i this case is equivalet to the o-egativity of mutual iformatio, gives H(V 1 ) + H(V 2 ) H(V 1, V 2 ). (1) This is ot a good lower boud for H(V 1 ) + H(V 2 ) for the radom variable uder cosideratio, as ca be see easily H(V 1 ) + H(V 2 ) H(V 1, V 2 ) + H(W 12 ). (2) I fact, (2) is simply the sub-modularity property of the etropy fuctio [6. The iequality (2) ca ideed be reached by applyig Ha s iequality i its coditioal form, yet a atural questio arises from this simple example: what is the form of the geeralized Ha s iequality which taes ito accout of the commo compoets? I this ote, we provide exactly such a ew set of iequalities. 1 Throughout this ote, { } is used to deote the collectios of radom variables, ad the resultig radom variable, such as V 1 ad V 2, is defied o the product of the alphabet of each of its elemets. The same questio ca be cosidered from aother perspective, that is: what is Ha s iequality whe the radom variables are i fact subsets of radom variables, istead of simple radom variables? I the above example, V 1 ad V 2 are themselves subsets of the set {W 1, W 2, W 12 }. Sice both perspectives may be of iterest, we will preset the mai theorem i two equivalet forms. The proof of this ew set of iequalities higes oly o the sub-modular property of the etropy fuctio, ad it relies o aother iequality previously discovered i [7, amely, the - way sub-modularity iequality. As a cosequece, this ew set of iequalities provides a coectio betwee Ha s iequality ad the -way sub-modularity iequality, as they are special cases of a wider spectrum of iequalities regardig subsets of radom variables. I additio to the well-ow Ha s iequalities, iequalities o the etropies of subsets of radom variables have bee cosidered i other wors, e.g. [8, [9. The ew iequalities we preset i this wor do ot appear to be icluded i these existig iequalities, though the itersectio-cover problem cosidered i those wor is ideed closely related to the proof idea of our mai result. This coectio is hardly surprisig, sice all these results are directly related to sub-modularity fuctios, which aturally appear i the itersectio-cover ad related coutig problems. The rest of this ote is orgaized as follows. I Sectio II, we first preset the mai results o the ew set of iequalities, ad the discuss its relatioship with Ha s iequality ad the -way sub-modular iequality. The proofs are give i Sectio III, ad Sectio IV gives a few cocludig remars. II. MAIN RESULT The set {1, 2,..., } is writte as I, ad its power set (i.e., the collectio of all its subsets) is writte as P(I ). Defie ˆP(I ) = P(I ) \ { }, i.e., ˆP(I ) is the collectio of all subsets of I excludig the empty set. For a set A, we write its cardiality as A. For simplicity, we state the results oly for discrete radom variables with fiite alphabets for which the etropy fuctios are always well-defied, however, they ca be exteded to cotiuous radom variables (or mixed radom variables) with well-defied differetial etropy fuctios. Theorem 1: Let {W A, A ˆP(I )} be a set of 2 1 discrete radom variables joitly distributed accordig to some

2 W 1 Fig. 1. W 1 W 12 W12 W2 W 2 W Subsets ad their relatio. probability mass fuctio o some fiite alphabets. Defie The h () = 1 ( ) A ˆP(I ): A = H(W A (j)), () W A (j) {W B : B A j} (4) h () 1 h () 2 h (). (5) We ca uderstad the set of radom variables {W B : i B}, i = 1, 2,...,, (6) as radom variables with certai commo compoets, or as subsets of radom variables istead of simple radom variables; see. Fig. 1 for a illustratio whe =. To mae this more explicit, it is beeficial to preset a alterative statemet of Theorem 1. Theorem 1 : Let {W i, i = 1, 2,..., N} be a set of radom variables joitly distributed accordig to some probability mass fuctio o some fiite alphabets. Let C i, i = 1, 2,..., be subsets of I N. Defie h () = 1 ( ) W A(j) W, the A ˆP(I ): A = {i 1,i 2,...,i j} A H(W A(j)) (7) (C i1 C i2 C ij ), (8) h () 1 h () 2 h (). (9) It is straightforward to verify that Theorem 1 ad Theorem 1 are equivalet. This result essetially provides the relatio amog the average etropy rates of the fixed cardiality itersectios of subsets of radom variables, however with special cosideratio of the specified commo parts. It should be oted that there is a layered structure i W A (j), which is geerated by taig itersectio of differet umber of subsets of a set. The followig results are complemetary to Theorem 1, i a similar maer as those to Ha s iequality [2 (pp ). Theorem 2: Let {W A, A ˆP(I )} be a set of 2 1 discrete radom variables joitly distributed accordig to some probability mass fuctio. Defie g () = 1 ( ) A ˆP(I ): A = W c A (j) = W I (1) \ W A (j). The H(W A (j) WA(j)), c (10) g () 1 g () 2 g (). (11) Combiig Theorem 1 ad Theorem 2, we have the followig theorem. Theorem : Let {W A, A ˆP(I )} be a set of 2 1 discrete radom variables joitly distributed accordig to some probability mass fuctio. Defie The f () = 1 ( ) A ˆP(I ): A = I(W A (j); WA(j)) c (12) f () 1 f () 2 f (). (1) As a special case to Theorem 1, we ca recover Ha s iequality o subsets, by lettig W A be a costat for all A > 1, for which W {i} ca be coveietly writte as W i. Propositio 4: Let {W i } be a set of discrete radom variables joitly distributed accordig to some probability mass fuctio. Defie h () = 1 ( ) H({W i : i A}). (14) The A ˆP(I ): A = h () 1 h () 2 h (). (15) As aother special case to Theorem 1, we ca recover the -way sub-modularity iequality [7, which is the followig propositio. Propositio 5: Let {W A, A ˆP(I )} be a set of 2 1 discrete radom variables joitly distributed accordig to some probability mass fuctio. The H({W B : i B}) H({W B : B j}). (16) The left had side of (16) is simply h () 1 ad the right had side is simply h (). For this reaso, it is temptig to state Propositio 5 as a corollary of Theorem 1. However, we refrai from doig so because Propositio 5 i fact plays a istrumetal role i our proof of Theorem 1. The proof of Propositio 5 ca be foud i [7. It should be oted that though Ha s iequality ad the -way sub-modularity iequality ca be recovered this way, this does ot mea Theorem 1 is stroger. Ideed the ew

3 [ 1 H(W 1, W 12, W 1, W 12 ) + H(W 2, W 12, W 2, W 12 ) + H(W, W 1, W 2, W 12 ) H(W 1, W 2, W 12, W 1, W 2, W 12 ) + H(W 12, W 12 ) + H(W 1, W, W 12, W 1, W 2, W 12 ) + H(W 1, W 12 ) 6 + H(W 2, W, W 12, W 1, W 2, W 12 ) + H(W 2, W 12 ) H(W 1, W 2, W, W 12, W 1, W 2, W 12 ) + H(W 12, W 1, W 2, W 12 ) + H(W 12 ) (17) [ 1 H(W 1, W 12, W 1, W 12 ) + H(W 2, W 12, W 2, W 12 ) + H(W, W 1, W 2, W 12 ) H(W 1, W 2, W 12, W 1, W 2, W 12 ) + H(W 1, W, W 12, W 1, W 2, W 12 ) + H(W 2, W, W 12, W 1, W 2, W 12 ) 6 1 H(W 1, W 2, W, W 12, W 1, W 2, W 12 ) (18) set of iequalities, or rather its geeral form o sub-modular fuctios (give i the ext sectio) ca be specialized to the -way sub-modularity iequality, however, sice the - way sub-modularity iequality is sufficiet to prove the ew set of iequalities, they are completely equivalet for geeral sub-modular fuctios. The ew set of iequalities o etropy fuctio ca be specialized to Ha s equality, yet the coditioal versio of Ha s iequality implies the sub-modularity of the etropy fuctio, ad thus the ew set of iequalities i its etropy form ad the coditioal versio of Ha s iequality are equivalet. I the laguage of classificatio of iformatio iequalities, these iequalities are all Shao type iequalities [10. Despite this relatioship, it is our hope that this ew set of iequalities may fid its applicatios i the future, just as Ha s iequality fids applicatios i [ [5 some 20 or 0 years after its discovery. Example: Cosider =, ad the radom variables i Theorem 1 are {W 1, W 2, W, W 12, W 1, W 2, W 12 }, for simplicity we have writte W {1,2,} as W 12, ad similarly for W {i,j} ad W {i} ; see Fig. 1. Theorem 1 the gives (17) o the top of this page. We ca also aively apply Ha s iequality o the three sets of radom variables, ad treat each of them as a radom variable, T 1 = {W 1, W 12, W 1, W 12 } T 2 = {W 2, W 12, W 2, W 12 } T = {W, W 1, W 2, W 12 }, which gives (18) o the top of this page. Note that whe all the radom variables are idepedet, (17) becomes equalities, however, (18) will ot give equalities i geeral. The form of (17) is ot obvious, particularly the coefficiets of the etropies are ot easy to determie a priori. III. PROOFS OF THE THEOREMS It is well ow that the etropy fuctio is sub-modular, thus the followig theorem regardig sub-modular fuctios suffices for the proof of Theorem 1. For otatioal simplicity, we reserve the calligraphic letters A, A ad B for o-empty subset of I, i.e., A ˆP(I ) (ad similarly for A ad B), ad will ot explicitly iclude this coditio from here o. Theorem 6: Let θ be a submodular fuctio o sets, i.e., for ay two sets C 1 ad C 2, Defie the θ(c 1 ) + θ(c 2 ) θ(c 1 C 2 ) + θ(c 1 C 2 ). (19) θ () 1 ( ) A: A = θ({b : B A j}), (20) θ () 1 θ () 2 θ (), (21) Note here C 1 ad C 2 should be uderstood as sets of subsets i the cotext of Theorem 1. We will eed the followig lemma from [7 to prove Theorem 6. It is clear that Propositio 5 follows directly from this lemma. Lemma 1 ([7 -way sub-modularity): Let θ be a submodular fuctio, the θ(c i ) θ(c, ), (22) C, =1 {j 1,j 2,...,j } I (C j1 C j2 C j ). (2) Proof of Theorem 6:

4 We first prove that θ () θ () 1. For otatioal simplicity, A i is used to deote I \ {i} for i = 1, 2,...,. By Lemma 1 we ca write for a fixed j that θ({b : B A i j}) We claim that C (j) i C (j) i = A =i θ(c (j) i ), (24) m A {B : B A m j}. (25) { {B : B j} i + j {B : B j + 1} i + j > (26) To see this, observe that due to its complete symmetry, C (j) i must be i the form {B : B v} for some iteger v. Clearly v j because of the special form of C (j) i. Furthermore, sice ay B that has at least j + 1 elemets i I, it must have at least j elemets i A i for i = 1, 2,...,, i.e., {B : B A i j} {B : B j + 1}, (27) it is clear that v j + 1. Next we show that v = j if ad oly if i + j. If i + j, the let us cosider a special choice of A A = {, 1,..., i + 1}, (28) ad thus for this choice I j {B : B A m j} (29) m A because I j {B : B A m j}, m =, 1,..., i + 1. (0) Thus v = j for this case. Next we show that if i + j >, the there does ot exist a set A such that (29) ca hold, ad it will follow that I j / C (j) i, ad thus v = j+1 for this case. We prove this by cotradictio. For (29) to hold, it must be true that This implies that I j {B : B A m j}, m A. (1) I j A m, m A. (2) However there are oly a total of j possible choices of A m such that I j A m, but at the same time A = i > j by assumptio. Thus a cotradictio arises by the pigeohole priciple. Thus (26) is ideed true. We ca ow write ( 1)θ () 1 = = 1 θ({b : B A i j}) 1 θ({b : B A i j}) 1 ( j)θ({b : B j}) + jθ({b : B j + 1}) = ( 1) θ({b : B j}) = ( 1)θ (), () which completes the proof for θ () θ () 1. To prove θ () θ () 1 for, we tae a similar approach as used i [2 (p. 491). We first limit the choice of A ad B to a -elemet subset of I, ad the taig a uiform choice over its ( 1)-elemet subsets. For each such -elemet subset, θ () θ () 1, ad the iequality remais true after taig the expectatio over all -elemet subset chose uiformly from I. This completes the proof. Theorem 1 ow directly follows from Theorem 6. Clearly, if the sub-modular fuctios i Theorem 6 are replaced by supermodular fuctios, the directio of the iequalities are simply reversed. With this observatio, Theorem 2 follows directly from the super-modularity of the coditioal etropy fuctio H(C C c ), C c = Ω \ C is the complemet of set C. This ca be see as follows. H(C 1 C c 1) + H(C 2 C c 2) = 2H(Ω) H(C c 1) H(C c 2) 2H(Ω) H((C 1 C 2 ) c ) H((C 1 C 2 ) c ) = H(C 1 C 2 (C 1 C 2 ) c ) + H(C 1 C 2 (C 1 C 2 ) c ), (4) the iequality is by the sub-modularity of the etropy fuctio ad the fact C c 1 C c 2 = (C 1 C 2 ) c C c 1 C c 2 = (C 1 C 2 ) c. The proofs of Theorem, Propositio 4 ad Propositio 5 are trivial, ad thus omitted here. IV. CONCLUDING REMARKS We provided a ew set of iequalities, which geeralizes the classic Ha s iequality o etropy rates of subsets of radom variables to iclude commo compoets. These iequalities provide a li betwee Ha s iequality ad the -way submodularity iequality, both of which belog to this ew set of iequalities as special cases. Admittedly, this ew set of iequality has yet to fid applicatios i iformatio theoretical problems. These iequalities, however, are ideed motivated by oe of author s recet wors [11, ad the cosideratio to iclude commo compoets comes from the commo message requiremet i broadcast chael. It appears possible to simplify the rather legthy proof give i [11 with the help of this ew set of iformatio

5 iequalities. It remais to be see whether this ew set of iequalities ca fid its applicatio else. REFERENCES [1 T. S. Ha, Noegative etropy measures of multivariate symmetric correlatios, Iform. Cotr., vol. 6, o. 2, pp , [2 T. M. Cover ad J. A. Thomas, Elemets of Iformatio Theory, New Yor: Wiley, [ A. Albaese, J. Blomer, J. Edmods, M. Luby, ad M. Suda, Priority ecodig trasmissio, IEEE Tras. Iformatio Theory, vol. 42, o. 6, pp , Nov [4 H. Wag ad P. Viswaath, Vector Gaussia multiple descriptio with two levels of receivers, IEEE Tras. Iformatio Theory, vol. 55, o. 1, pp , Ja [5 C. Tia, S. Mohajer, ad S. Diggavi, Approximatig the Gaussia multiple descriptio rate regio uder symmetric distortio costraits, IEEE Tras. Iformatio Theory, vol. 55, o. 8, pp , Aug [6 S. Fujishige, Polymatroidal depedece structure of a set of radom variables, Iform. Cotr., vol. 9, o. 1, pp , Oct [7 H. J. A. Harvey, R. Kleiberg, ad A. R. Lehma, O the capacity of iformatio etwors, IEEE Tras. Iformatio Theory, vol. 52, o. 6, pp , Ju [8 F. R. K. Chug, R. L. Graham, P. Fral, ad J. B. Shearer, Some itersectio theorems for ordered sets ad graphs, J. Combiatorial Theory, Ser. A, vol. 52, pp. 27, [9 M. Madima ad P. Tetali, Iformatio iequalities for joit distributios, with iterpretatios ad applicatios, IEEE Tras. Iformatio Theory, vol. 56, o. 6, pp , Ju [10 R. Yeug, A First Course i Iformatio Theory, Kluwer Academic/Pleum Publishers, New Yor, [11 C. Tia, Latet capacity regio: a case study o symmetric broadcast with commo messages, IEEE Tras. Iformatio Theory, vol. 57, o. 6, pp , Ju