SHANNON S information measures refer to entropies, conditional

Transcription

1 1924 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 43, NO. 6, NOVEMBER 1997 A Framework for Linear Information Inequalities Raymond W. Yeung, Senior Member, IEEE Abstract We present a framework for information inequalities, namely, inequalities involving only Shannon s information measures, for discrete rom variables. A region in IR 2 01, denoted by 0 3, is identified to be the origin of all information inequalities involving n rom variables in the sense that all such inequalities are partial characterizations of 0 3. A product from this framework is a simple calculus for verifying all unconstrained constrained linear information identities inequalities which can be proved by conventional techniques. These include all information identities inequalities of such types in the literature. As a consequence of this work, most identities inequalities involving a definite number of rom variables can now be verified by a software called ITIP which is available on the World Wide Web. Our work suggests the possibility of the existence of information inequalities which cannot be proved by conventional techniques. We also point out the relation between 0 3 some important problems in probability theory information theory. Index Terms Entropy, I-Measure, information identities, information inequalities, mutual information. Example 2: I. INTRODUCTION SHANNON S information measures refer to entropies, conditional entropies, mutual informations, conditional mutual informations. For information inequalities, we refer to those involving only Shannon s information measures for discrete rom variables. These inequalities play a central role in converse coding theorems for problems in information theory with discrete alphabets. This paper is devoted to a systematic study of these inequalities. We begin our discussion by examining the two examples below which exemplify what we call the conventional approach to proving such inequalities. Example 1: This is a version of the well-known data processing theorem. Let be rom variables such that form a Markov chain. Then In the above, the second equality follows from the Markov condition, while the inequality follows because is always nonnegative. Manuscript received August 10, 1995; revised February 10, The material in this paper was presented in part at the 1996 IEEE Information Theory Workshop, Haifa, Israel, June 9 13, The author is with the Department of Information Engineering, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong. Publisher Item Identifier S (97) The inequalities above follow from the nonnegativity of,,, respectively. In the conventional approach, we invoke certain elementary identities inequalities in the intermediate steps of a proof. Some frequently invoked identities inequalities are if if Proving an identity or an inequality using the conventional approach can be quite tricky, because it may not be easy to see which elementary identity or inequality should be invoked next. For certain problems, like Example 1, we may rely on our insight to see how we should proceed in the proof. But of course, most of our insight in problems is developed from the hindsight. For other problems like or even more complicated than Example 2 (which involves only three rom variables), it may not be easy at all to work it out by brute force. The proof of information inequalities can be facilitated by the use of information diagrams 1 [25]. However, the use of such diagrams becomes very difficult when the number of rom variables is more than four. 1 It was called an I-diagram in [25], but we prefer to call it an information diagram to avoid confusion with an eye diagram in communication theory /97$ IEEE

2 YEUNG: A FRAMEWORK FOR LINEAR INFORMATION INEQUALITIES 1925 In the conventional approach, elementary identities inequalities are invoked in a sequential manner. In the new framework that we shall develop in this paper, all identities inequalities are considered simultaneously. Before we proceed any further, we would like to make a few remarks. Let be any expressions depending only on Shannon s information measures. We shall call them information expressions, specifically linear information expressions if they are linear combinations of Shannon s information measures. Likewise, we shall call inequalities involving only Shannon s information measures information inequalities. Now if only if. Therefore, if for any expression we can determine whether it is always nonnegative, then we can determine whether any particular inequality always holds. We note that if only if. Therefore, it suffices to study inequalities. The rest of the paper is organized as follows. In the next section, we first give a brief review of -Measure [25] on which a few proofs will be based. In Section III, we introduce the canonical form of an information expression discuss its uniqueness. We also define a region called which is central to the discussion in this paper. In Section IV, we present a simple calculus for verifying information identities inequalities which can be proved by conventional techniques. In Section V, we further elaborate the significance of by pointing out its relations with some important problems in probability theory information theory. Concluding remarks are given in Section VI. II. REVIEW OF THE THEORY OF -MEASURE In this section, we give a review of the main results regarding -Measure. For a detailed discussion of -Measure, we refer the reader to [25]. Further results on -Measure can be found in [7]. Let be jointly distributed discrete rom variables, be a set variable corresponding to a rom variable. Define the universal set to be let be the -field generated by. The atoms of have the form, where is either or. Let be the set of all atoms of except for, which is by construction because Note that. To simplify notations, we shall use to denote to denote. Let. It was shown in [25] that there exists a unique signed measure on which is consistent with all Shannon s information measures via the following formal substitution of symbols: (1) ( ), i.e., for any (not necessarily disjoint) When When, we interpret (2) as, (2) becomes When, (2) becomes Thus (2) covers all the cases of Shannon s information measures. Let (2) (3) (4) (5) for some (6) Note that. Let. Define arbitrary one-to-one mappings let (7) (8) where for. Then where is a unique matrix (independent of ) with if if (9) (10) An important characteristic of is that it is invertible [25], so we can write (11) In other words, is completely specified by the set of values,, namely, all the joint entropies involving, it follows from (5) that is the unique measure on which is consistent with all Shannon s information measures. Note that in general is not nonnegative. However, if form a Markov chain, is always nonnegative [7]. As a consequence of the theory of -Measure, the information diagram was introduced as a tool to visualize the relationship among information measures [25]. Applications of information diagrams can be found in [7], [25], [26].

3 1926 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 43, NO. 6, NOVEMBER 1997 III. THE CANONICAL FORM In the rest of the paper, we shall assume that are the rom variables involved in our discussion. We observe that conditional entropies, mutual informations, conditional mutual informations can be expressed as a linear combination of joint entropies by using the following identity: (12) where. Thus any information expression can be expressed in terms of the joint entropies. We call this the canonical form of an information expression. Now for any their joint entropies correspond to a vector in, where we regard as the coordinates of. On the other h, a vector in is said to be constructible if there exist whose joint entropies are given by. We are then motivated to define is constructible As we shall see, not only gives a complete characterization of all information inequalities, but it also is closely related to some important problems in probability theory information theory. Thus a complete characterization of is of fundamental importance. To our knowledge, there has not been such a characterization in the literature (see Section V). Now every information expression can be expressed in canonical form. A basic question to ask is in what sense the canonical form is unique. Toward this end, we shall first establish the following theorem. Theorem 1: Let be measurable such that has zero Lebesque measure. Then cannot be identically zero on. We shall need the following lemma which is immediate from the discussion in [26, Sec. 6]. The proof is omitted here. Lemma 1: Let is constructible (cf., (11)). Then the first quadrant of is a subset of. Proof of Theorem 1: If has positive Lebesque measure, since has zero Lebesque measure hence has zero Lebesque measure, has positive Lebesque measure. Then cannot be a subset of, which implies that cannot be identically zero on. Thus it suffices to prove that has positive Lebesque measure. Using the above Lemma, we see that the first quadrant of, which has positive Lebesque measure, is a subset of. Therefore has positive Lebesque measure. Since is an invertible linear transformation of, its Lebesque measure must also be positive. This proves the theorem. The uniqueness of the canonical form for very general classes of information expressions follows from this theorem. For example, suppose are two polynomials of the joint entropies such that for all. Let.If is not the zero function, then has zero Lebesque measure. By the theorem, cannot be identical to zero on, which is a contradiction. Therefore is the zero function, i.e.,. Thus we see that the canonical form is unique for polynomial information expressions. We note that the uniqueness of the canonical form for linear information expressions has been discussed in [4] [2, p. 51, Theorem 3.6]. The importance of the canonical form will become clear in the next section. An application of the canonical form to recognizing the symmetry of an information expression will be discussed in Appendix II-A. We note that any invertible linear transformation of the joint entropies can be used for the purpose of defining the canonical form. Nevertheless, the current definition of the canonical form has the advantage that if are two sets of rom variables such that, then the joint entropies involving the rom variables in is a subset of the joint entropies involving the rom variables in. IV. A CALCULUS FOR VERIFYING LINEAR IDENTITIES AND INEQUALITIES In this section, we shall develop a simple calculus for verifying all linear information identities inequalities involving a definite number of rom variables which can be proved by conventional techniques. All identities inequalities in this section are assumed to be linear unless otherwise specified. Although our discussion will primarily be on linear identities inequalities (possibly with linear constraints), our approach can be extended naturally to nonlinear cases. For nonlinear cases, the amount of computation required is larger. The question of what linear combinations of entropies are always nonnegative was first raised by Han [5]. A. Unconstrained Identities Due to the uniqueness of the canonical form for linear information expressions as discussed in the preceding section, it is easy to check whether two expressions are identical. All we need to do is to express in canonical form. If all the coefficients are zero, then are identical, otherwise they are not. B. Unconstrained Inequalities Since all information expressions can be expressed in canonical form, we shall only consider inequalities in this form. The following is a simple yet fundamental observation which apparently has not been discussed in the literature. For any, always holds if only if. This observation, which follows immediately from the definition of, gives a complete characterization of all unconstrained inequalities (not necessary linear) in terms of. From this point of view, an unconstrained inequality is simply a partial characterization of. The nonnegativity of all Shannon s information measures form a set of inequalities which we shall refer to as the basic

4 YEUNG: A FRAMEWORK FOR LINEAR INFORMATION INEQUALITIES 1927 inequalities. We observe that in the conventional approach to proving information inequalities, whenever we establish an inequality in an intermediate step, we invoke one of the basic inequalities. Therefore, all information inequalities conditional information identities which can be proved by conventional techniques are consequences of the basic inequalities. These inequalities, however, are not nonredundant. For example,, which are both basic inequalities of the rom variables, imply again a basic inequality of. We shall be dealing with linear combinations whose coefficients are nonnegative. We call such linear combinations nonnegative linear combinations. We observe that any Shannon s information measure can be expressed as a nonnegative linear combination of the following two elemental forms of Shannon s information measures: i) ii), where. This can be done by successive (if necessary) application(s) of the following identities: (13) (14) (15) (16) (17) (18) (Note that all the coefficients in the above identities are nonnegative.) It is easy to check that the total number of Shannon s information measures of the two elemental forms is equal to (19) The nonnegativity of the two elemental forms of Shannon s information measures form a proper subset of the set of basic inequalities. We call the inequalities in this smaller set the elemental inequalities. They are equivalent to the basic inequalities because each basic inequality which is not an elemental inequality can be obtained by adding a certain set of elemental inequalities in view of (13) (18). The minimality of the elemental inequalities is proved in Appendix I. If the elemental inequalities are expressed in canonical form, then they become linear inequalities in. Denote this set of inequalities by, where is an matrix, define (20) Since the elemental inequalities are satisfied by any, we have. Therefore, if then i.e., always holds. Let, be the column -vector whose th component is equal to all the other components are equal to. Since a joint entropy can be expressed as a nonnegative linear combination of the two elemental forms of Shannon s information measures, each can be expressed as a nonnegative linear combination of the rows of. This implies that is a pyramid in the positive quadrant. Let be any column -vector. Then, a linear combination of joint entropies, is always nonnegative if. This is equivalent to say that the minimum of the problem (Primal) Minimize subject to is zero. Since gives ( is the only corner of ), all we need to do is to apply the optimality test of the simplex method [19] to check whether the point is optimal. We can obtain further insight in the problem from the Duality Theorem in linear programming [19]. The dual of the above linear programming problem is (Dual) Maximize subject to where By the Duality Theorem, the maximum of the dual problem is also zero. Since the cost function in the dual problem is zero, the maximum of the dual problem is zero if only if the feasible region (21) is nonempty. Theorem 2: is nonempty if only if for some, where is a column -vector, i.e., is a nonnegative linear combination of the rows of. Proof: We omit the simple proof that is nonempty if only if for some, where is a column -vector. Let If for some, then. Let (22) Since can be expressed as a nonnegative linear combination of the rows of (23) can also be expressed as a nonnegative linear combinations of the rows of. By (22), this implies for some. Thus always holds (subject to ) if only if it is a nonnegative linear combination of the elemental inequalities (in canonical form).

5 1928 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 43, NO. 6, NOVEMBER 1997 We now summarize the results in this section. For information expressions, let be the cost function subject to the elemental inequalities. Then apply the optimality test of the simplex method to the point. If is optimal, then always holds. If not, then may or may not always hold. If it always holds, it is not implied by the elemental inequalities. In other words, it cannot be proved by conventional techniques, namely, invoking the elemental inequalities. Han has previously studied unconstrained information inequalities involving three rom variables [5] as well as information inequalities which are symmetrical in all the rom variables involved [6], explicit characterizations of such inequalities were obtained. A discussion of these results is found in Appendix II. C. Constrained Inequalities Linear constraints on arise frequently in information theory. Some examples are 1),, are mutually independent if only if 2),, are pairwise-independent if only if 3) is a function of if only if. 4) form a Markov chain if only if. In order to facilitate our discussion, we now introduce an alternative set of notations for. We do not distinguish elements singletons of, we write unions of subsets of as juxtapositions. For any nonempty, we use to denote, i.e., (refer to Section II for the definition of ). We also define for nonempty When is a subspace of, we can easily modify the method in the last subsection by taking advantage of the linear structure of the problem. Let the constraints on be given by (24) where is a matrix (i.e., there are constraints). Following our discussion in the last subsection, a linear combination of joint entropies is always nonnegative under the constraint if the minimum of the problem Minimize subject to is zero. Let be the rank of. Since is in the null space of, we can write (25) where is a matrix whose columns form a basis of the orthogonal complement of the row space of, is a column -vector. Then the elemental inequalities can be expressed as in terms of, becomes (26) (27) which is a pyramid in (but not necessarily in the positive quadrant). Likewise, can be expressed as. With the constraints all expressions in terms of, is always nonnegative under the constraint if the minimum of the problem Minimize subject to to simplify notations. In general, a constraint is given by a subset of. For instance, for the last example above, When, there is no constraint. (In fact, there is no constraint if.) Parallel to our discussion in the preceding subsection, we have the following more general observation: Under the constraint, for any, always hold if only if. Again, this gives a complete characterization of all constrained inequalities in terms of. Thus in fact is the origin of all constrained inequalities, with unconstrained inequalities being a special case. In this the next subsection, however, we shall confine our discussion to the linear case. is zero. Again, since gives ( is the only corner of ), all we need to do is to apply the optimality test of the simplex method to check whether the point is optimal. By imposing the constraints in (24), the number of elemental inequalities remains the same, while the dimension of the problem decreases from to. Again from the Duality Theorem, we see that is always nonnegative if for some, where is a column - vector, i.e., is a nonnegative linear combination of the elemental inequalities (in terms of ). We now summarize the results in this section. Let the constraints be given in (24). For expressions, let. Then let be the cost function subject to the elemental inequalities (in terms of ) apply the optimality test to the point.if is optimal, then always holds, otherwise it may or may not always hold. If it always holds, it is not implied by the elemental inequalities. In other words, it cannot be proved by conventional techniques.

6 YEUNG: A FRAMEWORK FOR LINEAR INFORMATION INEQUALITIES 1929 D. Constrained Identities We impose the constraints in (24) as in the last subsection. As we have pointed out at the beginning of the paper, two information expressions are identical if only if always hold. Thus we can apply the method in the last subsection to verify all constrained identities that can be proved by conventional techniques. When are unconstrained, the uniqueness of the canonical form for linear information expressions asserts that if only if. However, when the constraints in (24) on are imposed, does not imply. We give a simple example to illustrate this point. Suppose we impose the constraint. Then every information expression can be expressed in terms of. Now consider (28) Note that the coefficients in the above expression are nonzero. But from the elemental inequalities, we have (29) V. FURTHER DISCUSSION ON We have seen that, but it is not clear whether.ifso, hence all information inequalities are completely characterized by the elemental inequalities. In the following, we shall use the notations when we refer to for a specific. For In -Measure notations, the elemental inequalities are,,. It then follows from Lemma 1 that. Inspired by the current work, the characterization of has recently been investigated by Zhang Yeung. They have found that (therefore in general), but, the closure of, is equal to [29]. This implies that all unconstrained (linear or nonlinear) inequalities involving three rom variables are consequences of the elemental inequalities of the same set of rom variables. However, it is not clear whether the same is true for all constrained inequalities. They also have discovered the following conditional inequality involving four rom variables which is not implied by the elemental inequalities: If, then (30) which imply that. We now discuss a special application of the method described in this subsection. Let us consider the following problem which is typical in probability theory. Suppose we are given that form a Markov chain, that are independent. We ask whether are always independent. This problem can be formulated in information-theoretic terms with the constraints represented by,,, we want to know whether they imply. Problems of such kind can be hled by the method described in this subsection. Our method can prove any independence relation which can be proved by conventional information-theoretic techniques. The advantage of using an information-theoretic formulation of the problem is that we can avoid manipulations of the joint distribution directly, which is awkward [8], if not difficult. It may be difficult to devise a calculus to hle independence relations of rom variables in a general setting, 2 because an independence relation is discrete in the sense that it is either true or false. On the other h, the problem becomes a continuous one if it is formulated in informationtheoretic terms (because mutual informations are continuous functionals), continuous problems are in general less difficult to hle. From this point of view, the problem of determining independence of rom variables is a discrete problem embedded in a continuous problem. 2 A calculus for independence relations has been devised by Massey [9] for the special case when the rom variables have a causal interpretation. If, in addition, the above inequality implies that This is a conditional independence relation which is not implied by the elemental inequalities. However, whether remained an open problem. Subsequently, they have determined that by discovering the following unconstrained inequality involving four rom variables which is not implied by the elemental inequalities of the same set of rom variables [30]: The existence of the above two inequalities indicates that there may be a lot of information inequalities yet to be discovered. Since most converse coding theorems are proved by means of information inequalities, it is plausible that some of these inequalities yet to be discovered are needed to settle certain open problems in information theory. In the remainder of the section, we shall further elaborate on the significance of by pointing out its relations with some important problems in probability theory information theory.

7 1930 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 43, NO. 6, NOVEMBER 1997 A. Conditional Independence Relations For any fixed number of rom variables, a basic question is what sets of conditional independence relations are possible. In the recent work of Matúš Studený [17], this problem is formulated as follows. Recall that let be the family of all couples where is the union of two, not necessarily different, singletons of. Having a system of rom variables with subsystems,, we introduce the notation where is the abbreviation of the statement is conditionally independent of given. For, means is determined by. The subsystem is presumed to be constant. A subfamily is called probabilistically ( -) representable if there exists a system, called a -representation, such that. The problem is to characterize the class of all -representable relations. Note that this problem is more general than the application discussed in Section IV-D. Now is equivalent to. If is a proper subset of, i.e., is not of elemental form, then can be written as a nonnegative combination of the corresponding elemental forms of Shannon s information measure. We observe that if only if each of the corresponding elemental forms of Shannon s information measures vanishes, that an elemental form of Shannon s information measure vanishes if only if the corresponding conditional independence relation holds. Thus it is actually unnecessary to consider,, for separately because it is determined by the other conditional independence relations. Let us now look at some examples. For As pointed out in the last paragraph, the couples are actually redundant. Let be a system of rom variables such that is not deterministic,,, are not functions of each other. Then it is easy to see that Thus is -representable. On the other h, is not - representable, because imply. The recent studies on the problem of conditional independence relations was launched by a seminal paper by Dawid [3], in which he proposed four axioms as heuristic properties of conditional independence. In information-theoretic terms, these four axioms can be summarized by the following statement: Subsequent work on this subject has been done by Pearl his collaborators in the 1980 s, their work is summarized in the book by Pearl [18]. Their work has mainly been motivated by the study of the logic of integrity constraints from databases. Pearl conjectured that Dawid s four axioms completely characterize the conditional independence structure of any joint distribution. This conjecture, however, was refuted by the work of Studený [20]. Since then, Matúš Studený have written a series of papers on this problem [10] [17], [20] [24]. So far, they have solved the problem for three rom variables, but the problem for four rom variables remains open. The relation between this problem is the following. Suppose we want to determine whether a subfamily of is -representable. Now each corresponds to setting to zero in. Note that is a hyperplane containing the origin in. Thus is - representable if only if there exists a in such that for all. Therefore, the problem of conditional independence relations is a subproblem of the problem of characterizing. B. Optimization of Information Quantities Consider minimizing given (31) (32) (33) where. This problem is equivalent to the following minimization problem. Minimize subject to (34) (35) (36) (37) As no characterization of is available, this minimization problem cannot be solved. Nevertheless, since, if we replace by in the above minimization problem, it becomes a linear programming problem which renders a lower bound on the solution. C. Multiuser Information Theory The framework for information inequalities developed in this paper provides new tools for problems in multiuser information theory. Consider the source coding problem in Fig. 1, in which are source rom variables,

8 YEUNG: A FRAMEWORK FOR LINEAR INFORMATION INEQUALITIES 1931 Fig. 1. A multiterminal source coding problem. the blocks on the left right are encoders decoders, respectively. The rom variables,, are the outputs of the corresponding encoders. Given,,, where, we are interested in the admissible region of the triple. Evidently,,, give the number of bits needed for the encoders. From the encoding decoding requirements, we immediately have,,,,, equal to zero. Now there are five rom variables involved in this problem. Then the intersection of the set containing all such that is the set of all possible vectors of the joint entropies involving given that they satisfy the encoding decoding requirements of the problem as well as the constraints on the joint entropies involving. Then is given as the projection of this set on the coordinates,,. In the same spirit as that in the last subsection, an explicit outer bound of, denoted by, is given by replacing by. We refer to an outer bound such as as an LP (linear programming) bound. This is a new tool for proving converse coding theorems for problems in multiuser information theory. The LP bound already has found applications in the recent work of Yeung Zhang [28] on a new class of multiterminal source coding problems. We expect that this approach will have impact on other problems in multiuser information theory. VI. CONCLUDING REMARKS We have identified the region as the origin all information inequalities. Our work suggests the possibility of the existence of information inequalities which cannot be proved by conventional techniques, this has been confirmed by the recent results of Zhang Yeung [29], [30]. A product from the framework we have developed is a simple calculus for verifying all linear information inequalities involving a definite number of rom variables possibly with linear constraints which can be proved by conventional techniques; these include all inequalities of such type in the literature. Based on this calculus, a software running on MATLAB called ITIP (Information-Theoretic Inequality Prover) has been developed by Yeung Yan [27], it is available on World Wide Web. The following session from ITIP contains verifications of Example 1 2, respectively, in Section I. >> ITIP( I(Y; Z) >= I(X; Z), I(X; Z Y) = 0 ) True >> ITIP( H(X,Y) H(Y) I(Y; X,Z) H(Y Z) >= 0 ) True We see from (19) that the amount of computation required is moderate when. Our work gives a partial answer to Han s question of what linear combinations of entropies are always nonnegative [5]. A complete answer to this question is impossible without further characterization of. The characterization of is a very fundamental problem in information theory. However, in view of the difficulty of some special cases of this problem [15], [17], [29], [30], it is not very hopeful that this problem can be solved completely in the near future. Nevertheless, partial characterizations of may lead to the discovery of some new inequalities which make the solutions of certain open problems in information theory possible. APPENDIX I MINIMALITY OF THE ELEMENTAL INEQUALITIES The elemental inequalities in set-theoretic notations have one of the following two forms: 1) ; 2), where. They will be referred to as -inequalities -inequalities, respectively. We are to show that all the elemental inequalities are nonredundant, i.e., none of them is implied by the others. For an -inequality (38) since it is the only elemental inequality which involves the atom, it is clearly not implied by the other elemental inequalities. Therefore, we only need to show that all -inequalities are nonredundant. To show that a -inequality is nonredundant, it suffices to show that there exists a measure on for that one. We shall show that the which satisfies all other elemental inequalities except -inequality (39) is nonredundant. To facilitate our discussion, we denote by we let be the atoms in, where (40) We first consider the case when, i.e.,. We construct a measure by if otherwise (41) where. In other words, is the only

9 1932 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 43, NO. 6, NOVEMBER 1997 atom with measure ; all other atoms have measure. Then is trivially true. It is also trivial to check that for any (42) for any such that (43) if. On the other h, if is a proper subset of, then contains at least two atoms, therefore, (44) This completes the proof for the -inequality in (39) to be nonredundant when. We now consider the case when, or. We construct a measure as follows. For the atoms in, let. (45) For,if is odd, it is referred to as an odd atom of, if is even, it is referred to as an even atom of. For any atom, we let (46) Consider (51) The nonnegativity of the second term above follows from (46). For the first term, is nonempty if only if (52) If this condition is not satisfied, then the first term in (51) becomes, (50) follows immediately. Let us assume that the condition in (52) is satisfied. Then by simple counting, we see that the number atoms in is equal to, where For example, for, there are atoms in namely, where or for. We check that This completes the construction of. We first prove that (47) We first consider the case when, i.e., Consider Then where the last equality follows from the binomial formula (48) for. This proves (47). Next we prove that satisfies all -inequalities. We note that for any, the atom is not in. Thus contains exactly one atom. If this atom is an even atom of, then the first term in (51) is either or (cf., (45)), (50) follows immediately. If this atom is an odd atom of, then the first term in (51) is equal to. This happens if only if have one common element, which implies that is nonempty. Therefore, the second term in (51) is at least, hence (50) follows. Finally, we consider the case when. Using the binomial formula in (48), we see that the number of odd atoms even atoms of in (49) are the same. Therefore, the first term in (51) is equal to if It remains to prove that satisfies all -inequalities except for (39), i.e., for any such that (50) is equal to otherwise. The former is true if only if, which implies that

10 YEUNG: A FRAMEWORK FOR LINEAR INFORMATION INEQUALITIES 1933 is nonempty, or that the second term is at least. Thus in either case (50) is true. This completes the proof that (39) is nonredundant. APPENDIX II SOME SPECIAL FORMS OF UNCONSTRAINED INFORMATION INEQUALITIES In this appendix, we shall discuss some special forms of unconstrained linear information inequalities previously investigated by Han [5], [6]. Explicit necessary sufficient conditions for these inequalities to always hold have been obtained. The relation between these inequalities the results in the current paper will also be discussed. It follows trivially from the elemental inequalities that is a sufficient condition for to always hold. The necessity of this condition can be seen by noting the existence of rom variables for each such that for all. This implies that all unconstrained linear symmetrical information inequalities are consequences of the elemental inequalities. We refer the reader to [5] for a more detailed discussion of symmetrical information inequalities. B. Information Inequalities Involving Three Rom Variables Consider. Let A. Symmetrical Information Inequalities An information expression is said to be symmetrical if it is identical under every permutation among. For example, for, the expression is symmetrical. This can be seen by permuting symbolically in the expression. Now let us consider the expression. If we replace by each other, the expression becomes, which is symbolically different from the original expression. However, both expression are identical to. Therefore, the two expressions are in fact identical, the expression is actually symmetrical although it is not readily recognized symbolically. The symmetry of an information expression in general cannot be recognized symbolically. However, it is readily recognized symbolically if the expression is in canonical form. This is due to the uniqueness of the canonical form as discussed in Section III. Consider a linear symmetrical information expression (in canonical form). As seen in Section IV-B, can be expressed as a linear combination of the two elemental forms of Shannon s information measures. It was shown in [5] that every symmetrical expression can be written in the form let Since is an invertible linear transformation of, all linear information expression can be written as, where It was shown in [6] that always holds if only if the following conditions are satisfied: (53) In terms of, the elemental inequalities can be expressed as, where where, for, Note that is the sum of all Shannon s information measures of the first elemental form, for, is the sum of all Shannon s information measures of the second elemental form conditioning on rom variables. From the discussion in Section IV-B, we see that always holds if only if is a nonnegative linear combination of the rows of. We leave it as an exercise for the reader to show that is a nonnegative linear combination of the rows of if only if the conditions in (53) are satisfied. Therefore, all unconditional linear inequalities involving three rom variables are consequences of the elemental inequalities. This result also implies that is the smallest pyramid containing.

11 1934 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 43, NO. 6, NOVEMBER 1997 ACKNOWLEDGMENT The author wishes to acknowledge the help of a few individuals during the preparation of this paper. They include I. Csiszár, B. Hajek, F. Matúš, Y.-O. Yan, E.-h. Yang, Z. Zhang. REFERENCES [1] T. M. Cover J. A. Thomas, Elements of Information Theory. New York: Wiley, [2] I. Csiszár J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems. New York: Academic, [3] A. P. Dawid, Conditional independence in statistical theory (with discussion), J. Roy. Statist. Soc., Ser. B, vol. 41, pp. 1 31, [4] T. S. Han, Linear dependence structure of the entropy space, Inform. Contr, vol. 29, pp , [5], Nonnegative entropy measures of multivariate symmetric correlations, Inform. Contr., vol. 36, pp , [6], A uniqueness of Shannon s information distance related nonnegativity problems, J. Combin.., Inform. Syst. Sci., vol. 6, no. 4, pp , [7] T. Kawabata R. W. Yeung, The structure of the I-Measure of a Markov chain, IEEE Trans. Inform. Theory, vol. 38, pp , May [8] J. L. Massey, Determining the independence of rom variables, in 1995 IEEE Int. Symp. on Information Theory (Whistler, BC, Canada, Sept , 1995). [9], Causal interpretations of rom variables, in 1995 IEEE Int. Symp. on Information Theory (Special session in honor of Mark Pinsker on the occasion of his 70th birthday) (Whistler, BC, Canada, Sept , 1995). [10] F. Matúš, Abstract functional dependency structures, Theor. Comput. Sci., vol. 81, pp , [11], On equivalence of Markov properties over undirected graphs, J. Appl. Probab., vol. 29, pp , [12], Ascending descending conditional independence relations, in Trans. 11th Prague Conf. on Information Theory, Statistical Decision Functions Rom Processes (Academia, Prague, 1992), vol. B, pp [13], Probabilistic conditional independence structures matroid theory: Background, Int. J. General Syst., vol. 22, pp , [14], Extreme convex set functions with many nonnegative differences, Discr. Math., vol. 135, pp , [15], Conditional independence among four rom variables II, Combin., Prob. Comput., to be published. [16], Conditional independence structures examined via minors, Ann. Math. Artificial Intell., submitted for publication. [17] F. Matúš M. Studený, Conditional independence among four rom variables I, Combin., Prob. Comput., to be published. [18] J. Pearl, Probabilistic Reasoning in Intelligent Systems. San Mateo, CA: Morgan Kaufman, [19] G. Strang, Linear Algebra Its Applications, 2nd ed. New York: Academic, [20] M. Studený, Attempts at axiomatic description of conditional independence, in Proc. Work. on Uncertainty Processing in Expert Systems, supplement to Kybernetika, vol. 25, nos. 1 3, pp , [21], Multiinformation the problem of characterization of conditional independence relations, Probl. Contr. Inform. Theory, vol. 18, pp. 3 16, [22], Conditional independence relations have no finite complete characterization, in Trans. 11th Prague Conf. on Information Theory, Statistical Decision Functions Rom Processes (Academia, Prague, 1992), vol. B, pp [23], Structural semigraphoids, Int. J. Gen. Syst., submitted for publication. [24], Descriptions of structures of stochastic independence by means of faces imsets (in three parts), Int. J. Gen. Syst., submitted for publication. [25], A new outlook on Shannon s information measures, IEEE Trans. Inform. Theory, vol. 37, pp , May [26], Multilevel diversity coding with distortion, IEEE Trans. Inform. Theory, vol. 41, pp , Mar [27] R. W. Yeung Y.-O. Yan, ITIP, [Online] Available WWW: ITIP. [28] R. W. Yeung Z. Zhang, Miltilevel distributed source coding, in 1997 IEEE Int. Symp. on Information Theory (Ulm, Germany, June 1997), p [29] Z. Zhang R. W. Yeung, A non-shannon type conditional information inequality, this issue, pp [30], On the characterization of entropy function via information inequalities, to be published in IEEE Trans. Inform. Theory.