Lecture 3: Oct 7, 2014

Transcription

1 Information and Coding Theory Autumn 04 Lecturer: Shi Li Lecture : Oct 7, 04 Scribe: Mrinalkanti Ghosh In last lecture we have seen an use of entropy to give a tight upper bound in number of triangles in an extremal graph in certain family. In this lecture we explore few more combinatorial applications of entropy. Before we get into the examples, let us recapitulate a few facts about entropy: H(Y X) H(Y ) H(X X... X n ) = H(X ) + n i= H(X i X... X i ) n i= H(X i) H(X) log U where U is the range of discrete random variable X with equality when X is distributed uniformly. The problems we discuss in this lecture require us to count various structures in certain extremal objects. In the first look, these problems seem unrelated to information theory or entropy. Nevertheless, we can use properties of entropy profitably to approach these problems. Some of the problems discussed in this lecture also appears in [Jai0]. Interested readers are rquested to refer to [Jai0] for details and for many other intersecting examples which are not covered here. Counting repetition in a square matrix Claim. Let A be an n n matrix such that each row and column has at most t distinct numbers. There is a number x such that x appears in A at least n /t times. Proof: Let X, Y, Z be a triple of random variables where X U, Y U and Z = A XY ( X and Y are independent ), i.e., we independently and uniformly randomly choose a row X, a column Y and set Z to the (X, Y )-th element of the matrix. Since each row has at most t distinct numbers, for all i, H(Z X = i) log t and hence H(Z X) = E i [H(Z X = i)] log t. Similarly, H(Z Y ) log t. So we want to establish relationship among H(Z X), H(Z Y ) and H(Z). To that end, we observe that Z is completely determined by X, Y and hence H(Z XY ) = 0. Also, X, Y are independent: H(XY ) = H(X) + H(Y ). Combining these two, we know that H(XY Z) = H(XY ) + H(Z XY ) = H(X) + H(Y ). Applying the chain rule in a different way and using the fact that conditioning reduces entropy, we

2 get that H(Z) = H(XY ) H(XY Z) = H(X) + H(Y ) H(X Z) H(Y XZ) H(X) + H(Y ) H(X Z) H(Y Z) Finally, noticing that H(X) H(X Z) = H(Z) H(Z X) and H(Y ) H(Y Z) = H(Z) H(Z Y ) gives that H(Z) H(Z) H(Z X) H(Z Y ) H(Z) H(Z X) + H(Z Y ) log t. Recall that H(Z) = ( ) ( ) z log. Thus, from the above we get that on average log ( ) is at most log t. Thus, there exists an element z for which log log t. In particluar, ( ) ( ) let z be the element which occurs with the highest probability. Then log log z, which gives log ( ) = z ( ) log z ( ) log log t. Thus, we get that /t which means that z occurs in at least n /t times in the matrix. Remark. The bound shown above is asymptotically tight: Consider a n n matrix and a t such that t n. Divide the matrix in t squares with sides n/t. Index the squares by [, t ]. Put the number i in the i-th square. Clearly, the requirements for the matrix is satisfied and each number appears n /t times. Shearer s Lemma We first start with an example similar to the one we saw in the last lecture. As we will see later, the bound here is easily implied by Shearer s lemma and the proof of Shearer s lemma is essentially a generalization of the proof here.. Areas under projection vs. Volume Let S = {(x i, y i, z i ) : i } be a set of n (distinct) points in R and let Q xy, Q yz, Q xz be the projections of S to XY, Y Z and XZ plains respectively. To be explicit: Q xy = {(x i, y i ) : i } and so on. We want to show: Claim. max { Q xy, Q yz, Q xz } n / Proof: Let X, Y, Z be random variables defined as (X, Y, Z) U S. Clearly, H(XY Z) = log n. Now (X, Y ) Q xy and so H(XY ) log Q xy. Similarly, H(Y Z) log Q yz and H(XZ) log Q xz. Now we use an inequality proved in last class:

3 Fact. H(XY ) + H(Y Z) + H(XZ) H(XY Z) We include a proof for the sake of completeness: Proof: H(XY ) + H(Y Z) + H(XZ) = H(Y ) + H(X Y ) + H(Y Z) + H(XZ) H(Y XZ) + H(X Y Z) + H(Y Z) + H(XZ) = H(XY Z) So, log Q xy + log Q yz + log Q xz H(XY ) + H(Y Z) + H(XZ) H(XY Z) = log n. Hence, max {log Q xy, log Q yz, log Q xz } log n/, or equivalently, max { Q xy, Q yz, Q xz } n / Remark. The bound shown above is asymptotically tight: Consider an axis-parallel solid cube with side lengths m with one corner placed at the origin. Let S be the integral points of this cube. Clearly, n = m and Q xy = Q yz = Q xz = m = n /.. Shearer s Lemma Consider the fact. and that the entropy is sub-additive. The fact that entropy is subadditive gives that for three variables X, Y and Z H(X) + H(Y ) + H(Z) H(XY Z). In the above example, we needed to bound H(XY ) + H(Y Z) + H(ZX), where each term is not the entropy of a single random variable, but the entropy of a subset of random variables. Since these subsets cover each variable X, Y and Z twice, we got H(XY ) + H(Y Z) + H(ZX) H(XY Z). To state Shearer s lemma, it is more convenient to think of the above in terms of distributions on the subsets of the random variables. Formally, the statement is as follows Lemma.4 (Shearer s Lemma) Let X = {X i : i } be a set of random variables. For any S, let us denote X S = {X i : i S}. Let D be an arbitrary distribution on (set of all subsets of ) and let µ be such that i P [i S] µ. Then E [H(X S)] µ H(X). Before we see the proof, let us see how the lemma implies the previous examples. Let us think of X = X, X = Y and X = Z. For subadditivity, we can consider D to be a uniform distribution on the sets {}, {}, {}. Then, a random set from D contains a given element i {,, } with probabiltiy / and the lemma gives H(X) + H(Y ) + H(Z) H(XY Z).

4 Similarly, taking D to be uniform over {, }, {, } and {, } gives µ = / and hence H(XY ) + H(Y Z) + H(ZX) H(XY Z). We now proceed to the proof of the lemma. Proof: E [H(X S)] = E E = E [ ] H(X i X S [i ] ) i S [ ] H(X i X [i ] ) i S S (i) H(X i X [i ] ) i by Chain rule H(X i X A ) H(X i X B ) for A B s is indicator function for sets = P [i S] H(X i X [i ] ) i µ H(X i X [i ] ) = µ H(X) i Number of graphs in a triangle-intersecting family In this section we consider families of graphs on vertex set for a fixed n. Definition. A family G of graphs is called intersecting if T, K G, T K contains an edge. Fact. For an intersecting family G with edges contained in a set E 0 ( ), G E 0 /. Proof: For any graph G G, let G c be a graph such that E(G c ) = E 0 \ E(G). Now G c G, because G G c does not have any edge. Since there are E 0 possible graphs on with edges contained in E 0, we have G E 0 /. Note that when E = ( ), we have any intersecting family G has at most ( n ) / many graphs. Definition. A family G of graphs is called -intersecting if T, K G, T K contains a. Now we prove the main lemma of this section: Lemma.4 If G is -intersecting then G (n ) /4. 4

5 Proof: For any R let us define G R to be the disjoint union of a clique on R and a clique on \ R. Formally, E(G R ) = {{i, j} : i, j R or i, j \ R}. Note that G R must have one edge: since at least vertices of a triangle is either in R or in \ R together and the edge connecting them is in G R. Since G is a -intersecting, T, K G, we have G R T K has at least one edge. Hence G R := {G R T : T G} is an intersecting family with edges contained in E(G R ). Consider a collection of random variables X = {X e : e ( ) } which is sampled as follows: select a graph G uniformly and randomly form G and for each edge e set X e = E(G) (e) i.e., X e = if e E(G) and 0 otherwise. Since there is a bijection mapping between a graph G and the corresponding values of X, we have H(X) = log G. We will now apply Shearer s lemma by considering a distribution D over subsets of ( ). We define the distribution D as follows: Pick a random R of size n/ (say n is even) and take the subset S = E(G R ). As before, we define X S = {X e e S}. Also, by symmetry, for each e ( ), PS [e S] = E(G R) = (n/ ( n ) = µ (say). Applying Shearer s lemma gives ) ( n ) E [H(X S)] µ H(X) = µ log G. It remains to bound the LHS in the above expression. To do so, we note that for any R and the corresponding S, a set of values for X S corresponds to a graph in G R. By the above discussion, we know that G R is an intersecting family with edges contained in E(G R ) and by Fact., G R E(GR). This gives that H(X S ) E(G R ) = ( ) n/. Combining this with the bound from Shearer s lemma gives log G ( ( ) ) ( n ( ( ) ) ( ) n/ µ = ) n/ n ( ) = n ( ) n n/ n/, and hence G (n ) /4 Remark.5 There exists a -intersecting family G with G (n ) /8: we fix a triangle and take any subset of remaining ( n ) edges to form a -intersecting family. Open Problem.6 There is a gap between the upper and lower bound of the size of a - intersecting family. Are those bounds tight? Can the gap be improved? References [Jai0] Jaikumar Radhakrishnan, Entropy and counting, IIT Kharagpur Golden Jubilee Volume, November 00,A link to the article. 5