Lecture 3: October 2, 2017

Transcription

1 Inforation and Coding Theory Autun 2017 Lecturer: Madhur Tulsiani Lecture 3: October 2, Shearer s lea and alications In the revious lecture, we saw the following stateent of Shearer s lea. Lea 1.1 (Shearer s Lea: distribution version) Let {X 1,..., X } be a set of rando variables. For any S [], let us denote X S = {X i : i S}. Let D be an arbitrary distribution on 2 [] (set of all subsets of []) and let µ be such that i [n] P [i S] µ. Then µ H(X 1,..., X ) E [H(X S )]. We saw soe alications of this lea in the revious lecture and will see soe ore in this one. However, let us first rove the lea. Proof: The roof of the lea follows sily fro the chain rule for entroy and the fact that conditioning reduces entroy (on average). [ ] E [H(X S)] = E E = E = i [n] [ [ i S i S H(X i X S [i 1] ) H(X i X [i 1] ) ] ] 1 S (i) H(X i X [i 1] ) i [n] P [i S] H(X i X [i 1] ) µ H(X i X [i 1] ) = µ H(X 1,..., X ) i [n] by Chain rule H(X i X A ) H(X i X B ) for A B 1 s is indicator function for set S 1

2 1.1 Counting grah hooorhiss Shearer s lea can be used to give an estiate of the nuber of ways of ebedding a sall grah G into a large grah H. For two grahs G : (V G, E G ) and H = (V H, E H ), an ebedding (also called a hooorhis) of G in H is defined as a function f : V G V H such that for all (u, v) E G, we have ( f (u), f (v)) V H. Note that the definition does not revent the iage of non-edge airs in E G fro being edges in E H. We will show an uer bound on the axiu nuber of ebeddings for a grah G into any H with at ost edges. For now, let us take G to be the 5-cycle with vertex set {1, 2, 3, 4, 5}. Consider any grah H with at ost edges and let F = (F(1),..., F(5)) be a collection of rando variables denoting an ebedding of G chosen uniforly fro the set of all ebeddings. Using Shearer s lea, we can write 2 H(F(1),..., F(5)) H(F(1), F(2)) + H(F(2), F(3)) + + H(F(5), F(1)). Since {1, 2} is an edge in G, the air (F(1), F(2)) ust corresond to an (ordered) edge in H. Since the nuber of edges in H is at ost, we get that H(F(1), F(2)) log(2). Using the sae bound for all ters on the right, we get H(F(1),..., F(5)) 5 2 log(2), which gives a bound of (2) 5/2 on the nuber of ebeddings. Exercise 1.2 Check that the exonent of 5/2 in the above bound is tight. The above ethod can also be used to give a tight estiate for any grah G (of constant size). In general, the exonent deends on a araeter known as the fractional indeendent set nuber of G. I will divide this roof in a few arts and add this as an extra roble in the hoework. The solution to this roble need not be subitted. The roof, along with any other cobinatorial alications can also be found in the surveys by Radhakrishnan [Rad03] and [Gal14]. 1.2 An alication to counting ries Consider the roble of finding the nuber of rie nubers less than or equal to a given nuber n. This quantity, denoted by π(n) is the subject of the faous rie nuber theore [Gol73] which shows that π(n) n ln n. 2

3 Here, f (n) g(n) is used to denote li n ( f (n)/g(n)) = 1. A weaker estiate was roved by Chebyshev, who roved that for sufficiently large n, c 1 n ln n π(n) c n 2 ln n. Chebyshev, indeed used (and roved) the following estiate to obtain his bounds: log n log n. An inforation theoretic roof of the above estiate was obtained by Kontoyiannis [Kon07, Kon08]. We sketch the arguent for one side of this estiate below. Let us first obtain a very sile lower bound on π(n). For a given n, let k = π(n) and let 1,..., k be the rie nubers in [n]. Let R be a unifor rando nuber in [n]. Using rie factorization, we can exress R as R = X 1 1 X k k, for rando variables X 1,..., X k. Using sub-additivity of entroy, we can write log n = H(R) = H(X 1,..., X k ) H(X 1 ) + + H(X k ). Now, notice that each rando variable X i takes value at ost log n since 2 X i X i i n. Thus, we have H(X i ) log(log n + 1) for each X i. This gives π(n) = k log n log(log n + 1). We can refine this estiate significantly by noticing that we actually used a terrible bound on H(X i ) for each i. This bound would hold only if X i was uniforly distributed in {0,..., log n}, which is quite far fro it s actual distribution. Each X i is aroxiately distributed as a geoetric rando variable with P [X i t] (1/ i ) t. Plugging in the entroy of a geoetric rando variable gives Substituting this bound gives H(X i ) log ( 1 log 1 1 ) log. log n log n 3 + o(log n).

4 2 Mutual Inforation The utual inforation is a quantity which easures the aount of deendence between two rando variables. Unlike correlation, which defines the rando variables to take values in the sae sace, the utual inforation can be defined for any two rando variables. The utual inforation between two rando variables X and Y is defined by the forula I(X; Y) = H(X) H(X Y) Using the chain rule for entroy, we can see that I(X; Y) = H(X) H(X Y) = H(Y) H(Y X) = H(X) + H(Y) H(X, Y). We can use the first two exressions to observe that I(X; Y) 0 and the last one to observe that I(X; Y) = I(Y; X). Exale 2.1 Consider the rando variable (X, Y) with X Y = 1, X {0, 1} and Y {0, 1} such that: 10 w. 1/3 (X, Y) = 01 w. 1/3 11 w. 1/3 Then, we can calculate the entroy and utual inforation as follows: H(X) = H(Y) = 1 3 log log 3 2 = log H(X, Y) = log 3 I(X; Y) = H(X) + H(Y) H(X, Y) = log Conditioning on a third rando variable Z, we can also define the conditional utual inforation I(X; Y Z) as I(X; Y Z) := E z [I(X Z = z; Y Z = z)] = E z [H(X Z = z) H(X Y, Z = z)] = H(X Z) H(X Y, Z). Consider the following exale of three rando variables. Exale 2.2 Consider the rando variable (X, Y, Z), X {0, 1}, Y {0, 1} and Z = X Y such that: 000 w. 1/4 011 w. 1/4 (X, Y, Z) = 101 w. 1/4 110 w. 1/4 4

5 We can check that in this case, X, Y are indeendent and thus I(X; Y) = 0. However, I(X : Y Z) = E z [I(X Z = z; Y Z = z)] = 1 2 I(X Z = 0; Y Z = 0) + 1 I(X Z = 1; Y Z = 1) 2 = 1 2 log log 2 = 1 The above exale illustrates that unlike entroy, it is not true that conditioning (on average) decreases the utual inforation. In the above exale, while I(X; Y) = 0, we have I(X; Y Z) = 1 which is in fact the axiu ossible. However, as in the case of entroy, utual inforation does obey a chain rule. Lea 2.3 I((X 1,..., X ); Y) = I(X i; Y X 1,..., X i 1 ) Proof: The chain rule for utual inforation is a sile consequence of the chain rule for entroy. We have I((X 1,..., X ); Y) = H(X 1,..., X ) H(X 1,..., X Y) = = = H(X i X 1,..., X i 1 ) H(X i Y, X 1,..., X i 1 ) [H(X i X 1,..., X i 1 ) H(X i Y, X 1,..., X i 1 )] I(X i ; Y X 1,..., X i 1 ) References [Gal14] [Gol73] David Galvin, Three tutorial lectures on entroy and counting, arxiv rerint arxiv: (2014). 2 Larry J Goldstein, A history of the rie nuber theore, The Aerican Matheatical Monthly 80 (1973), no. 6, [Kon07] Ioannis Kontoyiannis, Soe inforation-theoretic coutations related to the distribution of rie nubers, arxiv rerint arxiv: (2007). 3 [Kon08], Counting the ries using entroy, IEEE Inforation Theory Society Newsletter (2008),

6 [Rad03] Jaikuar Radhakrishnan, Entroy and counting, Coutational atheatics, odelling and algoriths 146 (2003). 2 6