EE 4TM4: Digital Communications II Information Measures

Transcription

1 EE 4TM4: Digital Commuicatios II Iformatio Measures Defiitio : The etropy H(X) of a discrete radom variable X is defied by We also write H(p) for the above quatity. Lemma : H(X) 0. H(X) = x X Proof: 0 p(x) implies log(/p(x)) 0. p(x) log p(x). Defiitio 2: The relative etropy or Kullback Leibler distace betwee two probability mass fuctios p(x) ad q(x) is defied as D(p q) = x X p(x)log p(x) q(x). Theorem : Jese s iequality: If f is a covex fuctio ad X is a radom variable, the Ef(X) f(ex). () Moreover, if f is strictly covex, the equality i () implies that X = EX with probability, i.e., X is a costat. Theorem 2: Let p(x), q(x), x X, be two probability mass fuctios. The with equality if ad oly if p(x) = q(x) for all x. D(p q) 0 Proof: Let A = {x : p(x) > 0} be the support set of p(x). The D(p q) = x A x A p(x)log p(x) q(x) p(x)log q(x) p(x) log x A = log x Aq(x) p(x) q(x) p(x) log x X q(x) = log = 0,

2 2 where the first iequality follows from Jeses iequality. Sice log t is a strictly cocave fuctio of t, we have the first iequality becomes equality if ad oly if q(x)/p(x) = everywhere, i.e., p(x) = q(x). Hece we have D(p q) = 0 if ad oly if p(x) = q(x) for all x. Defiitio 3: The type P x (or empirical probability distributio) of a sequece x,x 2,,x is the relative proportio of occurreces of each symbol of X, i.e., P x (a) = N(a x )/ for all a X, where N(a x ) is the umber of times the symbol a occurs i the sequece x X. Defiitio 4: Let P deote the set of types with deomiator. Defiitio 5: If P P, the the set of sequeces of legth ad type P is called the type class of P, deoted T(P), i.e., T(P) = {x X : P x = P}. Theorem 3: P = ( + X ) X (+) X. Theorem 4: If X,X 2,,X are draw i.i.d. accordig to Q(x), the the probability of x depeds oly o its type ad is give by Q (x ) = 2 (H(P x )+D(P x Q)). Proof: Q (x ) = Q(x i ) i= = a X Q(a) N(a x ) = a X Q(a) P x (a) = a X 2 P x (a)logq(a) = a X = 2 2 (P x (a)logq(a) P x (a)logp x (a)+p x (a)logp x (a)) a X ( P x (a)log P x (a) Q(a) +P x (a)logp x (a)) = 2 ( D(P x Q) H(P x )). Corollary : If x is i the type class of Q, the Theorem 5: For ay type P P, Q (x ) = 2 H(Q). (+) X 2H(P) T(P) 2 H(P). Proof: Note that the exact size of T(P) is give by ( )! T(P) = = P(a ),P(a 2 ),,P(a X ) (P(a ))!(P(a 2 ))! (P(a X ))!

3 3 We first prove the upper boud. Sice a type class must have probability, we have P (T(P)) = x T(P) P (x ) = x T(P) 2 H(P) = T(P) 2 H(P). Thus T(P) 2 H(P). Now for the lower boud. We first prove that the type class T(P) has the highest probability amog all type classes uder the probability distributio P, i.e., P (T(P)) P (T(ˆP)), for all ˆP P. We lower boud the ratio of probabilities, P (T(P)) P (T(ˆP)) = T(P) T(ˆP) a X = a X P(a)P(a) P(a) ˆP(a) ( P(a ),P(a 2),,P(a X ) ( ˆP(a ),ˆP(a 2),,ˆP(a X ) = a X ) ) (ˆP(a))! (P(a))! P(a)(P(a) ˆP(a)). a X P(a)P(a) a X P(a) ˆP(a) Now usig the simple boud (easy to prove by separately cosiderig the cases m ad m < ) m!! m, we obtai P (T(P)) P (T(ˆP)) a X = a X (P(a)) ˆP(a) P(a) P(a) (P(a) ˆP(a)) ( ˆP(a) P(a)) = ( ˆP(a) a X a X P(a)) = ( ) =. Hece P (T(P)) P (T(ˆP)). The lower boud ow follows easily from this result, sice = P (T(Q)) Q P Q P max Q P (T(Q)) = P (T(P)) Q P (+) X P (T(P)) = (+) X P (x ) x T(P) = (+) X x T(P) 2 H(P) = (+) X T(P) 2 H(P).

4 4 Theorem 6: For ay P P ad ay distributio Q, the probability of the type class T(P) uder Q is 2 D(P Q) to first order i the expoet. More precisely, Proof: We have Usig the bouds o T(P), we have + X 2 D(P Q) Q (T(P)) 2 D(P Q). Q (T(P)) = x T(P) = x T(P) Q (x ) 2 (D(P Q)+H(P)) = T(P) 2 (D(P Q)+H(P)). (+) X 2 D(P Q) Q (T(P)) 2 D(P Q). Theorem 7: Let X,X 2,,X be i.id. Q(x). The Pr{D(P x Q) > } 2 ( X log(+) ). Proof: Pr{D(P x Q) > } = Q (T(P)) P:D(P Q)> P:D(P Q)> P:D(P Q)> 2 D(P Q) 2 (+) X 2 = 2 ( X log(+) ). Theorem 8: Saov s theorem: Let X,X 2,,X be i.i.d. Q(x). Let E P be a set of probability distributios. The Q (E) = Q (E P ) (+) X 2 D(P Q), where P = argmid(p Q) is the distributio i E that is closest to Q i relative etropy. If, i additio, the P E set E is the closure of its iterior, the logq (E) D(P Q).

5 5 Proof: We first prove the upper boud: Q (E) = Q (T(P)) 2 D(P Q) max 2 D(P Q) = 2 mip E P D(P Q) 2 mip E D(P Q) 2 D(P Q) = (+) X 2 D(P Q). Note that P eed ot be a member of P. We ow come to the lower boud, for which we eed a ice set E, so that for all large, we ca fid a distributio i E P which is close to P. If we ow assume that E is the closure of its iterior (thus the iterior must be o-empty), the sice P is dese i the set of all distributios, it follows that E P is o-empty for all 0 for some 0. We ca the fid a sequece of distributios P such that P E P ad D(P Q) D(P Q). For each 0, Q (E) = Q (T(P)) Cosequetly, Q (T(P )) (+) X 2 D(P Q). limif logq (E) limif( X log(+) Combiig this with the upper boud establishes the theorem. We ca summarize the basic theorems cocerig types i four equatios: P (+) X, Q (x ) = 2 (D(P x Q)+H(P x )), T(P) 2 H(P), Q (T(P)) 2 D(P Q). D(P Q)) = D(P Q). Theorem 9: Weak Law of Large Numbers: Let X,X 2, be a sequece of idepedet radom variables havig a commo distributio Q, ad let E[X i ] = µ. The for ay > 0 { P X +X 2 + +X } µ 0 as

6 6 Proof: Let Y = X+X2+ +X by Chebyshev s iequality µ. Note that E[Y ] = 0 ad Var(Y ) = σ2, where σ2 = Var(X i ). Therefore, P{ Y } σ2 2 0 as Remark: This proof shows that the probability goes to zero at least as fast as. I fact, it is possible to derive a tighter boud. Without loss of geerality, we assume E[X i ] = 0. Note that P{ Y } = P{ Y 4 4 } E[Y 4 ] 4. Sice E(X +X 2 + +X ) 4 = E(X 4 )+3( )(E(X 2 )) 2, it follows that the probability goes to zero at least as fast as 2. by Remark: Accordig to Saov s theorem, the probability goes to zero expoetially fast with the expoet give mi P: x xp(x) µ D(P Q). Theorem 0: Cetral Limit Theorem: Let X,X 2, be a sequece of idepedet, idetically distributed radom variables, each with mea µ ad variace σ 2. The the distributio of X +X 2 + +X µ σ teds to the stadard ormal as. That is, { X +X 2 + +X µ P σ as. } a a e x2 /2 dx 2π Coectio betwee the cetral limit theorem ad the divergece: LetX,X 2, be a sequece of i.i.d. Beroulli radom variables with parameter p, i.e., P(X i = ) = p ad P(X i = 0) = p. Note that E[X i ] = p ad Var(X i ) = E[Xi 2] E[X i] 2 = p p 2 = p( p). Accordig to the cetral limit theorem, { X +X 2 + +X p } P [a,b] b e x2 /2 dx. p( p) 2π Note that here we are essetially coutig the total probability of type classes such that X +X 2 + +X [p+a p( p),p+b p( p) ], i.e., the empirical p x [p+a p( p)/,p+b p( p)/ ]. Now cosider the Taylor expasio of D(q p) at the eighborhood of p. Note that So which equals 0 whe q = p, ad D(q p) = qlog q p +( q)log q p. d dq D(q p) = log q q log p p, d 2 dq 2D(q p) = q( q), a

7 7 which equals p( p) whe q = p. Therefore, for q = p+x p( p)/, we have D(q p) 2p( p) (x p( p)/ ) 2 = x2 2. Cosequetly, the probability of the type class with empirical p x p+x p( p)/ is approximately e x2 2 = e x 2 /2, which gives the right expoet i the cetral limit theorem. Note that i the limit, oe ca replace the sum of the probabilities of type classes by itegral to recover the cetral limit theorem (the costat factor / 2π requires a more accurate approximatio usig Stirlig s formula). For the o-biary case, oe ca also establish a coectio betwee divergece ad the multi-dimesioal cetral limit theorem. Theorem : H(X) log X, where X deotes the umber of elemets i the rage of X, with equality if ad oly if X has a uiform distributio over X. Proof: Let u(x) = X be the uiform probability mass fuctio over X, ad let p(x) be the probability mass fuctio for X. The Hece by the o-egativity of relative etropy, D(p u) = p(x)log p(x) u(x) = log X H(X). 0 D(p u) = log X H(X). This result ca also be proved usig Lagragia multiplier. Coditioal etropy: H(Y X) = x X,y Yp(x,y)logp(y x) = x Xp(x) y Yp(y x)logp(y x) = x X Coditioal etropy: residual ucertaity Joit etropy: H(X,Y) = x X,y Y p(x,y)logp(x,y). We have H(X,Y) = H(X)+H(Y X) = H(Y)+H(X Y). More geerally, H(X ) = H(X )+H(X 2 X )+ +H(X X,,X ). p(x)h(y X = x). Mutual iformatio: I(X;Y) = (x,y) X Y p(x,y)log p(x,y) p(x)p(y) = H(X) H(X Y) = H(Y) H(Y X) = H(X)+H(Y) H(X,Y).

8 8 Note that I(X;Y) = D(p(x,y) p(x)p(y)). Therefore, I(X;Y) 0 with equality if ad oly if X ad Y are idepedet. Sice I(X;Y) 0, it follows that H(Y X) H(Y) H(X,Y) H(X)+H(Y). Moreover,H(g(X)) H(X) with equality ifg is oe-to-oe over the support ofx, i.e., the set{x X : p(x) > 0}. This is because H(X,g(X)) = H(g(X) X)+ H(X) = H(X) ad H(X,g(X)) = H(g(X)) + H(X g(x)) H(g(X)). Theorem 2: Fao s iequality: Suppose we wish to estimate a radom variable X with a distributio p(x). We observe a radom variable Y which is related to X by the coditioal distributio p(x y). From Y, we calculate a fuctio g(y) = ˆX, which is a estimate of X. We wish to boud the probability that ˆX X. We observe that X Y ˆX forms a Markov chai. Defie the probability of error P e = P{ ˆX X}. The H(P e )+P e log( X ) H(X Y), where H(P e ) = P e logp e ( P e )log( P e ). This iequality ca be weakeed to +P e log X H(X Y). Proof: Let E = if ˆX X ad E = 0 if ˆX = X. Note that H(E,X Y) = H(X Y)+H(E X,Y) = H(X Y). O the other had, H(E,X Y) = H(E Y)+H(X E,Y) H(E)+H(X E,Y) = H(P e )+H(X E,Y) = H(P e )+P(E = 0)H(X Y,E = 0)+P(E = )H(X Y,E = ) H(P e )+( P e )0+P e log( X ). Coditioal mutual iformatio: I(X;Y Z) = z Z p(z)i(x;y Z = z) = z Zp(z) x X,y Y p(x, y z) log p(x, y z) p(x z)p(y z).

9 9 Note that I(X;Y Z) = z Zp(z)D(p(x,y z) p(x z)p(y z)) = D(p(x,y z) p(x z)p(y z) p(z)). Also ote that I(X;Y Z) = H(X Z) H(X Y,Z) = H(Y Z) H(Y X,Z) = H(X Z)+H(Y Z) H(X,Y Z). The coditioal mutual iformatio I(X;Y Z) is oegative ad is equal to zero if ad oly if X ad Y are coditioally idepedet give Z, i.e., X Z Y form a Markov chai. Theorem 3: Data processig iequality: If X Y Z form a Markov chai, the I(X;Z) I(X;Y). Cosequetly, for ay fuctio g, I(X;g(Y)) I(X;Y). Proof: To prove the data processig iequality, we use the chai rule to expad I(X;Y,Z) i two ways as I(X;Y,Z) = I(X;Y)+I(X;Z Y) = I(X;Y) = I(X;Z)+I(X;Y Z) I(X;Z). Note that ulike etropy, o geeral iequality relatioship exists betwee the coditioal mutual iformatio I(X;Y Z) ad the mutual iformatio I(X;Y). There are, however two importat special cases. If X ad Z are idepedet, the This is because I(X;Y Z) = I(X;Y,Z) I(X;Y). If Z X Y form a Markov chai, the This is because I(X;Y Z) I(Z,X;Y) = I(X;Y). Typical Sequeces I(X;Y Z) I(X;Y). I(X;Y Z) I(X;Y). Let X,X 2, be a sequece of idepedet ad idetically distributed radom variables. The by the (weak) law of large umbers, for each x X, N(x x ) p(x) i probability. Thus, with high probability, the radom empirical pmf N(x X ) does ot deviate much from the true pmf p(x). For X p(x) ad (0,), defie the set of -typical -sequeces x (or the typical set i short) as T () (X) = {x : N(x x ) p(x) p(x) for all x X}.

10 0 Note that the typical set ca be viewed as the uio of type classes whose type is close to p(x). Also ote that p(x) = 0 implies N(x x ) = 0 because otherwise D(p x p) =. Lemma 2: Typical Average Lemma: Let x T () (X). The for ay oegative fuctio g(x) o X, ( )E(g(X)) g(x i ) (+)E(g(X)). Typical sequeces satisfy the followig properties: i= ) Let p(x ) = i= p X(x i ). The for each x T (X), 2 (H(X)+δ()) p(x ) 2 (H(X) δ()), where δ() = H(X). This follows by the typical average lemma with g(x) = logp(x). 2) The cardiality of the typical set is upper bouded as T () 2 (H(X)+δ()). This ca be show by summig the lower boud i property over the typical set. 3) If X,X 2, are i.i.d. with X i p X (x i ), the by the LLN, lim P{X T () } =. This result ca also be proved usig Theorem 7 i Lecture. Note that the size of the typical set is i geeral egligible compared with X. However, it captures almost all probability. 4) The cardiality of the typical set is lower bouded as T () ( )2 (H(X) δ()) for sufficietly large. This follows by property 3 ad the upper boud i property. Explai these properties from the perspective of method of types. Joitly Typical Sequeces Let (X,Y) p X,Y (x,y). The set of joitly -typical -sequeces is defied as { T () (X,Y) = (x,y ) : N(x,y x,y ) p X,Y (x,y) p X,Y(x,y) for all (x,y) X Y { Also defie the set of coditioally -typical sequeces as T () (Y x ) = y : (x,y ) T () properties of typical sequeces ca be exteded to joitly typical sequeces as follows. ) Let (x,y ) T () (X,Y) ad p(x,y ) = i= p X,Y(x i,y i ). The (a) x T () (X) ad y T () (Y), (b) 2 (+)H(X) p(x ) 2 ( )H(X) ad 2 (+)H(Y) p(y ) 2 ( )H(Y), }. } (X,Y). The (c) 2 (+)H(X Y) p(x y ) 2 ( )H(X Y) ad 2 (+)H(Y X) p(y x ) 2 ( )H(Y X), (d) 2 (+)H(X,Y) p(x,y ) 2 ( )H(X,Y ). 2) T () (X,Y) 2 (+)H(X,Y). 3) If p(x,y ) = i= p X,Y(x i,y i ), the lim P{(X,Y ) T () (X,Y)} =.

11 4) T () (X,Y) ( )2 ( )H(X,Y ) for sufficietly large. 5) For every x X, we have T () (Y x ) 2 (+)H(Y X). 6) Let X p X (x) ad Y = g(x). Let x T () i [ : ]. The followig property deserves special attetio. (X). The y T () (Y x ) if ad oly if y i = g(x i ) for Lemma 3: Coditioal Typicality Lemma: Let (X,Y) p(x,y). Suppose that x T () (X) ad Y p(y x ) = i= p Y X(y i x i ). The, for every >, The proof of this lemma follows by the LLN. lim P{(x,Y ) T () (X,Y)} =. The coditioal typicality lemma implies the followig additioal property of joitly typical sequeces. 5) If x T () (X) ad <, the for sufficietly large, Joit Typicality for a Triple of Radom Variables T () (Y x ) ( )2 ( )H(X Y). Let (X,Y,Z) p(x,y,z). The set of joitly -typical (x,y,z ) sequeces is defied as T () (X,Y,Z) = {(x,y,z ) : N(x,y,z x,y,z ) p(x,y,z) p(x,y,z) for all (x,y,z) X Y Z}. Suppose that (x,y,z ) T () (X,Y,Z) ad p(x,y,z ) = i= p X,Y,Z(x i,y i,z i ). The ) x T () ad (y,z ) T () (Y,Z), 2) p(x,y,z ) 2 H(X,Y,Z), 3) p(x,y z ) 2 H(X,Y Z), 4) T () (X y,z ) 2 (H(X Y,Z)+δ()), ad 5) if (y,z ) T () (Y,Z) ad <, the for sufficietly large, T () (X y,z ) 2 (H(X Y,Z) δ()). The followig two-part lemma will be used i may achievability proofs of codig theorem. Lemma 4: Joit Typicality Lemma: Let (X,Y,Z) p(x,y,z) ad <. The there exists δ() > 0 that teds to zero as 0 such that the followig statemets hold: ) If ( x,ỹ ) is a pair of arbitrary sequeces ad Z i= p Z X( z i x i ), the 2) If (x,y ) T () P{( x,ỹ, Z ) T () (X,Y,Z)} 2 (I(Y;Z X) δ()). ad Z i= p Z X( z i x i ), the for sufficietly large, P{(x,y, Z ) T () (X,Y,Z)} 2 (I(Y;Z X)+δ()).

12 2 Proof: To prove the first statemet, cosider P{( x,ỹ, Z T () (X,Y,Z))} = Similarly, for every sufficietly large, P{(x,y, Z ) T () which proves the secod statemet. z T () (Z x,ỹ ) p( z x ) T () (Z x,ỹ ) 2 (H(Z X) H(Z X)) 2 (H(Z X,Y )+H(Z X,Y )) 2 (H(Z X) H(Z X)) = 2 (I(Y;Z X) δ()). (X,Y,Z)} T () (Z x,y ) 2 (H(Z X)+H(Z X)) ( )2 (H(Z X,Y ) H(Z X,Y)) 2 (H(Z X)+H(Z X)) = 2 (I(Y;Z X)+δ()), REFERENCES [] T. Cover ad J. A. Thomas, Elemets of Iformatio Theory, 2d Editio. Hoboke, NJ: Wiley, [2] A. El Gamal ad Y.-H. Kim, Network Iformatio Theory. Cambridge, U.K.: Cambridge Uiversity Press, 20.