Chapter 8: Differential entropy. University of Illinois at Chicago ECE 534, Natasha Devroye

Transcription

1 Chapter 8: Differential entropy

2 Chapter 8 outline Motivation Definitions Relation to discrete entropy Joint and conditional differential entropy Relative entropy and mutual information Properties AEP for Continuous Random Variables

3 Motivation Our goal is to determine the capacity of an AWGN channel N Gaussian noise ~ N(0,PN) h X Wireless channel with fading Y = h X + N time time

4 Motivation Our goal is to determine the capacity of an AWGN channel N Gaussian noise ~ N(0,PN) h X Wireless channel with fading Y = h X + N C = 1 2 log = 1 2 h 2 P +P N P N log (1 + SNR) (bits/channel use)

5 Motivation need to define entropy, mutual information between CONTINUOUS random variables Can you guess? Discrete X, p(x): Continuous X, f(x):

6 Definitions - densities

7 Properties - densities

8 Properties - densities

9 Properties - densities

10 Quantized random variables = + f(x) FIGURE 8.1. Quantization of a continuous random variable. x

11 Quantized random variables = + = + f(x) f(x) x x FIGURE 8.1. Quantization of a continuous random variable. FIGURE 8.1. Quantization of a continuous random variable.

12 Quantized random variables = + = + f(x) f(x) x x FIGURE 8.1. Quantization of a continuous random variable. FIGURE 8.1. Quantization of a continuous random variable.

13 Differential entropy - definition

14 Examples f(x) a b x

15 Examples

16 Differential entropy - the good the bad and the ugly

17 Differential entropy - the good the bad and the ugly

18 Differential entropy - multiple RVs

19 Differential entropy of a multi-variate Gaussian

20 Parallels with discrete entropy... SUMMARY Definition defined by The entropy H(X) of a discrete random variable X is Properties of H H(X) = x Xp(x) log p(x). (2.156) 1. H(X) H b (X) = (log b a)h a (X). 3. (Conditioning reduces entropy) For any two random variables, X and Y, we have H(X Y) H(X) (2.157) with equality if and only if X and Y are independent. 4. H(X 1,X 2,...,X n ) n i=1 H(X i), with equality if and only if the X i are independent. 5. H(X) log X, with equality if and only if X is distributed uniformly over X. 6. H(p) is concave in p

21 Parallels with discrete entropy... Definition The relative entropy D(p q) of the probability mass function p with respect to the probability mass function q is defined by D(p q) = p(x) p(x) log x q(x). (2.158) Definition The mutual information between two random variables X and Y is defined as I(X; Y) = x X Alternative expressions p(x, y) log y Y H(X)= E p log H(X,Y)= E p log H(X Y)= E p log I(X; Y)= E p log p(x, y) p(x)p(y). (2.159) 1 p(x), (2.160) 1 p(x, Y), (2.161) 1 p(x Y), (2.162) p(x, Y) p(x)p(y), (2.163) D(p q) = E p log p(x) q(x). (2.164) Relative entropy: D(p(x, y) q(x,y)) = D(p(x) q(x)) + D(p(y x) q(y x))

22 Parallels with discrete entropy... Properties of D and I = 1. I(X; Y) = H(X) H(X Y) = H(Y) H(Y X) = H(X)+ H(Y) H(X,Y). 2. D(p q) 0 with equality if and only if p(x) = q(x), for all x X. 3. I(X; Y) = D(p(x, y) p(x)p(y)) 0, with equality if and only if p(x, y) = p(x)p(y) (i.e., X and Y are independent). 4. If X = m, and u is the uniform distribution over X, then D(p u) = log m H(p). 5. D(p q) is convex in the pair (p, q). Chain rules Entropy: H(X 1,X 2,...,X n ) = n i=1 H(X i X i 1,...,X 1 ). Mutual information: I(X 1,X 2,...,X n ; Y) = n i=1 I(X i; Y X 1,X 2,...,X i 1 )

23 Parallels with discrete entropy... = + Jensen s inequality. If f is a convex function, then Ef (X) f(ex). Log sum inequality. For n positive numbers, a 1,a 2,...,a n and b 1,b 2,...,b n, ( n n ) a i log a n i i=1 a i log a i b n i i=1 b (2.165) i i=1 i=1... with equality if and only if a i b i = constant.... Data-processing inequality. If X Y Z forms a Markov chain, I(X; Y) I(X; Z). Sufficient statistic. T(X) is sufficient relative to {f θ (x)} if and only if I(θ; X) = I(θ; T(X)) for all distributions on θ Fano s inequality. Let P e = Pr{ ˆX(Y) X}. Then H(P e ) + P e log X H(X Y). (2.166)

24 Differential entropy - the good the bad and the ugly

25 Relative entropy and mutual information

26 Properties

27 A quick example Find the mutual information between the correlated Gaussian random variables with correlation coefficient ρ What is I(X;Y)?

28 More properties of differential entropy

29 More properties of differential entropy

30 Examples of changes in variables

31 Concavity and convexity Same as in the discrete entropy and mutual information...

32 The AEP for continuous RVs The AEP for discrete RVs said... Theorem (AEP) If X 1,X 2,... are i.i.d. p(x), then 1 n log p(x 1,X 2,...,X n ) H(X) in probability. (3.2) Proof: Functions of independent random variables are also independent The AEP for continuous RVs says...

33 Typical sets One of the points of the AEP is to define typical sets. Typical set for discrete RVs... Typical set of continuous RVs...

34 Typical sets and volumes

35 Maximum entropy distributions For a discrete random variable taking on K values, what distribution maximized the entropy? Can you think of a continuous counter-part? [Look ahead to Ch.12, pg ]

36 Maximum entropy distributions [Look ahead to Ch.12, pg ]

37 Maximum entropy examples Prove 2 ways!

38 Maximum entropy examples Prove 2 ways!

39 Estimation error and differential entropy A counter part to Fano s inequality for discrete RVs... = { } Theorem (Fano s Inequality) For any estimator ˆX such that X Y ˆX, with P e = Pr(X ˆX), we have H(P e ) + P e log X H(X ˆX) H(X Y). (2.130) This inequality can be weakened to or 1 + P e log X H(X Y) (2.131) Why can t we use Fano s? P e H(X Y) 1. (2.132) log X

40 Estimation error and differential entropy

41 Summary SUMMARY h(x) = h(f ) = S f(x)log f(x)dx (8.81) f(x n ). =2 nh(x) (8.82) Vol(A (n) ϵ ). =2 nh(x). (8.83) H ([X] 2 n) h(x) + n. (8.84) h(n(0, σ 2 )) = 1 2 log 2πeσ 2. (8.85) h(n n (µ, K)) = 1 2 log(2πe)n K. (8.86) D(f g) = f log f 0. (8.87) g h(x 1,X 2,...,X n ) = n h(x i X 1,X 2,...,X i 1 ). (8.88) i=1

42 Summary = i=1 max EXX t =K h(x Y) h(x). (8.89) h(ax) = h(x) + log a. (8.90) I(X; Y) = f(x,y)log f(x,y) 0. (8.91) f(x)f(y) h(x) = 1 2 log(2πe)n K. (8.92) E(X ˆX(Y)) 2 1 2πe e2h(x Y). 2 nh (X) is the effective alphabet size for a discrete random variable. 2 nh(x) is the effective support set size for a continuous random variable. 2 C is the effective alphabet size of a channel of capacity C.