Information Theory and Statistics, Part I

Transcription

1 Information Theory and Statistics, Part I Information Theory 2013 Lecture 6 George Mathai May 16, 2013

2 Outline This lecture will cover Method of Types. Law of Large Numbers. Universal Source Coding. Large Deviation Theory. Examples of Sanov s Theorem. All illustrations are borrowed from the book.

3 Method of Types Definition : The type P x of a sequence x 1, x 2... x n is the relative proportion of the occurrences of each symbol of X P n x (a) = N(a x n ) n Ex: X ={1,2,3}. Let x= then P x (1) = 3/5, P x (2) = 1/5, P x (3) = 1/5. Hence P x = {3/5, 1/5, 1/5} Let P n denotes the set of types with denominator n. Ex: P 5 ={(0/5, 0/5, 5/5), (0/5, 1/5, 4/5)..(0/5, 5/5, 0/5)..(5/5, 0/5, 0/5)} If P P n, then the set of sequences of length n and type P is called the type class of P, denoted by T (P) T (P) = {x X n : P x = P} Ex: T (P x ) = {11123, 11132, 11213, }

4 Method of Types Theorem: P n (n + 1) X This bound is very high. One can achieve much better than this

5 Method of Types Theorem: If X 1, X 2,... X n are drawn IID according to Q(x), the probability of x depends only on its type and is given by Proof: Q n (x n n(h(px )+D(Px Q)) ) = 2 n Q n (x n ) = Q(x i ) i=1 = Q(a) N(a x n ) a X = Q(a) np x n (a) a X = a X 2 np xn (a) log Q(a) n(h(px )+D(Px Q)) = 2 Corollary : If x n is in the type class of Q, then Q n (x) = 2 nh(q)

6 Method of Types Theorem: Size of a type class T (P) For any type P P 1 (n + 1) X 2nH(P) T (P) 2 nh(p) where the exact value of T (P) is given by ( ) n T (P) = np(a1), np(a2),... np(a X ) So we look for the bounds. Upper Bound 1 P n (T (P)) = P n (x n ) x n T (P) = 2 nh(p) = T (P) 2 nh(p) x n T (P)

7 Method of Types Theorem:(Probability of type class) for any P P n and any distribution Q, the probability of the type class T (P) under Q n is 2 nd(p Q) to first order in the exponent. More precisely, Proof: 1 (n + 1) X 2 nd(p Q) Q n (T (P)) 2 nd(p Q) Q n (T (P)) = x n T (P) = x n T (P) Q n (x n ) n(h(px )+D(Px Q)) 2 n(h(px )+D(Px Q)) = T (P) 2 Using the bounds we found on T (P) we get 1 (n + 1) X 2 nd(p Q) Q n (T (P)) 2 nd(p Q)

8 Law of Large Numbers Given ɛ > 0 we can define a typical set TQ ɛ distribution Q n as of sequences for the T ɛ Q = {x n : D(P x n Q) ɛ} Then the probability that x n is not typical is 1 Q n (T ɛ Q) = Q n (T (P)) P:D(P Q)>ɛ 2 nd(p Q) P:D(P Q)>ɛ 2 nɛ P:D(P Q)>ɛ (n + 1) X 2 nɛ log(n+1) n(ɛ X ) = 2 n As n, then 1 Q n (T ɛ Q ) 0 = Pr(T ɛ Q ) 1

9 Law of Large Numbers Theorem: LetX 1, X 2... X n be i.i.d. P(x). Then log(n+1) n(ɛ X ) Pr{D(P x n P) > ɛ} 2 n and consequently, D(P x n P) 0with probability 1

10 Law of Large Numbers Definition: Strongly typical set A ɛ(n) Aɛ (n) = x n X n : 1 ɛ N(a x) P(a) n X ifp(a) 0 N(a x) = 0 Typical sets consists of sequences whose types does not differ from the true probabilities by more than ɛ X

11 Universal Source Coding Definition: A fixed - rate block code of rate R for source X 1, X 2... X n which has an unknown distribution Qconsists of two mapping: the encoder, and the decoder f n : X n {1, 2,... 2 nr } φ n : {1, 2,... 2 nr } X n where R is called the rate of the code. Probability of error for the code wrt distribution P (n) e = Q n (X n : φ n (f n (X n )) X n )

12 Universal Source Coding Definition: A rate R block code for a source will be called universal if the functions f n and φ n don t depend on the distribution Q and if P e (n) 0 as n if R > H(Q) Theorem: There exists a sequence of (2 nr, n) universal source codes such that P e (n) 0 for every source Q such that H(Q) < R

13 Universal Source Coding Proof: Let Consider the sequence log(n + 1) R n = R X n A = {x n X n : H(P x ) R n } Then A = T (P) P P n:h(p) R n 2 nh(p) P P n:h(p) R n 2 nrn P P n:h(p) R n (n + 1) X 2 nrn = 2 log(n+1) n(rn+ X n ) = 2 nr

14 Universal Source Coding Probability of decoding error P e (n) can be found by P (n) e = 1 Q n (A) = Q n (T (P)) P P n:h(p) R n (n + 1) X max P:H(P>R Qn (T (P)) n) (n + 1) X 2 n min P:H(P>Rn) D(P Q) As n, P (n) e 0

15 Large Deviation Theory Theory of large deviations concerns the asymptotic behaviour of remote tails of sequences of probability distributions. Let E be a subset of the set of probability mass functions. Q n (E) = Q n (E P n ) = x n :P x n E P n Q n (x n ) Q n (E) 1 If E contains relative entropy neighbourhood of Q Q n (E) 0 other wise

16 Large Deviation Theory Sanov s Theorem: Let X 1, X 2... X n be i.i.d. Q(x). Let E P be a set of probability distributions. Then where Q n (E) = Q n (E P n ) (n + 1) X 2 nd(p Q) P = arg min P E D(P Q) is the distribution E that is closest to Q in relative Entropy. If in addition, the set E is the closure of its interior, then Note: E is not in the typical set 1 n log Qn (E) D(P Q)

17 Large Deviation Theory Proof: Q n (E) = Q n (T (P)) P E P n 2 nd(p Q) P E P n max 2 nd(p Q) P E P n P E P n = 2 min P E Pn nd(p Q) P E P n 2 min P E nd(p Q) P E P n = P E P n 2 nd(p Q) (n + 1) X 2 nd(p Q)

18 Examples of Sanov s Theorem Task: Pr{ 1 n Set E is defined as n g j (X i ) α j, j = 1, 2,... k} i=1 E = {P : a P(a)g j (a) α j, j = 1, 2,..., k} To find te closest distribution in Eto Q. We minimize the D(P Q) subject tot he above constraint. The resulting functional J(P) = x P(x) log P(x) Q(x) + i Differentiating and we get P (x) = λ i P(x)g i (x) + ν x x Q(x)e i λ i g i (x) a X Q(a)e i λ i g i (x) P(x)