Asymptotic Equipartition Property - Seminar 3, part 1

Transcription

1 Asympoic Equipariion Propery - Seminar 3, par 1 Ocober 22, 2013 Problem 1 (Calculaion of ypical se) To clarify he noion of a ypical se A (n) ε and he smalles se of high probabiliy B (n), we will calculae he se for a simple example. Consider a sequence of i.i.d. binary random variables, X 1, X 2,..., X n, where he probabiliy ha X i = 1 is 0.6 (and herefore he probabiliy ha X i = 0 is 0.4). (a) Calculae H(X). (b) Wih n = 25 and ε = 0.1, which sequences fall in he ypical se A(n)? Wha is he probabiliy of he ypical se? How many elemens are here in he ypical se? (This involves compuaion of a able of probabiliies for sequences wih k 1 s, 0 k 25, and finding hose sequences ha are in he ypical se.) (c) How many elemens are here in he smalles se ha has probabiliy 0.9? (d) How many elemens are here in he inersecion of he ses in pars (b) and (c)? Wha is he probabiliy of his inersecion? Soluion. X : ( ) (a) H(x) = 0.4 log log = bis. (b) ε = 0.1, [H(X) ε, H(X) + ε] = [ , ]. A (n) ε = {x {0, 1} n : H(X) ε 1 n log p(xn ) H(X) + ε}, We generae a able of probabiliies using he following MATLAB code %Bernoulli AEP p=0.6; epsilon=0.1; 1

2 q=1-p; n=25; k=(0:n) ; Hx=enropy([p,q]); li=hx-epsilon; ls=hx+epsilon; for i=0:n combi(i+1)=nchoosek(n,i); end M=[k,combi,binocdf(k,n,p),-1/n*log2(p.^k.*q.^(n-k))]; The oupu able is e e e e e e e e e e e e e e e e e e e e The forh column conains - 1 n log p(xn ). The values wihin he range [ , ] are for 11 k 19 i=find(m(:,4)>=li & M(:,4)<=ls); M M(i,:) wih oupu 2

3 e e e e e e e e e The probabiliy ha he number of 1 s lies beween 11 and 19 is equal o F (19) F (10) = = Noe ha his is greaer han 1 ε, i.e., he n is large enough for he probabiliy of he ypical se o be greaer han 1 ε. The number of elemens in he ypical se can be found using he hird column. A ε (n) 19 ( ) 25 = = k k=11 (c) To find he smalles se B (n) of probabiliy 0.9, we can imagine ha we are filling a bag wih pieces such ha we wan o reach a cerain weigh wih he minimum number of pieces. To minimize he number of pieces ha we use, we should use he larges possible pieces. In his case, i corresponds o using he sequences wih he highes probabiliy. Thus we keep puing he high probabiliy sequences ino his se unil we reach a oal probabiliy of 0.9. Looking a he fourh column of he able, i is clear ha he probabiliy of a sequence increases monoonically wih k. Thus he se consiss of sequences of k = 25, 24;..., unil we have a oal probabiliy 0.9. Using he cumulaive probabiliy column, i follows ha he se B (n) consis of sequences wih k 13 and some sequences wih k = 12. The sequences wih k 13 provide a oal probabiliy of = o he se B (n). The remaining probabiliy of = should come from sequences wihk = 12. The number of such sequences needed o fill his probabiliy is a leas = p(x n ) = / = , which we round up o Thus he smalles se wih probabiliy 0.9 has = sequences. Noe ha he se B (n) is no uniquely defined - i could include any sequences wih k = 12. However, he size of he smalles se is a well defined number. (d) The inersecion of he ses A ε (n) and B (n) in pars (b) and (c) consiss of all sequences wih k beween 13 and 19, and sequences wih k = 12. The probabiliy of his inersecion=

4 = , and he size of his inersecion = = Problem 2 (Markov s inequaliy and Chebyshev s inequaliy) (a) (Markov s inequaliy.) For any non-negaive random variable X and any > 0, show ha P (X ) E(X). Exhibi a random variable ha achieves his inequaliy wih equaliy. (b) (Chebyshev s inequaliy.) Le Y be a random variable wih mean µ and variance σ 2. By leing X = (Y µ) 2, show ha for any ε > 0, Soluion. P ( Y µ > ε) σ2 ε 2. (a) E(X) = 0 xdf (x) = xdf (x) 0 xdf (x) + df (x) = P (X ). Rearranging sides and dividing by we ge, xdf (x) Example for = P (X ) E(X). X = { wih probabiliy µ 0 wih probabiliy 1 µ where µ. (b) In Markov inequaliy, ake X = (Y µ) 2. and noicing ha we ge ChebIn. P ((Y µ) 2 > ε 2) P E(Y µ)2 ε 2 = σ2 ε 2, ( (Y µ) 2 > ε 2) = P ( Y µ > ε) 4

5 Problem 3 (AEP and muual informaion) Le (X i, Y i ) be i.i.d. p(x, y). We form he log likelihood raio of he hypohesis ha X and Y are independen vs. he hypohesis ha X and Y are dependen. Wha is he limi of Soluion. Thus, p(xn )p(y n ) p(x n,y n ) independen. 1 n log p (Xn ) p (Y n ) p (X n, Y n ). 1 n log p (Xn ) p (Y n ) p (X n, Y n = 1 n ) n log p (X i ) p (Y i ) p (X i=1 i, Y i ) n log p (X i) p (Y i ) = 1 n E i=1 = p (X i, Y i ) ) = I(X, Y ). ( log p (X i) p (Y i ) p (X i, Y i ) 2 ni(x,y ), which will converge o 1 if X and Y are indeed Problem 4 (Piece of cake) A cake is sliced roughly in half, he larges piece being chosen each ime, he oher pieces discarded. We will assume ha a random cu creaes pieces of proporions: { ( 2 X = 3, ) ( 1 w.p , 5) 3 w.p. 1 4 Thus, for example, he firs cu (and choice of larges piece) may resul in a piece of size 3/5. Cuing and choosing from his piece migh reduce i o size (3/5)(2/3) a ime 2, and so on. How large, o firs order in he exponen, is he piece of cake afer n cus? Soluion. Le C i be he fracion of he piece of cake ha is cu a he ih cu, and le T n be he fracion of cake lef afer n cus. Then we have T n = C 1 C 2... C n = n i=1 C i. Hence lim 1 n log T n = 1 n n log C i = E(log C 1 ) i=1 = 3 4 log log 3 5. Problem 5 (The AEP and source coding) A discree memoryless source emis a sequence of saisically independen binary digis wih probabiliies p(1) = (0.005) and p(0) = The digis are aken 100 a a ime and a binary codeword is provided for every sequence of 100 digis conaining hree or fewer ones. 5

6 (a) Assuming ha all codewords are he same lengh, find he minimum lengh required o provide codewords for all sequences wih hree or fewer ones. (b) Calculae he probabiliy of observing a source sequence for which no codeword has been assigned. (c) Use Chebyshev s inequaliy o bound he probabiliy of observing a source sequence for which no codeword has been assigned. Compare his bound wih he acual probabiliy compued in par (b). Soluion. (a) The number of 100-bi binary sequences wih hree or fewer ones is ( ) ( ) ( ) ( ) = The required codeword lengh is log = 18. (Noe ha H(0.005) = , so 18 is quie a bi larger han he 4.5 bis of enropy.) (b) The probabiliy ha a 100-bi sequence has hree or fewer ones is 3 ( ) 100 (0.005) i (0.995) 100 i = i i=0 Thus he probabiliy ha he sequence ha is generaed canno be encoded is = (c) In he case of a random variable S n ha is he sum of n i.i.d. random variables X 1, X 2,..., X n, Chebyshev s inequaliy saes ha P ( S n nµ ε) nσ2 ε 2, where µ and σ 2 are he mean and he variance of X i, respecively. E(S n ) = nµ, V (S n ) = nσ 2. In his problem, n = 100, µ = 0.005, and σ 2 = (0.005)(0.995). Noe ha S if and only if S (0.005) 3.5, so we should choose ε = 3.5. Then P (S 100 4) 100(0.005)(0.995) (3.5) This bound is much larger han he acual probabiliy Problem 6 (Ses defined by probabiliies) Le X 1, X 2,... be an i.i.d. sequence of discree random variables wih enropy H(X). Le C n () = { x n X n : p(x n ) 2 n} denoe he subse of n-sequences wih probabiliies 2 n. 6

7 (a) Show C n () 2 n. (b) For wha values of does P (X n C n ()) 1? Proof. (a) Since he oal probabiliy of all sequences is less han 1, C n () min x n C n() p(xn ) 1 = C n () 2 n 1. (b) Since 1 n log p(xn ) H, if < H, he probabiliy ha p(x n ) > 2 n goes o 0, and if > H, he probabiliy goes o 1. Problem 7 (An AEP-like limi) Le X 1, X 2,... be i.i.d. o probabiliy mass funcion p(x). Find drawn according Soluion. log(x i ) are also i.i.d. and lim [P (X 1, X 2,..., X n )] 1/n. n lim [P (X 1, X 2,..., X n )] 1/n log[p (X1,X2,...,Xn)]1/n = lim 2 n = 2 lim 1 n log P (Xi) a.e. = 2 E(log p(x)) a.e. = 2 H(X) a.e. by he srong law of large numbers (assuming of course ha H(X) exiss). 7