Spring Information Theory Midterm (take home) Due: Tue, Mar 29, 2016 (in class) Prof. Y. Polyanskiy. P XY (i, j) = α 2 i 2j

Size: px

Start display at page:

Download "Spring Information Theory Midterm (take home) Due: Tue, Mar 29, 2016 (in class) Prof. Y. Polyanskiy. P XY (i, j) = α 2 i 2j"

Terence Peter Daniels
5 years ago
Views:

1 Sprig Iformatio Theory Midterm (take home) Due: Tue, Mar 29, 206 (i class) Prof. Y. Polyaskiy Rules. Collaboratio strictly prohibited. 2. Write rigorously, prove all claims. 3. You ca use otes ad textbooks. 4. All exercises are 0 poits. 2 Exercises Let X {0, } ad let Y be a oegative iteger-valued radom variable with joit distributio P XY (i, j) = α 2 i 2j where α is a ormalizatio costat. Fid H(X), H(Y ), H(X, Y ), H(Y X), H(X Y ), D(P Y X=0 P Y X=) ad D( P Y X= P Y X=0 ). 2 Let X be distributed accordig to the expoetial distributio with mea µ > 0, i.e., with desity p(x) = µ e x/µ {x 0}. Let a R. Compute the divergece D(P X+a P X ). 3 Let (X, Y ) be uiformly distributed i the uit l p -ball B p = {(x, y) : x p + y p }, where p (0, ). Also defie the l -ball B = {(x, y) : x, y }.. Compute I(X; Y ) for p = /2, p = ad p =. 2. (Bous) What do you thik I(X; Y ) coverges to as p 0. Ca you prove it? 4 Let X ad Y have fiite alphabets. Let C(P Y X ) = max PX I(X; Y ) be the capacity of PY X.. Is P X H(P X ) strictly cocave? 2. Fix P Y X. Is P X I(X; Y ) strictly cocave? 3. Fix P Y X with C(P Y X) > 0. Is P X I(X; Y ) strictly cocave? 4. Fix P X with H(P X ) > 0. Is P Y X I(X; Y ) strictly covex? 5. Is P XY I(X; Y ) covex, cocave, or either? 6. Is P Y X C(P Y v X) covex, coca e or either? 5 Let {Y k, k = 0,...} be a biary statioary Markov process defied as follows: Let Y 0 be a biary equiprobable radom variable, ad { δ P Yk+ Y k [b a] = δ b = a b a Fid I(Y 0 ; Y ). At what speed does I(Y 0 ; Y ) vaish with?

2 6 (Fiiteess of etropy) We have show that ay N-valued radom variable X, with E[X] < has H(X) E [X]h(/E [X]) <. Next let us improve this result.. Show that E[log X] < H(X) <. Moreover, show that the coditio of X beig iteger-valued is ot superfluous by givig a couterexample. 2. Show that if k P X (k) is a decreasig sequece, the H(X) < E[log X] <. Moreover, show that the mootoicity of pmf is ot superfluous by givig a couterexample. 7 Cosider the hypothesis testig problem: Questios:. Compute the Stei expoet. i.i.d. H 0 : X,..., X P = N (0, ), i.i.d. H : X,..., X Q = N (µ, ). 2. Compute the tradeoff regio E of achievable error-expoet pairs (E 0, E ). Express the optimal boudary i explicit form (elimiate the parameter). 3. Idetify the divergece-miimizig geodesic P (λ) ruig from P to Q, λ [0, ]. Verify that (E 0, E ) = (D(P (λ) P ), D(P (λ) Q)), 0 λ gives the same tradeoff curve. 4. Compute the Cheroff expoet. 8 Baby Saov. Let X be a fiite set. Let E be a covex subset of the simplex of probability distributios o X. Assume that E has o-empty iterior. Let X = (X,..., X ) be iid draw from some distributio P ad let π deote the empirical distributio, i.e., π = i= δ X i, which is a fuctio of X. Our goal is to show that E lim log = if D(Q P ). () P (π E) Q E a) Defie the followig set of joit distributios E {Q X : Q Xi E}. Show that if D(Q X P X ) = if D(Q P ), QX E Q E where P X = P. b) Cosider the coditioal distributio P X = P X π E. Show that PX E. c) Show that ( P (π E) exp d) For ay Q i the iterior of E, show that ) if D(Q P ) Q E,. P (π E) exp( D(Q P ) + o()),. (Hit: Use data processig as i the proof of the large deviatio theorem.) 2

3 e) Coclude (). Commet: Beefit of this proof compared to method of types is that it easily exteds to ifiite alphabets. 9 Let X j exp() be i.i.d. expoetial with mea. Sice MGF Ψ X (λ) does ot exist for all λ >, the result P[ X j γ] = exp{ Ψ X(γ) + o() } (2) j= prove i class does ot apply. Show (2) via the followig steps:. Apply Cheroff argumet directly to prove a upper boud: P[ X j γ] exp{ Ψ X(γ) } (3) j= 2. Fix a arbitrary A > 0 ad prove where u v = mi(u, v). P[ X j γ] P[ (X j A) γ], (4) j= j= 3. Apply the results show i class to ivestigate the asymptotics of the right-had side of (4). 4. Coclude the proof of (2) by takig A. 0 (Gibbs distributio) Let X be fiite alphabet, f : X R some fuctio ad E mi = mi f(x).. Usig I-projectio show that for ay E E mi the solutio of H (E) = max{h(x) : E [f(x)] E} is give by P X (x) = Z(β) e βf(x) for some β = β(e). Commet: I statistical physics x is state of the system (e.g. locatios ad velocities of all molecules), f(x) is eergy of the system i state x, P X is the Gibbs distributio ad β = T is the iverse temperatur of the system. I thermodyamic equillibrium, P X(x) gives fractio of time system speds i state x. 2. Show that dh (E) = β(e). de 3. Next cosider two fuctios f 0, f (i.e. two types of molecules with differet state-eergy relatios). Show that for E mi x0 f(x 0 ) + mi x f(x ) we have max H(X 0, X ) = max H 0 (E 0 ) + H (E ) (5) E [f 0 (X 0 )+f (X )] E E 0 +E E where H j (E) = max E [fj (X)] E H(X). 4. Further, show that for the optimal choice of E 0 ad E i (5) we have β 0 (E 0 ) = β (E ) (6) or equivaletly that the optimal distributio P X0,X is give by P X0,X (a, b) = Z 0 (β)z (β) e β(f 0(a)+f (b)) (7) 3

4 Remark: (7) also just follows from part by takig f(x 0, x ) = f 0 (x 0 ) + f (x ). The poit here is relatio (6): whe two thermodyamical systems are brought i cotact with each other, the eergy distributes amog them i such a way that β parameters (temperatures) equalize. 4

5 MIT OpeCourseWare Iformatio Theory Sprig 206 For iformatio about citig these materials or our Terms of Use, visit:

Lecture 7: October 18, 2017

Lecture 7: October 18, 2017 Iformatio ad Codig Theory Autum 207 Lecturer: Madhur Tulsiai Lecture 7: October 8, 207 Biary hypothesis testig I this lecture, we apply the tools developed i the past few lectures to uderstad the problem