Second-Order Asymptotics in Information Theory
|
|
- Rosa Barber
- 5 years ago
- Views:
Transcription
1 Second-Order Asymptotics in Information Theory Vincent Y. F. Tan Dept. of ECE and Dept. of Mathematics National University of Singapore (NUS) National Taiwan University November 2015 Vincent Tan (NUS) Second-Order Asymptotics NTU 1 / 109
2 Outline 1 Motivation, Background and History Vincent Tan (NUS) Second-Order Asymptotics NTU 2 / 109
3 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing Vincent Tan (NUS) Second-Order Asymptotics NTU 2 / 109
4 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding Vincent Tan (NUS) Second-Order Asymptotics NTU 2 / 109
5 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding 4 Channel Coding Vincent Tan (NUS) Second-Order Asymptotics NTU 2 / 109
6 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding 4 Channel Coding 5 Slepian-Wolf Coding Vincent Tan (NUS) Second-Order Asymptotics NTU 2 / 109
7 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding 4 Channel Coding 5 Slepian-Wolf Coding 6 Summary and Open Problems Vincent Tan (NUS) Second-Order Asymptotics NTU 2 / 109
8 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding 4 Channel Coding 5 Slepian-Wolf Coding 6 Summary and Open Problems Vincent Tan (NUS) Second-Order Asymptotics NTU 3 / 109
9 Transmission of Information Shannon s Figure 1 INFORMATION SOURCE TRANSMITTER RECEIVER DESTINATION SIGNAL RECEIVED SIGNAL MESSAGE MESSAGE NOISE SOURCE Shannon abstracted away information meaning, semantics treat all data equally Shannon s bits as a universal Figure currency 1 crucial abstraction for modern communication and computing systems Also relaxed computation and delay constraints to discover a fundamental limit: capacity, providing a goal-post to work toward Information theory Finding fundamental limits for reliable information transmission Saturday, June 11, 2011 Vincent Tan (NUS) Second-Order Asymptotics NTU 4 / 109
10 Transmission of Information Shannon s Figure 1 INFORMATION SOURCE TRANSMITTER RECEIVER DESTINATION SIGNAL RECEIVED SIGNAL MESSAGE MESSAGE NOISE SOURCE Shannon abstracted away information meaning, semantics treat all data equally Shannon s bits as a universal Figure currency 1 crucial abstraction for modern communication and computing systems Also relaxed computation and delay constraints to discover a fundamental limit: capacity, providing a goal-post to work toward Information theory Finding fundamental limits for reliable information transmission Saturday, June 11, 2011 Channel coding: Concerned with the maximum rate of communication in bits/channel use Vincent Tan (NUS) Second-Order Asymptotics NTU 4 / 109
11 Channel Coding (One-Shot) M X Y M f W ϕ A code is an triple C = {M, f, ϕ} where M is the message set The average error probability p err (C) is [ ] p err (C) := Pr M M where M is uniform on M Vincent Tan (NUS) Second-Order Asymptotics NTU 5 / 109
12 Channel Coding (One-Shot) M X Y M f W ϕ A code is an triple C = {M, f, ϕ} where M is the message set The average error probability p err (C) is [ ] p err (C) := Pr M M where M is uniform on M A non-asymptotic fundamental limit can be defined as M (W, ε) := sup { m N C s.t. m = M, p err (C) ε } Central problem in information theory to characterize M (W, ε). Vincent Tan (NUS) Second-Order Asymptotics NTU 5 / 109
13 Channel Coding (n-shot) M X n Y f W n n M ϕ Consider n independent uses of a discrete memoryless channel (DMC) W n Vincent Tan (NUS) Second-Order Asymptotics NTU 6 / 109
14 Channel Coding (n-shot) M X n Y f W n n M ϕ Consider n independent uses of a discrete memoryless channel (DMC) W n For vectors x n = (x 1,..., x n ) X n and y n := (y 1,..., y n ) Y n, the channel law is n W n (y n x n ) = W(y i x i ) i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 6 / 109
15 Channel Coding (n-shot) M X n Y f W n n M ϕ Consider n independent uses of a discrete memoryless channel (DMC) W n For vectors x n = (x 1,..., x n ) X n and y n := (y 1,..., y n ) Y n, the channel law is n W n (y n x n ) = W(y i x i ) i=1 Non-asymptotic fundamental limit for n uses of W M (W n, ε) Vincent Tan (NUS) Second-Order Asymptotics NTU 6 / 109
16 Background: Shannon s Channel Coding Theorem Shannon s (1948) noisy channel coding theorem and Wolfowitz s (1959) strong converse state that Vincent Tan (NUS) Second-Order Asymptotics NTU 7 / 109
17 Background: Shannon s Channel Coding Theorem Shannon s (1948) noisy channel coding theorem and Wolfowitz s (1959) strong converse state that Theorem (Shannon (1948), Wolfowitz (1959)) where is the capacity of the DMC. 1 lim n n log M (W n, ε) = C, ε (0, 1) C = max P I(P, W) Vincent Tan (NUS) Second-Order Asymptotics NTU 7 / 109
18 Background: Shannon s Channel Coding Theorem 1 lim n n log M (W n, ε) = C bits/channel use Channel coding theorem for DMCs is independent of ε (0, 1) Vincent Tan (NUS) Second-Order Asymptotics NTU 8 / 109
19 Background: Shannon s Channel Coding Theorem 1 lim n n log M (W n, ε) = C bits/channel use Channel coding theorem for DMCs is independent of ε (0, 1) 1 lim p err(c n ) n 0 C R Vincent Tan (NUS) Second-Order Asymptotics NTU 8 / 109
20 Background: Shannon s Channel Coding Theorem 1 lim n n log M (W n, ε) = C bits/channel use Channel coding theorem for DMCs is independent of ε (0, 1) 1 lim p err(c n ) n 0 C R Vincent Tan (NUS) Second-Order Asymptotics NTU 8 / 109
21 Background: Shannon s Channel Coding Theorem 1 lim n n log M (W n, ε) = C bits/channel use Channel coding theorem for DMCs is independent of ε (0, 1) 1 lim p err(c n ) n 0 C R Phase transition at capacity Vincent Tan (NUS) Second-Order Asymptotics NTU 8 / 109
22 Background: Second-Order Coding Rates What happens at capacity? Vincent Tan (NUS) Second-Order Asymptotics NTU 9 / 109
23 Background: Second-Order Coding Rates What happens at capacity? More precisely, what happens when log M n nc + L n for some L R? Vincent Tan (NUS) Second-Order Asymptotics NTU 9 / 109
24 Background: Second-Order Coding Rates What happens at capacity? More precisely, what happens when log M n nc + L n for some L R? Here L is known as the second-order coding rate of the code Vincent Tan (NUS) Second-Order Asymptotics NTU 9 / 109
25 Background: Second-Order Coding Rates What happens at capacity? More precisely, what happens when log M n nc + L n for some L R? Here L is known as the second-order coding rate of the code Note that L can be negative (cf. Hayashi (2008), Hayashi (2009)) Vincent Tan (NUS) Second-Order Asymptotics NTU 9 / 109
26 Background: Second-Order Coding Rates Assume rate of the code satisfies 1 n log M n = C + L ( 1 ) + o n n Vincent Tan (NUS) Second-Order Asymptotics NTU 10 / 109
27 Background: Second-Order Coding Rates Assume rate of the code satisfies 1 n log M n = C + L ( 1 ) + o n n 1 lim n perr(cn) L Vincent Tan (NUS) Second-Order Asymptotics NTU 10 / 109
28 Background: Second-Order Coding Rates Assume rate of the code satisfies 1 n log M n = C + L ( 1 ) + o n n 1 lim n perr(cn) 0.5 p err (C n ) = Φ( L V ) + o(1) 0 L Vincent Tan (NUS) Second-Order Asymptotics NTU 10 / 109
29 Background: Second-Order Coding Rates Assume rate of the code satisfies 1 n log M n = C + L ( 1 ) + o n n 1 lim n perr(cn) 0.5 p err (C n ) = Φ( L V ) + o(1) 0 L For an error probability ε, the optimum second-order coding rate is L (ε) := VΦ 1 (ε) Vincent Tan (NUS) Second-Order Asymptotics NTU 10 / 109
30 Error Exponents vs Normal Approximation For error exponent analysis, we fix rate R < C and study ε (W n, 2 nr ) = min{ε : (2 nr, ε)-code for W n } Most of the time, ε (W n, 2 nr ) exp( ne(r)) Vincent Tan (NUS) Second-Order Asymptotics NTU 11 / 109
31 Error Exponents vs Normal Approximation For error exponent analysis, we fix rate R < C and study ε (W n, 2 nr ) = min{ε : (2 nr, ε)-code for W n } Most of the time, ε (W n, 2 nr ) exp( ne(r)) For normal approximation or second-order analysis, we fix the error probability ε (0, 1) and seek R (W n, ε) = log M (W n, ε) n C + V n Φ 1 (ε) Vincent Tan (NUS) Second-Order Asymptotics NTU 11 / 109
32 Error Exponents vs Normal Approximation For error exponent analysis, we fix rate R < C and study ε (W n, 2 nr ) = min{ε : (2 nr, ε)-code for W n } Most of the time, ε (W n, 2 nr ) exp( ne(r)) For normal approximation or second-order analysis, we fix the error probability ε (0, 1) and seek R (W n, ε) = log M (W n, ε) n C + V n Φ 1 (ε) Some form of duality in the analyses... Vincent Tan (NUS) Second-Order Asymptotics NTU 11 / 109
33 Agenda : Part I Agenda for today s tutorial: Point-to-point communication 1 Most results in point-to-point information theory can be derived by understanding the fundamental limits of binary hypothesis testing (Strassen (1962)) Vincent Tan (NUS) Second-Order Asymptotics NTU 12 / 109
34 Agenda : Part I Agenda for today s tutorial: Point-to-point communication 1 Most results in point-to-point information theory can be derived by understanding the fundamental limits of binary hypothesis testing (Strassen (1962)) 2 Lossless source coding is an easy corollary of binary hypothesis testing (Strassen (1962)) Vincent Tan (NUS) Second-Order Asymptotics NTU 12 / 109
35 Agenda : Part I Agenda for today s tutorial: Point-to-point communication 1 Most results in point-to-point information theory can be derived by understanding the fundamental limits of binary hypothesis testing (Strassen (1962)) 2 Lossless source coding is an easy corollary of binary hypothesis testing (Strassen (1962)) 3 Prove the channel coding dispersion for DMCs 1 Strassen (1962) 2 Hayashi (2009) 3 Polyanskiy-Poor-Verdú (2010) 4 Tomamichel-Tan (2013) Vincent Tan (NUS) Second-Order Asymptotics NTU 12 / 109
36 Agenda : Part II An extension to a multiterminal (network) setting the Slepian-Wolf problem (Tan-Kosut (2014), Nomura-Han (2015)); Vincent Tan (NUS) Second-Order Asymptotics NTU 13 / 109
37 Agenda : Part II An extension to a multiterminal (network) setting the Slepian-Wolf problem (Tan-Kosut (2014), Nomura-Han (2015)); I will be talking about a subset of the material from my monograph. V. Y. F. Tan Asymptotic expansions in information theory with non-vanishing error probabilities Now Publishers Foundations and Trends in Communications and Information Theory Vincent Tan (NUS) Second-Order Asymptotics NTU 13 / 109
38 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding 4 Channel Coding 5 Slepian-Wolf Coding 6 Summary and Open Problems Vincent Tan (NUS) Second-Order Asymptotics NTU 14 / 109
39 Setup of the Binary Hypothesis Testing Problem In binary hypothesis testing, we are concerned with the problem H 0 : Z P H 1 : Z Q where P, Q P(Z) are distributions on the same space Z. Vincent Tan (NUS) Second-Order Asymptotics NTU 15 / 109
40 Setup of the Binary Hypothesis Testing Problem In binary hypothesis testing, we are concerned with the problem H 0 : Z P H 1 : Z Q where P, Q P(Z) are distributions on the same space Z. Alphabet Z may be assumed to be finite Vincent Tan (NUS) Second-Order Asymptotics NTU 15 / 109
41 Setup of the Binary Hypothesis Testing Problem In binary hypothesis testing, we are concerned with the problem H 0 : Z P H 1 : Z Q where P, Q P(Z) are distributions on the same space Z. Alphabet Z may be assumed to be finite Design a test δ : Z {0, 1} that outputs 0 if Z P and 1 otherwise Vincent Tan (NUS) Second-Order Asymptotics NTU 15 / 109
42 Various Error Probabilities For a given test δ : Z {0, 1}, we may define Vincent Tan (NUS) Second-Order Asymptotics NTU 16 / 109
43 Various Error Probabilities For a given test δ : Z {0, 1}, we may define Probability of false alarm P FA (δ) := z δ(z)p(z) = E P [δ(z)] Vincent Tan (NUS) Second-Order Asymptotics NTU 16 / 109
44 Various Error Probabilities For a given test δ : Z {0, 1}, we may define Probability of false alarm P FA (δ) := z δ(z)p(z) = E P [δ(z)] Probability of missed detection P MD (δ) := z (1 δ(z))q(z) = E Q [1 δ(z)] Vincent Tan (NUS) Second-Order Asymptotics NTU 16 / 109
45 Various Error Probabilities For a given test δ : Z {0, 1}, we may define Probability of false alarm P FA (δ) := z δ(z)p(z) = E P [δ(z)] Probability of missed detection P MD (δ) := z (1 δ(z))q(z) = E Q [1 δ(z)] Holy grail: Design a test δ such that P FA (δ) 0 while P D (δ) = 1 P MD (δ) 1 but impossible most of the time Vincent Tan (NUS) Second-Order Asymptotics NTU 16 / 109
46 A Measure of the Performance of a Test δ For given distributions P and Q and an ε (0, 1), we may define β 1 ε (P, Q) := This is the same as β 1 ε (P, Q) := inf {P MD(δ) : P FA (δ) ε} δ:z {0,1} inf A Z:P(A) ε Q(Ac ) where A represents the acceptance region for H 1, i.e., δ(z) = 1 iff z A. Vincent Tan (NUS) Second-Order Asymptotics NTU 17 / 109
47 A Measure of the Performance of a Test δ For given distributions P and Q and an ε (0, 1), we may define β 1 ε (P, Q) := This is the same as β 1 ε (P, Q) := inf {P MD(δ) : P FA (δ) ε} δ:z {0,1} inf A Z:P(A) ε Q(Ac ) where A represents the acceptance region for H 1, i.e., δ(z) = 1 iff z A. The larger the tolerance ε, the smaller β 1 ε Vincent Tan (NUS) Second-Order Asymptotics NTU 17 / 109
48 ε-hypothesis Testing Divergence The ε-hypothesis testing divergence is D ε h(p Q) := log β 1 ε(p, Q) 1 ε Measure of distinguishability of P from Q Vincent Tan (NUS) Second-Order Asymptotics NTU 18 / 109
49 ε-hypothesis Testing Divergence The ε-hypothesis testing divergence is D ε h(p Q) := log β 1 ε(p, Q) 1 ε Measure of distinguishability of P from Q Similar to divergence, we have non-negativity and data processing inequality D ε h(p Q) 0. D ε h(p Q) D ε h(pw QW) where PW(z ) = z P(z)W(z z) for any channel W : Z Z Vincent Tan (NUS) Second-Order Asymptotics NTU 18 / 109
50 ε-information Spectrum Divergence While D ε h (P Q) is very useful and fundamental, it is hard to compute (optimization over functions δ : Z {0, 1}). Vincent Tan (NUS) Second-Order Asymptotics NTU 19 / 109
51 ε-information Spectrum Divergence While D ε h (P Q) is very useful and fundamental, it is hard to compute (optimization over functions δ : Z {0, 1}). Define another related quantity: The ε-information spectrum divergence { { D ε s(p Q) := sup R : P z Z : log P(z) } } Q(z) R ε Information Spectrum Methods in Information Theory by T. S. Han (2003) Vincent Tan (NUS) Second-Order Asymptotics NTU 19 / 109
52 ε-information Spectrum Divergence { { D ε s(p Q) := sup R : P z Z : log P(z) } } Q(z) R ε ε 1 ε Density of log P(Z) Q(Z) when Z P R D ε s (P Q) is the largest point R for which the probability mass to the left is no larger than ε. Vincent Tan (NUS) Second-Order Asymptotics NTU 20 / 109
53 ε-information Spectrum Divergence { { D ε s(p Q) := sup R : P z Z : log P(z) } } Q(z) R ε The ε-information Spectrum Divergence is easy to estimate Vincent Tan (NUS) Second-Order Asymptotics NTU 21 / 109
54 ε-information Spectrum Divergence { { D ε s(p Q) := sup R : P z Z : log P(z) } } Q(z) R ε The ε-information Spectrum Divergence is easy to estimate If P n and Q n are product distributions, i.e., n P n (z n ) = P(z i ), Q n (z n ) = i=1 n Q(z i ), then the probability { P n z n Z n : log Pn (z n } ( n ) ) Q n (z n ) R = Pr log P(Z i) Q(Z i ) R i=1 i=1 and log P(Z i) Q(Z i ) (where Z i P) are iid random variables. Vincent Tan (NUS) Second-Order Asymptotics NTU 21 / 109
55 ε-information Spectrum Divergence { { D ε s(p Q) := sup R : P z Z : log P(z) } } Q(z) R ε The ε-information Spectrum Divergence is easy to estimate If P n and Q n are product distributions, i.e., n P n (z n ) = P(z i ), Q n (z n ) = i=1 n Q(z i ), then the probability { P n z n Z n : log Pn (z n } ( n ) ) Q n (z n ) R = Pr log P(Z i) Q(Z i ) R i=1 i=1 and log P(Z i) Q(Z i ) (where Z i P) are iid random variables. Probability is that of the tail of a sum of iid rvs easy! Vincent Tan (NUS) Second-Order Asymptotics NTU 21 / 109
56 Relation Between Divergences Lemma For every ε (0, 1) and η (0, 1 ε), we have D ε 1 s(p Q) log 1 ε Dε h(p Q) D ε+η s (P Q) + log 1 ε η Vincent Tan (NUS) Second-Order Asymptotics NTU 22 / 109
57 Relation Between Divergences Lemma For every ε (0, 1) and η (0, 1 ε), we have D ε 1 s(p Q) log 1 ε Dε h(p Q) D ε+η s (P Q) + log 1 ε η Proof: We only prove the lower bound: Let δ be the likelihood ratio test { δ(z) := 1 log P(z) } Q(z) γ where γ := D ε s(p Q) ξ. Vincent Tan (NUS) Second-Order Asymptotics NTU 22 / 109
58 Proof of Relation Between Divergences By definition of ε-information spectrum divergence, { E P [δ(z)] = P z Z : log P(z) } Q(z) }{{} γ ε =D ε s (P Q) ξ Vincent Tan (NUS) Second-Order Asymptotics NTU 23 / 109
59 Proof of Relation Between Divergences By definition of ε-information spectrum divergence, { E P [δ(z)] = P z Z : log P(z) } Q(z) }{{} γ ε =D ε s (P Q) ξ Next, we estimate E Q [1 δ(z)] = z z { Q(z)1 log P(z) } Q(z) > γ P(z) exp( γ)1 { log P(z) } Q(z) > γ exp( γ) Vincent Tan (NUS) Second-Order Asymptotics NTU 23 / 109
60 Proof of Relation Between Divergences By definition of ε-information spectrum divergence, { E P [δ(z)] = P z Z : log P(z) } Q(z) }{{} γ ε =D ε s (P Q) ξ Next, we estimate E Q [1 δ(z)] = z z { Q(z)1 log P(z) } Q(z) > γ P(z) exp( γ)1 { log P(z) } Q(z) > γ exp( γ) Thus, one has D ε 1 h(p Q) γ log 1 ε = 1 Dε s(p Q) ξ log 1 ε Proof is completed by taking ξ 0. Vincent Tan (NUS) Second-Order Asymptotics NTU 23 / 109
61 Basic Definitions Define the product distributions n n P (n) (z n ) = P i (z i ), Q (n) (z n ) = Q i (z i ). i=1 i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 24 / 109
62 Basic Definitions Define the product distributions n n P (n) (z n ) = P i (z i ), Q (n) (z n ) = Q i (z i ). The relative entropy is i=1 i=1 D(P Q) = z P(z) log P(z) Q(z), Vincent Tan (NUS) Second-Order Asymptotics NTU 24 / 109
63 Basic Definitions Define the product distributions n n P (n) (z n ) = P i (z i ), Q (n) (z n ) = Q i (z i ). i=1 i=1 The relative entropy is D(P Q) = z P(z) log P(z) Q(z), The relative entropy variance is V(P Q) := z [ P(z) log P(z) ] 2 Q(z) D(P Q). Vincent Tan (NUS) Second-Order Asymptotics NTU 24 / 109
64 Basic Asymptotic Expansions Lemma (Asymptotic Expansion for D ε s) Assume that V(P i Q i ) V for all i and for some V > 0. Then, D ε s(p (n) Q (n) ) = nd n + nv n Φ 1 (ε) + O(1) where D n := 1 n n D(P i Q i ), i=1 V n := 1 n n V(P i Q i ). i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 25 / 109
65 Basic Asymptotic Expansions Lemma (Asymptotic Expansion for D ε s) Assume that V(P i Q i ) V for all i and for some V > 0. Then, D ε s(p (n) Q (n) ) = nd n + nv n Φ 1 (ε) + O(1) where D n := 1 n n D(P i Q i ), i=1 V n := 1 n n V(P i Q i ). i=1 Corollary (Asymptotic Expansion for D ε s for Identical Distributions) If P i = P and Q i = Q for all i {1, 2,..., n} and P Q, then D ε s(p (n) Q (n) ) = nd + nvφ 1 (ε) + O(1) where D = D(P Q) and V = V(P Q). Vincent Tan (NUS) Second-Order Asymptotics NTU 25 / 109
66 Asymptotic Expansion for D ε s(p n Q n ) Moral: n (1 n n i=1 ) log P(Z i) Q(Z i ) D(P Q) d N (0, V(P Q)). Vincent Tan (NUS) Second-Order Asymptotics NTU 26 / 109
67 Asymptotic Expansion for D ε s(p n Q n ) Moral: n (1 n n i=1 ) log P(Z i) Q(Z i ) D(P Q) d N (0, V(P Q)). V(P Q) 1/2 D(P Q) Vincent Tan (NUS) Second-Order Asymptotics NTU 26 / 109
68 Berry-Esseen Theorem Theorem Let X 1,..., X n be independent random variables with Define E[X i ] = 0, E[X 2 i ] = σ 2 i, E[ X i 3 ] = T i σ 2 := 1 n Then for every n 1, ( sup Pr 1 σ n a R n σi 2, i=1 T := 1 n n i=1 ) n X i < a Φ(a) i=1 where Φ(a) = a 1 2π e t2 /2 dt. T i 6T σ 3 n Vincent Tan (NUS) Second-Order Asymptotics NTU 27 / 109
69 Berry-Esseen Theorem vs Central Limit Theorem Recall that CLT implies that ( ) 1 n Pr σ X i < a Φ(a) = Pr(Z < a) n for every a R. i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 28 / 109
70 Berry-Esseen Theorem vs Central Limit Theorem Recall that CLT implies that ( ) 1 n Pr σ X i < a Φ(a) = Pr(Z < a) n for every a R. i=1 Thus, the Berry-Esseen theorem quantifies the rate of convergence of the distribution function of to the standard Gaussian Z. 1 σ n n i=1 X i Vincent Tan (NUS) Second-Order Asymptotics NTU 28 / 109
71 Basic Asymptotic Expansions: Proof Now we show D ε s(p (n) Q (n) ) = nd n + nv n Φ 1 (ε) + O(1) Vincent Tan (NUS) Second-Order Asymptotics NTU 29 / 109
72 Basic Asymptotic Expansions: Proof Now we show D ε s(p (n) Q (n) ) = nd n + nv n Φ 1 (ε) + O(1) We may write ( Pr log P(n) (Z n ) Q (n) (Z n ) R ) = Pr ( n i=1 ) log P i(z i ) Q i (Z i ) R Vincent Tan (NUS) Second-Order Asymptotics NTU 29 / 109
73 Basic Asymptotic Expansions: Proof Now we show D ε s(p (n) Q (n) ) = nd n + nv n Φ 1 (ε) + O(1) We may write ( Pr log P(n) (Z n ) Q (n) (Z n ) R ) = Pr By the Berry-Esseen theorem, ( n ) Pr log P i(z i ) Q i (Z i ) R i=1 ( n i=1 ) log P i(z i ) Q i (Z i ) R ( ) R ndn = Φ ± c nvn n Vincent Tan (NUS) Second-Order Asymptotics NTU 29 / 109
74 Basic Asymptotic Expansions: Proof Now we show D ε s(p (n) Q (n) ) = nd n + nv n Φ 1 (ε) + O(1) We may write ( Pr log P(n) (Z n ) Q (n) (Z n ) R ) = Pr By the Berry-Esseen theorem, ( n ) Pr log P i(z i ) Q i (Z i ) R i=1 ( n i=1 ) log P i(z i ) Q i (Z i ) R ( ) R ndn = Φ ± c nvn n Constant c > 0 finite because V(P i Q i ) V > 0 for all i. Vincent Tan (NUS) Second-Order Asymptotics NTU 29 / 109
75 Basic Asymptotic Expansions: Proof Now we show D ε s(p (n) Q (n) ) = nd n + nv n Φ 1 (ε) + O(1) We may write ( Pr log P(n) (Z n ) Q (n) (Z n ) R ) = Pr By the Berry-Esseen theorem, ( n ) Pr log P i(z i ) Q i (Z i ) R i=1 ( n i=1 ) log P i(z i ) Q i (Z i ) R ( ) R ndn = Φ ± c nvn n Constant c > 0 finite because V(P i Q i ) V > 0 for all i. Now upper bound the RHS by ε and solve for R Vincent Tan (NUS) Second-Order Asymptotics NTU 29 / 109
76 Further Asymptotic Expansions Our real objective in this section is β 1 ε (P n, Q n ) := inf {P MD(δ) : P FA (δ) ε} δ:z {0,1} Vincent Tan (NUS) Second-Order Asymptotics NTU 30 / 109
77 Further Asymptotic Expansions Our real objective in this section is β 1 ε (P n, Q n ) := inf {P MD(δ) : P FA (δ) ε} δ:z {0,1} Now, we assume that P n and Q n are product distributions where the component distributions are identical, i.e., P n (z n ) = n P(z i ), Q n (z n ) = i=1 n Q(z i ). i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 30 / 109
78 Further Asymptotic Expansions Our real objective in this section is β 1 ε (P n, Q n ) := inf {P MD(δ) : P FA (δ) ε} δ:z {0,1} Now, we assume that P n and Q n are product distributions where the component distributions are identical, i.e., P n (z n ) = n P(z i ), Q n (z n ) = i=1 n Q(z i ). i=1 Recall that D ε h(p n Q n ) := log β 1 ε(p n, Q n ) 1 ε and D ε h is related to Dε s by simple bounds Vincent Tan (NUS) Second-Order Asymptotics NTU 30 / 109
79 Further Asymptotic Expansions Lemma Assume that P Q. Then, for every ε (0, 1), D ε h(p n Q n ) = nd + nvφ 1 (ε) + O(log n) where D = D(P Q) and V = V(P Q). Vincent Tan (NUS) Second-Order Asymptotics NTU 31 / 109
80 Further Asymptotic Expansions Lemma Assume that P Q. Then, for every ε (0, 1), D ε h(p n Q n ) = nd + nvφ 1 (ε) + O(log n) where D = D(P Q) and V = V(P Q). Corollary Assume that P Q. Then, for every ε (0, 1), β 1 ε (P n, Q n ) = exp ( nd nvφ 1 (ε) + O(log n) ) Vincent Tan (NUS) Second-Order Asymptotics NTU 31 / 109
81 Chernoff-Stein Lemma H. Chernoff C. Stein Vincent Tan (NUS) Second-Order Asymptotics NTU 32 / 109
82 Chernoff-Stein Lemma H. Chernoff C. Stein Corollary (Chernoff-Stein Lemma) lim 1 n n log β 1 ε(p n, Q n ) = D(P Q), ε (0, 1). Vincent Tan (NUS) Second-Order Asymptotics NTU 32 / 109
83 Chernoff-Stein Lemma H. Chernoff C. Stein Corollary (Chernoff-Stein Lemma) lim 1 n n log β 1 ε(p n, Q n ) = D(P Q), ε (0, 1). We have proved a refinement to the Chernoff-Stein Lemma, cf. log β 1 ε (P n, Q n ) = nd(p Q) + nv(p Q)Φ 1 (ε) + O(log n) Vincent Tan (NUS) Second-Order Asymptotics NTU 32 / 109
84 Further Asymptotic Expansions: Proof Recall that D ε s(p n Q n 1 ) log 1 ε Dε h(p n Q n ) D ε+η s (P n Q n ) + log 1 ε η Vincent Tan (NUS) Second-Order Asymptotics NTU 33 / 109
85 Further Asymptotic Expansions: Proof Recall that D ε s(p n Q n 1 ) log 1 ε Dε h(p n Q n ) D ε+η s (P n Q n ) + log 1 ε η From the lower bound, we obtain D ε s(p n Q n ) log 1 1 ε = nd + nvφ 1 (ε) + O(1) Vincent Tan (NUS) Second-Order Asymptotics NTU 33 / 109
86 Further Asymptotic Expansions: Proof Recall that D ε s(p n Q n 1 ) log 1 ε Dε h(p n Q n ) D ε+η s (P n Q n ) + log 1 ε η From the lower bound, we obtain D ε s(p n Q n ) log 1 1 ε = nd + nvφ 1 (ε) + O(1) From the upper bound, setting η = 1 n, we obtain Ds ε+η (P n Q n ) + log 1 ε = nd + ( nvφ 1 ε + 1 ) + 1 log n + O(1) η n 2 Taylor = nd + nvφ 1 (ε) + 1 log n + O(1) 2 Vincent Tan (NUS) Second-Order Asymptotics NTU 33 / 109
87 Summary of Binary Hypothesis Testing The main result of this section is 1 V(P Q) n log β 1 ε(p n, Q n ) D(P Q) + Φ 1 (ε). n Vincent Tan (NUS) Second-Order Asymptotics NTU 34 / 109
88 Summary of Binary Hypothesis Testing The main result of this section is 1 V(P Q) n log β 1 ε(p n, Q n ) D(P Q) + Φ 1 (ε). n We can be more precise about the third-order terms (omitted in this tutorial) Vincent Tan (NUS) Second-Order Asymptotics NTU 34 / 109
89 Summary of Binary Hypothesis Testing The main result of this section is 1 V(P Q) n log β 1 ε(p n, Q n ) D(P Q) + Φ 1 (ε). n We can be more precise about the third-order terms (omitted in this tutorial) Sometimes we can t guarantee that inf V(P i Q i ) > 0. i 1 In this case, we can use Chebyshev s inequality instead of the Berry-Esseen theorem to upper bound D ε s(p (n) Q (n) ) Vincent Tan (NUS) Second-Order Asymptotics NTU 34 / 109
90 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding 4 Channel Coding 5 Slepian-Wolf Coding 6 Summary and Open Problems Vincent Tan (NUS) Second-Order Asymptotics NTU 35 / 109
91 Setup for Lossless Source Coding x m ˆx f ϕ Illustration of the fixed-to-fixed length source coding problem. Vincent Tan (NUS) Second-Order Asymptotics NTU 36 / 109
92 Setup for Lossless Source Coding x m ˆx f ϕ Illustration of the fixed-to-fixed length source coding problem. An (M, ε)-code for the source P P(X ) consists of encoder f : X {1,..., M} decoder ϕ : {1,..., M} X such that the probability of error P ( {x X : ϕ(f (x)) x} ) ε. Vincent Tan (NUS) Second-Order Asymptotics NTU 36 / 109
93 Non-Asymptotic Fundamental Limit for Source Coding Define M (P, ε) := min{m : an (M, ε)-code for P} Vincent Tan (NUS) Second-Order Asymptotics NTU 37 / 109
94 Non-Asymptotic Fundamental Limit for Source Coding Define M (P, ε) := min{m : an (M, ε)-code for P} When we observe n independent and identically distributed realizations of the source, we are interested in M (P n, ε), where P n (x n ) = n P(x i ). i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 37 / 109
95 The Source Coding Theorem Shannon, in his seminal 1948 paper, showed using typicality arguments that Theorem (Shannon (1948)) For any discrete memoryless source and any ε (0, 1), 1 lim n n log M (P n, ε) = H(P) = x P(x) log 1 P(x), bits per source symb. Vincent Tan (NUS) Second-Order Asymptotics NTU 38 / 109
96 The Source Coding Theorem Shannon, in his seminal 1948 paper, showed using typicality arguments that Theorem (Shannon (1948)) For any discrete memoryless source and any ε (0, 1), 1 lim n n log M (P n, ε) = H(P) = x P(x) log 1 P(x), bits per source symb. The quantity H(P) is known as the entropy of the source. Vincent Tan (NUS) Second-Order Asymptotics NTU 38 / 109
97 The Source Coding Theorem Shannon, in his seminal 1948 paper, showed using typicality arguments that Theorem (Shannon (1948)) For any discrete memoryless source and any ε (0, 1), 1 lim n n log M (P n, ε) = H(P) = x P(x) log 1 P(x), bits per source symb. The quantity H(P) is known as the entropy of the source. Interpretation: Entropy is the smallest exponent of the size of sets in X n with P n -probability of at least 1 ε. Vincent Tan (NUS) Second-Order Asymptotics NTU 38 / 109
98 Refinements to the Source Coding Theorem Can we refine the remainder terms in log M (P n, ε) = nh(p) + o(n) bits per source symb. Vincent Tan (NUS) Second-Order Asymptotics NTU 39 / 109
99 Refinements to the Source Coding Theorem Can we refine the remainder terms in log M (P n, ε) = nh(p) + o(n) bits per source symb. In fact, this can be done very easily by using binary hypothesis testing! Vincent Tan (NUS) Second-Order Asymptotics NTU 39 / 109
100 Refinements to the Source Coding Theorem Can we refine the remainder terms in log M (P n, ε) = nh(p) + o(n) bits per source symb. In fact, this can be done very easily by using binary hypothesis testing! Strong and simple connection between binary hypothesis testing and fixed-length lossless source coding Vincent Tan (NUS) Second-Order Asymptotics NTU 39 / 109
101 Non-Asymptotic Achievability for Source Coding Lemma Let ε (0, 1). We have M (P, ε) β 1 ε (P, µ) = D ε h(p µ) log where µ is the counting measure, i.e., 1 1 ε µ(a) = A, A X Vincent Tan (NUS) Second-Order Asymptotics NTU 40 / 109
102 Non-Asymptotic Achievability for Source Coding Lemma Let ε (0, 1). We have M (P, ε) β 1 ε (P, µ) = D ε h(p µ) log where µ is the counting measure, i.e., 1 1 ε µ(a) = A, A X Remark that second argument β 1 ε (P, Q) can be any unnormalized measure (not necessarily a probability measure), i.e., we take Q = µ Vincent Tan (NUS) Second-Order Asymptotics NTU 40 / 109
103 Proof of Achievability for Source Coding Let T X be a typical set of symbols with P-probability 1 ε, i.e., P(T ) 1 ε Vincent Tan (NUS) Second-Order Asymptotics NTU 41 / 109
104 Proof of Achievability for Source Coding Let T X be a typical set of symbols with P-probability 1 ε, i.e., P(T ) 1 ε Search over all such sets for the one with the smallest cardinality. Vincent Tan (NUS) Second-Order Asymptotics NTU 41 / 109
105 Proof of Achievability for Source Coding Let T X be a typical set of symbols with P-probability 1 ε, i.e., P(T ) 1 ε Search over all such sets for the one with the smallest cardinality. Assign each symbol in T a unique index from {1,..., T } and those in X \ T the symbol 1. Vincent Tan (NUS) Second-Order Asymptotics NTU 41 / 109
106 Proof of Achievability for Source Coding Let T X be a typical set of symbols with P-probability 1 ε, i.e., P(T ) 1 ε Search over all such sets for the one with the smallest cardinality. Assign each symbol in T a unique index from {1,..., T } and those in X \ T the symbol 1. Clearly, error probability ε and hence M (P, ε) The RHS is exactly β 1 ε (P, µ). min T T X :P(T ) 1 ε Vincent Tan (NUS) Second-Order Asymptotics NTU 41 / 109
107 Non-Asymptotic Converse for Source Coding Lemma Let ε (0, 1). For any η (0, 1 ε), we have log M (P, ε) D ε+η s (P µ) log 1 η Proof follows from the idea as upper bounding D ε h with Dε+η s Vincent Tan (NUS) Second-Order Asymptotics NTU 42 / 109
108 Non-Asymptotic Converse for Source Coding Lemma Let ε (0, 1). For any η (0, 1 ε), we have log M (P, ε) D ε+η s (P µ) log 1 η Proof follows from the idea as upper bounding D ε h In summary, with Dε+η s D ε+η s (P µ) log 1 η log M (P, ε) D ε 1 h(p µ) log 1 ε Vincent Tan (NUS) Second-Order Asymptotics NTU 42 / 109
109 Asymptotic Expansion for Lossless Source Coding Define the entropy variance as V(P) = x Theorem (Strassen (1962)) P(x) For any source P with V(P) > 0, we have [ log 1 ] 2 P(x) H(P) log M (P n, ε) = nh(p) nv(p)φ 1 (ε) + O(log n) V. Strassen Vincent Tan (NUS) Second-Order Asymptotics NTU 43 / 109
110 Asymptotic Expansion for Source Coding: Proof The lower bound reads log M (P n, ε) D ε+η s (P n µ n ) log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 44 / 109
111 Asymptotic Expansion for Source Coding: Proof The lower bound reads log M (P n, ε) D ε+η s (P n µ n ) log 1 η Choose η = 1 n. Use the asymptotic expansion for D ε s: log M (P n, ε) D ε+ 1 n s (P n µ n ) + O(log n) = nh(p) nv(p)φ 1 ( ε + 1 n ) + O(log n) because D(P µ) = H(P), and V(P µ) = V(P). Vincent Tan (NUS) Second-Order Asymptotics NTU 44 / 109
112 Asymptotic Expansion for Source Coding: Proof The lower bound reads log M (P n, ε) D ε+η s (P n µ n ) log 1 η Choose η = 1 n. Use the asymptotic expansion for D ε s: log M (P n, ε) D ε+ 1 n s (P n µ n ) + O(log n) = nh(p) nv(p)φ 1 ( ε + 1 n ) + O(log n) because D(P µ) = H(P), and V(P µ) = V(P). Proof is completed by Taylor expanding Φ 1 ( ). Vincent Tan (NUS) Second-Order Asymptotics NTU 44 / 109
113 Asymptotic Expansion for Source Coding: Proof The upper bound reads log M (P n, ε) D ε h(p n µ n ) log 1 1 ε Vincent Tan (NUS) Second-Order Asymptotics NTU 45 / 109
114 Asymptotic Expansion for Source Coding: Proof The upper bound reads log M (P n, ε) D ε h(p n µ n ) log 1 1 ε The asymptotic expansion for D ε h then yields log M (P n, ε) nh(p) nv(p)φ 1 (ε) + O(log n) again because D(P µ) = H(P) and V(P µ) = V(P) Vincent Tan (NUS) Second-Order Asymptotics NTU 45 / 109
115 Lossless Source Coding: Remarks Main takeaway here is that log M (P n, ε) = nh(p) nv(p)φ 1 (ε) + O(log n) We can be more precise about O(log n) = 1 log n + O(1) 2 This requires some additional techniques. Vincent Tan (NUS) Second-Order Asymptotics NTU 46 / 109
116 Lossless Source Coding: Remarks Main takeaway here is that log M (P n, ε) = nh(p) nv(p)φ 1 (ε) + O(log n) We can be more precise about O(log n) = 1 log n + O(1) 2 This requires some additional techniques. Observe that fixed-length lossless source coding is nothing but binary hypothesis testing with Q taken to be the counting measure µ and P remains as P! Vincent Tan (NUS) Second-Order Asymptotics NTU 46 / 109
117 Lossless Source Coding: Remarks Main takeaway here is that log M (P n, ε) = nh(p) nv(p)φ 1 (ε) + O(log n) We can be more precise about O(log n) = 1 log n + O(1) 2 This requires some additional techniques. Observe that fixed-length lossless source coding is nothing but binary hypothesis testing with Q taken to be the counting measure µ and P remains as P! Proof follows directly from expansions of D ε h (Pn Q n ) and D ε s(p n Q n ) Vincent Tan (NUS) Second-Order Asymptotics NTU 46 / 109
118 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding 4 Channel Coding 5 Slepian-Wolf Coding 6 Summary and Open Problems Vincent Tan (NUS) Second-Order Asymptotics NTU 47 / 109
119 The Setup of the Channel Coding Problem m x y ˆm f W ϕ Illustration of the channel coding problem. Vincent Tan (NUS) Second-Order Asymptotics NTU 48 / 109
120 The Setup of the Channel Coding Problem m x y ˆm f W ϕ Illustration of the channel coding problem. An (M, ε) av -code for the channel W P(Y X ) consists of encoder f : {1,..., M} X decoder ϕ : X {1,..., M} such that the average probability of error 1 M M W(Y \ ϕ(m) f (m)) ε. m=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 48 / 109
121 Non-Asymptotic Fund. Limit for Channel Coding Define M av(w, ε) := max{m : an (M, ε) av -code for W} Vincent Tan (NUS) Second-Order Asymptotics NTU 49 / 109
122 Non-Asymptotic Fund. Limit for Channel Coding Define M av(w, ε) := max{m : an (M, ε) av -code for W} When we use the channel n times, we are interested in n Mav(W n, ε), where W n (y n x n ) = W(y i x i ). i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 49 / 109
123 Non-Asymptotic Fund. Limit for Channel Coding Define M av(w, ε) := max{m : an (M, ε) av -code for W} When we use the channel n times, we are interested in n Mav(W n, ε), where W n (y n x n ) = W(y i x i ). This means that the channel is stationary and memoryless. i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 49 / 109
124 Non-Asymptotic Fund. Limit for Channel Coding Define M av(w, ε) := max{m : an (M, ε) av -code for W} When we use the channel n times, we are interested in n Mav(W n, ε), where W n (y n x n ) = W(y i x i ). This means that the channel is stationary and memoryless. We also assume X and Y are finite so W is a DMC. i=1 Vincent Tan (NUS) Second-Order Asymptotics NTU 49 / 109
125 Non-Asymptotic Fund. Limit for Channel Coding Define M av(w, ε) := max{m : an (M, ε) av -code for W} When we use the channel n times, we are interested in M av(w n, ε), where W n (y n x n ) = n W(y i x i ). This means that the channel is stationary and memoryless. i=1 We also assume X and Y are finite so W is a DMC. Later we will also consider AWGN channels in which there is a power constraint Vincent Tan (NUS) Second-Order Asymptotics NTU 49 / 109
126 The Channel Coding Theorem Shannon, in his seminal 1948 paper, showed using typicality arguments that Theorem (Shannon (1948)) For any discrete memoryless channel and any ε (0, 1), 1 lim n n log M av(w n, ε) = C = max I(P, W), bits per channel use. P Vincent Tan (NUS) Second-Order Asymptotics NTU 50 / 109
127 The Channel Coding Theorem Shannon, in his seminal 1948 paper, showed using typicality arguments that Theorem (Shannon (1948)) For any discrete memoryless channel and any ε (0, 1), 1 lim n n log M av(w n, ε) = C = max I(P, W), bits per channel use. P Mutual information for an input distribution P and channel W is defined as I(P, W) = x P(x)D(W( x) PW) = x,y P(x)W(y x) log W(y x) PW(y) Vincent Tan (NUS) Second-Order Asymptotics NTU 50 / 109
128 The Channel Coding Theorem Shannon, in his seminal 1948 paper, showed using typicality arguments that Theorem (Shannon (1948)) For any discrete memoryless channel and any ε (0, 1), 1 lim n n log M av(w n, ε) = C = max I(P, W), bits per channel use. P Mutual information for an input distribution P and channel W is defined as I(P, W) = x P(x)D(W( x) PW) = x,y P(x)W(y x) log W(y x) PW(y) Interpretation: We can send up to C bits per channel use over W. Vincent Tan (NUS) Second-Order Asymptotics NTU 50 / 109
129 Refinements to the Channel Coding Theorem Can we refine the remainder terms in log M av(w n, ε) = nc + o(n) Vincent Tan (NUS) Second-Order Asymptotics NTU 51 / 109
130 Refinements to the Channel Coding Theorem Can we refine the remainder terms in log M av(w n, ε) = nc + o(n) This is not as simple as lossless source coding Vincent Tan (NUS) Second-Order Asymptotics NTU 51 / 109
131 Refinements to the Channel Coding Theorem Can we refine the remainder terms in log M av(w n, ε) = nc + o(n) This is not as simple as lossless source coding Requires understanding of non-asymptotic (finite blocklength) bounds Vincent Tan (NUS) Second-Order Asymptotics NTU 51 / 109
132 Refinements to the Channel Coding Theorem Can we refine the remainder terms in log M av(w n, ε) = nc + o(n) This is not as simple as lossless source coding Requires understanding of non-asymptotic (finite blocklength) bounds Careful asymptotic evaluations Vincent Tan (NUS) Second-Order Asymptotics NTU 51 / 109
133 Non-Asymptotic Achievability for Channel Coding Lemma (Feinstein (1954)) Let ε (0, 1) and W be any channel from X to Y. Then for any η (0, ε), we have log M av(w, ε) sup P D ε η s (P W P PW) log 1 η. Vincent Tan (NUS) Second-Order Asymptotics NTU 52 / 109
134 Non-Asymptotic Achievability for Channel Coding Lemma (Feinstein (1954)) Let ε (0, 1) and W be any channel from X to Y. Then for any η (0, ε), we have log M av(w, ε) sup P D ε η s (P W P PW) log 1 η. In fact, the original Feinstein s lemma (1954) works for max probability of error log M max(w, ε) sup P D ε η s (P W P PW) log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 52 / 109
135 Non-Asymptotic Achievability for Channel Coding Lemma (Feinstein (1954)) Let ε (0, 1) and W be any channel from X to Y. Then for any η (0, ε), we have log M av(w, ε) sup P D ε η s (P W P PW) log 1 η. In fact, the original Feinstein s lemma (1954) works for max probability of error log M max(w, ε) sup P D ε η s (P W P PW) log 1 η Again connection to binary hypothesis testing Vincent Tan (NUS) Second-Order Asymptotics NTU 52 / 109
136 Proof of Feinstein s Lemma : Part I Generate M codewords independently from P. This forms the codebook C = {x(1),..., x(m)}. Vincent Tan (NUS) Second-Order Asymptotics NTU 53 / 109
137 Proof of Feinstein s Lemma : Part I Generate M codewords independently from P. This forms the codebook C = {x(1),..., x(m)}. Given y, decode to m {1,..., M} if and only if log W(y x(m)) PW(y) > γ Vincent Tan (NUS) Second-Order Asymptotics NTU 53 / 109
138 Proof of Feinstein s Lemma : Part I Generate M codewords independently from P. This forms the codebook C = {x(1),..., x(m)}. Given y, decode to m {1,..., M} if and only if log W(y x(m)) PW(y) > γ Assume m = 1. Error events are { E 1 := E 2 := log W(Y X(1)) PW(Y) } γ { m 1 : log W(Y X( m)) > γ PW(Y) } Vincent Tan (NUS) Second-Order Asymptotics NTU 53 / 109
139 Proof of Feinstein s Lemma : Part II Probability of error is Pr(E) Pr(E 1 ) + Pr(E 2 ) Vincent Tan (NUS) Second-Order Asymptotics NTU 54 / 109
140 Proof of Feinstein s Lemma : Part II Probability of error is Pr(E) Pr(E 1 ) + Pr(E 2 ) First term is (related to information spectrum divergence) ( Pr(E 1 ) = Pr log W(Y X(1)) ) γ PW(Y) Vincent Tan (NUS) Second-Order Asymptotics NTU 54 / 109
141 Proof of Feinstein s Lemma : Part II Probability of error is Pr(E) Pr(E 1 ) + Pr(E 2 ) First term is (related to information spectrum divergence) ( Pr(E 1 ) = Pr log W(Y X(1)) ) γ PW(Y) Second term is Pr(E 2 ) = Pr M x,y M x,y ( m 1 : log W(Y X( m)) PW(Y) P(x)PW(y)1 ) > γ { log W(y x) PW(y) > γ P(x)W(y x) exp( γ)1 } { log W(y x) } PW(y) > γ M exp( γ) Vincent Tan (NUS) Second-Order Asymptotics NTU 54 / 109
142 Proof of Feinstein s Lemma : Part III Hence, there exists an (M, ε) av -code such that for every P and every γ, ( ε Pr log W(Y X(1)) ) γ + M exp( γ) PW(Y) or setting η := M exp( γ), ( ε η Pr log W(Y X(1)) log M ) PW(Y) η Vincent Tan (NUS) Second-Order Asymptotics NTU 55 / 109
143 Proof of Feinstein s Lemma : Part III Hence, there exists an (M, ε) av -code such that for every P and every γ, ( ε Pr log W(Y X(1)) ) γ + M exp( γ) PW(Y) or setting η := M exp( γ), ( ε η Pr log W(Y X(1)) log M ) PW(Y) η In other words, for every P and every η, we have an (M, ε) av -code s.t. log M η Dε η s (P W P PW) Vincent Tan (NUS) Second-Order Asymptotics NTU 55 / 109
144 Proof of Feinstein s Lemma : Part III Hence, there exists an (M, ε) av -code such that for every P and every γ, ( ε Pr log W(Y X(1)) ) γ + M exp( γ) PW(Y) or setting η := M exp( γ), ( ε η Pr log W(Y X(1)) log M ) PW(Y) η In other words, for every P and every η, we have an (M, ε) av -code s.t. log M η Dε η s (P W P PW) This completes the proof of Feinstein s lemma. Vincent Tan (NUS) Second-Order Asymptotics NTU 55 / 109
145 Second-Order Achievability for Channel Coding Feinstein s lemma says that log M av(w n, ε) sup D ε η s (P W n P PW n ) log 1 P P(X n ) η. Vincent Tan (NUS) Second-Order Asymptotics NTU 56 / 109
146 Second-Order Achievability for Channel Coding Feinstein s lemma says that log M av(w n, ε) sup D ε η s (P W n P PW n ) log 1 P P(X n ) η. Choose η = 1 n and where P(x n ) = which we assume to be unique. n P (x i ) i=1 P = arg max I(P, W) P Vincent Tan (NUS) Second-Order Asymptotics NTU 56 / 109
147 Second-Order Achievability for Channel Coding By the asymptotic expansion of D ε s, we obtain D ε η s ((P ) n W n (P ) n (P W) n ) = ni(p, W) + nu(p, W)Φ 1 (ε) + O(log n) Vincent Tan (NUS) Second-Order Asymptotics NTU 57 / 109
148 Second-Order Achievability for Channel Coding By the asymptotic expansion of D ε s, we obtain D ε η s ((P ) n W n (P ) n (P W) n ) = ni(p, W) + nu(p, W)Φ 1 (ε) + O(log n) Then using the bound on log M av(w n, ε), we obtain log M av(w n, ε) ni(p, W) + nu(p, W)Φ 1 (ε) + O(log n) = nc + nu(p, W)Φ 1 (ε) + O(log n) where the unconditional information variance is U(P, W) = [ P (x)w(y x) log W(y x) P W(y) C x,y ] 2 Vincent Tan (NUS) Second-Order Asymptotics NTU 57 / 109
149 Second-Order Achievability for Channel Coding Lemma For a DMC with unique capacity-achieving input distribution P, log M av(w n, ε) nc + nu(p, W)Φ 1 (ε) + O(log n) Vincent Tan (NUS) Second-Order Asymptotics NTU 58 / 109
150 Second-Order Achievability for Channel Coding Lemma For a DMC with unique capacity-achieving input distribution P, log M av(w n, ε) nc + nu(p, W)Φ 1 (ε) + O(log n) The second-order term contains U(P, W) = x,y P (x)w(y x) [ log W(y x) ] 2 P W(y) C Vincent Tan (NUS) Second-Order Asymptotics NTU 58 / 109
151 Second-Order Achievability for Channel Coding Lemma For a DMC with unique capacity-achieving input distribution P, log M av(w n, ε) nc + nu(p, W)Φ 1 (ε) + O(log n) The second-order term contains U(P, W) = x,y P (x)w(y x) [ log W(y x) ] 2 P W(y) C This is not quite right, but almost right; need a converse. Vincent Tan (NUS) Second-Order Asymptotics NTU 58 / 109
152 Second-Order Achievability for Channel Coding Lemma For a DMC with unique capacity-achieving input distribution P, log M av(w n, ε) nc + nu(p, W)Φ 1 (ε) + O(log n) The second-order term contains U(P, W) = x,y P (x)w(y x) [ log W(y x) ] 2 P W(y) C This is not quite right, but almost right; need a converse. We can refine the third-order term (Polyanskiy s thesis (2010)) Vincent Tan (NUS) Second-Order Asymptotics NTU 58 / 109
153 Second-Order Achievability for Channel Coding Lemma For a DMC with unique capacity-achieving input distribution P, log M av(w n, ε) nc + nu(p, W)Φ 1 (ε) + O(log n) The second-order term contains U(P, W) = x,y P (x)w(y x) [ log W(y x) ] 2 P W(y) C This is not quite right, but almost right; need a converse. We can refine the third-order term (Polyanskiy s thesis (2010)) Since Feinstein s lemma holds under the max error setting, the same bound holds for log M max(w n, ε). Vincent Tan (NUS) Second-Order Asymptotics NTU 58 / 109
154 Non-Asymptotic Converse for Channel Coding Lemma (Hayashi-Nagaoka (2003)) Let ε (0, 1) and η (0, 1 ε). Let W be any channel from X to Y. Then, log M av(w, ε) inf sup Q P(Y) P P(X ) D ε+η s (P W P Q) + log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 59 / 109
155 Non-Asymptotic Converse for Channel Coding Lemma (Hayashi-Nagaoka (2003)) Let ε (0, 1) and η (0, 1 ε). Let W be any channel from X to Y. Then, log M av(w, ε) inf sup Q P(Y) P P(X ) D ε+η s (P W P Q) + log 1 η Amazing duality with Feinstein s lemma log M av(w, ε) sup P P(X ) Ds ε η (P W P PW) log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 59 / 109
156 Non-Asymptotic Converse for Channel Coding Lemma (Hayashi-Nagaoka (2003)) Let ε (0, 1) and η (0, 1 ε). Let W be any channel from X to Y. Then, log M av(w, ε) inf sup Q P(Y) P P(X ) D ε+η s (P W P Q) + log 1 η Amazing duality with Feinstein s lemma log M av(w, ε) sup P P(X ) Ds ε η (P W P PW) log 1 η But the user can choose the output distribution Q Vincent Tan (NUS) Second-Order Asymptotics NTU 59 / 109
157 Non-Asymptotic Converse for Channel Coding Lemma (Hayashi-Nagaoka (2003)) Let ε (0, 1) and η (0, 1 ε). Let W be any channel from X to Y. Then, log M av(w, ε) inf sup Q P(Y) P P(X ) D ε+η s (P W P Q) + log 1 η Amazing duality with Feinstein s lemma log M av(w, ε) sup P P(X ) Ds ε η (P W P PW) log 1 η But the user can choose the output distribution Q Key in the second-order converse is how to choose Q so as to ensure the evaluation of RHS is easy Vincent Tan (NUS) Second-Order Asymptotics NTU 59 / 109
158 Proof of Hayashi-Nagaoka Lemma : Part I Fix any (M, ε) av -code for W. This induces the Markov chain J X Y Ĵ where J is uniformly distributed on {1,..., M}. This Markov chain induces the code distribution P (j, x, y, ĵ) = 1 JXYĴ 1{x = f (j)}w(y x)1{ĵ = ϕ(y)}. M Vincent Tan (NUS) Second-Order Asymptotics NTU 60 / 109
159 Proof of Hayashi-Nagaoka Lemma : Part I Fix any (M, ε) av -code for W. This induces the Markov chain J X Y Ĵ where J is uniformly distributed on {1,..., M}. This Markov chain induces the code distribution P (j, x, y, ĵ) = 1 JXYĴ 1{x = f (j)}w(y x)1{ĵ = ϕ(y)}. M Due to the data processing inequality for D ε h, we obtain D ε h(p W P Q) = D ε h(p XY P X Q Y ) D ε h(p JĴ P J QĴ) where QĴ is induced by the decoder ϕ applied to Q Y. Vincent Tan (NUS) Second-Order Asymptotics NTU 60 / 109
160 Proof of Hayashi-Nagaoka Lemma : Part I Fix any (M, ε) av -code for W. This induces the Markov chain J X Y Ĵ where J is uniformly distributed on {1,..., M}. This Markov chain induces the code distribution P (j, x, y, ĵ) = 1 JXYĴ 1{x = f (j)}w(y x)1{ĵ = ϕ(y)}. M Due to the data processing inequality for D ε h, we obtain D ε h(p W P Q) = D ε h(p XY P X Q Y ) D ε h(p JĴ P J QĴ) where QĴ is induced by the decoder ϕ applied to Q Y. Consider the test δ(j, ĵ) = 1{j ĵ} Vincent Tan (NUS) Second-Order Asymptotics NTU 60 / 109
161 Proof of Hayashi-Nagaoka Lemma : Part II The test satisfies E PJĴ [δ(j, Ĵ)] = Pr(J Ĵ) ε. Vincent Tan (NUS) Second-Order Asymptotics NTU 61 / 109
162 Proof of Hayashi-Nagaoka Lemma : Part II The test satisfies E PJĴ [δ(j, Ĵ)] = Pr(J Ĵ) ε. Furthermore, E PJ QĴ[δ(J, Ĵ)] = j,ĵ P J (j)qĵ(ĵ)1{j ĵ} = 1 j,ĵ P J (j)qĵ(ĵ)1{j = ĵ} = 1 ĵ QĴ(ĵ) j P J (j)1{j = ĵ} = 1 ĵ QĴ(ĵ) 1 M = 1 1 M Vincent Tan (NUS) Second-Order Asymptotics NTU 61 / 109
163 Proof of Hayashi-Nagaoka Lemma : Part III By the definition of the hypothesis-testing divergence, D ε h(p JĴ P J QĴ) log M + log(1 ε) Vincent Tan (NUS) Second-Order Asymptotics NTU 62 / 109
164 Proof of Hayashi-Nagaoka Lemma : Part III By the definition of the hypothesis-testing divergence, D ε h(p JĴ P J QĴ) log M + log(1 ε) By the relation between D ε s and D ε h, log M D ε h(p W P Q) + log 1 1 ε D ε+η s (P W P Q) + log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 62 / 109
165 Proof of Hayashi-Nagaoka Lemma : Part III By the definition of the hypothesis-testing divergence, D ε h(p JĴ P J QĴ) log M + log(1 ε) By the relation between D ε s and D ε h, log M D ε h(p W P Q) + log 1 1 ε D ε+η s (P W P Q) + log 1 η Maximize over P to make the bound code-independent log M sup P Ds ε+η (P W P Q) + log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 62 / 109
166 Proof of Hayashi-Nagaoka Lemma : Part III By the definition of the hypothesis-testing divergence, D ε h(p JĴ P J QĴ) log M + log(1 ε) By the relation between D ε s and D ε h, log M D ε h(p W P Q) + log 1 1 ε D ε+η s (P W P Q) + log 1 η Maximize over P to make the bound code-independent log M sup P Q is a free parameter. Minimize over it. Ds ε+η (P W P Q) + log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 62 / 109
167 Second-Order Converse for Channel Coding : Part I Fix an (M, ε) av -code. Starting from the Hayashi-Nagaoka converse for the channel W n, we obtain log M sup P for any fixed Q P(Y n ). D ε+η s (P W n P Q) + log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 63 / 109
168 Second-Order Converse for Channel Coding : Part I Fix an (M, ε) av -code. Starting from the Hayashi-Nagaoka converse for the channel W n, we obtain log M sup P for any fixed Q P(Y n ). D ε+η s (P W n P Q) + log 1 η We can replace the optimization over P P(X n ) to an optimization over input sequences x n X n : log M max x n D ε+η s (W n ( x n ) Q) + log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 63 / 109
169 Second-Order Converse for Channel Coding : Part I Fix an (M, ε) av -code. Starting from the Hayashi-Nagaoka converse for the channel W n, we obtain log M sup P for any fixed Q P(Y n ). D ε+η s (P W n P Q) + log 1 η We can replace the optimization over P P(X n ) to an optimization over input sequences x n X n : log M max x n D ε+η s (W n ( x n ) Q) + log 1 η Now choose Q P(Y n ) to be the convex combination Q(y n 1 ) = (PW) n (y n ) P n (X ) P P n(x ) Vincent Tan (NUS) Second-Order Asymptotics NTU 63 / 109
170 Second-Order Converse for Channel Coding : Part II Lemma Let θ i 0 be such that i θ i = 1. Then ( ) D ε s P θ i Q i inf {D εs(p Q } i ) + log 1θi i i Vincent Tan (NUS) Second-Order Asymptotics NTU 64 / 109
171 Second-Order Converse for Channel Coding : Part II Lemma Let θ i 0 be such that i θ i = 1. Then ( ) D ε s P θ i Q i inf {D εs(p Q } i ) + log 1θi i i From the previous derivations, log M max D ε+η x n s W n ( x n ) 1 P n (X ) P P n(x ) (PW) n + log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 64 / 109
172 Second-Order Converse for Channel Coding : Part II Lemma Let θ i 0 be such that i θ i = 1. Then ( ) D ε s P θ i Q i inf {D εs(p Q } i ) + log 1θi i i From the previous derivations, log M max D ε+η x n s W n ( x n ) 1 P n (X ) P P n(x ) (PW) n + log 1 η By sieving out the type of x n, one has log M max x n D ε+η s (W n ( x n ) (ˆP x nw) n ) + log P n (X ) + log 1 η Vincent Tan (NUS) Second-Order Asymptotics NTU 64 / 109
173 Second-Order Converse for Channel Coding : Part III log M max x n D ε+η s (W n ( x n ) (ˆP x nw) n ) + log P n (X ) + log 1 η }{{} =O(log n) Choose η = 1 n. By the asymptotic expansion of the information spectrum divergence, log M max ni(p, W) + nv(p, W)Φ 1 (ε) + O(log n). P P n(x ) where the conditional information variance is V(P, W) = P(x) [ W(y x) log W(y x) PW(y) D(W( x) PW) x y ] 2 Vincent Tan (NUS) Second-Order Asymptotics NTU 65 / 109
174 Second-Order Converse for Channel Coding : Part III log M max x n D ε+η s (W n ( x n ) (ˆP x nw) n ) + log P n (X ) + log 1 η }{{} =O(log n) Choose η = 1 n. By the asymptotic expansion of the information spectrum divergence, log M max ni(p, W) + nv(p, W)Φ 1 (ε) + O(log n). P P n(x ) where the conditional information variance is V(P, W) = P(x) [ W(y x) log W(y x) PW(y) D(W( x) PW) x y ] 2 Now, invoke continuity of P I(P, W) and P V(P, W) to replace P with P above Vincent Tan (NUS) Second-Order Asymptotics NTU 65 / 109
175 Second-Order Converse for Channel Coding : Part IV Lemma For a DMC with unique capacity-achieving input distribution P, log M av(w n, ε) nc + nv(p, W)Φ 1 (ε) + O(log n) Vincent Tan (NUS) Second-Order Asymptotics NTU 66 / 109
176 Second-Order Converse for Channel Coding : Part IV Lemma For a DMC with unique capacity-achieving input distribution P, log M av(w n, ε) nc + nv(p, W)Φ 1 (ε) + O(log n) This is almost the same as the achievability log M av(w n, ε) nc + nu(p, W)Φ 1 (ε) + O(log n) Vincent Tan (NUS) Second-Order Asymptotics NTU 66 / 109
177 Second-Order Converse for Channel Coding : Part IV Lemma For a DMC with unique capacity-achieving input distribution P, log M av(w n, ε) nc + nv(p, W)Φ 1 (ε) + O(log n) This is almost the same as the achievability log M av(w n, ε) nc + nu(p, W)Φ 1 (ε) + O(log n) So is U(P, W) = V(P, W)?? Vincent Tan (NUS) Second-Order Asymptotics NTU 66 / 109
178 Second-Order Asymptotics for Channel Coding Conditional information variance is V(P, W) = P(x) [ W(y x) log W(y x) PW(y) D(W( x) PW) x y Unconditional information variance is U(P, W) = x P(x) y W(y x) [ log W(y x) ] 2 PW(y) C ] 2 Vincent Tan (NUS) Second-Order Asymptotics NTU 67 / 109
179 Second-Order Asymptotics for Channel Coding Conditional information variance is V(P, W) = P(x) [ W(y x) log W(y x) PW(y) D(W( x) PW) x y Unconditional information variance is U(P, W) = x P(x) y W(y x) [ log W(y x) ] 2 PW(y) C ] 2 In general, if (X, Y) P W, var(i(x; Y)) = U(P, W) V(P, W) = E[var(i(X; Y)) X] by the law of total variance. Vincent Tan (NUS) Second-Order Asymptotics NTU 67 / 109
180 Second-Order Asymptotics for Channel Coding If we choose P to be capacity-achieving (i.e., I(P, W) = C), then by the KKT conditions, D(W( x) P W) = C Vincent Tan (NUS) Second-Order Asymptotics NTU 68 / 109
181 Second-Order Asymptotics for Channel Coding If we choose P to be capacity-achieving (i.e., I(P, W) = C), then by the KKT conditions, D(W( x) P W) = C and so V(P, W) = x = x P(x) y P(x) y [ W(y x) log W(y x) ] 2 P W(y) D(W( x) P W) [ W(y x) log W(y x) ] 2 P W(y) C = U(P, W) Vincent Tan (NUS) Second-Order Asymptotics NTU 68 / 109
182 Second-Order Asymptotics for Channel Coding Theorem (Strassen (1962)) For a DMC with unique capacity-achieving input distribution P, log Mav(W n, ε) = nc + nvφ 1 (ε) + O(log n) where V = V(P, W) Vincent Tan (NUS) Second-Order Asymptotics NTU 69 / 109
183 Second-Order Asymptotics for Channel Coding Theorem (Strassen (1962)) For a DMC with unique capacity-achieving input distribution P, log Mav(W n, ε) = nc + nvφ 1 (ε) + O(log n) where V = V(P, W) Direct part based on Feinstein Vincent Tan (NUS) Second-Order Asymptotics NTU 69 / 109
184 Second-Order Asymptotics for Channel Coding Theorem (Strassen (1962)) For a DMC with unique capacity-achieving input distribution P, log Mav(W n, ε) = nc + nvφ 1 (ε) + O(log n) where V = V(P, W) Direct part based on Feinstein Converse part based on Hayashi-Nagaoka with clever choice of Q Vincent Tan (NUS) Second-Order Asymptotics NTU 69 / 109
185 Second-Order Asymptotics for Channel Coding Theorem (Strassen (1962)) For a DMC with unique capacity-achieving input distribution P, log Mav(W n, ε) = nc + nvφ 1 (ε) + O(log n) where V = V(P, W) Direct part based on Feinstein Converse part based on Hayashi-Nagaoka with clever choice of Q We can optimize the third-order term; Usually O(log n) = 1 log n + O(1) 2 See Tomamichel-Tan (2013) and Altuğ-Wagner (2014). Vincent Tan (NUS) Second-Order Asymptotics NTU 69 / 109
186 Summary : Channel Coding Derived the second-order asymptotic expansions for DMCs and AWGN channels Vincent Tan (NUS) Second-Order Asymptotics NTU 70 / 109
187 Summary : Channel Coding Derived the second-order asymptotic expansions for DMCs and AWGN channels Achievability hinges on Feinstein s lemma or its generalized version Vincent Tan (NUS) Second-Order Asymptotics NTU 70 / 109
188 Summary : Channel Coding Derived the second-order asymptotic expansions for DMCs and AWGN channels Achievability hinges on Feinstein s lemma or its generalized version Converse hinges on Hayashi-Nagaoka s lemma with a good choice of output distribution Vincent Tan (NUS) Second-Order Asymptotics NTU 70 / 109
189 Outline 1 Motivation, Background and History 2 Binary Hypothesis Testing 3 Fixed-Length Lossless Source Coding 4 Channel Coding 5 Slepian-Wolf Coding 6 Summary and Open Problems Vincent Tan (NUS) Second-Order Asymptotics NTU 71 / 109
190 Setup of the Slepian-Wolf coding problem x1 n m 1 f 1 ϕ (ˆx 1 n, ˆxn 2 ) x n 2 f 2 m 2 Illustration of the Slepian-Wolf problem. Vincent Tan (NUS) Second-Order Asymptotics NTU 72 / 109
191 Setup of the Slepian-Wolf coding problem x1 n m 1 f 1 ϕ (ˆx 1 n, ˆxn 2 ) x n 2 f 2 m 2 Illustration of the Slepian-Wolf problem. Two correlated sources (X n 1, Xn 2 ) n i=1 P X 1 X 2 (x 1i, x 2i ). Vincent Tan (NUS) Second-Order Asymptotics NTU 72 / 109
192 Setup of the Slepian-Wolf coding problem x1 n m 1 f 1 ϕ (ˆx 1 n, ˆxn 2 ) x n 2 f 2 m 2 Illustration of the Slepian-Wolf problem. Two correlated sources (X n 1, Xn 2 ) n i=1 P X 1 X 2 (x 1i, x 2i ). Separately encoded Vincent Tan (NUS) Second-Order Asymptotics NTU 72 / 109
193 Setup of the Slepian-Wolf coding problem x1 n m 1 f 1 ϕ (ˆx 1 n, ˆxn 2 ) x n 2 f 2 m 2 Illustration of the Slepian-Wolf problem. Two correlated sources (X n 1, Xn 2 ) n i=1 P X 1 X 2 (x 1i, x 2i ). Separately encoded Both to be decoded at destination Vincent Tan (NUS) Second-Order Asymptotics NTU 72 / 109
194 The Slepian-Wolf theorem Sources to be compressed to nr 1 and nr 2 bits respectively. Vincent Tan (NUS) Second-Order Asymptotics NTU 73 / 109
195 The Slepian-Wolf theorem Sources to be compressed to nr 1 and nr 2 bits respectively. (R 1, R 2 ) achievable if there exists a sequence of (2 nr 1, 2 nr 2, n)-codes such that lim Pr ( (ˆX n n 1, ˆX 2 n ) (Xn 1, Xn 2 )) = 0. R(P XY ) is the set of all achievable (R 1, R 2 ) pairs. Vincent Tan (NUS) Second-Order Asymptotics NTU 73 / 109
196 The Slepian-Wolf theorem Sources to be compressed to nr 1 and nr 2 bits respectively. (R 1, R 2 ) achievable if there exists a sequence of (2 nr 1, 2 nr 2, n)-codes such that lim Pr ( (ˆX n n 1, ˆX 2 n ) (Xn 1, Xn 2 )) = 0. R(P XY ) is the set of all achievable (R 1, R 2 ) pairs. Slepian and Wolf (1973) R(P XY ) = {R 1 H(X 1 X 2 ), R 2 H(X 2 X 1 ), R 1 + R 2 H(X 1, X 2 )} D. Slepian J. Wolf Vincent Tan (NUS) Second-Order Asymptotics NTU 73 / 109
197 The Slepian-Wolf region R 2 H 2 H 2 1 R(P XY ) H 1 2 H 1 R 1 R 1 H(X 1 X 2 ) R 2 H(X 2 X 1 ) R 1 + R 2 H(X 1, X 2 ) Vincent Tan (NUS) Second-Order Asymptotics NTU 74 / 109
198 A Review of Slepian-Wolf coding Partition X n j randomly into exp(nr j ) bins, i.e., Pr(x n j B j (m j )) = exp( nr j ), j = 1, 2 Vincent Tan (NUS) Second-Order Asymptotics NTU 75 / 109
199 A Review of Slepian-Wolf coding Partition X n j randomly into exp(nr j ) bins, i.e., Pr(x n j B j (m j )) = exp( nr j ), j = 1, 2 Transmit bin index m j [1 : exp(nr j )] if X n j B(m j ). Vincent Tan (NUS) Second-Order Asymptotics NTU 75 / 109
200 A Review of Slepian-Wolf coding Partition X n j randomly into exp(nr j ) bins, i.e., Pr(x n j B j (m j )) = exp( nr j ), j = 1, 2 Transmit bin index m j [1 : exp(nr j )] if X n j B(m j ). At decoder, declare that (X n 1, Xn 2 ) B 1(m 1 ) B 2 (m 2 ) are the transmitted vectors iff (X n 1, Xn 2 ) T ɛ Vincent Tan (NUS) Second-Order Asymptotics NTU 75 / 109
201 A Review of Slepian-Wolf coding Partition X n j randomly into exp(nr j ) bins, i.e., Pr(x n j B j (m j )) = exp( nr j ), j = 1, 2 Transmit bin index m j [1 : exp(nr j )] if X n j B(m j ). At decoder, declare that (X n 1, Xn 2 ) B 1(m 1 ) B 2 (m 2 ) are the transmitted vectors iff (X n 1, Xn 2 ) T ɛ By standard typicality arguments, we need R 1 H(X 1 X 2 ) R 2 H(X 2 X 1 ) R 1 + R 2 H(X 1, X 2 ) Vincent Tan (NUS) Second-Order Asymptotics NTU 75 / 109
202 Setup for Second-Order Asymptotics for Slepian-Wolf We chose M 1 = exp(nr 1 ), M 2 = exp(nr 2 ) for the optimum rate region and sought rates (R 1, R 2 ) such that Pr ( (ˆX 1 n, ˆX 2 n ) (Xn 1, Xn 2 )) 0. Vincent Tan (NUS) Second-Order Asymptotics NTU 76 / 109
203 Setup for Second-Order Asymptotics for Slepian-Wolf We chose M 1 = exp(nr 1 ), M 2 = exp(nr 2 ) for the optimum rate region and sought rates (R 1, R 2 ) such that Pr ( (ˆX 1 n, ˆX 2 n ) (Xn 1, Xn 2 )) 0. Alternatively, we can fix (R 1, R 2 ) Bd(R(P XY)) and ε (0, 1) and choose M 1 = exp(nr 1 + nl 1 ), M 2 = exp(nr 2 + nl 2 ) Then seek (L 1, L 2 ) pair such that Pr ( (ˆX n 1, ˆX n 2 ) (Xn 1, Xn 2 )) ε + o(1). Vincent Tan (NUS) Second-Order Asymptotics NTU 76 / 109
204 Setup for Second-Order Asymptotics for Slepian-Wolf Note that we re operating on the boundary of the SW region! R 2 Bd(R(P XY )) H 2 H 2 1 (R 1, R 2 ) R(P XY ) H 1 2 H 1 R 1 Vincent Tan (NUS) Second-Order Asymptotics NTU 77 / 109
205 Definitions for Second-Order Asymptotics for SW (L 1, L 2 ) R 2 is (R 1, R 2, ε)-achievable if there exists a sequence of (n, M 1n, M 2n, ε n )-codes such that lim sup n lim sup n 1 n (log M 1n nr 1) L 1 1 n (log M 2n nr 2) L 2 and lim sup ε n ε. n Vincent Tan (NUS) Second-Order Asymptotics NTU 78 / 109
206 Definitions for Second-Order Asymptotics for SW (L 1, L 2 ) R 2 is (R 1, R 2, ε)-achievable if there exists a sequence of (n, M 1n, M 2n, ε n )-codes such that lim sup n lim sup n 1 n (log M 1n nr 1) L 1 1 n (log M 2n nr 2) L 2 and lim sup ε n ε. n L(ε; R 1, R 2 ) is the set of all (R 1, R 2, ε)-achievable (L 1, L 2 ) pairs Vincent Tan (NUS) Second-Order Asymptotics NTU 78 / 109
207 Second-Order Asymptotics for SW Joint work with Oliver Kosut Paper published in Feb 2014 issue of IT Transactions Vincent Tan (NUS) Second-Order Asymptotics NTU 79 / 109
On Third-Order Asymptotics for DMCs
On Third-Order Asymptotics for DMCs Vincent Y. F. Tan Institute for Infocomm Research (I R) National University of Singapore (NUS) January 0, 013 Vincent Tan (I R and NUS) Third-Order Asymptotics for DMCs
More informationAsymptotic Estimates in Information Theory with Non-Vanishing Error Probabilities
Asymptotic Estimates in Information Theory with Non-Vanishing Error Probabilities Vincent Y. F. Tan Dept. of ECE and Dept. of Mathematics National University of Singapore (NUS) September 2014 Vincent Tan
More informationEE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15
EE5139R: Problem Set 7 Assigned: 30/09/15, Due: 07/10/15 1. Cascade of Binary Symmetric Channels The conditional probability distribution py x for each of the BSCs may be expressed by the transition probability
More informationA Tight Upper Bound on the Second-Order Coding Rate of Parallel Gaussian Channels with Feedback
A Tight Upper Bound on the Second-Order Coding Rate of Parallel Gaussian Channels with Feedback Vincent Y. F. Tan (NUS) Joint work with Silas L. Fong (Toronto) 2017 Information Theory Workshop, Kaohsiung,
More informationLecture 4 Noisy Channel Coding
Lecture 4 Noisy Channel Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 9, 2015 1 / 56 I-Hsiang Wang IT Lecture 4 The Channel Coding Problem
More informationSecond-Order Asymptotics for the Gaussian MAC with Degraded Message Sets
Second-Order Asymptotics for the Gaussian MAC with Degraded Message Sets Jonathan Scarlett and Vincent Y. F. Tan Department of Engineering, University of Cambridge Electrical and Computer Engineering,
More informationLecture 4 Channel Coding
Capacity and the Weak Converse Lecture 4 Coding I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw October 15, 2014 1 / 16 I-Hsiang Wang NIT Lecture 4 Capacity
More informationTwo Applications of the Gaussian Poincaré Inequality in the Shannon Theory
Two Applications of the Gaussian Poincaré Inequality in the Shannon Theory Vincent Y. F. Tan (Joint work with Silas L. Fong) National University of Singapore (NUS) 2016 International Zurich Seminar on
More informationDispersion of the Gilbert-Elliott Channel
Dispersion of the Gilbert-Elliott Channel Yury Polyanskiy Email: ypolyans@princeton.edu H. Vincent Poor Email: poor@princeton.edu Sergio Verdú Email: verdu@princeton.edu Abstract Channel dispersion plays
More informationChannel Dispersion and Moderate Deviations Limits for Memoryless Channels
Channel Dispersion and Moderate Deviations Limits for Memoryless Channels Yury Polyanskiy and Sergio Verdú Abstract Recently, Altug and Wagner ] posed a question regarding the optimal behavior of the probability
More informationNational University of Singapore Department of Electrical & Computer Engineering. Examination for
National University of Singapore Department of Electrical & Computer Engineering Examination for EE5139R Information Theory for Communication Systems (Semester I, 2014/15) November/December 2014 Time Allowed:
More informationLecture 5 Channel Coding over Continuous Channels
Lecture 5 Channel Coding over Continuous Channels I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw November 14, 2014 1 / 34 I-Hsiang Wang NIT Lecture 5 From
More informationCovert Communication with Channel-State Information at the Transmitter
Covert Communication with Channel-State Information at the Transmitter Si-Hyeon Lee Joint Work with Ligong Wang, Ashish Khisti, and Gregory W. Wornell July 27, 2017 1 / 21 Covert Communication Transmitter
More informationChapter 4. Data Transmission and Channel Capacity. Po-Ning Chen, Professor. Department of Communications Engineering. National Chiao Tung University
Chapter 4 Data Transmission and Channel Capacity Po-Ning Chen, Professor Department of Communications Engineering National Chiao Tung University Hsin Chu, Taiwan 30050, R.O.C. Principle of Data Transmission
More informationSecret Key Agreement: General Capacity and Second-Order Asymptotics. Masahito Hayashi Himanshu Tyagi Shun Watanabe
Secret Key Agreement: General Capacity and Second-Order Asymptotics Masahito Hayashi Himanshu Tyagi Shun Watanabe Two party secret key agreement Maurer 93, Ahlswede-Csiszár 93 X F Y K x K y ArandomvariableK
More informationELEC546 Review of Information Theory
ELEC546 Review of Information Theory Vincent Lau 1/1/004 1 Review of Information Theory Entropy: Measure of uncertainty of a random variable X. The entropy of X, H(X), is given by: If X is a discrete random
More informationA Hierarchy of Information Quantities for Finite Block Length Analysis of Quantum Tasks
A Hierarchy of Information Quantities for Finite Block Length Analysis of Quantum Tasks Marco Tomamichel, Masahito Hayashi arxiv: 1208.1478 Also discussing results of: Second Order Asymptotics for Quantum
More informationCorrelation Detection and an Operational Interpretation of the Rényi Mutual Information
Correlation Detection and an Operational Interpretation of the Rényi Mutual Information Masahito Hayashi 1, Marco Tomamichel 2 1 Graduate School of Mathematics, Nagoya University, and Centre for Quantum
More informationStrong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach
Strong Converse Theorems for Classes of Multimessage Multicast Networks: A Rényi Divergence Approach Silas Fong (Joint work with Vincent Tan) Department of Electrical & Computer Engineering National University
More informationfor some error exponent E( R) as a function R,
. Capacity-achieving codes via Forney concatenation Shannon s Noisy Channel Theorem assures us the existence of capacity-achieving codes. However, exhaustive search for the code has double-exponential
More informationA new converse in rate-distortion theory
A new converse in rate-distortion theory Victoria Kostina, Sergio Verdú Dept. of Electrical Engineering, Princeton University, NJ, 08544, USA Abstract This paper shows new finite-blocklength converse bounds
More informationQuantum Sphere-Packing Bounds and Moderate Deviation Analysis for Classical-Quantum Channels
Quantum Sphere-Packing Bounds and Moderate Deviation Analysis for Classical-Quantum Channels (, ) Joint work with Min-Hsiu Hsieh and Marco Tomamichel Hao-Chung Cheng University of Technology Sydney National
More informationECE Information theory Final (Fall 2008)
ECE 776 - Information theory Final (Fall 2008) Q.1. (1 point) Consider the following bursty transmission scheme for a Gaussian channel with noise power N and average power constraint P (i.e., 1/n X n i=1
More informationChannels with cost constraints: strong converse and dispersion
Channels with cost constraints: strong converse and dispersion Victoria Kostina, Sergio Verdú Dept. of Electrical Engineering, Princeton University, NJ 08544, USA Abstract This paper shows the strong converse
More informationLecture 22: Final Review
Lecture 22: Final Review Nuts and bolts Fundamental questions and limits Tools Practical algorithms Future topics Dr Yao Xie, ECE587, Information Theory, Duke University Basics Dr Yao Xie, ECE587, Information
More informationECE Information theory Final
ECE 776 - Information theory Final Q1 (1 point) We would like to compress a Gaussian source with zero mean and variance 1 We consider two strategies In the first, we quantize with a step size so that the
More informationLecture 6 I. CHANNEL CODING. X n (m) P Y X
6- Introduction to Information Theory Lecture 6 Lecturer: Haim Permuter Scribe: Yoav Eisenberg and Yakov Miron I. CHANNEL CODING We consider the following channel coding problem: m = {,2,..,2 nr} Encoder
More informationA Tight Upper Bound for the Third-Order Asymptotics for Most Discrete Memoryless Channels
A Tight Upper Bound for the Third-Order Asymptotics for Most Discrete Memoryless Channels Marco Tomamichel, Member, IEEE and Vincent Y. F. Tan, Member, IEEE Abstract arxiv:.3689v3 [cs.it] 4 Oct 03 This
More informationCapacity of a channel Shannon s second theorem. Information Theory 1/33
Capacity of a channel Shannon s second theorem Information Theory 1/33 Outline 1. Memoryless channels, examples ; 2. Capacity ; 3. Symmetric channels ; 4. Channel Coding ; 5. Shannon s second theorem,
More informationRefined Bounds on the Empirical Distribution of Good Channel Codes via Concentration Inequalities
Refined Bounds on the Empirical Distribution of Good Channel Codes via Concentration Inequalities Maxim Raginsky and Igal Sason ISIT 2013, Istanbul, Turkey Capacity-Achieving Channel Codes The set-up DMC
More informationNecessary and Sufficient Conditions for High-Dimensional Salient Feature Subset Recovery
Necessary and Sufficient Conditions for High-Dimensional Salient Feature Subset Recovery Vincent Tan, Matt Johnson, Alan S. Willsky Stochastic Systems Group, Laboratory for Information and Decision Systems,
More informationSoft Covering with High Probability
Soft Covering with High Probability Paul Cuff Princeton University arxiv:605.06396v [cs.it] 20 May 206 Abstract Wyner s soft-covering lemma is the central analysis step for achievability proofs of information
More informationUpper Bounds on the Capacity of Binary Intermittent Communication
Upper Bounds on the Capacity of Binary Intermittent Communication Mostafa Khoshnevisan and J. Nicholas Laneman Department of Electrical Engineering University of Notre Dame Notre Dame, Indiana 46556 Email:{mhoshne,
More informationAppendix B Information theory from first principles
Appendix B Information theory from first principles This appendix discusses the information theory behind the capacity expressions used in the book. Section 8.3.4 is the only part of the book that supposes
More informationShannon s noisy-channel theorem
Shannon s noisy-channel theorem Information theory Amon Elders Korteweg de Vries Institute for Mathematics University of Amsterdam. Tuesday, 26th of Januari Amon Elders (Korteweg de Vries Institute for
More informationShannon s Noisy-Channel Coding Theorem
Shannon s Noisy-Channel Coding Theorem Lucas Slot Sebastian Zur February 13, 2015 Lucas Slot, Sebastian Zur Shannon s Noisy-Channel Coding Theorem February 13, 2015 1 / 29 Outline 1 Definitions and Terminology
More informationChapter 9 Fundamental Limits in Information Theory
Chapter 9 Fundamental Limits in Information Theory Information Theory is the fundamental theory behind information manipulation, including data compression and data transmission. 9.1 Introduction o For
More informationLecture 18: Shanon s Channel Coding Theorem. Lecture 18: Shanon s Channel Coding Theorem
Channel Definition (Channel) A channel is defined by Λ = (X, Y, Π), where X is the set of input alphabets, Y is the set of output alphabets and Π is the transition probability of obtaining a symbol y Y
More informationArimoto Channel Coding Converse and Rényi Divergence
Arimoto Channel Coding Converse and Rényi Divergence Yury Polyanskiy and Sergio Verdú Abstract Arimoto proved a non-asymptotic upper bound on the probability of successful decoding achievable by any code
More informationLecture 3: Channel Capacity
Lecture 3: Channel Capacity 1 Definitions Channel capacity is a measure of maximum information per channel usage one can get through a channel. This one of the fundamental concepts in information theory.
More informationX 1 : X Table 1: Y = X X 2
ECE 534: Elements of Information Theory, Fall 200 Homework 3 Solutions (ALL DUE to Kenneth S. Palacio Baus) December, 200. Problem 5.20. Multiple access (a) Find the capacity region for the multiple-access
More informationEE 4TM4: Digital Communications II. Channel Capacity
EE 4TM4: Digital Communications II 1 Channel Capacity I. CHANNEL CODING THEOREM Definition 1: A rater is said to be achievable if there exists a sequence of(2 nr,n) codes such thatlim n P (n) e (C) = 0.
More informationMemory in Classical Information Theory: A Brief History
Beyond iid in information theory 8- January, 203 Memory in Classical Information Theory: A Brief History sergio verdu princeton university entropy rate Shannon, 948 entropy rate: memoryless process H(X)
More informationThe Method of Types and Its Application to Information Hiding
The Method of Types and Its Application to Information Hiding Pierre Moulin University of Illinois at Urbana-Champaign www.ifp.uiuc.edu/ moulin/talks/eusipco05-slides.pdf EUSIPCO Antalya, September 7,
More informationA New Metaconverse and Outer Region for Finite-Blocklength MACs
A New Metaconverse Outer Region for Finite-Blocklength MACs Pierre Moulin Dept of Electrical Computer Engineering University of Illinois at Urbana-Champaign Urbana, IL 680 Abstract An outer rate region
More informationQuantum Achievability Proof via Collision Relative Entropy
Quantum Achievability Proof via Collision Relative Entropy Salman Beigi Institute for Research in Fundamental Sciences (IPM) Tehran, Iran Setemper 8, 2014 Based on a joint work with Amin Gohari arxiv:1312.3822
More informationEntropies & Information Theory
Entropies & Information Theory LECTURE I Nilanjana Datta University of Cambridge,U.K. See lecture notes on: http://www.qi.damtp.cam.ac.uk/node/223 Quantum Information Theory Born out of Classical Information
More informationEE5139R: Problem Set 4 Assigned: 31/08/16, Due: 07/09/16
EE539R: Problem Set 4 Assigned: 3/08/6, Due: 07/09/6. Cover and Thomas: Problem 3.5 Sets defined by probabilities: Define the set C n (t = {x n : P X n(x n 2 nt } (a We have = P X n(x n P X n(x n 2 nt
More informationlossless, optimal compressor
6. Variable-length Lossless Compression The principal engineering goal of compression is to represent a given sequence a, a 2,..., a n produced by a source as a sequence of bits of minimal possible length.
More informationLecture 7. Union bound for reducing M-ary to binary hypothesis testing
Lecture 7 Agenda for the lecture M-ary hypothesis testing and the MAP rule Union bound for reducing M-ary to binary hypothesis testing Introduction of the channel coding problem 7.1 M-ary hypothesis testing
More informationPerformance-based Security for Encoding of Information Signals. FA ( ) Paul Cuff (Princeton University)
Performance-based Security for Encoding of Information Signals FA9550-15-1-0180 (2015-2018) Paul Cuff (Princeton University) Contributors Two students finished PhD Tiance Wang (Goldman Sachs) Eva Song
More informationLecture 5: Channel Capacity. Copyright G. Caire (Sample Lectures) 122
Lecture 5: Channel Capacity Copyright G. Caire (Sample Lectures) 122 M Definitions and Problem Setup 2 X n Y n Encoder p(y x) Decoder ˆM Message Channel Estimate Definition 11. Discrete Memoryless Channel
More informationLECTURE 13. Last time: Lecture outline
LECTURE 13 Last time: Strong coding theorem Revisiting channel and codes Bound on probability of error Error exponent Lecture outline Fano s Lemma revisited Fano s inequality for codewords Converse to
More informationGeneralized Writing on Dirty Paper
Generalized Writing on Dirty Paper Aaron S. Cohen acohen@mit.edu MIT, 36-689 77 Massachusetts Ave. Cambridge, MA 02139-4307 Amos Lapidoth lapidoth@isi.ee.ethz.ch ETF E107 ETH-Zentrum CH-8092 Zürich, Switzerland
More informationThe PPM Poisson Channel: Finite-Length Bounds and Code Design
August 21, 2014 The PPM Poisson Channel: Finite-Length Bounds and Code Design Flavio Zabini DEI - University of Bologna and Institute for Communications and Navigation German Aerospace Center (DLR) Balazs
More informationLecture 7 Introduction to Statistical Decision Theory
Lecture 7 Introduction to Statistical Decision Theory I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw December 20, 2016 1 / 55 I-Hsiang Wang IT Lecture 7
More informationFeedback Capacity of a Class of Symmetric Finite-State Markov Channels
Feedback Capacity of a Class of Symmetric Finite-State Markov Channels Nevroz Şen, Fady Alajaji and Serdar Yüksel Department of Mathematics and Statistics Queen s University Kingston, ON K7L 3N6, Canada
More informationInformation measures in simple coding problems
Part I Information measures in simple coding problems in this web service in this web service Source coding and hypothesis testing; information measures A(discrete)source is a sequence {X i } i= of random
More informationNetwork coding for multicast relation to compression and generalization of Slepian-Wolf
Network coding for multicast relation to compression and generalization of Slepian-Wolf 1 Overview Review of Slepian-Wolf Distributed network compression Error exponents Source-channel separation issues
More informationDiscrete Memoryless Channels with Memoryless Output Sequences
Discrete Memoryless Channels with Memoryless utput Sequences Marcelo S Pinho Department of Electronic Engineering Instituto Tecnologico de Aeronautica Sao Jose dos Campos, SP 12228-900, Brazil Email: mpinho@ieeeorg
More informationLecture 2: August 31
0-704: Information Processing and Learning Fall 206 Lecturer: Aarti Singh Lecture 2: August 3 Note: These notes are based on scribed notes from Spring5 offering of this course. LaTeX template courtesy
More informationRecent Results on Input-Constrained Erasure Channels
Recent Results on Input-Constrained Erasure Channels A Case Study for Markov Approximation July, 2017@NUS Memoryless Channels Memoryless Channels Channel transitions are characterized by time-invariant
More informationLecture 11: Polar codes construction
15-859: Information Theory and Applications in TCS CMU: Spring 2013 Lecturer: Venkatesan Guruswami Lecture 11: Polar codes construction February 26, 2013 Scribe: Dan Stahlke 1 Polar codes: recap of last
More informationEE/Stat 376B Handout #5 Network Information Theory October, 14, Homework Set #2 Solutions
EE/Stat 376B Handout #5 Network Information Theory October, 14, 014 1. Problem.4 parts (b) and (c). Homework Set # Solutions (b) Consider h(x + Y ) h(x + Y Y ) = h(x Y ) = h(x). (c) Let ay = Y 1 + Y, where
More informationExercise 1. = P(y a 1)P(a 1 )
Chapter 7 Channel Capacity Exercise 1 A source produces independent, equally probable symbols from an alphabet {a 1, a 2 } at a rate of one symbol every 3 seconds. These symbols are transmitted over a
More informationLecture 8: Shannon s Noise Models
Error Correcting Codes: Combinatorics, Algorithms and Applications (Fall 2007) Lecture 8: Shannon s Noise Models September 14, 2007 Lecturer: Atri Rudra Scribe: Sandipan Kundu& Atri Rudra Till now we have
More informationAn Extended Fano s Inequality for the Finite Blocklength Coding
An Extended Fano s Inequality for the Finite Bloclength Coding Yunquan Dong, Pingyi Fan {dongyq8@mails,fpy@mail}.tsinghua.edu.cn Department of Electronic Engineering, Tsinghua University, Beijing, P.R.
More informationLecture 11: Quantum Information III - Source Coding
CSCI5370 Quantum Computing November 25, 203 Lecture : Quantum Information III - Source Coding Lecturer: Shengyu Zhang Scribe: Hing Yin Tsang. Holevo s bound Suppose Alice has an information source X that
More informationReliable Computation over Multiple-Access Channels
Reliable Computation over Multiple-Access Channels Bobak Nazer and Michael Gastpar Dept. of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA, 94720-1770 {bobak,
More informationCommunications Theory and Engineering
Communications Theory and Engineering Master's Degree in Electronic Engineering Sapienza University of Rome A.A. 2018-2019 AEP Asymptotic Equipartition Property AEP In information theory, the analog of
More informationSHARED INFORMATION. Prakash Narayan with. Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe
SHARED INFORMATION Prakash Narayan with Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe 2/40 Acknowledgement Praneeth Boda Himanshu Tyagi Shun Watanabe 3/40 Outline Two-terminal model: Mutual
More informationVariable Rate Channel Capacity. Jie Ren 2013/4/26
Variable Rate Channel Capacity Jie Ren 2013/4/26 Reference This is a introduc?on of Sergio Verdu and Shlomo Shamai s paper. Sergio Verdu and Shlomo Shamai, Variable- Rate Channel Capacity, IEEE Transac?ons
More informationCapacity of the Discrete Memoryless Energy Harvesting Channel with Side Information
204 IEEE International Symposium on Information Theory Capacity of the Discrete Memoryless Energy Harvesting Channel with Side Information Omur Ozel, Kaya Tutuncuoglu 2, Sennur Ulukus, and Aylin Yener
More informationThe Poisson Channel with Side Information
The Poisson Channel with Side Information Shraga Bross School of Enginerring Bar-Ilan University, Israel brosss@macs.biu.ac.il Amos Lapidoth Ligong Wang Signal and Information Processing Laboratory ETH
More informationVariable Length Codes for Degraded Broadcast Channels
Variable Length Codes for Degraded Broadcast Channels Stéphane Musy School of Computer and Communication Sciences, EPFL CH-1015 Lausanne, Switzerland Email: stephane.musy@ep.ch Abstract This paper investigates
More informationA Formula for the Capacity of the General Gel fand-pinsker Channel
A Formula for the Capacity of the General Gel fand-pinsker Channel Vincent Y. F. Tan Institute for Infocomm Research (I 2 R, A*STAR, Email: tanyfv@i2r.a-star.edu.sg ECE Dept., National University of Singapore
More information(Classical) Information Theory III: Noisy channel coding
(Classical) Information Theory III: Noisy channel coding Sibasish Ghosh The Institute of Mathematical Sciences CIT Campus, Taramani, Chennai 600 113, India. p. 1 Abstract What is the best possible way
More informationOn Composite Quantum Hypothesis Testing
University of York 7 November 207 On Composite Quantum Hypothesis Testing Mario Berta Department of Computing with Fernando Brandão and Christoph Hirche arxiv:709.07268 Overview Introduction 2 Composite
More informationECE 4400:693 - Information Theory
ECE 4400:693 - Information Theory Dr. Nghi Tran Lecture 8: Differential Entropy Dr. Nghi Tran (ECE-University of Akron) ECE 4400:693 Lecture 1 / 43 Outline 1 Review: Entropy of discrete RVs 2 Differential
More informationSHARED INFORMATION. Prakash Narayan with. Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe
SHARED INFORMATION Prakash Narayan with Imre Csiszár, Sirin Nitinawarat, Himanshu Tyagi, Shun Watanabe 2/41 Outline Two-terminal model: Mutual information Operational meaning in: Channel coding: channel
More informationSecret Key Agreement: General Capacity and Second-Order Asymptotics
Secret Key Agreement: General Capacity and Second-Order Asymptotics Masahito Hayashi Himanshu Tyagi Shun Watanabe Abstract We revisit the problem of secret key agreement using interactive public communication
More information18.2 Continuous Alphabet (discrete-time, memoryless) Channel
0-704: Information Processing and Learning Spring 0 Lecture 8: Gaussian channel, Parallel channels and Rate-distortion theory Lecturer: Aarti Singh Scribe: Danai Koutra Disclaimer: These notes have not
More informationA One-to-One Code and Its Anti-Redundancy
A One-to-One Code and Its Anti-Redundancy W. Szpankowski Department of Computer Science, Purdue University July 4, 2005 This research is supported by NSF, NSA and NIH. Outline of the Talk. Prefix Codes
More informationBlock 2: Introduction to Information Theory
Block 2: Introduction to Information Theory Francisco J. Escribano April 26, 2015 Francisco J. Escribano Block 2: Introduction to Information Theory April 26, 2015 1 / 51 Table of contents 1 Motivation
More informationLecture 2. Capacity of the Gaussian channel
Spring, 207 5237S, Wireless Communications II 2. Lecture 2 Capacity of the Gaussian channel Review on basic concepts in inf. theory ( Cover&Thomas: Elements of Inf. Theory, Tse&Viswanath: Appendix B) AWGN
More informationSource-Channel Coding Theorems for the Multiple-Access Relay Channel
Source-Channel Coding Theorems for the Multiple-Access Relay Channel Yonathan Murin, Ron Dabora, and Deniz Gündüz Abstract We study reliable transmission of arbitrarily correlated sources over multiple-access
More informationNotes 3: Stochastic channels and noisy coding theorem bound. 1 Model of information communication and noisy channel
Introduction to Coding Theory CMU: Spring 2010 Notes 3: Stochastic channels and noisy coding theorem bound January 2010 Lecturer: Venkatesan Guruswami Scribe: Venkatesan Guruswami We now turn to the basic
More informationOn Scalable Coding in the Presence of Decoder Side Information
On Scalable Coding in the Presence of Decoder Side Information Emrah Akyol, Urbashi Mitra Dep. of Electrical Eng. USC, CA, US Email: {eakyol, ubli}@usc.edu Ertem Tuncel Dep. of Electrical Eng. UC Riverside,
More informationHomework Set #2 Data Compression, Huffman code and AEP
Homework Set #2 Data Compression, Huffman code and AEP 1. Huffman coding. Consider the random variable ( x1 x X = 2 x 3 x 4 x 5 x 6 x 7 0.50 0.26 0.11 0.04 0.04 0.03 0.02 (a Find a binary Huffman code
More informationCoding on Countably Infinite Alphabets
Coding on Countably Infinite Alphabets Non-parametric Information Theory Licence de droits d usage Outline Lossless Coding on infinite alphabets Source Coding Universal Coding Infinite Alphabets Enveloppe
More informationLecture 10: Broadcast Channel and Superposition Coding
Lecture 10: Broadcast Channel and Superposition Coding Scribed by: Zhe Yao 1 Broadcast channel M 0M 1M P{y 1 y x} M M 01 1 M M 0 The capacity of the broadcast channel depends only on the marginal conditional
More informationLecture 15: Conditional and Joint Typicaility
EE376A Information Theory Lecture 1-02/26/2015 Lecture 15: Conditional and Joint Typicaility Lecturer: Kartik Venkat Scribe: Max Zimet, Brian Wai, Sepehr Nezami 1 Notation We always write a sequence of
More informationA Graph-based Framework for Transmission of Correlated Sources over Multiple Access Channels
A Graph-based Framework for Transmission of Correlated Sources over Multiple Access Channels S. Sandeep Pradhan a, Suhan Choi a and Kannan Ramchandran b, a {pradhanv,suhanc}@eecs.umich.edu, EECS Dept.,
More informationOn the Capacity of the Binary-Symmetric Parallel-Relay Network
On the Capacity of the Binary-Symmetric Parallel-Relay Network Lawrence Ong, Sarah J. Johnson, and Christopher M. Kellett arxiv:207.3574v [cs.it] 6 Jul 202 Abstract We investigate the binary-symmetric
More informationInformation Theory and Hypothesis Testing
Summer School on Game Theory and Telecommunications Campione, 7-12 September, 2014 Information Theory and Hypothesis Testing Mauro Barni University of Siena September 8 Review of some basic results linking
More informationA Novel Asynchronous Communication Paradigm: Detection, Isolation, and Coding
A Novel Asynchronous Communication Paradigm: Detection, Isolation, and Coding The MIT Faculty has made this article openly available. Please share how this access benefits you. Your story matters. Citation
More informationLecture 6: Gaussian Channels. Copyright G. Caire (Sample Lectures) 157
Lecture 6: Gaussian Channels Copyright G. Caire (Sample Lectures) 157 Differential entropy (1) Definition 18. The (joint) differential entropy of a continuous random vector X n p X n(x) over R is: Z h(x
More informationInformation Theory. Lecture 10. Network Information Theory (CT15); a focus on channel capacity results
Information Theory Lecture 10 Network Information Theory (CT15); a focus on channel capacity results The (two-user) multiple access channel (15.3) The (two-user) broadcast channel (15.6) The relay channel
More informationUnequal Error Protection Querying Policies for the Noisy 20 Questions Problem
Unequal Error Protection Querying Policies for the Noisy 20 Questions Problem Hye Won Chung, Brian M. Sadler, Lizhong Zheng and Alfred O. Hero arxiv:606.09233v2 [cs.it] 28 Sep 207 Abstract In this paper,
More informationSource and Channel Coding for Correlated Sources Over Multiuser Channels
Source and Channel Coding for Correlated Sources Over Multiuser Channels Deniz Gündüz, Elza Erkip, Andrea Goldsmith, H. Vincent Poor Abstract Source and channel coding over multiuser channels in which
More information