Refined Bounds on the Empirical Distribution of Good Channel Codes via Concentration Inequalities

Transcription

1 Refined Bounds on the Empirical Distribution of Good Channel Codes via Concentration Inequalities Maxim Raginsky and Igal Sason ISIT 2013, Istanbul, Turkey

2 Capacity-Achieving Channel Codes The set-up DMC T : X Y with capacity C = C(T ) = max P X I(X; Y ) (n, M)-code: C = (f, g) with encoder f : {1,..., M} X n and decoder g : Y n {1,..., M} Capacity-achieving codes: A sequence {C n } n=1, where each C n is an (n, M n )-code, is capacity-achieving if lim n 1 n log M n = C.

3 Capacity-Achieving Channel Codes Capacity-achieving input and output distributions: P X arg max P X I(X; Y ) (may not be unique) P X T P Y (always unique) Theorem (Shamai Verdú, 1997) Let {C n } be any capacity-achieving code sequence with vanishing error probability. Then 1 ( ) lim n n D P (Cn) P Y n Y n = 0, where P (Cn) Y is the output distribution induced by the code n C n when the messages in {1,..., M n } are equiprobable.

4 Capacity-Achieving Channel Codes lim n 1 n D(P Y n P Y n) = 0 Main message: channel output sequences induced by good code resemble i.i.d. sequences drawn from the CAOD P Y Useful implications: estimate performance characteristics of good channel codes by their expectations w.r.t. P Y n = (P Y )n often much easier to compute explicitly bound estimation accuracy using large-deviation theory (e.g., Sanov s theorem) Question: what about good codes with nonvanishing error probability?

5 Codes with Nonvanishing Error Probability Y. Polyanskiy and S. Verdú, Empirical distribution of good channel codes with non-vanishing error probability (2012) 1. Let C = (f, g) be any (n, M, ε)-code for T : max P ( g(y n ) j f(x n ) = j ) ε. 1 j M Then D ( P (C) ) P Y n Y n nc log M + o(n). 2. If {C n } n=1 is a capacity-achieving sequence, where each C n is an (n, M n, ε)-code for some fixed ε > 0, then 1 ( lim n n D P (Cn) Y n ) P Y n = 0. In some cases, the o(n) term can be improved to O( n).

6 Codes with Nonvanishing Error Probability D(P Y n PY n) nc log M + o(n) The same message: channel output sequences induced by good codes resemble i.i.d. sequences drawn from P Y Main technical tool: concentration of measure Our contribution: sharpening of the Polyanskiy Verdú bounds by identifying explicit expressions for the o(n) term

7 Preliminaries on Concentration of Measure Let Z 1,..., Z n Z be independent random variables. We seek tight bounds on the deviation probabilities P (f(z n ) r) for r > 0 where f : Z n R is some function with E[f(Z n )] = 0. Subgaussian tails: log E[e tf(zn) ] κt 2 /2, t > 0 = P (f(z n ) r) exp ) ( r2, r > 0 2κ

8 Preliminaries on Concentration of Measure Suppose that Z n is a metric space with metric d(, ). L 1 Wasserstein distance: for any µ, ν P(Z n ), W 1 (µ, ν) inf E[d(Z n, Z n )] Z n µ, Z n ν L 1 transportation cost inequalities (Marton): µ P(Z n ) satisfies a T 1 (c) inequality if W 1 (µ, ν) 2cD(ν µ), ν µ T 1 (c) implies concentration!

9 Preliminaries on Concentration of Measure L 1 transportation cost inequalities (Marton): µ P(Z n ) satisfies a T 1 (c) inequality if W 1 (µ, ν) 2cD(ν µ), ν µ Theorem (Bobkov Götze, 1999) A probability measure µ P(Z n ) satisfies T 1 (c) if and only if log E µ [e tf(zn) ] ct 2 /2 for all f with E µ [f(z n )] = 0 and f(z n ) f( z n ) f Lip sup z n z n d(z n, z n ) 1

10 Preliminaries on Concentration of Measure Endow Z n with the weighted Hamming metric n d(z n, z n ) = c i 1 {zi z i }, for some fixed c 1,..., c n > 0. i=1 Marton s coupling argument: Any product measure µ = µ 1... µ n P(Z n ) satisfies T 1 (c) (relative to d) with n c = 1 4 i=1 c 2 i By Bobkov Götze, this is equivalent to the subgaussian property [ ] n log E µ e tf(zn ) t2 c 2 i 8 for any f : Z n R with E µ f = 0 and f Lip 1 (another way to derive McDiarmid s inequality) i=1

11 Relative Entropy at the Output of a Code Consider a DMC T : X Y with T ( ) > 0, and let c(t ) = 2 max max T (y x) x X y,y Y ln T (y x) Theorem. Any (n, M, ε)-code C for T, where ε (0, 1/2), satisfies ( ) D P (C) P Y n Y n nc log M + log 1 n ε + c(t ) 2 log 1 1 2ε Remark: Polyanskiy and Verdú show that ( ) D P (C) P Y n Y n nc log M + a n for some constant a = a(ε)

12 Proof Idea: I Fix x n X n and study concentration of the function h x n(y n ) = log dp Y n X n =x n dp (C) Y n (y n ) around its expectation w.r.t. P Y n X n =x n: ( E[h x n(y n ) X n = x n P (C) ] = D P Y n X n =x n Y n ) Step 1: Because T ( ) > 0, the function h x n(y n ) is 1-Lipschitz w.r.t. scaled Hamming metric d(y n, ȳ n ) = c(t ) n i=1 1 {yi ȳ i }

13 Proof Idea: II Step 1: Because T ( ) > 0, the function h x n(y n ) is 1-Lipschitz w.r.t. scaled Hamming metric d(y n, ȳ n ) = c(t ) n i=1 1 {yi ȳ i } Step 2: Any product probability measure µ on (Y n, d) satisfies log E µ [ e tf(y n ) ] nc(t )2 t 2 for any f with E µ f = 0 and F Lip 1. Proof: tensorization of the Csiszár Kullback Pinsker inequality, followed by appeal to Bobkov Götze. 8

14 Proof Idea: III h x n(y n ) = log dp Y n X n =x n (y n ) dp (C) Y n ( E[h x n(y n ) X n = x n P (C) ] = D P Y n X n =x n Y n ) Step 3: For any x n, µ = P Y n X n =x n is a product measure, so ( ( ) ) ) P h x n(y n P (C) ) D P Y n X n =x n Y + r exp ( 2r2 n nc(t ) 2 Use this with r = c(t ) n log ε : ( ( ) ) P h x n(y n P (C) n ) D P Y n X n =x n Y + c(t ) n 2 log 1 1 2ε 1 2ε Remark: Polyanskiy Verdú show Var[h x n(y n ) X n = x n ] = O(n).

15 Proof Idea: IV Recall: ( ( ) ) P h x n(y n P (C) n ) D P Y n X n =x n Y + c(t ) n 2 log 1 1 2ε 1 2ε Step 4: Same as Polyanskiy Verdú, appeal to Augustin s strong converse to get log M log 1 ( ) P ε + D (C) P (C) n P Y n X n Y n X + c(t ) n 2 log 1 1 2ε ( ) D P (C) P Y n Y n ( P = D P Y n X n Y n ) ( P (C) P (C) X D P n Y n X n Y n P (C) nc log M + log 1 ε + c(t ) n 2 log 1 1 2ε X n )

16 Relative Entropy at the Output of a Code Theorem. Let T : X Y be a DMC with C > 0. Then, for any 0 < ε < 1, any (n, M, ε)-code C for T satisfies D ( P (C) P Y n Y n) nc log M + 2n (log n) 3/2 ( log n + log ( 2 X Y 2). 1 log n log ( ) ) ( ε ) log Y log n Remark: Polyanskiy and Verdú show that D ( P (C) Y n P Y n) nc log M + b n log 3/2 n for some constant b > 0.

17 Concentration of Lipschitz Functions Theorem. Let T : X Y be a DMC with c(t ) <. Let d : Y n Y n R + be a metric, and suppose that P Y n X n =x n, x n X n, as well as P Y n, satisfy T 1(c) for some c > 0. Then, for any ε (0, 1/2), any (n, M, ε)-code C for T, and any function f : Y n R we have ( ) P (C) Y f(y n ) E[f(Y n )] r n ) (nc 4 ε exp ln M + a n 8c f 2 Lip where Y n P Y n, and a c(t ) 1 2 ln 1 1 2ε. r 2, r 0

18 Proof Idea Step 1: For each x n X n, let φ(x n ) E[f(Y n ) X n = x n ]. Then, by Bobkov Götze, ( ) ( P f(y n ) φ(x n ) r X n = x n) 2 exp r2 2c f 2 Lip Step 2: By restricting to a subcode C with codewords x n X n satisfying φ(x n ) E[f(Y n )] + r, we can show that ( r f Lip 2c nc log M + a n + log 1 ), ε ) with M = MP (C) X (φ(x n ) E[f(Y n )] + r. Solve to get n ( ) P (C) X φ(x n ) E[f(Y n )] r 2e nc log M+a n+log 1 ε r 2 n 2c f 2 Lip Step 3: Apply union bound.

19 Empirical Averages at the Code Output Equip Y n with the Hamming metric n d(y n, ȳ n ) = Consider functions of the form f(y n ) = 1 n i=1 1 {yi ȳ i } n f i (y i ), where f i (y i ) f i (ȳ i ) L1 {yi ȳ i } for all i, y i, ȳ i. Then f Lip L/n. Since P Y n X n =x n for all xn and PY n are product measures on Y n, they all satisfy T 1 (n/4) (by tensorization) Therefore, for any (n, M, ε)-code and any such f we have P (C) ( Y f(y n ) E[f(Y n )] r ) n 4 ( ε exp nc log M + a ) n nr2 2L 2 i=1

20 Operational Significance A bound like C(T ) log M 1 n D( P (C) ) P a(t, ε) Y n Y n n quantifies trade-offs between minimal blocklength required for achieving a certain gap (in rate) to capacity with a fixed block error probability ε, and normalized divergence between output distribution induced by the code and the (unique) CAOD of the channel We have identified the precise dependence of a(t, ε) on the channel T and on the block error probability ε These results are similar to a lower bound on rate loss w.r.t. fully random block codes (whose average distance spectrum is binomially distributed) in terms of normalized divergence between the distance spectrum of a specific code and the binomial distribution (Shamai Sason, 2002).

21 Concentration of measure = powerful tool for studying nonasymptotic behavior of stochastic objects in information theory! For more information, see M. Raginsky and I. Sason, Concentration of Measure Inequalities in Information Theory, Communications and Coding, arxiv:

22 That s All, Folks!