Error Correctng Codes: Combnatorcs, Algorthms and Alcatons (Fall 2007) Lecture 9: Converse of Shannon s Caacty Theorem Setember 17, 2007 Lecturer: Atr Rudra Scrbe: Thanh-Nhan Nguyen & Atr Rudra In the last lecture, we stated Shannon s caacty theorem for the BSC, whch we restate here: Theorem 0.1. Let 0 < 1/2 be a real number. For every 0 < ε 1/2, the followng statements are true for large enough nteger n: () There exsts a real δ > 0, an encodng functon E : {0, 1} k {0, 1} n, and a decodng functon D : {0, 1} n {0, 1} k, where k (1 H( + ε))n such that the followng holds for every m {0, 1} k : P r [D(E(m) + e) m] 2 δn. nose e of BSC () If k (1 H() + ε)n then for every encodng and decodng functons E : {0, 1} k {0, 1} n and D : {0, 1} n {0, 1} k the followng s true for some m {0, 1} k : P r [D(E(m) + e) m] 1/2. nose e of BSC In today s lecture, we wll rove art () of Theorem 0.1. 1 Prelmnares Before we begn wth the roof we wll need a few results, whch we dscuss frst. 1.1 Chernoff Bound Chernoff bound states a bound on the tal of a certan dstrbuton that wll be useful for us. Here we state the verson of the Chernoff bound that we wll need. Prooston 1.1. For 1,, n, let X be a bnary random varable that takes a value of 1 wth robablty and a value of 0 wth robablty 1. Then the followng bounds are true: () P r [ n 1 X (1 + ε)n] e ε2 n/3 () P r [ n 1 X (1 ε)n] e ε2 n/3 Note that the exectaton of the sum n 1 X s n. The bound above states that the robablty mass s tghtly concentrated around the mean. 1
1.2 Volume of Hammng Balls We wll also need good uer and lower bounds on the volume of a Hammng ball. Recall that V ol q (0, n) B q (0, ρn) n ( n ) (q 1). We wll rove the followng result: Prooston 1.2. Let q 2 be an nteger and 0 1 1 q () V ol q (0, n) q Hq()n () V ol q (0, n) q Hq()n o(n) where recall that H q (x) x log q (q 1) x log q x (1 x) log q (1 x). be a real. Then for large enough n: Proof. We start wth the roof of (). Consder the followng sequence of relatons: 1 ( + (1 )) n n (1 ) n (1) n (1 ) n (2) n ( ) ( ) n (q 1) (1 ) n q 1 n ( ( ) n )(q 1) (1 ) n (q 1)(1 ) n ( ( ) n n )(q 1) (1 ) n (3) (q 1)(1 ) n ( ) n (q 1) (1 ) (1 )n. (4) q 1 In the above, (1) follows from the bnomal exanson. (2) follows by drong some terms from the summaton and (3) follows from that facts that 1 (as q 2 and 1/2) and (q 1)(1 ) n 1 (for large enough ( n). Rest of the stes follow from rearrangng the terms. n As q Hq()n q 1) (1 ) (1 )n, (4) mles that 1 V ol q (0, n)q Hq()n, whch roves (). We now turn to the roof of art (). For ths art, we wll need Strlng s aroxmaton for n! ( n ) n 2πn e λ 1 (n) < n! < ( n ) n 2πn e λ 2 (n), e e 2
where 1 λ 1 (n) 12n + 1 and λ 2(n) 1 12n. By the Strlng s aroxmaton, we have the followng nequalty: n! n (n)!((1 )n)! (n/e) n > (n/e) n ((1 )n/e) 1 e λ 1(n) λ 2 (n) λ 2 ((1 )n) (1 )n 2π(1 )n 1 l(n), (5) n (1 ) (1 )n where l(n) eλ 1 (n) λ 2 (n) λ 2 ((1 )n) 2π(1 )n. Now consder the followng sequence of relatons that comlete the roof: V ol q (0, n) (q 1) n (6) n (q 1) n > l(n) (7) n (1 ) (1 )n q Hq()n o(n). (8) In the above (6) follows by only lookng at one term. (7) follows from (5) whle (8) follows from the defnton of H q ( ) and the fact that for large enough n, l(n) s q o(n). 2 Converse of Shannon s Caacty Theorem for BSC We wll now rove art () of Theorem 0.1: the roof of the other art wll be done n the next lecture. Frst, we note that there s nothng to rove f 0, so for the rest of the roof we wll assume that > 0. For the sake of contradcton, assume that the followng holds for every m {0, 1} k : P r [D(E(m) + e) m] 1/2. nose e of BSC Fx an arbtrary message m {0, 1} k. Defne D m to be the set of receved words that are decoded to m by D, that s, D m {y D(y) m}. Note that by our assumton, the followng s true (where from now on we omt the exlct deendence of the robablty on the BSC nose for clarty): P r [E(m) + e D m ] 1/2. (9) 3
Further, by the Chernoff bound, P r[e(m) + e S m ] 2 Ω(γ2 n), (10) where S m s the shell of radus [(1 γ)n, (1 + γ)n] around E(m), that s, S m B 2 (E(m), (1 + γ)n) \ B 2 (E(m), (1 γ)n). (We wll set γ > 0 n terms of ε and at the end of the roof.) (9) and (10) along wth the unon bound mly the followng: P r [E(m) + e D m S m ] 1 2 2 Ω(γ2 n) 1 4, (11) where the last nequalty holds for large enough n. Next we uer bound the robablty above to obtan a lower bound on D m S m. It s easy to see that where P r [E(m) + e D m S m ] D m S m max, max max P r[e(m) + e y] max d (1 ) n d. y S m d [(1 γ)n,(1+γ)n] It s easy to check that d (1 ) n d s decreasng n d for 1/2. Thus, we have ( ) γn ( ) γn 1 1 max (1 γ)n (1 ) n (1 γ)n n (1 ) (1 )n 2 nh(). Thus, we have shown that ( ) γn 1 P r [E(m) + e D m S m ] D m S m 2 nh(), whch by (11) mles that D m S 1 ( ) γn 1 4 2 nh(). (12) Next, we consder the followng sequence of relatons: 2 n D m (13) m {0,1} k D m S m {0,1} k 1 ( ) γn 1 (14) 4 m {0,1} k 2 H()n 2 k 2 H()n γ log(1/ 1)n 2 > 2 k+h()n εn. (15) 4
In the above (13) follows from the fact that for m 1 m 2, D m1 and D m2 are dsjont. (14) ε follows from (12). (15) follows for large enough n and f we ck γ 2 log( 1 1). (Note that as 0 < < 1/2, γ Θ(ε).) (15) mles that k < (1 H() + ε)n, whch s a contradcton. The roof of art () of Theorem 0.1 s comlete. Remark 2.1. It can be verfed that the roof above can also work f the decodng error robablty s bounded by 2 βn (nstead of the 1/2 n art () of Theorem 0.1) for small enough β β(ε) > 0. 5