Lecture 6: Source coding, Typicality, and Noisy channels and capacity

Transcription

1 15-859: Iformatio Theory ad Applicatios i TCS CMU: Sprig 2013 Lecture 6: Source codig, Typicality, ad Noisy chaels ad capacity Jauary 31, 2013 Lecturer: Mahdi Cheraghchi Scribe: Togbo Huag 1 Recap Uiversal Source Codig: Lempel-Ziv Algorithm ad its optimality. Please see otes of last lecture. 2 Source Codig by Typicality 2.1 AEP ad Typical Set We have leared AEP i lecture 4, ad here is the defiitio oce agai: Theorem 1 (AEP If x 1, x 2,..., x are i.i.d p(x, the log p(x 1,x 2,...,x H(x i probability. Now, we ca defie a typical set give the defiitio of AEP: Defiitio 2 (Typical Set The typical set A ( (x 1, x 2,..., x χ ( with the property with respect to p(x is the set of sequeces 2 H(X+ p(x 1, x 2,..., x 2 H(X (1 Theorem 3 (Properties of Typical Set As a cosequece of AEP, we ca show that the set A ( comes with followig properties: 1. If is large eough, the P r(x 1 A ( A ( 2 H(X+, where A deotes the umber of elemets i the typical set. 2.2 Data Compressio with AEP We ca divide all sequeces i χ ito two sets: those i the typical set ad those i its complemet. This is illustrated i 2.2: Figure 6.1. Ecode each x 1 A ( by its idex i A ( (with a leadig zero. Also, ecode x 1 / A ( trivially (with a leadig oe. Such ecodig with esure uiqueess sice the decoder kows the distributio ad ca tell the elemets i typical set with the leadig bit. 1

2 Figure 1: Typical sets ad source codig Now we ca calculate the expected legth. E(l(x 1 = Σ x 1 χ P (x 1 l(x 1 = Σ x 1 A (P (x 1 l(x 1 + Σ x cp 1 A ( (x 1 l(x 1 Σ x 1 A (P (x 1 l((h(x Σ x cp 1 A ( (x 1 l(log χ + 2 (H + + log χ + 2 (H(x + (2 where = + log χ + 2 ca be made arbitrarily small by appropriate choice of ad. This ca be i terms tured ito a uiversal codig scheme for all source χ with bouded H(x h. (Roughly speakig, this is Csiszar-Körer Uiversal Source Codig. Here, we cosider oly source x s.t. x p(x = it X, which is used to limit the umber of uique distributio for uio boud. Also, we eed a bouded iitial source, so the typical set also works for small etropy sets. 3 Joit Typicality Defiitio 4 (Joitly Typical Set The set A ( of joitly typical sequeces (x, y with respect to the distributio p(x, y is the set of -sequeces with empirical etropies close to the true etropies: where p(x, y = i=1 p(x i, y i. A ( = {(x, y X Y : log p(x 1, y 1 log p(x 1 log p(y 1 H(x, y, H(x, H(y } (3 2

3 Theorem 5 (Joit AEP From the defiitio of joitly typical set, we have followig properties: 1. P r(x 1, y 1 A( 1 as 2. For large, (1 2 (H(x,y A ( 2 (H(x,y+ 3. If ( x 1, y 1 p(x 1 p(y 1 idepedet ad has the same margial distributio, but ot joit distributio, the (1 2 (I(x;y+3 P r( x 1, y 1 A( 2 (H(x,y+. Proof: 1. Accordig to the weak law of large umbers: > 0, 0, s.t. > 0, P r( log p(x 1 H(x 3, P r( log p(y 1 H(y 3, P r( log p(x 1,y 1 H(x, y 3. Property is easily proved with uio boud. 2. The upper boud follow from the oe variable AEP. To prove the lower boud, we take large eough so that: 3. For property 3, 1 P r(a ( = Σ (x p(x 1,y1 A( 1, y1 P r( x 1, y 1 A( = Σ (x p(x 1,y1 A( 1, y1 Ad the proof is similar for lower boud. A ( 2 (H(x,y (4 = 2 (H(x,y+ 2 (H(x 2 (H(y (by property 2 = 2 (I(x,y 3 (5 4 Chael Codig 4.1 Defiitio We ow prove what is perhaps the basic theorem of iformatio theory, the achievablility of chael capacity, first stated ad essetially proved by Shao i his origial 1948 paper. The result is rather couterituitive; if the chael itroduces errors, how ca oe correct them all? Ay correctio process is also subject to error, ad ifiitum. Shao used a umber of ew ideas to prove that iformatio ca be set reliably over a chael at all rates up to the chael capacity. Let s take a look at how chael work first, please see 4.1: Figure 6.2. A discrete chael ca be viewed as trasformig iput x to output y through probability p(y x. Meawhile, chael ca also be viewed as a probability trasformatio matrix as i 4.1: Figure

4 Figure 2: Illustratio of chael codig Figure 3: A probability trasformatio matrix 4.2 Examples Noiseless Chael Iput Output Probability Trasformatio Matrix: ( Noiseless Chael, Upredictable Output Iput Output Probability Trasformatio Matrix: ( 1/2 1/ /2 1/ Biary Symmetric Chael (BSC(p Iput Output Probability p 0 1 p 1 0 p p Trasformatio Matrix: ( 1 p p p 1 p Biary Erasure Chael (BEC(p 4

5 Iput Output Probability p 0 e p 1 e p p Trasformatio Matrix: ( 1 p p p p 4.3 Chael Capacity 4.4 Defiitio Defiitio 6 We defie chael capacity of a discreet memoryless chael as: C = max I(x; y (6 p(x Which is takig the best mutual iformatio amog all distributio i the trasformatio matrix. 4.5 Examples Noiseless Chael I(x; y = H(x = 1 for uiform x (lower boud Noiseless Chael, Upredictable Output I(x; y = H(x H(x y = 1, sice y determies x BSC(p I(x; y = H(y H(y x = H(y Σp(xH(y X = x = H(y h(p 1 h(p (7 For uiform x, uiform y, C = 1 h(p BEC(p C = max p(x H(y h(p H(y = H(y, erasure = H( erasure + H(y erasure = h(p + (1 ph(p 1 where p 1 = P r(x 1, which meas o ifo gai if erasure happes C = max p 1 (1 ph(p 1 = (1 p(p 1 = 1/2 (8 5