Lecture 11: Channel Coding Theorem: Converse Part

Size: px

Start display at page:

Download "Lecture 11: Channel Coding Theorem: Converse Part"

Maurice Shaw
5 years ago
Views:

1 EE376A/STATS376A Iformatio Theory Lecture - 02/3/208 Lecture : Chael Codig Theorem: Coverse Part Lecturer: Tsachy Weissma Scribe: Erdem Bıyık I this lecture, we will cotiue our discussio o chael codig theory. I the previous lecture, we proved the direct part of the theorem, which suggests if R < C (I), the R is achievable. Now, we are goig to prove the coverse statemet: If R > C (I), the R is ot achievable. We will also state some importat otes about the direct ad coverse parts. Recap: Commuicatio Problem Recall the commuicatio problem: J Uif{, 2,..., M} ecoder X Memoryless Chael P (Y X) Y decoder Ĵ Rate = R = bits chael use Probability of error = P e = P (Ĵ J) The mai result is C = C (I) = max PX I(X; Y ). Last week, we showed R is achievable if R < C (I). I this lecture, we are goig to prove that if R > C (I), the R is ot achievable. 2 Fao s Iequality Theorem (Fao s Iequality). Let X be a discrete radom variable ad ˆX = ˆX(Y ) be a guess of X based o Y. Let P e = P (X ˆX). The, H(X Y ) h 2 (P e ) + P e log( X ) where h 2 is the biary etropy fuctio. Proof. Let V = {X ˆX}, i.e. V is if X ˆX ad 0 otherwise. H(X Y ) H(X, V Y ) () = H(V Y ) + H(X V, Y ) (2) H(V ) + H(X V, Y ) (3) = H(V ) + v,y H(X V =v, Y =y)p (V =v, Y =y) (4) = H(V ) + y H(X V =0, Y =y)p (V =0, Y =y) + y H(X V =, Y =y)p (V =, Y =y) (5) = H(V ) + y H(X V =, Y =y)p (V =, Y =y) (6) H(V ) + log( X ) y P (V =, Y =y) (7) = H(V ) + log( X )P (V =) (8) = h 2 (P e ) + P e log( X ) (9) Readig: Chapter 7.9 ad 7.2 of Cover, Thomas M., ad Joy A. Thomas. Elemets of iformatio theory. Wiley, 2006.

2 where () is from data processig iequality, (2) is due to chai rule, (3) is because coditioig ca oly reduce (or ot chage) etropy. (4) directly follows from the defiitio of coditioal etropy. (6) is because whe V = 0, X = ˆX ad X is a fuctio of Y, so H(X V = 0, Y = y) = 0. Note that H(X V =, Y = y) is maximized whe P (X V =, Y = y) is uiformly distributed, which yields to log( X ). Hece, (7) follows. The ext step is just law of total probability, ad completes the proof. Note a weaker versio of Fao s iequality is H(X Y ) + P e log X (0) which will be useful later i provig the coverse theorem. This is also stated as P e H(X Y ) log X () Fao s iequality basically says that if H(X Y ) is large, i.e., if give Y, X has a lot of ucertaity, the ay estimator of X based o Y must have a large probability of error. 3 Proof of Coverse Part For ay scheme, H(J Y ) = H(J) H(J Y ) (2) = I(J; Y ) (3) = H(Y ) H(Y J) (4) = H(Y i Y i ) H(Y i Y i, J) (5) = = H(Y i ) H(Y i Y i, J) (6) H(Y i ) H(Y i Y i, X i, J) (7) H(Y i ) H(Y i X i ) (8) I(X i ; Y i ) (9) C (I) (20) where (2) is because J is uiformly distributed, (3), (4) ad (9) are directly from the defiitio of mutual iformatio, (5) is from the properties of joit/coditioal etropy, (6) ad (7) are due to the fact that coditioig ca oly decrease (or ot chage) etropy. Sice the chael is memoryless (i.e. Y i X i (Y i, J)), (8) follows. Next, (20) is because the capacity is the maximum of the mutual iformatio betwee iput ad output. 2

3 Now, for schemes with R, P e H(J Y ) C(I) = C(I) C(I) R R C(I) R where (2) is due to weaker versio of Fao s iequality, (22) is from the result obtaied with (20), ad (24) is because R. The result shows that whe R > C (I), there exists a positive lower boud o the probability of error, so R is ot achievable. 4 Geometric Iterpretatio of Coverse Part To iterpret the coverse theorem geometrically, cosider Fig., where a commuicatio chael is visualized with the correspodig typical sets. I this scheme, 2 R codewords are selected from X. After trasmissio through the chael, each codeword has a typical set of size 2 H(Y X) (we ll study the otio of coditioal typicality i more detail later). Also, ote that the typical set of Y has size 2 H(Y ), aturally idepedet from X. Now, ote that the typical chael outputs give X (i) s should ot itersect i order to have zero (2) (22) (23) (24) (25) Figure : Geometric iterpretatio of commuicatio problem probability of error. Hece, there must be at least 2 R 2 H(Y X) elemets i the typical chael outputs. By this volume argumet, Sice I(X; Y ) C, we must have R C. 2 H(Y ) 2 R 2 H(Y X) (26) 2 H(Y ) H(Y X) 2 R (27) 2 I(X;Y ) 2 R (28) I(X; Y ) R (29) 3

4 5 Some Notes o the Direct ad Coverse Parts 5. Commuicatio with Feedback Now assume X i is a fuctio of both J ad Y i (previously, it was a fuctio of oly J), so the ecoder kows what decoder receives. This is obviously a stroger ecoder, as it has more iformatio. However, it ca be verified that the proof of the coverse theorem is valid for memoryless chaels with feedback, as well. This ca be directly see from that the proof uses the properties of the chael oly at Eq. (8), which also holds whe feedback is allowed (because the Markov property still holds: Y i X i Y i, J). Moreover, achievability result is obvious as the feedback ca be igored. Therefore, the maximum achievable rate remais the same with feedback. O the other had, this settig icreases the reliability of the system, i.e. the probability of error vaishes faster; ad the schemes become simpler. Example. Recall the biary erasure chael (BEC) show i Fig. 2. Also recall that the capacity of BEC is C = p bits/chael use where p is the erasure probability. Cosider a biary erasure chael Figure 2: Biary erasure chael (image from Wikipedia) with feedback. A very simple scheme that achieves capacity would be to repeat each iformatio bit util it is correctly received by the decoder. With this scheme the probability that a bit is correctly set through the chael at oe attempt is p, at two attempts is p( p), ad so o. Hece, it follows a geometric distributio, whose mea is p. Therefore, we have p chael uses per iformatio bit Equivaletly, R = C. This approach ca be exteded to all memoryless chaels Practical Schemes I the proof of the direct part, we showed mere existece of schemes C of size at least 2 R ad arbitrarily small probability of error. We did ot explicitly give particular code costructios that achieve this. For practical schemes that eable commuicatio with rates arbitrarily close to the maximum mutual iformatio with reasoable complexity, we refer to: 2 Readig: Horstei, Michael. Sequetial trasmissio usig oiseless feedback. IEEE Trasactios o Iformatio Theory 9.3 (963): Additioal Refereces: Schalkwijk, J., ad Thomas Kailath. A codig scheme for additive oise chaels with feedback I: No badwidth costrait. IEEE Trasactios o Iformatio Theory 2.2 (966): Shayevitz, Ofer, ad Meir Feder. Optimal feedback commuicatio via posterior matchig. IEEE Trasactios o Iformatio Theory 57.3 (20): Li, Cheuk Tig, ad Abbas El Gamal. A efficiet feedback codig scheme with low error probability for discrete memoryless chaels. IEEE Trasactios o Iformatio Theory 6.6 (205):

5 . Low-desity parity-check (LDPC) codes 2. Polar codes The course EE388 - Moder Codig Theory is ecouraged for much more detail. I this course, we are goig to dedicate some time to either of these codes i lectures or homeworks. 5.3 Geeralizatio to Ifiite Alphabets Proof of the direct part assumed fiite alphabets; however it carries over to geeral case by approximatio / quatizatio. 5.4 P e vs P max We previously talked about P max which may be more reasoable for some practical systems, ad puts a more striget coditio tha P e : P e = P (Ĵ J) = m m P (Ĵ j J = j) j= P max = max P (Ĵ j J = j) j Claim. Give a codig scheme C such that P e 0 as, we ca fid aother scheme C, that achieves P max 0. Proof. By Markov s iequality, we have { j m : P ( Ĵ j J = j) 2P e } m 2 Hece, give C with C = m, R C ad P e, there exists a C such that C m/2 ad P max 2P e. The rate of this ew code is the R c log(m/2) = R ɛ for arbitrarily small ɛ whe is large. Note P max of C goes to zero as P e 0. Thus, capacity is the same regardless of whether reliable commuicatio is with respect to P e or P max. 5

Lecture 15: Strong, Conditional, & Joint Typicality

Lecture 15: Strong, Conditional, & Joint Typicality EE376A/STATS376A Iformatio Theory Lecture 15-02/27/2018 Lecture 15: Strog, Coditioal, & Joit Typicality Lecturer: Tsachy Weissma Scribe: Nimit Sohoi, William McCloskey, Halwest Mohammad I this lecture,