Lecture 7: Channel coding theorem for discrete-time continuous memoryless channel

Transcription

1 Lecture 7: Chael codig theorem for discrete-time cotiuous memoryless chael Lectured by Dr. Saif K. Mohammed Scribed by Mirsad Čirkić Iformatio Theory for Wireless Commuicatio ITWC Sprig 202 Let us first defie, for the radom sequeces X = [X,...,X ] ad Y = [Y...,Y ] ad their correspodig sequece realizatios x = [x,...,x ] ad y = [y...,y ] where x k,y k R, the followig probability desity fuctios pdfs: f X x = k= f Xx k as the iput pdf, f Y X y x = k= f Y Xy k x k as the memoryless chael trasitio pdf, ad f Y y as the output pdf. Each elemet X k or just X without the idex k are idepedet ad idetically distributed i.i.d. Further, due to the i.i.d. property of X ad the specified memoryless property imposed o the chael trasitio pdf, we have f Y y = k= f Yy k. Defie the so called ad the mutual iformatio quatity Let IX;Y fx,y x,y f X,Y x,ylog dxdy f X xf Y y x,y defie the mutual iformatio betwee the radom variables X ad Y. Further, let g : R R be the capacity cost fuctio mappig. The, we defie,n,λ code with a capacity cost S R as a system {u,a,...,u N,A N u i = [u i,... u i, ] R, A i R, A i A j = i j k= gu i,k S, i λ N N i= λ i, λ M max i=,...,n λ i, λ i P Y / A i X = u i, ad λ M λ. Theorem. Radom Codig Theorem We defie the capacity with cost costrait S as CS max IX;Y f X s.t. f X xgx S. x 2 This documet is a property of Commuicatio Systems Divisio, Departmet of Electrical Egieerig, Liköpig Uiversity, Swede. Copyright must be obtaied by writig to saif@isy.liu.se, erik.larsso@isy.liu.se prior to usage.

2 2 The theorem states that, For ay rate R which satisfies 0 R < CS, there exists a sequece of,2 R,λ codes for =,2,... such that λ 0 as. I what follows, a proof is preseted ad for that reaso, we eed to defie A ǫ {x,y : logf Xx hx < ǫ, logf Yy hy < ǫ, logf Y,Xy,x hy,x < ǫ, where hx x f Xxlogf X xdx. Further, we eed to itroduce the followig two lemmas. Lemma. f Xx X MC Y f Y Xy x P X,Y A ǫ > ǫ for sufficietly large VolA ǫ x,y A ǫ dxdy < 2hY,X+ǫ VolA ǫ > ǫ2 hy,x ǫ for sufficietly large We omit the explicit proofs sice they become trivial from the discrete case proofs i previous lectures. Lemma 2. Kow as the Packig Lemma For idepedet X ad Y, i.e., f Xx X f Xx X MC Y f Y Xy x the followig iequalities hold for sufficietly large, ǫ2 IX;Y+3ǫ < P X,Y A ǫ < 2 IX;Y 3ǫ. P X,Y A ǫ = x,y A ǫ f X xf Y y d xdy f X,Y x,y From the defiitio of A ǫ, we have 2 hx+ǫ < f X x < 2 hx ǫ, 2 hy+ǫ < f Y y < 2 hy ǫ. This gives us the upper boud P X,Y A ǫ < 2 hx+hy 2ǫ x,y A ǫ < 2 hx+hy 2ǫ 2 hy,x+ǫ lemma = 2 IX;Y 3ǫ, d xdy

3 ad aalogously the lower boud P X,Y A ǫ > 2 IX:Y+3ǫ, for sufficietly large. The proof of Theorem. relies o Shao s radom codig argumet, which utilizes the fact that if the error probability averaged over all possible codes goes to zero as, the there must exist at least oe code that provides a error probability smaller tha the average. Radom Code Geeratio: Choose a arbitrary ǫ > 0 Choose a pdf f X x s.t. x f Xxgxdx = S ǫ < S Geerate u i,k for all i =,...,N ad k =,..., idepedetly usig f X x. 2 Ecodig: Let us label the possible codewords u,...,u N as messages w =,...,N. Whe the trasmitter wats to commuicate message k, it will trasmit u k. 3 Decodig: Let us use the followig typical set decoder for the code C that, for a give chael 3 output y, estimates the trasmitted message as or be i error if i, if y A i C ad y / A j C j i ŵ C y = ad gu i,k k= S 0, otherwise i.e. decodig error, where A i C {y R ui,y A ǫ. The average error probability λu,...,u N N N i= λ iu,...,u N ad the error probability λ i u,...,u N P ŵy i X = ui are both determiistic for a particular realizatio of the code book C = u,...,u N. Each realizatio u i is draw from the iput pdf f X x, ad this is doe idepedetly over i. If we defie C = X,...,X i,...,x N, where X i are i.i.d. with the pdf f X, we ca say that f C x,...,x N = N i= f Xx i ad that the code book C is draw from f C x,...,x N. Now, the expectatio of λ take over C, i.e., the whole esemble of codes is { N E C { λc = EC λ i C = N E C {λ i X,...,X N = E C {λ X,...,X N, N N i= i= where the last equality utilizes the fact that the expectatio value is ivariat to ay reorderig of X,...,X N due to the i.i.d. property of X i over i. Before we cotiue further, let us itroduce a compact otatio A 0 C for the set {y gu,k k= S, which may seem strage sice the coditio does ot deped o y. Essetially, the set A 0 C will either cotai the whole y-space if the trasmitted codeword u fulfills the costrait o g, or it will be the ull set whe the

4 4 costrait is ot met. Hece, E C {λ C = f C C f Y X y u dy dc R N y;ŵ Cy = f C C f Y X y u dy dc R N A C A 2C A NC A 0C f C C f Y X y u dydc R N A C T N + f C C f Y X y u dydc i=2 R N A ic {{ + f C C f Y X y u dydc. R N A 0C T 0 We ca cotiue from this poit o by evaluatig the itegrals T 0,T,T 2,...,T N separately. Startig with T 0, ote that the coditio defiig the set A 0 does ot deped o y, which gives T 0 = C R N ; f gu,k C C f Y X y u dydc k= S R = f X u du < ǫ, u R ; k= gu,k S for sufficietly large where the last iequality follows from the weak law of large umbers. That is, k= gu,k E X {gx = S ǫ i probability as. Cotiuig with T, T = f C C f Y X y u dydc R N y;u,y/ A ǫ = f X u f Y X y u dydu R y;u,y/ A ǫ = f Y,X y,u dydu = P X,Y / A ǫ X = u < ǫ, A ǫ for sufficietly large where the last iequality follows from lemma. Lastly T i for i = 2,...,N, T i = f C C f Y X y u dydc R N y;u i,y A ǫ = f X u i f Y ydydu i R y;u i,y A ǫ = f X u i f Y ydydu i 2 IY;X 3ǫ, A ǫ were the last iequality follows from lemma 2. These upper bouds result i E C { λc < 2ǫ+N2 IY;X 3ǫ < 2ǫ+2 R 2 IY;X 3ǫ = 2ǫ+2 IY;X R 3ǫ, where R logn large such that E C { λc < 3ǫ. T i 0 represets the code rate. For R < IY;X 3ǫ, we ca choose sufficietly

5 5 SiceE C { λc < 3ǫ, followig Shao s radom codig argumet, there exists a code,2 R, that has a average error probability that is smaller tha 3ǫ. I essece, we will oly pick a code book that will yield a small average error probability, which also meas that oly those codewords that satisfy the capacity cost costrait will be icluded. Note that the decodig regios A i ca be made disjoit without icreasig the upper boud o the average error probability. So far, we have ot said aythig about the maximum error probability, which is the quatity of iterest i order to commuicate with a certai level of reliability. Nevertheless, give a upper boud o the average error probability, say 3ǫ, the best N/2 codewords smallest error probabilities have a maximum error probability that is smaller tha 6ǫ. If it were ot true, the the average error probability would ot be smaller tha 3ǫ due to the cotributio of the worst N/2 codewords each havig a error probability greater tha 6ǫ, i.e., N N 2 6ǫ = 3ǫ 3ǫ. Usig this fact, we ca costruct a ew code cosistig oly of the N/2 best codewords, which would the reduce the ew rate R to R = logn/2 = R. We ca coclude ow that sice S ǫ < S, it follows that R < IX;Y < CS. However by choosig ǫ > 0 to be arbitrarily small ad sufficietly large, we ca achieve ay rate less tha CS. Capacity cost of the discrete-time cotiuous memoryless AWGN chael with a average iput power costrait: Cosider the discrete-time memoryless additive white Gaussia oise chael AWGN, i.e., Y = X+N, with X R as the iput, Y R as the output ad N R as the AWGN with variace σ 2. Further, let us cosider a average power costrait o the iput, i.e. E[X 2 ] S. The capacity cost for this chael ca be computed usig 2, with the cost fuctio gx = x 2. CS = max IX;Y = max hy hy X f X :E X{X 2 S f X :E X{X 2 S hn = max hy f X :E X{X 2 S 2 log2πeσ2 = max Var{Y:E X{X 2 S 2 log = 2 log + S σ 2. Var{Y where the maximum is achieved whe X is Gaussia distributed with mea 0 ad variace S. Later i Lecture 9, we will see that 2 log + S σ 2 is ideed the capacity of this chael. σ 2