17.1 Channel coding with input constraints

Transcription

1 7. Chaels with iput costraits. Gaussia chaels. 7. Chael codig with iput costraits Motivatios: Let us look at the additive Gaussia oise. The the Shao capacity is ifiite, sice sup P I(X; X + Z) = achieved by X N ( 0, P ) ad P. But this is at the price of X ifiite secod momet. I reality, limitatio of trasmissio power costraits o the ecodig operatios costraits o iput distributio. Defiitio 7.. A (, M, ɛ) -code satisfies the iput costrait F [ M] F. (Without costrait, the ecoder maps ito A ). A if the ecoder is f F A Codewords all lad i a subset of A Defiitio 7. (Separable cost costrait). A chael with separable cost costrait is specified as follows:. A, B: iput/output spaces. PY A B X, =,, Cost c A R Iput costrait: average per-letter cost of a codeword x (with slight abuse of otatio) Example: A = B = R Average power costrait (separable): k = c ( x ) = c(x k ) P i= Peak power costrait (o-separable): x i P x P max x i A x A i 74

2 Defiitio 7.3. Some basic defiitios i parallel with the chael capacity without iput costrait. A code is a (, M, ɛ, P )-code if it is a (, M, ɛ)-code satisfyig iput costrait F {x c (x k) P } Fiite- fudametal limits: M ɛ-capacity ad Shao capacity Iformatio capacity (, ɛ, P ) = max{ M (, M, ɛ, P )-code} M max (, ɛ, P ) = max{m (, M, ɛ, P ) max-code} C ɛ (P ) = lim if log M (, ɛ, P ) C(P ) = lim C ɛ ( P ) ɛ 0 C i (P ) = lim if sup P X E[ k= c(x k)] P I(X ; Y ) Iformatio stability: Chael is iformatio stable if for all (admissible) P, there exists a sequece of chael iput distributios P X such that the followig two properties hold: i.p. X,Y i P ( X ; Y ) C i ( P ) (7.) P[c(X ) > P + δ] 0 δ > 0. (7.) Note: These are the usual defiitios, except that i C i ( P ), we are permitted to maximize I( X ; Y ) usig iput distributios from the costrait set {PX E[ k= c ( X k)] P } istead of the distributios supported o F. Defiitio 7.4 (Admissible costrait). P is a admissible costrait if x0 A s.t. c( x 0 ) P P X E[ c( X)] P. The set of admissible P s is deoted by Dc, ad ca be either i the form ( P 0, ) or [ P 0, ), where P 0 if x c( x ). A Clearly, if P Dc, the there is o code (eve a useless oe, with codeword) satisfyig the iput costrait. So i the remaiig we always assume P Dc. Propositio 7.. Defie f( P ) = sup P I. The X E[ c( X)] P ( X; Y ). f is cocave ad o-decreasig. The domai of f, dom f { x f( x) > } = Dc.. Oe of the followig is true: f(p ) is cotiuous ad fiite o (P 0, ), or f = o (P 0, ). Furthermore, both properties hold for the fuctio P C i (P ). 75

3 Proof. I () all statemets are obvious, except for cocavity, which follows from the cocavity of PX I( X; Y ). For ay P Xi such that E [ c( X i )] P i, i = 0,, let X λp X0 + λp X. The E [ c( X)] λp 0 + λp ad I(X; Y ) λi (X 0 ; Y 0 )+ λi( X ; Y ). Hece f( λp 0 + λp ) λf ( P 0 )+ λf( P ). The secod claim follows from cocavity of f ( ). To exted these results to C i ( P ) observe that for every P sup I(X ; Y ) P X E[c(X )] P is cocave. The takig lim if the same holds for C i (P ). A immediate cosequece is that memoryless iput is optimal for memoryless chael with separable cost, which gives us the sigle-letter formula of the iformatio capacity: Corollary 7. (Sigle-letterizatio). Iformatio capacity of statioary memoryless chael with separable cost: C i (P ) = f(p ) = sup I( X; Y ). E[c( X)] P Proof. C i (P ) f( P ) is obvious by usig P X = ( PX ). For, use the cocavity of f ( ), we have that for ay PX, I(X ; Y ) j= I( X j ; Yj) j= f(e[c(x j )]) f( E[c(X j )]) f(p ). j= 7. Capacity uder iput costrait C( P ) = C i ( P ) Theorem 7. (Geeral weak coverse). C ɛ (P ) C i(p ) ɛ? Proof. The argumet is the same as before: Take ay (, M, ɛ, P )-code, W X Y W ˆ. Apply Fao s iequality, we have h( ɛ) + ( ɛ) log M I(W ; W ˆ ) I(X ; Y ) sup I(X ; Y ) f(p ) PX E[c(X )] P Theorem 7. (Exteded Feistei s Lemma). Fix a radom trasformatio P Y X. P X, F X, γ > 0, M, there exists a ( M, ɛ) max -code with: Ecoder satisfies the iput costrait: f [ M] F X ; Probability of error boud: ɛp X (F ) P[i(X; Y ) < log γ] + M γ 76

4 Note: whe F = X, it reduces to the origial Feistei s Lemma. Proof. Similar to the proof of the origial Feistei s Lemma, defie the prelimiary decodig regios E c = {y i(c; y) log γ} for all c X. Sequetially pick codewords { c,..., c M } from the j set F ad the fial decodig regio { D,..., D M } where D j E cj / k = D k. The stoppig criterio is that M is maximal, i.e., x 0 F, PY [E x0 / j= M D j M j Dj X X = x 0 ] < ɛ x 0 X, P Y [ E x0 / = = x 0 ] < ( ɛ) [ x 0 F ] + [ x 0 F c ] average over x 0 P X, P[{ i( X; Y ) log γ}/ M j= D j] ( ɛ)p X(F ) + P X(F c ) = ɛp X (F ) From here, we ca complete the proof by followig the same steps as i the proof of Feistei s lemma (Theorem 5.3). Theorem 7.3 (Achievability). For ay iformatio stable chael with iput costraits ad P > P 0 we have C(P ) C i (P ) (7.3) Proof. Let us cosider a special case of the statioary memoryless chael (the proof for geeral iformatio stable chael follows similarly). So we assume P Y X P = ( Y X ). Fix. Sice the chael is statioary memoryless, we have PY X = (P Y X ). Choose a P X such that E[ c( X)] < P, Pick log M = ( I( X; Y ) δ) ad log γ = ( I( X; Y ) δ ). With the iput costrait set F = {x c( x k ) P }, ad iid iput distributio PX = P X, we apply the exteded Feistei s Lemma, there exists a (, M, ɛ, P ) max -code with the ecoder satisfyig iput costrait F ad the error probability ɛ PX P (F ) (i(x ; Y ) ( I(X; Y ) δ)) + exp( δ) b 0 0 as y WLLN ad statioary memoryless assumptio Also, sice E[c(X)] < P, by WLLN, we have PX (F ) = P ( c (x k) P ). ɛ ( + o( )) o( ) ɛ 0 as ɛ, 0, s.t. 0, (, M, ɛ, P ) max -code, with ɛ ɛ Therefore C ɛ (P ) log M = I( X; Y ) δ, δ > 0, P X s.t. E[ c( X)] < P C ɛ ( P ) sup lim(i ( X; Y ) δ) P X E[c(X)]< P δ 0 Cɛ( P ) sup I( X; Y ) = C i ( P ) = C i ( P ) P X E[ c( X)]< P where the last equality is from the cotiuity of Ci o (P 0, ) by Propositio 7.. Notice that for geeral iformatio stable chael, we just eed to use the defiitio to show that P ( i( X ; Y ) (C i δ)) 0, ad all the rest follows. 77

5 Theorem 7.4 (Shao capacity). For a iformatio stable chael with cost costrait ad for ay admissible costrait P we have C( P ) = C i ( P ). Proof. The case of P = P 0 is treated i the homework. So assume P > P 0. Theorem 7. shows C ( Cɛ( P ) i P ) ɛ, thus C(P ) C i(p ). O the other had, from Theorem 7.3 we have C(P ) C i (P ). Note: I homework, you will show that C(P 0 ) = C i (P 0 ) also holds, eve though C i (P ) may be discotiuous at P Applicatios 7.3. Statioary AWGN chael Z N (0, σ ) X + Y Defiitio 7.5 (AWGN). The additive Gaussia oise (AWGN) chael is a statioary memoryless additive-oise chael with separable cost costrait: A = B = R, c( x) = x, P Y X is give by Y = X + Z, where Z N ( 0, σ ) X, ad average power costrait EX P. I other words, Y = X + Z, where Z N ( 0, I ) Gaussia Note: Here white = ucorrelated = idepedet. Note: Complex AWGN chael is similarly defied: A = B = C, c (x) = x, ad Z CN ( 0, I ) Theorem 7.5. For statioary (C)-AWGN chael, the chael capacity is equal to iformatio capacity, ad is give by:. C(P ) = C i (P ) = log ( + P σ ) C(P ) = C i (P ) = log ( + P σ ) for AWGN for C-AWGN Proof. By Corollary 7., C i = sup I( X; X + Z) P X EX P The use Theorem 4.6 (Gaussia saddlepoit) to coclude X N (0, P ) (or CN (0, P )) is the uique caid. 78

6 Note: Sice Z N (0, σ ), the with high probability, Z cocetrates aroud σ. Similarly, due the power costrait ad the fact that Z X, the received vector Y lies i a l -ball of radius (P + σ ). Sice the oise ca at most perturb the codeword by σ i Euclidea distace, if we ca pack M balls of radius σ ito the l -ball of radius (P + σ ) cetered at the origi, the this gives a good codebook ad decisio regios. The packig umber is related to the volume ratio. Note that the volume of a l-ball of radius r i R is give by c r for some c ((P +σ )) costat c. The / = ( + P ) /. Take the log σ c (σ ) / ad divide by, we get log ( + P σ ). c 5 c 6 c 4 (P + σ ) c M c 3 c σ c7 c c 8 Theorem 7.5 applies to Gaussia oise. What if the oise is o-gaussia ad how sesitive is the capacity formula log( + SNR to the Gaussia assumptio? Recall the Gaussia saddlepoit result we have studied i Lecture 4 where we showed that for the same variace, Gaussia oise is the worst which shows that the capacity of ay o-gaussia oise is at least log( + SNR ). Coversely, it turs out the icrease of the capacity ca be cotrolled by how o-gaussia the oise is (i terms of KL divergece). The followig result is due to Ihara. Theorem 7.6 (Additive No-Gaussia oise). Let Z be a real-valued radom variable idepedet of X ad EZ <. Let σ = Var Z. The log ( + P σ ) sup I(X; X + Z) P log ( + D P X EX P σ ) + (P Z N (EZ, σ )). Proof. Homework. Note: The quatity D(P Z N (EZ, σ )) is sometimes called the o-gaussiaess of Z, where N ( EZ, σ ) is a Gaussia with the same mea ad variace as Z. So if Z has a o-gaussia desity, say, Z is uiform o [ 0, ], the the capacity ca oly differ by a costat compared to AWGN, which still scales as log SNR i the high-snr regime. O the other had, if Z is discrete, the D( P Z N ( EZ, σ )) = ad ideed i this case oe ca show that the capacity is ifiite because the oise is too weak Parallel AWGN chael Defiitio 7.6 (Parallel AWGN). A parallel AWGN chael with L braches is defied as follows: A = B = R L ; c(x) = L k= x k ; P Y L X L Y k = X k + Z k, for k =,..., L, ad Zk N ( 0, σ k ) are idepedet for each brach. Theorem 7.7 (Waterfillig). The capacity of L-parallel AWGN chael is give by C = L j= log + T σ j where log + (x) max(log x, 0), ad T 0 is determied by L P = T σj + j= 79

7 Proof. C i (P ) = = P sup I(X L ; Y L ) X L E[X i ] P I L sup sup P k P,P k 0 k= E[ Xk ] P k sup L ( + P k log ) P k P,P k 0 k= σ k (X k ; Y k ) with equality if X k N (0, P k ) are idepedet. So the questio boils dow to the last maximizatio problem power allocatio: Deote the Lagragia multipliers for the costrait P k P by λ Pk ad for the costrait P k 0 by µ k. We wat to solve max log( + ) µ k P σ k + λ(p P k ). k First-order coditio o P k gives that σk + P k = λ µ k, µ k P k = 0 therefore the optimal solutio is P k = T σk +, T is chose such that P = L k= T σk + Note: The figure illustrates the power allocatio via water-fillig. I this particular case, the secod brach is too oisy (σ too big) such that it is better be discarded, i.e., the assiged power is zero. Note: [Sigificace of the waterfillig theorem] I the high SNR regime, the capacity for AWGN chael is approximately log P, while the capacity for L parallel AWGN chael is approximately L log( P L ) L log P for large P. This L-fold icrease i capacity at high SNR regime leads to the powerful techique of spatial multiplexig i MIMO. Also otice that this gai does ot come from multipath diversity. Cosider the scheme that a sigle stream of data is set through every parallel chael simultaeously, with multipath diversity, the effective oise level is reduced to L, ad the capacity is approximately log(lp ), which is much smaller tha L log( P for P large. L ) 7.4* No-statioary AWGN Defiitio 7.7 (No-statioary AWGN). A o-statioary AWGN chael is defied as follows: A = B = R, c( x) = x, P Yj X j Y j = X j + Z j, where Z j N ( 0, σj ). 80

8 Theorem 7.8. Assume that for every T the followig limits exist: C i (T ) = lim j= P (T ) = lim T log+ σj T σj + the the capacity of the o-statioary AWGN chael is give by the parameterized form: C( T ) = C i ( T ) with iput power costrait P ( T ). Proof. Fix T > 0. The it is clear from the waterfillig solutio that j= sup I(X ; Y T ) = j= log+ σj, (7.4) where supremum is over all P X such that E[c(X )] T σ j +. (7.5) Now, by assumptio, the LHS of (7.5) coverges to P (T ). Thus, we have that for every δ > 0 C i P ( T ) δ) C i ( T ) (7.6) C i ( (P ( T ) + δ) C i ( T ) (7.7) j= Takig δ 0 ad ivokig cotiuity of P C i (P ), we get that the iformatio capacity satisfies C i ( P (T )) = C i (T ). The chael is iformatio stable. Ideed, from (6.7) P j e Var(i( X ; Y j )) = log j P j + σj log e ad thus j= Var (i(x j ; Y j )) <. From here iformatio stability follows via Theorem 6.9. Note: No-statioary AWGN is primarily iterestig due to its relatioship to the statioary Additive Colored Gaussia oise chael i the followig discussio. 7.5* Statioary Additive Colored Gaussia oise chael Defiitio 7.8 (Additive colored Gaussia oise chael ). A Additive Colored Gaussia oise chael is defied as follows: A = B = R, c( x) = x, P Y = + Gaussia process with spectral desity f Z (ω) > 0, ω [ j X j Y j X j Z j, where Z j is a statioary π, π]. 8

9 Theorem 7.9. The capacity of statioary ACGN chael is give by the parameterized form: C(T ) = π π T 0 log+ f Z (ω) dω P (T ) = π π T f Z (ω) + dω 0 Proof. Take, cosider the diagoalizatio of the covariace matrix of Z : Cov(Z ) = Σ = U Σ U, such that Σ = diag(σ,..., σ ) Sice Cov(Z ) is positive semi-defiite, U is a uitary matrix. Defie X = UX ad Y = UY, the chael betwee X ad Y is thus Ỹ = X + UZ, Cov( UZ ) = UCov(Z )U = Σ Therefore we have the equivalet chael as follows: By Theorem 7.8, we have that C = lim j= lim Ỹ = X + Z, Z j N (0, σ j ) idep across j log + T σ j T σj + = P (T ) = j = π π T 0 log+ dω. ( by Szegö, Theorem 5.6) f Z (ω) Fially sice U is uitary, C = C. Note: Noise is bor white, the colored oise is essetially due to some filterig. 8

10 7.6* Additive White Gaussia Noise chael with Itersymbol Iterferece Defiitio 7.9 (AWGN with ISI). A AWGN chael with ISI is defied as follows: A = B = R, c( x) = x, ad the chael law P Y X is give by Y k = j= h k j X j + Z k, k =,..., where Z k N (0, ) is white Gaussia oise, {h k, k =,..., } are coefficiets of a discrete-time chael filter. Theorem 7.0. respose H( ω) : Suppose that the sequece { h k } is a iverse Fourier trasform of a frequecy h k = π e iωk H( ω) dω. π 0 Assume also that H( ω) is a cotiuous fuctio o [0, π]. The the capacity of the AWGN chael with ISI is give by C(T ) = π π 0 log+ (T H(ω) )dω P (T ) = π + π T dω 0 H(ω) Proof. (Sketch) At the decoder apply the iverse filter with frequecy respose ω H(ω). The equivalet chael the becomes a statioary colored-oise Gaussia chael: Ỹ j = X j + Z j, where Z j is a statioary Gaussia process with spectral desity f Z(ω) =. H( ω) The apply Theorem 7.9 to the resultig chael. Remark: to make the above argumet rigorous oe must simply carefully aalyze the o-zero error itroduced by trucatig the decovolutio filter to fiite. 7.7* Gaussia chaels with amplitude costraits We have examied some classical results of additive Gaussia oise chaels. I the followig, we will list some more recet results without proof. Theorem 7. (Amplitude-costraied capacity of AWGN chael). For a AWGN chael Y i = X i + Z i with amplitude costrait X i A ad eergy costrait i= X i P, we deote the capacity by: C( A, P ) = max I( X; X + Z ). P X X A,E X P Capacity achievig iput distributio PX is discrete, with fiitely may atoms o [ A, A ]. Moreover, the covergece speed of lim A C( A, P ) = log( + P ) is of the order e O(A). For details, see [Smi7] ad [PW4, Sectio III]. 83

11 7.8* Gaussia chaels with fadig Fadig chaels are ofte used to model the urba sigal propagatio with multipath or shadowig. The received sigal Y i is modeled to be affected by multiplicative fadig coefficiet H i ad additive oise Z i : Y i = H i X i + Z i, Z i N ( 0, ) I the coheret case (also kow as CSIR for chael state iformatio at the receiver), the receiver has access to the chael state iformatio of H i, i.e. the chael output is effectively ( Y i, H i ). Wheever H j is a statioary ergodic process, we have the chael capacity give by: C( P ) = E[ log( + P H )] ad the capacity achievig iput distributio is the usual PX = N ( 0, P ). Note that the capacity C( P ) is i the order of log(p ) ad we call the chael eergy efficiet. I the o-coheret case where the receiver does ot have the iformatio of H i, o simple expressio for the chael capacity is kow. It is kow, however, that the capacity achievig iput distributio is discrete, ad the capacity This chael is said to be eergy iefficiet. C( P ) = O( log log P ), P (7.8) With itroductio of multiple atea chaels, there are edless variatios, theoretical ope problems ad practically uresolved issues i the topic of fadig chaels. We recommed cosultig textbook [TV05] for details. 84

12 MIT OpeCourseWare Iformatio Theory Sprig 06 For iformatio about citig these materials or our Terms of Use, visit: