Information Theory Tutorial Communication over Channels with memory. Chi Zhang Department of Electrical Engineering University of Notre Dame

Transcription

1 Iformatio Theory Tutorial Commuicatio over Chaels with memory Chi Zhag Departmet of Electrical Egieerig Uiversity of Notre Dame

2 Abstract A geeral capacity formula C = sup I(; Y ), which is correct for arbitrary sigle-user chaels without feedback, is itroduced i this tutorial. This ew capacity formula is obtaied by usig a geeral chael model without ay assumptios of the chael, itroducig the otio of if/supiformatio/etropy rates, ad fidig a ew tight coverse boud o the error probability. I Sectio we will show the error of the covetioal chael capacity formula. The i sectio2, we will give some defiitios used i the proofs of the capacity formula. Sectio 3 is devoted to the direct codig theorem. Sectio 4 will focus o the ew coverse boud ad the geeral capacity formula. We will see some useful results of the ew coverse boud i sectio 5. I sectio 6, we will see how feedback ca icrease the chael capacity whe the chael has memory. Shao s formula I. INTRODUCTION [] for chael capacity (the supremum of all rates R for which there exist sequeces of codes with vaishig error probability ad whose size grows with the block legth as exp(r)), C = max I(; Y ), () holds for memoryless chaels. If the chael has memory, the geeralizes to the familiar limitig expressio C = lim sup x 0 x I( ; Y ). (2) However, the capacity formula (2) does ot hold i full geerality. Let s see a example. Example: Cosider a biary chael where the output codeword is equal to the trasmitted codeword with probability l/2 ad idepedet of the trasmitted codeword with probability l/2. Obviously, the capacity of this chael is equal to 0. However the right-had side of (2) is equal to l/2 bit/chael use. Aother couter example ca be foud at Nedoma [6] The validity of (2) was proved for some specific class of chaels, like iformatio stable chaels. Researchers also try to get a capacity formula for iformatio ustable chaels; however, all these approaches rely o some assumptios of the chael. There is a call for a completely geeral formula for chael capacity. The desired chael capacity is expressed i terms of he probabilistic descriptio of the chael, which does t require ay assumptio of the chael, such as memorylessess, iformatio stability, statioarity, causality, etc. Such a formula is foud i A Geeral Formula for Chael Capacity [2], which will be the mai topic of this tutorial.

3 II. BACKGROUND 2 Before itroducig the ew chael capacity formula, we eed several defiitios. Defiitio : Limif i Probability If A is a sequece of radom variables, its limif i probability is the supremum of all the reals α for which P [A α] 0 as. Limsup i Probability If A is a sequece of radom variables, its limsup i probability is the ifimum of all the reals β for which P [A β] 0 as. Defiitio 2:A chael W with iput ad output alphabets, A ad B, respectively, is a sequece of coditioal distributios W = {W (y x ) = P Y (y x ); (x, y ) A B } =. Remarks: The chael defiitio i this paper is very geeral, without placig ay restrictio o the chael. It s also a reasoable chael model sice it captures the physical situatio to be modeled where block codewords are trasmitted through the chael. For the coveiece sake, i the proofs of this geeral chael capacity formula, we will assume that the iput ad output alphabets are fiite; however, the proofs do t deped o this assumptio. Defiitio 3: Give a joit distributio P Y (y x ) = P (x )W (y x ),the iformatio desity is the fuctio defied o A B : Remarks: i W (a ; b ) = log PY (b a ) P Y (b ) The distributio of the radom variable (/)i W (, Y ) where ad Y have joit distributio P Y will be referred to as the iformatio spectrum. The expected value of the iformatio spectrum is the ormalized mutual iformatio (/)I( ; Y ). Defiitio 4: If-Iformatio Rate The limif i probability of the sequece of radom variables (/)i W ( ; Y ) will be referred to as the the if-iformatio rate of the pair(,y) ad will be deoted as I(; Y ). Sup-Iformatio Rate The limsup i probability of the sequece of radom variables (/)i W ( ; Y ) will be referred to as the the sup-iformatio rate of the pair(,y) ad will be deoted as I(; Y ). Remarks: The itroductio of if/sup-iformatio/etropy rates eables us to deal with oergodic/ostatioary sources.

4 Usig these defiitios, the geeral capacity formula is 3 C = sup I(; Y ). (3) I (3), deotes a iput process i the form of a sequece of fiite-dimesioal distributios = { = (,..., )} =. We deote by Y = {Y = (Y,..., Y )} =.the correspodig output sequece of fiite-dimesioal distributios iduced by via the chael W = {W = P Y : A B } =, which is a arbitrary sequece of -dimesioal coditioal output distributios from A to B, where A ad B are the iput ad output alphabets, respectively. Before the proof of this ew capacity formula, we will show some importat properties of if-iformatio rate. Theorem : A arbitrary sequece of joit distributios (, Y) satisfies a) D( Y ) 0 b) I(; Y ) = I(Y ; ) c) I(; Y ) 0 d) I(; Y ) H(Y ) H(Y ), I(; Y ) H(Y ) H(Y ), I(; Y ) H(Y ) H(Y ) e) 0 H(Y ) < log B f) I(, Y ; Z) I(; Z) g) If I(; Y ) = I(; Y ) ad the iput alphabet is fiite, the I(; Y ) = lim (/)I( ; Y ) h) I(; Y ) lim if (/)I( ; Y ). i) (Data Processig Theorem) I( ; 3 ) I( ; 3 ), if for every, ad 3 are coditioally idepedet give 2. Defiitio 5: () The if-divergece rate D( Y ) is defied for two arbitrary processes U ad V as the limif i probability of the sequece of log-likelihood ratios log PU(U ) P V (V ) (2) The sup-etropy rate H(Y ) ad if-etropy H(Y ) are defied as the limsup ad limif, respectively, i probability of the ormalized etropy desity log P Y (y ). Similarly, the coditioal sup-etropy rate H(Y ) is the limsup i probability (accordig to P Y ) of log P Y (y x ). Proof: The proofs ca be foud at [2]. Here we will try to iterpret the above results. Remarks: ()May of the familiar properties satisfied by mutual iformatio tur out to be iherited by the ififormatio rate. (2)The oegativity of the if-divergece rate D( Y ) plays a key role i the proof of these properties,

5 4 just like its couterpart, the oegativity of divergece, i the proof of may of the mutual iformatio properties. (3)The proof of (e) ca be foud at [3]. That paper tells us that the miimum achievable source codig rate for ay fiite-alphabet source = = is equal to its sup-etropy rate H(), defied as the limsup i probability of (/) log /P ( ). (4)The proof of (g) ca also be foud at [3]. Whe (,Y) satisfies this property, the iput-output pair(,y) is called iformatio stable. III. DIRECT CODING THEOREM: Defiitio 6: A (, M, ɛ) code has block legth, M codewords, ad error probability ot larger tha ɛ. R 0 is a ɛ-achievable rate if, for every δ > 0, there exist, for all sufficietly large, (, M, ɛ) codes with rate log M > R δ. The maximum ɛ-achievable rate is called the ε-capacity C ɛ. The chael capacity C is defied as the maximal rate that is ɛ-achievable for all 0 < ɛ <. It follows immediately from the defiitio that C = lim ɛ 0 C ɛ Theorem 2 (Feistei s lemma): Fix a positive iteger ad 0 < ɛ <. For every γ > 0 ad iput distributio P x o A, there exists a (, M, ɛ) code for the trasitio probability W = P Y that satisfies ɛ P [ i W ( ; Y ) log M + γ] + exp( γ) (4) The direct part of the codig theorem follows Feistei s lemma ad the defiitios of capacity ad if-iformatio rate. Theorem 3: Proof: C sup I(; Y ). (5) Fix arbitrary 0 < ɛ < ad. We shall show that I(; Y ) is a ɛ-achievable rate by demostratig for every δ > 0 ad all sufficietly large, there exist (, M, exp( δ/4) + ɛ/2)codes with rate I(; Y ) δ < log M < I(; Y ) δ 2 If, i Theorem, we choose γ = δ/4,the the probability i (4) becomes (6) P [ i W ( ; Y ) log M + δ/4] P [ i W ( ; Y ) I(; Y ) δ/4] ɛ 2 (7)

6 5 where the secod iequality holds for all sufficietly large because of the defiitio of I(; Y ). I view of (7), Theorem 2 guaratees the existece of the desired codes. IV. CONVERSE CODING THEOREM: I this sectio, we will itroduce a ew coverse, which is the mai result of [2]. It s tight for every chael, ad it s obtaied without recourse to the Fao iequality. We will also compare this ew coverser boud with the covetioal Fao iequality, which is short of providig a tight coverse for some chaels. Theorem 4: Every (, M, ɛ) code satisfies ɛ P [ i W ( ; Y ) log M γ] exp( γ) (8) for every γ > 0,where places probability mass /M o each codeword. Proof: Deote β = exp( γ). Note first that the evet whose probability appears i (8) is equal to the set of atypical iput-output pairs L = {(a, b ) A B : P Y (a b ) β} (9) This is because the iformatio desity ca be writte as i W (a ; b ) = log P Y (a b ) P (a ) ad P (c i ) = /M for each of the M codewords c i A. We eed to show that Now, deotig the decodig set correspodig to c i by D i ad (0) P Y [L] ɛ + β () B i = {b B : P Y (c i b ) β} (2) We ca write = i= M P Y [L] = P Y [(c i, B i )] (3) i= M P Y [(c M i, B i D c i )] + P Y [(c i, B i Di )] (4) M i= i= M W (D c i c i ) + βp Y [ M D i ] (5) i= ɛ + β (6)

7 where the secod iequality is due to (2) ad the disjoitess of the decodig sets. 6 Remark: ()This coverse shows a dual of Feistei s result: the average error probability of ay code is essetially lower-bouded by the cumulative distributio fuctio of the iput-output iformatio desity evaluated at the code rate. (2)Theorem 4 gives a family (parameterized by γ) of lower bouds o the error probability. To obtai the best boud, we simply maximize the right-had side of (8) over γ. (3)A weaker boud Usig the Fao iequality, every (, M, ɛ) code satisfies log M ɛ [I( ; Y ) + h(ɛ)]; (7) where h is the biary etropy fuctio, is the iput distributio that places probability mass l/m o each of the iput codewords, ad Y is its correspodig output distributio. Usig (7), it is evidet that if R 0 is ɛ-achievable, the for every δ > 0. which, i tur, implies Thus, the geeral coverse i (2) follows by lettig ɛ 0 R δ < ɛ [ sup I( ; Y ) + h(ɛ) ] (8) R lim if ɛ sup I( ; Y ). (9) This weak boud is uable to provide the desired tight coverse because it depeds o the chael through the iput-output mutual iformatio (expectatio of iformatio desity) achieved by the code. However, the ew coverse boud oly depeds o the distributio of the iformatio desity achieved by the code, rather tha o just its expectatio. Theorem 5: Proof: C sup I(; Y ). (20) Usig the cocept of the iformatio spectrum [3], Theorem 4 tells us that if a reliable code sequece has rate R, the the mass of its iformatio spectrum lyig strictly to the left of R must be asymptotically egligible. With this isight, the proof i [2] use a cotradictio to prove the coverse part of the chael codig theorem.

8 V. OTHER RESULTS 7 We ca use the results above to fid upper ad lower bouds o C ɛ, the ɛ-capacity of the chael, for 0 < ɛ <. These bouds coicide at the poits where the ɛ-capacity is a cotiuous fuctio of ɛ. Theorem 6: For 0 < ɛ <, the ɛ-capacity C, satisfies C ɛ, where F,(R) deotes the limit of cumulative distributio fuctios C ɛ sup sup R : F (R) ɛ (2) C ɛ sup sup R : F (R) < ɛ (22) F (R) = lim sup P [ i W (, Y ) R]. (23) The bouds (2) (22) hold with equality, except possibly at the poits of discotiuity of C ε, of which there are, at most, coutably may. VI. GAUSSIAN FEEDBACK CAPACITY Pisker ad Ebert showed that feedback at most doubles the capacity of a o-white Gaussia chael; a simple proof ca be foud i Cover ad Pombra [4]. Lemma : For A,B oegative defiite matrices ad 0 λ Lemma 2: If ad Z are causally related, the ad Remark: λa + ( λ)b A λ B λ (24) h( Z) h(z) (25) K Z (26) The radom vector is causally related to Z if f(x, Z ) = f(z ) i= f(x i x i, z i ). Note that feedback codes ecessarily yield causally related (, Z ). Theorem 7: C,F B 2C. 2 log K +Z = 2 log 2 K +Z + 2 K Z (27) 2 log K +Z /2 K Z /2 (28) 2 log K +Z /2 /2 (29) = 2 2 log K +Z (30)

9 By maximizig each side, we have 2 C,F B C d (3) 8 REFERENCES [] C. E. Shao, A mathematical theory of commuicatio, Bell Syst. Tech..I., vol. 27, pp , , July-Oct [2] S. Verdu ad T. S. Ha, A Geeral Formula for Chael Capacity, IEEE Tras. Iform. Theory, July 994. [3] T. S. Ha ad S. Verdu, Approximatio theory of output statistics,ieee Tras. Iform. Theory, vol. 39, pp , May 993. [4] T. M. Cover ad S. Pombra, Gaussia feedback capacity, IEEE Tras. Iform. Theory, vol. 35, pp. 37C43, Ja [5] A. Feistei, A ew basic theorem of iformatio theory, IRE Tras. Iform. Theory, vol. IT-4, pp. 2-22, 954. [6] I. Nedoma, The capacitv of a discrete chael, i Proc. st Prague Cof. Iform. Theory, Statist. Decisio Fuctios, Radom Processes, Prague, 957, pp