A Coding Theorem for a Class of Stationary Channels with Feedback

Transcription

1 A Codg Theorem for a Class of Stationary Channels with Feedback Young-Han Kim Department of Electrical and Computer Engeerg University of California, San Diego La Jolla, CA 92093, USA yhk@ucsd.edu Abstract A codg theorem is proved for a class of stationary channels with feedback which the output Y n = f(x n n m, Z n n m) is the function of the current and past m symbols from the channel put X n and the stationary ergodic channel noise Z n. In particular, it is shown that the feedback capacity is equal to lim n p(x n y n ) n I(Xn Y n ), where I(X n Y n ) = P n I(Xi ; Y i Y i ) denotes the Massey directed formation from the channel put to the output, and the remum is taken over all causally conditioned distributions p(x n y n ) = Q n p(x i x i, y i ). The ma ideas of the proof are the Shannon strategy for codg with side formation and a new elementary codg technique for the given channel model without feedback, which is a sense dual to Gallager s lossy codg of stationary ergodic sources. A similar approach gives a simple alternative proof of codg theorems for fite state channels by Yang Kavčić Tatikonda, Chen Berger, and Permuter Weissman Goldsmith. I. INTRODUCTION In [], Massey troduced the mathematical notion of directed formation I(X n Y n ) = I(X i ; Y i Y i ), and established its operational meang by showg that the feedback capacity is upper bounded by the imum normalized directed formation, which can be general tighter than the usual mutual formation. Sce then there have been many attempts to show that Massey s directed formation is deed the feedback capacity, namely, n p(x n y n ) = lim n p(x n y n ) n I(Xn Y n ) () I(X i ; Y i Y i ), n where the remum is taken over all causally conditioned probabilities p(x n y n ) = n p(x i x i, y i ). For example, a heroic effort [7], [8], Tatikonda attacked the general nonanticipatory channel with feedback by combg Verdú Han formula for nonfeedback capacity [9], Massey directed formation, and Shannon strategy for channel side formation [6], as well as dynamic programmg for Markov decision processes. As the cost of generality, however, it is extremely difficult to establish a simple formula like (). See, for example, [8, Theorem 7.5] for a major hurdle provg the equivalence between a Verdú Han-type formula and (). More recently, Yang, Kavčić, and Tatikonda [20] and Chen and Berger [3] studied special cases of fite-state channels, based on Tatikonda s framework. A fite-state channel [6, Section 4.6] is described by the conditional probability distribution p(y n, s n x n s n ), (2) where s n denotes the channel state at time n. Usg a different approach based on Gallager s proof of the nonfeedback capacity [6, Section 5.9], Permuter, Weissman, and Goldsmith [5] proved various codg theorems for fite-state channels with feedback that, ter alia, subsume many results [20], [3] and establish the validity of () for decomposable fitestate channels without tersymbol terference (i.e., fitestate channels whose state evolves as an ergodic Markov cha, dependent of the channel put). The ma goal of this paper is to establish the validity of the feedback capacity formula () for a reasonably general class of channels with memory, the simplest manner. Towards this goal, we focus on stationary nonanticipatory channels of the form {, i =,..., m, Y i = g(xi m i, Zi i m ), i = m +, m + 2,..., (3) where the time-i channel output Y i on the output alphabet Y is given by a determistic map f : X m Z m Y of the current and past m channel puts Xi m i on the put alphabet X and the current and past m channel noises Zi m i on the noise alphabet Z. We assume that the noise process {Z n } n= is an arbitrary stationary ergodic process (without any mixg condition). The channel model (3) is rather simple and physically motivated. Yet this channel model is general enough to clude many important feedback communication models such as any additive noise fadg channels with tersymbol terference and decomposable fite-state channels without tersymbol terference. A notable exception is a famous fite-state channel called the trapdoor channel troduced by Blackwell [], the feedback capacity of which is established [4] /07/$25.00 c 2007 IEEE 856

2 The channel (3) has fite put memory the sense of Feste [5] and can be viewed as a fite-wdow slidgblock coder [7, Section 9.4] of put and noise processes (cf. primitive channels troduced by Neuhoff and Shields [3] which the noise process is memoryless). Compared to the general fite-state channel model (2), which the channel has fite put memory but the channel noise is memoryless, our channel model (3) has fite put memory but the noise has fite memory; recall that there is no mixg condition on the noise process {Z n } n=. Thus, the fite-state channel model and the fite slidg-block channel model nicely complement each other. Our ma result is to show that the feedback capacity C FB of the channel (3) is characterized by (). More precisely, we consider a communication problem depicted Figure. Here one wishes to communicated a message dex W W [2 nr ] Xn (W, Y n ) X i (W, Y i ) Z n Y i = g(x i i m, Zi i m ) Y n Ŵ n(y n ) Fig.. Feedback communication channel Y i = g(x i i m, Zi i m ). [2 nr ] := {, 2,..., 2 nr } over the channel (3). We assume that the channel noise process {Z i } is stationary ergodic and is dependent of the message W. The itial values of Y,..., Y m are set arbitrarily. They depend on the unspecified itial condition (X m+, 0 Z m+), 0 the effect of which vanishes from time m +. Thus the long term behavior of the channel is dependent of Y m. We specify a (2 nr, n) code with the encodg maps X n (W, Y n ) = (X (W ), X 2 (W, Y ),..., X n (W, Y n )), and the decodg map Ŵn : Y n [2 nr ]. The probability of error P e (n) is defed as P e (n) = Pr{Ŵn(Y n ) W }, where the message W is uniformly distributed over [2 nr ] and is dependent of {Z i }. We say that the rate R is achievable if there exists a sequence of (2 nr, n) codes with P e (n) 0 as n. The feedback capacity C FB is defed as the remum of all achievable rates. The nonfeedback capacity C is defed similarly, with codewords X n (W ) = (X (W ),..., X n (W )) restricted to be a function of the message W only. We will prove the followg result Section II. Theorem. The feedback capacity C FB of the channel (3) is given by n p(x n y n ) n I(Xn Y n ). (4) Our development has two major gredients. First, we use the codg theorem for the same channel without feedback, fully described [0]. This approach is somewhat different from the conventional approaches such as Shannon s random codebook generation and typicality decodg, Gallager s random codg exponent method, or Feste s imal codg argument. We use the strong typicality (relative frequency) decodg for n-dimensional er letters. A constructive codg scheme (up to the level of Shannon s random codebook generation) based on block ergodic decomposition of Nedoma [2] is developed, which uses a long codeword on the n-letter er alphabet, constructed as a concatenation of n shorter codewords. While each short codeword and the correspondg output fall to their own ergodic mode, the long codeword as a whole matas the ergodic behavior. To be fair, codebook construction of this type is far from new the literature, and our method is timately related to the one used by Gallager [6, Section 9.8] for lossy compression of stationary ergodic sources. Indeed, when the channel (3) has zero memory (m = 0), then the role of the put for our channel codg scheme is equivalent to the role of the coverg channel for Gallager s source codg scheme. Equipped with this codg method for nonfeedback slidgblock coder channels (3), the extension to the feedback case is relatively straightforward. The basic gredient for this extension is the Shannon strategy for channels with causal side formation at the transmitter [6]. As a matter of fact, Shannon himself observed that the major utility of his result is feedback communication. Followg is the first sentence of [6]: Channels with feedback from the receivg to the transmittg pot are a special case of a situation which there is additional formation available at the transmitter which may be used as an aid the forward transmission system. As observed by Caire and Shamai [2, Proposition ], the causality has no cost when the transmitter and the receiver share the same side formation our case, the past put (if decoded faithfully) and the past output (received from feedback) and the transmission can fully utilize the side formation as if it were known a priori. Thus, tuitively speakg, we can achieve the rate R = p(x n y n ) I(X i ; Y n i X i, Y i ) per n transmissions. Now a simple algebra shows that this rate is equal to the imal directed formation: I(X i ; Y n i X i, Y i ) = I(X n Y n ). (5) The above argument, while tuitively appealg, is not completely rigorous, however. Therefore, we will take more careful steps, by first provg the achievability of n I(U n ; Y n ) for all auxiliary random variables U n and Shannon strategies X i (U i, X i, Y i ), i =,..., n, and then showg that I(U n ; Y n ) reduces to I(X n Y n ) via pure algebra. II. PROOF OF THEOREM Recall our channel model {, i =,..., m, Y i = g(xi m i, Zi i m ), i = m +, m + 2,..., (6) 857

3 with the put X i and the stationary ergodic noise process {Z i }, as depicted Figure. We prove that the feedback capacity is given by n C FB,n = lim n p(x n y n ) n I(Xn Y n ), (7) where the remum is over all causally conditioned distributions n p(x n y n ) = p(x i x i, y i ). The followg lemma is crucial to the proof of Theorem. Lemma. Suppose a causally conditioned distribution p(y n x n ) is given. Then we have I(U n ; Y n ) = p(u n ),x i=f(u i,x i,y i ) p(x n y n ) I(Xn Y n ), (8) where the imum on the left hand side is taken over all jot distributions of the form p(u n, x n, y n ) = n ( p(ui )p(x i u i, x i, y i )p(y i x i, y i ) ) ( n ) = p(u i )p(x i u i, x i, y i ) p(y n x n ) with determistic p(x i u i, x i, y i ), i =,..., n, and the auxiliary random variables U i has the cardality bounded by U i X i Y i. Proof. Let q(u n, x n, y n ) be any jot distribution of the form (9) such that q(x i u i, x i, y i ), i =,..., n are determistic and that q(y n x n ) = p(y n x n ) (i.e., the jot distribution q(u n, x n, y n ) is consistent with the given causally conditioned distribution p(y n x n )). For (U n, X n, Y n ) q(u n, x n, y n ), it is easy to verify that Ui n is dependent of (U i, X i, Y i ), which implies that U i (X i, Y i ) Yi n forms a Markov cha. On the other hand, X i is a determistic function of (U i, Y i ) and thus X i (U i, Y i ) Yi n also forms a Markov cha. Similarly, we have the Markovity for U i (Y i, X i ) Yi n and X i (U i, Y i ) Yi n. Therefore, we have I(U i ; Y n U i ) = I(U i ; Yi n Y i, U i ) (0) = H(Yi n Y i, U i ) H(Yi n Y i, U i ) = H(Yi n Y i, X i ) H(Yi n Y i, X i ) () = I(X i ; Y n i X i, Y i ), where (0) follows from the dependence of U i and (U i, Y i ), and () follows from Markov relationships observed above. Now from the alternative expansion of the directed formation shown (5), we have q I(U n ; Y n ) = I(X n Y n ). q (9) Fally, by usg distributions of the form p(x i x i, y i ) = u i p(u i )p(x i u i, x i, y i ) with appropriately chosen p(u i ) and determistic p(x i u i, x i, y i ), we can represent any causally conditioned distribution n p(x i x i, y i ) = n ( p(ui )p(x i u i, x i, y i ) ), u n which implies the desired result. That the limit (7) is well-defed follows from the eradditivity 2 of nc FB,n. Thus, C FB,n = C FB,n. n n The converse was proved by Massey [, Theorem 3]. For any sequence of (2 nr, n) codes with P e (n), we have from Fano s equality nr I(W ; Y n ) + nɛ n = I(W ; Y i Y i ) + nɛ n = I(X i ; Y i Y i ) + nɛ n (2) = I(X n Y n ) + nɛ n, where ɛ n 0 as n. Here (2) follows from the codebook structure X i (W, Y i ) and the Markovity W (X i, Y i ) Y i. For the achievability, we show that there exists a sequence of codes that achieves C FB,n for each n. For simplicity, we assume that the alphabets X and Y are fite. A complete argument is given [0]. In the light of Lemma, it suffices to show that C FB,n = I(U n ; Y n ) (3) p(u n ),x i=f(u i,x i,y i ) is achievable, where the auxiliary random variables U i has the cardality bounded by U i X i Y i, and the imization is over all jot distributions of the form ( n ) p(u n, x n, y n ) = p(u i )p(x i u i, x i, y i ) p(y n x n ) with determistic p(x i u i, x i, y i ), i =,..., n. Codebook generation and encodg. Fix n and let p i (u i), i =,..., n, and f i : (u i, x i, y i ) x i, i =,..., n, achieve the imum of (3). We will also use the notation p (u n ) = n p i (u i) and f (u n, x n, y n ) = (f (u ),..., f n(u n, x n, y n )). For each k = k(l, n) = Ln 2 + n, L =, 2,..., we generate a (2 kr, k) code {X i (W, Y i )} k as summarized 2 that is, n C FB,n +n 2 C FB,n2 (n +n 2 )C FB,n +n 2, which is an easy consequence of the stationarity of the process {Z i } and the defition of the channel model (6), particular, Y i =, i =,..., m. 858

4 Figure 2. Here X (n), Ỹ(n), and Z (n) are copies the underlyg sequences X, Y, Z with every (Ln + )st symbol omitted. For each w {, 2,..., 2 kr }, we generate a codeword U (n) (w) = U Ln2 (w) of length Ln on the n-letter alphabet U U n dependently accordg to Ln p(u Ln2 ) = p (u ni (n )i+ ). This gives a 2 kr Ln codebook matrix with each entry drawn i.i.d. accordg to p (u n ). To communicate the message W = w, the transmitter chooses the codeword U (n) (w) = U Ln2 (w) and sends ( ) X (i )n+j = fj (i )n+j U j (w), X (i )n+, Ỹ (i )n+j (i )n+, i =,..., Ln, j =,..., n. Thus, the code function X n (w, Y n ) utilizes the codeword U (n) and the channel feedback Ỹ(n) only with the frame of n transmissions (each box Figure 2). Decodg. Upon receivg Y k, the receiver declares that the message Ŵ was sent if there is a unique Ŵ such that (U (n) (Ŵ ), Ỹ(n) ) A (Ln) ɛ (U n, Y n ), that is, (U (n) (Ŵ ), Ỹ(n) ) is jotly typical with respect to the jot distribution p(u n, y n ) specified by p (u n )p(z n ), x i = f i (u i, x i, y i ), and the defition of the channel (6). Otherwise, an error is declared. Analysis of the probability of error. We defe the followg events: E i = {(U (n) (i), Ỹ(n) ) A (Ln) ɛ (U n, Y n )}, i [2 kr ] Without loss of generality, we assume W = was sent. From [0, Lemma 7], U (n) () and Z (n) are jotly typical with high probability for L sufficiently large. Furthermore, Ỹ (n) is an n-letter blockwise function of ( X (n) (), Z (n) ), and thus of (U (n) (), Z (n) ). Therefore, the probability of the event E c that the tended codeword U (n) () is not jotly typical with Ỹ(n) vanishes as L. On the other hand, U (n) (i), i, is generated blockwise i.i.d. p (u n ) dependent of Y (n). Hence, from [4, Lemma 0.6.2], the probability of the event E i that U (n) (i) is jotly typical with Y (n) is bounded by Pr(E i ) 2 Ln(I(U n ;Y n ) δ), for all i, where δ 0 as ɛ 0. Consequently, we have Pr(Ŵ W ) Pr(Ec ) + Pr(E i ) 2 kr i=2 ɛ + 2 kr 2 Ln(I(U n ;Y n ) δ) 2ɛ if L is sufficiently large and kr = (Ln 2 + n)r < Ln(I(U n ; Y n ) δ). Thus by lettg L and then ɛ 0, we can achieve any rate R < C FB,n. Fally by Lemma, this implies that we can achieve C FB,n = p(x n y n ) n I(Xn Y n ), which completes the proof of Theorem. X Z x x 2 x 3 x 4 x 5 x 6 x 8 x 9 x 0 x x 2 x 3 x 5 x 6 x 7 x 8 x 9 x 20 z z 2 z 3 z 4 z 5 z 6 z 7 z 8 z 9 z 0 z z 2 z 3 z 4 z 5 z 6 z 7 z 8 z 9 z 20 z 2 Y y 2 y 3 y 4 y 5 y 6 y 7 y 8 y 9 y 0 y y 2 y 3 y 4 y 5 y 6 y 7 y 8 y 9 y 20 y 2 Y i = g(x i i m, Zi i m ) U (n) X (n) Z (n) Ỹ (n) u u 2 u 3 u 4 u 5 u 6 u 7 u 8 u 9 u 0 u u 2 u 3 u 4 u 5 u 6 u 7 u 8 x x 2 x 3 x 4 x 5 x 6 x 7 x 8 x 9 x 0 x x 2 x 3 x 4 x 5 x 6 x 7 x 8 z z 2 z 3 z 4 z 5 z 6 z 7 z 8 z 9 z 0 z z 2 z 3 z 4 z 5 z 6 z 7 z 8 ỹ 2 ỹ 3 ỹ 5 ỹ 6 ỹ 8 ỹ 9 ỹ ỹ 2 ỹ 4 ỹ 5 ỹ 7 ỹ 8 U (i )n+ i.i.d. p (u n ) X (i )n+j = fj (U (i )n+j j, X (i )n+, Ỹ (i )n+j (i )n+ ) Ỹ(i )n+ = f( X (i )n+, Z (i )n+ ) = f (U(i )n+, Z (i )n+ ) Fig. 2. Code, put, noise, and output sequences: n = 3, L = 2, m =. 859

5 III. CONCLUDING REMARKS Tradg off generality for transparency, we have focused on stationary channels of the form Y n = f(x n n m, Z n n m) and presented a simple and constructive proof of the feedback codg theorem. The Shannon strategy (Lemma ) has a fundamental role transformg the feedback codg problem to a nonfeedback one, which is then solved by a scalable codg scheme of constructg a long typical put-output sequence pair by concatenatg shorter nonergodic ones with appropriate phase shifts. This two-stage approach can be applied to other channel models and can give a straightforward codg theorem. For example, we can show that the semi-determistic fite-state channel p(y n, s n s n, x n ) = p(y n s n, x n )p(s n s n, x n, y n ) with s n = f(s n, x n, y n ) for some determistic function f (but without the assumption of decomposability) has the feedback capacity lower bounded by C FB n p(x n y n ) m s 0 n I(Xn Y n s 0 ). This result was previously shown by Permuter et al. [5, Section V] via a generalization of Gallager s random codg exponent method for fite state channels without feedback [6, Section 5.9]. Here we sketch a simple alternative proof. From a trivial modification of Lemma, the problem reduces to showg that m p(u n ),x i=f(u i,x i,y i ) s 0 n I(U n ; Y n s 0 ) (4) is achievable for each n. But the given Shannon strategy (p (u n ), x n = f (u n, x n, y n )) duces a new timevariant fite-state channel on the n-letter er alphabet as p(y k, s k s k, u k ). Hence we can use Gallager s random codg exponent method directly to achieve lim k p(u k ) m s 0 k I(Uk ; Y k s 0 ), which can be shown to be no less than our target n I(U ; Y s 0 ), because of the determistic evolution of the state S n = f(s n, X n, Y n ). We fally mention an important question that is not dealt with this paper. Our characterization of the feedback capacity n p(x n y n ) n I(Xn Y n ) (5) or any similar multi-letter expressions are general not computable and do not provide much sight on the structure of the capacity achievg codg scheme. One may ask whether a stationary or even Markov distribution is asymptotically optimal for the sequence of imizations (5). This problem has been solved for a few specific channel models such as certa classes of fite-state channels [3], [20], [5], [4] and stationary additive Gaussian noise channels [8], [9], sometimes with analytic expressions for the feedback capacity. In this context, the current development is just the first step toward the complete characterization of the feedback capacity. ACKNOWLEDGMENT The author wishes to thank Tom Cover, Bob Gray, and Haim Permuter for helpful discussions. REFERENCES [] D. Blackwell, Information theory, Modern Mathematics for the Engeer: Second Series. New York: McGraw-Hill, 96, pp [2] G. Caire and S. Shamai, On the capacity of some channels with channel state formation, IEEE Trans. Inf. Theory, vol. IT-45, no. 6, pp , 999. [3] J. Chen and T. Berger, The capacity of fite-state Markov channels with feedback, IEEE Trans. Inf. Theory, vol. IT-5, no. 3, pp , Mar [4] T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. New York: Wiley, [5] A. Feste, On the codg theorem and its converse for fite-memory channels, Information and Control, vol. 2, pp , 959. [6] R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 968. [7] R. M. Gray, Entropy and Information Theory. New York: Sprger- Verlag, 990. [8], Feedback capacity of the first-order movg average Gaussian channel, IEEE Trans. Inf. Theory, vol. IT-52, no. 7, pp , [9], Feedback capacity of stationary Gaussian channels, submitted to IEEE Trans. Inf. Theory, February [Onle]. Available: [0] Y.-H. Kim, A codg theorem for a class of stationary channels with feedback, submitted to IEEE Trans. Inf. Theory, January [Onle]. Available: [] J. L. Massey, Causality, feedback, and directed formation, Proc. International Symposium on Information Theory and its Applications, Honolulu, Hawaii, Nov. 990, pp [2] J. Nedoma, Über die Ergodizität und r-ergodizität stationärer Wahrschelichkeitsmasse, Z. Wahrschelichkeitstheorie und Verw. Gebiete, vol. 2, pp , 963. [3] D. L. Neuhoff and P. C. Shields, Channels with almost fite memory, IEEE Trans. Inf. Theory, vol. IT-25, no. 4, pp , 979. [4] H. Permuter, P. Cuff, B. Van Roy, and T. Weissman, Capacity of the trapdoor channel with feedback, submitted to IEEE Trans. Inform. Theory, [Onle]. Available: [5] H. Permuter, T. Weissman, and A. Goldsmith, Fite-state channels with time-variant determistic feedback, submitted to IEEE Trans. Inform. Theory, [Onle]. Available: [6] C. E. Shannon, Channels with side formation at the transmitter, IBM J. Res. Develop., vol. 2, pp , 958. [7] S. Tatikonda, Control under communication constrats, Ph.D. Thesis, Massachusetts Institute of Technology, Sept [8] S. Tatikonda and S. Mitter, The capacity of channels with feedback, Sept [Onle]. Available: [9] S. Verdú and T. S. Han, A general formula for channel capacity, IEEE Trans. Inf. Theory, vol. IT-40, no. 4, pp , July 994. [20] S. Yang, A. Kavčić, and S. Tatikonda, Feedback capacity of fitestate mache channels, IEEE Trans. Inf. Theory, vol. IT-5, no. 3, pp , Mar