On the Feedback Capacity of Stationary Gaussian Channels

Transcription

1 On the Feedback Capacity of Stationary Gaussian Channels Young-Han Kim Information Systems Laboratory Stanford University Stanford, CA Abstract The capacity of stationary additive Gaussian noise channels with feedback is characterized as solution to a variational problem. Toward this end, it is proved that the optimal feedback coding scheme is stationary. When specialized to the first-order autoregressive moving-average noise spectrum, this variational characterization yields a closed-form expression for the feedback capacity. In particular, this result shows that the celebrated Schalkwijk Kailath Butman coding scheme achieves the feedback capacity for the first-order autoregressive moving-average Gaussian channel, resolving a long-standing open problem studied by Butman, Schalkwijk Tiernan, Wolfowitz, Ozarow, Ordentlich, Yang Kavčić Tatikonda, and others. Introduction and summary We consider the additive Gaussian noise channel Y i = X i + Z i, i =, 2,..., where the additive Gaussian noise process {Z i } i= is stationary with Z n = (Z,..., Z n ) N n (0, K Z,n ) for each n =, 2,.... We wish to communicate a message W {,..., 2 nr } over the channel Y n = X n + Z n. For block length n, we specify a (2 nr, n) feedback code with codewords X n (W, Y n ) = (X (W ), X 2 (W, Y ),..., X n (W, Y n )), W =,..., 2 nr, satisfying the average power constraint n n i= EX2 i (W, Y i ) P and decoding function Ŵ n : R n {,..., 2 nr }. The probability of error P e (n) is defined by P e (n) = Pr{Ŵn(Y n ) W }, where the message W is uniformly distributed over {, 2,..., 2 nr } and is independent of Z n. We say that the rate R is achievable if there exists a sequence of (2 nr, n) codes with P e (n) 0 as n. The feedback capacity C FB is defined as the supremum of all achievable rates. We also consider the case in which there is no feedback, corresponding to the codewords X n (W ) = (X (W ),..., X n (W )) independent of the previous channel outputs. We define the nonfeedback capacity C, or the capacity in short, in a manner similar to the feedback case. Shannon [] showed that the nonfeedback capacity is achieved by water-filling on the noise spectrum, which is arguably one of the most beautiful results in information theory. This research is supported in part by NSF Grant CCR

2 More specifically, the capacity C of the additive Gaussian noise channel Y i = X i + Z i, i =, 2,..., under the power constraint P, is given by C = 2 log max{s Z(e iθ ), λ} S Z (e iθ ) dθ () where S Z (e iθ ) is the power spectral density of the stationary noise process {Z i } i= and the water-level λ is chosen to satisfy P = max{0, λ S Z (e iθ )} dθ. (2) Although () and (2) give only a parametric characterization of the capacity C(λ) under the power constraint P (λ) for each parameter λ 0, this solution is considered to be simple and elegant enough to be called closed-form. For the case of feedback, no such elegant solution exists. Most notably, Cover and Pombra [2] characterized the n-block feedback capacity C FB,n for arbitrary time-varying Gaussian channels via the asymptotic equipartition property (AEP) for arbitrary nonstationary nonergodic Gaussian processes as C FB,n = max K V,n,B n 2 log det(k V,n + (B n + I)K Z,n (B n + I) ) /n (3) det(k Z,n ) /n where the maximum is taken over all positive semidefinite matrices K V,n and all strictly lower triangular matrices B n of sizes n n satisfying tr(k V,n + B n K Z,n (B n ) ) np. Note that we can also recover the nonfeedback case by taking B n 0. When specialized to the stationary noise processes, the Cover Pombra characterization gives the feedback capacity as a limiting expression C FB = lim n C FB,n = lim max n K V,n,B n 2 log det(k V,n + (B n + I)K Z,n (B n + I) ) /n. (4) det(k Z,n ) /n Despite its generality, the Cover Pombra formulation of the feedback capacity falls short of what we can call a closed-form solution. It is very difficult, if not impossible, to obtain an analytic expression for the optimal (K V,n, B n) in (3) for each n. Furthermore, the sequence of optimal {K V,n, B n} n= is not necessarily consistent, that is, (K V,n, B n) is not necessarily a subblock of (K V,n+, B n+). Hence the characterization (3) in itself does not give much hint on the structure of optimal {K V,n, B n} n= achieving C FB,n, or more importantly, its limiting behavior. In this paper, we make one step forward by proving Theorem. The feedback capacity C FB of the Gaussian channel Y i = X i + Z i, i =, 2,..., under the power constraint P, is given by C FB = sup S V (e iθ ),B(e iθ ) 2 log S V (e iθ ) + + B(e iθ ) 2 S Z (e iθ ) S Z (e iθ ) where S Z (e iθ ) is the power spectral density of the noise process {Z i } i= and the supremum is taken over all power spectral densities S V (e iθ ) 0 and strictly causal filters B(e iθ ) = j= b je ijθ π satisfying the power constraint (S V (e iθ ) + B(e iθ ) 2 S Z (e iθ )) dθ P. dθ 5

3 Roughly speaking, this characterization shows the asymptotic optimality of a stationary solution (K V,n, B n ) in (3) and hence it can be viewed as justification for interchange of the order of limit and maximum in (4). Since Theorem gives a variational form of the feedback capacity, it remains to characterize the optimal (S V (eiθ ), B (e iθ )). In this paper, we provide a sufficient condition for the optimal solution using elementary arguments. This result, when specialized to the first-order autoregressive (AR) noise spectrum S Z (e iθ ) = / + βe iθ 2, < β <, yields a closed-form solution for feedback capacity as C FB = log x 0, where x 0 is the unique positive root of the fourth-order polynomial P x 2 = ( x2 ) ( + β x) 2. This result positively answers the long-standing conjecture by Butman [3, 4], Tiernan Schalkwijk [5, 6], and Wolfowitz [7]. In fact, we will obtain the feedback capacity formula for the first-order autoregressive moving average (ARMA) noise spectrum, generalizing the result in [8] and confirming a recent conjecture by Yang, Kavčić, and Tatikonda [9]. The rest of the paper is organized as follows. We prove Theorem in the next section. In Section 3, we derive a sufficient condition for the optimal (S V (eiθ ), B (e iθ )) and apply this result to the first-order ARMA noise spectrum to obtain the closed-form feedback capacity. We also show that the Schalkwijk Kailath Butman coding scheme [0,, 3] achieves the feedback capacity of the first-order ARMA Gaussian channel. 2 Proof of Theorem We start from the Cover-Pombra formulation of the n-block feedback capacity C FB,n in (3). Tracing the development of Cover and Pombra backwards, we express C FB,n as C FB,n = max h(y n ) h(z n ) = max I(V n ; Y n ) V n +B n Z n V n +B n Z n where the maximization is over all X n of the form X n = V n + B n Z n, resulting in Y n = V n + (I + B n )Z n, with strictly lower-triangular B n and multivariate Gaussian V n, independent of Z n, satisfying the power constraint E n i= X2 i np. Define C FB = sup S V (e iθ ),B(e iθ ) 2 log S V (e iθ ) + + B(e iθ ) 2 S Z (e iθ ) dθ S Z (e iθ ) where S Z (e iθ ) is the power spectral density of the noise process {Z i } i= and the supremum is taken over all power spectral densities S V (e iθ ) 0 and strictly causal filters B(e iθ ) = k= b ke ikθ satisfying the power constraint (S V (e iθ ) + B(e iθ ) 2 S Z (e iθ )) dθ P. In the light of Szegő Kolmogorov Krein theorem, we can express C FB also as C FB = sup h(y) h(z) {X i } where the supremum is taken over all stationary Gaussian processes {X i } i= of the form X i = V i + k= b kz i k where {V i } i= is stationary and independent of {Z i } i= such that EX0 2 P. We will prove that C FB = C FB. 6

4 We first show that C FB,n C FB for all n. Fix n and let (KV,n, B n) achieve C FB,n. Consider a process {V i } i= that is independent of {Z i } i= and blockwise i.i.d. with V (k+)n kn+ N n (0, KV,n ), k = 0, ±, ±2,.... Define a process {X i} i= as X (k+)n kn+ = V (k+)n kn+ + BnZ (k+)n kn+ for all k. Similarly, let Y i = X i + Z i, < i <, be the corresponding output process through the stationary Gaussian channel. Note that Y (k+)n kn+ = V (k+)n kn+ +(I+Bn)Z (k+)n kn+ for all k. For each t = 0,,..., n, define a process {V i (t)} i= as V i (t) = V t+i for all i and similarly define {X i (t)} i=, {Y i (t)} i=, and {Z i (t)} i=. Note that Y i (t) = X i (t) + Z i (t) for all i and all t = 0,,..., n, but X n (t) is not equal to V n (t) + BnZ n (t) in general. From the independence of V n and Vn+, 2n we can easily check that 2C FB,n = I(V n ; Y n ) + I(Vn+; 2n Yn+) 2n = h(v n ) + h(vn+) 2n h(v n Y n ) h(vn+ Y 2n n+) 2n h(v 2n ) h(v 2n Y 2n ) = I(V 2n ; Y 2n ) = h(y 2n ) h(z 2n ). By repeating the same argument, we get C FB,n kn (h(v ) h(z kn )), kn for all k. Hence, for all m =, 2,..., and each t = 0,..., n, we have C FB,n ( h(y m m (t)) h(z m (t)) ) + ɛ m = ( h(y m m (t)) h(z m ) ) + ɛ m where ɛ m absorbs the edge effect and vanishes uniformly in t as m. Now we introduce a random variable T uniform on {0,,..., n } and independent of {V i, X i, Y i, Z i } i=. It is easy to check the followings: (I) {V i (T ), X i (T ), Y i (T ), Z i (T )} i= is stationary with Y i (T ) = X i (T ) + Z i (T ). (II) {X i (T )} i= satisfies the power constraint EX 2 0(T ) = E[E(X 2 0(T ) T )] = n tr(k V,nB nk Z,n (B n) ) P. (III) {V i (T )} i= and {Z i (T )} i= are orthogonal; i.e., EV i (T )Z j (T ) = 0 for all i, j. (IV) Although there is no linear relationship between {X i (T )} and {Z i (T )}, {X i (T )} still depends on {Z i (T )} in a strictly causal manner. More precisely, for all i j, E(X i (T )Z j (T ) Z (T i )) = E ( E(X i (T )Z j (T ) Z (T i ), T ) Z (T i ) ) = E ( E(X i (T ) Z (T i ), T )E(Z j (T ) Z (T i ), T ) Z (T i ) ) = E ( E(X i (T ) Z (T i ), T )E(Z j (T ) Z (T i )) Z (T i ) ) and for all i, = E(X i (T ) Z i (T ))E(Z j (T ) Z i (T )), Var(X i (T ) V i (T ) Z i (T )) = E ( Var(X i (T ) V i (T ) Z i (T ), T ) Z i (T ) ) = 0. 7

5 Finally, define {Ṽi, X i, Ỹi, Z i } i= to be a jointly Gaussian stationary process with the same mean and autocorrelation as {V i (T ), X i (T ), Y i (T ), Z i (T )} i=. It is easy to check that {Ṽi, X i, Ỹi, Z i } also satisfies the properties (I) (IV) and hence that {Ṽi} and { Z i } are independent. It follows from these properties and the Gaussianity of {Ṽi, X i, Ỹi, Z i } that there exists a sequence {b k } k= such that X i = Ṽi + k= b Z k i k. Thus we have C FB,n ( h(y m m (T ) T ) h(z m ) ) + ɛ m ( h(y m m (T )) h(z m ) ) + ɛ m ( h( Ỹ m ) h(z m ) ) + ɛ m. m By letting m and using the definition of C FB, we obtain C FB,n h(ỹ) h(z) C FB. (5) For the other direction of the inequality, we use the notation C FB (P ) and C FB,n (P ) to stress the dependence on the power constraint P. Given ɛ > 0, let {X i = V i + k= b kz i k } i= achieve C FB (P ) ɛ under the power constraint P. The corresponding channel output is given as Y i = V i + Z i + b k Z i k (6) for all i = 0, ±, ±2,.... Now, for each m =, 2,..., we define a single-sided nonstationary process {X i (m)} i= in the following way: { Ui + V X i (m) = i + i k= k= b kz i k i m, U i + V i + m k= b kz i k i > m where U, U 2,... are i.i.d. N(0, ɛ). Thus, X i (m) depends causally on Z i for all i and m. Let {Y i (m)} i= be the corresponding channel output Y i (m) = X i (m) + Z i, i =, 2,..., for each m =, 2,.... We can show that there exists an m so that lim n n n EXi 2 (m ) P + 2ɛ i= and lim n n h(y n (m )) h(y) ɛ (7) where h(y) is the entropy rate of the stationary process defined in (6). Consequently, for n sufficiently large, n n EXi 2 (m ) n(p + 3ɛ) and n (h(y n (m )) h(z n )) C FB (P ) 2ɛ. Therefore, we can conclude that i= C FB,n (P + 3ɛ) C FB (P ) 2ɛ for n sufficiently large. Finally, using continuity of C FB (P ) = lim n C FB,n (P ) in P, we let ɛ 0 to get C FB (P ) C FB, which, combined with (5), implies that C FB (P ) = C FB (P ). 8

6 3 Example: First-order ARMA noise spectrum With the ultimate goal of an explicit characterization of C FB as a function of S Z and P, we wish to solve the optimization problem maximize subject to log( S V (e iθ ) + + B(e iθ ) 2 S Z (e iθ ) ) dθ B(e iθ ) strictly causal S V (e iθ ) 0 S V (e iθ ) + B(e iθ ) 2 S Z (e iθ ) dθ P. (8) Suppose that S Z (e iθ ) is bounded away from zero. Then, under the change of variable we rewrite (8) as S Y (e iθ ) = S V (e iθ ) + + B(e iθ ) 2 S Z (e iθ ), maximize subject to log S Y (e iθ ) dθ B(e iθ ) strictly causal S Y (e iθ ) + B(e iθ ) 2 S Z (e iθ ) S Y (e iθ ) (B(e iθ ) + B(e iθ ) + )S Z (e iθ ) dθ P. (9) Take any ν > 0, φ(e iθ ), ψ L, and ψ 2 (e iθ ), ψ 3 (e iθ ) L such that φ(e iθ ) > 0, log φ(e iθ ) L, ψ (e iθ ) = ν φ(e iθ ) 0, [ ] ψ (e iθ ) ψ 2 (e iθ ) 0 ψ 2 (e iθ ) ψ 3 (e iθ ) and A(e iθ ) := ψ 2 (ω) + νs Z (e iθ ) L is anticausal. Since any feasible B(e iθ ) and S Y (e iθ ) satisfy [ ] SY (e iθ ) + B(e iθ ) 0, + B(e iθ ) S Z (eiθ ) we have ([ SY ] [ + B ψ ]) ψ 2 tr + B S Z ψ 2 ψ 3 = φ S Y + ψ 2 ( + B) + ψ 2 ( + B) + ψ 3 S Z 0. From the fact that log x x for all x 0, we get the inequality log S Y log φ + φs Y = log φ + νs Y ψ S Y log φ + νs Y + ψ 2 ( + B) + ψ 2 ( + B) + ψ 3 S Z. (0) Further, since A L is anticausal and B H is strictly causal, AB L is strictly anticausal and A(e iθ )B(e iθ ) dθ = A(e iθ )B(e iθ ) dθ = 0. () 9

7 By integrating both sides of (0), we get log S Y = = log φ + νs Y + ψ 2 ( + B) + ψ 2 ( + B) + ψ 3 S Z log φ + ν ( (B+B+)S Z + P ) + ψ 2 ( + B) + ψ 2 ( + B) + ψ 3 S Z log φ + ψ 2 + ψ 2 + ψ 3 S Z + ν(s Z + P ) + AB + AB log φ + ψ 2 + ψ 2 + ψ 3 S Z + ν(s Z + P ) (2) where the second inequality follows from the power constraint in (9) and the last equality follows from (). Checking the equality conditions in (2), we find the following sufficient condition for the optimality of a specific (S Y (e iθ ), B(e iθ )). Lemma. Suppose S Z (e iθ ) is bounded away from zero. Suppose B(e iθ ) H is strictly causal with B(e iθ ) 2 S Z (e iθ ) dθ = P. (3) If there exists λ > 0 such that and that λ ess inf + B(e iθ ) 2 S Z (e iθ ) θ [,π) λ + B(e iθ ) B(eiθ )S Z (e iθ ) L is anticausal, then B(e iθ ) along with S V (e iθ ) 0 attains the feedback capacity. Now we turn our attention to the first-order autoregressive moving average noise spectrum S Z (z), defined by S Z (e iθ ) = + αe iθ 2 + βe iθ, α [, ], β (, ). (4) This spectral density corresponds to the stationary noise process defined by Z i +βz i = U i +αu i, where {U i } i= is a white Gaussian process with zero mean and unit variance. We find the feedback capacity of the first-order ARMA Gaussian channel in the following. Theorem 2. Suppose the noise process {Z i } i= has the power spectral density S Z (z) defined in (4). Then, the feedback capacity C FB of the Gaussian channel Y i = X i + Z i, i =, 2,..., under the power constraint P is given by C FB = log x 0 where x 0 is the unique positive root of the fourth-order polynomial and P x 2 = ( x2 )( + σαx) 2 ( + σβx) 2 (5) σ = sgn(β α) = {, β α,, β < α. 20

8 Proof sketch. Without loss of generality, we assume that α <. The case α = can be handled by a simple perturbation argument. When α <, S Z (e iθ ) is bounded away from zero, so that we can apply Lemma. Here is the bare-bone summary of the proof: We will take the feedback filter of the form B(z) = + βz + αz yz (6) σxz where x (0, ) to be an arbitrary parameter corresponding to each power constraint P (0, ) under the the choice of y = x2 +σαx. Then, we can show that B(z) satisfies σx +σβx the sufficient condition in Lemma under the power constraint P = B(e iθ ) 2 S Z (e iθ ) dθ = y 2 dθ xe iθ 2 = y2 x 2. The rest of the proof is the actual implementation of this idea. We skip the details. Although the variational formulation of the feedback capacity (Theorem ), along with the sufficient condition for the optimal solution (Lemma ), leads to the simple closed-form expression for the ARMA() feedback capacity (Theorem 2), one might be still left with somewhat uneasy feeling, due mostly to the algebraic and indirect nature of the proof. Now we take a more constructive approach and interpret the properties of the optimal feedback filter B. Consider the following coding scheme. Let V N(0, ). Over the channel Y i = X i + Z i, i =, 2,..., the transmitter initially sends X = V and subsequently refines the receiver s knowledge by sending X n = (σx) (n ) (V ˆV n ) (7) where x is the unique positive root of (5) and ˆV n = E(V Y,..., Y n ) is the minimum mean-squared error estimate of V given the channel output up to time n. We will show that lim inf n n I(V ; ˆV n ) ( ) 2 log x 2 while lim sup n n n Xi 2 P, which proves that the proposed coding scheme achieves the feedback capacity. Define, for n 2, i= Y n = d n V + U n + ( α) n (αu 0 βz 0 ) where d n = ( ) + σβx ( ( σαx) n )(σx) (n ). + σαx Then one can show that Y n can be represented as a linear combination of Y,..., Y n and hence that ( ( n 2 E(V ˆV n ) 2 E V d k Y k)). k=2 2

9 Furthermore, we can check that whence or equivalently, lim sup n E(V ( n k=2 d ky k ))2 E(V ( n k=2 d ky k ))2 x, 2 lim inf n On the other hand, for n 2, which converges to ( ) n log E(V ˆV n ) 2 log x 2 n I(V ; ˆV n ) ( ) 2 log. x 2 EX 2 n = x 2(n ) E(V ˆV n ) 2 x 2(n ) E x 2(n ) lim n n k=2 d2 k = ( V ( n k=2 ( + σαx)2 ( + σβx) 2 (x 2 ) = P. d k Y k)) 2, Hence, we have shown that the simple linear coding scheme (7) achieves the ARMA() feedback capacity. The coding scheme described above uses the minimum mean-square error decoding of the message V, or equivalently, the joint typicality decoding of the Gaussian random codeword V, based on the general asymptotic equipartition property of Gaussian processes shown by Cover and Pombra [2, Theorem 5]. Instead of the Gaussian codebook V, the transmitter initially sends a real number θ which is chosen from some equally spaced signal constellation Θ, say, Θ = {, +δ,..., δ, }, δ = 2/(2 nr ), and subsequently corrects the receiver s estimation error by sending θ ˆθ n (up to appropriate scaling as before) at time n, where ˆθ n is the minimum variance unbiased linear estimate of θ given Y n. Now we can verify that the optimal maximum-likelihood decoding is equivalent to finding θ Θ that is closest to ˆθ n, which results in the error probability P (n) e ( ) erfc c0 x 2n /2 2nR where erfc(x) = 2 π x exp( t2 )dt is the complementary error function and c 0 is a constant independent of n. This proves that the Schalkwijk Kailath Butman coding scheme achieves C FB = log x with doubly exponentially decaying error probability. 4 Concluding remarks Although it is still short of what we can call a closed-form solution in general, our variational characterization of Gaussian feedback capacity gives an exact analytic answer for a certain class of channels, as demonstrated in the example of the first-order ARMA Gaussian channel. Our development can be further extended in two directions. First, one can investigate properties of the optimal solution (S V, B ). Without much surprise, one can show that feedback increases the capacity if and only if the noise spectrum is white. 22

10 Furthermore, it can be shown that taking SV 0 does not incur any loss in maximizing the output entropy, resulting in a simpler maximin characterization of feedback capacity: ( π C FB = sup inf {b k } {a k } 2 log 2 ) 2 a k b k SZ (e iθ ) dθ k= k= where the supremum is taken over all {b k } satisfying k= b k 2 S Z (e iθ ) dθ P. Secondly, one can focus on the finite-order ARMA noise spectrum and show that the k-dimensional generalization of Schalkwijk Kailath Butman coding scheme is optimal for the ARMA spectrum of order k. This confirms many conjectures based on numerical evidences, including the recent study by Yang, Kavčić, and Tatikonda [9]. These results will be reported separately in [2]. References [] C. E. Shannon, Communication in the Presence of Noise, Proc. IRE, vol. 37, pp. 0 2, 949. [2] T. M. Cover and S. Pombra, Gaussian feedback capacity, IEEE Trans. Info. Theory, vol. 35, pp , January 989. [3] S. A. Butman, A general formulation of linear feedback communication systems with solutions, IEEE Trans. Info. Theory, vol. 5, pp , May 969. [4] S. A. Butman, Linear feedback rate bounds for regressive channels, IEEE Trans. Info. Theory, vol. 22, pp , May 976. [5] J. C. Tiernan and J. P. M. Schalkwijk, An upper bound to the capacity of the bandlimited Gaussian autoregressive channel with noiseless feedback, IEEE Trans. Info. Theory, vol. 20, pp. 3 36, May 974. [6] J. C. Tiernan, Analysis of the optimum linear system for the autoregressive forward channel with noiseless feedback, IEEE Trans. Info. Theory, vol. 22, pp , May 976. [7] J. Wolfowitz, Signalling over a Gaussian channel with feedback and autoregressive noise, J. Appl. Probab., vol. 2, pp , 975. [8] Y.-H. Kim, Feedback capacity of the first-order moving average Gaussian channel, submitted to IEEE Trans. Info. Theory, November [9] S. Yang, A. Kavčić, and S. Tatikonda, Linear Gaussian channels: feedback capacity under power constraints, in Proc. IEEE Int. Symp. Info. Theory, p. 72, [0] J. P. M. Schalkwijk and T. Kailath, A coding scheme for additive noise channels with feedback I: No bandwidth constraint, IEEE Trans. Info. Theory, vol. 2, pp , April 966. [] J. P. M. Schalkwijk, A coding scheme for additive noise channels with feedback II: Band-limited signals, IEEE Trans. Info. Theory, vol. 2, pp , April 966. [2] Y.-H. Kim, Feedback capacity of stationary Gaussian channels, in preparation. 23