Recursions for the Trapdoor Channel and an Upper Bound on its Capacity

Transcription

1 4 IEEE International Symposium on Information heory Recursions for the rapdoor Channel an Upper Bound on its Capacity obias Lutz Institute for Communications Engineering echnische Universität München Munich, 89 Germany Abstract he problem of maximizing the n-letter mutual information of the trapdoor channel is considered. It is shown that log 5.66 bits per use is an upper bound on the capacity of the trapdoor channel. his upper bound, which is the tightest upper bound known, proves that feedback increases the capacity. I. INRODUCION AND CHANNEL MODEL he trapdoor channel was introduced by David Blackwell in 96 is used by Robert Ash both as a book cover as an introductory example for channels with memory. he mapping of channel inputs to channel outputs can be described as follows. Consider a box that contains a ball that is labeled s {,}, where the index refers to time. Both the sender the receiver know the initial ball. In time slot, the sender places a new ball labeled x {,} in the box. In the same time slot, the receiver chooses one of the two ballss or x at rom while the other ball remains in the box. he chosen ball is interpreted as channel output y at time t while the remaining ball becomes the channel state s. he same procedure is applied in every future channel use. In time slot, for instance, the sender places a new ball x {,} in the box the corresponding channel outputy is eitherx or s. he transmission process is visualized in Fig.. Fig. a shows the trapdoor channel at time t when the sender places ball x t in the box. In the same time slot, the receiver chooses romly one of the two balls x t or s t as channel output, in the figure s t. Consequently, the upcoming channel state s t becomes x t see Fig. b. At time t+ the sender places a new ball x t+ in the box the receiver draws y t+ from s t x t+. able I depicts the probability of an output y t given an input x t state s t. ABLE I RANSIION PROBABILIIES OF HE RAPDOOR CHANNEL x t s t Py t x t,s t Py t x t,s t Despite the simplicity of the trapdoor channel, deriving its capacity seems challenging is an open problem. One feature that makes the problem cumbersome is that the distribution of the output symbols may depend on events s t x t+3 x t+ x t+ y t y t y t 3 x t a he trapdoor channel at time t. s t x t x t+4 x t+3 x t+ y t y t y t x t+ b he trapdoor channel at time t+. Here y t s t. Fig.. he trapdoor channel. happening arbitrarily far back in the past since each ball has a positive probability to remain in the channel over any finite number of channel uses. Instead of maximizing IX;Y one rather has to consider the multi-letter mutual information, i.e., limsup n IX n ;Y n. Let P n s denote the matrix of conditional probabilities of output sequences of length n given input sequences of length n where the initial state equals s. he following ordering of the entries of P n s is assumed. Row indices represent input sequences column indices represent output sequences. he i,jth entry of P n s, indicated as P n s i,j, is the conditional probability of the binary output sequence corresponding to the integer j given the binary input sequence corresponding the the integer i, i,j n. For instance, if n 3, then P 3 s denotes the conditional 5,3 probability that the channel input x,x,x 3,, will be mapped to the channel output y,y,y 3,,. Kobayashi Morita showed 3 that P n s, s {,}, satisfies the recursion laws Pn P n+ P n P n P n+ P n P n P n where the initial matrices are given by P P. Ahlswede Kaspi 4 derived the zero-error capacity of the trapdoor channel which equals.5 b/u. Permuter et al. 5 considered the trapdoor channel under the additional assumption of having a unit delay feedback link available from the receiver /4/$3. 4 IEEE 94

2 4 IEEE International Symposium on Information heory to the sender. hey established that the feedback capacity of the trapdoor channel is equal to the logarithm of the golden ratio. In this paper, we consider the problem of maximizing the n- letter mutual information of the trapdoor channel for any n N. We relax the problem by permitting distributions that are not probability distributions. he resulting optimization problem is convex but the feasible set is larger than the probability simplex. Using the method of Lagrange multipliers via a theorem presented in, we show that log 5.66 b/u is an upper bound on the capacity of the trapdoor channel. Specifically, the same absolute maximum log 5.66 b/u results for all trapdoor channels which process input blocks of even length n. And the sequence of absolute maxima corresponding to trapdoor channels which process inputs of odd lengths converges to log 5 b/u from below as the block length increases. Unfortunately, the absolute maxima of our relaxed optimization are attained outside the probability simplex. Otherwise we would have established the capacity. Nevertheless, log 5.66 b/u is, to the best of our knowledge, the tightest capacity upper bound. Moreover, this bound is less than the feedback capacity of the trapdoor channel. he notation used in this paper is as follows. he symbols N N refer to the natural numbers with without, respectively. he input corresponding to theith row ofp n s is denoted as x i. Further, I n denotes the n n identity matrix, Ĩ n is a n n matrix whose secondary diagonal entries are all equal towhile the remaining entries are all equal to, n denotes a column vector of length n consisting only of ones. he vector n is the transpose of n. he functions exp log indicate the exponential function to base the logarithm to base. If applied to a vector/matrix, log or exp of each element is taken a vector/matrix results. Finally, the symbol refers to the Hadarmard product, i.e., the entry wise product of two matrices. II. A LAGRANGE MULIPLIER APPROACH O HE A. Problem Formulation RAPDOOR CHANNEL We derive an upper bound on the capacity of the trapdoor channel. Specifically, for any n N, we find a solution to the optimization problem maximize P X n n subject to n IXn ;Y n s n n i j n i n k p i Pn s i,j log Pn s n k p k i,j Pn s k,j 3 p i 4 p k Pn s k,j j n. 5 Note that the pmf P X n represents the n -sequences p,...,p n where p i denotes the probability of the ith input sequence x i, i.e., the binary sequence corresponding to the integeri. Constraint 5 guarantees that the argument of the logarithm does not become negative. he feasible set, ined by 4 5, is convex. It includes the set of probability mass functions, but might be larger. o see this note that 5 is a weighted sum of all p k where each weight P n s k,j is non negative. Clearly, 4 5 are satisfied by probability distributions. However, there might exist distributions which involve negative values sum up to one but still satisfy 5. Moreover, the objective function n IX n ;Y n s is concave on the set of probability distributions, which follows by using the same arguments that show that mutual information is concave on the set of input probability distributions. Consequently, the optimization problem is convex every solution maximizes n IX n ;Y n s. In the following, we use the terminology max P X n n IX n ;Y n s. aking the limit of the sequence as n grows, one n N obtains either the capacity of the trapdoor channel or an upper bound on the capacity, depending on whether the limit is attained inside or outside the set of probability distributions. Since it does not matter whether the optimization is with respect to initial state or due to symmetry reasons, we do not have to distinguish between lower capacity upper capacity 6, Chapter 4.6 B. Using a Result from the Literature he reason for considering 5 not the more natural constraints p k for all k is that a closed form solution can be obtained by applying the method of Lagrange multipliers to 3 4. As a byproduct, 5 will be automatically satisfied. In particular, setting the partial derivatives of n IXn ;Y n s +λ n i p i 6 with respect to each of the p i equal to zero results in a closed form solution of the considered optimization problem. his was done in, heorem for general discrete memoryless channels which are square non singular. Note that P n s is square non singular see, for instance, Lemma II. b. Moreover, we assume that the channel P n s is memoryless by repeatedly using it over a large number of input blocks of length n. Consequently, might be an upper bound on the capacity of the channel P n s. he reason is that some input blocks possibly drive the channel P n s into the opposite state s, i.e., the upcoming input block would see the channel P n s whose is equal to of P n s by symmetry but not P n s. However, by assuming that the channel does not change over time, the sender always knows the channel state before a new block is transmitted. Hence, might be an upper bound even though it is attained on the set of probability distributions. Nevertheless, this issue 95

3 4 IEEE International Symposium on Information heory can be ignored if n goes to infinity because in the asymptotic regime the channel P n s is used only once. Indeed we are interested in the asymptotic regime since the limit of the sequence is also its supremum see heorem II.7 n N Remark II.9. In summary, it is valid to apply, heorem which yields n n log exp n P j,i H Yn X n x i attained at j where d k is given by n P j,k exp j i 7 p k C n dk, k,..., n 8 n i P j,i H Yn X n x i. 9 Clearly, p,...,p n is a probability distribution only if d k. Observe that the Lagrangian 6 does not involve the constraint 5. However, the proof of, heorem shows that n k p k equals Pn s k,j M exp λ P H Y n X n x i i j,i for all j n. Hence, 5 is satisfied. For computational reasons we write 7 in matrix vector notation, which reads n exp P Pn s log P n s n n log In the remainder, we use instead of 7 we find exact numerical expressions for in heorem II.7 below. C. Useful Recursions In this section, we derive recursions for P n s log P n s n P n s Pn s log P n s n, as stated in Lemma II.4 Lemma II.5. he recursions, interesting by themselves, are needed to prove the main result. Both expressions are ined next. Definition II.. a he conditional entropy vector h n s of P n s, s {,}, is h n s HY n X n x... HY n X n x n P n s log P n s n 3 where n N. b he weighted conditional entropy vector ω n s of P n s, s {,}, is where n N. ω n s P h n s 4 P Pn s log P n s n 5 We remark that h n s ω n s are column vectors with n entries. he following two lemmas provide tools that we need for the proof of Lemma II.4 Lemma II.5. Lemma II.. a he trapdoor channel matrices P n+ P n+, n N, satisfy the following recursions: P P n+ P n P 4 P n 4 P P 6 P n+ P n 4 P n 4 P P n 4 P n 4 P P 4 P n 4 P P n P. 7 P n b Let M P P n P M P n P P n. he inverses of P n+ P n+, n N, satisfy the following recursions: P P n+ M P P P M P n P 3M M 4P 8 P n+ 4P n M 3M M P P P n P n P n M P n n. 9 Proof. a: Substituting P n+ P n+ into P n+ P n+, where the four matrices are expressed as in, yields 6 7. b: wo versions of the matrix inversion lemma are 7 A A C D D CA D A B A A BD D D. Now divide 6 7 into four blocks of equal size. A twofold application of, first to P n+ P n+, subsequently, to each of the blocks of P n+ P n+ yields 8 9. A transformation relating P n to P n, P n to h n ω n to ω n is derived next. to P n, h n Lemma II.3. Let P n P n be trapdoor channel matrices, n N. We have the following identities. a P n ĨnP n Ĩ n P n ĨnP n Ĩ n. 3 96

4 4 IEEE International Symposium on Information heory b c d P n ĨnP 4 P n ĨnP 5 h n Ĩnh n 6 h n Ĩnh n. 7 ω n Ĩnω n 8 ω n Ĩnω n. 9 e he row sums of P n P n are. Proof. See 8. We can now state the recursive laws for the conditional entropy vector the weighted conditional entropy vector. Lemma II.4. For n, h n+ satisfies the recursion h n+ h h + Ĩnh + n 3 4 h + 4Ĩnh + 3 n 4 h + 3 4Ĩnh + 3 n he initial value for n is given by h. Proof. See Lemma II.5. a For n N, ω satisfies the recursion ω ω n+ ω n ω n 3 ω with initial value ω. b For n N, ω n+ satisfies the recursion ω n ω n+ Ĩ n ω n ω n n 3 Ĩ n ω n n with initial value ω. Proof. a: he proof is by induction. he case n can be verified using Definition II. b with P P. Now assume that 3 holds for some n. In order to show 3 for n+, we evaluate ω n+ using 5 replacing P n+ h n+ with 8 3. he details of the simplification steps can be found in 8. b: Recall the recursions Pn+ P n+ P n+ P 33 n+ P n+ P n+ P n+ P n+ P n+ P n+, 34 which follow from. Computing the first half i.e., the first n+ entries ofω n+, indicated asω n+, based on Definition II.b using yields ω n+ P n+ Pn+ log P n+ n+. 35 By inition, the right h side of 35 is ω n+. Hence, under consideration of 3, we have ω ω n+. 36 ω n It remains to expressω in 36 in terms ofω n. By the same argument as just used, the first half of the vector ω equals ω n. Since ω is a palindrome by assumption, the second half of ω equals Ĩn ω n. Hence, ω ω n Ĩ n ω n. 37 By replacing ω in 36 with 37, we obtain 3. he initial value ω follows from 36 for n noting that ω. Remark II.6. he recursions derived in Lemma II.4 II.5 are with respect to initial state s. hey can be transformed to recursions with respect to initial state s by using 6 8 from Lemma II.3. D. Proof of the Main Result In this section, we evaluate based on Lemma II.5. heorem II.7. Consider the convex optimization problem 3 to 5. he absolute maximum for input blocks of even length n is C n 5 log b/u for all n N. 38 For input blocks of odd length n, the absolute maximum is C n 5 5 log n +n log 4 b/u 39 where n N. Proof. Without loss of generality, the initial state is assumed to be s. Recall, which for input blocks of lengthn+k reads C n+k n+k log n+k exp ωn+k b/u 4 where n N k,. For n, a straightforward computation shows, using 3 3, that C log 5 4 b/u C log 5 b/u. Now assume that hold for some n. In particular, suppose that nexp ω 5 n 4 A palindrome is a finite sequence of symbols, numbers or elements, which reads the same backwards as forward. 97

5 4 IEEE International Symposium on Information heory n exp ωn 5 4 n 5. 4 We now show that hold if n is replaced by n+. By means of the recursions derived in Lemma II.5, we have n+ exp ωn+ n exp ω +exp ω n + nexp ω 43 n+ exp ωn+ n exp ωn +exp ωn n 44 + n exp ωn. 45 Observe that we used the property n exp Ĩn ω n n exp ωn in 44. Finally, using 4 under consideration of the induction hypotheses 4 4, we obtain C n+ n+ log + nexp ω 5 log b/u C n+ n+ log + n exp ωn 5 5 log n+ +n log 4 b/u. Remark II.8. Observe that lim n C n+ log 5, where convergence is from below. Hence, we have max n N C n 5 log b/u. Unfortunately, the distributions corresponding to involve negative probabilities otherwise the capacity of the trapdoor channel would have been established. We elaborate this issue in the following remark. Remark II.9. Note that condition 9 does not hold for all k,..., n. his can be verified as follows. For a trapdoor channel P n, condition 9 reads in matrix vector notation as dk P k n exp ω n n. 46 We now compute the second last row of Pn by the following recursive scheme. Applying the matrix inversion lemma in the form of to P n, which is written as in, subsequently taking the transpose, then replacing the right bottom block of this matrix, which is Pn, with the just obtained matrix times two where n is replaced by n, then applying the same procedure to the right bottom block of Pn so on until the right bottom block equals n P shows that the second last row of Pn equals n n. Further, using Lemma II.5, it follows that the second to last entry the last entry of ω n equals if n N is even, 4 if n N is odd. Inserting the gathered quantities into 46 yields { 3 n 3 if n is even d n 3 n 5 if n is odd. Hence, condition 9 does not hold for all k,..., n. III. CONCLUSIONS We have focused on the convex optimization problem 3 to 5 where the feasible set is larger than the probability simplex. An absolute maximum of the n-letter mutual information was established for any n N by using the method of Lagrange multipliers. he same absolute maximum log 5.66 b/u results for all even n the sequence of absolute maxima corresponding to odd block lengths converges from below to log 5 b/u as the block length increases. Unfortunately, all absolute maxima are attained outside the probability simplex. Hence, instead of establishing the capacity of the trapdoor channel, we have shown only that log 5 b/u is an upper bound on the capacity. o the best of our knowledge, this upper bound is the tightest known bound. Notably, this upper bound is strictly smaller than the feedback capacity 5. Moreover, the result gives an indirect justification that the capacity of the trapdoor channel is attained on the boundary of the probability simplex. ACKNOWLEDGMEN he author is supported by the German Ministry of Education Research in the framework of the Alexer von Humboldt-Professorship would like to thank Prof. Haim Permuter who suggested to use, heorem Moreover, the author wishes to thank Prof. Gerhard Kramer Prof. sachy Weissman for helpful discussions. REFERENCES D. Blackwell, Information heory, E. F. Beckenbach, Ed. McGraw-Hill Book Co., New York, 96, vol. Modern Mathematics for the Engineer. R. Ash, Information heory. Interscience Publishers, K. Kobayashi H. Morita, An input/output recursion for the trapdoor channel, in Proc. IEEE Int. Symp. Inf. heory, Lausanne, Switzerl, Jun. 3 Jul. 5, p R. Ahlswede A. H. Kaspi, Optimal coding strategies for certain permuting channels. IEEE rans. Inf. heory, vol. 33, no. 3, pp. 3 34, H. Permuter, P. Cuff, B. Van Roy,. Weissman, Capacity of the trapdoor channel with feedback, IEEE rans. Inf. heory, vol. 54, no. 7, pp , Jul R. G. Gallager, Information heory Reliable Communication. John Wiley & Sons, Inc., G. H. Golub C. F. van Van Loan, Matrix Computations, 3rd ed. he Johns Hopkins University Press, Lutz, Various views on the trapdoor channel an upper bound on its capacity, 4, available online: 98