Capacity of the Trapdoor Channel with Feedback

Transcription

1 Capacity of the Trapdoor Channel with Feedback Haim Permuter, Paul Cuff, Benjamin Van Roy and Tsachy Weissman Abstract We establish that the feedback capacity of the trapdoor channel is the logarithm of the golden ratio and provide a simple communication scheme that achieves capacity. As part of the analysis, we formulate a class of dynamic programs that characterie capacities of unifilar finite-state channels. The trapdoor channel is an instance that admits a simple analytic solution. Index Terms Bellman equation, chemical channel, constrained coding, directed information, feedback capacity, golden-ratio, infinite-horion dynamic program, trapdoor channel, value iteration. I. ITRODUCTIO David Blackwell, who has done fundamental work both in information theory and in stochastic dynamic programming, introduced the trapdoor channel in 96 [] as a simple two-state channel. The channel is depicted in Figure, and a detailed discussion of this channel appears in the information theory book by Ash [], where indeed the channel is shown on the cover of the book. The channel behaves as follows. Balls labeled 0 or are used to communicate through the channel. The channel starts with a ball already in it. To use the channel, a ball is inserted into the channel by the transmitter, and the receiver receives one of the two balls in the channel with equal probability. The ball that does not exit the channel remains inside for the next channel use. Channel Input 0 Output Fig.. The trapdoor(chemical) channel. Another appropriate name for this channel is chemical channel. This name suggests a physical system in which the concentrations of chemicals are used to communicate, such as might be the case in some cellular biological systems. The transmitter adds molecules to the channel and the receiver samples molecules randomly from the channel. The trapdoor channel is the most basic realiation of this type of channel; it has only two types of molecules and there are only three possible concentrations, (0, 0.5, ), or alternatively only one molecule remains in the channel between uses. Although the trapdoor channel is very simple to describe, its capacity has been an open problem for 45 years []. The ero-error capacity was found by Ahlswede et al. [3], [4] to be 0.5 bits per channel use. More recently, Kobayashi and Morita [5] derived a recursion for the conditional probabilities of output sequences of length n given the input sequences and used it to show that the capacity of this channel is strictly larger than 0.5 bits. Ahlswede This work was supported by ational Science Foundation (SF) through the grants CCR-03633, CCF , IIS and the SF CAREER grant. The authors are with the Department of Electrical Engineering, Stanford University, Stanford, CA 94305, USA. ( {haim, pcuff, bvr, tsachy}@stanford.edu) The name chemical channel is due to T. Cover.

2 and Kaspi [3] considered two modes of the channel called the permuting jammer channel and the permuting relay channel. In the first mode there is a jammer in the channel who attempts to frustrate the message sender by selective release of balls in the channel. In the second mode, where the sender is in the channel, there is a helper supplying balls of a fixed sequence at the input, and the sender is restricted to permuting this sequence. The helper collaborates with the message sender in the channel to increase his ability to transmit distinct messages to the receiver. Ahlswede and Kaspi [3] gave answers for specific cases of both situations and Kobayashi [6] established the answer to the general permuting relay channel. More results for specific cases of the permuting jammer channel can be found in [7], [8]. In this paper we consider the trapdoor channel with feedback. We derive the feedback capacity of the trapdoor channel by solving an equivalent dynamic programming problem. Our work consists of two main steps. The first step is formulating the feedback capacity of the trapdoor channel as an infinite-horion dynamic program, and the second step is finding explicitly the exact solution of that program. Formulating the feedback capacity problem as a dynamic program appeared in Tatikonda s thesis [9] and in work by Yang, Kav cić and Tatikonda [0], Chen and Berger [], and recently in a work by Tatikonda and Mitter []. Yang et. al. [0] have shown that if a channel has a one-to-one mapping between the input and the state, it is possible to formulate feedback capacity as a dynamic programming problem and to find an approximate solution by using the value iteration algorithm [3]. Chen and Berger [] showed that if the state of the channel is a function of the output then it is possible to formulate the feedback capacity as a dynamic program with a finite number of states. Our work provides the dynamic programming formulation and a computational algorithm for finding the feedback capacity of a family of channels called unifilar Finite State Channels (FSC s), which include the channels considered in [0], []. We use value iteration [3] to find an approximate solution and to generate a conjecture for the exact solution, and the Bellman equation [4] to verify the optimality of the conjectured solution. As a result, we are able to show that the feedback capacity of the trapdoor channel is log φ, where φ is the golden ratio, + 5. In addition, we present a simple encoding/decoding scheme that achieves this capacity. The remainder of the paper is organied as follows. Section II defines the channel setting and the notation throughout the paper. Section III states the main results of the paper. Section IV presents the capacity of a unifilar FSC in terms of directed information. Section V introduces the dynamic programming framework and shows that the feedback capacity of the unifilar FSC can be characteried as the optimal average reward of a dynamic program. Section VI shows an explicit solution for the capacity of the trapdoor channel by using the dynamic programming formulation. Section VII gives a simple communication scheme that achieves the capacity of the trapdoor channel with feedback and finally Section VIII concludes this work. II. CHAEL MODELS AD PRELIMIARIES We use subscripts and superscripts to denote vectors in the following ways: x j = (x... x j ) and x j i = (x i... x j ) for i j. Moreover, we use lower case x to denote sample values, upper case X to denote random variables, calligraphic letter X to denote the alphabet and X to denote the cardinality of the alphabet. The probability distributions are denoted by p when the arguments specify the distribution, e.g. p(x y) = p(x = x Y = y). In this paper we consider only channels for which the input, denoted by {X, X,...}, and the output, denoted by {Y, Y,...}, are from finite alphabets, X and Y, respectively. In addition, we consider only the family of FSC known as unifilar channels as considered by Ziv [5]. An FSC is a channel that, for each time index, has one of a finite number of possible states, s t, and has the property that p(y t, s t x t, s t, y t ) = p(y t, s t x t, s t ). A unifilar FSC also has the property that the state s t is deterministic given (s t, x t, y t ): Definition : An FSC is called a unifilar FSC if there exists a time-invariant function f( ) such that the state evolves according to the equation s t = f(s t, x t, y t ). () We also define a strongly connected FSC, as follows. Definition : We say that a finite state channel is strongly connected if for any state s there exists an integer T and an input distribution of the form {p(x t s t )} T t= such that the probability that the channel reaches state s from any starting state s, in less than T time-steps, is positive. I.e. T Pr(S t = s S 0 = s ) > 0, s S, s S. () t=

3 3 y t Message m Encoder x t (m, y t ) x t Unifilar Finite State Channel p(y t x t, s t ) s t = f(s t, x t, y t ) y t Decoder ˆm(y ) Estimated message ˆm Feedback y t Unit Delay y t Fig.. Unifilar FSC with feedback We assume a communication setting that includes feedback as shown in Fig.. The transmitter (encoder) knows at time t the message m and the feedback samples y t. The output of the encoder at time t is denoted by x t and is a function of the message and the feedback. The channel is a unifilar FSC and the output of the channel y t enters the decoder (receiver). The encoder receives the feedback sample with one unit delay. A. Trapdoor Channel is a Unifilar FSC The state of the trapdoor channel, which is described in the introduction and shown in figure, is the ball, 0 or, that is in the channel before the transmitter transmits a new ball. Let x t {0, } be the ball that is transmitted at time t and s t {0, } be the state of the channel when ball x t is transmitted. The probability of the output y t given the input x t and the state of the channel s t is shown in table I. TABLE I THE PROBABILITY OF THE OUTPUT y t GIVE THE IPUT xt AD THE STATE st. x t s t p(y t = 0 x t, s t ) p(y t = x t, s t ) The trapdoor channel is a unifilar FSC. It has the property that the next state s t is a deterministic function of the state s t, the input x t, and the output y t. For a feasible tuple, (x t, y t, s t ), the next state is given by the equation where denotes the binary XOR operation. s t = s t x t y t, (3) B. Trapdoor Channel is a Permuting Channel It is interesting to note, although not consequential in this paper, that the trapdoor channel is a permuting channel [6], where the output is a permutation of the input (Fig. 3). At each time t, a new bit is added to the sequence and the channel switches the new bit with the previous one in the sequence with probability 0.5. III. MAI RESULTS The capacity of the trapdoor channel with feedback is 5 + C = log. (4)

4 Fig. 3. places. The trapdoor channel as a permuting channel. Going from left to right, there is a probability of one half that two adjacent bits switch Furthermore, there exists a simple capacity achieving scheme which will be presented in Section VII. The problem of finding the capacity of a strongly connected unifilar channel (Fig. ) can be formulated as an average-reward dynamic program, where the state of the dynamic program is the probability mass function over the states conditioned on prior outputs, and the action is the stochastic matrix p(x s). By finding a solution to the average-reward Bellman equation we find the exact capacity of the channel. As a byproduct of our analysis we also derive a closed form solution to an infinite horion average-reward dynamic program with a continuous state-space. IV. THE CAPACITY FORMULA FOR A UIFILAR CHAEL WITH FEEDBACK The main goal of this section is to prove the following theorem which allows us to formulate the problem as a dynamic program. Theorem : The feedback capacity of a strongly connected unifilar FSC when initial state s 0 is known at the encoder and decoder can be expressed as C F B = sup lim inf {p(x t s t,y t )} t I(X t, S t ; Y t Y t ) (5) where {p(x t s t, y t )} t denotes the set of all distributions such that p(x t y t, x t, s t ) = p(x t s t, y t ) for t =,,.... Theorem is a direct consequence of Theorem 3 and eq. (6) in Lemma 4, which are proved in this section. For any finite state channel with perfect feedback, as shown in Figure, the capacity was shown in [7], [8] to be bounded as lim max p(x y ) t= max I(X Y s 0 ) C F B lim s 0 max min I(X Y s 0 ). (6) p(x y ) s 0 The term I(X Y ) is the directed information defined originally by Massey in [5] as I(X Y ) I(X t ; Y t Y t ). (7) The initial state is denoted as s 0 and p(x y ) is the causal conditioning distribution defined [7], [] as p(x y ) t= p(x t x t, y t ). (8) t= The directed information in eq. (6) is under the distribution of p(x n, y n ) which is uniquely determined by the causal conditioning, p(x y ), and by the channel. In our communication setting we are assuming that the initial state is known both to the decoder and to the encoder. This assumption allows the encoder to know the state of the channel at any time t because s t is a deterministic function of the previous state, input and output. In order to take into account this assumption, we use a trick of allowing a fictitious time epoch before the first actual use of the channel in which the input does not influence the output nor the state of channel and the only thing that happens is that the output equals s 0 and is In addition to feedback capacity, directed information has recently been used in rate distortion [9], [0], [], network capacity [], [3] and computational biology [4].

5 5 fed back to the encoder such that at time t = both the encoder and the decoder know the state s 0. Let t = 0 be the fictitious time before starting the use of the channel. According to the trick, Y 0 = S 0 and the input X 0 can be chosen arbitrarily because it does not have any influence whatsoever. For this scenario the directed information term in eq. (6) becomes I(X0 Y0 s 0 ) = I(X Y s 0 ). (9) The input distribution becomes p(x 0 {s 0, y }) = p(x y, s 0 ), (0) where p(x y, s 0 ) is defined as p(x y, s 0 ) t= p(x t x t, y t, s 0 ). Therefore, the capacity of a channel with feedback for which the initial state, s 0, is known both at the encoder and the decoder is bounded as lim max p(x y,s 0) max I(X Y s 0 ) C F B lim s 0 max min I(X Y s 0 ) () p(x y,s 0) s 0 Lemma : If the finite state channel is strongly connected, then for any input distribution p (x y, s 0 ) and any s 0 there exists an input distribution p (x y, s 0) such that Ip (X Y s 0 ) I p (X Y s 0) c () where c is a constant that does not depend on, s 0, s 0. The term I p (X Y s 0 ) denotes the directed information induced by the input distribution p (x y, s 0 ) where s 0 is the initial state. Similarly, the term I p (X Y s 0) denotes the directed information induced by the input distribution p (x y, s 0) where s 0 is the initial state. Proof: Construct p (x y, s 0) as follows. Use an input distribution, which has a positive probability of reaching s 0 in T time epochs, until the time that the channel first reaches s 0. Such an input distribution exists because the channel is strongly connected. Denote the first time that the state of the channel equals s 0 by L. After time L, operate exactly as p would (had time started then). amely, for t > L, p (x t x t, y t, s 0 ) = p (x t L x t L, y t L, s 0 ). Then Ip (X Y s 0 ) I p (X Y s 0) (a) Ip (X Y s 0 ) I p (X Y L, s 0) + H(L) (b) = p(l = l)i p (X Y s 0 ) p(l = l) ( I p (Xl Yl s l ) + I p (X l Y l s l, s 0) ) + H(L) l= l= (c) p(l = l)i p (X Y s 0 ) p(l = l)i p (Xl Y l s l ) l= l= + p(l = l)i p (X l Y l s l, s 0) + H(L) (d) l= p(l = l)l log Y + H(L) l= = (log Y E[L] + H(L)) (3) (a) from the triangle inequality and Lemma 3 in [7] which claims that for an arbitrary random variables (X, Y, S), the inequality I(X Y ) I(X Y S) H(S) always holds. (b) follows from using the special structure of p (x y, s 0). (c) follows from the triangle inequality. (d) follows from the fact that in the first absolute value l terms cancel and therefor only l terms remain where each one of them is bounded by I(X t ; Y t Y t ) Y. In the second absolute value there are l terms also bounded by Y.

6 6 The proof is completed by noting that H(L) and E(L) are upper bounded respectively, by H( L) and E( L) where L/T Geometric(p) and p is the minimum probability of reaching s 0 in less than T steps from any state s S. Because the random variable L/T has a geometric distribution, H( L) and E[ L] are finite and, consequently, so are H(L) and E(L). Theorem 3: The feedback capacity of a strongly connected unifilar FSC when initial state is known at the encoder and decoder is given by C F B = lim max {p(x t s t,y t )} t= I(X t, S t ; Y t Y t ), (4) Proof: The proof of the theorem contains four main equalities which are proven separately. C F B = lim = lim = lim = lim t= max min I(X Y s 0 ) (5) p(x y,s 0) s 0 max I(X Y S 0 ) (6) p(x y,s 0) max p(x y,s 0) t= max {p(x t s t,y t )} t= Proof of equality (5) and (6): As a result of Lemma, I(X t, S t ; Y t Y t ) (7) I(X t, S t ; Y t Y t ). (8) t= where, lim max I(X Y S 0 ) p(x y,s 0) (a) = lim (b) = lim (c) = lim (d) = lim max p(x y,s 0) min s 0 s 0 p(x y,s 0) p(s 0 ) max max p(x y,s 0) s 0 p(s 0 )I(X Y s 0 ) I(X Y s 0 ) I(X Y s 0 ) (9) max min I(X Y s 0 ). (0) p(x y,s 0) s 0 (a) follows from the definition of conditional entropy. (b) follows from exchanging between the summation and the maximiation. The exchange is possible because maximiation is over causal conditioning distributions that depend on s 0. (c) follows from Lemma. (d) follows from the observation that the distribution p (x y, s 0 ) that achieves the maximum in (9) and in (0) is the same: p (x y, s 0 ) = arg max p(x y,s 0) I(X Y s 0 ). This observation allows us to exchange the order of the minimum and the maximum. Equations (9) and (0) can be repeated also with max s0 instead of min s0 and hence we get lim max I(X Y S 0 ) = lim p(x y,s 0) max max I(X Y s 0 ). () p(x y,s 0) s 0 By using eq. (0) and (), we get that the upper bound and lower bound in () are equal and therefore eq. (5) and (6) hold.

7 7 Proof of equality (7): Using the property that the next state of the channel is a deterministic function of the input, output and current state we get, I(X Y S 0 ) = = (a) = (b) = = I(X t ; Y t Y t, S 0 ) t= H(Y t Y t, S 0 ) H(Y t X t, Y t, S 0 ) t= H(Y t Y t, S 0 ) H(Y t X t, Y t, S 0, S t (X t, Y t, S 0 )) t= H(Y t Y t, S 0 ) H(Y t X t, S t, Y t, S 0 ) t= I(S t, X t ; Y t Y t, S 0 ). () t= Equality (a) is due to the fact that s t is a deterministic function of the tuple (x t, y t, s 0 ). Equality (b) is due to the fact that p(y t x t, y t, s t, s 0 ) = p(y t x t, y t, s t, s 0 ). By combining eq. (6) and eq. () we get eq. (7). Proof of equality (8): It will suffice to prove by induction that if we have two input distributions {p (x t x t, y t, s 0 )} t and {p (x t x t, y t, s 0 )} t that induce the same distributions {p(x t s t, y t )} t then the distributions {p(s t, x t, y t )} t are equal under both inputs. First let us verify the equality for t = : p(s 0, x, y ) = p(s 0 )p(x s 0 )p(y s 0, x ). (3) Since p(s 0 ) and p(y s 0, x ) are not influenced by the input distribution and since p(x s 0 ) is equal for both input distributions then p(s 0, x, y ) is also for both input distributions. ow, we assume that p(s t, x t, y t ) is equal under both input distributions and we need to prove that p(s t, x t+, y t+ ) is also equal under both input distributions. The term p(s t, x t+, y t+ ) which can be written as, p(s t, x t+, y t+ ) = p(s t, y t )p(x t+ s t, y t )p(y t+ x t+, s t ). (4) First we notice that if p(s t, x t, y t ) is equal for both cases then necessarily p(s t, s t, x t, y t ) is also equal for both cases because s t is a deterministic function of the tuple (s t, x t, y t ) and therefore both input distributions induce the same p(s t, y t ). The distribution, p(x t+ s t, y t ), is the same under both input distributions by assumption and p(y t+ x t+, s t ) does not depend on the input distribution. The next lemma shows that it is possible to switch between the limit and the maximiation in the capacity formula. This is necessary for formulating the problem, as we do in the next section, as an average-reward dynamic program. Lemma 4: For any FSC the following equality holds: lim max min I(X Y s 0 ) = sup lim inf p(x y,s 0) s 0 {P (x t y t,x t,s 0)} t And, in particular, for a strongly connected unifilar FSC lim max {p(x t s t,y t )} t= I(X t, S t ; Y t Y t ) = t= sup lim inf {p(x t s t,y t )} t min s 0 I(X Y s 0 ). (5) I(X t, S t ; Y t Y t ) On the left-hand side of the equations appears lim because, as shown in [8], the limit exists due to the superadditivity property of the sequence. Proof: We are going to prove eq. (5) which hold for any FSC. For the case of unifilar channel, the left-hand side of eq. (5) is proven to be equal to the left side of eq. (6) in eq. (5)-(8). By the same arguments as in (5)-(8) also the right-hand side of (5) and (6) are equal. t= (6)

8 8 Define C max min I(X Y s 0 ). (7) p(x y,s 0) s 0 In order to prove that the equality holds we will use two properties of C that were proved in [8, Theorem 3]. The first property, is that C is a super additive sequence, namely, [ C log S ] n [ C n log S n The second property, which is a result of the first, is that ow, consider lim max min p(x y,s 0) s 0 I(X Y s 0 ) = sup ] + l [ C l ] log S. (8) l lim C = sup C (9) = sup = sup C max min I(X Y s 0 ) p(x y,s 0) s 0 sup min I(X Y s 0 ) {p(x t y t,x t,s s 0)} 0 t = sup sup {p(x t y t,x t,s 0)} t min I(X Y s 0 ) s 0 sup lim inf {p(x t y t,x t,s 0)} t min I(X Y s 0 )(30) s 0 The limit of the left side of the equation in the lemma implies that, ɛ > 0 there exists (ɛ) such that for all n > (ɛ), n max p(x n y n,s 0) min s0 I(X Y s 0 ) sup C ɛ. Let us choose j > (ɛ) and let p (x j y j ) be the input distribution that attains the maximum. Let us construct Then we get, p(x t y t, s 0 ) = p (x t t j+ y t t j+, s t j)p (x t j t j+ yt j t j+, s t j).... (3) sup lim inf {p(x t y t,x t,s 0)} min s 0 I(X Y s 0 ) lim inf min s 0 I p (X Y s 0 ) sup C ɛ (3) where I p (X Y s 0 ) is the directed information induced by the input p(x t y t, s 0 ) and the channel. The left inequality holds because p(x t y t, s 0 ) is only one possible input distribution among all {p(x t y t, s 0 )} t=. The right inequality holds because the special structure of p(x t y t, s 0 ) transforms the whole expression of normalied directed information into an average of infinite sums of terms that each term is directed information between blocks of length j. Because for each block the inequality holds, then it holds also for the average of the blocks. The inequality may not hold on the last block, but because we average over an increasing number of blocks its influence diminishes. V. FEEDBACK CAPACITY AD DYAMIC PROGRAMMIG In this section, we characterie the feedback capacity of the unifilar FSC as the optimal average-reward of a dynamic program. Further, we present the Bellman equation, which can be solved to determine this optimal average reward. A. Dynamic Programs Here we introduce a formulation for average-reward dynamic programs. Each problem instance is defined by a septuple (Z, U, W, F, P, P w, g). We will explain the roles of these parameters.

9 9 We consider a discrete-time dynamic system evolving according to t = F ( t, u t, w t ), t =,, 3,..., (33) where each state t takes values in a Borel space Z, each action u t takes values in a compact subset U of a Borel space, and each disturbance w t takes values in a measurable space W. The initial state 0 is drawn from a distribution P. Each disturbance w t is drawn from a distribution P w ( t, u t ) which depends only on the state t and action u t. All functions considered in this paper are assumed to be measurable, though we will not mention this each time we introduce a function or set of functions. The history h t = ( 0, w 0,..., w t ) summaries information available prior to selection of the tth action. The action u t is selected by a function µ t which maps histories to actions. In particular, given a policy π = {µ, µ,...}, actions are generated according to u t = µ t (h t ). ote that given the history h t and a policy π = {µ, µ,...}, one can compute past states,..., t and actions u,..., u t. A policy π = {µ, µ,...} is referred to as stationary if there is a function µ : Z U such that µ t (h t ) = µ( t ) for all t and h t. With some abuse of terminology, we will sometimes refer to such a function µ itself as a stationary policy. We consider an objective of maximiing average reward, given a bounded reward function g : Z U R. The average reward for a policy π is defined by ρ π = lim inf { E π t=0 g(z t, µ t+ (h t+ )) where the subscript π indicates that actions are generated by the policy π = (µ, µ,...). The optimal average reward is defined by ρ = sup ρ π. π }, B. The Bellman Equation An alternative characteriation of the optimal average reward is offered by the Bellman Equation. This equation offers a mechanism for verifying that a given level of average reward is optimal. It also leads to a characteriation of optimal policies. The following result which we will later use encapsulates the Bellman equation and its relation to the optimal average reward and optimal policies. Theorem 5: If ρ R and a bounded function h : Z R satisfy ρ + h() = sup u U ( g(, u) + ) P w (dw, u)h(f (, u, w)) Z (34) then ρ = ρ. Further, if there is a function µ : Z U such that µ() attains the supremum for each then ρ π = ρ for π = (µ 0, µ,...) with µ t (h t ) = µ( t ) for each t. This result follows immediately from Theorem 6. of [4]. It is convenient to define a dynamic programming operator T by ( (T h)() = sup g(, u) + u U ) P w (dw, u)h(f (, u, w)), for all functions h. Then, Bellman s equation can be written as ρ + h = T h. It is also useful to define for each stationary policy µ an operator (T µ h)() = g(, µ()) + P w (dw, µ())h(f (, µ(), w)). The operators T and T µ obey some well-known properties. First, they are monotonic: for bounded functions h and h such that h h, T h T h and T µ h T µ h. Second, they are non-expansive with respect to the sup-norm: for bounded functions h and h, T h T h h h and T µ h T µ h h h. Third, as a consequence of nonexpansiveness, T is continuous with respect to the sup-norm. 3 3 The proof of the properties of T are entirely analogous to the proofs of Propositions.. and..4 in [3, Vol. II]

10 0 C. Feedback Capacity as a Dynamic Program We will now formulate a dynamic program such that the optimal average reward equals the feedback capacity of a unifilar channel as presented in Theorem. This entails defining the septuple (Z, U, W, F, P, P w, g) based on properties of the unifilar channel and then verifying that the optimal average reward is equal to the capacity of the channel. Let β t denote the S -dimensional vector of channel state probabilities given information available to the decoder at time t. In particular, each component corresponds to a channel state s t and is given by β t (s t ) p(s t y t ). We take states of the dynamic program to be t = β t. Hence, the state space Z is the S -dimensional unit simplex. Each action u t is taken to be the matrix of conditional probabilities of the input x t given the previous state s t of the channel. Hence, the action space U is the set of stochastic matrices of dimension S X. The disturbance w t is taken to be the channel output y t. The disturbance space W is the output alphabet Y. The initial state distribution P is concentrated at the prior distribution of the initial channel state s 0. ote that the channel state s t is conditionally independent of the past given the previous channel state s t, the input probabilities u t, and the current output y t. Hence, β t (s t ) = p(s t y t ) = p(s t β t, u t, y t ). More concretely, given a policy π = (µ, µ,...), β t (s t ) = p(s t y t ) = p(s t, s t, x t y t ) x t,s t = = = = p(s t, s t, x t, y t yt ) p(y x t y t ) t,s t p(s t yt )p(x t s t, yt )p(y t s t, x t )p(s t s t, x t, y t ) p(y x t y t ) t,s t x t,s t β t (s t )p(x t s t, y t )p(y t s t, x t )p(s t s t, x t, y t ) x t,s t,s t β t (s t )p(x t s t, y t )p(y t s t, x t )p(s t s t, x t, y t ) x t,s t β t (s t )u t (s t, x t )p(y t s t, x t )(s t = f(s t, x t, y t )) x t,s t,s t β t (s t )u t (s t, x t )p(y t s t, x t )(s t = f(s t, x t, y t )), (35) where ( ) is the indicator function. ote that p(y t s t, x t ) is given by the channel model. Hence, β t is determined by β t, u t, and y t, and therefore, there is a function F such that t = F ( t, u t, w t ). The distribution of the disturbance w t is p(w t t, w t, u t ) = p(w t t, u t ). Conditional independence from t and w t given t is due to the fact that the channel output is determined by the previous channel state and current input. More concretely, p(w t t, w t, u t ) = p(y t β t, y t, u t ) = p(y t, x t, s t β t, y t, u t ) x t,s t = p(s t β t, u t )p(x t s t, β t, u t )p(y t x t, s t, β t, u t ) x t,s t = p(s t, x t, y t β t, u t ) x t,s t = p(y t β t, u t ) = p(w t t, u t ). (36) Hence, there is a disturbance distribution P w ( t, u t ) that depends only on t and u t. We consider a reward of I(Y t ; X t, S t y t ). ote that the reward depends only on the probabilities

11 p(x t, y t, s t y t ) for all x t, y t and s t. Further, p(x t, y t, s t y t ) = p(s t y t )p(x t s t, y t )p(y t x t, s t ) = β t (s t )u t (s t, x t )p(y t x t, s t ). (37) Recall that p(y t x t, s t ) is given by the channel model. Hence, the reward depends only on β t and u t. Given an initial state 0 and a policy π = (µ, µ,...), u t and β t are determined by y t. Further, (X t, S t, Y t ) is conditionally independent of y t given β t as shown in (37). Hence, g( t, u t ) = I(Y t ; X t, S t y t ) = I(X t, S t ; Y t β t, u t ). (38) It follows that the optimal average reward is [ ] ρ = sup lim inf π E π I(X t, S t ; Y t Y t ) = C F B. t= The dynamic programming formulation that is presented here is an extension of the formulation presented in [0] by Yang, Kav cić and Tatikonda. In [0] the formulation is for channels with the property that the state is deterministically determined by the previous inputs and here we allow the state to be determined by the previous outputs and inputs. VI. SOLUTIO FOR THE TRAPDOOR CHAEL The trapdoor channel presented in Section II is a simple example of a unifilar FSC. In this section, we present an explicit solution to the associated dynamic program, which yields the feedback capacity of the trapdoor channel as well as an optimal encoder-decoder pair. The analysis begins with a computational study using numerical dynamic programming techniques. The results give rise to conjectures about the average reward, the differential value function, and an optimal policy. These conjectures are proved to be true through verifying that they satisfy Bellman s equation. A. The Dynamic Program In Section V-C, we formulated a class of dynamic programs associated with unifilar channels. From here on we will focus on the particular instance from this class that represents the trapdoor channel. Using the same notation as in Section V-C, the state t would be the vector of channel state probabilities [p(s t = 0 y t ), p(s t = y t )]. However, to simplify notation, we will consider the state t to be the first component; that is, t p(s t = 0 y t ). This comes with no loss of generality the second component can be derived from the first since the pair sum to one. The action is a stochastic matrix u t = [ p(xt = 0 s t = 0) p(x t = s t = 0) p(x t = 0 s t = ) p(x t = s t = ) ]. (39) The disturbance w t is the channel output y t. The state evolves according to t = F ( t, u t, w t ), where using relations from eq. (3, 35) and Table I, we obtain the function F explicity as t = These expressions can be simplified by defining So that t u t(,) t u t(,)+0.5 t u t(,)+0.5( t )u t(,) if w t = 0 0.5( t )u t(,)+0.5 t u t(,) 0.5( t )u t(,)+0.5 t u t(,)+( t )u t(,) if w t =. t = γ t ( t )u t (, ), (40) δ t t u t (, ). (4) δ t +δ t γ t if w t = 0 γt δ t+γ t if w t =.

12 ote that, given t, the action u t defines the pair (γ t, δ t ) and vice-versa. From here on we will represent the action in terms of γ t and δ t. Because u t is required to be a stochastic matrix, δ t and γ t are constrained by 0 δ t t and 0 γ t t. Recall from eq. (38) that the reward function is given by g( t, u t ) = I(X t, S t ; Y t β t, u t ). This reward can be computed from the conditional probabilities p(x t, s t, y t β t, u t ). Using the expressions for these conditional probabilities provided in Table II, we obtain g( t, u t ) = I(X t, S t ; Y t β t, u t ) = H(Y t u t, β t ) H(Y t X t, S t, β t, u t ) ( = H t u t (, ) + t u t (, ) ( = H + δ ) t γ t + δ t + γ t, + ( t )u t (, ) ) t u t (, ) ( t )u t (, ) where, with some abuse of notation, we use H to denote the binary entropy function: H(q) = q ln q ( q) ln( q). TABLE II THE CODITIOAL DISTRIBUTIO p(x t, s t, y t β t, u t). x t s t y t = 0 y t = 0 0 β t u t (, ) ( β t )u t (, ) 0.5( β t )u t (, ) 0 0.5β t u t (, ) 0.5β t u t (, ) 0 ( β t )u t (, ) We now have a dynamic program the objective is to maximie over all policies π the average reward ρ π. The capacity of the trapdoor channel is the maximum of the average reward ρ. In the context of the trapdoor channel, the dynamic programming operator takes the form ( ( (T h)() = sup H 0 δ,0 γ + δ γ ) + δ + γ + + δ γ ( ) δ h + δ + γ ( γ h + δ γ (4) By Theorem 5, if we identify a scalar ρ and bounded function h that satisfy Bellman s equation, ρ + T h = h, then ρ is the optimal average reward. Further, if for each, T µ h = T h then the stationary policy µ is an optimal policy. B. Computational Study We carried out computations to develop an understanding of solutions to Bellman s equation. For this purpose, we used the value iteration algorithm, which in our context generates a sequence of iterates according to δ + γ J k+ = T J k, (43) initialied with J 0 = 0. For each k and, J k () is the maximal expected reward over k time periods given that the system starts in state. Since rewards are positive, for each, J k () grows with k. For each k, we define a differential reward function h k () J k () J k (0). These functions capture differences among values J k () for different states x. Under certain conditions such as those presented in [6], the sequence h k converges uniformly to a function that solves Bellman s equation. We will neither discuss such conditions nor verify that they hold. Rather, we will use the algorithm heuristically in order to develop intuition and conjectures. Value iteration as described above cannot be implemented on a computer because it requires storing and updating a function with infinite domain and optimiing over an infinite number of actions. To address this, we discretie the state and action spaces, approximating the state space using a uniform grid with 000 points in the unit interval and restricting actions δ and γ to values in a uniform grid with 4000 points in the unit interval. )).

13 3 We executed twenty value iterations. Figure 4 plots the function J 0 and actions that maximie the right-hand-side of eq. (43) with k = 0. We also simulated the system, selecting actions δ t and γ t in each time period to maximie this expression. This led to an average reward of approximately We plot in the right-bottom side of Figure 4 the relative state frequencies of the associated Markov process. ote that the distribution concentrates around four points which are approximately 0.36, 0.38, 0.63, and J Value function on the 0 th iteration, J 0 δ Action-parameter, δ PSfrag replacements γ Action-parameter, γ relative frequency Histogram of Fig. 4. Results from 0 value iterations. On the top-left side the value function J 0 is plotted. On the top-right and bottom-left the optimal action-parameters δ and γ with respect to 0 th iteration are plotted. On the bottom-right the relative state frequencies of the associated Markov process of with the policy that is optimal with respect to J 0 is plotted. C. Conjectures The results obtained from value iteration were, amaingly, close to the answers of two questions given in an information theory class at Stanford taught by Professor Thomas Cover. Here is a simplified version of the questions given to the class. () Entropy rate. Find the maximum entropy rate of the two-state Markov chain (Fig. 5) with transition matrix [ ] p p P =, (44) 0 where 0 p is the free parameter we maximie over. PSfrag replacements Fig. 5. The Markov chain of question. p p 0

14 4 () umber of sequences. To first order in the exponent, what is the number of binary sequences of length n with no two s in a row? The entropy rate of the Markov chain of question () is given by H(p) +p, and when maximiing over 0 p, we get that p = 3 5 and the entropy rate is It can be shown that the number of sequences of length n that do not have two s in a row is the n th number in the Fibonacci sequence. This can be proved by induction in the following way. Let us denote (n, 0 n) the number of sequences of length n with the condition of not having two s in a row that are ending with 0 and with respectively. For the sequences that end with 0 we can either add a next bit or 0 and for the sequences that end with we can add only 0. Hence n+ 0 = n 0 + n and n+ = n. 0 By repeating this logic, we get that n 0 behaves as a Fibonacci sequence. To first order in the exponent, the Fibonacci number behaves as lim n n log f n = log + 5 = 0.694, where the number, + 5, is called the golden ratio. The golden ratio is also known to be a positive number that solves the equation φ = φ, and it appears in many math, science and art problems [7]. As these problems illustrate, the number of typical sequences created by the Markov process given in question () is, to first order in the exponent, equal to the number of binary sequences that do not have two s in a row. Let us consider a policy for the dynamic program associated with a binary random process that is created by the Markov chain from question (see Fig 5). Let the state of the Markov process indicate if the input to the channel will be the same or different from the state of the channel. In other words, if at time t the binary Markov sequence is 0 then the input to the channel is equal to the state of the channel, i.e. x t = s t. Otherwise, the input to the channel is a complement to the state of the channel, i.e. x t = s t. This scheme uniquely defines the distribution p(x t s t, y t ): p(x t = s t s t, y t ) = { p if st = y t, if s t y t. This distribution is derived from the fact that for the trapdoor channel the state evolves according to equation (3) which can be written as s t y t = x t s t. (46) Hence, if s t y t then necessarily also x t s t. This means that the tuple (s t, y t ) defines the state of the Markov chain at time t and the tuple (x t, s t ) defines the state of the Markov chain at time t. Having the distribution p(x t s t, y t ), for the following four values of, {b 5, b 3 5, b 3 5, b 4 3 5}, the corresponding actions γ() and δ() which are defined in eq. (40,4) are: (45) γ() δ() b or b 5 ( ) b 3 or b 4 5 It can be verified, by using eq. (35), that the only values of ever reached are, { b 5, b 3 5 5, b 3, b 4 3 } 5, (47) and the transitions are a function of y t shown graphically in Figure 6. Our goal is to prove that an extension of this policy is indeed optimal. Based on the result of Question, we conjugate that the entropy rate of the average reward is 3 H( 5 ) 5 + ρ = = log (48) It is interesting to notice that all the numbers appearing above can be written in terms of the golden ratio, φ = 5. In particular, ρ = log φ, b = φ 3, b = φ, b 3 = φ and b 4 = 4 φ. By inspection of Figure 4, we let γ and δ be linear over the intervals [b, b ], [b, b 3 ], and [b 3, b 4 ] and we get the form presented in Table III.

15 PSfrag replacements 5 β t b b b 3 b 4 y t β t b b b 3 b 4 Fig. 6. The transition between β t and β t, under the policy δ, γ. γ() δ() b b 5 ( ) 3 b b b 3 b 4 5 TABLE III COJECTURED POLICY WHICH I THE EXT SECTIO WILL BE PROVE TO BE TRUE. We now propose differential values h() for [b, b 4 ]. If we assume that δ and γ maximie the right-hand-side of the Bellman equation (eq. 34) for [b, b 4 ] with h = h and ρ = ρ, we obtain ( ) h() = H ( 5 ) ρ + h(3 5), b b 3, (49) ( ) ( ) ( ) h() = H ρ+ 4 h(3 5)+ 4 h 5+ 4, b 3 b 4. (50) The equation for the range b b is implied by the symmetry relation: h() = h( ). If a scalar ρ and function h solve Bellman s equation, so do ρ and h + c for any scalar c. Therefore, there is no loss of generality in setting h(/) =. From eq. (49) we have that h() =, b b 3. (5) In addition, by symmetry considerations we can deduce that h( 5 ) = h(3 5) and from eq. (49) we obtain h( 5 ) = h(3 5) = ρ (5) Taking symmetry into consideration and applying eq. (50) twice we obtain, where c = log(3 5). By symmetry we obtain h() = H() + ρ + c, b 3 b 4, (53) h() = H() ρ + c, b b. (54) where c = log( 5 ) The conjectured policy ( γ, δ), which is given in Table III, and the conjectured differential value h, which is given in eq. (5)-(54), are plotted in Fig. 7.

16 PSfrag replacements 6 H() ρ + c H() + ρ + c Action-parameter, δ h b b b 3 b δ Action-parameter, γ γ ( ) Fig. 7. A conjecture about the optimal solution based on the 0 th value iteration of the DP which is shown in Fig. 4 and on the questions given by Professor Cover. On the top-left the conjectured differential value h() is plotted for [b, b 4 ]. On the top-right side and bottom-left the conjectured policy ( δ(), γ()) is plotted for [b, b 4 ] D. Verification In this section, we verify that the conjectures made in the previous section are correct. Our verification process proceeds as follows. First, we establish that if a function h : [0, ] R is concave, so is T h. In other words, value iteration retains concavity. We then consider a version of value iteration involving an iteration h k+ = T h k ρ. Since subtracting a constant does not affect concavity, this iteration also retains concavity. We prove that if a function h 0 is the pointwise maximum among concave functions that are equal to h in the interval [b, b 4 ] then each iterate h k is also concave and equal to h in this interval. Further, the sequence is pointwise nonincreasing. These properties of the sequence imply that it converges to a function h that again is concave and equal to h in the interval [b, b 4 ]. This function h together with ρ satisfies Bellman s Equation. Given this, Theorem 5 verifies our conjectures. We begin with a lemma that will be useful in showing that value iteration retains concavity. Lemma 6: Let ζ : [0, ] [0, ] R be concave on [0, ] [0, ] for all [0, ] and ψ() = sup ζ(δ, γ). δ [0,],γ [0, ] Then ψ : [0, ] R is concave. The proof of Lemma 6 is given in the appendix. Lemma 7: The operator T, defined in (4) retains concavity and continuity. amely, if h is concave then T h is concave, if h is continuous then T h is continuous. Proof (concavity): It is well-known that the binary entropy function H is concave, so the reward function H ( + δ γ ) + δ + γ

17 7 is concave in (δ, γ). ext, we show that if h() is concave then +δ γ h ( δ +δ γ ) is concave in (δ, γ). Let ξ = +δ γ and ξ = +δ γ. We will show that, for any α (0, ), ( ) ( ) ( ) δ δ αδ + ( α)δ αξ h + ( α)ξ h (αξ + ( α)ξ )h. (55) ξ ξ αξ + ( α)ξ Dividing both sides by (αξ + ( α)ξ ) we get ( ) δ + αξ αξ + ( αξ ) h ξ ( α)ξ αξ + ( αξ ) h ote that the last inequality is true because of the concavity of h. It follows that ( f(δ, γ) H + δ γ ) + δ + γ + + δ γ ( ) δ h + δ γ is concave in (δ, γ). Since ( δ ξ (T h)() = sup f(δ, γ), δ [0,],γ [0, ] ) ( ) αδ + ( α)δ h. (56) αξ + ( α)ξ + δ + γ h ( ) γ δ + γ it is concave by Lemma 6. ( ) δ Proof (continuity): ote that the binary entropy function H is continuous. Further, h +δ γ and ( ) h γ δ+γ, are continuous over the region {(δ, γ) δ 0, γ 0, δ + γ }. It follows that f(δ, γ) is continuous over the region {(δ, γ) δ 0, γ 0, δ + γ }. Hence, is continuous over [0, ]. (T h)() = sup f(δ, γ) δ [0,],γ [0, ] Let us construct value iteration function h k () as follows. Let h 0 () be the pointwise maximum among concave functions satisfying h 0 () = h() for [b, b 4 ], where h() is defined in eq.(5)-(54). ote that h 0 () is concave and that for / [b, b 4 ], h() is a linear extrapolation from the boundary of [b, b 4 ]. Let and (57) h k+ () = (T h k )() ρ, (58) h () lim sup h k (). (59) k The following lemma shows several properties of the sequence of function h k () including the uniform convergence. The uniform convergence is needed for verifying the conjecture, while the other properties are intermediate steps in proving the uniform convergence. Lemma 8: The following properties hold: 8. for all k 0, h k () is concave and continuous in 8. for all k 0, h k () is symmetric around, i.e. 8.3 for all k 0, h k () is fixed point for [b, b 4 ], i.e., h k () = h k ( ) (60) h k () = h(), [b, b 4 ], (6) and the stationary policy µ() = ( δ(), γ()), where ( δ(), γ()) are defined in Table III, satisfies (T µ h k )() = (T h k )() 8.4 h k () is uniformly bounded in k and, i.e., sup k sup h k () < (6) [0,]

18 8 8.5 h k () is monotonically nonincreasing in k, i.e. lim h k() = h () (63) k 8.6 h k () converges uniformly to h () Proof of 8.: Since h 0 () is concave and continuous and since the operator T retain continuity and concavity (see Lemma 7),it follows that h k () is concave and continuous for every k. Proof of 8.: We prove this property by induction. First notice that h 0 () is symmetric and satisfies h 0 () = h 0 ( ). ow let us show that if it holds for h k then it holds for h k+. Let f k (δ, γ) denote the expression maximied to obtain (T h k )(), i.e. ( f k (δ, γ) H + δ γ ) + δ + γ + + δ γ ( ) δ h k + δ + γ ) γ h k (. (64) + δ γ δ + γ otice that f k (δ, γ) = f k (γ, δ). Also observe that replacing the argument with in T h k yield the same result as exchanging between γ and δ. From those two observations follows that T h k () = T h k ( ) and from the definition of h k+ given in (58) follows that h k+ () = h k+ ( ). Proof of 8.3: We prove this property by induction. otice that h 0 satisfies h 0 () = h() for [b, b 4 ]. We assume that h k satisfies h k () = h() and then we will prove the property for h k+. We will show later in this proof that for [b, b 4 ], (T µ h k )() = (T h k )(). (65) Since (T µ h k )() ρ = h() for all [b, b 4 ] (see eq ) it follows that h k+ () = h() for all [b, b 4 ]. ow, let us show that (65) holds. Recall that in the proof of Lemma 7, eq. (64), we showed that f k (δ, γ) is concave in (δ, γ). The derivative with respect to δ is, f k (δ, γ) δ = log δ + γ + δ γ + + ( h k + γ ( δ + δ γ h k + δ γ δ + δ γ ) + γ δ + γ h k ) ( h k ( ) γ δ + γ γ δ + γ ). (66) The derivative with respect to γ is entirely analogous and can be obtained by mutually exchanging γ and δ. For [b, b 3 ], the action γ() = δ() = 3 5 γ() is feasible and = δ() = b δ()+ γ() + δ() γ() 4. Moreover, it is straightforward to check that the derivatives of f k are ero at ( γ(), δ()), and since f k is concave, ( γ(), δ()) attains the maximum. Hence, (T µ h)() = (T h)() for [b, b 3 ]. For [b 3, b 4 ], γ() = and δ() = 5 γ(). ote that and δ() are in [b δ()+ γ() + δ() γ(), b ] [b 3, b 4 ]. Using expressions for h() given in equations (53) and (54), we can write derivatives of f at ( δ(), γ()) as f( δ(), γ()) δ f( δ(), γ()) γ = log δ() γ() δ() = log δ() γ() γ() + + ρ = 0, (67) + + ρ 0. (68) otice that γ() is the maximum of the feasible set [0, ] and the derivative of f k with respect to γ at ( δ(), γ()) is positive. In addition, δ() is in the interior of the feasible set [0, ] and the derivative of f k with respect to δ at ( δ(), γ()) is ero. Since f k is concave, any feasible change in ( γ(), δ()) will decrease the value of the function. Hence, (T µ h k )() = (T h k )() for [b 3, b 4 ]. The situation for [b, b ] is completely analogous. Proof of 8.4: From Propositions , it follows that the maximum over of h k () is attained at = / and h k (/) = for all k. Further more because of concavity and symmetry the minimumm of h k () is attained at = 0 and =. Hence it is enough to show that h k (0) is uniformly bounded from below for all k. For = 0 let us consider the action γ = b b and δ = 0 and for b b 4 the action γ(), δ(). ow let us prove that under this policy h k (0) that is less or equal the optimal value is uniformly bounded. Under this policy h k+ (0) = (T h k )(0) ρ becomes h k (0) = c + αh k (0) + ( α) ρ (69)

19 9 ( ) where c and α are constant: c = H +b + b, α = b. Iterating the equation (69) k time we get k h k (0) = (c + α ρ) α i + α k h 0 (0). (70) Since α <, h k (0) is uniformly bounded for all k. Proof of 8.5: By Proposition 8. h k is concave for each k and by Proposition 8.3 h k () = h() for [b, b 4 ]. Since h 0 is the pointwise maximum of functions satisfying this condition, we must have h 0 h. It is easy to see that T is a monotonic operator. As such, h k h k+ for all k. Proposition 8.4 establishes that the sequence is bounded below, and therefore it converges pointwise. Proof of 8.6: By Proposition 8., each h k is concave and continuous. Further, by Proposition 8.5, the sequence has a pointwise limit h which is concave. Concavity of h implies continuity [8, Theorem 0.] over (0, ). Let h be the continuous extension of h from (0, ) to [0, ]. Since h is concave, h h. By Proposition 8.5, h k h. It follows from continuity of h k that h k h. Hence, h () = lim k h k () h () for [0, ]. Recalling that h h, we have h = h. Since the iterates h k are continuous and monotonically nonincreasing and their pointwise limit h is continuous, h k converges uniformly by Dini s Theorem [9]. The following theorem verifies our conjectures. Theorem 9: The function h and scalar ρ satisfy ρ + h = T h. Further, ρ is the optimal average reward and there is an optimal policy that takes actions δ t = δ( t ) and γ t = γ( t ) whenever t [b, b 4 ]. Proof: Since the sequence h k+ = T h k ρ converges uniformly and T is sup-norm continuous, h = T h ρ. It follows from Theorem 5 that ρ is the optimal average reward. Together with Proposition 8.3, this implies existence of an optimal policy that takes actions δ t = δ( t ) and γ t = γ( t ) whenever t [b, b 4 ]. i=0 VII. A CAPACITY-ACHIEVIG SCHEME In this section we describe a simple encoder and decoder pair that provides error-free communication through the trapdoor channel with feedback and known initial state. We then show that the rates achievable with this encoding scheme are arbitrarily close to capacity. It will be helpful to discuss the input and output of the channel in different terms. Recall that the state of the channel is known to the transmitter because it is a deterministic function of the previous state, input, and output, and the initial state is known. Let the input action, x, be one of the following: { 0, input ball is same as state x =, input ball is opposite of state Also let the output be recorded differentially as, { 0, received ball is same as previous ỹ =, received ball is opposite of previous where ỹ is undefined and irrelevant for our scheme. A. Encode/Decode Scheme Encoding. Each message is mapped to a unique binary sequence of actions, x n, that ends with 0 and has no occurrences of two s in a row. The input to the channel is derived from the action and the state as, x k = x k s k. Decoding. The channel outputs are recorded differentially as, ỹ k = y k y k, for k =,...,. Decoding of the action sequence is accomplished in reverse order, beginning with x = 0 by construction. Lemma 0: If x k+ is known to the decoder, x k can be correctly decoded. Proof: Table IV shows how to decode x k from x k+ and ỹ k+. Proof of case. Assume that x k =. At time k, just before the output is received, there are balls of both types in the channel. By symmetry we can assume that the ball that exits is labeled 0. Therefore, the ball labeled remains in the channel. According to the encoding scheme, x k+ = 0 because repeated s are not allowed, which