Stochastic Interpretation for the Arimoto Algorithm

Transcription

1 Stochastic Interpretation for the Arimoto Algorithm Serge Tridenski EE - Sstems Department Tel Aviv Universit Tel Aviv, Israel sergetr@post.tau.ac.il Ram Zamir EE - Sstems Department Tel Aviv Universit Tel Aviv, Israel zamir@eng.tau.ac.il arxiv: v2 cs.it] 11 Mar 2015 Abstract The Arimoto algorithm computes the Gallager function ma Q ρ,q for a given channel P and parameter ρ, b means of alternating maimization. Along the wa, it generates a sequence of input distributions Q 1, Q 2,..., that converges to the maimizing input Q. We propose a stochastic interpretation for the Arimoto algorithm. We show that for a random i.i.d. codebook with a distributionq k, the net distribution Q k+1 in the Arimoto algorithm is equal to the tpe Q of the feasible transmitted codeword that maimizes the conditional Gallager eponent conditioned on a specific transmitted codeword tpe Q. This interpretation is a first step toward finding a stochastic mechanism for on-line channel input adaptation. 1 Inde Terms Arimoto-Blahut algorithm, Gallager error eponent, natural tpe selection, channel input adaptation. I. INTRODUCTION Optimization of the transmitter output, under a set of constraints and channel conditions, is an important goal in digital communication. In information theoretic terms, it corresponds to selecting the channel input distribution Q that maimizes the mutual information to achieve the Shannon capacit C of the channel or the Gallager function to achieve the best error eponent ER. Since the channel conditions are often changing, practical schemes tend to graduall adapt to these conditions, using feedback from the receiver to the transmitter. A previous work 1], 2] found the phenomenon of natural tpe selection NTS in adaptive loss source coding: the empirical distribution or tpe Q of the first D-matching codeword in an ordered random codebook, generated i.i.d. according to a distribution Q, amounts asmptoticall as the word length goes to infinit to a single iteration from Q to Q of the Blahut algorithm for rate-distortion function computation 3]. B iterating the NTS step, the sequence of codebook distributions Q, Q, Q,... converges to the reproduction distribution Q that realizes the rate-distortion function RD. This result gives a stochastic flavor to the Blahut algorithm, b interpreting each iteration as an instance of the conditional limit theorem of large deviation theor. In this paper we provide a first step toward the channelcoding counterpart of NTS. The underling idea is that the tpe of a good codeword good in the sense of having a low individual error probabilit indicates the direction to which the codebook distribution should evolve. Similarl to 1 This work of S. Tridenski and R. Zamir was supported in part b the Israeli Science Foundation ISF, grant # 870/11 the source coding case, the tpe of the best codeword in the above sense amounts to a single iteration of the Arimoto error-eponent computation algorithm 4]. Thus, b iteration of the NTS step we obtain a sequence of input distributions, Q, Q, Q,..., that converges to Q, the input that maimizes the Gallager function ρ,q. B a proper selection of the parameter ρ, this input leads to the optimum random-coding error eponent ER at a coding rate R, and to the channel capacit C in the limit as ρ 0. Section II gives some background on the Arimoto algorithm and presents our main result. Section III proves the main result. Discussion is given in Section IV. II. BACKGROUND AND STATEMENT OF RESULT In our search for a dual phenomenon in channel coding we consider an alternating maimization algorithm for random coding eponent computation, discovered b Arimoto in ]. This algorithm is a refinement of Arimoto s earlier work from 1972 on capacit computation 5] and Blahut s work 3] on rate-distortion function and capacit computation. A. Capacit and Error Eponent Consider a discrete memorless channel, with a transition probabilit distribution P from the input to the output. Let Q denote an arbitrar probabilit assignment on the channel input alphabet. It is well known, 6], 7], that the capacit C of this channel is given b maimizing the mutual information IQ,P over the input Q: C = maiq,p, 1 Q where IQ,P = ] P QP log Q P. 2 Furthermore, for an coding rate R < IQ,P, if we generate M = 2 nr length-n codewords b an i.i.d. distribution Q, then the error probabilit in maimum likelihood decoding is bounded from above b for all 0 ρ 1, where P e ep n ρ,q ], 3 ρ, Q = log QP 1 ] 4

2 is the Gallager function defined in 6, eq ]. The optimum value of the parameter ρ that maimizes the error eponent in 3 decreases as the rate R increases, and vanishes as R approaches IQ,P. In particular, IQ,P = lim ρ 0 ρ,q/ρ. For a fied value of the parameter ρ, the channel input distribution that maimizes the Gallager function 4, Q ρ = argma ρ,q, 5 Q maimizes also the error eponent in 3. 2 It follows that Q ρ Q C, as ρ 0, where Q C is the capacit-achieving input distribution 1. Unfortunatel, however, ecept for smmetric channels where the optimum input is uniform 6], these optimum distributions do not have a closed-form solution. This motivates the Arimoto and Blahut iterative algorithms 5], 3], 4]. B. Arimoto s Alternating Maimization Instead of maimizing ρ, Q directl over Q which ma be a hard task, the Arimoto algorithm alternates between two maimization steps. To this end, the Gallager function 4 can be rewritten as 4] log = log QP 1 ] = ] ρ Φ QP, 6 Q where Φ achieving the equalit in 6 is a transition probabilit matri given b Φ QP 1 Q P 1. 7 If we keep Φ frozen and maimize the right hand side of 6 with respect to the probabilit vectorq, the maimizing distribution is given b 4] Q = 1 1/ρ P Φ ] ρ, 8 K where K is the normalizing constant that keeps Q = 1. The epression 7 now contains the previous distribution Q. Plugging 7 into 8 we get a recursive epression for the new distribution in terms of the previous one: Q = 1 ρ ] 1/ρ K Q P 1 Q P 1. 9 At this point we have completed onl one of the two maimization steps of the Arimoto algorithm and it is still unclear whether the new distribution Q, determined b 9, does not result in a lower value of the Gallager function 2 For a non-redundant channel input alphabet, the optimum input Q ρ is unique 6, chap. 4.5, corr. 3]. ρ, Q. As a net step, we observe that the right hand side epression in the definition 7 has an etra meaning of the maimizer of 6 as a function of a dumm transition probabilit matri Φ. Therefore, replacing Q with Q in both 7 and 6 we get ρ, Q and ascertain that ρ, Q ρ, Q. Furthermore, as shown in 4], an iterative application of the recursion 9, starting from an arbitrar nonzero probabilit vector Q 1, produces a monotonicall converging sequence ρ, Q k ր ma Q ρ, Q and Q k Q ρ, as k. Note that the limit of 9 asρ 0 corresponds to computing the capacit C. Specificall, b L Hopital s rule, 9 becomes Q = 1 K Qep P log P Q P. 10 which is one iteration in computing the capacit-achieving distribution in Arimoto s original algorithm from ]. Our goal is to find a stochastic mechanism which produces a new tpe of the form 9. Gallager s upper bound on error probabilit seems to be a relevant tool for this purpose. C. Conditional Gallager Eponent for a Given Codeword Suppose that M messages are used to communicate through an n-dimensional channel P not necessaril memorless. Each message m, 1 m M, is represented b a codeword m of length n, selected independentl with the probabilit measure Q. Let m be a fied channel input selected to represent a certain message m. Given this m, consider the probabilit to get an error at the decoder, after we have independentl generated according to Q the codewords for each of the M 1 alternative messagesm m and have sent the message m through the channel. This probabilit, denotedp e,m, can be bounded from above as P e,m M ρ ρ P 1 sρ m QP ] s, 11 where s > 0 and 0 ρ 1. This is the Gallager bound 6, eq ], ecept for the averaging over the transmitted word m. For the rest of the paper, we assume that both the probabilit measure Q and the channel P are memorless: Q = Q k, P = P k k. In this case, the conditional bound 11 depends onl on the tpe of the fied word m, and takes the form shown b the following lemma, which is proved in the Appendi: Lemma 1 Conditional Gallager eponent: For memorless Q and P, and a transmitted codeword m, P e,m ep n s,m ], 12

3 where R 1 n logm is the coding rate, Q m is the tpe of m, and s,ρ,q, Q Qlog P 1 sρ Q P s. 13 It can be shown that the bound 12 can be eponentiall tight for a certain choice of the parameters s and ρ, which is determined b the tpe Q m and the code rate R. Nevertheless, to enhance the relation to Gallager s analsis 6], we shall restrict attention to the generall suboptimal choice of s = 1, and use the simplified notation ρ,q, Q s = 1, ρ, Q, Q = Qlog ρ P 1 Q P 1 ]. 14 The conditional Gallager function m plas a similar role to the Gallager function ρ, Q, conditioned on a transmitted codeword of tpe Q m. In fact, it is eas to show from 4 and 14 that where ρ, Q = min Q D Q Q = ρ,q, Q+D Q Q Q log ] Q Q, denotes the divergence Kullback-Leibler distance between the distributions Q and Q. The relation 15 can be eplained b averaging 12 over the transmitted words m, and noting that the frequenc of codewords of tpe Q in a random code generated b a distribution Q is proportional to ep nd Q Q ; see 7]. Figure 1 shows the steps of the Arimoto algorithm with respect to both the conditional and unconditional Gallager functions, for a binar smmetric channel BSC0.2, at ρ = 0.1, starting from an initial input distribution Q 1 = 0.1,0.9, and converging to Q = 0.5,0.5. D. Statement of Main Result Let X m + m=1 be an infinite list of codewords of length n, selected independentl with the probabilit measure Q. This list can be viewed alternativel as a sequence of nested random codebooks of size M, C M X 1,X 2,...,X M, i.e. growing in size with each codeword. Let Q m denote the tpe of the codeword X m. We define the codeword with the maimum conditional Gallager eponent in two steps: 1 In each codebook C M choose the codeword with the maimum conditional Gallager eponent 12, at s = 1, with respect to this codebook s size M. eponent E ρ, Q, Q 0 k k ρ, Q* ρ, Q k iteration k Fig. 1. Steps of the Arimoto algorithm. 2 Among all the codewords chosen in 1 one for each M = 1,2,3,... choose the codeword which has the maimum such eponent, which corresponds to ma M ma m ρ logm n. 17 Note that implicit in the definition 17 is that each codeword competes at least with all its predecessors in the list, which means that the conditional Gallager eponent is maimized onl among the feasible codewords. Denote the inde of the codeword determined b 17 as N n. Then the tpe Q Nn of the codeword X Nn is close to Q : Theorem 1 Favorite tpe: Q Nn Q in prob. 18 n where Q is the maimizing distribution 9 in an iteration of the Arimoto algorithm. III. PROOF OF MAIN RESULT The following lemma gives an alternative optimization meaning to the maimizing distribution 9 in one iteration of the Arimoto algorithm. As we shall see later, this lemma serves as a basis for our result. Lemma 2 Standard optimization: Q = argma E0 ρ,q, Q ρ D Q Q = Q arg ma ρ,q, Q, 19 Q:D Q Q DQ Q where Q is the maimizing distribution 9 in an iteration of the Arimoto algorithm. Since ρ and Q are fied, in the remainder of this section we shall use the simplified notation EP ρ,q,p, and also R M 1 n logm. The P here stands for an arbitrar

4 channel input, and should not be confused with the channel transition distribution. Proof: Since EP is a linear function ofp and DP Q is a conve function of P, the following epression is a concave function of P : LP = EP ρ DP Q DQ Q ] + λ P 1, 20 for an ρ > 0 and λ. Differentiating 20 and equating the derivative to zero, we obtain the maimizing distribution of the LHS of 19: L P = log P 1 Q P 1 ρlog P Q ρ + λ = 0 P = 1 K Q P 1 Q P 1 ρ ] 1/ρ = Q. Observe that P = Q is also the maimizer of EP under the constraint DP Q DQ Q. Indeed, suppose there eist a better distribution P, such that E P > EP and D P Q DQ Q, but then also L P > LP, since ρ > 0, which is a contradiction. The ke idea of the proof of Theorem 1 is to compare upper and lower bounds on the maimum ma m E Qm m, 21 which is equivalent to the double maimum in 17, as shown below. Note that 21 is a random variable, because the tpe of the m-th codeword is random. As a result of the remaining lemmas, the upper bound on 21 becomes a function of the random tpe Q Nn, whereas the lower bound becomes a tight deterministic bound. We start with a pair of bounds which are valid with probabilit 1: Lemma 3: E Q Nn Nn for all M. ma E Qm M, 22 Proof: E Q Nn Nn = ma E Qm m m = ma ma E Qm M m M m = ma ma E Qm M M ma E Qm M, for all M. Aiming at Lemma 2, in a somewhat smmetric manner we would like to replace ma E Q m on the RHS of 22 with EQ, and R Nn on the LHS of 22 with D Q Nn Q. Both of these substitutions can be accomplished onl in probabilit, using properties of tpes, as the net two lemmas show. Lemma 4: Choose M n = epndq Q. Then for an ǫ > 0 and n sufficientl large, with high probabilit the RHS of 22 is further bounded from below as ma n E Qm Mn ma P:DP Q DQ Q EP Mn ǫ. Proof: Given a list of M n words, generated independentl with the probabilit measure Q, the tpes Q satisfing DQ Q < DQ Q will be found in the list with high probabilit. On the other hand, the tpes satisfing DQ Q > DQ Q with high probabilit will not be found in the list. If we choose the word which has the highest EQ in the list, as the RHS of 22 suggests, this procedure translates into the following optimization problem: maep subject to DP Q DQ Q. Equivalentl, for an ǫ > 0 and n sufficientl large, with high probabilit ma E Qm ma EP ǫ. n P:DP Q DQ Q Lemma 5: For anǫ > 0 andnsufficientl large, with high probabilit the LHS of 22 is further bounded from above as E Q Nn Nn E Q Nn ρd QNn Q +ǫ. Proof: Observe that the codeword X Nn has the minimal inde among the codewords of the tpe Q Nn in the list. For an given tpe Q define a random variable R Q min Q m =Q Rm, which corresponds to the inde of the first instance of the tpe Q in the list. Then Pr R Nn D Q Nn Q ǫ Pr R Q D Q Q ] ǫ Q Q Pr R Q D Q Q ǫ Q e n D Q Q ǫ e nd Q Q = Q e nǫ = e nǫ+o1.

5 Which implies that with high probabilit R Nn D Q Nn Q ǫ, and the lemma follows. Combining lemmas 3, 4, and 5, we obtain E Q Nn ρd QNn Q EP Mn 2ǫ. 23 ma P:DP Q DQ Q Since R Mn DQ Q as n, b Lemma 2 ma EP Mn P:DP Q DQ Q n Therefore, we can rewrite 23 as ma P EP ρdp Q. E Q Nn ρd QNn Q ma EP ρdp Q 2ǫ. P B continuit, appling Lemma 2 again, we have 18. IV. DISCUSSION It follows from 15, 19, the non-negativit of the divergence 7], and since Q ρ is the unique fied point of the Arimoto algorithm, that E0 ρ,q, 24 with equalit if and onl if Q is the optimum input Q ρ in 5. In particular, it can be shown directl with the help of Jensen s inequalit that ρdq Q ρ, Q. 25 Therefore, the tpe Q gives a conditional error eponent 12 which is higher than the unconditional error eponent ρ, Q b a gap of at least ρdq Q. Our results suggest the eistence of an interesting phenomenon that the lowest individual error probabilities in random codebooks occur not at the lowest rates, but as a result of a trade-off between the probabilities of good tpes and the number of competing codewords. When this trade-off is combined with the conditional Gallager eponent, we obtain a better tpe for random codebook generation, which surprisingl coincides with the outcome of the Arimoto algorithm and thereb guarantees convergence to the optimal random codebook distribution, for a given channel. One difficult in such an input adaptation scheme, is that it relies on channel knowledge, since the conditional Gallager eponent assumes maimum likelihood decoding. Furthermore, the optimum selection of the parameter ρ depends on the channel and the coding rate. Another issue is that the sstem requires estimation of the error eponent of each codeword, which is possible onl after man transmissions. And another weakness of the current analsis is that it assumes working at a fied rate R < C. So the current result is still far from our final goal of a universal adaptive scheme for channel input adaptation, that approaches the true channel capacit C. Some modification of the analsis is required to support a mechanism which is completel independent of the channel, and onl relies on tentative estimates of codeword goodness at the receiver, and a feedback to the transmitter. This future work will be reported elsewhere. Proof of Lemma 1 APPENDIX For memorless Q and P the bound 11 becomes M ρ P 1 sρ m QP s = M ρ P 1 sρ m Q k P s k k = M ρ P 1 sρ m = M ρ P 1 sρ k m,k n = M ρ P 1 sρ m,k QP s k QP s k QP s = M ρ ep n 1 n log P 1 sρ n m,k ] ρ QP s = M ρ ep n Q m log P 1 sρ ] ρ Q P s = ep n s,m ]. ACKNOWLEDGEMENT We thank Yuval Kochman for interesting discussions in the beginning of this research, and Nir Weinberger for helpful comments. REFERENCES 1] R. Zamir and K. Rose, Natural Tpe Selection in Adaptive Loss Compression, IEEE Trans. on Information Theor, vol. 47, no. 1, pp , Jan ] Y. Kochman and R. Zamir, Adaptive Parametric Vector Quantization b Natural Tpe Selection, in Proceedings of the Data Compression Conference, Snowbird, Utah, Mar ] R. Blahut, Computation of Channel Capacit and Rate-Distortion Functions, IEEE Trans. on Information Theor, vol. 18, no. 4, pp , Jul ] S. Arimoto, Computation of Random Coding Eponent Functions, IEEE Trans. on Information Theor, vol. 22, no. 6, pp , Nov ], An Algorithm for Computing the Capacit of Arbitrar Discrete Memorless Channels, IEEE Trans. on Information Theor, vol. 18, pp , Jan ] R. Gallager, Information Theor and Reliable Communication. John Wile & Sons, ] T. M. Cover and J. A. Thomas, Elements of Information Theor. New York: Wile, 1991.