Upper Bounds to Error Probability with Feedback

Transcription

1 Upper Bounds to Error robability with Feedbac Barış Naiboğlu Lizhong Zheng Research Laboratory of Electronics at MIT Cambridge, MA, {naib, lizhong Abstract A new technique is proposed for upper bounding the error probability of fied length bloc codes with feedbac. Error analysis is inspired by Gallager s error analysis for bloc codes without feedbac. Zigangirov-D yachov encoding scheme is analyzed with the technique on binary input channels and -ary symmetric channels. I. INTRODUCTION Shannon showed, [6], that capacity of the discrete memoryless channels DMCs) does not increase with feedbac. Later it was shown that the eponential decay rate of the error probability of fied length bloc codes can not eceed sphere pacing eponent even when there is feedbac, first by Dobrushin, [3], for symmetric channels and later by Sheverdyaev, [7], for general DMCs. In other words for the rates above the critical rate even the error eponent does not increase with feedbac, when we restrict ourselves to the fied length bloc codes. Characterizing the improvement in the error eponent for the rates below the critical rate is the most pressing open question in this stream of research. The first wor on the error analysis bloc codes with feedbac was by Berleamp []. He has obtained a closed form epression of the error eponent at zero rate for binary symmetric channels BSCs). Later Zigangirov [8] proposed an encoding scheme, for BSCs which reaches sphere pacing eponent for all rate over a critical rate R Zcrit. 2 Furthermore at zero-rate Zigangirov s encoding scheme reaches optimal error eponent at zero rate, which is derived by Berleamp in []. Later D yachov, [4], proposed a generalization of the encoding scheme of Zigangirov, and obtained a coding theorem for general DMCs. However the optimization problem in his coding theorem, is quite involved and does not allow for simplifications that will lead us to conclusions about the error eponents of general DMCs. In [4] after pointing out this fact, D yachov focuses on binary input channels and -ary symmetric channels and derives the error eponent epressions for these channels. There are a number of closely related models in which error eponent analysis has been successfully applied, lie variable-length bloc codes, fied length bloc codes with errors-and-erasure decoding, bloc codes on additive white Gaussian noise channels, fied/variable delay code on DMCs. We are refraining from discussing these variants mainly because understanding those variants will not help the reader much in understanding the wor at hand. 2 Evidently R Zcrit < R crit where R crit is the critical rate in the nonfeedbac case, i.e. the rate above which random coding eponent is equal to the sphere pacing eponent. Our approach will be similar to the D yachov s in the sense that we will first prove a coding theorem for general DMCs and then focus on particular cases and demonstrate its gains over non-feedbac encoding schemes. We will start with introducing the channel model we have at hand and the notation we use. After that we will consider the encoding schemes that uses feedbac and mae an error analysis which is very similar to the that of Gallager in [5]. Then we will use our results in different cases, to recover the results of Zigangirov [8] and D yachov [4] for binary input channels and to improve D yachov s results, [4], for -ary symmetric channels. 3 II. CHANNEL MODEL AND NOTATION We have a discrete memoryless channel with input alphabet X {, 2,..., X }, output alphabet Y {, 2,..., Y }. Channel transition probabilities are given by a X -by- Y matri W y ). In addition we assume that a noiseless, delay free feedbac lin eists from transmitter to the receiver. Thus transmitter learns the channel output at time before the transmission of the symbol at time +. A length n bloc code with feedbac for a messages set M {, 2,..., e nr } is a feedbac encoding scheme together with a decoding rule. The feedbac encoding scheme, Ψ, is a mapping from the set of possible output sequences, y j Y j for j {, 2,..., n}, to the set of possible input symbol assignment to the messages in the set M n Ψ : Y j X ) j The input letter for the message m M at time j given y j Y j is the m th element of Ψy j ), i.e., Ψ m y j ). Note that when there is no feedbac Ψy j ) Ψ j, y j Y j. The probability of observing a y i Y i conditioned on message m M is, { y i θ m } i W y j Ψ m y j )) j A decoding rule is simply a mapping from the set of all possible length n output sequences Y n to the message set 3 Indeed same improvement can be obtained within framewor of the analysis introduced by Zigangirov and D yachov with some fairly minor modifications.

2 M. Φ : Y n M We use a maimum a-posteriori probability MA) decoder. When there is a tie the message with the smaller inde is decoded. III. ERROR ANALYSIS The error probability of the message m M is, e,m {y n θ m} I{Φy n ) m} where I{ } is the indicator function. Note that for a MA decoder, for any ρ > 0, and η > 0 we have I{Φy n ) m} m {yn θ } η {y n θ } η Consequently e,m {y n θ m} {y n θ } η m If we introduce the short hand 2 3 Γ ρ,η,jy j ) X ny j o θ m 4 X ny j o η θ 5 m m M and recall that e m e,m e, we get, ) ρ 2) If Γ ρ,η,n y n ) 0 we can divide and multiply by Γ ρ,η,n y n ), i.e, e Thus e y n Y n y n Y n y n Y ξρ, η, y j, Ψ) 3) where ξρ, η, y n, Ψ) is given by, { 0 if Γρ,η,n y n ) 0 ξ y n Y if Γ ρ,η,n y n ) 0 Let us define αρ, η, Ψ) as αρ, η, Ψ) ma j {,...n} ma ξρ, η, y j Y yj, Ψ) 4) j ρ ρ Then e y n Y n Γρ,η,0y0 ) αρ, η, Ψ) n M ρ αρ, η, Ψ) n αρ, η, Ψ) e nρr+ln αρ,η,ψ)) 5) Now what we find an encoding scheme, Ψ, with small αρ, η, Ψ). For that we will focus on the encoding schemes that are repetitions of the same one step encoding function. IV. ONE STE ENCODING FUNCTIONS Let Q be the set -dimensional vectors whose all entries are non-negative. Let Υ be a parametric function, Υ : Q X R If q has at least two non-zero entries then Υ is defined as Υ ρ,η q, χ) y,m else Υ ρ,η q, χ) 0. Let us introduce the short hand, then we have W y χ m) ) η qm m W y χ ) η q ) ρ ϕm y j ) { y j θ m } η m ) ρ 6) ϕm y j ) W y j Ψ m y j )) η ϕm y j ) We can write ξ in terms of Υ as follows, ξρ, η, y j, Ψ) Υ ρ,η ϕ y j ), Ψy j )) 7) Note that ξρ, η, y j, Ψ) depends on the encoding in first j time units only through the dimensional vector ϕ y j ). Thus if we can find a χ q for each q Q such that Υ ρ,η q, χ q ) is small, we can use it to obtain a bloc code with small error probability. These mappings are what we call one step encoding function. An dimensional one step encoding function, X, is a mapping from Q to the X, i.e., We define αρ, η, X) as αρ, η, X) X : Q X ma Υ ρ,η q, Xq)) 8) q Q Lemma : For any ρ > 0, η > 0, dimensional one step encoding function X, and for any n there eists and bloc code such that e e nln αρ,η,x)+ρr) 9) where R n roof: Consider the encoding scheme, Ψ such that, Ψ m y j ) X m ϕ y j )) 0)

3 As a result of equations 4), 7) and 8) we get, αρ, η, Ψ) αρ, η, X) ) Using equation 5) we get the claim of the lemma. When we are calculating the achievable error eponents the role of the minimum of ln αρ, η, X) will be very similar to that of E 0 ρ) in the case without feedbac, [5]. V. EXAMLES: A. Achievability of Random Coding Eponent: In this subsection we will, as a sanity chec, re-derive the achievability of random coding eponent for all DMC using Lemma. Let η +ρ. For any >, at each q Q consider the set of all possible mappings of messages to the input letters, X, and calculate the epected value of Υ ρ,η q, χ), where the probability of each χ X is simply given by X )rχ,) where rχ, ) is the number of messages assigned to input letter X. Then one can show that [ ] E Υ ρ, q, χ) e E0ρ, ) q Q 2) +ρ where E 0 ρ, ) y W y ) +ρ )) +ρ Thus for each q Q there eist at least one χ q X such that Υ ρ, +ρ q, χ q) e E0ρ, ) 3) Thus for the one step encoding function Xq), such that Xq) χ q we have ln αρ, +ρ, X) E 0ρ, ) Using this together with the lemma we can conclude that: Corollary : For any input distribution ) on X, and ρ 0, ], R 0 any n > there eists a length n bloc code of the form given in equation 0) such that. e e n E0ρ, )+ρr) 4) Note that above description is not constructive in the sense that it proves the eistence of a one-step-encoding scheme, X with the desired properties but it does not tell us how to find it. Encoding scheme we will investigate below however does specify a X with the desired properties. B. Zigangirov-D yachov Encoding Scheme: In this subsection we will describe the Zigangirov-D yachov encoding scheme and apply Lemma to this encoding scheme on binary input channels and -ary symmetric channels. This encoding scheme was first described by Zigangirov, [8], for binary symmetric channels than generalized by D yachov to general DMCs. Consider a probability distribution ) on input alphabet X and a q Q. Without loss of generality we can assume that 4 i, j M, if i j then q i q j. Now 4 If this is not the case for a q, we can rearrange the messages m M, according to their q m in decreasing order. If two or more messages have same mass, q, we order them with respect to their inde numbers. we can define mapping χ for a given q and iteratively as follows: γ 0 ) 0 χ j arg min γ j ) supp ) i j:χ i q i γ j ) ) For assigning j M we first calculate for each input letter, X, the total mass of all of the messages that has already been assigned to, γ j ). Then we divide γ j ) s by the corresponding ) values and assign the message j M to the X, for which ) > 0 and γj ) ) is the minimum. If there is a tie we choose the input letter,, with larger ). If there is still a tie, we choose the input letter with smaller inde. ) roperties of Z-D Encoding Scheme: Now let us point out one property of this encoding scheme which will become handy later on. A Zigangirov-D yachov encoding scheme with ), will satisfy, ζ m qχm qm χ m) q ) X m M 5) where q γ,, χ ). In order to see this, simply consider the last message assigned to each input letter X. They will satisfy this property by construction. Since the messages that are assigned to the same letter prior to the last message should have at least the same mass as the last one, they will satisfy the property given in 5) too. Thus for any q Q and any input distribution ), the mapping created by a Zigangirov-D yachov encoding scheme, satisfies q )ζ m 0 X m M 6) In other words, with Zigangirov-D yachov encoding scheme, the mass of the q is distributed over the input letters in such a way that; when we consider all the mass distribution ecept an m M, it is a linear combination of ) and δ, s for χ m. Thus m I{χ }q ζm ) m q + χm I{χ }q ζ m )) For ρ, z ρ is a conve function of z thus E [z] ρ E [z ρ ], thus [ ] ρ m W y χ ) η q ) ρ W y ) η ) Let us define f m as f m y ζ m + χ m W y χ m ) ) q ζ m )) m q W y ) ηρ 7) m W y χ ) η q m q ) ρ

4 Using equation 7) we get, f m ζ m + χ m ζ m [ W y χ m ) ) W y ) η ) y W y χ m ) ) W y ) ηρ q )ζ m),ρ,η) eβχm ma{βχm,ρ,η),µχm ρη)} e y m ζm ρη) q )eµχm where i X, β i, ρ, η) and µ i ρη) are defined as, β i, ρ, η) ln X y W y i) ) µ iρη) ma i, X ln X y Consequently for ρ, η 0, Υ ρ,η q, χ )! ρ X )W y ) η W y i) ) W y ) ρη ] ρ y,m W y χm)) η qm m W y χ ) η q ) ρ m ) ρ m ) ρ f m m ) ρ e ma supp ) ma{β,ρ,η),µ ρη)} Thus for ρ for all input distributions and for all η > 0, ln αρ, η, X ) ma ma{β, ρ, η), µ ρη)} 8) supp ) For certain channels property given in equation 8) together with Lemma implies that sphere pacing eponent is achievable on an interval of the form [R Dcrit, R crit ]. Corollary 2: If for a DMC, ma µ ρ X +ρ ) E 0ρ) 9) on an interval of the form ρ [, ρ Dc ]. Then ln αρ, +ρ, X) E 0ρ) ρ [, ρ Dc ] roof: In order to see this, first recall that for the ) that maimizes E 0 ρ, ), i.e. satisfying we have, ma E 0ρ, ) E 0 ρ, ) E 0 ρ, ) β, +ρ, ρ) supp ) Now the statement simply follows the equation 8) Recall that sphere pacing eponent is given E sp R) ma ρ 0 E 0ρ) ρr Thus for each R there is a corresponding ρ R, which is the maimizer of the the epression above. As a result of Lemma and Corollary2 for the rates with ρ R [, ρ Dc ] sphere pacing eponent will be achievable as error eponent for the channels satisfying condition given in equation 9). Two family of channels that satisfy the condition 9) are Binary input channels and -ary symmetric channels. 2) Binary Input Channel: The binary input channel case have also been addressed by D yachov. We simply rederive his results here. A binary input channel is DMC for which X {0, } For ρ, η 0, using equation 8) we get, αρ, η, X) e ma{β0,η,ρ),β,η,ρ),µ0ρη),µρη)} 20) For the rates for which ρ R [, ρ Dc ] this will lead to sphere pacing eponent. For the rates for which ρ R > ρ Dc we will simply minimize the epression in equation 20) over,η and ρ to find the best possible error eponent achievable with this scheme. For the rates such that ρ R 0, ) one can also show that sphere pacing eponent is achievable. For that one needs to consider, f m X y X y W y χ m) ) m W y χ ) η q W y χ m) ) z mw y χ ) η + z m)w y χ ) η ) ρ It can be shown that, for ρ < and η <, z m f m 0. Furthermore as a result of equation 6) we now that z m χ m ). Thus for ρ <, η <, βχm,ρ,η) f m e αρ, η, X) e ma{β0,η,ρ),β,η,ρ)} If we consider that is the maimizer of E 0 ρ, ), we get ) ρ ln αρ, +ρ, X) E 0ρ) Recalling the definition of ρ R and E sp R) we conclude that sphere pacing eponent is achievable for R such that ρ R in0, ). 3) -ary Symmetric Channel: Let us consider -ary symmetric channel with <, i.e. { i j wi j) i j Note that for any ρ > 0, η > 0 and we have µρη) µ ρη) ) ) ) ) ρη + ) ) ) ρη + 2) Furthermore note that for any ρ > 0, η > 0, X and for ) / βρ, η) β, ρ, η) [ ) + ) ) ] )η + )η ) ρ Thus as a result of equation 8), for ρ αρ, η, X) ma{βρ, η), µρη)} 2) For 2 case these epressions are equivalent to those of Zigangirov in [8], which were specifically derived for BSCs. For 3, these epressions lead to αρ, η) s that are strictly

5 better than the corresponding quantities in [4] for all ρ >. Figure?? shows the resulting error eponents for a particular channel. In [4], equivalent of equation 2), has β D ρ, η) instead of βρ, η) where, β D ρ, η) ) )η + )η ) ρ + ) ) )ηρ + )ηρ ) Since the function z ρ is strictly conve in z for ρ >, βρ, η) > β D ρ, η) for all ρ > and η > 0. 4) Remars on Z-D Encoding Scheme: The two eample channels we have considered does not reveal the main weaness of the Zigangirov-D yachov encoding scheme. Consider the following four-by-four symmetric channel wy ), W y ) The error eponent of this channel is equal to sphere pacing eponent at all rates. In order to see why, consider any rate R > 0. We already now that R > 0 there is a fied list size L R such that: for any n there is a length n code with decoding list size L R whose error probability satisfies e e EspR)n. Using log 2 L R etra channel uses transmitter can specify the correct message among the decoded list, if the correct message is in the list. Since this etra time will become negligible as n increases we can conclude that sphere pacing eponent is achievable for this channel, when there is feedbac. However when we consider the Zigangirov-D yachov encoding scheme we can not reach the same conclusion. Clearly we should choose the ) to be the uniform distribution. Consider for eample the q, such that q [ ] any smart encoding scheme would assign the first two messages into input letters that are not consecutive. But Zigangirov- D yachov encoding scheme will not necessarily do so. Considering similar q s for low enough ρ s one can show that Z-D encoding scheme leads to an αρ, +ρ ) such that ln αρ, +ρ ) > E 0ρ). As a result this will imply that Z-D encoding scheme will have an error eponent strictly less than the sphere pacing eponent. On the other hand for all such anomalies we observed, we were also able to find a modification of Z-D encoding scheme, which employs a smarter encoding scheme for the first X messages, and which performs optimally. However we have yet to find a general modification on Z-D encoding scheme that wors for all q Q. analysis does help us to see how to generalize the Zigangirov- D yachov encoding scheme to the general DMCs. On a seperate note Burnashev, [2], considered the binary symmetric channels and showed that for all the rates between 0 and R Zcrit one can reach and an error eponent strictly higher than the ones given in [8]. Indeed a similar modification are possible within our frame wor too. We will present those in our journal paper on this subject. REFERENCES [] E. R. Berleamp. Bloc Coding with Noiseless Feedbac. h.d. thesis, Massachusetts Institute of Technology, Department of Electrical Engineering, 964. [2] M. V. Burnashev. On the reliability function of a binary symmetrical channel with feedbac. roblemy erdachi Informatsii, 24, No. :3 0, 988. [3] R. L. Dobrushin. An asymptotic bound for the probability error of information transmission through a channel without memory using the feedbac. roblemy ibernetii, vol 8:6 68, 962. [4] A. G. D yachov. Upper bounds on the error probability for discrete memoryless channels with feedbac. roblemy erdachi Informatsii, Vol, No. 4:3 28, 975. [5] R.G. Gallager. A simple derivation of the coding theorem and some applications. Information Theory, IEEE Transactions on,, No. :3 8, 965. [6] C. Shannon. The zero error capacity of a noisy channel. IEEE Transactions on Information Theory, Vol. 2, Iss 3:8 9, 956. [7] A.Yu. Sheverdyaev. Lower bound for error probability in a discrete memoryless channel with feedbac. roblemy erdachi Informatsii, 8, No. 4:5 5, 982. [8]. Sh. Zigangirov. Upper bounds for the error probability for channels with feedbac. roblemy erdachi Informatsii, Vol 6, No. 2:87 92, 970. VI. CONCLUSIONS AND FUTURE WOR A part from the minor improvement for -ary symmetric channels our results are essentially a rederivations of the results of Zigangirov, [8] and D yachov, [4]. However simplicity of the