The error exponent with delay for lossless source coding

Transcription

1 The error eponent with delay for lossless source coding Cheng Chang and Anant Sahai Wireless Foundations, University of California at Berkeley Abstract In channel coding, reliable communication takes place at rates below capacity at the fundamental cost of end-toend delay. Error eponents tell us how much faster convergence is when we settle for less rate. For lossless source coding, entropy takes the place of capacity and error eponents tell us how much faster convergence is when we use more rate. While in channel coding without feedback the block error eponent is a good proy for studying the more fundamental tradeoff with fied end-to-end delay, it is not so in source coding. Block-coding error eponents are quite conservative (despite being tight!) when it comes to the tradeoff with delay. onblock codes can achieve much better performance with fied delay and we present both the fundamental bound and how to achieve it in a delay-universal manner. The proof gives substance to Shannon s cryptic statement about how the duality between source and channel coding is like the duality between the past and the future. I. ITODUCTIO AD POBLEM SETUP Shannon closed his seminal paper on lossy source coding [] with an intriguing comment: [The duality between source and channel coding] can be pursued further and is related to a duality between past and future and the notions of control and knowledge. Thus we may have knowledge of the past and cannot control it; we may control the future but have no knowledge of it. While there has been much work eploring the relationship between source and channel coding, no connection to the past and the future has emerged so far. In this paper, we demonstrate such a connection by looking at the fundamental error eponent with respect to fied delay between the time a message is first made known to the encoder and its deadline at the decoder. For channel coding without feedback, the dominant error event is caused by the channel s future behavior in that errors can be forced by mild channel misbehavior during the period between the message arrival time and its deadline [2], [3]. This sort of mild misbehavior is eactly what is bounded by the traditional sphere-packing arguments for block coding. When feedback is allowed, the encoder can do flow-control and become robust to such mild misbehavior even when facing fied-delay deadlines [4], [3]. The fundamental limits are instead given by the focusing bound [3]. This paper develops a lossless source-coding counterpart to the focusing bound. This reveals that the dominant source of errors is the past misbehavior of the source, over which the encoder has no control. Since the communication medium is a noiseless fied-rate bit-pipe, the future is entirely under the control of the encoder. This bound is also asymptotically achievable using fied-to-variable codes whose variable-rate nature is smoothed out by using a queue. A. eview of block source coding results The discrete memoryless source S generates iid random variables i from a finite alphabet according to distribution p. Without loss of generality, assume p () > 0,. A rate block source coding system for n source symbols consists of a encoder-decoder pair (E n, D n) where E n : n {0, } n, D n : {0, } n n, E n( n ) = b n D n(b n ) = b n The probability of block decoding error is P ( n b n ) = P ( n D n(e n( n ))). Shannon proved that arbitrarily small error probabilities are achievable by letting n get big as long as the encoder rate is larger than the entropy of the source, > H(p ). Furthermore, it turns out that the error probability goes to zero eponentially in n. Lemma : (From [5]) For a discrete memoryless source p and encoder rate < log 2, ɛ > 0, K(ɛ) <, n 0, a block encoder-decoder pair E n, D n satisfying P ( n b n ) K(ɛ)2 n(e b() ɛ) This result is asymptotically tight, in the sense that for any sequence of encoder-decoder pairs E n, D n, () lim sup n n log 2 P ( n b n ) E b () (2) where E b () is defined as the block source coding reliability function with the form: E b () = min D(q p ) (3) q:h(q) Paralleling the definition of the Gallager function for channel coding [6], define E 0(ρ) = ( + ρ) log 2 [ Then, as mentioned as an eercise in [5]: p () +ρ ] (4) E b () = sup{ρ E 0(ρ)} (5) ρ0 = D(p ρ p ) where p ρ is the tilted distribution of parameter ρ satisfying H(p ρ ) =. The tilted distribution of parameter ρ (, ) of a distribution p is: p ρ () = p() +ρ Ps p(s) +ρ (6)

2 Source Encoding Decoding Time line b ( ) b 2 ( ) b 3 ( 2 ) b 4 ( 2 ) b 5 ( 3 ) b 6 ( 3 ) b 7 ( 4 ) b 8 ( 4 ) ˆ () ˆ 2 (2) ˆ 3 (3) ˆ 4 (4) Fig.. Delay-universal sequential source coding at = 2 H(p ρ ) ρ p 0 = p and as shown in [7], 0 and is generally strictly positive unless p is uniform on. Thus the optimal ρ corresponding to is unique unless p is uniform over which is a uninteresting case. B. Sequential Source Coding ather than being known in advance, the source symbols enter the encoder in a real-time fashion. We assume that the source S generates one source symbol per second from a finite alphabet. The j th source symbol j is not known at the encoder until time j. ate operation means that the encoder sends binary bit to the decoder every seconds. For obvious reasons, we focus on cases with H(p ) < < log 2. Definition : A sequential encoder-decoder pair E, D are sequence of maps. {E j}, j =, 2,... and {D j}, j =, 2,... The outputs of E j are the outputs of the encoder E from time j to j. E j : j {0, } j (j ) E j( j ) = b j (j ) + The outputs of D j are the decoding decisions of all the arrived source symbols by time j based on the received binary bits up to time j. D j : {0, } j D j(b j ) = b j d Where b j d (j) is the estimation, at time j, of j d and thus has end-to-end delay of d seconds. In a delay-universal scheme, the decoder emits revised estimates for all source-symbols so far. A rate = 2 delay-universal sequential source coding system is illustrated in Figure. For sequential source coding, it is important to study the symbol by symbol decoding error probability instead of the block coding error probability. Definition 2: A family of rate sequential source codes {(E d, D d )} are said to achieve delay-reliability E s() if and only if: i lim inf d d log 2 P ( i b i(i + d)) E s() II. UPPE BOUD O E s() To bound the best possible error eponent with fied delay, we consider a genie-aided encoder/decoder pair and translate the block-coding bounds of [5] to the fied delay contet. The arguments are analogous to the focusing bound derivation in [3] for the case of channel coding with feedback. Theorem : For fied-rate encodings of discrete memoryless sources, it is not possible to achieve an error eponent with fied-delay better than Es () = inf > E b() (7) Proof: For simplicity of eposition, we ignore integer effects arising from the finite nature of d,, etc. For every > and delay d, consider a code running over its fied-rate d noiseless channel till time. By this time, the decoder will have committed to estimates for the source symbols up to time i = d. The total number of bits used during this period is d. ow consider a genie that gives the encoder access to the first i source symbols at the beginning of time, rather than forcing the encoder to get the source symbols one at a time. Simultaneously, loosen the requirements on the decoder by only demanding correct estimates for the first i source symbols by the time d. In effect, the deadline for decoding the past source symbols is etended to the deadline of the i-th symbol itself. Any lower-bound to the error probability of the new problem is clearly also a bound for the original problem. Furthermore, the new problem is just a fied-length block-coding problem requiring the encoding of i source symbols into The rate per symbol is ( d) i = ( d) d = d bits. Theorem 2.5 in [5] tells us that such a code has a probability of error that is at least eponential in ie b (). Since i = d, this translates into an error eponent of at most E b () with parameter d. Since this is true for all >, we have a bound on the reliability function E s() with fied delay d: E s() inf > E b() The right hand side is defined to be Es (). The minimizing d tells how much of the past ( ) is involved in the dominant error event. This bound can be rewritten in terms of the ρ parameter to get a form paralleling the symmetric channel case from [3]. Corollary : where satisfies = E 0( ). Proof: For convenience, define E s() E 0( ) (8) G (ρ) = ρ E 0(ρ) (9) and notice that G ( ) = 0. As shown in [7]: dg (ρ) d 2 G (ρ) = H(p ρ ), 0 dρ dρ 2 G (0) = 0, G ( ) < 0 (0) 2

3 source symbol per second Block Encoder of length Fig. 2. G (ρ) Encoder Buffer (FIFO) Sequential Source Encoder bit per / seconds Decoder G (ρ) is a concave function as illustrated in Figure 2. Consider ρ 0 for which H(p ρ ) = 0. By concavity, we know that ρ <. Write: F (, ρ) = (ρ E0(ρ)) = ρ + G(ρ) Then for any (, log 2 ), F (,ρ) = H(pρ ) 2 F (, ρ), 0 ρ ρ 2 F (, 0) = 0, F (, ) < 0 () So for any (, log 2 ) let ρ() be so F (, ρ()) is maimized. ρ() is thus the unique solution to: Define = H(p ) = H(p ) = H(p ) H(p ρ() ) = 0 which satisfies H(p ) H(pρ ) = log 2 = Since H(p ) = 0, maimizes F (, ρ) over all ρ. Using (5) and the above analysis: E s() inf > = inf sup > ρ>0 = inf sup > ρ>0 E b() (ρ E0(ρ)) F (, ρ) sup F (, ρ) ρ>0 = F (, ) = (2) This proves the desired upper bound. III. ACHIEVABLE EPOETS To lower-bound the error eponent with delay, we give an eplicit fied-rate coding scheme that is a minor variation of the scheme analyzed in [8]. In [3], an achievable scheme was given for channel coding with feedback when there was an additional low-rate noise-free link that could carry flow-control information. In source-coding, there is just a noise-free link Fig. 3. Sequential source coding using a variable-length code and the flow control information is made implicit by using a prefi-free code. Just as in [3], the queuing behavior is what will determine the achieved delay error eponent. ather than repeating the from first principles derivation of [3], here we give an alternate derivation by applying Cramáer s theorem. A. A sequential variable length source coding scheme We are interested in the performance with asymptotically large delays d. A block-length is chosen that is much smaller than the target end-to-end delays, while still being large enough. For a discrete memoryless source and large blocklengths, the best possible variable-length code is given in [5] and consists of two stages: first describing the type of the block using O( log ) bits and then describing which particular realization has occurred by using a variable H( ) bits. The overhead O( log ) is asymptotically negligible and the code is also universal in nature. While the above code will also work, for simplicity of analysis, we instead consider a Shannon-code built for a particular tilted distribution for. Definition 3: The instantaneous code C,λ for λ > is a mapping from to a variable number of binary bits. C λ ( ) = b l( ) where l( ) is the codeword length for source sequence. The first bit is always and the rest of the codewords are the Shannon codewords based on the λ tilted distribution of p l( p ( ) +λ ) = + log 2 Ps p (s ) +λ 2 i= log 2 p ( i) +λ Ps p(s) +λ From (3), the longest code length l λ is l λ 2 log 2 p +λ min Ps p(s) +λ (3) (4) Where p min = min p (). The constant 2 is insignificant compared to. This variable-length code is turned into a fied rate code as follows: 3

4 Where the rate function is [9]: I() = inf D(ν p y ) ν: P Σ i= ν if(σ i )= ν is a distribution defined on Σ. It is shown in [9] that: Fig. 4. umber of bits in the buffer: t Definition 4: The sequential source coding scheme is illustrated in Figure 3. At time k, k =, 2,... the encoder E uses the variable length code C λ to encode the k th source block k = k (k )+ into a binary sequence b(k), b(k) 2,...b(k) l( k ). This codeword is pushed into a FIFO queue with infinite buffer-size. The encoder drains a bit from the queue every seconds. If the queue is empty, the encoder simply sends 0 s to the decoder until there are new bits pushed in the queue. The decoder knows the variable length code book and the prefi-free nature of the Shannon code guarantees that everything can be decoded correctly. B. Sequential error events The number of the bits k in the encoder buffer at time k, is a random walk process with negative drift and a reflecting barrier. An eample sample-path is illustrated in Figure 4. At time (k + d), the decoder can make an error in estimating k if and only if part of the variable length code for source block k is still in the encoder buffer. Since the FIFO queue drains deterministically, it means that when the k-th block s codeword entered the queue, it was already doomed to miss its deadline of d. Formally, for an error to occur, the number of bits in the buffer k d. Thus, meeting a specific end-to-end latency constraint over a fiedrate noiseless link is like the buffer overflow events analyzed in [8]. Define the random time t k to be the last time before time k when the queue was empty. A missed-deadline will occur only if P k i=t k + l( i) > (d + k). For arbitrary t k, define the error event k (t) = P ( l( i) > (d + )) i=t Using the following lemma, we will derive a tight, in the large deviation sense, upper bound on (t). Lemma 2: Cramáer theorem [9] Consider iid random variables Y i Σ = {σ,...σ Σ }, P Y i p y and f : Σ +, i = f(y i). Write S n = n n i= i. For any closed set F +. P (S n F ) (n + ) Σ 2 n inf F I() The initial is not really required since the decoder knows the rate at which source-symbols are arriving at the encoder. Thus, it knows when the queue is empty and does not need to even interpret the 0s it receives. Σ I() = sup{ρ log 2 ( ρ p yi 2 ρf(σi) )} i= Write I(, ρ) = ρ log 2 ( P Σ i= py i 2ρf(σ i) ), then > 0, ρ < 0, I(, ρ) < 0. Obviously I(, 0) = 0, which means that the ρ to maimize I(, ρ) is positive. This implies that I() is monotonically increasing with. Using the definition of l( ): log 2 ( p ( )2 λl( ) )} 2λ + log 2 [( p () +λ )( p () +λ ) λ ] = 2λ + ( + λ) log 2 [ p () +λ ] = 2λ + E 0(λ) (5) The 2λ is insignificant and so as a simple corollary of the Cramáer theorem, we have an upper bound on (t). (t) = P ( k Where: i=t+ = P ( λ l( i) (d + )) k i=t+ l( i) () 2 E (C λ,,k t,d) E (C λ,,, d) (d + ) () sup {ρ ρ + (d + ) ) log 2 ( p ( )2 ρl( ) )} (d + ) ()[λ log 2 ( p ( )2 λl( ) )] (d + ) ()(λ 2λ E0(λ)) = dλ + ()[G (λ) 2λ ] (6) C. Lower bound on E s() Theorem 2: For any ɛ > 0, by appropriate choice of, λ, it is possible to achieve an error eponent with delay of E 0( ) ɛ universally over large enough delays d for the satisfying = E 0( ). Proof: For the sequential source coding scheme of C,λ, the decoding error for the k th source block at time (k + d) is. t k is the last time before k when the buffer is empty. 4

5 = k P (t k = t, k P ( k i=t+ k i=t+ l( i) (d + )) l( i) (d + )) k ( + ) 2 E (C λ,,k t,d) k = 2 dλ ( + ) 2 (k t)[g (λ) λ ] The above equality is true for all, λ. Pick λ = ɛ. From the discussion at the end of previous section, we know that G (λ) > 0. Choose >. Then define K(ɛ,, ) = λ G (λ) (i + ) 2 i[g (λ) λ ] < i=0 which is guaranteed to be finite since dying eponentials dominate polynomials in sums. Thus: K(ɛ)2 dλ = K(ɛ,, )2 d(ρ ɛ) where the constant K(ɛ, ) does not depend on the delay in question. Since E 0( ) = and the encoder also is not targeted to the delay d, this scheme achieves the desired eponent delay-universally. This theorem effectively proves that Es () = E 0( ) is asymptotically achievable. For every realization of, the codelengths of this code differ from the optimal universal code by at most O(log()), which is insignificant compared to. Thus, this delay-eponent can also be attained universally over both discrete-memoryless sources and delays d. IV. A EAMPLE OF SUBOPTIMAL CODIG Large are not needed to do substantially better than blockcoding in the fied delay contet. The advantages of variablelength encoding are so great that even very simple VL-codes will outperform the best possible block-code. Consider the following ternary eample from [0]. Suppose p (a) = 0.9, p (b) = 0.05 and p (c) = Consider rate = 3. The best possible block coding error eponent is.474. Consider the following sub-optimal 2 variable length sequential coding scheme with = 2. Map (a, a) to 00, and other source symbol pairs are mapped to 000, 00,... Simple birth-death Markov chain analysis reveals that this achieves an error eponent of 6.27 with fieddelay despite being suboptimal even as a VL-code! V. COCLUSIOS AD FUTUE WOK The lossless source-coding version of the uncertainty focusing bound was developed and used to show that fied-tovariable length coding is optimal from an end-to-end latency point of view, even when the deadlines are specified in terms of a fied latency. In a very precise sense, lossless sourcecoding is like channel coding with feedback and using blockcodes results in a substantial loss in the error eponents. While the parametric form of the focusing bound is the same in the channel-coding and source-coding cases, the interpretation is a bit different. In source-coding, the dominant error events always involve only the past. As the rate varies, the only change is how much of the past is involved. For channel coding with feedback, both the past and the future are involved since the past is now known while the future remains only partially controllable. In channel-coding without feedback, only the future is involved since the past is entirely unknown. A similar story should hold for the error eponents of pointto-point lossy source coding. Once again, the source-coding contet guarantees that only the past behavior will matter in the dominant error event. The results here hint that VQ followed by variable-rate entropy-coding may also be optimal in the fied-delay setting. A more interesting direction is in etending our understanding of sequential distributed coding for correlated sources (Slepian-Wolf source coding). So far, we have only achievable eponents in general [7] and a good upper bound only for symmetric cases with side-information at the receiver []. For the case considered in [], it turns out that only the future behavior of the source matters. We suspect that depending on the rate point and the nature of the source, different combinations of the past and future will be involved in the dominant error event and the optimal codes will in general have a mied nature involving both queuing and binning. ACKOWLEDGMETS The authors thank Sanjoy Mitter, Tunc Simsek, Stark Draper, and Sandeep Pradhan for helpful discussions along the way and acknowledge the support of SF IT Award: CS in making this work possible. EFEECES [] C. E. Shannon, Coding theorems for a discrete source with a fidelity criterion, IE ational Convention ecord, vol. 7, no. 4, pp , 959. [2] M. S. Pinsker, Bounds on the probability and of the number of correctable errors for nonblock codes, Problemy Peredachi Informatsii, vol. 3, no. 4, pp , Oct./Dec [3] A. Sahai, Why block length and delay are not the same thing, IEEE Trans. Inform. Theory, submitted. [Online]. Available: sahai/papers/focusingbound.pdf [4], How to beat the sphere-packing bound with feedback, submitted to ISIT, [5] I. Csiszár and J. Körner, Information Theory. Budapest: Akadémiai Kiadó, 986. [6]. G. Gallager, Information Theory and eliable Communication. ew York, Y: John Wiley, 97. [7] C. Chang, S. Draper, and A. Sahai, Sequential random binning for streaming distributed source coding, In preparation. [8] F. Jelinek, Buffer overflow in variable length coding of fied rate sources, IEEE Transactions on Information Theory, vol. 4, pp , 968. [9] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications. Springer, 998. [0] S. Draper, C. Chang, and A. Sahai, Sequential random binning for streaming distributed source coding, ISIT, [] C. Chang and A. Sahai, Upper bound on error eponents with delay for lossless source coding with side-information, submitted to ISIT,