Rate-Distortion in Near-Linear Time

Transcription

1 -Distortion in Near-Linear Time IIT 28, Toronto, Canada, July 6 -, 28 Anit Gupta, ergio Verdù Tsachy Weissman, Department of Electrical Engineering, Princeton University, Princeton, NJ 854, {anitg,verdu}@princeton.edu Department of Electrical Engineering, tanford University, tanford, CA 9435, UA, tsachy@stanford.edu Department of Electrical Engineering, Technion, Haifa 32, Israel Abstract We present two results related to the computational complexity of lossy compression. The first result shows that for a memoryless source P with rate-distortion function, the rate-distortion pair ( + γ,d + ɛ) can be achieved with constant decoding time per symbol encoding time per symbol proportional to C (γ)ɛ C 2(γ). The second results establishes that for any given R, there exists a universal lossy compression scheme with O(ng(n)) encoding complexity O(n) decoding complexity, that achieves the point (R, D(R)) asymptotically for any ergodic source with distortion-rate function D( ), where g(n) is an arbitrary non-decreasing unbounded function. A computationally feasible implementation of the first scheme outperforms many of the best previously proposed schemes for binary sources with bloclengths of the order of. I. INTRODUCTION -distortion theory assures us that for an ergodic source with rate-distortion function, there exists a lossy compression scheme that achieves the rate-distortion pair (,D) asymptotically. However all achievability proofs in the literature rely on schemes with encoding complexity growing at least exponentially in the bloclength n. An interesting question is whether there exists a compression/decompression scheme with low complexity that can achieve the optimal performance asymptotically. Asymptotically optimal bloc codes based on sparse matrices have been proposed for the nonredundant source with Hamming distortion criterion, 2 3. Unfortunately, these schemes rely on the exponential time ML encoder for asymptotic optimality. Other schemes based on sparse-graph codes low complexity message-passing encoders are not theoretically optimal, but perform well in practice 4, 5. Recently, sparse graph codes have been constructed which combine asymptotically optimality for a general discrete memoryless source an arbitrary separable distortion criterion under ML encoding suboptimal quadratic encoders with good empirical performance 6. A universal, asymptotically optimal lossy compression scheme, for memoryless sources is proposed in 7, however the scheme performs far away from, for bloclengths of the order of for the simple case of compressing the Bernoulli source with Hamming distortion criterion. An asymptotically optimal low (polynomial) complexity compressor with near optimal empirical performance, has not yet been found in the literature. In this paper we construct lossy compressors by partitioning a source string of length n into words of length l each use a short bloc code (with bloclength equal to l) to encode each of these words (Figure ). The short code has the same rate as the long code (with bloclength equal to n). If the average distortion (over the source realization) achieved by the short code is less than D + ɛ, then the probability that the long code distortion exceeds D + ɛ vanishes asymptotically. We show that that there exists a short code with rate +γ that achieves average distortion less than D + ɛ has bloclength l g (γ)log(/ɛ)+g 2 (γ). By concatenating this code (as shown in Figure ) we obtain a lossy compressor with constant decoding time per symbol compression time proportional to C (γ)ɛ C2(γ). Using the code construction in Figure, we also show the existence of a universal lossy compressor with encoding complexity O(ng(n)) (for a nondecreasing unbounded g(n)) decoding complexity O(n), that asymptotically achieves the rate-distortion pair (R, D(R)) for any ergodic source with distortion-rate function D( ). The rest of the paper is organized as follows. In ection II we present the code design. In ection III we use the code design in ection II to obtain a scheme that has rate + γ, achieves distortion less than D + ɛ asymptotically with encoding time per symbol proportional to C (γ)ɛ C2(γ). In ection IV we show the existence of a universal lossy compression scheme with near-linear encoding complexity linear decoding complexity that asymptotically achieves the optimal rate-distortion tradeoff for any ergodic source. In ection IV we also show a near-linear complexity extension of the lossy compression scheme in ection III, that asymptotically achieves the optimal rate-distortion tradeoff. In contrast to the universal scheme which is only of theoretical interest, this scheme is also amenable to practical implementation as demonstrated by the empirical results in ection V. In ection V we show empirical results obtained when the scheme in ection III is used to encode binary sources with various distortion criteria compare its performance to other schemes in the literature. II. CODE CONTRUCTION The code construction in this section results in a long lossy compressor by bootstrapping a short lossy compression code C (consisting of 2 lr codewords) that encodes a vector of length l into a binary vector of length lr, with average distortion equal to Ed l (, 2,, l ; C ) = D, () where d l (s,s 2,,s l ; C ) is the distance from (s,s 2,,s l ), to its nearest neighbor in C. For an /8/$ IEEE 847

2 integer, we now construct a lossy compressor for a vector of length l = n as follows: l = g (γ)log ɛ + g 2(γ). (3) ource n C C C C C C l Proof. Denoting the distortion-rate function by R ( ), let D = R (+γ/2) = D g(γ), olve the optimization problem R( D) = min PŜ Ed(,Ŝ) D I(; Ŝ), (4) Encoding Fig.. nr Encoding procedure A. Encoding Given a source realization s, we partition it into words of length l each. Each word is encoded by the index of the nearest codeword in C (Figure ), since the code contains 2 lr codewords this can be done in l2 lr operations. Thus encoding time per symbol is equal to K 2 lr,wherek depends only on the underlying computational model. The index of the corresponding codeword is represented by lr bits. The output of the encoder is the sequence of blocs of lr bits each i.e. nr bits. B. Decoding Using the indices corresponding to each bloc, the decoder outputs the sequence of codewords corresponding to each of the words using l symbols for each word. Total number of operations required is equal to l = n, thus the decoding time per symbol is a constant K 2 which depends only on the underlying computational model. III. AYMPTOTIC OPTIMALITY AND COMPUTATIONAL COT PER YMBOL Theorem. Given a memoryless source, a separable bounded distortion criterion a point (,D) on the ratedistortion function, for arbitrary ɛ, γ > there exists a compression/decompression scheme with bloclength n rate R = +γ such that the probability that the normalized distortion between the source realization its reproduction exceeds D + ɛ vanishes with n, ) Encoding time per symbol is proportional to C (γ)ɛ C2(γ). 2) Decoding time per symbol is independent of ɛ γ. We will show this theorem using the following result. Lemma. Given a memoryless source with rate-distortion function, γ>, ɛ> D>, there exists a code C with rate R = +γ bloclength l, such that C (γ), C 2 (γ) do not dependent on ɛ. Ed l ( l, C ) <D+ ɛ, (2) lr to obtain P ( D) P ( D). For now choose an arbitrary integer Ŝ Ŝ, l. LetC l denote a rom codeboo consisting of 2 l(+γ) codewords of length l, such that all symbols are independent identically distributed with the distribution P ( D).Define Ŝ the following subsets of A l Âl D = {x, y : d l (x, y) D}, (5) T = x, y : P ( D) l log l(y x) 2 P ( D) + 3γ (y) 4. (6) The sets D c T c are composed of the pairs (x, y) such that the empirical averages l l i= d(xi, yi) l ( D) ( D) l i= log P (yi xi)/p (yi) differ from the expected value by a gap which is a function of γ. Therefore, Ŝ Ŝ using the Chernoff bound P ( D) lŝl(dc ) exp( l(f (γ)), (7) P ( D) lŝl(t c ) exp( l(f 2 (γ))), (8) where f (γ),f 2 (γ) are large deviations exponents. Define D(x) ={y Âl :(x, y) D}, (9) T (x) ={y Âl :(x, y) T}. () Averaging over source codeboo we have Pd l ( l, C l ) >D () = E( P ( D) (D( l ))) 2l(+γ) (2) E( P ( D) (D( l ) T( l ))) 2l(+γ) (3) E( P ( D) l(d(l ) T( l ) l ) 2 l(+3γ/4) ) 2l(+γ) (4) E P ( D) l(d(l ) T( l ) l )+2 lγ/4 (5) = P ( D) lŝl(d T)+2 lγ/4 (6) = P ( D) lŝl(dc T c )+2 lγ/4 (7) P ( D) lŝl(dc )+P ( D) lŝl(t c )+2 lγ/4 (8) 3(2 lf (γ) ), (9) where (4) follows from the definition in () (5) follows 848

3 from ( pβ ) M p + M β, (2) for p, β>, f (γ) =min{f (γ),f 2 (γ),γ/4}. (2) Using (9) averaging over code source, Ed l ( l, C l ) Pd l ( l, C l ) DD + Pd l ( l, C l ) >DD max (22) 3(D max D)2 l(f (γ)) + D, (23) We finally choose the bloclength l : l = f (γ) log 2 ɛ ( 3Dmax ). (24) From (24) (23) it follows that Ed l ( l, C l ) < D + ɛ, which implies that there exists at least one codeboo C for which (2) is satisfied, with the bloclength given in (24). Lemma may be compared to Theorem in 8, where it is shown that there exists a sequence of codes with rate bloclength n that achieve average distortion D n such that ( log n D n D = D () log n 2n + o n ), (25) where D (R) is the derivative of the distortion-rate function at R. In other words if we allow no slac in the rate, to achieve performance within ɛ of the optimal distortion the scheme in 8 requires bloclength growing at least as (/ɛ)log(/ɛ), whereas Lemma shows that if we allow a slac of γ in the rate, then n log(/ɛ) is enough to achieve a distortion D + ɛ. Note that (9) is reminiscent of Marton s reliability function result 9: There exist a sequence of bloc codes C(l) with bloclength l rate +γ, such that d l ( l, C(l)) satisfies l l log P d l ( l = e(γ), (26), C(l)) >D where e(γ) = min D(P P ), (27) P R(P,D) +γ where R(P,D) denotes the rate-distortion function of the source P. However Marton s result is not sufficient to prove Lemma for which we need a nonasymptotic bound on Pd l ( l, C(l)) >D. Proof of Theorem. Choose l from (3), construct a code for compressing a source with bloclength n = l, usingc as described in ection II. Let D (n) be the rom variable that denotes the normalized distortion between the source realization its reproduction obtained with this scheme, let D i be the normalized distortion achieved in the i th word, i =, 2,,.Then n PD (n) >D+ ɛ = P D i >D+ ɛ =. (28) where (28) follows from the law of large numbers, because D i are i.i.d. rom variables with E(D i ) <D+ ɛ. From ection II-A, the encoding time per symbol is proportional to 2 l(+γ) = C (γ)ɛ C2(γ) where i= C (γ) =2 g2(γ)(+γ), (29) C 2 (γ) =g (γ)(+γ). (3) The decoding complexity per symbol is a constant as demonstrated in ection II-B. Theorem establishes that for a given source P, R = R(P,D)+γ arbitrary ɛ >, there exists a lossy compression scheme, with rate R that achieves distortion D+ɛ asymptotically almost surely, with encoding time per symbol C (γ)ɛ C2(γ).Howeverif R>max R(P,D) (3) P then there exists a lossy compression scheme, with rate R that achieves guaranteed distortion less than D for any source on that alphabet. This scheme is described below: Marton 9 shows that if R satisfies (3), then there exists l such that if l l there exists a bloc-code C(l) with rate R bloclength l, such that for any sequence s l A l, d l (s l, C(l)) <D. Thus for any n = l, l l we can obtain a lossy compressor by concatenating copies of C(l) as in ection II. This compressor has guaranteed distortion less than D for any source realization any bloclength n = l with encoding time per symbol proportional to 2 lr. In particular if P achieves the maximum in (3) then for any rate close to (but slightly larger) than R(P,D), there exists a linear-time lossy compression scheme that guarantees distortion less than D. IV. ALMOT LINEAR-COMPLEXITY AYMPTOTICALLY OPTIMAL UNIVERAL LOY COMPREOR We present a sequence of universal encoding/decoding schemes indexed by bloclength n. These schemes have linear decoding complexity, O(ng(n)) encoding complexity (for an arbitrary non-decreasing unbounded function g(n)) asymptotic rate equal to R. Further, when used to compress an ergodic source with distortion-rate function D(R), the probability that the distortion between the source its reproduction exceeds D(R) vanishes with n. A constructive proof of the existence of such schemes is given below: Theorem 2. For a given R an arbitrary non-decreasing unbounded function g(n), there exists a sequence of universal encoding/decoding schemes indexed by n with asymptotic rate equal to R, encoding decoding complexities 849

4 O(ng(n)) O(n) respectively, such that n Pdn ( n, Ŝn ) >D(R)+ɛ =, (32) for any ɛ>, whered( ) is the distortion-rate function for the ergodic source n. Proof. We show this result assuming that g( ) is subexponential, it then follows a-fortiori for all other g( ). Encoder: Let log2 log l(n) = 2 g(n), (33) (R +) let C l(n) (R), denote the set of all codeboos with rate R, bloclength l(n) with symbols chosen from the reproduction alphabet Â. Given a source realization s n choose a codeboo from C l(n) (R) that achieves the minimum distortion when used to encode the sequence s n as shown in Figure. Describe the selected codeboo the encoding with nr +log 2 C l(n) (R) bits. An upper bound on the number of codeboos is given as C l(n) (R) Â l(n) 2l(n)R (34) Using (34) (33) the fact that g(n) is sub-exponential R l(n) 2 l(n)r n R +log 2 Â = R. (35) n n n Encoding taes n2 l(n)r C l(n) (R) operations. From (34) it can be shown that n2 l(n)r C l(n) (R) < n2 2l(n)(R+) for all sufficiently large n. Thus from (33) the encoding complexity is given as O(ng(n)). Decoder: Identify the codeboo using log 2 C l(n) (R) bits obtain the reproduction as described in ection II-B. Decoding taes n operations thus the decoding complexity is O(n). Optimality ee. A. An extension to Theorem We now modify the lossy compressor in ection III, such that it achieves the rate-distortion point (,D) asymptotically, with O(n) decoding O(ng(n)) encoding complexity. In contrast to the universal scheme which is only of theoretical interest, this compressor is also amenable to practical implementation as suggested by the results in ection V. Code Construction: Choose arbitrary c>. Let log2 g(n) l(n) = (36) +c ( ) log l(n) γ(n) =f, (37) l(n) where f ( ) is defined in (2) f (x) =inf{y :f (y) x}. (38) Choose a codeboo C l(n) with rate +γ(n), bloclength l(n), that satisfies Pd l ( l(n), C l(n) ) D 3(2 l(n)f (γ(n)) ), (39) such a codeboo is guaranteed to exist from (9). The lossy compressor is obtained by concatenating = n/l(n) copies of C l(n) as shown in Figure. It can be shown that n γ(n) = (see ), thus the asymptotic rate is equal to. From ection II-A encoding complexity is given as O(n2 (+γ(n))l(n) ), choosing n large enough such that γ(n) <cthe encoding complexity is upper bounded by O(ng(n)). The decoding complexity depends neither on l(n) nor on γ(n) is equal to O(n). Optimality: Let D i denote the distortion in the i th bloc. It can be shown that (see ) PD i >D n n 3(2 l(n)f (γ(n)) )=. (4) Therefore for any δ>there exists n(δ) such that for n> n(δ), PD i >D <δ.defining the binary rom variables B i ={D i >D}, ɛ = ɛ/d max we have P D n >D+ ɛ P B i >ɛ. (4) Let <δ<ɛ.forn>n(δ), from (4) the Chernoff bound we have PD n >D+ ɛ P B i >ɛ 2 n l(n) d(ɛ δ). (42) i= Therefore PD n >D+ ɛ =. (43) n V. EXPERIMENT i= The scheme in ection III, with fixed bloclength l bootstraps a short code with rate +γ average distortion D+ɛ to obtain a long code, such that when a source realization is compressed using this code, the probability that the resulting distortion exceeds D + ɛ vanishes asymptotically. Thus it can achieve the rate-distortion function in the strong excessdistortion sense. To further highlight this notion of guaranteed distortion we plot the percentile distortion as a function of rate. Where a p percentile distortion implies that in p percent of empirical runs the distortion is less than the plotted value. Empirically for bloclength of the order of thouss, this scheme outperforms existing schemes under the percentile distortion criterion. In each of these experiments we romly generate fix a short rom code with bloclength l = 22/(+γ). Thus, the encoding scheme has linear complexity in the bloclength n. Figure 2 shows the distortionrate performance of our scheme compared with the message passing based encoder applied to a regular sparse graph code described in 4 (with O(n 2 ) complexity) when used to compress a fair coin source with bloclength n =. Each point is obtained by running trials of encoding romly generated source vectors for each rate, choosing the bit error rate level such that 95% of the trials result in smaller bit error rate. In Figure 3 we compare the performance of our scheme when used for compressing the Bernoulli() source with bit error rate-distortion criterion. It is noted in that 85

5 .5 5 Algorithm in 4.5 cheme in 6 Bit Error Distortion Fig. 2. Bit Error (95% percentile criterion) for the fair coin source with bloclength= Fig. 4. Asymmetric distortion (44) (95% percentile criterion) for the fair coin source with bloclength=. Bit Error Algorithm in Fig. 3. Bit Error (95% percentile criterion) for the biased coin (bias=) source bloclength=. Average Bit Error shown for the scheme in. for this problem, even to beat the straight line connecting the points (, h()) (, ) on the (D, ) curve is a challenge for any encoder, no constructive schemes are nown that perform near the optimal distortion-rate function. The compressor proposed in (with O(n) complexity) is slightly better than the straight line, is one of the best implementable schemes for this problem in the literature. The asymptotic (n ) average distortion obtained with the scheme in is compared to our scheme with n = in Figure 4. It should be noted that while the curve for represents the average bit error rate, the curve for our scheme shows the stricter 95% percentile distortion criterion. In Figure 4 we show results for compressing fair coin flips with the asymmetric distortion criterion: d(, ) = 2d(, ) = 2. (44) This problem was also addressed in 6 using sparse graph codes. The new scheme also outperforms the scheme given in 6 (which has a time complexity of O(n 2 log 3 n)). In Figures 3 4 with biased coin asymmetric distortion, respectively, we use a short rom code with bloclength l = 22/(+γ), which is obtained in the following fashion. Choose a desired operating point (D, ) approximate the optimal binary reproduction distribution P Ḓ with a q-type ( P Ḓ ) over {, }. First construct a rom code over the ring {,,,q }, usinganr n matrix G, such that each of its elements is chosen from the set {,,,q } uniformly at rom. The codewords in the code are obtained by c i = G T u i,whereu i is a binary vector addition is modulo q. now apply a mapping Q : {,,,q } {, } to each c i such that Q maps the uniform probability distribution over {,,,q } to P Ḓ over {, }. Increasing bloclength while maintaining the same short code results in a further reduction in the probability of excess distortion. Narrowing the gap with the distortion-rate curve requires increasing the bloclength of the short code. These experimental results establish that the code construction in ection III is not only theoretically interesting but with small (computationally-feasible) wordlength outperforms existing constructive schemes for bloclengths of the order. REFERENCE Y. Matsunaga H. Yamamoto, A coding theorem for lossy data compression by LDPC codes, IEEE Transactions on Information Theory, vol. 49, pp , ep Miyae, Lossy data compression over Z q by LDPC code, IEEE International ymposium on Information Theory, eattle UA, july E. Martinian M. J. Wainwright, Low-density codes achieve the rate-distortion bound, Data Compression Conference, Utah, E. Maneva M. J. Wainwright, Lossy source encoding via messagepassing decimation over generalized codewords of LDGM codes, IEEE International ymposium on Information Theory, Adelaide, Austrailia, eptember Ciliberti, K. Mezard, R. Zecchina, Message passing algorithms for non-linear nodes data compression, arxiv. Online. Available: 6 A. Gupta. Verdú, Nonlinear sparse-graph codes for lossy compression, submitted to: IEEE Transactions on Information Theory. 7 I. Kontoiannis, An Implementable Lossy Version of the Lempel-Ziv Algorithm Part I: Optimality for Memoryless ources, IEEE Transactions on Information Theory, vol. 45, pp , Nov Z. Zhang, E. Yang, V. Wei, The redundancy of source coding with a fidelity criterion.. Known statistics, IEEE Transactions on Information Theory, vol. 43, no., pp. 7 9, June K. Marton, Error exponent for source coding with a fidelity criterion, IEEE Transactions on Information Theory, vol. 2, no. 2, pp , March 974. A. Gupta,. Verdu, T. Weissman, Almost-Linear lossy compression for memoryless sources, Draft, 28. J. Rissanen I. Tabus, -distortion without rom codeboos, Worshop on Information Theory Applications (ITA),UCD,