Achieving the Gaussian Rate-Distortion Function by Prediction

Transcription

1 Achievig the Gaussia Rate-Distortio Fuctio by Predictio Ram Zamir, Yuval Kochma ad Uri Erez Dept. Electrical Egieerig-Systems, Tel Aviv Uiversity Abstract The water-fillig solutio for the quadratic ratedistortio fuctio of a statioary Gaussia source is give i terms of its power spectrum. This formula aturally leds itself to a frequecy domai test-chael realizatio. We provide a alterative time-domai realizatio for the rate-distortio fuctio, based o liear predictio. This solutio has some iterestig implicatios, icludig the optimality at all distortio levels of vector-quatized differetial pulse code modulatio DPCM, ad a duality relatioship with decisio-feedback equalizatio DFE for iter-symbol iterferece ISI chaels. I. INTRODUCTION: RATE-DISTORTION AND PREDICTION The rate-distortio fuctio RDF of a statioary source with memory is give by a time domai formula, that is, as a limit of ormalized mutual iformatios associated with vectors of source samples. For a real valued source..., X, X, X 0, X, X,..., ad mea-squared distortio level D, the RDF ca be writte as, [], RD = lim if IX,..., X ; Y,..., Y where the ifimum is over all chaels X Y such that Y X D. A chael which realizes this ifimum is called a optimum test-chael. Whe the source is Gaussia, the RDF takes a explicit form i the frequecy domai. Assumig auto-correlatio fuctio R[k] = E{X X k }, ad power-spectrum Se jπf = k R[k]e jkπf, f:se jπf >θ π < f < π the RDF is give by the water fillig formula / Se jπf RD = / log De jπf df Se jπf = log df θ where De jπf is the distortio spectrum { De jπf θ, if Se = jπf > θ Se jπf, otherwise, ad θ = θd is the water level chose such that / / Dejπf df = D. For the special case of a memoryless white Gaussia source N0, σ, the power-spectrum is flat Se jπf = σ, ad the RDF is simplified to log σ D. 3 Our mai result is a predictive time-domai realizatio for the quadratic-gaussia RDF. The otio of etropypower ad the Shao lower boud provide a simple relatio betwee ad predictio, which motivates our result. Recall that that the etropy-power is the variace of a white Gaussia process havig the same etropy-rate as the source; for a Gaussia source the etropy-power is give by / P e X = exp log Se jπf df. 4 / The Shao lower boud states that for ay D RD log Pe X D with equality for distortio levels smaller tha or equal to the lowest value of the power spectrum: D mi f Se jπf i which case De jπf = θ = D. See []. I the cotext of Wieer s spectral-factorizatio theory, 4 quatifies the measquared error of the oe-step liear predictor of the source from its ifiite past []; that is, we also have P e X = if {a i} E X 5 a i X i. 6 Now, by the orthogoality priciple see [4], the error process of the optimum ifiite-order predictor sometimes called the iovatio process E = X a i X i has zero mea ad is white. Hece, i view of 3 ad 5, for small distortio levels the RDF of a Gaussia source with memory is equal to the RDF of its memoryless predictio error process at the same distortio level. We shall see later i Sectio II how the observatio above traslates ito a predictive test-chael which ca realize the RDF ot oly for small but for all distortio levels. Before that, we d like to cosider aother feature of the frequecy-domai solutio which motivates the predictive time-domai result of Sectio II. Note that the optimum test-chael that realizes the memoryless RDF 3 takes the form of a memoryless liear-additive oise chael: Y = βαx N with α = β = D/σ ad N N0, D. I the geeral case α ad β take the form of pre- ad post-filters. I view

2 N X Pre filter U Z Zq V Post filter H e jπf H e jπf Y Û Predictor fv L Fig.. Predictive Test Chael of that, oe way to uderstad the form of the geeral RDF i is to look o its discrete approximatio i Se jπf log i De jπfi as ecodig of parallel idepedet Gaussia sources, where source i is a memoryless Gaussia source X fi N0, Se jπfi ecoded at distortio De jπfi ; see [3]. Furthermore, the RDF ca be realized by a vector of parallel chaels of the form Y fi = β i α i X fi N i. This iterpretatio motivates practical, frequecy domai source codig schemes such as Trasform Codig ad Sub-bad Codig [5], which get close to the RDF of a Gaussia source with memory usig scalar quatizatio. Sectio II proposes a alterative formulatio for the RDF that is motivated by the time-domai quatizatio scheme of Differetial Pulse Code Modulatio DPCM. The goal of the ew formulatio is, like i the frequecy domai iterpretatio of 7, to traslate the ecodig of depedet source samples ito a series of ecodigs of idepedet sources. The task of removig the depedece i the time domai approach is achieved by predictio. Kim ad Berger [8] showed that we caot achieve the RDF of a auto-regressive AR Gaussia process by ecodig its iovatio process. Their scheme amouts to ope-loop predictio of the source. I this paper we show that RDF ca be achieved by embeddig the ecoder iside the predictio loop, i.e., by closed-loop predictio. After presetig ad provig our mai result i Sectios II ad III, respectively, we provide reflectios o the result ad its operatioal implicatios. Sectio IV discusses the spectral features of the solutio, Sectio V relates it to compressio of parallel sources, ad Sectio VI discusses implemetatio by Etropy Coded Dithered Quatizatio ECDQ. Fially, i Sectio VII we relate predictio i source codig to predictio i the area of chael equalizatio ad to recet observatios by Forey [4]. Like i [4], our aalysis is doe maily usig the properties of iformatio measures; from Wieer estimatio theory we eed oly two basic results: the orthogoality priciple ad the oe-step predictio error formula 6. 7 II. MAIN RESULT: A PREDICTIVE TEST CHANNEL Cosider the system i Figure, which cosists of three basic blocks: pre-filter H e jπf, a oisy chael embedded i a close loop, ad a post-filter H e jπf. The system parameters are derived from the water-fillig solutio -. The source samples {X } are passed through a pre-filter, whose phase is arbitrary ad its absolute squared frequecy respose is give by H e jπf = Dejπf Se jπf. 8 The pre-filter output, deoted U, is beig fed to the cetral block which geerates a process V accordig to the the followig recursio equatios: Û = fv, V,..., V L 9 Z = U Û 0 Zq = Z N V = Û Zq where N N0, θ is zero-mea additive white Gaussia oise AWGN idepedet of the iput process {U } whose variace is equal to the water level θ, ad f is some predictio fuctio for the iput U give the L past samples of the output process V, V,..., V L. Fially, the post-filter frequecy respose is the complex cojugate of that of the pre-filter, H e jπf = H e jπf. 3 The block from U to V is equivalet to the cofiguratio of differetial pulse code modulatio DPCM, [7], [5], with the DPCM scalar quatizer replaced by the AWGN chael Zq = Z N. I particular, by combiig the recursio equatios 9- it follows that this block satisfies the well kow DPCM error idetity, [7], V = U Zq Z = U N. 4 That is, the output V is a oisy versio of the iput U via the AWGN chael V = U N. I DPCM the predictio fuctio f is liear: L fv,..., V L = a i V i 5

3 N X U V Pre filter Post filter He jπf He jπf Fig.. Equivalet Chael where a,..., a L miimize the mea-squared oise predictio error L E U a i V i. 6 If {U } ad {V } are joitly Gaussia, the the best predictor of ay order is liear, so the miimum of 6 is also the the miimum mea squared error MMSE i estimatig U from the vector V,..., V L. We shall further elaborate o the relatioship with DPCM later. Note that while the cetral block is sequetial ad hece causal, the pre- ad post-filters are o-causal ad therefore their realizatio i practice requires large delay. Our mai result is the followig. Theorem : For ay statioary source with power spectrum Se jπf, the system of Figure satisfies Y EY X = D. 7 Furthermore, if the source X is Gaussia, ad if the fuctio f is the optimum liear predictor of U give the ifiite past V, V,..., i.e., L = i 9-, the for all IZ ; Z N = RD. 8 The proof is give i Sectio III. The mai feature of Theorem is the fact that the left-had side of 8 is a sigle letter mutual iformatio. Thus, i a sese the core of the ecodig process amouts to a memoryless AWGN testchael. Aother iterestig feature of the system is the relatioship betwee the predictio error process Z ad the origial process X. If X is a auto-regressive AR process, the i the limit of small distortio D 0, Z is roughly its iovatio process. Hece, ulike i ope-loop predictio [8], ecodig the iovatios i a closed-loop system is optimal i the limit of high-resolutio ecodig. We shall retur to that, as well as discuss the case of geeral resolutio, i Sectio IV. III. PROOF OF MAIN RESULT Note first that the error idetity 4 implies that the etire system of Figure is equivalet to the system depicted i Figure, cosistig of a pre-filter 8, a AWGN chael with oise variace θ, ad a post-filter 3. This is, i fact, oe of the equivalet forms of the forward chael realizatio of the quadratic-gaussia RDF [, Sectio 4.5], [9]. I particular, simple spectral aalysis shows that the power spectrum of the overall error process Y X is equal to the water fillig distortio spectrum De jπf i ; hece the total distortio i 7 is D as desired. For the secod part, sice the system of Figure coicides with the forward chael realizatio of the quadratic-gaussia RDF, for a Gaussia source we have I{X }; {Y } = I{U }; {V } = RD where I deotes mutual iformatio-rate betwee joitly statioary sources: I{X }; {Y } = lim IX,..., X ; Y,..., Y. 9 Hece, the proof will be completed by showig that I{U }; {V } = IZ ; Z N. 0 To that ed, cosider the coditioal mutual iformatio betwee U,..., U ad V,..., V give V 0 L = V 0, V,..., V L. Usig the chai rule we have IU,..., U ; V,..., V V L 0 = IU,..., U ; V i V i L. By the recursio equatios 9-, the i-th term i the sum ca be writte as IU,..., U ; V i V i L = IU,..., U i, U i Ûi, i U i,..., U ; V i Ûi V L = IU,..., U i, Z i, U i,..., U ; Z i N i V i L. Sice the AWGN N is idepedet of the source U,..., U ad of the past values of V, we have the Markov chai relatio Z i N i Z i, V i L U,..., U i, U i,..., U. Note that future values of the output process V i, V i,.. deped o N i, but this does ot affect the Markov chai above. This relatio implies that the i-th term above simplifies to IZ i ; Z i N i V i L. Util ow we did t eed to use the source Gaussiaity or the optimality of the predictio fuctio f. Takig these ito accout, ad lettig the predictor order L, the orthogoality priciple of MMSE estimatio implies that the estimatio error Z i is statistically idepedet of the measuremets V i, V i,.... Sice N i is idepedet of both Z i ad the past V i s, the i-th term above further simplifies to IZ i ; Z i N i. We obtai IU,..., U ; V,..., V V 0, V,... = IZ i ; Z i N i = IZ ; Z N where the secod equality follows sice the system is timeivariat ad all the processes are statioary. Usig the defiitio of mutual iformatio rate 9, ad otig that it remais uchaged if we coditio o the ifiite past, we arrive at 0 ad the proof is completed.

4 IV. PROPERTIES OF THE PREDICTIVE TEST-CHANNEL The followig observatios shed light o the behavior of the test chael of Figure. Predictio i the high resolutio regime. If the powerspectrum Se jπf is everywhere positive e.g., if {X } ca be represeted as a AR process, the i the limit of small distortio D 0, predictig U from its oisy past {V i = U i N i : i =,,...}, is equivalet to predictig U from its clea past {U, U,...}. Hece, i this limit the predictio error Z is equal to the iovatio process associated with U. Sice for small distortio the pre- ad post-filters 8, 3 are roughly all-pass filters, U has roughly the same power spectrum as X ; hece Z is i fact equivalet to the iovatio process of X. I particular, Z is a i.i.d. process whose variace is P e X = the etropy-power of the source 4. Predictio i the geeral case. Iterestigly, for geeral distortio D > 0, the predictio error Z is ot white, as the oisiess of the past does ot allow the predictor f to remove all the source memory. Nevertheless, the oisy versio of the predictio error Zq = Z N is white for every D > 0, because it amouts to predictig V from its ow ifiite past: sice N is white ad has zero-mea, Û, which is the optimal predictor for U, is also the optimal predictor for V = U N. This whiteess might seem at first a cotradictio, because Zq is the sum of a o-white process Z ad a white process N. However, {Z } ad {N } are ot idepedet processes, because Z depeds o past values of N through the past of V. Hece the chael Zq = Z N is ot quite a additive-oise chael; rather, it is sequetially-additive: the oise is idepedet of past ad preset chael iputs, but is ot idepedet of future iputs. These observatios imply that the chael Zq = Z N satisfies: IZ ; Z N Z N,..., Z N = IZ ; Z N, while i geeral Ī{Z }; {Z N } > IZ ; Z N. The chael whe the Shao lower boud holds. As log as D is smaller tha the lowest poit of the source spectrum i.e., De jπf = θ = D i, the quadratic- Gaussia RDF coicides with the Shao Lower Boud 5. I this case, the followig properties hold for the predictive test chael: The power spectra of U ad Y are the same ad are equal to Se jπf D. The power spectrum of V is equal to the power spectrum of the source Se jπf. Sice the white process Z N is the optimal predictio error of the process V from its ow ifiite past, its variace is P e V = P e X of 4. As a cosequece we have IZ ; Z N = hn0, P e X hn0, D = / logp e X/D X X K K idepedet pre-post filters Y ad predictio loops Fig. 3. Y K Parallel sources. joit quatizatio K-dim VQ which is ideed the Shao lower boud 5. V. VECTOR-QUANTIZED DPCM AND D PCM As metioed, the structure of the loop i the chael of Figure is of a DPCM quatizer, with the scalar quatizer replaced by the additive oise. However, if we wish to implemet the additive oise by a quatizer realizig the full sigle letter mutual iformatio IZ ; Z N, we must use vector quatizatio VQ. It is ot possible to do that alog the time domai, due to the sequetial ature of the system above. Nevertheless, we ca achieve the VQ gai by addig a spatial dimesio, if we joitly ecode a large umber of parallel sources, as happes, e.g., i video codig. See Figure 3. If we have oly oe source with decayig memory, we ca still achieve the rate distortio fuctio at the cost of large delay, by usig iterleavig. If we do ot use ay of the above, but restrict ourselves to scalar quatizatio, we have a pre/post filtered DPCM scheme. It follows from Theorem that i priciple, a pre/post filtered DPCM scheme is optimal up to the loss of the VQ gai at all distortio levels, ad ot oly at high resolutio. It is iterestig to metio that I the quatizatio literature, the ope loop predictio approach ivestigated i [8] is referred to as D PCM [7]. VI. A DUAL RELATIONSHIP WITH DECISION-FEEDBACK EQUALIZATION Cosider the real-valued discrete-time time-ivariat liear Gaussia chael arisig at the output of a sampled matched filter, Y = h k X k Z. k= Here h is the equivalet discrete-time impulse respose resultig from the cascade of the spectral shapig filter, the modulatio pulse, the cotiuous-time chael as well as the matched filter. It follows that He jπf is o-egative ad the autocorrelatio fuctio of Z is R ZZ k = E{Z k Z } = N 0 h k. Let X be a iid Gaussia radom process with power σ x. The the mutual iformatio ormalized per symbol betwee

5 Z Y X Z ˆX X Hz Fig. 4. Az FFE ˆD MMSE-DFE i predictive form. the iput ad output of the chael is I{X }, {Y } = / / log GZ DFE σ xhe jπf N 0 / X D ˆX df. 3 Capacity is achieved by usig a spectral shapig filter satisfyig the water-fillig power allocatio. As reflected i the expressio 3, capacity may be achieved by parallel AWGN codig over arrow frequecy bads as doe i practice i DMT/OFDM systems. A well kow alterative capacityachievig scheme which is based o predictio rather tha the Fourier trasform is offered by the caoical MMSE- DFE equalizatio structure used i sigle-carrier trasmissio. These observatios parallel those made i Sectio I with respect to the RDF. It is well kow that the capacity of liear Gaussia chaels ca be achieved usig MMSE-DFE coupled with AWGN codig. This has bee show usig differet approaches. Oe such approach which is particularly illumiatig i our cotext is based o liear predictio of the error sequece. We ow recout this result which was developed ad refied by umerous authors, see [6], [], [4] ad refereces therei. Our expositio closely follows that of Forey [4]. As a first step, let ˆX be the optimal MMSE estimator of X from the chael output sequece {Y }. Sice {X } ad {Y } are joitly Gaussia ad statioary this estimator is liear ad time ivariat. Deote the estimatio error, which is composed of residual ISI ad Gaussia oise, by D. The X = ˆX D 4 where {D } is idepedet of { ˆX } due to the orthogoality priciple ad Gaussiaity. Assumig correct decodig of past symbols, the decoder kows the past samples D, D,... ad may form a optimal predictor, ˆD, of the estimatio error D. The predictio error E = D ˆD has variace P e D, the etropy power of D. This predictio may the be added to X ˆ to form X. It follows that X = ˆX D = X ˆD D = X E, 5 Here we must actually break with assumptio that X is a Gaussia process. ad We implicitly assume that X are symbols of a capacityachievig AWGN code. The slicer should be viewed as a memoic aid where i practice a optimal decoder should be used. Furthermore, the use of a iterleaver ad log delay is ecessary. See [4] where { X } ad {E } are statistically idepedet. It follows that the residual estimatio error satisfies E{X X } = E{D ˆD } = P e D, 6 The chael 5 is ofte referred to as the backward chael. Furthermore, sice X ad E are i.i.d Gaussia, it is a AWGN chael. We have therefore derived the followig. Theorem : The mutual iformatio of the chael is equal to the scalar mutual iformatio I X ; X E of the memoryless chael 5. Proof: Let X = {X, X,...} ad D = {D, D,...}. Usig the chai rule of mutual iformatio we have I{X }, {Y } = h{x } hx {Y }, X = h{x } hx ˆX {Y }, X = h{x } hd {Y }, X = h{x } hd {Y }, D, X = h{x } hd ˆD {Y }, D = h{x } he {Y }, D = h{x } he 7 = I X ; X E, where 7 follows from successive applicatio of the orthogoality priciple [4]. It follows that / / log σ xhe jπf N 0 / df = log σ x P e D. 8 As a corollary, from 3, Theorem ad 8, we obtai the followig well kow result from Wieer theory, / σx N0 P e D = exp log df. / σxhe jπf N0 REFERENCES [] T. Berger. Rate Distortio Theory: A Mathematical Basis for Data Compressio. Pretice-Hall, Eglewood Cliffs, NJ, 97. [] J.M. Cioffi, G.P. Dudevoir, M.V. Eyuboglu, ad G.D. J. Forey. MMSE Decisio-Feedback Equalizers ad Codig - Part I: Equalizatio Results. IEEE Tras. Commuicatios, COM-43:58 594, Oct [3] T. M. Cover ad J. A. Thomas. Elemets of Iformatio Theory. Wiley, New York, 99. [4] G. D. Forey, Jr. Shao meets Wieer II: O MMSE estimatio i successive decodig schemes. I 4st Aual Allerto Coferece o Commuicatio, Cotrol, ad Computig, Allerto House, Moticello, Illiois, Oct [5] G.D.Gibso, T.Berger, T.Lookabaugh, D.Lidbergh, ad R.L.Baker. Digital Compressio for Multimedia: Priciples ad Stadards. Morga Kaufma Pub., Sa Fasisco, 998. [6] T. Guess ad M. K. Varaasi. A iformatio-theoretic framework for derivig caoical decisio-feedback receivers i gaussia chaels. IEEE Tras. Iformatio Theory, IT-5:73 87, Ja [7] N. S. Jayat ad P. Noll. Digital Codig of Waveforms. Pretice-Hall, Eglewood Cliffs, NJ, 984. [8] K. T. Kim ad T. Berger. Sedig a Lossy Versio of the Iovatios Process is Suboptimal i QG Rate-Distortio. I Proceedigs of ISIT- 005, Adelaide, Australia, pages 09 3, 005. [9] R. Zamir ad M. Feder. Iformatio rates of pre/post filtered dithered quatizers. IEEE Tras. Iformatio Theory, pages , September 996.