Performance Analysis and Optimal Filter Design for Sigma-Delta Modulation via Duality with DPCM

Transcription

1 Performace Aalysis ad Optimal Filter Desig for Sigma-Delta Modulatio via Duality with Or Ordetlich ad Uri Erez, Member, IEEE Abstract Samplig above the Nyquist rate is at the heart of sigma-delta modulatio, where the icrease i samplig rate is traslated to a reductio i the overall (mea-squared-error recostructio distortio. This is attaied by usig a feedback filter at the ecoder, i cojuctio with a low-pass filter at the decoder. The goal of this work is to characterize the optimal trade-off betwee the per-sample quatizatio rate ad the resultig mea-squared-error distortio, uder various restrictios o the feedback filter. To this ed, we establish a duality relatio betwee the performace of sigma-delta modulatio, ad that of differetial pulse-code modulatio whe applied to (discretetime bad-limited iputs. As the optimal trade-off for the latter scheme is fully uderstood, the full characterizatio for sigma-delta modulatio, as well as the optimal feedback filters, immediately follow. I. INTRODUCTION Aalog-to-digital (A/D ad digital-to-aalog (D/A coverters are essetial i moder electroics. I may cases, it is the quality of these coverters that costitutes the mai bottleeck i the system, ad cosequetly, dictates its etire performace. O the other had, as digital circuits are ow cosidered relatively cheap to implemet, the iterface betwee the aalog ad digital domais is ofte oe of the most expesive compoets i the system. Developig A/D ad D/A compoets that are o the oe had relatively simple, ad o the other had itroduce little distortio, is therefore of iterest. Ofte, the same A/D (or D/A compoet is applied to a variety of sigals with distict characterizatios. For this reaso, it is desirable to desig the data coverter to be robust to the characteristics of the iput sigal. Oe assumptio that caot be avoided is the badwidth of the sigal to be coverted, which dictates the miimal samplig rate, accordig to Nyquist s theorem. Beyod badwidth, however, oe would like to assume as little as possible about the iput sigal. A reasoable model for the iput sigal is therefore a stochastic oe, where the iput sigal is assumed to be a statioary Gaussia process with a give variace ad a arbitrary ukow power spectral desity (PSD withi the assumed badwidth, ad zero otherwise. I this paper, we adopt this compoud model which is rich eough to iclude a wide variety of processes. The robustess requiremet from the A/D The work of O. Ordetlich was supported by the Adams Fellowship Program of the Israel Academy of Scieces ad Humaities, a fellowship from The Yitzhak ad Chaya Weistei Research Istitute for Sigal Processig at Tel Aviv Uiversity ad the Feder Family Award. The work of U. Erez was supported by by the ISF uder Grat 557/3. O. Ordetlich ad U. Erez are with Tel Aviv Uiversity, Tel Aviv, Israel ( ordet,uri@eg.tau.ac.il. (or D/A coverter traslates to requirig that it iduces a small average distortio simultaeously for all processes withi our compoud model. Sigma-delta modulatio is a widely used techique for A/D as well as D/A coversio. The mai advatage offered by this type of modulatio is the ability to trade-off the samplig rate ad the umber of bits per sample required for achievig a target mea-squared error (MSE distortio. The iput to the sigma-delta modulator is a sigal sampled at times the Nyquist rate ( >. This over-sampled sigal is the quatized usig a R-bit quatizer. I much of the literature about sigma-delta modulatio, o stochastic model is assumed for the iput sigal. However, whe such a model is assumed, the beefit of over-samplig ca be easily uderstood from basic rate-distortio theoretic priciples: the (per-sample rate required to achieve distortio D for the over-sampled sigal is times smaller tha the rate required to achieve the same distortio for the sigal obtaied by samplig at the Nyquist rate. Thus, i priciple, icreasig the samplig rate should allow oe to use quatizers with lower resolutio, which is desirable i may applicatios. However, the rate-distortio theoretical property that guaratees a costat product of the umber of bits per sample eeded to achieve distortio D, ad the over-samplig ratio, is oly valid whe a very log block of samples is vectorquatized. I A/D ad D/A coversio, vector-quatizatio i high dimesios is a prohibitively complex operatio, ad quatizatio is ivariably doe via scalar uiform quatizers. Scalar quatizers aloe caot traslate the icrease of samplig rate to a sigificat reductio i the ecessary resolutio, but fortuately this problem ca be circumveted with the aid of appropriate sigal processig. I sigma-delta based coverters, the quatizatio oise is shaped usig a causal shapig filter embedded withi a feedback loop, see Figure. The filter coefficiets are chose i a maer that esures that most of the eergy of the shaped quatizatio oise lies outside the frequecy bad occupied by the over-sampled sigal. At the decoder, the quatized sigal is low-pass filtered, cacellig out the high-frequecies of the quatizatio oise process without effectig the sigal, such that the decoder s output is composed of the origial sigal corrupted by a low-pass oise process. Aother techique for compressig sources with memory, which explicitly models the source as a stochastic process, is differetial pulse-code modulatio (. I, a predictio filter is applied to the quatized sigal. The output of this filter is the subtracted from the source ad the result is fed to the quatizer, see Figure. At the decoder, the

2 quatized sigal is simply passed through the iverse of the predictio filter. The well-kow error idetity [] states that the output of the decoder is equal to the source plus the quatizatio error, just like i simple o-predictive quatizatio. The beefit of usig, however, is that the sigal fed to the quatizer is the error i predictig the source from its quatized past, rather tha the source itself. If the coefficiets of the predictio filter are chose appropriately, the variace of this error should be smaller tha the variace of the origial source, which traslates to a reductio i the umber of bits required from the quatizer for achievig a certai distortio. The performace of uder the assumptio of highresolutio quatizatio is well uderstood sice as early as the mid 60 s [] [3]. Uder this assumptio, the predictio filter should be chose as the optimal liear miimum measquared-error (MMSE predictio filter of the source process from its past [], ad the effect of the filtered quatizatio oise ca be eglected i the predictio process. While i most cases where is traditioally used, the high resolutio assumptio is well justified, it totally breaks dow for the class of bad-limited processes, which icludes the iput sigals to sigma-delta modulators. Ideed, the predictio error of such a process from its ifiite past has zero-variace, rederig the high-resolutio rate-distortio formulas completely useless. A. Coectio to Previous Work The coectio betwee ad sigma-delta modulatio, as two istaces of predictive codig, was kow from the outset. Ideed, both paradigms emerged from two Bell- abs patets authored by CC Cutler [4], [5] i 95 ad 954. I fact, by addig appropriate pre- ad post-filters to the sigma-delta modulator, as depicted i Figure 3, the iput to the quatizer, as well as the fial recostructio of the sigal, become idetical to those i the architecture of Figure, see [6, Sectio II], [7, Chapter 3..4]. Thus, oe may implemet via either of the architectures of Figure or Figure 3. However, a importat aspect of our iterest i sigma-delta modulators as a meas of data-coversio rather the datacompressio, is that it dictates that the assumptios oe ca make o the statistics of the iput sigal must be miimal. Cosequetly, we cosider a compoud class of sources that cosists of all statioary Gaussia processes with variace σ X whose PSD is limited to some predefied frequecy bad. Ufortuately, for this compoud class, is usatisfactory, as its performace depeds ot oly o the variace ad badwidth, but rather, o the explicit form of the PSD. O the other had, for ay choice of C(Z, sigma-delta modulatio, as depicted i Figure, attais the same performace for all sources withi the class. The duality result we establish here, is that the performace of sigma-delta modulatio for ay source i the compoud class is equal to that of desiged for a bad-limited statioary process with a flat PSD. Data coverters ofte operate at very high rates, ad it therefore makes sese to impose various costraits o the sigma-delta feedback filter C(Z, such as cofiig it to be a fiite impulse respose (FIR filter with a limited umber of taps. For a give desired MSE distortio level, our goal is to fid the costraied sigma-delta feedback filter C(Z that miimizes the quatizatio rate w.r.t. all sources i the compoud class, ad to characterize the attaied rate. This goal is differet tha the oe pursued i [4], where the optimal ucostraied filters w.r.t. a kow PSD were foud. The problem of fidig the optimal N-tap FIR sigmadelta feedback filter C(Z for a compoud family of sources similar to ours, was cosidered i [6]. The optimal filter was claimed i [6] to be the Nth order MMSE predictio filter C(Z = ( Z N of a badpass statioary process from its past, ad for a fixed target MSE distortio the required quatizatio rate was foud to decrease liearly with N. Such as statemet is obviously iaccurate, as it violates Shao s rate-distortio theorem. The major drawback of [6] is that it (implicitly makes the high-resolutio assumptio that the variace of the quatizer s iput is solely dictated by the target sigal X }, whereas the cotributio of the quatizatio oise to this variace ca be eglected. As discussed above, for over-sampled processes this assumptio may ot be valid eve whe the quatizer s resolutio is very high. I particular, usig the filter C(Z = (Z N from [6], the eergy of the quatizatio oise withi the frequecy bad occupied by the sigal ideed decreases expoetially with N. However, the oise s eergy outside this bad icreases rapidly with N, ad for ay quatizatio resolutio it will become much greater tha σx for N large eough, makig the high-resolutio assumptio iapplicable. I this case, the dyamic rage of the quatizer will be exceeded ad overload errors would frequetly occur. It therefore follows that i the aalysis of sigma-delta modulators oe should ot make high-resolutio assumptios, but rather must take ito accout the effect of the filtered quatizatio oise o the variace of the quatizer s iput. Fortuately, i the aalysis of modulators the highresolutio assumptio has bee overcome i [8]. It was show that for ay distortio level ad ay statioary Gaussia source, the architecture iduces a rate-distortio optimal test chael, provided that the predictio filter is chose as the optimal filter for predictig the source from its quatized past, ad i additio water-fillig pre- ad post-filters are applied. The aalysis of [8], which takes ito accout the effect of the quatizatio oise, ca therefore be used to obtai the optimal feedback filter ad its correspodig performace for a system applied to a over-sampled statioary Gaussia source. I this paper, we leverage the results from [8] to the aalysis of sigma-delta modulators, by establishig a appropriate duality betwee the two architectures. B. Cotributios et S be the compoud class of all discrete-time statioary Gaussia sources with variace σx ad PSD that is zero for all / [/,/],. Note that this class correspods to uiformly samplig a compoud class of cotiuous-time

3 3 statioary Gaussia processes with variace σx ad PSD that is zero for all f > f max, at a samplig rate of f max samples/per secod. et X } be a discrete-time statioary Gaussia process with PSD SX σx for / ( = 0 for / < <, ( ad ote that X } S. Our mai result, derived i Sectio II, is that for ay process X } from the compoud class S, the test chael iduced by the sigma-delta modulator (Figure achieves exactly the same rate-distortio fuctio as that of the test chael (Figure with iput X }. More specifically, for such processes, for ay choice of σ ad predictio filter C(Z i the test chael of Figure, the same choice of C(Z together with the choice σ = σ / / C( d i Figure, yields the same compressio rate ad the same distortio. While this result is simple to derive, it has a very pleasig cosequece: the problem of optimizig the filter C(Z i sigma-delta modulatio w.r.t. ay sigal i S, uder ay set of costraits, ca be cast as a equivalet problem of optimizig the predictio filter w.r.t. iput X } uder the same set of costraits. Furthermore, i Sectio II-A, we formalize a similar duality betwee ad sigma-delta modulatio for a frequecy-weighted-mea-squared-error distortio measure. I this case SX ( is replaced with a PSD that depeds o the distortio s weight fuctio. I priciple, recastig the sigma-delta optimizatio problem as a MMSE predictio problem may be derived directly from the formulas characterizig its performace, as give i Propositio. Nevertheless, establishig the equivalece betwee sigma-delta modulatio ad, i the specific form described above, is isightful as it allows to borrow kow results from the literature about the latter. Havig recast the filter optimizatio problem for sigma-delta as that of optimal liear predictio, we ca readily obtai the solutio uder costraits for which a explicit solutio was lackig i the literature, or was cumbersome to derive. Oe may questio the relevace of the test chael of Figure ad its iformatio-theoretic aalysis to the practical, resource limited, problem of A/D ad D/A coversio via sigma-delta modulators. To that ed, i Sectio III we replace the AWGN chael from Figure with a simple scalar uiform (dithered quatizer of fiite support, which is suitable for implemetatio withi A/D ad D/A coverters. As log as overload does ot occur, the effect of applyig the scalar quatizer is equivalet to that of a additive oise chael. We show that the rate-distortio trade-off derived for sigma-delta modulatio i Sectio II remais valid with high probability, with a costat additive excess-rate pealty for usig scalar quatizatio. The purpose of this excess-rate is to esure that a overload evet, which jeopardizes the stability of the ( system, occurs with low probability. The stochastic model we assume for the iput process allows us to tackle the issue of stability i a systematic ad rigourous maer, ad the tradeoff betwee the excess-rate ad the overload probability is aalytically determied. Clearly, a sigma-delta modulator ca oly perform well if overload errors are rather rare. Our stability aalysis i Sectio III is based o avoidig overload evets w.h.p., ad does ot aim to cosider the effect of such evets o the distortio oce they occur. I geeral, the overload probability of the scheme described i Sectio III decreases double expoetially with the excess-rate of the quatizer w.r.t. the mutual iformatio. Thus, takig a excess rate of bits will usually yield a sufficietly low overload probability. However, sigma-delta quatizers are ofte employed with a oe-bit quatizer. I this case, the overload error probability caot be very low. Cosequetly, the desiger would eed to guaratee that the effect of overload errors is local i time, ad does ot drive the system out of stability. There are various restrictios oe ca place o C(Z i pursuit of the latter goal. The issue of maitaiig stability whe overload errors are uavoidable is outside the scope of this paper. Nevertheless, we stress that our mai result is of great relevace to this settig, as it shows that the filter C(Z should be chose as the optimal MMSE predictio filter ofx } from its oisy past uder the stability esurig restrictios. II. MAIN RESUT We begi by itroducig some basic otatio that will be used i the sequel. For a discrete sigal c }, the Z-trasform is defied as C(Z c Z, ad the Fourier trasform as = C( C(Z Z=e j = = c e j. For a discrete (real statioary process X } with zero-mea ad autocorrelatio fuctio R X [k] E(X +k X we defie the power-spectral desity (PSD as the Fourier trasform of the autocorrelatio fuctio S X ( R X [k]e jk. k= The PSD of a cotiuous statioary process is defied i a aalogous maer. Assume X (t is a cotiuous statioary bad-limited Gaussia process with zero mea ad variaceσx, whose PSD is zero for all frequecies f > f max, but otherwise ukow. The Nyquist samplig rate for this process is f max samples per secod. Sice our focus here is o quatizatio of oversampled sigals, we assume thatx (t is sampled uiformly with rate of f max samples per secod for some >. The obtaied sampled process X } is therefore a discrete statioary Gaussia process with zero mea ad variace σx

4 4 N N ( 0,σ H( X C(Z N U U +N ˆX Fig.. The test chael correspodig to the sigma-delta modulatio architecture, with the sigma-delta quatizer replaced by a AWGN chael. The iput is assumed to be over-sampled at times the Nyquist rate. N N ( 0,σ H( X U U +N + V ˆX C(Z Fig.. The test chael correspodig to the architecture, with the quatizer replaced by a AWGN chael. The iput is assumed to be over-sampled at times the Nyquist rate. X C(Z C(Z N U N N ( 0,σ U +N C(Z H( ˆX Fig. 3. A test chael correspodig to the sigma-delta modulatio with pre-filter C(Z ad post-filter from Figure.. This test-chael is equivalet to that C(Z whose PSD is zero for all / [/,/], but otherwise ukow. Our goal is to characterize the rate-distortio tradeoff obtaied by a sigma-delta modulator, modeled as the test chael from Figure, whose iput is X }. To that ed, we establish a equivalece betwee the performace obtaied by this test chael for ay statioary bad-limited Gaussia process with variaceσx ad the performace obtaied by the test chael from Figure, which models a compressio system, for a statioary flat bad-limited Gaussia process with variace σx. The performace of the latter is ow well uderstood [8], ad, as we shall show, ca be traslated to a simple characterizatio of the performace of sigma-delta modulatio. First, we recall the derivatio of the distortios attaied by the test chaels from Figure ad Figure, ad the scalar mutual iformatio I(U ;U + N ad I(U ;U +N betwee the iput ad output of the additive white Gaussia oise (AWGN chaels embedded withi the two test chaels. The test chaels i Figure ad Figure do ot immediately iduce a output distributio from which a radom quatizatio codebook with rate I(U ;U + N ad MSE distortio D ca be draw. The reaso for this is the sequetial ature of the compressio, which seems to coflict with the eed of usig high-dimesioal quatizers, as required for attaiig a quatizatio error distributed as N with compressio rate I(U ;U +N. Fortuately, this difficulty, which is also preset i decisio feedback equalizatio for itersymbol iterferece chaels, ca be overcome with the help of a iterleaver [8] [0]. Thus, the scalar mutual iformatio I(U ;U +N ca ideed be iterpreted as the compressio rate eeded to achieve the distortio attaied by the test chaels i Figure ad Figure. We elaborate further about this i subsectio II-B. Moreover, i Sectio III we show that I(U ;U +N is closely related to the required quatizatio rate i a sigma-delta modulator that applies a uiform scalar quatizer of fiite support. We begi with the test chael i Figure, that correspods

5 5 to sigma-delta modulatio, with the sigma-delta quatizer replaced by a AWGN chael with zero mea ad variace σ. The filter C(Z is assumed to be strictly causal. Propositio : For ay Gaussia statioary process X } with variaceσx whose PSD is zero for all / [/,/], the test chael from Figure achieves MSE distortio D = σ / / C( d, ad its scalar mutual iformatio satisfies I(U ;U +N = ( log + C( d + σ X σ. Proof. From Figure, we have that ad therefore U U +N = X = X c N, (3 +(δ c N, where δ is the discrete idetity filter. Usig the fact that X } is a low-pass process, passig it through the filter H( has o effect, ad hece ˆX = h (U = X +N +h (δ c N. The MSE distortio attaied by the test chael from Figure is therefore D = E(X ˆX = σ / C( d. / The scalar mutual iformatio betwee the quatizer s iput U ad output U +N is give by I(U ;U +N = h(u +N = ( log + E(U h(n (4, (5 σ where (4, as well as (5, follow from the statistical idepedece of N ad U. Usig (3, the variace of U is E(U = σx +σ C( d. (6 Substitutig (6 ito (5 establishes the secod part of the propositio. Next, we aalyze the test chael i Figure, that correspods to compressio with the quatizer replaced by a AWGN chael with zero mea ad variace σ. As i the test chael of Figure, the filter C(Z is strictly causal. The distortio correspodig to this test chael, as well as I(U ;U +N, were already foud i [8, Theorem ] for the special case where C(Z is the optimal MMSE ifiite legth predictio filter of X from all past samples of the process X +N }. The All logarithms i this paper are take to base. followig straightforward propositio characterizes the rate ad distortio for ay choice of the causal filter C(Z ad ay value of σ. Propositio : For a Gaussia statioary process X } with variace σx ad PSD SX σx for / ( = 0 for / < <, (7 the test chael from Figure achieves MSE distortio D = σ, ad its scalar mutual iformatio satisfies I(U ;U +N = ( log + C( d + σ X / σ C( d. / Proof. From Figure, we have that U V = X = U Substitutig (8 i (9 yields V c V (8 +N +c V (9 = X +N. (0 Usig the fact that X } is a low-pass process, as before, we obtai ˆX = h (X = X +N +h N. ( Sice N } is AWGN with variace σ, the variace of the filtered process h N is σ /. Thus, D = E(X ˆX = σ. As i the aalysis of the test chael from Figure, the scalar mutual iformatio betwee U give by ad U I(U ;U +N = ( log + Now, substitutig (0 i (8 gives U = (δ c X ad the variace of U is therefore E(U = + = σ X / / + N E(U σ c N, SX ( C( d SN ( C( d C( d + σ. C( d. is ( (3

6 6 Substitutig (3 ito ( establishes the secod part of the propositio. Remark : I propositios ad we derived the scalar mutual iformatio betwee the iput ad output of the AWGN test chaels embedded i Figures ad, respectively. As will become clear i Sectio III, the scalar mutual iformatio is closely related to the required quatizatio rate whe a scalar memoryless quatizer is used withi the sigma-delta or modulator. I [8], [0], the directed iformatio was show to be related to the required quatizatio rate whe the quatizer is followed by a etropy coder. Here, we do ot cosider applyig etropy codig to the quatizer s output as we require that the desiged modulator be robust to the statistics of the iput process, whereas etropy codig is very sesitive to the process statistics. Moreover, if the desig of a A/D (or D/A is cosidered, the appropriate merit for the modulator s complexity is the umber of quatizatio levels withi the scalar quatizer, which are ot reduced by icorporatig a etropy coder. Our mai result ow follows immediately from Propositios ad. Theorem : et X } be ay Gaussia statioary process with variaceσx whose PSD is zero for all / [/,/], let X } be a flat low-pass Gaussia statioary process with PSD as i (7, ad let C(Z be a strictly causal filter. The test chael from Figure with σ = D / / C( d, ad the test chael from Figure with σ = D, both achieve MSE distortio D ad their scalar mutual iformatio satisfy I(U ;U +N = log ( + + σ X D = I(U / ;U C( d / +N C( d. This theorem idicates that for ay statioary bad-limited Gaussia process with variace σx, the sigma-delta test chael from Figure achieves exactly the same rate-distortio trade-off as that of the test chael from Figure with a statioary flat bad-limited Gaussia iput with the same variace, provided that the AWGN variaces are scaled accordig to (. Thus, Theorem provides a uified framework for aalyzig the performace of sigma-delta modulatio ad. A great advatage offered by such a uified framework, is that ay result kow for ca be traslated to a correspodig result for sigma-delta modulatio, ad vice versa. Theorems ad Corollary below costitute two importat examples of such results. Theorem : et X } be a Gaussia statioary process with variace σx whose PSD is zero for all / [/,/] ad let C be a family of strictly causal filters. Defie the virtual process S } as a Gaussia statioary process with PSD as i (7, ad the virtual process W } as a Gaussia i.i.d. radom process statistically idepedet of S } with variace D, D > 0. et σ D = mi C(Z C E(S c (S +W CD(Z = argmie(s c (S +W. C(Z C If the filter C(Z i the sigma-delta test chael from Figure belogs to C ad the MSE distortio attaied by this test chael is D, the I(U ;U +N ( log with equality if C(Z = C D (Z. + σ D D, (4 Theorem states that for a target distortio D, the sigmadelta filter which miimizes the required compressio rate is the optimal liear time-ivariat MMSE estimator, withi the class of costraits C, for S from the past of the oisy process S +W }. For example, if C cosists of all strictly causal fiite-impulse respose (FIR filters of legth p, the optimal filter C(Z is the optimal predictor of S from the samples S + W,...,S p + W p }, which ca be easily calculated i closed-form. The optimal sigma-delta filter desig problem was studied by several authors, uder various assumptios [], [6], [] [5]. However, to the best of our kowledge, the simple expressio from Theorem for the optimal filter as the optimal predictor of S from the past of S + W } is ovel. The refereces most relevat to Theorem, are perhaps [3] ad [4], [5]. I [3], Spag ad Schultheiss formulated a optimizatio problem for fidig the best FIR filter with p coefficiets i a sigma-delta modulator with a scalar quatizer, uder a fixed overload probability. Their optimizatio problem ca be solved umerically, but o closed form solutio was give. I [4] ad [5] the desig of a optimal ucostraied sigma-delta filter was studied, uder the assumptio of a fixed scalar quatizer which ca oly be scaled i order to cotrol the overload probability. Equatios that characterize the optimal filter were derived. However, the obtaied expressios usually yield filters with a ifiite umber of taps, ad do ot provide the solutio to the costraied problem. It is also worth metioig that for the case of a statioary Gaussia process X } with = (samplig at the Nyquist rate ad kow PSD, the optimal ifiite legth filter uder the assumptio of high-resolutio quatizatio is kow to equal the optimal predictio filter of X from its (clea past []. As already metioed i the itroductio, the high-resolutio assumptio ever holds whe > ad therefore this result is iapplicable for over-sampled sigals. Proof of Theorem. By Propositio, if the test chael

7 7 from Figure achieves MSE distortio D, we must have σ = D / / C( d. By Theorem, the correspodig mutual iformatio I(U ;U + N is equal to the mutual iformatio I(U ;U +N i the test chael from Figure withx = S,N = W adσ = D. Thus, I(U ;U = log ( +N = I(U + E(S c (S +W D ;U +N, (5 where we have used (8, (0, ad (, to arrive at (5. It follows that amog all filters ic, the filter that miimizes (5 is CD (Z, ad that it attais (4 with equality. It is iterestig to ote [8] that sice W } is a i.i.d. process with variace D ad C(Z is strictly causal, the mutual iformatio (5 ca also be writte as I(U ;U +N ( = log E(S +W c (S +W. D (6 Thus, the optimal predictor of S from the past of S +W } is idetical to the optimal predictor of S +W from its past samples. Whe C(Z is take as the (uique ifiite order optimal oe-step predictio filter of S + W from its past samples, the predictio error variace is the etropy power of the process S +W } [6], which equals / log(ss(+ Dd = ( D( + σ X. (7 D Moreover, the ifiite order predictio error E pred S +W c (S +W is i this case a white process. This, together with (7 implies that for the optimal ucostraied sigma-delta filter C(Z we must have S E pred( C( ( D+S S ( ( / = ( D + σ X, [, (8 D Combiig (6, (7, ad (8 yields the followig corollary. Corollary : et X } be a Gaussia statioary process with variaceσx whose PSD is zero for all / [/,/]. If the test chael from Figure attais MSE distortio D, the I(U ;U +N ( log + σ X. (9 D with equality if ad oly if C(Z is a strictly causal filter satisfyig ( (/ + σ X C( D [ =, ] / (0 (+ σ XD / [, ], ad σ = D / / C( d = D (/. (+ σ XD Remark : Note that the existece of a strictly causal filter C(Z which satisfies (0 is guarateed by Wieer s spectralfactorizatio theory [6] due to the readily verified fact that log C( d =. The optimal filter iduces a two-level frequecy respose for C(. I [0] Østergaard ad Zamir used sigma-delta modulatio to attai the optimal multiple-descriptio ratedistortio regio. Iterestigly, the optimal filter C(Z i their scheme also iduced a two-level respose for C(. We also ote that the optimality of the ucostraied filter specified by (0 ca be deduced as a special case of [4, Sectio IV]. Remark 3: Note that for the optimal ucostraied filter C(Z specified by (0, the pre- ad post-filters from Figure 3 have o effect as log as the PSD of the iput sigal X } is zero for all / [/,/]. However, filters with a fiite umber of taps will ever icur a flat frequecy respose i the iterval[/,/], ad for such filters the systems from Figure ad Figure 3 will ot be equivalet. Remark 4: The output of the test chael from Figure (as well as that from Figure is of the form ˆX = X + E, where E has zero mea ad variace D, ad is statistically idepedet of X. This estimate ca be further improved by applyig scalar MMSE estimatio for X from ˆX. This boils dow to producig the estimate ˆ X = α ˆX, where α = σ X σ X +D. Cosequetly, the obtaied MSE distortio is reduced to D = E(X α ˆX = σ X D σ X +D. It is straightforward to verify [7] that with this improvemet, the sigma-delta test chael from Figure with C(Z ad σ as specified i Corollary attais I(U ;U +V = σ log(, X D which is the optimal rate-distortio fuctio for a statioary Gaussia sourcex } with PSD as i (7. It follows that the sigma-delta test chael from Figure with C(Z ad σ as specified i Corollary is miimax optimal for the class of all statioary Gaussia sources with variace σx ad PSD that

8 8 equals zero for all / [/,/], i.e., o other system ca achieve MSE distortio D with a smaller compressio rate, uiversally for all sources i this class. A. Extesio to Frequecy-Weighted Mea Squared Error Distortio I may applicatios, higher values of distortio are acceptable i certai frequecy bads while smaller distortio is permitted i other bads. The MSE distortio measure is iadequate for such scearios, ad a commoly used distortio measure, that (partially captures such perceptual effects, is the frequecy-weighted mea squared error (FWMSE criterio. Uder this criterio, the distortio is measured as D FWMSE P(S E (d, ( where P( is a o-egative weight fuctio, ad S E ( is the PSD of the error process E X ˆX. Note that for P( =, [,, the FWMSE criterio reduces to the MSE oe. The ext theorem shows that the costraied optimal sigma-delta filter uder the FWMSE criterio is the optimal costraied predictio filter of a oisy process defied accordig to the weight fuctio P(. Theorem 3: et X } be a Gaussia statioary process with variaceσx whose PSD is zero for all / [/,/], P( a weightig fuctio which forms a valid PSD, ad C a family of strictly causal filters. Defie the virtual process S } as a Gaussia statioary process with PSD SX FWMSE σx P( for / ( = 0 for / < <, ( ad the virtual process W } as a Gaussia i.i.d. radom process statistically idepedet of S } with variace D FWMSE, D FWMSE > 0. et σd FWMSE = mi E(S c (S +W C(Z C CD FWMSE (Z = argmie(s c (S +W. C(Z C If the filter C(Z i the sigma-delta test chael from Figure belogs to C ad the FWMSE distortio w.r.t. P( attaied by this test chael is D FWMSE, the ( I(U ;U +N log with equality if C(Z = C D FWMSE (Z. + σ D FWMSE D FWMSE Sketch of proof:. The proof is fairly similar to that of Theorem. Thus, for brevity, we omit the full proof ad oly highlight its mai steps: Repeat the derivatio of Propositio where ow the MSE distortio is replaced by FWMSE distortio. Note that this has o effect o I(U ;U +N. Repeat the derivatio of Propositio where the PSD of the iput process is (, rather tha (7. Note that this chages I(U ;U + N, but has o effect o the attaied distortio., It follows that the test chael for the process S } uder MSE distortio is equivalet to the sigmadelta test chael with iput X } uder FWMSE distortio, i the sese that i both chaels if the attaied distortio is D FWMSE (uder the appropriate distortio measure, the I(U ;U = log ( +N = I(U + E(S c (S +W D FWMSE ;U +N B. Sigma-Delta Modulatio with a Iterleaved Vector Quatizer The goal of this short subsectio is to give the test chael from Figure a operatioal meaig, i.e., to show how the AWGN from the figure ca be replaced with a lossy source code of rate R = I(U ;U + N whose icurred quatizatio oise is distributed as N. As already metioed, the key idea is to use a iterleaver [8] [0], as we ow recall. Assume that X }, the iput process to the sigmadelta modulator, has a decayig memory, such that X is essetially idepedet of all samples of sufficietly distat samplig times. I order to compress a N-dimesioal vector x = [X,...,X N ], cotaiig N cosecutive samples of the process X }, we first split it ito K vectors x k = [X (km+,...,x km ], k =,...,K, where M N/K. Now, we ca apply K parallel sigma-delta modulators, oe for each such vector, where the oly couplig betwee the K parallel systems is through the quatizatio step, which is applied joitly o all of them, as depicted i Figure 4. By our assumptio that X } has decayig memory, if M is large eough the K iputs that eter the quatizer Q( = [Q (,...,Q K ( ] are i.i.d. radom variables distributed as U from Figure. For large eough K, stadard rate-distortio argumets imply that there exists a vector quatizer with rate I(U ;U +N that iduces quatizatio oise distributed as N. III. SIGMA-DETA MODUATION WITH A SCAAR UNIFORM QUANTIZER The previous subsectio showed how to replace the AWGN chael i Figure with a vector quatizer whose rate is arbitrarily close to R = I(U ;U + N ad whose iduced quatizatio oise is distributed as N. The iputs to the vector quatizer are vectors of i.i.d. Gaussia compoets. Thus, ay off-the-shelf rate distortio optimal vector quatizer for a i.i.d. Gaussia source ca be used. The total sigma-delta compressio system that is obtaied is therefore simple i the sese that it oly requires the vector quatizer to be good for quatizig a i.i.d. Gaussia source, which is.

9 x x M C(Z C(Z N, N K, U, U K, Q( Q (. U, +N, Q K( U K, +N K, H( H( ˆx ˆx K 9 Fig. 4. K parallel sigma-delta modulators coupled by a K-dimesioal quatizer Q(. a stadard task, rather tha requirig it to be a good quatizer for a bad-limited Gaussia source. However, the sigma-delta modulatio architecture is maily used for A/D ad D/A coversio. I such applicatios, vector quatizatio is typically out of the questio, ad simple uiform scalar quatizers of fiite support are used. For such quatizers, the quatizatio error is composed of two mai factors []: graular errors that correspod to the quatizatio error i the case where the iput sigal falls withi the quatizer s support, ad overload errors that correspod to the case where the iput sigal falls outside the quatizer s support. Due to the feedback loop, iheret to the sigma-delta modulator, errors of the latter kid, whose magitude is ot bouded, may have a disastrous effect as they jeopardize the system s stability. I order to avoid such errors, the support of the quatizer has to be chose appropriately. As the support of the quatizer determies its rate for a give quatizatio resolutio, the overload probability ca be cotrolled by icreasig the quatizatio rate. We shall show that, give that overload errors did ot occur, the quatizatio oise ca be modeled as a additive oise. Thus, the test chael from Figure accurately predicts the total distortio icurred by a sigma-delta A/D (or D/A i this case. Moreover, the overload probability is a doubly expoetially decreasig fuctio of R I(U ;U + N, where R are the umber of levels i the scalar quatizer. Thus, fixig the desired overload error probability as P ol, we may achieve the MSE distortio predicted by the test chael from Figure (characterized i Propositio with a scalar quatizer whose ( rate( is I(U ;U +N +δ(p ol, where δ(p ol = O log log P ol. et Q R,σ ( be a uiform quatizer with quatizatio step σ ad R quatizatio levels, such that the quatizer support is [Γ/,Γ/, where Γ R σ, see Figure 5. Our goal is to aalyze the distortio ad overload probability attaied by a sigma-delta modulator that uses a Q R,σ ( quatizer, as a fuctio of R ad σ. As discussed i Sectio I-B, oe ca try to limit the effect of overload errors by placig various costraits o C(Z. Here, we restrict attetio to cotrollig the overload probability Q R,σ (x Fig. 5. A illustratio of Q R,σ ( for R = ad σ = /3. Clearly, if we employ the scalar sigma-delta modulator o a log eough iput sequece, a overload evet will evetually occur. As discussed above, the effects of overload errors ca be amplified due to the feedback loop, ad i this case the average MSE may sigificatly grow. We therefore split the iput sequece ito fiite blocks of legthn, ad iitialize the memory of the filter C(Z with zeros before the begiig of each ew block. This makes sure that the effect of a overload error i the origial system is restricted to the block where it occurs. The aalysis is made much simpler by itroducig a subtractive dither [7]. Namely, let Z } be a sequece of i.i.d. radom variables uiformly distributed over the iterval [ σ /, σ /. I order to quatize U, we add Z to it before applyig the quatizer, ad subtract Z afterwards, such that the obtaied result is Q R,σ (U + ( Z Z. Addig ad subtractig U, we get U + Q R,σ (U +Z (U +Z, ad the quatizatio error is therefore N Q R,σ (U 3 +Z (U +Z (3 The mai result i this sectio is the followig. Theorem 4: et D be the MSE distortio attaied by the test chael i Figure with a filter C(Z of fiite legth, ad I(U ;U +N the scalar mutual iformatio betwee the iput ad output of the AWGN chael i the same figure. x

10 0 X C(Z N U Z + Q R,σ ( U +N H( ˆX Fig. 6. A sigma-delta modulator with a dithered scalar uiform quatizer. The iput is assumed to be over-sampled at [ times the Nyquist rate, ad the dither sequece Z } is assumed to be a i.i.d. sequece of radom variables uiformly distributed over the iterval σ σ /, / ad statistically idepedet X }. For ay 0 < P ol < the scalar sigma-delta modulator from Figure 6 applied o a sequece of N cosecutive source samples with quatizatio rater = I(U ;U +N +δ(p ol attais MSE distortio smaller tha D(+o N (, P ol give that overload did ot occur. I additio, the overload probability is smaller tha P ol, where o N ( 0 as N icreases, ad δ(p ol ( log 3 l P ol. (4 N Proof. et Q σ Z (x be the operatio of roudig x to the earest poit i the (ifiite lattice σ Z. It is easy to verify that for ay x [Γ/,Γ/ we have ( Q R,σ (x = Q σ σ σ Z x+. (5 Applyig (3 therefore yields that if overload did ot occur i the th sample, i.e., if U +Z Γ/, we have ( N = Q U σ Z +Z + 3σ ( U +Z + 3σ. (6 Dealig with the overload evet of the quatizer directly is rather ivolved. Istead, as doe i [8], we first cosider a referece system with a ifiite-support quatizer (R = ad aalyze its performace. If the magitude of the iput to the ifiite-support quatizer ever exceeds Γ/ withi the processed block, the clearly the referece system is completely equivalet to the origial system withi this block. Thus, it suffices to fid the average distortio of the referece system ad the probability that the iput to its quatizer exceeds Γ/ withi a block. I what follows we will therefore assume that the quatizatio oise is give by (6 regardless of whether or ot U + Z Γ/, ad accout for the overload probability later. Assumig that the dither sequece Z } is draw statistically idepedet of the process X }, the Crypto emma, see, e.g. [7, emma 4..], implies that N } is a i.i.d. sequece of radom variables uiformly distributed over the iterval [ σ /, σ /, statistically idepedet of X }. Note that N has zero mea ad variace σ. Followig this reasoig, the referece sigma-delta data coverter depicted i Figure 6 (with a ifiite-support quatizer is equivalet to the test chael from Figure withn Uiform ([ σ /, σ / istead of N N(0,σ. Thus, the average MSE distortio attaied by the referece scalar sigma-delta modulator from Figure 6 is as give i Propositio up to a multiplicative factor of+o N ( that accouts for edge effects. These effects are the by-product of the operatio of ullig the filter memory at the begiig of each ew block, which icurs temporal o-statioarities. I particular, if the filter C(Z has taps, the oly after samples withi the block the statistics of the process U } will coverge to its statioary distributio. However, if the block legth is sufficietly large w.r.t. the filter legth ad the iverse of the MSE distortio, the ifluece of these effects vaishes. Next, we tur to aalyze the probability that a overload error occurs withi a block of legth N, as a fuctio of R adi(u ;U +N. Sice this evet is equivalet to the evet that at the referece system some iput to the quatizer exceeds Γ/ i magitude withi the block, it suffices to upper boud the probability of the latter evet. Assume the referece scalar sigma-delta modulator from Figure 6 is applied to a vector x = [X,...,XN ] of N cosecutive samples of the process X }, where the memory of the filter C(Z is iitialized with zeros. Defie the evet O k Uk + Nk > Γ/} ad the evet O N k O k. By the uio boud, we have P ol Pr(O N Pr(O k. (7 k= The radom variable Uk + Nk = Xk + (δ k c k Nk is a liear combiatio of a Gaussia radom variable Xk ad statistically idepedet uiform radom variables Nk }. I [9, emma 4] the probability that a radom variable of this type exceeds a certai threshold was bouded i terms of its variace. Applyig this boud to Uk +Nk

11 yields Pr ( Uk +N k > Γ/ exp σ R 8E(U k Γ +Nk = exp 8 ( E(Uk +E(Nk, where i the last equality we have used the defiitio of Γ ad the fact that Uk ad Nk are statistically idepedet. Equivaletly, we may write Pr (O k exp σ R 8σ (+ E(U k σ (R ( = exp 3 log + E(U k σ = exp 3 } (RI(U k ;Uk +Nk, (8 where we have used (5 i the last equality. Substitutig (8 ito (7 gives N P ol exp 3 } (RI(U k ;U k +N k. (9 k= Note that E(Uk = σx k +σ m= c k is mootoically odecreasig i k ad is give by (6 for values of k that are greater tha the legth of the filter c k. We ca therefore further boud (9 as P ol N exp 3 } (RI(U ;U +N, (30 where I ( U ;U +N is as give i Propositio. To summarize, we have show that the referece system achieves the same MSE distortio D as characterized by Propositio up to a +o N ( multiplicative term, ad that the probability that oe of the quatizer iput samples exceeds Γ/ i magitude withi a block of legth N, is bouded by (30. For our origial system whose quatizer has fiite support of [Γ/, Γ/, this meas that the overload probability is also upper bouded by the RHS of (30. Moreover, the average distortio it achieves if overload did ot occur is the same as that of the referece system coditioed o the evet that O did ot occur. Deote this coditioed expected distortio by D O ad the expected distortio coditioed o the evet that O did occur by D O. For the referece system, we have D(+o N ( = Pr(OD O +Pr(OD O Pr(OD O, ad therefore D O D(+o N( P ol. This shows that the scalar sigma-delta system from Figure 6, whose quatizer has limited support [Γ/, Γ/, with R = I(U ;U +N +δ(p ol achieves the same average MSE distortio as the test chael from Figure up to a multiplicative factor of (+o N (/(P ol, with block error } } probability smaller tha N exp 3 } δ. Thus, Propositio characterizes the rate-distortio tradeoff achieved by the scalar sigma-delta system up to the aforemetioed factor ad a costat rate pealty δ(p ol, that depeds o the target overload error probability. To be more precise, for ay 0 < P ol <, takig the rate pealty as i (4 guaratees that the overload error probability is smaller tha P ol. ACKNOWEDGEMENTS We thak Ja Østergaard ad Ram Zamir for their valuable commets o a earlier versio of this mauscript. REFERENCES [] N. S. Jayat ad P. Noll, Digital codig of waveforms: priciples ad applicatios to speech ad video. Eglewood Cliffs, NJ: Pretice-Hall, 984. [] R. A. McDoald, Sigal-to-oise ad idle chael performace of differetial pulse code modulatio systems particular applicatios to voice sigals, Bell System Techical Joural, vol. 45, o. 7, pp. 3 5, 966. [3] R. Gray ad D. Neuhoff, Quatizatio, IEEE Trasactios o Iformatio Theory, vol. 44, o. 6, pp , Oct 998. [4] C. Cutler, Differetial quatizatio of commuicatio sigals, 95, US Patet,605,36. [5], Trasmissio systems employig quatizatio, 960, US Patet,97,96 (filed 954. [6] S. Tewksbury ad R. Hallock, Oversampled, liear predictive ad oiseshapig coders of order >, IEEE Trasactios o Circuits ad Systems, vol. 5, o. 7, pp , Jul 978. [7] M. Derpich, Optimal source codig with sigal trasfer fuctio costraits, Ph.D. dissertatio, Uiversity of Newcastle, 009. [8] R. Zamir, Y. Kochma, ad U. Erez, Achievig the Gaussia ratedistortio fuctio by predictio, IEEE Trasactios o Iformatio Theory, vol. 54, o. 7, pp , July 008. [9] T. Guess ad M. K. Varaasi, A iformatio-theoretic framework for derivig caoical decisio-feedback receivers i Gaussia chaels, IEEE Trasactios o Iformatio Theory, vol. IT-5, pp , Ja 005. [0] J. Østergaard ad R. Zamir, Multiple-descriptio codig by dithered delta-sigma quatizatio, IEEE Trasactios o Iformatio Theory, vol. 55, o. 0, pp , Oct 009. [] P. Noll, O predictive quatizig schemes, The Bell System Techical Joural, vol. 57, o. 5, pp , May 978. [] M. A. Gerzo ad P. G. Crave, Optimal oise shapig ad dither of digital sigals, i Audio Egieerig Society Covetio 87, 989. [3] H. Spag III ad P. Schultheiss, Reductio of quatizig oise by use of feedback, IRE Trasactios o Commuicatios Systems, vol. 0, o. 4, pp , Dec 96. [4] M. Derpich, E. Silva, D. Quevedo, ad G. Goodwi, O optimal perfect recostructio feedback quatizers, IEEE Trasactios o Sigal Processig, vol. 56, o. 8, pp , Aug 008. [5] M. Derpich ad J. Østergaard, Improved upper bouds to the causal quadratic rate-distortio fuctio for Gaussia statioary sources, IEEE Trasactios o Iformatio Theory, vol. 58, o. 5, pp , May 0. [6] T. Berger, Rate distortio theory: A mathematical basis for data compressio. Eglewood Cliffs, NJ: Pretice-Hall, 97. [7] R. Zamir, attice Codig for Sigals ad Networks. Cambridge: Cambridge Uiversity Press, 04. [8] A. Be-Yishai ad O. Shayevitz, The Gaussia chael with oisy feedback: Near-capacity performace via simple iteractio, 04. [Olie]. Available: [9] O. Ordetlich ad U. Erez, Precoded iteger-forcig uiversally achieves the MIMO capacity to withi a costat gap, IEEE Trasactios o Iformatio Theory, vol. 6, o., pp , Ja 05.