EE-597 Notes Quantization

Transcription

1 EE-597 Notes Quantization Phil Schniter June, 4 Quantization Given a continuous-time and continuous-amplitude signal (t, processing and storage by modern digital hardware requires discretization in both time and amplitude, as accomplished by an analogto-digital converter (ADC. We will typically wor with discrete-time quantities (n (indeed by time variable n which we assume were sampled ideally and without aliasing. Here we discuss various ways of discretizing the amplitude of (n so that it may be represented by a finite set of numbers. This is generally a lossy operation, and so we analyze the performance of various quantization schemes and design quantizers that are optimal under certain assumptions. A good reference for much of this material is [].. Memoryless Scalar Quantization Memoryless scalar quantization of continuous-amplitude variable is the mapping of to output y when lies within interval X : { < + },,,..., L. The are called decision thresholds, and the number of quantization levels is L. The quantization operation is written y Q(. When {y,..., y L }, quantizer is called midtread, else midrise. Quantization error defined q : Q( If is a r.v. with pdf p ( and liewise for q, then quantization error variance is σ q E{q } q p q (qdq ( ( Q( p (d + ( y p (d ( A special quantizer is the uniform quantizer: y + y, for,,...,l, +, for finite, +, L+. c P. Schniter, 999

2 (a (b Q( Q( y y q( q( Figure : (a Uniform and (b non-uniform quantization Q( and quantization error q(. Uniform Quantizer Performance for large L: For bounded input (, ma, uniform quantization with + and L ma, and with y / and y + / (for >, the quantization error is well approimated by a uniform distribution for large L: { / q /, p q (q else. Why? As L, p ( is constant over X for any. Since q y X, it follows that p q (q X will have uniform distribution for any. With (, ma and with and y as specified, q ( /, /] for all (see Fig.. Hence, for any, { / q ( /, /], p q (q X else. q / / Figure : Quantization error for bounded input and midpoint y. c P. Schniter, 999

3 In this case, from (, σ q / [ q 3 q / dq 3 ] / / ( (3 If we use R bits to represent each discrete output y and choose L R, then σq ( ma L 3 ma R and SNR [db] log ( σ σ q ( log 3 σ R ma ( 6.R log 3 ma σ. Recall that the epression above is only valid for σ small enough to ensure (, ma. For larger σ, the quantizer overloads and the SNR decreases rapidly. Eample. (SNR for Uniform Quantization of Uniformly-Distributed Input: For uniformly distributed, can show ma/σ 3, so that SNR 6.R. Eample. (SNR for Uniform Quantization of Sinusoidal Input: For a sinusoidal, can show ma/σ, so that SNR 6.R (Interesting since sine waves are often used as test signals. Eample.3 (SNR for Uniform Quantization of Gaussian Input: Though not truly bounded, Gaussian might be considered as approimately bounded if we choose ma 4σ and ignore residual clipping. In this case SNR 6.R MSE-Optimal Memoryless Scalar Quantization Though uniform quantization is convenient for implementation and analysis, non-uniform quantization yeilds lower σ q when p ( is non-uniformly distributed. By decreasing q( for frequently occuring (at the epense of increasing q( for infrequently occuring, the average error power can be reduced. Lloyd-Ma Quantizer: MSE-optimal thresholds { } and outputs {y } can be determined given an input distribution p (, and the result is the Lloyd-Ma quantizer. Necessary conditions on { } and {y } are σ q for {,...,L} and σ q y for {,...,L}. Using (, /b b a f(d f(b, /a b f(d f(a, and above, a σ q ( y p ( ( y p ( y +y, {...L},, L+, (4 σq + + ( y p (d y y + p (d, {...L} (5 p (d It can be shown that above are sufficient for global MMSE when log p (/, which holds for uniform, Gaussian, and Laplace pdfs, but not Gamma. Note: c P. Schniter, 999 3

4 optimum decision thresholds are halfway between neighboring output values, optimum output values are centroids of the pdf within the appropriate interval, i.e., are given by the conditional means y E{ X } p ( X d p (, X Pr( X d, + + p (d. p (d Iterative Procedure to Find { } and {y }:. Choose ŷ.. For,...,L, given ŷ and ˆ, solve (5 for ˆ +, given ŷ and ˆ +, solve (4 for ŷ +. end; 3. Compare ŷ L to y L calculated from (5 based on ˆ L and L+. Adjust ŷ accordingly, and go to step. Lloyd-Ma Performance for large L: As with the uniform quantizer, can analyze quantization error performance for large L. Here, we assume that the pdf p ( is constant over X for {,...,L}, the input is bounded, i.e., (, ma for some (potentially large ma. So with assumption and definition we can write and thus, from (, σ q becomes p ( p (y for, y X : +, P : Pr{ X } p (y (where we require P σ q P + ( y d. (6 For MSE-optimal {y }, now σ q y P + ( y d y + +, which is epected since the centroid of a flat pdf over X is simply the midpoint of X. Plugging y into epression (6, σ q P [ ( / + / 3] + 3 P 3 [ (+ / / 3 ( / + / 3] P 3 ( 3 P. (7 c P. Schniter, 999 4

5 Note that for uniform quantization (, the epression above reduces to the one derived earlier. Now we minimize σ q w.r.t. { }. The tric here is to define α : 3 p (y so that σ q p (y 3 α 3. For p ( constant over X and y X, α 3 p (y y p (d C we have the following constrained optimization problem: min α 3 s.t. α C. {α } This may be solved using Lagrange multipliers. (a nown constant, Aside. (Optimization via Lagrange Multipliers: Consider the problem of minimizing N-dimensional real-valued cost function J(, where (,,..., N t, subject to M <N real-valued equality constraints f m( a m, m,..., M. This may be converted into an unconstrained optimization of dimension N +M by introducing additional variables λ (λ,..., λ M t nown as Lagrange multipliers. The uncontrained cost function is J u(, λ J( + X m λ m`fm( a m, and necessary conditions for its minimization are Ju(, λ J( + X m λ m fm( (8 Ju(, λ fm( am for m,..., M. (9 λ The typical procedure used to solve for optimal is the following:. Equations for n, n,..., N, in terms of {λ m} are obtained from (8.. These N equations are used in (9 to solve for the M optimal λ m. 3. The optimal {λ m} are plugged bac into the N equations for n, yielding optimal { n}. Necessary conditions are ( ( l, α 3 α + λ α C λ 3α l α l λ/3 l ( ( α 3 + λ α C α C, λ which can be combined to solve for λ: λ ( 3 C C λ 3. L Plugging λ bac into the epression for α l, we find α l C /L, l. c P. Schniter, 999 5

6 Using the definition of α l, the optimal decision spacing is C L 3 p (y 3 p (d L 3 p (y, and the minimum quantization error variance is σq min L p (y 3 ( p (y 3 3 p (d. ( ma 3 3 p (d L 3 p (y An interesting observation is that α 3 l, the lth interval s optimal contribution to σ q, is constant over l..3 Entropy-Coding Binary Scalar Encoding: Previously we have focused on the memoryless scalar quantizer y Q(, where y taes a value from a set of L reconstruction levels. By coding each quantizer output in binary format, we transmit (store the information at a rate (cost of R log L bits/sample. If, for eample, L 8, then we transmit at 3 bits/sample. Say we can tolerate a bit more quantization error, e.g., as results from L 5. We hope that this reduction in fidelity reduces our transmission requirements, but with this simple binary encoding scheme we still require R 3 bits/sample! Idea Bloc Coding: Let s assign a symbol to each bloc of 3 consecutive quantizer outputs. We need a symbol alphabet of size 5 3 5, which is adequately represented by a 7-bit word ( 7 8. Transmitting these words requires only 7/3.33 bits/sample! Idea Variable Length Coding: Assume some of the quantizer outputs occur more frequently than others. Could we come up with an alphabet consisting of short words for representing frequent outputs and longer words for infrequent outputs that would have a lower average transmission rate? Eample.4 (Variable Length Coding: Consider the quantizer with L 4 and output probabilities indicated on the right. Straightforward -bit encoding requires average bit rate of bits/sample, while the variable length code on the right gives average R P P n bits/sample. output P code y.6 y.5 y 3. y 4.5 (Just enough information about Entropy: Q: Given an arbitrarily comple coding scheme, what is the minimum bits/sample required to transmit (store the sequence {y(n}? A: When random process {y(n} is i.i.d., the minimum average bit rate is R min H y + ɛ, c P. Schniter, 999 6

7 Notes: where H y is the entropy of random variable y(n in bits: H y P log P, and ɛ is an arbitrarily small positive constant [], [3]. Entropy obeys H y log L. The left inequality occurs when P for some, while the right inequality occurs when P /L for every. The term entropy refers to the average information of a single random variable, while the term entropy rate refers to a sequence of random variables, i.e., a random process. When {y(n} is not independent (the focus of later sections, a different epression for R min applies. Though the minimum rate is well specified, the construction of a coding scheme which always achieves this rate is not. Eample.5 (Entropy of Variable Length Code: Recalling the setup of Eample.4, we find that H y `.6 log log.5 +. log. +.5 log.5.49 bits. Assuming i.i.d. {y(n}, we have R min.49 bits/sample. Compare to the variable length code on the right which gave R.55 bits/sample. output P code y.6 y.5 y 3. y 4.5 Huffman Encoding: Given quantizer outputs y or fied-length blocs of outputs (y j y y l, the Huffman procedure constructs variable length codes that are optimal in certain respects [3]. For eample, when the probabilities of {P } are powers of / (and {y(n} is i.i.d., the entropy rate of a Huffman encoded output attains R min. Aside. (Huffman Procedure (Binary Case:. Arrange ouput probabilities P in decreasing order and consider them as leaf nodes of a tree.. While there eists more than one node: Merge the two nodes with smallest probability to form a new node whose probability equals the sum of the two merged nodes. Arbitrarily assign and to the two branches of the merging pair. 3. The code assigned to each output is obtained by reading the branch bits sequentially from root note to leaf node Eample.6 (Huffman Encoder Attaining R min: In Aside., a Huffman code was constructed for the output probabilities listed on the right. Here H y `.5 log log log.5.75 bits, so that R min.75 bits/sample (with the i.i.d. assumption. Since the average bit rate for the Huffman code is also R bits/sample, Huffman encoding attains R min for this output distribution. output P code y.5 y.5 y 3.5 y 4.5 c P. Schniter, 999 7

8 .4 Quantizer Design for Entropy Coded Sytems Say that we are designing a system with a memoryless quantizer followed by an entropy coder, and our goal is to minimize the average transmission rate for a given σq (or vice versa. Is it optimal to cascade a σq-minimizing (Lloyd-Ma quantizer with a rate-minimizing code? In other words, what is the optimal memoryless quantizer if the quantized outputs are to be entropy coded? A Compander Formulation: To determine the optimal quantizer,. consider a companding system: a memoryless nonlinearity c( followed by uniform quantizer,. find c( minimizing entropy H y for a fied error variance σ q. c( ma L Figure 3: Compander curve: nonuniform input regions mapped to uniform output regions (for subsequent uniform quantization. First we must epress σ q and H y in terms of c(. Fig. 3 suggests that, for large L, the slope c ( : dc(/d obeys c ( X ma/l, so that we may write ma Lc (. X Assuming large L, the σq -approimation (7 can be transformed as follows. σ q P ma 3L ma 3L ma 3L P c ( X p ( c ( X since P p ( X for large L p ( c ( d. c P. Schniter, 999 8

9 Similarly, H y P log P ( p ( log p ( X p ( log p ( X p (log p (d }{{} h : differential entropy h log ma L constant + p (d }{{} p ( log X p (log c (d p (log ma Lc ( }{{} d + p (log c (d ( Entropy-Minimizing Quantizer: Our goal is to choose c( which minimizes the entropy rate H y subject to fied error variance σq. We employ a Lagrange technique again, minimizing the cost p (log c (d under the constraint that the quantity ma p ( ( c ( d equals a constant C. This yields the unconstrained cost function ( J u c (, λ [ p (log c ( + λ (p ( ( c ( ] C d, ( } {{ } φ(c (,λ with scalar λ, and the unconstrained optimization problem becomes min J u(c (, λ. c (,λ The following technique is common in variational calculus [4]. Say a ( minimizes a (scalar cost J ( a(. Then for any (well-behaved variation η( from this optimal a (, we must have ɛ J( a ( + ɛη( ɛ where ɛ is a scalar. Applying this principle to our optimization problem, we search for c ( such that η(, ɛ J ( u c ( + ɛη(, λ. ɛ From ( we find (using log a log e log e a J u ma ɛ ɛ ɛ φ( c ( + ɛη(, λ ɛ d ( ( [p (log ɛ (e log e c ( + ɛη( + λ p ( ( c ( + ɛη( ] C d ɛ ma [ ( η( ( 3η( ] log (ep ( c ( + ɛη( λp ( c ( + ɛη( d p ( ( c ( [ log (e λ ( c ( ] η(d c P. Schniter, ɛ

10 and to allow for any η( we require log (e λ ( c ( c λ (. log e }{{} a constant! Applying the boundary conditions, { c(ma ma c( } c(. Thus, for large-l, the quantizer that minimizes entropy rate H y for a given quantization error variance σq is the uniform quantizer. Plugging c( into (, the rightmost integral disappears and we have ma H y uniform h log, }{{ L } and using the large-l uniform quantizer error variance approimation (3, H y uniform h log ( σ q. It is interesting to compare this result to the information-theoretic minimal average rate for transmission of a continuous-amplitude memoryless source of differential entropy h at average distortion σ q [, ]: R min h log ( πe σ q. Comparing the previous two equations, we find that (for a continous-amplitude memoryless source uniform quantization prior to entropy coding requires log ( πe 6.55 bits/sample more than the theoretically optimum transmission scheme, regardless of the distribution of. Thus,.55 bits/sample (or.5 db using the 6.R relationship is the price paid for memoryless quantization..5 Adaptive Quantization Previously have considered the case of stationary source processes, though in reality the source signal may be highly non-stationary. For eample, the variance, pdf, and/or mean may vary significantly with time. Here we concentrate on the problem of adapting uniform quantizer stepsize to a signal with unnown variance. This is accomplished by estimating the input variance ˆσ (n and setting the quantizer stepsize appropriately: (n φ ˆσ (n. L Here φ is a constant that depends on the distribution of the input signal whose function is to prevent input values greater than σ (n from being clipped by the quantizer (see Fig. 4; comparing to non-adaptive step size relation ma /L, we see that φ ˆσ (n ma. c P. Schniter, 999

11 Q( φˆσ L p ( ˆσ φ ˆσ Figure 4: Adaptive quantization stepsize (n φ ˆσ /L As long as the reconstruction levels {y } are the same at encoder and decoder, the actual values chosen for quantizer design are arbitrary. Assuming integer values as in Fig. 4, the quantization rule becomes (n (n midrise, y(n ( (n midtread. (n AQF and AQB: Fig. 5 shows two structures for stepsize adaptation: (a adaptive quantization with forward estimation (AQF and (b adaptive quantization with bacward estimation (AQB. The advantage of AQF is that variance estimation may be accomplished more accurately, as it is operates directly on the source as opposed to a quantized (noisy version of the source. The advantage of AQB is that the variance estimates do not need to be transmitted as side information for decoding. In fact, practical AQF encoders transmit variance estimates only occasionally, e.g., once per bloc. (a (b (n y(n y(n (n (n y(n y(n Q channel Q Q channel Q (n Variance Estimator channel Variance Estimator Variance Estimator Figure 5: (a AQF and (b AQB. Bloc Variance Estimation: When operating on finite blocs of data, the structures in Fig. 5 c P. Schniter, 999

12 perform variance estimation as follows: Bloc AQF: ˆσ (n N Bloc AQB: ˆσ (n N N (n i i N ( y(n i (n i i N is termed the learning period and its choice may significantly impact quantizer SNR performance: choosing N too large prevents the quantizer from adapting to the local statistics of the input, while choosing N too small results in overly noisy AQB variance estimates and ecessive AQF side information. Fig. 6 demonstrates these two schemes for two choices of N. (a AQF, N (b AQB, N (c AQF, N (d AQB, N Figure 6: Bloc AQF and AQB estimates of σ (n superimposed on (n for N 8, 3. SNR acheived: (a.6 db, (b 8.8 db, (c. db, and (d 8.8 db. c P. Schniter, 999

13 Recursive Variance Estimation: The recursive method of estimating variance is as follows Recursive AQF: ˆσ (n αˆσ (n + ( α (n Recursive AQB: ˆσ (n αˆσ (n + ( α (y(n (n. where α is a forgetting factor in the range < α < and typically near to. This leads to an eponential data window, as can be seen below. Plugging the epression for ˆσ (n into that for ˆσ (n, ˆσ (αˆσ (n α (n + ( α (n + ( α (n α ˆσ ( (n + ( α (n + α (n. Then plugging ˆσ (n into the above, ( ˆσ (n α( αˆσ (n 3 + ( α (n 3 + ( α (n + α (n α 3ˆσ ( (n 3 + ( α (n + α (n + α (n 3. Continuing this process N times, we arrive at ˆσ (n ( α N i α i (n i + α N ˆσ (n N. Taing the limit as N, α < ensures that ˆσ (n ( α α i (n i. i References [] N.S. Jayant and P. Noll, Digital Coding of Waveforms, Englewood Cliffs, NJ: Prentice-Hall, 984. [] T. Berger, Rate Distortion Theory, Englewood Cliffs, NJ: Prentice-Hall, 97. [3] T.A. Cover and J.A. Thomas, Elements of Information Theory, New Yor, NY: Wiley, 99. [4] A.P. Sage and C.C. White, III, Optimum Systems Control, nd Ed., Englewood Cliffs, NJ: Prentice- Hall, 977. c P. Schniter, 999 3

14 (a AQF, lambda (b AQB, lambda (c AQF, lambda (d AQB, lambda Figure 7: Eponential AQF and AQB estimates of σ (n superimposed on (n for λ.9,.99. (a.5 db, (b 8. db, (c. db, (d 4. db. c P. Schniter, 999 4