Quantization C.M. Liu Perceptual Signal Processing Lab College of Computer Science National Chiao-Tung University http://www.csie.nctu.edu.tw/~cmliu/courses/compression/ Office: EC538 (03)5731877 cmliu@cs.nctu.edu.tw
Contents Quantization Problem Uniform Quantization Adaptive Quantization Nonuniform Quantization Entropy-Coded Quantization Vector Quantization Rate-Distortion Function
Quantization 3 Definition: The process of representing a large possibly infinite set of values with a smaller set. Example: Source: Real numbers in the [-10.0, 10.0] Quantization Q(x) x+0.5 [-10.0, -10.0] {-10, -9,, -1, 0, 1,,, 9, 10} Scalar vs. vector quantization Scalar: applied to scalars Vector: applied to vectors
The Quantization Process 4 Two aspects Encoder mapping Map a range of values to a codeword If source is analog A/D converter Knowledge of the source can help pick more appropriate ranges Decoder mapping Map the codeword to a value in the range If output is analog D/A converter Knowledge of the source distribution can help pick better approximations Quantizer encoder + decoder
5 Quantization Example
6 Quantizer Input-Output Map
Quantization Problem Formulation 7 Input: X random variable f X (x) probability density function (pdf) Output: {b i } i0..m decision boundaries {y i } i1..m reconstruction levels Discrete processes are often approximated by continuous distributions E.g.: Laplacian model of pixel difference If source is unbounded, then first/last decision boundaries ±
Quantization Error 8 Q( x) yi iff bi 1 < x b i Mean squared quantization error σ q M i 1 ( x Q( x) ) b b i i 1 f dx ( x y ) f dx i X X Quantization error is a.k.a. Quantization noise Quantizer distortion
Quantization Problem Formulation () 9 Bit rates w/ fixed-length codewords R log M E.g.: M 8 R 3 Quantizer design problem Given: input pdf f X (x) & M Find: decision boundaries {b i } and Reconstruction levels {y i } Such that: MSQE is minimized
Quantization Problem Formulation (3) Bit rates w/ variable-length codewords R depends on boundary selection Example 10 M i i i y P l R 1 ) ( i i b b X i dx x f y P 1 ) ( ) ( M i b b X i i i dx x f l R 1 1 ) (
Quantization Problem Formulation (4) 11 Rate-optimization formulation Given: Distortion constraint σ q D* Find: {b i }, {y i }, binary codes Such that: R is minimized Distortion-optimization formulation Given: Rate constraint R R* Find: {b i }, {y i }, binary codes Such that: σ q is minimized
Uniform Quantizer 1 All intervals of the same size i.e., boundaries are evenly spaced (Δ) Outer intervals may be an exception Reconstruction Usually the midpoint is selected Midrise quantizer Zero is not an output level Midtread quantizer Zero is an output level
Midrise vs. Midtread Quantizer 13 Midrise Midtread
Uniform Quantization of Uniform Source 14 Input Uniform [-X max, X max ] Output M-level uniform quantizer Δ X max M σ M iδ i 1 1 Δ q x Δ dx ( i 1) Δ i 1 X max 1
Alternative MSQE Derivation 15 Consider quantization error instead: q x Q(x) q [-Δ/, Δ/] σ q 1 Δ Δ Δ q dq Δ 1 SNR(dB) 10log 10log 10 10 σs 10log σq ( M ) 0log 10 10 ( X ) 1 ( X ) ( n max 1 Δ ) 6.0n db 10log 10 max 1 1 X M max
Examples (8 1,, 3 bits/pixel) 16 Darkening, contouring & dithering
Uniform Quantization of Nonuniform Sources 17 Example nonuniform source: x [-100, 100], P(x [-1, 1]) 0.95 Problem Design an 8-level quantizer The naïve approach leads to 95% of samples values represented by two numbers: -1.5 and 1.5 Max quantization error (QE) 1.5 Min QE 11.5 (!) Consider an alternative Step 0.3 Max QE 98.5, however 95% of the time QE < 0.15
Optimizing MSQE 18 Numerically solvable for specific PDF
19 Example Optimum Step Sizes
0 QE for 3-bit Midrise Quantizer
Overload/Granular Regions 1 The step selection Tradeoff between overload noise and granular noise. max granular value f1, f1 4 "4σ loading" stdev
Variance Mismatch Effects
3 Variance Mismatch Effects ()
Distribution Mismatch Effects 4 8-level quantizers, SNR
Adaptive Quantization 5 Idea Instead of a static scheme, adapt to the actual data: Mean, variance, pdf Forward adaptive (off-line) Divide source in blocks Analyze block statistics Set quantization scheme Side channel Backward adaptive (on-line) Adaptation based on quantizer output No side channel necessary
Forward Adaptive Quantization (FAQ) 6 Choosing block size Too large Not enough resolution Increased latency Too small More side channel information Assuming a mean of zero Variance estimate: ˆ σ 1 N 1 q x i 0 n + i N
Speech Quantization Example 7 16-bit speech 3-bit fixed
Speech Quantization Example () 8 16-bit speech 3-bit FAQ Block 18 samples 8-bit variance quantization
FAQ Refinement 9 So far we assumed uniform pdf Refinement Assume uniform pdf but Record min/max values for each block Example: Sena image 8x8 blocks 3-bit quantization Overhead 16/8x8 0.5 bits/pixel
FAQ Refinement Example 30 Original: 8 bits/pixel Quantized: 3.5 bits/pixel
Backward Adaptive Quantization (BAQ) 31 Observation Only encoder sees input Adaptation can only be based on quantized output Problem How do we deduce mismatch information from output only? It is possible, if we know the pdf and we are very patient
Jayant Quantizer 3 Idea If input falls in the outer levels Expand step size If input falls in the inner levels Contract step size The product of expansions & contraction should be 1 Multipliers: M k If input S n-1 falls in the k th interval, then step is multiplied by M k Inner M k < 1, outer M k > 1 Δ Δ n M l ( n 1) n 1
33 3-bit Jayant Quantizer Output Levels
Jayant Example 34 M 0 M 4 0.8, M 1 M 5 0.9 M M 6 1.0, M 3 M 7 1., Δ 0 0.5 Input: 0.1-0. 0. 0.1-0.3 0.1 0. 0.5 0.9 1.5
Picking Jayant Multipliers Δ min / Δ max to prevent under/overflow. Adaption speed affected by γ. 35 > M k k k M k P l k k l k M k k k P M k N n M k n P l l l M Let N n P M M M k k k k k k k k k 0 0 0 0 0 0 1 1, where, where 1, 1 1 γ γ γ
Jayant Example 36 Ringing
Jayant Performance 37 Expands more rapidly than contracts to avoid overload errors. Robustness over changing input statistics.
Non-uniform Quantization 38 Idea: Pick the boundaries such that error is minimized i.e., smaller/bigger step for smaller/bigger values e.g.:
39 Non-uniform Quantization-- pdfoptimized Quantization Problem: Given f X, minimize MSQE: σ q M i 1 b b i i 1 ( x y ) f dx i X Set derivative w.r.t. y j to zero and solve for y j : y j b b j j 1 b b j j 1 x f f X X ( x) dx ( x)dx Set derivative w.r.t. b j to zero and solve for b j : ( y y ) b + j j+ 1 j
Non-uniform Quantization-- Lloyd- 40 Max Algorithm Observation: Circular dependency b/w b j and y j Lloyd/Max/Lukaszewics/Steinhaus approach: Solve the two iteratively until an acceptable solution is found Example:
Non-uniform Quantization-- Lloyd- 41 Max Algorithm () Boundaries: { b 1, b,, b M/-1 } b 0 0, b M/-1 MAX_INPUT Reconstruction levels: { y 1, y,, y M/-1 } y 1 b b 0 1 x f X b1 ( x) dx f ( x)dx b 0 X One equation, two unknowns: b 1, y 1 Pick a value for b 1 (e.g. b 1 1), solve for y 1 and continue: y x f ( x) dx f ( x)dx y + b1 y1 and so on until all { b n } and { y m } are found b b 1 X b b 1 X
Non-uniform Quantization-- Lloyd- 4 Max Algorithm (3) Terminating condition: y M yˆm ε yˆ M b + M 1 ym 1 y M b b M M / / 1 x f X bm / ( x) dx f ( x)dx b M / 1 X Else: pick a different b 1 & repeat
43 Non-uniform Quantization Example: pdf- Optimized Quantizers Significant improvement over the uniform quantizer.
Lloyd-Max Quantizer Properties 44 1. Mean OUTPUT Mean INPUT. Variance OUTPUT Variance INPUT 3. MSQE: σ q M x σ y j P b j 1 j 1 [ x b ] 4. If N is a random variable representing QE E [ XN ] σ q 5. Quantizer output and QE are orthogonal (uncorrelated) E [ Q X ) N b, b, K, ] 0 ( 0 1 b M j
Mismatch Effects 45 4-bit Laplacian pdf-optimized quantizer
Companded Quantization (CQ) 46 Compressor Uniform quantizer Expander
47 CQ Example: Compressor
48 CQ Example: Uniform Quantizer
49 CQ Example: Expander
50 CQ Example: Equivalent Non-uniform Quantizer
Vector Quantization x, y Definition x [ x(1) x()... x( N )] y [ y(1) y()... y( N )] x( i), y( i), 1 i N : real random variables : N - dimensional random vector the vector y has a special distribution in that it may only take one of L (deterministic ) vector values in N R
Vector Quantization (c.1) Vector quantization y Q() x the vector quantization of x may be viewed as a pattern recognition problem involving the classification of the outcomes of the random variable x into a discrete number of categories or cell in N-space in a way that optimizes some fidelity criterion, such as mean square distortion.
Vector Quantization (c.) VQ Distortion L D P( x Ck) E{ d( x, yk) x C k k} 1 d( x, y ) are typically the distance measures k N in R, including l, l, l norm 1 VQ Optimization minimize the average distortion D.
Vector Quantization (c.3) Two conditions for optimality Nearest Neighbor Selection Q( x) y iff k d( x, y k, ) x C k d( x, y j ) for k j,1 j L. minimize average distortion y k arg min arg y y min D k... x C arg min k > applied to partition the N-dimensional space into cell { C, k L} when the joint pdf y d ( x, y ) f x E { d ( x, y ) x C ( ξ... ξ ) d ξ... 1 n 1 d ξ k 1 f x () k N } is known.
55
Appendix C: Rate-Distortion Functions Introduction Rate-Distortion Function for a Gaussian Source Rate-Distortion Bounds Distortion Measure Methods
1. Introduction Considering question Given a source-user pair and a channel, under what conditions is it possible to design a communication system that reproduces the source output for the user with an average distortion that does not exceed some specified upper limit D? The capacity (C) of a communication channel. The rate distortion function ( R(D) )of a source-user pair. Rate-distortion function R(D) A communication system can be designed that achieves fidelity D if and only if the capacity of the channel that connects the source to user exceeds R(D). The lower limit for data compression to achieve a certain fidelity subject to a predetermined distortion measure D.
1. Introduction (cont.) Equations representations : Distortion D: D d ( q ) p ( x ) q ( y x ) ρ ( x, y ) dxdy M utual information: q ( y x ) I ( q ) p ( x ) q ( y x ) log dxdy q ( y ) Rate distortion function R(D): R ( D ) inf I ( q ), Q { q ( y x ): d ( q ) D } q Q D ρ ( x, y ): distortion measure for the source word x ( x 1,..., x ) reproduced as y (y,..., y ) ρ n n 1 ( x,y ) n ρ ( x, y ) t 1 The fam ily F ρ { ρ n, 1 n < } is called the single-letter fidelity criterion generated by ρ. n d t t 1 n
. Rate-Distortion Bounds Introduction Rate-Distortion Function for A Gaussian Source R(D) for a memoryless Gaussian source Source coding with a distortion measure Rate-Distortion Bounds Conclusions
3. Rate-Distortion Function for A Gaussian Source Rate-Distortion for a memoryless Gaussian source The minimum information rate (bpn) necessary to represent the output of a discrete-time, continuous-amplitude, memoryless stationary Gaussian source based on an MSE distortion measure per symbol. 1 log ( x D), σ Rg ( D) 0, 0 D σx D σ x
3. Rate-Distortion Function for A Gaussian Source (c.1) Source coding with a distortion measure (Shannon, 1959) There exists a coding scheme that maps the source output into codewords such that for any given distortion D, the minimum rate R(D) bpn is sufficient to reconstruct the source output with an average distortion that is arbitrarily close to D. Transform the R(D) to distortion-rate function D(R) D g ( R) 10log 10 R Expressin db D g σ x ( R) 6R + 10log 10 σ x
3. Rate-Distortion Function for A Gaussian Source (c.) Comparison between different quantizations
4. Rate-Distortion Bounds Source: Memoryless, continuous-amplitude source with zero mean and finite variance σ x with respect to the MSE distortion measure. Upper bound According to the theorem of Berger (1971), it implies that the Gaussian source requires the maximum rate among all other sources for a specified level of mean square distortion. 1 σx RD D R D D ( ) log g( ), 0 σx R DR ( ) D( R) σ g x
Lower bound (Shannon lower bound) where 4. Rate-Distortion Bounds (c.1) ) ( * * 1 ) ( log 1 ) ( ) ( x H R e R D ed x H D R π π D ed e D R e x H e x f x x x g x x x x * log 1 log 1 log 1 ) ( log 1 ) ( 1 ) ( For Gaussian source : σ π σ π σ π πσ σ ξ ξ ξ d f f x H x H n x n x def ) ( )log ( ) ( entropy ) :differential ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( * * R D R D R D D R D R D R g g
4. Rate-Distortion Bounds (c.) For Gaussian source, the rate-distortion, upper bound and lower bound are all identical to each other. The bound of differential entropy 10log 10log 10 10 D * D D g * ( R) 6R 6[ H ( R) ( R) 6[ H 6[ R g g ( x) H( x)] ( D) R ( x) H( x)] ( D)] The differential entropy is upper boundedby g * H g ( x)
4. Rate-Distortion Bounds (c.3) Rate-distortion R(D) to channel capacity C For C > R g (D) The fidelity (D) can be achieved. For R(D)<C< R g (D) Achieve fidelity for stationary source May not achieve fidelity for random source For C<R(D) Can not be sure to achieve fidelity
4. Rate-Distortion Bounds (c.4)
5. Distortion Measure Methods { r } GG( m, σ, r) k exp c( x m) rc k and c Γ( 1/ r) For different r: r 1 Laplacian pdf r Gaussian pdf r 0 constant pdf r uniform pdf Γ( 3 / r) σ Γ( 1 / r)
Problems 69 Homeworks (Sayood 3 rd, pp. 70-71) 3, 6. References http://en.wikipedia.org/wiki/rate_distortion_theory. J.R. Deller, J.G. Proakis, and J.H.L. Hansen, Discrete-Time Processing of Speech Signals, IEEE Press