C.M. Liu Perceptual Signal Processing Lab College of Computer Science National Chiao-Tung University

Transcription

1 Quantization C.M. Liu Perceptual Signal Processing Lab College of Computer Science National Chiao-Tung University Office: EC538 (03)

2 Contents Quantization Problem Uniform Quantization Adaptive Quantization Nonuniform Quantization Entropy-Coded Quantization Vector Quantization Rate-Distortion Function

3 Quantization 3 Definition: The process of representing a large possibly infinite set of values with a smaller set. Example: Source: Real numbers in the [-10.0, 10.0] Quantization Q(x) x+0.5 [-10.0, -10.0] {-10, -9,, -1, 0, 1,,, 9, 10} Scalar vs. vector quantization Scalar: applied to scalars Vector: applied to vectors

4 The Quantization Process 4 Two aspects Encoder mapping Map a range of values to a codeword If source is analog A/D converter Knowledge of the source can help pick more appropriate ranges Decoder mapping Map the codeword to a value in the range If output is analog D/A converter Knowledge of the source distribution can help pick better approximations Quantizer encoder + decoder

5 5 Quantization Example

6 6 Quantizer Input-Output Map

7 Quantization Problem Formulation 7 Input: X random variable f X (x) probability density function (pdf) Output: {b i } i0..m decision boundaries {y i } i1..m reconstruction levels Discrete processes are often approximated by continuous distributions E.g.: Laplacian model of pixel difference If source is unbounded, then first/last decision boundaries ±

8 Quantization Error 8 Q( x) yi iff bi 1 < x b i Mean squared quantization error σ q M i 1 ( x Q( x) ) b b i i 1 f dx ( x y ) f dx i X X Quantization error is a.k.a. Quantization noise Quantizer distortion

9 Quantization Problem Formulation () 9 Bit rates w/ fixed-length codewords R log M E.g.: M 8 R 3 Quantizer design problem Given: input pdf f X (x) & M Find: decision boundaries {b i } and Reconstruction levels {y i } Such that: MSQE is minimized

10 Quantization Problem Formulation (3) Bit rates w/ variable-length codewords R depends on boundary selection Example 10 M i i i y P l R 1 ) ( i i b b X i dx x f y P 1 ) ( ) ( M i b b X i i i dx x f l R 1 1 ) (

11 Quantization Problem Formulation (4) 11 Rate-optimization formulation Given: Distortion constraint σ q D* Find: {b i }, {y i }, binary codes Such that: R is minimized Distortion-optimization formulation Given: Rate constraint R R* Find: {b i }, {y i }, binary codes Such that: σ q is minimized

12 Uniform Quantizer 1 All intervals of the same size i.e., boundaries are evenly spaced (Δ) Outer intervals may be an exception Reconstruction Usually the midpoint is selected Midrise quantizer Zero is not an output level Midtread quantizer Zero is an output level

13 Midrise vs. Midtread Quantizer 13 Midrise Midtread

14 Uniform Quantization of Uniform Source 14 Input Uniform [-X max, X max ] Output M-level uniform quantizer Δ X max M σ M iδ i 1 1 Δ q x Δ dx ( i 1) Δ i 1 X max 1

15 Alternative MSQE Derivation 15 Consider quantization error instead: q x Q(x) q [-Δ/, Δ/] σ q 1 Δ Δ Δ q dq Δ 1 SNR(dB) 10log 10log σs 10log σq ( M ) 0log ( X ) 1 ( X ) ( n max 1 Δ ) 6.0n db 10log 10 max 1 1 X M max

16 Examples (8 1,, 3 bits/pixel) 16 Darkening, contouring & dithering

17 Uniform Quantization of Nonuniform Sources 17 Example nonuniform source: x [-100, 100], P(x [-1, 1]) 0.95 Problem Design an 8-level quantizer The naïve approach leads to 95% of samples values represented by two numbers: -1.5 and 1.5 Max quantization error (QE) 1.5 Min QE 11.5 (!) Consider an alternative Step 0.3 Max QE 98.5, however 95% of the time QE < 0.15

18 Optimizing MSQE 18 Numerically solvable for specific PDF

19 19 Example Optimum Step Sizes

20 0 QE for 3-bit Midrise Quantizer

21 Overload/Granular Regions 1 The step selection Tradeoff between overload noise and granular noise. max granular value f1, f1 4 "4σ loading" stdev

22 Variance Mismatch Effects

23 3 Variance Mismatch Effects ()

24 Distribution Mismatch Effects 4 8-level quantizers, SNR

25 Adaptive Quantization 5 Idea Instead of a static scheme, adapt to the actual data: Mean, variance, pdf Forward adaptive (off-line) Divide source in blocks Analyze block statistics Set quantization scheme Side channel Backward adaptive (on-line) Adaptation based on quantizer output No side channel necessary

26 Forward Adaptive Quantization (FAQ) 6 Choosing block size Too large Not enough resolution Increased latency Too small More side channel information Assuming a mean of zero Variance estimate: ˆ σ 1 N 1 q x i 0 n + i N

27 Speech Quantization Example 7 16-bit speech 3-bit fixed

28 Speech Quantization Example () 8 16-bit speech 3-bit FAQ Block 18 samples 8-bit variance quantization

29 FAQ Refinement 9 So far we assumed uniform pdf Refinement Assume uniform pdf but Record min/max values for each block Example: Sena image 8x8 blocks 3-bit quantization Overhead 16/8x8 0.5 bits/pixel

30 FAQ Refinement Example 30 Original: 8 bits/pixel Quantized: 3.5 bits/pixel

31 Backward Adaptive Quantization (BAQ) 31 Observation Only encoder sees input Adaptation can only be based on quantized output Problem How do we deduce mismatch information from output only? It is possible, if we know the pdf and we are very patient

32 Jayant Quantizer 3 Idea If input falls in the outer levels Expand step size If input falls in the inner levels Contract step size The product of expansions & contraction should be 1 Multipliers: M k If input S n-1 falls in the k th interval, then step is multiplied by M k Inner M k < 1, outer M k > 1 Δ Δ n M l ( n 1) n 1

33 33 3-bit Jayant Quantizer Output Levels

34 Jayant Example 34 M 0 M 4 0.8, M 1 M M M 6 1.0, M 3 M 7 1., Δ Input:

35 Picking Jayant Multipliers Δ min / Δ max to prevent under/overflow. Adaption speed affected by γ. 35 > M k k k M k P l k k l k M k k k P M k N n M k n P l l l M Let N n P M M M k k k k k k k k k , where, where 1, 1 1 γ γ γ

36 Jayant Example 36 Ringing

37 Jayant Performance 37 Expands more rapidly than contracts to avoid overload errors. Robustness over changing input statistics.

38 Non-uniform Quantization 38 Idea: Pick the boundaries such that error is minimized i.e., smaller/bigger step for smaller/bigger values e.g.:

39 39 Non-uniform Quantization-- pdfoptimized Quantization Problem: Given f X, minimize MSQE: σ q M i 1 b b i i 1 ( x y ) f dx i X Set derivative w.r.t. y j to zero and solve for y j : y j b b j j 1 b b j j 1 x f f X X ( x) dx ( x)dx Set derivative w.r.t. b j to zero and solve for b j : ( y y ) b + j j+ 1 j

40 Non-uniform Quantization-- Lloyd- 40 Max Algorithm Observation: Circular dependency b/w b j and y j Lloyd/Max/Lukaszewics/Steinhaus approach: Solve the two iteratively until an acceptable solution is found Example:

41 Non-uniform Quantization-- Lloyd- 41 Max Algorithm () Boundaries: { b 1, b,, b M/-1 } b 0 0, b M/-1 MAX_INPUT Reconstruction levels: { y 1, y,, y M/-1 } y 1 b b 0 1 x f X b1 ( x) dx f ( x)dx b 0 X One equation, two unknowns: b 1, y 1 Pick a value for b 1 (e.g. b 1 1), solve for y 1 and continue: y x f ( x) dx f ( x)dx y + b1 y1 and so on until all { b n } and { y m } are found b b 1 X b b 1 X

42 Non-uniform Quantization-- Lloyd- 4 Max Algorithm (3) Terminating condition: y M yˆm ε yˆ M b + M 1 ym 1 y M b b M M / / 1 x f X bm / ( x) dx f ( x)dx b M / 1 X Else: pick a different b 1 & repeat

43 43 Non-uniform Quantization Example: pdf- Optimized Quantizers Significant improvement over the uniform quantizer.

44 Lloyd-Max Quantizer Properties Mean OUTPUT Mean INPUT. Variance OUTPUT Variance INPUT 3. MSQE: σ q M x σ y j P b j 1 j 1 [ x b ] 4. If N is a random variable representing QE E [ XN ] σ q 5. Quantizer output and QE are orthogonal (uncorrelated) E [ Q X ) N b, b, K, ] 0 ( 0 1 b M j

45 Mismatch Effects 45 4-bit Laplacian pdf-optimized quantizer

46 Companded Quantization (CQ) 46 Compressor Uniform quantizer Expander

47 47 CQ Example: Compressor

48 48 CQ Example: Uniform Quantizer

49 49 CQ Example: Expander

50 50 CQ Example: Equivalent Non-uniform Quantizer

51 Vector Quantization x, y Definition x [ x(1) x()... x( N )] y [ y(1) y()... y( N )] x( i), y( i), 1 i N : real random variables : N - dimensional random vector the vector y has a special distribution in that it may only take one of L (deterministic ) vector values in N R

52 Vector Quantization (c.1) Vector quantization y Q() x the vector quantization of x may be viewed as a pattern recognition problem involving the classification of the outcomes of the random variable x into a discrete number of categories or cell in N-space in a way that optimizes some fidelity criterion, such as mean square distortion.

53 Vector Quantization (c.) VQ Distortion L D P( x Ck) E{ d( x, yk) x C k k} 1 d( x, y ) are typically the distance measures k N in R, including l, l, l norm 1 VQ Optimization minimize the average distortion D.

54 Vector Quantization (c.3) Two conditions for optimality Nearest Neighbor Selection Q( x) y iff k d( x, y k, ) x C k d( x, y j ) for k j,1 j L. minimize average distortion y k arg min arg y y min D k... x C arg min k > applied to partition the N-dimensional space into cell { C, k L} when the joint pdf y d ( x, y ) f x E { d ( x, y ) x C ( ξ... ξ ) d ξ... 1 n 1 d ξ k 1 f x () k N } is known.

55 55

56 Appendix C: Rate-Distortion Functions Introduction Rate-Distortion Function for a Gaussian Source Rate-Distortion Bounds Distortion Measure Methods

57 1. Introduction Considering question Given a source-user pair and a channel, under what conditions is it possible to design a communication system that reproduces the source output for the user with an average distortion that does not exceed some specified upper limit D? The capacity (C) of a communication channel. The rate distortion function ( R(D) )of a source-user pair. Rate-distortion function R(D) A communication system can be designed that achieves fidelity D if and only if the capacity of the channel that connects the source to user exceeds R(D). The lower limit for data compression to achieve a certain fidelity subject to a predetermined distortion measure D.

58 1. Introduction (cont.) Equations representations : Distortion D: D d ( q ) p ( x ) q ( y x ) ρ ( x, y ) dxdy M utual information: q ( y x ) I ( q ) p ( x ) q ( y x ) log dxdy q ( y ) Rate distortion function R(D): R ( D ) inf I ( q ), Q { q ( y x ): d ( q ) D } q Q D ρ ( x, y ): distortion measure for the source word x ( x 1,..., x ) reproduced as y (y,..., y ) ρ n n 1 ( x,y ) n ρ ( x, y ) t 1 The fam ily F ρ { ρ n, 1 n < } is called the single-letter fidelity criterion generated by ρ. n d t t 1 n

59 . Rate-Distortion Bounds Introduction Rate-Distortion Function for A Gaussian Source R(D) for a memoryless Gaussian source Source coding with a distortion measure Rate-Distortion Bounds Conclusions

60 3. Rate-Distortion Function for A Gaussian Source Rate-Distortion for a memoryless Gaussian source The minimum information rate (bpn) necessary to represent the output of a discrete-time, continuous-amplitude, memoryless stationary Gaussian source based on an MSE distortion measure per symbol. 1 log ( x D), σ Rg ( D) 0, 0 D σx D σ x

61 3. Rate-Distortion Function for A Gaussian Source (c.1) Source coding with a distortion measure (Shannon, 1959) There exists a coding scheme that maps the source output into codewords such that for any given distortion D, the minimum rate R(D) bpn is sufficient to reconstruct the source output with an average distortion that is arbitrarily close to D. Transform the R(D) to distortion-rate function D(R) D g ( R) 10log 10 R Expressin db D g σ x ( R) 6R + 10log 10 σ x

62 3. Rate-Distortion Function for A Gaussian Source (c.) Comparison between different quantizations

63 4. Rate-Distortion Bounds Source: Memoryless, continuous-amplitude source with zero mean and finite variance σ x with respect to the MSE distortion measure. Upper bound According to the theorem of Berger (1971), it implies that the Gaussian source requires the maximum rate among all other sources for a specified level of mean square distortion. 1 σx RD D R D D ( ) log g( ), 0 σx R DR ( ) D( R) σ g x

64 Lower bound (Shannon lower bound) where 4. Rate-Distortion Bounds (c.1) ) ( * * 1 ) ( log 1 ) ( ) ( x H R e R D ed x H D R π π D ed e D R e x H e x f x x x g x x x x * log 1 log 1 log 1 ) ( log 1 ) ( 1 ) ( For Gaussian source : σ π σ π σ π πσ σ ξ ξ ξ d f f x H x H n x n x def ) ( )log ( ) ( entropy ) :differential ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( * * R D R D R D D R D R D R g g

65 4. Rate-Distortion Bounds (c.) For Gaussian source, the rate-distortion, upper bound and lower bound are all identical to each other. The bound of differential entropy 10log 10log D * D D g * ( R) 6R 6[ H ( R) ( R) 6[ H 6[ R g g ( x) H( x)] ( D) R ( x) H( x)] ( D)] The differential entropy is upper boundedby g * H g ( x)

66 4. Rate-Distortion Bounds (c.3) Rate-distortion R(D) to channel capacity C For C > R g (D) The fidelity (D) can be achieved. For R(D)<C< R g (D) Achieve fidelity for stationary source May not achieve fidelity for random source For C<R(D) Can not be sure to achieve fidelity

67 4. Rate-Distortion Bounds (c.4)

68 5. Distortion Measure Methods { r } GG( m, σ, r) k exp c( x m) rc k and c Γ( 1/ r) For different r: r 1 Laplacian pdf r Gaussian pdf r 0 constant pdf r uniform pdf Γ( 3 / r) σ Γ( 1 / r)

69 Problems 69 Homeworks (Sayood 3 rd, pp ) 3, 6. References J.R. Deller, J.G. Proakis, and J.H.L. Hansen, Discrete-Time Processing of Speech Signals, IEEE Press