BASICS OF COMPRESSION THEORY

Transcription

1 BASICS OF COMPRESSION THEORY Why Compression? Task: storage and transport of multimedia information. E.g.: non-interlaced HDTV: 0x0x0x = Mb/s!! Solutions: Develop technologies for higher bandwidth Find ways to reduce the number of bits to transmit without compromising the information content

2 Types of Compression Lossless (entropy coding) Does not lose information (i.e., the signal is perfectly reconstructed after decompression) Produces a variable bit-rate It is not guaranteed to actually reduce the data size Lossy Loses some information (i.e., the signal is not perfectly reconstructed after decompression) Produces any desired constant bit-rate Lossless Compression

3 Data Sources Model the data production as a source of symbols. A source has a vocabulary of N symbols Each symbol represented with log N bits. speech: bits/sample: N = =, symbols color image: x bits/sample: N = =x0 symbols x image blocks: x bits/block: N = =0 symbols Lossless Compression Coding: translate each symbol in the vocabulary into a binary codeword. Codewords may have different binary lenghts. Example: symbols (a,b,c,d): a (00) Æ 000 b (0) Æ 00 c (0) Æ 0 d () Æ Goal: minimize the average symbol length

4 Average Symbol Length l(i) = binary length of i -th symbol Source emits M symbols (one every T seconds) Symbol i has been emitted m(i) times: M = Number of bits been emitted: L = Average length per symbol: L = Average bit rate: L / T Â Â N i= N i= Â m(i) l(i) N i = m(i) l(i) M m(i) Minimum Average Symbol Length Basic idea for reducing the average symbol length: assign shorter codewords to symbols that appear more frequently, longer codewords to symbols that appear less frequently Theorem: the average binary length of the the encoded symbols is always greater than or equal to the entropy H of the source What is the entropy of a source of symbols? We need some basic concept of statistic first...

5 Symbol Probabilities Probability P(i) of a symbol: corresponds to its relative frequency: m(i)/m Axioms of probability: 0 P(i) Â Average symbol length: N i = P(i) = N N L = Â m(i) M l(i) = Â P(i)l(i) i= ( ) i = Entropy Entropy (definition): H = - Â The entropy is a function of a given source of symbols (more precisely: of a probability distribution The entropy is highest (equal to log N ) if all symbols are equally probable N i = P(i)log P(i) The entropy is small (always 0) when some symbols that are much more likely to appear than other symbols

6 Example: N= { } {,0,0,0} H = - log ( ) = H = 0 Case Study : Huffman coding The Huffman code assignment procedure is based on a binary tree structure. The leaves of binary tree associated with the list of probabilities. Take the two smallest probabilities in the list and make the corresponding nodes siblings. Generate an intermediate node as their parent and label the branch from parent to one of the child nodes and the branch from parent to the other child 0.. Replace the probabilities and associated nodes in the list by the single new intermediate node with the sum of the two probabilities. If the list contains only one element, quit. Otherwise, go to step. Codeword formation: Follow the path from the root of the tree to the symbol, and accumulates the labels of all the branches.

7 N = symbols: {a,b,c,d,e,f,g,h} bits per symbol (N = =) P(a) = 0.0, P(b)=0.0, P(c)=0.0, P(d)=0.09, P(e)=0., P(f)=0., P(g)=0., P(h)=0. Average length per symbol (before coding): Entropy:H = -  i = Example  L = P(i) = i= bits/symbol P(i)log P(i) =. bits/symbol = 0. xl Average length per symbol (with Huffman coding): L Huf =. bits/symbol Efficiency of the code: H L Huf = 9%. Original codewords Huffman codewords 0 P(h)=0. 0 P(g)= P(f)= P(e)= P(d)=0.09 P(c)=0.0 P(b)= P(a)=

8 Huffman Coding (cont d) Probabilities are estimated from the relative frequencies To decode the bit stream, the decoder needs the Huffman table Decoding the bit stream: without coding: = bcbgfb with Huffman coding: = ghaed (variable-length) we can decode it because it is a prefix code (no codeword is the prefix of another codeword) Do we always reduce the length? No! We reduce the average symbol length Case Study : CCITT Group -D fax encoding Facsimile: bilevel image. x page scanned (left-to-right) at 00 dpi:. Mb ( pixels on each line) Coding strategy: Count each run of white/black pixels Encode the run-lengths with Huffman coding Note that the probabilities of run-lengths are not the same for black and white pixels Decompose each run-length n as n =kx+m (with 0 m<), and create Huffman tables for both k and m Special codewords for end-of-line, end-of-page, synchronization Average compression ratio on text documents: 0-to-

9 Lossy Compression Lossy Compression Goal: minimize the distortion for a given compression ratio Ideally, we would optimize the system based on perceptual distortion (difficult to compute) We ll need a few more concepts from statistics

10 Some Definitions x = the original value of a sample ˆ x = the sample value after compression and decompressio e = ˆ x - x = error (a.k.a. noise, a.k.a. distortion) s x = power (variance) of the signal s e = power (variance) of the error Signal-to-noise ratio :SNR = 0log 0 s ( x s e ) Lossy Compression: Simple Examples Subsampling : retain only samples on a subsampling grid (spatial/temporal) Has fundamental limitations: see sampling theorem Quantization: quantize with fewer levels N (larger quantization step D=V/N ) SNR PCM = 0log 0 s ( x s e ) = 0log 0 ( N a ) Can we do better than simple quantization (PCM)?

11 Differential PCM If the (n-)-th sample has value x(n-), what is the most probable value for x(n)? Simplest case: the predicted value for x(n) is x(n-) Differential PCM (DPCM): instead of quantizing and transmitting x(n), quantize and transmit the residual d(n)=x(n)-x(n-) Decoding: x ˆ (n) = x(n -) + d ˆ (n) where d ˆ (n) is the quantized version of d(n) We will show that SNR DPCM > SNR PCM More precisely: we ill compare SNR DPCM and SNR PCM when the same number of bits is used to quantize x(n) (in PCM) and d(n) (in DPCM) SNR of DPCM Coding What is the equivalent quantization error? e( n) = x ˆ ( n) - x( n) = x( n -) + d ˆ ( n) - x( n) = d ˆ ( n) - d( n) = e d ( n) So the equivalent quantization error is equal to the quantization error for the residual. In particular, s e = s ed

12 SNR of DPCM Coding (cont d) Now we can compute SNR DPCM : s 0 log x s 0 0 log x SNR DPCM = = 0 s e s e d Ê s 0 log x s d ˆ s 0 Á 0 log x = = 0 + SNR Ë s d s d d s e d where SNR d is the signal-to-noise ratio consequent to the quantization of d(n) IMPORTANT: SNR d = SNR PCM Because it only depends on the number of quantization levels! SNR of DPCM Coding (cont d) Now, we expect that (in general), s Hence, 0log x 0 s > 0 d s d < s x and therefore, for the same number of quantization levels (i.e., for the same bit-rate in output), we have SNR DPCM > SNR PCM

13 Example x(n) d(n) d(n) Closed-loop Encoding Problem: to be able to reconstruct x ˆ (n) = x(n -) + d ˆ (n) the decoder must know x(n-)!! Closed-loop differential quantization: rather than d(n) = x(n) - x(n - ) we compute (at the encoder) d(n) = x(n) - x ˆ (n - ) so that x ˆ (n) = x ˆ (n -) + d ˆ (n)

14 x(n) d (n) d ˆ (n) Quantizer ˆ x (n - ) Predictor ˆ x (n) ˆ d (n) (a) ˆ x (n) ˆ x (n - ) Predictor (b) Figure : Closed-loop differential encoder (a) and decoder (b) Predictive Encoding Higher-order linear prediction: instead of d(n) = x(n) - x ˆ (n - ) quantize and transmit M d(n) = x(n) - a m x ˆ (n - m) Â m = The coefficients are computed based on large segment of the signal The decoder must know the coefficients (need to transmit them) Note: DPCM is the simplest case of predictive encoding

15 Differential PCM (cont d) Differential PCM not only allows to reduce the quantization noise but also the signal entropy Example: x(n) d(n) P(x) P(d) Transform Coding Consider segments formed by M samples. E.g.: image frame x pixel block (M =) a segment of speech Apply a suitable invertible transformation (typically a matrix multiplication) x()...,x(m) Æ y(),...,y(m) (channels) Quantize and transmit y(),...,y(m)

16 x x() y() y ˆ () x() x() T y() y() Q Q Q y ˆ () y ˆ () ˆ y x(m) y(m) Q M y ˆ (M) (a) y ˆ () x ˆ () ˆ y ˆ y () ˆ y () T - ˆ x () ˆ x () ˆ x y ˆ (M) x ˆ (M) (b) Figure : Transform coder (a) and decoder (b). Some Definitions y ˆ (),..., y ˆ (M) = quantized versions of y()...,y(m) x ˆ (),..., x ˆ (M) = inverse transform of y ˆ (),..., y ˆ (M) Distortion (error): e (n) = ˆ X x (n) - x(n) e Y (n) = y ˆ (n) - y(n) Overall distortion: D = X M D Y = M Â Â M n = M n= s e x (n) s ey (n)

17 Channel Power Distribution Without transformation (y(n)=x(n), PCM case), all channels have the same error variance s ex because the source is stationary. Therefore, D X = D Y = s e x = s ey = D PCM Otherwise, the channels y(x) will (in general) have different variances s e y( n) (as a consequence of the transformation). This turns out to be the key factor in transform coding. Transform Coding (cont d) Important: quantizers in different channels may have different numbers of levels Bit budget B : if we we quantize y(),...,y(m) using B(),...,B(M) bits respectively, it must be M B(n) = B Â n= Optimal bit allocation: allocate more bits to those channels that have the highest variance It can be shown that the transformation increases the equivalent quantization SNR The transformation also reduces the signal entropy!

18 Discrete Cosine Transform (DCT) The DCT is a kind of transform coding with matrix T defined as Ï Ô t pm = Ì Ô Ó Ê M cos p M p ( m + ˆ Ë ) p =,,..., M - Ê M cos p M p ( m + ˆ ) Ë p = 0 It has the property that different channels represent the signal power along different (spatial or temporal) frequencies (similarly to the (discrete) Fourier transform What Do the DCT Coefficients Represent? The DCT coefs represent the spatial frequency content within a MxM image block The (0,0) coefficient contains is called DC coefficient, and is proportional to the average of the MxM image values in the block As we move to the right of the block, the coefficients represent the energy in higher horizontal frequencies As we move down the block, the coefficients represent the energy in higher vertical frequencies

19 Examples of x DCTs Horizontal centerband frequency x f x x f x x f x Vertical centerband frequency y y f y image x f y image DCT f x DCT x f x y y f y image x f y image DCT f x DCT y image f y DCT x f x y image f y DCT y image f y DCT Subband Coding It is a kind of transform coding (although operating on the whole signal) It is not implemented as a matrix multiplication but as a bank of filters followed by decimation (I.e., subsampling) Again, each channel represents the signal energy content along different frequencies

20 A Simple Scheme of Subband Coding ( channels) Low-pass filter Frequency response Signal band Decimate by Q Interpolate by x(n) f Transmission ˆ x (n) Decimate by Q Interpolate by High-pass filter f N-Band Subband Coding Band n- Band n Band n+ H(f) Hz

21 Coding - Conclusions Lossless coding Can only compress so much (depending on source entropy) Produces variable size bit rate Lossy coding Produces any desired constant bit-rate Uses DPCM or transform coding for improved SNR In general: x(n) Lossy coder Lossless coder ˆ x (n) References A. Gersho, R. Gray, Vector Quantization and Signal Compression, Kluwer Academic Publisher, 99

22 Some Details for the More Curious... Orthonormal Transformations We will consider linear and orthogonal transformations: È x() È y() X = Í... Y = Í... Î Í x(m) Î Í y(m) Then Y=TX where T is a M xm orthonormal matrix (i.e. T - =T T, det(t)=±) Fact : If T is orthonormal, then D X = D Y

23 Distortion Without transformation (T=I, PCM case): There are B/M bits per channel, therefore: D X = D PCM = s e x = Cs x B / M With transformation T: Fact : Under optimal bit allocation, N ( n= ) N D Y = C s y (n ) B N Coding Gain The coding gain (CG) measures the performance of transform coding with respect to direct quantization: CG = D PCM /D x = D PCM /D y = s x N ( s n= y(n) ) N Thus, transform coding (under optimal bit allocation) always performs at least as well as PCM It can be shown that the coding gain reaches its maximum when T is the Karhunen-Loeve transform, i.e. when the channels are decorrelated