Entropy Encoding Using Karhunen-Loève Transform

Transcription

1 Entropy Encoding Using Karhunen-Loève Transform Myung-Sin Song Southern Illinois University Edwardsville Sept 17, 2007

2 Joint work with Palle Jorgensen.

3 Introduction In most images their neighboring pixels are correlated and thus contain redundant information. Task : to find less correlated representation of the image, then perform redundancy reduction and irrelavancy reduction. Redundancy reduction removes duplication from the signal source (image) Irrelavancy reduction omits parts of the signal that will not be noticed by the Human Visual System (HVS).

4 Definition Spatial Redundancy : correlation between neighboring pixel values. Spectral Redundancy : correlation between different color planes or spectral bands. Aim : Reduce the number of bits needed to represent an image by removing redundancies as much as possible.

5 Outline Figure: Outline of the wavelet image compression process.

6 Image Decomposition using Forward Wavelet Transform A 1-level wavelet transform of an N M image can be represented as a 1 h 1 f v 1 d 1 where the subimages h 1, d 1, a 1 and v 1 each have the dimension of N/2 by M/2.

7 Image Decomposition using Forward Wavelet Transform-cont d a 1 = Vm 1 Vn 1 : ϕ A (x, y) = ϕ(x)ϕ(y) = i j h ih j ϕ(2x i)ϕ(2y j) h 1 = Vm 1 Wn 1 : ψ H (x, y) = ψ(x)ϕ(y) = i j g ih j ϕ(2x i)ϕ(2y j) v 1 = Wm 1 Vn 1 : ψ V (x, y) = ϕ(x)ψ(y) = i j h ig j ϕ(2x i)ϕ(2y j) d 1 = Wm 1 Wn 1 : ψ D (x, y) = ψ(x)ψ(y) = i j g ig j ϕ(2x i)ϕ(2y j) ϕ : the father function in sense of wavelet. ψ : is the mother function in sense of wavelet. V space : the average space and the from multiresolution analysis (MRA). W space : the difference space from MRA. h : low-pass filter coefficients g : high-pass filter coefficients.

8 Test Image Figure: Prof. Jorgensen in his office.

9 First-level Decomposition Figure: 1-level Haar Wavelet Decomposition of Prof. Jorgensen

10 Second-level Decomposition Figure: 2-level Haar Wavelet Decomposition of Prof. Jorgensen

11 Quantization The number of bits needed to store the wavelet transformed coefficients by reducing the precision of the values. This is a many-to-one mapping, meaning that it is a lossy process resulting in lossy compression.

12 Quantization-cont d Definition Let X be a set, and K be a discrete set. Let Q and D be mappings Q : X K and D : K X. Q and D are such that x D(Q(x)) x D(d), for all d K Applying Q to some x X is called quantization, and Q(x) is the quantized valued of x. Likewise, applying D to some k K is called dequantization and D(k) is the dequantized value of k.

13 Thresholding Another method of quantization is thresholding. Thresholding is a method of data reduction where it puts 0 for the pixel values below the thresholding value or something other appropriate value. Soft thresholding is defined as follows: 0 if x λ T soft (x) = x λ if x > λ x + λ if x < λ (1)

14 Thresholding Hard thresholding is as follows: { 0 if x λ T hard (x) = x if x > λ (2) where λ R + and x is a pixel value.

15 Entropy Encoding Entropy encoding further compresses the quantized values in lossless manner which gives better compression in overall. It uses a model to accurately determine the probabilities for each quantized value and produces an appropriate code based on these probabilities so that the resultant output code stream will be smaller than the input stream.

16 Various Entropy Encoding Schemes Kahunen-Loève Transform Shannon-Fano Entropy Huffman Coding Arithmetic Encoding Kolmogorov Entropy

17 Shannon-Fano Entropy For each data on an image, ie. pixel, a set of probabilities p i is computed, where n i=1 p i = 1. The entropy of this set gives the measure of how much choice is involved, in the selection of the pixel value on average.

18 Shannon-Fano Entropy-cont d Definition Shannon s entropy E(p 1, p 2,..., p n ) which satisfy the following: E is a continuous function of p i. E should be steadily increasing function of n. If the choice is made in k successive stages, then E = sum of the entropies of choices at each stage, with weights corresponding to the probabilities of the stages. E = k n i=1 p i log p i. k controls the units of the entropy, which is bits. logs are taken base 2.

19 Shannon-Fano Entropy-cont d Shannon-Fano entropy encoding is done according to the probabilities of data and the method is as follows: The data is listed with their probabilities in decreasing order of their probabilities. The list is divided into two parts that has roughly equal probability. Start the code for those data in the first part with a 0 bit and for those in the second part with a 1. Continue recursively until each subdivision contains just one data.

20 Example An example with letters in the text would better depict how the mechanism works. Suppose we have a text with letters a, e, f, q, r with the following probability distribution: Letter Probability a 0.3 e 0.2 f 0.2 q 0.2 r 0.1

21 Example-cont d Then applying the Shannon-Fano entropy encoding scheme on the above table gives us the following assignment. Letter Probability code a e f q r Note that instead of using 8-bits to represent a letter, 2 or 3-bits are being used to represent the letters in this case.

22 Description of the Algorithm for Karhunen-Loève transform entropy encoding 1. Perform the wavelet transform for the whole image. (i.e., wavelet decomposition.) 2. Do quantization to all coefficients in the image matrix, except the average detail. 3. Subtract the mean: Subtract the mean from each of the data dimensions. This produces a data set whose mean is zero. 4. Compute the covariance matrix cov(x, Y ) = n i=1 (X i X)(Y i Ȳ ). n 1

23 5. Compute the eigenvectors and eigenvalues of the covariance matrix. 6. Choose components and form a feature vector(matrix of vectors), (eig 1,..., eig n ). Eigenvectors are listed in decreasing order of the magnitude of their eigenvalues. Eigenvalues found in step 5 are different in values. The eigenvector with highest eigenvalue is the principle component of the data set. 7. Derive the new data set. Final Data = Row Feature Matrix Row Data Adjust.

24 Row Feature Matrix : the matrix that has the eigenvectors in its rows with the most significant eigenvector (i.e., with the greatest eigenvalue) at the top row of the matrix. Row Data Adjust : the matrix with mean-adjusted data transposed. That is, the matrix contains the data items in each column with each row having a separate dimension.

25 Karhunen-Loeve Expansion Consider an ensemble of a large number N of similar objects, of which Nw α, α = 1, 2,..., ν where the relative frequency w α satisfies the probability axioms: w α 0, ν w α = 1 α=1 Assume that each type specified by a value of the index α is represented by f α (ξ) in a real domain [a, b], which we normalize by b a f α (ξ) 2 dξ = 1

26 Karhunen-Loeve Expansion-cont d Let {ψ i (ξ)}, i = 1, 2,..., be a complete set of orthonomal base functions defined on [a, b] Then any function f α (ξ) can be expanded as f α (ξ) = i=1 x (α) i ψ i (ξ) (3) with b xi α = ψi (ξ)f α (ξ)dξ. (4) a Here, xi α is the component of f α in ψ i coordinate system. With the normalization of f α we have i=1 x α i 2 = 1 (5)

27 Karhunen-Loeve Expansion-cont d Then substituting (4) in (3) gives f α (ξ) = b a f α (ξ)[ in definition of ONB. i=1 ψ i (ξ)ψ i(ξ)]dξ = i=1 ψ i (ξ) f α ψ i (6)

28 Karhunen-Loeve Expansion-cont d Let H = L 2 (a, b). and ψ i : H l 2 (Z) U : l 2 (Z) l 2 (Z) where U is a unitary operator Note that the distance is invariant under a unitary transformation. Thus, using another coordinate system {φ j } in place of {ψ i }, would not change the distance.

29 Karhunen-Loeve Expansion-cont d Let {φ j }, j = 1, 2,..., be another set of ONB functions instead of {ψ i (ξ)}, i = 1, 2,...,. Let yj α be the component of f α in {φ j } where it can be expressed in terms of xi α by a linear relation where y α j = i=1 U is a unitary operator matrix φ j, ψ i x α i = i=1 U : l 2 (Z) l 2 (Z), U i,j = φ j, ψ i = b a U i,j x α i φ j (ξ)ψ i(ξ)dξ

30 Karhunen-Loeve Expansion-cont d Also, xi α relation where U 1 i,j can be written in terms of y α j x α i = j=1 ψ i, φ j y α j = = U i,j and U i,j = U j,i f α (ξ) = i=1 under the following j=1 U 1 i,j yj α x α i (ξ)ψ i (ξ) = y α i (ξ)φ i (ξ) So U(x i ) = (y i ) and i=1 x i α ψ i (ξ) = j=1 y j α φ j (ξ) b xi α =< ψ i, f α >= ψ i (ξ)f (α) (ξ)dξ a

31 Karhunen-Loeve Expansion-cont d The squared magnitude x (α) i 2 of the coefficient for ψ i in the expansion of f (α) can be considered as a good measure of the average in the ensemble Q i = n α=1 w (α) x (α) i 2 can be considered as the measure of importance of {ψ i }. Q i 0, Q i = 1 i

32 Karhunen-Loeve Expansion-cont d The entropy function in terms of the Q i s is defined as S({ψ i }) = i Q i log Q i. We are interested in minimizing the entropy, that is if {Θ j } is one such optimal coordinate system, we shall have S({Θ j }) = min {ψj }S({ψ i })

33 Karhunen-Loeve Expansion-cont d Let G(ξ, ξ ) = α w α f α (ξ)f α (ξ ). Then G is a Hermitian matrix and Q i = G(i, i) = α w α xi α xi α where normalization of Qi = 1 give us trace G = 1 where the trace means the diagonal sum. Then define a special function system {Θ k (ξ)} as the set of eigen-functions of G, i.e. b so GΘ k (ξ) = λ k Θ k (ξ). a G(ξ, ξ )Θ k (ξ)dξ = λ k Θ k (ξ). (7)

34 Karhunen-Loeve Expansion-cont d When the date are not functions but vectors v α s whose components are x (α) i in the ψ i coordinate system, we have G(i, i )ti k = λ kti k (8) i where ti k is the i th component of the vector Θ k in the coordinate system {ψ i }. So we get ψ : H (x i ) and also Θ : H (t i ). The two ONBs result in x α i = k c α k tk i for all i, c α k = i ti k xi α which is Karhunen-Loève expansion of f α (ξ) or vector v α.

35 Karhunen-Loeve Expansion-cont d Then {Θ k (ξ)} is the K-L coordiate system dependent on {w α } and {f α (ξ)}. Then we arrange the corresponding functions or vectors in the order of eigenvalues λ 1 λ 2... λ k 1 λ k....

36 Karhunen-Loeve Expansion-cont d Q i = G i,i = ψ i Gψ i = k A ikλ k where A ik = t k double stochastic matrix. Then λ 1 0 G = U U 1 0 λ k i ti k which is a

37 Karhunen-Loeve Expansion-cont d Theorem: If two probability distributions λ k, k = 1, 2,..., and Q i, i = 1, 2,... are related by Q i = k A ikλ k then S(G) = λ k log λ k Q i log Q i. k=1 i=1

38 References M.-S. Song, Wavelet Image Compression, Operator Theory, Operator Algebras, and Applications, Contemp. Math., vol. 414, American Mathematical Society, Providence P. E. T. Jorgensen and M.-S. Song, Entropy Encoding using Hilbert Space and Karhunen-Loeve Transforms Journal of Mathematical Physics to appear, P. E. T. Jorgensen and M.-S. Song, Comparison of Discrete and Continuous Wavelet Transforms Springer Encyclopedia of Complexity and Systems Science, to appear, M.-S. Song, Entropy Encoding in Wavelet Image Compression preprint, 2007 P. E. T. Jorgensen. Analysis and probability: wavelets, signals, fractals, volume 234 of Graduate Texts in Mathematics. Springer, New York, 2006.

39 References-cont d Smith, L. I., A Tutorial on Principal Components Analysis. tutorials/principal compo A. Skodras, C. Christopoulos, and T. Ebrahimi. Jpeg 2000 still image compression standard IEEE signal processing magazine. IEEE Signal processing Magazine, 18:36?58, Sept B. E. Usevitch. A tutorial on modern lossy wavelet image compression: Foundations of jpeg 2000 IEEE Signal processing Magazine, 18:22 35, Sept Watanabe, S., Karhunen-Loève Expansion and Factor Analysis Theoretical Remarks and Applications Transactions of the Fourth Prague Conference on Information Theory Statistical Decision Functions Random Process. (Adademia Press 1965). Ash, R. B. Information theory. Corrected reprint of the 1965 original (Dover Publications, Inc., New York, 1990).