An Algorithm for Quantization of Discrete Probability Distributions

Transcription

1 An Algorith for Quantization of Discrete Probability Distributions Yuriy A. Reznik Qualco Inc., San Diego, CA Eail: Abstract We study the proble of quantization of discrete probability distributions, arising in universal coding, as well as other applications. We show, that in any situations this proble can be reduced to the covering proble for the unit siplex, yielding precise characterization in the high-rate regie. We present siple and asyptotically optial algorith for solving this proble. Perforance of this algorith is studied and copared with several known solutions. I. INTRODUCTION The proble of coding of probability distributions surfaces any ties in the history of source coding. First universal codes, developed in late 1960s, such as Lynch-Davisson [9], [21], cobinatorial [28], and enuerative codes [7] used lossless encoding of frequencies of sybols in an input sequence. The Rice achine [24], developed in early 1970 s, transitted quantized estiate of variance of source s distribution. Two-step universal codes, developed by J. Rissanen in 1980s, explicitly estiate, quantize, and transit paraeters of distribution as a first step of the encoding process [25], [26]. Vector quantization techniques for two-step universal coding were proposed in [4], [30]. In practice, twostep coding was often ipleented by constructing a Huffan tree, then encoding and transitting this code tree, and then encoding and transitting the data. Such algoriths becoe very popular in 1980s and 1990s, and were used, for exaple, in ZIP archiver [17], and JPEG iage copression standard [16]. In recent years, the proble of coding of distributions has attracted a new wave of interest coing fro other fields. For exaple, in coputer vision, it is now custoary to use histogra-derived descriptions of iage features. Exaples of such descriptors include SIFT [20], SURF [1], and CHoG [2], differentiating ainly in a way they quantize histogras. Several other uses of coding of distributions are described in [11]. To the best of our knowledge, ost related prior studies were otivated by optial design of two-step universal codes [25], [26], [30], [4], [15]. In such context, quantization of distributions becoes a sall sub-proble in a coplex rate optiization process, and final solutions yield few insights about it. In this paper, we treat quantization of distributions as a stand-alone proble. In Section 2, we explain setting of the proble and survey relevant analytic results. In Section 3, we propose and study an algorith for solving this proble. In Section 4, we provide coparisons with other known techniques. Conclusions are drawn in Section 5. II. DESCRIPTION OF THE PROBLEM Let A = {r 1,..., r }, <, denote a discrete set of events, and let Ω denote the set of probability distributions over A: { Ω = [ω 1,..., ω ] R ωi 0, } i ω i = 1. (1) Let p Ω be an input distribution that we need to encode, and let Q Ω be a set of distributions that we will be able to reproduce. We will call eleents of Q reconstruction points or centers in Ω.

2 We will further assue that Q is finite Q <, and that its eleents are enuerated and encoded by using fixed-rate code. The rate of such code is R(Q) = log 2 Q bits. By d (p, q) we will denote a distance easure between distributions p, q Ω. In order to coplete traditional setting of the quantization proble, it reains to assue that input quantity p Ω is produced by soe rando process, e.g. a eoryless process with density θ over Ω. Then the proble of quantization can be forulated as iniization of the average distance to the nearest reconstruction point (cf. [14, Lea 3.1]) d(ω, θ, R) = inf E p Ω Q Ω p θ Q 2 R in d(p, q), (2) q Q However, we notice that in ost applications, best possible accuracy of the reconstructed distribution is needed instantaneously. For exaple, in the design of a two-step universal code, saple-derived distribution is quantized and iediately used for encoding of this saple [26]. Siilarly, in coputer vision / iage recognition applications, histogras of gradients fro an iage are extracted, quantized, and used right away to find nearest atch for this iage. In all such cases, instead of iniizing the expected distance, it akes ore sense to design a quantizer that iniizes the worst case- or axial distance to the nearest reconstruction point. In other words, we need to solve the following proble 1 d (Ω, R) = We next survey soe known results about it. A. Achievable Perforance Liits inf Q Ω Q 2 R ax p Ω in d(p, q). (3) q Q We note that the proble (3) is purely geoetric in nature. It is equivalent to the proble of covering of Ω with at ost 2 R balls of equal radius. Related and iediately applicable results can be found in Graf and Luschgy [14, Chapter 10]. First, observe that Ω is a copact set in R (unit 1-siplex), and that its volue in R can be coputed as follows [29]: λ (Ω ) = ak k! k k k= a= 2 Here and below we assue that 3. Next, we bring result for asyptotic covering radius [14, Theore 10.7] = ( 1)!. (4) li 2 R d (Ω, R) = C λ (Ω ), (5) R where C > 0 is a constant known as covering coefficient for the unit cube C = inf 2 R d ([0, 1], R). (6) R 0 1 The dual proble: R(ε) = inf log Q Ω : ax p Ω in 2 Q, q Q d(p,q) ε ay also be posed. The quantity R(ε) can be understood as Kologorov s ε-entropy for etric space (Ω, d) [18].

3 The exact value of C depends on a distance easure d(p, q). For exaple, for L nor it is known that d (p, q) = p q = ax i p i q i, C, = 1 2. Hereafter, when we work with specific L r - nors: ( ) 1/r d r (p, q) = p q r = p i q i r (7) we will attach subscripts r to covering radius d (.) and other expressions to indicate the type of nor being used. By putting all these facts together, we obtain: Proposition 1. With R : where C,r are soe known constants. d r(ω, R) C,r i ()! 2 We now note, that with large the leading ter in (8) turns into = e ( ) 1 ()! + O 2 R, (8) which is a decaying function of. This highlights an interesting property of this quantization proble, distinguishing it fro well known cases, such as quantization of the unit ( 1)-diensional cube. III. AN ALGORITHM FOR CODING OF DISTRIBUTIONS The following algorith 2 can be viewed as a custo designed lattice quantizer. It is interesting in a sense that its lattice coincides with the concept of types in universal coding [8]. A. Choice of Lattice Given soe integer n 1, we define a set: { Q n = [q 1,..., q ] Q qi = k i, } n i k i = n, (10) where n, k 1,..., k Z +. Paraeter n serves as a coon denoinator to all fractions, and can be used to control the density and nuber of points in Q n. By analogy with the concept of types in universal coding [8] we will refer to distributions q Q n as types. For sae reason we will (soewhat inforally) call Q n a type lattice. Several exaples of sets Q n are shown in Figure 1. 2 An earlier publication related to this technique is [3]. (9)

4 Figure 1. Exaples of type lattices and their Voronoi cells (for L-nors) in 3 diensions. B. Quantization The task of finding the nearest type in Q n can be solved by using the following siple algorith 3 : Algorith 1. Given p, n, find nearest q = [ ] k 1 n,..., k n : 1) Copute values (i = 1,..., ) k i = np i + 1 2, n = i k i. 2) If n = n the nearest type is given by: k i = k i. Otherwise, copute errors and sort the such that δ i = k i np i, 1 2 δ j 1 δ j2... δ j 1 2, 3) Let = n n. If > 0 then decreent d values k i with largest errors [ k k ji = ji j = i,..., 1, k j i 1 i =,...,, otherwise, if < 0 increent values k i with sallest errors [ k k ji = ji + 1 i = 1,...,, k j i i = + 1,...,. The correctness of this algorith in ters of L, L 1, and L 2 -nors is self-evident. C. Enueration and Encoding As entioned earlier, the nuber of types in lattice Q n depends on the paraeter n. It is essentially the nuber of partitions of n into ters k k = n: ( ) n + 1 Q n =. (11) 1 In order to encode a type with paraeters k 1,..., k, we need to obtain its unique index ξ(k 1,..., k ). We suggest to copute it as follows: n 2 k j 1 ( n i j 1 l=1 ξ(k 1,..., k n ) = k ) l + j 1 + k n 1. (12) j 1 j=1 i=0 3 This algorith is siilar in concept to Conway and Sloane s quantizer for lattice A n [5, Chapter 20], but it is naturally scaled and reduced to find solutions within (-1) siplex.

5 This forula follows by induction (starting with = 2, 3, etc.), and it ipleents lexicographic enueration of types. For exaple: ξ(0, 0,..., 0, n) = 0, ξ(0, 0,..., 1, n 1) = 1,... ξ(n, 0,..., 0, 0) = ( ) n+ 1. Siilar cobinatorial enueration techniques were discussed in [7], [27], [28]. Once index is coputed, it is transitted by using direct binary representation at rate: R(n) = ( log n+ ) 2. (13) D. Perforance Analysis The set of types Q n is related to so-called lattice A n in lattice theory [5, Chapter 4]. It can be understood as a bounded subset of A n with n = 1 diensions, which is subsequently scaled, and placed in the unit siplex. Using this analogy, we can show that vertices of Voronoi cells (so called holes) in type lattice Q n are located at: q i = q + v i, q Q n, i = 1,..., 1, (14) where v i are so-called glue vectors [5, Chapter 21]: v i = 1 n [ i,..., i }{{} i ties ],..., i. (15) }{{}, i We next copute axiu distances (covering radii). i ties Proposition 2. Let a = /2. The following holds: ( ax in d (p, q) = 1 p Ω q Q n 1 1 n ), (16) ax in d 2 (p, q) = 1 a( a) p Ω q Q n n (17) ax in d 1 (p, q) = 1 2a( a). (18) p Ω q Q n n Proof: We use vectors (15). The largest coponent values appear when i = 1 or i = 1. E.g. for i = 1: [ v 1 = 1, 1,..., ] 1 n. This produces L - radius. The largest absolute su is achieved when all coponents are approxiately the sae in agnitude. This happens when i = a: [ ] v a = 1 a,..., a a, n }{{},..., a. }{{} a ties a ties This produces L 1 - radius. L 2 nor is the sae for all vectors v i, i > 0. It reains to evaluate distance / rate characteristics of type-lattice quantizer: d r[q n ](Ω, R) = in n: Q n 2 R ax p Ω in q Q n d r (p, q).

6 We report the following. Theore 1. Let a = /2. Then, with R : d [Q n ](Ω, R) 2 R d 2[Q n ](Ω, R) 2 R d 1[Q n ](Ω, R) 2 R 1 1 ( 1)!, (19) a( a) ( 1)!, (20) 2a( a) ( 1)!. (21) Proof: We first obtain asyptotic (with n ) expansion for the rate of our code (13): This iplies that R = ( 1) log 2 n log 2 ( 1)! + O ( 1 n). n 2 R ( 1)!. (22) Stateents of theore are obtained by cobination of this relation with expressions (16-18). 1) Optiality: We now copare the result of Theore 1 with theoretical asyptotic estiates for covering radius for Ω (8). As evident, the axiu distance in our schee decays with the rate R as: 1 2 d [Q n ](Ω, R) 2 which atches the decay rate of theoretical estiates. The only difference is in a leading constant factor. For exaple, under L nor, such factor in expression (8) is ( ) log. 1 = 2 + O Our algorith, on the other hand, uses a factor 1 1, which starts with 1 when = 2, and tends to 1 with. 2 2) Perforance in ters of KL-distance: All previous results are obtained using L-nors. Such distance easures are coon in coputer vision and iage recognition applications [1], [20], [22]. In source coding, ain interest presents Kullback-Leibler (KL) distance: d KL (p, q) = D(p q) = i R, p i log 2 p i q i. (23) This is not a true distance easure, so precise analysis is coplicated. Yet, by observing 4 that for sall distances between p and q: d KL (p, q) 1 (p i q i ) 2 (24) 2 ln 2 q i i 4 For exaple, by considering Tailor series of h(p) = i p i log p i at point q: h(p) = h(q) + (p q) T h(q) (p q)t 2 h(q) (p-q) +..., we note that D(p q) is just a tail of this series: D(p q) = h(q) h(p) + (p q) T h(q).

7 Figure 2. Two 10-point lattices: Q 3 (left), and Q 2 (right). The second has noticeably saller cells. we can at least say, that for a cell in the iddle of the siplex { q i = } 1, KL-distances to the nearest holes q will satisfy: [ d KL (q, q) d2 (q, q) ] ( ) 2 2 ln 2 = 1 a( a) 1 = O. (25) 2 ln 2 n 2 n 2 As evident fro (24), KL-radii of our cells will increase as we ove point q closer to the boundary of the siplex. However, in the asyptotic sense (and excluding boundary points), their agnitude will still decrease at the rate of O (n 2 ). This follows, for exaple, fro inequality: (p i q i ) 2 i q i i (p i q i ) (ax 2 1 i q i ). By translating n to bitrate (cf. (22)), we obtain ) d KL (q, q) = O (2 2R. (26) Precise lower bounds, supporting this siple estiate, can be obtained by using Pinsker inequality [23], and its recent refineents [10]. E. Additional iproveents 1) Introducing bias: As easily observed, type lattice Q n places reconstruction points with k i = 0 precisely on edges of the probability siplex Ω. This is not best placeent fro quantization standpoint, particularly when n is sall. This placeent can be iproved by using biased types: q i = k i + β n + β, i = 1,...,, where β 0 is a constant that defines shift towards the iddle of the siplex. In traditional source coding applications, it is custoary to use β = 1/2 [19]. In our case, setting β = 1/ appears to work best for L-nors, as it introduces sae distance to edges of the siplex as covering radius of the lattice. 2) Using dual type lattice Q n: Another idea for iproving perforance of our quantization algorith is to define and use dual type lattice Q n. Such a lattice consists of all points: q = q + v i, q Q n, q Ω i = 0,..., 1, where v i are the glue vectors (15). The ain advantage of using dual lattice is thinner covering in high diensions (cf. [5, Chapter 2]). But even at sall diensions, it ay soeties be useful. An exaple of such a situation for = 3 is shown in Figure. 2.

8 Figure 3. L 1-radius vs rate characteristics d 1[H](R), d 1[GM](R), d 1[Q n](r) achievable by Huffan-, Gilbert-Moore-, and type-based quantization schees. IV. COMPARISON WITH TREE-BASED QUANTIZATION TECHNIQUES Given a probability distribution p Ω, one popular in practice way of copressing it is to design a prefix code (for exaple, a Huffan code) for this distribution p first, and then encode the binary tree of such a code. Below we suarize soe known results about perforance of such schees. By denoting by l 1,..., l lengths of prefix codes, recalling that they satisfy Kraft inequality [6], and noting that 2 l i can be used to ap lengths back to probabilities, we arrive at the following set: Q tree = { [q 1,..., q ] Q q i = 2 l i, i 2 l i 1 }. There are several specific algoriths that one can eploy for construction of codes, producing different subsets of Q tree. Below we only consider the use of classic Huffan and Gilbert-Moore [13] codes. Soe additional tree-based quantization schees can be found in [11]. Proposition 3. There exists a set Q GM Q tree, such that where and C n = 1 n+1( 2n n ) is the Catalan nuber. d KL[Q GM ](R GM ) 2, (27) d 1[Q GM ](R GM ) 2 ln 2, (28) d [Q GM ](R GM ) 1, (29) R GM = log 2 Q GM = log 2 C = log 2 + O(1), (30) Proof: We use Gilbert-Moore code [13]. Upper bound for KL-distance is well known [13]. L 1 bound follows by Pinsker inequality [23]: d KL (p, q) 1 d 2 ln 2 1(p, q) 2. L bound is obvious: p i, q i (0, 1). Gilbert-Moore code uses fixed assignent (e.g. fro left to right) of letters to the codewords. Any

9 binary rooted tree with leaves can serve as a code. The nuber of such trees is given by the Catalan nuber C. Proposition 4. There exists a set Q H Q tree, such that where d KL[Q H ](R H ) 1, (31) d 1[Q H )](R H ) 2 ln 2, (32) d [Q H ](R H ) 1 2, (33) R H = log 2 Q H = log 2 + O (). (34) Proof: We use Huffan code. Its KL-distance bound is well known [6]. L 1 bound follows by Pinsker inequality [23]. L bound follows fro sibling property of Huffan trees [12]. It reains to estiate the nuber of Huffan trees T with leaves. Consider a skewed tree, with leaves at depths 1, 2,..., 1, 1. The last two leaves can be labeled by ( ) 2 cobinations of letters, whereas the other leaves - by ( 2)! possible cobinations. Hence T ( ) 2 ( 2)! = 1!. Upper bound is 2 obtained by arbitrary labeling all binary trees with leaves: T <! C, where C is the Catalan nuber. Cobining both we obtain: 1 < log ln 2 2 T log 2 < ( 2 1 ln 2). We present coparison of axial L 1 distances achievable by tree-based and type-based quantization schees in Figure 3. We consider cases of = 5 and = 10 diensions. It can be observed that the proposed type-based schee is ore efficient and ore versatile, allowing a wide range of possible rate/distance tradeoffs. V. CONCLUSIONS The proble of quantization of discrete probability distributions is studied. It is shown, that in any cases, this proble can be reduced to the covering radius proble for the unit siplex. Precise characterization of this proble in high-rate regie is reported. A siple algorith for solving this proble is also presented, analyzed, and copared to other known solutions. ACKNOWLEDGEMENTS The author would like to thank Prof. K. Zeger (UCSD) for reviewing and providing very useful coents on the initial draft of this paper. The author also wishes to thank Prof. B. Girod and his students V. Chandrasekhar, G. Takacs, D. Chen, S. Tsai (Stanford University), and Dr. R. Grzeszczuk (Nokia Research Center, Palo Alto) for introduction to the field of coputer vision, and collaboration that propted study of this quantization proble [3]. REFERENCES [1] H. Bay, A. Ess, T. Tuytelaars, L. Van Gool, SURF: Speeded Up Robust Features, Coputer Vision and Iage Understanding (CVIU), vol. 110, no. 3, pp , [2] V. Chandrasekhar, G. Takacs, D. Chen, S. Tsai, R. Grzeszczuk, B. Girod, CHoG: Copressed histogra of gradients A low bit-rate feature descriptor, in Proc. Coputer Vision and Pattern Recognition (CVPR 09), pp , [3] V. Chandrasekhar, Y. Reznik, G. Takacs, D. Chen, S. Tsai, R. Grzeszczuk, and B. Girod, Quantization Schees for Low Bitrate Copressed Histogra of Gradient Descriptors, in Proc. Coputer Vision and Pattern Recognition (CVPR 10), [4] P. A. Chou, M. Effros, and R. M. Gray, A vector quantization approach to universal noiseless coding and quantization, IEEE Trans. Inforation Theory, vol. 42, no. 4, pp , [5] J. H. Conway and N. J. A. Sloane, Sphere Packings, Lattices and Groups. New York: Springer-Verlag, [6] T. M. Cover and J. M. Thoas, Eleents of Inforation Theory. New York: John Wiley & Sons, 2006.

10 [7] T. Cover, Enuerative source coding, IEEE Trans. Infor. Theory, vol. 19, pp , Jan [8] I. Csiszár, The ethod of types, IEEE Trans. Inf. Theory, vol.44, no. 66, pp , [9] L. D. Davisson, Coents on Sequence tie coding for data copression, Proc. IEEE, vol. 54, p Dec [10] A. A. Fedotov, P. Harreoës, and F. Topsøe, Refineents of Pinsker s inequality, IEEE Trans. Inf. Theory, vol. 49, no. 6, pp , [11] T. Gagie, Copressing Probability Distributions, Inforation Processing Letters, vol. 97, no. 4, pp , [12] R. Gallager, Variations on a thee by Huffan, IEEE Trans. Infor. Theory, vol.24, no.6, pp , Nov [13] E. N. Gilbert and E. F. Moore, Variable-Length Binary Encodings, The Bell Syste Tech. Journal, vol. 7, pp , [14] S. Graf, and H. Luschgy, Foundations of Quantization for Probability Distributions. Berlin: Springer-Verlag, [15] T. S. Han, and K. Kobayashi, Matheatics of Inforation and Coding. Boston: Aerican Matheatical Society, [16] ITU-T and ISO/IEC JTC1, Digital Copression and Coding of Continuous-Tone Still Iages, ISO/IEC ITU-T Rec. T.81, Sept [17] P. W. Katz, PKZIP. Data copression utility, version 1.1, [18] A. N. Kologorov and V. M. Tikhoirov, ε-entropy and ε-capacity of sets in etric spaces, Uspekhi Math. Nauk, vol. 14, no. 2, pp. 3 86, (in Russian) [19] R. E. Krichevsky and V. K. Trofiov, The Perforance of Universal Encoding, IEEE Trans. Inforation Theory, vol. 27, pp , [20] D. Lowe, Distinctive Iage Features fro Scale-Invariant Keypoints, International Journal of Coputer Vision, vol. 60, no. 2, pp , [21] T. J. Lynch, Sequence tie coding for data copression, Proc. IEEE, vol. 54, pp , [22] K. Mikolajczyk and C. Schid, Perforance Evaluation of Local Descriptors, IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10, pp , [23] M. S. Pinsker, Inforation and Inforation Stability of Rando Variables and Processes, Probley Peredachi Inforacii, vol. 7, AN SSSR, Moscow (in Russian). [24] R. F. Rice and J. R. Plaunt, Adaptive variable length coding for efficient copression of spacecraft television data, IEEE Trans. Co. Tech., vol. 19, no.1, pp , [25] J. Rissanen, Universal coding, inforation, prediction and estiation, IEEE Trans. Infor. Theory, vol. 30, pp , [26] J. Rissanen, Fisher Inforation and Stochastic Coprexity, IEEE Trans. Infor. Theory, vol. 42, pp , [27] J. P. M. Schalkwijk, An algorith for source coding, IEEE Trans. Infor. Theory, vol. 18, pp , May [28] Yu. M. Shtarkov and V. F. Babkin, Cobinatorial encoding for discrete stationary sources, in Proc. 2nd Int. Syp. on Inforation Theory, Akadéiai Kiadó, 1971, pp [29] D. Soerville, An Introduction to the Geoetry of n Dientions. New York: Dover, [30] K. Zeger, A. Bist, and T. Linder, Universal Source Coding with Codebook Transission, IEEE Trans. Counications, vol. 42, no. 2, pp , 1994.