Compression and Predictive Distributions for Large Alphabet i.i.d and Markov models

Transcription

1 2014 IEEE International Syposiu on Inforation Theory Copression and Predictive Distributions for Large Alphabet i.i.d and Markov odels Xiao Yang Departent of Statistics Yale University New Haven, CT, Eail: Andrew R. Barron Departent of Statistics Yale University New Haven, CT, Eail: Abstract This paper considers coding and predicting sequences of rando variables generated fro a large alphabet. We start fro the i.i.d odel and propose a siple coding distribution forulated by a product of tilted Poisson distributions which achieves close to optial perforance. Then we extend to Markov odels, and in particular, tree sources. A context tree based algorith is designed according to the frequency of various contexts in the data. It is a greedy algorith which seeks for the greatest savings in codelength when constructing the tree. Copression and prediction of individual counts associated with the contexts again uses a product of tilted Poisson distributions. Ipleenting this ethod on a Chinese novel, about 20.56% savings in codelength is achieved copared to the i.i.d odel. I. INTRODUCTION Large alphabet data copression does not have the luxury of diinishing per sybol redundancy. Luckily, sybols in a large alphabet are usually not equally probable in practice. For exaple, in Chinese characters, a subset of 964 characters covers 90% inputs in Chinese[1] though the vocabulary size is no saller than 106,230 in total [2]. Coding and prediction of strings of rando variables generated fro an i.i.d odel have been considered for the large alphabet setting with the restriction that the ordered probability or count list rapidly decreasing [3], or satisfies an envelope class property [4]. Although this i.i.d odel is not the best for copression or prediction when there is dependence between successive characters, it serves as an analytical tool that ore coplicated odels can be based on, and helps understand the behavior of coding and predictive distributions. In this paper, we propose a siple coding ethod particularly applicable for large alphabet copression and prediction probles, which also perfors alost optially. Suppose a string of rando variables X =(X 1,...,X N ) is generated independently fro a discrete alphabet A of size. We allow the string length N to be variable. A special case is when N is given as a fixed nuber, or it can be rando. In either case, X is a eber of the set X of all finite length strings 1[ X = X n = n=0 1[ {x n =(x 1,...,x n ):x i 2A,i=1,...,n}. n=0 Our goal is to code/predict the string X. Now suppose given N, each rando variable X i is generated independently according to a probability ass function in a paraetric faily P = {P (x) : 2 R } on A. Thus ny P (X 1,...,X N N = n) = P (X i ) for n = 1, 2,... We are interested in the class of all distributions with P (j) = j paraeterized by the siplex ={ =( 1,..., ): j 0, P j=1 j =1,j=1,...,}. Let N = (N 1,...,N ) denote the vector of counts for sybol 1,...,. The observed saple size N is the su of the counts N = P j=1 N j. Both P (X) and P (X N = n) have factorizations based on the distribution of the counts and i=1 P (X N =n) =P (X N) P (N N =n), P (X) = P (X N) P (N). The first factor of the two equations is the unifor distribution on the set of strings with given counts, which does not depend on. The vector of counts N fors a sufficient statistic for. Modeling the distribution of the counts is essential for foring codes and predictions. In the particular case of all i.i.d. distributions paraeterized by the siplex, the distribution P (N N = n) is the ultinoial(n, ) distribution. In the above, there is a need for a distribution of the total count N. Of particular interest is the case that the total count is taken to be Poisson, because then the resulting distribution of individual counts akes the independent. Poisson sapling is a standard technique to siplify analysis [5][6]. Here we give particular attention to the target faily P = {P (N): j 0, j=1,...,}, in which P (N) is the product of Poisson( j ) distribution for N j,j=1,...,. It akes the total count N Poisson( su ) with su = P j=1 j and yields the ultinoial(n, ) distribution by conditioning on N = n, where j = j / su. And the induced distribution on X is P (X) =P (X N)P (N) /14/$ IEEE 2504

2 2014 IEEE International Syposiu on Inforation Theory Ideally the true probability distribution P (X) could be used to code the sequence, if were known. The regret induced by using instead of P is R(, P,X) = 1 (X) 1 P (X), where is arith base 2 (We don t worry about integer constraint). Here we can construct by choosing a probability distribution for the counts and then use the unifor distribution for the distribution of strings given the counts, written as P unif. That is (X) =P unif (X N)(N). (1) Then the regret becoes R(, P,X) = P (N) (N) = R(, P,N). However, in reality is usually unknown. Given the faily P, we could instead use the best candidate with hindsight Pˆ(X) = ax 2 (P (X)) (corresponding to in 2 (1/P (X))), and copare it to (X). The axiization is equivalent to axiizing for the count probability, as the unifor distribution dose not depend on, i.e. ax 2 (P (X)) = P unif (X N) ax 2 P (N) = P unif (X N) Pˆ(N). Then the proble becoes: given the faily P, how to choose to iniize the axiized regret in ax X R(, Pˆ,X)=in ax N Pˆ(N) (N). For the regret, the axiu can be restricted to a set of counts instead of the whole space. A traditional choice being S,n = {(N 1,...,N ): P j=1 N j =n, N j 0,j=1,...,} associated with a given saple size n, in which case the iniax regret is in ax Pˆ(N) N2S,n (N). As is failiar in universal coding [7][8], the noralized axiu likelihood (NML) distribution nl (N) = Pˆ(N) C(S,n ) is P the unique pointwise iniax strategy when C(S,n )= N2S,n Pˆ(N) is finite, and C(S,n ) is the iniax value. When is large, the NML distribution can be unwieldy to copute for copression or prediction. Instead we will introduce a slightly suboptial coding distribution that akes the counts independent and show that it is nearly optial for every S,n 0 with n 0 not too different fro a target n. Indeed, we advocate that our siple coding distribution is preferable to use coputationally when is large even if the saple size n were known in advance. To produce our desired coding distribution we ake use of two basic principles. One is that the ultinoial faily of distributions on counts atches the conditional distribution of N 1,...,N given the su N when unconditionally the counts are independent Poisson. Another is the inforation theory principle [9][10][11] that the conditional distribution given a su (or average) of a large nuber of independent rando variables is approxiately a product of distributions, each of which is the one closest in relative entropy to the unconditional distribution subject to an expectation constraint. This iniu relative entropy distribution is an exponential tilting of the unconditional distribution. N In the Poisson faily with distribution j j e j /N j!, exponential tilting (ultiplying by the factor e anj ) preserves the Poisson faily (with the paraeter scaled to je a ). Those distributions continue to correspond to the ultinoial distribution (with paraeters j = j / su ) when conditioning on the su of counts N. A particular choice of a =ln( su /N ) provides the product of Poisson distributions closest to the ultinoial in regret. Here for universal coding, we find the tilting of individual axiized likelihood that akes the product of such closest to the Shtarkov s NML distribution. This greatly siplifies the task of approxiate optial universal copression and the analysis of its regret. Indeed, applying the axiu likelihood step to a Poisson count k produces a axiized likelihood value of M(k) = k k e k /k!. We call this axiized likelihood the Stirling ratio, as it is the quantity that Stirling s approxiation shows near (2 k) 1/2 for k not too sall. We find that this M(k) plays a distinguished role in universal large alphabet copression, even for sequences with sall counts k. This easure M has a product extension to counts N 1,...,N, M(N) =M(N 1 ) M(N ). Although M has an infinite su by itself, it is noralizable when tilted for every positive a. The tilted Stirling ratio distribution is P a (N j )= N Nj j e Nj e anj, (2) N j! C a with the noralizer C a = P 1 k=0 kk e (1+a)k /k!. The coding distribution we propose and analyze is siply the product of those tilted one-diensional axiized Poisson likelihood distributions for a value of a we will specify later a (N) =P a (N) =P a (N 1 ) P a (N ). If it is known that the total count is n, then the regret is a siple function of n and the noralizer C a. The choice of the P tilting paraeter a given by the oent condition E a j=1 N j = n iniizes the regret over all positive a. Moreover, value of a depends only on the ratio between the size of the alphabet and the total count /n. Fig. 1 displays a as a function of /n solved nuerically. Given an alphabet with sybols and a string generated fro it of length n, one can look at the plot and find the a desired according to the /n given, and then use the a to code the data. 2505

3 2014 IEEE International Syposiu on Inforation Theory a * /n Fig. 1. Relationship between a and n. As copared to i.i.d class, Markov sources are uch richer and ore realistic. Suppose given N, each rando variable X i is generated according to a probability ass function depending on its context (string of sybols preceding it). Following Willies notations in [12], a tree source can be deterined by a context set S. Eleents of S are strings of sybols fro A or concatenation of others and prefixes of the contexts. others represents copleents of the contexts in S with a coon prefix. Prefix of a context is the part except the last sybol. For exaple, the prefix for s = ab is a, and for s = a equals nothing. The collection of distributions is P S = {P s (x) : s 2 S R,s 2S}, where S is the paraeter set defined later. For siplicity, we require the order of the odel no larger than T 2{0, 1, 2,...}, so S2C T, where C T is the class of tree sources with order T or less. For each context s 2S with a given S, let sx denote the probability of sybol x 2Ashowing up after s, for all x 2A. Then s =( s1,..., s ) lies in the set S = { s =( s1,..., s ):x 2A, sx 0, X x2a sx =1}. Again, we could take advantage of factorizations based on the distribution of the counts N S =(N s ) s2s, where N s = (N s1,...,n s ) is the count for all sybols given context s 2S and P (X N =n) =P (X N S ) P (N S N =n), P (X) =P (X N S ) P (N S ). Picking the distribution for the total count to be Poisson again leads to the target faily P S = {P (N S ): sj 0, j=1,...,, s 2S}, in which P (N S ) is the product of Poisson( sj ) distribution for N sj,j=1,..., and s 2S. And the induced distribution on X is P (X) =P (X N S )P (N S ). Fig. 2. An exaple context tree. There are two sources of cost involved in using a tree odel. One is the coding cost C(X) for variables given the context. Another is the description cost D(S) for describing the tree. Overall, we want to find which uses shorter codelength for sequences generated fro an unknown tree source S2C T. That is, to iniize ax ax S2C T X2S C(S,X) = ax S2C T ax X2S (C(X)+D(S)). for a given set of sequences P P S. One choice of S could be S,n,S = {N S : s2s j=1 N sj = n, N sj 0,j = 1,...,, s 2S}associated with a given saple size n. We use the sae coding distribution as given in equation (2) for count variables conditional on each given context s. The coding distribution for the counts given s is siply the product as (N s ) =P a s (N s ) =P as (N s1 ) P as (N s ), (3) with a properly chosen a s for each context s 2S. The tilting paraeter a s can be chosen according to the ratio of alphabet size and the context count N s = P j=1 N sj for all s 2 S [3]. Using the tilted distribution P as as a coding distribution, the regret is siply a su of the individual regrets. It is clear that when the context set S is provided, the coding distribution and the regret it induces is easy to obtain. Yet the questions is how to construct the context set to begin with. We adopt a ethod siilar to Rissanen s approach in [13]. Different fro the i.i.d odel in which regret can be solely relied on to evaluate the perforance, for Markov odels different targets associated with different odels need to be taken account of. Hence, we focus on the total codelength C(S,X) and use it as a standard to evaluate the perforance of different odels and coding distributions. We use a greedy algorith to build the context tree with details discussed in Section III-C. An illustrative exaple tree with a siple alphabet A = {a, b, c, d} is given in Fig.2. II. I.I.D CLASS Let S be any set of counts, then the axiized regret of using as a coding strategy given a class P of distributions when the vector of counts is restricted to S is R(, P,S) = ax ax P 2P P (N). N2S (N) 2506

4 2014 IEEE International Syposiu on Inforation Theory Theore 1: Let P a be the distribution specified in equation (2) (Poisson axiized likelihood, tilted and noralized). The regret of using a product of tilted distributions a = j=1 P a for a given vector of counts N =(N 1,...,N ) is R ( a, P,N)=aN e + C a. Let S,n be the set of count vectors with total count n be defined as before, then R ( a, P,S,n )=an e + C a. (4) Let a be the choice of a satisfying the following oent condition E Pa X j=1 N j = E Pa N 1 = n. (5) Then a is the iniizer of the regret in expression (4). Write R,n =in a R( a, P,S,n). When = o(n), the R,n is near ne 2 in the following sense. d 1 2 e apple R ne,n 2 r apple (1 + ), (6) n where d 1 = O ( n )1/3. When n = o(), the R,n is near n ne in the following sense. 1+(1 d 2 ) n apple R,n n ne apple 1+ n + d 3 (7) n 2 e 2 ( ne). where d 2 = O( n ), and d 3 = 1 2 p When n = b, the R,n = c, where the constant c = a b e + C a, and a is such that E Pa N 1 = b. Proof: The expression of the regret is fro the definition. The fact that a is the iniizer can be seen by taking partial derivative with respect to a of expression (4). Details of proof can be found in [3]. Reark 1: The regret depends only on the nuber of paraeters, the total counts n and the tilting paraeter a. The optial tilting paraeter is given by a siple oent condition in equation (5). Reark 2: The regret R,n is close to the iniax level in all three cases listed in Theore 1. The ain ters in the = o(n) and n = o() cases are the sae as the iniax regret given in [14] except the ultiplier for (ne/) here is /2 instead of ( 1)/2 for the sall scenario. For the n = b case, the R,n is close to the iniax regret in [14] nuerically. Corollary 1: Let P be a faily of ultinoial distributions with total count n. Then the axiized regret R( a, P,S,n) has an upper bound within n above the upper bound in Theore 1. Proof: This can be easily seen by the following equation Pˆ(X 1,..., X N N = n) P unif (X 1,..., X n N 1,..., N ) a (N 1,..., N ) j=1 = M(N j) (N = n) Pˆsu a (N 1,...,N ) ' n + j=1 M(N j) a (N 1,...,N ). (8) Here ˆsu = n, hence the ter n is Stirling s approxiation of (N = n). 1/Pˆsu Reark 3: The n arises because here includes description of the total N while the ore restrictive target regards it as given. A. Coding cost III. TREE SOURCE For each context s 2S, we use a product of tilted Stirling ratio distributions with paraeter a s as in equation (3) to code the vector of counts N s. The coding distribution is the product of all the as (N s ), i.e. S a (N S )= Y s2s as (N s ). Corollary 2: Using independent tilted Stirling ratio distribution S a to code the counts in S,n,S, the regret equals R(P S, S a,s,n,s )= X (a s N s e + C as ). s2s This can be easily seen by applying the definition. B. Description cost To describe a given context set S, we use the following rule D(S) =1+N branches (1 + A ), where N branches is the nuber of labeled branches in the tree. Here labeled eans a specified sybol in the alphabet. For instance, N branches =5in the exaple tree. The first bit is used to describe if the odel is nondegenerate (i.i.d or Markov). For each branch other than others, we could use bits to convey which sybol it is, and then another 1 bit to say if it is nondegenerate. Our exaple tree uses 1 + 5(1 + 4) = 16 bits. C. Using codelength to construct the tree Here we use the exaple in Fig.2 to describe how we construct the tree. Starting fro the root which conditions on nothing (the i.i.d odel), we choose the first sybol (c) that produces the ost savings (if any) in codelength to condition on. Next, we consider two possible ways to expand the tree: one is another sybol (a) that akes the ost savings; the other is to further condition on one ore sybol (b) preceding the first found one (c). After calculating and coparing savings produced by these two candidates, we then decide which one to pick again by choosing the one with the greatest savings. Continue these procedures until conditioning provides no ore 2507

5 2014 IEEE International Syposiu on Inforation Theory!!!!!!!!!!!!!!!! others!!!!!!! Fig. 3. Context tree for Fortress Beseiged. savings or the axiu nuber of sybols to condition on (T ) is reached, we have the context tree built. At each layer, the label others represents all other sybols that are not picked up in that layer, for exaple, others includes b and d in the first layer in the exaple. When expanding one branch fro the tree, we are conditioning on one ore sybol. This leads to a different target odel. We calculate the coding cost by adding up the iniized codelength for the target odel and the iniax regret by using Szpankowski s approxiation [14], because the iniized regret of using the tilted Stirling ratio distribution is very close to the iniax level. Then the overall codelength is used for coparison in tree construction. IV. A REAL EXAMPLE We apply the proposed ethod to a conteporary Chinese novel naed and translated as Fortress Besieged [15]. The book contains 216,601 characters in total, and it is encoded in GB18030, the largest official Chinese character set which contains 70,244 characters [16]. We start fro the i.i.d odel which uses 1,954,777 bits. The first single character to condition on is, which produces 12,631 bits of savings. It is actually the first character of the protagonist s first nae in the novel. It is indeed not surprising that seeing the first character highly indicates the next thing to see is the second for a two-character nae. We restrict the order of the Markov odel to be no larger than 5, but in fact no context exceeding two characters is chosen. There are 342 branches in the tree, aong which 95 are in the first layer, and 5 of the extends to the second layer. In fact, second layer branches are picked up only after ost first layer branches have already been chosen. A sall part of the tree is displayed in Fig.3. The dots on the right stand for the rest of the odel that cannot be shown. And the blank cell in the iddle of the first layer is actually the space sybol. The total savings aounts to 401,922 bits (about 20.56%) as copared to the i.i.d odel. Please note that existing odels for tree sources are ostly designed for sall alphabet copression, hence direct coparison with which would not be quite fair. V. CONCLUSION We consider a copression and prediction proble under both large alphabet i.i.d odel and bounded tree odels, and design a ethod to construct the context tree. Cobining this ethod with tilted Stirling ratio distribution as coding distributions for sybols with given contexts, we have a convenient and efficient way for copression and prediction. Ipleenting the proposed ethod on a Chinese novel, considerable savings in codelength is produced copared to the i.i.d odel. ACKNOWLEDGMENT The authors would like to thank Prof. Jun ichi Takeuchi, Prof. Wojciech Szpankowski and Prof. Narayana Santhana. Furtherore, we thank for the invitation to present part of the work in The Sixth Workshop on Inforation Theoretic Methods in Science and Engineering (Tokyo, Aug, 2013) and Center for Science of Inforation at Purdue University (West Lafayette, Oct, 2013). Soe ideas of the subsequent work are based on feedback fro these presentations. REFERENCES [1] I. of Applied Linguistice Ministry of Education. (2008, Nov) 2007 report on language use in china. [Online]. Available: / htl [2] Wikipedia, Chinese characters, Deceber [Online]. Available: [3] X. Yang and A. Barron, Large alphabet copression and predictive distributions through poissonization and tilting, arxiv: v1 [stat.me], [4] A. G. S. Boucheron and E. Gassiat, Coding on countably infinite alphabets, IEEE Transactions on Inforation Theory, vol. 55, no. 1, Jan [5] W. Feller, An Introduction to Probability Theory and Its Applications. Wiley, 1950, vol. 1. [6] H. D. J. Archarya, Tight bounds for universal copression of large alphabets, IEEE International Syposiu on Inforation Theory, [7] Y. M. Shtarkov, Universal sequential coding of single essages, Probles of Inforation Transission, vol. 23, no. 3, pp , [8]. Xie and A. R. Barron, Miniax redundancy for the class of eoryless sources, IEEE Transactions on Inforation Theory, vol. 43, pp , May [9] I. Csiszar, I-divergence geoetry of probability distributions and iniization probles, The Annals of Probability, vol. 3, no. 1, pp , Feb [10], Sanov property, generalized I-projection and a conditional liit theore, The Annals of Probability, vol. 12, no. 3, pp , Jan [11] J. V. Capenhout and T. Cover, Maxiu entropy and conditional probability, IEEE Transactions on Inforation Theory, vol. 27, no. 4, July [12] F. M. J. Willes, Y. Shtarkov, and T. Tjalkens, The context-tree weighting ethod: basic properties, IEEE Transactions on Inforation Theory, vol. 41, no. 3, pp , [13] J. Rissanen, A Universal Data Copression Syete, IEEE Transactions on Inforation Theory, vol. 29, no. 5, pp , [14] W. Szpankowski and M. J. Weinberger, Miniax redundancy for large alphabets, Inforation Theory Proceedings, June [15] Wikipedia, Fortress beseiged, October [Online]. Available: Besieged [16], Gb18030 stardard, January [Online]. Available: