Poissonization and universal compression of envelope classes

Transcription

1 Possonzaton and unversal compresson o envelope classes Jayadev Acharya, Ashkan Jaarpour, Alon Orltsky, and Ananda Theertha Suresh ECE UCSD, {jacharya, ashkan, alon, asuresh}@ucsd.edu Abstract Posson samplng s a method or elmnatng dependence among symbols n a random sequence. It helps mprove algorthm desgn, strengthen bounds, and smply proos. We relate the redundancy o xed-length and Posson-sampled sequences, use ths result to derve a smple ormula or the redundancy o general envelope classes, and apply ths ormula to obtan smple and tght bounds on the redundancy o powerlaw and exponental envelope classes, n partcular answerng a queston posed n [1]. I. INTRODUCTION Unversal compresson s the desgn o a sngle scheme to encode an unknown source rom a known class o dstrbutons. Let P be a dstrbuton over A. From the source codng theorem we know that H(P bts are necessary and sucent to encode P. Any dstrbuton Q over A mples a code that uses log Q(a bts to encode a A. The code mpled by the underlyng dstrbuton uses log P (a bts and expected code length s the entropy H(P. The (worst-case redundancy o a dstrbuton Q wth respect to P s sup log P (a a A Q(a, the largest derence between the number o bts used and the optmal. We assume the dstrbutons belong to a known class P. Then ˆR(P = n Q sup sup P P a A log P (a Q(a, s the worst case redundancy, or mnmax regret o P. de Let ˆP (a = sup P P P (a be the largest probablty assgned to symbol a by a dstrbuton n P. Then, S(P de = ˆP (a a A s the Shtarkov sum o P. Shtarkov [2] showed that ˆR(P = log (S(P, and the optmal dstrbuton assgns probablty ˆP (a/s(p to a A. In ths work we study codng o..d. samples rom unknown dstrbutons. For a dstrbuton P on an underlyng alphabet X let P n be the product dstrbuton P P... P over X n. Let P belong to a known class P. Let P n be the class o all dstrbutons P n wth P P. Codng length-n sequences over X equvalently mples encodng symbols rom A = X n. In such block encodng we treat each x n 1 X n as a symbol a. As beore we consder the class o all dstrbutons (not only..d. Q n over X n to dene ˆR(P n de = n sup sup log P n (x n 1 Q n P n x n 1 X n Q n (x n 1, as the worst case redundancy, or mnmax regret o P n. The class P s sad to be unversal ˆR(P = o(n,.e., redundancy s sublnear n the block-length. The most extensvely studed [3 12] class o dstrbutons s Ik n, the collecton o all..d. dstrbutons over length-n sequences rom an underlyng alphabet o sze k, e.g., over [k]. It s now well establshed that 1 For k = o(n 2 For n = o(k ˆR(I n k = k 1 2 log n k + k 2 + o(k, ˆR(I n k = n log k n + o(n. In partcular [3, 4] showed that class o all dstrbutons over N has nnte redundancy. Whle the problem was tradtonally studed n the settng o nte underlyng alphabet sze and large block lengths, however n numerous applcatons the underlyng natural alphabet best descrbng the data mght be large. For example, a text document only contans a small set o the entre dctonary, a natural mage has many possble pxel values. The class o all..d. dstrbutons over N s thereore too large to provde meanngul compresson schemes or all dstrbutons. These mpossblty results led researchers to consder natural subclasses o all..d. dstrbutons and then compress sequences generated rom dstrbutons n these or to consder all..d. dstrbutons but decompose sequences nto natural components and compress each part separately. We now descrbe three such approaches that have been proposed n recent years to characterze large alphabet compresson. [13] propose to encode the order o symbols (pattern n the sequences and the dctonary separately. [14, 15] studed the compresson o patterns and n partcular show that patterns can be compressed wth sublnear worst case redundancy.e., they are unversally compressble. [16] consder the class M o all monotone dstrbutons over N. They rst show that even ths class s not unversally compressble. Despte ths, they desgned unversal codes or

2 monotone dstrbutons wth nte entropy. [17] study the redundancy o M k, monotone dstrbutons over alphabet sze k, as a uncton o k and the block length n. In ths paper, we consder a thrd, natural and elegant approach, proposed by [1]. They study a arly general class o dstrbutons, called envelope classes, whch as the name suggests are dstrbutons that are bounded by an envelope. They provde general bounds on the redundancy o envelope classes. The upper bounds on the worst-case redundancy are obtaned by boundng the Shtarkov sum. They provde bounds on the more strngent (or lower bounds average case redundancy, where nstead o the supremum over all possble sequences, they consder the expected log rato,.e., the KL dvergence. For ths quantty, they provde a more complex employng the redundancy-capacty theorem. We now ntroduce the Posson samplng model, and later use t to provde redundancy bounds on envelope classes. II. POISSON MODEL Posson samplng provdes smpled analyss n a number o problems n machne learnng and statstcs [18]. To the best o our knowledge, the rst applcaton o Posson samplng n unversal codng was n [19], and [20] to prove near-optmal bounds on the pattern redundancy. More recently, [21] also apply Posson samplng to provde smpler codng schemes or compressng..d. dstrbutons. In ths work we are nterested n codng/predcton over countably nnte alphabets descrbed bounded by an envelope. Let po(λ be a Posson dstrbuton wth parameter λ and po(λ, µ de λ λµ = e µ! s the probablty o µ under a po(λ dstrbuton. For a dstrbuton P over X, recall that P n s a dstrbuton over X n. Smlarly, we dene a dstrbuton P po(n over X de = X X 2 X 3... as ollows. Frst generate n po(n, then sample the dstrbuton..d. n tmes. Thereore the probablty o any sequence x n 1 X s P po(n (x n 1 = po(n, n P n (x n 1 = po(n, n P (x. Smlar to P n, let P po(n de = {P po(n : P P} n be the class o dstrbutons over X va samplng a dstrbuton..d. po(n tmes. Under ths model, the advantage s that the number o appearances o symbols (multplctes are ndependent o each other. Note that even though the length o the sequence s random, t s concentrated around n as a Posson random varable. For large n, the length n o a Posson sampled sequence s concentrated around n. Usng ths, we show n Theorem 2 that t s sucent to consder P po(n nstead o P n. We rst descrbe some propertes o Posson sampled dstrbutons. A. Propertes o Posson samplng The multplcty o a symbol n a sequence s the number o tmes t appears n t. The type o a sequence x n 1 over X = {a 1,..., } s τ(x n 1 de = (µ(a 1, µ(a 2,..., the tuple o multplctes o the symbols n the sequence x n 1 [22, 23]. For example, a =, or = 1,..., 6 denotes the possble outcomes o a de. Then the sequence o outcomes 2, 3, 1, 6, 1, 3, 3, 4, 6 has type τ = (2, 1, 3, 1, 0, 2. Any product dstrbuton assgns the same probablty to sequences wth the same type. Thereore, or most statstcal problems, under ndependent samplng, t suces to consder only the type as the random varable. When X = k, ths corresponds to the balls and bns model wth n balls and k bns. The man dculty n analyzng such models s the dependences that arse among the multplctes, e.g., they add to n [18]. However, we consder po(n samplng, the dstrbuton P po(n has the ollowng propertes. Lemma 1 ([18]: For any dstrbuton P po(n, 1 condtoned on n, the dstrbuton on X n s P n or dscrete dstrbutons. 2 A symbol a X wth P (a = p appears po(np tmes ndependently o all other symbols. Ths shows that the multplctes are ndependent under Posson samplng. In the next result we relate the redundancy o Posson samplng to xed length samplng. In partcular, we show that t suces to consder P po(n. Theorem 2: For any ɛ > 0, or n > n 0 (ɛ, P ˆR(P po(n(1 ɛ 1 ˆR(P n ˆR(P po(n + 1. III. RELATED WORK AND NEW RESULTS Denton 3: The envelope class o a uncton s the class P de = {(p 1, p 2,... : p, and p 1 + p = 1} o all dstrbutons such that the symbol probablty s bounded by the value o the uncton at that pont. Analogous to P n, we can dene P n and Ppo(n. The study o unversal codng or envelope classes was ntated n [1], where they provde upper and lower bounds on the redundancy. They show that = the redundancy s nnte or any n, and thereore consder only absolutely ntegrable envelopes. Ther upper bound states that [ ˆR(P n mn n ( + u 1 ] log n + O(1. u n 2 u They prove a more complex lower bound on the averageredundancy. More recently, [21] provde bounds o smlar order or envelope classes usng Posson samplng. po(n We provde smple bounds on ˆR(P n terms o the redundancy o a smple one dmensonal class o dstrbutons characterzed by a sngle parameter λ max, as de = {po(λ : λ < λ max }.

3 Ths s the class o all Posson dstrbutons wth parameter bounded above. Ths class s smple enough, so that we can bound ts redundancy tghtly. For an envelope characterzed by, let λ max l be the smallest nteger such that j 1 ɛ. j l Ths also mples that j l λ max j n(1 ɛ. We now state our man result on envelope classes. Theorem 4: For the envelope class P, =l ˆR (POI(λ max ˆR ( P po(n de = n. Let ˆR (, where λ max = n. Snce, we use as a prmtve, we now show smple bounds that wll be used to compute redundances o specc classes later. Lemma 5: For λ 1, ˆR POI(λ = log (2 exp( λ, and or λ 1, 2(λ + 1 ˆR POI(λ 2 +. We now consder power-law and exponental envelopes and apply Theorem 4 to obtan sharp bounds on redundances o these classes mprovng the prevous results. To prove the ecacy o these bounds we apply them on the Power-law and exponental envelopes. Denton 6: The power-law envelope class wth parameters α > 1 and c s the class o dstrbutons Λ c α over N such that N, p c. α Denton 7: The exponental-law envelope class wth parameters α and c s the class o dstrbutons Λ ce α over N such that N, p ce α. [1] obtan bounds on the redundancy o these classes. For power-law envelopes they show that or large n, C 0 n 1 α n ˆR(Λc α ( 2cn α 1 1 α (log n 1 1 α + O(1, (1 where C 0 s a constant (uncton o α and c. For exponental envelopes they prove that log 2 n 8α (1 + o(1 ˆR(Λ n ce α log2 n 2α + O(1. [24] mprove the bounds or Λ n ce and show that α ˆR(Λ n ce α = log2 n 4α + O(log n log log n. More recently, [25] extend the arguments o [24] to nd tght unversal codes or the larger class o sub-exponental dstrbutons, whch have strctly aster decay than power-law classes, but slower than exponental. However these results do not nd the optmal redundancy o power-law envelopes. Ther technques seem to rely on the act that exponental envelopes restrct the support szes to be poly-logarthmc and break down or more heavy-taled dstrbutons, such as the power-law envelopes. We show that smply applyng Theorem 4 to these classes and boundng the resultng expressons gves tght redundancy bounds. In partcular, or Λ n c α, we show that Theorem 8: For large n (cn 1/α [ α + 1 ] 2 α 1 log 3 n 1 ˆR(Λc α (cn 1/α[ α ] α 1 + log For Λ n ce α, we prove that Theorem 9: For large n ˆR(Λ n ce α = log2 n 4α + O(log c log n. Remark Ths result can also be generalzed to provde tght bounds or sub-exponental envelope class consdered n [25]. In the next three sectons we prove these results. A. Proo o Theorem 2 IV. PROOFS For x n 1 X n, let ˆP n (x n 1 = sup P n P P n (x n n 1 be the maxmum lkelhood (ML probablty o x n 1. Then, S(P n = x n 1 X n ˆP n (x n 1. By the rst part o Lemma 1, t ollows that S(P po(n = n po(n, n S(P n. (2 By condtonng on x n 1 n the expresson or S(P n+1 t s easy to see that S(P n S(P n+1. Thereore, S(P po(n (a = n 0 po(n, n S(P n (b S(P n po(n, n (c 1 2 S(Pn, n n where (a ollows rom Equaton (2, (b rom monotoncty o S(P n and (c rom the act that medan o a Posson dstrbuton close to ts mean. Takng logarthms gves the second part o the theorem. The rst part o the theorem uses (2 agan along wth tal bounds on Posson random varables and s omtted here. It essentally uses the act that a po(n(1 ɛ random varable exceeds n wth exponentally small probablty. We omt the proo due to lack o space.

4 B. Proo o Theorem 4 For..d. samplng types are a sucent statstc o the sequence, namely all sequences wth the same type have the same probablty. Let τ(p n be the dstrbuton nduced by P n over types. Let τ(p n = {τ(p n : P P} be all dstrbutons over types rom dstrbutons o the orm P n. By the second tem n Lemma 1, or a dstrbuton P = (p 1, p 2,,... over X = {1, 2,...} τ(p po(n = (po(np 1, po(np 2,..., where each coordnate s an ndependent Posson dstrbuton. We rst show that the redundancy o sequences s the same as the redundancy o types. Lemma 10: ˆR(τ(P n = ˆR(P n and ˆR(τ(P po(n = ˆR(P po(n. Proo: Any..d. dstrbuton assgns the same probablty to all sequences wth the same type. Thereore, such sequences have the same maxmum lkelhood probablty. We show that the Shtarkov sums o the two classes are the same and hence they have same redundancy. S(P n = ˆP n (x n 1 = ˆP n (x n 1 x n 1 X n τ x n 1 :τ(xn 1 =τ = τ ˆP n (τ = S(τ(P n. The proo also apples to Posson samplng. Under Posson samplng, the type o a dstrbuton s a tuple o ndependent random varables, namely the multplctes. In general suppose P be a collecton o product (ndependent dstrbutons over A B,.e., each element n P s a dstrbuton o the orm P 1 P 2, where P 1 and P 2 are dstrbutons over A and B respectvely. Let P A and P B be the class o margnals over A and B respectvely. It can be shown, e.g., [20], that the redundancy o P s at most the sum o the margnal redundances. Lemma 11 (Redundancy o products: For a collecton P o product dstrbutons over A B, ˆR(P ˆR(P A + ˆR(P B. Furthermore, P = P A P B then equalty holds. Proo: For any (a, b A B, Now, sup P 1 (ap 1 (b sup P 1 (a sup P 2 (b. (P 1,P 2 P P 1 P A P 2 P B S(P = sup (P (a,b A B 1,P 2 P P 1 (ap 2 (b sup P (a sup Q(b = S(P A S(P B, P a A 1 P A P b B 2 P B where the nequalty ollows rom the equaton above, and the lemma ollows by takng logarthms. When all possble combnatons o margnals s possble, then the nequalty becomes an equalty. The lemma generalzes to more than two product classes. Smlar to the characterzaton o τ(p po(n t ollows that ( τ P po(n = { (po(λ 1, po(λ 2... : λ n, } λ = n. Notce that each λ λ max de = n, and thereore the class margnal dstrbutons o element s a Posson random varable wth parameter λ max and hence a subset o. Thereore, Lemma 11 yelds ˆR τ(p po(n ˆR ( 1 + ˆR ( For the lower bound note or any choce o λ < λ max or l corresponds to a dstrbuton n P po(n. In other words, all product dstrbutons n l l are vald projectons o a dstrbuton n P po(n along the coordnates l. Thereore, or the elements along these coordnates the redundancy o types under Posson samplng s determned precsely by Lemma 11. ˆR τ(p po(n ˆR l + ˆR l Combnng these wth Lemma 10 proves the theorem. C. Proo o Lemma 5 The Posson dstrbuton assgnng hghest probablty to N s po(. Thereore the ML dstrbuton o j n POI(λ s arg max POI(λ P ( = { po( po(λ Usng ths, the Shtarkov sum o the class s S POI(λ λ = =0 (a = 1 + λ otherwse. e! + λ λ e! λ +1 λ =0 (e! λ e λ,! where (a uses that µ po(λ, µ = 1. Usng the second expresson shows the case λ 1. Usng the rst along wth the ollowng Strlng s approxmaton proves the case o λ > 1. Lemma 12 (Strlng s Approxmaton: For any n 1, 1 there s a θ n ( 12n+1, 1 12n such that n! = ( n n 2n e θ n. e

5 D. Power-law: Proo o Theorem 8 Proo: By Denton 6, or power-law class Λ c α, λ max = cn. Let b de = (cn 1/α, then λ max α 1 or b and λ max < 1 otherwse. Then, (a n ˆR(Λc α b ˆR( + >b (b ( log 2 + b max + ˆR( + 1 where (a ollows rom Theorem 4 and (b rom Lemma 5. We consder the two summatons separately. For the rst term, we note that or λ 1, 2 + / < 3 λ and use t wth the ollowng smplcaton. B log B = log BB B! B, whch ollows rom B! > (B/e B. Thereore, ( b max b ( cn log 2 + < log 3 α = b log(3 + α 2 where (a ollows snce b = (cn 1/α. Takng the second term, =b+1 log(2 e λmax (a b (a < (cn 1/α( log(3 + α 2 =b+1 λ max = cn ( 1 (cn α log, =b+1 1 α = c1/α α 1 n1/α, where (a uses e x 2 e x, and (b ollows by usng s = b + 1 = (cn 1/α + 1 n =s 1 r s 1 (s 11 r. (x 1 r (r 1 Combnng these results proves the theorem. For the lower bound we only use the terms λ max or j l. Usng the lower bound on S(POI(λ rom Lemma 5 and agan applyng the Strlng s approxmaton (lower bound nstead o upper proves the result. Snce the deas are the same, we omt the proo. E. Exponental class: Proo o Theorem 9 Proo: By Denton 7 or Λ n ce α, log(cn α and only λ max 1. Let b de = log(cn α. Then smlar to the proo o power-law class we derve the two summatons and bound them ndvdually. Once agan or λ > 1, 2 + / < 3 λ. Thereore ( b max log 2 + = b log b log[cne α ] 2 =b = b log b α(b < b log(3 + b2 α 4. log(2 e λmax + For 1, the second term, snce e αb = cn, >b log(2 e λmax λ max = =b =b cne α 1 1 e α. Substtutng b = log(cn/α proves the upper bound. Once agan the lower bound s very smlar and s omtted. REFERENCES [1] S. Boucheron, A. Garver, and E. Gassat, Codng on countably nnte alphabets, IEEE Transactons on Inormaton Theory, vol. 55, no. 1, pp , [2] Y. M. Shtarkov, Unversal sequental codng o sngle messages, Problems o Inormaton Transmsson, vol. 23, no. 3, pp. 3 17, [3] J. Keer, A uned approach to weak unversal source codng, IEEE Transactons on Inormaton Theory, vol. 24, no. 6, pp , Nov [4] L. Davsson, Unversal noseless codng, IEEE Transactons on Inormaton Theory, vol. 19, no. 6, pp , Nov [5] L. D. Davsson, R. J. McElece, M. B. Pursley, and M. S. Wallace, Ecent unversal noseless source codes, IEEE Transactons on Inormaton Theory, vol. 27, no. 3, pp , [6] F. M. J. Wllems, Y. M. Shtarkov, and T. J. Tjalkens, The context-tree weghtng method: basc propertes. IEEE Transactons on Inormaton Theory, vol. 41, no. 3, pp , [7] T. Cover, Unversal portolos, Mathematcal Fnance, vol. 1, no. 1, pp. 1 29, January [8] J. Rssanen, Fsher normaton and stochastc complexty, IEEE Transactons on Inormaton Theory, vol. 42, no. 1, pp , January [9] W. Szpankowsk, On asymptotcs o certan recurrences arsng n unversal codng, Problems o Inormaton Transmsson, vol. 34, no. 2, pp , [10] Q. Xe and A. R. Barron, Asymptotc mnmax regret or data compresson, gamblng and predcton, IEEE Transactons on Inormaton Theory, vol. 46, no. 2, pp , [11] A. Orltsky and N. Santhanam, Speakng o nnty [..d. strngs], tt, vol. 50, no. 10, pp , oct [12] W. Szpankowsk and M. J. Wenberger, Mnmax redundancy or large alphabets, n ISIT, 2010, pp [13] J. Åberg, Y. M. Shtarkov, and B. J. M. Smeets, Multalphabet codng wth separate alphabet descrpton, n Proceedngs o Compresson and Complexty o Sequences, [14] A. Orltsky, N. Santhanam, and J. Zhang, Unversal compresson o memoryless sources over unknown alphabets, IEEE Transactons on Inormaton Theory, vol. 50, no. 7, pp , July [15], Always Good Turng: Asymptotcally optmal probablty estmaton, Scence, vol. 302, no. 5644, pp , October , see also Proceedngs o the 44th Annual Symposum on Foundatons o Computer Scence, October [16] D. Foster, R. Stne, and A. Wyner, Unversal codes or nte sequences o ntegers drawn rom a monotone dstrbuton, IEEE Transactons on Inormaton Theory, vol. 48, no. 6, pp , June [17] G. I. Shamr, Unversal source codng or monotonc and ast decayng monotonc dstrbutons, IEEE Transactons on Inormaton Theory, vol. 59, no. 11, pp , [18] M. Mtzenmacher and E. Upal, Probablty and computng - randomzed algorthms and probablstc analyss. Cambrdge Unv. Press, [19] J. Acharya, H. Das, and A. Orltsky, Tght bounds on prole redundancy and dstngushablty, n NIPS, 2012.

6 [20] J. Acharya, H. Das, A. Jaarpour, A. Orltsky, and A. Suresh, Tght bounds or unversal compresson o large alphabets, n Inormaton Theory Proceedngs (ISIT, 2013 IEEE Internatonal Symposum on, 2013, pp [21] X. Yang and A. R. Barron, Large alphabet codng and predcton through possonzaton and tltng, n Workshop on Inormaton Theoretc Methods n Scence and Engneerng, [22] H. Das, Compettve tests and estmators or propertes o dstrbutons, Ph.D. dssertaton, UCSD, [23] T. Cover and J. Thomas, Elements o Inormaton Theory, 2nd Ed. Wley Interscence, [24] D. Bontemps, Unversal codng on nnte alphabets: Exponentally decreasng envelopes, IEEE Transactons on Inormaton Theory, vol. 57, no. 3, pp , [25] D. Bontemps, S. Boucheron, and E. Gassat, About adaptve codng on countable alphabets, IEEE Transactons on Inormaton Theory, vol. 60, no. 2, pp , 2012.