On the Finite-Length Performance of Universal Coding for k-ary Memoryless Sources

Transcription

1 Forty-ghth Annual Allerton Conference Allerton House, UIUC, Illnos, USA Septeber 9 - October, 00 On the Fnte-Length Perforance of Unversal Codng for -ary Meoryless Sources Ahad Bera and Faraarz Fer School of lectrcal and Coputer ngneerng Georga Insttute of Technology, Atlanta GA 3033, USA al: bera, fer}@ece.gatech.edu Abstract In ths paper, we nvestgate the perforance of unversal codng schees on fnte-length eoryless sequences. Rssanen deonstrated that for the unversal copresson of -ary eoryless sources, expected redundancy for regular codes s asyptotcally lower bounded by logn for alost all sources. Xe and Barron derved the nax expected redundancy for -ary eoryless sources, whch characterzes the axu redundancy over all possble source paraeters. It does not provde uch nforaton about dfferent source paraeter values. Ths paper s a fnte-length extenson to Rssanen s result. Our treatent n ths paper s probablstc. In partcular, we derve a lower bound on the probablty easure of the sources that are not copressble wth a redundancy saller than a certan fracton of log n. In other words, we deonstrate a lower bound on the redundancy for a gven percentle of sources. We deonstrate that as the length of the eoryless sequence decreases, the redundancy tends to becoe sgnfcant and coparable to the entropy of the sequence. I. INTRODUCTION Recently, the aount of data that s beng stored n storage systes has been ncreasng wth a very hgh rate. Hence, the developent of copresson for storage systes has ganed a lot of nterest. In any cases, one coplete database ay be copressed to less than one tenth of ts orgnal sze. The redundancy n the data ay be leveraged to sgnfcantly reduce the cost of data antenance as well as data transsson. However, ost applcatons requre that ndvdual fles be retreved and updated separate fro the rest of the database. Therefore, the copressed data for ndvdual fles need to be ndependently retrevable and updateable. On the other hand, the ndvdual fles are relatvely very sall. Moreover, the fles ay coe fro varous natures, whch calls for unversal copresson. It s well nown that f a sall fle s copressed alone, exstng unversal copresson technques are unable to acheve good perforance. Rssanen s result descrbes asyptotc achevable copresson rates for unversal copresson of ndvdual fles,. However, t does not provde uch nsght on the perforance of unversal codng for a sall gven fle. The nax redundancy 3 s concerned wth the axu redundancy over all paraeters. However, t does not characterze the redundancy for other paraeters. Snce Shannon s wor on the analyss of councaton systes 4, any researchers have contrbuted towards the developent of source codng schees wth the copresson rate as close as possble to the entropy rate of the nforaton source. It s well nown that usng a prefx-free code, a statonary ergodc nforaton source cannot be copressed at a rate lower than the entropy rate of the source 5. Thus, the goal of source codng s to acheve a copresson rate that approaches the entropy rate. In ths paper, we focus on the unversal copresson of eoryless sources wth a fnte-alphabet sze. Let S denote a unversal eoryless nforaton source wth -ary alphabetα = α,...,α }. Further, denoteθ = θ,...,θ Theta as the vector n the -densonal splex of source paraeters such that for a sybol X generated by S, PX = α = θ, for. Let hθ be the entropy rate of S gven paraeters θ,.e., hθ = j θ j logθ j. Fnally, we use the notaton X n = X,...,X n to present a vector of length n of d rando varables generated by S. Let lx n L denote the regular length functon that descrbes the codeword assocated wth the sequence X n. Denote R n l,θ as the expected redundancy of the code wth length functon l. and paraeter vector θ on a sequence of length n, defned as the dfference between the expected codeword length and the entropy of the sequence X n : R n l,θ = lx n nhθ. For an asyptotcally optal code wth length functon lx n, Rnl,θ n 0 as n for all θ. Provded that the statstcs of the nforaton source are nown, Huffan bloc codng acheves the entropy rate wth a redundancy saller than bt per source sybol 5. However, the assupton of nown source statstcs fals to hold for any practcal applcatons. We usually cannot assue a pror nowledge on the statstcs of the source although we stll wsh to copress the unnown statonary ergodc source to ts entropy rate. Ths s nown as the unversal copresson proble. The axu average redundancy for a length functon of a code wth length functon l s gven as R n l = ax θ Θ R n l,θ, whch s lower bounded by the nax average redundancy R n = n l L ax θ Θ R n l,θ 3. The leadng ter of the average nax redundancy s asyptotcally log n. Accordng to Rssanen s results, for the unversal copresson of eoryless sources wth /0/$ I 740

2 unforly dstrbuted paraeter vector θ, the redundancy of regular codes s asyptotcally lower bounded byr n l,θ δ logn, for all ǫ > 0 and alost all sources. These results were later extended n 6, 7 to ore general classes of sources. The asyptotc lower bound s tght snce there exst codng schees that acheve the bound asyptotcally, 8. We wll forally state Rssanen s result n Sec. II. In ths paper, we study the redundancy for the unversal copresson n fnte-length rege. As the frst step, we consder unversal codng for -ary eoryless sources. The wor can be vewed as the extenson of Rssanen s probablstc treatent to the fnte-length sequences. The rest of ths paper s organzed as follows. In Secton II, we forally state the fnte-length unversal copresson proble followed by our an result. In Secton III, we present the proof of the an result. In Secton IV, we deonstrate the sgnfcance of the an result through two an exaples. Fnally, the concluson s gven n Secton V. II. PROBLM STATMNT AND MAIN RSULTS In ths secton, we forally state the fnte-length redundancy proble and present our an result and ts plcatons. We focus on two-part codes, where the copresson schee utlzes bts for the dentfcaton of an estate for the unnown source paraeters. The estate paraeter s then used for the copresson of the sequence. It has been deonstrated that the two-part assupton brngs about a sall O redundancy ter 9. Ths corresponds to possble estate ponts n the paraeter space for the unnown paraeter. In other words, the copresson schee chooses the best aong estate ponts n the paraeter space as t copresses the nput sequence. Let Φ = φ,...,φ } denote the set of all estate ponts. Note that each φ s a pont n the -densonal splex of θ. For each sequence X n, there s an estate pont β = βx n Φ that nzes the redundancy. Let r denote the nuber of tes sybol α appears n the sequence X n. Let f denote the relatve frequency of sybol α,.e., f = r /n. Let µ θ denote the probablty easure defned over a eoryless source wth paraeter vector θ as µ θ X n = PX n θ = θ r...θr. We requre the length functon l θ X n be regular,.e., l θ X n log µ θ X n X n, 3 where µ θ X n s the eoryless probablty easure defned by the paraeter vectorθ. Note that the requreent 3 s not restrctve snce all codes that we now are regular. Let l β X n denote the length functon nduced by β Φ. Further denote µ β X n as the probablty easure nduced by β: µ β X n = β r...β r. 4 In ths paper logx always denotes the logarth of x n base. Increasng results n an exponental growth n the nuber of estate ponts and ore accurate estate for the unnown source paraeter vector hence better copresson. On the other hand, ncreasng drectly ncreases the copresson overhead. Therefore, t s desrable to fnd the best that nzes the total codeword length as lx n = n +l βx n }. 5 Snce we assued the code s regular, we ay use 3 to bound the redundancy rate } R n θ n +log µ β X n nhθ. 6 Usng 4, we get R n θ n +n } nhθ. 7 = Our goal s to better characterze the lower bound n 7. In, Rssanen proved an asyptotc lower bound on the unversal copresson of paraetrc sources that can be represented wth paraeters. The followng asyptotc lower bound on the redundancy rate of unversal codng of - ary nforaton sources s a drect consequence of Rssanen s result: Theore Let S denote a -ary eoryless nforaton source wth paraeter vectorθ. LetX n denote a sequence of lengthnproduced bys. Let lx n denote any regular length functon for unversal copresson. Then, for all paraeters θ, except n a set of asyptotcally Lebesgue volue zero, we have l n R n θ ǫ, ǫ > 0, 8 logn Whle Theore descrbes the asyptotc fundaental lts of the unversal copresson of eoryless nforaton sources, t does not provde uch nsght for the case of fnte-length n. Moreover, the result excludes an asyptotcally volue zeros set of paraeter vectors θ that has non-zero volue for any fnte n. In 3, Xe and Barron derve the expected nax redundancy R n for eoryless sources, where they deonstrate that R n = n log +log π Γ/ Γ/ +o, 9 where Γx = 0 t x e t dt s the gaa functon. The nax redundancy characterzes the axu redundancy on the space of all possble paraeter vectors but does not ply uch about the rest of the space of the paraeter vectors. In ths paper, we derve a lower bound on the probablty that a source wth paraeter vector θ s copressed wth redundancy rate R n θ > R 0 for any R 0 > 0. In other words, we fnd a lower bound on PR n θ > R 0. Usng ths 74

3 result, we deonstrate the fundaental lts of the unversal copresson for fnte-length n. The followng s our an result: Theore Let S denote a -ary eoryless nforaton source wth paraeter vector θ that follows the Drchlet dstrbuton. Let X n denote a sequence of length n produced by S. Let lx n denote any regular length functon. Let ǫ be a real nuber such that 0 < ǫ <. Then, the probablty that R n θ s greater than ǫ tes the asyptotc bound s lower bounded as follows: P R n θ logn ǫ where B = Γ Γ + B en ǫ, 0 π. Note that B π for. Proof: See Secton III. Note that t s straghtforward to deduce Theore fro Theore by lettng n. Note that for any ǫ such that B > 0, R ǫ = ǫ logn s a lower bound on the nax redundancy snce there exsts a source that has a redundancy of at least R ǫ. We wll later deonstrate that ths lower bound s actually tght, and gves bac the nax redundancy. en ǫ III. PROOF OF TH MAIN RSULT In ths secton, we present the proof of Theore. We brea down the an proof to a few nteredate leas that consttute the an proof. Frst, we splfy 7 as Lea The redundancy rate R n θ s lower bounded by } θ R n θ n +n X n. = Proof: Ths s straghtforward snce hθ = = f log θ. In order to bound the redundancy n Lea, n the followng, we bound = f θ log X n. Note that ths ter plctly depends on snce s a functon of. Lea Assue that β Φ such that ; Dθ β Dθ φ. Then, we have where θ X n Dθ β, = Dx y = x j log j= xj y j. Proof: See Appendx I Note that Dθ s the non-negatve Kullbac Lebler dvergence between the probablty easures defned by θ and β. We use a probablstc treatent n order to bound Dθ β for a certan percentage of source paraeters. We assue that the paraeter vector θ follows the Drchlet dstrbuton. The Drchlet pror dstrbuton s partcularly nterestng snce t results n unfor redundancy over the paraeter vector space, whch results n the acheveent of the nax expected redundancy 3, 0. The Drchlet probablty dstrbuton for the paraeter vector θ s gven by: pθ = Γ/ Γ/ j= θj, 3 where Γ. s the gaa functon defned n Theore. In order to bound the redundancy rate R n θ, n the followng, we fnd an upper bound on the Lebegue easure of the volue defned by Dθ β < δ n the - densonal splex of θ. Snce β Φ, the total easure of the volue defned by φ Φ; Dθ φ < δ ay be upper bounded as well. Ths provdes wth a lower bound on the easure of the sources wth R n θ δ. Lea 3 Let β = β,...,β and θ = θ,...,θ be a fxed and a varable pont n the -densonal splex of θ, respectvely. If θ follows the Drchlet dstrbuton, the probablty PDθ β < δ s upper bounded by PDθ β < δ Γ δ Γ. 4 + Moreover, P φ Φ; Dθ φ < δ s upper bounded by P φ Φ; Dθ φ < δ Γ δ Γ. + 5 Proof: See Appendx II. Lea 3 states that the probablty easure that s covered PDθ β < δ does not depend on β when θ follows the Drchlet pror. In other words, the choce of the set of the paraeter ponts Φ does not affect the perforance of the copresson. We are now equpped to prove the an result gven n Theore. Proof: Proof of Theore Lea 3 bounds the probablty easure of the paraeter vectors that ay be copressed wth a sall redundancy. The condton n Lea s satsfed f < logn for ponts close to soe φ Φ. Ths s because = on and thus the dstance between the ponts s ω/ n. Usng Leas and, R n θ n n +ndθ φ }. 6 Therefore, P R n θ logn ǫ 7 74

4 R PrR n θ,l>r 0 P 0 n = 8 Ths Wor n = 8 Mnax n = 3 Ths Wor n = 3 Mnax n = 8 Ths Wor n = 8 Mnax n = 5 Ths Wor n = 5 Mnax P 0 R PrR n θ,l>r 0 P 0 n = 5b Ths Wor 0 n = 5b Mnax n = b Ths Wor n = b Mnax 0 n = 8b Ths Wor n = 8b Mnax n = 3b Ths Wor n = 3b Mnax P 0 Fg.. Redundancy level R 0 as a functon of percentle of sources P 0 wth R nl,θ > R 0. Alphabet sze: =. P n n = n n P n +ndθ φ } ǫ Dθ φ ǫ Γ Γ + π n logn n } δ loge logn The last nequalty s obtaned usng Lea 3. Here, δ s gven by δ = ǫ n logn n. Carryng out the nzaton n 0 leads to the desred result n Theore. IV. LABORATION OF TH RSULTS AND XAMPLS In ths secton, we llustrate the results through two exaples. In Fgures and, the x-axs s a percentle P 0 and they-axs represents a redundancy levelr 0. The sold curves deonstrate the derved lower bound on the redundancy as a functon of the percentle of the sources that have redundancy beyond the specfed level based on Theore,.e., we have PR n θ R 0 P 0. In other words, at least a fracton P 0 of the sources that are chosen fro the Drchlet pror have an expected redundancy that s greater than R 0. A. Mnax Redundancy of Two-Part Codes: Lower Bound Note that the sold curves are a lower bound on the nax redundancy for all values of P 0 > 0. Ths s due to the fact that a non-zero percentage of the sources ay not be copressed beyond the level R 0 hence the worst case s ncluded n ths regon. Therefore, we ay use our result to derve a lower bound on the nax redundancy of two-part codes when P 0 0 as a sde product: R n,p logn logb log + loge, Fg.. Redundancy level R 0 as a functon of percentle of the sources P 0 wth R nl,θ < R 0. Alphabet sze: = 56. where B s defned n Theore. Ths ay be rewrtten as follows: + R n,p R n +logγ log. 3 e Note that t s straghtforward to deonstrate that the lower bound R n,p R n as,.e., the effect of two-part assupton wll be neglgble. B. xaple : = Frst, for a Bernoull nforaton source,.e., =, we deonstrate the fundaental lts for the copresson of fnte-length sequences. The plots are gven for varous n. The arers n ths Fgure labeled as Mnax deonstrate the nax redundancy for a two-part code. For a Bernoull source, the nax redundancy of the two-part code s gven by 9: R n,p = R n + πe log R n +.048, 4 where R n s the nax redundancy and s defned n 9. As shown n Fgure, at least 40% of Bernoull sequences of length n = 3 n = 8 ay not be copressed beyond a redundancy of bts per sybol. Also, at least 60% of Bernoull sequences of length n = 3 n = 8 ay not be copressed beyond a redundancy of bts per sybol. Note that as n, the redundancy approaches the nax redundancy for ore sources, and asyptotcally redundancy approaches the nax redundancy for all sources as gven n Theore. Also, usng we get the followng lower bound on the nax redundancy of two-part codes for = : R n,p logn+ log πe whch s ndeed equal to the nax redundancy., 5 743

5 C. xaple : = 56 Next, we consder = 56, whch s a coon practce to use the byte as a sybol. In Fgure, the achevable redundancy s deonstrated for four dfferent values of n. Here, the redundancy s easured n bts per source sybol byte. We observe that for n = 5b, we have R n l,θ 300 bts/sybol for alost all sources. Ths s consdered sgnfcant specally for sources wth sall entropy rate. For exaple, f the entropy rate s around bt/sybol, there wll be alost always ore than 50% redundancy n the copresson of a sequence of length n = 5b. We also observe that the two-part assupton does not ncur further redundancy for large. V. CONCLUSION In ths paper, we nvestgated the redundancy rate of unversal codng schees on eoryless nput sequences n the fnte-length rege. We derved a lower bound on the probablty easure of nforaton sources that ay not be copressed beyond any certan redundancy level. Our result ay be vewed as the fnte-length extenson of the prevous asyptotc results. Our result ay be used to evaluate the perforance of unversal source codng on fnte-length sequences. We observe that redundancy ay be very sgnfcant n the copresson of fnte-length lowentropy nforaton sources. Proof: = n! r,...,r j rj=n = APPNDIX I PROOF OF LMMA = r!...r! θr...θr θ X n 6 r n log θ X n, 7 n! s,...,s s!...s! θs...θs θ θ log j sj=n X n, 8 where s = r and s j = r j, j. Note that ths could now be vewed as tang expectatons over a dfferent set of varables. Thus, 8 could be rewrtten as θ log = θ X n. 9 Ths ay be lower bounded usng the fact φ Φ, Dθ φ Dθ β, whch proves the cla. APPNDIX II PROOF OF LMMA 3 Proof: Let fθ = Dθ β = = θ log usng Taylor expanson, we get θ. Then, where Lθ,β = loge = θ. Note that Lθ,β δ, where δ > 0, defnes an ellpsod on the -densonal splex of θ. It s straghtforward to deonstrate that the volue of the ellpsod s gven by δ V β = C loge β, 3 = where C = Γ/ Γ+/ s the volue of the - densonal unt ball. Moreover, snce θ follows the Drchlet dstrbuton, the probablty easure of the covered ellpsod s gven by PLθ,β δ = V β Γ/ Γ/ β j= = Γ δ Γ. 3 + Snce Dθ β Lθ,β, the volue defned by Dθ β < δ s equal to the volue Lθ,β < δ, whch copletes the proof of the frst cla. Although the volue of the ellpsod depends on the pont n the paraeter space, the volue of the ellpsod defned by Dθ β < δ does not depend on β. For the second cla, there are choces for φ., Dθ φ defnes a easure that has the sae upper bound. Thus, the second cla follows drectly fro the frst cla usng the unon bound. RFRNCS J. Rssanen, Unversal codng, nforaton, predcton, and estaton, Inforaton Theory, I Transactons on, vol. 30, no. 4, pp , Jul 984., Coplexty of strngs n the class of Marov sources, Inforaton Theory, I Transactons on, vol. 3, no. 4, pp , Jul Q. Xe and A. Barron, Mnax redundancy for the class of eoryless sources, Inforaton Theory, I Transactons on, vol. 43, no., pp , ar C.. Shannon, A Matheatcal Theory of Councaton, The Bell Syste Techncal Journal, vol. 7, pp , , Jul, Oct T. M. Cover and J. A. Thoas, leents of Inforaton Theory. John Wley and sons, N. Merhav and M. Feder, The nax redundancy s a lower bound for ost sources, n Data Copresson Conference, 994. DCC 94. Proceedngs, , pp M. Feder and N. Merhav, Herarchcal unversal codng, Inforaton Theory, I Transactons on, vol. 4, no. 5, pp , sep F. Wlles, Y. Shtarov, and T. Tjalens, The context-tree weghtng ethod: basc propertes, Inforaton Theory, I Transactons on, vol. 4, no. 3, pp , May P. D. Grunwald, The nu descrpton length prncple. The MIT Press, R.. Krchevsy and V. K. Trofov, The perforance of unversal encodng, Inforaton Theory, I Transactons on, vol. 7, no., pp , March 98. Dθ β Lθ,β+Oθ 3,