A Tutorial of The Context Tree Weighting Method: Basic Properties

Transcription

1 A uoril of h on r Wighing Mhod: Bic ropri Zijun Wu Novmbr 9, 005 Abrc In hi uoril, ry o giv uoril ovrvi of h on r Wighing Mhod. W confin our dicuion o binry boundd mmory r ourc nd dcrib qunil univrl d comprion procdur, hich chiv dirbl coding diribuion for r ourc ih unknon modl nd unknon prmr. ompuionl nd org compliy of h propod procdur r boh linr in h ourc qunc lngh.. Inroducion In our cl, lrnd Huffmn cod. For mll ourc lphb, hough, hv fficin coding only if u long block of ourc ymbol. I i hrfor dirbl o hv n fficin coding procdur h ork for long block of ourc ymbol. Huffmn coding i no idl for hi iuion. Arihmic coding chiv hi gol. Whn no priori probbiliy diribuion knoldg of h ourc i vilbl, on u univrl coding lgorihm. For fini mmory ourc modl, Winbrgr, Ziv nd Lmpl dvlopd qunil lgorihm for univrl coding, hr rificil prmr K i involvd. o void chooing prmr, Willm, hrkov nd jlkn propod ighing mhod. hi mhod im h ourc probbiliy givn p ymbol nd hn combin i ih rihmic lgorihm. W ill oulin h bic of h mhod in hi uoril.. Binry Boundd Mmory r ourc A binry r ourc gnr qunc of digi uming vlu in h lphb {0,}. W dno by n m h qunc m m+ n nd llo m nd n o b infinily lrg. For n<m h qunc i mpy, dnod byφ. h iicl bhvior of binry fini mmory r ourc cn b dcribd by mn of uffi. hi uffi i collcion of binry ring (k, ih k=,,,. W rquir i o b propr nd compl. roprn of h uffi impli h no ring in i uffi of ny ohr ring in. ompln gurn h ch mi-infini qunc (ring h uffi h blong o. hi uffi i uniqu inc i propr. n n n A boundd mmory r ourc h uffi h ifi l ( for ll. W y h h ourc h mmory no lrgr hn. W dfin uffi funcion β (, hich mp mi-infin qunc ono hir uniqu uffi in. finiion : h cul n-ymbol probbilii for boundd mmory r ourc

2 ih uffi nd prmr vcor r ( X =,, = ( X = 0,, = θ ( for ll. h cul block probbilii r no produc of cul n-ymbol probbilii, i.. ( X 0 =,, = ( X = τ τ = τ τ τ,, All ourc ih h m uffi r id o hv h m modl. h of ll r modl hving mmory no lrgr hn i clld h modl cl. I i poibl o pcify modl in hi modl cl by nurl cod by ncoding h uffi rcurivly. h cod of i h cod of h mpy ring λ. h cod of ring i void if l ( = ; ohri, i i 0 if nd follod by h cod of h ring 0 nd if. If u hi nurl cod, h numbr of bi h r ndd o pcify modl i qul o Γ (, hr finiion : Γ (, h co of modl ih rpc o modl cl i dfind Γ ( = - + {:, l ( } hr i i umd h. o fr, v propod modl for h ourc. And lrdy kno rihmic coding i good qunil mhod in h n h i individul coding rdundncy i no mor hn bi. No our k i o find diribuion im hich bridg bn h modl nd rihmic coding mhod. In h folloing, dfin h coding rdundncy, dcrib h rdundncy uppr bound of rihmic coding, nd hn ighing mhod diribuion im. β 3. od nd Rdundncy W um h boh h ncodr nd h dcodr hv cc o h p ourc ymbol =, o h implicily h uffi h drmin h probbiliy diribuion 0 0 of h fir ourc ymbol, i vilbl o hm. W dno h funcionl rlionhip L 0 bn ourc qunc nd codord o b c (. h lngh of h codord, in 0 binry digi, i dnod L(. W rric ourlv o prfi cod hr. 0 h codord lngh L( drmin h individul rdundnci. 0 finiion 3: h individul rdundncy ρ (,, of qunc givn h p ymbol 0, ih rpc o ourc ih modl nd prmr vcor, i dfind

3 0 ρ (,, = 0 L( log 0 (,, 0 W only conidr qunc ih poiiv probbiliy (,,. 4. Arihmic oding uppo h h ncodr nd dcodr boh hv cc o, h i clld h coding diribuion (, {0,}, = 0,,, W rquir h hi diribuion ifi (φ =, ( = (, X = 0 + (, X, for ll = {0,}, = 0,,, nd ( > 0, for ll poibl {0,} ( hrom : Givn coding diribuion (, {0,}, = 0,,, h Eli lgorihm chiv codord lngh L ( h ify L ( log ( < + for ll poibl {0,}. h codord form prfi cod. W y h h coding individul coding rdundncy i ly l hn bi. h Eli lgorihm combin n ccpbl coding rdundncy ih dirbl qunil implmnion. h numbr of oprion i linr in h ourc qunc lngh. If r rdy o ccp lo of mo bi coding rdundncy, r no lf ih h problm of finding good, qunilly vilbl, coding diribuion. 5. robbiliy Eimion h probbiliy h mmoryl ourc ih prmr θ gnr qunc ih zro nd b on i ( θ θ b. If igh hi probbiliy ovr ll θ ih (, -irichl diribuion obin h o-clld Krichvky-rofimov im ( [5]. finiion 4: h Krichvky-rofimov imd probbiliy for qunc conining 0 zro nd b 0 on i dfind = b (, ( θ θ dθ 0 π ( θ θ 3

4 hi imor h propri h r lid in h lmm h follo. Lmm : h K- probbiliy imor (, cn b compud qunilly, i.., (0,0 =, nd for 0 nd b 0 + ( +, = (, nd + b + b + (, b + = (, ( + b + ifi, for + b, h folloing inquliy: + b b (, = ( ( + b b + b 6. oding for n Unknon r ourc A. finiion of h on-r Wighing Mhod onidr h c hr hv o compr qunc hich i (uppod o b gnrd by r ourc, ho uffi nd prmr vcor r unknon o h ncodr nd h dcodr. W ill dfin ighd coding diribuion for hi iuion. finiion 5: h con r i of nod lbld, hr i (binry ring ih lngh l ( uch h 0 l(. Ech nod ih l ( <, pli up ino o nod, 0 nd. h coun mu ify = 0 + nd b 0 + b = b finiion 6: o ch nod, ign ighd probbiliy dfind : (, b + (, b 0 for for 0 l ( l ( = (3 hich i h con r oghr ih h ighd probbilii of h nod i clld ighd con r. W dfin our ighd coding diribuion ( ( for ll {0,}, = 0,,,, hr λ i h roo 0 λ 0 nod of h con r. (4 W cn chck h h ighd coding diribuion ifi (. B. An Uppr Bound on h Rdundncy finiion 7: L 4

5 z for 0 z < γ ( z log z + for z h bic rul concrning h con-r ighing chniqu cn b d no. horm : h individul rdundnci ih rpc o ny ourc ih modl nd prmr vcor r uppr-boundd by 0 ρ (,, < Γ ( + γ ( + for ll {0,}, for ny qunc of p ymbol 0. h hr rm on h righ hnd id of h inquliy rprn uppr bound of modl rdundncy, prmr rdundncy nd coding rdundncy rpcivly. orollry: Uing h coding diribuion in (4, h codord lngh L ( r 0 uppr boundd by L ( < min(min log +Γ ( + γ ( + (,, 0 0. Implmnion of h on-r Wighing Mhod Encoding nd coding: W um h nod conin h pir (, b, h imd probbiliy (, b nd h ighd probbiliy. Whn nod i crd, h coun nd b r md 0, h probbilii (, b nd r md. W hn u h upd chm indicd by ( & (3 o g h coding diribuion. hn h ncoding nd dcoding procdur follo hrough ih h Eli lgorihm. ompliy Iu: For ch ymbol hv o vii + nod. om of h nod hv o b crd fir. From hi i follo h h ol numbr of llocd nod cnno b mor hn ( +. hi mk h org compliy no mor hn linr in. No lo h h numbr of nod cnno b mor hn +, h ol numbr of nod in. h compuionl compliy, i.. h numbr of ddiion, muliplicion, nd diviion, i proporionl o h numbr of nod h r viid, hich i ( +. hrfor, hi compliy i lo linr in. 5

6 REFERENE []. M. ovr nd J. A. hom, Elmn of Informion hory, N York: Wily, 99. [] J. Ziv nd A. Lmpl, A univrl lgorihm for qunil d comprion, IEEE rn. Inform. hory, vol. I-3, NO.3, My 977. [3] M. J. Winbrgr, A. Lmpl, nd J. Ziv, A qunil lgorihm for h univrl coding of fini mmory ourc, IEEE rn. Inform. hory, vol. 38, pp , My 99. [4] G. G. Lngdon, An inroducion o rihmic coding. IBM Journl of Rrch nd vlopmn, 8: 35-49, 984 [5] J. Rinn nd G. G. Lngdon, Univrl modling nd coding, IEEE rn. Inform.hory, vol. I-7, pp. -3, Jn