Context model automata for text compression (published in The ComputerJournal, 41 (7), , 1998)

Transcription

1 Context model utomt for text ompression (pulished in The ComputerJournl, 41 (7), , 1998) Psi Fränti Deprtment of Computer Siene, University of Joensuu, Box 111, FIN Joensuu, FINLAND Timo Htkk Deprtment of Informtion Tehnology, Lppeenrnt University of Tehnology Box 20, FIN Lppeenrnt, FINLAND Astrt Finite-stte utomt offer n lterntive pproh to implement ontext models where the sttes in the utomt nnot in generl e ssigned y single ontext. Despite the potentil of this pproh, it mkes the design of the modelling more prolemti euse the ext ehviour of the model is not known. Here we propose simple formlism ontext model utomt (CMA) tht gives n ext interprettion for the minimum ontext elonging to eh stte in the utomton. The formlism is generl enough to simulte ontext models suh s PPM nd GDMC. Using the CMA formlism s our tool, we study the ehviour of the ove two ontext models. Key words: Text ompression, ontext modelling, PPM models, Mrkov models Numer of words: 8100 (inluding figures nd tles) Numer of lines: 1000 (inluding figures nd tles) Numer of figures: 14 Numer of tles: 3 Correspondene: Psi Fränti Deprtment of Computer Siene, University of Joensuu P.O. Box 111, FIN Joensuu, FINLAND Phone: Fx: Emil: frnti@s.joensuu.fi

2 1. Introdution Text ompression is typilly performed y sttistil or ditionry-sed methods [1, 2]. The input file is proessed y single pss strting with n initil model or ditionry. The method then dpts to the soure dt during the ompression. Only the dt tht is lredy ompressed (thus lso known y the deoder) n e used in the model. In this wy the deompression n e done synhronously. Ditionry-sed methods deompose the soure dt into sustrings whih re ompressed seprtely either y pointer to n erlier ourrene of the string in the soure dt (LZ77 vrints) [3], or y pointer to dynmilly onstruted ditionry (LZ78 vrints) [4]. The pointers re stored using fixed length oding, or n entropy oding method suh s Huffmn oding [5]. Better ompression performne is hieved y sttistil ompression t the ost of lower throughput nd higher memory onsumption. In symol-sed ontext models eh symol is predited y ontext, whih is omintion of few preeding symols. A different sttistil model is onstruted for eh ontext on the sis of the previously oded dt. Eh model onsists of tle of suessor symols nd their frequenies. The tul oding n e performed y rithmeti oding using the implementtion given y Witten et l. [6]. Vrile-order ontexts hve een widely used in PPM models (predition y prtil mth), originlly proposed in [7]. The highest order ontext is first pplied. If the symol is not predited y the ontext, n espe symol is oded nd nother lower level ontext is used. If no other ontext n e pplied, the symol is eventully oded using zero-order model. The size of the highest order ontext is kept reltively smll (three in [1]) to redue the memory requirements nd the lerning ost. Dt ompression n oneptully e divided into two seprte omponents: modelling nd oding. Modelling is the key question in the design of ompression lgorithm wheres the oding n e optimlly done using rithmeti oding. From this point of view, it is esy to see tht the sttistil models re more generl thn the ditionry-sed models sine they expliitly seprte the two omponents from eh other. In ft, ommonly used ditionrysed methods n e simulted y symol-wise sttistil models. For exmple, Lngdon hs given sttistil ounterprt for LZ78 in [8]. A third pproh to text ompression is the use of finite stte utomt. An utomton onsists of set of sttes nd trnsitions etween the sttes. Eh trnsition orresponds to single oding event: oding of symol or prt of it. All sttes must inlude outgoing trnsitions for every possile oding inident tht might pper. An implementtion known s Dynmi Mrkov Compression (DMC) uses inry lphet in the utomton [9]. Symols of lrger lphets must e hndled itwise. It hs een shown tht DMC tully implements finite-ontext model [10]. Generlized Dynmi Mrkov Compression (GDMC), proposed y Teuhol nd Rit [11], generlizes the ide of DMC to multisymol lphets. The sttes in the utomton orrespond to the ontexts nd the trnsitions to the seletion of the next stte. Espe strtegy is tken to redue the numer of outgoing trnsitions in eh stte. GDMC 2

3 implements ontext model with potentilly stronger pilities thn the PPM model ut its ext ehviour is not understood very well. We propose simple formlism, ontext model utomt (CMA), for modelling the ontexts in finite-stte utomton. It gives n ext interprettion for the minimum ontext elonging to eh stte in the utomton. We will show tht oth PPM nd GDMC follow the CMA formlism. We thus otin ontext interprettion for the sttes of GDMC utomton. CMA n lso e used s prtil tool for nlyzing the ehviour of the GDMC nd PPM models, nd to design new ones. A more theoretil pproh hs een tken y Bunton [12] for studying the ontexts in fulltle model where eh stte inludes omplete set of trnsitions. The ontexts re defined s set of strings desried y regulr expressions. This gives n ext ut not neessrily n esy interprettion for the ontexts of the sttes. Our pproh does not tke into ount the prtil ontexts inluded in the sttes ut it gives simpler interprettion for the ontexts, whih n e esily utilized in prtil pplitions. We fous here on models tht uses the espe strtegy, in whih only suset of the entire lphet my pper in eh stte. This is more fithful for the trditionl ontext models suh s PPM. Using CMA formlism s our tool, we study the ehviour of the GDMC nd PPM ontext models. CMA gives useful informtion out the struture of the utomton y giving ontext interprettion for the sttes. CMA, however, models only n existing utomton nd it does not define the wy the utomton is onstruted. The dynmi onstrution of CMA is therefore the key question when designing new utomton-sed ompression sheme. In this study, we limit the disussion to the existing models underlying in PPM nd GDMC. In PPM, the retion of new sttes depends on the ontext length, nd in GDMC, on the sttistis of the stte. PPM is more greedy in the model expnsion nd it dpts quiker to the soure dt. The model growth is ontrolled y limiting the ontext length y predefined mximum. GDMC, on the other hnd, hs no upper limit for the ontext length ut it uses frequeny sed heuristi to ontrol the model growth. Due to its lzier expnsion, the resulting ontexts re not neessrily unique. Although CMA defines minimum ontext length for eh stte, the utomton hs longer prtil ontexts. It is noted tht the CMA model itself sys nothing out the ompression performne. It is therefore eqully importnt to onsider lso the sttistil model integrted with the ontext model. For exmple, oth PPM nd GDMC use greedy stte seletion (the first preditive stte is lwys pplied). Better results ould e otined using informtion theoreti stte seletion, sed on the expeted ompression performne of the sttes. However, it is not ler how the expeted ompression performne should e mesured. In the rest of the pper, we will fous on ontext modelling only, nd settle for the existing PPM nd GDMC methods in the sttistil modelling. 3

4 2. Context model utomt The min omponents of CMA-sed ompression lgorithm re: Context model utomton (CMA) Sttistil model Entropy oder The ojet of CMA is to model ontexts only. For eh input symol the urrent stte of the utomton determines the ontext prediting the symol to e oded. The sttistil model then lultes the proility distriution using the symol frequenies inluded in the hosen ontext. The tul oding is performed y n entropy oder (usully rithmeti oding) on the sis of the determined proility distriution. After the oding, the sttistis re updted. CMA implements the ontext model s follows. Eh stte in the utomton orresponds to single ontext hving its own sttistis. The sttes inlude lso trnsitions for ll possile oding events tht might pper. A trnsition leds to the stte to e used next. The utomton is deterministi: its ehviour depends only on the input (soure dt nd prmeters), i.e., the utomton works lwys in the sme wy for the sme input. The sttes do not neessrily inlude trnsitions for ll possile symols nd the espe strtegy is pplied insted. Only trnsitions for lredy seen symols n pper in the sttes. The rest re oded y n espe-trnsition () leding to nother stte. If the symol does not exist in the following stte, n espe pth is followed until proper stte is rehed nd the symol n e oded. The espe strtegy elimintes lso the zero-frequeny prolem [1]. An exmple of CMA nd its pplition for oding smple soure string re illustrted in Figure 1. The min phses of CMA-sed ompression lgorithm re desried in Figure 2. The input dt is proessed symol y symol nd the utomton is onstruted dynmilly during the oding proess. The ompression strts with n initil utomton from its initil stte. The initil stte typilly inludes trnsitions for ll possile symols nd it rries no ontext. After symol hs een oded nd sttistis updted, the utomton n e updted pplying two si loning opertions: Crete new stte Crete new trnsition The modifition of the utomton thus orresponds to the dynmi modelling in PPM-sed ompression shemes. The initil utomton does not need to e very omplited. It is more importnt tht the utomton dpts to the input dt. The key questions of CMA re when nd how to lone. Solutions for these questions will e disussed in Setion 3. 4

5 NULL CODING EXAMPLE: Soure string: "" Sttes: NULL NULL Coded symols: es Figure 1: Exmple of ontext model utomton for lphet = {,, }. Compress(File): { C = C 0 /* initil stte */ REPEAT Symol = GetNextSymol(File); C = CompressSymol(C, Symol); UNTIL Symol=EOF; } CompressSymol(C, Symol): returns next stte. { IF EXISTS( C.Trnsition(Symol) ) THEN EnodeSymol(C, Symol); UpdteSttistis(C, Symol); nextc = C.Trnsition(Symol); Updte utomton ELSE EnodeSymol(C, ); UpdteSttistis(C, ); nextc = CompressSymol(C.trnsition(), Symol); Updte utomton RETURN nextc; } Figure 2: Pseudo ode for the CMA oding proess Forml desription of CMA The following symols nd definitions will e used to desrie CMA: Alphet of ll symols Stte in the utomton C Initil stte C 0 Order of the ontext in C C Any trnsition (S D) T Soure stte for trnsition S Destintion stte for trnsition D Espe trnsition Lel of trnsition Lel(T) 5

6 There re two kinds of trnsitions in the utomton: tul trnsitions nd espe trnsitions. Lel() is defined to e n empty string. Note tht the -symol still needs to e oded. Pth desries sequene of trnsitions orresponding to trversl in the utomton. Lel of pth denotes the tention of ll non- lels of the trnsitions long the pth. The lel of trversed pth orresponds to fixed numer of previously oded symols. Using the ove symols nd definitions, n utomton is defined to e ontext model utomton (CMA) if it meets the following riteri: Definition 1: (deterministiity property): There is only one outgoing trnsition from eh stte with the sme lel. Furthermore, only one possile outgoing -trnsition exists in eh stte. Definition 2: (termintion property): The utomton must terminte in finite numer of steps for ny soure string. Definition 3: (ontext property): The ontext in eh stte C is the longest ommon suffix of the lels of ll possile pths from C 0 to C. The order of the ontext in C is the length of tht suffix string, nd is denoted y C. Definition 4: (suffix property): Consider trnsition T in the form S D. The ontext of D is Suffix(S)+Lel(T). In other words, the next ontext fter n tul trnsition is lwys suffix of the ontext of the previous stte, ontented with the symol just oded. The termintion property sttes tht the utomton must e le to enode ny soure string nd terminte within finite time, i.e., it reognizes ll the lnguges over the lphet. This mens tht ll espe pths from ny stte must eventully led to stte tht is ple of enoding the input symol. In prtie, this is usully solved so tht ll espe pths end up in the initil stte of the utomton, whih is ple of oding ll lphet symols. The ontext property defines the ontext of stte C s the longest uniquely determined ontext. For exmple, onsider the stte in Figure 1. The lels of ll possile inoming pths of length 3 re nd. Their longest ommon suffix is, whih is therefore the ontext of tht stte due to the ontext property. The tul ontext, however, my still e longer thn C. In the previous exmple, there exists longer (ut only prtilly unique) ontext ( or )+ euse the inoming pth with the lel nnot pper. The following orollries re dedued diretly from the CMA definitions: Corollry 1: If there is n tul trnsition from S to D, then D S + 1. Corollry 2: If there is n -trnsition from S to D, then D S. Corollry 3: All tul trnsitions inoming to D hve the sme lel. Furthermore, if there is t lest one tul trnsition inoming to D then the lst non- trnsition long ll pths leding to D must hve the sme lel. 6

7 The orollry 1 sttes tht the order of the ontext nnot e inresed y more thn one due to single trnsition. This n e verified oth y the ontext nd suffix properties. The suffix property imposes lso tht the ontext nnot expnd due to n espe trnsition (orollry 2). The suffix property is rther strit. Consider n -trnsition etween two sttes with ontext of order 1 or greter. The suffix property implies tht these ontexts must hve the sme lst symol (orollry 3). A weker definition would hve stted tht D = Suffix(S+Lel(T)) ut we dopt here the stronger definition Dynmi onstrution of CMA CMA gives ontext interprettion for the sttes of utomton. However, CMA models only n existing utomton nd does not sy nything out how the utomton is onstruted. The dynmi onstrution of CMA must therefore e defined y n initil utomton nd y set of loning opertions. We define CMA-sed ompression method reursively y the following three riteri: Requirement 1: The initil utomton is CMA. Requirement 2: The new utomton due to loning opertion is CMA. Requirement 3: Cloning does not hnge the ontexts of existing sttes. These first two requirements re rther loose nd they give gret freedom in the design of the ontext model. The third requirement, however, defines tht loning nnot hnge the ontexts of n existing utomton. Thus, the ontext of stte is determined t the time of its retion nd loning opertions my generte only lol hnges in the utomton. In this wy, we n esily keep trk of the wy the utomton develops. 7

8 3. CMA implementtions of GDMC nd PPM models In the following setion we will show how the GDMC nd PPM models n e implemented under the CMA onept. We will first disuss the loning opertions introdued in GDCM nd show tht they diretly implement CMA. The PPM model is then simulted using the sme loning opertions, nd y mking only minor modifitions to the CompressSymol funtion GDMC model The initil model of GDMC is given in Figure 3. It is lerly deterministi utomton nd it n enode ll symols y finite numer of steps euse ll sttes hve n espe pth leding to the initil stte, whih ontins trnsitions for ll lphet symols. If the sttes re ssigned y the ontexts shown in Figure 3, the utomton will lso meet the ontext nd suffix properties. Thus, the first requirement for dynmi CMA is fulfilled Cloning opertions in GDMC: GDMC involves two loning opertions: (1) reting new sttes, nd (2) reting new trnsitions (shortuts). A new stte n e reted fter symol hs een oded in the THEN-rnh of the CompressSymol funtion (see Figure 2). The ojets of the opertion re the urrent stte nd the next stte in the utomton. Denote the lst oded symol y nd the orresponding trnsition y S D. The loning is performed y reting new ontext S nd redireting the originl trnsition from S D to S S. An -trnsition S D is lso reted, see Figure 4. The deision when to lone depends on the frequeny distriution in the ontext, nd is issued in Setion 3.3. Note tht the loning in GDMC is performed only if the soure stte diretly predits the next symol. It is thus never pplied fter n espe pth. GDMC involves lso the so-lled shortut opertion. Shortut is new trnsition tht ypsses omintion of n -trnsition followed y n tul trnsition, see Figure 5. Note tht it is possile to trverse long -pth until orretly lelled trnsition is found. In this se, only the lst -trnsition is involved in the shortut opertion, see Figure 6. Alterntive shortut strtegies hve een disussed in [11]. For exmple: (1) reting diret shortut S 1 D, or (2) reting ll possile shortuts S 1 D, S 2 D, nd so on. It is lso noted tht in GDMC, it is llowed to rete new stte diretly fter shortut hs een reted. NULL d Figure 3: The initil model of GDMC. d 8

9 D D' S D C D' C S S S S Figure 4: Cloning of GDMC. Figure 5: Shortut of GDMC. S1 S n-1 Sn Shortut D Figure 6: Long pth of -trnsitions CMA properties fter ontext loning: The new utomton fter the ontext loning is deterministi nd the inlusion of the espe trnsition does not rek the termintion property. The ontext of the soure stte S is not ffeted y the loning nd so we n limit the exmintion of the ontext nd suffix properties to S nd D only. D denotes here the stte D fter loning. The ontext property sttes tht the ontext of stte is the longest ommon suffix of the lels of ll pths from C 0 to the stte. As the only inoming trnsition to S is S S, the ontext of the new stte n e defined y S (tention of S with ). This fulfills oth the ontext nd suffix properties. The lel of the longest ommon pth inoming to D is either the originl ontext D, or S due to loning. Thus, due to the ontext property D = Min(D, S), where Min is defined s the shorter of the two strings. Considering tht the utomton ws CMA efore loning, we know tht: D = Suffix(S)+ (suffix property). It follows tht D = Min(Suffix(S)+, S) = Suffix(S)+ = D. In other words, the ontext of D does not hnge due to loning nd therefore D fulfills the ontext property. Furthermore, the loning hs no effet on ny suessor of D. The suffix property of D n lso e verified esily: D = Suffix(S)+Lel() = Suffix(S) whih is true euse D = Suffix(S)+. The modified utomton therefore meets the CMA properties lso fter the loning of new stte. In the previous exmintions we hve ssumed tht no trnsition is direted to the stte itself. It is possile tht tul trnsitions in GDMC re sometimes direted to the stte itself. Even so, ll the ove onlusions re still vlid (proof is omitted here). 9

10 CMA properties fter shortut: An -trnsition is trversed in GDMC only when there is not outgoing trnsition lelled y the urrent symol to e oded. Duplite lels re thus never reted nd the utomton remins deterministi. The termintion property is lso preserved sine new espe trnsitions re never reted y shortuts. The retion of shortut does not ffet the sttes S nd C of Figure 5 in ny wy, nd so only the stte D needs to e exmined. The lel of the longest ommon pth inoming to D is either D or Suffix(S)+ due to the new trnsition. Considering tht the utomton ws lso CMA efore the modifition, we know tht D = Suffix(C)+ nd C = Suffix(S). It thus follows tht D = Min(D, Suffix(S)+) = Min(Suffix(C)+, Suffix(S)+) (1) The ft tht C = Suffix(S) implies Suffix(C)+ Suffix(S)+. The ove eqution therefore simplifies to: D = Suffix(C)+ = D. (2) This gives proof to the ontext nd suffix properties for D. Moreover, sine D = D, the shortut does not ffet the sueeding sttes of D in ny wy. Thus, the modified utomt due to the retion of shortut meet the CMA properties. Note tht the lterntive shortut strtegies disussed in [11] would meet ll the CMA riteri, too PPM model In PPM, the oding of symol strts lwys t the highest level N mx. If the symol nnot e oded y the urrent ontext, lower level ontexts re rehed using espe trnsitions. After proper ontext hs een found nd the symol hs een oded, the proess should then ontinue t the highest level gin. The orollry 1 of the CMA properties, however, sttes tht the order of ontext nnot inrese y more thn one due to single trnsition. For exmple, suppose tht the lst three oded symols were (ssuming N mx =3), ut the next symol d is predited only y the ontext. After d hs een oded, the next ontext ording to the PPM model should e d. But there nnot e diret trnsition d in CMA. In order to perform the PPM simultion, the mening of trnsition is more relxed thn in GDMC. Originlly trnsition orresponds not only to the oding of symol ut it lso indites the stte to e used for the next symol. The loning nd retion of trnsitions re defined in the sme wy s in GDMC ut the oding is not neessrily ontinued from the stte pointed out y the trnsition. Insted, the next stte is the new stte due to loning. The CompressSymolPPM funtion simulting the PPM model is defined in Figure 7. The funtion tkes the symol to e oded nd n N-level ontext s input, nd outputsthe next ontext to e used, whih is loted t level N+1 (or t level N mx in se of N=N mx ). If trnsition for the input symol exists, the symol is ompressed nd the trnsition is followed 10

11 for otining the ontext for the next symol. The resulting stte is then returned s the output of the funtion. This is trivil se where no loning is needed. In the se when the orret trnsition does not exist, the routine follows n espe pth reursively through lower levels nd when proper stte is found nd the symol oded, the funtion egins ktrking mong the trversed espe pth. New sttes re loned during the ktrking nd the result of the reursion is stte t the sme level s the urrent one. If the urrent stte is t level N mx, shortut is reted for pointing to the newly otined stte. Otherwise (whih hppens t the lower levels during the reursion) new stte is loned t the level N+1. This proess is illustrted in Figure 8 where d is the input symol nd the ontext. Sine trnsition d does not exist the funtion is reursively lled nd the stte d is eventully otined s result. The simultion is now lmost omplete ut we still need to define the initil utomton for PPM. In priniple, the utomton of Figure 3 ould e pplied ut the originl lgorithm in [7] uses slightly different initilistion. This is implemented y the utomton shown in Figure 9. The first stte ( stti ) hs trnsitions for ll possile symols in the lphet nd the sttistil model ssigned to the stte is ssumed to e stti. This is sometimes referred s -1 level ontext in the literture. The seond stte ( dptive ) hs only n espe trnsition to the stti stte. In this wy the model my etter dpt to the input dt euse it hs no trnsitions for unused symols s urden. In our implementtion, we let lso the stti stte to dpt to the input dt. The initil model of Figure 9 is theoretilly not len euse the dptive stte ontrdits the suffix property of CMA (inoming trnsitions should not hve different lels). This hs no prtil onsequenes euse the exeption remins lol nd the rest of the dynmilly onstruted utomton will still meet ll the CMA riteri. If we hd dopted the weker version of the suffix property (stting tht D = Suffix(S+Lel(T)) the initil model would hve een proper CMA. CompressSymolPPM(C, Symol): returns next stte. { IF EXISTS( C.Trnsition(Symol) ) THEN EnodeSymol(C, Symol); UpdteSttistis(C, Symol); nextc = C.Trnsition(Symol); ELSE EnodeSymol(C, ); nextc = CompressSymolPPM(C.trnsition(), Symol); CreteTrnsition(C, nextc, Symol); IF C < N mx THEN nextc = CloneContext(C, nextc, Symol); RETURN nextc; } Figure 7: Compression funtion for CMA in order to simulte PPM. The opertions modifying the utomton re enhned. <Figure 8 is not inluded in this file due to tehnil prolems. Only pper opy exists!!!> 11

12 Figure 8: Exmple of the use of the CompressSymolPPM funtion. NULL stti NULL dptive Figure 9: Initil model for PPM Properties of the GDMC nd PPM models In the following, we will study the ehviour of the GDMC nd PPM ontext models using the CMA formlism s our tool. The ontexts in CMA n e esily determined during the oding proess y ssigning the sttes with their level numer. Both in GDMC nd PPM, the level numer C of new stte C is lredy known t the time it is loned. Due to the suffix property, it is lwys one higher thn tht of the soure stte involved in the loning, see Figure 4. The tul ontext is then the C lst oded symols Cloning sttes: The key questions for reting new sttes re: when to lone nd how to lone. PPM lones lwys fter symol hs een oded. Greedy ontext expnsion is usully dvntgeous, see [1, p. 149] nd [2], ut the growth of the utomton must e limited in some wy to keep the memory onsumption resonle. The strightforwrd solution of PPM limits the ontext lengths to predefined mximum N mx. GDMC, on the other hnd, hs no upper limit for the ontext length ut it uses the following heuristi to ontrol the growth. Consider tht the trnsition S D ws trversed. A new stte is reted if the differene i f D DD f S D i (3) 12

13 exeeds predefined loning onstnt (CC). Here DD i refers to suessor stte of D, nd f to trnsition frequeny. This gives fesile solution for limiting the growth of the utomton. Best ompression results were reported with very smll CC-vlues [11]. Note tht unounded ontext lengths hve lso een proposed for PPM in [13]. This method limits the model growth y loning only when ontext is used seond time. Another signifint differene etween the GDMC nd PPM models lies in the wy the utomton is trversed. When oding long sequene of espe trnsitions, the PPM model llows ktrking mong the trversed espe pth. New sttes re lso loned during the ktrking. In this wy, PPM retes ll possile ontexts from the urrent level up to the level N mx wheres GDMC expnds the utomton only y one ontext t time. PPM is therefore more greedy in the ontext loning. After new stte hs een loned, there is question whether the oding proess should ontinue from the urrent stte D (like in GDMC), or from the new stte tht ws reted in the loning (like in PPM). At first sight there seems to e no point to ontinue from the newly reted stte euse it hs not yet lerned ny trnsitions. The ontext nnot therefore predit the next symol nd n espe trnsition must e followed k to the stte D. The trversed espe trnsition, on the other hnd, does not ost ny its sine its proility is 1. Besides, if the oding ontinues from the new stte it n diretly lern the next symol to e oded, nd in this wy the stte dpts fster to the soure dt. Figure 10 demonstrtes the ehviour of the GDMC nd PPM models when smple string is oded. PPM lerns ll existing ontexts up to the upper limit, wheres GDMC expndeds only up to the seond level. None of the seond level ontexts hs lerned ny trnsitions yet. The fster model growth of PPM n e summrised in the following three properties: (1) greedy loning riterion, (2) multiple loning opertions in single oding step, (3) oding is ontinued diretly from the new stte fter loning. NULL NULL NULL Figure 10: GDMC (left) nd PPM (right) utomt fter the oding of string. In GDMC, loning ws performed when ontext ws used seond time. In PPM, N mx ws set to 3. 13

14 Trnsitions: Consider trnsition S D (ssuming S <N mx ) nd denote D - S y the inrese of ontext length due to the trnsition. In PPM, ll suh trnsitions inrese the ontext extly y one. This is onsequene of the greedy loning riterion in whih the retion of new trnsition is diretly followed y loning opertion. Only exeptions re the sttes loting t the level N mx. Espe trnsitions derese the ontext lwys y one. In GDMC, trnsitions n lso e direted to the sme, or to lower level in the utomton. For exmple, when oding the symol in the stte of Figure 10, trnsition will e reted. If the stte is not immeditely loned the trnsition will remin. This is possile due to the lzy loning riterion. Trnsitions n lso e direted to themselves. It my hppen when the sme symol is repeted in the soure dt. Espe trnsitions n e direted in GDMC to ny lower level. For exmple, if the stte ws loned fter trversing the trnsition (the new stte eing ), the resulting espe trnsition ( ) would derese the ontext y two. These kind of ritrry jumps etween different levels re possile euse the loning riterion does not tke into ount the ontexts of the sttes ut it depends on the frequenies only Prtil ontexts: The ontext property defines the minimum order of the ontext nd implies tht there might e higher order prtil ontext in the stte. Consider stte C with n N-order ontext. Denote ll possile inoming (N+1)-length pths to C y xc, nd the set of ll possile symols x y X. The stte C is defined to inlude prtil ontext of order N+1 if the set X is true suset of the full symol set (X ). In other words, there exists t lest one symol in the lphet tht nnot preede the ontext. In the se of full tle strtegy (where ll sttes inlude trnsitions for ll lphet symols), eh longer ontext implies prtil ontext in ll of its suffix ontexts [12]. For exmple, if the stte exists in the utomton, the prtil ontext in is ( \)+ euse the stte will never e used for the input string. Similrly, ll other longer ontexts redue ertin string omintions from the ontext of. In the se of espe strtegy, the question of prtil ontexts eomes more omplited euse of the espe trnsitions. For exmple, the stte in Figure 11 (left) does not inlude prtil ontext ( or )+ despite the stte exists in the utomton. The stte is indeed not rehed with the input. However, the ontext of stte is defined merely on the sis of the history. The exlusion of ertin string omintions depending on the future symols is known s the exlusion priniple. Nevertheless, it is not onsidered s prtil ontext euse the stte n still e rehed y ll possile input strings x (x ). Vrile order ontext models using espe strtegy re not so strit out wht is the ext ontext of stte. When longer ontext () dpts to more nd more different trnsitions, the effet of the string is redued in the shorter ontext. It is therefore grdully trnsferring towrds the ontext ( or )+ ut it my never reh tht sttus. 14

15 Despite the ove disussion, the CMA utomt n still inlude prtil ontexts in the strit sense. Figure 11 illustrtes how this kind of ontext is reted y the loning opertion of GDMC. The new stte inludes prtil ontext ( or )+ euse it nnot e rehed when oding the string (due to the stte nd the trnsition ). This origintes from the lzy loning riterion of GDMC, whih llows sttes to e loned fter they hve lredy een ypssed y shortuts. The diret onsequene is tht the longest possile ontext is not lwys utilized. In PPM model, this kind of prtil ontexts nnot pper euse of the greedy loning riterion. NULL NULL prtil ontext Figure 11: Exmple of prtil ontext in GDMC Duplite ontexts: Consider the utomton of Figure 11 (left). Assume tht the trnsition NULL ws trversed nd the stte is then loned. This results in two sttes with identil ontexts, see Figure 12. The originl stte will e rehed using the existing trnsitions nd, the new one using the redireted trnsition NULL. Due to the suffix property, the loning opertion retes new ontext whose order is one higher thn tht of the soure trnsition. Duplite ontexts re therefore generted lwys when the trversed trnsition lredy inreses the ontext y one: D = S +1. The min effet of the duplites is tht the new stte will inlude prtil ontext. For exmple, the new ontext in Figure 12 inludes prtil ontext ( or )+. Duplites my lso e generted for D < S +1 if the soure stte S is lredy duplite ontext. In this se, there exist two rnhes in the utomton with similr ontexts. It is unler wht is the dvntge/disdvntge of the duplite ontexts. Presumly they slow down the lerning proess euse the sme set of trnsitions must first e lerned in the originl ontext, nd only then they n e lerned in the new (duplite) ontext. If so desired, the existene of duplites ould e esily voided y llowing loning only when D < S +1. Note tht duplite ontexts nnot pper in PPM model due to the greedy loning riterion. 15

16 NULL Figure 12: Exmple of duplite ontexts Proility estimtion Both PPM nd GDMC use frequeny-sed proility model. The proility for trnsition is lulted y dividing its frequeny ount y the sum of ll frequeny ounts of the outgoing trnsitions. The only exeption is the espe trnsition. In GDMC, espe trnsitions re treted s ny other trnsitions nd they thus hs their own frequeny ounters. In PPM the proility for n espe trnsition is lulted y predefined funtion known s Espe method C [14]: p q q (4) Here q is the numer of outgoing trnsitions in the stte, nd is the sum of their frequeny ounts, i.e., the totl numer of times the stte hs een visited. It is noted tht q equls to the numer of times the espe trnsition hs een used. Both methods thus hve the sme intuition ut the tul proility vlues my differ euse of different initilistion nd updte strtegies. In PPM, the sttistis for new ontext strt from srth nd the frequeny of new trnsition is lwys set to 1. In GDMC, the frequeny of new trnsition (referring to Figure 5) is estimted y: f S D f S C f C D Effsum C S (5) where Effsum(C S) is the sum of ll trnsition ounts in C exluding the ones existing lso in S. This orresponds to the exlusion priniple (see elow). Due to this initilistion the frequenies re stored s rel numers insted of integer domin. See [11] for detils of this proility modelling method. Both PPM nd GDMC pply the so-lled exlusion priniple when lulting the effetive proility for symol. If espe trnsition ws just trversed, it is known tht the urrent symol nnot e ny of the symols predited y the previous stte. All these symols n therefore e exluded from the proility lultions. The exlusion priniple lwys improves the ompression euse otherwise proportion of the proility spe would e wsted for impossile oding events. 16

17 After the symol hs een oded, its frequeny ount is inremented y one. The updte is restrited only to the preditive stte; the ounters in the lower order sttes re not ltered. This priniple, known s updte exlusion, is ommonly used in PPM nd lso in GDMC. An lterntive strtegy would e to follow the espe pth to the initil stte nd updte the frequenies in ll sttes long the pth. For exmple, if the symol ws predited y the ontext, the updte would then onsider lso the sttes nd. 4. Empiril results The files of Clgry orpus [1] were ompressed using the GDMC nd PPM models with the prmeter setup of Tle 1. We wnt to find out wht kind of ontexts (nd how long) the models rete, how fst they dpt to the soure, nd wht kind of trnsitions there exist in the utomt. Additionl tests were performed y vrying the min prmeters of the models. The im is to find out how the prmeter setup ffets the performne of the ompression, nd how well the ontext model n e ontrolled y the prmeters. The throughput of the CMAsed implementtion does not signifintly differ from the previous implementtions of PPM nd GDMC. In the literture, the mximum ontext length of PPM is often set to 3 ut we use N mx =4 euse it gives lower it rte for ll files in Corpus, exept the file geo. The verge it rtes for vrious N mx -vlues (1 to 7) re (3.62, 2.85, 2.46, 2.34, 2.32, 2.33, 2.34). In ddition, we tested lso GDMC with N mx =4 (referred s GDMC-4). The results re summrised in Tle 2. The verge it rtes for GDMC nd GDMC-4 re 2.52 nd The GDMC it rtes re 7 % higher thn tht of PPM on verge. Slightly etter results were otined only for the three longest files (ook1, ook2 nd pi). The enefit of the unlimited ontext length remins mrginl: 2 % on verge nd the mximum improvement is 6 % for the file pi. PPM lerns ll ontexts up to the mximum length tht there re to e lerned. GDMC is more piky in loning nd lerns only out % of the sme ontexts. It lso lerns lrge numer of longer ontexts. The totl numer of ontexts in GDMC is typilly out three times s muh s in PPM ut only out 60 % of the ontexts re unique; the rest re duplites. Tle 1: Summry of the ontext nd proility model prmeters. PPM GDMC Context Context length limit: N mx = 4 No upper limit model Cloning riterion: Clone lwys Depends on sttistis Multiple loning: Yes No Sttistil Proility estimtion: Frequeny sed Frequeny sed model Initil sttistis: Strting from srth Derived from suessor Espe proility: PPM model C Own frequeny ounter for 17

18 Tle 2: Performne omprison (it rtes) of the vrious methods. File Size PPM-4 GDMC GDMC-4 i ook ook geo news oj oj pper pper pi prog progl progp trns Averge In PPM, the input symols were predited y ontext length of 3.27 on verge. In GDMC nd GDMC-4, the orresponding numers were 4.38 nd 2.94 (exluding oj1 nd pi). In pi, the verge length of the predited ontext ws s lrge s 1823 due to the high repetition of zero ytes (lrge homogenous white res in the inry imge). We lso oserved tht GDMC uses lmost lwys the longest ville ontext in the utomton. This indites tht, even if there re prtil ontexts in the utomton (esides the duplites), they re rrely used. It ws shown in Setion 3.3 tht duplites in GDMC n e voided y modifying the loning riterion ordingly. Our experiments show tht this modifition does not hve ny signifint effet on the ompression performne. Smll improvement (0-3 %) ws hieved in se of 9 files out of 14, ut the it rte inresed in the se of geo (4 %) nd pi (16 %). This is most likely explined y the nture of their dt, whih is not tken into ount in the ompression. The file pi, for exmple, is inry imge ut the ontext models proess the dt one yte t time. GDMC utilizes the itwise dependenies ross the yte oundries somewht etter sine it is ple of modelling prtil ontexts (involved lso in the duplite ontexts). Figure 13 illustrtes the soure of the ode its. In PPM, 55 % of the ode its originte from the highest level (N=4). The ode its in GDMC originte from wider vriety of levels, ut overll, the highest levels (N 4) ontriute y lesser mount (38 %) thn the highest level in PPM. Even if there re gret numer of longer ontexts in GDMC, they re only suffix extensions of those ppering lso t the levels N 4. GDMC thus relies more on the lower level ontexts thn PPM. Figure 13 demonstrtes lso the growth of the models. The lower level ontexts re used more often in the erly stge of ompression. When the ompression proeeds nd the models dpt to the dt, the higher levels eome more ommonly used. The overll it rte dereses over the ourse of time. PPM expnds the ontexts fster thn GDMC, whih is shown in etter ompression performne in the erly stges, see Figure

19 Code its (kb) PPM % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 Compressed symols % 3 4 Code its (kb) GDMC 1 10 % 20 % 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 Compressed symols % >4 Figure 13: Soure level of ode its s funtion time (ll files). Error! Not vlid link. Figure 14: Bit rtes of GDMC reltive to PPM s funtion of time (verge vlues). In GDMC, 32 % of the ode its origintes from espe symols wheres the orresponding numer in PPM is 14 %. By ompring the trnsition sttistis, it ws shown tht GDMC lerns fewer trnsitions thn PPM. The numer of trnsitions (exluding espe trnsitions) per ontext re 2.00 (PPM) nd 1.04 (GDMC) on verge. This is prtly explined y the longer ontexts of GDMC. They n predit ertin symols more urtely thn the lower level ontexts ut the totl numer of trnsitions (predited symols) remins smller. Espe trnsitions re therefore used more often. The destintion level of the trnsitions, however, did not signifintly differ etween the models. Only very few trnsitions led to the lower levels in GDMC, see Tle 3. Tle 3: Destintion level of trnsitions in respet to the soure level. +1 sme -1 lower PPM 34 % 66 % 0.0 % 0.0 % GDMC 95 % 4 % 0.7 % 0.0 % GDMC-4 42 % 56 % 2.4 % 0.0 % PPM nd GDMC hve their differenes lso in the proility model. For exmple, they use different initil utomt (see Figures 3 nd 9). The initil utomton of PPM gives 1 % lower it rte on verge. A more signifint differene is the initilistion of the frequeny ounter when new trnsition is reted. In PPM, the frequenies strt from srth wheres GDMC uses more sophistited frequeny estimtion tehnique (see Setion 3.4). Aording to our experiments, the results of GDMC re out 10 % worse if the PPM-like proility model is pplied. This implies tht the model might e utilized lso in PPM ut unfortuntely this is not the se. The GDMC proility model hs een designed nd optimized merely within the GDMC ontext model nd we were not le to dopt the method in PPM. 19

20 5. Conlusions A simple formlism ws introdued to desrie ontext model utomt (CMA). It provides strong tool to simulte existing ontext modelling methods suh s PPM nd GDMC. The ehviour of the ove two methods were studied oth nlytilly nd experimentlly from the CMA point of view. The min dvntges of the PPM nd GDMC models n e summrised s follows: PPM: Fst dpttion to soure Better initil model Aury of sttistis GDMC: No upper limit for ontext size Prtil ontexts Sophistited proility model The min dvntge of PPM over GDMC is its fster dpttion to the soure dt. This origintes from the following properties: (1) greedy loning riterion, (2) multiple loning opertions in single oding step, (3) oding is ontinued diretly from the new stte fter loning. PPM n lern up to N mx new ontexts per oded symol wheres GDMC n lern only one. The PPM model expnds quiker in the erly prt of ompression ut the growth eomes smoother over the ourse of time. The GDMC model, on the other hnd, keeps on growing with the sme rte throughout the ompression euse there is no upper limit for the ontext length. The initil model of PPM is slightly etter (1 %) thn tht of GDMC euse the zero-order ontext hs no trnsitions for unused symols s urden. Furthermore, the sttistis of the PPM ontexts re more urte thn tht of GDMC. The highest level ontexts in PPM re reted immeditely when they first pper in the soure dt. The sttistis thus onentrte on the relevnt ontexts only. In GDMC it tkes longer to uild the sme ontexts. There re lso gret numer of higher level nd duplite ontexts in the GDMC utomton. The lerning is therefore ineffiiently distriuted over wider numer of ontexts resulting in less urte sttistis. The unounded ontext length of GDMC is oth n dvntge nd disdvntge. It llows more flexiility in the ontext length ut it lso results in unneessrily long ontexts euse the ontext expnsion is not properly ontrolled. This defets the lerning of the sttistis nd etter loning riterion should therefore e designed. The other enefits of GDMC re lso questionle. The potentil of prtil/duplite ontexts is their ility to model more effiiently ontexts eyond the yte oundry (suh s the inry dt of the file pi). Unfortuntely extr ontexts rek up the sttistis into too mny ontexts resulting in less urte predition in the se of norml dt files. There is lso no diret ontrol of the onditions under whih the duplites re reted nd they re reted more or less in rndom fshion. Moreover, we were not le to trnsfer the sophistited estimtion of trnsition frequenies into the PPM model. Its min enefit is most likely to ompenste the slower lerning effiieny of GDMC, i.e., the method tries to predit the frequeny tht the PPM model would hve lredy lerned due to its fster dpttion. To sum up, CMA provides formlism to desrie ontext model utomt. It gives useful informtion out the struture of the utomton. Potentil weknesses, suh s duplites nd 20

21 ritrrily long ontexts, n therefore e esily ontrolled if so desired. CMA, on the other hnd, models only the struture of the utomton without giving ny diret informtion out the ompression performne. It is therefore importnt to onsider lso the sttistil model when designing CMA-sed ompression method. In prtiulr, informtion theoreti stte seletion nd the mixing of the proility estimtes from different sttes hve een shown to improve ompression performne of PPM model y 5-12 % [15]. Aknowledgements The work of Psi Fränti ws supported y grnt from the Ademy of Finlnd, nd the work of Timo Htkk y grnt from the Finnish Culturl Foundtion. 21

22 Referenes 1. Bell, T.C., Clery, J.G. nd Witten, I.H. (1990) Text Compression, Prentie Hll, Englewood Cliffs, NJ. 2. Willims, R.N. (1991) Adptive Dt Compression, Kluwer Ademi Pulishers, Norwell, Msshusetts. 3. Ziv, J. nd Lempel, A. (1977) A universl lgorithm for sequentil dt ompression. IEEE Trnstions on Informtion Theory, 23, Ziv, J. nd Lempel, A. (1978) Compression of individul sequenes vi vrile-rte oding. IEEE Trnstions on Informtion Theory, 24, Huffmn, D. (1952) A method for the reonstrution of minimum redundny odes. Pro. of the IRE, 40, Witten, I.H., Nel, R.M. n Clery, J.G. (1987) Arithmeti oding for dt ompression. Communitions of the ACM, 30, Clery, J.G. nd Witten, I.H. (1984) Dt ompression using dptive oding nd prtil string mthing, IEEE Trnstions on Communitions, 32, Lngdon, G.G. (1983) A note on the Ziv-Lempel model for ompressing individul sequenes. IEEE Trnstions on Informtion Theory, 29, Cormk, G.V. nd Horspool, R.N.S. (1987) Dt ompression using dynmi mrkov modelling, The Computer Journl, 30, Bell, T.C. nd Mofft, A. (1989) A note on the DMC dt ompression sheme, The Computer Journl, 32, Teuhol, J. nd Rit, T. (1993) Applition of finite-stte model to text ompression, The Computer Journl, 36, Bunton, S. (1995) The struture of DMC, Proeedings Dt Compression Conferene, Snowird, Uth, Clery, J.G. nd Tehn, W.J. (1997) Unounded length ontexts for PPM, The Computer Journl, 40, Mofft, A. (1990) Implementing the PPM dt ompression sheme, IEEE Trnstions on Communitions, 38, Bunton, S. (1997) Semntilly motivted improvements for PPM vrints, The Computer Journl, 40,