On asymptotically optimal methods of prediction and adaptive coding for Markov sources

Transcription

1 On asymptotically optimal methods of pediction and adaptive coding fo Makov souces Bois Ya. Ryabko Infomation Technology Univesity, Copenhagen, Denmak Flemming Topsøe Depatment of Mathematics, Univesity of Copenhagen, Denmak Summay. The poblem of pedicting a sequence x, x 2, geneated by a discete souce with unknown statistics is consideed. Each lette x t+ is pedicted using infomation on the wod x x 2 x t only. In fact, this poblem is a classical poblem which has eceived much attention. Its histoy can be taced back to Laplace. To estimate the efficiency of a method of pediction, thee quantities ae consideed: the pecision as given by the Kullback-Leible divegence, the memoy size of the pogam needed to implement the method on a compute and the time equied, measued by the numbe of binay opeations needed at each time instant. A method is pesented fo which the memoy size and the aveage time is close to the minimum. The esults can eadily be tanslated to esults about adaptive coding. Keywods. Pediction, adaptive coding, univesal coding, ε-capacity, ε-net, imaginay sliding window. Intoduction The poblem of pediction and the closely elated poblem of adaptive coding of time seies is well known in infomation theoy, pobability theoy and statistics. The poblem can be taced back to Laplace cf. Felle [8] whee the poblem is efeed to as the poblem of succession. Pesently, the poblem of pediction is investigated by many eseaches because of its pactical applications and impotance fo pobability theoy, statistics, patten ecognition, cybenetics and othe theoetical sciences. An extensive eview and list of efeences can be found in Algoet, [2]. 0 Reseach of both authos suppoted in pat by the Calsbeg Foundation. We shall investigate the elation between complexity and pecision of pediction methods. This poblem is inteesting fom a pactical point of view. To ealize this, imagine you want to pedict x t+ knowing the sequence x x 2 x t. Then you have to deal with a gowing amount of data as t inceases. This esults in an incease of the time of calculation and of the amount of memoy space equied. Though pactical methods have been devised and studied empiically, thee is a lack of studies whee the elation between complexity and pecision of pediction methods is investigated theoetically. Such theoetical esults will be useful, e.g. when judging the efficiency of concete pediction methods in expeiments based on eal souces. Hee, the appaently simplest classes of poblems ae consideed as a fist step towads an undestanding of the connection between complexity and pecision. Namely, we conside a souce with unknown statistics which geneates sequences x x 2 of lettes fom a finite alphabet A = {a,, a n } and the models we conside ae eithe Benoulli o Makov models of fixed connectivity. The undelying tue distibution, which is unknown except fo the estiction given by the model assumption, is indicated by the lette p. We imagine that we have a compute at ou disposal fo solving the pediction poblem. Now, let us have a specific method of pediction in mind. As input we conside any finite sting x x 2 x t of lettes fom A and as output we equie that at each time instant t we eceive non-negative numbes p a x x t,, p a n x x t which ae estimates of the unknown conditional pobabilities pa x x t,, pa n x x t, i.e. of the pobabilities px t+ = a i x x t ; i =,, n. The set p a i x x t ; i n is the pediction. The pecision of a pediction method is measued by the divegence between p and p, and the complexity of a method is chaacteized by two numbes: The time of calculation at each time instant

2 in bit opeations and the memoy size in bits of the compute which is necessay in ode to execute the pogam defining the method. This appoach is natual fom a pactical point of view and well known in Compute Science, see, fo example, []. It is in confomity with methods used fo pediction and fo simila poblems of leaning and statistics which ae based on the theoy of finite state machines cf. [7] and [3]. The poblem which Laplace consideed cf. also the ecent contibution by Kichevskii, [3] was to estimate the pobability that the sun will ise tomoow, given that it has isen evey day since the ceation. Using ou teminology, we can say that Laplace estimated p and p, whee {, } is the alphabet sun ises, sun does not ise and the length of is the numbe of days since the ceation. Instead of viewing the pediction p a x x t as pobabilities, one may view p a x x t as a stake on the lette a A. This moe game theoetical view also goes back to Laplace. Futhe consideations of a game theoetical natue wee suggested by many authos Kelly [0], Topsøe [20], Ryabko [6], Fede,Mehav and Gutman [7] and Rissanen [5]. The poblem of adaptive and univesal coding is closely elated to the pediction poblem and was investigated in Kichevskii [2] and Ryabko [7]. Fom a mathematical point of view the poblems ae identical and can, theefoe, be investigated togethe cf. also futhe explanatoy emaks in the next section. We shall suggest two pediction methods fo Makov souces, which ae, espectively, asymptotically optimal o nea optimal in aveage and asymptotically optimal with pobability one. The one method is deteministic, wheeas the othe uses andomization. Regading andomization cf. also [4], it is impotant to ealize that this device is consideed as an extenal device, an oacle, which can be consulted fom time to time. Theefoe, the use of andomization does not in itself pose exta demands on the memoy of the main compute. In ode to implement the andomizing mechanism, a pseudoandom numbe geneato will always be used and this demands sepaate devices memoy etc.. As stated, these demands ae not consideed to intefee with memoy management etc. elated to the main compute. Cetain theoetical consideations demonstate the possibility to geneate efficient andomizing agents with low complexity, and this justifies the view taken. We acknowledge that in pactice the exteio device and the methods in question ae built togethe using the same compute. 2 Definitions Conside an alphabet A = {a,, a n } with n 2 lettes and denote by A t the set of wods x x t of length t fom A. Let p be a souce which geneates lettes fom A. Fomally, p is a pobability distibution on the set of wods of infinite length o, moe simply, p = p t t is a consistent set of pobabilities ove the sets A t ; t. By M 0 A we denote the set of Benoulli souces ove A, and by M k A the set of Makov souces ove A of connectivity memoy k ; k. We use M to denote the model unde consideation. Fomally, M could be any set of souces but fo this pape we only conside the cases M = M 0 A and M = M k A with k a fixed natual numbe. In fact, we shall mainly focus on the case M = M 0 A as esults fo the geneal Makovian case can be deduced fom esults fo the Benoulli case. Denote by D the Kullback-Leible divegence and conside the souce p and a method γ of pediction. Fo a deteministic method of pediction, the pecision is chaacteized by the divegence γ,p x x t = D p x x t p γ x x t = a A pa x x t log pa x x t p γa x x t. Hee, and in the sequel, log denotes natual logaithms we also need logaithms to the base 2 which ae denoted by log 2. As usual, high pecision means divegence close to zeo. We will also conside methods of pediction which allow andomization. Then, we emphasize that p γa x... x t ; a A is a andom distibution even fo fixed x x t. Thus, fo these methods we define the pecision as follows γ,p x... x t = E γ Dp x... x t p γ x... x t = a A pa x... x t E γ log pa x...x t p γ a x...x t, whee E γ denotes mean value. Note that γ,p may also be consideed as the edundancy when the pediction is used fo coding. Let us comment on the elation to coding in moe detail. We use p to stand fo the tue conditional distibution p x x t and p to stand fo the coesponding pediction possibly chosen afte invoking andomization. An obseve can constuct a pefix-fee code with codelength 2

3 κ a x x t log 2 p a x x n fo any lette a A since Shannon s oiginal eseach, it has been well known, cf. e.g. Gallage [9] o Cove and Thomas [4], that, using block codes with lage block length o moe moden methods of aithmetic coding, the appoximation may be as accuate as you like. An ideal obseve would base coding on the tue distibution p and not on the pediction p. The diffeence in pefomance measued by aveage code length is given by pa x x t log 2 p a x x t a A a A pa x x t log 2 pa x x t = a A pa x x t log 2 pa x x t p a x x t. Thus this excess, the edundancy, is apat fom the unit in bits athe than in natual units exactly the pecision defined above. We shall in fact mostly efe to edundancy athe than divegence o pecision in what follows. In the final section with concluding emaks we shall etun to coding aspects and include a discussion of adaptive coding. Fo fixed t, γ,p is a andom vaiable. This has nothing to do with the use of andomization and only eflects that x, x 2,, x t ae andom vaiables. We define the aveage divegence at time t by D t p γ = E p t γ,p = x x t A t px x t γ,p x x t 2 Related to this quantity we define the maximum aveage divegence at time t by D t M γ = sup D t p γ 3 p M and the limiting maximum aveage divegence by D M γ = lim sup D t M γ. 4 t We also efe to D M γ simply as limiting edundancy. The dependence on the method and on the model can be emphasized by speaking of limiting edundancy of the method γ unde the model M. It is impotant to develop methods which, in pinciple, can be ealized on any compute. Theefoe, a method γ will also depend on a paamete elating to the compute available. This paamete could be the memoy size. Fo the methods we shall discuss it is moe convenient to use a paamete w expessing the size of a window x t w+ x t which is used fo the pediction. The asymptotic popeties which we shall study investigates the pefomance of a method as given by limiting edundancy when the method is ealized on moe and moe poweful computes. Equivalently, and this will be ou pefeed view, one may study the necessay equiements on memoy space and time in ode to achieve a lowe and lowe limiting edundancy. 3 A method which is asymptotically optimal on the aveage fo Benoulli souces We shall descibe a method α 0 which is based on esults fom univesal coding theoy, cf.[2]. The method is based on the fequencies of the lettes in A in a window x x w of size w. By ν a x x w we denote the fequency of a in x x w, i.e. the numbe of i w with x i = a. Fo the method α 0, x x w is used to obtain the fequencies ν a x x w and these will then be the basis fo pediction of x w+, x w+2, accoding to the fomula p α 0 a x x t = ν ax x w + ; w + n t w. 5 As we ae only inteested in limiting behaviou, it is not impotant how x,, x w ae pedicted. In ode to stoe the numbes ν a x x w ; a A we could equie S w = n log 2 w + o just n log 2 w+ bits. In pinciple, we could have been less demanding and only equied a memoy w + n of log 2 bits. Howeve, this might n incease computing time since the actual fequencies ν a x,, x w may then not be so easy to access. We also point out that we conside n as fixed wheeas lage and lage w will be involved. Theefoe, the quantities suggested basically only diffe by an additive constant. Futhemoe, we ae not inteested in the fine details egading actual implementations on a compute. Taking the above consideations into account, we simplify the discussion by taking Sw = n log 2 w 6 as the equied memoy size. 3

4 When the compute pesents the esults 5 of the pediction at a given time instant, we assume that the pobabilities involved ae epesented i.e. pinted as factions. The time needed fo this is c n log 2 w + counted in bit opeations. Hee c is a constant which is chaacteistic of the actual compute used. We again simplify and take T w = Olog w 7 to be the time equiement. We need some lemmas. Lemma. Fo the method α 0, the limiting edundancy fo the Benoulli model M 0 A can be uppe bounded as follows: D M 0 A α 0 n /w + 8 Poof. Though this is known fom [7], we give the details of the poof fo the convenience of the eade. Conside a Benoulli souce p and an intege t w. We employ the geneal inequality Dµ η + a A µa 2 /ηa, valid fo any distibutions µ and η ove A follows fom the elementay inequality log x x, and find: D t p α 0 = E p tdp x x t p x x t w i = E p wdp p x x w + px x w x x w A w a A = + a A pa 2 w + n ν a x x w + w i=0 pa 2 w + n i + pa i pa w i = + w + n w + w w + pa i+ pa w i i + i=0 w+ w + j j=0 + w + n w + pa a A pa a A pa j pa w+ j = n w +. As this holds fo any t w and any Benoulli souce p, 8 follows. It may be emaked that if, instead of 5, we define the method α by the fomula p α a x x t = ν ax x w + 2 w + n/2 ; t w, then the edundancy fo small values of w will be less than fo α 0 see [2], [7], howeve, the asymptotic popeties fo memoy size going to infinity will not be changed, wheeas the analysis needed to eplace Lemma will be moe complicated. In ode to each a given small value of = D M 0 A α 0, a cetain size of the window is equied as shown by 8. By 6 this imposes a condition on the memoy size S α0 demanded by the method. Thus, the equied memoy size may be consideed to be a function of = D M 0 A α 0 and we may wite S = S α0. In the same manne, the time of pediction may be consideed to be a function of : T = T α0. Simila consideations apply to any method γ. Lemma and the consideations above suffice in ode to establish the appopiate uppe bounds fo the complexity of the method α 0. Howeve, we must also develop esults that pemit the deivation of lowe bounds and these esults must apply to any method. Othewise, nea optimality of any paticula method, such as e.g. α 0, cannot be ascetained. The technique we shall use in the seach fo lowe bounds uses the notions of ε-capacity and ε-nets developed by Kolmogoov and Tihomiov []. Let MA denote the set of pobability distibutions ove the alphabet A and conside an ε > 0. A subset Γ MA is a 2ε net fo MA if, fo evey p, q Γ with p q thee does not exist a distibution λ MA fo which both Dp λ < ε and Dq λ < ε hold. By C ε MA we denote the ε-capacity of MA defined as the maximum of all numbes log N fo which thee exists a 2ε-net Γ MA with N elements. Befoe we state the capacity bound we need, it is convenient to point out the following auxiliay esult: Lemma 2. Let p = p,, p n, q = q,, q n and π = π,, π n be pobability distibutions. Then maxdp π, Dq π 8 p q 2 9 with denoting l -nom total vaiation. Poof. By Pinske s inequality cf. Csiszá and Köne [5], Dp π 2 p π 2 and Dq π 2 q π 2. Then 8 p q 2 8 p π + q π 2 4

5 8 2 max p π, q π 2 max 2 q π 2 maxdp π, Dq π. We can then pove a key lemma: Lemma 3. C ε MA n 2 log 2 + O. 0 ε Poof. Put δ = 8ε and denote by Γ MA the set of distibutions p = p,, p n such that all coodinates p i with i n ae of the fom p i = k i δ with k,, k n non-negative integes. We show that Γ is a 2ε-net. Indeed, if p Γ, q Γ and p q, thee exists i n such that p i q i. Then p i q i δ, hence we obtain fom Lemma 2 that fo any distibution π, maxdp π, Dq π 8 δ2 = ε, and it follows that Γ is a 2ε-net. Thus C ε M 0 A log Γ denoting numbe of elements in. Let K = δ and denote by a n K the numbe of solutions of the inequality k + + k n K with all the k i s non-negative integes. It is then clea that Γ a n K. Now, a n K = K+ j= K j + n 2 n 2 j=0 j n 2 = n 2! K+ 0 K + n. n! n 2! x n 2 dx Putting things togethe, we find that C ε M 0 A log δ n n! = n log 2 ε + O as claimed. We may note that the O-tem in Lemma 3 is appoximately n log n apply Stiling s fomula. The theoem below shows that the method α 0 is close to being optimal fo 0. Theoem. i. Fo 0, whee = D M 0 A α 0, we have n S α0 n log 2 = n log 2 + O and T α0 = Olog. 2 ii. Fo any pediction method γ we have fo 0, whee = D M 0 A γ, S γ n 2 T γ c log 2 + O, 3 log, 4 whee c is a positive constant also depending on n. Poof. The poof of i was indicated above in connection with Lemma. In ode to establish the moe difficult pat ii of the theoem, conside an > 0 and any method γ of pediction with D M 0 A γ =. We shall pove that and 2 hold fo any such method. In ode to make the ideas of the poof clea, we fist teat the simple case when γ is a deteministic method. In that case the fomula D t p γ = E p tdp p γ x x t 5 holds fo all t and p MA. Choose t so lage that D t p γ fo all p MA. Then, fo each p, thee must exist at least one sting x x t such that Dp p γ x x t. Now, let Γ = {p,, p N } be a 2-net. We can then find stings x i x i t; i N such that Dp i p γ x i x i t ; i =,, N. Accoding to the definition of a 2-net, the distibutions p γ x i x i t; i N must be distinct, hence the N input stings x i x i t; i N must be distinct too and the compute must be able to distinguish between them. This means that the compute must have a memoy of at least log 2 N bits. As N may be chosen equal to exp C MA, 3 now follows fom 0. Then we conside the geneal case of a method which may involve andomization. In that case we find fom and 2 that D t p γ = E p te γ Dp p γ x x t. 6 By Jensen s inequality in the fom Elog x we obtain log Ex D t p γ E p tdp λ γ x x t 7 whee we have defined λ γ x x t MA by λ γ a x x t = E γ p γa x x t ; a A. 5

6 Choose t so lage that D t p γ fo all p M 0 A. Again, let Γ = {p,, p N } be a 2-net. Let us conside a distibution p i Γ. Then thee exists at least one sting x i x i t fo which E γ Dp i p γ x i x i t. Fom 7 and the assumption about the pefomance of the method γ we obtain the inequality We should note that Dp i λ γ x i x i t. Dp j λ γ x i x i t > fo all p j Γ, j i because Γ is a 2-net. The deteministic distibutions λ γ x i x i t, i N ae thus all diffeent, hence also the andom distibutions p γ x i x i t, i N ae diffeent. So we found N input stings such that the N output objects povided by the method γ the above indicated andom distibution ae all distinct. The compute must theefoe be able to distinguish between these N input stings, and this equies a memoy of at least log 2 N bits. The lowe bound 3 now follows as befoe by efeence to 0. As to the lowe bound 4, this is diectly connected with the poof of 3. Indeed, evey pediction method must be able to pint the distibution which is pedicted at each time instant. And, as we saw in the poof of, if D M 0 A γ, then, at some time instant t, the method could lead to any of exp C MA many distibutions whethe andom o not as the distibution pedicted fo x t+. So the compute must contain a code to distinguish between these distibutions. Such a code must contain a codewod of bit length at least log 2 exp C MA. If the coesponding distibution is to be pinted and the compute must be capable of doing that then a look-up of the codewod in question is equied, and this takes time measued in bit opeations of at least the bit length of the codewod. Theefoe, fo a positive constant which is chaacteistic fo the compute used, T γ c log 2 exp C MA and 4 follows fom 0. Remak. An inspection of the poof shows that the given lowe bounds hold in a slightly moe geneal setting fo which lim sup in the defining elation 4 is eplaced by lim inf. 4 A method which is asymptotically optimal with pobability one Fist, we notice that the method α 0 has a seious shotage as the fist w lettes on which the method is based could exhibit a bad statistics. On fist sight, it is tempting to impove on this by using the window x t w+ x t athe than x x w fo the pediction of x t+. With this change, the method will be good with pobability one and not only in aveage. But then, afte pedicting evey lette x t+, we would have to move the window: x t+ should be included in the window and x t w+ emoved. This will equie w log 2 n bits athe than the n log 2 w bits which suffice fo α 0. In moe detail, the point is that fo any of the w positions making up the window, we would have to know which lette was obseved at that time instant, so that even if n = 2, we would need w bits of memoy fo this pupose. When the pecision goes to 0, the paamete w goes to infinity and the method with a sliding window would need constant/ bits of memoy as compaed to n log/ bits fo α 0. In ode to peseve the elatively small demand fo memoy fo α 0 while at the same time impoving the pediction by using also the late lettes, we popose to use the method β 0 of the imaginay sliding window fom [9] elated methods ae used in compute science, e.g. egading so-called paging, cf. [4], Chapte 3. At each time instant t we need only woy about t w we keep tack of cetain numbes ν t a,, ν t a n, which we think of as fequencies, and use these to pedict x t+ accoding to the fomula p β 0 a x x t = νt a + w + n ; t w. The fequencies ν t a ae andom vaiables as they depend on x,, x t. We shall now explain how these fequencies ae defined. The stating fequencies ν w a,, ν w a n ae the tue fequencies in the window x x w. The new featue of β 0 is that afte pediction of x t+ t w, we change the fequencies as follows: Fist, we consult the oacle and choose a lette at andom accoding to the pobabilities ν t a/w, a A. If a j is chosen, we define the new fequency fo this lette using the convention ν t+ a j = ν t a j. Afte this, we add to the fequency of x t+ ν t+ x t+ = ν t x t+ +. If a x t+ and a a j, 6

7 we do not change its fequency ν t+ a = ν t a. The new fequencies ν t+ a i ; i n ae then used to pedict x t+2, and afte this the fequencies ae again changed, x t+3 is pedicted and so on. The memoy size equied fo the method β 0 is n log 2 w+ bits, and can moe conveniently be taken to be S w = n log 2 w 8 just as fo the method α 0, cf. 6. Howeve, in taking 8 as expession fo the memoy size, we ignoe the asymptotically negligible demand fo memoy esulting fom the use of a andom numbe geneato. This is also justifyed by the consideations detailed in the intoduction. When estimating the time of pediction, we likewise ignoe the complexity of a andom numbe geneato and only conside the tansfomation to values with pobabilities ν t a/w; a A. Using fast methods of tansfomation as descibed in [8], one finds that the aveage time of tansfomation pe lette is Olog w bit opeations. Using popeties of the imaginay sliding window, see [9] and also the emaks below, and appealing to calculations simila to those fo the poof of Theoem, we now obtain the following esult: Theoem 2. Let > 0 be given, put w = n log 2 e/ and apply the method β 0 of pediction with this paamete value w. Then, fo evey p M 0 A, P lim sup β0,px,, x t =, t S β0 n log 2 + O, T α0 = O log. Remaks. As the lowe bounds 3 and 4 still apply, the esult shows that the method β 0 is not all that fa fom being optimal. A emak on the poof is also in place. It is essential that the statistics of the numbes ν t a i eally do behave as they should, i.e. that they convege to the coesponding tue fequencies. We point out that the poof of this cucial fact is easily accomplished when tansfoming the poblem in a natual way into a poblem of calculating the invaient distibution associated with the Makov chain which models the updating method of the imaginay sliding window. 5 Makov souces In 2 we indicated that extensions of the key esults to cove the geneal Makov case ae possible. We take this up now. The tick is to view a Makov souce p M k A as esulting fom A k Benoulli souces. We illustate this idea by an example. So assume that A = {O, I}, k = 2 and assume that the souce p M 2 A has geneated the sequence OOIOIIOOIIIOIO. We epesent this sequence by the following fou subsequences: I I, O I I O, I O I, O IO. These fou subsequences contain lettes which follow afte OO, afte OI, afte IO and afte II, espectively. By definition, p M k A if pa x x t = pa x t m+ x t, fo all k m t, all a A and all x x t A t. Theefoe, each of the fou geneated subsequences may be consideed to be geneated by a Benoulli souce. Futhe, it is possible to econstuct the oiginal sequence if we know the fou = A k subsequences and the two = k fist lettes of the oiginal sequence. In ode to pedict, it is enough to stoe in the memoy A k methods fo Benoulli souces α 0 o β 0, one coesponding to each wod in A k. Thus, in the example, the lette x 3 which follows afte OO is pedicted based on the Benoulli method coesponding to the OO- subsequence = II, then x 4 is pedicted based on the Benoulli method coesponding to x 2 x 3, i.e. to the OI- subsequence = OIIO, and so foth. It will not be impotant how to pedict x x 2 o, in geneal, x x k. The methods fo M k A which ae obtained in this way by using eithe α 0 fo all A k subsequences o else β 0 fo all these subsequences, we denote by α k and β k, espectively. Fo the associated memoy size and the associated aveage time of calculation by lette we find the following esult: Theoem 3. i. Denoting the limiting aveage divegence fo the method α k by, then, as 0, and S αk = n k n log 2 + O 9 T αk = O log. 20 ii. If, fo the method β k, P lim sup βk,px x t t =, 7

8 then, as, and and S βk n k n log 2 + O 2 T βk = O log. 22 iii. Fo evey pediction method γ fo M k A, S γ nk n 2 log 2 + O 23 T γ c log, 24 whee c is a positive constant which also depends on n and k. Poof. As it was shown above, evey Makov souce p M k A can be pesented as esulting fom A k Benoulli souces. So, the compute has to stoe A k tables in ode to pedict each lette x t using the coesponding above descibed method fo M 0 A. That is why the memoy size equied fo the suggested method is A k times moe than fo the coesponding method fo a single Benoulli souce, but the time of pediction is asymptotically the same. It is not impotant how to pedict the fist lettes x,, x k, because we ae inteested in asymptotic behaviou of the method. Fo example, we can ascibe equal pobabilities A k to all stings x x k. The same epesentation of a Makov souce may be used in ode to obtain the lowe bounds. In fact, in the case of Makov souces, the dimension of the paamete space is equal to A A k, unlike the Benoulli case when the dimension is A. Using a simila estimation of the ε-capacity as befoe, we can obtain the lowe bounds of the memoy size. 6 Concluding emaks Fom a pactical point of view, the β methods have an exta advantage ove the α methods which is connected also with the fact that the β methods ae good with pobability wheeas α methods ae only good in aveage. The point is that the vey natue of the β methods make them obust to changes in time of the statistics of the souce. Note that the methods developed α as well as β wee designed solely with the aim of obtaining good methods fo special stationay souces. Clealy, futhe eseach should boaden the scope. But even though we did not aim at developing methods fo non-stationay souces it so happens that the notion of the imaginay sliding window is pefectly suited to handle souces with changes in time of the statistics. This pape is focused on theoetical consideations and has the pimay aim to study pefomance in tems of complexity, especially we have endeavoued to initiate eseach which gives tight lowe bounds of pefomance possibilities which ae close to the optimally achievable. On the moe pactical side, it would be inteesting to study the pefomance of ou methods and methods developed by othe authos in eal expeiments based on eal data. Acknowledgements. The authos had helpful discussions with Joe Suzuki in Novosibisk, July 998, and late with Jyki Katajainen, Pete Andeasen and Pete Haemoës. Some of the comments eceived esulted in claifications egading the use of andomization. Thanks ae due to two efeees who equested cetain expansions and eaangements of the mateial which has made the pape moe eadable and self-contained. The meticulous wok of the efeees is gatefully acknowledged. Refeences. Aho A.V., Hopcoft J.E. and Ullman J.D., The design and analysis of compute algoithms. Addison-Wesley, Cambidge, Algoet P., Univesal Schemes fo Leaning the Best Nonlinea Pedicto Given the Infinite Past and Side Infomation, IEEE Tans. Infom. Theoy, v. 45, pp , Cove T.M., Feedman A.M. and Hellman M.E., Optimal Finite Memoy Leaning Algoithms fo the Finite Sample Poblem, Infomation and Contol, v. 30, pp , Cove T.M. and Thomas J.A., Elements of Infomation Theoy. John Wiley & Sons, New Yok, Csiszá I. and Köne J., Infomation Theoy, Coding Theoems fo Discete Memoyless Systems. Akademiai Kiado, Budapest Fede M. and Mehav N., Univesal pediction, IEEE Tans. Infom. Theoy, v. 44, pp , Fede M., Mehav N. and Gutman M., Univesal pediction of individual sequences, IEEE Tans. Infom. Theoy, v. 38, pp ,

9 8. Felle W., An Intoduction to Pobabability Theoy and Its Applications, vol.. John Wiley & Sons, New Yok, Vedu S., Fifty yeas of Shannon Theoy, IEEE Tans. Infom. Theoy, v. 44, pp , Gallage R. G., Infomation Theoy and Reliable Communication. John Wiley & Sons, New Yok, Kelly J.L., A new intepetation of infomation ate, Bell System Tech. J., v. 35, pp , Kolmogoov A.N. and Tihomiov V.M., Epsilon-entopy and epsilon-capacity of sets in metic spases, Uspechi Math. Nauk. v. 4, pp. 2-86, 959. Russian oiginal, tanslated into English. 2. Kichevsky R., Univesal Compession and Retival. Kluve Academic Publishes, Dodecht, Kichevskii R., Laplace s Law of Succession and Univesal Encoding, IEEE Tans. Infom. Theoy, v. 44, pp , Motwani R. and Raghavan, P., Randomized Algoithms. Cambidge Univesity Pess, Cambidge, Rissanen J., Univesal coding, infomation, pediction, and estimation, IEEE Tans. Infom. Theoy, v. 30 pp , Ryabko B. Ya., The complexity and effectiveness of pediction algoithms, J. of Complexity, v. 0, pp , Ryabko B. Ya., Pediction of andom sequences and univesal coding, Poblems Infom. Tansm. v. 24, pp , Ryabko B. Ya., A fast on-line adaptive code, IEEE Tans. Infom. Theoy, v. 38, pp , Ryabko B. Ya., Data compession by an imaginay sliding window, Poblems Infom. Tansmission v. 32, pp , Topsøe F, Infomation Theoetical Optimization Techniques, Kybenetika, v. 5, pp. 8-27, Topsøe F., Game Theoetical Equilibium, Maximum Entopy and Minimum Infomation Discimination, in Mohammad-Djafai et al eds.: Maximum Entopy and Bayesian Methods. Kluwe, Dodecht,