Gusztav Morvai. Hungarian Academy of Sciences Goldmann Gyorgy ter 3, April 22, 1998

Transcription

1 A simple radmized algrithm fr csistet sequetial predicti f ergdic time series Laszl Gyr Departmet f Cmputer Sciece ad Ifrmati Thery Techical Uiversity f Budapest 5 Stczek u., Budapest, Hugary gyrfi@if.bme.hu Gabr Lugsi Departmet f Ecmics, Pmpeu Fabra Uiversity Ram Trias Fargas 5-7, 85 Barcela, Spai, lugsi@upf.es Gusztav Mrvai Research Grup fr Ifrmatics ad Electrics Hugaria Academy f Scieces 5 Gldma Gyrgy ter 3, Budapest, Hugary mrvai@if.bme.hu April, 998 Classicati: C3. The wrk f the secd authr was supprted by DGES grat PB96-3

2 Abstract We preset a simple radmized prcedure fr the predicti f a biary sequece. The algrithm uses ideas frm recet develpmets f the thery f the predicti f idividual sequeces. We shw that if the sequece is a realizati f a statiary ad ergdic radm prcess the the average umber f mistakes cverges, almst surely, t that f the ptimum, give by the Bayes predictr.

3 Itrducti We address the prblem f sequetial predicti f a biary sequece. A sequece f bits y y y :::f g is hidde frm the predictr. At each time istat i = :::, the bit y i; is revealed ad the predictr is asked t guess the value f ext utcme y i.thus, the predictr's decisi, at time i, is based the value f y i; =(y ::: y i; ). We als assume that the predictr has access t a sequece f i.i.d. radm variables U U :::, uifrmly distributed [ ], s that the predictr ca use U i i frmig a radmized decisi fr y i.frmally, the strategy f the predictr is a sequece g = fg i g i= f decisi fuctis g i : f g i; [ ]!f g ad the radmized predicti frmed at time i is g i (y i; U i ). The predictr pays a uit pealty each time a mistake is made. After ruds f play, the rmalized cumulative lss the strig y is L (g U )= X i= I fgi (y i; U i )6=y i g where I detes the idicatr fucti. Whe cfusi is caused, we will simply write L (g) =L (g U ). I geeral, we dete We als write L m(g U )= ; m + X i=m I fgi (y i; U i )6=y i g : bl (g) =EL (g U ) ad b L m (g) =EL m (g U ) fr the expected lss f the radmized strategy g. I this paper we assume that y y :::are realizatis f the radm variables Y Y ::: draw frm the biary-valued ergdic prcess fy g ; (which is idepedet f the radmizig variables U U :::). I this case there is a fudametal it fr the predictability f the sequece. This is stated i the ext lemma: Therem Fr ay predicti strategy g ad statiary ergdic prcess fy g ;, if! L (g) L almst surely, where h L = E mi PfY =jy ; ; g PfY =jy ; ; gi is the miimal (Bayes) prbability f errr f ay decisi fr the value f Y iite past Y ; ; =(::: Y ;3 Y ; Y ; ). based the

4 Prf. A easy applicati f the Azuma-Hedig iequality fr sums f buded martigale diereces (Hedig [8], Azuma []) shws that fr ay predicti strategy g ad >, P I particular, ( L (g) ; X i=! L (g) ; Pfg i (Y i; U i ) 6= Y i jy i; X i= ; g > It is well kw (see, e.g., [6]) that fr ay predictr g i, U ::: U ) e ; : Pfg i (Y i; U i ) 6= Y i jy i; ; g! = almst surely: Pfg i (Y i; U i ) 6= Y i jy i; ; g Pfg i (Y i; ; ) 6= Y i jy i; ; g = mi PfY i =jy i; ; g PfY i =jy i; ; g where g i is the ptimal (Bayes) decisi fr Y i based Y; i; give by ( if g i (y;) i; PfYi =jy; = i; = y;g i; = therwise. Nte that by statiarity, g = g = def = g. Therefre, if! L (g) if! X i= mi PfY i =jy i; ; g PfY i =jy i; ; g almst surely. Fially, we te that by the ergdic therem (see, e.g., Stut [3]) the average the righthad side cverges almst surely t L, s the prf is ished. Based Therem the fllwig deiti is meaigful. Deiti Apredicti strategy g is called csistet if fr all ergdic prcesses fy g ;,! L (g) =L almst surely: Therefre, csistet strategies asympttically achieve the best pssible lss fr all ergdic prcesses. The rst questi is, f curse, if such a strategy exists. The armative aswer may be easily deduced frm earlier results f Orstei ad Bailey as fllws: Therem There exists a csistet predicti scheme. Prf. Orstei [] prved that there exists a sequece f fuctis f i : f g i! [ ], i = ::: such that fr all ergdic prcesses fy g ;,! f (Y ; ; )=PfY =jy ;g almst surely: ()

5 (A simpler estimatr with the same cvergece prperty was itrduced by Mrvai, Yakwitz, ad Gyr [].) Bailey [3] shwed that fr such estimatrs, fr all ergdic prcesses! X i= j(f i (Y i) ; PfY i+ =jy i gj = almst surely: () Ideed, () ad Breima's geeralized ergdic therem (see, e.g., Alget []) yield (). Oce such a sequece ff i g f estimatrs is available, we may dee a (-radmized) predicti scheme by By [6, Therem.], P g (Y ; ) 6= Y jy; ; g (y ; )= ; P ( if f; (y ; ) therwise. g (Y; ; ) 6= Y jy; ; f; (Y ; ) ; P Y =jy ; ; therefre jl (g) ; L j L (g) ; + + X i= X i= P L (g) ; + + X i= X i= P g i (Y i; ) 6= Y i jy i; g i (Y i; ) 6= Y i jy i; ; P P g (Y; i; ) 6= Y i jy; i; ; L X i= P g i (Y i; ) 6= Y i jy i; f i; (Y i; ) ; P X i= Y i =jy i; P g (Y; i; ) 6= Y i jy; i; g (Y i; ; ) 6= Y ijy i; ; ; L : ; The rst term f the right had side teds t zer almst surely by the Hedig-Azuma iequality [8], [] by a similar argumet that was used i the prf f Therem. The secd e cverges t zer almst surely by () ad the third term teds t zer almst surely by the ergdic therem. Ufrtuately, all kw estimatrs satisfyig () are either very cmplicated r eed s large amuts f data that their practical use is urealistic. Therefre, desigig a simple direct algrithm is called fr. 3

6 A simple csistet algrithm I this secti we preset a simple predicti strategy, ad prve its csistecy. It is mtivated by sme recet develpmets frm the thery f the predicti f idividual sequeces (see, e.g., Vvk [4], Feder, Merhav, ad Gutma [7], Littleste ad Warmuth [9], Cesa-Biachi et al. [5]). These methds predict accrdig t a cmbiati f several predictrs, the s-called experts. The mai idea i this paper is that if the sequece t predict is draw frm a statiary ad ergdic prcess, cmbiig the predictis f a small ad simple set f apprpriately chse predictrs (the s-called experts) suces t achieve csistecy. First we dee a iite sequece f experts h () h () ::: as fllws: Fix a psitive iteger k, ad fr each s f g k ad y f g dee bp k (y y ; js) = fk <i<: y i; fk <i<: y i; i;k = s y i = yg i;k = sg >k+: (3) = is deed t be =. Als, fr k +we dee P b k (y y; js) = =. I ther wrds, bp k (y y ; js) is the prprti f the appearaces f the bit y fllwig the strig s amg all appearaces f s i the sequece y ;. Als itrduce the fucti F k (z) =I fz[:5;=k :5+=k]g z ; :5+=k =k + I fz>:5+=kg : The expert h (k) deed by is a sequece f fuctis h (k) : f g ; [ ]!f g, = ::: ( if h (k) u<fk ( P b (y; u)= k ( y ; jy;k)) ; therwise, = :::: That is, expert h (k) lks fr all appearaces f the last see strig y ; ;k f legth k i the past ad predicts accrdig t the larger f the relative frequecies f 's ad 's fllwig the strig. The fucti F k ly plays a rle if these frequecies are clse t =. I such a case a radmized predicti is made. (Nte that F k (z) is ctiuus ad it diers frm I fz=g ly if jz ; =j < =k.) The prpsed predicti algrithm prceeds as fllws: Let m = ::: be a egative iteger. Fr m < m+, the predicti is based up a weighted majrity f predictis f the experts h () ::: h (m+ ) as fllws: g (y ; u)= 8 >< >: P m+ k= if u< F k( b k ( y ; jy ; ;k ))w (k) P m+ w k= (k) therwise, = ::: where w (k) is the weight fexperth (k) deed by the past perfrmace f h (k) as w m(k) = ad w (k) =e ;m b L ; m (h (k) ) > m 4

7 where m = q 8l( m+ )= m. Recall that bl ; m (h(k) )= ; m ; X P h (k) i (y i; U i ) 6= y i i= m is the average umber f mistakes made by expert h (k) betwee times m ad ;. The weight feach expert is therefre expetially decreasig with the umber f its mistakes this part f the data. Our mai result is the csistecy f this simple predicti scheme: Therem 3 The predicti scheme g deed abve is csistet. I the prf we use a beautiful result f Cesa-Biachi et al. [5]. It states that, give a set f N experts, ad a sequece f xed legth, there exists a radmized predictr whse umber f mistakes is t mre tha that f the best predictr plus abut q (=) l N fr all pssible sequeces y. The simpler algrithm ad statemet cited belw is due t Cesa-Biachi [4]: Lemma Let h ~ () ::: ~ h (N ) be a ite cllecti f predicti strategies (experts). The if the predicti strategy ~g is deed by ~g t (y t; u)= 8 >< >: if u< therwise, t = :::, where fr all k = ::: N P N k= P ~h (k) (y t; U)= ~w t (k) P N k= ~w t (k) ~w (k) = ad ~w t (k) b =e ; L t; ( ~ h (k)) t> with = q 8lN=, the fr every y f g, bl (~g) mi bl ( ~ l N h )+s (k) k= ::: N : Prf f Therem 3. By Lemma, we have that the expected umber f errrs cmmitted by g a segmet m ::: m+ ; is buded, fr ay y m+ ; m f gm,as 3 bl m+ ; m (g) = E 4 m+ X; 5 m mi L b k m+ I fgi (y i; i= m U i )6=y i g m+ ; m (h(k) )+ = mi bl m+ ; k= ::: m (h(k) )+ 5 s l( m+ ) s m l( m+ ) m

8 where the last equality fllws frm the fact that sice < m+, all experts h (k) with k m+ predict zer with prbability = up t time (ad therefre they are idetical t h (m+ ) ). Therefre, detig = blg c+,fray sequece y y :::, therefre where b L (g) = blg c; X m= X blg c; m= +( ; =+) bl (g) mi bl h (k) + k= ::: mi bl h (k) + k= ::: mi bl k= ::: mi bl k= ::: mi bl k= ::: m L b m+ ; m (g)+(; =+)b L=(g) s l( m+ ) mi bl m+ ; k= ::: m (h(k) mi k= ::: blg c; X m= blg c bl = (h(k) )+ s m l m+ s m+ l m+ vu u t m= p q blg X c h (k) + l blg c + m= p q p h (k) + l lg + ( p ; ) h (k) + c s lg + c = p l p ; :84: m A l() A ( ; =+) + s m= ( ; = + ) l A It fllws frm McDiarmid's iequality (McDiarmid [] see als [6, Therem 9.]) that fr ay sequece y, P L (g U ) ; b L (g) > e ; : Therefre, if L ad b L are w evaluated the radm sequece Y Y :::,we btai sup! L (g U ) sup@ mi bl (h (k) lg + )+cs A :! k= ::: = sup mi bl (h (k) )! k= ::: Thus, it remais t shw thatfray ergdic prcess Y Y :::, sup! mi k= ::: This will fllw easily frm the fllwig lemma: almst surely: bl (h(k) ) L almst surely: (4) 6

9 Lemma Fr each k, sup! where fr ay s f g k, L b (h (k) ) ; P g (k) (Y ; ) 6= ;k Y k almst surely g (k) (s) = is the Bayes decisi fr Y give Y ; ;k. ( if P Y =jy ; ;k = s therwise Prf. Nte that bl (h (k) ) = = X i= E X i= I (k) fh i (Y i; U i )6=Y i g jy E I (k) fh i (Y i; U i )6=Y i g jy i= ; X i= I fg (k) (Y i; i;k )6=Y ig! + X i= I fg (k) (Y i; i;k )6=Y ig : Fr the secd term the right-had side, it fllws frm the ergdic therem that X I! fg (k) (Y i; i;k )6=Y ig = P g (k) (Y ; ) 6= ;k Y almst surely: Therefre, it suces t shw that sup! X i= T see this, write X i= E = i= E I (k) fh i (Y i; U i )6=Y i g jy X ; X I fg (k) (Y i; i;k )6=Y ig i= k almst surely: I (k) fh i (Y i; U i )6=Y i g jy ; I fg (k) (Y i; i;k )6=Y ig i= X i E hy i + h (k) i (Y i; U i ) ; Y i h (k) i (Y i; U i )jy ; Yi ; g (k) (Y i; i;k )+Y i g (k) (Y i; i;k ) (usig that if a b f g the I fa6=bg = a + b ; ab) = X i Y i +(; Y i )E hh (k) i (Y i; U i )jy ; Yi ; ( ; Y i )g (k) (Y i; = = X ( ; Y i ) F k ( P bk i= i= X ( ; Y i ) F k ( P bk i= + X i= i ( Yi; jy i; i ( Yi; i;k )) ; g(k) (Y i; jy i; i;k ) i;k )) ; F k(pfy i =jy i; i;k g)! i;k ) ( ; Y i ) F k (PfY i =jy i; i;k )g;g(k) (Y i; )! i;k : (5) 7

10 Nw it fllws frm the ergdic therem that! ad therefre max yf g sf g k b P k i! (y Y ; js) ; P P b k i; i ( Y jy i; )) ; i;k P Y i =jy i; ) i;k = Y = yjy ; ;k = s = almst surely almst surely s by the ctiuity ff k,wehave, fr the rst term the right-had side f (5), that! X ( ; Y i ) F k ( P bk i= i ( Y i; jy i; i;k )) ; F k (PfY i =jy i; i;k g) = almst surely: Fr the secd term the right-had side f (5), te that by the ergdic therem, almst surely, X ( ; Y i ) F k P Y i =jy i; i;k ; g (k) (Y i; i;k ) i= = h E ( ; Y ) F k P Y =jy ; ;k ; g (k) (Y ; )i ;k = h E he ( ; Y ) F k P Y =jy ; ;k ; g (k) (Y ;) ii ;k Y ; ;k = h E ; P Y =jy ; ;k F i k P Y =jy ; ;k ; g (k) (Y i; i;k ) E ; P Y =jy ; ;k I fjpfy =jy ;k ; g;=j=kg! E k I fjpfy =jy ; k ;k g;=j=kg ad Lemma is prved. Nw we retur t the prf f Therem 3. Sice h P g (k) (Y ; ) 6= ;k Y = E mi PfY =jy ; ;k g PfY =jy ; ;k gi it fllws frm the martigale cvergece therem ad Lebesgue's dmiated cvergece therem that h i P g (k) (Y ; ;k ) 6= Y = E mi PfY =jy ; ;k g PfY =jy ; ;k g k! k! Therefre, we cclude that k! sup! sup k!! k! k = : L b (h(k) ) ; L = E b L (h (k) ) ; P (By Lemma ad (6)): h mi PfY =jy ; ;g PfY =jy ; ;g i = L : (6) g (k) (Y ; ;k ) 6= Y + P g (k) (Y ; ;k ) 6= Y ; L 8

11 Fially, fr a xed >, chse the psitiveiteger K such that sup! b L (h(k) ) ; L <. The sup mi bl (h (k) ) sup bl (h (K) )! k= :::! L + : Sice was arbitrary, (4) is established, ad the prf f the therem is ished. Remarks.. The prpsed estimate is clearly easy t cmpute. Oe merely has t keep track f the expected cumulative lsses b L ; m (h(k) ) fr k = :::.. We see frm the aalysis that fr ay sequece y y ::: ad fr all, bl (g) mi bl k= ::: h (k) +3 s lg + I ther wrds, the algrithm is guarateed t perfrm almst well as the best expert. The rate f cvergece t L depeds the behavir f the best expert. 3. The fucti F k is deed smewhat arbitrarily. All that's eeded fr csistecy is that F k (z) is ctiuus ad it ly diers frm I fz=g i a shrikig eighbrhd f = as k!. Fr ite-sample behavir the chice f F k may be a imprtat issue, hwever, we cat er ay gd ituiti this. Actually, F k (z) =I fz=g is the mst atural chice, but ctiuity ff k is eeded i ur aalysis. We dtkw if it is ecessary. : 9

12 3 Predicti with side ifrmati I this secti we apply the same ideas t the seemigly mre dicult classicati (r patter recgiti) prblem. The setup is the fllwig: let f(x Y )g =; be a statiary ad ergdic sequece f pairs takig values i R d f g. The prblem is t predict the value f Y give the data (X D ; ), where we dete D ; =(X ; Y ; ). The predicti prblem is similar t the e studied i the previus secti with the excepti that the sequece f X i 's is als available t the predictr. Oe may thik abut the X i 's as side ifrmati. We may frmalize the predicti prblem as fllws. A (radmized) predicti strategy is a sequece g = fg i g i= f decisi fuctis g i : f g i; R d i [ ]!f g s that the predicti frmed at time i is g i (y i; x i U i). The rmalized cumulative risk fr ay xed pair f sequeces x y is w R (g U )= X i= I fgi (y i; x i U i)6=y i g We als use the shrt tati R (g) =R (g U ). Dete the expected risk f the radmized strategy g by br (g) =ER (g U ): Similarly t the tati f the previus secti, we write R m (g U )= ; m + X i=m I fgi (y i; x i U i)6=y i g ad b R m (g) =ER m (g U ): We assume that the radmizig variables U U :::are idepedet f the prcess f(x Y )g. Just like i the case f predicti withut side ifrmati, the fudametal it is give by the Bayes prbability f errr: Therem 4 Fr ay predicti strategy g ad statiary ergdic prcess f(x Y )g =;, where if! R (g) R almst surely, h i R = E mi PfY =jy ; ; X ;g PfY =jy ; ; X ;g : The prf f this lwer bud is similar t that f Therem, the details are mitted. It fllws frm results f Mrvai, Yakwitz, ad Gyr [] that there exists a predicti strategy g such that fr all ergdic prcesses, R (g)! R almst surely. (The result f [] shuld be cmplemeted with a argumet similar appearig i the prf f Therem t btai the abve statemet. T avid repetiti, the details are agai mitted.) The algrithm f Mrvai, Yakwitz, ad Gyr, hwever, is t useful i practice, as it requires

13 a astrmical data size. The mai message f this secti is a simple csistet prcedure with a practical appeal. The idea, agai, is t cmbie the decisis f a small umber f simple experts i a apprpriate way. We dee a iite array f experts h (k `), k ` = ::: as fllws. Let P` = fa` j j = ::: m`g be a sequece f ite partitis f the feature space R d, ad let G` be the crrespdig quatizer: G`(x) =j if x A` j R d,we write G`(x )frthese- With sme abuse f tati, fr ay ad x quece G`(x ) ::: G`(x ). Fix psitive itegers k `, ad fr each s f g k ad z f ::: m`g k+ ad y f g dee bp (k `) (y y ; fk <i<: y i; x js z) = = i;k s G`(x i i;k) =z y i = yg fk <i<: y i; = i;k s G`(xi )=z g i;k >k+: (7) = is deed t be =. Als, fr k +we dee b P (k `) (y y ; x js z) ==. The expert h (k `) is w deedby h (k `) (y ; therwise, x u)= ( if u<fk ( b P (k `) ( y ; x jy ; ;k G`(x ;k))) = ::: where F k is deed i the previus secti. That is, expert h (k `) quatizes the sequece x accrdig t the partiti P`, ad lks fr all appearaces f the last see quatized strigs y ; ;k G`(x ;k) f leght k i the past. The it predicts accrdig t the larger f the relative frequecies f 's ad 's fllwig the strig. The prpsed algrithm cmbies the predictis f these experts similarly t that f the previus secti. This way bth the legth f the strig t be matched ad the resluti f the quatizer are adjusted depedig the data. The frmal deiti is as fllws: Fr ay m = :::,if m < m+, the predicti is based up a weighted majrity f predictis f the ( m+ ) experts h (k `), k l m+ as fllws: g (y ; x u)= 8 > < >: Pk ` if u< m+ F k( P b (k `) ( y ; x jy ; ;k G`(x ;k)))w (k `) P k ` m+ w (k `) therwise, where w (k `) is the weight f expert h (k `) deed by the past perfrmace f h (k `) as w m(k `) = ad w (k `) =e ;m b R ; m (h (k `) ) > m where m = q 8l( m+ ) = m. Fr the csistecy f the methd, we eed sme atural cditis the sequece f partitis. We assume the fllwig: (a) the sequece f partitis is ested, that is, ay cell f P`+ is a subset f a cell f P`, ` = :::

14 (b) each partiti P` is ite (c) if diam(a) =sup x ya kx ; yk detes the diameter f a set, the fr each sphere S cetered at the rigi max diam(a` j) =: `! j A` j \S6= Therem 5 Assume that the sequece fpartitis P` satises the three cditis abve. The the patter recgiti scheme g deed abve satises! R (g) =R almst surely fr ay statiary ad ergdic prcess f(x Y )g =;. Prf f Therem 5. Exactly the same way as i the rst part f the prf f Therem 3, we btai that fr ay statiary ad ergdic prcess f(x Y )g =;, sup! R (g U ) sup! Thus, it remais t shw that = sup! mi k = ::: ` = ::: ; mi k = ::: ` = ::: ; br (h (k `) )+cs lg + br (h (k `) ) almst surely: C A sup! mi k = ::: ` = ::: ; br (h (k `) ) R almst surely: T prve this, we use the fllwig lemma, whse prf is easily btaied by cpyig that f Lemma : Lemma 3 Fr each k, sup! R(h b (k `) ) ; P g (k `) (Y ; ;k X ;k) 6= Y k almst surely where fr ay s f g k ad z f ::: m`g k+, ( if P Y g (k `) (s z) = =jy ; ;k = s G`(X;k) =z therwise is the Bayes decisi fr Y give Y ; ;k G`(X ;k). Nw we retur t the prf f Therem 5. Fr x `, the sequeces PfY =jy ; ;k G`(X ;k)g ad PfY =jy ; ;k G`(X ;k)g k = :::

15 are martigales, ad they cverge almst surely t PfY =jy ; ; G`(X ; )g ad PfY =jy ; ; G`(X ; )g respectively. Sice the sequece f partitis P` is ested, ad by (c), the sequeces PfY =jy ; ; G`(X ; )g ad PfY =jy ; ; G`(X ; )g l = ::: are martigales ad they cverge almst surely t PfY =jy ; ; X ; g ad PfY =jy ; ; X ; g: Thus, it fllws frm Lebesgue's dmiated cvergece therem that Sice P l! k! E h mi = E PfY =jy ; ;k G`(X ;k )g PfY =jy ; ;k G`(X )gi ;k h mi PfY =jy ; ; X ;g PfY =jy ; ; X ;g i = R : g (k `) (Y ; ;k X ;k) 6= Y = E h mi PfY =jy ; ;k G`(X ;k)g PfY =jy ; ;k G`(X ;k)g i we cclude that `! k! Nw it fllws easily that sup! sup `! k!! R b (h (k `) ) ; R R b (h (k `) ) ; P ; R + P g (k `) (Y ; ;k X ) 6= ;k Y = almst surely: sup! mi k = ::: ` = ::: ; br (h (k `) ) R g (k `) (Y ; ;k X ;k) 6= Y almst surely ad the prf f the therem is ished. Ackwledgemet. We thak Nicl Cesa-Biachi fr teachig us all wee eeded t kw abut predicti with expert advise. 3

16 Refereces [] P. Alget. The strg law f large umbers fr sequetial decisis uder ucertaiity. IEEE Trasactis Ifrmati Thery, 4:69{634, 994. [] K. Azuma. Weighted sums f certai depedet radm variables. Thku Mathematical Jural, 68:357{367, 967. [3] D.H. Bailey. Sequetial schemes fr classifyig ad predictig ergdic prcesses. PhD thesis, Stafrd Uiversity, 976. [4] N. Cesa-Biachi. Aalysis f tw gradiet-based algrithms fr -lie regressi. I Prceedigs f the th Aual Cferece Cmputatial Learig Thery, pages 63{7. ACM Press, 997. [5] N. Cesa-Biachi, Y. Freud, D.P. Helmbld, D. Haussler, R. Schapire, ad M.K. Warmuth. Hw t use expert advice. Jural f the ACM, 44(3):47{485, 997. [6] L. Devrye, L. Gyr, ad G. Lugsi. A Prbabilistic Thery f Patter Recgiti. Spriger-Verlag, New Yrk, 996. [7] M. Feder, N. Merhav, ad M. Gutma. Uiversal predicti f idividual sequeces. IEEE Trasactis Ifrmati Thery, 38:58{7, 99. [8] W. Hedig. Prbability iequalities fr sums f buded radm variables. Jural f the America Statistical Assciati, 58:3{3, 963. [9] N. Littleste ad M. K. Warmuth. The weighted majrity algrithm. Ifrmati ad Cmputati, 8:{6, 994. [] C. McDiarmid. O the methd f buded diereces. I Surveys i Cmbiatrics 989, pages 48{88. Cambridge Uiversity Press, Cambridge, 989. [] G. Mrvai, S. Yakwitz, ad L. Gyr. Nparametric iferece fr ergdic, statiary time series. Aals f Statistics, 4:37{379, 996. [] D.S. Orstei. Guessig the ext utput f a statiary prcess. Israel Jural f Mathematics, 3:9{96, 978. [3] W.F. Stut. Almst sure cvergece. Academic Press, New Yrk, 974. [4] V.G. Vvk. Aggregatig strategies. I Prceedigs f the Third Aual Wrkshp Cmputatial Learig Thery, pages 37{383. Assciati f Cmputig Machiery, New Yrk, 99. 4