Divergence criteria for improved selection rules

Size: px

Start display at page:

Download "Divergence criteria for improved selection rules"

Mitchell Mathews
5 years ago
Views:

1 Divergece criteria for improved selectio rules. Berliet ad I. Vajda N ovem b er 3, 00 bstract t the basis of combiatorial methods i desity estimatio itroduced by Devroye ad Lugosi is the so-called Sche é selectio rule. We show by a examples that this rule based o L errors may ot brig the selectio closer to optimality tha tossig of a coi. s i ay estimatio problem, the choice of a criterio is at the heart of the matter. The optimality of the Sche é estimate is perceived di eretly by di eret divergece criteria. We show that the L oracle iequality satis ed by the Sche é estimate ca be exteded to divergeces. It ca be also exteded to estimates associated with selectio rules based o divergeces. s the L rule, the ew rules are applicable to ay selectio problem i desity estimatio. MS 99 subject classi catio: 6 G 05. Key Words: Desity estimatio. Noparametric estimatio. Selectio of estimates. Iformatio divergece. Optimality. Combiatorial methods. Itroductio ad basic cocepts I the book of Devroye ad Lugosi (00), the authors cosidered the statistical model with R d -valued observatios X ; : : : ; X i.i.d. by a probability desity f o R d ad two estimates f (i) = f (i) (; X ; : : : ; X ); i f; g () of this desity. They were iterested i the problem how to select for each realizatio of X ; : : : ; X the better of these estimates i the sese of L -error. Obviously, the optimal but practically uachievable selectio is f (0) = >< >: f () f () if otherwise: jf () fj < They proposed a practically achievable approximatio to this selectio called Sche é estimate which is selected by by the rule >< f () f if f () ( ) = < f () ( ) ; (3) >: otherwise f () jf () I3M, UMR CNRS 549, Uiversity of Motpellier II, Motpellier Cedex, Frace Istitute of Iformatio Theory ad utomatio, SCR, 0 Prague, Czech Republic fj; ()

2 where is the so-called Sche é set for f () ; f () ad = f () ; f () = x : f () (x) > f () (x) (4) () = X I (X i ) ; B d ; (5) i= is the empirical probability measure o the -algebra of Borel sets B d. Chapter 6 of the cited book cotais a umber of argumets i favour of the Sche é selectio rule (3). However, the ext example demostrates that the favorizatio of the Sche é rule is problematic i some cases. s above, I () deotes the idicator fuctio. Example. Cosider f uiform o the closed iterval [0; ] R ad the correspodig ordered sample X : ; : : : ; X : : For the estimates of f we get so that f () = I(X : x X : + ) ad f () = I(X : x X : ) = (X : ; X : + j; ( ) = 0 f () = X : + X : ad f () = 0 f () ( ) = jx : + X : j > X : exceeds with probability the absolute deviatio ( ) = 0: f () Cosequetly the Sche é rule selects the estimate f () achievig the L -error R jf () fj = ( X : ) whereas the estimate f () achieves the error R jf () fj = X : ; so that is strictly better i the L -sese with the probability Pr(X : + X : < ) = =. The book of Devroye ad Lugosi (00) presets a systematic theory dealig with properties ad applicatios of the Sche é selectio f: This theory is based o Theorem 6. which compares the errors jf (0) fj = mi jf () fj; This fudametal theorem ca be give the form of the iequality jf fj 3 jf (0) fj + 4 f ( ) (6) jf () fj ad jf fj:

3 where is the Sche é set for f () ; f (). This iequality states that the selectio f ca achieve the error level 3 R jf (0) fj up to the uiversal error term appearig o the right. This iequality was applied ot oly i Chapters 7 7 of the Devroye-Lugosi book, but also i subsequet papers, amog them i Berliet, Biau ad Rouviere (005 a,b). The latter papers observed that the L -error criterio R jf gj for the estimates g beig formally probability desities is a special case of the more geeral -divergece criterio D (f; g) de ed for arbitrary probability desities f; g by the formula f D (f; g) = g : (7) g Here (t) is oegative ad covex i the domai t (0; ), strictly covex ad vaishig at the poit t = (for details about formula (7) ad the basic properties of -divergeces used below, see Csiszár (967a) or Liese ad Vajda (97, 006). The L -error is the -divergece for (t) = jt j, called total variatio ad deoted by V (f; g), i. e. V (f; g) = jf gj = sup f g : () B d Other examples are the squared Helliger distace the squared Le Cam distace H (f; g) = p f p g for (t) = p t, (9) LC (f; g) = ad the iformatio divergece I(f; g) = (f g) f + g f l f g for (t) = (t ) (t + ) ; (0) for (t) = t l t: () Natural motivatio for the alterative -divergece error criteria is the eed to work with estimates covereget i the topologies stroger tha that iduced by the total variatio (cf. Csiszár 967b ad Österreicher ad Vajda (003). This paper itroduces a ew motivatio achieved i Example 3 below by extedig the framework of Example through admittig o-uiform desities with uit supports o R: I this exteded settig Example 3 demostrates that for some desities f the alterative -divergece error criteria exhibit with positive probabilities optimality of the estimate g = f () at the same time whe the L -error exhibits the optimality of g = f (). Sice the optimality of the Sche é estimates f is perceived di eretly by di eret - divergece error criteria, it is importat to see whether or how the fudametal Devroye Lugosi iequality (6) ca be exteded from the total variatio criteria V (f; f ) = jf fj ad V (f; f (0) ) = 3 jf (0) fj ()

4 to the more geeral -divergece criteria D (f; f) = f f f This problem is solved i Sectio 3. ad D (f; f (0) ) = f (0) f f (0)! : (3) Sectio 4 itroduces a replacemet of the Sche é L -based selectio rule by a more geeral -divergece selectio rule ad solves a problem parallel to that of Sectio 3, amely whether or how the Devroye Lugosi iequality (6) ca be exteded to these estimates ad to the more geeral -divergece criteria. For the obvious reasos, i this paper the attetio is restricted to the estimates () which are a.s. probability desities themselves. Metric divergece criteria of errors Let us start with the followig basic properties of -divergeces eeded i the sequel: (i) The rage of values is 0 D (f; g) (0) + (0) (4) where (0); (0) are smooth extesios of (t), (t) = t (=t) to the poit t = 0. I (4) D (f; g) = 0 if ad oly if f = g a. s. ad D (f; g) = (0) + (0) if (for ite (0) + (0) if ad oly if) f?g (disjoit supports). (ii) The symmetry D (f; g) = D (g; f) for all f; g holds if ad oly if = fot the adjoit fuctio de ed i (i). (iii) The mootoicity property deals with relatios betwee -divergeces of the distributios () = D (; ) D (f; g) f; () = g; B d ad -divergeces of restrictios of these distributios o sub--algebras S B d of the Borel -algebra B d de ed by formula fs D (; js) = D (f S ; g S ) = g S for S-measurable versios f S ; g S of desities f; g. It states that the orderig D (f; gjs) D (; js) D (; ) D (f; g) (5) holds. If the equality i (5) takes place the we say that S preserves the -divergece D (f; g). It is kow (see e.g. Corollary.9 i Liese ad Vajda (97)) that if a sub- -algebra S is su ciet for the pair ff; gg the the equality takes place i (5), i.e. the 4 g S

5 su ciet S allways preserves the -divergece D (f; g). (iv) Fially, the spectral represetatio says that if a sub--algebra S B d is geerated by a ite or coutable B d -measurable partitio P of R d (spectrum of S, i symbols we write S = S(P)) the D (f; gjs) = X P R g : R f g : (6) Example. Cosider for every B d the partitio P = (; c ) of R d ad the P- geerated (or, more simply, -geerated) algebra S := S (; c ) B d (7) cosistig of the sets R d ; ; c ; ;: The the geeral spectral represetatio (6) implies V (f; gjs ) = X f g = f g : () Bf; c g B From () ad () we see that the fudametal Devroye Lugosi iequality (6) ca be give the form V (f; f) 3V (f; f (0) ) + V (; js ) (9) for the Sche é set of the estimates f () ad f () : If i () is the Sche é set (f; g) of f ad g the the absolute di erece o the right of () ca be replaced by the ordiary di erece. Moreover, it is see from () that the S preserves V (f; g) so that the formula () ca be exteded ad speci ed as follows V (f; gjs ) = V (f; g) = B f g : (0) The followig sectios exted the Devroye Lugosi theorem (6), or equivaletly (9), to the error criteria D(f; g) for probability desities f; g o (R d ; B d ) satisfyig similar metric properties as the total variatio criterio V (f; g) amely the re exivity the symmetry ad the triagle iequality D(f; g) = 0 if ad oly if f = g a. s.; () D(f; g) = D(g; f) for all f; g () D(f; g) D(f; h) + D(h; g) for all f; g; h: (3) We restrict ourselves to the metric divergece criteria de ed as powers D(f; g) = D (f; g) ; > 0 5

6 of -divergeces D (f; g) satisfyig ()-(3). These-divergeces achieve ite uppr bouds (0) + (0) = (0) < (4) (see (ii) for the equality ad Csiszár (967b) for the iteess). To provide a su cietly rich class of such criteria, let us itroduce the class of - divergeces D (f; g) = D (f; g); R: (5) Here the covex fuctios (t) are give i the domai t > 0 by the formula (t) = j j ( ) (t + ) (t = + ) (6) if ( ) 6= 0; ad by the correspodig limits 0 (t) = j t j =; (7) (t) = t l t + (t + ) l () t + otherwise. The subclass of these divergeces for 0 was proposed (with a di eret parametrizatio) by Österreicher ad Vajda (003). The extesio to < 0 was proposed recetly by Vajda (00). It is easy to verify for all f; g the formulas ad D 0 (f; g) = V (f; g) (total variatio, ()); (9) D (f; g) = H (f; g) (Helliger, (9)); (30) D (f; g) = 4 LC (f; g) (Le Cam, (0)) (3) D (f; g) = I (f; (f + g)=) + I(g; (f + g)=): (3) I the ppedix we demostrate that the powers D(f; g) := D (f; g) = maxf;g, R (33) of the divergeces (5) satisfy () (3), i. e. that they are metric divergece criteria. 3 Sche é selectio rule This sectio exteds the fudametal Devroye Lugosi iequality (6) for the Sche é estimates f from the total variatio error criteria () to the more geeral -divergece criteria (3). The ext example provides a motivatio for this extesio. 6

7 Example 3. Mai result of this sectio is the followig theorem. This theorem ad its proof refer to the lower ad upper error bouds L (V ) D (f; g) U (V ) (34) achieved for a give covex by the -divergeces D (f; g) o the class of desities f; g satisyig the total variatio coditio V (f; g) = V; 0 V : By Propositio.7 i Liese ad Vajda (97), the upper boud is for geeral give by the formula U (V ) = V c where c = (0) + (0) (cf. (4)) (35) ad the lower boud L (V ) is covex ad strictly icreasig i the variable V from the miimum L (0) = 0 to the maximum L () = (0) + (0) = c. Hece the strictly icreasig ad cocave iverse fuctio L (D) : [0; c ]! [0; ] (36) allways exists. For such that the powers D (f; g) are metrics o the space of desities f; g (4) implies c = (0) < : (37) Theorem. Let f be a estimated distributio o R d, f (0) ; f () ad f () the estimates cosidered i (), () with the correspodig Sche é set ad f the Sche é estimate resultig from the selectio rule (3). The for every metric divergece criterio D(f; g) = D (f; g) D (f; f) D f (0) ; f + c L D f (0) where L ad c are give by (36) ad (37) ; f = + f ( ) (3) Proof. Cosider the radom variables E ij = I f = f (i) ; f (0) = f (j) where X i;j= E ij = : (39) By the triagle iequality ad symmetry of D(f; g), ad by the de itio of E ii ; D (f ; f) D f (0) ; f + It su ces to prove that for i 6= j D f; f (0) Eij c X i;j= D f; f (0) Eij = D f (0) ; f + D f; f (0) E + D f; f (0) E : (40) L D f (0) ; f = + f ( ) E ij : (4) 7

8 We restrict ourselves to E : For E the proof is similar. By the de itio of E ad (35), (36), D f; f (0) E = D f () ; f () E c V f () ; f () E = c V f () ; f () js E c V f () ; fjs + V f () ; fjs E c V f () ; f + V () ; js + V ( ; js ) E c V f (0) ; f + V ( ; js ) E h c D f (0) ; f = + V ( ; js )i E : L where we bouded the sum of the total variatios i the third lie above by This completes the proof. V f () ; f + V () ; js + V ( ; js ) V f () ; f + V () ; js + V ( ; js ) V f () ; f + V ( ; js ) : The ext corollary reformulates the result of Theorem i a simpler but slightly weaker form. Corollary. For 0 <, uder the assumptios ad otatios of Theorem, D (f; f) c 3L D f (0) ; f = + 4 f ( ) (4) Proof. Clear from (3) by takig ito accout the iequalities D f (0) ; f U V f (0) ; f = c V f (0) ; f h c L D f (0) ; f i = obtaied from (34), (36) ad also the iequality (a) + (b) (a + b) obtaied from Jese s iequality for the cocave fuctio (x) = x. The ext example demostrates that Theorem geeralizes the Devroye ad Lugosi iequality (6). Example 4. Put D(f; g) = D 0 (f; g) = V (f; g)= (cf. (9)). The c 0 = 0 (0) = =; L 0 (V ) = U 0 (V ) = V ; 0 V ad L 0 (D) = D. Hece Theorem implies D 0 (f; f) D 0 f (0) ; f + D 0 f (0) ; f + f ( )

9 or, equivaletly, which coicides with (6) ad (9). V (f; f) 3V f (0) ; f + 4 f ( ) The ext example illustrates cotributios of Theorem ad its Corollary beyod the framework of Devroye ad Lugosi. Example 5. Put D(f; g) = D (f; g) =, i.e. take the Le Cam error criterio LC(f; g)= (cf. (3)). The parts (ii) ad (iii) of Theorem i the ppedix imply that c = =, U (V ) = V=6 ad L (V ) =! + V= + V= = (V=) V = : 4 Therefore L (D) = 4 p D ad for the Sche é selectio f of Devroye ad Lugosi we get from Theorem the relatio D (f; f) D f (0) ; f + 4D f (0) ; f = + f ( ) i. e. LC (f ; f) LC f (0) ; f + r LC f (0) ; f + jf ( )j where is the Sche é set of the iitial estimates f () ad f (). Corollary implies for the same f ad as before the alterative iequality i. e. D (f ; f) LC (f ; f) 3 (0) 4D f ; f + 4 s 3 LC f (0) ; f + 4 f ( ) = f ( ) : We see that the rate of covergece of the Le Cam error LC (f; f) to zero garateed by our theory for the Sche é estimate is strictly below the rate of the Le Cam error LC ; f achieved by the ideal estimate f (0). Oe ca deduce from the kow properties of the lower boud L (V ) ad its iverse L (D) that similar result ca be expected also for other divergece errors D (f; f) with strictly covex everywhere. f (0) 9

10 4 Divergece selectio rule This sectio is a cotiuatio of Sectio 3 where the estimatio errors are still evaluated by metric divergece criteria of the type D(f; g) = D (f; g) ; > 0 but the Sche é selectio (3) of Devroye ad Lugosi (00) is replaced by a more geeral selectio. Oe arrives quite aturally at such geeralizatio if he applies the same criteria also to the de itio of the optimal estimate f (0) ad its practical approximatio f. I other words, the geeralizatio cosists i the replacemet of the L -based de itio () by the divergece based de itio < f (0) = : f () f () if D f () otherwise: ; f < D f () ; f ad the L -based Sche é selectio rule (3) by the divergece selectio rule < f () f if D () ; js < D () ; js = : otherwise: f () This rule uses the empirical distributio de ed by (5), the estimates (i) (B) = f (i) ; B B d ; i f; g E of the probability distributio f; ad the sub--algebra S B d preservig the divergece D () ; (), i. e. satisfyig the equality D () ; () = D () ; () js (43) (44) (cf. (5)): (45) Next follows the mai result of this sectio where f is the estimated distributio ad S is sub--algebra preservig the divergece D(f () ; f () ) of the estimates f () ; f () i (43) ( e.g. the itersectio of all sub--algebras S B d preservig this divergece). Theorem. The estimate f resultig from the metric divergece selectio rule (44) satis es the iequality D (f ; f) 3D f (0) ; f + D (; js ) : (46) Proof. We ca start with the equality (40) valid i the preset situatio as well. It su ces to prove that for i 6= j D f; f (0) Eij D f (0) ; f + D ( ; js ) E ij 0

11 where E ij is de ed by (39) for f; f (0) give by (43), (44). Usig repeatedly the triagle iequality ad relatios (45) ad (5) we obtai D f ; f (0) E = D f (0) ; f () D f () E = D f () ; fjs + D f () ; f () js E ; fjs E D f () ; f + D () ; js + D ( ; js ) E D f () ; f + D () ; js + D ( ; js ) E D f () ; f + D () ; js + D ( ; js ) E D f () ; f + D ( ; js ) E = D f (0) ; f + D ( ; js ) E : I the same maer we obtai D f ; f (0) which completes the proof of (46). E D f (0) ; f + D ( ; js ) E The ext corollary presets a di eret expressio of the error term i (46). Corollary. The estimate f resultig from the selectio rule (44) employig a metric divergece D(f; g) = D (f; g) satis es the iequality D (f; f) 3D f (0) ; f + + c sup f (B) (47) BS where f (0) ; f ad S are the same as i Theorem ad c = (0) < : Proof. By Propositio.7 i Liese ad Vajda (97) ad (), D ( ; js ) c V ( ; js ) ad V ( ; js ) = sup S j() ()j ad the rest follows from Theorem ad (37).. s i the previous sectio, our rst step is to verify that Theorem geeralizes the Devroye Lugosi result (6). Example 6. Puttig D(f; g) = V (f; g) i Theorem ad usig the fact that by (0) the sub--algebra S preserves the total variatio V (f () ; f () ) of the estimates f () ; f () ; we get V (f; f) 3V f (0) ; f + V (; js ) : This coicides with the equivalet form (9) of the Devroye-Lugosi iequality (6). Most importat from the poit of view of applicatios is the complexity of the sub-algebra S B d which appears i the right-had error terms of (46) ad (47). It depeds

12 o the complexity of the used divergece error criterio D(f; g) ad the complexity of the estimates f () ; f (). I the previous example we have see that if D(f; g) is as simple as the total variatio V (f; g), the S is the simple -algebra S geerated by just oe set the Sche é set of the estimates f () ; f () irrespectively of how complex these estimates are. I the followig example we shall see the opposite extreme, amely simple estimates f () ; f () leadig to a simple -algebra S = S B geerated by just oe set B speci ed by these estimates, irrespectively of how complex the divergece criterio D(f; g) is. More precisely, B does ot deped o this criterio at all. Example 7. Let the sample X ; : : : ; X be govered by a bell-shaped desity f o R ad cosider the sample mea ad variace = X X i ad = i= X (X i ) ; i= ad also the followig cetral cover set Let f be estimated by Cauchy type desities ad where b = B = fx : jx j < 3 g: (4) f () (x) = [ + (x ) ] f () b (x) = I(x B ) [ + (x ) ] arctg 3 = arctg 3 : I (49) we used the fact that the coditio I(x B ) cuts away from f () (x) two tail probabilities of the size 3 3 f () dx = [ + x ] = + arctg( 3) = arctg 3 so that the f () -probability of the sample cetral cover set is /b. The likelihood ratio f () =f () is piecewise costat, f () (x) b if x f () (x) = B 0 otherwise; where b is the ormalizig factor used i (49). Therefore the sub--algebra S B = fr; B ; B; c ;g B geerated by the cetral cover set B of (4) is su ciet for the family ff () ; f () g. By what was said i Sectio, this meas that S B preserves for every (49)

13 covex the -divergece D (f () ; f () ). I other words, the sub--algebra S cosidered i Theorem ad Corollary is S B. Hece, by Theorem ad formula (6), for every metric divergece criterio D(f; g) = D (f; g) with > 0 D (f; f) 3D f (0) ; f + 4 X 3 (B) f R 5 B f : (50) B BfB ;B c g By Corollary, simpler but i geeral weaker variat of the result (50) is the iequality D (f; f) 3D f (0) ; f + + (0) f (B ) : (5) B Next follows a theorem which geeralizes ad makes precise the pheomea observed i the last example. Theorem 3. If the metric divergece criterio D(f; g) is a -divergece power with (t) strictly covex i the whole domai t > 0 the a sub--algebra S B d preserves D(f () ; f () ) i the sese D f () ; f () js = D f () ; f () if ad oly if S is su ciet for ff () ; f () g. Proof. Let D(f () ; f () ) = D (f () ; f () ) for some > 0. By the Corollary above, the metricity of D (f; g) implies D (f () ; f () ) (0) <. Hece, by Corollary.9 i Liese ad Vajda (97), the equality D (f () ; f () ) = D (f () ; f () js ) takes place if ad oly if S is su ciet. From this theorem we see that fuctios strictly covex everywhere de e the most complex divergece criteria for which the -algebra S is simple oly if the estimates f () ; f () are simple eough. Example 4 illustrated such situatio. 5 ppedix For practical applicatios of the results of Sectios 3 ad 4 oe eeds cocrete metric divergece criteria D(f; g) = D (f; g) with kow ad simple upper ad lower boud U (V ) ad L (V ) itroduced i (34). For this purpose he ca use the criteria from theclass >< whe < D(f; g) = D (f; g) () for () = maxf; g = >: whe > : itroduced i (5) (). The followig theorem summarizes basic relevat properties of the divergeces D (f; g). For the proof we refer to Vajda (00). (5) 3

14 Theorem. (i) D (f; g) are -divergeces with fuctios (t) strictly covex i the domai t > 0 whe 6= 0. (ii) The lower bouds of the divergeces D (f; g), R uder the costrait V (f; g) = V are give for 0 V by the formulas L (V ) = jj ( ) " + V = + # =! V (53) if ( ) 6= 0 ad otherwise by the correspodig limits L 0 (V ) = V=; L (V ) = + V l + V + V l V : (54) (iii) The upper bouds of the divergeces D (f; g), R uder the costrait V (f; g) = V are U (V ) = c V where c > 0 is cotiuous i the variable R; give by the formula whe < 0 >< jj + c = (0) = l whe = (55) >: whe 0; 6= : (iv) The powers D (f; g) () give i (5) are metrics i the space of probability desities f; g. Remark. Puttig = 0 i (iv) of Theorem oe obtais amog other the triagle iequality p D0 (f; g) p D 0 (f; h) + p D 0 (h; g) for the divergece D 0 (f; g) = V (f; g)= which is weaker tha the classical triagle iequality D 0 (f; g) D 0 (f; h) + D 0 (h; g) (56) obtaied by applyig the L -orm argumet to the total variatio V (f; g). Usig the cotiuity of the divergeces D (f; g) i the variable R we ca deduce from (56) that more sophisticated argumets tha those used to prove Theorem lead to stroger triagle iequalities also for the remaiig divergeces D (f; g), R, i particular for those with close to 0. ckowledgemet. This research was supported by the grats G µcr 0/07/3 ad MŠMT M05. 4

15 Refereces. Berliet, G. Biau ad L. Rouviere (005a): Parameter selectio i modi ed histogram estimates. Statistics 39, Berliet, G. Biau ad L. Rouviere (005b): Optimal L badwith selectio for variable kerel estimates. Statistics ad Probability Letters 74,.6-7. I. Csiszár (967a): Iformatio-type measures of di erece of probability distributios ad idirect observatios. Studia Sci. Math. Hugarica, I. Csiszár (967b): O topological properties of f-divergeces. Studia Sci. Math. Hugarica, L. Devroye ad G. Lugosi (00): Combiatorial Methods i Desity Estimatio. Spriger, Berli. F. Liese ad I. Vajda (97): Covex Statistical Distaces. Teuber, Leipzig. F. Liese ad I. Vajda (006): O divergeces ad iformatios i statistics ad iformatio theory. IEEE Tras. Iform. Theory 5, 0, F. Österreicher ad I. Vajda (003): ew class of metric divergeces o probability spaces ad its statistical applicatios.. Ist. Statist. Math. 55, I. Vajda (00): O metric f-divergeces of probability measures. Kyberetika (submitted). 5

Convergence of random variables. (telegram style notes) P.J.C. Spreij

Convergence of random variables. (telegram style notes) P.J.C. Spreij Covergece of radom variables (telegram style otes).j.c. Spreij this versio: September 6, 2005 Itroductio As we kow, radom variables are by defiitio measurable fuctios o some uderlyig measurable space