arxiv: v1 [math.st] 12 Dec 2018

Transcription

1 DIVERGENCE MEASURES ESTIMATION AND ITS ASYMPTOTIC NORMALITY THEORY : DISCRETE CASE arxiv: v1 [ath.st] 1 Dec 018 Abstract. 1) BA AMADOU DIADIÉ AND 1,,4) LO GANE SAMB 1. Itroductio 1.1. Motivatios. I this paper, we study the covergece of epirical discrete probability disributios supported o a fiite set. Let throughoutthe followigx = {c 1,c,,c r }r be afiite coutablespace. The distributios probability o X are fiite diesioal vectors p i { } PX) = p = p c ) c X : p c 0, c X ad c X A divergece easure o PX) is a fuctio 1.1) D : PX)) p,q) R Dp,q) p c = 1 such that Dp,q) = 0 for ay p such that p,p) i the doai of applicatio of D. The fuctio D is ot ecessarily a applicatio. Ad if it is, it is ot always syetrical ad it does either have to be a etric. I lack of syetry, the followig ore geeral otatio is ore appropriate : 1.) D : P 1 X) P X) R p,q) Dp,q), where P 1 X) ad P X) are two failies of distributios probability o X, ot ecessarily the sae. To better explai our cocer, let us itroduce soe of the ost celebrated divergece easures.. We ay preset the followig divergece easure : let p,q) PX) PX) with X = {c 1,c,,c r }, p = p 1,,p r ) ad q = q 1,,q r ) two probabilities distributio o X. 1) The L -divergece easure : 1.3) D L p,q) = r p j q j ). j=1 1

2 1) BA AMADOU DIADIÉ AND 1,,4) LO GANE SAMB ) The faily of Reyi s divergece easures idexed by α 1, α > 0, kow uder the ae of Reyi-α : 1.4) D R,α p,q) = 1 r α 1 log p α jq 1 α j. j=1 3) The faily of Tsallis divergece easures idexed by α 1, α > 0, also kow uder the ae of Tsallis-α : 1.5) D T,α p,q) = 1 r p α α 1 jq 1 α j 1 ; j=1 4) The Kulback-Leibler divergece easure r 1.6) D KL p,q) = p j logq j /p j ). j=1 The latter, the Kullback-Leibler easure, ay be iterpreted as a liit case of both the Reyi s faily ad the Tsallis oe by lettig α 1. As well, for α ear 1, the Tsallis faily ay be see as derived fro D R,α p,q) based o the first order expasio of the logarith fuctio i the eighborhood of the uity. Fro this sall saple of divergece easures, we ay give the followig rearks. a) The L -divergece easure is both a applicatio ad a etric o P, where P is the class of probability easures o R d such that p j < +. j b) For both the Reyi ad the Tsallis failies, we ay have coputatio probles ad lack of syetry. Let give exaples. It is clear fro the very for of these divergece easures that we do ot have syetry, uless for the special case where α = 1/. Both failies are build o the followig fuctioal I α p,q) = j p α jq 1 α j 1.. Previous work ad ai cotributios. Our ai cotributio ay be suurized as follows, for data sapled fro oe or two ukow rado variables, we derive alost sure covergecy ad cetral liit theores for epirical φ divergeces 1.3. Overview of the paper.

3 DIVERGENCE MEASURES ESTIMATION AND ITS ASYMPTOTIC NORMALITY THEORY : DISCRETE CASE3. Distributio liit for epirical φ divergece.1. Notatio ad defiitios. Before we state the ai results we eed a few defiitios. Defie the epirical probability distributio geerated by i.i.d. rado variables X 1,,X fro the distributio probability p as.1) = p c ) c X, where p cj = 1 1 cj X i ) i.i.d. ad q is defied i the sae way by Y 1,,Y q that is.) q = q ) c c X, where q cj = 1 1 cj Y i ) Defiitio 1. The φ-divergece betwee the two probability distributios p ad q is give by.3) Jp,q) = φp j,q j ) where φ : [0,1] R is a easurable fuctio o which we will ake the appropriate coditios. The results o the fuctioal Jp,q) will lead to those o the particular cases of the Reyi, Tsallis, ad Kullback-Leibler easures... Mai resuls. Siceforafixedj {1,,r}, p cj has a bioial distributio with paraeters ad success probability p j, therefore.4) E[ p cj ] = p j ad V p cj ) = p j1 p j ). Furtherore, by the strog law of large ubers we have that p cj coverges alost surely ad hece i probability) to p j for every fixed j {1,,r}. By the theore cetral liit.5) i=1 i=1 p cj p j N0,1), as +, pj 1 p j ) where we use the sybol to deote covergece i distributio. Also for a fixed j {1,,r}, we have.6) q cj q j N0,1), as +. qj 1 q j ) More geerally sice is a saple of size fro a ultioial distributio with probabilities p, therefore see Lo et al. 016)) P p p) N r 0,Σp)) as + where N r 0,Σp)) is the ultioial covariace atrix give by p 1 1 p 1 ) p 1 p p 1 p r p p 1 p 1 p ) p p r ) Σp) = p r p 1 p r p p r 1 p r )

4 4 1) BA AMADOU DIADIÉ AND 1,,4) LO GANE SAMB 3. Asyptotic theory for φ-divergece easure 3.1. Boudess assuptio ad otatios. Defie Let D = {j {1,,,r} such that p j,q j κ > 0} Jp,q) = ϕp j,q j ) whereϕ : [0,1] Ris aesurablefuctio havigcotiuoussecodorderpartial derivatives defied as follows : φ 1) φ φ 1 s,t) = s,t), φ1) s,t) = s t s,t) ad φ ) 1 s,t) = φ φ) ss,t), s,t) = φ s,t), φ) t 1, s,t) = φ),1 s,t) = φ s t s,t). Set A 1,p = φ 1) 1 p j,q j ), A,q = φ 1) p j,q j ) A 3,q = φ 1) 1 q j,p j ), ad A 4,p = φ 1) q j,p j ). Based o.1) ad.), we will use the followig epirical φ-divergeces. J,q) = ϕ p cj,q j ), Jp, q ) = ϕp j, q cj ), ad J, q ) = ϕ p cj, qcj ). Set 3.1) a = sup p cj p j, ad c, = axa,b ). b = sup q cj q j, 3.. Stateets of the ai results. The first cocers the alost sure efficiecy of the estiators. Theore 1. Let X a fiite coutable space ad p,q) PX)), ad ad q be geerated by i.i.d. saples X 1,,X p ad Y 1,,Y q. The the followig asyptotic results hold for the epirical φ-divergeces a) Oe saple 3.) 3.3) + J,q) Jp,q) a Jp, q ) Jp,q) b A 1,p, a.s A,q, a.s

5 DIVERGENCE MEASURES ESTIMATION AND ITS ASYMPTOTIC NORMALITY THEORY : DISCRETE CASE5 3.4) b) Two saples J p li su, q ) Jp,q) A 1,p +A,q a.s,) +,+ ) c, where a, b ad c, are as i 3.1). The secod cocers the asyptotic orality of the estiators. Theore. Let V 1,p = p j 1 p j )φ 1) 1 p j,q j )) ad V,q = q j 1 q j )φ 1) p j,q j )). Uder the sae assuptios as i theore 1, the followig cetral liit theores hold for epirical φ-divergeces a) Oe saple : as +, 3.5) J p,q) Jp,q)) N 0,V 1,p ), 3.6) Jp, q ) Jp,q)) N 0,V,q ), b) Two saples : as + ad +, ) 1/ 3.7) J p V 1,p +V, q ) Jp,q)) N 0,1),q II - Direct extesios. Quite a few uber of divergece easures are ot syetrical. Aog these o-syetrical easures are soe of the ost iterestig oes. For such easures, estiators of the for J,q), Jp, q ) ad J, q ) are ot equal to Jg, ), J q,f) ad J q, ) respectively. I oe-sided tests, we have to decide whether the hypothesis p = q, for q kow ad fixed, is true based o data fro p. I such a case, we ay use the statistics oe of the statistics J,q) ad Jq, ) to perfor the tests. We ay have iforatio that allows us to prefer oe of the. If ot, it is better to use both of the, upo the fiiteess of both Jp,q) ad Jq,p), i a syetrized for as 3.8) J s) p,q) = Jp,q)+Jq,p). The sae situatio applies whe we face double-side tests, i.e., testig p = q fro data geerated by p et q. Asyptotic a.e. efficiecy. Theore 3. Uder the sae assuptios as i theore 1, the followig hold

6 6 1) BA AMADOU DIADIÉ AND 1,,4) LO GANE SAMB a) Oe saple : 3.9) 3.10) b) Two saples : 3.11),) +,+ ) Asyptotic Norality. Deote V 3,q = Js),q) J s) p,q) a Js) p, q ) J s) p,q) b Js), q ) J s) p,q) c, 1 A 1,p +A 4,p ) a.e., 1 A,q +A 3,q ) a.e., 1 A 1,p +A,q +A 3,q +A 4,p ), a.e.. q j 1 q j )φ 1) 1 q j,p j )) ad V 4,p = p j 1 p j )φ 1) q j,p j )). We have V 1,4,p = V 1,p +V 4,p ad V,3,q = V,q +V 3,q. Theore 4. Uder the sae assuptios as i theore 1, the followig hold a) Oe saple : as + 3.1) 3.13) V 1,4,p V,3,q ) J s),g) J s) p,q) N0,1), ) J s) p, q ) J s) p,q) N0,1). b) Two saples : as,) +,+ ) 3.14) V 1,4,p +V,3,q ) 1/ ) J s), q ) J s) p,q) N0,1). RearkTheproofoftheseextesioswillotbegivehere, sicetheyarestraight cosequeces of the ai results. As well, such cosideratios will ot be ade agai for particular easures for the sae reaso. 4. Particular Cases 4.1. Reyi ad Tsallis failies. These two failies are expressed through the fuctioal 4.1) I α p,q) = p α jq 1 α j, α > 0, α 1, which is of the for of the φ divergece easure with φx,y) = x α y 1 α, x,y) {p j,q j ), j D}.

7 DIVERGENCE MEASURES ESTIMATION AND ITS ASYMPTOTIC NORMALITY THEORY : DISCRETE CASE7 A- a)- The asyptotic behavior of the Tsallis divergece easure. Deote We have A T,α,1 =: α α 1 p j /q j ) α 1 ad A T,α, =: p j /q j ) α. Corollary 1. Uder the sae assuptios as i theore 1, ad for ay α > 0, α 1, the followig hold a) Oe saple : b) Two saples : D T,α,q) D T,α p,q) a D T,α p, q ) D T,α p,q) b A T,α,1 a.s, A T,α, a.s. D T,α p li su,g ) D T,α p,q) A T,α,1 +A T,α, a.s.,) +,+ ) c, Deote σt,α,1 p,q) = α α 1) p j p j /q j ) α p j p j /q j ) α 1 σt,α,p,q) = q j p j /q j ) α q j p j /q j ) α. We have Corollary. Uder the sae assuptios as i theore 1, ad for ay α > 0, α 1, the followig hold DT,α,q) D T,α p,q)) N0,σ T,α,1p,q), as +, ad as DT,α p, q ) D T,α p,q)) N0,σT,α, p,q) as +,,) +,+ ), ) 1/ σt,α, p,q)+σ T,α,1 p,q) D T,α, q ) D T,αp,q)) N0,1). As to the syetrized for T,α p,q) = D T,αp,q)+D T,α g,f), D s)

8 8 1) BA AMADOU DIADIÉ AND 1,,4) LO GANE SAMB we eed the suppleetary otatios: α A T,α,3 = j /p j ) α 1 q α 1, A T,α,4 = q j /p j ) α, σt,α,3 σ = α 1) q j q j /p j ) α q j q j /p j ) α 1 We have ad σ T,α,4 = p j q j /p j ) α p j q j /p j ) α ). Corollary 3. Let Assuptios?? ad?? hold ad let BD) be satisfied. The for ay α > 0, α 0, ad Deote We also have D s) T,α,g) D s) T,α p,q) A T,α,1 +A T,α,4 )/ =: A s) T,α,1 a a.s, D s) T,α f, q ) D s) T,α p,q) A T,α, +A T,α,3 )/ =: A s) T,α, b a.s, D s) T,α,g ) D s) T,α p,q),) +,+ ) c, σ T,α,1:4 p,q) = σ T,α,1 p,q)+σ T,α,4 p,q), σ T,α,:3 p,q) = σ T,α, p,q)+σ T,α,3 p,q). A s) T,α,1 +As) T,α, a.s. Corollary 4. Let Assuptios?? ad?? hold ad let BD) be satisfied. The for ay α > 0, α 0, we have ) D s) T,α,g) D s) T,α p,q) N0,σT,α,1:4p,q)), as +, D s) T,α f, q ) D s) T,α p,q) ) N0,σ T,α,:3p,q)), as +, ad as,) +,+ ), ) 1/ ) σt,α,1:4 p,q)+σ T,α,:3 p,q) D s) T,α,g ) D s) T,α p,q) N0,1). A-b)- The asyptotic behavior of the Reyi-α divergece easure. The treatet of the asyptotic behavior of the Reyi-α, α > 0, α 1 is obtaiedfroparta) a) byexpasiosadby theapplicatioofthedelta ethod. We first reark that D R,α p,q) = 1 α 1 logi αp,q)).

9 DIVERGENCE MEASURES ESTIMATION AND ITS ASYMPTOTIC NORMALITY THEORY : DISCRETE CASE9 Corollary 5. Uder the sae assuptios as i theore 1, ad for ay α > 0, α 1, the followig hold ad Deote We have, + D R,α,g) D R,α p,q) a D R,α f, q ) D R,α p,q) b σ R,α,1p,q) = σ T,α,1 p,q) I αp,q) D R,α,g ) D R,α p,q) c, A T,α,1 I α p,q) =: A R,α,1 a.s, A T,α, I α p,q) =: A R,α, a.s, A R,α,1 +A R,α, a.s. ad σr,α,p,q) = σ T,α, p,q) Iαp,q). Corollary 6. Let Assuptios?? ad?? hold ad let BD) be satisfied. The for ay α > 0, α 1, ) D R,α,g) D R,α p,q) N0,σR,α,1 p,q)), as +, ) D R,α f, q ) D R,α p,q) N0,σR,α,p,q)), as +, ad as,) +,+ ) ) 1/ ) D R,α,g ) D R,α p,q) N0,1). σ R,α, p,q)+σ R,α,1 p,q) As to the syetrized for D s) R,α p,q) = D R,αp,q) D R,α g,f), we eed the suppleetary otatios: A R,α,3 = A T,α,3 I α p,q), A R,α,4 = A T,α,4 I α p,q) σ R,α,3 p,q) = σ T,α,3 p,q) I α p,q) ad σ R,α,4 p,q) = σ T,α,4 p,q) I α p,q). Corollary 7. Let Assuptios?? ad?? hold ad let BD) be satisfied. The for ay α > 0, α 1, ad D s) R,α,g) D s) R,α p,q) A R,α,1 +A R,α,4 )/ =: A s) R,α,1 a, a.s. D s) R,α f, q ) D s) R,α p,q) A R,α, +A R,α,3 )/ =: A s) R,α, a, D s) R,α,g ) D s) R,α p,q),) +,+ ) c, A s) R,α,1 +As) R,α,

10 10 1) BA AMADOU DIADIÉ AND 1,,4) LO GANE SAMB Deote We also have σ R,α,1:4 p,q) = σ R,α,1 p,q)+σ R,α,4 p,q) σ R,α,:3 p,q) = σ R,α, p,q)+σ R,α,3 p,q) Corollary 8. Uder the sae assuptios as i theore 1, ad for ay α > 0, α 1, the followig hold D s) R,α,g) D s) R,α p,q) ) N0,σ R,α,1:4p,q)), as + ) D s) R,α f, q ) D s) R,α p,q) N0,σR,α,:3 p,q)), as + ad as,) +,+ ), ) 1/ ) σr,α,:3 p,q)+σ R,α,1:4 p,q) D s) R,α,g ) D s) R,α p,q) N0,1). B - Kulback-Leibler Measure 5. Proofs To keep the otatio siple, we itroduce the two followig otatios : therefore a = sup cj For ay j D, set cj = p cj p j ad cj q = q cj q j, j D,, b = sup q p j ) = cj cj q, ad c, = axa,b ). ad g q j ) = cj q We will use i the followig the ea value theore ad the delta ethod. For oe saple estiatio, defie J,q) = ϕ p cj,q j) For a fixed j D, we have 5.1) ϕ p cj,q j ) = ϕp j + cj,q j ) = ϕp j,q j )+ cj φ 1) 1 p j +θ 1,j cj,q j ) by applyig the ea value theore to the fuctio.) φ.),q j ) ad where θ 1,j is soe uber lyig betwee 0 ad 1. I the sequel, ay θ i,j, i = 1,, satisfies θ i,j < 1.

11 DIVERGENCE MEASURES ESTIMATION AND ITS ASYMPTOTIC NORMALITY THEORY : DISCRETE CASE11 By applyig agai the ea values theore to the fuctio.) φ 1) 1.),q j), we have φ 1) 1 p j +θ 1,j cj,q j ) = φ 1) 1 p j,q j )+θ 1,j cj φ ) 1 p j +θ,j cj,q j ) We ca write 5.1) as ϕ p cj,q j ) = ϕp j,q j )+ cj φ 1) 1 p j,q j )+θ 1,j cj ) φ ) 1 p j +θ,j cj,q j ) Now we have J,q) Jp,q) = cj φ 1) 1 p j,q j ) 5.) + θ 1,j cj ) φ ) 1 p j +θ,j cj,q j ), hece Therefore J,q) Jp,q) a J,q) Jp,q) a + a φ 1) 1 p j,q j ) A 1,p +a φ ) 1 p j +θ,j cj,q j ), φ ) 1 p j +θ,j cj,q j ). We kow that A 1,p < ad φ ) 1 p j +θ,j cj,q j ) φ ) 1 p j,q j ) < as. This proves 3.). Forula 3.3) is obtaied i a siilar way. We oly eed to adapt the result cocerig the first coordiate to the secod. The proof of 3.4) coes by splittig c φ p j, q ) φp cj j,q j ) ), ito the followig two ters φ p cj, q ) φp cj j,q j )) = φ p cj, q ) φp cj j, q )) cj + φp j, q cj ) φp j,q j )) I,1 +I,

12 1 1) BA AMADOU DIADIÉ AND 1,,4) LO GANE SAMB we have where J p,q) Jp,q)) = q p j )φ 1) 1 p j,q j )+ R, R, = θ 1,j cj ) φ ) 1 p j +θ,j cj,q j ). Fro.5) ad for a fixed j D, we have therefore Hece where q p j )φ 1) 1 p j,q j ) N q p j ) N0,p j 1 p j )) as +, 0,p j 1 p j )φ 1) 1 p j,q j )) ), as +. q p j )φ 1) 1 p j,q j ) N 0,V 1,p ), as +, V 1 = p j 1 p j )φ 1) 1 p j,q j )) sice X 1,X, is a sequece of i.i.d. replicatios of X. Let show that R, = 0 P 1). We have 5.3) R, a φ ) 1 p j +θ,j cj,q j ) Let show that a = o P 1) By the Bieayé-Tchebychev iequality, we have, for ay ǫ > 0 ad for ay j {1,,r} P ) ǫ p cj p j) ǫ) = P p cj p j p j1 p j ), 1/4 ǫ 1/ which iplies that a coverges i probability to 0 as +. Fially fro 5.3) we have R, P 0 as + which iplies J p,q) Jp,q)) N 0,V 1,p ), as +. This eds the proof of 3.5). The result 3.6) is obtaied by a syetry arguet by swappig the role of p ad q.

13 DIVERGENCE MEASURES ESTIMATION AND ITS ASYMPTOTIC NORMALITY THEORY : DISCRETE CASE13 Now, it reais to prove Forula 3.7) of the theore. Let us use bi-variate Taylor-Lagrage-Cauchy forula to get, J, q ) Jp,q) = cj φ 1) 1 p j,q j )+ cj q φ 1) p j,q j ) 1 cj ) φ ) 1 + cj cj q φ ) ) 1, + cj q ) φ ) u cj,v cj ). where u cj,v cj ) = p j +θ cj, q j +θ j cj q ). Thus we get J, q ) Jp,q) = 1 q p j )φ 1) 1 p j,q j )+ 1 g q j )φ 1) p j,q j )+R,, where R, is give by 1 cj p ) φ ) 1 + cj cj q φ ) 1, + cj q ) φ ) ) ) u cj,v cj. But we have where with q p j )φ 1) 1 p j,q j ) = N 1)+o P 1) q q j )φ 1) p j,q j ) = N )+o P 1), N 1) N 0,V 1,p ) ad N ) N 0,V,q ) V,q = q j 1 q j )φ 1) p j,q j )) Sice N 1) ad N ) are idepedet, we have 1 q p j )φ 1) 1 p j,q j )+ 1 q q j )φ 1) p j,q j ) = N 0, V 1,p + V ),q ) ) o P +o P. Therefore, we have J, q ) Jp,q) = N 0, V 1,p + V ) ),q o P +o P )+R,.

14 14 1) BA AMADOU DIADIÉ AND 1,,4) LO GANE SAMB Hece 1 J, q ) Jp,q)) = N 0,1) + o P 1 1 V 1,p + V,q V 1,p + V,q + o P 1 1 V 1,p + V,q + 1 R,. V 1,p + V,q That leads to J p V 1,p +V, q ) Jp,q)) = N 0,1)+o P 1)+ R,,,q V 1,p +V,q sice /V 1,p +V,q ) ad /V 1,p +V,q ) are bouded, ad the o P 1 ) 1 = o P = o P 1) V 1,p + V V,q 1,p +V,q ad o P 1 ) 1 = o P = o P 1). V 1,p + V,q V 1,p +V,q It reaistoprovethat V 1,p+V,q R,r = o P 1). But we haveby the cotiuity assuptios o φ ad o its partial derivatives ad by the uifor coverges of cj ad cj q to zero, that R,r V 1,p +V,q 1 a φ ) ) 1 p j,q j )+o1)) V 1,p +V,q + 1 b φ ) ) p j,q j )+o1)) V 1,p +V,q + 1 a b ) φ ) p j,q j )+o1)) V 1,p +V,q As previously, we have a = o P 1), b = o P 1) ad a b = o P 1). Fro there, the coclusio is iediate.

15 DIVERGENCE MEASURES ESTIMATION AND ITS ASYMPTOTIC NORMALITY THEORY : DISCRETE CASE15 Refereces Lo, G.S.016). Weak CovergeceIA). Sequeces of rado vectors. SPAS Books Series. Sait-Louis, Seegal - Calgary, Caada. Doi : /sbs/ Arxiv : ISBN : E-ail address: aadou-diadie.ba@edu.ugb.e E-ail address: gae-sab.lo@ugb.edu.s