A SUMMARY ON ENTROPY STATISTICS

Transcription

1 A SUARY ON ENTROPY STATISTICS Esteban,.D. and orales, D. Departamento de Estadístca e I.O. Facultad de atemátcas Unversdad Complutense de adrd ADRID SPAIN). Abstract Wth the purpose to study as a whole the major part of entropy measures cted n the lterature, a mathematcal expresson s proposed n ths paper. In favour of ths mathematcal tool s the fact that most entropy measures can be obtaned as a partcular or a lmt case of the H ϕ1,ϕ2 -entropy functonal, and therefore, all those propertes whch are proved for the functonal are also true for ts partcularzatons. Entropy estmates are obtaned by replacng probabltes by relatve frequences and ther asymptotc dstrbutons are obtaned. To fnsh the asymptotc varance of many entropy statstcs are tabulated. Keywords and phrases: entropy, asymptotc dstrbuton. AS 1991 Subject Classfcaton: prmary 62B10; secondary 62E20. 1 INTRODUCTION Let X, β X, P ) P be an statstcal space, where X = {x 1,..., x }, = {P = p 1,..., p ) t /p 0 and =1 p = 1} and β X s the σ-feld of all the subsets of X. For any P, the -entropy s defned by the followng expresson: ) =1 v ϕ 1 p ) P ) = h =1, v ϕ 2 p ) where v > 0, = 1,...,, s the weght assocated to the element x of X. Furthermore we suppose that ϕ 1 : 0, 1) IR, ϕ 2 : 0, 1) IR and h: IR IR are any of the 3-uples of functons appearng n table 1. In table 1, v and functons hx), ϕ 1 x) and ϕ 2 x) are gven for the followng entropy measures: 1) Shannon 14], 2) Reny 12], 3) Aczel-Daróczy 1], 4) Aczel-Daróczy 1], 5) The research n ths paper was supported n part by DGICYT Grants N.PB and by Complutense Unversty grant N.PR161/ Ther fnancal support s gratefully acknowledged 1

2 Aczel-Daróczy 1], 6) Varma 19], 7) Varma 19], 8) Kapur 8], 9) Havdra-Charvat 7], 10) Armoto 2], 11) Sharma-ttal 15], 12) Sharma-ttal 15], 13) Taneja 18], 14) Sharma-Taneja 16], 15) Sharma-Taneja 17], 16) Ferrer 5], 17) Sant anna-taneja 13], 18) Sant anna-taneja 13], 19) Bels-Guasu 3] and Gl 6], 20) Pcard 10], 21) Pcard 10], 22) Pcard 10] and 23) Pcard 10]. TABLE 1 easure hx) ϕ 1 x) ϕ 2 x)v 1 x x log x x v 2 1 r) 1 log x x r x v 3 x x r log x x r v 4 s r) 1 log x x r x s v 5 1/s) arctan x x r sns log x) x r coss log x) v 6 m r) 1 log x x r m+1 x v 7 mm r)) 1 log x x r/m x v 8 1 t) 1 log x x t+s 1 x s v 9 1 s) 1 x 1) x s x v 10 t 1) 1 x t 1) x 1/t x v 11 1 s) 1 e x 1) s 1)x log x x v 12 1 s) 1 x s 1 r 1 1) x r x v 13 x x r log x x v 14 s r) 1 x x r x s x v 15 sn s) 1 x x r sns log x) x v ) log1 + λ) x 1 + λx) log1 + λx) x v λ λ 17 x x log 18 x snxs) ) snsx) 2 sns/2) snsx) 2 sns/2) log 2 sns/2) 19 x x log x x w 20 x log x 1 v 21 1 r) 1 log x x r 1 1 v 22 1 s) 1 e x 1) s 1) log x 1 v 23 1 s) 1 x r 1 s 1 1) x r 1 1 v ) x x v v Estmaton of populaton -entropes can be done by estmatng the probablty vector P wth the relatve frequency vector P = p 1,..., p ) t assocated to a smple random sample of sze n. In ths paper we show that the asymptotc dstrbuton of n P ) P )] s N 0, σ 2 ) where σ 2 = =1 t 2 p =1 ) 2, t p and we tabulate the values of t appearng n the expresson of ts asymptotc varance. On the bass of ths result, a confdence nterval for P ) can be gven and hypotheses about P ) can be 2

3 tested. 2 ASYPTOTIC DISTRIBUTION OF -STATISTICS If f C A) denotes that the real valued functon f has a contnuous dervatve of th order n the set A, then we obtan the followng result. Theorem 2.1. Suposse that h C 1 IR), ϕ 1 C 1 0, 1)), ϕ 2 C 1 0, 1)) and p > 0, = 1,...,. If the realatve frequency estmator of P = p 1,..., p ), P, s based on a smple random sample of sze n, then where n 1 2 H ϕ 1,ϕ 2 P ) L P )] n N 0, σ 2 ) ) 2 σ 2 = T t ΣT = t 2 p t p =1 =1 Σ = p δ j p j )),j=1,..., = dagp ) P P t T = t 1,..., t ) t and ) =1 t = h v ϕ 1 p ) =1 vϕ 1 p ) =1 v ϕ 2 p ) v ϕ 2 p ) =1 v ϕ 1 p ) v ϕ 2 p ) =1 2, = 1,..., v ϕ 2 p )) Proof. By the mean value theorem where P P 2 < P P 2. We conclude that P ) = P ) + =1 P ) p p p ), n H ϕ 1,ϕ 2 P ) P )] and nt t P P ) have asymptotcally the same dstrbuton c.f. Rao 11], p.385). Fnally applyng the Central Lmt Theorem, the results follows. In table 2, the expressons of the values t obtaned n Theorem 2.1 are gven. 3

4 TABLE 2 easure t Shannon 14] 1 + log p ) Reny 12] Aczel-Daróczy 1] r ] 1 1 r pr 1 p r =1 p r 1 =1 ) r log p ) p r rp r 1 =1 p r log p =1 ) ] 2 p r Aczel-Daróczy 1] s r) 1 =1 p r ) 1 =1 p s ) rp r 1 =1 p s ] =1 sps 1 p r Aczel-Daróczy 1] Varma 19] Varma 19] Kapur 8] Havdra-Charvat 7] Armoto 2] Sharma-ttal 15] Sharma-ttal 15] s p r 1 =1 p r sns log p ) 2 1 ) =1 p r coss log p ) 2 p r coss log p )] =1 r sns log p ) + s coss log p )) =1 p r coss log p ) p r 1 r coss log p ) s sns log p )) ] =1 p r sns log p ) r m + 1 ) 1 p r m p r m+1 m r =1 r ) 1 m 2 m r) pr/m) 1 p r/m =1 1 t) 1 ) =1 p t+s 1 1 =1 ) 1 p s t + s 1)p t+s 2 t 1) p 1/t) 1 =1 p s sps 1 1 s) 1 sp s 1 =1 p 1/t ) t 1 ] =1 p t+s log p ) exp {s 1) } =1 p log p r ) s r 1 r pr 1 p r r 1 =1 4

5 easure TABLE 2 cont.) t Taneja 18] Sharma-Taneja 16] p r r log p ) 1 rp r 1 s r ] sp s 1 Sharma-Taneja 17] 1 sn s pr 1 r sns log p ) + s coss log p )] Ferrer 5] 1 + log1 + λp )) Sant anna-taneja 13] Sant anna-taneja 13] ) snp s) log 2 sns/2) ] s cosp s) + p snp s) s cosp s) snp s) + log snp ) s) 2 sns/2) 2 sns/2) =1 ) 1 Bels-Guasu; Gl w log p + 1) w p + =1 =1 ) 2 w w p log p w p 3], 6] Pcard 10] Pcard 10] Pcard 10] v p 1 =1 ] 1 v v p r 2 =1 ] v p r 1 1 { v p 1 =1 ] } 1 =1 v p v exp s 1) =1 v Pcard 10] v p r 2 =1 ] 1 =1 =1 ] ) r s 1 v v p v s 1 The followng result gves a necessary and suffcent condton for σ 2 = 0. Proposton 2.1. Let S n = n 1/2 T t P P ) be the frst order term n the Taylor s expanson of P ) around P. Then, S n = 0 for all n wth probablty one f and only f σ 2 = 0 Proof. If S n = 0 a.s., then V S n ] = 0 for every n IN and therefore σ 2 = lm n V S n] = 0 5

6 On the other hand t s easy to check that V S n ] = σ 2, and therefore σ 2 = 0 mples S n = 0 a.s. Wth regard to Theorem 2.1, t s necessary to determne the asymptotc dstrbuton of the -statstcs when the asymptotc varance become zero. If A = a j ),j=1,..., wth a j = 2 H ϕ1,ϕ 2 P ), then we obtan the followng result p p j Theorem 2.2. Assume that h C 2 IR), ϕ 1 C 2 0, 1)), ϕ 2 C 2 0, 1)) and p > 0, = 1,..., n. If σ 2 = 0 and the relatve frequency estmator of P, P, s based on a random sample of sze n, then 2n P ) L P )] n =1 β χ 2 1, where the χ 2 1 s are ndependent and the β s are the egenvalues of AΣ. Proof. By proposton 2.1 and the mean value theorem P ) = P ) P 2 P ) t P ) ) p p j where P P 2 < P P 2. We conclude that,j=1..., P P ), 2n P ) P )] and n P P ) t A P P ) have asymptotcally the same dstrbuton c.f. Rao 11], p.385). Fnally, applyng the Central Lmt Theorem and well known facts about quadratc forms of normal varates, the result follows. A partcular but mportant case of Theorem 2.2 appears when P = U = 1/,..., 1/). Under ths asumpton, a ch-square asymptotc dstrbuton s obtaned. Theorem 2.3. Assume that h C 2 IR), ϕ 1 C 2 0, 1)) and ϕ 2 C 2 0, 1)). If P = U, v = v and P s based on a random sample of sze n, then where 2n θ) U)] b L n χ 2 1. ) b = h ϕ1 1/) ϕ2 1/)ϕ ϕ 2 1/) 11/) ϕ 1 1/)ϕ 21/) ] 2 ϕ 2 1/) 2] 1. 6

7 Proof. Followng the steps of the proof of Theorem 2.2, we get that 2n θ) U)] and n P U) t A P U) have asymptotcally the same dstrbuton, and therefore the results follows. In table 3, the expressons of the values b obtaned n Theorem 2.3 are gven. TABLE 3 7

8 easure Shannon 14] Reny 12] b 1 r Aczel-Daróczy 1] 2r 1 Aczel-Daróczy 1] rr 1) ss 1)]s r) 1 Aczel-Daróczy 1] 2r 1 Varma 19] m r 1 Varma 19] rm 3 Kapur 8] 1 t) 1 t + s 1)t + s 2) ss 1)] Havdra-Charvat 7] Armoto 2] Sharma-ttal 15] Sharma-ttal 15] s 1 s t 1 t 1 1 s r 1 s Taneja 18] 1 r 2r 1 rr 1) log ] Sharma-Taneja 16] s r) 1 rr 1) r ss 1) s ] Sharma-Taneja 17] Ferrer 5] 1 r sn s) 1 2rs s) coss log ) rr 1) s 2 ) sns log )] λ + λ) 1 Sant anna-taneja 13] s 1 2 cots 1 ) s 1 cscs 1 )) 2 ] Sant anna-taneja 13] 1 s 2 )) sns/) sns/) 1 + log s2 coss/)) 2 ] 2 sns/2) 2 sns/2) 2 sns/2) sns/) 3 STATISTICAL APPLICATIONS The prevous result gvng the asymptotc dstrbuton of -entropy statstcs, n a smple random samplng, can be used n varous settngs to construct confdence ntervals and to test statstcal hypotheses based on one or more samples. 8

9 a).- Test for a predcted value of the populaton entropy. To test H 0 : P ) = D 0 aganst H 1 : P ) D 0, we reject the null hypothess f n 1/2 P ) ) D 0 T a = σ > z α/2, where σ s obtaned from σ 2 n theorem 2.1 when p s replaced by p and z α s the 1 α)-quantle of the standard normal dstrbuton. In ths context an approxmate 1 α level confdence nterval for P ) s gven by P ) σz α/2 n 1/2, Hϕ 1,ϕ 2 P ) + σz α/2 n 1/2 ). Furthermore the mnmum sample sze gvng a maxmun error ε at a confdence level 1 α, s σ 2 z 2 ] α/2 n = + 1. ε 2 b).- Test for a common predcted value of r populaton entropes. To test H 0 : P 1 ) =... = P r ) = D 0, we reject the null hypotheses f T b = r n j P ) 2 j ) D 0 > χ 2 r,α j=1 σ 2 j where n j s the sze of the ndependent sample n the jth populaton, σ j s are obtaned from σ when p s replaced n theorem 2.1 by p j), = 1,...,, j = 1,..., r, and χ 2 r,α s the 1 α)-quantle of the ch square dstrbuton wth r degrees of freedom. In ths context an approxmate 1 α confdence nterval for the dfference of entropes correspondng to ndependent populatons s gven by P 1 ) P 2 ) ± z α/2 σ 2 1 n 1 + σ2 2 n 2. Furthermore, for n = n 1 = n 2, the mnmum sample sze gvng a maxmum error ε at a confdence level 1 α, s σ σ 2 ] 2 n = )z2 α/ ε 2 9

10 c).- Test for the equalty of r populaton entropes. To test H 0 : P 1 ) =... = P r ), we reject the null hypotheses f T c = r j=1 n j P ) 2 j ) H σ j 2 > χ 2 r 1,α, where and n j and σ j are defned above. H = rj=1 n j H ϕ 1,ϕ 2 Pj ) σ j 2 rj=1 n j σ j 2, d).- Test for dscrete unformty. To test H 0 : P = U, we reject the null hypothess f T d = 2n P ] ) P ) b > χ 2 1,α. Entropc test of unformty ncludng that consdered n Example 3.1 have been studed n Festauerová and Vajda 4]. Ths test s specally nterestng because t can be used to test for goodness-of-ft to a completely specfed dstrbuton. In ths sense, we are usng the dea that ann and Wald 9] suggested,.e. to take ntervals wth equal probabltes. To fnsh, we gve an example to llustrate ths procedure. Example 3.1. The followng sample was smulated from a Normal dstrbuton wth mean 2 and standard desvaton 1.1: To test for H 0 : Data fron N 2, 1.1), we take sx ntervals: 10

11 I 1 =, ) =, 0.933) I 2 = , ) = 0.933, 1.527) I 3 = , 2) = 1.527, 2) I 4 = 2, ) = 2, 2.473) I 5 = , ) = 2.473, 3.067) I 6 = , ) = 3.067, ) wth the property P N 2, 1.1) I ) = 1, = 1,..., 6. 6 We use the Shannon entropy statstc, so we reject the null hypothess f T = 2 n log H P ] ) > χ 2 1,0.05 Now, p 1 = 0.14, p 2 = 0.18, p 3 = 0.18, p 4 = 0.18, p 5 = 0.14, p 6 = 0.18, n = 50, H P ) = 6 =1 p log p = 1.785, T = and χ 2 5,0.05 = Furthermore, the classcal ch square statstc s S = n =1 p 1 ) 2 = Thus both procedures behaves smlarly and the concluson s that we cannot reject the null hypothess. References 1] J. Aczél and Z. Daróczy: Characterserung der entropen postver ordnung und der Shannonschen entrope. Act.ath.Acad.Sc.Hunger ), ] S. Armoto: Informaton-theoretcal consderatons on estmaton problems. Informaton and Control ), ]. Bels and S. Guasu: A quanttatve qualtatve measure of nformaton n cybernetcs systems. IEEE Trans. Inf. Th. IT ), ] J. Festauerová and I. Vajda: Testng system entropy and predcton error probablty. IEEE Trans. on Systems, an and Cybernetcs, ),

12 5] C. Ferrer: Hypoentropy and related heterogenety dvergence measures. Statstca ), ] P. Gl: eddas de ncertdumbre e nformacón en problemas de decsón estadístca. Revsta de la Real Academa de CC. Exactas, Físcas y Naturales de adrd, LXIX 1975), ] J. Havdra and F. Charvat: Concept of structural α-entropy. Kybernetka, ), ] J.N. Kapur: Generalzed entropy of order α and type β. The ath. Semnar, ), ] H.B. ann and A. Wald: On the choce of the number of class ntervals n the applcaton of the ch-squared test. Ann. ath. Statst., ), ] C.F. Pcard: The use of Informaton theory n the study of the dversty of bologcal populatons. Proc. Ffth Berk. Symp. IV 1979), ] C.R. Rao: Lnear statstcal nference and ts applcatons. 2nd. ed. John Wley. New York, 1973). 12] A. Reny: On the measures of entropy and nformaton. Proc. 4th Berkeley Symp. ath. Statst. and Prob., ), ] A.P. Sant anna and I.J. Taneja: Trgonometrc entropes, Jensen dfference dvergences and error bounds. Infor. Sc., ), ] C.E. Shannon: A mathematcal theory of communcaton. Bell. System Tech. J., ), ] B.D. Sharma and D.P. ttal: New non-addtve measures of relatve nformaton. J. Comb. Inform.& Syst. Sc., ), ] B.D. Sharma and I.J. Taneja, I.J.: Entropy of type α, β) and other generalzed measures n nformaton theory. etrka, ), ] B.D. Sharma and I.J. Taneja: Three generalzed addtve measures of entropy. Elect. Infor. Kybern, ),

13 18] I.J. Taneja: A study of generalzed measures n nformaton theory. Ph.D. Thess. Unversty of Delh, 1975). 19] R.S. Varma: Generalzatons of Reny s entropy of order α. J. ath. Sc., ),