New Entropy Estimators with Smaller Root Mean Squared Error

Transcription

1 Joural of Moder Applied Statistical Methods Volume 4 Issue 2 Article New Etropy Estimators with Smaller Root Mea Squared Error Amer Ibrahim Al-Omari Al al-bayt Uiversity, Mafraq, Jorda, alomari_amer@yahoo.com Follow this ad additioal works at: Part of the Applied Statistics Commos, Social ad Behavioral Scieces Commos, ad the Statistical Theory Commos Recommeded Citatio Al-Omari, Amer Ibrahim (205) "New Etropy Estimators with Smaller Root Mea Squared Error," Joural of Moder Applied Statistical Methods: Vol. 4 : Iss. 2, Article 0. DOI: /jmasm/ Available at: This Regular Article is brought to you for free ad ope access by the Ope Access Jourals at DigitalCommos@WayeState. It has bee accepted for iclusio i Joural of Moder Applied Statistical Methods by a authorized editor of DigitalCommos@WayeState.

2 New Etropy Estimators with Smaller Root Mea Squared Error Cover Page Footote The author thaks the editor ad the referees for their helpful ad valuable commets that substatially improved this paper. This regular article is available i Joural of Moder Applied Statistical Methods: iss2/0

3 Joural of Moder Applied Statistical Methods November 205, Vol. 4, No. 2, Copyright 205 JMASM, Ic. ISSN New Etropy Estimators with Smaller Root Mea Squared Error Amer Ibrahim Al-Omari Al al-bayt Uiversity Mafraq, Jorda New estimators of etropy of cotiuous radom variable are suggested. The proposed estimators are ivestigated uder simple radom samplig (SRS), raked set samplig (RSS), ad double raked set samplig (DRSS) methods. The estimators are compared with Vasicek (976) ad Al-Omari (204) etropy estimators theoretically ad by simulatio i terms of the root mea squared error (RMSE) ad bias values. The results idicate that the suggested estimators have less RMSE ad bias values tha their competig estimators itroduced by Vasicek (976) ad Al-Omari (204). Keywords: Shao etropy; simple radom samplig, raked set samplig; double raked set samplig; root mea square error. Itroductio The raked set samplig was first suggested by McItyre (952) to estimate a mea of pasture ad forage yields. It is a cost efficiet samplig procedure alterative to the commoly used simple radom samplig scheme. The RSS is useful i situatios where the visual orderig of a set of uits ca be doe easily, but the exact measuremet of the uits is difficult or expesive. Let the variable of iterest X has a probability desity fuctio (pdf) g(x) ad a cumulative distributio fuctio (cdf) G(x), with mea μ ad variace σ 2. Let g (i:) (x) ad G (i:) (x) be the pdf ad cdf of the ith order statistic, X (i:), ( i ) of a radom sample of size. The pdf ad the cdf of X (i:), respectively, are give by i i g( i : ) x G x Gx g x, x i Amer Ibrahim Al-Omari is Faculty of Sciece, i the Departmet of Mathematics. at: alomari_amer@yahoo.com. 88

4 AMER IBRAHIM AL-OMARI ad j j G : x G x G x, x i j j, 2 i: x dx ad variace i: i: x i : g x dx i:. The raked set samplig method ca be describes as follows: with mea x g 2 Step. Radomly select 2 uits from the target populatio. Step 2. Allocate the 2 selected uits radomly ito sets, each of size. Step 3. Without yet kowig ay values for the variable of iterest, rak the uits withi each set with respect to a variable of iterest. This may be based o a persoal professioal judgmet or based o a cocomitat variable correlated with the variable of iterest. Step 4. The sample uits are selected for actual measuremet by icludig the ith smallest raked uit of the ith sample (i =, 2,, ). Step 5. Repeat Steps through 4 for r cycles to obtai a sample of size r for actual measuremet. It is of iterest to ote here that eve if 2 uits are selected from the populatio, but oly of them are measured for compariso with a simple radom samplig of the same size. Let the measured RSS uits are deoted by X (:), X 2(2:),, X (:). The i RSS estimator of the populatio mea is defied as XRSS Xii :. Takahasi ad Wakimoto (968) provided the mathematical theory of the RSS ad showed that 2 i: i: RSS 2 ii: g x g x,, Var X. i i i Al-Saleh ad Al-Kadiri (2000) suggested double raked set samplig (DRSS) method for estimatig the populatio mea to icrease the efficiecy of the estimators for fixed sample size. The DRSS method ca be described as: 2 89

5 NEW ENTROPY ESTIMATORS Step. Radomly choose 2 samples of size each from the target populatio. Step 2. Apply the RSS method described above o the 2 samples i Step. This step yields samples of size each. Step 3. Reapply the RSS method agai o the samples obtaied i Step 2 to obtai a sample of size from the DRSS data. The cycle ca be repeated r times if eeded to obtai a sample of size r uits. Let X be a cotiuous radom variable with probability desity fuctio gx ( ) ad cumulative distributio fuctio G(x). The etropy H [g(x)] of the radom variable is defied by Shao (948a, 948b) as H g x g x log g x dx. () The problem of etropy estimatio of a cotiuous radom variable is cosidered by may authors. Vasicek's (976) suggested a estimator of etropy based o spacig's as log dg p H g x dp, 0 dp (2) where the estimatio is foud by replacig the distributio fuctio G(x) by the empirical distributio fuctio G (x), ad usig the differece operator istead of d the differetial operator. The the derivative G p is estimated by a dp fuctio of the order statistics. Let X, X 2,, X be a simple radom sample of size from G(x) ad X () < X (2) < < X () be the order statistics of the sample. The Vasicek's (976) estimator of H [g(x)] is defied as HV log X X 2 (3) m i m i m i m 90

6 AMER IBRAHIM AL-OMARI where m < / 2 is a positive iteger kow as the widow size, X (i - m) = X () if P. i m, ad X (i + m) = X () if i m. He proved that HVm H g x as m, m, ad 0. Va Es (992) suggested a estimator of etropy based o spacigs as m HVEm X X i m i log m log m i m km k (4) ad proved the cosistecy ad the asymptotic ormality of the estimator uder some coditios. Ebrahimi, Pflughoeft, ad Soofi (994) adjusted the weights of Vasicek (976) estimator to have a smaller weights ad proposed a etropy estimator give by where HE X X log (5) m i m i m i im i, i m, m i 2, m i m, i, m i, m where X (i-m) = X () for i m ad X (i+m) = X () for i m. Ebrahimi et al. (994) showed by simulatio that their estimator has a smaller bias ad mea squared error tha Vasicek (976) estimator. Also, they proved that P. HEm H g x m m as,, 0. Noughabi ad Noughabi (203) suggested a ew estimator of etropy of a ukow cotiuous probability desity fuctio as 9

7 NEW ENTROPY ESTIMATORS HNN log s, m, (6) m i i where ad gˆ X i gˆ X, i m, i 2 m/ si, m, m i m, Xi m X im gˆ X, m i, i Xi X j k, where h is badwidth ad k is a kerel fuctio h h j satisfies k xdx. They proved that HNN P m H g x as, m, m / 0. Note that the kerel fuctio i Noughabi ad Noughabi (203) is selected to be the stadard ormal distributio ad the badwidth h is chose to be h =.06s -/5, where s is the sample stadard deviatio. To estimate the etropy H [g(x)] of a ukow cotiuous probability desity fuctio g(x), Noughabi ad Arghami (200) suggested a etropy estimator give by where HN X X log (7) m i m i m i cim, im, ci 2, m i m,, m i, ad X (i-m) = X () if i m ad X (i+m) = X () for i m. Correa (995) suggested a modified etropy estimator to have smaller mea squared error i the form 92

8 AMER IBRAHIM AL-OMARI HC m i im j i X X j i j i m log, im X X 2 j i j i m (8) im where X X i j. 2m j i m Al-Omari (204) suggested three estimators of etropy of a ukow cotiuous probability desity fuctio g(x) usig SRS, RSS, ad DRSS methods. Based o SRS his first suggested estimator is defied as AHESRS X X log (9) m i m i m i im where X (i-m) = X () for i m, X (i+m) = X () for i m, ad, i m, 2 i 2, m i m,, m i, 2 (0) The secod ad third estimators suggested by Al-Omari (204), based o RSS ad DRSS respectively, are give by ad AHERSS X X * * log () m i m i m i im AHEDRSS X X ** ** log (2) m i m i m i im * * * * where Xi m X () for i m ad X X im ( ) for i m, ad 93

9 NEW ENTROPY ESTIMATORS X X ** ** X im () for i m ad i m X for i m. ** ** ( ) For more about etropy estimators, see Choi, Kim, ad Sog (2004), Park, Park (2003), Goria, Leoeko, Mergel, ad Novi Iverardi (2005) ad Choi (2008). The remaiig part of this paper is orgaized as follows. The suggested etropy estimators are give i the sectio, Proposed Estimators. Next, a simulatio study is coducted to compare the ew estimators with their couterparts suggested by Vasicek (976) ad Al-Omari (204). Fially, some coclusios ad suggestios for further works. The proposed estimators The coefficiet of the etropy estimators i Ebrahimi et al. (994), Noughabi ad (0) (0) (0) Arghami (200), ad Al-Omari (204) are adjusted. Let X, X 2,..., X be a simple radom sample of size from G (x). Based o SRS the first suggested estimator is give by where SHESRS X X 0 0 log (3) m i m i m i im, i m, 4 i 2, m i m,, m i, 4 (4) (0) (0) (0) (0) Xi m X () for i m ad X X im ( ) for i m. Comparig (3) with (3), we have 94

10 AMER IBRAHIM AL-OMARI Let 0 0 SHESRSm log X i m X im i im 2 HVSRSm log (5) () () () (: ) (2: ) ( : ) HVSRS m i 2m 8 log 5 X, X,..., X be a RSS of size, Vasicek (976) etropy estimator usig RSS as cosidered by Mahdizadeh (202) is give by HVRSS log X X 2 i (6) m i m i m i m Based o the RSS uits etropy estimator is X (), (: ) X (), (2: ), X () ( : ), the secod suggested SHERSS X X log (7) m i m i m i im () () () () where i is defied as i (4), ad Xi m X () for i m ad X X im ( ) for i m. Comparig (6) with (7) to have Assume that () () SHERSSm log X i m X im i im 2 HVRSSm Log (8) HVRSS m i 2m 8 log 5 X, X,..., X is a DRSS sample of size. The third (2) (2) (2) (: ) (2: ) ( : ) suggested etropy estimator has the form i 95

11 NEW ENTROPY ESTIMATORS SHEDRSS X X 2 2 log (9) m i m i m i im (2) (2) (2) (2) where i is defied as i (4), ad Xi m X () for i m ad X X im ( ) for i m. Based o DRSS method Mahdizadeh (202) showed that Vasicek (976) estimator will be SHEDRSS log X X (20) m i m i m i m Comparig (9) with (20) to get 2 2 SHEDRSSm log X i m X im i im 2 HVDRSSm log (2) HVDRSS ME Remark : The etropy m i 2m 8 log 5 H f of a empirical maximum etropy desity ME f which is related to HVSRS ad SHESRS ca be computed followig Theil (980) as: i ME 2 2log 2 H f HVSRS log 2 SHESRS log SHESRS log 5 (22) ME Remark 2: If i (22), the H f SHESRS. I the followig two theorems, we compared the suggested estimators with Vasicek (967) ad Al-Omari (204). 96

12 AMER IBRAHIM AL-OMARI Theorem : The suggested estimators have the followig properties: (0) (0) (0) a) Let X, X 2,..., X be SRS of size, the SHESRS m > HVSRS m. () () () b) Let X (), X (2),..., X ( ) be a RSS of size, the SHERSS m > HVRSS m. (2) (2) (2) c) Let X (), X (2),..., X ( ) be a DRSS of size, the SHEDRSS m > HVDRSS m. Proof: The proof of (a), (b), (c), is straightforward by usig (5), (8), (2), respectively, where 2 m log I the followig theorem, we compare our suggested etropy estimators with their competitors i Al-Omari (204). Theorem 2: Based o the suggested estimators ad Al-Omari (204) etropy respectively, we have SHEj m > AHEj m, j = SRS, RSS, DRSS. Proof: Compare (9) with (3) based o SRS to obtai SHESRS m 2m 6 AHESRS m log, 5 ad sice 2 m log 6 0, the the case of SRS holds. Also, compare () with (7) 5 based o RSS, ad (2) with (9) usig DRSS to complete the proof of this theorem. The followig theorem proves the cosistecy of the suggested estimators SHESRS m, SHERSS m, ad SHEDRSS m. Theorem 3: Let Ω be the class of cotiuous desities with fiite etropies ad let X, X 2,, X be a radom sample from g Ω. If, m, m/ 0, the SHEj m, (j = SRS, RSS, DRSS) coverges i probability to H [g(x)]. Proof: Based o the simple radom samplig, from (5) we have 97

13 NEW ENTROPY ESTIMATORS SHESRS m 2m 8 HVSRS m log, 5 ad Vasicek (976) showed that HVSRS m coverges i probability to H [g(x)] ad sice 2 m log 8 coverges to zero as goes to ifiity, the we proved the 5 case of the SRS. Follow the same approach ad use (8) ad (2) to prove the theorem for RSS ad DRSS estimators, respectively. Methodology Simulatio study A simulatio was coducted to ivestigate the performace of the suggested etropy estimators with Vasicek (976) ad Al-Omari (204) etropy estimators usig samplig methods cosidered i this study. The compariso is based o the root mea squared errors (RMSEs) ad bias values of the estimators for 0000 samples geerated from the uiform, expoetial ad the stadard ormal distributios usig SRS, RSS ad DRSS methods. The selectio of the optimal values of the widow size of m for a give value is as yet a ope problem i the etropy estimatio. Therefore, we used the heuristic formula m 0.5 suggested by Wieczorkowski ad Grzegorzewski (999) to select m ad to compute the RMSEs of etropy estimators. I this study, we cosidered the sample ad widow sizes as give i Table. Table. The sample ad widow sizes cosidered i this simulatio Sample size = 0 = 20 = 30 Widow size m 5 m 0 m 5 Also, the performace of the RMSE of the suggested estimators for samples geerated from the uiform, expoetial ad stadard ormal distributios is evaluated based o the quatity Q N HVjm N 00, N SHEj m, AHEj m, j SRS, RSS, DRSS. HVj m 98

14 AMER IBRAHIM AL-OMARI The results are summarized i Tables 2-6. Also, we compared the suggested estimators of etropy with their competitors suggested by Al-Omari (204) ad the results preseted i Table 7 are take from Al-Omari (204). Based o these results observe the followig. The suggested etropy estimators usig SRS, RSS ad DRSS methods are more efficiet tha their competitors HV m based o the same method for all cases cosidered i this study. As a example, from Table 3, with = 0 ad m = 3 for the expoetial distributio with H [g(x)] = usig RSS method, the RMSE ad bias value of SHERSS m are ad compared to ad the RMSE ad bias of HVRSS m. The SHEDRSS m is superior to the other suggested estimators, SHERSS m ad SHESRS m uder the uiform, expoetial ad ormal distributios. From Table, cosider the case of = 20 ad m = 4 uder the uiform distributio whe H [g(x)] = 0, it ca be oted that the RMSE values of SHEDRSS m, SHERSS m, ad SHESRS m are , ad , respectively. The ature of the uderlyig distributio as well as the value of H [g(x)] affect o the efficiecy of the estimator usig the same method. As a example, the Q values with = 30 ad m = 3 SHERSS m for the uiform, expoetial, ad the stadard ormal distributios are , ad , respectively. However, the values of Q for the uiform distributio with H [g(x)] = 0 are SHE m superior to their couterparts for the expoetial ad ormal distributios. Fially, the suggested etropy estimators are foud to be more efficiet tha their competitors i Al-Omari (204) etropy estimators usig SRS, RSS ad DRSS schemes for the same widow ad sample sizes. For illustratio, assume that = 30 ad m = 8 whe the uderlyig distributio is the stadard ormal, from Table 4, the RMSE of SHERSS m is compared to which is the RMSE of AHERSS m as show i Table 7. 99

15 NEW ENTROPY ESTIMATORS Table 2. The Mote Carlo RMSEs ad bias values of HV m ad SHE m for the uiform distributio with H [g(x)] = 0. SRS HV m SHE m RSS Q SHE m HV m SHE m m Bias RMSE Bias RMSE Bias RMSE Bias RMSE Q SHE m Table 2 cotiued o ext page 00

16 AMER IBRAHIM AL-OMARI Table 3. The Mote Carlo RMSEs ad bias values of HV m ad SHE m for the expoetial distributio with H [g(x)] =. SRS HV m SHE m RSS Q SHE m HV m SHE m m Bias RMSE Bias RMSE Bias RMSE Bias RMSE Q SHE m Table 3 cotiued o ext page 0

17 NEW ENTROPY ESTIMATORS Table 4. The Mote Carlo RMSEs ad bias values of HV m ad SHE m for the stadard ormal distributio ad H [g(x)] =.49. SRS HV m SHE m RSS Q SHE m HV m SHE m m Bias RMSE Bias RMSE Bias RMSE Bias RMSE Q SHE m Table 4 cotiued o ext page 02

18 AMER IBRAHIM AL-OMARI Table 5. The Mote Carlo RMSEs ad bias values of HV m ad SHE m for the uiform distributio with H [g(x)] = 0 ad expoetial distributio with H [g(x)] = usig DRSS. SRS HV m m SHE SHEm RSS Q HV m m SHE Q SHEm m Bias RMSE Bias RMSE Bias RMSE Bias RMSE Table 5 cotiued o ext page 03

19 NEW ENTROPY ESTIMATORS

20 AMER IBRAHIM AL-OMARI Table 6. The Mote Carlo RMSEs ad bias values of HV m ad SHE m for the stadard ormal distributio ad H [g(x)] =.49. HV m SHE m m Bias RMSE Bias RMSE Q SHE m

21 NEW ENTROPY ESTIMATORS Table 7. The Mote Carlo RMSEs ad bias values of AHEj m, j = SRS, RSS, DRSS (Al-Omari, 204). AHESRS m Q AHESRS AHERSS m Q AHERSS AHEDRSS m Q AHEDRSS m Bias RMSE Bias RMSE Bias RMSE Uiform distributio with H [g(x)] = Expoetial distributio with H [g(x)] = Stadard ormal distributio with H [g(x)] =

22 AMER IBRAHIM AL-OMARI Coclusio Three etropy estimators are suggested usig SRS, RSS, ad DRSS methods. The cosistecy of these estimators is proved as well as some properties are reported. Based o theoretical ad umerical comparisos the suggested etropy estimators are more efficiet tha Vasicek (976) ad Al-Omari (204) etropy estimators. However, the suggested estimators of etropy i this paper ca be exteded by cosiderig other samplig methods such as the multistage RSS ad media RSS methods. Ackowledgemets The author thaks the referees for their helpful ad valuable commets that substatially improved this paper. Refereces Al-Omari, A. I. (204). Estimatio of etropy usig radom samplig. Joural of Computatio ad Applied Mathematics, 26, doi:0.06/j.cam Al-Saleh, M. F. & Al-Kadiri, M. A. (2000). Double raked set samplig. Statistics ad Probability Letters, 48(2), doi:0.06/s (99) Choi, B. (2008). Improvemet of goodess of fit test for ormal distributio based o etropy ad power compariso. Joural of Statistical Computatio ad Simulatio, 78(9), doi:0.080/ Choi, B., Kim, K., & Sog, S. H. (2004). Goodess of fit test for expoetiality based o Kullback-Leibler iformatio. Commuicatio i Statistics-Simulatio ad Computatio, 33(2), doi:0.08/sac Goria, M. N., Leoeko, N. N., Mergel, V. V., & Novi Iverardi, P. L. (2005). A ew class of radom vector etropy estimators ad its applicatios i testig statistical hypotheses. Joural of Noparametric Statistics, 7(3), doi:0.080/

23 NEW ENTROPY ESTIMATORS Correa, J. C. (995). A ew estimator of etropy. Commuicatio i Statistics-Theory Methods, 24(0), doi:0.080/ Ebrahimi, N., Pflughoeft, K., & Soofi, E. S. (994). Two measures of sample etropy. Statistics & Probability Letters, 20(3), doi:0.06/ (94) Mahdizadeh, M. (202). O the use of raked set samples i etropy based test of fit for the Laplace distributio. Revista Colombiaa de Estadística, 35(3), McItyre, G. A. (952). A method for ubiased selective samplig usig raked sets. Australia Joural of Agricultural Research, 3(4), doi:0.07/ar Noughabi, H. A. & Noughabi, R. A. (203). O the etropy estimators. Joural of Statistical Computatio ad Simulatio, 83(4), doi:0.080/ Noughabi, H. A. & Arghami, N. R. (200). A ew estimator of etropy. Joural of the Iraia Statistical Society, 9(), Park, S. & Park, D. (2003). Correctig momets for goodess of fit tests based o two etropy estimates. Joural of Statistical Computatio ad Simulatio, 73(9), doi:0.080/ Shao, C. E. (948a). A mathematical theory of commuicatios. Bell System Techical Joural 27(3), doi:0.002/j tb0338.x Shao, C. E. (948b). A mathematical theory of commuicatios. Bell System Techical Joural 27(4), doi:0.002/j tb0097.x Takahasi, K. & Wakimoto, K. (968). O the ubiased estimates of the populatio mea based o the sample stratified by meas of orderig. Aals of the Istitute of Statistical Mathematics, 20(), -3. doi:0.007/bf Theil, J. (980). The etropy of maximum etropy distributio. Ecoomics Letters, 5(2), doi:0.06/ (80) Va Es, B. (992). Estimatig fuctioals related to a desity by class of statistics based o spacigs. Scadiavia Joural of Statistics, 9(), Vasicek, O. (976). A test for ormality based o sample etropy. Joural of the Royal Statistical Society, B, 38,

24 AMER IBRAHIM AL-OMARI Wieczorkowski, R. & Grzegorzewsky, P. (999). Etropy estimators - improvemets ad comparisos. Commuicatio i Statistics-Simulatio ad Computatio, 28(2), doi:0.080/