BENFORD S LAW AND HOSMER-LEMESHOW TEST

Transcription

1 Journal of Mathematical Sciences: Avances an Applications Volume 4, 6, Paes Availale at DI: BFRD S LAW AD SMR-LMSW TST ZRA JASAK LB Banka Sarajevo Bosnia an erzeovina zoranjasak@nla Astract Benfor s law is loarithmic law for istriution of leain iits It s name y Frank Alert Benfor [] who formulate mathematical moel Before him, the same oservation was mae y Simon ewcom This law has chane usual preassumption of equal proaility of each iit on each position in numer Testin proceure y osmer-lemeshow test for Benfor s law is presente Such test can e, particularly, use to etect anomalies in samples of two or more partitions Introuction In article ote on the frequency of use of the ifferent iits in natural numers [], Simon ewcom asserte that the ten iits o not occur with equal frequency must e evient to any one makin much use of loarithmic tales, an noticin how much faster the first paes wear out than the last ones The first sinificant fiure is oftener than any other iit, an the frequency iminishes up to ewcom i not ive mathematical explanation of this oservation, just relative frequencies which were verifie later [] Mathematics Suject Classification: 6, 6Q Keywors an phrases: Benfor s law, osmer-lemeshow test, ata partitions, ecils Receive Septemer 8, 6 6 Scientific Avances Pulishers

2 58 ZRA JASAK The same phenomenon was re-iscovere y Benfor (38) [] who ave the mathematical formulation P lo [ D ] In next tale (Tale ), proailities for first leain iits are presente Tale Proailities of first leain iits Diits Proailities This law is extene to roups of leain an non-leain iits Practical prolem is how to test conformity to this law In this paper, test euce from osmer-lemeshow test is propose osmer-lemeshow Test Introuction osmer-lemeshow test is propose as a tool to asses fit of the loistic reression moel ([3], p 47-56) in case when population is ivie on two isjunctive supopulations, partitions Gooness-of-fit statistic C is otaine y calculatin the Pearson chisquare statistic form tale of oserve an estimate expecte frequencies A formula efinin the calculation of C is:

3 BFRD S LAW AD SMR-LMSW TST 5 ( n π ) C k n π ( k k π ) k k k k ere is numer of roups, n k is total numer of sujects in the k-th roup, c k enotes the numer of covariate patterns in the k-th ecile, k ck y j j is the numer of responses amon the c k covariate pattern an is the averae estimate proaility c m j π j π k k n j k Main preassumptions for this test are: Sample is ivie on two separate supopulations corresponin to cases of presence an asence of some property Proailities for covariance pattern, unique comination of values of preictor variales, are π k an πk for presence an asence of some property, respectively; their sum is for k-th ecile stimate of expecte frequencies are m j π j an mj ( π j ), respectively, for the cell corresponin to y an y rows same ere Sum of oserve an expecte frequencies for k-th ecile are the, k k k k,,, enote sample Y values, expecte Y values, sample Y values, expecte Y values, numer of oservations in roup, respectively

4 6 ZRA JASAK Central prolem of this test is how to make roups of values osmer an Lemeshow ([3], p 48) propose two strateies With the first metho, percentiles of risk, use of roups result in the first roup containin the n n sujects havin the smallest estimate proailities an the last roup containin the n sujects n havin the larest estimate proailities With the secon metho, use of roups results in cutpoints efine at the values k, k,,, an the roups contain all sujects whose estimate proaility etween ajacent cutpoints Preferre stratey, y authors, ([3], p 5) is to use eciles of risk Connection to Benfor s law Benfor s law is known as a stron tool for etectin anomalies There are numerous text concernin theoretical an practical issues of this law Usual approach in testin conformance to the Benfor s law is to consier ata as one sample, with no ifference either any element elons to any of two istinct supopulations nly criterion is leain or non-leain iit or roups of iits There is a lot of examples in which such approach is oth unpractical an has some eficiencies This is specially case in finance an similar areas Suppose we want to analyse some financial ata set (accountin, payments, ) consistin of input (creit) an output (eit) transactions It s common to mere those ata into one sample an conuct statistical test If we on t iffer them in some way we can loose possile important information aout anomalies on one, either creit or eit, sie We can test them separately ut in this case we o not have whole context Most of existin testin proceures, enerally, o not consier such ifference an treat them as they are mere in one roup For plausile investiation, it s important sometimes to make such ifference for analyse them in some context, for example, etectin anomalies, money launry etc

5 BFRD S LAW AD SMR-LMSW TST 6 ne of possiilities is to use osmer-lemeshow statistics, what is propose in this paper Accorin to Benfor s law, we ivie ata in G roups, corresponin to leain iits {,, 3, 4, 5, 6, 7, 8, } Suppose ata are ivie in two partitions (suroups), marke y Y (for creits) an Y (for eits), respectively For test we nee next values: j : numer of oservations in -th roup for partition j {, } Accorin to this is j j, j {, } : numer of oservations in -th roup Accorin to this is : Benfor s theoretical proaility for -th roup P [ D ] lo : expecte numer of elements in the -th roup iven Y, calculate y j ( Y j ) lo j j, {,, 3, 4, 5, 6, 7, 8, }, j {, } j j j, j {, }

6 ZRA JASAK 6 Specificities for this case are: Proaility for oth partitions in roup is < Sum of expecte numer of cases in roup must not e equal to the sum of oserve cases Appropriate statistic is [4]: ( ) ( ) G After such preassumptions for roups, we have ( ) ( ) ( ) ( ) ( ) ( ) () Another two ways to write this formula are: ; (a)

7 BFRD S LAW AD SMR-LMSW TST 63 () This statistic has χ istriution with G erees of freeom We can consier that this statistic is sum of two statistics, If we o not make ifference etween partitions, we have next ( ) (( ) ( ) ) ( ) [( ) ( )] ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( ) ( )( ) ( ) () This statistic has χ istriution with G erees of freeom, in this case is G Secon factor on the riht sie can e simplifie in next way:

8 ZRA JASAK 64 ( )( ) ( ) S ( ) ( ) ( ) ( ) ( ) Last factor on riht sie is harmonic mean of an Consierin inequality,,,,, > y x a y x a y x a from (), we have ( ) ( ) ( )( ) ( ),

9 BFRD S LAW AD SMR-LMSW TST 65 or ( ) ( ) First factor on the riht sie is ientical to This means that is more conservative than, althouh they have the same numer of erees of freeom 3 umerical xamples For emonstration, sample of financial payments is taken, consiste of 33,563 items, of which,44 are creits (input) an,3 are eits (output) In Tale, frequencies for leain iits for all three cateories are presente Tale Frequencies of leain iits in sample Diits All_Items Creits Deits,6 5,38 3,84 6,4 3,83,74 3 4,7,77,45 4 4,48,83,8 5 3,,76,43 6,366, ,87, , , Total 33,563,44,3

10 66 ZRA JASAK Graphical presentation of relative frequencies is on next iaram Diaram Relative frequencies of leain iits for creits, eits an all transactions, compare to Benfor s law The first thin we can note is that frequencies for input an output transactions are consieraly ifferent in comparison to frequencies if they are taken toether, if we make no ifference of cateory Frequencies of iit 4 are notaly ier in input an output transactions ut it s not so visile on whole sample level First step is first iit test Stanar are in Tale 3 χ test is conucte an results

11 BFRD S LAW AD SMR-LMSW TST 67 Tale 3 Chi-square test Diits j j ( ) ( ) j j j j j,7, , ,4 5,5 37, ,7 4, ,48 3,558 63, ,, , ,366,4635 4, ,87, , ,536,7683 3, ,54, , Value of χ statistic is 465 what is sinificantly ier than 7 ; 5 tale value χ 467 As the secon, osmer-lemeshow test is conucte Calculations are in Tale 4

12 Tale 4 Calculation of statistic Diits Total ( ) ( ),7 5,38 6,358 4, ,84 3, ,836 6,4 3,83 3, ,4,74,64443, ,7,77,65485,8587,45,53454, ,48,83, ,8644,8,3336, ,,76,6864,344, , ,366,54,476, ,87,86,38, ,536 6, , Total 33,564,44,44, ,3,3,37347

13 BFRD S LAW AD SMR-LMSW TST 6 Value of statistic is, , , This is sinificantly ier than value of stanar chi-square test conucte as a first step Suppose, for the moment, that oserve frequencies in this example in total are the same as expecte an that frequencies in partitions are as in Tale 5

14 Tale 5 Calculation of statistic Diits Total ( ) ( ),4 6,3 6,358 6, ,84 3, ,7586 5, 3,85 3, ,63,6,64443, ,3,6,65485, ,583,53454, ,53,7,587563,835,83,3336,7687 5,658,68,6864, ,47,433,476, ,46,3,38, ,77,,86684, , Total 33,564,44,44,453,3,3,35447

15 BFRD S LAW AD SMR-LMSW TST 7 Value of statistic is,453, ,564 5; Since critical value is χ 4 67 we nee to reject hypothesis that leain iits in this example follow Benfor s law n the other sie, we have that 553, 5447 This means that we shoul not reject hypothesis if we make separate tests on partitions At the same time, value of chi-square test for whole sample is 8 This means that we shoul not reject hypothesis for whole sample In the another wors, we have ifferent conclusions for the same sample, epenin on either we ivie sample or not At the same way, it s possile to have nonconformity on one of sies an conformity on oth sies 4 Discussion Main oal of this paper is to analyse possiility to use osmer- Lemeshow test to test conformity of sample to Benfor s law In this sense, roupin of values, y leain iits instea of ecils is propose By this, we can have roups for first leain iits, roups for leain two iits etc Avantae of this approach is that we can etect contriution of any roup in whole level of anomalies, even in case when test oes not etect anomalies on whole sample level Another way is to ivie interval [, ), k,,, k k in n suintervals, where n is aritrary chosen natural numer [4] This can e extene to iits or roups of iits on other positions Prolems can arrise with i frequencies in some roups ext step is to eneralize this proceure on m partitions, corresponin to values Y j, j {,,, m }, with G roups in each partition Assumptions in this case are:

16 7 ZRA JASAK j : numer of oservations in -th roup for partition j {,,, m } Accorin to this is m G j j, j {,,, m } j : numer of oservations in -th roup Accorin to j this is G m j j : Benfor s theoretical proaility for -th roup P [ D ] lo m j : expecte numer of elements in the -th roup iven Y, calculate y j j ( Y j ) lo j j, {,, 3, 4, 5, 6, 7, 8, }, j {,,, m } m j j G j j j, j {,,, m } Specificities for this case are: G Proaility for all partitions in roup is <

17 BFRD S LAW AD SMR-LMSW TST 73 Sum of expecte numer of cases in roup must not e equal to the sum of oserve cases m j m j j j In this case, statistic can e interprete as sum m G j j, j j j j This statistic has G m erees of freeom This means that G is the iest numer of partitions 5 Conclusion In this paper, use of osmer-lemeshow test for ooness-of-fit for Benfor s law is propose This means that roupin is accorin to leain iits is use instea of ecils Calculations show that, in this variant, test is more sensitive to anomalies than stanar χ test if it s possile to ivie sample in two or more partitions It s metho is easy to implement this test in xcel or similar prorams References [] Simon ewcom, ote on the frequency of use of the ifferent iits in natural numers, American Journal of Mathematics 4(/4) (88), 3-4 [] Frank A Benfor, The law of anomalous numers, Proceeins of the American Philosophical Society 78(4) (38), [3] Davi osmer an Stanley Lemeshow, Applie Loistic Reression, n ition, p 48 [4] Zoran Jasak, Benfor s law an arithmetic sequences, Journal of Mathematical Sciences: Avances an Applications 3 (5), -6 ISS [5]