The Newcomb-Benford Law: Theory and Applications

Transcription

1 The Newcomb-Benford Law: Theory and Applications Christoph Richard Universität Erlangen Bielefeld, richard 1 / 28

2 introduction history Simon Newcomb 1881 fractional part logarithm tables: only first pages worn out heavily (digit 1) '- s"1 + s"' - etc. is the mantissa of the logarithm of the limiting ratio. We thus reach the conclusion: The law of probability of the occurrence of numbers is such that all mantissa? of their logarithms are equally probable. In other words, every part of a table of anti-logarithms is entered with equal frequency. We thus find the required probabilities of occurrence in the case of the first two significant digits of a natural number to be: Dig. First Digit. Second Digit In the case of the third figure the probability will be nearly the same for each digit, and for the fourth and following ones the difference will be inappreciable. argument: natural random numbers X > 0 should obey: log 10 (X ) mod 1 uniformly distributed on [0, 1) this implies (frequency first digit k) = log 10 ((k + 1)/k) 2 / 28

3 introduction history THE LAW OF ANOMALOUS NUMBERS 553 The numbers counted in each group is given in the last column Frank Benford 1938 of Table (independently I. of Newcomb) TABLE I NATURAL NUMBERS PERCENTAGE OF TIMES THE 1 TO 9 ARE USED AS FIRST DIGITS IN NUMBERS, AS DETERMINED BY 20,229 OBSERVATIONS First Digit T itle - _ - _ - _ -~ount C A Rivers, Area B Population C Constants D Newspapers E Spec. Heat F Pressure G H.P. Lost H Mol. Wgt I Drainage J Atomic Wgt K n-, i/n, *.** L Design M Digest N Cost Data X-RayVolts P Am. League Q Black Body R Addresses S n', n2... n! T Death Rate Average Probable Error At the foot of each column of Table I the average percentage is given for each first digit, and also the probable error of the average. These averages can be better studied if the decimal point is moved two places to the left, making the sum of all the averages unity. The frequency of first l's is then seen to be 0.306, which is about equal to the common logarithm of 2. The frequency of first 2's is 0.185, which is slightly greater than the logarithm of 3/2. The difference 3 - is n a does not obey NBL (see below) data tuned through rounding (Diaconis, Freedman 79) publication after Bethe et al, The multiple scattering of electrons 3 / 28

4 introduction example example: areas of the 271 states of the world in km 2 rel. Haeufigkeit Ziffer (CIA world factbook 2007) 4 / 28

5 introduction example distribution appears to be scale-invariant and base-invariant: Flaechen in miles^2 Flaechen im Oktalsystem rel. Haeufigkeit rel. Haeufigkeit Ziffer Ziffer 5 / 28

6 introduction universality universality of the NBL-digit distribution random numbers from natural phenomena often satisfy: positive values values range over several orders of magnitude composed of many (nearly) independent factors not artificially processed (rounded, truncated, etc.) Then typically NBL arises. mathematical explanation? 6 / 28

7 digit functions digit functions D 10 (x) leading digit of x > 0 in decimal representation Graph der Ziffernfunktion D D(x) e 01 1e+00 1e+01 1e+02 1e+03 x logarithmisch D 10 on (0, ) Borel-measurable: D 1 10 ({k}) = n Z [k 10n, (k + 1) 10 n ) 7 / 28

8 definition NBL D b (x) leading digit of x > 0 in base b representation, b N, b > 2 Definition (Newcomb-Benford Law) Let X > 0 be a positive random variable. X satisfies the Newcomb-Benford law for base b (b-nbl), if P(D b (X ) = k) = log b (1 + 1/k) (k = 1,..., b 1) Which X satisfy b-nbl? For practically every sequence (x n ) n N of b-nbl random numbers the empirical digit frequencies converge to the b-nbl probability. 8 / 28

9 definition NBL almost sure convergence of empirical frequencies 1 n card ({1 i n : D 10(x i ) = 1}) P(D 10 (X ) = 1) = log 10 (2) Konvergenz der emp. Ziffernhaeufigkeiten Haeufigkeit(D(n)=1)fuer n=1..n N reason: almost sure convergence of the empirical distribution function (Glivenko-Cantelli theorem, strong law of large numbers) 9 / 28

10 strong NBL generalisation: Definition (strong Newcomb-Benford Law) Let X > 0 be a positive random variable. (i) X satisfies the strong Newcomb-Benford law for base b (b-snbl), if for all real numbers 1 α β < b we have P( n Z : α b n X < β b n ) = log b (β/α). (ii) X satisfies the strong Newcomb-Benford law (snbl), if X b-snbl for all b > 2. We have b-snbl b-nbl. (choose α := k, β := k + 1) natural b-nbl data appears to be even snbl. 10 / 28

11 strong NBL Theorem Let X > 0 be a positive random variable. Then the following are equivalent: (i) X is b-snbl. (ii) log b (X ) mod 1 is uniformly distributed on [0, 1). reason: Let 1 α β < b be arbitrary real numbers. Then P( n Z : α b n X < β b n ) = P ( n Z : log b (α) + n log b (X ) < log b (β) + n) = P (log b (X ) mod 1 [log b (α), log b (β))) But this characterises the distribution! 11 / 28

12 strong NBL remarks: examples of b-snbl random variables: X := b U+V, U unif[0, 1), im(v ) Z (cf. Leemis et al 2000) b 1 -snbl b 2 -NBL (X := 10 U, b 1 := 10, b 2 := 3) simple example of b-nbl, but not b-snbl? 12 / 28

13 scale invariance Let X b-nbl for b > 2. Then also Y := 2X b-nbl: P (D b (Y ) = 1) = P (D b (X ) = 2) + P (D b (X ) = 3) ( ) ( ) ( ) = log b + log 2 b = log 3 b 1 (analogously for other digits) = P (D b (X ) = 1) Definition (b-scale invariance) A positive random variable X > 0 is b-scale invariant, if for all c > 0 we have: for all real numbers 1 α β < b we have P( n : α b n X < β b n ) = P( n : α b n cx < β b n ). 13 / 28

14 scale invariance Theorem (b-scale invariance I) Let X > 0 be a positive random variable. Then the following are equivalent: (i) X is b-scale invariant. (ii) X is b-snbl. reason: Check characterisation of previous theorem. Let 1 α β < b be arbitrary real numbers. Then P( n : α b n cx < β b n ) = P (log b (cx ) mod 1 [log b (α), log b (β))). Use that uniform distribution is the only translation invariant distribution. 14 / 28

15 scale invariance other digit frequencies D (l) b (x) l leading digits of x > 0 in base b (e.g. D(3) Theorem (b-scale invariance II) For a positive random variable X > 0 are equivalent: (i) X is b-snbl. 10 (π) = (3, 1, 4)) (ii) For all l N we have: for all choices of k 1,..., k l «P D (l) b (X ) = (k1,..., k 1 l) = log b 1 +. (k 1 k l ) b (iii) For all c > 0 and all l N we have: for all choices of k 1,..., k l P D (l) b (cx ) = (k1,..., k l) = P D (l) b (X ) = (k1,..., k l). (k 1 k l ) b := k l + k l 1 b k 2b l 2 + k 1b l 1 15 / 28

16 scale invariance reason: (i) (ii): choose α := k 1 b k l b 1 l, β := α + b 1 l (ii) (i): Approximate α, β by numbers with finitely many digits. Use left continuity in α, β. (i) (iii): analogously 16 / 28

17 universality Theorem (universality of snbl) Let X 1,..., X n independent identically distributed positive random variables with density. We then have for all b > 2: for all real numbers 1 α β < b ( lim n P n : α b n ) n X i < β b n = log b (β/α). i=1 Hence n i=1 X i approximately satisfies snbl for n large enough. We have approximately scale- and (!) base-invariance. Statement remains true for special discrete random variables (see below) generalisations (see Miller, Nigrini 2008) 17 / 28

18 universality simulation standard normal distribution n = 1, 2, 3, 4 Histogram of D(x) Histogram of D(x) rel. Haeufigkeit rel. Haeufigkeit Ziffer Ziffer Histogram of D(x) Histogram of D(x) rel. Haeufigkeit rel. Haeufigkeit Ziffer Ziffer 18 / 28

19 universality proof rests on CLT for S 1 -random variables: Lemma (Lévy 1939) Let Y 1,..., Y n independent identically distributed S 1 -random variables, whose image is not contained in a regular polygon. Then the distribution of n i=1 Y i converges to the uniform distribution on S 1. Universality theorem follows with Y i := log b (X i ). Newcomb uses essentially the same argument! 19 / 28

20 universality almost-nbl X = b Y with V(Y ) large should approximately obey b-nbl, e.g.: Theorem (Dümbgen, Leuenberger 2008) Let X = b Y with Y N(µ, σ 2 ) and σ 1/6. Let h(m) = m!/m m. Then we have for all l N and for all choices of k 1,..., k l ( ) P D (l) b (X ) = (k 1,..., k l ) ) 1 1 log b (1 + 3h( 36σ2 ). (k 1 k l ) b For σ = 1 already 3h(36) ! other example: X Ex(λ) (Engel, Leuenberger 2003) 20 / 28

21 deterministic NBL numbers Diaconis 1977: snbl for sequences (x n ) n N of positive numbers equidistribution of (log b (x n )) n N modulo 1: 1 lim N N card ({log b(x 1 ) mod 1,..., log b (x N ) mod 1} [a, b]) = b a for every interval [a, b] [0, 1] Above theorems hold mutatis mutandis, if probabilities are replaced by limits of empirical densitites, e.g.: 1 lim N N card ({1 n N : D b(x n ) = k}) log b (1+1/k) (N ) examples: 2 n, n!, Fibonacci numbers counter-examples: n a, log b (n) 21 / 28

22 deterministic NBL numbers What s wrong with (n) n N? empirical frequencies 1 N card ({n {1,..., N} : D 10(n) = 1}) Dichten existieren nicht Haeufigkeit(D(n)=1) natural limit frequency? N 22 / 28

23 deterministic NBL numbers snbl in dynamical systems snbl near stable fixpoints, e.g.: Theorem (Berger, Bunimovic, Hill 2004) Let T (x) = αx(1 x) with α (0, 1). Then the following are equivalent: (i) Orbit (T (n) (x)) n N is b-snbl for all x 0 sufficiently close to 0. (ii) log b α / Q. results for non-autonomous systems (n!, e n, Fibonacci, Newton iteration,... ) results for differential equations 23 / 28

24 applications Goodness-of-fit tests tests of distribution hypotheses: example area of countries Newcomb-Benford law (discrete): χ 2 -test of NBL with 8 degrees of freedom value of χ 2 -statistic: , p-value: strong Newcomb-Benford law (continuous): Kolmogorov-Smirnov-test of log(x ) mod 1 for uniform distribution 226 largest values (without bindings), two-sided test value of KS-statistic: , p-value: In both examples NBL hypothesis consistent with data. specific goodness-of-fit tests for NBL (Nigrini 2000, Posch 2005) 24 / 28

25 applications Economy some examples for applications in economics: tax fraud detection book on special methods (Nigrini 2000) being used by tax offices implemented in bookkeeping software (e.g. Audit Commander) inflation rate unbereinigt vs. saisonal angepasst (Posch 2005) gross national product of different countries and regimes (Hellan, Nye 2002) 25 / 28

26 applications election analysis presidential election Iran 2009 candidates: Ahmadinejad, Mousavi, Karroubi, Rezaei analysis of Boudewijn F. Roukema (Torun) data of 366 districts analysis 1. digit: A: 1 too seldom, 2 too frequent; K: 7 too frequent Should NBL hold? (number of people per district) specialist for NB-analysis of election results: Walter R. Mebane (Michigan) distribution of second digit robust against small deviations of NBL deviations in 12 largest districts for K, R deviations to pre-election vote estimates for K,R suspicion: votes for K,R transferred to A 26 / 28

27 universality more universal distributions given the above assumptions on a data set, one frequently observes Newcomb-Benford law: proportions of numbers with first digit k approximately log 10 (1 + 1/k). Zipf s law: n-th smallest value approx. Cn α for n large (C > 0, α > 0). Pareto distribution: proportion of numbers with at least m digits approximately exponentially distributed for large m (normal distribution, if data concentrated about mean) 27 / 28

28 references some references T.Tao, Benford s law, Zipf s law, and the Pareto distribution P. Mörters, Benfords Gesetz über die Verteilung der Ziffern (2001) online-collection T.P. Hill, The first digit phenomenon, American Scientist 86 (1998), M.Nigrini, Digital Analysis Using Benford s Law: Test Statistics for Auditors, Global Audit Publications Iran election N.Hüngerbühler, Benfords Gesetz über führende Ziffern: Wie die Mathematik Steuersündern das Fürchten lehrt, EducETH 28 / 28