S6880 #6. Random Number Generation #2: Testing RNGs

Transcription

1 S6880 #6 Random Number Generation #2: Testing RNGs

2 1 Testing Uniform RNGs Theoretical Tests Outline 2 Empirical Tests for Independence Gap Tests Runs Test Coupon Collectors Test The Poker Test 3 Other Tests Tests for Uniformity Tests of Pairs, and k-tuples 4 Test Suites for RNGs Test Suites for RNGs (WMU) S6880 #6 S6880, Class Notes #6 2 / 20

3 Theoretical Tests no realizations of the generator needed, decide whether the period is large enough. Use spectral test. It requires the assumption that the generator is of full period. The method compares, for n-tuples (x 1,..., x n ) = x, the Fourier transformation of the probability function f (x) of theoretically uniform with respect to the Fourier transformation of the probability function g(x) of the generator. (WMU) S6880 #6 S6880, Class Notes #6 3 / 20

4 Example, Knuth (1969) for mixed-congruential generator. Denote the Fourier transforms of f and g by F and G, respectively. f U{0, 1,, m 1} n U(R). F = exp{ 2πi m (s x)}f (x), s = (s 1,..., s n ) x R { 1, if s = 0 = (0,, 0) = 0, elsewhere Can show that G(s) 0 for some s 0 which indicate the departures from random uniformity. Let A = {s : s 0, n 1 k=0 s jα k m 0}, ν n = min s, c n = πn/2 νn n (Knuth, 1969) s A )!m Larger value of c n generally indicate more randomness. Knuth recommended c 2, c 3, c for minimal acceptability of the generator ideally, c n > 1, n. ( n 2 (WMU) S6880 #6 S6880, Class Notes #6 4 / 20

5 1 Testing Uniform RNGs Theoretical Tests Outline 2 Empirical Tests for Independence Gap Tests Runs Test Coupon Collectors Test The Poker Test 3 Other Tests Tests for Uniformity Tests of Pairs, and k-tuples 4 Test Suites for RNGs Test Suites for RNGs (WMU) S6880 #6 S6880, Class Notes #6 5 / 20

6 Gap Tests Empirical tests need realizations of the generator. Fix 0 < α < β < 1 and consider the lengths of gaps for which U i (α, β). [ ) u 4 u 7 u 2 u 1 u 5 u 6 u 3 0 α β 1 Gap lengths are 0, 3, 0,... Theoretically, if U i s are independent, then the distribution of gap length should be geometric with parameter P(α < U i < β) = β α and that successive gap lengths are independent. Can use a chi-square test. (WMU) S6880 #6 S6880, Class Notes #6 6 / 20

7 Gaps Test Example U 1 =.563, U 2 =.624, U i 1 (U i 1 + U i 2 ). Then U 3 =.187, U 4 =.811,... > u <- c(.563,.624, numeric(48)) > for(i in 3:50)u[i] <- zapsmall(u[i-2]+u[i-1]) %% 1 > u [1] [11] [21] [31] [41] Take α =.4 and β =.6 then the gap lengths are 0, 7, 0, 1, 0, 0, 1, 0, 0, 3, 0, 5, 0, 4, 0, 5, 0, 0, 1, 0, 0, 4, 0, 5 (WMU) S6880 #6 S6880, Class Notes #6 7 / 20

8 Gaps Test Example, cont d k > 7 O n = 24 E Note: Gap length Geometric(.2) (since β α =.2) > round(c(dgeom(0:7,.2), pgeom(7,.2,lower=f))*24, 2) [1] (WMU) S6880 #6 S6880, Class Notes #6 8 / 20

9 Runs Test runs up = increasing subsequences runs down = decreasing subsequences Can consider runs up only runs down only runs up and down (WMU) S6880 #6 S6880, Class Notes #6 9 / 20

10 Runs Tests, cont d 1 Total number of runs, T = 5 in the example above. See nonparametric books for critical values for n < 25. For n 25, use approximation test with continuity correction: T N ( 2n 1, 3 16n 29) 90 2 Number of runs of each given length (Levene & Wolfowitz, 1944): #terms 1, runs up and down e.g. above : 1, 2, 1, 1, 3 length of a run = #terms, runs up only or runs down only e.g. above : 2, 1, 2, 4 (WMU) S6880 #6 S6880, Class Notes #6 10 / 20

11 Runs Tests, cont d In a sequence of length n: { #runs of length i, for i = 1, 2,, 5 n i = #runs of length 6 or greater, for i = 6. Denote E(R i ), the expected number of runs of length i, for i = 1,, 5; and E(R 6 ), the expected number of runs of length 6 or greater. It can be shown E(R i ) = { (2i 2 +6i+2)n (i+3)! 2i3 +6i 2 2i 8 (i+3)!, i = 1, 2,, 5 2(7n 41) 8!, i = 6. (WMU) S6880 #6 S6880, Class Notes #6 11 / 20

12 Runs Tests, cont d Let cov 1 (R) = [a ij ] = (symmetric) T = 1 n 6 6 [n i E(R i )][n j E(R j )]a ij i=1 j=1 Under H 0 of independence, T χ 2 (6) for large n (WMU) S6880 #6 S6880, Class Notes #6 12 / 20

13 Coupon Collectors Test Denote L, the length of sequences needed to collect all integers 0, 1,, k 1. Assume that each integer has probability 1 k of being collected (discrete uniform!) Then the length has distribution P(L = n) = 1 k 1 ( ) k 1 k n 1 ( 1) j (k 1 j) n 1, n = k, k + 1, j j=0 E(L) = k ( 1 k + 1 k ) 1 Var(L) = k 2( 1 k (k 1) ) 1 2 k ( 1 k + 1 k ) 1 from von Schelling (1954), American Mathematics Monthly. Observe sequence of digits until obtaining m complete sets and then use chi-square test. (WMU) S6880 #6 S6880, Class Notes #6 13 / 20

14 The Poker Test Consider various patterns analogous to poker hands in groups of five consecutive digits in a sample. For example, say partition in g = 5 different hands C 1 = { five of a kind }, C 2 = { full house, four of a kind }, C 3 = { three of a kind, two pairs }, C 4 = { one pair }, C 5 = { all different }. n groups of five consecutive digits were sampled, O i = observed # of times (groups) having hand C i. H 0 : digits i.i.d. discrete uniform. Then calculate Use χ 2 (4) test. E i = np(c i H 0 ) = expected #times having C i (WMU) S6880 #6 S6880, Class Notes #6 14 / 20

15 Other Tests include (but not limited to) permutation test digit frequency test serial correlation test (WMU) S6880 #6 S6880, Class Notes #6 15 / 20

16 1 Testing Uniform RNGs Theoretical Tests Outline 2 Empirical Tests for Independence Gap Tests Runs Test Coupon Collectors Test The Poker Test 3 Other Tests Tests for Uniformity Tests of Pairs, and k-tuples 4 Test Suites for RNGs Test Suites for RNGs (WMU) S6880 #6 S6880, Class Notes #6 16 / 20

17 Tests for Uniformity Kolmogorov-Smirov test: max F n (x) x using empirical C.D.F. F n. Chi-square test based on dividing (0,1) into intervals (WMU) S6880 #6 S6880, Class Notes #6 17 / 20

18 Tests of Uniformity of k-tuples {(U ki,, U ki +k 1)}, dividing [0, 1] k into a number of small regions. These types of tests require a large number of observations. (WMU) S6880 #6 S6880, Class Notes #6 18 / 20

19 1 Testing Uniform RNGs Theoretical Tests Outline 2 Empirical Tests for Independence Gap Tests Runs Test Coupon Collectors Test The Poker Test 3 Other Tests Tests for Uniformity Tests of Pairs, and k-tuples 4 Test Suites for RNGs Test Suites for RNGs (WMU) S6880 #6 S6880, Class Notes #6 19 / 20

20 Test Suites for Random Number Generators Knuth s TAOCP (The Art Of Computer Programming, Vol. 2) was the de facto standard test suite for a long time since George Marsaglia s Diehard test batteries, 1995 to present NIST RNG test suite, 2001 TestU01: L Ecuyer, P. & Simard, R. (2007) ACM Trans. on Math. Software 33(4) (WMU) S6880 #6 S6880, Class Notes #6 20 / 20

21 Test Suites for Random Number Generators Knuth s TAOCP (The Art Of Computer Programming, Vol. 2) was the de facto standard test suite for a long time since George Marsaglia s Diehard test batteries, 1995 to present NIST RNG test suite, 2001 TestU01: L Ecuyer, P. & Simard, R. (2007) ACM Trans. on Math. Software 33(4) (WMU) S6880 #6 S6880, Class Notes #6 20 / 20

22 Test Suites for Random Number Generators Knuth s TAOCP (The Art Of Computer Programming, Vol. 2) was the de facto standard test suite for a long time since George Marsaglia s Diehard test batteries, 1995 to present NIST RNG test suite, 2001 TestU01: L Ecuyer, P. & Simard, R. (2007) ACM Trans. on Math. Software 33(4) (WMU) S6880 #6 S6880, Class Notes #6 20 / 20

23 Test Suites for Random Number Generators Knuth s TAOCP (The Art Of Computer Programming, Vol. 2) was the de facto standard test suite for a long time since George Marsaglia s Diehard test batteries, 1995 to present NIST RNG test suite, 2001 TestU01: L Ecuyer, P. & Simard, R. (2007) ACM Trans. on Math. Software 33(4) (WMU) S6880 #6 S6880, Class Notes #6 20 / 20

24 Test Suites for Random Number Generators Knuth s TAOCP (The Art Of Computer Programming, Vol. 2) was the de facto standard test suite for a long time since George Marsaglia s Diehard test batteries, 1995 to present NIST RNG test suite, 2001 TestU01: L Ecuyer, P. & Simard, R. (2007) ACM Trans. on Math. Software 33(4) (WMU) S6880 #6 S6880, Class Notes #6 20 / 20