Lehmer Random Number Generators: Introduction

Transcription

1 Lehmer Random Number Generators: Introduction Revised version of the slides based on the book Discrete-Event Simulation: a first course LL Leemis & SK Park Section(s) 21, 22 c 2006 Pearson Ed, Inc Discrete-Event Simulation Lehmer Random Number Generators: Introduction 1/48

2 Random Number Generators Historically there are three types of generators table look-up generators hardware generators algorithmic (software) generators Algorithmic generators are widely accepted because they meet all of the following criteria: randomness - output passes all reasonable statistical tests of randomness controllability - able to reproduce output, if desired portability - able to produce the same output on a wide variety of computer systems efficiency - fast, minimal computer resource requirements documentation - theoretically analyzed and extensively tested Discrete-Event Simulation Lehmer Random Number Generators: Introduction 2/48

3 Algorithmic Generators An ideal random number generator produces output such that each value in the interval 00 < u < 10 is equally likely to occur A good random number generator produces output that is (almost) statistically indistinguishable from an ideal generator We will discuss a good random number generator satisfying all our criteria Discrete-Event Simulation Lehmer Random Number Generators: Introduction 3/48

4 Conceptual Model Conceptual Model: Choose a large positive integer m This defines the set X m = {1, 2,, m 1} Fill a (conceptual) urn with the elements of X m Each time a random number u is needed, draw an integer x at random from the urn and let u = x/m Each draw simulates a sample of an independent identically distributed sequence of Uniform(0, 1) The possible values are 1/m, 2/m, (m 1)/m It is important that m be large so that the possible values are densely distributed between 00 and 10 Discrete-Event Simulation Lehmer Random Number Generators: Introduction 4/48

5 Conceptual Model 00 and 10 are impossible This is important for some random variates We would like to draw from the urn with replacement For practical reasons, we will draw without replacement If m is large and the number of draws is small relative to m, then the distinction becomes irrelevant Discrete-Event Simulation Lehmer Random Number Generators: Introduction 5/48

6 Lehmer s Algorithm Lehmer s algorithm (also called Linear Congruential Generator) for random number generation is defined in terms of two fixed parameters: modulus m, a fixed large prime integer multiplier a, a fixed integer in X m The integer sequence x 0, x 1, is defined by the iterative equation x i+1 = g(x i ) with g(x) = ax mod m x 0 X m is called the initial seed Discrete-Event Simulation Lehmer Random Number Generators: Introduction 6/48

7 Lehmer Generators Because of the mod operator, 0 g(x) < m However, 0 must not occur since g(0) = 0 Since m is prime, g(x) 0 if x X m If x 0 X m, then x i X m for all i 0 If the multiplier and prime modulus are chosen properly, a Lehmer generator is statistically indistinguishable from drawing from X m with replacement Note, there is nothing random about a Lehmer generator For this reason, it is called a pseudo-random generator Discrete-Event Simulation Lehmer Random Number Generators: Introduction 7/48

8 Intuitive Explanation ax/m m g(x) ax 0 x a m 2m 3m 4m 5m ax When ax is divided by m, the remainder is likely to be any value between 0 and m 1 Discrete-Event Simulation Lehmer Random Number Generators: Introduction 8/48

9 Parameter Considerations The choice of m is dictated, in part, by system considerations On a system with 32-bit 2 s complement integer arithmetic, is a natural choice With 16-bit or 64-bit integer representation, the choice is not obvious In general, we want to choose m to be the largest representable prime integer Given m, the choice of a must be made with great care Discrete-Event Simulation Lehmer Random Number Generators: Introduction 9/48

10 Simple example (Lehmer Generator) Effects of choosing different values for the multiplier Switch to Random WS 1 Click Here Discrete-Event Simulation Lehmer Random Number Generators: Introduction 10/48

11 Example - Period If m = 13 and a = 6 with x 0 = 1 then the sequence is 1, 6, 10, 8, 9, 2, 12, 7, 3, 5, 4, 11, 1, The ellipses indicate the sequence is periodic If m = 13 and a = 7 with x 0 = 1 then the sequence is 1, 7, 10, 5, 9, 11, 12, 6, 3, 8, 4, 2, 1 Because of the 12, 6, 3 and 8, 4, 2, 1 patterns, this sequence appears less random If m = 13 and a = 5 then 1, 5, 12, 8, 1, or 2, 10, 11, 3, 2, or 4, 7, 9, 6, 4, This less-than-full-period behavior is obviously undesirable Discrete-Event Simulation Lehmer Random Number Generators: Introduction 11/48

12 Central Issues Given a modulus m a Full Period sequence contains all the numbers in the set (0, m) before repeating itself For a chosen (a, m) pair, does the function g() generate a full-period sequence? If a full period sequence is generated, how random does the sequence appear to be? Can ax mod m be evaluated efficiently and correctly? Integer overflow can occur when computing ax Discrete-Event Simulation Lehmer Random Number Generators: Introduction 12/48

13 Full Period Considerations Theorem If x 0 X m and the sequence x 0, x 1, x 2 is produced by a Lehmer generator with multiplier a and (prime) modulus m then there is a positive integer p with p m 1 such that x 0, x 1, x 2 x p 1 are all different and x i+p = x i i = 0, 1, 2, That is, the sequence is periodic with fundamental period p In addition (m 1) mod p = 0 Discrete-Event Simulation Lehmer Random Number Generators: Introduction 13/48

14 Full Period Multipliers If we pick any initial seed x 0 X m and generate the sequence x 0, x 1, x 2, then x 0 will occur again Further x 0 will reappear at index p that is either m 1 or a divisor of m 1 The pattern will repeat forever We are interested in choosing full-period multipliers where p = m 1 Discrete-Event Simulation Lehmer Random Number Generators: Introduction 14/48

15 Example - Many full period sequences Full-period multipliers generate a virtual circular list with m 1 distinct elements (a, m) = (6, 13) (a, m) = (7, 13) Discrete-Event Simulation Lehmer Random Number Generators: Introduction 15/48

16 Finding Full Period Multipliers Algorithm p = 1; x = a; while (x!= 1) { p++; x = (a x)% m; /* beware of a x overflow */ } if(p == m 1) /* a is a full-period multiplier */ else /* a is not a full-period multiplier */ This algorithm corresponds to a naif implementation of Lehmer methods with x 0 = 1 It simply counts how many numbers are generated before getting back 1 It does not exploit any property of the method It is a slow-but-sure way to test for a full-period multiplier Discrete-Event Simulation Lehmer Random Number Generators: Introduction 16/48

17 Frequency of Full-Period Multipliers Given a prime modulus m, how many corresponding full-period multipliers are there? Theorem If m is prime and p 1, p 2,, p r are the (unique) prime factors of m 1 then the number of full-period multipliers in X m is (p 1 1)(p 2 1) (p r 1) p 1 p 2 p r (m 1) Discrete-Event Simulation Lehmer Random Number Generators: Introduction 17/48

18 Simple example (full period multipliers) Compare sequences generated using different multipliers Switch to Random WS 2 Click Here Discrete-Event Simulation Lehmer Random Number Generators: Introduction 18/48

19 Example If m = = 2, 147, 483, 647 then since the prime decomposition of m 1 is m 1 = = the number of full period multipliers is ( ) ( ) = 534, 600, Therefore, approximately 25% of the multipliers are full-period Discrete-Event Simulation Lehmer Random Number Generators: Introduction 19/48

20 Finding All Full-Period Multipliers Once one full-period multiplier has been found, then all others can be found in O(m) time Algorithm i = 1; x = a; while (x!= 1) { if(gcd(i, m 1) == 1) /* if i and (m 1) are relative prime a i mod m is a full-period multiplier*/ i++ x = (a * x) % m; /* beware a x overflow */ } Recall that two numbers are relative prime if they do not share any common divisor (except 1) Discrete-Event Simulation Lehmer Random Number Generators: Introduction 20/48

21 Finding All Full-Period Multipliers Theorem If a is any full-period multiplier relative to the prime modulus m then each of the integers a i mod m X m i = 1, 2, 3,, m 1 is also a full-period multiplier relative to m if and only if i and m 1 are relatively prime Discrete-Event Simulation Lehmer Random Number Generators: Introduction 21/48

22 Example If m = 13 then we know from previous examples there are 4 full period multipliers and one of them is a = 6 Then, since 1, 5, 7, and 11 are relatively prime to mod 13 = mod 13 = mod 13 = mod 13 = 11 Equivalently, if we knew a = 2 is a full-period multiplier 2 1 mod 13 = mod 13 = mod 13 = mod 13 = 7 Discrete-Event Simulation Lehmer Random Number Generators: Introduction 22/48

23 Simple example (finding full period multipliers) Simple Lehmer generator with a = 5 and m = 23 Switch to Random WS 3 Click Here Discrete-Event Simulation Lehmer Random Number Generators: Introduction 23/48

24 Example If m = we already observed that there are 534,600,000 integers relatively prime to m 1 The first few are i = 1, 5, 13, 17, 19 a = 7 is a full-period multiplier relative to m and therefore 7 1 mod = mod = mod = mod = mod = are full-period multipliers relative to m Discrete-Event Simulation Lehmer Random Number Generators: Introduction 24/48

25 Modulus Compatibility Definition The multiplier a is modulus-compatible with (prime modulus) m if and only if the remainder r = m mod a is less than the quotient q = m/a For implementation, efficiency and portability reasons, it is useful to choose the multiplier a to be both Full period Modulus-compatible Discrete-Event Simulation Lehmer Random Number Generators: Introduction 25/48

26 Other Multipliers and Considerations for m = there are multipliers a that are full period of these are modulus compatible (smaller than the m) In the next lecture, we will discuss statistical tests for these numbers, but a lot of research has already been done Nonrepresentative Subsequences: What if only 20 random numbers were needed and you chose seed x 0 = ? Resulting 20 random numbers: Discrete-Event Simulation Lehmer Random Number Generators: Introduction 26/48

27 Randomness Randomness properties are not less important that full period capabilities Choose the FPMC (Full Period Modulus Compatible) multiplier that gives most random sequence No universal definition of randomness exists In 2-space, (x 0, x 1 ), (x 1, x 2 ), (x 2, x 3 ), form a lattice structure For any integer k 2, the points (x 0, x 1,, x k 1 ), (x 1, x 2,, x k ), (x 2, x 3,, x k+1 ), form a lattice structure in k-space Numerically analyze uniformity of the lattice Discrete-Event Simulation Lehmer Random Number Generators: Introduction 27/48

28 Random Numbers Falling In The Planes (a, m) = (23, 401) (a, m) = (66, 401) Discrete-Event Simulation Lehmer Random Number Generators: Introduction 28/48

29 Scatter Plot Of 400 Pairs Initial seed x 0 = Initial seed x 0 = Discrete-Event Simulation Lehmer Random Number Generators: Introduction 29/48

30 Observations on Randomness In previous figure, no lattice structure is evident Appearance of randomness is an illusion If all m 1 = points were generated, lattice would be evident Herein lies distinction between ideal and good RNGs Discrete-Event Simulation Lehmer Random Number Generators: Introduction 30/48

31 Example Plotting all pairs (x i, x i+1 ) for m = would give a black square Any tiny square should appear (approximately) the same But if we were Zooming in we would see results similar to those depicted in the following figure obtained for multipliers a = and a = on the next slide Discrete-Event Simulation Lehmer Random Number Generators: Introduction 31/48

32 Scatter Plots for m = Multiplier a = Multiplier a = Further justification for using a = over a = Discrete-Event Simulation Lehmer Random Number Generators: Introduction 32/48

33 Tests for Randomness Pseudo-random number generators are intrinsically non-random, however we accept the fact that they apppear random, if the generation method (parameters of the Lehmer Generator) are not explicitly known Statistical tests can be performed on sequences of generated random numbers to quantify their randomness The more statistical tests are passed by a generator the better its quality is assumed The chi-square test is commonly used to quantify the randomness of long, but finite (and in any case much shorter than the period of the generator) sequences of (generated) random numbers Discrete-Event Simulation Lehmer Random Number Generators: Introduction 33/48

34 Simple example (preliminary observations on randomness) Filtering the sequence of numbers produced by Lehmer generator to appear more random Switch to Random WS 4 Click Here Discrete-Event Simulation Lehmer Random Number Generators: Introduction 34/48

35 Chi-square statistic Let x be a random variable assuming values in interval X = (0, m) Suppose that the possible values of x be partitioned into k adjacent intervals (a 0, a 1 ], (a 1, a 2 ],, (a k 1, a k ) to form a new (discrete) random variable y that assumes values in the set Y k = {0, 1,, k 1} Based on a large number n of independent observations of y, for each (possible) value y j, let N j be the number of times y j occurs η j = np j be the expected number of times y j occurs, where p j is the probability of an outcome (observation) in (a j 1, a j ] 1 Then the resulting non-negative quantity V = k j=1 (N j η j ) 2 η j is known as a chi-square statistic 1 The fact that (a k 1, a k ) is open is not relevant for practical purposes Discrete-Event Simulation Lehmer Random Number Generators: Introduction 35/48

36 Chi-square test If the sample of the random variable x, represented by the sequence X = {x 1, x 2,, x n } is truly an iid random variate sample from the desired distribution and if n/k is sufficiently large (> 10) then the statistic V is approximately a X 2 random variate with n 1 degrees of freedom In these conditions, let v1 = χ2 (k 1, α/2) be the percentile such that Pr{X 2 (n 1) χ 2 (k 1, α/2)} = α/2 v2 = χ2 (k 1, 1 α/2) be the percentile such that Pr{X 2 (k 1) χ 2 (k 1, 1 α)} = 1 α/2 then Pr{v1 < V v 2 } = 1 α, ie, the Chi-square statistic V belongs to the interval (v1, v 2 ) with confidence level (1 α) Typically the confidence level α is chosen to be equal to 005 Discrete-Event Simulation Lehmer Random Number Generators: Introduction 36/48

37 Chi-square percentile table df χ χ 2 01 χ χ 2 05 χ 2 10 χ 2 90 χ 2 95 χ χ 2 99 χ Discrete-Event Simulation Lehmer Random Number Generators: Introduction 37/48

38 Chi-square test interpretation With the help of the Chi-square percentile table it is possible to accept the test hypothesis using the following criteria: A value of V is considered acceptable with 95% confidence if it falls between the columns 2 χ 025 and χ 975 of the table A value of V is refused if it falls before column χ 005 or after column χ 995 A value of V is considered dubious otherwise More detailed distinctions can be made according to the following criteria A value of V is considered dubiousif it falls between the columns χ 005 and χ 001 or between the columns χ 99 and χ 995 A value of V is considered quasi-dubious if it falls between the columns χ 01 and χ 025 or between the columns χ 975 and χ 99 A test of this type is called Two-Tailed test 2 columns report values for different degrees of freedom Discrete-Event Simulation Lehmer Random Number Generators: Introduction 38/48

39 Chi Square Density function 2 025(9)= (9)= (9)= (9)=1902 Discrete-Event Simulation Lehmer Random Number Generators: Introduction 39/48

40 Empirical tests of randomness based on the Chi-square test Different tests of randomness can be performed on the sequence produced by a random number generator using the Chi-square test The most common ones are: Uniformity Test - Is the histogram flat Bivariate Uniformity Test - Serial test Gap Test of independence Many other tests have been proposed to check the properties of the Random Number Generator using other points of view (see Knuth DE, The Art of Computer Programming, Vol2: Seminumerical Algorithms ) Discrete-Event Simulation Lehmer Random Number Generators: Introduction 40/48

41 Uniformity Test Divide the interval (0, 1) into k subintervals of equal length Generate U 1, U 2,,U n It is recommended to choose k 100 and n/k 5 Let N j be the number of the n U i s in the j-th subinterval Then V = k n k j=1 ( N j n ) 2 k is approximately a X 2 random variate with k 1 degrees of freedom Discrete-Event Simulation Lehmer Random Number Generators: Introduction 41/48

42 Simple example (uniformity test) Application of the Chi-square test to check uniformity Sensitivity of the test with respect to length of sequence Switch to Random WS 5-6 Click Here Discrete-Event Simulation Lehmer Random Number Generators: Introduction 42/48

43 Serial Test 2-dimensional version of the Uniformity test to assess the independence between successive observations Divide the interval (0, 1) into k subintervals of equal length Generate U 1, U 2,,U 2n If the U i s are really iid, then the non-overlapping pairs (U 1, U 2 ), (U 3, U 4 ),, (U 2n 1, U 2n ) are iid random vectors uniformly distributed in the square (0, 1) 2 Count the number of outcomes that fall in each sub-square (ie, let N ij be the number of pairs in the (i, j)-th sub-square) Then V = k2 n k i=1 j=1 k (N ij n ) 2 k 2 is approximately a X 2 random variate with k 2 1 degrees of freedom Obviously this test can be generalized to higher dimensions Discrete-Event Simulation Lehmer Random Number Generators: Introduction 43/48

44 Gap Test The Gap test checks the randomness of the generator with a non-local view Define two values α and β (α < β) within the interval (0, 1) Let q, equal to (β α), be the probability that a random number U is in the interval (α, β) Generate U 1, U 2,,U 2n Transform the {U i } sequence into the {I i } sequence using the following definition I i = { 1 if Ui (α, β) 0 Otherwise Identify consecutive subsequences I j, I j+1,, I j+r, I j+(r+1) such that I j and I j+(r+1) = 1, but the other elements in the subsequence are = 0 This is a gap of length r Discrete-Event Simulation Lehmer Random Number Generators: Introduction 44/48

45 Gap Test (cont) Perform a chi-square test using the different (possible) lengths of the gaps as categories and the probabilities as follows p r = q(1 q) r, r = 0, 1, 2, The size of a gap may grow up to, thus we consolidate the gaps, choosing and arbitrary upper value k and counting together all the gaps of length (k 1) or greater The probability of this consolidated event is Pr{r k 1} = (1 q) k 1 Since the gaps have a geometric distribution with parameter (1 δ), the expected gap length is E[r] = 1 q q Small values of q = (α β) yield gaps of large size Discrete-Event Simulation Lehmer Random Number Generators: Introduction 45/48

46 Gap Test (cont) The choice of k and of q have a large impact on the length of the sequence of random numbers that we use for the test the larger is the value of k the less local is the randomness test large values of k make the probability of having gaps of length larger of k 2 small small values of q makes the test more focused the length of the sequence of random numbers n may be evaluated in the following manner m n E[r] Pr{r k 1} = m q(1 q) k 2 where m is the minimum number of expected gaps of length (k 1) Usually we choose m 10 Discrete-Event Simulation Lehmer Random Number Generators: Introduction 46/48

47 Final Considerations How do we use these tests to check the quality of a random Number Generator? If the generator to be tested is truly awful, then it may be sufficient to use some of these tests to see if failures a consistently produced If the generator is not so bad (but also not so good) then there will be some test that will reveal its deficiency, but unfortunately there are no simple rules for finding the test that identifies this situation If the generator is good, then it should fail any of these tests approximately only αx100% of the time where α is the confidence level of the test (ie, we repeat the tests many times and we should observe only a very limited number of failures) Discrete-Event Simulation Lehmer Random Number Generators: Introduction 47/48

48 Good Random Number Generators The quality of a random Number Generator is vital for providing reliable simulation results The Mathematic Department of the Hiroshima University is known to develop and to maintain random number generators of high quality The Marsenne Twister Generator is a very fast random number generator of period p = (> ) m-mat/mt/emthtml It shows very good statistical randomness It is the default Pseudo Random Number Generator of many contemporary languages Discrete-Event Simulation Lehmer Random Number Generators: Introduction 48/48