Random numbers and generators

Transcription

1

2 Chapter 2 Random numbers and generators Random numbers can be generated experimentally, like throwing dice or from radioactive decay measurements. In numerical calculations one needs, however, huge set of random numbers and this make gambling not practical. For that reason one relies on so-called Pseudo Random Numbers. Pseudo random numbers are not really random in the common day notion. They are generated by algorithms such that their statistical properties follow as close as possible those of truly random numbers. For that reason many different algorithms and many different tests for randomness have been designed. There are algorithms for many different distributions. The working horses, however, are algorithms to produce random numbers uniformly distributed in the interval [0, 1] such that the probability P(0 a x b 1) = b a. Random numbers produced by computer programs should be statistically reliable (i.e. pass statistics tests); have long periods (on a finite computer any generator will have some period, but it should be much longer than any number of random numbers used in the analysis); long are periods larger than 2 60 ; reproducible, i.e., give the same sequences when started from the begin in order to allow to repeat calculations; restartable, i.e., there should be a method to save the current status and restart from that at some later time; portable, should give identical results for different computer architectures and compiler.; efficient (fast), since one needs (very) many random numbers; 13

3 14 CHAPTER 2. RANDOM NUMBERS AND GENERATORS have disjoint subsets, since ideally one wants to use them in parallel machines; Finally one should take serious the warning reiterated in Press book Numerical Recipes [PrTeVe99]. The random number generators supplied by standard compilers (including the ANSI standard) are usually not good and with relatively small periods. If you really want to do scientifically sound work and need many random number (more than a few thousand) please rely on better algorithms (like RANLUX discussed further down). Different ideas are pursued by so-called Quasi Random Number Generators. There one constructs sequences of numbers filling a d-dimensional hypercube as uniform as possible, more uniform then pseudo random numbers. Such algorithms exist for moderately small dimensions (cf. [PrTeVe99]).

4 2.1. LINEAR CONGRUENTIAL GENERATORS Linear congruential generators This is simple, fast, restartable but with small periods and not reliable. Its simplest form (D. Lehmer: 1948) is based on integer multiplication and addition: where x n+1 = (a x n + c) mod m (2.1) a multiplier 0 a < m m modulus 0 < m c increment 0 c < m x 0 start value 0 x 0 < m (2.2) This gives a sequence of numbers between 0 and m. From this one computes real random numbers with r n = x n /m [0, 1). (2.3) Obviously the maximum period is m. The value of c is often chosen to be zero. Originally one used values of m that are powers of 2, since then the modulo operation amounts to a truncation of the binary representation of the number. A notorious bad choice was IBM s very first random number generator RANDU with a = 65539, c = 0 and m = It proved to have bad correlations. For such choices Knuth [Kn81] claims the best alternative is a = , c = 0 and m = The period depends very much on the seed x 0. For instance for the last mentioned values the period varies between 2 17 for seed 123 and 2 24 for seed values 0, 2, 124. Somewhat better choices seem to be those of l Ecuyer a = , c = 0, m = Park-Miller a = 7 5 = 16807, c = 0, m = (2.4) Whenever one uses a choice with c = 0 this is also a fixed point of the algorithm and thus one has to avoid it (check for that value). Algorithmic detail: A problem associated with the implementation of such random number algorithms in higher-level languages is that the multiplication would lead to overflows, numbers larger than the largest allowed integer. A method to multiply two 32-bit integers modulo a 32-bit constant in 32-bit arithmetic has been suggested by Schrage. It is based on approximate factorization of m, m = a q + r where q = [m/a], r = m moda. (2.5)

5 16 CHAPTER 2. RANDOM NUMBERS AND GENERATORS If r < q and a number 0 < z < m 1 then both and then { az mod m = a(z mod q) {0, 1,...m 1}, r[z/q] {0, 1,...m 1}. a(z modq) r[z/q] if this is 0, a(z modq) r[z/q] + m otherwise. (2.6) (2.7) For the Park-Miller numbers this gives m = = , a = 7 5 = 16807, q = [m/a] = , r = Problem: Implement the Park-Miller algorithm and compute a sequence of real numbers [0, 1). Estimate mean value, variance and higher moments. Plot ( X k N 1/(k + 1)) N vs. 1/N; what do you observe? Problem: One could formulate the linear congruential generator also for real numbers x n [0, 1), using m = 1.0 and defining the modulo operation via x mod 1 = x [x]. What values of a and c are possible? Implement this and compute mean value and several moments. For the linear congruential method the random numbers are correlated in subtle ways. If one considers tuples of k subsequent numbers as representing a point in k-dimensional space (r 1,r 2,...r k ), the points often fill parallel k 1- dimensional planes in the k-dimensional hypercube. This will be discussed when we present tests in Section

6 2.2. FIBONACCI TYPE GENERATORS Fibonacci type generators The Fibonacci sequence 1, 1, 2, 3, 5, 8, 13, 21, comes from starting with the numbers 1 and 1 and finding the next number by adding the preceeding two. If we add the number modulo 10 we get the sequence a random sequence (is it?) of integers. 1, 1, 2, 3, 5, 8, 3, 1, 4, 5, 9, 4,... Problem: Do you find a cycle in the modified Fibonacci sequence of integer numbers {0, 1, 2,...,9}? This gives rise to another method to construct pseudo random number: x n+1 = (x n + x n 1 ) mod m. (2.8) This allows much larger periods than just m. Implementation is also possible for real numbers [0, 1] if m = 1.0 and the modulo operation is understood accordingly x mod 1 = x [x]. More general we speak of a Lagged Fibonacci sequence for x n+1 = (x n p x n q ) mod m, (2.9) where denotes a binary operation (like addition or subtraction or some logical operation). For careful choices of p and q one can achieve a period up to (2 p 1)(2 q 1). An often used choice of values is p = 24 and q = 55. For its implementation one has to keep an array of length 55 in memory, which can be reused cyclically. To start (or restart) the generator one needs this array. For the first start one can build such an array with help of a simple linear congruential generator. A variant of the lagged Fibonacci generator is called Shift Register Generator. It uses the logical operation exclusive or (XOR) for the operation. For two bits a and b the corresponding multiplication table for the resulting a XOR b is a\ b Each new (signed) 32-bit integer number is thus computed by this operation (for each of the 32 bits) from two other, earlier computed integers. E.g. for p = 147 and q = 250 the new number could be

7 18 CHAPTER 2. RANDOM NUMBERS AND GENERATORS x(n-147) x(n-250) x(n) Each number is interpreted as a signed integer x [ m,m] with m = Since the sign is also random, one just adds m, it the number is negative, before normalizing to get a real number in [0, 1]. The Marsaglia-Zaman Generator[MaZa94] is a lagged Fibonacci (subtraction) generator with a so-called carry bit: = x n p x n q c n 1, x n =, c n = 0 if 0, x n = + m, c n = 1 if < 0. (2.10) The resulting random numbers are in [0,m]. For p = 10, r = 24 and m = 2 24 one find periods O( ). A particularly efficient and well tested generator of that type, but introducing extra holes in the sequence, is RANLUX (by Lüscher [Lu94], see also [Ja94]). It exists for C and F90, single and double precision, suitable for computing long vectors of random numbers and portable to all computers with IEEE-754 compliance. Since 1998 another popular pseudorandom number generator is the Mersenne Twister by Matsumoto and Nishimura. The name stems form the period which is chosen as a Mersenne prime number. After some time (longer than for the lagged Fibonacci) high-quality random numbers are produced. A drawback is that form a sequence of, e.g., 624 (for the variant MT19937) all future numbers can be predicted, thus it is unusable for cryptography. Further improvement of the quality of generators comes from combining different generators, reshuffling random numbers and reshuffling the seed tables. Finally here a quote from Marsaglia: Random number generators are like sex: when it s good, it s wonderful, when it s bad it s still pretty good.

8 2.3. TESTING RANDOM NUMBER QUALITY Testing random number quality Intelligence is what the IQ-tests test. In that sense we use statistical tests to probe the quality of pseudo random number generators. In this section we assume that the random numbers should be uniformly distributed and x n [0, 1] Simple tests These are the usual statistical tests, i.e., computation of moments. For a uniform distribution one expects X k = 1 N N i=1 x k i = 1 k O(1/ N). (2.11) Even if these are correct, the numbers may be correlated (statistically not independent). This can be checked by computing the autocorrelation (see Chapter 5) function. Again, for a uniform distribution of uncorrelated numbers one expects If this is not the case, we expect X n X n+k = X n 2 = 1 4. (2.12) X n X n+k X n X n+k exp( k/ξ), (2.13) where ξ denotes the correlation length. This is discussed in detail in Chapter 6. A trivial test is for symmetry. Instead of the sequence (X n ) we take (1 X n ). The results should still be that of a uniform distribution. Another, graphical, test (which will be elaborated in more detail in Section 2.3.4) is pairing subsequent numbers (x i,x i+1 considering them as points in the unit square. Do they fill up the square uniformly (inspection by eye)? This test is also called parking-lot test for obvious reasons χ 2 -test This is a test for the distribution density directly. One first builds (from the random number sequence) a histogram of values and then compares the histogram with the expected distribution density function. Fig. 2.1 gives examples of such histograms for a (hopefully) uniform distribution. How did we construct it? Decide on the number of bins nbins (e.g. 100) and initialize a histogram array h(1:nbins)=0. The bin size is 1/nbins.

9 20 CHAPTER 2. RANDOM NUMBERS AND GENERATORS N = N = N = Figure 2.1: Histograms for 10 5, 10 6 and 10 7 uniformly distributed random numbers (produced with the intrinsic routine F90-routine RANDOM NUMBER under Intel ifort v9.0). Compute N random numbers. For each compute first the slot (bin number i) by i=[r/binsize]+1 where [n] denotes the largest integer in n. Add 1 to the bin entry: h(i)=h(i)+1. At the end normalize the histogram to total area 1, i.e. h=h/(binsize*nran). How to estimate the quality of the resulting histogram? We first define a norm for the difference by adding the squares of the differences between the histogram values and the values of the tested distribution density (computed at the central values of the respective bins): χ 2 bins i (y i n th,i ) 2 The theoretically expected number of hits in a bin is where n th,i = f i n ran d i, n th,i. (2.14) n ran n bins y i n th,i f i is the number of random numbers is the number of bins is the number of hits in bin number i is the expected number of hits in bin number i is the value of f(x) at the central value of the bin d i is the width of bin number i with i d i f i = 1. (2.15)

10 2.3. TESTING RANDOM NUMBER QUALITY 21 With this we find another representation of χ 2 1 n ran y 2 i f i d i n ran. (2.16) The value of χ 2 determines the likelihood of the fit, in this case of the quality of the representation of the correct distribution in n bins bins. It follows a χ 2 -distribution of ν n bins 1 degrees of freedom. Its most likely value is ν, other values have to read from tables (or evaluation) of the distribution F χ 2(ν,x). This distribution has mean value ν and variance 2ν. E.g. for 100 bins one finds P(χ 2 113) = F χ 2(99, 113) = meaning that values below 113 are quite probably, larger values would be worrysome Kolmogorov-Smirnov Test Whereas the χ 2 -test needs binning (or discrete distributions) the Kolmogorov- Smirnov test works for continuous random variables. Given such a set of numbers one computes the maximum distance (at any point) between the cumulative distribution F(x) and the function constructed from the data: F n (x) = number of x i below x, i.e. x i x n. (2.17) This is a function with n steps, but if the set of random numbers is sorted increasingly then the maximum distance D obs = max F n (x) F(x) (2.18) can be computed easily. For n data points the probability to get some D is P(D > D obs ) = Q KS ( D ( ( n / n )). (2.19) The positive function Q KS is monotonically decreasing from Q KS (0) = 1 to Q KS ( ) = 0 and given by Q KS (z) = 2 ( 1) j 1 e 2j2 z 2. (2.20) j=1 It is found in tables (or can be calculated).

11 22 CHAPTER 2. RANDOM NUMBERS AND GENERATORS Figure 2.2: The random numbers of the mentioned generator (a = 137, c = 187 and m = 256) are aligned in 2-d plots! Spectral test This is an impressive visual test. One identifies k subsequent random numbers with the coordinates of a point in a k-dimensional hypercube. A simple example is shown in Fig The linear congruential algorithm with a = 137, c = 187 and m = 256 gives its number-pairs arranged along parallel lines. Sometimes one has to compute many numbers in order to exhibit this feature. Fig. 2.3 shows a small patch of the unit square produced in a run of 10 9 number for the generator with a = 16897, c = 0 and m = For linear congruential generators often such an arrangement of parallel k.1- dimensional hyperplanes is found when plotting k-tuples. Another example is Fig. 2.4, where the parameters a = 65539, c = 0 and m = There triples of numbers were used as points in a cube. Here is the MATHEMATICA code for that example: ApRandom := (Rannew = Ranold 65539; Ranold = Mod[Rannew, 32768]; Return[Ranold/32768]) Ranold = pts = Table[Point[{ApRandom, ApRandom, ApRandom}], {2000}] Show[Graphics3D[pts, ViewPoint -> {-1.529, 1.578, 2.573}]] The distance between such neighboring planes can be used to formulate the spectral test, described in more detail in Knuth s book [Kn81].

12 2.3. TESTING RANDOM NUMBER QUALITY Figure 2.3: Some numbers out of a run with 10 9 of the generator (a = 16897, c = 0 and m = ) are aligned! Figure 2.4: Triples of random numbers produced with a linear congruential generator (a = 65539, c = 0 and m = 2 16 ) fill parallel planes.

13 24 CHAPTER 2. RANDOM NUMBERS AND GENERATORS Problem: For the linear congruential generator with a = 65539, c = 0 and m = 2 16 do the spectral test for D = 3. Then use a = and m = and compare the result.

14 2.4. NON-UNIFORM DISTRIBUTIONS 25 y M f ( x) X nein ja 0 a r 1 b x Figure 2.5: Rejection method: If r 2 f(r 1 ) (with r 1 (a,b) and r 2 (0,M) uniformly distributed), then r 2 is accepted. 2.4 Non-uniform distributions Often one wants random numbers with a particular distribution. Although a Metropolis method like discussed in Chapter 4 is possible, depending on the distribution other approaches may be more efficient. Some distributions can be achieved by special tricks: X = max(x 1,X 2 ) has the distribution F 1 (x)f 2 (x). X = min(x 1,X 2 ) has the distribution F 1 (x) + F 2 (x) F 1 (x)f 2 (x). Due to this we may easily construct some distributions with help of a set of uniformly distributed random number u i : X = max(u 1,U 2 ) F(X) = x 2, f(x) = 2x, X = max(u 1,U 2,...U t ) F(X) = x t, f(x) = t x t 1. (2.21) Rejection method This is discussed in more detail in Chapter 4. The distribution density function f(x) is embedded in a bounding rectangle (a,b) (A,B) and a random number r 1 chosen uniformly (a,b) is accepted if r 2 f(r 1 ) for another random number r 2 chosen uniformly (A,B). (See Fig. 2.5.) A variant of this is the Metropolis algorithm, also discussed in Chapter 4.

15 26 CHAPTER 2. RANDOM NUMBERS AND GENERATORS u F(x) r x Figure 2.6: Uniformly distributed random numbers u lead to random numbers x with distribution density f(x) Inversion method The normalized probability density distribution f(x) defines the distribution function F X (x) = P(X x) = x dx f X (x). (2.22) It raises monotonically from F( ) = 0 to F( ) = 1. The crucial observation is: If a random variable Y is uniformly distributed, then X = F 1 (Y ) has the distribution density f(x). To show this we use the relation for the distribution of a function of a random variable. If a random variable y has the distribution density f Y (y) then the random variable X g(y ) has the density f X (x) = i where the sum runs over all points y i where x = g(y i ). In our case f Y = 1 in the unit interval. Furthermore x = g(y) = F 1 X (y) y = F X(x) dy dx = f X(x) = which is just (2.23). The method is exemplified in Fig. 2.6: f Y (y i (x)) g (y i (x)), (2.23) 1. Find a random number u uniformly distributed in (0, 1). ( ) 1 dy = 1 dx g (y) (2.24) 2. Compute x = F 1 (u) where F 1 is the inverse distribution function. 3. Then x has the distribution density f(x). The crucial problem usually is to invert F(x). If this is not possible analytically (and often it is not), then tabulation of F and piecewise interpolation can help.

16 2.4. NON-UNIFORM DISTRIBUTIONS 27 Simple examples are X = U 1/t F(x) = x t, X = ln U F(x) = exp( x). (2.25) Please take care of the normalization depending on the interval applicable! Gaussian normal distribution This is very often needed and there is an efficient method [BoMuMa58] to get such random variable. 1. Compute two random numbers r 1, r 2 uniformly distributed in (0, 1), and map them to ( 1, 1) with u i = 2r i Compute s = u u 2 2. If s 1 then reject this pair and start again above. If s < 1 continue with the next step. 3. Compute the pair of numbers x 1 = u 1 2 ln s s, x 2 = u 2 2 ln s s. (2.26) These two numbers are independent (!) and normally distributed with mean 0 and variance 1. The only disadvantage of this algorithm is that one has to compute logarithm and square root. Problem: Simulate radioactive decay. For N(t) not yet decayed nuclei during t the number of decaying nuclei is N(t) = N(t)λ t. This is simulated in the following way: In each time step t = 1 the number N(t) is reduced by 1 if a random number r (uniform in (0,1) is smaller than λ (for a choice 0 < λ < 1). Plot N(t) and log N(t) as functions of t/λ. Study values of N(0) = 10, 100, Do the functions depend on the starting value?

17 28 CHAPTER 2. RANDOM NUMBERS AND GENERATORS Bibliography BoMuMa58 G.E.P. Box, M.E. Muller and G. Marsaglia, Ann. Math. Stat. 28 (1958) 610. Ja90 F. James, Comp. Phys. Comm. 60 (1990) 329 Ja94 F. James, Comp. Phys. Comm. 79 (1994) 111; ibid. 97 (1996) 357. Kn81 D.E. Knuth, The art of computer programming, vol. 2 Lu94 M. Lüscher, Comp. Phys. Comm. 79 (1994) 100. Ma85 G. Marsaglia, in Computer Science and Statistics: The Interface; ed. C. Billard (Elsevier: 1985) MaZa94 G. Marsaglia and A. Zaman, Computers in Physics 8 (1994) 117. PrTeVe99 W. H. Press et al., Numerical Recipes, 2nd ed. (Cambridge UP: 1999) (Addison-Wesley, Reading, MA: 1981).