411 Pseudorandom Numbers To draw the winning number for several million dollars in a lottery, people would generally not trust a computer They would r

Transcription

1 CHAPTER 4 RANDOM NUMBER GENERATION 1 Pierre L'Ecuyer Universite de Montreal 2 41 INTRODUCTION Random numbers are the nuts and bolts of simulation Typically, all the randomness required by the model is simulated by a random number generator whose output is assumed to be a sequence of independent and identically distributed (IID) U(0 1) random variables (ie, continuous random variables distributed uniformly over the interval (0 1)) These random numbers are then transformed as needed to simulate random variables from dierent probability distributions, such as the normal, exponential, Poisson, binomial, geometric, discrete uniform, etc, as well as multivariate distributions and more complicated random objects In general, the validity of the transformation methods depends strongly on the IID U(0 1) assumption But this assumption is false, since the random number generators are actually simple deterministic programs trying to fool the user by producing a deterministic sequence that looks random What could be the impact of this on the simulation results? Despite this problem, are there \safe" generators? What about the generators commonly available in system libraries and simulation packages? If they are not satisfactory, how can we build better ones? Which onesshould be used, and where is the code? These are some of the topics addressed in this chapter 1 Chapter 4 of the Handbook on Simulation, Ed: Jerry Banks, Wiley, 1998 Version: February 27, Departement d'informatique et de recherche operationnelle, Universite de Montreal, CP 6128, Succ Centre-Ville, Montreal, H3C 3J7, Canada lecuyer@iroumontrealca

2 411 Pseudorandom Numbers To draw the winning number for several million dollars in a lottery, people would generally not trust a computer They would rather prefer a simple physical system that they understand well, such as drawing balls from one or more container(s) to select the successive digits of the number (as done, for example, by Loto Quebec each week in Montreal) Even this requires many precautions: The balls must have identical weights and sizes, be well mixed, and be changed regularly to reduce the chances that some numbers come out more frequently than others in the long run Such a procedure is clearly not practical for computer simulations, which often require millions and millions of random numbers Several other physical devices to produce random noise have been proposed and experiments have been conducted using these generators These devices include gamma ray counters, noise diodes, and so on [47, 62] Some of these devices have been commercialized and can be purchased to produce random numbers on a computer But they are cumbersome and they may produce unsatisfactory outputs, as there may besignicant correlation between the successive numbers Marsaglia [90] applied a battery of statistical tests to three such commercial devices recently and he reports that all three failed the tests spectacularly As of today, the most convenient and most reliable way of generating the random numbers for stochastic simulations appears to be via deterministic algorithms with a solid mathematical basis These algorithms produce a sequence of numbers which are in fact not random at all, but seem to behave like independent random numbers that is, like a realization of a sequence of IID U(0 1) random variables Such a sequence is called pseudorandom and the program that produces it is called a pseudorandom number generator In simulation contexts, the term random is used instead of pseudorandom (a slight abuseof language, for simplication) and we do so in this chapter The following denition is taken from L'Ecuyer [62, 64] Denition 1 A (pseudo)random number generator is a structure G =(S s 0 T U G), where S is a nite set of states, s 0 2 S is the initial state (or seed), the mapping 2

3 T : S! S is the transition function, U is a nite set of output symbols, and G : S! U is the output function The state of the generator is initially s 0 and evolves according to the recurrence s n = T (s n;1 ), for n = ::: At step n, the generator outputs the number u n = G(s n ) The u n, n 0, are the observations, andare also called the random numbers produced by the generator Clearly, the sequence of states s n is eventually periodic, since the state space S is nite Indeed, the generator must eventually revisit a state previously seen that is, s j = s i for some j > i 0 From then on, one must have s j+n = s i+n and u j+n = u i+n for all n 0 The period length is the smallest integer > 0 such that for some integer 0 and for all n, s +n = s n The smallest with this property is called the transient Often, = 0 and the sequence is then called purely periodic Note that the period length cannot exceed jsj, the cardinality of the state space Good generators typically have their very close to jsj (otherwise, there is a waste of computer memory) 412 Example: A Linear Congruential Generator Example 1 The best-known and (still) most widely used types of generators are the simple linear congruential generators (LCGs) [41, 57, 60, 82] The state at step n is an integer x n and the transition function T is dened by the recurrence x n =(ax n;1 + c) modm (1) where m > 0, a > 0, and c are integers called the modulus, the multiplier, and the additive constant, respectively Here, \mod m" denotes the operation of taking the least nonnegative residue modulo m In other words, multiply x n;1 by a, add c, divide the result by m, and put x n equal to the remainder of the division One can identify s n with x n and the state space S is the set f0 ::: m; 1g To produce values in the interval [0 1], one can simply dene the output function G by u n = G(x n )=x n =m When c = 0, this generator is called a multiplicative linear congruential generator (MLCG) The maximal period length for the LCG is m in general For the MLCG it 3

4 cannot exceed m ; 1, since x n = 0 is an absorbing state that must be avoided Two popular values of m are m =2 31 ;1andm =2 32 But as discussed later, these values are too small for the requirements of today's simulations LCGs with such small moduli are still in widespread use, mainly because of their simplicity and ease of implementation, but we believe that they should be discarded and replaced by more robust generators For a concrete illustration, let m = 2 31 ; 1 = , c = 0, and a = These parameters were originally proposed in [83] Take x 0 = Then x 1 = mod m = u 1 = =m =0: x 2 = mod m = u 2 = =m =0: x 3 = mod m = u 3 = =m =0: and so on 413 Seasoning the Sequence with External Randomness In certain circumstances one may want to combine the deterministic sequence with external physical noise The simplest and most frequently used way of doing this in simulation contexts is to select the seed s 0 randomly If s 0 is drawn uniformly from S, say by picking balls randomly from a container or by tossing fair coins, the generator can be viewed as an extensor of randomness: It stretches a short, truly random seed into a longer sequence of random-looking numbers Denition 1 can easily be generalized to accommodate this possibility: Add to the structure a probability distribution dened on S and say that s 0 is selected from In some contexts, one may want to rerandomize the state s n of the generator every now and then, or to jump ahead from s n to s n+ for some random integer For example, 4

5 certain types of slot machines in casinos use a simple deterministic random number generator, which keeps running at full speed (ie, computing its successive states) even when there is nobody playing with the machine Whenever a player hits the appropriate button and some random numbers are needed to determine the winning combination (eg, in the game of Keno) or to draw a hand of cards (eg, for poker machines), the generator provides the output corresponding to its current state Each time the player hits the button, he or she selects a, as just mentioned This is random (although not uniformly distributed) Since typical generators can advance by more than 1 million states per second, hitting the button at the right time to get a specic state or predicting the next output value from the previous ones is almost impossible One could go further and select not only the seed, but also some parameters of the generator at random For example, for a MLCG, one may select the multiplier a at random from a given set of values (for a xed m) or select the pairs (a m) at random from a given set Certain classes of generators for cryptographic applications are dened in a way that the parameters of the recurrence (eg, the modulus) are viewed as part of the seed and must be generated randomly for the generator to be safe (in the sense of unpredictability) After observing that physical phenomena by themselves are bad sources of random numbers and that the deterministic generators may produce sequences with too much structure, Marsaglia [90] decided to combine the output of some random number generators with various sources of white and black noise, such as music, pictures, or noise produced by physical devices The combination was done by addition modulo 2 (bitwise exclusive-or) between the successive bits of the generator's output and of the binary les containing the noise The result was used to produce a CD-ROM containing 48 billion random bits, which appear to behave as independent bits distributed uniformly over the set f0 1g Such a CD-ROM may beinteresting but is no universal solution: Its use cannot match the speed and convenience of a good generator, and some applications require much more random numbers than provided on this disk 5

6 414 Design of Good Generators How can one build a deterministic generator whose output looks totally random? Perhaps a rst idea is to write a computer program more or less at random that can also modify its own code in an unpredictable way However, experience shows that random number generators should not be built at random (see Knuth [57] for more discussion on this) Building a good random number generator may look easy on the surface, but it is not It requires a good understanding of heavy mathematics The techniques used to evaluate the quality of random number generators can be partitioned into two main classes: The structural analysis methods (sometimes called theoretical tests) and the statistical methods (also called empirical tests) An empirical test views the generator as ablack box It observes the output and applies a statistical test of hypothesis to catch up signicant statistical defects An unlimited number of such tests can be designed Structural analysis, on the other hand, studies the mathematical structure underlying the successsive values produced by the generator, most often over its entire period length For example, vectors of t successive output values of a LCG can be viewed as points in the t-dimensional unit hypercube [0 1] t It turns out that all these points, over the entire period of the generator, form a regular lattice structure As a result, all the points lie in a limited number of equidistant parallel hyperplanes, in each dimension t Computing certain numerical gures of merit for these lattices (eg, computing the distances between neighboring hyperplanes) is an example of structural analysis Statistical testing and structural analysis is discussed more extensively in forthcoming sections We emphasize that all these methods are in a sense heuristic: None ever proves that a particular generator is perfectly random or fully reliable for simulation The best they can do is improve our condence in the generator 415 Overview of What Follows We now give an overview of the remainder of this chapter In the next section we portray our ideal random number generator The desired properties include uniformity, independence, long period, rapid jump-ahead capability, ease of implementation, and 6

7 eciency in terms of speed and space (memory size used) In certain situations, unpredictability isalso an issue We discuss the scope and signicance of structural analysis as a guide to select families of generators and choose specic parameters Section 43 covers generators based on linear recurrences This includes the linear congruential, multiple recursive, multiply-with-carry, Tausworthe, generalized feedback shift register generators, all of which have several variants, and also dierent types of combinations of these We study their structural properties at length Section 44 is devoted to methods based on nonlinear recurrences, such asinversive and quadratic congruential generators, as well as other types of methods originating from the eld of cryptology Section 45 summarizes the ideas of statistical testing In Section 46 we outline the specications of a modern uniform random number package and refers to available implementations We also discuss parallel generators briey 42 DESIRED PROPERTIES 421 Unpredictability and \True" Randomness From the user's perspective, an ideal random number generator should be like a black box producing a sequence that cannot be distinguished from a truly random one In other words, the goal is that given the output sequence (u 0 u 1 :::) and an innite sequence of IID U(0 1) random variables, no statistical test (or computer program) could tell which is which with probability larger than 1/2 An equivalent requirement is that after observing any nite number of output values, one cannot guess any given bit of any given unobserved number better than by ipping a fair coin But this is an impossible dream The pseudorandom sequence can always be determined by observing it suciently, since it is periodic Similarly, forany periodic sequence, if enough computing time is allowed, it is always possible to construct a statistical test that the sequence will fail spectacularly To dilute the goal we may limit the time of observation of the sequence and the computing time for the test This leads to the introduction of computational complexity into the picture More specically, we now consider a family of generators, fg k k = 1 2 :::g, indexed by an integral parameter k equal to the number of bits required to 7

8 represent the state of the generator We assume that the time required to compute the functions T and G is (at worst) polynomial in k We also restrict our attention to the class of statistical tests whose running time is polynomial in k Since the period length typically increases as 2 k, this precludes the tests that exhaust the period A test is also allowed to toss coins at random, so its outcome is really a random variable We saythat the family fg k g is polynomial-time perfect if, for any polynomial time statistical test trying to distinguish the output sequence of the generator from an innite sequence of IID U(0 1) random variables, the probability that the test makes the right guess does not exceed 1=2 +e ;k, where is a positive constant An equivalent requirement is that no polynomial-time algorithm can predict any given bit of u n with probability of success larger than 1=2+e ;k, after observing u 0 ::: u n;1, for some >0 This setup is based on the idea that what cannot be computed in polynomial time is practically impossible to compute if k is reasonably large It was introduced in cryptology, where unpredictability isakey issue (see [4, 6, 59, 78] and other references given there) Are ecient polynomial-time perfect families of generators available? Actually, nobody knows for sure whether or not such a family exists But some generator families are conjectured to be polynomial-time perfect The one with apparently the best behavior so far is the BBS, introduced by Blum, Blum, and Shub [4], explained in the next example Example 2 The BBS generator of size k is dened as follows The state space S k is the set of triplets (p q x) such thatp and q are (k=2)-bit prime integers, p + 1 and q +1 are both divisible by 4,andxis a quadratic residue modulo m = pq, relatively prime to m (ie, x can be expressed as x = y 2 mod m for some integer y that is not divisible by p or q) The initial state (seed) is chosen randomly from S k, with the uniform distribution The state then evolves as follows: p and q remain unchanged and the successive values of x follow therecurrence x n = x 2 n;1 mod m: At each step, the generator outputs the k least signicant bits of x n (ie, u n = x n mod 2 k), where k K log k for some constant K The relevant conjecture here 8

9 is that with probability at least 1 ; e ;k for some > 0, factoring m (ie, nding p or q, given m) cannot be done in polynomial time (in k) Under this conjecture, the BBS generator has been proved polynomial-time perfect [4, 124] Now, a down-to-earth question is: How large should be k to be safe in practice? Also, how small should be K? Perhaps no one really knows A k larger than a few thousands is probably pretty safe but makes the generator too slow for general simulation use Most of the generators discussed in the remainder of this chapter are known not to be polynomial-time perfect However, they seem to have good enough statistical properties for most reasonable simulation applications 422 What Is a Random Sequence? The idea of a truly random sequence makes sense only in the (abstract) framework of probability theory Several authors (see, eg, [57]) give denitions of a random sequence, but these denitions require nonperiodic innite-length sequences Whenever one selects a generator with a xed seed, as in Denition 1, one always obtains a deterministic sequence of nite length (the length of the period) which repeats itself indenitely Choosing such a random number generator then amounts to selecting a nite-length sequence But among all sequences of length of symbols from the set U, for given and nite U, which ones are better than others? Let juj be the cardinality of the set U If all the symbols are chosen uniformly and independently from U, each of the juj possible sequences of symbols from U has the same probability of occurring, namely juj ; So it appears that no particular sequence (ie, no generator) is better than any other A pretty disconcerting conclusion! To get out of this dead end, one must take a dierent point ofview Suppose that a starting index n is randomly selected, uniformly from the set f1 2 ::: g, and consider the output vector (or subsequence) u n =(u n ::: u n+t;1 ), where t Now, u n is a (truly) random vector We would like u n to be uniformly distributed (or almost) over the set U t of all vectors of length t This requires juj t,sincethere are at most dierent values of u n in the sequence For < juj t, the set = fu n 1 9

10 n g can cover only part of the set U t Then one may ask to be uniformly spread over U t For example, if U is a discretization of the unit interval [0 1], such as U = f0 1=m 2=m : : : (m ; 1)=mg for some large integer m, and if the points of are evenly distributed over U t, they are also (pretty much) evenly distributed over the unit hypercube [0 1] t Example 3 Suppose that U = f0 1=100 2=100 ::: 99=100g and that the period of the generator is = 10 4 Here we have juj = 100 and = juj 2 In dimension 2, the pairs u n = (u n u n+1 ) can be uniformly distributed over U 2, and this happens if and only if each pair of successive values of the form (i=100 j=100), for 0 i j < 100 occurs exactly once over the period In dimension t > 2, we have juj t = 10 2t points to cover but can cover only 10 4 of those because of the limited period length of our generator In dimension 3, for instance, we can cover only 10 4 points out of 10 6 We would like those 10 4 points that are covered to be very uniformly distributed over the unit cube [0 1] 3 An even distribution of over U t, in all dimensions t, will be our basis for discriminating among generators The rationale is that under these requirements, subsequences of any t successive output values produced by the generator, from a random seed, should behave much like random points in the unit hypercube This captures both uniformity and independence: If u n = (u n ::: u n+t;1 ) is generated according to the uniform distribution over [0 1] t, the components of u n are independent and uniformly distributed over [0 1] This idea of looking at what happens when the seed is random, for a given nite sequence, is very similar to the scanning ensemble idea of Compagner [11, 12], except that we use the framework of probability theory instead The reader may have already noticed that under these requirements, will not look at all like a random set of points, because its distribution over U t is too even (or superuniform, as some authors say [116]) But what the foregoing model assumes is that only a few points are selected at random from the set In this case, the best one can do for these points to be distributed approximately as IID uniforms is to take superuniformly distributed over U t For this to make some sense, must be 10

11 several orders of magnitude larger than the number of output values actually used by the simulation To assess this even distribution of the points over the entire period, some (theoretical) understanding of their structural properties is necessary Generators whose structural properties are well understood and precisely described may look less random, but those that are more complicated and less understood are not necessarily better They may hide strong correlations or other important defects One should avoid generators without convincing theoretical support As a basic requirement, the period length must be known and huge But this is not enough Analyzing the equidistribution of the points as just discussed, which is sometimes achieved by studying the lattice structure, usually gives good insight on how the generator behaves Empirical tests can be applied thereafter, just to improve one's condence 423 Discrepancy A well-established class of measures of uniformity for nite sequences of numbers are based on the notion of discrepancy This notion and most related results are well covered by Niederreiter [102] We only recall the most basic ideas here Consider the N points u n = (u n ::: u n+t;1 ), for n = 0 ::: N ; 1, in dimension t, formed by (overlapping) vectors of t successive output values of the generator For any hyper-rectangular box aligned with the axes, of the form R = Q t j=1[ j j ), with 0 j < j 1, let I(R) be the number of points u n falling into R, and V (R) = Q t j=1( j ; j ) be the volume of R Let R be the set of all such regions R, and D (t) N = max jv (R) ; I(R)=Nj: R2R This quantity is called the t-dimensional (extreme) discrepancy of the set of points fu 0 ::: u N;1 g If we impose j = 0 for all j that is, we restrict R to those boxes which have one corner at the origin, then the corresponding quantity is called the star discrepancy, denoted by D (t) N Other variants also exist, with richer R 11

12 A low discrepancy value means that the points are very evenly distributed in the unit hypercube To get superuniformity of the sequence over its entire period, one might want tominimize the discrepancy D (t) or D (t) for t =1 2 ::: A major practical diculty with discrepancy is that it can be computed only for very special cases For LCGs, for example, it can be computed eciently in dimension t = 2, but for larger t, the computing cost then increases as O(N t ) In most cases, only (upper and lower) bounds on the discrepancy are available Often, these bounds are expressed as orders of magnitude as a function of N, are dened for N =, and/or are averages over a large (specic) class of generators (eg, over all full-period MLCGs with a given prime modulus) Discrepancy also depends on the rectangular orientation of the axes, in contrast to other measures of uniformity, such as the distances between hyperplanes for LCGs (see Section 434) On the other hand, it applies to all types of generators, not only those based on linear recurrences We previously argued for superuniformityover the entire period, which means seeking the lowest possible discrepancy When a subsequence of length N is used (for N ), starting, say, at a random point along the entire sequence, the discrepancy of that subsequence should behave (viewed as a random variable) as the discrepancy of a sequence of IID U(0 1) random variables The latter is (roughly) of order O(N ;1=2 ) for both the star and extreme discrepancies Niederreiter [102] shows that the discrepancy of full-period MLCGs over their entire period (of length = m ; 1), on the average over multipliers a, is of order O(m ;1 (log m) t log log(m + 1)) This order is much smaller (for large m) than O(m ;1=2 ), meaning superuniformity Over small fractions of the period length, the available bounds on the discrepancy are more in accordance with the law of the iterated logarithm [100] This is yet another important justication for never using more than a negligible fraction of the period Suppose now that numbers are generated in [0 1] with L fractional binary digits This gives resolution 2 ;L, which means that all u n 's are multiples of 2 ;L It then follows ([102]) that D (t) N 2 ;L for all t 1 and N 1 Therefore, as a necessary condition for the discrepancy to be of the right order of magnitude, the resolution 2 ;L must be 12

13 small enough for the number of points N that we plan to generate: 2 ;L should be much smaller than N ;1=2 A too coarse discretization implies a too large discrepancy 424 Quasi-random Sequences The interest in discrepancy stems largely from the fact that deterministic error bounds for (Monte Carlo) numerical integration of a function are available in terms of D (t) N and of a certain measure of variability of the function In that context, the smaller the discrepancy, the better (because the aim is to minimize the numerical error, not really to imitate IID U(0 1) random variables) Sequences for which the discrepancy of the rst N values is small for all N are called low-discrepancy or quasi-random sequences [102] Numerical integration using such sequences is called quasi-monte Carlo integration To estimate the integral using N points, one simply evaluates the function (say, a function of t variables) at the rst N points of the sequence, takes the average, multiplies by the volume of the domain of integration, and uses the result as an approximation of the integral Specic low-discrepancy sequences have been constructed by Sobol', Faure, and Niederreiter, among others (see [102]) Owen [106] gives a recent survey of their use In this chapter we concentrate on pseudorandom sequences and will not discuss quasi-random sequences further 425 Long Period Let us now return to the desired properties of pseudorandom sequences, starting with the length of the period What is long enough? Suppose that a simulation experiment takes N random numbers from a sequence of length Several reasons justify the need to take N (see, eg, [21, 64, 86, 102, 112]) Based on geometric arguments, Ripley [112] suggests that N 2 for linear congruential generators The papers [75, 79] provide strong experimental support for this, based on extensive empirical tests Our previous discussion also supports the view that must be huge in general Period lengths of 2 32 or smaller, which are typical for the default generators of many operating systems and software packages, are unacceptably too small Such period 13

14 lengths can be exhausted in a matter of minutes on today's workstations Even =2 64 is a relatively small period length Generators with period lengths over are now available 426 Eciency Some say that the speed of a random number generator (the number of values that it can generate per second, say) is not very important for simulation, since generating the numbers typically takes only a tiny fraction of the simulation time But there are several counterexamples, such as for certain large simulations in particle physics [26], or when using intensive Monte Carlo simulation to estimate with precision the distribution of a statistic that is fast to compute but requires many random numbers Moreover, even if a fast generator takes only, say, 5% ofthesimulation time, changing to another one that is 20 times slower will approximately double the total simulation time Since simulations often consume several hours of CPU time, this is signicant The memory size used by a generator might also be important in general, especially since simulations often use several generators in parallel, for instance to maintain synchronization for variance reduction purposes (see Section 46 and [7, 60] for more details) 427 Repeatability, Splitting Facilities, and Ease of Implementation The ability to replicate exactly the same sequence of random numbers, called repeatability, is important for program verication and to facilitate the implementation of certain variance reduction techniques [7, 55, 60, 113] Repeatability is a major advantage of pseudorandom sequences over sequences generated by physical devices The latter can of course be stored on disks or other memory devices, and then reread as needed, but this is less convenient than a good pseudorandom number generator that ts in a few lines of code in a high-level language A code is said to be portable if it works without change and produces exactly the same sequence (at least up to machine accuracy) across all \standard" compilers and 14

15 computers A portable code in a high-level language is clearly much more convenient than a machine-dependent assembly-language implementation, for which repeatability is likely to be more dicult to achieve Ease of implementation also means the ease of splitting the sequence into (long) disjoint substreams and jumping quickly from one substream to the next In Section 46 we showwhy this is important For this, there should be an ecient way to compute the state s n+ for any large, given s n For most linear-type generators, we knowhowtodo that But for certain types of nonlinear generators and for some methods of combination (such asshuing), good jump-ahead techniques are unknown Implementing a random number package as described in Section 46 requires ecient jump-ahead techniques 428 Historical Accounts There is an enormous amount of scientic literature on random number generation Law and Kelton [60] present a short (but interesting) historical overview Further surveys and historical accounts of the old days are provided in [47, 53, 119] Early attempts to construct pseudorandom number generators have given rise to all sorts of bad designs, sometimes leading to disatrous results An illustrative example is the middle-square method, which works as follows (see, eg, [57, 60]) Take a b-digit number x i;1 (say, in base 10, with b even), square it to obtain a 2b-digit number (perhaps with zeros on the left), and extract the b middle digits to dene the next number x i To obtain an output value u i in [0 1), divide x i by10 b The period length of this generator depends on the initial value and is typically very short, sometimes of length 1 (such as when the sequence reaches the absorbing state x i =0) Hopefully, it is no longer used Another example of abadgenerator is RANDU (see G4 in Table 1) 15

16 43 LINEAR METHODS 431 Multiple-Recursive Generator Consider the linear recurrence x n =(a 1 x n;1 + + a k x n;k ) mod m (2) where the order k and the modulus m are positive integers, while the coecients a 1 ::: a k are integers in the range f;(m ; 1) ::: m ; 1g Dene Z m as the set f0 1 ::: m;1g on which operations are performed modulo m The state at step n of the multiple recursive generator (MRG) [57, 62, 102] is the vector s n = (x n ::: x n+k;1 ) 2 Z k m The output function can be dened simply by u n = G(s n ) = x n =m, which gives a value in [0 1], or by a more rened transformation if a better resolution than 1=m is required The special case where k = 1isthe MLCG mentioned previously The characteristic polynomial P of (2) is dened by P (z) =z k ; a 1 z k;1 ;;a k : (3) The maximal period length of (2) is = m k ; 1, reached if and only if m is prime and P is a primitive polynomial over Z m, identied here as the nite eld with m elements Suppose that m is prime and let r =(m k ; 1)=(m ; 1) The polynomial P is primitive over Z m if and only if it satises the following conditions, where everything is assumed to be modulo m (see [57]) (a) [(;1) k+1 a k ] (m;1)=q 6= 1 for each prime factor q of m ; 1 (b) z r mod P (z) =(;1) k+1 a k (c) z r=q mod P (z) has degree > 0 for each prime factor q of r, 1<q<r For k = 1 and a = a 1 (the MLCG case), these conditions simplify to a 6= 0 (mod m) and a (m;1)=q 6= 1 (mod m)foreach prime factor q of m;1 For large r, nding the factors q to check condition (c) can be too dicult, since it requires the factorization of r In this 16

17 case, the trick is to choose m and k so that r is prime (this can be done only for prime k) Testing primality of large numbers (using probabilistic algorithms, for example, as in [73, 111]) is much easier than factoring Given m, k, and the factorizations of m ; 1 and r, primitive polynomials are generally easy to nd, simply by random search If m is not prime, the period length of (2) has an upper bound typically much lower than m k ; 1 For k =1and m =2 e, e 4, the maximum period length is 2 e;2, which is reached if a 1 = 3 or 5 (mod 8) and x 0 is odd [57, p 20] Otherwise, if m = p e for p prime and e 1, and k > 1 or p > 2, the upper bound is (p k ; 1)p e;1 [36] Clearly, p = 2 is very convenient from the implementation point of view, because the modulo operation then amounts to chopping-o the higher-order bits So to compute ax mod m in that case, for example with e = 32 on a 32-bit computer, just make sure that the overow-checking option or the compiler is turned o, and compute the product ax using unsigned integers while ignoring the overow However, taking m =2 e imposes a big sacrice on the period length, especially for k > 1 For example, if k =7and m =2 31 ; 1 (a prime), the maximal period length is (2 31 ;1) 7 ; But for m =2 31 and the same value of k, the upper bound becomes (2 7 ; 1)2 31;1 < 2 37, which is more than times shorter For k = 1 and p = 2, an upper bound on the period length of the ith least signicant bit of x n is max(1 2 i;2 ) [7], and if a full cycle is split into 2 d equal segments, all segments are identical except for their d most signicant bits [20, 26] For k > 1 and p =2,the upper bound on the period length of the ith least signicant bit is (2 k ; 1)2 i;1 So the low-order bits are typically much too regular when p =2 For k =7and m =2 31, for example, the least signicant bit has period length at most 2 7 ; 1 = 127, the second least signicant bit has period length at most 2(2 7 ; 1) = 254, and so on Example 4 Consider the recurrence x n = 10205x n;1 mod 2 15, with x 0 = The rst eight values of x n, in base 10 and in base 2, are x 0 = = x 1 = =

18 x 2 = = x 3 = 7581 = x 4 = = x 5 = 1093 = x 6 = = x 7 = = : The last two bits are always the same The third least signicant bit has a period length of 2, the fourth least signicant bit has a period length of 4, and so on Adding a constant c as in (1) can slightly increase the period-length The LCG with recurrence (1) has period length m if and only if the following conditions are satised ([57, p 16]) 1 c is relatively prime to m 2 a ; 1 is a multiple of p for every prime factor p of m (including m itself if m is prime) 3 If m is a multiple of 4, then a ; 1 is also a multiple of 4 For m = 2 e 4, these conditions simplify to c is odd and a mod 4 = 1 But the loworder bits are again too regular: The period length of the ith least signicant bit of x n is at most 2 i A constant c can also be added to the right side of the recurrence (2) One can show (see [62]) that a linear recurrence of order k with such a constant term is equivalent to some linear recurrence of order k + 1with no constant term As a result, an upper bound on the period length of such a recurrence with m = p e is (p k+1 ; 1)p e;1, which is much smaller than m k for large e and k All of this argues against the use of power-of-2 moduli in general, despite their advantage in terms of implementation It favors prime moduli instead Later, when 18

19 discussing combined generators, we will also be interested in moduli that are the products of a few large primes 432 Implementation for Prime m For k > 1 and prime m, for the characteristic polynomial P to be primitive, it is necessary that a k and at least another coecient a j be nonzero From the implementation point ofview,itisbesttohave only two nonzero coecients that is, a recurrence of the form x n =(a r x n;r + a k x n;k ) mod m (4) with characteristic trinomial P dened by P (z) =z k ; a r z k;r ; a k Note that replacing r by k ; r generates the same sequence in reverse order When m is not a power of 2, computing and adding the products modulo m in (2) or (4) is not necessarily straightforward, using ordinary integer arithmetic, because of the possibility ofoverow: The products can exceed the largest integer representable on the computer For example, if m =2 31 ; 1 and a 1 = 16807, then x n;1 can be as large as 2 31 ; 2, so the product a 1 x n;1 can easily exceed 2 31 L'Ecuyer and C^ote [76] study and compare dierent techniques for computing a product modulo a large integer m, using only integer arithmetic, so that no intermediate result ever exceeds m Among the general methods, working for all representable integers and easily implementable in a high-level language, decomposition was the fastest in their experiments Roughly, this method simply decomposes each of the two integers that are to be multiplied in two blocks of bits (eg, the 15 least signicant bits and the 16 most signicant ones, for a 31-bit integer) and then cross-multiplies the blocks and adds (modulo m) just as one does when multiplying large numbers by hand There is a faster way to compute ax mod m for 0 < a x < m, called approximate factoring, which works under the condition that a (m mod a) <m: (5) This condition is satised if and only if a = i or a = bm=ic for i< p m (here bxc denotes the largest integer smaller or equal to x, so bm=ic is the integer division of m by i) To 19

20 implement the approximate factoring method, one initially precomputes (once for all) the constants q = bm=ac and r = m mod a Then, for any positive integer x<m, the following instructions have the same eect as the assignment x ax mod m, but with all intermediate (integer) results remaining strictly between ;m and m [7, 61, 107]: y x bx=qc a(x ; yq) ; yr IF x<0 THEN x x + m END As an illustration, if m =2 31 ; 1 and a = 16807, the generator satises the condition, since < p m In this case, one has q = and r = 2836 Hormann and Deringer [51] give adierent method, which is about as fast, for the case where m =2 31 ; 1 Fishman [41, p 604] also uses a dierent method to implement the LCG with m =2 31 ; 1 and a = , which does not satisfy (5) Another approach is to represent all the numbers and perform all the arithmetic modulo m in double-precision oating point This works provided that the multipliers a i are small enough so that the integers a i x n;i and their sum are always represented exactly by the oating-point values A sucient condition is that the oating-point numbers are represented with at least dlog 2 ((m ; 1)(a a k ))e bits of precision in their mantissa, where dxe denotes the smallest integer larger or equal to x On computers with good 64-bit oating-point hardware (most computers nowadays), this approach usually gives by far the fastest implementation (see, eg, [68] for examples and timings) 433 Jumping Ahead To jump ahead from x n to x n+ with an MLCG, just use the relation x n+ = a x n mod m =(a mod m)x n mod m: 20

21 If many jumps are to be performed with the same, the constant a mod m can be precomputed once and used for all subsequent computations Example 5 Again, let m = , a = 16807, and x 0 = Suppose that we want to compute x 3 directly from x 0, so =3 One easily nds that mod m = and x 3 = x 0 mod m = , which agrees with the value given in Example 1 Of course, we are usually interested in much larger values of, but the method works the same way For the LCG, with c 6= 0,onehas x n+ = a x n + c(a ; 1) a ; 1! mod m: To jump ahead with the MRG, one way is to use the fact that it can be represented as a matrix MLCG: X n = AX n;1 mod m, where X n is s n represented as a column vector and A is a k k square matrix Jumping ahead is then achieved in the same way as for the MLCG: X n+ = A X n mod m =(A mod m)x n mod m: Another way is to transform the MRG into its polynomial representation [64], in which jumping ahead is easier, and then apply the inverse transformation to recover the original representation 434 Lattice Structure of LCGs and MRGs A lattice of dimension t, in the t-dimensional real space IR t, is a set of the form L = 8 < tx : V = j=1 9 = z j V j j each z j 2 Z (6) where Z is the set of all integers and fv 1 ::: V t g is a basis of IR t The lattice L is thus the set of all integer linear combinations of the vectors V 1 ::: V t,andthese vectors are called a lattice basis of L The basis fw 1 ::: W t g of IR t which satises V 0 i W j = ij for all 1 i j t (where the prime means \transpose" and where ij = 1 if i = j, 0 21

22 otherwise) is called the dual of the basis fv 1 ::: V t g, and the lattice generated by this dual basis is called the dual lattice to L Consider the set T t = fu n =(u n ::: u n+t;1 ) j n 0 s 0 =(x 0 ::: x k;1 ) 2 Z k mg (7) of all overlapping t-tuples of successive values produced by (2), with u n = x n =m, from all possible initial seeds Then this set T t is the intersection of a lattice L t with the t-dimensional unit hypercube I t =[0 1) t For more detailed studies and to see how to construct a basis for this lattice L t and its dual, see [23, 57, 73, 77] For t k it is clear from the denition of T t that each vector (x 0 ::: x t;1 ) in Z t m can be taken as s 0, so T t = Z t m=m = ( Z t =m) \ I t that is, L t is the set of all t-dimensional vectors whose coordinates are multiples of 1=m, andt t is the set of m t points in L t whose coordinates belong to f0 1=m ::: (m ; 1)=mg For a full-period MRG, this also holds if we x s 0 in the denition of T t to any nonzero vector of Z k m, and then add the zero vector to T t In dimension t > k, the set T t contains only m k points, while Z t m=m contains m t points Therefore, for large t, T t contains only a small fraction of the t-dimensional vectors whose coordinates are multiples of 1=m For full-period MRGs, the generator covers all of T t except the zero state in one cycle In other cases, such as for MRGs with nonprime moduli or MLCGs with powerof-2 moduli, each cycle covers only a smaller subset of T t, and the lattice generated by that subset is often equal to L t, but may in some cases be a strict sublattice or subgrid (ie, a shifted lattice of the form V 0 + L where V 0 2 IR t and L is a lattice) In the latter case, to analyze the structural properties of the generator, one should examine the appropriate sublattice or subgrid instead of L t Consider, for example, an MLCG for which m is a power of 2, a mod 8 = 5, and x 0 is odd The t-dimensional points constructed from successive values produced by this generator form a subgrid of L t containing one-fourth of the points [3, 50] For a LCG with m a power of 2 and c 6= 0, with full period length = m, the points all lie in a grid that is a shift of the lattice L t associated with the corresponding MLCG (with the same a amd m) The value of c determines only the shifting and has no other eect on the lattice structure 22

23 10 u n u n Figure 1: All pairs (u n u n+1 ) for the LCG with m = 101 and a = 12 Example 6 Figures 1 to 3 illustrate the lattice structure of a small, but instructional, LCGs with (prime) modulus m = 101 and full period length = 100, in dimension t =2 They show all 100 pairs of successive values (u n u n+1 ) produced by these generators, for the multipliers a = 12, a = 7, and a = 51, respectively In each case, one clearly sees the lattice structure of the points Any pair of vectors forming a basis determine a parallelogram of area 1/101 This holds more generally: In dimension t, the vectors of any basis of L t determine a parallelepiped of volume 1=m k Conversely, any set of t vectors that determine such a parallelepiped form a lattice basis The points are much more evenly distributed in the square for a = 12 than for a = 51, and slightly more evenly distributed for a = 12 than for a =7 The points of L t are generally more evenly distributed when there exists a basis comprised of vectors of similar lengths One also sees from the gures that all the points lie in a relative small number of equidistant parallel lines In Figure 3, only two lines contain all the points and this leaves large empty spaces between the lines, which is bad 23

24 10 u n u n Figure 2: All pairs (u n u n+1 ) for the LCG with m = 101 and a =7 In general, the lattice structure implies that all the points of T t lie on a family of equidistant parallel hyperplanes Among all such families of parallel hyperplanes that cover all the points, take the one for which the successive hyperplanes are farthest apart The distance d t between these successive hyperplanes is equal to 1=`t, where `t is the length of a shortest nonzero vector in the dual lattice to L t Computing a shortest nonzero vector in a lattice L means nding the combination of values of z j in (6) giving the shortest V This is a quadratic optimization problem with integer variables and can be solved by a branch-and-bound algorithm, as in [15, 40] In these papers the authors use an ellipsoid method to compute the bounds on the z j for the branch-and-bound This appears to be the best (general) approach known to date and is certainly much faster than the algorithm given in [23] and [57] This idea of analyzing d t was introduced by Coveyou and MacPherson [18] through the viewpoint of spectral analysis For this historical reason, computing d t is often called the spectral test The shorter the distance d t, the better, because a large d t means thick empty slices 24

25 u n u n Figure 3: All pairs (u n u n+1 ) for the LCG with m = 101 and a = 51 of space between the hyperplanes One has the theoretical lower bound where t d t d t = 1 (8) t mk=t is a constant which depends only on t and whose exact value is currently known only for t 8 [57] So, for t 8 and T 8, one can dene the gures of merit S t = d t =d t and M T = min ktt S t, which lie between 0 and 1 Values close to 1 are desired Another lower bound on d t, for t>k, is (see [67]) d t 0 1+ j=1 a 2 j 1 A ;1=2 : (9) This means that an MRG whose coecients a j are small is guaranteed to have a large (bad) d t Other gures of merit have been introduced to measure the quality of random number generators in terms of their lattice structure For example, one can count the minimal number of hyperplanes that contain all the points or compute the ratio of lengths of 25

26 the shortest and longest vectors in a Minkowski-reduced basis of the lattice For more details on the latter, which istypically much more costly to compute than d t, the reader can consult [77] and the references given there These alternative gures of merit do not tell us much important information in addition to d t Tables 1 and 2 give the values of d t and S t for certain LCGs and MRGs All these generators have full period length The LCGs of the rst table are well known and most are (or have been) heavily used For m = 2 31 ; 1, the multiplier a = was found by Fishman and Moore [42] in an exhaustive search for the MLCGs with the best value of M 6 for this value of m It is used in the GPSS/H simulation environment The second multiplier, a = 16807, was originally proposed in [83], is suggested in many simulation books and papers (eg, [7, 107, 114]) and appears in several software systems such as the SLAM II and SIMAN simulation programming languages, MATLAB [94], the IMSL statistical library [54], and in operating systems for the IBM and Macintosh computers It satises condition (5) The IMSL library also has available the two multipliers and , with the same modulus, as well as the possibility of adding a shue to the LCG The multiplier a = was proposed in [108], is recommended in [60, 92] among others, and is used in software such as the SIMSCRIPT II5 and INSIGHT simulation programming languages Generator G4, with modulus m = 2 31 and multiplier a = 65539, is the infamous RANDU generator, used for a long time in the IBM/360 operating system Its lattice structure is particularly bad in dimension 3, where all the points lie in only 15 parallel planes Law and Kelton [60] give a graphical illustration Generator G5, with m =2 32, a = 69069, and c = 1, is used in the VAX/VMS operating system The LCG G6, with modulus m = 2 48, multiplier a = , and constant c = 11, is the generator implemented in the procedure drand48 of the SUN Unix system's library [117] G7, whose period length is slighly less than 2 40, is used in the Maple mathematical software We actually recommend none of the generators G1 to G7 Their period lengths are too short and they fail many statistical tests (see Section 45) In Table 2, G8 and G9 are two MRGs of order 7 found by a random search for multipliers with a \good" lattice structure in all dimensions t 20, among those giving 26

27 Table 1: Distances between hyperplanes for some LCGs G1 G2 G3 G4 G5 G6 G7 m 2 31 ; ; ; ; 11 k a c ; ; ; ; 12 S S S S S S S =m 465E E E E E E E-12 d E E E E E E E-6 d E E E E E E-4 d E E E E E E-3 d E E-3 d E d E d d d d d d d d d d d d d d d d d d d d d d d