Finally, a theory of random number generation

Transcription

1 Finally, a theory of random number generation F. James, CERN, Geneva Abstract For a variety of reasons, Monte Carlo methods have become of increasing importance among mathematical methods for solving all kinds of computational problems in science and medecine. As the size and complexity of such problems increases, the quality of the random number generator becomes critical even as it becomes difficult if not impossible to verify whether a given generator is good enough for the problem at hand. All this has changed in recent years as it becomes clear that the work of Martin Lüscher provides the first operational definition of randomness in the sense required for Monte Carlo calculations, and that the generator he proposed is the first for which there is a convincing argument that it must produce random sequences in which no defect can be observed. Not surprisingly, after six years of intensive use, no departures from randomness have been found in his generator. paper presented at the Fifth Internatioal Workshop On Mathematical Methods in Scattering Theory and Biomedical Technology Corfu, Greece, October 2001 appears in the proceedings published as Scattering and Biomedical Engineering D. Fotiadis and C. Massalas (eds.) World Scientific (2002) 1

2 1 Introduction. Pseudorandom Numbers The random numbers discussed here are technically referred to as pseudorandom, meaning that they are not really random in the sense of being unpredictable and unrepeatable, but they should appear in every way to be really random and are unpredictable to someone who does not know the algorithm used to generate them. Random numbers of this sort are of the greatest (and still increasing) importance in many areas of computing, particularly in scientific applications involving largescale calculations. The importance of the methods making use of random numbers is partly due to the discovery in the early days of computing that a very simple and fast numerical algorithm could produce long sequences of numbers suitable for calculations which soon became known as Monte Carlo, in honour of the famous gambling casino located there. 2 Traditional Random Number Generators There is a large body of knowledge concerning pseudorandom number generators built up over more than fifty years. The time-honoured procedure for making a new improved RNG usually goes as follows: 1. Choose an algorithm based on a simple arithmetic operation known from experience to work reasonably well. 2. Adjust available constants to give the longest possible period and to avoid or minimize all known defects. 3. If necessary apply additional tricks such as shuffling or combining two algorithms. 4. Do extensive testing of the sequences to demonstrate that the resulting algorithm is good. Some years later, even the best RNGs produced according to this method are invariably found to fail some stringent test or give the wrong answer in some calculation. Nowhere in the above is there any credible reason why the algorithm should produce random or even random-looking numbers. It is in fact amazing that the usual operations like integer multiply and modulo a large integer, which are perfectly repeatable and predictable, should produce random-looking output. The difficulty in understanding this random behaviour is often attributed to the fact that it is hard even to define what one means by random. In particular, one common definition is based on algorithmic complexity. Essentially, it says that the randomness of a sequence is related to the length of the 2

3 shortest algorithm that could generate the sequence. This is intuitively easy to understand for the limiting cases: a very random sequence could not be generated by any algorithm except the sequence itself, which is the longest possible algorithm; and a very bad sequence (like all zeros) can be generated by a very short algorithm. However, this definition is not very helpful for evaluating the quality of a particular RNG, and it is relatively easy to produce examples where increasing the complexity of the algorithm decreases the quality of the generator. Similarly, definitions based on passing statistical tests are unusable because the number of possible tests is uncountably infinite. 3 Classical mechanical systems A radically different approach has been attempted in recent years, based on dynamical systems with strongly stochastic properties. Perhaps because it is so different, this work seems to have been largely ignored by the traditional random number community. Classical dynamical systems can be divided into two categories, those that conserve energy (Hamiltonian systems) and those that do not (dissipative). The behaviour of dissipative systems on long timescales has been the subject of considerable interest in recent years because they often show chaotic properties and unusual behaviour associated with attractors (especially strange attractors) which are closed or restricted orbits within the total available space of motion. However, dissipative systems, although they can be chaotic, are not good candidates for modeling random numbers because they do not fill up the whole available space (do not generate all possible sets of numbers). Hamiltonian systems turn out to be the interesting systems for our purposes. Traditionally, at least in Western countries, interest in Hamiltonian systems has been concentrated on those for which the Hamiltonian is separable and integrable, that is, exactly soluble. Fortunately, however, the Russian school of mathematicians and physicists has been interested in the properties of systems which cannot be solved exactly. It has long been known, for instance that a system as simple as the double pendulum is not soluble and indeed behaves very unpredictably. Largely through the work of Kolmogorov and Arnold, a whole theory was developed, establishing a hierarchy of chaotic behaviour of such systems. At the bottom of the hierarchy, just above the soluble (completely predictable) systems, come the ergodic systems which sweep out the entire space available to them, but not necessarily in a random way, where this randomness has a very precise meaning indicated below. At the top of the randomness hierarchy are systems called C-systems and K-systems. Most importantly, it was shown that this hierarchy is real: Every system at a given level in the hierarchy possesses all the properties of all the lower levels, but fails to possess at least one property of all the higher levels. And it is possible to find at least one system at every level (of which there are an infinite number). 3

4 4 Kolmogorov K-systems K-systems are a class of continuous (classical) Hamiltonian systems whose important property for our purposes is the mixing property. Let x i and x j represent the state of the dynamical system at two times i and j. Let {A 1 } and {A 2 } be any two subspaces of the entire allowed space of points x, with measures (volumes relative to the total allowed volume) respectively A 1 and A 2. Then the dynamical system is said to be a 2-system if P (x i {A 1 } and x j {A 2 }) = A 1 A 2 for all i and j sufficiently far apart, and for all subspaces {A}. Similarly, a 3-system has the same property for three points and three subvolumes, etc. A K-system has the same property for an arbitrarily large number of points and subvolumes. This property for K-systems, called K-mixing or strong mixing, is easily seen to be nothing other than statistical independence, which is exactly what we mean by randomness. Each point is (asymptotically) independent of all the other points in the sense that the probability of finding it in any volume is independent of all the other states it has been in. 5 Random Number Generators based on K-systems 5.1 MIXMAX G. Savvidy and his co-workers in Armenia may have been the first to recognize the possibility of generating demonstrably random numbers using K-systems. Unfortunately, their generator MIXMAX, based on matrix multiplication, is slow and clumsy to use and never attracted much attention. Some of their work is published in J. Comp. Phys. 97 (1991) 566 and 97(1991) 573. Further papers appear only in conference proceedings and internal reports. 5.2 Other chaotic maps Several other random number generators have been proposed based on different dynamical systems known to have strong chaotic or stochastic properties. The most popular systems are the logistic map and Arnold s cat map. These generators suffer from the same kind of problems as MIXMAX. Often the period is not even known. In addition, they may yield non-uniform random numbers which must be transformed back into a uniform distribution. 4

5 5.3 RANLUX Contrary to the way the above generators were conceived, Martin Lüscher started with an existing random number generator RCARRY with very good practical properties (fast, uniform, convenient, known long period, etc.) and demonstrated that it had the properties of a K-system and showed how to make it into a high-quality RNG. 6 Is RANLUX a K-system? The reasons why RANLUX is expected to produce very high quality random numbers are well explained in the original paper of Lüscher [Comp. Phys. Comm. 79(1994) 100]. His paper should be consulted for the details; only a rough outline of the key points is given here. The theory of C-systems and K-systems is given in Arnold and Avez, Ergodic Problems of Classical Mechanics, Addison-Wesley, The RCARRY algorithm RANLUX is based on RCARRY, the generator proposed by Marsaglia and Zaman. RCARRY has a period of the order of , is quite fast for a long-period generator, and was at first believed to have good statistical properties, but by the time Lüscher developed the RANLUX algorithm it was already known to fail some tests. The basic algorithm works on an array of 24 numbers, each of which is a floating-point number between zero and one, with a 24-bit mantissa (IEEE single-precision format). It can also work with an array of 24-bit integers which yield exactly the same output. The choice between floating-point and integer array depends on which is faster on the particular processor and does not have any effect on the actual numbers produced. Lüscher had the idea to consider RCARRY as a 24-dimensional discrete dynamical system, and study the properties of the corresponding continuous system. He gives convincing arguments why one can neglect the carry bit for the purpose of studying the dynamical system, and shows that the linear transformation defining the system dynamics can be represented as a matrix composed of elements all of which are either zero or ±1 and having the following properties: The determinant is one (area-conserving). All eigenvalues are complex and distinct. No eigenvalue has modulus one (hyperbolic map). There are four eigenvalues with maximal absolute value = One can conclude from the above that the continuous system is a C-system and therefore a K-system. The Kolmogorov entropy is positive and the Lyapunov exponent is

6 Note however, that all the above is true of RCARRY, the RNG proposed by Marsaglia and Zaman, which was already known (as pointed out in Phys. Rev. Lett. by Ferrenberg et. al.) to have defects. It is therefore not sufficient to identify the dynamical system as the discrete approximation to a continuous K-system. 6.2 Short-term correlations: The Lyapunov exponent The mixing property is only an asymptotic property, and the rate at which the mixing sets in is given by the Kolmogorov entropy. A system with positive Kolmogorov entropy loses information about its past exponentially fast. From Arnold and Avez, p. 43, we have that the entropy = i ν i, where the sum runs over the (positive) exponents of divergence, ν i = ln( λ i ), and λ i are the eigenvalues of the transformation matrix with modulus greater than 1. The Lyapunov exponent determines the rate of divergence of nearby trajectories in state space, and the non-randomness of RCARRY can be viewed as due to the Lyapunov exponent being too small. This can be seen graphically in a very striking way, by plotting the average separation as a function of time, of trajectories that start with minimum separation in the 24-dimensional state space (Fig.1 of Lüscher s paper). From this one sees that after generating a first set of 24 random numbers, one has to wait about sixteen iterations before the next set of 24 numbers is completely independent of the starting configuration. The proposed new generator therefore consists of generating 24 numbers with RCARRY, and throwing away the next = 365 numbers before accepting 24 more, etc. The number 389 is chosen because it is prime and this turns out to preserve the period of the generator in spite of wasting most of the numbers. This highest luxury level with P = 389 is guaranteed to give random numbers with no detectable defects. RANLUX also offers the option of choosing smaller values of P which result in a faster generator of lower quality. For example at P = 223 no defects have yet been detected, but the theory indicates that it could happen under sufficiently stringent conditions. 6.3 Long-term correlations: The period Unlike true continuous K-systems, which cannot have any long-term correlations, RCARRY and RANLUX are only discrete approximations to continuous K-systems. They therefore have a finite period and describe closed orbits in the 24-dimensional state space. Thus, like all RNG s with a finite period, they consume their state space as they progress through it, creating a forbidden zone they cannot re-enter, which violates the definition of K-mixing and introduces defects which grow as the algorithm generates more numbers. It is easy to estimate the size of the resulting correlations. Given that a million years is less than µsec, one can estimate that the total number of random numbers that will be generated on the earth in the next million years will not exceed 6

7 If all these were generated by RANLUX, it would not consume more than of its period. Long-term correlations should therefore exist, but at a level more than one hundred orders of magnitude too small to be detected. Another important point concerns the relation between the discrete trajectories and the full state space. Every discrete trajectory fills (almost) 1/48th of the full (discrete) space of states and thus is essentially ergodic. In the continuum system only almost every trajectory is guaranteed to fill the space; in this sense the discrete trajectories represent the typical case. 7 Testing RANLUX Of course it is important to test the random numbers produced by RANLUX. However, this testing is fundamentally different from the testing of traditional RNG s. Traditional RNG s must be tested because, apart from the testing, there is no reason to believe they are at all random. Indeed, it always turns out that even after extensive testing, defects are eventually found, and sometimes understood. Experience shows that testing is necessary but not sufficient. RANLUX, on the other hand, has a good underlying theory, so the purpose of testing is to make sure that the theory has been understood, applied and programmed correctly. Testing is not necessary to prove the randomness of the algorithm. 7.1 The spectral test It was known already by the time of Lüscher s paper that RCARRY is closely related to a linear congruential generator of enormous modulus and multiplier. It is therefore possible to apply the powerful spectral test to establish its underlying lattice structure. Since this is a full-period test, it is in some sense not very relevant for a generator for which it is physically impossible to generate the full period, but it is nevertheless good to know the lattice structure in low dimensions. While RCARRY fails this test badly, discarding improves the lattice structure exactly as one would expect from the Lyapunov exponent and the exponential divergence of nearby trajectories. 8 Related work It is interesting to consider the possible relation between RANLUX and other proposed generators besides RCARRY. The existing random number literature makes very little reference to Lüscher s work. One notable exception is Donald Knuth s Art of Computer Programming, of which volume two is devoted mostly to pseudorandom number generation. The latest edition of this classic makes several references 7

8 to Lüscher s work, and even makes the rather strong statement (in italics in the original!): This method has modest theoretical support and no known defects. However, upon closer inspection one notices that Knuth refers only to one aspect of Lüscher s theory (the discarding) and makes no mention of K-systems or Kolmogorov entropy or even Lyapunov exponents. He gives the impression that discarding is another trick, like shuffling, and can be used to improve any generator. He fails to mention, for example, that in the case of RANLUX, one can calculate exactly how many random numbers must be rejected in order to eliminate all detectable correlations. An important question is whether the theory of K-systems can explain why other generators work, and why they often don t work very well. To my knowledge, RCARRY is the only algorithm studied extensively from this angle, but it is likely that similar reasoning can be applied to other existing generators to explain their good and bad properties and to indicate whether they can be improved. In particular, ordinary Fibonacci generators that operate in the space of real numbers can immediately be mapped to dynamical systems. In all these cases the discrete trajectory only fills a very small part of the full space of discrete states, i.e. there are zillions of disjoint trajectories, an indication that one could expect long-term correlations. Bit-shuffling algorithms, such as the Mersenne Twistor are, on the other hand, not obviously related to an underlying continuous dynamical system. This kind of analysis could lead to finding better algorithms (for example, K- systems with bigger Lyapunov exponents requiring less discarding). To conclude, there is no doubt that reasoning based on dynamical systems gives the best understanding we currently have of why certain RNG s work. This does not exclude the possible existence of other theories that could shed more light on this question in the future, but it makes arguments for randomness based on testing look very weak. 8