Random numbers? What do we want?

Transcription

1 Random numbers? What do we want? 1 Produce randomly-looking sequences; eg: bit sequences: integers from 0 to 99 (say): 31, 83, 02, 72, 54, 26, real numbers between 0 and 1

2 Random numbers? What do we want? 1 Produce randomly-looking sequences; eg: bit sequences: integers from 0 to 99 (say): 31, 83, 02, 72, 54, 26, real numbers between 0 and 1 Physical devices: dices, balls, roulette wheels, thermal noise in resistances of electronic circuits, devices that rely on quantum physics, and so on

3 Random numbers? What do we want? 1 Produce randomly-looking sequences; eg: bit sequences: integers from 0 to 99 (say): 31, 83, 02, 72, 54, 26, real numbers between 0 and 1 Physical devices: dices, balls, roulette wheels, thermal noise in resistances of electronic circuits, devices that rely on quantum physics, and so on Contains true entropy, but not very convenient, not reproducible, not always reliable, and no (or little) mathematical analysis Can improve reliability by combining bits (XOR) Many such devices on the market

4 Random numbers? What do we want? 1 Produce randomly-looking sequences; eg: bit sequences: integers from 0 to 99 (say): 31, 83, 02, 72, 54, 26, real numbers between 0 and 1 Physical devices: dices, balls, roulette wheels, thermal noise in resistances of electronic circuits, devices that rely on quantum physics, and so on Contains true entropy, but not very convenient, not reproducible, not always reliable, and no (or little) mathematical analysis Can improve reliability by combining bits (XOR) Many such devices on the market Algorithmic generators (or pseudo-random): Once the parameters and the seed are selected, everything else becomes deterministic But much more convenient No special hardware required

5 1 Computer games: Good looking appearance may suffice 2

6 1 Computer games: Good looking appearance may suffice 2 2 Stochastic simulation (Monte Carlo): Simulate the behavior of complex systems Want to reproduce the relevant statistical properties of the mathematical model Typical RNG: imitates a sequence U 0, U 1, U 2, of independent random variables uniform over the interval (0,1)

7 1 Computer games: Good looking appearance may suffice 2 2 Stochastic simulation (Monte Carlo): Simulate the behavior of complex systems Want to reproduce the relevant statistical properties of the mathematical model Typical RNG: imitates a sequence U 0, U 1, U 2, of independent random variables uniform over the interval (0,1) 3 Lotteries, casino machines, Internet gambling, etc It should not be possible (or practical) to make an inference that provides an advantage in guessing the next numbers Stronger requirements than for simulation

8 1 Computer games: Good looking appearance may suffice 2 2 Stochastic simulation (Monte Carlo): Simulate the behavior of complex systems Want to reproduce the relevant statistical properties of the mathematical model Typical RNG: imitates a sequence U 0, U 1, U 2, of independent random variables uniform over the interval (0,1) 3 Lotteries, casino machines, Internet gambling, etc It should not be possible (or practical) to make an inference that provides an advantage in guessing the next numbers Stronger requirements than for simulation 4 Cryptology: Requirements are even stronger Observing any part the output should not help guessing (with reasonable effort) any other part

9 Requirements for stochastic simulation 3 We want to use an RNG to imitate a sequence U 0, U 1, U 2, of independent random variables, uniformly distributed over (0, 1)

10 Requirements for stochastic simulation 3 We want to use an RNG to imitate a sequence U 0, U 1, U 2, of independent random variables, uniformly distributed over (0, 1) To generate variates from other distributions, we apply transformations to these U j For example, inversion: if X j = F 1 (U j ), then X j imitates a random variable with distribution function F

11 Requirements for stochastic simulation 3 We want to use an RNG to imitate a sequence U 0, U 1, U 2, of independent random variables, uniformly distributed over (0, 1) To generate variates from other distributions, we apply transformations to these U j For example, inversion: if X j = F 1 (U j ), then X j imitates a random variable with distribution function F Simulation on multiple processors: need multiple streams of random numbers

12 Requirements for stochastic simulation 3 We want to use an RNG to imitate a sequence U 0, U 1, U 2, of independent random variables, uniformly distributed over (0, 1) To generate variates from other distributions, we apply transformations to these U j For example, inversion: if X j = F 1 (U j ), then X j imitates a random variable with distribution function F Simulation on multiple processors: need multiple streams of random numbers Also for comparing similar systems with common random numbers Suppose we simulate a communication network, or a telephone call center, or a supply chain or logistic system, or a factory, or the dynamic management of an investment portfolio, etc We want to compare two similar configurations (or control policies) for a given system The difference in performance will be due partly to the difference of configuration, and partly to stochastic noise We want to minimize this second part

13 Basic idea: simulate the two configurations with the same uniform random numbers U j, used at exactly the same places There are several theoretical results concerning the efficiency improvement (variance reduction) that this brings This requires good synchronization of the random numbers, and this can be complicated to implement and manage when the two configurations do not need the same number of U j s (eg, we may need one random variate in one case and not in the other case) 4

14 Basic idea: simulate the two configurations with the same uniform random numbers U j, used at exactly the same places There are several theoretical results concerning the efficiency improvement (variance reduction) that this brings This requires good synchronization of the random numbers, and this can be complicated to implement and manage when the two configurations do not need the same number of U j s (eg, we may need one random variate in one case and not in the other case) 4 A solution: RNG with multiple streams and substreams Each stream can be seen as a virtual RNG It is also partitioned into substreams We should be able to create as many independent streams as we want Current state start start next stream substream substream

15 Basic idea: simulate the two configurations with the same uniform random numbers U j, used at exactly the same places There are several theoretical results concerning the efficiency improvement (variance reduction) that this brings This requires good synchronization of the random numbers, and this can be complicated to implement and manage when the two configurations do not need the same number of U j s (eg, we may need one random variate in one case and not in the other case) 4 A solution: RNG with multiple streams and substreams Each stream can be seen as a virtual RNG It is also partitioned into substreams We should be able to create as many independent streams as we want Current state start start next stream substream substream

16 Basic idea: simulate the two configurations with the same uniform random numbers U j, used at exactly the same places There are several theoretical results concerning the efficiency improvement (variance reduction) that this brings This requires good synchronization of the random numbers, and this can be complicated to implement and manage when the two configurations do not need the same number of U j s (eg, we may need one random variate in one case and not in the other case) 4 A solution: RNG with multiple streams and substreams Each stream can be seen as a virtual RNG It is also partitioned into substreams We should be able to create as many independent streams as we want Current state start start next stream substream substream

17 Basic idea: simulate the two configurations with the same uniform random numbers U j, used at exactly the same places There are several theoretical results concerning the efficiency improvement (variance reduction) that this brings This requires good synchronization of the random numbers, and this can be complicated to implement and manage when the two configurations do not need the same number of U j s (eg, we may need one random variate in one case and not in the other case) 4 A solution: RNG with multiple streams and substreams Each stream can be seen as a virtual RNG It is also partitioned into substreams We should be able to create as many independent streams as we want Current state start start next stream substream substream

18 start start next stream substream substream Suppose that for a sequence of customers arriving to a system, we must generate: (1) the arrival time of each customer; (2) a service time at server A for each customer; (3) a service time at server B for groups of 10 to 20 customers To maintain synchonization, we can use one stream for each of these

19 5 Current state start start next stream substream substream Suppose that for a sequence of customers arriving to a system, we must generate: (1) the arrival time of each customer; (2) a service time at server A for each customer; (3) a service time at server B for groups of 10 to 20 customers To maintain synchonization, we can use one stream for each of these

20 Example of a Java interface (in SSJ) 6 public interface RandomStream { public void resetstartstream (); } Reinitializes the stream to its initial state public void resetstartsubstream (); Reinitializes the stream to the beginning of its current substream public void resetnextsubstream (); Reinitializes the stream to the beginning of its next substream public double nextdouble (); Returns a U(0, 1) (pseudo)random number, using this stream, after advancing its state by one step public int nextint (int i, int j); Returns a (pseudo)random number from the discrete uniform distribution over the integers {i, i + 1,, j}, using this stream

21 public class MRG32k3a implements RandomStream { } An implementation based on a combined MRG with period length near 2 181, partitioned into streams whose initial states are steps apart The substreams have length 2 76 public MRG32k3a(); Constructs a new stream 7 public class LFSR113 implements RandomStream { } An implementation based on a combined LFSR, with period length near public LFSR113(); Constructs a new stream This system (with MRG32k3a) has been adopted by leading simulation and statistical software such as SAS, Arena, Witness, Simul8, Automod, ns2, MATLAB, etc

22 Non-uniform variate generators 8 Continuous or discrete Default method: inversion public class RandomVariateGen { public RandomVariateGen (RandomStream s, Distribution dist) Creates a new generator from the distribution dist, using stream s public double nextdouble() By default, generates a variate by inversion

23 Non-uniform variate generators 8 Continuous or discrete Default method: inversion public class RandomVariateGen { public RandomVariateGen (RandomStream s, Distribution dist) Creates a new generator from the distribution dist, using stream s public double nextdouble() By default, generates a variate by inversion Also specialized (faster) generators for certain distributions public class Normal extends RandomVariateGen { public Normal (RandomStream s, double mu, double sigma); Constructs a new normal rv generator public static double double sigma); Generates a new normal random variate using stream s nextdouble (RandomStream s, double mu,

24 Example: comparing two configurations: 9 RandomStream genarr = new RandomStream(); // Inter-arriv RandomStream genserva = new RandomStream(); // Serveur A RandomStream genservb = new RandomStream(); // Serveur B

25 Example: comparing two configurations: 9 RandomStream genarr = new RandomStream(); // Inter-arriv RandomStream genserva = new RandomStream(); // Serveur A RandomStream genservb = new RandomStream(); // Serveur B for (int rep = 0; rep < n; rep++) { genarrresetnextsubstream(); genservaresetnextsubstream(); genservbresetnextsubstream(); --- simuler configuration } genarrresetstartsubstream(); genservaresetstartsubstream(); genservbresetstartsubstream(); --- simuler configuration 2 ---

26 class BatchServer { RandomStream genarr = new RandomStream(); // Interarrival times RandomStream genserva = new RandomStream(); // For server 1 RandomStream genservb = new RandomStream(); // For server 2 Tally statsojourn = new Tally ("Sojourn times in one run"); Tally statdiff = new Tally ("Differences on averages"); 10

27 class BatchServer { RandomStream genarr = new RandomStream(); // Interarrival times RandomStream genserva = new RandomStream(); // For server 1 RandomStream genservb = new RandomStream(); // For server 2 Tally statsojourn = new Tally ("Sojourn times in one run"); Tally statdiff = new Tally ("Differences on averages"); public void simulatediffcrn (int n) { for (int rep = 0; rep < n; rep++) { setserviceparamb ( 0075); simulateonerun(); double mean1 = statsojournaverage(); 10

28 class BatchServer { RandomStream genarr = new RandomStream(); // Interarrival times RandomStream genserva = new RandomStream(); // For server 1 RandomStream genservb = new RandomStream(); // For server 2 Tally statsojourn = new Tally ("Sojourn times in one run"); Tally statdiff = new Tally ("Differences on averages"); public void simulatediffcrn (int n) { for (int rep = 0; rep < n; rep++) { setserviceparamb ( 0075); simulateonerun(); double mean1 = statsojournaverage(); genarr resetstartsubstream(); genserv1resetstartsubstream(); genserv2resetstartsubstream(); setserviceparamb ( 0080); simulateonerun(); statdiffadd (statsojournaverage() - mean1); 10

29 class BatchServer { RandomStream genarr = new RandomStream(); // Interarrival times RandomStream genserva = new RandomStream(); // For server 1 RandomStream genservb = new RandomStream(); // For server 2 Tally statsojourn = new Tally ("Sojourn times in one run"); Tally statdiff = new Tally ("Differences on averages"); public void simulatediffcrn (int n) { for (int rep = 0; rep < n; rep++) { setserviceparamb ( 0075); simulateonerun(); double mean1 = statsojournaverage(); genarr resetstartsubstream(); genserv1resetstartsubstream(); genserv2resetstartsubstream(); setserviceparamb ( 0080); simulateonerun(); statdiffadd (statsojournaverage() - mean1); genarr resetnextsubstream(); genserv1resetnextsubstream(); genserv2resetnextsubstream(); } } 10

30 class BatchServer { } RandomStream genarr = new RandomStream(); // Interarrival times RandomStream genserva = new RandomStream(); // For server 1 RandomStream genservb = new RandomStream(); // For server 2 Tally statsojourn = new Tally ("Sojourn times in one run"); Tally statdiff = new Tally ("Differences on averages"); public void simulatediffcrn (int n) { for (int rep = 0; rep < n; rep++) { setserviceparamb ( 0075); simulateonerun(); double mean1 = statsojournaverage(); genarr resetstartsubstream(); genserv1resetstartsubstream(); genserv2resetstartsubstream(); setserviceparamb ( 0080); simulateonerun(); statdiffadd (statsojournaverage() - mean1); genarr resetnextsubstream(); genserv1resetnextsubstream(); genserv2resetnextsubstream(); } } static public void main (String[] args) { BatchServer q = new BatchServer (); qsimulatediffcrn (10000); // pairs of runs with CRNs Systemoutprintln (qstatdiffreportandcistudent (095)); } 10

31 } private void simulateonerun () { statsojourninit(); serverainit(); binservbinit(); Siminit(); serverb = new ServerB(); serverbschedule(00); new Customer()schedule(00); Simstart(); } class Customer extends Process { public void actions() { new Customer()schedule (ExponentialGennextDouble (genarr, arrivalrate)); double arrivaltime = Simtime(); serverarequest(1); delay (ExponentialGennextDouble (genserva, serviceparama)); serverarelease(1); if (binservbwaitlist()size() >= 9 &&!serverbbusy) serverbresume(); binservbtake(1); // Blocked until service ends statsojournupdate (Simtime() - arrivaltime); if (statsojournnumberobs() >= numcust) Simstop(); } } class ServerB extends Process { boolean busy = false; int batchsize; // Current batch size public void actions() { while (true) { if (binservbwaitlist()size() < 10) { busy = false; suspend(); // Wait for enough customers }; busy = true; // Starts serving new batch batchsize = Mathmin (20, binservbwaitlist()size()); delay (ExponentialGennextDouble (genservb, serviceparamb)); binservbput (batchsize); // Unblocks customers in batch } } } 11

32 Multiple streams are very useful (on a single processor), for example for: 12 comparing similar systems sensitivity analysis, gradient estimation by finite differences optimization of sample function, etc using a (simplified) similar system for an external control variate

33 Multiple streams are very useful (on a single processor), for example for: 12 comparing similar systems sensitivity analysis, gradient estimation by finite differences optimization of sample function, etc using a (simplified) similar system for an external control variate Can also be used on multiple processors

34 Multiple streams are very useful (on a single processor), for example for: 12 comparing similar systems sensitivity analysis, gradient estimation by finite differences optimization of sample function, etc using a (simplified) similar system for an external control variate Can also be used on multiple processors In SSJ, a RandomStream can also be an iterator on a randomized quasi-monte Carlo point set

35 How to Define and Design an RNG? 13 A (Pseudo)random number generator (RNG): S, finite state space; s 0, seed (initial state); f : S S, transition function; s n = f(s n 1 ) U, output set, often U = (0, 1); g : S U, output function u n = g(s n )

36 How to Define and Design an RNG? 13 A (Pseudo)random number generator (RNG): S, finite state space; s 0, seed (initial state); f : S S, transition function; s n = f(s n 1 ) U, output set, often U = (0, 1); g : S U, output function u n = g(s n ) s 0

37 How to Define and Design an RNG? 13 A (Pseudo)random number generator (RNG): S, finite state space; s 0, seed (initial state); f : S S, transition function; s n = f(s n 1 ) U, output set, often U = (0, 1); g : S U, output function u n = g(s n ) g s 0 u 0

38 How to Define and Design an RNG? 13 A (Pseudo)random number generator (RNG): S, finite state space; s 0, seed (initial state); f : S S, transition function; s n = f(s n 1 ) U, output set, often U = (0, 1); g : S U, output function u n = g(s n ) g s 0 f s 1 u 0

39 How to Define and Design an RNG? 13 A (Pseudo)random number generator (RNG): S, finite state space; s 0, seed (initial state); f : S S, transition function; s n = f(s n 1 ) U, output set, often U = (0, 1); g : S U, output function u n = g(s n ) g s 0 f s 1 g u 0 u 1

40 How to Define and Design an RNG? 13 A (Pseudo)random number generator (RNG): S, finite state space; s 0, seed (initial state); f : S S, transition function; s n = f(s n 1 ) U, output set, often U = (0, 1); g : S U, output function u n = g(s n ) g s 0 f s 1 g f f s n g f s n+1 g f u 0 u 1 u n u n+1 Period: ρ S s i+ρ = s i i τ Assume τ = 0

41 How to Define and Design an RNG? 13 A (Pseudo)random number generator (RNG): S, finite state space; s 0, seed (initial state); f : S S, transition function; s n = f(s n 1 ) U, output set, often U = (0, 1); g : S U, output function u n = g(s n ) f s ρ 1 g f s 0 g f s 1 g f f s n g f s n+1 g f u ρ 1 u 0 u 1 u n u n+1 Period: ρ S s i+ρ = s i i τ Assume τ = 0

42 14 f f f f f f f s ρ 1 s 0 s 1 s n s n+1 g g g g g u ρ 1 u 0 u 1 u n u n+1 Aim: By observing only the output (u 0, u 1, ), it should be hard to distinguish it from the realizations of iid uniform random variables over U Utopia: cannot distinguish better than by flipping a fair coin That is, passes all statistical tests

43 14 f f f f f f f s ρ 1 s 0 s 1 s n s n+1 g g g g g u ρ 1 u 0 u 1 u n u n+1 Aim: By observing only the output (u 0, u 1, ), it should be hard to distinguish it from the realizations of iid uniform random variables over U Utopia: cannot distinguish better than by flipping a fair coin That is, passes all statistical tests Also want high speed, ease of implementation, and perfect reproducibility Compromise between speed / good statistical behavior / predictability

44 14 f f f f f f f s ρ 1 s 0 s 1 s n s n+1 g g g g g u ρ 1 u 0 u 1 u n u n+1 Aim: By observing only the output (u 0, u 1, ), it should be hard to distinguish it from the realizations of iid uniform random variables over U Utopia: cannot distinguish better than by flipping a fair coin That is, passes all statistical tests Also want high speed, ease of implementation, and perfect reproducibility Compromise between speed / good statistical behavior / predictability With random seed s 0, an RNG is like a gigantic roulette wheel Selecting s 0 at random and generating t random numbers means spinning the wheel and taking u = (u 0,, u t 1 )

45 The uniform distribution over [0, 1] t : The multiset Ψ t = {(u 0,, u t 1 ) = (g(s 0 ),, g(s t 1 )), s 0 S} is viewed as the sample space, approximation of [0, 1] t 15

46 The uniform distribution over [0, 1] t : The multiset Ψ t = {(u 0,, u t 1 ) = (g(s 0 ),, g(s t 1 )), s 0 S} is viewed as the sample space, approximation of [0, 1] t 15 We suggest: Want Ψ t uniformly spread over [0, 1] t for all t up to some t 0

47 The uniform distribution over [0, 1] t : The multiset Ψ t = {(u 0,, u t 1 ) = (g(s 0 ),, g(s t 1 )), s 0 S} is viewed as the sample space, approximation of [0, 1] t 15 We suggest: Want Ψ t uniformly spread over [0, 1] t for all t up to some t 0 Need a measure of uniformity of Ψ t (or a measure of discrepancy from the uniform distribution) Several possible definitions Important: Must be efficiently computable For this, must understand the mathematical structure of Ψ t This is why most good RNGs are based on linear recurrences

48 The uniform distribution over [0, 1] t : The multiset Ψ t = {(u 0,, u t 1 ) = (g(s 0 ),, g(s t 1 )), s 0 S} is viewed as the sample space, approximation of [0, 1] t 15 We suggest: Want Ψ t uniformly spread over [0, 1] t for all t up to some t 0 Need a measure of uniformity of Ψ t (or a measure of discrepancy from the uniform distribution) Several possible definitions Important: Must be efficiently computable For this, must understand the mathematical structure of Ψ t This is why most good RNGs are based on linear recurrences Why not insist that Ψ t behaves as a typical set of random points instead? We need that behavior only for the tiny fraction of Ψ t that we use

49 The uniform distribution over [0, 1] t : The multiset Ψ t = {(u 0,, u t 1 ) = (g(s 0 ),, g(s t 1 )), s 0 S} is viewed as the sample space, approximation of [0, 1] t 15 We suggest: Want Ψ t uniformly spread over [0, 1] t for all t up to some t 0 Need a measure of uniformity of Ψ t (or a measure of discrepancy from the uniform distribution) Several possible definitions Important: Must be efficiently computable For this, must understand the mathematical structure of Ψ t This is why most good RNGs are based on linear recurrences Why not insist that Ψ t behaves as a typical set of random points instead? We need that behavior only for the tiny fraction of Ψ t that we use Generalization: measure the uniformity of Ψ I = {(u i1,, u it ) s 0 S} for selected sets I = {i 1, i 2,, i t } of nonsuccessive indexes Make sure that Ψ I is uniform for all I J, for a given family J

50 Multiple Recursive Generator (MRG) 16 x n = (a 1 x n a k x n k ) mod m, u n = x n /m Max period length: ρ = m k 1, for m prime

51 Multiple Recursive Generator (MRG) 16 x n = (a 1 x n a k x n k ) mod m, u n = x n /m Max period length: ρ = m k 1, for m prime Structure of Ψ t : (x 0,, x k 1 ) can take any value in {0, 1,, m 1} k, then x k, x k+1, are determined by the linear recurrence So, (x 0,, x k 1 ) (x 0,, x k 1, x k,, x t 1 ) is a linear mapping

52 It follows (details omitted) that Ψ t = L t [0, 1) t where 17 L t = { v = t z i v i z i Z }, a lattice in R t, with i=1 v 1 = (1, 0,, 0, x 1,k,, x 1,t 1 )/m v 2 = (0, 1,, 0, x 2,k,, x 2,t 1 )/m v k = (0, 0,, 1, x k,k,, x k,t 1 )/m v k+1 = (0, 0,, 0, 1,, 0) v t = (0, 0,, 0, 0,, 1)

53 u n 0 1 u n+1 LCG with m = 101 and a 1 = 12;

54 u n 0 1 u n+1 LCG with m = 101 and a 1 = 12; v 1 = (1, 12)/m, v 2 = (0, 1)

55 u n u n+1 x n = x n 1 mod and u n = x n /

56 u n u n+1 x n = x n 1 mod and u n = x n /

57 Uniformity can be measured in terms of this lattice structure in various dimensions 21

58 Uniformity can be measured in terms of this lattice structure in various dimensions 21 Can also take into account projections on subsets of coordinates I = {i 1, i 2,, i t }: Ψ I = {(u i1,, u it ) s 0 = (x 0,, x k 1 ) Z k m} = L I [0, 1) t Can make computer searches for good parameters wrt a criterion that account for the uniformity of a selected set of projections

59 Uniformity can be measured in terms of this lattice structure in various dimensions 21 Can also take into account projections on subsets of coordinates I = {i 1, i 2,, i t }: Ψ I = {(u i1,, u it ) s 0 = (x 0,, x k 1 ) Z k m} = L I [0, 1) t Can make computer searches for good parameters wrt a criterion that account for the uniformity of a selected set of projections Remark: it can be proved that if 1 + a 2 i a 2 i t is small, then the lattice structure of Ψ I is bad

60 Efficient Implementations 22 Requires efficient computation of ax mod m for large m

61 Efficient Implementations 22 Requires efficient computation of ax mod m for large m Approximate factoring Valid if (a 2 < m) or (a = m/i where i 2 < m) Uses integer arithmetic Precompute q = m/a and r = m mod a Then, y = x/q ; x = a(x yq) yr; if x < 0 then x = x + m

62 Efficient Implementations 22 Requires efficient computation of ax mod m for large m Approximate factoring Valid if (a 2 < m) or (a = m/i where i 2 < m) Uses integer arithmetic Precompute q = m/a and r = m mod a Then, y = x/q ; x = a(x yq) yr; if x < 0 then x = x + m Floating point in double precision Valid if a m < 2 53 double m, a, x, y; int k; y = a x; k = y/m ; x = y k m;

63 Efficient Implementations 22 Requires efficient computation of ax mod m for large m Approximate factoring Valid if (a 2 < m) or (a = m/i where i 2 < m) Uses integer arithmetic Precompute q = m/a and r = m mod a Then, y = x/q ; x = a(x yq) yr; if x < 0 then x = x + m Floating point in double precision Valid if a m < 2 53 double m, a, x, y; int k; y = a x; k = y/m ; x = y k m; Straightforward 64-bit integer arithmetic On 64-bit computers, just make sure that am < 2 63

64 Decomposition in powers of 2 Suppose a = ±2 q ± 2 r and m = 2 e h for small h (Wu 1997 for h = 1; L Ecuyer and Simard 1999 for h > 1) To compute y = 2 q x mod m, decompose x = x e q x 1 ; 23 q bits (e q) bits x = x 1 x 0 For h = 1, y is obtained by swapping x 0 and x 1 For h > 1, requires a single multiplication of x 1 by h, plus a few shifts, masks, additions, and subtractions

65 Decomposition in powers of 2 Suppose a = ±2 q ± 2 r and m = 2 e h for small h (Wu 1997 for h = 1; L Ecuyer and Simard 1999 for h > 1) To compute y = 2 q x mod m, decompose x = x e q x 1 ; 23 q bits (e q) bits x = x 1 x 0 For h = 1, y is obtained by swapping x 0 and x 1 For h > 1, requires a single multiplication of x 1 by h, plus a few shifts, masks, additions, and subtractions

66 Lagged-Fibonacci (widely used, but bad idea): 24 x n = (±x n r ± x n k ) mod m All vectors (u n, u n+k r, u n+k ) lie in only two planes!

67 Lagged-Fibonacci (widely used, but bad idea): 24 x n = (±x n r ± x n k ) mod m All vectors (u n, u n+k r, u n+k ) lie in only two planes! Same problem with add-with-carry and subtract-with-borrow RNGs Common mistake: if we cannot exhaust the period, then we are okay

68 Lagged-Fibonacci (widely used, but bad idea): 24 x n = (±x n r ± x n k ) mod m All vectors (u n, u n+k r, u n+k ) lie in only two planes! Same problem with add-with-carry and subtract-with-borrow RNGs Common mistake: if we cannot exhaust the period, then we are okay Variants that skip values are recommended by Luscher (1994) and Knuth (1997)

69 Lagged-Fibonacci (widely used, but bad idea): 24 x n = (±x n r ± x n k ) mod m All vectors (u n, u n+k r, u n+k ) lie in only two planes! Same problem with add-with-carry and subtract-with-borrow RNGs Common mistake: if we cannot exhaust the period, then we are okay Variants that skip values are recommended by Luscher (1994) and Knuth (1997) Safer AWC/SWB versions proposed by Couture and L Ecuyer (1995), Klapper and Goresky (2003) But no significant advantage over MRGs

70 Combined MRGs Consider two MRGs (or more) evolving in parallel: 25 Define the two combinations: x 1,n = (a 1,1 x 1,n a 1,k x 1,n k ) mod m 1, x 2,n = (a 2,1 x 2,n a 2,k x 2,n k ) mod m 2 z n := (x 1,n x 2,n ) mod m 1 ; u n := z n /m 1 ; w n := (x 1,n /m 1 x 2,n /m 2 ) mod 1

71 Combined MRGs Consider two MRGs (or more) evolving in parallel: 25 Define the two combinations: x 1,n = (a 1,1 x 1,n a 1,k x 1,n k ) mod m 1, x 2,n = (a 2,1 x 2,n a 2,k x 2,n k ) mod m 2 z n := (x 1,n x 2,n ) mod m 1 ; u n := z n /m 1 ; w n := (x 1,n /m 1 x 2,n /m 2 ) mod 1 One can show (L Ecuyer 1996) that {w n, n 0} is the output sequence of yet another MRG, with modulus m = m 1 m 2, and {u n, n 0} is almost the same sequence if m 1 and m 2 are close Can reach period length (m k 1 1)(m k 2 1)/2

72 Combined MRGs Consider two MRGs (or more) evolving in parallel: 25 x 1,n = (a 1,1 x 1,n a 1,k x 1,n k ) mod m 1, x 2,n = (a 2,1 x 2,n a 2,k x 2,n k ) mod m 2 Define the two combinations: z n := (x 1,n x 2,n ) mod m 1 ; u n := z n /m 1 ; w n := (x 1,n /m 1 x 2,n /m 2 ) mod 1 One can show (L Ecuyer 1996) that {w n, n 0} is the output sequence of yet another MRG, with modulus m = m 1 m 2, and {u n, n 0} is almost the same sequence if m 1 and m 2 are close Can reach period length (m k 1 1)(m k 2 1)/2 Permits efficient implementation of an MRG with large m and several large nonzero coefficients

73 Combined MRGs Consider two MRGs (or more) evolving in parallel: 25 x 1,n = (a 1,1 x 1,n a 1,k x 1,n k ) mod m 1, x 2,n = (a 2,1 x 2,n a 2,k x 2,n k ) mod m 2 Define the two combinations: z n := (x 1,n x 2,n ) mod m 1 ; u n := z n /m 1 ; w n := (x 1,n /m 1 x 2,n /m 2 ) mod 1 One can show (L Ecuyer 1996) that {w n, n 0} is the output sequence of yet another MRG, with modulus m = m 1 m 2, and {u n, n 0} is almost the same sequence if m 1 and m 2 are close Can reach period length (m k 1 1)(m k 2 1)/2 Permits efficient implementation of an MRG with large m and several large nonzero coefficients Tables of good parameters and codes in L Ecuyer (1999) and L Ecuyer and Touzin (2000)

74 Combined MRGs Consider two MRGs (or more) evolving in parallel: 25 x 1,n = (a 1,1 x 1,n a 1,k x 1,n k ) mod m 1, x 2,n = (a 2,1 x 2,n a 2,k x 2,n k ) mod m 2 Define the two combinations: z n := (x 1,n x 2,n ) mod m 1 ; u n := z n /m 1 ; w n := (x 1,n /m 1 x 2,n /m 2 ) mod 1 One can show (L Ecuyer 1996) that {w n, n 0} is the output sequence of yet another MRG, with modulus m = m 1 m 2, and {u n, n 0} is almost the same sequence if m 1 and m 2 are close Can reach period length (m k 1 1)(m k 2 1)/2 Permits efficient implementation of an MRG with large m and several large nonzero coefficients Tables of good parameters and codes in L Ecuyer (1999) and L Ecuyer and Touzin (2000) Packages with multiple streams are available

75 MRG32k3a (L Ecuyer 1999) J = 2, k = 3, m 1 = , a 11 = 0, a 12 = , a 13 = , m 2 = , a 21 = , a 22 = 0, a 23 =

76 MRG32k3a (L Ecuyer 1999) J = 2, k = 3, m 1 = , a 11 = 0, a 12 = , a 13 = , m 2 = , a 21 = , a 22 = 0, a 23 = Combination: z n = (x 1,n x 2,n ) mod m 1

77 MRG32k3a (L Ecuyer 1999) J = 2, k = 3, m 1 = , a 11 = 0, a 12 = , a 13 = , m 2 = , a 21 = , a 22 = 0, a 23 = Combination: z n = (x 1,n x 2,n ) mod m 1 Corresponding MRG: k = 3, m = m 1 m 2 = , a 1 = , a 2 = , a 3 =

78 MRG32k3a (L Ecuyer 1999) J = 2, k = 3, m 1 = , a 11 = 0, a 12 = , a 13 = , m 2 = , a 21 = , a 22 = 0, a 23 = Combination: z n = (x 1,n x 2,n ) mod m 1 Corresponding MRG: k = 3, m = m 1 m 2 = , a 1 = , a 2 = , a 3 = Period ρ = (m 3 1 1)(m 3 2 1)/

79 #define norm e-10 /* 1/(m1+1) */ #define m #define m #define a #define a13n #define a #define a23n double s10, s11, s12, s20, s21, s22; double MRG32k3a () { long k; double p1, p2; /* Component 1 */ p1 = a12 * s11 - a13n * s10; k = p1 / m1; p1 -= k * m1; if (p1 < 00) p1 += m1; s10 = s11; s11 = s12; s12 = p1; /* Component 2 */ p2 = a21 * s22 - a23n * s20; k = p2 / m2; p2 -= k * m2; if (p2 < 00) p2 += m2; s20 = s21; s21 = s22; s22 = p2; /* Combination */ if (p1 <= p2) return ((p1 - p2 + m1) * norm); else return ((p1 - p2) * norm); } 27

80 Jumping ahead (to make streams and substreams): If x n = (x n k+1,, x n ) t and 28 A = a k a k 1 a 1 then and therefore x n = Ax n 1 mod m x n+ν = (A ν mod m)x n mod m

81 Generators Based on Linear Recurrences in F 2 29 Matrix linear recurrence over F 2 ( {0, 1}, mod 2): x n = Ax n 1, (k-bit state vector) y n = Bx n, (w-bit output vector) u n = w y n,j 1 2 j = y n,0 y n,1 y n,2, (output) j=1

82 Generators Based on Linear Recurrences in F 2 29 Matrix linear recurrence over F 2 ( {0, 1}, mod 2): x n = Ax n 1, (k-bit state vector) y n = Bx n, (w-bit output vector) u n = w y n,j 1 2 j = y n,0 y n,1 y n,2, (output) j=1 Each coordinate of x n and of y n follows the linear recurrence x n,j = (α 1 x n 1,j + + α k x n k,j ), with characteristic polynomial P (z) = z k α 1 z k 1 α k 1 z α k = det(a zi) Max period length ρ = 2 k 1 is reached iff P (z) is primitive over F 2

83 With clever choice of A, implementation involves bit shifts, xor s, and s, etc Special cases: Tausworthe, linear feedback shift register (LFSR), GFSR, twisted GFSR, Mersenne twister, xorshift, polynomial LCG, etc 30

84 With clever choice of A, implementation involves bit shifts, xor s, and s, etc Special cases: Tausworthe, linear feedback shift register (LFSR), GFSR, twisted GFSR, Mersenne twister, xorshift, polynomial LCG, etc 30 Would also like the number N 1 of nonzero α j s to be roughly around k/2

85 With clever choice of A, implementation involves bit shifts, xor s, and s, etc Special cases: Tausworthe, linear feedback shift register (LFSR), GFSR, twisted GFSR, Mersenne twister, xorshift, polynomial LCG, etc 30 Would also like the number N 1 of nonzero α j s to be roughly around k/2 To jump ahead, again: x n+ν = (X ν mod 2) }{{} precompute x n mod 2 This is time-expensive when k is large A more efficient algorithm proposed by Haramoto, L Ecuyer, Matsumoto, Nishimura, Panneton (2006)

86 Equidistribution For j = 1,, t, partition the jth axis of [0, 1) t into 2 q j equal parts: this gives 2 q boxes, where q = q q t If each box contains exactly 2 k q points from Ψ t, then Ψ t is called (q 1,, q t )-equidistributed Means that all t-dim vectors, up to q j bits of resolution for each coordinate j, appear the same number of times 31

87 Equidistribution For j = 1,, t, partition the jth axis of [0, 1) t into 2 q j equal parts: this gives 2 q boxes, where q = q q t If each box contains exactly 2 k q points from Ψ t, then Ψ t is called (q 1,, q t )-equidistributed Means that all t-dim vectors, up to q j bits of resolution for each coordinate j, appear the same number of times 31 Can express the q bits of interest as Mx 0 for some matrix M One has (q 1,, q t )-equidistribution iff M has full rank

88 32 00 u n u n+1 10 Toy example: LFSR generator with Ψ t = 1024 = 2 10

89 32 00 u n u n+1 10 Toy example: LFSR generator with Ψ t = 1024 = 2 10

90 32 00 u n u n+1 10 Toy example: LFSR generator with Ψ t = 1024 =

91 32 00 u n u n+1 10 Toy example: LFSR generator with Ψ t = 1024 =

92 For general set of indices I = {i 1, i 2,, i t }, resolution gap: δ I = min( k/t, w) max{l : Ψ I is (l,, l)-equidist} Potential figures of merit: 33 J = max I J δ I or V J = I J δ I where J is a selected class of sets I Choice of J is a question of compromise

93 Tausworthe or LFSR generator (Tausworthe 1965): 34 x n = (a 1 x n a k x n k ) mod 2, u n = x nν+j 1 2 j, j=1 A = 1 1 ν a k a k 1 a 1 Often, only two nonzero a j s: bad!

94 Generalized feedback shift register (GFSR) (Lewis and Payne 1973): 35 v n = (a 1 v n a r v n r ) mod 2 = (v n,0,, v n,w 1 ) t, y n = v n, u n = w v n,j 1 2 j j=1 State has rw bits, but maximal period length is 2 r 1 Usually, only two nonzero a j s: v n = (a q v n q + a r v n r ) mod 2 characteristic polynomial is too lean

95 Twisted GFSR (Matsumoto and Kurita 1992, 1994): 36 v n = (v n+m r + Av n r ) mod 2 y n = v n or y n = T v n Maximal period length is 2 rw 1 Flag carrier: TT800, with period length

96 Mersenne Twister (Matsumoto and Nishimura 1998): 37 v n = (v n+m r + A(v (w p) n r v (p) n r+1 )) mod 2 y n = T v n Maximal period length is 2 rw p 1 Flag carrier: MT19937, with period length

97 Linear output tempering Take (carefully selected) B I to improve the uniformity of the output 38 Example: Matsumoto and Kurita s tempering for the TGFSR: x n = trunc w (x n ) s n = x n (x n u) r n = s n ((s n s 1 ) & b) t n = r n ((r n s 2 ) & c) y n = t n (t n l) where w = 32, s 1,, s 4 are integers, and b and c are bit vectors

98 The WELL RNGs (Well-Equidistributed Long-period Linear) (Panneton, L Ecuyer, et Matsumoto 2004) Idea: Build the transition matrix A by placing simple operations on 32-bit blocs (exclusive-or s, shifts, bit masks, etc) at strategic places We optimise the figure of merit 39 1 = w (min( k/l, w) max{t : Ψ t is (l,, l)-equidist}), l=1 under the constraints: (1) maximal period and (2) comparable speed with MT19937 Specific generators for period lengths ranging from to Better equidistribution than MT19937 for same speed and period length Also larger number N 1 of nonzero coefficients in P (z)

99 Examples of WELL generators, vs MT, for w = 32 bits: 40 Name k r N 1 1 WELL19937a WELL19937c WELL44497a WELL44497b MT

100 Impact of N 1 too small and/or 1 too large Experiment: start from an initial state with a single bit at 1 Try all k possibilities and average the outputs after each transition WELL19937 vs MT19937; 06 moving averate over 1000 transitions Number of transitions

101 Combined F 2 -Linear Generators 42 Run two or more smaller generators with relatively prime period lengths Combine outputs by bitwise xor Equivalent to a single larger F 2 -linear generator Advantages: easier to implement and jump ahead with smaller components

102 Nonlinear Generators 43 Cubic Congruential: x n = (ax 3 n 1 + 1) mod m, u n = x n /m Explicit Inversive: x n = (an + c) mod m, u n = (x 1 n mod m)/m Arbitrary permutation implemented as a table Can combine several tables: efficient AES, SHA-1, etc Used in cryptology Chaotic dynamical systems? No

103 Combined linear/nonlinear generators 44 All the F 2 -linear generators discussed here fail (of course) a statistical test that measures the (binary) linear complexity

104 Combined linear/nonlinear generators 44 All the F 2 -linear generators discussed here fail (of course) a statistical test that measures the (binary) linear complexity We would like: to eliminate this linear structure; to keep some theoretical guarantees for the uniformity; a fast implantation

105 Combined linear/nonlinear generators 44 All the F 2 -linear generators discussed here fail (of course) a statistical test that measures the (binary) linear complexity We would like: to eliminate this linear structure; to keep some theoretical guarantees for the uniformity; a fast implantation L Ecuyer and Granger-Picher (2003): Large F 2 -linear generator combined with a small nonlinear one, via XOR

106 Combined linear/nonlinear generators 44 All the F 2 -linear generators discussed here fail (of course) a statistical test that measures the (binary) linear complexity We would like: to eliminate this linear structure; to keep some theoretical guarantees for the uniformity; a fast implantation L Ecuyer and Granger-Picher (2003): Large F 2 -linear generator combined with a small nonlinear one, via XOR Theorem: If the linear component is (q 1,, q t )-equidistributed, then this holds also for the combination

107 Combined linear/nonlinear generators 44 All the F 2 -linear generators discussed here fail (of course) a statistical test that measures the (binary) linear complexity We would like: to eliminate this linear structure; to keep some theoretical guarantees for the uniformity; a fast implantation L Ecuyer and Granger-Picher (2003): Large F 2 -linear generator combined with a small nonlinear one, via XOR Theorem: If the linear component is (q 1,, q t )-equidistributed, then this holds also for the combination Empirical tests: excellent behavior, more robust than linear generators

108 Some generators used in SSJ 45 gen time: CPU time to generate 10 9 uniform random numbers jump time: time to get a new stream (jump ahead) 10 6 times

109 Some generators used in SSJ 45 gen time: CPU time to generate 10 9 uniform random numbers jump time: time to get a new stream (jump ahead) 10 6 times Java JDK 15, 24 GHz 64-bit computer RNG period gen time jump time LFSR WELL WELL LFSR MT * MRG31k3p MRG32k3a RandRijndael

110 Empirical statistical Tests 46 Hypothesis H 0 : {u 0, u 1, u 2, } are iid U(0, 1) rv s We know that H 0 is false, but can we detect it?

111 Empirical statistical Tests 46 Hypothesis H 0 : {u 0, u 1, u 2, } are iid U(0, 1) rv s We know that H 0 is false, but can we detect it? Test: Define a statistic T, function of the u i, whose distribution under H 0 is known (or approx) Reject H 0 if value of T is too extreme Power and efficiency of the test strongly depend on the class of alternatives Different tests detect different deficiencies

112 Empirical statistical Tests 46 Hypothesis H 0 : {u 0, u 1, u 2, } are iid U(0, 1) rv s We know that H 0 is false, but can we detect it? Test: Define a statistic T, function of the u i, whose distribution under H 0 is known (or approx) Reject H 0 if value of T is too extreme Power and efficiency of the test strongly depend on the class of alternatives Different tests detect different deficiencies Utopian ideal: T mimics the rv of practical interest Not easy Ultimate dream: Build an RNG that passes all the tests? Formally impossible

113 Empirical statistical Tests 46 Hypothesis H 0 : {u 0, u 1, u 2, } are iid U(0, 1) rv s We know that H 0 is false, but can we detect it? Test: Define a statistic T, function of the u i, whose distribution under H 0 is known (or approx) Reject H 0 if value of T is too extreme Power and efficiency of the test strongly depend on the class of alternatives Different tests detect different deficiencies Utopian ideal: T mimics the rv of practical interest Not easy Ultimate dream: Build an RNG that passes all the tests? Compromise: Build an RNG that passes most reasonable tests Tests that fail are hard to find Formalization: computational complexity framework Formally impossible

114 Example: A Collision Test 47 Partition the box [0, 1) t into k = d t cubic boxes of equal sizes Generate n points (u i,, u i+t 1 ) in [0, 1) t Let X j = number of points in box j Number of collisions: C = k 1 j=0 max(0, X j 1) Under H 0, C Poisson with mean λ = n 2 /k, for large k, small λ If we observe c collisions, we compute the right and left p-values as p + (c) = P[X c X Poisson(λ)], p (c) = P[X c X Poisson(λ)], We reject H 0 if p + (c) is consistently very close to 0 (too many collisions) or p (c) is consistently very close to 1 (too few collisions)

115 Example: Birthday spacings 48 Partition [0, 1) t into k = d t cubic boxes and generate n points as before Let I 1 I 2 I n the box numbers where the points fall Compute the spacings S j = I j+1 I j, 1 j n 1 Let S (1),, S (n 1) be the sorted spacings Number of collisions between the spacings: Y = n 1 j=1 I[S (j+1) = S (j) ]

116 Example: Birthday spacings 48 Partition [0, 1) t into k = d t cubic boxes and generate n points as before Let I 1 I 2 I n the box numbers where the points fall Compute the spacings S j = I j+1 I j, 1 j n 1 Let S (1),, S (n 1) be the sorted spacings Number of collisions between the spacings: Y = n 1 j=1 I[S (j+1) = S (j) ] For large k, Y is approx Poisson with mean λ = n 3 /(4k)

117 Example: Birthday spacings 48 Partition [0, 1) t into k = d t cubic boxes and generate n points as before Let I 1 I 2 I n the box numbers where the points fall Compute the spacings S j = I j+1 I j, 1 j n 1 Let S (1),, S (n 1) be the sorted spacings Number of collisions between the spacings: Y = n 1 j=1 I[S (j+1) = S (j) ] For large k, Y is approx Poisson with mean λ = n 3 /(4k) If Y takes the value y, the right p-value is p + (y) = P[X y X Poisson(λ)]

118 Other examples of tests 49 Nearest pairs of points in [0, 1) t Sorting card decks (poker, etc) Rank of random binary matrix Linear complexity of binary sequence Measures of entropy Complexity measures based on data compression Etc

119 The TestU01 software 50 [L Ecuyer and Simard, ACM Trans on Math Software, 2007] Implements a large variety of statistical tests and RNGs (hardware or software), written in C, freely available Also contains predefined batteries of tests: SmallCrush: quick check, 15 seconds; Crush: 96 test statistics, 1 hour; BigCrush: 144 test statistics, 6 hours; Rabbit: for bit strings Many widely-used generators fail these batteries unequivocally Some test results ρ = period length; t-32 and t-64 gives the CPU time to generate 10 8 random numbers Number of failed tests (p-value < or > ) in each battery

120 Results of test batteries applied to some well-known RNGs Generator log 2 ρ t-32 t-64 SmallCrush Crush BigCrush LCG in Microsoft VisualBasic LCG(2 31, 65539, 0) (6) LCG(2 32, 69069, 1) (2) 106 (2) LCG(2 32, , 0) (4) LCG(2 46, 5 13, 0) (2) LCG(2 48, , 11), Unix (1) JavautilRandom (3) 21 (1) LCG(2 48, 5 19, 0) (2) LCG(2 48, , 0) (5) LCG(2 48, , 0) (5) LCG(2 59, 13 13, 0) (1) 17 (5) LCG(2 63, 5 19, 1) LCG(2 31 1, 16807, 0), Wide use (9) LCG(2 31 1, , 0) (7) LCG(2 31 1, , 0), (SAS) (4) LCG(2 31 1, , 0) (5) LCG(2 31 1, , 0) (4) LCG( ,, 0), in Maple (2) 34 (1) LCG(2 61 1, , 0) (4) 3 (1) 51

121 Generator log 2 ρ t-32 t-64 SmallCrush Crush BigCrush Wichmann-Hill, in Excel (3) 22 (8) CombLec Knuth(38) (1) 2 ran2, in Numerical Recipes CLCG Knuth(39) (1) 3 (2) MRGk DengLin (2 31 1, 2, 46338) (1) 11 (1) 19 (2) DengLin (2 31 1, 4, 22093) (1) 2 4 (2) DX DX Marsa-LFIB CombMRG MRG31k3p (1) MRG32k3a SSJ + others MRG63k3a LFib(2 31, 55, 24, +) (5) LFib(2 31, 55, 24, ) ran3, in Numerical Recipes (1) 11 (1) 17 (2) LFib(2 48, 607, 273, +) Unix-random (2) 101 (3) Unix-random (1) 57 (6) Unix-random (3) Unix-random (1) 8 11 (1) 52

122 Generator log 2 ρ t-32 t-64 SmallCrush Crush BigCrush Knuth-ran array Knuth-ranf array SWB(2 24, 10, 24) (2) SWB(2 24, 10, 24)[24, 48] (1) 16 (1) SWB(2 24, 10, 24)[24, 97] SWB(2 24, 10, 24)[24, 389] SWB(2 32 5, 22, 43) (1) 8 17 SWB(2 31, 8, 48) (2) 8 (2) 11 Mathematica-SWB (2) 15 (3) SWB(2 32, 222, 237) (2) GFSR(250, 103) (4) GFSR(521, 32) GFSR(607, 273) Ziff T (4) TT (4) 14 (3) MT19937, widely used WELL1024a WELL19937a (1) 2 LFSR LFSR Marsa-xor32 (13, 17, 5) (10) Marsa-xor64 (13, 7, 17) (1) 7 53

123 Generator log 2 ρ t-32 t-64 SmallCrush Crush BigCrush Matlab-rand (1) Matlab-LCG-Xor (normal) (1) SuperDuper-73, in S-Plus (1) 25 (3) SuperDuper R-MultiCarry (1) 40 (4) KISS KISS Brent-xor4096s ICG(2 31 1, 22211, ) (8) EICG(2 31 1, , 1) (6) SNWeyl (12) Coveyou (5) Coveyou LFib(2 64, 17, 5, ) LFib(2 64, 55, 24, ) LFib(2 64, 607, 273, ) LFib(2 64, 1279, 861, ) ISAAC AES (OFB) AES (CTR) (1) AES (KTR) SHA-1 (OFB) SHA-1 (CTR)

124 Conclusion 55 A flurry of computer applications require RNGs A poor generator can severely bias simulation results, or permit one to cheat in computer lotteries or games, or cause important security flaws Don t trust blindly the RNGs of commercial or other wisely-used software, especially if they hide the algorithm (proprietary software) Some software products have good RNGs; check what it is RNGs with multiple streams are available from my web page in Java, C, and C++