Stochastic Simulation

Transcription

1 Stochastic Simulation APPM 7400 Lesson 3: Testing Random Number Generators Part II: Uniformity September 5, 2018 Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

2 RNGs: Tests for Uniformity χ 2 test serial test Kolmogorov-Smirnov test Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

3 RNGs: Tests for Uniformity χ 2 Test Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

4 RNGs: Tests for Uniformity χ 2 Test Break up the unit interval into k bins. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

5 RNGs: Tests for Uniformity χ 2 Test Break up the unit interval into k bins. If data are uniform, expect n/k in each bin. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

6 RNGs: Tests for Uniformity χ 2 Test Break up the unit interval into k bins. If data are uniform, expect n/k in each bin. Do a χ 2 test to compare observed and expected values in each bin. (Make sure that you expect at least 5 in each bin.) Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

7 χ 2 Test In our sample with n = 100,000 and k = 20: Bin Observed Expected (partial table). Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

8 χ 2 Test In our sample with n = 100,000 and k = 20: W = k (O i E i ) E i i=1 Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

9 χ 2 Test In our sample with n = 100,000 and k = 20: W = k (O i E i ) E i i=1 Compare to the critical value χ (19) = WE PASSED! Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

10 RNGs: Tests for Uniformity Serial Test (For Uniformity and Independence) Idea: Bunch up the data into m-dimensional vectors. If the individual uniform values are independent, the vectors should be uniformly distributed in the m-dimensional unit cube. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

11 Serial Test (For Uniformity and Independence) Example: 2 dimensions Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

12 Serial Test (For Uniformity and Independence) k by k grid Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

13 Serial Test (For Uniformity and Independence) k by k grid let O ij be the number of observations in the ijth bin Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

14 Serial Test (For Uniformity and Independence) k by k grid let O ij be the number of observations in the ijth bin if uniform and members of pairs independent, expect (n/2)/k 2 in each bin Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

15 Serial Test (For Uniformity and Independence) k by k grid let O ij be the number of observations in the ijth bin if uniform and members of pairs independent, expect (n/2)/k 2 in each bin now a standard χ 2 test Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

16 Serial Test (For Uniformity and Independence) k by k grid let O ij be the number of observations in the ijth bin if uniform and members of pairs independent, expect (n/2)/k 2 in each bin now a standard χ 2 test (Really we are testing for a local independence.) Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

17 Serial Test (For Uniformity and Local Independence) For our sample, I used a 20 by 20 grid. Of the 50,000 pairs... We expect 125 in each cell... W = 400 i=1 (O i E i ) 2 E i Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

18 Serial Test (For Uniformity and Local Independence) For our sample, I used a 20 by 20 grid. Of the 50,000 pairs... We expect 125 in each cell... W = 400 i=1 (O i E i ) 2 E i χ (399) = WE PASSED! Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

19 Serial Test (For Uniformity and Local Independence) Incidentally, k W χ (k 1) result failed passed passed Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

20 RNGs: Tests for Uniformity χ 2 test serial test Kolmogorov-Smirnov test Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

21 RNGs: Tests for Uniformity Kolmogorov-Smirnov Test Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

22 RNGs: Tests for Uniformity Kolmogorov-Smirnov Test Let F(x) = P(X x) be the cdf for the distribution. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

23 RNGs: Tests for Uniformity Kolmogorov-Smirnov Test Let F(x) = P(X x) be the cdf for the distribution. In the uniform(0,1) case: F(x) = x,0 x 1. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

24 RNGs: Tests for Uniformity Kolmogorov-Smirnov Test Let F(x) = P(X x) be the cdf for the distribution. In the uniform(0,1) case: F(x) = x,0 x 1. Compare this to the empirical distribution function : ˆF n (x) = #X i in the sample x n Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

25 RNGs: Tests for Uniformity Empirical and Hypothesized uniform CDFs solid line is the empirical d.f. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

26 RNGs: Tests for Uniformity Empirical and Hypothesized uniform CDFs (dashed line is uniform cdf) solid line is the empirical d.f. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

27 RNGs: Tests for Uniformity Empirical and Hypothesized uniform CDFs (dashed line is uniform cdf) (We can use the KS test for any distribution.) solid line is the empirical d.f. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

28 K-S: Test for a Distribution If X 1,X 2,...,X n really come from the distribution with cdf F, the distance should be small. D = D n = max ˆF n (x) F(x) x Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

29 K-S: Test for a Distribution Computing the test statistic: Suppose we simulate 7 uniform(0,1) s and get: (obviously simplified) Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

30 K-S: Test for a Distribution Put them in order: Now the empirical cdf is: ˆF 7 (x) = 0 for x < 0.1 1/7 for 0.1 x < 0.2 3/7 for 0.2 x < 0.4 4/7 for 0.4 x < 0.5 5/7 for 0.5 x < 0.6 6/7 for 0.6 x < for x 0.9 Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

31 K-S: Test for a Distribution Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

32 K-S: Test for a Distribution D 7 = Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

33 K-S: Test for a Distribution Let X (1),X (2),...,X (n) be the ordered sample. Then D n can be computed as D n = max{d + n,d n } where D + n D n = max 1 i n { i n F(X (i)) } = max 1 i n { F(X(i) ) i 1 n } (assuming non-repeating values) Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

34 K-S: Test for a Distribution We reject that this sample came from the proposed distribution if the empirical cdf is too far away from the true cdf of the proposed distribution. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

35 K-S: Test for a Distribution We reject that this sample came from the proposed distribution if the empirical cdf is too far away from the true cdf of the proposed distribution. ie: We reject if D n is too large. Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

36 K-S: Test for a Distribution We reject that this sample came from the proposed distribution if the empirical cdf is too far away from the true cdf of the proposed distribution. ie: We reject if D n is too large. How large is large? Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

37 K-S: Test for a Distribution In the 1930 s, Kolmogorov and Smirnov showed that lim P( nd n t) = 1 2 n ( 1) i 1 e 2i2 t 2. i=1 So, for large sample sizes, you could assume that P( nd n t) 1 2 ( 1) i 1 e 2i2 t 2. and find the value of t that makes the right hand side 1 α for an α level test. i=1 Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

38 K-S: Test for a Distribution For small samples, people have worked out and tabulated critical values, but there is no nice closed form solution. J. Pomerantz (1973) J. Durbin (1968) Good approximations for n > 40: α c.v n n n n n Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

39 K-S: Test for a Distribution For our small sample of size 7, D 7 = From a table, the critical value for a 0.05 level test for n = 7 is Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

40 K-S: Test for a Distribution For our small sample of size 7, D 7 = From a table, the critical value for a 0.05 level test for n = 7 is WE PASSED! Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

41 K-S: Test for a Distribution For our large sample of size 100,000, D = The approximate critical value for a 0.05 level test for n = 100,000 is Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24

42 K-S: Test for a Distribution For our large sample of size 100,000, D = The approximate critical value for a 0.05 level test for n = 100,000 is WE PASSED! Lesson 3: Testing Random Number GeneratorsPart II: Uniformity Stochastic Simulation September 5, / 24