MONTE CARLO TESTING AND VERIFICATION OF NUMERICAL ALGORITHM IMPLEMENTATIONS. David D. Pokrajac Abdullah-Al-Zubaer Imran Predrag R.

Transcription

1 MONTE CARLO TESTING AND VERIFICATION OF NUMERICAL ALGORITHM IMPLEMENTATIONS David D. Pokrajac Abdullah-Al-Zubaer Imran Predrag R. Bakic

2 OUTLINES Objectives Validation Monte Carlo Approach Numerical Algorithms Validation Software Testing Software Verification Application in Partial Volume Approximation Results Discussion & Conclusions Acknowledgements 2

3 PARTIAL VOLUME APPROXIMATION 1 Voxel V n 2 V 1 Adipose tissue x c n 1 2 Skin x 1 x 2 Ligament-Adipose boundary Cooper s ligament 3

4 OBJECTIVES To develop a statistical test in order to assess the correctness of a numerical algorithm implementation To propose a Monte Carlo method of an approximation algorithm To illustrate the methodology on partial volumes 4

5 VALIDATION Software testing Verification Software testing determine whether the algorithm in quest is correctly implemented Software verification comprises techniques determine the adequacy of the developed algorithm Software testing by providing a limited set of test cases with known outputs Difficulty to examine a variety of potential inputs to software In software verification, it is often of interest to determine the accuracy of an approximation algorithm Validation can be performed empirically 5

6 MONTE CARLO AS THE REPLACEMENT The application of Monte Carlo approach for validation of a class of numerical software. The method is developed for a class of multiple integral computation problems and demonstrated on a related problem of partial volume computation A statistics developed and what shows: H0: algorithm implementation is correct The statistics has asymptotical standard normal distribution 6

7 BASICS Goal: calculate integrals I i = D f i x 1, x 2,, x k 1 The Monte Carlo approach as follows: dx 1 dx 2 dx k 1, i = 1,, T a) Uniformly sample N MC independent points x j = x j,1, x j,2,, x j,k [0,1] k, j = 1, N MC b) Determine: N i = x j x j,k f i x j,1, x j,2,, x j,k 1, j = 1,, N MC c) Compute an approximation I MC,i = N i N MC For a randomly chosen x j [0,1] k, the probability that x j,k f i x j,1, x j,2,, x j,k 1 is equal to I i. A random variable N i follows a Binomial distribution with expectation N MC I i and variance N MC I i 1 I i 7

8 SOFTWARE TESTING Assume: I a,i = g i x 1, x 2,, x D k 1 1,, T is implemented. dx 1 dx 2 dx k 1, i = g i x 1, x 2,, x k 1 = F f i x 1, x 2,, x k 1 are suitable chosen functions Consider a set of functions f i (i,,t): the algorithm provides an exact solution Under H 0 that the algorithm is correctly implemented, for large enough T, random variable Z has approximately Gaussian distribution Variance: σ X 2 + σ Y 2 2σ XY (X and Y: Two random variables) Z = Z s Z s Z = s2 ε 2 T + 3s2 I a 1 I a TN MC 2 s ε 2 = 1 T 1 T i=1 ε i 2 ε 2 2 ε 2 = 1 T T 2 ε i i=1 8

9 SOFTWARE VERIFICATION A random variable Z defined as: Z = X Y = 1 T T i=1 ε i 2 1 N MC 1 1 T T i=1 I MC,i 1 I MC,i Z, is asymptotically Gaussian, with mean equal to E ε A 2 and variance which square root is bounded 1T σ ε2 σ I MC 1 I MC N MC 1 σ Z 1 T σ ε2 + σ I MC 1 I MC N MC 1 The boundaries for standard deviation of the estimate can be obtained when a squared root σ of variance is estimated using a sample standard deviation s as: s Z,min s Z s Z,max, s Z,min = 1 T s ε2 s I MC 1 I MC N MC 1 s Z,max = 1 T s ε2 + s I MC 1 I MC N MC 1 9

10 PRACTICAL APPLICATION Case p 1 (6 bits) p 2 (6 bits) 1.Skin and air 0 p Air 0 2.Cooper s ligament; fat p Fat Cooper s ligament; 0 p Dens 1 dense e 4. Skin; dense tissue 0 p Skin 3 5. Skin; fat tissue 0 p Skin 2 6. Skin; Cooper s ligament 1-p Skin p Skin 2 Label (4 bits) Skin, Cooper s ligament and dense tissue 8. Skin, Cooper s ligament and fat tissue p Cooper p Skin 3 p Cooper p Skin 2 Skin Two-material ligament Three-material ligament 10

11 RESULTS COMPUTED P-VALUES FOR STATISTICAL TEST OF ALGORITHM FOR CORRECT IMPLEMENTATION OF PARTIAL VOLUME ALGORITHM OBTAINED FOR DIFFERENT COMBINATIONS OF T AND N MC T 1e6 1e5 1e4 1e5 1e4 5e6 1e6 1e5 N MC 1e5 1e5 1e5 1e4 1e p-value

12 RESULTS (CONT D) ESTIMATED APPROXIMATION ERROR (MEAN AND STANDARD DEVIATION BOUNDARIES): N MC = 63. MSE A FROM CORRESPONDING TO Z IS INCLUDED FOR COMPARISON Voxels containing T Z s Z,min s Z,max MSE A Skin 1,597, e e e e-05 Ligaments and Compartmental Tissue 6,435, e e e e-04 12

13 INCORRECT IMPLEMENTATION (3-MATERIAL PV) T=10,000 N=10,000 z = 2.847e 04 σ z 2 = e 05 p value = e 31 H 0 can be rejected! 13 Histogram of ε for N=10,000, T=10,000

14 CORRECT IMPLEMENTATION (3-MATERIAL PV) T=10,000 N=10,000 z = e 09 σ 2 z = e 07 p value = H 0 cannot be rejected! 14 Histogram of ε for N=10,000, T=10,000

15 DISCUSSION AND CONCLUSIONS The accuracy of the Monte Carlo approximation is of secondary importance Manual evaluation of test cases needed to test a complex algorithm is not feasible This distinguishes the proposed approach from other approaches that may provide only the point estimate of the approximation error The proposed approach can be easily extended whenever the estimation using an analog of I MC,i = N i is possible and N MC where N i follows binomial distribution. 15

16 ACKNOWLEDGEMENTS David D. Pokrajac Associate Dean of Research & Analytics Delaware State University Predrag R. Bakic Associate Professor of Radiology University of Pennsylvania Alton Thompson Provost and Executive Vice President for Academic Affairs Delaware State University Department of Computer & Information Sciences Delaware State university 16

17 Thanks 17