Full-core Reactor Analysis using Monte Carlo: Challenges and Progress

Transcription

1 Full-core Reactor Analysis using Monte Carlo: Challenges and Progress Spring 2018 Colloquia Paul K. Romano Argonne National Laboratory University of California, Berkeley Department of Nuclear Engineering March 12, 2018 exascaleproject.org

2 Monte Carlo for reactor analysis Monte Carlo is the most accurate particle transport method: Exact representation of geometry Exact representation of physical phenomena Why isn t Monte Carlo the method of choice for full-core reactor analysis? 2

3 Challenges Primary challenges outlined in Martin s 2012 paper [1]: Prohibitive computational time for acceptable statistics Excessive demand on computer memory Slow convergence of the fission source Apparent versus true variance Accounting for multiphysics feedback Adapting to future computing architectures 3

4 Challenges 1 Memory use 2 Performance Time for acceptable statistics Fission source convergence Future computer architectures 3 Accuracy of solution Estimating true variance Multiphysics 4

5 Existing use of MC Research and test reactors Small reactors Cross section generation Validation and one-off calculations 5

6 Target Problem LWR 6 Calculate reaction rates (7) in each nuclide (300) in each fuel pellet subdivided into 10 rings (200 million) Converged solution requires 1012 particles

7 Memory Requirements MC simulations require memory for: Nuclear data Geometry description Material compositions Tallies Particles 7

8 Memory: Nuclear Data Continuous-energy cross sections for all nuclides at a single temperature require 1 GB Need data at temperatures from 300 K to 2500 K Without sacrificing accuracy, would need a temperature spacing of 10 K to 25 K 100 GB required 8

9 Memory: Materials 200 million unique materials 300 nuclides per material 4 bytes for nuclide identifier, 8 bytes for atom density 750 GB required 9

10 Memory: Tallies 200 million unique materials 300 nuclides per material 7 transmutation reaction rates per nuclide 24 bytes per reaction rate 10 TB required 10

11 Approaches: Tallies/Materials 1 Domain decomposition 2 Data decomposition 3 Wait it out 11

12 Domain Decomposition Each fission generation is broken into stages where particles are communicated between domains 12

13 Domain Decomposition Can, in theory, handle both material and tally data Negative impact on performance due to: Cost of communicating particles between nodes Load imbalances due to non-uniform particle distribution Studies have shown that in worst-case, probably 2 slower [2] Using buffering and non-blocking communication can significantly reduce cost [3, 4] Non-trivial implementation Used in Mercury, Shift, RMC 13

14 Data Decomposition Arbitrarily divide tally bins over multiple nodes At tallying event, send scores to separate process responsible for accumulating tallies Simplest implementation: compute/server processes [5] Improvements: using a PGAS library [6], using multithreading [7] As of yet, no real proposal for dealing with material compositions Relatively simple to implement compared to DD 14

15 Wait it out? From Kord Smith s 2013 paper [8]: Regardless of the high-fidelity methods that might be employed, it is very likely that implementation will require domain decomposition or other memory management approaches since available memory per computational node is far less than 1 TB. 15

16 Wait it out? 24 DIMMs 128 GB/DIMM = 3 TB 16

17 Putting it together... Domain Decomp. Data Decomp. Just Wait Tallies Materials Overhead 0 100% 0 25% 0 Complexity Hard Modest None 17

18 Appraches: Nuclear Data 1 Polynomial expansion in temperature [9] 2 Target motion sampling (TMS) [10] 3 Windowed multipole method [11] 18

19 Temperature Expansion For each cross section and energy point, fit the following regression in T: N a N i σ(t) T i/2 + b i T i/2 + c i=1 Requires pregenerated tabulated cross sections at every 1 K for regression fitting Runtime cost shown to be small i=1 19

20 Target Motion Sampling Assume distribution of target motion distribution of neutron path lengths Use rejection method to sample path length based on distribution Only requires 0 K cross sections Note: still compatible with surface tracking Can track on regions with a continuously-varying T Rejections can increase variance of tallies; using elevated basis temperature can help somewhat [12, 13] 20

21 Windowed Multipole Use a pole representation to write cross section at 0 K σ = 1 ir j Re E E p j Can then analytically Doppler broaden: σ(t) = 1 2E r j Re ir j πw (z) C ξ π j j pj u, ξ 2 ξ 21

22 Windowed Multipole Use windows to reduce number of pole evaluations Runtime cost same as evaluating tabulated cross sections Requires complicated preprocessing of cross sections 22

23 Putting it together... Expansion TMS Multipole Memory 17 GB 1 GB MBs Overhead 10 25%? 0 Data preparation Cumbersome None Complex T distributions Used in MCNP Serpent OpenMC None of the methods handle S(α,β) or URR! 23

24 Improving MC performance Engineering Approaches Improvements in basic algorithms Improvements in coupling algorithms Variance reduction Computer science approaches Increasing CPU clock frequency Increasing number of CPU cores Architecture-specific optimization Improving convergence of fission source 24

25 Source convergence Two separate problems: 1 Convergence source in fewer iterations 2 Knowing when you re actually converged State of affairs: Too many papers to list Most production codes don t have anything built-in Not a significant part of overall time-to-solution 25

26 Basic Algorithms Lots of work done on improving performance of cross section lookups: Unionized energy grids [14] Logarithmic mapping techniques [15] Fractional cascading [16] Not as much focus on tallies Fundamental limit is memory latency [17, 18] 26

27 Coupling Algorithms Once heat transfer/fluid dynamics is considered, you have non-linear system: Reduce time by: ψ = F 1 (T) T = F 2 (ψ) Not fully converging each iteration [19] Iterate on low-order operator [20] Using Newton method [21] 27

28 Architectural trends Most things are not going in our favor: CPU frequencies leveled off long ago Improvements in CPUs are not matched by improvements in memory subsystem Increasing levels of data-level parallelism are largely out of reach for Monte Carlo However, Hardware vendors keep pushing single-thread performance Core counts have continued to increase (Moore s law) Memory capacity per node continues to increase 28

29 Architecture-specific optimization Lots of interest in vectorization [22] Event-based algorithm [23] FPGAs? 29

30 Summary Lots of progress has been made towards full-core reactor analysis with MC: Memory issues have been mostly solved Better understanding of issues with multi-physics coupling Many challenges lay ahead: Application use still limited by performance Robust algorithms for multi-physics coupling Best utilizing new CPU architectures 30

31 References I [1] William R. Martin. Challenges and prospects for whole-core Monte Carlo analysis. Nucl. Eng. Technol., 44(2): , [2] Nicholas Horelik, Andrew Siegel, Benoit Forget, and Kord Smith. Monte Carlo domain decomposition for robust nuclear reactor analysis. Parallel Comput., 40: , [3] Tara M. Pandya, Seth R. Johnson, Thomas M. Evans, Gregory G. Davidson, Steven P. Hamilton, and Andrew T. Godfrey. Implementation, capabilities, and benchmarking of Shift, a massively parallel Monte Carlo radiation transport code. J. Comput. Phys., 308: , [4] Jingang Liang, Kan Wang, Yishi Qiu, Xiaoming Chai, and Shenglong Qiang. Domain decomposition strategy for pin-wise full-core Monte Carlo depletion calculation with the Reactor Monte Carlo code. Nucl. Eng. Technol., 48: , [5] Paul K. Romano, Andrew R. Siegel, Benoit Forget, and Kord Smith. Data decomposition of Monte Carlo particle transport simulations via tally servers. J. Comput. Phys., 252:20 36,

32 References II [6] Nan Dun, Hajime Fujita, John R. Tramm, Andrew A Chien, and Andrew R. Siegel. Data decomposition in Monte Carlo neutron transport simulations using global view arrays. Int. J. High Perform. Comput. Appl., 29(3): , [7] Hong-Fei Liu, Peng Ge, Sheng-Peng Yu, Jing Song, and Xiao-Lei Zheng. Data decomposition method for full-core Monte Carlo transport-burnup calculation. Nucl. Sci. Tech., 29:20:1 8, February [8] Kord Smith and Benoit Forget. Challenges in the development of high-fidelity LWR core neutronics tools. In International Conference on Mathematics and Computational Methods Applied to Nuclear Science & Engineering, Sun Valley, Idaho, May [9] Gokhan Yesilyurt, William R. Martin, and Forrest B. Brown. On-the-fly Doppler broadening for Monte Carlo codes. Nucl. Sci. Eng., 171(3): , July [10] Tuomas Viitanen and Jaakko Leppänen. Explicit treatment of thermal motion in continuous-energy Monte Carlo tracking routines. Nucl. Sci. Eng., 171: ,

33 References III [11] Colin Josey, Pablo Ducru, Benoit Forget, and Kord Smith. Windowed multipole for cross section doppler broadening. J. Comput. Phys., 307: , [12] Tuomas Viitanen and Jaakko Leppänen. Target motion sampling temperature treatment technique with elevated basis cross-section temperatures. Nucl. Sci. Eng., 177(1):77 89, May [13] Tuomas Viitanen and Jaakko Leppänen. Effects of the Target Motion Sampling temperature treatment method on the statistics and performance. Ann. Nucl. Energy, 82: , [14] Jaakko Leppänen. Two practical methods for unionized energy grid construction in continuous-energy Monte Carlo neutron transport calculation. Ann. Nucl. Energy, 36: , [15] Forrest B. Brown. New hash-based energy lookup algorithm for Monte Carlo codes. Technical Report LA-UR , Los Alamos National Laboratory, Los Alamos, New Mexico,

34 References IV [16] Amanda L. Lund, Andrew R. Siegel, Benoit Forget, Colin Josey, and Paul K. Romano. Using fractional cascading to accelerate cross section lookups in Monte Carlo neutron transport calculations. In Joint Int. Conf. Mathematics and Computation, Supercomputing in Nuclear Applications, and the Monte Carlo method, Knoxville, Tennessee, Apr [17] John R. Tramm and Andrew R. Siegel. Memory bottlenecks and memory contention in multi-core monte carlo transport codes. Ann. Nucl. Energy, 82: , [18] Paul K. Romano, Andrew R. Siegel, and Ronald O. Rahaman. Influence of the memory subsystem on Monte Carlo code performance. In Joint Int. Conf. Mathematics and Computation, Supercomputing in Nuclear Applications, and the Monte Carlo method, Knoxville, Tennessee, Apr [19] Daniel F. Gill, David P. Griesheimer, and David L. Aumiller. Numerical methods in coupled Monte Carlo and thermal-hydraulic calculations. Nucl. Sci. Eng., 185: , January [20] Sterling Harper, Kord Smith, and Benoit Forget. Faster Monte Carlo multiphysics using temperature derivatives. In PHYSOR, Cancun, Mexico, Apr

35 References V [21] Manuele Aufiero and Massimiliano Fratoni. A new approach to the stabilization and convergence acceleration in coupled Monte Carlo CFD calculations: The Newton method via Monte Carlo perturbation theory. Nucl. Eng. Technol., 49: , [22] Emeric Brun, Stephane Chauveau, and Fausto Malvagi. PATMOS: A prototype Monte Carlo transport code to test high performance architectures. In Int. Conf. Mathematics & Computational Methods Applied to Nuclear Science & Engineering, Jeju, Korea, Apr [23] Steven P. Hamilton, Stuart R. Slattery, and Thomas M. Evans. Multigroup Monte Carlo on GPUs: Comparison of history- and event-based algorithms. Ann. Nucl. Energy, 113: ,

36 Acknowledgments This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of two U.S. Department of Energy organizations (Office of Science and the National Nuclear Security Administration) responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering, and early testbed platforms, in support of the nation s exascale computing imperative. 36