HPC Use for Earthquake Research

Transcription

1 HPC Use for Earthquake Research Christine Goulet, Ph.D. Executive Science Director for Special Projects Southern California Earthquake Center () Thanx to T. Jordan, P. Maechling, Y. Cui, S. Callaghan, K. Milner, K. Olsen and others HPC User Forum April 17, 2018

2 Southern California Earthquake Center () Mission: solve problems in earthquake system science Understand and model the physics of earthquake processes, from tectonic stress to fault rupture to ground motions Use SoCal as our natural laboratory Large collaborative center Multi-disciplinary Planning Committee develops yearly science plan Support fundamental AND applied science Over 1,000 participants 18 core institutions, 44 domestic participating institutions, 13 international NSF & USGS funded center Additional support from PG&E, W.M. Keck Foundation, NSF SSI programs HPC User Forum April 17,

3 Institutions HPC User Forum April 17,

4 Risk = 3D (Deaths, Dollars, Downtime) = Hazard Exposure Vulnerability Resilience Faulting, shaking, tsunami, landsliding, liquefaction Extent & density of built environment Structural & nonstructural fragility Response capabilities, insurance, cat bonds Land-Use Policies & Planning Earthquake Engineering & Building Codes Preparedness & Response Earthquake system science characterizes seismic hazards key information for reducing risk and increasing resilience HPC User Forum April 17,

5 Conclusion: a paradigm shift for better resilience Risk reduction can be achieved by Making risk-informed decisions (design engineers, society) Prioritizing risk-informed research topics: improve hazard assessment where it matters most (researchers) Hazard assessment will improve by Using validated physics-based simulation models to learn about earthquake processes Reducing epistemic uncertainty Collect more data Refine and validate models Quantify uncertainty and its range HPC User Forum April 17,

6 Immediate motivation Limited recorded datasets... Most complete record databases by PEER (UC Berkeley) NGA-West2: CA and similar tectonic regions NGA-East: CEUS and similar tectonic regions HPC User Forum April 17,

7 Essential Ingredients for Ground Motion Computations 1. Earthquake rupture forecasts (ERFs) detailed representation of fault geometry rupture models that capture the complexities of dynamic fault failure Long-term rate of earthquakes 2. 3D models of geologic structure large-scale crustal heterogeneity sedimentary basin structure shallow site properties 3. Calculation of wave propagation and attenuation efficient anelastic wave propagation codes nonlinear models of near-surface response HPC User Forum April 17,

8 ShakeOut Scenario M7.8 earthquake simulation on Southern San Andreas Fault (deterministic band f = 0-1 Hz; stochastic band f = 1-10 Hz) HPC HPC User User Forum, April April 17, 17,

9 ShakeOut Scenario M7.8 earthquake simulation on Southern San Andreas Fault (deterministic band f = 0-1 Hz; stochastic band f = 1-10 Hz) HPC User Forum April 17,

10 TeraShake Simulations of M7.7 Earthquake on the San Andreas Fault SE to NW rupture NW to SE rupture Santa Barbara Los Angeles Santa Barbara Los Angeles San Diego San Diego Simulations indicated strong 3D focusing of ground motions from strike-slip faults in Southern California Quantified the importance of source directivity and basin excitation effects in earthquake forecasting and PSHA HPC User Forum April 17,

11 The ShakeOut Scenario M7.8 Earthquake on Southern San Andreas Fault Scenario Results M7.8 mainshock Broadband ground motion simulation (0-10 Hz) Large aftershocks M7.2, M7.0, M6.0, M5.7 10, ,000 landslides 1,600 fire ignitions $213 billion in direct economic losses 300,000 buildings significantly damaged Widespread infrastructure damage 270,000 displaced persons 50,000 injuries 1,800 deaths Long recovery time shaking intensity Great Southern California ShakeOut Exercise Impacts Largest emergency response exercise in US history Demonstrated that existing disaster plans are inadequate for an event of this scale Motivated reformulation of system preparedness and emergency response Scientific basis for the LA Seismic Safety Task Force report, Resilience by Design HPC User Forum April 17,

12 ground-motion simulation platforms Seismic band mantle waves crustal waves basin waves strongly scattered waves 100 s 10 s 1 s 0.1 s 0.01s period.01 Hz.1 Hz 1 Hz Earthquake engineering band 10 Hz 100 Hz frequency tanks, tall buildings short buildings, houses stiff structures, NPPs, equipment simulation computations BroadBand Platform 1D Deterministic Stochastic Development and production software (runs on clusters and PCs) CyberShake 3D Deterministic Complete physics-based PSHA (runs on HPC) BroadBand CyberShake 3D Deterministic CyberShake Stochastic BroadBand Platform High-F 3D Deterministic Physics-based development for new physics (runs on HPC and clusters) HPC User Forum April 17,

13 HPC User Forum April 17,

14 Broadband Platform (BBP) BBP is an open-source distribution Broadband Hz (deterministic up to ~ 1Hz) Simple source and path (1D) 7 alternative simulation codes Fully validated for spectral response 1.5 year validation project Multiple rounds of validation/improvements Independent review panel Continuation of work in 2018 Used for large ground motion characterization projects SRL Special Focus on BBP HPC User Forum April 17, 2018 Validation 9 papers - Jan

15 CyberShake: PSHA from Physics-Based Simulations + + = Earthquake Rupture Forecast (e.g. UCERF) Kinematic source models 3D velocity model (from F3DT) Seismograms and intensity measures Hazard Curves and fault disaggregation M, R, ε disaggregation Hazard maps HPC User Forum April 17,

16 CyberShake Study 17.3, Central CA Calculations for 2 velocity models for each of 438 sites, 1 Hz simulations, 40,000 + earthquakes Used OLCF Titan and NCSA Blue Waters Averaged 1295 nodes (CPU + GPU) for 31 days, maximum of ,000 node-hours consumed (21.6M core-hours) Pegasus, HTCondor, Globus used for workflows Workflow tools scheduled 15,581 jobs to both systems Transferred 308 TB of intermediate data between the two systems Hazard map, 3D model Generated 285 million two-component seismograms 43 billion intensity measures Workflow tools managed 777 TB of data 10.7 TB of output data automatically staged back for archival storage HPC User Forum April 17,

17 CyberShake Study 17.3 Results: Velocity Model Comparison 1D model 3D model HPC User Forum April 17,

18 CyberShake Study 17.3 Results: CCA and LA CyberShake First simulation-based PSHA model Combined CyberShake map Used as database for machine-learning of California Earthquake Early Warning system Leading model in USGS Urban Seismic Hazard Mapping Project Source of new design maps under consideration by building code update HPC User Forum April 17,

19 High-F Project Pushing deterministic ground-motion simulations to higher frequencies (~10Hz) while Improving improving computational efficiency Adding more realistic physics: fault roughness near-fault plasticity frequency-dependent attenuation topography small-scale nearsurface heterogeneity near-surface nonlinearity Image: Roten et al Image: K. Olsen, Y. Cui, A. Chourasia HPC User Forum April 17,

20 VERY GOOD Very rough, fresh unweathered surfaces VERY GOOD GOOD Very rough, fresh unweathered s Rough, slightly weathered, iron stained surface GOOD Rough, slightly weathered surfaces VERY GOOD Very rough, fresh unweathered surfaces SURFACE CONDITIONS OF DISCONTINUITIES (Predominantly bedding planes) VER Very DECREASING INTERLOCKING GOOD OF ROCK PIECES FAIR Rough, slightly weathered, iron s Smooth, moderately weathered and altered sur POOR GOO Roug FAIR Smooth, moderately weathered a optimistic sequently, ting a linast squares n the1994 cept of the which the the Mohr of a variower term to consolnto a come criterion xamples to equations, Mass Ratquate as a on to geocularly for ntheintrondex (GSI) and Hoek, x wassubasses in a ), Marinos d Marinos mating the ferent georelated to the condi- Tables4.3 asses and hestrength properties s the freetateunder om isconintact rock es separatith clean, rounded nd altered h stronger s rounded nd altered GSI FOR HETEROGENEOUS ROCK MASSES SUCH AS FLYSCH Including plasticity in near-fault material (Marinos P. and present Hoek E., 2000) in an unfavorable orientation be too precise. From a description with respect Quoting of the lithology, to the a range discontinuities structure STRUCTURE excavation from and surface conditions face, 33 these (particularly Rock The strength description properties of the andhoek Brown their measurement strength 95 of the bedding planes), choose a box in the chart. Locate the position in the box criterion in this section includes all the data in that corresponds to the condition of the discontin uities and estimate the average thiswork The description up to2002. of the Hoek Brown strength criterion in this section includes all the data in value of GSI The from the shear contours. strength Do not 3D Drucker-Prager plasticity with parameters INTACT attempt of surfaces to BLOCKY well OR be too MASSIVE intact precise. Quoting rocksinterlocked a range undisturbed failures. Note that the Hoek-Brown from Hoek-Brown rock thiswork Generalized up to2002. Hoek Brown strength from apply 33 to to 37 that is structurally more are realistic prone than controlled to giving deterioration GSI = 35. rock specimens as or a rock massive result mass in consisting situ strength criterion include in AWP-ODC (GPU and CPU versions on criterion criterion Where does weak not apply to structurally controlled failures. Where unfavourably Generalized Hoek Brown strength oriented continuous of changes planar weak planar in discontinuities moisture planes are of content present, cubical are 90 rock with few widely these will blocks spaced will be dominate formed by three The generalized Hoek Brown strength criterion criterion the present behaviour of the rock mass. The strength of some rock masses reduced by is expressed in terms of the major and reduced in an if unfavorable water present. orientation intersecting When work- discontinuity sets BW) minor The generalized principal stresses, Hoek Brown and istrength modified criterion from the presence of groundwater and this discontinuities with respect to the excavation can be face, allowed these for by a slight shift to the equation is expressed (4.13) asfollows(figure4.22) in terms of the major and right in the columns Hoek-Brown Strength Criterion ing for with for fair, Fractured rocks poor and in very Rock the poor fair conditions. Masses to very Water pressure poor does 80 minor principal stresses, and is modified from not will change dominate the value of GSI and it is dealt with by using effective stress analysis. a equation (4.13) asfollows(figure4.22) σ σ 1 = σ 3 + σ ci m 3 categories, the rock a shift mass to the behavior. right may be b + s (4.14) COMPOSITION σ ci The shear AND STRUCTURE a made strength for wet of conditions. surfaces BLOCKY well VERY in Water rocks BLOCKY interlocked, pressure undisturbed σ σ where 1 = σ m 3 + σ b is a ci m 3 mi reduced b + s (4.14) that : tabulated are by rock type, e.g.: σ ci value of the material slate, shale, A. prone Thick siltstone: bedded, to deterioration is dealt mi with very 7 by blocky effective sandstone as partially a stress rock result mass analysis. disturbed consisting mass with 70 constant m i for intact rock and isgiven by of sandstones, changes The effect breccias, in of conglomerates: moisture pelitic coatings of content cubical the bedding mi 17 multi-faceted will blocks be formed angular by three blocks 70 where m b is a reduced value of the material planes is minimized by the confinement of constant GSI 100 diorite, gabbro, granite: mi 25 m b = m i i exp for intact rock and isgiven by reduced (4.15) the if rock water mass. In is shallow present. tunnels intersecting When or slopes formed working with these rocks bedding in planes the may fair cause to structurally very poor discontinuity by 4 or more sets joint A 28 14D STRUCTURE DECREASING sets SURFACE 60 GSI Table4.5 m b = m givesvaluesof i exp m GSI D i for awidevariety (4.15)! ci : tables with controlled range of instability. values by rock type of categories, empirical equations a shift (e.g., Chang, to the 2006) right BLOCKY/DISTURBED/SEAMY may be rock s= types, exp and sand aareconstantsfor therock massgiven by 9 3D INTACT OR MASSIVE intact Table4.5 a= 1 givesvaluesof m i for awidevariety of made descriptive for B. Sandstone with rock specimens stone partially and or massive or silty shale in situ siltstone grades wet (Brown, conditions. 1981), C. field Sand- VERY Water estimates BLOCKY interlocked, pressure folded D. Siltstone with angular blocks E. Weak rock types, and GSI 6 (e GSI/15 sand aareconstantsfor 100 e 20/3 ) therock massgiven s= exp by is dealt (4.16) thin with interlayers of similar multi-faceted stone layers shale with by effective siltstone stress in analysis. formed disturbed with by mass sand- many with intersecting 50 GSI: describes degree of fracturing or clayey B 9 3D rock and with weathering few widely spaced D isa factor that depends upon the discontinuity angular blocks sets. Persistence a= s= 1 exp GSI 100 value (0-100) assigned based charts 6 (e GSI/15 9 3D e 20/3 ) (4.17) (4.16) siltstone discontinuities amounts formed by sandstone 3 Parameters: approximates STRUCTURE 100 at large depth (> 1 km) of 4 bedding or more planes joint sets DECREASING or schistosity 40 layers SURFACE QUALIT 80 D isa= a factor 1 that depends upon the degree of (e GSI/15 e 20/ 3 ) (4.17) Hoek-Brown parameter mi disturbancetowhich therock masshasbeen subjected Unconfined by blast damage compressive C,D,E and G may be more or D is a factor that depends and stress upon relaxation. the degree It ofless folded than Ilustrated but variesfrom0 disturbancetowhich strength, for! undisturbed ci of therock intact insitu masshasbeen rock rock masses subjected 1 for Geological byvery blast disturbed damage Strength rock and stress masses; Index, relaxation. guidelines ItTectonic deformation, faulting and this does not change the strength. to for variesfrom0 theselection GSI for of undisturbed appropriateinsitu values rock for masses D are discussed to 1 for very insection disturbed loss of continuity rock moves with thesefew widely spaced Degree of disturbance rock masses; D = 0 guidelines for Theuniaxial theselection compressive of appropriate strength values of for therock categories to F and D are discontinuities H. mass discussed is obtained insection by setting σ 3 = 0 in equation massive blocky disturbed disintegrated deformed, Theuniaxial (4.14), giving compressive strength of therock folded mass is obtained σ c = σ ci s a by setting σ 3 = 0 in equation (4.14), giving G. Undisturbed silty (4.18) or clayey shale with or without a few very and, thetensilestrength is σ c = σ ci s a (4.18) thin sandstone layers Roten et al. (2014, 2015, 2016) σ t = sσ ci and, thetensilestrength (4.19) m is b σ t = sσ ci m b (4.19) apply to made structurally for wet conditions. controlled rock specimens Water failures. pressure conditions of the discontinuities, estimate or massive in situ Where weak is dealt planar with structural by effective rock with planes stress few are widely analysis. the average value of GSI. Do not try to spaced Table4.4 GSI valuescharacterizing schistosemetamorphic rock masseson thebasisof foliation and disconti to 37 is more realistic than stating that will dominate the rock mass behavior. GSI = 35. Note that the table does not INTACT BLOCKY well OR MASSIVE intact BLOCKY/DISTURBED/SEAMY DISINTEGRATED poorly interlocked undisturbed folded interlocked, with F. Tectonically deformed, intensively rock specimens or rock massive in angular consisting folded/faulted, situ heavily blocks broken sheared 90 clayey rock shale mass siltstone with broken deformed of cubical formed blocks by with formed many mixure intersecting sandstone by three of angular and layers forming an almost intersecting discontinuity discontinuity rounded sets. chaotic rock sets Persistence structure pieces of bedding planes or schistosity80 : Means deformation HPC User after Forum tectonic April disturbance 17, 2018 H. Tectonically deformed silty or clayey shale forming a chaotic structure with pockets of clay. Thin layers of sandstone are 70 transformed into small rock pieces. BLOCKY well VERY BLOCKY interlocked, DISINTEGRATED poorly LAMINATED/SHEARED lack undisturbed partially rock mass locked, disturbed consisting heavily of blockiness mass broken with due rock to mass close spacing inter- of cubical multi-faceted blocks with formed mixure angular of by weak of three blocks angular schistosity and or shear planes intersecting formed discontinuity rounded by 4 or more rock setsjoint pieces sets 60 OF ROCK PIECES SURFACE CONDITIONS ERLOCKING OF ROCK PIECES SURFACE CONDITIONS DECREASING INTERLOCKING OF ROCK PIECES SUR DECREASING 80 SURF Roten et al N/A N/

21 ShakeOut with Drucker-Prager Plasticity (Roten et al., 2016) HPC User Forum April 17,

22 AWP-ODC NVIDIA GPUs A first 4-Hz nonlinear M7.7 earthquake simulation on the southern San Andreas Fault conducted using 4,200 Blue Waters GPUs 100% of parallel efficiency achieved for both linear/ nonlinear versions of AWP-ODC up to 8,192 GPUs Accelerated time-to-solution from original nonlinear 0.68sec to 0.29sec per iteration Blue Waters PAID project provided additional support. Intel Xeon Phi Mega Lattice Updates per Second (MLUPS): Xeon Phi KNL 7290 achieves 2x speedup over NVIDIA K20X, 97% of NVIDIA Tesla P100 performance Performance on 9,000 nodes of Cori-II equivalent to performance of over 20,000 K20X GPUs at 100% scaling Open Source: 6.5x Speedup of CyberShake SGT version on Cray XK7 compared to XE6 at nodeto-node level 22

23 AWP-ODC on Sunway TaihuLight Tsinghua University/Wuxi SC Center ported open source AWP-ODC using Sunway OpenACC and fully optimized the code on TaihuLight Sustained 15-Pflop/s or 12.5% of the peak achieved, a ACM Gordon Bell Finalist in Mw7.2 Tangshan earthquake scenario and high-fidelity simulation using 10 million cores A spatial resolution of 25m 320km x 320km x 60km Frequency of up to 10Hz Included non-linear near-fault physics (Haohuan Fu, Conghui He, Bingwei Chen, Zekun Yin, Zhenguo Zhang, Wenqiang Zhang, Tingjian Zhang, Wei Xue, Weiguo Liu, Wanwang Yin, Guangwen Yang, Xiaofei Chen (2017) 15-Pflops Nonlinear Earthquake Simulation on Sunway TaihuLight: Enabling Depiction of Realistic 10 Hz Scenarios, SC 17, Nov 13-16, Denver, 2017) 23

24 EDGE Achieved 10.4 DP-PFLOPs on Cori Phase 2 Extreme-scale Discontinuous Galerkin Environment: Seismic wave propagation through DG-FEM Focus: Problem settings with high geometric complexity Fused simulations to explore intersimulation parallelism Tuned for Haswell, Knights Landing, Skylake, and Knights Mill through LIBXSMM library Cori Phase 2 weak scaling: Sustained 10.4 DP-PFLOPs (38% of peak) Theta strong scaling: Sustained 3.4 DP- PFLOPs (40% of peak) for a 100x increase in node count Open Source: Collaboration of SDSC, Intel, and through Intel PCC program (Breuer et al., ISC 17) Weak scaling study on Cori-II. Shown are hardware and non-zero peak efficiencies in flat mode. O denotes the order and C the number of fused simulations. (Breuer et al., ISC 17) Strong scaling study on Theta. Shown are hardware and non-zero peak efficiencies in flat mode. O denotes the order and C the number of fused simulations. HPC User Forum April 17,

25 Takeaways Ground motions simulations on HPC are a great learning tool for scientific research Simulations are already impacting design, improved hazard will help us to reduce seismic risk Continuous development needed Collaboration is essential What s next? Replace ERF with earthquake simulator Go to up to higher frequencies, extend the range of applications Test and implement additional physics Formally evaluate uncertainties Use simple models as benchmarks in the evaluation of complex models Validate, validate, validate... HPC User Forum April 17, 2018 Photo: Michael Collier, A land in motion 25

26 Partial list of collaborators: T. H. Jordan, P. J. Maechling], Y. Cui, S. Callaghan], K. Olsen, R. Taborda, J. Bielak, R. Graves, K. Milner, S. Day, D. Roten,, F. Silva, E. Poyraz, J. Zhou, A. Chourasia, K. Withers, W. Savran, Z. Shi, E. Deelman, G. Juve, K. Vahi, G. Bauer, J. Hill, A. Pope, H. Karaoglu, Y. Isbiliroglu, D. Restrepo, A. Pitarka, N. Abrahamson, R. Archuleta, G. Atkinson, J. Crempien, S.K. Song, D. Mu, A. Breuer, H. Miyake Earthquakes not a question of if... but when. Thank you! HPC User Forum April 17, 2018

27 HPC User Forum April 17,

28 HPC User Forum April 17, 2018

29 Not all damage is structural... HPC User Forum April 17,

30 Resilience by Design Report of the Los Angeles Mayoral Seismic Task Force (Lucy Jones, chair) Los Angeles will have the nation's toughest earthquake safety rules LA Times Released Dec 8, 2014 This Report s approach to evaluating the severity of the risk relies on the ShakeOut Scenario created by a multidisciplinary team convened by the Multi-Hazards Demonstration Project of the USGS Team included USGS, CGS, FEMA,, and nearly 200 other partners in government, academia, emergency response, and industry. An ambitious plan to strengthen buildings fortify water supply and distribution system enhance reliable telecommunications HPC User Forum April 17,

31 Not a question of if... Future earthquakes in California constitute ~ 66% of the long-term national seismic risk HPC User Forum April 17,