1st Committee Meeting

1st Committee Meeting Eric Liu Committee: David Darmofal, Qiqi Wang, Masayuki Yano Massachusetts Institute of Technology Department of Aeronautics and Astronautics Aerospace Computational Design Laboratory

Outline 1 Introduction and Motivation 2 Existing Work 3 Expected Contributions 4 Logistics E. Liu (MIT) 1st Committee Meeting 2 / 41

Outline 1 Introduction and Motivation 2 Existing Work 3 Expected Contributions 4 Logistics E. Liu (MIT) 1st Committee Meeting 3 / 41

Motivation: Next Generation CFD Constantly-increasing computational power allows for higher-fidelity simulations of multiscale problems; e.g., unsteady RANS LES/DES/DNS aeroacoustics Most of these problems are unsteady Need for automatic error control and methods that can utilize DOFs efficiently (a) Axial velocity at combustor/turbine interface (Medic 2006) (b) Pressure field of SSLV (Mavriplis 2007) E. Liu (MIT) 1st Committee Meeting 4 / 41

Motivation: Space-Time Anisotropy Many unsteady problems are anisotropic in space-time e.g., wave-propagation: wave support propagation distance Appropriate mesh anisotropy efficiency gains (Yano et al. 2011) Recognized in the space-time literature: Key to an effective space-time adaptation strategy is attributing the error indicator to spatial versus temporal resolution using a measure of space-time anisotropy. (Luo, Fidkowski, 2011) Simplex meshes can match arbitrary solution anisotropy Consider a simple 1D + 1 example... E. Liu (MIT) 1st Committee Meeting 5 / 41

Motivation: DOF Reduction 0.5 1.5 s=1 Ideal 1D 0 0.25 1 "Realistic" 1D 0 0.2 0.4 0.6 1 (a) 1D Meshes t x (b) Ideal Simplex Consider a 1D, stationary shock problem (Left) Ideal 1D mesh with 2 elements is shown Shock-tracking unreasonable; more realistic, 5 element mesh also shown Now consider a 1D, moving shock problem (Right) Ideal, 3 element simplex mesh is shown Shock path highlighted in red (s = 1) E. Liu (MIT) 1st Committee Meeting 6 / 41

Motivation: DOF Reduction t t x (a) Uniform Quad (N uni elem) x (b) Adapted Quad (O( N uni log N uni) elem) Assumed current state-of-the-art in ST adaptation: e.g., Bangerth 1999, Hartmann 2001, Fidkowski 2011 Sample mesh shown in (b) Hierarchical, (usually) tensor-product mesh structure 1 hanging node per edge E. Liu (MIT) 1st Committee Meeting 7 / 41

Motivation: DOF Reduction t (a) Anisotropic Simplex x Simplex-adaptive ST potential: N 2D = C 1 N 1D elements! Best-case scenario N 1D = C 2 + O(log N uni ), C 2 = number of elements spanning shock C 1 is a constant dependent on: maximum element anisotropy, discretization dissipation, dispersion properties Result: dimension of the problem is reduced E. Liu (MIT) 1st Committee Meeting 8 / 41

Sample Works: Adaptive Space-Time Methods Bar-Yoseph (1989) First application of DG to space-time problems 1D + 1 and 2D + 1 scalar advection Hartmann (2001, 2002) DWR-driven (isotropic) mesh adaptation 1D + 1 Burgers and Euler (a) Hartmann (2002) Luo, Fidkowski (2011, 2012) Applies anisotropy measures of Leicht, Hartmann (2010) Linear solve cost nearly reduced to spatial linear solve (Richter 2009) Navier-Stokes Note: All prior works (all examples here) use hierarchical meshes. E. Liu (MIT) 1st Committee Meeting 9 / 41

Objective Development of an autonomous, robust, and practical solution strategy for (certain) unsteady problems using an adaptive, space-time DG method. Approaches High-order DG space-time solver DWR-based error estimation Mesh-optimization adaptation scheme (Yano 2012) Metric-based mesh adaptation (BAMG, EPIC/MADCAP) E. Liu (MIT) 1st Committee Meeting 10 / 41

General Approach Higher-order solutions with the Discontinuous Galerkin (DG) method p-th order polynomial basis on each element; discontinuous between elements Nearest-neighbor stencil y x T H u(x,y) Anisotropic simplex meshes for complex geometries Output-based mesh adaptation: Error estimation on integrated outputs of interest (e.g., lift, drag) Element-wise localization of error estimates E. Liu (MIT) 1st Committee Meeting 11 / 41

Output Adaptive Solution Strategy Problem: Find output J(u) where u satisfies governing equations DGFEM: Find discrete output, J h,p, with u h,p V h,p s.t. R h,p (u h,p, v h,p ) = 0 v h,p V h,p. Output-based adaptation: estimate and control the error: ε J(u) J h,p (u h,p ). E. Liu (MIT) 1st Committee Meeting 12 / 41

Error Estimation Dual-Weighted Residual (DWR) of Becker and Rannacher (2001): ε = R h,p (u h,p, ψ) = R h,p[uu h,p ](u u h,p, ψ ψ h,p ) R h,p (u h,p, ψ h,p ) Localized error indicator: η K R h,p (u h,p, ψ h,p K ), where p > p. Primal solution features (left) and dual solution features (right). E. Liu (MIT) 1st Committee Meeting 13 / 41

Mesh Optimization via Error Sampling and Synthesis (MOESS) Driven by DWR (a posteriori) error estimator Resulting mesh resolves primal and dual solution features No a priori assumptions about solution irregularity Robust method can start from almost any mesh (a) Mesh-Metric Duality E. Liu (MIT) 1st Committee Meeting 14 / 41

Mesh Optimization via Error Sampling and Synthesis (MOESS) Mesh optimization (relaxed): M = arg inf M E(M) s.t. C(M) = DOF target 1 Sample error response to local metric modifications 2 Synthesize {M Ki, η Ki } pairs into local approximations of (roughly) E M 3 Solve optimization problem via steepest descent E. Liu (MIT) 1st Committee Meeting 15 / 41

Outline 1 Introduction and Motivation 2 Existing Work 3 Expected Contributions 4 Logistics E. Liu (MIT) 1st Committee Meeting 16 / 41

Overview Yano (2012) produced a series of preliminary adaptive space-time results. Results Included... 1D + 1/2D + 1 Wave Equation (not shown) 1D + 1/2D + 1 Euler Equation Yano s results are now presented to further motivate space-time schemes. Validation We will show the entropy mode (1D) and shear mode (2D) We will also show a 1D + 1 Riemann Problem (modified Sod s) E. Liu (MIT) 1st Committee Meeting 17 / 41

Euler Equation Validation (1D + 1) (Yano 2012) (a) entropy (b) mesh (c) convergence Gaussian perturbation of density at constant pressure Output: entropy perturbation at final time Adaptation reduces DOF usage substantially Mesh anisotropy is nearly perfect, with 3 elements spanning time Hence the effective dimension is reduced E. Liu (MIT) 1st Committee Meeting 18 / 41

Euler Equation Validation (2D + 1) (Yano 2012) (a) density Excitation of shear mode using bivariate Gaussian with the procedure of Wang (2006) High level of anisotropy observed in the mesh (b) mesh E. Liu (MIT) 1st Committee Meeting 19 / 41

Euler Equation Validation (2D + 1) (Yano 2012) Output: momentum perturbation at the final time Expected convergence rate: h 2p ; correlates to N p DOF Adapted cases superconverge Obtains 2D rates even though problem is 3D Appropriate anisotropy dimensionality reduction in 2D, N2p/3 DOF in 3D E. Liu (MIT) 1st Committee Meeting 20 / 41

(Modified) Sod s Problem: Inviscid Shock (1D + 1) (Yano 2012) Initial Conditions: x < 0: (ρ L, v L, p L ) = (1.0, 0.0, 1.0) x > 0: (ρ R, v R, p R ) = (0.4, 0.0, 0.4) (a) density (b) mesh Output: density perturbation at final time Space-time adaptation appears effective for this problem Shock-capturing (Barter 2008) not robust for stronger shocks E. Liu (MIT) 1st Committee Meeting 21 / 41

Outline 1 Introduction and Motivation 2 Existing Work 3 Expected Contributions 4 Logistics E. Liu (MIT) 1st Committee Meeting 22 / 41

Overview Objective Development of an autonomous, robust, and practical solution strategy for (certain) unsteady problems using an adaptive, space-time DG method. Challenges Performance: space-time is expensive without special handling of time Robustness: current shock-capturing scheme lacks robustness Expected Contributions Preconditioning, solver efficiency improvements Shock-capturing robustness and error estimation improvements Demonstration of capability on complex 2D + 1 problems (Time permitting) Exploration of adaptive space-time schemes applied to near-turbulent flow E. Liu (MIT) 1st Committee Meeting 23 / 41

Preconditioning: Long Journey Ahead Currently, we run steady 3D cases at... p = 2 or p = 3 with 1M DOFs (e.g., 100K p = 2 elements) Insufficient for RANS problems Potentially workable for Euler shock problems Memory Usage for p = 2 primal (p = 3 dual) (Jacobian Only) N elem p = 2, 3D p = 3, 3D p = 2, 4D p = 3, 4D 100K 9.31GB 37.3GB 402GB 2.19TB 1M 93.1GB 373GB 4.02TB 21.9TB 10M 931GB 3.73TB 40.2TB 219TB 100M 9.31TB 37.3TB 402TB 2.19PB For reference, the upcoming ACDL cluster will have 40 32GB = 1.28TB or about 50,000 4D elements. E. Liu (MIT) 1st Committee Meeting 24 / 41

Preconditioning Ignoring causality of time = high cost Time-direction has strong hyperbolic structure Potentially reduce cost with appropriate preconditioning Care needed so that causality is not violated May also reduce cost with solver modifications/optimizations Some Ideas Parallel decomposition cut purely in the time direction? Time-slab structure upper block triangular Jacobian Insert cut planes and use cut cell techniques? Use mesher capable of internal, material boundaries? Line-preconditioning: order elements along strongly causal lines and perform Gauss-Seidel-like sweep E. Liu (MIT) 1st Committee Meeting 25 / 41

Reducing Jacobian Storage t t (a) Time slabs via cut cells x (b) Meshing with time slabs x Time-slabs can help with the storage issue: Time-slabs allow serialization in time Store Jacobian for only one slab at a time Traditional domain decomposition (e.g., METIS) in space to parallelize Try avoiding Jacobian storage for p + 1 truth surrogate: Strong solvers (e.g., GMRES) implemented matrix-free Weak smoothers (e.g., block Jacobi) that do not require Jacobian storage E. Liu (MIT) 1st Committee Meeting 26 / 41

Shock-Capturing Currently use a scheme that governs artificial viscosity with a PDE Pros PDE adds artificial viscosity smoothly across elements Reduces oscillations/solution pollution downstream compared to element-wise constant viscosity Cons Lack of robustness for strong shocks (steady and space-time settings) Increased cost due to extra PDE coupled to flow equations How to handle error estimation? Don t necessarily want to adapt to features of the shock PDE Cons may be from poor tuning or incorrect extension to space-time Try p = 0 subdivision technique of Huerta, Casoni, Peraire (2011)? Error estimation questions remain with or without the shock PDE E. Liu (MIT) 1st Committee Meeting 27 / 41

Complex 2D + 1 Problems 3D + 1 problems are probably too cost-prohibitive (memory). Instead, consider more complex 2D + 1 problems, for example... Flow through rotor-stator rows: many complex interactions Stationary/rotating blade rows Interactions between wakes and downstream blades Shock-boundary layer interaction Complex heat-transfer problem Large, unsteady force variations Other problems possible; may try 3D + 1 depending on progress (c) Entropy, transonic turbine (Paniagua 2008) (d) Density, 3 stator-rotor rows (Cizmas 1998) E. Liu (MIT) 1st Committee Meeting 28 / 41

Complex 2D + 1 Problems: Lead-up Cannot start solving rotor-stator problems tomorrow. Work toward goal with simpler problems: 1D + 1 Sod s Problem, other traditional shock-tube problems Shock-tube with moving walls 2D + 1 Euler, Navier-Stokes, then RANS-SA with subsonic, then transonic flow Shock-tube, extruded Isolated airfoil with periodic BCs, time-varying BCs Euler-only: isolated airfoil with specified vortex-wake inflow Single blade rotor Multiple airfoils (stationary, then rotating) (Hopefully) Comparison with industry code from Pratt E. Liu (MIT) 1st Committee Meeting 29 / 41

Broader Classes of Problems Have noted that space-time makes sense for problems propagating few, complex phenomena with limited feature width (shocks, waves, vortices, etc.) What about flows with less distinct/obvious features? Near-turbulent or onset-turbulent flow Rotorcraft Or PDE problems from other fields? Solid Mechanics (e.g., elasticity) Maxwell s Equations, Magnetohydrodynamics Shallow water (e.g., climate problems) E. Liu (MIT) 1st Committee Meeting 30 / 41

Outline 1 Introduction and Motivation 2 Existing Work 3 Expected Contributions 4 Logistics E. Liu (MIT) 1st Committee Meeting 31 / 41

Proposed Timeline Time Task Now - 09/12 Develop, defend thesis proposal 08/12-12/12 Improve preconditioning 1/13-4/13 Improve shock capturing implementation (robustness, error estimation) 5/13-10/13 Apply method to complex 2D + 1 problems (time allowing) Investigate method effectiveness on nearturbulent flow 11/13-4/14 Write, defend thesis E. Liu (MIT) 1st Committee Meeting 32 / 41

Completed Coursework Major: Aerospace Computational Engineering 2.25 Advanced Fluid Mechanics A- 6.255 Optimization Methods A 16.110* Flight Vehicle Aerodynamics A 16.13 Aerodynamics of Viscous Fluids A 16.225 Computational Mechanics of Materials B+ 16.920* Numerical Methods for PDEs A 16.930 Adv. Topics in Numerical Methods for PDEs A 18.335 Introduction to Numerical Methods A+ 18.336 Numerical Methods for PDEs A+ Minor: Theoretical Computer Science 6.849 Geometric Folding Algorithms A+ 6.851 Advanced Data Structures A 6.854 Advanced Algorithms A 6.856 Randomized Algorithms A 18.409 Topics in Theoretical CS A * Indicates that the course was taken as an undergraduate. E. Liu (MIT) 1st Committee Meeting 33 / 41

Backup Slides E. Liu (MIT) 1st Committee Meeting 34 / 41

Motivation: DOF Reduction Ignoring the dissipation issue... Consider feature-propagation in 1D. Let: L propagation distance, l feature width (=0 for shocks) x element size (1D), N elem,1d = l x Then the number of elements needed scales like: L l Current (isotropic) methods: Potential (anisotropic) method: Dimension Reduction x x = L l N2 elem,1d L l AR x x = 1 AR L l N2 elem,1d = Ω(N elem,1d) 1 L Generalizes to higher-dimensions: AR l N2 elem,nd+1 = Ω(N elem,nd) Dimension reduction saves DOFs by exploiting higher AR to reduce the effect of L x E. Liu (MIT) 1st Committee Meeting 35 / 41

Euler Equation Implementation The Euler Equations can be written: F(U(x)) = F ij x j = 0, x Ω where Ω is a space-time domain and = ( t, x 1, x 2, x 3 ). F : R 5 R 5 4 is defined as: ρ ρu 1 ρu 2 ρu 3 ρu 1 ρu 2 1 F = + p ρu 1u 2 ρu 1 u 3 ρu 2 ρu 1 u 2 ρu 2 2 + p ρu 2u 3 ρu 3 ρu 1 u 3 ρu 2 u 3 ρu 2 3 + p ρe (ρe + p)u 1 (ρe + p)u 2 (ρe + p)u 3 Roe Flux Requirements Reduces to full-upwinding for faces with normal (t, 0, 0, 0) Reduces to standard Roe for faces with normal (0, x 1, x 2, x 3 ) E. Liu (MIT) 1st Committee Meeting 36 / 41

Euler Equation Implementation Standard Roe F Roe = 1 2 (F n,r + F n,l ) 1 2 A Roe (U R U L ) with eigenvalues v n ± a and v n. Space-Time Roe Apply Roe s method to Un 0 + F spatial n s Roe-averaging unchanged (for normal velocities, etc. use n s, the spatial normal, which need not have unit length) Eigenvalues become v n ± a n s + n t and v n + n t Validation Verified that Riemann invariants were preserved along characteristic lines and propagated at the correct speed. We will show the entropy mode (1D) and shear mode (2D) E. Liu (MIT) 1st Committee Meeting 37 / 41

Preconditioning: Long Journey Ahead The cost issue is severe. Let s look at some (rough) numbers: Number of Basis Functions 2D 3D 4D 1 2 (p + 1)(p + 2) 1 6 (p + 1)(p + 2)(p + 3) 1 24 (p + 1)... (p + 4) (p = 0) 1 1 1 (p = 1) 3 4 5 (p = 2) 6 10 15 (p = 3) 10 20 35 (p = 4) 15 35 70 (p = 5) 21 56 126 E. Liu (MIT) 1st Committee Meeting 38 / 41

Preconditioning: Long Journey Ahead Currently, we run 3D cases at... p = 2 or p = 3 100K p = 2 elements or 1M DOFs Insufficient for RANS problems Potentially workable for Euler shock problems Consider memory usage for p = 2 primal (p = 3 dual) solves: Memory Usage for 3D and 4D (Jacobian+GMRES) N elem p = 2, 3D p = 3, 3D p = 2, 4D p = 3, 4D 10K 1.6GB 5.2GB 44.7GB 229GB 30K 5.0GB 15.6GB 134GB 688GB 100K 16.8GB 52.2GB 447GB 2.30TB 300K 50.3GB 156GB 1.3TB 6.9TB 1M 167GB 522GB 4.5TB 23TB For reference, the upcoming ACDL cluster will have 40 32GB = 1.28TB GB or about 50,000 4D elements. E. Liu (MIT) 1st Committee Meeting 39 / 41

3D + 1 Capability Interesting engineering problems are essentially all fully 3D. 3D + 1 adaptive space-time simulations pose meshing challenges: Anisotropic, metric-based simplex meshing in 4D Ability to handle internal (material) boundaries Visualization Impossible to even look at the entire mesh Even in 3D, looking at 2D mesh slices is minimally informative Very challenging problem! Simplify by considering simple geometry only. Two current ideas for meshing: Generalize method of Loseille (edge-split primitives) to 4D 4D Anisotropic Delaunay (disadvantage: leaves no path toward more complex geometries) Depending on time, may stick with solving more complex 2D problems; e.g., stator-rotor flows E. Liu (MIT) 1st Committee Meeting 40 / 41

Disk Space Requirements: Space-Time vs. Method of Lines Memory Usage for 3D and 4D Solution Vector N elem p = 2, 3D p = 3, 3D p = 2, 4D p = 3, 4D 1M 0.447GB 0.894GB 2.68GB 6.26GB 1.8M 0.804GB 1.61GB 4.83GB 11.2GB 10M 4.47GB 8.94GB 26.8GB 62.6GB 100M 44.7GB 89.4GB 268GB 626GB New ACDL cluster has enough memory for at most p = 3 adjoint on 1.8M elements in 3D. Require 100-1000 checkpoints for unsteady adjoint solves 80GB to 800GB storage for p = 2, 3D, 1.8M elem Checkpoints are subset of 10 4 to 10 5 time-steps Space-time solution for the same problem size requires 500GB Assumes 100 time-spanning elements (= 100M total) But ACDL cluster may be unable to solve such a large problem E. Liu (MIT) 1st Committee Meeting 41 / 41