Parallel-in-Time Optimization with the General-Purpose XBraid Package

Transcription

1 Parallel-in-Time Optimization with the General-Purpose XBraid Package Stefanie Günther, Jacob Schroder, Robert Falgout, Nicolas Gauger 7th Workshop on Parallel-in-Time methods Wednesday, May 3 rd, 2018 LLNL-PRES This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA Lawrence Livermore National Security, LLC Lawrence Livermore National Laboratory LLNL-PRES

2 Outline Motivation: parallelizing optimization problems Parallel-in-time overview: Multigrid reduction in time (MGRT) XBraid-adjoint code interface : Open source implementation Non-intrusively uses existing user code Example: user-interface for simple scalar ODE problem Lawrence Livermore National Laboratory LLNL-PRES

3 Motivation: PDE constrained optimization Example 1 Objective: Lift maximization Design: Amplitudes of actuation Base flow Runtimes Simulation: 2.5h Optimization: 1,152h Optimized flow 54% lift increase 1. Ötzkaya, Nemili et al., 2015 Lawrence Livermore National Laboratory LLNL-PRES

4 Problem description: Optimization with unsteady PDEs Optimize object function J, with a design variable u (continuous) min 1 T Z T 0 J(y(t),u)dt While satisfying constraint of the forward in time process, (continuous) with state variable y and initial condition g y(t) + c(y(t),u)=0, t 8t 2 (0,T) y(0) = g Lawrence Livermore National Laboratory LLNL-PRES

5 Problem description: Optimization with unsteady PDEs Optimize object function J, with a design variable u (discrete) min 1 NX J(y i,u) J n := J(y n,u) N i=1 While satisfying constraint of the forward in time process, (discrete) with state variable y and initial condition g y n = (y n 1,u), n =1,...,N y 0 = g Lawrence Livermore National Laboratory LLNL-PRES

6 First Order Optimality Conditions Form Lagrangian L = NX (J n +(ȳ n ) T ( n 1 y n )) i 1. State equations: y n = (y n 1,u ), n =1,...,N y 0 = g 2. Adjoint equations: ȳ n = r y nj n +( y n ) T ȳ n, n = N,...,1 ȳ N+1 =0 3. Design equation: NX r u J n +( u n 1 ) T ȳ n 1 =0 n=1 Lawrence Livermore National Laboratory LLNL-PRES

7 Nested Optimization Approach nitial design u i For i = 1, 2, State equations solve: y n = (y n 1,u i ), n =1,...,N y 0 = g 2. Adjoint equations solve: 3. Design update: ȳ n y n ȳ n = r y nj n +( y n ) T ȳ n, n = N,...,1 ȳ N+1 =0 u i+1 = u i B 1 i! NX (r u J n n +( 1 u ) T ȳ n ) n=1 Time Parallel! Time Parallel! Lawrence Livermore National Laboratory LLNL-PRES

8 Outline Motivation: parallelizing optimization problems Parallel-in-time overview: Multigrid reduction in time (MGRT) XBraid-adjoint code interface : Open source implementation Non-intrusively uses existing user code Example: user-interface for simple scalar ODE problem Lawrence Livermore National Laboratory LLNL-PRES

9 Parallel-in-time overview: Approach leverages spatial multigrid research Motivation: Clock speeds are stagnate but speedup still possible through more concurrency à Parallel-in-time is needed The Multigrid V-cycle smoothing (relaxation) prolongation (interpolation) Error on the fine grid restriction Error approximated on a smaller coarse grid Lawrence Livermore National Laboratory LLNL-PRES

10 Multigrid reduction in time (MGRT) General one-step method for a forward evolution process y n = (y n 1,u i ), n =1,...,N n the linear setting (for simplicity), sequential marching forward solve Ay C B y 0 y 1. y N 1 0 C A = g 0 g 1. g N 1 C A g We solve this system iteratively with multigrid reduction (MGR 1979) Replace sequential O(N) method with O(N) parallel alternative Coarsens only in time à non-intrusive, i.e. is "arbitrary" Lawrence Livermore National Laboratory LLNL-PRES

11 Multigrid reduction in time (MGRT) General one-step method for a backward evolution process n the linear setting (for simplicity), sequential marching forward solve Āȳ ȳ n = r y nj n +( y n ) T ȳ n, n = N,...,1 0 T T T 1 0 C B ȳ 0 ȳ 1. ȳ N 1 0 C A = r y 0J 0 r y 1J 1. r y N J N 1 C A We solve this system iteratively with multigrid reduction (MGR 1979) Replace sequential O(N) method with O(N) parallel alternative Coarsens only in time à non-intrusive, i.e. is "arbitrary" Lawrence Livermore National Laboratory LLNL-PRES

12 MGRT for forward solve T 0 T 1 t 0 t 1 t 2 t 3 DT = mdt dt t N F-point (fine grid only) C-point (coarse & fine grid) Relaxation is highly parallel Alternates between F-points and C-points F-relaxation = block Jacobi on each coarse time interval F-relaxation Coarse system approximates fine system m Approximate impractical with (e.g., time rediscretization with ) m A = C A ) B = C A ṁ T Apply recursively for multilevel hierarchy Lawrence Livermore National Laboratory LLNL-PRES

13 A broader summary of MGRT Expose concurrency in the evolution dimension with multigrid Non-intrusive, with unchanged fine-grid problem Converges to same solution as sequential marching C A Only store C-points to minimize storage Optimal for variety of parabolic problems t 0 t 1 t 2 t 3 Converges in ~10 iterations for any coarsening factor Extends to nonlinear problems with FAS formulation n specialized two-level setting, MGRT Parareal Large speedups available, but in a new way Time stepping is already O(N) Useful only beyond a crossover More time steps à more speedup potential Example strong scaling of the 2D heat eqn Much future work to do... dt t N Lawrence Livermore National Laboratory LLNL-PRES

14 Outline Motivation: parallelizing optimization problems Parallel-in-time overview: Multigrid reduction in time (MGRT) XBraid-adjoint code interface : Open source implementation Non-intrusively uses existing user code Example: user-interface for simple scalar ODE problem Lawrence Livermore National Laboratory LLNL-PRES

15 XBraid-adjoint: open source & non-intrusive Solve for y, ȳ, ū := ( J/ u) T (reduced gradient of J w.r.t. design u) Wrap existing user code to obtain time parallelism Standard XBraid: forward in time solve 1. Step: 2. Clone 3. Sum: 4. SpatialNorm: ky n k 5. Buf[Un]Pack 6. nit 7. Free 8. Access y n = (y n 1,u) y m = y n + y m teration k of XBraid: y k+1 XBraid(y k,u k ) J J(y k,u k ) Lawrence Livermore National Laboratory LLNL-PRES

16 XBraid-adjoint: open source & non-intrusive Solve for y, ȳ, ū := ( J/ u) T (reduced gradient of J w.r.t. design u) Wrap existing user code to obtain time parallelism XBraid-Adjoint 1. ObjectiveT: J(y n,u) 2. Step_diff: ȳ n =( y n ) T ȳ n+1 ū +=( u n ) T ȳ n+1 3. ObjectiveT_diff: ȳ n += r y nj n ū += r u J n teration k of XBraid-adjoint: ȳ k+1 XBraid adjoint(y k, ȳ k,u k ) ū J(y k,u k ) u Functions 2 and 3 allow XBraid to compute ȳ n = r y nj n +( y n ) T ȳ n+1 u i+1 = u i B 1 i! NX (r u J n n +( 1 u ) T ȳ n ) n=1 Reduced gradient: ū Lawrence Livermore National Laboratory LLNL-PRES

17 XBraid example: ex-01-adjoint.c Solve for the reduced gradient ū := ( J/ u) T (later use in optimization) Begin with simple problem: J(y, )=1/T s.t. y t = y y(0) = 1 Z T First, user must define objects: App and Vector 0 kykdt Lawrence Livermore National Laboratory LLNL-PRES

18 XBraid example: ex-01-adjoint.c Step(): y n = (y n 1,u)... Lawrence Livermore National Laboratory LLNL-PRES

19 XBraid example: ex-01-adjoint.c Step_diff(): ȳ n =( y n ) T ȳ n+1 ū +=( u n ) T ȳ n+1... Lawrence Livermore National Laboratory LLNL-PRES

20 XBraid example: ex-01-adjoint.c ObjectiveT() and ObjectiveT_diff() are similar nitialize and run XBraid-adjoint:... ū ( J/ u) T Lawrence Livermore National Laboratory LLNL-PRES

21 XBraid example: ex-01-adjoint.c Run it! Lawrence Livermore National Laboratory LLNL-PRES

22 XBraid example: ex-01-optimization.c XBraid-adjoint solver Generically solves adjoint equations, backwards in time Generically computes reduced gradients ū à Designed to work with many optimization methods! Example: ex-01-optimization.c mplements simple reduced space optimization loop (This is a template for implementing your favorite optimization method) Requires: - Global objective evaluation function (and derivative) - Design update function - Halting metric (e.g., gradient norm) Again, the goal is to wrap existing user code! Lawrence Livermore National Laboratory LLNL-PRES

23 Model Problem: 1D advection-diffusion Advection dominated, with Van Der Pol oscillator on left boundary y t + y x y xx =0, =10 5 Tracking objective: Minimize difference of space-time averaged solution to preset value When used with one-shot strategies, the max speedup is 25x runtime primal (sec) N = N = N = cores primal adjoint Scaling of primal (solid lines) and adjoint (dashed lines) XBraid solvers runtime adjoint (sec) Lawrence Livermore National Laboratory LLNL-PRES

24 Summary and Conclusions Parallel time integration is needed on future architectures, especially for adjoint-based optimization techniques New XBraid-adjoint non-intrusively couples with existing codes Available in May, 2018 release References BRAD PARALLEL MULTGRD N TME 1. Günther, Gauger, and Schroder. A Non-ntrusive Parallel-in-Time Approach for Simultaneous Optimization with Unsteady PDEs. Optimization Methods and Software, (submitted), (2018). 2. Günther, Gauger, and Schroder. A Non-ntrusive Parallel-in-Time Adjoint Solver with the XBraid Library. CVS, Springer, (accepted), (2017). Lawrence Livermore National Laboratory LLNL-PRES

25 Thank You! Any Questions? A good read, Parallel Time ntegration with Multigrid, SAM J. Sci. Comp. Open Source XBraid Code BRAD PARALLEL MULTGRD N TME Our Team Rob Falgout Stefanie Günther Matthieu Lecouvez Tzanio Kolev Jacob Schroder Scott MacLachlan Hans De Sterck Stephanie Friedhoff Collaborators CU Boulder (Manteuffel, McCormick, O'Neill), Red Deer (Howse), U Stuttgart (Roehrle, Hessenthaler), Kaiserslautern (Günther, Gauger), LLNL (Woodward, Top) Lawrence Livermore National Laboratory LLNL-PRES

26 Done Lawrence Livermore National Laboratory LLNL-PRES