STCE. Mixed Integer Programming for Call Tree Reversal. J. Lotz, U. Naumann, S. Mitra

Size: px

Start display at page:

Download "STCE. Mixed Integer Programming for Call Tree Reversal. J. Lotz, U. Naumann, S. Mitra"

Vivian Davidson
5 years ago
Views:

1 Mixed Integer Programming or J. Lotz, U. Naumann, S. Mitra LuFG Inormatik 12: Sotware and Tools or Computational Engineering () RWTH Aacen University and Cemical Engineering, Carnegie Mellon University [CSC16, Albuquerque, NM, Oct , 2016]

2 Outline Motivation Conclusion and Outlook Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

3 Outline Motivation Conclusion and Outlook Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

4 Motivation Ceckpointing o Adjoint Simulations Consider solution x τ = x(τ) o initial value problem (IVP) dx dt = (t, x), t 0, x = x(t) IRn, x(0) = x 0 at target time τ > 0. Gradient-based calibration o initial condition x 0 beneits rom adjoint x 0 = dxτ T dx 0 x τ IR n, were x τ IR n is gradient o, e.g, least-squares objective matcing solution o x τ to given observations. Computation o x 0 amounts to solution o adjoint IVP d x dt T d(t, x) = x, τ t 0, x = x(t) IR n, x(τ) = x τ. dx Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

5 Motivation Data Flow Reversal W.l.o.g, explicit Euler time stepping yields or t = τ/m primal x i+1 = x i + t (x i ), i = 0,..., m 1 algoritmic adjoint symbolic adjoint x i = x i+1 + t d T (x i ) x i+1, i = m 1,..., 0 dx x i = x i+1 + t d T (x i+1 ) x i+1, i = m 1,..., 0 dx Note use o primal iterates in reverse order! A. Griewank: Acieving Logaritmic Growt o Temporal and Spatial Complexity in Reverse Automatic Dierentiation, Opt. Met. Sotw. 1, (1992). Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

6 Motivation DAG Reversal Problem: Recovery o V = n + q non-persistent (vertex) values in reverse order (v 5,..., v 1 ); extreme cases: store-all, recompute-all Objective: Minimization o Primal Reevaluation Cost (PRC) or given upper bound M on Persistent Memory Requirement (PMR) Complexity: Fixed Cost (n + q) Minimum Memory Data Flow Reversal by reduction rom Vertex Cover solvable by O(n + q) instances o Fixed Memory Minimum Cost Data Flow Reversal U.N.: DAG Reversal is NP-Complete, J. Disc. Alg. 7(4), (2009) G = (V, E) n = 2; q = 5 Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

7 Outline Motivation Conclusion and Outlook Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

8 Problem Description and Computational Complexity primal 1 g augmented primal adjoint 1 store arguments restore arguments 1000 Objective: Reversal sceme R : E {0, 1} E minimizing PRC or upper bound M on PMR and given annotated call tree T = (V, E) Extreme Cases: R = 0 (ully split; ceckpoint none); R = 1 (ully joint; ceckpoint all) U.N.: is NP-Complete, LNCSE 64, (2008). Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

9 Example: Let MEM = R=(0,0): g g R=(1,1): +1 g g g MEM=1220, OPS= MEM=1110, OPS= Notation: Set S o subprogram calls; Number n[i] o subprogram calls in i S; Set χ(i) o callees; PMR m[i] = n[i] j=0 m[i] j; Reversal sceme R = (r[i]) i S ; PMR M[i] ater augmented primal run; PMR M[i] prior to adjoint run; PMR M[i] o argument ceckpoint Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

10 Example: Let MEM = Greedy Heuristics Smallest Memory Increase starts rom R = 1 and yields... Largest Memory Decrease (LMD) starts rom R = 0 and yields... R=(0,1): g MEM=1110, OPS=1000 g R=(1,0): +1 g g g Largest Memory Increase (LMI) remains at R = 1 as R = (1, 0) ineasible 1000 MEM=1120, OPS= Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

11 Mixed Integer Programming Formulation (Objective) We aim to balance PRC S r[] c[], cost induced by writing/reading ceckpoints (memory traic) c S r[] and implementation eort due to number o ceckpointed routines ĉ S ˆr[] c(r) = r[] c[] + c r[] + ĉ ˆr[], S S S were ˆr[] {0, 1} vanises unless at least one call o is ceckpointed, c[] denotes te PRC o and c and ĉ are used to weig impacts due to memory traic and implementation eort, respectively. Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

12 Mixed Integer Programming Formulation (Constraints) We deine PMR or eac subprogram s ater execution o its augmented primal M[g] = m[g]+ [ ] (1 r[]) M[] + r[] M[]. χ[g] and prior to execution o its adjoint, e.g, M[g] n[g] = M[ i ] m[] i ( ) M[g] + r[g] M[g] Te maximum PMR is reaced prior to execution o adjoint subprograms, ence, max g S M[g]n[g] must not exceed te upper bound M. J. Lotz, U.N., S. Mitra: Mixed Integer Programming or, SIAM CSC (2016). Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

13 Numerical Results (Toy) g g i 50 For M = 125 we get MIP LMI LMD R (1,0,0,1) (0,0,1,0) (1,1,0,0) PMR c(r) c[] = 200, c[] = 120, c[i] = 30, and c[g] = 40. Te cost c[] is cosen to be muc iger tan te corresponding memory requirement to mimic some time consuming operation in (e.g, ile I/O) witout eect on PMR. Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

14 Numerical Results (Towards Reality) We consider te solution o an elliptic boundary value problem wit PETSc v3.3. A discrete adjoint version was generated using dco/c++ and AMPI. Bars visualize te overead due to PRC. Statistics: 1500 source iles, 2717 subprograms instrumented based on dco/c++ count mode, 443k subprogram calls, PMR or R = 0 equal to 94GB, runtime o MIP analysis/optimization equal to 40s ,000 8,000 64,000 Memory Bound in MB LMI LMD CPLEX J. Lotz, U.N., M. Scanen: Discrete Adjoints o PETSc troug dco/c++ and Adjoint MPI, Euro-Par 2013 Parallel Processing, Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

15 Outline Motivation Conclusion and Outlook Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

16 Conclusion Tecnical Callenges automated instrumentation o large code bases ( clang) application to call graps requires conservatism and abstract interpretation or annotation combination wit static data low analysis desirable deinition o corresponding adjoint code design patterns is work in progress U.N.: Adjoint Code Design Patterns, AD2016. L. Hascoët, U.N., V. Pascual: To Be Recorded Analysis in Reverse-Mode Automatic Dierentiation, Future Generation Computer Systems 21(8): , Elsevier (2005). Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

17 Outlook Result Ceckpointing g g 1 g g MEM=1110, OPS= U.N.: Te Art o Dierentiating Computer Programs, SIAM (2012). Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

18 Outlook Binomial Ceckpointing 0:9 0:9 0:3 4:9 4:9 4:9 0:3 0:3 +4 4:6 7:9 4:6 4:6 * 0:0 1:3 1:3 1:3 0: * * 4:4 5:6 4:4 * * 1:1 2:3 1: recursive bisection (dynamic programming) local recompute all repeated accesses to ceckpoints A. Griewank, A. Walter: Algoritm 799: revolve: an implementation o ceckpointing or te reverse or adjoint mode o computational dierentiation, ACM Transactions on Matematical Sotware 26(1):19 45, ACM (2000). Lotz, Naumann, Mitra, CSC16, Albuquerque, NM, Oct ,

The Art of Differentiating Computer Programs

The Art of Differentiating Computer Programs Uwe Naumann LuFG Informatik 12 Software and Tools for Computational Engineering RWTH Aachen University, Germany naumann@stce.rwth-aachen.de www.stce.rwth-aachen.de