Nested Differentiation and Symmetric Hessian Graphs with ADTAGEO

Transcription

1 Nested Differentiation and Symmetric Hessian Graphs with ADTAGEO ADjoints and TAngents by Graph Elimination Ordering Jan Riehme Andreas Griewank Institute for Applied Mathematics Humboldt Universität zu Berlin 7th December th Euro AD Workshop Aachen, Germany

2 Outline 1 Introduction 2 Nested Automatic Differentiation 3 Propagation within ADTAGEO 4 Hessian Graphs in ADTAGEO 5 Symmteric Hessian Graphs in ADTAGEO 6 Conclusion and Outlook

3 Introduction Introduction

4 Introduction Maintain a Live -DAG Eliminate as soon as possible as many vertecies as possible. DAG represents the active variables alive at any one time. Small graph Memory savings ADTAGEO performs vertex elimination whenever (i) An active variable is deallocated/destroyed (ii) An active variable is overwritten

5 Introduction Instant Elimination Looking at a function call: void f( double x1, double x2, double & z ) { double y1 = x1*x2, y2 = x1+x2; z = x1*y1*y2; } inside scope of f after leaving scope of f x1 c y1,x2 x2 x1 x2 c y1,x1 y1 c z,x1 c z,y1 c y2,x2 c y2,x1 c z,y2 y2 c z,x1 c z,x2 z z

6 may sound complicated... Perfect fitting into OOP scenario (i) is covered by Destructor (assuming it exists in language) (ii) is covered by assignment operator Prototype in C++ Proof of concept optimized for understanding, not optimized for speed Heavy use of class map from the Standard Template Library to store partials locally at every node (edges in graph) Rapid prototyping (First Order Derivatives): 140 lines of code for +-*/ and sin, cos, exp Rapid prototyping Hessian, Second Order Elimination: 100 additional lines of code for Hessian elimination

7 DAGLAD - Local graph storage class daglad{ private: double val; map<daglad*, double> args; map<daglad*, double> uses; hurry //function value //arguments = incoming edges //used by = outgoing edges public: daglad() {...}; //constructor void eliminate() {...}; //eliminate current vertex ~daglad() { eliminate();...}; //destructor void operator = (...) { eliminate();...}; // asgnm. friend dagdoub operator + (...); // arithmetic operators friend double operator % (...);... // retrieval op }; /* class daglad */

8 ADTAGEO Local Graph Representation Program: y = x1 * x2 * x3; z = x3 * x4 * y; x1 x2 x3 x4 y.args y y.uses z Note: Every partial is stored as attribute of a forward and corresponding reverse arcs.

9 ADTAGEO Overview No specification of independents/dependents Choose mixture of forward / reverse mode by variable allocation No tape on disc, Graph is managed and reduced memory Correctness of derivatives has to be ensured (full accumulation) No top-level routine concept Access to derivatives at any time = Nested Automatic Differentiation = Graph represents the sparsity structure of the Jacobian BUT: ADTAGEO is not only dynamic sparsity propagation ADTAGEO computes derivatives in sparse mode, therefore no structural zeros are computed No need to propagate a seed matrix / directions /... No need for Jacobian compression

10 ADTAGEO Derivative Retrieval Easy mode: Redeclare (required) variables to be of type daglad Retrieve first order partials y x on theee arc between variables x and y somewhere in the code using the % operator y[j]%x[i] y j x i x and y are vectors in your C++ program) Advanced mode: Check/prepare/write code for better performance Right mixture of forward and reverse mode Hybrid Forward-Reverse-sweeps Using propagators of ADTAGEO to ensure fully accummulated derivatives

11 Nested Automatic Differentiation Nested Automatic Differentiation

12 ?????? Newtons Method Suppose we have For fixed u = û lets look at F = F (y, u) : R 2 R F (y, û) = 0 = Newton s Method needs F y = f%y in ADTAGEO. How about solving F (y, û) = 0 and taking the derivative y(u) (û) =??????? u

13 ?????? Newtons Method Subprogram: #include "daglad.h" void newton(daglad &y, daglad &u, double &eps) { daglad f; // f (y, u) function value daglad t = 3*eps; // f /f, update while (fabs(t.val()) > eps ) // terminate loop if f /f ɛ { f = sin(exp(y)) - cos(exp(y)) * u * u; // compute f t = f / ( f%y ); // compute update using ADTAGEO y = y-t; // update iterate } } // end of newton

14 ?????? Main program Newton Main Program: main(){ double eps, dy; daglad y, u; cout << eps = ; cin >> eps; } y=0; u=0.5; // initialise newton(y,u,eps); // call newton method dy = y%u ; // retrieve derivative, AD // end of newton

15 Nested Automatic Differentiation Pain-free!!! Suppose we have For fixed u = û lets look at F = F (y, u) : R 2 R F (y, û) = 0 = Newton s Method needs F y = f%y in ADTAGEO. How about solving F (y, û) = 0 and taking the derivative ADTAGEOs answer: y(u) (û) =??????? u y(u) (û) = y%u u

16 Nested Automatic Differentiation Application Least Squares Scenario: 1 min y y 2 s.t. F (y, u) = 0 (y,u) 2 Derivatives needed in Newton J = F y = F%y Rn n Derivatives needed in Gauss - Newton K = y u = y%u Rn m with m n

17 Nested Automatic Differentiation Application Function eval y(y, u) F (y, u) = 0 with u fixed by Newton r = F (y, u) R n While r > ɛ Compute residual J = y r R n n [L, U] = LUdecomp(J) dy = LUsolve(L, U, r) y = y dy r = F (y, u) R n return y passive: partial Jacobian of r wrt. y passive: get LU decomposition of J active: compute update dy on y Apply update Compute residual

18 Nested Automatic Differentiation Application Least Squares with Gauss - Newton Target state y is given Choose initial control u and state y y = eval y(y, u) While y y 2 >2 ɛ Compute state y K = u y R n m passive: Reduced Jacobian of y wrt. u [Q, R] = QRdecomp(K) passive du = Backsolve(R, Q T (y y )) active u = u du Apply update on control y = eval y(y, u) R n Compute new state y u = u, y = y solution (u, y )

19 Nested Automatic Differentiation Application Example 11.1 from Nocedal/Wright: [Rheinboldt] [ ] y F (y, u) = A + φ(y) = 0 y R 5, u R 3, A R 5 8 u φ : R 5 R 5 non - linear State y: flight configuration of an aircraft y 1... roll y 2... pitch y 3... yaw y 4... incremental angle of attack y 5... sideslip angle Control u: controls of the aircraft u 1... elevator u 2... aileron u 3... rudder Implicit function y = y(u) : R 3 R 5 for fixed control u

20 Nested AD Numerical results lsqr START **Y** = [ ] START y = [ ] START u = [ ]... END res = 2.63e-13 END **Y** = [ ] END y = [ ] END u = [ ]

21 Nested AD Complexity Considerations Assumption (A1) : fixed number of Newton steps small number of Gauss - Newton steps Computation of y = eval y(y, u) (J R n n, Forward mode) OPS(u y) OPS(J 1 F ) = OPS((y, u) J) n3 n OPS((y, u) F ) Computation of F u (K R n m, Reverse mode) OPS((y, u) F ) m OPS((y, u) F ) u

22 Nested AD Optimal Reduced Jacobian All together we have OPS((y, u) u ) OPS(u K) + OPS(u J 1 F ) = OPS((y, u) F u ) + OPS(J 1 ) m OPS((y, u) F ) + OPS(u y) and since m <= n we have m OPS((y, u) F ) OPS(u y) and therefore we have for the Reduced Jacobian K in Gauss-Newton OPS((y, u) u ) OPS(u y) in contrast to general AD assumption OPS((y, u) u ) n OPS(u y)

23 Nested AD Verifying Complexity Central differences on discretization of x y + ɛ e y = 0 with N gridpoints in each direction y i,j = y(i h, j h), i = 1(1)n, j = 1(1)n y i 1,j + y i+1,j + y i,j 1 + y i,j+1 4y }{{ i,j + h 2 ɛ e yi,j = 0 } approximation of x y i,j So far: results for dense LU decomposition N n A P

24 Nested AD Verifying Complexity LU dense, active LU dense, passive

25 Propagation within ADTAGEO Propagation within ADTAGEO

26 Propagation within ADTAGEO Setup Example (from Andreas Book): y = f (x) = exp(x 1 ) sin(x 1 + x 2 ) v -1 v 3 1 v 4 v 0 1 v 1 cos(v 1 ) v 3 Main Propagation Pinciple Instant Propagation Whenever a vertex v i got the contributions v j c ij from all of its predecessors v j with j i, than the now fully accumulated v i = j i v i c ij has to be propagated immediately to successors of v i.

27 Propagation within ADTAGEO Forward, F ẋ Propagating ẋ = (1, 1) T ẋ 1 = 1 v -1 v 3 v 3 + v cos(v 3 1 ) 1 ẋ 2 = 1 12 v 0 1 v 1 cos(v 1 ) 2 cos(v 1 ) v 3 v 4

28 Propagation within ADTAGEO Notes Propagation does not require any searches, since successors of vertex v i can be visited in arbitrary order, thus especially as stored in the successors set of v i Propagation touches every arc exactly once Propagation appears always on a fixed graph, that might change again after propagation finished Propagation in reverse mode too Propagation might start somewhere in the graph This is in fact Nested Automatic Differentiation Evaluation of vector products with partial Jacobians (e.g. Jacobians of parts of the original function)

29 Propagation within ADTAGEO Notes Lemma The simple propagation technique from above computes F ẋ iff the vertices v i,i = 1... n, selected by seeding with ẋ R n forms a vertex cut in the graph. To have a Lemma is nice, but it tells a bad story: We have to search a vertex cut for seeding, that contains all x i selected by ẋ, and add some x j = 0. That is we have to propagate zeros. Really???

30 ADTAGEO Sparse Propagation Sparse propagation Add a level number to vertices instead of counting contributions Store every vertex on first visit in a priority queue Q sorted by ascending levels (F ẋ) or descending levels (ȳf ) Remove every finished vertex from Q (somehow) If propagation stops, but Q is not empty Take first vertex v i from Q and continue propagation from v i until Q is empty Lemma (i) Sparse Propagation in ADTAGEO always terminates. (ii) All zeros in the initial direction ẋ [ȳ] can be ignored.

31 Hessian Graphs in ADTAGEO Hessian Graphs in ADTAGEO

32 Hessian Graphs in ADTAGEO First European AD-Workshop in Nice, 2005: Storing sparse local Hessians in ADTAGEO utilizing symmetry Second order elimination rules preserving symmetry Example (from Andreas Book): y = f (x) = exp(x 1 ) sin(x 1 + x 2 ) v -1 v 3 1 v 0 1 v 1 -v 3 cos(v 1 ) v 3 1 v 4

33 ADTAGEO Annotated Hessian Arcs Annotated Hessian arcs... from v i pointing towards the predecessors of v i with the fully accumulated adjoint v i = i k v kc ki of v i v -1 v 0 1 v 1 Look at Reverse propagation of v 4 from v 4 v 4 v (cos( v 4 cos(v 1 ) + 1 ) v 3 ) v 3 v 4 1 -v 3 v 3 cos(v 1 ) v 3 v 3 v 4 1 v 4 v 4 v 4 v 4 cos(v 1 ) v 4 cos(v 1 )

34 Symmetric Second Order Propagation Second order propagation to compute ȳf ẋ Combine reverse propagation of v i with special second order propagation rules (similiarity to elimination rules), which incoporates v i from forward propagation of ẋ. v -1 v 3 1 v 0 1 v 1 -v 3 v 3 cos(v 1 ) v 3 1 v 4 v 4

35 Symmteric Hessian Graphs in ADTAGEO Symmteric Hessian Graphs in ADTAGEO

36 Symmteric Hessian Graphs in ADTAGEO You might ask: Where is the symmetric Hessian graph from Andreas book? v -1 v2 v-1 v 4 1 v 4 v 4 v 4 -v 3 v 3 v 3 v 4 v 3 v 0 v 1 v 1 v 0 v 3 v 3

37 Symmteric Hessian Graphs in ADTAGEO You might ask: Where is the symmetric Hessian graph from Andreas book? v -1 v 4 v -1 v 4 v 4 v 3 v 4 v 3 v 0 v 1 v 1 v 0 Here it is!! v 3 v 3

38 Conclusion and Outlook Conclusion You have seen ADTAGEO as a new AD concept You have seen Pain-free Nested Automatic Differentation You heard about Optimal Reduced Jacobian You have seenautomatic Exploitation of Sparsity You have seen Symmetrix Hessian Graphs and Propagation Outlook Easy to exploit Partial Separability Possible to detectpartial Separability Automatic deallocation reordering (vectors,..) Keep non differential vertices to get generalized derivatives ADTAGEO ALLEGRO : optimize for speed

39 Thank you Thank you very much!