Exploiting Fill-in and Fill-out in Gaussian-like Elimination Procedures on the Extended Jacobian Matrix

Transcription

1 2nd European Workshop on AD 1 Exploiting Fill-in and Fill-out in Gaussian-like Elimination Procedures on the Extended Jacobian Matrix Andrew Lyons (Vanderbilt U.) / Uwe Naumann (RWTH Aachen)

2 2nd European Workshop on AD 2 Outline 1. Elementary Row/Column Operations in E 2. Sparse Gaussian Elimination 3. Algorithms for exploiting fill

3 2nd European Workshop on AD 3 Motivation Given function y = F(x), F : R n R m as evaluation procedure v i = x i, v i = ϕ i (v h ) h i, y i = v n+p+i, i = 1,...,n i = n + 1,...,n + p + m i = 1,...,m where l k v l is an argument of ϕ k. Want Jacobian matrix F = F (x) ( ) j=1,...,m yj (x) x i i=1,...,n R m n

4 2nd European Workshop on AD 4 The Extended Jacobian Extended system, from evaluation procedure: 0 = E(x;v) = (ϕ i (u h ) v i ) h i i=1,...,n+p+m Differentiate with respect to v = (v i ) i=n+p+m to get: E = E (x;v) (c i,j ) j=1,...,(n+p+m) i=1,...,(n+p+m) I R(n+p+m) (n+p+m)

5 2nd European Workshop on AD 5 Example y 1 = x 1 x 2 x 2 3, y 2 = x 2 1 x 2 x 3, y 3 = x 1 x 2 2 x 3 v 1 = x 1 ; v 2 = x 2 ; v 3 = x 3 v 4 = v 1 v 3 ; v 5 = v 1 v 2 ; v 6 = v 2 v 3 v 7 = v 4 v 6 ; v 8 = v 4 v 5 ; v 9 = v 5 v 6 y 1 = v 7 ; y 2 = v 8 ; y 3 = v 9

6 2nd European Workshop on AD 6 Example continued c 4,1 0 c 4, c 5,1 c 5, c 6,2 c 6, c 7,4 0 c 7, c 8,4 c 8, c 9,5 c 9, v 8 v 9 c 8,5 c 8,4 c 9,5 v 4 c 9,6 v 7 c 7,4 c 7,6 v 5 v 6 c 5,1 c 4,1 c 5,2 c 6,2 c 4,3 c 6,3 v 1 v 2 v 3

7 2nd European Workshop on AD 7 Elementary Row and Column Operations Correspond to back- and front-eliminations of edges in the Linearized computational graph: V = (v i ) i=1,...,n+p+m (v i, v j ) E i j Rows contain in-edges, columns contain out-edges eliminating all in- or out-edges equates to vertex elimination not the most general elimination technique - see face elimination

8 2nd European Workshop on AD 8 Elementary Row and Column Operations v 8 v 9 v 7 v 5 v 4 v 6 v 1 v 2 v 3

9 2nd European Workshop on AD 9 Elementary Row and Column Operations v 8 v 9 v 7 v 4 v 6 v 1 v 2 v 3

10 2nd European Workshop on AD 10 Digression/Review Sparse Gaussian Elimination Part I - Symbolic Fill Prediction Compressed row storage (CRS): a 0 b 0 α : a b c d e f g h c d 0 e κ : f 0 g 0 0 h ρ : Problem: Need to allocate memory for fill-in that occurs in L + U during the elimination process Solution: use a graph model to symbolically predict where such fill-in will occur, and allocate space for it in CRS scheme.

11 2nd European Workshop on AD 11 Sparse Gaussian Elimination Part I - Symbolic Fill Prediction directed graph captures structure under symmetric row/column permutations Finding a pivot sequence that minimizes fill is NP-hard (Rose/Tarjan 78, corrected later by Gilbert) for jacobians, emphasis has been on ops: elimination of a vertex costs num. predecessors * num. successors ops minimization for Jacobian accumulation is NP-hard (Naumann2005) there is a lot of theory behind perfect elimination graphs with respect to fill, but there are no perfect elimination computational graphs with respect to ops

12 2nd European Workshop on AD 12 Sparse Gaussian Elimination Part II - Numerical Phase Symbolic phase works for all matrices with same sparsity pattern. searching through indices is a large part of the cost?

13 2nd European Workshop on AD 13 Sparse Gaussian Elimination and Jacobian Accumulation They re Similar vertex elimination E is analogous to A This relationship is hardly new, research draws from the large body of research in sparse linear systems Griewank/Reese 91 - Markowitz heuristic Pryce introduced a scheme for crout-doolittle A = LU factorization.

14 2nd European Workshop on AD 14 Sparse Gaussian Elimination and Jacobian Accumulation They re Different 4 mults, 3 fill-in lower triangular E means the computational graph is acyclic edge elimination sequences terminate don t eliminate all vertices (not a big deal) fill-out (potential for exploitation)

15 2nd European Workshop on AD 15 Technique 1 v k v l v i v j v k v l v k v l v i v j v k v l Back elimination of (k,l)

16 2nd European Workshop on AD 16 Technique 2 v k v l v h v i v j v k v l 1... v k v l v h v i v j v k v l 1...

17 2nd European Workshop on AD 17 Maximum Immediate Successor Enumeration We have extra freedom: is a partial order. we can symmetrically permute E in order to get more nonzero elements one off the diagonal. Problem: Find a topological sort of G that maximizes the number of edges from v i to v i+1

18 2nd European Workshop on AD 18 Maximum Immediate Successor Enumeration v 6 v 3 v 5 v 7 v 1 v 2 v 8 v 9 v 4

19 2nd European Workshop on AD 19 MISE is NP-complete Reduction from covering by bicliques (CCBS): CCBS is NP-complete for bipartite graphs. Every immediate successor corresponds to exactly one biclique. MISE is NP-complete

20 2nd European Workshop on AD 20 To do: support for general edge elimination sequences (heuristics?) speed up fill prediction