Fundamentals Algorithms for Linear Equation Solution. Gaussian Elimination LU Factorization

Transcription

1 Fundamentals Algorithms for Linear Equation Solution Gaussian Elimination LU Factorization J. Roychowdhury, University of alifornia at erkeley Slide

2 Dense vs Sparse Matrices ircuit Jacobians: typically 3N-4N non-zeros compare against N2 for dense J. Roychowdhury, University of alifornia at erkeley Slide 2

3 Why Sparsity / O(n) Matters Matri storage for dense: N (N = size of matri = no. of rows/cols) N=, dense = billion nonzeros (8G storage) sparse: 4N = 4 nonzeros (3.2M storage) 2 25, less omputation how to eploit sparsity of A in solving A=b? lower bound: O(N) i.e.: at the least, need to look at each non-zero core concepts: Gaussian Elimination, LU factorization J. Roychowdhury, University of alifornia at erkeley Slide 3

4 Why are Matrices Sparse? Stems from local connectivity of graph each node: connected only to a few other nodes Incidence Matri: sparse Some eceptions power (gnd) nodes substrate networks etracted networks parasitic caps J. Roychowdhury, University of alifornia at erkeley i2 i3 i4 i5 {z A i6 i7 i8 i i9 i2 i3 i4 ~ i5 = : i6 A A } i 9 {z } ~ib Slide 4

5 Solution via omputing the Inverse ~ = J ~g (~ ), with J sparse matri sparse, but inverse dense still N2 storage computation even worse: N3! J. Roychowdhury, University of alifornia at erkeley Slide 5

6 Gaussian Elimination: Eample The A = b problem: z 2 A 2 2:5 :5 } :5 :5 ~ ~b { z } { z } { = 5 A 5 Step : divide first eqn. by diagonal element 2:5 :5 :5 :5 J. Roychowdhury, University of alifornia at erkeley : = 5 A 5 Slide 6

7 Gaussian Elim. Eample (contd.) From Step : 2:5 :5 :5 :5 Step 2: zero column : = 5 A 5 by subtracting eqn., scaled, from every other :5 :5 :5 :5 J. Roychowdhury, University of alifornia at erkeley : = 5 Slide 7

8 Gaussian Elim. Eample (contd.) From Step :5 :5 :5 :5 : = 5 Step 3: normalize eqn. 2 by dividing by diagonal element of row :5 :5 J. Roychowdhury, University of alifornia at erkeley : = 5 Slide 8

9 Gaussian Elim. Eample (contd.) From Step :5 :5 : = 5 Step 4: zero column 2 by subtracting eqn. 2, scaled, from every other J. Roychowdhury, University of alifornia at erkeley : = :5A A Slide 9

10 Gaussian Elim. Eample (contd.) From Step : = :5A A Step 5: eqn. 3: zero pivot, cannot normalize solution: swap eqns 3 and 4, then J. Roychowdhury, University of alifornia at erkeley : = 2 A A Slide

11 Gaussian Elim. Eample (contd.) From Step Step 6: zero column J. Roychowdhury, University of alifornia at erkeley 2 6: = 2 A A :5 3 = 2 A A 4 5 4:5 Slide

12 Gaussian Elim. Eample (contd.) From Step :5 3 = 2 A A 4 5 4:5 Steps 7, 8: normalize eqn. 4, zero col. J. Roychowdhury, University of alifornia at erkeley :5 3 = 2 A A 4 5 6:5 Slide 2

13 Gaussian Elim. Eample (contd.) From Step :5 3 = 2 A A 4 5 6:5 Steps 9, : normalize eqn. 5, zero col. J. Roychowdhury, University of alifornia at erkeley :25 2 :75 3 = 6:25A A 4 5 8:25 Solution Slide 3

14 Gaussian Elim. to LU Factorization Summary of Gaussian Elimination For i = :N normalize equation (row + RHS) by diagonal pivot zero all entries of column i (ecept diagonal, = ) (swap row/col if necessary first, to ensure non-zero pivot) by subtracting scaled versions of the ith equation In practice: LU Factorization tweak of Gaussian Elimination strategy zero only lower columns GE with tweak is equivalent to factoring A: A = LU, with L/U structured: lower/upper triangular LU factorization: 3-step strategy for solving A=b factor A = LU ( difficult ) forward and back substitute ( easy ): Ly = b, U = y key features can factor A only once and re-use with multiple RHS vectors crucial fact: L and U often retain sparsity J. Roychowdhury, University of alifornia at erkeley Slide 4

15 LU vs Inversion for Sparse Matrices J. Roychowdhury, University of alifornia at erkeley Slide 5

16 Forward Substitution Forward Substitution: solve Ly=b for y L= Key: solve successively for y, y2,..., yn y = b =l y2 = (b2 l2 y )=l22 y3 = (b3 l3 y l32 y2 )=l33 yn = (bn ln y ln 2 y2 l(n;n ) yn )=lnn J. Roychowdhury, University of alifornia at erkeley Slide 6

17 ack Substitution ack Substitution: solve U=y for U = Solve successively for N,,2, N = yn =un N N = (yn u(n ;N) N )=u(n ;N ) = (y u2 2 u3 3 u N N )=u J. Roychowdhury, University of alifornia at erkeley Slide 7

18 rout LU Factorization: 33 Eample z a3 z a3 l a23 A a33 l3 L } l22 l32 l33 { z A@ U } u2 { u3 u23 A a2 = l u2 a3 = l u3 a2 = l2 a22 = l2 u2 + l22 a23 = l2 u3 + l22 u23 a3 = l3 a32 = l3 u2 + l32 a33 = l3 u3 + l32 u23 + l33 Go in submatri order from top left first col of L; first row of U; second col of L; Doolittle: diagonal s are in L, not U { a = l A } a2 a22 a32 uii are unknowns, lii =. Factorize in place : save memory note: aij used only once, to compute lij or uij overwrite elements of A as L/U elements are computed J. Roychowdhury, University of alifornia at erkeley Slide 8

19

20

21

22

23 Re-ordering for Sparsity (not just for handling zero pivots) re-order matri: make it sparse towards top left A L=U re-order: swap rows and 4, cols and 4 Ar Lr =Ur A J. Roychowdhury, University of alifornia at erkeley A A % fill-in No fill-in Slide 23

24 Fill-in Reduction via Re-Ordering J. Roychowdhury, University of alifornia at erkeley Slide 24

25 Sparse Matri Technology Heuristics are key achieve balance between fill-in vs error growth control cost of re-ordering error growth == stability ~= zero pivot issues best reordering for minimum fill-in: NP-complete problem i.e., simply re-ordering is much worse than dense LU! Structural vs Numerical factorization structural : no numerics involved numerical compute a good re-ordering (using heuristics) find locations of fill-ins; pre-allocate memory, set up pointers run the LU algorithm (numerical calculations) structural often more epensive than numerical J. Roychowdhury, University of alifornia at erkeley Slide 25

26 Sparse Matri Packages SPIE3's SMP: hard-coded () current-controlled equations at end) Kundert's SPARSE v.3 () partial/full pivoting options (internally) well-structured, modular HSL/NAG: MA28, ZA28, MA58, (Fortran) long-time industry staple Demmel group: superlu () superblock (small dense-ish) matri operations, leveraging LAS, cache utilization advanced heuristics (a la iterative methods) superlu MT/DIST variants for parallel machines Tim Davis (U. Florida): UMFPAK (++) MATLA's lu(...), A\b for A sparse Dense matri packages LINPAK (Fortran) LAPAK (Fortran), -LAPAK (), LAPAK++ (++) MATLA's inv(a), A\b for A dense J. Roychowdhury, University of alifornia at erkeley Slide 26