Direct Methods for Sparse Linear Systems: MATLAB sparse backslash

Transcription

1 SIAM 2006 p. 1/100 Spiecal Thanks to: Special thanks to: Direct Methods for Sparse Linear Systems: MATLAB sparse backslash Tim Davis University of Florida

2 SIAM 2006 p. 4/100 Sparse matrices arise in... computational fluid dynamics, finite-element methods, statistics, time/frequency domain circuit simulation, dynamic and static modeling of chemical processes, cryptography, magneto-hydrodynamics, electrical power systems, differential equations, quantum mechanics, structural mechanics (buildings, ships, aircraft, human body parts...), heat transfer, MRI reconstructions, vibroacoustics, linear and non-linear optimization, financial portfolios, semiconductor process simulation, economic modeling, oil reservoir modeling, astrophysics, crack propagation, Google page rank, 3D computer vision, cell phone tower placement, tomography, multibody simulation, model reduction, nano-technology, acoustic radiation, density functional theory, quadratic assignment, elastic properties of crystals, natural language processing, DNA electrophoresis,...

3 For problems this important, I can't resist to ask: Can you solve Ax=b faster?

4 SIAM 2006 p. 15/100 Sparse data structures compressed sparse column format column j is Ai[Ap[j]... Ap[j+1]-1], ditto in Ax Thus, A(:,j) is easy in MATLAB; A(i,:) hard A = Ap: [0, 3, 6, 8, 10] Ai: [0, 1, 3, 1, 2, 3, 0, 2, 1, 3 ] Ax: [4.5,3.1,3.5,2.9,1.7,0.4,3.2,3.0,0.9,1.0]

5 SIAM 2006 p. 21/100 Sparse lower triangular solve, x=l\b x = b for j = 1:n if (x(j) 0) x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) end end

6 SIAM 2006 p. 22/100 Sparse lower triangular solve, x=l\b x = b for j = 1:n if (x(j) 0) x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) end end O(n+flops) time too high the problem: for j=1:n if (x(j) 0) need pattern of x before computing it

7 SIAM 2006 p. 23/100 Sparse lower triangular solve, x=l\b x = b for j = 1:n if (x(j) 0) x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) end end b i 0 x i 0 x j l ij x i L

8 SIAM 2006 p. 24/100 Sparse lower triangular solve, x=l\b x = b for j = 1:n if (x(j) 0) x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) end end b i 0 x i 0 x j 0 l ij 0 x i 0 x j l ij x i L

9 SIAM 2006 p. 25/100 Sparse lower triangular solve, x=l\b x = b for j = 1:n if (x(j) 0) x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) end end b i 0 x i 0 x j 0 l ij 0 x i 0 let G(L) have an edge j i if l ij 0 l ij x j x i L

10 SIAM 2006 p. 26/100 Sparse lower triangular solve, x=l\b x = b for j = 1:n if (x(j) 0) x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) end end b i 0 x i 0 x j x j 0 l ij 0 x i 0 let G(L) have an edge j i if l ij 0 let B = {i b i 0} and X = {i x i 0} L l ij x i

11 SIAM 2006 p. 27/100 Sparse lower triangular solve, x=l\b x = b for j = 1:n if (x(j) 0) x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) end end b i 0 x i 0 x j x j 0 l ij 0 x i 0 let G(L) have an edge j i if l ij 0 let B = {i b i 0} and X = {i x i 0} L l ij x i then X = Reach G(L) (B)

12 SIAM 2006 p. 28/100 Sparse lower triangular solve, x=l\b Lower triangular matrix L Graph G L

13 SIAM 2006 p. 29/100 Sparse lower triangular solve, x=l\b Lower triangular matrix L Graph G L If B = {4}

14 SIAM 2006 p. 30/100 Sparse lower triangular solve, x=l\b Lower triangular matrix L Graph G L If B = {4} then X = {4, 9, 12, 13, 14}

15 SIAM 2006 p. 31/100 Sparse lower triangular solve, x=l\b Lower triangular matrix L Graph G L If B = {4, 6} then X = {6, 10, 11, 4, 9, 12, 13, 14}

16 SIAM 2006 p. 32/100 Sparse lower triangular solve, x=l\b function x = lsolve(l,b) x = b for j = 1:n if (x(j) 0) x(j+1:n) = x(j+1:n) - L(j+1:n,j)*x(j) Time: O(n + flops), need X to get O(flops)

17 SIAM 2006 p. 33/100 Sparse lower triangular solve, x=l\b function x = lsolve(l,b) X = Reach(L, B) x = b for each j in X x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j)

18 SIAM 2006 p. 34/100 Sparse lower triangular solve, x=l\b function x = lsolve(l,b) X = Reach(L, B) x = b for each j in X x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) function X = Reach(L, B) for each i in B do if (node i is unmarked) dfs(i) function dfs(j) mark node j for each i in L j do if (node i is unmarked) dfs(i) push j onto stack for X

19 SIAM 2006 p. 35/100 Sparse lower triangular solve, x=l\b function x = lsolve(l,b) X = Reach(L, B) x = b for each j in X x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) function X = Reach(L, B) for each i in B do if (node i is unmarked) dfs(i) function dfs(j) Total time: O(flops) mark node j for each i in L j do if (node i is unmarked) dfs(i) push j onto stack for X

20 SIAM 2006 p. 36/100 Sparse lower triangular solve, x=l\b function x = lsolve(l,b) X = Reach(L, B) x = b for each j in X x(j+1:n) = x(j+1:n) - L(j+1:n,j) * x(j) function X = Reach(L, B) for each i in B do if (node i is unmarked) dfs(i) function dfs(j) which can be less than n mark node j for each i in L j do if (node i is unmarked) dfs(i) push j onto stack for X

21 SIAM 2006 p. 41/100 Sparse Cholesky, LL T = A [ ] [ ] [ ] L 11 l T 12 l 22 L T 11 l 12 l 22 = A 11 a 12 a T 12 a 22

22 SIAM 2006 p. 42/100 Sparse Cholesky, LL T = A [ ] [ ] [ ] L 11 l T 12 l 22 L T 11 l 12 l 22 = A 11 a 12 a T 12 a factorize L 11 L T 11 = A 11

23 SIAM 2006 p. 43/100 Sparse Cholesky, LL T = A [ ] [ ] [ ] L 11 l T 12 l 22 L T 11 l 12 l 22 = A 11 a 12 a T 12 a factorize L 11 L T 11 = A solve L 11 l 12 = a 12 for l 12

24 SIAM 2006 p. 44/100 Sparse Cholesky, LL T = A [ ] [ ] [ ] L 11 l T 12 l 22 L T 11 l 12 l 22 = A 11 a 12 a T 12 a factorize L 11 L T 11 = A solve L 11 l 12 = a 12 for l l 22 = a 22 l T 12 l 12

25 SIAM 2006 p. 45/100 Sparse Cholesky, LL T = A [ ] [ ] [ ] L 11 l T 12 l 22 L T 11 l 12 l 22 = A 11 a 12 a T 12 a factorize L 11 L T 11 = A solve L 11 l 12 = a 12 for l l 22 = a 22 l12 T l 12 for k = 1 to n solve L 11 l 12 = a 12 for l 12 l 22 = a 22 l12 T l 12

26 SIAM 2006 p. 46/100 Sparse Cholesky, LL T = A [ ] [ ] [ ] L 11 l T 12 l 22 L T 11 l 12 l 22 = A 11 a 12 a T 12 a factorize L 11 L T 11 = A solve L 11 l 12 = a 12 for l l 22 = a 22 l12 T l 12 for k = 1 to n solve L 11 l 12 = a 12 for l 12 l 22 = a 22 l12 T l 12 an up-looking method accessed compute kth row not accessed

27 SIAM 2006 p. 47/100 Sparse Cholesky: etree elimination tree arises in many direct methods Compute nonzero pattern of x=l\b for a Cholesky L in time O( x ), the number of nonzeros in x...

28 SIAM 2006 p. 48/100 Sparse Cholesky: etree Elimination tree T : pruning the graph of L. Consider computing kth row of L: accessed computed not accessed x k

29 SIAM 2006 p. 49/100 Sparse Cholesky: etree Elimination tree T : pruning the graph of L. Consider computing kth row of L: l ki 0 x i 0 x i l ki x k

30 SIAM 2006 p. 50/100 Sparse Cholesky: etree Elimination tree T : pruning the graph of L. Consider computing kth row of L: l ji x i x j l ki 0 x i 0 (l ji 0 and x i 0) x j 0 l ki x k

31 SIAM 2006 p. 51/100 Sparse Cholesky: etree Elimination tree T : pruning the graph of L. Consider computing kth row of L: l ji x i x j l ki 0 x i 0 (l ji 0 and x i 0) x j 0 l kj 0 x j 0 l ki l kj x k

32 SIAM 2006 p. 52/100 Sparse Cholesky: etree Elimination tree T : pruning the graph of L. Consider computing kth row of L: l ki 0 x i 0 x i (l ji 0 and x i 0) x j 0 l ji x j l kj 0 x j 0 l ki l kj x k Thus, l ki redundant for X = Reach(B).

33 SIAM 2006 p. 53/100 Sparse Cholesky: etree Elimination tree T : pruning the graph of L. Consider computing kth row of L: l ki 0 x i 0 x i (l ji 0 and x i 0) x j 0 l ji x j l kj 0 x j 0 l ki l kj x k Thus, l ki redundant for X = Reach(b). parent(i) = min{j > i l ji 0}; other edges redundant

34 SIAM 2006 p. 54/100 Sparse Cholesky: etree Elimination tree T : pruning the graph of L. Consider computing kth row of L: l ki 0 x i 0 x i (l ji 0 and x i 0) x j 0 l ji x j l kj 0 x j 0 l ki l kj x k Thus, l ki redundant for X = Reach(b). parent(i) = min{j > i l ji 0}; other edges redundant L k = Reach(A 1:k,k ) in O( L k ) time

35 SIAM 2006 p. 55/100 Sparse Cholesky: etree A

36 SIAM 2006 p. 56/100 Sparse Cholesky: etree A Cholesky factor L of A

37 SIAM 2006 p. 57/100 Sparse Cholesky: etree A Cholesky factor L of A elimination tree Can read off zero patterns of L by zero patterns of A + etree. Problem: Why we can compute zero patterns of L in O~(n^2) time, but not L itself?

38 SIAM 2006 p. 66/100 Sparse Cholesky: overview Symbolic analysis: fill-reducing ordering, Ā = PAP T = LL T etree of Ā: nearly O( A ) depth-first postordering of etree: O(n) column counts of L: nearly O( A ) some methods find L: O( L ) or less Numeric factorization: up-looking left-looking, supernodal Postorder (Left, Right, Root)

39 SIAM 2006 p. 68/100 Sparse Cholesky: left-looking access L(k:n,1:k-1) compute kth column for k = 1 to n x = A(k:n,k) for each j in Reach(L,A(1 : k,k)) x(k:n) = x(k:n) - L(k:n,j) * L(k,j) L(k,k) = sqrt(x(k)) L(k+1:n,k) = x(k) / L(k,k)

40 SIAM 2006 p. 69/100 Sparse Cholesky: left-looking L k = Reach(L,A(1 : k,k)) for k = 1 to n x = A(k:n,k) for each j in Reach(L,A(1 : k,k))

41 SIAM 2006 p. 70/100 Sparse Cholesky: left-looking for k = 1 to n x = A(k:n,k) for each j in Reach(L,A(1 : k,k)) x(k:n) = x(k:n) - L(k:n,j) * L(k,j)......

42 SIAM 2006 p. 71/100 Sparse Cholesky: left-looking for k = 1 to n x = A(k:n,k) for each j in Reach(L,A(1 : k,k)) x(k:n) = x(k:n) - L(k:n,j) * L(k,j) L(k,k) = sqrt(x(k)) L(k+1:n,k) = x(k) / L(k,k)

43 SIAM 2006 p. 75/100 Sparse Cholesky: supernodal column 4 etree: Adjacent columns of L often have identical pattern a chain in the elimination tree can exploit dense submatrix operations

44 SIAM 2006 p. 76/100 Sparse Cholesky: supernodal block left-looking k 1 k 2 for jth supernode: jth supernode, w = k 2 k 1 columns of L

45 SIAM 2006 p. 77/100 Sparse Cholesky: supernodal block left-looking for jth supernode: (1) sparse block matrix multiply

46 SIAM 2006 p. 78/100 Sparse Cholesky: supernodal block left-looking for jth supernode: (1) sparse block matrix multiply (2) dense Cholesky

47 SIAM 2006 p. 79/100 Sparse Cholesky: supernodal block left-looking for jth supernode: (1) sparse block matrix multiply (2) dense Cholesky (3) dense block Lx = b T solve