7. LU factorization. factor-solve method. LU factorization. solving Ax = b with A nonsingular. the inverse of a nonsingular matrix

Transcription

1 EE507 - Computational Techniques for EE 7. LU factorization Jitkomut Songsiri factor-solve method LU factorization solving Ax = b with A nonsingular the inverse of a nonsingular matrix LU factorization algorithm effect of rounding error sparse LU factorization 7-

2 Factor-solve approach to solve Ax = b, first write A as a product of simple matrices A = A A 2 A k then solve (A A 2 A k )x = b by solving k equations A z = b, A 2 z 2 = z,..., A k z k = z k 2, A k x = z k examples Cholesky factorization (for positive definite A) k = 2, A = LL T sparse Cholesky factorization (for sparse positive definite A) k = 4, A = PLL T P LU factorization 7-2

3 Complexity of factor-solve method #flops = f +s f is cost of factoring A as A = A A 2 A k (factorization step) s is cost of solving the k equations for z, z 2,... z k, x (solve step) usually f s example: positive definite equations using the Cholesky factorization f = (/3)n 3, s = 2n 2 LU factorization 7-3

4 Multiple right-hand sides two equations with the same matrix but different right-hand sides Ax = b, A x = b factor A once (f flops) solve with right-hand side b (s flops) solve with right-hand side b (s flops) cost: f +2s instead of 2(f +s) if we solve second equation from scratch conclusion: if f s, we can solve the two equations at the cost of one LU factorization 7-4

5 LU factorization LU factorization without pivoting A = LU L unit lower triangular, U upper triangular does not always exist (even if A is nonsingular) LU factorization (with row pivoting) A = PLU P permutation matrix, L unit lower triangular, U upper triangular exists if and only if A is nonsingular (see later) cost: (2/3)n 3 if A has order n LU factorization 7-5

6 Solving linear equations by LU factorization solve Ax = b with A nonsingular of order n factor-solve method using LU factorization. factor A as A = PLU ((2/3)n 3 flops) 2. solve (PLU)x = b in three steps permutation: z = P T b (0 flops) forward substitution: solve Lz 2 = z (n 2 flops) back substitution: solve Ux = z 2 (n 2 flops) total cost: (2/3)n 3 +2n 2 flops, or roughly (2/3)n 3 this is the standard method for solving Ax = b LU factorization 7-6

7 Multiple right-hand sides two equations with the same matrix A (nonsingular and n n): Ax = b, A x = b factor A once forward/back substitution to get x forward/back substitution to get x cost: (2/3)n 3 +4n 2 or roughly (2/3)n 3 exercise: propose an efficient method for solving Ax = b, A T x = b LU factorization 7-7

8 Inverse of a nonsingular matrix suppose A is nonsingular of order n, with LU factorization A = PLU inverse from LU factorization A = (PLU) = U L P T gives interpretation of solve step: evaluate in three steps x = A b = U L P T b z = P T b, z 2 = L z, x = U z 2 LU factorization 7-8

9 Computing the inverse solve AX = I by solving n equations AX = e, AX 2 = e 2,..., AX n = e n X i is the ith column of X; e i is the ith unit vector of size n one LU factorization of A: 2n 3 /3 flops n solve steps: 2n 3 flops total: (8/3)n 3 flops conclusion: do not solve Ax = b by multiplying A with b LU factorization 7-9

10 LU factorization without pivoting partition A, L, U as block matrices: A = a A 2, L = A 2 A 22 a and u are scalars 0, U = L 2 L 22 L 22 unit lower-triangular, U 22 upper triangular of order n u U 2 0 U 22 determine L and U from A = LU, i.e., a A 2 A 2 A 22 = = 0 u U 2 L 2 L 22 0 U 22 u U 2 u L 2 L 2 U 2 +L 22 U 22 LU factorization 7-0

11 recursive algorithm: determine first row of U and first column of L u = a, U 2 = A 2, L 2 = (/a )A 2 factor the (n ) (n )-matrix A 22 L 2 U 2 as A 22 L 2 U 2 = L 22 U 22 this is an LU factorization (without pivoting) of order n cost: (2/3)n 3 flops (no proof) LU factorization 7-

12 Example LU factorization (without pivoting) of A = write as A = LU with L unit lower triangular, U upper triangular A = = 0 0 l 2 0 l 3 l 32 u u 2 u 3 0 u 22 u u 33 LU factorization 7-2

13 first row of U, first column of L: = 0 0 / /4 l u 22 u u 33 second row of U, second column of L: 9 4 /2 2 9 = 7 9 3/4 8 /2 = /2 9/4 0 u22 u 23 l 32 0 u /2 /6 0 u 33 third row of U: u 33 = 9/4+/32 = 83/32 conclusion: A = = 0 0 /2 0 3/4 / / /32 LU factorization 7-3

14 Not every nonsingular A can be factored as A = LU A = = 0 0 l 2 0 l 3 l 32 u u 2 u 3 0 u 22 u u 33 first row of U, first column of L: = l u 22 u u 33 second row of U, second column of L: 0 2 = u 22 = 0, u 23 = 2, l 32 0 =? 0 l 32 u22 u 23 0 u 33 LU factorization 7-4

15 LU factorization (with row pivoting) if A is n n and nonsingular, then it can be factored as A = PLU P is a permutation matrix, L is unit lower triangular, U is upper triangular not unique; there may be several possible choices for P, L, U interpretation: permute the rows of A and factor P T A as P T A = LU also known as Gaussian elimination with partial pivoting (GEPP) cost: (2/3)n 3 flops LU factorization 7-5

16 Proof: by induction; show that if every nonsingular (n ) (n ) matrix has an LU factorization then the same is true for nonsingular n n-matrices if A is nonsingular, A cannot have an entirely zero column if a is zero, one can permute the rows of A such that à = P T A = ã à 2 à 2 à 22 where Ã22 has size (n ) (n ) and ã 0 the Schur complement of ã in à is à 22 ã à 2 à 2 and we know that it is nonsingular if à is nonsingular LU factorization 7-6

17 by assumption, this matrix can be factorized as à 22 ã à 2 à 2 = P 2 L 22 U 22 this provides the LU factorization of A: ã à A = P 2 à 2 à 22 0 ã à = P 2 0 P 2 P2 TÃ2 P2 TÃ22 0 ã à = P 2 0 P 2 P2 TÃ2 L 22 U 22 +(/ã )P2 T Ã2Ã2 0 0 ã à = P 2 0 P 2 (/ã )P2 TÃ2 L 22 0 U 22 LU factorization 7-7

18 so if we define 0 P = P, L = 0 P 2 0 (/ã )P2 T, U = Ã2 L 22 ã Ã 2 0 U 22 then P is permutation matrix, L is unit lower triangular, U is upper triangular and A = PLU LU factorization 7-8

19 Example = / / /3 8/ /9 the factorization is not unique; the same matrix can be factored as = / LU factorization 7-9

20 Effect of rounding error 0 5 x x 2 = 0 exact solution: x = 0 5, x 2 = 0 5 let us solve the equations using LU factorization, rounding intermediate results to 4 significant decimal digits we will do this for the two possible permutation matrices: P = 0 0 or P = 0 0 LU factorization 7-20

21 first choice of P: P = I (no pivoting) = L, U rounded to 4 decimal significant digits L = 0 5, U = forward substitution 0 z 0 5 z 2 = 0 = z =, z 2 = 0 5 back substitution x x 2 = 0 5 = x = 0, x 2 = error in x is 00% LU factorization 7-2

22 second choice of P: interchange rows = L, U rounded to 4 decimal significant digits 0 L = 0 5, U = 0 forward substitution z z 2 = 0 = z = 0, z 2 = backward substitution 0 x x 2 = 0 = x =, x 2 = error in x, x 2 is about 0 5 LU factorization 7-22

23 Sparse linear equations if A is sparse, it is usually factored as P and P 2 are permutation matrices A = P LUP 2 interpretation: permute rows and columns of A and factor à = PT AP T 2 à = LU choice of P and P 2 greatly affects the sparsity of L and U: many heuristic methods exist for selecting good permutations in practice: #flops (2/3)n 3 ; exact value is a complicated function of n, number of nonzero elements, sparsity pattern LU factorization 7-23

24 Conclusion different levels of understanding how linear equation solvers work: highest level: x = A\b costs (2/3)n 3 ; more efficient than x = inv(a)*b intermediate level: factorization step A = P LU followed by solve step lowest level: details of factorization A = P LU for most applications, level is sufficient in some situations (e.g., multiple right-hand sides) level 2 is useful level 3 is important only for experts who write numerical libraries LU factorization 7-24

25 References Lecture notes on LU Factorization, EE03, L. Vandenberhge, UCLA LU factorization 7-25