Math 59 Lecture 2 (Tue Mar 5) Gaussian elimination and LU factorization (II) 2 Gaussian elimination - LU factorization For a general n n matrix A the Gaussian elimination produces an LU factorization if the pivots are nonzero: a a 2 a n L M Mn m 2 U a () An 22 a () 2n m n m n2 a (n ) so that A LU The matrix L is a unit lower triangular matrix and a a (2) 22 a (n ) are pivots The elements m ik are the multipliers The LU factorization can be summarized by the following algorithm (in Matlab m-file form) Algorithm 2 LU factorization with Gaussian Elimination Given A R n n this algorithm computes a unit lower triangular matrix L and an upper triangular matrix U such that A LU The entries of U are stored in the upper triangular part of A and the lower triangular entries are stored the lower triangular part of A (without storing the ones on the diagonal) LU gausslu(a) for k:n- for ik+:n A(ik) A(ik)/A(kk) for jk+:n A(ij) A(ij) - A(ik)A(kj) If we use matrix-vector operations we may use a single loop
2 Math 59 LU gausslu(a) for k:n- A(k+:nk)A(k+:nk)/A(kk) A(k+:nk+:n)A(k+:nk+:n)-A(k+:nk)A(kk+:n) For each k it needs n k divisions (n k) 2 multiplications and (n k) 2 subtractions - 2(n k) 2 + (n k) flops Totally k n it needs 2(n ) 2 + (n ) + 2(n 2) 2 + (n 2) + + 2 + 2 + 2 2 + + (n ) 2 2n3 3 For an m n matrix A and an integer k min{m n} the matrix a a k A k a k a flops formed by taking the entries from the leading k rows and columns of A is the kth leading principal submatrix of A Theorem 2 Existence and Uniqueness of LU factorization If all the leading principal submatrices A k of A R n n are invertible then the Gaussian elimination process can complete and A has a unique LU factorization Proof We show the completion of the Gaussian elimination process by induction Because a the first leading principal submatrix is nonzero we can determine the elementary lower triangular matrix M such that a a 2 a n a () 22 a () 2n M A a () n2 a () Since M m 2 m n A m 2 m n a a 2 a n a () 22 a () 2n a () n2 a ()
Math 59 3 Use the block form the second leading principal submatrix A 2 is 0 a a 2 A 2 m 2 0 a () 22 Because 0 det A 2 det m 2 a a 2 det 0 a () 22 a a () 22 and det A 2 0 it must be a () 22 0 So we can determine M 2 and perform another step of Gaussian elimination Suppose we have performed the Gaussian elimination k steps and we have U k u k u n A M M u (k ) u (k ) kn k u (k ) nk u (k ) where Note Then from we have U k a a k a (k 2) k k u k a k a (k ) k k u n a n a (k ) k n M M k m k Lk 0 m k m kk I m n m nk Because det A k 0 and det A k det L k det U k u k u n Lk 0 u (k ) u (k ) kn A I u (k ) nk u (k ) A k L k Uk Uk u k u (k ) u k u (k ) Uk u k det u (k ) a a () 22 u (k )
4 Math 59 we have u (k ) 0 So we can perform the kth step of the Gaussian elimination By induction the Gaussian elimination can be completed and we eventually have A LU For uniqueness suppose A L U L 2 U 2 Since A is invertible so are U U 2 Note L L 2 are always invertible Then from L U L 2 U 2 L 2 L U 2 U The matrix L 2 L is still unit lower triangular and U 2 U is still upper triangular So L 2 L U U 2 I L L 2 U U 2 Note it is not necessary to require that A is invertible The condition in Theorem 2 can be relaxed to all the leading principal submatrices A 0 A n are invertible In this case if A is singular then u 0 but the LU factorization is still unique 22 Numerical instability of LU factorization The Gaussian elimination or the LU factorization process may break down when a (k ) 0 for some k n This is rare in practice However quite often a (k ) can be very small In this case the computed LU factorization may not be accurate 0 Example 2 Let A In exact arithmetic 0 0 M 0 4 U A () 0 0 4 So for L M A LU 0 0 4 0 0 0 4 Suppose we compute an LU factorization in a finite floating point decimal system with t 3 Then µ 2 0 2 0005 The (22) entry of U will be The computed L and U will be 0 ˆL 0 4 fl( 0 4 ) fl ( (0000 )0 4) 0 4 0 Û 0 0 4 Then L ˆL 0 U Û 0 0 0
Math 59 5 and A ˆLÛ 0 0 0 0 0 0 showing that the compute LU factorization has a big backward error Note that A is wellconditioned (κ (A) A A 4) Such a big backward error is not acceptable The cause of this problem is that a big number (0 4 ) is created during the process which introduces big error in our results In general such a situation may happen We try to control the magnitudes of the entries involved in the whole elimination process so that the error can be reduced In the above example if we perform the Gaussian elimination on the matrix P A 0 the exact LU factorization of P A is P A LU L 0 0 with P 0 0 U 0 0 and in the same finite floating point number system (t 3) the computed L and U would be 0 L 0 Ũ 0 Then and L L 0 U Ũ 0 0 0 0 P A LŨ 0 0 0 0 This idea leads to a modified Gaussian elimination called Gaussian elimination with partial pivoting 23 Permutation matrices Definition 2 A matrix P R n n is a permutation if P e T α e T α 2 e T α n where (α α 2 α n ) is a permutation of ( 2 n)
6 Math 59 Let i < j n The elementary permutation determined by i j is P ij e T e T i e T j e T i+ e T j e T i e T j+ e T n 0 0 e e i e j e i+ e j e i e j+ e n By convention P ii I for any i A general permutation can be expressed as a product of elementary permutations This is because a permutation (α α 2 α n ) can be generated from ( 2 n) by interchanging a pair of numbers once a time Proposition 22 If P is a permutation then P P T I If P ij is an elementary permutation then P ij P T ij Example 22 For n 3 all permutations of ( 2 3) are and P ij P T ij P 2 ij I ( 2 3) ( 3 2) (2 3) (2 3 ) (3 2) (3 2 ) The corresponding permutation matrices are 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Among them the 2nd 3rd and the 6th are elementary permutations and 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ( 2 3) (2 3) (2 3 )
Math 59 7 a T For a given matrix A 2 a T Rn p and B b b n R m n a T n P ij A a T a T i a T j a T j a T ị a T n BP ij b b i b j b i+ b j b i b j+ b n 2 0 0 Example 23 Let A 2 3 and P 2 0 0 Then 0 4 5 0 0 P 2 A 2 3 2 0 4 5 For convenience for any i n P ii I n AP 2 2 3 2 2 4 0 5