This can be accomplished by left matrix multiplication as follows: I

Transcription

1 1 Numerical Linear Algebra 11 The LU Factorization Recall from linear algebra that Gaussian elimination is a method for solving linear systems of the form Ax = b, where A R m n and bran(a) In this method one first forms the augmented system [A b] and then uses the three elementary row operations to put this system into row echelon form (or upper triangular form) A solution x is then obtained by back substitution, or back solving, starting with the component x n We now show how the process of bringing a matrix to upper triangular form can be performed by left matrix multiplication The key step in Gaussian elimination is to transform a vector of the form a α b where a R k, 0 α R, and b R n k 1, into one of the form a α 0 This can be accomplished by left matrix multiplication as follows: I k k 0 0 a α = a α 0 α 1 b I (n k 1) (n k 1) b 0 The matrix I k k α 1 b I (n k 1) (n k 1) is called a Gaussian elimination matrix This matrix is invertible with inverse I k k α 1 b I (n k 1) (n k 1) We now use this basic idea to show how a matrix can be put into upper triangular form Suppose a1 v1 T A = C n m, u 1 Ã 1 with 0 a 1 C, u 1 C m 1, v 1 C n 1, and Ã1 C (m 1) (n 1) Then using the first row to zero out u 1 amounts to left multiplication of the matrix A by the matrix 1 0, u 1 a 1 1 I

2 2 to get (*) where Define and observe that 1 0 a1 v1 T C I n m = u 1 Ã 1 u 1 a 1 u 1 a 1 A 1 = Ã1 u 1 v T 1 /a L 1 = C I m m and U 1 = [ = I u 1 a 1 ] a1 v1 T 0 A 1 a1 v1 T C 0 A m n 1 Hence (*) becomes 1 A = U 1, or equivalently, A = L 1 U 1 Note that L 1 is unit lower triangular (ones on the mail diagonal) and U 1 is block uppertriangular with one 1 1 block and one (m 1) (n 1) block on the block diagonal The multipliers are usually denoted u/a = [µ 21, µ 31,, µ m1 ] T If the (1, 1) entry of A 1 is not 0, we can apply the same procedure to A 1 : if a2 v2 T A 1 = C (m 1) (n 1) u 2 Ã 2 with a 2 0, letting and forming 2 A 1 = u 1 a 2 I 0 L 2 = C I (m 1) (m 1), u 2 a a2 v2 T = I u 2 Ã 1 a2 v2 T 0 A Ũ2 C (m 1) (n 1), 2 where A 2 C (m 2) (n 2) This process amounts to using the second row to zero out elements of the second column below the diagonal Setting 1 0 a v T L 2 = 0 L and U 2 =, 2 we have [ A = 0 L Ũ2 ] a v T = U 0 A 2, 1 or equivalently, A = L 2 L 1 U 2 Here U 2 is block upper triangular with two 1 1 blocks and one (m 2) (n 2) block on the diagonal, and again L 2 is unit lower triangular We can continue in this fashion at most m 1 times, where m = min{m, n},

3 3 If we can proceed m 1 times, then m 1 L A = U m 1 = U is upper triangular provided that along the way that the (1, 1) entries of A, A 1, A 2,, A m 2 are nonzero so the process can continue Define L = ( m 1 L 1 1 ) 1 = L 1 L 2 L m 1 The matrix L is square unit lower triangular, and so is invertable Moreover, A = LU, where the matrix U is the so called row echelon form of A In general, a matrix T C m n is said to be in row echelon form if for each i = 1,, m 1 the first non-zero entry in the (i + 1) st row lies to the right of the first non-zero row in the i th row Let us now suppose that m = n and A C n n is invertible Writing A = LU as a product of a unit lower triangular matrix L C n n (necessarily invertible) and an upper triangular matrix U C n n (also nessecarily invertible in this case) is called the LU factorization of A Remarks (1) If A C n n is invertible and has an LU factorization, it is unique (2) One can show that A C n n has an LU factorization iff for 1 j n, the upper left j j principal submatrix a 11 a ij a j1 a jj is invertible (3) Not every [ invertible ] A C n n has an LU-factorization 0 1 Example: 1 0 Typically, one must permute the rows of A to move nonzero entries to the appropriate spot for the elimination to proceed Recall that a permutation matrix P C n n is the identity I with its rows (or columns) permuted: so P R n n is orthogonal, and P 1 = P T Permuting the rows of A amounts to left multiplication by a permutation matrix P T ; then P T A has an LU factorization, so A = P LU (called the PLU factorization of A) (4) Fact: Every invertible A C n n has a (not necessarily unique) PLU factorization (5) The LU factorization can be used to solve linear systems Ax = b (where A = LU C n n is invertible) The system Ly = b can be solved by forward substitution (1 st equation gives x 1, etc), and Ux = y can be solved by back-substitution (n th equation gives x n, etc), giving the solution to? Ax = LUx = b

4 4 Example: We now use the procedure outlined above to compute the LU factorization of the matrix A = A = = We now have and L = L 1 L 2 = 2 1 A = = U = , = The Cholesky Factorization We now consider the application of the LU factorization to a symmetric positive definite matrix, but with a twist Suppose the n n matrix H is symmetric and we have performed the first step in the procedure for computing the LU factorization of H so that 1 H = U 1 Clearly, U 1 is no-longer symmetric (assuming L 1 is not the identity matrix) To recover symmetry we could multiply U 1 on the right by the upper triangular matrix L T 1 so that 1 HL T 1 = U 1 L T 1 = H 1 We claim that H 1 necessarily has the form [ h(1,1) 0 H 1 = 0 Ĥ 1 ],

5 where h (1,1) is the (1, 1) element of H and Ĥ1 is an (n 1) (n 1) symmetric matrix For example, consider the matrix H = In this case, we get 1 HL T 1 = = If we now continue this process with the added feature of multiplying on the right by L T j as we proceed, we obtain HL T = D, or equivalently, H = LDL T, where L is a unit lower triangular matrix and D is a diagonal matrix Note that the entries on the diagonal of D are not necessarily the eigenvalues of H since the transformation HL T is not a similarity transformation Observe that if it is further assumed that H is positive definite, then the diagonal entries of D are necessarity all positive and the factorization H = LDL T can always be obtained, ie no zero pivots can arise in computing the LU factorization (see exercises) Let us apply this approach by continuing the computation of the LU factorization for the matrix given above Thus far we have Next 1 HL T 1 = = HL T 1 L T 2 = = ,

6 6 giving the desired factorization H = LDL T = Note that this implies that the matrix H is not positive definite We make one final comment on the positive definite case When H is symmetric and positive definite, an LU factorization always exists and we can use it to obtain a factorization of the form H = LDL T, where L is unit lower triangular and D is diagonal with positive diagonal entries If D = diag(d 1, d 2,, d n ) with each d i > 0, the D 1/2 = diag( d 1, d 2,, d n ) Hence we can write H = LDL T = LD 1/2 D 1/2 L T = ˆLˆL T, where ˆL = LD 1/2 is a non-singular lower triangular matrix The factorization H = ˆLˆL T where ˆL is a non-singular lower triangula matrix is called the Cholesky factorization of H In this regard, the process of computing a Chlesky factorization is an effective means for determining is a symmetric matrix is positive definite 13 The QR Factorization Recall the Gram-Schmidt orthogonalization process for a sequence of linearly independent vectors a 1,, a n R n Define q 1,, q n inductively, as follows: set p 1 = a 1, q 1 = p 1 / p 1, j 1 p j = a j a j, q j q i for 2 j n, and i=1 q j = p j / p j For 1 j n, q j Span{a 1,, a j }, so each p j 0 by the lin indep of {a 1,, a n } Thus each q j is well-defined We have {q 1,, q n } is an orthonormal basis for Span{a 1,, a n } Also a k Span{q 1,, q k } 1 k n, so {q 1,, q k } is an orthonormal basis of Span{a 1,, a k } Define r jj = p j and r ij = a j, q i for 1 i < j n,

7 7 we have: a 1 = r 11 q 1, a 2 = r 12 q 1 + r 22 q 2, a 3 = r 13 q 1 + r 23 q 2 + r 33 q 3, a n = n r in q i i=1 Set A = [a 1 a 2 a n ], R = [r ij ], and Q = [q 1 q 2 q n ], where r ij = 0, i > j Then A = QR, where Q is unitary and R is upper triangular Remarks (1) If a 1, a 2, is a linearly independent sequence, apply Gram-Schmidt to obtain an orthonormal sequence q 1, q 2, such that {q 1,, q k } is an orthonormal basis for Span{a 1,, a k }, k 1 (2) If the a j s are linearly dependent, for some value(s) of k, a k Span{a 1,, a k 1 }, so p k = 0 The process can be modified by setting r kk = 0, not defining a new q k for this iteration and proceeding We end up with orthogonal q j s Then for k 1, the vectors {q 1,, q k } form an orthonormal basis for Span{a 1,, a l+k } where l is the number of r jj = 0 Again we obtain A = QR, but now Q may not be square (3) The classical Gram-Schmidt algorithm as described does not behave well computationally This is due to the accumulation of round-off error The computed q j s are not orthogonal: q j, q k is small for j k with j near k, but not so small for j k or j k An alternate version, Modified Gram-Schmidt, is equivalent in exact arithmetic, but behaves better numerically In the following pseudo-codes, p denotes a temporary storage vector used to accumulate the sums defining the p j s

8 8 Classic Gram-Schmidt Modified Gram-Schmidt For j = 1,, n do For j = 1,, n do p := a j p := aj For i = 1,, j 1 do For i = 1,, j 1 do r ij = a j, q i r ij = p, q i p := p r ij q i p := p r ij q i r jj := p r jj = p q j := p/r jj q j := p/r jj The only difference is in the computation of r ij : in Modified Gram-Schmidt, we orthogonalize the accumulated partial sum for p j against each q i successively Theorem 11 Suppose A R m n with m n Then for which unitary Q R m m upper triangular R R m n A = QR If Q R m n denotes the first n columns of Q and R R n n denotes the first n rows of R, then R A = QR = [ Q ] = Q R 0 Moreover (a) We may choose an R to have nonnegative diagonal entries (b) If A is of full rank, then we can choose R with positive diagonal entries, in which case the condensed factorization A = Q R is unique (and thus if m = n, the factorization A = QR is unique since then Q = Q and R = R) Proof If A has full rank, apply the Gram-Schmidt Define as above, so Q = [q 1,, q n ] R m n and R = [rij ] R n n A = Q R Extend {q 1,, q n } to an orthonormal basis {q 1,, q m } of R m, and set R Q = [q 1,, q m ] and R = C m n, so A = QR 0 As r jj > 0 in the G-S process, we have (b) Uniqueness follows by induction passing through the G-S process again, noting that at each step we have no choice Remarks

9 (1) In practice, there are more efficient and better computationally behaved ways of calculating the Q and R factors The idea is to create zeros below the diagonal (successively in columns 1, 2, ) as in Gaussian Elimination, except we now use Householder transformations (which are unitary) instead of the unit lower triangular matrices L j (2) A QR factorization is also possible when m < n A = Q[R 1 R 2 ], where Q C m m is unitary and R 1 C m m is upper triangular Every A R m n has a QR-factorization, even when m < n Indeed, if there always exist Q R m k R R k n rank(a) = k, and a permutation matrix P R n n such that ( ) with orthonormal columns, full rank upper triangular, AP = QR Moreover, if A has rank n (so m n), then R R n n is nonsingular On the other hand, if m < n, then R = [R 1 R 2 ], where R 1 R k k is nonsingular Finally, if A R m n, then the same facts hold, but now both Q and R can be chosen to be real matrices QR-Factorization and Orthogonal Projections Let A R m n have condensed QR-factorization A = Q R Then by construction the columns of Q form an orthonormal basis for the range of A Hence P = Q Q T is the orthogonal projector onto the range of A Similarly, if the condensed QR-factorization of A T is A T = Q 1 R1, then P 1 = Q 1 QT 1 is the orthogonal projector onto Ran(A T ) = ker(a), and so I Q 1 QT 1 is the orthogonal projector onto ker(a) The QR-factorization can be computed using either Givens rotations or Householder reflections Although, the approach via rotations is arguably more stable numerically, it is more difficult to describe so we only illustrate the approach using Householder reflections QR using Householder Reflections 9

10 10 Given w R n we can associate the matrix U = I 2 wwt w T w which reflects R n across the hyperplane Span{w} The matrix U is call the Householder reflection across this hyperplane Given a pair of vectors x and y with x 2 = y 2, and x y, there is a Householder reflection such that y = Ux: Proof (x y)(x y)t U = I 2 (x y) T (x y) x 2 y T x Ux = x 2(x y) x 2 2y T x + y 2 since x = y = x 2(x y) x 2 y T x 2( x 2 y T x) = y QR using Householder Reflections We now describe the basic deflation step in the QR-factorization α0 a A 0 = T 0 b 0 A 0 Set ν 0 = ( α0 b 0 ) 2 Let H 0 be the Householder transformation that maps ( ) α0 ν 0 e 1 : b T 0 H 0 = I 2 wwt w T w where w = ( α0 ) ν b 0 e 1 = 0 ( ) α0 ν 0 b 0 Thus, H 0 A = [ ν0 a T 1 0 A 1 ]

11 A problem occurs if ν 0 = 0 or ( ) α0 = 0 b 0 In this case, permute the offending column to the right bringing in the column of greatest magnitude Now repeat with A 1 If the above method if implemented by always permuting the column of greatest magnitude into the current pivot column, then AP = QR gives a QR-factorization with the diagonal entries of R nonnegative and listed in the order of descending magnitude 11