Lecture 13 Stability of LU Factorization; Cholesky Factorization. Songting Luo. Department of Mathematics Iowa State University

Transcription

1 Lecture 13 Stability of LU Factorization; Cholesky Factorization Songting Luo Department of Mathematics Iowa State University MATH 562 Numerical Analysis II ongting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 1 / 18

2 Outline 1 Stability of LU Factorization 2 Cholesky Factorization 3 Linear Algebra Software Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 2 / 18

3 Outline 1 Stability of LU Factorization 2 Cholesky Factorization 3 Linear Algebra Software Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 3 / 18

4 Stability of LU without Pivoting For A LU computed without pivoting LŨ A ` δa, }δa} }L}}U} Opɛ machineq This is close to backward stability, except that we have }L}}U} instead of }A} in the denominator Instability of Gaussian elimination can happen only if one or both of the factors L and U is large relative to size of A Unfortunately, }L} and }U} can be arbitrarily large (even for well-conditioned A), e.g., j j A Therefore, the algorithm is unstable Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 4 / 18 j

5 Stability of LU with Partial Pivoting With pivoting, all entries of L are in r 1, 1s, so }L} Op1q. To measure growth in U, we introduce the growth factor ρ max ij u ij max ij a ij, and hence }U} Opρ}A}qq We then have PA LU LŨ PA ` δa, }δa} }A} Opρɛ machineq If l ij ă 1 for each i ą j, (i.e., there is no tie for the pivoting), then P P for sufficiently small ɛ machine If ρ Op1q, then the algorithm is backward stable In fact, ρ ď 2 m 1, so by definition ρ is a constant but can be very large Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 5 / 18

6 The Growth Factor ρ can indeed be as large as 2 m 1. Consider matrix» fi» fi» ffi ffi ffi ffi fl ffi ffi ffi ffi fl where growth factor ρ 16 2 m ρ 2 m 1 is as large as ρ can get. It can be catastrophic in practice Theoretically, Gaussian elimination with partial pivoting is backward stable according to formal definition However, in the worst case, Gaussian elimination with partial pivoting may be unstable for practical values of m Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 6 / 18 fi ffi ffi ffi ffi fl

7 The Growth Factor in Practice Good news: Large ρ occurs only for very skewed matrices. Experimentally, one rarely see very large ρ Probability of large ρ decreases exponentially in ρ. If you pick a billion matrices at random, you will almost certainly not find one for which Gaussian elimination is unstable In practice, ρ is no larger than Op? mq. However, this behavior is not fully understood yet In conclusion, Gaussian elimination with partial pivoting is backward stable In theory, its error may grow exponentially in m In practice, it is stable for matrices of practical interests Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 7 / 18

8 Outline 1 Stability of LU Factorization 2 Cholesky Factorization 3 Linear Algebra Software Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 8 / 18

9 Hermitian Positive-Definite Matrices Symmetric matrix A P R mˆm is symmetric positive definite (SPD) if x T Ax ą 0 for x P R m zt0u Hermitian matrix A P C mˆm is Hermitian positive definite (HPD) if x Ax ą 0 for x P C m zt0u If A is m ˆ m HPD and X P C mˆm has full column rank, then X AX is HPD Any principal submatrix (picking some rows and corresponding columns) of A is HPD and a ii ą 0. HPD matrices have positive real eigenvalues and orthogonal eigenvectors Note: A positive-definite matrix does not need to be symmetric or Hermitian! A real matrix A is positive definite iff A ` A T is SPD. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 9 / 18

10 Cholesky Factorization Key idea: take advantage and preserve the properties of symmetry and positive-definiteness in factorization Eliminate below diagonal and to the right of diagonal j j j a11 w α 0 α w {α A w K w{α I 0 K ww {a 11 j j j α α w {α R 1A w{α I 0 K ww {a 11 0 I 1 R 1 where α? a 11, a 11 ą 0 K ww {a 11 is principal submatrix of HPD A 1 R therefore is HPD, with positive diagonal entries 1 AR 1 1 and Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 10 / 18

11 Cholesky Factorization Apply recursively to obtain A pr 1R 2 R mqpr m R 2 R 1 q R R, r jj ą 0 which is known as Cholesky factorization Question: Is R simply union of kth rows of R k (or R union of kth columns of R k )? Yes. Hint: Write R k in a form similar to L k I ` l k e k in LU. Existence and uniqueness: every HPD matrix has a unique Cholesky factorization Exists because algorithm for Cholesky factorization always works for HPD matrices Is unique since once α? a 11 is determined at each step, entire column w{α is determined Question: How to check whether a Hermitian matrix is positive definite? Answer: Run Cholesky factorization and it would succeeds iff the matrix is positive definite. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 11 / 18

12 Algorithm of Cholesky Factorization Factorize HPD matrix A Algorithm: Cholesky factorization R A for k 1 to m for j k ` 1 to m r j,j:m Ð r j,j:m r k,j:m r kj {r kk r k,k:m Ð r k,k:m {? r kk Operation count mÿ mÿ k 1 j k`1 2pm jq 2 mÿ k 1 j 1 kÿ j mÿ k 2 m 3 {3 Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 12 / 18 k 1

13 Stability Theorem The computed Cholesky factor R satisfies R R A ` δa, i.e., Cholesky factorization is backward stable }δa} }A} Opɛ machineq, Forward errors in R is } R R}{}R} OpκpAqɛ machine q, which may be large for ill-conditioned A. Solve Ax b for positive definite A Factorize A R R, solve R y b, solve Rx y Operation count is m 3 {3 Algorithm is backward stable: pa ` Aq x b, } A} }A} Opɛ machineq Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 13 / 18

14 LDL Factorization Cholesky factorization is sometimes given by A LDL where D is diagonal matrix and L is unit lower triangular matrix It avoids computing square roots Analogously, LU factorization can also be written as LDU, where U is unit upper triangular Question: How is R in A R R related to the L and U factors of A LU? U DL? DR, where? D diagp? d 11,,? d mm q Hermitian indefinite systems can be factorized with PAP T LDL, but D is block diagonal with 1 ˆ 1 and 2 ˆ 2 blocks. Its cost is similar to Cholesky factorization and is about 50% of Gaussian elimination. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 14 / 18

15 Outline 1 Stability of LU Factorization 2 Cholesky Factorization 3 Linear Algebra Software Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 15 / 18

16 Software for Linear Algebra LAPACK: Linear Algebra PACKage ( Standard library for solving linear systems and eigenvalue problems Depends on BLAS (Basic Linear Algebra Subprograms) Note: Uses Fortran conventions for matrix arrangements C-version: CLAPACK MATLAB e.g., Factorize A: lupaq and cholpaq Solve Ax b: x Azb Uses back/forward substitution for triangular matrices Uses Cholesky factorization for positive-definite matrices Uses LU factorization with column pivoting for nonsymmetric matrices Uses Householder QR for least squares problems Uses LAPACK and other packages internally Solvers for sparse matrices (e.g., SuperLU, TAUCS) Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 16 / 18

17 Some Examples Example BLAS routines: Matrix-vector multip.: dgemv; Matrix-matrix multip: dgemm LU Factorization Solve linear system Est. cond General Symmetric General Symmetric LAPACK dgetrf dpotrf/dsytrf dgesv dposv/dposvx dgecon LINPACK dgefa dpofa/dsifa dgesl dposl/dsisl dgeco MATLAB lu chol rcond Linear least squares Eigenvalue/vector SVD QR Solve Rank-deficient General Sym. LAPACK dgeqrf dgesl dgelsy/s/d dgeev dsyev dgesvd LINPACK dqrdc dqrsl dqrst dsvdc MATLAB qr eig eig svd For BLAS, LINPACK, and LAPACK, first letter s stands for single-precision real, d for double-precision real, c for single-precision complex, and z for double-precision complex. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 17 / 18

18 Using LAPACK Routines in C Programs LAPACK was written in Fortran 77. Special attention is required when calling a Fortran subroutine from C. Key differences between C and Fortran Storage of matrices: column major (Fortran) versus row major (C/C++) Argument passing for subroutines in C and Fortran: pass by reference (Fortran) and pass by value (C/C++) To find a function name, refer to LAPACK Users? Guide. or search netlib.org To find out arguments for a given function, search on netlib.org Note the difference on precision Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH562 MATH 562 Numerical Analys 18 / 18