Numerical Methods in Matrix Computations

Transcription

1 Ake Bjorck Numerical Methods in Matrix Computations Springer

2 Contents 1 Direct Methods for Linear Systems Elements of Matrix Theory Matrix Algebra Vector Spaces Submatrices and Block Matrices Operation Counts in Matrix Algorithms Permutations and Determinants The Schur Complement Vector and Matrix Norms Eigenvalues The Singular Value Decomposition Gaussian Elimination Methods Solving Triangular Systems Gaussian Elimination and LU Factorization LU Factorization and Pivoting Variants of LU Factorization Elementary Elimination Matrices Computing the Matrix Inverse Perturbation Analysis Scaling and componentwise Analysis Hermitian Linear Systems Properties of Hermitian Matrices The Cholesky Factorization Inertia of Symmetric Matrices Symmetric Indefinite Matrices Error Analysis in Matrix Computations Floating-Point Arithmetic Rounding Errors in Matrix Operations Error Analysis of Gaussian Elimination 95 xi

3 xii Contents Estimating Condition Numbers Backward Perturbation Bounds Iterative Refinement of Solutions Interval Matrix Computations Banded Linear Systems Band Matrices Multiplication of Band Matrices LU Factorization of Band Matrices Tridiagonal Linear Systems Envelope Methods Diagonally Dominant Matrices Implementing Matrix Algorithms BLAS for Linear Algebra Software Block and Partitioned Algorithms Recursive Matrix Multiplication Recursive Cholesky and LU Factorizations Sparse Linear Systems Storage Schemes for Sparse Matrices Graphs and Matrices Graph Model of Cholesky Factorization Ordering Algorithms for Cholesky Factorization Sparse Unsymmetric Matrices Permutation to Block Triangular Form Linear Programming and the Simplex Method Structured Linear Equations Kronecker Products and Linear Systems Toeplitz and Hankel Matrices Vandermonde Systems Semiseparable Matrices The Fast Fourier Transform Cauchy-Like Matrices Notes and Further References 200 References Linear Least Squares Problems Introduction to Least Squares Methods The Gauss-Markov Model Projections and Geometric Characterization The Method of Normal Equations Stability of the Method of Normal Equations Least Squares Problems and the SVD SVD and the Pseudoinverse Perturbation Analysis 232

4 Contents xiii SVD and Matrix Approximation Backward Error Analysis Principal Angles Between Subspaces Orthogonal Factorizations Elementary Orthogonal Matrices QR Factorization and Least Squares Problems Golub-Kahan Bidiagonalization Gram-Schmidt QR Factorization Loss of Orthogonality and Reorthogonalization MGS as a Householder Method Partitioned and Recursive QR Factorization Condition Estimation and Iterative Refinement Rank-Deficient Problems Numerical Rank Pivoted QR Factorizations Rank-Revealing Permutations Complete QR Factorizations The QLP Factorization Modifying QR Factorizations Stepwise Variable Regression Structured and Sparse Least Squares Kronecker Products Tensor Computations Block Angular Least Squares Problems Banded Least Squares Problems Sparse Least Squares Problems Block Triangular Form Regularization of Ill-Posed Linear Systems TSVD and Tikhonov Regularization Least Squares with Quadratic Constraints Bidiagonalization and Partial Least Squares The NIPALS Algorithm Least Angle Regression and /] Constraints Some Special Least Squares Problems Weighted Least Squares Problems Linear Equality Constraints Linear Inequality Constraints Generalized Least Squares Problems Indefinite Least Squares Total Least Squares Problems Linear Orthogonal Regression The Orthogonal Procrustes Problem 391

5 xiv Contents 2.8 Nonlinear Least Squares Problems Conditions for a Local Minimum Newton and Gauss-Newton Methods Modifications for Global Convergence Quasi-Newton Methods Separable Least Squares Problems Iteratively Reweighted Least Squares Nonlinear Orthogonal Regression Fitting Circles and Ellipses 413 References Matrix Eigenvalue Problems Basic Theory Eigenvalues of Matrices The Jordan Canonical Form The Schur Decomposition Block Diagonalization and Sylvester's Equation Perturbation Theory Gersgorin's Theorems General Perturbation Theory Perturbation Theorems for Hermitian Matrices The Rayleigh Quotient Bounds Numerical Range and Pseudospectra The Power Method and Its Generalizations The Simple Power Method Deflation of Eigenproblems Inverse Iteration Rayleigh Quotient Iteration Subspace Iteration The LR and QR Algorithms The Basic LR and QR Algorithms The Practical QR Algorithm Reduction to Hessenberg Form The Implicit Shift QR Algorithm Enhancements to the QR Algorithm The Hermitian QR Algorithm Reduction to Real Symmetric Tridiagonal Form Implicit QR Algorithm for Hermitian Matrices The QR-SVD Algorithm Skew-Symmetric and Unitary Matrices Some Alternative Algorithms The Bisection Method Jacobi's Diagonalization Method 532

6 Contents Jacobi SVD Algorithms Divide and Conquer Algorithms Some Generalized Eigenvalue Problems Canonical Forms Solving Generalized Eigenvalue Problems The CS Decomposition Generalized Singular Value Decomposition Polynomial Eigenvalue Problems Hamiltonian and Symplectic Problems Functions of Matrices The Matrix Square Root The Matrix Sign Function The Polar Decomposition The Matrix Exponential and Logarithm Nonnegative Matrices with Applications The Perron-Frobenius Theory Finite Markov Chains Notes and Further References 602 References Iterative Methods Classical Iterative Methods A Historical Overview A Model Problem Stationary Iterative Methods Convergence of Stationary Iterative Methods Relaxation Parameters and the SOR Method Effects of Non-normality and Finite Precision Polynomial Acceleration Krylov Methods for Hermitian Systems General Principles of Projection Methods The One-Dimensional Case The Conjugate Gradient (CG) Method Rate of Convergence of the CG Method The Lanczos Process Indefinite Systems Block CG and Lanczos Processes Krylov Methods for Non-Hermitian Systems The Arnoldi Process Two-Sided Lanczos and the BiCG Method The Quasi-Minimal Residual Algorithm Transpose-Free Methods Complex Symmetric Systems 685

7 xvi Contents 4.4 Preconditioned Iterative Methods Some Preconditioned Algorithms Gauss-Seidel and SSOR Preconditioners Incomplete LU Factorization Incomplete Cholesky Factorization Sparse Approximate Inverse Preconditioners Block Incomplete Factorizations Preconditioners for Toeplitz Systems Iterative Methods for Least Squares Problems Basic Least Squares Iterative Methods Jacobi and Gauss-Seidel Methods Krylov Subspace Methods GKL Bidiagonalization and LSQR Generalized LSQR Regularization by Iterative Methods Preconditioned Methods for Normal Equations Saddle Point Systems Iterative Methods for Eigenvalue Problems The Rayleigh-Ritz Procedure The Arnoldi Eigenvalue Algorithm The Lanczos Algorithm Reorthogonalization of Lanczos Vectors Convergence of Arnoldi and Lanczos Methods Spectral Transformation The Lanczos-SVD Algorithm Subspace Iteration for Hermitian Matrices Jacobi-Davidson Methods Notes and Further References 771 References 772 Mathematical Symbols 783 Flop Counts 785 Index 787