Lecture 16 Methods for System of Linear Equations (Linear Systems) Songting Luo Department of Mathematics Iowa State University MATH 481 Numerical Methods for Differential Equations Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 1 / 19
Outline 1 Direct Methods 2 Iterative Methods 3 Stationary Methods 4 Convergence of Jacobi, Gauss-Seidel, SOR Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 2 / 19
Solving Linear System Ax b General Properties Two classes of numerical methods: direct and iterative Direct Methods: terminates after finite many steps gives exact solution, except for roundoff errors accuracy governed by condition number examples: Gaussian elimination, LU, QR, SVD, Cholesky, tec. Iterative Methods: produces a sequence of approximate solutions never produces exact solution, except by accident accuracy governed by approximation error examples: fixed point iteration, stationary methods, non-stationary methods. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 3 / 19
Brief Review of Direct Methods We review direct methods briefly. Using Gaussian elimination, LU factorization as examples: Gaussian elimination: how it works? LU factorization; forward substitution; backward substitution. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 4 / 19
Overview of Iterative Methods General Schemes x p0q initial guess iterative process: x pn`1q gpx pnq q (stationary methods, equivalent to fixed point iterations) or x pn`1q g pnq px pnq q (non-stationary methods) For either way: error: e pnq x pnq x ( with x the true solution); e n }e pnq } usual behavior: e n`1 «ce p n for some constant c: convergence of order p. x satisfies x gp xq if g is continuous ô x is a fixed point. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 5 / 19
Example: Rooting Finding One-dimensional Solve fpxq 0 by rewriting it as x gpxq, then iterate x pn`1q gpx pnq q. Assume g is differentiable to all necessary orders (for simplicity), so: e pn`1q x pn`1q x gpx pnq q gp xq g 1 p xqe pnq ` 1 2 g2 p xqpe pnq q 2 ` e n`1 «g 1 p xq e n What if g 1 p xq 0? What about high-dimensional? p linear convergenceq «g 1 p xq n e 0 p converges if g 1 p xq ă 1q Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 6 / 19
Stationary Methods Stationary Methods for Linear Systems Include: Jacobi Iteration Gauss-Seidel Iteration SOR (successive Over-relaxation) Iteration. Basic approach: want to solve Ax b, split A B ` C with B non-singular, easy to invert. converges if ρpb 1 Cq ă 1. pb ` Cqx b Bx Cx ` b x B 1 Cx ` B 1 b Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 7 / 19
Jacobi Iteration Jacobi Iteration A L ` D ` U where L is strictly lower triangular, D is diagonal, and U is strictly upper triangular. pl ` D ` Uqx b Then Jacobi iteration is given as Dx pl ` Uqx ` b x D 1 pl ` Uqx ` D 1 b x pn`1q D 1 pl ` Uqx pnq ` D 1 b Clearly, ρpd 1 pl ` Uqq is relevant for convergence. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 8 / 19
Gauss-Seidel Iteration Gauss-Seidel Iteration pl ` D ` Uqx b pd ` Lqx Ux ` b x pd ` Lq 1 Ux ` pd ` Lq 1 b Then Gauss-Seidel iteration is given as x pn`1q pd ` Lq 1 Ux pnq ` pd ` Lq 1 b Clearly, ρppd ` Lq 1 Uq is relevant for convergence. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 9 / 19
Jacobi v.s. Gauss-Seidel Both methods take Opn 2 q operations per iteration for a full matrix. They usually take Opnq iterations, which give total Opn 3 q operations (no gain compared to Direct Methods). Therefore, such methods are usually for sparse matrices. What is the difference between Jacobi and Gauss-Seidel? Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 10 / 19
SOR Iteration SOR SOR is generalization of Gauss-Seidel. Given x pnq, let x pn`1q GS Gauss-Seidel. So, Define: x pn`1q SOR x pn`1q GS xpnq ` ωpx pn`1q x pnq ` px pn`1q x pnq q GS where ω is the relaxation parameter ω ă 1: under-relaxation ω 1: SOR GS ω ą 1: over-relaxation GS x pnq q p1 ωqx pnq ` ωx pn`1q GS x pn`1q by Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 11 / 19
SOR: Matrix Formulation Look at i-th equation: ř n j 1 a ijx j b i. SOR: x pn`1q i Re-arrange terms: p1 ωqx pnq i ` ω i 1 ÿ rb i a ij x pn`1q a ii j 1 j nÿ j i`1 a ij x pnq j s x pn`1q pd ` ωlq 1 rp1 ωqd ωlsx pnq ` ωpd ωlq 1 b Clearly, ρppd ` ωlq 1 rp1 ωqd ωlsq is relevant for convergence Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 12 / 19
Convergence of Jacobi, Gauss-Seidel, SOR Notations For A L ` D ` U, let Goals J D 1 pl ` Uq G pd ` Lq 1 U Spωq pd ` ωlq 1 rp1 ωqd ωls. explicitly estimate ρ, if possible. if not possible, at least prove ρ ă 1 for SOR, determine optimal ω with minimum ρ. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 13 / 19
Convergence Theorem If J is non-negative (i.e., all entries ě 0), then exactly one of the following occurs: ρpjq ρpgq 0 0 ă ρpgq ă ρpjq ă 1 ρpjq ρpgq 1 1 ă ρpjq ă ρpgq Special Case If all diagonal entries of A are positive, and all others are negative, then Jacobi, Gauss-Seidel either both work or both donot work. If they work, Gauss-Seidel is faster. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 14 / 19
Convergence cont ed Definition A is strictly diagonally row dominant if a kk ą ÿ a kj @ k j k A is strictly diagonally column dominant if a kk ą ÿ a jk @ k j k Theorem If A is strictly diagonally dominant, both Jacobi and Gauss-Seidel converge. Proof. Use row dominant as example. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 15 / 19
Convergence cont ed Theorem Gauss-Seidel converges if A is real symmetric, positive definite. Proof. ρ ă 1? Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 16 / 19
Convergence cont ed Definition Let J D 1 pl ` Uq be the Jacobi iteration matrix, and define Jpαq D 1 pαl ` 1 Uq, α 0. α A is consistently ordered if the eigenvalues of Jpαq are independent of α. Corollary If A is consistently ordered, all eigenvalues of J are in pairs pλ, λq. Proof. observe Jp1q J, Jp 1q J. Theorem If A is block tridiagonal, A is consistently ordered. Proof. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 17 / 19
Convergence cont ed Theorem ρpspωqq ě w 1. SOR can only possibly work if ω P p0, 2q. Proof. Theorem If A is consistently ordered, the eigenvalues µ of J, and λ 0 of Spωq are related by pλ ` ω 1q 2 λω 2 µ 2 Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 18 / 19
Convergence cont ed Theorem If A is consistently ordered, if the eigenvalues of J are real, and ρpjq ă 1, then $ 1 ω ` 1 c & 2 ω2 ρpjq ` ωρpjq 1 ω ` 1 4 ω2 rρpjqs 2, ρpspωqq if 0 ď ω ď ω b, ω 1, % if ω b ď ω ď 2, where In particular, ρpspω b qq ω b 1. 2 ω b 1 ` a1 rρpjqs. 2 It is better to take ω a little larger than ω b, rather than a little smaller. Songting Luo ( Department of Mathematics Iowa State University[0.5in] MATH481 MATH 481 Numerical Metho 19 / 19