Iterative Methods Splitting Methods 1
Direct Methods Solving Ax = b using direct methods. Gaussian elimination (using LU decomposition) Variants of LU, including Crout and Doolittle Other decomposition methods including QR Advantages Algorithm terminates in a fixed number of iterations Can take advantage of sparsity (to some extent) 2
Direct Methods Possible disadvantages? We have to store the matrix, or at least know what the entries are No obvious way to get a solution that is close enough to the exact solution for whatever purpose we have in mind No way to reduce the operation count unless we have a special sparse structure. 3
Iterative methods - scalar case Can we apply what we know about root-finding to the problem of finding the solution to a linear system of equations? Newton s method, when applied to a scalar linear equation converges in one step: r = b ax f(x) =b ax Newton : x 1 = x 0 b ax a = b a 4
Iterative methods - scalar case We can formulate a fixed point problem with the function g(x) defined as g(x) = 1 m (b ax)+x = 1 a m x + b m where we choose m so that g 0 (x) < 1 (choose m =2a, for example). g(x) = 1 2 x + 5 6 f(x) =5 3x 5
Iterative methods - matrix case We can apply Newton s Method to the function F (x) =b Ax where x, b 2 R n and A 2 R n n. We get x k+1 = x k [F 0 (x k )] 1 F (xk ) = x k + A 1 (b Ax k ) This also gives us the answer in exactly one step! 6
Iterative methods - matrix case Appling the fixed point iteration to F (x) =b get g(x) =x + M 1 (b Ax) Ax, we where now M 2 R n n. The fixed point iteration then looks like x k+1 = x k + M 1 (b Ax k ) The choice of M will affect the convergence of the scheme. 7
Comparison Newton step: x k+1 = x k + A 1 (b Ax k ) Fixed point step: x k+1 = x k + M 1 (b Ax k ) The Newton step is no different than using a direct method, since we have to invert A at each step. The fixed point approach is promising, if we can understand how different choices for M affect convergence of the scheme. 8
A simple iteration scheme For the scalar case, we have that g(x) = 1 m (b ax)+x = 1 a m x + b m which will converge to a root of f(x) =b ax if g 0 (x) = 1 a m < 1 and the closer m is to a, the faster the convergence. In fact, if m = a, we get convergence in one step. 9
A simple iteration scheme In the matrix case, we have g(x) = x + M 1 (b Ax) = I M 1 A x + M 1 b we also want M to be as close to A as is feasible, while still being easily invertible. If M = A 1, we converge in one step. Do we also have to have a condition like g 0 (x) < 1? 10
General splitting method In a general splitting method, we write A = P Q This leads to the iterative scheme P x k+1 = Qx k + b or x k+1 = P 1 Qx k + P 1 b The matrix P 1 Q is sometimes called the iteration matrix. 11
General splitting method For our method, we have P M and Q M A. Then x k+1 = M 1 (M A) x k + M 1 b = I M 1 A x k + M 1 b The splitting matrix is P 1 Q (I M 1 A) 12
Classic splitting methods Some classic schemes. Note that for each scheme, M approximates A. x k+1 = I M 1 A x k + M 1 b M = D M = L The Jacobi iteration The Gauss-Seidel iteration M =(! 1 1)D + L The Successive Over Relaxation or SOR Method where D is the diagonal of A and L is the lower triangular portion of A (including diagonal). The matrix M for each of these choices is easy to invert! 13
Simple Iteration Algorithm x k = I M 1 A x k 1 + M 1 b Given an initial guess x 0, compute r 0 = b solve Mz 0 = r 0. Ax 0, and For k =1, 2,..., Set x k = x k 1 + z k 1. Compute r k = b Ax k. Matrix vector multiply Solve Mz k = r k should be easy to solve! if kz k k< Note that z k is the increment x k x k 1. Stop. end 14
Convergence? Let A be an n n matrix. matrix is defined as The spectral radius of a (A) = max { : is an eigenvalue of A} The following are equivalent (A) < 1 A k! 0 as k!1, and A k x! 0 as k!1for any vector x. If one of the above conditions is true, they are all true. 15
Convergence Let e k = A 1 b x k.then e k =(I M 1 A)e k 1 =...=(I M 1 A) k e 0 Then show this! ke k kapplek I M 1 A k kke 0 k The iteration will converge if and only if we have (I M 1 A) < 1 16
Convergence Depending on which norm we use, however, we might not see linear convergence. All we can say is that to get ke k kapple ", we require that k satisfy log(") k log ki M 1 Ak This only says that we need at least this many iterations; in practice the convergence can be quite slow. 17
Convergence of Splitting methods Some relevant matrix properties for A 2 R n n : Symmetric : A T = A Positive definite : x T Ax > 0 for any x 2 R n Symmetric, positive definite (SPD) : A is symmetric, and all of its eigenvalues are positive, Reminder : If A is symmetric, then the eigenvalues of A are positive if and only if A is positive definite. Diagonally dominant : P n j=1,i6=j a ij < a ii 18
Convergence of splitting methods Suppose A 2 R n n is a non-singular matrix. Then If A is strictly diagonally dominant, then the Jacobi method converges for any starting vector x 0. If A is symmetric, positive definite, then the Gauss- Seidel method converges for any starting vector x 0. If A is symmetric, positive definite and 0 <! < 2, then the SOR method converges for any starting vector x 0. Note : There are many examples of matrices for which the above conditions do not hold, and yet the iteration using one of the three methods, converges. 19
Examples of convergence Jacobi SOR, with optimal! Gauss-Seidel Convergence for model tridiagonal [1, 2, 1] matrix. 20
Examples of convergence SOR Gauss-Seidel Jacobi Convergence for a diagonally dominant system and poorly chosen SOR parameter. 21
Examples of convergence Gauss-Seidel Errors initially increase Pathological convergence for a highly non-normal matrix 22