Solving Linear Systems Iterative Solutions Methods Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) Linear Systems Fall 2015 1 / 12
Introduction We continue looking how to solve linear systems of the form Ax = b where A = (a ij ) is an n n matrix and b = (b i ) and x = (x i ) are two n 1 vectors. In these slides, we will focus on iterative methods. Here is a link to iterative methods from Larson s book. Philippe B. Laval (KSU) Linear Systems Fall 2015 2 / 12
Jacobi Iteration For the purpose of this explanation, consider the system when A is a 11 x 1 + a 12 x 2 + a 13 x 3 = b 1 3 3, that is: a 21 x 1 + a 22 x 2 + a 23 x 3 = b 2 a 31 x 1 + a 32 x 2 + a 33 x 3 = b 3 Solving for x 1 in the first equation, for x 2 in the second equation and x 3 in the third equation, we obtain: x 1 = 1 ( a 12 x 2 a 13 x 3 + b 1 ) a 11 x 2 = 1 ( a 21 x 1 a 23 x 3 + b 2 ) a 22 x 3 = 1 ( a 31 x 1 a 32 x 2 + b 3 ) a 33 Philippe B. Laval (KSU) Linear Systems Fall 2015 3 / 12
Jacobi Iteration With D = a 11 0 0 0 a 22 0 0 0 a 33 x = D 1 ((D A) x + b)., we can write the second system as In other words, solving Ax = b is the same as solving x = D 1 ((D A) x + b). This is known as a fixed point problem, one of the techniques used to solve such problems is an iterative technique. Philippe B. Laval (KSU) Linear Systems Fall 2015 4 / 12
Jacobi Iteration We define a sequence x (n) as follows: 1 Make a first guess, call it x (0) 2 Define the sequence iteratively, using the Jacobi iteration scheme x (n+1) = D 1 ( (D A) x (n) + b ) 3 Stop when x (n+1) x (n) < tolerance Where x is the norm or the magnitude of x. There are many possible norms which can be used. The most commonly used in linear algebra is the l 2 norm defined by x 2 = x1 2 + x 2 2 +... + x n 2 if x = (x 1, x 2,..., x n ). We often drop the subscript 2 for the norm and simply write x. Of course, this assumes that the sequence defined as such converges. In that case, the limit of the sequence is the fixed point. But how do we know the sequence of vectors x (n) will converge? Philippe B. Laval (KSU) Linear Systems Fall 2015 5 / 12
Jacobi Iteration In MATLAB, the l 2 norm norm of a vector x is norm(x,2) or simply norm(x). See help norm for other available norms. Before we answer the question about convergence, let us make a remark. The Jacobi iteration scheme, x (n+1) = D 1 ( (D A) x (n) + b ), can be written as a system as follows: x (n+1) 1 = 1 ( ) a 12 x (n) 2 a 13 x (n) 3 + b 1 a 11 x (n+1) 2 = 1 ( ) a 21 x (n) 1 a 23 x (n) 3 + b 2 a 22 x (n+1) 3 = 1 ( ) a 31 x (n) 1 a 32 x (n) 2 + b 3 a 33 This is the form we use when we program it. Philippe B. Laval (KSU) Linear Systems Fall 2015 6 / 12
Jacobi Iteration Definition A matrix A = (a ij ) is said to be diagonally dominant if for each row of A, the magnitude of the diagonal entry in a row is larger than or equal to the sum of the magnitudes of all the other entries in that row, in other words if a ii a ij for each i. When a strict inequality is used instead, the j i matrix is said to be strictly diagonally dominant. Theorem If A is strictly diagonally dominant, then the Jacobi iteration scheme will converge for every choice of x (0). It is possible for the Jacobi iteration scheme to converge even if A is not diagonally dominant, but it is not guaranteed. Furthermore, in this case, the choice of x (0) is very important. Philippe B. Laval (KSU) Linear Systems Fall 2015 7 / 12
Gauss-Seidel Iteration The Gauss-Seidel iteration is a variant of Jacobi iteration that usually improves convergence. It always uses the latest value of a particular variable. In Jacobi s iteration, we use the old value (x (n) i for i = 1..n) until all the new values (x (n+1) i for i = 1..n) have been computed. The Gauss-Seidel scheme uses a new variable as soon as it is computed. For our 3 3 system, the new scheme is: x (n+1) 1 = 1 ( ) a 12 x (n) 2 a 13 x (n) 3 + b 1 a 11 x (n+1) 2 = 1 ( ) a 21 x (n+1) 1 a 23 x (n) 3 + b 2 a 22 x (n+1) 3 = 1 ( ) a 31 x (n+1) 1 a 32 x (n+1) 2 + b 3 a 33 Philippe B. Laval (KSU) Linear Systems Fall 2015 8 / 12
Gauss-Seidel Iteration Gauss-Seidel can also be written as a matrix equation similar to Jacobi iteration: x (n+1) = D 1 ( (D A) x (n) + b ). However, in order to do so, one must replace the matrix D A (which is A without its diagonal) by L U where L is the lower triangular part of A without its diagonal and U is the upper triangular part of A without its diagonal (see assignment). The condition for the sequence of vectors obtained with Gauss-Seidel iteration to converge is the same as for Jacobi iteration. More specifically, we have the theorem: Theorem If A is strictly diagonally dominant, then the Gauss-Seidel iteration scheme will converge for every choice of x (0). Philippe B. Laval (KSU) Linear Systems Fall 2015 9 / 12
Gauss-Seidel Iteration To solve the system Ax = b using Gauss-Seidel iteration, one proceeds as follows: 1 Make a first guess, call it x (0) 2 Define the sequence ( x (n)) iteratively, using the Gauss-Seidel iteration scheme. 3 Stop when x (n+1) x (n) < tolerance Philippe B. Laval (KSU) Linear Systems Fall 2015 10 / 12
Assignment To be turned in by September 13, before midnight. For question 1, email me the m-file. For questions 2 & 3, type the solutions/answers either as a latex file, or using MS Word and email me your answers. 1. Implement Jacobi iteration as a MATLAB function with the format: function [x error niter flag] = my_jacobi(a,x,b,maxiter,tol) where: INPUT A: the matrix of the system Ax = b x: the first guess vector Ax = b b: The vector in the system maxiter: the maximum number of iteration to perform tol: the tolerance OUTPUT x: the solution vector error: x (n+1) x (n) niter: the number of iterations it took flag: Indicates whether a solution was found within the specified number of iterations. 0 means a solution was found, 1 means no solution was found. Philippe B. Laval (KSU) Linear Systems Fall 2015 11 / 12
Assignment 2. Write the Gauss-Seidel iteration scheme in matrix format, following the hint in the slides. 3. Test your code on the two systems below. Report what you find and try to give an explanation. 4x y + z = 7 4x 8y + z = 21 2x + y + 5z = 15 2x + y + 5z = 15 4x 8y + z = 21 4x y + z = 7 Philippe B. Laval (KSU) Linear Systems Fall 2015 12 / 12