3.2 Iterative Solution Methods for Solving Linear

Transcription

1 22 CHAPTER 3. NUMERICAL LINEAR ALGEBRA 3.2 Iterative Solution Methods for Solving Linear Systems Introduction We continue looking how to solve linear systems of the form Ax = b where A = (a ij is an n n matrix and b = (b i and x = (x i are two n 1 vectors. In this section, we will focus on iterative methods. Here is a link from Larson s book for more information on iterative methods Jacobi Iteration The method we describe here is named after Carl Gustav Jacob Jacobi ( , a mathematician from Prussia (Germany. It makes two assumptions: The system has a unique solution. The element on the main diagonal of A are all non-zero. This may require switching rows and/or columns of A to achieve this. For the purpose of this explanation, we consider the system when A is 3 3, a 11 x 1 + a 12 x 2 + a 13 x 3 = b 1 that is: a 21 x 1 + a 22 x 2 + a 23 x 3 = b 2. The technique described here easily a 31 x 1 + a 32 x 2 + a 33 x 3 = b 3 extends to n n matrices. Jacobi iteration is described as follows: 1. We solve for x 1 in the first equation, for x 2 in the second equation and x 3 in the third equation, to obtain: x 1 = 1 a 11 ( a 12 x 2 a 13 x 3 + b 1 x 2 = 1 a 22 ( a 21 x 1 a 23 x 3 + b 2 x 3 = 1 a 33 ( a 31 x 1 a 32 x 2 + b 3 (3.1 We can see why the diagonal elements have to be non-zero. They are in the denominator. 2. Next, we make an initial approximation (guess of the solution (x 1, x 2, x Next, we substitute the current values for (x 1, x 2, x 3 in the right-hand side of each equation of the system 3.1 and compute the new values for (x 1, x 2, x 3. This is called our first approximation. 4. Using these new values for (x 1, x 2, x 3, we repeat step 3 to get our second approximation. 5. Repeating steps 3 and 4 will give a sequence of approximations.

2 3.2. ITERATIVE SOLUTION METHODS FOR SOLVING LINEAR SYSTEMS23 It turns out that this sequence of approximations often converges to the solution of the original system. We ll discuss when a little later. We now focus on rewriting Jacobi iteration in terms of matrices as follows: 1. With D = D 1 ((D A x + b. a a a 33, we can write the second system as x = 2. In other words, solving Ax = b is the same as solving x = D 1 ((D A x + b. This is known as a fixed point problem, one of the techniques used to solve such problems is an iterative technique. We can now express Jacobi iteration in terms of matrices. We define our sequence of iterations x (n as follows: 1. Make a first guess, call it x (0 2. Define the sequence iteratively, using the Jacobi iteration scheme x (n+1 = D 1 ( (D A x (n + b 3. Stop when x (n+1 x (n < tolerance Where x is the norm or the magnitude of x. Remark Let us make a few remarks before continuing. 1. There are many possible norms which can be used. The most commonly used in linear algebra is the l 2 norm defined by x 2 = x x x2 n if x = (x 1, x 2,..., x n. We often drop the subscript 2 for the norm and simply write x. 2. In MATLAB, the l 2 norm norm of a vector x is norm(x,2 or simply norm(x. See help norm for other available norms. 3. Another commonly used norm in linear algebra is the l 1 norm. It is defined n by x 1 = x i if x = (x 1, x 2,..., x n. i=1 Of course, everything described above assumes that the sequence defined as such converges. In that case, the limit of the sequence is the fixed point. But how do we know the sequence of vectors x (n will converge? Before we answer the question about convergence, let us note that the Jacobi iteration scheme, x (n+1 = D ( 1 (D A x (n + b, can be written as a system as follows: x (n+1 1 = 1 ( a 12 x (n 2 a 13 x (n 3 + b 1 a 11 x (n+1 2 = 1 ( a 21 x (n 1 a 23 x (n 3 + b 2 a 22 x (n+1 3 = 1 ( a 31 x (n 1 a 32 x (n 2 + b 3 a 33

3 24 CHAPTER 3. NUMERICAL LINEAR ALGEBRA This is the form we use when we program it. We now turn our attention to the convergence of the sequence of approximations. Definition A matrix A = (a ij is said to be diagonally dominant if for each row of A, the magnitude of the diagonal entry in a row is larger than or equal to the sum of the magnitudes of all the other entries in that row, in other words if a ii a ij for each i. When a strict inequality is used instead, the j i matrix is said to be strictly diagonally dominant. Theorem If A is strictly diagonally dominant, then the Jacobi iteration scheme will converge for every choice of x (0. It is possible for the Jacobi iteration scheme to converge even if A is not diagonally dominant, but it is not guaranteed. Furthermore, in this case, the choice of x (0 is very important. Example Which of the matrices is diagonally dominant? Gauss-Seidel Iteration The Gauss-Seidel iteration is a variant of Jacobi iteration that usually improves convergence. It always uses the latest value of a particular variable. In Jacobi s iteration, we use the old value (x (n i for i = 1..n until all the new values (x (n+1 i for i = 1..n have been computed. The Gauss-Seidel scheme uses a new variable as soon as it is computed. For our 3 3 system, the new scheme is: x (n+1 1 = 1 ( a 12 x (n 2 a 13 x (n 3 + b 1 a 11 x (n+1 2 = 1 ( a 21 x (n+1 1 a 23 x (n 3 + b 2 a 22 x (n+1 3 = 1 ( a 31 x (n+1 1 a 32 x (n b 3 a 33 Gauss-Seidel can also be written as a matrix equation similar to Jacobi iteration: x (n+1 = D 1 ( (D A x (n + b. However, in order to do so, one must replace the matrix D A (which is A without its diagonal by L U where L is the lower triangular part of A without its diagonal and U is the upper triangular part of A without its diagonal (see assignment.

4 3.2. ITERATIVE SOLUTION METHODS FOR SOLVING LINEAR SYSTEMS25 The condition for the sequence of vectors obtained with Gauss-Seidel iteration to converge is the same as for Jacobi iteration. More specifically, we have the theorem: Theorem If A is strictly diagonally dominant, then the Gauss-Seidel iteration scheme will converge for every choice of x (0. To solve the system Ax = b using Gauss-Seidel iteration, one proceeds as follows: 1. Make a first guess, call it x (0 2. Define the sequence ( x (n iteratively, using the Gauss-Seidel iteration scheme. 3. Stop when x (n+1 x (n < tolerance Exercises 1. Implement Jacobi iteration as a MATLAB function with the format: function [x error niter flag] = xyz_jacobi(a,x,b,maxiter,tol where: xyz are the first three letters of your last name. INPUT A: the matrix of the system Ax = b x: the first guess vector Ax = b b: The vector in the system maxiter: the maximum number of iteration to perform tol: the tolerance OUTPUT x: the solution vector error: x (n+1 x (n niter: the number of iterations it took flag: Indicates whether a solution was found within the specified number of iterations. 1 means a solution was found, 0 means no solution was found. 2. Write the Gauss-Seidel iteration scheme in matrix format, following the hint in the slides. Explain what you are doing and why. You don t have to implement this as a MATLAB function. 3. Test your code on the two systems below. Report what you find and try to give an explanation. 4x y + z = 7 (a 4x 8y + z = 21 2x + y + 5z = 15

5 26 CHAPTER 3. NUMERICAL LINEAR ALGEBRA (b 2x + y + 5z = 15 4x 8y + z = 21 4x y + z = 7

6 Bibliography [1] M. C, S. M, Y. Z, V. C. L, Big data: related technologies, challenges and future prospects, Springer, [2] J. D, Big data, data mining, and machine learning: value creation for business leaders and practitioners, John Wiley & Sons, [3] L. G, What s this all about?, Time, 186 (2015, pp [4] H. A. K, Big Data: techniques and technologies in geoinformatics, CRC Press, [5] J. N. K, Data-Driven Modeling & Scientific Computation: Methods for Complex Systems and Big Data, Oxford University Press, [6] F. L R. I, Google le nouvel einstein, Science & Vie, 1138 (2012, pp [7] K.-C. L, H. J, L. T. Y, A. C, Big Data: Algorithms, Analytics, and Applications, CRC Press, [8] T. MP, Will our data drown us, IEEE Spectrum, (2015, pp [9] T. P, Giving your body a "check engine" light, IEEE Spectrum, (2015, pp [10] L. S, Should you get paid for your data, IEEE Spectrum, (2015, pp [11] E. S, Their prescription: Big data, IEEE Spectrum, (2015, pp [12] S. Q. Y, Big data analysis for bioinformatics and biomedical discoveries, CRC Press,