Next topics: Solving systems of linear equations

Next topics: Solving systems of linear equations 1 Gaussian elimination (today) 2 Gaussian elimination with partial pivoting (Week 9) 3 The method of LU-decomposition (Week 10) 4 Iterative techniques: Jacobi method (Week 11) 5 Revision (Week 12) 6 Continuous Assessment Test (Week 13 - Monday, 10th December, 7.00-9.00pm) Dana Mackey (DIT) Numerical Methods 1 / 58

Solving systems of linear equations Example: A mathematician has three children called X, Y and Z. Next year, Y will be half the age of X, while X will be three times as old as Z. In 5 years time, X will be twice as old as the sum of the current ages of the other two. How old are the children? Dana Mackey (DIT) Numerical Methods 2 / 58

Review of matrices Let m, n IN. A rectangle of numbers written like a 11 a 12... a 1n a 21 a 22... a 2n A =...... a m1 a m2... a mn where each a ij IR is called a matrix with m rows and n columns or an m n matrix. The above matrix may be denoted by [a ij ] 1 i m 1 j n (or just by [a ij ]). The number a ij is called the (i,j) entry. It lies in the i th row and j th column. Dana Mackey (DIT) Numerical Methods 3 / 58

The entries like a 11, a 22, a 33 and so on are called the diagonal entries. These entries form the main diagonal. Notice that the main diagonal finishes in the bottom right corner only if the matrix has the same number of rows as columns, i.e. we have a square matrix. A matrix composed of only one column (that is an m 1 matrix) is called a column matrix or vector. Two matrices are equal if and only if they have the same size and the corresponding entries are all equal. For each m and n in IN, we use M m,n for the set of all m n matrices. The m n matrix whose entries are all zero is called the m n zero matrix and is denoted 0 m,n or simply 0. Dana Mackey (DIT) Numerical Methods 4 / 58

Matrix addition If A = [a ij ] and B = [b ij ] are two matrices in M m,n then we define A + B M m,n by a 11 + b 11 a 12 + b 12... a 1n + b 1n a 21 + b 21 a 22 + b 22... a 2n + b 2n A + B = [a ij + b ij ] =...... a m1 + b m1 a m2 + b m2... a mn + b mn For example, [ ] 1 2 1 + 4 3 0 [ ] 0 3 2 = 5 4 1 [ ] 1 1 3. 9 1 1 Note: We can only add two matrices if they are of the same size. Dana Mackey (DIT) Numerical Methods 5 / 58

Scalar multiplication If A = [a ij ] M m,n and k IR then we define the product of A with the scalar number k to be ka M m,n given by ka 11 ka 12... ka 1n ka 21 ka 22... ka 2n ka = [ka ij ] =...... ka m1 ka m2... ka mn This operation is called scalar multiplication. Note that ka = [ka ij ] = [a ij k] so the notation Ak will also be used, and then ka = Ak. For example 5 [ ] 1 2 = 4 3 [ ] 5 10 and 0 20 15 [ ] 1 2 = 4 3 [ ] 0 0. 0 0 Dana Mackey (DIT) Numerical Methods 6 / 58

Inner product The next question is how to multiply two matrices. We start with a special case. Let n IN and let A be a 1 n matrix (a row matrix) and let B be an n 1 column matrix. That is A = [ ] b 2 a 1 a 2... a n while B =.. We b n define the (inner) product of A and B to be b 1 AB = [ ] b 2 a 1 a 2... a n. = a 1b 1 + a 2 b 2 + + a n b n. b n b 1 Dana Mackey (DIT) Numerical Methods 7 / 58

Note that the inner product is of a row and a column of the same length (the length of a row or column matrix is the number of entries in it) and that the answer is a real number (not a matrix). For example [ ] 2 2 1 3 5 = 2( 2) + ( 1)5 + 3(3) = 4 5 + 9 = 0. 3 Dana Mackey (DIT) Numerical Methods 8 / 58

Matrix multiplication Let m, n and p IN. Suppose A M m,n and B M n.p. Then we define the product of A and B to be the m p matrix formed as follows: The (i,j) entry of AB is the inner product of the i th row of A with the j th column of B. [ ] 1 3 2 1 2 0 1 Example: Let A = and B = 0 4 2 1. Then A M 3 1 1 2,3 2 0 2 3 and B M 3,4 implies AB M 2,4 and AB = [ 2 0 1 3 1 1 ] 3 2 1 1 0 4 2 1 = 2 0 2 3 [ 0 6 2 ] 1 5 5 10 7 Dana Mackey (DIT) Numerical Methods 9 / 58

In general AB BA for matrices In order to multiply two matrices, A by B, the number of columns of A must equal the number of rows of B. In other words, if A M m,n and B M p,q then to form AB we must have that n = p. To form BA we must have q = m. In particular we note that AB BA in this case. Even if we take two matrices A and B so that AB and BA are the [ same ] size, we cannot [ ] 1 2 5 6 assume that AB = BA. For example, if A = and B = then 3 4 7 8 [ ][ ] 1 2 5 6 AB = = 3 4 7 8 [ 19 ] 22 43 50 while [ ][ ] 5 6 1 2 BA = = 7 8 3 4 [ ] 23 34 AB. 31 46 Dana Mackey (DIT) Numerical Methods 10 / 58

Elementary Row operations Given a matrix, the following operations on the rows of that matrix are called (elementary) row operations. 1 Interchange rows, e.g. 0 1 2 3 1 3 2 9 4 R 1 R 2 3 1 3 0 1 2 2 9 4 2 Multiply a row by a non-zero real number 1 3 1 3 1 0 1 2 R 1 1 3 R 3 1 1 0 1 2 2 9 4 2 9 4 3 Add a scalar multiple of one row to another 1 1 1 3 1 1 0 1 2 R 3 R 3 2R 3 1 1 0 1 2 = 2 9 4 2 2(1) 9 2 3 4 2( 1) 1 1 3 1 0 1 2 0 8 1 3 6 Dana Mackey (DIT) Numerical Methods 11 / 58

Echelon form A matrix A = [a ij ] is said to be in echelon form if a ij = 0 for i > j. For 1 2 2 3 example, 0 2 3 17 is in echelon form. A square matrix in 0 0 5 1 echelon form is called upper triangular. Any matrix can be converted to echelon form by using elementary row operations. The steps involved are as follows. Let A = [a ij ] M m,n. Dana Mackey (DIT) Numerical Methods 12 / 58

1 Look at the first column. We want all entries below the first one, a 11, to be zero. If necessary (that is, if a 11 = 0), interchange two rows so that a 11 0. 2 Add multiples of the first row to the other rows to ensure a 21, a 31,...,a m1 are all zero. For example, add a 21 row 1 to row 2, etc. 3 Move to the second column. We want all the entries below a 22, that is a 32, a 42, etc., to be zero. If necessary, interchange two rows (not the first row) so that a 22 0. 4 Add multiples of the second row to ensure a 32,a 42,...,a m2 are all zero. 5 Move along the columns in this way until the matrix is in echelon form. a 11 Dana Mackey (DIT) Numerical Methods 13 / 58

Example: 0 1 2 Convert the matrix 3 1 3 to echelon form. 2 9 4 Dana Mackey (DIT) Numerical Methods 14 / 58

Systems of linear equations A linear equation in one variable is an equation of the form ax = b where a and b are given real numbers, and x is a variable. A linear equation in n variables is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b where a 1,...,a n,b IR. For example, 2x 1 + 3x 2 = 7 is a linear equation in two variables; 2x 3y + 4z = 5 is a linear equation in three variables. Dana Mackey (DIT) Numerical Methods 15 / 58

We can also consider a system of m equations in n variables. That is, a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 2 + a 22 x 2 + + a 2n x n = b 2.. a m1 x 1 + a m2 x 2 + + a mn x n = b m. The system of m equations above can also be written as one matrix equation: a 11 a 12... a 1n x 1 b 1 a 21 a 22... a 2n x 2....... = b 2. a m1 a m2... a mn x n b m and we will usually write this as AX = B. When written in this form, a solution of the system will be a n 1 column matrix with n elements (or n-vector). A linear system may have no solutions, many solutions or just a unique solution. Dana Mackey (DIT) Numerical Methods 16 / 58

The method of Gaussian Elimination for solving systems Suppose the system is AX = B where A M m,n and B M m,1. Form the augmented matrix [A. B] and convert this matrix to echelon form to get a matrix [A. B ]. The echelon system A X = B has the same solutions as the original system AX = B but is much easier to solve. This process of solving a linear system is called Gaussian elimination. Example: Solve the system of linear equations, x 2 + 2x 3 = 3 x 1 + 2x 2 + 3x 3 = 2 x 1 2x 2 2x 3 = 0 Dana Mackey (DIT) Numerical Methods 17 / 58

More examples Solve, by Gaussian elimination, the following systems of linear equations. (i) x 1 2x 2 2x 3 = 4 3x 1 x 2 x 3 = 7 6x 1 +x 2 +x 3 = 5. (ii) x 1 +x 2 +x 3 x 4 = 0 3x 1 x 2 x 3 +2x 4 = 0 4x 1 +5x 2 +x 3 x 4 = 0 9x 1 +6x 2 +2x 3 x 4 = 0 Dana Mackey (DIT) Numerical Methods 18 / 58

Gaussian Elimination with partial pivoting Example: Consider the following system 2 3 x 1 + 2 7 x 2 + 1 5 x 3 = 43 15 1 3 x 1 + 1 7 x 2 1 2 x 3 = 5 6 1 5 x 1 3 7 x 2 + 2 5 x 3 = 12 5 (1) Exercise: Using Gaussian elimination and back substitution show that the exact solution is x 1 = 1, x 2 = 7 and x 3 = 1. Dana Mackey (DIT) Numerical Methods 19 / 58

Now write the system using four decimal digit rounding arithmetic 0.6667x 1 + 0.2857x 2 + 0.2000x 3 = 2.867 0.3333x 1 + 0.1429x 2 0.5000x 3 = 0.8333 (2) 0.2000x 1 0.4286x 2 + 0.4000x 3 = 2.400 Remarks: 1 To obtain the 4-digit floating point approximation of a number, we use the rounding method (e.g. 2 3 = 0.66666... gives 0.6667.) 2 Whenever performing a calculation (e.g. addition or multiplication) involving two or more numbers, the 4-digit rounded form is used for each number and then the result of the calculation is rounded as well. For example, 0.1234 1.567 = 0.193368... = 0.1934. This is the source of roundoff error accummulation. 3 When performing row operations to get the echelon form, the entries below the pivot are implicitly set equal to zero, as opposed to being explicitly calculated, in order to avoid unnecessary calculations. Dana Mackey (DIT) Numerical Methods 20 / 58

The first step of the Gaussian elimination yields 0.6667x 1 + 0.2857x 2 +0.2000x 3 = 2.867 + 0.0001x 2 0.6000x 3 = 0.5997 (3) 0.5143x 2 +0.3400x 3 = 3.260 where the row operations performed were R 2 0.3333 0.6667 R 1 = R 2 0.4999 R 1 R 3 0.2000 0.6667 R 1 = R 3 0.3 R 1 and the above entries have been calculated as follows: and so on. 0.1429 0.2857 0.4999 = 0.1429 0.1428 = 0.0001 0.8333 2.867 0.4999 = 0.8333 1.433 = 0.5997 0.4286 0.2857 0.3 = 0.4286 0.08571 = 0.5143 Dana Mackey (DIT) Numerical Methods 21 / 58

The next step of the Gaussian elimination gives 0.6667x 1 + 0.2857x 2 +0.2000x 3 = 2.867 which gives the solution: x 3 = 1.000 (exact) x 2 = 3.000 (relative error=57%) and x 1 = 2.715 (relative error=200%)! + 0.0001x 2 0.6000x 3 = 0.5997 (4) 3086x 3 = 3087 Dana Mackey (DIT) Numerical Methods 22 / 58

Remark: The large errors obtained in the previous example were obtained because of a cancellation (or loss of significance) error which occurred while working on the first column. This introduced a small pivot, 0.0001, in the second column which amplified the error. We cannot always avoid cancellation errors but at least we can try to avoid the use of very small pivots. Dana Mackey (DIT) Numerical Methods 23 / 58

The partial pivoting strategy is the simplest scheme which eliminates the use of small pivots. For each column j let M i = max A ij j i and let i 0 be the smallest value of i (i > j) for which this maximum is achieved. Then interchange rows j and i 0. In other words, while working on column j, look for the entry with the largest absolute value below the pivot (A ij with i j). If this largest entry appears in more than one row, take the one closest to the pivot row, i 0, and swap it with the current pivot row, j. Exercise: Apply the technique of Gaussian elimination with partial pivoting to system (2) and verify if the error has improved. Dana Mackey (DIT) Numerical Methods 24 / 58

0.6667 0.2857 0.2000 2.867 0.3333 0.1429 0.5000 0.8333 0.2000 0.4286 0.4000 2.400 0.6667 0.2857 0.2000 2.867 ( R 2 0.4999 R 1 ) R 3 0.3 R 1 0 0.0001 0.6000 0.5997 (R 2 R 3 ) 0 0.5143 0.3400 3.260 0.6667 0.2857 0.2000 2.867 0 0.5143 0.3400 3.260 (R 3 0.0001944 R 2 ) 0 0.0001 0.6000 0.5997 0.6667 0.2857 0.2000 2.867 0 0.5143 0.3400 3.260 0 0 0.5999 0.603 Partial pivoting gives the solution x 3 = 1.001, x 2 = 7.000 and x 1 = 1.000. Dana Mackey (DIT) Numerical Methods 25 / 58

Exercise: Consider the following system 3x 1 + x 2 + 4x 3 x 4 = 7 2x 1 2x 2 x 3 + 2x 4 = 1 5x 1 + 7x 2 + 14x 3 8x 4 = 20 x 1 + 3x 2 + 2x 3 + 4x 4 = 4 1 Show, by direct substitution, that the exact solution is (1,-1,1,-1). 2 Solve the system using Gaussian elimination without pivoting and calculate the relative errors. 3 Solve the system using Gaussian elimination with partial pivoting and calculate the relative errors. Dana Mackey (DIT) Numerical Methods 26 / 58

Scaled partial pivoting Consider now the example 0.7x 1 + 1725x 2 = 1739 0.4352x 1 5.433x 2 = 3.271 The exact solution is x 1 = 20 and x 2 = 1. Partial pivoting would leave the equations unchanged since 0.7 is the largest entry in the first column. Gaussian elimination then gives hence x 2 = 1.001 and x 1 = 17.14. 0.7x 1 + 1725x 2 = 1739 1077x 2 = 1078 Dana Mackey (DIT) Numerical Methods 27 / 58

The large errors obtained in this case are due to the fact that the entries 0.7 and 0.4352 are not compared with the other coefficients in their corresponding rows. For instance, 0.7 > 0.4352 but 0.7 1725 < 0.4325 5.433 If, instead, we choose the pivot which is largest in magnitude relative to the other coefficients in the equation, we need to swap the rows and use 0.4352 as pivot 0.4352x 1 5.433x 2 = 3.271 0.7x 1 + 1725x 2 = 1739 which gives x 2 = 1.000 and x 2 = 20.000. Dana Mackey (DIT) Numerical Methods 28 / 58

Scaled partial pivoting Construct a scale vector as follows S i = max 1 j n a ij, for each 1 i n (S i is the largest absolute value in row i.) During step i of the Gaussian elimination (column i), let ( ) aji M i = max i j n and let j 0 be the smallest j for which this maximum occurs. If j 0 > i then we interchange rows i and j 0. Note: The scale vector S does not change during the Gaussian elimination process. S j Dana Mackey (DIT) Numerical Methods 29 / 58

Exercise: Consider the system 3x 1 + x 2 + 4x 3 x 4 = 7 2x 1 2x 2 x 3 + 2x 4 = 1 5x 1 + 7x 2 + 14x 3 8x 4 = 20 x 1 + 3x 2 + 2x 3 + 4x 4 = 4 Recall that the exact solution of this system is (1,-1,1,-1). Calculate the solution using Gaussian elimination with scaled partial pivoting and compare the errors with those obtained using (simple) partial pivoting. Dana Mackey (DIT) Numerical Methods 30 / 58

The LU decomposition method Consider the system of equations, written in matrix form where A M n,n is a square matrix and X, B M n,1. AX = B (5) The LU decomposition (or factorisation) of the matrix A consists of writing the matrix A as the product of a lower triangular matrix L and an upper triangular matrix U, such that A = LU. (Recall that a matrix is called upper triangular if all entries below the main diagonal are zero and lower triangular if all entries above the main diagonal are zero.) Dana Mackey (DIT) Numerical Methods 31 / 58

The procedure of solving the system of equations (5) using LU decomposition is the following. Step 1: Determining the LU decomposition The matrices L and U can be determined by writing (in the case of a 3 3 matrix, for example) a 11 a 12 a 13 1 0 0 u 11 u 12 u 13 A = a 21 a 22 a 23 = L U = l 21 1 0 0 u 22 u 23 a 31 a 32 a 33 l 31 l 32 1 0 0 u 33 and solving this system for the entries l ij and u ij. Dana Mackey (DIT) Numerical Methods 32 / 58

Step 2: Solving the LU decomposed system After the LU factorisation, system (5) becomes LUX = B which can be solved as two successive problems: (1) LY = B; (2) UX = Y. Each of these two systems is easy to solve since the matrix of coefficients is either upper or lower triangular. Dana Mackey (DIT) Numerical Methods 33 / 58

Example: Solve the following system using the LU decomposition method. x 1 + x 2 x 3 = 1 x 1 + 2x 2 2x 3 = 5 2x 1 + x 2 + x 3 = 10 Dana Mackey (DIT) Numerical Methods 34 / 58

The Inverse of a Matrix Let A M n,n. If there exists B M n,n such that AB = I = BA then we say A is invertible and B is an inverse of A. We write B = A 1. Note that if B is an inverse of A, then A is an inverse of B. Procedure to invert a matrix: We can use elementary [ ] row operations to invert a 2 2 invertible matrix. 1 2 Start with A = and augment the identity matrix to form the 3 4 2 4 matrix 1 2. 1 0. 3 4. 0 1 Now use elementary row operations to convert the left hand side to the identity matrix but performing the row operations across the entire row. Dana Mackey (DIT) Numerical Methods 35 / 58

When the left hand matrix becomes equal to the identity, the right hand matrix will be the inverse of A. Example: [ ] [ ] 1 2 1 0 R 2 R 2 3R 1 1 2 1 0 3 4 0 1 0 2 3 1 Hence the inverse of R 2 1 2 R 2 [ 1 0 2 1 0 1 3 2 1 2 [ ] [ 1 2 2 1 is 3 3 4 2 1 2 [ R 1 R 1 +R 2 1 0 2 1 0 2 3 1 This procedure can be used to invert any n n matrix, if it is invertible. ]. ]. ] Dana Mackey (DIT) Numerical Methods 36 / 58

0 1 2 Example: Find the inverse of the matrix A = 1 2 3. 1 2 2 2 2 1 The answer is A 1 = 1 2 2. Check that AA 1 = I = A 1 A. 0 1 1 Note: A square matrix is invertible if and only if it does not contain a zero on the main diagonal after it has been reduced to echelon form by elementary row operations. 1 2 3 For example, the matrix A = 3 1 5 has no inverse! 2 3 2 Dana Mackey (DIT) Numerical Methods 37 / 58

Solving systems of linear equations The inverse of a matrix gives a new method for solving a system of linear equations: If AX = B then X = A 1 B provided A is invertible. Exercise: Solve the following system of equations by inverting the matrix of coefficients. x + 2y + 3z = 1 x y + 4z = 4 2x 2y + z = 8 Dana Mackey (DIT) Numerical Methods 38 / 58

Properties of the LU decomposition method: 1: Not every matrix has an LU-decomposition. For example, 0 0 1 A = 0 1 0 1 0 1 cannot be written as the product of a lower triangular and upper triangular matrices. However, given any invertible matrix A, it is always possible to rearrange the rows of A so that the resulting matrix does have an LU-decomposition. A sufficient condition for a matrix A to have an LU decomposition is that Gaussian elimination can be performed on A without row interchanges. Dana Mackey (DIT) Numerical Methods 39 / 58

2. When a matrix has an LU-decomposition, that decomposition is not unique. Given the factorisation A = LU, if we define the new matrices L = LD and U = D 1 U where D is any invertible diagonal matrix, then L is lower triangular, U is upper triangular and A = L U so we found another LU-factorisation. In fact, once a matrix A admits an LU-factorisation, there are an infinite number of choices for the matrices L and U! Dana Mackey (DIT) Numerical Methods 40 / 58

3. The LU-decomposition is unique up to scaling by a diagonal matrix. This means that, if A admits two LU-factorisations, then we must have for some diagonal matrix D. A = L 1 U 1 = L 2 U 2 L 1 = L 2 D and U 2 = DU 1 Note: A product of upper (or lower) triangular matrices is also an upper (or lower) triangular matrix. Dana Mackey (DIT) Numerical Methods 41 / 58

4. If A admits a LU-factorisation then we can find matrices L and U such that L has 1 s along the diagonal. (Similarly, we can find a decomposition in which U has 1 s along the diagonal.) 5. The decomposition A = LU is unique if we require that L (or U) has 1 s along its diagonal! Dana Mackey (DIT) Numerical Methods 42 / 58

LU Factorization using Gaussian Elimination Consider the following example 1 4 5 1 4 5 1 4 5 A = 4 20 32 = 0 4 12 = 0 4 12 5 32 64 0 12 39 0 0 3 where the row operations used were as follows R 2 R 2 4R 1 R 3 R 3 5R 1 R 3 R 3 3R 2 On the other hand, the LU factorization of the matrix A is 1 4 5 1 0 0 1 4 5 A = 4 20 32 = 4 1 0 0 4 12 = LU 5 32 64 5 3 1 0 0 3 Dana Mackey (DIT) Numerical Methods 43 / 58

We notice that the upper triangular echelon form of the matrix A yields the matrix U in the LU decomposition. The matrix L is constructed as a lower triangular matrix with 1 s along the main diagonal and the negatives of the Gaussian elimination multipliers in the appropriate positions. An algorithm for constructing an LU decomposition for a matrix A can be constructed as follows. Initialize the matrix L as the identity matrix and U as the matrix A. Perform Gaussian elimination on the matrix U and insert the row multipliers into the appropriate positions in the matrix L. When the echelon form for U has been achieved, L will be a lower triangular matrix. The LU decomposition is complete. Dana Mackey (DIT) Numerical Methods 44 / 58

Example Find the LU factorization of the matrix 4 12 8 4 A = 1 7 18 9 2 9 20 20 3 11 15 14 Gaussian elimination: 1 0 0 0 4 12 8 4 L = 0 1 0 0 0 0 1 0 U = 1 7 18 9 2 9 20 20 0 0 0 1 3 11 15 14 1 0 0 0 4 12 8 4 L = 1/4 1 0 0 1/2 0 1 0 U = 0 4 16 8 0 3 16 18 3/4 0 0 1 0 2 9 11 R 2 1 4 R 1 R 3 1 2 R 1 R 4 3 4 R 1 Dana Mackey (DIT) Numerical Methods 45 / 58

1 0 0 0 4 12 8 4 L = 1/4 1 0 0 1/2 3/4 1 0 U = 0 4 16 8 0 0 4 12 R 3 3 4 R 2 3/4 1/2 0 1 0 0 1 7 1 0 0 0 4 12 8 4 R 4 1 2 R 2 L = 1/4 1 0 0 1/2 3/4 1 0 U = 0 4 16 8 0 0 4 12 3/4 1/2 1/4 1 0 0 0 4 R 4 1 4 R 3 which completes the LU decomposition. Exercise: Check that A = LU! Dana Mackey (DIT) Numerical Methods 46 / 58

Why does this work? Consider the matrices 1 0 0 1 0 0 M 1 = 2 1 0, M 2 = 0 1 0. 3 0 1 0 4 1 Exercise 1: Show that multiplying a matrix A by the matrix M 1 has the same effect as performing the row operations R 2 R 2 + 2R 1 and R 3 R 3 + 3R 1, while multiplication by the matrix M 2 has the same effect as performing the row operation R 3 R 3 + 4R 2. Exercise 2 Calculate the inverse matrices M 1 M1 1 M 1 2. 1, M 1 2 and the product Dana Mackey (DIT) Numerical Methods 47 / 58

Remarks 1 The Gaussian elimination procedure can be viewed as repeated multiplication by lower triangular matrices and the inverse of their product yields the matrix L in the LU decomposition. 2 Note that we have avoided row interchanges in the previous examples. If such row operations are needed (for example when performing Gaussian elimination with pivoting), the procedure is slightly different. (We won t cover this case now!) 3 The old procedure is referred to as LU decomposition by direct calculation or Doolittle s method. Dana Mackey (DIT) Numerical Methods 48 / 58

Iterative methods for systems of linear equations Linear systems arising from practical applications tend to be large and their coeffficient matrices will often be sparse (that is, only a small percentage of entries are non-zero). For such systems, iterative methods are generally more efficient than direct methods. Basic Idea: Iterative methods for systems of linear equations are similar to the algorithms for finding roots of nonlinear algebraic equations. For example, consider the linear system where A is an n n matrix and B is an n-vector. AX = B (6) Dana Mackey (DIT) Numerical Methods 49 / 58

This can be converted to a fixed-point problem: X = TX + C for some (new) n n matrix T, called the iteration matrix, and n-vector C. An iterative method for solving this problem consists of starting with a suitable initial guess, X (0) and then constructing the iterative sequence X (k) as follows X (k+1) = TX (k) + C, k = 1,2,... This algorithm is terminated when two successive vectors X (k) and X (k+1) are close enough in a sense which remains to be defined. Dana Mackey (DIT) Numerical Methods 50 / 58

Definition: Let A be an n n matrix. If we can write A = M N, where M and N are n n matrices and M is invertible, then the pair (M,N) is called a splitting of the matrix A. If we have a splitting A = M N in Equation (1) above then the system can also be written as MX = NX + B or X = M 1 NX + M 1 B = TX + C where we let T = M 1 N and C = M 1 B. Dana Mackey (DIT) Numerical Methods 51 / 58

1. The Jacobi method This method is based on splitting the coefficient matrix A as follows: A = D (L + U) where D is the diagonal part of A, while L and U are the strictly lower and upper triangular parts of A, respectively. (So, according to the definition above, we have M = D and N = L + U.) Note that L and U in this context are not related at all to the matrices L, U we would get from a LU-decomposition of A. Example: Find the Jacobi splitting for the matrix 5 1 2 A = 3 9 4 1 2 7 Dana Mackey (DIT) Numerical Methods 52 / 58

Recall, from the definition of a splitting, that the matrix M has to be invertible. In the case of the Jacobi splitting this means that the diagonal part of A has to be invertible so all diagonal elements have to be non-zero. If at least one diagonal element of A happens to be zero, the equations in the system must be reordered so that the zero entry is relocated to a non-diagonal position. The Jacobi method iteration formula X (k+1) = TX (k) + C where T = D 1 (L + U) and C = D 1 B. Dana Mackey (DIT) Numerical Methods 53 / 58

Exercise 1: Consider the following system of equations 5x 1 + x 2 + 2x 3 = 10 3x 1 + 9x 2 + 4x 3 = 14 x 1 + 2x 2 7x 3 = 33. Show that the exact solution is x1 = 1, x 2 = 3, x 3 = 4 and that the iteration sequence associated with the Jacobi method is given by x (k+1) 1 = 1 (k) [10 x 2 2x (k) 3 ] 5 x (k+1) 2 = 1 (k) [ 14 + 3x 1 4x (k) 3 ] 9 x (k+1) 3 = 1 (k) [ 33 x 1 2x (k) 2 ] 7 Find an approximate solution such that { } max x (k) 1 x1, x (k) 2 x2, x (k) 3 x3 0.01 Dana Mackey (DIT) Numerical Methods 54 / 58

Solution k x (k) 1 x (k) 2 x (k) 3 x (k) 1 1 x (k) 1 + 3 x (k) 1 4 0 0 0 0-1 3-4 1 2.0000-1.5556 4.7143 1.0000 1.4444 0.7143 2 0.4253-2.9841 4.5555-0.5746 0.0159 0.5555 3 0.7746-3.4384 3.9224-0.2254-0.4384-0.0775 4 1.1187-3.0407 3.8425 0.1187-0.0407-0.1575 5 1.0711-2.8904 4.0053 0.0711 0.1095 0.0053 6 0.9759-2.9787 4.0415-0.0240 0.0213 0.0415 7 0.9791-3.0264 4.0027-0.0208-0.0264 0.0027 8 1.0042-3.0081 3.9895 0.0042-0.0081-0.0105 9 1.0058-2.9939 3.9983 0.0058 0.0061-0.0017 Dana Mackey (DIT) Numerical Methods 55 / 58

2. The Gauss-Seidel method An obvious improvement in the Jacobi method is to use x (k+1) 1 and x (k+1) 2 as soon as they have been calculated rather than wait for the next iteration. This modification yields the Gauss-Seidel iteration algorithm: x (k+1) 1 = 1 (k) [10 x 2 2x (k) 3 ] 5 x (k+1) 2 = 1 (k+1) [ 14 + 3x 1 4x (k) 3 ] 9 x (k+1) 3 = 1 (k+1) [ 33 x 1 2x (k+1) 2 ] 7 Exercise 2: From the approximation equations above, show that the Gauss-Seidel method is based on the splitting A = (D L) U where D, U and L have the same meaning as in the Jacobi method. Dana Mackey (DIT) Numerical Methods 56 / 58

Exercise 3 : Redo Exercise 1 above using the Gauss-Seidel method. k x (k) 1 x (k) 2 x (k) 3 x (k) 1 1 x (k) 1 + 3 x (k) 1 4 0 0 0 0-1 3-4 1 2.0000-0.8889 4.7460 1.0000 2.1111 0.7460 2 0.2794-3.5718 3.7337-0.7206-0.5718-0.2663 3 1.2209-2.8080 4.0864 0.2209 0.1920 0.0864 4 0.9270-3.0627 3.9716-0.0730-0.0627-0.0283 5 1.0239-2.9794 4.0093 0.0239 0.0205 0.0092 6 0.9922-3.0067 3.9969-0.0078-0.0067-0.0030 We have obtained the same order of accuracy after only 6 steps therefore the Gauss-Seidel method is faster than the Jacobi method. Dana Mackey (DIT) Numerical Methods 57 / 58

Convergence properties for the Jacobi and Gauss-Seidel methods A matrix is called strictly diagonally dominant if, for each row, the absolute value of the diagonal element is strictly larger than the sum of absolute values of the other elements. If A is strictly diagonally dominant then both the Jacobi and Gauss-Seidel algorithms will converge for any choice of the initial vector approximation! Note that strict diagonal dominance is a sufficient condition: it does guarantee convergence but convergence might still occur even if the condition is not satisfied. Dana Mackey (DIT) Numerical Methods 58 / 58