Process Model Formulation and Solution, 3E4

Transcription

1 Process Model Formulation and Solution, 3E4 Section B: Linear Algebraic Equations Instructor: Kevin Dunn Department of Chemical Engineering Course notes: Dr Benoît Chachuat 06 October

2 Why solve linear algebraic equations? Consider the modelling of an isothermal batch reactor, where a reversible reaction A B is taking place Assumptions: Perfect mixing No change in volume due to the reaction Modelling goals: Predict the concentrations of species A and B in the reactor at steady state 2

3 Motivation and topics Problem formulation We consider linear algebraic equations of the general form: a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2 a n1 x 1 + a n2 x a nn x n = b n The a s and b s are constant coefficients The number of equations, n, is the same as the number of variables Equations that do not conform to the foregoing form are nonlinear! 3

4 Outline Mathematical background Gauss Elimination LU Decomposition and Matrix Inversion Gauss-Seidel Method 4

5 Matrix notation A is an m n matrix is a rectangular array of elements, a 11 a 12 a 1n a 21 a 22 a 2n A = a m1 a m2 a mn Row and column vectors are 1 n and m 1 matrices, respectively A matrix is said to be square when m = n Special types of square matrices: Diagonal matrix Upper triangular matrix Lower triangular matrix 2 D = 6 4 d d d nn U = 6 4 u 11 u 12 u 1n 0 u 22 u 2n 0 0 u nn L = 6 4 l l 21 l 22 0 l n1 l n2 l nn

6 Matrix operations Addition/subtraction: only possible between two m n matrices, 2 3 a 11 ± b 11 a 12 ± b 12 a 1n ± b 1n a 21 ± b 21 a 22 ± b 22 a 2n ± b 2n A ± B = a m1 ± b m1 a m2 ± b m2 a mn ± b mn Multiplication: only possible between m p and p n matrices, 2 3 px A B = a ik b kj k=1 5 Calculate the product of A = 4 B A? with B =» How about 6

7 Matrix operations (cont d) Inverse: A 1 is called the inverse of the square matrix A if A 1 A = A A 1 = I Theorem Invertible matrices can be characterized using the so-called determinant A has a unique inverse A 1 if and only if det A 0 (non-singular) Transpose: Transform rows into columns and vice-versa, a 11 a 12 a 1n a 11 a 21 a m1 a 21 a 22 a 2n A =, a 12 a 22 a m2 AT = a m1 a m2 a mn a 1n a 2n a mn 7

8 Solution of linear algebraic equations Matrices provide a concise notation for linear algebraic equations: a 11 a 12 a 1n b 1 a 21 a 22 a 2n A x = b, with: A =, b = b 2 a n1 a n2 a nn Theorem The system A x = b has a unique solution x if and only if det A 0 b n Non-singular case: Singular case: 8

9 Gauss elimination: foreword First appears in Chapter 8 of the Chinese mathematical test The Nine Chapters of the Mathematical Art, written as early as 150 BC! Invented independently by C F Gauss in his book Theory of Motion of Heavenly Bodies published in Principles and naive Gauss elimination 2 Pitfalls 3 Improved Gauss elimination: pivoting 4 Variant: Gauss-Jordan 9

10 The elimination of unknowns Consider the two linear equations: { a11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 Idea: Combine the equations algebraically to eliminate one of the unknowns 1 Multiplying eq 1 by a 21 and eq 2 by a 11 gives: { a21 a 11 x 1 + a 21 a 12 x 2 = a 21 b 1 a 11 a 21 x 1 + a 11 a 22 x 2 = a 11 b 2 2 Subtracting the resulting equations gives: from which one obtains: (a 21 a 12 a 11 a 22 ) x 2 = a 21 b 1 a 11 b 2, x 2 = a 21b 1 a 11 b 2 a 21 a 12 a 11 a 22, and x 1 = b 1 a 12 x 2 a 11 = a 22b 1 a 12 b 2 a 21 a 12 a 11 a 22 10

11 Gauss elimination: principles Extend and formalize the elimination of unknown approach to systems with more than 2 equations Two-step approach: (Case n = 3) 8 < : 8 < : 1 Forward elimination: convert A into upper triangular form 8 < a 11x 1 + a 12x 2 + a 13x 3 = b 1 a 21x 1 + a 22x 2 + a 23x 3 = b 2 a 31x 1 + a 32x 2 + a 33x 3 = b 3 a 11x 1 + a 12x 2 + a 13x 3 = b 1 a 22x 2 + a 23x 3 = b 2 a 33x 3 = b 3 2 Back substitution: solve upper triangular system 8 >< : a 11x 1 + a 12x 2 + a 13x 3 = b 1 a 22x 2 + a 23x 3 = b 2 a 33x 3 = b 3 >: x 3 = b 3 a 33 x 2 = b 2 a 23 x 3 a 22 x 1 = b 1 a 12 x 2 a 13 x 3 a 11 Main difficulty: How to formulate the upper triangular system? 11

12 Gauss elimination: procedure Step 1 of forward elimination: eliminate x 1 from rows 2,, n Initial system: a 11 x 1 + a 12 x 2 + a 13 x a 1n x n = b 1 a 21 x 1 + a 22 x 2 + a 23 x a 2n x n = b 2 a n1 x 1 + a n2 x 2 + a n3 x a nn x n = b n 1st pivot equation Multiply row 1 by a 21 a 11 : a 11 0 the 1st pivot element a 21 x 1 + a 21 a 11 a 12 x a 21 a 11 a 1n x n = a 21 a 11 b 1 Subtract the result from row 2, ( a 22 a ) ( 21 a 12 x a 2n a ) 21 a 1n x n = b 2 a 21 b 1 a 11 a 11 a }{{}}{{}}{{ 11 } a 22 a 2n b 2 12

13 Gauss elimination: procedure Step 1 of forward elimination: eliminate x 1 from rows 2,, n Repeat same steps for rows 3,, n: Multiply row 1 by a k1 a 11 Subtract the result from row k a 11 x 1 + a 12 x 2 + a 13 x a 1n x n = b 1 a 22 x 2 + a 23 x a 2n x n = b 2 a 32 x 2 + a 33 x a 3n x n = b 3 a n2 x 2 + a n3 x a nnx n = b n 13

14 Gauss elimination: procedure (cont d) Step 2 of forward elimination: eliminate x 2 from rows 3,, n Modified subsystem: a 22 x 2 + a 23 x a 2n x n = b 2 a 32 x 2 + a 33 x a 3n x n = b 3 a n2 x 2 + a n3 x a nnx n = b n 2nd pivot equation Multiply row 2 by a 32 a 22 : a 22 0 the 2nd pivot element a 32x 2 + a 32 a 22 a 23x a 32 a 22 Subtract ( the result ) from row 3, ( ) a 33 a 32 a 22 a 23 x a 3n a 32 a a 2n 22 }{{}}{{} a 33 a 3n a 2nx n = a 32 a 22 b 2 x n = b 3 a 32 a 22 b 2 } {{ } b 3 14

15 Gauss elimination: procedure (cont d) Step 2 of forward elimination: eliminate x 2 from rows 3,, n Repeat same steps for rows 4,, n: Multiply row 2 by a k2 a 22 Subtract the result from row k a 11 x 1 + a 12 x 2 + a 13 x a 1n x n = b 1 a 22 x 2 + a 23 x a 2n x n = b 2 a 33 x a 3n x n = b 3 a n3 x a nnx n = b n 15

16 Gauss elimination: procedure (cont d) After step n 1 of forward elimination: Upper triangular system: a 11 x 1 + a 12 x 2 + a 13 x a 1n x n = b 1 a 22 x 2 + a 23 x a 2n x n = b 2 a 33 x a 3n x n = b 3 a nn (n 1) x n = b n (n 1) Back substitution: Solve row n for x n, then back-substitute the result into row (n 1) to solve for x n 1, and so forth x n = b(n 1) n a (n 1) nn ; x i = b (i 1) i n j=i+1 a (i 1) ii a (i 1) ij x j, for i = n 1,, 1 16

17 Gauss elimination: example Use Gauss elimination to solve the linear algebraic equations x + y + z + w = 0 x 2y + 2z 2w = 4 x + 4y 4z + w = 2 x 5y 5z 3w = 4 17

18 Gauss elimination: algorithm statement Forward elimination: Back substitution: Loop: row = 1:n-1 Loop: i = (row+1):n factor = a[i,row] / a[row,row] Loop: j = row:n # or use row+1:n a[i,j] = a[i,j] - factor * a[row,j] End Loop b[i] = b[i] - factor * b[row] End Loop print(a) End Loop x[n] = b[n] / a[n,n] Loop: row = n-1:1 steps of -1 sums = b[row] Loop: j = row+1, n sums = sums - a[row,j] * x[j] End Loop x[row] = sums / a[row,row] End Loop Counting floating-point operations (FLOPs): 2 3 n3 + O(n 2 ) 1 2 n2 + O(n) Addition/subtraction and multiplication/division considered the same Forward elimination dominates as n becomes large n = 10 3 : Gauss elimination FLOPs ( 1 CPU sec) n = 10 4 : Gauss elimination FLOPs ( 1, 000 CPU sec) 18

19 Gauss elimination: sensitivity to round-off error Consider the following linear algebraic equations: { x x 2 = x x 2 = The first pivot element is a 11 = 00003: close to zero! Application of Gauss elimination yields: x 2 = 2 3, x 1 = x 1 very sensitive to subtractive cancellation: Significant ɛ rel Figures x 2 x 1 [%]

20 Improved Gauss elimination: pivoting Consider the following linear algebraic equations: 2x 2 + 3x 3 = 8 4x 1 + 6x 2 + 7x 3 = 3 2x 1 + x 2 + 6x 3 = 5 The first pivot element is a 11 = 0 Application of naive Gauss elimination results in a division by zero! Similar problem if pivot element close to zero Workaround Permute the system rows/columns so that the pivot elements are always the largest elements: this is known as pivoting Partial Pivoting: Switch rows so that the largest element becomes the pivot element Complete Pivoting: Switch rows and columns: rarely used Pivoting also minimizes round-off error 20

21 Improved Gauss elimination: partial pivoting Step 1 of forward elimination: eliminate x 1 from (n 1) rows Initial System: a 11 x 1 + a 12 x 2 + a 13 x a 1n x n = b 1 a 21 x 1 + a 22 x 2 + a 23 x a 2n x n = b 2 a n1 x 1 + a n2 x 2 + a n3 x a nn x n = b n Permute row 1 with row l 1 such that a l11 = max i {1,,n} a i1 Apply 1st step of forward elimination to permuted system: a l11x 1 + a l12x 2 + a l13x a l1nx n = b l1 a 22 x 2 + a 23 x a 2n x n = b 2 a 32 x 2 + a 33 x a 3n x n = b 3 a n2 x 2 + a n3 x a nnx n = b n 21

22 Improved Gauss elimination: partial pivoting (cont d) Step 2 of forward elimination: eliminate x 2 from (n 2) rows Apply same procedure as in Step 1 to the new subsystem: a 22 x 2 + a 23 x a 2n x n = b 2 a 32 x 2 + a 33 x a 3n x n = b 3 a n2 x 2 + a n3 x a nnx n = b n After step n 1 of forward elimination: Upper triangular system: a l11x 1 + a l12x 2 + a l13x a l1nx n = b l1 a l x a l x a l x 2n n = b l 2 a l x a l x 3n n = b l 3 a (n 1) l nn x n = b (n 1) l n 22

23 Improved Gauss elimination: example Use Gauss elimination with partial pivoting to solve the linear algebraic equations 2x 2 + 3x 3 = 8 4x 1 + 6x 2 + 7x 3 = 3 2x 1 x 2 + 6x 3 = 5 5x 1 + x 3 = 2 5x 2 3x 3 = 1 x 1 3x 2 + 2x 3 = 3 23

24 Variant: Gauss-Jordan method Main ideas: Pursue the elimination step until the identity matrix is obtained, rather than an upper triangular matrix Back substitution step no longer necessary, but FLOPs increased to n 3 + O(n 2 ) At each step: 1 Normalize the current row by dividing it by its pivot element 2 Eliminate an unknown from all other equations, not just the subsequent equations Use the Gauss-Jordan method to solve the linear algebraic equations 2x 2 + 3x 3 = 8 4x 1 + 6x 2 + 7x 3 = 3 2x 1 + x 2 + 6x 3 = 5 24

25 Gauss elimination: wrap-up Most fundamental methods for solving simultaneous linear algebraic equations Easy to automate by a computer program Extremely effective for solution of many engineering problems Solutions may be sensitive to round-off error and scaling Use partial pivoting strategy Make computations using extended precision Readings: Chapter 9 in: S C Chapra, and R P Canale, Numerical Methods for Engineers, McGraw Hill, 5th/6th Edition 25

26 Why do things differently? Idea: Modify the time-consuming elimination step so that it only involves operations on the coefficient matrix A Why? Provides an efficient means to solve equations with same coefficient matrix A, but many different right-hand-side vectors b Provides an efficient means to compute the matrix inverse A 1 Outline: 1 LU Decomposition 2 Matrix Inverse 3 Error Analysis and System Condition 26

27 LU decomposition for linear equation solving a 11 a 1n A x = b, with: A = a n1 a nn, b = Suppose A can be decomposed into the product L U where: l u 11 u 12 u 1n L = l 21 l 22 0, U = 0 u 22 u 2n, l n1 l n2 l nn 0 0 u nn b 1 b n Substitution steps: 1 Forward Substitution step: find y such that Ly = b 2 Back Substitution step: find x such that Ux = y Main difficulty: How to calculate a LU decomposition? 27

28 Performing LU decomposition Main variants: Doolittle decomposition: 1 s on the diagonal of L Crout decomposition: 1 s on the diagonal of U our focus The good news! Gauss elimination can be used to decompose A into L U Obtaining the upper triangular matrix U: a 11 a 1n A = a n1 a nn Gauss elimination U = a 11 a 12 a 1n a 22 a 2n a (n 1) nn with: a (k) ij = a (k 1) ij a(k 1) ik a (k 1) kk a (k 1) kj, i, j = k + 1,, n 28

29 Performing LU decomposition Main variants: Doolittle decomposition: 1 s on the diagonal of L Crout decomposition: 1 s on the diagonal of U our focus The good news! Gauss elimination can be used to decompose A into L U Obtaining the lower triangular matrix L: By application of the Gauss elimination procedure, we have: b 1 b 2 b n 1 = l 21 1 l n1 l n2 1 b 1 b 2 b (n 1) n why? 29

30 Performing LU decomposition (cont d) Obtaining the lower triangular matrix L: At the first step of Gauss elimination: b i = b i a i1 a 11 b 1, i 2 2 b b a b n 7 5 = 6 4 a 11 1 a n1 a b 1 b 2 b n 7 5 = a 21 a 11 a n1 a 11 At the second step of Gauss elimination: b i b 1 b 2 b n = a 21 a 11 1 a n1 a 11 a 32 a 22 a n2 a » b1 b b = b i a i2 a 22 b 2, i b 3 b n b 2 b n

31 Performing LU decomposition (cont d) Obtaining the lower triangular matrix L: And so on for n 1 steps: b 1 b 2 b n = a 21 a 11 1 a n1 a 11 a 32 a 22 a n2 a 22 1 a (n 2) n,n 1 a (n 2) n 1,n 1 {z } = L b 1 b 2 b (n 1) n

32 LU decomposition: procedure At each step k = 1,, n 1: 1 Calculate and store factor elements in kth column of L: l ik = a(k 1) ik a (k 1) kk, i = k + 1,, n 2 Perform Gauss elimination in kth column of A Remarks: a (k) ij = a (k 1) ij l ik a (k 1) kj, i, j = k + 1,, n Compact representation: store factor elements l ik in lower part of coefficient matrix Corresponding entries are converted to zeros anyway Pivoting: should be performed as in Gauss elimination to avoid division by zero and mitigate round-off errors (LU decomp) Keep track of pivoting by using an order vector p Complexity: number of FLOPs is 2 3 n3 + O(n 2 ) 32

33 LU decomposition for linear equation solving: example Perform an LU decomposition of the matrix A (without pivoting), then find the solution to the linear equations Ax = b, where: A = 1 3 1, and b =

34 LU decomposition: application to matrix inversion Idea: (do you see why?) Compute the inverse A 1 of a (nonsingular) square matrix A in a column-by-column fashion: The 1st column of A 1 can be obtained by solving the linear equations Ax = e 1, with e 1 = [ ] T The 2nd column of A 1 can be obtained by solving the linear equations Ax = e 2, with e 2 = [ ] T And so on Procedure: Perform LU decomposition of A only once Perform back- and forward-substitution steps n times, for each right-hand-side vector e 1,, e n Complexity: Number of FLOPs for matrix inverse is 4 3 n3 + O(n 2 ) 34

35 LU decomposition for matrix inversion: example Perform an LU decomposition of the matrix A, then calculate its inverse A 1, for: A =

36 Error analysis: did everything go well? Ways to check conditioning of the system Ax = b Multiply A 1 by A and assess whether the result is close to identity: A A 1? I Invert A 1 and assess whether the result is close to A: [A 1 ] 1? A Example: Hilbert matrix (MATLAB, single precision) 2 A A 1 = Goal: Obtain a single number as an indicator of ill-conditioning

37 Vector and matrix norms A norm is real-valued function that provides a measure of the size of vectors/matrices (Some) Vector norms: Euclidean norm x 2 = n k=1 x 2 k (Some) Matrix norms: Taxicab norm n x 1 = x k k=1 Uniform norm x = max 1 k n x k Frobenius norm n n A e = i=1 j=1 a 2 ij Column-sum norm n A 1 = max a ij 1 j n i=1 Row-sum norm A = max 1 i n n a ij j=1 2 Given A = , calculate A e, A 1 and A 37

38 Matrix condition number The condition number of a matrix A, denoted cond(a), is cond(a) = A A 1 cond(a) 1: The larger cond(a), the more ill-conditioned! For linear algebraic equations Ax = b, 1 b cond(a) b x x cond(a) b b Implications: If the a s are known to t-digit precision and cond(a) 10 c, then the solution x may only be valid up to t c digits! Example: Hilbert matrix (cont d) 2 A = /2 1/3 1/4 1/5 1/2 1/3 1/4 1/5 1/6 1/3 1/4 1/5 1/6 1/7 1/4 1/5 1/6 1/7 1/8 1/5 1/6 1/7 1/8 1/ , cond(a) = A 1 A

39 LU decomposition and matrix inversion: wrap-up Readings: Chapter 10 in: S C Chapra, and R P Canale, Numerical Methods for Engineers, McGraw Hill, 5th/6th Edition 39

40 Elimination vs iterative methods Ax = b, A IR n n, b IR n Elimination methods: (Gauss elimination, LU decomposition) Returns an estimate of the solution after a constant number of operations FLOPs = 2 3 n3 + O(n 2 ) Sensitive to round-off errors when cond(a) 1 Situation gets worse as n increases Iterative methods: (Gauss-Seidel, Jacobi) Starts with an initial guess that is successively refined to get closer to the solution An estimate of the solution is obtained upon convergence But, the method may not converge Well suited for large-scale systems of linear equations Number of FLOPs not known a priori 40

41 Jacobi and Gauss-Seidel methods: principles Idea of iterative methods: Create a sequence of estimates x (0), x (1),, x (k), converging to the solution x of the linear equations Ax = b (provided it exists) Notion of algorithms: x (k+1) = A (x (k) ), k 0; x (0) given 1 Need to pick-up an initial guess x (0) 2 Need to define the algorithm A 3 Need to specify a termination criterion Crucial questions: Does the algorithm converge? If so, how fast is the convergence? 41

42 Jacobi method: algorithm statement a 11 x 1 + a 12 x 2 + a 13 x a 1n x n = b 1 a 21 x 1 + a 22 x 2 + a 23 x a 2n x n = b 2 a n1 x 1 + a n2 x 2 + a n3 x a nn x n = b n At a given iteration k 0: Consider the 1st equation: a 11 x 1 + a 12 x 2 + a 13 x a 1n x n = b 1 Principle: Use current estimates x (k) 2, x (k) 3,, x n (k) to calculate a new estimate x (k+1) 1, x (k+1) 1 = b 1 a 12 x (k) 2 a 13 x (k) 3 a 1n x n (k) a 11 a 11 a 11 a 11 Caution: The x 1 update requires a

43 Jacobi method: algorithm statement (cont d) Next, consider the 2nd equation: a 21 x 1 + a 22 x 2 + a 23 x a 2n x n = b 2 Principle: use current estimates x (k) 1, x (k) 3,, x n (k) to calculate a new estimate x (k+1) 2, x (k+1) 2 = b 1 a 21 x (k) 1 a 23 x (k) 3 a 2n x n (k) a 22 a 22 a 22 a 22 Caution: The x 2 update requires a 22 0 Jacobi method: update rule For each variable x i, i = 1,, n, x (k+1) i = 1 a ii b i n a ij x (k) j, a ii 0 j=1 j i using the previous estimates x (k) 1,, x n (k) 43

44 Jacobi method: initialization and termination Initialization: If a good guess for the solution is not known, x (0) 1,, x n (0) can be chosen randomly! (0) Eg, x 1 = = x n (0) = 0 The initial guess plays no role in whether convergence takes place If the procedure converges from one point, it converges from any starting point Some initial guesses will have fewer iterations than others Termination: Stop when the relative change in the iterates becomes small, x (k) i x (k 1) i x (k) < ɛrel i, for each i = 1,, n i with ɛ rel i a user-specified tolerance or, stop when the x (k) x (k 1) x (k) ratio of norms is below ɛ rel 44

45 Jacobi method: initialization Good initial guesses for m i? 45

46 Jacobi method: example Use the Jacobi method to solve the linear algebraic equations 2x 1 + 3x 2 2x 3 = 7 x 1 + 2x 2 + x 3 = 9 x 1 + x 2 + 2x 3 = 8 from the starting point x (0) 1 = x (0) 2 = x (0) 3 = 0, and with termination tolerances 1%, 01%, 001% Tol 1% 01% 001% # Iter x x x

47 Jacobi method: example (cont d) 45 4 x1 x2 x Points Iterations 47

48 Jacobi method: algorithm statement Initialization: maxit: maximum number of iterations tol: user-specified tolerance x[:,1]: initial guess for the solution Loop: k = 1, maxiter Loop: i = 1, n subs = 0 Loop: j = 1, n Test: i ~= j? subs = subs + a[i,j] * x[j,k] End Test End Loop x[i,k+1] = ( b[i] - subs ) / a[i,i] rel_diff = norm(x[:,k+1] - x[:,k]) / norm(x[:,k+1]) End Loop Test: rel_diff <= tol? Exit End Test End Loop 48

49 Gauss-Seidel method: a variant Idea of the Gauss-Seidel method: Update each element x i, always using the latest available values of the other elements x 1,, x i 1, x i+1,, x n Jacobi vs Gauss-Seidel Jacobi: Defer use of the new values until the next iteration Gauss-Seidel: Use each new value immediately in the next equation Gauss-Seidel method: update rule For each variable x i, i = 1,, n, x (k+1) i = 1 i 1 b i a ij x (k+1) j a ii j=1 n j=i+1 a ij x (k), a ii 0, using the latest estimates x (k+1) 1,, x (k+1) i 1 and x (k) i+1,, x n (k) Using the best available estimates usually accelerates convergence Typically, 2 faster compared with the Jacobi method 49 j

50 Gauss-Seidel method: example 1 Use the Gauss-Seidel method to solve the linear algebraic equations 2x 1 + 3x 2 2x 3 = 7 x 1 + 2x 2 + x 3 = 9 x 1 + x 2 + 2x 3 = 8 from the starting point x (0) 1 = x (0) 2 = x (0) 3 = 0, and with termination tolerances 1%, 01%, 001% Tol 1% 01% 001% # Iter x x x

51 Gauss-Seidel method: example 1 (cont d) 4 35 x1 x2 x Points Iterations 51

52 Gauss-Seidel method: example 2 Use the Gauss-Seidel method to solve the linear algebraic equations 2x 1 3x 2 2x 3 = 7 x 1 + 2x 2 + x 3 = 9 x 1 + x 2 + 2x 3 = x1 x2 x3 50 Points Iterations 52

53 Jacobi and Gauss-Seidel methods: convergence Difficulty: convergence of iterative methods is not guaranteed! Sufficient condition for convergence: diagonal dominance 1 In each row, the magnitude of the diagonal entry is larger than or equal to the sum of the magnitudes of all the off-diagonal entries: n a ii a ij, for all i = 1,, n j=1 j i 2 In at least one row, the magnitude of the diagonal entry is strictly larger than the sum of the magnitudes of all the off-diagonal entries: a ii > n a ij, for at least one i = 1,, n j=1 j i 53

54 Jacobi and Gauss-Seidel methods: convergence (cont d) Which of the following linear algebraic systems Ax = b are diagonally dominant: A = 1 3 2, A = 1 3 2, A = Important remarks: 1 The diagonal dominance condition is rather conservative, yet easy to check 2 An iterative method may or may not convergence for a non-diagonally dominant system 3 Diagonal dominance can be obtained by simply pivoting the rows of the system 4 Pivoting the columns can also help, although this modifies the order in the solution vector x 54

55 Jacobi and Gauss-Seidel methods: relaxation Idea: Reduce the adaptation step in order for the algorithm to be less aggressive How? Mix a certain fraction of the full-step Jacobi/Gauss-Seidel with the previous estimate x (k+1) i = ωx (k+1) i ω = 1: Full-step Jacobi/Gauss-Seidel + (1 ω)x (k) i 0 < ω < 1: Under-relaxation achieves more reliable convergence 1 < ω 2: Over-relaxation speeds-up convergence ω > 2: Always diverges Relaxed Gauss-Seidel method: update rule For each variable x i, i = 1,, n, x (k+1) i = (1 ω)x (k) i + ω i 1 b i a ij x (k+1) j a ii j=1 n j=i+1 a ij x (k), a ii 0, j 55

56 Relaxed Gauss-Seidel method: example Use the relaxed Gauss-Seidel method to solve the linear equations 2x 1 3x 2 2x 3 = 7 x 1 + 2x 2 + x 3 = 9 x 1 + x 2 + 2x 3 = 8 from the starting point x (0) 1 = x (0) 2 = x (0) 3 = 0 and with ω = Points 4 3 x1 x2 x Iterations 56

57 The final words Alternative methods for linear equation solving: Method Stability Accuracy Gauss Elimination Affected by General (with partial pivoting) round-off error LU Decomposition General Affected by round-off error Gauss-Seidel May diverge Excellent General General Breadth of Application Diagonally dominant systems Readings: Chapter 112 in: S C Chapra, and R P Canale, Numerical Methods for Engineers, McGraw Hill, 5th/6th Edition 57