5 Solving Systems of Linear Equations

Transcription

1 106 Systems of LE 5.1 Systems of Linear Equations 5 Solving Systems of Linear Equations 5.1 Systems of Linear Equations System of linear equations: a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b a m1 x 1 + a m2 x a mn x n = b m

2 107 Systems of LE 5.1 Systems of Linear Equations Matrix form: A- given matrix, vector b - given; vector of unknowns x a 11 a 12 a 1n a 21 a 22 a 2n a m1 a m2 a mn x 1 x 2. = b 1 b 2 x n b m. or Ax = b (8) Suppose, m > n overdetermined system; Does a system thiswith systemore have equations a solution? than unknowns has usually no solution m < n underdetermined system; How a system many with solutions fewer equations does it have? than unknowns has usually infinitely many solutions m = n awhat system about withethe solution same number in this case? of equations and unknowns has usually a single unique solution we will deal only with this case now

3 108 Systems of LE 5.2 Classification 5.2 Classification Two main types of systems of linear equations: systems with full matrix most of the values are nonzero how storage store in a 2D it? array sparse matrix most of the matrix values are zero storing How in store a full such matrix matrices? system would be waste of memory different sparse matrix storage schemes Quite different strategies for solution of systems with full or sparse matrices

4 109 Systems of LE 5.2 Classification Problem Transformation Common strategy is to modify the problem (8) such that the solution remains the same modified problem more easy to solve What kind of transformations do not change the solution? Possible to multiply the both sides of the equation (8) with an arbitrary nonsingular matrix M without the change in solution. To check it, notice that the solution of MAz = Mb is: z = (MA) 1 Mb = A 1 M 1 Mb = A 1 b = x.

5 110 Systems of LE 5.2 Classification For example, M = D diagonal matrix, or M = P permutation matrix NB! Although, theoretically the multiplication of (8) with nonsingular matrix M does not change the solution, we will see later that it may change the numerical process of the solution and the exactness of the solution... The next question we ask: what type of systems are easy to solve?

6 111 Systems of LE 5.3 Triangular linear systems 5.3 Triangular linear systems If the system matrix A has a row i with a nonzero only on the diagonal, it is easy to calculate x i = b i /a ii ; if now there is row j where except the diagnal a j j 0 the only nonzero is at position a ji, we find that x j = (b j a ji x i )/a j j and again, if there exist a row k such that a kk 0 and a kl = 0 if l {i, j}, we can have x k = (b k a ki x i a k j x j )/a kk etc. Such systems easy to solve called triangular systems. With rearranging rows and unknowns (columns) it is possible to transform the system to Lower Triangular form L or Upper Triangular form U: l u 11 u 12 u 1n l L = 21 l , U = 0 u 22 u 2n l n1 l n2 l nn 0 0 u nn

7 112 Systems of LE 5.3 Triangular linear systems l u 11 u 12 u 1n l L = 21 l , U = 0 u 22 u 2n l n1 l n2 l nn 0 0 u nn Solving system Lx = b is called Forward Substitution : x 1 = ( b 1 /l 11 x i = b i i 1 j=1 l i jx j )/l ii Solving system Ux = b is called Back Substitution : x n = b n /u nn ) x i = (b i n j=i+1 u i jx j /u ii But how to transform an arbitrary matrix to a triangular form?

8 113 Systems of LE 5.4 Elementary Elimination Matrices 5.4 Elementary Elimination Matrices for a 1 0: 1 0 a 2 /a 1 1 a 1 a 2 = a 1 0. In general case, if a = a 1,a 2,...,a n and a k 0: M k a = m k m n 0 1 a 1. a k a k+1. a n = a 1. a k 0. 0, where m i = a i /a k, i = k + 1,...,n.

9 114 Systems of LE 5.4 Elementary Elimination Matrices Matrix M k is called also elementary elimination matrix or Gauss transformation The divider a k is called pivot (juhtelement) 1. M k is nonsingular = Why? being lower triangular and unit diagonal 2. M k = I me T k, where m = 0,...,0,m k+1,...,m n T and e k is column k of unit matrix 3. L k = (de f ) M 1 k = I + me T k. 4. If M j, j > k is some other elementary elimination matrix with multiplication vector t, then M k M j = I me T k tet j + me T k tet j = I me T k tet j, due to e T k t = 0.

10 115 Systems of LE 5.5 Gauss Elimination and LU Factorisation 5.5 Gauss Elimination and LU Factorisation apply series of Gauss elimination matrices from the left: M 1, M 2,...,M n 1, taking M = M n 1 M 1 we get the linear system: MAx = M n 1 M 1 Ax = M n 1 M 1 b = Mb upper triangular = easy to solve. The process is called Gauss Elimination Method (GEM)

11 116 Systems of LE 5.5 Gauss Elimination and LU Factorisation Denoting U = MA and L = M 1, we get that L = M 1 = (M n 1 M 1 ) 1 = M 1 1 M 1 n 1 = L 1 L n 1 is lower unit triangular (ones on the diagonal) = A = LU. Expressed in an algorithm:

12 117 Systems of LE 5.5 Gauss Elimination and LU Factorisation Algoritm 5.1. LU-factorisation using Gauss elimination method (GEM) do k=1,...,n-1 # cycle over matrix columns if a kk ==0 then stop # stop in case pivot == 0 do i=k+1,n m ik = a ik /a kk enddo do i=k+1,n do j=k+1,n # coefficient calculation in column k # applying transformations to a i j = a i j m ik a k j # the rest of the matrix enddo enddo enddo NB! In practical implementation: For storing m ik use corresponding elements in A (will be zeroes anyway)

13 118 Systems of LE 5.6 Number of operations in GEM 5.6 Number of operations in GEM Finding operation counts for Alg. 5.1: Replace loops with corresponding sums over number of particular operations: ( n 1 n 1 + i=1 j=i+1 n j=i+1 n k=i+1 2 ) = n 1 i=1 ((n i) + 2(n i) 2 ) = 2 3 n3 + O(n 2 ) used that m i=1 ik = m k+1 /(k+1)+o(m k ) (which is enough for finding the number of operations with the highest order) How Number many of operationsthere for forward are solving and backward a triangular substitution system? for L and U is O(n 2 )

14 119 Systems of LE 5.7 GEM with row permutations = the whole system solution Ax = b takes 2 3 n3 + O(n 2 ) operations 5.7 GEM with row permutations If pivot == 0 GEM won t work Row permutations or partial pivoting may help For numerical stability, also the pivot must not be small Example 5.1 Consider matrix A = non-singular but LU-factorisation impossible without row permutations

15 120 Systems of LE 5.7 GEM with row permutations But on contrary, the matrix 1 1 A = 1 1 has the LU-factorisation 1 1 A = 1 1 = = LU But, with what A being is wrong actually with singular matrixmatrix! A?

16 121 Systems of LE 5.7 GEM with row permutations Example 5.2. Small pivots Consider A = ε 1 1 1, with ε such that 0 < ε < ε mach in given floating point system (i.e. 1 + ε = 1 in floating point arithmetics) Without row permutation we get (in floating-point arithmetics): M = 1 0 1/ε 1 = L = 1 0 1/ε 1, U = ε /ε = ε 1 0 1/ε But then LU = 1 0 1/ε 1 ε 1 0 1/ε = ε A

17 122 Systems of LE 5.7 GEM with row permutations Using row permutation the pivot is 1; multiplier ε = M = 1 0 ε 1 = L = 1 0 ε 1, U = ε = in floating point arithmetics = LU = 1 0 ε = 1 1 ε 1 OK!

18 123 Systems of LE 5.7 GEM with row permutations Algoritm 5.2. LU-factorisation with GEM using row permutations do k=1,...,n-1 # cycle over matrix columns Find index p such that: # looking for the best pivot a pk a ik, k i n # in given column if p k then interchange rows k and p if a kk = 0 then continue with next k # skip such column do i=k+1,n m ik = a ik /a kk enddo do i=k+1,n do j=k+1,n a i j = a i j m ik a k j enddo enddo enddo # multiplier calculation in column k # transformation application # to the rest of the matrix

19 124 Systems of LE 5.7 GEM with row permutations As a result, MA = U, where U upper-triangular, OK but actually so far? M = M n 1 P n 1 M 1 P 1 M 1 is still not lower-triangular? any more, although it is still denoted by L but we have is it triangular? still triangular L knowing the permutations P = P n 1 P 1 in advance would give PA = LU, where L indeed lower triangular matrix

20 125 Systems of LE 5.7 GEM with row permutations But, do instead we really of row need exchanges to actually weperform can perform the rowjust exchanges appropriate explicitly? mapping of matrix (and vector) indexes We start with unit index mapping p = 1,2,3,4,...,n If rows i and j need to be exchanged, we exchange corresponding values pi and p j In the algorithm, take everywhere a pi j instead of a i j (and other arrays correspondingly) To solve the system Ax = b (8) How does the whole algorithm look like now? Solve the lower triangular system Ly = Pb with forward substitution Solve the upper triangular system Ux = y backward substitution The term partial pivoting comes from the fact that we are seeking for the best pivot only in the current column (starting from the diagonal and below) of the matrix

21 126 Systems of LE 5.7 GEM with row permutations Complete pivoting the best pivot is chosen from the whole remaining part of the matrix This means exchanging both rows and columns of the matrix PAQ = LU, where P and Q are permutation matrices. The system is solved in three stages: Ly = Pb; Uz = y and x = Qz Although numerical stability is better in case of complete pivoting it is rarely used, because more costly usually not needed

22 127 Systems of LE 5.8 Reliability of the LU-factorisation with partial pivoting 5.8 Reliability of the LU-factorisation with partial pivoting Introduce the vector norm: x = max i x i, x R n and corresponding matrix norm: Ax A = sup x R n, x R n. x Looking at the rounding errors in GEM, it can be shown that actually we find L, and U which satisfy the relation: where error E can be estimated: PA = LU E, E nε L U (9)

23 128 Systems of LE 5.8 Reliability of the LU-factorisation with partial pivoting where ε is machine epsilon In practice we replace the system PAx = Pb with the system LU = Pb. = the system we are solving is actually (PA + E) x = Pb for finding the approximate solution x. How far is it from the real solution? From the matrix perturbation theory: Let us solve the system Ax = b (10) where A R n n and b R n are given and x R n is unknown. Suppose, A is given with an error à = A + δa. Perturbed solution satisfies the system of linear equations (A + δa)(x + δx) = b. (11)

24 129 Systems of LE 5.8 Reliability of the LU-factorisation with partial pivoting Theorem 5.1. Let A be nonsingular and δa be sufficiently small, such that δa A (12) Then (A + δa) is nonsingular and δx x 2κ(A) δa A, (13) where κ(a) = A A 1 is the condition number. Proof of the theorem ( perturb-toest/perturb-proof.pdf) (EST) ( perturb-toest.pdf)

25 130 Systems of LE 5.8 Reliability of the LU-factorisation with partial pivoting Remarks 1. The result is true for an arbitrary matrix norm derived from the corresponding vector norm 2. Theorem 5.1 says, that in case of small condition number a small relative error in matrix A can cause small only aor small large? error in the solution x 3. If condition number is big, what everything can happen? can happen 4. It is not simple to calculate condition number but still, it can be estimated Combining the result (9) with Theorem 5.1, we see that: GEM forward error can be estimated as follows: δx x 2nεκ(A)G, where the coefficient G = L U A is called growth factor

26 131 Systems of LE 5.8 Reliability of the LU-factorisation with partial pivoting Conclusions If G is not large, then well-conditioned matrix gives fairly good answer (i.e. O(nε)). In case of partial pivoting, the elements of matrix L are 1. Nevertherless, examples, where with well-conditioned matrix A the elements of U exponentially large in comparison with the elements in A. But these examples are more like academic in practice rarely one finds such (and the method can be used without any fair)