Solving Linear Systems 1 Review We start with a review of vectors and matrices and matrix-matrix and matrixvector multiplication and other matrix and

Transcription

1 Solving Linear Systems 1 Review We start with a review of vectors and matrices and matrix-matrix and matrixvector multiplication and other matrix and vector operations. Vectors operations and norms: Let x =[x 1 ;x ; ;x n ] t and y =[y 1 ;y ; ;y n ] t Then x + y =[x 1 + y 1 ;x + y ; ;x n + y n ] x =[ x 1 ; x ; ; x n ] t >>x = [1,,,-]; >>y = [,,,1]; >> x+y [ -] >> *x [,,,-10] Norms of vectors jjxjj 1 = jx 1 j + jx j + + jx n j jjxjj 1 = max(jx 1 j; jx j; ; jx n j) jjxjj = p x 1 + x + + x n jjxjj p =(x p 1 + xp + + xp n) 1=p Properties of norms (i) jjxjj 0 1

2 (ii) jjxjj =0, x =0 (iii) jjx + yjj» jjxjj + jjyjj (iv) jj xjj = j jjjxjj Matrix operations and norms Let us consider two matrices A = a 11 a 1m... B = a n1 a nm b 11 b 1m... : b n1 b nm The matrices A and B have n rows and m columns. If n = m, A and B are square matrices, otherwise they are rectangular matrices. We define the following matrix operations C = A + B, c ij = a ij + b ij C = A, c ij = a P ij n C = AB, c ij = a k=1 ikb kj AB = BA in general C = A t, c ij = a ji A = A t in general Determinant of square matrices for n =1,det(A) =a 11 for n =,det(a) =a P 11 a a 1 a 1 n for n>; det(a) = a j=1 ij( 1) i+j M ij where M ij is the determinant of the matrix obtained by deleting the i th row and j th column det(ab) =det(a)det(b) det(a t )=det(a) det( A) = n det(a) det(a) = 0 if and only if A is singular, i.e., A does not have aninverse matrix A has an inverse A 1 if and only if AA 1 = A 1 A = I

3 where I is the n n identity matrix. Direct methods for solving linear systems We start with an example x 1 +x x = 1 x 1 x +x + x = -1 x 1 +x 1x 8x = 1 x 1 + x + x = 1 The matrix formulation can be written as x 1 x x x = An upper-triangular system: x 1 x x x = 0 8 Use the backward substitution to solve the system starting from the last equation 1. Equation, x = 8 yields x =. Equation, x +x = with x = yields x =0. Equation, x + x = 0, yields x =1. Equation 1, x 1 + x + x = yields x 1 =1. A lower-triangular system x 1 x x x = 1 Use forward substitution to solve the system

4 1. Equation 1, x 1= yields x 1 = 1. Equation, x 1 + x = yields x = 1. Equation, x 1 + x = 1 yields x =0. Equation, x 1 +x x =, yields x =0 Matlab program for backward substitution function [x]=backward(u,b) %Input: U an upper triangular nxn matrix % b a vector of length n %Ouput: x a vector of length n solution to Ux = b % [n,m] = size(b); x(n,:) = b(n,:)/u(n,n); for i = n-1:-1:1 x(i,:) = (b(i,:) - U(i,i+1:n)*x(i+1:n,:)')/U(i,i); Matlab program for forward substitution function [x]=forward(l,b) %input: L a lower triangular n x n matrix % b a nxm matrix %output: x a n x m matrix solution of Lx=b % [n,m] = size(b); x(1,:) = b(1,:)/l(1,1); for i = :n x(i,:) = (b(i,:) - L(i,1:i-1)*x(1:i-1,:))/L(i,i);.1 Naive Gaussian elimination Gaussian elimination helps transform a general system Ax = b into an equivalent, i.e., have the same solution as Ax = b, upper-triangular system Ux = ~ b. using the operations (i) E j $ E j j E i ; j =0 (ii)e j $ E i (row interchanges) General form

5 a 11 x 1 + a 1 x + + a 1n x n = b 1 a 1 x 1 + a x + + a n x n = b.. a n1 x 1 + a n x + + a nn x n = b n into upper triangular system u 11 x 1 + u 1 x + + u 1n x n = ~ b 1 u x + + u n x n = ~ b... unnxn = ~ bn Let us start with an example: x 1 x x x = 1 Step 1 in Gaussian elimination: E 1 E E 1! E E E 1! E E + E 1! E x 1 x x x = 1 8 Step in Gaussian elimination: E 1 E E E! E E +E! E x 1 x x x = 1 1 Step : will make the a = 0 which is already the case. In practice we just do it to avoid checking for zero entries. Solve the resulting system using backward substitution

6 1. Equation, 1x = 1 yields x =1. Equation, x +1x = 1 yields x =0. Equation, x x x = yields x =. Equation 1, x 1 + x +x = yields x = 1. Matlab program for Gaussian elimination with no pivoting function [x]=gausselim(a,b) %Input: A is a nxn matrix % B is a nxm matrix %Output: x is nxm matrix solution of Ax=b % [n,m] = size(b); for i = 1:n for j = i+1:n c = A(j,i)/A(i,i); A(j,i:n) = A(j,i:n) - c*a(i,i:n); b(j,1:m) = b(j,1:m) - c*b(i,1:m); x = backward(a,b); Breakdown of Gaussian elimination: Consider the following example: x 1 x x = We write the system using an augmented matrix formulation and perform Gaussian elimination Step 1:

7 E 1 E E1! E E E 1! E Step : Cannot use E to eliminate x from equation E. E E! E will not eliminate x from E. Instead we interchange the rows E $ E to obtain E 1 E! E E! E Now we use the backward substitution algorithm to compute the solution x 1 =; x =1; x =1. Stability of Gaussian elimination and pivoting strategies Let us consider the example: ffl 1. 1+ffl. with exact solution x 1 =1; x =1. Set ffl =10 8 and apply Gaussian elimination with four significant digits in the decimal system to obtain E 1 1 ffl 1. 1+ffl E E 1 =ffl! E Rounding-off leads to the system ffl Solving leads to the incorrect answer x 1 = 0 and x =1.

8 Now if we interchange E 1 and E prior to applying Gaussian elimination we obtain E! E 1. E 1! E ffl ffl Gaussian elimination leads to the system. 0 1 ffl=. 1 ffl= Rounding-off yields Finally, we obtain the correct solution x 1 =1,x =1. In the first system the problem was caused by the division by a small number. Row interchanges that uses largest possible pivot and help reduce of effect of round-off errors. Next we will discuss several pivoting strategies. Partial pivoting We consider the general system in augmented form a 11 a 1 a 1n. b 1 a 11 a 1 a 1n. b..... b n a 11 a 1 a 1n. b 1 Step1: (i)select smallest p such that ja p;1 j = max n i=1 ja i;1j (ii) Interchange rows p and 1 (iii) Perform Gaussian elimination E j j E 1! E j, j = a j1 =a 11 j =; n. Step : (i) Select p such that ja p; j = max n j= ja jj, where a i;j are the updated entries. (ii) Interchange rows p and. 8

9 (iii) Perform elimination E j j E! E j, j = a j =a j =; ; n. Step i: (i) Select p such that ja p;i j = max n j=i ja jij (ii) Interchange rows p and i. (iii) Perform elimination E j j E i! E j, j = a ji =a ii j = i +1; n. Let us apply Gaussian with partial pivoting to the following system Step 1: since ja 1 j > ja 11 j; ja 1 j,interchange rows three and one to obtain Applying Gaussian elimination leads to E 1! E E1=! E E E1=! E Step : = 1=.. 1= 0 = =. 1= Since ja j > ja j,interchange rows two and three to obtain 1 0 = = Using Gaussian elimination to obtain.. 1= 0 = 1=.. 1= 9

10 E 1! E E1=! E E E1=! E 1 0 = =.. 1= 0 0 =.. = Now apply the backward substitution algorithm to computer solution x 1 =1,x =1,x = 1 and IP =[; 1; ]. Remark: Partial pivoting can be fooled by a wrong scaling of the equations. For instance, if we multiply the first equation in the system (??) by10 =ffl we obtain a new system =ffl. (1+e)10 =ffl. 1 Partial pivoting results in using the first equation to perform Gaussian elimination. This will lead again to incorrect results for small values of ffl =ffl 0 =ffl. (1 + e)10 =ffl. 1 (1+ffl)=ffl We may remedy this problem using scaled-column pivoting where we (i) compute the weights s i = max n j=1 ja i;jj from the original matrix At i th Gaussian step we (ii) compute the smallest integer p such that ja p;i j=s i = max n j=i ja j;ij=s j (iii) interchange rows p and i (iii) Apply Gaussian elimination Number of operations for Gaussian elimination solving Ax = b a 1;1 a 1;n. a 1;n a n;1 a n;n. a n;n+1 Number of operations in step i: Λ= : (n i)+(n i) Λ (n i +1)=(n i) Λ (n i +) 10

11 ± : (n i) Λ (n i +1) In order to compute the total number of operations we will need the following identities: P m j=1 1=m P m j=1 j = m(m +1)= P m j=1 j = m Λ (m +1)Λ (m +1)= (the proof is obtained using induction) There for the total number of Λ and / is nx i=1 (n i)(n i +)=(n + n) The total number of + and n 1 X i=1 n 1 X i=1 X n 1 (n +1) j=1 =(n +n n)= ß n =; for n>>1: X n 1 i + (n i)(n i +1)=(n n)= ß n =; n>>1 The number of operation for backward substitution is as follows P Λ= : n j j=1 =(n + n)= P ± : n 1 j j=1 =(n n)= The total solution cost: Λ= : n =+n n= + : n + n = n= Additional cost for partial pivoting: P n 1 j=1 j =(n 1)n= Remarks: (i) Cost of a comparison is comparable to cost of an addition (ii) Cost of a multiplication and division is greater that that of an addition( not true in matlab) (iii) Gaussian elimination is the most expensive part of the computation (iv) Backward substitution cost ß n = Λ = and ß n = ±. i=1 i 11

12 . Solving systems with multiple right-hand sides Consider m systems with same matrix and multiple right-hand sides. If all the right-hand sides are available at once we may use an augmented matrix formulation and apply Gaussian elimination at once and solve m upper-triangular systems as described below a 1;1 a 1;n. b 1;1 b 1;m a n;1 a n;n. b n;1 b n;m We illustrate the procedure with partial pivoting on the example Step 1 : interchange rows 1 and and apply Gaussian elimination = 1= 1=. 1 0 = 1= =. 0 0 = 8= 1=.. 0 Step : Apply Gaussian elimination with no row interchanges = 1= 1= = 1=. 1= = Step : no row interchanges 1

13 = 1= 1= = Using backward substitution we obtain = = X1 =[ 1; ; 0; 1] and X =[8=9; 19=9; 1=; =1] Inverse of matrix A To compute the inverse of a matrix A we need to solve the n systems AX = I, where I is the n n identity matrix. Now let us consider the augmented matrix [A : I] for the following example Step 1: interchange row 1 and and apply Gaussian elimination = 1= = 0 = = Step : interchange row and 1 0 = =. 1 0 = = 0 0 =. = 1 = We use backward substitution to solve the system and obtain A 1 = =1 1=1 =1 =1 =1 =1 1=1 = = 1

14 Computational Cost: Gaussian elimination and backward substitution: Λ= :n = n=, (n = +n =+n =) + :n = n =+n= Matrix factorization Theorem: If A is strictly diagonally dominant, i.e., ja i;i j > P n j=1;j=i ja i;jj, for i = 1; ; n, then the Gaussian elimination with pivoting is stable with respect to round-off errors. Example of a diagonally dominant matrix LU Factorization Theorem: If Gaussian elimination can applied to a matrix A without row interchanges then we can find a lower triangular matrix L and an upper triangular matrix U such that A = LU. First, we illustrate the procedure by the example: Step 1: no row interchanging Step : Step : 1

15 U = Thus, L = The reader may check that LU = A. Applications Solving Ax = b is equivalent to solving LUx = b by setting Ux = z and (i) first solve Lz = b (ii) then solve Ux = z A general algorithm for LU factorization Doolitle factorization where l i;i =1; i =1; ;n u 1;j = a 1;j ; j =1; ;n for k =1 n P 1 k 1 u k;k =(a k;k P l j=1 k;ju j;k )=l k;k k 1 l j;k =(a j;k P l i=1 j;iu i;k )=u k;k ; j =1; k 1 k 1 u k;j =(a k;j l i=1 k;iu i;j )=l k;k ;j = k +1; n Crout factorization Set u i;i =1,i P =1; n k 1 l k;k =(a k;k P l j=1 k;ju j;k )=u k;k k 1 l j;k =(a j;k P l i=1 j;iu i;k )=u k;k ; j = k +1; n k 1 u k;j =(a k;j l i=1 k;iu i;j )=l k;k ; j = k +1; n Program for LU factorization with no row interchanges function [L,U]=mylu(A) 1

16 %function computes the lu factorization of A with no pivoting %input: A %output: L a lower triangular matrix % U an upper triangular matrix % [n,m] = size(a); U(1,:) = A(1,:); d = zeros(1,n); L = diag(d+1,0); for i = 1:n-1 L(i+1:n,i) = A(i+1:n,i)/U(i,i); for j = i+1:n A(j,i:n) = A(j,i:n) - L(j,i)*U(i,i:n); U(i+1,i+1:n) = A(i+1,i+1:n);. LU factorization with row interchanges Theorem: If A is a non singular matrix then there exist a lower triangular matrix L, an upper triangular matrix U and a permutation matrix P (obtained by interchanging rows of the identity) such that LU = PA We illustrate the procedure on an example U = L = P = Step 1: interchange row 1 and and apply Gaussian elimination to obtain U = 1 0 1= = 0 L = = P = Step : interchange rows and and apply Gaussian elimination U = = L ~ = P = = 1= L = L ~ + I = = 1= 1 where LU = PA. In order to solve Ax = b we multiply the system by P to obtain PAx = Pb which leads to LUx = Pb. 1

17 We solve the system by solving (i) Lz = Pb (ii) Ux = z For instance, we solve Ax =[; 0; ] 0 = b by computing (i) Pb =[; ; 0] (ii) Lz = Pb yields z =[; ; 9=] 0 (iii) Ux = z leads to x =[ 1; 1; 1] 0. Matlab program for Lu factorization with partial pivoting function [L,U,P,mperm]=mylupiv(A) %PA=LU factorization with partial pivoting %input: matrix A %output: L lower triangular matrix % U upper triangular % P permutation matrix % mperm number of row interchanges % [n,m] = size(a); P = eye(n); U(1,:) = A(1,:); d = ones(1,n); L = zeros(n); mperm = 0; for i = 1:n-1 [mx,p]=max(abs(a(i:n,i))); p = p-1+i; if (p ~= i) mperm = mperm + 1; % rows interchanges in A tmp=a(p,i:n); A(p,i:n)=A(i,i:n); A(i,i:n)=tmp; % rows interchanges in L tmp= L(i,1:i-1); L(i,1:i-1)=L(p,1:i-1); L(p,1:i-1) = tmp; % rows interchanges in P tmp=p(p,:); P(p,:)=P(i,:); P(i,:)=tmp; U(i,i:n) = A(i,i:n); 1

18 % % compute l factors L(i+1:n,i) = A(i+1:n,i)/U(i,i); %Gaussian elimination for j = i+1:n A(j,i:n) = A(j,i:n) - L(j,i)*U(i,i:n); U(n,n) = A(n,n); L = L+diag(d); I. Determinant ofa The method of co-factors requires n! operations, for instance, for n = 100, we need operations. Thus, on a machine with 10 1 flops it will take10 18 years to compute the determinant of A. On the other-hand, using LU factorization it will take less than a second to compute the determinant as det(a) =det(l) Λ det(u) =(Π n i=1 l i;i)(π n i=1 u i;i) =Π n i=1 u i;i In general when PA = LU det(a) =( 1) m (Π n i=1 l i;i)(π n i=1 u i;i) =( 1) m (Π n i=1 u i;i), m is the numberofrow interchanges during Gaussian elimination. II. Solving Ax = b. The LU factorization is useful for solving Ax = b if all right-hand sides are not known at the beginning of the computation.. LU factorization of special matrices Symmetric positive matrices Theorem: If A is a symmetric positive matrix, i.e., x 0 Λ Ax > 0, for all x = 0, there exist a lower triangular matrix L such that A = L Λ L t Theorem: A symmetric matrix A is positive definite matrix if and only if det(a k ) > 0 for k =1; ; n, where A k = a 1;1 a 1;k..... a k;1 a k;k 18

19 Cholesky Factorization for i =1; ; n l i;i = q a i;i P i 1 k=1 l i;k l j;i =(a j;i P i 1 k=1 l j;kl i;k )=l i;i ; j = i +1; n Applications In order to solve Ax = LL t x = b we solve (i) Lz = b (ii) L t x = z Remarks (i) Cholesky factorization costs ß n = operations, i.e., half the cost of the regular LU factorization. (ii) It is stable with respect to round-off errors. (iii) Square root is expensive use A = LDL t, l i;i =1. Let us consider an example >[L,p] = chol(a) A = One may check that det(a 1 )=> 0 det(a )=8 => 0 det(a) = 00 > 0 and A 0 = A. Then A is a positive definite matrix with L = Matlab program for Cholesky factorization 19

20 function U = mychol(a) %function computes Cholesky factorization %input: matrix A positive definite % %output: U an upper matrix such as U'U=A % [n,m]=size(a); L(:,1) = A(:,1)/sqrt(A(1,1)); for i=:n nz = max([i-1,0]); nz L(i,i) =sqrt(a(i,i) - norm(l(i,1:nz),)^); for j=i+1:n L(j,i) = (A(j,i) - L(j,1:i-1)*L(i,1:i-1)')/L(i,i); U = L'; Tridiagonal matrices We consider the tridiagonal matrix where a i;j =0ifji jj > 1. a 11 a 1; 0 0 a 1 a a a a an 1;n 0 0 a n;n 1 a nn Gaussian elimination can be applied to factor this matrix as A = LU very efficientlyusing only O(n) operations where u 11 u 1; 0 0. l L =. 0 l u u.. 0 U = u un 1;n 0 0 l n;n u nn Algorithm: u 11 = a 11 ; u 1 = a 1 l 1 = a 1 =a 11 0

21 fori =; ; n u i;i = a ii l i;i 1 Λ u i 1;i ; u i;i+1 = a i;i+1 l i+1;i = a i+1;i =u i;i u n;n = a n;n l n;n 1 Λ u n 1;n Banded matrices Matlab program for banded matrices function [L,U]=mylu(A,nl,nu) %Lu factorization with no pivoting %input: A matrix % nl number of lower diagonals % nu number of upper diagonals %output: L lower triangular matrix % U upper triangular matrix % [n,m] = size(a); U = zeros(n); U(1,1:min(n,1+nu)) = A(1,1:min(1+nu,n)); d = zeros(1,n); L = diag(d+1,0); for i = 1:n-1 L(i+1:min(i+nl,n),i) = A(i+1:min(i+nl,n),i)/U(i,i); for j = i+1:min(i+nl,n) A(j,i:min(i+nu,n)) = A(j,i:min(i+nu,n)) - L(j,i)*U(i,i:min(i+nu,n)); U(i+1,i+1:min(i+1+nu,n)) = A(i+1,i+1:min(i+1+nu,n));. Sensitivity to errors in data and stability If we start with an error in the data for instance in A or b what is the error in x, where Ax = b. Let the exact problem be Ax = b. Let an approximate problem be A~x = b + ~ b and subtract the approximate problem from the exact problem to obtain A(x ~x) = ~ b 1

22 which can be written as (x ~x) =A 1 ~ b Taking the norm of both sides we obtain jj(x ~x)jj = jja 1 ~ bjj The relative error leads to jj(x ~x)jj jjxjj = jjbjjjja 1 ~ bjj jjxjjjjbjj» jjbjjjja 1 jjjj~ bjj jjxjjjjbjj Using and leads to jj(x ~x)jj jjxjj jja 1 ~ bjj»jja 1 jjjj ~ bjj: jjbjj = jjaxjj» jjajjjjxjj = jja 1 jjajj jj ~ bjj jjjbjj =»(A)jj ~ bjj jjbjj ; where»(a) =jja 1 jjajj is the condition number. Noting that ~ b = b A~x = r we can write jj(x ~x)jj jjxjj»»(a) jjrjj jjjbjj which gives a relationship between the residual and the relative error in the solution. Remarks 1.»(A)» 1. if»(a) >> 1 the system is ill-conditioned. if»(a) = O(1) the system is well conditioned. Small errors in b for ill-conditioned systems result in large errors in x Example: Consider the following matrix A =

23 jjajj 1 = ; jja 1 jj 1 =1: 10 leads to»(a) =jjajj 1 jja 1 jj 1 =:8 10 For instance if jj ~ bjj 1 =jjbjj 1 =10, then jjx ~xjj1 jjxjj1 ß :8: Now, if we assume the approximate problem to be (A + ~ A)~x = b + ~ b; then we can prove that jj(x ~x)jj jjxjj =»(A) ψ! jj Ajj ~ jjajj + jj~ bjj + O(jj Ajj ~ ) : jjjbjj Example in matlab >A = [ +10^(-1) 1; 1]; >b = [ + ^(-1); ]; >A b Iterative refinement Algorithm This algorithm is useful for ill-conditioned systems (»(A) >> 1). 1. Perform LU = PA with t digits. Solve LUx 0 = Pb. Compute r 0 = b Ax 0 using t digits (double precision). Solve LUx c = Pr 0. update x 0 as x 0 = x 0 + xc. if jjx c jj <fflstop else go to

24 Remarks: (i) If machine zero is u =10 d and» 1 (A) =10 q and using double precision to compute b Ax the x 0 has approximately min(d; k Λ (d q)) correct digits where k is the iteration number in the refinement algorithm Thus for instance when k =1,x has min(d; d q) correct digits when k =,x has min(d; (d q)) correct digits (ii) each iteration costs O(n ) operations not including the LU factorization. Example: Using the decimal system with t= significant digits we solve»»» 0:980:9 x1 0: = 0:090: x 0:10 Using the LU factorization we obtain»» :11 1:99 X 0 = X :1 1 = X :99 1 =» :00 :00