Solving Linear Systems of Equations

Transcription

1 1 Solving Linear Systems of Equations Many practical problems could be reduced to solving a linear system of equations formulated as Ax = b This chapter studies the computational issues about directly and iteratively solving Ax = b A Linear System of Equations Direct Solution for Ax = b LU-decomposition by Gaussian Elimination Gaussian Elimination with Partial Pivoting Crout Algorithm for A = LU Cholesky Algorithm for A = LL t (A is positive definite) Vector and Matrix Norms Matrix Condition Number (Cond(A) = A A 1 ) Sensitity and Error Bounds Iterative Solutions Jacobi method Gauss-Seidel method Other methods Applications

2 2 Overdetermined, Underdetermined, Homogeneous Systems a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2 a m1 x 1 + a m2 x a mn x n = b m Ax = b Definition: A linear system is said to be overdetermined if there are more equations than unknowns (m > n), underdetermined if m < n, homogeneous if b i = 0, 1 i m x + y = 1 x + y = 3 x + y = 2 (A) x y = 3 (B) x y = 1 (C) 2x + 2y = 4 x + 2y = 2 2x + y = 5 x y = 2 (A) has no solution, (B) has unique solution, (C) has infinitely many solutions (D) x + 2y + z = 1 2x + 4y + 2z = 3 (E) x + 2y + z = 5 2x y + z = 3 (D) has no solution, (E) has infinitely many solutions

3 3 Some Special Matrices A = [a ij ] R n n Diagonal if a ij = 0 i j Lower if a ij = 0 if j > i Unit lower if A is lower- with a ii = 1 Lower Hessenberg if a ij = 0 for j > i + 1 Band matrix with bandwidth 2k + 1 if a ij = 0 for i j > k A band matrix with bandwidth 1 is diagonal A band matrix with bandwidth 3 is tridiagonal A band matrix with bandwidth 5 is pentadiagonal A lower and upper Hessenberg matrix is tridiagonal A 1 = , A 2 = , A 3 = , A 4 =

4 4 A Direct Solution of Linear Systems A linear system 2x + y + z = 5 4x 6y = 2 2x + 7y + 2z = 9 A matrix representation Ax = b, or x y z = Solution using MATLAB A = [2, 1, 2; 4, 6, 0; 2, 7, 2]; b = [5, 2, 9] ; x = A\b (x = [1; 1; 2])

5 5 Elementary Row Operations (1) Interchange two rows: A r A s (2) Multiply a row by a nonzero real number: A r αa r (3) Replace a row by its sum with a multiple of another row: A s αa r + A s E 1 =, E 2 = 0 0 2, E 3 = 1 0 1, Example A = , E 1 A = , E 2 A = , E 3 A = Let L 3 = 0 1 1, L 2 = 1 0 1, L 1 = 2 1 0, A = Then L 3 L 2 L 1 A = = U (Upper ) A = (L 1 1 L 1 2 L 1 3 )U = LU, where L is unit lower

6 6 Computing An Inverse Matrix By Elementary Row Operations A = = I E 1 A = = E 1 I E 2 E 1 A = = E 2 E 1 I E 3 E 2 E 1 A = = E 3 E 2 E 1 I E 4 E 3 E 2 E 1 A = = E 4 E 3 E 2 E 1 I = A 1 where the elementary matrices are E 1 = 1 1 0, E 2 = 2 0 1, E 3 = 1 2 0, E 4 = 1 0 3

7 7 LU-Decomposition If a matrix A can be decomposed into the product of a unit lower- matrix L and an upper- matrix U, then the linear system Ax = b can be written as LUx = b The problem is reduced to solving two simpler triangular systems Ly = b and Ux = y by forward and back substitutions A = , L 3 = 0 1 1, L 2 = 1 0 1, L 1 = Then where L 3 L 2 L 1 A = = U A = L 1 1 L 1 2 L 1 3 U = LU A = = = LU If b = [5, 2, 9] t, then y = [5, 12, 2] t and x = [1, 1, 2] t B = , b = x = 1 2 1

8 8 Analysis of Gaussian Elimination Algorithm for i = 1, 2,, n 1 for k = i + 1, i + 2,, n β a ki /a ii if a ii 0 a ki β (β = m ki ) for j = i + 1, i + 2,, n a kj a kj β a ij The Worst Computational Complexity is O( 2 3 n3 ) 1 # of divisions are (n 1) + (n 2) = n(n 1) 2 2 # of multiplications are (n 1) 2 + (n 2) = n(n 1)(2n 1) 6 3 # of subtractions are (n 1) 2 + (n 2) = n(n 1)(2n 1) 6

9 9 The Analysis of Gaussian Elimination and Back Substitution to solve Ax=b 2 3 n n2 7n 6 R 1 : a 11 x 1 + a 12 x a 1n x n = b 1 R 2 : a 21 x 1 + a 22 x a 2n x n = b 2 R i : a i1 x 1 + a i2 x a in x n = b i R n : a n1 x 1 + a n2 x a nn x n = b n By Gaussian Elimination, we need C 1 = [ n k=1 (k+1)(k 1) + n k=1 k(k 1)] flops to reduce the above linear system of equations equivalent to the following upper triangular system R 1 : u 11 x 1 + u 12 x u 1n x n = c 1 R i : u ii x i + + u in x n = c i R n : u nn x n = c n We need C 2 = n k=1 (2k 1) = n 2 flops to solve an upper triangular linear system of equations Therefore, the total number of flops of solving Ax = b is summarized as C 1 + C 2 = [ n k=1 (k + 1)(k 1) + n k=1 k(k 1)] + n k=1 (2k 1) = 2 n k=1 k 2 n + n k=1 k n = 2n(n+1)(2n+1) 6 + n(n+1) 2 2n = n 6 (4n2 + 6n n ) = 2 3 n n2 7 6 n

10 10 PA=LU Let A R 4 4 and L 3 P 3 L 2 P 2 L 1 P 1 A = U by Gaussian Elimination with Partial Pivoting Denote L 1 = 0 α 1 α 2 α 3, L 2 = α α 5 0 1, L 3 = α 6 1 Without loss of generality, let P 2 = Then P 2 L 1 = (P 2 L 1 P 1 2 )P 2 = L (2) 1 P 2, where L (2) 1 = 0 α 3 α 2 α 1 Theorem: For any A R n n, there exists a permutation matrix P such that P A = LU, where L is unit lower and U is upper

11 11 Gaussian Elimination with partial Pivoting Algorithm for i = 1, 2,, n p(i)=i for i = 1, 2,, n 1 (a) select a pivotal element a p(j),i such that a p(j),i = max i k n a p(k),i (b) p(i) p(j) (c) for k = i + 1, i + 2,, n m p(k),i = a p(k),i /a p(i),i for j = i + 1, i + 2,, n a p(k),j = a p(k),j m p(k),i a p(i),j An example A = P 23 P 13 A = LU =

12 12 Matlab Codes for Gaussian Elimination with Partial Pivoting %% %% %% gaussppm - drive of Gaussian Elimination wit Partial Pivoting %% %% %% fin=fopen( gaussmatdat, r ); n=fscanf(fin, %d,1); A=fscanf(fin, %f,[n n]); A=A ; b=fscanf(fin, %f,n); X=gausspivot(A,b,n) %% %% %% gausspivotm - Gaussian elimination with Partial Pivoting %% %% %% function X=gausspivot(A,b,n) if (abs(det(a))<eps) disp(sprintf( A is singular with det=%f\n,det(a))) end C=[A, b]; % Gaussian Elimination with Partial Pivoting % for i=1:n-1 [pivot, k]=max(abs(c(i:n,i))); if (k>1) temp=c(i,:); C(i,:)=C(i+k-1,:); C(i+k-1,:)=temp; end m(i+1:n,i)= -C(i+1:n,i)/C(i,i); C(i+1:n,:)=C(i+1:n,:)+m(i+1:n,i)*C(i,:); end % Back substitution % X=zeros(n,1); %% Let X be a column vector of size n X(n)=C(n,n+1)/C(n,n); for i=n-1:-1:1 X(i)=(C(i,n+1)-C(i,i+1:n)*X(i+1:n))/C(i,i); end

13 13 Doolittle s LU Factorization Algorithm: A R n n, A = LU, L is unit lower-, U is upper- for k = 0, 1,, n 1 L kk 1 for j = k, k + 1,, n 1 U kj A kj k 1 s=0 L ks U sj for i = k + 1, k + 2,, n 1 L ik [ A ik k 1 s=0 L isu sk ] /Ukk A = = = LU

14 14 Crout s LU Factorization Algorithm: A R n n, A = LU, L is lower-, U is unit upper- for k = 0, 1,, n 1 U kk 1 for i = k, k + 1,, n 1 L ik A ik k 1 s=0 L is U sk for j = k + 1, k + 2,, n 1 U kj [ A kj k 1 s=0 L ksu sj ] /Lkk A = = = LU

15 15 Cholesky Algorithm Algorithm: A R n n, A = LL t, A is positive definite and L is lower- for j = 0, 1,, n 1 L jj [ A jj ] j 1 1/2 k=0 L2 jk for i = j + 1, j + 2,, n 1 L ij [ A ij j 1 k=0 L ikl jk ] /Ljj A = = = LL t

16 16 Vector Norms Definition: A vector norm on R n is a function that satisfies τ : R n R + = {x 0 x R} (1) τ(x) > 0 x 0, τ(0) = 0 (2) τ(cx) = c τ(x) c R, x R n (3) τ(x + y) τ(x) + τ(y) x, y R n Hölder norm (p-norm) x p = ( n i=1 x i p ) 1/p for p 1 (p=1) x 1 = n i=1 x i (Mahattan or City-block distance) (p=2) x 2 = ( n i=1 x i 2 ) 1/2 (Euclidean distance) (p= ) x = max 1 i n { x i } ( -norm)

17 17 Matrix Norms Definition: A matrix norm on R m n is a function that satisfies τ : R m n R + = {x 0 x R} (1) τ(a) > 0 A O, τ(o) = 0 (2) τ(ca) = c τ(a) c R, A R m n (3) τ(a + B) τ(a) + τ(b) A, B R m n Consistency Property: τ(ab) τ(a)τ(b) A, B (a) τ(a) = max{ a ij 1 i m, 1 j n} (b) A F = [ mi=1 nj=1 a 2 ij] 1/2 (Fröbenius norm) Subordinate Matrix Norm: A = max x 0 { Ax / x } (1) If A R m n, then A 1 = max 1 j n ( m i=1 a ij ) (2) If A R m n, then A = max 1 i m ( nj=1 a ij ) (3) Let A R n n be real symmetric, then A 2 = max 1 i n λ i, where λ i λ(a)

18 18 Matrix Condition Number Cond(A) = A A 1 For the matrix A = , A 1 = Cond 1 (A) = A 1 A 1 1 = 6 45 = 27 Cond (A) = A A 1 = 8 35 = 28 Cond 2 (A) = A 2 A 1 2 = =

19 19 Sensitivity and Error Bounds Let x and ˆx be the solutions to Ax = b and Aˆx = ˆb, respectively, where ˆx = x + x, ˆb = b + b Then b = Ax A x or x b A Thus x x x = A 1 b A 1 b A 1 b A b Cond(A) b b Similarly, suppose that (A + E) x = b and Ax = b, we have Further derivations to get x x E Cond(A) A ( x b x Cond(A) b + E ) A

20 20 Iterative Methods for Solving Linear Systems 1 Iterative methods are msot useful in solving large sparse system 2 One advantage is that the iterative methods may not require any extra storage and hence are more practical 3 One disadvantage is that after solving Ax = b 1, one must start over again from the beginning in order to solve Ax = b 2 Jacobi Method Given Ax = b, write A = C M, where C is nonsingular and easily invertible Then Ax = b (C M)x = b Cx = Mx + b x = C 1 Mx + C 1 b x = Bx + c, where B = C 1 M, c = C 1 b Suppose we start with an inital x (0), then x (1) = Bx (0) + c and x (k+1) = Bx (k) + c

21 21 Jacobi Iterative Method for Solving Linear Systems Suppose x is a solution to Ax = b, then x (1) x = (Bx (0) + c) (Bx + c) = B(x (0) x) x (k) x = B k (x (0) x) x (k) x B k x (0) x B k x (0) x x (k) x 0 as k if B = C 1 M < 1 For simplest computations, 1-norm or -norm is used The simplest choice of C, M with A = C M is C = Diag(A), M = (A C) Theorem: Let x (0) R n be arbitrary and define x (k+1) = Bx (k) + c for k = 0, 1, If x is a solution to Ax = b, then the necessary and sufficient condition for x (k) x is B < 1 Theorem: If A is diagonally dominant, ie, a ii > j i a ij for 1 i n A = C M and B = C 1 M, then n B = max 1 i n b ij = max a ij 1 i n j=1 j i a ii < 1 Let Example: Let Then A = C = B = C 1 M = , b = [ 11 12, M = ], x (0) = , [ 0 0, c = C 1 b = ] [ ] x (1) = Bx (0) + c = [ ], x (2) = [ ], x (3) = [ ]

22 22 Gauss-Seidel Method Given Ax = b, write A = C M, where C is nonsingular and easily invertible Jacobi: C = diag(a 11, a 22,, a nn ), M = (A C) Gauss-Seidel: A = (D L) U = C M, where C = D L, M = U for Jacobi Let x (0) R n be nonzero, then Cx (k+1) = Mx (k) + b implies that (D L)x (k+1) = Ux (k) + b Dx (k+1) = Lx (k+1) + Ux (k) + b x (k+1) 1 = 1 a 11 ( nj=2 a 1j x (k) j + b 1 ) x (k+1) ( i = 1 a ii i 1 j=1 a ijx (k+1) j n j=i+1 a ij x (k) ) j + b i, for i = 2, 3,, n The difference between Jacobi and Gauss-Seidel iteration is that in the latter case, one is using the coordinates of x (k+1) as soon as they are calculated rather than in the next iteration The program for Gauss-Seidel is much simpler

23 23 Convergence of Gauss-Seidel Iterations Theorem: If A is diagonally dominant, ie, a ii > j i a ij for 1 i n Gauss-Seidel iteration converges to a solution of Ax = b Then Proof: Denote A = (D L) U and let j 1 α j = a ji, β j = n a ji, and R j = i=1 i=j+1 β j a jj α j Since A is diagonally dominant, then a jj > α j + β j, thus R j = Therefore, R = Max 1 j n R j < 1 β j a jj α j < a jj α j a jj α j = 1 1 j n The remaining problem is to show that B = Max x =1 Bx R < 1, where B = C 1 M = (D L) 1 U Let x = 1 and y = Bx, then y = Max 1 i n y i = y k for some k Then y = Bx = (D L) 1 Ux (D L)y = Ux Dy = Ly + Ux y = D 1 (Ly + Ux) Then y k = 1 a kk ( k 1 i=1 a kiy i n i=k+1 a ki x i ) y = y k 1 a kk (α k y + β k x ) which implies that for all x with x = 1, Bx = y Thus, B = Max x =1 Bx R < 1 β k a kk α k = R k < 1

24 24 Example for Gauss-Seidel Iterations Example: A = = b = [ ] , x (0) = [ 0 0 ] = D L U Then Moreover, Thus x (1) 1 = 1 10 ( a 12x (0) 2 + b 1 ) = 11 x (1) 2 = 1 10 ( a 21x (1) b 2 ) = 098 x (2) 1 = 1 10 ( a 12x (1) 2 + b 1 ) = 1002 x (2) 2 = 1 10 ( a 21x (2) b 2 ) = x (1) = [11, 098] t x (2) = [1002, 09996] t x (3) = [100004, ] t