Engineering Computation

Transcription

1 Engineering Computation Systems of Linear Equations_1 1

2 Learning Objectives for Lecture 1. Motivate Study of Systems of Equations and particularly Systems of Linear Equations. Review steps of Gaussian Elimination. Examine how roundoff error can enter and be magnified in Gaussian Elimination 4. Introduce Pivoting and Scaling as defenses against roundoff. 5. Consider what an engineer can do to generate well formulated problems.

3 Systems of Equations In the previous lecture we have tried to determine the value x, satisfying f(x)=0, where f is a scalar function or a system of non linear equations. : Now we try to obtain the values x 1,x,, x n, satisfying the system of linear equations A x=b with the same number n of equations and unknowns:

4 Systems of Equations Linear Algebraic Equations a 11 x 1 + a 1 x + a 1 x + + a 1n x n = b 1 a 1 x 1 + a x + a x + + a n x n = b.. a n1 x 1 + a n x + a n x + + a n x n = b n where all a ij 's and b i 's are constants. In matrix form: a11 a1 a1 a1n x1 b1 a1 a a an x b a1 a a an x = b an1 an an ann xn bn or simply n x n n x 1 n x 1 [A]{x} = {b} 4

5 Systems of Equations where A is the coefficients square matrix and b vector of independent terms. Many of the engineering fundamental equations are based on conservation laws. In mathematical terms, these principles lead to balance or continuity equations relating the system behavior with respect to the amount of the magnitude being modelled and the external stimuli acting on the system. b1 P b At each node, Balance = = Force Equilibrium F b1 V 1 H 1 H V 1 H 1 V 6 equations - 6 unknowns F x = 0 F y = 0 5

6 Systems of Equations Some special types of matrices: Symmetric matrix Identity matrix Diagonal matrix Upper triangular matrix 6

7 Systems of Equations Lower triangular matrix Banded matrix Half band width Sparse matrix: many zeros (See Matlab sparse and spconvert) All elements are null with the exception of those in a band centered around the main diagonal. This matrix has a band width of and has the name of tridiagonal. 7

8 Systems of Equations Graphic Solution: Systems of equations are hyperplanes (straight lines, planes, etc.). The solution of a system is the intersection of these hyperplanes.. Compatible and determined system.. Normal vectors are linearly independent.. Determinant of A is non-null.. Unique solution. 8

9 Systems of Equations. Incompatible system,. Linearly dependent vectors.. Null determinant of A.. No solution.. Compatible but undetermined system.. Linearly dependent vectors. Null determinant of A.. A infinite number of solutions.. Compatible and determined system. Linearly independent vectors.. Nonnull determinant of A, but close to zero.. There exists a solution but it is difficult to find precisely.. It is an ill conditioned system leading to numerical errors. 9

10 Gauss elimination Naive Gauss elimination method: The Gauss method has two phases: Forward elimination and backsustitution. In the first, the system is reduced to an upper triangular system: ( a 1 / a11) add pivot Pivot equation First, the unknown x 1 is eliminated. To this end, the first row (pivot row) is multiplied by -a 1 /a 11 and added to the second row. The same is done with all other succesive rows (n-1 times) until only the first equation contains the first unknown x 1. 10

11 Gauss elimination 1. Forward Elimination (Row Manipulation): a. Form augmented matrix [A b]: a11 a1 a1n x1 b1 a11 a1 a1n b1 a1 a an x b a1 a an b = ==> a a a x b a a a b n1 n nn n n n1 n nn n b. By elementary row manipulations, reduce [A b] to [U b'] where U is an upper triangular matrix: DO i = 1 to n-1 DO k = i+1 to n Row(k) = Row(k) +(-a ki /a ii ) * Row(i) ENDDO ENDDO 11

12 Gauss elimination flops 1. Forward Elimination (Row Manipulation): DO k = 1 to n 1 DO i = k+1 to n r = -A(i,k)/A(k,k) DO j = k+1 to n A(i,j)=A(i,j) +r A(k,j) ENDDO B(i) = B(i) +r B(k) ENDDO Complexity: Operation Counting, (+-) plus (*/) is 1 operation, ENDDO Inner loop: n j=k+1 1= n -(k +1)+1= n-k n i=k+1 Second loop: [ ] [ ] + (n -k) = ( + n) k (n k) = (n + n) (n + 1)k + k 1

13 Gauss elimination flops Operation Counting Forward Elimination : DO k = 1 to n 1 ENDDO m i 1... m m(m 1)(m 1)/6 m / + O(m = = + + = ) i = 1 n 1 k = 1 [( n n) ( n 1) k k ] n 1 n 1 n 1 = (n + n) 1 (n + 1) k + k k= 1 k= 1 k= 1 (n 1))(n) = (n + n)(n 1) (n + 1) + (n 1)(n)(n 1) 6 n = + O(n ) 1

14 Gauss elimination flops Back Substitution Solve upper triangular system [U]{x} = {b } x n = b' n / u nn DO i = n-1 to 1 by (-1) END Outer Loop: [ ] n 1 n 1 n (n i) = (1+ n) 1 i i= 1 i= 1 i= 1 u11 u1 u1 u1n x1 b1 0 u u u n x b 0 0 u un x = b u x b n nn n n Inner Loop: j= i+ 1 1 = n (i + 1) + 1 = n-i Total Effort, number of Floating Point Operations, (FLOPS), Gauss method: n / + O(n ) {Elim} + n / + O(n) {Back Subs} = n / + O(n ) = O(n ) = (1+ n)(n 1) (n 1)n n = + O(n) Ch&C 9..1 considers one (+-) and one (*/) as different flops: Naive Gaussian elimination, No of flops= n /+O(n ) {mult/div} + n /+O(n) {add/substr}= n / + O(n ) For backsubstitution: n flops Total effort: n / + O(n ) =O(n ) Solution by Cramer's Rule: n! flops. Never used with a computer 14

15 Pivoting strategy Floating point operations, mantissa with 4 digits, rounding 1st step 5 x 1 - x +x = 5-1 Elimination factors - x x +4 x = (-/5)=/5=0.6000e0 x 1 + x + x =7 7 -/5= e0 (Exact solution: [1,1,1].5000e e1.1000e1.000e1.5000e e1.1000e1.000e1 1st step -.000e1.1799e1.4000e1.799e e-.4600e1.4599e1.000e1.000e1.000e1.7000e e1.1400e1.500e1 15

16 Pivoting strategy.5000e e1.1000e1.000e e-.4600e1.4599e1 nd step.5000e e1.1000e1.000e e-.4600e1.4599e e1.1400e1.500e1 Elim factor:-.800e1/(-.1000e-)=800 x = 17490/ x = (.4599e e1.1001e1)/(-.1000e- )= =( )/(-0.001)= /(-0.001) = 6 x 1 = (-x +x )/5=( )/5=( )/5= =0/5 = 4 (Exact solution: [1,1,1]) Strategy (pivoting): reduce de abs value of elim factors e5.1749e = e = e5+.5e1.1749e5. Coefficients in the last equation, after step, have values 10 or 10 4 times greater than those in the original system.. Small rounding errors for them are big errors for other coefficients.. Relative SCALING of coefficientes can affect a lot!! 16

17 Pivoting strategy 1 Pivoting effect: reduce as much as possible the abs values of elimination factors, protecting against increase of the values of the coefficients. We still operate with 4 digits mantissa and rounding, and partial pivoting.5000e e e1.000e e1.1799e1.4000e1.799e1.000e1.000e1.000e1.7000e1 1st step as before nd step: permuting + elimination m =-(-0.001/.8)=.6e-4 x = 1 x = ( )/.8 = e e e1.000e e-.4600e1.4599e e1.1400e1.500e1 Look for a pivot (step i): greatest abs value coef in column i from diag ii-element x 1 = (-x +x )/5=(-1+)/5=1 (Exact solution: [1,1,1])!!!.6e e e e The exact solution isn t usually obtained, but pivoting is absolutely necessary to reduce rounding errors!! 17

18 Gauss elimination (example) Goals: 1. Best accuracy (i.e. minimize error). Parsimony (i.e. minimize effort) Possible Problems: A. Zero on diagonal term by zero. B. Many floating point operations (flops) cause numerical precision problems and propagation of errors. C. System may be ill-conditioned: det[a] 0. D. No solution or an infinite # of solutions: det[a] = 0. Possible Remedies: A. Carry more significant figures (double precision). B. Pivoting : choose elimination factor as small as possible in abs value (one case is when a diag elem is close to zero) C. Scaling: to reduce round-off error. 18

19 Gauss elimination (pivoting) PIVOTING A. Row pivoting (Partial Pivoting) Partial Pivoting, No Scaling: At each step i, find (along column i) max k a ki for k = i, i+1, i+,..., n Partial Pivoting, Scaling At each step i, find (along column i) max k a ki /size(matrix coef row k) for k = i, i+1, i+,..., n -> Move corresponding row to pivot position.. Avoids zero a ii. Protects against growing of coefficients & minimizes round-off. Maintains diagonal dominance.. Row pivoting does not affect the order of the variables.. Included in any good Gaussian Elimination routine. 19

20 Gauss elimination (pivoting) B. Column pivoting - Reorder remaining variables x j for j = i,...,n so get largest a ji Column pivoting changes the order of the unknowns, x i, and thus leads to complexity in the algorithm. Not usually done. C. Complete or Full pivoting Performing both row pivoting and column pivoting. (If [A] is symmetric, needed to preserve symmetry.) SCALING a. Express all equations (and variables) in comparable units so all elements of [A] are about the same size. b. If that fails, and max j a ij varies widely across the rows, replace each row i by: a ij <- a ij /max a ij. This makes the largest coefficient a ij of each equation equal to 1 and the largest element of [A] equal to 1 or -1 NOTE: Routines generally do not scale automatically; scaling can cause round-off error too! (Partial pivoting with scaling integrates pivoting and scaling to select the pivots) 0

21 LU factorization LU factorization - The LU factorization is a method that uses gaussian elimination techniques to transform the matrix A in a product of triangular matrices (L is lower triangular, U is upper traingular). This is specially useful to solve systems with different vectors b, because the same decomposition of matrix A can be used to evaluate in an efficient form, by forward and backward sustitution, all cases. The situation of a system with the same coefficient matrix and several right hand vectors is quite common in engineering. This is because usually coefficient matrix represent geometry/connectivity and physical properties in the balance equations represented in the system, while the independent right hand vectors represent actions on the system, like mechanical loads corresponding to different loading hypothesis. A pure factorization A=L U is possible in some cases, but in general a factorization with pivoting is the method to apply. It is always possible if matrix A is nonsingular, and protects against rounding errors. In practice, pivoting is needed: P A = L U (P is a square matrix that results of row permutations of the Identity matrix) 1

22 LU factorization P A=LU A X=B Initial system naming U X=Y Back substitution of Y: U X=Y Upper triangular system solve for X= (x n,,x 1 ): solution P A X=P B, L U X=P B L Y=P B, Lower triangular system, solve for Y= (y 1,...,y n )

23 LU factorization LU decomposition is very much related to Gauss method, because the upper triangular matrix U appears in the LU decomposition. Surprisingly, during the Gauss elimination procedure, this matrix L is obtained, but one is not aware of this fact. The factors we use to get zeroes below the main diagonal are elements of this matrix L. Let us consider first an example of LU factorization without pivoting: 4 A= m 1 =-1/ m 1 =-1/ Frobenius matrices M 1 = M =1-1/ 1 0-1/ = 4 0 1/ 1 0-1/ / 1 0-1/ 0 M 1 A (1) (=A) = A (), det(m 1 )=1 m =-((-1/)/(1/))=1 4 = 0 1/ 1 M A () = A () = U, det(m )=1 M M 1 A= U ; M M 1 A = U ; A = U = u 11 u u, det(a)=det(u) (no pivoting) A= M 1-1 M -1 U = L U

24 LU factorization (no pivoting) M 1 = -1/ 1 0 M -1 1 = -1/ / / 0 1 M = M -1 = M 1-1 M -1 = 1/ 1 0 1/ -1 1 Coefficients of L under the diagonal: elimination factors with changed sign In general, P A = L U (using pivoting strategies), and det(a)=det(u) (-1) number of row permutations in P P A=L U number of FLOPS : O(n /), 1 flop=1(*/) plus 1(+-) 4

25 LU Gaussian factorization, nonscaled partial pivoting P A= Rows indices, initial situation Each step: substeps: 1.- Pivot selection and possible row permutation,.- Elimination. 1st step: 1st column, 7=max( 1, 4, 7), pivot element, permute 1st and rd rows pivot element m 1 =-4/7 m 1 =-1/ /7 /7 /7 +1/7 6/7 16/7. nd step: st column, 6/7=max( /7, 6/7 ), pivot element, permute nd and rd rows /7 6/7 16/7 +4/7 /7 /7 pivot element m =-(/7)/(6/7)=-1/ when permuting, full rows are permuted /7 6/7 16/7 +4/7 +1/ L U= L= P is orthogonal: P T =P -1 U /7 16/ / /7 1/ 1 P= P A=L U, check! 5

26 A= LU Gaussian factorization, nonscaled partial pivoting Rows indices. 1st step: Substep 1: permute 1st and rd rows P 1 = Substep : eliminate with Frobenius matrix M 1 = -4/ / nd step: Substep 1: permute nd and rd rows P = Substep : eliminate with Frobenius matrix M = / 1 M P M 1 P 1 A=U, after some algebra: PA=LU, L =1, U =7 (6/7), P =(-1) permut, A =1 A 1 =M 1 P 1 A /7 /7 0 6/7 16/7 A =M P A 1 = =M P M 1 P 1 A=U /7 16/7 0 0 L= 1/7 1 0 P= 4/7 1/

27 LU factorization, nonscaled partial pivoting Row indices 1 A st step: m 1 =-4/7 m 1 =-1/ /7-1/7 /7 +1/7 /7 16/7. nd step (permutation isn t needed): /7-1/7 /7 m =(-/7)/(-1/7)=/1 +1/7 -/1 6/ = 4/ /7 -/ /7 / /1 P A L U 7

28 LU factorization, SCALED partial pivoting Row Indices i 1 A Row Sizes d st step: max( 1 /, 4 /6, 7 /1)=4/6, pivot: 4, nd row Row Indices i Row Sizes d 6 1 m 1 =-1/4 m 1 =-7/ /4 /4 / +7/4 1/4-11/ d 6 1. nd step (choose pivot): max( /4 /, 1/4 /1=1/48, pivot: 1/4, rd row i 1 d /4 1/4-11/ +1/4 /4 / m =(-/4)/(1/4)=-/ /4 1/4-11/ +1/4 /1 6/ = 7/ /4 / /4-11/ 0 0 6/1 P A L U 8

29 Matlab LU factorization (nonscaled partial pivoting) >> A1=[1 ;4 5 6;7 1 5] A1 = >> P*A1 ans = >> L*U ans = >> >> [L,U,P]=lu(A1) L = U = P = >> Pivoting 1st example revisited: A=[5,-,1;-,1.799,4;,,] [L1,U1,P1]=lu(A) 9

30 LU decomposition (Complexity) Basic Approach Consider A x = b a) Gauss-type "decomposition" of A into L U n / flops A x = b becomes L U x = b; let U x y b) First solve L y = b for y, lower triangular n / flops c) Then solve U x = y for x by back substitution n / flops Practice: USE PIVOTING!!: P A=L U Ax=b ->> P A x = P b; L U x=p b; L y=p b ->> y, U x=y ->> x, the solution 0

31 LU decomposition LU Decomposition Variations Doolittle L 1 U General A=L 1 U [ ] 1 0 L 1 = a ij 1 U= a a 0 a [ ] 11 ij Crout L U 1 General A=L U 1 a 0 L= a ij a [ ] 11 nn 1 a [ ] ij U 1 = 0 1 Cholesky L L T Positive Definite Symmetric A=L L T a 0 L= a ij a [ ] 11 nn Cholesky works only for Positive Definite Symmetric matrices!! nn. There are other important matrix factorizations. One of them is A=Q R, Q orthogonal and R, upper triangular. It is used in robust matrix eigenvalue calculations. 1

32 Cholesky factorization PROPERTY: A=L L T, with l ii 0, i=1,,n (L: Lower triangular matrix) is a necesary and sufficient condition for A to be a positive definite matrix (x T A x >0 x 0 (vector), x T A x =0 only if x=0 function [L,indic] = Choles(A); % Input: A Symmetric and Positive Definite matrix, % Output: % L: Lower triangular matrix, L*L'=A % indic: =0 if no problems, =1 if problems indic=0; n = size(a,1); L = zeros(n); for k=1:n s=0;for p=1:k-1,s=s+l(k,p)^;end; aux1=a(k,k)-s; if (aux1 <0 abs(aux1)<=1e-1), indic=1,aux1,return end L(k,k)=sqrt(aux1); for i=k+1:n s=0;for p=1:k-1,s=s+l(i,p)*l(k,p);end L(i,k)=(A(i,k)-s)/L(k,k); end % for i end % for k A= L l l 1 l 0 l n1 l n l nn L T l 11 l 1 l n1 0 l l n 0 0 l nn In A=L L T, identify element to element at both sides, by columns for k=1:n l kk =sqrt(a kk - p=1:k-1 (l kp ) ) for i=k+1 : n l ik =(a ik - p=1:k-1 l ip l kp )/l kk end end No. of flops=o(n /6)

33 Matrix Inversion Definition of a matrix inverse: A A -1 = I, A x = b => x = A -1 b First Rule: Don t do it. (more effort and possibly numerical unstabilities) Consider: A A -1 = I, n systems with the columns i k (k=1,,n) of I as RHS vectors 1) Create the augmented matrix: [A I ] ) Apply simultaneously elimination under and over the diagonal (Gauss-Jordan elimination): at the end one gets: [ I A -1 ]. This type of elimination can also be applied to a system Ax=b, but the number of flops is higher than performing gaussian elimination plus back substitution, and should nt be used.. Nevertheless, Gauss-Jordan method flops to calculate the inverse A -1 takes O(n ) and it is adequate instead of solving n systems with RHS i k (k=1,,n) by L U method.

34 Matrix Inversion. Gauss-Jordan method 4 A= m 1 =-1/ m 1 = 1/ / 11/ 0 8/ 1/ -1/ 1 0 1/ 0 1 A -1 =?, A A -1 =I linear systems in parallel. Same coeff matrix m 1 =-/(16/)=-/8 m =-(8/)/(16/)=-1/ m 1 =(-1/8)/(5/)=-1/0 m =(-11/)/(5/)=-/15 0 1/8 0 16/ 11/ 0 0 5/ / / 9/8 -/8 0-1/ 1 0 1/ -1/ 1 /5 /0-1/0-16/15 6/15 -/15 1/ -1/ 1 Full 1st row * (1/) Full nd row * (/16) Full rd row * (/5) /5 1/0-7/0-1/5 1/40-11/40 1/5-1/5 /5 A -1 Partial pivoting, scaled or not, should be applied. At each step/column search the pivot under the diagonal at each column/step, for not breaking the zeros structure 4

35

36