Linear Algebraic Equations
Linear Equations: a + a + a + a +... + a = c 11 1 12 2 13 3 14 4 1n n 1 a + a + a + a +... + a = c 21 2 2 23 3 24 4 2n n 2 a + a + a + a +... + a = c 31 1 32 2 33 3 34 4 3n n 3...... a + a + a + a +... + a = c m1 1 m2 2 m3 3 m4 4 mn n m m equations in n unknowns Where a ij are constant coefficients, c j are constants.
Brief Review of Matrices [ A] a11 a12... a1 n a a... a............ am1 am2... amn 22 2n = [m*n] matri İf m=n square matri If m=1, [1*n] row vector and if n=1 [m*1] column vector [ ] C c1 c2... cn b b.. b = 1 b 2 = m
Matri is symmetric if: a ij =a ji Matri is diagonal if a i j=0 for i j for all i Matri is identity Matri, I, if a i j=0 for i j for all i and a ii =1 Upper and Lower Triangular Matrices Banded matri Transpose of a matri
Matri Operations [A]=[B] if both matrices have the same m rows, n columns and a ij =b ij for all i,j Addition and Subtraction: [C]=[A]+[B], c ij =a ij +b ij [C]=[B]+[A] = [A]+[B] Commutative ([A]+[B])+[C]= [A]+([B]+[C]) Associative
Multiplication by a constant: k[a]=g g ij =ka ij Multiplication of Matrices: [C]= [A] [B] m*l m*n n*l c ij = Σa ik b kj (AB)C=A(BC) (Associative) A(B+C)=AB + AC (Distributive) AB BA (is not commutative)
Trace of a Matri: Augmentation: [ ] tr A n = a i= 1 ii Matri A augmented by an identity Matri: [ ] a a a 11 12 13 = 22 23 31 32 33.1 0 0 AI, a a a.0 1 0 a a a.0 0 1 AX Matri A augmented by B: = B a a a. b A, B a a a. b [ ] 11 12 13 1 = 22 23 2 a a a. b 31 32 33 3
Inverse of a Matri A 1 A 1 If (Inverse of Matri A)İs such that 1 1 A A= AA = I AX = B Multiply both sides by A 1 Since 1 A A= I 1 1 A AX = A B X = 1 A B
DETERMINANT (Defined for square matrices only) 22 a a a A a a a [ ] 11 12 13 = 22 23 a a a a 31 32 33 a 11 12 D = = a11a22 a12a21 a21 a22 Determinant of Matri A a a a 11 12 13 D= a a a 22 23 a a a 31 32 33 33 or a a a a a a a a a D = a a a = a a + a 11 12 13 22 23 23 22 22 23 11 12 13 a32 a33 a31 a33 a31 a32 31 32 33 a a a a a a a a a a a a D= a a a = a a + a 11 12 13 22 23 12 13 12 13 22 23 11 31 a32 a33 a32 a33 a22 a23 31 32 33 a a a
Gauss Elimination a + a + a + a +... + a = c 11 1 12 2 13 3 14 4 1n n 1 a + a + a + a +... + a = c 21 2 2 23 3 24 4 2n n 2 a + a + a + a +... + a = c 31 1 32 2 33 3 34 4 3n n 3...... a + a + a + a +... + a = c n1 1 n2 2 n3 3 n4 4 nn n n n equations in n unknowns
When n is small: (Graphical Method) a + a = b 11 1 12 2 1 a + a = b 21 2 2 2 Solve for 2 a = + b 11 2 1 1 a12 a = + b 21 2 a22 = slope* + intercept 2 1 1 3 + 2 = 18 + 2 = 2 3 = + 9 2 1 = + 1 2 2 1 2 1
No Solution Infinite Solutions Ill Conditioned
When n is small Cramer s Rule a11 a12 a13 1 b1 a a a = b 22 23 2 2 a31 a32 a 33 3 b 3 1 = b a a 1 12 13 b a a 2 23 b a a 1 32 33 D D is the determinant of the coefficient Matri A
When n is small: Elimination of the unknowns a + a = b 11 1 12 2 1 a + a = b 21 2 2 2 a a + a a = a b 21 11 1 12 2 21 1 a a + a a = a b 11 1 12 2 1 a22a112 a21a122 = a11b2 a21b1 2 = a b a a a b a a 1 21 1 22 11 12
Naive Gauss Elimination a + a + a + a +... + a = c 11 1 12 2 13 3 14 4 1n n 1 a + a + a + a +... + a = c 21 2 2 23 3 24 4 2n n 2 a + a + a + a +... + a = c 31 1 32 2 33 3 34 4 3n n 3...... a + a + a + a +... + a = c n1 1 n2 2 n3 3 n4 4 nn n n Step 1: eliminate 1 from the second upto nth equation a21 Multiply the first equation by: a21 a11 a11 a a a a a a211 + a122 + a133 + a144 +... + a1 nn = c1 a a a a a 21 11 11 11 11 11 Subtract it from the second equation: the result is: a a a a a a + a a +... + a a = c c or 21 22 12 2 23 13 3 2n 1n n 2 1 a11 a11 a11 a11 a' + a' +... + a' = b' 22 2 23 3 2n n
Repeat the same procedure to eliminate 1 from the other equations a + a + a + a +... + a = c 11 1 12 2 13 3 14 4 1n n 1... a' + a' + a' +... + a' = c' 22 2 23 3 24 4 2n n 2... a' + a' + a' +... + a' = c'...... 32 2 33 3 34 4 3n n 3... a' + a' + a' 4 +... + a' = c' n2 2 n3 3 n4 nn n n First equation is the pivot equation and a 11 is the pivot coefficient Repeat the above procedure to eliminate 2 from the third upto nth equation a + a + a + a +... + a = c 11 1 12 2 13 3 14 4 1n n 1... a' + a' + a' +... + a' = c' 22 2 23 3 24 4 2n n 2... + a" + a" +... + a" = c"......... a" 33 3 34 4 3n n 3 + a" +... + a" = c" + n3 3 n4 4 nn n n
Repeat the procedure (n-1) times till n-1 is eliminated from the nth equation. a + a + a + a +... + a = c 11 1 12 2 13 3 14 4 1n n 1... a' + a' + a' +... + a' = c' 22 2 23 3 24 4 2n n 2... + a" + a" +... + a" = c"...... Upper triangular system is formed. 33 3 34 4 3n n 3 n... + a = c ( 1) ( n 1) nn n n Determine i by back substitution n = c a ( n 1) n ( n 1) nn i = n ( i 1) ( i 1) i ij j j=+ i 1 c a a ( i 1) ii
Forward elimination Back Sustitution
Eample (Pr.9.10): 2 + = 1 1 5 + 2 + 2 = 4 3 3 + + = 5 3 Back Substitution 3 2.. +..... = 1 1... 0.5 + 4.5 = 6. 5 2 3... 0.5 + 2. 5 = 3. 5 2 3 10 = = 5 2 1 = [ 6.5 4.5 ] = 32 0.5 2 3 1 1 = [ 1 ( 1) 3 ] = 14 2 Forward Elimination 2.. +...... = 1 1... 0.5 + 4.5 = 6.5 2 3... 2. = 10 Check your result!!! 3
Eample 3 + 2 = 12 1 + 2 + 3 = 11 3 2 2 = 2 3 Augmented Matri 3 12 3 11 2 2 We shall use 3 digits Row2-1/3*Row1... Row3-1/3*Row1... 3 12 0 2.333 2.334 7.004 0 1.334 2.332 5.992 Row3-(-1.334/2.333)*Row1.. 3 12 0 2.333 2.334 7.004 0 0 1 1.993 X 3 =1.993, 2 =(7.004-2.334*1.993)/2.333=1.008, X 1 =(12-2*1.993+1*1.008)/3=3.007 (actual result is 2,1,3)
Operation Count Elimination: Outer Loop k 1 2.. k Middle Loop i 2,n 3,n.. k+1,n FLOPS [n-1][1+n] [n-2][n] [n-k][1n+2- k] n-1 n,n [1][3] Number of Mult&Div= n 3 3 + On 2 ( ) Back Substitution: 2 On ( ) Therefore, the number of operations is proportional to n 3 /3
Pitfalls: 1. Division by zero: If the pivot element is zero i.e. a 11 =0 2. Round off errors (especially when n is large) 3. Ill conditioned systems (when the determinant D 0 but, D~0 Improvements: 1. Use more significant figures (time?) 2. Pivoting Partial pivoting: Determine the largest coefficient in the column below the pivot element and switch rows (equations). Complete pivoting: Determine the largest element (in rows and columns and switch the rows (equations) and columns ( s) Too costly!!! 3. Scaling (normalization- make the highest coefficient equal to unity)
Eample (Partial pivoting): No Pivoting 0.0003 + 1.566 = 1.569 0.3454 2.436 = 1.018 We shall use 4 digit accuracy. With Pivoting 0.3454 2.436 = 1.018 0.0003 + 1.566 = 1.569 Multiply the first eq. By 0.3454/0.0003= 1 151 Multiply the first eq. By 0.0003/0.3454= 0.0009 0.0003 + 1. 566 = 1.569... 1804 = 1805 2 0.3454 2.436 = 1.018... + 1.5682 = 1.5681 2 X 2 = 1.001 X 1 =(1.569-1.566*1.001)/.0003= 3.333 X 2 = 0.9999 X 1 =(1.018+2.436*0.9999)/0.3454= 9.9993
Eample (Scaling): We shall use 3 significant digits + 50000 = 50000... +... = 2 + 50000 = 50000... 50000 = 50000 X 2 =1 and 1 =0 Pivot, but do not scale... +... = 2 2 + 50000 = 50000... +... = 2...50000 = 50000 2 If we scale the equations Pivot 0.00002 + = 1... + =... + = 2 0.00002 + = 1 + =... 2 2 = 1 X 2 =1 and 1 =1 2
Eample (Partial Pivoting)... 2... + 2 = 0 2 4 2 + 2 + 3 + 2 = 2 3 4 4 3... +... = 7 4 6 + 6 5.. = 6 3 4 Eliminate 6 1-6 -5 6 0 1.6667 5 3.6667-4 0-3.6667 4 4.3333-11 0 2 0 2 0 Eliminate 6 1-6 -5 6 0-3.6667 4 4.3333-11 0 0 6.8181 5.6364-9 0 0 2.1818 4.3636-6 0 2 0 2 0 2 2 3 2 2 4 3 0 1 7 6 1 6 5 6 Pivot change Use back substitution to obtain: X 4 = -1.2188; 3 = -0.3125; 2 = 1.2188; 1 = -.5313 6 1-6 -5 6 2 2 3 2-2 4-3 0 1-7 0 2 0 2 0 Pivot change 6 1-6 -5 6 0-3.6667 4 4.3333-11 0 1.6667 5 3.6667-4 0 2 0 2 0 Eliminate 6 1-6 -5 6 0-3.6667 4 4.3333-11 0 0 6.8181 5.6364-9 0 0 0 2.5600-3.12
Pseudo Code
Comments on pseudocode Equations are not scaled but scaled values of the elements are used to determine whether pivoting is required. Diagonal term is monitored to determine near-zero occcurrence s i : is the value of the maimum element on ith row. a: Coefficient matri, b: column vector of a set of equations a=b n: number of equations (or unknowns) to be solved. tol, a small value defined by the user. If the diagonal element <tol the matri is considered as singular and er is set to -1. er is to be returned by the program.
Gauss Jordan Augmented Matri Interchange row 1&4 and normalize... 2... + 2 = 0 2 4 2 + 2 + 3 + 2 = 2 3 4 4 3... +... = 7 4 6 + 6 5.. = 6 3 4 Eliminate 1 0.1667-1 -0.8333 1 0 1.6667 5 4.3333-11 0-3.6667 4 3.6667-4 0 2 0 2 0 Normalize 3rd row and eliminate 1 0 0 0.0400-0.58 0 1 0-2.800 1.56 0 0 1 0.8267-1.32 0 0 0 2.5600-3.12 0 2 0 2 0 2 2 3 2 2 4 3 0 1 7 6 1 6 5 6 Interchange Row 2&3, normalize & eliminate all elements in the second column. Normalize 4th row and eliminate 1 0.1667-1 -0.8333 1 2 2 3 2-2 4-3 0 1-7 0 2 0 2 0 1 0-0.8182-0.6364 0.5 0 1-1.0909-1.1818 3 0 0 6.8182 5.6364-9 0 0 2.1818 4.3636-6 1 0 0 0-0.5313 0 1 0 0 1.2188 0 0 1 0-0.3125 0 0 0 1-1.2188 Requires more work (n 3 /2, compared to n 3 /3 İn Gauss elimination)
Gauss-Jordan Method Eliminate the unknown from all but one of the equations. Pivot and Normalize 2 + = 1 1 5 + 2 + 2 = 4 3 3 + + = 5 3 Eliminate 2 1 1 1 1 1/3 1/3 5/3 5 2 2 4 /5 2/5 4/5 3 1 1 5 1 1/2 1/2 1/2 1 1/3 1/3 5/3 1 1/3 1/3 5/3 1 0 0 14 0 1/15 1/15 37/15 0 1 1 37 0 1 1 37 0 1/6 5/6 7/6 0 1 5 7 0 0 6 30 1 0 0 14 0 1 1 37 0 0 1 5 1 0 0 14 0 1 0 32 0 0 1 5 Right Column is the result. No back substitution
Inverse of a matri Using Gauss-Jordan: AA -1 =I Augment A with I 1 A = 3 0 1 1 0 2 Change Row 1&2 and normalize 1 1 0 0 1 0 0.3333 0 0.3333 0 3 0 1 0 1 0 1 1 0 0 1 0 2 0 0 1 1 0 2 0 0 1 Normalize and eliminate Eliminate 1 0 0.3333 0 0.3333 0 1 0 0.3333 0 0.3333 0 0 1 1.6667 1 0.3333 0 0 1 1.6667 1 0.3333 0 0 0 1.6667 0 0.3333 1 0 0 1.6667 0 0.3333 1 Normalize and eliminate 1 0 0 0 0.4 0.2 0 1 0 1 0 1 0 0 1 0 0.2 0.6 1 A 0 0.4 0.2 = 1 0 1 0 0.2 0.6 One can also use LU decomposition eplained in the book (pg 273).
Iterative Methods Guess a solution and improve your solution The Equations must be diagonally dominant Consider: a + a + a = b 11 1 12 2 13 3 1 a + a + a = b 21 2 2 23 3 2 a + a + a = b 31 1 32 2 33 3 3 Solve the first eq. For 1, second for 2 and the third for 3 : 1 2 3 = = = b a a a 1 12 2 13 3 11 b a a a 2 21 3 3 22 b a a a 3 31 1 32 2 33
Iterative Method 1: Gauss-Seidel 1. Assume i =0 (i=0, 1,2, n) 2. Solve for 1 from the first eq. 3. Solve for 2 from the second eq. (use 1 obtained in step 2) 4. Solve for 3 from the 3rd eq. (use 1 and 2 values) 5. Solve for 1 from the 1st eq. (use the latest values available) 6. Repeat until: j j 1 i i ε ai, = *100% < ε j s i 1 2 3 b a a = a11 b a a = a22 b a a = a 1 12 2 13 3 2 21 3 3 3 31 1 32 2 33
8 + = 8 3 7 + 2 = 4 3 2 + + 9 = 12 3 = 1 0.125 + 0.125 3 = 0.571+ 0.143 + 0.286 2 1 3 = 1.333 0.222 + 0.111 3 0 1 2 3 4 5 6 7 8 1 0.000000 1.000000 1.039718 0.996848 1.000565 0.999930 1.000010 0.999999 1.000000 2 0.000000 0.714000 1.014759 0.996559 1.000390 0.999942 1.000008 0.999999 1.000000 3 0.000000 1.031746 0.989544 1.001082 0.999831 1.000022 0.999997 1.000000 1.000000 ε s <10-6
PseudoCode for GaussSeidel
Start Iteration Iter=1 Do While Iter<MaIterations Sentinel=1 For i= 0 to n-1 Old=(i) (i)= BC(i) For j= 0 to n-1 If i<>j Then (i)=(j)-ac(i,j)*(j) Net j (i)=lambda*(i)+(1-lambda)*old = λ + (1 λ) new new old i i i 0< λ < 2 Relaation: λ=1 Original Gauss-Seidel λ<1 underrelaation λ>1 overrelaation
Iterative Method 2: Jacobi Compute a set of new values before using them in the equations. Gauss-Seidel Jacobi
Eample: 8 + = 8 3 7 + 2 = 4 3 2 + + 9 = 12 3 = 1 0.125 + 0.125 3 = 0.571+ 0.143 + 0.286 2 1 3 = 1.333 0.222 + 0.111 3 0 3 4 5 6 7 8 9 10 11 1 0.0000 1.0000 1.0953 0.9940 0.9926 1.0010 1.0005 0.9999 1.0000 1.0000 1.0000 1.0000 2 0.0000 0.5710 1.0952 1.0272 0.9901 0.9985 1.0009 1.0001 0.9999 1.0000 1.0000 1.0000 3 0.0000 1.3330 1.0476 0.9683 0.9983 1.0027 0.9999 0.9998 1.0000 1.0000 1.0000 1.0000 Jacobi 1 0 0.0000 1 1.0000 2 1.0397 3 0.9968 4 1.0006 5 0.9999 6 1.0000 Gauss-Seidel ε s <10-4 2 0.0000 0.7140 1.0148 0.9966 1.0004 0.9999 1.0000 3 0.0000 1.0317 0.9895 1.0011 0.9998 1.0000 1.0000
Matri Norms and Matri Condition Number Norm is the largest sum of the columns A = ma n 1 1 j n i = 1 a ij Norm is largest sum of the rows A = ma 1 i n n j = 1 a ij Uniform Matri Norm or Row-Sum Norm Condition Number of Matri A (>1) 1 Cond [ A] = A A
System of Nonlinear equations N nonlinear equations in n unknowns f (,,,,,... ) = 0 1 3 4 f (,,,,,... ) = 0... 2 3 4 f (,,,,,... ) = 0 n 3 4 n n n Sometimes you can use a fied point iteration or an iteration similar to Gauss Seidel Iteration for linear equations. Newton Raphson method for n variables Newton Raphson Method for one variable: f( ) = f( ) + '( )( f ) ( 0 f f f = f + ) + ( ) +... + ( ) + HOTerms 0 i+ 1 1, i 1, i 1, i i+ 1 i i i+ 1 i 1, i+ 1 1, i 1, i+ 1 1, i 2, i+, i n, i+ 1 n, i 1 2 1 = i f2, i f2, i f2, i f( i ) f2, i+ 1 = f2, i + ( 1, i+ 1 1, i) + ( 2, i+ 1 2, i) +... + ( n, i+ 1 n, i) + HOTerms 0 1 2 1 f '( i )... f f f f = f + ( ) + ( ) +... + ( ) + HOTerms 0 ni, ni, ni, ni, + 1 ni, 1, i+ 1 1, i 2, i+, i ni, + 1 ni, 1 2 1 f f f f f f + +... + = f + +... + 1, i 1, i 1, i 1, i 1, i 1, i 1, i+, i+ 1 n, i+ 1 1, i 1, i 2, i n, i n n f f f f f f + +... + = f + +... +... 2, i 2, i 2, i 2, i 2, i 2, i 1, i+, i+ 1 n, i+, i 1, i 2, i n n f f f f f f + +... + = f + +... + ni, ni, ni, ni, ni, ni, 1, i+, i+ 1 ni, + 1 ni, 1, i 2, i ni, n n ni,
J = i i+ 1 Bi J i f f f... d d d f f f.. 1 1 1 1 i 2 i n i 2 2 2 = d1 d i 2 d i n i............ fn fn fn.. d d d 1 i 2 i n i B i f1 f1 f1 f1, i + 1, i + 2, i... + n, i 1 i 2 i n i f2 f2 f 2 f2, i + 1, i + 2, i... + n, i = 1 i 2 i n i... fn fn fn fni, + 1, i+ 2, i... + ni, 1 i 2 i n i Solve for i+1 using any one of the methods discussed and repeat till there is convergence Or, if we let u i+1 = i+1 - i then J iu = i+ 1 Ci Where C i f1, i f 2, i =... fni, Solve for u i+1, i+1 =u i+1 + i
or Let: δ ki, = ki, + 1 ki, Aδ = B Then: i i i A i f f f... d d d f f f.. 1 1 1 1 i 2 i n i 2 2 2 = Ji = d1 d i 2 d i n i............ fn fn fn.. d d d 1 i 2 i n i B i f1, i f 2, i =... f2, i δ i δ1, i δ 2, i =... δ2, i = + +δ ki, 1 ki, ki,