Solving Linear Systems Using Gaussian Elimination How can we solve? 1
Gaussian elimination Consider the general augmented system:
Gaussian elimination Step 1: Eliminate first column below the main diagonal. Subtract a j1 a 11 the first row from the j th row, j =,,...,n. Pivot!
Gaussian elimination Step : Eliminate second column below the main diagonal. Subtract a(1) j a (1) the second row from the j th row, j =,,...,n.!
Gaussian elimination Repeat:. Subtract a() j a (). Subtract a() j a () the third row from the j th row, j =,,...,n:! (A () b () ) the fourth row from the j th row, j =,,...,n:! (A () b () ) n 1. Subtract a (n ) a j,n 1 (n ) n 1,n 1. the (n 1) row from the j th row, j = n:! (A (n 1) b (n 1) )
Gaussian Elimination After n 1 steps an upper triangular system is obtained: This system can now be solved using backward substitution.
Backward Substitution Upper triangular system Backward substitution: For k = n, n 0 x k = 1 nx (k 1) @b (k 1) k a kj a kk j=k+1 (k 1) x j 1,...,1 compute 1 A
Cost of Gaussian elimination The cost of eliminating entries below the 1 st pivot: Pivot a 11 a 1 a 1 a 1 a 1 b 1 a 1 a a a a b a 1 a a a a b a 1 a a a a b a 1 a a a a b n = 8
Cost of Gaussian elimination The cost of eliminating entries below the 1 st pivot: Pivot 11 1 1 1 1 1 1 1 1 1 n = Each row operation costs n + 1 multiplies/divides and n subtractions: Total operations per row: n +1 There are n 1 rows to eliminate: Total operations: (n + 1)(n 1) = n(n 1) + n 1 9
Cost of Gaussian elimination The cost of eliminating entries below the nd pivot: Pivot 11 1 1 1 1 1 0 a (1) a (1) a (1) a (1) b (1) 0 a (1) a (1) a (1) a (1) b (1) 0 a (1) a (1) a (1) a (1) b (1) 0 a (1) a (1) a (1) a (1) b (1) n = Each row operation costs n multiplies/divides and n 1 subtractions: Total operations per row: n 1 There are n rows to eliminate: Total operations: (n 1)(n ) = (n 1)(n ) + n 10
Cost of Gaussian elimination The cost of eliminating entries below the rd pivot: 11 1 1 1 1 1 0 a (1) a (1) a (1) a (1) b (1) a () a () b () Pivot 0 0 a () 0 0 a () a () a () b () 0 0 a () a () a () b () n = Each row operation costs n 1 multiplies/divides and n subtracts: Total operations per row: n There are n rows to eliminate: Total operations: (n )(n ) = (n )(n ) + n 11
Cost of Gaussian elimination The cost of eliminating entries below the th pivot: 11 1 1 1 1 1 0 a (1) a (1) a (1) a (1) b (1) 0 0 a () Pivot 0 0 0 a () a () a () b () a () b () 0 0 0 a () a () b () n = Each row operation costs n multiplies/divides and n subtracts: Total operations per row: n There are n rows to eliminate: Total operations: (n )(n ) = (n )(n ) + n 1
Cost of Gaussian elimination After the (n 1) st step the matrix is in upper triangular form: 11 1 1 1 1 1 0 a (1) a (1) a (1) a (1) b (1) 0 0 a () a () a () b () 0 0 0 a () a () b () 0 0 0 0 a () b () n = 1
Cost of Gaussian elimination We can count the total floating point operations (FLOPs): Total FLOPs = = nx k=1 ((n k + 1)(n k)+(n k)) nx 1 (n k + 1)(n k)+ k=1 = n(n 1) + 1 n(n 1) = n + 1 n n nx 1 (n k) k=1 n we ignore lower order powers of n We say that elimination is an n algorithm, which is considered expensive. 1
What about the back solve? a 11 a 1 a a1 a 1 a 1 0 a a a a 0 0 a a a 0 0 0 a a x 1 x x x = b 1 b b b 0 0 0 0 a x b x = b /a! 1 op x =(b a x )/a! ops One op is a multiplication or divide; Ignore subtractions for now; we will add them in momentarily. x =(b a x x a x )/a! ops x =(b a x a x x a x )/a! ops x 1 =(b 1 a 1 x a 1 x a 1 x x a 1 x )/a 11! ops 1
What about the cost of the back solve? Step k in the back solve requires k multiplications and k additions. So the total work for a back solve is : nx k=1 k = n(n + 1) 1 n Ignore lower order powers of n, but pay attention to the coefficient of the highest order power of n. or 1 n = n if we include subtractions, as well as divisions and multiplications. We say that a back solve is an order n operation, which is considerably cheaper than the original elimination. 1
Pivoting What happens when one of the diagonals is zero? We have to pivot! Partial pivoting Complete pivoting 1