MATH 3511 Lecture 1 Solving Linear Systems 1 Dmitriy Leykekhman Spring 2012 Goals Review of basic linear algebra Solution of simple linear systems Gaussian elimination D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 1
Linear System of Equations Linear system of m equations with n unknowns x 1, x 2,, x n we write as a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2 a m1 x 1 + a m2 x 2 + + a mn x n = b m, where a ij R and b i R, 1 i m, 1 j n D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 2
Linear System of Equations From the coefficients a ij we form a matrix a 11 a 12 a 1n a 21 a 22 a 2n A = a m1 a m2 a mn We also define vectors b and x b = b 1 b 2 b m, x = x 1 x 2 x n Notice that A R m n, x R n, and b R m D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 3
Matrix-Vector Multiplication Let A R m n and x R n The i-th component of the matrix-vector product y = Ax is defined by y i = n a ij x j (1) j=1 ie, y i is the dot product (inner product) of the i-th row of A with the vector x y i = a i1 a i2 a in x 1 x 2 x n D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 4
Matrix-Vector Multiplication Another useful point of view is to look at entire vector y = Ax y 1 y i y m = a 11 a 12 a 1n a i1 a i2 a in a m1 a m2 a mn x 1 x 2 x n = x 1 a 11 a i1 a m1 + x 2 a 12 a i2 a m2 + + x n a 1n a in a mn Thus, y is a linear combination of the columns of matrix A D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 5
Definitions Let v 1,, v n be some vectors in R n and c 1,, c n be some numbers An expression of the form c 1 v 1 + + c n v n is called a linear combination of v 1,, v n A set of all possible linear combinations of vectors v 1,, v n is called a linear span of v 1,, v n Vectors v 1,, v n are linearly dependent if there exist numbers c 1,, c n not all zeros such that c 1 v 1 + + c n v n = 0 If such numbers do not exist we say that vectors v 1,, v n are linearly independent D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 6
Solution of Linear Systems Let A R m n and x R n Then Ax is a linear combination of the columns of A Hence Ax = b has a solution if b R m can be written as a linear combination of the columns of A Solvability: Ax = b is solvable for every b R m iff the columns of A span R m (necessary n m) Uniqueness: If Ax = b has a solution, then the solution is unique iff the columns of A are linearly independent (necessary n m) Hence, for any b R m, the system Ax = b has a unique solution iff n = m and the columns of A are linearly independent D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 7
Solution of Triangular Systems A matrix L R n n is called lower triangular matrix if all matrix entries above the diagonal are equal to zero, ie, if l ij = 0 for j > i A matrix U R n n is called upper triangular matrix if all matrix entries below the diagonal are equal to zero, ie, if l ij = 0 for i > j A Linear system with lower (upper) triangular matrix can be solved by forward substitution (backward substitution) D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 8
Solution of Triangular Systems Example Consider the system Forward substitution gives 2 0 0 1 4 0 4 3 3 x 1 x 2 x 3 = x 1 = 4/2 = 2, x 2 = (2 1 2)/4 = 0, x 3 = (5 4 2 3 0)/3 = 1 4 2 5 D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 9
Solution of Triangular Systems Example Consider the system 2 4 2 0 2 4 0 0 3 Backward substitution gives x 1 x 2 x 3 = x 3 = 6/3 = 2, x 2 = (2 4 2)/( 2) = 3, x 1 = (4 2 2 4 3)/2 = 6 4 2 6 D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 10
Solution of Triangular Systems Back substitution Solution of Upper Triangular Systems (Row-Oriented Version) Input: Upper triangular matrix U R n n, right hand side vector b R n Output: Solution x R n of Ux = b Mathematically, x i = b i n u ij x j /u ii, if u ii 0 j=i+1 MATLAB code that overwrites b with the solution to Ux = b if all(diag(u)) == 0 disp( the matrix is singular ) else b(n) = b(n)/u(n,n); for i = n-1:-1:1 b(i)= (b(i) - U(i,i+1:n)*b(i+1:n))/U(i,i); end end D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 11
Solution of Triangular Systems Back substitution Solution of Upper Triangular Systems (Column-Oriented Version) Input: Upper triangular matrix U R n n, right hand side vector b R n Output: Solution x R n of Ux = b MATLAB code that overwrites b with the solution to Ux = b if all(diag(u)) == 0 disp( the matrix is singular ) else for end j = n:-1:2 b(j) = b(j)/u(j,j) ; b(1:j-1) = b(1:j-1) - b(j)*u(1:j-1,j); end b(1) = b(1)/u(1,1); D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 12
Gaussian Elimination Gaussian elimination for the solution of a linear system transforms the system Ax = b into an equivalent system with upper triangular matrix This is done by applying three types of transformations to the augmented matrix (A b) Type 1: Replace an equation with the sum of the same equation and a multiple of another equation; Type 2: Interchange two equations; and Type 3: Multiply an equation by a nonzero number Once the augmented matrix (A b) is transformed into (U y), where U is an upper triangular matrix, we can use the techniques discussed previously to solve this transformed system Ux = y D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 13
Gaussian Elimination (cont) Consider a linear system Ax = b with A = 1 2-1 2 1 0, b = 0 2-1 2 2 1 The augmented system is 2 1 0 2 1 2-1 0-1 2 2 1 Step 1 1 2-1 0 0-3 2 2 0 4 1 1 (1) (2) (2) 2 (1) (3) (3) + 1 (2) D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 14
Gaussian Elimination (cont) Consider a linear system Ax = b with A = 1 2-1 2 1 0, b = 0 2-1 2 2 1 The augmented system is 2 1 0 2 1 2-1 0-1 2 2 1 Step 2 1 2-1 0 0-3 2 2 0 0 11/3 11/3 (1) (2) (3) (3) + 4/3 (2) Solving the triangular system we obtain x 3 = 1, x 2 = 0, x 1 = 1 D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 14
Gaussian Elimination We need to modify Gaussian elimination for two reasons: improve numerical stability (change how we perform pivoting) make it more versatile (leads to LU-decomposition) D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 15
Gaussian Elimination (cont) Partial pivoting: In step i, find row j with j > i such that a ji a ki for all k > i and exchange rows i and j Such numbers a ji we call pivots Again consider the augmented system 1 2-1 0 2 1 0 2-1 2 2 1 Step 1 2 1 0 2 1 2-1 0-1 2 2 1 (1) (2) (2) (1) (3) D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 16
Gaussian Elimination (cont) Partial pivoting: In step i, find row j with j > i such that a ji a ki for all k > i and exchange rows i and j Such numbers a ji we call pivots Again consider the augmented system 1 2-1 0 2 1 0 2-1 2 2 1 Step 2 2 1 0 2 0 3/2-1 -1 0 5/2 2 2 (1) (2) (2) 1/2 (1) (3) (3) ( 1/2) (2) D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 16
Gaussian Elimination (cont) Partial pivoting: In step i, find row j with j > i such that a ji a ki for all k > i and exchange rows i and j Such numbers a ji we call pivots Again consider the augmented system 1 2-1 0 2 1 0 2-1 2 2 1 Step 3 2 1 0 2 0 5/2 2 2 0 3/2-1 -1 (1) (2) (3) (3) (2) D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 16
Gaussian Elimination (cont) Partial pivoting: In step i, find row j with j > i such that a ji a ki for all k > i and exchange rows i and j Such numbers a ji we call pivots Again consider the augmented system 1 2-1 0 2 1 0 2-1 2 2 1 Step 4 2 1 0 2 0 5/2 2 2 0 0 11 8 11 8 (1) (2) (3) (3) 3/2 5/2 (2) Solving the triangular system we obtain x 3 = 1, x 2 = 0, x 1 = 1 D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 16
Gaussian Elimination Why do we need partial pivoting? It doesn t matter if we use exact arithmetic Without partial pivoting with floating point arithmetic the method can be unstable How expensive is Gaussian elimination? Using n j 2 = j=1 n(n + 1)(2n + 1) 6 we can calculate that for large n the number of flops in the Gaussian elimination with partial pivoting approximately equal to 2n 3 /3 D Leykekhman - MATH 3511 Introduction to Computational Mathematics Linear Systems 1 17