Chapter : Systems of Linear Equations February, 9 Systems of linear equations Linear systems Lecture A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and b are constant real or complex numbers The constants a i is called the coefficient of x i, and b the constant term of the equation A system of linear equations (or linear system) is a finite collection of linear equations in same variables For instance, a linear system of m equations in n variables x, x,, x n can be written as a x + a x + + a n x n = b a x + a x + + a n x n = b () a m x + a m x + + a mn x n = b m A solution of a linear system () is a tuple (s, s,, s n ) of numbers that makes each equation a true statement when the values s, s,, s n are substituted for x, x,, x n, respectively The set of all solutions of a linear system is called the solution set of the system Theorem Any system of linear equations has one and only of the following conclusions (a) no solution; (b) unique solution; (c) infinitely many solutions A linear system is called consistent if it has at least one solution, and inconsistent if it has no solution Geometric interpretation The following three linear systems x +x = 3 (a) x x = x x = 4 (b) x +x = 3 x x = 5 x x = 4 (c) x +x = 3 4x +x = 6 6x +3x = 9 have no solution, a unique solution, and infinitely many solutions, respectively See Figure Note: A linear equation of two variables represents a straight line in R A linear equation of three variables represents a plane in R 3 In general, a linear equation of n variables represents a hyperplane in the
x x x o x o x o x Figure : No solution, unique solution, and infinitely many solutions n-dimensional Euclidean space R n Matrices of a linear system Definition The augmented matrix of the general linear system () is the table a a a n b a a a n b a m a m a mn b m and the coefficient matrix of () is a a a n a a a n a m a m a mn () (3)
Systems of linear equations can be represented by matrices Operations on equations (for eliminating variables) can be represented by appropriate row operations on the corresponding matrices For example, x +x x 3 = x 3x +x 3 = 8 3 8 3x +x +4x 3 = 7 3 4 7 [Eq ] [Eq ] [Eq 3] 3[Eq ] x +x x 3 = 5x +5x 3 = x +x 3 = 4 ( /5)[Eq ] ( /)[Eq 3] x +x x 3 = x x 3 = x 5x 3 = R R R 3 3R 5 5 4 ( /5)R ( /)R 3 5 [Eq 3] [Eq ] R 3 R x +x x 3 = x x 3 = 4x 3 = 4 4 4 ( /4)[Eq 3] ( /4)R 3 x +x x 3 = x x 3 = x 3 = [Eq ] + [Eq 3] [Eq ] + [Eq 3] x +x = 3 x = 3 x 3 = R + R 3 R + R 3 3 3 [Eq ] [Eq ] R R x = x = 3 3 x 3 = 3
Elementary row operations Lecture Definition 3 There are three kinds of elementary row operations on matrices: (a) Multiplying all entries of one row by a nonzero constant; (b) Interchanging two rows; (c) Adding a multiple of one row to another row Definition 4 Two linear systems in same variables are called equivalent if they have the same solutions, that is, their solution sets are the same A matrix A is said to be row equivalent to a matrix B, written A B, if there is a sequence of elementary row operations that changes A to B Theorem 5 If the augmented matrices of two linear systems are row equivalent, then the two systems have the same solution set In other words, elementary row operations do not change solution set Proof It is trivial for the row operations (b) and (c) in Definition 3 Consider the row operation (a) in Definition 3 Without loss of generality, we may assume that a multiple of the first row is added to the second row Let us only exhibit the first two rows as follows a a a n b a a a n b (4) a m a m a mn b m Do the row operation (Row ) + c(row ) We obtain a a a n b a + ca a + ca a n + ca n b + cb a 3 a 3 a 3n b 3 a m a m a mn b m (5) Let (s, s,, s n ) be a solution of (4), that is, a i s + a i s + + a in s n = b i, i m (6) In particular, a s + a x + + a n s n = b, (7) a s + a x + + a n s n = b (8) Multiplying c to both sides of (7), we have ca s + ca + + ca n s n = cb (9) Adding both sides of (8) and (9), we obtain (a + ca )s + (a + ca )s + + (a n + ca n )s n = b + cb () This means that (s, s,, s n ) is a solution of (5) 4
Conversely, let (s, s,, s n ) be a solution of (5), ie, () is satisfied and the equations of (6) are satisfied except for i = Since a s + a x + + a n s n = b, multiplying c to both sides we have Note that () can be written as c(a s + a + + a n s n ) = cb () (a s + a s + + a n s n ) + c(a s + a s + + a n s n ) = b + cb () Subtracting () from (), we have This means that (s, s,, s n ) is a solution of (4) Row echelon forms a s + a s + + a n s n = b Lecture 3 Definition A matrix is called in row echelon form if it satisfies the following two conditions: (a) All zero rows are gathered near the bottom; (b) The first nonzero entry of a row, called the leading entry of that row, is ahead of the first nonzero entry of the next row A matrix in row echelon form is said to be in reduced row echelon form if it satisfies two more conditions: (c) The leading entry of every nonzero row is ; (d) Each leading entry is the only nonzero entry in its column A matrix in (reduced) row echelon form is called a (reduced) row echelon matrix Note Sometimes we call row echelon forms just as echelon forms and row echelon matrices as echelon matrices without mentioning the word row Row echelon form pattern The following are two typical row echelon matrices, where the solid squares represent arbitrary nonzero numbers, and the stars represent arbitrary numbers, including zero The following are two typical reduced row echelon matrices, 5
Theorem Every matrix is row equivalent to one and only one reduced row echelon matrix In other words, every matrix has a unique reduced row echelon form (or canonical form) Proof See Appendix A Definition 3 If a matrix A is row equivalent to a row echelon matrix E, we say that A has the row echelon form E; if E is further a reduced row echelon matrix, then we say that A has the reduced row echelon form (or canonical form) E Row reduction algorithm Definition 4 A pivot position of a matrix A is a location of entries of A that corresponds to a leading entry in a row echelon form of A A pivot column (pivot row) is a column (row) of A that contains a pivot position Algorithm (Row Reduction Algorithm) (Proof of Theorem ) () Begin with the leftmost nonzero column, which is a pivot column; the top entry is a pivot position () If the entry of the pivot position is zero, select a nonzero entry in the pivot column, interchange the pivot row and the row containing this nonzero entry (3) If the pivot position is nonzero, use elementary row operations to reduce all entries below the pivot position to zero, (and the pivot position to and all entries above and below the pivot position to zero for reduced row echelon form) (4) Cover the pivot row and the rows above it; repeat () (3) to the remaining submatrix Solving linear system Example Find all solutions for the linear system x +x x 3 = x +x +4x 3 = 3x +3x +4x 3 = Solution Perform the row operations: 4 3 3 4 7 4 The system is equivalent to which means the system has a unique solution R R R 3 3R R + R 3 R + R 3 x = 7 x = 4 x 3 = 3 6 3 7 4 Example Solve the linear system x x +x 3 x 4 = x x +x 3 +x 4 = 4x 4x +4x 3 = 4 x +x x 3 +x 4 = 3 ( /3)R R 3 R R R 6
Solution Do the following row operations: R R R 3 4R 4 4 4 4 3 R 4 + R R + R The linear system has the following parametric description { x = + x x 3 x 4 = 4 4 () [ ] [] () (/)R R 3 R R 4 + (/)R We see that the variables x, x 3 can take arbitrary numbers; they are called free variables Let x = c, x 3 = c, where c, c R Then x = + c c, x 4 = All solutions of the system are given by The general solutions may be written as x x x 3 = x 4 x = + c c x = c x 3 = c x 4 = + c + c where c, c R Set c = c =, ie, set x = x 3 =, we have a particular solution x =, Basic solutions For the corresponding homogeneous linear system Ax = of the above example, that is, x x +x 3 x 4 = x x +x 3 +x 4 = 4x 4x +4x 3 = x +x x 3 +x 4 = we have 4 4 4 () [ ] [] () Let x = c, x 3 = c, where c, c R Then x = c c, x 4 = All solutions of the homogeneous system are given by x = c c x = c x 3 = c x 4 = 7
The general solutions may be written as x x x 3 x 4 = c + c Set c =, c =, ie, x =, x 3 =, we have x = and x 4 = ; we obtain one basic solution x = for the homogeneous system Ax = Set c =, c =, ie, x =, x 3 =, we have x = and x 4 = ; we obtain another basic solution x = for the homogeneous system Ax = Example 3 The linear system with the augmented matrix 5 3 3 4 has no solution because its augmented matrix has the row echelon form () ( 3) [7] The last row represents a contradictory equation = Theorem 5 A linear system is consistent if and only if the row echelon form of its augmented matrix contains no row of the form [,, b ], where b Example 4 Solve the linear system whose augmented matrix is A = 3 6 3 3 7 4 4 6 5 4 Solution Interchanging Row and Row, we have 3 6 3 3 7 4 4 6 5 4 4 4 5 6 R 3R R 4 R R R 3 8
Then the system is equivalent to 4 4 5 6 4 3 6 R 4 + R R 4 + 3 R 3 R 3 R R 3 () [] () [ ] [] () x = x x 4 + x 5 x 3 = + x 4 x 5 x 6 = The unknowns x, x 4 and x 5 are free variables Set x = c, x 4 = c, x 5 = c 3, where c, c, c 3 are arbitrary The general solutions of the system are given by x = c c + c 3 x = c x 3 = + c c 3 x 4 = c x 5 = c 3 x 6 = The general solution may be written as x = + c + c + c 3, c, c, c 3 R Definition 6 A variable in a consistent linear system is called free if its corresponding column in the coefficient matrix is not a pivot column 3 Vector Space R n Vectors in R and R 3 Lecture 4 Let R be the set of all tuples (u, u ) of real numbers Each tuple (u, u ) is called a point in R For each point (u, u ) we associate a column [ u u ] 9
called the vector associated to the point (u, u ) The set of all such vectors is still denoted by R So by a point in R we mean a tuple (x, y) and, by a vector in R we mean a column Similarly, we denote by R 3 the set of all tuples [ x y ] (u, u, u 3 ) of real numbers, called points in R 3 We still denote by R 3 the set of all columns u u, u 3 called vectors in R 3 For example, (, 3, ), ( 3,, ), (,, ) are points in R 3, while are vectors in R 3 3, 3 Definition 3 The addition, subtraction, and scalar multiplication for vectors in R are defined by [ ] [ ] [ ] u v u + v + =, u v u + v [ u ] [ ] v = u v [ ] u c = u, [ ] u v, u v [ cu cu ] Similarly, the addition, subtraction, and scalar multiplication for vectors in R 3 are defined by u v u + v u u 3 + v v 3 = u + v u 3 + v 3, Vectors in R n u u u 3 c v v u u u 3 v 3 = = u v u v u 3 v 3 cu cu cu 3 Definition 3 Let R n be the set of all tuples (u, u,, u n ) of real numbers, called points in R n We still use R n to denote the set of all columns u u u n,
of real numbers, called vectors in R n The vector = is called the zero vector in R n The addition and the scalar multiplication in R n are defined by u v u v u n + c u u u n v n = = cu cu cu n u + v u + v u n + v n The set R n together with the addition and scalar multiplication is called the Euclidean n-space Proposition 33 For vectors u, v, w in R n and scalars c, d in R, () u + v = v + u, () (u + v) + w = u + (v + w), (3) u + = u, (4) u + ( u) =, (5) c(u + v) = cu + cv, (6) (c + d)u = cu + du (7) c(du) = (cd)u (8) u = u Subtraction can be defined in R n by Linear combinations u v = u + ( v) Definition 34 A vector v in R n is called a linear combination of vectors v, v,, v k in R n if there exist scalars c, c,, c k such that v = c v + c v + + c k v k The set of all linear combinations of v, v,, v k is called the span of v, v,, v k, denoted Span {v, v,, v k } = {c v + c v + + c k v k : c, c,, c k R} Example 3 The span of a single nonzero vector in R 3 is a straight line through the origin For instance, Span 3 = t 3 : t R,
is a straight line through the origin, having the parametric form x = 3t x = t, t R x 3 = t Eliminating the parameter t, the parametric equations reduce to two equations about x, x, x 3, { x +3x = x +x 3 = Example 3 The span of two nonproportional vectors in R 3 is a plane through the origin For instance, Span 3, = s + t 3 : s, t R is a plane through the origin, having the following parametric form x = s +3t x = s t x 3 = s +t Eliminating the parameters s and t, the plane can be described by a single equation Example 33 Given vectors in R 3, v =, v = 3x 5x 7x 3 =, v 3 =, v 4 = (a) Every vector b = [b, b, b 3 ] T in R 3 is a linear combination of v, v, v 3 (b) The span of {v, v, v 3 } is the whole vector space R 3 3, v 5 = (b) The vector v 4 can be expressed as a linear combination of v, v, v 3, and it can be expressed in one and only one way as a linear combination v 4 = v + 3v v 3 (d) The vector v is not a linear combination of v, v 3, and v 4 (e) The vector v 5 can be written as linear combinations of v, v 3, and v 4 in more than one ways In fact, v 5 = 6v v 3 = 3v 3 + v 4 = 3v + v 3 + v 4 Solution (a) Let b = x v + x v + x 3 v 3 Then the vector equation has the following matrix form x x = b b x 3 b 3 5 6 5 Perform row operations: b b b 3 b b b b 3
Thus b b b b b 3 + b b b b 3 b + b + b 3 b b b 3 x = b b 3, x = b + b + b 3, x 3 = b b b 3 So b is a linear combination of v, v, v 3 (b) Since b is arbitrary, we have Span {v, v, v 3 } = R 3 Example 34 Consider vectors in R 3, v =, v =, v 3 = ; u =, v = 5 7 (a) The vector u can be expressed as linear combinations of v, v, v 3 in more than one ways For instance, u = v + v + v 3 = 4v + 3v = v v + v 3 (b) The vector v can not be written as a linear combination of v, v, v 3 Geometric interpretation of vectors Multiplication of matrices Definition 35 Let A be an m n matrix and B an n p matrix, a a a n a a a n A = a m a m a mn, B = The product (or multiplication) of A and B is an m p matrix c c c p c c c p C = = AB c m c m c mp whose (i, k)-entry c ik, where i m and k p, is given by c ik = b b b p b b b p b n a n b np n a ij b jk = a i b k + a i b k + + a in b nk j= Proposition 36 Let A be an m n matrix, whose column vectors are denoted by a, a,, a n Then for any vector v in R n, v v Av = [a, a,, a n ] = v a + v a + + v n a n v n Proof Write A = a a a n a a a n a m a m a mn, v = v v v n 3
Then Thus a = Av = = = a a a m = v, a = a a a m a a a n a a a n a m a m a mn,, a n = v v v n a v + a v + + a n v n a v + a v + + a n v n a m v + a m v + + a mn v n a v a v a m v a a a m + + v a v a v a m v a a a m + + + + v n a n a n a mn a n v n a n v n a mn v n a n a n a mn = v a + v a + + v n a n Theorem 37 Let A be an m n matrix Then for any vectors u, v in R n and scalar c, (a) A(u + v) = Au + Av, (b) A(cu) = cau 4 The four forms of linear systems Lecture 5 A general system of linear equations can be written as a x + a x + + a n x n = b a x + a x + + a n x n = b a m x + a m x + + a mn x n = b m (4) We introduce the column vectors: a = a a m,, a n = a n a mn ; x = x x n ; b = b b m ; 4
and the coefficient matrix: A = a a a n a a a n = [ a, a,, a n ] a m a m a mn Then the linear system (4) can be expressed by (a) The vector equation form: (b) The matrix equation form: x a + x a + + x n a n = b, Ax = b, (c) The augmented matrix form: [ a, a,, a n b ] Theorem 4 The system Ax = b has a solution if and only if b is a linear combination of the column vectors of A Theorem 4 Let A be an m n matrix The following statements are equivalent (a) For each b in R m, the system Ax = b has a solution (b) The column vectors of A span R m (c) The matrix A has a pivot position in every row Proof (Exhibit for m = 3 and n = 4) (a) (b) and (c) (a) are obvious (a) (c): Suppose A has no pivot position for at least one row; that is, A ρ ρ ρ k ρ k A Ak Ak, where ρ, ρ,, ρ k are elementary row operations, and A k is a row echelon matrix Let e m = [,,, ] T Clearly, the system [A k e m ] is inconsistent Let ρ i denote the inverse row operation of ρ i, i k Then [A k e m ] ρ k [A k b ] ρ k ρ [A b k ] ρ [A b k ] Thus, for b = b k, the system [A b] has no solution, a contradiction Example 4 The following linear system has no solution for some vectors b in R 3 x +x 3 +3x 4 = b x +4x +6x 3 +7x 4 = b x +x +x 3 +x 4 = b 3 The row echelon matrix of the coefficient matrix for the system is given by 3 4 6 7 R R 3 4 6 7 R R 3 3 R 3 R 3 3 5
Then the following systems have no solution 3 R 3 + R 4 6 7 R 3 R 3 Thus the original system has no solution for b =, b = b 3 = 3 3 5 Solution structure of a linear System Homogeneous system 3 4 6 7 R + R A linear system is called homogeneous if it is of the form Ax =, where A is an m n matrix and is the zero vector in R m Note that x = is always a solution for a homogeneous system, called the zero solution (or trivial solution); solutions other than the zero solution are called nontrivial solutions Theorem 5 A homogeneous system Ax = has a nontrivial solution if and only if the system has at least one free variable Moreover, # { pivot positions } + # { free variables } = # { variables } Example 5 Find the solution set for the nonhomogeneous linear system x x +x 4 +x 5 = x +x x 3 4x 4 3x 5 = x x +x 3 +3x 4 +x 5 = x +x +x 3 +x 4 3x 5 = Solution Do row operations to reduce the coefficient matrix to the reduced row echelon form: R + R 4 3 R 3 R 3 3 R 4 + R () [ ] () [] [ ] Then the homogeneous system is equivalent to { x = x x 4 x 5 ( )R R 3 + R R 4 + R x 3 = x 4 +x 5 The variables x, x 4, x 5 are free variables Set x = c, x 4 = c, x 5 = c 3 We have x c c c 3 x x 3 x 4 = c c + c 3 c = c + c + c 3 x 5 c 3 6
Set x =, x 4 =, x 5 =, we obtain the basic solution v = Set x =, x 4 =, x 5 =, we obtain the basic solution v = Set x =, x 4 =, x 5 =, we obtain the basic solution v 3 = The general solution of the system is given by In other words, the solution set is Span {v, v, v 3 } x = c v + c v + c 3 v 3, c, c, c 3, R Theorem 5 Let Ax = be a homogeneous system If u and v are solutions, then the addition and the scalar multiplication u + v and cu are also solutions In other words, any linear combination of solutions for a homogeneous system is again a solution Theorem 53 Let Ax = be a homogeneous linear system, where A is an m n matrix with p pivot positions Then the system has n p free variables and n p basic solutions The n p basic solutions can be obtained as follows: Select one of the free variables; set the free variable equal to and all other free variables equal to Nonhomogeneous systems A linear system Ax = b is called nonhomogeneous if b The homogeneous linear system Ax = is called its corresponding homogeneous linear system Proposition 54 Let v and w be solutions of a nonhomogeneous system Ax = b Then the difference v w is a solution of the corresponding homogeneous system Ax = Theorem 55 (Structure of Solution Set) Let v be a solution of a nonhomogeneous system Ax = b Then solutions of Ax = b are given by x = v + u, where u are solutions of the corresponding homogeneous system Ax = 7
y v cu v+ cu v x x+3y=6 x+3y= Figure : Solution sets of non-homogeneous system and its homogeneous system Example 5 For the system x + 3y = 6 (its homogeneous system x + 3y = ), the solution is given by [ ] [ ] [ ] x 3 3/ = + c = v + cu, c R y Example 53 Find the solution set for the nonhomogeneous linear system x x +x 4 +x 5 = x +x x 3 4x 4 3x 5 = 3 x x +x 3 +3x 4 +x 5 = x +x +x 3 +x 4 3x 5 = 3 Solution Do row operations to reduce the augmented matrix to the reduced row echelon form: R + R ( )R 4 3 3 R 3 R R 3 + R 3 3 3 R 4 + R R 4 + R () [ ] () [] [ ] Then the nonhomogeneous system is equivalent to { x = +x x 4 x 5 x 3 = x 4 +x 5 The variables x, x 4, x 5 are free variables Set x = c, x 4 = c, x 5 = c 3 We obtain the general solutions x + c c c 3 x x 3 x 4 = c c + c 3 c = + c + c + c 3 x 5 c 3 x = v + c v + c v + c 3 v 3, c, c, c 3 R 8
In particular, setting x = x 4 = x 5 =, we obtain the particular solution v = The solution set of Ax = b is given by S = v + Span {v, v, v 3 } Example 54 The augmented matrix of the nonhomogeneous system x x +x 3 +3x 4 +3x 5 +4x 6 = x +x x 3 3x 4 4x 5 5x 6 = x +x x 5 4x 6 = 5 x +x +x 3 +3x 4 +x 5 x 6 = has the reduced row echelon form () [ ] 3 () [3] [] 3 () Then a particular solution x p for the system and the basic solutions of the corresponding homogeneous system are given by 3 x p = 3, v =, v = 3, v 3 = The general solution of the system is given by x = x p + c v + c v + c 3 v 3 Example 55 For what values of a and b the linear system x +x +ax 3 = x +bx = 3x +x +x 3 = has nontrivial solutions Answer: 8a + (3a )(b 4) = 6 Linear dependence and independence Lecture 6 Starting with motivation of two or more vectors Then conclude to the following formal definition Definition 6 Vectors v, v,, v p in R n are called linearly independent provided that if c v + c v + + c p v p = for some scalars c, c,, c p, then c = c = = c p = The vectors v, v,, v p are called linearly dependent if there exist constants c, c,, c p, not all zero, such that c v + c v + + c p v p = 9
Example 6 The vectors v =, v = Solution Consider the linear system x 3 + x, v 3 = + x 3 3 3 = in R 3 are linearly independent The system has the only zero solution x = x = x 3 = Thus v, v, v 3 are linearly independent Example 6 The vector v =, v =, v 3 = 3 in R 3 are linearly dependent 5 Solution Consider the linear system x 3 5 + x + x 3 3 6 3 5 = The system has one free variable There is nonzero solution Thus v, v, v 3 are linearly dependent Example 63 The vectors,,, 3 in R 3 are linearly dependent 3 Theorem 6 The basic solutions of any homogeneous linear system are linearly independent Proof Let v, v,, v k be basic solutions of a linear system Ax =, corresponding to free variables x j, x j,, x jk Consider the linear combination c v + c v + + c k v k = Note that the j i coordinate of c v + c v + + c k v k is c ji, i k It follows that c ji i k This means that v, v,, v k are linearly independent = for all Theorem 63 Any set of vectors containing the zero vector is linearly dependent Theorem 64 Let v, v,, v p be vectors in R n If p > n, then v, v,, v p are linearly dependent Proof Let A = [v, v,, v p ] Then A is an n p matrix, and the equation Ax = has n equations in p unknowns Recall that for the matrix A the number of pivot positions plus the number of free variables is equal to p, and the number of pivot positions is at most n Thus, if p > n, there must be some free variables Hence Ax = has nontrivial solutions This means that the column vectors of A are linearly dependent Theorem 65 Let S = {v, v,, v p } be a set of vectors in R n, (p ) Then S is linearly dependent if and only if one of vectors in S is a linear combination of the other vectors Moreover, if S is linearly dependent and v, then there is a vector v j with j such that v j is a linear combination of the preceding vectors v, v,, v j
Note The condition v in the above theorem can not be deleted For example, the set { [ ] [ ]} v =, v = is linearly dependent But v is not a linear combination of v Theorem 66 The column vectors of a matrix A are linearly independent if and only if the linear system has the only zero solution Ax = Proof Let A = [a, a,, a n ] Then the linear system Ax = is the vector equation x a + x a + + x n a n = Then a, a,, a n are linear independent is equivalent to that the system has only the zero solution Corollary 67 A linear system Ax = b has unique solution if and only if Ax = has the only zero solution