Linear equations in linear algebra

Linear equations in linear algebra Samy Tindel Purdue University Differential equations and linear algebra - MA 262 Taken from Differential equations and linear algebra Pearson Collections Samy T. Linear equations Differential equations 1 / 81

Outline 1 Systems of linear equations 2 Row reduction and echelon form 3 Vector equations 4 Matrix equation Ax = b 5 Solution sets of linear systems 6 Linear dependence 7 Introduction to linear transformations 8 The matrix of a linear transformation Samy T. Linear equations Differential equations 2 / 81

Systems of linear equations General form of a m n linear system: a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2. (1) a m1 x 1 + a m2 x 2 + + a mn x n = b m System coefficients: The real numbers a ij System constants: The real numbers b i Homogeneous system: When b i = 0 for all i Samy T. Linear equations Differential equations 4 / 81

Example of linear system Linear system in R 3 : x 1 + x 2 + x 3 = 1 x 2 x 3 = 2 x 2 + x 3 = 6 Unique solution by substitution: x 1 = 5, x 2 = 4, x 3 = 2 Samy T. Linear equations Differential equations 5 / 81

Notation R n Definition of R n : Set of ordered n-uples of real numbers (x 1,..., x n ) Matrix notation: An element of R n can be seen as a row or column n-vector x 1 x (x 1,..., x n ) 2 [x 1,..., x n ]. x n Samy T. Linear equations Differential equations 6 / 81

Related matrices Matrix of coefficients: For the system (1), given by a 11 a 12 a 13... a 1n a A = 21 a 22 a 23... a 2n....... a m1 a m2 a m3... a mn Augmented matrix: For the system (1), given by a 11 a 12 a 13... a 1n b 1 A a = 21 a 22 a 23... a 2n b 2........ a m1 a m2 a m3... a mn b m Samy T. Linear equations Differential equations 7 / 81

Vector formulation Example of system in R 3 : x 1 +3x 2 4x 3 = 1 2x 1 +5x 2 x 3 = 5 x 1 +6x 3 = 3 Related matrices: 1 3 4 1 A = 2 5 1, b = 5, x = 1 0 6 3 x 1 x 2 x 3 Augmented matrix: 1 3 4 1 A = 2 5 1 5 1 0 6 3 Samy T. Linear equations Differential equations 8 / 81

Linear systems in R 3 General system in R 3 : a 11 x 1 + a 12 x 2 + a 13 x 3 = b 1 a 21 x 1 + a 22 x 2 + a 23 x 3 = b 2 a 31 x 1 + a 32 x 2 + a 33 x 3 = b 3 Geometrical interpretation: Intersection of 3 planes in R 3 Samy T. Linear equations Differential equations 9 / 81

Sets of solutions Possible sets of solutions: In the general R n case we can have No solution to a linear system A unique solution An infinite number of solutions Definition 1. Consider a linear system given by (1). Then 1 If there is at least one solution the system is consistent 2 If there is no solution the system is inconsistent Samy T. Linear equations Differential equations 10 / 81

Elementary row operations Operations leaving the system unchanged: 1 Permute equations 2 Multiply a row by a nonzero constant 3 Add a multiple of one equation to another equation Example of system: x 1 +3x 2 4x 3 = 1 2x 1 +5x 2 x 3 = 5 x 1 +6x 3 = 3 Samy T. Linear equations Differential equations 12 / 81

Example of elementary row operations Example of system: x 1 +3x 2 4x 3 = 1 2x 1 +5x 2 x 3 = 5 x 1 +6x 3 = 3 Permute R 1 and R 2 : Denoted by P 12 2x 1 +5x 2 x 3 = 5 x 1 +3x 2 4x 3 = 1 x 1 +6x 3 = 3 Samy T. Linear equations Differential equations 13 / 81

Example of elementary row operations (2) Example of system: x 1 +3x 2 4x 3 = 1 2x 1 +5x 2 x 3 = 5 x 1 +6x 3 = 3 Multiply R 2 by 2: Denoted by M 2 ( 2) 2x 1 +5x 2 x 3 = 5 4x 1 10x 2 +2x 3 = 10 x 1 +6x 3 = 3 Samy T. Linear equations Differential equations 14 / 81

Example of elementary row operations (3) Example of system: x 1 +3x 2 4x 3 = 1 2x 1 +5x 2 x 3 = 5 x 1 +6x 3 = 3 Add 2R 3 to R 1 : Denoted by A 31 (2) 3x 1 +3x 2 +8x 3 = 7 2x 1 +5x 2 x 3 = 5 x 1 +6x 3 = 3 Samy T. Linear equations Differential equations 15 / 81

Elementary operations in matrix form Example of system: x 1 +3x 2 4x 3 = 1 2x 1 +5x 2 x 3 = 5 x 1 +6x 3 = 3 Adding 2R 3 to R 1 : 1 3 4 1 2 5 1 5 1 0 6 3 A 31 (2) 3 3 8 7 2 5 1 5 1 0 6 3 Samy T. Linear equations Differential equations 16 / 81

Row-echelon matrices Definition 2. A m n matrix is row-echelon whenever 1 If there are rows consisting only of 0 s They are at the bottom of the matrix 2 1st nonzero entries of each row have a triangular shape Called leading entries 3 All entries in a column below a leading entry are 0 s Samy T. Linear equations Differential equations 17 / 81

Reduced row-echelon matrices Definition 3. A reduced row-echelon matrix is a matrix A such that 1 A is row-echelon 2 All leading entries are = 1 3 Any column with a leading 1 has zeros everywhere else Samy T. Linear equations Differential equations 18 / 81

Example of row-echelon matrix Row-echelon matrix: 1 1 1 4 0 1 3 5 0 0 1 2 Related system: x 1 +x 2 x 3 = 4 +x 2 3x 3 = 5 x 3 = 2 Solution of the system: very easy thanks to a back substitution x 3 = 2, x 2 = 11, x 1 = 5 Strategy to solve general systems: Reduction to a row-echelon system Samy T. Linear equations Differential equations 19 / 81

Reduction to a row-echelon system Algorithm: 1 Start with a m n matrix A. If A = 0, go to step 7. 2 Pivot column: leftmost nonzero column. Pivot position: topmost position in the pivot column. 3 Use elementary row operations to put 1 in the pivot position. 4 Use elementary row operations to put zeros below pivot position. 5 If all rows below pivot are 0, go to step 7. 6 Otherwise apply steps 2 to 5 to the rows below pivot position. 7 The matrix is row-echelon. Samy T. Linear equations Differential equations 20 / 81

Example of reduction First operations: 3 2 5 2 1 1 1 1 1 1 1 1 1 1 1 1 P 12 3 2 5 2 A 12( 3), A 13 ( 1) 0 1 2 1 1 0 3 4 1 0 3 4 0 1 2 3 Iteration: 1 1 1 1 1 1 1 1 1 1 1 1 M 2 ( 1) 0 1 2 1 A 23(1) 0 1 2 1 M 3(1/4) 0 1 2 1 0 1 2 3 0 0 0 4 0 0 0 1 Samy T. Linear equations Differential equations 21 / 81

Reduced row-echelon matrices Definition 4. A reduced row-echelon matrix is a matrix A such that 1 A is row-echelon 2 All leading entries are = 1 3 Any column with a leading 1 has zeros everywhere else Samy T. Linear equations Differential equations 22 / 81

Example of reduced row-echelon Example: 1 9 26 1 9 26 1 0 8 0 14 28 M 2(1/14) 0 1 2 A 21( 9), A 23 (14) 0 1 2 0 14 28 0 14 28 0 0 0 In particular rank(a) = 2 Remark: One can also 1 Obtain the row-echelon form 2 Get the reduced row-echelon working upward and to the left Samy T. Linear equations Differential equations 23 / 81

Basic idea to solve linear systems Aim: Solve a linear system of equations Strategy: 1 Reduce the augmented matrix to row-echelon 2 Solve the system backward thanks to the row-echelon form Samy T. Linear equations Differential equations 24 / 81

Example of system (1) System: Augmented matrix: 3x 1 2x 2 +2x 3 = 9 x 1 2x 2 +x 3 = 5 2x 1 x 2 2x 3 = 1 3 2 2 9 A = 1 2 1 5 2 1 2 1 Samy T. Linear equations Differential equations 25 / 81

Example of system (2) Row-echelon form of the augmented matrix: 1 2 1 5 A 0 1 3 5 0 0 1 2 System corresponding to the augmented matrix: x 1 2x 2 +x 3 = 5 x 2 +3x 3 = 5 x 3 = 2 Solution set: S = {(1, 1, 2)} Samy T. Linear equations Differential equations 26 / 81

Solving with reduced row-echelon System: Augmented matrix: x 1 +2x 2 x 3 = 1 2x 1 +5x 2 x 3 = 3 x 1 +3x 2 +2x 3 = 6 1 2 1 1 A = 2 5 1 3 1 3 2 6 Samy T. Linear equations Differential equations 27 / 81

Solving with reduced row-echelon (2) Reduced row-echelon form of the augmented matrix: 1 0 0 5 A 0 1 0 1 0 0 1 2 System corresponding to the augmented matrix: x 1 = 5 x 2 = 1 x 3 = 2 Solution set: S = {(5, 1, 2)} Samy T. Linear equations Differential equations 28 / 81

Comparison between reduced and non reduced Pros and cons: For Gauss-Jordan, the final backward system is easier to solve The reduced row-echelon form is costly in terms of computations Overall, Gauss is more efficient than Gauss-Jordan for systems Main interest of Gauss-Jordan: One can compute the inverse of a matrix Samy T. Linear equations Differential equations 29 / 81

Ex. of system with number of solutions System: x 1 2x 2 +2x 3 x 4 = 3 3x 1 +x 2 +6x 3 +11x 4 = 16 2x 1 x 2 +4x 3 +4x 4 = 9 (2) Row-echelon form of the augmented matrix: 1 2 2 1 3 A 0 1 0 2 1 0 0 0 0 0 Consistency: We have r = r = 2 Consistent system with infinite number of solutions Samy T. Linear equations Differential equations 30 / 81

Ex. of system with number of solutions (2) Consistency: From row-echelon form we have 4 variables 2 leading entries No equation of the form 1 = 0 Therefore we have a Consistent system with infinite number of solutions Samy T. Linear equations Differential equations 31 / 81

Ex. of system with number of solutions (3) Rule for systems with number of solutions: Choose as free variables those variables that do not correspond to a leading 1 in row-echelon form of A Application to system (2): Free variables: x 3 = s and x 4 = t Solution set: S = {(5 2s 3t, 1 2t, s, t); s, t R} = {(5, 1, 0, 0) + s ( 2, 0, 1, 0) + t ( 3, 2, 0, 1); s, t R} Samy T. Linear equations Differential equations 32 / 81

Vectors Fact: A tuple u 1 u u = 2. can be geometrically interpreted as a vector Illustration of an addition: u n Samy T. Linear equations Differential equations 34 / 81

Algebraic operations on vectors (1) 1 Commutativity of addition: For all u, v R n, u + v = v + u 2 Associativity of addition: For all u, v, w R n, (u + v) + w = u + (v + w) 3 Existence of a zero vector: There exists 0 R n such that v + 0 = v 4 Existence of additive inverses in R n : For all v R n, there exists v R n such that v + ( v) = 0 Samy T. Linear equations Differential equations 35 / 81

Algebraic operations on vectors (2) 5 Unit property: For all v R n, we have 1 v = v 6 Associativity of scalar multiplication: (rs)v = r(s v) 7 Distributivity 1: 8 Distributivity 2: r(u + v) = r u + r v (r + s)v = r v + s v Samy T. Linear equations Differential equations 36 / 81

Linear combinations Definition 5. Let v 1,..., v p family of vectors in R n. c 1,..., c p family of scalars in R Then a linear combination of v 1,..., v p with weights c 1,..., c p is y = c 1 v 1 + + c p v p, Samy T. Linear equations Differential equations 37 / 81

Example in R 2 Example: Consider the vectors of R 2 : Then v 1 = Set of linear combinations: [ ] 1, and v 1 2 = 2v 1 v 2 = [ ] 4 3 [ ] 2 1 Samy T. Linear equations Differential equations 38 / 81

Spanning subset of R n Definition 6. Let v 1,..., v p family of vectors in R n. Consider the collection of x R n which can be written as a linear combination x = c 1 v 1 + + c k v k, we say that x Span {v 1,..., v p } Samy T. Linear equations Differential equations 39 / 81

Example in R 3 Illustration of spans: Span{v} (left) and Span{u, v} (right) Samy T. Linear equations Differential equations 40 / 81

Matrix Definition 7. A m n matrix is a rectangular array of numbers with m rows and n columns Example of 2 3 matrix: A = [ 3 2 2 3 1 5 0 5 4 3 7 ] Samy T. Linear equations Differential equations 42 / 81

Index notation Recall: we have [ 3 2 A = 0 5 3 4 7 Index notation: For the matrix A we have 2 3 1 5 ] a 12 = 2 3, a 21 = 0, a 23 = 3 7 Index notation for a m n matrix: a 11 a 12 a 13... a 1n a A = 21 a 22 a 23... a 2n....... a m1 a m2 a m3... a mn Samy T. Linear equations Differential equations 43 / 81

Vectors A row 3-vector: a = [ 2 3 1 5 ] 4 7 A column 5-vector: 1 4 b = π 67 3 Samy T. Linear equations Differential equations 44 / 81

Multiplication as a linear combination Definition 8. Let A a m n matrix x R n Then x 1 Ax = [ ] x a 1 a 2 a n 2. = x 1a 1 + + x n a n x n Samy T. Linear equations Differential equations 45 / 81

Example Matrices: Product: A = [ 1 2 0 2 ] Ax = and x = [ 0 2 ] [ 2 1 ] Samy T. Linear equations Differential equations 46 / 81

Rules for multiplications Rules to follow: A (u + v) = Au + Av A (cu) = c (Au) A 0 = 0 Distributive law Associative law Absorbing state Samy T. Linear equations Differential equations 47 / 81

Vector formulation Example of system in R 3 : x 1 +3x 2 4x 3 = 1 2x 1 +5x 2 x 3 = 5 x 1 +6x 3 = 3 Related matrices: 1 3 4 1 A = 2 5 1, b = 5, x = 1 0 6 3 Vector formulation of the system: Ax = b x 1 x 2 x 3 Samy T. Linear equations Differential equations 48 / 81

Existence of solutions Theorem 9. Let A a m n matrix Then the following statements are equivalent 1 For each b R m, the system Ax = b has a solution 2 Each b R m is a linear combination of columns of A 3 The columns of A span R m 4 A has a pivot in every row Samy T. Linear equations Differential equations 49 / 81

Example Matrix of a generalized system: 1 3 4 b 1 A = 4 2 6 b 2 3 2 7 b 3 Row-echelon form: 1 3 4 b 1 A 0 14 10 b 2 + 4b 1 0 0 0 b 1 1b 2 2 + b 3 Conclusion: There is a solution only if b 1 1 2 b 2 + b 3 = 0 This is due to the fact that we have a pivot on 2 rows only The columns of A don t span R 3 Samy T. Linear equations Differential equations 50 / 81

Homogeneous systems Proposition 10. For a m n matrix A, consider the system Then we have A x = 0. 1 The system is always consistent with trivial solution x = 0. 2 The system has a nontrivial solution iff the equation has at least one free variable iff a row with no pivot in row-echelon form of A Samy T. Linear equations Differential equations 52 / 81

Example of homogeneous system Matrix: System: Augmented matrix: 0 2 3 A = 0 1 1 0 3 7 A x = 0. 0 2 3 0 A = 0 1 1 0 0 3 7 0 Samy T. Linear equations Differential equations 53 / 81

Example of homogeneous system (2) Reduced echelon form of A : 0 1 0 0 0 0 1 0 0 0 0 0 Number of solutions: No pivot in 3rd row infinite number of solutions Set of solutions: S = {(s, 0, 0); s R}. Samy T. Linear equations Differential equations 54 / 81

2nd example of homogeneous system Matrix: System: Augmented matrix: 3 5 4 A = 3 2 4 6 1 8 A x = 0. 3 5 4 0 A = 3 2 4 0 6 1 8 0 Samy T. Linear equations Differential equations 55 / 81

2nd example of homogeneous system (2) Reduced echelon form of A : 1 0 4 0 3 0 1 0 0 0 0 0 0 Number of solutions: No pivot in 3rd row infinite number of solutions Set of solutions: for t R, of the form x = t 4 3 0 1 Samy T. Linear equations Differential equations 56 / 81

Inhomogeneous systems Theorem 11. Assume Equation Ax = b is consistent p is a particular solution of Ax = b Then the general solution of Ax = b can be written as w = p + v h, where v h is the general solution of Ax = 0 Samy T. Linear equations Differential equations 57 / 81

General solution for inhomogeneous systems Solution of Ax = b vs solution of Ax = 0: Samy T. Linear equations Differential equations 58 / 81

Example of inhomogeneous system Matrices: System: 3 5 4 7 A = 3 2 4, b = 1 6 1 8 4 A x = b. Augmented matrix: 3 5 4 7 A = 3 2 4 1 6 1 8 4 Samy T. Linear equations Differential equations 59 / 81

Example of inhomogeneous system (2) Reduced echelon form of A : 1 0 4 1 3 0 1 0 2 0 0 0 0 Number of solutions: No pivot in 3rd row, no equation of the form 0 = 1 infinite number of solutions Set of solutions: of the form 4 1 3 x = 2 + t 0 0 1 Samy T. Linear equations Differential equations 60 / 81

Definition Definition 12. Let v 1,..., v p be a family of vectors in R n. Then 1 If there exist c 1,..., c p not all zero such that c 1 v 1 + + c p v p = 0, we say that {v 1,..., v p } is linearly dependent 2 If we have c 1 v 1 + + c p v p = 0 = c 1 = c 2 = = c p = 0, we say that {v 1,..., v p } is linearly independent Samy T. Linear equations Differential equations 62 / 81

Examples Simple examples: The family {v} is linearly dependent iff v = 0 The family {v 1, v 2 } is linearly dependent iff v 2 = c v 1 If 0 is an element of {v 1,..., v p } then v 1,..., v p are linearly dependent If p > n and v 1,..., v p R n then v 1,..., v p are linearly dependent Samy T. Linear equations Differential equations 63 / 81

Example in R 3 Family of vectors: We consider v 1 = (1, 2, 1), v 2 = (2, 1, 1), v 3 = (8, 1, 1) Then v 1, v 2, v 3 are linearly dependent Proof: The system can be written as: c 1 v 1 + c 2 v 2 + c 3 v 3 = 0 c 1 +2c 2 +8c 3 = 0 2c 1 c 2 +c 3 = 0 c 1 +c 2 +c 3 = 0 Samy T. Linear equations Differential equations 64 / 81

Example in R 3 (2) Proof (ctd): System in row-echelon form 1 2 8 0 0 1 3 0 0 0 0 0 One row is 0, so that we have linear dependence Explicit linear dependence: We solve for c c 3 = t, c 2 = 3t, c 1 = 2t Then choosing (arbitrary choice) t = 1 we get 2v 1 3v 2 + v 3 = 0 Samy T. Linear equations Differential equations 65 / 81

Linear dependence and systems Proposition 13. Let Then v 1,..., v p be a family of vectors in R n. A = [v 1,..., v p ] defined as a M n,p (R) matrix The vectors v 1,..., v p are linearly dependent iff the system Ax = 0 has a nontrivial solution Samy T. Linear equations Differential equations 66 / 81

Example of application Family of vectors: We consider v 1 = (1, 2, 1), v 2 = (2, 1, 1), v 3 = (8, 1, 1) Recall: We have obtained a row-echelon form 1 2 8 0 0 1 3 0 0 0 0 0 One row is 0, so that the system Ax = 0 has a nontrivial solution Conclusion: v 1, v 2, v 3 are linearly dependent Samy T. Linear equations Differential equations 67 / 81

Definition of mapping Definition 14. Let Then: n and m two integers 1 A mapping is a rule which assigns to each vector v R n a vector w = T (v) R m 2 Notation: T : R n R m Samy T. Linear equations Differential equations 69 / 81

Linear transformation Definition 15. Let n and m two integers A mapping T : R n R m Then T is a linear transformation if it satisfies: 1 T (u + v) = T (u) + T (v) 2 T (c v) = c T (v) Samy T. Linear equations Differential equations 70 / 81

Domain and codomain Vocabulary: For T : R n R m linear R n is called domain of T R m is called codomain of T T (R n ) R m is called range of T Samy T. Linear equations Differential equations 71 / 81

Examples of linear transformations Projection: Domain = R 3, Codomain = R 3, Range = Plane Rotation: Domain = R 2, Codomain = R 2, Range = R 2 Samy T. Linear equations Differential equations 72 / 81

Matrix form Theorem 16. Let T : R n R m be a linear transformation (e 1,..., e n ) canonical basis of R n Then T is described by the matrix transformation T (x) = A x, where A is the matrix defined by A = [T (e 1 ), T (e 2 ),..., T (e n )] Samy T. Linear equations Differential equations 74 / 81

Example of matrix form Linear transformation: Defined as T : R 2 R 3 such that T (e 1 ) = T (1, 0) = (2, 3, 1), T (e 2 ) = T (0, 1) = (5, 4, 7) Matrix form: T (x) = A x with A = [T (e 1 ), T (e 2 )] = 2 5 3 4 1 7 General expression: 2x 1 + 5x 2 T (x) = A x = 3x 1 4x 2 x 1 + 7x 2 Samy T. Linear equations Differential equations 75 / 81

Examples of transformations in R 2 Reflection on x 1 -axis: A 1 = [ ] 1 0 0 1 Samy T. Linear equations Differential equations 76 / 81

Examples of transformations in R 2 (2) Reflection on line x 2 = x 1 : A 2 = [ ] 0 1 1 0 Samy T. Linear equations Differential equations 77 / 81

Examples of transformations in R 2 (3) Rotation: [ ] cos(ϕ) sin(ϕ) A 3 = sin(ϕ) cos(ϕ) Samy T. Linear equations Differential equations 78 / 81

Composition of transformations in R 2 Reflection on x 1 then reflection on x 2 = x 1 : A = A 2 A 1 = [ ] [ ] 0 1 1 0 = 1 0 0 1 [ ] 0 1 1 0 Resulting transformation: Rotation with ϕ = π 2 Samy T. Linear equations Differential equations 79 / 81

Properties of T according to its matrix Theorem 17. Let T : R n R m linear transformation A its standard matrix form Then 1 T is onto iff columns of A span R m 2 T is one-to-one iff columns of A are linearly independent Samy T. Linear equations Differential equations 80 / 81

Example from R 2 to R 3 Transformation: For x R 2 3x 1 + x 2 T (x) = 5x 1 + 7x 2 x 1 + 3x 2 Matrix: 3 1 A = 5 7 1 3 Properties of T : 1 Only two columns in A = is not onto in R 3 2 Columns of A are independent = T is one-to-one Samy T. Linear equations Differential equations 81 / 81