Lectures on Linear Algebra for IT by Mgr. Tereza Kovářová, Ph.D. following content of lectures by Ing. Petr Beremlijski, Ph.D. Department of Applied Mathematics, VSB - TU Ostrava Czech Republic
2. Systems o Linear Equations 1. Systems of Linear Equations 2. Matrix Notation 3. Elementary Row Operations 4. Echelon Forms 5. Pivots 6. Solutions of Linear System 7. Gaussian Elimination 8. Gauss-Jordan Elimination 9. Numerical Notes
2.1 System of Linear Equations Definition 1 A system of m linear equations with n variables x 1,..., x n is a set of equations of the form: a 11 x 1 + + a 1n x n = b 1.. =. (S) a m1 x 1 + + a mn x n = b m where a ij for i = 1,..., m, j = 1,..., n are coefficients and b i for i = 1,..., m are right sides of the system (S).
2.1 System of Linear Equations Example 1 System of linear equations from the Application III. 2u 1 u 2 = 0.1112 u 1 +2u 2 u 3 = 0.1112 u 2 +2u 3 u 4 = 0.1112 u 3 +2u 4 u 5 = 0.1112 u 4 +2u 5 = 0.1112 The system has 5 equations with 5 variables u 1,..., u 5...
2.1 System of Linear Equations A solution of the system is a list (s 1,..., s n ) of numbers that makes each equation a true statement when substituted for variables x 1,..., x n respectively. The set of all possible solutions is called the solution set of the linear system. Two linear systems are called equivalent if they have the same solution set. A system of linear equations has 1. no solution, or 2. exactly one solution, or 3. infinitely many solutions A linear system i said to be consistent if it has either one solution or infinitely many solutions. A system is inconsistent if it has no solution.
2.2 Matrix Notation The essential information of a linear system (S) can be recorded compactly in a rectangular array that is called the augmented matrix of the system. a 11... a 1n b 1...... a m1... a mn b m. The matrix A and the vector b, a 11... a 1n A =..... a m1... a mn, b = b 1. b m, are called the coefficient matrix and the right side of the system (S). If the vector of variables x 1,..., x n is denoted by x = [x 1,..., x n ] we can write the system as the matrix equation: Ax = b.
2.3 Elementary Row Operations Systematic procedure for solving linear system will be described next. The basic strategy is to replace one system with an equivalent system (i.e., one with the same solution set) that is easier to solve. Definition 2 By elementary row operations we understand the following basic operations on equations (rows) of the system: E1 (Interchange) Interchange two rows. E2 (Scaling) Multiply all entries in a row by a nonzero constant. E3 (Replacement) Replace one row by the sum of itself and a multiple of another row.
2.3 Elementary Row Operations Row operations can be applied to any matrix, not merely to one that corresponds to an augmented matrix of some linear system. Definition 3 Two matrices are called row equivalent if there is a sequence of elementary row operations that transforms one matrix into the other. Important is to mention that row operations are reversible. If two rows of the system (S) are interchanged, they can be returned to the original position by another interchange. If a row is scaled by a nonzero constant c, then multiplying the new row by 1/c produces the original row. If a row j is replaced by the sum of c-multiple of a row i with the row j, to reverse the operation, add c times the row i to a (new) row j and obtain the original row j.
Example 2 2.3 Elementary Row Operations Elimination procedure with and without matrix notation: 2x 1 + x 2 = 0 [ 2 1 0 x 1 3x 2 = 10 1 3 10 ] r1 r 2 We keep x 1 in one of the equations and eliminate it from the other equation. To do so, interchange equation 1 with equation 2 (row operation E1) to obtain first equation with coefficient 1 for x 1. x 1 3x 2 = 10 2x 1 + x 2 = 0 [ 1 3 10 2 1 0 ] r 2 + 2r 1 Now add 2 times equation 1 to equation 2 and write the result of this operation (row operation E3) in place of the second equation. [ ] x 1 3x 2 = 10 1 3 10 5x 2 = 20 0 5 20 1 5 r 2 To simplify the system even more we can scale the second equation by the constant 1 (row operation E2). 5 [ ] x 1 3x 2 = 10 1 3 10 x 2 = 4 0 1 4 From the second equation x 2 = 4. After substituting for x 2 into the first equation we obtain x 1 = 2.
2.3 Elementary Row Operations Theorem 1 If the augmented matrices of two linear systems are row equivalent, then the two systems are equivalent (have the same solution set). Warning! If we apply an operation on a matrix that is not transformed properly, we can generate errors. For instance the following way of performing row operations does not produce equivalent system. 1 1 1 1 2 1 1 0 2 0 1 1 r 3 r 2 1 1 1 1 0 1 0 1 0 1 0 1 +r 2 1 1 1 1 0 1 0 1 0 0 0 0 To avoid such mistakes when performing replacement operations, we select one row to be unchanged and add its multiples to another rows as convenient. In the next example row operations already yield the system that is equivalent. 1 1 1 1 2 1 1 0 2 0 1 1 2r 1 2r 1 1 1 1 1 0 1 1 2 0 2 1 1 2r 2 1 1 1 1 0 1 1 2 0 0 1 3
2.3 Elementary Row Operations We will use the row operations on augmented matrix to determine the answer to the following two questions: Is the system consistent? (Does at least one solution exist?) If a solution exists, is it the only one? (Is the solution unique?)
2.4 Echelon Forms Definition 4 A matrix is in echelon form (or row echelon form) if: 1. All nonzero rows are above any rows of all zeros. 2. Each leading entry (the leftmost nonzero entry) of a nonzero row is strictly to the right of a leading entry of the row above it. These two conditions imply, that all entries in a column below a leading entry are zeros. A matrix in an echelon form is in the reduced row echelon form (or row canonical form) if: 3. The leading entry in each nonzero row is 1 and it is the only nonzero entry in it s column. Example 3 row echelon form 1 4 1/3 7 0 0 0 2 2 3 0 0 0 0 1/5 reduced row echelon form 1 0 0 1 2 3 0 0 1 3 0 0 0 1 4
Example 4 2.4 Echelon Forms Given matrices are in echelon form. The leading entries ( ) have any nonzero value. The starred entries (*) may have any value (including zero). 0 0 0 0 0 0 0 0 0 0 The next matrices are corresponding matrices in reduced echelon form. 1 0 0 0 1 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 1 Theorem 2 Any nonzero matrix can be row reduced (transformed by elementary row operations) into more than one matrix in echelon form. Uniqueness of the Reduced Echelon Form Each matrix is row equivalent to one and only one reduced echelon matrix.
2.5 Pivots Definition 5 A pivot position in a matrix A is a location in A that corresponds to a leading 1 in the reduced echelon form of A. A pivot column is a column of A that contains a pivot position. A pivot is a nonzero entry in a pivot position that is used as needed to create zeros via row operations. Example 5 1 0 2 0 1 0 1 3 0 3 0 0 0 1 4 1 4 1/3 7 0 0 0 2 2 3 0 0 0 0 1/5
2.5 Pivots Definition 6 A pivot position in a matrix A is a location in A that corresponds to a leading 1 in the reduced echelon form of A. A pivot column is a column of A that contains a pivot position. A pivot is a nonzero entry in a pivot position that is used as needed to create zeros via row operations. Example 6 1 0 2 0 1 0 1 3 0 3 0 0 0 1 4 1 4 1/3 7 0 0 0 2 2 3 0 0 0 0 1/5 }{{} pivot columns Pivots at pivot positions are displayed in red. Many fundamental concepts of basic LA can be connected with pivot positions in a matrix.
Example 7 2.5 Pivots Location of pivot columns and pivot positions of A. 0 3 6 4 9 1 r 4 1 2 1 3 1 2 3 0 3 1 r 1 4 5 9 7 1 4 5 9 7 0 2 4 6 6 0 5 10 15 15 0 3 6 4 9 1 4 5 9 7 0 2 4 6 6 0 0 0 5 0 0 0 0 0 0 A = 5 2 r 2 + 3 2 r 2 1 4 5 9 7 1 2 1 3 1 2 3 0 3 1 0 3 6 4 9 1 4 5 9 7 0 2 4 6 6 0 0 0 0 0 0 0 0 5 0 +r 1 +2r 1 r3 r 4 The left matrix is an echelon form of A, thus reveals all the pivots (in red) and pivot columns. Below are displayed the pivot positions (in blue) and pivot columns in A. 0 3 6 4 9 1 2 1 3 1 2 3 0 3 1 1 4 5 9 7
2.6 Solutions of Linear System Theorem 3 A linear system is inconsistent if and only if the right most column of the augmented matrix is a pivot column - that is, if and only if an echelon form of the augmented matrix has a row of the form [ 0... 0 b ] with b nonzero. Example 8 (System with No Solution) An augmented matrix of a system was row reduced to the following echelon form: 1 2 1 1 0 1 1 2 0 0 0 3 This matrix corresponds to the equivalent reduced system: x 1 + 2x 2 x 3 = 1 x 2 x 3 = 2 0 = 3 The last equation 0 = 3 can not be satisfied for any choice of numbers for x 1, x 2, x 3. The system has no solution (is inconsistent).
2.6 Solutions of Linear System Example 9 (System with Unique Solution) An augmented matrix of a system was row reduced to the following echelon form: 1 2 1 1 0 1 1 2 0 0 1 3 The echelon form of an augmented matrix corresponds to the system: x 1 +2x 2 x 3 = 1 x 2 x 3 = 2 x 3 = 3 x 1 +2x 2 ( 3) = 1 x 2 ( 3) = 2 x 1 +2x 2 = 2 x 2 = 1 x 1 + 2( 1) = 2 x 1 = 0 The system has unique solution x 1 = 0, x 2 = 1, x 3 = 3. Notice: All the columns of the coefficient matrix are the pivot columns. This fact together with the consistency of the system implies that the solution is unique.
2.6 Solutions of Linear System Example 10 (System with Infinitely Many Solutions) An augmented matrix of a system was row reduced to the following echelon form: 1 1 1 1 1 0 0 1 1 2 0 0 0 0 0 The echelon form of an augmented matrix corresponds to the system: x 1 +x 2 x 3 +x 4 = 1 x 3 x 4 = 2 x 1 = 1 x 2 + x 3 x 4 x 0 = 0 3 = 2 + x 4 x 1 = 1 x 2 + (2 + x 4 ) x 4 = 3 x 2 The system has infinitely many solutions x 1 = 3 x 2, x 3 = 2 + x 4, we may choose x 2, x 4 arbitrarily. The variables x 1 and x 3 corresponding to pivot columns are called basic variables. The other variables x 2, x 4 are called free variables. The system has infinitely many solutions if it is consistent and when there is at least one free variable.
2.7 Gaussian Elimination Gaussian Elimination (row reduction algorithm) 1. Forward Phase (Row reduction of augmented matrix into an echelon form.) 2. Back Substitution Example 11 (Inconsistent Linear System ) Solve the next linear system: 2x 1 + x 2 = 2 x 1 + 2x 2 x 3 = 1 4x 1 + 5x 2 2x 3 = 1 By a sequence of elementary row operations we obtain: 2 1 0 2 1 2 1 1 4 5 2 1 2 r 1 2r 1 2 1 0 2 0 3 2 0 0 3 2 5 r 2 2 1 0 2 0 3 2 0 0 0 0 5 The last equation 0x 1 + 0x 2 + 0x 3 = 5 has no solution for any choice of numbers for variables x 1, x 2, x 3. Therefore the system is inconsistent.
2.7 Gaussian Elimination Example 12 (Consistent Linear System with Unique Solution) Solve the linear system: 2x 2 + 3x 3 = 2 x 2 + x 3 = 0 x 1 + x 3 = 4 By a sequence of elementary row operations we obtain: 0 2 3 2 0 1 1 0 1 0 1 4 r 3 r 1 1 0 1 4 0 1 1 0 0 2 3 2 2r 2 1 0 1 4 0 1 1 0 0 0 1 2 Last column of the augmented matrix is not a pivot column and so the system is consistent. Each column of the coefficient matrix is a pivot column (there are no free variables) and so the system has a unique solution. By back substitution we obtain the only solution: x 3 = 2 x 2 = x 3 = 2 x 1 = 4 x 3 = 2
2.7 Gaussian Elimination Example 13 (Consistent System with Infinitely Many Solutions) Solve the linear system: x 1 + x 2 + x 3 = 1 x 1 x 3 = 1 x 2 + 2x 3 = 0 By a sequaence of elementary row operations we obtain: 1 1 1 1 1 0 1 1 0 1 2 0 r 1 1 1 1 1 0 1 2 0 0 1 2 0 +r 2 1 1 1 1 0 1 2 0 0 0 0 0 Last column of the augmented matrix is not a pivot column and so the system is consistent. Only two from three columns of the coefficient matrix are pivot columns. Therefore there is one free variable and so the system has infinitely many solutions. The free variable corresponds to the no pivot column which is x 3. We can choose any value for x 3 and so we set x 3 = t, where t R is a parameter. Further we proceed by back substitution to obtain all the solutions.
Example 13 (continuation) 2.7 Gaussian Elimination The linear system corresponding to the echelon form of the augmented matrix of the original system is: x 1 + x 2 + x 3 = 1 (1) x 2 2x 3 = 0 (2) 0 = 0 (3) From the equation (2) we express x 2 in terms of x 3, it is x 2 = 2x 3. Then after substituing x 2 into equation (1) we obtain x 1 = 1 + x 3. The free variable x 3 acts as a parameter. We can set x 3 = t and write all the solutions as: x 1 = 1 + t x 2 = 2t x 3 = t Note: Whenever a system is consistent and has free variables, the solution set has many parametric descriptions. For inst. in previous system we could treat x 2 as a parameter and set x 2 = p. Then we obtain x 3 = 1 2 p, x 1 = 1 p + 1, which is 2 different parametric description of the same solution set.
Gauss Jordan Elimination 2.8 Gauss Jordan Elimination 1. Forward Phase (Row reduction of augmented matrix into an echelon form.) 2. Backward Phase (Row reduction of augmented matrix into reduced echelon form (canonical)) (a) scaling rows by reciprocal of leading entries (pivots become 1 s) (b) working upward to the left, creating zeros in columns above each pivot via elementary row operations Example 14 For instance we can continue with the Backward Phase from the echelon form of the augmented matrix from Example 12: 1 0 1 4 0 1 1 0 r 3 r 3 1 0 0 2 0 1 0 2 0 0 1 2 0 0 1 2 Solution of the system is in the last column. Corresponding equations are x 1 = 2, x 2 = 2 and x 3 = 2.
2.9 Numerical Notes Gaussian elimination is effective when solving small linear systems by hand, but also for computer programs solving systems of hundreds up to thousands of equations. The method is very effective also for solving systems of greater number of equations that have some special structure of nonzero elements. For linear systems with greater number of equations exist more effective methods which are developed currently. Gaussian elimination is not suitable for parallel computer implementation. Efficiency of an algorithm based on Gaussian Elimination (m = n): (It is usually measured in flops floating point operations) 1. Forward phase can take 1 6 (2n + 1)(n + 1)n multiplications, which is approximately 1 3 n3 for n moderately large say n 30. 2. Backward phase (or back substitution) can take 1 2 n(n 1) multiplications, it is approximately 1 2 n2 for n large.