Copyright 2005, W.R. Winfrey
Topics Preliminaries Echelon Form of a Matrix Elementary Matrices; Finding A -1 Equivalent Matrices LU-Factorization
Topics Preliminaries Echelon Form of a Matrix Elementary Matrices; Finding A -1 Equivalent Matrices LU-Factorization
Preliminaries The primary goal is to deal with loose ends from the first set of lectures Develop systematic procedure for solving linear systems that works directly with the matrix form Establish conditions for a solution to a linear system to exist Establish conditions for a unique solution to a linear system Re-examine solutions to homogeneous systems Develop a systematic procedure for computing A 1 Discover equivalent conditions to nonsingularity Will also examine a special technique for solving linear systems: LU decomposition
Topics Preliminaries Echelon Form of a Matrix Elementary Matrices; Finding A -1 Equivalent Matrices LU-Factorization
Echelon Form of a Matrix Solve a system using elimination of variables x 1 x 2 2x 3-1 x 1 x 2 2x 3-1 x 1-2x 2 x 3-5 0-3x 2 - x 3-4 3x 1 x 2 x 3 3 0-2x 2-5x 3 6 x 1 x 2 2x 3-1 0 3x 2 x 3 4 0 2x 2 5x 3-6 x 1 x 2 2x 3-1 0 6x 2 2x 3 8 0 6x 2 15x 3-18 x 1 x 2 2x 3-1 0 6x 2 2x 3 8 0 0 13x 3-26 x 1 x 2 2x 3-1 0 6x 2 2x 3 8 0 0 x 3-2
Echelon Form of a Matrix Continuing the process on the previous slide gives x 1 1, x 2 2 and x 3-2 In matrix form, started with and transformed it into 1 1 2 x 1-1 1-2 1 x -5 3 1 1 x 3 2 3 1 1 2 x 1-1 1 0 0 x 1 1 0 1 1 3 x 4 3 0 1 0 x 2 0 0 1 x -2 0 0 1 x -2 2 2 3 3 Row echelon form Reduced row echelon form
Echelon Form of a Matrix Soon, will express the process as Pivot Columns 1 1 2-1 1 1 2-1 1 1 2-1 1-2 1-5 0-3 -1-4 0-3 -1-4 3 1 1 3 0-2 -5 3 0 0-13 3 26 3 Pivots 1 1 2-1 1 0 0 1 0 1 1 3 4 3 0 1 0 2 0 0 1-2 0 0 1-2
Echelon Form of a Matrix Defn - An mxn matrix A is said to be in reduced row echelon form if it has the following properties a) If any rows consist of entirely zeros, they are at the bottom of the matrix b)if a row does not consist entirely of zeros, the first nonzero entry in that row is 1 c) If rows i and i + 1 are two successive rows which do not consist entirely of zeros, then the first nonzero entry of row i + 1 is to the right of the first nonzero entry of row i d)if a column contains the first nonzero entry of some row, then all the other entries in that column are zero Defn - If an mxn matrix A satisfies properties (a), (b) and (c), it is said to be in row echelon form
Echelon Form of a Matrix Examples - Row Echelon Form 1 5 0 2-2 4 0 1 0 3 4 8 0 0 0 1 7-2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 3 5 7 9 0 0 0 0 1-2 3 0 0 0 0 0 1 2 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
Echelon Form of a Matrix Examples - Reduced Row Echelon Form 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 1 2 0 0 1 0 0 1 2 3 0 0 0 0 0 1 0 0 0-2 4 0 1 0 0 4 8 0 0 0 1 7-2 0 0 0 0 0 0 0 0 0 0 0 0
Echelon Form of a Matrix Examples - Not Reduced Row Echelon Form 1 2 0 4 0 0 0 0 0 0 1-3 1 0 3 4 0 2-2 5 0 0 1 2 1 0 3 4 0 1-2 5 0 1 2 2 0 0 0 0 1 2 3 4 0 1-2 5 0 0 1 2 0 0 0 0
Echelon Form of a Matrix Comments - Row Echelon Form Can define column echelon form and reduced column echelon form in a similar manner. Basically, they are just the transpose of the corresponding row forms Close connection between echelon forms and the solution of linear equations
Echelon Form of a Matrix Defn - An elementary row operation on a matrix A is any one of the following operations a) Interchange rows i and j of matrix A b)multiply row i of A by c 0 c) Add c times row i of A to row j of A, i j
Echelon Form of a Matrix Comment The first elementary row operation, exchange two rows, is needed to deal with zeros in the pivot positions. For example, consider 2x2 1 0 2 x1 1 0 2 1 3x14 x22 3 4 x 2 2 3 4 2 No multiple of the first row will remove the 3 in the second row. The solution with equations is just to exchange the equations. The solution with matrices is just to exchange the rows. 3x14 x22 3 4 x1 2 3 4 2 2x2 1 0 2 x 2 1 0 2 1 This problem can also occur as an intermediate step in solving a larger system and has the same fix
Echelon Form of a Matrix Comment - The elementary row operations on an mxn matrix A can be accomplished by multiplying A by a specially chosen mxm matrix. This is useful for proofs, but not for practical computations since too many operations are required.
Echelon Form of a Matrix Elementary Row Operations Interchange rows i and j of matrix A 0 1 0 a a a a a a a a 1 0 0 a a a a a a a a 0 0 1 a a a a a a a a 11 12 13 14 21 22 23 24 21 22 23 24 11 12 13 14 31 32 33 34 31 32 33 34 In general, rows i and j of A can be interchanged by multiplying A by the mxm matrix E defined as 1 if k i and k j epq 0 except for ekk 0 if k i or k j and e 1, e 1 E is a special case of a permutation matrix ij ji
Echelon Form of a Matrix Elementary Row Operations Multiply row i of A by c 0 1 0 0 11 12 13 14 11 12 13 14 0 c 0 a21 a22 a23 a24 ca21 ca22 ca23 ca24 0 0 1 a a a a a a a a a a a a a a a a 31 32 33 34 31 32 33 34 In general, row i of A can be multiplied by c by multiplying A by the mxm matrix E defined as 0 if p q epq 1 if p q i c if p q i
Echelon Form of a Matrix Elementary Row Operations Add c times row i of A to row j of A, i j 1 0 0a11 a12 a13 a 14 0 1 0 a21 a22 a23 a24 0 c 1 a a a a 31 32 33 34 a11 a12 a13 a 14 a21 a22 a23 a24 ca21 a31 ca22 a32 ca23 a33 ca24 a 34 In general, can add c times row c p j and q i i to row j of A by multiplying epq 1 p q A by the mxm matrix E defined as 0 otherwise
Echelon Form of a Matrix Comments The appropriate elementary matrix E may be obtained by performing the desired operation on the rows of the mxm identity matrix I m Can define elementary matrices F to perform elementary column operations by doing the corresponding operations on the columns of the identity matrix. The matrices F are applied by post multiplying, i.e. AF
Echelon Form of a Matrix Defn - An mxn matrix A is row equivalent to an mxn matrix A if A can be obtained by applying a finite sequence of elementary row operations to A The basic method for solving the system AX B is to apply elementary row operations to the augmented matrix [A:B] to get a new augmented matrix [C:D] such that the CX D is easy to solve If [C:D] is in row echelon form, the method is called Gaussian Elimination If [C:D] is in reduced row echelon form, the method is called Gauss-Jordan Elimination
Echelon Form of a Matrix Comments In first set of lectures, we argued that the solution to a system of linear equations is unchanged if we do any of the following a) Interchange two equations b)multiply a given equation by a nonzero constant c) Add a constant times one equation to another equation When the system is viewed in terms of matrices, the above operations become elementary row operations on the augmented matrix.
Echelon Form of a Matrix Theorem - Every nonzero mxn matrix A is row equivalent to a matrix in row echelon form Proof Let A = [ a ij ] be a nonzero mxn matrix. Find the first (from the left) nonzero column, s, then find the row, r, containing the first (from the top) nonzero entry in column s. Define a matrix B = [ b ij ] as follows. If r = 1, B = A. If r 1, interchange rows 1 and r of A to get B. Then b 1s 0. Multiply the first row of B by 1/b 1s to get a matrix C = [ c ij ] such that c 1s = 1. Consider the other entries in column s. If c hs 0, 2 h m, add -c hs times the first row to row h.
Echelon Form of a Matrix Proof (continued) - The elements of column s in rows 2, 3,, m of C are now zero. Call the resulting matrix D. Consider the (m-1)xn submatrix A 1 of D obtained by deleting the first row of D. Repeat the above process with A 1 in place of A. Continuing this process yields a matrix in row echelon form that is row equivalent to A. QED
Echelon Form of a Matrix Theorem - Every nonzero mxn matrix A is row equivalent to a matrix in reduced row echelon form Proof - Let A be a nonzero mxn matrix. Apply the method of the preceding theorem to obtain a matrix H in row echelon form that is row equivalent to A. Suppose that rows 1, 2,, r of H are nonzero and that the leading ones in these rows occur in columns c 1, c 2,, c r, where c 1 < c 2 < L < c r.
Echelon Form of a Matrix Proof (continued) - Starting with the last nonzero row of H, add suitable multiples of this row to all rows above it to create zeros in column c r above the one in row r. Repeat this process with rows r 1, r 2,, 2 to make all entries above a leading one equal to zero. This gives a matrix in reduced row echelon form that is row equivalent to H, which is row equivalent to A. QED Note: Can show that the reduced row echelon matrix is unique
Echelon Form of a Matrix Theorem - Let A and C be mxn matrices. If the system AX b has an augmented matrix [A:b] which is row equivalent to an augmented matrix [C:d], then the systems AX b and CX d have the same solution. Proof - The elementary row operations on the augmented matrices correspond to the following operations on the equations, which leave the solution unchanged. a) Interchange two equations b) Multiply a given equation by a nonzero constant c) Add a constant times one equation to another equation So, the two systems have the same solution QED
Echelon Form of a Matrix Corollary - If A and C are row equivalent mxn matrices, then the homogeneous systems AX 0 and CX 0 are equivalent. Proof - Form the augmented matrices [A:0] and [C:0]. Since A and C are row equivalent, [A:0] and [C:0] are row equivalent. By the preceding theorem, AX 0 and CX 0 have the same solution. QED
Echelon Form of a Matrix Comments If Gauss-Jordan reduction is used, the solution is just read off. If Gaussian elimination is used, the solution must be found by back substitution Although we have talked about the solution, we have not established any conditions that ensure that a solution will exist or that it will be unique
Echelon Form of a Matrix Example 1 2 3 4 5 6 0 1 2 3-1 7 0 0 1 2 3 7 0 0 0 1 2 9 : CD x1 2x2 3x3 4x4 5x5 6 x2 2x3 3x4 - x5 7 x3 2x4 3x5 7 x 2x 9 4 5 Set x 5 = r and backsolve x1-1-10r x2 25r x3-11 r x4 9-2r Infinite number of solutions
Echelon Form of a Matrix Example 1 2 3 4 5 CD : 0 1 2 3 6 0 0 0 0 1 The last equation is 0x 1 +0x 2 +0x 3 +0x 4 1, which cannot be satisfied Clearly, several things can happen and a careful and organized analysis of the problem is required
Echelon Form of a Matrix Homogeneous Systems Homogeneous system, AX 0, of m equations in n unknowns occurs often applications Example 1 0 0 0 2 0 0 0 1 0 3 0 0 0 0 1 4 0 0 0 0 0 0 0 Set x 5 r, then determine x 4-4r, x 3-3r and x 1-2r. There is no equation determining x 2, so set x 2 = s. Note that the set of solutions is determined/controlled by the pair of independent parameters r and s. More about this later
Homogeneous Systems Theorem - A homogeneous system of m linear equations in n unknowns always has a nontrivial solution if m < n, i.e. if the number of unknowns exceeds the number of equations Proof - Let B be a matrix in reduced row echelon form that is row equivalent to A. The homogeneous systems AX 0 and BX 0 are equivalent. Let l be the number of nonzero rows of B. Then l m. Since m < n, then l < n. So, we are solving l equations in n unknowns and can solve for l of the unknowns in terms of the remaining n - l unknowns, which can take any values. So, BX 0 and thus AX 0 have nontrivial solutions QED Linear Algebra Echelon Form of a Matrix
Topics Preliminaries Echelon Form of a Matrix Elementary Matrices; Finding A -1 Equivalent Matrices LU-Factorization
Elementary Matrices; Finding A -1 Preliminaries Earlier, concept of nonsingular, or invertible, nxn matrix A was introduced. For an invertible matrix A, there is a matrix A -1 such that A A -1 = I and A - 1 A = I. However, at this point, there is no way to compute A -1 Have defined three elementary row operations Type I - exchange two rows Type II - multiply a row by a nonzero constant Type III - add a multiple of one row to another Each elementary row operation can be performed on a matrix A by multiplying A by an appropriate elementary matrix E
Elementary Matrices; Finding A -1 Defn - An nxn elementary matrix of Type I, Type II or Type III is a matrix obtained from the identity matrix I n by performing a single elementary row operation of Type I, Type II or Type III, respectively Theorem - Let A be an mxn matrix and let an elementary row operation of Type I, Type II or Type III be performed on A to yield matrix B. Let E be the elementary matrix obtained from I m by performing on it the same elementary row operation that was performed on A. Then B = EA
Elementary Matrices; Finding A -1 Theorem - If A and B are mxn matrices, then A is row equivalent to B if and only if B EkEk-1 E2E1A k where are mxm elementary matrices. E i i 1 Proof - Suppose A is row equivalent to B. By definition, B can be obtained by applying a finite number of elementary row operations to A. Each elementary row operation can be accomplished by multiplying by an appropriate elementary matrix. So the transformation from A to B can be accomplished by multiplying by a sequence of k elementary matrices. So B E E E E A E k k-1 2 1 i i 1
Elementary Matrices; Finding A -1 Proof (continued) Now suppose that we have B EkEk-1 E2E1A where each E i is an elementary matrix. Since B is obtained by applying a sequence of elementary row operations to A, then A is row equivalent to B. QED
Elementary Matrices; Finding A -1 Theorem - An elementary matrix is nonsingular and its inverse is an elementary matrix of the same type Proof - Will use the definition of nonsingularity, i.e. find a matrix that serves as an inverse. Type I - Let E switch rows i and j. Then the effects of E may be undone by applying E again, i.e. EE = I. So, E is invertible and E -1 = E Type II - Let E multiply the ith row by c 0. Let F multiply the ith row by 1/c. Then EF = FE = I. So, E is invertible and E -1 = F
Elementary Matrices; Finding A -1 Proof (continued) Type III - Let E add c times the ith row to the jth row. Let F add (-c) times the ith row to the jth row. Then EF = FE = I. So, E is invertible and E -1 = F QED
Elementary Matrices; Finding A -1 Theorem - Let A be a an nxn matrix and let the homogeneous system AX 0 have only the trivial solution X 0. Then A is row equivalent to I n. Proof - Let B be the matrix in reduced row echelon form that is row equivalent to A. The systems AX 0 and BX 0 are equivalent, so BX 0 has the trivial solution only. Now examine the structure of B. Since B is in reduced row echelon form, it has a similar appearance to I n. Let r n be the number of nonzero rows of B. If r < n, we have proved that BX 0 has a nontrivial solution. So, r = n, i.e. every row of B is nonzero. The first nonzero entry in each row appears to the right of the first one in the previous row, so B = I n QED
Elementary Matrices; Finding A -1 Theorem - A is nonsingular if and only if A is a product of elementary matrices Proof - Let A be nonsingular and consider the system AX 0. Then A -1 ( AX ) A -1 0 X 0. So, AX 0 has the trivial solution only. By the theorem on the the previous slide, A is row equivalent to I n. So, there exist elementary k matrices E i such that 1 I i n EkEk-1 E2E1A Then - 1-1 -1-1 A EkEk- 1 E2E1 E1 E2 Ek- 1Ek 1 Each E ī is an elementary matrix. Let A E Each E i is nonsingular 1E2 El- 1El and A is nonsingular since it is the product of nonsingular matrices QED
Elementary Matrices; Finding A -1 Theorem - A is nonsingular if and only if A is row equivalent to I n Proof - Let A be nonsingular. By the previous theorem, A can be expressed as the product of elementary row matrices. A EkEk-1 E2E1 Then A AInE so A is row kek-1 E2E1I n equivalent to I n Let A be row equivalent to I n. Then In E Each E i has an inverse. kek-1 E2E1A Operate on each side with inverses A E E E E I E E E E E ī -1-1 -1-1 -1-1 -1-1 1 2 k-1 k n 1 2 k-1 k each 1is an elementary matrix, so A is nonsingular QED
Theorem - The homogeneous system of n linear equations in n unknowns, AX 0, has a nontrivial solution if and only if A is singular Proof - Let AX 0 have a nontrivial solution. Suppose A is nonsingular. Then A -1 exists. So A -1 ( AX ) A -1 0 X 0, i.e. the trivial solution. This is a contradiction since we know that AX 0 has a nontrivial solution. So, A is singular. Let A be singular. If the system AX 0 has the trivial solution only, then A is row equivalent to I n. By the previous theorem, A is nonsingular, which is a contradiction. So, AX 0 has a nontrivial solution QED Linear Algebra Elementary Matrices; Finding A -1
Elementary Matrices; Finding A -1 Equivalent statements for nxn matrix A A is nonsingular AX 0 has the trivial solution only A is row equivalent to I n The system AX B has a unique solution for every nx1 matrix B A is a product of elementary matrices
Elementary Matrices; Finding A -1 Computing A -1 Have shown that if A is nonsingular, then it is row equivalent to I n. Also, have shown that it can be expressed as the product of elementary matrices. I E E E E A A E E E E -1-1 -1-1 n k k-1 2 1 1 2 k-1 k - 1 Then -1-1 -1-1 -1 A E E E So, A -1 k-1e k EkEk- 1 E E can be represented as the product of the elementary matrices that reduce A to I n Create the partitioned matrix n A I and apply the operations that reduce A to I n k 1 2 2 1 E E E E A I I A -1 k-1 2 1 n n
Elementary Matrices; Finding A -1 Computing A -1 - Example Compute inverse of A 1 1 3 1 2 3 0 1 1 1 1 3 1 0 0 1 1 3 1 0 0 1 2 3 0 1 0 0 1 0-1 1 0 0 1 1 0 0 1 0 1 1 0 0 1 1 1 3 1 0 0 1 1 0-2 3-3 0 1 0-1 1 0 0 1 0-1 1 0 0 0 1 1-1 1 0 0 1 1-1 1 1 0 0-1 2-3 -1 2-3 -1 0 1 0-1 1 0 A -1 1 0 0 0 1 1-1 1 1-1 1
Elementary Matrices; Finding A -1 Theorem - An nxn matrix A is singular if and only if A is row equivalent to a matrix B that has a row of zeros Proof - Let A be singular. Apply Gauss-Jordan reduction to A to get a matrix B in reduced row echelon form which is row equivalent to A. B cannot be I n since if it were, A would be nonsingular. Since B is in reduced row echelon form, B must have at least one row of zeros at the bottom. Let a A be row equivalent to a matrix B that has a row of all zeros.
Elementary Matrices; Finding A -1 Proof (continued) - The matrix B is singular since there does not exist a matrix C such that BC = CB = I n. (To see this, let the ith row of B consist of all zeros. The ith row of BC is generated by taking the ith row of B and multiplying each column of C by it. So, the ith row of BC is all zeros, but BC = I n. So, B must be singular.) A is row equivalent to B, so B E k E k-1 E 2 E 1 A If A is nonsingular then so is B since it is the product of nonsingular matrices. Since we know B is singular, then A must be singular. QED
Elementary Matrices; Finding A -1 The preceding theorem gives a way to determine if A is singular. Also, the determination of singularity/nonsingularity can be made in the process of computing A -1 Form the augmented matrix A I n, then put it into reduced row echelon form to get. C D If C = I n then A is nonsingular and D = A -1 If C I n then C has a row of zeros and thus is singular. So, A is singular and A -1 does not exist
Elementary Matrices; Finding A -1 Theorem - If A and B are nxn matrices such that AB = I n, then BA = I n. Thus A is nonsingular and B = A -1 Proof - First show that if AB = I n then A is nonsingular. Suppose that A is singular. Then A is row equivalent to a matrix C that has a row of zeros, i.e. C E, where is a kek-1 E2E1A k E i i 1 set of elementary matrices. CB EkEk-1 E2E1AB so CB is row equivalent to AB. Since C has a row of zeros, so does CB. AB is singular since it is row equivalent to a matrix with a row of zeros. However AB = I n, which is nonsingular. Contradiction. So, A is nonsingular and A -1 exists. A -1 ( AB ) = A -1 I n so B = A -1 QED
Topics Preliminaries Echelon Form of a Matrix Elementary Matrices; Finding A -1 Equivalent Matrices LU-Factorization
Equivalent Matrices Comments Have looked at elementary row operations, classified them as Type I, Type II and Type III, and argued that for a given row operation there is an elementary matrix E such that EA performs the operation on A Can also define elementary column operations and classify them as Type I, Type II and Type III. For a given elementary column operation, can find a matrix E such that AE performs the operation on A
Equivalent Matrices Defn - If A and B are two mxn matrices, then A is equivalent to B if B is obtained from A by a finite sequence of elementary row or elementary column operations Theorem - If A is any nonzero mxn matrix, then A is equivalent to a partitioned matrix of the form I 0 0 0 r r n-r m-r r m-r n-r Proof - By elementary row operations, transform A into a matrix B in reduced row echelon form.
Equivalent Matrices Proof (continued) - Interchange columns of B (Type I operations) to transform B into a matrix of the form Ir run-r m-r0r m-r0n-r where r is the number of nonzero rows in B. The only difference between this matrix and the desired form is the matrix r U n-r. The entries may be cleared by elementary column operations subtracting appropriate multiples of columns of Ir from run-r I giving r r0n-r m-r0r m-r0n-r m-r0r m-r0n-r which is equivalent to A QED
Equivalent Matrices Theorem - Two mxn matrices A and B are equivalent if and only if B = PAQ for some nonsingular matrices P and Q Proof - Let A and B be equivalent. Then B can be obtained from A by a sequence of elementary k row operations performed by E i and a i 1 sequence of column operations performed by Then B EkEk- 1 E2E1AF 1F2 Fl -1F l The matrices P EkEk-1 E2E1and Q F1F 2 Fl- 1Fl are nonsingular since they are products of nonsingular matrices. l F i i 1
Equivalent Matrices Proof (continued) Let B = PAQ where P and Q are nonsingular. Since P is nonsingular, it can be represented as a product of elementary row matrices, i.e. P EkEk-1 E2E1 Similarly, Q can be represented as the product of elementary column matrices Q F1F 2 Fl- 1Fl So, B E Thus B is kek-1 E2E1AF 1F2 Fl -1F l equivalent to A. QED
Equivalent Matrices Theorem - An nxn matrix A is nonsingular if and only if A is equivalent to I n Proof - Let A be nonsingular. Then A is row equivalent to I n and, thus, A is equivalent to I n Let A be equivalent to I n. Then by the previous theorem, there exist nonsingular matrices P and Q such that I n = PAQ. Since P and Q are nonsingular, P -1 and Q -1 exist. So A = P -1 I n Q -1. Since A is the product of nonsingular matrices, then A is nonsingular. QED
Topics Preliminaries Echelon Form of a Matrix Elementary Matrices; Finding A -1 Equivalent Matrices LU-Factorization
LU-Factorization Comments Current method of solving linear equations, Gauss-Jordan reduction, consists of forming the augmented matrix A B and applying Gauss- Jordan reduction to it to obtain a matrix n I C where C is the solution. Works fine for small systems. For large systems, there are more efficient methods that do not require getting the reduced row echelon form, or even the row echelon method
LU-Factorization Observations Upper triangular system AX = B a11 a12 a13 a1n b1 0 a22 a23 a2n b 2 0 0 a33 a3n b3 aii 0 for 1 i n 0 0 0 0 ann bn System may be solved by back substitution a nn x n b n x n b n a nn n - n a x a x b x b a x a n-1, n-1 n-1 n-1, n n-1 n-1 n-1 n-1, n n-1, n-1 n x b - a x a j n -1, n - 2,,1 j j jk k jj kj1
LU-Factorization Observations Lower triangular system AX = B a11 0 0 0 b 1 a21 a22 0 0 b2 a a a 0 b a 0 for 1 i n 31 32 33 3 an1 an2 an3 ann b a x b x b a 11 1 1 1 1 11 a x a x b x b -a x a 21 1 22 2 2 2 2 21 1 22 j-1 x b -a x a j 2,3,, n j j jk k jj k1 n ii System may be solved by forward substitution
LU-Factorization Comments If the coefficient matrix has either of the triangular forms, the system is easy to solve Will develop a method for solving a general system AX = B by a method based on these observations
LU-Factorization Defn - An nxn matrix A is said to be in triangular form when it satisfies the following properties a) A is either an upper or lower triangular matrix b)all of the diagonal entries of A are nonzero If the coefficient matrix A of the system AX = B is of triangular form, then the system is said to be of triangular form
LU-Factorization Defn - Let A be a nonsingular matrix and suppose that there is a lower triangular matrix L and an upper triangular matrix U such that A = LU. Then A is said to have an LU factorization or an LU decomposition Method - To solve AX = B where A = LU, express the system as LUX = B. Let UX = Z, then LZ = B. Solve this system by forward substitution to get Z, then solve UX = Z by back substitution to get X.
LU-Factorization Comments If we need to solve AX = B 1, AX = B 2,, AX = B k where { B 1, B 2,, B k } are not all known at the same time, it does not make sense to do the same Gaussian Elimination steps k times, needing about n 3 /3 operations each time Options are Compute A -1 and compute X = A -1 B i, 1 i k Compute LU = A and solve the two triangular systems each time.
LU-Factorization Comments Computing A -1 requires about n 3 operations and solving each system requires about n 2 operations Computing the factorization LU requires about n 3 /3 operations and solving each system requires about n 2 operations If A has a special structure, e.g. if A is tridiagonal, that structure may be partially reflected in L and U, but not necessarily in A -1
LU-Factorization Example 1-1 0 0 1 0 0 0 1-1 0 0-1 2-1 0-1 1 0 0 0 1-1 0 A 0-1 2-1 0-1 1 0 0 0 1-1 0 0-1 2 0 0-1 1 0 0 0 1 A -1 4 3 2 1 3 3 2 1 2 2 2 1 1 1 1 1 L Forward solving and back solving can be done very efficiently since the lower triangular zeros of L and the upper triangular zeros of U are in known locations. The cost of solving a tridiagonal system is proportional to n U
LU-Factorization Comments A band matrix has a ij = 0 except in the band i - j w The half bandwidth is w = 1 for diagonal matrix, w = 2 for a tridiagonal matrix, and w = n for a full matrix. Solving a band system requires about w 2 n operations A w w w w L U
LU-Factorization Example - Consider AX = B 6-2 -4 4 x1 2 3-3 -6 1 x -4-12 8 21-8 x 3 8-6 0-10 7 x -43 2 4 Can show A has an LU factorization L 1 0 0 0 6-2 -4 4 1 2 1 0 0 0-2 -4-1 U -2-2 1 0 0 0 5-2 -1 1-2 1 0 0 0 8
LU-Factorization Example (continued) A = LU, let Z = UX and solve LZ = B 1 0 0 0 z1 2 1 2 1 0 0 z 2-4 -2-2 1 0 z 3 8-1 1-2 1 z 4-43 z1 2, 1 z1 z2-4 z2-5 2-2z1-2z2 z3 8 z3 2 -z z - 2z z -43 z -32 1 2 3 4 4
LU-Factorization Example (continued) Solve UX = Z by back substitution 6-2 -4 4 x1 2 0-2 -4-1 x2-5 0 0 5-2 x 3 2 0 0 0 8 x -32 4 8x4-32 x4-4 5x3-2x4 2 x3-1.2-2x2-4x3 - x4-5 x2 6.9 6x - 2x - 4x 4x 2 x 4.5 1 2 3 4 1
LU-Factorization Questions How do we find L and U? Are L and U unique? Are there other computational issues?
LU-Factorization Computing L and U Initialize L with the identity (leave the entries below the diagonal blank) and initialize U with A 1 0 0 0 6-2 -4 4 1 0 0 3-3 -6 1 L U 1 0-12 8 21-8 1-6 0-10 7 Will clear the entries below the diagonal of U, a column at a time, by subtracting multiples of rows of U from rows below them. The multipliers used are recorded in L in the location corresponding to position in U being cleared
LU-Factorization Computing L and U (continued) Clear u 21 1 0 0 0 1 2 1 0 0 L 1 0 1 U 6-2 -4 4 0-2 -4-1 - 12 8 21-8 -6 0-10 7 Clear u 31 1 0 0 0 1 2 1 0 0 L - 2 1 0 1 U 6-2 -4 4 0-2 -4-1 0 4 13 0-6 0-10 7
LU-Factorization Computing L and U (continued) Clear u 41 1 0 0 0 1 2 1 0 0 L - 2 1 0-1 1 U 6-2 -4 4 0-2 -4-1 0 4 13 0 0-2 -14 11 Clear u 32 1 0 0 0 1 2 1 0 0 L - 2-2 1 0-1 1 U 6-2 -4 4 0-2 -4-1 0 0 5-2 0-2 -14 11
LU-Factorization Computing L and U (continued) Clear u 42 1 0 0 0 1 2 1 0 0 L - 2-2 1 0-1 1 1 U 6-2 -4 4 0-2 -4-1 0 0 5-2 0 0-10 12 Clear u 43 1 0 0 0 1 2 1 0 0 L - 2-2 1 0-1 1-2 1 U 6-2 -4 4 0-2 -4-1 0 0 5-2 0 0 0 8
LU-Factorization Computing L and U (continued) When doing this on a computer with a large matrix A, can return the results in the space allocated to A. L always has ones on its diagonal, so it is not necessary to store them. Also, it is not necessary to store the lower triangular part of U since it always consists of zeros. For our example, results could be returned as 6-2 -4 4 1 2-2 -4-1 -2-2 5-2 -1 1-2 8
LU-Factorization Uniqueness Can also express 6-2 -4 4 3-3 -6 1-12 8 21-8 -6 0-10 7 as the product L * U * of 2 0 0 0 3-1 -2 2 * 1-1 0 0 * 0 2 4 1 L U -4 2 1 0 0 0 5-2 -2-1 -2 2 0 0 0 4 So lower and upper triangular matrices are not unique
LU-Factorization Uniqueness (continued) Note that 6-2 -4 4 6 0 0 0 1-1 3-2 3 2 3 0-2 -4-1 0-2 0 0 0 1 2 1 2 U 0 0 5-2 0 0 5 0 0 0 1-2 5 0 0 0 8 0 0 0 8 0 0 0 1 So 1 0 0 0 6 0 0 0 1-1 3-2 3 2 3 1 2 1 0 0 0-2 0 0 0 1 2 1 2 A - 2-2 1 0 0 0 5 0 0 0 1-2 5-1 1-2 1 0 0 0 8 0 0 0 1
LU-Factorization L * Uniqueness (continued) Also note that 2 0 0 0 1 0 0 0 2 0 0 0 1-1 0 0 1 2 1 0 0 0-1 0 0-4 2 1 0-2 -2 1 0 0 0 1 0-2 -1-2 2-1 1-2 1 0 0 0 2 U * 3-1 -2 2 3 0 0 0 1-1 3-2 3 2 3 0 2 4 1 0 2 0 0 0 1 2 1 2 0 0 5-2 0 0 5 0 0 0 1-2 5 0 0 0 4 0 0 0 4 0 0 0 1
LU-Factorization Uniqueness (continued) Finally, 2 0 0 0 3 0 0 0 6 0 0 0 0-1 0 0 0 2 0 0 0-2 0 0 0 0 1 0 0 0 5 0 0 0 5 0 0 0 0 2 0 0 0 4 0 0 0 8 LU * * 1 0 0 0 6 0 0 0 1-1 3-2 3 2 3 1 2 1 0 0 0-2 0 0 0 1 2 1 2-2 - 2 1 0 0 0 5 0 0 0 1-2 5-1 1-2 1 0 0 0 8 0 0 0 1
LU-Factorization Uniqueness (continued) Theorem - If A = L 1 D 1 U 1 and also A = L 2 D 2 U 2 where the Ls are lower triangular with unit diagonal, the Us are upper triangular with unit diagonal, and the Ds are diagonal matrices with no zeros on the diagonal, then L 1 = L 2, D 1 = D 2, and U 1 = U 2. That is, the LDU factorization of A is unique.
LU-Factorization Other Computational Issues Compute the LU factorization of 1 0 0 1 1 1 L 1 0 U 1 1 3 1 2 5 8 1 0 0 1 1 1 L 1 1 0 U 0 0 2 1 2 5 8 L 1 0 0 1 1 1 1 1 0 U 0 0 2 2 1 0 3 6 A 1 1 1 1 1 3 2 5 8 Can t continue. Go back and fix A
LU-Factorization Other Computational Issues (continued) Problem is that rows of A are in the wrong order 1 0 0 1 1 1 1 1 1 PA 0 0 1 1 1 3 2 5 8 0 1 0 2 5 8 1 1 3 Permutation Matrix
LU-Factorization Other Computational Issues (continued) 1 0 0 1 1 1 L 1 0 U 2 5 8 1 1 1 3 L 1 0 0 1 1 1 L 2 1 0 U 0 3 6 1 1 1 3 1 0 0 1 1 1 2 1 0 U 0 3 6 1 0 1 0 0 2
LU-Factorization Other Computational Issues (continued) 1 0 0 1 1 1 1 0 0 1 1 1 PA 0 0 1 1 1 3 2 1 0 0 3 6 LU 0 1 0 2 5 8 1 0 1 0 0 2 Comments This has to be done often in practical problems The matrices L and U may be used directly to solve AX = B without considering permutations Need to consider P only when trying to reconstruct A. Then A = P 1 LU