Chapter 2. Solving Linear Equations. Po-Ning Chen, Professor. Department of Computer and Electrical Engineering. National Chiao Tung University

Transcription

1 Chapter 2 Solving Linear Equations Po-Ning Chen, Professor Department of Computer and Electrical Engineering National Chiao Tung University Hsin Chu, Taiwan 3000, R.O.C.

2 2. Vectors and linear equations 2- What is this course Linear Algebra about? Algebra ( 代數 ) The part of mathematics in which letters and other general symbols are used to represent numbers and quantities in formulas and equations. Linear Algebra ( 線性代數 ) To combine these algebraic symbols (e.g., vectors) in a linear fashion. So, we will not combine these algebraic symbols in a nonlinear fashion in this course! [ ] { x x x 2 = Example of nonlinear equations for x = : x 2 x /x 2 = [ ] { x x + x 2 =4 Example of linear equations for x = : x 2 x x 2 =

3 2. Vectors and linear equations 2-2 The linear equations can always be represented as matrix operation, i.e., Example of linear equations. Ax = b. Hence, the central problem of linear algorithm is to solve a system of linear equations. { ] x 2y = 3x +2y = [ ][ x y = [ ] A linear equation problem can also be viewed as a linear combination problem for column vectors (as referred by the textbook as the column picture view). In contrast, the original linear equations (the red-color one above) is referred by the textbook as the row picture view. Column picture view: x [ 3 ] + y [ 2 2 We want to find the scalar coefficients of column vectors [ ] another vector. ] = [ 3 [ ] ] and [ ] 2 to form 2

4 2. Matrix equation 2-3 From now on, we will focus on the matrix equation because it can be easily extended to higher dimension ( 2). In terminologies, Ax = b. Ax = b will be referred to as matrix equation. A will be referred to as coefficient matrix. x will be referred to as unknown vector. More terminologies: The elements of a matrix A as well as its size are indexed first by row, and then by column. Example. [ ] a, a A 2 3 =,2 a,3 a 2, a 2,2 a 2,3

5 2. Other representations of matrix 2-4 In MATLAB: MATLAB ( Matrix Laboratory ) is a numerical computing environment, innovated by Cleve Moler, and developed by MathWorks. It allows matrix manipulation, plotting of functions and data, implementation of algorithms, creation of user interfaces, and interfacing with programs in other languages. A matrix is represented by A=[ 2 3 ; 2 5-2] [ ] a, a A 2,3 =,2 a,3 = a 2, a 2,2 a 2,3 A vector x = [ ] is inputted as x=[ ; - ; ] [ ] % Again, row first, and then column. % Rows are separated by semicolon ;

6 2. Other representations of matrix 2-5 The matrix multiplication b = Ax is then given by b=a x Note: The textbook refers to this asdot product (equivalently, inner product) in MATLAB, which is not entirely correct. MATLAB treated this as the usual matrix multiplication. An element of a matrix A can be extracted by row and column indexes. [ ] a, a A(,2) % This is exactly a,2 in A 2,3 =,2 a,3. a 2, a 2,2 a 2,3 The row vector of a matrix A can be extracted by row index. A(,:) % This is exactly [ a, a,2 a,3 ] in A2,3. The submatrix of a matrix A can be extracted by row and column index ranges. [ ] a, a A(:2,:2) % This is exactly,2 in A a 2, a 2,3. 2,2

7 2. Other representations of matrix 2-6 The matrix multiplication b = Ax can then be rewritten by two forms: % Inner product with row vectors b=[a(,:)*x ; A(2,:)*x] % Linear combination of column vectors b=a(:,)*x()+a(:,2)*x(2)+a(:,3)*x(3) The above MATLAB rules for matrix manipulation can be easily extended: A(:,2)=[] % Eliminate the second column. A(:2,:)=[] % Eliminate the first two rows. A=[A; a 3, a 3,2 a 3,3 ] % Add a third row to matrix A. 3 v=[3:5] % v = 4 5

8 2. Other representations of matrix 2-7 Some useful MATLAB functions appear in textbook Problems. 0 0 A=eye(3) % Generate a 3-by-3 identity matrix A = A=ones(3) % Generate a 3-by-3 all-one matrix A = A=zeros(3) % Generate a 3-by-3 all-zero matrix A = v=ones(3,) % Generate a 3-by- all-one vector v = 0 A=zeros(3,) % Generate a 3-by- all-zero vector A = 0 0

9 2.2 The idea of elimination 2-8 Is there a systematic method to solve the linear equations? Answer: Forward elimination and back(ward) substitution. Example. { x 2y = 3x +2y = Step : Forward elimination. Eliminate in a forward sequential order, i.e., x, y,... { x 2y =...() 3x +2y =...(2) { Multiplier x 2y =...() (3x 3x)+(2y ( 6y)) = ( 3)...(2) 3 () { x 2y =...() 8y = 8...(2) 3 ()

10 2.2 The idea of elimination 2-9 Step 2: Back(ward) substitution. Substitute in a backward sequential order, i.e.,..., y, x. { x 2y =...(i) 8y = 8...(ii) { x 2y =...(i) y = { x 2 =... y = substitute y = { x = 3 y = (i) After forward elimination, the equations form an upper triangle. { x 2y = 8y = 8 In terminology, the element (e.g., x in the first equation) that is used to eliminate the same unknown in the remaining equations is called pivot. Thepivots are on the diagonal of the upper triangle after elimination.

11 2.2 Failure of forward elimination 2-0 Can the systematic method fail? Answer: If Ax = b has no solution or has infinitely many solutions, then some pivots will disappear during the process of forward elimination. Example. { x 2y =...() 2x 4y =...(2) { Multiplier x 2y =...() (2x 2x)+( 4y ( 4y)) = ( 2)...(2) 2 () { x 2y =...() 0 = 9...(2) 2 () No solution since 0 9! Hence, the next pivot disappears (or was canceled). If obtaining 0 = 0, then there are infinitely many solutions for the linear equations.

12 2.2 Failure of forward elimination 2- Can the systematic method fail if the solution exists and is unique? Answer: Yes, if the sequence for forward elimination is not properly ordered. Example. Sequence of choosing pivots in forward elimination {x from (), y from (2), z from (3)}. x 2y + z =...() 2x 4y + z =...(2) x + y z = 22...(3) x 2y + z =...() z = 9...(2) 2 () 3y 2z = 2...(3) () No y from (2) can be found! This has a unique solution (x, y, z) =(2,, 9). In such case, one can simply switch (2) with any of the remaining equations (e.g., (3)) of which its y still exists (i.e., has a coefficient other than zero).

13 2.3 Forward elimination/back substitution in matrix form2-2 The forward elimination/back substitution can be easily done in matrix form. Example. Thepivots are the diagonal elements x 2...() with 2 the first pivot y = 8...(2) z 0...(3) x 2...() with 2 the first pivot 0 y = 4...(2) 2 () 0 5 z 2...(3) ( ) () x 2...() 0 y = 4...(2 )with the second pivot 0 5 z 2...(3 ) x 2...() 0 y = 4...(2 )with the second pivot z 8...(3 ) (2 ) Based on the above, back substitution can be straightforwardly followed. The above procedure transforms Ax = b into Ux = c, whereu is an upper triangular matrix.

14 2.3 Forward elimination/back substitution in matrix form2-3 We summarize the algorithmic procedure of forward elimination for n n matrix A and n unknowns as follows. Use the first diagonal element as the first pivot to create zeros below it. If the diagonal element is zero, switch the row with any row below it, for which its respective element at the same column is non-zero. Use the new second diagonal element as the second pivot to create zeros below it. If the diagonal element is zero, switch the row with any row below it, for which its respective element at the same column is non-zero. Use the new third diagonal element as the third pivot to create zeros below it.... If the diagonal element is zero, switch the row with any row below it, for which its respective element at the same column is non-zero. Repeat the above procedure until either an upper-triangular U is resulted or the next pivot does not exist even with row switching.

15 2.3 Forward elimination/back substitution in matrix form2-4 This algorithmic procedure can be graphically illustrated as: Terminology: When a set of linear equations has either no solution or infinite many solutions, it is often referred to as a singular system.

16 2.3 Elimination using matrices 2-5 Reminder of terminologies (in English) Sigma notation: The dot or inner product can be expressed using the so-called sigma notation. n v w = v j w j. m by n matrix: The terminology to describe the size of a matrix with m rows and n columns. Entry of a matrix: a i,j in the ith row and jth column of matrix A is referred to as an entry of A. Component of a vector: v j in a vector v = [ v v 2 v n ] is referred to as a component of v. j=

17 ( ) The matrix form of forward elimination The forward elimination process can be implemented by multiplying the elimination matrix. Example. Thepivots are the diagonal elements x 2...() with 2 the first pivot y = 8...(2) z 0...(3) x 2...() with 2 the first pivot 0 y = 4...(2) 2 () 0 5 z 2...(3) ( ) () x 2...() 0 y = 4...(2 )with the second pivot 0 5 z 2...(3 ) x 2 0 y = z 8...()...(2 )with the second pivot...(3 ) (2 ) Based on the above, back substitution can be straightforwardly followed.

18 2.3 The matrix form of forward elimination 2-7 The forward elimination process can then be implemented by multiplying the elimination matrices E 2,, E 3, and E 3,2 in sequence x 0 ( ) }{{}}{{}}{{} E 3,2 E 3, E 2, which immediately gives = }{{} E 3, x = ( ) 0 }{{} E 3, }{{} E 2, For matrix multiplications, associative law is true (e.g., (E 3, E 2, )A = E 3, (E 2, A)) but commutative law is false (e.g., EA AE).

19 2.3 The matrix form of forward elimination 2-8 We now understand why the forward elimination process can be implemented by multiplying the elimination matrix. What if a row exchange is needed (because the next diagonal element is zero) during the process? Answer: Use the row exchange matrix. Specifically, insert the row exchange matrix in-between the elimination matrices. Denote by P i,j the row exchange matrix that exchanges row i and row j by left-multiplying it. P i,j is an identity matrix with row i and j reversed. For example, 0 0 P 2,3 = 0 0 The original row The original row 2.

20 ( ) The matrix form of forward elimination 2-9 Example. Thepivots are the diagonal elements x 2...() with 2 the first pivot y = 8...(2) (Replace the middle 9 by 8) z 0...(3) x 2...() with 2 the first pivot 0 0 y = 4...(2) 2 () 0 5 z 2...(3) ( ) () x 2...() y = 2...(2 )with the second pivot 0 0 z 4...(3 ) x 2...() y = 2...(2 )withthe second pivot z 4...(3 ) 0 (2 ) Based on the above, back substitution can be straightforwardly followed.

21 2.3 The matrix form of forward elimination 2-20 The forward elimination process can then be implemented by multiplying E 2,, E 3,, P 2,3 and E 3,2 in sequence x ( ) }{{}}{{}}{{}}{{} E 3,2 P 2,3 E 3, E 2, = }{{} E 3, }{{} P 2, ( ) 0 }{{} E 3, }{{} E 2, which immediately gives x =

22 2.3 The augmented matrix for forward elimination 2-2 Since we need to multiply both A and b by E = E 3,2 P 2,3 E 3, E 2, in order to solve Ax = b, e.g., x ( ) }{{}}{{}}{{}}{{} E 3,2 P 2,3 E 3, E 2, = ( ) }{{}}{{}}{{}}{{} E 3,2 P 2,3 E 3, E 2, we can introduce a so-called augmented matrix [ A b ] so that x can be solved by directly computing E [ A b ] = E =

23 2.3 Appendix 2-22 If we place E 2, on the right, what would happen then? Answer: Row operations will become column operations. For example, = (row ) = (row ) where (row 2) = (row 2) 2 (row ) (row 3) = (row 3) becomes where = (column 2) = (column 2) (column ) = (column ) 2 (column 2) (column 3) = (column 3) Exchange 2 row column

24 2.3 Appendix 2-23 If we place P 2,3 on the right, what would happen then? Answer: Row operations will become column operations. For example, = (row ) = (row ) where (row 2) = (row 3) (row 3) = (row 2) becomes where = (column ) = (column ) (column 3) = (column 2) (column 2) = (column 3) Exchange 2 3 row column

25 2.3 Appendix 2-24 The below definition gives what is required by Problem 8. [ ] a b (Problem 8, Section 2.3) The determinant of M = is det(m) = c d ad bc. Subtract l times row from row 2 (i.e., row 2 - l row ) to produce a new M. Show that det(m )=det(m) for every l. When l = c/a, the product of pivots equals the determinant: (a)(d lb) equals ad bc.

26 2.3 Appendix 2-25 Definition (Determinant 決定因子 ): Adeterminant, often denoted as det(a) orsimplydeta, is a determinant factor [ associated ] with a square v matrix. For example, for a matrix formed by,wherev and w are 2-dimensional (column) vectors, the determinant is the (signed) area of the parallelogram formed by v and w, andisgivenby [ ] v v deta = det 2 = w w 2 v v 2 w w 2 = v w 2 v 2 w w Area of parallelogram = v w sin(θ) (θ :fromvto w counterclockwisely) w = v w ( ) 2 v w cos 2 (θ) = v w v w θ v = v 2 w 2 (v w) 2 Area of parallelogram = v w sin(2π θ) ifa = [ ] w w 2 v v 2

27 2.3 Appendix 2-26 We can say more about determinant. The determinant of a matrix formed by v w u, wherev, w and u are 3-dimensional (column) vectors, the determinant is the (signed) volume of the parallelepiped formed by v, w and u.... (We will introduce the determinant in great detail in Chapter 5.)

28 2.3 Appendix 2-27 The below definition gives what is required by Problem 30. (Problem 30, Section 2.3) Write M = [ ] [ ] 0 A = and B =. 0 [ ] as a product of many factors (a) What matrix E subtractsrowfromrow2tomakerow2ofem smaller? (b) WhatmatrixF subtracts row 2 of EM from row to reduce row of FEM? (c) ContinueE s and F s until (many E s and F s) times (M) is(a or B). (d) E and F are the inverses of A and B! Moving all E s and F s to the right side will give you the desired result M = product of A s and B s. [ ] a b This is possible for all matrices M = > 0thathavead bc =. c d

29 2.3 Appendix 2-28 Definition (Positivity of a matrix): A positive matrix is a matrix in which all the elements are greater than zero. It is often denoted by A>0. Particularly note that a positive matrix is different from a positive definite matrix, which we will introduce in Chapter 6.

30 2.4 Rules for matrix operations 2-29 Let us revisit matrix operations and summarize the rules that they obey. Matrix addition is simply a term-by-term addition. The definition of matrix multiplication is repeated below. Definition (Product of matrices): The product of two matrices is defined as the inner products of row vectors and column vectors respectively from the first matrix and the second matrix, counted from the left. Specifically as an example, [ ] [ a [b A 2 3 B 3 2 = ] a b 2 b a b ] [ ] 2 a b a 2b a = a b 2, 2b 2 a 2 b a 2 b 2 a 2 where for the two matrices on the extreme left of the above equation, 2 is the number of vectors, and 3 is the dimension of the vectors. Note that the dimension of the vectors shall be the same, otherwise the inner product operation cannot be performed

31 2.4 Rules for matrix operations 2-30 Rules or laws that matrix operations obey. Commutative law for addition: A + B = B + A. Commutative law for multiplication: AB = BA. Distributive law from the left: A(B + C) =AB + AC. Distributive law from the right: (A + B)C = AC + BC. Distributive law with respect to scalar: α(a + B) =αa + αb. Associative law for addition: (A + B)+C = A +(B + C). Associative law for multiplication: (AB)C = A(BC). A p A q = A p+q, if p and q are positive integers A p A q = A p+q, if p or q is a negative integer, and A exists A 0 = I, if A is a square matrix, Power law: where I is an identity matrix Note: We can talk about the product AA only when A is a square matrix.

32 2.4 Rules for matrix operations 2-3 The way to prove these rules may require the below two pictures. Column picture of matrix product: From the definition of matrix product, we can see that [ ] [[ ] [ ] ] [ ] a [b A 2 3 B 3 2 = ] a a b 2 = a 2 a b a 2 a b 2 b a b 2 2 a 2 b a 2 b 2 or we can simply write A 2 3 B 3 2 = A [ b b 2 ] = [ Ab Ab 2 ]. In summary, the jth column of product AB is given by Ab j,whereb j is the jth column of B. 2 2

33 2.4 Rules for matrix operations 2-32 Row picture of matrix product: From the definition of matrix product, we can see that [ ] [ [ ]] [ a [b A 2 3 B 3 2 = ] a b b 2 a b a a b 2 = [ ] b ] 2 2 b b 2 a 2b a 2b 2 or we can simply write A 2 3 B 3 2 = a 2 [ ] a B = a 2 [ ] a B a. 2B 2 2 In summary, the ith row of product AB is given by a i B,wherea i ith row of A. is the

34 2.4 Block manipulation of matrices 2-33 The above column or row picture indicates that we can manipulate the matrix in blocks. For example, matrix B can be divided into two blocks when calculating the matrix product. AB = A [ b b 2 ] = [ Ab Ab 2 ]. Recall that we can solve the linear equations using the augmented matrix [ A b ], which consists of two blocks of different sizes. To generalize the idea, we can represent matrix A in the form [ ] I I I A = = I I I 0 0 0

35 2.4 Block manipulation of matrices 2-34 We can treat blocks as number entries in matrix operation, which is usually simpler. Example. [ ] [ ] A, A,2 B, B +,2 A 2, A 2,2 B 2, B 2,2 = [ ] A, + B, A,2 + B,2 A 2, + B 2, A 2,2 + B 2,2 and ][ ] [A, A,2 B, B,2 A 2, A 2,2 B 2, B 2,2 = [ ] A, B, + A,2 B 2, A, B,2 + A,2 B 2,2 A 2, B, + A 2,2 B 2, A 2, B,2 + A 2,2 B 2,2

36 2.4 Block manipulation of matrices 2-35 By block manipulation, the product AB does not have to be calculated/represented in terms of rows of A and columns of B, butcanbedoneinaviceversa way. [ ] [ a [b AB = ] a a b 2 = b a b ] 2 2 a 2b a Representation based on Inner product 2b = [ ] [ d ] c c 2 d = c d + c 2 d 2 Representation based on Outer product 2 The representation using inner product is more convenient in the sense that we can have an entry-wise identification of elements in A m n B n p. Hence, the (i, j)th entry of A m n B n p is the inner product of row i of A and column j of B, i.e., a ib j = b j a i. Hence, we need to perform m p inner products to obtain A m n B n p.

37 2.4 Block manipulation of matrices 2-36 The representation using outer product can represent AB as sum of all outer products of column i of A and row j of B, i.e., d A m n B n p = [ ] d 2 c c 2 c n. = d n n c i d i. Hence, we need to perform n outer products to obtain A m n B n p. i=

38 2.4 Block manipulation of matrices 2-37 By block manipulation, theforward elimination canbedoneina block-by-block fashion. Example. and [ ] [ ] 0 a b E 2, F = ca = c d E 2, F = = = [ I 0 CA I ] [ A B C D [ ] a b 0 d ca b [ ] A B C CA A D CA B [ ] A B 0 D CA B ] The matrix D CA B is important in linear algebra; hence, it gets itself a terminology Scher complement, named after Issai Scher.

39 2.4 Appendix 2-38 What is the (i, j)th entry of ABC? Solution: The (i, j)th entry of AD is the inner product of row i of A and column j of D, i.e. a i d j. The column j of D = BC is the matrix multiplication of matrix B and column j of C, i.e., d j = Bc j. Therefore, the answer is a i Bc j = [ a i, a i,2 ] B c 2,j.. c,j Exercise: What is the (i, j)th entry of ABCD?

40 2.4 Appendix 2-39 Exercise: Problems 27, 28 and 37 in textbook. (Problem 27, Section 2.4) Show that the product of upper triangular matrices is always upper triangular: x x x x x x AB = 0 x x 0 x x = x 0 0 x 0 0 Proof using dot products (Row times column)(row2ofa) (column of B)=0. Which other dot products gives zeros? Proof using full matrices (Column times row) Drawx s and 0 s in (column 2 of A) times (row 2 of B). Also show (column 3 of A) times (row 3 of B). (Problem 28, Section 2.4) Draw the cuts in A (2 by 3) and B (3 by 4) and AB to show how each of the four multiplications rules is really a block multiplication: () Matrix A times columns of B. Columns of AB (2) Rows of A times the matrix B. Rows of AB (3) Rows of A times columns of B. Inner products (numbers in AB) (3) Columns of A times rows of B. Outer products (matrices add to AB)

41 2.4 Appendix 2-40 (Problem 37, Section 2.4) To prove (AB)C = A(BC), use the column vectors b,...,b n of B. First suppose that C has only one column c with entries c,...,c n : AB has columns Ab,...,Ab n and then (AB)c = c Ab + + c n Ab n. Bc has one column c b + +c n b n and then A(Bc)equalsA(c b + +c n b n ). Linearity gives equality of those two sums. This proves (AB)c = A(Bc). The same is true for all other of C. Therefore (AB)C = A(BC). Apply to inverses: If BA = I and AC = I, prove that the left-inverse B equals the right-inverse C.

42 2.5 Inverse matrices 2-4 Definition (Inverse matrix): The matrix A is invertible if there exists a matrix A such that A A = I and AA = I. A is invertible only when A is a square matrix. A is invertible if, and only if, Ax = b has a unique solution. This unique solution must be A b. If A is invertible, thenax = 0 cannot have a non-zero solution. A n n is invertible if, and only if, forward elimination (with possibly row exchange) should give n non-zero pivots. (This test is what you can use now!) If A is invertible if, and only if, the determinant of A (i.e., det(a)) is non-zero. (We will come back to this in Chapter 5.) Hence, for a square matrix, either both left and right inverses do not exist (det(a) = 0) or both left and right inverses exist (det(a) 0).

43 2.5 Inverse matrices 2-42 The inverse matrix of A is unique. Proof: If L and R are both the inverse of A, then L = LI = L(AR) =(LA)R = IR = R. In fact, the above proof also verifies that for a square matrix A, itsleft inverse L always equals its right inverse R. Again, it is not possible for a square matrix without a left (respectively, right) inverse to have a right (respectively, left) inverse. Again, it is not possible for a square matrix to have distinct left and right inverses.

44 2.5 Inverse matrices 2-43 Lemma. Suppose A and B are both square matrices. Then, AB has an inverse if, and only if, both A and B have inverses. Proof:. if-part: If both A and B have inverses, then (AB)(B A )=A(BB )A = AIA = AA = I. Hence, AB has an inverse B A. 2. only-if-part: Suppose AB has an inverse C. Then, it is not possible that A or B has no inverse because if A had no inverse (meaning that both left and right inverses of A do not exist), then ABC = I implies that the square matrix A has a right inverse BC, a contradiction. On the other hand, if B has no inverse (meaning that both left and right inverses of B do not exist), then CAB = I implies that the square matrix B has a left inverse CA, a contradiction.

45 2.5 How to find the inverse matrix of A 2-44 How to find A if it exists? Answer: Gauss-Jordan method. What is Gauss-Jordan method? Answer: To repeat forward elimination and back substitution n times for an n-by-n matrix A. Example. Suppose A is a 3-by-3 matrix. Find the solutions x, x 2 and x 3 for Ax = e 0, Ax 2 = e 2 0 Then, 0 and Ax 3 = e A [ x x 2 x 3 ] = [ Ax Ax 2 Ax 3 ] = [ e e 2 e 3 ] = I, which means A = [ x x 2 x 3 ].

46 2.5 How to find the inverse matrix of A 2-45 Gauss said We can solve the three linear equations in one time by using the augmented matrix. Ax = e [ Ax 2 = e 2 = A e e 2 Ax 3 = e 3 ] [ ] e 3 = A I. 2 0 Example. Suppose A = [ ] A I = ()...(2)...(3)...()...( )...(2) ( 2) ()...(2 )...(3) 0 ()...(3 )...( )...(2 )...(3 ) ( 2 3) (2)

47 2.5 How to find the inverse matrix of A 2-46 What is the next step after obtaining Answer: Back substitution to solve x, x,2 x = x 2,, x 2 = x 2,2 and x 3 = x 3, x 3,2 x,3 x 2,3. x 3,3 Example (Continue). x = x, x 2, = x 3, ( )x 2, (0)x 3, 2 /2 ( )x 3, 3/2 /3 4/3 =

48 2.5 How to find the inverse matrix of A 2-47 and x 2 = x,2 x 2,2 = x 3,2 0 ( )x 2,2 (0)x 3,2 2 ( )x 3,2 3/2 2/3 4/3 = 2 2 and x 3 = x,3 x 2,3 = x 3,3 0 ( )x 2,3 (0)x 3,3 2 0 ( )x 3,3 3/2 4/3 =

49 2.5 How to find the inverse matrix of A 2-48 Jordan said We can also solve [ x x 2 x 3 ] by backward elimination. Example. (Continue) (3) ()...(2)...(3)...()...(2) ( 3/4) (3) = (2 )...(3)...() (0) (3) = ( )...(2 )...( ) ( 2/3) (2 )=( )...(2 )...(3) Based on this new augmented matrix, we can solve x, x 2 and x 3 simply by dividing the pivot in each row (i.e., to make the pivot equal to ). The resultant matrix is said to be in its reduced echelon form.

50 2.5 How to find the inverse matrix of A ( )/2...(2 )/(3/2)...(3)/(4/3) Hence, A = Gauss-Jordan method: Multiply [ A I ] by A to get [ I A ] using forward elimination and backward elimination.

51 2.5 Terminologies 2-50 Row echelon form: A form of matrices satisfying Every all-zero row is below non-all-zero rows. The leading coefficient (the first nonzero number from the left, also called the pivot) of a non-all-zero row is always strictly to the right of the leading coefficient of the row above it. Example. Upper triangular matrix obtained from forward elimination. (row) Reduced echelon form: A form of matrices in row echelon form with the leading coefficient being one and also being the only non-zero entry in the column. 0 Example

52 2.5 MATLAB revisited 2-5 In MATLAB: The Gauss-Jordan method can be performed as: I = eye(n); % Define the n-by-n identity matrix. R = rref([a I]); % Obtain the row reduced echelon form. X = R(:, n+:n+n); % Pick up the last n columns to form A. You can also obtain the inverse matrix directly: X=inv(A) % A has to be a square matrix. Question: If inv() has already fulfilled the purpose of finding inverse, why we need rref()?

53 2.5 Gauss-Jordan method revisited 2-52 Example. Find the right inverse R 3 2 of a non-square matrix A 2 3 = [ a a 2 ] a 3. (Note that this cannot be done by inv().) r, [ ] A r 2, = 0 r 3, [ = Form the augmented matrix A ] 0 r,2 [ ] 0 A r 2,2 0 r 3,2 [ [ ] [ ]] [ [ ] 0 = Find E 2 2 such that EA E E = I Ea 3 E 0 [ ] [ ] r, = E Ea 3 r 3, r 2, 0 = [ ] [ ] The right inverse R is not unique! r,2 0 = E Ea 3 r 3,2 r 2,2 E [ ]] 0

54 2.5 Gauss-Jordan method revisited 2-53 [ ] 2 0 Further suppose A =. Then, 2 [ ] [ ] [ ] = [ ] [ ] [ ] = [ 2 0 ] [ ] [ ] = Hence, [ E = 2 0 ][ ][ ] = [ Forward elimination...backward elimination...pivot normalization ].

55 2.5 Gauss-Jordan method revisited 2-54 The problem is reduced to r, [ ] EA r 2, = E 0 r 3, r,2 [ ] EA r 2,2 = E 0 r 3,2 = [ ] [ ] r, r 2, r 3, r,2 r 2,2 r 3,2 = = [ [ ] ] We can let r 3, = r 3,2 = 0 and obtain [ ] [ ] [ ] r, 2 3 = E = r 2, 0 [ ] [ ] [ 3] r,2 0 So, the right inverse is R = 3 = E = 2 r 2,2 3 Question: Can we choose r 3, = r 3,2 =? Hint: R = [ E 0 0 ]. [ E Ea3 [ r3, r 3,2 ] r 3, r 3,2 ].

56 2.5 Gauss-Jordan method revisited 2-55 In MATLAB: The Gauss-Jordan method can be performed as: I = eye(2); % Define the 2-by-2 identity matrix. A = [2-0; - 2 -]; % Define A. S = rref([a I]); % Obtain [ the row reduced ] echelon form. 0 2 % S = R = [S(:, 4:5); 0 0 ]; % Pick up the last 2 columns to form R. Important summary: In the process of Gauss-Jordan method, we solve and obtain directly the right inverse in the above example. Using the same technique, we may obtain the left inverse, whichisthe product oftheforwardeliminationmatrices,thebackwardeliminationmatrices and the pivot normalization matrix. In the above example, A 2 3 does not have the left inverse because no matrix L 3 2 can make L 3 2 A 2 3 = I 3 3.

57 2.5 Diagonal and tridiagonal matrices 2-56 () The inverse matrix of a diagonal square matrix (if the inverse exists) is a diagonal square matrix. (2) The inverse matrix of a tridiagonal square matrix may however be a dense square matrix (since a diagonal matrix is also a tridiagonal matrix). Definition (Diagonal and tridiagonal matrices): A square matrix is a diagonal matrix if all the off-diagonal entries are zero. Likewise, a tridiagonal matrix is one that all the entries are zero except those in the main diagonal and two adjacent diagonals (which can be zeros). Some may define the tridiagonal matrix as one that all entries in the main diagonal and two adjacent diagonals are non-zeros. In such case, its inverse is always dense. It is also worth knowing that antidiagonal entries of a matrix A are a,n,a 2,n,a 3,n 2,...,a n,. Definition (Dense matrix): A matrix is dense if all the entries are nonzeros.

58 2.5 Triangular matrix and singular matrix 2-57 If L is lower triangular with s on the diagonal, so is its inverse. A square triangular matrix is invertible if, and only if, all its diagonals are nonzeros. Definition (Singular matrix): A square matrix is called singular if it has no inverse.

59 2.5 Symmetric and determinant 2-58 The inverse matrix of a symmetric matrix (if the inverse exists) is symmetric. Definition (Symmetric matrix): A square matrix is symmetric if its (i, j)th entry is equal to its (j, i)th entry for every i and j. Recall that the determinant is the signed area (volume, etc) of the parallelogram (parallelepiped, etc) formed by the column vectors of a matrix A. The Gauss-Jordan method can also be used to determine the determinant, which is equal to the product of all the pivots before normalization. 2 0 Example. Suppose A = [ A I ] G.J.Method = det(a) = =4.

60 2.6 Elimination = Factorization : A = LU 2-59 Definition (Factorization): Factorization is the decomposition of an object (e.g., a number ora polynomialoramatrix) into a product ofotherobjects, named factors, such that when being multiplied together, one gets the original. Example. 5 = 3 5 x 2 3x +2=(x )(x 2) A = LU Triangular factorization: How to find lower triangular L and upper triangular U such that A = LU? Answer: Gauss-Jordan method (Actually, just forward elimination. Why? See the below example). Forward elimination produces a matrix E such that EA = U. The inverse of lower triangular E is also lower triangular L = E. This immediately gives A = E U = LU.

61 ( ) Elimination = Factorization : A = LU 2-60 Example (3) (3 ) () with 2 the first pivot...(2)...() with 2 the first pivot...(2) 2 ()...(3) ( ) ()...()...(2 )with the second pivot...()...(2 )with the second pivot...(3 ) (2 )

62 2.6 Elimination = Factorization : A = LU 2-6 The forward elimination process can then be implemented by multiplying the elimination matrices E 2,, E 3, and E 3,2 in sequence ( ) }{{}}{{}}{{} E 3,2 E 3, E 2, = Then we have that after forward elimination E 3,2 E 3, E 2, [ A I ] = [ EA E ] = [ U E ] where E = E 3,2 E 3, E 2,. Next, we need to determine L = E = E 2, E 3, E 3,2.Thisiseasybecauseevery E i,j is formed by changing just one off-diagonal entry from an identity matrix.

63 2.6 Elimination = Factorization : A = LU 2-62 What we need to do is to negate the only non-zero off-diagonal entry! = 0 0, 0 0 = }{{ } 0 ( ) 0 }{{ } ( ) 0 E3,2 E3, and } 0 0 {{ } E2, = More surprisingly, when they are multiplied in reverse order, the product is simply to place all these non-zero entries at their respective position! 0 0 L = E = E2, E 3, E 3,2 = 2 0

64 2.6 Elimination = Factorization : A = LU 2-63 Intuitions behind the above example Why the inverse of E i,j is simply to negate the multiplier of E i,j? Answer: The inverse action of (i) l i,j (j) is simply (i)+l i,j (j). Why the product of E2, E 3, E 3,2 respective position? Answer: E 3, Hence, = only updates row 3 of E 3,2 is to place all the non-zero entries in their by row, i.e., (3 )=(3)+l 3, (). (3 )= { (3), if () = 0 (such as the (, 2)th and (, 3)th entries of E 3,2 ) l 3,, if (3) = 0 and () = Hint of a formal proof: [I + (E3, I)][I + (E 3,2 I)] =I + (E 3, I) + (E 3,2 I) + (E 3, I)(E 3,2 I) = I + (E 3, I) + (E 3,2 I). Similar interpretation can be done for E 2, onto E 3, E 3,2. We summarize the observation in the below lemma.

65 2.6 Elimination = Factorization : A = LU 2-64 Lemma: For an invertible matrix A, if no row exchange is necessary during forward elimination, thenitslower triangular factor L and upper triangular factor U satisfy A = LU; L has on all main diagonal entries, and multipliers below the diagonal entries. When a row of A starts with r zeros, so does the rows of L (because the multipliers are zeros). U has (non-normalized) pivots on its main diagonal. When a column of A starts with c zeros, so does the columns of U (because forward eliminations will not change these zeros). Example. Suppose A = a, a,2 a,3 a 2, a 2,2 a 2,3. a 3, a 3,2 a 3,3

66 2.6 Elimination = Factorization : A = LU 2-65 Then a, a,2 a,3 a 2, a 2,2 a 2,3 a 3, a 3,2 a 3,3 a, a,2 a,3 0 a 2,2 (a 2, /a, )a,2 a 2,3 (a 2, /a, )a,3 0 a 3,2 (a 3, /a, )a,2 a 3,3 (a 3, /a, )a,3...()...(2) (a 2, /a, ) ()...(3) (a 3, /a, ) () Then, we know 0 0 a, a,2 a,3 L = a 2, /a, 0 and U = 0 a 2,2 (a 2, /a, )a,2 a 2,3 (a 2, /a, )a,3 a 3, /a,? 0 0? Remember the formula of multiplier: l 2, = a 2, /a, and l 3, = a 3, /a,. By LU decomposition, Ax = b can be solved by { Lc = b (forward substitution) Ux = c (back substitution)

67 ( ) Elimination = Factorization : A = LU 2-66 What if row exchange is needed during forward elimination? Example (3) () with 2 the first pivot...(2) (Replace the middle 9 by 8)...() with 2 the first pivot...(2) 2 ()...(3) ( ) ()...()...(2 )with the second pivot...(3 )...()...(2 )with the second pivot...(3 ) 0 (2 )

68 2.6 Elimination = Factorization : A = LU 2-67 The forward elimination process can then be implemented by multiplying E 2,, E 3,, P 2,3 and E 3,2 in sequence ( ) }{{}}{{}}{{}}{{} E 3,2 P 2,3 E 3, E 2, = Then we have after forward elimination E 3,2 P 2,3 E 3, E 2, [ A I ] = [ EA E ] = [ U E ] where E = E 3,2 P 2,3 E 3, E 2,. Next, we need to determine L = E = E 2, E 3, P 2,3 E 3,2. This is easy because every E ij is formed by changing just one off-diagonal entry from an identity matrix, and the inverse of P 2,3 is itself!

69 2.6 Elimination = Factorization : A = LU 2-68 What we need to do is to negate the only non-zero off-diagonal entry! = 0 0, 0 0 = }{{ } 0 ( ) 0 }{{ } ( ) 0 E3,2 E3, L = E = E2, E 3, P 2,3 E 3,2 = and } 0 0 {{ } E2, = Not a lower triangular matrix!

70 2.6 Elimination = Factorization : A = LU 2-69 How can we guarantee there is no row exchanges necessary for finding A = LU? No row exchanges during forward elimination if, and only if, all the upper-left square sub-matrices of A are invertible. (See Problem 24 on p. 06.) (Problem 24, Section 2.6) Which invertible matrices allow A = LU (elimination without row exchanges)? Good question! Look at each of the square upper left submatrices of A. All upper left k by k submatrices A k must be invertible [ (sizes ][ k =,...,n). ] Lk 0 Uk Explain that answer: A k factors into because LU =. 0 In such case A k = L k U k,wherea k (similarly, L k and U k ) consists of the first k rows and the first k columns of A. [ a, a,2 a,3 ] [ 0 0 ][ u, u,2 u,3 ] a 2, a 2,2 a 2,3 = l 2, 0 0 u 2,2 u 2,3 a 3, a 3,2 a 3,3 l 3, l 3,2 0 0 u 3,3

71 2.6 Elimination = Triple Factorization : A = LDU 2-70 Now assume again no row exchange is needed during forward elimination. Sometimes, we wish to make the diagonal entries of U being all ones. Hence, a diagonal matrix D is introduced. Example }{{}}{{} D U The advantage of triple factorization is that for symmetric matrix A = LDU, we have L = U, where U is the transpose of matrix U (i.e., the (i, j)th entry of U is the (j, i)th entry of U). Hence, A = LDL.

72 2.6 The cost of elimination 2-7 Question: Again, assume with no row exchanges during elimination. What will be the complexity of using forward elimination and back substitution to solve Ax = b? Answer: i) Forward elimination for matrix A, operations exact approximation a i, 0fori =2,...,n n(n ) n 2 a i,2 0fori =3,...,n. (n )(n 2). (n ) 2. a i,n 0fori = n total n i(i ) = 3 n(n2 ) Note: For example, n(n ) = n operations/row (n ) rows. i=2 3 n3 Note: The blue-colored numbers are used for step ii) in the next page, which requires for example, operation/row (n ) rows.

73 2.6 The cost of elimination 2-72 ii) Forward elimination for vector b, we need n i=2 (i ) = 2n(n ) computations. iii) Back substitution Ux = c requires or / operations exact x n = c n /u n,n x n =(c n u n,n x n )/u n,n. 2. x = n total Hence, ii) andiii) requiren 2 multiplications. n i= i = n(n +) 2 Using MATLAB, the above algorithm requires about second to solve Ax = b with n = 000. Doubling n makes the algorithm 2 3 = 8 times slower. Example. n = seconds =.574 days.

74 2.6 The cost of elimination 2-73 Question: How about the complexity for calculating A (without row exchange) using Gauss-Jordan? Does it require n ( 3 n(n2 ) + n 2) multiplications as we need to solve Ax i = e i for i =,...,n? Answer: i) Forward elimination for augmented matrix [ A I ], operations exact a i, 0fori =2,...,n 2n(n ) a i,2 0fori =3,...,n. (2n )(n 2). a i,n 0fori = n (n +2) total n (n + i)(i ) = n(n )(5n +2) 6 i=2

75 2.6 The cost of elimination 2-74 ii) Backward elimination for augmented matrix [ A I ], operations exact a i,n 0fori = n,..., (n +)(n ) a i,n 0fori = n 2,...,. (n +2)(n 2). a i,2 0fori = (2n ) total n (2n i)i = n(n )(4n +) 6 iii) Pivot normalization: n 2 divisions. Three steps in total, 2 n(n )(3n +)+n2 = 2 n(3n2 ) 3 2 n3. i= We can also find the inverse of A by solving Ax i = e i for i =,...,n separately. In such case, the forward elimination part in slide 2-7 (step i)) can be shared among all n equations, while steps ii) andiii) in slide 2-72 (steps ii) andiii)) must be repeated n times. This yields 3 n(n2 ) + n n n3 multiplications. Backward substitution is more efficient than backward elimination.

76 2.6 The cost of elimination 2-75 Again, assume with no row exchanges during elimination. What will be the complexity of using forward elimination and back substitution to solve Ax = b, if A is a band matrix with w non-zero diagonals both below and above the main diagonal? Answer: i) Forward elimination for matrix A, operations exact approximation a i, 0fori =2,...,w+ w w w 2 a i,2 0fori =3,...,w+2. w w. w 2. a i,n w 0fori = n w +,...,n. w w. w 2. a i,n 0fori = n w 2 total w i= i 2 + w 2 (n w) = 6 w(6nw 4w2 3w +) Note: The blue-colored numbers are used for step ii). nw 2

77 2.6 The cost of elimination 2-76 ii) Forward elimination for vector b, we need w i= i +(n w)w = w(2n w ) 2 computations. iii) Back substitution Ux = c requires operations exact x n = c n /u n,n x n =(c n u n,n x n )/u n,n. 2. x n w =(c n w u n w,n x n )/u n w,n w. w +. x = w + total (w + )(2n w) 2 Hence, ii) andiii) requiren( + 2w) w( + w) computations. A band-(2w +) matrix A (if no row exchange is needed during forward elimination) satisfies that its factors L and U are both band matrices with bandwidth of w +.

78 2.7 Transpose and permutation 2-77 Definition (Matrix transpose): The transpose A of a matrix A is simply to exchange the rows and columns. Hence, the (i, j)th entry of A is the (j, i)th entry of A. Note: A matrix transpose is sometimes denoted by A or A T. For example, A is used by MATLAB. A T is adopted by the textbook. (We will use this notation in the sequel!) Properties of transpose Sum Property: (A + B) T = A T + B T Proof: This can be proved by listing all the matrix entries of both sides.

79 2.7 Transpose and permutation 2-78 Product Property: (AB) T = B T A T Proof: First, prove using Sum Property of Transpose. For a matrix A and vector b, (Ab) T = ([ a a 2 ] b ) T Then, =(b a + b 2 a 2 + ) T = b a T + b 2 a T 2 + = b T a T 2 = b T A T. a T (AB) T =(A [ b b 2 ]) T = [ Ab Ab 2 ]T = b T A T b T 2A T =. b T b T 2. A T = B T A T.

80 2.7 Transpose and permutation 2-79 Inverse Property: (A T ) =(A ) T Proof: A A = I (A A) T = I T = I; hence, the Product Property of Transpose gives A T (A ) T = I, which implies (A ) T is the inverse of A T.

81 2.7 Inner product revisited 2-80 Recall the difference between inner product and outer product (for real-valued vectors) and Inner product v w = w T v = [ ] [ ] v w w 2 v 2 Outer product v w = vw T = [ v v 2 = w v + w 2 v 2 ] [ ] [w ] v w w 2 = v w 2 v 2 w v2w 2 To memorize it, for inner product, T is placed inner (in-between v and w) for outer product, T is placed outer (outside v and w).

82 2.7 Inner product revisited 2-8 The dot product we have introduced in Chapter should be only termed as a product between two vectors, which is denoted by a dot. It happens to satisfy the three axioms of an inner product. Definition: A mapping from V Vto F, denoted by,, is an inner product if for every x, y, z Vand a F,. Positive-definiteness: x, x 0 with equality only when x =0 2. Symmetry: x, y = y, x { ax, y = a x, y 3. Linearity: x + y, z = x, z + y, z Vector dot product is an inner product with V = R n and F = R (set of all real numbers). It is then possible to parallel linear algebra with calculus along this conception.

83 2.7 Inner product revisited 2-82 Example. What is the calculus for function transpose operation, such as in (Ax) T y = x T (A T y), where A is the difference matrix? Inner product of two vectors can be expressed in terms of transpose as x, y = x y = y T x. Hence, (Ax) T y = x T (A T y) can be expressed as y,ax = A T y, x. Inner product of two real functions can be defined as x(t),y(t) x(t)y(t)dt. We then reasonably guess the calculus counterpart of difference matrix A is derivative. I.e., we guess ( ) ( ) T y(t), x(t) = y(t),x(t). t t The big question is what ( t) T is?

84 2.7 Inner product revisited 2-83 Given that x(t) dt < and y(t) dt <, integration by parts gives us y(t),x (t) = y(t) x(t) t dt = y(t)x(t) y(t) t x(t)dt (Integration by parts) ( = y(t) ) x(t)dt = y (t),x(t) t Hence, ( ) T Transpose of differentiation is = t t. Verification: 0 0 Ax = 0 x = 0 x x 2 x and A T y = x 3 x y = 0 0 (y 2 y ) (y 3 y 2 ) y 3

85 2.7 Inner product revisited 2-84 Summary Inner product is a more general concept, and has much more extensional use than what we have introduced here.

86 2.7 Symmetric matrix 2-85 We can re-define symmetric matrix (different from that given in slide 2-58) using transpose. Definition (Symmetric matrix): A matrix A is said to be symmetric if A T = A. Example. Prove that A T A is a symmetric matrix. Proof: (A T A) T = A T A; hence, it is symmetric by definition. Remarks. Based on the above definition, we can even introduce skew-symmetric matrix defined as A T = A. One advantage of symmetric matrices is that its triple factorization has a symmetric form. In other words, A = LDL T if A is symmetric (and no row exchange during forward elimination). This can save half efforts in, e.g., forward elimination (from n 3 /3downto n 3 /6). (See Slide 2-7.)

87 2.7 Symmetric matrix 2-86 Question: Again, assume without row exchanges during elimination. What will be the complexity of forward elimination for a symmetric A? Answer: i) Forward elimination for matrix A, operations exact a i, 0fori =2,...,n (n ) + n = 2 (n )(n +2) a i,2 0fori =3,...,n (n ) = (n 2)(n +) 2.. a i,n 0fori = n 2= 2 n 4 total 2 (i )(i +2)= n(n )(n +4) 6 i=2 ii) During the above eliminations, we only need to retain each multiplier (i.e., l i,j ) and pivot (i.e., u i,i )toforml and D. (Note that in the above process, we know that A will be reduced to LDL T,whichis of course known after knowing L and D.) [ a, a,2 a,3 a 2, a 2,2 a 2,3 a 3, a 3,2 a 3,3 ] = [ 0 0 l 2, 0 l 3, l 3,2 ][ u, 0 u 2,2 0 0 u 3,3 ] L T

88 ( ) Factorization with row exchange: PA = LU 2-87 What if row exchange is needed during forward elimination? Example. (This is exactly the same example that appears on slide 2-66!) () with 2 the first pivot (2) (Replace the middle 9 by 8) (3) () with 2 the first pivot (2) 2 () (3) ( ) () () (2 )with the second pivot (3 ) () (2 )with the second pivot (3 ) 0 (2 )

89 2.7 Factorization with row exchange: PA = LU 2-88 The forward elimination process can then be implemented by multiplying E 2,, E 3,, P 2,3 and E 3,2 in sequence ( ) }{{}}{{}}{{}}{{} E 3,2 P 2,3 E 3, E 2, = Then we have that after forward elimination E 3,2 P 2,3 E 3, E 2, [ A I ] = [ EA E ] = [ U E ] where E = E 3,2 P 2,3 E 3, E 2,. Next, we need to determine L = E = E 2, E 3, P 2,3 E 3,2. This is easy because every E ij is formed by changing just one off-diagonal entry from an identity matrix, and the inverse of P 2,3 is itself!

90 2.7 Factorization with row exchange: PA = LU 2-89 What we need to do is to negate the only non-zero off-diagonal entry! = 0 0, 0 0 = }{{ } 0 ( ) 0 }{{ } ( ) 0 E3,2 E3, L = E = E2, E 3, P 2,3 E 3,2 = and } 0 0 {{ } E2, = Not a lower triangular matrix! Then, we obtain A = LU =

91 2.7 Factorization with row exchange: PA = LU 2-90 Hence, P A = 0 0 A = P LU = LU = Factorization with row exchange: Find permutation matrix P such that PA = LU. The pre-permutation matrix that can force zero row exchange is not unique! For (another) example, check the below for the above example A =

92 2.7 Factorization with row exchange: PA = LU 2-9 The MATLAB command for LU-factorization is [L,U,P]=lu(A) Note: The MATLAB will make row exchanges such that the next pivot is the largest! This is the reason why for A = 4 8 3, the first pivot that MATLAB chooses is 4!

93 2.7 Factorization with row exchange: PA = LU 2-92 Some useful notes A permutation matrix P (which has the rows of identity matrix I in any order) satisfies P T = P. Proof: Permutation does not change the inner product; hence (P x) T (P y) = x T y.sincexand y are arbitrary, P T P = I. It is not necessary true that P = P as P may not be symmetric. There are n! permutation matrices of size n n. The determinant of P is either (when the number of row changes is even) or (when the number of row changes is odd). We can also use forward elimination to make symmetric of a matrix A, i.e., A = ES where E is an elementary row operation matrix, and S is symmetric. The idea is simply to make the lower triangle entry a i,j the same as a j,i (instead of making a i,j =0). [ ] [ ][ ] Example (Problem 34 on page 9). A = = = ES

94 2.7 Factorization with row exchange: PA = LU 2-93 [ ] 2 (Problem 34, Section 2.7) Write A = as the product EH of an 3 4 elementary row operation matrix E and a symmetric matrix H. Definition (Elementary matrix): An elementary (row operation) matrix is a matrix that differs from the identity matrix by one single elementary row operation, which includes. row exchanges; 2. row multiplications; and 3. row additions. In fact, we can use triple factorization to symmetricize a matrix. A = LDU = L(U T ) }{{ (U } T DU) = ES. }{{} E S