Math 1553 Introduction to Linear Algebra

Math 1553 Introduction to Linear Algebra Lecture Notes Chapter 2 Matrix Algebra School of Mathematics The Georgia Institute of Technology Math 1553 Lecture Notes for Chapter 2 Introduction, Slide 1

Section 2.1: Matrix Operations Terminology Suppose A is an m n matrix. Then a ij or A ij is the ij th entry a ii are diagonal entries diagonal matrix: all non-diagonal entries are 0 zero matrix: all entries are 0 Sums and Scalar Multiples Same as for vectors: component-wise, so matrices must be same size to add. Basic rules: A + B = B + A (A + B) + C = A + (B + C) r(a + B)rA + rb (r + s)a = ra + sa (rs)a = r(sa) A + 0 = A Math 1553 Lecture Notes for Chapter 2 Section 2.1: Matrix Operations, Slide 2

Matrix Multiplication If A = m n and B = n p, then AB is m p and (AB) ij = k a ik b kj = r i b j where r i is the i th row of A, and b j is the j th column of B. Clicker Question Are there A, B 0 such that AB = 0? 1. Yes 2. No Math 1553 Lecture Notes for Chapter 2 Section 2.1: Matrix Operations, Slide 3

Matrix Multiplication and Linear Transforms Fact. T AB = T A T B. So: matrix multiplication composition of linear transforms Why? It is enough to check that T AB(e i) = T A T B(e i). T AB(e i) = AB(e i) = i th col of AB r 1ci r 2c i =. r mc i = Ac i = A(Be i) = A(T B(e i)) = T A T B(e i) Another view: AB = A(c 1 c p) = (Ac 1 Ac p). Math 1553 Lecture Notes for Chapter 2 Section 2.1: Matrix Operations, Slide 4

Properties of Matrix Multiplication A(BC) = (AB)C A(B + C) = AB + AC (B + C)A = BA + CA r(ab) = (ra)b = A(rB) I ma = A = AI n, where I m is the m n identity matrix. Associativity Multiplication is associative because function composition is (just check with the formula). Communativity In general, AB BA AB = AC does not mean that B = C AB = 0 does not mean that A or B is 0 Math 1553 Lecture Notes for Chapter 2 Section 2.1: Matrix Operations, Slide 5

Section 2.2 The Inverse of a Matrix A = n n matrix. A is invertible (or non-singular) if there is a matrix A 1 with AA 1 = A 1 A = I n Where A 1 is the inverse of A. For example, ( ) 1 ( 2 1 1 1 = 1 1 1 2 Q. Can you guess the inverse of ( 1 1 0 1 )? ) Q. Find a matrix that is not invertible. Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 6

Transforms and Inverses Clicker Question Which of the followiong linear transforms of R 3 correspond to invertible matrices? projection to xy-plane rotation about z-axis by π reflection through origin reflection through xy-plane Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 7

The 2 2 Case ( a b Fact. Let A = c d ). Then det(a) = ad bc is the determinant of A. ( d b If det(a) 0 then A is invertible and A 1 = 1 ad bc c a If det(a) = 0 then A is not invertible. Why? If A 0 we can verify this by calculating A 1 A or AA 1. If deta = 0, then 1/(ad bc) is undefined, and so A 1 is also undefined. ). Example. ( 1 2 3 4 ) 1 ( 4 2 = 1 2 3 1 ). Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 8

Solving Linear Systems via Inverses Fact. If A is invertible, then Ax = b has exactly one solution, namely x = A 1 b. Why? Example. Solve 2x + 3y + 2z = 1 x + 3z = 1 2x + 2y + 3z = 1 Using 2 3 2 1 0 3 2 2 3 1 = 6 5 9 3 2 4 2 2 3 Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 9

Some Facts If A, B are invertible matrices. A 1 is inveritble and (A 1 ) 1 = A AB is invertible and (AB) 1 = B 1 A 1 A T is invertible and (A T ) 1 = (A 1 ) T Q. What is (ABC) 1? A. (ABC) 1 = ((AB)C) 1 = C 1 (AB) 1 = C 1 B 1 A 1. Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 10

An Algorithm for Finding A 1 Suppose A = n n matrix. Row reduce (A I n) If reduction has form (I n B) then A is invertible and B = A 1. Otherwise, A is not invertible. 1 0 4 1 Example. Find 0 1 2. 0 3 4 A. 1 0 4 1 0 0 0 1 2 0 1 0 0 3 4 0 0 1 Thus, the inverse is 1 6 2 0 2 1 0 3/2 1/2. 1 0 4 1 0 0 0 1 2 0 1 0 0 0 2 0 3 1 1 0 0 1 6 2 0 1 0 0 2 1 0 0 1 0 3/2 1/2 Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 11

Why Does This Work? First answer: we can think of the algorithm as simultanenously solving Ax 1 = e 1 Ax 2 = e 2 and so on. But the columns of A 1 are A 1 e i, which is x i. There is another explanation, which uses elementary matrices. Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 12

Elementary Matrices Definition. operation. An elementary matrix, E, is one that differs by I n by one row Fact. If E is an elementary matrix for some row operation, then EA differs from A by some row operation. Why? To verify, check each row operation. Fact. Elementary matricies are invertible. Observation. An n n matrix A is invertible iff it is row equivalent to I n. In this case, the sequence of row operations taking A to I n also takes I n to A 1. This gives us a second explanation of the algorithm. Why is it true? E k E k 1 E 2E 1A = I n. If we multiply on right by A 1, then E k E k 1 E 2E 1I n = A 1. The above is an expression for the inverse of A. Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 13

Cryptography Encode letters A,B,C,..., Z by 1, 2, 3,..., 26. Choose a matrix, A, say n n. Break messages into blocks of size n, which gives us (a set of) vectors. Apply A to each vector to get encrypted message. 2 3 1 Example. A = 2 0 1, and the encoded message is 0 0 6 What is the encoded message? We can also ask: 112 52 36 After decoding one message, can you use the same matrix to decode other messages? Can you decode (76, 37, 42) T? (81, 36, 72) T? Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 14

Structural Engineering Suppose we put 3 downward forces on an elastic beam. By Hooke s law, the vertical displacements at those three points y 1, y 2, y 3 are given by a linear transformation. f 1 y 1 A f 2 f 3 = y 2 y 3 If we want to achieve a certain displacement, use A 1 to find the required forces. Math 1553 Lecture Notes for Chapter 2 Section 2.2: The Inverse of a Matrix, Slide 15

Section 2.3 Characteristics of Invertible Matrices The Invertible Matrix Theorem Suppose A = n n matrix, and T A : R n R n the associated linear transform. The following are equivalent. a) A is invertible b) A is row equivalent to I n c) A has n pivots d) Ax = 0 has only 0 solution e) columns of A are linearly independent f) T I is one-to-one g) Ax = b is consistent for all b in R n h) columns of A span R n i) T A is onto j) A has a left inverse k) A has a right inverse l) A T is invertible Math 1553 Lecture Notes for Chapter 2 Section 2.3: Characteristics of Invertible Matrices, Slide 16

The Invertible Matrix Theorem One possible road map. Math 1553 Lecture Notes for Chapter 2 Section 2.3: Characteristics of Invertible Matrices, Slide 17

The Invertible Matrix Theorem (IMT) There are two kinds of square matricies, invertible (or non-singular), and non-intertible (or singluar) matricies. For invertible matrices, all of the conditions in the IMT hold. And for a non-invertible matrix, the negations of the statements hold. Q. State the negations of statements (a) to (l). Clicker Question Are the following equivalent? m) rows of A span R n n) rows of A are linearly independent o) Ax = b has exactly one solution for all b in R n p) det(a) 0, where det(a) is the volume of the parallelepiped formed by the columns of A q) A 2 is invertible Math 1553 Lecture Notes for Chapter 2 Section 2.3: Characteristics of Invertible Matrices, Slide 18

Invertible Functions A function f : X Y is invertible if there is a function g : Y X, so f g = g g. That is, g f(x) = x for all x X and g f(x) = x for all x X Fact 1. If a function has an inverse, it is unique. Call it f 1. Fact 2. Invertible functions are one-to-one and onto. Fact 3. Suppose A = n n matrix, and T A the associated linear transform. Then T A is invertible as a function if and only if A is invertible. And in this case, (T A) 1 = T A 1. Why? T A invertible implies (by Fact 2) T A is onto, which implies (by the IMT) that A is invertible. And A invertible implies that T A T A 1 = T AA 1 = T I = id., and T A 1 T A = id. Math 1553 Lecture Notes for Chapter 2 Section 2.3: Characteristics of Invertible Matrices, Slide 19

An Additional Example Q. Determine whether matrix A is invertible. A = 1 0 2 3 1 2 5 1 9 A. It isn t necessary to find the inverse. Instead, we may use the Invertible Matrix Theorem by checking whether we can row reduce to obtain three pivot columns, or three pivot positions. A = 1 0 2 3 1 2 5 1 9 1 0 2 0 1 4 0 1 1 1 0 2 0 1 4 0 0 3 By inspection, there are three pivot positions, so A is invertible by the IMT (statement c). Math 1553 Lecture Notes for Chapter 2 Section 2.3: Characteristics of Invertible Matrices, Slide 20

Section 2.5 Matrix Decompositions Recall. If we want to solve Ax = b, we can find A 1 and obtain x by A 1 b. Calculating A 1 with row reduction is not efficient, especially for large matrices. The LU decomposition is an efficient method for solving Ax = b. An LU factorization of A = m n is the product A = LU, where L = m m lower triangular matrix U = m n echelon form of A To solve Ax = b, we write Ax = LUx = b, and then 1. solve Ly = b, to obtain y, 2. solve Ux = y to obtain x. This approach only uses back substitution, not elimination. Also note that L is a unit lower triangular matrix, meaning that all the elements on its diagonal are equal to 1. U is an upper triangular matrix Historical note: the LU Decomposition was introduced by Alan Turing. Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 21

Composition and Transformations The LU decomposition provides another example where we can view a matrix multiplication as a composition. Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 22

Example Calculation of a LU Factorization A = 6 0 2 24 1 8 12 1 3 = 1 0 0 4 1 0 2 1 1 6 0 2 0 1 0 0 0 1 = LU Matrix L records negatives of the row operations. Be careful to go in order, and do not make any row swaps. Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 23

Why Does This Method Work? Row operations are elementary matrices, so E 4E 3E 2E 1A = U A = (E 4E 3E 2E 1) 1 U = E 1 1 E 1 2 E 1 3 E 1 4 U = LU All the elementary matrices are lower triangular. It can be shown that the inverse of each E exists and is lower triangular, the product of lower triangular matrices is lower triangular, and so L exists and is lower triangular. Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 24

Clicker Question Clicker Question Which of the following statements are true? 1. Every matrix has an LU factorization 2. LU factorizations are unique 3. Computing the LU is faster than row reduction Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 25

Using LU To Solve a Linear System Q. Solve Ax = 4 19 6, with the same A as in the previous example. A. Step 1: Solve Ly = b. Step 2: Solve Ux = y. Ly = Ux = 4 19 6 4 3 1 y = x = 4 3 1 1 3 1 Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 26

Further Remarks Non-Square Matrices The LU factorization can be computed for non-square matrices. ( ) ( ) ( ) 2 1 3 1 0 2 1 3 = 4 4 1 2 1 0 6 5 Row Permutations The above procedure for LU factorization works when we don t need to swap rows in row reduction. If a row-permutation is needed, we introduce a permutation matrix, P, so that P A = LU, or A = P LU. Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 27

Application: LU Factorization in Electrical Engineering In an electrical circuit, current i and voltage v often change by a linear transformation (by Ohm s law and Kirchoff s law). ( v1 So A i 1 ) = ( v2 i 2 ) for some transfer matrix A. Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 28

Circuit Examples A = ( 1 R1 0 1 ) current unchanged, voltage decreases proportional to current ( A = 1 0 1/R 2 1 voltage unchanged, current decreases proportional to voltage If we string these together we get a ladder circuit. The transfer matrix for the ladder circuit is the product of the matrices for the components. Why does this make sense? Think about function composition. ) Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 29

Example Consider the following circuit. The matrix for this circuit is ( 1 0 1/R 2 0 ) ( 1 R1 0 1 ) ( = 1 R 1 1/R 2 1 + R 1/R 2 Can ( you reverse) engineer a ladder circuit whose transfer matrix is 1 8? 0.5 5 Use the LU decomposition. ) Math 1553 Lecture Notes for Chapter 2 Section 2.5: Matrix Decompositions, Slide 30

Section 2.8: Subspaces of R n A subspace of R n is a subset V with three properties. 1. The zero vector is in V. 2. If u, v are vectors in V, then u + v is also in V. 3. If u is a vector in V and c is in R, then cu V. Note the following. We say that a subspace is closed under addition and scalar multiplication. We can represent (or visualize) spans as lines or planes that pass through the origin. If V = span{v 1, v 2,..., v k }, then we say V is the subspace generated by vectors v 1, v 2,..., v k. Example. If v 1, v 2 are vectors in R n, then H = span{v 1, v 2} is a subspace. Why? We can verify the three properties above for vectors v 1, v 2. A Non-Example A line that does not pass through the origin is not a span. Why not? Math 1553 Lecture Notes for Chapter 2 Section 2.8: Subspaces of R n, Slide 31

Clicker Question Clicker Question Consider V = { a b c d Is V a subspace of R 4? 1. Yes 2. No R4 : ad bc = 0}. Math 1553 Lecture Notes for Chapter 2 Section 2.8: Subspaces of R n, Slide 32

Column Space and Null Space Suppose A is an m n matrix. We define the following subspaces. The column space of A, Col(A), is the span of the columns, in R m. The null space of A, Nul(A), is the set of solutions to Ax = 0, in R n. Example. A = 1 1 1 1 1 1 Then Col(A) = span{ 1 1 1. Nul(A) is x + y = 0, a line in R 2. } = line in R 3. Math 1553 Lecture Notes for Chapter 2 Section 2.8: Subspaces of R n, Slide 33

Bases Let V be a subspace in R n. A basis for V is a set of vectors {v 1, v 2,..., v k } such that 1. V = span{v 1,..., v k } 2. the vectors v i are linearly independent We refer to k as the dimension of subspace V, and k = dim(v ). Example. The standard basis for R n is the set e 1, e 2,..., e n. Q. What is a basis for the set of Fibonacci sequences in R? Math 1553 Lecture Notes for Chapter 2 Section 2.8: Subspaces of R n, Slide 34

Example Q. Find a basis for the null space and a basis for the column space of 1 1 1 A = 1 1 1. 1 1 1 A. Nul(A) is the span of solutions to Ax = 0, which for this matrix we can show is given by 1 1 y 1 0 + z 0 1 So the basis for nullspace of A is { 1 1 0, 1 0 1 }. Math 1553 Lecture Notes for Chapter 2 Section 2.8: Subspaces of R n, Slide 35

Further Remarks Note that In general, the pivot columns of A form a basis for Col(A). Be careful to not use the reduced pivot columns in constructing the basis for Col(A). If A is a square matrix with dimensions n n, then A is invertible iff the columns of A form a basis for R n. Math 1553 Lecture Notes for Chapter 2 Section 2.8: Subspaces of R n, Slide 36

Section 2.9 Dimension and Rank Bases as Coordinate Systems Suppose V is a subspace of R n, B = {b 1, b 2,..., b k } is a basis for V, and x is a vector in V. Then we can write x uniquely as c 1b 1 + c 2b 2 + + c k b k We write [x] B = c 1 c 2.. c k The vector [x] B is called the coordinate vector of x relative to B, or the B coordinate vector of x. Math 1553 Lecture Notes for Chapter 2 Section 2.9 Dimension and Rank, Slide 37

Example Q. Suppose b 1 = 1 0 1, b 2 = 1 1 1, x = 5 3 5, and V = span{b 1, b 2}. What is a basis for V, and what is the coordinate vector of x relative to B? A. The set B = {b 1, b 2} is a basis for the subspace V. The coordinate vector, [x] B, is found by solving c 1b 1 + c 2b 2 = x We find the values of c 1 and c 2 by solving a system. 1 1 ( ) 5 0 1 c1 = 3 c 1 1 2 5 This leads to [x] B = ( 2 3 ). Math 1553 Lecture Notes for Chapter 2 Section 2.9 Dimension and Rank, Slide 38

Dimension Definition. Let V be a subspace of R n. We define dim V to be the number of vectors in a basis for V. Note: the basis for {0} is {}. That is, dim{0} = 0. Clicker Question If U, V are 2 dimensional subspaces of R 4, what are the possible dimensions for U V? a) 0 b) 1 c) 2 d) 3 e) 4 Math 1553 Lecture Notes for Chapter 2 Section 2.9 Dimension and Rank, Slide 39

Rank Theorem We define the following. rank(a) = dim Col(A) = # pivot columns dim Nul(A) = # non pivot columns = # free variables Now recall that for an m n matrix A, (# pivot columns) + (# free variables) = n. By substituting the above definitions into this expression, we obtain the Rank Theorem. Rank Theorem If A is an m n matrix, then rank(a) + dim Nul(A) = n. Example. If A = 1 1 1 1 1 1 1 1 1, then rank(a) = 1, and dim Nul(A) = 2. Math 1553 Lecture Notes for Chapter 2 Section 2.9 Dimension and Rank, Slide 40

Clicker Question Clicker Question If A and B are 3 3 matrices, and rank(a) = rank(b) = 2 then what are the possible values of rank(ab) a) 0 b) 1 c) 2 d) 3 e) 4 Math 1553 Lecture Notes for Chapter 2 Section 2.9 Dimension and Rank, Slide 41

Two More Theorems Basis Theorem If V is a k dimensional subspace of R n, then any k linearly independent vectors of V form a basis for V any k vectors that span V form a basis for V Invertible Matrix Theorem (m) cols of A form a basis for R n (n) Col(A) = R n (o) dim Col(A) = n (p) rank(a) = n (q) Nul(A) = {0} (r) dim Nul(A) = 0 Why? Math 1553 Lecture Notes for Chapter 2 Section 2.9 Dimension and Rank, Slide 42

Chapter 2 in a Nutshell Products AB BA T AB = T A T B Inverses if AB = BA = I, then B = A 1 T 1 A = T A 1 Ax = b implies x = A 1 b application is flexibility matrix, D forces=displacement Find A 1 by row reducing (A I) to get (I A 1 ) Another perspective: E k E k 1 E 2E 1A = I, where each E i is an elementary matrix. This implies that E k E k 1 E 2E 1 = A 1. Math 1553 Lecture Notes for Chapter 2 Chapter 2 in a Nutshell, Slide 43

Chapter 2 in a Nutshell Invertible Matrix Theorem The following are equivalent A is invertible A can be reduced to I A has n pivots NulA = {0} T A is one-to-one T A is onto Ax = b is consistent for all b ranka = n etc... Math 1553 Lecture Notes for Chapter 2 Chapter 2 in a Nutshell, Slide 44

Chapter 2 in a Nutshell LU Decompositions Let A = LU, where L is lower triangular, U is in echelon form. Easy to solve for many b. Algorithm Step 1. Solve Ly = b Step 2. Solve Ux = y To find L, U, we row reduce A to echelon form to obtain U, then use negative of row operations to obtain L. Be careful to not row swap and only use lower row replacement. Application: circuits A = ( 1 R1 0 1 ) ( A = 1 0 1/R 2 1 ) Math 1553 Lecture Notes for Chapter 2 Chapter 2 in a Nutshell, Slide 45

Chapter 2 in a Nutshell Subspaces A subspace is a non-empty subset closed under taking linear combinations. Two important subspaces are Col(A), which is the span of the columns of matrix A. Nul(A), which is the span of the solutions to Ax = 0. A basis for a subspace, W, is the set of linearly independent vectors that spans W. The dimension of a subspace is the number of elements in the basis. Use row reduction to find a basis for Col(A) or Nul(A). Pivot columns are used to identify the basis for Col(A). Parametric form is used to identify the basis for Nul(A). Coordinate Vector If B = {b 1,..., b k } is a basis for a subspace, then we find the B-coordinates for vector x, are found by solving (b 1, b 2,..., b k )c = x for vector c. Math 1553 Lecture Notes for Chapter 2 Chapter 2 in a Nutshell, Slide 46

Chapter 2 in a Nutshell Rank Theorem Let A be an m n matrix, then rank(a) + dim(nul(a)) = n. Basis Theorem Suppose V is a k-dimensional subspace of R n. Then Any k linearly independent vectors form a basis. Any k spanning vectors form a basis. Math 1553 Lecture Notes for Chapter 2 Chapter 2 in a Nutshell, Slide 47