Math 403 Linear Algebra Kenneth Massey Carson-Newman College February 15, 2012 C-N Math 403 - Massey, 1 / 65
Linear Algebra Introduction Linear essentially means flat, and algebra is about how objects interact. Key to working efficiently and elegantly in higher dimensions. Nonlinear models are linearized; continuous problems are discretized to yield linear algebra problems. C-N Math 403 - Massey, 2 / 65
Toy Example Introduction Each wagon requires 4 cogs and 1 sprocket. Each tricycle requires 2 cogs and 3 sprockets. Hermey has used 22 cogs and 18 sprockets today. How many wagons and tricycles did he make? C-N Math 403 - Massey, 3 / 65
Toy Example Introduction Two ways to visualize the equations: intersection of lines linear combination of vectors C-N Math 403 - Massey, 4 / 65
Toy Example Introduction Add a Part: Wagons require 7 widgets and tricycles requre 3 widgets. How many widgets did Hermey use? Add a Toy: Racecars require 5 cogs and 2 sprockets. Another elf used 122 cogs and 58 sprockets. How many toys of each type did he make? Add Part and Toy? C-N Math 403 - Massey, 5 / 65
Vectors in R n Introduction list of numbers / coordinates, ordered n-tuple, array lower case letters at end of alphabet, e.g. x R n (from context or bold faced) subscript notation, dimension n column / row vectors (transpose x T toggles) zero vector 0 two operations: addition and scalar multiplication C-N Math 403 - Massey, 6 / 65
Properties of Vector Algebra Introduction For any x, y, z R n, and scalars R, closure: cx + y R n associative: (x + y) + z = x + (y + z ) and (c 1 c 2 )x = c 1 (c 2 x ) commutative: x + y = y + x distributive: (c 1 + c 2 )x = c 1 x + c 2 x and c(x + y) = cx + cy identities: x + 0 = x and 1(x ) = x additive inverse: x + ( x ) = 0 C-N Math 403 - Massey, 7 / 65
Geometric View Introduction length and direction add head to tail e.g. if u + v = w then w u = v Definition Euclidean length/norm x = n i=1 x 2 i Definition unit vector u = 1, e.g. u = x x C-N Math 403 - Massey, 8 / 65
Linear Combinations Introduction Definition x is a linear combination of a set of vectors {v i } n i=1 iff constants/scalars c i such that x = n c i v i i=1 C-N Math 403 - Massey, 9 / 65
Matrix A matrix is an array/table/grid of catenated vectors. Introduction denoted by capital letters A R m n, size m n, m rows, n columns a j denotes the j th column ãi T denotes the ith row a ij is the entry in row i, column j A = [a 1 a 2 a n ] = ã T 1 ã T 2. ã T m = [a ij ] C-N Math 403 - Massey, 10 / 65
Matrix-Vector Multiplication Introduction Definition Let A R m n and x R n. Then the matvec product is defined by Ax = n x j a j j =1 This is a linear combination of the columns of A using coefficients from the vector x. Think of matrix-vector multiplication as a function A : R n R m. C-N Math 403 - Massey, 11 / 65
Linear System Introduction A linear system of equations with m equations and n unknowns (variables, degrees of freedom) can be written compactly as Ax = b where A R m n is the coefficient matrix x R n is the vector of unknowns b R m is the RHS vector The augmented matrix for the system is ^A = [A b] R m (n+1) C-N Math 403 - Massey, 12 / 65
Solutions Introduction Consider the linear system Ax = b siumultaneous intersection of m hyper-planes in R n linear combination of n vectors in R m consistent if a solution exists C-N Math 403 - Massey, 13 / 65
Elementary Row Operations Introduction Equivalent systems have the same solution set. Any elementary row operation (ERO) yields an equivalent ( ) system. Swap two rows. Multiply a row by a non-zero constant. Add a multiple of one row to another row. Each ERO is invertible. C-N Math 403 - Massey, 14 / 65
Gauss Elimination Introduction Gauss (Jordan) elimination uses EROs to systematically create a sequence of increasingly simple equivalent systems. [ 4 2 22 1 3 18 ] [ 1 3 18 0 1 5 ] [ 1 3 18 2 1 11 [ 1 0 3 0 1 5 ] ] [ 1 3 18 0 5 25 ] C-N Math 403 - Massey, 15 / 65
Row Echelon Form Introduction Row echelon form (REF): The first non-zero entry in a row (pivot) is to the right of the pivot above it. Zero rows are below all non-zero rows. Reduced row echelon form (RREF) has the additional property: Each pivot equals 1, and is the only non-zero in its column. C-N Math 403 - Massey, 16 / 65
Back-Substitution Introduction Use EROs to put the augmented matrix in REF: 3 4 1 35 2 4 8 18 1 0 1 1 2 3 11 0 0 1 Then use back-substitution to solve the system. C-N Math 403 - Massey, 17 / 65
REF Solutions Introduction Consider the (R)REF of the augmented matrix, but ignore the all zero rows. If a column contains a pivot, then that variable is determined. Otherwise, that variable is a free parameter. Existence: If pivot in each row, then consistent. Uniqueness: If pivot in each column, then no free params, so if consistent then unique. C-N Math 403 - Massey, 18 / 65
Three Planes Introduction C-N Math 403 - Massey, 19 / 65
Solutions Most likely case is highlighted. m = n m < n m > n unique infinite (redundancy) none (inconsistent) infinite (under-determined) none (inconsistent) none (over-determined, inconsistent) unique (redundancy) infinite (redundancy) Introduction C-N Math 403 - Massey, 20 / 65
Parametric Solution Introduction Solution possibilities: inconsistent point (unique) line (1 free param) plane (2 free param) etc [ ] 1 0 2 3 solved by x = 0 1 4 5 3 5 + 0 2 4 t 1 C-N Math 403 - Massey, 21 / 65
Homogeneous Solution Introduction Ax = 0 (the RHS is zero) is called homogeneous Definition The kernel (null space) of A R m n is Ker(A) = {x R n Ax = 0} trivial solution 0 Ker(A) If x, y Ker(A) then cx + y Ker(A). C-N Math 403 - Massey, 22 / 65
General Solution Introduction Theorem Let p be a particular solution to Ax = b. The general solution to Ax = b is the set: {p + h} h Ker(A)} C-N Math 403 - Massey, 23 / 65
Example Introduction Consider Ax = b. Find the general solution and Ker(A) if the RREF of the augmented matrix is: 1 2 0 0 5 1 0 0 1 0 4 2 0 0 0 1 6 3 3 pivots; x 2 and x 5 are free x = p + (t 1 h 1 + t 2 h 2 ) C-N Math 403 - Massey, 24 / 65
Quiz Introduction Write 5 6 as a linear combination of: 12 1 0, 0 2 4, and 0 3 5. 6 C-N Math 403 - Massey, 25 / 65
Example Introduction Consider the linear system: 2x + 5y = 7 6x + αy = β Under what conditions: unique soln (2α 5 6 0) infinite solns no soln C-N Math 403 - Massey, 26 / 65
Interpolation Introduction Set up and solve (Octave) linear system to: 1. Find parabola through three points. 2. Find circle through three points. 3. Find cubic with given local min and max. C-N Math 403 - Massey, 27 / 65
Economy Model Introduction Five sector closed economy represented by.1.2.2.35.25.1.1.2.1.25 A =.15.3.15.1.25.4.15.2.2.25.25.25.25.25 0 Solve the homogeneous equation Ax x = 0 for a non-trivial solution. C-N Math 403 - Massey, 28 / 65
Canonical Basis Vectors Matrix Algebra e i is a column vector with all zeros except for a one in position i, The identity matrix is I n = [e 1 e 2 e n ] R n n In R 3, visualize e 1, e 2, e 3 ( i, j, k in physics) What is e 2 R 4? Any vector is a lin comb of CBV s 2 5 = 2e 1 + 5e 2 + e 3 1 C-N Math 403 - Massey, 29 / 65
Matrix Arithmetic Matrix Algebra addition and scalar multiplication are defined entrywise (sizes must match) linear combination of matrices, e.g. c 1 A + c 2 B equality, additive identity/inverse, associative/commutative/distributive properties are inherited as you d expect. C-N Math 403 - Massey, 30 / 65
Transpose Matrix Algebra Definition The transpose of the matrix A R m n is A T = a T 1 a T 2. a T n Rn m (A T ) T = A (ca + B) T = ca T + B T C-N Math 403 - Massey, 31 / 65
Special Matrices Matrix Algebra Definition A square matrix A is symmetric if A T = A. Definition A matrix D is diagonal if i j = d ij = 0. notation D = diag(2, 5, 3) square diagonal matrices are symmetric for any square matrix A, A + A T is symmetric C-N Math 403 - Massey, 32 / 65
Matrix Multiplication Matrix Algebra Definition Let A R m p and B R p n. Then AB = [Ab 1 Ab 2 Ab n ] R m n (m p)(p n) = m n Each column of AB is a linear combination of the columns of A. Distributive, but not generally commutative. C-N Math 403 - Massey, 33 / 65
Matrix Algebra Lemma Ae i = a i Lemma (AB)x = A(Bx ) Proof. (AB)x = x j (Ab j ) = A( x j b j ) = A(Bx ) C-N Math 403 - Massey, 34 / 65
Theorem Matrix multiplication is associative, i.e. (AB)C = A(BC ) Proof: show each column matches. Matrix Algebra ((AB)C )e j = (AB)(Ce j ) = A(B(Ce j )) = A((BC )e j ) = (A(BC ))e j Therefore, the entire matrices are equal. C-N Math 403 - Massey, 35 / 65
Selecting from Matrices Matrix Algebra pick out a column Ae j = a j pick out a row ei T A = ãi T pick out an entry ei T Ae j = a ij C-N Math 403 - Massey, 36 / 65
Vector Products Matrix Algebra If x, y R n, the inner/dot product is: x T y = x i y i R. the outer product is: xy T R m n each column is a multiple of x each row is a multiple of y C-N Math 403 - Massey, 37 / 65
Lemma x T A T = (Ax ) T = x T A T Proof. Matrix Algebra (Ax ) T = ( xi a i ) T = x i ai T = x i ei T A T ( ) T = xi e i A T = x T A T C-N Math 403 - Massey, 38 / 65
Matrix Algebra Theorem (AB) T = B T A T Proof: show each row matches. e T i (AB) T = ((AB)e i ) T = (A(Be i )) T = (Be i ) T A T = (e T i B T )A T C-N Math 403 - Massey, 39 / 65
Matrix Algebra Alt. Proof: show each entry matches. e T i ((AB) T )e j = e T j (AB)e i = (e T j A)(Be i ) = ã T j b i = b T i ã j = (e T i B T )(A T e j ) = e T i (B T A T )e j C-N Math 403 - Massey, 40 / 65
Four Views of Matrix Mult. Matrix Algebra Each column of AB is a linear combination of A s columns: (AB)e k = Ab k Each row of AB is a linear combination of B s rows: ek T(AB) = ã k TB Entries are inner products of A s rows with B s columns: [AB] ij = ãi T b j AB is the sum of outer products of A s columns and B s rows: AB = a i b i T C-N Math 403 - Massey, 41 / 65
Matrix Algebra Multiplying by Diagonal Matrices Diagonal matrices commute. post-multiplying by a diagonal matrix scales the columns. pre-multiplying by a diagonal matrix scales the rows. Multiplying by the identity does nothing: if A R m n, then AI n = I m A = A C-N Math 403 - Massey, 42 / 65
Inverse Matrix Matrix Algebra Definition Let A be a square matrix. Then A is invertible (non-singular) if there exists a matrix B such that AB = BA = I The inverse is unique. Proof: Suppose AB = BA = I and AC = CA = I. Then B = IB = CAB = CI = C. Denote the inverse by A 1. Pairs: B is A s inverse and vice-versa. C-N Math 403 - Massey, 43 / 65
Inverses Matrix Algebra To verify that two matrices are inverses, show that the product is I. (Fact to be proved later: if A square and AB = I, then BA = I.) (A 1 ) 1 = A. diag(d ii ) 1 = diag(1/d ii ) [ ] 1 [ ] a b d b If = c d 1 ad bc c a (provided ad bc 0) (AB) 1 = B 1 A 1 C-N Math 403 - Massey, 44 / 65
Using the Inverse Matrix Algebra To solve 2x = 6, multiply both sides by 2 1. Being invertible means that multiplication is reversible. If A is invertible, then Ax = b x = A 1 b. Be careful about non-commutativity. If AX = B then X = A 1 B, but if XA = B then X = BA 1. C-N Math 403 - Massey, 45 / 65
Cancellation Matrix Algebra Be careful about cancellation. Check [ that AB ] = AC [, but ] B C [: ] 2 3 8 4 5 2 A =, B =, C = 4 6 5 5 3 1 Prove that if A is invertible and AB = AC, then B = [ C. ] [ ] 1 2 2 6 Let A =. Then AB = 0 2 4 1 3 even though neither A nor B is zero, so neither A nor B could be invertible. C-N Math 403 - Massey, 46 / 65
Finding the Inverse Matrix Algebra AB = I corresponds to n systems of the form Ab i = e i. Row reduce the augmented matrix. If [A I ] [I B], then B = A 1. Octave inv(a) C-N Math 403 - Massey, 47 / 65
Matrix Algebra Elementary Row Operation Matrices Illustrate how each ERO can be accomplished by pre-multiplication by an EROM. 2 4 6 26 3 6 1 9 2 7 0 11 EROs are invertible and yield equivalent systems since Ax = b EAx = Eb. C-N Math 403 - Massey, 48 / 65
Inverse Using EROMs Matrix Algebra The RREF can be computed using a sequence of EROMs: rref(a) = E k E 2 E 1 A If rref(a) = I, then A 1 = E k E 2 E 1 A = E1 1 E2 1 E 1 k C-N Math 403 - Massey, 49 / 65
Invertible Matrix Theorem Matrix Algebra Theorem (IMT) Let A R n n. The following are equivalent: 1. rref(a) has a pivot in every column. 2. rref(a) has a pivot in every row. 3. rref(a) = I 4. Ax = y is consistent y R n 5. Ax = 0 has only the trivial solution 6. Ker(A) = {0} 7. Ax = y has a unique solution y R n C-N Math 403 - Massey, 50 / 65
Invertible Matrix Theorem Matrix Algebra IMT continued Let A R n n. The following are equivalent: 8. B R such that AB R = I 9. B L such that B L A = I 10. B such that AB = BA = I 11. A is invertible (non-singular) 12. A T is invertible 13. A is the product of EROMs C-N Math 403 - Massey, 51 / 65
Parts of the proof: Matrix Algebra (5) implies (7) since if Ax = y and Av = y then A(x v) = 0, so x v = 0 and so x = v (4) implies (8) since Ab i = e i has a soln (8) implies (4) since x = B R y solves Ax = y (3) implies (9) since E k E 1 A = rref(a) = I (9) implies (6) since if Ax = 0 then x = B L Ax = 0 (8) and (9) imply (10) since B R = B L AB R = B L (12) follows since (A T ) 1 = (A 1 ) T (13) follows since A = E 1 1 E 1 k C-N Math 403 - Massey, 52 / 65 Matrix Algebra Matrix Algebra True or False? 1. A k = (A k ) 1 = (A 1 ) k 2. (I + A) 1 = I + A 1 3. (I + A) 1 = (I + A 1 ) 1 A 1 4. (A + B) 1 B = (B 1 A + I ) 1 Solve for X by doing the same thing to both sides of the equation. 5. C (I + 2XC )C = BC 2 6. (A AX ) 1 = X 1 B C-N Math 403 - Massey, 53 / 65
Lucas Sequences Matrix Algebra Given p and q, define the recursive sequence: x 0 = 0, x 1 = 1, x k = px k 1 + qx k 2 Fibonacci p = q = 1 Mersenne p = 3, q = 2 Create[ an update ] matrix [ ] U such that xk xk 1 = U x k 1 x k 2 Find U 1. Write x k in terms of x k+1 and x k+2. Fact: U 2 = pu + qi Find the zenzizenzizenzic of U. Estimate lim k x k x k 1 C-N Math 403 - Massey, 54 / 65
Make Your Own Matrix Algebra 1. Pick p, q, r and create an update matrix for x k = px k 1 + qx k 2 + rx k 3 2. Verify that U 3 = pu 2 + qu + ri 3. Pick x 0, x 1, x 2 and compute through x 9. 4. Compute U 9 and use it to compute x 9. 5. Compute U 1. 6. Write a formula for x k in terms of x k+1, x k+2, and x k+3. 7. Estimate lim k x k x k 1 8. Name the sequence after yourself. C-N Math 403 - Massey, 55 / 65
Matrix Algebra Theorem Suppose A, B are square. If I AB is invertible, then so is I BA. Find the flaw in this proof. Note that B(I AB) = (I BA)B. Post-multiply both sides by (I AB) 1 B 1 : I = (I BA)B(I AB) 1 B 1. Thus (I BA) 1 = B(I AB) 1 B 1. C-N Math 403 - Massey, 56 / 65
Matrix Algebra Constructive proof. Let C = I + B(I AB) 1 A. Show (I BA) 1 = C by demonstrating that C (I BA) = I Proof by contradiction. Assume I BA is not invertible. Then by IMT x 0 with (I BA)x = 0. Then (I BA)x = 0 = A(I BA)x = 0 = (I AB)Ax = 0. But since I AB is invertible, Ker(I AB) = {0}, so Ax = 0. Therefore 0 = (I BA)x = x BAx = x 0, which is a contradiction. C-N Math 403 - Massey, 57 / 65
Functions of Matrices Matrix Algebra Let f (x ) = x 2 +2x 3 x. Then compute f (A) = (A 2 + 2A 3I )A 1 Powers of A commute with each other. (AB) 2 A 2 B 2 (A + B) 1 A 1 + B 1 C-N Math 403 - Massey, 58 / 65
Triangular Matrices Matrix Algebra L is lower triangular if i < j = L ij = 0 L = U is upper triangular if i > j = U ij = 0 U = Diagonal matrices are both upper and lower triangular. C-N Math 403 - Massey, 59 / 65
Block Matrices Matrix Algebra Matrices may be partitioned into sub-matrix blocks. A = AB = 1 2 3 4 3 2 1 0 2 4 6 8 [ A11 A 12 A 21 = [ ] [ ] B1 A 22 B 2 A 11 A 12 A 21 A 22 = ], B = 4 1 2 3 5 0 1 1 [ = [ ] A11 B 1 + A 12 B 2 A 21 B 1 + A 22 B 2 B 1 B 2 ] C-N Math 403 - Massey, 60 / 65
Block Matrices Matrix Algebra Blocks may: reveal special structure/patterns represent different components of the application be assigned to separate processors C-N Math 403 - Massey, 61 / 65
Block Examples Matrix Algebra [ ] 1 [ ] I 0 I 0 1. = X I X I [ ] A 0 2. Under what condition is M = is I A its own inverse? 3. [ Find the inverse of a block diagonal matrix: A11 A22] 4. [ Solve for the ] [ inverse ] of [ a block ] triangular: A11 A 12 X Y I 0 = 0 A 22 Z W 0 I C-N Math 403 - Massey, 62 / 65
Matrix Algebra Theorem The inverse of an upper tri. matrix exists iff i u ii 0. The inverse is upper tri. Proof outline (by induction). Assume [ ] true [ for] n [ n. Write ] UB = I n+1 as α v T β x T 1 0 = 0 W 0 Y 0 I n Then let αβ = 1, Y = W 1, and αx T + v T Y = 0. C-N Math 403 - Massey, 63 / 65
LU Factorization Matrix Algebra Let A = 1 4 2 2 11 13. 3 18 25 1. Find EROMs such that ( E k )A = U, where U is upper triangular. 2. Find lower triangular L such that A = LU. This is a LU factorization/decomposition. 3. Find EROMs such that ( E k )A = I. 4. Write A as the product of EROMs. C-N Math 403 - Massey, 64 / 65
Using the LU Matrix Algebra To solve Ax = LUx = b: 1. Let y = Ux and solve L(y) = b. 2. Solve Ux = y. Use the LU to solve Ax = 3 3. 11 How would you solve x T A = [5, 20, 9]? C-N Math 403 - Massey, 65 / 65