Math 403 Linear Algebra

Transcription

1 Math 403 Linear Algebra Kenneth Massey Carson-Newman College February 15, 2012 C-N Math Massey, 1 / 65

2 Linear Algebra Introduction Linear essentially means flat, and algebra is about how objects interact. Key to working efficiently and elegantly in higher dimensions. Nonlinear models are linearized; continuous problems are discretized to yield linear algebra problems. C-N Math Massey, 2 / 65

3 Toy Example Introduction Each wagon requires 4 cogs and 1 sprocket. Each tricycle requires 2 cogs and 3 sprockets. Hermey has used 22 cogs and 18 sprockets today. How many wagons and tricycles did he make? C-N Math Massey, 3 / 65

4 Toy Example Introduction Two ways to visualize the equations: intersection of lines linear combination of vectors C-N Math Massey, 4 / 65

5 Toy Example Introduction Add a Part: Wagons require 7 widgets and tricycles requre 3 widgets. How many widgets did Hermey use? Add a Toy: Racecars require 5 cogs and 2 sprockets. Another elf used 122 cogs and 58 sprockets. How many toys of each type did he make? Add Part and Toy? C-N Math Massey, 5 / 65

6 Vectors in R n Introduction list of numbers / coordinates, ordered n-tuple, array lower case letters at end of alphabet, e.g. x R n (from context or bold faced) subscript notation, dimension n column / row vectors (transpose x T toggles) zero vector 0 two operations: addition and scalar multiplication C-N Math Massey, 6 / 65

7 Properties of Vector Algebra Introduction For any x, y, z R n, and scalars R, closure: cx + y R n associative: (x + y) + z = x + (y + z ) and (c 1 c 2 )x = c 1 (c 2 x ) commutative: x + y = y + x distributive: (c 1 + c 2 )x = c 1 x + c 2 x and c(x + y) = cx + cy identities: x + 0 = x and 1(x ) = x additive inverse: x + ( x ) = 0 C-N Math Massey, 7 / 65

8 Geometric View Introduction length and direction add head to tail e.g. if u + v = w then w u = v Definition Euclidean length/norm x = n i=1 x 2 i Definition unit vector u = 1, e.g. u = x x C-N Math Massey, 8 / 65

9 Linear Combinations Introduction Definition x is a linear combination of a set of vectors {v i } n i=1 iff constants/scalars c i such that x = n c i v i i=1 C-N Math Massey, 9 / 65

10 Matrix A matrix is an array/table/grid of catenated vectors. Introduction denoted by capital letters A R m n, size m n, m rows, n columns a j denotes the j th column ãi T denotes the ith row a ij is the entry in row i, column j A = [a 1 a 2 a n ] = ã T 1 ã T 2. ã T m = [a ij ] C-N Math Massey, 10 / 65

11 Matrix-Vector Multiplication Introduction Definition Let A R m n and x R n. Then the matvec product is defined by Ax = n x j a j j =1 This is a linear combination of the columns of A using coefficients from the vector x. Think of matrix-vector multiplication as a function A : R n R m. C-N Math Massey, 11 / 65

12 Linear System Introduction A linear system of equations with m equations and n unknowns (variables, degrees of freedom) can be written compactly as Ax = b where A R m n is the coefficient matrix x R n is the vector of unknowns b R m is the RHS vector The augmented matrix for the system is ^A = [A b] R m (n+1) C-N Math Massey, 12 / 65

13 Solutions Introduction Consider the linear system Ax = b siumultaneous intersection of m hyper-planes in R n linear combination of n vectors in R m consistent if a solution exists C-N Math Massey, 13 / 65

14 Elementary Row Operations Introduction Equivalent systems have the same solution set. Any elementary row operation (ERO) yields an equivalent ( ) system. Swap two rows. Multiply a row by a non-zero constant. Add a multiple of one row to another row. Each ERO is invertible. C-N Math Massey, 14 / 65

15 Gauss Elimination Introduction Gauss (Jordan) elimination uses EROs to systematically create a sequence of increasingly simple equivalent systems. [ ] [ ] [ [ ] ] [ ] C-N Math Massey, 15 / 65

16 Row Echelon Form Introduction Row echelon form (REF): The first non-zero entry in a row (pivot) is to the right of the pivot above it. Zero rows are below all non-zero rows. Reduced row echelon form (RREF) has the additional property: Each pivot equals 1, and is the only non-zero in its column. C-N Math Massey, 16 / 65

17 Back-Substitution Introduction Use EROs to put the augmented matrix in REF: Then use back-substitution to solve the system. C-N Math Massey, 17 / 65

18 REF Solutions Introduction Consider the (R)REF of the augmented matrix, but ignore the all zero rows. If a column contains a pivot, then that variable is determined. Otherwise, that variable is a free parameter. Existence: If pivot in each row, then consistent. Uniqueness: If pivot in each column, then no free params, so if consistent then unique. C-N Math Massey, 18 / 65

19 Three Planes Introduction C-N Math Massey, 19 / 65

20 Solutions Most likely case is highlighted. m = n m < n m > n unique infinite (redundancy) none (inconsistent) infinite (under-determined) none (inconsistent) none (over-determined, inconsistent) unique (redundancy) infinite (redundancy) Introduction C-N Math Massey, 20 / 65

21 Parametric Solution Introduction Solution possibilities: inconsistent point (unique) line (1 free param) plane (2 free param) etc [ ] solved by x = t 1 C-N Math Massey, 21 / 65

22 Homogeneous Solution Introduction Ax = 0 (the RHS is zero) is called homogeneous Definition The kernel (null space) of A R m n is Ker(A) = {x R n Ax = 0} trivial solution 0 Ker(A) If x, y Ker(A) then cx + y Ker(A). C-N Math Massey, 22 / 65

23 General Solution Introduction Theorem Let p be a particular solution to Ax = b. The general solution to Ax = b is the set: {p + h} h Ker(A)} C-N Math Massey, 23 / 65

24 Example Introduction Consider Ax = b. Find the general solution and Ker(A) if the RREF of the augmented matrix is: pivots; x 2 and x 5 are free x = p + (t 1 h 1 + t 2 h 2 ) C-N Math Massey, 24 / 65

25 Quiz Introduction Write 5 6 as a linear combination of: , 0 2 4, and C-N Math Massey, 25 / 65

26 Example Introduction Consider the linear system: 2x + 5y = 7 6x + αy = β Under what conditions: unique soln (2α 5 6 0) infinite solns no soln C-N Math Massey, 26 / 65

27 Interpolation Introduction Set up and solve (Octave) linear system to: 1. Find parabola through three points. 2. Find circle through three points. 3. Find cubic with given local min and max. C-N Math Massey, 27 / 65

28 Economy Model Introduction Five sector closed economy represented by A = Solve the homogeneous equation Ax x = 0 for a non-trivial solution. C-N Math Massey, 28 / 65

29 Canonical Basis Vectors Matrix Algebra e i is a column vector with all zeros except for a one in position i, The identity matrix is I n = [e 1 e 2 e n ] R n n In R 3, visualize e 1, e 2, e 3 ( i, j, k in physics) What is e 2 R 4? Any vector is a lin comb of CBV s 2 5 = 2e 1 + 5e 2 + e 3 1 C-N Math Massey, 29 / 65

30 Matrix Arithmetic Matrix Algebra addition and scalar multiplication are defined entrywise (sizes must match) linear combination of matrices, e.g. c 1 A + c 2 B equality, additive identity/inverse, associative/commutative/distributive properties are inherited as you d expect. C-N Math Massey, 30 / 65

31 Transpose Matrix Algebra Definition The transpose of the matrix A R m n is A T = a T 1 a T 2. a T n Rn m (A T ) T = A (ca + B) T = ca T + B T C-N Math Massey, 31 / 65

32 Special Matrices Matrix Algebra Definition A square matrix A is symmetric if A T = A. Definition A matrix D is diagonal if i j = d ij = 0. notation D = diag(2, 5, 3) square diagonal matrices are symmetric for any square matrix A, A + A T is symmetric C-N Math Massey, 32 / 65

33 Matrix Multiplication Matrix Algebra Definition Let A R m p and B R p n. Then AB = [Ab 1 Ab 2 Ab n ] R m n (m p)(p n) = m n Each column of AB is a linear combination of the columns of A. Distributive, but not generally commutative. C-N Math Massey, 33 / 65

34 Matrix Algebra Lemma Ae i = a i Lemma (AB)x = A(Bx ) Proof. (AB)x = x j (Ab j ) = A( x j b j ) = A(Bx ) C-N Math Massey, 34 / 65

35 Theorem Matrix multiplication is associative, i.e. (AB)C = A(BC ) Proof: show each column matches. Matrix Algebra ((AB)C )e j = (AB)(Ce j ) = A(B(Ce j )) = A((BC )e j ) = (A(BC ))e j Therefore, the entire matrices are equal. C-N Math Massey, 35 / 65

36 Selecting from Matrices Matrix Algebra pick out a column Ae j = a j pick out a row ei T A = ãi T pick out an entry ei T Ae j = a ij C-N Math Massey, 36 / 65

37 Vector Products Matrix Algebra If x, y R n, the inner/dot product is: x T y = x i y i R. the outer product is: xy T R m n each column is a multiple of x each row is a multiple of y C-N Math Massey, 37 / 65

38 Lemma x T A T = (Ax ) T = x T A T Proof. Matrix Algebra (Ax ) T = ( xi a i ) T = x i ai T = x i ei T A T ( ) T = xi e i A T = x T A T C-N Math Massey, 38 / 65

39 Matrix Algebra Theorem (AB) T = B T A T Proof: show each row matches. e T i (AB) T = ((AB)e i ) T = (A(Be i )) T = (Be i ) T A T = (e T i B T )A T C-N Math Massey, 39 / 65

40 Matrix Algebra Alt. Proof: show each entry matches. e T i ((AB) T )e j = e T j (AB)e i = (e T j A)(Be i ) = ã T j b i = b T i ã j = (e T i B T )(A T e j ) = e T i (B T A T )e j C-N Math Massey, 40 / 65

41 Four Views of Matrix Mult. Matrix Algebra Each column of AB is a linear combination of A s columns: (AB)e k = Ab k Each row of AB is a linear combination of B s rows: ek T(AB) = ã k TB Entries are inner products of A s rows with B s columns: [AB] ij = ãi T b j AB is the sum of outer products of A s columns and B s rows: AB = a i b i T C-N Math Massey, 41 / 65

42 Matrix Algebra Multiplying by Diagonal Matrices Diagonal matrices commute. post-multiplying by a diagonal matrix scales the columns. pre-multiplying by a diagonal matrix scales the rows. Multiplying by the identity does nothing: if A R m n, then AI n = I m A = A C-N Math Massey, 42 / 65

43 Inverse Matrix Matrix Algebra Definition Let A be a square matrix. Then A is invertible (non-singular) if there exists a matrix B such that AB = BA = I The inverse is unique. Proof: Suppose AB = BA = I and AC = CA = I. Then B = IB = CAB = CI = C. Denote the inverse by A 1. Pairs: B is A s inverse and vice-versa. C-N Math Massey, 43 / 65

44 Inverses Matrix Algebra To verify that two matrices are inverses, show that the product is I. (Fact to be proved later: if A square and AB = I, then BA = I.) (A 1 ) 1 = A. diag(d ii ) 1 = diag(1/d ii ) [ ] 1 [ ] a b d b If = c d 1 ad bc c a (provided ad bc 0) (AB) 1 = B 1 A 1 C-N Math Massey, 44 / 65

45 Using the Inverse Matrix Algebra To solve 2x = 6, multiply both sides by 2 1. Being invertible means that multiplication is reversible. If A is invertible, then Ax = b x = A 1 b. Be careful about non-commutativity. If AX = B then X = A 1 B, but if XA = B then X = BA 1. C-N Math Massey, 45 / 65

46 Cancellation Matrix Algebra Be careful about cancellation. Check [ that AB ] = AC [, but ] B C [: ] A =, B =, C = Prove that if A is invertible and AB = AC, then B = [ C. ] [ ] Let A =. Then AB = even though neither A nor B is zero, so neither A nor B could be invertible. C-N Math Massey, 46 / 65

47 Finding the Inverse Matrix Algebra AB = I corresponds to n systems of the form Ab i = e i. Row reduce the augmented matrix. If [A I ] [I B], then B = A 1. Octave inv(a) C-N Math Massey, 47 / 65

48 Matrix Algebra Elementary Row Operation Matrices Illustrate how each ERO can be accomplished by pre-multiplication by an EROM EROs are invertible and yield equivalent systems since Ax = b EAx = Eb. C-N Math Massey, 48 / 65

49 Inverse Using EROMs Matrix Algebra The RREF can be computed using a sequence of EROMs: rref(a) = E k E 2 E 1 A If rref(a) = I, then A 1 = E k E 2 E 1 A = E1 1 E2 1 E 1 k C-N Math Massey, 49 / 65

50 Invertible Matrix Theorem Matrix Algebra Theorem (IMT) Let A R n n. The following are equivalent: 1. rref(a) has a pivot in every column. 2. rref(a) has a pivot in every row. 3. rref(a) = I 4. Ax = y is consistent y R n 5. Ax = 0 has only the trivial solution 6. Ker(A) = {0} 7. Ax = y has a unique solution y R n C-N Math Massey, 50 / 65

51 Invertible Matrix Theorem Matrix Algebra IMT continued Let A R n n. The following are equivalent: 8. B R such that AB R = I 9. B L such that B L A = I 10. B such that AB = BA = I 11. A is invertible (non-singular) 12. A T is invertible 13. A is the product of EROMs C-N Math Massey, 51 / 65

52 Parts of the proof: Matrix Algebra (5) implies (7) since if Ax = y and Av = y then A(x v) = 0, so x v = 0 and so x = v (4) implies (8) since Ab i = e i has a soln (8) implies (4) since x = B R y solves Ax = y (3) implies (9) since E k E 1 A = rref(a) = I (9) implies (6) since if Ax = 0 then x = B L Ax = 0 (8) and (9) imply (10) since B R = B L AB R = B L (12) follows since (A T ) 1 = (A 1 ) T (13) follows since A = E 1 1 E 1 k C-N Math Massey, 52 / 65 Matrix Algebra Matrix Algebra True or False? 1. A k = (A k ) 1 = (A 1 ) k 2. (I + A) 1 = I + A 1 3. (I + A) 1 = (I + A 1 ) 1 A 1 4. (A + B) 1 B = (B 1 A + I ) 1 Solve for X by doing the same thing to both sides of the equation. 5. C (I + 2XC )C = BC 2 6. (A AX ) 1 = X 1 B C-N Math Massey, 53 / 65

53 Lucas Sequences Matrix Algebra Given p and q, define the recursive sequence: x 0 = 0, x 1 = 1, x k = px k 1 + qx k 2 Fibonacci p = q = 1 Mersenne p = 3, q = 2 Create[ an update ] matrix [ ] U such that xk xk 1 = U x k 1 x k 2 Find U 1. Write x k in terms of x k+1 and x k+2. Fact: U 2 = pu + qi Find the zenzizenzizenzic of U. Estimate lim k x k x k 1 C-N Math Massey, 54 / 65

54 Make Your Own Matrix Algebra 1. Pick p, q, r and create an update matrix for x k = px k 1 + qx k 2 + rx k 3 2. Verify that U 3 = pu 2 + qu + ri 3. Pick x 0, x 1, x 2 and compute through x Compute U 9 and use it to compute x Compute U Write a formula for x k in terms of x k+1, x k+2, and x k Estimate lim k x k x k 1 8. Name the sequence after yourself. C-N Math Massey, 55 / 65

55 Matrix Algebra Theorem Suppose A, B are square. If I AB is invertible, then so is I BA. Find the flaw in this proof. Note that B(I AB) = (I BA)B. Post-multiply both sides by (I AB) 1 B 1 : I = (I BA)B(I AB) 1 B 1. Thus (I BA) 1 = B(I AB) 1 B 1. C-N Math Massey, 56 / 65

56 Matrix Algebra Constructive proof. Let C = I + B(I AB) 1 A. Show (I BA) 1 = C by demonstrating that C (I BA) = I Proof by contradiction. Assume I BA is not invertible. Then by IMT x 0 with (I BA)x = 0. Then (I BA)x = 0 = A(I BA)x = 0 = (I AB)Ax = 0. But since I AB is invertible, Ker(I AB) = {0}, so Ax = 0. Therefore 0 = (I BA)x = x BAx = x 0, which is a contradiction. C-N Math Massey, 57 / 65

57 Functions of Matrices Matrix Algebra Let f (x ) = x 2 +2x 3 x. Then compute f (A) = (A 2 + 2A 3I )A 1 Powers of A commute with each other. (AB) 2 A 2 B 2 (A + B) 1 A 1 + B 1 C-N Math Massey, 58 / 65

58 Triangular Matrices Matrix Algebra L is lower triangular if i < j = L ij = 0 L = U is upper triangular if i > j = U ij = 0 U = Diagonal matrices are both upper and lower triangular. C-N Math Massey, 59 / 65

59 Block Matrices Matrix Algebra Matrices may be partitioned into sub-matrix blocks. A = AB = [ A11 A 12 A 21 = [ ] [ ] B1 A 22 B 2 A 11 A 12 A 21 A 22 = ], B = [ = [ ] A11 B 1 + A 12 B 2 A 21 B 1 + A 22 B 2 B 1 B 2 ] C-N Math Massey, 60 / 65

60 Block Matrices Matrix Algebra Blocks may: reveal special structure/patterns represent different components of the application be assigned to separate processors C-N Math Massey, 61 / 65

61 Block Examples Matrix Algebra [ ] 1 [ ] I 0 I 0 1. = X I X I [ ] A 0 2. Under what condition is M = is I A its own inverse? 3. [ Find the inverse of a block diagonal matrix: A11 A22] 4. [ Solve for the ] [ inverse ] of [ a block ] triangular: A11 A 12 X Y I 0 = 0 A 22 Z W 0 I C-N Math Massey, 62 / 65

62 Matrix Algebra Theorem The inverse of an upper tri. matrix exists iff i u ii 0. The inverse is upper tri. Proof outline (by induction). Assume [ ] true [ for] n [ n. Write ] UB = I n+1 as α v T β x T 1 0 = 0 W 0 Y 0 I n Then let αβ = 1, Y = W 1, and αx T + v T Y = 0. C-N Math Massey, 63 / 65

63 LU Factorization Matrix Algebra Let A = Find EROMs such that ( E k )A = U, where U is upper triangular. 2. Find lower triangular L such that A = LU. This is a LU factorization/decomposition. 3. Find EROMs such that ( E k )A = I. 4. Write A as the product of EROMs. C-N Math Massey, 64 / 65

64 Using the LU Matrix Algebra To solve Ax = LUx = b: 1. Let y = Ux and solve L(y) = b. 2. Solve Ux = y. Use the LU to solve Ax = How would you solve x T A = [5, 20, 9]? C-N Math Massey, 65 / 65

65