Copyright 2005, W.R. Winfrey
Topics Preliminaries Systems of Linear Equations Matrices Algebraic Properties of Matrix Operations Special Types of Matrices and Partitioned Matrices Matrix Transformations
Topics Preliminaries Systems of Linear Equations Matrices Algebraic Properties of Matrix Operations Special Types of Matrices and Partitioned Matrices Matrix Transformations
Preliminaries Focus on two major problems Solution of systems of linear equations Eigenvalue problem Solution of these problems will introduce many new ideas, such as vector spaces, inner products and linear transformations, that are useful independently of these two problems Focus on linear problems because They occur naturally in many applications They occur as first approximations to many nonlinear problems
Preliminaries Three Examples Example #1 2x y 2z 10 3x y 2z 1 5x 4y 3z 4 Add first equation to second 2x y 2z 10 5x 11 5x 4y 3z 4 x 11 5, y 14 55, z 147 55
Preliminaries Three Examples Example #2 2x y 2z 10 3x y 2z 1 x 2y 4z 8 This system has no solution
Preliminaries Three Examples Example #3 2x y 2z 10 3x y 2z 1 x 2y 4z 9 This system has infinitely many solutions x 11 5, y 2z 28 5
Preliminaries Three Examples Questions How do we generalize the notation to handle large numbers of variables? How do we know that the procedure suggested in Example #1 will always work? Is the solution in Example #1 unique? How can we reach the conclusions in Example #2 and Example #3 in a systematic way that can become an algorithm for a computer program? Is there hidden structure in the problem that can give us additional insight?
Topics Preliminaries Systems of Linear Equations Matrices Algebraic Properties of Matrix Operations Special Types of Matrices and Partitioned Matrices Matrix Transformations
Systems of Linear Equations System of m equations in n unknowns a 11 x 1 a 12 x 2 a 1n x n b 1 a 21 x 1 a 22 x 2 a 2n x n b 2 a m1 x 1 a m2 x 2 a mn x n b m
Systems of Linear Equations Comments If a system has a solution, call it consistent If a system doesn t have a solution, call it inconsistent If b 1 b 2 b m 0, the system is called homogeneous. A homogeneous system always has the trivial solution x 1 x 2 x n 0 If two systems have the same solution, then they are called equivalent. The solution strategy for linear systems is to transform the system through a series of equivalent systems until the solution is obvious
Systems of Linear Equations Elementary Operations on Systems 1) Switch two equations 2) Multiply an equation by nonzero constant 3) Add multiple of one equation to another The application of any combination of elementary operations to a linear system yields a new linear system that is equivalent to the first
Systems of Linear Equations Third Elementary Operation a i1 x 1 a i2 x 2 a in x n b i a j1 x 1 a j2 x 2 a jn x n b j Add a times ith row to jth row to get aa a x aa a x aa a x ab b i1 j1 1 i2 j2 2 in jn n i j Let s 1, s 2,, s n be a solution of the original system a i1 s 1 a i2 s 2 a in s n b i a j1 s 1 a j2 s 2 a jn s n b j aa a s aa a s aa a s ab b i1 j1 1 i2 j2 2 in jn n i j
Systems of Linear Equations Elementary Operations 1) Switch two equations 2) Multiply an equation by non zero constant 3) Add multiple of one equation to another
Topics Preliminaries Systems of Linear Equations Matrices Algebraic Properties of Matrix Operations Special Types of Matrices and Partitioned Matrices Matrix Transformations
Matrices Introduce a new notation which aids the solution of systems of linear equations and gives insight into the solution process and into the structure of linear systems
Matrices Recall earlier example 2x 1 x 2 2x 3 10 3x 1 x 2 2x 3 1 5x 1 4x 2 3x 3 4 In the solution process, the x variables are just placeholders. The real work is done on the coefficients and the right-hand sides
Matrices Express system on previous slide as 2 1 2 x 1 10 3 1 2 x 1 5 4 3 x 4 2 3 3x3 matrix 3x1 matrix 3x1 matrix Original system can recovered by multiplying column of x variables by rows of 3x3 Operations used to solve system can be performed simultaneously on rows of 3x3 and entries in 3x1
Matrices General Case System of m equations in n unknowns a 11 x 1 a 12 x 2 a 1n x n b 1 a 21 x 1 a 22 x 2 a 2n x n b 2 a m1 x 1 a m2 x 2 a mn x n b m Define the mxn matrix A a a a a a a A 11 12 1n 21 22 2n a a a m1 m2 mn
Matrices General Case The mxn system can be written as or AX B a a a x b a a a x b 11 12 1n 1 1 21 22 2n 2 2 a a a x b m1 m2 mn n m Recall that if b 1 b 2 b m 0, the system is called homogenous and may be written as AX 0
General Case i th row of A j th column of A Matrices a i1 a i2 a in 1 i m a a a 1 j 2 j mj 1 jn Will sometimes write A [ a ij ] Will sometimes write (A) ij for a ij
Matrices Comments a) If m n then A is called a square matrix b) For a square matrix, the elements a 11, a 22,, a nn constitute the main diagonal of A c) Two matrices, A [ a ij ] and B [ b ij ], are equal if a ij b ij for 1 i m, 1 j n
Matrices Comment In terms of matrices, the two fundamental problems in linear algebra are 1) Linear systems - solve AX = B 2) Eigenvalues & eigenvectors - solve AX = lx
Matrices Basic Operations on Matrices a) Addition b) Scalar multiplication c) Matrix multiplication
Matrices Addition Adding matrices means adding corresponding elements, e.g. 1 2 3 3 2 1 4 4 4 4 5 6 6 5 4 10 10 10 In general, let A [ a ij ] and B [ b ij ] be two mxn matrices, then C A + B is a matrix C [ c ij ] such that c ij a ij + b ij i,j Note: sizes of matrices must be the same 1 2 1 2 3 3 4 4 5 6 undefined
Matrices Scalar Multiplication Scalar multiplication means multiplying each element of a matrix by the same scalar, e.g. 1 2 2 4 2 3 4 6 8 Let A [ a ij ] and r R. Then C r A, where C [ c ij ] is defined as c ij r a ij i,j
Matrices Matrix Multiplication Simple Example. Will want to express a11x1 a12x2 b a11 a 1 12 x1 b as 1 a21x a 1 a22x2 b2 21 a22 x2 b 2 so, need to have matrix multiplication to work like e.g. a a x a x a x a a x a x a x 11 12 1 11 1 12 2 21 22 2 21 1 22 2 1 2 5 6 15 27 16 28 19 22 3 4 7 8 35 47 36 48 43 50
Matrices Matrix Multiplication Summation n i 1 2 n i i 1 1 2 2 n n i1 i1 Properties 1) 2) 3) n d d d d ra r a r a r a n n n r s a ra s a i1 i i i i1 i i i1 i i n n cd c d i1 i i1 n m a ij i1 j1 i m j1 n a ij i1
Matrices Matrix Multiplication Recall If A is mxn and X is nx1, then 2 1 1 11 12 1 11 1 12 2 1 2 21 22 2 21 1 22 2 2 2 1 k k k k k k a x a a x a x a x b a a x a x a x b a x 1 1 2 1 1 n k k k n k k k n mk k k a x a x a x AX Row of A times the column of X
Matrices Matrix Multiplication If A is mxn and X is nx1, can also express AX as a a a AX x a x a x a 11 12 1n a a a mn 21 22 2n 1 2 n m1 m2 Will call this a linear combination of the columns of A. The coefficients are the elements of X
Matrices Matrix Multiplication a11 a12 a b b b 1n What about a a a n b b b 11 12 1p 21 22 2 21 22 2 p a a a b b b m1 m2 mn n1 n2 np A B The basic idea is to multiply each column of B by A b11 b b 12 1p b21 b22 b 2 p AB A A A b b b n1 n2 np?
Matrices Matrix Multiplication Examine first column of product A b b b 11 21 k1 n k1 1k k1 2k k1 n1 n n k1 a b a b a b mk k1 1st row A x 1 st column B 2nd row A x 1 st column B mth row A x 1 st column B
Matrices Matrix Multiplication Defn. Let A [ a ij ] be an mxn matrix and let B [ b ij ] be an nxp matrix. The product of A and B, AB C [ c ij ], is the mxp matrix defined by c ij n a ik b kj a i1 b 1 j a i2 b 2 j a in b nj k1 for i 1,2,,m j 1,2,, p
Matrices Matrix Multiplication Comments BA is defined only if p=m If p=m so that BA is defined, then BA is nxn and AB is mxm, i.e. different sizes If AB and BA are the same size, they may not be equal, i.e. they may not commute
Matrices Systems of Equations Consider a 11 x 1 a 12 x 2 a 1n x n b 1 a 21 x 1 a 22 x 2 a 2n x n b 2 a m1 x 1 a m2 x 2 a mn x n b m Define a a a x b a a a x b A X B 11 12 1n 1 1 21 22 2n 2 2 a m1 am2 amn x n b m Express system as AX B
Matrices Systems of Equations Since the solution of the system involves the a and b values only, will often work with the augmented matrix a a a b a a a b 11 12 1n 1 21 22 2n 2 a a a b m1 m2 mn m
Matrices Transpose Defn. Let A [ a ij ] be an mxn matrix. The transpose of A, A T [ a ijt ] is the nxm matrix defined by a ij T a ji A 1 2 T 1 3 A 3 4 2 4
Topics Preliminaries Systems of Linear Equations Matrices Algebraic Properties of Matrix Operations Special Types of Matrices and Partitioned Matrices Matrix Transformations
Algebraic Properties of Matrix Operations Consider algebraic properties of 1) Matrix addition 2) Matrix multiplication 3) Scalar multiplication 4) Transpose Purposes 1) Establish relationships between matrices and known algebraic systems 2) Provide rules for manipulating matrices
Algebraic Properties of Matrix Operations Theorem (Matrix Addition) Let A, B and C be mxn matrices Closure Associativity Identity Inverse Commutivity 1) A + B is an mxn matrix 2) A + (B + C) = (A + B) + C 3) There is a unique mxn matrix m 0 n such that A + m 0 n = m 0 n + A = A for every matrix A 4) For every mxn matrix A, there is a unique mxn matrix D such that A + D = D + A = m 0 n 5) A + B = B + A
Algebraic Properties of Matrix Operations Theorem (Matrix Multiplication) - a) If A, B and C are matrices of the appropriate sizes, then A(BC) (AB)C b) If A, B and C are matrices of the appropriate sizes, then (A + B)C AC + BC c) If A, B and C are matrices of the appropriate sizes, then C(A + B) CA + CB
Algebraic Properties of Matrix Operations Proof a) Let A [ a ij ] be mxn, B [ b ij ] be nxp and C [ c ij ] be pxq BC kj A BC AB it AB C p t1 ij n k1 ij bc kt n tj p a b c a b c ik kt tj ik kt k1 t1 k1 t1 ab ik kt n p n p n n p a b c a b c a b c p ik kt tj ik kt tj ik kt tj t1 k1 t1 k1 k1 t1 tj
Algebraic Properties of Matrix Operations Proof (continued) b) Let A [ a ij ] be mxn, B [ b ij ] be mxn and C [ c ij ] be nxp A B a b ik A BC c) Proof similar to b) ik n n n n aik bik ckj aikckj bikckj k1 k1 k1 ik kj ik kj k1 k1 n a c b c AC BC QED
Algebraic Properties of Matrix Operations Comments on Matrix Multiplication AB need not equal BA Let 1 2 4 6 0 0 A B AB 2 4 2 3 0 0 So, we can have AB 0, but A 0 and B 0 1 2 2 1 2 7 Let A, B, C 2 4 3 2 5 1 8 5 then AB AC but B C, i.e. can't cancel 16 10
Algebraic Properties of Matrix Operations Properties of Scalar Multiplication If r and s are real numbers and A and B are matrices, then r (sa) (rs) A (r + s) A ra + sa r (A + B) ra + rb A (rb) r (AB) (ra) B
Algebraic Properties of Matrix Operations Properties of Transpose If r is a scalar and A and B are matrices, then (A T ) T A (A + B) T A T + B T (AB) T B T A T (ra) T ra T
Topics Preliminaries Systems of Linear Equations Matrices Algebraic Properties of Matrix Operations Special Types of Matrices and Partitioned Matrices Matrix Transformations
Special Matrices and Partitioned Matrices Defn - An nxn matrix A [ a ij ] is called a diagonal matrix if a ij 0 for i j, i.e. the terms off the main diagonal are all zero Defn - A scalar matrix is a diagonal matrix whose diagonal elements are all equal Defn - The scalar matrix I n [ a ij ], where a ii 1 and a ij 0 for i j is called the nxn identity matrix. The name comes from the following property. Let A be any mxn matrix, then A I n A and I m A A
Special Matrices and Partitioned Matrices Matrix Powers Recall that matrix multiplication is associative, i.e. if A, B and C have the proper dimensions, then A(BC) (AB)C, so the parentheses are unnecessary and the product can be written as ABC If A is an nxn matrix and p is a positive integer, can define p A AA A factors Again, if A is an nxn matrix, adopt the convention p A 0 I n
Special Matrices and Partitioned Matrices Matrix Powers The following laws of exponents hold for nonnegative integers p and q and any nxn matrix A 1 ) A p A q A p + q 2 ) (A p ) q A pq Caution. Without additional assumptions on A and B, cannot do the following 1 ) define A p for negative integers p 2) assert that (AB) p A p B p
Special Matrices and Partitioned Matrices Triangular Matrices An nxn matrix A [ a ij ] is called upper triangular if a ij 0 for i > j An nxn matrix A [ a ij ] is called lower triangular if a ij 0 for i < j Note: A diagonal matrix is both upper and lower triangular The nxn zero matrix is both upper and lower triangular
Special Matrices and Partitioned Matrices Symmetry Defn - A matrix A is called symmetric if A T A Defn - A matrix A is called skew-symmetric if A T A Comment - If A is skew-symmetric, then the diagonal elements of A are zero Comment - Any square matrix A can be written as the sum of a symmetric matrix and a skewsymmetric matrix T T A 1 A A 1 A A 2 2 symmetric skew-symmetric
Special Matrices and Partitioned Matrices Symmetry - Example A symmetric matrix is often indicative of some symmetry in the original problem being analyzed Consider a mass m suspended from a spring of spring constant k. Let x be the displacement of the mass from equilibrium. Hooke s Law is F kx. Combining Hooke s Law with Newton s Second Law gives m d 2 x dt 2 kx k k xt acos t sin t m m m k
Special Matrices and Partitioned Matrices Symmetry - Example Consider a system of springs and masses. Let x 1, x 2 and x 3 be the deviations of the masses from their equilibrium positions (at equilibrium, x 1 0, x 2 0 and x 3 0) k 01 k 12 k 23 k 34 m 1 m 2 m 3
Special Matrices and Partitioned Matrices Symmetry - Example Can argue that the equations of motion for the masses may be written as m1 0 0 x1 k01 k12 k12 0 x1 0 0 m2 0 x2 k12 k12 k23 k23 x2 0 0 0 m3 x3 0 k23 k23 k34 x3 0 Comments Note that the mass matrix is symmetric Note that the spring matrix is symmetric. This reflects the fact that a spring is a symmetric device Determination of vibrational frequencies will have to be deferred until the eigenvalue problem is discussed Symmetric matrices also appear in the analysis of networks of resistors and voltage sources
Special Matrices and Partitioned Matrices Partitioning of Matrices Defn - Let A [ a ij ] be an mxn matrix. A submatrix of A is obtained by deleting some, but not all, of the rows and columns of A Example - Let 1 2 3 4 A 2 4 3 5 3 0 5 3 some submatrices of A are 1 2 4 3 5,, 2 3 0 3 5 3
Special Matrices and Partitioned Matrices Partitioning of Matrices Primary interest is in submatrices obtained by partitioning, i.e. by drawing horizontal and vertical lines between rows and columns of a matrix. Consider A a a a a a a a a a a a a a a a a a a a a 11 12 13 14 15 21 22 23 24 25 31 32 33 34 35 41 42 43 44 45
Special Matrices and Partitioned Matrices Partitioning of Matrices A can be written asa A A A A 11 12 21 22 where A A a a a a a A 11 12 13 14 15 11 a 12 21 a22 a23 a24 a25 a a a a a A 31 32 33 34 35 21 a 22 41 a42 a43 a44 a45
Special Matrices and Partitioned Matrices Partitioning of Matrices A could also be partitioned as a11 a12 a13 a14 a15 a a a a a A A A A a a a a a A A A a a a a a 21 22 23 24 25 11 12 13 31 32 33 34 35 21 22 23 41 42 43 44 45 (Note: Definitions of A ij have changed from previous slide)
Special Matrices and Partitioned Matrices Partitioning of Matrices Define the matrix B as b11 b12 b13 b 14 b21 b22 b23 b24 B11 B 12 B b31 b32 b33 b 34 B21 B 22 b41 b42 b43 b44 B31 B32 b51 b52 b53 b 54 Direct computation shows that the product AB may be written as A B A B A B A B A B A B AB A B A B A B A B A B A B 11 11 12 21 13 31 11 12 12 22 13 32 21 11 22 21 23 31 21 12 22 22 23 32 So, the product can be done in pieces
Special Matrices and Partitioned Matrices Partitioning of Matrices Multiplication by partitioning works only if A ( mxn ) and B ( nxp ) are partitioned compatibly (conformal partitioning) A n 1 m 1 m 2 m q n 2 n h A A A A A A 11 12 1h 21 22 2h A A A q1 q2 qh where A ij is m i x n j and q h m m n n i i1 j1 j B p 1 p 2 p h B B B B B B 11 12 1k 21 22 2k B B B h1 h2 hk where B ij is n i x p j and h n n p p i i1 j1 k j n 1 n 2 n h
Special Matrices and Partitioned Matrices Partitioning of Matrices - Example Consider powers of a 3n x 3n matrix A, which has the partitioned form P In 0 A 0 P In 0 0 P where P is an nxn matrix, I n is the nxn identity matrix and 0 is the nxn zero matrix
Special Matrices and Partitioned Matrices Partitioning of Matrices - Example Then 2 P 2P I 2 2 A 0 P 2P 2 0 0 P s s P P P 1 2 s s1 s2 s A 0 P P 1 s s s1 0 0 P s 3 2 P 3P 3P A 0 P 3P 0 0 P 3 3 2 3 s 2,3,4, Note: Easy to do if the matrix is in partitioned form. Hard to see pattern otherwise.
Special Matrices and Partitioned Matrices Nonsingular Matrices Defn - An nxn matrix A is called nonsingular or invertible if there exists an nxn matrix B such that AB BA I n Comments If B exists, then B is called the inverse of A If B does not exist, then A is called singular or noninvertible At this point, the only available tool for showing that A is nonsingular is to show that B exists
Special Matrices and Partitioned Matrices Nonsingular Matrices Theorem - If the inverse of a matrix exists, then that inverse is unique Proof - Let A be a nonsingular nxn matrix and let B and C be inverses of A. Then AB BA I n and AC CA I n B B I n B(AC) (BA)C I n C C so the inverse is unique. QED Notation - If A is a nonsingular matrix. The inverse of A is denoted by A 1 Comment - For nonsingular matrices, A, can define A raised to a negative power as A k (A 1 ) k k > 0
Special Matrices and Partitioned Matrices Nonsingular Matrices Theorem - If A and B are both nonsingular matrices, then the product AB is nonsingular and (AB) 1 B 1 A 1 Proof - Consider the following products AB ( B 1 A 1 ) AB B 1 A 1 AI n A 1 AA 1 I n and ( B 1 A 1 ) AB B 1 A 1 AB B 1 I n B B 1 B I n Since we have found a matrix C such that C(AB) (AB)C I n, AB is nonsingular and its inverse is C B 1 A 1 QED
Special Matrices and Partitioned Matrices Nonsingular Matrices Theorem - If A 1, A 2,, A r are nonsingular matrices, then A 1 A 2 A r is nonsingular and A 1 A 2 A r 1 A 1 1 r A r1 A 2 1 A 1 1
Special Matrices and Partitioned Matrices Nonsingular Matrices Theorem - If A is a nonsingular matrix, then A 1 is nonsingular and (A 1 ) 1 A Proof - Since A 1 A AA 1 I n, then A 1 is nonsingular and its inverse is A. So (A 1 ) 1 A QED Comment - Cannot prove this using laws of exponents for matrix powers since this theorem is needed to prove those laws Comment - The previous four theorems involving inverses are essentially algebraic, i.e. the details of the matrix product are not used
Special Matrices and Partitioned Matrices Nonsingular Matrices Comment - Have observed earlier that AB AC does not necessarily imply that B C. However, if A is an nxn nonsingular matrix and AB AC, then B C. A 1 (AB) A 1 (AC) (A 1 A) B (A 1 A)C B C Comment - Have observed earlier that AB n 0 n does not imply that A n 0 n or B n 0 n. However, if A is an nxn nonsingular matrix and AB n 0 n, then B n 0 n. A 1 (AB) A 1 n0 n (A 1 A) B n 0 n B n 0 n
Special Matrices and Partitioned Matrices Nonsingular Matrices Theorem - If A is a nonsingular matrix, then A T is nonsingular and (A T ) 1 (A 1 ) T Proof - By an earlier theorem, (AB) T B T A T for any two matrices A and B. Since A is nonsingular, A 1 A AA 1 I n. Applying the relationship on transposes gives (A 1 A) T A T (A 1 ) T I nt I n (AA 1 ) T (A 1 ) T A T I nt I n Since A T (A 1 ) T I n and (A 1 ) T A T I n, A T is nonsingular and its inverse is (A 1 ) T, i.e. (A T ) 1 (A 1 ) T QED
Special Matrices and Partitioned Matrices Linear Systems and Inverses A system of n linear equations in n unknowns may be written as AX = B, where A is nxn matrix. If A is nonsingular, then A 1 exists and the system may be solved by multiplying both sides by A 1 A 1 (AX) = A 1 B (A 1 A)X = A 1 B X = A 1 B
Special Matrices and Partitioned Matrices Linear Systems and Inverses Comment - Although X = A 1 B gives a simple expression for the solution, its primary usage is for proofs and derivations At this point we have no practical tool for computing A 1 Even with a tool for computing A 1, this method of solution is usually numerically inefficient. The only exception is if A has a special structure that lets A 1 have a simple relationship to A
Topics Preliminaries Systems of Linear Equations Matrices Algebraic Properties of Matrix Operations Special Types of Matrices and Partitioned Matrices Matrix Transformations
Matrix Transformations Let A [ a ij ] be an mxn matrix. Let R m be the set of all mx1 matrices and let R n be set of all nx1 matrices. If X R m and Y = AX. Then Y R n and A can be interpreted as a function (or mapping or transformation) from R m into R n R m is the domain of the function The range or image of A is the set of all Y R n such that Y = AX for some X R m Will focus primarily on R 2 and R 3 and will interpret corresponding 2x1 and 3x1 matrices as two-dimensional and three-dimensional points
Matrix Transformations Example - Reflections Reflections in the x-axis A 1 0 0 1 Reflections in the y-axis A 1 0 0 1
Matrix Transformations Example - Reflections Reflections in the origin A 1 0 0 1 x,y y x, y x
Matrix Transformations Example - Projection onto xy Plane A 1 0 0 0 1 0 0 0 0 Can also view this as a mapping from R 3 to R 2 using the matrix 1 0 0 A 0 1 0
Matrix Transformations Example - Scaling A r 0 0 0 r 0 0 0 r If r > 1, A is called a dilation If 0 < r < 1, A is called contraction
Matrix Transformations Example - Rotation Let A cos sin sin cos x y x y cos sin sin cos A x y