Linear Algebra. James Je Heon Kim

Transcription

1 Linear lgebra James Je Heon Kim (jjk9columbia.edu) If you are unfamiliar with linear or matrix algebra, you will nd that it is very di erent from basic algebra or calculus. For the duration of this session, we will be focusing on de nitions of such concepts as linear equations, matrices, determinants, vector spaces, inner products, linear transformations, eigenvalues and eigenvectors, and their applications to interesting mathematical problems. Texts: Linear lgebra and Its pplications (3rd Edition) ddison Wesley c23, by David. Lay (DL) Module Properties of Matrices System of Linear Equation DL (Recommended):..6, 22, 3.6.8, 4, 5.7., 3, 5,.8., 3, 5, 2 Module 2 Linear Independence Inverse of Matrix Linear Systems and Inverses Determinants DL(Recommended): 2..3, 5, 7,, 5, 6, 9, , 9,, 9, , 9, Eigenvalues/Eigenvectors ramer s Rule Orthogonality(?) DL(Recommended): 5..3, 4, 5, 7, , ,, , 5, , , 9, Module 3

2 Linear lgebra: Module. Module Topics:.. Linear Equation System of Linear Equations Method of Substitution Gaussian Elimination Gauss-Jordan Elimination Matrix Methods a x + a 2 x 2 + ::: + a n x n = b 4x 5x = x! rearranged! x 2 = 2( p 6 x ) + x 3! rearranged!..2 Systems of Linear Equations ollection of one or more linear equations involving ollection of one or more linear equations involving same variables say x ; :::; x n. n example is x 2x 2 + x 3 = 2x 2 8x 3 = 8 4x + 5x 2 + 9x 3 = 9 More generally, we might have a system of m equations in n unknowns a x + a 2 x 2 + ::: + a n x n = b a 2 x + a 22 x 2 + ::: + a 2n x n = b 2... a m x + a m2 x 2 + ::: + a mn x n = b m Examples for this workshop were drawn from Lay (23) Linear lgebra and Its pplications, Leon(22) Linear lgebra with pplications (6th Edition), and Harvard University s Political Science Math (P)refresher. 2

3 The set of all possible solutions is called the solution set of the linear system. Two linear systems are called equivalent if they have the same solution sets. Finding the solution set of a system of two linear equations in two variables is easy because it amounts to nding the intersection of two lines. x + x 2 = x 2x 2 = 3 x + x 2 = 3 x + x 2 = 2x 4x 2 = 8 2x 2x 2 = 6 If there are three equations in three variables, each equation would de ne a plane in a 3 dimensional space. Same reasoning applies. If there are more than 3 variables, the intersection of hyperplanes would determine the solution set...3 Strategies for Solving a System In order to solve the system of linear equations, we can utilize all or one of the following three strategies:. Substitution 2. Elimination (of variables) 3. Matrix Methods Method of Substitution Steps:. Solve one equation for one variable, say x, in terms of the other variables in the equation 2. Substitute the expression for x into the other m equations, resulting in a new system of m equations in n unknowns. 3. Repeat steps & 2 until one equation in one unknown, say x n. We now have a value for x n. 4. ackward substitution: substitute x n into previous equation(s). Repeat using the successive expressions of each variable in terms of the other variables to nd the values of all x i s. Using substitution, solve: x 2x 2 = x + 3x 2 = 3 3

4 Using substitution, solve: x 2x 2 + x 3 = 2x 2 8x 3 = 8 4x + 5x 2 + 9x 3 = 9 Method of Elimination a) Elementary Row Operations Elementary row operations are used to transform the equations of a linear system, while maintaining an equivalent linear system equivalent in the sense that the same values of xj solve both the original and transformed systems. These operations are. (Replacement) dd one row to a multiple of another row. 2. (Interchange) Interchange two rows. 3. (Scaling) Multiply all entries in a row by a nonzero constant. Take Example 2, for instance, we can: dd the st row to the 2nd row of equation to get: x 2x 2 = x 2 = 2 Then use backward substitution to get x = 3 x 2 = 2 We can apply the similar procedure to solve for Example 3 x 2x 2 + x 3 = 2x 2 8x 3 = 8 4x + 5x 2 + 9x 3 = 9 (i) To do so, add 4 times st row to the 3rd row of equation. To get a new 3rd row of equation. (ii) Now, multiply equation 2 by 2 in order to obtain as the coe cient for x 2. (This calculation will simplify the arithmetic for the next step.) 4

5 (iii) Use the x 2 in equation 2 to eliminate the 3x 2 in equation 3 in order to obtain: (iv) The new system has a triangular form: We can use backward substitution to nd the solution set. heck to see if (29, 6, 3) is a solution of the original system. b) Gaussian Elimination Method by which we start with some linear system of m equations in n unknowns and use the elementary equation operations to eliminate variables until we arrive at an equivalent system of the form: a x + a 2x 2 + a 3x 3 + ::: + a nx n = b a 22x 2 + a 23x 3 + ::: + a 2nx n = b 2 a 33x 3 + ::: + a 3nx n = b 3 : : : : : : a mnx n = b m The bold faced coe cients are referred to as pivots. (Essentially similar to the method shown above) c) Gauss-Jordan Elimination 5

6 Takes the Gaussian elimination method one step further. Once the linear system is in the reduced form shown in the preceding section, elementary row operations and Gaussian eliminations are used to and (i) hange the coe cient of the pivot term in each equation to (ii) Eliminate all terms above each pivot in its column, resulting in a reduced, equivalent system. For a system of m equations in n unknowns, a typical reduced system would be: x = b x 2 = b 2 x 3 = b 3. :. :. : x n = b m d) Matrix Method Matrices provide an easy and e cient way to represent linear systems such as: a x + a 2 x 2 + ::: + a n x n = b a 2 x + a 22 x 2 + ::: + a 2n x n = b 2... a m x + a m2 x 2 + ::: + a mn x n = b m The m n coe cient matrix is an array of mn real numbers arranged in m rows by n columns. = a a 2 ::: a n a 2 : a 2n : : : a m a m2 ::: a mn The RHS of the linear system is represented by the vector b = b b 2 : ugmented matrix: when we append b to the coe cient matrix ; we get the augmented matrix ^ = [jb] b m 6

7 ^ = a a 2 ::: a n a 2 : a 2n : : : a m a m2 ::: a mn b b 2 : b m The rules of elementary row operations apply here. Gaussian elimination of the augmented matrix results in a row echelon form. a a 2 ::: a n b a 22 a 23 a 2n b 2 : : : a mn b m Note: (i) ll nonzero rows are above any rows of all zeros. (ii) Each leading entry of a row is in a column to the right of the leading entry of the row above it. (iii) ll entries in a column below a leading entry are zeros. Reduced row echelon form Satis es the following additional conditions (iv) The leading entry in each nonzero row is. (v) Each leading is the only nonzero entry in its column. a b a 22 b 2 : ::: : a mn b m In essence, this is the matrix equivalent of a linear system after Gauss-Jordan elimination. Exercise Solve the system using matrix method: x + x 2 = x x 2 = 3 x + 2x 2 = 2 7

8 Exercise 2 Solve the system using matrix method x + x 2 x 3 + 3x 4 = 3x + x 2 x 3 x 4 = 2x x 2 x 3 x 4 = Exercise 3 In the downtown section of a certain city, two sets of one way streets intersect as shown below. The average hourly volume tra c entering and leaving this section during rush hour is given in the diagram. Determine the amount of tra c between each of the four intersections. Rank, Uniqueness and onsistency s the exercises suggest, there are two fundamental questions that we may want to address when we are solving for a system of linear equations ) Is the system consistent? (Does a solution exist?) 2) If a solution exists, is it unique? (Is there one and only one solution?) We can determine whether one in nite, or no solutions exist if we know () the number of equations m, (2) the number of unknowns n, and (3) the rank of the matrix representing the linear system. Rank: number of nonzero rows in its row echelon form. 8

9 Example 4 Example 5 Example Rank = b 4 5 b 2 b 3 b 4 Rank = 2 ; b j 6= Rank = 4 Let be the coe cient matrix and ^ = [jb] be the augmented matrix. Then,. rank rank  ugmenting with b can never result in more zero rows than originally in itself. Suppose row i in is all zeros and that b i is non-zero. ugmenting with b will yield a non-zero row i in ^: 2. rank rows y de nition of a rank.3 3. rank columns Suppose there are more rows than columns (otherwise the previous rule applies). Each column can contain at most one pivot. y pivoting, all other entries in a column below the pivot are zeroed. Hence, there will only be as many non-zero rows as pivots, which will equal the number of columns. Existence of solutions:. Exactly one solution rank = rank  = rows = columns Necessary condition for a system to have a unique solution: that there be exactly as many equations as unknowns. 2. In nite solutions rank = rank; and columns > rank If a system has a solution and has more unknowns than equations, then it has in nitely many solutions. 9

10 3. No solution rank < rank  Then there is a zero-row i in s reduced echelon that corresponds to a non-zero row i in ^ s reduced echelon. Row i of the translates to the equation: x i + x i2 + ::: + x in = b i Where b i 6= : Hence, the system has no solution. Exercise 7 x + 2x 2 + x 3 = 2x + 4x 2 + 2x 3 = 3 Exercise 8 x + x 2 + x 3 + x 4 + x 5 = 2 x + x 2 + x 3 + 2x 4 + 2x 5 = 3 x + x 2 + x 3 + 2x 4 + 3x 5 = 2 2 Linear lgebra: Module 2 2. Module Topics: 2 Linear Independence Matrix lgebra Inverse Determinants ramer s Rule Eigenvalues & Eigenvectors 2.. Linear Independence Let v ; v 2 ; ::::; v n be a set of n vectors each of which is of order m. Then the set of vectors is linearly dependent if there exist scalars a ; a 2 ; :::; a n at least one of which is not such that Or a v + a 2 v 2 + a 3 v 3 + ::: + a n v n = 2 Examples for this workshop were drawn from Lay (23) Linear lgebra and Its pplications, Leon(22) Linear lgebra with pplications (6th Edition), and Harvard University s Political Science Math (P)refresher.

11 V a = Example 9 Is there linear dependence? v = 2 3 v 2 = v 3 = 2 Hint: We can try to augment these vectors into a matrix and solve for a reduced row echelon form and see if we can nd some relationship between the rows. If we can determine some functional linear relationship between these vectors, we can say that there is linear dependence Matrix lgebra Matrix: matrix is an array of mn real numbers arranged in m rows by n column. a a 2 ::: a n = a 2 a 22 a 2n : : a m a m2 ::: a mn Matrix addition (or subtraction): Let and be two m n matrices. Then a + b a 2 + b 2 ::: a n + b n + = a 2 + b 2 a 22 + b 22 ::: a 2n + b 2n : a m + b m a m2 + b m2 :::: a mn + b mn Note: and must be the same size! Example = Example = Example = is Scalar Multiplication: Given the scalar s, the scalar multiplication of s

12 s = s a a 2 ::: a n a 2 a 22 a 2n : : a m a m2 ::: a mn = sa sa 2 ::: sa n sa 2 sa 22 sa 2n : : sa m sa m2 ::: sa mn Matrix multiplication: If is an m n matrix and is a k n matrix, then their product = is an m n matrix where So If = c ij = a i b j + a i2 b 2j + ::: + a ik b kj 2 3 = and = 4 5 then, =? 3 6 Note: the number of columns of the rst matrix must equal the number of rows of the second matrix. The size of the matrices (including the resulting product) must be: (m k)(k n) = (m n) Using matrix multiplication, we can now write the linear equation. a a 2 ::: a n Let be a m n coe cient matrix a 2 a 22 a 2n : : a m a m2 ::: a mn x be a m vector array of unknowns x x 2 : x m b be a m vector array of constants b b 2 : b m Then, 2

13 x = b a x + a 2 x 2 + ::: + a n x m = b a 2 x + a 22 x 2 + ::: + a 2n x m = b 2 : : a m x + a m2 x 2 + ::: + a mn x m = b m 2..3 Law of Matrix lgebra. ssociative ( + ) + = + ( + ) () = () 2. ommutative + = + Question: Is multiplication of matrices commutative? 6= or =?an you think of an example that can show whether one or the other? 3. Distributive ( + ) = + ( + ) = + 2 Exercise 3 If = 3 4 Solve a) () b) () c) ( + ) d) + If = and = then, 2 =? 3 =? n =? and = 2. Transpose Transpose of the m n matrix is the n m matrix T interchanging the rows and columns of. obtained by The following rules apply for transposed matrices a) ( + ) T = T + T b) ( T ) T = c).(s) T = s T 3

14 Exercise 4 = for T T and () T = Solve d) () T = T T 2..4 The Inverse of a Matrix Identity Matrix: The n n identity matrix I n is the matrix whose diagonal elements are and all o -diagonal elements are. So I n = a a 2 ::: a n a 2 a 22 a 2n : : a m a m2 ::: a mn where all diagonal a = a 22 = a 33 = ::: = a mn = and all other o -diagonal elements are. Inverse Matrix: n n matrix is nonsingular or invertible if there exists an n n matrix such that = = I n is the inverse of. If there is no such ; then is singular or noninvertible. Invertible: a Let = c b d (2:) If ad bc 6=, then is invertible (or nonsingular) d b and = ad bc c a Properties of Inverse. If the inverse exists, it is unique 2. nonsingular! nonsingular ( ) = 3. and nonsingular! nonsingular () = 4

15 4. nonsingular! ( T ) = ( ) T Exercise 5 Find if = 2 = Determinants If = (a) is a matrix, then will have a multiplicative inverse if and only if a 6= : Thus, if we de ne, det() = a then will be nonsingular if and only if det() 6= : Hence, the determinant will equal zero when the inverse does not exist. Since the inverse of a, a, does not exist when a =, we let the determinant of a be a: 2 2 Matrix a a Let = 2 a 2 a 22 In order to nd its determinant,. Multiply the second row of by a 2. Subtract a 2 times the rst row from the new second row There are two alternatives to this method: a) Take the rst upper left hand corner element and eliminate its corresponding row and column elements; then multiply with the remaining element(s). Do the same for all other rst row elements alternating subtraction and addition. b) Simply multiply the diagonal elemtns together and subtract the product of the upper left hand element in the matrix to that of the other. The determinant is of the form: a a 22 a 2 a 2 6= We say that is nonsingular only if a a 22 determinant of a 2 2 matrix as a 2 a 2 6= : We then de ne the 5

16 a a 2 a 2 a 22 = a a 22 a 2 a 2 = a ja 22 j a 2 ja 2 j a a 2 a 3 a 2 a 22 a 23 a 3 a 32 a 33 Extending this to a 3 3 matrix we get = a a 22 a 23 a a 32 a 33 a 2 a 23 2 a 3 a 33 + a 3 a 2 a 22 a 3 a 32 For n 2, the determinant of an n n matrix = [a ij ] is given by nx det = a det a 2 det 2 +:::+( ) +n a n det n = ( ) +j a j det j For a triangular or diagonal matrices, the determinant is just the product of the diagonal terms. Example: R = r r 2 r 3 r 22 r 23 r 33 j= Then jrj = r r 22 r 23 r 33 = r r 22 r 33 Exercise 6 Exercise Shortcut for a 3 3 matrix: a a 2 a 3 a 2 a 22 a 23 a 3 a 32 a 33 = (a a 22 a 33 + a 2 a 23 a 3 + a 2 a 32 a 3 ) (a 3 a 22 a 3 + a 2 a 2 a 33 + a 23 a 32 a ) 6

17 Properties of determinants. jj = j T j 2. If results from by interchanging two rows, thenjj = j j 3. If two rows of are equal, then jj = 4. If a row of consists of all zeros, then jj = 5. If is obtained by multiplying a row of by a scalar s, then jj = jsj 6. If is obtained from by adding to the ith row of the jth row (i 6= j) multiplied by a scalar s, then jj = jj 7. If no row interchanges and no scalar multplicatinos of a single row are used to compute the row echelon form R from the n n coe cient matrix, then jj = jrj 8. square matrix is nonsingular if and only if its determinent is not zero 9. jj = jj jj. If is nonsingular, then jj 6= and j j = jj Knowing the determinant is useful. For instance, we can calculate the matrix inverse if we know its determinant: = jj adj() entry is ji: where adj() [adjoint of ] is a nn matrix whose (i,j)th So, if we see equation 2., adj = a22 a 2 a 2 a For a 3 3 matrix, the adjoint is the transpose of the cofactors 2 2 If = adj() = 2 3 Using above de nition of an inverse, calculate = 7

18 2..5 ramer s Rule Let be a n n nonsingular matrix and let b 2 R n : Let i be the matrix obtained by replacing the ith column of by b. If x is the unique aolution to x = b; then x i = det(i) det() for i = ; 2; :::; n Use ramer s Rule to solve x + 2x 2 + x 3 = 5 2x + 2x 2 + x 3 = 6 x + 2x 2 + 3x 3 = 9 Note: ramer s rule gives us a convenient method for writing the solution to a n n system of linear equations in terms of determinants. However, we must evaluate n+ determinents of order n if we wish to use ramer s Rule. Evaluating even two of these determinants generally involves more computation than solving the system using Gaussian elimination technique Eigenvalues & Eigenvectors onsider the following: Given a k k matrix, can we nd a scalar and a nonzero vector x such that x = x? The answer to this question is a "yes". In fact, there exists k (possibly nondistinct) values for and in ntely many vectors x satisfying this condition. This is known as the eigenvalue/eigenvector problem. 8

19 How do we calculate them? 2..7 Some useful application of linear algebra: Linear Regression lthough linear algebra may seem impractical for rst time users, it is a very useful tool for understanding the theoretical basis of linear regression. To illustrate, if we have a collection of data where the dependent variable is y and explanatory variables are series of X s, we can represent them using matrix notations as: y = y y 2 : y n X = x x 2 ::: x k x 2 x 22 : : : : x n x n2 ::: x nk If we wish to represent the relationship between these two variables y and X using linear regression, they are: y = X + " where represents the coreponding coe cient vector based on the data y and X, and " is the random error term. The most popular method of estimating is by using the least squares method where we seek to minimize the sum of the squared elements of ": That is, we seek to minimize nx " 2 i = "T " = (y X) T (y X) i= 9

20 One way to do this is to nd the derivative of the above equation with respect to and then set it equal to zero and solve for : This yields: X T X = X T y If we minimize this term, we can obtain the coe cient estimate ^ ^ = (X T X) X T y So we can actually try this with real data to see if it works. Let y be the price in thousands of dollars Let x be the lot size (in hundreds of square yards). Let x 2 be the number of bedrooms y = 22 5 X = Find the linear relationship between y and X. 2