POLI 7 - Mathematical and Statistical Foundations Prof S Saiegh Fall Lecture Notes - Class 4 October 4, Linear Algebra The analysis of many models in the social sciences reduces to the study of systems of equations Moreover, some of the most frequently used models are linear models Today, we will cover the simplest possible systems of equations linear systems We will also examine some of the techniques for solving such systems In particular, we will rely on Matrix Algebra, and important tool in mathematical social science Linear Systems Consider the following two equations: 3x + x 7 x + x 3 These are typical linear equations straight lines They are called linear because their graphs are In general, an equation is linear if it has the form: a x + a x + + a n x n b The letters a,, a n and b stand for fixed numbers, such as 3 and 7 in the first equation These numbers are called parameters The letters x,, x n stand for variables The key feature of the general form of a linear equation is that each term of the equation contains at most one variable, and that variable appears only to the first power rather than to the second, third, or some other power
Linear equations are the most elementary ones that can arise In addition, the typically describe geometric objects such as lines or planes Linear systems have the added advantage that we can often calculate exact solutions to the equations Linearity, however, is usually a simplifying assumption: the real world is nonlinear While many models have a natural linear structure, many other ones are better described by a system of nonlinear equations In those latter cases, though, we can still use calculus to take the derivative of those equations and convert them to an approximating linear system For example, the best linear approximation to the graph of a nonlinear function at any point on its graph is the tangent line to the graph at that point Let s take a look now at different techniques for solving systems of linear equations Systems of Linear Equations Consider the problem of solving the linear systems of equations such as or 3x + x 7 x + x 3 x + x + x 3 5 x x 3 The general linear system of m equations in n unknowns can be written a x + a x + + a n x n b a x + a x + + a n x n b a m x + a m x + + a mn x n b m In this system, the a ij s and b i s are given real numbers The number a ij is the coefficient of the unknown x j in the ith equation A solution of this system of equations is an n-tuple of real numbers x, x,,,, x n which satisfies each of the m equations in the system For instance, going back to our examples x, x solves the first system And, x 5, x, x 3 solves the second
For any linear system such as the one described above, we are usually interested in answering the following three questions: Does a solution exist? How many solutions are there? 3 Is there an efficient algorithm that computes actual solutions? There are essentially three ways of solving such systems: substitution, elimination of variables, and 3 matrix methods Substitution Substitution is the method usually taught in beginning algebra classes To use this method, first we solve one equation of the system for one variable, say x n, in terms of the other variables in the equation Then, we substitute this expression for x n into the other m equations The result will be a new system of m equations in the n unknowns x,,,, x n We can continue this process until we reach a system with just a single equation, a situation which is easily solved Finally, we can use our expressions of one variable in terms of the others to find all the x i s Let s go back to our initial example, 3x + x 7 x + x 3 From the second equation we have x 3 x and on substitution for x in the first we get 3x + (3 x ) 7, which leads to the solution x, x That these values satisfy our system of equations is easily verified Thus we have shown that a solution exists in this case The question whether this is the only solution possible remains to be answered 3
Consider now the equations, 3x + x 7 6x + x 4 Casual inspection tells us that the left-hand sides of the two equations have the same relation as the right-hand sides; ie the second equation is double the first Therefore, in effect we have only one equation But if there is only one equation involving two unknowns we can have an infinite number of solutions In this example, it is easy to verify that the values x θ and x 7 3θ satisfy the system of equations, whatever the value of θ While the substitution method is straightforward, it is not a method around which we can generate general solutions of linear systems Elimination of Variables The method that is most conducive to theoretical analysis is the elimination of variables Consider now the equations, x x 8 3x + x 3 We can eliminate the variable x from this system by multiplying the first equation by 3 to obtain 3x + 6x 4 and adding this new equation to the second one The result is 7x or x 3 To find x, we substitute x 3 back into our second or first equation to compute that x Note that we chose to multiply the first equation by 3 precisely so that when we added the new equation to the second one, we would eliminate x from the system More generally, when we want to solve a system of m equations by elimination of variables, we can use the coefficient of x in the first equation to eliminate the term x from all the equations below it We can then do the same thing with the remaining terms, until we eliminate enough variables to have a simplified system that can be solved by substitution 4
Matrix Methods When we perform the method of elimination of variables, we have to carefully keep track of the coefficients a ij and the b i s of the system each time we transform our system Matrices In order to do these operations in a careful manner, it usually makes sense to simplify the representation of linear systems with the use of matrices A matrix is a rectangular array of numbers The idea is to treat such arrays as single objects To explicitly indicate this intention, we should enclose the array within brackets: 3 7 3 5 7 Instead of brackets, sometimes you will see that parentheses or double ruling on both sides are also used: 3 7 3 7 3 5 3 5 7 7 The numbers that constitute a matrix are called the element entries of the matrix We typically refer to the elements by their row and column numbers, in that order So, for example, the (,) element of the matrix presented above is the number 5; the (,3) is 7; and so on If a matrix has n rows and m columns, it has altogether n m elements The dimensions of a matrix are the number of rows and columns it contains A matrix that has n rows and m columns is said to be of order n by m, or n m Therefore, the matrix presented above is of order 3 by 3 or 3 3 In giving the order of a matrix, we always mention the number of rows first, followed by the number of columns We are now almost ready to start making our matrices work for us; but first, it is convenient to introduce some more terminology 5
We typically use a single letter as a label for a matrix and also to use letters to designate its elements so that we may refer to matrices with arbitrary elements To distinguish between the letter designations of matrices and those of their elements, we shall follow the convention of using boldface capital letters for matrices, and lowercase, ordinary letters for their elements An example is A a a a m a a a m a n a n a nm Here we use the label A for the n m matrix whose typical element is a ij Since the exact numerical values of n and m are not specified, it is not possible to write the matrix in full So, we use dots to indicate the elements not written It is often informative to write A n m ((a ij )) to indicate that the matrix A with n rows and m columns has a typical element a ij A vector is an ordered set of numbers arranged in either a row or a column Therefore, matrices containing only one row are often called row vectors Similarly, matrices containing only one column are called column vectors For example, [ 3 7 is a row vector, and 3 is a column vector 7 We typically denote a vector by a boldfaced lower case letter, as in a If A is an n m matrix and n equals m, then A is a square matrix Several particular types of square matrices are worth noting: A symmetric matrix, A, is one in which a ij a ji for all i and j For example, A 3 7 3 5 7 6
A diagonal matrix is a square matrix whose only nonzero elements appear on the main diagonal, moving from upper left to lower right For example, B 7 5 A scalar matrix is a diagonal matrix with the same value in all diagonal elements For example, C 5 5 5 An identity matrix is a scalar matrix with ones on the diagonal This is always denoted I A subscript is sometimes included to indicate its size, or order For example, Algebraic Manipulation of Matrices I 3 Let s take a look now at some basic arithmetic operations of matrix algebra Equality of Matrices Two matrices (or vectors) are equal if and only if they have the same dimensions and their corresponding elements are equal A B if and only if a ij b ij for all i and j So, for example if A [ 3 4 and B [ x x 4 then A B implies that x and x 3 Transposition The transpose of a matrix A, denoted A, is obtained by creating the matrix whose mth row is the mth column of the original matrix 7
Thus, if B A, each column of A will appear as the corresponding row of B If A is n m, A is m n For example, A 3 5 5 6 4 5 3 4, A 5 6 3 4 3 5 5 4 An equivalent definition of the transpose of a matrix is: B A b ij a ji for all i and j The definition of a symmetric matrix implies that if A is symmetric, A A For any A, (A ) A Finally, the transpose of a column vector, a, is a row vector: Matrix Addition and Subtraction a [ a a a n We define addition (subtraction) of matrices in terms of addition (subtraction) of their corresponding elements, C A + B [ a ij + b ij Matrices cannot be added unless they have the same dimensions, in which case they are said to be conformable for addition So, for example, if [ a a A a a [ b b and B b b then their sum is A + B [ (a + b ) (a + b ) (a + b ) (a + b ) By way of numerical example 8
[ 3 4 + [ 5 6 7 8 [ ( + 5) ( + 6) (3 + 7) (4 + 8) A zero matrix or null matrix is one whose elements are all zero [ 6 8 In the addition of matrices, the zero matrix plays the same role as the scalar in scalar addition; that is, A + A We can also extend the operation of subtraction to matrices precisely as if they were scalars by performing the operation element by element Thus, A B [ a ij b ij It follows that matrix addition is commutative, A + B B + A, and associative, (A + B) + C A + (B + C), and that (A + B) A + B Multiplication by a Scalar Let k be an ordinary number (scalar) and A ((a ij )) be any matrix Then ka ((ka ij )) That is, to multiply a matrix by a scalar, we multiply each element of the matrix by that number It is easy to verify that multiplication of a matrix by a positive integer is the same as repeated addition For example, if A [ 3 4, 9
[ 4 6 A A + A 8 It is also the case that B n m A n m B n m + ( )A n m Matrix Multiplication Matrices are multiplied by using the inner product Let a be a row vector and b be a column vector Then, the inner product (or dot product) of a and b is a scalar and is written a b a b + a b + + a n b n For example a b [ 3 4 3 8 (3) + 3(8) + 4() 35 Notice that in the expression above, each term a j b j equals b j a j ; hence a b b a Matrix-Vector Multiplication: Let A be a matrix and v a column vector such that the number of columns of A equals the number of elements in v Then, the product A times v, written Av, is a column vector c whose ith element is equal to the inner product of the ith row of A with v For example, the first element of c is [ 4 5 4 5 c 5
the second element of c is [ 4 5 35 and so on and so forth So, the vector c is NOTE: c If A is a 3 matrix, and Av is defined (where v is a column vector), then we know that v has 3 elements If C is a 3 matrix, and u is a column vector with elements only, then Cu is not defined Matrix Multiplication: For an n m matrix A and an m p matrix B, the product matrix, 5 35 39 4 C AB, is an n p matrix whose ijth element equals the inner product of the row i of A and column j of B To avoid confusion, we can use the following notation: let a i denote the ith row of matrix A, and a i be the ith column of a matrix Then, So, for example: [ 3 4 [ 5 6 7 8 C AB c ij a i b j [ (5) + (7) (6) + (8) 3(5) + 4(7) 3(6) + 4(8) [ 9 43 5 Notice that the element c 43 in our new matrix is the dot product of a b [ 3 4 [ 5 7
To multiply two matrices, the number of columns in the first must be the same as the number of rows in the second, in which case, they are conformable for multiplication A simple way to check the conformability of two matrices for multiplication is to write down the dimensions of the operation, for example, (n m) times (m p) The inner dimensions must be equal; the result has dimensions equal to the outer values Multiplication of matrices is generally not commutative For example, [ 4 [ 3 AB 6 5 3, 4 5 3 4 5 but BA 4 6 5 [ 3 4 5 () + 4(4) (3) + 4(5) () + 4( ) () + 6(4) (3) + 6(5) () + 6( ) () + 5(4) (3) + 5(5) () + 5( ) 8 6 5 33 4 5 5 In other cases, even when the product AB exists, the product BA may not be defined This is the case, for example, if A if of order ( 3) while B is of order (3 7) In general, however, even if AB and BA do have the same dimensions, they will not be equal In view of thus, we define premultiplication and posmultiplication of matrices In the product AB, B is premultiplied by A, while A is posmultiplied by B Some general rules for matrix multiplication are as follows: Associative Law: (AB)C A(BC) Assuming that the matrices are conformable, we can get the product ABC either by postmultiplying AB by C or premultiplying BC by A Distributive Law: A(B + C) AB + AC Assuming that the factors of AB are conformable and that those of AC are also conformable, the product A(B + C), where parentheses signify operation that has priority, is equivalent to AB + AC Similarly, given conformability, (E + F)G EG + FG
Transpose of a product: (AB) B A By direct extension, (ABC) C B A Zero Matrix A conformable matrix of zeros produces the expected result: A Identity Matrix In ordinary algebra we have the number, which has the property that its product with any number is the number itself We now introduce an analogous concept in matrix algebra Consider the following matrices: A [ 5 3 4 and B [ The product of A times B and the product of B times A in this case are both equal to A as can be easily verified Notice that the matrix B is an identity matrix of order In matrix multiplication, the identity matrix is analogous to the scalar Elementary Operations We are going to focus now on some further properties of matrices and the matrix as an operator The main objective is to introduce the concept of inverse of a matrix Formally, there are three types of elementary row operations that can be carried out on a matrix: interchanging two rows, multiplying each element of a row by a nonzero scalar, and 3 adding a nonzero multiple of one row to another Each of these operations on the rows of a matrix can be carried out by premultiplying the given matrix by an appropriate elementary row operator To get the appropriate elementary row operator, all that we have to do is carry out the required operations on an n n identity matrix, if the given matrix is of order n m 3
For example, suppose we are given A 3 To interchange rows and 3, we premultiply A by E which is obtained by interchanging the first and third rows of I 3 Clearly, E A 3 3 Notice that by premultiplying A by E we interchanged the first and third rows of A Now, suppose that we want to multiply the second row of the same matrix A by a scalar, say, (-8) The appropriate elementary row operator for this is E 8 which, it may be noted, is constructed by performing the required operation: multiplying the second row of I 3 by (-8) We can verify that E A 8 3 3 8 which is A with its second row multiplied by (-8) as desired Finally, suppose that we wish to add twice the second row of A to the first 4
Once again, we perform the desired operation on I 3 and use the resulting matrix as an elementary row operator: And, we can verify that E 3 A E 3 3 4 3 Elementary column operations can be defined similarly They are equivalent to postmultiplication by appropriate elementary column operators (of order m m, if A is of order n m) which can be constructed by carrying out the specified elementary column operations on the identity matrix of the appropriate order Echelon Matrices Consider the following matrix, 3 B 4 4 7 and, the following sequence of elementary row operations on B: Subtract row from row Subtract twice row of the resulting matrix from its row 3 3 Subtract the new row from the new row 3 To perform these operations, the appropriate row operators are: E for operation (), E E 3 for operation (), for operation (3), 5
Applying these in sequence, we get E B E (E B) 3 4 4 7 3 4 7 3 4 7, 3, and E 3 (E E B) 3 3 The last matrix illustrates the concept of an echelon matrix Notice that each row begins with more zeros than does the previous row Definition A row of a matrix is said to have k leading zeros if the first k elements of the row are all zeros and the (k + )th element of the row is not zero With this terminology, a matrix is in row echelon form if each row has more leading zeros than the row preceding it So, for example, 3 4 5 6 is in echelon row form Each non-zero entry in each row of a matrix in row echelon form is called a pivot In this case, the pivots are the numbers, 4, and 6, respectively Definition A row echelon matrix in which each pivot is a and in which each column containing a pivot contains no other nonzero entries is said to be in reduced row echelon form So, for example, 6
is in reduced echelon form Inverse of a Square Matrix Earlier today we covered the matrix operations of addition, subtraction, and multiplication In arithmetic and ordinary algebra, however, there is also an operation of division Can we define an analogous operation for matrices? Strictly speaking, there is no such thing as division of one matrix by another; but there is an operation that accomplishes the same thing as division does in arithmetic and scalar algebra In arithmetic, we know that multiplying by is the same thing as dividing by More generally, given any nonzero scalar a, we can speak of multiplying by a instead of dividing by a The multiplication by a has the property that aa a a This prompts the question, for a matrix A, can we find a matrix B such that where I is an identity matrix of order n BA AB I n In order for the equality in the previous expression to hold, AB and BA must be of order n n; but AB is of order n n only if A has n rows and B has n columns, and BA is of order n n only if B has n rows and A has n columns Therefore, the equality will only hold if A and B are both of order n n This leads to the following definition: Definition 3 Given a square matrix A, if there exists a square matrix B, such that BA AB I then B is called the inverse matrix (or simply the inverse) of A, and A is said to be invertible Not all square matrices are invertible, though A square matrix that does not have an inverse is said to be singular 7
A square matrix that possesses an inverse is said to be nonsingular To illustrate the concept of inverse of a matrix, consider the following example: Given a matrix, A [ 3 4 it is easy to verify that the matrix satisfies the relations [ B 4 3 BA AB I Therefore the matrix B is the inverse of A Similarly, given the matrix C D 3 4 5 4 5 5 satisfies the relations CD DC I Hence D is the inverse of C A Procedure to Calculate the Inverse of a Matrix If It Exists Let us first confine attention to finding a matrix B, given a matrix A, such that the premultiplication requirement BA I is satisfied 8
For the moment, we are not concerned with the postmultiplication requirement that AB should also be equal to I Our task, then, is to find a premultiplying matrix B n n that transforms A n n into the identity matrix I This reminds us of the procedure described earlier in connection with deriving an echelon matrix If we can find a sequence of row operations that transforms the given matrix A n n into I, then the premultiplying matrix must be the B n n matrix we are looking for Let see whether the method works and, if so, how Suppose the given square matrix is A [ 3 4 The first part of our procedure is to find a sequence of elementary row operations that transforms A into an echelon form In the present case this task is easily accomplished: subtract three times the first row from the second or, which is the same thing, premultiply A by [ E 3 This yields [ E A 3 [ 3 4 [, Note that this echelon matrix does not have any row consisting entirely of zeros There are ones all the way from top-left down to bottom-right of the principal diagonal Also the elementary row operator created zero elements everywhere below the principal diagonal What remains now is to perform additional row operations so that all entries above the principal diagonal become zero In the present case, this is accomplished by subtracting the second row from the first or, which is the same thing, premultiplying by 9
[ E giving E E A [ [ [ Finally, the product E E E [ [ 3 [ 4 3 is the matrix B we are looking for, satisfying the relation BA I We have to be careful, though, when we do this last step: notice the order in which the matrices are entered, E on the left of E We can also check if our calculation is correct: [ [ [ 4 3 3 4 Moreover, we can also easily verify that the matrix which satisfies the premultiplication requirements, also satisfies the postmultiplication requirement: [ 3 4 [ 4 3 [ In fact, it can be shown mathematically that this holds true in general It may also be noted that if a square matrix has an inverse, then this inverse is unique Suppose that it is not and that C is a different inverse of A Then CAB CAB, but (CA)B IB B and C(AB) C, which would be a contradiction if C did not equal B Properties of the Inverse We just learned a way to find a square matrix B such that BA I If the matrix B exists, it is the inverse of A, denoted From the definition, B A,
A A I In addition, by premultiplying by A, postmultiplying by A, and then canceling terms, we find as well AA I The following facts about the behavior of the inverse are also worth noting: If a row of A is all zeros, then A does not exist If a column of A is all zeros, then A does not exist If a row of A is a multiple of another row, then A does not exist If a column of A is a multiple of another row, then A does not exist If a matrix is of the form with d i, then D D d d d 3 d d d 3 Let A be a square invertible matrix, then (A ) A Let A be a square invertible matrix, then (A ) (A ) Let A and B be square invertible matrices, then AB is invertible, and (AB) B A Gaussian Elimination Algorithm for Finding A Besides the method outlined above, there are other ways to compute the inverse of a square matrix, if it exists A particularly useful algorithm uses Gaussian elimination to do this Given a matrix A 4 4 4
and the identity matrix, I 3, we can obtain the matrix [A I 4 4 4 which is called the augmented matrix of A Just as before, we can perform the elementary row operations on the augmented matrix [A I: Interchange two rows (R i R j ) Multiply any row by a nonzero scalar (kr i ) 3 Add a multiple of one row to another row (kr i + R j ) Algorithm Start at the top row Is there a nonzero entry in the diagonal position? No if possible, interchange with a lower row to get a nonzero entry if not possible, then A does not exist Yes divide each element in the row by the diagonal element (pivot) to get a in the pivot position 3 Add multiples of the row being considered to the lower rows to sweep out any nonzero entries below the pivot 4 Go to the next lower row and repeat steps and 3 5 Continue as in 4 until all rows are considered You will now have an echelon matrix 6 Now begin from the bottom row and work up to sweep out nonzero entries above the pivots 7 When you reach the form [I B then B A So, for example, given the augmented matrix [A I, R +R 4R 4 4 4 +R 3 8 8 4 8 8 4 R R 3 8 8 4,,
8 8 4 8 R R 3, 8 R 3+R 8, R 3 +R 8 R 8 +R 8, 8 8 8 A 8 4 4 4 8 4 Using Matrices to Solve Systems of Linear Equations At the beginning of today s class, we looked at two ways of solving linear systems of equations Now, we are going to examine the application of matrix algebra to the solution of systems of equations First, we should note that matrices provide a convenient way of collecting sets of equations and equations involving sums of values For example, suppose we have the following system of equations: x + 3x x x 3 we can arrange the coefficients as: [ 3 and call this array, the coefficient matrix We can also collect the values corresponding to the right-hand side of the system and represent them with a column vector [ 3 3
Vector Representation of a System of Linear Equations Consider now the following equations in two unknowns: x + 3x 5 3x + x 5 We can form three column vectors, corresponding to the coefficients of x, those of x, and the the values corresponding to the right-hand side of the system: [ [ [ 3 5 a, b, c 3 5 Now the given set of equations can be expressed compactly as x a + x b c To check this out, we note that the vector equation just written is equivalent to [ [ [ 3 5 x + x 3 5 which by virtue of the definition of scalar multiplication, becomes [ [ x 3x + 3x x [ 5 5 which, in turn, by virtue of the definition of addition is the same as [ (x + 3x ) (3x + x ) and now the definition of equality gives x + 3x 5 3x + x 5 [ 5 5 Elementary Operations Let us consider now the following two equations in two unknowns: x + 3x 5 3x 6x 3 4
We can solve this set of equations using the method of elimination of variables In this case, multiplication of the first equation by gives 4x + 6x Adding this to the second equation in the initial set, we get 7x 7, which leads to the solution x, x These same steps can be performed by matrix multiplication Let s see: First, we should write the initial set of equations in matrix form [ 3 3 6 [ x x [ 5 3 If we premultiply both sides of this matrix equation by E [ we get [ 4 6 3 6 [ x x [ 3 which is equivalent to the same operation that we preformed earlier Now let us premultiply the last expression by E [ the result is [ 7 3 6 [ x x [ 7 3 which gives the equation 7x 7, obtained earlier, and the second equation in the initial set Using the Inverse of a Matrix to Solve Systems of Equations We have already seen that a system of linear equations can be compactly express as a single matrix equation 5
Consider the following two equations with two unknowns x + 3x 4x + 9x We can rewrite them as [ 3 4 9 [ x x [ If [ 3 4 9 exists, premultiplication of both sides of the previous expression gives [ 3 4 9 [ 3 4 9 [ x x [ 3 4 9 [ that is [ [ x x [ 3 4 9 [ or [ x x [ 3 4 9 [ thus giving the solution required [ [ 3 In this case, exists and it is 4 9 Therefore, the required solution is or x and x [ x x [ 9 3 6 6 4 6 6 9 3 6 6 4 6 6 [ If we check this answer by direct substitution in the initial set of equations, we get + + [ thus demonstrating that this is indeed a solution to the system 6
As another example, consider the following system of three equations in three unknowns: x + x + 3x 3 x + 3x + 5x 3 x + 5x + 9x 3 3 Writing this system as a matrix equation, we have 3 3 5 5 9 x x x 3 3 If we premultiply of both sides of the previous expression by 3 3 5 5 9 we get 3 3 3 3 5 5 9 x x x 3 3 3 3 which is x x x 3 giving the solution x, x, and x 3 ; which on direct substitution in the original equations yields + + + 3 + + 5 + 3 thus proving that x, x, and x 3 is indeed a solution to the given system of equations 7
Generalizing from these examples, if A is an n n matrix, and x and b are both column vectors having n elements, the former consisting of unknowns and the latter of known constants, then a solution to the system of equations Ax b can be obtained by premultiplying both sides of the equation by A, if it exists This is so because is the same as A Ax A b Ix A b by virtue of the definition of the inverse (A A I), and this in turn is the same as x A b by virtue of the definition of the identity matrix That x A b satisfies the given equation system can be seen by substitution A(A b) Ib b More About Simultaneous Linear Equations We are now ready to answer the questions about existence and uniqueness of solutions that were posed at the beginning of today s lecture We are going to rely on two important concepts, that of linear dependence among a collection of vectors, and that of the rank of a matrix The latter concept is used in discussing equation systems with no solution, one unique solution and infinitely many solutions Definition 4 Linear Combination Given a set of vectors, a sum of scalar multiples of vectors, all containing the same number of elements, is called a linear combination of vectors in the set So, for example, given a [ 3 and b [ 8
any sum such as k a + k b, where k and k are any two scalars, not both zero, is a linear combination of a and b Linear combinations of what are known as unit vectors are worth special mentioning An n-tuple is called a unit vector is all except one of its elements are zero and one element is unity Any n-tuple can be written as a linear combination of the corresponding set of unit vectors For example, 3 5 3 + 5 Definition 5 Linear Dependence A set of vectors is linearly dependent if any one of the vectors in the set can be written as a linear combination of the others For example, the set of vectors 3 5, 6, is linearly dependent because one of them is twice another: 6 3 5 For another example, if a [ [ 3, b 3 8 4 [, and c 4 then a + b c, so a, b, and c are linearly dependent Any of the three possible pairs of them, however, are not linearly dependent One way to check whether a given set of vectors is linearly dependent is the following: [ [ Suppose we want to check whether, and are linearly dependent 4 3 9
If they are, we know that there exist, by definition, numbers k and k, not both zero, such that [ [ [ k + k 4 3 Note that this vector equation is equivalent to the following two equations in two unknowns (k and k ): solving which we find k k k + k 4k + 3k Therefore, in this case, there are no k and k, not both zero,such that the vector expression is satisfied We can conclude that the two vectors are not linearly dependent, or that they are linearly independent Another way of checking whether a given set of vectors is linearly dependent is the following: Concatenate the given set of vectors into a matrix, Derive an echelon form from it, 3 If the echelon form has one or more rows containing nothing but zeros, declare the collection as linearly dependent Suppose we want to determine whether the following vectors are linearly dependent: 4 7, 5, 8 3 6 9 Stacking the transposes of these vectors, we get the following matrix: 3 A 4 5 6 7 8 9 Now, lets get in into echelon form: 3 4 5 6 4R +R 7R 7 8 9 +R 3 3 3 3 6 6,
3 3 6 6 3 6 3 R 6R +R 3 Bingo! This set of vectors is linearly dependent 3 6 3 Once a matrix is transformed into an echelon matrix by elementary row operations, we can count the number of nonzero rows in the resulting echelon matrix Definition 6 Rank of a Matrix The rank of a matrix is the number of nonzero rows in its row echelon form So, in the preceding example, we say that the rank of original matrix A The usual notation for the rank of a matrix A is r(a) The rank is zero only for a zero (null) matrix All other matrices have positive (greater than zero) rank The rank of an n m matrix exceeds neither n nor m The rank of a square matrix determines whether the matrix has an inverse A square matrix of order n n is said to be of full rank if its rank is n A square matrix of full rank has an inverse; such matrices are said to be nonsingular Square matrices with less than full rank are said to be singular, and they are not invertible Rank - The Fundamental Criterion Using the ranks of two matrices associated with simultaneous linear equations, it is possible to determine whether the equations have no solution, one solution, or infinitely many solutions Consider a system of m linear equations in n unknowns, x, x,, x n, a x + a x + + a n x n b a x + a x + + a n x n b a m x + a m x + + a mn x n b m where m does not necessarily has to be equal to n 3
We can write this system of equations in matrix from as AX B, where a a a m a a a m A X a n a n a nm x x x n B We can solve the system by considering the augmented matrix [A B and using Gaussian elimination Consider the following system of equations: b b b m The coefficient matrix A into the [A B matrix: 3x 5x 3 x + x 3 6x + x 3 5 6 [A B and the matrix B 3 5 3 3 6 3 3 can be augmented Using Gaussian elimination, 3 5 3 3 R R 6 3 3 5 3 6 3 8 3 4 6 3R +R 6R +R 3 3 4 4 6 8 R 4R +R 3 3 3 5 3 6 3 8 3 4 6 3 4 4 6 3 4,,,, 3
3 4 R 3+R 4 Our reduced echelon matrix corresponds to the system x x 4 Therefore, this system has a unique solution: x and x 4 Consider now the following system of equations: The augmented into the [A B matrix is: Using Gaussian elimination, [A B x + x 4 3 x + x 5 x 3 + x 4 5 x + x 3 3 5 5 3 5 5 3 5 5 Our reduced echelon matrix corresponds to the system x + x 4 3 x x 4 5 x 3 + x 4 5 It is clear that there is no single solution to this system of equations 33
For any value of x 4, the system of equations, determines the corresponding values of x, x and x 3 Note also that x 4 can be anything So, we can assign any arbitrary number θ as its value, and rewrite the system as x 3 θ x 5 (3 θ) θ 5 x 3 5 θ This system has many solutions (the system is underdetermined) We can easily check this by substituting back the values for the coefficients: If x 4, then x x 5 x 3 5 θ But if x 4 75, then x 5 x 5 x 3 5 θ Finally, consider the following system of equations: The coefficient matrix A into the [A B matrix: a + b + c 3 a + b + c a + 3b + 3c 6 3 3 [A B and the matrix B 3 3 3 6 3 6 can be augmented 34
Using Gaussian elimination, 3 R +R R 3 3 6 +R 3 3 3 R +R 3 3 3 3 5 The last row corresponds to the equation (a) + (b) + ()c 5 The left-hand side of this equation is always zero and thus can never equal 5 So, there is no tuple a, b, c which solves this equation This system has no solution (the system is overdetermined) Homogeneous Equations Consider a system in which al the bj s on the right-hand side are : a x + a x + + a n x n a x + a x + + a n x n a m x + a m x + + a mn x n Such a system is called homogeneous We can write this system of equations in matrix from as AX, where a a a m a a a m A X a n a n a nm x x x n, B Any homogeneous system has at least one solution: x x x n Such a solution is usually called the trivial solution Using the Rank Criterion Given a system of equations, let A be the coefficient matrix and let [A B be the augmented matrix 35
Then, 36