The Four Fundamental Subspaces

Transcription

1 The Four Fundamental Subspaces Introduction Each m n matrix has, associated with it, four subspaces, two in R m and two in R n To understand their relationships is one of the most basic questions in linear algebra What we want to do, first, is to define the subapaces in terms of the row vectors and the column vectors of the given matrix, and then to find description of them This means that we want to find a basis for each, ie, find a linearly independent set of vectors in terms of which each vector in the subspace can be written uniquely as a linear combination of the basis elements Of course, since the dimension of a vector space is the number of elements in a basis, we will have, in each case, the dimension of the subapace The key tool that is used to answer all these questions is the reduction of a matrix to row-echelon form All the basic informtion on the subspaces is encoded in the reduced matrix, sometimes directly, sometimes indirectly as we shall see In the following, we will use a particular matrix to illustrate the discussion The matrix we often use is which has row-echelon form U = Before proceeding, you should verify this claim The Four Subspaces The Row Space of A This subspace is just the subspace spanned by the rows of the matrix A Since each elementary row operation that reduces A to its row-echelon form U either simply intershanges rows or forms linear combinations of the existing rows, the subspace spanned by the rows of A is exactly the same as the subspace spanned by the rows of U Now the non-zero row vectors of the row-echelon form each contains a leading entry So, for example, in the i th row suppose that the leading entry occurs in the j th column Then all entries below that leading entry are zero (u ik = for k > j) It follows that the non-zero rows of the row-echelon form are linearly independent

2 In our concrete example, the first and second rows of U are linearly independent since = (,,, ) = c (,,, ) + c (,,, ) = (c, c, c + c, c + c ) clearly implies that c = c = In general a row-echelon form π c π c π c π r c r c r+ c m the second and third rows ( π ) and ( π ) are independent just as in the concrete example Obviously, there can be at most m non-zero rows in the row-echelon form The number of these non-zero rows is called the rank of the matrix A ; it is often denoted r Since these non-zero rows are linearly independent and span the row space, the number of non-zero rows is the dimension of the subspace In other words, the dimension of the row space is just r and r m The Null Space of A The null space of the matrix A, usually denoted by N(A) is defined as N(A) = {x R n Ax = } Otherwise said, it is just the set of all solutions of the linear homogeneous equation Ax = On the other hand, since the operations of putting a matrix in rowechelon form simply replaces the original equations with an equivalent system, the set of solutions does not change, ie, the set of solutions to Ax = is exactly the

3 same as the set of solutions of Ux = Each of the r non-zero rows of the rowechelon form U has a leading entry and the columns containing those leading entries correspond to the basic variables; the other n r columns correspond to the free variables and these free variables can be chosen arbitrarily So we may successively assign a value of to a particular free variable, zero to the others and solve Ux = by back substitution The n r vectors so obtained then form a basis for N(A) and dim N(A) = n r The number n r is sometimes called the nullity of A; N(A) is also called the kernel of A and is written ker(a) Notice the crucial fact that rank (A) + nullity (A) = n Example Consider with REF U = The columns of U which contain a leading entry correspond to x and x so that the free variables are x and x 4 Set x =, x 4 = and solve the resulting equations to obtain the equations x = and x + = hence x = (,,, ) is one solution Setting x = and x 4 = leads to the equations x + = and x +x +x 4 = so that x = and x = / Hence x = (,, /, ) is the other solution These vectors are linearly independent and form a basis for N(A) Here we see that dim (N(A)) =, rank (A) = and +=4! The Column Space of A The column space of A is also called the range of A (which it is if we consider the mat x Ax!) and is usually denoted R(A) Our object is to find a basis for R(A) as well as its dimension To do this, we must realize at the outset that the space spanned by the columns of A and that spanned by the columns of U, its row-echelon form, are not the same While the elementary row operations leave the row space and null space unchanged, the columns are significantly altered In our running example: so using the first and third columns, It is unfortunate that row space starts with the letter r as does the term rank One must be careful in using this letter!

4 + 9 = while no linear combination of the columns of U = 5 can produce a vector with a non-zero third component! The key observation we need to make is that the homogeneous equation Ax = expresses a linear dependence relation between the columns of A whose coefficients are just the components of the vector x But since Ax = if and only if Ux =, each dependence relation on the columns of A is matched by a linear dependence relation on the columns of U with exactly the same coefficients! Again, going back to the example, the last column is dependent on the first and third: 5 = + 9 while the last column of U is again dependent on the first and third columns of U with the same coefficients: = + Now, to find a basis for R(A) we first find a basis for R(U) which is eas: the basis for R(U) is formed just by the columns that contain a pivot (or, what is the same thing, the columns corresponding to the basic variables); there will be r of them We then choose the columns of the original matrix A corresponding to the columns of U that contain the pivots It is important to notice that the dimension of R(A) is just the rank of A and that is exactly the dimension of the row space of A Otherwise said, the number of independent columns is exactly the number of independent rows of A This leads to the statement, which is not at all obvious for, say a 7 matrix that the column rank (the number of linearly independent columns of A) is exactly the row rank of A! So we have another crucial relation; 4

5 row rank (A) = column rank (A) In particular, for a square n n matrix, if the rows are linearly independent, then so are the columns To see how this fits with our running example, since with REF U = it is the first and third columns that contain the pivots of U and hence a basis for R(A) is formed by the first and third columns of A namely and 9 4 The Null Space of A The columns of A are the rows of A and we can take the transpose of A y = to get y = (,,, ) (a row vector) The vector y is sometimes called a left null vector of A and the null space of A is then called the left null space of A We use this terminology below Clearly the product y A is a linear combination of the rows of A and produces a row-vector Now the dimension of N(A ) is easy to find Indeed, for any matrix the number of basic variables + the number of free variables is just the number of columns of the matrix In light of what we have just done, this means that, for an m n matrix we see once again that: rank (A) + nullity (A) = dim R(A) + N(A) = n Now apply this formula to the matrix A which has m columns Here the rowrank is the same as the column rank and that is r hence r + dim N(A ) = m and so dim N(A ) = m r So much for the question of the dimension of the null space of A What about finding a basis? Suppose that we have a LU-decomposition where some row interchanges must be done then we have the factoriaation P LU which can be written as L PU Let the last m r rows of U be zero rows Then the last m r row of L P are a basis for the left null space 5

6 5 Summary The foregoing is somtimes put together in the form of a Theorem: Theorem Fundamental Theorem of Linear Algebra, Part I: (a) R(A ) = row space of A has dimension r (b) N(A) = null space of A has dimension n r (c) R(A) = column space of (A has dimension r (d) N(A ) = null space of (A) has dimension m r Orthogonality Recall that orthogonality of vectors is defined in terms of an inner product Here, we take the usual dot product of two vectors as the inner product and say that two vectors, x, y R n are orthogonal provided x, y = n x i y i = We note that the vector is orthogonal to all vectors in R n i= The associated norm (or length) is then given by x = x, x If we look at the norm of x y and expand in terms of the inner product, we find x y = x y, x y = x, x y y, x y = x, x xy y, x + y, y = x x, y + y This means that the Pythagorean Theorem x y = x + y holds if and only if x, y = It also means that the inner product can be used to give a definition of the angle between two vectors To see this we start by using the Law of Cosines which says that if the vectors x and y form two sides of a triangle with included angle θ, then the third side is x y and x + y x y cos (θ) = x y But, from the expression for x y computed above, we find that x y cos (θ) = x, y, or x, y = x y cos (θ) 6

7 Now, we can introduce the idea of orthogonal subspaces In R, for example, the z-axis is a one dimensional subspace and any vector lying in that subspace is clearly orthogonal to any vector lying in the (x, y)-plane This simple idea generalizes to the notion of orthogonal subspaces in higher dimensional spaces Definition Two subspaces, V and W of the space R n are called orthogonal subspaces provided every vector v V is orthogonal to every vector w W If we look at the four fundamental subspaces of a matrix, we notice, first, that N(A) and R(A ) are both subspaces of R n while the other two subspaces, N(A ) and R(A) are subspaces of R m The most important fact about these two pairs of subspaces, other than their dimension, is that they are orthogonal pairs of subspaces Let us see why this is so I : The N(A) is orthogonal to R(A ) To see this, suppose x N(A) The equation Ax = can be written out, symbolically, as follows A x = row row row m x x x n = This is equivalent to m equations formed by taking inner products of the rows of A with the vector x Hence the vector x is orthogonal to all the rows of the matrix A But the rows of A are the columns of A So if v is in the column space of A = R(A ), it is a linear combination of the columns of A we have x, v = Hence N(A) R(A ) II : The left null space N(A ) is orthogonal to the column space R(A) The simple way to prove this statement is to apply statement (I) to the matrix A Or, we can write y ( ) y y y m col col col n = ( ) from which we see that y is orthogonal to every column of A Therefore it is orthogonal to every linear combination of columns, that is, every y N(A ) is orthogonal to every x R(A) 7

8 Let us look at an example Example Let ( 4 8 ) with REF U = ( 4 ) The second column is basic and the other threee variables are free Therefore, if we set each free variable equal to one, in turn, and solve Ux =, we can compute the basis for the null space N(A) to be, 4, It is easy to check that these vectors are all orthogonal to the rows of A as they should be according to our result Since the row rank is equal to the column rank, the column space of A is onedimensional and is spanned by the one basic column (, ) On the other hand, the left null-space is found by combining the rows of A to produce the zero row in the matrix U Since this is done by adding row + row, y = (, ) and the inner product of the second column with this last vector is clearly Finally, we have the following important definition: Definition If V is a subspace of R n the set of all vectors orthogonal to all the vectors of V is called the orthogonal compliment of V We denote the orthogonal compliment of V by V Note that if some vector v were orthogonal to N(A) but was not in the row space of A then adding v as an extra row of A would enlarge the row space without changing the null-space But we have the formula dimr(a ) + dim N(A) = number of columns so it is impossible for the row space to change dimension by adding v Hence we have R(A ) = N(A) Similar reasoning with A leads to the following final result 8

9 Theorem 4 Fundamental Theorem of Linear Algebra part II N(A) = (R(A )), R(A) = N(A) N(A ) = (R(A)), R(A) = (N(A )) One final remark: The last equality means: Ax = b has a solution if and only if b is orthogonal to N(A ) Or, said another way b is in the column space if and only if it is orthogonal to every solution y of the transposed homogeneous equation A y = From this fact, it follows that Ax = b is solvalble for every right-hand vector b if and only if the transposed homogeneous equation has only the zero solution This statement is the finitie dimensional analog of what is known as the Fredholm Alternative Theorem: either Ax = b has a solution for every b or A y = has non-trivial solutions, but not both 9