Lecture 1: Basic Concepts

Transcription

1 ENGG 5781: Matrix Analysis and Computations Lecture 1: Basic Concepts First Term Instructor: Wing-Kin Ma This note is not a supplementary material for the main slides. I will write notes such as this one when certain concepts cannot be put on slides. This time, the aim is to elaborate upon subspace concepts at a more fundamental level. 1 Subspace and Linear Independence 1.1 Subspace A nonempty subset S of R m is called a subspace of R m if, for any α, β R, x, y S = αx + βy S. It can be verified that if S R m is a subspace and a 1,... a n S, then any linear combination of a 1,... a n, i.e., n α ia i, where α R n, also lies in S. Given a collection of vectors a 1,... a n R m, the span of {a 1,... a n } is defined as { span{a 1,... a n } = y = α i a i α R }. n In words, span{a 1,... a n } is the set of all possible linear combinations of the vectors a 1,... a n. It is easy to verify that span{a 1,... a n } is a subspace for any given a 1,... a n. In the literature, span is commonly used to represent a subspace. For example, we can represent R m by In fact, any subspace can be written as a span: R m = span{e 1,..., e m }. Theorem 1.1 For every subspace S R m, there exists a positive integer n and a collection of vectors a 1,..., a n R m such that S = span{a 1,..., a n }. In the literature you would notice that we take the result in Theorem 1.1 for granted in a way that is almost like a common sense and without elaboration. There is an easy way to prove Theorem 1.1, but the proof requires some result in linear independence. We relegate the proof to the later part of this note. 1.2 Linear Independence A set of vectors {a 1,... a n } R m is said to be linearly independent if α i a i 0, for all α R n with α 0. Otherwise, it is called linearly dependent. A subset {a i1,..., a ik }, where {i 1,..., i k } {1,..., n} with i j i l for any j l, and 1 k n, is called a maximal linearly independent subset of {a 1,..., a n } if it is linearly independent and is not contained by any other linearly independent subset of {a 1,..., a n }. From the above definitions, we see that 1

2 1. if {a 1,..., a n } is linearly independent, then any a j cannot be a linear combination of the set of the other vectors {a i } i {1,...n},i j ; 2. if {a 1,..., a n } is linearly dependent, then there exists a vector a j such that it is a linear combination of the other vectors; 3. {a i1,..., a ik } is a maximal linearly independent subset of {a 1,..., a n } if and only if {a i1,..., a ik, a j } is linearly dependent for any j {1,..., n} \ {i 1,..., i k }. Thus, roughly speaking, we may see a linearly independent {a 1,..., a n } as a non-redundant or sufficiently different set of vectors, and a maximal linearly independent {a i1,..., a ik } as an irreducibly non-redundant set of vectors for representing the whole vector set {a 1,..., a n }. It can be easily shown that for any maximal linearly independent subset {a i1,..., a ik } of {a 1,..., a n }, we have span{a 1,..., a n } = span{a i1,..., a ik }. We also have the following results which are very basic and we again take them almost for granted: 1. Let {a 1,..., a m } R m be a linearly independent vector set. Suppose y span{a 1,..., a m }. Then the coefficient α for the representation y = α i a i is unique; i.e., there does not exist a β R n, β α, such that y = n β ia i. 2. If {a 1,... a n } R m is linearly independent, then n m must hold. The first result is simple: if there exists a β α such that y = n β ia i, then we have n (α i β i )a i = y y = 0, which contradicts the linear independence of {a 1,..., a m }. For the second result, I show you in the proof in [2]. The proof is done by induction. Suppose m = 1. Then it is easy to see that n 1 must hold. Next suppose m 2. We need to show that n m. Partition the a i s as [ ] bi a i =, i = 1,..., n, c i where b i R m 1, c i R. If c 1 =... = c n = 0, then the linear independence of {a 1,..., a n } implies that {b 1,..., b n } is also linearly independent. By induction, we know that for any collection of n linearly independent vectors in R m 1, it must hold that n m 1. It follows that {b 1,..., b n } must satisfy n m 1 too; hence, we get n m. If c i 0 for some i, we need more work. Assume w.l.o.g. that c n 0 (we can always reshuffle the ordering of a 1,..., a n ). For any α 1,..., α n 1 R, choose α n as follows ( α n = 1 n 1 ) α i c i. c n Then we see that [ n 1 α i a i = α ib i n 1 c ] [ n 1 ( )] i c n α i b n = α i b i c i c n b n 0 0 Let b i = b i (c i /c n )b n for ease of explanation. From the above equation, we see that the linear independence of {a 1,..., a n } implies that { b 1,..., b n 1 } is linearly independent. By induction, we know that { b 1,..., b n 1 } must satisfy n 1 m 1; hence, we get n m. The proof is complete. 2

3 1.3 Orthogonality and Orthonormality A collection of vectors a 1,..., a n R m is said to be orthogonal if a T i a j = 0 for all i, j with i j, and orthonormal if a T i a j = 0 for all i, j with i j and a i 2 = 1 for all i. Some basic results arising from such a definition are as follows. 1. An orthogonal or orthonormal collection of vectors is also linearly independent. 2. Let {a 1,..., a n } R m be an orthonormal set of vectors. Suppose y span{a 1,..., a n }. Then the coefficient α for the representation y = α i a i is uniquely given by α i = a T i y, i = 1,..., n. 3. Let {a 1,..., a n } R m be a linearly independent set of vectors. There exists an orthonormal set of vectors {q 1,..., q n } such that span{a 1,..., a n } = span{q 1,..., q n }. The first and second results above are easy to verify. The proof of the third result is constructive and is a consequence of the Gram-Schmidt procedure; see the main slides for the proof. 1.4 Proof of Theorem 1.1 We now prove Theorem 1.1. Suppose that the subspace S does not equal {0}; the case of S = {0} is trivial to prove. Let n be a positive integer and a 1,..., a n S. We have span{a 1,..., a n } S, and the reason is that any linear combination of vectors in S also lies in S. Now we wish to show that there exists a linearly independent collection of a 1,..., a n S such that S span{a 1,..., a n }. We adopt a constructive proof. First, pick any nonzero vector a 1 S. If any y S can be written as y = α 1 a 1 for some α 1, then we finish. If not, consider the following recursive process. Suppose that we have previously picked a linearly independent collection of vectors a 1,..., a k 1 S, but there exists y S such that y k 1 α ia i for any α i s. Note that we have the case of k = 1 already. Then we pick a k as any y S that satisfies y k 1 α ia i for any α i s. Clearly, a 1,..., a k are linearly independent. If any y S can be written as y = k α ia i for some α i s, then it means S span{a 1,..., a k } and we finish. Otherwise, we increase k by one and repeat the above steps. The question is whether the recursive process above will stop; if yes, our proof is complete. Suppose that the process has reached an iteration number of k > m. On one hand, the process says that {a 1,..., a k } is linearly independent. On the other hand, if {a 1,..., a k } is linearly independent, then k m must hold (a take-it-for-granted fact in linear independence). Hence, by contradiction, the process must stop; k = m is the largest possible iteration number. The proof is complete. 3

4 2 Basis and Dimension Let subspace S R m with S {0}. A set of vectors {b 1,..., b k } R m is called a basis for S if {b 1,..., b k } is linearly independent and S = span{b 1,..., b k }. Taking S = span{a 1,..., a n } as an example, any maximal linearly independent subset of {a 1,..., a n } is a basis for S. From the definition of bases, the following facts can be shown: 1. A subspace may have more than one basis. 2. A subspace always has an orthonormal basis (if it does not equal {0}). 3. All bases for a subspace S have the same number of elements; i.e., if {b 1,..., b k } and {c 1,..., c l } are both bases for S, then k = l. Given a subspace S R m with S {0}, the dimension of S is defined as the number of elements of a basis for S. Also, by convention, the dimension of the subspace {0} is defined as zero. The notation dim S is used to denote the dimension of S. Some examples are as follows: We have dim R m = dim span{e 1,..., e m } = m. If {a i1,..., a ik } is a maximal linearly independent subset of {a 1,..., a n }, then dim span{a 1,..., a n } = k. We have the following properties for subspace dimension. 1. Let S 1, S 2 R m be subspaces. If S 1 S 2, then dim S 1 dim S Let S be a subspace of R m. We have dim S = m if and only if S = R m. 3. Let S 1, S 2 R m be subspaces. We have dim(s 1 + S 2 ) dim S 1 + dim S 2 ; note that the notation X + Y = {x + y x X, y Y} denotes the sum of two subsets X, Y R m. 3 Projections onto Subspaces Let S R m be a nonempty closed set, and let y R m be any given vector. A projection of y onto S is any solution to the following problem min z z S y 2 2. In words, a projection of y onto S is a point in S that is closest to y in the Euclidean sense. In general, there may be more than one such closest point. If every y R m has only one projection of y onto S, we will use the following notation Π S (y) = arg min z S z y 2 2. to denote the projection of y onto S. We are interested in projections onto subspaces. Such concepts play a crucial role in linear algebra and matrix analysis. Consider the following theorem: Theorem 1.2 Let S be a subspace of R m. 1. For every y R m, there exists a unique vector y s S that minimizes z y 2 over all z S. Thus, we can write Π S (y) = arg min z S z y

5 2. Given y R m, we have y s = Π S (y) if and only if y s S, z T (y s y) = 0, for all z S. (1) The above theorem is a special case of the projection theorem in convex analysis and optimization [1, Proposition B.11], which deals with projections onto closed convex sets. In the following we provide a proof that is enough for the subspace case. Proof: First, we should mention that there always exists a vector in S at which the minimum of z y 2 2 over all z S is attained; in this claim we only need S to be closed. This result is shown by applying the Weierstrass theorem, and readers are referred to [1, proof of Proposition B.11] for details. Second, we show the sufficiency of Statement 2. Let y s S be a vector that minimizes z y 2 over all z S. Since z y 2 2 y s y 2 2 for all z S, and we have The above equation is equivalent to z y 2 2 = z y s + y s y 2 2 = z y s (z y s ) T (y s y) + y s y 2 2, z y s (z y s ) T (y s y) 0, for all z S. z z T (y s y) 0, for all z S; the reason is that z S implies z y s S, and the converse is also true. Now, suppose that there exists a point z S such that z T (y s y) 0. Then, by choosing z = α z, where α = z T (y s y)/ z 2 2, one can verify that z zT (y s y) < 0 and yet z S. Thus, by contradiction, we must have z T (y s y) = 0 for all z S. Third, we show the necessity of Statement 2. Suppose that there exists a vector ȳ s S such that z T (ȳ s y) = 0 for all z S. The aforementioned condition can be rewritten as (z ȳ s ) T (ȳ s y) = 0, for all z S, where we have used the equivalence z ȳ s S z S. Now, for any z S, we have z y 2 2 = z ȳ s (z ȳ s ) T (ȳ s y) + ȳ s y 2 2 = z ȳ s ȳ s y 2 2 ȳ s y 2 2. (2) The above inequality implies that ȳ s minimizes z y 2 over all z S. This, together with the previous sufficiency proof, completes the proof of Statement 2. In addition, we note that the equality in (2) holds if and only if z ȳ s = z. This implies that ȳ s is the only minimizer of z y 2 over all z S. Thus, we also obtain the uniqueness claim in Statement 1.. Theorem 1.2 has many implications, e.g., in least squares and orthogonal projections which we will learn in later lectures. In the next section we will consider an application of Theorem 1.2 to subspaces. 5

6 4 Orthogonal Complements Let S a nonempty subset in R m. The orthogonal complement of S is defined as the set S = { y R m z T y = 0 for all z S }. It is easy to verify that S is always a subspace even if S is not. From the definition, we see that 1. any z S, y S are orthogonal, i.e., S consists of all vectors that are orthogonal to all vectors of S; 2. we have either S S = {0} or S S = ; i.e., if we exclude 0, the sets S and S are non-intersecting. The following theorem is a direct consequence of the projection theorem in Theorem 1.2. Theorem 1.3 Let S be a subspace of R m. 1. For every y R m, there exists a unique 2-tuple (y s, y c ) S S such that y = y s + y c. Also, such a (y s, y c ) is given by y s = Π S (y), y c = y Π S (y). 2. The projection of y onto S is given by Π S (y) = y Π S (y). Proof: Let us rephrase the problem in Statement 1 as follows: Given a vector y R m, find a vector y s such that y s S and y y s S. This problem is exactly the same as (1) in Theorem 1.2. Thus, by Theorem 1.2 the solution is uniquely given by y s = Π S (y). Also, we have y c = y Π S (y) S. To show Statement 2, let us consider the following problem: Given a vector y R m, find a vector ȳ c such that ȳ c S and y ȳ c (S ), or equivalently, ȳ c S, z T (y ȳ c ) = 0, for all z S. By Theorem 1.2, the solution is uniquely given by ȳ c = Π S (y). On the other hand, the above conditions are seen to satisfy if we choose ȳ c = y Π S (y) S. It follows that Π S (y) = y Π S (y). movmov Armed with Theorem 1.3, we can easily prove the following results. Property 1.1 The following properties hold for any subspace S R m : 1. S + S = R m ; 2. dim S + dim S = m; 3. (S ) = S. 1 1 We also have the following result: Let S be any subset (and not necessarily a subspace) in R m. Then, we have (S ) = span S, where span S is defined as the set of all finite linear combinations of points in S. 6

7 Proof: Statement 1 is merely a consequence of the first statement in Theorem 1.3. For Statement 2, let us assume S does not equal either {0} or R m ; the cases of {0} and R m are trivial. Let {u 1,..., v k } and {v 1,..., v l } be orthonormal bases of S and S, respectively; here note that k = dim S, l = dim S. We can write S + S = span{u 1,..., u k, v 1,..., v l }. Also, from the definition of orthogonal complements, it is immediate that u T i v j = 0 for all i, j. Hence, {u 1,..., u k, v 1,..., v l } is an orthonormal basis for S + S, and consequently we have dim(s + S ) = k + l. Moreover, since S + S = R m, we also have dim(s + S ) = m. The result m = dim S + dim S therefore holds. The proof of Statement 3 is as follows. One can verify from the orthogonal complements definition that y S implies y (S ). On the other hand, any y (S ) satisfies y = Π (S ) (y) = y Π S (y) = y (y Π S (y)) = Π S (y), where the first equation above is due to the equivalence y S y = Π S (y) (which is easy to verify), and the second equation above is due to the second statement of Theorem 1.3. The equality y = Π S (y) shown above implies that y S. Let us give an example on the application of Property 1.1. It is known that dim N (A) = n rank(a). The above result can be shown by Property 1.1. First, we use N (A) = R(A T ). Second, from the second result in Property 1.1 we have n = dim R(A T ) + dim R(A T ). Third, it is known that dim R(A T ) = rank(a T ) = rank(a). Combining these results together gives dim N (A) = n rank(a). References [1] D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, Mass., U.S.A., 2nd edition, [2] S. Boyd and L. Vandenberghe. Introduction to Applied Linear Algebra Vectors, Matrices, and Least Squares Available online at 7