Elementary linear algebra

Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The fundamental concepts are linear combination, linear dependence, basis, subspace. The material of this chapter can be found in Zhang (Chapter 1) and Halmos (Chapters I-III). Let F be a given field, usually we consider F = R, C Example 1.1.1 The set of all rational numbers, Q, is a field. Definition 1.1.2 A vector space V over the field F is a set of elements (vectors) with two operations + (addition) and (scalar multiplication) such that (A) (V, +) is an Abelian group, i.e., 1. commutative: x + y = y + x, for all x, y V. 2. associative: x + (y + z) = (x + y) + z, for all x, y, z V. 3. neutral element: there exists in V a unique vector 0 such that x + 0 = x for all x V. 4. inverse: for every x V, there is a unique vector x (called the inverse of x) such that x + ( x) = 0. (B) Scalar multiplication: α x or simply αx, α F, x V. 1. distributive: α(βx) = (αβ)x, for all α F, x V. 2. 1x = x, for all x V. (C) Distributive (scalar multiplication over addition) 1. α(x + y) = αx + αy, for all α F, x, y V. 1

2. (α + β)x = (αx + βx), for all α, β F, x V. Example 1.1.3 R n over R, C n over C or R, P = the set of all polynomial in the indeterminant t, P n = the set of polynomials in t of degree no greater than n 1, and F = the set of real valued functions. Remark 1.1.4 If F is replaced by a ring, V is called a module Group Representation Theory. Definition 1.1.5 (linear combination) Let x 1,..., x n V. The vector x = α 1 x 1 + α n x n, where α 1,..., α n F, is called a linear combination of x 1,..., x n. Example 1.1.6 1. The vector x = (1, 3) is a linear combination of x 1 = (1, 0) and x 2 = (1, 2) since (1, 3) = 1 2 (1, 0)+ 3 2 (1, 2) by direct observation or solving a 2 2 system of linear equations. 2. If A is an m n matrix and x = (x 1,..., x n ) T is an n 1 vector, the product Ax is indeed x 1 a 1 + +x n a n, a linear combination of the columns, a 1,..., a n, of A. Definition 1.1.7 (linear dependence and independence) A finite set of vectors S = {x 1,..., x n } is linearly dependent if α 1 x 1 + + α n x n = 0 where α 1,..., α n F are not all zero, i.e., there is a nontrivial zero combination among x 1,..., x n. Otherwise the set is linearly independent, i.e., α 1 x 1 + + α n x n = 0 implies that α 1,..., α n are all zero. Sometimes we say that x 1,..., x n are linearly independent or dependent instead of the set S. In case S V is infinite, we shall say that the set S is linearly independent if every finite subset of S is such. Otherwise, S is linearly dependent. Remark 1.1.8 1. Every set containing a linearly dependent set is also linearly dependent. Every subset of a linearly independent set is also linear independent. 2. The empty set of vectors in linearly independent. α 1 x 1 + + α n x n = 0 implies that α 1,..., α n are all zero can be interpreted as if α 1 x 1 + + α n x n = 0, then there is no index i for which α i 0. Exercise 1.1.9 Show that the three vectors x, x 1, x 2 in the last example are linearly dependent but any two of them are linearly independent. Theorem 1.1.10 (Halmos p.9) The set of nonzero vectors x 1,..., x n is linearly dependent if and only if some x k, 2 k n is a linear combination of the preceding ones. 2

Proof ( ) Let k be the first integer between 2 and n for which x 1,..., x k are linearly dependent (if worst comes worst, k = n will do it). Then α 1 x 1 + + α n x n = 0, for some α 1,..., α k not all zero. Now we cannot have α k = 0, otherwise we should have a linear dependence relation among x 1,..., x k 1, contrary to the definition of k. Hence x k = α 1 x 1 + + α k 1 x k 1. α k α k Thus we have the necessity. ( ) Since every set containing a linearly dependent set is also linearly dependent, we have the sufficiency. Corollary 1.1.11 The set of nonzero vectors x 1,..., x n is linearly dependent if and only if some x k is a linear combination of the others. Definition 1.1.12 Let S V be a subset of vectors. The span of S, denoted by span S, is the set of all linear combinations of finitely many vectors of S. The vector space V is spanned by S if span S = V. Example 1.1.13 Let V = R 3, S 1 = {(1, 0, 0), (0, 1, 0)}, and S 2 = {( 1, 2, 0), (0, 1, 0)}. Then span S 1 and span S 2 are the xy-plane and V is spanned by the set S 3 = {(1, 0, 0), (0, 1, 0), (0, 0, 1)}. Definition 1.1.14 A basis for a vector space V is a set B V of linearly independent vectors such that B spans V. V is finite dimensional if it has a finite basis. Remark 1.1.15 Infinite dimensional vector space normed spaces, Hilbert spaces, Banach spaces. The proof of the existence of a basis for an infinite dimensional vector space involves Zorn s lemma. We will only study finite dimensional vector spaces. From now on, if we do not specify, all vector spaces are finite dimensional. Theorem 1.1.16 If B = {x 1,..., x n } is a basis for V, then every x V can be written as x = α 1 x 1 + + α n x n and α 1,... α n F are uniquely determined by x. Proof Surely x can be expressed as a linear combination of x 1,..., x n since B is a basis. If x = α 1 x 1 + + α n x n = β 1 x 1 + + β n x n, then (α 1 β 1 )x 1 + + (α n β n )x n = 0. By the linear independence of B, we have α i = β i for all i = 1,..., n. The above theorem justifies the following 3

Definition 1.1.17 If x = α 1 x 1 +... + α n x n, the vector α 1 α 2 x B :=. Fn α n is called the coordinate vector of x with respect to the basis B = {x 1,..., x n }. Theorem 1.1.18 (Extension to a basis, Halmos p.11) If {x 1,..., x k } is a linearly independent set of a finite dimensional vector space V, then it can be extended to a basis, i,e., we can find vectors x k+1,..., x n V such that x 1,..., x k, x k+1,..., x n is a basis for V. Proof Since V is finite dimensional, there is a basis, say {y 1,..., y n }. We consider the set S of vectors x 1,..., x k, y 1,..., y n, in this order. Since S is linearly dependent (?), applying Theorem 1.1.10 to S, let y i be the first vector such that linear dependence occurs among x 1,..., x k, y 1,..., y i. Then consider the set S x 1,..., x k, y 1,..., y i 1, y i+1,..., y n. The deletion of y i would not reduce the span of the remaining vectors. If S is linearly independent, then we are done. Otherwise we apply Theorem 1.1.10 until we reach a basis. Theorem 1.1.19 (Halmos p.13) Every basis for a finite dimensional vector space V contains the same number of vectors. Proof Let {x 1,..., x n } and {y 1,..., y m } be two bases for V. Consider the set S of vectors y m, x 1,..., x n, which is clearly linearly dependent. Applying Theorem 1.1.10 to have the set S of vectors y m, x 1,..., x i 1, x i+1,..., x n, such that span S = V. Then add y m 1 in front of them and apply the same argument. Continuing in this way, the x s will not be exhausted before the y s, otherwise the remaining y s would have to be linear combinations of the ones already incorporated into S but y s are linearly independent. So n m. By symmetry m n. So n = m. The above theorem justifies the following 4

Definition 1.1.20 (dimension) The dimension of a finite dimensional vector space V is the number of vector in any basis for V, denoted by dim V. Corollary 1.1.21 If dim V = n, then any set of m vectors in V with m > n is linearly dependent. Proof By Theorem 1.1.18. Exercise 1.1.22 Show that dim R n = n, dim C n = n, dim P n = n by providing a basis for each. Show that P and F are infinite dimensional. Definition 1.1.23 A nonempty subset W V of a vector space is called a subspace of V if it is also a vector space under the same operations. Example 1.1.24 1. The subspaces {0} (will simply be denoted by 0) and V are called trivial subspaces of V. 2. Let V = R 3. Then any planes and any lines that pass through the origin are nontrivial subspaces. 3. If S V, then span S is a subspace of V. 4. The solution set of the system of linear equations Ax = 0 is a subspace of R n (C n ), where A is an m n real (complex) matrix. Exercise 1.1.25 Show that a nonempty set W V is subspace of V if and only if αu+βv W for all α, β F and u, v W, i.e., W is closed with respect to addition and scalar multiplication. Theorem 1.1.26 (Halmos p.17) The intersection of any (possibly infinite) collection of subspaces is a subspace. Proof By the preceding exercise. Exercise 1.1.27 Find an example to show that the union of subspaces may not be a subspace. Definition 1.1.28 If V 1, V 2 V are subspaces of V, then the sum of V 1 and V 2, denoted by V 1 + V 2, is the set V 1 + V 2 = {v 1 + v 2 : v 1 V 1, v 2 V 2 }. It is a subspace of V containing V 1 V 2. Exercise 1.1.29 Show that the sum V 1 + V 2 is the smallest subspace containing V 1 and V 2. Theorem 1.1.30 Let V 1 and V 2 be subspaces of V. dim V 1 + dim V 2 = dim(v 1 + V 2 ) + dim(v 1 V 2 ). 5

Proof Let B = {x 1,..., x k } be a basis for V 1 V 2. Extend B to a basis for V 1 : B 1 = {x 1,..., x k, x k+1,..., x k+i }, i.e., dim V 1 = k + i and extend B to a basis for V 2 : B 2 = {x 1,..., x k, x k+i+1,..., x k+i+j }, i.e., dim V 2 = k + j, via Theorem 1.1.18. We claim that B := B 1 B 2 = {x 1,..., x k+i+j } is a basis for V 1 + V 2. Clearly span B = V 1 + V 2 because span B 1 = V 1 and span B 2 = V 2 and span B 1 + span B 2 span B 2 (?). If α 1 x 1 + + α k+i+j x k+i+j = 0, then So α 1 x 1 + + α k+i x k+i = α k+i+1 x l+i+1 α k+i+j x k+i+j V 1 V 2 (?). α 1 x 1 + + α k+i x k+i = β 1 x 1 + + β k x k α k+i+1 x k+i+1 α k+i+j x k+i+j = β 1 x 1 + + β k x k for some scalars β s. From the first equation, we have β m = α m, m = 1,..., k, α m = 0, m = k + 1,..., k + i, and from the second equation, β m = 0, m = 1,..., k, α k+i+m = 0, m = 1,..., j (?). Thus all α s are zero. Definition 1.1.31 The sum V 1 + V 2 is called a direct sum if V 1 V 2 = 0 and will be denoted by V 1 V 2. It is called a direct sum of V if V 1 V 2 = V in addition. Exercise 1.1.32 Describe explicitly the sum of V 1 = span {(1, 0, 0)} and V 2 = span {(1, 1, 0), (0, 0, 1)}? Is it a direct sum of R 3? Theorem 1.1.33 If V 1 and V 2 are subspace of V. Then V = V 1 V 2 if and only if every x V is uniquely written in the form x = x 1 + x 2 where x 1 V 1, x 2 V 2. Proof ( ) Since V = V 1 V 2, each x V can be expressed as x = x 1 + x 2, x 1 V 1, x 2 V 2. Suppose x = y 1 +y 2 where y 1 V 1, y 2 V 2. Then x 1 +x 2 = y 1 +y 2 which implies that x 1 y 1 = y 2 x 2 V 1 V 2. So x 1 = y 1 and x 2 = y 2. ( ) Now we clearly have V = V 1 + V 2. It suffices to show that V 1 V 2 = 0. Let x V 1 V 2. Write x = 0 + x and x = x + 0. By the uniqueness of expression, x = 0 follows immediately. Theorem 1.1.34 Suppose V 1 V 2 is a direct sum. If B 1 = {x 1,..., x n } is a basis for V 1 and B 2 = {y 1,..., y m } is a basis for V 2, then B = {x 1,..., x n, y 1,..., y m } is a basis for V 1 V 2. Thus dim(v 1 V 2 ) = dim V 1 + dim V 2. Proof It is clear that span B = V 1 V 2. Suppose α 1 x 1 + + α n x n + β 1 y 1 + + β m y m = 0, for some scalars α s and β s. Rearranging yields α 1 x 1 + + α n x n = β 1 y 1 + β m y m 6

which is in V 1 V 2. So α s and β s are all zero (?). Thus B is linearly independent. Problems 1. Show that R n is a vector space over R but not over C. 2. Show that P n, the set of all polynomials (with complex coefficients) of degree less than or equal to n 1 together with the zero polynomial (we include it because the degree of the zero polynomial is not defined), is a vector space. What happens if complex coefficients are replaced by real coefficients? 3. Let V be a vector space. Prove that if x, y V and α F, then (a) 0 + x = x. (b) 0 = 0. (c) α 0 = 0. (d) 0 x = 0. (e) If αx = 0, then either x = 0 or α = 0. (f) x = ( 1)x. (g) y + (x y) = x (here x y = x + ( y)). 4. Under what conditions on the scalar α are the vectors (1 + α, 1 α) and (1 α, 1 + α) in C 2 linearly independent? What is the answer if C 2 is replaced by R 2. 5. Show that the vectors (1, 0) and (0, 1) and (1, 1) are linearly dependent by expressing one as a linear combination of the others. 6. Show that the vector x = (6, 7, 3) is a linear combination of the vectors x 1 = (2, 3, 1), x 2 = (3, 5, 2) and x 3 = (1, 1, 0). 7. Is the set {(x, y) R 2 : x 2y = 0} a subspace of R 3? How about {(x, y) R 2 : x 2y = 1}? 8. Let V be a vector space. Show that the following are bases for V. (a) any maximal linearly independent set B = {x 1,..., x n }, i.e., B is linearly independent and any set properly containing B would be linearly dependent, and (b) any minimal spanning set B = {x 1,..., x n }, i.e., the span of B is V and any proper subset of B would not span V. 7

9. (a) What is the dimension of C n over C? What is the dimension of R n over R? Is C n a vector space over R? If so, what is the dimension? (b) Every complex vector space V, i.e., over C, can be turned into a real vector space easily. How to achieve that and what is the dimension of V over R? 10. Show that the space of polynomials P in the variable t (i.e, p(t) := t 2 +3t+2 is an element of P) with complex coefficients is not finite dimensional by giving an infinite linearly independent set. 11. Prove that if W V is a subspace of the vector space V and dim W = dim V, then W = V. 12. Let V be a vector space and L, M, N V subspaces of V. (a) Show that the equation L (M +N) = (L M)+(L N) is not necessarily true by an example. (b) Prove that L (M + (L N)) = (L M) + (L N). 13. Suppose that V is vector space and V 1, V 2 V are subspaces. Show that V 1 V 2 = V if and only if dim V 1 + dim V 2 = dim V. 1.2 Linear maps and matrices Definition 1.2.1 Let U and V be vector spaces over the same field F. A linear map T : U V is a map that satisfies T (α 1 x 1 + α 2 x 2 ) = α 1 T (x 1 ) + α 2 T (x 2 ), for all x 1, x 2 U, α 1, α 2 F. When U = V, we call T : V V a linear operator or simply operator. Example 1.2.2 1. The zero map is a linear map. 2. The identity map I : V V is a linear operator. 3. A rotation on R 2 by an angle θ is a linear operator. 4. The differential operator d dt : P P is a linear map defined as d dt f = df dt, the derivative of f(t). Exercise 1.2.3 1. A0 = 0 for any linear map. 2. The set of all linear maps, L(U, V ), from U to V is a vector space. Show that if T : U V is an isomorphism, then the inverse T 1 : V U is linear. 8

Definition 1.2.4 The set Ker T = {x U : T (x) = 0} is called the kernel of T. The set Im T = {T (x) : x U} is called the range of T. Exercise 1.2.5 Show that Ker T and Im T are subspaces of U and V respectively. Theorem 1.2.6 Let T : U V be a linear map. The following are equivalent. 1. T is injective. 2. The kernel of T is 0. 3. T maps linearly independent set into linearly independent set, i.e., if {x 1,..., x n } is linearly independent, so is {T (x 1 ),..., T (x n )} Proof (1) (2): Clear. (2) (1): Suppose T (x 1 ) = T (x 2 ). Then T (x 1 x 2 ) = 0 and thus x 1 = x 2 and hence T is injective. (2) (3): Let {x 1,..., x n } be a linearly independent set. Let α 1 T (x 1 ) + + α n T (x n ) = 0. Then T (α 1 x 1 +... + α n x n ) = 0 (?). By (2), α 1 x 1 +... + α n x n = 0 and thus α s are all zero (?). So {T (x 1 ),..., T (x n )} is linearly independent. (3) (2): Suppose T (x) = 0. It implies that T (x) is linearly dependent and thus x is linearly dependent. So x = 0. Theorem 1.2.7 Let T : U V be a linear map. Then dim U = dim Ker T + dim Im T. Proof Let {x 1,..., x k } be a basis for Ker T. Extend it to a basis {x 1,..., x k, x k+1, x n } for U. The set {T (x k+1 ),..., T (x n )} is a basis for Im T since each element of Im T is of the form T (x) for some x = α 1 x 1 +... + α n x n. Now T (x) = α 1 T (x 1 ) +... + α n T (x n ) = α k+1 T (x k+1 ) + + α n T (x n ). If α k+1 T (x k+1 ) + + α n T (x n ) = 0, then T (α k+1 x k+1 + + α n x n ) = 0 and thus α k+1 x k+1 + + α n x n Ker T. So α s are all zero (?). Definition 1.2.8 An isomorphism T : U V is a bijective linear map and the spaces U and V are said to be isomorphic. In this case, dim U = dim V by the above theorem. Indeed U and V are isomorphic if and only if they have the same dimension. For if they have the same dimension, we fix (ordered) bases B 1 = {x 1,..., x n } and B 2 = {y 1,..., y n } for U and V respectively, and define the isomorphism T : U 9

V in the obvious way: T (x i ) = y i, i = 1,..., n and extend it by linearity, i.e., T ( n i=1 α i x i ) = n i=1 α i y i. In particular if V = F n and choose a basis B 2 = {e 1,..., e n } for F n. The natural isomorphism T sends n i=1 α i x i to n i=1 α i e i. An m n matrix A in F naturally induces a linear map T : F n F m by T (x) = Ax, x F n. Example 1.2.9 Let 1 2 A = 0 3. 2 1 Then T : R 2 R 3 is defined as T ( ) x1 x 2 = x 1 + 2x 2 3x 2 2x 1 x 2 To save space we often write T (x 1, x 2 ) T = A(x 1, x 2 ) T = (x 1 + 2x 2, 3x 2, 2x 1 x 2 ) T. Conversely if T : U V is a linear map, and if we fix bases B = {x 1,..., x n } and C = {y 1,..., y m } for U and V, respectively, we have. n T (x i ) = a ji y j, i = 1,..., m. j=1 Definition 1.2.10 (matrix representation) The m n matrix A = (a ij ), denoted by MB C (T ), is called the matrix representation of T with respect to the bases B and C, or simply called the matrix of T. If x = n k=1 α k x k, then n n n n n T (x) = α k T (x k ) = a jk α k y j = ( a jk α k )y j. k=1 k=1 j=1 j=1 k=1 Thus (T (x)) C = M C B (T )x B, for all x V, i.e., the coordinate vector of T (x) with respect to the basis C is the product of the the matrix of T with respect to the bases B and C and the coordinate vector of x with respect to B. Moreover MB C (T ) is uniquely determined by the above formula (?). 10

Theorem 1.2.11 Let S, T : U V be linear maps and let B, C be bases of U and V. Indeed MB C is an isomorphism defined by M B C(aS + bt ) = am B C(S) + bm B C (T ), a, b scalars, from L(U, V ) to F m n. Theorem 1.2.12 Let U, V, W be vectors spaces with bases B, C, D. Let S : U V and T : V W be linear maps. Then M D B (T S) = M D C (T )M C B (S). Proof Let A := MC D(T ) and E := M B C (S). Then for all x U, and the desired result follows. (T S(x)) D = (T (S(x)) D = A(S(x)) C = AEx B, Corollary 1.2.13 M C B (I)M B C (I) = M B C (I)M C B (I) = I, i.e., M C B (I) 1 = M B C (I). The matrix M(I) C B (the matrix representation of the identity operator I : V V with respect to B and C) is called the matrix of change of bases from B to C, which gives the relation between the coordinate vectors x B and x C of x V : x C = M(I) C Bx B. Exercise 1.2.14 Show that if S : V V is the linear map defined by Sx i = y i, i = 1,..., n, then M B B (S) = M(I)C B. The following offers the relationship between M(T ) B 2 B 1 and M(T ) C 2 C 1 if B 1 and C 1 are bases for U and B 2 and C 2 are bases for V. Theorem 1.2.15 M C 2 B 2 (T ) = M C 2 C 1 (I)M C 1 B 1 (T )(M B 2 B 1 (I)) 1. Corollary 1.2.16 If T : V V is an operator and B and C are bases for V, then M C C (T ) = M C B (I)M B B (T )(M C B (I)) 1. Definition 1.2.17 Two n n matrices A and B are similar if there exists a nonsingular matrix P such that B = P AP 1. Exercise 1.2.18 Show that similarity is an equivalence relation on F n n. Problems 1. Is the map T : R 2 R defined by T (x 1, x 2 ) T = x 2 1 + x 2 linear? 2. A linear operator P : V V is a projection if P 2 = P. Show that P is a projection if and only if Im P Im (I P ) = V. 11

3. Show that if V = V 1 V 2, then P x = x 1, x = x 1 + x 2, x 1 V 1, x 2 V 2, for each x V, defines a projection on V. 4. Given the map T : R 3 R 4 defined by T (x 1, x 2, x 3 ) T = (0, x 1 + x 2, x 3, x 1 x 3 ) T. (a) Show that T is a linear map. (b) Find its matrix representation with respect to the standard bases of R 3 and R 4. (c) Find Ker T and Im T. 5. Show that the following are equivalent. (a) T : V V is bijective. (b) Im T = V. (c) Ker T = 0. 6. Show that AB = I if and only if BA = I, where A and B are n n matrices (a) by showing that if (i) AB = I, then the columns of B are linearly independent by considering the system of equations Bx = 0, (ii) the systems Bc i = e i, i = 1,..., n have solutions, where {e 1,..., e n } is the standard basis of F n, (iii) BA = I. 1.3 Inner product and adjoint Inner product is a new structure given to a vector space that enables us to introduce the some geometric notions like length and angle. Definition 1.3.1 An inner product of a vector space V over R or C is a map (, ) : V V R such that for all x, y, z V and scalar α 1. (x, x) 0 and (x, x) = 0 if and only if x = 0. 2. (x + y, z) = (x, z) + (x, z). 3. (αx, y) = α(x, y). 4. (x, y) = (y, x). V is called an inner product space. The norm or length x of a vector x V is x = (x, x). Two vectors x, y V are orthogonal if (x, y) = 0. 12

Example 1.3.2 1. R n and C n are inner product spaces with the standard inner products (x, y) = y T x and (x, y) = y x respectively. 2. P has an inner product (f, g) = 1 0 f(t)g(t) dt. Exercise 1.3.3 Show that for all x, y V and scalar α 1. (x, αy) = α(x, y). 2. (i) x 0, (ii) αx = α x, (iii) (triangle inequality) x + y x + y. 3. (cosine law) x + y 2 = x 2 + y 2 + 2(x, y). Theorem 1.3.4 (Cauchy-Schwarz inequality) For all x, y in an inner product space V, (x, y) x y. Equality holds if and only if x and y are linearly dependent. Proof To avoid triviality we assume x 0. Consider z = y the projection of y orthogonal to x. Now (y, x) (x, x) x, 0 (z, z) = (z, y) = (y, y) (y, x) (x, y), (x, x) and the desired Cauchy-Schwarz inequality follows. If equality holds, it implies (z, z) = 0 and hence y is a scalar multiple of x, i.e., linearly dependent. Exercise 1.3.5 Picture the geometry of the proof. Definition 1.3.6 Let x, y V be nonzero vectors. Then the cosine of the angle θ (0 θ π) between x and y is cos θ = (x, y) x y. Thus the cosine law becomes x + y 2 = x 2 + y 2 + 2 x y cos θ. Theorem 1.3.7 (Gram-Schmidt process) Let {x 1,..., x k } be a linearly independent set in V, the there is an orthogonal set {y 1,..., y k } such that span {x 1,..., x i } = span {y 1,..., y i }, i 1,..., k. Thus every inner product space V has an orthonormal basis. 13

Proof Let y 1 = x 1. Set In order to have (y i, y j ) = 0 for j i, y i = x i α 1 y 1 α i 1 y i 1. α j = (x i, y j )/(y j, y j ), j = 1,..., i 1. It is clear that the span of x s and the span of y s are identical. Since {x 1,..., x k } is linearly independent, y s are nonzero. Normalization yields orthonormal vectors. Definition 1.3.8 Let S V be a subset. The orthogonal complement of S is the set S := {x : (x, y) = 0 for all y S}. Exercise 1.3.9 Show that S is a subspace of V. Theorem 1.3.10 Let W V be a subspace. Then dim W + dim W = dim V. Proof If W = 0 or V, then it is trivial. Suppose W V is a proper subspace of V. Let {x 1,..., x k } be an orthonormal basis for W. Extend it to a basis {x 1,..., x k, x k+1,..., x n}. Apply Gram-Schmidt process to get an orthonormal basis {x 1,..., x k, x k+1,..., x n }. Then {x k+1,..., x n } is a basis of W (check!). Theorem 1.3.11 Let T : V V be a linear operator on the inner product space. There exists a unique T : V V such that (T x, y) = (x, T y), called the adjoint of T. Proof It suffices to show that for any y V, there exists a unique element z V (depends on y and T ) such that (T x, y) = (x, z) for all x V. Let {x 1,... x n } be an orthonormal basis of V. We define n z := (T x i, y)x i. i=1 Suppose (T x, y) = (x, z 1 ) = (x, z 2 ), then (x, z 1 z 2 ) = 0 for all x V and thus z 1 = z 2 by putting x = z 1 z 2. Exercise 1.3.12 1. Prove (T ) = T. 2. Prove that (T ) 1 = (T 1 ) if T is bijective. Definition 1.3.13 1. T is normal if T T = T T. 2. T is self-adjoint if T = T. 14

3. T is positive semi-definite if (T x, x) 0 for all x V and positive definite if (T x, x) > 0 for all nonzero x. 4. T is an isometry if T T = I, i.e., (T x, T y) = (x, y) for all x, y V. Exercise 1.3.14 Show that positive semi-definite operator are self-adjoint. (Hint: consider (A(x + y), x + y) and (A(x + iy), x + iy) if F = C. Complexify V if F = R) Definition 1.3.15 Let A be an n n matrix. 1. A is normal if A A = AA. 2. A is Hermitian if A = A and is real symmetric if A is real and A T = A. 3. A is positive semi-definite if x Ax 0 for all x F n and positive definite if x Ax > 0 for all nonzero x. 4. A is unitary if A A = I and is orthogonal if A is real and A T A = I. Example 1.3.16 (matrix representation of adjoint) Let B = {x 1,..., x n } be an orthonormal basis for V. If T : V V is a linear operator. Let A = (a ij ) = MB B (T ), i.e., T (x j ) = n i=1 a ij x i, j = 1,..., n. Since B is orthonormal, we have a ij = (T x j, x i ). Therefore, a ji = (T x i, x j ) = (x i, T x j ) = (T x j, x i ). Thus M B B (T ) = (a ji ), the conjugate transpose of the matrix representation T. Exercise 1.3.17 1. Show that if T is normal, positive semi-definite, positive definite, unitary, respectively, so is the matrix representation of T with respect to an orthonormal basis. i.e., Definition 1.3.15 is just the matrix version of Definition 1.3.13 in the sense of the previous example. 2. Deduce that positive semi-definite matrices are Hermitian. Exercise 1.3.18 Show that a matrix A is unitary if and only if the columns (rows) of A are orthonormal with respect to the standard inner product of F n. Exercise 1.3.19 What is the matrix version of Exercise 1.3.12? Problems 1. Prove that if x and y are in a complex inner product space, then 4(x, y) = x + y 2 x y 2 + i x + iy 2 i x iy 2. 2. (Parallelogram identity) Show that x + y 2 + x y 2 = 2 x 2 + 2 y 2. 15

3. *(Jordan-von Neumann) Let V be a complex vector space with a norm. Then there exists an inner product (, ) on V such that x = (x, x) if and only if it satisfies the parallelogram law. 4. (a) Let V be a real inner product space. Show that x, y V are orthogonal if and only if x + y 2 = x 2 + y 2. (b) Let V be a real inner product space. If x = y, then x y and x + y are orthogonal. (c) Show that (a) is false if V is a complex vector space. (d) Let V be a complex inner product space. Show that x, y V are orthogonal if and only if αx + βy 2 = αx 2 + βy 2 for all α, β C. 5. Let x 1,..., x n be n unit vectors of a real inner product space V such that x i x j = 1 for all i j. Find the angle between x i and x j and the angle between s x i and s x j where x = (x 1 + + e n )/(n + 1). Find a geometric interpretation of the vectors for n = 2 and n = 3. 6. Let C n n be the vector space of complex n n matrices. Show that (A, B) = tr AB is an inner product on C n n where tr A = n i=1 a ii is the trace of A. How about R n n and C m n, (m n)? 7. Apply Gram-Schmidt process to the three vectors (1, 1, 1), ( 1, 2, 2) and (1, 4, 0). 8. Let V be an inner product space with inner product (, ). Let, be an inner product of V. Then there exists a unique positive definite operator T such that x, y = (T x, y), for all x, y V. What is the matrix version for C n with standard inner product (, )? 9. Let V be a complex inner product space. Show that if T : V V such that (T x, x) = 0 for all x V, then T = 0. Is it true if V is over R? 10. Given T : V V a linear operator on V. Show that the following are equivalent. (a) T is an isometry. (b) T maps an orthonormal basis into an orthonormal basis. (c) T preserves norm, i.e., T x = x for all x V. 11. Show that the set of n n Hermitian (real symmetric) matrices is a vector space. Is it true for unitary matrices and normal matrices? 12. Show that the set of n n unitary (orthogonal) matrices is a group. 16

13. (orthogonal projection) Let V = V 1 V 2 be an orthogonal sum, i.e., V 1 V 2. Show that the map P : V V 1 defined by P x = x 1 if x = x 1 + x 2 where x 1 V 1, x 2 V 2, is an orthogonal projection, i.e., P 2 = P and P = P. Conversely prove that if P is an orthogonal projection, then V = Im P Im (I P ) is an orthogonal direct sum. 14. (reflection) Let r V be a nonzero vector in the real inner product space V. Show that the map s r : V V defined by (x, r) s r (x) = x 2 (r, r) r is a unitary operator. What is s r (r)? and what is the image of x V such that x r. What is the geometric interpretation? 1.4 Some numerical characteristics of matrices Let A be an m n matrix. The image and the kernel of A are the subspaces Im A = {Ax : x F n } of F m and Ker A = {x F n : Ax = 0} of F n respectively. The rank of A is rank A = dim Im A. Theorem 1.4.1 Let A and B be matrices of sizes m n and n p, respectively. Then rank AB = rank B dim(im B Ker A). In particular rank A + rank B n rank AB min{rank A, rank B}. A scalar λ is an eigenvalue of an n n A if there exists a nonzero vector x such that Ax = λx, or equivalently (A λi)x = 0 has nontrivial solution. To compute the eigenvalues of A, we solve the characteristic equation det(λi A) = 0. By the fundamental theorem of algebra, an n n complex matrix has n eigenvalues in C. Let A be an m n matrix. The square roots of the eigenvalues of A A are called the singular values of A. Problems Let A be an m n matrix. Show that for any n m matrix B,. dim Im A + dim Ker A = dim Im BA + dim Ker BA 17

Chapter 2 Matrix decompositions 2.1 Schur s triangularization theorem Two n n matrices are unitarily similar if B = UAU 1 for some unitary matrix U. Unitarily similarity is an equivalence relation on F n n. Theorem 2.1.1 (Schur s triangularization theorem) Let A be an n n complex matrix. Then there exists a unitary matrix U such that UAU 1 is upper triangular. If A is real, then U may be chosen as real orthogonal. Theorem 2.1.2 (Spectral theorem for normal matrices) Let A be an n n normal matrix. Then there exists a unitary matrix U such that UAU 1 = diag (λ 1,..., λ n ) where λ s are the eigenvalues of A. If A is real normal, then U may be chosen as real orthogonal. Corollary 2.1.3 (Spectral theorem for Hermitian matrices) Let A be an n n Hermitian matrix. Then there exists a unitary matrix U such that UAU 1 = diag (λ 1,..., λ n ) where λ s in R are the eigenvalues of A. If A is real symmetric, then U may be chosen as real orthogonal. Corollary 2.1.4 (Spectral theorem for unitary matrices) Let A be an n n unitary matrix. Then there exists a unitary matrix U such that UAU 1 = diag (λ 1,..., λ n ) where λ s of modulus 1 are the eigenvalues of A. If A is real orthogonal, then U may be chosen as real orthogonal. Two m n matrices are unitarily equivalent if B = UAV for some nonsingular matrices U and V. Unitarily equivalence is an equivalence on F m n. Theorem 2.1.5 (singular value decomposition SVD) Let A be an m n complex matrix. Then there exist unitary matrices U (m m) and V (n n) such that 18

A = USV where S = { [diag (s1,..., s r, 0,..., 0) 0] if m n ( diag (s 1,..., s r, 0,..., 0) 0 ) if m > n. If A is real, then U and V may be chosen as real orthogonal. Corollary 2.1.6 Let A be an n n complex matrix. There exist positive semidefinite matrices P 1, P 2 and unitary matrices U 1 and U 2 such that A = U 1 P 1 = P 2 U 2. Two m n matrices are equivalent if B = P AQ for some nonsingular matrices P and Q. Exercise 2.1.7 Show that equivalence is an equivalence relation on F m n. Theorem 2.1.8 Let A be an m n matrix with ranka = r. nonsingular matrices S and T such that (I ( r 0 m r 0) ) if m n SAT = Ir 0 m r if m > n 0 Then there exist The form is call the Hermite form of A which is completely determined by the rank of A. 19