# Math 350 Fall 2011 Notes about inner product spaces. In this notes we state and prove some important properties of inner product spaces.

Save this PDF as:
Size: px
Start display at page:

Download "Math 350 Fall 2011 Notes about inner product spaces. In this notes we state and prove some important properties of inner product spaces."

## Transcription

1 Math 350 Fall 2011 Notes about inner product spaces In this notes we state and prove some important properties of inner product spaces. First, recall the dot product on R n : if x, y R n, say x = (x 1,..., x n ) and y = (y 1,..., y n ), then their dot product is given by x y = x 1 y x n y n. The dot product is very useful in understanding geometry in R n. For instance, one can compute the length of a vector, the distance between two points, and the angle between two vectors using the dot product. What we will do now can be thought of as an effort to generalize the dot product to an abstract vector space. The generalization will be called an inner product. 1. Real inner product spaces Suppose V is a vector space over R. A (real) inner product on V is a map, : V V R (i.e. an association of a number v, w R for every v, w V ) that satisfies the following axioms: (i) v, v 0 for all v V ; (ii) v, v = 0 if and only if v = 0; (iii) v 1 + v 2, w = v 1, w + v 2, w for all v 1, v 2, w V, and cv, w = c v, w for all v, w V and all c R; (iv) v, w = w, v for all v, w V. A vector space over R, with a real inner product, is called a real inner product space. For instance, the dot product on R n is a real inner product on R n. (Check!) In fact one should think of an inner product as a generalization of the dot product. Natural examples of real inner products arise in the study of abstract vector spaces over R. For instance, if V = C[a, b], where C[a, b] is the vector space over R of all real-valued continuous functions defined on an interval [a, b], then a real inner product on V is defined by f, g = b a f(x)g(x)dx for all f, g V. More generally, if h is a (fixed) positive continuous function on [a, b], then one can define a real inner product on C[a, b] by f, g = b f(x)g(x)h(x)dx a 1 for all f, g C[a, b].

2 2 (So there are in general many possible inner products on a given vector space; one should always specify what inner product one uses unless there is a standard one and that it is understood that the standard one is being used.) Also, if V = M n n (R), the vector space of all n n matrices with real entries, then a real inner product on V can be defined by A, B = Trace(B t A). One should check that the above are examples of real inner products by using the axioms. We mention some properties of real inner products. Suppose now V is a real inner product space, with inner product,. 1. If w V is fixed, and one defines a map T : V R such that T (v) = v, w, then T is a linear map. This is evident from Axiom (iii) above. 2. We have and w, v 1 + v 2 = w, v 1 + w, v 2 for all v 1, v 2, w V, w, cv = c w, v for all v, w V and c R. This is because w, v 1 +v 2 = v 1 +v 2, w by Axiom (iv); then one applies Axiom (iii), and use Axiom (iv) again to conclude that w, v 1 +v 2 = v 1, w + v 2, w = w, v 1 + w, v 2 as desired. Similarly one can prove the assertion about w, cv. It follows that if w V is fixed, then the map S : V R, defined by S(v) = w, v, is also a linear map. 3. It follows from the above two properties that 0, w = w, 0 = 0 for all w V. One way of seeing this is to quote the theorem that says every linear map applied to the zero vector yields zero; another way is to repeat that proof, and observe for instance that 0, w = 0 + 0, w = 0, w + 0, w = 2 0, w, which implies 0, w = If one wants to show that a vector v V is zero, one just need to show that v, v = 0 by Axiom (ii). 5. Alternatively, we have the following little, but remarkably useful, lemma: Lemma 1. Suppose v V is such that v, w = 0 for all w V. Then v = 0. The proof is one line: just take w = v. Then v, v = 0, which implies v = 0 as we have just observed. 6. Recall that two vectors in R n are said to be orthogonal to each other if their dot product is zero. This suggests the following terminology in our setting: Two vectors v, w V are said to be orthogonal to each other if and only if v, w = 0. A way of remembering the above lemma is then to say that the zero vector is the only vector that is orthogonal to every vector in V.

3 3 2. Complex inner product spaces Next, we look at the situation where the scalars are complex numbers rather than real numbers. The definition of an inner product in this setting is very similar to the one we had when the vector spaces are over R. However, as we will point out below, there will be some subtle (but important) differences, and one should always be careful about these. First, let s review the standard dot product in C n. If we denote this dot product by (, ), then (z, w) = z 1 w z n w n if z = (z 1,..., z n ) and w = (w 1,..., w n ). Note that one has the complex conjugates over the w j s. The complex conjugates were there because one would like to have (z, z) = z 2 0 for all z C n, where z is the Euclidean norm of z, defined by z = ( z z n ) 1/2 ; if the complex conjugates were absent from the definition of (z, w), then it would no longer be true that (z, z) = z 2. On the other hand, the presence of the complex conjugates over the w j s leads to the following fact: if z, w C n and if c C, then (z, cw) = c(z, w). In other words, one cannot simply pull a scalar out of the second argument of this dot product ; one has to put a complex conjugate over the scalar if one pulls it out from the second argument of the dot product. On the other hand, it is still true that (cz, w) = c(z, w) for all z, w C n and all c C. Thus the first and second argument of the dot product in C n behaves slightly differently. In general, if V is a vector space over C, when we define a complex inner product on V, one would like to build this difference into our definition. Thus we are led to the following. Suppose now V is a vector space over C. A (complex) inner product on V is a map, : V V C (i.e. an association of a number v, w C for every v, w V ) that satisfies the following axioms: (i) v, v 0 for all v V (in particular v, v is real for every v V ); (ii) v, v = 0 if and only if v = 0; (iii) v 1 + v 2, w = v 1, w + v 2, w for all v 1, v 2, w V, and cv, w = c v, w for all v, w V and all c C; (iv) v, w = w, v for all v, w V. A vector space over C, with a complex inner product, is called a complex inner product space. For instance, the dot product on C n is a complex inner product on C n. (Check! For instance, Axiom (iv) can be checked by beginning with the right hand side of the identity, and observing that (w, z) = w 1 z w n z n = w 1 z w n z n = z 1 w z n w n = (z, w)

4 4 for all z, w C n.) Before we turn to more examples of complex inner products, let s make some observations first. Suppose now V is a complex inner product space, with inner product,. 1. If w V is fixed, and one defines a map T : V C such that T (v) = v, w, then T is a linear map. This is evident from Axiom (iii) above. 2. We have w, v 1 + v 2 = w, v 1 + w, v 2 for all v 1, v 2, w V, and w, cv = c w, v for all v, w V and c C. These statements are very similar to the corresponding statements we had about real inner products, except for the complex conjugate over the c in the second identity. In fact, one can prove the first statement about w, v 1 + v 2 in exactly the same way we did for the real inner product; we leave that to the reader. On the other hand, suppose v, w V and c C. Then we have w, cv = cv, w = c v, w = c v, w = c w, v, proving the second identity. It follows that if w V is fixed, then the map S : V C, defined by S(v) = w, v, is NOT a linear map, contrary to the case of real inner products. (It is sometimes called a conjugate linear map, because S(v 1 + v 2 ) = S(v 1 ) + S(v 2 ) and S(cv) = cs(v), but we will not need this piece of terminology.) 3. It follows from the above two properties that 0, w = w, 0 = 0 for all w V. One way of seeing this is to quote the theorem that says every linear map applied to the zero vector yields zero; thus 0, w = 0, from which it follows that w, 0 = 0, w = 0 = 0. Another way is to do this directly, and observe for instance that w, 0 = w, = w, 0 + w, 0 = 2 w, 0, which implies w, 0 = 0. The next three properties works exactly the same way they did in a real inner product space: 4. If one wants to show that a vector v V is zero, one just need to show that v, v = 0 by Axiom (ii). 5. Alternatively, again we have the following little, but remarkably useful, lemma: Lemma 2. Suppose v V is such that v, w = 0 for all w V. Then v = 0. The proof is the same as in the case of real inner product spaces: just take w = v. Then v, v = 0, which implies v = 0 as we have just observed. 6. Recall that two vectors in C n are said to be orthogonal to each other if their dot product is zero. This suggests the following terminology in our setting: Two vectors v, w V are said to be orthogonal to each other if and only if v, w = 0.

5 5 A way of remembering the above lemma is then to say that the zero vector is the only vector that is orthogonal to every vector in V. Now we turn to some natural examples of complex inner product spaces. The first example is the following. Suppose V is the vector space over C of all complex-valued continuous functions defined on an interval [a, b], then a complex inner product on V is defined by f, g = b a f(x)g(x)dx for all f, g V. More generally, if h is a (fixed) positive continuous function on [a, b], then one can define a complex inner product on V by f, g = b a f(x)g(x)h(x)dx for all f, g V. Also, if V = M n n (C), the vector space of all n n matrices with complex entries, then a complex inner product on V can be defined by A, B = Trace(B A). Here B is the conjugate transpose of B; in other words, B is what one obtains if one first take the transpose of B, and then put complex conjugates over every entry entry of the resulting matrix. One should check that the above are examples of complex inner products by using the axioms. Note that the complex conjugate is always somehow involved in the second argument of the complex inner product in these examples. A piece of notation: Many definitions and results below hold for both real and complex inner product spaces. If we do not want to emphasize whether it is a real or complex inner product space, we just say inner product space. If there is a difference between real and complex inner product spaces, we will say so explicitly. All inner product spaces we have will be over R or C, and on R n and C n, we will always use the dot product as our (standard) inner product unless stated otherwise. 3. Norms, Distances and Angles If V is a inner product space, with inner product,, then the length of a vector v in V is by definition v = v, v. For instance, if V = R n with the standard inner product, and if v R n, then v is the usual Euclidean norm of the vector v. Similarly if V = C n with the standard inner product. In general, if V is a inner product space, with inner product,, and if is the norm associated to the inner product, then we have: (i) v 0 for all v V ;

6 6 (ii) v = 0 if and only if v = 0; (iii) cv = c v for all v V and for all scalars c; (iv) For any v, w V, we have (1) v + w v + w. The first three properties are very easy to verify using the definition of, and we leave it to the reader. The last one says the length of the sum of two vectors is less than or equal to the sum of the lengths of the two vectors. In R n or C n this is commonly known as the triangle inequality, because it asserts that the length of any side of a triangle cannot exceed the sum of the lengths of the other two sides. We will adopt the same terminology, and call (iv) above the triangle inequality. The proof of the triangle inequality makes use of the following Cauchy-Schwarz inequality: (2) v, w v w for all v, w V. We defer the proofs of these until a point where we would have a better conceptual understanding of the proof of these inequalities. (c.f. Section 8.) A function : V R satisfying Properties (i) to (iv) above are usually called a norm on V. It is not a linear function since in general v + w v + w. The above says that if V is an inner product space, then the inner product on V defines a natural norm on V. The study of a general real or complex vector space with a norm is an interesting topic in itself, but we will not go into that. Now given two points v, w V, we define the distance between v and w to be d(v, w) = v w. This definition makes sense whenever one has a norm on V, but the reader may feel free to assume that the norm arises from an inner product on V just for the sake of concreteness. It follows from the Properties (i) to (iv) of the norm above that (i) d(v, w) 0 for all v, w V ; (ii) d(v, w) = 0 if and only if v = w; (iii) d(v, w) = d(w, v) for all v, w V ; (iv) d(u, w) d(u, v) + d(v, w) for all u, v, w V. A function d(, ): V V R satisfying these properties is usually called a metric on V. The above says that a norm on a vector space V will naturally define a metric on V. Example 1. Suppose V = C[0, 1], the vector space of all continuous real-valued functions on [0, 1]. Define an inner product on V by f, g = 1 0 f(x)g(x)dx for all f, g V. Define the norm and the distance using the inner product. Compute x, e 2x, x, e 2x and check that x, e 2x x e 2x in this case. Also, compute the distance between x and e 2x.

7 7 Recall that on R n, if θ [0, π] is the angle between two non-zero vectors v and w, then v, w = v w cos θ. It follows that ( ) v, w θ = cos 1. v w Motivated by this, we make the following definition. Suppose V is a real inner product space, with inner product,. Suppose is the norm associated to the given inner product. For any non-zero vectors v, w V, we define the angle between v and w to be cos 1 ( v, w v w This defines an angle in [0, π] since we are taking the arc-cosine of a real number between 1 and 1, thanks to the Cauchy-Schwarz inequality (2). We then recover the identity v, w = v w cos(θ) in any real inner product space, if v, w are any non-zero vectors in V and θ is the angle between them. Example 2. Following the notations of Example 1 above, compute the angle between x and e 2x. Recall that two vectors v, w in an inner product space are said to be orthogonal to each other if and only if v, w = 0. It follows that if v, w are non-zero vectors in a real inner product space, then they are orthogonal to each other if and only if the angle between them is π/2. This is in turn the motivation for the definition of orthogonality. From now on, will always denote the norm defined by an inner product. ). 4. Orthogonal and orthonormal sets Suppose now V is an inner product space (real or complex). A set of vectors {v 1,..., v k } in V is said to be orthogonal if v i, v j = 0 for all i j. It follows that {v 1,..., v k } is orthogonal, if and only if any two distinct vectors from the set is orthogonal to each other. Furthermore, a set of vectors {v 1,..., v k } in V is said to be orthonormal, if first it is orthogonal, and second the length of every vector in the set is equal to 1. For example, the standard basis in R n or C n are both orthogonal and orthonormal; the set {(1, 1), (1, 1)} in R 2 is orthogonal but not orthonormal. Orthogonal sets of vectors enjoy a lot of good properties. For example, one has the Pythagoras theorem: Proposition 1. If v, w V and {v, w} is orthogonal, then v + w 2 = v 2 + w 2.

8 8 In fact, to prove this, one simply observes that v + w 2 = v + w, v + w = v, v + v, w + w, v + w, w, and the last expression in this identity is v 2 + w 2 since v, w = 0 = w, v. More generally, we have: Proposition 2. If the set {v 1,..., v k } is orthogonal, then v v k 2 = v v k 2. In particular, if the set {v 1,..., v k } is orthonormal, and if v = a 1 v a k v k, then v 2 = a a k 2. Next, we have: Proposition 3. Suppose S = {v 1,..., v k } is an orthonormal set of vectors. Suppose v is in the span of S. We claim that and v = v, v 1 v v, v k v k v 2 = v, v v, v k 2. In fact, any v in the span of S can be written as v = a 1 v a k v k for some scalars a 1,..., a k. By taking the inner product of this equation with v j, we get v, v j = a 1 v a k v k, v j = a 1 v 1, v j + + a k v k, v j. Since v i, v j = 0 whenever i j, and v j, v j = v j 2 = 1 2 = 1, we get v, v j = a j, and by plugging this back into the equation v = a 1 v a k v k, we obtain the first conclusion of the proposition. The second conclusion in the proposition follows from the first one and the previous proposition. The second conclusion is sometimes called Parseval s identity. Corollary 1. Suppose V is an inner product space, and {v 1,..., v n } is an orthonormal set that forms a basis of V. Then for any v V, we have and v = v, v 1 v v, v n v n v 2 = v, v v, v n 2. This is because then any v V is in the span of {v 1,..., v n }, and we can apply the previous proposition. A basis of V that forms an orthonormal set is called an orthonormal basis of V. Thus by this corollary, in an inner product space, it is particularly easy to express a vector in terms an orthonormal basis. Since it is advantageous to work with orthonormal basis, a natural question is then the following: when is an orthonormal set in V a basis of V? We have the following proposition:

9 9 Proposition 4. If {v 1,..., v k } is orthonormal, then it is linearly independent. In fact, suppose {v 1,..., v k } is orthonormal. If a 1,..., a k are scalars such that a 1 v a k v k = 0, then taking inner product of this with v 1, we get a 1 = 0; similarly, taking inner product with v 2, we get a 2 = 0. In fact, by taking inner product with v j, we get a j = 0 for all j = 1,..., k. Thus {v 1,..., v k } is linearly independent. We then have the following corollaries: Corollary 2. If {v 1,..., v k } is orthonormal and spans V, then {v 1,..., v k } is a basis of V. Corollary 3. If {v 1,..., v k } is orthonormal and dim(v ) = k, then {v 1,..., v k } is a basis of V. Example 3. (a) Show that {(1, 0, 1, 0), (0, 1, 0, 1), (1, 0, 1, 0)} is an orthogonal set of vectors in R 4. (b) Let W be the span of these vectors. Find an orthonormal basis of W. Example 4. Suppose W = P 2 (R), the vector space of all polynomials of degree 2 with real coefficients. Define an inner product on W by p, q = p(x)q(x)dx. Let u 1 = 1, u 2 = 3x, and u 3 = 5(3x 2 1)/2. Show that {u 1, u 2, u 3 } is an orthonormal basis of W. It will be shown that every finite dimensional inner product space admits an orthonormal basis. To do that, we need to first study orthogonal projections. 5. Orthogonal projections Suppose V is an inner product space over a field F, where F = R or C. Suppose v, w V and w 0. We define a function from F to R by c v cw, and ask when this function achieves its minimum as c varies over F. The answer is given by the following proposition: Proposition 5. Suppose v, w V and w 0. Then the function c v cw achieves a global minimum at only one point, namely at (3) c = v, w w, w. Proof. Since the minimum of v cw is achieved precisely when the minimum of v cw 2 is achieved, we look at the latter function instead. Now suppose first F = R. Then v cw 2 = v cw, v cw = v, v + 2c v, w + c 2 w, w

10 10 is a quadratic polynomial in a real variable c, and the coefficient of c 2 is positive. Thus v cw 2 achieves a global minimum; in fact d dc v cw 2 = d dc ( v, v 2c v, w + c2 w, w ) = 2 v, w + 2c w, w, so setting this equal zero, we see that the minimum of v cw 2 occurs precisely when 2 v, w + 2c w, w = 0, i.e. when c = v, w w, w. The case when F = C can be dealt with similarly, except now one has a minimization problem involving one complex variable (or two real variables). We leave the details to the reader. Next, we observe that (3) holds if and only if v cw, w = 0, i.e. if and only if v cw is orthogonal to w. Motivated by this, we call cw the orthogonal projection of v onto w, if and only if v, w c = w, w. The above proposition then says the orthogonal projection of v onto w is the closest point to v on the span of w. More generally, suppose V is an inner product space (real or complex), and suppose W is a finite dimensional subspace of V. Proposition 6. For any v V, there exists one and only one vector π(v) W such that v π(v), w = 0 for all w W. If {w 1,..., w k } is orthogonal and form a basis of W, then π(v) is given by where c 1,..., c k are scalars such that π(v) = c 1 w c k w k c 1 = v, w 1 w 1, w 1,..., c k = v, w k w k, w k. In addition, the function from W to R, defined by w v w, achieves a global minimum at only one point, namely when w = π(v). We will call this π(v) the orthogonal projection of v onto W. In particular, if {w 1,..., w k } is an orthonormal basis of W, then the orthogonal projection of v onto W is given by v, w 1 w v, w k w k. Proof of Proposition 6. Suppose v V is given, and {w 1,..., w k } is an orthogonal set that forms a basis of W. To prove the uniqueness assertion in the proposition, suppose v 0 is in W and v v 0, w = 0 for all w W. We will determine v 0 explicitly. Since {w 1,..., w k } is a basis of W, we can write v 0 = c 1 w c k w k

11 11 for some scalars c 1,..., c k. Now by assumption, v v 0, w = 0 for all w W. This implies v (c 1 w c k w k ), w j = 0 for all j = 1,..., k. It follows that v, w j c j w j, w j = 0 for all j = 1,..., k. Thus c j = v, w j for all j = 1,..., k. w j, w j This proves that v 0 is uniquely determined, and proves the uniqueness assertion in the proposition. On the other hand, if v V is given, and if one defines π(v) to be c 1 w c k w k, where c j = v, w j for all j = 1,..., k, w j, w j then it is easy to show that v π(v), w j = 0 for all j = 1,..., k. Since {w 1,..., w k } is a basis of W, this shows that v π(v), w = 0 for all w W. This proves the existence part of the proposition, and proves the given formula for π(v). A multivariable minimization gives the last part of the proposition, which we omit. Example 5. Recall that in Example 3, W is the subspace of R 4 spanned by the orthogonal set {(1, 0, 1, 0), (0, 1, 0, 1), (1, 0, 1, 0)}. Compute the orthogonal projection of (1, 2, 3, 4) onto W. Example 6. Let V = P 4 (R), the vector space of all polynomials of degree 4 with real coefficients. Define an inner product on V by p, q = p(x)q(x)dx. Recall that in Example 4, if W = P 2 (R) is a subspace of V, then u 1 = 1, u 2 = 3x, and u 3 = 5(3x 2 1)/2 form an orthonormal basis of W. If p(x) = x 4 V, find the orthogonal projection of p(x) onto W. Corollary 4 (Bessel s inequality). Suppose V is an inner product space (real or complex), and that {v 1,..., v k } is an orthonormal set in V. Then for any v V, we have v 2 v, v v, v k 2, with equality holding if and only if v is in the span of {v 1,..., v k }. Proof. In fact, if W is the span of {v 1,..., v k }, and w is the orthogonal projection of v onto W, then v w, w = 0, which implies by the Pythagoras theorem (Proposition 1) that But v 2 = v w 2 + w 2 w 2. w = v, v 1 v v, v k v k,

12 12 and thus by Pythagoras theorem again (c.f. Proposition 2), w 2 = v, v v, v k 2. Together this proves the desired inequality. The equality holds in the inequality if and only if v w 2 = 0, i.e. if and only if v = w. This is equivalent to saying that v is in W, i.e. that v is the span of {v 1,..., v k }. 6. Orthogonal complements Suppose V is an inner product space. If W is a subspace of V, the orthogonal complement of W in V is the set of all vectors in V that are orthogonal to every vector in W, and is usually denoted as W. In other words, W = {v V : v, w = 0 for all w W.} Example 7. Find the orthogonal complement of the span of {(1, 0, i, 1), (0, 1, 1, i)} in C 4. Example 8. Suppose V = C[a, b] is equipped with the inner product f, g = b f(x)g(x)dx as in Section 1. If W is the vector subspace of all constant functions in V, show that the orthogonal complement of W consists of all functions a f C[a, b] such that b f(x)dx = 0. a Suppose now W is a finite dimensional subspace of V. It then follows from the previous section that if v V and π(v) is the orthogonal projection of v onto W, then v π(v) W. In particular, every vector v V can be written as the sum of a vector in W and a vector in W ; in fact, v = π(v) + (v π(v)), where π(v) W and v π(v) W. The intersection of W with W is {0}. Thus V is the direct sum of W and W. In particular, if V is finite dimensional, then dim(v ) = dim(w ) + dim(w ). Example 9. Suppose V is a finite dimensional inner product space (real or complex), and W is a subspace of V. For every v V, let π 1 (v) be the orthogonal projection of v onto W, and let π 2 (v) be the orthogonal projection of v onto W. If Id denotes the identity map on V, show that Hence show that Id = π 1 + π 2, and π 1 π 2 = 0 = π 2 π 1. π 2 1 = π 1 and π 2 2 = π 2. Conclude that both π 1 and π 2 are diagonalizable.

13 13 7. The construction of orthogonal sets and the Gram-Schmidt process We now turn to the construction of orthogonal sets in an inner product space. Suppose {v 1,..., v k } is a subset of an inner product space V. Define u 1 = v 1, u 2 = v 2 orthogonal projection of v 2 onto the span of {u 1 }, u 3 = v 3 orthogonal projection of v 2 onto the span of {u 1, u 2 }, etc. In general, for j = 2,..., k, define u j to be u j = v j orthogonal projection of v j onto the span of {u 1,..., u j 1 }. Then it is easy to see that u i, u j = 0 for all i > j, and that the span of {u 1,..., u k } is the same as the span of {v 1,..., v k }. It follows that u 1 = v 1, u 2 = v 2 v 2, u 1 u 1, u 1 u 1, etc, and in general u 3 = v 3 v 3, u 1 u 1, u 1 u 1 v 3, u 2 u 2, u 2 u 2, u j = v j v j, u 1 u 1, u 1 u 1 v j, u j 1 u j 1, u j 1 u j 1 for j = 2,..., k. The process of constructing {u 1,..., u k } from {v 1,..., v k } is called the Gram-Schmidt process. It constructs for us an orthogonal set of vectors {u 1,..., u k } that has the same span as the given set {v 1,..., v k }. Notice that if {v 1,..., v k } are linearly independent to begin with, then so is {u 1,..., u k }. In particular, all of the u j s are then not equal to zero. By replacing u j by u j / u j, we can obtain an orthonormal set of vectors that has the same span as the given {v 1,..., v k }. In particular, if V is a finite dimensional inner product space, then V has an orthonormal basis. (Just begin with an arbitrary basis of V and apply Gram-Schmidt.) Example 10. Suppose V = P 4 (R), the vector space of all real polynomials of degree 4, equipped with inner product p(x), q(x) = 1 0 p(x)q(x)dx. Let W be the subspace of V spanned by {1, x, x 2 }. Find an orthonormal basis of W. Also find an orthonormal basis of W, the orthogonal complement of W in V.

14 14 8. Proof of the Cauchy-Schwarz and Triangle inequality We finally have the conceptual framework that helps us understand the proof of the Cauchy-Schwarz inequality (2) and the triangle inequality (1). To prove the Cauchy-Schwarz inequality, suppose V is an inner product space over C. Suppose v, w V are given. If w = 0, the Cauchy-Schwarz inequality is trivial. So suppose w 0. Then we minimize v cw 2 as c varies over C. The minimum is achieved when v, w c = w, w, as we have seen in the section about orthogonal projections; on the other hand, the minimum value of v cw 2 as c varies is certainly non-negative. Thus we have 0 v cw 2 = v, v c w, v c v, w + c 2 w, w = v 2 2Re (c v, w ) + c 2 w 2 2 = v 2 v, w 2 2 w, w + v, w w, w w 2 = v 2 v, w 2 v, w 2 2 w 2 + w 2 = v 2 v, w 2 w 2. The Cauchy-Schwarz inequality (2) then follows. A similar argument works when V is an inner product space over R. Next, to prove the triangle inequality, observe that v+w 2 = v+w, v+w = v, v + v, w + w, v + w, w = v 2 +2Re v, w + w 2. Now we invoke Cauchy-Schwarz: So 2Re v, w 2 v, w 2 v w. v + w 2 v v w + w 2 = ( v + w ) 2. The triangle inequality (1) follows. 9. The transpose and the adjoint of a matrix Next we turn to another set of concepts revolving around the adjoint of a linear operator. To begin with, recall that the transpose of a matrix is obtained by replacing its rows by its columns. If A is a matrix, its transpose is usually written as A t. The conjugate transpose of a matrix is what one obtains by transposing the matrix and putting complex conjugates over every entry; if A is a matrix, its conjugate transpose is usually written A. The conjugate transpose of a matrix is also sometimes called the adjoint of a matrix.

15 15 Note that if A is a real (resp. complex) m n matrix, one can think of A as a linear map from R n (resp. C n ) to R m (resp. C m ), and one can think of A t (resp. A ) as a linear map from R m (resp. C m ) into R n (resp. C n ). Also, if A, B are two matrices of the correct sizes, then (AB) t = B t A t and (AB) = B A. If we think of a vector in R n as a column vector, then the dot product of x and y in R n can be identified with y t x, where y t x is the matrix product of y t with x. Similarly, if we think of a vector in C n as a column vector, then for z, w C n, the dot product (z, w) can be identified with the matrix product of w z. It follows that if A is an m n matrix with real entries, then for all x R n and all y R m, we have Ax, y = x, A t y ; here, denote the standard inner products on R m and R n. This can be seen by just noticing that Ax, y = y t Ax = y t (A t ) t x = (A t y) t x = x, A t y. One also has y, Ax = A t y, x for all x R n and y R m. Similarly, if A is an m n matrix with complex entries, then for all z C n and all w C m, we have Az, w = z, A w, as well as w, Az = A w, z. These are important identities, and should in fact be thought of as defining A t and A when A is a given matrix. To illustrate the usefulness of these identities, we prove: Proposition 7. If A is a real m n matrix, then the kernel of A is the orthogonal complement of the range of A t in R n. Proof. Simply note that y is in the orthogonal complement of the range of A t y, A t x = 0 for all x R n Ay, x = 0 for all x R n Ay = 0 y is in the kernel of A. Similarly, one can prove that if A is a real m n matrix, then the kernel of A t is the orthogonal complement of the range of A in R m. It follows that 1. The range of A is the orthogonal complement of the kernel of A t in R m ; 2. The range of A t is the orthogonal complement of the kernel of A in R n ; 3. R n is the direct sum of the kernel of A and the range of A t ;

16 16 4. R m is the direct sum of the range of A and the kernel of A t. In particular, the equation Ax = b is solvable, if and only if b is orthogonal to the kernel of A t. The reader should formulate the complex version of these statements, when A is a complex m n matrix and A is its conjugate transpose. These generalizes to the setting where we have adjoints of linear maps. But before we can define that, we need the following Riesz representation theorem. 10. Riesz representation theorem Suppose V is a vector space over a field F. Suppose f : V F is a linear map. Then f is called a linear functional on V, and the set of all linear functionals on V is usually denoted as V. V is commonly known as the dual space of V. It is also a vector space, and in fact if V is finite dimensional, then so is V, and in fact dim(v ) = dim(v ). Suppose now V is an inner product space over F, where F = R or C. If w V, then the map f : V F, defined by f(v) = v, w, is a linear functional on V. The following Riesz representation theorem says that these are all the examples we have if V is finite dimensional; in fact different w defines different linear maps from V to F. Theorem 5. Suppose V is a finite dimensional inner product space over F, where F = R or C. If f : V F is a linear map, then there exists an unique w V such that f(v) = v, w for all v V. Proof. We carry out the proof when F = C. The proof when F = R is an easy modification of this proof and left to the reader. Suppose f : V C is a linear map. We first show the existence of w such that f(v) = v, w for all v V. If f is identically zero, then f(v) = v, 0 for all v V, so it suffices to take w = 0. Otherwise the rank of f is strictly bigger than zero, but since the rank of f is also at most 1 (since f maps V into C and dim(c) = 1), we have the rank of f = 1. This shows that the kernel of f is an n 1 dimensional subspace of V. Let W be the kernel of f. Then the orthogonal complement W of W in V is one-dimensional. Let w 0 be an unit vector in W, and let c = f(w 0 ). We claim that f(v) = v, cw 0 for all v V. In fact, it suffices to show that this is true for all v W and for v = w 0, since V is the direct sum of W and the subspace spanned by {w 0 }. But if

17 17 v W, then f(v) = 0 = v, cw 0 by our construction of W and w 0. Also, if v = w 0, then and also f(v) = f(w 0 ) = c, v, cw 0 = w 0, cw 0 = c. Thus we have f(v) = v, cw 0 for all v V, as desired. Next, suppose there exists w 1, w 2 V such that both f(v) = v, w 1 and f(v) = v, w 2 for all v V. Then v, w 1 = v, w 2 for all v V, i.e. v, w 1 w 2 = 0 for all v V. This shows, by Lemma 2, that w 1 w 2 = 0, i.e. w 1 = w 2. Thus we have the uniqueness assertion of the theorem. 11. The transpose and the adjoint of a linear map We are now in the position to define the transpose and the adjoint of a general linear map. Suppose V, W are finite dimensional inner product spaces over R. If T : V W is a linear map, one can find a linear map T t : W V such that T v, w = v, T t w for all v V, w W. This is because if w W is given, then the map v T v, w defines a linear functional on V. Thus by the Riesz representation theorem, there exists a unique element v 0 V such that T v, w = v, v 0 for all v V. This v 0 certainly depends on w, and we set T t w = v 0. This defines a map from W to V, and one can show that the map T t such defined is linear. We call T t the transpose of the linear map T. Similarly, if V, W are finite dimensional inner product spaces over C, then for every linear map T : V W, there exists a linear map T : W V such that T v, w = v, T w for all v V, w W. This linear map T is called the adjoint of the linear map T. Example 11. Show that if T : C n C m is a linear map given by T (z) = Az, where A is an m n matrix with complex entries and one thinks of z C n as a column vector, then T : C m C n is given by T (w) = A w for all w C m, where A is the conjugate transpose of A and we think of w in C m as a column vector.

18 18 Example 12. Suppose V is the vector space over R consisting of the set of all linear combinations of sin x, sin(2x), sin(3x) and sin(4x). We define an inner product on V by f(x), g(x) = 2π 0 f(x)g(x)dx. Let T : V V be the linear map defined by T (f(x)) = f (x) for all f V. Show that T t = T. 12. The spectral theorem for matrices The rest of the article is devoted to proving the spectral theorem in the setting of a finite dimensional real or complex inner product space. For the sake of exposition, we will first state the results for matrices, and then state the corresponding results for linear operators. From now on all matrices will be square matrices (say n n), and all linear operators will be from a finite dimensional inner product space V into itself. First we recall some definitions. First we recall some definitions. An n n real matrix P is said to be orthogonal if and only if P P t = P t P = I; here P t is the transpose of the matrix P. It is known that if P is an n n real matrix, then the following statements are equivalent: (i) P is orthogonal; (ii) P t P = I; (iii) the columns of P form an orthonormal basis of R n ; (iv) P P t = I; (v) the rows of P form an orthonormal basis of R n. An n n complex matrix U is said to be unitary if and only if U U = UU = I; here U is the conjugate transpose (or adjoint) of the matrix U. It is known that if U is an n n complex matrix, then the following statements are equivalent: (i) U is unitary; (ii) U U = I; (iii) the columns of U form an orthogonal basis of C n ; (iv) UU = I; (v) the rows of U form an orthogonal basis of C n. An n n real matrix A is said to be symmetric if and only if A t = A. An n n complex matrix A is said to be Hermitian (or self-adjoint) if and only if A = A. An n n complex matrix A is said to be normal if and only if AA = A A = I. The first theorem is a characterization of (real) symmetric matrices: Theorem 6. Suppose A is an n n real matrix. Then the following are equivalent:

19 19 (a) A is symmetric; (b) R n has an orthonormal basis that consists of eigenvectors of A; (c) there exists an orthogonal matrix P, and a diagonal matrix D, such that A = P DP 1. Note that it follows from the last statement of the above theorem that any symmetric matrix is diagonalizable over R. Also, one should observe that in last statement of the above theorem, one could also have written A = P DP t, since P t = P 1 when P is an orthogonal matrix. Similarly, we have the following characterization of Hermitian matrices: Theorem 7. Suppose A is an n n complex matrix. equivalent: Then the following are (a) A is Hermitian; (b) C n has an orthogonal basis that consists of eigenvectors of A, and all eigenvalues of A are real; (c) there exists an unitary matrix U, and a diagonal matrix D with real entries, such that A = UDU 1. Finally we have the following characterization of normal matrices: Theorem 8. Suppose A is an n n complex matrix. equivalent: Then the following are (a) A is normal; (b) C n has an orthogonal basis that consists of eigenvectors of A; (c) there exists an unitary matrix U, and a diagonal matrix D, such that A = UDU 1. Again, it follows from the last statement of the two theorems above that any self-adjoint matrix is diagonalizable over C, and so is any normal matrix. Also, one should observe that in last statement of the two theorems above, one could also have written A = UDU, since U = U 1 when U is a unitary matrix. In each of the three theorems above, the equivalence of (b) and (c) is an easy exercise. So the key is to prove the equivalence between (a) and (b). Before we do that, to put these theorems in perspective, we note the role played by orthogonality here. For instance, Theorem 8 can be seen as a characterization of complex n n matrices A for which C n admits an orthonormal basis consisting of eigenvectors of A. The class of such linear operators turns out to be quite small, as one sees from the theorem; in fact the set of normal matrices is nowhere dense in the set of complex n n matrices. On the other hand, if we drop the assumption of orthogonality on the basis of C n, i.e. if we are interested instead in the set of all complex matrices for

20 20 which C n admits a basis (not necessarily orthonormal) consisting of eigenvectors of A, then we will get the set of all diagonalizable complex n n matrices, and this set turns out to be dense in the set of complex n n matrices. 13. The spectral theorem for linear operators It turns out that the above three theorems have a counterpart that is stated in terms of linear operators. First we make the following definitions. If V is a real inner product space, and T : V V is linear, then T is said to be symmetric if and only if T t = T. If V is a complex inner product space, and T : V V is linear, then T is said to be self-adjoint (or Hermitian) if and only if T = T, and T is said to be normal if and only if T T = T T. One can show that every symmetric operator on a finite dimensional real inner product space extends to a self-adjoint operator on the complexification of the real inner product space, and every self-adjoint operator on a complex inner product space is normal. We now have a characterization of symmetric operators on real inner product spaces: Theorem 9. Suppose V is a finite dimensional real inner product space, and T : V V is linear. Then the following are equivalent: (a) T is symmetric; (b) V has an orthonormal basis that consists of eigenvectors of T ; (c) V has an ordered orthonormal basis such that the matrix representation of T with respect to this basis is diagonal. Similarly, we have the following characterization of self-adjoint operators on complex inner product spaces: Theorem 10. Suppose V is a finite dimensional complex inner product space, and T : V V is linear. Then the following are equivalent: (a) T is self-adjoint; (b) V has an orthonormal basis that consists of eigenvectors of T, and all eigenvalues of T are real; (c) V has an ordered orthonormal basis such that the matrix representation of T with respect to this basis is diagonal with real entries. Finally we have the following characterization of normal operators on complex inner product spaces: Theorem 11. Suppose V is a finite dimensional complex inner product space, and T : V V is linear. Then the following are equivalent:

21 21 (a) T is normal; (b) V has an orthonormal basis that consists of eigenvectors of T ; (c) V has an ordered orthonormal basis such that the matrix representation of T with respect to this basis is diagonal. Again the equivalence of (b) and (c) in the three theorems above are very easy. We leave them to the reader. Observe that if one uses the standard inner products on R n and C n, then a real n n matrix A is symmetric if and only if the linear operator x Ax is a symmetric operator on R n, and a complex n n matrix A is self-adjoint (resp. normal) if and only if the linear operator z Az is a self-adjoint (resp. normal) operator on C n. Thus to prove the equivalence of (a) and (b) in Theorems 6, 7 and 8, we only need to prove the equivalence of (a) and (b) in Theorems 9, 10 and 11; in fact it is beneficial to take this abstract point of view even if one is only interested in proving the results about matrices. Since it is relatively easy to prove that (b) implies (a) in Theorems 9, 10 and 11, we leave them to the reader. In the next three sections, we prove that (a) implies (b) in Theorems 9, 10 and Proof of Theorem 10 In this section we prove that (a) implies (b) in Theorem 10. Suppose T : V V is a self-adjoint linear operator on a finite dimensional complex inner product space V. First we prove the following proposition: Proposition 8. All the eigenvalues of T are real. Proof. Suppose λ C is an eigenvalue of T, and v 0 is the corresponding eigenvector of T. Then T v = λv, so Since v 0, we get and thus λ is real. λ v, v = T v, v = v, T v = λ v, v. λ = λ, Next, we want to prove that V has an orthonormal basis consisting of eigenvectors of T. The proof here is by induction on the dimension of V. It comes in two steps. First, given T as above, we find an eigenvector v 1 of T. Without loss of generality we may assume that v 1 = 1. Let W be the orthogonal complement of v 1 in V. We will then show 1 that W is a T -invariant subspace of V, and that the restriction of T to W is still self-adjoint. One can then invoke our induction hypothesis, and conclude that W has an orthonormal basis {v 2,..., v n } that consists 1 This is a crucial place where orthogonality comes in; it is not just any complement of the span of v 1 in V that is T -invariant, but the orthogonal complement of v 1 that is T -invariant.

22 22 of eigenvectors of T. Then one can check that {v 1,..., v n } is an orthonormal basis of V consisting of eigenvectors of T. As a result, we only need to prove the following: (i) Every self-adjoint linear operator T : V V on a finite dimensional complex inner product space V has an eigenvector; (ii) If v 1 is an eigenvector of T and W is the orthogonal complement of v 1 in V, then W is T -invariant, and that T W : W W is self-adjoint. We begin with (ii). If v 1 is an eigenvector of T, say with eigenvalue λ ( R by the previous proposition), and if W is the orthogonal complement, then for any w W, we have T w, v 1 = w, T v 1 = w, T v 1 = λ w, v 1 = 0. Thus T w is in W for all w W. This proves that W is T -invariant. Next, since T = T, W is also T -invariant. Thus if U is the restriction of T to W, then, U is the restriction of T to W ; in fact if w 1, w 2 W, then w 1, U w 2 = Uw 1, w 2 = T w 1, w 2 = w 1, T w 2 = w 1, Uw 2, which shows U w 2 = Uw 2 for all w 2 W. (Here one uses Uw 2 W.) It follows that U = U, and thus the restriction of T to W is self-adjoint. We next prove (i). The easiest proof is to invoke the fundamental theorem of algebra, which says that every complex polynomial has a root in the complex plane. In particular, since V is finite dimensional, there exists some λ 1 C such that det(t λ 1 I) = 0. This says T has an eigenvalue λ 1, and thus T has an eigenvector v 1 corresponding to λ Proof of Theorem 11 In this section, suppose V is a finite dimensional complex inner product space, and T : V V is a normal linear operator. We will prove that V has a basis that consists of eigenvectors of T, thereby proving that (a) implies (b) in Theorem 11. Again the idea is to prove the following: (i) T has an eigenvector in V ; (ii) If v 1 is an eigenvector of T and W is the orthogonal complement of v 1 in V, then W is T -invariant, and that T W : W W is normal. Once these are proved, the proof that (a) implies (b) in Theorem 11 can be completed as in the case of Theorems 10. We leave that to the reader. First we prove an important property about eigenvectors of normal operators. Proposition 9. Suppose T : V V is normal. If v is an eigenvector of T with eigenvalue λ, then v is also an eigenvector of T with eigenvalue λ, and conversely.

23 23 Proof. By normality of T, we know that T λi and T λi commutes for all λ C. Thus for any v V and any λ C, we have (T λi)v, (T λi)v = 0 (T λi)v, (T λi)v = 0. But this says (T λi)v = 0 (T λi)v = 0. So we have the desired assertion. We are now ready to prove statement (ii) above. Suppose again that T : V V is a complex normal operator. If v 1 is an eigenvector of T, say with eigenvalue λ, let W be the orthogonal complement of v 1. Now if w W, we want to show that T w W. In other words, we want to show T w, v 1 = 0. But T w, v 1 = w, T v 1 = w, λv 1 = λ w, v 1 = 0, where the second equality follows from the previous proposition. Thus W is T - invariant. Similarly, one can show that W is T -invariant. Hence, if U is the restriction of T to W, then for all w 1, w 2 W, we have w 1, U w 2 = Uw 1, w 2 = T w 1, w 2 = w 1, T w 2, which shows U w 2 = T w 2 for all w 2 W. (Here one uses T w 2 W.) From normality of T, it then follows that U U = UU, and thus the restriction of T to W is still a normal operator. Next, we turn to a proof of (i). Again the most direct one is to use the fundamental theorem of algebra, as in the first proof of (i) in Theorem 10. We leave the details to the reader. 16. Proof of Theorem 9 We now give the modifications of the proof of Theorem 10 that are necessary to prove that (a) implies (b) in Theorem 9. Thus in this section, unless otherwise stated, we assume that V is a real inner product space, and T : V V is a symmetric linear operator. As in the proof of Theorem 10, we only need to prove the following: (i) T has an eigenvector in V ; (ii) If v 1 is an eigenvector of T and W is the orthogonal complement of v 1 in V, then W is T -invariant, and that T W : W W is symmetric. The proof of (ii) in our current context is the same as that of the corresponding statement in the proof of Theorem 10. One just needs to keep in mind that if V is a real inner product space, then any eigenvalue of a symmetric linear operator T : V V is real (by definition). One then replaces any occurrence of T in the previous proof by T t. We leave the details to the reader. To prove (i) in our current context, we use a variational method. Let S be the unit sphere in V, i.e. S = {v V : v = 1}. It is a compact subset of V by the finite dimensionality of V. Now let F (v) = T v, v. This defines a smooth function

24 24 from S to R. Thus F achieves its maximum at some v 1 S. We can then show that v 1 is an eigenvector of T, pretty much in the same way that we had when we gave the second proof of (i) in the previous section. In fact, we then have ( ) d v1 + tw dt F = 0. t=0 v 1 + tw But the left hand side of this equation is equal to d 1 dt t=0 v 1 + tw 2 T (v 1 + tw), v 1 + tw which in turn is given by v 1, w + w, v 1 v 1 4 T v 1, v v 1 2 ( T v 1, w + T w, v 1 ), i.e. ( v 1, w + w, v 1 ) T v 1, v 1 + T v 1, w + T w, v 1. Since we are working on a real inner product space, and since T t = T, this implies Thus letting λ 1 = T v 1, v 1, we get 2 v 1, w T v 1, v T v 1, w = 0. T v 1 λ 1 v 1, w = 0 for all w V. Thus T v 1 = λ 1 v 1, and T has an eigenvector v 1. This completes our proof that (a) implies (b) in Theorem 9.

### 1. General Vector Spaces

1.1. Vector space axioms. 1. General Vector Spaces Definition 1.1. Let V be a nonempty set of objects on which the operations of addition and scalar multiplication are defined. By addition we mean a rule

### LINEAR ALGEBRA W W L CHEN

LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008. This chapter is available free to all individuals, on the understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,

### DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.

DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

### MATRICES ARE SIMILAR TO TRIANGULAR MATRICES

MATRICES ARE SIMILAR TO TRIANGULAR MATRICES 1 Complex matrices Recall that the complex numbers are given by a + ib where a and b are real and i is the imaginary unity, ie, i 2 = 1 In what we describe below,

### Linear Algebra Massoud Malek

CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### Math Linear Algebra II. 1. Inner Products and Norms

Math 342 - Linear Algebra II Notes 1. Inner Products and Norms One knows from a basic introduction to vectors in R n Math 254 at OSU) that the length of a vector x = x 1 x 2... x n ) T R n, denoted x,

### Spectral Theorem for Self-adjoint Linear Operators

Notes for the undergraduate lecture by David Adams. (These are the notes I would write if I was teaching a course on this topic. I have included more material than I will cover in the 45 minute lecture;

### Ir O D = D = ( ) Section 2.6 Example 1. (Bottom of page 119) dim(v ) = dim(l(v, W )) = dim(v ) dim(f ) = dim(v )

Section 3.2 Theorem 3.6. Let A be an m n matrix of rank r. Then r m, r n, and, by means of a finite number of elementary row and column operations, A can be transformed into the matrix ( ) Ir O D = 1 O

### The following definition is fundamental.

1. Some Basics from Linear Algebra With these notes, I will try and clarify certain topics that I only quickly mention in class. First and foremost, I will assume that you are familiar with many basic

### Linear Algebra Lecture Notes-II

Linear Algebra Lecture Notes-II Vikas Bist Department of Mathematics Panjab University, Chandigarh-64 email: bistvikas@gmail.com Last revised on March 5, 8 This text is based on the lectures delivered

### Chapter 6: Orthogonality

Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products

### MATH 240 Spring, Chapter 1: Linear Equations and Matrices

MATH 240 Spring, 2006 Chapter Summaries for Kolman / Hill, Elementary Linear Algebra, 8th Ed. Sections 1.1 1.6, 2.1 2.2, 3.2 3.8, 4.3 4.5, 5.1 5.3, 5.5, 6.1 6.5, 7.1 7.2, 7.4 DEFINITIONS Chapter 1: Linear

### October 25, 2013 INNER PRODUCT SPACES

October 25, 2013 INNER PRODUCT SPACES RODICA D. COSTIN Contents 1. Inner product 2 1.1. Inner product 2 1.2. Inner product spaces 4 2. Orthogonal bases 5 2.1. Existence of an orthogonal basis 7 2.2. Orthogonal

### Numerical Linear Algebra

University of Alabama at Birmingham Department of Mathematics Numerical Linear Algebra Lecture Notes for MA 660 (1997 2014) Dr Nikolai Chernov April 2014 Chapter 0 Review of Linear Algebra 0.1 Matrices

### 08a. Operators on Hilbert spaces. 1. Boundedness, continuity, operator norms

(February 24, 2017) 08a. Operators on Hilbert spaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ [This document is http://www.math.umn.edu/ garrett/m/real/notes 2016-17/08a-ops

### Math 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination

Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column

### 6 Inner Product Spaces

Lectures 16,17,18 6 Inner Product Spaces 6.1 Basic Definition Parallelogram law, the ability to measure angle between two vectors and in particular, the concept of perpendicularity make the euclidean space

### LINEAR ALGEBRA MICHAEL PENKAVA

LINEAR ALGEBRA MICHAEL PENKAVA 1. Linear Maps Definition 1.1. If V and W are vector spaces over the same field K, then a map λ : V W is called a linear map if it satisfies the two conditions below: (1)

### A PRIMER ON SESQUILINEAR FORMS

A PRIMER ON SESQUILINEAR FORMS BRIAN OSSERMAN This is an alternative presentation of most of the material from 8., 8.2, 8.3, 8.4, 8.5 and 8.8 of Artin s book. Any terminology (such as sesquilinear form

### MATH 20F: LINEAR ALGEBRA LECTURE B00 (T. KEMP)

MATH 20F: LINEAR ALGEBRA LECTURE B00 (T KEMP) Definition 01 If T (x) = Ax is a linear transformation from R n to R m then Nul (T ) = {x R n : T (x) = 0} = Nul (A) Ran (T ) = {Ax R m : x R n } = {b R m

### Mathematics Department Stanford University Math 61CM/DM Inner products

Mathematics Department Stanford University Math 61CM/DM Inner products Recall the definition of an inner product space; see Appendix A.8 of the textbook. Definition 1 An inner product space V is a vector

### Linear Algebra. Min Yan

Linear Algebra Min Yan January 2, 2018 2 Contents 1 Vector Space 7 1.1 Definition................................. 7 1.1.1 Axioms of Vector Space..................... 7 1.1.2 Consequence of Axiom......................

### MATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators.

MATH 423 Linear Algebra II Lecture 33: Diagonalization of normal operators. Adjoint operator and adjoint matrix Given a linear operator L on an inner product space V, the adjoint of L is a transformation

### MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2.

MATH 304 Linear Algebra Lecture 23: Diagonalization. Review for Test 2. Diagonalization Let L be a linear operator on a finite-dimensional vector space V. Then the following conditions are equivalent:

### Designing Information Devices and Systems II

EECS 16B Fall 2016 Designing Information Devices and Systems II Linear Algebra Notes Introduction In this set of notes, we will derive the linear least squares equation, study the properties symmetric

### Dot Products. K. Behrend. April 3, Abstract A short review of some basic facts on the dot product. Projections. The spectral theorem.

Dot Products K. Behrend April 3, 008 Abstract A short review of some basic facts on the dot product. Projections. The spectral theorem. Contents The dot product 3. Length of a vector........................

### homogeneous 71 hyperplane 10 hyperplane 34 hyperplane 69 identity map 171 identity map 186 identity map 206 identity matrix 110 identity matrix 45

address 12 adjoint matrix 118 alternating 112 alternating 203 angle 159 angle 33 angle 60 area 120 associative 180 augmented matrix 11 axes 5 Axiom of Choice 153 basis 178 basis 210 basis 74 basis test

### Math 413/513 Chapter 6 (from Friedberg, Insel, & Spence)

Math 413/513 Chapter 6 (from Friedberg, Insel, & Spence) David Glickenstein December 7, 2015 1 Inner product spaces In this chapter, we will only consider the elds R and C. De nition 1 Let V be a vector

### Lecture 7: Positive Semidefinite Matrices

Lecture 7: Positive Semidefinite Matrices Rajat Mittal IIT Kanpur The main aim of this lecture note is to prepare your background for semidefinite programming. We have already seen some linear algebra.

### Elementary linear algebra

Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The

### Recall the convention that, for us, all vectors are column vectors.

Some linear algebra Recall the convention that, for us, all vectors are column vectors. 1. Symmetric matrices Let A be a real matrix. Recall that a complex number λ is an eigenvalue of A if there exists

### Math 108b: Notes on the Spectral Theorem

Math 108b: Notes on the Spectral Theorem From section 6.3, we know that every linear operator T on a finite dimensional inner product space V has an adjoint. (T is defined as the unique linear operator

### MTH 2032 SemesterII

MTH 202 SemesterII 2010-11 Linear Algebra Worked Examples Dr. Tony Yee Department of Mathematics and Information Technology The Hong Kong Institute of Education December 28, 2011 ii Contents Table of Contents

### MATH 1120 (LINEAR ALGEBRA 1), FINAL EXAM FALL 2011 SOLUTIONS TO PRACTICE VERSION

MATH (LINEAR ALGEBRA ) FINAL EXAM FALL SOLUTIONS TO PRACTICE VERSION Problem (a) For each matrix below (i) find a basis for its column space (ii) find a basis for its row space (iii) determine whether

### Homework set 4 - Solutions

Homework set 4 - Solutions Math 407 Renato Feres 1. Exercise 4.1, page 49 of notes. Let W := T0 m V and denote by GLW the general linear group of W, defined as the group of all linear isomorphisms of W

### Recall: Dot product on R 2 : u v = (u 1, u 2 ) (v 1, v 2 ) = u 1 v 1 + u 2 v 2, u u = u u 2 2 = u 2. Geometric Meaning:

Recall: Dot product on R 2 : u v = (u 1, u 2 ) (v 1, v 2 ) = u 1 v 1 + u 2 v 2, u u = u 2 1 + u 2 2 = u 2. Geometric Meaning: u v = u v cos θ. u θ v 1 Reason: The opposite side is given by u v. u v 2 =

### Lecture notes: Applied linear algebra Part 1. Version 2

Lecture notes: Applied linear algebra Part 1. Version 2 Michael Karow Berlin University of Technology karow@math.tu-berlin.de October 2, 2008 1 Notation, basic notions and facts 1.1 Subspaces, range and

### Analysis Preliminary Exam Workshop: Hilbert Spaces

Analysis Preliminary Exam Workshop: Hilbert Spaces 1. Hilbert spaces A Hilbert space H is a complete real or complex inner product space. Consider complex Hilbert spaces for definiteness. If (, ) : H H

### Chapter 4 Euclid Space

Chapter 4 Euclid Space Inner Product Spaces Definition.. Let V be a real vector space over IR. A real inner product on V is a real valued function on V V, denoted by (, ), which satisfies () (x, y) = (y,

### Inner product spaces. Layers of structure:

Inner product spaces Layers of structure: vector space normed linear space inner product space The abstract definition of an inner product, which we will see very shortly, is simple (and by itself is pretty

### INNER PRODUCT SPACE. Definition 1

INNER PRODUCT SPACE Definition 1 Suppose u, v and w are all vectors in vector space V and c is any scalar. An inner product space on the vectors space V is a function that associates with each pair of

### Econ 204 Supplement to Section 3.6 Diagonalization and Quadratic Forms. 1 Diagonalization and Change of Basis

Econ 204 Supplement to Section 3.6 Diagonalization and Quadratic Forms De La Fuente notes that, if an n n matrix has n distinct eigenvalues, it can be diagonalized. In this supplement, we will provide

### Lecture 23: 6.1 Inner Products

Lecture 23: 6.1 Inner Products Wei-Ta Chu 2008/12/17 Definition An inner product on a real vector space V is a function that associates a real number u, vwith each pair of vectors u and v in V in such

### Inner products. Theorem (basic properties): Given vectors u, v, w in an inner product space V, and a scalar k, the following properties hold:

Inner products Definition: An inner product on a real vector space V is an operation (function) that assigns to each pair of vectors ( u, v) in V a scalar u, v satisfying the following axioms: 1. u, v

### Math 291-2: Lecture Notes Northwestern University, Winter 2016

Math 291-2: Lecture Notes Northwestern University, Winter 2016 Written by Santiago Cañez These are lecture notes for Math 291-2, the second quarter of MENU: Intensive Linear Algebra and Multivariable Calculus,

### NORMS ON SPACE OF MATRICES

NORMS ON SPACE OF MATRICES. Operator Norms on Space of linear maps Let A be an n n real matrix and x 0 be a vector in R n. We would like to use the Picard iteration method to solve for the following system

### x 1 x 2. x 1, x 2,..., x n R. x n

WEEK In general terms, our aim in this first part of the course is to use vector space theory to study the geometry of Euclidean space A good knowledge of the subject matter of the Matrix Applications

### Linear algebra 2. Yoav Zemel. March 1, 2012

Linear algebra 2 Yoav Zemel March 1, 2012 These notes were written by Yoav Zemel. The lecturer, Shmuel Berger, should not be held responsible for any mistake. Any comments are welcome at zamsh7@gmail.com.

### Linear Models Review

Linear Models Review Vectors in IR n will be written as ordered n-tuples which are understood to be column vectors, or n 1 matrices. A vector variable will be indicted with bold face, and the prime sign

### Vectors in Function Spaces

Jim Lambers MAT 66 Spring Semester 15-16 Lecture 18 Notes These notes correspond to Section 6.3 in the text. Vectors in Function Spaces We begin with some necessary terminology. A vector space V, also

### Linear Algebra I. Ronald van Luijk, 2015

Linear Algebra I Ronald van Luijk, 2015 With many parts from Linear Algebra I by Michael Stoll, 2007 Contents Dependencies among sections 3 Chapter 1. Euclidean space: lines and hyperplanes 5 1.1. Definition

### Math 4A Notes. Written by Victoria Kala Last updated June 11, 2017

Math 4A Notes Written by Victoria Kala vtkala@math.ucsb.edu Last updated June 11, 2017 Systems of Linear Equations A linear equation is an equation that can be written in the form a 1 x 1 + a 2 x 2 +...

### MATH 583A REVIEW SESSION #1

MATH 583A REVIEW SESSION #1 BOJAN DURICKOVIC 1. Vector Spaces Very quick review of the basic linear algebra concepts (see any linear algebra textbook): (finite dimensional) vector space (or linear space),

### THE MINIMAL POLYNOMIAL AND SOME APPLICATIONS

THE MINIMAL POLYNOMIAL AND SOME APPLICATIONS KEITH CONRAD. Introduction The easiest matrices to compute with are the diagonal ones. The sum and product of diagonal matrices can be computed componentwise

### University of Colorado at Denver Mathematics Department Applied Linear Algebra Preliminary Exam With Solutions 16 January 2009, 10:00 am 2:00 pm

University of Colorado at Denver Mathematics Department Applied Linear Algebra Preliminary Exam With Solutions 16 January 2009, 10:00 am 2:00 pm Name: The proctor will let you read the following conditions

### Quantum Computing Lecture 2. Review of Linear Algebra

Quantum Computing Lecture 2 Review of Linear Algebra Maris Ozols Linear algebra States of a quantum system form a vector space and their transformations are described by linear operators Vector spaces

### Functional Analysis Review

Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all

### Finite-dimensional spaces. C n is the space of n-tuples x = (x 1,..., x n ) of complex numbers. It is a Hilbert space with the inner product

Chapter 4 Hilbert Spaces 4.1 Inner Product Spaces Inner Product Space. A complex vector space E is called an inner product space (or a pre-hilbert space, or a unitary space) if there is a mapping (, )

### MATH 115A: SAMPLE FINAL SOLUTIONS

MATH A: SAMPLE FINAL SOLUTIONS JOE HUGHES. Let V be the set of all functions f : R R such that f( x) = f(x) for all x R. Show that V is a vector space over R under the usual addition and scalar multiplication

### Introduction to Linear Algebra, Second Edition, Serge Lange

Introduction to Linear Algebra, Second Edition, Serge Lange Chapter I: Vectors R n defined. Addition and scalar multiplication in R n. Two geometric interpretations for a vector: point and displacement.

### 235 Final exam review questions

5 Final exam review questions Paul Hacking December 4, 0 () Let A be an n n matrix and T : R n R n, T (x) = Ax the linear transformation with matrix A. What does it mean to say that a vector v R n is an

### MATH 304 Linear Algebra Lecture 34: Review for Test 2.

MATH 304 Linear Algebra Lecture 34: Review for Test 2. Topics for Test 2 Linear transformations (Leon 4.1 4.3) Matrix transformations Matrix of a linear mapping Similar matrices Orthogonality (Leon 5.1

### 5 Compact linear operators

5 Compact linear operators One of the most important results of Linear Algebra is that for every selfadjoint linear map A on a finite-dimensional space, there exists a basis consisting of eigenvectors.

### Lecture 2: Linear operators

Lecture 2: Linear operators Rajat Mittal IIT Kanpur The mathematical formulation of Quantum computing requires vector spaces and linear operators So, we need to be comfortable with linear algebra to study

### Algebra II. Paulius Drungilas and Jonas Jankauskas

Algebra II Paulius Drungilas and Jonas Jankauskas Contents 1. Quadratic forms 3 What is quadratic form? 3 Change of variables. 3 Equivalence of quadratic forms. 4 Canonical form. 4 Normal form. 7 Positive

### MATH 23a, FALL 2002 THEORETICAL LINEAR ALGEBRA AND MULTIVARIABLE CALCULUS Solutions to Final Exam (in-class portion) January 22, 2003

MATH 23a, FALL 2002 THEORETICAL LINEAR ALGEBRA AND MULTIVARIABLE CALCULUS Solutions to Final Exam (in-class portion) January 22, 2003 1. True or False (28 points, 2 each) T or F If V is a vector space

### 12x + 18y = 30? ax + by = m

Math 2201, Further Linear Algebra: a practical summary. February, 2009 There are just a few themes that were covered in the course. I. Algebra of integers and polynomials. II. Structure theory of one endomorphism.

### Spectral Theorems in Euclidean and Hermitian Spaces

Chapter 9 Spectral Theorems in Euclidean and Hermitian Spaces 9.1 Normal Linear Maps Let E be a real Euclidean space (or a complex Hermitian space) with inner product u, v u, v. In the real Euclidean case,

### Real symmetric matrices/1. 1 Eigenvalues and eigenvectors

Real symmetric matrices 1 Eigenvalues and eigenvectors We use the convention that vectors are row vectors and matrices act on the right. Let A be a square matrix with entries in a field F; suppose that

### Spectral Theory, with an Introduction to Operator Means. William L. Green

Spectral Theory, with an Introduction to Operator Means William L. Green January 30, 2008 Contents Introduction............................... 1 Hilbert Space.............................. 4 Linear Maps

### A Brief Outline of Math 355

A Brief Outline of Math 355 Lecture 1 The geometry of linear equations; elimination with matrices A system of m linear equations with n unknowns can be thought of geometrically as m hyperplanes intersecting

### Linear Algebra: Matrix Eigenvalue Problems

CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given

### LECTURE VI: SELF-ADJOINT AND UNITARY OPERATORS MAT FALL 2006 PRINCETON UNIVERSITY

LECTURE VI: SELF-ADJOINT AND UNITARY OPERATORS MAT 204 - FALL 2006 PRINCETON UNIVERSITY ALFONSO SORRENTINO 1 Adjoint of a linear operator Note: In these notes, V will denote a n-dimensional euclidean vector

### 1. Foundations of Numerics from Advanced Mathematics. Linear Algebra

Foundations of Numerics from Advanced Mathematics Linear Algebra Linear Algebra, October 23, 22 Linear Algebra Mathematical Structures a mathematical structure consists of one or several sets and one or

### I teach myself... Hilbert spaces

I teach myself... Hilbert spaces by F.J.Sayas, for MATH 806 November 4, 2015 This document will be growing with the semester. Every in red is for you to justify. Even if we start with the basic definition

### Typical Problem: Compute.

Math 2040 Chapter 6 Orhtogonality and Least Squares 6.1 and some of 6.7: Inner Product, Length and Orthogonality. Definition: If x, y R n, then x y = x 1 y 1 +... + x n y n is the dot product of x and

### The Spectral Theorem for normal linear maps

MAT067 University of California, Davis Winter 2007 The Spectral Theorem for normal linear maps Isaiah Lankham, Bruno Nachtergaele, Anne Schilling (March 14, 2007) In this section we come back to the question

### 22.3. Repeated Eigenvalues and Symmetric Matrices. Introduction. Prerequisites. Learning Outcomes

Repeated Eigenvalues and Symmetric Matrices. Introduction In this Section we further develop the theory of eigenvalues and eigenvectors in two distinct directions. Firstly we look at matrices where one

### LINEAR ALGEBRA REVIEW

LINEAR ALGEBRA REVIEW JC Stuff you should know for the exam. 1. Basics on vector spaces (1) F n is the set of all n-tuples (a 1,... a n ) with a i F. It forms a VS with the operations of + and scalar multiplication

### 2. Review of Linear Algebra

2. Review of Linear Algebra ECE 83, Spring 217 In this course we will represent signals as vectors and operators (e.g., filters, transforms, etc) as matrices. This lecture reviews basic concepts from linear

### MATH 220: INNER PRODUCT SPACES, SYMMETRIC OPERATORS, ORTHOGONALITY

MATH 22: INNER PRODUCT SPACES, SYMMETRIC OPERATORS, ORTHOGONALITY When discussing separation of variables, we noted that at the last step we need to express the inhomogeneous initial or boundary data as

### Quadratic forms. Here. Thus symmetric matrices are diagonalizable, and the diagonalization can be performed by means of an orthogonal matrix.

Quadratic forms 1. Symmetric matrices An n n matrix (a ij ) n ij=1 with entries on R is called symmetric if A T, that is, if a ij = a ji for all 1 i, j n. We denote by S n (R) the set of all n n symmetric

### Chapter 2 Linear Transformations

Chapter 2 Linear Transformations Linear Transformations Loosely speaking, a linear transformation is a function from one vector space to another that preserves the vector space operations. Let us be more

### Chapter 3 Transformations

Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases

### 1 Last time: least-squares problems

MATH Linear algebra (Fall 07) Lecture Last time: least-squares problems Definition. If A is an m n matrix and b R m, then a least-squares solution to the linear system Ax = b is a vector x R n such that

### HILBERT SPACES AND THE RADON-NIKODYM THEOREM. where the bar in the first equation denotes complex conjugation. In either case, for any x V define

HILBERT SPACES AND THE RADON-NIKODYM THEOREM STEVEN P. LALLEY 1. DEFINITIONS Definition 1. A real inner product space is a real vector space V together with a symmetric, bilinear, positive-definite mapping,

### MATH 31 - ADDITIONAL PRACTICE PROBLEMS FOR FINAL

MATH 3 - ADDITIONAL PRACTICE PROBLEMS FOR FINAL MAIN TOPICS FOR THE FINAL EXAM:. Vectors. Dot product. Cross product. Geometric applications. 2. Row reduction. Null space, column space, row space, left

### Diagonalizing Matrices

Diagonalizing Matrices Massoud Malek A A Let A = A k be an n n non-singular matrix and let B = A = [B, B,, B k,, B n ] Then A n A B = A A 0 0 A k [B, B,, B k,, B n ] = 0 0 = I n 0 A n Notice that A i B

### Archive of past papers, solutions and homeworks for. MATH 224, Linear Algebra 2, Spring 2013, Laurence Barker

Archive of past papers, solutions and homeworks for MATH 224, Linear Algebra 2, Spring 213, Laurence Barker version: 4 June 213 Source file: archfall99.tex page 2: Homeworks. page 3: Quizzes. page 4: Midterm

### Math 113 Solutions: Homework 8. November 28, 2007

Math 113 Solutions: Homework 8 November 28, 27 3) Define c j = ja j, d j = 1 j b j, for j = 1, 2,, n Let c = c 1 c 2 c n and d = vectors in R n Then the Cauchy-Schwarz inequality on Euclidean n-space gives

### The value of a problem is not so much coming up with the answer as in the ideas and attempted ideas it forces on the would be solver I.N.

Math 410 Homework Problems In the following pages you will find all of the homework problems for the semester. Homework should be written out neatly and stapled and turned in at the beginning of class

### Supplementary Notes on Linear Algebra

Supplementary Notes on Linear Algebra Mariusz Wodzicki May 3, 2015 1 Vector spaces 1.1 Coordinatization of a vector space 1.1.1 Given a basis B = {b 1,..., b n } in a vector space V, any vector v V can

### Linear Algebra. Workbook

Linear Algebra Workbook Paul Yiu Department of Mathematics Florida Atlantic University Last Update: November 21 Student: Fall 2011 Checklist Name: A B C D E F F G H I J 1 2 3 4 5 6 7 8 9 10 xxx xxx xxx

### Linear algebra and applications to graphs Part 1

Linear algebra and applications to graphs Part 1 Written up by Mikhail Belkin and Moon Duchin Instructor: Laszlo Babai June 17, 2001 1 Basic Linear Algebra Exercise 1.1 Let V and W be linear subspaces

### Throughout these notes we assume V, W are finite dimensional inner product spaces over C.

Math 342 - Linear Algebra II Notes Throughout these notes we assume V, W are finite dimensional inner product spaces over C 1 Upper Triangular Representation Proposition: Let T L(V ) There exists an orthonormal

### UNDERSTANDING THE DIAGONALIZATION PROBLEM. Roy Skjelnes. 1.- Linear Maps 1.1. Linear maps. A map T : R n R m is a linear map if

UNDERSTANDING THE DIAGONALIZATION PROBLEM Roy Skjelnes Abstract These notes are additional material to the course B107, given fall 200 The style may appear a bit coarse and consequently the student is

### 4.2. ORTHOGONALITY 161

4.2. ORTHOGONALITY 161 Definition 4.2.9 An affine space (E, E ) is a Euclidean affine space iff its underlying vector space E is a Euclidean vector space. Given any two points a, b E, we define the distance

### Chapter 5. Basics of Euclidean Geometry

Chapter 5 Basics of Euclidean Geometry 5.1 Inner Products, Euclidean Spaces In Affine geometry, it is possible to deal with ratios of vectors and barycenters of points, but there is no way to express the