10 Orthogonality Orthogonal subspaces

Transcription

1 10 Orthogonality 10.1 Orthogonal subspaces In the plane R 2 we think of the coordinate axes as being orthogonal (perpendicular) to each other. We can express this in terms of vectors by saying that every vector in one axis is orthogonal to every vector in the other. Let V be an inner product space and let S and T be subsets of V. We say that S and T are orthogonal, written S T, if every vector in S is orthogonal to every vector in T: S T s t for all s S, t T Example In R 3 let S be the x 1 x 2 -plane and let T be the x 3 -axis. Show that S T. Solution Let s S and let t T. We can write s = [x 1,x 2,0] T and t = [0,0,x 3 ] T so that s,t = s T t = [ x 1 x 2 0 ] 0 0 = 0. Therefore, s t and we conclude that S T Example In R 3 let S be the x 1 x 2 -plane and let T be the x 1 x 3 - plane. Is it true that S T? Solution Although the planes S and T appear to be perpendicular in the informal sense, they are not so in the technical sense. For instance, the vector e 1 = [1,0,0] T is in both S and T, yet e 1,e 1 = e T 1 e 1 = [ ] 1 0 = 1 0, 0 so e 1 e 1. Viewing the first e 1 in S and the second e 1 in T, we see that S T. x 3 1

2 10 ORTHOGONALITY 2 Orthogonal complement. Let S be a subspace of V. The orthogonal complement of S (in V), written S, is the set of all vectors in V that are orthogonal to every vector in S: S = {v V v s for all s S}. For instance, in R 3 the orthogonal complement of the x 1 x 2 -plane is the x 3 -axis. On the other hand, the orthogonal complement of the x 3 -axis is the x 1 x 2 -plane. It follows using the inner product axioms that if S is a subspace of V, then so is its orthogonal complement S. The next theorem says that in order to show that a vector is orthogonal to a subspace it is enough to check that it is orthogonal to each vector in a spanning set for the subspace. Theorem. Let {b 1,b 2,...,b n } be a set vectors in V and let S = Span{b 1,b 2,...,b n }. A vector v in V is in S if and only if v b i for each i. Proof. For simplicity of notation, we prove only the special case when n = 2 so that S = Span{b 1,b 2 }. If v is in S, then it is orthogonal to every vector in S, so, in particular, v b 1 and v b 2. Now assume that v V and v b 1 and v b 2. If s S, then we can write s = αb 1 +αb 2 for some α 1,α 2 R.

3 10 ORTHOGONALITY 3 So using properties of the inner product we get v,s = v,α 1 b 1 +α 2 b 2 = α 1 v,b 1 +α 2 v,b 2 (ii), (iii), and (iv) = α 1 0+α 2 0 v b 1 and v b 2 = 0. Therefore, v s. This shows that v is orthogonal to every vector in S so that v S. Theorem. If A is a matrix, then the orthogonal complement of the row space of A is the null space of A: (RowA) = NullA. Proof. Let Abe anm n matrixandlet b 1,b 2,...,b m be the rowsofawritten as columns so that RowA = Span{b 1,b 2,...,b m }. Let x be a vector in R n. The product Ax is the m 1 matrix obtained by taking the dot products of x with the rows of A: Ax = b T 1. b T m x = b T 1 x. b T mx so, in view of the previous theorem, x is in (RowA) if and only if b T i x = 0 for each i and this holds if and only if x is in NullA Example Let S be the subspace of R 4 spanned by the vectors [1,0, 2,1] T and [0,1,3, 2] T. Find a basis for S. Solution We first note that S = RowA, where [ ] A = According to the theorem, S = (RowA) = NullA so we need only find a basis for the null space of A. The augmented matrix we write to solve Ax = 0 is already in reduced row echelon form (RREF): [ ] so NullA = {[2t s, 3t+2s,t,s] T t,s R}. We have 2t s 2 1 3t+2s t = t 3 1 +s 2 0, s 0 1

4 10 ORTHOGONALITY 4 and therefore {[2, 3,1,0] T,[ 1,2,0,1] T } is a basis for NullA (and hence S ). The next theorem generalizes the notion that the shortest distance from a point to a plane is achieved by dropping a perpendicular. Theorem. Let S be a subspace of V, let b be a vector in V and assume that b = s 0 + r with s 0 S and r S. For every s S we have dist(b,s 0 ) dist(b,s). Proof. Let s S. We have b s 2 = (b s 0 )+(s 0 s) 2 = b s s 0 s 2 Pythagorean theorem b s 0 2 (notethat the Pythagoreantheoremappliessince b s 0 = r S ands 0 s S so that (b s 0 ) (s 0 s)). Therefore, dist(b,s 0 ) = b s 0 b s = dist(b,s) Least squares Suppose that the matrix equation Ax = b has no solution. In terms of distance, this means that dist(b,ax) is never zero, no matter what x is.

5 10 ORTHOGONALITY 5 Instead of leaving the equation unsolved, it is sometimes useful to find an x 0 that is as close to being a solution as possible, that is, for which the distance from b to Ax 0 is less than or equal to the distance from b to Ax for every other x. This is called a least squares solution. A least squares solution can be used, for instance, to find the line that best fits some data points (see Example ). Least squares. Let A be an m n matrix and let b be a vector in R n. If x = x 0 is a solution to A T Ax = A T b. then, for every x R n, dist(b,ax 0 ) dist(b,ax). Such an x 0 is called a least squares solution to the equation Ax = b. Proof. Let x 0 R n and assume that A T Ax 0 = A T b. Letting s 0 = Ax 0, we have A T (b s 0 ) = A T (b Ax 0 ) = 0, so that b s 0 NullA T = (RowA T ) (Theorem in Section 10.1) = (ColA) = S, where S = {Ax x R n } (the product Ax can be interpreted as the linear combination of the columns of A with the entries of x as scalar factors, so S is the set of all linear combinations of the columns of A, which is ColA). This shows that the vector r = b s 0 is in S, so that b = s 0 +r with s 0 S and r S. By the last theorem of Section 10.1, for every x R n, we have dist(b,ax 0 ) = dist(b,s 0 ) dist(b,ax) Example Use a least squares solution to find a line that best fits the data points (1,2), (3,2), and (4,5).

6 10 ORTHOGONALITY 6 Solution Here is the graph: If we write the desired line as c+dx = y, then ideally the line would go through all three points giving the system c+d = 2 c+3d = 2 c+4d = 5, which can be written as the matrix equation Ax = b, where A = 1 1 [ c 1 3, x =, b = d] The least squares solution is obtained by solving the equation A T Ax = A T b. We have [ ] A T A = 1 1 [ ] = and [ ] A T b = 2 2 = [ ] 9, 28 so the matrix equation has corresponding augmented matrix [ ] [ ] [ ] [ [ ] ]

7 10 ORTHOGONALITY 7 Therefore, c = 5 7 and d = 6 7 and the best fitting line is y = x, which is the line shown in the graph. (We could tell in advance that the matrix equation Ax = b has no solution since the points are not collinear. In general, however, it is not necessary to first check that an equation has no solution before applying the least squares method since, if it has a solution, then that is what the method will produce.) We can use the last example to explain the terminology least squares. The method gives a line that minimizes the sum of the squares of the vertical distances from the points to the lines. This sum of squares is s 2 1 +s2 2 +s2 3, with s 1, s 2, and s 3 as indicated: We have s 2 1 +s2 2 +s2 3 = ( 2 (c+d) ) 2 + ( 2 (c+3d) ) 2 + ( 5 (c+4d) ) 2 = b Ax 2 = dist(b,ax) 2 A least squares solution x = [c,d] T makes dist(b,ax) as small as possible and therefore makes s 2 1 +s 2 2 +s 2 3 as small as possible as well Example Use a least squares solution to find a parabola that best fits the data points ( 1,8), (0,8), (1,4), and (2,16). Solution Here is the graph:

8 10 ORTHOGONALITY 8 Ideally, a parabola would go through all four points, so if we write its equation as c+dx+ex 2 = y, then c d +e = 8 c = 8 c +d +e = 4 c+2d+4e = 16, which can be written as the matrix equation Ax = b, where A = , x = c 8 d, b = 8 4. e The least squares solution is obtained by solving the equation A T Ax = A T b. We have A T A = = and A T b = = 36 28,

9 10 ORTHOGONALITY 9 so the matrix equation has corresponding augmented matrix Therefore, c = 5, d = 1, and e = 3, and the best fitting parabola has equation y = 5 x+3x 2, which is the parabola shown in the graph Gram-Schmidt process The standard basis vectors e 1,e 2,...,e n in R n have two useful properties: they are pairwise orthogonal (any two are orthogonal), each is a unit vector (a vector of norm one). If we have a basis for an inner product space V that does not already have these properties, then we can change it into a basis that does by using the Gram-Schmidt process.

10 10 ORTHOGONALITY 10 Orthonormal set. Let {b 1,b 2,...,b s } be a set of vectors in the inner product space V. The set is orthogonal if b i,b j = 0 for all i j (the vectors are pairwise orthogonal). The set is orthonormal if it is orthogonal and each vector is a unit vector. Any orthogonal set of nonzero vectors can be changed into an orthonormal set by dividing each vector by its norm. For instance, {[1,1] T,[ 1,1] T } is an orthogonalset in R 2 and eachofthesevectorshas norm 2, so{[ 1 2, 1 2 ] T,[ 1 2, 1 2 ] T } is an orthonormal set: Let V be an inner product space. The Gram-Schmidt process requires a formula for the projection of a vector on a subspace S.

11 10 ORTHOGONALITY 11 Theorem. Let S be a subspace of V and let {u 1,u 2,...,u s } be an orthonormal basis for S. Let b be a vector in V and let p = s b,u i u i. i=1 Then p S and b p S : The vector p is the projection of b on S. Proof. We prove only the special case s = 2 so that {u 1,u 2 } is a basis for S and p = b,u 1 u 1 + b,u 2 u 2. Since p is a linear combination of u 1 and u 2, it is in S. We have b p,u 1 = b ( b,u 1 u 1 + b,u 2 u 2 ),u1 = b,u 1 b,u 1 u 1,u 1 b,u 2 u 2,u 1 (iii) and (iv) = b,u 1 b,u 1 (1) b,u 1 (0) u 1 is unit vector and u 1 u 2 = 0, so b p is orthogonal to u 1. Similarly, b p is orthogonal to u 2. Since S = Span{u 1,u 2 }, it follows from the theorem in Section 10.1 that b p is in S. We now describe the Gram-Schmidt process in the case of three initial vectors. Suppose that {b 1,b 2,b 3 } is a basis for an inner product space V. The Gram- Schmidt processusesthese vectorsto produce anorthonormal basis{u 1,u 2,u 3 } for V:

12 10 ORTHOGONALITY 12 Here are the first two steps of the process (the view is looking straight down at the plane spanned by b 1 and b 2 and we write v 1 v to mean v v): u 1 = b 1 b 1 u 2 = b 2 p 1 b 2 p 1 where p 1 = b 2,u 1 u 1 Here is the third step of the process: u 3 = b 3 p 2 b 3 p 2 where p 2 = b 3,u 1 u 1 + b 3,u 2 u 2 The general statement is as follows:

13 10 ORTHOGONALITY 13 Gram-Schmidt process. Let {b 1,b 2,...,b n } be a basis for the inner product space V. Define vectors u 1,u 2,...,u n recursively by u 1 = b 1 b 1 u k = b k p k 1 b k p k 1, where p k 1 k 1 = b k,u i u i (k > 1) Then {u 1,u 2,...,u n } is an orthonormal basis for V. Moreover, Span{u 1,u 2,...,u k } = Span{b 1,b 2,...,b k } for each k. i=1 We will not give a detailed proof. However, we note that for each k the vector b k p k 1 is in the orthogonal complement of Span{u 1,u 2,...,u k 1 } by the preceding theorem so that u k u i for 1 i < k Example Let b 1 = [1,2,2,4] T, b 2 = [ 2,0, 4,0] T, and b 3 = [ 1,1,2,0] T, and let S be the span of these vectors. Apply the Gram-Schmidt process to {b 1,b 2,b 3 } to obtain an orthonormal basis {u 1,u 2,u 3 } for S. Solution Sincethe givenvectorsarein R 4, the innerproductisthe dotproduct and the norm of a vector x is given by the formula x = x T x = x 2 1 +x2 2 +x2 3 +x2 4. We use the formula αx = α x to simplify computations. First, Next, and u 1 = b 1 b 1 = [1,2,2,4]T [1,2,2,4] T = 1 5 [1,2,2,4]T. p 1 = b 2,u 1 u 1 = [ 2,0, 4,0] T, 1 5 [1,2,2,4]T u 1 = 2u 1 = 2 5 [1,2,2,4]T b 2 p 1 = [ 2,0, 4,0] T [1,2,2,4]T = 4 5 [ 2,1, 4,2]T, so u 2 = b 4 2 p 1 b 2 p 1 = 5 [ 2,1, 4,2]T 4 5 [ 2,1, 4,2]T = 1 5 [ 2,1, 4,2]T.

14 10 ORTHOGONALITY 14 Finally, p 2 = b 3,u 1 u 1 + b 3,u 2 u 2 = [ 1,1,2,0] T, 1 5 [1,2,2,4]T u 1 + [ 1,1,2,0] T, 1 5 [ 2,1, 4,2]T u 2 = u 1 u 2 = 1 5 [1,2,2,4]T 1 5 [ 2,1, 4,2]T = 1 5 [3,1,6,2]T and b 3 p 2 = [ 1,1,2,0] T 1 5 [3,1,6,2]T = 2 5 [ 4,2,2, 1]T, so u 3 = b 2 3 p 2 b 3 p 2 = 5 [ 4,2,2, 1]T 2 5 [ 4,2,2, 1]T = 1 5 [ 4,2,2, 1]T. Therefore, { 1 5 [1,2,2,4]T, 1 5 [ 2,1, 4,2]T, 1 5 [ 4,2,2, 1]T } is an orthonormal basis for S Example Let S be the span of the functions 1, x, and x 2 in C [0,1]. Apply the Gram-Schmidt process to {1,x,x 2 } to obtain an orthonormal basis {u 1,u 2,u 3 } for S. Solution The inner product in C [0,1] is given by f,g = 1 f(x)g(x)dx. Let 0 b 1 = 1, b 2 = x, and b 3 = x 2. First, so b 1 = b 1,b 1 = 1,1 = 1 0 1dx = 1, u 1 = b 1 b 1 = 1. Next, ( 1 p 1 = b 2,u 1 u 1 = x,1 u 1 = and b 2 p 1 = x 1 2, 0 ) xdx u 1 = 1 2 u 1 = 1 2 so u 2 = b 2 p 1 b 2 p 1 = x 1 2 x 1 2 = x 1 2 = 2 3(x (x 1 2 ) = 3(2x 1). 2 )2 dx

15 10 ORTHOGONALITY 15 Finally, and so p 2 = b 3,u 1 u 1 + b 3,u 2 u 2 = x 2,1 u 1 + x 2, 3(2x 1) u 2 ( 1 ) ( 3 1 ) = x 2 dx u 1 + x 2 (2x 1)dx u 2 = 1 3 u u = 1 3 (1)+ ( ) 3 6 3(2x 1) = x 1 6 b 3 p 2 = x 2 (x 1 6 ) = x2 x+ 1 6, u 3 = b 3 p 2 b 3 p 2 = x2 x+ 1 6 x 2 x+ 1 6 = x 2 x (x2 x+ 1 6 )2 dx = 6 5(x 2 x+ 1 6 ) = 5(6x 2 6x+1). Therefore {1, 3(2x 1), 5(6x 2 6x+1)} is an orthonormal basis for S. 10 Exercises 10 1 Let x 1 = [ 1,0,1] T, x 2 = [0,1, 1] T, and x 3 = [1,1,1] T. Show that S T, where S = Span{x 1,x 2 } and T = Span{x 3 } Let S be the subspace of P 3 spanned by x and x 2. Find S, where the inner product on P 3 is as in Section 9.5 with x 1 = 1, x 2 = 0, and x 3 = 1. Hint: By the first theorem of Section 10.1 a polynomial p = ax 2 +bx+c is in S if and only if it is orthogonal to both x and x 2.

16 10 ORTHOGONALITY Let S be the subspace of R 4 spanned by the vectors [1,3, 4,0] T and [ 2, 6,8,1] T. (a) Find a basis for S. (b) Verify that the basis vectors you found in (a) are orthogonal to the given vectors Use a least squares solution to find a line that best fits the data points (1,2), (2,4), and (3,3) Usealeastsquaressolutiontofindacurveoftheformy = acosx+bsinx that best fits the data points (0,1), (π/2,2), and (π,0). Answer: y = 1 2 cosx+2sinx Let {u 1,u 2,u 3 } be an orthonormal set in an inner product space V. Prove that u 1, u 2, and u 3 are linearly independent. Hint: Suppose that α 1 u 1 +α 2 u 2 +α 3 u 3 = 0. Apply,u 1 to both sides of this equation and use properties of the inner product to conclude that α 1 = 0. Note that similarly α 2 = 0 and α 3 = 0. You may use the fact that 0,v = 0 for every vector v V Let v and w be vectors in an inner product space V and assume that w is nonzero. Let p = v,w w,w w The vector p is the projection of v on w.

17 10 ORTHOGONALITY 17 (a) Show that (v p) w. (b) Find p when v = [1,2] T and w = [3,1] T and sketch Let b 1 = [0,0, 1,1] T, b 2 = [1,0,0,1] T, and b 3 = [1,0, 1,0] T, and let S be the span of these vectors in R 4. Apply the Gram-Schmidt process to {b 1,b 2,b 3 } to obtain an orthonormal basis {u 1,u 2,u 3 } for S. Answer: { 1 2 [0,0, 1,1] T, 1 6 [2,0,1,1] T, 1 3 [1,0, 1, 1] T } Let S be the span of the functions x, x+1, and x 2 1 in C [0,1]. Apply the Gram-Schmidt process to {x,x+1,x 2 1} to obtain an orthonormal basis {u 1,u 2,u 3 } for S. Answer: { 3x, 3x+2, 5(6x 2 6x+1)}.