Chapter 5 Orthogonality

Transcription

1 Matrix Methods for Computational Modeling and Data Analytics Virginia Tech Spring 08 Chapter 5 Orthogonality Mark Embree embree@vt.edu Ax=b version of February 08 We needonemoretoolfrom basic linear algebra to tackle the applications that lie ahead. In the last chapter we saw that basis vectors provide the most economical way to describe a subspace, but not all bases are created equal. Clearly the following three pairs of vectors all form bases for IR : apple 0, apple 0 ; apple p / p /, apple p / p / ; apple 0, apple,.00. For example, the vector x =[, ] T can be written as a linear combination of all three sets of vectors. In the first two cases, apple = apple apple = p apple p apple p / / p + 0 p, / / the coefficients multiplying against the basis vectors are no bigger than kxk = p. In the third case, apple apple apple = , the coefficients 999 and 000 are much bigger than kxk. Small changes in x require significant swings in these coefficients. For example, if we change the second entry of x from to 0.9, apple.9 apple apple = , the coefficients change by 00, which is 000 times the change in x!

2 6 The first two bases above are special, for the norm of each vector is one, and the vectors are orthogonal. The vectors in the third basis form a small angle. Indeed, all these bases are linearly independent, but, extrapolating from Animal Farm, some bases are more linearly independent than others. This chapter is devoted to the art of orthogonalizing a set of linearly independent vectors. The main tool of this craft is the orthogonal projector. 5. Projectors Among the most fundamental operations in linear algebra is the projection of a vector into a particular subspace. Such actions are encoded in a special class of matrices. Definition 8 A matrix P IR n n is a projector provided P = P. If additionally P = P T, then P is called an orthogonal projector. The simplest projectors send all vectors into a one-dimensional subspace. Suppose that w T v =. Then P := vw T is a projector, since P =(vw T )(vw T )=v(w T v)w T = vw T = P. We say that P projects onto span{v}, since and along span{w}?, since R(P) =span{v}, N(P) =span{w}? = {x IR n : x T w = 0}. Example Consider the vectors Confirm that w T v =, so v = is a projector. It sends the vector apple apple 0, w =. P = vw T = x = apple 0 0 apple,

3 6 to Px = apple 0 0 apple = apple 0, which is contained in R(P) =span(v). The remainder, span{w} x Px = is indeed orthogonal to span(w). apple apple 0 apple = x Px w x The remainder x Px plays a key role in our development. Notice that P annihilates this vector, P(x Px) =Px P x = Px Px = 0, which means that x Px N(P). v Px The projector P sends the vector x to Px R(P) = span{v}, while the remainder x Px is in N(P) = span{w}?, i.e., x Px is orthogonal to w. span{v} For the projector in the last example, P 6= P T, so we call P an oblique projector. For such projectors, the remainder x Px generally forms oblique = not 90 an oblique angle with R(P). In this course we shall most often work with orthogonal projectors, for which P = P T. Of course this implies N(P) =N(P T ), and so, the Fundamental Theorem of Linear Algebra s N(P T )? R(P) becomes N(P)? R(P). Since x Px N(P), the remainder is always orthogonal to R(P). This explains why we call such a P an orthogonal projector. To build an orthogonal projector onto a one-dimensional subspace, we need only a single vector v. Set P = vv T, and, to give P = P, require that v T v = kvk =, i.e., v must be a unit vector. This P projects onto and along span{v}, since R(P) =span{v}, N(P) =span{v}?. The notation span{v}? denotes the set of all vectors orthogonal to span{v}. We illustrate these ideas in the following example. Example Use the unit vector v from the last example to build apple 0 P = vv T = 0 0,

4 64 which projects vectors onto the x axis. For the same x used before, we have apple apple apple 0 Px = = Now observe the orthogonality of the remainder apple 0 x Px = with the projection subspace R(P) =span{v}. Did you notice that, in contrast to the first example, the projection Px has smaller norm than x, kpxk applekxk? Such reduction is a general trait of orthogonal projectors. x Px x v Px span{v} The orthogonal projector P sends the vector x to Px R(P) =span{v}, while the remainder x Px is contained in N(P) =span{v}?, i.e., x Px is orthogonal to v and Px. Proposition If P is an orthogonal projector, then kpxk apple kxk for all x IR n. Proof. Since P is an orthogonal projector, P T = P and P = P. Hence Px is orthogonal to x Px: (Px) T (x Px) =x T P T x x T P T Px = x T Px x T P x = x T Px xpx = 0. Now apply the Pythagorean Theorem to the orthogonal pieces Px R(P) and x Px N(P), kxk = kpx +(x Px)k = kpxk + kx Pxk, and conclude the result by noting that kx Pxk 0. student experiments 5.8. To get an appreciation for projectors, in matlab construct the orthogonal projector P onto apple span. Now construct 500 random vectors x = randn(, ). Produce a figure showing all 500 of these vectors in the plane (plotted as blue dots), then superimpose on the plot the values of Px (plotted as red dots). Use axis equals to scale the axes in the same manner. Repeat the experiment, but now in three dimensions, with P the orthogonal projector onto 8 9 < = span 4 5 : ;. Use matlab s plot command plot x and Px in three dimensional space.

5 65 5. Orthogonalization Now we arrive at the main point of this chapter: the transformation of basis into an orthonormal basis for the same set. Definition 9 A basis {q,...,q n } is orthonormal provided q T j q k = 0 for all j 6= k (the vectors are orthogonal); kq j k = for j =,..., n (the vectors are normalized). We can summarize orthonormality quite neatly if we arrange the basis in the matrix Q = [ q q q n ]. Then Q T Q contains all possible inner products between q j and q k : since q T j q j = kq jk =. q Tq q Tq q Tq n Q T q T Q = q q Tq q Tq n = I, qn T q qn T q qn T q n Definition 0 A matrix Q IR n n is unitary if Q T Q = I. If Q IR m n with m > n and Q T Q = I, then Q is subunitary. student experiments 5.9. Suppose Q IR m n with m < n. Is it possible that Q T Q = I? Explain Suppose that Q IR m n is subunitary. Show that QQ T is a projector. What is R(Q) in terms of q,...,q n? We are now prepared to orthogonalize a set of vectors. Suppose we have a basis a,...,a n for a subspace S C m. We seek an orthonormal basis q,...,q n for the same subspace, which we shall span{a } build one vector at a time. The goal is to first construct unit vector q a P a so that span{q } = span{a } and then to build unit vector q orthogonal to q so that span{q, q } = span{a, a } and then unit vector q orthogonal to q and q so that span{q, q, q } = span{a, a, a } a a span{a } The setting: basis vectors a and a that are neither unit length, nor orthogonal.

6 66 and so on, one q j vector at a time, until we have the whole new basis: span{q, q,...,q n } = span{a, a,...,a n } = S. span{a } The first of these steps is easy, for we get q by normalizing a : a P a a q = ka k a. The next step is decisive, and here we use insight gained from our quick study of orthogonal projectors. Figuratively speaking, we must remove the part of a in the direction q, leaving the portion of a orthogonal to q. Since q is a unit vector, q First normalize a to obtain q with span{q } = span{a }. a span{a } P := q q T is an orthogonal projector onto span{q } = span{a }, so P a = q q T a a P a a span{a } is the part of a in the direction q. Remove that from a to get bq := a P a. And since P bq = P (a P a )=0, q P a a span{a } Compute cq = a P a, the portion of a that is orthogonal to q. spot that bq N(P ), which is orthogonal to R(P )=span{q }, so bq? q. If you prefer a less high-falutin explanation, just compute q T bq = qt a q T q qt a = qt a q T a = 0. So we have constructed a vector bq orthogonal to q such that span{q, bq } = span{a, a }. Since we want not just an orthogonal basis, but an orthonormal basis, we adjust bq by scaling it to become the unit vector q := kbq k bq. q a P a a span{a } The orthogonalization process for subsequent vectors follows the same template. To construct q, we first remove from a its components in the q and q directions: q P a a Normalize q = cq /kcq k to get a unit vector in the cq direction. span{a } bq := a P a P a, where P = q q T, where for j =,, q T j bq = qt j a q T j q qt a q T j q qt a = q T j a q T j a = 0,

7 67 using the orthonormality of q and q. Normalize bq to get q := kbq k bq. Now the general pattern should be evident. For future vectors, construct bq k+ = a k+ k  P j a k+, with P j := q j q T j, j= then normalize giving q k+ = kbq k+ k bq k+, q k+? span{q,...,q k } which extends the orthonormal basis in one more direction: span{q,...,q k, q k+ } = span{a,...,a k, a k+ }. This algorithm is known as the Gram Schmidt process. student experiments 5.. Under what circumstances will this procedure break down? That is, will you ever divide by zero when trying to construct q k+ = bq k+ /kbq k+ k? 5.. Show that P k = P + + P k is an orthogonal projector. With this notation, explain how we can compactly summarize each Gram Schmidt step as q k+ = (I P k)a k+ k(i P k )a k+ k. 5. Gram Schmidt is QR factorization The Gram Schmidt process is a classical way to orthogonalize a basis, and you can execute the process by hand when the vectors are short (m is small) and there are few of them (n is small). As with Gaussian elimination, we want to automate the process for larger problems and just as Gaussian elimination gave us the A = LU factorization of a square matrix, so Gram Schmidt will lead to a factorization of a general rectangular matrix. Let us work through the arithmetic behind the orthogonalization of three linearly independent vectors a, a, and a, and along the

8 68 way define some quantities r j,k that arise in the process: bq := a q := kbq k bq = r, bq bq := a q q T a = a r, q q := kbq k bq = r, bq bq := a q q T a q q T a = a r, q r, q q := kbq k bq = bq. r, To summarize, we have defined 8 >< q T j a k, j < k; r j,k = kbq j k, j = k; >: 0, j > k. With this notation, three steps of the Gram Schmidt process become: r, q = a r, q = a r, q r, q = a r, q r, q, or, collecting the a j vectors on the left hand side: a = r, q + 0q + 0q a = r, q + r, q + 0q a = r, q + r, q + r, q. Stack the orthonormal basis vectors as the columns of the subunitary matrix Q IR m n Q = [ q q q ], and note that a = a = 4 q q q 5 4 q q q r, r, r, 0 5

9 69 a = 4 q q q 5 4 r, r, r, which we organize in matrix form as r, r, r, 4 a a a 5 = 4 q q q r, r, r, We summarize the entire process as: A = QR, where Q IR m n is subunitary and R IR n n is upper triangular. Now change your perspective: let A IR m n be any matrix with linearly independent columns, m n. Performing the Gram Schmidt process on those columns yields the decomposition A = QR, which is known as the QR factorization. But wait, there s more! Suppose b R(A), so there exists some x IR n such that Ax = b. Substitute the QR factorization to obtain QRx = b. Since we have written b = Q(Rx), we see that b R(Q): since b was any vector in the column space of A, we conclude that R(A) R(Q). Similarly, if b R(Q), then there exists x IR n such that Qx = b. Since R is invertible, splice I = RR into this last equation to obtain QR(R x)=b: so A(R x)=b, hence b R(A), and R(Q) R(A). Since R(Q) and R(A) each contain the other, they must be equal. Finally, notice that P = QQ T IR m m is an orthogonal projector: 5, P = QQ T QQ T = QQ T = P, since the columns of Q are orthonormal, Q T Q = I, and (QQ T ) T = (Q T ) T Q T = QQ T. In fact, one can confirm that QQ T 6 = 4 q q n q T. q T n 7 5 = q q T + q nq T n. Notice that R(P) =R(Q), so P projects onto R(Q) =R(A). Theorem 7 Let A = QR be a QR factorization of a matrix A IR m n with linearly independent columns, with Q IR m n and R IR n n. Then R(Q) =R(A) and QQ T is an orthogonal projector onto R(A). 5.4 QR solves systems Suppose we wish to solve Ax = b for a square matrix A with linearly independent columns. The linear independence implies that

10 70 A is invertible. Rather than perform Gaussian elimination, we could alternatively replace A with its QR factorization, QRx = b. Since Q is subunitary, Q T Q = I, so the Q term on the left can be cleared by premultiplying by Q T : Rx = Q T b. (5.) Since R is upper triangular, this Rx = Q T b system can be solved for x by back-substitution, just as with the last step in Gaussian elimination. We can formally write the solution as x = R Q T b. In general, the R factor in a QR factorization will be different from the U factor in an LU factorization, though both are upper triangular matrices. But is R invertible? Yes: since the columns of A are linearly independent, the diagonal entries of R obtained via the Gram Schmidt process are always nonzero. student experiments 5.. The discussion above assumed A was a square matrix. Suppose A IR m n with m > n, and A has n linearly independent columns. If b R(A), write down a solution x to Ax = b in terms of the QR factorization of A. Is this solution unique? 5.5 Least squares, take two Suppose that, instead of solving Ax = b, we instead wish to find x to minimize min kb Axk, xc n the least squares problem discussed in the last chapter. We saw there that x minimized the misfit b Ax when A T Ax = A T b, and that this x was unique provided N(A) ={0}. Throughout this chapter we have assumed that a,...,a n are linearly independent, giving the matrix A = [ a a a n ] linearly independent columns: hence N(A) = {0}, and the least squares solution is the unique x that satisfies Substitute A = QR for A to get A T Ax = A T b. (QR) T (QR)x =(QR) T b

11 7 which is equivalent to R T Q T QRx = R T Q T b. Since Q is subunitary (its columns are orthogonal), Q T Q = I, so our x solves R T Rx = R T Q T b. (5.) Note that R T is a lower-triangular matrix with nonzero entries on the main diagonal, so it is invertible. Multiply both sides of (5.) by (R ) to obtain Rx = Q T b, (5.) which, remarkably enough, is the same equation we obtained in (5.) for solving Ax = b when b R(A). But is the agreement of equations (5.) and (5.) really so remarkable? Insert I = Q T Q on the right-hand side of (5.) to obtain Rx = Q T QQ T b matlab captures this idea in its backslash command, \. When you type A\b, you will obtain x = A b when A is a square invertible matrix, and the x that minimizes kb Axk when A IR m n with m > n having linearly independent columns. = Q T (QQ T )b = Q T b R, noting that QQ T is the orthogonal projector onto R(A), and b R = QQ T b is the vector given by the FTLA decomposition: b =(QQ T b)+(i QQ T )b = b R + b N. So implicitly, equation (5.) projects b IR n to b R, to give an equation Rx = Q T b R that has a solution x.