Section 6.4 The Gram Schmidt Process
Motivation The procedures in 6 start with an orthogonal basis {u, u,..., u m}. Find the B-coordinates of a vector x using dot products: x = m i= x u i u i u i u i Find the orthogonal projection of a vector x onto the span W of u, u,..., u m: m x u i proj W (x) = u i. u i u i i= Problem: What if your basis isn t orthogonal? Solution: The Gram Schmidt process: take any basis and make it orthogonal.
The Gram Schmidt Process Procedure The Gram Schmidt Process Let {v, v,..., v m} be a basis for a subspace W of R n. Define:. u = v v u. u = v proj Span{u }(v ) = v u u u v3 u v3 u 3. u 3 = v 3 proj Span{u,u }(v 3) = v 3 u u u u u u. m m. u m = v m proj Span{u,u,...,u m } (vm) = vm v m u i u i u i u i Then {u, u,..., u m} is an orthogonal basis for the same subspace W. Remark In fact, for every i between and n, the set {u, u,..., u i } is an orthogonal basis for Span{v, v,..., v i }. i=
The Gram Schmidt Process Example : Two vectors Find an orthogonal basis {u, u } for W = Span{v, v }, where v = and v =. Run Gram Schmidt:. u = v. u = v Why does this work? First we take u = v. Because u v, we can t take u = v. Fix: let L = Span{u }, and let u = (v ) L = v proj L (v ). By construction, u u =, because L u. v u u = =. u u W v = u Remember: This is an orthogonal basis for the same subspace. Span{u, u } = Span{v, v } v u = (v ) L L
The Gram Schmidt Process Example : Three vectors Find an orthogonal basis {u, u, u 3} for W = Span{v, v, v 3} = R 3, where v = v = 3 v 3 =. Run Gram Schmidt:. u = v. u = v v u u = = u u v3 u v3 u 3. u 3 = v 3 u u u u u u 3 = 4 = Remember: This is an orthogonal basis for the same subspace W.
The Gram Schmidt Process Example, continued 3 v =, v =, v 3 = G S u =, u =, u 3 = Why does this work? Check: Once we have u and u orthogonal, let W = Span{u, u }, and u 3 = (v 3) W = v 3 proj W3 (u 3). By construction, W u 3, so u u 3 = = u u 3. u u = " u u 3 = " u u 3 = " proj W (v 3 ) W u u v 3 u 3 L y z x
The Gram Schmidt Process Example 3: Vectors in R 4 Find an orthogonal basis {u, u, u 3} for W = Span{v, v, v 3}, where 4 v = v = 4 4 v 3 =. Run Gram Schmidt:. u = v. u = v 5/ v u u = 4 u u 4 6 4 = 5/ 5/ 5/ v3 u v3 u 3. u 3 = v 3 u u u u u u 4 5/ = 4 5/ 5 5/ = 5/
QR Factorization Recall: A set of vectors {v, v,..., v m} is orthonormal if they are orthogonal unit vectors: v i v j = when i j, and v i v i =. Orthonormal A matrix Q has orthonormal columns if and only if Q T Q = I. QR Factorization Theorem Let A be a matrix with linearly independent columns. Then A = QR where Q has orthonormal columns and R is upper-triangular with positive diagonal entries. The columns of A are a basis for W = Col A. The columns of Q are equivalent basis coming from Gram Schmidt (as applied to the columns of A), after normalizing to unit vectors. The columns of R come from the steps in Gram Schmidt.
Procedure: QR Factorization Through an example Find the QR factorization of A =. (The columns of A are the vectors v, v, v 3 from example.) Step : Run Gram Schmidt and solve for v, v, v 3 in terms of u, u, u 3. u = v = v = u v u u = v u = v u = v = u + u u u v3 u v3 u u 3 = v 3 u u u u u u = v 3 u u = v 3 = u + u + u 3
QR Factorization Through an example, continued Step : Write A = Q R, where Q has orthogonal columns u, u, u 3 and R is upper-triangular (with s on the diagonal) as shown below. v = u v = u + u v 3 = u + u + u 3 A v v v 3 = u u u 3 Q R first column of A = u u u 3 = u = v second column of A = u u u 3 = u + u = v third column of A = u u u 3 = u + u + u 3 = v 3
QR Factorization Through an example, continued A = Q R = Step 3: To get Q and R, scale the columns of Q to get unit vectors, and scale the rows of R by the opposite factor. / / / = / / / / / /. It doesn t change the product: the entries in the ith column of Q multiply by the entries in the ith row of R. The final QR decomposition is: / / A = QR Q = / / R =
QR Factorization Through a second example 4 Find the QR factorization of A = 4 4. (The columns are vectors from example 3.) Step : Run Gram Schmidt and solve for v, v, v 3 in terms of u, u, u 3: u = v = v = u 5/ v u u = v u = v 3 u u u = 5/ u 3 = v 3 5/ 5/ v3 u v3 u u u = v 3 + 4 u u u u 5 u = v = 3 u + u v3 = 4 u + u3 5
QR Factorization Through a second example, continued v = u v = 3 u + u v3 = u 4 u + u3 5 Step : Write A = Q R, where Q has orthogonal columns u, u, u 3 and R is upper-triangular with s on the diagonal. 5/ Q = u u u 3 = 5/ 5/ 5/ 3/ R = 4/5
QR Factorization Through a second example, continued 5/ A = Q R Q = 5/ 5/ 5/ 3/ R = 4/5 Step 3: To get Q and R, normalize the columns of Q and scale the rows of R: Q = u / u u / u u 3/ u 3 u 3/ u u R = u 4/5 u u 3 The final QR decomposition is / / / A = QR Q = / / 3 / / / / / R = 5 4.
Extra: computing determinants Consider the QR factorization of an invertible n n matrix: A = QR. det(r) is easy to compute because it is upper-triangular det(q) = ± (see below) Why: Q is orthonormal, Q T Q = I n, so Q T = Q. Also det(q T ) = det(q), = det(i n) = det(q T Q) = det(q T ) det(q) = det(q) ; so det(q) can take only two values: ±. Determinant (up to sign) If v, v,..., v n are the columns of A, and u, u,..., u n are the vectors obtained by applying Gram Schmidt, then det(a) = det(q) det(r) = ± u u u n ; Because the (i, i) entry of R is u i. Moreover, det(r) > so det(q) has the same sign as det(a).
Extra: computing eigenvalues The QR algorithm Let A be an n n matrix with real eigenvalues. Here is the algorithm: A = Q R A = R Q QR factorization swap the Q and R = Q R find its QR factorization A = R Q swap the Q and R = Q 3R 3 find its QR factorization et cetera Theorem The matrices A k converge to an upper triangular matrix whose diagonal entries are the eigenvalues of A. Moreover, the convergence is fast! The QR algorithm The algorithm above gives a computationally efficient way to find the eigenvalues of a matrix.