Worksheet for Lecture Name: Section.4 Gram-Schmidt Process Goal For a subspace W = Span{v,..., v n }, we want to find an orthonormal basis of W. Example Let W = Span{x, x } with x = and x =. Give an orthogonal basis of W. Solution Step I Take v = x. Step II Take v = x proj v x. More precisely, take v = x x v v = + + + + = =. 4 { Then v =, v = } is an orthogonal basis of W. Theorem (The Gram Schmidt Process) Given a basis {x,..., x p } for a nonzero subspace W of R n, define v = x v = x x v v v = x x v v x v v v v.. v p = x p x p v v x p v p v p v p v p Then {v,..., v p } is an orthogonal basis for W. Example Let x =, x =, x =. Give an orthogonal basis of W = Span{x, x, x }, and then give an orthonormal basis of W. Solution Step I Set v = x =.
Step II Set /4 v = x x v v = + + + + + + = /4 /4. /4 To make our computation later easier, we can use v = 4 v =. Step III Set v = x x v v x v v v v = + + + + + + + + + ( ) + + + = 4 = / /. / It might be better to use v = v =. Therefore, an orthogonal basis of W is { },,. { /4 } (Remark, it is not necessary to use v, v, v, and, /4 /4, / / /4 / correct answer.) is also a
Step IV Normalize them: / u = v v = + + + = / /. / / u = v v = ( ) + + + = / / /. u = v v = + ( ) + + = /. / / QR Factorization If A is an m n matrix with linearly independent columns, then A can be factored as A = QR, where Q is an m n matrix whose columns form an orthonormal basis for Col(A) and R is an n n upper triangular invertible matrix with positive entries on its diagonal. Example We use the example above, with A = Solution We use the computation above with / / u = / /, u = / / / /, u = / / /. Then the needed matrix Q is / / Q = / / / / / / / / / Since Q T Q = I, we must have Q T A = Q T QR R = Q T A
4 So we have / / / / R = / / / / / / / / = / / /. Application of QR decomposition In practical problems, we sometimes want to solve linear equations Ax = b, but the system is very inconsistent. For example, we expect certain number y should be of the form a sin x + b cos x, for some number a and b. So we conduct experiments to get lots of data points (x i, y i ). Then we hope to solve the equation y = a sin x + b cos x y = a sin x + b cos x y n = a sin x n + b cos x n To solve for a and b, we need to solve a linear system like sin x sin x y sin x sin x y... sin x n sin x n y n But we all know that experiments come with errors, and in practice, there is almost no way for this system with say n = to be consistent. Least-square solution We hope to get a pair (a, b) that closest to being a solution. Here is the theoretic meaning of this: the system isconsistent exactly when the column y sin x cos x vector y =. lies in the span of. and., a.k.a. Col(A). In general y n sin x n cos x n this is not the case, so we choose the projection of y to Col(A). The projection, which sin x cos x can be written as a. + b. is the vector in Col(A) that is closest to the sin x n cos x n experimental result y. This is called the least-square solution. In terms of formula, if the matrix A has a QR factorization, we have ˆx = R Q T b. (Heuristically, we are doing x = A b = R Q b = R Q T b. Except that this does not make sense as Q is not a square matrix. So even though Q T Q = I, it doesn t make Q exist.)
True/False Questions () If {v, v, v } is an orthogonal basis for W, then multiplying v by a scalar c gives a new orthogonal basis {v, v, cv }. True. () If A = QR, where Q has orthonormal columns, then R = Q T A. True. From A = QR, we deduce that Q T A = Q T QR = R. () If W = Span{x, x, x } with {x, x, x } linearly independent, and if {v, v, v } is an orthogonal set of nonzero vectors in W, then {v, v, v } is a basis for W. True. The condition says that dim W =. An orthogonal set is always linearly independent; so {v, v, v } is a basis for W. (4) If x is not in a subspace W, then x proj W x is not zero. True. x is in the subspace W if and only if x = proj W x. () In a QR factorization, say A = QR (when A has linearly independent columns), the columns of Q form an orthonormal basis for the columns space of A. True. Exercise Using Gram-Schmidt process to produce an orthogonal basis for { } W = Span, 9 9. Solution Set v =. Then set v = x x v v = 9 ( ) + 9 ( ) + ( 9) + ( ) 9 + ( ) + + ( ) 4 = 9 9 4 =.
So an orthogonal basis of W is { 4 },. Exercise Using Gram-Schmidt Span{ process to produce an orthonormal basis for W = 4, 4 }. 7 Solution Set v = 4. Then set Normalize them as follows: v = x x v v = 4 + 4 7 7 + 4 + 4 = 4 4 = 4 4. 7 8 u = v v = + 4 + u = v v = 4 4 4 + 4 + ( 8) = 4 = / 4 /. = 8 4 = 4 So an orthonormal basis of W is { / / /, / } /. Exercise Give the QR factorization for the matrix 9 A = 7 / / /.
Solution Set v =. Then set 9 v = x x v v = 7 9 + 7 + ( ) ( ) + + + ( ) + 9 = 7 7 =. We normalize them as follows: / u = v v = + + ( ) + = = / /. / / u = v v = ( ) + + + = =. / / / So the matrix Q is / / Q = / / / / / / The corresponding matrix R is ( ) 9 ( ) R = Q T / / / / A = 7 / / / / =. 7