Math 9 Applied Linear Algebra Lecture : Orthogonal Projections, Gram-Schmidt Stephen Billups University of Colorado at Denver Math 9Applied Linear Algebra p./
Orthonormal Sets A set of vectors {u, u,..., u p } in R n is called an orthonormal set if. It is orthogonal.. Each vector has length. If the orthonormal set {u, u,..., u p } spans a vector space W, then {u, u,..., u p } is called an orthonormal basis for W. Math 9Applied Linear Algebra p./
Orthogonal Matrices Recall that v is a unit vector if v = v v = v T v =. Suppose U = [u u u ] where {u, u, u } is an orthonormal set. Then U T U = u T u T u T [u u u ] = u T u u T u u T u u T u u T u u T u u T u u T u u T u = = It can be shown that UU T = I also. So U = U T (such a matrix is called an orthogonal matrix). (NOTE: U must be square to be orthogonal). Math 9Applied Linear Algebra p./
THEOREM U T U = I. An m n matrix U has orthonormal columns if and only if THEOREM Let U be an m n matrix with orthonormal columns, and let x and y be in R n. Then a. Ux = x b. (Ux) (Uy) = x y c. (Ux) (Uy) = if and only if x y =. Proof of part b: (Ux) (Uy) = Math 9Applied Linear Algebra p./
Section. Orthogonal Sets Review: by = y u u u u is the orthogonal projection of onto. y u Suppose {u,..., u p } is an orthogonal basis for W in R n. For each y in W, y = y u y up u u u + + u p u p u p Math 9Applied Linear Algebra p./
EXAMPLE Suppose {u, u, u } is an orthogonal basis for R and let W =Span{u, u }. Write y in R as the sum of a vector by in W and a vector z in W. W ƒ y z W u y u Math 9Applied Linear Algebra p./
Solution: Write y = y u u u + y u u u u u + y u u u u where by= y u u u + y u u u u u z = y u u u u. To show that z is orthogonal to every vector in W, show that z is orthogonal to the vectors in {u, u }. Since z u = = = z u = = = Math 9Applied Linear Algebra p./
THEOREM 8 THE ORTHOGONAL DECOMPOSITION THEOREM Let W be a subspace of R n. Then each y in R n can be uniquely represented in the form y =by + z where by is in W and z is in W. In fact, if {u,..., u p } is any orthogonal basis of W, then and z = y by. by = y u y up u u u + + u p u p u p The vector by is called the orthogonal projection of y onto W. Math 9Applied Linear Algebra p.8/
y z W y =proj W y Math 9Applied Linear Algebra p.9/
EXAMPLE: Let u =, u =, and y =. Observe that {u, u } is an orthogonal basis for W =Span{u, u }. Write y as the sum of a vector in W and a vector orthogonal to W. Solution: proj W y = by = y u u u u + y u u u u = ( ) + ( ) = z = y by = = 9 Math 9Applied Linear Algebra p./
Geometric Interpretation of Orthogonal Projections y y y u u u u u y u u u u u Math 9Applied Linear Algebra p./
THEOREM 9 The Best Approximation Theorem Let W be a subspace of R n, y any vector in R n, and by the orthogonal projection of y onto W. Then by is the point in W closest to y, in the sense that y by < y v for all v in W distinct from by. y z W y =proj W y Math 9Applied Linear Algebra p./
Outline of Proof Let v in W distinct from by. Then v by is also in W (why?) z = y by is orthogonal to W y by is orthogonal to v by y v = (y by) + (by v) = y v = y by + by v. y v > y by Hence, y by < y v. Math 9Applied Linear Algebra p./
EXAMPLE Find the closest point to y in Span{u, u } where y =, u =, and u =. Solution: by= y u u u u + y u u u u = ( ) + ( ) = Math 9Applied Linear Algebra p./
Another View of matrix Multiplication Part of Theorem below is based upon another way to view matrix multiplication where A is m p and B is p n AB = h col A col A col p A i row B row B. row p B = (col A) (row B) + + (col p A) (row p B) Math 9Applied Linear Algebra p./
For example = = h i + h i = Math 9Applied Linear Algebra p./
h So if U = u u u p i. Then U T = u T u T.. So u T p UU T = u u T + u u T + + u p u T p T `UU y = `u u T + u u T + + u p u T p y = `u u T y + `u u T y + + `up u T p y = u `ut y + u `ut y + + u p `ut p y = (y u ) u + (y u ) u + + (y u p ) u p `UU T y = (y u ) u + (y u ) u + + (y u p ) u p Math 9Applied Linear Algebra p./
THEOREM If {u,..., u p } is an orthonormal basis for a subspace W of R n, then If U = h proj W y = (y u ) u + + `y u p up u u u p i, then proj W y =UU T y for all y in R n. Outline of Proof: proj W y = y u y up u u u + + u p u p u p = (y u ) u + + `y u p up = UU T y. Math 9Applied Linear Algebra p.8/
Section. The Gram-Schmidt Process Goal: Form an orthogonal basis for a subspace W. EXAMPLE: Suppose W =Span{x, x } where x = Find an orthogonal basis {v, v } for W. and x =. Math 9Applied Linear Algebra p.9/
Let v = x =. by= proj v x = x v v v v and v = x by = x x v v v v = = (component of x orthogonal to x ) Math 9Applied Linear Algebra p./
EXAMPLE Suppose {x, x, x } is a basis for a subspace W of R. Describe an orthogonal basis for W. Solution: Let v = x and v = x x v v v v. {v, v } is an orthogonal basis for Span{x, x }. Let v = x x v v v v x v v v v (component of x orthogonal to Span{x, x }) Note that v is in W. Why? {v, v, v } is an orthogonal basis for W. Math 9Applied Linear Algebra p./
Theorem : The Gram-Schmidt Process Given a basis {x,..., x p } for a subspace W of R n, define v = x v = x x v v v v v = x x v v v v x v v v v. v p = x p x p v v v v x p v v v v Then {v,..., v p } is an orthogonal basis for W and x p v p v p v p v p Span{x,..., x p } =Span{v,..., v p } Math 9Applied Linear Algebra p./
EXAMPLE Suppose {x, x, x }, where x =, x =, x =, is a basis for a subspace W of R. Describe an orthogonal basis for W. Solution: v = x = and Math 9Applied Linear Algebra p./
cont. v = x x v v v v = = 9 9 Replace v with v : v = 9 9 = 9 8 (optional step - to make v easier to work with in the next step) Math 9Applied Linear Algebra p./
cont. v = x x v v v v x v v v v v = 9 9 8 = 9 8 = Math 9Applied Linear Algebra p./
cont. Rescale (optional): v = Orthogonal Basis for W : {v, v, v } = 8 >< >:, 9 8, 9 >= >; Math 9Applied Linear Algebra p./