Math 191 Applied Linear Algebra Lecture 1: Inner Products, Length, Orthogonality Stephen Billups University of Colorado at Denver Math 191Applied Linear Algebra p.1/
Motivation Not all linear systems have solutions. EXAMPLE: No solution to 1 5 x 1 x 5 = 5 exists. Why? 8 9 < Ax is a point on the line spanned by 1 = 5 and b is not on the line. So Ax b for all : ; x. x.5.5 1.5 1 0.5 1 x 1 Math 191Applied Linear Algebra p./
Approximate Solutions Instead find bx so that Abx lies closest to b. x.5.5 1.5 1 0.5 1 x 1 Using techniques in this chapter, we will find that bx= 1. 0 5, so that Abx = 1.. 8 5. Math 191Applied Linear Algebra p./
Observation: Segment joining Abx and b is perpendicular ( or orthogonal) to the set of solutions to Ax = b. Need to develop fundamental ideas of length orthogonality orthogonal projections The key to all of these concepts is the Inner Product. Math 191Applied Linear Algebra p./
The Inner Product Inner product or dot product of u = u 1 u.. 5 and v = v 1 v.. 5 : u n v n u v = u T v = h u 1 u u n i v 1 v.. 5 = u 1 v 1 + u v + + u n v n v n Note that v u =v 1 u 1 + + v n u n = u 1 v 1 + + u n v n = u v Math 191Applied Linear Algebra p.5/
THEOREM 1 Let u, v and w be vectors in R n, and let c be any scalar. Then a. u v = v u b. (u + v) w = u w + v w c. (cu) v =c (u v) = u (cv) d. u u 0, and u u = 0 if and only if u = 0. Combining parts b and c, one can show (c 1 u 1 + + c p u p ) w =c 1 (u 1 w) + + c p (u p w) Math 191Applied Linear Algebra p./
Length of a Vector For v = v 1 v.., the length or norm of v is the nonnegative scalar v defined 5 by v n v = v v = p v1 + v + + v n and v = v v. For example, if v = a b 5, then v = a + b (distance between 0 and v) Observation: For any scalar c, cv = c v Math 191Applied Linear Algebra p./
Distance in R n The distance between u and v in R n : dist(u, v) = u v. This agrees with the usual formulas for R and R. Let u = (u 1, u ) and v = (v 1, v ). Then u v = (u 1 v 1, u v ) and dist(u, v) = u v = (u 1 v 1, u v ) = q (u 1 v 1 ) + (u v ) Math 191Applied Linear Algebra p.8/
Orthogonal Vectors [dist (u, v)] = u v = (u v) (u v) = (u) (u v) + ( v) (u v) = = u u u v + v u + v v = u + v u v [dist (u, v)] = u + v u v Math 191Applied Linear Algebra p.9/
Previous slide showed that [dist (u, v)] = u + v u v Similarly, we can show that [dist (u, v)] = u + v + u v Since [dist (u, v)] = [dist (u, v)], u v =. Two vectors u and v are said to be orthogonal (to each other) if u v = 0. Also note that if u and v are orthogonal, then u + v = u + v. THEOREM THE PYTHAGOREAN THEOREM Two vectors u and v are orthogonal if and only if u + v = u + v. Math 191Applied Linear Algebra p.10/
Orthogonal Complements If a vector z is orthogonal to every vector in a subspace W of R n, then z is said to be orthogonal to W. The set of vectors z that are orthogonal to W is called the orthogonal complement of W and is denoted by W (read as W perp ). Math 191Applied Linear Algebra p.11/
Row, Null and Columns Spaces THEOREM Let A be an m n matrix. Then the orthogonal complement of the row space of A is the nullspace of A, and the orthogonal complement of the column space of A is the nullspace of A T : (Row A) =Nul A, (Col A) =Nul A T. Why? (See complete proof in the text) Consider Ax = 0: r 1 x 0 r x 0 Note that Ax = =. where r 1,..., r m are the rows of A. Thus, x 5. 5 r m x 0 is orthogonal to each row of A. So x is orthogonal to Row A. Math 191Applied Linear Algebra p.1/
EXAMPLE Let A = 1 0 1 5. 0 8 9 >< 0 1 Basis for Nul A = 1 5 >:, >= 0 5 and therefore Nul A is a plane in R. >; 0 1 8 9 >< 1 >= Basis for Row A = 0 5 and therefore Row A is a line in R. >: >; 1 8 9 < Basis for Col A = 1 = 5 : ; and therefore Col A is a line in R. 8 9 < Basis for Nul A T = = 5 : 1 ; and therefore Nul AT is a line in R. Math 191Applied Linear Algebra p.1/
Section. Orthogonal Sets A set of vectors {u 1, u,..., u p } in R n is called an orthogonal set if u i u j = 0 whenever i j. EXAMPLE: Is 8 >< >: 1 1 0 5, 1 1 0 5, 0 0 1 9 >= 5 >; an orthogonal set? Solution: Label the vectors u 1, u, and u respectively. Then u 1 u =, u 1 u =, u u = Therefore, {u 1, u, u } is an orthogonal set. Math 191Applied Linear Algebra p.1/
THEOREM Suppose S = {u 1, u,..., u p } is an orthogonal set of nonzero vectors in R n and W =span{u 1, u,..., u p }. Then S is a linearly independent set and is therefore a basis for W. Partial Proof: Suppose c 1 u 1 + c u + + c p u p = 0 (c 1 u 1 + c u + + c p u p ) = 0 (c 1 u 1 ) u 1 + (c u ) u 1 + + (c p u p ) u 1 = 0 c 1 (u 1 u 1 ) + c (u u 1 ) + + c p (u p u 1 ) = 0 c 1 (u 1 u 1 ) = 0 Since u 1 0, u 1 u 1 > 0 which means c 1 =. In a similar manner, c,...,c p can be shown to by all 0. So S is a linearly independent set. Math 191Applied Linear Algebra p.15/
Orthogonal Basis An orthogonal basis for a subspace W of R n is a basis for W that is also an orthogonal set. Question: Why would we want to have an orthogonal basis? Ans: It makes it easy to calculate the coordinates relative to the basis. EXAMPLE: Suppose S = {u 1, u,..., u p } is an orthogonal basis for a subspace W of R n and suppose y is in W. Find c 1,...,c p so that y =c 1 u 1 + c u + + c p u p. Solution: y = (c 1 u 1 + c u + + c p u p ) y u 1 = (c 1 u 1 + c u + + c p u p ) u 1 y u 1 =c 1 (u 1 u 1 ) + c (u u 1 ) + + c p (u p u 1 ) y u 1 =c 1 (u 1 u 1 ) c 1 = y u 1 u 1 u 1 Similarly, c =, c =,..., c p = Math 191Applied Linear Algebra p.1/
THEOREM 5 Let {u 1, u,..., u p } be an orthogonal basis for a subspace W of R n. Then each y in W has a unique representation as a linear combination of u 1, u,..., u p. In fact, if then y =c 1 u 1 + c u + + c p u p c j = y u j u j u j (j = 1,..., p). Math 191Applied Linear Algebra p.1/
EXAMPLE: Express y = 5 as a linear combination of the orthogonal basis 8 >< >: 1 1 0 5, 1 1 0 5, 0 0 1 9 >= 5. >; Solution: y u 1 u 1 u 1 = y u u u = y u u u = Hence y = u 1 + u + u Math 191Applied Linear Algebra p.18/
Orthogonal Projections For a nonzero vector u in R n, suppose we want to write y in R n as the the following y = (multiple of u) + (multiple a vector to u). (y αu) u =0 y u α (u u) =0 = α = by= y u u (orthogonal projection of y onto u) u u and z = y y u u (component of y orthogonal to u) u u Math 191Applied Linear Algebra p.19/
Example Let y = 8 5 and u = 1 5. Compute the distance from y to the line through 0 and u. Solution: by= y u u u u = Distance from y to the line through 0 and u = distance from by to y = by y = Math 191Applied Linear Algebra p.0/
Orthonormal Sets A set of vectors {u 1, u,..., u p } in R n is called an orthonormal set if 1. It is orthogonal.. Each vector has length 1. If the orthonormal set {u 1, u,..., u p } spans a vector space W, then {u 1, u,..., u p } is called an orthonormal basis for W. Math 191Applied Linear Algebra p.1/
Orthogonal Matrices Recall that v is a unit vector if v = v v = v T v = 1. Suppose U = [u 1 u u ] where {u 1, u, u } is an orthonormal set. Then U T U = u T 1 u T u T 5 [u 1 u u ] = u T 1 u 1 u T 1 u u T 1 u u T u 1 u T u u T u u T u 1 u T u u T u 5 = 5 = It can be shown that UU T = I also. So U 1 = U T (such a matrix is called an orthogonal matrix). (NOTE: U must be square to be orthogonal). Math 191Applied Linear Algebra p./
THEOREM U T U = I. An m n matrix U has orthonormal columns if and only if THEOREM Let U be an m n matrix with orthonormal columns, and let x and y be in R n. Then a. Ux = x b. (Ux) (Uy) = x y c. (Ux) (Uy) = 0 if and only if x y = 0. Proof of part b: (Ux) (Uy) = Math 191Applied Linear Algebra p./