Chapter 6 Orthogonality and Least Squares
Section 6.1 Inner Product, Length, and Orthogonality
Orientation Recall: This course is about learning to: Solve the matrix equation Ax = b Solve the matrix equation Ax = λx Almost solve the equation Ax = b We are now aiming at the last topic. Idea: In the real world, data is imperfect. Suppose you measure a data point x which you know for theoretical reasons must lie on a plane spanned by two vectors u and v. x v u Due to measurement error, though, the measured x is not actually in Span{u, v}. In other words, the equation au + bv = x has no solution. What do you do? The real value is probably the closest point to x on Span{u, v}. Which point is that?
The Dot Product We need a notion of angle between two vectors, and in particular, a notion of orthogonality (i.e. when two vectors are perpendicular). This is the purpose of the dot product. Definition The dot product of two vectors x, y in R n is x 1 y 1 x 2 x y =. y 2. x n y n def = x 1y 1 + x 2y 2 + + x ny n. Thinking of x, y as column vectors, this is the same as x T y. Example 1 4 2 5 = ( 1 2 3 ) 4 5 = 1 4 + 2 5 + 3 6 = 32. 3 6 6
Properties of the Dot Product Many usual arithmetic rules hold, as long as you remember you can only dot two vectors together, and that the result is a scalar. x y = y x (x + y) z = x z + y z (cx) y = c(x y) Dotting a vector with itself is special: Hence: x x 0 x 1 x 1 x x = 0 if and only if x = 0. x 2. x 2. = x 1 2 + x2 2 + + xn 2. x n x n Important: x y = 0 does not imply x = 0 or y = 0. For example, ( 1 0) ( 0 1) = 0.
The Dot Product and Length Definition The length or norm of a vector x in R n is x = x x = x1 2 + x 2 2 + + x n 2. Why is this a good definition? The Pythagorean theorem! ( 3 4) 4 3 2 + 4 2 = 5 3 ( = 4) 3 2 + 4 2 = 5 3 Fact If x is a vector and c is a scalar, then cx = c x. ( ) ( ( ) 6 = 3 8 4) 2 = 2 3 = 10 4
The Dot Product and Distance Definition The distance between two points x, y in R n is dist(x, y) = y x. This is just the length of the vector from x to y. Example Let x = (1, 2) and y = (4, 4). Then ( ) dist(x, y) = y x = 3 = 3 2 2 + 2 2 = 13. x y x y 0
Unit Vectors Definition A unit vector is a vector v with length v = 1. Example The unit coordinate vectors are unit vectors: 1 e 1 = 0 0 = 1 2 + 0 2 + 0 2 = 1 Definition Let x be a nonzero vector in R n. The unit vector in the direction of x is the x vector x. This is in fact a unit vector: x scalar x = 1 x = 1. x
Unit Vectors Example Example What is the unit vector in the direction of x = u = ( ) 3? 4 x ( ) x = 1 3 = 1 32 + 4 2 4 5 ( ) 3. 4 x u 0
x y Orthogonality Definition Two vectors x, y are orthogonal or perpendicular if x y = 0. Notation: x y means x y = 0. Why is this a good definition? The Pythagorean theorem / law of cosines! y y x x x and y are perpendicular x 2 + y 2 = x y 2 x x + y y = (x y) (x y) x x + y y = x x + y y 2x y x y = 0 Fact: x y x y 2 = x 2 + y 2
Orthogonality Example 1 Problem: Find all vectors orthogonal to v = 1. 1 We have to find all vectors x such that x v = 0. This means solving the equation x 1 1 0 = x v = x 2 1 = x 1 + x 2 x 3. x 3 1 The parametric form for the solution is x 1 = x 2 + x 3, so the parametric vector form of the general solution is x 1 1 1 x = x 2 = x 2 1 + x 3 0. x 3 0 1 1 For instance, 1 0 1 1 1 1 because 1 0 1 1 1 = 0.
Orthogonality Example 1 1 Problem: Find all vectors orthogonal to both v = 1 and w = 1. 1 1 Now we have to solve the system of two homogeneous equations x 1 1 0 = x v = x 2 1 = x 1 + x 2 x 3 x 3 1 x 1 1 0 = x w = x 2 1 = x 1 + x 2 + x 3. x 3 1 In matrix form: ( ) ( ) 1 1 1 rref 1 1 0 The rows are v and w. 1 1 1 0 0 1 The parametric vector form of the solution is x 1 1 x 2 = x 2 1. x 3 0
Orthogonality General procedure Problem: Find all vectors orthogonal to some number of vectors v 1, v 2,..., v m in R n. This is the same as finding all vectors x such that 0 = v T 1 x = v T 2 x = = v T m x. Putting the row vectors v T 1, v T 2,..., v T m into a matrix, this is the same as finding all x such that v T 1 v 1 x v2 T.. x = v 2 x. = 0. vm T v m x Important The set of all vectors orthogonal to some vectors v 1, v 2,..., v m in R n is the null space of the m n matrix In particular, this set is a subspace! v T 1 v2 T... vm T
Orthogonal Complements Definition Let W be a subspace of R n. Its orthogonal complement is W = { v in R n v w = 0 for all w in W } read W perp. Pictures: W is orthogonal complement A T is transpose The orthogonal complement of a line in R 2 is the perpendicular line. [demo] W W The orthogonal complement of a line in R 3 is the perpendicular plane. [demo] W W The orthogonal complement of a plane in R 3 is the perpendicular line. [demo] W W
Poll Poll Let W be a plane in R 4. How would you describe W? A. The zero space {0}. B. A line in R 4. C. A plane in R 4. D. A 3-dimensional space in R 4. E. All of R 4.
Orthogonal Complements Basic properties Let W be a subspace of R n. Facts: 1. W is also a subspace of R n 2. (W ) = W 3. dim W + dim W = n 4. If W = Span{v 1, v 2,..., v m}, then Let s check 1. W = all vectors orthogonal to each v 1, v 2,..., v m = { x in R n x v i = 0 for all i = 1, 2,..., m } v T 1 = Nul v2 T... vm T Is 0 in W? Yes: 0 w = 0 for any w in W. Suppose x, y are in W. So x w = 0 and y w = 0 for all w in W. Then (x + y) w = x w + y w = 0 + 0 = 0 for all w in W. So x + y is also in W. Suppose x is in W. So x w = 0 for all w in W. If c is a scalar, then (cx) w = c(x 0) = c(0) = 0 for any w in W. So cx is in W.
Orthogonal Complements Computation 1 1 Problem: if W = Span 1, 1, compute W. 1 1 By property 4, we have to find the null space of the matrix whose rows are ( 1 1 1 ) and ( 1 1 1 ), which we did before: Nul ( ) 1 1 1 1 = Span 1 1 1 1. 0 v T 1 Span{v 1, v 2,..., v m} = Nul v2 T.. vm T
Orthogonal Complements Row space, column space, null space Definition The row space of an m n matrix A is the span of the rows of A. It is denoted Row A. Equivalently, it is the column span of A T : It is a subspace of R n. Row A = Col A T. We showed before that if A has rows v T 1, v T 2,..., v T m, then Hence we have shown: Fact: (Row A) = Nul A. Span{v 1, v 2,..., v m} = Nul A. Replacing A by A T, and remembering Row A T = Col A: Fact: (Col A) = Nul A T. Using property 2 and taking the orthogonal complements of both sides, we get: Fact: (Nul A) = Row A and Col A = (Nul A T ).
Orthogonal Complements Reference sheet Orthogonal Complements of Most of the Subspaces We ve Seen For any vectors v 1, v 2,..., v m: v T 1 Span{v 1, v 2,..., v m} = Nul v2 T.. vm T For any matrix A: Row A = Col A T and (Row A) = Nul A (Col A) = Nul A T Row A = (Nul A) Col A = (Nul A T )