Section 6. Inner Product, Length, and Orthogonality
Orientation Almost solve the equation Ax = b Problem: In the real world, data is imperfect. x v u But due to measurement error, the measured x is not actually in Span{u, v}. But you know, for theoretical reasons, it must lie on that plane. What do you do? The real value is probably the closest point, on the plane, to x. New terms: Orthogonal projection ( closest point ), orthogonal vectors, angle.
The Dot Product The dot product encodes the notion of angle between two vectors. We will use it to define orthogonality (i.e. when two vectors are perpendicular) Definition The dot product of two vectors x, y in R n is This is the same as x T y. Example x y x 2 x y =. y 2. x n y n def = x y + x 2y 2 + + x ny n. 4 2 5 = ( 2 3 ) 4 5 = 4 + 2 5 + 3 6 = 32. 3 6 6
Properties of the Dot Product Many usual arithmetic rules hold, as long as you remember you can only dot two vectors together, and that the result is a scalar. x y = y x (x + y) z = x z + y z (cx) y = c(x y) Dotting a vector with itself is special: Hence: x x 0 x x x x = 0 if and only if x = 0. x 2. x 2. = x 2 + x2 2 + + xn 2. x n x n Important: x y = 0 does not imply x = 0 or y = 0. For example, ( 0) ( 0 ) = 0.
The Dot Product and Length Definition The length or norm of a vector x in R n is x = x x = x 2 + x 2 2 + + x n 2. Why is this a good definition? The Pythagorean theorem! ( 3 4) 4 3 2 + 4 2 = 5 3 ( = 4) 3 2 + 4 2 = 5 3 Fact If x is a vector and c is a scalar, then cx = c x. ( ) ( ( ) 6 = 3 8 4) 2 = 2 3 = 0 4
The Dot Product and Distance The following is just the length of the vector from x to y. Definition The distance between two points x, y in R n is dist(x, y) = y x. Example Let x = (, 2) and y = (4, 4). Then ( ) dist(x, y) = y x = 3 = 3 2 2 + 2 2 = 3. x y x y 0
Unit Vectors Definition A unit vector is a vector v with length v =. Example The unit coordinate vectors are unit vectors: e = 0 0 = 2 + 0 2 + 0 2 = Definition Let x be a nonzero vector in R n. The unit vector in the direction of x is the x vector x. Is this really a unit vector? scalar x x = x =. x
Unit Vectors Example Example What is the unit vector in the direction of x = u = ( ) 3? 4 x ( ) x = 3 = 32 + 4 2 4 5 ( ) 3. 4 x u 0
x y Orthogonality Definition Two vectors x, y are orthogonal or perpendicular if x y = 0. Notation: Write it as x y. Why is this a good definition? The Pythagorean theorem / law of cosines! y y x x x and y are perpendicular x 2 + y 2 = x y 2 x x + y y = (x y) (x y) x x + y y = x x + y y 2x y x y = 0 Fact: x y x y 2 = x 2 + y 2 (Pythagorean Theorem)
Orthogonality Example Problem: Find all vectors orthogonal to v =. We have to find all vectors x such that x v = 0. This means solving the equation x 0 = x v = x 2 = x + x 2 x 3. x 3 The parametric form for the solution is x = x 2 + x 3, so the parametric vector form of the general solution is x x = x 2 = x 2 + x 3 0. x 3 0 For instance, 0 because 0 = 0.
Orthogonality Example Problem: Find all vectors orthogonal to both v = and w =. Now we have to solve the system of two homogeneous equations x 0 = x v = x 2 = x + x 2 x 3 x 3 x 0 = x w = x 2 = x + x 2 + x 3. x 3 In matrix form: ( ) ( ) rref 0 The rows are v and w. 0 0 The parametric vector form of the solution is x x 2 = x 2. x 3 0
Orthogonality General procedure Problem: Find all vectors orthogonal to v, v 2,..., v m in R n. This is the same as finding all vectors x such that 0 = v T x = v T 2 x = = v T m x. Putting the row vectors v T, v T 2,..., v T m into a matrix, this is the same as finding all x such that v T v x v2 T.. x = v 2 x. = 0. vm T v m x The key observation The set of all vectors orthogonal to some vectors v, v 2,..., v m in R n is the null space of the m n matrix: In particular, this set is a subspace! v T v2 T... vm T
Orthogonal Complements Definition Let W be a subspace of R n. Its orthogonal complement is W = { v in R n v w = 0 for all w in W } W is orthogonal complement A T is transpose Pictures: The orthogonal complement of a line in R 2 is the perpendicular line. W read W perp. W The orthogonal complement of a line in R 3 is the perpendicular plane. W W The orthogonal complement of a plane in R 3 is the perpendicular line. W W
Orthogonal Complements Basic properties Facts: Let W be a subspace of R n.. W is also a subspace of R n 2. (W ) = W 3. dim W + dim W = n 4. If W = Span{v, v 2,..., v m}, then W = all vectors orthogonal to each v, v 2,..., v m = { x in R n x v i = 0 for all i =, 2,..., m } v T = Nul v2 T... vm T Property 4 v T Span{v, v 2,..., v m} = Nul v2 T.. vm T
Orthogonal Complements Row space, column space, null space Definition The row space of an m n matrix A is the span of the rows of A. It is denoted Row A. Equivalently, it is the column span of A T : It is a subspace of R n. Row A = Col A T. We showed before that if A has rows v T, v T 2,..., v T m, then Span{v, v 2,..., v m} = Nul A. Hence we have shown: (Row A) = Nul A. Other Facts: (Col A) = Nul A T. (Replacing A by A T, and remembering Row A T = Col A) (Nul A) = Row A and Col A = (Nul A T ). (Using property 2 and taking the orthogonal complements of both sides)
Reference sheet Orthogonal Complements of Most of the Subspaces We ve Seen For any vectors v, v 2,..., v m: v T (Span{v, v 2,..., v m}) = Nul v2 T.. vm T For any matrix A: Row A = Col A T thus (Row A) = Nul A (Col A) = Nul A T Row A = (Nul A) Col A = (Nul A T )
Extra: Practice proving a set is subspace Example Let s check W is a subspace. Is 0 in W? Yes: 0 w = 0 for any w in W. Closed under addition: Suppose x, y are in W. So x w = 0 and y w = 0 for all w in W. Then (x + y) w = x w + y w = 0 + 0 = 0 for all w in W. So x + y is also in W. Closed under scalar product: Suppose x is in W. So x w = 0 for all w in W. If c is a scalar, then (cx) w = c(x 0) = c(0) = 0 for any w in W. So cx is in W.