Lecture 23: 6.1 Inner Products

Lecture 23: 6.1 Inner Products Wei-Ta Chu 2008/12/17

Definition An inner product on a real vector space V is a function that associates a real number u, vwith each pair of vectors u and v in V in such a way that the following axioms are satisfied for all vectors u, v, and w in V and all scalars k. u, v= v, u u + v, w= u, w+ v, w ku, v= k u, v u, u0 and u, u= 0 if and only if u = 0 A real vector space with an inner product is called a real inner product space. 2008/12/17 Elementary Linear Algebra 2

Euclidean Inner Product on R n If u = (u 1, u 2,, u n ) and v = (v 1, v 2,, v n ) are vectors in R n, then the formula v, u= u v = u 1 v 1 + u 2 v 2 + + u n v n defines v, uto be the Euclidean product on R n. The four inner product axioms hold by Theorem 4.1.2. 2008/12/17 Elementary Linear Algebra 3

Properties of Euclidean Inner Product Theorem 4.1.2 If u, v and w are vectors in R n and k is any scalar, then u v = v u (u + v) w = u w + v w (k u) v = k (u v) v v 0; Further, v v = 0 if and only if v = 0 Example (3u + 2v) (4u + v) = (3u) (4u + v) + (2v) (4u + v ) = (3u) (4u) + (3u) v + (2v) (4u) + (2v) v =12(u u) + 11(u v) + 2(v v) 2008/12/17 Elementary Linear Algebra 4

Euclidean Inner Product vs. Inner Product The Euclidean inner product is the most important inner product on R n. However, there are various applications in which it is desirable to modify the Euclidean inner product by weighting its terms differently. More precisely, if w 1,w,,w 2,,w n are positive real numbers, and if u=(u 1,u 2,,u n ) and v=(v 1,v 2,,v n ) are vectors in R n, then it can be shown which is called weighted Euclidean inner product with weights w 1,w 2,,w n 2008/12/17 Elementary Linear Algebra 5

Example Suppose that some physical experiment can produce any of n possible numerical values x 1,x 2,,x n. We perform m repetitions of the experiment and yield these values with various frequencies; that is, x 1 occurs f 1 times, x 2 occurs f 2 times, and so forth. f 1 +f+ +f 2 + +f n =m. Thus, the arithmetic average, or mean, is 2008/12/17 Elementary Linear Algebra 6

Example If we let f=(f 1,f 2,,f n ), x=(x 1,x 2,,x n ), w 1 =w 2 = =w n =1/m Then this equation can be expressed as the weighted inner product 2008/12/17 Elementary Linear Algebra 7

Weighted Euclidean Product Let u = (u 1, u 2 ) and v = (v 1, v 2 ) be vectors in R 2. Verify that the weighted Euclidean inner product u, v= 3u 1 v 1 + 2u 2 v 2 satisfies the four product axioms. Solution: Note first that if u and v are interchanged in this equation, the right side remains the same. Therefore, u, v= v, u. If w = (w 1, w 2 ), then u + v, w= 3(u 1 +v 1 )w 1 + 2(u 2 +v 2 )w 2 = (3u 1 w 1 + 2u 2 w 2 ) + (3v 1 w 1 + 2v 2 w 2 )= u, w+ v, w which establishes the second axiom. 2008/12/17 Elementary Linear Algebra 8

Weighted Euclidean Product ku, v=3(ku 1 )v 1 + 2(ku 2 )v 2 = k(3u 1 v 1 + 2u 2 v 2 ) = k u, v which establishes the third axiom. v, v= 3v 1 v 1 +2v 2 v 2 = 3v 12 + 2v 22.Obviously, v, v= 3v 12 + 2v 22 0. Furthermore, v, v= 3v 12 + 2v 22 = 0 if and only if v 1 = v 2 = 0, That is, if and only if v = (v 1,v 2 )=0. Thus, the fourth axiom is satisfied. 2008/12/17 Elementary Linear Algebra 9

Definition If V is an inner product space, then the norm (or length) of a vector u in V is denoted by u and is defined by u = u, u ½ The distance between two points (vectors) u and v is denoted by d(u,v) and is defined by d(u, v) = u v If a vector has norm 1, then we say that it is a unit vector. 2008/12/17 Elementary Linear Algebra 10

Norm and Distance in R n If u = (u 1, u 2,, u n ) and v = (v 1, v 2,, v n ) are vectors in R n with the Euclidean inner product, then d 1/ 2 1/ 2 ( u, v) u v u v, u v ( u v ) ( u v ) ( u v ) ( u v )... ( u v ) 2 2 2 1 1 2 2 n n 2008/12/17 Elementary Linear Algebra 11

Weighted Euclidean Inner Product The norm and distance depend on the inner product used. If the inner product is changed, then the norms and distances between vectors also change. For example, for the vectors u = (1,0) and v = (0,1) in R 2 with the Euclidean inner product, we have u 1 and d( u, v) u v (1, 1) ( 1) However, if we change to the weighted Euclidean inner product u, v= 3u 1 v 1 + 2u 2 v 2, then we obtain 1 2 2 2 u u, u 1/ 2 (3 1 1 2 0 0) 1/ 2 3 d( u, v) u v (1, 1),(1, 1) 1/ 2 [3 1 1 2 ( 1) ( 1)] 1/ 2 5 2008/12/17 Elementary Linear Algebra 12

Unit Circles and Spheres in Inner Product Space If V is an inner product space, then the set of points in V that satisfy u = 1 is called the unite sphere or sometimes the unit circle in V. In R 2 and R 3 these are the points that lie 1 unit away form the origin. 2008/12/17 Elementary Linear Algebra 13

Unit Circles in R 2 Sketch the unit circle in an xy-coordinate system in R 2 using the Euclidean inner product u, v= u 1 v 1 + u 2 v 2 Sketch the unit circle in an xy-coordinate system in R 2 using the Euclidean inner product u, v= 1/9u 1 v 1 + 1/4u 2 v 2 Solution If u = (x,y), then u = u, u ½ = (x 2 + y 2 ) ½, so the equation of the unit circle is x 2 + y 2 = 1. If u = (x,y), then u = u, u ½ = (1/9x 2 + 1/4y 2 ) ½, so the equation of the unit circle is x 2 /9 + y 2 /4 = 1. 2008/12/17 Elementary Linear Algebra 14

Inner Products Generated by Matrices u 1 v 1 u 2 v 2 Let u and v= be vectors in R n (expressed as n1 : : un vn matrices), and let A be an invertible nn matrix. If u v is the Euclidean inner product on R n, then the formula u, v= Au Av defines an inner product; it is called the inner product on R n generated by A. Recalling that the Euclidean inner product u v can be written as the matrix product v T u, the above formula can be written in the alternative form u, v= (Av) T Au, or equivalently, u, v= v T A T Au 2008/12/17 Elementary Linear Algebra 15

Inner Product Generated by the Identity Matrix The inner product on R n generated by the nn identity matrix is the Euclidean inner product: Let A = I, we have u, v= Iu Iv = u v The weighted Euclidean inner product u, v= 3u 1 v 1 + 2u 2 v 2 is the inner product on R 2 generated by since 3 0 A 0 2 3 0 3 0 u 3 0 u u, v v 0 2 0 2 u 2 0 2 u 2 1 1 v 1 v2 v 1 v2 3u1v 1 2 u2 2 2008/12/17 Elementary Linear Algebra 16

Inner Product Generated by the Identity Matrix In general, the weighted Euclidean inner product u, v= w 1 u 1 v 1 + w 2 u 2 v 2 + + w n u n v n is the inner product on R n generated by 2008/12/17 Elementary Linear Algebra 17

An Inner Product on M 22 If 1 2 1 2 U u u and V v v u u v v 3 4 3 4 are any two 22 matrices, then U, V= tr(u T V) = tr(v T U) = u 1 v 1 + u 2 v 2 + u 3 v 3 + u 4 v 4 defines an inner product on M 22 For example, if 1 2 1 0 U and V 3 4 3 2 then U, V= 1(-1)+2(0) + 3(3) + 4(2) = 16 The norm of a matrix U relative to this inner product is U U, U u u u u 1/ 2 2 2 2 2 1 2 3 4 and the unit sphere in this space consists of all 22 matrices U whose entries satisfy the equation U = 1, which on squaring yields u 12 + u 22 + u 32 + u 42 = 1 2008/12/17 Elementary Linear Algebra 18

An Inner Product on P 2 If p = a 0 + a 1 x + a 2 x 2 and q = b 0 + b 1 x + b 2 x 2 are any two vectors in P 2, then the following formula defines an inner product on P 2 : p, q= a 0 b 0 + a 1 b 1 + a 2 b 2 The norm of the polynomial p relative to this inner product is 1/ 2 2 2 2 p p, p a a a 0 1 2 and the unit sphere in this space consists of all polynomials p in P 2 whose coefficients satisfy the equation p = 1, which on squaring yields a 02 + a 12 + a 22 =1 2008/12/17 Elementary Linear Algebra 19

Theorem 6.1.1 (Properties of Inner Products) If u, v, and w are vectors in a real inner product space, and k is any scalar, then: 0, v= v, 0= 0 u, v + w= u, v+ u, w u, kv=k u, v u v, w= u, w v, w u, v w= u, v u, w 2008/12/17 Elementary Linear Algebra 20

Example u 2v, 3u + 4v = u, 3u + 4v 2v, 3u+4v = u, 3u+ u, 4v 2v, 3u 2v, 4v = 3 u, u+ 4 u, v 6 v, u 8 v, v = 3 u 2 + 4 u, v 6 u, v 8 v 2 = 3 u 2 2 u, v 8 v 2 2008/12/17 Elementary Linear Algebra 21

Example We are guaranteed without any further proof that the five properties given in Theorem 6.1.1 are true for the inner product on R n generated by any matrix A. u, v + w= (v+w) T A T Au =(v T +w T )A T Au =(v T A T Au) + (w T A T Au) = u, v+ u, w [Property of transpose] [Property of matrix multiplication] 2008/12/17 Elementary Linear Algebra 22

Lecture 23: 6.2 Angle and Orthogonality Wei-Ta Chu 2008/12/17

Theorem 6.2.1 Theorem 6.2.1 (Cauchy-Schwarz Inequality) If u and v are vectors in a real inner product space, then u, v u v The inequality can be written in the following two forms The Cauchy-Schwarz inequality for R n (Theorem 4.1.3) follows as a special case of this theorem by taking u, vto be the Euclidean inner product u v. 2008/12/17 Elementary Linear Algebra 24

Theorem 6.2.2 Theorem 6.2.2 (Properties of Length) If u and v are vectors in an inner product space V, and if k is any scalar, then : u 0 u = 0 if and only if u = 0 ku = k u u + v u + v (Triangle inequality) Proof of (d) (Property of absolute value) (Theorem 6.2.1) 2008/12/17 Elementary Linear Algebra 25

Theorem 6.2.3 Theorem 6.2.3 (Properties of Distance) If u, v, and w are vectors in an inner product space V, and if k is any scalar, then: d(u, v) 0 d(u, v) = 0 if and only if u = v d(u, v) = d(v, u) d(u, v) d(u, w) + d(w, v) (Triangle inequality) 2008/12/17 Elementary Linear Algebra 26

Angle Between Vectors Cauchy-Schwarz inequality can be used to define angles in general inner product cases. We define to be the angle between u and v. 2008/12/17 Elementary Linear Algebra 27

Example Let R 4 have the Euclidean inner product. Find the cosine of the angle between the vectors u = (4, 3, 1, -2) and v = (-2, 1, 2, 3). 2008/12/17 Elementary Linear Algebra 28

Orthogonality Definition Two vectors u and v in an inner product space are called orthogonal if u, v= 0. Example (U, V= tr(u T V) = tr(v T U) = u 1 v 1 + u 2 v 2 + u 3 v 3 + u 4 v 4 ) If M 22 has the inner project defined previously, then the matrices U 1 1 0 and 1 0 0 2 0 are orthogonal, since U, V= 1(0) + 0(2) + 1(0) + 1(0) = 0. V 2008/12/17 Elementary Linear Algebra 29

Orthogonal Vectors in P 2 Let P 2 have the inner product p, q p( x) q( x) dx and let p = x and q = x 2. 1 Then p q 1 1 1/ 2 1 1/ 2 1/ 2 2 p, p xxdx x dx 1 1 1 1/ 2 1 1/ 2 1/ 2 2 2 4 q, q x x dx x dx 1 1 2 3 p, q xx dx x dx 0 1 1 1 1 because p, q= 0, the vectors p = x and q = x 2 are orthogonal relative to the given inner product. 2 3 2 5 2008/12/17 Elementary Linear Algebra 30

Theorem 6.2.4 (Generalized Theorem of Pythagoras) If u and v are orthogonal vectors in an inner product space, then Proof u + v 2 = u 2 + v 2 2008/12/17 Elementary Linear Algebra 31

Example Since p = x and q = x 2 are orthogonal relative to the inner product 1 p, q p( x) q( x) dx on P 2. 1 It follows from the Theorem of Pythagoras that p + q 2 = p 2 + q 2 Thus, from the previous example: 2 2 2 2 2 2 2 16 p+ q ( ) ( ) 3 5 3 5 15 We can check this result by direct integration: 1 2 2 2 p+ q p+ q, p+ q ( x x )( x x ) dx 1 1 1 1 2 3 4 2 2 16 x dx 2 x dx x dx 0 3 5 15 1 1 1 2008/12/17 Elementary Linear Algebra 32

Orthogonality Definition Let W be a subspace of an inner product space V. A vector u in V is said to be orthogonal to W if it is orthogonal to every vector in W, and the set of all vectors in V that are orthogonal to W is called the orthogonal complement ( 正交補餘 ) of W. If V is a plane through the origin of R 3 with Euclidean inner product, then the set of all vectors that are orthogonal to every vector in V forms the line L through the origin that is perpendicular to V. L V 2008/12/17 Elementary Linear Algebra 33

Theorem 6.2.5 Theorem 6.2.5 (Properties of Orthogonal Complements) If W is a subspace of a finite-dimensional inner product space V, then: W is a subspace of V. (read W perp ) The only vector common to W and W is 0; that is,w W = 0. The orthogonal complement of W is W; that is, (W ) = W. 2008/12/17 Elementary Linear Algebra 34

Proof of Theorem 6.2.5(a) Note first that 0, w=0 for every vector w in W, so W contains at least the zero vector. We want to show that the sum of two vectors in W is orthogonal to every vector in W (closed under addition) and that any scalar multiple of a vector in W is orthogonal to every vector in W (closed under scalar multiplication). 2008/12/17 Elementary Linear Algebra 35

Proof of Theorem 6.2.5(a) Let u and v be any vector in W, let k be any scalar, and let w be any vector in W. Then from the definition of W, we have u, w=0 and v, w=0. Using the basic properties of the inner product, we have u+v, w= u, w+ v, w= 0 + 0 = 0 ku, w= ku, w= k(0) = 0 Which proves that u+v and ku are in W. 2008/12/17 Elementary Linear Algebra 36

Proof of Theorem 6.2.5(b) The only vector common to W and W is 0; that is,w W = 0. If v is common to W and W, then v, v=0, which implies that v=0 by Axiom 4 for inner products. 2008/12/17 Elementary Linear Algebra 37

Theorem 6.2.6 Theorem 6.2.6 If A is an mn matrix, then: The nullspace of A and the row space of A are orthogonal complements in R n with respect to the Euclidean inner product. The nullspace of A T and the column space of A are orthogonal complements in R m with respect to the Euclidean inner product. 2008/12/17 Elementary Linear Algebra 38

Proof of Theorem 6.2.6(a) We must show that if a vector v is orthogonal to every vector in the row space, then Av=0, and conversely, that if Av=0, then v is orthogonal to every vector in the row space. Assume that v is orthogonal to every vector in the row space of A. Then in particular, v is orthogonal to the row vectors r 1, r 2,, r m of A; that is 2008/12/17 Elementary Linear Algebra 39

Proof of Theorem 6.2.6(a) The linear system Ax=0 can be expressed in dot product notation as And it follows that v is a solution of this system and hence lies in the nullspace of A. 2008/12/17 Elementary Linear Algebra 40

Proof of Theorem 6.2.6(a) Conversely, assume that v is a vector in the nullspace of A, so Av=0. It follows that But if r is any vector in the row space of A, then r is expressible as a linear combination of the row vectors of A, say Thus Which proves that v is orthogonal to every vector in the row space of A. 2008/12/17 Elementary Linear Algebra 41

Example (Basis for an Orthogonal Complement) Let W be the subspace of R 5 spanned by the vectors w 1 =(2, 2, -1, 0, 1), w 2 =(-1, -1, 2, -3, 1), w 3 =(1, 1, -2, 0, - 1), w 4 =(0, 0, 1, 1, 1). Find a basis for the orthogonal complement of W. Solution The space W spanned by w 1, w 2, w 3, and w 4 is the same as the row space of the matrix 2 2 1 0 1 1 1 2 3 1 A 1 1 2 0 1 0 0 1 1 1 By Theorem 6.2.6, the nullspace of A is the orthogonal complement of W. In Example 4 of Section 5.5 we showed that 1 1 1 0 v 0 and 1 1 v2 0 0 0 1 form a basis for this nullspace. Thus, vectors v 1 = (-1, 1, 0, 0, 0) and v 2 = (-1, 0, -1, 0, 1) form a basis for the orthogonal complement of W. 2008/12/17 Elementary Linear Algebra 42

Remarks In any inner product space V, the zero space {0} and the entire space V are orthogonal complements. If A is an matrix, to say that Ax=0 has only the trivial solution is equivalent to saying that the orthogonal complement of the nullspace of A is all of R n, or equivalently, that the row space of A is all of R n. 2008/12/17 Elementary Linear Algebra 43

Theorem 6.2.7 (Equivalent Statements) If A is an mn matrix, and if T A : R n R n is multiplication by A, then the following are equivalent: A is invertible. Ax = 0 has only the trivial solution. The reduced row-echelon form of A is I n. A is expressible as a product of elementary matrices. Ax = b is consistent for every n1 matrix b. Ax = b has exactly one solution for every n1 matrix b. det(a) 0. The range of T A is R n. T A is one-to-one. The column vectors of A are linearly independent. The row vectors of A are linearly independent. The column vectors of A span R n. The row vectors of A span R n. The column vectors of A form a basis for R n. The row vectors of A form a basis for R n. A has rank n. A has nullity 0. The orthogonal complement of the nullspace of A is R n. The orthogonal complement of the row of A is {0}. 2008/12/17 Elementary Linear Algebra 44