Contents Appendix D (Inner Product Spaces W-5 Index W-63
Inner city space W-49
W-5 Chapter :
Appendix D Inner Product Spaces The inner product, taken of any two vectors in an arbitrary vector space, generalizes the dot product of two vectors in R n or C n. For two column vectors x and y R n we can form two different vector products, namely and the outer product xy T := x. x n the standard inner product ( y,..., y n = x T y := ( x,..., x n y. y n x y... x y n.. x n y... x n y n R n,n = x y +... + x n y n R. Here we interpret the two respective vectors as by n or n by matrices and multiply according to the rules of matrix multiplication. The outer product is a dyadic product since it creates an n by n dyad from two vectors, of which the first appears in column and the second in row form. It allows us to express matrix multiplication as a sum of rank dyadic generators; see Sections 6.,., and the proof of Lemma 3 in Section.. A matrix product can be written as the sum of dyads of the columns and rows of the two matrix factors as follows: c Copyright by Frank Uhlig; Work in progress ; latest update of 9th November. W-5
W-5 Appendix D: A mn B nk = = b a... a n. b k a ( b +... + a n ( b k R m,k, where the vectors a i denote the columns of the first factor A and the b j denote the rows of the second factor B. The standard inner product x T y of two vectors in R n is the same as the dot product x y R of the two vectors. It was introduced in Chapter and interprets the first factor as a row and the second one as a column vector. The inner or dot product is also handy to express matrix multiplication, namely A mn B nk = ã. ã m b... bk ã b... ã b k =.. ã m b... ã m b k R m,k, where the ã i now denote the rows of A and the b j the columns of the second factor B. In addition, the inner or dot product helps define angles and orthogonality of two vectors in R n, see Chapters through. We start by listing four fundamental properties of the standard inner product of two vectors.... : R n R n R. Proposition : (Real Inner Product The standard inner product of two vectors x and y R n is defined as x y := x T y = ( y n x,..., x n. = x i y i R. y i= n It satisfies the following four properties: (a x y = y x for all x, y R n ; (b x (y + z = x y +x y for all x, y, z R n ; (c (αx y = x (αy = α(x y for all x, y R n and all α R. (d x x for all x R n,and x x= R if and only if x = R n. For two complex vectors x, y C n, several modifications are in order in the definition and the properties of an inner product due to the effects of complex conjugation, see Appendix A.
Inner Product Spaces W-53 Proposition : (Complex Inner Product The standard inner product of two vectors x and y C n is defined as x y := x y = ( y n x,..., x n. = x i y i C. y i= n It satisfies the following four properties: (a x y = y x for all x, y C n ; (b x (y + z = x y +x y for all x, y, z C n ; (c (αx y = x (αy = α(x y for all x, y C n and all α C. (d x x for all x C n,and x x= C if and only if x = C n. Proof: We deduce the four properties for the complex inner product only. The properties of the real inner product in Proposition follow immediately by dropping all complex conjugation bars in this proof. (a x y = i x iy i = i x iy i = i y ix i = y x since double conjugation c gives c back for any c in C. (b x (y + z = i x i(y i +z i = i x iy i + x i z i = x y +x z. (c (αx y = i (αx iy i = α i x iy i = αx y and i (αx iy i = i x i(αy i = x (αy. (d x x = i x i as the sum of real squares. And equality holds precisely when x i =foreachi=,..., n, orwhenx= C n. The standard dot or inner product of R n (or C n serves very well in many aspects of linear algebra, such as when defining angles, orthogonality, and the length of vectors. More generally, an inner product can be defined in an arbitrary vector space V by requiring the four properties of a dot product; see Appendix C for abstract vector spaces. Definition : Let V be an arbitrary vector space over a field of scalars F. ( A function..,.. : V V F that maps any two vectors f and g V to the scalar f,g in F is bilinear if..,.. is linear in each of its arguments, i.e., if αf +βg,h = αf, h + βg,h and x, δu+ɛv = x, δu + x, ɛv for all scalars α, β, δ, ɛ F and all vectors x, u, v, f, g, h V. ( A bilinear function..,.. : V V C (or R, operating on a complex (or real vector space V,isaninner product on V if it satisfies the following four properties. (a x, y = y,x for all x, y V ; (b x, (y + z = x, y + x, y for all x, y, z V ; (c (αx,y = x, (αy = α x, y for all x, y V and all α C.
W-54 Appendix D: (d x, x for all x V,and x, x = C if and only if x = V. Note that if V is a vector space with the scalar field R, then the complex conjugation in parts (a and (c above should simply be dropped. For V = R n and two vectors x, y R n, the standard dot product x, y := x y clearly defines an inner product on R n. We can express the dot product as x y = x T Iy via the n by n identity matrix I. ThematrixI n is symmetric with n positive eigenvalues equal to on its diagonal. Positive definite matrices generalize the properties of I; see Section.3. For example, every positive definite matrix P = P T R n,n can be expressed as a matrix product P = A T A with a nonsingular real square matrix A. For any P = A T A that is positive definite, we may set x, y P := x T Py = x T A T Ay =(Ax T (Ay and thereby obtain an inner product x, y P := R n R n R that differs from the ordinary dot product x y = x, y I. All one needs to do to verify this statement, is to show that..,.. P is bilinear and satisfies the four standard properties of an inner product of Definition, see Problem below. Proposition 3: (a Every bilinear form f(x, y :R n R n Rcan be expressed as f(x, y =x T Sy for a real symmetric n by n matrix S. (b If P = P T R n,n is a positive definite real matrix, then x, y P := x T Py R defines an inner product on R n. (c If P = P C n,n is a positive definite complex matrix, then x, y P := x Py C defines an inner product on C n. Example : (a To write f(x, y =3x y x y +3x 3 y x y +4x 4 y 4 : R 4 R 4 R in the form x T Sy with S = S T R 4,4, we define the diagonal entries s ii of S as the coefficients of x i y i in f, orass =3, s =, s 33 =,ands 44 =4. For i>jwe set the off-diagonal entries s ij in S equal to half the coefficient of x i y j and then define s ji = s ij.thuss = =s and s 3 =.5=s 3.Thus S=.5.5 = ST. 4 (b Determine whether x, y := x T ( 3 3 y is an inner product on R. Clearly x.y is bilinear in ( both x and y R since it is matrix generated. The 3 generating matrix S := with x.y = x.y 3 S = x T Sy is symmetric, i.e., S = S T, and hence its eigenvalues are real according to Section.. Using the trace and determinant conditions of Theorem 9.5, we observe that the two eigenvalues of S add to and multiply to 9 =. Thus both eigenvalues of S must be positive real, making S = S T positive definite and x.y = x.y S an inner product on R according to Proposition 3.
Inner Product Spaces W-55 (c Determine whether u, v := u T ( v is an inner product on C. Clearly the function u.v maps ( any two complex -vectors to a complex ( number ( and it is bilinear. For X = = X we observe that X = ( ( ( and X = =. Therefore X has the eigenvalues and and X is not positive definite. Thus u, v = u, v X is not necessarily an inner product on C n since Proposition 3 does not apply. In fact, u, v X violates the fourth property (d of inner products for u = v = e C : e,e X = ( ( ( = ( ( = C. Therefore u, v X is not an inner product on C. Inner products can help us measure and navigate in abstract vector spaces, such as in spaces of functions. As an example, we now consider the space of continuous functions F [,] := {f : [,] R f continuous} defined on the interval [, ] R. This space is infinite dimensional, see Section 7.(b. By setting f,g := f(xg(x dx for all functions αu + βv,w = (αu + βvw dx=α uw dx + β f,g F [,],wehavemadef [,] into an inner product space. Clearly f,g is linear in both of its function variables f and g since integration is linear in the sense that u + vdx= udx+ vdx. Identities such as vw dx = α u, w + β v, w prove the properties (b and (c of an inner product. Next we observe that property (a holds since f,g = fg dx = gf dx = g, f. And the first part of property (d f,f = f (x dx holds for any integrable function f since f (x. To show that f,f =forf F [,] implies that f =on[,] requires more thought: Every function f F [,] is continuous. If f is not the zero function on [, ], then f(x forsome x [, ]. By continuity there is an interval [a, b], a<b, with x [a, b] and f(x>ɛ> for some given ɛ>andallx [a, b]. Using the additivity of the integral over its domain of integration, we observe that f,f = b a f (x dx = a b f dx + f dx (b aɛ >. a f dx + b f dx Consequently if f is continuous and f,f =, then f= on[,]. Thus
W-56 Appendix D: f,g := f(xg(x dx is an inner product on F [,] = {f :[,] R f continuous }. Different inner product can be defined on F [,] by setting f,g w := f(xg(xw(x dx for an arbitrary continuous weight function w F [,] that is positive on [, ], see Problem 5. Inner products define angles and orthogonality in abstract vector spaces V just as the standard dot products do in R n and C n ; recall Section.. If both f and g V and V is an inner product space with the inner product..,.., then the angle between f and g is defined with respect to a given inner product..,.. by the formula cos (f,g := f,g f,f g, g. And f V is orthogonal to g V,orf g V,if f,g = for the inner product..,.. of V.Here f,g may be complex for two functions f and g when V is a complex vector space. In this case the complex valued cosine function cos(z is used. Example : (a In V = F [,] with the inner product f,g := f(xg(x dx, find the angle between the two functions f(x =andg(x=x V. To find the angle we have to evaluate three different inner products: f,g = x dx=x =, f,f = dx =x =,and g, g = x dx = x 3 3 = 3. Therefore cos (f,g = f,g 3 = f,f g, g =.Andthe 3 ( 3 angle between f and g has the radian measure of arccos. Note that the inner product contains the weight function w(x =. (b Show that the two functions h(x =andk(x=x are orthogonal in V of part (a with its given inner product. We evaluate h, k = (x dx = x dx dx = x x = =. (c Find an orthogonal basis for the subspace span{x, x } F [,] with respect to the inner product f,g := f(xg(x dx.
Inner Product Spaces W-57 Here we use the modified Gram Schmidt process of Chapter for the functions u = x and u = x. v := u = x ; v := v,v u u,v v. We have v,v = x dx = x3 3 = 3 and u,v = x 3 dx = x4 4 = 4. Thus v = x and v = 3 u 4 v = 3 x 4 x are orthogonal in F [,] with respect to the particular inner product. Normalizing the v i with respect to the given inner product..,.. makes w = v,v v = 3xand w = v,v v = 4 5 x 3 5 x since v,v =/3and v,v =/( 5. The students should check this assertion by evaluating v,v = v (xv (x dx, v,v, and v,v. Inner products define vector norms in a natural way. Proposition 4: If..,.. is an inner product on a real vector space V,then x..,.. := x, x / : V R defines a vector norm for every x V with the following properties: ( x..,.. for all x V,and x..,.. = R if and only if x = V. ( αx..,.. = α x..,.. for all vectors x V and all scalars α R. (3 x, y x..,.. y..,.. for all vectors x, y V. (Cauchy Schwarz inequality (4 x + y..,.. x..,.. + y..,.. for all vectors x, y V. (triangle inequality A vector norm.. is called induced by the inner product..,.. if.. =..,.. / as it is in Proposition 4. General vector norms without an underlying inner product can, however, be solely defined by the three norm defining properties (, (, and (4 of Proposition 4. Proofs of both the Cauchy Schwarz and the triangle inequality for the standard euclidean norm x = x x of R n or C n are outlined in Section. and Problems 3 and 6 in Section..P. Definition : A function g(x :V Ris a vector norm on a real vector space V if it satisfies the properties (, (, and (4 of Proposition 4.
W-58 Appendix D: Example 3: The function g(x := max n { x i } : R n R is a vector norm on R n. i= To see this we check the three conditions of a vector norm: Clearly g(x for all x R n,andg(x = if and only if max x i = R,orifandonly if x = R n, establishing property (. Next g(x satisfies property ( since g(αx =max{ αx i } =max{ α x i } = α max{ x i } = α g(x. Finally, the triangle inequality α + β α + β for scalars α, β R helps us prove property (4: g(x + y =max{ x i + y i } max{ x i + y i } max{ x i } +max{ y i } = g(x+g(y. Thus g(x is a vector norm. In Example 5 we learn that g is not induced by any inner product of R n. A vector norm.. measures the length of vectors in V, just as the standard euclidean norm u := u T u measures the length of vectors in R n via the standard dot product. Example 4: (a Find the length of the standard unit vector e R in ( terms of the 3 vector norm that is induced by the inner product x, y = x T y of 3 Example (a. We compute e..,.. = e,e..,.. = ( ( 3 3 Therefore e..,.. =. ( = ( ( 3 =. (b Find the norm of the function f(x =x in the function space V of Example (a. We have f,f = x 4 dx = x5 5 = 5, giving f the length, or norm f..,.. = f,f / = 5. (c Find the length of the vector w = for the norm that is induced by the inner product x, y := x T y on R 3. 3 Clearly the matrix is positive definite as a positive diagonal matrix. Therefore..,.. is an inner product according to Proposition 3. Next we 3 compute the induced vector norm of w:
Inner Product Spaces W-59 w..,.. = w, w = ( = ( 6 3 = ++ = 5, or w..,.. = 5. Note that in the euclidean norm w := w T Iw = w T w = 6. ( x (d Show that for x = R, the function f(x = 3x +4x x +3x : x R Ris an induced vector norm. We have f (x = 3x +4x x +3x = ( ( ( 3 x x x = 3 x ( 3 x T x; see Example 6 in Section.3 for more on quadratic forms 3 ( 3 such as f. ThematrixA:= = A 3 T is real symmetric with eigenvalues that sum to its trace 6 and that multiply to its determinant 9 4 = 5, according to Theorem 9.5. Thus ( the eigenvalues of A are 5 and, making A positive 3 definite and x, y := x T y an inner product on R 3 due to Proposition 3. This inner product induces f(x asanormonr. All inner products....,.. that are induced by an inner product..,.. satisfy the parallelogram identity. Proposition 5: If....,.. is the induced vector norm for the inner product..,.. of an arbitrary real vector space V, then the parallelogram identity ( x + y..,.. + x y..,.. = x..,.. + y..,.. holds for all x, y V. The parallelogram identity states that the sum of the squared lengths of the two sides x and y of any parallelogram equals the average of the lengths of its two diagonals x + y and x y squared for any induced vector norm. Proof: To prove the parallelogram identity in a real vector space we expand its left hand side by using the properties of the norm inducing inner product.
W-6 Appendix D: ( x + y..,.. + x y..,.. = ( x + y,x + y + x y,x y = x, x + x, y + y,y + x, x x, y + y,y = x, x + y,y = x..,.. + y..,... Example 5: The maximum vector norm x := max n { x i } of R n from Example 3 is i= not induced by any inner product of R n since it does not satisfy the parallelogram identity. ( ( For example, in R we have for x = and y = that x = y = ( ( = x y and x + y =sincex+y= and x y =.Thus ( x+y + x y = (4 + = 5 = + = x + y. Proposition 4 makes every inner product space a normed vector space. However, in Example 5 and more generally in functional analysis, it has been shown that not all vector norms derive from inner products. To complete our elementary explorations of inner product spaces and normed vector spaces, we mention without proof that all normed vector spaces whose norm.. satisfies the parallelogram identity of Proposition 5 can be made into an inner product space by setting x, y := x + y x y. Moreover, the parallelogram identity can be generalized to complex vector spaces, but this is beyond the scope of this appendix and elementary linear algebra. A.D.P Problems. Show that the function f(x, y =x y x y +4x 3 y 3 +x y 3 : R 3 R 3 R is a bilinear function for x, y R 3. Is this function an inner product on R 3?Doesfinduce a norm on R 3?. (a Show that x, y A := x T Ay is a bilinear function for every real n by n matrix A. (b Show that if P = P T R n,n is positive definite, then P can be expressed as P = A T A for some nonsingular real matrix A. (Hint: Use Chapter : Diagonalize P orthogonally as U T PU = D = D D for a positive diagonal matrix D and extract P from this matrix equation. (c Show: If P = P T R n,n is positive definite, then x, y P := x T Py is an inner product on R n. (Hint: Use part (b.
Inner Product Spaces W-6 3. Test whether the following functions are (a 8. Orthogonalize the two functions f(x = bilinear and (b inner products on their respective spaces: and g(x = x F [,] with respect ( h(x, y =x T to the weighted inner product f,g w := 4 y for x, y f(xg(xx dx. R 3. 9. Orthogonalize the three functions f(x = ( 4, g(x = x, and h(x = x in F ( k(x, y =x T y for x, y R [,]. with respect to the inner product f,g := ( 4 (3 l(x, y =x T y for x, y R. f(xg(x dx. ( 4 (4 m(x, y =x T y for x, y R.. Show that the standard euclidean vector norm x := x T x for x R n satisfies the (5 n(x, y =x 3x y +4y on R. parallelogram identity. (6 p(x, y = x y on R.. Show that x := n x i is a vector norm 4. (a Find the length of the two vectors x = (... ( and y =... R n for both the standard euclidean vector norm and for the maximum vector norm. (b Find the cosine of the angle between x and y R n in part (a for ( the standard inner product and the 3. ( inner product x T........... y 3 of R n. 5. If w(x > is continuous on the interval [, ], prove that f,g w := f(xg(xw(x dx is an inner product on the space of continuous functions F [,]. 6. Construct a positive and continuous weight function w(x so that the two functions f(x =andg(x=x F [,] become orthogonal with respect to the inner product f,g w := f(xg(xw(x dx, if possible. 7. Construct a positive and continuous weight function w(x so that the two functions f(x =andg(x=x F [,] are not orthogonal with respect to the inner product f,g w := f(xg(xw(x dx, if possible. i= on R n. Is it induced by an inner product or not?. Examine whether the vector norm x of R n in the previous problem is an induced vector norm. 3. (a Assume that the norm.. is an induced norm on V.If u =4, u+v =6, and u v = 5, what is the length of v? What is the distance between u and v? (b Repeat part (a for u =7. 4. Let V be a real inner product space. Let.. be the vector norm that is induced by the inner product..,.. on V. Show that x, y = x + y x y 4 holds for all x, y V. 5. Let {u,..., u k } be an orthonormal set of vectors in an inner product space of finite dimension n k. Prove that x span{u,..., u k } if and only if x = x, u +... + x, u k for the induced vector norm. ( i 6. (a Does x, y := x y define an i inner product on C? (b Repeat ( part (a for x, y := i i x y. i (c Repeat part (a for x, y := x y x y.
W-6 Appendix D: (d Are any of the functions in parts (a through (c bilinear? 7. In a normed vector space V we define the distance function between any two vectors as d(x, y := x y. Show that (a d(x, y =d(y, x, (b d(x, y d(x, z+d(z,y for all x, y, and z V. (c Evaluate the distance between x = ( 4 3 ( and y = 3 R 4 for the euclidean norm and the maximum norm. 8. (a Assume that x, y H := x T Hx and x, y K := x T Kx are two bilinear forms on R n with H and K both n by n and positive definite. If x, y H = x, y K for all x, y R n, show that H = K as matrices. (b Show: If x T Ay = for a real symmetric matrix A and all vectors x, y R n,then A=O n. 9. Let x = ( i +i 3 ( and y = i i C 3. For the standard euclidean vector norm of C 3, compute (a x and y, (b the distance d(x, y, (c y x, (d x+y,and (e the angle between x and y C 3.. In F [,] with the standard inner product f,g := f(xg(x dx, determine the length of the two functions f(x =3x and g(x =x +x F [,],aswellasthe cosine of the angle between f and g.. Consider the function f(p, q :P n P n R defined by f(p, q =p(q( + p(q( + 3p(3q(3, where P n denotes the real variable polynomials of degree not exceeding n. (a Show that f is bilinear on P n P n. (b Show that f induces a norm.. on P m for all m. (c Show that f does not induce a norm on any P m if m>. (d Find x 3x and x x 3 +4x in the vector norm that is induced by f. (e Write down a formula for the induced norm p and p P.
Index A angle W-56 B bilinear form W-53 C Cauchy Schwarz inequality W-57 complex conjugation W-54 D dimension infinite W-55 distance W-6 dot product W-5 dyad W-5 E euclidean norm W-57, W-59 F form bilinear W-53 quadratic W-59 function bilinear W-53 distance W-6 weight W-56 G Gram Schmidt process W-57 I induced norm W-57 inequality Cauchy Schwarz W-57 triangle W-57 inner product W-5, W-53 M matrix multiplication W-5 positive definite W-54 symmetric W-54 multiplication matrix W-5 N norm euclidean W-57, W-59 induced W-57 maximum W-6 vector W-57 O orthogonal W-56 P parallelogram identity W-59 W-63
W-64 Index product dot W-5 dyadic W-5 inner W-5 W-53 outer W-5 Q quadratic form W-59 T triangle inequality W-57 V vector complex W-5 inner product W-5 norm W-57 maximum W-6 orthogonal W-56 outer product W-5 space W-57 W weight function W-56
Frank Uhlig Department of Mathematics Auburn University Auburn, AL 36849 53 tla@auburn.edu http://www.auburn.edu/ uhligfd