Chpter 3 Inner Products nd Norms The geometry of Eucliden spce relies on the fmilir properties of length nd ngle. The bstrct concept of norm on vector spce formlizes the geometricl notion of the length of vector. In Eucliden geometry, the ngle between two vectors is governed by their dot product, which is itself formlized by the bstrct concept of n inner product. Inner products nd norms lie t the hert of nlysis, both liner nd nonliner, in both finite-dimensionl vector spces nd infinite-dimensionl function spces. It is impossible to overemphsize their importnce for both theoreticl developments, prcticl pplictions, nd in the design of numericl solution lgorithms. We begin this chpter with discussion of the bsic properties of inner products, illustrted by some of the most importnt exmples. Mthemticl nlysis is founded on inequlities. The most bsic is the Cuchy Schwrz inequlity, which is vlid in ny inner product spce. The more fmilir tringle inequlity for the ssocited norm is then derived s simple consequence. Not every norm rises from n inner product, nd, in more generl norms, the tringle inequlity becomes prt of the definition. Both inequlities retin their vlidity in both finite-dimensionl nd infinite-dimensionl vector spces. Indeed, their bstrct formultion helps us focus on the key ides in the proof, voiding ll distrcting complictions resulting from the explicit formuls. In Eucliden spce R n, the chrcteriztion of generl inner products will led us to n extremely importnt clss of mtrices. Positive definite mtrices ply key role in vriety of pplictions, including minimiztion problems, lest squres, mechnicl systems, electricl circuits, nd the differentil equtions describing dynmicl processes. Lter, we will generlize the notion of positive definiteness to more generl liner opertors, governing the ordinry nd prtil differentil equtions rising in continuum mechnics nd dynmics. Positive definite mtrices most commonly pper in so-clled Grm mtrix form, consisting of the inner products between selected elements of n inner product spce. The test for positive definiteness is bsed on Gussin elimintion. Indeed, the ssocited mtrix fctoriztion cn be reinterpreted s the process of completing the squre for the ssocited qudrtic form. So fr, we hve confined our ttention to rel vector spces. Complex numbers, vectors nd functions lso ply n importnt role in pplictions, nd so, in the finl section, we formlly introduce complex vector spces. Most of the formultion proceeds in direct nlogy with the rel version, but the notions of inner product nd norm on complex vector spces requires some thought. Applictions of complex vector spces nd their inner products re of prticulr importnce in Fourier nlysis nd signl processing, nd bsolutely essentil in modern quntum mechnics. 2/25/4 88 c 24 Peter J. Olver
v v v 3 v 2 v v 2 v Figure 3.. The Eucliden Norm in R 2 nd R 3. 3.. Inner Products. The most bsic exmple of n inner product is the fmilir dot product v ; w = v w = v w + v 2 w 2 + + v n w n = n i = v i w i, (3.) between (column) vectors v = ( v, v 2,..., v n ) T, w = ( w, w 2,..., w n ) T lying in the Eucliden spce R n. An importnt observtion is tht the dot product (3.) cn be identified with the mtrix product v w = v T w = ( v v 2... w w 2 v n ).. (3.2) between row vector v T nd column vector w. The dot product is the cornerstone of Eucliden geometry. The key fct is tht the dot product of vector with itself, v v = v 2 + v2 2 + + v2 n, is the sum of the squres of its entries, nd hence, s consequence of the clssicl Pythgoren Theorem, equl to the squre of its length; see Figure 3.. Consequently, the Eucliden norm or length of vector is found by tking the squre root: v = v v = v 2 + v2 2 + + v2 n. (3.3) Note tht every nonzero vector v hs positive length, v, while only the zero vector hs length =. The dot product nd Eucliden norm stisfy certin evident properties, nd these serve to inspire the bstrct definition of more generl inner products. Definition 3.. An inner product on the rel vector spce V is piring tht tkes two vectors v, w V nd produces rel number v ; w R. The inner product is required to stisfy the following three xioms for ll u, v, w V, nd c, d R. 2/25/4 89 c 24 Peter J. Olver w n
(i) Bilinerity: c u + d v ; w = c u ; w + d v ; w, u ; c v + d w = c u ; v + d u ; w. (3.4) (ii) Symmetry: v ; w = w ; v. (3.5) (iii) Positivity: v ; v > whenever v, while ; =. (3.6) A vector spce equipped with n inner product is clled n inner product spce. As we shll see, given vector spce cn dmit mny different inner products. Verifiction of the inner product xioms for the Eucliden dot product is strightforwrd, nd left to the reder. Given n inner product, the ssocited norm of vector v V is defined s the positive squre root of the inner product of the vector with itself: v = v ; v. (3.7) The positivity xiom implies tht v is rel nd non-negtive, nd equls if nd only if v = is the zero vector. Exmple 3.2. While certinly the most bsic inner product on R 2, the dot product v w = v w + v 2 w 2 is by no mens the only possibility. A simple exmple is provided by the weighted inner product v ; w = 2v w + 5v 2 w 2, v = ( v v 2 ) ( ) w, w =. (3.8) w 2 Let us verify tht this formul does indeed define n inner product. The symmetry xiom (3.5) is immedite. Moreover, c u + d v ; w = 2(cu + dv )w + 5(cu 2 + dv 2 )w 2 = c(2u w + 5u 2 w 2 ) + d(2v w + 5v 2 w 2 ) = c u ; w + d v ; w, which verifies the first bilinerity condition; the second follows by very similr computtion. (Or, one cn use the symmetry xiom to deduce the second bilinerity identity from the first; see Exercise.) Moreover, ; =, while v ; v = 2v 2 + 5v2 2 > whenever v, since t lest one of the summnds is strictly positive, verifying the positivity requirement (3.6). This serves to estblish (3.8) s n legitimte inner product on R 2. The ssocited weighted norm v = 2v 2 + 5v2 2 defines n lterntive, non-pythgoren notion of length of vectors nd distnce between points in the plne. A less evident exmple of n inner product on R 2 is provided by the expression v ; w = v w v w 2 v 2 w + 4v 2 w 2. (3.9) 2/25/4 9 c 24 Peter J. Olver
Bilinerity is verified in the sme mnner s before, nd symmetry is obvious. Positivity is ensured by noticing tht v ; v = v 2 2v v 2 + 4v2 2 = (v v 2 )2 + 3v 2 2, nd is strictly positive for ll v. Therefore, (3.9) defines nother inner product on R 2, with ssocited norm v = v 2 2v v 2 + 4v2 2. Exmple 3.3. Let c,..., c n be set of positive numbers. The corresponding weighted inner product nd weighted norm on R n re defined by n v ; w = c i v i w i, v = v ; v = n c i vi 2. (3.) i = The numbers c i > re the weights. The lrger the weight c i, the more the i th coordinte of v contributes to the norm. Weighted norms re prticulrly importnt in sttistics nd dt fitting, where one wnts to emphsize certin quntities nd de-emphsize others; this is done by ssigning suitble weights to the different components of the dt vector v. Section 4.3 on lest squres pproximtion methods will contin further detils. Inner Products on Function Spce Inner products nd norms on function spces ply n bsolutely essentil role in modern nlysis nd its pplictions, prticulrly Fourier nlysis, boundry vlue problems, ordinry nd prtil differentil equtions, nd numericl nlysis. Let us introduce the most importnt exmples. Exmple 3.4. Let [, b] R be bounded closed intervl. Consider the vector spce C [, b] consisting of ll continuous sclr functions f(x) defined for x b. The integrl of the product of two continuous functions f ; g = b i = f(x) g(x) dx (3.) defines n inner product on the vector spce C [, b], s we shll prove below. The ssocited norm is, ccording to the bsic definition (3.7), b f = f(x) 2 dx, (3.2) nd is known s the L 2 norm of the function f over the intervl [, b]. The L 2 inner product nd norm of functions cn be viewed s the infinite-dimensionl function spce versions of the dot product nd Eucliden norm of vectors in R n. For exmple, if we tke [, b] = [, 2 π ], then the L2 inner product between f(x) = sin x nd g(x) = cos x is equl to π/2 sin x ; cos x = sin x cos x dx = π/2 2 sin2 x = 2. 2/25/4 9 c 24 Peter J. Olver x =
Similrly, the norm of the function sin x is π/2 sin x = (sin x) 2 dx = π 4. One must lwys be creful when evluting function norms. For exmple, the constnt function c(x) hs norm π/2 π = 2 dx = 2, not s you might hve expected. We lso note tht the vlue of the norm depends upon which intervl the integrl is tken over. For instnce, on the longer intervl [, π ], π = 2 dx = π. Thus, when deling with the L 2 inner product or norm, one must lwys be creful to specify the function spce, or, equivlently, the intervl on which it is being evluted. Let us prove tht formul (3.) does, indeed, define n inner product. First, we need to check tht f ; g is well-defined. This follows becuse the product f(x)g(x) of two continuous functions is lso continuous, nd hence its integrl over bounded intervl is defined nd finite. The symmetry requirement is immedite: f ; g = b f(x) g(x) dx = g ; f, becuse multipliction of functions is commuttive. The first bilinerity xiom c f + d g ; h = c f ; h + d g ; h, mounts to the following elementry integrl identity b [ ] b c f(x) + d g(x) h(x) dx = c f(x) h(x) dx + d b g(x) h(x) dx, vlid for rbitrry continuous functions f, g, h nd sclrs (constnts) c, d. The second bilinerity xiom is proved similrly; lterntively, one cn use symmetry to deduce it from the first s in Exercise. Finlly, positivity requires tht f 2 = f ; f = b f(x) 2 dx. This is cler becuse f(x) 2, nd the integrl of nonnegtive function is nonnegtive. Moreover, since the function f(x) 2 is continuous nd nonnegtive, its integrl will vnish, b f(x) 2 dx = if nd only if f(x) is the zero function, cf. Exercise. This completes the demonstrtion tht (3.) defines bon fide inner product on the function spce C [, b]. 2/25/4 92 c 24 Peter J. Olver
w θ v Figure 3.2. Angle Between Two Vectors. Remrk: The L 2 inner product formul cn lso be pplied to more generl functions, but we hve restricted our ttention to continuous functions in order to void certin technicl complictions. The most generl function spce dmitting this importnt inner product is known s Hilbert spce, which forms the foundtion for modern nlysis, [26], including Fourier series, [5], nd lso lies t the hert of modern quntum mechnics, [, 4, 22]. One does need to be extremely creful when trying to extend the inner product to more generl functions. Indeed, there re nonzero, discontinuous functions with zero L 2 norm. An exmple is {, x =, f(x) = which stisfies f 2 = f(x) 2 dx = (3.3), otherwise, becuse ny function which is zero except t finitely mny (or even countbly mny) points hs zero integrl. We will discuss some of the detils of the Hilbert spce construction in Chpters 2 nd 3. The L 2 inner product is but one of vst number of importnt inner products on functions spce. For exmple, one cn lso define weighted inner products on the function spce C [, b]. The weights long the intervl re specified by (continuous) positive sclr function w(x) >. The corresponding weighted inner product nd norm re b b f ; g = f(x) g(x) w(x) dx, f = f(x) 2 w(x) dx. (3.4) The verifiction of the inner product xioms in this cse is left s n exercise for the reder. As in the finite-dimensionl versions, weighted inner products ply key role in sttistics nd dt nlysis. 3.2. Inequlities. There re two bsolutely fundmentl inequlities tht re vlid for ny inner product on ny vector spce. The first is inspired by the geometric interprettion of the dot product 2/25/4 93 c 24 Peter J. Olver
on Eucliden spce in terms of the ngle between vectors. It is nmed fter two of the founders of modern nlysis, Augustin Cuchy nd Hermn Schwrz, who estblished it in the cse of the L 2 inner product on function spce. The more fmilir tringle inequlity, tht the length of ny side of tringle is bounded by the sum of the lengths of the other two sides is, in fct, n immedite consequence of the Cuchy Schwrz inequlity, nd hence lso vlid for ny norm bsed on n inner product. We will present these two inequlities in their most generl, bstrct form, since this brings their essence into the spotlight. Specilizing to different inner products nd norms on both finite-dimensionl nd infinite-dimensionl vector spces leds to wide vriety of striking, nd useful prticulr cses. The Cuchy Schwrz Inequlity In two nd three-dimensionl Eucliden geometry, the dot product between two vectors cn be geometriclly chrcterized by the eqution v w = v w cos θ, (3.5) where θ mesures the ngle between the vectors v nd w, s drwn in Figure 3.2. Since cos θ, the bsolute vlue of the dot product is bounded by the product of the lengths of the vectors: v w v w. This is the simplest form of the generl Cuchy Schwrz inequlity. We present simple, lgebric proof tht does not rely on the geometricl notions of length nd ngle nd thus demonstrtes its universl vlidity for ny inner product. Theorem 3.5. Every inner product stisfies the Cuchy Schwrz inequlity v ; w v w, v, w V. (3.6) Here, v is the ssocited norm, while denotes bsolute vlue of rel numbers. Equlity holds if nd only if v nd w re prllel vectors. Proof : The cse when w = is trivil, since both sides of (3.6) re equl to. Thus, we my suppose w. Let t R be n rbitrry sclr. Using the three bsic inner product xioms, we hve v + t w 2 = v + t w ; v + t w = v 2 + 2t v ; w + t 2 w 2, (3.7) Russins lso give credit for its discovery to their comptriot Viktor Bunykovskii, nd, indeed, mny uthors ppend his nme to the inequlity. Recll tht two vectors re prllel if nd only if one is sclr multiple of the other. The zero vector is prllel to every other vector, by convention. 2/25/4 94 c 24 Peter J. Olver
with equlity holding if nd only if v = t w which requires v nd w to be prllel vectors. We fix v nd w, nd consider the right hnd side of (3.7) s qudrtic function, p(t) = t 2 + 2bt + c, where = w 2, b = v ; w, c = v 2, of the sclr vrible t. To get the mximum milege out of the fct tht p(t), let us look t where it ssumes its minimum. This occurs when its derivtive vnishes: p (t) = 2t + 2b =, nd thus t t = b = v ; w w 2. Substituting this prticulr minimizing vlue into (3.7), we find v 2 2 v ; w 2 w 2 + v ; w 2 w 2 = v 2 v ; w 2 w 2. Rerrnging this lst inequlity, we conclude tht v ; w 2 w 2 v 2, or v ; w 2 v 2 w 2. Tking the (positive) squre root of both sides of the finl inequlity completes the proof of the Cuchy Schwrz inequlity (3.6). Q.E.D. Given ny inner product on vector spce, we cn use the quotient cos θ = v ; w v w (3.8) to define the ngle between the elements v, w V. The Cuchy Schwrz inequlity tells us tht the rtio lies between nd +, nd hence the ngle θ is well-defined, nd, in fct, unique if we restrict it to lie in the rnge θ π. For exmple, using the stndrd dot product on R 3, the ngle between the vectors v = (,, ) T nd w = (,, ) T is given by cos θ = 2 2 = 2, nd so θ = 3 π =.472..., i.e., 6. On the other hnd, if we use the weighted inner product v ; w = v w +2v 2 w 2 +3v 3 w 3, then cos θ = 3 2 =.6782..., whereby θ =.835482.... 5 Thus, the mesurement of ngle (nd length) is dependent upon the choice of n underlying inner product. Similrly, under the L 2 inner product on the intervl [, ], the ngle θ between the polynomils p(x) = x nd q(x) = x 2 is given by cos θ = x ; x2 x x 2 = x 3 dx x 2 dx x 4 dx = 3 4 5 = 5 6, 2/25/4 95 c 24 Peter J. Olver
so tht θ =.25268 rdins. Wrning: One should not try to give this notion of ngle between functions more significnce thn the forml definition wrrnts it does not correspond to ny ngulr properties of their grph. Also, the vlue depends on the choice of inner product nd the intervl upon which it is being computed. For exmple, if we chnge to the L 2 inner product on the intervl [, ], then x ; x 2 = x 3 dx =, nd hence (3.8) becomes cos θ =, so the ngle between x nd x 2 is now θ = 2 π. Orthogonl Vectors In Eucliden geometry, prticulrly noteworthy configurtion occurs when two vectors re perpendiculr, which mens tht they meet t right ngle: θ = 2 π or 3 2 π, nd so cos θ =. The ngle formul (3.5) implies tht the vectors v, w re perpendiculr if nd only if their dot product vnishes: v w =. Perpendiculrity lso plys key role in generl inner product spces, but, for historicl resons, hs been given more suggestive nme. Definition 3.6. Two elements v, w V of n inner product spce V re clled orthogonl if their inner product vnishes: v ; w =. Orthogonlity is remrkbly powerful tool in ll pplictions of liner lgebr, nd often serves to drmticlly simplify mny computtions. We will devote ll of Chpter 5 to detiled explortion of its mnifold implictions. Exmple 3.7. The vectors v = (, 2 ) T nd w = ( 6, 3 ) T re orthogonl with respect to the Eucliden dot product in R 2, since v w = 6 + 2 ( 3) =. We deduce tht they meet t 9 ngle. However, these vectors re not orthogonl with respect to the weighted inner product (3.8): ( ) ( ) 6 v ; w = ; = 2 6 + 5 2 ( 3) = 8. 2 3 Thus, the property of orthogonlity, like ngles in generl, depends upon which inner product is being used. Exmple 3.8. respect to the inner product p ; q = The polynomils p(x) = x nd q(x) = x 2 2 re orthogonl with p(x) q(x) dx on the intervl [, ], since x ; x 2 2 = x ( x 2 ) ( 2 dx = x 3 2 x ) dx =. They fil to be orthogonl on most other intervls. For exmple, on the intervl [, 2], The Tringle Inequlity x ; x 2 2 2 = x ( x 2 ) 2 ( 2 dx = x 3 2 x ) dx = 3. The fmilir tringle inequlity sttes tht the length of one side of tringle is t most equl to the sum of the lengths of the other two sides. Referring to Figure 3.3, if the 2/25/4 96 c 24 Peter J. Olver
v + w w v Figure 3.3. Tringle Inequlity. first two side re represented by vectors v nd w, then the third corresponds to their sum v + w, nd so v + w v + w. The tringle inequlity is direct consequence of the Cuchy Schwrz inequlity, nd hence holds for ny inner product spce. Theorem 3.9. inequlity The norm ssocited with n inner product stisfies the tringle v + w v + w (3.9) for every v, w V. Equlity holds if nd only if v nd w re prllel vectors. Proof : We compute v + w 2 = v + w ; v + w = v 2 + 2 v ; w + w 2 v 2 + 2 v w + w 2 = ( v + w ) 2, where the inequlity follows from Cuchy Schwrz. Tking squre roots of both sides nd using positivity completes the proof. Q.E.D. Exmple 3.. The vectors v = 2 nd w = 2 sum to v + w = 3 Their Eucliden norms re v = 6 nd w = 3, while v + w = 7. The tringle inequlity (3.9) in this cse sys 7 6 + 3, which is vlid. Exmple 3.. Consider the functions f(x) = x nd g(x) = x 2 +. Using the L 2 norm on the intervl [, ], we find 23 f = (x ) 2 dx = 3, g = (x 2 + ) 2 dx = 5, 77 f + g = (x 2 + x) 2 dx = 6. 77 The tringle inequlity requires 6 3 + 23 5, which is true. 2/25/4 97 c 24 Peter J. Olver 3 2 2.
The Cuchy Schwrz nd tringle inequlities look much more impressive when written out in full detil. For the Eucliden inner product (3.), they re n v i w i n v 2 n i wi 2, i = i = i = (3.2) n (v i + w i ) 2 n vi 2 + n wi 2. i = Theorems 3.5 nd 3.9 imply tht these inequlities re vlid for rbitrry rel numbers v,..., v n, w,..., w n. For the L 2 inner product (3.2) on function spce, they produce the following splendid integrl inequlities: b b b f(x) g(x) dx f(x) 2 dx g(x) 2 dx, (3.2) b [ ] b b 2 f(x) + g(x) dx f(x) 2 dx + g(x) 2 dx, which hold for rbitrry continuous (nd, in fct, rther generl) functions. The first of these is the originl Cuchy Schwrz inequlity, whose proof ppered to be quite deep when it first ppered. Only fter the bstrct notion of n inner product spce ws properly formlized did its innte simplicity nd generlity become evident. 3.3. Norms. Every inner product gives rise to norm tht cn be used to mesure the mgnitude or length of the elements of the underlying vector spce. However, not every norm tht is used in nlysis nd pplictions rises from n inner product. To define generl norm on vector spce, we will extrct those properties tht do not directly rely on the inner product structure. Definition 3.2. A norm on the vector spce V ssigns rel number v to ech vector v V, subject to the following xioms for ll v, w V, nd c R: (i) Positivity: v, with v = if nd only if v =. (ii) Homogeneity: c v = c v. (iii) Tringle inequlity: v + w v + w. As we now know, every inner product gives rise to norm. Indeed, positivity of the norm is one of the inner product xioms. The homogeneity property follows since c v = c v ; c v = c 2 v ; v = c v ; v = c v. Finlly, the tringle inequlity for n inner product norm ws estblished in Theorem 3.9. Here re some importnt exmples of norms tht do not come from inner products. i = i = 2/25/4 98 c 24 Peter J. Olver
Exmple 3.3. Let V = R n. The norm of vector v = ( v v 2... v n ) T is defined s the sum of the bsolute vlues of its entries: v = v + v 2 + + v n. (3.22) The mx or norm is equl to the mximl entry (in bsolute vlue): v = sup { v, v 2,..., v n }. (3.23) Verifiction of the positivity nd homogeneity properties for these two norms is strightforwrd; the tringle inequlity is direct consequence of the elementry inequlity + b + b for bsolute vlues. The Eucliden norm, norm, nd norm on R n re just three representtives of the generl p norm v p = n p v i p. (3.24) This quntity defines norm for ny p <. The norm is limiting cse of s p. Note tht the Eucliden norm (3.3) is the 2 norm, nd is often designted s such; it is the only p norm which comes from n inner product. The positivity nd homogeneity properties of the p norm re strightforwrd. The tringle inequlity, however, is not trivil; in detil, it reds n p v i + w i p n p v i p + n p w i p, (3.25) i = nd is known s Minkowski s inequlity. A proof cn be found in [97]. i = Exmple 3.4. There re nlogous norms on the spce C [, b] of continuous functions on n intervl [, b]. Bsiclly, one replces the previous sums by integrls. Thus, the L p norm is defined s f p = i = i = b p f(x) p dx. (3.26) In prticulr, the L norm is given by integrting the bsolute vlue of the function: f = b f(x) dx. (3.27) The L 2 norm (3.2) ppers s specil cse, p = 2, nd, gin, is the only one rising from n inner product. The proof of the generl tringle or Minkowski inequlity for p, 2 is gin not trivil, [97]. The limiting L norm is defined by the mximum f = mx { f(x) : x b }. (3.28) 2/25/4 99 c 24 Peter J. Olver
Exmple 3.5. Consider the polynomil p(x) = 3x 2 2 on the intervl x. Its L 2 norm is 8 p 2 = (3x 2 2) 2 dx = 5 =.8974.... Its L norm is p = mx { 3x 2 2 : x } = 2, with the mximum occurring t x =. Finlly, its L norm is p = 3x 2 2 dx 2/3 2/3 = (3x 2 2) dx + (2 3x 2 ) dx + ( = 4 3 2 3 ) + 8 3 2 3 + ( 2/3 4 3 2 3 ) = 6 3 2/3 (3x 2 2) dx 2 3 2 = 2.3546.... Every norm defines distnce between vector spce elements, nmely d(v, w) = v w. (3.29) For the stndrd dot product norm, we recover the usul notion of distnce between points in Eucliden spce. Other types of norms produce lterntive (nd sometimes quite useful) notions of distnce tht, nevertheless, stisfy ll the fmilir properties: () Symmetry: d(v, w) = d(w, v); (b) d(v, w) = if nd only if v = w; (c) The tringle inequlity: d(v, w) d(v, z) + d(z, w). Unit Vectors Let V be fixed normed vector spce. The elements u V with unit norm u = ply specil role, nd re known s unit vectors (or functions). The following esy lemm shows how to construct unit vector pointing in the sme direction s ny given nonzero vector. Lemm 3.6. If v is ny nonzero vector, then the vector u = v/ v obtined by dividing v by its norm is unit vector prllel to v. Proof : We compute, mking use of the homogeneity property of the norm: u = v v = v v =. Q.E.D. Exmple 3.7. The vector v = (, 2 ) T hs length v 2 = 5 with respect to the stndrd Eucliden norm. Therefore, the unit vector pointing in the sme direction is u = v = ( ) ( ) 5 =. v 2 5 2 2 5 2/25/4 c 24 Peter J. Olver
On the other hnd, for the norm, v = 3, nd so ũ = v v = 3 ( ) = 2 ( ) 3 2 3 is the unit vector prllel to v in the norm. Finlly, v = 2, nd hence the corresponding unit vector for the norm is û = v = ( ) ( ) = 2. v 2 2 Thus, the notion of unit vector will depend upon which norm is being used. Exmple 3.8. Similrly, on the intervl [, ], the qudrtic polynomil p(x) = x 2 2 hs L2 norm p 2 = ( x 2 ) 2 ( 2 dx = x 4 x 2 + 4 ) dx = 7 6. Therefore, u(x) = p(x) p = 6 7 x2 5 7 is unit polynomil, u 2 =, which is prllel to (or, more correctly, sclr multiple of) the polynomil p. On the other hnd, for the L norm, p = mx { x 2 2 x } = 2, nd hence, in this cse ũ(x) = 2p(x) = 2x 2 is the corresponding unit function. The unit sphere for the given norm is defined s the set of ll unit vectors S = { u = } V. (3.3) Thus, the unit sphere for the Eucliden norm on R n is the usul round sphere For the norm, it is the unit cube S = { x 2 = x 2 + x2 2 + + x2 n = }. S = { x R n x = ± or x 2 = ± or... or x n = ± }. For the norm, it is the unit dimond or octhedron S = { x R n x + x 2 + + x n = }. See Figure 3.4 for the two-dimensionl pictures. In ll cses, the closed unit bll B = { u } consists of ll vectors of norm less thn or equl to, nd hs the unit sphere s its boundry. If V is finite-dimensionl normed vector spce, then the unit bll B forms compct subset, mening tht it is closed nd bounded. This bsic topologicl fct, which is not true in infinite-dimensionl 2/25/4 c 24 Peter J. Olver
.5.5.5 - -.5.5 - -.5.5 - -.5.5 -.5 -.5 -.5 - - - Figure 3.4. Unit Blls nd Spheres for, 2 nd Norms in R 2. spces, underscores the fundmentl distinction between finite-dimensionl vector nlysis nd the vstly more complicted infinite-dimensionl relm. Equivlence of Norms While there re mny different types of norms, in finite-dimensionl vector spce they re ll more or less equivlent. Equivlence does not men tht they ssume the sme vlue, but rther tht they re, in certin sense, lwys close to one nother, nd so for most nlyticl purposes cn be used interchngebly. As consequence, we my be ble to simplify the nlysis of problem by choosing suitbly dpted norm. Theorem 3.9. Let nd 2 be ny two norms on R n. Then there exist positive constnts c, C > such tht c v v 2 C v for every v R n. (3.3) Proof : We just sketch the bsic ide, leving the detils to more rigorous rel nlysis course, cf. [25, 26]. We begin by noting tht norm defines continuous function f(v) = v on R n. (Continuity is, in fct, consequence of the tringle inequlity.) Let S = { u = } denote the unit sphere of the first norm. Any continuous function defined on compct set chieves both mximum nd minimum vlue. Thus, restricting the second norm function to the unit sphere S of the first norm, we cn set c = u 2 = min { u 2 u S }, C = U 2 = mx { u 2 u S }, (3.32) for certin vectors u, U S. Note tht < c C <, with equlity holding if nd only if the the norms re the sme. The minimum nd mximum (3.32) will serve s the constnts in the desired inequlities (3.3). Indeed, by definition, c u 2 C when u =, (3.33) nd so (3.3) is vlid for ll u S. To prove the inequlities in generl, ssume v. (The cse v = is trivil.) Lemm 3.6 sys tht u = v/ v S is unit vector in the first norm: u =. Moreover, by the homogeneity property of the norm, u 2 = v 2 / v. Substituting into (3.33) nd clering denomintors completes the proof of (3.3). Q.E.D. 2/25/4 2 c 24 Peter J. Olver
norm nd 2 norm Figure 3.5. Equivlence of Norms. norm nd 2 norm Exmple 3.2. For exmple, consider the Eucliden norm 2 nd the mx norm on R n. According to (3.32), the bounding constnts re found by minimizing nd mximizing u = mx{ u,..., u n } over ll unit vectors u 2 = on the (round) unit sphere. Its mximl vlue is obtined t the poles, when U = ± e k, with e k =. Thus, C =. The miniml vlue is obtined when u = components, whereby c = u = / n. Therefore, ( n,..., ) hs ll equl n n v 2 v v 2. (3.34) One cn interpret these inequlities s follows. Suppose v is vector lying on the unit sphere in the Eucliden norm, so v 2 =. Then (3.34) tells us tht its norm is bounded from bove nd below by / n v. Therefore, the unit Eucliden sphere sits inside the unit sphere in the norm, nd outside the sphere of rdius / n. Figure 3.5 illustrtes the two-dimensionl sitution. One significnt consequence of the equivlence of norms is tht, in R n, convergence is independent of the norm. The following re ll equivlent to the stndrd ε δ convergence of sequence u (), u (2), u (3),... of vectors in R n : () the vectors converge: u (k) u : (b) the individul components ll converge: u (k) i u i for i =,..., n. (c) the difference in norms goes to zero: u (k) u. The lst cse, clled convergence in norm, does not depend on which norm is chosen. Indeed, the bsic inequlity (3.3) implies tht if one norm goes to zero, so does ny other norm. An importnt consequence is tht ll norms on R n induce the sme topology convergence of sequences, notions of open nd closed sets, nd so on. None of this is true in infinite-dimensionl function spce! A rigorous development of the underlying topologicl nd nlyticl properties of compctness, continuity, nd convergence is beyond the scope of this course. The motivted student is encourged to consult text in rel nlysis, e.g., [25, 26], to find the relevnt definitions, theorems nd proofs. 2/25/4 3 c 24 Peter J. Olver
Exmple 3.2. Consider the infinite-dimensionl vector spce C [, ] consisting of ll continuous functions on the intervl [, ]. The functions hve identicl L norms f n (x) = { nx, x n,, n x, f n = sup { f n (x) x } =. On the other hnd, their L 2 norm /n f n 2 = f n (x) 2 dx = ( nx) 2 dx = 3n goes to zero s n. This exmple shows tht there is no constnt C such tht f C f 2 for ll f C [, ]. Thus, the L nd L 2 norms on C [, ] re not equivlent there exist functions which hve unit L 2 norm but rbitrrily smll L norm. Similr comprtive results cn be estblished for the other function spce norms. As result, nlysis nd topology on function spce is intimtely relted to the underlying choice of norm. 3.4. Positive Definite Mtrices. Let us now return to the study of inner products, nd fix our ttention on the finitedimensionl sitution. Our immedite gol is to determine the most generl inner product which cn be plced on the finite-dimensionl vector spce R n. The resulting nlysis will led us to the extremely importnt clss of positive definite mtrices. Such mtrices ply fundmentl role in wide vriety of pplictions, including minimiztion problems, mechnics, electricl circuits, nd differentil equtions. Moreover, their infinite-dimensionl generliztion to positive definite liner opertors underlie ll of the most importnt exmples of boundry vlue problems for ordinry nd prtil differentil equtions. Let x ; y denote n inner product between vectors x = ( x x 2... x n ) T nd y = ( y y 2... y n ) T, in R n. We begin by writing the vectors in terms of the stndrd bsis vectors: x = x e + + x n e n = n i = x i e i, y = y e + + y n e n = n j = y j e j. (3.35) To evlute their inner product, we will pply the three bsic xioms. We first employ the bilinerity of the inner product to expnd x ; y = n i = x i e i ; n j = y j e j = n i,j = x i y j e i ; e j. 2/25/4 4 c 24 Peter J. Olver
Therefore we cn write x ; y = n i,j = k ij x i y j = x T K y, (3.36) where K denotes the n n mtrix of inner products of the bsis vectors, with entries k ij = e i ; e j, i, j =,..., n. (3.37) We conclude tht ny inner product must be expressed in the generl biliner form (3.36). The two remining inner product xioms will impose certin conditions on the inner product mtrix K. Symmetry implies tht k ij = e i ; e j = e j ; e i = k ji, i, j =,..., n. Consequently, the inner product mtrix K is symmetric: K = K T. Conversely, symmetry of K ensures symmetry of the biliner form: x ; y = x T K y = (x T K y) T = y T K T x = y T K x = y ; x, where the second equlity follows from the fct tht the quntity is sclr, nd hence equls its trnspose. The finl condition for n inner product is positivity. This requires tht x 2 = x ; x = x T K x = n i,j = k ij x i x j for ll x R n, (3.38) with equlity if nd only if x =. The precise mening of this positivity condition on the mtrix K is not s immeditely evident, nd so will be encpsulted in the following very importnt definition. Definition 3.22. An n n mtrix K is clled positive definite if it is symmetric, K T = K, nd stisfies the positivity condition x T K x > for ll x R n. (3.39) We will sometimes write K > to men tht K is symmetric, positive definite mtrix. Wrning: The condition K > does not men tht ll the entries of K re positive. There re mny positive definite mtrices which hve some negtive entries see Exmple 3.24 below. Conversely, mny symmetric mtrices with ll positive entries re not positive definite! Remrk: Although some uthors llow non-symmetric mtrices to be designted s positive definite, we will only sy tht mtrix is positive definite when it is symmetric. But, to underscore our convention nd remind the csul reder, we will often include the superfluous djective symmetric when speking of positive definite mtrices. 2/25/4 5 c 24 Peter J. Olver
Our preliminry nlysis hs resulted in the following chrcteriztion of inner products on finite-dimensionl vector spce. Theorem 3.23. Every inner product on R n is given by where K is symmetric, positive definite mtrix. x ; y = x T K y, for x, y R n, (3.4) Given ny symmetric mtrix K, the homogeneous qudrtic polynomil q(x) = x T K x = n i,j = k ij x i x j, (3.4) is known s qudrtic form on R n. The qudrtic form is clled positive definite if q(x) > for ll x R n. (3.42) Thus, qudrtic form is positive definite if nd only if its coefficient mtrix is. ( ) 4 2 Exmple 3.24. Even though the symmetric mtrix K = hs two 2 3 negtive entries, it is, nevertheless, positive definite mtrix. Indeed, the corresponding qudrtic form q(x) = x T K x = 4x 2 4x x 2 + 3x2 2 = (2x x 2 )2 + 2x 2 2 is sum of two non-negtive quntities. Moreover, q(x) = if nd only if both 2x x 2 = nd x 2 =, which implies x = lso. This proves positivity for ll nonzero x, nd hence K > is indeed positive definite mtrix. The corresponding inner product on R 2 is x ; y = ( x x 2 ) ( 4 2 2 3 ) ( ) y = 4x y 2x y 2 2x 2 y + 3x 2 y 2. On the other hnd, despite the fct tht the mtrix K = entries, it is not positive definite mtrix. Indeed, writing out y 2 q(x) = x T K x = x 2 + 4x x 2 + x2 2, ( ) 2 2 hs ll positive we find, for instnce, tht q(, ) = 2 <, violting positivity. These two simple exmples should be enough to convince the reder tht the problem of determining whether given symmetric mtrix is or is not positive definite is not completely elementry. With little prctice, it is not difficult to red off the coefficient mtrix K from the explicit formul for the qudrtic form (3.4). Exercise shows tht the coefficient mtrix K in ny qudrtic form cn be tken to be symmetric without ny loss of generlity. 2/25/4 6 c 24 Peter J. Olver
Exmple 3.25. Consider the qudrtic form q(x, y, z) = x 2 + 4xy + 6y 2 2xz + 9z 2 depending upon three vribles. The corresponding coefficient mtrix is K = 2 2 6 whereby q(x, y, z) = ( x y z ) 2 2 6 9 9 Note tht the squred terms in q contribute directly to the digonl entries of K, while the mixed terms re split in hlf to give the symmetric off-digonl entries. The reder might wish to try proving tht this prticulr mtrix is positive definite by estblishing positivity of the qudrtic form: q(x, y, z) > for ll nonzero ( x, y, z ) T R 3. Lter, we will devise simple, systemtic test for positive definiteness. Slightly more generlly, qudrtic form nd its ssocited symmetric coefficient mtrix re clled positive semi-definite if x y z q(x) = x T K x for ll x R n. (3.43) A positive semi-definite mtrix my hve null directions, mening non-zero vectors z such tht q(z) = z T K z =. Clerly ny nonzero vector z ker K tht lies in the mtrix s kernel defines null direction, but there my be others. A positive definite mtrix is not llowed to hve null directions, nd so ker K = {}. As consequence of Proposition 2.39, we deduce tht ll positive definite mtrices re invertible. (The converse, however, is not vlid.) Theorem 3.26. If K is positive definite, then K is nonsingulr. ( ) Exmple 3.27. The mtrix K = is positive semi-definite, but not positive definite. Indeed, the ssocited qudrtic form q(x) = x T K x = x 2 2x x 2 + x2 2 = (x x 2 )2 is perfect squre, nd so clerly non-negtive. However, the elements of ker K, nmely the sclr multiples of the vector (, ) T, define null directions, since q(c, c) =. ( ) b Exmple 3.28. By definition, generl symmetric 2 2 mtrix K = is b c positive definite if nd only if the ssocited qudrtic form stisfies q(x) = x 2 + 2bx x 2 + cx2 2 > (3.44) for ll x. Anlytic geometry tells us tht this is the cse if nd only if >, c b 2 >, (3.45) i.e., the qudrtic form hs positive leding coefficient nd positive determinnt (or negtive discriminnt). A direct proof of this elementry fct will pper shortly. 2/25/4 7 c 24 Peter J. Olver.
Furthermore, qudrtic form q(x) = x T K x nd its ssocited symmetric mtrix K re clled negtive semi-definite if q(x) for ll x nd negtive definite if q(x) < for ll x. A qudrtic form is clled indefinite if it is neither positive nor negtive semi-definite; equivlently, there exist one or more points x + where q(x + ) > nd one or more points x where q(x ) <. Detils cn be found in the exercises. Grm Mtrices Symmetric mtrices whose entries re given by inner products of elements of n inner product spce ply n importnt role. They re nmed fter the nineteenth century Dnish mthemticin Jorgen Grm not the metric mss unit! Definition 3.29. Let V be n inner product spce, nd let v,..., v n V. The ssocited Grm mtrix v ; v v ; v 2... v ; v n v K = 2 ; v v 2 ; v 2... v 2 ; v n.......... (3.46) v n ; v v n ; v 2... v n ; v n is the n n mtrix whose entries re the inner products between the chosen vector spce elements. Symmetry of the inner product implies symmetry of the Grm mtrix: k ij = v i ; v j = v j ; v i = k ji, nd hence K T = K. (3.47) In fct, the most direct method for producing positive definite nd semi-definite mtrices is through the Grm mtrix construction. Theorem 3.3. All Grm mtrices re positive semi-definite. The Grm mtrix (3.46) is positive definite if nd only if v,..., v n re linerly independent. Proof : To prove positive (semi-)definiteness of K, we need to exmine the ssocited qudrtic form n q(x) = x T K x = k ij x i x j. i,j = Substituting the vlues (3.47) for the mtrix entries, we find q(x) = n i,j = v i ; v j x i x j. Bilinerity of the inner product on V implies tht we cn ssemble this summtion into single inner product q(x) = n i = x i v i ; n j = x j v j = v ; v = v 2, 2/25/4 8 c 24 Peter J. Olver
where v = x v + + x n v n lies in the subspce of V spnned by the given vectors. This immeditely proves tht K is positive semi-definite. Moreover, q(x) = v 2 > s long s v. If v,..., v n re linerly independent, then v = if nd only if x = = x n =, nd hence q(x) = if nd only if x =. Thus, in this cse, q(x) nd K re positive definite. Q.E.D. Exmple 3.3. Consider the vectors v = 2, v 2 = 3. For the stndrd 6 Eucliden dot product on R 3, the Grm mtrix is ( ) ( ) v v K = v v 2 6 3 =. v 2 v v 2 v 2 3 45 Since v, v 2 re linerly independent, K >. Positive definiteness implies tht the ssocited qudrtic form q(x, x 2 ) = 6x 2 6x x 2 + 45x2 2 is strictly positive for ll (x, x 2 ). Indeed, this cn be checked directly using the criteri in (3.45). On the other hnd, for the weighted inner product the corresponding Grm mtrix is ( ) v ; v K = v ; v 2 v 2 ; v v 2 ; v 2 x ; y = 3x y + 2x 2 y 2 + 5x 3 y 3, (3.48) = ( ) 6 2. (3.49) 2 27 Since v, v 2 re still linerly independent (which, of course, does not depend upon which inner product is used), the mtrix K is lso positive definite. In the cse of the Eucliden dot product, the construction of the Grm mtrix K cn be directly implemented s follows. Given column vectors v,..., v n R m, let us form the m n mtrix A = ( v v 2... v n ). In view of the identifiction (3.2) between the dot product nd multipliction of row nd column vectors, the (i, j) entry of K is given s the product k ij = v i v j = v T i v j of the i th row of the trnspose A T with the j th column of A. In other words, the Grm mtrix cn be evluted s mtrix product: For the preceding Exmple 3.3, A = 3 ( 2, nd so K = A T 2 A = 3 6 6 K = A T A. (3.5) ) 3 2 6 = ( 6 3 3 45 Theorem 3.3 implies tht the Grm mtrix (3.5) is positive definite if nd only if the columns of A re linerly independent vectors. This implies the following result. 2/25/4 9 c 24 Peter J. Olver ).
Proposition 3.32. Given n m n mtrix A, the following re equivlent: () The n n Grm mtrix K = A T A is positive definite. (b) A hs linerly independent columns. (c) rnk A = n m. (d) ker A = {}. Chnging the underlying inner product will, of course, chnge the Grm mtrix. As noted in Theorem 3.23, every inner product on R m hs the form v ; w = v T C w for v, w R m, (3.5) where C > is symmetric, positive definite m m mtrix. Therefore, given n vectors v,..., v n R m, the entries of the Grm mtrix with respect to this inner product re k ij = v i ; v j = v T i C v j. If, s bove, we ssemble the column vectors into n m n mtrix A = ( v v 2... v n ), then the Grm mtrix entry k ij is obtined by multiplying the i th row of A T by the j th column of the product mtrix C A. Therefore, the Grm mtrix bsed on the lterntive inner product (3.5) is given by K = A T C A. (3.52) Theorem 3.3 immeditely implies tht K is positive definite provided A hs rnk n. Theorem 3.33. Suppose A is n m n mtrix with linerly independent columns. Suppose C > is ny positive definite m m mtrix. Then the mtrix K = A T C A is positive definite n n mtrix. The Grm mtrices constructed in (3.52) rise in wide vriety of pplictions, including lest squres pproximtion theory, cf. Chpter 4, nd mechnicl nd electricl systems, cf. Chpters 6 nd 9. In the mjority of pplictions, C = dig (c,..., c m ) is digonl positive definite mtrix, which requires it to hve strictly positive digonl entries c i >. This choice corresponds to weighted inner product (3.) on R m. Exmple 3.34. Returning to the sitution of Exmple 3.3, the weighted inner product (3.48) corresponds to the digonl positive definite mtrix C = 3 2. 5 Therefore, the weighted Grm mtrix (3.52) bsed on the vectors K = A T C A = ( 2 3 6 reproducing (3.49). ) 3 2 3 2 5 6 2, 3 is 6 ), = ( 6 2 2 27 2/25/4 c 24 Peter J. Olver
The Grm mtrix construction is not restricted to finite-dimensionl vector spces, but lso pplies to inner products on function spce. Here is prticulrly importnt exmple. Exmple 3.35. Consider vector spce C [, ] consisting of continuous functions on the intervl x, equipped with the L 2 inner product f ; g = f(x) g(x) dx. Let us construct the Grm mtrix corresponding to the simple monomil functions, x, x 2. We compute the required inner products ; = 2 = x ; x = x 2 = x 2 ; x 2 = x 2 2 = Therefore, the Grm mtrix is dx =, ; x = x 2 dx = 3, ; x2 = x 4 dx = 5, x ; x2 = x dx = 2, x 2 dx = 3, x 3 dx = 4. K = ; ; x ; x 2 x ; x ; x x ; x 2 x 2 ; x 2 ; x x 2 ; x 2 = 2 2 3 3 4 3 4 5. As we know, the monomil functions, x, x 2 re linerly independent, nd so Theorem 3.3 implies tht this prticulr mtrix is positive definite. The lert reder my recognize this prticulr Grm mtrix s the 3 3 Hilbert mtrix tht we encountered in (.67). More generlly, the Grm mtrix corresponding to the monomils, x, x 2,..., x n hs entries k ij = x i ; x j = x i+j 2 dt =, i, j =,..., n +. i + j Therefore, the monomil Grm mtrix is the (n + ) (n + ) Hilbert mtrix (.67): K = H n+. As consequence of Theorems 3.26 nd 3.33, we hve proved the following non-trivil result. Proposition 3.36. The n n Hilbert mtrix H n is positive definite. In prticulr, H n is nonsingulr mtrix. Exmple 3.37. Let us construct the Grm mtrix corresponding to the functions, cos x, sin x with respect to the inner product f ; g = π π f(x) g(x) dx on the intervl 2/25/4 c 24 Peter J. Olver
[ π, π ]. We compute the inner products ; = 2 = π cos x ; cos x = cos x 2 = sin x ; sin x = sin x 2 = π π dx = 2π, ; cos x = π π π cos 2 x dx = π, ; sin x = π π π sin 2 x dx = π, cos x ; sin x = Therefore, the Grm mtrix is simple digonl mtrix: K = definiteness of K is immeditely evident. π π cos x dx =, sin x dx =, π 2π π π cos x sin x dx =.. Positive 3.5. Completing the Squre. Grm mtrices furnish us with n lmost inexhustible supply of positive definite mtrices. However, we still do not know how to test whether given symmetric mtrix is positive definite. As we shll soon see, the secret lredy ppers in the prticulr computtions in Exmples 3.2 nd 3.24. You my recll the importnce of the method known s completing the squre, first in the derivtion of the qudrtic formul for the solution to q(x) = x 2 + 2b x + c =, (3.53) nd, lter, in fcilitting the integrtion of vrious types of rtionl nd lgebric functions. The ide is to combine the first two terms in (3.53) s perfect squre, nd so rewrite the qudrtic function in the form ( q(x) = x + b ) 2 + c b2 =. (3.54) As consequence, ( x + b ) 2 = b2 c 2, nd the well-known qudrtic formul x = b ± b 2 c follows by tking the squre root of both sides nd then solving for x. The intermedite step (3.54), where we eliminte the liner term, is known s completing the squre. We cn perform the sme kind of mnipultion on homogeneous qudrtic form q(x, x 2 ) = x 2 + 2bx x 2 + cx2 2. (3.55) 2/25/4 2 c 24 Peter J. Olver
In this cse, provided, completing the squre mounts to writing q(x, x 2 ) = x 2 + 2bx x 2 + cx2 2 = (x + b x 2 ) 2 + c b2 x 2 2 = y2 + c b2 (3.56) The net result is to re-express q(x, x 2 ) s simpler sum of squres of the new vribles y = x + b x 2, y 2 = x 2. (3.57) It is not hrd to see tht the finl expression in (3.56) is positive definite, s function of y, y 2, if nd only if both coefficients re positive: >, c b 2 >. Therefore, q(x, x 2 ), with equlity if nd only if y = y 2 =, or, equivlently, x = x 2 =, This conclusively proves tht conditions (3.45) re necessry nd sufficient for the qudrtic form (3.55) to be positive definite. Our gol is to dpt this simple ide to nlyze the positivity of qudrtic forms depending on more thn two vribles. To this end, let us rewrite the qudrtic form identity (3.56) in mtrix form. The originl qudrtic form (3.55) is ( ) ( ) q(x) = x T b x K x, where K =, x =. (3.58) b c x 2 Similrly, the right hnd side of (3.56) cn be written s ( ) q (y) = y T D y, where D = c b 2 y 2 2. ( ) y, y =. (3.59) y 2 Anticipting the finl result, the equtions (3.57) connecting x nd y cn themselves be written in mtrix form s ( ) ( y = L T y x x or = + b ) ( ) x 2, where L T = y b. 2 x 2 Substituting into (3.59), we find y T D y = (L T x) T D (L T x) = x T L D L T x = x T K x, where K = LD L T (3.6) is the sme fctoriztion (.56) of the coefficient mtrix, obtined erlier vi Gussin elimintion. We re thus led to the reliztion tht completing the squre is the sme s the LD L T fctoriztion of symmetric mtrix! Recll the definition of regulr mtrix s one tht cn be reduced to upper tringulr form without ny row interchnges. Theorem.32 sys tht the regulr symmetric mtrices re precisely those tht dmit n LD L T fctoriztion. The identity (3.6) is therefore vlid 2/25/4 3 c 24 Peter J. Olver
for ll regulr n n symmetric mtrices, nd shows how to write the ssocited qudrtic form s sum of squres: q(x) = x T K x = y T D y = d y 2 + + d n y2 n, where y = LT x. (3.6) The coefficients d i re the digonl entries of D, which re the pivots of K. Furthermore, the digonl qudrtic form is positive definite, y T D y > for ll y, if nd only if ll the pivots re positive, d i >. Invertibility of L T tells us tht y = if nd only if x =, nd hence positivity of the pivots is equivlent to positive definiteness of the originl qudrtic form: q(x) > for ll x. We hve thus lmost proved the min result tht completely chrcterizes positive definite mtrices. Theorem 3.38. A symmetric mtrix K is positive definite if nd only if it is regulr nd hs ll positive pivots. As result, squre mtrix K is positive definite if nd only if it cn be fctored K = LD L T, where L is specil lower tringulr, nd D is digonl with ll positive digonl entries. Exmple 3.39. Consider the symmetric mtrix K = 2 2 6. Gussin elimintion produces the fctors 9 L = 2, D = 2, L T = 2. 6 in its fctoriztion K = LD L T. Since the pivots the digonl entries, 2, 6 in D re ll positive, Theorem 3.38 implies tht K is positive definite, which mens tht the ssocited qudrtic form stisfies q(x) = x 2 + 4x x 2 2x x 3 + 6x2 2 + 9x2 3 >, for ll x = ( x, x 2, x 3 )T. Indeed, the LD L T fctoriztion implies tht q(x) cn be explicitly written s sum of squres: q(x) = x 2 + 4x x 2 2x x 3 + 6x2 2 + 9x2 3 = y2 + 2y2 2 + 6y2 3, (3.62) where y = x + 2x 2 x 3, y 2 = x 2 + x 3, y 3 = x 3, re the entries of y = L T x. Positivity of the coefficients of the yi 2 (which re the pivots) implies tht q(x) is positive definite. Exmple 3.4. Lets test whether the mtrix K = 2 3 2 3 4 is positive definite. 3 4 8 When we perform Gussin elimintion, the second pivot turns out to be, which immeditely implies tht K is not positive definite even though ll its entries re positive. (The third pivot is 3, but this doesn t help; ll it tkes is one non-positive pivot to disqulify mtrix from being positive definite.) This mens tht the ssocited qudrtic form q(x) = x 2 + 4x x 2 + 6x x 3 + 3x2 2 + 8x 2 x 3 + 8x2 3 ssumes negtive vlues t some points; for instnce, q( 2,, ) =. 2/25/4 4 c 24 Peter J. Olver
A direct method for completing the squre in qudrtic form goes s follows. The first step is to put ll the terms involving x in suitble squre, t the expense of introducing extr terms involving only the other vribles. For instnce, in the cse of the qudrtic form in (3.62), the terms involving x re x 2 + 4x x 2 2x x 3 which we write s (x + 2x 2 x 3 ) 2 4x 2 2 + 4x 2 x 3 x2 3. Therefore, where q(x) = (x + 2x 2 x 3 ) 2 + 2x 2 2 + 4x 2 x 3 + 8x2 3 = (x + 2x 2 x 3 )2 + q(x 2, x 3 ), q(x 2, x 3 ) = 2x 2 2 + 4x 2 x 3 + 8x2 3 is qudrtic form tht only involves x 2, x 3. We then repet the process, combining ll the terms involving x 2 in the remining qudrtic form into squre, writing This gives the finl form q(x 2, x 3 ) = 2(x 2 + x 3 ) 2 + 6x 2 3. q(x) = (x + 2x 2 x 3 ) 2 + 2(x 2 + x 3 ) 2 + 6x 2 3, which reproduces (3.62). In generl, s long s k, we cn write where q(x) = k x 2 + 2k 2 x x 2 + + 2k n x x n + k 22 x2 2 + + k nn x2 n ( = k x + k 2 x k 2 + + k ) 2 n x k n + q(x 2,..., x n ) = k (x + l 2 x 2 + + l n x n ) 2 + q(x 2,..., x n ), l 2 = k 2 k = k 2 k,... l n = k n k = k n k, (3.63) re precisely the multiples ppering in the mtrix L obtined from Gussin Elimintion pplied to K, while n q(x 2,..., x n ) = kij x i x j i,j = 2 is qudrtic form involving one less vrible. The entries of its symmetric coefficient mtrix K re kij = k ji = k ij l j k i, for i j. Thus, the entries of K tht lie on or below the digonl re exctly the sme s the entries ppering on or below the digonl of K fter the the first phse of the elimintion process. 2/25/4 5 c 24 Peter J. Olver