QUADRATIC FORMS AND DEFINITE MATRICES. a 11 a 1n. a n1 a nn a 1i x i

Transcription

1 QUADRATIC FORMS AND DEFINITE MATRICES 1 DEFINITION AND CLASSIFICATION OF QUADRATIC FORMS 11Definitionofaquadraticform LetAdenoteannxnsymmetricmatrixwithrealentriesand letxdenoteanncolumnvectorthenq=x AxissaidtobeaquadraticformNotethat For example, consider the matrix andthevectorxqisgivenby Q =x Ax = (x 1 x n ) = (x 1,x,,x n ) a 11 a 1n a n1 a nn a 1i x i a ni x i =a 11 x 1 +a 1x 1 x + +a 1n x 1 x n +a 1 x x 1 +a x + +a nx x n a n1 x n x 1 +a n x n x + +a nn x n = i j a ij x i x j A = [ 1 1 ] [ ] [ ] Q =x 1 Ax = [x 1 x ] 1 [ ] x = [x 1 +x x 1 + x ] x = +x 1x +x 1 x +x = +4x 1x +x ( x n ) (1) 1ClassificationofthequadraticformQ=x Ax:Aquadraticformissaidtobe: a:negativedefinite:q<whenx = b:negativesemidefinite:q forallxandq =forsomex = c:positivedefinite:q>whenx = d:positivesemidefinite:q forallxandq=forsomex = e:indefinite:q>forsomexandq<forsomeotherx Date: February 19, 8 1

2 QUADRATIC FORMS AND DEFINITE MATRICES Considerasanexamplethe3x3diagonalmatrixDbelowandageneral3elementvectorx D = 1 4 The general quadratic form is given by Q =x Ax = [x 1 x x 3 ] 1 x 1 x 4 x 3 = [x 1 x 4x 3 ] x 1 x x 3 = +x +4x 3 Notethatforanyrealvectorx =,thatqwillbepositive,becausethesquareofanynumber is positive, the coefficients of the squared terms are positive and the sum of positive numbers is always positive Also consider the following matrix E = The general quadratic form is given by Q =x Ax = [x 1 x x 3 ] x 1 x x 3 = [ x 1 +x x 1 x x 3 ] = x 1 + x 1x +x 1 x x x 3 = x 1 + x 1x x x 3 = [x 1 x 1x ] x x 3 = x 1 [x x 1x ] x 3 x 1 x x 3 Notethatindependentofthevalueofx 3,thiswillbenegativeifx 1 andx areofoppositesignor equaltooneanothernowconsiderthecasewhere x 1 > x WriteQas Q = x 1 + x 1x x x 3 Thefirst,third,andfourthtermsareclearlynegativeButwith x 1 > x, x 1 > x 1x sothatthefirsttermismorenegativethanthesecondtermispositive,andsothewholeexpression isnegativenowconsiderthecasewhere x 1 < x WriteQas Q = x 1 + x 1x x x 3 Thefirst,third,andfourthtermsareclearlynegativeButwith x 1 < x, x > x 1x sothatthethirdtermismorenegativethanthesecondtermispositive,andsothewholeexpression isnegativethusthisquadraticformisnegativedefiniteforanyandallrealvaluesofx =

3 QUADRATIC FORMS AND DEFINITE MATRICES 3 13GraphicalanalysisWhenxhasonlytwoelements,wecangraphicallyrepresentQin3dimensions A positive definite quadratic form will always be positive except at the point where x =Thisgivesanicegraphicalrepresentationwheretheplaneatx=boundsthefunctionfrom below Figure 1 shows a positive definite quadratic form FIGURE1 PositiveDefiniteQuadraticForm3x 1 +3x 1 x Q Similarly,anegativedefinitequadraticformisboundedabovebytheplanex=Figureshows a negative definite quadratic form

4 4 QUADRATIC FORMS AND DEFINITE MATRICES FIGURE NegativeDefiniteQuadraticForm x 1 x 1 x Q Apositivesemi-definitequadraticformisboundedbelowbytheplanex=butwilltouchthe planeatmorethanthesinglepoint(,),itwilltouchtheplanealongaline Figure3showsa positive semi-definite quadratic form Anegativesemi-definitequadraticformisboundedabovebytheplanex=butwilltouchthe planeatmorethanthesinglepoint(,) Itwilltouchtheplanealongaline Figure4showsa negative-definite quadratic form Anindefinitequadraticformwillnotliecompletelyaboveorbelowtheplanebutwilllieabove forsomevaluesofxandbelowforothervaluesofxfigure5showsanindefinitequadraticform 14 Note on symmetry The matrix associated with a quadratic form B need not be symmetric However, no loss of generality is obtained by assuming B is symmetric We can always take definite and semidefinite matrices to be symmetric since they are defined by a quadratic form Specifically consideranonsymmetricmatrixbanddefineaas 1 (B +B ),Aisnowsymmetricandx Ax = x Bx DEFINITE AND SEMIDEFINITE MATRICES 1Definitionsofdefiniteandsemi-definitematricesLetAbeasquarematrixofordernand letxbeannelementvectorthenaissaidtobepositivesemidefiniteiffforallvectorsx

5 QUADRATIC FORMS AND DEFINITE MATRICES 5 FIGURE3 PositiveSemi-DefiniteQuadraticForm +4x 1x +x x Q FIGURE4 NegativeSemi-DefiniteQuadraticForm +4x 1x x x Q ThematrixAissaidtobepositivedefiniteiffornonzerox x Ax () x Ax > (3)

6 6 QUADRATIC FORMS AND DEFINITE MATRICES FIGURE5 IndefiniteQuadraticForm +4x 1x +x x Q LetAbeasquarematrixofordernThenAissaidtobenegative(semi)definiteiff-Aispositive (semi)definite Diagonal elements of positive definite matrices Theorem1LetAbeapositivedefinitematrixofordermThen If A is only positive semidefinite then a ii >,i = 1,,,m a ii,i = 1,,,m ProofLete i bethem-elementvectorallofwhoseelementsarezerossavetheith,whichisunity Forexampleifm=5andi=thene = [,1,,, ]IfAispositivedefinite,becausee i isnot thenullvector,wemusthave But e i Ae i >,i = 1,,,m (4) e i Ae i = a ii,i = 1,,,m (5) If A is positive semidefinite but not positive definite then repeating the argument above we find a ii = e i Ae i,i = 1,,,m (6)

7 QUADRATIC FORMS AND DEFINITE MATRICES 7 3 Factoring positive definite matrices(cholesky factorization) Theorem LetAbeapositivedefinitematrixofordernThenthereexistsalowertriangularmatrixT such that A = TT (7) Proof Define T as follows T = t 11 t 1 t t 31 t 3 t 33 t n1 t n t n3 t nn (8) NowdefineTT = TT = t 11 t 1 t t 31 t 3 t 33 t n1 t n t n3 t nn t 11 t 1 t 31 t n1 t t 3 t n t 33 t n3 t nn t 11 t 11 t 1 t 11 t 31 t 11 t n1 t 1 t 11 t 1 +t t 1 t 31 +t t 3 t 1 t n1 +t t n t 31 t 11 t 31 t 1 +t 3 t t 31 +t 3 +t 33 t 31 t n1 +t 3 t n +t 33 t n3 t n1 t 11 t n1 t 1 +t n t t n1 t 31 +t n t 3 +t n3 t 33 Σ n i=1 t ni (9) NowdefineA=TT andcomparelikeelements A =TT a 11 a 1 a 13 a 1n a 1 a a 3 a n a 31 a 3 a 33 a 3n = a n1 a n a n3 a nn t 11 t 11 t 1 t 11 t 31 t 11 t n1 t 1 t 11 t 1 +t t 1 t 31 +t t 3 t 1 t n1 +t t n t 31 t 11 t 31 t 1 +t 3 t t 31 +t 3 +t 33 t 31 t n1 +t 3 t n +t 33 t n3 t n1 t 11 t n1 t 1 +t n t t n1 t 31 +t n t 3 +t n3 t 33 Σ n i=1 t ni (1) Solvethesystemnowforeacht ij asfunctionsofthea ij Thesystemisobviouslyrecursive becausewecansolvefirstfort 11,thent 1,etcAschematicalgorithmisgivenbelow

8 8 QUADRATIC FORMS AND DEFINITE MATRICES t = ± t 11 = ± a 11,t 1 = a 1 t 11,t 31 = a 13 t 11,,t n1 = a 1n t 11 a a 1 a 11,t 3 = a 3 t 1 t 31 t,,t n = a n t 1 t n1 t 33 = ± a 33 t 31 t 3 = ± a 33 a 13 a 11 t ( a3 t 1 t 31 t 43 = a 34 t 31 t 41 t 3 t 4,,t n3 = a 3n t 31 t n1 t 3 t n (11) t 33 t 33 This matrix is not unique because the square roots involve two roots The standard procedure is to make the diagonal elements positive Consider the following matrix as an example F = 4 9 WecanfactoritintothefollowingmatrixT T = 1 anditstransposet ThenTT =F TT = 1 1 = Characteristic roots of positive definite matrices Theorem3LetAbeasymmetricmatrixofordernLet λ i,i=1,,nbeitscharacteristicrootsthenifais positivedefinite, λ i >,foralli ProofBecauseAissymmetric,chooseanorthonormalsetofeigenvectorsQClearlyQ AQ = Λ, where ΛisadiagonalmatrixwiththeeigenvaluesofAonthediagonalNowconsideranyoneof therowsofq ThisisoneoftheeigenvectorsofADenoteitbyq i Thenclearly 5 Nonsingularity of positive definite matrices q i Aq i = λ i > (1) Theorem4LetAbeasymmetricmatrixofordernIfAispositivedefinitethenr(A)=n ProofBecauseAissymmetric,chooseanorthonormalsetofeigenvectorsQClearlyQ AQ = Λ, where ΛisadiagonalmatrixwiththeeigenvaluesofAonthediagonalBecauseQisorthogonal itsinverseisitstransposeandwealsoobtainthataq=qλnowbecauseaispositivedefinite, allthecharacteristicrootsonthediagonalof ΛarepositiveThustheinverseof Λisjustamatrix with the reciprocal of each characteristic root on its diagonal Thus Λ is invertible Because QΛ istheproductoftwoinvertiblematrices,itisinvertiblethusaqisinvertible,andbecauseqis invertible, this means A is invertible and of full rank See Dhrymes[1, Proposition 61] or Horn and Johnson[4] t )

9 QUADRATIC FORMS AND DEFINITE MATRICES 9 Theorem5IfAintheabovetheoremismerelypositivesemidefinitethenr(A) <n ProofBecauseAispositivesemidefinite,weknowthat λ i,i=1,,,ntheproofisbasedon showingthatatleastoneoftherootsiszerowecandiagonalizeaas Consequently, for any vector y, Q AQ = Λ (13) y Q AQy = Σ n i=1 λ iy i (14) Now,ifxisanynonnullvector,bythesemidefinitenessofAwehave wherenowweset x Ax = x QQ AQQ x = x QΛQ x = Σ n i=1 λ iy i, (15) y = Q x (16) Because x is nonnull then y is also nonnull, because Q is orthogonal and thus non-singular Ifnoneofthe λ i iszero,15impliesthatforanynonnullx x Ax >, (17) thusshowingatobepositivedefiniteconsequently,atleastoneofthe λ i,i=1,,,n,mustbe zeroandtheremustexistatleastonenonnullxsuchthat But this shows that x Ax = Σ n i=1 λ iy i = (18) 6 Factoring symmetric positive definite matrices r(a) < n (19) Theorem6LetAbeasymmetricmatrix,ofordermThenAispositivedefiniteifandonlyifthereexists amatrixsofdimensionnxmandrankm(n m)suchthat Itispositivesemidefiniteifandonlyif A = S S r(s) < m Proof(Dhrymes[1, Proposition 61] or Horn and Johnson[4]) If A is positive(semi)definite then, asintheproofoftheorem56,wehavetherepresentation A = QΛQ HereQisanorthonormalsetofeigenvectorsand Λisadiagonalmatrixwiththeeigenvaluesof A on the diagonal Taking we have S = Λ 1/ Q

10 1 QUADRATIC FORMS AND DEFINITE MATRICES A = S S If A is positive definite, Λ is nonsingular and thus r(s) = m IfAismerelypositivesemidefinitethenr(Λ) <mandhence Thisprovesthefirstpartofthetheorem On the other hand suppose r(s) < m A = S S () andsisnxmmatrix(n m)ofrankmletxbeanynonnullvectorandnote x Ax = xs Sx (1) Therightsideoftheequationaboveisasumofsquaresandthusiszeroifandonlyif Sx = () IftherankofSism,equationcanbesatisfiedonlywithnullxHenceAispositivedefinite Soforanyx x Ax = x S Sx, andifsisofranklessthanm,thereexistsatleastonenonnullxsuchthat Sx = Consequently, there exists at least one nonnull x such that x Ax = which shows that A is positive semidefinite but not positive definite 7 Using naturally ordered principal minors to test for positive definiteness 71 Definition of naturally ordered(leading) principal minors The naturally ordered principle minorsofamatrixaaredefinedasdeterminantsofthematrices a 11 a 1 a 1 a a k1 a k a 1k a k a kk k = 1,,,n (3) AprincipalminoristheminorofaprincipalsubmatrixofAwhereaprincipalsubmatrixisa matrixformedfromasquarematrixabytakingasubsetconsistingofnrowsandcolumnelements from the same numbered columns The natural ordering considers only those principal minors that fall along the main diagonal Specifically for a matrix A, the naturally ordered principal minors are

11 QUADRATIC FORMS AND DEFINITE MATRICES 11 a 11, or schematically a 11 a 1 a 1 a, a 11 a 1 a 13 a 1 a a 3 a 31 a 3 a 33, a 11 a 1 a 1 a a n1 a n a 1n a n a nn (4) FIGURE 6 Naturally Ordered Principle Minors of a Matrix Theorem7 LetAbeasymmetricmatrixofordermThenAispositivedefiniteiffitsnaturallyordered principal minors are all positive Foraproof,seeGantmacher[,p36]orHadley[3,p6-6] As an example consider the matrix G1

12 1 QUADRATIC FORMS AND DEFINITE MATRICES G1 = 4 9 Elementa 11 =4>Nowconsiderthefirstnaturallyoccurringprincipalxsubmatrix 4 9 = 36 4 = 3 > Now consider the determinant of the entire matrix 4 9 = (4)(9)() + () () () + () () () (9) () () () () () (4) () () = = = 64 > This matrix is then positive definite 8 Characteristic roots of positive semi-definite matrices Let A be a symmetric matrix of order nlet λ i,i=1,,nbeitscharacteristicrootsifaispositivesemi-definitethen λ i i = 1,,,nandatleastone λ i = 9 Using principal minors to test for positive definiteness and positive semidefiniteness 91Definitionofprincipalminors Aprincipalminoroforderrisdefinedasthedeterminantofa principalsubmatrixaprincipalsubmatrixisdefinedasfollowsifaisamatrixofordern,and wewipeoutroftherowsandthecorrespondingrcolumnsaswell,theresulting(n-r)x(n-r) submatrix is called a principal submatrix of A The determinant of this matrix is called a principal minorofaanotherwaytowriteaprincipalminoroforderpis Note:Amatrixofordermhas principal minors of order p where ( ) i1,i A,i 3,,i p = i 1,i,i 3,,i p ( ) m = p ( ) m p a i1 i 1, a i1 i, a i i 1, a i i, a ip i 1, a ip i m! p! (m p)! Forexamplea4x4matrixhas1principalminoroforder4,(thematrixitself),4principalminors oforder3,6principleminorsoforder,and4principleminorsoforder1foratotalof15principal minors a i1 i p a i i p a ip i p (5)

13 QUADRATIC FORMS AND DEFINITE MATRICES 13 Asanexample,considerthefollowingmatrixoforder3whichhas1principalminoroforder3, 3principalminorsoforder, ( ) 3 3! (3) () = =! (3 )! (1) () (1) (1) = 6 = 3 and3principleminorsoforder1foratotalof7principalminors G = Order Order Order14 9 Nowconsiderthegeneral4x4matrixAandsomeofitsprincipalminors A = a 11 a 1 a 13 a 14 a 1 a a 3 a 4 a 31 a 3 a 33 a 34 a 41 a 4 a 43 a 44 A ( ) = a 11 a 13 a 31 a 33 A (14 14 ) = a 11 a 1 a 14 a 1 a a 4 a 41 a 4 a 44 A (4 4 ) = a a 4 a 4 a 44 A (3 3 ) = a 33 (6) 9 A test for for positive definiteness and positive semidefiniteness using principle minors Theorem 8 A matrix A is positive semidefinite iff all the principal minors of A are non-negative ForaproofseeGantmacher[,p37] As an example consider the matrix G G = Thediagonalelementsareallpositivesothe1testispassedNowconsidertheprincipalx minors 4 =8 = 8 > 6 = 1 4 = 8 > = 4 16 = 8 > TheseareallpositiveandsowepassthextestNowconsiderthedeterminantoftheentire matrix

14 14 QUADRATIC FORMS AND DEFINITE MATRICES = ()(4)(6) + () (4) () + () (4) () () (4) () (4) (4) () (6) () () = = = This determinant is zero and so the matrix is positive semidefinite but not positive definite 93 Some characteristics of negative semidefinite matrices The results on positive definite and positive semidefinite matrices have counterparts for negative definite and semidefinite matrices a: A negative semidefinite matrix is negative definite only if it is non-singular b:letabeanegativedefinitematrixofordermthen c:ifaisonlynegativesemidefinitethen a ii < i = 1,,m a ii i = 1,,m d:letabeasymmetricmatrixofordermthenaisnegativedefiniteiffitsnaturallyordered (leading)principal minors alternate in sign starting with a negative number The naturally ordered principle minors of a matrix A are defined as determinants of matrices a 11 a 1 a 1k a 1 a a k k = 1,,,m (7) a k1 a k a kk As an example consider the matrix E = 1 1 Elementa 11 =- <Nowconsiderthefirstnaturallyoccurringprincipalxsubmatrix 1 1 = 4 1 = 3 > Now consider the determinant of the entire matrix 1 1 = ( )( )( ) + (1) () () + (1) () () ( ) () () ( ) () () ( ) (1) (1) = = 6 < This matrix is then negative definite e:letabeasymmetricmatrixoforderm Aisnegativesemidefiniteiffthefollowinginequalities hold: ( ) ( 1) p i1, i A,, i p (8) i 1, i,, i p [1 i 1 i,, i p m,p = 1,,,m]

15 QUADRATIC FORMS AND DEFINITE MATRICES 15 HereA()isthedeterminantofthesubmatrixofAwithprowsandcolumnsofA,ie,it isaprincipleminorofa ( ) i1,i,,i p A i 1,i,,i p = a i1 i 1 a i1 i a i i 1 a i i a ip i 1 a ip i (9) Fora3x3matrixthismeansthatallthediagonalelementsarenon-positive,allxprincipal minors are non-negative and the determinant of the matrix is non-positive Consider asanexamplethematrixg3 G3 = Thediagonalelementsareallnegativesothe1testispassedNowconsidertheprincipal x minors 1 1 =4 1 = 3 > 1 1 = 4 1 = 3 > 1 1 = 4 1 = 3 > TheseareallpositiveandsowepassthextestNowconsiderthedeterminantofthe entire matrix G3 = = [( )( )( )] + [(1) ( 1) ( 1)] + [(1) ( 1) ( 1)] [( 1) ( ) ( 1)] [( 1) ( 1) ( )] [(1) (1) ( )] = ( ) ( ) ( ) = This determinant is zero and so the matrix is negative semidefinite but not negative definite f: Characteristic roots of negative definite matrices LetAbeasymmetricmatrixofordermandlet λ i,i=1,,mbeitsrealcharacteristic rootsifaisnegativedefinitethen λ i < i =1,,,m g: Characteristic roots of negative semi-definite matrices LetAbeasymmetricmatrixofordermandlet λ i,i=1,,mbeitsrealcharacteristic roots If A is negative semi-definite then λ i i = 1,,,mandatleastone λ i =

16 16 QUADRATIC FORMS AND DEFINITE MATRICES 1 Example problems Determine whether the following matrices are positive definite, positive semidefinite, negative definite, negative semidefinite, or indefinite 1 A = B = C = D = E = SECOND ORDER CONDITIONS FOR OPTIMIZATION PROBLEMS AND DEFINITENESS CONDITIONS ON MATRICES 31 Restatement of second order conditions for optimization problems with variables Theorem9Supposethatf(x 1,x )anditsfirstandsecondpartialderivativesarecontinuousthroughouta diskcenteredat(a,b)andthat f x 1 (a,b) = f x (a,b) = Then a:fhasalocalmaximumat(a,b)if f (a, b) < and f canalsowritethisasf 11 < andf 11 f f 1 f x > at (a, b) b:fhasalocalminimumat(a,b)if f (a, b) > and f canalsowritethisasf 11 > andf 11 f f c:fhasasaddlepointat(a,b)if f f 11 f f1 < at (a, b) f x d:thetestisinconclusiveat(a,b)if f f x someotherwaytodeterminethebehavioroffat(a,b) Theexpression f x 1 f x f x [ f x 1 x ] > at (a, b)we [ f x 1 x ] > at (a, b)we 1 > at (a, b) [ ] f x 1 x < at (a, b)wecanalsowritethisas [ f x 1 x ] iscalledthediscriminantoff [ f x 1 x ] = at (a, b)ifthiscasewemustfind 3 Expressing the second order conditions in terms of the definiteness of the Hessian of the objective function

17 QUADRATIC FORMS AND DEFINITE MATRICES 17 31SecondorderconditionsforalocalmaximumTheHessianofafunctionfisthenxnmatrixof second order partial derivatives, that is f 11 f 1 f 1n f 1 f f n H = (3) f n1 f n f nn We can write the discriminant condition as the determinant of the Hessian of the objective functionfwhentherearejustvariablesinthefunctionas f [ f x ] f = f 11 f 1 x 1 x f 1 f = H (31) Thesecondorderconditionforalocalmaximumisthenthat f 11 < and f 11 f 1 f 1 f > which is just the condition that H is negative definite 3 Second order conditions for a local minimum The second order condition for a local minimum is that f 11 > and f 11 f 1 f 1 f > which is just the condition that H is positive definite 33 Extension of condition on Hessian to more than two variables The second order conditions, for localmaximaandminimabasedonthesignoff 11 andthediscriminantwrittenintermsofwhether the Hessian of the objective function is positive or negative, extend to problems involving objective functions with more than variables 4 CONVEXITY AND CONCAVITY AND DEFINITENESS CONDITIONS ON HESSIAN MATRICES 41DefinitionofconcavityLetSbeanonemptyconvexsetinR n Thefunctionf:S R 1 issaid tobeconcaveonsiff(λx 1 +(1-λ)x ) λf(x 1 )+(1-λ)(x )foreachx 1,x Sandforeach λ [,1] Thefunctionfissaidtobestrictlyconcaveiftheaboveinequalityholdsasastrictinequalityfor eachdistinctx 1 x Sandforeach λ (,1) 4 Characterizations of concave functions a:thefunctionfiscontinuousontheinteriorofs b:thefunctionfisconcaveonsifandonlyiftheset {(x,y):x S,y f(x)}isconvexthis setiscalledthehypographoff ItisasubsetofR Thusconcavityoffisequivalentto convexity of its hypograph c:theset {x S,f(x) α }isconvexforeveryreal α d:adifferentiablefunctionfisconcaveonsifandonlyif f (x) f ( x) + f ( x) (x x)foreachdistinctx, x S This implies that tangent line is above the graph e: A twice differentiable function f is concave iff the Hessian H(x) is negative semidefinite for eachx S

18 18 QUADRATIC FORMS AND DEFINITE MATRICES f:letfbetwicedifferentiable TheniftheHessianH(x)isnegativedefiniteforeachx S,fisstrictlyconcave Furtheriffisstrictlyconcave,thentheHessianH(x)isnegative semidefinite for each x S g:everylocalmaximumoffoveraconvexsetw Sisaglobalmaximum h:iff ( x) =foraconcavefunctionthen, xistheglobalmaximumoffovers 43DefinitionofconvexityLetSbeanonemptyconvexsetinR n Thefunctionf:S R 1 issaid tobeconvexonsiff ( λx 1 + (1 λ)x ) λf(x 1 ) + (1 λ )f (x )foreachx 1,x Sandfor each λ [,1]Thefunctionfissaidtobestrictlyconvexiftheaboveinequalityholdsasastrict inequalityforeachdistinctx 1,x, Sandforeach λ (,1) 44 Characteristics of convex functions a:thefunctionfiscontinuousontheinteriorofs b:thefunctionfisconvexonsifandonlyiftheset {(x,y):x S,y f(x)}isconvexthisset iscalledtheepigraphoffitisasubsetofr Thusconcavityoffisequivalenttoconvexity of its epigraph c:theset {x S,f(x) α }isconvexforeveryreal α d:adifferentiablefunctionfisconvexonsifandonlyif f(x) f ( x ) + f ( x) (x x)foreachdistinctx, x S This implies that tangent line is below the graph e: A twice differentiable function f is convex iff the Hessian H(x) is positive semidefinite for eachx S f:letfbetwicedifferentiablethenifthehessianh(x)ispositivedefiniteforeachx S,fis strictly concave Further if f is strictly concave, then the Hessian H(x) is positive semidefiniteforeachx S g:everylocalminimumoffoveraconvexsetw Sisaglobalminimum h:iff ( x) =foraconvexfunctionthen, xistheglobalminimumoffovers 5 LINEAR CONSTRAINTS AND BORDERED MATRICES 51 Definition of a quadratic form with linear constraints Let the quadratic form be given by Q =x Ax = (x 1 x n ) with a set of m linear constraints represented by x B = a 11 a 1n a n1 a nn ( x n ) (3) (x 1 x x n ) b 11 b 1 b 1 b b n1 b n b 1m b m b nm = (33)

19 QUADRATIC FORMS AND DEFINITE MATRICES 19 5 Graphical analysis Consider the indefinite matrix A given by [ ] A = Thequadraticformisgivenby [ ] [ ] Q =x Ax = [x 1 x ] [ x ] = [ x 1 +x x 1 + x ] x = +x 1x +x 1 x +x = +4x 1x +x =4x ( x 1 ) (34) (35) Thegraphin3dimensionsincontainedinfigure7 FIGURE7 IndefiniteQuadraticForm x 1 +4x 1x x Q x 5 5 whereitisclearthatqtakesbothpositiveandnegativevalues ( ) 1 Ifwerestrictourattentiontovaluesofx 1 andx wherex 1 =x oramatrixb = then 1 thefunctionwillbepositiveforallvaluesofx =asisobviousfromthelastlineofequation35to seethismoreclearly,drawaverticalplanethroughthegraphabovealongthex 1 =x lineinfigure 8

20 QUADRATIC FORMS AND DEFINITE MATRICES FIGURE 8 Indefinite Quadratic Form with Restrictions 5 Q x 4 If we combine figure 8 with the plane divding the positive and negative orthants, the positive definiteness of the quadratic from subject to the constraint is even more obvious as shown in figure 9 FIGURE 9 Indefinite Quadratic Form with Restrictions 5 Q x 5 5 Nowalongthesetofpointswherex 1 =x,thefunctionisalwayspositiveexceptwherex 1 =x =Sothisfunctionispositivedefinitesubjecttotheconstraintthatx 1 =x 53DefinitionofaborderedmatrixwithconstraintsDefinetheborderedmatrixH B asfollows

21 QUADRATIC FORMS AND DEFINITE MATRICES 1 H B = a 11 a 1 a 1n b 11 b 1 b 1m a 1 a a n b 1 b b m a n1 a n a nn b n1 b n b nm b 11 b 1 b n1 b 1 b b n b 1m b m b nm (36) OntherightoftheAmatrixweappendthecolumnsoftheBmatrixIftherearethreeconstraints, thenthematrixh B willhaven+3columnsoringeneraln+mcolumns BelowtheAmatrixwe appendthetransposeofthebmatrix,onerowatatimeasweaddconstraintssoifm=,thenh B willhaven+rowsforthecaseofonelinearconstraintweobtain H B = a 11 a 1 b 11 a 1 a b 1 (37) b 11 b 1 54 Definiteness of a quadratic form subject to linear constraints 541ConstructingminorsofH B Todeterminethedefinitenessofthequadraticforminequation 3subjecttoequation33constructthematrixH B inequation36 Thedefinitenessischeckedby analyzingthesignsofthenaturallyorderprincipalminorsofh B startingwiththeminorthathas m+1rowsandcolumnsofthematrixaalongwiththebordersforthoserowsandcolumns For example,ifm=1,thenthefirstminorwecheckis a 11 a 1 b 11 a 1 a b 1 b 11 b 1 (38) Then we check a 11 a 1 a 13 b 11 a 1 a a 3 b 1 a 31 a 3 a 33 b 31 b 11 b 1 b 31 (39) andsoforthingeneralwearecheckingthesignsofminorswithp+mrows,wherepgoesfrom m+1tontheminorswecheckcanbewrittenasbelowwhentherearemconstraints

22 QUADRATIC FORMS AND DEFINITE MATRICES a 11 a 1 a 1p b 11 b 1 b 1m a 1 a a p b 1 b b m a p1 a p a pp b p1 b p b pm b 11 b 1 b p1 b 1 b b p b 1m b m b pm 54 Definiteness of a quadratic form subject to one linear constraint Construct the(n+1)x(n+1) matrix H B asin37wheretheconstraintequationisnowgivenby b 11 x 1 +b 1 x + +b n1 x n = (41) Supposethatb 1 =IfthelastnleadingprincipalminorsofH B havethesamesign,qispositive definite on the constraint set Ifthelastnleadingprincipalminorsalternateinsign,thenQisnegativedefiniteontheconstraint(Simon[5, Section 163]) 543GeneralconditionforamatrixtobepositivedefinitesubjecttoasetoflinearconstaintsIfthedeterminantofH B andtheselastn-mleadingprincipalminorsallhavethesamesignas ( 1) m,thenq ispositivedefiniteontheconstraintsetx B =Withoneconstraint,m=1,sothatthefirstminor is negative as are all subsequent ones 544 General condition for a matrix to be negative definite subject to a set of linear constaints If determinantofh B hasthesamesignas ( 1) p andiftheselastn-mleadingprincipalminorsalternate insign,thenthequadraticformqisnegativedefiniteontheconstraintsetx B = Withone constraint,m=1,andsopstartsat,sothatthefirstminorispositive,thesecondnegativeandso forth 545 If both of these conditions 543 and 544 are violated by nonzero leading principal minors, then Q isindefiniteontheconstraintsetx B = (4)

23 QUADRATIC FORMS AND DEFINITE MATRICES 3 REFERENCES [1] Dhrymes,PJMathematicsforEconometrics-3 r deditionnewyork:springer-verlag, [] Gantmacher, FR The Theory of Matrices Vol I New York: Chelsea Publishing Company, 1977 [3] Hadley, G Linear Algebra Reading: Addison Wesley Publishing Company, 1961 [4] Horn, RA and CR Johnson Matrix Analysis Cambridge: Cambridge University Press, 1985 [5] Simon, CP, and L Blume Mathematics for Economists New York: WW Norton and Company, 1994