Elements of Mtrix Algebr Klus Neusser Kurt Schmidheiny September 30, 2015 Contents 1 Definitions 2 2 Mtrix opertions 3 3 Rnk of Mtrix 5 4 Specil Functions of Qudrtic Mtrices 6 4.1 Trce of Mtrix......................... 6 4.2 Determinnt............................ 6 4.3 Inverse of Mtrix........................ 7 5 Systems of Equtions 8 6 Eigenvlue, Eigenvector nd Spectrl Decomposition 9 7 Qudrtic Forms 11 8 Prtitioned Mtrices 12 9 Derivtives with Mtrix Algebr 13 10 Kronecker Product 14 1
Foreword These lecture notes re supposed to summrize the min results concerning mtrix lgebr s they re used in econometrics nd economics. For deeper discussion of the mteril, the interested reder should consult the references listed t the end. 1 Definitions A mtrix is rectngulr rry of numbers. Here we consider only rel numbers. If the mtrix hs n rows nd m columns, we sy tht the mtrix is of dimension (n m). We denote mtrices by cpitl bold letters: 11 12... 1m A = (A) ij = ( ij ) = 21 22... 2m...... n1 n2... nm The numbers ij re clled the elements of the mtrix. A (n 1) mtrix is column vector with n elements. Similrly, (1 m) mtrix is row vector with m elements. We denote vectors by bold letters. 1 = 2. n b = (b 1, b 2,..., b m ). A (1 1) mtrix is sclr which is denoted by n itlic letter. The null mtrix (O) is mtrix ll elements equl to zero, i.e. ij = 0 for ll i = 1,..., n nd j = 1,..., m. A qudrtic mtrix is mtrix with the sme number of columns nd rows, i.e. n = m. A symmetric mtrix is qudrtic mtrix such tht ij = ji for ll i = 1,..., n nd j = 1,..., m. A digonl mtrix is qudrtic mtrix such tht the off-digonl elements 2
re ll equl to zero, i.e. ij = 0 for i j. The identity mtrix is digonl mtrix with ll digonl elements equl to one. The identity mtrix is denoted by I or I n. A qudrtic mtrix is sid to be upper tringulr whenever ij = 0 for i > j nd lower tringulr whenever ij = 0 for i < j. Two vectors nd b re sid to be linerly dependent if there exists sclrs α nd β both not equl to zero such α + βb = 0. Otherwise they re sid to be linerly independent. 2 Mtrix opertions Equlity Two mtrices or two vectors re equl if they hve the sme dimension nd if their respective elements re ll equl: A = B ij = b ij for ll i nd j Trnspose Definition 1. The mtrix B is clled the trnspose of mtrix A if nd only if b ij = ji for ll i nd j. The mtrix B is denoted by A or A T. Tking the trnspose of some mtrix is equivlent to interchnging rows nd columns. If A hs dimension (n m) then B hs dimension (m n). Tking the trnspose of column vector gives row vector nd vice vers. In generl we men vectors to be column vectors. Remrk 1. For ny mtrix A, (A ) = A. For symmetric mtrices A = A. Addition nd Subtrction The ddition nd subtrction of mtrices is only defined for mtrices with the sme dimension. 3
Definition 2. The sum (difference) between two mtrices A nd B of the sme dimension is given by the sum (difference) of its elements, i.e. C = A + B c ij = ij + b ij for ll i nd j We hve the following clcultion rules: A + O = A ddition of null mtrix A B = A + ( B) A + B = B + A kommuttivity (A + B) + C = A + (B + C) ssocitivity (A + B) = A + B Product Definition 3. The inner product (dot product, sclr product) of two vectors nd b of the sme dimension (n 1) is sclr (rel number) defined s: n b = b = 1 b 1 + 2 b 2 + + n b n = i b i. The product of sclr c nd mtrix A is mtrix B = ca with b ij = c ij. Note tht ca = Ac when c is sclr. Definition 4. The product of two mtrices A nd B with dimensions (n k) nd (k m), respectively, is given by the mtrix C with dimension (n m) such tht k C = A B c ij = is b sj for ll i nd j Remrk 2. The mtrix product is only defined if the number of columns of the first mtrix is equl to the number of rows of the second mtrix. Thus, lthough A B my be defined, B A is only defined if n = m. Thus for qudrtic mtrices both A B nd B A re defined. Remrk 3. The product of two mtrices is in generl not commuttive, i.e. A B B A. 4 s=1
Remrk 4. The product A B my lso be defined s c ij = (C) ij = i b j where i denotes the i-th row of A nd b j the j-th column of B. Under the ssumption tht dimensions gree, we hve the following clcultion rules: AI = A, IA = A AO = O, OA = O (AB)C = A(BC) = ABC ssocitivity A(B + C) = AB + AC distributivity (B + C)A = BA + CA distributivity c(a + B) = ca + cb distributivity with sclr c (AB) = B A reverse ordering fter trnspose (ABC) = C B A 3 Rnk of Mtrix The column rnk of mtrix is the mximl number of linerly independent columns. The row rnk of mtrix is the mximl number of linerly independent rows. A mtrix is sid to hve full column (row) rnk if the column rnk (row rnk) equls the number of columns (rows). For qudrtic mtrices the column rnk is lwys equl to the row rnk. In this cse we just spek of the rnk of mtrix. The rnk of qudrtic mtrix is denoted by rnk(a). For qudrtic mtrices we hve: rnk(ab) min{rnk(a), rnk(b)} rnk(a ) = rnk(a) rnk(a) = rnk(aa ) = rnk(a A) 5
4 Specil Functions of Qudrtic Mtrices In this section only qudrtic (n n) mtrices with dimensions re considered. 4.1 Trce of Mtrix Definition 5. The trce of mtrix A, denoted by tr(a), is the sum of its digonl elements: tr(a) = The following clcultion rules hold: n ii tr(ca) = c tr(a) tr(a ) = tr(a) tr(a + B) = tr(a) + tr(b) tr(ab) = tr(ba) tr(abc) = tr(bca) = tr(cab) 4.2 Determinnt The determinnt cn be computed ccording to the following formul: n A = ij ( 1) i+j A ij for some rbitrry j The determinnt, computed s bove, is sid to be developed ccording to the j-th column. The term ( 1) i+j A ij is clled the cofctor of the element ij. Thereby A ij is mtrix of dimension ((n 1) (n 1)) which is obtined by deleting the i-th row nd the j-th column. A ij = i n 11 1 1 1 j ij nj 1n in nn 6
If t lest two columns (rows) re linerly dependent, the determinnt is equl to zero nd the inverse of A does not exist. The mtrix is clled singulr in this cse. If the mtrix is nonsingulr then ll columns (rows) re linerly independent. If column or row hs just zeros s its elements, the determinnt is equl to zero. If two columns (rows) re interchnged, the determinnt chnges its sign. For n = 2 nd n = 3, the determinnt is given by trctble formul: n = 2 : A = 11 22 12 21 22 23 n = 3 : A = 11 32 33 12 13 21 32 33 + 12 13 31 22 23 = 11 22 33 11 23 32 21 12 33 + 21 13 32 + 31 12 23 31 13 22 Clcultion rules for the determinnt re: A = A AB = A B ca = c n A 4.3 Inverse of Mtrix If A is qudrtic mtrix, there my exist mtrix B with property AB = BA = I. If such mtrix exists, it is clled the inverse of A nd is denoted by A 1. The inverse of mtrix cn be computed s follows ( 1) 1+1 A 11 ( 1) 2+1 A 21... ( 1) n+1 A n1 A 1 = 1 ( 1) 1+2 A 12 ( 1) 2+2 A 22... ( 1) n+2 A n2 A...... ( 1) 1+n A 1n ( 1) 2+n A 2n... ( 1) n+n A nn where A ij is the mtrix of dimension (n 1) (n 1) obtined from A by deleting the i-th row nd the j-th column. 7
A ij = i n 11 1 1 1 j ij nj 1n in nn The term ( 1) i+j A ij is clled the cofctor of ij. For n = 2, the inverse is given by ( ) A 1 1 22 12 =. 11 22 12 21 21 11 We hve the following clcultion rules if both A 1 nd B 1 exist: ( ) A 1 1 = A (AB) 1 = B 1 A 1 (A ) 1 = ( A 1) A 1 = A 1 order reversed 5 Systems of Equtions Consider the following system of n equtions in m unknowns x 1,..., x m : 11 x 1 + 12 x 2 + + 1m x m = b 1 21 x 1 + 22 x 2 + + 2m x m = b 2... n1 x 1 + n2 x 2 + + nm x m = b n If we collect the unknowns into vector x = (x 1,..., x m ), the coefficients b 1,..., b m in to vector b, nd the coefficients ( ij ) into mtrix A, we cn 8
rewrite the eqution system compctly in mtrix form s follows: 11 12... 1m x 1 21 22... 2m x 2....... } n1 n2... {{ nm x m }}{{} A x b 1 b = 2. A x = b b n }{{} b This eqution system hs unique solution if n = m, i.e. if A is qudrtic mtrix, nd A is nonsingulr, i.e. A 1 exists. The solution is then given by x = A 1 b Remrk 5. To chieve numericl ccurcy it is preferble not to compute the inverse explicitly. There re efficient numericl lgorithms which cn solve the eqution system without computing the inverse. 6 Eigenvlue, Eigenvector nd Spectrl Decomposition In this section we only consider qudrtic mtrices of dimension n n. Eigenvlue nd Eigenvector A sclr λ is sid to be n eigenvlue for the mtrix A if there exists vector x 0 such A x = λx The vector x is clled n eigenvector corresponding to λ. If x is n eigenvector then α x, α 0, is lso n eigenvector. Eigenvectors re therefore not unique. It is therefore sometimes useful to normlize the length of the eigenvectors to one, i.e. to chose the eigenvector such tht x x = 1. 9
Chrcteristic eqution In order to find the eigenvlues nd eigenvectors of mtrix, one hs to solve the eqution system: A x = λx = λi x (A λ I)x = 0 This eqution system hs nontrivil solution, x 0, if nd only if the mtrix (A λ I) is singulr, or equivlently if nd only if the determinnt of (A λ I) is equl to zero. This leds to n eqution in the unknown prmeter λ: A λ I = 0. This eqution is clled the chrcteristic eqution of the mtrix A nd corresponds to polynomil eqution of order n. The n solutions of this eqution (roots) re the eigenvlues of the mtrix. The solutions my be complex numbers. Some solutions my pper severl times. Eigenvectors corresponding to some eigenvlue λ cn be obtined from the eqution (A λ I)x = 0. We hve the following importnt reltions: n tr(a) = λ i nd A = n Eigenvlues nd eigenvectors of symmetric mtrices If A is symmetric mtrix, ll eigenvlues re rel nd there exist n linerly independent eigenvectors x 1,..., x n with the properties x ix j = 0 for i j nd x ix i = 1, i.e the eigenvectors re orthogonl to ech other nd of length one. If we collect of the eigenvectors into n (n n) mtrix X = (x 1,..., x n ), we cn write C C = CC = I. If we collect ll the eigenvlues into digonl mtrix Λ, λ 1 0... 0 0 λ Λ = 2... 0......, 0 0... λ n 10 λ i
we cn digonlize the mtrix A s follows: C AC = C CΛ = IΛ = Λ. This implies tht we cn decompose A into the sum of n mtrices s follows: n A = CΛC = λ i x i x i where the mtrices x i x i hve ll rnk one. The bove decomposition is clled the spectrl decomposition or eigenvlue decomposition of A. The inverse of the mtrix A is now esily clculted by n A 1 = CΛ 1 C 1 = x i x λ i. i s C 1 = C. Remrk 6. Note tht beside symmetric mtrices mny other mtrices, but not ll mtrices, re lso digonlizble. 7 Qudrtic Forms For vector x R n nd qudrtic mtrix A of dimension (n n) the sclr function n n f(x) = x Ax = x i x j ij = x Ax j=1 is clled qudrtic form. The qudrtic form x Ax nd therefore the mtrix A is clled positive (negtive) definite, if nd only if x Ax > 0(< 0) for ll x 0. The property tht A is positive definite implies tht λ i > 0 for ll i ll eigenvlues re positive A > 0 the determinnt is positive A 1 exists tr(a) > 0 11
The first property cn serve s n lterntive definition for positive definite mtrix. The qudrtic form x Ax nd therefore the mtrix A is clled nonnegtive definite or positive semi-definite, if nd only if x Ax 0 for ll x. For nonnegtive definite mtrices we hve: λ i 0 A 0 tr(a) 0 for ll i the determinnt is nonnegtive The first property cn serve s n lterntive definition for nonnegtive definite mtrix. Theorem 1. If the mtrix A of dimension (n m), n > m, hs full rnk then A A is positive definite nd AA is nonnegtive definite. 8 Prtitioned Mtrices Consider qudrtic mtrix P of dimensions ((p + q) (r + s)) which is prtitioned into the (p r) mtrix P 11, the (p s) mtrix P 12, the (q r) mtrix P 21 nd the (q s) mtrix P 22 : ( ) P 11 P 12 P = P 21 P 22 Assuming tht dimensions in the involved multiplictions gree, two prtitioned mtrices re mulitplied s ( ) ( ) ( ) P 11 P 12 Q 11 Q 12 P 11 Q 11 + P 12 Q 21 P 11 Q 12 + P 12 Q 22 = P 21 P 22 Q 21 Q 22 P 21 Q 11 + P 22 Q 21 P 21 Q 12 + P 22 Q 22 Assuming tht P 1 11 exists, the determinnt of prtitioned mtrix is P 11 P 12 P 21 P 22 = P 11 P 22 P 21 P 1 11 P 12 12
nd the inverse is ( ) 1 ( ) P 11 P 12 P 1 11 + P 1 11 P 12 F 1 P 21 P 1 11 P 1 11 P 12 F 1 = P 21 P 22 F 1 P 21 P 1 11 F 1 where F = P 22 P 21 P 1 11 P 12 non-singulr. The determinnt of block digonl mtrix is P 11 O O P 22 = P 11 P 22 nd its inverse is, ssuming tht P 1 11 nd P 1 22 exist, ( P 11 O O P 22 ) 1 ( P 1 11 O = O P 1 22 9 Derivtives with Mtrix Algebr A liner function f from the n-dimensionl vector spce of rel numbers, R n, to the rel numbers, R, f : R n R is determined by the coefficient vector = ( 1,..., n ) : ). y = f(x) = x = n i x i = 1 x 1 + 2 x 2 + + n x n where x is column vector of dimension n nd y sclr. The derivtive of y = f(x) with respect to the column vector x is defined s follows: y x = x x = y/ x 1 1 y/ x 2. = 2. = y/ x n The simultneous eqution system y = Ax cn be viewed s m liner functions y i = ix where i denotes the i-th row of the (m n) dimensionl mtrix A. Thus the derivtive of y i with respect to x is given by y i x = ix x 13 = i n
Consequently the derivtive of y = Ax with respect to row vector x cn be defined s y x = Ax x = y 1 / x 1 y 2 / x. = 2. = A. y n / x The derivtive of y = Ax with respect to column vector x is therefore y x = Ax x = A. For qudrtic mtrix A of dimension (n n) nd the qudrtic form x Ax = n n j=1 x ix j ij the derivtive with respect to the column vector x is defined s x Ax = (A + A )x. x If A is symmetric mtrix this reduces to: x Ax x = 2Ax. The derivtive of the qudrtic form x Ax with respect to the mtrix elements ij is given by x Ax ij = x i x j. Therefore the derivtive with respect to the mtrix A is given by x Ax A = xx. 10 Kronecker Product The Kronecker Product of m n Mtrix A with p q Mtrix B is mp nq Mtrix A B defined s follows: 11 B 12 B... 1n B A B = 21 B 22 B... 21 B...... m1 B m1 B... mn B 14 n
The following clcultion rules hold: (A B) + (C B) = (A + C) B (A B) + (A C) = A (B + C) (A B)(C D) = (AC) (BD) (A B) 1 = A 1 B 1 tr(a B) = tr(a)tr(b) References [1] Amemiy, T., Introduction to Sttistics nd Econometrics, Cmbridge, Msschusetts: Hrvrd University Press, 1994. [2] Dhrymes, P.J., Introductory Econometrics, New York : Springer-Verlg, 1978. [3] Greene, W.H., Econometrics, New York: Mcmilln, 1997. [4] Meyer, C.D., Mtrix Anlysis nd Applied Liner Algebr, Phildelphi: SIAM, 2000. [5] Strng, G., Liner Algeb nd its Applictions, 3rd Edition, Sn Diego: Hrcourt Brce Jovnovich, 1986. [6] Mgnus, J.R., nd H. Neudecker, Mtrix Differentition Clculus with Applictions in Sttistics nd Econometrics, Chichester: John Wiley, 1988. 15