Matrix Algebra Tutorial With Examples in Matlab

Transcription

1 Matr Algebra Tutoral Wth Eamples Matlab by Klaus Moelter Departmet of Agrcultural ad Appled Ecoomcs Vrga Tech emal: web: Specfcally desged as a / day "bare-boes" troducto to Matr Algebra usg Matlab for begg Graduate Ecoometrcs studets Commets & suggestos are always welcome!

2 Notes o Notato Followg stadard coveto (as used ecoometrcs) we wll de the rows of a matr by "" (..), ad the colums by "j"(j :k). Thus, the total umber of rows ad colums are gve by "" ad "k", respectvely. For both row ad colum vectors elemets wll be deed by "" (..). Bold-faced otato wll be used for vectors ad matrces. Addtoal otato wll be troduced o a case-by-case bass. Module I. Basc Vector ad Matr Operatos (see Matlab scrpt scrpt ma_mod) Vector & Matr deftos ad the Traspose operator Vectors: or k Specal case: Scalar ( k ) Matrces:, k > Colum vector Row vector y y y y [ ] 3 Traspose operator [ ] 3 y y y y Matr eamples, k : X 3, k : m m M m m m3 m Traspose operator: X M m m m 3 m m m 3

3 Matr addto / subtracto: a a a3 b b b3 A a a a B b b b a + b a + b a3 + b3 A + B a b a b a3 b a b a b a3 b3 A B a b a b a3 b Note: For ths to work the two matrces must have detcal dmesos,.e. A B, k A k B Specal Matrces: Square matr: k (e: matr X above) Dagoal matr: A square matr wth o-zero elemets o ts dagoal, ad zeros everywhere else. A specal case s the detty matr I, whch has a dagoal of "'s". I Matlab, smply use eye() to create a detty matr of sze. (see scrpt) Symmetrc matr: A square matr for whch m m, j,.e. elemets above the dagoal are mrror mages of elemets below the dagoal. Eample: j j M For ay symmetrc matr: M' M Vector ad Matr multplcato: The smplest case s the multplcato of a row vector by a colum vector: b a [ a a a3 ] b b b a b a b + a b + a b [ ] 3 3 Ths s called the er product of two vectors. Rule: For ths to work the colum dmeso of the frst vector must be equal to the row dmeso of the secod,.e. k a b. Note: Wheever you multply a row vector by a colum vector, the result wll be a scalar ( matr). I geeral, for row vector a ad colum vector b (where k a b ), we get: a b ab Kowg ths result allows us to perform ay multplcato of coformable vectors ad matrces.

4 3 Some geeral rules:. Matrces must be coformable for multplcato,.e. the colum dmeso of the frst matr must be equal to the row dmeso of the secod,.e. k A B. The resultg ew matr wll be of dmeso A by k B 3. Deotg a dvdual cell the resultg matr for row, colum j as m j, use the followg rule to fll these cells: mj (row of frst matr) * (colum j of secod matr) Eamples: Usg a ad b from above: b a b a b a b a 3 ba ba ba 3 b3a b3a b3a Usg X ad M from above: m + m m + m M X m m m m + + m3 + m3 m3 + m3 frst row of M frst colum of X frst row of M secod colum of X secod row of M frst colum of X secod row of secod colum of M X thrd row of M frst colum of X thrd row of M secod colum of X Note: Ay vector or matr ca be pre- or post-multpled by a scalar. Multplcato s the performed elemet-by-elemet. Usg aga M from above, ad scalar s: sm sm sm sm sm sm sm 3 3 Also, for ay detty matr I ad coformable matr M we have: MI M ad / or IM M (thus the ame "detty matr"). Some mportat specal cases: For colum vector wth rows (or elemets), we get. For ay two colum vectors a ad b of equal legth, we ca thus derve results such as:

5 4 ' ( a - b) ( a - b) ( a b ) ' ( a + b) ( a + b) ( a + b ) The -vector: Cosder a colum vector of legth wth all ""'s. Such a vector s ofte deoted as. Some useful results: For scalar s: s s s s ( by matr of ""'s) For, ad some colum vector of equal legth :. I Matlab, use oes(,) to create a -(colum) vector of legth. Combg some of these results: The summed devatos of the elemets of colum vector from ts mea ca be epressed as: ( ) ( ) ( ) Useful Rules: For coformable matrces A, B, C: A B B A ( ) ( A B ) C A ( B C) ( ) A B + C A B + A C Bedg the Rules: The "Dot" operator Matlab Matlab's "dot" operator performs elemet-by-elemet multplcato ad dvsos for matrces that have detcal dmesos. Strctly speakg, ths s ot a permssble "tetbook procedure", but t ca be very useful programmg. Eample:

6 5 A B A./B A.*B Practce: Q: Let 3 0 A B C D E Compute by had: a. A+B b. C*A c. (C*A) d. A C e. A C f. C*D g. D*C h. 3*B. B j. E *C k. E *E l. E*E Check your work usg Matlab (see scrpt ma_mod for solutos)

7 6 Q: Usg vector, oes-vector ( ), ad/or detty matr I ( ), derve the followg epressos:, Q.3: Cosder the vector. Fd A s.t. [ ] Fd B s.t. B. A (ote the traspose at the ed). a Combe mapulatos of ad vector a a to obta a a + a + a Q.4: Usg the "dot" operator combato wth ut vector ad detty matr I, compute : (by had) the followg dervatos of vector [ ] 3 / where s the mea of the elemets of. 4

8 7 Module II. Workg wth Parttoed Matrces (see Matlab scrpt ma_mod) More otes o otato (or: "a cofusg coveto"): Whe performg matr mapulatos, cludg parttog, we ofte wat to refer to dvdual rows or colums. To follow Greee ad some other popular ecoometrcs tetbooks, we wll geercally de colums by "c j " (j..k, as before), ad rows by ther trasposes, e.g. (..., as before). So we thk of each row as a trasposed colum wth dmeso k. (However, some cases, we wll devate from ths coveto, just to keep you o your toes...) Ay matr X ca be vsualzed as beg composed by sub-matrces or vectors. Fragmetg a matr to sub-elemets s called "parttog" a matr. Cosder the followg matr: X There are may dfferet ways to partto X. Here are some eamples: E.: Partto to colums [ ] X c c c c 3 4 where 3 4 c c c c E.: Partto to rows 3 4 X E. 3: 3 4 where X X X X X X where 3 4 X 3 4 X X

9 8 E.4: X X X where X X 3 4 X X X [ ] X There are two basc "rules" for parttog a matr:. Adjacet sub-matrces eed to share the same dmeso alog the "glue" le. Together, the sub-matrces eed to fully cota all orgal elemets, ad thus yeld the orgal matr. Perhaps the best & most mportat aspect of parttoed matrces s that they allow us to compactly epress complcated matr products. Parttog va colums (as the frst eample above) or rows (as eample ) s especally mportat ths respect. For eample, cosder matr X from above. Assume we wat to compute or at least vsualze the geeral structure of the matr product M X X. A elemet-by-elemet approach wll be tedous. Istead, we partto X to colum vectors as above ad proceed from there. From before: X [ c c c 3 c 4 ] Whe performg matr operatos usg parttoed matrces, t s best to look at each elemet of the parttoed matr as a " by " (.e. "pseudo-scalar"), but to keep the actual dmesos of each submatr md. I ths case, we ca vew X as a by 4 "row vector". Naturally, each of these pseudo-scalars has dmeso (4 by ) as s clear from specto of the orgal X. Trasposg a parttoed matr. Take the traspose of the orgal parttoed matr (treatg each sub-matr as a pseudo-scalar). Traspose each sub-matr. here: c c X c 3 c 4 Now we ca use basc matr multplcato to derve the product. Frst, let's double-check that the two costructs are coformable ther pseudo-dmesos. X' s pseudo- 4 by. X s pseudo by 4. The

10 9 result wll be pseudo- 4 by 4. Naturally, coformablty also has to hold actual dmesos: X' s 4 by 4, ad so s X. The actual dmesos of the product wll thus also be a 4 by 4. Thus, we get: c c cc cc3 cc4 cc cc cc3 cc4 X X c 3 c c 3 c c 3 c 3 c 3 c 4 c 4c c 4c c 4c3 c 4c The result s a symmetrc matr. Ths s always true for ay "X'*X" operato, regardless of the orgal dmesos of X. Practce: Q: Cosder by k matr X ad by vector y. Deote the rows of X as ',.., ad the colums of X as c j, j k. Also, deote the elemets of y as y. Epress the followg matr products frst terms of, the terms of c j (plus ay y 's f eeded). () X'y () X'X () XX' Q: Cosder the followg vectors: a a a a b b b b c c c c y y y y z z z z Compute a b ad ab ' (show the full results wth all dvdual elemets). The cosder the 3 3 matrces A [ a b c] ad X [ y z]. Show the full results wth all dvdual elemets for A X. The epress A X terms of vectors a, b, c,, y, ad z.

11 0 Module III. Lear Depedece ad Colum Rak (see Matlab scrpt ma_mod3) Geeral deftos & rules. Whe we talk about the "rak" of a matr, we usually refer to ts colum rak.. The colum rak of a matr s the umber of ts learly depedet colums. 3. The colum rak of a matr ca ever eceed the umber of ts rows (mportat f <k) Eamples: 0 4 A rak(a ) (why?) A rak(a ) A 3 4 rak(a 3 ) (why?) A rak(a 4 ) 3 (lear depedece ca be subtle...see scrpt ma_mod3) More geeral deftos & rules:. A matr X has "full rak" whe rak(x) k. A full-rak matr s ofte called "o-sgular".. A rak-defcet matr s called "sgular". 3. A full-rak matr has a o-zero determat (though determats are oly defed for square matrces). The determat of a sgular matr s Full rak s eeded for the verse of a matr to est (though verses are oly defed for square matrces). 5. rak(a*b) m(rak(a), rak(b)) 6. rak(a) rak(a'a) rak(aa')

12 Module IV. Matr Iverses ad Systems of Lear Equatos (see Matlab scrpt ma_mod4) Iverses are oly defed for square, o-sgular matrces. Cosder square ( ) matr A. The verse of A s defed as follows: - B A B A I (detty matr) Thus: - - A A A A I Importat rules: ( ABC) C B A ( A' ) ( A ) ' There are three cases ecoometrcs where the verse plays a mportat role:. I desty fuctos of multvarate dstrbutos. For smplfcato of comple matr multplcatos 3. Solvg systems of lear equatos To llustrate (): I a product of matrces, ay matr or cluster of matrces that s mmedately pre-or post-multpled by ts verse wll "cacel out". Eample: ( ) - B ACD ACD F BIF BF To llustrate (3): Frst, let's see how a system of lear equatos ca be epressed matr terms, usg the lear regresso model as a eample. I lear regresso aalyss, we stpulate the followg geeral relatoshp for a gve dvdual (or frm, household etc) betwee depedet varable ("outcome") y ad a vector of eplaatory varables ( s set to "" for all f the regresso model cotas a tercept term): [ ] k ' y β + β + + β (omttg ay error terms for ow) k k We ca stack these equatos for all.. dvduals: y β + β + + β k k y β + β + + β k k y β + β + + β k k Net, we ote that the left had sde ca be compactly epressed as a vector y [ y y y ] a gve dvdual, the rght had sde ca be wrtte as a er product of vectors:. For

13 β + β + + β β k k [ ] β β k, β β k where We ca ow wrte the etre system as: β k β k where y β X β X β k We ow eed the verse to solve systems of lear equatos (assumg X s full rak). Chagg otato from β to b (the covetoal symbol for the regresso estmate of β), we have: y Xb where y ( ), X ( k), b ( k) ( ) ( ) X X X y X X X Xb b Recall that X'X s always square, so the full rak codto of X s the oly assumpto eeded for ths verse to est.

14 3 Module V. Quadratc Forms, Characterstc Roots, ad Matr Decomposto (see Matlab scrpt ma_mod5) Cosder a () symmetrc matr A ad a () vector. The the scalar q A s called a "quadratc form". Ths structure occurs may optmzato problems. It s also used to assess the "defteess" of a matr. Specfcally:. If q>0 for all ozero (.e for all that do ot cota zeros eclusvely), A s "postve defte".. If q<0 for all ozero, A s "egatve defte". 3. If q 0 for all ozero, A s "oegatve defte" or "postve semdefte". The defteess of a matr becomes a ssue three ma stuatos Ecoometrcs:. Whe comparg the relatve "magtude" of two matrces: Assume we have two symmetrc matrces of equal dmesos (e.g. the varace-covarace matrces of two competg estmators), A ad B. We would lke to determe whch matr s "larger" (e.g. we would lke to choose the estmator wth "less ose",.e. wth the smaller varace-covarace matr). We could the compute the quadratc form of the dfferece ad eame ts magtude,.e. q ( A - B). If q>0, the dfferece s postve defte, ad we would coclude that A s "larger" tha B. The opposte holds f q<0.. I Cholesky factorzato (CF) of a matr. I may ecoometrc aalyses we would lke to represet a symmetrc, postve defte matr as the product of a lower ad a upper tragular matr. Postve-defteess s a requremet for ths to work. The CF, tur, s eeded to draw from certa multvarate dstrbutos ad to ascerta postve-defteess of varace-covarace matrces mamum lkelhood estmato (MLE). Eample: a a a3 l 0 0 l l l3 A a a a L l l 0 L 0 l l where 3 3 a3 a3 a 3 l3 l3 l l 3 the relatoshp betwee the elemets of A ad L has to be such that ALL' holds. Ths mples: 3 3 a l a l l a l l a l l a l l l l a l l l The Matlab commad for the CF s L chol(a). If A s ot postve defte, ad error message occurs. However, ote that Matlab chol(a) produces a upper tragular. So to recostruct A, use L'*L (ad NOT L*L'). 3. I optmzato problems, to assure that a mamum s foud, the matr of secod dervatves ("Hessa matr") must be egatve-defte. It must be postve-defte for a mmum.

15 4 Assessg Defteess: The oly relable way to assess the defteess property of a matr s by specto of ts egevalues or characterstc roots. Cosder ( ) symmetrc matr A. Characterstc roots ad ther correspodg characterstc vectors are defed va the followg equalty: AC CΛ where all matrces are by. The colums of C are called "egevectors" or "characterstc vectors" of A. Matr Λ s a dagoal matr wth the egevalues of A o ts dagoal. If all egevalues > 0, A s postve defte. If all of them are <0, A s egatve defte. If some are >0, some 0, A s postve sem-defte. I Matlab, the vector of egevalues ca be obtaed quckly usg eg(a). If both egevalues ad egevectors are requred, use [C L]eg(A). C Correspods to matr C above, ad L correspods to matr Λ. You ca verfy that C ad L are correct usg the rule: A CΛC' Ths s called the "spectral decomposto" of A. Importat rules:. If A s pos. def., so s ts verse.. If A s by k wth full colum rak, ad >k, the A'A s pos. def. ad AA' s oegatve def. (mportat for lear regresso models) 3. If two matrces A ad B are both pos. def, ad every egevalue of A > egevalue of B whe sorted from smallest to largest, (A - B) s pos. def (so ths s ultmately how we ca fgure out whch matr s "larger") 4. The rak of a matr s the umber of ts o-zero egevalues. Practce: Cosder A Create a separate Matlab scrpt ad call t m5_practce. Your program should accomplsh the followg: create symmetrc A'*A, call t A get the set of egevalues ad the matr of egevectors for A. verfy that A-80, A-84, ad A-85 hold (Greee p ) a. Deduce the rak of A by spectg the set of egevalues. b. Would you epect the determat of A to be zero? Why or why ot? Verfy ths usg det( ) c. Are the colums A learly depedet why or why ot? d. Verfy A's rak usg rak().

16 5 Matr "Calculus" Scalar-valued fuctos: Cosder a geeral fucto that relates a vector to a scalar y,.e. ( ) where [ ] y f k ( ) f By coveto, the set of dervatves s uderstood to be a k colum vector of the form: f f f ( ) g ( ) where "g" stads for "gradet" f k f ( ) (If for some reaso you wsh to obta a row vector stead, use the otato ) Eample: f f ( ) ( ) + 6 f ( ) [ ] Takg secod dervatves yelds a symmetrc k k matr: f f f k f f f f ( ) k H f f f k k k ( ) where "H" stads for "Hessa". For our eample: f ( ) Some coveet results for scalar valued lear fuctos of (see Greee Apped A, p. 839): a a a a A + ( A A ) ( A f A s symmetrc)

17 6 Systems of equatos (or "vector-valued" fuctos): Now cosder a set of scalar-valued fucto y y y f f f ( ) ( ) ( ) Here s the full set of k parameters cotaed the system. Not all equatos may cota all k elemets, but that's OK. We ca compactly wrte ths as ( ) y f I ths case, coveto dctates that frst dervatves are take wth respect to '. The resultg matr of frst dervatves wll have dmeso by k ad take the followg form: f f f k f f f f ( ) k f f f k If a certa elemet of s ot represeted a gve equato, the correspodg dervatve wll be automatcally set to "0", s.t. the resultg matr s always by k. Dervatves of equato systems play a mportat role regresso models for the estmato of varace-covaraces for fuctos of orgal estmators (usg the so called "Delta method"). If the system s lear wthout ay cross-terms, a coveet result s avalable: y A A A ad A' where A s ( k)