Review of Important Concepts

Transcription

1 Appedix 1 Review f Imprtat Ccepts I 1 AI.I Liear ad Matrix Algebra Imprtat results frm liear ad matrix algebra thery are reviewed i this secti. I the discussis t fllw it is assumed that the reader already has sme familiarity with these tpics. The specific ccepts t be described are used heavily thrughut the bk. Fr a mre cmprehesive treatmet the reader is referred t the bks Nble ad Daiel 1977] ad Graybill 1969]. All matrices ad vectrs are assumed t be real. AI.I.I Defiitis Csider a m x matrix A with elemets aij, i = 1,2,..., m; j = 1,2,...,. A shrthad tati fr describig A is Al ij = aij The traspse f A, which is deted by AT, is defied as the x m matrix with elemets aji r. AT]ij = aji A square matrix is e fr which m =. A square matrix is symmetric if AT = A. The rak f a matrix is the umber f liearly idepedet rws r clums, whichever is less. The iverse f a square x matrix is the square x matrix A-1 fr which A- 1A=AA-1 =1 where 1 is the x idetity matrix. The iverse will exist if ad ly if the rak f A is. If the iverse des t exist, the A is sigular. The determiat f a square x matrix is deted by. It is cmputed as = E aijcij j=1 567

2 568 APPENDIX 1. REVIEW OF IMPORTANT CONCEPTS A1.1. LINEAR AND MATRIX ALGEBRA 569 where C ij = (_l)i+iu.; M ij is the determiat f the submatrix f A btaied by deletig the ith rw ad jth clum ad is termed the mir f aij. Cij is the cfactr f aij Nte that ay chice f i fr i = 1,2,..., will yield the same value fr. A quadratic frm Q is defied as Q = EEaijXiXj. j=l I defiig the quadratic frm it is assumed that aji = aij This etails lss i geerality sice ay quadratic fucti may be expressed i this maer. Q may als be expressed as Q=xTAx where x = Xl X2... x]t ad A is a square x matrix with aji = aij r A is a symmetric matrix. A square x matrix A is psitive semidefiite if A is symmetric ad xtax ~ 0 fr all x =f.. If the quadratic frm is strictly psitive, the A is psitive defiite. Whe referrig t a matrix as psitive defiite r psitive semidefiite, it is always assumed that the matrix is symmetric. The trace f a square x matrix is the sum f its diagal elemets r tr(a) = E aii A partitied i x matrix A is e that is expressed i terms f its submatrices. A example is the 2 x 2 partitiig Al.l.2 Special Matrices A = All A 12]. A2l A22 Each "elemet" Aij is a submatrix f A. The dimesis f the partitis are give as k x 1 k x ( -1) ] (m - k) x 1 (m - k) x ( -l). A diagal matrix is a square x matrix with aij = 0 fr i -:f. j r all elemets ff the pricipal diagal are zer. A diagal matrix appears as a ll 0 A = ~. a at] A diagal matrix will smetimes be deted by diag(alb a22,..., a). The iverse f a diagal matrix is fud by simply ivertig each elemet the pricipal diagal. A geeralizati f the diagal matrix is the square x blck diagal matrix All 0 0] A22 0 A= A k k i which all submatrices Aii are square ad the ther submatrices are idetically zer. The dimesis f the submatrices eed t be idetical. Fr istace, if k = 2, All might have dimesi 2 x 2 while A22 might be a scalar. If all Au are sigular, the the iverse is easily fud as Als, the determiat is A-I 11 A- 1 = : A square x matrix is rthgal if A - I 22 = IT det(a ii). A-l=A T ~ ]. Fr a matrix t be rthgal the clums (ad rws) must be rthrmal r if A - I kk A = al a2.. a] where a, detes the ith clum, the cditis T {O fr i -:f. j aiaj= 1 ~.. lr l = J must be satisfied. A imprtat example f a rthgal matrix arises i mdelig f data by a sum f harmically related siusids r by a discrete Furier series. As a example, fr eve V2 V2 A=_1_ I V1 1 cs 211' -L cs 21r( j) si 211' si 21r(i- l) V2... V cs 21r{-l) 1 21rj(-l) si 21r{-l). si 21r(i- l)(-l) V2... V2 cs...

3 570 APPENDIX 1. REVIEW OF IMPORTANT CONCEPTS Al.l. LINEAR AND MATRIX ALGEBRA 571 is a rthgal matrix. This fllws frm the rthgality relatiships fr i, j = 0,1,...,/2 ad fr i, j = 1, 2,..., /2-1 -l 2k' 2k' {O i:l= j 1T' 't 1T' J " E cs--cs-- = '2 't = J = 1,2,..., '2-1 k=o i = j = 0, j ~. 21T'ki. 21rkj s L.J si--si-- = -Uij k=o 2 ad fially fr i = 0, 1,..., /2; j = 1,2,..., /2-1 ~ 21rki. 21rkj 0 L.J cs--si-- =. k=o These rthgality relatiships may be prve by expressig the sies ad csies i terms f cmplex expetials ad usig the result -l (2 ) Eexp j~kl k=o =6 k l fr k, 1= 0,1,..., - 1 Oppeheim ad Schafer 1975]. A idemptet matrix is a square x matrix which satisfies A 2=A. This cditi implies that A' = A fr 1 ~ 1. A example is the prjecti matrix A = H(HTH)-IH T where H is a m x full rak matrix with m >. A square x Teplitz matrix is defied as A]ij = ai-j r a a_i a-2... a_(_l)] a a_i... a_(-2) A= 7... (A1.1) a-l a-2 a-3. a Each elemet alg a rthwest-sutheast diagal is the same. If i additi, a-k = ak, the A is symmetric Teplitz. Al.l.3 Matrix Maipulati ad Frmulas Sme useful frmulas fr the algebraic maipulati f matrices are summarized i this secti. Fr x matrices A ad B the fllwig relatiships are useful. (AB)T (A T )- 1 (AB)-l det(a T) det(ea) det(ab) det(a- 1 ) tr(ab) tr(atb) Als, fr vectrs x ad y we have BTA T (A-1)T B- 1A-1 e (e a scalar) det(b) 1 tr(ba) E i=1 j=1 EA]ijB]ij' v"x = tr(xyt). It is frequetly ecessary t determie the iverse f a matrix aalytically. T d s e ca make use f the fllwig frmula. The iverse f a square x matrix is C T A- 1 = where C is the square x matrix f cfactrs f A. The cfactr matrix is defied by Cl ij = (-l)i+ j M ij where M ij is the mir f aij btaied by deletig the ith rw ad jth clum f A. Ather frmula which is quite useful is the matrix iversi lemma (A + BCD)-l = A-I - A-IB(DA-IB + C- 1 )- IDA- 1 where it is assumed that A is x, B is x m, C is m x m, ad D is m x ad that the idicated iverses exist. A special case kw as Wdbury's idetity results fr B a x 1 clum vectr u, C a scalar f uity, ad D a 1 x rw vectr u T The, T)-1 = A-I _ A-luuTA-I (A +uu.. Partitied matrices may be maipulated accrdig t the usual rules f matrix algebra by csiderig each submatrix as a elemet. Fr multiplicati f partitied

4 572 APPENDIX 1. REVIEW OF IMPORTANT CONCEPTS 1 matrices the submatrices which are multiplied tgether must be cfrmable. As a illustrati, fr 2 x 2 partitied matrices AB = A11 A12 ] B ll B12 ] A21 A22 B21 B22 AllBll + A12B2l AllB12 + A12B22 ] A2lB ll + A22B2l A2lB12 + A22B22. The traspsiti f a partitied matrix is frmed by traspsig the submatrices f the matrix ad applyig T t each submatrix. Fr a 2 x 2 partitied matrix All A 12] T _ Afl Arl] A2l A22 - Ai2 Ar2. The extesi f these prperties t arbitrary partitiig is straightfrward. Determiati f the iverses ad determiats f partitied matrices is facilitated by emplyig the fllwig frmulas. Let A be a square x matrix partitied as The, A _ All A12 ] _ k x k k x ( - k) ] - A 2l A22 - ( - k) x k ( - k) x ( - k) A-I _ (All - A12A2l A2l)-1 -(All - Al2A2l A21)-1Al2A2l ] - -(A22 - A2lA1l A I2)-1 A2lA1l (A22 - A2lA1l A I2)- 1 det(a22) det(aii - Al2A221 A2l) det(aii) det(a22 - A2IAIlAI2) where the iverses f All ad A22 are assumed t exist. Al.l. LINEAR AND MATRIX ALGEBRA 573 b. the pricipal mirs are all psitive. (The ith pricipal mir is the determiat f the submatrix frmed by deletig all rws ad clums with a idex greater tha i.) If A ca be writte as i (A1.2), but C is t full rak r the pricipal mirs are ly egative, the A is psitive semidefiite. 3. If A is psitive defiite, the tile Iverse exists ad may be fud frm (A1.2) as A-I = (C-I)T(C-I). 4. Let A be psitive defiite. If B is a m x matrix f full rak with m ~, the BAB T is als psitive defiite. 5. If A is psitive defiite (psitive semidefiite), the Al.l.5 a. the diagal elemets are psitive (egative) b. the determiat f A, which is a pricipal mir, is psitive (egative). Eigedecmpsti f Matrices A eigevectr f a square x matrix A is a x 1 vectr v satisfyig Av = AV (A1.3) fr sme scalar A, which may be cmplex. A is the eigevalue f A crrespdig t the eigevectr v. It is assumed that the eigevectr is rmalized t have uit legth r vtv = 1. If A is symmetric, the e ca always fid liearly idepedet eigevectrs, althugh they will t i geeral be uique. A example is the idetity matrix fr which ay vectr is a eigevectr with eigevalue 1. If A is symmetric, the the eigevectrs crrespdig t distict eigevalues are rthrmal r vtvj = 6 i j ad the eigevalues are real. If, furthermre, the matrix is psitive defiite (psitive semidefiite), the the eigevalues are psitive (egative). Fr a psitive semidefiite matrix the rak is equal t the umber f zer eigevalues. The defiig relati f (Al.3) ca als be writte as Al.l.4 Therems A VI V2... v ] = AIVI A2V2... AV] Sme imprtat therems used thrughut the text are summarized i this secti. 1. A square x matrix A is ivertible (sigular) if ad ly if its clums (r rws) are liearly idepedet r, equivaletly, if its determiat is zer. I such a case, A is full rak. Otherwise, it is sigular. 2. A square x matrix A is psitive defiite if ad ly if r where V A AV=VA VI V2. V ] diag(ai' A2',A). (Al.4) a. it ca be writte as If A is symmetric s that the eigevectrs crrespdig t distict eigevalues are A=CC T (A1.2) rthrmal ad the remaiig eigevectrs are chse t yield a rthrmal eigevectr set, the V is a rthgal matrix. As such, its iverse is VT, s that where C is als x ad is full rak ad hece ivertible, r (Al.4)

5 574 APPENDIX 1. REVIEW OF IMPORTANT CONCEPTS A1.2. PROBABILITY, RANDOM PROCESSES, TIME SERIES MODELS 575 becmes Als, the iverse is easily determied as A VAV T EAiViV. The extesi t a set f radm variables r a radm vectr x = Xl X2 X]T with mea E(x) = Px ad cvariace matrix is the multivariate Gaussia PDF E (x - Px)(x - Px)T] = Cx A-I = V T - 1A- lv-1 VA-1V T 1 E~ViV. ' A fial useful relatiship fllws frm (A1.4) as Al.2 det(v) det(v- 1) IIx, Prbability, Radm Prcesses, ad Time Series Mdels A assumpti is made that the reader already has sme familiarity with prbability thery ad basic radm prcess thery. This chapter serves as a review f these tpics. Fr thse readers eedig a mre extesive treatmet the text by Papulis 1965] prbability ad radm prcesses is recmmeded. Fr a discussi f time series mdelig see Kay 1988]. Al.2.1 Useful Prbability Desity Fuctis A prbability desity fucti (PDF) which is frequetly used t mdel the statistical behavir f a radm variable is the Gaussia distributi. A radm variable x with mea p,x ad variace O'~ is distributed accrdig t a Gaussia r rmal distributi if the PDF is give by p(x) = J2~U~ exp - 2~~ (x - p",y] The shrthad tati x tv N(P,x, O'~) is fte used, where f'v meas "is distributed accrdig t." If x f'v N(, O'~), the the mmets f x are k)_{ 1 3 (k-1)0'~ keve E( x - 0 k dd < x < 00. (Al.5) ( ) 1 1 T -1 ] px = If i exp --2(X- P x ) C x (x-px). (21r) det (C x ) (Al.6) Nte that C x is a x symmetric matrix with Cx]ij = E{Xi - E(Xi)]Xj - E(xj)]} = COV(Xi' Xj) ad is assumed t be psitive defiite s that C x is ivertible. If C x is a diagal matrix, the the radm variables are ucrrelated. I this case (A1.6) factrs it the prduct f N uivariate Gaussia PDFs f the frm f (A1.5), ad hece the radm variables are als idepedet. If x is zer mea, the the higher rder jit mmets are easily cmputed. I particular the furth-rder mmet is E(XiXjXkXI) = E(XiXj)E(XkXI) + E(XiXk)E(xjXI) + E(XiXl)E(xjXk). If x is liearly trasfrmed as y=ax+b where A is m x ad b is m x 1 with m $ ad A full rak (s that Cy is sigular), the y is als distributed accrdig t a multivariate Gaussia distributi with ad E(y) = P y = Apx + b E (y - py)(y - py)t] = Cy = ACxA T. Ather useful PDF is the X 2 distributi, which is derived frm the Gaussia distributi. If x is cmpsed f idepedet ad idetically distributed radm variables with Xi f'v N(, 1), i = 1,2,...,, the ~ 2 2 Y = L..JXi f'v X where X~ detes a X 2 radm variable with degrees f freedm. The PDF is give as p(y) = { 2':(t>yt- 1 exp(-!y) fr y 2:: 0 fry<o where I'(u) is the gamma itegral. The mea ad variace f yare E(y) var(y) 2.