Matrix Algebra from a Statistician s Perspective BIOS 524/ Scalar multiple: ka

Transcription

1 Matrix Algebra from a Statisticia s Perspective BIOS 524/546. Matrices... Basic Termiology a a A = ( aij ) deotes a m matrix of values. Whe =, this is a am a m colum vector. Whe m= this is a row vector..2. Basic Operatios ka ka.2.. Scalar multiple: ka = ( kaij ). Note that ( ) A = Adeotes kam ka m the egatio of A Additio: A ad B are coformal for additio if they have the same umber of rows ad colums. a + b a + b A+ B= ( aij + bij ). am + bm am + b m Commutative: A+ B= B+ A A+ B+ C = A+ B + C Associative: ( ) ( ) Distributive: k( A+ B) = ka+ k B ad ( ) c+ k A = ca+ ka.2.3. Matrix Multiplicatio: A ad B are coformal for the matrix multiplicatio AB if the umber of colums of A equals the umber of rows of B. AB ab ik kj k = Not commutative: AB BA i geeral. I fact, BA does ot exist if A ad B are ot coformal for this operatio. A BC = AB C Associative: ( ) ( ) Distributive: A( B+ C) = AB+ AC ad ( ) = + A+B C AC BC A ad B commute if AB = BA.2.4. Traspositio: The traspose of a m matrix A is deoted by A, which is the m matrix A ( A ) = A ( a ji ) ( A+ B) = A + B AB = B A ( )

2 Matrix Algebra from a Statisticia s Perspective BIOS 524/ Some Basic Types of Matrices.3.. Square: matrix.3.2. Symmetric: A = A. (also square) d d Diagoal: D = 0 0 d diag (( 2 ) ) ( ) ( ) d d d = D ad diag A = a a a, where A is 22 square Idetity: I = Matrix of oes: J = m ; J = J ; J = j Null matrix: Om = ; O = O Triagular matrices: Upper triagular: a a2 a3 a 0 a22 a23 a2 0 0 a33 a a A lower triagular matrix has the zeros above ad to the right of the diagoal but arbitrary values elsewhere.

3 Matrix Algebra from a Statisticia s Perspective BIOS 524/ Submatrices ad partitioed matrices. 2.. Some Termiology ad Basic Results 2... A submatrix is the result of retaiig certai rows ad colums of the origial matrix (or strikig out certai row ad colums). With IML, use the matrix reductio operators to retai specific rows ad colums. Example: If A is 0 0 ad you wat to create a ew matrix B with oly colums {2, 3, 5} ad rows {, 2, 3} of A the use this code: B=A[{2, 3, 5},{, 2, 3}] Partitioed Matrix A A2 A c A A A A = c where A ij is a m i j matrix ad is called the ij th block. Ar Ar2 Arc Give the blocks, the partitioed matrix ca be costructed i IML by use of the cocateatio operators ( ad // ). A block-diagoal matrix is a partitioed matrix of the form A 02 0 c 02 A22 02c A = with off-diagoal blocks cosistig of ull matrices. I 0r 0r2 Arc IML the BLOCK fuctio may be used to costruct a block-diagoal matrix. Example: A={ 2, 3 4}; A2={5 6 7, 8 9 0}; B=Block(A,A2); This results i B = Sums, ad Products of Partitioed Matrices B B2 B v The matrix A as defied above ad B2 B22 B2v B = are coformal for Bu Bu2 Buv additio, where is a p i q j matrix, if r=u, c=v, p i =m i, ad q j = j for i = ()r ad j = ()c. Thus B ij ( ij ij ) A+ B A + B.

4 Matrix Algebra from a Statisticia s Perspective BIOS 524/ The matrix A ad B as defied above are coformal for multiplicatio (AB) if F F2 F v v =p. The product AB ca be writte as F F F AB = where Fr Fr2 Frv c Fij = AikBkj if Aik ad Bkj are coformal for AikBkj, i = ()r, j = ()v, k = k = ()c, ad c=u. AB + A2B2 AB2 + A2B22 Whe r=c=u=v=2, for example, the AB =. A2B + A22B2 A2B2 + A22B Some Results o the Product of a Matrix ad a Colum Vector Let A = ( a ) be a partitioed matrix where is the i th a 2 a ai colum of A. Let x = ( x x 2 x ) be a vector. The Ax = xkakis a liear k = combiatio of colums of A with coefficiets beig the elemets of x For ay colum vector y ad oull vector x there exists a matrix A such that y=ax For ay two m matrices A ad B, the A=B if ad oly if Ax=Bx for every x (that coforms) Expasio of a Matrix i Terms of Its Rows, Colums, or Elemets Let A = ( r r 2 r m ) be a partitioed matrix where r i is the i th row of A ad let e be the i th colum of the idetity matrix I (a colum vector with a i the i th i row ad 0 s elsewhere). The ) m i= m A = er i iis a expasio of A Let A = ( a be a partitioed matrix where is the i th a 2 a ai colum of A ad let u be the j th j row of the idetity matrix I. The A= au i jis a expasio of A. m A= ij i j i j i= j= a eu where eu is a m matrix with a i the ijth positio ad 0 s elsewhere. j=

5 Matrix Algebra from a Statisticia s Perspective BIOS 524/ Special case: If A=I the m Im = ee i i. 3. Liear depedece ad idepedece. See i-class otes. 3.. Defiitios 3.2. Some Basic Results 4. Liear spaces: Row ad colum spaces. 4.. Some Defiitios, Notatio, ad Basic Relatioships ad Properties i= 4... Liear Spaces (Vector Space): V= o-empty set of m matrices. For every A V ad B V we have 4... A+ B V : closed uder additio ka V : closed uder scalar multiplicatio There exists 0 V such that A+ 0= A Colum space of a matrix: Liear space (vector space) spaed by the colums of a matrix A deoted C A. ( ) Row space of a matrix: Liear space spaed by the rows of a matrix A deoted R ( A) Notes: A = A R A = C A C( ) R( ) ad ( ) ( ) ( ) C I = R : The liear space of all -tuples or -dimesio Euclidea space. m m R = R R deotes the liear space of all m real-valued matrices Subspaces: The set U V is called a subspace of V if U is a liear space. See pages Bases See i-class otes: If S V is a set of k liearly idepedet matrices that spas V. The S is a basis for V Note Theorem o page Let S = { A, A2,, Ak} be a basis for V. The for ay A V there is a uique set of scalars { x x x },,, k 2 such that k A = xia i Dimesio of a matrix. See pages Rak of a Matrix. See pages Some Basic Results o Partitioed Matrices ad o Sums of Matrices 4.6. Fidig a set of liearly idepedet colum vectors for a give m matrix A Use row operators to covert A to reduced row echelo form Colums with a i oe positio ad 0 elsewhere idicates that the correspodig colum of A is a basis vector Otherwise, colums of the reduced row echelo form correspod to colums of A that ca be writte as liear combiatios of the basis vectors. The elemets of these colums are the coefficiets of the liear combiatio. 5. Trace of a (Square) Matrix. 5.. Defiitio ad Basic Properties i=

6 Matrix Algebra from a Statisticia s Perspective BIOS 524/ Suppose A = ( a ij ) tr( A) = a i= Liear fuctio tr( ka) = ktr( A) ii tr( A+ B) = tr( A) + tr( B) 5.2. Trace of a Product is a (square) matrix. The trace of A is defied to be, the sum of the diagoal elemets of A. Suppose A is m ad B is m. The tr( AB) ab. m = i= j= tr( AB) = tr( B A ) = tr( BA) 5.3. Some Equivalet Coditios 6. Geometrical cosideratios. 6.. Defiitios: Norm, Distace, Agle, Ier Product, ad Orthogoality 6... Ier Product deoted, : A liear fuctioal that maps a pair of vectors to a real umber Suppose that uvz,, V ad that k is a real umber. The a ier product of u ad v, u, v, has the followig properties. u, v = v, u ku, v = k u, v u+ v, z = u, z + v, z uu, 0with uu, = 0 if ad oly if u= Examples of a ier product (ca you verify the properties above?) Suppose V= R. The usual ier product is defied as u, v = uv. 3 Suppose V= R. Oe possible ier product is defied as u, v = uv + 2uv 2 2+ uv 3 3. Suppose V is the liear space of all m matrices. Let AB, V. Oe possible ier product is defied as AB, = tr ( AB. ) Norm, a measure of distace The orm of a vector with respect to a ier product is defied ad deoted by u = u, u. This is iterpreted is the legth of the vector u The distace betwee two vectors u ad v, with respect to a ier product, is the orm of u v: u v, The agle, θ, betwee two vectors u ad v, is defied by cosθ = u v u v. ij ji

7 Matrix Algebra from a Statisticia s Perspective BIOS 524/ Two vectors u ad v, are said to be orthogoal, with respect to a ier product, if cosθ = 0. This is equivalet to u, v = 0. Orthogoality (perpedicularity) betwee vectors is deoted by u v Orthogoal ad Orthoormal Sets Gram-Schmidt orthogoalizatio method (see hadout ad class otes) 6.3. Schwarz Iequality u, v u v. This is useful i some proofs Orthoormal Bases The vectors of a orthoormal basis may be viewed as coordiates. E.g., if { v, v2,, v} is a orthoormal set of vectors that spas a liear space V, the ay y V may be writte as y = aivi where i= y 2 2 = ai i=. We may represet y with the -tuple ( a a 2 a ) with respect to the coordiate system { v v v }, 2,, QR-decompositio of a full rak matrix A. See sectio 6.4(c) o pages This decompositio may be attaied by way of the Gram-Schmidt orthogoalizatio. 7. Liear systems: Cosistecy ad Compatibility 7.. Some Basic Termiology 7... System of liear equatios: Ax = b, where A is m, x is, ad b is m p liear systems: AX = B, where A is m, X is p, ad B is m p The liear system AX = 0 is said to be homogeeous Cosistecy = is a cosistet system if b C A. That is, b is a liear combiatio of Ax b ( ) the colums of A with the coefficiets beig the elemets of x. I other words, cosistecy meas there exists at least oe solutio to the system of equatios AX = B is a cosistet system if ad oly if oe of the followig holds (they are all equivalet): C B C A ( ) ( ) Every colum of B is cotaied i C ( A) C( AB, ) = C ( A) rak ( AB, ) = rak ( A ) 7.3. Compatibility AX = B is said to be compatible if ad oly if for every vector k that is orthogoal to each colum of A is also orthogoal to each colum of B. That is, for every k such that ka = 0 we have that kb = The liear system AX = B is compatible if ad oly if it is cosistet. A proof of this theorem is give o pages 73-74, but we will postpoe provig this util we develop more tools. With these tools the proof will be evidet.

8 Matrix Algebra from a Statisticia s Perspective BIOS 524/ The liear system AAX = AB i X is cosistet. * * Proof: It suffices to show that the liear system AX = B, where X * = AX ad * B = AB, is compatible. This is trivial sice if ka = 0 the kaa = 0 ad * kb = kab = Some results TAA = TA R AAT = R AT C( ) C ( ) ad ( ) ( ) Proof: See page 75. rak = rak ( TAA) ( TA ) ad rak ( ) = rak ( ) AAT AT. This follows directly from the previous result Lettig T=I, we have C( AA ) = C ( A ) ad rak ( AA ) = rak ( A) It follows from the last result that if A is full rak, the AA is osigular (a matrix with both full row-rak ad full colum-rak). 8. Iverse matrices. 8.. Some Defiitios ad Basic Results 8.2. Properties of Iverse Matrices 8.3. Premultiplicatio or Postmultiplicatio by a Matrix of Full Colum or Row Rak 8.4. Orthogoal Matrices 8.5. Some Basic Results o the Raks ad Iverses of Partitioed Matrices 9. Geeralized iverses. 9.. Defiitio, Existece, ad a Coectio to the Solutio of Liear Systems 9.2. Some Alterative Characterizatios 9.3. Some Elemetary Properties 9.4. Ivariace to the Choice of a Geeralized Iverse 9.5. A Necessary ad Sufficiet Coditio for the Cosistecy of a Liear System 9.6. Some Results o the Raks ad Geeralized Iverses of Partitioed Matrices 9.7. Extesio of Some Results o Systems of the Form AX = B 0. Idempotet matrics. 0.. Defiitio ad Some Basic Properties 0.2. Some Basic Results. Liear systems... Some Termiology, Notatio, ad Basic Results.2. Geeral Form of a Solutio.3. Number of Solutios.4. A Basic Result o Null Spaces.5. A Alterative Expressio for the Geeral Form of a Solutio.6. Equivalet Liear Systems.7. Null ad Colum Spaces of Idempotet Matrices.8. Liear Systems With Nosigular Triagular or Block-Triagular Coefficiet Matrices.9. A Computatioal Approach.0. Liear Combiatios of the Ukows.. Absorptio.2. Extesios to Systems of the Form AXC = B 2. Projectios ad projectio matrices. 2.. Some Geeral Results, Termiology, ad Notatio

9 Matrix Algebra from a Statisticia s Perspective BIOS 524/ Projectio of a Colum Vector 2.3. Projectio Matrices 2.4. Least Squares Problem 2.5. Orthogoal Complemets 3. Determiats. 4. Liear, biliear, ad quadratic forms. 4.. Some Termiology ad Basic Results 4.2. Noegative Defiite Quadratic Forms ad Matrices 4.3. Decompositio of Symmetric ad Symmetric Noegative Defiite Matrices 4.4. Geeralized Iverses of Symmetric Noegative Defiite Matrices 4.5. LDU, UDU, ad Cholesky Decompositios 4.6. Skew-Symmetric Matrices 4.7. Trace of a Noegative Defiite Matrix 4.8. Partitioed Noegative Defiite Matrixes 4.9. Some Results o Determiats 4.0. Geometrical Cosideratios 4.. Some Results o Raks ad Row ad Colum Subspaces ad o Liear Systems 4.2. Projectios, Projectio Matrices, ad Orthogoal Complemets 5. Matrix differetiatio. 6. Kroecker products ad the Vec ad Vech operators. 7. Itersectios ad sums of subspaces. 8. Sums ad differeces of matrics. 9. Miimizatio of a secod-degree polyomial (i variables) subject to liear costraits. 20. The Moore-Perose iverse. 2. Eigevalues ad eigevectors. 22. Liear trasformatios.