Why learn matrix algebra? Vectors & Matrices with statistical applications. Brief history of linear algebra

Transcription

1 R Vectors & Matrices with statistical applicatios x RXX RXY y RYX RYY Why lear matrix algebra? Simple way to express liear combiatios of variables ad geeral solutios of equatios. Liear statistical models (regressio, aova) geeralize to ay # of predictors & resposes. ax ax ax ax x 1 b 1 x b y ˆi x x ŷ X uivariate respose Michael Friedly Psychology Strog relatios betwee algebra, geometry & statistical cocepts Ŷ XB multivariate respose Goal: a readig kowledge of matrix expressios to aid i uderstadig statistical cocepts. 2 Brief history of liear algebra Ideas first arose i relatio to solvig systems of equatios i astroomy & geodesy (1700s) Determiig the shape of the earth from measures of latitude ad arc legth (3 eq., 3 ukows) Calculatig the orbits of plaets, e.g., Satur, Jupiter (6 eq., 6 ukows) Pierre-Louis Moreau de Maupertius The ma who flatteed the earth (Portrait from 1739) His crowig glory was a jourey to Laplad, makig measures of the legth of 1 o of latitude, ad showig that they were smaller ear the poles tha at the equator. rc legths measured from Dukirk to Barceloa 3 4

2 Brief history of liear algebra By ~ 1800, Gauss developed Gaussia elimiatio to solve such problems, ad least squares to deal with fallible measuremets Still required proper otatio & algebra ( m x ) 1848: J.J. Sylvester itroduced matrix (Lati: womb) for array of umbers, with a sigle symbol. 1855: rthur Cayley defied matrix multiplicatio i relatio to systems of equatios 1858: Cayley develops algebra, icludig iverse, -1 Now, there was a geeral otatio for solvig m equatios i ukows! 5 Vectors & matrices matrix is a rectagular array of umbers, with r rows ad c colums. B 12 3 a a i1, 2, r a21 a22 ( aij ), j 1,2, c a31 a b11 b12 b b21 b22 b23 Traspose operatio: T = [a ji ], iterchages rows ad colums 1 2 B Vectors & matrices Special vectors & matrices vector is just a oe colum matrix Sometimes writte i trasposed (row) form to save space. y y 13 y T y ll of these forms defie y as a 3 x 1 colum vector 8 uit vector: zero vector: 1 j 0 cotrast vectors: ci 0 c c Square matrix: x : same # rows/cols Symmetric matrix: = T, or a ij =a ji B B Diagoal matrix: a ij = 0 for i j D D

3 Special vectors & matrices Operatios o vectors & matrices Idetity matrix: diagoal w/ a ii = 1 I Uit matrix: all a ij = 1 I J 1 1 j j Zero matrix: all a ij = Why: acts like 1 i multiplicatio I = Why: coveiet way to sum vectors & matrices a T j = i Why: acts like 0 i additio + 0 = -B = 0 10 dditio & subtractio: add correspodig elemets. Must have same shape a b a31 b3 1 ai bi a2b 2 3 2B32 aij bij a21b21 a22 b22 a b 3 3 a b a b a 31 b31 a32 b B 4 6 B 4 4 B Properties: same as for scalars order does t matter Commutative: + B = B + ssociative: + (B + C) = ( + B) + C vector geometry y x 11 Operatios o vectors & matrices Partitioed matrices Scalar multiplicatio: multiply each elemet by a scalar. Def : partitioed matrix has its rows & colums divided ito sub-matrices ka 1 ka1 kai ka c m ca11 ca1 caij cam1 cam.9y y x 1.5x I r12 r13 RI r21 1 r23 r 31 r32 1 Vector geometry Statistical examples: R x RXX RXY y RYX RYY XX Xy yx yy X y X y Same matrix, just with ames for the sub-matrices Makes it easier to express sets of variables 13

4 Partitioed matrices Vector & matrix multiplicatio dditio ad subtractio is defied for partitioed matrices if all submatrices i correspodig positios are of the same size ad shape Symbolically, B11 B12 11 B11 12 B B21 B22 21 B21 22 B22 14 Note that ier dimesios must match! 15 Vector & matrix multiplicatio Geometry of vector products I a geometric represetatio, the scalar product relates to the agle betwee 2 vectors: ab a b cos 16 Orthogoal vectors (=90) have the property that b = 0 1 ab xy Correlatio (= cos ) = x y 17

5 Matrix product Matrix product The matrix product, B, is defied oly if the # of colums of = # of rows of B The, ad B are coformable for multiplicatio lgebraic view: Each elemet, c ij is the vector product of row i of times col j of B Matrix product Matrix product: examples rc * B cs = C rs Diagram view a i b j c ij B Vector formula Scalar formula a i b j = c ij b1 j b a a a a b c 2 j i1 i 2 ic ik kj ci j k 1 b cj 20 B 21

6 Visualizig matrix product Right-mult: liear combiatio of colums multiplyig by B is the liear combiatio of the colums of usig coefficiets from B Right multiplyig by a matrix is just more of the same. Each colum of the result is a differet liear combiatio of the colums of Visualizig matrix product Left-mult: liear combiatio of rows Why multiply like this? To express systems of liear equatios: multiplyig by B is the liear combiatio of the rows of B usig coefficiets from From: 24 Solutio: x = -1 b whe -1 exists (m=, eq. idepedet) 25

7 Properties of matrix multiplicatio Properties of matrix multiplicatio 1. ssociative BC ( ) ( BC ) 5. Zero 0 0 rc cs rs 2. Distributive 3. NOT commutative (i geeral) ( B) C CBC B ( C) BC B B 6. Traspose of a product ( B) B T T T ( BZ) Z B T T T T 4. Idetity I I rc cc rr rc ll of these properties are aalogous to ordiary (scalar) algebra, except for (3) ad (6). Why? Matrix powers Matrix powers For a square matrix, ( x ) : etc, for p Square roots too: If B 2 =, the B is also the square root of, i.e., B = 1/2 e.g., e.g., so, 1/ B /2 I applicatios (e.g., MP II-1), matrix powers provide a simple way to compute paths through a etwork, represeted by (0/1) values i a matrix. 28 The idea of the square root of a matrix was fudametal i the developmet of factor aalysis, where Thurstoe defied factors as R 29

8 Vectors & matrices i regressio The geeral liear regressio model, y X X X i 0 1 i1 2 i2 p ip i Matrix products i regressio ll calculatios are based o the sums ad sums of squares from the followig matrix products (show for p=1 predictor): has the followig form i terms of vectors ad matrices: or, y 1 x11 x 1 1p 0 1 y 1 x21 x 2 2p 1 2 y 1 x1 x p p y X 1 ( p1) ( p1) We ca represet these all with partitioed matrices: XX Xy yx yy X y X y 31 Liear combiatios of vectors Give: vectors x 1, x 2, x 3,... (same legth) liear combiatio is a weighted sum of the form Liear combiatios of vectors Simple example: a x1 b x2 c x3 a, b, c: scalars e.g., 3 x x 2 7x 3 Why: liear models use liear combiatios: yˆ x x x

9 Liear idepedece Liear idepedece: example set of vectors, x 1, x 2, x is liearly depedet if: 1. Oe x i ca be expressed as a liear combiatio of the others; or, equivaletly: 2. There are some scalars, a 1, a 2, a, ot all zero, such that 0 a 1 x 1 + a 2 x a x = 0 = 0 0 Otherwise, the vectors are liearly idepedet. Why: liear idepedece rak of a matrix, # of degrees of freedom 34 Whe does this arise? You iclude such composite measures Ipsatized scores: divide all by the total Sample size (N) < # of variables (p) Cosequeces: Most aalyses will fail, give errors, etc. 35 Rak of a matrix The idea of rak of a matrix (or set of vectors) is a fudametal idea i matrix algebra ad statistical applicatios. Def: rak( ) r( ) = # of liearly idepedet rows (or colums) of r x c Properties: # liearly idepedet rows = # liearly idepedet colums r( ) rak ever greater tha smaller dimesio r( B) = mi[ r(), r(b) ] rak of product = smaller of separate raks Geometric idea: rak = # of dimesios (of a vector space) Statistical idea: rak = degrees of freedom = # of liearly idepedet variables 36 37

10 Matrix iverse: -1 Iverse of a umber: I ordiary arithmetic, divisio (iverse of multiplicatio) is essetial for solvig equatios Equally we ca regard this is multiplyig both sides by the iverse of the costat 4 x x 4 x Matrix iverse: -1 Iverse of a matrix: Divisio is ot defied for matrices, but most square matrices have a matrix iverse, -1, that plays a similar role i solvig equatios. The iverse of a x matrix,, is defied as a matrix -1 such that its product with gives the idetity matrix: -1 = -1 = I ( x ) Matrix iverse: basic properties If a iverse, -1 exists, it is uique No iverse exists if x = 0 (i.e., r()=0) or, i geeral, if r() < is sigular det() = = 0 Ordiary iverse defied oly for square, o-sigular matrices Ca also defie a geeralized iverse, such that = ad 40 41

11 Matrix iverse: 2 x 2 Properties of matrix iverse The iverse of a 2 x 2 matrix is easy to calculate: e.g., Note: a b 1 d b 1 d b 1 c d ad bc c a c a ( 2) I No iverse if = ad bc = 0, e.g., Properties of matrix iverse Determiats I geeral, to show or verify that a matrix K is the iverse of matrix L, show that K L = L K = I 44 45

12 3 x 3 matrix: Copy first 2 cols to the right Multiply diagoals dd / subtract Determiats Determiats: cofactors Geeral method: expad by cofactors of a give row or colum Mior of a ij :M ij = determiat of submatrix removig the ith row ad jth colum of. Cofactor of a ij :C ij = (-1) (i+j) M ij (sigs alterate) For row i : For col j: For a 12 : M 12 a 12-1 M (-22+12) = Determiats: cofactors Determiats: geometry Expad by row 1: M D: det() = area of parallelogram det u a b vdet ad bc c d M 12 M 13 3D: det() = volume D: det() = hyper-volume (What happes if u, v, w are liearly depedet?) Correlatio matrices: I geeral: 0det( ) 1 R p p 49 1 r det 1 r r Sigular Ucorrelated, R=I 50

13 2 0 D 0 2 D 4 Geometry: 2 x 2 2 x 2 matrices ca be visualized by drawig their row (or colum) vectors. This illustrates the determiat as the area of the parallelogram B 2 4 B 0 Geometry: Iverse The iverse of a 2 x 2 matrix ca be visualized by drawig its row vectors i the same plot. This shows that: The shape of -1 is a 90 o rotatio of the shape of. -1 is small i the directios where is large; det( -1 )= 1/det() The vector a 2 is at right agles to a 1 ad a 1 is at right agles to a Diagoal matrix Geeral matrix Sigular matrix Matrix fuctios Summary Basic matrix fuctios are provided i base R: matrix(), c(), rbid(), cbid(), t(), %*%, [,] diag(), det(), solve(), crossprod() The matlib package provides some more: Rak: R(), trace: tr(), legth: le() Iverse: iv() May more for liear algebra ad vector diagrams Matrices & vectors: shorthad otatio Matrix: 2-way table; vector: 1-way collectio of #s lgebra: dditio, subtractio: like ordiary arithmetic Multiplicatio: a x = liear combiatio; x = set of them Use: represet a liear model: y = X + Iverse: Matrix divisio Solve liear equatios: x = b x = -1 b std.errors Determiat: size of a square matrix Rak = # liearly idepedet rows, cols, equatios 53 54