R Vectors & Matrices with statistical applicatios x RXX RXY y RYX RYY Why lear matrix algebra? Simple way to express liear combiatios of variables ad geeral solutios of equatios. Liear statistical models (regressio, aova) geeralize to ay # of predictors & resposes. ax ax ax ax 1 1 2 2 3 3 1 x 1 b 1 x b y ˆi x x ŷ X 0 1 1 2 2 uivariate respose Michael Friedly Psychology 6140 1 Strog relatios betwee algebra, geometry & statistical cocepts Ŷ XB multivariate respose Goal: a readig kowledge of matrix expressios to aid i uderstadig statistical cocepts. 2 Brief history of liear algebra Ideas first arose i relatio to solvig systems of equatios i astroomy & geodesy (1700s) Determiig the shape of the earth from measures of latitude ad arc legth (3 eq., 3 ukows) Calculatig the orbits of plaets, e.g., Satur, Jupiter (6 eq., 6 ukows) Pierre-Louis Moreau de Maupertius The ma who flatteed the earth (Portrait from 1739) His crowig glory was a jourey to Laplad, makig measures of the legth of 1 o of latitude, ad showig that they were smaller ear the poles tha at the equator. rc legths measured from Dukirk to Barceloa 3 4
Brief history of liear algebra By ~ 1800, Gauss developed Gaussia elimiatio to solve such problems, ad least squares to deal with fallible measuremets Still required proper otatio & algebra ( m x ) 1848: J.J. Sylvester itroduced matrix (Lati: womb) for array of umbers, with a sigle symbol. 1855: rthur Cayley defied matrix multiplicatio i relatio to systems of equatios 1858: Cayley develops algebra, icludig iverse, -1 Now, there was a geeral otatio for solvig m equatios i ukows! 5 Vectors & matrices matrix is a rectagular array of umbers, with r rows ad c colums. B 12 3 a a 11 12 i1, 2, r 32 15 0 a21 a22 ( aij ), j 1,2, c 23 7 1 a31 a32 1 7 3 b11 b12 b13 2 4 6 b21 b22 b23 Traspose operatio: T = [a ji ], iterchages rows ad colums 1 2 B3 2 7 4 3 6 7 Vectors & matrices Special vectors & matrices vector is just a oe colum matrix Sometimes writte i trasposed (row) form to save space. y 31 6 7 12 y 13 y T 13 6 7 12 y 31 6 7 12 ll of these forms defie y as a 3 x 1 colum vector 8 uit vector: zero vector: 1 j 0 cotrast vectors: 0 0 0 1 1 1 ci 0 c 1 1 1 1 1 c 2 3 1 1 1 Square matrix: x : same # rows/cols 22 2 10 11 9 Symmetric matrix: = T, or a ij =a ji 22 2 10 10 9 B B 33 33 Diagoal matrix: a ij = 0 for i j D 22 3 0 0 1 D 33 9 7 1 3 3 5 1 9 4 9 7 1 7 3 5 1 5 4 4 0 0 0 2 0 0 0 6 9
Special vectors & matrices Operatios o vectors & matrices Idetity matrix: diagoal w/ a ii = 1 I 22 1 0 0 1 Uit matrix: all a ij = 1 I 33 1 1 J 1 1 j j 1 1 32 3 3 Zero matrix: all a ij = 0 0 23 0 0 0 0 0 0 1 0 0 0 1 0 0 0 1 Why: acts like 1 i multiplicatio I = Why: coveiet way to sum vectors & matrices a T j = i Why: acts like 0 i additio + 0 = -B = 0 10 dditio & subtractio: add correspodig elemets. Must have same shape a b 1 1 11 11 12 12 a31 b3 1 ai bi a2b 2 3 2B32 aij bij a21b21 a22 b22 a b 3 3 a b a b a 31 b31 a32 b32 10 2 5 3 15 5 5 1 B 4 6 B 4 4 B 8 10 0 2 Properties: same as for scalars order does t matter Commutative: + B = B + ssociative: + (B + C) = ( + B) + C vector geometry y x 11 Operatios o vectors & matrices Partitioed matrices Scalar multiplicatio: multiply each elemet by a scalar. Def : partitioed matrix has its rows & colums divided ito sub-matrices ka 1 ka1 kai ka c m ca11 ca1 caij cam1 cam.9y y x 1.5x 1 3 1 2 2 4 0 0 3 5 15 2 10 30 3 4 6 8 0 0 I 0 0 0 0 0 1 r12 r13 RI r21 1 r23 r 31 r32 1 Vector geometry 12 11 12 43 7 8 9 21 22 Statistical examples: R 1 2 3 4 5 6 10 11 12 7 8 21 10 11 x RXX RXY y RYX RYY XX Xy yx yy X y X y Same matrix, just with ames for the sub-matrices Makes it easier to express sets of variables 13
Partitioed matrices Vector & matrix multiplicatio dditio ad subtractio is defied for partitioed matrices if all submatrices i correspodig positios are of the same size ad shape 1 2 3 4 5 6 7 8 9 2 2 0 3 4 3 1 0 1 5 5 5 0 2 1 7 10 8 Symbolically, 11 12 B11 B12 11 B11 12 B12 21 22 B21 B22 21 B21 22 B22 14 Note that ier dimesios must match! 15 Vector & matrix multiplicatio Geometry of vector products I a geometric represetatio, the scalar product relates to the agle betwee 2 vectors: ab a b cos 16 Orthogoal vectors (=90) have the property that b = 0 1 ab 1 1 1 2 0 1 xy Correlatio (= cos ) = x y 17
Matrix product Matrix product The matrix product, B, is defied oly if the # of colums of = # of rows of B The, ad B are coformable for multiplicatio lgebraic view: Each elemet, c ij is the vector product of row i of times col j of B 18 19 Matrix product Matrix product: examples rc * B cs = C rs Diagram view a i b j c ij B Vector formula Scalar formula a i b j = c ij b1 j b a a a a b c 2 j i1 i 2 ic ik kj ci j k 1 b cj 20 B 21
Visualizig matrix product Right-mult: liear combiatio of colums multiplyig by B is the liear combiatio of the colums of usig coefficiets from B Right multiplyig by a matrix is just more of the same. Each colum of the result is a differet liear combiatio of the colums of 22 23 Visualizig matrix product Left-mult: liear combiatio of rows Why multiply like this? To express systems of liear equatios: multiplyig by B is the liear combiatio of the rows of B usig coefficiets from From: http://eli.thegreeplace.et/2015/visualizig-matrix-multiplicatio-as-a-liear-combiatio 24 Solutio: x = -1 b whe -1 exists (m=, eq. idepedet) 25
Properties of matrix multiplicatio Properties of matrix multiplicatio 1. ssociative BC ( ) ( BC ) 5. Zero 0 0 rc cs rs 2. Distributive 3. NOT commutative (i geeral) ( B) C CBC B ( C) BC B B 6. Traspose of a product ( B) B T T T ( BZ) Z B T T T T 4. Idetity I I rc cc rr rc 1 2 31 0 1 2 3 4 5 6 0 1 4 5 6 ll of these properties are aalogous to ordiary (scalar) algebra, except for (3) ad (6). Why? 26 27 Matrix powers Matrix powers For a square matrix, ( x ) : 2 3 2 etc, for p Square roots too: If B 2 =, the B is also the square root of, i.e., B = 1/2 e.g., 2 4 0 4 04 0 16 0 0 3 0 30 3 0 9 e.g., 2 1 2 1 21 2 7 10 3 4 3 43 4 15 22 so, 1/2 16 0 4 0 B 0 9 0 3 1/2 I applicatios (e.g., MP II-1), matrix powers provide a simple way to compute paths through a etwork, represeted by (0/1) values i a matrix. 28 The idea of the square root of a matrix was fudametal i the developmet of factor aalysis, where Thurstoe defied factors as R 29
Vectors & matrices i regressio The geeral liear regressio model, y X X X i 0 1 i1 2 i2 p ip i Matrix products i regressio ll calculatios are based o the sums ad sums of squares from the followig matrix products (show for p=1 predictor): has the followig form i terms of vectors ad matrices: or, y 1 x11 x 1 1p 0 1 y 1 x21 x 2 2p 1 2 y 1 x1 x p p y X 1 ( p1) ( p1) 1 1 30 We ca represet these all with partitioed matrices: XX Xy yx yy X y X y 31 Liear combiatios of vectors Give: vectors x 1, x 2, x 3,... (same legth) liear combiatio is a weighted sum of the form Liear combiatios of vectors Simple example: a x1 b x2 c x3 a, b, c: scalars e.g., 3 x 1 + 2 x 2 7x 3 Why: liear models use liear combiatios: yˆ x x x 1 1 2 2 3 3 32 33
Liear idepedece Liear idepedece: example set of vectors, x 1, x 2, x is liearly depedet if: 1. Oe x i ca be expressed as a liear combiatio of the others; or, equivaletly: 2. There are some scalars, a 1, a 2, a, ot all zero, such that 0 a 1 x 1 + a 2 x 2 + + a x = 0 = 0 0 Otherwise, the vectors are liearly idepedet. Why: liear idepedece rak of a matrix, # of degrees of freedom 34 Whe does this arise? You iclude such composite measures Ipsatized scores: divide all by the total Sample size (N) < # of variables (p) Cosequeces: Most aalyses will fail, give errors, etc. 35 Rak of a matrix The idea of rak of a matrix (or set of vectors) is a fudametal idea i matrix algebra ad statistical applicatios. Def: rak( ) r( ) = # of liearly idepedet rows (or colums) of r x c Properties: # liearly idepedet rows = # liearly idepedet colums r( ) rak ever greater tha smaller dimesio r( B) = mi[ r(), r(b) ] rak of product = smaller of separate raks Geometric idea: rak = # of dimesios (of a vector space) Statistical idea: rak = degrees of freedom = # of liearly idepedet variables 36 37
Matrix iverse: -1 Iverse of a umber: I ordiary arithmetic, divisio (iverse of multiplicatio) is essetial for solvig equatios Equally we ca regard this is multiplyig both sides by the iverse of the costat 4 x 1 1 8 8 2 4 4 x 4 x 38 39 Matrix iverse: -1 Iverse of a matrix: Divisio is ot defied for matrices, but most square matrices have a matrix iverse, -1, that plays a similar role i solvig equatios. The iverse of a x matrix,, is defied as a matrix -1 such that its product with gives the idetity matrix: -1 = -1 = I ( x ) Matrix iverse: basic properties If a iverse, -1 exists, it is uique No iverse exists if x = 0 (i.e., r()=0) or, i geeral, if r() < is sigular det() = = 0 Ordiary iverse defied oly for square, o-sigular matrices Ca also defie a geeralized iverse, such that = ad 40 41
Matrix iverse: 2 x 2 Properties of matrix iverse The iverse of a 2 x 2 matrix is easy to calculate: e.g., Note: a b 1 d b 1 d b 1 c d ad bc c a c a 3 2 1 4 4 2 1 4 2 431 ( 2) 1 3 141 3 1 1 1 3 2 1 4 2 1 14 0 I 1 4 14 1 3 14 0 14 No iverse if = ad bc = 0, e.g., 2 3 6 9 42 43 Properties of matrix iverse Determiats I geeral, to show or verify that a matrix K is the iverse of matrix L, show that K L = L K = I 44 45
3 x 3 matrix: Copy first 2 cols to the right Multiply diagoals dd / subtract Determiats Determiats: cofactors Geeral method: expad by cofactors of a give row or colum Mior of a ij :M ij = determiat of submatrix removig the ith row ad jth colum of. Cofactor of a ij :C ij = (-1) (i+j) M ij (sigs alterate) For row i : For col j: For a 12 : M 12 a 12-1 M 12 0-1 (-22+12) = 0 46 47 Determiats: cofactors Determiats: geometry Expad by row 1: M 11 1 8 6 11 2D: det() = area of parallelogram det u a b vdet ad bc c d M 12 M 13 3D: det() = volume D: det() = hyper-volume (What happes if u, v, w are liearly depedet?) Correlatio matrices: I geeral: 0det( ) 1 R p p 49 1 r det 1 r r12 1 12 2 12 Sigular Ucorrelated, R=I 50
2 0 D 0 2 D 4 Geometry: 2 x 2 2 x 2 matrices ca be visualized by drawig their row (or colum) vectors. This illustrates the determiat as the area of the parallelogram 3 1 2 4 10 1 2 B 2 4 B 0 Geometry: Iverse The iverse of a 2 x 2 matrix ca be visualized by drawig its row vectors i the same plot. This shows that: The shape of -1 is a 90 o rotatio of the shape of. -1 is small i the directios where is large; det( -1 )= 1/det() The vector a 2 is at right agles to a 1 ad a 1 is at right agles to a 2 2 1 2 1 1 2 3 1 2 1 1 2 1 1 1 1 1 1 2 1 Diagoal matrix Geeral matrix Sigular matrix 51 52 Matrix fuctios Summary Basic matrix fuctios are provided i base R: matrix(), c(), rbid(), cbid(), t(), %*%, [,] diag(), det(), solve(), crossprod() The matlib package provides some more: Rak: R(), trace: tr(), legth: le() Iverse: iv() May more for liear algebra ad vector diagrams Matrices & vectors: shorthad otatio Matrix: 2-way table; vector: 1-way collectio of #s lgebra: dditio, subtractio: like ordiary arithmetic Multiplicatio: a x = liear combiatio; x = set of them Use: represet a liear model: y = X + Iverse: Matrix divisio Solve liear equatios: x = b x = -1 b std.errors Determiat: size of a square matrix Rak = # liearly idepedet rows, cols, equatios 53 54