Introduction to Matrix Algebra

Transcription

1 Introduction to Clayton Webb CRMDA University of Kansas August 2015

2 Motivation Outline Where we are headed Introduction Motivation We learned how to calculate correlation and regression coefficients in introductory statistics. - ˆρ x,y = Cov(x,y) σ x σ y - ˆβ1 = ( (Y i Ȳ )(X 1i X 1 ))( (X 2i X 2 ) 2 ) (( (Y i Ȳ )(X 2i X 2 )))( (X 1i X 1 )(X 2i X 2 )) ( (X 1i X 1 ) 2 )( (X 2i X 2 ) 2 ) ( (X 1i X 1 )(X 3i X 3 )) 2 When we did these toy examples we had small samples - 20,30,etc. In applied analysis we tend to have larger N. Trying to calculate regression and correlation coefficients this way becomes intractable.

3 Motivation Outline Where we are headed Introduction Motivation Linear algebra (matrix algebra) allows for algebraic manipulation on a larger scale. Data analysis requires a basic understanding of matrix algebra. - Many important texts and articles are written in matrix notation. - What s going on under the hood? - Linear algebra is necessary for statistics. OLS MLE Time Series Our data are already shaped like large matrices. We might as well learn how to use them.

4 Motivation Outline Where we are headed Introduction Outline 1 Defintions 2 - Addition - Subtraction - - Division () 3 - Correlation - Regression - R

5 Motivation Outline Where we are headed Introduction Where we are headed Correlation R = S 1 VS 1 Regression β = (X X ) 1 X Y

6 Basics Special Matrices Basics Matrix A matrix is a rectangular array of numbers, parameters, variables, or other values. a b c d e f g h i j k l a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33

7 Basics Special Matrices Basics Matrix Elements - the individual values in a matrix. Dimensions - The dimensions of a matrix are defined by the number of rows (r) and columns (c). - Rows (r) - the values in the horizontal lines. - Columns (c) - the values in the vertical lines. - A r c matrix has r rows and c columns. - Unless stated otherwise, rows are always given before columns when discussing dimensions. Matrices are usually represented in texts with capital letters.

8 Basics Special Matrices Basics Matrix A double subscript refers to a particular value in a matrix. The row value is given then the column value is given. a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 a 12 Second value in row one a 22 Second value in row two a 31 First value in row three

9 Basics Special Matrices Basics Vector Vectors are special matrices that contain only one row or column. Row vector - r 1 matrix. Column vector - 1 c matrix. r = [ b 1 b 2 b 3 ] c = Vectors are usually represented in texts with lower case letters so readers can differentiate between matrices and vectors.

10 Basics Special Matrices Basics Scalar A scalar is a single number. 1, 2, and 3 are all scalars. Scalars are not 1 1 matrices, they are just numbers. The difference is important because the mathematical operations that can be performed on matrices and vectors are different from the operations that can be performed on scalars. For example, there is a difference between scalar and matrix multiplication. - Scalar multiplication - a matrix or vector is multiplied by a scalar to produce a new matrix or vector. - Matrix multiplication - a matrix or vector is multiplied by another matrix or vector to produce a new matrix or vector.

11 Basics Special Matrices Special Matrices Square matrix - a matrix with the same number of rows and columns (r=c). Sometimes referred to as n n matrices. Triangular matrix - a special kind of square matrix containing several zero elements. - Upper triangular - a square matrix where all the elements above the main diagonal are zero. - Lower tirangular - a square matrix where all the elements below the main diagonal are zero. - A diagonal matrix is upper and lower triangular.

12 Basics Special Matrices Special Matrices Diagonal matrix - a matrix that only contains elements on the main diagonal. All off diagonal elements are 0. Identity matrix - a matrix that only contains ones on the main diagonal. All off diagonal elements are 0 Symmetrical matrix - a square matrix that is equal to its transpose. - Transpose - a matrix that changes rows to columns and columns to rows. - Transpose A is A or A T

13 Basics Special Matrices Special Matrices Transpose A = a b c p q r u v w A = a p u b q v c r w

14 Conformability Conformable matrix - a matrix is conformable if its dimensions are suitable for defining some operation. The conformability of a matrix to an operation depends on the operation. - Addition and subtraction - A and B must have the same dimensions. - - A must be square. - - c of the lead (first) matrix must equal r in the lag (second) matrix. 3 3 and yes 3 2 and yes 3 3 and no

15 Addition (subtraction) of matrices is simply adding (subtracting) the corresponding elements of two matrices. These operations are only possible if the matrices have the same dimensions. That is, the matrices must have the same number of rows and columns. The matrix of sums (differences) will have the same dimensions as the original matrix. A = B =

16 Addition = A + B = C =

17 Subtraction A B = C = =

18 In linear algerbra, there are several different types of multiplication. We will discuss three: - Matrix multiplication - The multiplication of two matrices. - Scalar multiplication - The multiplication of all the elements of a matrix by a scalar. - Vector multiplication - The multiplication of a row vector by a column vector.

19 Order Order is very important for matrix multiplication. Changing the order can give you completely different answers. AB BA You can also get different shapes of matrices. There is an easy rule to keep track of what answer you will get. 1 Write the dimensions of the lead and lag matrices. 2 If the inner values match the matrix is conformable. 3 The outer values tell you the rows and columns for the solution. This is also an easy way to check if you are setting up your matrices correctly.

20 Scalar Scalar multiplication - multiplying a matrix by a scalar. s > 1 scale up. s < 1 scale down. Order does not matter = =

21 Scalar = =

22 Vector Vector multiplication - multiplication of two vectors. Order matters and 3 1 produces and 1 3 produces 3 3 [ ] 1 4 = = = 60 3

23 Vector 1 4 [ ] = =

24 Matrix There are several types of matrix multiplication, but people typically use matrix multiplication to refer to a specfic operation. Order matters. You can multiply matrices by vectors. Square matrices have special features. a b c p q r u v w α β γ λ µ ν ρ σ τ = aα + bλ + cρ aβ + bµ + cσ aγ + bν + cτ pα + qλ + rρ pβ + qµ + rσ pγ + qν + rτ uα + vλ + wρ uβ + vµ + wσ uγ + vν + wτ

25 Matrix α β γ λ µ ν ρ σ τ a b c p q r u v w = αa + βp + γu αb + βq + γv αc + βr + γw λa + µp + νu λb + µq + νv λc + µr + νw ρa + σp + τu ρb + σq + τv ρc + σr + τw

26 Matrix (1) + 6(4) + 9(3) 9(1) + 6(3) + 9(2) 9(2) + 6(3) + 9(4) = 8(1) + 7(4) + 7(3) 8(1) + 7(3) + 7(2) 8(2) + 7(3) + 7(4) 5(1) + 8(4) + 8(3) 5(1) + 8(3) + 8(2) 5(2) + 8(3) + 8(4) = =

27 Matrix (9) + 1(8) + 2(5) 1(6) + 1(7) + 2(8) 1(9) + 1(7) + 2(8) = 4(9) + 3(8) + 3(5) 4(6) + 3(7) + 3(8) 4(9) + 3(7) + 3(8) 3(9) + 2(8) + 4(5) 3(6) + 2(7) + 4(8) 3(9) + 2(7) + 4(8) = =

28 Matrix [ ] [ ] 9(9) + 6(8) + 9(5) 9(6) + 6(7) + 9(8) = 8(9) + 7(8) + 7(5) 8(6) + 7(7) + 7(8) [ ] [ ] = =

29 Matrix 1 1 [ ] (9) + 1(8) 1(6) + 1(7) 1(9) + 1(7) = 4(9) + 3(8) 4(6) + 3(7) 4(9) + 3(7) 3(9) + 2(8) 3(6) + 2(7) 3(9) + 2(7) = =

30 The invere of a matrix A is A 1 A matix only has an inverse if it is square and non-singular. Square matrix - a matrix with the same number of rows (r) and columns (c). Singular matrix - a matrix with linear dependence between at least two rows or columns. Non-singular matrix - a matrix without linear dependence. A matrix with a non-zero determinant. Later we will see how to use determinants to test for linear dependence.

31 Multiplying a matrix by its inverse reduces it to an identity matrix. AA 1 = I = A 1 A This condition can be used to test whether or not you have found the inverse of a matrix. The inverse of a matrix in linear algebra perfroms the same function as a reciprocal in regular algerbra = 1

32 A 1 = 1 adj(a) adj(a) = A A A 1 Inverse A Determinant adj(a) Adjoint The inverse of a matrix is the adjoint matrix divided by the determinant.

33 Determinant A determinant is a useful quantity because it is used to calculate the inverse of a matrix. The logic of calculating a determinant is easiest to understand with a simple 2 2 example. Consider the following matrix. [ ] a b A = p q The equation for the determinant is A = a q b p

34 Determinant This formula works for a 2 2 matrix but will not for larger matrices. For larger matrices we will need to use the Laplace expansion. This process requires minors and cofactors. Minor - the determinant of a submatrix formed by deleting the ith row and the jth column of a matrix. Cofactor - a minor with a specific sign.

35 Determinant - Minor a b c p q r u v w M a = q v r w, M b = p u r w, M c = p u q v, M p = b v c w, M v = a p c r, M w = a p b q,

36 Determinant - Minor To calculate a determinant using minors one chooses a column or row, constructs the minors, and calculates the determinants of the submatrices. M a = q r v w, M b = p r u w, M c = p q u v, A = a[q(w) r(v)] b[p(w) r(u)] + c[p(v) q(u)]

37 Determinant - Minor M 8 = A = , M 3 = , M 2 = , A = 8[4(3) 7(1)] 3[6(3) 7(5)] + 2[6(1) 4(5)] = 8[5] 3[ 17] + 2[ 14] = = 63

38 Determinant - Cofactor A cofactor is a minor with a perscribed sign. The rule for the sign is C rc = ( 1) r+c M rc r is the row number. c is the column number M rc is the minor for the value in row (r) and column (c). - If r + c is even then the cofactor is positive. - If r + c is odd then the cofactor is negative. The cofactor is a more general form of a minor.

39 Determinant - Cofactor Now we can specify the formula for the determinant without worrying about the sign for each minor. A = a b c p q r u v w The cofactors for the first row are C a = q r v w, C b = p u r w, C c = p u q v,

40 Determinant - Cofactor Using the cofactors, the determinant can be calculated A = a C a + b C b + c C c This is the same formula as before, but we don t have to keep track of the sign if we use the cofactor rule. This general form is the basis for the Laplace expansion. We can extend the formula to a 4 4 square matrix. A = a C a + b C b + c C c + d C d

41 Determinant - Laplace Expansion A 5 5 square matrix. And so on. A = a C a + b C b + c C c + d C d + e C e We just need to remember to change the signs for the cofactors based on the rule. Assuming we are using the first row for the 5 5 the minor form is A = a M a b M b + c M c d M d + e M e

42 Determinant - Laplace Expansion The expansion becomes more cumbersome to do by hand as you add additional rows and columns. You have to find the determinants for each of the 4 4 cofactors to find the determinant for a 5 5. For each 4 4 you have to find all the determinants for each of the 3 3 cofactors for these matrices. For each of the 3 3 cofactors you have to find the determinants for all the 2 2 cofactors. The general form is useful, but we would not do this without a computer.

43 Determinant - One can use the properties of matrices and determinants to make determinants easier to calculate. 1 Adding or subtracting a nonzero multiple of one row (or column) from another row (or column) will have no effect on the determinant. 2 Interchanging any two rows or columns of a matrix will change the sign, but not the absolute value, of the determinant. 3 Multiplying the elements of any row or column by a constant will cause the determinant to be multiplied by the constant.

44 Determinant - 4 The determinant of a triangular matrix is equal to the product of the elements on the principal diagonal. 5 The determinant of a matrix equals the determinant of its transpose: A = A. 6 If all the elements of any column or row are zero, the determinant is zero. 7 If two rows or columns are identitical or proportional they are linearly dependent. This means the determinant is zero.

45 Determinant - The determinant can be used as a test for linear dependence. In practice, our columns will be our variables and our rows will be our observations. If we need the inverse to calculate quantities of interest, having rows or columns that are equal or proportional will create problems. - Rank (ρ) - the maxium number of linearly indpendent rows or columns in the matrix. - In a square matrix c = r. So we can say the matrix is n n. ρ(a) = n, A is non-singular. No linear dependence. ρ(a) < n, A is singular and there is linear dependence.

46 Adjoint We have calculated the determinant. To calculate the inverse we need to calculate the adjoint matrix. Adjoint matrix (AdjA)- the transpose of a cofactor matrix. Cofactor matrix (C A ) - a matrix in which every element of the matrix is replaced with its cofactor. a b c A = p q r u v w, C A = C a C b C c C p C q C r C u C v C w, AdjA = C a C p C u C b C q C v C c C r C w

47 Adjoint Steps for calculating an adjoint matrix 1 Construct the cofactor matrix. 2 Calculate the determinant for each sub-cofactor matrix to calculate the cofactor matrix. 3 Find the transpose of the cofactor matrix. A =

48 Adjoint C A =

49 Adjoint C A = 4(3) 7(1) 6(3) 7(5) 6(1) 4(5) 3(3) 2(1) 8(3) 2(5) 8(1) 3(5) 3(7) 2(4) 8(7) 2(6) 8(4) 3(6) C A =

50 Adjoint C A = C A = , AdjA = C A =

51 Inverse Now that we have the determinant and the adjoint matrix we can calculate the inverse matrix. A 1 = 1 A AdjA The inverse matrix for the example is A 1 = =

52 Inverse We can check to see if we calculated the inverse correctly by multiplying the original matrix by the inverse. AA 1 = AA 1 = I = A 1 A = Note: Your results will rarely be this clean. You can check your answers in R and you should be able to get close.

53 Matrix algebra follows slightly different rules than regular algebra. Understanding the similarities and differences is important for understanding how you can manipulate matrices. Commutative law Associative law Distributive law

54 Commutative Law The commutative law says you can swap numbers around and still get the same answer when you add or multiply. a + b = b + a a b = b a Matrix addition is commutative. A + B = B + A Matrix multiplication is not commuatative. A B B A

55 Associative Law The associative law says that it doesn t matter how you group numbers. Matrix addition is associative. (a + b) + c = a + (b + c) (a b) c = a (b c) (A + B) + C = A + (B + C) Matrix multiplication is associative. (A B) C = A (B C)

56 Distributive Law The distributive law says that multiplying a number by a group of numbers added together is the same as doing each multiplication separately. a (b + c) = a b + a c The distributive law also applies to matrices. A (B + C) = A B + A C (B + C) A = B A + C A

57 Summary The order of multiplication matters with matrices. Pre- and Post- multiplication can give you different answers. - Scalar multiplication is commutative. - ka = Ak You can move terms for addition, subtraction, and multiplication because terms are associative, but you have to be careful about the rules of conformability. You can distribute terms with matrices.

58 Special Cases Identity matrices are commutative, associative, and distributive for multiplication. AIB = IAB = ABI = AB Transpose doesn t distribute how you might expect. (AB) = B A Exponential operator only works with square matrices. The inverse of a transpose. A 4 = AAAA (A ) 1 = (A 1 )

59 Correlation Regression Application Correlation 1 Organize your variables into columns of a matrix. 2 Generate a vector of column means. 3 Center your data matrix by subtracting the column mean from each observation. 4 Compute a cross product matrix by multiplying the centered matrix by its inverse. 5 Divide the cross product matrix by the number of observations (or n 1) to produce the variance covariance matrix. - Variances - diagonal - Covariances - offdiagonal

60 Correlation Regression Application Correlation 6 Create a diagonal matrix containing only the variances. 7 Create a diagonal matrix of standard deviations by taking the square root of this matrix. 8 Pre and post multiply the variance covariance matrix by the inverse of the standard deviation matrix. 9 The correlation coefficients are on the off diagonal. The diagonal should be ones because each variable is perfectly correlated with itself.

61 Correlation Regression Application Regression 1 Calculate the regression coefficients with the following formula. (X X ) 1 X Y = β 2 Estimate the predicted Y using the coefficients. 3 Calculate the errors using Y Ŷ

62 Correlation Regression Application Regression 4 Calculate the sum of the squared errors 5 Calculate the variance covariance matrix for the errors. 6 Pull the variances from the diagonal. 7 Calculate the standard errors for the coefficients.

63 Correlation Regression