Unit 3: Matrices Juan Luis Melero and Eduardo Eyras September 2018 1
Contents 1 Matrices and operations 4 1.1 Definition of a matrix....................... 4 1.2 Addition and subtraction of matrices.............. 4 1.3 Multiplication by a number................... 5 1.4 Multiplication of matrices.................... 5 1.4.1 Triangulation of matrices (Gaussian elimination).... 7 1.5 Special matrices.......................... 7 1.5.1 Unit matrix........................ 7 1.5.2 Zero matrix........................ 8 1.5.3 Triangular matrix..................... 8 1.5.4 Diagonal matrix...................... 8 1.6 Matrix transposition....................... 8 2 Determinant of a matrix 9 2.1 Definition and calculus of the determinant........... 9 2.2 Properties of the determinant.................. 11 2.2.1 Determinant of a multiplication of matrices...... 11 2.2.2 Determinant of the inverse of a matrix......... 12 2.2.3 Determinant of the transposed matrix......... 12 2.2.4 Determinant of a matrix multiplied by a number.... 12 2.2.5 Determinant of triangular and diagonal matrices.... 12 2.2.6 Changing rows and columns............... 13 2.2.7 Multiplying a row or a column by a scalar....... 13 2.2.8 Determinant when we apply linear combinations.... 13 2.2.9 Determinant with linearly dependent rows or columns. 14 3 Rank of a matrix 15 3.1 Definition of the rank of a matrix................ 15 3.2 Calculation of the rank of a matrix............... 16 4 Inverse of a matrix 17 4.1 Definition of inverse matrix................... 17 4.2 Properties of the inverse of a matrix............... 17 4.3 Calculation of the inverse of a matrix.............. 18 5 Exercises 21 2
6 R practical 23 6.1 Addition and subtraction of matrices.............. 23 6.2 Multiplication of a matrix by a number............. 23 6.3 Multiplication of matrices.................... 24 6.4 Matrix transposition....................... 25 6.5 Determinant of a matrix..................... 25 6.6 Rank of a matrix......................... 26 6.7 Inverse of a matrix........................ 26 3
1 Matrices and operations 1.1 Definition of a matrix A matrix is a set of elements organized into rows and columns. All rows have the same length and all columns have the same length. Figure 1: Distribution of rows and columns in a matrix The dimension of a matrix is the number of rows and columns that it has. It is indicated as rows columns. a b c a a b d e f b c d g h i c e f 3 3 3 1 1.2 Addition and subtraction of matrices To add or subtract two matrices you simply add or subtract every element. ( ) ( ) ( ) a b e f a + e b + f + = c d g h c + g d + h ( ) ( ) ( ) a b e f a e b f = c d g h c g d h Notice that to add or subtract two or more matrices they must have the same dimensions. 3 2 4
1.3 Multiplication by a number To multiply a matrix by a number you simply multiply each element of the matrix by the number. ( ) ( ) a b αa αb α = c d αc αd Given the addition of matrices and the multiplication by real numbers, the set of all matrices of a given dimension m n, usually denoted as M m n (R), is a vector space. 1.4 Multiplication of matrices Not all matrices can be multiplied by another matrix. It is possible to multiply two matrices if the number of columns of the first one is the same as the number of rows of the second one. A M m k (R), B M k n (R) L = AB M m n (R) To multiply two matrices, we multiply the elements of each row in the first matrix by the elements of each row in the second matrix: For example: l ij = a i1 b 1j + a i2 b 2j + + a ik b kj = k a ir b rj r=1 Figure 2: Multiplication of two matrices in M 3 3 (R). In particular, we show the elements involved in the calculation of the element l 12. l 12 = a 11 b 12 + a 12 b 22 + a 13 b 32 5
Examples of matrix multiplications: [ ] [ ] c a b = ac + bd d [ ] [ ] [ ] a b e ae + bf = c d f ce + df [ ] [ ] [ ] a b e f ae + bg af + bh = c d g h ce + dg cf + dh Matrix multiplication is NOT commutative in general. This can be intuitively seen from the fact that we can multiply two matrices of different dimensions as long as the number columns of the first matrix and the number of rows of the second one coincide. However, this does not ensure that we can invert the order and maintain the same condition: A M m k (R), B M k n (R) L = AB M m n (R) L = BA = B k n (R)A m k (R) Cannot be multiplied Moreover, even for square matrices the multiplication is not commutative: ( ) ( ) a b e f = c d g h ( ) ae + bg......... ( ) ( ) e f a b = g h c d ( ea + fc )......... Matrix multiplication is associative and ( directionally ) distributive with respect to the matrix addition: A(BC) = (AB)C A(B + C) = AB + AC (A + B)C = AC + BC 6
1.4.1 Triangulation of matrices (Gaussian elimination) A triangular matrix is a matrix whose elements above or below the principal diagonal are all 0 (see section 1.5.3. A triangular matrix has some particular properties that are useful, as will be discussed in the next sections, and will also help solving linear equations. Additionally, all the properties of a matrix (the rank, the determinant, etc.) do not change if we exchange a row or a column by a linear combination of that particular row or column with the other rows or columns, we change the order of the rows or we multiply or divide the whole row by a number. Accordingly, we will apply linear combinations to generate new rows or columns with 0 in the desired place. In general: Consider the matrix M m n (R): Let s show it with an example: a 11... a 1n..... a m1... a mn Consider the matrix M 3 4 (R): 3 2 1 1 5 3 4 2 1 1 1 1 We triangulate the matrix with the following operations: a 11... a 1n..... 0... a mn 3 2 1 1 R 3 R 1 1 1 1 1 5 3 4 2 R 1 R 2 3 2 1 1 R 2 R 2 3R 1 1 1 1 1 R 2 R 3 5 3 4 2 R 3 R 3 5R 1 1 1 1 1 1 1 1 1 0 1 4 2 0 1 4 2 0 2 9 3 R 3 R 3 2R 2 0 0 1 1 Now the matrix is triangular. 1.5 Special matrices 1.5.1 Unit matrix The unit matrix I has in its main diagonal 1 and the rest of elements are 0. It is the neutral element for multiplication of matrices: 7
I = 1.5.2 Zero matrix ( ) 1 0 M 0 1 2x2 (R), A M 2x2 (R), AI = IA = A The zero matrix O is a matrix whose elements are all 0. It is the neutral element for the sum of matrices: ( ) 0 0 O = M 0 0 2x2 (R), A M 2x2 (R), A + O = A 1.5.3 Triangular matrix A triangular matrix is a matrix in which the elements above or below the main diagonal are all zeros. For example: a 0 0 a b c A = b c 0 B = 0 d e d e f 0 0 f 1.5.4 Diagonal matrix A diagonal matrix has all elements that are not in the main diagonal equal to zero: a 0 0 A = 0 b 0 0 0 c 1.6 Matrix transposition Transposition of matrices consists in changing the rows by the columns. Accordingly, rows become columns and columns become rows. If A is m n, the transpose of A (A T ) is n m. A M m n (R) A T M n m (R) ( A T ) T = A 8
Figure 3: Matrix transposition of 3x3 matrices and 3x2 matrices In some cases, the transpose is the same as the original matrix. Then, we say that the matrix is symmetric. If A M n n (R), A T = A = A is symmetric [a ij ] = [a ji ] For example: a 11 a b A = a a 22 c b c a 33 2 Determinant of a matrix 2.1 Definition and calculus of the determinant The determinant of a matrix is a real number that you can calculate from a matrix, and it is only defined for square matrices. The way to calculate the determinant depends on the dimension of the matrix. For matrices 1 1: For matrices 2 2: For matrices 3 3 A = ( a ), det(a) = a ( ) a b A =, det(a) = ad bc c d a b c A = d e f g h i 9
The determinant is defined as det(a) = a 11 a 22 a 33 + a 12 a 23 a 31 + a 13 a 21 a 32 a 13 a 22 a 31 a 11 a 23 a 32 a 12 a 21 a 33 which can be derived from either the Rule of Sarrus (see Figure 4) or the cofactors method (see below). Figure 4: Graphically view of Role of Sarrus to calculate determinants of 3 3 matrices. Each square represents a multiplication. The arrowhead represents the numbers you have to multiply. Blue arrows are the multiplications you add, red arrows are the multiplications you subtract. The dashed grey line represents the diagonals of the matrix. To calculate the determinant for matrices of dimensions 4 4 and higher we use the definitions of Minor and Cofactor of a matrix element. M ij is called the Minor of element a ij or the ij-th minor, and it is defined as the determinant of the matrix obtained by deleting the i-th row and the j-th column. C ij is called the Cofactor of element a ij or the ij-th cofactor and it is defined as C ij = ( 1) i+j M ij The cofactor matrix is the matrix generated by the cofactors: C 11... C 1n C =... C m1... C mn With these definitions, we can calculate the determinant as the sum of entries in a row or in a column each multiplied by their corresponding cofactor. 10
Using a fixed row i: det(a) = n a ij Cij = a i1 C i1 + a i2 Ci2 + + a in C in j=1 Using a fixed column j: det(a) = n a ij C ij = a 1j C 1j + a 2j C 2j + + a nj C nj i=1 For instance, using row 1 (and denoting the determinant with vertical bars): a b c A = d e f g h i a b c det(a) = d e f g h i = ( 1)1+1 a e f h i +( 1)1+2 b d f g i +( 1)1+3 c d e g h = = a e h f i b d g f i + c d g e h = aei + bfg + cdh ceg bdi afh 2.2 Properties of the determinant 2.2.1 Determinant of a multiplication of matrices The determinant of a multiplication of a matrix is the multiplication of the determinants. A, B, M n n (R) det(ab) = det(a)det(b) Although the multiplication of matrices is not commutative, the determinant of a matrix multiplication is commutative by the commutative property of the product in R: det(ab) = det(a)det(b) = det(b)det(a) = det(ba) 11
2.2.2 Determinant of the inverse of a matrix Given a matrix A, the definition of its inverse A 1 is such that: A 1 A = AA 1 = I The determinant of the inverse matrix is the inverse of the determinant of the matrix. det(a 1 ) = 1 det(a) We will see more about the inverse of a matrix in the next section. 2.2.3 Determinant of the transposed matrix The determinant of the transpose of a matrix is the same as the determinant of the matrix. For this reason, it is not important if the vectors are rows or columns, when the determinant is calculated. det(a) = det(a T ) 2.2.4 Determinant of a matrix multiplied by a number The determinant of a matrix multiplied by a number is the determinant of the matrix times the number to the n th power, where n is the dimension of the matrix. A M n n (R) det(λa) = λ n det(a) det(λa) = det(λi n A) = det(λi n )det(a) = λ n det(a) 2.2.5 Determinant of triangular and diagonal matrices The determinant of triangular and diagonal matrices is the multiplication of the values in the main diagonal. For example: a b c A = 0 d e det(a) = adf 0 0 f 12
a 0 0 A = b c 0 det(a) = acf d e f a 0 0 A = 0 b 0 det(a) = abc 0 0 c 2.2.6 Changing rows and columns Exchanging a row or a column of a matrix changes the sign of the determinant. One sign change by row or column change. ( ) ( ) row v row w det = det row w row v det ( col v col w ) = det ( col w col v ) where u, v, represent row or column vectors. 2.2.7 Multiplying a row or a column by a scalar If a matrix is multiplied by a number in a row or a column, then the determinant is multiplied by that number. For instance: ( ) ( ) λa λb a b det = λdet c d c d For this reason, if a matrix has all its rows or columns multiplied by a number, the determinant is multiplied by λ n (see section 2.2.4). 2.2.8 Determinant when we apply linear combinations The determinant of a matrix will not change if we exchange one row or column for a linear combination of that row (column) with other rows (columns). For instance: ( ) ( ) a b a b + λa det = det, λ R c d c d + λc 13
2.2.9 Determinant with linearly dependent rows or columns When a row or a column of a matrix is linearly dependent to the others, the determinant is 0. This may happen in different ways. For instance, one row (or column) is proportional to another one: ( ) a b det = aλb λab = 0 λa λb Particular cases are when two rows (or columns) are equal: ( ) a b det = ab ba = 0 a b or when one row (or column) is zero: ( ) a 0 det = a 0 c 0 = 0 c 0 We may also have one row (or column) that is a linear combination of other rows (or columns): a b c det d e f = 0 a + d b + e c + f In fact, this is a very effective way to test linear independence between vectors. Proposition: Row or column vectors in a square matrix are linearly dependent if and only if the determinant is 0. Proof: We have to prove both directions. 1. If rows/columns linearly dependent = determinant = 0 2. If determinant = 0 = rows/columns linearly dependent Proof of 1: Assume rows or columns (from now, vectors) are linearly dependent. Then there are λ i 0 such that λ 1 v 1 + +λ n v n = 0. Hence, v i is a linear combination of the other vectors for any i. Changing a vector by a linear combination with the other vectors does not change the determinant (section 14
2.2.8). In particular, we can change v i by v i + (λ 1 v 1 +... no ith term + λ n v n )/λ i. We thus end up with a whole row or column of 0. This makes the determinant 0. Proof of 2: Proof by contradiction. If determinant = 0 = rows/columns linearly dependent is the same as saying that if vectors linearly independent = determinant 0. Performing linear combinations and triangulating the matrix (Gauss or Gauss-Jordan method), if the vectors are linearly independent, then there is no zero vector at the end. The determinant of a triangular matrix is the multiplication of the elements of its principal diagonal (section 2.2.5). As there is no zero vector, then the multiplication of the elements of the main diagonal (the determinant) cannot be 0. Q.E.D. 3 Rank of a matrix 3.1 Definition of the rank of a matrix The rank of the matrix is the maximum number of linearly independent rows or columns. A M m n (R), rank(a) N The rank of a matrix is determined by the non-zero minor of highest possible dimension (recall the definition of minor provided before). To calculate the rank of a matrix, we will calculate the determinant of the matrix and of the minors until we find one different from 0. For a square matrix: For non-square matrices: A M n n (R) rank(a) n det(a) = 0 rank(a) < n A M m n (R) size of the largest square submatrix with non-zero determinant. rank(a) min(m, n) 15
3.2 Calculation of the rank of a matrix We can calculate the rank of the matrix using determinants or using Gauss method. Using determinants we have to find the largest square submatrix whose determinant is not zero. For example: 1 2 1 2 A = 2 3 1 2 3 5 0 0 Let s first try for a 3 3 submatrix. 1 2 1 det(u) = 2 3 1 3 5 0 = 1 ( 3) 0+( 2) 5 1+3 2 1 3 ( 3) 1 ( 2) 2 0 1 5 1 = 0 det(u 3 3 ) = 0 rank(u) < 3 Let s try now with a submatrix 2 2 det(u) = 1 2 2 3 = 1 ( 3) ( 2) (2) = +1 0 det(u 2 2 ) 0 rank(u) = 2 rank(a) = 2. We can do it also using Gauss method, i.e. triangulating the matrix. 1 2 1 2 1 2 1 2 1 2 1 2 A = 2 3 1 2 0 1 3 6 0 1 3 6 3 5 0 0 0 1 3 6 0 0 0 0 There are two non-zero rows. Therefore, rank(a) = 2 16
4 Inverse of a matrix 4.1 Definition of inverse matrix Matrix division is not a possible operation. However, there is a way to apply an operation to a matrix to obtain the unit matrix: the inverse matrix. The inverse matrix of a square matrix A M n n (R) is another square matrix A 1 M n n (R), such that: AA 1 = A 1 A = I n Proposition: the inverse of a matrix, if exists, is unique. Proof: Suppose that there are two inverse matrices B and C of the matrix A. Then they satisfy AB = BA = I and AC = CA = I. B = BI = B(AC) = (BA)C = IC = C Q.E.D. Not all matrices have inverse. For a matrix to have inverse, it has to be square and the determinant must be 0 (condition of existence). Proof: A M n n (R) is invertible (non-singular) det(a) 0 AA 1 = I det(aa 1 ) = det(i) det(a)det(a 1 ) = det(i) Accordingly det(a)det(a 1 ) = 1 det(a), det(a 1 ) 0 det(a)det(a 1 ) = 1 det(a 1 ) = 1 det(a) 4.2 Properties of the inverse of a matrix The inverse of the inverse matrix is the original matrix. 17
If A is invertible A 1 (A 1 ) 1 = A. If A, B are invertible, then AB is also invertible and: (AB) 1 = B 1 A 1 AB (AB) 1 = ABB 1 A 1 = AI n A 1 = AA 1 = I n If A is invertible, then its transpose is also invertible and: ( ) A T 1 ( ) = A 1 T The inverse of the power of a matrix is the inverse matrix to the same power: ( ) A k 1 = (A... (k times)... A) 1 = ( A 1) k The inverse of a matrix multiplied by a number is the inverse of the number multiplied by the inverse matrix (λa) 1 = 1 λ A 1 For matrices in general AC = BC does not imply A = B. It is only true when C 1 exists, then AC = BC = A = B 4.3 Calculation of the inverse of a matrix To calculate the inverse of a matrix, we use the cofactor matrix. We define the adjoint matrix as the transpose of the cofactor matrix: Adj(A) = C T The inverse of the matrix is the adjoint matrix over the determinant of the matrix. A 1 = 1 det(a) Adj(A) 18
General example in 2 2: Therefore: A 1 1 = det(a) Adjt(A) AAdj(A) = det(a) I n ( ) ( ) T ( ) ( ) a b c11 c AAdj(A) = 12 a b d b = = c d c 21 c 22 c d c a ( ) ( ) ( ) ad bc 0 det(a) 0 1 0 = = = det(a) 0 ad bc 0 det(a) 0 1 ( ) 1 A Adj(A) = det(a) I n A det(a) Adj(A) = I n A 1 = With this, you can also show that: ( ) 1 det(a) Adj(A) A = I n 1 det(a) Adj(A) We can calculate the matrix by its definition, having the elements as variables and solving the linear equations system. Example for M 2 2 (R): ( ) ( ) a b x y A = A 1 = (x, y, z, t not known) c d z t AA 1 = I 2 ( ) ( ) ( ) a b x y 1 0 = c d z t 0 1 ax + bz = 1 cx + dz = 0 ay + bt = 0 cy + cd = 1 19
( ) 1 2 Example: Consider the matrix A =. Find the inverse matrix. 1 3 ( ) ( ) ( ) x y 1 2 1 0 = z t 1 3 0 1 x y = 1 2x + 3y = 0 z t = 0 2z + 3t = 1 x = 3 5, y = 2 5, z = 1 5, t = 1 5 ( ) 3/5 2/5 A 1 = = 1 ( ) 3 2 1/5 1/5 5 1 1 20
5 Exercises Ex. 1 Given the matrices A and B, compute the matrix products AB and BA. ( ) 1 1 1 1 3 A =, B = 1 0 10 7 1 2 1 Ex. 2 Show that det(ab) = det(a)det(b) for the two matrices: ( ) ( ) a b α β A =, B = c d γ δ ( ) ( ) 1 2 1 2 Ex. 3 Given the matrices A =, B =, calculate their 2 3 3 6 determinant. What is their rank? What are the implications of these results? Ex. 4 For this problem assume that we know the following: if X is an m m matrix, Y is an m n matrix, 0 is the zero matrix and I is the identity matrix of appropriate size, then: ( ) X Y det = det(x) 0 I Given A an m n matrix and B an n m matrix. Prove that: ( ) 0 A det = det(ab). B I ( ) ( ) 0 A I 0 Hint: consider the product B I B I Ex. 5 Calculate the inverse of the matrix A = 1 5 2 Ex. 6 Consider the matrix A = 1 1 4 0 2 1 1. Calculate the cofactor matrix C. 21 ( ) 1 3 2 2
2. Calculate the adjoint matrix Adj(A) = C T 3. Calculate the determinant for A 4. Calculate the inverse A 1, using the previous results. Ex. 7 A matrix is called orthogonal if it is built from column vectors that are mutually perpendicular. A matrix is called orthonormal if it is orthogonal and all its column vectors have unit length. Show that the inverse of a 2 2 orthonormal matrix is its transpose. Hint: show that for A T A = I to be true, A has to be orthonormal. 22
6 R practical 6.1 Addition and subtraction of matrices To add or subtract matrices we use the basic operators + and. # Define the matrices > m <- matrix (c(1, 0, 2, 3), 2,2) > n <- matrix (c(-1, 5, 3, -1), 2,2) # Operate with the matrices > m+n [,1] [,2] [1,] 0 5 [2,] 5 2 > m-n [,1] [,2] [1,] 2-1 [2,] -5 4 # If the matrices are not of # the same dimensions, an error will be thrown > p <- matrix (c(1,3,0,-1,5,2), 2, 3) > m+p Error in m + p : non - conformable arrays 6.2 Multiplication of a matrix by a number To multiply a matrix by a number we use the symbol in between the matrix and the number. # Introduce the matrix > m <- matrix (c(1, 0, 2, 3), 2,2) > m [,1] [,2] [1,] 1 2 [2,] 0 3 23
> 5*m [,1] [,2] [1,] 5 10 [2,] 0 15 6.3 Multiplication of matrices The multiplication of matrices can be done in two ways: the false matrix multiplication, which consists in multiplying each element (like addition/- subtraction operation, but multiplying the elements), or the true matrix multiplication. # Define two matrices > m <- matrix (c(1, 0, 2, 3), 2,2) > n <- matrix (c(-1, 5, 3, -1), 2,2) #" False " multiplication # Multiply each element > m*n [,1] [,2] [1,] -1 6 [2,] 0-3 #" True " multiplication > m %*% n [,1] [,2] [1,] 9 1 [2,] 15-3 # Remember that the matrix multiplication # is not commutative, in general. >n %*% m [,1] [,2] [1,] -1 7 [2,] 5 7 # If two matrices cannot be multiplied # because of the dimensions, an error will be risen > p <- matrix (c(1,3,0,-1,5,2), 2, 3) > p %*% m 24
Error in p %*% m : non - conformable arguments 6.4 Matrix transposition R has a command to transpose matrices and arrays. It is the same as we saw for vectors in Unit 1: Vector Spaces. # Introduce the matrix > m <- matrix (c(1, 0, 2, 3), 2,2) # Transpose the matrix > mt <- t(m) > mt [,1] [,2] [1,] 1 0 [2,] 2 3 # We can also prove that the transposition # of a transposed matrix returns the # original matrix > mtt <- t(mt) > mtt [,1] [,2] [1,] 1 2 [2,] 0 3 # We can use conditional operators, as well. > mtt == m [,1] [,2] [1,] TRUE TRUE [2,] TRUE TRUE 6.5 Determinant of a matrix To calculate the determinant of a matrix we use the command det() in R. # Define a matrix > m <- matrix (c(1, 0, 2, 3), 2,2) # Compute the determinant 25
> det (m) [1] 3 # If the matrix is not square, # it raises an error > p <- matrix (c(1,3,0,-1,5,2), 2, 3) > det (p) Error in determinant. matrix ( x, logarithm = TRUE,...) : x must be a square matrix As an exercise, you can try to test the properties of determinants by making up a matrix and testing it. 6.6 Rank of a matrix To calculate the rank of a matrix we will use the command qr() in R. The rank is stored in qr()$rank # Define a non - square matrix > m <- matrix (c(1, -2, 3, 2, -3, 5, 1, 1, 0, 2, 2, 0), 3, 4) # Compute the rank > rank <- qr(m)$ rank > rank [1] 2 6.7 Inverse of a matrix To compute the inverse of a matrix we will use the command inv() which is included in the package matlib. # Install the package if not installed yet > install. packages (" matlib ") # Load the package if installed > library (" matlib ") # Introduce the matrix > m <- matrix (c(1, -1, 2, 3), 2, 2) > mi <- inv (m) > mi 26
[,1] [,2] [1,] 0.6-0.4 [2,] 0.2 0.2 # Just to be sure... > 5*mi [,1] [,2] [1,] 3-2 [2,] 1 1 27
References [1] Howard Anton. Introducción al álgebra lineal. 2003. [2] Marc Peter Deisenroth; A Aldo Faisal and Cheng Soon Ong. Mathematics for Machine Learning. 2018. [3] Michael Friendly. Inverse of a matrix, 2018. [4] Soren Hojsgaard. Introduction to linear algebra with R. 2011. [5] Jordi Villà and Pau Rué. Elements of Mathematics: an embarrasignly simple (but practical) introduction to algebra. 2011. 28