Matrix Arithmetic
Matrix Addition and Subtraction Addition and subtraction act element-wise on matrices. In order for the addition/subtraction (A B) to be possible, the two matrices A and B must have the same size. Then, we simply add/subtract the corresponding elements of each matrix to get the result: (A + B) ij = A ij + B ij (A B) ij = A ij B ij For example: ( ) 2 3 + 1 2 ( ) 4 5 = 6 3 ( ) 6 8 7 5
Example: Addition and Subtraction a) Compute A + B, if possible: 2 3 1 4 5 6 A = 1 1 1 B = 1 0 4 2 2 1 3 4 3 We can add the matrices because they have the same size. 6 8 5 A + B = 0 1 5 5 6 4 b) Compute A H, if possible: ( ) 1 2 A = H = 3 5 ( 6 5 ) 10 0.1 0.5 0.9 We cannot subtract these matrices because they don t have the same size.
Scalar Multiplication αa Multiplying a matrix by a scalar also works element-wise: (αa) ij = αa ij For example, 2 ( ) 2 3 = 1 2 ( ) 4 6 2 4
Geometry of Scalar Multiplication Scalar multiplication changes the length of a vector but not the overall direction (although a negative scalar will scale the vector in the opposite direction through the origin).
Geometry of Vector Addition/Subtraction Geometrically, vector addition is witnessed by placing the two vectors, a and b, tail-to-head. The result, a + b, is the vector from the open tail to the open head. When subtracting vectors as a b we simply add b to a. The vector b has the same length as b but points in the opposite direction. a+b a b a b a-b
Linear Combinations A linear combination is constructed from a set of terms {v 1, v 2,..., v n } by multiplying each term by a scalar constant and adding the result: n c = α 1 v 1 + α 2 v 2 + + α n v n = α i v n The coefficients α i are scalar constants and the terms, {v i } can be scalars, vectors, or matrices. Most often, we will consider linear combinations where the terms {v i } are vectors. i=1
Elementary Linear Combinations The simplest linear combination might involve columns of the identity matrix (elementary vectors): 3 1 0 0 2 = 3 0 2 1 + 4 0 4 0 0 1 We can easily picture this linear combination as a breakdown into parts where the parts give directions along the 3 coordinate axis.
Linear Combination of Matrices Write the matrix A = following matrices: {( ) 1 0, 0 0 Solution: ( ) 1 3 A = = 1 4 2 ( ) 1 3 as a linear combination of the 4 2 ( ) 0 1, 0 0 ( ) 1 0 + 3 0 0 ( ) 0 0, 1 0 ( ) 0 1 + 4 0 0 ( )} 0 0 0 1 ( ) 0 0 + 2 1 0 ( ) 0 0 0 1
Matrix Multiplication
Matrix Multiplication Matrix Multiplication is NOT done element-wise. Matrix Multiplication is NOT commutative = ORDER MATTERS!! AB BA!!
A Notational Note For the remainder of this course, unless otherwise specified, we will consider vectors to be columns rather than rows. This makes the notation more simple because if x is a column vector, x 1 x 2 x =. x n then we can automatically assume that x T is a row vector: x T = ( x 1 x 2... x n ).
The Inner Product (Dot Product) The inner product of two vectors, x and y, written x T y, is defined as the sum of the product of corresponding elements in x and y: x T y = n x i y i. If we write this out for two vectors with 4 elements each, we d have: y 1 i=1 x T y = ( ) x 1 x 2 x 3 x 4 y 2 y 3 = x 1y 1 + x 2 y 2 + x 3 y 3 + x 4 y 4 y 4 Note: The inner product between vectors is only possible when the two vectors have the same number of elements!
The Inner Product The inner product defines how we multiply ROW VECTOR COLUMN VECTOR The product of two rows or two columns is not defined! We will see why shortly!
Example - Inner Products Let x = 1 2 4 0 y = 3 2 3 5 1 v = 2 5 3 u = 1 3 3 7 2 2 If possible, compute the following inner products: a. x T y 3 x T y = ( 1 2 4 0 ) 5 1 7 = ( 1)(3) + (2)(5) + (4)(1) + (0)(7) = 3 + 10 + 4 = 11 (1)
Example - Inner Products Let x = 1 2 4 0 y = 3 2 3 5 1 v = 2 5 3 u = 1 3 3 7 2 2 If possible, compute the following inner products: b. x T v This is not possible because x and v do not have the same number of elements c. v T u 2 v T u = ( 3 2 5 3 2 ) 1 3 3 2
Inner Products It should be clear from the definition that for all vectors x and y, x T y = y T x. Also, if we take the inner product of a vector with itself, the result is the sum of squared elements in that vector: x T x = n x 2 i = x 2 1 + x2 2 + + x2 n. i=1
Check Your Understanding Let 1 1 0.5 10 2 v = 3 4 e = 1 1 1 p = 0.1 0.2 0 u = 4 2 3 2 s = 2 3 5 1 0.2 1 If possible, compute the following inner products: a. v T e d. p T u b. e T v e. v T v c. v T s
Check Your Understanding - Solution Let 1 1 0.5 10 2 v = 3 4 e = 1 1 1 p = 0.1 0.2 0 u = 4 2 3 2 s = 2 3 5 1 0.2 1 If possible, compute the following inner products: a. v T e = 15 d. p T u = 6.2 b. e T v = 15 e. v T v = 55 c. v T s = not possible
Matrix Product AB Matrix multiplication is just a series of inner products done simultaneously in one operation. As with vectors, the operation is not always possible. The order in which you multiply matrices makes a big difference!
Matrix Multiplication - Definition Let A m n and B k p be matrices. The matrix product AB is possible if and only if n = k; that is, when the number of columns in A is the same as the number of rows in B. If this condition holds, then the dimension of the product, AB is m p and the (ij)-entry of the product AB is the inner product of the i th row of A and the j th column of B: (AB) ij = A i B j
Steps to Compute a Matrix Product Let 2 3 A = 1 4 and B = 5 1 ( ) 0 2 2 3 When we first get started with matrix multiplication, we often follow a few simple steps: 1) Write down the matrices and their dimensions. Make sure the inside dimensions match - those corresponding to the columns of the first matrix and the rows of the second matrix: A B (3 2)(2 2) If these dimensions match, then we can multiply the matrices. If they don t, multiplication is not possible.
Steps to Compute a Matrix Product Let 2 3 A = 1 4 and B = 5 1 ( ) 0 2 2 3 2) Now, look at the outer dimensions - this will tell you the size of the resulting matrix. A B (3 2)(2 2) So the product AB is a 3 2 matrix.
Steps to Compute a Matrix Product Let 2 3 A = 1 4 and B = 5 1 ( ) 0 2 2 3 (AB) 11 (AB) 12 AB = (AB) 21 (AB) 22 (AB) 31 (AB) 32 3) Finally, we compute the product of the matrices by multiplying each row of A by each column of B using inner products. (AB) 11 = A 1 B 1 (AB) ij = A i B j
Steps to Compute a Matrix Product Let 2 3 A = 1 4 and B = 5 1 ( ) 0 2 2 3 AB = = (2)(0) + (3)(2) (2)( 2) + (3)( 3) ( 1)(0) + (4)(2) ( 1)( 2) + (4)( 3) (5)(0) + (1)(2) (5)( 2) + (1)( 3) 6 13 8 10 (2) 2 13
Check Your Understanding Suppose we have A 4 6 B 5 5 M 5 4 P 6 5 Circle the matrix products that are possible to compute and write the dimension of the result. AM MA BM MB PA PM AP A T P M T B
Check Your Understanding - Solution Suppose we have A 4 6 B 5 5 M 5 4 P 6 5 Circle the matrix products that are possible to compute and write the dimension of the result. AM MA BM MB PA PM AP A T P M T B 5 6 5 4 6 4 4 5 4 5
Check Your Understanding Let A = ( 1 1 0 1 ) ( ) 2 1 1 2 2 M = 1 2 0 1 2 ( ) 1 0 1 0 C = 1 1 0 0 Determine the following matrix products, if possible. a. AC b. AM c. AC T
Check Your Understanding - Solution Let ( ) ( ) 1 1 2 1 1 2 2 A = M = 0 1 1 2 0 1 2 ( ) 1 0 1 0 C = 1 1 0 0 Determine the following matrix products, if possible. ( ) 0 1 1 0 a. AC = 1 1 0 0 b. AM = ( 1 1 1 1 ) 0 1 2 0 1 2 c. AC T = not possible
One very important thing to keep in mind is that matrix multiplication is not commutative! As we see from the previous exercises, it s quite common to be able to compute a product AB where the reverse product, BA is not even possible to compute. Even if both products are possible it is almost never the case that AB equals BA
Multiplication by a Diagonal Matrix 2 0 0 1 2 3 D = 0 3 0 A = 1 1 2 0 0 2 2 1 3 2 4 6 DA = 3 3 6 4 2 6 In doing this multiplication, we see that the effect of multiplying the matrix A by a diagonal matrix on the left is that the rows of the matrix A are simply scaled by the entries in the diagonal matrix.
Matrix-Vector Product A matrix-vector product works exactly the same way as matrix multiplication. Again we must match the dimensions to make sure they line up correctly. Suppose we have A m n. Then we can multiply a row vector v T 1 m on the left: v T A works because v T A (1 m)(m n) = The result will be a 1 n row vector. Or, we can multiply an n 1 column vector x on the right: Ax works because A x (m n) (n 1) = The result will be a m 1 column vector.
Matrix-Vector Product Matrix-vector multiplication works the same way as matrix multiplication: we simply multiply rows by columns until we ve completed the answer. As we will see in the near future, there are many ways to view matrix multiplication. Multiplying a matrix by a vector amounts to taking a linear combination of its rows or columns.
Matrix-Vector Products Let 2 3 A = 1 4 v = 5 1 ( ) 3 2 2 q = 1 3 Determine whether the following matrix-vector products are possible. When possible, compute the product. a. Aq Not Possible: Inner dimensions do not match A q (3 2) (3 1) b. Av 2 3 1 4 5 1 ( ) 3 = 2 2(3) + 3(2) 12 1(3) + 4(2) = 5 5(3) + 1(2) 17
Linear Combination of Columns 2 3 ( ) Av = 1 4 3 2 5 1 2(3) + 3(2) = 1(3) + 4(2) = 5(3) + 1(2) 2 3 12 3 1 + 2 4 = 5 (3) 5 1 17
Why Does this Viewpoint Matter? Suppose I had a matrix of data, containing columns for height, weight, and length. If I wanted to make a new variable, that was equal to 0.5 height + 2 weight + 0.75 length All I would have to do is multiply my data matrix by the vector 0.5 v = 2 0.75
Vector Outer Product There is one last type of product to consider: A column vector times a row vector. Regardless of the vectors size, this will always be possible. The dimensions of the outcome? x (m 1) yt = M m n (1 n) = The result is a matrix! Treat this product the same way we treat any matrix product, by multiplying row column until we ve run out of rows and columns.
Vector Outer Product - Example 3 1 Let x = 4 and y = 5. Then, 2 3 3 xy T = 4 ( 1 5 3 ) 3 15 9 = 4 20 12 2 2 10 6 As you can see by performing this calculation, a vector outer product will always produce a matrix whose rows are multiples of each other!
Special Cases of Matrix Multiplication
The Identity Matrix The Identity Matrix, I, is to matrices as the number 1 is to scalars. It is the multiplicative identity. For any matrix (or vector) A, multiplying A by the identity matrix on either side does not change A: AI = A IA = A This is easy to verify since the identity matrix is simply a diagonal matrix with ones on the diagonal.
The Matrix Inverse For certain square matrices, A, an inverse matrix, written A 1, exists such that AA 1 = I A 1 A = I A MATRIX MUST BE SQUARE TO HAVE AN INVERSE NOT ALL SQUARE MATRICES HAVE AN INVERSE Only full-rank, square matrices are invertible. We will talk about rank soon. For now, understand that the inverse matrix serves like the multiplicative inverse in scalar algebra: (2)(2 1 ) = (2)( 1 2 ) = 1 Multiplying a matrix by its inverse (if it exists) yields the multiplicative identity, I.
The Matrix Inverse A square matrix which has an inverse is equivalently called: Non-singular Invertible
Don t cancel that!! Ax = λx A x = λ x????????????? A m n x n 1 = λ 1 1 x m 1 A is a matrix and λ is a scalar! A = λ is nonsense!
Cancellation implies multiplication by an inverse Canceling numbers in algebra: 2x = 2y 2x = 2y 1 2 2x = 1 2 2y 1x = 1y Canceling matrices in linear algebra: Ax = b A 1 Ax = A 1 b Ix = A 1 b x = A 1 b So inverses are great - WHEN THEY EXIST!