Matrix-Matrix Multiplication

Transcription

1 Chapter Matrix-Matrix Multiplication In this chapter, we discuss matrix-matrix multiplication We start by motivating its definition Next, we discuss why its implementation inherently allows high performance Motivating Example: Rotations Consider the transformation R : R R that, given a vector x R returns y Rx, where y is the vector x but rotated through an angle ρ Example The rotation R so defined is a linear transformation Rαx αrx: One can first scale x by a factor α and then rotate it through an angle ρ or one can first rotate x through an angle ρ and then scale the result by a factor α Rx + y Rx +Ry: One can first add the vectors and then rotate, or one can rotation each of them first, and then add the results We need a picture to accompany this! Now, let s see what the matrix is that represents R We need to determine how e 0 0 and e 0 are transformed by the transformation R Simple trigonometry tells cos ρ 0 sin ρ us that R and R We need a picture to 0 sin ρ cos ρ accompany this! Thus, cos ρ sin ρ Rx Ax where A sin ρ cos ρ 55

2 56 Chapter Matrix-Matrix Multiplication Next, let us consider the transformation S : R R that rotates vectors through an angle σ Then cos σ sin σ Sx Bx where B sin σ cos σ Finally, consider the transformation T that rotates vectors through an angle ρ + σ so that cosρ + σ sinρ + σ T x Cx where C sinρ + σ cosρ + σ We notice that T x RSx: rotating a vector through angle ρ + σ can be accomplished by first rotating that vector through angle σ and then rotating the result through angle ρ Thus, Cx ABx The question is, how are A, B, and C related when Cx ABx? Recall that the jth row of matrix C, c j, equals Ce j so that cos ρ sin ρ cos σ c 0 Ce 0 ABe 0 Ab 0 sin ρ cos ρ sin σ cos ρ cos σ sin ρ sin σ sin ρ cos σ + cos ρ sin σ cos ρ sin ρ c Ce ABe Ab sin ρ cos ρ sin ρ cos σ cos ρ sin σ cos ρ cos σ sin ρ sin σ sin σ cos σ so that cosρ + σ sinρ + σ cos ρ cos σ sin ρ sin σ sin ρ cos σ cos ρ sin σ C sinρ + σ cosρ + σ sin ρ cos σ + cos ρ sin σ cos ρ cos σ sin ρ sin σ This is consistent with what we learned in high school: from trigonometry we remember that cosρ + σ cos ρ cos σ sin ρ sin σ sinρ + σ sin ρ cos σ + cos ρ sin σ What we will see next, in more generality, is that C is the matrix product of A and B: C ABx ABx In other words, C is defined to be the matrix that represents the composition of transformations R and S, which is then written as C AB Composing Linear Transformations Theorem Let L A : R k R m and L B : R n R k both be linear transformations and, for all x R n, define the function L C : R n R m by L C x L A L B x Then L C x is a linear transformations

3 Composing Linear Transformations 57 Proof: Let x, y R n and α, β R Then L C αx + βy L A L B αx + βy L A αl B x+βl B y αl A L B x + βl A L B y αl C x+βl C y Now, let linear transformations L A, L B, and L C be represented by matrices A R m k, B R k n, and C R m n, respectively Then Cx L C x L A L B x ABx The matrix-matrix multiplication product is defined so that C A B AB It composes A and B into C just like L C is the composition of L A and L B Remark 3 If A is m A n A matrix, B is m B n B matrix, and C is m C n C matrix, then for C AB to hold it must be the case that m C m A, n C n B, and n A m B The question now becomes how to compute C given matrices A and B For this, we are going to use and abuse the unit basis vectors e j Let C γ 0,0 γ 0, γ 0,n α 0,0 α 0, α 0,k γ,0 γ, γ,n, A α,0 α, α,k γ m,0 γ m, γ m,n α m,0 α m, α m,k β 0,0 β 0, β 0,n β,0 β, β,n and B β k,0 β k, β k,n Recall that γ i,j e T i Ce j : Ce j picks out the jth column of C and e T i Ce j then picks out the ith element of the jth column Recall that C AB is defined to be the matrix such that Cx ABx for all x Then γ i,j e T i Ce j e T i ABe j The result of Be j is the jth column of B, b j Thus, e T i ABe j e T i Ab j which equals the ith element of the vector Ab j By the definition of matrix-vector multiplication, the ith element of vector Ab j is given by the dot product of the ith row of A viewed as a column vector with the vector b j Now, the ith row of A and jth column of B are respectively given by αi,0 α i, α i,k and β 0,j β,j β k,j,

4 58 Chapter Matrix-Matrix Multiplication so that k γ i,j e T i ABxe j α i,0 β 0,j + α i, β,j + + α i,k β k,j α i,p β p,j p0 Definition 4 Matrix-matrix multiplication Let A R m k, B R k n, and C R m n Then the matrix-matrix multiplication product C AB is computed by setting k γ i,j α i,p β p,j α i,0 β 0,j + α i, β,j + + α i,k β k,j p0 As a result of this definition Cx ABx ABx and can drop the parentheses, unless they are useful for clarity: Cx ABx Example 5 In 7 and Exercise 6, notice that Q P P Example 6 In we notice that cosρ + σ sinρ + σ cos ρ sin ρ cos σ sin σ C sinρ + σ cosρ + σ sin ρ cos ρ sin σ cos σ cos ρ cos σ sin ρ sin σ sin ρ cos σ cos ρ sin σ sin ρ cos σ + cos ρ sin σ cos ρ cos σ sin ρ sin σ Remark 7 We emphesize that for matrix-matrix multiplication to be a legal operations, the row and column dimensions of the matrices must obey certain constraints Whenever we talk about dimensions being conformal, we mean that the dimensions are such that the encountered matrix multiplications are valid operations The following triple-nested loops compute C : AB + C:

5 Composing Linear Transformations 59 for j 0,, n for i 0,, m for p 0,, k k γ i,j : α i,p β p,j + γ i,j γ i,j : α i,p β p,j +γ i,j p0 for j:n for i:m for p:k Ci,j + Ai,p * Bp,j; end end end Figure : Simple triple-nested loop for computing C : AB + C for j 0,, n for i 0,, m for p 0,, k γ i,j : ǎ T i b j + γ i,j for j:n for i:m Ci,j + SLAP_Dot Ai,:, B:,j ; end end Figure : Algorithm and M-script from Figure with inner loop replaced by a dot product for j 0,, n for i 0,, m for p 0,, k c j : Ab j + c j for j:n C:,j + SLAP_Gemv SLAP_NO_TRANSPOSE,, A, B:,j,, C:,j ; end Figure 3: Algorithm and M-script from Figure with inner two loops replaced by a matrix-vector multiplication for i 0,, m for j 0,, n for p 0,, k The two outer-most loops sweep over all elements in C, and the inner loop computes the inner product of the ith row of A with the jth column of B If originally C 0, then the above algorithm computes C : AB Remark 8 When computing C AB + C the three loops can be nested in 3! 6 different ways We will examine this more, later In Figures 3 we show how an ordering that places the loop indexed by j first, the triplenested loop can be viewed as a double nested loop with dot products as the body which

6 60 Chapter Matrix-Matrix Multiplication then implements the third loop or as a single loop with matrix-vector multiplications as its loop 3 Special Cases of Matrix-Matrix Multiplication We now show that if one treats scalars, column vectors, and row vectors as special cases of matrices, then many operations we encountered previously become simply special cases of matrix-matrix multiplication In the below discussion, consider C AB where C R m n, A R m k, and B R k n m n k scalar multiplication In this case, all three matrices are actually scalars: γ0,0 α0,0 β0,0 α0,0 β 0,0 so that matrix-matrix multiplication becomes scalar multiplication Example 9 Let A 4 and B 3 Then AB 4 3 n,k scal γ 0,0 γ,0 α 0,0 α,0 Now the matrices look like β0,0 α 0,0 β 0,0 α 0,0 β,0 β 0,0 α 0,0 β 0,0 α,0 β 0,0 α 0,0 α,0 γ m,0 α m,0 α 0,0 β m,0 β 0,0 α m,0 α m,0 In other words, C and A are vectors, B is a scalar, and the matrix-matrix multiplication becomes scaling of a vector Example 0 Let A AB 3 3 and B 4 Then

7 3 Special Cases of Matrix-Matrix Multiplication 6 m,k scal Now the matrices look like γ0,0 γ 0, γ 0,n α0,0 β0,0 β 0, β 0,n α 0,0 β0,0 β 0, β 0,n α 0,0 β 0,0 α 0,0 β 0, α 0,0 β 0,n In other words, C and B are just row vectors and A is a scalar The vector C is computed by scaling the row vector B by the scalar A Example Let A 4 and B 3 Then AB m,n dot The matrices look like γ0,0 α0,0 α 0, α 0,k β 0,0 β,0 β k,0 k α 0,p β p,0 p0 In other words, C is a scalar that is computed by taking the dot product of the one row that is A and the one column that is B Example Let A 3 and B Then 0 AB k outer product γ 0,0 γ 0, γ 0,n γ,0 γ, γ,n α 0,0 α,0 β0,0 β 0, β 0,n γ m,0 γ m, γ m,n α m,0

8 6 Chapter Matrix-Matrix Multiplication Example 3 Let A AB 3 3 and B α 0,0 β 0,0 α 0,0 β 0, α 0,0 β 0,n α,0 β 0,0 α,0 β 0, α,0 β 0,n α m,0 β 0,0 α m,0 β 0, α m,0 β 0,n Then n matrix-vector product γ 0,0 γ,0 α 0,0 α 0, α 0,k α,0 α, α,k β 0,0 β,0 γ m,0 α m,0 α m, α m,k β k,0 m row vector-matrix product γ0,0 γ 0, γ 0,n α0,0 α 0, α 0,k β 0,0 β 0, β 0,n β,0 β, β,n β k,0 β k, β k,n so that γ 0,j k p0 α 0,pβ p,j Example 4 Let A 0 0 and B Then AB 4 0 Exercise 5 Let e i R m equal the ith unit basis vector and A R m n Show that e T i A ǎ T i, the ith row of A

9 4 Properties of Matrix-Matrix Multiplication 63 4 Properties of Matrix-Matrix Multiplication Let us examine some properties of matrix-matrix multiplication: Theorem 6 Let A, B, and C be matrices of conforming dimensions Then ABC ABC In other words, matrix multiplication is associative Proof: Let e j equal the jth unit basis vector Then ABCe j ABc j ABc j ABCe j ABCe j Thus, the columns of ABC equal the columns of ABC, making the two matrices equal Theorem 7 Let A, B, and C be matrices of conforming dimensions Then A + BC AC + BC and AB + C AB + AC In other words, matrix multiplication is distributative Remark 8 Matrix-matrix multiplication does not commute: Only in very rare cases does AB equal BA Indeed, the matrix dimensions may not even be conformal Theorem 9 Let A R m k and B R k n Then AB T B T A T Before proving this theorem, we first note that Lemma 0 Let A R m n Then e T i A T Ae i T and A T e j e T j A T The proof of this lemma is pretty obvious: The ith row of A T is clearly the ith column of A, but viewed as a row, etc Proof: of Theorem 9 We prove that the i, j element of AB T equals the i, j element of B T A T : e T i AB T e j

10 64 Chapter Matrix-Matrix Multiplication < i, j element of C equals j, i element of C T > e T j ABe i < Associativity of matrix multiplication > e T j ABe i <x T y y T x> Be i T e T j A T < Lemma 0 > e T i B T A T e j 5 Multiplying partitioned matrices Theorem Let C R m n, A R m k, and B R k n Let m m 0 + m + m M, m i 0 for i 0,, M ; n n 0 + n + n N, n j 0 for j 0,, N ; and k k 0 + k + k K, k p 0 for p 0,, K Partition C 0,0 C 0, C 0,N A 0,0 A 0, A 0,K C,0 C, C,N C, A A,0 A, A,K C M,0 C M, C M,N A M,0 A M, B 0,0 B 0, B 0,N A M,K and B,0 B, B,N B, B K,0 B K, B K,N, with C i,j R m i n j, A i,p R m i k p, and B p,j R kp n j Then C i,j K p0 A i,pb p,j Remark If one partitions matrices C, A, and B into blocks, and one makes sure the dimensions match up, then blocked matrix-matrix multiplication proceeds exactly as does a regular matrix-matrix multiplication except that individual multiplications of scalars commute while in general individual multiplications with matrix blocks submatrices do not

11 5 Multiplying partitioned matrices 65 Example 3 Consider A If A , A, B , B 0, and AB : 0, and B 4 0 Then AB A 0 A B 0 B A 0 B 0 + A B : }{{}}{{} A B }{{} }{{} B 0 }{{} A } {{ } A 0 B } {{ } A B A 0 } 4 0 {{ } B } {{ } AB

12 66 Chapter Matrix-Matrix Multiplication Corollary 4 In Theorem partition C and B by columns and do not partition A In other words, let M, m 0 m; N n, n j, j 0,, n ; and K, k 0 k Then C c 0 c c n and B b 0 b b n so that c0 c c n C AB A b0 b b n Ab0 Ab Ab n Example By moving the loop indexed by j to the outside in the algorithm for computing C AB + C we observe that for j 0,, n for i 0,, m for p 0,, k c j : Ab j + c j or for j 0,, n for p 0,, k for i 0,, m c j : Ab j + c j

13 5 Multiplying partitioned matrices 67 Corollary 6 In Theorem partition C and A by rows and do not partition B In other words, let M m, m i i, i 0,, m ; N, n 0 n; and K, k 0 k Then C c T 0 c T and A ã T 0 ã T c T m ã T m so that c T 0 c T c T m C AB ã T 0 ã T ã T m B ã T 0 B ã T B ã T m B Example In the algorithm for computing C AB + C the loop indexed by i can be moved to the outside so that for i 0,, m for j 0,, n for p 0,, k c T i : ã T i B + ct i or for i 0,, m for p 0,, k for j 0,, n c T i : ã T i B + ct i

14 68 Chapter Matrix-Matrix Multiplication Corollary 8 In Theorem partition A and B by columns and rows, respectively, and do not partition C In other words, let M, m 0 m; N, n 0 n; and K k, k p, p 0,, k Then A a 0 a a k and B bt 0 bt bt k so that C AB a 0 a a k bt 0 bt bt k a 0 b T 0 + a bt + + a k bt k Example In the algorithm for computing C AB + C the loop indexed by p can be moved to the outside so that for p 0,, k for j 0,, n for i 0,, m C : a p bt p + C or for p 0,, k for i 0,, m for j 0,, n C : a p bt p + C Example 30 In Theorem partition C into elements scalars and A and B by rows and columns, respectively, and do not partition C In other words, let M m, m i,

15 5 Multiplying partitioned matrices 69 i 0,, m ; N N, n j, j 0,, n ; and K, k 0 k Then γ 0,0 γ 0, γ 0,n ã T 0 γ,0 γ, γ,n C, A ã T, and B b 0 b b n γ m,0 γ m, γ m,n ã T m so that γ 0,0 γ 0, γ 0,n γ,0 γ, γ,n C γ m,0 γ m, γ m,n ã T 0 b 0 ã T 0 b ã T 0 b n ã T b 0 ã T b ã T b n ã T m b 0 ã T m b ã T m b n ã T 0 ã T ã T m b0 b b n As expected, γ i,j ã T i b j : the dot product of the ith row of A with the jth row of B Example In the algorithm for computing C AB + C the loop indexed by p which computes the dot product of the ith row of A with the jth column of B can be moved to the inside so that

16 70 Chapter Matrix-Matrix Multiplication for j 0,, n for i 0,, m for p 0,, k γ i,j : ã T i b j + γ i,j or for i 0,, m for j 0,, n for p 0,, k γ i,j : ã T i b j + γ i,j Notice that the algorithm on the left already appeared after Example?? while the one on the right appeared after Example 7 6 Summing it all up Figure 4 summarizes how matrix-matrix multiplication can be implemented in terms of operations that we studies in the previous chapter It explains how the six different orderings of the loops exhibit themselves as matrix-matrix multiplication is viewed in a layered fashion

17 6 Summing it all up 7 for j 0,, n c j : Ab j + c j for j 0,, n for i 0,, m γ i,j : ǎ T i b j + γ i,j for j 0,, n for p 0,, k c j : β p,j a p + c j for j 0,, n for i 0,, m for p 0,, k for j 0,, n for p 0,, k for i 0,, m C AB + C for i 0,, m č T i : ǎ T i B +čt i for i 0,, m for j 0,, n γ i,j : ǎ T i b j + γ i,j for i 0,, m for p 0,, k č T i : α i,pˇbt p +č T i for i 0,, m for j 0,, n for p 0,, k for i 0,, m for p 0,, k for j 0,, n for p 0,, k C : a pˇbt p + C for p 0,, k for i 0,, m č T i : α i,pˇbt p +č T i for p 0,, k for j 0,, n c j : β p,j a p + c j for p 0,, k for i 0,, m for j 0,, n for p 0,, k for j 0,, n for i 0,, m Figure 4: Algorithms for implementing C : AB + C