CS 4424 Matrix multiplication

Transcription

1 CS 4424 Matrix multiplication 1

2 Reminder: matrix multiplication Matrix-matrix product. Starting from a 1,1 a 1,n A =.. and B = a n,1 a n,n b 1,1 b 1,n.., b n,1 b n,n we get AB by multiplying A by all columns of B (or all rows of A by B). Explicitly, AB = a 1,1 b 1,j + + a 1,n b n,j.. a n,1 b 1,j + + a n,n b n,j 2

3 2 2 matrix multiplication With we get A = a 1,1 a 1,2 and B = b 1,1 b 1,2, a 2,1 a 2,2 b 2,1 b 2,2 AB = a 1,1b 1,1 + a 1,2 b 2,1 a 1,1 b 1,2 + a 1,2 b 2,2. a 2,1 b 1,1 + a 2,2 b 2,1 a 2,1 b 1,2 + a 2,2 b 2,2 3

4 Naive algorithm for i = 1,..., n for j = 1,..., n c i,j = 0 for k = 1,..., n c i,j = c i,j + a i,k b k,j Total: n 3 mul, n 3 n 2 add. 4

5 Main results 1. One can multiply matrices of size n using O(n 3 ) operations (naive algorithm) O(n log 2 7 ) operations using Strassen s algorithm (1969) (... many improvements) O(n 2.39 ) operations using Coppersmith and Winograd s algorithm (1990). We let ω be any number such that matrix multiplication can be done in O(n ω ) operations, so that ω One can invert matrices (and do many other things) in O(n ω ). 5

6 Practical aspects Many of these algorithms offer no interest for practical computations. One uses the naive algorithm (or a couple of variants of it) for sizes up to 100. For large sizes, algorithms by Strassen (ω = 2.81) and Pan (ω = 2.77) are sometimes used. None of the other ones is useful (threshold too high). For matrices with double entries, optimizing data access is more important. 6

7 Strassen s algorithm Similar to Karatsuba s algorithm: find an improvement for a base case: 2 2 matrices use it recursively. For the 2 2 case, given A = a 1,1 a 1,2 and B = a 2,1 a 2,2 b 1,1 b 1,2, b 2,1 b 2,2 we compute 7 linear combinations of the a i,j and b i,j multiply them pairwise recombine the results 7

8 Formulas q 1 = (a 1,1 a 1,2 )b 2,2 q 2 = (a 2,1 a 2,2 )b 1,1 q 3 = a 2,2 (b 1,1 + b 2,1 ) q 4 = a 1,1 (b 1,2 + b 2,2 ) q 5 = (a 1,1 + a 2,2 )(b 2,2 b 1,1 ) q 6 = (a 1,1 + a 2,1 )(b 1,1 + b 1,2 ) q 7 = (a 1,2 + a 2,2 )(b 2,1 + b 2,2 ) and c 1,1 = q 1 q 3 q 5 + q 7 c 1,2 = q 4 q 1 c 2,1 = q 2 + q 3 c 2,2 = q 2 q 4 + q 5 + q 6 8

9 n n matrices Suppose that we have to multiply A and B in size n, with n = 2 k. We break them into blocks: A = A 1,1 A 1,2 A 2,1 A 2,2 and B = B 1,1 B 1,2 B 2,1 B 2,2 where A i,j et B i,j have size n/2 n/2. The formulas we used for the case 2 2 still work. They allow us to multiply A and B using 7 products in size n/2 O(n 2 ) extra operations. 9

10 Complexity analysis Let MM(n) be the cost of multiplication in size n. Then we have MM(n) 7MM(n/2) + λn 2 and so MM(n) Cn log(7) log(2) Cn Proof: master theorem. 10

11 Beyond Strassen More generally: if you find an algorithm that does k multiplications in size n then you can take ω = log(n)/ log(k) n = 2: k = 7 is optimal n = 3: k = 23 is known; k = 21 would improve ω n = 4: k = 49 n = 5: k = 100 is known; k = 91 would improve ω Remark for a given n and k, many attemps done to find algorithms by computer search. 11

12 Rectangular matrices 12

13 Notation Let s write n, m, p for the number of multiplications it takes to multiply matrices (n, m) (m, p). Prop. (we ve seen that before). If n, n, n = k then we can take Prop. If n, m, p = k then we can take ω = log(k) log(n). ω = 3 log(k) log(mnp). 13

14 Steps of the proof 1. (block matrices) mm, nn, pp m, n, p m, n, p. 2. (permutations) m, n, p = n, p, m = p, m, n. 3. (conclusion) mnp, mnp, mnp m, n, p n, p, m p, m, n m, n, p m, n, p m, n, p k 3 so we can take ω = log(k3 ) log(mnp) = 3 log(k) log(mnp). 14

15 Step 1: block matrices Suppose we have to multiply A of size (mm, nn ) by B of size (nn, pp ). We can decompose them into blocks: A 1,1 A 1,n A =.., B = A m,1 A m,n B 1,1 B 1,p.., B n,1 B n,p where: each A i,j is a matrix of size (m, n ) each B i,j is a matrix of size (n, p ). 15

16 Step 1: block matrices Their product is C = C 1,1 C 1,p.., C m,1 C m,p where each C i,j is a block of size (m, p ) To compute AB, we apply the algorithm in size (m, n, p) each of the products is done on blocks of size (m, n, p ) so the total number of multiplications is m, n, p m, n, p. 16

17 Step 2: easy permutations The transpose of a matrix is obtained by switching rows and columns. Prop. B t A t = (AB) t. A = = A t = Consequence: (with A of size (m, n) and B of size (n, p)) m, n, p = p, n, m. 17

18 Polynomial notation 18

19 Multiplication algorithms We have seen several algorithms for multiplying things polynomials (Karatsuba, FFT) power series matrices which all have the same structure 1. compute combinations of the inputs 2. multiply them pairwise 3. recombine the products to get the result. 19

20 We want to describe the multiplication Polynomial notation (a 0 + a 1 X)(b 0 + b 1 X) = a 0 b 0 + (a 1 b 0 + a 0 b 1 )X + a 1 b 1 X 2. We can describe this operation using a polynomial in variables (A 0, A 1 ), (B 0, B 1 ), (C 0, C 1, C 2 ): P poly2 = A 0 B 0 C 0 + A 0 B 1 C 1 + A 1 B 0 C 1 + A 1 B 1 C 2. translation compute A 0 B 0 and add it to C 0 compute A 0 B 1 and add it to C 1 compute A 1 B 0 and add it to C 1 compute A 1 B 1 and add it to C 2 20

21 Polynomial notation for Karatsuba We can rewrite the polynomial P poly2 as P poly2 = A 0 B 0 (C 0 C 1 ) + (A 0 + A 1 )(B 0 + B 1 )C 1 + A 1 B 1 (C 2 C 1 ) This is the same polynomial, just written differently. Now, the translation is: compute A 0 B 0, add the result to C 0, subtract it from C 1 compute (A 0 + A 1 )(B 0 + B 1 ), add the result to C 1 compute A 1 B 1, add the result to C 2, subtract it from C 1. 21

22 Polynomial notation for matrices For the multiplication of 2 2 matrices, we have P mat2 = A 1,1 B 1,1 C 1,1 + A 1,2 B 2,1 C 1,1 + A 1,1 B 1,2 C 1,2 + A 1,2 B 2,2 C 1,2 + A 2,1 B 1,1 C 2,1 + A 2,2 B 2,1 C 2,1 + A 2,1 B 1,2 C 2,2 + A 2,2 B 2,2 C 2,2. translation compute A 1,1 B 1,1 and add it to C 1,1, compute A 1,2 B 2,1 and add it to C 1,1,... 22

23 Polynomial notation for Strassen s algorithm We can rewrite P mat2 as P mat2 = (A 1,1 A 1,2 )B 2,2 (C 1,1 C 1,2 ) + (A 2,1 A 2,2 )B 1,1 C 2,1 + A 2,2 (B 1,1 + B 2,1 )( C 1,1 + C 2,1 ) + A 1,1 (B 1,2 + B 2,2 )(C 1,2 C 2,2 ) +... translation compute (A 1,1 A 1,2 )B 2,2, and add it to C 1,1, subtract it from C 1,

24 Polynomial notation for (1, 2) (2, 3) Let s compute ] [A 1,1 A 1,2 B 1,1 B 1,2 B 1,3 = B 2,1 B 2,2 B 2,3 ] [C 1,1 C 1,2 C 1,3. This gives P mat123 = A 1,1 B 1,1 C 1,1 + A 1,2 B 2,1 C 1,1 + A 1,1 B 1,2 C 1,2 + A 1,2 B 2,2 C 1,2 + A 1,1 B 1,3 C 1,3 + A 1,2 B 2,3 C 1,3. 24

25 Polynomial notation for (2, 3) (3, 1) Let s compute This gives A 1,1 A 1,2 A 1,3 A 2,1 A 2,2 A 2,3 C 1,1 C 2,1 C 3,1 = C 1,1. C 2,1 P mat231 = A 1,1 B 1,1 C 1,1 + A 1,2 B 2,1 C 1,1 + A 1,3 B 3,1 C 1,1 + A 2,1 B 1,1 C 2,1 + A 2,2 B 2,1 C 2,1 + A 2,3 B 3,1 C 2,1. 25

26 Comparison P mat123 = A 1,1 B 1,1 C 1,1 + A 1,2 B 2,1 C 1,1 + A 1,1 B 1,2 C 1,2 + A 1,2 B 2,2 C 1,2 + A 1,1 B 1,3 C 1,3 + A 1,2 B 2,3 C 1,3. Conclusion: P mat231 = A 1,1 B 1,1 C 1,1 + A 1,2 B 2,1 C 1,1 + A 1,3 B 3,1 C 1,1 + A 2,1 B 1,1 C 2,1 + A 2,2 B 2,1 C 2,1 + A 2,3 B 3,1 C 2,1. up to replacing A i,j by C j,i, B i,j by A i,j and C i,j by B j,i, these are the same polynomials so an algorithm for P mat231 gives an algorithm for P mat123 so 2, 3, 1 = 1, 2, 3 true in general 26

27 Approximate algorithms: an example with power series 27

28 Modular multiplication Reminder: to multiply two polynomials A, B modulo a polynomial P (with deg(p ) = d) compute C = AB return D = C rem P. Alternative solution when all the roots r 1,..., r d of P are known compute the values of A and B at r 1,..., r d return the polynomial E such that E(a i ) = A(r i )B(r i ) deg(e) < d Prop: D = E. 28

29 Remark: FFT multiplication Given A, B: evaluate A and B at roots of unity of high enough order n, by FFT multiply the values do an inverse FFT to get C = AB In general, this will return AB rem(x n 1). When n is large enough, there is no reduction, and we get AB. 29

30 Interpolation Prop. Given two different elements r 0, r 1 in k and two values v 0, v 1, the unique polynomial P such that P (r 0 ) = v 0, P (r 1 ) = v 1, deg(p ) < 2 is P = v 0 X r 1 r 0 r 1 + v 1 X r 0 r 1 r 0. Remark: similar formula with more points, but we won t need it. 30

31 Two similar situations 1. computing modulo P = X 2 (a 0 + a 1 X)(b 0 + b 1 X) mod X 2 = a 0 b 0 + (a 0 b 1 + a 1 b 0 )X naive: 3 multiplications no improvement possible over the naive algorithm 2. computing modulo P = X 2 1 = (X 1)(X + 1) (a 0 + a 1 X)(b 0 + b 1 X) mod (X 2 1) = a 0 b 0 + a 1 b 1 + (a 0 b 1 + a 1 b 0 )X naive: 4 multiplications Karatsuba: 3 multiplications evaluation / interpolation: 2 multiplications C = (a 0 + a 1 )(b 0 + b 1 ) X (a 0 a 1 )(b 0 b 1 ) X

32 More generally Computing modulo P = X 2 r 2 = (X r)(x + r): (a 0 + a 1 X)(b 0 + b 1 X) rem (X 2 r 2 ) = a 0 b 0 + r 2 a 1 b 1 + (a 0 b 1 + a 1 b 0 )X naive: 4 multiplications Karatsuba: 3 multiplications evaluation / interpolation: 2 multiplications C = (a 0 + a 1 r)(b 0 + b 1 r) X + r 2r (a 0 a 1 r)(b 0 b 1 r) X r 2r. 32

33 An approximate solution Let s suppose we can compute with real coefficients, and take r = 10 10, so we are computing modulo X We multiply ( X) and ( 4 + 5X), which gives ( X)( 4 + 5X) rem (X ) = X. This is (of course) very close to ( X)( 4 + 5X) rem X 2 = X. We can get the former using two multiplications, but the latter requires three. 33

34 Making this formal We introduce a new variable ε. Then, to compute (a 0 + a 1 X)(b 0 + b 1 X) rem X 2 we do the following: we compute C ε = (a 0 + a 1 X)(b 0 + b 1 X) rem (X 2 ε 2 ) = (a 0 b 0 + a 1 b 1 ε 2 ) + (a 0 b 1 + a 1 b 0 )X. We do this by evaluation / interpolation: we replace ε by 0. C ε = (a 0 + a 1 ε)(b 0 + b 1 ε) X + ε 2ε (a 0 a 1 ε)(b 0 b 1 ε) X ε. 2ε 34

35 Using the polynomial notation Multiplication modulo X 2 is represented by P = A 0 B 0 C 0 + A 0 B 1 C 1 + A 1 B 0 C 1. Multiplication modulo X 2 ε 2 is P ε = A 0 B 0 C 0 + ε 2 A 1 B 1 C 0 + A 0 B 1 C 1 + A 1 B 0 C 1 = P + ε 2 Q. The evaluation / interpolation algorithm shows P ε = (A 0 + A 1 ε)(b 0 + B 1 ε)(εc 0 + C 1 ) 2ε (A 0 A 1 ε)(b 0 B 1 ε)(εc 0 + C 1 ). 2ε 35

36 Summary We get (A 0 + A 1 ε)(b 0 + B 1 ε)(εc 0 + C 1 ) 2ε or (A 0 A 1 ε)(b 0 B 1 ε)(εc 0 + C 1 ) 2ε = P+ε 2 Q, (A 0 + A 1 ε)(b 0 + B 1 ε)(εc 0 + C 1 ) (A 0 A 1 ε)(b 0 B 1 ε)(εc 0 + C 1 ) 2 2 = εp+ε 3 Q. The left-hand side gives us an algorithm using 2 multiplications with coefficients involving ε that computes ε times what we want plus an higher-order term Q that we can dismiss afterwards. 36

37 Approximate algorithms for matrix multiplication 37

38 Multiplication of partial matrices We consider the product AB = C, with A = a 1,1 a 1,2 a 2,1 0, B = b 1,1 b 1,2 b 2,1 b 2,2, C = c 1,1 c 1,2 c 2,1 c 2,2. The naive algorithm uses 6 multiplications: P = A 1,1 B 1,1 C 1,1 + A 1,2 B 2,1 C 1,1 + A 1,1 B 1,2 C 1,2 + A 1,2 B 2,2 C 1,2 + A 2,1 B 1,1 C 2,1 + A 2,1 B 1,2 C 2,2. It is optimal. 38

39 The algorithm described by An approximate algorithm P ε = (A 1,2 + εa 1,1 )(B 1,2 + εb 2,2 )C 1,2 (A 2,1 + εa 1,1 )B 1,1 (C 1,1 + εc 2,1 ) A 1,2 B 1,2 (C 1,1 + C 1,2 + εc 2,2 ) A 2,1 (B 1,1 + B 1,2 + εb 2,1 )C 1,1 (A 1,2 + A 2,1 )(B 1,2 + εb 2,1 )(C 1,1 + εc 2,2 ) gives us P ε = εp + ε 2 Q, for some higher-order term Q. 39

40 Rectangular matrix multiplication We can use this trick to compute AB = C with a 1,1 a 1,2 A = a 2,1 a 2,2, B = b 1,1 b 1,2 b 2,1 b 2,2 a 3,1 a 3,2, C = c 1,1 c 1,2 c 2,1 c 2,2. This gives us an algorithm for 3, 2, 2 using 10 multiplications with coefficients involving ε that computes ε times what we want plus an higher-order term Q that we can dismiss afterwards. 40

41 Towards an algorithm for square matrices As we did for usual algorithms, we can apply permutations. This give algorithms for 3, 2, 2, 2, 3, 2 and 2, 2, 3 using 10 multiplications with coefficients involving ε that compute ε times what we want plus higher-order terms that we can dismiss afterwards. 41

42 Formalization An algorithm using k multiplications with coefficients involving ε that compute ε r 1 times m, n, p plus higher-order terms. is called an approximate algorithm of order r for m, n, p. Prop Given an approximate algorithm of order r for m, n, p, involving k multiplications, we can deduce a (normal) algorithm for m, n, p involving km(r) multiplications. Proof: all operations are done using power series addition and multiplication, modulo ε r. 42

43 Block products Prop. Suppose we have an algorithm of order r for m, n, p an algorithm of order r for m, n, p Then we can deduce an algorithm of order r + r 1 for mm, nn, pp. Proof. We are doing a product m, n, p of matrices of size m, n, p. 1. We recursively compute products of the form ε r 1 m, n p + ε r 2. We plug them into our approximate algorithm for m, n, p. This gives ε r 1 ε r 1 mm, nn, pp + ε r +r 1 43

44 Using block multiplication We have approximate algorithms of order 2 for 3, 2, 2, 2, 3, 2 and 2, 2, 3, with 10 multiplications. This gives us an approximate algorithm of order 4 for 12, 12, 12, with 1000 multiplications. and thus an approximate algorithm of order 7 for 12 2, 12 2, 12 2, with multiplications. and thus an approximate algorithm of order 10 for 12 3, 12 3, 12 3, with multiplications. and thus an approximate algorithm of order 3N + 1 for 12 N, 12 N, 12 N, with 1000 N multiplications. and thus an algorithm for 12 N, 12 N, 12 N, with 1000 N M(3N + 1) multiplications. 44

45 Conclusion Each of these algorithms gives us a value for ω: ω N = log(1000n M(3N+1)) log(12 N ) = log(1000n ) log(12 N ) + log(m(3n+1)) log(12 N ) so that ω N 3 log(10) log(12) = 3 log(10) log(12) + log(m(3n+1)) log(12 N ) 45

46 In practice i 12 i ratio vs. naive algo i 12 i ratio vs. Strassen

47 Swiss cheese matrices 47

48 Using sparsity We consider matrices with many zero entries (we ll call them motives). Theorem. Consider two motives A and B, of sizes (k, m) and (m, n). Suppose that you can compute A B using l products (possibly using approximate algorithms). Let f be the number of products in the naive product A B. Then we can take ω 3 log l log f. Simple examples. gives ω = 3 for the naive algorithm gives ω = 2.81 for Strassen. 48

49 Better examples 1. Back the previous example: and We had ω 3 log(10)/ log(12) 2.78 Now, l = 5, f = 6 = ω 3 log(5)/ log(6) The product of and can be done with l = 17, whereas f = 26 = ω 3 log(17)/ log(26)

50 Tensor powers We write A (s) and B (s) the sth tensor powers of A and B. On the previous example, A = a 1,1 a 1,2 a 2,1 0, the powers A (1), A (2) and A (3) are (I am not writing the actual coefficients) 50

51 Rows and columns Each column of A (s) is indexed by an s-uple µ of the form µ = (µ 1,..., µ s ), with 1 µ i m. Example. For s = 3, this gives: Same thing for the rows of B (s) ,1,2 51

52 Counting zeros Prop. Let k 1,..., k m be the numbers of non-zero entries in the columns 1,..., m in A. Let C µ be a column in A (s), with µ = (µ 1,..., µ s ). Then the number of non-zero entries in C µ is K(µ) = k µ1 k µs. If we let σ i be the number of µ i s that are equal to i, we have K(µ) = k σ 1 1 kσ m m. Example. µ = (2, 1, 2) = σ = (1, 2) = K(µ) = = 2. Same thing for the columns of B (s) ; we write N(µ) = n σ 1 1 nσ m m. 52

53 Cleaning We fix σ 1,..., σ m. There are M σ = s! σ 1! σ 2! σ m! choices µ = (µ 1,..., µ s ) for the columns of A (s). For all these columns, Same things for the rows of B (s), with K(µ) = K σ = k σ 1 1 kσ m m. N(µ) = N σ = n σ 1 1 nσ m m. We remove all other columns of A (s) and rows of B (s) et we call A σ and B σ what is left. 53

54 Cleaning Example: σ = (2, 1). We find the columns / rows (1, 1, 2), (1, 2, 1) and (2, 1, 1), with K σ = 4, M σ = 3, N σ = 8. The remaining A σ and B σ : and

55 From sparse to full Let U and V be any matrices of sizes (K σ, M σ ) and (M σ, N σ ). Prop. On can find G and Q such that U = GA σ, V = B σ Q. and thus UV = G(A σ B σ )Q. So If we can compute A B with l product, then K σ, M σ, N σ l s and thus log(l s ) ω 3 log(k σ M σ N σ ). 55

56 Remember that K σ M σ N σ = Asymptotically This is one of the terms in the expansion of s! σ 1! σ 2! σ m! (k 1n 1 ) σ1 (k m n m ) σ m. (k 1 n k m n m ) s. There are ( ) s + m 1 m 1 terms in the expansion. So, there is at least one choice of σ such that K σ M σ N σ (k 1n k m n m ) s ). ( s+m 1 m 1 56

57 Asymptotically This gives ( ) s + m 1 log(k σ M σ N σ ) s log(k 1 n k m n m ) log( ) m 1 or 1 log(k σ M σ N σ ) 1 s log(k 1 n k m n m ) log( ( ) s+m 1 ) Since ( ) s + m 1 m 1 we get (s + m 1) m 1 m 1 1 log(k σ M σ N σ ) 1 s log(k 1 n k m n m ) (m 1) log(s + m 1). 57

58 Conclusion Let f be the number of operations in the naive product A B. Then f = k 1 n k m n m. So we get With s : log(l s ) ω 3 s log f (m 1) log(s + m 1). ω 3 log l log f. 58