Fast Matrix Product Algorithms: From Theory To Practice

Transcription

1 Introduction and Definitions The τ-theorem Pan s aggregation tables and the τ-theorem Software Implementation Conclusion Fast Matrix Product Algorithms: From Theory To Practice Thomas Sibut-Pinote Inria, École Polytechnique, France Éric Schost University of Waterloo, ON,Canada Specfun Seminar February 8th, /22

2 Motivation Complexity of matrix product complexity of linear algebra; ω = inf { θ it takes n θ operations to multiply in M n (K) } [2, 3]; Strassen 69 : ω < 2.81 (used in practice); Le Gall 14 : ω < (theoretical). Malgré leurs performances asymptotiques, aucun des algorithmes de cette section ne semble devoir être implémenté sur machine dans un proche avenir Abdeljaoued & Lombardi, /22

3 Motivation Complexity of matrix product complexity of linear algebra; ω = inf { θ it takes n θ operations to multiply in M n (K) } [2, 3]; Strassen 69 : ω < 2.81 (used in practice); Le Gall 14 : ω < (theoretical). Malgré leurs performances asymptotiques, aucun des algorithmes de cette section ne semble devoir être implémenté sur machine dans un proche avenir Abdeljaoued & Lombardi, 2003 Can we bridge the gap a little? 2/22

4 Problem Statement Let m, n, p denote the bilinear map: M m,n (K) M n,p (K) M m,p (K) (A, B) A B. Goal: determine the arithmetic complexity of m, n, p. 3/22

5 Problem Statement Let m, n, p denote the bilinear map: M m,n (K) M n,p (K) M m,p (K) (A, B) A B. Goal: determine the arithmetic complexity of m, n, p. Known: naive algorithm in mnp operations: n i 1, m, j 1, p, [AB] i,j = a i,k b k,j. k=1 3/22

6 Problem Statement Let m, n, p denote the bilinear map: M m,n (K) M n,p (K) M m,p (K) (A, B) A B. Goal: determine the arithmetic complexity of m, n, p. Known: naive algorithm in mnp operations: i 1, m, j 1, p, [AB] i,j = Can we do better? n a i,k b k,j. k=1 3/22

7 Strassen s Algorithm Strassen s algorithm: 2, 2, 2 in 7 multiplications (instead of = 8): α 1 = (a 1,2 a 2,2 ), β 1 = (b 2,1 + b 2,2 ), p 1 = α 1 β 1 α 2 = (a 2,1 a 1,1 ), β 2 = (b 1,2 + b 1,1 ), p 2 = α 2 β 2 α 3 = a 1,1, β 3 = (b 1,2 b 2,2 ), p 3 = α 3 β 3 α 4 = a 2,2, β 4 = (b 2,1 b 1,1 ), p 4 = α 4 β 4 α 5 = (a 2,1 + a 2,2 ), β 5 = b 1,1, p 5 = α 5 β 5 α 6 = (a 1,2 + a 1,1 ), β 6 = b 2,2, p 6 = α 6 β 6 α 7 = (a 1,1 + a 2,2 ), β 7 = (b 1,1 + b 2,2 ), p 7 = α 7 β 7 4/22

8 Strassen s Algorithm Strassen s algorithm: 2, 2, 2 in 7 multiplications (instead of = 8): α 1 = (a 1,2 a 2,2 ), β 1 = (b 2,1 + b 2,2 ), p 1 = α 1 β 1 α 2 = (a 2,1 a 1,1 ), β 2 = (b 1,2 + b 1,1 ), p 2 = α 2 β 2 α 3 = a 1,1, β 3 = (b 1,2 b 2,2 ), p 3 = α 3 β 3 α 4 = a 2,2, β 4 = (b 2,1 b 1,1 ), p 4 = α 4 β 4 α 5 = (a 2,1 + a 2,2 ), β 5 = b 1,1, p 5 = α 5 β 5 α 6 = (a 1,2 + a 1,1 ), β 6 = b 2,2, p 6 = α 6 β 6 α 7 = (a 1,1 + a 2,2 ), β 7 = (b 1,1 + b 2,2 ), p 7 = α 7 β 7 c 1,1 = p 1 + p 4 p 6 c 1,2 = p 4 + p 5 c 2,1 = p 3 + p 6 c 2,2 = p 2 + p 3 p 5 + p 7 ( ) c1,1 c C = 1,2 c 2,1 c 2,2 4/22

9 Strassen s Algorithm Strassen s algorithm: 2, 2, 2 in 7 multiplications (instead of = 8): α 1 = (a 1,2 a 2,2 ), β 1 = (b 2,1 + b 2,2 ), p 1 = α 1 β 1 c 1,1 = p 1 + p 4 p 6 α 2 = (a 2,1 a 1,1 ), β 2 = (b 1,2 + b 1,1 ), p 2 = α 2 β 2 c 1,2 = p 4 + p 5 α 3 = a 1,1, β 3 = (b 1,2 b 2,2 ), p 3 = α 3 β 3 c 2,1 = p 3 + p 6 α 4 = a 2,2, β 4 = (b 2,1 b 1,1 ), p 4 = α 4 β 4 α 5 = (a 2,1 + a 2,2 ), β 5 = b 1,1, p 5 = α 5 β 5 c 2,2 = p 2 + p 3 p 5 + p 7 ( ) α 6 = (a 1,2 + a 1,1 ), β 6 = b 2,2, p 6 = α 6 β 6 c1,1 c C = 1,2 α 7 = (a 1,1 + a 2,2 ), β 7 = (b 1,1 + b 2,2 ), p 7 = α 7 β 7 c 2,1 c 2,2 Observe: C = p 1 γ 1 + p 2 γ 2 + p 3 γ 3 + p 4 γ 4 + p 5 γ 5 + p 6 γ 6 + p 7 γ 7. where γ 1 = E 1,1, γ 2 = E 2,2, γ 3 = E 2,1 + E 2,2, γ 4 = E 1,1 + E 1,2, γ 5 = E 1,2 E 2,2, γ 6 = E 2,1 E 2,2, γ 7 = E 2,2 E i,j canonical basis 4/22

10 Strassen s Algorithm Strassen s algorithm: 2, 2, 2 in 7 multiplications (instead of = 8): α 1 = (a 1,2 a 2,2 ), β 1 = (b 2,1 + b 2,2 ), p 1 = α 1 β 1 c 1,1 = p 1 + p 4 p 6 α 2 = (a 2,1 a 1,1 ), β 2 = (b 1,2 + b 1,1 ), p 2 = α 2 β 2 c 1,2 = p 4 + p 5 α 3 = a 1,1, β 3 = (b 1,2 b 2,2 ), p 3 = α 3 β 3 c 2,1 = p 3 + p 6 α 4 = a 2,2, β 4 = (b 2,1 b 1,1 ), p 4 = α 4 β 4 α 5 = (a 2,1 + a 2,2 ), β 5 = b 1,1, p 5 = α 5 β 5 c 2,2 = p 2 + p 3 p 5 + p 7 ( ) α 6 = (a 1,2 + a 1,1 ), β 6 = b 2,2, p 6 = α 6 β 6 c1,1 c C = 1,2 α 7 = (a 1,1 + a 2,2 ), β 7 = (b 1,1 + b 2,2 ), p 7 = α 7 β 7 c 2,1 c 2,2 Observe: C = p 1 γ 1 + p 2 γ 2 + p 3 γ 3 + p 4 γ 4 + p 5 γ 5 + p 6 γ 6 + p 7 γ 7. where γ 1 = E 1,1, γ 2 = E 2,2, γ 3 = E 2,1 + E 2,2, γ 4 = E 1,1 + E 1,2, γ 5 = E 1,2 E 2,2, γ 6 = E 2,1 E 2,2, γ 7 = E 2,2 E i,j canonical basis 7 Tensor notation: α i β i γ i. 4/22

11 Tensors and algorithms General tensor notation identified with a bilinear map: m p n m, n, p = a i,k b k,j c i,j. j=1 k=1 Representing m, n, p as r α i β i γ i gives an algorithm. Example: The elementary tensor (a 1,2 + a 3,5 ) b 2,4 (c 1,4 + c 2,4 ) reads as the algorithm tmp (a 1,2 + a 3,5 ) b 2,4 c 1,4 tmp c 2,4 tmp 5/22

12 Composition t t : computes the composition of two tensors. To multiply A of size (mm, nn ) by B of size (nn, pp ), decompose A and B into blocks: A 1,1 A 1,n B 1,1 B 1,p A =.., B =.. A m,1 A m,n B n,1 B n,p where A i,j of size (m, n ), B j,k of size (n, p ). If t = m, n, p and t = m, n, p : t t mm, nn, pp. 6/22

13 Composition t t : computes the composition of two tensors. To multiply A of size (mm, nn ) by B of size (nn, pp ), decompose A and B into blocks: A 1,1 A 1,n B 1,1 B 1,p A =.., B =.. A m,1 A m,n B n,1 B n,p where A i,j of size (m, n ), B j,k of size (n, p ). If t = m, n, p and t = m, n, p : t t mm, nn, pp. Also set t k = t t t m }{{} k, n k, p k. k times 6/22

14 Direct Sum of Tensors t t : computes two independent matrix products in parallel. We will denote s t for t t t. }{{} s times 7/22

15 Rank and ω Definition (Rank of a Tensor t) { R(t) := min r t can be written as } r x i y i z i 8/22

16 Rank and ω Definition (Rank of a Tensor t) { R(t) := min r t can be written as } r x i y i z i R( m, n, p ) is the minimal number of multiplications for m, n, p. 8/22

17 Rank and ω Definition (Rank of a Tensor t) { R(t) := min r t can be written as } r x i y i z i R( m, n, p ) is the minimal number of multiplications for m, n, p. Definition (Linear Algebra Exponent) ω := inf{τ There exists an algorithm to multiply n n matrices in O(n τ ) additions and multiplications}( [2, 3]) 8/22

18 Rank and ω Definition (Rank of a Tensor t) { R(t) := min r t can be written as } r x i y i z i R( m, n, p ) is the minimal number of multiplications for m, n, p. Definition (Linear Algebra Exponent) ω := inf{τ There exists an algorithm to multiply n n matrices in O(n τ ) additions and multiplications}( [2, 3]) Theorem inf{τ R( n, n, n ) = O(n τ )} = ω 8/22

19 Back to Strassen s Algorithm Theorem (Strassen 69) R ( 2, 2, 2 ) 7, hence ω log 2 (7) Idea: R ( 2 k, 2 k, 2 k ) 7 k by induction on k. Cut into blocks of size 2 k 1 and proceed recursively. 9/22

20 Back to Strassen s Algorithm Theorem (Strassen 69) R ( 2, 2, 2 ) 7, hence ω log 2 (7) Idea: R ( 2 k, 2 k, 2 k ) 7 k by induction on k. Cut into blocks of size 2 k 1 and proceed recursively. Lemma R ( m, n, p ) r R ( mnp, mnp, mnp ) r 3. Idea: If we can do m, n, p in r operations, then we can obtain n, p, m and p, m, n in r operations. Then we compose them. 9/22

21 Back to Strassen s Algorithm Theorem (Strassen 69) R ( 2, 2, 2 ) 7, hence ω log 2 (7) Idea: R ( 2 k, 2 k, 2 k ) 7 k by induction on k. Cut into blocks of size 2 k 1 and proceed recursively. Lemma R ( m, n, p ) r R ( mnp, mnp, mnp ) r 3. Idea: If we can do m, n, p in r operations, then we can obtain n, p, m and p, m, n in r operations. Then we compose them. Theorem R ( m, n, p ) r ω 3 log(r) log(mnp). R ( mnp, mnp, mnp ) r 3 ; Proceed recursively for (mnp) k, (mnp) k, (mnp) k just like for the 2, 2, 2 case. 9/22

22 Bini s Approximate Algorithms ( 79) Idea: K K[ε] Definition (degenerate rank of a tensor t) r R(t) := min{r t(ε), t(ε) = u i (ε) v i (ε) w i (ε) with t(ε) = ε q 1 t + ε q t 1 (ε) and q > 0}. Algorithmically, one can obtain t by computing t(ε) modulo ε q. 10/22

23 Bini s Approximate Algorithms ( 79) Idea: K K[ε] Definition (degenerate rank of a tensor t) R(t) := min{r t(ε), t(ε) = r u i (ε) v i (ε) w i (ε) with t(ε) = ε q 1 t + ε q t 1 (ε) and q > 0}. Algorithmically, one can obtain t by computing t(ε) modulo ε q. Theorem (Bini 79) Consequence: ω < R ( m, n, p ) r ω 3 log(r) log(mnp) 10/22

24 The τ-theorem Theorem (τ-theorem, Schönhage 81) If and then ( s ) R m i, n i, p i r, s (m i n i p i ) β = r, ω 3 β. Consequence (Schönhage again): ω < Crucial for recent records (including Le Gall 14: ω < ) 11/22

25 Towards a Practical Use of the τ-theorem Theoretical Obstacles The τ-theorem gives great bounds on ω but it is not seen as a way to build concrete matrix product algorithms (non-effective proofs). Degenerate rank rank relies on the fact that computing with polynomials is asymptotically negligible compared with scalars. Theoretical Contributions More constructive proof of the τ-theorem (an algorithm). Get rid of ε and use the τ-theorem constructively! (for specific kinds of tensors) 12/22

26 Sketch of the constructive proof ( s m i, n i, p i ) k 13/22

27 Sketch of the constructive proof ( s k m i, n i, p i ) µ=(µ 1,,µ s ) µ 1+ +µ s =k (several matrix products (M, N, P)) 13/22

28 Sketch of the constructive proof ( s k m i, n i, p i ) µ=(µ 1,,µ s ) µ 1+ +µ s =k ( k µ 1,, µ s ) s m µ i i }{{} M, s n µ i i }{{} N, s p µ i i }{{} } {{ } ( µ) k matrix products (M, N, P) in parallel P 13/22

29 Sketch of the constructive proof ( s k m i, n i, p i ) µ=(µ 1,,µ s ) µ 1+ +µ s =k Suppose t(ε) is a degeneration of ( k µ 1,, µ s ) s m µ i i }{{} M, s n µ i i }{{} N, s p µ i i }{{} } {{ } ( µ) k matrix products (M, N, P) in parallel s m i, n i, p i. In the same way, t(ε) k µ t µ (ε). P 13/22

30 Sketch of the constructive proof ( s k m i, n i, p i ) µ=(µ 1,,µ s ) µ 1+ +µ s =k Suppose t(ε) is a degeneration of ( k µ 1,, µ s ) s m µ i i }{{} M, s n µ i i }{{} N, s p µ i i }{{} } {{ } ( µ) k matrix products (M, N, P) in parallel s m i, n i, p i. In the same way, t(ε) k µ t µ (ε). P 1 Choose one specific t µ (ε) we can do ( k µ) M, N, P matrix products in parallel effectively (with ε s). 2 Compute M l, N l, P l = M l 1, N l 1, P l 1 M, N, P recursively like previously, using t µ to gain operations at each stage. 13/22

31 Visualizing the exponent in the proof ( 2, 1, 2 1, 3, 1 ) 2 = 4, 1, 4 2 2, 3, 2 1, 9, 1 Figure : Direct Sum, iterated once, of two matrix products 2, 1, 2 and 1, 3, 1 14/22

32 Pan s aggregation tables ( 84) Builds a family of tensors computing independent matrix products to improve ω: Input: table with various tensors. Example: x i,0 y 0,k ε 2 z k,i m, 1, p m 1 p 1 i=0 k=0 εu 0,k,i εv k,i,0 w 0,0 1, (m 1)(p 1), 1 m 1 p 1 i=0 k=0 Every row gives a matrix product (actually, some variables to adjust); Aggregate terms by summing over columns, m 1 p 1 here: t = (x i,0 + εu 0,k,i ) (y 0,k + εv k,i,0 ) (ε 2 z k,i + w 0,0 ). i=0 k=0 t = ε 2 ( m, 1, p 1, (m 1)(p 1), 1 ) + t 2 15/22

33 Correction term m 1 p 1 t = (x i,0 + εu 0,k,i ) (y 0,k + εv k,i,0 ) (ε 2 z k,i + w 0,0 ) i=0 k=0 To apply the τ-theorem we want: t = ε 2 ( m, 1, p 1, (m 1)(p 1), 1 ) + terms of higher degree in ε Let us remove terms of degree 0 and 1, hence the corrected term: t 1 = t We get the output: ( m 1 ) x i,0 ( p 1 i=0 k=0 ) y 0,k w 0,0. t 1 = ε 2 ( m, 1, p 1, (m 1)(p 1), 1 ) + ε 3 t 2 Hence R( m, 1, p 1, (m 1)(p 1), 1 ) mp + 1. Consequence: ω < 2.55 with m = 4,p = 4. 16/22

34 Combined use with the τ-theorem Every matrix variable appears with the same degree in ε: homogenous tensor. Theorem (S,S-P 12) Let t be a homogenous tensor. If we apply the algorithm of the constructive proof of the τ-theorem to t, for any µ and k > 1, the resulting tensor t µ (ε) can be written as where t 1 does not contain any ε. Consequence t µ (ε) = ε q t 1, Set ε = 1 in t µ (ε): get an ε-free tensor computing disjoint matrix products. Even better: set ε = 1 in t(ε) before extracting t µ from t(ε) k. We can get rid of the ε while still benefiting from the τ-theorem! 17/22

35 Example Example: 2 4, 9, 4 in 243 multiplications (instead of 2 (4 9 4) = 288) with: m 1 p 1 t 1 = (x i,0 + εu 0,k,i ) (y 0,k + εv k,i,0 ) (ε 2 z k,i + w 0,0 ) i=0 k=0 ( m 1 ) x i,0 i=0 ( p 1 ) y 0,k w 0,0. k=0 with m = p = 4, k = 2 and µ = (1, 1) in the τ-theorem. This gives an ω-equivalent of /22

36 Example Example: 2 4, 9, 4 in 243 multiplications (instead of 2 (4 9 4) = 288) with: m 1 p 1 t 1 = (x i,0 + εu 0,k,i ) (y 0,k + εv k,i,0 ) (ε 2 z k,i + w 0,0 ) i=0 k=0 ( m 1 ) x i,0 i=0 ( p 1 ) y 0,k w 0,0. k=0 with m = p = 4, k = 2 and µ = (1, 1) in the τ-theorem. This gives an ω-equivalent of Better, with the same tensor: µ = (4, 2), k = 6, m = p = 4: , 81, 256 matrix products in multiplications, ω-equivalent /22

37 Example Example: 2 4, 9, 4 in 243 multiplications (instead of 2 (4 9 4) = 288) with: m 1 p 1 t 1 = (x i,0 + εu 0,k,i ) (y 0,k + εv k,i,0 ) (ε 2 z k,i + w 0,0 ) i=0 k=0 ( m 1 ) x i,0 i=0 ( p 1 ) y 0,k w 0,0. k=0 with m = p = 4, k = 2 and µ = (1, 1) in the τ-theorem. This gives an ω-equivalent of Better, with the same tensor: µ = (4, 2), k = 6, m = p = 4: , 81, 256 matrix products in multiplications, ω-equivalent Even better, not built explicitly: µ = (10, 5), ω-equivalent /22

38 Software implementation in OCaml Parse degenerate tensors as Pan-style aggregation tables; Compose tensors symbolically; Extract a given coefficient µ m µ i i τ-theorem;, n µ i i Test of tensors by applying them to random matrices;, p µ i i following the Maple code generation which computes the rank of a subterm of a power of tensor without actually computing it; C++ code generation implementing a given tensor. 19/22

39 Specifics of doing this in OCaml Static typing much helpful; Caveat: some algebraic computations had to be recoded; Symbolic computations on algorithms akin to compilation passes: AST manipulation; Some interaction with Maple : generating code to do some computations; Parametricity: Export possible to Latex, C++, Maple. 20/22

40 How to Use this Result and Implementation, Future Work Roadmap of use Try out new or modified Pan Tables extract good algorithms; Optimize corresponding code as much as possible (cache, other algorithms at leaves,...). Future work Finish trying out all Pan tables. This work showed improvements in ω are not purely theoretical results. Adapt other theoretical improvements to build concrete tensors? For instance, the laser method? 21/22

41 Thank you for your attention! Any questions? 22/22