(Group-theoretic) Fast Matrix Multiplication

Transcription

1 (Group-theoretic) Fast Matrix Multiplication Ivo Hedtke Data Structures and Efficient Algorithms Group (Prof Dr M Müller-Hannemann) Martin-Luther-University Halle-Wittenberg Institute of Computer Science 30th of January, 2013 updated on 11th March, 2013 Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

2 1 Fast Matrix Multiplication An Overview 2 Group-theoretic Fast Matrix Multiplication An Introduction 3 Properties of the Triple Product Property and the Search for Triple Product Property Triples A Summary Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

3 Part1 Fast Matrix Multiplication An Overview Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

4 Naive Matrix Multiplication: Row Column b 1,1 b 1,j b 1,n b 2,1 b 2,j b 2,n b p,1 b p,j b p,n a 1,1 a 1,2 a 1,p a i,1 a i,2 a i,p a m,1 a m,2 a m,p c 1,1 c 1,j c 1,n c i,1 c i,j c i,n c m,1 c m,j c m,n A R m p, B R p n, C := AB R m n c i,j = (AB) i,j = p l=1 a i,l b l,j Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

5 Fast Fast complexity of other problems Fast Matrix Multiplication parallel comp Fast= Complexity sparse matrices Fast= Time small matrices asymptotic runtime shape/ form/ layout special types of matrices computer architecture Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

6 Naive Matrix Multiplication: Row Column for (int i = 0; i < N; i++) { for (int j = 0; j < M; j++) { aux = 00; for (int k = 0; k < P; k++) { aux += A[i][k]*B[k][j]; } Result[i][j] = aux; } a 1,1 a 1,2 a 1,p } a i,1 a i,2 a i,p a m,1 a m,2 a m,p b 1,1 b 1,j b 1,n b 2,1 b 2,j b 2,n b p,1 b p,j b p,n c 1,1 c 1,j c 1,n c i,1 c i,j c i,n c m,1 c m,j c m,n A R m p, B R p n, C := AB R m n c i,j = (AB) i,j = p a i,l b l,j l=1 Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

7 Folgendes Diagramm konnte ich aus den Daten erstellen: Naive Matrix Multiplication: Performance scalar additions n 3 n 2 scalar multiplications n 3 2n 3 n 2 FLOPs MFLOPS FLOP = Floating Point Operation FLOPs = Floating Point Operations FLOPS = Floating Point Operations per Second N Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

8 RAM Model (random-access machine) 1 Instructions are executed one after another, with no concurrent operations Every instruction takes the same amount of time, at least up to small constant factors Unbounded amount of available memory Memory stores words of size O(logn) bits where n is the input size Any desired memory location can be accessed in unit time For numerical and geometric algorithms, it is sometimes also assumed that words can be represent real numbers accurately Exact arithmetic on arbitrary real numbers can be done in constant time 1 [D Ajwani and H Meyerhenke: Realistic Computer Models In: M Müller-Hannemann and S Schirra: Algorithm Engineering Bridging the Gap between Algorithm Theory and Practice, LNCS 5971] Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

9 Real Architecture 2 < 1 KB Registers 1 ns Size < 256 MB Caches 10 ns Speed < 8 GB Main Memory 5-70 ns > 20 GB Hard Disk 10 ms 2 [D Ajwani and H Meyerhenke: Realistic Computer Models In: M Müller-Hannemann and S Schirra: Algorithm Engineering Bridging the Gap between Algorithm Theory and Practice, LNCS 5971] Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

10 Loop Unrolling & Tiling saxpy = Single-precision real Alpha X Plus Y: α x + y (BLAS Level 1) DO 50 I = MP1,N,4 SY(I) = SY(I) + SA*SX(I) SY(I+1) = SY(I+1) + SA*SX(I+1) SY(I+2) = SY(I+2) + SA*SX(I+2) SY(I+3) = SY(I+3) + SA*SX(I+3) 50 CONTINUE C uses row-major storage double uses 64 bit per number assume our cache block size is 128 byte cache block size of 16 numbers Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

11 ATLAS = Automatically Tuned Linear Algebra Software make[5]: Leaving directory /tune/blas/gemm SUCCESSFUL FINISH FOR /xummsearch BEST USER CASE 241, NB=56: MFLOP hedtke$ /Test -O TIME TEST FOR METHODS OF MATRIX MULTIPLICATION C = A*(8A), where A is a (n x n) random matrix with n = method time (sec) NaivStandard 156 MFLOPS NaivOnArray NaivLoopUnrollingFour StrassenNaiv 19GB RAM WinogradOriginal BLAS 8 GFLOPS Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

12 Strassen s algorithm 3 [ ] a11 a 12 A = a 21 a 22 [ ] b11 b 12 B = b 21 b 22 [ ] h1 + h 4 h 5 + h 7 h 3 + h 5 AB = h 2 + h 4 h 1 + h 3 h 2 + h 6 wikimediaorg/ wikipedia/commons/e/ ef/strassen_knuth_ Prize_lecturejpg h 1 := (a 11 + a 22 )(b 11 + b 22 ) h 2 := (a 21 + a 22 )b 11 h 3 := a 11 (b 12 b 22 ) h 4 := a 22 (b 21 b 11 ) h 5 := (a 11 + a 12 )b 22 h 6 := (a 21 a 11 )(b 11 + b 12 ) h 7 := (a 12 a 22 )(b 21 + b 22 ) 3 [Volker Strassen: Gaussian Elimination is not Optimal, Numer Math 13 (1969), ] Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

13 although originally it appeared in the equivalent trilinear version and was applied The Exponent of Matrix Multiplication to the evaluation of one rather than two matrix products We will end this section by representing a diagram of the progress in the reduction of the exponent M(n) := number of field operations in characteristic 0 required to multiply Diagram (~(~) is the best exponent announced by time ~ ) two n n matrices ω := inf { r R : M(n) = O(n r ) } 3,0" 2,8- coco ~-[ST69] -"t oj (r) [P80] I 'I [P791,-q ~ "~z~- [S80] [R82] / [(3Wl (P80b]" ' I I I I I I I I T ~ t t 1 [Victor Pan: How to Multiply Matrices Faster, LNCS 179, 1984] Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

14 Coppersmith, Winograd and Williams ω Don Coppersmith and Shmuel Winograd 4 ω Virginia Vassilevska Williams [Don Coppersmith and Shmuel Winograd, Matrix multiplication via arithmetic progressions, J Symbolic Comput 9 (1990), ] 5 [Virginia Vassilevska Williams, Multiplying matrices faster than Coppersmith-Winograd, Proceedings of the 44th Symposium on Theory of Computing, STOC 12] Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

15 Group-theoretic Fast Matrix Multiplication An Introduction Part2 Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

16 Cohn and Umans Henry Cohn and Christopher Umans 6 6 [Henry Cohn and Christopher Umans, A Group-theoretic Approach to Fast Matrix Multiplication, Proceedings of the 44th Annual Symposium on Foundations of Computer Science, October 2003, Cambridge, MA, IEEE Computer Society (2003), ] Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

17 basics: algebraic complexity theory n,p,m := n,p,m C n,p,m k : k n p k p m k n m, (A,B) AB bilinear complexity field k, fin dim k-vs U, V and W, k-bil map η: U V W k-bil algo: 1 i r: f i U, g i V, w i W : η(u,v) = r i=1 f i(u)g i (v)w i u U,v V (f 1,g 1,w 1 ;;f r,g r,w r ) is called k-bil algo of length r for η R(η) := min length of all k-bil algo for η k-algebra A: R(A) := rank of its k-bil multiplication map restriction of a bilinear map bil maps φ: U V W, φ : U V W restriction: linear maps σ: U U, τ: V V, ζ : W W : φ = ζ φ (σ τ) we write φ φ if restriction of φ to φ Fact: If φ φ, then R(φ) R(φ ) Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

18 TPP basics n,p,m := n,p,m C, n,p,m k : k n p k p m k n m, (A,B) AB R(η) rank of bilinear algorithm R(A) rank of algebra R(n,p,m) := R( n,p,m ) R(n) := R(n,n,n) R(G) : R(C[G]) (AB) s,u = ( )( t T A s,tb t,u s S,t T A s,ts 1 t Q(X) = {xy 1 : x,y X} TPP: S,T,U G fulfill the TPP: s Q(S), t Q(T), u Q(u) stu = 1 s = t = u = 1 ˆt T,u U Bˆt,uˆt 1 u G realizes n,p,m : S,T,U G, S = n, T = p, U = m and S, T, U fulfill the TPP In this case we call (S,T,U) a TPP triple of G Its size is npm TPP (subset) capacity: β(g) = max{npm : G realizes n, p, m } {d i } character degrees of G r-character capacity: D r (G) = i d r i ) Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

19 Part3 Properties of the Triple Product Property and the Search for Triple Product Property Triples A Summary Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

20 Properties of the TPP / TPP triples If S, T and U fulfill the TPP then [Murthy, arxiv: ] S + T + U G + 2 [Hedtke, arxiv: v2] Q(X) Q(Y ) = 1 X Y {S,T,U} [Hedtke, arxiv: v2] Q(S) + Q(T) + Q(U) G + 2 [Hedtke, arxiv: v2] If S, T and U fulfill the TPP, then there exists a triple S, T and U with S = S, T = T, U = U and S T = T U = S U = 1 which also fulfills the TPP Let G be a group If (S,T,U) is a TPP triple of G, then (dsa,dtb,duc) is a TPP triple for all a,b,c,d G, too basic TPP triple: 1 S T U academics/other-fellows/ emeritus-fellows/peter-neumann/ [P M Neumann, A note on the triple product property for subsets of finite groups, LMS J Comput Math 14 (2011), ] Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

21 New Characterizations of the TPP [me & S Murthy, Search and test algorithms for triple product property triples, Groups Compl Crypt, Vol 4 Issue 1 (2012), p ] Theorem Three subsets S, T and U of G form a TPP triple (S,T,U) iff (i) 1 S T U, (ii) Q(T) Q(U) = 1 and (iii) Q(S) Q(T)Q(U) = 1 more general: (S 1,S 2,S 3 ) is a TPP triple if and only if for all π Sym(3) 1 S 1 S 2 S 3, Q(S π2 ) Q(S π3 ) = 1, and Q(S π1 ) Q(S π2 )Q(S π3 ) = 1 Theorem Let G be a group and (S,T,U) a basic TPP triple of subsets of G such that either (i) two members, say S and T, are subgroups of G which permute, or (ii) one member, say S, is a normal subgroup of G Then S T U G Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

22 New Characterizations of the TPP Let C be a finite set and C = {C 1,,C k } a partition of it A set X C is called a subtransversal for C with support supp C (X) = T C if for all C i C X C i = { 1 Ci T, 0 otherwise special case: C = set of cosets of a subgroup S of a group G, then any subtransversal T for S \ G will be called a subtransversal for S in G Theorem Let G be a group, S a subgroup of G, and T, U subsets of G 1 If (S,T,U) is a TPP triple of G then T and U are subtransversals for S in G such that supp S\G (T) supp S\G (U) = {S} (*) 2 If T and U are also subgroups of G, and T and U are subtransversals for S in G satisfying (*) then (S,T,U) is a TPP triple of G Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

23 Search for TPP triples Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

24 Theoretical insights obtained from experiments Conjecture 74 For any group G, β g (G) D 3 (G) Conjecture 75 Let D 2n denote the dihedral group of order 2n (i) If n is a multiple of 3, then β(d 2n ) = β g (D 2n ) = 8 3 n (ii) If n is not a multiple of 3, then β(d 2n ) < 8 3 n and β g(d 2n ) = 2n Conjecture 76 If G is a group with a cyclic subgroup of index 2, then β(g) 4 3 G Theorem There is no group G that realizes 3,3,3 such that R(G) < 23, or that realizes 4,4,4 such that R(G) < 49 Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25

25 5 5 matrices (techn rep in preparation) Theorem There is no group G that realizes 5,5,5 such that R(G) < 100 Suppose G realizes 5,5,5 If G has a subgroup H of index 2, then H realizes 3, 3, 3 If G realizes 5,5,5 and G 72, then G has no abelian subgroups of index 2 If G is group with R(G) < 100 that realizes 5,5,5, then G is non-abelian and 45 G 72 No group G of order 64 fulfills R(G) < 100 and realizes 5,5,5 Search candidates: The final list contains ten groups of order 48 and two of order 54 better search algorithm for k,k,k TPP triples Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, / 25