Solving Sparse Integer Linear Systems. Pascal Giorgi. Dali team - University of Perpignan (France)

Transcription

1 Solving Sarse Integer Linear Systems Pascal Giorgi Dali team - University of Perignan (France) in collaboration with A. Storjohann, M. Giesbrecht (University of Waterloo), W. Eberly (University of Calgary), G. Villard (ENS Lyon) LMC - MOSAIC Seminar, June 2, 26

2 Motivations Large linear systems are involved in many mathematical alications over a eld : I integers factorization [Odlyzko 999], I discrete logarithm [Odlyzko 999 ; Thome 23], over the integers : I number theory [Cohen 993], I grou theory [Newman 972], I integer rogramming [Aardal, Hurkens, Lenstra 999]

3 Problem Let A a non-singular matrix and b a vector dened over Z. Problem : Comute x = A b over the rational numbers A = x = A b = C A, b = C A C A. Main diculty : exression swell

4 Problem Let A a non-singular matrix and b a vector dened over Z. Problem : Comute x = A b over the rational numbers A = x = A b = C A C A, b = C A. Main diculty : exression swell and take advantage of sarsity

5 Interest in linear algebra Integer linear systems are central in recent linear algebra algorithms I Determinant [Abbott, Bronstein, Mulders 999 ; Storjohann 25] I Smith Form [Eberly, Giesbrecht, Villard 2] I Nullsace, Kernel [Chen, Storjohann 25] I Diohantine solutions [Giesbrecht 997 ; Giesbrecht, Lobo, Saunders 998 ; Mulders, Storjohann 23 ; Mulders 24]

6 Algorithms for non-singular system solving Dense matrices : I Gaussian elimination and CRA,! O~(n!+ log jjajj) bit oerations I P-adic lifting [Monck, Carter 979 ; Dixon 982],! O~(n 3 log jjajj) bit oerations I High order lifting [Storjohann 25],! O~(n! log jjajj) bit oerations Sarse matrices : I P-adic lifting or CRA [Wiedemann 986 ; Kaltofen, Saunders 99],! O(n 2 (log(n) + log jjajj)) bit oerations with non-zero elts.

7 P-adic algorithm with matrix inversion Scheme to comute A b : (-) B := A mod (-2) r := b for i := to k (2-) x i := B:r mod (2-2) r := (=)(r A:x i ) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

8 P-adic algorithm with matrix inversion Scheme to comute A b : (-) B := A mod O~(n 3 log jjajj) (-2) r := b for i := to k k = O~(n) (2-) x i := B:r mod O~(n 2 log jjajj) (2-2) r := (=)(r A:x i ) O~(n 2 log jjajj) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

9 P-adic algorithm with matrix inversion Scheme to comute A b : (-) B := A mod O~(n 3 log jjajj) (-2) r := b for i := to k k = O~(n) (2-) x i := B:r mod O~(n 2 log jjajj) (2-2) r := (=)(r A:x i ) O~(n 2 log jjajj) (3-) x := P k i= x i : i (3-2) rational reconstruction on x Main oerations : matrix inversion and matrix-vector roducts

10 Dense linear system in ractice Ecient imlementations are available : LinBox. [ IML library [ Details : level 3 BLAS-based matrix inversion over rime eld with LQUP factorization [Dumas, Giorgi, Pernet 24] with Echelon form [Chen, Storjohann 25] level 2 BLAS-based matrix-vector roduct use of CRT over the integers rational number reconstruction half GCD [Schonage 97] heuristic using integer multilication [NTL library]

11 Timing for dense linear system solving use of LinBox library on Pentium 4-3.4Ghz, 2Go RAM random dense linear system with coecients over 3 bits : n time :6s 4:3s 3:s 99:6s 236:8s 449:2s random dense linear system with coecients over 2 bits : n time :8s 2:9s 9:5s 299:7s 76:4s MT erformances imrovement by a factor comare to NTL's tuned imlementation

12 what does haen when matrices are sarse? we consider sarse matrices with O(n) non zero elements,! matrix-vector roduct needs only O(n) oerations.

13 Scheme to comute A b : (-) B := A mod certainly dense (-2) r := b for i := to k (2-) x i := B:r mod dense roduct (2-2) r := (=)(r A:x i ) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

14 Sarse linear system and P-adic lifting P-adic lifting doesn't imrove comlexity as in dense case.,! comuting the modular inverse is roscribed due to ll-in Solution [Wiedemann 986 ; Kaltofen, Saunders 99] :,! use modular minimal olynomial instead of inverse

15 Sarse linear system and P-adic lifting P-adic lifting doesn't imrove comlexity as in dense case.,! comuting the modular inverse is roscribed due to ll-in Solution [Wiedemann 986 ; Kaltofen, Saunders 99] :,! use modular minimal olynomial instead of inverse Let A 2 Z n n of full rank and b 2 Z n. Then x = A b can be exressed as a linear combination of the Krylov subsace fb; Ab; :::; A n bg Let () = c + c + ::: + d 2 Z [] be the minimal olynomial of A

16 Sarse linear system and P-adic lifting P-adic lifting doesn't imrove comlexity as in dense case.,! comuting the modular inverse is roscribed due to ll-in Solution [Wiedemann 986 ; Kaltofen, Saunders 99] :,! use modular minimal olynomial instead of inverse Let A 2 Z n n of full rank and b 2 Z n. Then x = A b can be exressed as a linear combination of the Krylov subsace fb; Ab; :::; A n bg Let () = c + c + ::: + d 2 Z [] be the minimal olynomial of A A b = c (c b + c 2 Ab + ::: + A d b) {z }

17 Sarse linear system and P-adic lifting P-adic lifting doesn't imrove comlexity as in dense case.,! comuting the modular inverse is roscribed due to ll-in Solution [Wiedemann 986 ; Kaltofen, Saunders 99] :,! use modular minimal olynomial instead of inverse Let A 2 Z n n of full rank and b 2 Z n. Then x = A b can be exressed as a linear combination of the Krylov subsace fb; Ab; :::; A n bg Let () = c + c + ::: + d 2 Z [] be the minimal olynomial of A A b = c (c b + c 2 Ab + ::: + A d b) {z } x

18 P-adic algorithm for sarse systems Scheme to comute A b : (-) := minoly (A) mod (-2) r := b for i := to k X deg (2-) x i := [] [i] A i r mod i= (2-2) r := (=)(r A:x i ) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

19 P-adic algorithm for sarse systems Scheme to comute A b : (-) := minoly (A) mod O~(n 2 log jjajj) (-2) r := b for i := to k X k = O~(n) deg (2-) x i := [] [i] A i r mod O~(n 2 log jjajj) i= (2-2) r := (=)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

20 P-adic algorithm for sarse systems Scheme to comute A b : (-) := minoly (A) mod (-2) r := b for i := to k X k = O~(n) deg (2-) x i := [] [i] A i r mod O~(n 2 log jjajj) i= (2-2) r := (=)(r A:x i ) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

21 Integer sarse linear system in ractice use of LinBox library on Itanium II -.3Ghz, 28Go RAM random non-singular sarse linear system with coecients over 3 bits and non zero elements er row. system order Male 64:7s 849s 98s CRA-Wied 4:8s 68s 7s 3857s 452s P-adic-Wied :2s 3s 693s 2629s 834s Dixon :9s s 42s 78s 429s

22 Integer sarse linear system in ractice use of LinBox library on Itanium II -.3Ghz, 28Go RAM random non-singular sarse linear system with coecients over 3 bits and non zero elements er row. system order Male 64:7s 849s 98s CRA-Wied 4:8s 68s 7s 3857s 452s P-adic-Wied :2s 3s 693s 2629s 834s Dixon :9s s 42s 78s 429s main dierence : (2-) x i = B:r mod (Dixon) P (2-) x i := deg [] i= [i] A i r mod (P-adic-Wied) Remark : n sarse matrix alications is far from level 2 BLAS in ractice.

23 Our objectives In ratice : Integrate level 2 and 3 BLAS in integer sarse solver In theory : Imrove bit comlexity of sarse linear system solving =) O~(n ) bits oerations with < 3?

24 Integration of BLAS in sarse solver Our goals : minimize the number of sarse matrix-vector roducts. maximize the number of level 2 and 3 BLAS oerations.,! Block Wiedemann algorithm seems to be a good candidate Let s be the blocking factor of Block Wiedemann algorithm. then I the number of sarse matrix-vector roduct is divided by roughly s. I order s matrix oerations are integrated.

25 Block Wiedemann and P-adic Relace vector rojections by block of vectors rojections s f ( u A i s A@ v A f b is st column of v Let A 2 Z n n of full rank, b 2 Z n and n = m s. One can use a column of the minimal generating matrix olynomial P 2 Z [x] s s of sequence fua i vg to exress A of block krylov subsace fv ; Av ; : : : ; A m vg b as a linear combination

26 Block Wiedemann and P-adic Relace vector rojections by block of vectors rojections s f ( u A i s A@ v A f b is st column of v Let A 2 Z n n of full rank, b 2 Z n and n = m s. One can use a column of the minimal generating matrix olynomial P 2 Z [x] s s of sequence fua i vg to exress A of block krylov subsace fv ; Av ; : : : ; A m vg b as a linear combination the cost to comute P is : I O~(s 3 m) eld o. [Beckermann, Labahn 994 ; Kaltofen 995 ; Thome 22], I O~(s! m) eld o. [Giorgi, Jeannerod, Villard 23].

27 Block Wiedemann and P-adic Scheme to comute A b : (-) r := b for i := to k (2-) v ; := r (2-2) P := block minoly fua i vg mod (2-3) x i := linear combi (A i v ; P) mod (2-4) r := (=)(r A:x i ) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

28 Block Wiedemann and P-adic Scheme to comute A b : (-) r := b for i := to k (2-) v ; := r k = O~(n) (2-2) P := block minoly fua i vg mod O~(s 2 n log jjajj) (2-3) x i := linear combi (A i v ; P) mod O~(n 2 log jjajj) (2-4) r := (=)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

29 Block Wiedemann and P-adic Scheme to comute A b : (-) r := b for i := to k (2-) v ; := r k = O~(n) (2-2) P := block minoly fua i vg mod O~(s 2 n log jjajj) (2-3) x i := linear combi (A i v ; P) mod O~(n 2 log jjajj) (2-4) r := (=)(r A:x i ) (3-) x := P k i= x i : i (3-2) rational reconstruction on x Not satisfying : comutation of block minoly. at each stes How to avoid the comutation of the block minimal olynomial?

30 Alternative to Block Wiedemann Exress the inverse of the sarse matrix through a structured form,! block Hankel/Toelitz structures Let u 2 Z s n U = and v 2 Z n s u ua. ua m C s.t. following matrices are non-singular B C A, V v Av : : : Am v A 2 Zn n

31 Alternative to Block Wiedemann Exress the inverse of the sarse matrix through a structured form,! block Hankel/Toelitz structures Let u 2 Z s n U = and v 2 Z n s u ua. ua m C s.t. following matrices are non-singular B C A, V v Av : : : Am v A 2 Zn n then we can dene the block Hankel matrix H = UAV = 2 m 2 3 m+ C. A, m m 2m C i = ua i v 2 Z s s and thus we have A = VH U

32 Alternative to Block Wiedemann Nice roerty on block Hankel matrix inverse [Gohberg, Krunik 972, Labahn, Choi, Cabay 99] H = : : :.... C A : : : C A {z } {z } : : :.... C A : : : C A {z } {z } H T H 2 T 2 where H, H 2 are block Hankel matrices and T, T 2 are block Toelitz matrices

33 Alternative to Block Wiedemann Nice roerty on block Hankel matrix inverse [Gohberg, Krunik 972, Labahn, Choi, Cabay 99] H = : : :.... C A : : : C A {z } {z } : : :.... C A : : : C A {z } {z } H T H 2 T 2 where H, H 2 are block Hankel matrices and T, T 2 are block Toelitz matrices Block coecients in H, H 2, T, T 2 come from Hermite Pade aroximants of H(z) = + 2 z + : : : + 2m z 2m 2 [Labahn, Choi, Cabay 99]. Comlexity of H Jeannerod, Villard 23]. reduces to olynomial matrix multilication [Giorgi,

34 Alternative to Block Wiedemann Scheme to comute A b : (-) H(z) := 2m X i= ua i v :z i mod (-2) comute H mod from H(z) (-3) r := b for i := to k (2-) x i := V H U:r mod (2-2) r := (=)(r A:x i ) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

35 Alternative to Block Wiedemann Scheme to comute A b : (-) H(z) := 2m X i= ua i v :z i mod O~(sn 2 log jjajj) (-2) comute H mod from H(z) O~(s 2 n log jjajj) (-3) r := b for i := to k k = O~(n) (2-) x i := V H U:r mod O~((n 2 + sn) log jjajj) (2-2) r := (=)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

36 Alternative to Block Wiedemann Scheme to comute A b : (-) H(z) := 2m X i= ua i v :z i mod O~(sn 2 log jjajj) (-2) comute H mod from H(z) O~(s 2 n log jjajj) (-3) r := b for i := to k k = O~(n) (2-) x i := V H U:r mod O~((n 2 + sn) log jjajj) (2-2) r := (=)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i : i (3-2) rational reconstruction on x Not yet satisfying : alying matrices U and V is too costly

37 Alying block Krylov subsaces V v Av : : : A m v A 2 Z n n and v 2 Z n s can be rewrite as V v A + v A + : : : + A v A Therefore, alying V to a vector corresonds to : m linear combinations of columns of v m alications of A

38 Alying block Krylov subsaces V v Av : : : A m v A 2 Z n n and v 2 Z n s can be rewrite as V v A + v A + : : : + A v A Therefore, alying V to a vector corresonds to : m linear combinations of columns of v m alications of A O(m sn log jjajj) O(mn log jjajj)

39 Alying block Krylov subsaces V v Av : : : A m v A 2 Z n n and v 2 Z n s can be rewrite as V v A + v A + : : : + A v A Therefore, alying V to a vector corresonds to : m linear combinations of columns of v m alications of A O(m sn log jjajj) How to imrove the comlexity?

40 Alying block Krylov subsaces V v Av : : : A m v A 2 Z n n and v 2 Z n s can be rewrite as V v A + v A + : : : + A v A Therefore, alying V to a vector corresonds to : m linear combinations of columns of v m alications of A O(m sn log jjajj) How to imrove the comlexity? ) using secial block rojections u and v

41 Candidates as suitable block rojections Considering A 2 Z n n non-singular and n = m s. Let us denote K(A; v) := v Av A m v 2 Z n n A suitable block rojection is dened through the trile such that : (R; u; v) 2 Z n n Z s n Z n s. K(RA; v) and K((RA) T ; u T ) are non-singular, 2. R can be alied to a vector with O~(n) oerations, 3. u, u T, v and v T can be alied to a vector with O~(n) oerations.

42 Candidates as suitable block rojections Considering A 2 Z n n non-singular and n = m s. Let us denote K(A; v) := v Av A m v 2 Z n n A suitable block rojection is dened through the trile such that : (R; u; v) 2 Z n n Z s n Z n s. K(RA; v) and K((RA) T ; u T ) are non-singular, 2. R can be alied to a vector with O~(n) oerations, 3. u, u T, v and v T can be alied to a vector with O~(n) oerations. Conjecture : for any non-singular A 2 Z n n block rojection (R; u; v) and sjn there exists a suitable

43 A structured block rojection Let u and v be dened as follow u = v T = u : : : u m um+ : : : u 2m... v : : : v m vm+ : : : v 2m... u n m+ : : : u n v n m+ : : : v n C A 2 Z s n C A 2 Z s n where u i and v i are chosen randomly from a sucient large set.

44 A structured block rojection Let u and v be dened as follow u = v T = u : : : u m um+ : : : u 2m... v : : : v m vm+ : : : v 2m... u n m+ : : : u n v n m+ : : : v n C A 2 Z s n C A 2 Z s n where u i and v i are chosen randomly from a sucient large set. oen question : Let R diagonal and v as dened above, is K(RA; v) necessarily non-singular? We rooved it for case s = 2 but answer is still unknown for s > 2

45 Our new algorithm Scheme to comute A b : (-) choose block rojection u and v (-2) choose R and A := R:A, b := R:b (-3) H(z) := X 2m i= ua i v :z i mod (-4) comute H mod from H(z) (-5) r := b for i := to k (2-) x i := V H U:r mod (2-2) r := (=)(r A:x i ) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

46 Our new algorithm Scheme to comute A b : (-) choose block rojection u and v (-2) choose R and A := R:A, b := R:b (-3) H(z) := X 2m i= ua i v :z i mod O~(n 2 log jjajj) (-4) comute H mod from H(z) O~(s 2 n log jjajj) (-5) r := b for i := to k k = O~(n) (2-) x i := V H U:r mod O~((mn + sn) log jjajj) (2-2) r := (=)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i : i (3-2) rational reconstruction on x

47 Our new algorithm Scheme to comute A b : (-) choose block rojection u and v (-2) choose R and A := R:A, b := R:b (-3) H(z) := X 2m i= ua i v :z i mod O~(n 2 log jjajj) (-4) comute H mod from H(z) O~(s 2 n log jjajj) (-5) r := b for i := to k k = O~(n) (2-) x i := V H U:r mod O~((mn + sn) log jjajj) (2-2) r := (=)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i : i (3-2) rational reconstruction on x taking the otimal m = s = n gives a comlexity of O~(n 2:5 log jjajj)

48 High level imlementation LinBox roject (Canada-France-USA) : Our tools : BLAS-based matrix multilication and matrix-vector roduct olynomial matrix arithmetic (block Hankel inversion),! FFT, Karatsuba, middle roduct fast alication of H is needed to get O~(n 2:5 log jjajj)

49 High level imlementation LinBox roject (Canada-France-USA) : Our tools : BLAS-based matrix multilication and matrix-vector roduct olynomial matrix arithmetic (block Hankel inversion),! FFT, Karatsuba, middle roduct fast alication of H is needed to get O~(n 2:5 log jjajj) I Lagrange's reresentation of H at the beginning (Horner's scheme) I use evaluation/interolation on olynomial vectors,! use Vandermonde matrix to have dense matrix oerations

50 Is our new algorithm ecient in ractice?

51 Performances use of LinBox library on Itanium II -.3Ghz, 28Go RAM random non-singular sarse linear system with coecients over 3 bits and non zero elements er row. system order Male 64:7s 849s 98s CRA-Wied 4:8s 68s 7s 3857s 452s P-adic-Wied :2s 3s 693s 2629s 834s Dixon :9s s 42s 78s 429s Our algo. 2:4s 5s 6s 75s 426s The exected n imrovement is unfortunately amortized by a high constant in the comlexity.

52 Sarse solver vs Dixon's algorithm 6 sarsity = elts/row Our algo. Dixon 6 sarsity = 3elts/row Our algo. Dixon Time ( 4 s) 3 Time ( 4 s) System order System order 44 Our algorithm erformances are deending on matrix sarsity

53 Sarse solver vs Dixon's algorithm 4 sarsity=.7% sarsity=.3% sarsity=.% crossover line 2 seed u System order The sarser the matrices are, the earlier the crossover aears

54 Practical eect on blocking factors n blocking factor value is theoretically otimal Is this still true in ractice?

55 Practical eect on blocking factors n blocking factor value is theoretically otimal Is this still true in ractice? system order =, otimal block = block size timing 723s 5264s 459s 3833s 4332s system order = 2, otimal block 4 block size timing 4472s 35967s 3854s 2852s 3738s

56 Practical eect on blocking factors n blocking factor value is theoretically otimal Is this still true in ractice? system order =, otimal block = block size timing 723s 5264s 459s 3833s 4332s system order = 2, otimal block 4 block size timing 4472s 35967s 3854s 2852s 3738s best ractical blocking factor is certainly deending on the ratio of sarse matrix/dense matrix oerations eciency

57 Conclusions We rovide a new aroach for solving sarse integer linear systems : I imrove the comlexty by a factor n (heuristic). I allow eciency by minimizing sarse matrix oerations and maximizing BLAS use. We introduce secial block rojections for sarse linear algebra,! inverse of sarse matrix in O(n 2:5 ) eld o. drawback : not taking advantage of low degree minimal olynomial

58 Conclusions We rovide a new aroach for solving sarse integer linear systems : I imrove the comlexty by a factor n (heuristic). I allow eciency by minimizing sarse matrix oerations and maximizing BLAS use. We introduce secial block rojections for sarse linear algebra,! inverse of sarse matrix in O(n 2:5 ) eld o. drawback : not taking advantage of low degree minimal olynomial On going work : I rovide an automatic choice of block dimension (non square?) I roove conjecture for secial block rojections I how to handle the case of singular matrix? I how to introduce fast matrix multilication in the comlexity?