Solving Sparse Rational Linear Systems. Pascal Giorgi. University of Waterloo (Canada) / University of Perpignan (France) joint work with

Transcription

1 Solving Sparse Rational Linear Systems Pascal Giorgi University of Waterloo (Canada) / University of Perpignan (France) joint work with A. Storjohann, M. Giesbrecht (University of Waterloo), W. Eberly (University of Calgary), G. Villard (ENS Lyon) ISSAC'26, Genova - July, 26

2 Problem Let A a non-singular matrix and b a vector dened over Z. Problem : Compute x = A b over the rational numbers A = B x = A b = B C A, b = C A B C A. Main diculty : expression swell

3 Problem Let A a non-singular matrix and b a vector dened over Z. Problem : Compute x = A b over the rational numbers A = B x = A b = B C A, b = C A B C A. Main diculty : expression swell and take advantage of sparsity

4 Motivations Large linear systems are involved in many mathematical applications Over a nite eld : integers factorization [Odlyzko 999], discrete logarithm [Odlyzko 999 ; Thome 23]. Over the integers : number theory [Cohen 993], group theory [Newman 972], integer programming [Aardal, Hurkens, Lenstra 999]. Rational linear systems are central in recent linear algebra algorithms I Determinant [Abbott, Bronstein, Mulders 999 ; Storjohann 25] I Smith form [Giesbrecht 995 ; Eberly, Giesbrecht, Villard 2] I Nullspace, Kernel [Chen, Storjohann 25]

5 Algorithms for non-singular system solving Dense matrices : I Gaussian elimination and CRA,! O~(n!+ log jjajj) bit operations I P-adic lifting [Monck, Carter 979 ; Dixon 982],! O~(n 3 log jjajj) bit operations I High order lifting [Storjohann 25],! O~(n! log jjajj) bit operations Sparse matrices : I P-adic lifting or CRA [Kaltofen, Saunders 99],! O~(n 2 log jjajj) bit operations with non-zero elts.

6 P-adic algorithm with matrix inversion Scheme to compute A b [Dixon 982] : (-) B := A mod p (-2) r := b for i := to k (2-) x i := B:r mod p (2-2) r := (=p)(r A:x i ) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x

7 P-adic algorithm with matrix inversion Scheme to compute A b [Dixon 982] : (-) B := A mod p O~(n 3 log jjajj) (-2) r := b for i := to k k = O~(n) (2-) x i := B:r mod p O~(n 2 log jjajj) (2-2) r := (=p)(r A:x i ) O~(n 2 log jjajj) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x

8 P-adic algorithm with matrix inversion Scheme to compute A b [Dixon 982] : (-) B := A mod p O~(n 3 log jjajj) (-2) r := b for i := to k k = O~(n) (2-) x i := B:r mod p O~(n 2 log jjajj) (2-2) r := (=p)(r A:x i ) O~(n 2 log jjajj) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x Main operations : matrix inversion and matrix-vector products

9 Dense linear system solving in practice Ecient implementations are available : LinBox. [ Use tuned BLAS oating-point library for exact arithmetic matrix inversion over prime eld [Dumas, Giorgi, Pernet 24] BLAS matrix-vector product with CRT over the integers Rational number reconstruction half GCD [Schonage 97] heuristic using integer multiplication [NTL library] random dense linear system with 3 bits entries (P4-3.4Ghz) n Time :6s 4:3s 3:s 99:6s 236:8s 449:2s performances improvement of a factor over NTL's tuned implementation

10 What does happen when matrices are sparse? We consider sparse matrices with O(n) non zero elements,! matrix-vector product needs only O(n) operations.

11 Sparse linear system and P-adic lifting Computing the modular inverse is proscribed due to ll-in Solution [Kaltofen, Saunders 99] :,! use modular minimal polynomial instead of inverse

12 Sparse linear system and P-adic lifting Computing the modular inverse is proscribed due to ll-in Solution [Kaltofen, Saunders 99] :,! use modular minimal polynomial instead of inverse Wiedemann approach (986) Let A 2 IF n n of full rank and b 2 IF n. Then x = A b can be expressed as a linear combination of the Krylov subspace fb; Ab; :::; A n bg Let f A () = f + f + ::: + f d d 2 IF[] be the minimal polynomial of A

13 Sparse linear system and P-adic lifting Computing the modular inverse is proscribed due to ll-in Solution [Kaltofen, Saunders 99] :,! use modular minimal polynomial instead of inverse Wiedemann approach (986) Let A 2 IF n n of full rank and b 2 IF n. Then x = A b can be expressed as a linear combination of the Krylov subspace fb; Ab; :::; A n bg Let f A () = f + f + ::: + f d d 2 IF[] be the minimal polynomial of A A b = f (f b + f 2 Ab + ::: + f d A d b) {z }

14 Sparse linear system and P-adic lifting Computing the modular inverse is proscribed due to ll-in Solution [Kaltofen, Saunders 99] :,! use modular minimal polynomial instead of inverse Wiedemann approach (986) Let A 2 IF n n of full rank and b 2 IF n. Then x = A b can be expressed as a linear combination of the Krylov subspace fb; Ab; :::; A n bg Let f A () = f + f + ::: + f d d 2 IF[] be the minimal polynomial of A A b = f (f b + f 2 Ab + ::: + f d A d b) {z } x Applying minpoly in each lifting steps cost O~(nd) eld operations, then giving a worst case complexity of O~(n 3 log jjajj) bit operations.

15 Sparse linear system solving in practice use of LinBox library on Itanium II -.3Ghz, 28Gb RAM random systems with 3 bits entries and elts/row (plus identity) system order Maple 64:7s 849s 98s CRA-Wied 4:8s 68s 7s 3857s 452s P-adic-Wied :2s 3s 693s 2629s 834s Dixon :9s s 42s 78s 429s

16 Sparse linear system solving in practice use of LinBox library on Itanium II -.3Ghz, 28Gb RAM random systems with 3 bits entries and elts/row (plus identity) system order Maple 64:7s 849s 98s CRA-Wied 4:8s 68s 7s 3857s 452s P-adic-Wied :2s 3s 693s 2629s 834s Dixon :9s s 42s 78s 429s main dierence : (2-) x i = B:r mod p (Dixon) P (2-) x i := deg A f f i= f i A i r mod p (P-adic-Wied) Remark : n sparse matrix applications is far from level 2 BLAS in practice.

17 Our objectives In practice : Integrate level 2 and 3 BLAS in integer sparse solver In theory : Improve bit complexity of sparse linear system solving =) O~(n ) bits operations with < 3?

18 Our alternative to Block Wiedemann Express the inverse of the sparse matrix through a structured form,! block Hankel/Toeplitz structures Let u 2 IF s n and v 2 IF n s s.t. following matrices are non-singular U = B u ua. ua m C B C A, V = v Av : : : Am v A 2 IFn n

19 Our alternative to Block Wiedemann Express the inverse of the sparse matrix through a structured form,! block Hankel/Toeplitz structures Let u 2 IF s n and v 2 IF n s s.t. following matrices are non-singular U = B u ua. ua m C B C A, V = v Av : : : Am v A 2 IFn n then we can dene the block Hankel matrix H = UAV = B 2 m 2 3 m+ C. A, m m 2m C i = ua i v 2 IF s s and thus we have A = VH U

20 Alternative to Block Wiedemann Nice property on block Hankel matrix inverse [Gohberg, Krupnik 972, Labahn, Choi, Cabay 99] H = B : : :.... C A B : : : C A {z } {z } B : : :.... C A B : : : C A {z } {z } H T H 2 T 2 where H, H 2 are block Hankel matrices and T, T 2 are block Toeplitz matrices

21 Alternative to Block Wiedemann Nice property on block Hankel matrix inverse [Gohberg, Krupnik 972, Labahn, Choi, Cabay 99] H = B : : :.... C A B : : : C A {z } {z } B : : :.... C A B : : : C A {z } {z } H T H 2 T 2 where H, H 2 are block Hankel matrices and T, T 2 are block Toeplitz matrices Block coecients in H, H 2, T, T 2 come from Hermite Pade approximants of H(z) = + 2 z + : : : + 2m z 2m 2 [Labahn, Choi, Cabay 99]. Complexity of H reduces to polynomial matrix multiplication [Giorgi, Jeannerod, Villard 23].

22 Alternative to Block Wiedemann Scheme to compute A b : (-) H(z) := 2m X i= ua i v :z i mod p (-2) compute H mod p from H(z) (-3) r := b for i := to k (2-) x i := V H U:r mod p (2-2) r := (=p)(r A:x i ) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x

23 Alternative to Block Wiedemann Scheme to compute A b : (-) H(z) := 2m X i= ua i v :z i mod p O~(sn 2 log jjajj) (-2) compute H mod p from H(z) O~(s 2 n log jjajj) (-3) r := b for i := to k k = O~(n) (2-) x i := V H U:r mod p O~((n 2 + sn) log jjajj) (2-2) r := (=p)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x

24 Alternative to Block Wiedemann Scheme to compute A b : (-) H(z) := 2m X i= ua i v :z i mod p O~(sn 2 log jjajj) (-2) compute H mod p from H(z) O~(s 2 n log jjajj) (-3) r := b for i := to k k = O~(n) (2-) x i := V H U:r mod p O~((n 2 + sn) log jjajj) (2-2) r := (=p)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x Not yet satisfying : applying matrices U and V is too costly

25 Applying block Krylov subspaces V = v Av : : : A m va 2 IF n n and v 2 IF n s can be rewrite as V = v A + A v A + : : : + A m v A Therefore, applying V to a vector corresponds to : m linear combinations of columns of v m applications of A

26 Applying block Krylov subspaces V = v Av : : : A m va 2 IF n n and v 2 IF n s can be rewrite as V = v A + A v A + : : : + A m v A Therefore, applying V to a vector corresponds to : m linear combinations of columns of v m applications of A O(m sn log jjajj) O(mn log jjajj)

27 Applying block Krylov subspaces V = v Av : : : A m va 2 IF n n and v 2 IF n s can be rewrite as V = v A + A v A + : : : + A m v A Therefore, applying V to a vector corresponds to : m linear combinations of columns of v m applications of A O(m sn log jjajj) How to improve the complexity?

28 Applying block Krylov subspaces V = v Av : : : A m va 2 IF n n and v 2 IF n s can be rewrite as V = v A + A v A + : : : + A m v A Therefore, applying V to a vector corresponds to : m linear combinations of columns of v m applications of A O(m sn log jjajj) How to improve the complexity? ) using special block projections u and v

29 Candidates as suitable block projections Considering A 2 IF n n non-singular and n = m s. Let us denote K(A; v) := v Av A m v 2 IF n n Conjecture : for any non-singular A 2 IF n n and sjn there exists a suitable block projection (R; u; v) 2 IF n n IF s n IF n s such that :. K(RA; v) and K((RA) T ; u T ) are non-singular, 2. R can be applied to a vector with O~(n) operations, 3. u, u T, v and v T can be applied to a vector with O~(n) operations.

30 A structured block projection Let v be dened as follow v T = B v : : : v m vm+ : : : v 2m... v n m+ : : : v n C A 2 IF s n where v i 's are chosen randomly from a sucient large set.

31 A structured block projection Let v be dened as follow v T = B v : : : v m vm+ : : : v 2m... v n m+ : : : v n C A 2 IF s n where v i 's are chosen randomly from a sucient large set. open question : Let R diagonal and v as dened above, is K(RA; v) necessarily non-singular? We prooved it for case s = 2 but answer is still unknown for s > 2

32 Our new algorithm Scheme to compute A b : (-) choose structured blocks u and v (-2) choose R and A := R:A, b := R:b (-3) H(z) := X 2m i= ua i v :z i mod p (-4) compute H mod p from H(z) (-5) r := b for i := to k (2-) x i := V H U:r mod p (2-2) r := (=p)(r A:x i ) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x

33 Our new algorithm Scheme to compute A b : (-) choose structured blocks u and v (-2) choose R and A := R:A, b := R:b (-3) H(z) := X 2m i= ua i v :z i mod p O~(n 2 log jjajj) (-4) compute H mod p from H(z) O~(s 2 n log jjajj) (-5) r := b for i := to k k = O~(n) (2-) x i := V H U:r mod p O~((mn + sn) log jjajj) (2-2) r := (=p)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x

34 Our new algorithm Scheme to compute A b : (-) choose structured blocks u and v (-2) choose R and A := R:A, b := R:b (-3) H(z) := X 2m i= ua i v :z i mod p O~(n 2 log jjajj) (-4) compute H mod p from H(z) O~(s 2 n log jjajj) (-5) r := b for i := to k k = O~(n) (2-) x i := V H U:r mod p O~((mn + sn) log jjajj) (2-2) r := (=p)(r A:x i ) O~(n log jjajj) (3-) x := P k i= x i :p i (3-2) rational reconstruction on x taking the optimal m = s = p n gives a complexity of O~(n 2:5 log jjajj)

35 High level implementation LinBox project (Canada-France-USA) : Our tools : BLAS-based matrix multiplication and matrix-vector product polynomial matrix arithmetic (block Hankel inversion),! FFT, Karatsuba, middle product fast application of H is needed to get O~(n 2:5 log jjajj)

36 High level implementation LinBox project (Canada-France-USA) : Our tools : BLAS-based matrix multiplication and matrix-vector product polynomial matrix arithmetic (block Hankel inversion),! FFT, Karatsuba, middle product fast application of H is needed to get O~(n 2:5 log jjajj) I Lagrange's representation of H at the beginning (Horner's scheme) I use evaluation/interpolation on polynomial vectors,! use Vandermonde matrix to have dense matrix operations

37 Is our new algorithm ecient in practice?

38 Comparing performances use of LinBox library on Itanium II -.3Ghz, 28Gb RAM random systems with 3 bits entries and elts/row (plus identity) system order extra memory Maple 849s 98s O() CRA-Wied 68s 7s 3 857s 452s 28 s O(n) P-adic-Wied 3s 693s 2 629s 8 34s 2 s O(n) Dixon s 42s 78s 429s 257s O(n 2 ) Our algo. 5s 6s 75s 426s 937s O(n :5 ) The expected p n improvement is unfortunately amortized by a high constant in the complexity.

39 Sparse solver vs Dixon's algorithm 6 sparsity = elts/row Our algo. Dixon 6 sparsity = 3elts/row Our algo. Dixon Time ( 4 s) 3 Time ( 4 s) System order System order 44 Our algorithm performances are depending on matrix sparsity

40 Practical eect of blocking factors p n blocking factor value is theoretically optimal Is this still true in practice?

41 Practical eect of blocking factors p n blocking factor value is theoretically optimal Is this still true in practice? system order =, optimal block = block size timing 723s 5264s 459s 3833s 4332s system order = 2, optimal block 4 block size timing 4472s 35967s 3854s 2852s 3738s

42 Practical eect of blocking factors p n blocking factor value is theoretically optimal Is this still true in practice? system order =, optimal block = block size timing 723s 5264s 459s 3833s 4332s system order = 2, optimal block 4 block size timing 4472s 35967s 3854s 2852s 3738s best practical blocking factor is dependent upon the ratio of sparse matrix/dense matrix operations eciency

43 Conclusions We provide a new approach for solving sparse integer linear systems : I improve the complexty by a factor p n (heuristic). I improve eciency by minimizing sparse matrix operations and maximizing BLAS use. drawback : not taking advantage of low degree minimal polynomial We propose special block projections for sparse linear algebra,! inverse of sparse matrix in O(n 2:5 ) eld op.

44 Conclusions We provide a new approach for solving sparse integer linear systems : I improve the complexty by a factor p n (heuristic). I improve eciency by minimizing sparse matrix operations and maximizing BLAS use. drawback : not taking advantage of low degree minimal polynomial We propose special block projections for sparse linear algebra,! inverse of sparse matrix in O(n 2:5 ) eld op. Ongoing work : I provide an automatic choice of block dimension (non square?) I prove conjecture for our structured block projections I handle the case of singular matrix I introduce fast matrix multiplication in the complexity

45 Sparse solver vs Dixon's algorithm 4 sparsity=.7% sparsity=.3% sparsity=.% crossover line 2 speed up System order The sparser the matrices are, the earlier the crossover appears