The Generalized Sieve Kernel

Transcription

1 The Generalized Sieve Kernel The Algorithmic Ant and the Sandpile Léo Ducas 1 Based on joint work in progress with M. Albrecht, E. Postlethwaite, G. Herold, E. Kirshanova, M. Stevens Cryptology Group, CWI, Amsterdam, The Netherlands Lattice Coding Crypto Meeting London, Sept Supported by a Veni Innovational Research Grant from NWO ( ). Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

2 T h`e A l g o r i t h m i`c A n t `a n`dffl t h`e S`a n`d p i l e Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

3 O n`c e u p`o nffl `affl tˇi m`e... Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

4 O n`c e u p`o nffl `affl tˇi m`e t h`eˇr`e wˆa s `a nffl `a n t. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

5 O n`c e u p`o nffl `affl tˇi m`e t h`eˇr`e wˆa s `a nffl `a n t. xor a bit 7,b jr z,bc_nneg ld hl,0 xor a sbc hl,bc push hl pop bc ld a,1 A nffl `ãl g o r i t h m i`c `a n t. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

6 T h`e Q u`e e nffl `o f `a n tṡ sfi u m m`o n`e dffl t h`e `ãl g o r i t h m i`c `a n t, Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

7 T h`e Q u`e e nffl `o f `a n tṡ sfi u m m`o n`e dffl t h`e `ãl g o r i t h m i`c `a n t, S`e e t h i s sfi`a n`dffl p i l e. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

8 Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

9 I wˆa n t i t f l åt! Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

10 x100 L`oˆo k i n`g `c l oşfi`eˇrffl `åt t h`e S`a n`d p i l e, t h`e `ãl g o r i t h m i`c `a n t p`o n`d`eˇr s. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

11 O n`e `gˇr`a i nffl `åt t h`e tˇi m`e, I sfi h`ãl l p u l l t h`e sfi`a n`dffl `d`o w n h i l l. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

12 O n`e `gˇr`a i nffl `åt t h`e tˇi m`e, I sfi h`ãl l p u l l t h`e sfi`a n`dffl `d`o w n h i l l. S`o h`o p`e dffl t h`e `a n t. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

13 B u t t h`e sfi`a n`d p i l e i s w h i m sfi i`c ãl, `e àc hffl `e x c a vˆåtˇi`o nffl i s `affl p u zˇz l e `o f i tṡ `o w nffl. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

14 B u t t h`e sfi`a n`d p i l e i s w h i m sfi i`c ãl, `e àc hffl `e x c a vˆåtˇi`o nffl i s `affl p u zˇz l e `o f i tṡ `o w nffl. x100 Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

15 B u t t h`e sfi`a n`d p i l e i s w h i m sfi i`c ãl, `e àc hffl `e x c a vˆåtˇi`o nffl i s `affl p u zˇz l e `o f i tṡ `o w nffl. x100 C`o lˇu m n s `år`e tˇi`e dffl i nffl m yṡfi t eˇr i`o u s wˆa yṡ: t o p u sfi hffl `o n`e `d`o w nffl, `o n`e m u sfi t fˇi n`dffl t h`e r i`g h t `c o m b i n`åtˇi`o nffl. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

16 U n sfi u r`e h`o w t o p r`oˆc e e dffl,? Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

17 U n sfi u r`e h`o w t o p r`oˆc e e dffl, T h`e `a n t `c ãl lṡ `affl `c o m p u t eˇrffl sfi`cˇi`e n tˇi sfi t. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

18 L`eˇt s b r`e a kffl t h i s `āp`år t. Figure: Annie Easley (NASA / NACA) 1 From Lattices to Sandpiles 2 Finding a grain of sand: Progress on SVP from Sieving 3 Flattening the Pile: Progress on lattice reduction from Sieving Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

19 From Lattices to Sandpiles Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

20 Lattices! y b2 x b1 Definition A lattice L is a discrete subgroup of a finite-dimensional Euclidean vector space. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

21 Bases of a Lattice b 2 b 2 b 1 b 1 Good Basis G of L Bad Basis B of L G B : easy (randomization); B G : hard (LLL, BKZ, Lattice Sieve...). Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

22 An important invariant: the Volume For any two bases G, B of the same lattice Λ: det(gg t ) = det(bb t ). We can therefore define: vol(λ) = det(gg t ). Geometrically: the volume of any fundamental domain of Λ. Let G be the Gram-Schmidt Orthogonalization of G G is not a basis of Λ, nevertheless: vol(λ) = det(g G t ) = gi. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

23 An important invariant: the Volume For any two bases G, B of the same lattice Λ: det(gg t ) = det(bb t ). We can therefore define: vol(λ) = det(gg t ). Geometrically: the volume of any fundamental domain of Λ. Let G be the Gram-Schmidt Orthogonalization of G G is not a basis of Λ, nevertheless: vol(λ) = det(g G t ) = gi. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

24 What is a Good basis Recall that, independently of the basis G it holds that: vol(λ) = gi. Therefore, it is somehow equivalent that max i gi is small min i gi is large κ(g) = max i gi / min i gi is small Good basis max b i min b i Bad basis max b i min b i Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

25 What is a Good basis Recall that, independently of the basis G it holds that: vol(λ) = gi. Therefore, it is somehow equivalent that max i gi is small min i gi is large κ(g) = max i gi / min i gi is small Good basis max b i min b i Bad basis max b i min b i Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

26 What is a Good basis Recall that, independently of the basis G it holds that: vol(λ) = gi. Therefore, it is somehow equivalent that max i gi is small min i gi is large κ(g) = max i gi / min i gi is small Good basis max b i min b i Bad basis max b i min b i Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

27 Bases and Fundamental Domains Each basis defines a parallelepipedic tiling. v t t v Round off Algorithm [Lenstra, Babai]: Given a target t Find s v L at the center the tile. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

28 Bases and Fundamental Domains Each basis defines a parallelepipedic tiling. v t t v Round off Algorithm [Lenstra, Babai]: Given a target t Find s v L at the center the tile. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

29 Bases and Fundamental Domains Each basis defines a parallelepipedic tiling. v t t v Round off Algorithm [Lenstra, Babai]: Given a target t Find s v L at the center the tile. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

30 Round off Algorithm t B 1 t B RoundOff Algorithm [Lenstra,Babai]: Use B to switch to the lattice Z n ( B 1 ) round each coordinate (square tiling) switch back to L ( B) t = B 1 t; v = t ; v = B v Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

31 Round off Algorithm t B 1 t B RoundOff Algorithm [Lenstra,Babai]: Use B to switch to the lattice Z n ( B 1 ) round each coordinate (square tiling) switch back to L ( B) t = B 1 t; v = t ; v = B v Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

32 Round off Algorithm t B 1 v t B RoundOff Algorithm [Lenstra,Babai]: Use B to switch to the lattice Z n ( B 1 ) round each coordinate (square tiling) switch back to L ( B) t = B 1 t; v = t ; v = B v Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

33 Round off Algorithm v t B 1 v t B RoundOff Algorithm [Lenstra,Babai]: Use B to switch to the lattice Z n ( B 1 ) round each coordinate (square tiling) switch back to L ( B) t = B 1 t; v = t ; v = B v Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

34 Nearest-Plane Algorithm There is a better algorithm (NearestPlane) based on Gram-Schmidt Orth. B of a basis B: Decoding radius with G Decoding radius with B Worst-case distance: 1 2 b i 2 (Approx-CVP) Correct decoding of t = v + e where v Λ if (BDD) e 1 2 min b i Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

35 Profile of a Basis Good basis max b i min b i Bad basis max b i min b i Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

36 Profile of a Basis Good basis max b i min b i Bad basis max b i min b i log bi* i Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

37 Profile of a Basis Good basis max b i min b i Bad basis max b i min b i log bi* Good basis Flat profile i Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

38 Local Modification B = T * Q Local blocks [i : j] of T correspond to a projected sublattice L [i:j] We can work locally: modify this block, affecting only b i... b j Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

39 Local Modification B = T * Q Local blocks [i : j] of T correspond to a projected sublattice L [i:j] We can work locally: modify this block, affecting only b i... b j Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

40 Local Modification B = T * Q Local blocks [i : j] of T correspond to a projected sublattice L [i:j] We can work locally: modify this block, affecting only b i... b j x100 Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

41 Local Improvement Find the shortest vector v of the projected sublattice L [i:j] `affl p u zˇz l e `o f i tṡ `o w nffl. t h`e r i`g h t `c o m b i n`åtˇi`o nffl. Construct a unimodular matrix U such that T [i:j] U = [v,,,... ]. Apply U (locally). The new b i = v got shorter! The other b i+1,..., b j will change as well Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

42 Local Improvement Find the shortest vector v of the projected sublattice L [i:j] `affl p u zˇz l e `o f i tṡ `o w nffl. t h`e r i`g h t `c o m b i n`åtˇi`o nffl. Construct a unimodular matrix U such that T [i:j] U = [v,,,... ]. Apply U (locally). The new b i = v got shorter! The other b i+1,..., b j will change as well Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

43 Local Improvement Find the shortest vector v of the projected sublattice L [i:j] `affl p u zˇz l e `o f i tṡ `o w nffl. t h`e r i`g h t `c o m b i n`åtˇi`o nffl. Construct a unimodular matrix U such that T [i:j] U = [v,,,... ]. Apply U (locally). The new b i = v got shorter! The other b i+1,..., b j will change as well x100 Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

44 Lattice reduction (e.g. BKZ-b) b: Blocksize Run the local improvements for consecutive blocks: [1 : b], [2 : b + 1], [3 : b + 2],..., [n b : n], [n b + 1 : n],... [n 1 : n] This is called a tour. Repeat tours until satisfication (or convergence). Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

45 Lattice reduction (e.g. BKZ-b) b: Blocksize Run the local improvements for consecutive blocks: [1 : b], [2 : b + 1], [3 : b + 2],..., [n b : n], [n b + 1 : n],... [n 1 : n] This is called a tour. Repeat tours until satisfication (or convergence). Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

46 BKZ in action Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

47 T h`a n k s f o rffl t h`e l e cˇtˇu r`e, b u t... xor a bit 7,b jr z,bc_nneg ld hl,0 xor a sbc hl,bc push hl pop bc ld a,1 h`o w sfi h`o u l dffl I sfi`o l vfle t h`oşfi`e SVP p u zˇz l eṡ? Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

48 T h`a n k s f o rffl t h`e l e cˇtˇu r`e, b u t... xor a bit 7,b jr z,bc_nneg ld hl,0 xor a sbc hl,bc push hl pop bc ld a,1 h`o w sfi h`o u l dffl I sfi`o l vfle t h`oşfi`e SVP p u zˇz l eṡ? Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

49 Shortest Vector from Lattice Sieving: a Few Dimensions for Free 2 2 EUROCRYPT 2018 Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

50 Two classes of Algorithms for SVP The Shortest Vector Problem I : The basis B of an n-dimensional lattice L O: A shortest non-zero vector v L Algorithm Running time Memory Enumeration n n/2e 2 O(n) poly(n) Sieving 3 [2.292n+o(n), 2.415n+o(n) ] [2.2075n+o(n), 2.292n+o(n) ] The paradox In theory, Sieving is faster. In pratice it is quite a lot slower. 3 Given complexities are heuristic, heavily supported by experiments. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

51 Two classes of Algorithms for SVP The Shortest Vector Problem I : The basis B of an n-dimensional lattice L O: A shortest non-zero vector v L Algorithm Running time Memory Enumeration n n/2e 2 O(n) poly(n) Sieving 3 [2.292n+o(n), 2.415n+o(n) ] [2.2075n+o(n), 2.292n+o(n) ] The paradox In theory, Sieving is faster. In pratice it is quite a lot slower. 3 Given complexities are heuristic, heavily supported by experiments. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

52 Two classes of Algorithms for SVP The Shortest Vector Problem I : The basis B of an n-dimensional lattice L O: A shortest non-zero vector v L Algorithm Running time Memory Enumeration n n/2e 2 O(n) poly(n) Sieving 3 [2.292n+o(n), 2.415n+o(n) ] [2.2075n+o(n), 2.292n+o(n) ] The paradox In theory, Sieving is faster. In pratice it is quite a lot slower. 3 Given complexities are heuristic, heavily supported by experiments. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

53 Many trade-offs Time complexity n n n n NV '08 MV '10 BDGL16 BGJ '15 WLTB '11 LdW '15 / BL '15 ZPH '13 BGJ '14 Laa '15 Time =Space n n n n n Space complexity Our main contribution can also be applied to other sieving algorithms. Implementation limited to the version of [Micciancio Voulgaris 2010]. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

54 Many trade-offs Time complexity n In this work n n n NV '08 MV '10 BDGL16 BGJ '15 WLTB '11 LdW '15 / BL '15 ZPH '13 BGJ '14 Laa '15 Time =Space n n n n n Space complexity Our main contribution can also be applied to other sieving algorithms. Implementation limited to the version of [Micciancio Voulgaris 2010]. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

55 Results Heuristic claim, asymptotic One can solve SVP in dimension n with a call to Sieve in dimension n d where d = Θ(n/ log n). Heuristic claim, concrete One can solve SVP in dimension n making a call to Sieve in dimension i for each i = 2... n d for d n ln(4/3) ln(n/2πe) (d 15 for n = 80) Experimental claim: A bogey A Sieve implem. almost on par with enumeration (within a factor 4 in dims 70 80), still with room for many improvements. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

56 Results Heuristic claim, asymptotic One can solve SVP in dimension n with a call to Sieve in dimension n d where d = Θ(n/ log n). Heuristic claim, concrete One can solve SVP in dimension n making a call to Sieve in dimension i for each i = 2... n d for d n ln(4/3) ln(n/2πe) (d 15 for n = 80) Experimental claim: A bogey A Sieve implem. almost on par with enumeration (within a factor 4 in dims 70 80), still with room for many improvements. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

57 Results Heuristic claim, asymptotic One can solve SVP in dimension n with a call to Sieve in dimension n d where d = Θ(n/ log n). Heuristic claim, concrete One can solve SVP in dimension n making a call to Sieve in dimension i for each i = 2... n d for d n ln(4/3) ln(n/2πe) (d 15 for n = 80) Experimental claim: A bogey A Sieve implem. almost on par with enumeration (within a factor 4 in dims 70 80), still with room for many improvements. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

58 Sieving Algorithm 1 Sieve(L) L a set of N random vectors from L where N (4/3) n/2. while (v, w) L 2 such that v w < v do v v w end while return L The above runs in heuristic time (4/3) n+o(n). Many concrete and asymptotic improvements: [Nguyen Vidick 2008, Micciancio Voulgaris 2010, Laarhoven 2015, Becker Gamma Joux 2015, Becker D. Gamma Laarhoven 2015,... ]. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

59 Sieving Algorithm 2 Sieve(L) L a set of N random vectors from L where N (4/3) n/2. while (v, w) L 2 such that v w < v do v v w end while return L The above runs in heuristic time (4/3) n+o(n). Many concrete and asymptotic improvements: [Nguyen Vidick 2008, Micciancio Voulgaris 2010, Laarhoven 2015, Becker Gamma Joux 2015, Becker D. Gamma Laarhoven 2015,... ]. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

60 More than SVP Note that Sieve returns N (4/3) n short vectors, not just a shortest vector. Definition (Gaussian Heuristic: Expected length of the shortest vector) gh(l) = n/2πe vol(l) 1/n. Observation (heuristic & experimental) The output of Sieve contains almost all vectors of length 4/3 gh(l): { L := Sieve(L) = x L s.t. x } 4/3 gh(l). Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

61 Sieve then Lift Main idea: Sieve in a projected sub-lattice, and lift all candidate solutions. SubSieve(L, d) Set L = L(b 1,..., b d ) Set L = π L (L) Compute L = Sieve(L ) left part of L, dim=d right part of L, dim=n d Hope that πl (s) L (1) Lift all v L from L to L and take the shortest (Babai alg.) Pessimistic prediction for (1) Optimistic prediction for (1) gh(l) 4/3 gh(l ). n d gh(l) 4/3 gh(l ). n Similar to linear pruning for enum. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

62 Sieve then Lift Main idea: Sieve in a projected sub-lattice, and lift all candidate solutions. SubSieve(L, d) Set L = L(b 1,..., b d ) Set L = π L (L) Compute L = Sieve(L ) left part of L, dim=d right part of L, dim=n d Hope that πl (s) L (1) Lift all v L from L to L and take the shortest (Babai alg.) Pessimistic prediction for (1) Optimistic prediction for (1) gh(l) 4/3 gh(l ). n d gh(l) 4/3 gh(l ). n Similar to linear pruning for enum. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

63 Sieve then Lift Main idea: Sieve in a projected sub-lattice, and lift all candidate solutions. SubSieve(L, d) Set L = L(b 1,..., b d ) Set L = π L (L) Compute L = Sieve(L ) left part of L, dim=d right part of L, dim=n d Hope that πl (s) L (1) Lift all v L from L to L and take the shortest (Babai alg.) Pessimistic prediction for (1) Optimistic prediction for (1) gh(l) 4/3 gh(l ). n d gh(l) 4/3 gh(l ). n Similar to linear pruning for enum. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

64 Sieve then Lift Main idea: Sieve in a projected sub-lattice, and lift all candidate solutions. SubSieve(L, d) Set L = L(b 1,..., b d ) Set L = π L (L) Compute L = Sieve(L ) left part of L, dim=d right part of L, dim=n d Hope that πl (s) L (1) Lift all v L from L to L and take the shortest (Babai alg.) Pessimistic prediction for (1) Optimistic prediction for (1) gh(l) 4/3 gh(l ). n d gh(l) 4/3 gh(l ). n Similar to linear pruning for enum. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

65 With BKZ pre-processing To ensure (1), we need the basis to be as reduced as possible We can easily afford BKZ preprocessing with block-size b = n/2 Using simple BKZ models 4 we can predict gh(l) and gh(l ) Heuristic claim SubSieve(L, d) algorithm will successfully find the shortest vector of L for some d = Θ(n/ ln n). Improve time & memory by a sub-exponential factor 2Θ(n/ log n) 4 The Geometric Series Assumption Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

66 With BKZ pre-processing To ensure (1), we need the basis to be as reduced as possible We can easily afford BKZ preprocessing with block-size b = n/2 Using simple BKZ models 4 we can predict gh(l) and gh(l ) Heuristic claim SubSieve(L, d) algorithm will successfully find the shortest vector of L for some d = Θ(n/ ln n). Improve time & memory by a sub-exponential factor 2Θ(n/ log n) 4 The Geometric Series Assumption Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

67 Quasi-HKZ preprocessing Idea: Attempt stronger pre-processing. Algorithm 3 SubSieve + (L, d) L Sieve(L ) L = {Lift L L(v) for v L} for j = 0... n/2 1 do v j = arg min s L π (v0...v j 1 ) (s) end for return (v 0... v n/2 1 ) Insert (v 0... v n/2 1 ) as the new b 1... b n/2 Repeat SubSieve + (L, d) for d = n 1, n 2,..., d min Hope that iteration d min + 1 provided a quasi-hkz basis. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

68 Concrete prediction with quasi-hkz preprocessing Pessimistic prediction for (1) d n ln 4/3 ln(n/2π) Optimistic prediction for (1) d n ln 4/3 ln(n/2πe) d pessimistic simulation pessimistic approximation optimistic simulation optimistic approximation n Figure: Predictions of the maximal successful choice of d min. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

69 Baseline Implementation (V0) Re-implemented GaussSieve [Micciancio Voulgaris 2010] No gaussian sampling Initial sphericity of L doesn t seem to matter Initial vectors can be made much shorter speed-up Prevent collisions using a hash table Terminate when the ball 4/3 gh(l) is half-saturated Sort only periodically Can use faster data-structures Vectors represented in bases B and GramSchmidt(B) Required to work in projected-sublattices Kernel in c++, control in python Calls to fpylll to maintain B and GramSchmidt(B) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

70 Baseline Implementation (V0) Re-implemented GaussSieve [Micciancio Voulgaris 2010] No gaussian sampling Initial sphericity of L doesn t seem to matter Initial vectors can be made much shorter speed-up Prevent collisions using a hash table Terminate when the ball 4/3 gh(l) is half-saturated Sort only periodically Can use faster data-structures Vectors represented in bases B and GramSchmidt(B) Required to work in projected-sublattices Kernel in c++, control in python Calls to fpylll to maintain B and GramSchmidt(B) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

71 Baseline Implementation (V0) Re-implemented GaussSieve [Micciancio Voulgaris 2010] No gaussian sampling Initial sphericity of L doesn t seem to matter Initial vectors can be made much shorter speed-up Prevent collisions using a hash table Terminate when the ball 4/3 gh(l) is half-saturated Sort only periodically Can use faster data-structures Vectors represented in bases B and GramSchmidt(B) Required to work in projected-sublattices Kernel in c++, control in python Calls to fpylll to maintain B and GramSchmidt(B) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

72 Baseline Implementation (V0) Re-implemented GaussSieve [Micciancio Voulgaris 2010] No gaussian sampling Initial sphericity of L doesn t seem to matter Initial vectors can be made much shorter speed-up Prevent collisions using a hash table Terminate when the ball 4/3 gh(l) is half-saturated Sort only periodically Can use faster data-structures Vectors represented in bases B and GramSchmidt(B) Required to work in projected-sublattices Kernel in c++, control in python Calls to fpylll to maintain B and GramSchmidt(B) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

73 Baseline Implementation (V0) Re-implemented GaussSieve [Micciancio Voulgaris 2010] No gaussian sampling Initial sphericity of L doesn t seem to matter Initial vectors can be made much shorter speed-up Prevent collisions using a hash table Terminate when the ball 4/3 gh(l) is half-saturated Sort only periodically Can use faster data-structures Vectors represented in bases B and GramSchmidt(B) Required to work in projected-sublattices Kernel in c++, control in python Calls to fpylll to maintain B and GramSchmidt(B) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

74 XOR-POPCNT trick (V0 V1) Already used in Sieving [Fitzpatrick et al. 2015]. More generally know as SimHash [Charikar 2002]. Idea: Pre-filter pairs (v, w) L with a fast compressed test. Choose a spherical code C = {c 1... c k } S n and a threshold t k/2 Precompute compressions ṽ = Sign( v, c i ) {0, 1} k Only test v ± w v if HammingWeight(v w) k/2 t. Asymptotic speed-up Θ(n/ log n)? In practice, k = 128 (2 words), t = 18: about 10 cycles per pairs. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

75 Progressive Sieving (V1 V2) Concurrently and independetly invented in [Mariano Laarhoven 2018]. Idea: Increase the dimension progressively. Recursively, Sieve in the lattice L(b 1,... b n 1 ) Start the sieve in dimension n with many short-ish vectors Fresh vectors get reduced much faster thanks to this initial pool. Refer to [Mariano Laarhoven 2018] for a full analysis of this trick. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

76 Dimensions for Free (V2 V3) Apply the quasi-hkz preprocessing strategy Do not force the choice of d min Simply increase d until the shortest vector is found. d pessimistic simulation pessimistic approximation optimistic simulation optimistic approximation Experimental average n Figure: Predictions experiments for d min. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

77 Performances T (sec. ) n Fit V0: n Fit V1: n Fit V2: n Fit V3: Fit Enum: 2 V0 (Sieve) V1 (Sieve) V2 (Sieve) V3 (SubSieve) fplll's Pruned Enum n lnn Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59 n

78 Comparison to other Sieving implementation Algorithms V0 V1 V2 V3 [MV10] [FBB + 14] [ML17] [HK17] Features XOR-POPCNT trick x x x x pogressive sieving x x SubSieve x LSH (more mem.) tuple (less mem.) x x Dimension Running times n = s 49s 8s 0.9s 464s 79s 13s 1080s n = s 10s 23933s 4500s 250s 33000s n = s s 94700s CPU freq. (GHz) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

79 Summary Sieving vs. Sieving Exploit all outputs of Sieve Dimensions for Free Our implementation is 10x faster than all previous Sieving It does not use LSH techniques: further speed-up expected Sieving vs. Enumeration Only a factor 4x slower than Enum for dimensions Guesstimates a cross-over at dim 90 with further improvements (LSH/LSF, fine-tuning, vectorization,... ) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

80 Summary Sieving vs. Sieving Exploit all outputs of Sieve Dimensions for Free Our implementation is 10x faster than all previous Sieving It does not use LSH techniques: further speed-up expected Sieving vs. Enumeration Only a factor 4x slower than Enum for dimensions Guesstimates a cross-over at dim 90 with further improvements (LSH/LSF, fine-tuning, vectorization,... ) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

81 A l l t h`åt wˆo r kffl f o rffl `affl sfi i n`g l e `gˇr`åi nffl `o f sfi`a n`d! xor a bit 7,b jr z,bc_nneg ld hl,0 xor a sbc hl,bc push hl pop bc ld a,1 M u sfi t I r`eṗ`e åt `ãl l `o f t h i s f o rffl `e àc hffl `gˇr`åi nffl? Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

82 A l l t h`åt wˆo r kffl f o rffl `affl sfi i n`g l e `gˇr`åi nffl `o f sfi`a n`d! xor a bit 7,b jr z,bc_nneg ld hl,0 xor a sbc hl,bc push hl pop bc ld a,1 M u sfi t I r`eṗ`e åt `ãl l `o f t h i s f o rffl `e àc hffl `gˇr`åi nffl? Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

83 H u mffl. L`eˇt m`e t h i n kffl. M`a y bfle wfle `d`o nffl t n`e e dffl t o r`eṗ`e åt `ãl l `o f t h i s... Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

84 H u mffl. L`eˇt m`e t h i n kffl. M`a y bfle wfle `d`o nffl t n`e e dffl t o r`eṗ`e åt `ãl l `o f t h i s... Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

85 The Generalized Sieve Kernel (G6K, pronounced /ζe.si.ka/) 5 5 Work in Progress with M. Albrecht, E. Postlethwaite, G. Herold, E. Kirshanova, M. Stevens Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

86 1st design principle: Go Green! Idea: Recycle vectors between overlapping blocks. Rather than an function serving as an SVP oracle, design a stateful machine that takes advantages of the overlapping instances. In other words: A nffl A l g o r i t h m i`c A n t `o nffl `affl S`a n`d p i l e, `c år r yˇi n`g `affl bˆàg `o f vfle cˇt o r s `o nffl i tṡ bˆàc kffl. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

87 1st design principle: Go Green! Idea: Recycle vectors between overlapping blocks. Rather than an function serving as an SVP oracle, design a stateful machine that takes advantages of the overlapping instances. In other words: A nffl A l g o r i t h m i`c A n t `o nffl `affl S`a n`d p i l e, `c år r yˇi n`g `affl bˆàg `o f vfle cˇt o r s `o nffl i tṡ bˆàc kffl. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

88 Moving with a bag of vectors Relations between the projected sublattices: L = L [1:n] L [1:n 1]... L [1:2]... L [1:1] π π π L [2:n] L [2:n 1]... L [2:2]. L [n 1:n] L [n 1:n 1] π L [n:n] π can be inverted in many ways. Choose π 1 to be the Babai lift: the shortest of all possible lifts. All maps, π 1, π preserve shortness somewhat Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

89 Moving with a bag of vectors Relations between the projected sublattices: L = L [1:n] L [1:n 1]... L [1:2]... L [1:1] π π π L [2:n] L [2:n 1]... L [2:2]. L [n 1:n] L [n 1:n 1] π L [n:n] π can be inverted in many ways. Choose π 1 to be the Babai lift: the shortest of all possible lifts. All maps, π 1, π preserve shortness somewhat Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

90 Moving with a bag of vectors Relations between the projected sublattices: L = L [1:n] L [1:n 1]... L [1:2]... L [1:1] π π π L [2:n] L [2:n 1]... L [2:2]. L [n 1:n] L [n 1:n 1] π L [n:n] π can be inverted in many ways. Choose π 1 to be the Babai lift: the shortest of all possible lifts. All maps, π 1, π preserve shortness somewhat Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

91 Moving with a bag of vectors L = L [1:n] L [1:n 1]... L [1:2] L [1:1] π π π L [2:n] L [2:n 1]... L [2:2].. L [n 1:n] L [n 1:n 1] π L [n:n] Change of context/block [l:r] : transform the vectors in the bag Extend-Right : Shrink-Left : π 1 Extend-Left : π (do nothing) (Babai lift) (project) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

92 2nd design principle: be flexible BKZ theory use exact-svp for each block consecutively, but maybe we re better off making different choices. Maintain a cadidate for insertion at each position Decide where to insert after sieving Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

93 3rd design principle: seize opportunities Algorithm 4 Sieve(L) L a set of N random vectors from L where N (4/3) n/2. while (v, w) L 2 such that v w < v do v v w end while return L Even if v w v, it could be worth considering the lifts of v w. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

94 The abstract machine State: A lattice basis B Positions 0 l l r d. [l : r] the sieving context, and [l : r] the lifting context. A database db of N vectors in L [l:r] (preferably short). Insertion candidates c l,..., c l where c i L [i:r] or c i =. Instructions: Sieve (S): make vector shorter, improve insertion candidates Extend Right, Shrink Left, Extend Left (ER, SL, EL): change the sieve-context, updating the database Insert (I): update the basis and the database Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

95 The ideal BKZ with G6K BKZ can be written very simply: Repeat {S; I; ER; } When starting the second Sieve, vectors are already quite short No need to restart progressive sieving from the beginning. The ER bug. It turns out that ER is not very compatible with our fastest sieve implementation. Somehow, the Sieve gets stuck in a subspace. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

96 The ideal BKZ with G6K BKZ can be written very simply: Repeat {S; I; ER; } When starting the second Sieve, vectors are already quite short No need to restart progressive sieving from the beginning. The ER bug. It turns out that ER is not very compatible with our fastest sieve implementation. Somehow, the Sieve gets stuck in a subspace. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

97 Pump Before: SubSieve f : Reset 0,f,f, (ER, S) d f, I 0, I 1,..., I d f. No issues with EL Progressive-Sieving toward the left instead. Can now Sieve again after insertion Can now insert the best candidate rather than a pre-chosen one Pump l,l,r,s : Reset l,r,r, pump-up {}}{ (EL, S) r l, pump-down {}}{ (I, S) r l. Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

98 WorkOut Workout: Pumps of increasing strength WorkOut κ,β,f,f +,s : Pump κ,κ+β f +,κ+β,s, Pump κ,κ+β 2f +,κ+β,s, Pump κ,κ+β 3f +,κ+β,s,... Pump κ,κ+f,κ+β,s, Termination condition can vary (e.g. fixed number of dims for free, or reached satisfying shortest vector) steps size of pump strength is not necessarly 1 Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

99 Pump and Jump Block is left somewhat reduced by the pump in the previous block: no need for a full workout. Many short vectors inserted, little improvement left around here: directly Jump far away. PumpnJumpBKZ β,f,j : Pump 0,f,β, Pump j,j+f,j+β, Pump 2j,2j+f,2j+β,... Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

100 Implementation 3 layers c++: multi-threaded heavy duty operation (Sieve, db updates) cython: middleware, basis maintainance python: control, tuning, and monitoring Several Sieve inside: Standard Gauss-Sieve (mono-threaded) Mem = 2.208n+o(n), Time = 2.415n+o(n) Becker-Gama-Joux with 1 level of filtration (multi-threaded) k-sieve k = 2, 3 (multi-threaded) Mem = 2.208n+o(n), Time = 2.349n+o(n) Mem = 2.208n+o(n), Time = 2.349n+o(n) Mem = 2.189n+o(n), Time = 2.372n+o(n) Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

101 Performances: Exact-SVP time (core-hours) Time in seconds for exact-svp 10 3 BKZ+pruned enum (fplll) G6K WorkOut dim About 4 extra dims for free Cross-over with enum at dim 70 Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

102 Records: SVP-challenges time (core-hours) SVP-challenges 10 6 BKZ+pruned enum RSR/discrete pruning G6K WorkOut dim Solved challenges up to dim 155, with 80 cores in 14 days About 400x faster than previous records Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

103 Records: LWE-challenges Red: solved (prior) Blue: solved (ours) Green: unsolved. New cost-balancing trick improving upon the prediction of [AGVW17] Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

104 Stay tuned Paper to be finalized Implementation will be made open-source Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59

105 Thanks!? Léo Ducas (CWI, Amsterdam) G6K or, the Alg. Ant and the Sandpile 24 Sept / 59