Beyond Hellman s Time-Memory Trade-Offs with Applications to Proofs of Space

Transcription

1 Beyond Hellman s Time-Memory Trade-Os with Applications to Proos o Space Hamza Abusalah 1, Joël Alwen 1, Bram Cohen 2, Danylo Khilko 3, Krzyszto Pietrzak 1, and Leonid Reyzin 4 1 Institute o Science and Technology Austria, habusalah jalwen pietrzak@ist.ac.at 2 Chia Network, bram@chia.network 3 ENS Paris, dkhilko@ukr.net 4 Boston University, reyzin@cs.bu.edu Abstract. Proos o space (PoS) were suggested as more ecological and economical alternative to proos o work, which are currently used in blockchain designs like Bitcoin. The existing PoS are based on rather sophisticated graph pebbling lower bounds. Much simpler and in several aspects more eicient schemes based on inverting random unctions have been suggested, but they don t give meaningul security guarantees due to existing time-memory trade-os. In particular, Hellman showed that any permutation over a domain o size N can be inverted in time T by an algorithm that is given S bits o auxiliary inormation whenever S T N (e.g. S = T N 1/2 ). For unctions Hellman gives a weaker attack with S 2 T N 2 (e.g., S = T N 2/3 ). To prove lower bounds, one considers an adversary who has access to an oracle : [N] [N] and can make T oracle queries. The best known lower bound is S T Ω(N) and holds or random unctions and permutations. We construct unctions that provably require more time and/or space to invert. Speciically, or any constant k we construct a unction [N] [N] that cannot be inverted unless S k T Ω(N k ) (in particular, S = T N k/(k+1) ). Our construction does not contradict Hellman s time-memory trade-o, because it cannot be eiciently evaluated in orward direction. However, its entire unction table can be computed in time quasilinear in N, which is suicient or the PoS application. Our simplest construction is built rom a random unction oracle g : [N] [N] [N] and a random permutation oracle : [N] [N] and is deined as h(x) = g(x, x ) where (x) = π((x )) with π being any involution without a ixed point, e.g. lipping all the bits. For this unction we prove that any adversary who gets S bits o auxiliary inormation, makes at most T oracle queries, and inverts h on an ɛ raction o outputs must satisy S 2 T Ω(ɛ 2 N 2 ). 1 Introduction A proo o work (PoW), introduced by Dwork and Naor [DN93], is a proo system in which a prover P convinces a veriier V that he spent some computation with

2 respect to some statement x. A simple PoW can be constructed rom a unction H( ), where a proo with respect to a statement x is simply a salt s such that H(s, x) starts with t 0 s. I H is modelled as a random unction, P must evaluate H on 2 t values (in expectation) beore he inds such an s. The original motivation or PoWs was prevention o spam and denial o service attacks, but today the by ar most important application or PoWs is securing blockchains, most prominently the Bitcoin blockchain, whose security is based on the assumption that the majority o computing power dedicated towards the blockchain comes rom honest users. This results in a massive waste o energy and other resources, as this mining is mostly done on dedicated hardware (ASICs) which has no other use than Bitcoin mining. In [DFKP15] proos o space (PoS) have been suggested as an alternative to PoW. The idea is to use disk space rather than computation as the main resource or mining. As millions o users have a signiicant amount o unused disk space available (on laptops etc.), dedicating this space towards securing a blockchain would result in almost no waste o resources. Let [N] denote some domain o size N. For convenience we ll oten assume that N = 2 n is a power o 2 and identiy [N] with {0, 1} n, but this is never crucial and [N] can be any other eiciently samplable domain. A simple idea or constructing a PoS is to have the veriier speciy a random unction : [N] [N] during the initialization phase, and have the prover compute the unction table o and sort it by the output values. 5 Then, during the proo phase, to convince the veriier that he really stores this table, the prover must invert on a random challenge. We will call this approach simple PoS ; we will discuss it in more detail in 1.3 below, and explain why it miserably ails to provide any meaningul security guarantees. Instead, existing PoS [DFKP15,RD16] are based on pebbling lower bounds or graphs. These PoS provide basically the best security guarantees one could hope or: a cheating prover needs Θ(N) space or time ater the challenge is known to make a veriier accept. Unortunately, compared to the (insecure) simple PoS, they have two drawbacks which make them more diicult to use as replacement or PoW in blockchains. First, the proo size is quite large (several MB instead o a ew bytes as in the simple PoS or Bitcoin s PoW). Second, the initialization phase requires two messages: the irst message, like in the simple PoS, is sent rom the veriier to the prover speciying a random unction, and second message, unlike in the simple PoS, is a commitment rom the prover to the veriier. 6 I such a pebbling-based PoS is used as a replacement or PoW in a blockchain design, the irst message can be chosen non-interactively by the miner (who plays the role o the prover), but the commitment sent in the second message is more 5 must have a short description, so it cannot be actually random. In practice the prover would speciy by, or example, a short random salt s or a cryptographic hash unction H, and set (x) = H(s, x). 6 Speciically, the prover computes a graph labelling o the vertices o a graph (which is speciied by the PoS) using, and then a Merkle tree commitment to this entire labelling, which must be sent back to the veriier. 2

3 tricky. In Spacemint (a PoS-based decentralized cryptocurrency [PPK + 15]), this is solved by having a miner put this commitment into the blockchain itsel beore he can start mining. As a consequence, Spacemint lacks the nice property o the Bitcoin blockchain where miners can join the eort by just listening to the network, and only need to speak up once they ind a proo and want to add it to the chain. 1.1 Our Results In this work we resurrect the simple approach towards constructing PoS based on inverting random unctions. This seems impossible, as Hellman s timememory trade-os which are the reason or this approach to ail can be generalized to apply to all unctions (see Section 1.4). For Hellman s attacks to apply, one needs to be able to evaluate the unction eiciently in orward direction. At irst glance, this may not seem like a real restriction at all, as inverting unctions which cannot be eiciently computed in orward direction is undecidable in general. 7 However, we observe that or unctions to be used in the simple PoS outlined above, the requirement o eicient computability can be relaxed in a meaningul way: we only need to be able to compute the entire unction table in time linear (or quasilinear) in the size o the input domain. We construct unctions satisying this relaxed condition or which we prove lower bounds on time-memory trade-os beyond the upper bounds given by Hellman s attacks. Our most basic construction o such a unction g : [N] [N] is based on a unction g : [N] [N] [N] and a permutation : [N] [N]. For the lower bound proo g and are modelled as truly random, and all parties access them as oracles. The unction is now deined as g (x) = g(x, x ) where (x) = π((x )) or any involution π without ixed points. For concreteness we let π simply lip all bits, denoted (x) = (x ). Let us stress that does not need to be a permutation it can also be a random unction 8 but we ll state and prove our main result or a permutation as it makes the analysis cleaner. In practice where one has to instantiate and g with something eicient one would rather use a unction, because it can be instantiated with a cryptographic hash unction like SHA-3 or (truncated) AES, 9 whereas we don t have good candidates or suitable permutations (at the very least needs to be one-way; and, unortunately, all 7 Consider the unction : N {0, 1} {0, 1} {0, 1} where (s, T ) = (b, T ) with b = 1 i the Turing machine T stops in s steps. Here deciding i (1, T ) has a pre-image at all requires to solve the halting problem. 8 I is a unction, we don t need π; the condition (x) = π((x )) can be replaced with simply (x) = (x ), x x. Note than now or some x there s no output g (x) at all (i.e., i x x : (x) (x )), and or some x there s more than one possible value or g (x). This is a bit unnatural, but such a g can be used or a PoS in the same way as i were a permutation. 9 As a concrete proposal, let AES n : {0, 1} 128 {0, 1} 128 {0, 1} n denote AES with the output truncated to n bits. We can now deine, g by a random key k {0, 1} 128 as (x) = AES n(k, 0 x n 1 ) and g(x) = AES n(k, 1 x n 1 ). As in practice n will be something like 30 50, which corresponds to space (which is n 2 n bits) 3

4 candidates we have or one-way permutations are number-theoretic and thus much less eicient). In Theorem 2 we state that or g as above, any algorithm which has a state o size S (that can arbitrarily depend on g and ), and inverts g on an ɛ raction o outputs, must satisy S 2 T Ω(ɛ 2 N 2 ). This must be compared with the best lower bound known or inverting random unctions (or permutations) which is ST = Ω(ɛN). We can urther push the lower bound to S k T Ω(ɛ k N k ) by Fig. 1. Illustration o lower bounds. Orange: the ST = Ω(N) lower bound or inverting random permutations or unctions. Dark green: the ideal bound where either T or S is Ω(N) as achieved by the pebbling-based PoS [DFKP15,RD16] (more precisely, the bound approaches the dark green line or large N). Light green: the lower bound S 2 T = Ω(N 2 ) or T N 2/3 or our most basic construction as stated in Theorem 2. Pink: the restriction T N 2/3 on T we need or our proo to go through can be relaxed to T N t/(t+1) by using t-wise collisions instead o pairwise collisions in our construction. The pink arrow shows how the bound improves by going rom t = 2 to t = 3. Purple: we can push the S 2 T = Ω(N 2 ) lower bound o the basic construction to S k T = Ω(N k ) by using k 1 levels o nesting. The purple arrow shows how the bound improves by going rom k = 2 to k = 3. nesting the construction; in the irst iteration o this nesting one replaces the inner unction with g. 10 These lower bounds are illustrated in Figure 1. In this paper we won t give a proo or the general construction, as the proo or the general construction doesn t require any new ideas, but just gets in the Gigabyte to Petabyte range. Using AES with the smallest 128 bit blocksize is suicient as 2n The dream version would be a result showing that one needs either S = Ω(N) or T = Ω(N) to invert. Our results approach this as k grows showing that S = T = Ω(N k/(k+1) ) is required. 4

5 more technical. We also expect the basic construction to be already suicient or constructing a secure PoS. Although or g there exists a time-memory trade-o S 4 T O(N 4 ) (say, S = T N 4/5 ), which is achieved by nesting Hellman s attack, 11 we expect this attack to only be o concern or extremely large N. 12 A caveat o our lower bound is that it only applies i T N 2/3. We don t see how to break our lower bound i T > N 2/3, and the restriction T N 2/3 seems to be mostly related to the proo technique. One can improve the bound to T N t/(t+1) or any t by generalizing our construction to t-wise collisions. One way to do this i is a permutation and t divides N is as ollows: let g : [N] t [N] and deine g (x) = g(x, x 1,..., x t 1 ) where or some partition S 1,..., S N/t, S i = t o [N] the values (x), (x 1 ),..., (x t 1 ) contain all elements o a partition S i and x 1 < x 2 <... < x t Proos o Space A proo o space as deined in [DFKP15] is a two-phase protocol between a prover P and a veriier V, where ater an initial phase P holds a ile F o size N, 13 whereas V only needs to store some small value. The running time o P during this phase must be at least N as P has to write down F which is o size N, and we require that it s not much more, quasilinear in N at most. V on the other hand must be very eicient, in particular, its running time can be polynomial in a security parameter, but must be basically independent o N. Then there s a proo execution phase which typically will be executed many times over a period o time in which V challenges P to prove it stored F. The security requirement states that a cheating prover P who only stores a ile F o size signiicantly smaller than N either ails to make V accept, or must invest a signiicant amount o computation, ideally close to P s cost during initialization. Note that we cannot hope to make it more expensive than that as a cheating P can always just store the short communication during initialization, and then reconstruct all o F beore the execution phase. 11 Inormally, nesting Hellman s attack works as ollows. Note that i we could eiciently evaluate g (.), we could use Hellman s attack. Now to evaluate g we need to invert. For this make a Hellman table to invert, and use this to semi-eiciently evaluate g (.). More generally, or our construction with nesting parameter k (when the lower bound is S k T Ω(N k )) the nested Hellman attack applies i S 2k T O(N 2k ). 12 The reason is that or this nested attack to work, we need tables which allow to invert with very high probability, and in this case the tables will store many redundant values. So the hidden constant in the S 4 T O(N 4 ) bound o the nested attack will be substantial. 13 We use the same letter N or the space committed by an honest prover in a PoS as we used or the domain size o the unctions discussed in the previous section to emphasize that in our construction o a PoS rom a unction these will be roughly the same. We ll come back to this in Remark 2 in the next section. 5

6 1.3 A Simple PoS that Fails Probably the irst candidate or a PoS scheme that comes to mind is to have during the initalization phase V send the (short) description o a random behaving unction : [N] [N] to P, who then computes the entire unction table o and stores it sorted by the outputs. During proo execution V will pick a random x [N], and then challenge P to invert on y = (x). 14 An honest prover can answer any challenge y by looking up an entry (x, y) in the table, which is eicient as the table is sorted by the y s. At irst one might hope this provides good security against any cheating prover; intuitively, a prover who only stores N log N bits (i.e., uses space suicient to only store N output labels o length log N) will not have stored a value x 1 (y) or most y s, and thus must invert by brute orce which will require Θ(N) invocations to. Unortunately, even i is modelled as a truly random unction, this intuition is totally wrong due to Hellman s time-memory trade-os, which we ll discuss in the next section. The goal o this work is to save this elegant and simple approach towards constructing PoS. As discussed beore, or our unction g : [N] [N] (deined as g (x) = g(x, x ) where (x) = (x )) we can prove better lower bounds than or random unctions. Instantiating the simple PoS with g needs some minor adaptions. V will send the description o a unction g : [N] [N] [N] and a permutation : [N] [N] to P. Now P irst computes the entire unction table o and sorts it by the output values. Note that with this table P can eiciently invert. Then P computes (and sorts) the unction table o g (using that g (x) = g(x, 1 ((x))). Another issue is that in the execution phase V can no longer compute a challenge as beore i.e. y = g (x) or a random x as it cannot evaluate g. Instead, we let V just pick a random y [N]. The prover P must answer this challenge with a tuple (x, x ) s.t. (x) = (x ) and g(x, x ) = y (i.e., g (x) = y). Just sending the preimage x o g or y is no longer suicient, as V is not able to veriy i g (x) = y without x. Remark 1 (Completeness and Soundness Error). This protocol has a signiicant soundness and completeness error. On the one hand, a cheating prover P who only stores, say 10%, o the unction table, will still be able to make V accept in 10% o the cases. On the other hand, even i g behaves like a random unction, an honest prover P will only be able to answer a 1 1/e raction ( 63%) o the challenges y [N], as some will simply not have a preimage under g. 15 When used as a replacement or PoW in cryptocurrencies, neither the soundness nor the completeness error are an issue. I this PoS is to be used in a context 14 Instead o storing all N tuples (x, (x)) (sorted by the 2nd entry), which takes 2N log N bits, one can compress this list by almost a actor 2 using the act that the 2nd entry is sorted, and another actor 1 1/e by keeping only one entry whenever there are multiple tuples with the same 2nd entry, thus requiring 0.632N log N bits. 15 Throwing N balls in N bins at random will leave around N/e bins empty, so g s outputs will miss N/e 0.37 N values in [N]. 6

7 where one needs negligible soundness and/or completeness, one can use standard repetition tricks to ampliy the soundness and completeness, and make the corresponding errors negligible. 16 Remark 2 (Domain vs. Space). When constructing a PoS rom a unction with a domain o size N, the space the honest prover requires is around N log N bits or the simple PoS outlined above (where we store the sorted unction table o a unction : [N] [N]), and roughly twice that or our basic construction (where we store the unction tables o g : [N] [N] and : [N] [N]). Thus, or a given amount N o space the prover wants to commit to, it must use a unction with domain N N / log(n ). In particular, the time-memory tradeos we can prove on the hardness o inverting the underlying unction translate directly to the security o the PoS. 1.4 Hellman s Time-Memory Trade Os Hellman [Hel80] showed that any permutation p : [N] [N] can be inverted using an algorithm that is given S bits o auxiliary inormation on p and makes at most T oracle queries to p( ), where (Õ below hides log(n)o(1) actors) S T Õ(N) e.g. when S = T N 1/2. (1) Hellman also presents attacks against random unctions, but with worse parameters. A rigorous bound was only later proven by Fiat and Naor [FN91] where they show that Hellman s attack on random unctions satisies S 2 T Õ(N 2 ) e.g. when S = T N 2/3. (2) Fiat and Naor [FN91] also present an attack with worse parameters which works or any (not necessarily random) unction, where S 3 T Õ(N 3 ) e.g. when S = T N 3/4. (3) The attack on a permutation p : [N] [N] or a given T is easy to explain: Pick any x [N] and deine x 0, x 1,... as x 0 = x, x i+1 = p(x i ), let l N 1 be minimal such that x 0 = x l. Now store the values x T, x 2T,..., x (l mod T )T in a sorted list. Let us assume or simplicity that l 1 = N, so x 0,..., x l 1 cover the entire domain (i this is not the case, one picks some x not yet covered and makes a new table or the values x 0 = x, x 1 = p(x 0 ),...). This requires storing S = N/T values. I we have this table, given a challenge y to invert, we just apply p to y until we hit some stored value x it, then continue applying p to x (i 1)T until 16 To decrease the soundness error rom 0.37 to negligible, the veriier can ask the prover to invert g on t N independent random challenges in [N]. In expectation g will have a preimage on 0.63 t challenges. The probability that say at least 0.5 t o the challenges have a preimage is then exponentially (in t) close to 1 by the Cherno bound. So i we only require the prover to invert hal the challenges, the soundness error becomes negligible. 7

8 we hit y, at which point we ound the inverse p 1 (y). By construction this attack requires T invocations to p. The attack on general unctions is more complicated and gives worse bounds as we don t have such a nice cycle structure. In a nutshell, one computes several dierent chains, where or the jth chain we pick some random h j : [N] [N] and compute x 0, x 1,..., x n as x i = (h j (x i 1 )). Then, every T th value o the chain is stored. To invert a challenge y we apply (h 1 ( )) sequentially on input y up to T times. I we hit a value x it we stored in the irst chain, we try to invert by applying (h 1 ( )) starting with x (i 1)T. 17 I we don t succeed, continue with the chains generated by (h 2 ( )), (h 3 ( )),... until the inverse is ound or all chains are used up. This attack will be successul with high probability i the chains cover a large raction o s output domain. 1.5 Samplability is Suicient or Hellman s Attack One reason the lower bound or our unction g : [N] [N] (deined as g (x) = g(x, x ) where (x) = (x )) does not contradict Hellman s attacks is the act that g cannot be eiciently evaluated in orward direction. One can think o simpler constructions such as g (x) = g(x, 1 (x)) which also have this property, but observe that Hellman s attack is easily adapted to break g. More generally, Hellman s attack doesn t require that the unction can be eiciently computed in orward direction, it is suicient to have an algorithm that eiciently samples random input/output tuples o the unction. This is possible or g as or a random z the tuple (z), g((z), z) is a valid input/output: g ((z)) = g((z), 1 ((z)) = g((z), z). To adapt Hellman s attack to this setting where we just have an eicient input/output sampler σ or replace the (h i ( )) s in the attack described in the previous section with σ (h i ( )). 1.6 Lower Bounds De, Trevisan and Tulsiani [DTT10] (building on work by Yao [Yao90], Gennaro- Trevisan [GT00] and Wee [Wee05]) prove a lower bound or inverting random permutations, and in particular show that Hellman s attack as stated in Eq.(1) is optimal: For any oracle-aided algorithm A, it holds that or most permutations p : [N] [N], i A is given advice (that can arbitrarily depend on p) o size S, makes at most T oracle queries and inverts p on ɛn values, we have S T Ω(ɛN). Their lower bound proo can easily be adapted to random unctions : [N] [N], but note that in this case it is no longer tight, i.e., matching Eq.(2). Barkan, Biham, and Shamir [BBS06] show a matching S 2 T Ω(N 2 ) lower bound or a restricted class o algorithms. 17 Unlike or permutations, there s no guarantee we ll be successul, as the challenge might lie on a branch o the unction graph dierent rom the one that includes x (i 1) T. 8

9 1.7 Proo Outline The starting point o our proo is the S T Ω(ɛN) lower bound or inverting random permutations by De, Trevisan and Tulsiani [DTT10] mentioned in the previous section. We sketch their simple and elegant proo, with a minor adaption to work or unctions rather than permutations in Appendix A. The high level idea o their lower bound proo is as ollows: Assume an adversary A exists, which is given an auxiliary string aux, makes at most T oracle queries and can invert a random permutation p : [N] [N] on an ɛ raction o [N] with high probability (aux can depend arbitrarily on p). One then shows that given (black box access to) A aux ( ) de = A(aux, ) it s possible to compress the description o p rom log(n!) to log(n!) bits or some > 0. As a random permutation is incompressible (ormally stated as Fact 1 in 2 below), the bits we saved must come rom the auxiliary string given, so S = aux. To compress p, one now inds a subset G [N] where (1) A inverts successully, i.e., or all y p(g) = {p(x) : x G} we have A p aux(y) = p 1 (y) and (2) A never makes a query in G, i.e., or all y G all oracle queries made by A p aux(y) are in [N] G (except or the last query which we always assume is p 1 (y)). The compression now exploits the act that one can learn the mapping G p(g) given aux, an encoding o the set p(g), and the remaining mapping [N] G p([n] G). While decoding, one recovers G p(g) by invoking A p aux( ) on all values p(g) (answering all oracle queries using [N] G p([n] G), the irst query outside [N] G will be the right value by construction). Thus, we compressed by not encoding the mapping G p(g), which will save us G log(n) bits, however we have to pay an extra G log(en/ G ) bits to encode the set p(g), so overall we compressed by G log( G /e) bits, and thereore S G assuming G 2e. Thus the question is how large a set G can we choose. A simple probabilistic argument, basically picking values at random until it s no longer possible to extend G, shows that we can always pick a G o size at least G ɛn/t, and we conclude S ɛn/t assuming T ɛn/2e. In the De et al. proo, the size o the good set G will always be close to ɛn/t, no matter how A aux actually behaves. In this paper we give a more ine grained analysis introducing a new parameter T g as discussed next. The T g parameter. Inormally, our compression algorithm or a unction g : [N] [N] goes as ollows: Deine the set I = {x : A g aux(g(x)) = x} o values where A g aux inverts g(i), by assumption I = ɛn. Now we can add values rom I to G as long as possible, every time we add a value x, we spoil up to T values in I, where we say x gets spoiled i A g aux(g(x)) makes oracle query x, and thus we will not be able to add x to G in the uture. As we start with I = ɛn, and spoil at most T values or every value added to G, we can add at least ɛn/t values to G. This is a worst case analysis assuming A g aux really spoils close to T values every time we add a value to G, but potentially A g aux behaves nicer and on average spoils less. In the proo o Lemma 1 we take advantage o this and extend G as 9

10 long as possible, ending up with a good set G o size at least ɛn/2t g or some 1 T g T. Here T g is the average number o elements we spoiled or every element added to G. This doesn t help to improve the De et al. lower bound, as in general T g can be as large as T in which case our lower bound S T g Ω(ɛN) coincides with the De at al. S T Ω(ɛN) lower bound. 18 But this more ine grained bound will be a crucial tool to prove the lower bound or g. Lower Bound or g. We now outline the proo idea or our lower bound S 2 T Ω(ɛ 2 N 2 ) or inverting g (x) = g(x, x ), (x) = (x ) assuming g : [N] [N] [N] is a random unction and : [N] [N] is a random permutation. We assume an adversary A g, aux exists which has oracle access to, g and inverts g : [N] [N] on a set J = {y : g (A,g aux(y)) = y} o size J = ɛn. I the unction table o is given, g : [N] [N] is a random unction that can be eiciently evaluated, and we can prove a lower bound S T g Ω(ɛN) as outlined above. At this point, we make a case distinction, depending on whether T g is below or above T. I T g < T our S T g Ω(ɛN) bound becomes S 2 T Ω(ɛ 2 N 2 ) and we are done. The more complicated case is when T g T where we show how to use the existence o A,g aux to compress instead o g. Recall that T g is the average number o values that got spoiled while running the compression algorithm or g, that means, or every value added to the good set G, A,g aux made on average T g resh queries to g. Now making resh g queries isn t that easy, as it requires inding x, x where (x) = (x ). We can use A,g aux which makes many such resh g queries to compress : when A,g aux makes two queries x, x where (x) = (x ), we just need to store the irst output (x), but won t need the second (x ) as we know it is (x). For decoding we also must store when exactly A,g aux makes the queries x and x, more on this below. Every time we invoke A,g aux or compression as just outlined, up to T outputs o may get spoiled in the sense that A,g aux makes an query that we need to answer at this point, and thus it s no longer available to be compressed later. As A,g aux can spoil up to T queries on every invocation, we can hope to invoke it at least ɛn/t times beore all the queries are spoiled. Moreover, on average A,g aux makes T g resh g queries, so we can hope to compress around T g outputs o with every invocation o A,g aux, which would give us around T g ɛn/t compressed values. This assumes that a large raction o the resh g queries uses values o that were not spoiled in previous invocations. The technical core o our proo is a combinatorial lemma which we state and prove in 5, which implies that it s always possible to ind a sequence o inputs to A,g aux such that this is the case. 18 Note that or the adversary as speciied by Hellman s attack against permutations as outlined in 1.4 we do have T g = T, which is not surprising given that or permutations the De at al. lower bound matches Hellman s attack. 10

11 Concretely, we can always ind a sequence o inputs such that at least T g ɛn/32t values can be compressed Notation and Basic Facts We use brackets like (x 1, x 2,...) and {x 1, x 2,...} to denote ordered and unordered sets, respectively. We ll usually reer to unordered sets simply as sets, and to ordered sets as lists. [N] denotes some domain o size N, or notational convenience we assume N = 2 n is a power o two and identiy [N] with {0, 1} n. For a unction : [N] [M] and a set S [N] we denote with (S) the set {(S[1]),..., (S[ S ])}, similarly or a list L [N] we denote with (L) the list ((L[1]),..., (L[ L ])). For a set X, we denote with x X that x is assigned a value chosen uniormly at random rom X. Fact 1 (rom [DTT10]) For any randomized encoding procedure Enc : {0, 1} r {0, 1} n {0, 1} m and decoding procedure Dec : {0, 1} r {0, 1} m {0, 1} n where we have m n log(1/δ). Pr x {0,1} n,ρ {0,1} r[dec(ρ, Enc(ρ, x)) = x] δ Fact 2 I a set X is at least ɛ dense in Y, i.e., X Y, X ɛ Y, and Y is known, then X can be encoded using X log(e/ɛ) bits. To show this we use the inequality ( ) n ɛn (en/ɛn) ɛn, which implies log ( n ɛn) ɛn log(e/ɛ). 3 A Lower Bound or Functions The ollowing theorem is basically rom [DTT10], but stated or unctions not permutations. Theorem 1. Fix some ɛ 0 and an oracle algorithm A which on any input makes at most T oracle queries. I or every unction : [N] [N] there exists a string aux o length aux = S such that then Pr y [N] [(A aux(y)) = y] ɛ T S Ω(ɛN). (4) The theorem ollows rom Lemma 1 below using Fact 1 as ollows: in Fact 1 let δ = 0.9 and n = N log N, think o x as the unction table o a unction : [N] [N]. Then Enc(ρ, aux, ) N log N log(1/0.9), together with the upper bound on the encoding rom Eq.(6) this implies Eq.(4). Note that the extra assumption that T ɛn/40 in the lemma below doesn t matter, as i it s not satisied the theorem is trivially true. For now the value T g in the lemma below is not important and the reader can just assume T g = T. 19 The constant 32 here can be decreased with a more ine-grained analysis, we opted or a simpler proo rather than optimising this constant. 11

12 Lemma 1. Let A, T, S, ɛ and be as in Theorem 1, and assume T ɛn/40. There are randomized encoding and decoding procedures Enc, Dec such that i : [N] [N] is a unction and or some aux o length aux = S then Pr y [N] [(A aux(y)) = y] ɛ Pr ρ {0,1} r[dec(ρ, Enc(ρ, aux, )) = ] 0.9 (5) and the length o the encoding is at most or some T g, 1 T g T. Enc(ρ, aux, ) N log N ɛn + S + log(n) (6) }{{} 2T g = 3.1 Proo o Lemma 1 The Encoding and Decoding Algorithms. In Algorithms 1 and 2, we always assume that i A aux(y) outputs some value x, it makes the query (x) at some point. This is basically w.l.o.g. as we can turn any adversary into one satisying this by making at most one extra query. I at some point A aux(y) makes an oracle query x where (x) = y, then we also w.l.o.g. assume that right ater this query A outputs x and stops. Note that i A is probabilistic, it uses random coins which are given as input to Enc, Dec, so we can make sure the same coins are used during encoding and decoding. The Size o the Encoding. We will now upper bound the size o the encoding o G, (Q ), ( q 1,..., q G ), ([N] {G 1 Q }) as output in line (15) o the Enc algorithm. Let T g := B / G be the average number o elements we added to the bad set B or every element added to the good set G, then G ɛn/2t g. (7) To see this we note that when we leave the while loop (see line (8) o the algorithm Enc) it holds that B J /2 = ɛn/2, so G = B /T g J /2T g = ɛn/2t g. G: Instead o G we will actually encode the set π 1 (G) = {c 1,..., c G }. From this the decoding Dec (who gets ρ, and thus knows π) can then reconstruct G = π(π 1 (G)). We claim that the elements in c 1 < c 2 <... < c G are whp. at least ɛ/2 dense in [c G ] (equivalently, c G 2 G /ɛ). By Fact 2 we can thus encode π 1 (G) using G log(2e/ɛ)+log N bits (the extra log N bits are used to encode the size o G which is required so decoding later knows how to parse the encoding). To see that the c i s are ɛ/2 dense whp. consider line (9) in Enc which states c := min{c > c : y c {J \ B}}. I we replace J \ B 12

13 Algorithm 1 Enc 1: Input: A, aux, randomness ρ and a unction : [N] [N] to compress. 2: Initialize: B, G :=, c := 1 3: Throughout we identiy [N] with {0,..., N 1}. 4: Pick a random permutation π : [N] [N] (using random coins rom ρ) 5: Let J := {y : (A aux(y)) = y}, J = ɛn The set J where A inverts. I A is probabilistic, use random coins rom ρ. 6: For i = 0,..., N 1 deine y i := π(i). Randomize the order 7: For y J let the list q(y) contain all queries made by A (y) except the last query (which is x s.t. (x) = y). 8: while B < J /2 do While the bad set contains less than hal o J 9: c := min{c > c : y c {J \ B}} Increase c to the next y c in J \ B 10: G := G y c Add this y c to good set 11: B := B ((q(y c)) J) Add spoiled queries to bad set 12: end while 13: Let G = {g 1,..., g G }, Q = (q(g 1),..., q(g G )), and deine Q = (q 1,..., q G ), q i q(g i) to contain only the resh queries in Q by deleting all but the irst occurrence o every element. E.g. i (q(g 1), q(g 2)) = ((1, 2, 3, 1), (2, 4, 5, 4)) then (q 1, q 2) = ((1, 2, 3), (4, 5)). 14: Let G 1 = {A aux(y) : y G} 15: Output an encoding o (the set) G, (the lists) (Q ), ( q 1,..., q G ), ([N] {G 1 Q }) and (the string) aux. with J, then the c i s would be whp. close to ɛ dense as J is ɛ dense in [N] and the y i are uniormly random. As B < J /2, using J \ B instead o J will decrease the density by at most a actor 2. I we don t have this density, i.e., c G > 2 G /ɛ, we consider encoding to have ailed. (Q ): This is a list o Q elements in [N] and can be encoded using Q log N bits. ( q 1,..., q G ): Require G log T bits as q i q i T. A more careul argument (using that the q i are on average at most T g) requires G log(et g ) bits. ([N] {G 1 Q }): Requires (N G Q ) log N bits (using that G 1 Q = and G 1 = G ). aux: Is S bits long. Summing up we get Enc(ρ, aux, ) = G log(2e 2 T g /ɛ) + (N G ) log N + S + log N as by assumption T g T ɛn/40, we get log N log(2e 2 T g /ɛ) 1, and urther using (7) we get as claimed. Enc(ρ, aux, ) N log N ɛn 2T g + S + log N 13

14 Algorithm 2 Dec 1: Input: A, ρ and the encoding (G, (Q ), ( q 1,..., q G ), ([N] {G 1 Q }), aux). 2: Let π be as in Enc. 3: Let (g 1,..., g G ) be the elements o G ordered as they were added by Enc (i.e., π 1 (g i) < π 1 (g i+1) or all i). 4: Invoke A ( ) aux( ) sequentially on inputs g 1,..., g G using (Q ) to answer A aux s oracle queries. I A is probabilistic, use the same random coins rom ρ as in Enc. 5: Combine the mapping G 1 Q (G 1 Q ) (which we learned in the previous step) with [N] {G 1 Q } ([N] {G 1 Q }) to learn the entire [N] ([N]) 6: Output ([N]) 4 A Lower Bound or g(x, 1 ((x))) For a permutation : [N] [N] and a unction g : [N] [N] [N] we deine g : [N] [N] as g (x) = g(x, x ) where (x) = (x ) or equivalently g (x) = g(x, 1 ((x)) Theorem 2. Fix some ɛ > 0 and an oracle algorithm A which makes at most T (N/4e) 2/3 (8) oracle queries and takes an advice string aux o length aux = S. I or all unctions : [N] [N], g : [N] [N] [N] and some aux o length aux = S we have Pr [g (A,g aux(y)) = y] ɛ (9) y [N] then T S 2 Ω(ɛ 2 N 2 ). (10) The theorem ollows rom Lemma 2 below as we ll prove thereater. Lemma 2. Fix some ɛ 0 and an oracle algorithm A which makes at most T (N/4e) 2/3 oracle queries. There are randomized encoding and decoding procedures Enc g, Dec g and Enc, Dec such that i : [N] [N] is a permutation, g : [N] [N] [N] is a unction and or some advice string aux o length aux = S we have Pr [g (A,g aux(y)) = y] ɛ y [N] then Pr ρ {0,1} r[dec g(ρ,, Enc g (ρ, aux,, g)) = g] 0.9 (11) Pr ρ {0,1} r[dec (ρ, g, Enc (ρ, aux,, g)) = ] 0.9. (12) Moreover or every ρ, aux,, g there is a T g, 1 T g T, such that Enc g (ρ, aux,, g) N 2 log N ɛn + S + log N (13) }{{} 2T g = g 14

15 and i T g T Enc (ρ, aux,, g) log N! ɛnt g }{{} 64T = + S + log N. (14) We irst explain how Theorem 2 ollows rom Lemma 2 using Fact 1. Proo (o Theorem 2). The basic idea is to make a case analysis; i T g < T we compress g, otherwise we compress. Intuitively, our encoding or g achieving Eq.(13) makes both and g queries, but only g queries spoil g values. As the compression runs until all g values are spoiled, it compresses better the smaller T g is. On the other hand, the encoding or achieving Eq.(12) is derived rom our encoding or g, and it manages to compresses in the order o T g values o or every invocation (while spoiling at most T o the values), so the larger T g the better it compresses. Concretely, pick, g uniormly at random (and assume Eq.(9) holds). By a union bound or at least a 0.8 raction o the ρ Eq.(11) and Eq.(12) hold simultaneously. Consider any such good ρ, which together with, g ixes some T g, 1 T g T as in the statement o Lemma 2. Now consider an encoding Enc,g where Enc,g (ρ, aux,, g) outputs (, Enc g (ρ, aux,, g)) i T g < T, and (g, Enc (ρ, aux,, g)) otherwise. I T g < T we use (13) to get Enc,g (ρ, aux,, g) = + Enc g (ρ, aux,, g) + g ɛn/2t g +S+log N and now using Fact 1 (with δ = 0.8) we get S ɛn/2t g log N log(1/0.8) > ɛn/2 T log N log(1/0.8) and thus T S 2 Ω(ɛ 2 N 2 ) as claimed in Eq.(10). I T g T then we use Eq.(14) and Fact 1 and again get S ɛnt g /64T log N log(1/0.8) which implies Eq.(10) as T g T. Algorithm 3 Enc g 1: Input: A, ρ, aux,, g 2: Compute the unction table o g : [N] [N], g (x) = g(x, x ) where (x) = (x ). 3: Invoke E g Enc(A, g, aux, ρ) 4: Let g be the unction table o g([n] 2 ) = g(1, 1)... g(n, N), but with the N entries (x, x ) where (x) = (x ) deleted. 5: Output E g, g, aux. 15

16 Algorithm 4 Dec g 1: Input: A, ρ, and the encoding (E g, g, aux) o g. 2: Invoke g Dec(A, ρ, aux, E g ). 3: Reconstruct g rom g and g (this is possible as is given). 4: Output g([n] 2 ) Algorithm 5 Enc 1: Input: A, ρ, aux,, g 2: Invoke E g Enc(A, g, aux, ρ) Compute the same encoding o g as Enc g did. 3: For G E g, let G G, G = {z 1,..., z G } be as deined in proo o Lemma 2. 4: Initialize empty lists L, T, C :=. 5: or i = 1 to G do 6: Invoke A,g aux (z i). Using random coins rom ρ i A is probabilistic. 7: For each pair o queries x, x (made in this order during invocation) where (x) = (x ) and neither (x ) nor (x) is in L C, let (t, t ) be the indices (1 t < t T ) speciying when during invocation these queries were made. Append (t, t ) to T and append (x ) to C. 8: Append all images o oracle queries to made during invocation o A,g aux (z i) to L, except i the value is in L C. Append the images o all resh queries which were not compressed. 9: end or 10: Let L 1 (similarly C 1 ) contain the inputs corresponding to L, i.e., add x to L 1 when adding (x) to L. 11: Output an encoding o G, the list o values o queries L, the list o tuples T and the remaining outputs ([N L 1 C 1 ]) which were neither in the list L nor compressed. Algorithm 6 Dec 1: Input: A, ρ, aux, g and encoding (G, L, T, ([N L 1 2: Let G = {z 1,..., z G }. 3: or i = 1 to G do 4: Invoke A aux ( ),g ) using the lists L and T. 5: end or 6: For L 1 C 1 C 1 ])) o. (z i) reconstructing the answers to the irst oracle (which should be, C 1 as in Enc, we have learned the mapping (L 1 C 1 )). Reconstruct all o ([N]) by combining this with ([N] (L 1 7: Output ([N]) ) ((L 1 C 1 )). Proo (o Lemma 2). 16

17 The Encoding and Decoding Algorithms. The encoding and decoding o g are depicted in Algorithms 3 and 4, and those o in Algorithms 5 and 6. A,g aux( ) can make up to T queries in total to its oracles (.) and g(.). We will assume that whenever a query g(x, x ) is made, the adversary made queries (x) and (x ) beore. This is basically without loss o generality as we can turn any adversary into one adhering to this by at most tripling the number o queries. It will also be convenient to assume that A,g aux only queries g on its restriction to g, that is, or all g(x, x ) queries it holds that (x) = (x ), but the proo is easily extended to allow all queries to g as our encoding will store the unction table o g on all uninteresting inputs (x, x ), (x) (x ) and thus can directly answer any such query. As in the proo o Lemma 1, we don t explicitly show the randomness in case A is probabilistic. The Size o the Encodings. We will now upper bound the size o the encodings output by Enc g and Enc in Algorithms 3 and 5 and hence prove Eq.(13) and Eq.(14). Eq.(13) now ollows almost directly rom Theorem 1 as our compression algorithm Enc g or g : [N] [N] [N] simply uses Enc to compress g restricted to g : [N] [N], and thus compresses by exactly the same amount as Enc. It remains to prove an upper bound on the length o the encoding o by our algorithm Enc as claimed in Eq.(14). Recall that Enc (as used inside Enc g ) deines a set G such that or every y G we have (1) A,g aux(y) inverts, i.e., g (A,g aux(y)) = y and (2) never makes a g query x where g (x) G. Recall that T g in Eq.(13) satisies T g = ɛn/2 G, and corresponds to the average number o resh g queries made by A,g aux( ) when invoked on the values in G. Enc invokes A,g aux( ) on a careully chosen subset G = (z 1,..., z G ) o G (to be deined later). It keeps lists L, C and T such that ater invoking A,g aux( ) on G, L C holds the outputs to all queries made. Looking ahead, the decoding Dec will also invoke A,g aux( ) on G, but will only need L and T (but not C ) to answer all queries. The lists L, T, C are generated as ollows. On the irst invocation A,g aux(z 1 ) we observe up to T oracle queries made to g and. Every g query (x, x ) must be preceded by queries x and x where (x) = (x ). Assume x and x are the queries number t, t (1 t < t T ). A key observation is that by just storing (t, t ) and (x), Dec will later be able to reconstruct (x ) by invoking A,g aux(z 1 ), and when query t is made, looking up the query (x) in L (its position in L is given by t), and set (x ) = (x). Thus, every time a resh query (x ) is made we append it to L, unless earlier in this invocation we made a resh query (x) where (x ) = (x). In this case we append the indices (t, t ) to the list T. We also add (x ) to a list C just to keep track o what we already compressed. Enc now continues this process by invoking A,g aux( ) on inputs z 2, z 3,..., z G G and inally outputs and encoding o G, an encoding o the list o images o resh queries L, an encoding o the list o colliding indices T, aux, and all values o that were neither compressed nor queried. 17

18 In the sequel we show how to choose G G such that G ɛn/8t and hence it can be encoded using G log N + log N where the extra log N is used to encode G. We also show that T G T g /4 and urthermore that we can compress at least one bit per element o T. Putting things together we get Enc (ρ, aux,, g) log N! G (T g /4 log N) + S + log N. And i log N T g /8, we get Eq.(14) Enc (ρ, aux,, g) log N! ɛnt g /64T + S + log N. Given G such that G ɛn/2t g, the subset G can be constructed by careully applying Lemma 3 which we prove in Sec. 5. Let (X 1,..., X G ), (Y 1,..., Y G ) be two sequences o sets such that Y i X i [N] and X i T such that Y i and X i respectively correspond to g and queries in G consecutive executions o A,g aux( ) on G. 20 Given such sequences Lemma 3 constructs a subsequence o ) are resh. As executions G G whose corresponding g queries (Y i1,..., Y i G a g query is preceded by two queries, such a subsequence induces a sequence (Z i1,..., Z i G ) o queries that are not only resh or g but also resh or. Furthermore, such a sequence covers y I /16T where y = I / G is the average coverage o Y i s and I [N] is their total coverage. However, Lemma 3 considers a g query (x, x ) Y i to be resh i either x / i 1 j=1 X j or x / i 1 j=1 X j, i.e., i at least one o x, x is resh in the i th execution, then the pair is considered resh. For compressing both x, x need to be resh. To enorce that and apply Lemma 3 directly, we apply Lemma 3 on augmented sets X 1,..., X G such that whenever X i, Y i are selected, the corresponding Z i contains exactly Z i /2 pairs o queries that are resh or both g and. We augment X i as ollows. For every X i and every query x made in the i th step, add 1 ((x)) to X i. This augmentation results in X i such that X i 2T as originally we have X i T. Applying Lemma 3 on Y 1,..., Y G and such augmented sets X 1,..., X G yields G such that the total number o resh colliding queries is o size at least y I 16 2T = ɛn G ɛn 32T = ɛnt g 16T. Thereore the total number o resh colliding pairs, or equivalently T, is ɛnt g /32T as claimed. Furthermore, Lemma 3 guarantees that G ɛn/8t. 21 What remains to show is that or each colliding pair in T we compress by at least one bit. Recall that the list T has exactly as many entries as C. However 20 Here is how these sets are compiled. Note that i q is an query then q [N], and i q is a g query then q [N] 2. In the i th execution, both X i, Y i are initially empty and later will contain only elements in [N]. Thereore or each query q, i q = (x, x ) is a g query we add two elements x and x to Y i, and i q = x is an query we add the single element x to X i. Furthermore as a g query (x, x ) is preceded by two queries x, x, then Y i X i, and as the max number o queries is T we have X i T. 21 G corresponds to l in the proo o Lemma 3. 18

19 entries in T are colliding pairs o indices (t, t ) and entries in C are images o size log N. Per each entry (t, t ) in T we compress i the encoding size o (t, t ) is strictly less than log N. Here is an encoding o T that achieves this. Instead o encoding each entry (t, t ) as two indices which costs 2 log T and thereore we save one bit per element in T assuming T N/2, we encode the set o colliding pairs among all possible query pairs. Concretely, or each z G we obtain a set o colliding indices o size at least T g /4. Then we encode this set o colliding pairs T g /4 among all possible pairs 22, which is upper bounded by T 2, using ( ) T 2 log T 2 g 4eT log T g /4 4 T g bits, and thereore, given that T g T and T (N/4e) 2/3, we have that log N log 4eT 2 /T g 1 and thereore we compress by at least one bit or each pair, i.e., or each element in T, and that concludes the proo. 5 A Combinatorial Lemma In this section we state and prove a lemma which can be cast in terms o the ollowing game between Alice and Bob. For some integers n, N, M, Alice can choose a partition (Y 1,..., Y n ) o I [N], and or every Y i also a superset X i Y i o size X i M. The goal o Bob is to ind a subsequence 1 b 1 < b 2 <... < b l such that Y b1, Y b2,..., Y bl contains as many resh elements as possible, where ater picking Y bi the elements i k=1 X b k are not resh, i.e., picking Y bi spoils all o X bi. How many resh elements can Bob expect to hit in the worst case? Intuitively, as every Y bi added spoils up to M elements, he can hope to pick up to l I /M o the Y i s beore most o the elements are spoiled. As the Y i are on average o size y := I /n, this is also an upper bound on the number o resh elements he can hope to get with every step. This gives something in the order o y ( I /M) resh elements in total. By the lemma below a subsequence that contains about that many resh elements always exists. Lemma 3. For M, N N, M N and any disjoint sets Y 1,..., Y n [N] n Y i = I, i j : Y i Y j = i=1 and supersets (X 1,..., X n ) where i [n] : Y i X i [N], X i M 22 Note that T 2 is an upper bound on all possible pairs o queries, however as we have that t < t or each pair (t, t ), we can cut T 2 by at least a actor o 2. Other optimizations are possible. This extra saving one can use to add extra dummy pairs o indices to separate executions or decoding. The details are tedious and do not aect the bound as we were generous to consider T 2 to be the size o possible pairs. 19

20 there exists a subsequence 1 b 1 < b 2 <... < b l n such that the sets Z bj = Y bj \ k<j X bk (15) have total size l Z bj = j=1 l I Z bj y 16M j=1 where y = I /n denotes the average size o the Y i s. Proo. Let (Y a1,..., Y am ) be a subquence o (Y 1,..., Y n ) that contains all the sets o size at least y/2. By a Markov bound, these large Y ai s cover at least hal o the domain I, i.e. i [m] Y ai > I /2. (16) We now choose the subsequence (Y b1,..., Y bl ) rom the statement o the lemma as a subsequence o (Y a1,..., Y am ) in a greedy way: or i = 1,..., m we add Y ai to the sequence i it adds a lot o resh elements, concretely, assume we are in step i and so ar have added Y b1,..., Y bj 1, then we ll pick the next element, i.e., Y bj := Y ai, i the resh elements Z bj = Y bj \ k<j X bk contributed by Y bj are o size at least Z bj > Y bj /2. We claim that we can always add at least one more Y bj as long as we haven t yet added at least I /4M sets, i.e., j < I /4M. Note that this then proves the lemma as l Z bj j=1 l Y bj /2 ly/4 I /4M y/4 = y I /16M. j=1 It remains to prove the claim. For contradiction assume our greedy algorithm picked (Y b1,..., Y bl ) with l < I /4M. We ll show that there is a Y at (with a t > b l ) with Y at \ j i=1 X b i Y at /2 which is a contradiction as this means the sequence could be extended by Y bl+1 = Y at. We have l i=1 X bi I /4M M = I /4. This together with (16) implies i [m] Y ai \ l i=1x ai > i [m] Y ai /2. By Markov there must exist some Y at with Y at \ l i=1x ai Y at /2 as claimed. 20

21 6 Conclusions In this work we showed that existing time-memory trade-os or inverting unctions can be overcome, i one relaxes the requirement that the unction is eiciently computable, and just asks or the unction table to be computed in (quasi)linear time. We showed that such unctions have interesting applications towards constructing proos o space. The ideas we introduced can potentially also be used or related problems, like memory-bound or memory-hard unctions. Acknowledgements Hamza Abusalah, Joël Alwen, and Krzyszto Pietrzak were supported by the European Research Council, ERC consolidator grant ( TOCNeT). Leonid Reyzin grateully acknowledges the hospitality and support o IST Austria, where much o this work was perormed. He was also supported, in part, by US NSF grants , , and A Proo o Lemma 1 ollowing [DTT10] Fig. 2. The dierent subsets o the input domain o : [N] [N]. R is a random subset o size N/T, I are the values x where A aux((x)) = x. G is the subset o I R o x where A aux((x)) makes no oracle queries to values in R (except or the last query x). In this section we sketch the proo o Theorem 1 ollowing the proo rom [DTT10], just marginally adapting it so it applied to unctions not just permutations. Let I = {x : A aux((x)) = x}, J = (I) = {y : (A aux(y)) = y} For x I let q((x)) denote the queries made by A aux ((x)) but without the inal query x. By assumption I = ɛn. Pick a random subset R [N] o size R = N/T (or this use the randomness ρ given to the encoding). Let G denote 21