HASH FUNCTIONS. Mihir Bellare UCSD 1

Transcription

1 HASH FUNCTIONS Mihir Bellare UCSD 1

2 Hashing Hash functions like MD5, SHA1, SHA256, SHA512, SHA3,... are amongst the most widely-used cryptographic primitives. Their primary purpose is collision-resistant data compression, but they have many other purposes and properties as well. A good hash function is often treated like a magic wand... Mihir Bellare UCSD 2

3 Collision resistance (CR) Definition: A collision for a function h : D {0, 1} n is a pair x 1, x 2 D of points such that h(x 1 ) = h(x 2 ) but x 1 x 2. If D > 2 n then the pigeonhole principle tells us that there must exist a collision for h. Mihir Bellare UCSD 3

4 Collision resistance (CR) Definition: A collision for a function h : D {0, 1} n is a pair x 1, x 2 D of points such that h(x 1 ) = h(x 2 ) but x 1 x 2. If D > 2 n then the pigeonhole principle tells us that there must exist a collision for h. Mihir Bellare UCSD 4

5 Collision resistance (CR) Definition: A collision for a function h : D {0, 1} n is a pair x 1, x 2 D of points such that h(x 1 ) = h(x 2 ) but x 1 x 2. If D > 2 n then the pigeonhole principle tells us that there must exist a collision for h. We want that even though collisions exist, they are hard to find. Mihir Bellare UCSD 5

6 Collision-resistance of a function family The formalism considers a family H : Keys(H) D R of functions, meaning for each K Keys(H) we have a map H K : D R defined by H K (x) = H(K, x). Game CR H procedure Initialize K $ Keys(H) Return K procedure Finalize(x 1, x 2 ) If (x 1 = x 2 ) then return false If (x 1 D or x 2 D) then return false Return (H K (x 1 ) = H K (x 2 )) Let Adv cr H (A) = Pr [ CR A H true ]. Mihir Bellare UCSD 6

7 Collision-resistance Game CR H procedure Initialize K $ Keys(H) Return K procedure Finalize(x 1, x 2 ) If (x 1 = x 2 ) then return false If (x 1 D or x 2 D) then return false Return (H K (x 1 ) = H K (x 2 )) The Return statement in Initialize means that the adversary A gets K as input. The key K here is not secret! Adversary A takes K and tries to output a collision x 1, x 2 for H K. A s output is the input to Finalize, and the game returns true if the collision is valid. Mihir Bellare UCSD 7

8 Example Let E: {0, 1} k {0, 1} n {0, 1} n be a blockcipher. Let H: {0, 1} k {0, 1} 2n {0, 1} n be defined by Alg H(K, x[1]x[2]) y E K (E K (x[1]) x[2]); Return y Let s show that H is not collision-resistant by giving an efficient adversary A such that Adv cr H (A) = 1. Mihir Bellare UCSD 8

9 Example Let E: {0, 1} k {0, 1} n {0, 1} n be a blockcipher. Let H: {0, 1} k {0, 1} 2n {0, 1} n be defined by Alg H(K, x[1]x[2]) y E K (E K (x[1]) x[2]); Return y Let s show that H is not collision-resistant by giving an efficient adversary A such that Adv cr H (A) = 1. Idea: Pick x 1 = x 1 [1]x 1 [2] and x 2 = x 2 [1]x 2 [2] so that E K (x 1 [1]) x 1 [2] = E K (x 2 [1]) x 2 [2] Mihir Bellare UCSD 9

10 Example Alg H(K, x[1]x[2]) y E K (E K (x[1]) x[2]); Return y Idea: Pick x 1 = x 1 [1]x 1 [2] and x 2 = x 2 [1]x 2 [2] so that adversary A(K) E K (x 1 [1]) x 1 [2] = E K (x 2 [1]) x 2 [2] x 1 0 n 1 n ; x 2 [2] 0 n ; x 2 [1] E 1 K (E K (x 1 [1]) x 1 [2] x 2 [2]) return x 1, x 2 Then Adv cr H (A) = 1 and A is efficient, so H is not CR. Note how we used the fact that A knows K and the fact that E is a blockcipher! Mihir Bellare UCSD 10

11 Exercise Let E: {0, 1} k {0, 1} l {0, 1} l be a blockcipher. Let D be the set of all strings whose length is a positive multiple of l. Define the hash function H: {0, 1} k D {0, 1} l as follows: Alg H(K, M) M[1]M[2]... M[n] M C[0] 0 l For i = 1,..., n do B[i] E(K, C[i 1] M[i]); C[i] E(K, B[i] M[i]) Return C[n] Show that H is not CR by giving an efficient adversary A such that (A) = 1. Adv cr H Mihir Bellare UCSD 11

12 Keyless hash functions We say that H: Keys(H) D R is keyless if Keys(H) = {ε} consists of just one key, the empty string. In this case we write H(x) in place of H(ε, x) or H ε (x). Practical hash functions like MD5, SHA1, SHA256, SHA512, SHA3,... are keyless. Mihir Bellare UCSD 12

13 SHA1 Alg SHA1(M) // M < 2 64 V SHF1( 5A ED9EBA1 8F1BBCDC CA62C1D6, M ) return V Alg SHF1(K, M) // K = 128 and M < 2 64 y shapad(m) Parse y as M 1 M 2 M n where M i = 512 (1 i n) V EFCDAB89 98BADCFE C3D2E1F0 for i = 1,..., n do V shf1(k, M i V ) return V Alg shapad(m) // M < 2 64 d (447 M ) mod 512 Let l be the 64-bit binary representation of M y M 1 0 d l // y is a multiple of 512 return y Mihir Bellare UCSD 13

14 shf1 Alg shf1(k, B V ) // K = 128, B = 512 and V = 160 Parse B as W 0 W 1 W 15 where W i = 32 (0 i 15) Parse V as V 0 V 1 V 4 where V i = 32 (0 i 4) Parse K as K 0 K 1 K 2 K 3 where K i = 32 (0 i 3) for t = 16 to 79 do W t ROTL 1 (W t 3 W t 8 W t 14 W t 16 ) A V 0 ; B V 1 ; C V 2 ; D V 3 ; E V 4 for t = 0 to 19 do L t K 0 ; L t+20 K 1 ; L t+40 K 2 ; L t+60 K 3 for t = 0 to 79 do if (0 t 19) then f (B C) (( B) D) if (20 t 39 OR 60 t 79) then f B C D if (40 t 59) then f (B C) (B D) (C D) temp ROTL 5 (A) + f + E + W t + L t E D ; D C ; C ROTL 30 (B) ; B A ; A temp V 0 V 0 +A ; V 1 V 1 +B ; V 2 V 2 +C ; V 3 V 3 +D ; V 4 V 4 +E V V 0 V 1 V 2 V 3 V 4 ; return V Mihir Bellare UCSD 14

15 SHA1 calculator Mihir Bellare UCSD 15

16 Applications of hash functions The killer-app is hashing before digitally signing, but we ll discuss a few others: primitive in cryptographic schemes tool for security applications tool for non-security applications Mihir Bellare UCSD 16

17 Password verification Client A has a password PW that is also held by server B A authenticates itself by sending PW to B over a secure channel (TLS) A PW PW B PW Problem: The password will be found by an attacker who compromises the server. Mihir Bellare UCSD 17

18 Password verification Client A has a password PW and server stores PW = H(PW ). A sends PW to B (over a secure channel) and B checks that H(PW ) = PW A PW PW B PW Server compromise results in attacker getting PW which should not reveal PW as long as H is one-way, which we will see is a consequence of collision-resistance. But we will revisit this when we consider dictionary attacks! Mihir Bellare UCSD 18

19 File comparision A has a large file F A and B has a large file F B. For example, music collections. They want to know whether F A = F B A sends F A to B and B checks whether F A = F B A F A F A B F B Problem: Transmission could be slow. Mihir Bellare UCSD 19

20 Compare-by-hash A has a large file F A and B has a large file F B and they want to know whether F A = F B A computes h A = H(F A ) and sends it to B, and B checks whether h A = H(F B ). A F A h A B F B Collision-resistance of H guarantees that B does not accept if F A F B! Mihir Bellare UCSD 20

21 Virus protection An executable may be available at lots of sites S 1, S 2,..., S N. Which one can you trust? Provide a safe way to get the hash h = H(X ) of the correct executable X. Download an executable from anywhere, and check hash. Mihir Bellare UCSD 21

22 Birthday attacks Let H : {0, 1} k D {0, 1} n be a family of functions with D > 2 n. The q-trial birthday attack finds a collision with probability about q 2 2 n+1. So a collision can be found in about q = 2 n+1 2 n/2 trials. Mihir Bellare UCSD 22

23 Recall Birthday Problem $ for i = 1,..., q do y i {0, 1} n if i, j (i j and y i = y j ) then COLL true Pr [COLL] = C(2 n, q) q2 2 n+1 Mihir Bellare UCSD 23

24 Birthday attack Let H : {0, 1} k D {0, 1} n. adversary A(K) $ for i = 1,..., q do x i D ; y i H K (x i ) if i, j (i j and y i = y j and x i x j ) then return x i, x j else return FAIL Mihir Bellare UCSD 24

25 Analysis of birthday attack Let H : {0, 1} k D {0, 1} n. adversary A(K) $ for i = 1,..., q do x i D ; y i H K (x i ) if i, j (i j and y i = y j and x i x j ) then return x i, x j else return FAIL What is the probability that this attack finds a collision? adversary A(K) $ for i = 1,..., q do x i D ; y i H K (x i ) if i, j (i j and y i = y j ) then COLL true We have dropped things that don t much affect the advantage and focused on success probability. So we want to know what is Pr [COLL]. Mihir Bellare UCSD 25

26 Analysis of birthday attack Birthday for i = 1,..., q do $ y i {0, 1} n if i, j (i j and y i = y j ) then COLL true Adversary A for i = 1,..., q do $ x i D ; y i H K (x i ) if i, j(i j and y i = y j ) then COLL true Pr [COLL] = C(2 n, q) Pr [COLL] =? Are the two collision probabilities the same? Mihir Bellare UCSD 26

27 Analysis of birthday attack Birthday for i = 1,..., q do $ y i {0, 1} n if i, j (i j and y i = y j ) then COLL true Adversary A for i = 1,..., q do $ x i D ; y i H K (x i ) if i, j(i j and y i = y j ) then COLL true Pr [COLL] = C(2 n, q) Pr [COLL] =? Are the two collision probabilities the same? Not necessarily, because $ on the left y i {0, 1} n $ on the right x i D ; y i H K (x i ) Mihir Bellare UCSD 27

28 Analysis of birthday attack We say that H : {0, 1} k D {0, 1} n is regular if every range point has the same number of pre-images under H K. That is if we let then H is regular if H 1 K (y) = {x D : H K (x) = y} H 1 D K (y) = 2 n for all K and y. In this case the following processes both result in a random output Process 1 y $ {0, 1} n return y Process 2 x $ D; y $ H K (x) return y Mihir Bellare UCSD 28

29 Analysis of birthday attack If H: {0, 1} k D {0, 1} n is regular then the birthday attack finds a collision in about 2 n/2 trials. Mihir Bellare UCSD 29

30 Analysis of birthday attack If H: {0, 1} k D {0, 1} n is regular then the birthday attack finds a collision in about 2 n/2 trials. If H is not regular, the attack may succeed sooner. So we want functions to be close to regular. It seems MD4, MD5, SHA1, SHA2, SHA3,... have this property. Mihir Bellare UCSD 30

31 Birthday attack times Function n T B MD MD SHA SHA SHA SHA SHA T B is the number of trials to find collisions via a birthday attack. Mihir Bellare UCSD 31

32 Compression functions A compression function is a family h : {0, 1} k {0, 1} b+n {0, 1} n of hash functions whose inputs are of a fixed size b + n, where b is called the block size. E.g. b = 512 and n = 160, in which case h : {0, 1} k {0, 1} 672 {0, 1} 160 x v h K h K (x v) Mihir Bellare UCSD 32

33 The MD transform Design principle: To build a CR hash function where D = {0, 1} 264 : H : {0, 1} k D {0, 1} n First build a CR compression function h : {0, 1} k {0, 1} b+n {0, 1} n. Appropriately iterate h to get H, using h to hash block-by-block. Mihir Bellare UCSD 33

34 MD setup Assume for simplicity that M is a multiple of b. Let M b be the number of b-bit blocks in M, and write M = M[1]... M[l] where l = M b. i denote the b-bit binary representation of i {0,..., 2 b 1}. D be the set of all strings of at most 2 b 1 blocks, so that M b {0,..., 2 b 1} for any M D, and thus M b can be encoded as above. Mihir Bellare UCSD 34

35 MD transform Given: Compression function h : {0, 1} k {0, 1} b+n {0, 1} n. Build: Hash function H : {0, 1} k D {0, 1} n. Algorithm H K (M) m M b ; M[m + 1] m ; V [0] 0 n For i = 1,..., m + 1 do v[i] h K (M[i] V [i 1]) Return V [m + 1] M[1] M[2] 2 0 n h K h K h K H K (M) Mihir Bellare UCSD 35

36 MD preserves CR Assume h is CR H is built from h using MD Then H is CR too! This means No need to attack H! You won t find a weakness in it unless h has one H is guaranteed to be secure assuming h is. For this reason, MD is the design used in many current hash functions. Newer hash functions use other iteration methods with analogous properties. Mihir Bellare UCSD 36

37 MD preserves CR Theorem: Let h : {0, 1} k {0, 1} b+n {0, 1} n be a family of functions and let H : {0, 1} k D {0, 1} n be obtained from h via the MD transform. Given a cr-adversary A H we can build a cr-adversary A h such that Adv cr H (A H) Adv cr h (A h) and the running time of A h is that of A H plus the time for computing h on the outputs of A H. Implication: h CR Adv cr h (A h) small Adv cr H (A H) small H CR Mihir Bellare UCSD 37

38 How A h works Let (M 1, M 2 ) be the H K -collision returned by A H. The A h will trace the chains backwards to find an h k -collision. Mihir Bellare UCSD 38

39 Case 1: M 1 b M 2 b Let x 1 = 2 V 1 [2] and x 2 = 1 V 2 [1]. Then h K (x 1 ) = h K (x 2 ) because H K (M 1 ) = H K (M 2 ). But x 1 x 2 because 1 2. Mihir Bellare UCSD 39

40 Case 2: M 1 b = M 2 b x 1 2 V 1 [2] ; x 2 2 V 2 [2] If x 1 x 2 then return x 1, x 2 Mihir Bellare UCSD 40

41 Case 2: M 1 b = M 2 b x 1 2 V 1 [2] ; x 2 2 V 2 [2] If x 1 x 2 then return x 1, x 2 Else // V 1 [2] = V 2 [2] Mihir Bellare UCSD 41

42 Case 2: M 1 b = M 2 b x 1 2 V 1 [2] ; x 2 2 V 2 [2] If x 1 x 2 then return x 1, x 2 Else // V 1 [2] = V 2 [2] x 1 M 1 [2] V 1 [1] ; x 2 M 2 [2] V 2 [1] If x 1 x 2 then return x 1, x 2 Mihir Bellare UCSD 42

43 Case 2: M 1 b = M 2 b x 1 2 V 1 [2] ; x 2 2 V 2 [2] If x 1 x 2 then return x 1, x 2 Else // V 1 [2] = V 2 [2] x 1 M 1 [2] V 1 [1] ; x 2 M 2 [2] V 2 [1] If x 1 x 2 then return x 1, x 2 Else // V 1 [1] = V 2 [1] Mihir Bellare UCSD 43

44 Case 2: M 1 b = M 2 b x 1 2 V 1 [2] ; x 2 2 V 2 [2] If x 1 x 2 then return x 1, x 2 Else // V 1 [2] = V 2 [2] x 1 M 1 [2] V 1 [1] ; x 2 M 2 [2] V 2 [1] If x 1 x 2 then return x 1, x 2 Else // V 1 [1] = V 2 [1] x 1 M 1 [1] 0 n ; x 2 M 2 [1] 0 n Return x 1, x 2 Mihir Bellare UCSD 44

45 Exercise Let h: K {0, 1} 2b {0, 1} b be a compression function. Define H: K {0, 1} 4b {0, 1} b as follows: Alg H(K, M) M 1 M 2 M V 1 h(k, M 1 ); V 2 h(k, M 2 ) V h(k, V 1 V 2 ) return V Above, by M 1 M 2 M, we mean that M 1 is the first 2b bits of M and M 2 is the rest, so that M 1 = M 2 = 2b. Show that if h is collision-resistant then so is H. Do this by stating and proving an analogue of the Theorem above. Mihir Bellare UCSD 45

46 How are compression functions designed? Let E : {0, 1} b {0, 1} n {0, 1} n be a block cipher. Let us define keyless compression function h : {0, 1} b+n {0, 1} n by Question: Is H collision resistant? h(x v) = E x (v). Mihir Bellare UCSD 46

47 How are compression functions designed? Let E : {0, 1} b {0, 1} n {0, 1} n be a block cipher. Let us define keyless compression function h : {0, 1} b+n {0, 1} n by Question: Is H collision resistant? h(x v) = E x (v). We seek an adversary that outputs distinct x 1 v 1, x 2 v 2 satisfying E x1 (v 1 ) = E x2 (v 2 ). Mihir Bellare UCSD 47

48 How are compression functions designed? Let E : {0, 1} b {0, 1} n {0, 1} n be a block cipher. Let us define keyless compression function h : {0, 1} b+n {0, 1} n by Question: Is H collision resistant? h(x v) = E x (v). We seek an adversary that outputs distinct x 1 v 1, x 2 v 2 satisfying E x1 (v 1 ) = E x2 (v 2 ). Answer: NO, h is NOT collision-resistant, because the following adversary A has Adv cr h (A) = 1: adversary A x 1 0 b ; x 2 1 b ; v 1 0 n ; y E x1 (v 1 ) ; v 2 Ex 1 2 (y) Return x 1 v 1, x 2 v 2 Mihir Bellare UCSD 48

49 How are compression functions designed? Let E : {0, 1} b {0, 1} n {0, 1} n be a block cipher. Let us define keyless compression function h : {0, 1} b+n {0, 1} n by h(x v) = E x (v) v. The compression function of SHA1 is underlain in this way by a block cipher E : {0, 1} 512 {0, 1} 160 {0, 1} 160. The compression function of SHA256 is underlain in this way by a block cipher E : {0, 1} 512 {0, 1} 256 {0, 1} 256. Mihir Bellare UCSD 49

50 Cryptanalytic attacks So far we have looked at attacks that do not attempt to exploit the structure of H. Can we do better than birthday if we do exploit the structure? Ideally not, but functions have fallen short! Mihir Bellare UCSD 50

51 Cryptanalytic attacks against hash functions 1993,1996 : collisions found in compression function md5 of MD5 in 2 16 time [dbbo,do]. But these did not yield collisions for MD , 2005 : collisions found in RIPEMD, SHA0 [WaFeLaYu]. 2005, 2006: collisions can be found in MD5 in 1 minute [WaFeLaYu, LeWadW, Kl] 2015: collisions found in compression function sha1 of SHA1 [SKP]. But these do not yield collisions for SHA : collisions found for SHA1 in time [SBKAM] : Google, Microsoft and Mozilla browsers stop accepting SHA1-based certificates. Mihir Bellare UCSD 51

52 SHA1 collision Mihir Bellare UCSD 52

53 SHA1 collision news Mihir Bellare UCSD 53

54 SHA2 The SHA256 and SHA512 hash functions are still viewed as secure, meaning the best known attack is the birthday attack. These are based on the MD paradigm we discussed above. Mihir Bellare UCSD 54

55 SHA3 National Institute for Standards and Technology (NIST) held a world-wide competition to develop a new hash function standard. Contest webpage: Requested parameters: Design: Family of functions with 224, 256, 384, 512 bit output sizes Security: CR, one-wayness, near-collision resistance, others... Efficiency: as fast or faster than SHA2-256 Mihir Bellare UCSD 55

56 SHA3 Submissions: 64 Round 1: 51 Round 2: 14: BLAKE, Blue Midnight Wish, CubeHash, ECHO, Fugue, Grostl, Hamsi, JH, Keccak, Luffa, Shabal, SHAvite-3, SIMD, Skein. Finalists: 5: BLAKE, Grostl, JH, Keccak, Skein. SHA3: 1: Keccak Mihir Bellare UCSD 56

57 SHA3: The Sponge construction f : {0, 1} r+c {0, 1} r+c is a (public, invertible!) permutation. d is the number of output bits, and c = 2d. SHA3 does not use the MD paradigm used by SHA1 and SHA2. Shake(M, d) Extendable-output function, returning any given number d of bits. Mihir Bellare UCSD 57