Count-Min Sketches for Estimating Password Frequency within Hamming Distance Two

Transcription

1 Count-Min Sketches for Estimating Password Frequency within Hamming Distance Two Leah South and Dougas Stebia Schoo of Mathematica Sciences, Queensand University of Technoogy, Brisbane, Queensand, Austraia Abstract. The count-min sketch is a usefu data structure for recording and estimating the frequency of string occurrences, such as passwords, in sub-inear space with high accuracy. However, it cannot be used to draw concusions on groups of strings that are simiar, for exampe cose in Hamming distance. This paper introduces a variant of the count-min sketch which aows for estimating counts within a specified Hamming distance of the queried string. This variant can be used to prevent users from choosing popuar passwords, ike the origina sketch, but it aso aows for a more efficient method of anaysing password statistics. Keywords: count-min sketch, Boom fiter, password frequency, approximate string matching 1 Introduction The use of passwords for identity verification is widespread. There is a ong ine of research on anayzing the security and guessabiity of password [8, 3]. In arge onine systems that we see on the Internet today, an important characteristic that affects the overa security of the system is that passwords within the system shoud not be too popuar. In an idea setting, of course, users woud create a unique password that is hard to guess, and not popuar, so that ony that user and no one ese coud access their account. However, users tend to choose passwords that are easy to remember and famiiar to them, such as dictionary words, or perhaps strings associated with the system in question. This tendency means that certain passwords are used with higher frequency, making them popuar. If an attacker knew the distribution of passwords, they coud use its statistics and guess the most popuar passwords first. This is known as the statistica guessing technique [9]. When there is a high percentage of popuar passwords, the attacker can compromise a high percentage of accounts. For exampe, the 009 breach of RockYou.com s 3 miion account password database showed that the most popuar password (13456) was used by 0.9% of a accounts, and the next 4 most popuar passwords (1345, , password, and ioveyou) were used by another 0.8% of a accounts. Ceary, the system was not screening passwords for popuarity. As a resut, a statistica guessing attack woud ead to miions of accounts being compromised. In order to prevent a successfu statistica guessing

2 Leah South and Dougas Stebia attack such as this, it is common to imit the number of guesses; more recenty, it has been proposed [9] to imit the popuarity of passwords: when users try to set their password to a string that is used in a percentage of accounts above some threshod, it is rejected and the user is required to choose another password. In order to keep track of password popuarity, some sort of system which counts passwords must be used. Onine sites with a arge number of users are best suited for systems which restrict popuar passwords, as such sites are at high risk of trawing attacks, in which attackers aim to guess the passwords to many accounts without targeting any singe account. How can we store password information in a way that aows us to cacuate frequency when users attempt to register a password? The simpest technique for cacuating frequency during password registration woud be to store a separate tabe of passwords aong with their frequency. This is undesirabe both for efficiency reasons (since the size of the tabe grows ineary in the number of distinct passwords) and for security reasons (since it immediatey provides an attacker with the fu distribution of passwords). Best-practice recommendations for storing passwords for ogin invove storing sated password hashes for each account; the set of such passwords does not admit statistica anaysis since, by design, the hash of the same password under different sats yieds different, seemingy independent, outputs, thus yieding no information about the frequency with which a password is used. 1.1 Boom Fiters and Count-Min Sketches The Boom fiter [] can be used [10] to store in sub-inear space a tabe representing a dictionary of prohibited passwords. The system is setup as foows. A w h tabe T of bits is used, aong with h independent hash functions hash 1,..., hash h : {A Z, a z, 0 9,... } {1,..., w}. Each word x in the dictionary of prohibited passwords is hashed under each hash function hash k, and the entry T hashk (x),k is set to 1. When a proposed password y is to be tested for membership in the ist of prohibited passwords, if a of the vaues T hashk (y),k are equa to 1, then y is deemed to be prohibited, but if at east one of those tabe entries is zero, then y is not prohibited. There are no fase negatives, meaning that it is impossibe for a password that is prohibited to not be recognized as such, but there may be fase positives, meaning that some passwords that are not prohibited may, due to coisions on a rows, sti be identified by the tabe as prohibited. Assuming the hash functions are independent random functions, the fase positive rate (1 (1 h w )N ) h, where N is the number of prohibited passwords originay added to the tabe [10]. The count-min sketch [5] enhances the Boom fiter by storing a tabe of integers, not bits. The update(x, c) function records that string x has been used c more times by adding c to each tabe entry T hashk (x),k. The estimate(x) function returns an estimate on the number of times that string x has been used by computing min{t hashk (x),k : 1 k h}. The use of count-min sketches for recording password frequency was proposed by Schechter, Herey, and Mitzenmacher [9], who propose preventing users from registering with passwords whose

3 Count-Min Sketches for Estimating Password Frequency within Distance 3 current popuarity is above a certain threshod. As with the Boom fiter, fase negatives cannot occur, meaning that for any password x it is impossibe for the estimate(x) to return a vaue ower than the sum of the c over a cas of update(x, c). However, fase positives may sti occur, meaning that estimate(x) may over-estimate the frequency of password x, due to coisions across a hash functions. For a singe hash, the expected error due to coisions is N/w, where N is the tota sum of the counts of a passwords; this is because the tota count of N wi spread approximatey eveny across a w coumns in the tabe. By the Markov inequaity, the error of the count min-sketch (the minimum across a hash functions) is at most N/w with probabiity at east 1 ( 1 )h. 1. Contributions When used for passwords, the Boom fiter and the count-min sketch can be usefu in prohibiting certain passwords or imiting the frequency with which any password is registered. However, they cannot accommodate any reation between simiar passwords. For exampe, a user who tries to register the password password and finds that it is prohibited may try again with p@ssword or passw rd. An attacker, knowing this prevaence for eetspeak, may aso make use of these simiarities in an attack strategy that targets amost popuar passwords. Since the Boom fiter and count-min sketch use independent random passwords, they ose any semantic connection between such simiar strings. In this work, we expore a variant of the count-min sketch that aows one to compute estimates not ony for the given string but for a strings within a fixed Hamming distance of the given string. This technique aows a system to detect frequenty used passwords at the registration phase. Our technica approach is to introduce widcard characters and then record and estimate based on a strings that can be constructed with widcard characters within the required Hamming distance. This technique imposes a computationa overhead of ( d), where is the ength of the password in question and d is the maximum Hamming distance. This compares favouraby with the naive technique which woud have a computationa overhead of ( α d ), where α is the size of the aphabet. We provide fase positive error rates as we, and compared with the naive method our technique provides better accuracy for the same size tabe for a wide range of password engths. Our technique can naturay be extended to higher Hamming distances. Reated Work Whie the Boom fiter and count-min sketch are widey used, there are severa other methods to store estimate counts, some of which may be usefu in obtaining statistics on groups of simiar passwords. There have been some systems in the past which have stored ceartext passwords aongside their counts. This method is mosty outdated due to the ack of security it provides. If attackers gain access to password databases such as these, a highy successfu statistica guessing attack can be carried out because the

4 4 Leah South and Dougas Stebia actua passwords are visibe and there are no fase positives - the exact counts are known. Using hash functions is preferabe to this method. Decision trees, as suggested by Bergadano, Crispo and Ruffo, can be used to determine password membership. These decision trees consist of nodes for particuar attributes, arcs for vaues of these attributes and eaves for cassification of whether or not the password has been used before in that database [1]. When using decision trees, there is a training phase in which the nodes, arcs and eaves are chosen to produce the best resuts. For each update to the database, the training phase must be repeated. Due to this, decision trees can be used for determining membership but are not as practica when frequent updates may be invoved. There has aso been some work on the topic of determining popuarity of simiar words. This incudes work which aows for checking membership of words within Levenshtein distance 1, that is words which have ony one insertion, deetion or substitution. Manber and Wu [7] suggested an approach simiar to the origina Boom fiter, with the main difference being that the membership of words within Levenshtein distance 1 are checked in the estimation stage. It is proposed that if any password within distance 1 appears to be positive, the password cannot be used [7]. As with a Boom fiter-based techniques, this ony records the binary data of whether two simiar passwords have been used, not counts on how frequenty simiar passwords are used. 3 Construction The adaptation of the origina count-min sketch that is introduced in this paper aows for the popuarity estimations of words within Hamming distance zero to two, that is words that differ by up to two substitutions. Like the count-min sketch, this adaptation consists of three main parts: hashing the passwords, updating the tabe and estimating the counts. In Appendix A, we provide a worked exampe of each of the three operations hashing, updating, and estimating. 3.1 Hash We represent passwords x as integers X. Any canonica representation of a word as an integer wi suffice. For expository convenience, we can imagine a mapping of an -etter word x = x 1... x where each character x i is coded as a two-digit integer X i {00,..., 98}; the integer 99 is reserved to represent a widcard character *. The vector X 1,..., X is then viewed as an integer X = i=1 100i 1 X i. This suffices to encode for exampe a passwords that can be typed using characters on a standard US keyboard. As in [6], each hash function hash k is a Carter Wegman -universa hash function [4] of the form hash k (x) = (((a k X + b k ) mod p k ) mod w) + 1,

5 Count-Min Sketches for Estimating Password Frequency within Distance 5 where X is the canonica integer representation of the word x, p k is a prime number much arger than w (reca w is the width of the tabe), say p k = 31 1 or p i = 61 1, and a k and b k are chosen uniformy at random from {0,..., p k 1}. We can simpify to using the same prime p = p 1 = = p h across a hash functions, but a a k and b k need to be chosen independenty to ensure negigibe probabiity of coisions; otherwise the benefits of using mutipe hash functions wi be eiminated. We define the vector-wise hash function hash(x) where the kth entry of hash(x) consists of hash k (x) with a k and b k. 3. Update The function update(t, x, a, b, c) T updates the sketch based on the previous tabe T, the password x to be updated, the vaues or vectors of a and b for the hash functions and the amount of times c that the password is being added. The update agorithm in this work differs greaty to that in the traditiona count-min sketch. Firsty, the count for the actua password is updated. Like the origina countmin sketch, this is done by finding the hash of the password for each hash function then updating the count at these positions in the tabe. Next, the updates are done for words within distance 1. To do this, a widcard character is introduced. This widcard character, which for our canonica encoding above is denoted by 99, is used to group simiar passwords together: for exampe if the word abcd, or a, b, c, d, is represented in integer form as 1,, 3, 4, then 99,, 3, 4 represents *bcd, a four etter words ending in bcd. A widcard character indicates that any character coud go in its pace. After the count of the actua password is updated, a widcard passwords within Hamming distance 1 are updated by creating = ength(x) variations of the password, each with a different character from the initia word repaced with 99, then hashing these variations and updating their positions in the tabe. Next, the widcard words within Hamming distance of the password x are hashed and their positions in the tabe are updated. In Agorithm 1, update, using the widcard character means that additiona updates are done for adding widcard words within distance 1. The aternative, naive technique, woud be to search a α passwords within distance 1 during the estimate agorithm, where α is the size of the aphabet. When the password ength α, the proposed technique is much more efficient than the naive technique. 3.3 Estimate The function estimate(t, x, a, b) is used to obtain estimates of the count of passwords at exacty Hamming distances 0, 1, and of x and is specified in fu in Agorithm. 1 1 Note that we have estimate return estimates for counts of passwords at exacty, rather than within, the specified Hamming distance for carity of exposition; estimates

6 6 Leah South and Dougas Stebia Agorithm 1 update(t, x, a, b, c) 1: T T : H hash(x, a, b) 3: for k = 1 to h do 4: T H k,k T H k,k + c 5: for i = 1 to = ength(x) do 6: x x 7: x i 99 8: H hash(x, a, b) 9: for k = 1 to h do 10: T H k,k T H k,k + c 11: for i = 1 to 1 do 1: for j = i + 1 to do 13: x x 14: x i 99 15: x j 99 16: H hash(x, a, b) 17: for k = 1 to h do 18: T H k,k H k,k 19: return T At distance 0. When the specified Hamming distance d is zero, that is, when the count of the actua word is desired, the estimate process is very simiar to the origina count-min sketch. The h different hashes of the password and the counts at a h positions in the tabe are found. These counts can then be put into a vector est 0 of ength h, where each eement represents the count at a different hash function. The resuting estimate est 0 is the minimum of these counts. At distance 1. At a maximum Hamming distance of one, the process is sighty more compex. First, estimates est 1 i, i = 1,...,, for the frequency of words that may differ from x in the ith character; each of these estimates can be found using the technique estimating the (distance 0) occurrences of the widcard word. In other words, est 1 i is the (distance 0) estimate for the widcard word x where the ith character of x has been repaced with the specia widcard character. Once estimates est 1 i for the frequency of words that may differ from x in the ith character have been found, we can find an estimate est 1 for the number of occurrences of words at distance 1 from x by summing est 1 i for a i, then subtract est 0. The reason for the subtraction is as foows. As expained in the update section, when a singe password x is added into the database, the counts for the exact password and the counts for a = ength(x) passwords within distance 1 are a increased. In the estimation stage, the counts for a passwords with one widcard character, or within distance 1, are summed together. This means that the count within the specified distance can be found by summing the estimates exacty at a distances ess than or equa to.

7 Count-Min Sketches for Estimating Password Frequency within Distance 7 for the exact password has been incuded times in the cacuation, when it shoud ony have been incuded once. To overcome this probem, the count of that exact password mutipied by the ength of the password is subtracted. At distance. The frequency estimate est for passwords at distance is found in a simiar manner to that of distance 1. For each of the ( widcard passwords of x that may differ in the ith and jth characters, we compute (distance 0) estimates est i,j of the widcard word x where the ith and jth characters of x have been repaced with the specia widcard character. We then sum these estimates of words within distance. However, again we have overcounted. The words at precisey distance 1 have been counted 1 times. Simiary, the word at precisey distance 0 has been counted ( times. Agorithm estimate(t, x, a, b) 1: H hash(x, a, b) : for k = 1 to h do 3: est 0 k T Hk,k 4: est 0 min k {est 0 k} 5: for i = 1 to = ength(x) do 6: x x 7: x i 99 8: H hash(x, a, b) 9: for k = 1 to h do 10: est 1 i,k T H k,k 11: est 1 i min k {est 1 i,k} 1: est 1 ( ) i est1 0 i est 13: for i = 1 to 1 do 14: for j = i + 1 to do 15: x x 16: x i 99 17: x j 99 18: H hash(x, a, b) 19: for k = 1 to h do 0: 1: est i,j,k T H k,k est i,j min k {est i,j,k} ( : est i,j est i,j ( 1) est 1 ( 3: return est 0, est 1, est ) ) est 0

8 8 Leah South and Dougas Stebia 4 Anaysis of Construction 4.1 Error Like the origina count-min sketch, coisions can occur in this adaptation. These coisions resut in some error, causing either sight overestimations of counts or fase positives. The error in estimating exact words can be expressed fairy simpy. In determining this error reative to the origina sketch, it is important to note that the tota count for this method is not the same as what it woud be in the origina count-min sketch. In this adaptation of the sketch, the tota count across the sketch is (1 + + ( ) times arger than the actua tota of passwords used. This additiona count is the resut of the update stage, in which the password itsef, a passwords within distance 1 and a ( combinations of passwords within distance are added into the sketch. Let ˆN = (1 + + ( )N denote the tota count on the new sketch, where N is the tota number of (non-distinct) passwords entered. In genera, hash outputs are expected to spread approximatey eveny across a coumns of the tabe, making the average count per hash the quotient of the tota sum of counts and the width w of the tabe: ˆN/w. The expected maximum error is therefore ˆN/w, which may occur, for exampe, if an estimate is done on a password which is not present but happens to have the same hash across a h hash functions as other passwords which have a count of ˆN/w. The expected maximum error for estimating counts of passwords at distance 1 can be derived from the formua for distance 1 in Agorithm : ( ) error(est 1 ) error est 1 i + error( est 0 ). i Since each term of i est1 i is cacuated as distance-0 estimate, the error of each term is the same as the error in a distance-0 estimate, and thus error(est 1 ) error(est 0 ) + error( est 0 ) error(est 0 ) = ˆN w. For passwords at distance, there is another increase in error. Based on the estimate for words at distance in Agorithm, the expected maximum error is error(est ) error i,j est i,j + error(( 1) est 1 ) + error (( ) ) est 0 Since each term of i,j est i,j is cacuated as distance-0 estimate, the error of each term is the same as the error in a distance-0 estimate, and thus ( ) (( ) ) error(est ) error(est 0 ) + error(( 1) est 1 ) + error est 0..

9 Count-Min Sketches for Estimating Password Frequency within Distance 9 Simpifying and substituting, we get ( ) ( ) error(est ) error(est 0 ) + ( 1) error(est 1 ) + error(est 0 ) (( ) ( )) + ( 1)() + error(est 0 ) ( ( ) ) = + ( 1) error(est 0 ) ( ( ) ( )) = + 4 error(est 0 ) ( ) ( ) = 6 error(est 0 ˆN ) = 6 w. 4. Comparison with Naive Method The purpose of the adaptation is to provide a more effective method of estimating counts of passwords within specified Hamming distances. Whie the count-min sketch can be used to do this, this naive method using the origina sketch incrementing the entry for every one of the α ( d) words within distance d in an aphabet of size α woud be ess efficient and have a higher error rate. In this section, a comparison of efficiency and error is carried out. A of the foowing graphs have the same response variabe: the expected average error as a mutipe of N w where N is the tota number of passwords entered (aso the tota count in the origina sketch) and w is the width of the tabe. Estimates at distance 0. When estimating how many times a specific word appears in a database (i.e. estimating passwords at distance zero), the origina count-min sketch is more effective. The expected maximum error in estimating counts is N ˆN w in the origina method and w, or (1 + + ( )N/w, in this variation. Therefore the expected maximum error for estimating counts at distance zero wi aways be ( times arger in this variation. However, this increase in error is not a major probem since this sketch was not designed for estimating exact word counts. Estimates at distance 1. The benefits of using this modified sketch are more evident when estimating counts at distance 1 of a password. In order to estimate counts at distance 1 of a specified password using the origina sketch, mutipe estimates woud have to be done. More specificay, the number of estimates woud be α, where α is the size of the aphabet and is the ength of the password. Since the error in estimating each individua password is N w, the expected maximum error in estimating α passwords is αn/w. With the new method, the expected maximum error in estimating words at distance 1 is ˆN/w = (1 + + ( )N/w. A comparison of these errors can be seen in Figure 1, where the size of the aphabet has been set to α = 7 (6 uppercase and 6 owercase etters, 10

10 10 Leah South and Dougas Stebia numbers, and 10 specia characters). From this graph, it can be seen that the error for the proposed method is ower unti the ength of the password reaches nine characters. Error factor 1, naive scheme proposed scheme Password ength Fig. 1. Error as a mutipe of N/w for passwords at distance 1, using a count-min sketch supporting maximum distance of, compared with naive scheme. Note that this cacuation above assumes that the update stage is as described in Section 3., aowing for estimates within distance, even though we are currenty estimating distance 1. If the sketch is not going to be used to estimate counts at distance at a, then it is possibe to remove the distance section. Updates for passwords at distance woud then not be incuded, making the expected maximum error for words at distance 1 equa to (1 + )N/w. A graph comparing the expected maximum error using the traditiona method and this variation in which ony distance 1 is incuded can be seen beow, in Figure. For this graph, the size of the aphabet is again α = 7. The proposed method has a ower error than the naive method unti the ength of the password reaches 36 characters. Estimates at distance. For passwords at distance, ( α estimates woud have to be done using the origina sketch. This makes the error for the origina sketch α ( ( N/w whereas our proposed technique has an error of 6 ( (1 + + )N/w. When the size of the aphabet is α = 7, the resuting differences in error can be seen in Figure 3: the error when using our proposed scheme is better than the naive method when the ength of the word is ess than 4 characters. 5 Exampe Parameter Instantiation By using the error estimates found in previous sections, it is possibe to estimate the size of the tabe required in order to obtain certain eves of accuracy. To do this, the Markov inequaity can be appied to a expected maximum errors. The resuts are as foows:

11 Count-Min Sketches for Estimating Password Frequency within Distance 11 Error factor,000 1,000 naive scheme proposed scheme Password ength Fig.. Error as a mutipe of N/w for passwords at distance 1, using a count-min sketch supporting maximum distance of 1, compared with naive scheme naive scheme proposed scheme Error factor Password ength Fig. 3. Error as a mutipe of N/w for passwords at distance, compared with naive scheme.

12 1 Leah South and Dougas Stebia When estimating the number of times a specific word appears, there is an error of at most (1 + + ( ) )N/w with probabiity of at east 1 ( 1 )h, where h is the number of hash functions. There is an error of at most 4(1 + + ( )N/w with probabiity of at east 1 ( 1 )h when estimating words within distance 1. There is an error of at most 1 ( ( (1 + + )N/w with probabiity of at east 1 ( 1 )h when estimating words within distance. The foowing exact parameter cacuations are done for distances one and two, but not for Hamming distance zero because, if the user desires a certain error rate for estimating exact words, it is preferabe to use the origina count-min sketch. If the purpose of this sketch was to estimate passwords within distance 1 with an error of at most 1% with probabiity of at east 99.9%, then the width w of the tabe and the number h of hash functions woud have to be as foows: 4(1 + + ( ) = 1 w ( ) h 1 = = w = = h 6.64 ( The number of hash functions woud have to be 7 or more and the width of the tabe woud depend on the ength of the password. If the ength of the passwords is 6 characters, then the width of the tabe woud have to be at east 48,400. This size woud be smaer if the tabe ony incuded passwords within distance zero to one, as suggested previousy. Simiary, if the sketch was needed to estimate passwords within distance with an error of at most 1% with probabiity of at east 99.9%, then the number of hash functions woud sti be at east 7 but the width of the tabe woud be: 1 ( ( (1 + + ) = 1 w 100 ( ( ) ( = w = For 6-character passwords, this woud make the width 338,800. Whie this width seems arge, it is possibe that the fase positive (or error) rate coud be higher when estimating words within specified distances. )) ( )) 6 Concusion When no restrictions are paced on passwords choices, users tend to choose popuar passwords. This eaves systems vunerabe to statistica guessing attacks. By imiting the percentage of popuar passwords, these attacks are not as successfu. In order to do this, efficient toos must be avaiabe to track password usage. The count-min sketch can be used to estimate the counts of password usage within a system. However, the count-min sketch is not as effective when estimating the

13 Count-Min Sketches for Estimating Password Frequency within Distance 13 counts of passwords within specified Hamming distances. In this paper, we have proposed a variant of the count-min sketch using widcard characters that can be used to cacuate estimates of words that are cose in Hamming distance. As with the origina count-min sketch, there wi never be fase negatives where the estimate agorithm under-reports usage of the password but there may be fase positives meaning the estimate agorithm may over-report usage of the password due to coisions. We have cacuated the error rate for estimation as the Hamming distance increases, which aows for cacuation of sketch size for a given error rate. For a reasonabe aphabet size, the error rates in our proposed method are ower than they woud be using the naive approach on a standard count-min sketch for a wide range of password engths, up to 36- and 4-character passwords when estimating passwords within distances 1 and respective. Our technique is most suited when a passwords in the database have the same ength. In future work, it may be desirabe to deveop another variation in which edit distance, such as Levenshtein distance, is used, rather than Hamming distance, to take into account passwords that are cose due to deetions or insertions of characters. Other future work may be to consider the impact of using fractiona contributions for passwords within a certain distance, where the fraction is either a function of the Hamming distance divided by the password ength, or based on some mode of textua simiarity: for exampe, in eetspeak, 5 is a common substitution for s. References 1. F. Bergadano, B. Crispo, G. Ruffo. Proactive password checking with decision trees. Computer and Communications Security, 4(1):67-77, B. H. Boom. Space/time trade-offs in hash coding with aowabe errors. Communications of the ACM, 13(7):4 46, Juy J. Bonneau. The science of guessing: anayzing an anonymized corpus of 70 miion passwords. In Proc. 01 IEEE Symposium on Security and Privacy (S&P), J. Lawrence Carter and M.N. Wegman. Universa casses of hash functions. Journa of Computer and System Sciences, 18(: , G. Cormode, S. Muthukrishnan. An improved data stream summary: the count-min sketch and its appications. Journa of Agorithms, 55(1):58-75, Apri G. Cormode, S. Muthukrishnan. Approximating data with the count-min sketch. IEEE Software, 9(1):64 69, January/February U. Manber, S. Wu. An agorithm for approximate membership checking with appication to password security. Information Processing Letters, 50(1): , J.O. Piam. On the incomparabiity of entropy and margina guesswork in brute-force attacks. In Proc. INDOCRYPT 000, LNCS, vo. 1977, pp S. Schechter, C. Herey, M. Mitzenmacher. Popuarity is everything: A new approach to protecting passwords from statistica guessing attacks. In Proc. 5th USENIX Conference on Hot Topics in Security (HotSec), E. Spafford. Preventing weak password choices. Purdue University Computer Science Technica Reports, paper 875, report number 91-08,

14 14 Leah South and Dougas Stebia A Exampe Count-Min Sketch Cacuation Fix the tabe T to be of width w = 101 and height h =. A.1 Hash First we show how the hash vaues can be cacuated for a singe password, say abcd, under two hash functions. We encode each character as two-digit integer, say abcd x = 1,, 3, 4, then find the integer representation, X = Set a common prime p = 3571, and for each of the two hash functions hash k, choose the parameters a k and b k at random moduo p; for exampe, a = [1151, 111] and b = [941, 1433]. The hashes of abcd are as foows: hash 1 (X) = (( ) mod 3571) mod 101 = 0 hash (X) = (( ) mod 3571) mod 101 = 83. The vector-wise hash is thus hash(abcd, a, b) = 0, 83. A. Update We now show how to update the tabe T to record the use of a password, first updating just the entries for the password itsef, then the entries for passwords within Hamming distance. Suppose the tabe T is currenty as foows: coumn row 1 row Update at distance 0. Now, suppose we ca update(t, x = abcd, a, b, c = 1) T, which is meant to increment (since c = 1) the use of the password abcd. Assume a and b are as in the previous subsection. Then we have that hash(abcd) = 0, 83. Thus, we increment the 0th entry of row 1 and the 83rd entry of row, to obtain T : coumn row 1 row

15 Count-Min Sketches for Estimating Password Frequency within Distance 15 Update at distance 1. Next, we increment a four widcard passwords within distance 1 of abcd, namey *bcd, a*cd, ab*d, and abc*. This is done by computing the corresponding hashes hash(*bcd) = 38, 17 hash(a*cd) = 37, 47 hash(ab*d) = 37, 63 hash(abc*) = 78, 1 and updating a the corresponding entries of the tabe accordingy. Notice that a few partia coisions occur, for exampe, both ab*d and a*cd coide under hash 1, but fortunatey do not coide under hash. In our tabe extract, we ony see a few of the 8 updates since not a coumns are shown (though we imagine a the updates are appied): coumn row 1 row Update at distance. Finay, we increment a ( 4 = 6 widcard passwords within distance of abcd; namey hash(**cd) = 55, 8 hash(*b*d) = 55, 33 hash(*bc*) = 31, 36 hash(a**d) = 18, 7 hash(a*c*) = 95, 66 hash(ab**) = 95, 8 In our tabe extract, we again ony see a few of the 1 updates: coumn row 1 row A.3 Estimate Suppose we now estimate(t, x = bbcd, a, b) to obtain the estimate est 1 of the frequency of passwords at distance 1 of bbcd. Estimate at distance 0. First, we need to compute an estimate est 0 for the number of times bbcd itsef has been used. We do this by computing hash k (bbcd) and retrieving the corresponding ce from row i, then taking the minimum. In this case, hash(bbcd) = 76, 53. The 76th entry in the first row and the 53rd entry in the second row are both 0, so this yieds a (correct) estimate that the exact password bbcd has been seen 0 times before.

16 16 Leah South and Dougas Stebia Estimate at distance 1. To compute an estimate est 1 for the number of times any password at distance 1 of bbcd has been used, we first need to compute estimates for the number of times each of the four widcard passwords *bcd, b*cd, bb*d, and bbc* has been used before. hash(*bcd) = 38, 17 = est 1 1 = min{1, 1} = 1 hash(b*cd) = 100, 8 = est 1 = min{0, 0} = 0 hash(bb*d) = 100, 60 = est 1 3 = min{0, 0} = 0 hash(bbc*) = 76, 63 = est 1 4 = min{0, 1} = 0 Then, we sum these vaues and subtract the estimate for the number of times the string bbcd itsef was used: ( ) est 1 = est 1 i est 0 = 1. i