8 APPENDIX. E[m M] = (n S )(1 exp( exp(s min + c M))) (19) E[m M] n exp(s min + c M) (20) 8.1 EMPIRICAL EVALUATION OF SAMPLING

Transcription

1 8 APPENDIX 8.1 EMPIRICAL EVALUATION OF SAMPLING We wish to evauate the empirica accuracy of our samping technique on concrete exampes. We do this in two ways. First, we can sort the eements by probabiity and mae events of drawing an eement in the top 1, or the top 1, top 1, etc. We show the resuts for two random θ with different distributions in Figure 8 for 5, sampes. our method cosey matches the histogram of the true distribution. For a more comprehensive evauation, we sampe 3 vaues of θ and compute the reative error for exact samping and our approximate samping. See Figure 8. We aso present in Figure 7 the amortized speedups obtained by our method. The amortized cost is defined as the time needed to train the index, added to the runtime of samping 1, sampes. E[m M] = (n )(1 exp( exp(s min + c M))) (19) E[m M] n exp(s min + c M) () Pr[ne Smin+c M > x] = Pr[M S min c < n(n/x)] (1) = Pr[(max i S y i + G i ) S min c < n(n/x)] () Pr[max i S G i c < n(n/x)] (3) 8. LEARNING TRAINING SET For the earning experiment, we show the set of images D that we maximized the probabiity of. See Figure 9. The common theme of the images is the presence of water. 8.3 VALUE OF c For a of the proofs, we wi incude the resut with the approximate MIPS with an error of c. To recover the origina resuts in the paper, set c =. And thus, Pr[n() + G c < n(n/x)] (4) Pr[ nec e G > x] (5) Pr[exponentia( nec ) > x] (6) 8.4 SAMPLING Theorem 3.. For Agorithm 1, E[m] nec Proof. m is the number of Gumbes that are arger than B = M S min c. Gumbes can be defined by n( n(u i )) where U i is a uniform random variabe on the interva [, 1]. Thus, we can thin of each point having a uniform sampe U i and finding paces where n( n(u i )) > M S min c (17) U i > exp( exp(s min + c M)) (18) Thus, if we can find paces where U i > exp( exp(s min +c M)), then we have the vaue of m. The number of points where this occurs is distributed according to Bin(n, 1 exp( exp(s min +c M)). Thus, E[ne Smin+c M ] E[exponentia( nec nec )] = (7) Putting it a together, E[m] = E[E[m M]] nec (8) Theorem 3.3. For Agorithm, the sampe is an exact sampe with probabiity 1 δ for δ = exp( n e c ). Proof. the eements not in S T have vaues y i S min + c and G i B = n( n(1 /n)). Further, there exists an eement in S with y i S min and with G i = max G j. As ong as there is an eement in S that exceeds a the eements not in S T, the sampe wi be exact. Pr[not exact sampe] (9)

2 Runtime (in seconds) wordvec - Naive samping cost wordvec - Amortized cost wordvec - MIPS samping cost Runtime (in seconds) ImageNet - Naive samping cost ImageNet - Amortized cost ImageNet - MIPS samping cost Dataset size Speed-up wordvec - Samping speed-up ImageNet - Samping speed-up Figure 7: Left and Center: empirica comparison of the runtime of samping for a 1, randomy chosen θ on Word Embeddings and Image Net for varying fraction of the data. The amortized cost is defined as the time necessary for the samping in addition to the training time of the index. Right: Evauation of the samping speed-up for both datasets for varying fraction of the data. Frequency True Empirica Frequency True Empirica Bins Average Error Rate Average Error Rate Naive Samping Average Error Rate MIPS Samping Figure 8: Left and Center: Two randomy chosen θ with different bin distributions. We see that the empirica samping cosey matches the true distribution for a the bins. Right: Evauation of the reative error on 3 sampes of θ for the exact samping and our samping technique. The error bars are for the average error rate between the empirica distribution and the true distribution for both exact samping and our method. We see that the error rates are not statisticay significanty different.

3 Let r be the argest vaue of e yi for eements in S. Then for a i S, e yi (, r] and further, for a i S, e yi re c. We can scae these vaues of S and denote them as q i = ey i r where q i [, 1]. For the estimate Ẑ we wi draw sampes with repacement from S and denote the set as T. Denote the sampes as y (j) and the scaed versions as q (j) = ey(j) r. We use the estimate: Ẑ = i S e yi + Ẑ = 1 + X S T (n )r e yi (36) i T q (j) (37) Figure 9: The set of images D used in the earning experiment. a of the images contain water, though the content of the images is quite different. Pr[ max G j < n( n(1 /n)) + c] (3) Pr[n() n( n(u)) < n( n(1 /n)) + c] (31) Pr[( n(1 /n))e c < n(u)] (3) Pr[ n e c < n(u)] (33) Pr[exp( n e c ) > U] (34) exp( n e c ) (35) Thus, the probabiity of faiure, δ, is bounded by exp( n e c ) 8.5 PARTITION FUNCTION ESTIMATE Theorem 3.4. Agorithm 3 returns an unbiased estimate Ẑ and for 1 3 ɛ ne c n(1/δ), then with 1 δ probabiity, Ẑ ɛ Proof. Define = i eyi. Let S be the indices of the argest eements of {y i } and S = [1, n] \ S. Denote S 1 = i S eyi and 1 = i S eyi. Thus, the true partition function is = i S eyi + i S eyi = 1 + S 1. E[Ẑ] = (n )re[q(1) ] + 1 (38) E[Ẑ] = (n )r 1 e yi S r + 1 = S = i S (39) This is because S = n. Thus, Ẑ is an unbiased estimator of. However, we are concerned if it is we concentrated about its mean. Let Q be a random variabe as the scaed sampe from S and Q be the empirica mean over sampes. Thus, E[Q] = S 1 (n )r. Ẑ Ẑ Therefore, = (n )r = (n )r q (j) S 1 (4) q (j) (n )re[q] (41) Ẑ = (n )r 1 q (j) E[Q] (4) Ẑ = (n )r Q E[Q] (43) Pr[ Ẑ > ɛ] = Pr[(n )r Q E[Q] > ɛ( S )] (44) = Pr[(n )r Q E[Q] > ɛ((n )re[q] + 1 )] (45)

4 = Pr[ Q E[Q] > ɛ(e[q] + 1 )] (46) (n )r Pr[ Q E[Q] > ɛ(e[q] + e c )] (47) n If we use Chernoff (use a convexity argument to bound the MGF in terms of the mean as on page of Concentration of Measure for the Anaysis of Randomised Agorithms by Dubhashi and Panconesi) Pr[ j Q (j) E[Q] > δe[q]] exp( 1 3 δ E[Q]) (48) Pr[ Q E[Q] > δe[q]] exp( 1 3 δ E[Q]) (49) Pr[ Q E[Q] > a] exp( a E[Q] ) (5) Combining these two by setting a = ɛ(e[q] + e c n ), Pr[ Q E[Q] > ɛ(e[q] + e c )] n (51) exp( 1 3 ɛ (E[Q] + e c 1 n ) E[Q] ) (5) exp( e c ɛ 3 n ) (53) 1 Thus, as ong as 3 ɛ ne c n(1/δ), then with 1 δ probabiity, Ẑ (1 ± ɛ) returns an unbiased estimate Ẑ and for = 1 3 ɛ n n(1/δ), then with 1 δ probabiity, Ẑ (1 ± ɛ) 8.6 EXPECTATION ESTIMATE Theorem 3.5. Agorithm 4 returns an estimate ˆF such that ˆF F ɛc with probabiity δ if and Proof. Reca Ĵ = i S 8n e c ɛ n(4/δ) ɛ nec n(/δ) J = i e yi f i e yi f i + n e yi f i i T Thus, F = J/ and ˆF = Ĵ/Ẑ. To show that ˆF F = Ẑ J ɛc with probabiity 1 δ, we wi show that Ẑ Ĵ ɛ C J ɛ C each with probabiity 1 δ/. These wi be shown as two separate parts Part One Because 4 3 ɛ nec n(/δ) from Theorem 3.4, with probabiity 1 δ/ then Ẑ ɛ Part Two Ẑ Ĵ = Ĵ Ẑ Ĵ Ẑ ɛ C ɛ Ẑ For the second one is written as the foowing emma Lemma 8.1. J ɛ C with probabiity 1 δ/ for 8n ɛ n(4/δ) Proof. the smaest eement in S has probabiity e yi / 1/. Thus, for the argest eement not in S, e yi / e c /. Define Q as the random variabe of the vaue of samping i uniformy from [1, n] \ S and returning f Q = eyi i e c C

5 Q [ 1, 1] and that E[Q] = 1 n Ce i S c eyi f i. The eements of T are sampes {y (j) } i and {f (i) } i and thus we can define Q (j) = ey(j) e c f (j) C J Ĵ = 1 e yi f i + n i S = (n )Cec E[Q] 1 From Hoeffding s Inequaity, Thus, Pr[ E[Q] 1 j e y(j) f (j) Q (i) Q (i) > t] exp( t ) Pr[ J Ĵ (n )Cec > t] exp( t ) Defining t = ɛ (n )e c j we get that (n )Cec t = ɛ C, so Pr[ J Ĵ > ɛ C] exp( ɛ 8(n ) e c ) Ca the LSH instances {L i } i and for the i th instance, set the ower tuned vaue to be S i, = (c/)(i 1) M 1 M and the higher tuned vaue to be S i,1 = (c/)i M 1 M. Thus, S 1 S = c/. Further, set the faiure probabiity of each LSH instance to be δ = δn LSH so that with high probabiity, each of the LSH instances wi not fai to find each of the top vaues. At query time, hash the query θ and et B i be the bucets of neighbors from L i. From the LSH guarantee, there are a sma constant number of eements in B i that are smaer than S i, and with high probabiity, a eements arger than S i,1 wi be in B i. Find the neighboring pair of LSH instances L i and L i+1 where B i+1 and B i. Coect = B i+1 eements B B i B i+1 where the eements are arger than S i, (a but a constant number wi be arger than S i, ). Then return the eements S = B B i+1. any eements arger than S i+1,1 wi be in S with high probabiity because they wi be contained in B i+1. So max x S θ φ(x) S i+1,1. Further, by construction, min x S θ φ(x) S i,. Thus, the technique returns the approximate top eements with high probabiity with a gap of S i+1,1 S i, = (c/) = c. This technique wi have a tota runtime of O( + (og() + og(1/δ)) og(n)n ρ ) where ρ < 1. Thus, we have a subinear approximate top eement MIPS technique. Pr[ J Ĵ > ɛ C] exp( ɛ 8n e c ) Thus, the concusion of the Lemma foows for 8n e c ɛ n(4/δ) With this emma, the concusion of the theorem foows. Theorem 3.6. There exists a MIPS technique that returns the approximate top eements in subinear time. Proof. For the data structure, we create a sequence of LSH instances that are tuned to vaues that are c/ apart. Thus, if θ M 1 and φ(x) M, then θ φ(x) M 1 M. And we create n LSH = 4M1M c instances.