Lecture 7: Testing Distributions

Transcription

1 CSE 5: Sublinear (and Streaming) Algorithm Spring 014 Lecture 7: Teting Ditribution April 1, 014 Lecturer: Paul Beame Scribe: Paul Beame 1 Teting Uniformity of Ditribution We return today to property teting and a urpriing application of F etimation (or equivalently l -norm approximation) for the problem of teting the cloene to uniform of a probability ditribution. We conider dicrete ditribution on the domain [n]. Each uch ditribution i given by a probability vector p = (p 1,..., p n ). The uniform ditribution U on [n] ha U i = 1/n for all i [n]. The property teting problem we focu on firt i to do the following: Accept if p i uniform Reject if p i ε-far from uniform. Thi definition i ambiguou without a notion of ditance between ditribution. The mot natural notion of ditance between ditribution i the l 1 ditance between their probability vector, where p q 1 = i p i q i = max A [n]p(a) q(a). p 1 1 i called the total variation ditance between ditribution, while max A [n]p(a) q(a) i called the tatitical ditance. Another notion that we will conider i the l ditance between ditribution, p q = p i q i. While l 1 ditance i the mot natural, it will turn out that analyzing the l ditance will be the mot ueful algorithmically. i 1

2 Thee can be cloely related if p = (1, 0,..., 0) then p U 1 = /n and p U 1 but if p = (/n,..., /n, 0,..., 0) and q = (0,..., 0, /n,..., /n) then p q 1 = but p q i only / n. The latter i the larget gap poible. Oberve that for any probability ditribution p on [n], 1/ n p p 1 = 1 and p q /n 1/ p q p q 1 n 1/ p q. Naive algorithm for l 1 ditance from uniformity: Chooe ample and compute the ample ditribution p a an approximation to p. Compute U p 1 to etimate p U 1. The problem with thi approach i that p p 1 i huge unle = Ω(n). (Otherwie, the algorithm i trying to etimate n different quantitie with o(n) ample. In thi cae, every p would be ditance 1 o(1) from uniform.) Alternative idea: Ue an algorithm for property teting with repect to l ditance to derive an algorithm for property teting with repect to l ditance. The reaon for the utility of the l ditance i cloely related to the connection of F to the ize of elf-join. Oberve that the colliion probability for independent ample from p, P a,b p [a = b] = p i = p. Alo P a,b U [a = b] = 1/n. Then the quare of the l ditance, p U = = = (p i 1 n ) (p i p i n + 1 n ) n p i p i + n n n ) = p n + 1 n ) = p 1 n, or equivalently p = 1 n + p U. Thi ugget ampling to get a good etimate of the colliion probability, p. What error will we need?

3 l ditance: If p U > ε then p = 1/n + p U > 1/n + ε. On the other hand U = 1/n. In order to eparate p from U, we would need to eparate 1/n from omething > 1/n + ε, even with error. It uffice to have an additive error in computing p of at mot ε /. l 1 ditance: If p U 1 > ε then p U > ε/ n o p U > ε /n. Therefore, p = 1/n + p U > 1/n + ε /n = 1/n(1 + ε ). In thi cae we need to ditinguih 1/n from omething larger than 1/n (1 + ε ). It i natural to conider multiplicative error in thi cae. With multiplicative error 1 ± ε / oberve that if p U 1 1 > ε then p would evaluate to trictly more than 1/n (1 + ε )(1 ε /) = 1/n(1 + ε / ε 4 /) 1/n(1 + ε /) which i the larget that the value could be if p = U. Property teting algorithm for uniformity: Chooe independent { ample x 1,..., x from p. 1 if x i = x j Let Y ij = 0 otherwie. Output X = Y ij/ ( ). Analyi By definition, E(Y ij = p. Therefore, ince X i the average of the Y ij, E(X) = p and o X i an unbiaed etimator for p. ( ) V ar( X) = V ar( Y ij ) = E([ Y ij E( Y ij )] ) = E([ Y ij E(Y ij )] ).

4 We write Ŷij = Y ij E(Y ij ) = Y ij p and note that E(Ŷij) = 0. Then ( ) V ar( X) = E([ Ŷ ij ] ) = E( Ŷ ij Ŷ kl ) k<l = E( Ŷ ij Ŷ kl ) = k<l E(Ŷ ij) + k<l {i,j,k,l} = E(ŶijŶkl)) + k<l {i,j,k,l} =4 E(ŶijŶkl). Now if i, j, k, l are all ditinct, the random variable Ŷij and Ŷkl are independent o E(ŶijŶkl) = E(Ŷij) E(Ŷkl) = 0 0 = 0 and hence the third term in the um i 0. More generally, for any i < j and k < l, E(ŶijŶkl) = E((Y ij p )(Y kl p )) = E(Y ij Y kl p (E(Y ij + E(Y kl ) + p 4 = E(Y ij Y kl p 4 + p 4 = E(Y ij Y kl p 4 < E(Y ij Y kl ). In particular, E(Ŷ ij) E(Yij) = E(Y ij ) = p ince Y ij i an indicator variable. Therefore the firt term in the um i ( ) p. Alo, for every i < j and k < l uch that {i, j, k, l} =, the event Y i,j Y k,l i the event that all the ample indexed by them produce the ame value. For any three ample, thi probability i preciely n p i = p. For each of the ( ) choice of three ample, there are 6 way that thi can correpond to i < j and k < l: if i = l or k = j then there i only one way to extend thi to three ample, if either i = k or j = l there are a further two way to order the remaining indice. Putting thi together, we have V ar( ( ) ( ) Y ij ) p + 6 p. 4

5 Therefore, V ar(x) = V ar( ( ) Y ij )/ ( ) p + 6 ( ) p ( ) = p ( 1) + 4( ) p ( 1) < p ( 1) + 4 p. l teting quality approximation for p : Thi require an ε / additive approximation. Oberve that p, p 1 o V ar(x) 1/binom + 4/ < 5/ for 5. Therefore, by Chebyhev inequality, P[ X p ε /] V ar(x) ε /) 1/, for = O(ε ), in particular = 60/ε. Therefore, for contant ε, only a contant number of ample are required to tet the proximity p to uniform ditribution uing the l error meaure. l 1 teting quality approximation for p : Thi require a 1 ± ε / multiplicative approximation. For convenience, write ε 0 = ε /. Again via Chebyhev inequality, P[ X p ε p ] V ar(x) ε p 4 [ ] 1 p ε p 4 ( 1) + 4 p = + 4 p ( 1)ε 0 p ε 0 p 4 Since p 1/n, 1/6 ( 1)ε 0 p for 4 n/ε 0. Since p p, we have p / p 4 1/ p. Since p 1/ n, 4 p ε 0 p 4 4 ε 0 p 1/6 for 4 n/ε 0. Therefore, for = 4 n/ε 0 ample with probability at leat /, we obtain a 1 ± ε 0 factor approximation for p. 5

6 In the application to teting uniformity with repect to l 1 ditance we have ε 0 = ε / and hence O(ε 4 n) ample uffice. (Note that the upper bound p / p 4 1/ p n that we ued i aymptotically optimal. Conider a ditribution which ha probability p n = 1/ n, p i = 1/n for i n n and p i = 0 otherwie. Thi ditribution ha p /n and p 1/n /.) Improvement and a Lower Bound The original l ditance teter for uniformity i baed on a teter due to Goldreich and Ron []. The verion here and extenion to l 1 ditance i due to Batu et al. []. Note that the 4-th power dependence on the invere ditance 1/ε i not optimal. The exact power i a bit le of an iue in property teting becaue unlike the treaming cae, the comparion i with a polynomial in n rather than log n. An aymptotically optimal dependence of Θ(ε n) wa hown by Paninki []. The baic idea imilarly involve colliion but intead the algorithm extimate the ditance baed on the number of ditinct ample. Thi avoid the large variance one can get if there are certain element, uch a in the example above where the probability of a triple colliion i too large. A lower bound of Ω( n) on the number of ample needed i not hard to how uing a ditribution related to the hard intance for element ditinctne teting. Conider the ditribution p that ha probability 1/n for all element larger than εn, but ha probability /n for the firt εn and the ret 0. Thi ha p U 1 = ε, but i = o( n) then the algorithm will not ee any colliion with probability near 1. Extenion One can extend thi algorithm to one that tet the ditance to any fixed known ditribution q uing the above algorithm for the uniform ditribution: Group the element into bin baed on their probabilitie o that every element ha the ame probability up to a 1 + ε factor, down to probabilitie at mot ε/(n log n) ay. (Element with maller probability occur too rarely in total probability to matter.) Thi give O(log n / log(1 + ε)) = O(ε 1 log n) bin. The algorithm will firt apply the naive ampling algorithm to etimate the probability of each bin. Since there are only the ize of O(ε 1 log n) bin, Chernoff bound imply that all ize of the correponding bin for p can be etimated with mall additive error uing only ε 1 log O(1) n ample. If any of thee i too far from that of q, the algorithm will reject. Within each bin, the ditribution i approximately uniform and the above tet can be applied, provided that the bin ha ufficiently large probability under q. Again, if the error on the bin i too large, the algorithm will reject. If p and q are both given a input, then the ample complexity required i larger, ε Θ(1) n / but part of the general idea i imilar to that of the uniform cae. Namely, one ample, element from both ditribution and compare the colliion within the p ample and within the q ample and compare thi to the colliion between the p and q ample. The variance of thi tet i not good if one of the ditribution contain ome element that occur too frequently. However, by firt filtering out the high probability element (thoe with probability Ω(n / )) and checking that thoe agree for the two ditribution uing the naive algorithm, Batu et al. derive the above bound. 6