Inequalities for the L 1 Deviation of the Empirical Distribution

Transcription

1 Inequalities for the L 1 Deviation of the Emirical Distribution Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, Marcelo J. Weinberger June 13, 2003 Abstract We derive bounds on the robability that the L 1 distance between the emirical distribution of a sequence of indeendent identically distributed random variables and the true distribution is more than a secified value. We also derive a generalization of Pinsker s inequality relating the L 1 distance to the divergence. Key words: Sanov s theorem, Pinsker s inequality, large deviations, L 1 distance, divergence, variational distance, Chernoff bound. 1 Preliminaries Let A denote the finite set {1,..., a}. For two robability distributions P and Q on A let P Q 1 = a P k Qk k=1 denote the variational, or L 1, distance between P and Q. For a sequence of symbols x m = x 1,..., x m A m, let ˆP x m be the emirical robability distribution on A defined by ˆP x mj = 1 m m 1x i = j, 1 i=1 Hewlett-Packard Laboratories, 1501 Page Mill Rd., Palo Alto, CA and Stanford University, Stanford, CA 94305, tsachy@stanford.edu. Hewlett-Packard Laboratories, 1501 Page Mill Rd., Palo Alto, CA 94304, eord@hl.h.com, seroussi@hl.h.com, marcelo@hl.h.com. Princeton University, Princeton, NJ 08544, verdu@rinceton.edu. 1

2 where 1 denotes the indicator function of the secified event. For robability distributions P and Q on A, let DP Q = a k=1 P k log P k Qk 2 denote the divergence between P and Q, where throughout log denotes the natural logarithm. For 0 1, 2 1 let D B 1 2 = 1 log log denote the binary divergence, whereas for 1, 2 / [0, 1] 2 we set D B 1 2 =. The following conventions imlied by continuity are adoted: for c > 0, c/0 =, c/ = 0, c =, log =, e = 0. Additionally, in 2 and 3 it is assumed that 0 log0/0 = 0 and 0 log 0 = 0. For [0, 1/2, we define and, by continuity, set ϕ1/2 = 2. ϕ = log 1 For a robability distribution P on A, we define Note that π P 1/2 for any P. π P = max minp A, 1 P A. 5 A A Finally, throughout we take the minimum of a function over an emty set to be. 2 Results In this work, we rove the following theorem on the robability that the L 1 distance between the emirical distribution of a sequence of indeendent identically distributed random variables and the true distribution is more than a secified value. Theorem 2.1 Let P be a robability distribution on the set A = {1,..., a}. Let X m = X 1, X 2,..., X m be indeendent identically distributed random variables distributed according to 4 2

3 P. Then, for all ɛ > 0, Additionally, lim m 1 m log P r P ˆP X m 1 ɛ = min D B P A + ɛ A A 2 P A. 6 P r P ˆP X m 1 ɛ 2 a 2e m[min A A D B P A+ɛ/2 P A] 7 2 a 2e mϕπ P ɛ 2 /4. 8 We also strengthen Pinsker s inequality [3], Problem 3.17; [2], Lemma , relating the L 1 distance to the divergence, as follows. Theorem 2.2 Let P and Q be two robability distributions on the set A = {1,..., a}. Then DP Q P Q 1 2 ϕπ Q. 9 Theorems 2.1 and 2.2 are discussed in Section 3 and roved in Section 4. 3 Discussion Pinkser s inequality states that P Q 1 2DP Q. 10 Theorem 2.2 strengthens 10 since, by Proosition 4.3 below, ϕπ Q 2 with equality if and only if π Q = 1/2. The method-of-tyes argument underlying the roof of Sanov s Theorem [2], Theorem , and Pinsker s inequality, can be used to directly derive the well-known bound for the L 1 norm P r P ˆP X m 1 ɛ n + 1 a 1 e m[min P : P P 1 ɛ DP P ] 11 n + 1 a 1 e mɛ2 /2. 12 It follows from Sanov s Theorem and 6 in Theorem 2.1 that the exonent in 11 is the same as in 7. A minor advantage of 7 is the imroved factor multilying the exonential decay, 3

4 i.e., it avoids the olynomial factor of In comaring 8 and 12, we see that 8 not only retains the imroved factor multilying the exonential decay, but, more imortantly, may also have a faster exonential decay. Again, this follows from the fact that ϕπ P 2 with equality if and only if π P = 1/2 Proosition 4.3. One may wonder why weakening 7 to 8 is useful at all. In many situations, the comlicated deendence on ɛ of the exonent in 7 leads to an intractable analysis. The usual simlifying ste involves alying the Pinsker, Hoeffding, or Bernstein inequalities see, e.g., [1]. The Pinsker and Hoeffding inequalities have the disadvantage of droing the deendence on the underlying distribution by assuming a worst case behavior. Bernstein s inequality, on the other hand, is useful only for small ɛ. The bound 8 avoids these disadvantages by scaling the ɛ 2 form of the exonents of the Hoeffding and Pinsker aroaches with the otimum distribution-deendent factor. One concrete alication of this tool is in the roof of Theorem 2 of [4], in which numerous bounds on P r P ˆP X m 1 ɛ for different P s and ɛ s must be aggregated through a series of union bounds and otimizations over the constituent ɛ s. The comlicated deendence on ɛ of the exonent in 7 makes the aggregation of bounds of this form rohibitively comlex. The simler exonent of 8, on the other hand, leads to a tractable analysis. The resence of ϕπ P in the exonent of 8 allows the final aggregated exonent to retain a deendence on the constituent distributions P unlike 11, which in [4] corresond to the er inut channel outut robabilities of a discrete memoryless channel. It should be noticed that other refinements of Pinsker s inequality considered in the literature see [5], [6, Corollary 1.4], and references therein are still indeendent of the underlying distribution. Rather, the idea is to add higher owers of P Q 1 to the lower bound on DP Q e.g., one such result states that DP Q 1/2 P Q /36 P Q In the binary case, the Chernoff bounding technique, as oosed to the method-of-tyes, also yields the multilicative factor of 7. In fact, the arguments behind our roof of 7 can be viewed as extending the Chernoff bounding technique to the non-binary case. 4

5 4 Proofs of main theorems The roofs of Theorems 2.1 and 2.2 will make use of the following four roositions, which are roved in Aendix A. Proosition 4.1 For [0, 1/2], D B + ɛ D B 1 + ɛ 1. Proosition 4.2 For [0, 1/2] D B + ɛ inf ɛ 0,1 ] ɛ 2 = ϕ. Proosition 4.3 The function ϕ is strictly decreasing for [0, 1/2]. Proosition 4.4 For all distributions Q on A = {1,..., a}, min DP Q = min D P : P B QA + ɛ Q 1 ɛ A A 2 QA. Proof of Theorem 2.1: By Sanov s Theorem cf., e.g., Equation of Theorem in [2], lim m 1 m log P r P ˆP X m 1 ɛ = min where 13 follows from Proosition 4.4, roving 6. P : P P 1 ɛ = min A A D B DP P P A + ɛ 2 P A, 13 To rove 7 and 8, we start with the well-known fact that for any distribution Q on A which, together with a union bound, imlies that Q P 1 = 2 maxqa P A, 14 A A P r ˆP X m P 1 ɛ P r ˆPX ma P A ɛ A A 5

6 For ɛ > 0 and A = A,, clearly P r ˆP X ma P A ɛ/2 = 0. For the other subsets of A, the standard Chernoff bounding technique alied to the binary random variable 1X i A shows that P r ˆPX ma P A 2 ɛ e md BP A+ɛ/2 P A. 16 Combining 16 with 15 results in P r ˆP X m P 1 ɛ A A:A A, e md BP A+ɛ/2 P A 2 a 2e m[min A A D B P A+ɛ/2 P A], roving 7. Now, min D B A A = min A A D B P A + ɛ 2 P A minp A, 1 P A + ɛ 2 min A A ϕminp A, 1 P Aɛ 2 4 = ϕπ P ɛ 2, 4 minp A, 1 P A where 17, 18, and 19 follow, resectively, from Proosition 4.1, Proosition 4.2 the case ɛ/2 > 1 minp A, 1 P A follows by our conventions, and Proosition 4.3 and 5, comleting the roof of 8. Proof of Theorem 2.2: We have DP Q min DP Q P : P Q 1 P Q 1 = min A A D B QA + P Q 1 2 QA 20 ϕπ Q P Q 2 1, 21 4 where 20 follows from Proosition 4.4 and 21 follows from Proositions 4.1, 4.2, and 4.3, as in equations 17 through 19 in the roof of Theorem 2.1. The roof is comleted by rearranging terms. 6

7 Aendix A Proofs of roositions Proof of Proosition 4.1: If < ɛ, the result holds by the conventions suorting the definition 3. For ɛ, let f, ɛ = D B + ɛ D B 1 + ɛ 1. We show that f, ɛ 0 for [ɛ, 1/2] by showing below that f/ =1/2 0 and 2 f/ 2 0 for ɛ, 1/2]. The claim then follows for ɛ, 1/2] since f1/2, ɛ = 0. The claim for = ɛ follows by continuity. By definition, Differentiating we obtain and f f, ɛ = + ɛ log + ɛ 1 + ɛ log 1 + ɛ ɛ log 1 ɛ 1 ɛ log ɛ. 2ɛ = log + ɛ log1 ɛ + log1 + ɛ log ɛ 2ɛ 1, A.1 2 f 1 = 2 + ɛ 1 ɛ ɛ ɛ + 2ɛ 2ɛ 2 1 [ 2 = 2ɛ 1 2 ɛ ɛ + 1 ] From A.2 we see that 2 / f 2 non-decreasing for x [0, 1/2]. From A.1 we have = 2ɛ [ ɛ ɛ 2 f ]. A.2 0 for ɛ, 1/2], as claimed above, since x1 x is =1/2 = 2 log1 + 2ɛ 2 log1 2ɛ 8ɛ 7

8 so that f/ =1/2,ɛ=0 = 0. Differentiating with resect to ɛ we have 2 f ɛ =1/2 = = ɛ ɛ ɛ 8, 2 which is non-negative for ɛ [0, 1/2. This imlies that f/ =1/2 0 for all ɛ [0, 1/2, again, as claimed above. Proof of Proosition 4.2: For a given [0, 1/2], consider the function fɛ = D B + ɛ ϕɛ 2 = + ɛ log + ɛ + 1 ɛ log 1 ɛ 1 Let f ɛ be the derivative of fɛ. We will show below that: 1. f ɛ 0 for ɛ [0, 1/2 ]. 2. f ɛ 0 for ɛ [1/2, 1 2]. 3. f ɛ 0 for ɛ [1 2, 1 ]. ɛ2 1 2 log 1. The first roerty of f ɛ and f0 = 0, together, imly that fɛ 0 for ɛ [0, 1/2 ]. Similarly, roerties 2 and 3 and the readily verified f1 2 = 0 imly that fɛ 0 for [1/2, 1 ]. Thus, fɛ 0 with equality at ɛ = 1 2, which roves the roosition. We now verify the above roerties 1-3 of f ɛ. Let After differentiating we have and gɛ = f ɛ = log + ɛ log 1 ɛ 1 g ɛ = 1 + ɛ ɛ 2ϕ g ɛ = 1 + ɛ ɛ 2, 2ɛ 1 2 log 1. from which we see that g ɛ 0 for ɛ [0, 1/2 ] and g ɛ 0 for ɛ [1/2, 1 ]. Thus, gɛ is concave for ɛ [0, 1/2 ], which, together with the fact that g0 = g1/2 = 0, 8

9 imlies Proerty 1 above. Analogously, roerties 2 and 3 follow from the convexity of gɛ for ɛ [1/2, 1 ] and the fact that g1/2 = g1 2 = 0. Proof of Proosition 4.3: Differentiating ϕ see 4 with resect to yields [ ϕ 1 = log 1 ]. Thus, to show that ϕ < 0 for 0, 1/2, it suffices to show that g = log 1 To this end, note that g1/2 = 0 and that the derivative of g is < 0. g = = [ 1 = 1 1 ] 2. In articular, g > 0 for 1/2. Continuity arguments comlete the roof for = 1/2 and = 0. Proof of Proosition 4.4: The argument is similar to a ste in the roof of Pinsker s inequality [3], Problem 3.17; [2], Lemma For distributions P and Q on A let AP, Q = {a A : P a Qa}. It is then not difficult to see that P Q 1 = 2P AP, Q QAP, Q. A.3 Therefore min DP Q P : P Q 1 ɛ [ ] = min min DP Q A A P : P Q 1 ɛ,ap,q=a [ ] min min D BP A QA A.4 A A P : P Q 1 ɛ,ap,q=a [ ] = min min D BQA + P A QA QA A.5 A A P :P A QA ɛ/2,ap,q=a = min D B QA + ɛ QA, A.6 A A 2 9

10 where A.4 follows from the data rocessing inequality [3], Lemma 3.11, A.5 follows from A.3, and A.6 follows from the fact that D B q is non-decreasing in for q, continuity, and, in the event that the set in A.5 is emty, from our conventions. The reverse inequality follows by letting A = arg min D B QA + ɛ A A 2 QA. If QA + ɛ/2 > 1 then D B QA + ɛ/2 QA =, in which case the reverse inequality is trivial. If QA + ɛ/2 1, define the distribution P as P k = { QA +ɛ/2qk QA for k A 1 QA ɛ/2qk 1 QA for k A \ A, and check that DP Q = D B QA + ɛ/2 QA and P Q 1 = ɛ. References [1] L. Devroye, L. Györfi, G. Lugosi, A Probabilistic Theory of Pattern Recognition, Sringer- Verlag New York, Inc., [2] T. M. Cover and J. A. Thomas, Elements of Information Theory, Wiley, New York, [3] I. Csiszár and J. Körner, Information Theory: Coding Theorems for Discrete Memoryless Systems, Akadémiai Kiadó, Budaest, [4] T. Weissman, E. Ordentlich, G. Seroussi, S. Verdú, M. J. Weinberger, Universal Discrete Denoising: Known Channel, HP Labs. Tech. Re., HPL , February Submitted to IEEE Trans. Inform. Theory. [5] F. Tosøe, Some Inequalities for Information Divergence and Related Measures of Discrimination, IEEE Trans. Inform. Theory, vol. IT-46, , July [6] F. Tosøe, Bounds for Entroy and Divergence for Distributions over a Two-element Set, J. of Inequalities in Pure and Alied Mathematics, vol. 2,