This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and

Transcription

1 This article appeared in a ournal published by Elsevier. The attached copy is furnished to the author for internal non-coercial research and education use, including for instruction at the authors institution and sharing with colleagues. Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited. In ost cases authors are peritted to post their version of the article (e.g. in Word or Tex for) to their personal website or institutional repository. Authors requiring further inforation regarding Elsevier s archiving and anuscript policies are encouraged to visit:

2 Statistics and Probability Letters 86 (204) 9 98 Contents lists available at ScienceDirect Statistics and Probability Letters ournal hoepage: Tight lower bound on the probability of a binoial exceeding its expectation Spencer Greenberg a,, Mehryar Mohri a,b a Courant Institute of Matheatical Sciences, 25 Mercer Street, New Yor, NY 002, United States b Google Research, 76 Ninth Avenue, New Yor, NY 00, United States a r t i c l e i n f o a b s t r a c t Article history: Received 2 Noveber 20 Received in revised for Deceber 20 Accepted 2 Deceber 20 Available online 22 Deceber 20 Keywords: Binoial distribution Lower bound Expected value Relative deviation Machine learning We give the proof of a tight lower bound on the probability that a binoial rando variable exceeds its expected value. The inequality plays an iportant role in a variety of contexts, including the analysis of relative deviation bounds in learning theory and generalization bounds for unbounded loss functions. 20 Elsevier B.V. All rights reserved.. Motivation This paper presents a tight lower bound on the probability that a binoial rando variable exceeds its expected value. If the binoial distribution were syetric around its ean, such a bound would be trivially equal to /2. And indeed, when the nuber of trials for a binoial distribution is large, and the probability p of success on each trial is not too close to 0 or to, the binoial distribution is approxiately syetric. With p is fixed, and sufficiently large, the de Moivre Laplace theore tells us that we can approxiate the binoial distribution with a noral distribution. But, when p is close to 0 or, or the nuber of trials is sall, substantial asyetry around the ean can arise, as illustrated by Fig., which shows the binoial distribution for different values of and p. The lower bound we prove has been invoed several ties in the achine learning literature, starting with wor on relative deviation bounds by Vapni (998), where it is stated without proof. Relative deviation bounds are useful bounds in learning theory that provide ore insight than the standard generalization bounds because the approxiation error is scaled by the square root of the true error. In particular, they lead to sharper bounds for epirical ris iniization, and play a critical role in the analysis of the generalization bounds for unbounded loss functions (Cortes et al., 200). This binoial inequality is entioned and used again without proof or reference in Anthony and Shawe-Taylor (99), where the authors iprove the original wor of Vapni (998) on relative deviation bounds by a constant factor. The sae clai later appears in Vapni (2006) and iplicitly in other publications referring to the relative deviations bounds of Vapni (998). To the best of our nowledge, there is no publication giving an actual proof of this inequality in the achine learning literature. Our search efforts for a proof in the statistics literature were also unsuccessful. Instead, soe references suggest in fact that such a proof is indeed not available. In particular, we found one attept to prove this result in the context Corresponding author. E-ail address: sgg247@nyu.edu (S. Greenberg) /$ see front atter 20 Elsevier B.V. All rights reserved.

3 92 S. Greenberg, M. Mohri / Statistics and Probability Letters 86 (204) 9 98 Fig.. Plots of the probability of getting different nubers of successes, for the binoial distribution B(, p), shown for three different values of, the nuber of trials, and p, the probability of a success on each trial. Note that in the second and third iage, the distribution is clearly not syetrical around its ean. of the analysis of soe generalization bounds (Jaeger, 2005), but the proof is not sufficient to show the general case needed for the proof of Vapni (998), and only pertains to cases where the nuber of Bernoulli trials is large enough. A concise proof of this inequality for the special case where p (Rigollet and Tong, 20) was also recently brought 2 to our attention. However that proof technique does not see to readily extend to the general case. We also derived an alternative straightforward proof for the sae special case using Slud s inequality following a suggestion of Luc Devroye (private counication). However, the proof of the general case turned out to be ore challenging. Our proof therefore sees to be the first rigorous ustification of this inequality, which is needed, aong other things, for the analysis of relative deviation bounds in achine learning. In Section 2, we start with soe preliinaries and then give the presentation of our ain result. In Section, we give a detailed proof of the inequality. 2. Main result The following is the standard definition of a binoial distribution. Definition. A rando variable X is said to be distributed according to the binoial distribution with paraeters (the nuber of trials) and p (the probability of success on each trial), if for = 0,,..., we have P[X = ] = p ( p). () The binoial distribution with paraeters and p is denoted by B(, p). It has ean p and variance p( p). The following theore is the ain result of this paper. Theore. For any positive integer and any probability p such that p >, let X be a rando variable distributed according to B(, p). Then, the following inequality holds: P X E [X] > 4, (2) where E[X] = p is the expected value of X. The lower bound is never reached but is approached asyptotically when = 2 as p fro the right. Note that when 2 = 2, the case p = is excluded fro consideration, due to our assuption p >. In words, the theore says that a 2 coin that is flipped a fixed nuber of ties always has a probability of ore than /4 of getting at least as any heads as the expected value of the nuber of heads, as long as the coin s chance of getting a head on each flip is not so low that the expected value is less than or equal to. The inequality is tight, as illustrated by Fig. 2.. Proof Our proof is based on the following series of leas and corollaries and aes use of Cap Paulson s noral approxiation to the binoial cuulative distribution function (Johnson et al., 995, 2005; Lesch and Jese, 2009). We start with a lower bound that reduces the proble to a sipler one. Lea. For all =, 2,..., and p (, + ], the following inequality holds: P X B(,p) [X E[X]] P X B(, )[X + ].

4 S. Greenberg, M. Mohri / Statistics and Probability Letters 86 (204) Fig. 2. This plot depicts P[X E[X]], the probability that a binoially distributed rando variable X exceeds its expectation, as a function of the trial success probability p. Each colored line corresponds to a different nuber of trials, = 2,,..., 8. Each colored line is dotted in the region where p, and solid in the region that our proof pertains to, where p >. The dashed horizontal line at represents the value of the lower bound. Our theore is 4 equivalent to saying that for all positive integers (not ust the values of shown in the plot), the solid portions of the colored lines never cross below the dashed horizontal line. As can be seen fro the figure, the lower bound is nearly et for any values of. Proof. Let X be a rando variable distributed according to B(, p) and let F(, p) denote P[X E[X]]. Since E[X] = p, F(, p) can be written as the following su: F(, p) = p ( p). = p We will consider the sallest value that F(, p) can tae for p (, ] and a positive integer. Observe that if we restrict p to be in the half open interval I = (, ], which represents a region between the discontinuities of F(, p) which result fro p, then we have p (, ] and so p =. Thus, we can write p I, = 0,,..., F(, p) = p ( p). The function p F(, p) is differentiable for all p I and its differential is F(, p) = ( p) p ( p). p = = Furtherore, for p I, we have p, therefore p (since in our su ), and so F(,p) 0. The inequality is p in fact strict when p 0 and p since the su ust have at least two ters and at least one of these ters ust be positive. Thus, the function p F(, p) is strictly increasing within each I. In view of that, the value of F(, p) for p I + is lower bounded by li p + F(, p), which is given by li F(, p) = p + = P X B(, =+ Therefore, for =, 2,...,, whenever p (, + ] we have F(, p) P X B(, )[X + ]. )[X + ]. Corollary. For all p (, ), the following inequality holds: P X B(,p) [X E[X]] ax P {,..., } X B(, )[X ].

5 94 S. Greenberg, M. Mohri / Statistics and Probability Letters 86 (204) 9 98 Fig.. For = 2, 22,..., 72 and, this plot depicts the values of P X B(, )[X ] as colored dots (with one color per choice of ), against the values of the bound fro Lea 2, which are shown as short horizontal lines of atching color. The upper bound that we need to deonstrate,, is 4 shown as a blue horizontal line. Proof. By Lea, the following inequality holds P X B(,p) [X E[X]] The right-hand side is equivalent to in P {,..., } X B(, )[X + ]. in P {,..., } X B(, )[X ] = ax which concludes the proof. {,..., } P X B(, )[X ] In view of Corollary, in order to prove our ain result, it suffices that we upper bound the expression P X B(, )[X ] = =0 by for all integers 2 and. Note that the case = is irrelevant since the inequality p > assued for 4 our ain result cannot hold in that case, due to p being a probability. The case = 0 can also be ignored since it corresponds to p. Finally, the case = is irrelevant, since it corresponds to p >. We note, furtherore, that when p = that iediately gives P X B(,p) [X E[X]] =. 4 Now, we introduce soe leas which will be used to prove our ain result. Lea 2. The following inequality holds for all =, 2,..., : β θ + P X B(, )[X ] Φ γ, , β + γ, where Φ: x x e s2 2 ds is the cuulative distribution function for the standard noral distribution and β 2π, γ,, and θ are defined as follows: β = + + 2/, γ, =, θ = 7 2 / 2/ Proof. Our proof aes use of Cap Paulson s noral approxiation to the binoial cuulative distribution function (Johnson et al., 995, 2005; Lesch and Jese, 2009), which helps us reforulate the bound sought in ters of the noral distribution. The Cap Paulson approxiation iproves on the classical noral approxiation by using a non-linear transforation. This is useful for odeling the asyetry that can occur in the binoial distribution. The Cap Paulson bound can be stated as follows (Johnson et al., 995, 2005): c µ P X B(,p)[X ] Φ σ p( p)

6 S. Greenberg, M. Mohri / Statistics and Probability Letters 86 (204) where c = ( b)r /, µ = a, σ = br 2/ + a, a = 9 9, b = ( + )( p), r = p p Plugging in the definitions for all of these variables yields / c µ (+)( p) 9 + p + 9 Φ = Φ σ 2/. (+)( p) 9 + p + 9 Applying this bound to the case of interest for us where p = and =, yields c µ σ = α + γ,, β + γ, with α = + / +, and with β = ( + + )2/. Thus, we can write P X B(, )[X ] Φ α + γ, β + γ, () To siplify this expression, we will first upper bound α in ters of β. To do so, we consider the ratio α = ( + )/ ( ) + β ( + = [( + )/ ] + )2/ ( + + )2/ ( +. )/ Let λ = ( + )/, which we can rearrange to write follows: α = λ [(λ )] β (λ )λ 2 λ = λ + λ + λ 2 λ. = λ + λ The expression is differentiable and its differential is given by d α = ( + λ + λ2 ) λ(2λ + ) + dλ β ( + λ + λ 2 ) 2 λ 2 = ( λ 2 ) ( + λ + λ 2 ) 2 + λ 2 = 8(λ )4 0(λ ) 0(λ ) λ 2 + λ + λ 2 2., with λ (, 2 / ]. Then, the ratio can be rewritten as For λ (, 2 ], λ , thus, the following inequality holds: 8(λ ) 4 + 0(λ ) + 0(λ ) 2 8(2 ) 4 + 0(2 ) + 0(2 ) < 9. Thus, the derivative is positive, so α is an increasing function of β λ on the interval (, 2 ] and its axiu is reached for λ = 2. For that choice of λ, the ratio can be written = = θ , which upper bounds α β. Since Φ[x] is a strictly increasing function, using α θβ yields Φ α + γ, β θ + Φ γ,. β + γ, β + γ, (4)

7 96 S. Greenberg, M. Mohri / Statistics and Probability Letters 86 (204) 9 98 We now bound the ter ( The quadratic function ( ) for =, 2,...,, achieves its iniu ) at =, giving ( ) ( ). Thus, in view of () and (4), we can write P B, Φ β θ + γ, β + γ, This concludes the proof. As illustrated by Fig., the upper bound provided by Lea 2 closely approxiates the true value. Lea. Let β = + + 2/ and γ, = for > and =, 2,...,. Then, the following inequality holds for θ = 7 2 : Φ 2 β θ + γ, β + γ, Φ β θ +. β + Proof. Since for =, 2,...,, we have γ,, the following inequality holds: β θ + γ, β + γ, β θ + ax γ γ [0,] β + γ. Since β = + + 2/ > 0, the function φ: γ β θ+ γ β +γ is continuously differentiable for γ [0, ]. Its derivative is given by φ (γ ) = γ +β (2 θ) 6(β +γ ) /2. Since 2 θ 0.56 < 0, φ (γ ) is non-negative if and only if γ β (θ 2). Thus, φ(γ ) is decreasing for γ < β (θ 2) and increasing for values of γ larger than that threshold. That iplies that the shape of the graph of φ(γ ) is such that the function s value is axiized at the end points. So ax γ [0,] φ(γ ) = ax(φ(0), φ()) = ax( β θ, β θ+ ). The inequality β β + θ β θ+ holds if and only if β β + (β + )θ 2 (β θ + )2, that is if β θ(9θ 6).022. But since β is a decreasing function of, it has β as its upper bound, and so this necessary requireent always holds. That eans that the axiu value of φ(γ ) for γ [0, ] occurs at γ =, yielding the upper bound β θ+ β, which concludes the proof. + Corollary 2. The following inequality holds for all 2 and 2: P X B(, )[X ] Proof. By Leas 2 and, we can write β θ + P X B(, )[X ] Φ β + Furtherore β = + + 2/ is a decreasing function of. Therefore, for 2, it ust always be within the range β [li β, β 2 ] = 0, [0, ], which iplies 2 2/ / β θ + β + β ax θ + 2 β + βθ + ax. β [0,β 2 ] β + The derivative of the differentiable function g: β βθ+ β+ is given by g (β) = (β+2)θ 6(β+) /2. We have that (β + 2)θ 6θ 6.77 > 0, thus g (β) 0. Hence, the axiu of g(β) occurs at β 2, where g(β 2 ) is slightly saller than Thus, we can write β θ + βθ + Φ Φ ax Φ[0.5968] < β + β [0,β 2 ] β +

8 S. Greenberg, M. Mohri / Statistics and Probability Letters 86 (204) Now, Hence, the following holds: Φ β θ + β + is a decreasing of function of, thus, for 2 it is axiized at = 2, yielding = 0.752, as required. The case = is addressed by the following lea. Lea 4. Let X be a rando variable distributed according to B(, ). Then, the following equality holds for any 2: P [X ] 4. Proof. For 2, define the function ρ by ρ() = P [X ] = = +. =0 The value of the function for = 2 is given by ρ(2) = =. Thus, to prove the result, it suffices to 4 show that ρ is non-increasing for 2. The derivative of ρ is given for all 2 by ρ () = ( ) 2 + (2 ) log. Thus, for 2, ρ () 0 if and only if 2 + (2 ) log 0. Now, for 2, using the first three ters of the expansion log = =, we can write (2 ) log (2 ) where the last inequality follows fro the proof = , 0 for 2. This shows that ρ () 0 for all 2 and concludes We now coplete the proof of our ain result, by cobining the previous leas and corollaries. Proof of Theore. By Corollary, we can write P X B(,p) [X E[X]] ax P {,..., } X B(, )[X ] = ax P X B(, )[X ], ax P {2,..., } X B(, )[X ] ax 4, = 4 = 4, where the last inequality holds by Corollary 2 and Lea 4. Corollary. For any positive integer and any probability p such that p <, let X be a rando variable distributed according to B(, p). Then, the following inequality holds: P X E [X] > 4, (5) where E[X] = p is the expected value of X. Proof. Let G(, p) be defined as G(, p) P X p E [X] = p ( p) and let F(, p) be defined as before as F(, p) P X E [X] = p ( p). =0 = p

9 98 S. Greenberg, M. Mohri / Statistics and Probability Letters 86 (204) 9 98 Then, we can write, for q = p, G(, p) = G(, q) ( q) = ( q) q =0 = ( q) t q t t = t= ( q) t= q t ( q) t q t = F(, q) = F(, p) > 4 with the inequality at the end being an application of Theore, which holds so long as q >, or equivalently, so long as p <. 4. Conclusion We presented a rigorous ustification of an inequality needed for the proof of relative deviations bounds in achine learning theory. A nuber of first attepts to find such a proof in the literature, or to prove this result ourselves, indicated that the inequality is not straightforward. Nevertheless, a sipler proof of this inequality is liely possible, and we ay present such a sipler result in the future. Acnowledgent We than Luc Devroye for discussions about the topic of this wor. References Anthony, M., Shawe-Taylor, J., 99. A result of Vapni with applications. Discrete Appl. Math. 47, Cortes, C., Mansour, Y., Mohri, M., 200. Learning Bounds for Iportance Weighting. MIT Press, Vancouver, Canada. Jaeger, S.A., Generalization bounds and coplexities based on sparsity and clustering for convex cobinations of functions fro rando classes. J. Mach. Learn. Res. 6, Johnson, N., Kep, A., Kotz, S., Univariate Discrete Distributions. In: Wiley Series in Probability and Statistics, vol.. Wiley & Sons. Johnson, N., Kotz, S., Balarishnan, N., 995. Continuous Univariate Distributions. In: Wiley Series in Probability and Matheatical Statistics: Applied Probability and Statistics, vol. 2. Wiley & Sons. Lesch, S.M., Jese, D.R., Soe suggestions for teaching about noral approxiations to Poisson and binoial distribution functions. Aer. Statist. 6, Rigollet, P., Tong, X., 20. Neyan Pearson classification, convexity and stochastic constraints. J. Mach. Learn. Res. 2, Vapni, V.N., 998. Statistical Learning Theory. Wiley-Interscience. Vapni, V.N., Estiation of Dependences Based on Epirical Data. Springer-Verlag.