s: 1 (corresponding author); 2

Transcription

1 Internatonal Workshop Stochastc Programmng for Implementaton and Advanced Applcatons (STOPROG-01) July 3 6, 01, Nernga, Lthuana ISBN L Sakalauskas, A Tomasgard, S WWallace (Eds): Proceedngs Vlnus, 01, pp The Assocaton of Lthuanan Serals, Lthuana, 01 do:10500/stoprog0117 LAW OF SMALL NUMBERS AS CONCENTRATION INEQUALITIES FOR SUMS OF INDEPENDENT RANDOM SETSAND RANDOM SET VALUED MAPPINGS Vladmr I Norkn 1, Roger J-B Wets 1 Department of Operatons Research, Insttute of Cybernetcs, Kev Department of Mathematcs, Unv of Calforna, Davs E-mals: 1 norkn@comua (correspondng author); rbwets@ucdavsedu Abstract In the paper we study concentraton of sample averages (Mnkowsk's sums) of ndependent bounded random sets and set valued mappngs around ther expectatons Sets and mappngs are consdered n a Hlbert space Concentraton s formulated n the form of exponental bounds on probabltes of normalzed large devatons In a sense, concentraton phenomenon reflects the law of small numbers, descrbng non-asymptotc behavor of the sample averages We sequentally consder concentraton nequaltes for bounded random varables, functons, vectors, sets and mappngs, dervng next nequaltes from precedng cases Thus we derve concentraton nequaltes wth explct constants for random sets and mappngs from the sharpest avalable (Talagrand type) nequaltes for random functons and vectors The most explct nequaltes are obtaned n case of dscrete dstrbutons The obtaned results contrbute to substantaton of the Monte Carlo method n nfnte dmensonal spaces Keywords: concentraton nequalty, Talagrand's nequalty, random set, random set-valued mappng, Mnkowsk's averages, large devaton, law of small numbers 1 Introducton Concentraton nequaltes descrbe tal behavor of the probabltes of large devatons for sums of ndependent random varables from ther mean They, lke lmt theorems, determne rate of convergence n the law of large numbers Moreover, they are vald for any fnte sums and thus express, n a sense, a law of small numbers Classcal results of ths type are Bernsten's, Chernoff's and Hoeffdng's exponental nequaltes for bounded random varables There are extensons of these results to random vectors and random functons (see, eg (Talagrand, 1994, 1996), (Pflug, 003), (Nemrovsk, 004), (Boucheron et al, 005), (Stenwart and Chrstmann, 008), (Shapro et al, 009)) In the present paper we derve such nequaltes for random sets (and mappngs) Frst nonasymptotc result of ths type was obtaned by (Artsten, 1984) for fnte dmensonal ndependent random sets It exponentally bounds the probablty that Housdorff dstance between sample average set and ts mean exceeds a gven threshold The result s essentally fnte dmensonal snce the bound heavly depends on the dmenson number, the more the dmenson the worse the bounds Unlke ths, contemporary concentraton nequaltes are dmenson free (eg are vald for Banach space valued random varables), but explot dfferent complexty measures for the range of the random obect Asymptotc large devaton result for random sets n a separable Banach space was obtaned n (Cerf, 1999) We derve explct non-asymptotc concentraton nequaltes for bounded random sets and mappngs n a separable nfnte dmensonal Hlbert space In the present paper we sequentally revew and update concentraton nequaltes for bounded random varables, functons, vectors, random sets and set valued mappngs, dervng next nequaltes from precedng cases Especally sharp and explct results are obtaned n case of dscrete dstrbutons A closely related asymptotc result, a central lmt theorem for random sets wth a dscrete dstrbuton, was obtaned n (Cresse, 1979) Thus the man contrbuton of the paper conssts n the followng We mprove (Artsten's, 1984) concentraton results for bounded random sets makng the estmates much sharper and equally applcable to fnte and nfnte dmensonal (separable Hlbert space) cases The estmates stll reman nonasymptotc, e vald for any fnte sum of ndependent random sets (unlke (Cerf, 1999)) We derve concentraton nequaltes from the sharpest avalable analogues results for random vectors and random functons Concentraton nequaltes for random mappngs are completely new, although they were obtaned only for a dscrete dstrbuton Extensons of these results to general dstrbutons and to unbounded random sets and mappngs are stll open problems 94

2 LAW OF SMALL NUMBERS AS CONCENTRATION INEQUALITIES FOR SUMS OF INDEPENDENT RANDOM Concentraton nequaltes for sums of ndependent random varables and ther extensons Classcal concentraton nequaltes for random varables are due to Bernsten, Chernoff and Hoeffdng (see, eg (Boucheron et al, 005), (Stenwart and Chrstmann, 008)) Next statements extend concentraton nequaltes to random functons Further they are used for dervaton of concentraton nequaltes for random vectors and sets 1 Theorem (see, eg, (Boucheron et al, 005)) Let ( Y, d ) be a separable metrc space wth metrc d, ( Ξ, ΣP, ) be a probablty space and F := { f : Y Ξ R } be a set of functons f : Y Ξ R contnuous n y Y for all Ξ, Σ -measurable n and E = 0 for all y Y y Furthermore, let b 0 be a constant such that y y f b for all y Y varables,, 1, we defne a random varable G : Z R by G t > 0 and all γ > 0, we have { } P G > E G + t b e t f y For ndependent random : = sup f y( ) Then, for all The followng theorem presents a refnement of the (Talagrand s, 1994, 1996) nequalty Theorem (Talagrand s nequalty n d case, see (Stenwart and Chrstmann, 008)) Let b 0, σ 0, and n 1 Moreover, let ( Ξ, Σ, µ ) be a probablty space and F be a countable set of Σ -measurable functons such that E f = 0, µ µ f σ y Y =1 E and f b for all f F We wrte Z: = Ξ and P : =µ, E denotes expectaton over P Furthermore, for ndependent random varables,, 1 we defne a random varable Gn: Z R by G have lemmas : = sup f ( ) Then, for all t > 0, we f F =1 { σ } P G EG + t( + be G ) + tb 3 e To use Theorems 1, one needs estmates of the quantty G, whch are gven n the followng 3 Lemma ( E G n case of fnte Ξ ) Suppose the probablty space s dscrete, Ξ contans N Ξ elements, and sup f ( ) M f F, Ξ F, Ξ Then E G MF, Ξ NΞ 4 Lemma ( E G n case of fnte F, (Boucheron et al, 005, Theorem 33)) Suppose the famly F fs fnte and bounded, e t contans F number of elements and sup f F, Ξ f ( ) M Then G M ln F 3 Concentraton nequaltes for sums of random vectors n a separable Hlbert space In the present secton we derve concentraton nequaltes for sums S = =1 of ndependent bounded random vectors wth values n a separable nfnte dmenson Hlbert space H In the next secton these results wll be used to derve concentraton nequaltes for random sets Smlar results for fnte dmensonal regular Banach spaces (obtaned n a dfferent way) can be found n (Nemrovsk, 004) 95

3 V I Norkn, R J-B Wets Concentraton nequaltes for sums S = of ndependent random vectors =1 can be obtaned from such nequaltes (Theorems 1, ) for random functons, snce for any s H holds s = sup f s f H: f 1 31 Theorem Let,, 1 be ndependent random varables (n a separable Hlbert space) such that E b, E E σ for all, then for any t > 0 holds and (for d case) 1 σ tb P S ES > + e, ( σ + σ ) 1 t b σ tb P S ES + + e 3 As an llustraton we apply Theorem 31 to derve concentraton nequaltes for frequences n the law of large numbers The next theorem establshes rate of convergence (of order 1 / ) and the exponental decay of the probabltes of large devatons for frequences n the law of large numbers wth countable realzatons Later t s used for establshng concentraton nequaltes for random mappngs 3 Theorem Let { 1,,} be a sequence of ndependent random varables takng values from a fnte or countable collecton { 1,,} wth probabltes { p1, p,} = p respectvely Denote λ = { λ1,, λ,} a random vector of frequences such that λ s the quotent of { } =1 httng the value (If { } =1 does not ht the value then λ = 0 ) Then for any and t > 0 λ p = ( λ p ) where ( ) 1/ 4 P{ λ p > 1 + t} e, t P λ p > 1+ t e, 3 4 Concentraton nequaltes for bounded random sets Concentraton nequaltes from Theorem 31 can be extended to random sets The frst result of ths type for bounded m -dmensonal random sets was obtaned by (Artsten, 1984) The estmate of the probablty of large devatons of Mnkowsk's averages from ts expectaton n (Artsten, 1984) depends 1 on the sze of the devaton t through a multpler 1 / t m standng before the exponental term, so t makes sense only for a fnte dmensonal case An asymptotc large devaton result for random sets n a separable Banach space was presented n (Cerf, 1999) X, e Borel-measurable vector functons from some probablty space Consder random sets { } ( Ξ, ΣP, ) to the space of non-empty closed subsets of a separable Hlbert space H wth norm = H A random vector : Ξ H s called a selecton of X f wth probablty one holds X Defne the expectaton E X of X to be { + } EX = E : s a selecton of X ande < H 96

4 LAW OF SMALL NUMBERS AS CONCENTRATION INEQUALITIES FOR SUMS OF INDEPENDENT RANDOM Defne the devaton of a set A H from a set B H as ( A, B) = sup nf a b and Hausdorff dstance between sets A and B as A B { A B B A } a A b B H H (, ) = max (, ), (, ) In partcular for a H denote dst( a, B) = ( a, B) Denote coa a convex hull of A H, Defne a sample average set 1 1 S = X = s = : s a selecton of X = 1 = 1 A = sup x A x In the next theorems we extend and sharpen fnte dmensonal results by (Artsten, 1984) to a separable nfnte dmensonal Hlbert space 41 Theorem Let X,, X 1 be ndependent closed valued random sets wth X = E cox, H ( cox, X ) b < + wth probablty one, EH ( cox, X ) σ for all Denote Then for any t > 0 we have and for d { X } 1 1 D = H cox, X =1 =1 tb P D E D + e ( σ + σ ) t b tb P D E D + + e 3 If { X } take on values from a fnte collecton of sze N X, then the term can be replaced by b N m In a fnte dmensonal case, { } ( ) b m 1+ ln X ; E D n the nequaltes X R, the term E D n the nequaltes can be replaced by Theorem 41 states a concentraton result for convex valued random sets { cox } In a fnte m dmensonal case, X R, the result can be extended by means of Shapley-Folkman lemma to nonconvex valued mappngs smlar to (Arsten, 1984) In an nfnte dmensonal case ths s not possble In 1 the next theorem we obtan the desred result for one-sded dstance X, =1 E cox1 4 Theorem Let X,, X 1 be d closed valued random sets n a Hlbert space H, X b < + wth probablty one, E X σ, and X= E cox Then for any t > 0 and we have 1 σ tb P X, X + e, =1 ( + ) 1 t σ σ b σ tb P X, X e + + =1 3 97

5 V I Norkn, R J-B Wets 5 Concentraton nequaltes for random sets and mappngs n case of a dscrete dstrbuton In ths secton we establsh concentraton nequaltes for bounded random set valued mappngs n the case when the random mappng has a dscrete dstrbuton If the mappng s constant then these nequaltes become concentraton nequaltes for random sets In a smlar settng (Cresse, 1979) obtans a central lmt theorem for random sets wth fnte dscrete dstrbuton Unlkely, our results are nonasymptotc, vald for countable dscrete dstrbutons and are extended to random mappngs Assume that () s a dscrete random varable takng at most countable number of values { 1,,} wth probabltes { p1, p,} = p ; () { S1( x), S( x ),} are mappngs defned on a set space; 98 n X R wth convex values n a Banach () mappngs { S } are unformly bounded on X, S( X ) = sup sup x X S ( x) < + Defne a random mappng S(, x) : = S( x) f =, and let {, = 1,,} are d realzatons of 1 the random varable, sample average mappng ( S, x ) = S (, x ) and the expectaton mappng S x p S x { s s p s S x } E (, ) = ( ) = = : ( ) Denote S( x) = ( S1( x), S( x ),), S ( X ) = supx X S ( x) < +, ( ) 1/ =1 S( X ) = S < + 51 Theorem In condtons ()-() for any t > 0 and the followng estmates hold true for the unform dstance ( S, S) = sup x X ( S (, x), S( U E H E, x)) and graph dstance H( gphs, gphe S) between graphs of mappngs S and S, { ( gphs, gph S) > (1 + t) S( X ) } P{ U S ES + t S X } ( ) P H E (, ) > (1 ) ( ) exp 4 These estmates have a non-asymptotc character, do not depend on the dstrbuton p and contan explct constants If appled to random sets wth dscrete dstrbuton, ths theorem gves a sharper result than Theorem 41 6 Conclusons We have obtaned a number of new explct concentraton results for vectors (Theorem 31), random sets (Theorems 41, 4) and random set valued mappngs (Theorem 51) n nfnte dmensonal spaces In a sense, concentraton phenomenon reflects the law of small numbers, descrbng non-asymptotc behavor of sample averages One nterestng consequence of Theorem 31 s a concentraton result for frequences n the law of large numbers wth countable number of realzatons (Theorem 3) One more nterestng consequence (of Theorem 51) s that n case of a dscrete dstrbuton a unform law of large numbers for random mappngs, e lm U( S, E S) = 0, holds wth probablty one wthout any semcontnuty assumptons on the nvolved (convex valued unformly bounded) mappngs { S( x )} (compare to (Shapro and Xu, 007)) All these results contrbute to the substantaton of the Monte Carlo method n nfnte dmensonal spaces Extensons of these results to general dstrbutons and to unbounded random sets and mappngs are stll open problems Aknowledgements Vladmr Norkn s work on the paper was supported by a Fulbrght scholarshp whle stayng at the Department of Mathematcs of the Unversty of Calforna, Davs (October 010 September 011) and

6 LAW OF SMALL NUMBERS AS CONCENTRATION INEQUALITIES FOR SUMS OF INDEPENDENT RANDOM by a grant of the Ukranan State Fund for Fundamental Research Ф401/016 Roger Wets' research was supported n part by a grant of the Army Research Offce (ARO) W911NS References Artsten, Z (1984) Convergence of sums of random sets, n Ambertzuman, R and Wel, W (Ed), Stochastc Geometry, Geometrc Statstcs and Stereology, Vol 65 of Teubner-Texte zur Math, Teubner Verlag, Lepzg, pp Boucheron, S; Bousquet, O and G Lugos (005) Theory of classfcaton: A survey of some recent advances, ESAIM: Probablty and Statstcs 9: Cerf, R (1999) Large devatons for sums of d random compact sets, Proceedngs of the Amercan Mathematcal Socety, Vol 17, pp Cresse, N (1979) A central lmt theorem for random sets, Zetschrft für Wahrschenlchketstheore und Verwandte Gebete 49: Ledoux, M (001) The Concentraton of Measure Phenomenon, Vol 89 of Mathematcal surveys and monographs, Amercan Mathematcal Socety, Provdence Nemrovsk, A S (004) Regular Banach spaces and large devatons of random sums, Avalable Pflug, G (003) Stochastc optmzaton and statstcal nference, n Ruszczyńsk, A and Shapro, A (Ed), Stochastc Programmng, Vol 10 of Handbooks n Operatons Research and Management Scence, Elsever, pp Shapro, A; Dentcheva, D and Ruszczyńsk, A (009) Lectures on Stochastc Programmng: Modelng and Theory, SIAM, Phladelpha Shapro, A and Xu, H (007) Unform laws of large numbers for set-valued mappngs and subdfferentals of random functons, Journal of Mathematcal Analyss and Applcatons 35: Stenwart, I and Chrstmann, A (008) Support Vector Machnes, Sprnger, New York Talagrand, M (1994) Sharper bounds for gaussan and emprcal processes, The Annals of Probablty : Talagrand, M (1996) New concentraton nequaltes n product spaces, Inventones Mathematcae 16: