NONPARAMETRIC LEAST SQUARES ESTIMATION OF A MULTIVARIATE CONVEX REGRESSION FUNCTION

Transcription

1 NONPARAMETRIC LEAST SQUARES ESTIMATION OF A MULTIVARIATE CONVEX REGRESSION FUNCTION Emilio Seijo an Bohisattva Sen Columbia University arxiv: v2 [math.st] 0 Jul 200 July 0, 200 Abstract This paper eals with the consistency of the least squares estimator of a convex regression function when the preictor is multiimensional. We characterize an iscuss the computation of such an estimator via the solution of certain quaratic an linear programs. Mil sufficient conitions for the consistency of this estimator an its subifferentials in fixe an stochastic esign regression settings are provie. We also consier a regression function which is known to be convex an componentwise nonincreasing an iscuss the characterization, computation an consistency of its least squares estimator. Introuction Consier a close, convex set X R, for, with nonempty interior an a regression moel of the form Y = φx + ɛ where X is a X-value ranom vector, ɛ is a ranom variable with Eɛ X = 0, an φ : R R is an unknown convex function. Given inepenent observations X,Y,...,X n,y n from such a moel, we wish to estimate φ by the metho of least squares, i.e., by fining a convex function ˆφ n which minimizes the iscrete L 2 norm Yk ψx k 2 2 among all convex functions ψ efine on the convex hull of X,..., X n. In this paper we characterize the least squares estimator, provie means for its computation, stuy its finite sample properties an prove its consistency. The problem just escribe is a nonparametric regression problem with known shape restriction convexity. Such problems have a long history in the statistical literature with seminal papers like Brunk 955, Grenaner 956 an Hilreth 954 written more than 50 years ago, albeit in simpler settings. The former two papers eal with the estimation of monotone functions while the latter iscusses least squares estimation of a concave function whose omain is a subset of the real line. Since then, many results on ifferent nonparametric shape restricte regression problems have been publishe. For instance, Brunk 970 an, more recently, Zhang 2002 have enriche the literature concerning isotonic

2 regression. In the particular case of convex regression, Hanson an Pleger 976 prove the consistency of the least squares estimator introuce in Hilreth 954. Some years later, Mammen 99 an Groeneboom et al. 200 erive, respectively, the rate of convergence an asymptotic istribution of this estimator. Some alternative methos of estimation that combine shape restrictions with smoothness assumptions have also been propose for the one-imensional case; see, for example, Birke an Dette 2006 where a kernel-base estimator is efine an its asymptotic istribution erive. Although the asymptotic theory of the one-imensional convex regression problem is well unerstoo, not much has been one in the multiimensional scenario. The absence of a natural orer structure in R, for >, poses a natural impeiment in such extensions. A convex function on the real line can be characterize as an absolutely continuous function with increasing first erivative see, for instance, Follan 999, Exercise 42.b, page 09. This characterization plays a key role in the computation an asymptotic theory of the least squares estimator in the one-imensional case. By contrast, analogous results for convex functions of several variables involve more complicate characterizations using either secon-orer conitions as in Duley 977, Theorem 3., page 63 or cyclical monotonicity as in Rockafellar 970, Theorems 24.8 an 24.9, pages Interesting ifferences between convex functions on R an convex functions on R are given in Johansen 974 an Bron steĭn 978. Recently there has been consierable interest in shape restricte function estimation in multiimension. In the ensity estimation context, Cule et al eal with the computation of the nonparametric maximum likelihoo estimator of a multiimensional log-concave ensity, while Cule an Samworth 2009, Schuhmacher et al an Schuhmacher an Dümbgen 200 iscuss its consistency an relate issues. Seregin an Wellner 2009 stuy the computation an consistency of the maximum likelihoo estimator of convex-transforme ensities. This paper focuses on estimating a regression function which is known to be convex. To the best of our knowlege this is the first attempt to systematically stuy the characterization, computation, an consistency of the least squares estimator of a convex regression function with multiimensional covariates in a completely nonparametric setting. In the fiel of econometrics some work has been one on this multiimensional problem in less general contexts an with more stringent assumptions. Estimation of concave an/or componentwise nonecreasing functions has been treate, for instance, in Banker an Mainiratta 992, Matzkin 99, Matzkin 993, Beresteanu 2007 an Allon et al The first two papers efine maximum likelihoo estimators in semiparametric settings. The estimators in Matzkin 99 an Banker an Mainiratta 992 are shown to be consistent in Matzkin 99 an Mainiratta an Sarath 997, respectively. A maximum likelihoo estimator an a sieve least squares estimator have been efine an techniques for their computation have been provie in Allon et al an Beresteanu 2007, respectively. The metho of least squares has been applie to multiimensional concave regression in Kuosmanen We take this work as our starting point. In agreement with the techniques use there, we efine a least squares estimator which can be compute by solving a quaratic program. We argue that this estimator can be evaluate at a single point by fining the solution to a linear program. We then show that, uner some mil regularity conitions, our estimator can be use to consistently estimate both, the convex function an its subifferentials. Our work goes beyon those mentione above in the following ways: Our metho oes not require any tuning parameters, which is a major rawback for most nonparametric regression methos, such as kernel-base proceures. The choice of the tuning parameters is especially problematic in higher imensions, e.g., kernel base methos woul require the choice of a matrix of banwiths. The sets of assumptions that most authors have use to stuy the estimation of a multiimensional convex regression function are more restrictive an of a ifferent nature than the ones in this paper. As oppose to the maximum likelihoo approach use in Banker an Mainiratta 992, Matzkin 99, Allon et al. 2

3 2007 an Mainiratta an Sarath 997, we prove the consistency of the estimator keeping the istribution of the errors unspecifie; e.g., in the i.i.. case we only assume that the errors have zero expectation an finite secon moment. The estimators in Beresteanu 2007 are sieve least squares estimators an assume that the observe values of the preictors lie on equiistant gris of rectangular omains. By contrast, our estimators are unsieve an our assumptions on the spatial arrangement of the preictor values are much more relaxe. In fact, we prove the consistency of the least squares estimator uner both fixe an stochastic esign settings; we also allow for heterosceastic errors. In aition, we show that the least squares estimator can also be use to approximate the graients an subifferentials of the unerlying convex function. It is har to overstate the importance of convex functions in applie mathematics. For instance, optimization problems with convex objective functions over convex sets appear in many applications. Thus, the question of accurately estimating a convex regression function is inee interesting from a theoretical perspective. However, it turns out that convex regression is important for numerous reasons besies statistical curiosity. Convexity also appears in many applie sciences. One such fiel of application is microeconomic theory. Prouction functions are often suppose to be concave an componentwise nonecreasing. In this context, concavity reflects ecreasing marginal returns. Concavity also plays a role in the theory of rational choice since it is a common assumption for utility functions, on which it represents ecreasing marginal utility. The intereste reaer can see Hilreth 954, Varian 982a or Varian 982b for more information regaring the importance of concavity/convexity in economic theory. The paper is organize as follows. In Section 2 we iscuss the estimation proceure, characterize the estimator an show how it can be compute by solving a positive semiefinite quaratic program an a linear program. Section 3 starts with a escription of the eterministic an stochastic esign regression schemes. The statement an proof of our main results are also inclue in Section 3. In Section 4 we provie the proofs of the technical lemmas use to prove the main theorem. Section A, the Appenix, contains some results from convex analysis an linear algebra that are use in the paper an may be of inepenent interest. 2 Characterization an finite sample properties We start with some notation. For convenience, we will regar elements of the Eucliian space R m as column vectors an enote their components with upper inices, i.e, any z R m will be enote as z = z, z 2,..., z m. The symbol R will stan for the extene real line. Aitionally, for any set A R we will enote as Conv A its convex hull an we ll write Conv X,..., X n instea of Conv {X,..., X n }. Finally, we will use, an to enote the stanar inner prouct an norm in Eucliian spaces, respectively. For X = {X,..., X n } X R, consier the set K X of all vectors z = z,..., z n R n for which there is a convex function ψ : X R such that ψx j = z j for all j =,...,n. Then, a necessary an sufficient conition for a convex function ψ to minimize the sum of square errors is that ψx j = Z j n for j =,...,n, where { } Z n = argmin Y k z k 2. 2 z K X The computation of the vector Z n is crucial for the estimation proceure. We will show that such a vector exists an is unique. However, it shoul be note that there are many convex functions ψ satisfying ψx j = Z j n for all j =,...,n. Although any of these functions can play the role of the least squares 3

4 estimator, there is one such function which is easily evaluate in Conv X,..., X n. For computational convenience, we will efine our least squares estimator ˆφ n to be precisely this function an escribe it explicitly in 7 an the subsequent iscussion. In what follows we show that both, the vector Z n an the least squares estimator ˆφ n are well-efine for any n ata points X,Y,...,X n,y n. We will also provie two characterizations of the set K X an show that the vector Z n can be compute by solving a positive semiefinite quaratic program. Finally, we will prove that for any x Conv X,..., X n one can obtain ˆφ n x by solving a linear program. 2. Existence an uniqueness We start with two characterizations of the set K X. The evelopments here are similar to those in Allon et al an Kuosmanen Lemma 2. Primal Characterization Let z = z,..., z n R n. Then, z K X if an only if for every j =,...,n, the following hols: { z j = inf θ k z k : θ k =, θ k X k = X j, θ 0, θ R }, n 3 where the inequality θ 0 hols componentwise. Proof: Define the function g : R R by { } g x = inf θ k z k : θ k =, θ k X k = x, θ 0, θ R n 4 where we use the convention that inf = +. By Lemma A. in the Appenix, g is convex an finite on the X j s. Hence, if z j satisfies 3 then z j = g X j for every j =,...,n an it follows that z K X. Conversely, assume that z K X an g X j z j for some j. Note that g X k z k for any k from the efinition of g. Thus, we may suppose that g X j < z j. As z K X, there is a convex function ψ such that ψx k = z k for all k =,...,n. Then, from the efinition of g X j there exist θ 0 R n with θ 0 0 an θ θn 0 = such that θ 0 X θ n 0 X n = X j an θ0 k ψx k = θ0 k zk < z j = ψx j = ψ θ0 k X k, which leas to a contraiction because ψ is convex. We now provie an alternative characterization of the set K X base on the ual problem to the linear program use in Lemma 2.. Lemma 2.2 Dual Characterization Let z R n. Then, z K X if an only if for any j =,...,n we have { } z j = sup ξ, X j + η : ξ, X k + η z k k =,...,n, ξ R, η R. 5 Moreover, z K X if an only if there exist vectors ξ,...,ξ n R such that ξ j, X k X j z k z j k, j {,...,n}. 6 4

5 Proof: Accoring to the primal characterization, z K X if an only if the linear programs efine by 3 have the z j s as optimal values. The linear programs in 5 are the ual problems to those in 3. Then, the uality theorem for linear programs see Luenberger 984, page 89 implies that the z j s have to be the corresponing optimal values to the programs in 5. To prove the secon assertion let us first assume that z K X. For each j {,...,n} take any solution ξ j,η j to 5. Then by 5, η j = z j ξ j, X j an the inequalities in 6 follow immeiately because we must have ξ j, X k + η j z k for any k {,...,n}. Conversely, take z R n an assume that there are ξ,...,ξ n R satisfying 6. Take any j {,...,n}, η j = z j ξ j, X j an θ to be the vector in R n with components θ k = δ k j, where δ k j is the Kronecker δ. It follows that ξ j, X k + η j z k k =,...,n so ξ j,η j is feasible for the linear program in 5. In aition, θ is feasible for the linear program in 3 so the weak uality principle of linear programming see Luenberger 984, Lemma, page 89 implies that ξ, X j + η z j for any pair ξ,η which is feasible for the problem in the right-han sie of 5. We thus have that z j is an upper boun attaine by the feasible pair ξ j,η j an hence 5 hols for all j =,...,n. Both, the primal an ual characterizations are useful for our purposes. The primal plays a key role in proving the existence an uniqueness of the least squares estimator. The ual is crucial for its computation. Lemma 2.3 The set K X is a close, convex cone in R n an the vector Z n satisfying 2 is uniquely efine. Proof: That K X is a convex cone follows trivially from the efinition of the set. Now, if z K X, then there is j {,...,n} for which z j > g X j with the function g efine as in 4. Thus, there is θ 0 R n with θ 0 0 an θ θn 0 = such that θ 0 X θ0 n X n = X j an n θk 0 zk < z j. Setting δ = 2 z j n θk 0 zk it is easily seen that for all ζ n zk δ, z k + δ we still have n θk 0 ζk < ζ j an thus ζ K X. Therefore we have shown that for any z K X there is a neighborhoo U of z with U R n \ K X. Therefore, K X is close an the vector Z n is uniquely etermine as the projection of Y,...,Y n R n onto the close convex set K X see Conway 985, Theorem 2.5, page 9. We are now in a position to efine the least squares estimator. Given observations X,Y,...,X n,y n from moel, we take the nonparametric least squares estimator to be the function ˆφ n : R R efine by { } ˆφ n x = inf θ k Zn k : θ k =, θ k X k = x, θ 0, θ R n 7 for any x R. Here we are taking the convention that inf = +. This function is well-efine because the vector Z n exists an is unique for the sample. The estimator is, in fact, a polyheral convex function i.e., a convex function whose epigraph is a polyheral; see Rockafellar 970, page 72 an satisfies, as a consequence of Lemma A., ˆφ n x = sup {ψx}, ψ K X,Zn where K X,Zn is the collection of all convex functions ψ : R R such that ψx j Z j n for all j =,...,n. Thus, ˆφ n is the largest convex function that never excees the Z j n s. It is immeiate that ˆφ n is inee a convex function as the supremum of any family of convex functions is itself convex. The primal characterization of the set K X implies that ˆφ n X j = Z j n for all j =,...,n. 5

6 2.2 Finite sample properties In the following lemma we state some of the most important finite sample properties of the least squares estimator efine by 7. Lemma 2.4 Let ˆφ n be the least squares estimator obtaine from the sample X,Y,...,X n,y n. Then, i ii iii ψx k ˆφ n X k Y k ˆφ n X k 0 for any convex function ψ which is finite on Conv X,..., X n ; ˆφ n X k Y k ˆφ n X k = 0; Y k = ˆφ n X k ; iv the set on which ˆφ n < is Conv X,..., X n ; v for any x R the map X,..., X n,y,...,y n ˆφ n x is a Borel-measurable function from R n+ into R. Proof: Property i follows from Moreau s ecomposition theorem, which can be state as: Consier a close convex set C on a Hilbert space H with inner prouct, an norm. Then, for any x H there is only one vector x C C satisfying x x C = argmin ξ C { x ξ }. The vector x C is characterize by being the only element of C for which the inequality ξ x C, x x C 0 hols for every ξ C see Moreau 962 or Song an Zhengjun Taking ψ to be κ ˆφ n an letting κ vary through 0, gives i i from i. Similarly, i i i follows from i by letting ψ to be ˆφ n ±. Property i v is obvious from the efinition of ˆφ n. To see why v hols, we first argue that the map X,..., X n,y,...,y n Z n is measurable. This follows from the fact that Z n is the solution to a convex quaratic program an thus can be foun as a limit of sequences whose elements come from arithmetic operations with X,..., X n, Y,...,Y n. Examples of such sequences are the ones prouce by active set methos, e.g, see Bolan 997; or by interior-point methos see Kapoor an Vaiya 986 or Mehrotra an Sun 990. The measurability of ˆφ n x follows from a similar argument, since it is the optimal value of a linear program whose solution can be obtaine from arithmetic operations involving just X,..., X n,y,...,y n an Z n e.g., via the well-known simplex metho; see Noceal an Wright 999, page 372 or Luenberger 984, page Computation of the estimator Once the vector Z n efine in 2 has been obtaine, the evaluation of ˆφ n at a single point x can be carrie out by solving the linear program in 7. Thus, we nee to fin a way to compute Z n. An here the ual characterization proves of vital importance, since it allows us to compute Z n by solving a quaratic program. Lemma 2.5 Consier the positive semiefinite quaratic program min n Y k z k 2 subject to ξ k, X j X k z j z k k, j =,...,n ξ,...,ξ n R, z R n. 8 6

7 Then, this program has a unique solution Z n in z, i.e., for any two solutions ξ,...,ξ n, z an τ,...,τ n,ζ we have z = ζ = Z n. This solution Z n is the only vector in R n which satisfies 2. Proof: From Lemma 2.2 if ξ,...,ξ n, z belongs in the feasible set of this program, then z K X. Moreover, for any z K X there are ξ,...,ξ n R such that ξ,...,ξ n, z belongs to the feasible set of the quaratic program. Since the objective function only epens on z, solving the quaratic program is the same as getting the element of K X which is the closest to Y. This element is, of course, the uniquely efine Z n satisfying 2. The quaratic program 8 is positive semiefinite. This implies certain computational complexities, but most moern nonlinear programming solvers can hanle this type of optimization problems. Some examples of high-performance quaratic programming solvers are CPLEX, LINDO, MOSEK an QPOPT. Here we present two simulate examples to illustrate the computation of the esti Figure : The scatter plot an nonparametric least squares estimator of the convex regression function when a φx = x 2 left panel; b φx = x + x 2 right panel. mator when = 2. The first one, epicte in Figure a correspons to the case where φx = x 2. Figure b shows the convex function estimator when the regression function is the hyperplane φx = x + x 2. In both cases, n = 256 observations were use an the errors were assume to be i.i.. from the stanar normal istribution. All the computations were carrie out using the MOSEK optimization toolbox for Matlab an the run time for each example was less than 2 minutes in a stanar esktop PC. We refer the reaer to Kuosmanen 2008 for aitional numerical examples although the examples there are for the estimation of concave, componentwise nonecreasing functions, the computational complexities are the same. 2.4 The componentwise nonincreasing case We now consier the case where the regression function φ is assume to be convex an componentwise nonincreasing. The evelopments here are quite similar to those in the convex case, so we omit some of the etails. Given the observe values X,Y,...,X n,y n, we write Q X for the collection of all vectors z R n for which there is a convex, componentwise nonincreasing function ψ satisfying ψx j = z j for 7

8 every j =,...,n. We will enote by R + an R, respectively, the nonnegative an nonpositive orthants of R. We now have the following characterizations. Lemma 2.6 Let z R n. Then, z Q X if an only if the following hols for every j =,...,n: { } z j = inf θ k z k : θ k =, ϑ + θ k X k = X j, θ 0, θ R n,ϑ R +. Proof: The proof is very similar to that of Lemma 2.. The ifference being that we use Lemma A.2 an the function { } hx = inf θ k z k : θ k =, ϑ + θ k X k = x, θ 0, θ R n,ϑ R + instea of using Lemma A. an the function g. The analogous ual characterization here is given in the following lemma. Its proof is just an application of the uality theorem of linear programming, so we omit it. Lemma 2.7 Let z R n. Then, z Q X if an only if for every j =,...,n we have } z j = sup { ξ, X j + η : ξ, X k + η z k k =,...,n, ξ R, η R. Moreover, z Q X if an only if there exist vectors ξ,...,ξ n R such that ξ j, X k X j z k z j k, j {,...,n}. Just as in the previous case, we can use both characterizations to show the existence an uniqueness of the vector { } W n = argmin Y k z k 2 z Q X an then efine the nonparametric least squares estimator by { } ˆϕ n x = inf θ k Wn k : θ k =,ϑ + θ k X k = x,θ R n +,ϑ R +. Here, the vector W n can also be compute by solving the corresponing quaratic program min n Y k z k 2 subject to ξ k, X j X k z j z k k, j =,...,n ξ,...,ξ n R, z Rn. which iffers from the program 8 just because here the ξ j s have to be nonpositive. The estimator enjoys analogous finite imensional properties to those liste in Lemma 2.4. For the sake of completeness, we inclue them in the following lemma. Lemma 2.8 Let ˆϕ n be the convex, componentwise nonincreasing least squares estimator obtaine from the sample X,Y,...,X n,y n. Then, 8

9 i ii iii ψx k ˆϕ n X k Y k ˆϕ n X k 0 for any convex, componentwise nonincreasing function ψ which is finite on Conv X,..., X n ; ˆϕ n X k Y k ˆϕ n X k = 0; Y k = ˆϕ n X k ; iv the set on which ˆϕ n < is Conv X,..., X n + R + ; v for any x R the map X,..., X n,y,...,y n ˆϕ n x is a Borel-measurable function from R n+ into R. 3 Consistency of the least squares estimator The main goal of this paper is to show that in an appropriate setting the nonparametric least squares estimator ˆφ n escribe above is consistent for estimating the convex function φ on the set X. In this context, we will prove the consistency of ˆφ n in both, fixe an stochastic esign regression settings. Before proceeing any further we woul like to introuce some notation. For any Borel set X R we will enote by B X the σ-algebra of Borel subsets of X. Given a sequence of events A n n= we will be using the notation [A n i.o.] an [A n a.a.] to enote lim A n an lim A n, respectively. Now, consier a convex function f : R R. This function is sai to be proper if f x > for every x R. The effective omain of f, enote by Domf, is the set of points x R for which f x <. The subifferential of f at a point x R is the set f x R of all vectors ξ satisfying the inequality ξ,h f x + h f x h R. The elements of f x are calle subgraients of f at x see Rockafellar 970. For a set A R we enote by A, A an A its interior, closure an bounary, respectively. We write ExtA = R \ A for the exterior of the set A an iama := sup x,y A x y for the iameter of A. We also use the sup-norm notation, i.e., for a function g : R R we write g A = sup x A g x. To avoi measurability issues regaring some sets, specially those involving the ranom set-value functions { ˆφ n x} x X, we will use the symbols P an P to enote inner an outer probabilities, respectively. We refer the reaer to Van er Vaart an Wellner 996, pages 6-5, for the basic properties of inner an outer probabilities. In this context, a sequence of not necessarily measurable functions Ψ n n= from a probability space Ω,F, P into R is sai to converge to a function Ψ almost surely see a.s. Van er Vaart an Wellner 996, Definition.9.-iv, page 52, written Ψ n Ψ, if P Ψ n Ψ =. We will use the stanar notation PA for the probabilities of all events A whose measurability can be easily inferre from the measurability of the ranom variables { ˆφ n x} x X, establishe in Lemma 2.4. Our main theorems hol for both, fixe an stochastic esign schemes, an the proofs are very similar. They iffer only in minor steps. Therefore, for the sake of simplicity, we will enote the observe values of the regressor variables always with the capital letters X n. For any Borel set X R, we write N n X = #{ j n : X j X}. The quantities X n an N n X are non-ranom uner the fixe esign but ranom uner the stochastic one. 9

10 3. Fixe Design In a fixe esign regression setting we assume that the regressor values are non-ranom an that all the uncertainty in the moel comes from the response variable. We will now list a set of assumptions for this type of esign. The one-imensional case has been proven, uner ifferent regularity conitions, in Hanson an Pleger 976. A We assume that we have a sequence X n,y n n= satisfying Y k = φx k + ɛ k where ɛ n n= is an i.i.. sequence with E ɛ j = 0, E ɛ 2 = σ 2 < an φ : R R is a proper convex j function. A2 The non-ranom sequence X n n= is containe in a close, convex set X R with X an X Domφ. A3 We assume the existence of a Borel measure ν on X satisfying: i {X B X : νx = 0} = {X B X : X has Lebesgue measure 0}. ii n N nx νx for any Borel set X X. Conition A may be replace by the following: A4 We assume that we have a sequence X n,y n n= satisfying Y k = φx k + ɛ k where φ : R R is a proper convex function an ɛ n n= is an inepenent sequence of ranom variables satisfying i Eɛ n = 0 n N an lim n n E ɛ k > 0. ii n= Varɛ 2 n n 2 <. iii sup n N {E ɛ 2 n } <. Uner these conitions we efine σ 2 := lim n n n j = E ɛ 2. j The raison etre of conition A4 is to allow the variance of the error terms to epen on the regressors. We make the istinction between A an A4 because in the case of i.i.. errors it is enough to require a finite secon moment to ensure consistency. 3.2 Stochastic Design In this setting we assume that X n,y n n= is an i.i.. sequence from some Borel probability measure µ on R +. Here we make the following assumptions on the measure µ: A5 There is a close, convex set X R with X such that µx R =. Also, y 2 µ x, y <. X R 0

11 A6 There is a proper convex function φ : R R with X Domφ such that whenever X,Y µ we have E Y φx X = 0 an E Y φx 2 = σ 2 <. Thus, φ is the regression function. A7 Denoting by ν = µ R the x-marginal of µ, we assume that {X B X : νx = 0} = {X B X : X has Lebesgue measure 0}. We wish to point out some conclusions that one can raw from these assumptions. Consier the class of functions { } K µ := ψ : R R ψ is convex with ψx 2 ν x <. Then for any X X the following hols ψxy φxµ x, y = 0 ψ K µ ; X R so we get that φ is in fact the element of K µ which is the closest to Y in the Hilbert space L 2 X R,B X R,µ. This follows from Moreau s ecomposition theorem see the proof of Lemma 2.4. Aitionally, conitions {A5-A7} allow for stochastic epenency between the error variable Y φx an the regressor X. Although some level of epenency can be put to satisfy conitions {A2-A4}, the measure µ allows us to take into account some cases which wouln t fit in the fixe esign setting even by conitioning on the regressors. 3.3 Main results We can now state the two main results of this paper. The first result shows that assuming only the convexity of φ, the least squares estimator can be use to consistently estimate both φ an its subifferentials φx. Theorem 3. Uner any of {A-A3}, {A2-A4} or {A5-A7} we have, i P sup{ ˆφ n x φx } 0 for any compact set X X =. x X ii For every x X an every ξ R lim lim ˆφ n x + hξ ˆφ n x φx + hξ φx lim almost surely. n h 0 h h 0 h iii Denoting by B the unit ball w.r.t. the Eucliian norm we have P ˆφ n x φx + ɛb a.a. = ɛ > 0, x X. iv If φ is ifferentiable at x X, then sup { ξ φx } a.s. 0. ξ ˆφ n x

12 Our secon result states that assuming ifferentiability of φ on the entire X allows us to use the subifferentials of the least squares estimator to consistently estimate φ uniformly on compact subsets of X. Theorem 3.2 If φ is ifferentiable on X, then uner any of {A-A3}, {A2-A4} or {A5-A7} we have, P sup x X ξ ˆφ n x 3.4 Proof of the main results { ξ φx } 0 for any compact set X X =. Before embarking on the proofs, one must notice that there are some statements which hol true uner any of {A-A3}, {A2-A4} or {A5-A7}. We list the most important ones below, since they ll be use later. For any set X X we have N n X n a.s. νx. 9 The strong law of large numbers implies that for any Borel set X X with positive Lebesgue measure we have N n X X k X k n Y k φx k a.s. 0 0 an also lim Y k φx k 2 = σ 2 a.s. n n k n We woul like to point out that in the case of conition A4, A4-iii allows us to obtain 0 from an application of a version of the strong law of large number for uncorrelate ranom variables, as it appears in Chung 200, page 08, Theorem Similarly, conition A4-ii implies that we can apply a version the strong law of large numbers for inepenent ranom variables as in Williams 99, Lemma 2.8, page 8 or in Follan 999, Theorem 0.2, page 322 to obtain. For any Borel subset X X with positive Lebesgue measure, #{n N : X n X} a.s. + 2 Proof of Theorem 3.. We will only make istinctions among the esign schemes in the proof if we are using any property besies 9, 0, or 2. For the sake of clarity, we ivie the proof in steps. Step I: We start by showing that for any set with positive Lebesgue measure there is a uniform ban aroun the regression function over that set such that ˆφ n comes within the ban at least at one point for all but finitely many n s. This fact is state in the following lemma prove in Section 4.. 2

13 Lemma 3. For any set X X with positive Lebesgue measure we have, P inf x X { ˆφ n x φx } M i.o. = 0 M > σ. νx Step II: The iea is now to use the convexity of both, φ an ˆφ n, to show that the previous result in fact implies that the sup-norm of ˆφ n is uniformly boune on compact subsets of X. We achieve this goal in the following two lemmas whose proofs are given in Sections 4.2 an 4.3 respectively. Lemma 3.2 Let X X be compact with positive Lebesgue measure. Then, there is a positive real number K X such that P inf x X { ˆφ n x} < K X i.o. = 0. Lemma 3.3 Let X X be a compact set with positive Lebesgue measure. Then, there is K X > 0 such that P sup{ ˆφ n x} K X i.o. x X = 0. Step III: Convex functions are etermine by their subifferential mappings see Rockafellar 970, Theorem 24.9, page 239. Moreover, having a uniform upper boun K X for the norms of all the subgraients over a compact region X imposes a Lipschitz continuity conition on the convex function over X see Rockafellar 970, Theorem 24.7, page 237; the Lipschitz constant being K X. For these reasons, it is important to have a uniform upper boun on the norms of the subgraients of ˆφ n on compact regions. The following lemma prove in Section 4.4 states that this can be achieve. Lemma 3.4 Let X X be a compact set with positive Lebesgue measure. Then, there is K X > 0 such that P sup ξ ˆφ n x x X { ξ } > K X i.o. = 0. Step IV: For the next results we nee to introuce some further notation. We will enote by µ n the empirical measure efine on R + by the sample X,Y,...,X n,y n. In agreement with Van er Vaart an Wellner 996, given a class of functions G on D R +, a seminorm on some space containing G an ɛ > 0 we enote by N ɛ,g, the ɛ covering number of G with respect to. Although Lemmas 3.5 an 3.7 may seem unrelate to what has been one so far, they are crucial for the further evelopments. Lemma 3.5 prove in Section 4.5 shows that the class of convex functions is not very complex in terms of entropy. Lemma 3.7 is a uniform version of the strong law of large numbers which proves vital in the proof of Lemma 3.8. Lemma 3.5 Let X X be a compact rectangle with positive Lebesgue measure. For K > 0 consier the class G K,X of all functions of the form ψx Y φx X X where ψ ranges over the class D K,X of all proper convex functions which satisfy a ψ X K ; b {ξ} [ K,K ]. ξ ψx x X 3

14 Then, for any ɛ > 0 we have lim N ɛ,g K,X,L X R,µ n < almost surely, n an there is a positive constant A ɛ <, epening only on X,..., X n, K an X, such that the covering numbers N ɛ n n j = Y j φx j,g K,X,L X R,µ n are boune above by A ɛ, for all n N, almost surely. The proofs of Lemmas 3.7 an 3.8 given in Sections 4.7 an 4.8 respectively are the only parts in the whole proof where we must treat the ifferent esign schemes separately. To make the argument work, a small lemma prove in Section 4.6 for the set of conitions {A2-A4} is require. We inclue it here for the sake of completeness an to point out the ifference between the schemes. Lemma 3.6 Consier the set of conitions {A2-A4} an a subsequence n k such that lim n k k n k j = E ɛ 2 j = σ 2. Let X m m= be a an increasing sequence of compact subsets of X satisfying νx m. Then, lim lim m k n k { j n k :X j X m } E ɛ 2 j = σ 2. We are now reay to state the key result on the uniform law of large numbers. Lemma 3.7 Consier the notation of Lemma 3.5 an let X X be any finite union of compact rectangles with positive Lebesgue measure. Then, { } a.s. sup ψx ψ D K,X j Y j φx j 0. n { j n:x j X} Step V: With the ai of all the results prove up to this point, it is now possible to show that Lemma 3. is in fact true if we replace M by an arbitrarily small η > 0. The proof of the following lemma is given in Section 4.8. Lemma 3.8 Let X X be any compact set with positive Lebesgue measure. Then, i P inf x X {φx ˆφ n x} η i.o. = 0 η > 0, ii P sup{φx ˆφ n x} η i.o. x X = 0 η > 0. Step VI: Combining the last lemma with the fact that we have a uniform boun on the norms of the subgraients on compacts, we can state an prove the consistency result on compacts. This is one in the next lemma proof inclue in Section 4.9. Lemma 3.9 Let X X be a compact set with positive Lebesgue measure. Then, i P inf x X { ˆφ n x φx} < η i.o. = 0 η > 0, 4

15 ii P sup{ ˆφ n x φx} > η i.o. x X = 0 η > 0, iii sup{ ˆφ n x φx } a.s. 0. x X Step VII: We can now complete the proof of Theorem 3.. Consier the class C of all open rectangles R such that R X an whose vertices have rational coorinates. Then, C is countable an R C R = X. Observe that Lemmas 3.2 an 3.3 imply that for any finite union A := R R m of open rectangles R,...,R m C there is, with probability one, n 0 N such that the sequence ˆφ n n=n 0 is finite on Conv A. From Lemma 3.9 we know that the least squares estimator converges at all rational points in X with probability one. Then, Theorem 0.8, page 90 of Rockafellar 970 implies that i hols if X is replace by the convex hull of a finite union of rectangles belonging to C. Since there are countably many of such unions an any compact subset of X is containe in one of those unions, we see that i hols. An application of Theorem 24.5, page 233 of Rockafellar 970 on an open rectangle C containing x an satisfying C X gives i i an i i i. Note that i v is a consequence of i i i. Proof of Theorem 3.2. To prove the esire result we nee the following lemma whose proof is provie in Section 4.0 from convex analysis. The result is an extension of Theorem 25.7, page 248 of Rockafellar 970, an might be of inepenent interest. Lemma 3.0 Let C R be an open, convex set an f a convex function which is finite an ifferentiable on C. Consier a sequence of convex functions f n n= which are finite on C an such that f n f pointwise on C. Then, if X C is any compact set, sup x X ξ f n x { ξ f x } 0. Defining the class C of open rectangles as in the proof of Theorem 3., one can use a similar argument to obtain Theorem 3.2 from an application of Theorem 3. an the previous lemma. 3.5 The componentwise nonincreasing case The regression function φ is now assume to be convex an componentwise nonincreasing. Recalling the notation efine in Section 2.4, we now have that Theorems 3. an 3.2 still hol with ˆφ n replace by ˆϕ n. In view of the fact that the proof of the results is very similar to that when φ is just convex, we omit the proof an sketch the main ifferences. The proof of the main results in Section 3 relie essentially on two key facts: i The finite sample properties of ˆφ n establishe in Lemma 2.4. ii The vector ˆφ n X,..., ˆφ n X n R n is the L 2 projection of Y,..., Y n on the close, convex cone K X of all evaluations of proper convex functions on X,..., X n. Also, note that φx,...,φx n K X. We know from Lemma 2.8 that ˆϕ n has similar finite sample properties as its convex counterpart. Note that if φ is convex an componentwise nonincreasing φx,..., φx n Q X an ˆϕ n X,..., ˆϕ n X n R n is the L 2 projection of Y,...,Y n onto Q X. From these consierations an the nature of the arguments use to prove Theorems 3. an 3.2, it follows that all but one of those arguments carry forwar to the componentwise nonincreasing case; the 5

16 only ifference being the entropy calculation of Lemma 3.5. At some point in that proof, one breaks the rectangle [ K,K ] into a family of subrectangles in orer to approximate the subifferentials of the class D K,X. It is easily seen that the same argument hols in the componentwise nonincreasing case if one instea uses a partition of [ K,0] to approach the subifferentials of the corresponing class D K,X for componentwise nonincreasing convex functions. By oing this, the resulting function g will be convex an componentwise nonincreasing an 30, 3 an 32 will still hol for the corresponing class H n,ɛ. Then, the conclusions of Lemma 3.5 are also true for the componentwise nonincreasing case an we can conclue that our main results are vali in this case too. 4 Proofs of the lemmas Here we prove the lemmas involve in the proof of the main theorem. To prove these, we will nee aitional auxiliary results from matrix algebra an convex analysis, which may be of inepenent interest an are prove in the Appenix. 4. Proof of Lemma 3. We will first show that the event [ infx X { ˆφ n x φx } M i.o. ] has probability zero. Uner this event, there is a subsequence n k such that inf x X { ˆφ nk x φx } M k N. Then 0 implies that for this subsequence, with probability one, we have lim k {Y j ˆφ nk X j } M. 3 N nk X X j X On the other han, it is seen by solving the corresponing quaratic programming problems; see, e.g., Exercise 6.2, page 484 of Noceal an Wright 999 that for any η > 0, m N { } inf ξ j 2 : ξ j η, ξ R m = η 2, 4 m j m m j m { } inf ξ j 2 : ξ j η, ξ R m = η 2. 5 m m j m j m For 0 < δ < M, using 5 with η = M δ together with 2 an 3 we get that, with probability one, we must have n k lim Y j ˆφ nk X j 2 νxm δ 2. Letting δ 0 we actually get lim k n k j = k n k j = n k Y j ˆφ nk X j 2 νxm 2 > σ 2 = lim n k k n k j = which is impossible because ˆφ nk is the least squares estimator. Therefore, P inf x X { ˆφ n x φx } M i.o. = 0. Y j φx j 2 a.s. 6

17 A similar argument now using 4 gives P sup x X { ˆφ n x φx } M i.o. = 0, which completes the proof of the lemma. Before we prove Lemmas 3.2 an 3.3, we nee some aitional results from matrix algebra. For convenience, we state them here, but postpone their proofs to Section A.2 in the Appenix. We first introuce some notation. We write e j R for the vector whose components are given by e k j = δ j k, where δ j k is the Kronecker δ. We also write e = e e for the vector of ones in R. For α {,} we write { } R α = θ k α k e k : θ 0,θ R for the orthant in the α irection. For any hyperplane H efine by the normal vector ξ R an the intercept b R, we write H = {x R : ξ, x = b}, H + = {x R : ξ, x > b} an H = {x R : ξ, x < b}. For r > 0 an x 0 R we will write Bx 0,r = {x R : x x 0 < r }. We enote by R the space of matrices enowe with the topology efine by the 2 norm where A 2 = sup x { Ax } an can be shown to be equal to the largest singular value of A; see Harville Lemma 4. Let r > 0. There is a constant R r > 0, epening only on r an, such that for any ρ 0,R r there are ρ,ρ > 0 with the property: for any α {,} an any -tuple of vectors β = {x,..., x } R such that x j Bα j r e j,ρ j =,...,, there is a unique pair ξ α,β,b α,β, with ξ α,β R, ξ α,β = an b α,β > 0 for which the following statements hol: i β form a basis for R. ii x,..., x H α,β := {x R : ξ α,β, x = b α,β }. iii min j { ξj } > 0. α,β iv B0,ρ H α,β. v {x R : x ρ } R α H + α,β. vi B α j r e j,ρ {x R : ξ α,β, x < 0} for all j =,...,. ρ vii For any w B 0, 6 3ρ an w 2 B 8 α, ρ 8 we have { } j min X β w + tw 2 w > 0 t j where X β = x,..., x R is the matrix whose j th column is x j. Figure 2a illustrates the above lemma when = 2 an α =,. The lemma states that whatever points x an x 2 are taken insie the circles of raius ρ aroun α r e an α 2 r e 2, respectively, B0,ρ an {x R : x ρ } R α are containe, respectively, in the half-spaces H α,β an H +. Assertion vi i of α,β 7

18 x 2 x x - x -2 Figure 2: Explanatory iagram for a Lemma 4. left panel; b Lemma 4.2 right panel. the lemma implies that all the points in the half line {w +tw 2 w } t shoul have positive co-orinates with respect to the basis β as they o with respect to the basis {α j e j }. We refer the reaer to Section j = A.2. for a complete proof of Lemma 4.. We now state two other useful results, namely Lemma 4.2 an Lemma 4.3, but efer their proofs to Section A.2.2 an Section A.2.3 respectively. Lemma 4.2 Let r > 0 an consier the notation of Lemma 4. with the positive numbers ρ, ρ an ρ as efine there. Take 2 vectors {x ±,..., x ± } R such that x ±j B±r e j,ρ an for α {,} write β α = {x α, x α 2 2,..., x α }, ξ α = ξ α,βα, b α = b α,βα an H α = H α,β, all in agreement with the setting of Lemma 4.. Then, if K = Conv x ±,..., x ± we have: i K = α {,} {x R : ξ α, x b α }. ii K = α {,} {x R : ξ α, x < b α }. iii K = α {,} Conv x α,..., x α. iv K = α {,} {x R : ξ α, x = b α } α {,} {x R : ξ α, x b α }. v B0,ρ K. vi B0,ρ ExtK. Figure 2b illustrates Lemma 4.2 for the two-imensional case. Intuitively, the iea is that as long as the points x ± an x ±2 belong to B±r e,ρ an B±r e 2,ρ, respectively, we will have B0,ρ an B0,ρ as subsets of K an ExtK, respectively. Lemma 4.3 Let [a,b] R be a compact rectangle an r > 0, with r < 2 if 3. For each α {,} write z α = a + +α j j = 2 b j a j e j so that {z α } α {,} is the set of vertices of [a,b]. Then, there is ρ > 0 such that if x α Bz α + r z α z α,ρ α {,}, then [a,b] Conv x α : α {,}. 8

19 Z 2 X -, A 2 α0 =, X, Z -, Z, X* Z -,- Z,- B 2 x X -,- X,- B A X * z 0 Z Figure 3: Explanatory iagram for a Lemma 4.3 left panel; b Lemma 3.2 right panel. Figure 3a escribes Lemma 4.3 in the two-imensional case. As long as the points x ±,± are chosen in the balls of raius ρ aroun z ±,± + r z ±,± z,, Conv x ±,± will contain Conv z±,±. 4.2 Proof of Lemma 3.2 Since any compact subset of X is containe in a finite union of compact rectangles, it is enough to prove the result when X is a compact rectangle [a,b] X. Let r = 4 min k {b k a k } an choose ρ 0, 4 r, ρ > 0 an 0 < ρ < 2 r such that the conclusions of Lemmas 4. an 4.2 hol for any α {,} an any β = z,..., z R with z j Bα j r e j,ρ. Take N N such that N max k {bk a k } < 32 ρ 6 an ivie X into N rectangles all of which are geometrically ientical to N [0,b a]. Let C be any one of the rectangles in the gri an choose any vertex z 0 of C satisfying z 0 = argmax z C { max j { z j a j,b j z j }}. Then, from the efinition of z 0 an r, there is α 0 {,} such that Bz 0,r z 0 + R α0 X. Aitionally, efine B = B z 0, ρ 6 B 2 = B z 0 + 3ρ, ρ 8 α 0, 8 A j = Bz 0 + α j 0 r e j,ρ z 0 + R α0 j =,...,,, A j = Bz 0 α j 0 r e j,ρ j =,...,. 9

20 Observe that all the sets in the previous isplay have positive Lebesgue measure an that the A j s are not necessarily containe in X. Let M = φ X σ, M 0 >, M = M + M 0 an K C > 6M. min{νb,νb 2,νA,...,νA } Also, notice that C B because of 6. We will argue that P inf { ˆφ n x} K C i.o. = 0. 7 x C From Lemma 3., we know that [ { P inf ˆφ n x φx } ] < M 0 a.a. =, 8 x A j j = { so there is, with probability one, n 0 N such that inf x A j ˆφ n x φx } < M 0 for any n n 0 an any j =,...,. Assume that the event [ inf x C { ˆφ n x} < K C i.o. ] is true. Then, there is a subsequence n k such that inf x C { ˆφ nk x} < K C for all k N. Fix any k n 0. We know that there is X C B such that ˆφ nk X K C. In aition, for j =,...,, there are Z j α 0 j A j such that ˆφ nk Z j α 0 j φz α j 0 j < M 0, which in turn implies ˆφ nk Z j α 0 j < M. Pick any Z α j 0 Take any x B 2. We will show the existence of X Conv A j an let K = Conv Z ±,..., Z ± = z0 +Conv Z ± z 0,..., Z ± z 0. Z α 0,..., Z α 0 such that x Conv X, X, as shown in Figure 3b for the case = 2. We will then show that the existence of such an X implies that φx ˆφ nk x > M 0. 9 Consequently, since x is an arbitrary element of B 2 we will have [ ] [ inf { { ˆφ n x} K C i.o. inf ˆφ n x φx } ] < M 0 a.a. x C x A j j = [ ] inf { φx ˆφ nk x } M 0 i.o.. x B 2 But from Lemma 3., the event on the right is a null set. Taking 8 into account, we will see that 7 hols an then complete the argument by taking K X = max C {K C }. To show the existence of X consier the function ψ : R R given by ψt = X + tx X. The function ψ is clearly continuous an satisfies ψ0 = X an ψ = x B 2 K. That B 2 K is a consequence of Lemma 4., i v. The set K is boune, so there is T > such that ψt ExtK = R \ K. The intermeiate value theorem then implies that there is t,t such that X := ψt K. Observe that by Lemma 4.2 i i i we have K = Zα,..., Z α. α {,} Conv Lemma 4. i implies that {Z α 0 z 0,..., Z α 0 z 0} forms a basis of R so we can write X z 0 = j = θ j Z α j 0 j z 0. Moreover, Lemma 4. vi i implies that θ j > 0 for every j =,..., as θ = θ,...,θ = Z α 0 z 0,..., Z α 0 z 0 X z 0. Here we apply Lemma 4. vi i with w = X B, w 2 = x B 2 an t >. For α {,} consier the pair ξ α,b α R R as efine in Lemma 4.2 for the set of vectors {Z ± z 0,..., Z ± z 0 } here we move the origin to z 0. Observe that Lemma 4. i i implies that ξ α0, Z α j 0 j z 0 = 20

21 b α0 for all j =,...,. Consequently, ξ α0, X z 0 = b α0 j = θ j, but since X K, Lemma 4.2 i v implies that ξ α0, X z 0 b α0 an hence j = θ j. Aitionally, for α α 0 we can write ξ α, X z 0 as j = θ j ξ α, Z α j 0 j z 0 = α j =α j 0 θ j b α + α j α j 0 θ j ξ α, Z α j 0 j z 0 < b α 20 as ξ α, Z α j z 0 = b α by Lemma 4. i i an ξ α, Z α j z 0 < 0 by Lemma 4. vi for every j =,...,. Since ξ α, w z 0 = b α for all w Conv Z α,..., Z α an all α {.}, 20 an the fact that X K imply that X Conv Z α 0,..., Z α 0. Hence ˆφ n X j = θ j ˆφ nk Z j α < M. We therefore have j Since X B an we have By using the triangle inequality we get the following bouns ˆφ nk X < M, ˆφ nk X < K C, 2 X + t X X = x. 22 An from Lemma 4. i v an the fact that ξ α0, X = b α0 we also obtain 0 z 0 X < 8 ρ ρ < z 0 x < 2 ρ. 24 z 0 X ρ. 25 From 22 we know that t = X X x X. Using the triangle inequality with 23, 24 an 25 one can fin lower an upper bouns for X X as X X X z 0 z 0 X an x X as x X x z 0 + z 0 X, respectively, to obtain t 7 5. Then, 2 an 22 imply ˆφ nk x ˆφ t nk X + t ˆφ nk X 2 7 K C M < M. Consequently, This proves 9 an completes the proof. φx ˆφ nk x > M M = M Proof of Lemma 3.3 Assume without loss of generality that X is a compact rectangle. Let {z α : α {,} } be the set of vertices of the rectangle. Then, there is r 0, such that Bz α,r X α {,}. Recall that from Lemma 4.3, there is 0 < ρ < 2 r such that for any {η α : α {,} } if η α Bz α + r 2 z α z α,ρ then X Conv η α : α {,}. Let A α = Bz α + 2 r z α z α, ρ 2 an M σ 0 > an choose min{νa α :α {,} } M = x Conv sup { φx }. α {,} A α 2

22 Take K X > M 0 + M. Since P α {,} [ ] inf { ˆφ n x φx } < M 0, a.a. = x A α by Lemma 3., there is, with probability one, n 0 N such that for any n n 0 we can fin η α A α, α {,}, such that ˆφ n η α φη α < M 0. It follows that ˆφ n η α K X α {,}. Now, using Lemma 4.3 we have X Conv η α : α {,} an the convexity of ˆφ n implies that ˆφ n x K X for any x X. 4.4 Proof of Lemma 3.4 Assume that X = [a,b] is a rectangle with vertices {z α : α {,} }. The function ψx = inf η ExtX { x η } is continuous on R so there is x X such that ψx = inf x X {ψx}. Observe that ψx > 0 because x X X. By Lemma 4.3, there is a r < 2 ψx for which there exists ρ < 4 r such that whenever η α A α := B z α r zα z α z α z α,ρ for any α {,} an K z = Conv z α + 2 r zα z α : α {,} z α z α K η = Conv η α : α {,} we have Let M 0 > σ min{νa α :α {,} } an M R be such that X K z K η K η X. 26 P inf x X { ˆφ n x} M 0 i.o. = 0 an M = sup {φx}. α {,} A α x Conv From Lemmas 3. an 3.2 we can fin, with probability one, n 0 N such that inf x X { ˆφ n x} > M 0 an inf x Aα { ˆφ n x φx } < M 0 for any n n 0. Define M = M + M 0 K X = 4 b a r min j {b j a j } M an take any n n 0. Then, for any α {,} we can fin η α A α such that ˆφ n η α φη α < M 0. Then, 26 implies that ˆφ n x M x X. Take then x X an ξ ˆφ n x. A connecteness argument, like the one use in the proof of Lemma 3.2, implies that there is t > 0 such that x + t ξ K η. But then we must have t > r min j {b j a j } 2 ξ b a as a consequence of 26, since the smallest istance between K z an X is r min j {b j a j } 2 b a an K η ExtK z. This can be seen by taking a look at Figure 4, which shows the situation in the two imensional case. Thus, using the efinition of subgraients, r min j {b j a j } ξ,ξ ξ, t ξ ˆφ n x + t ξ ˆφ n x 2M 2 ξ b a which in turn implies ξ K X. We have therefore shown that, with probability one, we can fin n 0 N such that ξ K X ξ ˆφ n x, x X, n n 0. This completes the proof. 22

23 K η K z X Figure 4: The smallest istance between K z an X is at least r min j {b j a j } 2 b a. 4.5 Proof of Lemma 3.5 The result is obvious for conitions {A-A3} an {A5-A7} when σ 2 = 0. So we assume that σ 2 > 0 for {A-A3} an {A5-A7}. Let ɛ > 0 an M = sup x X { x }. Choose δ > 0 satisfying 22M+K + n ɛ < δ < n j = Y j φx j 2M+K + n ɛ n j = Y j φx j 27 for n large. Notice that δ is well-efine an the quantity on the left is positive, finite an boune away from 0 as lim n n j = Y j φx j > 0 a.s. uner any set of regularity conitions for {A2-A4}, conitions A4-i an A4-iii imply that we can apply the version of the strong law of large number for uncorrelate ranom variables, as it appears in Chung 200, page 08, Theorem 5..2 to the sequence ɛ j j = ; for {A-A3} an {A5-A7} this is immeiate as σ 2 > 0. The efinition of the class D K,X implies that all its members are Lipschitz functions with Lipschitz constant boune by K, a consequence of Rockafellar 970, Theorem 24.7, page 237. Hence, 27 implies that Now, efine N n N by N n = implies sup x y <δ x,y X,ψ D K,X { ψx ψy } n ɛ n j = Y j φx j. iamx δ 2K δ, where enotes the ceiling function. Observe that 27 N n iamx 2K 22M + K + Y j φx j. 28 ɛ n j = 23