DELTA METHOD, INFINITE DIMENSIONAL

Transcription

1 DELTA METHOD, INFINITE DIMENSIONAL Let T n be a sequence of statistics with values in some linear topological space D converging to some θ D. Examples are the empirical distribution function in the Skorohod space D[, + ] and the empirical measure indexed by functions belonging to some class F on the space l (F) of real-valued bounded functions on F equipped with the topology of uniform convergence. Assume that the statistics T n satisfies a limit result in the sense that r n (T n θ) converges in distribution to some limit for some sequence of positive numbers r n +. Many important statistics can be writtenintheform (T n ), i.e., as a transformation of T n,wherethemap is defined on asubsetd of D and takes values in some other linear topological space F.Hence,it is of considerable interest to obtain conditions on such that the expressions r n ( (T n ) (θ)) also converge in distribution. Results of this type go back to the work of von Mises [36] and are known as the von Mises calculus or as the delta method. Basic ingredients of the delta method are the weak convergence in the spaces D and F, respectively, and suitable notions of directional derivatives of. Since in some applications the statistics of interest T n are not Borel measurable, we make use of the extended weak convergence theory in metric spaces. Hence, it is assumed throughout that the linear topological spaces D and F are metrizable, i.e., there exist metrics on D and F that are compatible with their topologies. Important special cases are linear normed spaces. However, it will not be required that these spaces are separable and complete, respectively. THEORY Weak Convergence Let us first introduce concepts that are needed to deal with nonmeasurable maps. If (, F, P) is a probability space and Y an arbitrary map from to the extended reals, the outer integral E Y of Y is defined by E Y := inf{e(u) :U Y, U is measurable and E(U) exists} with E(U) denoting the expectation of U.The outer probability P (B) of an arbitrary subset B of is given by P (B) := inf{p(a) :B A, A F}. Now, let ( n, F n, P n ) be a sequence of probability spaces. A sequence of maps X n : n D is said to converge weakly to a Borel probability measure L on D, denoted by X n L, if lim n E f (X n ) = fdl holds for every real-valued bounded and continuous function f on D. IfX is a Borel measurable map from some probability space to D and has probability distribution L,itisalso said that X n converges weakly (or in distribution) to X and denoted by X n X. Weshall need that the convergence of a sequence of maps X n in outer probability implies its weak convergence, and that the converse is true if X n converges weakly to a constant c D. The most important result from weak convergence theory for deriving the delta method is the extended continuous mapping theorem. Let D n be subsets of D and g n : D n F (n 0) satisfy the following property for every sequence x n D n : for every subsequence n, if x n x and x D 0,theng n (x n ) g 0 (x). Let X n : n D n and X be Borel measurable with values in D 0 such that g 0 (X) is Borel measurable. Then X n X implies the weak convergence g n (X n ) g 0 (X) onf. For an introduction to extended weak convergence the reader may consult [35, Section 1] and for a proof of the latter result [34, Section 18.2]. Hadamard Directional Differentiability Several notions of directional differentiability of mappings on linear topological spaces have been introduced (e.g. [4]). It has been suggested by Reeds [22] and further developed by Fernholz [11] and Gill [13] to depart from the Gateaux and Fréchet directional differentiability concepts to that of Hadamard. D Encyclopedia of Statistical Sciences, Copyright 2006 John Wiley & Sons, Inc. 1

2 2 DELTA METHOD, INFINITE DIMENSIONAL Amap defined on a subset D of D with values in F is called Hadamard directionally differentiable at θ D if there exists a mapping θ : D F such that (θ + t n h n ) (θ) lim = n θ (h) (1) t n holds for any h D and for any sequences t n converging to 0+ and h n having the property θ + t n h n D and converging to h in D. In some recent work Hadamard directional differentiability appears under different names. For example, it is called Bouligand differentiability (or B-differentiabilty) in [23] and semidifferentiability in [25] as in the setvalued situation below. We note that linearity of the Hadamard directional derivative θ ( ) isnot required. Indeed, θ ( ) is often not linear if is given by inequality constraints as can be seen from the examples given later in this note. However, as an immediate consequence of the definition we observe that θ ( ) ispositively homogenous if it exists, i.e., it holds θ (th) = t θ (h) for all t 0andh D.Furthermore,ifθ is an interior point of D, the mapping θ ( )iscontinuous on D (see [30]). Since the domain D of is not open in D in many applications, the following extension of Hadamard directional differentiability becomes important. For a subset of D the map is said to be Hadamard directionally differentiable at θ D tangentially to if the limit (1) exists for all sequences t n 0+ and h n having the form h n = θn θ with θ tn n and converging to h. In the latter case, the Hadamard directional derivative θ ( ) is defined on the contingent or tangent cone to at θ, i.e., on the set { T (θ) = h D : h = lim n θ n θ t n for some sequences θ n θ in and t n 0 + The cone T (θ) is nonempty only if θ belongs to the topological closure cl of.ifθ belongs to the interior of, we have T (θ) = D. Hence, the tangential version of Hadamard directional differentiability extends the original notion. Furthermore, the cone T (θ) is always closed. It is convex if the set is convex, where it has the form T (θ) = cl { x θ t : }. t > 0, x }.If is a linear subspace of D,we have T (θ) = cl if θ cl. Additional information on tangent and related cones can be found in [2]. Clearly, θ ( ) is again positively homogenous, i.e., θ (th) = t θ (h)forallt 0 and h T (θ), and continuous on T (θ). Moreover, Hadamard directional differentiability at θ D implies Gateaux directional differentiability at θ D. Both notions are equivalent if D and F are normed spaces and is locally Lipschitz continuous around θ. Hadamard directional differentiability at θ D also implies compact directional differentiability at θ, i.e., there exists a positively homogeneous mapping A : D F such that r(th) t (θ + h) = (θ) + A(h) + r(h) and converges to 0 as t 0+ uniformly with respect to h varying in any compact subset of D. The converse holds if, in addition, A is continuous. Another important property of Hadamard directional differentiability is the chain rule: If : F is Hadamard directionally differentiable at θ tangentially to and : D F G (with G denoting another metrizable linear topological space) is Hadamard directionally differentiable at (θ) tangentially to ( ) D, then the composite mapping is Hadamard directionally differentiable at θ tangentially to, where θ : T (θ) F and (θ) : T ( ) ( (θ)) G, and the chain rule ( ) θ = (θ) θ is valid. For proofs of these statements it is referred to [30]. Delta Method Theorem 1. Let D and F be metrizable linear topological spaces, be a subset of D, : F a mapping and assume that the following two conditions are satisfied: (i) The mapping is Hadamard directionally differentiable at θ tangentially to with derivative θ ( ) : T (θ) F. (ii) For each n, X n : n are maps such that r n (X n θ) X holds for some sequence r n + and some random element X that takes its values in T (θ). Then we have r n ( (X n ) (θ)) θ (X). Furthermore, if is convex, the sequence

3 DELTA METHOD, INFINITE DIMENSIONAL 3 r n [ (X n ) (θ) θ (X n θ)] converges to zero in outer probability. Proof. For each n we define D n :={h D : θ + h rn } and g n : D n F, g n (h) := r n ( (θ + h rn ) (θ)) for all h D n.thenwe conclude from (i) that for every subsequence h n D n converging to h, henceh T (θ), the sequence g n satisfies g n (h n ) θ (h). In addition, θ (X) is Borel measurable since θ ( ) is continuous on T (θ). Therefore, the extended continuous mapping theorem for weak convergence implies g n (r n (X n θ)) = r n ( (X n ) (θ)) θ (X), i.e., the first assertion. To prove the second one, we consider the mapping ĝ n : D n F F,whereĝ n (h) := (g n (h), θ (h)) for all h D n. The mapping is well-defined since D n is a subset of the cone T (θ) due to the convexity of. Since θ ( ) is continuous on T (θ), we have for every subsequence h n D n converging to h that ĝ n (h n ) converges to θ (h) θ (h). Hence, the extended continuous mapping theorem implies ĝ n (r n (X n θ)) θ (X) θ (X). Again using the continuous mapping theorem the difference r n ( (X n ) (θ)) θ (r n(x n θ)) of the components of ĝ n (r n (X n θ)) converges weakly to θ (X) θ (X). Hence, the sequence r n( (X n ) (θ)) r n θ (X n θ) converges weakly to 0 and, thus, also in outer probability. For measurable maps X n and by imposing a separability condition on the limit X, the delta method is an immediate consequence of the almost sure representation theorem (e.g., [35, Theorem ]). The latter approach has been used e.g. in [26, Chapter 6] and in [33] to prove the above result in the measurable situation. The above proof follows the ideas in [34, Section 20] for the nonmeasurable case. The corresponding result in [34] makes use of the stronger requirement on that θ is linear. Normal vs Non-Normal Limits If the weak limit X of the original sequence r n (X n θ) is a normal (or Gaussian) random element in D, the question arises whether the limit θ (X) of the transformed sequence r n ( (X n ) (θ)) is also normal in F. The answer is yes if D and F are normed spaces, X is normal and the Hadamard directional derivative θ is linear (in addition to continuity) on the linear subspace of D containing the values of X n and θ. To prove this observation, let us recall that the Borel measurable θ (X) is a normal random element iff f, θ (X) is a real normal random variable for every f in the dual space F to F. Here,, denotes the dual pairing of elements in a normed space and its dual, respectively. For any f F we use the identity f, θ (X) = ( θ ) f, X with the linear dual mapping ( θ ) : F. It remains to note that the real random variable ( θ ) f, X is normal as ( θ ) f is an element of. However, in general, the cone T (θ) is not a linear subspace of D and the Hadamard directional derivative θ is not linear. Hence, θ (X) is the nonlinear image of a cone-valued random element and, hence, asymptotic normality of the transformed sequence is lost in general. This aspect is discussed in many papers dealing with asymptotics in (constrained) M-estimation and stochastic programming, respectively (e.g., [14,29,9]). Higher Order Delta Methods The first group of variants of the delta method deals with higher order expansions of the mapping. Let us derive, for example, a second order result. The mapping : D F is called second order Hadamard directionally differentiable at θ tangentially to if it is first order Hadamard directionally differentiable at θ tangentially to with derivative θ and if for all h T (θ) the limit θ (h) := lim n + (θ + t n h n ) (θ) t n θ (h n) 1 2 t2 n exists for all sequences h n and t n such that θ + t n h n, h n converging to h in D and t n converging to 0+. Then θ is also continuous on T (θ) and it holds θ (th) = t2 θ (h) forall t 0andh T (θ). If the mapping is twice continuously differentiable at θ, then θ (h) coincides with the second order term in the Taylor expansion of (θ + h). Theorem 2. Let D and F be metrizable linear topological spaces, be a convex subset of D and : F be second order

4 4 DELTA METHOD, INFINITE DIMENSIONAL Hadamard directionally differentiable at θ tangentially to. Furthermore, let X n : n be maps such that r n (X n θ) X holds for some sequence r n + and some random element X that takes its values in T (θ). Then r 2 n [ (X n) (θ) θ (X n θ)] 1 2 θ (X), and r 2 n [ (X n) (θ) θ (X n θ) 1 2 θ (X n θ)] converges to 0 in outer probability. Proof. We argue as in the proof of Theorem 1 by means of the extended continuous mapping theorem for weak convergence. For every n we set again D n :={h D : θ + h } and consider g rn n : D n F, g n (h) := r 2 n ( (θ + h ) (θ) 1 rn rn θ (h)), for proving the first assertion and ĝ n : D n F F, ĝ n (h) := (g n (h), θ (h)) for the second. It remains to note that g n and ĝ n are welldefined since D n T (θ) holds due to the convexity of. The above result generalizes to higher order Hadamard directional derivatives in a straightforward way. Set-Valued Delta Method In some applications one might be concerned with a set-valued mapping from a subset D of D to F. For simplicity we assume that is closed-valued, i.e., the values (θ) are closed subsets of F for every θ D. DenotingbyCL(F) the set of all nonempty closed subsets of F, may also be regarded as a single-valued mapping from D to CL(F). Hence, in order to extend the delta method to the set-valued case, we need a topology on CL(F). A variety of such topologies is known from the literature (see [5] for an overview). Some of them are defined by distance functions to sets, i.e, by the functions d(x, ) fromf to the extended reals, where d(x, A) := inf d(x, y) for every subset A of F y A and d denotes a metric that is compatible with the topology of F. Forexample, thewijsman topology τ W,theAttouch-Wets topology τ AW and the Hausdorff topology τ H are such topologies on CL(F). A sequence A n in CL(F) converges to A with respect to τ W, τ AW and τ H iff the sequence d(, A n ) of distance functions converges to d(, A) pointwise, uniformly on bounded subsets of F and uniformly on F, respectively. The Wijsman topology is metrizable iff F is separable. The Attouch-Wets topology is metrizable, for example, by the metric d AW (A, B) := 2 k min{1, sup d(x, A) k=1 d(x,x 0 ) k d(x, B) } (A, B CL(F)), for some fixed x 0 F. The Hausdorff topology is metrized by the uniform distance of distance functions, i.e., by d H (A, B) := sup d(x, A) d(x, B) x F (A, B CL(F)). Clearly, the metrics d AW and d H depend on the metric d being compatible with the topology on F. Thetopologiesτ AW and τ W coincide with the topology of Painlevé-Kuratowski set convergence if the linear space F is finitedimensional ([5, Section 5.2], [25, Chapter 4]). Now, we assume that CL(F) is equipped with the Attouch-Wets topology τ AW.Itissaid that the map : D CL(F) issemidifferentiable at a pair (θ, x) F, x (θ), tangentially to if there exists a mapping (θ,x) : T (θ) CL(F) such that the limit (θ + t n h n ) x (θ,x)(h) = lim n t n exists for every h T (θ) and for all sequences t n converging to 0+ and h n converging to h in D and having the property θ + t n h n. We observe again that the semiderivative (θ,x) is continuous on T (θ) and positively homogeneous if it exists, i.e., it holds 0 (θ,x) (0), (θ,x) (th) = t (θ,x) (h) for all t > 0 and h T (θ). If is singlevalued, semidifferentiability of at (θ, x) with x = (θ) is equivalent to Hadamard directional differentiability of at θ (tangentially to ). Semidifferentiability of setvalued maps has been introduced in [21]. If F is finite-dimensional, the semidifferentiability of at (θ, x) is equivalent to the Hadamard directional differentiability of the function d(, ( )) at (x, θ) [3]. Further properties and relations to contingent derivatives and protodifferentiability of set-valued mappings are discussed in [24,15,3] for finite-dimensional spaces D and F.

5 DELTA METHOD, INFINITE DIMENSIONAL 5 Theorem 3. Let D and F be metrizable linear topological spaces, be a subset of D, be a mapping from to CL(F) endowed with the Attouch-Wets topology τ AW and let the following two conditions be satisfied: (i) The mapping : CL(F) issemidifferentiable at (θ, x) F tangentially to with semiderivative (θ,x) ( ) : T (θ) CL(F) forsomex (θ). (ii) For each n, X n : n are maps such that r n (X n θ) X holds for some sequence r n + and some random element X that takes its values in T (θ). Then we have r n ( (X n ) x) (θ,x)(x) (2) in the sense of extended weak convergence in the metrizable space (CL(F), τ AW ). Proof. We argue again as in the proof of Theorem 1 by means of the extended continuous mapping theorem for weak convergence, but this time for weak convergence in the metric space (CL(F), d AW ). Here, we set again D n :={h D : θ + h } and consider the mapping g n : D n CL(F), g n (h) := rn r n ( (θ + h ) x) forallh D rn n and all natural n. The above set-valued delta method is proved in [15] in the measurable situation, i.e., in case that the r n (X n θ) are random closed sets converging in distribution to the random closed set X. In that case the convergence (2) is equivalent to the convergence in distribution of {d(y, r n ( (X n ) x))} y F as stochastic processes to {d(y, (θ,x) (X))} y F if F is finitedimensional [28, Theorem 2.5]. Extensions of the results in [15] and of Theorem 3 to the more general setting of linear topological Hausdorff spaces can be found in [17,18]. If CL(F) is endowed with the topology τ W, is continuous and has convex images, and the space D satisfies certain additional conditions, the weak convergence result (2) can be supplemented by classical delta theorems for countably many selections of that are Hadamard directionally differentiable at θ and form a Castaing representation of [7, Section 4]. The paper [7] also contains higher order extensions of Theorem 3 assuming higher order semidifferentiability of. For more information on random sets, their distribution and their convergence in distribution it is referred to [1,20,28]. EXAMPLES Many examples of the delta method are described in the recent monographs [35] and [34]. In particular, we refer to [35, Chapter 3.9.4] and [34, Chapter 20.3] for examples that lead to mappings having linear Hadamard directional derivatives, e.g., the Wilcoxon statistic, empirical quantiles, Nelson-Aalen estimator. The delta method for the bootstrap is discussed in [35, Chapter 3.9.3]. In the following, we provide a few recent examples of Hadamard directional differentiable functions from variational and nonsmooth analysis. Later we show their potential when applying the delta method to some models in M- and Z-estimation. Examples of Hadamard Directionally Differentiable Mappings (a) Infimal Value Mappings. Let X be a set and D := l (X) denote the linear normed space of real-valued bounded functions defined on X with the norm θ := sup x X θ(x) for θ D. We define to be the mapping that assigns to each θ D its infimum on X, i.e., (θ) := inf θ(x). (3) x X Furthermore, let S(θ, ε) :={x X : θ(x) (θ) + ε} denote the set of all ε-approximate minimizers of θ D. The following result is due to [19]. Proposition 1. The infimal value mapping is Hadamard directionally differentiable at any θ 0 D and we have for each h D that θ 0 (h) = lim ε 0 inf x S(θ 0,ε) h(x). (4) Proof. Let θ 0 and h be in D, t n be a sequence of positive numbers tending to 0 and h n be a sequence converging in D to h D. Then

6 6 DELTA METHOD, INFINITE DIMENSIONAL we obtain for any n and x n S(θ 0, t 2 n ) the estimates (θ 0 + t n h n ) (θ 0 ) (θ 0 + t n h n )(x n ) θ 0 (x n ) S(θ 0 ) ={x 0 }.Itissaidthatθ 0 satisfies a second order growth condition around x 0 on X if there exist a constant c > 0andaneighborhood U of x 0 such that + t 2 n t nh(x n ) + t n h n h +t 2 n 1 and, hence, lim sup ( (θ n tn 0 + t n h n ) (θ 0 )) lim ε 0 inf x S(θ0,ε) h(x). Now, let x n S(θ 0 + t n h n, t 2 n ). Then we have (θ 0 + t n h n ) (θ 0 ) (θ 0 + t n h n )( x n ) θ 0 ( x n ) t 2 n t nh n ( x n ) t 2 n t n t n inf x S(θ 0 +tnhn,t 2 n) inf x S(θ 0,2t 2 n+2tn hn ) h(x) t n h n h t 2 n h(x) t n h n h t 2 n, where the latter inequality is due to the fact that the inclusion S(θ 0 + θ, ε) S(θ 0,2ε + 2 θ ) is valid for any ε>0andθ D. Weconclude lim inf n 1 t n ( (θ 0 + t n h n ) (θ 0 )) lim inf h(x) ε 0 x S(θ 0,ε) and, hence, the proof is complete. Due to (4) θ 0 ( ) is not linear in general. Proposition 1 extends the corresponding result in [31] for the case D = C(X) withx being a compact metric space. To verify that is even second order Hadamard directionally differentiable at some θ 0 in D, some smoothness conditions on θ 0 and on the directions h are needed. Let X be a compact subset of a Euclidean space with norm and scalar product,. Let D := Lip (X) be the linear normed space of real-valued Lipschitz continuous functions on X with the norm θ := sup θ(x) x X { θ(x) θ( x) + sup x x } : x, x X, x x. Let be the linear subspace of D formed by all functions that are Hadamard directionally differentiable at x 0,wherex 0 is the unique minimizer of some given θ 0, i.e., θ 0 (x) θ 0 (x 0 ) + c x x 0 2, for all x X U. (5) Furthermore, some information on the curvature of X is needed. Therefore, we introduce the second order tangent set TX 2 (x, d) to X at x X in direction d T X (x) by TX 2 (x, d) :={w :inf z X x + td t2 w z =o(t 2 )}. Second order tangent sets areclosedand,inaddition,convexifx is convex. The set X is called second order regular at x 0 if for any d T X (x 0 )andany sequences x n X and t n > 0 such that x n = x 0 + t n d t2 n w n, t n 0, t n w n 0 the condition lim n inf{ w n z : z TX 2 (x 0, d)} =0 holds. For example, polyhedral convex sets are second order regular at any of their elements and it holds 0 TX 2 (x 0, d) = T TX (x 0 )(d) for every d T X (x 0 ). If θ 0 is twice continuously differentiable in a neighborhood of x 0 and X is second order regular at x 0,thefollowing second order optimality condition is necessary and sufficient for the second order growth condition (5) to hold: 2 θ 0 (x 0 )d, d + inf z T 2 X (x 0,d) θ 0 (x 0 ), z > 0 (6) for all d C(x 0 ) \{0}. Here, C(x 0 ):={d T X (x 0 ): θ 0 (x 0 ), d =0} and θ 0 (x 0 )and 2 θ 0 (x 0 ) denote the gradient and Hessian of θ 0 at x 0, respectively. The second term on the left hand side of (6) vanishes if θ 0 (x 0 ) = 0andX is polyhedral, respectively. Proposition 2. Assume that θ 0 is twice continuously differentiable in a neighborhood of its unique minimizer x 0 and that θ 0 satisfies the second order growth condition (5) around x 0 on X. Furthermore, let X be second order regular at x 0. Then the infimal value mapping defined on D by (3) is second order Hadamard directionally differentiable at θ 0 tangentially to

7 DELTA METHOD, INFINITE DIMENSIONAL 7 and θ 0 (h) = holds for every h. { inf 2h x d C(x 0 ) 0 (d) + 2 θ 0 (x 0 )d, d } + inf θ 0 (x 0 ), z (7) z T X 2 (x 0,d) The above result is a special case of [6, Theorem 4.156]. It simplifies considerably if the set X is polyhedral. In the latter case we obtain the following expansion of at θ 0 by combining Propositions 1 and 2: (θ 0 + th) = (θ 0 ) + th(x 0 ) t2 inf d C(x 0 ) { } 2h x 0 (d) + 2 θ 0 (x 0 )d, d + o(t 2 ). Minimizing Set Mappings. As in the previous example we consider the linear normed space Lip (X) of real-valued functions on a compact subset X of a Euclidean space and the set-valued map : D CL (F) defined by (θ) := S(θ) = arg min θ(x). (8) x X Let be the linear subspace of D containing all functions that are Hadamard directionally differentiable at x 0 with x 0 being the unique minimizer of some function θ 0 in, i.e., (θ 0 ) ={x 0 }. Proposition 3. Let the assumptions of Proposition 2 be satisfied. Assume, in addition, that for every h the infimum on the right hand side of (7) is attained at a unique element d(h) of the cone C(x 0 ). Then is semidifferentiable at (θ 0, x 0 ) tangentially to and it holds (θ 0,x 0 )(h) = d(h) for every h. Proposition 3 is also a consequence of [6, Theorem 4.156]. In general, is no longer semidifferentiable at (θ 0, x 0 ) if the minimizing set S(θ 0 ) is not a singleton. However, in the special case of two-stage stochastic programs with multiple solutions such a semidifferentiability result was established in [8]. Implicit Functions. Let D, F and G be linear normed spaces and F be complete, i.e., a Banach space. Let U and V be neighborhoods of θ 0 D and y 0 F, respectively. Let f be a function from U V to G with f (θ 0, y 0 ) = 0. Suppose the function to be given implicitly as a solution of the equation f (θ, (θ)) = 0 for θ D near θ 0. In order to derive conditions on the existence and Hadamard directional differentiability of such an implicitly defined function, we assume that f has at (θ 0, y 0 )partial Hadamard directional derivatives f θ,(θ 0,y 0 ) (from D to G) andf y,(θ 0,y 0 ) (from F to G) with respect to θ and y, respectively. The following result [23, Corollary 3.4] forms an extension of the classical implicit function theorem to the case of Hadamard directionally differentiable functions. Proposition 4. In addition, assume that, for some l>0andeachy V, f (, y) islipschitz continuous on U with modulus l and that (a) the partial Hadamard directional derivative f y,(θ 0,y 0 ) is strong at (θ 0, y 0 ), i.e., for each ɛ>0 there are neighborhoods U ɛ of θ 0 D and V ɛ of the origin in F such that for each θ U ɛ the function h f (θ, y 0 + h) f (θ, y 0 ) f y,(θ 0,y 0 )(h) is Lipschitz continuous on V ɛ with modulus ɛ. (b) f y,(θ 0,y 0 ) (V y 0) is a neighborhood of the origin in G and f y,(θ 0,y 0 ) has an inverse on that neighborhood which is Lipschitz continuous. Then there exist neighbourhoods U and V of θ 0 and y 0, respectively, and a Lipschitz continuous function from U to V such that (θ 0 ) = y 0, (θ) is the unique solution of f (θ, y) = 0 for each θ U and is Hadamard directionally differentiable at θ 0 with θ 0 (h) = [f y,(θ 0,y 0 ) ] 1 (9) ( f θ,(θ 0,y 0 )(h)) (h D).

8 8 DELTA METHOD, INFINITE DIMENSIONAL Examples of the Infinite Dimensional Delta Method M-Estimation. Let us consider the following problem of constrained M-estimation or, with other words, the stochastic programming model (cf. [27]) min { θ(x) := E[f (x, ξ)] : x X}, (10) where f is an extended real-valued function defined on R m R s, X is a compact polyhedral set (i.e., a polytope) in R m, ξ is an R s -valued random vector on some probability space and E denotes expectation. We assume that the function f is a normal integrand (cf. [25, Chapter 14]) and, hence, that f (x, ξ) is also a random variable for every x X. Furthermore, let there exist some nonnegative function a on R s such that E[a(ξ)] < and f (x, ξ) f ( x, ξ) a(ξ) x x ( x, x X). (11) Let ξ 1, ξ 2,..., ξ n,... be a sequence of independent random variables having the same probability distribution as ξ and we consider the sequence of minimization problems min { θ n (x) := 1 n f (x, ξ i ):x X}. (12) n i=1 Solutions x n of (12) are called M-estimators of asolutionx 0 to (10). Due to (11) the function θ is in D := Lip(X)andthemapsθ n take values in D. Proposition 5. Assume, in addition, that θ is twice continuously differentiable in a neighborhood of its unique minimizer x 0 X and that θ satisfies the second order growth condition (5) around x 0 in X. Furthermore, assume that n 1 2 (θ n θ) converges in distribution tosomerandom elementη with values in the linear subspace of functions in Lip(X) being Hadamard directionally differentiable at x 0.Thenwehave n 1 2 (x n x 0 ) d(η) (13) = arg min d C(x 0 ) {2η x 0 (d) + 2 θ(x 0 )d, d }, if the minimization problem on the right hand side of (13) has a unique solution. Here, C(x 0 ):={d T X (x 0 ): θ(x 0 ), d =0}. The limit is normal if x 0 is an interior point of X. For the proof it remains to appeal to Proposition 3. Note that the minimization problem on the right hand side of (13) is a random quadratic program with linear cone constraints. For related results and extensions we refer to [10,12,16,32,33,37]. Z-Estimation. Let F be a Banach space, B an open subset of F and G a linear normed space. We consider the linear normed space D := l (B, G) of all uniformly norm-bounded functions from B to G. Letθ 0 D have the zero y 0 B, be the subset of maps in D with at least one zero and be a map that assigns to each element in one of its zeros and has the property (θ 0 ) = y 0. We consider the implicit equation f (θ, y) := θ(y) = 0 and obtain that is Hadamard directionally differentiable at θ 0 with θ 0 (h) = [(θ 0 ) y 0 ] 1 ( h(y 0 )) for all h D, if θ 0 is Hadamard directionally differentiable in y 0 and the derivative (θ 0 ) y 0 satisfies the conditions (a) and (b) of Proposition 4. Now, let y n be the estimators of y 0 by solving the equations θ n (y) = 0 and assume that the sequence n 1 2 (θ n θ 0 ) converges in distribution in D to some random element η. Then we conclude from Theorem 1 that n 1 2 (y n y 0 ) [(θ 0 ) y 0 ] 1 ( η(y 0 )). Acknowledgments The author s work was supported by the DFG research centre Mathematics for key technologies (FZT 86) in Berlin. The author wishes to thank Alexander Shapiro (Georgia Institute of Technology) and Petr Lachout (Charles University Prague) for valuable comments. REFERENCES 1. Artstein, Z. (1983). Distributions of random sets and random selections, Isr. J. Math. 46, Aubin, J.-P., and Frankowska, H. (1990). Set- Valued Analysis, Birkhäuser, Boston. 3. Auslender, A., and Cominetti, R. (1991). A comparative study of multifunction differentiability with applications in mathematical programming, Math. Oper. Res. 16,

9 DELTA METHOD, INFINITE DIMENSIONAL 9 4. Averbukh, V. I., and Smolyanov, O. G. (1967). The theory of differentiation in linear topological spaces, Russ. Math. Surv. 22 (1967), Beer G. (1993). Topologies on Closed and Closed Convex Sets, Kluwer, Dordrecht. 6. Bonnans, J. F., and Shapiro, A. (2000). Perturbation Analysis of Optimization Problems, Springer-Verlag, New York. 7. Dentcheva, D. (2001). Approximations, expansions and univalued representation of multifunctions, Nonlinear Anal., Theory Methods Appl. 45 (2001), Dentcheva, D., and Römisch, W. (2000). Differential stability of two-stage stochastic programs, SIAM J. Optim. 11, Dupačová, J. (1991). On non-normal asymptotic behavior of optimal solutions for stochastic programming problems and on related problems of mathematical statistics, Kybernetika 27, Dupačová, J., and Wets, R. J.-B. (1988). Asymptotic behavior of statistical estimators and of optimal solutions of stochastic optimization problems, Ann. Stat. 16, Fernholz, L. T. (1983). Von Mises calculus for statistical functionals, Lecture Notes in Statistics 19, Springer-Verlag, New York. 12. Geyer, C. J. (1994). On the asymptotics of constrained M-estimation, Ann. Stat. 22, Gill, R. D. (1989). Non- and semi-parametric maximum likelihood estimators and the von Mises method (Part I), Scand. J. Stat. 16, King, A. J. (1986). Asymptotic behavior of solutions in stochastic optimization: Nonsmooth Analysis and the derivation of non-normal limit distributions, Ph.D. dissertation, Dept. Mathematics, Univ. Washington, Seattle. 15. King, A. J. (1989). Generalized delta theorems for multivalued mappings and measurable selections, Math. Oper. Res. 14, King, A. J., and Rockafellar, R. T. (1993). Asymptotic theory for solutions in statistical estimation and stochastic programming, Math. Oper. Res. 18, Lachout, P. (1994). A general version of delta theorem, Acta Univ. Carolinae Math. et Phys. 35, Lachout, P. (2002). The delta theorem in general setup, in: Limit Theorems in Probability and Statistics II (I. Berkes, E. Csáki. M. Csörgö Eds.),János Bolyai Mathematical Society, Budapest, Lachout, P. (2004). Personal communication. 20. Norberg, T. (1984). Convergence and existence of random set distributions, Ann. Probab. 12, Penot, J.-P. (1984). Differentiability of relations and differential stability of perturbed optimization problems, SIAM J. Control Optim. 22, Reeds, J. A. (1976). On the Definition of von Mises Functionals, Research Report S-44, Department of Statistics, Harvard University, Cambridge, MA. 23. Robinson, S. M. (1991). An implicit-function theorem for a class of nonsmooth functions, Math. Oper. Res. 16, Rockafellar, R. T. (1989). Proto-differentiability of set-valued mappings and its application in optimization, Ann. Inst. Henri Poincaré, Anal. Non Linéaire 6, Suppl., Rockafellar, R. T., and Wets, R. J.-B. (1998). Variational Analysis, Springer, Berlin. 26. Rubinstein, R. Y., and Shapiro, A. (1993). Discrete Event Systems, Sensitivity Analysis and Stochastic Optimization by the Score Function Method, Wiley, Chichester. 27. Ruszczyński, A., and Shapiro, A. (Eds.) (2003). Handbooks in Operations Research and Management Science, Volume 10: Stochastic Programming, Elsevier, Amsterdam. 28. Salinetti, G., and Wets, R. J.-B. (1986). On the convergence in distribution of measurable multifunctions (random sets), normal integrands, stochastic processes and stochastic infima, Math. Oper. Res. 11, Shapiro, A. (1989). Asymptotic properties of statistical estimators in stochastic programming, Ann. Stat. 17, Shapiro, A. (1990). On concepts of directional differentiability, J. Optimization Theory Appl. 66 (1990), Shapiro, A. (1991). Asymptotic analysis of stochastic programs, Ann. Oper. Res. 30, Shapiro, A. (2000). On the asymptotics of constrained local M-estimators, Ann. Stat. 28, Shapiro, A. (2000). Statistical inference of stochastic optimization problems, in: Probabilistic Constrained Optimization (S. P. Uryasev Ed.), Kluwer, Dordrecht, van der Vaart, A. W. (1998). Asymptotic Statistics, Cambridge University Press.

10 10 DELTA METHOD, INFINITE DIMENSIONAL 35. van der Vaart, A. W., and Wellner, J. A. (1996). Weak Convergence and Empirical Processes, Springer,NewYork. 36. von Mises, R. (1947). On the asymptotic distribution of differentiable statistical functions, Ann. Math. Stat. 18, Wets, R. J.-B. (1991). Constrained estimation: consistency and asymptotics, Appl. Stochastic Models Data Anal. 7, FURTHER READING Huber, P. J. (1981). Robust Statistics, Wiley, New York. Rieder, H. (1994). Robust Asymptotic Statistics, Springer, New York. Serfling, R. J. (1980). Approximation Theorems of Mathematical Statistics, Wiley, New York. van de Geer, S. (2000). Empirical Processes in M-Estimation, Cambrigde University Press. See also EMPIRICAL PROCESSES; LINEAR PROGRAMMING; MATHEMATICAL PROGRAMMING; M-ESTIMATORS;andSTATISTICAL DIFFERENTIALS, METHOD OF. WERNER RÖMISCH