A robust, adaptive M-estimator for pointwise estimation in heteroscedastic regression

Transcription

1 Bernoulli 203), 2014, DOI: /13-BEJ533 A robust, adaptive M-estimator for pointwise estimation in eteroscedastic regression MICHAËL CHICHIGNOUD * and JOHANNES LEDERER ** Seminar for Statistics, ETH Züric, Rämistrasse 101, CH-8092 Züric, Switzerland. * cicignoud@stat.mat.etz.c; ** lederer@stat.mat.etz.c We introduce a robust and fully adaptive metod for pointwise estimation in eteroscedastic regression. We allow for noise and design distributions tat are unknown and fulfill very weak assumptions only. In particular, we do not impose moment conditions on te noise distribution. Moreover, we do not require a positive density for te design distribution. In a first step, we study te consistency of locally polynomial M-estimators tat consist of a contrast and a kernel. Afterwards, minimax results are establised over unidimensional Hölder spaces for degenerate design. We ten coose te contrast and te kernel tat minimize an empirical variance term and demonstrate tat te corresponding M-estimator is adaptive wit respect to te noise and design distributions and adaptive Huber) minimax for contamination models. In a second step, we additionally coose a data-driven bandwidt via Lepski s metod. Tis leads to an M-estimator tat is adaptive wit respect to te noise and design distributions and, additionally, adaptive wit respect to te smootness of an isotropic, multivariate, locally polynomial target function. Tese results are also extended to anisotropic, locally constant target functions. Our data-driven approac provides, in particular, a level of robustness tat adapts to te noise, contamination, and outliers. Keywords: adaptation; Huber contrast; Lepski s metod; M-estimation; minimax estimation; nonparametric regression; pointwise estimation; robust estimation 1. Introduction We introduce a new metod for pointwise estimation in eteroscedastic regression tat is adaptive wit respect to te model, in particular, wit respect to te noise and te design distribution Dadaptive) and te smootness of te regression function S-adaptive). Let us first briefly summarize te related literature. First, te seminal paper 14] contains a proof of te asymptotic normality of M-estimators for te location parameter in regular models. Furtermore, te series of papers 30 33] provide minimax results for nonparametric regression. More recently, a block median metod was used in 5] to prove te asymptotic equivalence between Gaussian regression and omoscedastic regression for deterministic designs and possibly eavy-tailed noises. Using a blockwise Stein s Metod wit wavelets, tis leads to an S-adaptive estimator tat is adaptive optimal over Besov spaces wit respect to te L 2 -risk and adaptive optimal over isotropic Hölder classes wit respect to te punctual risk. Moreover, using an estimate of te noise density at 0 and a plug-in metod, tis also leads to a D-adaptive estimator. However, in contrast to tis paper, only omoscedastic regression is considered and multivariate regression functions, in particular anisotropic functions, are not allowed for. Next, a modified version of Lepski s metod was applied for omoscedastic regression in 27]. Finally, local M- estimators, also for regression models wit degenerate designs, were intensively studied in te ISI/BS

2 A fully adaptive pointwise M-estimator 1561 case of Gaussian regression: S-adaptivity results of a local least squares estimator were derived in 8], sup-norm S-minimax results were establised in 9], and te effect of degenerate designs on te L 2 -norm was investigated wit wavelet-type estimators in 1]. However, in contrast to tis paper, pointwise estimation wit random, possibly degenerate designs and eteroscedastic, possibly eavy-tailed noises as not been included. Wat is te main idea beind our approac? Consider te estimation of t 0 R in te translation model Y g t 0 ) for a probability density g. Te M-estimator ˆt of t 0 corresponding to te contrast ρ ) and te sample Y 1,...,Y n of Y is ten It olds tat see 14 16]) ˆt := arg min t ρy i t). n ˆt t 0 ) L ρ ) 2 dg N 0, AV), were AV := n ρ dg) 2, 1.1) G is te distribution of Y t 0, ρ ) and ρ ) are te first and second derivatives of te contrast ρ ), and L indicates convergence in law. In oter words, ˆt is asymptotically normal wit asymptotic variance AV. Tis result suggests tat an optimal estimator is obtained by minimizing te asymptotic variance. Moreover, te Crámer Rao Inequality and see 14]) inf ρ ρ ) 2 dg ρ dg) 2 = IG) ) 1, 1.2) were I ) is te Fiser information and te infimum is taken over all twice differentiable contrasts, imply tat tis M-estimator is efficient. Huber proposed in 14], Proposal 3, to minimize an estimate of te above asymptotic variance since te distribution G is not available in practice) over te family of Huber contrasts teir definition is given below). He also conjectured tat te corresponding estimator is minimax for certain contamination models for more details, see Section A.1 in te arxiv version). More recently, in 2], an M-estimator wit a contrast tat minimizes an estimate of te asymptotic variance was introduced for te parametric model, its asymptotic normality was proved, and especially Huber contrasts indexed by teir scale and a family of l p losses were considered. In a first step, we derive general properties of M-estimators suc as pointwise risk bounds. Tis includes, in particular, S-minimax results for degenerate designs and allows us to recover results in 7] see Teorem 1 and Remark 1). In a second step, we ten consider a local M- estimator tat consists of a contrast and a kernel tat minimize an estimate of te variance and sow, in particular, tat tis estimator mimics te oracle, wic minimizes te true variance. Our data-driven approac can be used, for example, for te selection of te scale of te Huber contrast wit an adaptive robustness wit respect to outliers or for te selection of a suitable even noncentered or nonconvex) support tat takes a maximal number of points around x 0 into account cf. 12] for te latter objective). Finally, we sow tat our estimator is, under some restrictions on te design and te noise level see Condition 3), D-adaptive for various sets of contrasts and kernels wit finite entropy.

3 1562 M. Cicignoud and J. Lederer We finally study simultaneous D- and S-adaptation for anisotropic target functions. In a first step, we study te case of isotropic target functions, were te standard Lepski s metod see 23, 24]) can be applied. To tis end, we assume tat te variance of te estimator is decreasing wit respect to te bandwidt and plug-in an estimate of te minimal variance for te D-adaptation to apply Lepski s metod for te S-adaptation see Section 4.1). Tis yields te first estimator in eteroscedastic regression wit random designs and eavy-tailed noise distributions tat is simultaneously D- and S-adaptive and optimal in a sense describe later. Furtermore, we note tat applications of Lepski s metod to nonlinear estimators are still nonstandard and can only be found in a small number of examples in te literature 6,26,27]. In a next step, we extend our results to anisotropic target functions. For tis, we restrict ourselves to locally constant target functions and omoscedastic regression wit uniform design and apply a modification of Lepski s metod given in 19,22] to construct an optimal, simultaneously S- and D-adaptive estimator. Tis is te first application of Lepski s metod to nonlinear estimators of anisotropic target functions and yields a selection of an anisotropic bandwidt wic is of great interest for applications in te context of image denoising cf. 17]), for example. Altoug we consider estimation problems, our approac may also be useful for inference, for example, for te construction of confidence bands. Wile confidence bands for parametric estimation are derived from central limit teorems see 1.1)), confidence bands for nonparametric regression are especially desired to be adaptive wit respect to te smootness of te target function. Te construction of suc S-adaptive confidence bands is more difficult tan in te parametric case see 13]), but since Lepski-type procedures ave already been used in tis context, see 10], Teorem 1 and Corollary 1, we expect tat our approac may be useful for te construction of S-adaptive confidence bands for regression wit possibly eavy-tailed noises see Section 5 for a discussion of some tecnical aspects). Eventually, if for example te smootness is known, our approac may be used, plugging an estimate of te variance in te confidence band, to obtain D-adaptive confidence bands, wic are, in particular, adaptive wit respect to te design and te noise distributions. Te structure of tis paper is as follows: In te following section, we first introduce an estimator wic satisfies a risk bound see Teorem 1). So, S-minimax results are deduced over Hölder spaces see Corollaries 1, 2 and 3). We ten provide a coice for te contrast and te kernel see Teorem 2) via te minimization of a nonasymptotic variance. Ten, we provide a coice for te bandwidt for isotropic, locally polynomial target functions see Teorem 3) and for anisotropic, locally constant target functions see Teorem 4). After tis, we give a discussion on our assumptions and an outlook in Section 5. Te proofs are finally conducted in Section 6 and in te Appendix. For conciseness, only te crucial proofs are presented ere. For te remaining proofs and more details, in particular, on te parametric model and on a comparison to classical results, we refer to te longer version available on arxiv and te webpages of te autors. 2. Preliminary definitions and results In tis section, we give some preliminary definitions and results. After specifying te model, we introduce a first estimator and ten, we present a risk bound and S-minimax properties of tis estimator.

4 A fully adaptive pointwise M-estimator 1563 Let us first specify te model. Te observations X i,y i ),...,n satisfy te set of equations Y i = f X i ) + σx i )ξ i, i = 1,...,n, 2.1) and are distributed according to te probability measure P := P n) f wit associated expectation E := E n) f. We aim at estimating te target function f : 0, 1] d M,M] for M>0) at a given point x 0 on 0, 1) d. Te target function is assumed to be smoot, more specifically, it is assumed to belong to a Hölder class see Definition 4 below). Te target function is obscured by te second part of te above model, te noise. Te noise variables ξ i ) i 1,...,n areassumedtobe distributed independently according to te densities g i ) wit respect to te Lebesgue measure on R. Te noise densities g i ) may be unknown but are assumed to be symmetric. We stress tat we do not impose, unlike in te literature on te median cf. 5]), any moment assumptions on te noise, and we do not require tat te noise densities are positive at 0. We postpone te detailed discussion on te assumptions to te end of te next section. Te noise level σ : 0, 1] d 0, ) is assumed to be bounded, but may also be unknown. Usually, te noise level is te variance of te noise, owever, tis is not te case if te noise distributions do not ave any moments, for example. Finally, te design points X i ) i 1,...,n are assumed to be distributed independently and identically according to te density μ ) wit respect to te Lebesgue measure on R. We assume tat μ ) vanises at most finitely many points. For ease of exposition, we also assume tat X i ) i 1,...,n and ξ i ) i 1,...,n are mutually independent. Next, we introduce an estimator of f x 0 ) wit a local polynomial approac LPA) for a fixed bandwidt, a fixed kernel, and a fixed contrast. Te key idea of te LPA, as described for example in 18] orin34], Capter 1, is to approximate te target function in a neigborood of size 0, 1] d of a given point x 0 by a polynomial. To start, we define for a fixed m N te set P := {p = p 1,...,p d ) N d :0 p m} wit p =p 1 + +p d and denote its cardinality by P. Te cardinality P is exponential in d and enters te bounds derived below as a factor. For any multi-indexed column vector t = t p1,...,p d R : p P) R P and for any x 0, 1] d, we ten define te desired polynomial as ) x P t x) := t x0 U := ) x p x0 t p. p P Here, z p := z p 1 1 zp d d for all z R d, and te division by is understood coordinate wise. Next, for M>0, we define F := {P t : t M,M] P } as a set of polynomials of degree at most m. We now specify wat we mean by a kernel and a contrast: Definition 1. A function K : R d 0, ) is called kernel function) if it as te following properties: 1. K ) as a not necessarily symmetric) support wic is a ypercube aving edge lengt one and contains te origin; 2. K < and Kx)dx = 1. For ease of exposition, we set := d j=1 j and use te notation K ) := K x 0 )/)/ at some points. Moreover, we define te neigborood of x 0 of size as V :=

5 1564 M. Cicignoud and J. Lederer {x R d : K x) > 0} and assume for simplicity tat te kernel is cosen suc tat V 0, 1] d. Next, we specify wat we mean by a contrast: Definition 2. A function ρ : R 0, ) is called contrast function) if it as te following properties: 1. ρ ) is convex, symmetric and ρ0) = 0; 2. te derivative ρ ) of ρ ) is 1-Lipscitz and bounded; 3. te second derivative ρ ) of ρ ) is defined Lebesgue almost everywere and is 1-Lipscitz wit respect to te measure P. Moreover, ρ 1. Te constants in te Lipscitz condition and te boundedness condition in te last definition are set to 1 for ease of exposition only. Well-known contrasts are te Huber contrast see 14]), for any scale γ>0 and z R, { z ρ H,γ z) := 2 /2, if z γ, γ z γ/2 ), oterwise, 2.2) and te contrast induced by te arctan function see 30]) ρ arc,γ z) := γzarctanz/γ ) γ 2 2 ln 1 + z 2 /γ 2). 2.3) Note tat te square loss and te absolute loss do not satisfy te above definition. However, tey can be mimicked by te Huber contrast wit γ small median) and γ large mean). Let us define, for any function ζ, te empirical measure as P n ζ := n 1 n ζx i,y i ). We can now combine a kernel and a contrast to obtain te λ-lpa estimator fˆ λ x 0 ) of f x 0 ) defined as: ˆ f λ := arg min f F P nλf ), were λf )x, y) := ρ y fx) ) K x) for x 0, 1] d and y R. 2.4) Te coefficients of te estimated polynomial can be considered as estimators of te derivatives of te function f at x 0. In tis paper, owever, we focus on te estimation of f x 0 ) A first risk bound In tis section, we present a risk bound for te estimator introduced above. Tis estimator involves, in particular, fixed contrasts, kernels and bandwidts. To ease te presentation, we introduce some additional definitions. First, we define te best approximation of te target f in F as { f 0 := arg min sup fx) f x) } : f F,fx 0 ) = f x 0 ) 2.5) x V

6 A fully adaptive pointwise M-estimator 1565 and te associated bias term as b F) := sup x V f 0 x) f x). 2.6) Te minimum is not necessarily unique, but all minimizers work for our derivations. We ten fix a multi-indexed vector t 0 = tp 0 1,...,p d ) p P suc tat P t 0 = f 0. We recall tat te entropy wit bracketing of a set of functions A for a given radius u>0 wit respect to a pseudo)metric is te logaritm of te minimal number of pairs of functions f j) 1,f j) 2 ) A A suc tat for any f A, tere is a couple f j) 1,f j) 2 ) suc tat f j) 1 f f j) 2 and f j) 1,f j) 2 ) u. Here, in particular, H F ) denotes te entropy wit bracketing of F wit respect to te pseudometric EP n λ f 1 ) λ f 2 )] 2,f 1,f 2 F, were λ f )x, y) := ρ y fx) ) K x) for x 0, 1] d and y R. 2.7) Te entropy H F ) cannot be calculated if te probability law is unknown. However, it can be upper bounded invoking an upper bound for te pseudometric. For tis, one may use tat EP n λ f 1 ) λ f 2 ) ] 2 K t 1) t 2) 1 due to te continuity of ρ ) and te definition of F. Here, t 1) and t 2) M,M] P are suc tat P t 1) = f 1 and P t 2) = f 2, respectively. Terefore, te entropy H F ) can be bounded by P times te entropy of M,M] wit respect to te Euclidean distance multiplied by K tis is, in particular, independent of n). As a next step, we introduce te condition under wic we derive te risk bound. Condition 1. Let ρ ) be a contrast, K ) a kernel, n {1, 2,...}, and 0, 1] d. We say tat Condition 1 is satisfied if te smallest eigenvalue of te matrix 1 n E U Xi x 0 ) ) U Xi x 0 ρ ] ) σx i )ξ i K X) is positive, n 1, and, defining δ := 2 P 2 E K X) ] b F) + 54 ρ E K 2X)]+ K / n ) ] n lnn P ) H 1/2 F u) du + H, F 1)) 1 tat 4 ) 1 b F) + δ inf x V n E ρ )] σx)ξ i. 2.8) Condition 1 can be interpreted in te following sense: n must be sufficiently large and appropriate for te setting under consideration. In particular,, as a function of n, is usually

7 1566 M. Cicignoud and J. Lederer cosen suc tat 0 and n as n to satisfy Condition 1. We postpone a detailed discussion of Condition 1 to after te main result of tis section. Te variance term of te estimator is crucial for te following. To state it explicitly, we need to introduce some more notation: First, we introduce λ similarly as λ in 2.7)) as λ f )x, y) := ρ y fx) ) K x) for x 0, 1] d and y R. We ten introduce te crucial quantity EP n λ Vλ) := f )] 2 + ρ K ln 2 n)/ ) n 2 EP n λ f. 2.9) ) We call it nonasymptotic variance, since it plays te role of te variance in te risk bounds in te teorems below. From Condition 1 and Definitions 1 and 2, we conclude tat Vλ) <. Te term ρ ln K 2 n) n depends on and n. However, te bandwidt is typically cosen suc tat n for n so tat tis term vanises asymptotically. Additionally, besides te normalization in front of te first term, a dependence on is given troug λ. We will discuss tis after giving te main result of tis section. If = 1,...,1) parametric case), te nonasymptotic variance Vλ) tends towards te asymptotic variance AVλ) defined in 1.1) as n. Te main result of tis section reads as te following. Teorem 1. Let λ be as in 2.4), n {1, 2,...}, and 0, 1] d suc tat Condition 1 is satisfied. Ten, for all q 1, E fˆ λ x 0 ) f x 0 ) q C q b F) H 1/2 F u) du + 4H ] ) F 1) Vλ) q ln 2 n) q Mq n n 2 for a constant C q C q = 4q P 68 q Gammaq) works, were Gamma ) is te classical Gamma function). Te proof can be easily deduced integrating te result of Proposition 3 and using Proposition 2 te propositions can be found in te Appendix, see also te more detailed arxiv version). Remark 1. In contrast to Huber s asymptotic results see 14] and also 2,30 33]), te above teorem olds for finite but sufficiently large) sample sizes n. We note tat te desired variance term Vλ) is found up to constants, wic are of minor interest for tis paper. Moreover, a wide range of designs including degenerate designs, e.g.) and noise levels including zero noise, e.g.) is covered. Let us compare tis result to 7]: assume tat d = 1 and te noise σ X i )ξ i ) i is identically and independently normal distributed wit variance σ>0, and consider te local Huber estimator wit ρ ) = ρ H,lnn) ) 2.2), were γ = lnn), and te indicator kernel K ) = 1 1/2,1/2] ). As we mentioned above, te Huber estimator, wit a large parameter γ, mimics

8 A fully adaptive pointwise M-estimator 1567 te local least squares estimator. Indeed it olds Vλ) σ n n. x 0 +/2 x 0 /2 μx) dx Te term on te rigt-and side is te classical standard deviation of te local least squares estimator. Teorem 1 ten implies te results of 7], Teorem 1 and Proposition 1, in te Gaussian case and extends tem to eteroscedastic, eavy-tailed noises. Remark 2. Wile te above bound is to te best of our knowledge already a new result, te final goal is to provide a specific λ tat minimizes tis bound since te second term 2 q M q /n 2 is neglectable and since te bias term b F) is independent of λ. However, te bandwidt, wic accounts for te smootness of te target function, is included in Vλ). Tis makes simultaneous D- and S-adaptation difficult. Te specific dependences of te numerator and te denominator on can be deduced from EP n λ f )] 2 ρ = μx)k 2 x) σx)z )] 2 n 1 g i z) dz dx 2.10) i and EP n λ f ) = μx)k x) ρ σx)z ) n 1 i g i z) dz dx. 2.11) We study tis in detail in te following section for tree examples. Discussion of Condition 1. Te condition > 0 is fulfilled in many examples. Indeed, wit a cange of variables and by te definition of K ), we obtain 1 n = E U Xi x 0 ) ) U Xi x 0 ρ ] ) σx i )ξ i K X) Ux)U x)μx 0 + x)kx) ρ σx 0 + x)z ) 1 n g i z) dz dx. According to 34], Lemma 1.6, a sufficient condition for > 0 is tus tat μx 0 + x)kx) ρ σx 0 + x)z ) 1 g i z) dz>0 2.12) n for all x in some set in te kernel support wit positive Lebesgue measure. Recall tat μx 0 + ) is positive almost everywere in te support of K ) since μ ) vanises only at finitely many points. Te condition > 0 is tus fulfilled if inf x V ρ σx)z ) 1 n g i z) dz> )

9 1568 M. Cicignoud and J. Lederer Tis condition is satisfied, for example, for all densities g i ) and bounded σ ) if te contrast function is strictly convex. Tis olds true for ρ arc,γ ) see 2.3)). Te Huber contrast ρ H,γ ) see 2.2)), owever, is strictly convex on te interval γ,γ) only. It olds tat ρ H,γ ) = 1 γ,γ] ); terefore, te densities g i ) ave to satisfy te additional constraint inf x V 1 γ,γ] σx)z ) 1 n g i z) dz>0 to ensure > 0 in tis case. If we assume, for simplicity, tat te noise level is constant σ ) σ>0, te last constraint simplifies to γ/σ γ/σ n 1 n g i z) dz >0. So even for te Huber contrast wit a fixed γ>0, te assumption > 0 is weaker tan te standard assumption in te literature of g i ) being positive and continuous in te origin for all i {1, 2,...,n}. For te oter crucial part of te condition, we first note tat for 0, te quantity on te rigt-and side of 2.8) tends to a positive constant if σ ) is continuous in x 0. Indeed, since ρ is P-continuous, it olds tat 1 inf x V n E ρ σx)ξ i )] 1 n E ρ )] σx 0 )ξ i 2.14) as 0. Similarly as above, te quantity on te rigt-and side can be lower bounded by a positive constant for many contrasts and noise densities. We now give te rate for te quantity δ. It olds tat δ 1 E K X) ] E K 2 b F) + X) lnn) + K ] lnn), n n wic sould be cf. 2.8)) bounded by a constant. Here, indicates te asymptotic dependence on n. Te above display corresponds to a so-called bias variance decomposition up to a factor lnn). We also note tat if f is continuous as assumed in te standard literature, te bias term tends to zero as 0. In te literature, one typically cooses some couple of positive constants α 1,α 2 ), and σ ) and some bandwidt = 1,..., d ) suc tat n α 1/d lnn) ) α 2 j lnn) ) 1 for all j = 1,...,d, 2.15) were we assume tat n is sufficiently large suc tat te above inequalities can old. For appropriate α 1,α 2 ), Condition 1 is ten satisfied for n sufficiently large in many examples. Example 1. If, for example, te design is uniform μ ) 1) and te noise level omoscedastic σ ) σ>0), it olds tat const and tus δ b F) + lnn)/ n. Coosing a bandwidt = n as in 2.15) wit α 1 = 1 and α 2 = 4, Condition 2.8) is satisfied for n sufficiently large.

10 A fully adaptive pointwise M-estimator 1569 Example 2. For degenerated designs, owever, it is possible tat 0as 0. For example, let d = 1, te noise level be omoscedastic σ ) σ>0), and s + 1 μ ) = x s x 0 ) x 0 s 1 0,1] ) 2.16) s+1 wit s> 1 and x 0 0, 1] see 7]). Te density explodes for s<0) or vanises for s>0) at x 0, so tat one will eiter ave a lot or very little observations in te vicinity of x 0.Tisis reflected in δ recall tat d = 1 and tus 0, 1]): δ b F) + lnn) n s+1. So, similarly as above, one may coose a bandwidt like in 2.15) wit α 1 = 1/s + 1) and α 2 = 4/s + 1). We finally note tat te concrete form of Condition 1 is due to te application of deviation inequalities for bounded empirical processes. Similarly, we could relax te boundedness condition on te empirical processes involved to Bernstein conditions see, e.g., 35]). Tis allows to incorporate unbounded contrasts suc as te least squares contrast and te factor ρ in Condition 1 sould be replaced by te factor Eρ σ X)ξ)] 2, were ξ would be a sub-gaussian random variable S-minimax results In tis section, we deduce some corollaries adapted to simple examples from te above results. To start, we recall te notion of S-minimaxity. To tis end, let fx 0 ) be an estimator of f x 0 ) and S a set of functions. For any q>0, we define te maximal risk of f and te S-minimax risk for x 0 and S as R n,q f,s]:= sup E f S fx 0 ) f x 0 ) q and R n,q S]:=inf f R n,q f,s], 2.17) respectively. Te infimum on te rigt-and side is taken over all estimators. We can now define te S-minimax rates of convergence and te asymptotic) S-minimax estimators: Definition 3. A sequence φ n is an S-minimax rate of convergence, and te estimator ˆ f is an asymptotic) S-minimax estimator wit respect to te set S if 0 < lim inf n φ q n R n,q S] lim sup n φn q R n,q f,s] ˆ <. We can give some simple examples for one dimensional target functions, tat is, d = 1. We call H 1 β,l,m) Hölder space, wit parameters β,l,m > 0, te set of β -times differentiable functions f : 0, 1] R suc tat f j) M for all j {0, 1,..., β } and satisfied te Hölder continuity f β ) x) f β ) y) L x y β β for all x,y 0, 1] d. Te following corollary can now be easily deduced from Teorem 1.

11 1570 M. Cicignoud and J. Lederer Corollary 1. Consider te model in Example 1, tat is, uniform design μ ) 1) and omoscedastic noise level σ ) σ>0). Let β,l and M be positive parameters. Moreover, let fˆ λ be defined as in 2.4) wit m = β, n 1/2β+1), ρ ) = ρ arc,1 ) as in 2.3), and K ) := 1 1/2,1/2] ). Ten, it olds tat and EP n λ f )] 2 = 1 n EP n λ f ) = 1 n E ρ arc,1 σ ξ i) ] 2, Eρ arc,1 σ ξ i) lim sup n qβ/2β+1) R n,q f ˆλ, H 1 β,l,m) ) <. n Te rate n β/2β+1) is a standard S-minimax rate in te context Gaussian noise see 34], Capter 2). Here, owever, tis rate is acieved for a large class of noise distributions. Similarly, one can deduce te next corollary. Corollary 2. Consider te model in Example 2, tat is, a degenerate design as in 2.16) wit s> 1 and a omoscedastic noise level σ ) σ>0). Let β,l, and M be positive parameters. Moreover, let fˆ λ be defined as in 2.4) wit m = β, n 1/2β+s+1), ρ ) = ρ arc,1 ) as in 2.3), and K ) := 1 { 1/2,1/2} ). Ten, it olds tat and EP n λ f )] 2 = EP n λ f ) = s x s x 0 ) s+1 1 n s x s x 0 ) s+1 1 n E ρ arc,1 σ ξ i) ] 2, Eρ arc,1 σ ξ i) lim sup n qβ/2β+s+1) R n,q f ˆλ, H 1 β,l,m) ) <. n Tus, te rate n β/2β+s+1) is acieved. Tis rate is S-minimax in te nonparametric regression wit omoscedastic Gaussian noise see 7]). Note tat we ave only considered examples wit omoscedastic noises ere. For eteroscedastic noises, te dependence on can be very involved for some contrast functions cf. equations 2.10) and 2.11)). But, as igligted by te next example, tis is not always te case. Corollary 3. Consider te model 2.1) wit d = 1, a degenerate design as in 2.16) wit s> 1, a eteroscedastic noise level σ ) x 0 α, 0 α s/2, and a noise ξ i ) i wit finite variance.

12 A fully adaptive pointwise M-estimator 1571 Let β,l and M be positive parameters. Moreover, let ˆ f λ be defined as in 2.4) wit m = β, n 1/2β+s 2α+1), ρ ) = ρ H,lnn) ) as in 2.2), and K ) := 1 { 1/2,1/2} ). Ten, it olds tat EP n λ f )] 2 s+2α, EP n λ f ) s and lim sup n qβ/2β+s 2α+1) R n,q f ˆλ, H 1 β,l,m) ) <. n Tis result illustrates te effect of small noise levels on te rate and te possible compensations to degenerate unfavorable) designs. In particular, if α = s/2, we get te standard minimax rate n β/2β+1) as in Corollary 1. We assume tat α is smaller tan s/2, since oterwise te noise level is very small and te bandwidt cosen is tus as small as possible. We also recall tat te noise level is assumed to be bounded so tat we only consider te case α A D-adaptive estimator for fixed bandwidts In tis section, we discuss te selection of te combined function λ, tat is, of te kernel and te contrast. For tis, we introduce an oracle tat minimizes te bound in Teorem 1 above and ten provide an estimator tat mimics tis oracle. Tis estimator is ten D-adaptive, tat is, adaptive wit respect to te noise and te design distributions. To tis end, we first introduce := ϒ K as te set of possible combined functions λ as in 2.4) for a given set of contrasts ϒ, a given set of kernels K, and a fixed bandwidt 0, 1] d. For example, one may consider a subset of te set of Huber functions indexed by te scale γ>0 as set of contrasts ϒ := {ρ H,γ ) : γ>0}. An example for te set of kernels is te set of indicator functions wit different supports as K := { 1 Su) ) : u 1/2, 1/2] d} for Su) := 1/2 + u 1, 1/2 + u 1 ] 1/2 + u d, 1/2 + u d ]. Tis contains, in particular, te symmetric indicator kernel 1 S0) ). In tis section, te bandwidt is fixed so tat te bias term b F) in Teorem 1 is of minor importance; we ten introduce te oracle as te minimizer of te variance 2.9) To mimic te oracle λ, we propose te estimator λ λ := arg min Vλ). 3.1) λ λ := arg min Vλ), λ P n λ were Vλ) fˆ λ )] 2 + ρ K ln 2 n)/ ) n 2 :=. P n λ fˆ λ ) 3.2)

13 1572 M. Cicignoud and J. Lederer Note tat we estimate te target function f by fˆ λ and EP n λ f )] 2 and EP n λ f ) by teir empirical versions P n λ fˆ λ )] 2 and P n λ fˆ λ ), respectively. Te explicit expressions for te numerator and te denominator can be obtained using P n λ ˆ f λ ) ] 2 = 1 n P n λ ˆ f λ ) = 1 n K 2 X i) ρ Y i ˆ f λ X i ) )] 2 K X i )ρ Y i ˆ f λ X i ) ). We now sow tat te estimator ˆ f λ tat results from 2.4) and 3.2) performs up to constants as well as te oracle fˆ λ. For tis, we define H F ) as te entropy wit bracketing of F wit respect to te pseudo)metric EP n κf1,λ 1 ) κf 2,λ 2 ) ] 2 EP n λ 1 f 1) λ 2 f 2) ] 2 3.3) for any f 1,f 2 F,λ 1,λ 2, were κf,λ) := λ f )/ EP n λ f )] 2 + ρ K ln 2 n)/ n ). We compute in te Appendix a bound for tis entropy for te set of Huer contrasts indexed by te scale. Before giving te main result of tis section, we give te necessary assumptions. Condition 2. Let = ϒ K be a set of functions as in 2.4) were ϒ is a set of contrasts as in Definition 2 and K is a set of kernels as in Definition 1, n {1, 2,...}, and 0, 1] d. We say tat Condition 2 is satisfied if te smallest eigenvalue defined in Condition 1) is positive, n ln 4 n), and, defining for any λ δ 2 P 2 λ) := E K X) ] b F) + it olds for all λ 4 b F) + δ λ)) 1 inf x V n and 54 ρ E K 2X)]+ K / n ) n lnn P ) H 1/2 F u) du + H F 1)) 1 Condition 3. Additionally, we say tat Condition 3 is satisfied if, defining s λ) := 1 2 K ) δ λ) + b F) ] it olds for all λ ], E ρ )] σx)ξ i. 3.4) K ρ 2 ) n ln n P ) + s λ) 1 0 ) H 1/2 F u) du + H F 1), 1 2 K min { EP n λ f ), EP n λ f )] 2}. 3.5)

14 A fully adaptive pointwise M-estimator 1573 We discuss te above conditions after te following result. Teorem 2. Let be a set of functions as in 2.4), n {1, 2,...}, and 0, 1] d suc tat Conditions 2 and 3 are satisfied. Ten, for all q 1, E ˆ f λ x 0) f x 0 ) q T q b F) H 1/2 F u) du + 4H ] F 1) Vλ ln ) ) q + 52M)q n) n n 2 for a constant T q T q = 2q P 117 q Gammaq) works, were Gamma ) is te classical Gamma function). Remark 3. Apart from te given assumptions, te estimator ˆ f λ x 0) does not premise knowledge about te noise level σ ) and te densities g i ) and μ ) but acieves up to constants te optimal variance term Vλ ) for all suc functions. Te estimator is tus called D-adaptive optimal wit respect to te set ). For example, for te Huber contrast 2.2) indexed by te scale γ, ϒ := {ρ H,γ ) : γ>0}, te estimator is D-adaptive minimax Huber minimax) for te set of contamination models, see Section A.1 in te arxiv version. Finally, we mention tat appropriate coices of te bandwidt in te above result lead to S-minimax results. Discussion of Conditions 2 and 3. Condition 2 limits te possible sets of combined functions and tus, in particular, te sets of possible contrast functions ϒ. It demands tat all possible combined functions λ fulfill Condition 1, wic ten leads to consistent estimators see Proposition 1) and to sets of contrast wit finite entropy. Condition 2 demands, in particular, tat te rigt-and side of 3.4) is positive and, since te rigt-and side of 3.4) is upper bounded by 1, tat sup ρ ϒ ρ does not increase too rapidly wit n. In te following, we illustrate tese restrictions wit an example. We consider a omoscedastic model σ ) σ>0) and ϒ equal to a set of Huber contrasts ρ H,γ ) as in 2.2) wit scale parameter γ γ,γ + ], γ + γ > 0. It olds tat sup ρ ϒ ρ H,γ = γ +. Tis implies tat γ + must not increase too rapidly wit n. Moreover, it must old tat 1 n E ρ H,γ σ ξ i) ] γ/σ 1 = γ/σ n γ /σ 1 g i z) dz γ /σ n g i z) dz>0. For noise densities tat are positive and continuous in te origin, tis condition is verified for all γ > 0. For more involved noise densities vanised at te origin), owever, γ as to be cosen sufficiently large. Condition 3 is similar to Condition 2 since s λ) δ λ). However, te terms in te minimum on te rigt-and side of 3.5), can be small for a certain design and noise level. Te second term, vanises if σ ) 0 since ρ 0) = 0. Moreover, if te design degenerates as in 2.16)) wit alarges, EP n λ f ) and EP n λ f )] 2 ten tend to zero faster tan s λ) as n cf. 2.10) and 2.11)). Tis is due to te estimation of EP n λ f ) and EP n λ f )] 2 : if σ ) 0 or if te design degenerates, te above terms are small cf. 2.9)), and tus, te estimation error of tem wic is related to s )) obstructs teir beavior.

15 1574 M. Cicignoud and J. Lederer 4. A D-adaptive and S-adaptive estimator In tis section, we introduce an estimator of f x 0 ) tat is simultaneously S- and D-adaptive. For tis, we apply te data-driven procedure introduced above to select te contrast and te kernel and a modification of te data-driven Lepski s metod to select te bandwidt. In te first part, we consider isotropic, locally polynomial target functions, in te second part anisotropic, locally constant functions. To simplify te exposition, we present asymptotic results only. Te LPA is designed for functions tat can be locally approximated by polynomials. Tis is, for example, te case for Hölder classes, wic we define similarly as in 3]) as Definition 4. Let β := β 1,...,β d ) ]0, + d suc tat β 1 = = β d =: β, and let L,M > 0. Te function s : 0, 1] d M,M] belongs to te anisotropic Hölder Class H d β,l,m) if for all x,x 0 0, 1] d p S β x 0,1] d sx) Ps)x x 0 ) L sup p sx) x p 1 1 x p d d M, d x j x 0,j β j and were Ps)x x 0 ) is te Taylor polynomial of s of order β at x 0, and x j and x 0,j are te jt components of x and x 0, respectively. Te parameter β is usually unknown; tus, it is desirable to ave an estimator tat is adaptive wit respect to β. Tis motivates te following definition, were := {ψ n β)} β M is a given family of normalizations for a set of parameters M: Definition 5. Te family is called admissible if tere exists an estimator ˆ f n suc tat lim sup n sup β M j=1 ψn q β)r n,q f ˆn, H d β,l,m) ) <. Te estimator ˆ f n is ten called -adaptive in te S-minimax sense. We distinguis two cases in te following: First, we consider te special case of isotropic Hölder classes, tat is, β 1 = =β d. Tese classes only require a common bandwidt for all dimensions tat is cosen wit te standard version of Lepski s metod see 24] and 23]). Afterwards, we allow for anisotropic Hölder classes. Tese classes necessitate a separate bandwidt for every dimension of te domain under consideration. Te standard version of Lepski s metod is not applicable in tis case, because it requires a monotonous bias. We circumvent tis problem using a modified version of Lepski s metod as described in 19] and 22].

16 A fully adaptive pointwise M-estimator A fully adaptive estimator for isotropic, locally polynomial functions Here, we consider isotropic Hölder classes wit β 0,m+ 1], were m is te degree of te estimator fˆ λ and may be cosen arbitrarily large. Terefore, only one bandwidt iso = 1 = = d > 0 as to be selected. Geometrically, tis means tat we select a ypercube in R d wit edge lengt iso as domain of interest in contrast to te anisotropic case, were we select a yperrectangle wit edge lengts 1,..., d ). A major issue is te coice of te bandwidt. In te following, we assume tat te variance term Vλ iso )/n d iso ) for see Definitions 2.4) and 2.9)) λ iso f )x, y) := ρ y fx) ) K iso x) for all x 0, 1] d,y R, is decreasing in te bandwidt so tat we can apply Lepski s metod. Tis imposes an additional restriction on te design and te noise. After te main result of tis section, we give some examples for designs and noises tat fulfill tis restriction. Next, we introduce te set of bandwidts H iso :=, + ], were 0 < < + < 1 are defined as cf. 2.15)) := ln6/d n) n 1/d and + := 1 lnn). 4.1) Since te inequality < + as to be satisfied, n is required to be large enoug. We ten introduce te isotropic M-estimator for any iso H iso as were fˆ iso iso := arg min P n λ iso f ), f F λ iso = arg min Vλ iso ) λ iso and V ) is defined in 2.9). Eventually, we introduce a net Hɛ iso := { iso H iso, m N : iso = + ɛ m },ɛ 0, 1), suc tat 1 Hɛ iso n and ten apply Lepski s metod for isotropic functions see 24] and 23]) to define te data-driven bandwidt ĥ iso : { ĥ iso := max iso H iso ɛ : fˆ iso for all iso Hiso ɛ iso x 0) fˆ iso iso x 0) 15 2 B + isoɛ n) ) V λ ) iso n, iso )d } 4.2) suc tat iso iso, were iso ɛ n) := 11 lnn Hɛ iso ) and B := H 1/2 F u) du + 4H F 1). ln 2 n) We now obtain on isotropic Hölder classes H iso d β,l,m) := H dβ,..., β), L, M), for all β,l,m > 0 te following result:

17 1576 M. Cicignoud and J. Lederer Teorem 3. Let be a set of combined functions as in 2.4) and n {1, 2,...} suc tat Conditions 2 and 3 are satisfied for all iso H iso. Ten, for any x 0 0, 1) d, any β 0,m+ 1], and any L>0, tere exists a universal positive constant C>0 suc tat R n,q f ˆ ĥiso iso x 0), H iso d β,l,m)] C inf iso H iso { Ld β iso + iso ɛn) Vλ ) } q iso as n. n d iso Remark 4. Tis oracle inequality like result sows te simultaneous S- and D-adaptation of te estimator. It generalizes results in 5], wic rely on te asymptotic equivalence of te block median metod, in two important aspects: First, it allows for eteroscedastic regression models wit random designs. Second, it does not require tat te noise densities are positive at teir median and tus allows for a wider range densities. Finally, we note tat Lepski s metod as been used for locally constant M-estimators in 27] but to te best of our knowledge never to locally polynomial M-estimators as it is done ere. Remark 5. If only S-adaptation is considered, te conditions on n can be considerably relaxed. In fact, assuming tat Vλ ) is known, te estimator fˆ λ of 2.4) can be applied instead of fˆ ĥiso iso iso, were iso is selected from 4.2) replacing V λ ) by Vλ iso ). Te Conditions 2 and 3 can ten iso be replaced by Condition 1. Remark 6. Te variance term is decreasing for settings wit indicator kernels and omoscedastic noise levels as one can ceck easily starting from 2.9)); for settings wit indicator kernels, Huber contrasts, σ ) = 1 + x 0 α for a α 0, 1/2], and d = 1; and for many oter settings. On te contrary, te variance term can be increasing, for example, if te noise level is symmetric in x 0 and convex. Corollary 4. Consider te model in Example 1 in te previous section wit μ ) 1uniform design) and σ ) 1omoscdastic noise level). For any β 0,m+ 1] and any L>0, it olds tat ) n qβ/2β+d) lim sup R n,q f ˆ ĥiso n lnn) iso x 0), H iso d β,l,m)] <. Tis corollary can be deduced minimizing te term on te rigt-and side of te last teorem wit a standard bias/variance trade-off. Remark 7. Te rate lnn)/n) β/2β+1) in te above corollary is admissible cf. Definition 5) over isotropic Hölder spaces and is asymptotically optimal see 4] and 24]) up to te logaritm lnn), wic is te usual price for te adapativity see Section 5 for more details). Moreover, te approac used to deduce te above corollary presumes uniform designs and omescedastic noises; owever, more elaborate approaces, peraps similar to te ones in 8], may lead to comparable results for degenerate designs.

18 A fully adaptive pointwise M-estimator A fully adaptive estimator for anisotropic, locally constant functions In tis part, we allow for anisotropic Hölder classes and bandwidts. In return, we restrict ourselves to locally constant functions, tat is, m = 0 and tus P =1) and F = M,M]. Moreover, we restrict ourselves to uniform designs μ ) 1) and omoscedastic σ ) σ 0) and identically distributed noise g i ) g ) for all i = 1,...,n). For tis setting, we introduce an S- and D-adaptive estimator of f x 0 ). Te main properties of tis estimator are given in Teorem 4. We introduce an estimator for eac bandwidt in te set H :=, + ] d, were and + are defined in te previous section. For tis, we define te variance term as ρ σ z)] 2 gz)dz + ρ K ln 2 n)/ n d ) 2 Vρ, K) := ρ, 4.3) σz)gz) dz and te oracle for a set of contrasts ϒ and a set of kernels K as ρ,k ) := arg min Vρ, K). 4.4) ρ ϒ,K K Next, we introduce an estimator of te variance term as 1/n) n ρ Y i fˆ λ + X i ))] 2 + ρ K ln 2 n)/ Vρ, K) := 1/n) n ρ Y i fˆ λ + X i )) n d ) 2, 4.5) were ˆ f λ + is defined in 2.4) wit λ = λ +f )x, y) := ρy f x))k +x), and an estimator of te oracle as ˆρ, ˆK) := arg min Vρ, K). 4.6) ρ ϒ,K K We stress tat te variance term V, te oracle ρ,k ), and teir estimators V and ˆρ, ˆK) are independent of te bandwidt. We can finally introduce te desired estimator ˆ f for all H: ˆ f := arg min f F n 1 i ˆρ Y i fx i ) ) ˆK X i ). 4.7) Te crucial step is now te coice of te bandwidt wit a modified version of Lepski s metod see 19] and 20]). First, we define for all a,b R te scalar a b := maxa, b) and for all, H te vector := 1 1,..., d d ). We ten consider te two families of Locally Constant Approximation LCA) estimators provoked by 4.7)) { ˆ f } H and { ˆ f, := ˆ f }, H 2. Note tat fˆ, = fˆ, commutativity). Similarly as above, we ten introduce a net H ɛ := {,..., )} { H: j = 1,...,d m j N : j = + ɛ m j },ɛ 0, 1), suc tat H ɛ n

19 1578 M. Cicignoud and J. Lederer and set ani ɛ n) := 11 lnn H ɛ ). We finally select te bandwidt according to { ĥ := max H ɛ : fˆ, x 0 ) fˆ x 0 ) 16 B + aniɛ n) ) V ˆρ, ˆK) n } for all H ɛ suc tat. 4.8) Te maximum is taken wit respect to te order wic we define as d j=1 j dj=1 j. Note, in particular, tat te rigt-and side of 4.8) is decreasing wit respect to tis order. Te above coice of te bandwidt leads to te estimator fˆ ĥ wit te following properties: Teorem 4. Let be a set of combined functions as in 2.4) and let n {1, 2,...} suc tat Conditions 2 and 3 are satisfied for all H. Ten, for any x 0 0, 1) d, any β 0, 1] d, and any L>0, tere exists a universal constant C suc tat R n,q ˆ f ĥx 0 ), H d β,l,m) ] C inf H { L d j=1 β j j + ani ɛ n) } q Vρ,K ). n We can also derive te following corollary from Teorem 4 via a bias/variance trade-off. Corollary 5. For any β 0, 1] d and any L>0, it olds tat ) n q β/2 β+1) lim sup R n,q f ˆ ĥx 0 ), H d β,l,m) ] <, n lnn) were β := j 1/β j ) 1 is te armonic average. Remark 8. In contrast to te previous part, only locally constant functions are considered ere, wic is due to te bias term cf. Lemma 6). To te best of our knowledge, te presented coice of te bandwidt is te first application of te anisotropic Lepski s principle 22], see also 11,19, 20]) for te selection of an anisotropic bandwidt for nonlinear M-estimators. We also note tat, comparing te adaptive rate lnn)/n) β/2 β+1) wit te optimal rate in te wite noise model see 20]), for example, one finds tat tis rate is nearly optimal. We finally refer to te remarks after Teorem Discussion Let us detail on te assumptions and restrictions and igligt some open problems: 1. Instead of assuming tat te densities g i ) are symmetric cf. 14,29]), it is sufficient tat te sum i g i ) is symmetric. We are, owever, not aware of examples were tis generalization is relevant.

20 A fully adaptive pointwise M-estimator Te variance of te median estimator is 1/4g 2 0)) wic implies a strong sensitive to te noise density at 0. Moreover, te estimation of g0) see 5], e.g.) requires many observations near fx 0 ) in practice. On te contrary, Huber contrast wit scale γ allowed by our approac), te denominator of te variance term 2.9) depends on te mass of te noise density on te interval γ,γ] instead of te mass at To estimate te variance term 2.9), we plug an estimate Y i fˆ λ of te residuals. Condition 2 ensures te consistency of all estimators in. Tis is considerably restrictive on te initial family. Tis problem can be circumvented using a pre-estimator e.g., wit te contrast 2.3)) instead of fˆ λ for te estimation of te variance. 4. Lepski s metod is very sensitive to outliers see 27]). To complement it wit te adaptive robustness of te estimator via te minimization of te variance term can tus be interesting for many applications. 5. Te variance term and its empirical version do not depend on te bias term see Teorem 1, Definition 3.2) and Remark 2) and, more generally, not on te specific model. Te procedure presented in tis paper may tus be interesting for oter models, suc as ig dimensional settings cf. 21]), for example. 6. Te quantity 15 2B + iso ɛ n)) in te tresold term in 4.2) contains te factor lnn) and known but large constants. For applications, it sould usually be cosen considerably smaller see 23]) and can probably be tuned wit te propagation metod 28], for example. 7. As mentioned in te Introduction, Lepski-type procedures are also useful to get S-adaptive confident bands see 10] and references terein). Tis requires deviation inequalities tat can be derived along te presented lines see Proposition 3) but also a lower bound for te bias term of te estimator cf. 10], Condition 3, Section 3.2 and Section 3.5 for Discussion), wic seems not to be available ere, since robust M-estimators and tus te bias term do not ave explicit expressions. For our purposes, we circumvent tis issue by using te bias term of te criterions s derivative as an estimator of te expected criterion s derivative, see Lemmas 1 and 2. However, tis way, we only obtain an upper bound. We terefore suggest to establis first S-adaptive confidence bands for te criterion s derivative viewed as an estimator and ten, using te smootness of te contrast, confidence bands wit respect to a pointwise semi-norm or sup-norm. 6. Proofs of te main results Let us introduce some additional notation to simplify te exposition. For tis, we introduce F δ := { f = P t F : t t 0 l1 δ } 6.1) as a ball in F wit radius δ>0 centered at f 0. Furtermore, we denote te column vector of partial derivatives of te criterion P n λ ) defined in 2.4)) by D λ P t ) := ) P n λp t ) for all t R P, 6.2) t p p P

21 1580 M. Cicignoud and J. Lederer and te parametric expectation wit respect to te distribution E 0 of X, f 0 X) + σx)ξ) by E 0 D λ ) ]. 6.3) Next, for all t R P, we introduce te Jacobian matrix J D of E 0 D λ ] as JD P t ) ) p,q P := E 0 D p λ t P t) ] ) q p,q P = E 0 ]) P n λp t ), 6.4) t q t p p,q P were D p λ ) is te pt component of D λ ). Te Jacobian matrix exists according to Definition 2 and Fubini s teorem. Furtermore, te sup-norm on R P is denoted by l, and te vector of coefficients of te estimated polynomial fˆ λ is denoted by ˆt λ. Moreover, we set c λ := EP n λ f ) 6.5) and b := b F). We finally define λ := ρ K and for any z 0 B z := H 1/2 F u) du + 4H F 1) ln z + 2z n) ln 2 n). 6.6) 6.1. Auxiliary results Te following propositions are basic for te proofs of te main results. Te proofs of te propositions are given in te Appendix. Proposition 1. Let = ϒ K be a set of functions as in 2.4) were ϒ is a set of contrasts as in Definition 2 and K is a set of kernels as in Definition 1. Let n {1, 2,...} and 0, 1] d be suc tat Condition 2 is satisfied. Ten, P λ { fˆ λ F δ λ)}) 1 n 2, were δ ) is defined in Condition 2. Te following proposition allows us to control te deviations of te process D λ ). Proposition 2. For any z 0, it olds tat P sup sup λ f F δ λ) were B z is defined in 6.6). D λ f ) E D λ f )] l EP n λ f )] 2 + λ ln2 n)/ n B z n ) 2 P exp z), Tis proposition is directly deduced from Massart s Inequality see te arxiv version for details).

22 A fully adaptive pointwise M-estimator 1581 Proposition 3. Let be a set of functions as in 2.4), n {1, 2,...}, and 0, 1] d be suc tat Condition 2 is satisfied. Ten, for any z 0, it olds tat { P sup f ˆλ x 0 ) f x 0 ) ] } Vλ)Bz 2 3b ) fˆ λ F δ λ n λ)} 2 P exp z). λ { We note tat te constants 2 and 3 can be replaced by o1). Proposition 4. Let = ϒ K be a set of functions as in 2.4) were ϒ is a set of contrasts as in Definition 2 and K is a set of kernels as in Definition 1. Let n {1, 2,...} and 0, 1] d be suc tat Condition 2 is satisfied. Ten, P ) 1 5/n 2, were := λ { Vλ) 2 3 Vλ), 6 Vλ)]}. We note tat te constants 2/3 and 6 can be replaced by o1) Proof of Teorem 2 First, we set := λ { Vλ) 2 3 Vλ), 6 Vλ)]}. Ten, we observe tat, since ˆ F, sup f F fx 0 ) M, and f x 0 ) M, te risk can be bounded by E ˆ fˆλ x 0) f x 0 ) q = E ˆ fˆλ x 0) f x 0 ) q 1 + E ˆ fˆλ x 0) f x 0 ) q 1 c E ˆ fˆλ x 0) f x 0 ) q 1 + 2M) q P c). Using Proposition 4, Lemma 3, te last inequality, and simple computations, we obtain E ˆ fˆλ x 0) f x 0 ) q E ˆ fˆλ x 0) f x 0 ) q M) q /n 2 fˆλ 2 E q ˆ fˆλ x 0) f x 0 ) 3b 6 3Vλ ) )B q 0 6.7) n q 3b + 6 3Vλ ) )B q M) q /n 2. n Let us now bound te first term on te rigt-and side of te last inequality. To do so, we note tat on te event V λ ) Vλ ) Vˆλ) Vˆλ) ) Using te last inequality and integrating te result of Proposition 3 wit ε = 10 z + get for more details see te arxiv version) E ˆ fˆλ x 0) f x 0 ) 3b 6 3Vλ ) )B q 0 Vλ 1 T q b + ) )B q 0. n + n 2z ln 2 n),we

23 1582 M. Cicignoud and J. Lederer From 6.7) and te last inequality, te teorem can be deduced Proof of Teorem 3 For ease of exposition, we set B 0 = B cf. 6.6)), k := iso, and ˆk := ĥ iso. Ten, one may verify tat te oracle bandwidt { k := arg min Ldk β + c B 0 + iso ɛ n) ) Vλ k ) } k H iso nk d is well defined, were c is a constant cosen suc tat bot terms are equal at te point k.next, from Propositions 1 and 4 wit = k,...,k), it follows tat P k Hɛ iso, λ k : fˆ iso k / F ) δk λ k) n 2 n 1 6.9) k H iso ɛ and k H iso ɛ P c ) k k Hɛ iso 5 n 2 5n 1, 6.10) were k := is defined in Proposition 4. Tus, we may restrict our considerations to te event k Hɛ iso,λ k { fˆ iso k F δk λ k)} k, since we are only interested in te asymptotic beavior. We now introduce kɛ Hiso ɛ suc tat kɛ k ɛ 1 kɛ. Control of te risk on te event {k ɛ ˆk} Wit te triangular inequality and Lemma 3, we obtain fˆ ˆk iso x 0 ) f x 0 ) q 1 k ɛ ˆk 2 q 1 fˆ ˆk iso x 0 ) fˆ k ɛ iso x 0) q 1 k ɛ ˆk + fˆ k ɛ iso x 0) f x 0 ) q ). 6.11) Te first term on te rigt-and side of te last inequality is controlled using te procedure 4.2) to obtain E fˆ ˆk iso x 0 ) fˆ k ɛ iso x 0) q ] V λ 1 k ɛ ˆk E 15 k )B ɛ 0 + iso ɛ n))] q 2. nk ɛ ) d On te event k Hɛ iso k,wegetsimilarlyasin6.8) E fˆ ˆk iso x 0 ) ˆ f k ɛ iso x 0) q 1 k ɛ ˆk ] 45 Vλ k )B 0 + iso ɛ n))) q ɛ 6. nk ɛ ) d