The Asymptotic Minimax Constant for Sup-Norm. Loss in Nonparametric Density Estimation

Transcription

1 The Asymptotic Minimax Constant or Sup-Norm Loss in Nonparametric Density Estimation ALEXANDER KOROSTELEV 1 and MICHAEL NUSSBAUM 2 1 Department o Mathematics, Wayne State University, Detroit, MI 48202, USA 2 Weierstrass Institute, Mohrenstr. 39, D Berlin, Germany March 1999 Abstract We develop the exact constant o the risk asymptotics in the uniorm norm or density estimation. This constant has irst been ound or nonparametric regression and or signal estimation in Gaussian white noise. Hölder classes or arbitrary smoothness index β > 0 on the unit interval are considered. The constant involves the value o an optimal recovery problem as in the white noise case, but in addition it depends on the maximum o densities in the unction class. Keywords: Density estimation, exact constant, optimal recovery, uniorm norm risk, white noise RUNNING TITLE: Asymptotic minimax density estimation To whom correspondence should be addressed. 1

2 1 Introduction and Main Result Recently in Korostelev (1993) an asymptotically minimax exact constant has been ound or loss in the uniorm norm, or Gaussian nonparametric regression when the parameter set is a Hölder unction class. This risk bound represents an analog o the now classical L 2 -minimax constant o Pinsker (1980) valid or a Sobolev unction class. Donoho (1994) extended Korostelev s (1993) result to signal estimation in Gaussian white noise and showed it to be related to nonstochastic optimal recovery. Here we consider density estimation rom i. i. d. data with a sup-norm loss. Consider a sample X 1,..., X n o i. i. d. observations having a probability density = (x) in the interval 0 x 1. Let β, L be some positive constants, and let Σ(β, L) be the class o densities Σ(β, L) = {g : 1 0 g = 1, g 0, and g β (x 1 ) g β (x 2 ) L x 1 x 2 β β, 0 x 1, x 2 1} where β is the greatest integer strictly less than β. Assume that the density belongs a priori to Σ(β, L). Consider an arbitrary estimator ˆ n = ˆ n (x) measurable w.r.t. the observations X 1,..., X n. We deine the discrepancy o ˆn (x) and the true density (x) by the sup norm ˆ n where = sup (x). 0 x 1 Denote by the probability distribution o the observations X 1,..., X n, and by E (n) the expectation w.r.t.. Let w(u), u 0, be a continuous increasing unction which admits a polynomial majorant w(u) W 0 (1 + u γ ) with some positive constants W 0, γ, and such that w(0) = 0. Introduce the minimax risk r n = r n (w( ); β, L, b) = in ˆ n sup Σ(β,L,b) 2 E (n) w(ψn 1 ˆ n ) (1)

3 where ψ n = ((log n)/n) β/(2β+1) is the optimal rate o convergence (c. Khasminskii (1978), Stone (1982), Ibragimov and Khasminskii (1982)). The main goal o this paper is to ind the exact asymptotics o the risk (1). To do this we need two additional deinitions. First, note that the densities in (β, L) are uniormly bounded, i. e. B = B (β, L) = max Σ(β,L) max (x) < +. (2) 0 x 1 An argument or this based on imbedding theorems, as well as urther inormation on the value o B is given in an appendix (section 4). Secondly, denote by Σ 0 (β, L) an auxiliary class o unctions on the whole real line: Σ 0 (β, L) = { : β (x 1 ) β (x 2 ) L x 1 x 2 β β, x 1, x 2 R 1 }. Let g 2 denote the L 2 -norm o g. Deine the constant { } A β = max g(0) g 2 1, g Σ 0 (β, 1). (3) Theorem. For any β > 0, L > 0 and or any loss unction w(u) the minimax risk (1) satisies: lim r n = w (C) n where ( ) 2B L 1/β β/(2β+1) C = C(β, L, B ) = A β, 2β + 1 and the constants B = B (β, L) and A β are deined by (2) and (3) respectively. The proo o the corresponding upper and lower asymptotic risk bounds is developed in sections 2 and 3. A more concise argument based on asymptotic equivalence o experiments in the Le Cam sense is possible (c. Nussbaum (1996)), but only in the case β > 1/2, and under an additional assumption that the densities are uniormly 3

4 bounded away rom 0. While asymptotic equivalence is known to ail or β 1/2 (c. Brown and Zhang (1998)), our method here yields the sup-norm constant or density estimation or all β > 0. The proo via asymptotic equivalence can be ound in the technical report (Korostelev and Nussbaum (1995)). 2 Upper Asymptotic Bound Let g be a solution o the extremal problem in (3), g Σ 0 (β, 1). The correctness o this deinition ollows rom Micchelli and Rivlin (1977), and, as shown by Leonov (1997), g has a compact support. Consider also the solution g 1 Σ 0 (β, 1) o the dual extremal problem I g is the solution o (3) then g 1 (u) = A 1 β min { g 1 2 g 1 (0) = 1, g 1 Σ 0 (β, 1)}. (4) g(a1/β β u) (c. section 2.2 o Donoho (1994)); hence g 1 2 = A (2β+1)/2β β. Since g is o compact support, so is g 1 ; let S be a constant such that g 1 (u) = 0 or u > S. Put K(u) = g 1 (u)/ g 1, u R 1 and choose the bandwidth h n = (Cψ n /L) 1/β. For an arbitrary small ixed ɛ > 0 deine regular grid points in the interval [0, 1] by x k = ɛkh n, k = 0,..., M, where M = M(n, ɛ) = (ɛh n ) 1 is assumed integer. Put M 0 = [S/ɛ] + 1, and introduce the kernel estimator n at the inner grid points n(x k ) = (nh n ) 1 n K((X i x k )/h n ), k = M 0,..., M M 0. i=1 4

5 Lemma 1. There exists a constant p 0 > 0 such that or any α > 0 the inequality holds sup ( max Σ(β,L) M 0 k M M 0 n(x k ) (x k ) (1 + α)cψ n ) p 0 M α. Proo. Deine the bias and stochastic terms by b nk = E (n) [ n(x k )] (x k ), and z nk = n(x k ) E (n) [ n(x k )]. For any α > 0 the ollowing inequalities are true: M M 0 k=m 0 ( max M 0 k M M 0 ( n(x k ) (x k )) (1 + α)cψ n ) = ( max M 0 k M M 0 (z nk + b nk ) (1 + α)cψ n ) ( max M 0 k M M 0 z nk (1 + α)cψ n (z nk (1 + α)cψ n sup Σ 0 (β,l), (0)=0 M M 0 k=m 0 max b nk ) M 0 k M M 0 h 1 n K(u/h n )(u) du ) (z nk (1 + α)cψ n (1 sup K(u)(u) du )) Σ 0 (β,1), (0)=0 where the standard renormalization technique applies (see Donoho (1994)). Deine K δ (u) = δ 2/(2β+1) g(δ 2/(2β+1) u)/ g or any δ > 0, where g is again the solution o (3). The optimal recovery identity (Micchelli and Rivlin (1977), Donoho (1994)) implies that sup sup Σ 0 (β,1) z 2 1 K δ (u)(u) du (0) + δ 5 K δ (u)z(u)du = δ2β/(2β+1) A β,

6 hence sup Σ 0 (β,1), (0)=0 A choice δ = A (2β+1)/2β β yields K δ (u) = A 1/β β K δ (u)(u) du + δ K δ 2 = δ 2β/(2β+1) A β. g(a1/β β u)/ g = g 1 (u)/ g 1 = K(u) and hence 1 sup K(u)(u) du = A (2β+1)/2β β K 2. Σ 0 (β,1), (0)=0 By urther calculation we obtain nh n /(B K 2 2) Cψ n A (2β+1)/2β 2 β K 2 = ( 2β + 1 and that or or any ɛ < 1 and any n, satisying log n)1/2 log n > (2β + 1)(log ɛ 1 + β 1 log(l/c)) we have 2 ( 2β + 1 log n)1/2 2 log M. Thus, the latter sum o probabilities can be estimated rom above by M M 0 k=m 0 ( nh n /(B K 2 2) z nk (1 + α) 2 log M). Note that where nh n /(B K 22) z nk = n 1/2 n i=1 ξ ik ξ ik = h n /(B K 2 2) (h 1 n K((X i x k )/h n ) E (n) [h 1 n K((X i x k )/h n )]), i = 1,..., n, k = M 0,..., M M 0. 6

7 The random variables ξ ik, i = 1,..., n, are independent or any ixed k, and E (n) [ξ ik ] = 0, V ar (n) [ξ ik ] = B 1 (x k ) + o n (1) 1 + o n (1) (5) where o n (1) 0 as n uniormly in i, k, and Σ(β, L). Moreover, or any integer m, m 3, the ollowing bounds hold: E (n) ξ ik m (h n /(B K 2 2)) m/2 (2B S 2 m H m /h m 1 n ) = 2B Sh n (λ/ h n ) m (6) where H = max u R 1 K(u) and λ = 2H / B K 2 2. The Chebyshev exponential inequality, known as Cherno s upper bound, yields ( max M 0 k M M 0 ( n(x k ) (x k )) (1 + α)cψ n ) M M 0 k=m 0 (n 1/2 n ξ in (1 + α) 2 log M) i=1 M exp ( c(1 + α) 2 log M)(E (n) [exp(cξ in / n)]) n. Using (5) and (6), we can estimate the moment generating unction as ollows: E (n) [exp(cξ in / n)] 1 + c2 2n V ar(n) [ξ ik ] + 1 m! ( c ) m E (n) m 3 n ξ ik m 1 + c2 2n (1 + o n(1)) + 2B Sλ 3 c 3 n 1 nh n m! ( λc ) m 3 nhn m c2 2n (1 + o n(1) + 4B Sλ 3 c nhn exp(λc/ nh n )) exp ( c2 2n (1 + o n(1))). (7) 7

8 The latter inequality is true or any c = o n ( nh n ) as n. I we choose c = 2 log M, then (7) implies that ( max M 0 k M M 0 ( n(x k ) (x k )) (1 + α)cψ n ) M exp ( 2(1 + α) log M) exp ((1 + o n (1)) log M) M exp ( (1 + α) log M) = M α or any n large enough. The probability o the random event { min M 0 k M M 0 ( n(x k ) (x k )) (1 + α)cψ n } admits the same upper bound, and this proves the lemma. To extend the deinition o n(x k ) to the grid points x k which are close to the endpoints o the interval [0,1], we take a kernel K 0 (u) with the support [0,1] satisying the orthogonality conditions 1 0 K 0 = 1, and 1 0 u j K 0 = 0, j = 1,..., β. Put n n(x k ) = (nκh n ) 1 K 0 ((X i x k )/(κh n )), k = 0,..., M 0 1 (8) i=1 where a small positive constant κ is chosen in Lemma 2 below. For the grid points x k [1 Sh n, 1] we deine n(x k ) = (nκh n ) 1 n K 0 ((x k X i )/(κh n )), k = M M 0 + 1,..., M. i=1 Put M = {0,..., M 0 1} {M M 0 + 1,..., M}. 8

9 Lemma 2. There exist constants p 0 and p 1 such that or any n and or any α > 0 the inequality holds sup Σ(β,L) ( max n(x k ) (x k ) (1 + α)cψ n ) p 0 M αp 1. (9) k M Proo. To prove (9), it suices to derive the upper bound or the probability ( max ( 0 k<m 0 n(x k ) (x k )) (1 + α)cψ n ) p 0 M αp 1. (10) The bias b nk o the estimator (8) at any point x k is O((κh n ) β ) as n (see Devroye and Györi (1985)). Choose κ so small that b nk Cψ n /2, k = 0,..., M 0 1. Taking into account our choice o κ, and ollowing the lines o the proo o Lemma 1, or all n large enough we have the inequalities ( max ( 0 k<m 0 n(x k ) (x k )) (1 + α)cψ n ) where M 0 1 k=0 M 0 1 k=0 M 0 1 k=0 (z nk (1 + α)cψ n Cψ n /2) ( nψ 1/β n (n 1/2 z nk (1/2 + α)c log n) n ξ ik (1/2 + α) log M) i=1 ξ ik = ψn 1/β /C 2 (2β + 1)( 1 K 0 ( X i x k ) E (n) κh n κh n [ 1 K 0 ( X i x k )]). κh n κh n Similarly to (7) we obtain the inequality E (n) [exp(cξ in/ n)] exp ( c2 2n V ar(n) [ξ in](1 + o n (1))) 9

10 with the only dierence that the variance V ar (n) [ξ in] σ0 2 is bounded by some constant σ which is not necessarily 1, as in (7). Note that M 0 is independent o n. Applying Chebyshev s exponential inequality, we have that uniormly in Σ(β, L) ( max ( 0 k<m 0 n(x k ) (x k )) (1 + α)cψ n ) M 0 exp ( c(1/2 + α) log M) exp ( c2 σ (1 + o n(1))). Under the choice c = log M/σ 2 0, the latter ormula yields the upper bound M 0 exp ( α log M) M 2σ0 2 0 M α/(2σ2 0 ). This completes the proo o (10), and the lemma ollows. The derivatives (m) (x), m = 1,..., β, o a density Σ(β, L), can be estimated in the sup-norm with the minimax rate O(h β m n ) as n. We need the ollowing version o the upper bound. Lemma 3. For any m, m = 1,..., β, there exists an estimator (m) n and positive constants p 0, p 1, and C 1 such that or any n and or any α > 0 the inequality holds sup ( max Σ(β,L) 0 k M (m) n (x k ) (m) (x k ) (1 + α)c 1 h β m n ) p 0 M α p 1. Proo. Note that the upper bound in this lemma is crude since C 1 is not necessarily optimal. Choose the kernel K 0 (u) as in Lemma 2, i.e. K 0 (u) has support in [0, 1] and satisies the orthogonality conditions. Assume that K 0 has β + 1 continuous derivatives. For a ixed m, m β, put and (m) n (x k ) = ( 1)m h 1+m n (m) n (x k ) = 1 h 1+m n n i=1 n i=1 K (m) 0 ( X i x k h n ) i 0 x k 1/2 K (m) 0 ( x k X i h n ) i 1/2 < x k 1 10

11 where K (m) 0 is the mth derivative o K 0. Standard arguments show that at each point the bias term is bounded rom above by C 2 h β m n in Σ(β, L) and x k [0, 1]. Take C 1 > 2C 2. Then with a positive constant C 2 uniormly where z (m) nk = (m) n ( max 0 k M (m) n (x k ) (m) (x k ) (1 + α)c 1 h β m n ) ( max 0 k M z(m) nk (1/2 + α)c 1h β m n ) (x k ) E (n) [ n (m) (x k )] are zero-mean random variables. Following the lines o the proo o Lemma 2, we ind that or all n large enough the latter probability is bounded rom above by 2M exp( c(1/2 + α) C 1 log M) exp( c 2 σ 2 m 2 ) with an arbitrary positive c and a constant σ 2 m > 0 independent o n. Choose C 1 > 8σ 2 m, and put c = (1/2 + α)c 1 log M/σ 2 m. Direct calculations show that the latter bound turns into 2M exp( 1 (1/2 + α) 2 C 2 2σm 2 1 log M) 2M 1 4(1/2+α)2 2M 4α which proves the lemma. Proo o Theorem: upper risk bound. Take the estimators n and (m) n as in Lemmas 1-3. For any x [x k, x k+1 ) continue n as the polynomial approximation n(x) = n(x k ) + β m=1 Uniormly in Σ(β, L) we have the inequality 1 m! (m) n (x k )(x x k ) m, x k x < x k+1, k = 0,..., M 1. n L(εh n ) β / β! + max n(x k ) (x k ) + 0 k M β m=1 1 m! (εh n) m (m) max n (x k ) (m) (x k ) 0 k M 11

12 where the irst term in the right-hand side appears rom the Taylor expansion o the density unctions Σ(β, L). When the complementary events to those in Lemmas 1-3 hold, then n (1 + α)(c + C 2 ε)ψ n with a positive constant C 2 independent o n, α and ε. Applying Lemmas 1-3, we have sup ( n (1 + α)(c + C 2 ε)ψ n ) p 2 M αp 3 (11) Σ(β,L) where p 2 = (1 + β )p 0 and p 3 = min[1; p 1 ]. Take an arbitrary small α 0, and put α j = jα 0, u j = (C + C 2 ε)(1 + α j ), j = 1, 2,.... Finally, or any continuous loss unctions w(u) with the polynomial majorant, we obtain rom (11) that sup Σ(β,L) E (n) w(ψn 1 n ) w((1 + α 0 )(C + C 2 ε)) + W 0 (1 + u γ j+1 ) p 2M jα 0p 3. Since the latter sum is vanishing as n, and α 0, ε are arbitrary small, the upper bound ollows. 3 Lower Asymptotic Bound We irst ormulate a lemma in a general ramework. For each j = 1,..., M let Q j,ϑ, ϑ [ 1, 1] be a dominated amily o distributions on some measurable space (X j, F j ). Let R = [ 1, 1] M, θ R and let Q θ = M Q j,θj, θ R be the amily o product measures indexed by θ = (θ 1,..., θ M ). Deine θ M = max 1 j M θ j. Lemma 4. Let π j be discrete prior distributions with inite support on [ 1, 1], and consider the Bayes risks r j,t (π j ) = in ˆϑ j [ 1,1] ( Q j,ϑ ˆϑj ϑ > T 12 ) π j (dϑ), j = 1,..., M (12)

13 where the inimum is taken over nonrandomized estimators ˆϑ j o ϑ depending only on data rom X j. Let ˆθ denote nonrandomized estimators o θ depending on the whole data vector x = (x j ),...,M, x j X j, let π = M π j and consider the Bayes risk r T (π) = in ˆθ ( ) M Q θ ˆθ θ > T π(dθ). Then or any T > 0 M r T (π) = 1 (1 r j,t (π j )). Proo. The j-th Bayes risk r j,t (π j ) with data x j rom X j can be ound as ollows. Let Q j,xj be the posterior distribution or ϑ and Q j be the marginal distribution or x j ; then [ 1,1] ( ) Q j,ϑ ˆϑj ϑ > T π j (dϑ) = 1 g j,t (x j, ˆϑ j (x j ))Q j (dx j ) where g j,t is the posterior gain g j,t (x j, t) = Q j,xj ( t ϑ T ). I S j is the inite support o π j then Q j,xj is concentrated on S j [ 1, 1]. For any t [ 1, 1] we have g j,t (x j, t) = Q j,xj ({ϑ}). ϑ S j : t ϑ T This unction o t takes only initely many values, and a maximum in t is attained on some closed interval t [t min (x j ), t max (x j )]. For uniqueness, take ˆϑ j(x j ) = t max (x j ) as a Bayes estimator. We then have max t [ 1,1] g j,t (x j, t) = g j,t (x j, ˆϑ j(x j )), (13) r j,t (π j ) = 1 g j,t (x j, ˆϑ j(x j ))Q j (dx j ). (14) 13

14 Consider now the global problem: we have r T (π) = in ˆθ ( = in 1 ˆθ = 1 sup ˆθ ( ) M Q θ ˆθ θ > T π(dθ) ) ( M χ [ T,T ] (ˆθ j θ j ) g T (x, ˆθ(x)) Q θ (dx) ) π(dθ) M Q j (dx j ) (15) where g T (x, u) is the posterior gain (or u = (u j ),...,M ): Then (13) implies M M g T (x, u) = Q j,xj ( u j ϑ T ) = g j,t (x j, u j ). max g(x, u) = M u R Thus a Bayes estimator o θ is max t [ 1,1] g j,t (x j, t) = M g j,t (x j, ˆϑ j(x j )). ˆθ (x) = (ˆϑ j(x j )),...,M, and rom (15) and (14) we obtain r T (π) = 1 g T (x, ˆθ M (x)) Q j (dx j ) M = 1 g j,t (x j, ˆϑ j(x j ))Q j (dx j ) M = 1 (1 r j,t (π j )). Back in our density problem, take a small value ɛ = ɛ(α) (0, 1); the inal choice o ɛ will be made below. Let Σ(β, L) be such that β (x) is constant in an interval x [t 1, t 2 ], t 2 t 1 ɛ, and (x) B /(1 + ɛ) or x [t 1, t 2 ] (c. lemma A. 3 below). 14

15 Set 0 = (t 1 ); then 0 B /(1 + ɛ). Consider again the solution g 1 Σ 0 (β, 1) o the extremal problem (4); recall g 1 2 = A (2β+1)/2β β and that S is such such that g 1 (u) = 0 or u > S. Deine g ɛ (u) = g 1 (u S) ɛg 1 (ɛ(u 2S(1 + ɛ 1 ))), u R. As is easily seen, g ɛ = 0, g 2 ɛ = (1 + ɛ) g and g ɛ Σ 0 (β, 1) or ɛ suiciently small. Set l n = h n 2S(1 + 1/ɛ) and redeine M = M(n, ɛ) rom section 2 as M = [n 1/((2β+1) (1+ɛ)) ]. Introduce a amily o unctions (x; θ) = (x; θ 1,..., θ M ) = (x) + Lh β n M θ j g ɛ (h 1 n (x a j )), 0 x 1, (16) where a 1 = t 1, a j+1 a j = l n, j = 1,..., M, θ = (θ 1,..., θ M ) R = [ 1, 1] M. The density (x; θ) diers rom (x) only in the interval [t 1, t 1 + Ml n ] [t 1, t 2 ] or n large since Mh n 0 as n or any ixed ɛ. Since β is constant on x [t 1, t 2 ], we obtain that or ɛ small enough and n suiciently large (x; θ) Σ(β, L) or θ R. Put or shortness ( ;θ) = θ and E (n) ( ;θ) = E(n) θ. Deine intervals J j = [a j, a j + l n ), j = 1,..., M, and let P j,θj be the conditional distribution o X 1 given that X 1 J j when θ obtains. Let κ(, ) be the Kullback-Leibler inormation number: or laws P 1, P 2 such that P 1 P 2 κ(p 1, P 2 ) = log dp 1 dp 2 dp 1. Consider also κ 2 2(P 1, P 2 ) = κ (P 1, P 2 ) = es sup P 1 ( log dp ) 2 1 dp 1, dp 2 log dp 1 dp 2. 15

16 Lemma 5. Let ϑ [0, 1] and consider the quantities κ = κ(p 1, P 2 ), κ 2 = κ 2 (P 1, P 2 ) and κ = κ (P 1, P 2 ) or measures P 1 = P j,ϑ, P 2 = P j, ϑ and j = 1,..., M. Set µ = 2(1 + ɛ) 2 /(2β + 1), n 0 = nl n 0. (17) Then uniormly over j = 1,..., M, as n (i) κ = 2ϑ 2 µ 0 n 1 0 log n (1 + o(1)) or some positive constant µ 0 = µ 0 (β, L, ɛ), µ 0 µ (ii) κ 2 2 = 2κ(1 + o(1)) (iv) κ 2 = O(n 1 0 log n). Proo. Deine The distribution P j,ϑ has density η j = ln 1 (x)dx. J j j (x; ϑ) = ( (x) + ϑlh β ng ɛ (h 1 n (x a j )))/l n η j, x J j. Observe that (x) = 0 + o(1) and η j = 0 + o(1) uniormly in j and x. In the sequel we use notation o (1), O (1) or quantities which are o(1) or O(1) as n uniormly over x J j and j = 1,..., M. Recall 0 B /(1 + ɛ). Deine urther z j (x) = Lh β ng ɛ (h 1 n (x a j ))/ (x); we then obtain j (x; ϑ) = l 1 n (1 + ϑz j (x))(1 + o (1)), x J j. (18) Now g ɛ = 0 entails z j (x) (x)dx = 0 and as a consequence ( z j (x) j (x; ϑ)dx = ϑln 1 ) zj 2 (x)dx (1 + o (1)). (19) 16

17 Note the ollowing relation: or 0 < z 0 Note also log 1 + z 1 z = 2z + O(z3 ). (20) z 2 j (x) = O (h 2β n ) (21) and the ollowing equalities o order o magnitude (denoted ), which are immediate consequences o our deinitions: h 2β n (log n/n) 2β/(2β+1) n 1 0 log n. (22) Proo o (i). We have κ = log 1 + ϑz j(x) 1 ϑz j (x) j(x; ϑ)dx; consequently, in view o (19) and (20) κ = 2ϑ = 2ϑ 2 l 1 n z j (x) j (x; ϑ)dx + O( z j (x) 3 ) ( ) zj 2 (x)dx (1 + o (1)) + O ((n 1 0 log n) 3/2 ). (23) Note that ln 1 zj 2 (x)dx = ln L 2 h (2β+1) n (1 + ɛ) g (1 + o (1)). Recall g = A (2β+1)/β β ; an evaluation o the r. h. s. above yields ln 1 z 2 j (x)dx = (B / 0 (1 + ɛ))µn 1 0 log n(1 + o (1)). (24) Set µ 0 = (B / 0 (1 + ɛ))µ; then µ 0 depends on ɛ, β, B = B (β, L) and 0 = (t 1 ), and the unction can be selected to depend only on β and L (c. Lemma A.3). The inequality 0 B /(1 + ɛ) now completes the proo o (i). 17

18 Proo o (ii). We have ( κ = log 1 + ϑz ) 2 j(x) j (x; ϑ)dx 1 ϑz j (x) (2ϑzj = (x) + O ((n 1 0 log n) 3/2 ) ) 2 j (x; ϑ)dx ( ) = 4ϑ 2 zj 2 (x) j (x; ϑ)dx + O ((n 1 0 log n) 2 ) ( ) = 4ϑ 2 ln 1 zj 2 (x)dx (1 + o (1)) + O ((n 1 0 log n) 3/2 ) so that (ii) ollows rom (23) and (24). Proo o (iii). This is an immediate consequence o (18), (21) and (22). Let us state a result on large deviations or sums o i. i. d. random variables. Let Z, Z 1, Z 2,... be a sequence o independent real random variables with common law Q. Lemma 6. Assume (i) E Q Z = 0, Var Q Z = 1 (ii) there exists a positive constant C such that Z C Q a.s. Let x n be a sequence such that x n, x n = o(n 1/2 ). Then or every δ > 0 we have n Pr Q (n 1/2 Z i > x n ) exp ( x 2 n(1 + δ)/2 ) (1 + o(1)), n i=1 uniormly over all Q ulilling (i) and (ii) or a given constant C. Proo. For the moment generating unction o Z we have an expansion E exp(tz) = 1 + t 2 /2 + φ with a remainder term satisying φ t 3 C 3 e C /3! uniormly over the class o distributions ulilling (i) and (ii). Hence uniormly over Q the ollowing lower bound holds: 18

19 lim n (x 2 n ) log Pr Q ((Z Z n )/(x n n) > 1) ( 1/2) (see Wentzell (1990), Theorem 4.4.1, or Freidlin and Wentzell (1984), Section 5.1, Example 4.) Thus, or all n large uniormly over Q satisying (i), (ii) we have log Pr Q ((Z Z n )/ n > x n ) ( 1/2 δ)x 2 n and the lemma ollows. For measures P 1, P 2 and P 0 = P 1 + P 2 let Π(P 1, P 2 ) be the testing ainity between P 1 and P 2 Π(P 1, P 2 ) = min(dp 1 /dp 0, dp 2 /dp 0 )dp 0. Let ν be natural and consider the ν-old product measure P ν j,ϑ o P j,ϑ with itsel, or ixed ϑ [0, 1] and or ϑ, and j = 1,..., M. Lemma 7. Let ϑ [0, 1] assume that n 0 (1 ɛ) ν n 0 (1 + ɛ). Then i ɛ is suiciently small, Π(P ν j,ϑ, P ν j, ϑ ) 2n ϑ2 µ (1 + o(1)) uniormly over j = 1,..., M, where µ = (1 + ɛ) 6 /(2β + 1). Proo. It is well known that i P 1 P 2 and P 2 P 1 then Π(P 1, P 2 ) = P 1 (dp 2 /dp 1 1) + P 2 (dp 1 /dp 2 > 1). 19

20 Set P 1 = P ν j,ϑ, P 2 = P ν j, ϑ and consider i. i. d. random variables λ 1,..., λ ν, having the law o λ = log (dp j, ϑ /dp j,ϑ ) under P j,ϑ. Then Note that P 1 (dp 2 /dp 1 1) = P ν j,ϑ ( ν ) λ i 0. (25) i=1 Eλ = κ(p j,ϑ, P j, ϑ ), Varλ = κ 2 2(P j,ϑ, P j, ϑ ) κ 2 (P j,ϑ, P j, ϑ ) = 2κ(P j,ϑ, P j, ϑ )(1 + o (1)) according to lemma 5. Set λ i = (λ i Eλ)/(Varλ) 1/2, i = 1,..., ν; then (25) takes the orm P 1 (dp 2 /dp 1 1) = P ν j,ϑ ( ν 1/2 ) ν λ i ν 1/2 Eλ/(Varλ) 1/2. We use lemma 6 or a lower bound to this large deviation probability. i=1 Note that Varλ 1 = 1, and λ 1 = λ Eλ /(Varλ) 1/2 (κ 2 2(P j,ϑ, P j, ϑ )) 1/2 2κ (P j,ϑ, P j, ϑ ) which according to lemma 5 is uniormly bounded or all suiciently large n. This lemma also yields ν 1/2 Eλ/(Varλ) 1/2 (1 + ɛ) 1/2 n 1/ /2 (κ(p j,ϑ, P j, ϑ )) 1/2 (1 + o (1)) (1 + ɛ)ϑµ 1/2 (log n) 1/2 (26) or suiciently large n. Moreover since (cp. (22)) ν n 0 n 2β/(2β+1) (log n) 1/(2β+1) 20

21 it ollows that the r. h. s. o (26) is o order (logν) 1/2, hence o(ν 1/2 ). Thus lemma 6 is applicable or x n = (1 + ɛ)ϑµ 1/2 (log n) 1/2 : or every δ > 0 ( P 1 (dp 2 /dp 1 1) exp 1 ) 2 x2 n(1 + δ) (1 + o (1)). Selecting δ = ɛ, we obtain we obtain P 1 (dp 2 /dp 1 1) n ϑ2 (1+ɛ) 4 µ/2 (1 + o (1)) = n ϑ2 µ (1 + o (1)). For P 2 (dp 1 /dp 2 1) this lower bound is proved analogously. Deine numbers ν j = n χ Jj (X i ), j = 1,..., M. (27) i=1 The joint distribution o ν = (ν 1,..., ν M ) under θ does not depend θ; call it ν. Lemma 8. For the event { } N n = sup,...,m ν j /n 0 1 < ɛ where n 0 is given by (17), we have ν (N n ) 1. Proo. Note that ν j is a sum o i. i. d. Bernoulli random variables χ Jj (X i ), i = 1,..., n with expectation J j and variance ( J j )(1 J j ). Let n j = n J j. Bennett s inequality (Shorack, Wellner (1986), Appendix A, p. 851) yields or any ɛ > 0 ν ( ν j n j n j ɛ ) exp( ɛ n 1/2 j C ɛ ) (28) or a constant C ɛ. Observe ln 1 J j = 0 + o(1) uniormly in j, hence n j /n 0 1 uniormly. Note also ν j /n 0 1 ν j /n j 1 (n j /n 0 ) + n j /n

22 Select ɛ ɛ/3 and n suiciently large such that n j /n 0 1 < ɛ ; then (28) and M = [n 1/((2β+1) (1+ɛ)) ] imply the assertion. Proo o Theorem: lower risk bound. to the Gaussian case in Korostelev (1993). We omit those details which are similar It suices to prove that or an arbitrary estimator ˆ n and or any small α > 0 lim in n sup Σ(β,L,b) Standard arguments show that this is implied by ( ˆ ) n > (1 α)cψ n = 1. lim in n sup θ R ( θ ˆθ n θ M ) > 1 α = 1 (29) where ˆθ n = (ˆθ n1,..., ˆθ nm ) is an arbitrary estimator o θ = (θ 1,..., θ M ), θ M = max 1 j M θ j. For the intervals J j = [a j, a j + l n ) deine conditional empirical distribution unctions F nj (t) = ν 1 j n χ [aj,a j +tl n )(X i ), t [0, 1], j = 1,..., M, i=1 where ν j are deined in (27). Though the random variables F nj under θ are dependent via the sample X 1,..., X n, they are conditionally independent given the number o sample points in each J j. Thus or sets D 1,..., D M in the appropriate sample space Let j,θ j,ν j ( θ Fn1 D 1,..., F ) nm D M ν 1 = n 1,..., ν M = n M = (30) = M ( ) θ Fnj D j ν j = n j. be the conditional distribution o the process F nj given ν j ; deine also a conditional empirical or the complement o M J j in [0, 1] and let 0,ν its conditional ( ) distribution given ν = (ν 1,..., ν M ). Then θ,ν = M j,θ j,ν j 0,ν represents 22

23 the conditional distribution o the whole sample X 1,..., X n given ν. Recall that ν is the joint θ -distribution o ν, which is is independent o θ R. Put C n = { } ˆθ n θ M > 1 α. Consider a prior distribution π = M π j on R where each π j has inite support in [ 1, 1]. Then in ˆθ n sup θ (C n ) in P θ R ˆθ n R (n) N n In view o lemma 8 it now suices to prove in in ν N n ˆθ n Applying Lemma 4 we obtain in ˆθ n ν (N n ) in ν N n in ˆθ n θ,ν (C n) ν (dν)π(dθ) θ,ν (C n) π(dθ). θ,ν (C n) π(dθ) 1 + o(1). (31) θ,ν (C n) π(dθ) 1 M (1 r j,1 α (π j )) (32) where r j,1 α (π j ) is the Bayes risk (12) or Q j,θj = j,θ j,ν j, T = 1 α. Now let us estimate this Bayes risk in each o the M (conditionally) independent problems, or ν N n. Note that each measure j,θ j,ν j can be construed as coming rom an i. i. d. sample o size ν j governed by the conditional distribution o X 1 given J j ; i. e. by P j,θj. Consider a test o the hypothesis θ j = θ + j = 1 α/2 vs. θ j = θ j = (1 α/2). Let π j be uniorm on {θ + j, θ j }; then we have (c. e. g. Strasser (1985), (4)) Now apply lemma 7, noting that r j,1 α (π j ) 1 (n) Π(P, ). 2 j,θ + j,ν j j,θ j,ν j Π(, j,θ + j,ν j j,θ j,ν j ) = Π(P ν j j,θ + j, P ν j ) j,θ j and that on N n we have n 0 (1 ɛ) ν j n 0 (1 + ɛ). We get r j,1 α (π j ) n (1 α/2)2 µ (33) 23

24 or all j = 1,..., M i n is large enough. Hence or the r. h. s. in (32) we obtain a lower bound 1 M We get Mn (1 α/2)2 µ = (1 + o(1))n µ ( 1 n (1 α/2)2 µ ) ( 1 exp Mn (1 α/2)2 µ ). (34) or an exponent µ = 1/(2β + 1)(1 + ɛ) (1 α/2) 2 µ = 1/(2β + 1)(1 + ɛ) (1 α/2) 2 (1 + ɛ) 6 /(2β + 1). ) For given α > 0, ɛ can be chosen such that µ > 0. In that case exp ( Mn (1 α/2)2 µ 0 and (34) implies (31). 4 Appendix: Analytic Facts The act that densities o the class Σ(β, L) are uniormly bounded in sup-norm ollows rom standard imbedding theorems. Lemma A 1. For any L > 0 and β > 0 B (β, L) = max Σ(β,L) max (x) < +. 0 x 1 Proo. Apply Theorem 17.4 o Besov, Il in and Nikol skii (1979), using the act that is bounded in L 1 -norm on [0, 1]. For β 1 the value o B (β, L) can be ound. Lemma A 2. For any L > 0 and 0 < β 1 B (β, L) = ((β + 1)/β) β/(β+1) L 1/(β+1) i L (β + 1)/β, = 1 + L/(β + 1) i L (β + 1)/β. 24

25 Proo. It can be shown that the extremal density is (x) = max(((0) Lx β ), 0), x [0, 1]. An easy calculation rom (x)dx = 1 yields (0). Lemma A 3. For any L > 0 and β > 0, and every ɛ (0, 1) there are t 1, t 2 [0, 1], 0 < t 2 t 1 ɛ and a unction Σ(β, L) such that or all x [t 1, t 2 ], (x) B (β, L)/(1 + ɛ), (35) β (x) = β (t 1 ). Proo. Let be a solution in o the problem (2), i. e. = B (β, L). Let ɛ (0, ɛ) and let t 1, t 2 [0, 1], t 2 t 1 = ɛ be such that (x) B (β, L)/(1 + ɛ/2) or x [t 1, t 2 ]. Since Σ(β, L) is continuous on [0, 1], such t 1, t 2 [0, 1] exist or suiciently small ɛ. Let m = β, γ = β m and let t 0 [t 1, t 2 ] be such that (m) (t 0 ) (m) (x), or x [t 1, t 2 ]. Since (m) is continuous, such a t 0 exists. Deine a unction g 0 by g 0 (x) = (m) (t 0 ) (m) (x), x [t 1, t 2 ] = (m) (t 0 ) (m) (t 2 ), x (t 2, 1] = (m) (t 0 ) (m) (t 1 ), x [0, t 1 ). Note that g 0 (x) 0, x [0, 1] and g 0 L t 2 t 1 γ = L ɛ γ. Let Q be the integral operator Qg(t) = t 0 g(u)du, t [0, 1] and deine g = Qm g 0 (m-old application o Q). Then g(x) 0, x [0, 1] and g g 0 L ɛ γ. (36) 25

26 Deine = + g. Since (m) (t) = (m) (t 0 ) on [t 1, t 2 ] while (m) (t) (m) (t) is constant outside (t 1, t 2 ), it ollows that (m) (x 1 ) (m) (x 2 ) L x 1 x 2 γ, x 1, x 2 [0, 1]. Furthermore, and by (36) L ɛ γ. Deining = /, we see that is a density in Σ(β, L). Moreover, (x) B (β, L)/(1 + ɛ/2) or x [t1, t 2 ]. By selecting ɛ suiciently small, we achieve (35). Acknowledgements This work was supported by the Deutsche Forschungsgemeinschat, Sonderorschungsbereich 373 Quantiication and Simulation o Economic Processes, Berlin, Germany. The research o the irst author was supported by the International Science Foundation under Grant MCF000. The authors grateully acknowledge the help o Alexander Tsybakov with the concise proo o lemma A.1. Reerences Besov, O. V., Il in, V. P. and Nikol skii, S. M. (1979). Integral Representation o Functions and Imbedding Theorems, Vol. II. V. H. Winston & Sons, Washington, DC / John Wiley, New York. Brown, L. D. and Zhang, C.-H. (1998). Asymptotic nonequivalence o nonparametric experiments when the smoothness index is 1/2. Ann. Statist

27 Devroye, L., Gyori, L. (1985). Nonparametric Density Estimation: the L 1 -view. John Wiley, N. Y. Donoho, D. (1994). Asymptotic minimax risk (or sup norm loss): solution via optimal recovery. Probab. Theor. Rel. Fields Freidlin, M., and Wentzell, A. (1984), Random Perturbation o Dynamical Systems, Springer, New York Ibragimov, I. A., Khasminskii, R. Z. (1982). Bounds or the risks o non-parametric regression estimates. Theory Probab. Appl Khasminskii, R.Z. (1978). A lower bound on the risk o non-parametric estimates o densities in the uniorm metric. Theory Probab. Appl Korostelev, A. P. (1993). Exact asymptotically minimax estimator or nonparametric regression in uniorm norm. Theory Probab. Appl Korostelev, A. P. and Nussbaum, M. (1995). Density estimation in the uniorm norm and white noise approximation. Preprint No. 154, Weierstrass Institute, Berlin. Leonov, S.L (1997). On the solution o an optimal recovery problem and its applications in nonparametric regression. Math. Meth. Stat Micchelli, C. and Rivlin, T. (1977) A survey o optimal recovery. In: C. Micchelli and T. Rivlin (Eds.) Optimal Estimation in Approximation Theory Plenum Press, New York. Nussbaum, M. (1996). Asymptotic equivalence o density estimation and Gaussian white noise. Ann. Statist

28 Pinsker, M. S. (1980). Optimal iltering o square integrable signals in Gaussian white noise. Problems Inorm. Transmission (1980) Shorack, G. and Wellner, J. (1986). Empirical Processes with Applications to Statistics. Wiley, New York Stone, C. (1982). Optimal global rates o convergence or nonparametric regression. Ann. Statist Strasser, H. (1985). Mathematical Theory o Statistics. Walter de Gruyter, Berlin. Wentzell, A., (1990). Theorems on Large Deviations or Markov Stochastic Processes. Kluwer, Dordrecht. 28