Uniform Convergence Rates for Nonparametric Estimation

Transcription

1 Uniform Convergence Rates for Nonparametric Estimation Bruce E. Hansen University of Wisconsin October 2004 Preliminary and Incomplete Abstract Tis paper presents a set of rate of uniform consistency results for kernel estimators of density functions and regressions functions. We generalize te eisting literature by allowing for stationary strong miing multivariate data wit infinite support and kernels wit unbounded support. Researc supported by te National Science Foundation. Department of Economics, 1180 Observatory Drive, University of Wisconsin, Madison, WI

2 1 Introduction Tis paper presents a set of rate of uniform consistency results for kernel estimators of density functions and regressions functions. We generalize te eisting literature by allowing for stationary strong miing multivariate data wit infinite support and kernels wit unbounded support. Te kernel estimators tat we eamine were first introduced by Rosenblatt (1956) for density estimation, by Nadaraya (1964) and Watson (1964) for regression estimation. Te local linear estimator was introduced by Stone (1977) and came into prominence troug te work of Fan (1992, 1993). Andrews (1995) provides a compreensive set of results concerning te uniform consistency of kernel estimators, but is rates are not sarp. Masry (1996) derived sarp rates for uniform almost sure convergence, but confined attention to te case of bounded regression support, and placed overly restrictive conditions on te regression functions. Fan and Yao (2003) also ave a set of results, but are quite restrictive in application. In tis paper we attempt to provide a general set of results wit broad applicability. Our main result is te uniform convergence of a sample average functional, wic can be easily used for application to density and regression estimation. Te conditions imposed on te functional are quite general, allowing for kernels wit unbounded support (suc as te standard normal), so long as tey satisfy a Lipscitz condition. Te data are allowed to be generated eiter from a random sample or from a stationary strong miing time series. Te support fortedataisallowedtobeinfinite, and our convergence results include te entire sample space rater tan being restricted to a compact subset. Te rate of decay for te bandwidt is allowed considerable fleibility. Our proof metod is a generalization of tat in Fan and Yao (2003), and like teirs is based on an eponential inequality from Bosq (1998). We also borrow te trimming argument of Andrews (1995) to allow for unbounded regression support. Section 2 of te paper presents te main results: a variance bounded and te rate of convergence for te sample average functional. Section 3 provides an application to density estimation, and Section 4 to regression estimation. Te proofs are in te Appendi. 1

3 2 Basic Results Let {X i,y i } R d R be a sequence of random vectors. We are interested in averages of te form Ĝ () = 1 n µ Xi Y i G (1) were G() :R d R and = cn γ (2) is a bandwidt, were 0 <c< and 0 <γ<1/d. For typical applications, te function G is eiter a kernel or te product of a kernel wit a polynomial. Assumption 1 For all, 0 R d,someλ < and η>1/γ ten ( Λ, 1 G() Λ η, > 1, (3) G() G( 0 ) Λ 0, (4) Equation (3) imposes boundedness and a tail condition related to te bandwidt. Equation (4) is a Lipscitz condition. We require te following moment and smootness conditions on te observations. Let f Y X (y ) and f X () denote te conditional density of Y i given X i, and te marginal density of X i, respectively, and for any j 1 let f j ( 0, j ) denote te joint density of (X 0,X j ). Assumption 2 {X i,y i } is strictly stationary and ergodic, wit strong miing coefficients α m am β for some a< and β> 2s 2 (5) s 2 for some s>2. Furtermore,E Y i+j s <, and for all j 0 and all t s sup E Y i+j t X i = f X () Ψ 1 <, (6) for j d +1, sup f j ( 1, 2 ) Ψ 2 <, (7) 1, 2 and for large and some 1 <µ E ( Y i X i = ) f X () Ψ 3 µ. (8) Note tat equations (6) and (8) involve te product of te conditional mean and te marginal density, and tus are not very restrictive. For independent data, f j ( 0, j )=f X ( 0 )f X ( j ), so (7) 2

4 olds wen te density f X () is bounded. If te support of X i is bounded, ten we can take µ = in (8) We first describe a uniform bound on te variance of Ĝ(). Teorem 1 Under Assumptions 1 and 2, tere is J< suc tat Var³Ĝ () Jd n. (9) We now present our main result concerning te rate of convergence for Ĝ(). Teorem 2 Under Assumptions 1 and 2, if in addition β> 3 µ s 2 + d µ µ 1 µ (1 + γd)+3+7γd 2 s (s 2) (1 γd) (10) ten were ³ sup Ĝ() EĜ() = O p d r n R d r n = µ log n 1/2 d. n (11) Te rate for te bandwidt is controlled by (10) wic is satisfied for sufficiently large β. In particular, in te case of independent data or eponential decay for te miing coefficients, we ave β = and (10) is automatically satisfied. Te rigt side of equation (10) is increasing in γ, so te inequality is least restrictive by selecting γ close to zero (so te bandwidt declines to zero slowly). Te limiting case (γ =0)is β> 3 µ s 2 + d µ µ µ s 2 s 2 Furtermore, if Y i as all moments finite and te support of X i is bounded, ten s = and µ =, and tis simplies to β>d In particular, for d =1, ten te inequality is β>5/2, te restriction used by Fan and Yao (2003), Lemma 6.1. Tus Teorem 2 generalizes teir result to te case of mutlivariate data wit unbounded support. 3

5 3 Density Estimation Consider te estimation of f X (), te density of X i.letk(u) :R R denote a kernel function and let K() = dy k ( j ) be a product kernel. Let be a bandwidt. Te kernel density estimator of f X () is ˆf X () = 1 n d j=1 µ Xi K. It is asymptotically optimal to set = cn γ wit γ =1/ (d +4), wic we now assume. Assumption 3 (a) R R k(u)du =1;(b) k( u) =k(u); k(u) satisfies (3) and (4) for some η>d+4. Te bandwidt satisfies = cn 1/(d+4). We can use Teorem 2 to obtain te uniform rate of convergence for f X (). Teorem 3 If Assumption 2 olds wit Y i =1and s =, Assumption 3 olds, and β> 1 4 µ d µ (2d +4)+6+5d µ 1 (12) ten sup ˆf X () f X () = O p (r n ) R d were r n = n 2/(d+4) log 1/2 n Alternative results for te uniform rate of convergence for kernel density estimates ave been provided by Andrews (1995, Teorem 1) and Fan and Yao (2003, Teorem 5.3). Andrews result is more general in allowing for near-epoc-dependent arrays but obtains a slower rate of convergence. Fan and Yao obtained te same rate of convergence, but teir result is restricted to univariate data, compact support for X i, and kernels wit compact support. 4

6 4 Nadaraya-Watson Estimator Consider te estimation of te conditional mean m() =E (Y i X i = ). Let te multivariate kernel K and bandwidt be defined as in te previous section. Te Nadaraya-Watson estimator of m() is ˆm() = P n Y ik ³ Xi P n K ³ Xi Te local linear (LL) estimator of m() is obtained from a weigted regression of Y i on X i. Letting Ã! 1 z i = ten te LL estimator is m() = P n Y ik ³ Xi P n K ³ Xi P n z0 i K ³ Xi P n z0 i K ³ Xi We introduce te following smootness condition. Assumption 4 For some δ>0, X i. ³ Pn z iz i K ³ Pn z iz i K sup m(1 ) 0 f X ( 2 ) < 1 2 δ sup 2 m( 1 )f X ( 2 ) < 1 2 δ ³ Xi ³ Xi 1 P n z iy i K 1 P n z ik ³ Xi ³ Xi Observe tat Assumption 4 does not require te regression function and its derivatives to be bounded. Rater, m() and 2 m() are required to not diverge faster tan f X () and f X () decline to zero in te tails. For any positive sequence δ n define A n = Teorem 4 Under Assumptions 2, 3, 4, and (10), n R d : f() δ n o. sup ˆm() m() = O p δ 2 n r n. A n 5

7 Alternative results for te uniform rate of convergence for te Nadaraya-Watson estimator ave been provided by Andrews (1995, Teorem 1). His results allow for near-epoc-dependent arrays but obtains a slower rate of convergence. Teorem 5 Under [conditions] sup m() m() = O p δ 2 n r n. A n Te conditions and proof are incomplete. Tis result complements tat of Masry (1996). Masry obtains almost sure convergence over compact sets, but imposed stronger conditions on te regression function. 6

8 References [1] Andrews, Donald W.K. (1995): Nonparametric kernel estimation for semiparametric models, Econometric Teory, 11, [2] Bosq, D. (1998): Nonparametric Statistics for Stocastic Processes: Estimation and Prediction (2nd ed.). Lecture Notes in Statistics 110. Springer-Verlag. [3] Fan, Jianqing (1992): Design-adaptive nonparametric regression, Journal of te American Statistical Association, 87, [4] Fan, Jianqing (1993): Local linear regression smooters and teir minima efficiency, Annals of Statistics, 21, [5] Fan, Jianqing and Qiwei Yao (2003): Nonlinear Time Series: Nonparametric and Parametric Metods. Springer-Verlag. [6] Masry, Elias (1996): Multivariate local polynomial regression for time series: Uniform strong consistency and rates, Journal of Time Series Analysis, 17, [7] Nadaraya, E. A. (1964): On estimating regression, Teory of Probability and Its Applications, 9, [8] Rosenblatt, M. (1956): Remarks on some non-parametric estimates of a density function, Annals of Matematical Statistics, 27, [9] Stone, C.J. (1977): Consistent nonparametric regression, Annals of Statistics, 5, [10] Watson, G.S. (1964): Smoot regression analysis, Sankya Series A, 26,

9 5 Appendi ProofofTeorem1. WLOG assume tat Λ 1 and Ψ 1. From (3) we observe tat G () d Λ d + Λ η d ΛV dη R d 1 >1 η 1 were V d is te volumn of te unit spere in R d. Since G () Λ It follows tat for any 1 t s G () t d Λ t 1 ΛV dη R d η 1 Λs V d η Θ. (13) η 1 ³ Let U ni () =Y i G Xi. By a cange of variables, for any 1 t s, using (13) and (6), µ E U ni () t y G t u t f Y X (y u) f X (u) dudy µ = d G (u) t y t f Y X (y u) dy f X ( u) du ΘΨ 1 d. (14) We now develop several alternative bounds on te covariances Cov (U n0 (),U nj ()). First, by te Caucy-Scwarz inequality and (14) wit t =2, Cov (U n0 (),U nj ()) Var(U n0 ()) 2EU n0 () 2 2ΘΨ 1 d. (15) Second, take j d +1and observe tat µ µ Cov (U n0 (),U nj ()) E Y 0 Y j EY 0 EY j X0 Xj G G µ µ +(EY 0 ) 2 E X0 Xj G G +(E U n0() ) 2. (16) We eamine te terms on te rigt-and-side of (16). By cange of variables, (13), and (7), µ µ µ µ E X0 Xj u0 uj G G = G G f j (u 0,u j ) du 0 du j = 2d G (u 0 ) G (u j ) f j ( u 0, u j ) du 0 du j 2d Θ 2 sup f j ( 1, 2 ) 1, 2 2d Θ 2 Ψ 2. (17) 8

10 Let g j (y 0,y j ) denote te conditional density of (Y 0,Y j ) given X 0 =. Using (3), a cange of variables, conditioning, (13), Davydov s Lemma, (6), and te assumption α j aj β, we find µ µ µ E Y 0 Y j EY 0 EY j X0 Xj G G µ µ ΛE Y 0 Y j EY 0 EY j X0 G µ = Λ y 0 y j EY 0 EY j u G g j (y 0,y j u) f X (u)dudy 0 dy j = d Λ y 0 y j EY 0 EY j G (u) g j (y 0,y j u) f X ( u)dudy 0 dy j d ΛΘ sup E ( Y 0 Y j EY 0 EY j X 0 = ) f X () d ΛΘα 1 2/s j sup (E ( Y 0 s X 0 = ) f X ()) 1/s (E ( Y j s X 0 = ) f X ()) 1/s d aλθψ 1 j β[1 2/s]. (18) Equations (16)-(18) combine to sow tat for j d +1 ³ Cov (U n0 (),U nj ()) aλθψ 1 j β[1 2/s] d + (EY 0 ) 2 Ψ 2 + Ψ 2 1 Θ 2 2d. (19) Tird, using Davydov s Lemma, (14) wit t = s, and te assumption α j aj β, Cov (U n0 (),U nj ()) 16α 1 2/s j (E U ni () s ) 2/s 16aj β(1 2/s) ΘΨ 1 2d/s were te final inequality olds since 2/s 2 β (1 2/s) under (5). Te bounds (15), (19) and (20) sow tat 16aj β(1 2/s) ΘΨ 1 d[2 β(1 2/s)] (20) nv ar ³Ĝ () = 1 n E Ã U ni () EU ni ()! 2 Var(U n0 ()) d X j=d+1 X dx Cov (U n0 (),U nj ()) j=1 Cov (U n0 (),U nj ()) j= d Cov (U n0 (),U nj ()) 9

11 2ΘΨ 1 d (1 + 2d) d X j=d+1 X ³ aλθψ 1 j β(1 2/s) d + (EY 0 ) 2 Ψ 2 + Ψ 2 1 Θ 2 2di j= d 16aΘΨ 1 j β(1 2/s) d[2 β(1 2/s)] 2ΘΨ 1 (1 + 2d) d + 2aΛΘΨ ³ 1 β 1 2 d +2 (EY 0 ) 2 Ψ 2 + Ψ 2 1 Θ 2 d s aΘΨ 1 β 1 2 d s 1 J d wic is (9) wit J =2ΘΨ 1 (1 + 2d)+ 2aΛΘΨ ³ 1 β (EY 0 ) 2 Ψ 2 + Ψ 2 1 Θ aΘΨ 1 s 1 β 1 2 (21) s 1 Tis is (9). For te final inequality we ave used te fact tat for δ>0 and k 1 Tis completes te proof. X j=k+1 j δ 1 k δ 1 d = 1 δk δ. ProofofTeorem2. We start by introducing some notation. First, define µ Xi V ni () =Y i G Ã 1 Y i r 2/s n 2! and ˆV () = 1 n V ni (). V ni () as been truncated so tat V ni () EV ni () rn 2/s. Second, for µ defined in Assumption 2 let τ n =2 1 d 1/(µ 1) r n and set A = { : τ n }. Te region A can be covered wit µ τ d n N = 1+d = d(1+dµ/2(µ 1)) r n 2 d µ n dµ/2(µ 1) (22) log n ypersperes of te form A j = : j 1+d ª r n, wic are centered at j and ave radius 1+d r n.leta c = { : >τ n }. 10

12 We sow below tat sup Ĝ() EĜ() = sup ˆV () E ˆV () + O p ( d r n ). (23) sup ˆV () E ˆV () = ma ˆV ( j ) E ˆV ( j ) + O p ( d r n ) (24) A 1jN sup ˆV () E ˆV () = O p ( d r n ) (25) A c ˆV ( j ) E ˆV ( j ) = O p ( d r n ). (26) ma 1jN Togeter, tese establis tat sup Ĝ() EĜ() = O p ( d r n ) as desired. It remains to sow (23)-(26), wic we take sequentially. Proofof(23):First, P µ ³ d r n 1 sup Ĝ() ˆV () > 0 Ã! P ma Y i > r 2/s n 1in 2 µ np Y i s > 1 2 s rn µ µ 2 2 s E Y i s 1 Y i s > 1 2 s rn 2 0 using Markov s inequality since r n 0 as n. Hence sup Ĝ() ˆV () = o p ( d r n ). (27) Second, by a cange of variables and using (13) sup E Ĝ() ˆV () sup d sup d d Θ y rn 2/s /2 y rn 2/s /2 G (u) du sup à r 2/s n 2 µ u G y f Y X (y u) f X (u) dydu G (u) y f Y X (y u) f X ( u) dydu Ã! 1 s sup y f Y X (y ) dyf X () y rn 2/s /2 µ y s f Y X (y ) dyf X () 2 s ΘΨ 1 d r n, (28)! 11

13 te final inequality using (6) and te fact tat for s>2 and n large r 2(s 1) s n r n. Equation (23) follows from (27) and (28). Proofof(24):Te Lipscitz condition (4) and te radius of A j imply tat for all A j and tus for all A j µ µ Xi j X i G G Λ 1 j Λ d r n ˆV () ˆV ( j ) 1 n µ µ Y i Xi j X i G G ˆµ n Λ d r n were ˆµ n = 1 n Y i = O p (1). Similarly E ˆV () E ˆV ( j ) E ˆV () ˆV ( j ) E Y i Λ d r n. Terefore sup ˆV () E ˆV () = ma sup ˆV () E ˆV () A 1jN A j = ma ˆV ( j ) E ˆV ( j ) 1jN + ma ³ sup ˆV () ˆV ( j ) + E ˆV ( j ) E ˆV () 1jN A j ˆV ( j ) E ˆV ( j ) +(ˆµ n + E Y i ) Λ d r n ma 1jN wic implies (24). Proofof(25):For η defined in Assumption 1, let δ n = d 1/η r n. Observetatsinceη>1/γ ten (1 + γd) /2η <γand µ n 1/2η δ n = d O ³n γ+(1+γd)/2η = o(1). (29) log n If X i τ n δ n and A c = { : >τ n }, ten X i µ Xi G G (δ n ) Λδ η n δ n and = d r n 12

14 by (3). Hence µ µ E Y i sup Xi G A c 1( X i τ n δ n ) Λ d r n E Y i. Furtermore using (8), µ µ E Y i sup Xi G A c 1( X i >τ n δ n ) E ( Y i 1( X i >τ n δ n )) = E ( Y i X i = ) f X ()d >τ n δ n 2Ψ 3 µ d τ n δ n O ³(τ 1 µ n δ n ) O τ n 1 µ O( d r n ) te second-to-last inequality using (29) and te final inequality by te definition of τ n. Tus E sup ˆV () E ˆV µ µ () 2E Y i sup Xi G = O( d r n ). A c A c (25) follows by application of Markov s inequality. Proofof(26): Define à m 2 X σ 2 m() =E (V ni () EV ni ())!. By Teorem 1 and n sufficiently large, σ 2 m() mj d (observe tat (5) olds under (10)). Since V ni EV ni rn 2/s, by Teorem 1.3 of Bosq (1998), for all, q (0, 1] and ε>0 P ³ ˆV () E ˆV () >ε Ã! 1/2 ε 4ep 2 n 32qσ 2 [1/q] ()+8q 1 εrn 2/s +11nqα [1/q] 1+ 4 εrn 2/s Ã! ε 2 n 4ep 32J d +8q 1 εrn 2/s +12anq 1+β ε 1/2 rn 1/s were te second inequality olds for n sufficiently large and te assumption on te miing coefficients. 13

15 Set q = rn 1 2/s and ε = Mr n d for M>J.Tis gives te bound P ³ ˆV () E ˆV d () >Mr n µ 4ep M 2 rn 2 2d n 32J d +8M d +12aM 1/2 nr n (1+β)(1 2/s) 1/2 1/s d/2 = 4n M/40 +12aM 1/2 d(3/2+β(1 2/s) 3/s)/2 n (3/2 β(1 2/s)+3/s)/2 (log n) λ were λ =(β (1 2/s)+1/2 3/s) /2. Hence µ P ˆV ( j ) E ˆV ( j ) >M d r n ma 1jN NP ³ ˆV ( j ) E ˆV ( j ) >M d r n 4Nn M/40 +12a Nn(3/2 β(1 2/s)+3/s)/2 (log n) λ M 1/2 d(3/2+β(1 2/s) 3/s)/2. (30) We now sow tat te rigt-and-side of (30) is o(1) for M sufficiently large, wic implies (26). Indeed, using definition (22) Nn M/40 = d(1+dµ/2(µ 1)) 2 d µ n dµ/2(µ 1) n M/40 log n n γd(1+dµ/2(µ 1))+dµ/2(µ 1) M/40 wic is = o(1) wen M>40γd(1 + dµ/2(µ 1)) + 20dµ/ (µ 1). Furtermore, Nn (3/2 β(1 2/s)+3/s)/2 (log n) λ d(3/2+β(1 2/s) 3/s)/2 = = Ã n dµ/(µ 1)+3/2 β(1 2/s)+3/s d(7/2+dµ/(µ 1)+β(1 2/s) 3/s)! 1/2 (log n) λ dµ/2(µ 1) ³ n γd(7/2+dµ/(µ 1)+β(1 2/s) 3/s)+dµ/(µ 1)+3/2 β(1 2/s)+3/s1/2 (log n) λ dµ/2 = o(1) since µ 7 γd 2 + d µ µ µ 1 + β d µ s s µ µ 2 β s s < 0 under (10). Tus (30) is o(1) wic establises (26) as desired. We ave sown (23)-(26), wic completes te proof. ProofofTeorem3. In te notation of Teorem 2, ˆfX () = d Ĝ () wit G() =K(). Since k() satisfies Assumption 1, so does te product K(). Equation (12) is (10) substituting s = and γ =1/(d +4). Ten by Teorem 2 sup ˆf X () E ˆf X () = d sup Ĝ() EĜ() = O p (r n ) R d R d 14

16 were µ log n 1/2 r n = n d/(d+4) = log1/2 n n n 2/(d+4). It is well known tat under tese assumptions E ˆf X () f X () =O 2 = O ³n 2/(d+4) O (r n ). Togeter we obtain sup ˆf X () f X () = O p (r n ). R d ProofofTeorem4.Lete i = Y i m(x i ). Ten P ³ n (Y i m()) K Xi ˆm() m() = P ³ n K Xi P n (e i + m(x i ) m()) K = P ³ n K Xi à = ˆf X () 1 1 µ Xi n d e i K ³ Xi + 1 n d Now consider te terms on te rigt and side. First, by a Taylor epansion, Teorem 3, and te definition of A n µ! Xi (m(x i ) m()) K. sup ˆfX () 1 f X () 1 sup f X () 2 sup ˆfX () f X () δ 2 n O p (r n ), A n A n A n Second, by Teorem 2 and te fact tat E (e i X i )=0 Tird, by Teorem 2 1 sup n d A n sup A n 1 n d µ (m(x i ) m()) K µ Xi e i K = O p (r n ). µ Xi E (m(x i ) m()) K µ Xi = O p (r n ). 15

17 Now, letting denote a point intermediate between and u, µ 1 d E (m(x Xi i) m()) K = 1 µ u d (m(u) m()) K f X (u)du = (m( u) m()) K (u) f X ( u)du µ = 2 m() 0 uu 0 K (u) f X ( )du tr 2 m( )uu 0 K (u) f X ( u)du. Tus for sufficiently small µ 1 d E (m(x Xi i) m()) K Ã 2 uu 0 K (u) du m(1 ) 0 f X ( 2 ) + = O 2 O p (r n ) Togeter, tese complete te proof. sup 1 2 δ sup 1 2 δ! 2 m( 1 )f X ( 2 ) 16