Bootstrap confidence intervals in nonparametric regression without an additive model

Transcription

1 Bootstrap confidence intervals in nonparametric regression witout an additive model Dimitris N. Politis Abstract Te problem of confidence interval construction in nonparametric regression via te bootstrap is revisited. Wen an additive model olds true, te usual residual bootstrap is available but it often leads to confidence interval under-coverage; te case is made tat tis under-coverage can be partially corrected using predictive as opposed to fitted residuals for resampling. Furtermore, it as been unclear to date if a bootstrap approac is feasible in te absence of an additive model. Te main trust of tis paper is to sow ow te transformation approac put fort by Politis (2010, 2013) in te related setting of prediction intervals can be found useful in order to construct bootstrap confidence intervals witout an additive model. 1 Introduction Consider regression data of te type {(Y t, ), t = 1,..., n}. For simplicity of presentation, te regressor is assumed univariate and deterministic; te case of a multivariate regressor is andled similarly. As usual, it will be assumed tat Y 1,...,Y n are independent but not identically distributed. Attention focuses primarily on te first two moments of te response Y t, namely µ( ) = E(Y t ) and σ 2 ( ) = Var(Y t ). (1) In te nonparametric setting, te functions µ( ) and σ( ) are considered unknown but assumed to possess some degree of smootness (differentiability, etc.). Tere are many approaces towards nonparametric estimation of te functions µ and σ, e.g., wavelets and ortogonal series, smooting splines, local polynomials, and kernel smooters. For concreteness, tis paper will focus on one of te oldest metods, namely te Nadaraya-Watson (N-W) kernel estimators; see Li and Racine (2007) and te references terein. Beyond point estimates of te functions µ and σ, it is important to be able to additionally provide interval estimates in order to ave a measure of teir statistical accuracy. Suppose, for example, tat a practitioner is interested in te expected response to be observed at a future point. A confidence interval for µ( ) is ten desirable. Under regularity conditions, suc a confidence interval can be given eiter via a large-sample Dimitris N. Politis University of California at San Diego, La Jolla, CA , USA, dpolitis@ucsd.edu 1

2 2 Dimitris N. Politis normal approximation, or via a resampling approac; see e.g. Freedman (1981), Härdle and Bowman (1988), Härdle and Marron (1991), Hall (1993), or Neumann and Polzel (1998). Typical regularity conditions for te above bootstrap approaces involve te assumption of an additive model wit respect to independent and identically distributed (i.i.d.) errors. In Section 2, we revisit te usual model-based bootstrap for regression adding te dimension of employing predictive as opposed to fitted residuals as advocated by Politis (2010, 2013) in a related context. More importantly, in Section 3 we address te problem of constructing a bootstrap confidence interval for µ( ) witout an underlying additive model. Te model-free approac developed in tis paper is totally automatic, relieving te practitioner from te need to find an optimal transformation towards additivity and variance stabilization; tis is a significant practical advantage because of te multitude of suc proposed transformations, e.g. te Box/Cox power family, ACE, AVAS, etc. see Linton et al. (1997) and te references terein. Te finite-sample simulations provided in Section 4 confirm te viability and good performance of te model-free confidence intervals. 2 Model-based nonparametric regression 2.1 Nonparametric regression wit an additive model An additive model for nonparametric regression is given by te equation Y t = µ( )+σ( ) ε t, t = 1,..., n, (2) wit ε t i.i.d. (0,1) from an (unknown) distribution F. Te N-W estimator of µ(x) is defined as n ( ) ( ) x xi x xi m x = Y i K wit K = K( x x ) i n k=1 K ( x x ) (3) k were is te bandwidt, and K(x) is a symmetric kernel function wit K(x)dx = 1. Similarly, te N-W estimator of σ 2 (x) is given by s 2 x = M x m 2 x were M x = n Y i 2 K ( x x ) i. For t = 1,..., n, let e t = (Y t m xt )/s xt denote te fitted residuals, and ẽ t = (Y t m (t) )/s (t) te predictive residuals. Here, m x (t) and M x (t) denote te estimators m x and M x respectively computed from te delete-y t dataset: {(Y i, x i ), i = 1,...,t 1 and i = t + 1,..., n}. As before, define s (t) = M x (t) t (m (t) ) 2. Coosing te bandwidt is often done by cross-validation, i.e., picking to minimize t=1 n ẽ2 t, or its L 1 analog: t=1 n ẽ t. 2.2 Model-based confidence intervals Consider te problem of constructing a confidence interval for te regression function µ( ) at a point of interest. A normal approximation to te distribution of te estimator m xf implies an approximate (1 α)100% equal-tailed, confidence interval for µ( ) given by: [m xf + v xf z(α/2), m xf + v xf z(1 α/2)] (4)

3 Bootstrap confidence intervals in nonparametric regression witout an additive model 3 were v 2 = s 2 n K 2 ( x i ) wit K defined in eq. (3), and z(α) being te α quantile of te standard normal. If te density (e.g. istogram) of te design points x 1,..., x n can be tougt to approximate a given functional sape (say, f( )) for large n, ten te large-sample approximation n K 2 ( x f x i K 2 (x)dx ) n f( ) (5) can be used wic relies on te assumption tat K(x)dx = 1; see e.g. Li and Racine (2007). Interval (4) may be problematic in two respects: (a) it ignores te bias of m x, so it must be eiter explicitly bias-corrected, or a suboptimal bandwidt must be used to ensure undersmooting; and (b) it is based on a Central Limit Teorem wic may not be a good finite-sample approximation if te errors are skewed and/or leptokurtic, or wen te sample size is not large enoug. For bot above reasons, practitioners often prefer bootstrap metods over te normal approximation interval (4). Wen using fitted residuals, te following algoritm is te well-known residual bootstrap pioneered by Freedman (1981) in a linear regression setting, and extended to nonparametric regression by Härdle and Bowman (1988), and oter autors. As an alternative, we also propose te use of predictive residuals for resampling as advocated by Politis (2010, 2013) in a related context. Te predictive residuals ave an empirical distribution tat as similar sape as tat of te fitted residuals but it as larger scale. Tis is a finite-sample penomenon only but it may elp alleviate te well-known penomenon of under-coverage of bootstrap confidence intervals. Our goal is to approximate te distribution of te confidence root: µ( ) m xf by tat of its bootstrap counterpart. RESAMPLING ALGORITHM FOR MODEL-BASED CONFIDENCE INTERVALS FOR µ( ) 1. Based on te {(Y t, ),t = 1,..., n} data, construct te estimates m x and s rom wic te fitted residuals e i, and predictive residuals ẽ i are computed for i = 1,..., n. 2. For te traditional model-based bootstrap approac (MB), let r i = e i n 1 j e j, for i = 1,..., n. For te predictive residual approac (PRMB) as in Politis (2010), let r i = ẽ i n 1 j ẽ j, for i = 1,..., n. a. Sample randomly (wit replacement) te residuals r 1,..., r n to create te bootstrap pseudo-residuals r1,..., r n wose empirical distribution is denoted by ˆF n. b. Create pseudo-data in te Y domain by letting Yi = m xi + s xi ri, for i = 1,..., n. c. Based on te pseudo-data {(Yt, ),t = 1,..., n}, re-estimate te functions µ(x) and σ(x) by te kernel estimators m x and s x (wit same kernel and bandwidts as te original estimators m x and s x ). d. Calculate a replicate of te bootstrap confidence root: m xf m. 3. Steps (a) (d) in te above are repeated B times, and te B bootstrap root replicates are collected in te form of an empirical distribution wit α quantile denoted by q(α). 4. Ten, a (1 α)100% equal-tailed confidence interval for µ( ) is given by: [m xf + q(α/2), m xf + q(1 α/2)]. (6) Remark 2.1 As in all nonparametric smooting problems, coosing te bandwidt is often a key issue due to te ever-looming problem of bias; te addition of a bootstrap algoritm as above furter complicates tings. Different autors ave used various tricks to account for te bias. For example, Härdle and Bowman (1988)

4 4 Dimitris N. Politis construct a kernel estimate for te second derivative µ (x), and use tis estimate to explicitly correct for te bias; te estimate of te second derivative is known to be consistent but it is difficult to coose its bandwidt. Härdle and Marron (1991) estimate te (fitted) residuals using te optimal bandwidt but te resampled residuals are ten added to an oversmooted estimate of µ; te bootstrapped data are ten smooted using te optimal bandwidt. Neumann and Polzel (1998) use only one bandwidt but it is of smaller order tan te mean square error optimal rate; tis undersmooting of curve estimates was first proposed by Hall (1993) and is peraps te easiest teoretical solution towards confidence band construction altoug te recommended degree of undersmooting for practical purposes is not obvious. Remark 2.2 An important feature of all bootstrap procedures is tat tey can andle joint confidence intervals, i.e., confidence regions, wit te same ease as te univariate ones. Tis is especially true in regression were simultaneous confidence intervals are typically constructed in te form of confidence bands; te details are well-known in te literature and are omitted due to lack of space. 3 Model-free nonparametric regression 3.1 Nonparametric regression witout an additive model We now revisit te nonparametric regression setup but in a situation were a model suc as eq. (2) can not be considered to old true (not even approximately). As an example of model (2) not being valid, consider te setup were te skewness and/or kurtosis of Y t depends on, and tus centering and studentization will not result in i.i.d. ness. Te dataset is still {(Y t, ), t = 1,..., n} were te regressor is univariate and deterministic, and te variables Y 1,Y 2,... are independent but not identically distributed. Define te conditional distribution D x (y) = P{Y f y = x} were (Y f, ) represents te random response Y f associated wit regressor. Attention still focuses on constructing an interval estimate of µ( ) = E(Y f ) = y D xf (dy). Trougout tis section, we will assume tat te function D x (y) is continuous in bot x and y. Consequently, we can estimate D x (y) by te local (weigted) empirical distribution ˆD x (y) = n ( x xi 1{Y i y} K ) ; (7) tis is just a N-W smooter of te variables 1{Y t y}, t = 1,..., n. Estimator ˆD x (y) enjoys many desirable properties, including asymptotic consistency, but is discontinuous as a function of y. To construct a continuous (and differentiable) estimator, let b be a positive bandwidt parameter and Λ(y) be a (differentiable) distribution function tat is strictly increasing, and define D x (y) = n ( ) y Yi Λ b Under regularity conditions, Li and Racine (2007, Teorem 6.2) sow tat ) K( x xi. (8) Var( D x (y)) = O( 1 n ) and Bias( D x (y)) = O( 2 + b 2 ) (9)

5 Bootstrap confidence intervals in nonparametric regression witout an additive model 5 assuming tat 0, b 0, n and n( 3 + b 3 ) = o(1); to minimize te asymptotic Mean Squared Error of D x (y), te optimal bandwidts are c n 1/5 and b c b n 2/5 for some positive constants c, c b. Recall tat te Y t s are non-i.i.d. only because tey do not ave identical distributions. Since tey are continuous random variables, te probability integral transform is applicable. If we let η i = D xi (Y i ) for i = 1,..., n, ten η 1,..., η n are i.i.d. Uniform(0,1). Of course, D x ( ) is not known but we can define u i = D xi (Y i ) for i = 1,..., n; (10) by te consistency of D x ( ), we can now claim tat u 1,..., u n are approximately i.i.d. Uniform(0,1). Using eq. (10) and following te Model-free Prediction Principle of Politis (2010), te quantity Π xf = n 1 n ˆD x 1 f (u i ) (11) was proposed as an L 2 optimal predictor of Y f, i.e., an approximation to te conditional expectation µ( ) = E(Y f ). Note tat ˆD xf (y) is a step function in y, and tus not invertible; te notation ˆD x 1 f denotes te quantile inverse. Alternatively, one could propose te quantity n 1 n D 1 (u i ) were a true inverse is used; te difference between te two is negligible, and definition (11) is straigtforward. Note tat Π xf is defined as a function of te approximately i.i.d. variables u 1,..., u n ; as suc, it may be amenable to te original i.i.d. bootstrap of Efron (1979). Two questions arise: (a) is te estimator Π xf quite different from te standard N-W estimator m xf? and (b) could m xf itself be bootstrapped using i.i.d. resampling? Te answers to tese questions are NO and YES respectively, due to te following fact. To motivate it, recall tat te N-W estimator m x can be expressed alternatively as n ( ) x xi 1 m x = Y i K = y ˆD x (dy) = ˆD x 1 (u)du. (12) 0 Te last equality in (12) is te identity y F(dy) = 1 0 F 1 (u)du tat olds true for any distribution F. Fact 3.1 Assume tat D x (y) is continuous in x, and differentiable in y wit derivative tat is everywere positive on its support. Ten, Π xf and m xf are asymptotically equivalent, i.e., n (Π xf m xf ) = o p (1) for any tat is not a boundary point. One way to prove te above is to sow tat te average appearing in (11) is close to a Riemann sum approximation to te integral at te RHS of (12) based on a grid of n points. Te law of te iterated logaritm for order statistics of uniform spacings can be useful ere; see Devroye (1981) and te references terein. Remark 3.1 Te above line of arguments indicates tat tere is a variety of estimators tat are asymptotically equivalent to m xf in te sense of Fact 3.1. For example, te Riemann sum M 1 M k=1 ˆD 1 (k/m) is suc an approximation as long as M n. A stocastic approximation can also be concocted as M 1 M 1 ˆD (W i ) were W 1,...,W M are i.i.d. generated from a Uniform(0,1) distribution and M n.

6 6 Dimitris N. Politis 3.2 Bootstrap algoritm for model-free confidence intervals Let ˆµ( ) denote our cosen estimator of µ( ) = E(Y f ), i.e., eiter m xf or Π xf, or even one of te oter asymptotically equivalent estimators discussed in Remark 3.1. Our goal is to approximate te distribution of te confidence root: µ( ) ˆµ( ) by tat of its bootstrap counterpart. Te algoritm reads as follows. RESAMPLING ALGORITHM FOR MODEL-FREE CONFIDENCE INTERVALS FOR µ( ) 1. Based on te {(Y t, ),t = 1,..., n} data, construct te estimates ˆD x ( ) and D x ( ), and use eq. (10) to obtain te transformed data u 1,..., u n tat are approximately i.i.d. Uniform (0,1). a. Sample randomly (wit replacement) te transformed data u 1,..., u n to create bootstrap pseudo-data u 1,..., u n. b. Use te quantile inverse transformation ˆD x 1 to create bootstrap pseudo-data in te Y domain, i.e., let Y n = (Y1,...,Y n) were Yt = ˆD x 1 t (ut ). Note tat Yt is paired wit te original design point; ence, te bootstrap dataset is {(Yt, ),t = 1,..., n}. c. Based on te pseudo-data {(Yt, ),t = 1,..., n}, re-estimate te conditional distribution D x ( ); denote te bootstrap estimates by ˆD x( ) and D x( ). d. Calculate a replicate of te bootstrap confidence root: ˆµ( ) ˆµ ( ) were ˆµ ( ) equals eiter y ˆD (dy) = ˆD (u)du or n 1 n 1 ˆD (u i ) according to weter ˆµ() was cosen as m xf or Π xf. 2. Steps (a) (d) in te above are repeated B times, and te B bootstrap root replicates are collected in te form of an empirical distribution wit α quantile denoted by q(α). 3. Ten, te Model-Free (MF) (1 α)100% equal-tailed, confidence interval for µ( ) is [ ˆµ( )+q(α/2), ˆµ( )+q(1 α/2)]. (13) Remark 3.2 An alternative way to implement step 1(a) of te above algoritm is: a. Generate bootstrap pseudo-data u 1,...,u n i.i.d. from an exact Uniform (0, 1) distribution. If te above coice is made, ten tere is no need to use eq. (10) to obtain te transformed data u 1,...,u n ; in tis sense, te smoot estimator D x ( ) is not needed, and te step function ˆD x ( ) suffices for te algoritm. Te downside to te above proposal is tat te option to use predictive u data is unavailable. To elaborate, recall tat Politis (2010) defined te model-free predictive u data as follows. Let D (t) denote te estimator D xt as computed from te delete-y t dataset, i.e., {(Y i, x i ), i = 1,...,t 1 and i = t + 1,..., n}. Now let u (t) t = D (t) (Y t ) for t = 1,..., n. (14) Te u (t) t variables are te model-free analogs of te predictive residuals ẽ t of Section 2. Remark 3.3 We can now define Predictive Model-Free (PMF) confidence intervals for µ( ). Te PMF Resampling Algoritm is identical to te above wit one exception; replace step 1(a) wit te following: a. Sample randomly (wit replacement) te predictive u data u (1) 1,..., u(n) n to create bootstrap pseudo-data u 1,...,u n.

7 Bootstrap confidence intervals in nonparametric regression witout an additive model 7 Remark 3.4 Recall tat te model-free L 1 optimal predictor of Y f is given by te median{ D 1 (u i )}; see Politis (2010, 2013). Terefore, by analogy to Fact 3.1, we ave: median{ D 1 (u i )} = D 1 (median{u i }) D 1 (1/2) since te u i s are approximately Uniform (0,1). Hence, if te practitioner wanted to estimate te median (as opposed to te mean) of te conditional distribution ofy f given, ten te local median D x 1 f (1/2), could be bootstrapped using i.i.d. resampling in te same manner tat median{ D 1 (u i )} can be bootstrapped. 4 Simulations 4.1 Wen a nonparametric regression model is true Te building block for te simulation in Section 4.1 is model (2) wit µ(x) = sin(x), σ(x) = 1/2, and errors ε t i.i.d. N(0,1) or two-sided exponential (Laplace) rescaled to unit variance. Knowledge tat te variance σ(x) is constant was not used in te estimation, i.e., σ(x) was estimated from te data. For eac distribution, 500 datasets eac of size n = 100 were created wit te design points x 1,..., x n being equi-spaced on (0, 2π), and N-W estimates of µ(x) = E(Y x) and σ 2 (x) = Var(Y x) were computed using a normal kernel in R. Confidence intervals wit nominal level α = 0.90 were constructed using te two metods presented in Section 2.2: Traditional Model-Based (MB) and Predictive Residual Model-Based (PRMB); te two metods presented in Section 3.2: Model-Free (MF) of eq. (13), and Predictive Model-Free (PMF) from Remark 3.3; and te NORMAL approximation interval (4). Te smooting kernel Λ in eq. (8) was taken to be te standard normal density. All required bandwidts were computed by L 1 cross-validation. For eac type of interval, te corresponding empirical coverage level (CVR) and average lengt (LEN) were recorded togeter wit te (empirical) standard error associated wit eac average lengt. Tables 1, 2, 3, and 4, summarize our findings, and contain a number of important features. Te standard error of te reported coverage levels over te 500 replications is By construction, tis simulation problem as some symmetry tat elps us furter appreciate te variability of te CVRs. To elaborate, note tat for any x [0, π] we ave µ(x) = µ(2π x) and te same symmetry olds for te derivatives of µ(x) as well due to te sinusoidal structure. Hence, te expected CVRs sould be te same for = 0.15π and 1.85π in all metods. So for te NORMAL case of Table 1, te CVR would be better estimated by te average of and 0.845, i.e., closer to 0.837; similarly, te PMF CVR for te same points could be better estimated by te average of and 0.878, i.e., Te NORMAL intervals are caracterized by under-coverage even wen te true distribution is Normal. Tis under-coverage is more pronounced wen = π/2 or 3π/2 due to te ig bias of te kernel estimator at te points of a peak or valley tat te normal interval (4) sweeps under te carpet. Te lengt of te NORMAL intervals is quite less variable tan tose based on bootstrap; tis is not surprising since te extra randomization from te bootstrap is expected to inflate te overall variances. Altoug regression model (2) olds true ere, te MB intervals sow pronounced under-coverage; tis is a penomenon well-known in te bootstrap literature. As previously mentioned, te predictive residuals ave generally larger scale tan te fitted ones. Consequently, te PRMB intervals are wider, and manage to partially correct te under-coverage of te MB intervals.

8 8 Dimitris N. Politis MB PRMB MF PMF Normal Table 1 Empirical coverage levels (CVR) of confidence intervals according to different metods at several points spanning te interval (0, 2π). Nominal coverage was 0.90, and sample size n = 100; error distribution: i.i.d. Normal. MB PRMB MF PMF Normal Table 2 (Average) lengts (LEN) wit standard errors below tem of te confidence intervals reported in Table 1. MB PRMB MF PMF Normal Table 3 As in Table 1 but wit error distribution: i.i.d. Laplace. Te performance of MF intervals is better tan tat of MB intervals despite te fact tat te former are constructed witout making use of eq. (2). However, as wit te MB intervals, te MF intervals also sow a tendency towards under-coverage. Te PMF intervals appear to nicely correct te MF under-coverage in te Normal case altoug in te Laplace case tey yield an over-correction. However, even wit tis over-correction, te PMF coverages are closer to te nominal in most entries of Tables 1 and 3 wit only a few exceptions in Table 3 were te PRMB intervals are more accurate. 4.2 Wen a nonparametric regression model is not true In tis subsection, we investigate te performance of different confidence intervals in te absence of model (2). For easy comparison wit Section 4.1, we will keep te same (conditional) mean and variance, i.e.,

9 Bootstrap confidence intervals in nonparametric regression witout an additive model 9 MB PRMB MF PMF Normal Table 4 As in Table 2 but wit error distribution: i.i.d. Laplace. MB PRMB MF PMF Normal Table 5 As in Table 1 but wit error distribution (15): non-i.i.d. skewed. MB PRMB MF PMF Normal Table 6 As in Table 2 but wit error distribution (15): non-i.i.d. skewed. we will generate independent Y data suc tat E(Y x) = sin(x), Var(Y x) = 1/2, and design points x 1,..., x 100 equi-spaced on (0, 2π). However, te error structure ε x = (Y E(Y x))/ Var(Y x) as skewness and/or kurtosis tat depends on x, tereby violating te i.i.d. assumption. For our simulation, we considered ε x = c xz +(1 c x )W c 2 x +(1 c x ) 2 (15) were c x = x/(2π) for x [0, 2π], and Z N(0, 1) independent of W tat will eiter be distributed as 1 2 χ to capture a canging skewness, or as 5 t 5, to capture a canging kurtosis; note tat EW = 0 and EW 2 = 1. Our results are summarized in Tables 5, 6, 7, and 8. Te findings are qualitatively similar to tose in Section 4.1. Te PMF intervals are te undisputed winners ere in terms of coverage accuracy. By contrast,

10 10 Dimitris N. Politis MB PRMB MF PMF Normal Table 7 As in Table 1 but wit error distribution (15): non-i.i.d. kurtotic. MB PRMB MF PMF Normal Table 8 As in Table 2 but wit error distribution (15): non-i.i.d. kurtotic. te NORMAL and te MB bootstrap intervals sow pronounced under-covarage; interestingly, tese are te two metods tat most practitioners use at te moment. References 1. Devroye, L. (1981). Laws of te iterated logaritm for order statistics of uniform spacings, Ann. Probab., vol. 9, no. 6, pp Efron, B. (1979). Bootstrap metods: anoter look at te jackknife, Ann. Statist., 7, Freedman, D.A. (1981). Bootstrapping regression models, Ann. Statist., 9, Hall, P. (1993), On Edgewort expansion and bootstrap confidence bands in nonparametric curve estimation, J. Roy. Statist. Soc., Ser. B, 55, Härdle, W. and Bowman, A.W. (1988). Bootstrapping in nonparametric regression: local adaptive smooting and confidence bands, J. Amer. Statist. Assoc., 83, Härdle, W. and Marron, J.S. (1991). Bootstrap simultaneous error bars for nonparametric regression, Ann. Statist., 19, Li, Q. and Racine, J.S. (2007). Nonparametric Econometrics, Princeton Univ. Press, Princeton NJ. 8. Linton, O.B., Cen, R., Wang, N. and Härdle, W. (1997). An analysis of transformations for additive nonparametric regression, J. Amer. Statist. Assoc., 92, Neumann, M. and Polzel, J. (1998). Simultaneous bootstrap confidence bands in nonparametric regression, J. Nonparam. Statist., 9, Politis, D.N. (2010). Model-free Model-fitting and Predictive Distributions, Discussion Paper, Department of Economics, Univ. of California San Diego. Retrievable from: ttp://escolarsip.org/uc/item/67j6s Politis, D.N. (2013). Model-free Model-fitting and Predictive Distributions, to appear as a Discussion Paper in te journal Test in 2013.