THE EFFICIENCY OF BIAS CORRECTED ESTIMATORS FOR NONPARAMETRIC KERNEL ESTIMATION BASED ON LOCAL ESTIMATING EQUATIONS

Transcription

1 THE EFFICIENCY OF BIAS CORRECTED ESTIMATORS FOR NONPARAMETRIC KERNEL ESTIMATION BASED ON LOCAL ESTIMATING EQUATIONS Göra Kauerma, Marlee Müller ad Raymod J. Carroll April 18, 1998 Abstract Stuetzle ad Mittal (1979) for ordiary oparametric kerel regressio ad Kauerma ad Tutz (1996) for oparametric geeralized liear model kerel regressio costructed estimators with lower order bias tha the usual estimators, without the eed for devices such as secod derivative estimatio ad multiple badwidths of differet order. We derive a similar estimator i the cotext of local (multivariate) estimatio based o estimatig fuctios. As expected, this lower order bias is bought at a cost of icreased variace. Surprisigly, whe compared to ordiary kerel ad local liear kerel estimators, the bias corrected estimators icrease variace by a factor idepedet of the problem, depedig oly o the kerel used. The variace icrease is approximately 40% ad more for kerels i stadard use. However, the variace icrease is still less tha that icurred whe udersmoothig a local quadratic regressio estimator. Key words ad phrases: Bias Reductio; Bootstrap; Estimatig Equatios; Geeralized Liear Models; Local Liear Regressio; Noparametric Regressio. Short title. Proportioal Variace Iflatio for Bias Reductio Göra Kauerma is Wisseschaftlicher Assistet, Fachgebiet Statistik ud Wirtschaftsmathematik, Techische Uiversität Berli, Fraklistraße 28/29, D Berli. Marlee Müller is Wisseschaftliche Assisteti, Istitut für Statistik ud Ökoometrie, Humboldt Uiversität zu Berli, Spadauerstraße 1, D Berli. Raymod J. Carroll is Professor of Statistics, Nutritio ad Toxicology, Departmet of Statistics, Texas A&M Uiversity, College Statio, TX Carroll s research was supported by a grat from the Natioal Cacer Istitute (CA 57030), ad was partially completed while visitig the Istitut für Statistik ud Ökoometrie, Soderforschugsbereich 373, Humboldt Uiversität zu Berli, with partial support from a seior Alexader vo Humboldt Foudatio research award.

2 1 INTRODUCTION Noparametric fuctio estimatio is greatly complicated by the problem of bias, ad this has importat cosequeces for iferetial methods such as cofidece bads ad test statistics. For example, cosider the problem of ordiary oparametric regressio to estimate the regressio fuctio Θ(x 0 ) of a respose Y o a predictor X evaluated at a value x 0 :Θ(x 0 )=E(Y X=x 0 ). Local liear regressio estimators are solutios to the locally weighted estimatig equatio 0= w i (x 0 ) Y i β 0 β 1 (X i x 0 )} (1,X i x 0 ) T, (1) where the itercept is the regressio estimate Θ(x 0,h) ad the weights w i (x 0 ) icrease with the distace of X i to x 0. Cosider the case that the weights are kerel weights K h (X i x 0 ) = h 1 K(X i x 0 )/h} with a symmetric kerel desity fuctio K( ). Let Θ (2) (x) be the secod derivative of the fuctio Θ(x), let f x (x) be the desity fuctio of X ad ad suppose that the variace of Y give X is costat ad equal to σ 2. The it is well kow (Fa & Gijbels, 1996 is a coveiet referece) that if x 0 is iterior to the support of X, the bias ad the variace are give approximately by } bias Θ(x 0,h) } var Θ(x 0,h) (1/2)h 2 Θ (2) (x 0 ) σ 2 hf x (x 0 )} 1 z 2 K(z)dz; (2) K 2 (z)dz. (3) Results similar to (2) (3) hold for ordiary kerel regressio, geeralized liear models (Fa, Heckma & Wad, 1995) ad more geerally for local estimatig equatios (Carroll, Ruppert & Welsh, 1998), see sectio 2 for defiitios. I practice, oe must estimate the badwidth, ad this is usually doe by miimizig a estimate of the mea squared error based o (2) (3), see Ruppert (1998) for review. For these badwidth estimators, for which h is proportioal to 1/5, the squared bias ad variace are set equal. The et effect is that while the estimators are optimal i a mea squared error sese, the (squared) bias is approximately the same as the variace. Thus, while optimal badwidth estimators give good fuctio estimates, the fact that their squared bias approximately equals the variace has importat implicatios for ifereces. example, if oe igores the bias, cofidece itervals for the regressio fuctio at a poit x 0 have coverage levels asymptotically smaller tha the omial. To overcome this problem, a stadard techique is to estimate the secod derivative fuctio Θ (2) (x 0 ), ad the subtract the estimated bias from Θ 0 (x 0,h), a techique employed either effectively or explicitly by Härdle & Bowma 1 For

3 (1988), Härdle & Marro (1991), Eubak & Speckma (1993) ad Hall (1993), amog others. The result is to remove the bias, ad asymptotically ot chage the variace, because the h 2 i (2) is of small order, ad ay error i estimatig the secod derivative is of eve lower asymptotic order. There are a few difficulties with this geeral approach. First, all the above refereced papers iclude the eed for a secod badwidth, which we will call g. The secod badwidth is eeded effectively to estimate the secod derivative fuctio ad hece ecessarily coverges to zero more slowly tha the mai badwidth h. The secod badwidth g must be estimated as well, ad this if usually much harder to accomplish tha estimatig h itself. Secod, outside the cotext of estimatig a mea fuctio (Kauerma & Tutz, 1996; Carroll, Ruppert & Welsh, 1998), while it is possible to work out formulae similar to (2) (3), it is either clear how best to estimate the vector of secod derivative fuctios (ad their secod badwidths) or whether such estimatio leads to reasoable bias reductio properties i small samples. To address these cocers, Kauerma & Tutz suggest a alterative method of bias reductio. While the method is give i its estimatig equatio form i sectio 2, here we derive it i ordiary kerel regressio. The estimated regressio fuctio has the algebraic expressio Θ(x 0,h)= 1 w i (x 0 )Y i / 1 w i (x 0 ), ad from this oe deduces that Θ(x 0,h) Θ(x 0 )=b +E = w i (x 0 ) Θ(X i ) Θ(x 0 )} w i (x 0 ) + w i (x 0 ) Y i Θ(X i )}. w i (x 0 ) The term b determies the bias, while the term E is a mea zero radom variable which determies the variace. The simplest device to estimate bias is to plug i Θ( ) forθ( )ib, leadig to the bias corrected estimator Θ c (x 0,h) = Θ(x 0,h) w i (x 0 ) Θ(X i ) Θ(x } 0 ) / w i (x 0 ) (4) = w i (x 0 ) 2 Θ(x 0 ) Θ(X } i ) / w i (x 0 ). The estimator (4) is a bias corrected estimator which was called the twicig estimator by Stuetzle ad Mittal (1979): it has bias of lower order tha the usual kerel estimator. discuss the geeralizatio of this estimator to the estimatig equatio cotext. I sectio 2 we Oe would expect that bias corrected estimators such as (4) should have larger variace tha the ordiary estimator. We cosider this questio for estimatig equatios usig local average ad local liear kerel methods with symmetric kerel K( ). Our coclusio is surprisig: idepedet of the problem, bias corrected estimators are more variable by a factor depedig oly o the 2

4 kerel, amely c(k) = K(z2 )K(z 3 )2K(z 1 ) K(z 2 z 1 )}2K(z 1 ) K(z 3 z 1 )dz 1 dz 2 dz 3 } K 2. (5) (z)dz For the Gaussia kerel, c(gaussia) = 1.44, while for the Epaechikov kerel, c(epaechikov) = A alterative device to remove bias is direct udersmoothig, e.g., usig a local quadratic regressio with a badwidth h 1/5 from local liear regressio. The icrease i the variace, d(k) say, from this ca be computed from results of Ruppert & Wad (1994). For the Gaussia kerel, d(gaussia) = 1.69, while for the Epaechikov kerel, d(epaechikov) = variace icreases are larger tha for the bias corrected estimators (4). 2 LOCAL ESTIMATING EQUATIONS AND ESTIMATES OF BIAS I parametric problems, estimatio of a possibly vector valued parameter Θ is typically based o a ubiased estimatig fuctio ψ( ), so that if the data are geerically deoted by Ỹi (i =1,..., ), the Θ is the solutio to the estimatig equatio 0= 1 ψ(ỹi, Θ). By a ubiased estimatig fuctio, we mea that Eψ(Ỹ,Θ) = 0. The choices of ψ( ) arewell kow. For example, whe the data Both Ỹ cosist oly of a respose Y adθisthemeaofy, ψ(ỹ,θ) = Y Θ. I geeralized liear models, the data Ỹ cosist of a respose Y ad covariates Z. The mea is µ(z T Θ), the variace is proportioal to V (Z T Θ), ad the estimatig fuctio } is the quasilikelihood score ψ(ỹ, Θ) = Zµ(1) (Z T Θ) Y µ(z T Θ) /V (Z T Θ), where µ (1) ( ) isthe first derivative of the fuctio µ( ). Noparametric regressio ca be thought of as a varyig coefficiet model (Kauerma & Tutz, 1996), where the coefficiet Θ varies with a covariate X. A estimate of Θ(x 0 ) ca be obtaied usig local polyomials of order p 0 as follows. Defie G p (v) =(1, v,..., v p ) T, ad let the weights w i (x 0 ) be as i Sectio 1. Suppose that Θ(x) is a vector of legth q. Let is the Kroecker product, so that for example (a, c) (b1,b2,b3) = (ab 1,ab 2,ab 3,cb 1,cb 2,cb 3 ). Defie B T =(β0 T,..., βt p ). The Θ(x 0 ) is the itercept β 0 i the solutio to the equatio 0= 1 } ] w i (x 0 )G p (X i x 0 ) ψ [Ỹi, G T p (X i x 0 ) I q B. (6) 3

5 For these local polyomial estimators, formulae similar to (2) (3) hold. I fact, if p = 1, the bias is still give by (2), while the variace is the same as (3) except that σ 2 is replaced by } g 1 (x)l(x)g T (x), where g T ( ) is the traspose of g 1 ( ), χ(ỹ, Θ) = ( / Θ)ψ(Ỹ, Θ), g(x) = E[χỸ,Θ(X)} X = x] adl(x)=e[ψỹ,θ(x)}ψt Ỹ, Θ(X)} X = x]. A bias corrected estimator is costructed as follows. Defie B (x 0 )= 1 The, by a Taylor series expasio, [ }] w i (x 0 ) G p (X i x 0 )G T p (X i x 0 ) χ Ỹi, Θ(X i ). Θ(x 0 ) Θ(x 0 ) b + E, (7) where E = e p B 1 (x 0 ) 1 } w i (x 0 )G p (X i x 0 ) ψ Ỹi, Θ(X i ) b = e p B 1 0) 1 ( } ] }) w i (x 0 )G p (X i x 0 ) ψ [Ỹi, G T p (X i x 0 ) I q B ψ Ỹi, Θ(X i ). with e p be the q q(p + 1) matrix of zeros except that the first q q submatrix is the idetity matrix. Just as i (4), the idea is to estimate the terms i the bias, leadig to the estimator Θ c (x 0 ) = Θ(x 1 0 ) e p B (x 0)C (x 0 ); (8) C (x 0 ) = 1 ( } ] w i (x 0 )G p (X i x 0 ) ψ [Ỹi, G T p (X i x 0 ) I q B ψ Ỹi, Θ(X }) i ) ; B (x 0 ) = 1 [ w i (x 0 ) G p (X i x 0 )G T p (X i x 0 ) χ Ỹi, Θ(X }] i ). I the case of the likelihood score ψ( ) with local averages, p =0,G p (v) = 1 ad the bias corrected estimator derived from (8) reproduces the bias corrected estimator of Kauerma & Tutz. I the appedix, we show that for local averages (p = 0) ad local liear smoothig (p =1), with badwidth h 1/5 the bias corrected estimator is always more variable asymptotically tha the ucorrected estimator (6) by the factor (5). 3 DISCUSSION There are two geeral ways to correct for bias i oparametric regressio: (a) estimate the secod derivative fuctio directly ad subtract a multiple of it from the usual estimator; ad (b) bias correct idirectly either by udersmoothig (applyig a local liear badwidth to a local quadratic 4

6 estimator) or the twicig techique. The major difficulty with method (a) is the eed for a secod badwidth. We have show that methods (b) are more variable tha method (a), by a costat factor idepedet of the problem. Betwee the two possibilities i method (b), the twicig estimator is asymptotically less variable. The twicig estimator shows aother advatage with respect to applicatio. If the bias corrected estimators (4) ad (8) are used, it is simple to estimate their variace: take ay variace estimator for a ucorrected regressio fuctio which is already typically available i the literature, ad multiply it by the variace iflatio factor (5). REFERENCES Carroll, R. J., Ruppert, D. & Welsh, A. (1998). Noparametric estimatio via local estimatig equatios, with applicatios to utritio calibratio. Joural of the America Statistical Associatio, to appear. Eubak, R. L. & Speckma, P. L. (1993). Cofidece bads i oparametric regressio. Joural of the America Statistical Associatio, 88, Fa, J., Heckma, N. E. & Wad, M. P. (1995). Local polyomial kerel regressio for geeralized liear models. Joural of the America Statistical Associatio, 90, Fa, J. & Gijbels, I. (1996): Local Polyomial Modelig ad its Applicatios. Lodo: Chapma & Hall. Hall, P. (1993). O Edgeworth expasio ad bootstrap cofidece bads i oparametric curve estimatio. Joural of the Royal Statistical Society, Series B, 55, Härdle, W. & Bowma, A. W. (1988). Bootstrappig oparametric regressio: local adaptive smoothig ad cofidece bads. Joural of the America Statistical Associatio, 83, Härdle, W. & Marro, J. S. (1991). Bootstrap simultaeous error bars for oparametric regressio. Aals of Statistics, 19, Kauerma, G. & Tutz, G. (1996). O model diagostics ad bootstrappig i varyig coefficiet models. Submitted. Ruppert, D. (1998). Empirical bias badwidth selectio. Joural of the America Statistical Associatio, to appear. Ruppert, D., & Wad, M. P. (1994). Multivariate locally weighted least squares regressio. Aals of Statistics, 22, Stuetzle, W. & Mittal, Y. (1979). Some commets o the asymptotic behavior of robust smoothers. I Smoothig Techiques for Curve Estimatio, editors T. Gasser ad M. Roseblatt, Spriger Lecture Notes, 757,

7 Appedix A PROOFS OF THEOREMS I what follows, we will assume that h 1/5, ad we will use the otatio to mea equality to terms of order o p (h 2 ). Recall that f x ( ) is the desity of X. The kerel fuctio K( ) is symmetric. Defie g(x) =E[χỸ,Θ(X)} X = x] adl(x)=e[ψỹ,θ(x)}ψt Ỹ, Θ(X)} X = x]. Our argumet here is heuristic but ca be justified uder strog regularity coditios, e.g. X ad K are compactly supported, the desity of X is bouded away from zero o its support, etc. We provide details i the case of local averages, i.e., p = 0 i (6). The argumets are similar i the local liear case (p = 1) because the expasio (9) give below still holds i this case, but with (10) replaced by (2). Carroll, et al. (1998) show that Θ(x) Θ(x) C (x) h 2 r(x)+d (x), where (9) B (x) = 1 K h (X i x)χỹi, Θ(x)}; C (x) = B 1 (x) 1 K h (X i x)ψỹi, Θ(x)}; D (x) = B 1 (x) 1 K h (X i x)ψỹi, Θ(X i )}; r(x) = (1/2)Θ (2) (x)+f x (1) (x)θ(1) (x)/f x (x)} z 2 K(z)dz. (10) The variace of Θ(x) is approximately Σ=hf x (x)} 1 } K 2 (z)dz g 1 (x)l(x)g T (x), where g T ( ) is the traspose of g 1 ( ). Usig these expasios, we have that Θ c (x 0 ) Θ(x 0 ) B 1 (x 0 ) 1 [ K h (X i x 0 ) ψỹi, Θ(X i )} ψỹi, Θ(x 0 )} + χỹi, Θ(x 0 )} Θ(x ] 0 ) Θ(x 0 )} B 1 (x 0) 1 K h (X i x 0 )χỹi, Θ(x 0 )}[2 Θ(x 0 ) Θ(x 0 )} Θ(X i ) Θ(x 0 )}] +B 1 (x 0) 1 K h (X i x 0 )[ψỹi, Θ(X i )} ψỹi,θ(x 0 )}] B 1 (x 0 ) 1 K h (X i x 0 )χỹi, Θ(x 0 )}2D (x 0 ) D (X i )} +h 2 B 1 (x 0) 1 K h (X i x 0 )χỹi, Θ(x 0 )}2r(x 0 ) r(x i )} 6

8 +B 1 (x 0 ) 1 K h (X i x 0 )ψỹi, Θ(X i )} ψỹi,θ(x 0 )}} = G 1 + G 2 + G 3. It is easily show that G 2 h 2 r(x 0 ). Further, by calculatios of first ad secod momets, it follows that G 3 h 2 r(x 0 ). Fially, we have that B (x) =f x (x)g(x)+o p (h 2 ). From this, it follows that Θ c (x 0 ) Θ(x 0 ) f x (x 0 )g(x 0 )} 1 1 Now defie The it is easily see that L i (x) =f x (x)g(x)} 1 1 Θ c (x 0 ) Θ(x 0 ) f x (x 0 )g(x 0 )} 1 1 K h (X i x 0 )χỹi, Θ(x 0 )}2D (x 0 ) D (X i )}. j=1 K h (X j x)ψỹj, Θ(X j )} K h (X i x 0 )χỹi, Θ(x 0 )}2L i (x 0 ) L i (X i )}. (11) Now write out (11) ito a double sum i i ad j, iterchage the idices, elimiate the terms i which i ad j are equal sice they are of order (h) 1 = o p (h 2 ) ad use the assumptio that K( ) is symmetric to get that Θ c (x 0 ) Θ(x 0 ) f x (x 0 )g(x 0 )} 1 1 ψỹi, Θ(X i )}M (x 0,X i ), where (12) M (x 0,X i ) = 1 j=1,j i K h (X j x 0 )χỹj, Θ(x 0 )} 2Kh (X i x 0 ) f x (x 0 )g(x 0 ) K } h(x j X i ) f x (X j )g(x j ) Recall the defiitio of c(k) i (5). The right side of (12) is a mea zero radom variable, ad its variace is easily calculated to be c(k)σ1 + o(1)}, as claimed. 7