Bias Correction and Higher Order Kernel Functions

Transcription

1 Bias Correction and Higher Order Kernel Functions Tien-Chung Hu 1 Department of Mathematics National Tsing-Hua University Hsinchu, Taiwan Jianqing Fan Department of Statistics University of North Carolina Chapel Hill, N.C Abstract. Kernel density estimates are frequently used, based on a second order kernel. Thus, the bias inherent to the estimates has an order of O(h~). In this note, a method of corr~cting the bias in the kernel density estimates is provided, which reduces the bias to a smaller order. Effectively, this method produces a higher order kernel based on a second order kernel. For a kernel function K, the functions and 1 K(lc-l)( )/ f~oo K(lc-l)(z)/zdz z z are kernels of order k, under some mild conditions. lcompleted while visiting Department of Statistics, University of North Carolina, Chapel Hill. Abbreviated title. Bias Correction. AMS 1980 subject classification. Primary 62G05. Secondary 62G20. Key words and phrases. Bias correction, higher order kernel, kernel density estimate, nonparametrics. 1

2 1 Introduction Consider data that can be thought of as a random sample from a distribution having an unknown density. It is common practice to summarize the data with some kinds of statistics. Unless the form ofthe density is known, it is also very helpful to examine graphical representations and overall structures of the data. Kernel density estimates provide a useful tool for these purpose. See Silverman (1986), Eubank (1988), Miiller (1988), HardIe (1990) and Wahba (1990) for many examples of this, and good introductions to the general subject area. Great efforts have been made to select a bandwidth for a kernel density estimate based on a second order kernel, because such an estimate is easily explainable. A large amount of recent progress has been obtained on data based smoothing parameter selection, see. Rice (1984), Marron (1988), Hall et al. (1990), Jones, Marron and Park (1990), Chiu (1990), Fan and Marron (1990) and among others. Most of these bandwidth selectors have extremely fast rates of convergence to their theoretical optimal. However, since the second order kernel is used in the density estimate, the bias inherent to the estimate is always of order n- 2!5, no matter how good an automatic bandwidth selector is. This amount of bias may sometimes obscure the interesting features such the number of modes and height of the underlying density at the modes. In such cases, bias correction to the kernel density estimate is desirable. The discussion on this issue forms the core of this paper. For a set of random sample Xl,"', X n, a kernel density estimator is defined by (See Rosenblatt (1956)) in(x) = n~n ~I( (X ~nxj), (1.1) where K is a kernel function and h n is a bandwidth. We will concentrate on how the bia.s of (1.1) can be estimated for a nonrandom bandwidth h n. A method of correcting bias is given in section 2. Effectively, we give a method for constructing a higher order kernel based on a second order kernel. This provides a new insight to the effect of an higher order kernel. 2

3 There are several methods of constructing a higher order kernel. Schucany and Sommers (1977) propose a method based on the generalized jackknife to higher order kernels. A useful class of higher order kernel based on Gaussian density can be found in Wand and Schucany (1990). BerliJ:!-et (1990) using the idea of reproducing kernel in a Hilbert space to construct a class of higher order kernel and discuss its consequences. The optimalities of higher order kernels' are discussed in Millier (1984), Gasser et al. (1985), and Granovsky and Miiller (1990). Mathematically, most of methods above are directly targeted at finding a function ](r satisfying 1: ](r(z)dz =1,1: zqkr(z)dz = O,q = 1,,T -1, and 1: Izr Kr(z)ldz < 00. We take a different approach from the pioneering work by correcting bias directly. As a result of bias correction, a class of higher order kernel is constructed. Section 2 gives a precise formulation, and discussion, of the main results. Proofs are in section 3. 2 Main Results Let's illustrate how the bias of the kernel density can be corrected. Mathematical justifications are given in section 3. Observe that the kernel density estimator (1.1) is an unbiased estimator of J~oo f(x - hny)k(y)dy: E]n(x) =1:f(x - hny)k(y)dy. (2.1) Taking derivatives j times with respective to h n yields an unbiased estimate of the functional (viewing x as a fixed point) (2.2) and the unbiased estimate of OJ is given by A A ()i OJ(X) =-.fn(x), ah~ (2.3) 3

4 where in is the kernel density estimate defined by (1.1). Let's assume that the unknown density has k bounded continuous derivatives. Now, the Taylor expansion of f(x) yields f(x) = f(x-hny+hny) = k-l 1 L 1f(i)(x - hny)(hny)i + O(h~) j=o J. (2.4) Multiplying K(y) and then integrating both sides of (2.4) with respect to y, we have k-l (-hn)j f(x) = L., 8j(x) +O(hn), j=o J. where the fact J~oo K(y)dy = 1 is used. Thus, one can use OJ(x) to correct the bias k-l ( h )j 8o(x) - f(x) = - L -.; 8j(x) + O(h~) j=l J. of kernel density estimate (1.1). In other words, a bias-corrected estimate is defined as k-l (-hn)j fb(x) = L " 8j(x). j=o J. Let's give a simpler formula for the bias-corrected estimate (2.5). Lemma 2.1. If K( ) has bounded k th derivative, then k (2.5) where K,(Z) = zlk(l)(z), and OJ was defined by (2.3). By Lemma 2.1, the bias-corrected estimate (2.5) can be written as with (2.6) (2.7) 4

5 where the identity that was used. Thus, effectively the efforts of bias correction of kernel density estimate produce another kernel function Wk(') defined by (2.7). As intuitively expected, Wk( x) is a k th order kernel, which is justified by Theorem 1. If the kernel function K (.) satisfies 1: K(y)dy = 1 and 1:ly2k K(k)(y)ldy < 00, then the function Wk(') is a k th order kernel: 1: Wk(x)dx = 1, 1:x"Wk(x)dx = O,Jor s = 1"" k - 1, and (2.8) Since Wk( x) is a k th order kernel, it follows that Theorem 2. Let K satisfy the condition of Theorem 1 and let f( ) have k th bounded continuous derivative. Then, f(k)(x)100 A Efb(x) = f(x) - k! -00 x k K(x)dxh~(1 + 0(1)). Thus, the bias-corrected estimate does have the order of bias as expected. Since W k is a k th order kernel, a similar conclusion holds for the Mean Integrated Square Error (MISE). Remark 1. When K is symmetric, the kernel function (2.7) is also symmetric. In such a case, if k = 2r - 1 is an odd integer, then W 2r - 1 is also a kernel of order 2r which can easily justified by (2.8), and satisfies 5

6 Example 1. Let's take a standard normal density as a kernel function. Then, 1 K(x) = </>(x) = /icexp(-x 2 /2) y21l" </>(I)(x) = (-1)IH,(x)</>(x), where H,(x) is the Hermite polynomial of order 1. Thus, by (2.7), is a k th order kernel with k-1 ( k ) Wk(X) = L 1= (-x)ih,(x)</>(x)/l! if k = 2r - 1 if k = 2r. (2.9) Note that also that W 2r - 1 ( x) is a kernel of order 2r with These kernel functions are different from the kernel functions derived by Wand and Schucany (1990). The following table list the first few kernel functions (2.9), which is computed by a computer program. k 2 (_x 2 + 2)t/>(x) 3 (x - 7x 2 + 6)t/>(x)/2 4 (_x x - 48x2 + 24)t/>(x)/6 5 (x 8-26x x - 360x )t/>(x)/24 6 (_x x 8-495x x x )t/>(x)/120 7 (x 12-57r o +1095x8-8625x x x )t/>(x)/ It appears that the higher kernels produced by (2.9) are quite complicated, which make them less useful. However, a simple method is possible. Observe that for 1 2:: 1 i:1((1)(x )dx = O. 6

7 By integration by parts, we obtain 100 x'k(k-l)(x)dx = { 0, -00 (_l)(k-l)(k - I)!, Thus, if J~oo IK(k-l)(x)/xldx < 00, then by (2.10) if 1= 0".., k - 2 ifl =k-1 (2.10) (2.11) is a kernel of order k. Theorem 3. Let K (x) be a kernel function satisfying and x'k(l-l)(x) --+ 0, as Ixl--+ oo,lor 1= 1,'",k-1. Then, Kk defined by (2.11) is a kernel function satisfying if 1= 1,,,,,k - 1 if 1 = k Remark 2. When K( ) is a even function, then K(2r-l)(0) = O. Thus, if K(2r)(0) exits. In other words, the function!(2r-l)(x)/x is well defined at point x = O. Consequently, if J~oo IK(2r-l)(x)ldx < 00, then J~oo IK(2r-l)(x)l/xdx < 00 and K 2r- 1(X) is a kernel of order 2r, if other conditions of Theorem 3 is satisfied. Example 1. (continued) If K (x) = </>(x), then </>(2r-l)(X)/[X1:</>(2r-l)(x)/xdx] = ( _ly</>(2r-l)(x) 2 r - 1 (r - 1)!x. 7

8 is a kernel function of order 2r. The result is found in Wand and Schucany (1990). See also Wand and Schucany (1990) for the kernel functions K2r-1 (x), r = 1",,,5. Example 2. Let K(x) = ~ 1;X 2 be the standard Cauchy density. Then, (1 + x 2 )K(2r-1)(x) + 2(2r - l)xk(2r-2)(x) + (2r - 1)(2r - 2)K(2r-3)(x) = O. The recursive formula is used to compute higher order kernels. The following table gives the higher order kernel function resulting from (2.11). The renormalization constants and f~oo K~r-1(x)dx are computed by using numerical integration. Table 2: Cauchy density based kernels of order 2-8 order 2r K2r-l (x) f~~ K~r_l(x)dx 2 1«1 ';x2) (x~-I) «1 +x 2 ) (3x -10x~+3) (l+x2) S0(x 6-7x + 7x~ -1) l~x Example 3. Let Kn(x) = cn(l- x 2 )+. be a kernel function, where Cn is a normalization constant. Then, by (2.11) _ -1 n j ( n ) (2 J ')' 2j-2r K n,2r-1 - Cn,r ~(-1) j (2j _ 2r + 1)!x l[1xl$11' for r = 1"",[n/2], h C - 2"n ( 1)j ( n ) (2j)! Th r 11. T bl' h 1 were n,r - L.Jj=r - j (2j-2r+1)!(2j-2r+1)' e 10 owmg a e gives t e resu t of K 8,2r-1(X) for x E [-1,1]. Table 3: Polynomial based kernels of order 4-8 order 2r KS.2r-l (x) f~<>o K~r_l(x)dx 4 ~(1 - x 2 )t(1-5x 5 ) nhr(1 - x2)~(3-26x 2 +39x 4 ) j;\lijil (1 - x 2 )+ (35-385x x 4-715x6) If one is interested in finding a fourth order kernel, a simpler one would be K 4.3(X) = 8

9 3 Proofs 3.1 Proof of Lemma 2.1 Since differentiation is a linear operator, we need only to show for the case n = 1. We use the induction to prove the result. Note that Lemma 2.1 holds for j = O. Assume that Lemma 2.1 holds for j = m. Then, A 0 Om+I(X) A = oh n Om(X) = (-~=::m! t ( m ) [KI+I(X ~ Xl) +(l + m + l)ki(x ~ Xl)] Il! n 1=0 1 n n (_1)m+lm! [ m + 1 x - Xl X - Xl = h~+2 (m + 1)!Km+ l ( h n ) + (m + 1)Ko( h n ) Eam,IK1+I(x ~ Xl)], 1=0 n where Combining the last two displays yields that ij ()_(-1)m+I(m+1)!~(m+1)K(X-XI)ll' m+i x - hm+2 L..J 1 h.. n 1=0 1 n Thus, Lemma 2.1 holds for 1= m Proof of Theorem 1 Let's give two simple Lemmas, which will be used in the proof of Theorem 1. min(r k) ( r)( s ) (r +s ) Lemma 3.1. Li=~(O,k-") i k _ i = k. 9

10 Proof. Think of products consisting of T good and 8 bad products. Choosing k products is equivalent to selecting i good products and k - i bad products, for all possible i. Lemma 3.2. Under the condition of Theorem 1, where T 8 = f~oo x 8 K(x)dx. Proof. Integration by parts j times yields the results. Proof of Theorem 1. By Lemma 3.2 and the definition of Wk, we have for 1 ~ 8 ~ k _ ~ ( k ) ( ) T 8 L.J (-1) 1=1 1 8 (3.1) By Lemma 3.1, the summation in (3.1) can be written as t( -1)/-1 ( k ) mii:8) ( 1 ) ( 8-1 ) 1=1 1 i=1 i i - 1 ; t.~(-1)1-1 (; )( : )( : =~ ) = tt(-1)/_l( k) (k-i) (8-1) i=1 I=i i k - 1 i - 1 = t [E( _1)l+i-l ( k - i )] ( ~ ) ( ~ - 1 ),=1 1=0 1 t t - 1 (3.2) Note that L(-1)1 - t = 1=0 1 k-i (k' ) { 0, if i < k 1, if i = k 10

11 Thus, by (3.1) and (3.2) if s < k if s = k Similarly, by (3.1) we have This completes the proof.. L W,<x)dx = t.<-l)'( ~)= 1. ACKNOWLEDGEMENTS We would also like to express our sincere thanks to Professor J.S. Marron for many helpful discussions. References [1] Berlinet, A. (1990) Reproducing kernels and finite order kernels. Manuscript. [2] Chiu, S.T. (1990). Bandwidth selection for kernel density estimation, Ann. Statist., to appear. [3] Eubank, R. 1. (1988). Spline Smoothing and Nonparametric Regression. Dekker, New York. [4] Fan, J. and Marron, J.S. (1990). Best possible constant for bandwidth selection. Institute of Mimeo Series #2041, University of North Carolina, Chapel Hill. [5] Gasser, T., MUller, H.G., and Mammitzsch, V. (1985). Kernels for nonparametric curve estimation. J. Roy. Statist. Soc. Ser. B, 47, [6] HardIe, W. (1990). Applied Nonparametric Regression. Cambridge University Press, Boston. [7] Hall, P., Sheather, S. J., Jones, M.C. and Marron, J.S. (1990). On optimal data-based bandwidth selection in kernel density estimation. Biometrika, to appear. [8] Jones, M. C., Marron, J. S. and Park, B. U. (1990). A simple root n bandwidth selector. Annals of Statistics, to appear. [9] Marron, J. S. (1988). Automatic smoothing parameter selection: a survey. Empirical Economics, 13,

12 [10] MUller, H.G. (1984). Smooth optimum kernel estimators of densities, regression curves and modes. Ann. Statist., 12, [11] Milller, H.G. (1988). Nonparametnc Analysis of Longitudinal Data. Springer Verlag, Berlin. [12] Granovsky, B. and Milller, H.-G. (1990). Optimizing kernel methods for the nonparametric estimation of functions and characteristic points: a unifying variational principle. Manuscript. [13] Rice, J. (1984). Bandwidth choice for nonparametric regression. Ann. Statist., 12, [14] Rosenblatt, M. (1956). Remarks on some nonparametric estimates ofa density function, Ann. Math. Statist., 42, [15] Silverman, B. W. (1986). Density Estimation for Statistics and Data Analysis. Chapman and Hall, London. [16] Wahba, G. (1990). Spline Models for Observational Data. SIAM, Philadelphia. [17] Wand, M. P. and Schucany, W.R. (1990). Gaussian-based kernels. Canadian J. Statist., 18, [18] Schucany, W. R. and Sommers, J. P. (1977). Improvement of kernel type density estimators. J. Amen. Statist. Assoc., 72,