The Priestley-Chao Estimator

Transcription

1 Te Priestley-Cao Estimator In tis section we will consider te Pristley-Cao estimator of te unknown regression function. It is assumed tat we ave a sample of observations (Y i, x i ), i = 1,..., n wic are described by te model Y i = m(x i ) + ɛ i, i = 1,..., n were m( ) is te unknown conditional mean function wic as k 2 continuous derivatives on (a, b) and te errors (ɛ 1,..., ɛ n ) are uncorrelated wit zero mean and constant variance, σ 2. It will also be assumed tat te design variables (x 1,..., x n ) are equally-spaced in [a, b] so tat x i = i δ i = 1,..., n were δ = (b a)/n. It is required to estimate m nonparametrically using te available data. Te estimator wic we consider ere is tat proposed by Priestley and Cao (1972) wic is defined by ˆm(x) = δ n ( ) x xi K Y i for x (a, b) K( ) is called te kernel function wic is assumed to be symmetric about zero, be suc tat K(u) 2 du < and ave a finite second moment (ie. u 2 K(u)du = σk 2 < ). A common coice is a symmetric probability density function (pdf) suc as te standard Normal pdf or a symmetric Betatype pdf on te finite interval [, 1]. Te constant is called te smooting parameter or bandwidt and controls ow spread out te kernel functions are about eac x i. Te Priestley-Cao estimator is a weigted average of te response variables Y 1,..., Y n and te weigt associated wit Y i is K( x x i ) wit its actual value being determined by te proximity of x to x i relative to te value of. It is a linear function of te Y i s and ence is referred to as a linear smooter. Note tat if te data are not equally-spaced ten te estimator is expressed as ˆm(x) = 1 n ( ) x xi (x i x i )K Y i for x (a, b) i=2 Figure 1 below sows te a scatterplot of te simulated regression data wit four Pristley-Cao estimates calculated using different smooting parameters superimposed. Te kernel used was te biweigt function defined by { 15 K(t) = (1 16 t2 ) 2, < t < 1 0, oterwise 1

2 Wen = 0.3 (te red line) te estimate is quite wiggly and overfits te data but, as te value of increases, te estimates become more smoot and increasingly oversmoot te features in te data. Subjectively, te estimate wit = 0.1 appears to be closest to te true underlying regression function. Simulated Regression Data wit Priestley Cao Smoots y x Figure 1: Scatterplot of te simulated regression data wit fitted Priestley- Cao regression estimates. True curve - black line; = red line; = blue line; = green line; = orange line. A biweigt kernel was used in eac case. Exact Mean and Variance of ˆm(x). It will now be assumed, witout loss of generality, tat a = 0 and b = 1 so tat δ = 1/n. Te exact mean and variance of te estimator ˆm at te point x are given by and E[ ˆm(x)] = 1 n n ( x xi K ) m(x i ) V [ ˆm(x)] = σ2 n 2 2 n ( ) x xi 2 K Te E[ ˆm(x)] can be seen to be a smooted version of te true regression function values. 2

3 To see wat is driving te form of te mean and variance we will look at deriving asymptotic expressions for tem bot. Asymptotic Mean and Variance of ˆm(x). Here, we will make te following furter assumptions: (i) Te kernel K is symmetric about zero and is supported on [, 1]. (ii) Te bandwidt = (n) is a sequence satisfying 0 and n as n. (iii) Te point x at wic te estimation is taking place satisfies < x < 1 for all n n 0, were n 0 is fixed. Now, E[ ˆm(x)] = Let u = x t E[ ˆm(x)] = = 0 ( ) 1 x t K m(t)dt + O(n ) as n ie. as δ 0. du = dt x/ (1 x)/ x/ (1 x)/ dt = du. Hence, K(u)m(x u)du + O(n ) K(u) [ m(x) um (1) (x) + u2 2 ] du + o( 2 ) + O(n ) Provided < x < 1 ten, as 0, x/ and (1 x)/. Terefore, E[ ˆm(x)] = m(x) K(u)du m (1) (x) uk(u)du + 2 = m(x) + 2 u 2 K(u)du + o( 2 ) + O(n ) as 0 K(u)du m (1) (x) uk(u)du u 2 K(u)du + o( 2 ) + O(n ) as 0, using a Taylor series expansion of m(x u) and also te fact tat te support of K is [, 1]. Hence, E[ ˆm(x)] = m(x) + 2 σ 2 K + o( 2 ) + O(n ) as n and 0 3

4 Terefore, as n and 0 we ave tat E[ ˆm(x)] m(x) + 2 σ 2 K Tis expression indicates tat te bias[ ˆm(x)] = E[ ˆm(x)] m(x) will be large wen te value of m (2) (x) is large wic will occur at te peaks and trougs of te function m. We also ave tat te V [ ˆm(x)] = σ2 n ( ) x xi 2 K n 2 2 ( ) = δσ 2 1 x t 2 0 K dt + O(n ) as n 2 = δσ2 ( ) 1 x t 2 0 K dt + O(n ). Now, again let u = x t du = dt dt = du. Hence, V [ ˆm(x)] δσ2 = δσ2 = O(δ/) since K(u) 2 du as n K(u) 2 du as n K(u) 2 du <. ie. V [ ˆm(x)] = O( 1 n ); since δ = 1/n. Te mean squared error (mse) of ˆm at te point x is mse[ ˆm(x)] = [bias[ ˆm(x)]] 2 + V [ ˆm(x)] 4 4 m(2) (x) 2 σk 4 + δσ2 K(u) 2 du 4

5 Notes: (i) ˆm(x) is terefore consistent in mse provided 0 and δ = 1 0 as n. ie. n as n. n (ii) It can be seen tat one strategy for reducing te bias of te estimator is to make te bandwidt,, small. However, tis will increase te value of te variance of ˆm(x). Also, increasing to reduce te variance will ave te effect of increasing te bias. Hence, tere is a trade-off between te bias and variance of te estimator wic is controlled by te value of. (iii) We can use calculus to find an expression for tat value of wic minimizes te asymptotic mean-squared error. ie. = [ σ 2 K(u) 2 ] 1/5 du n /5 (m (2) (x)) 2 σ 4 K as δ = 1/n. We ave = O(n /5 ). If we substitute back into te asymptotic mse ten mse( ) = O(n 4/5 ). ie. wen using te Priestley-Cao estimator to estimate te regression function m at te point x te asymptotically optimal bandwidt converges to zero at te rate n /5 and te mse( ˆm(x)) using converges to zero at te rate n 4/5. Tis expression for is elpful from a teoretical point of view but is not directly useful in practice as we would not know te values of m (2) (x) and σ 2 for a real set of data. (iv) All oter tings being equal, we can regard te mse[ ˆm(x)] as a function of te kernel K troug te terms σ 4 K and K(u) 2 du. Consequently, we may wis to find tat kernel function K wic minimizes mse[ ˆm(x)]. Tis is a problem involving te calculus of variations and te solution was sown by Benedetti (1977) to be te so-called Epanecnikov kernel given by: K(t) = 3 4 (1 t2 ) 1 t 1 However, it turns out tat te loss in efficiency wen using a nonoptimal kernel is quite small (see Silverman (1986) for a discussion of tis issue in te context of kernel density estimation) and so K may be cosen using computational and/or smootness considerations. 5

6 (v) Beaviour near te boundary. Suppose now tat te point of estimation x is suc tat x <. From te above, we ave tat E[ ˆm(x)] = x/ (1 x)/ K(u) [ +o( 2 ) + O(n ) m(x) um (1) (x) + u2 2 Wen < x < 1 we can replace te lower and upper endpoints of integration by and 1, respectively. Consider now te case wen x = α for α (0, 1). ie. x (0, ), te left-and boundary region. For suc an x, x/ = α < 1 and (1 x)/ <. Terefore, te upper and lower limits of integration in te expectation are α and, respectively. Hence, α E[ ˆm(x)] = m(x) + 2 α K(u)du m (1) (x) α uk(u)du ] du u 2 K(u)du + o( 2 ) + O(n ) as 0, Now, denote te incomplete moments of K by µ l,α = α ul K(u)du for l = 0, 1, 2,.... Hence, E[ ˆm(x)] = m(x)µ 0,α + m (1) (x)µ 1,α + 2 µ 2,α + o( 2 ) + O(n ) Since µ 0,α 1, we will no longer in general ave consistency at points x <. At te zero boundary point itself we will ave E[ ˆm(x)] = 1m(x) + O(). We can see tat consistency is only acieved in te 2 boundary region wen m(x) = 0 tere. Since te location of te boundary is usually known, we can adapt ˆm(x) to acieve consistency. A simple approac is to normalize ˆm(x) by dividing it by µ 0,α at eac x. Tis will acieve consistency in te boundary region but te bias will be O() tere. Tere are a variety of furter modifications wic can be carried out wen estimating in a boundary region to acieve O( 2 ) bias everywere but tese will not be covered in tis module. Clearly toug, te asymptotic variance will still be O((n) ) wen x <. Te discussion ere as focussed on te left-and boundary (0, ) but tere are analogous results wen x (1, 1), te rigt-and boundary. 6

7 In te main discussion above we ave focussed on estimating m at a single point x wic is not in a boundary region and te mse[ĝ(x)] summarizes ow good te estimator is at tat point. Now suppose tat we use te Priestley- Cao estimator to estimate te entire function m in te interval (0, 1) using smooting parameter. We can quantify globally ow good our estimator is by integrating te asymptotic mse[ ˆm(x)] wit respect to x (0, 1). ie. ignoring possible boundary effects. We ave 0 mse[ ˆm(x)]dx = MISE( ˆm) = m (2) (x) 2 σ 4 K + δσ2 Te value of wic minimizes te asymptotic MISE( ˆm) is opt = [ σ 2 K(u) 2 du 0 m(2) (x) 2 dxσ 4 K ] 1/5 n /5 K(u) 2 du wic is again O(n /5 ) and te ensuing MISE( ˆm) is O(n 4/5 ). Te quantity 0 m(2) (x) 2 dx can be interpreted as a an overall measure of te rougness of te function m. Large values correspond to a function wit large fluctuations wile small ones are synonymous wit a smoot curve. Tis means tat smaller values of te global are required for estimating fluctuating regression functions m, wile larger values will be required for smooter m. References Benedetti, J. P. (1977) On te nonparametric estimation of regression functions. Journal of Te Royal Statistical Society, Series B, Vol 39, pp Priestley, M. E. and Cao, M. T. (1972) Nonparametric function fitting. Journal of Te Royal Statistical Society, Series B, Vol 34, pp Silverman, B. W. (1986) Density Estimation for Statistics and Data Analysis. (Capman and Hall/CRC Monograps on Statistics and Applied Probability) (Hardcover) 7