Basic Nonparametric Estimation Spring 2002

Transcription

1 Basic Nonparametric Estimation Spring 2002 Te following topics are covered today: Basic Nonparametric Regression. Tere are four books tat you can find reference: Silverman986, Wand and Jones995, Hardle990, Hardle994C 38 in te Handbook of Econometrics vol 4. Key Words: Density estimate, conditional expectation, kernel smooting, k-nnk nearest neigborood, local polynomial, seriessieve, spline, curse of dimensionality, optimal rate of convergence, bias-variance trade-off, undersmooting, oversmooting, pointwise asymptotic distribution, pointwise confidence interval, uniform confidence interval, automatic selection of bandwidt, uniform rate of convergence. Basic View: Tere can be many meanings to nonparametrics. One meaning is optimization over a set of function. For example, given te sample of observations x,..., x n, you are asked to find a distribution function under wic te joint probability of x,..., x n is maximized. You can convince yourself tat tis is te empirical distribution: Essentially, you are looking for probability masses on x,..., x n, say p,..., p n, suc tat p p n = and p p n is maximized. Convince yourself tat te solution is p =... = p n =. Tis is n also called nonparametric maximum likeliood. But tis is not wat we will discuss ere. Te meaning of nonparametric for now is density estimate and estimation of conditional expectations. Density estimate: motivation For tis part you sould read te first several pages in Hardle994. One motivation is tat you first use te istogram to estimate te density: #ofx i in x, x + = 2 n 2 n t= x x i x + = i= x 2 xi Now x is te uniform density over,, tis is called te uniform kernel. Of 2 course you can replace by anoter density function K, not neccessary over,, and plug in into were te uniform density appears, to get ˆf x = n t= K x x i. Anoter motivation, probably easier to understand: you estimate te distribution function F x by ˆF x = n n t=] x i x but you can t differentiate it to get te density. So you smoot out ˆF x by replacing x i x by G x i x were G is any smoot distribution function so tat G =, G = 0, and 0. In practice, you just take to be some small but fixed number, like 0.. So you ave ˆF x = n n t= G x i x, differentiate tis to get ˆf x = n t= K x i x for K = G te density corresponding to G. Higer dimension: if x R d a d-dimension vector, ten just take G to be a d-dimensional distributional and do te same ting, ˆF x = n n t= G x i x. Now wen you differentiate it don t forget tat you ave to do tat to eac of te x,..., x d, so you will pull out d of to get: ˆf x = d xi x K t=

2 were now K is te multivariate density corresponding to te multivariate distribution function G. Conditional expectation: motivation You want to estimate E y x or more generally E g y x for some function g. You can also estimate tings like te conditional median of y give x, say med y x, or any oter conditional quantiles. Local Weigting: Te wole ting about nonparametric estimate of E y x is just use observations for wic x i is close to x. Te obvious to do is to take a neigborood around x and average over tose y i for wic x i is wit tis neigborood. Te size of te neigborood sould srink to 0 but not too fast. More generally you take weigted average over all y i but you give more weigts to tose y i for wic te x i tat is close to x, and you give less weigt to tose y i for wic te x i is far away from x. See Stone977. For weigts W n x, x i suc tat i= W n x, x i = tis is wat weigt means; 2 W n x, x i 0 if x i x; 3 max i n W n x, x i 0 as n, you estimate E y x by i= W n x, x i Y i. Similarly, you estimate E y 2 x by i= W n x, x i Yi 2, and F y x by i= W n x, x i Y i y. Anytings you do parametrically, if you do tat only for x i close to x, ten you become nonparametric. You can run least square, quantile regression, and even maximum likeliood over smallsrinking neigborood of x, and call tat nonparametric st st... Depending on ow you define te weigt sequence W n x, x i, te local weigting type nonparametric estimators are classified into many many different categories. Te most important ones are kernel smooting, k-nearest neigborood, local polynomials. Asymptotically tey are all equivalent to eac oter. Kernel smooting is te easiest to use so tis is te only ting we will deal wit in details. In additions to tese local nonparametric estimate, tere are also global nonparametric estimates including seriessieve estimator and splines. Seriessieve estimation as been paid increasingly attention recently. But few people would boter wit splines. Te reason again is tat seriessieve is extremely easy to do, you just need to run least square! And just remember to say tat te number of regressors included as n, altoug in practice you just include some finite number of terms. But te focus today is kernel. Kernel smooting: If you use density weigting for te weigts W n x, x i, ten you get te kernel estimator of E y x. If x i is one-dimensional, let W n x, x i = i= i= K x x i i= K x x i tis very definition i= W n x, x i =. Ten te kernel estimate is defined by: W n x, x i Y i = K x x i i= K i= x x i Y i = K x x i Yi i= K x x i For multidimension x i R d, use te multidimension density estimate W n x, x i = and te kernel estimator of E y x is given by: W n x, x i Y i = i= i= K x x i d d i= K x x i Y i = 2 Yi d i= K x x i d i= K x x i. By K x x d i K x x d i i=,

3 Anoter view of kernel estimator: You can tink of E y x = E y x f x f x = γ x f x were γ x E y x f x = yf y, x dy. So you may estimate γ x and f x seperately. Of course ˆf x = n d i= K x x i. For ˆγ x, plug in te kernel density estimate for x, y: ˆf x, y = n d+ i= K x x i K yi y into yf y, x dy, and use a cange of variable u = y i y /: y ˆf y, x dy = d = d x xi K x xi K i= i= y K yi y dy y i + u K u du = d x xi K y i Yet anoter view for γ x: you can tink of y ˆf y, x dy as ydp, were P is te measure over y defined by P y i y, x i = x. Note tat P y i y, x i = x = d P y dx i y, x i x, you can estimate P y i y, x i x by n n i= y i y G x i x /. Differentiating tis, you estimate d P y dx i y, x i x by n d i= y i y K x i x /. Plug in tis estimate of P into ydp : yd ˆP = = d yd d y i y K x i x / i= K x i x / i= yd y i y = d i= K x i x / y i Note: Now note tat to study convergence rate, asymp distribution, etc, we only need to worry about ˆγ x, and we may ignore discussing ˆf x. Te reason is tat ˆf x is just a special case of ˆγ x, in wic you take y i identically. Convenient forms of kerneldensity function include tose on pp2303 Hardle994 C8 vol Handbook: Uniform kernel x ; 2 Triangular kernel: u u ; and 2 quartic, epanecniknov, gaussian, etc. Read pp2303, and try to use gauss or matlab to grap out wat eac of tese look like. To get consistency, you want 0 as n 0 so tat you don t ave a bias, you also want d so tat you ave enoug observations. More on tis below. Estimating Derivatives: It is also easy to estimate derivatives of γ xincluding f x. As long as your kernel is smoot differentiable, you could just simply differentiate your ˆγ x and f x. To estimate te kt derivative of γ x, denoted by γ k x, use ˆf k x = k+d i= 3 i= K k xi x y i

4 Don t forget tat eac time you differentiate wrt x in K x i x, you will pull out one, so altogeter you pull out k of tem. Also remember tat wen x is d-dimension, te derivative can be taken wit respect to different combinations out of x,..., x d, as long as te degree of differentiation wit respect to x,..., x d sum to k. Sometimes people write f k and similarly K k using very long notation. For example, f k is written as: for positive integers λ,..., λ d, let λ λ d = k. Ten write f k x as f x. k x λ... xλ d d k-nnnearest neigborood and local polynomials: Tese are te oter two major weigting scemes for W ni x. Bot of tese are asymptotically equivalent to kernel weigting. k-nn: Insteading of using observations wose x i are wit far away from x, you may also just use your k closest neigbors, assuming you are standing at te point x. Look at te example in pp3 Hardle990 for te precise definition. You may weigt your k nearest neigbors equally, or you may give even iger weigt to te ones wo just live next to you and give decreasing weigts to te ones increasing furter away from you. You can of course use any kernel density weigt K. In tis case you may take te bandwidt to be te distance between x and te kt nearest neigbor x i. Giving equal weigts to eac of te k neigbors is te same as saying using an uniform kernel density weigt. If you do tis ten tere is an easy relation between and k: k n 2 d f x. For te case were d =, ceck to see tat tis relation makes sense to you by verfitying te equivalence between kernel and k-nn in Table 3.2. pp46 Hardle990. For local polynomial, read pp30-32 Hardle990. You must know ow to run a polynomial regression using all observations, just regress y on, x, x 2, all cross products, cross products of all powers..., up to degree k. If you run tis kt degree polynomial regression only over a neigborood for wose x i wit distance from x: x i x, ten you get te local polynomial estimate. Te degreee k of local polynomial corresponding to te order of te kernel, wic we will talk about below. Series and Splines: Tere is also anoter ting called sieve estimator, wic is very similar to, and you can tink tat it is just same as, series estimator. Te only difference between series and local polynomials is tat you run te polynomials using all observations, instead of only a srinking neigborood x, x +. However, now instead of fixing k, te degree of polynomial, you promise to let k. But tis is noting more tan a promise, you don t actually do it. Just take a number k, not too small. Instead of using polynomials, wic tends to be igly correlated, you may also use family of ortogonal series of functions, like trigonometric function, legendre polynomials, laguerre polynomials. If you are interested you can find more information about tese function in a recent book by Judd998. Splines is based on find a twice differentiable function g x tat minimizes i= y i g x i 2 + λ g x 2 dx, for some λ > 0. Te reason to include te second term is to penalize te rougness of te estimate g, oterwise te minization problem is ill-defined and you will simply take ĝ x i = y i, no good. Tis will give you a cubic polynomial wit continuous second derivatives. Of course instead of penalizing by λ g x 2 dx, you migt tink of all oter kinds of penalizations. But tis as not been muc used oter tan simple case wit one dimension x. If you are interested read Hardle990 sec 3.4 pp56. 4

5 Now back to te local nonparametric estimators. Curse of dimensionality: For a given bandwidtwindow size, te iger dimension x, te less data you ave in a neigborood wit bandwidt. Say all your x is on te unit interval 0,, you only look at observations witin a small bandwidt, te total number of observations is n. If d =, ten te effective number of observations you ave in tat neigborood is. If d = 2, te effective number of observations you ave is 2, so on so fort. So for general d, te number of observations you ave in a d dimension cube wit lengt is just d. Since is small, goes to 0, of course you get less and less observations as d get larger. Anoter view of tis is tat, fixing te number of total observations, te average distance between any pair of observations gets larger and larger as d increasesyou find tat your data becomes sparser and sparser in iger dimensions. Optimal rate of convergence for nonparametric estimates: Suppose te true function γ x is pt degree differentiable, all pt derivative bounded uniformly over x. Ten te optimal bandwidt opt you sould use is n, and te best rate at wic your estimate ˆγ x can approac γ x is O p n p. Now we will explain wy. Reading is in sec 4 of Hardle990. You may consult te original Stone papers if you are interested. Te problem ere is te bias and variance tradeoff. Basically, by using a bandwidt of size you are doing two tings: you are using observations x i tat can be at most away from x to approximate te true function γ x at point x; 2 you are using all d observations in tis small cube of lengt. Step gives you a bias in estimating γ x, and step 2 gives you te variance in te estimate. Te smaller te, te smaller te bias, but te less observations you ave, tus te large te variance. Te larger te, te larger te bias, but ten you ave more observations wic reduces te variance in your estimate. To get consistency, you want to get rid of bot te bias and te variance. To get rid of bias, simply take 0, as n 0. To get rid of variance, you want to ave infinite number of observations as n, but te number of observations you ave is d, so you also want d. If you ave bot 0 and d, ten your estimate is consistent. However, you want more tan consistency. You want ˆγ x to converge as fast as possible, to do tat you need to balance te trade off between bias and variance. To do tis, we need to find more precise representations of te bias and te variance. Te bias is of order p : Because you know your function γ x ave p bounded derivatives, so you could do a taylor expansion of te approximation error γ x i g x around x up to degree p. Now eac term is a polynomial in x i x and remember tat x i x. Just assume tat you asomeow knows te coefficients in tose p expansion terms, so tat you don t ave to worry about tese terms. But te last term, te pt order term, an order of x i x p p cannot be furter expanded anymore, so you ave to live up wit it. Te order of tis error is p. Tese are different ways to get rid of te first p terms in te taylor expansion. Higer order kernel is one way, wic we will see below. Use p t order local polynomial is anoter waywic is just exactly doing a taylor expansion of order p but we won t get into tis. Te variation is of order O p. Just tink of te simplest problem of estimating a d 5

6 mean using a sample average. Wen you ave n observations, using a central limit teorem, you know tat d n x µ N 0,? = O p. So tat x µ = O p n. Now wit a window size te number of observations you ave is d, by analogy to te simple case of x, te precision of your estimate, of course, sould be O p. d So tat you total error, bias plus estimation error, is of te order of p +, find a to d minize tis, you can easily see tat opt = O n. A more formal way to tink of te total error is MSEmean square error = bias 2 + variance. Now bias 2 = 2p, and variance =. We will see tese in detail for kernel regressions. Ten te pointwise optimal rate of convergence d is given by: O p opt = O = O n p d It sould ten be clear tat it is not possible to ave n convergence for nonparametric p estimates since <. Sometimes 2 n/4 rate of convergence is needed for getting rid of te p second order terms for semiparametric estimators. You will need p > d/2 to get > /4. So te more regressors you ave, te more smoot your regression function as to be in order to get te same rate of convergence. Optimal rate for derivative estimates: Wat is te optimal bandwidt for estimating γ k x and wat is te best convergence rate. It turns out te optimal bandwidt is of te same order as tat of estimating γ x itself. Tis can be seen again from te same bias and variance tradeoff. bias: Now if γ x as p bounded derivatives, ten γ k x as p k bounded derivatives. So by analogy, te bias is of te order of p k. variance: Again you ave d observations effectively. However, eac differentiation amplify te error by an order of so after taking k derivative you amplify te error by, see k equation. So te estimation error is of te order of k. Te total error is of order d p k + k. Find a to minimize tis again you see tat d opt = n. Note tat opt does not depend on k at all. Ten te best convergence rate is: n p k = k = d n Te order of bias and variance will be developed in detail for kernel estimates below. Higer order kernels Te kernel weigts sould integrate to, as required for a density, so K u du =. If you use a density K tat is symmetric around 0, ten uk u du = 0. But if you insist tat te kernel K must be a density, ten K u 0, u, and so u 2 K u du > 0. A iger order kernel is tose K tat as some similarity to a symmetric density but is not a density. Precisely, for tese K, K u udu = 0, but may be negative for some u. Tis allows you to find suc K so tat K u u 2 du = 0. To see graps of example, see 6 p k

7 e.g figure 3. and 3.2 in pp60 and pp6 in Hardle990. Precisely, a kernel of order r is defined as tose K for wic: K u du = K u u q du = 0, q =,..., r u r K u du < for examples of tese kernel can be found in table 4.5. pp35 Hardle990. Bias of kernel estimates: use te cange of variable u = x i x, and bear in mind tat te integration notation is over te d-dimensional x i, and finally do a taylor expansion of γ x + u, Eˆγ x γ x =E x xi d K Y i γ x = E x d K xi Y i γ x i= x =E d K xi x m x i f x i dx i γ x = E d K xi γ x i dx i γ x r = K u γ x + u du γ x = j γj u j k u du + r γ r x u r K u du j! r! j= If your kernel is of te order r, ten eac of te integrals u j k u du = 0 in te first summation. So you are left wit only te last term, so tat bias = γ. Note tat you can do tis only if your true underlying function γ x as p r bounded derivatives. So if your function γ x as pt bounded derivatives and your kernel is of order r, ten te bias = minp,r. Optimally, if you know your γ x is pt degree smoot, ten you sould use a pt degree kernel to reduce bias. But you never know wat p-smootness your function is. So wy not always use a very ig order kernel? Tat is because a ig order kernel will neccessary make te variance big, as seen below. In te basic case wen you just use a symmetric density for K, it is of order r = 2 so tat te bias is of order 2. Variance of kernel estimates: Tis one is easier tan te bias since iger order kernel does not play any role ere, let g x i = E yi 2 x i f x i : x xi V ar d K Y i i= = x xi n 2 2d V ar K Y i = x 2d E K 2 xi Yi 2 x 2 xi 2d EK Y i i= = x xi d d K2 E y 2 i x i f xi dx i E x 2 n d K xi Y i = x xi d d K2 g x i dx i + O = n d K 2 u g x + u du + O n = d K 2 u g x du + d K 2 u g x udu + O n = d K 2 u g x du + O d + O = O n d 7

8 Bias and Variance of derivatives of kernel estimates: You could verify te formulas for bias and variance for γ k x by carrying troug te same exercise of taking expectation, cange of variable in integration. An additional tools tat are needed ere is integration by parts, assuming K u is bounded and all te first kt derivatives dimins at te boundary, you can verify tat: K k u γ u du = k K u γ k u du. Exercise: Use integration by parts to obtain te bias formula for kernel estimates of γ k x, and convince yourself tat iger order kernel plays te same role ere. Te variance formula can be calculated as before witout aving to worry about integration by parts. Asymptotic Distribution, Confidence band To sow ow precise your nonparametric estimate is at eac point x, it is neccessary to derive its asymptotic distribution and obtain te confidence interval at eac point x. Section 4 in Hardle990 contains detailed discussion. Te derivation of te asymptotic distribution depends on wat bandwidt you use, if you use opt, were means te same order as, ten te asymptotic distribution will depend on bot te bias and te variance. On te oter and, if you use << opt were << means smaller by an order of magnitude, i.e., opt 0, ten tere is no bias in te asymptotic distribution but te convergence rate is not te fastest. To look at tis in more detail, consider te simple example for d =, r = 2, for one-dimensional x and for a standard density kernel wit only order 2, ten opt = n = n 5. To compute te asymptotic distribution of opt ˆm x m x = 2 opt ˆm x m x, for ˆm x = ˆγx ˆfx, first linear tis in terms of ˆγ x and ˆf x by simply taking first order taylor expansion: ˆm x m x γ x ˆγ x γ x ˆf x f x f x f x 2 As seen above: Eˆγ x γ x = 2 2 γ x u 2 K u du. Note tat for γ x = m x f x, γ x = m x f x + 2f x m x + m x f x. So for te density were m x, E ˆf x f x = 2 2 f x. Terefore E 2 opt ˆm x m x = γ x 2 f = 2 m f f x = 2 f 2m x f x + m x f x f x m f + 2m f + mf m f f x u 2 K u du. To compute te variance, follows from te variance calculation for kernel estimation to get: V ar ˆγ x γ x g x K 2 u du, for g x = E y 2 x f x. For density estimate were y, g x = f x, so tat V ar ˆf x f x g x K 2 u du. It is also neccessary to calculate te covariance between ˆγ x and ˆf x, you may use te type of cange-of-variable calculation to get: Cov ˆγ x γ x, ˆf x f x γ x K 2 u du. u 2 K u du 8

9 Ten use te delta metod to obtain te variance of V ar ˆm x m x : V ar ˆm x m x = V ar = K 2 u du f ˆγ x m f ˆf f 2 E y 2 x f x 2 m2 m x γ x + f f f = E y 2 x m x 2 K 2 u du = f x f x σ2 x K 2 u du To summarize: ˆm x m x d m x f x + 2m x f x N 2f x u 2 K u du, f x σ2 x K 2 u du If instead of using opt = n /5, you use a undersmoot bandwidt << n /5, say = n /6, ten te bias term will disappear and you don t ave to worry about it anymore. In tat case te asymptotic distribution simplies to: d ˆm x m x N 0, f x σ2 x K 2 u du Terefore, if you don t want to boter wit te complicated bias term in drawing your confidence interval, ten you sould use a undersmoot bandwidt, like = n /4. However, if you insist on using te optimal opt and want to draw te confidence interval around your estimated ˆm x, ten you will need to be able to estimate te bias term consistently. However, te bias term involve second derivatives m x, or for tat matter γ x. And γ x can NOT be estimated consistently using opt. Instead, you will need to use a seperate and oversmooted bandwidt, say g = n /6 to estimate γ x consistently. Te reason is simply tat ˆγ x as a bigger variance tan ˆγ x, by an order of 2. To reduce tis larger bias you need more data and so you need a larger oversmooted bandwidt g. In fact you can sow tat te variance of ˆγ x is O /5, so if you use opt = n /5, te variance won t go to 0. But if you use g = n /6, ten it will. Tis g is called pilot bandwidt in Hardle990 s terminology. Uniform confidence band: Te above asymptotic distribution is for deriving te Pointwise confidence band. Te uniform band is te two curves for wic te probability tat tey encompass te ENTIRE true m x function is α%. Constructing uniform band is difficult. You migt consult Hardle990 sec 4.3 if you are interested. But I suggest against doing tat in first reading. You need to know owever, te difference between pointwise confidence band and uniform band. Automatic bandwidt selection: Two ways: cross validation and penalizing functions, read Hardle990 pp Cross Validation: You want a good fit of your estimate, try to minimize i= ˆm x i m x i 2, but you don t know te true m x i, so you estimate it using y i : min i= ˆm x i y i 2. But 9

10 ten you are in trouble, since as 0, ˆm x i = y i, so you ave perfect fit 0, same problem tat motivates te spline estimator. Anoter way to tink about tis, write ˆm x i y i 2 = i= ˆm x i m x i ɛ i 2 = i= ˆm x i m x i 2 + ɛ 2 i 2 i= ˆm x i m x i ɛ i Te first term is wat you want. Te second term is not related to and can be ignored. Te tird term causes trouble, its expectation: E i= xi x j K j= ɛ j ɛ i = i= K 0 σ 2 = σ2 K 0 2 Tis is not zero and in fact not negligiblesame order as first term. Te problem is due to te correlation between ɛ i and itself. Of course if ɛ i is not used to estimate m x i ten tis term wouldn t exist to cause problem, terefore leading to te leave-one-out estimate ˆm i x i = n j=,j i K x j x i i= yi, and define te cross-validation function tat is to be minimized over as: min CV = i= m i x i m x i 2. According to Hardle990 sec 5. tis will automatically give you te optimal bandwidt, no matter ow smoot your m x is. Of course tis statement needs to be constrained by te order of te kernel function you use. Terefore te point of cross-validation is to get rid of te nonzero expectation due to correlation of ɛ i wit itself, in 2, anoter way to get rid of tis term is just to subtract it, if you know σ 2. But you don t know σ 2, but you can still estimate tis term consistently by K 0 n i= y i ˆm x i 2. Tis leads to te penalizing function metods, read pp54 Hardle990, to minimizer over : G = ˆm x i y i 2 + 2K 0 n i= y i ˆm x i 2 i= G as te same property as CV. Bias reduction by Jacknifing: skip te rest of tis note if you don t find it interesting. Tese are Extra material Anoter way to tink about bias reduction is by Jacknifing. Read Hardle990 sec 4.6in particular page 43 for details. It is essentially equivalent to ig order kernel. It doesn t make any difference if you are just running a simple kernel regression. So if you are just running a simple kernel regression don t boter wit Jacknifing. However, if you ave some objective function tat are only convex wit positive K, say, if you want to run a nonparametric quantile regression and want to reduce bias, ten operationally te Jacknife metod is very useful in preserving te convexity of te objective function. See for example, te recent paper Honore and Powell998 for illustration. Uniform rate of convergence: In addition to te pointwise rate. It is sometimes useful to obtain optimal bandwidt and optimal uniform convergence rate, i.e., for sup x X ˆγ x γ x. Te necceesary steps are again to consider te bias-variance tradeoff. Te bias for sup x X ˆγ x γ x for a rt order kernel is te same before p, noting tat te bias calculation is completely uniform in x. Te variance calculation, on te oter and, need to 0

11 /2 be modified. It turns out te variance is of te order O d p Eˆγ x = O p d, i.e. sup x X ˆγ x /2. Te mysterious comes from considering te variance at an increasing number of points simultaneous and by te application of te Bernstein inequality, a form of te exponential inequality: Berstein inequality: Tis is one of te many exponential inequality used to bound te tail probability of sums of mean 0 random variables. To see a clean and brief discussion of it you may read pp 93 of Pollard984. It basically says tat for z,..., z n independent of eac oter, eac z i bounded by B, V = V ar z V ar z n, S n = z z n, η P S n > η 2 exp 2. Tis says tat S n looks rougly like normal distribution 2 V + 3 Bη wit variance V in te tails. Te term Mη turns out to be not important and may in fact 3 be ignored. /2, Variance for uniform rate: To sow tat sup x X ˆγ x Eˆγ x = O p d te first ting is to partition te compact set X into side lengt n L squares, for some big number L. If X is compact, say just te unit cube, ten you got about n dl suc squares wit side lengt n L. Let x j, j =,..., n dl be te center of eac of te squares. As long as L is big enoug, te difference between γ x Eˆγ x and γ x 2 Eˆγ x 2 for x and x 2 in te same square is small enoug and may be ignored, because now te difference is of order n L d /2. << d Ten rougly speaking, supx X ˆγ x Eˆγ x sup xj,j=,...,n ˆγ x j Eˆγ x dl j. Now for eac x j, ˆγ x j Eˆγ x j is sums of mean 0 random variables, te sum of variance V, as ave been calculated above, is about D, for some constant D. Assuming bot Y d i and K are bounded, ten B = E for some E > 0. remember also tat saying tat d d /2 sup ˆγ x Eˆγ x = O p x X means lim d /2 P M sup ˆγ x Eˆγ x > M 0. n x X Now time to use te Bernstein inequality: d /2 d /2 P sup ˆγ x Eˆγ x > M < P sup ˆγ x j Eˆγ x j > M x X x j,j=,...,n dl n dl 2 exp M 2 d n dl 2 exp M2D 2 = 2 exp dl M M,n 0. 2D D + E d 3 M d d Ten minimize te sum of bias + error = p + O p d bandwidt and te best convergence rate. /2 you will find te optimal