NADARAYA WATSON ESTIMATE JAN 10, 2006: version 2. Y ik ( x i

Size: px

Start display at page:

Download "NADARAYA WATSON ESTIMATE JAN 10, 2006: version 2. Y ik ( x i"

Aileen Heath
6 years ago
Views:

1 NADARAYA WATSON ESTIMATE JAN 0, 2006: version 2 DATA: (x i, Y i, i =,..., n. ESTIMATE E(Y x = m(x by n i= ˆm (x = Y ik ( x i x n i= K ( x i x EXAMPLES OF K: K(u = I{ u c} (uniform or box kernel K(u = u 2 if u, 0 else (Epanecnikov kernel K(u = exp( u 2 /(2σ 2 (normal kernel?? Does K need to integrate to???? Does K need to be positive?? Nadaraya (964, Teory of Probability and its Applications Watson (964 Sankya series A COMMENT: tis estimator as fallen out of favour, replaced by local polynomial estimators. But it s wort studying - easier asymptotics. 0-0

2 IN R, ksmoot produces te Nadaraya Watson estimate. Usage: ksmoot(x, y, kernel = c("box", "normal", bandwidt = 0.5, range.x = range(x, n.points = max(00, lengt(x, x.points Arguments: x: input x values y: input y values kernel: te kernel to be used. bandwidt: te bandwidt. Te kernels are scaled so tat teir quartiles (viewed as probability densities are at +/- 0.25*bandwidt. range.x:te range of points to be covered in te output. n.points:te number of points at wic to evaluate te fit. x.points:points at wic to evaluate te smooted fit. If missing, n.points are cosen uniformly to cover range.x. 0-

3 ?? For te uniform kernel, wat value of c is ksmoot using?? HOMEWORK: Jan 0 #: For te normal kernel, wat value of σ 2 is ksmoot using? EXAMPLES IN R (see R.txt ksmoot wit normal kernel simulation study to look at bias and variance effect of sample size 0-2

4 IN SMOOTHING WE STUDY (conditional on te x i s: Bias( ˆm (x =E( ˆm (x m(x Var( ˆm (x MSE( ˆm (x = E[ ˆm (x m(x] 2 = Bias 2 + Var IMSE( ˆm = MSE( ˆm (x dx WHY? to understand ˆm and te effect of to derive an asymptotic formula for MSE and IMSE for plug-in estimates of. ASYMPTOTICS INVOLVE smootness assumptions on m conditions on te x i s n, n 0 but not too fast properties of K THESE IDEAS ARE COMMON TO MANY/ALL smooting metods 0-3

5 ASYMPTOTIC RESULTS For rigorous presentation, see te posted notes. THROUGHOUT ASSUME THAT: Y i = m(x i + ɛ i, ɛ i iid mean 0 variance σ 2. te x i s live on [a, b], a finite interval. eiter: te x i s are realizations of iid random variables wit density f OR te x i s ave a regular design density f, tat is x i f(t dt = i/(n+. a In bot cases, f must be bounded and bounded away from 0 on [a, b]. K(u = 0 for u > (can be relaxed to include normal density and K is Lipsitz ( K(x K(u < C x u for some C. 0-4

6 MOST COMMON AND SIMPLEST CASE: and just for fixed design (random X i s is similar. In addition to previous assumptions, suppose tat K: u K(u du = 0, u 2 K(u du 0 m and f exist and are continuous on [a, b] n, = n 0, n 4 n THEN for interior x [a + n, b n ] E( ˆm (x m(x = 2 (mf (x m(xf (x 2f(x and var( ˆm (x = = 2 (mf (x m(xf (x 2f(x = σ 2 nf(x σ 2 nf(x 0-5 u 2 K(u du K(u du + o( 2 ν 2 (K ν 0 (K + o(2 K 2 ( (u du ( 2 + o K(u du n ν 0 (K 2 ( (ν 0 (K 2 + o n (

7 For boundary x, tat is, x [a, a + n or (b n, b], E( ˆm (x m(x = (mf (x m(xf (x f(x ν (K, x, ν 0 (K, x, + o( and var( ˆm (x = σ2 nf(x ν 0 (K 2 (, x, [ν 0 (K, x, ] 2 + o n were (b x/ ν j (K, x, = s j K(s ds. (a x/ NOTE: for x interior, ν j (K, x, = s j K(s ds ν j (K 0-6

8 MORE GENERAL CASE: by using a pt order kernel and assuming n p+2, 0 m (p and f (p exist and are continuous on [a, b]. THEN, for interior x: E( ˆm (x m(x = p (mf(p (x m(xf (p (x p!f(x and var( ˆm (x is as in (. ν p (K du ν 0 (K + o( p (2 For boundary x we still get E( ˆm (x m(x = B(x, + o( and var( ˆm (x = V (x, /(n + o(/(n. 0-7

9 A pt order kernel K satisfies ν 0 (K K(u du 0 ν j (K u j K(u du = 0, j =,, p ν p (K < and 0 COMMENT: consider x i equi-spaced and suppose y i x k i. THEN for x i not a boundary point: if K is an order 2 kernel, ŷ i x k i for k = 0, ; if K is an order 4 kernel, ŷ i x k i for k = 0,, 2, 3. etc SEE R CODE. 0-8

10 USING THE ASYMPTOTICS - equations (2 and (: For a fixed interior x, te tat minimizes MSE( ˆm(x is opt = Cn /(2p+ Tis yields asymptotic MSE of order n 2p/(2p+. For a boundary x, te tat minimizes MSE( ˆm(x is opt = C n /3 Tis yields asymptotic MSE of order n 2/3. NOTE: parametric rate of convergence is n, muc smaller/faster. 0-9

11 SOME o/o notation: DETERMINISTIC SEQUENCES Let a n, b n and c n be tree sequences of numbers. Ten a n = b n + o(c n iff a n b n = o(c n iff lim n a n b n c n = 0. a n = b n + O(c n iff a n b n = O(c n iff lim sup n a n b n c n <. RANDOM SEQUENCES Let a n, b n and c n be tree sequences, at least one of wic is random. a n = b n + o p (c n iff a n b n = o p (c n iff a n b n c n 0 in probability. a n = b n +O p (c n iff a n b n = O p (c n iff lim lim P { a n b n C} =. C n c n 0-0

12 RANDOM SEQUENCE EXAMPLES: suppose tat X i are iid E(X i = µ and var(x i = σ 2. Let X n = n X i/n, s 2 = n (X i X 2 /(n and ˆσ 2 = n (X i X 2 /n. Ten X = µ + o p ( ˆσ 2 s 2 = O p (/n DETERMINISTIC SEQUENCE EXAMPLE: Suppose tat n p+2 n. Ten = o( p n n 2 n 0-

13 FOR THE PROOF OF THE GENERAL RESULTS (2 and ( wit x i s non-random: implies ˆm (x = E[ ˆm (x] = n K ( x i x n K ( x i x Yi n m(x ik ( x i x n K ( x i x and ( n var[ ˆm (x] = σ 2 K2 x i x [ n K ( x i x ] 2 0-2

14 CONSIDER x in te interior. THEN (to be sown (n + n K ( xi x = f(xν 0 (K + f (p (xν p (K p p! + o(p (3 (n + n K 2 ( xi x f(xν 0 (K 2 (4 (n + n m(x i K ( xi x = m(xf(xν 0 (K +(mf p (xν p (K p p! + o(p (5 If tose tings are true ten, wit some work, we can sow ( and (2. 0-3

15 TO PROVE THE THREE EQUATIONS USE: LEMMA (proven in 998/99 notes. Let te x i s ave a regular design on [a, b] wit f bounded and bounded away from 0. Let g be Lipsitz on [a, b], M wit support [, ] and Lipsitz. Ten for all 0 > 0, tere exists C so tat (n + n ( xi x g(x i M for all n, x and for all wit 0. b a M ( u x g(u f(u du Te proof is tedious, but for intuition: suppose X i s are iid wit density f. Take expected value of te sum. PROOFS OF EQUATIONS: just (3 - oters are similar. C n 2 0-4

16 MORE HOMEWORK Jan 0 #2: For tis exercise, consider te normal kernel, used for te Nadaraya-Watson estimate of m. (a Find te expressions for te asymptotic bias and variance of ˆm (x. (b Find te optimal value of for estimating m(x at an interior x. (c Carry out a simulation to study ow te bias and variance of ˆm (x depend on. Use te regression data as in my simulation study. For fixed x consider te bias and variance as a function of. Do tis for x = 0, 0.2, 0.4, 0.6, 0.8,.0. Relate wat you see to te asymptotic expressions for te bias and variance in (a. PLEASE HAND IN: a write-up (word-processed or neatly and-written wit your calculations, describing wat you did in your simulation study, and wat you conclude. Include any relevant plots. Also include welldocumented easy to read code. Part of your evaluation will be on ow clearly you write tings up. 0-5

The Priestley-Chao Estimator

The Priestley-Chao Estimator Te Priestley-Cao Estimator In tis section we will consider te Pristley-Cao estimator of te unknown regression function. It is assumed tat we ave a sample of observations (Y i, x i ), i = 1,..., n wic are