Nonparametric Regression. Changliang Zou

Transcription

1 Nonparametric Regression Institute of Statistics, Nankai University

2 Smoothing parameter selection An overall measure of how well m h (x) performs in estimating m(x) over x (0, 1) is given by the average squared error defined by: ASE = n 1 { m h (x i ) m(x i )} 2 Its population version is EASE = E(ASE X) We may choose to use y i as an estimate of m(x i ); the average residual sum of square ARSS(h) = n 1 { m h (x i ) y i } 2

3 Smoothing parameter selection ARSS(h) can be arbitrarily small by letting h 0 We can express this as ARSS(h) = 1 σ 2 (x i )ɛ 2 i + 1 { m h (x i ) m(x i )} 2 n n 2 n { m h (x i ) m(x i )} σ(x i )ɛ i Taking expectations, we have E{ARSS(h) X} = 1 σ 2 (x i ) + EASE(h) 2K(0) n n 2 h σ 2 (x i ) fh (x i ) E[ARSS(h)] E[ASE(h)]. The third term depends on h 1 and will clearly get large as h 0. This will mean that the ARSS(h) is minimized with respect to h when h 0.

4 Cross-validation To overcome this problem, we can use the cross-validation approach and replace m h (x i ) by the leave-one-out version. The least squares cross-validation function is then given by CV(h) = n 1 { m h, i (x i ) y i } 2, where We now have that 1 m h, i (x i ) = n f h (x i ) K h (x i x j )y j. j i E{CV(h) X 1,..., X n }} = EASE(h) + 1 n σ 2 (x i ) since the cross product now has expectation zero. Thus, using the data to find an h-value which minimizes CV(h) should, on average, result in the h-value which minimizes the EASE.

5 Asymptotic normal distribution For some regularity conditions and h = cn 1/5, conditioned on X n 2/5 { m h (x) m(x)} ( { D m N c 2 (x) µ 2 (K) 2 + m (x)f } ) (x), σ2 (x)r(k) f (x) cf (x) Pointwise Confidence Intervals Global Confidence Bands

6 Local polynomial estimator: derivation Taylor expansion for sufficiently smooth functions m(t) m(x) + m (x)(t x) + + m (p) (x)(t x) p /p! Consider a weighted least squares problem arg min β {y i β 0 β 1 (x i x) β p (x i x) p } 2 K h (x i x). resulting estimate for β provides estimates for m (ν) (x), ν = 0, 1,..., p.

7 Local polynomial estimator: Matrix expression Let X x be and { W x = h 1 diag K 1 x 1 x (x 1 x) p... 1 x n x (x n x) p ( x1 x h ),..., K ( )} xn x The least squares problem is then to minimize the weighted sum-of-squares function (y X x β) W x (y X x β) with respect to the parameter β. The solution is β = (X x W x X x ) 1 X W x y provided X x W x X x is a nonsingular matrix. h

8 Local polynomial estimator: Matrix expression The quantity m(x) is then estimated by the fitted intercept parameter β 0 = e 1 β, where the vector e 1 is of length p + 1 and has a 1 in the first position and 0 s elsewhere. Remarks: Provides an easy way of estimating derivatives of the function m(x) p is usually taken to be one (local linear) or three (local cubic regression) When p = 0, β 0 is equivalent to the Nadaraya-Watson estimator

9 Local linear estimator When p = 1 (local linear), we can express the estimator as m 1 (x) = 1 n {s 2 (x, h) s 1 (x, h)(x i x)}k h (x i x) y i s 2 (x, h)s 0 (x, h) s 2 1 (x, h), where s r (x, h) = 1 n (x i x) r K h (x i x). This is still a linear function of the y i s and so is a linear smoother in the sense that we have previously defined.

10 Local linear estimator: Asymptotic MSE properties By Taylor expansion of m(x i ) E{ m 1 (x)} m(x) = e 1 (X x W x X x ) 1 X x W x { 1 2 m (x)} {(x1 x) 2,..., (x n x) 2} + To find the leading bias term for general functions m, note that ( ) n 1 X s0 (x, h) s x W x X x = 1 (x, h) s 1 (x, h) s 2 (x, h) and n 1 X x W x { (x1 x) 2,..., (x n x) 2} = ( s2 (x, h)) s 3 (x, h) )

11 Asymptotic expression We can approximate the functions s r (x, h) by integrals 1 s r (x, h) = h 1 (u x) r K((u x)/h)f (u)du + O p ( Var{s r (x, h)}) 0 (1 x)/h = h r u r K(u)f (x + uh)du + O p ( Var{s r (x, h)}) x/h = h r f (x) u r K(u)du{1 + o(1) + O p (1/ nh)} By the symmetry and compact support of K, the odd moments of K are all zero and so we have s 0 (x, h) = f (x) + o p (1) s 1 (x, h) = o p (h) s 2 (x, h) = h 2 f (x)µ 2 (K){1 + o p (1)} s 3 (x, h) = o p (h 3 ).

12 Local linear estimator: Asymptotic Variance Some straightforward matrix algebra then leads to Bias{ m 1 (x) X} = 1 2 h2 µ 2 (K)m (x) + o p (h 2 ) To derive the asymptotic variance of m 1 (x) we have Var{ m 1 (x) X} = e 1 where V = diag{σ 2 (x 1 ),..., σ 2 (x n )}. (X x W x X x ) 1 (X x W x VW x X x ) ( X x W x X x ) 1 e1

13 Local linear estimator: Asymptotic Variance Using approximations analogous to those used above n 1 ( X x W x VW x X x ) = n 1 ( Kh 2 (x i x)µ 2 (K) 1 x i x x i x (x i x) 2 ( h = 1 µ 2 (K)R(K){1 + o p (1)} O p (1/ ) nh) O p (1/ nh) O p (h) These expressions can be combined to obtain Var{ m 1 (x) X} = 1 nh µ 2(K) R(K) f (x) + o p{(nh) 1 }. )

14 Behaviour Near the Boundary In the above analysis of the mse properties of the local linear estimator, we have assumed that the point of estimation, x, is an interior point in that it satisfies h < x < 1 h Consider now the left-hand boundary where x < h. The reason we have to consider the boundary separately is that the kernel function K((u x)/h) placed over x overspills the boundary at x = 0 when u < x h and so part of its support is devoid of data.

15 Behaviour Near the Boundary Suppose now that x (0, h) is expressed as x = αh where 0 α 1 Define the incomplete moments of K by µ l,α = α 1 ul K(u)du for l = 0, 1, 2,.... For such an x we have that ( ) n 1 X µ0,α (K){1 + o x W x X x = p (1)} hµ 1,α (K){1 + o p (1)} hµ 1,α (K){1 + o p (1)} h 2 µ 2,α (K){1 + o p (1)} and n 1 X x W x { (x1 x) 2,..., (x n x) 2} = ( h 2 µ 2,α (K){1 + o p (1)} h 3 µ 3,α (K){1 + o p (1)} )

16 Behaviour Near the Boundary It then follows that where Bias{ m 1 (x) X} = 1 2 h2 Q α (K)m (x) + o p (h 2 ), Q α (K) = µ2 2,α (K) µ 1,α(K)µ 3,α (K) µ 2,α (K)µ 0,α (K) µ 2 1,α (K) Therefore, the left-hand boundary bias of the local linear estimator is of the same order (ie. O(h 2 )) as its bias when estimating m at interior points, although the kernel-dependent constant is different This is also the case for estimating m at x-values in the right-hand boundary region, which can be shown in an analogous manner

17 Comparison with N-W estimator for boundary behaviour Suppose that the point of estimation x is such that x < h. Bias{ m 0 (x) X} = x/h (1 x)h { K(u)uhm (x) + O(h 2 ) } du + o p (h) When h < x < 1 h, we can replace the lower and upper endpoints of integration by -1 and 1, respectively Consider now the case when x = αh for α (0, 1) We have Bias{ m 0 (x) X} = hm (x)µ 1,α (K) + h2 2 m (x)µ 2,α (K) + The bias will be O(h) there The asymptotic variance will still be O{(nh) 1 } when x < h.