Department of Statistics Purdue University West Lafayette, IN USA

Transcription

1 Effect of Mean on Variance Funtion Estimation in Nonparametric Regression by Lie Wang, Lawrence D. Brown,T. Tony Cai University of Pennsylvania Michael Levine Purdue University Technical Report #06-08 Department of Statistics Purdue University West Lafayette, IN USA October 2006

2 Effect Of Mean On Variance Function Estimation In Nonparametric Regression Lie Wangl, Lawrence D. Brown1, T. Tony Cail and Michael Levine2 March 16, 2006 Abstract Variance function estimation in nonparametric regression is considered and the minimax rate of convergence is derived. We are particularly interested in the effect of the unknown mean on the estimation of the variance function. Our results indicate that, contrary to the common practice, it is often not desirable to base the estimator of the variance function on the residuals from an optimal estimator of the mean. Instead it is desirable to use estimators of the mean with minimal bias. In addition the results also correct the optimal rate claimed in Hall and Carroll (1989, JRSSB). Keywords: Minimax estimation, nonparametric regression, variance estimation. AMS 2000 Subject Classification: Primary: 62G08, 62G20. Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, PA The research of Tony Cai was supported in part by NSF Grant DMS Department of Statistics, Purdue University, West Lafayette, IN

3 1 Introduction Consider the heteroscedastic nonparametric regression model pi= f(x,)+l/~(xz)zi, i=l,..., n, (1) where x, = iln and zi are independent with zero mean, unit variance and uniformly bounded fourth moments. Both the mean function f and variance function V are defined on [O,1] and are unknown. The main object of interest is the variance function V. The estimation accuracy is measured both globally by the mean integrated squared error and local.1~ by the mean squared error at a point We wish to study the effect of the unknown mean f on the estimation of the variance function V. In particular, we are interested in the case where the difficulty in estimation of V is driven by the degree of smoothness of the mean f. The effect of not knowing the mean f on the estimation of V has been studied before in Hall and Carroll (1989). The main conclusion of their paper is that it is possible to characterize explicitly how the smoothness of the unknown mean function influences the rate of convergence of the variance estimator. In association with this they claim an explicit minimax rate of convergence for the variance estimator under pointwise risk. For example, they state that the "classical" rates of convergence (np4l5) for the twice differentiable variance function estimator is achievable if and only if f is in the Lipschitz class of order at least 113. More precisely, Hall and Carroll (1989) stated that, under the pointwise mean squared error loss the minimax rate of convergence for estimating V is rniwcin-20r+l, n-28+1) (4) if f has cr derivatives and V has,b derivatives. We shall show here that this result is in fact incorrect. In the present paper we revisit the problem in the same setting as in Hall and Carroll (1989). We show that the minimax rate of convergence under both the pointwise squared error and global integrated mean squared error is

4 if f has a derivatives and V has p derivatives. The derivation of the minimax lower bound is involved and is based on a moment matching technique and a two-point testing argument. A key step is to study a hypothesis testing problem where the alternative hypothesis is a Gaussian location mixture with a special moment matching property. The minimax upper bound is obtained using kernel smoothing of the squared first order differences. Our results have two interesting implications. Firstly, if V is known to belong to a regular parametric model, such as the set of positive polynomials of a given order, the cutoff for the smoothness of f on the estimation of V is 114, not 112 as stated in Hall and Carroll (1989). That is, if f has at least 114 derivative then the minimax rate of convergence for estimating V is solely determined by the smoothness of V as if f were known. On the other hand, if f has less than 114 derivative then the minimax rate depends on the relative smoothness of both f and V and will be completely driven by the roughness of f. Secondly, contrary to the common practice, our results indicate that it is often not desirable to base the estimator? of the variance function V on the residuals from an optimal estimator f^ of f. In fact, the result shows that it is desirable to use an f with minimal bias. The main reason is that the bias and variance off have quite different effects on the estimation of V. The bias of f cannot be removed or even reduced in the second stage smoothing of the squared residuals, while the variance of f^ can be incorporated easily. The paper is organized as follows. Section 2 presents an upper bound for the minimax risk while Section 3 derives a rate-sharp lower bound for the minimax risk under both the global and local losses. The lower and upper bounds together yield the minimax rate of convergence. Section 4 discusses the obtained results and their implications for practical variance estimation in the nonparametric regression. The proofs are given in Section 5. 2 Upper bound In this section we shall construct a kernel estimator based on the square of the first order differences. Such and more general difference based kernel estimators of the variance function have been considered, for example, in Miiller and Stadtmiiller (1987 and 1993). For estimating a constant variance, difference based estimators has a long history. See von Neumann (1941, 1942), Rice (1984), Hall, Kay and Titterington (1990) and Munk, Bissantz, Wagner and Freitag (2005).

5 Define the Lipschitz class Aa(M) in the usual way: Aa(M) = {g: for all 0 5 x,y 5 1, 5 = 0,..., 101-1, ~g(~)(x)l 5 M, and lg(laj)(x) - g(laj)(y)l 5 M Jx - yla'} where 101 is the largest integer less than cu and a' = cu - LcuJ We shall assume that f E Aa(Mf) and V E AP(MV). We say that the function f "has cu derivatives" if f E Aa(Mf ) and V "has /3 derivatives" if V E A~(&). For i = 1,2,..., n - 1, set Di = yi - yi+l. Then one can write where 6, = f (xi)- f(~,+~), v,; = J;(v(xi) + V(xi+l)) and has zero mean and unit variance. We construct an estimator by applying kernel smoothing to the squared differences D: which have means 6: + 22V,. Let K(x) be a kernel function satisfying K(x)issupportedon[-1,1], S_: ~(x)x~dx = 0 for i = 1,2,..., 1/31 and It is well known in kernel regression that special care is needed in order to avoid significant, sometimes dominant, boundary effects. We shall use the boundary kernels with asymmetric support, given in Gasser and Miiller (1979 and 1984), to control the boundary effects. For any t E [O, 11, There exists a boundary kernel function Kt(x) with support [-I, t] satisfying the same conditions as K (x), i.e. ~~(x)x~dx = 0 for i = 1,2,..., 1/31 j- 1 1 t ~~~(x)dx 5 je < oo for all t E [o, I].

6 We can also make Kt(x) + K(x) as t + 1 (but this is not necessary here). See Gasser, Miiller and Mammitzsch (1985). For any 0 < h < a, x E [O,l], and i = 1,2,..., n - 1, let (xz+xz+1)/21 J(xz+xzl)/2 K ) when x E (h, 1 - h), (xz+xz+1)/2 x z + x z l ~ 2 K t when x = th for some t E [O,l], (x'+x'+1)/21kt(-y)du when x = 1 - th for some t E J(zz+xz-1)/2 [O, 11, h where for convenience we take the integral from 0 to (x1+x2)/2 instead of from (x1+x0)/2 to (xl + x2)/2 when i = 1, and integral from (xn-1 + xn-2)/2 to 1 when i = n - 1. Then we can see that for any 0 < x < 1, c:!: K,h(z) = 1. Define estimator Q as Similar to the mean function estimation problem, the optimal bandwidth hn can be easily seen to be hn = ~(n-l/(l+~fl)) for V E A~(Mv). For this optimal choice of the bandwidth, we have the following theorem. Theorem 1 Under the regresszon model (1) where xi = i/n and zt aare independent wzth zero mean, unit variance and uniformly bounded fourth moments, let the estimator Q be given as in (7) with the bandwidth h = ~(n-l/(l+~fl)). Then there exists some constant Co > 0 depending only on a, P, Mf and Mv such that for suficiently large n, SUP f EA"(M~),VEA~(MV) sup E(~(x*) - ~ (z*))~ 5 CO. ma~{n-~o, osx* n 1+211) (8) and Remark 1: The uniform rate of convergence given in (8) yields immediately the pointwise rate of convergence that for any fixed point x, E [O,1] Remark 2: The upper bound given in (8) is smaller than the minimax risk lower bound given in Hall and Carroll (1989). The lower bound in their paper is incorrect and the upper bound not optimal. See Sections 3 and 4 for further discussion.

7 3 Lower Bound In this section, we derive a lower bound for the minimax risk of estimating the variance function V under the regression model (1). The lower bound shows that the upper bound given in the previous section is rate-sharp. As in Hall and Carroll (1989) we shall assume in the lower bound argument that the errors are normally distributed, i.e., ti J N(0,l). Theorem 2 Under the regression model (1) with y % N(0, 1)) inf SUP Ellv - V II; > C1. max{r~-~~, G ~EA~(M~),VEAP(MV) and for any fixed x, E (0,l) inf SUP E(~(x,) v ~EA~(M~),vEA~(Mv) - ~ (2,))~ ZP n-m) 20 > C1. ma~{n-~~, n-'+zp) where Cl > 0 is a constant depending only on a, P, Mf and Mv. (10) (11) It follows immediately from Theorems 1 and 2 that the minimax rate of convergence for estimating V under both the global and local losses is show The proof of this theorem can be naturally divided into two parts. The first step is to This part is standard and relatively easy. Brown and Levine (2006) contains a detailed proof of this assertion for the case p = 2. Their argument can be easily generalized to other values of,b. We omit the details. The proof of the second step, is much more involved. The derivation of the lower bound (13) is based on a moment matching technique and a two-point testing argument. One of the main steps is to study a complicated hypothesis testing problem where the alternative hypothesis is a Gaussian location mixture with a special moment matching property. iid More specifically, let X1,..., X, - P and consider the following hypothesis testing problem between H~ : P = po = N(O,I + 8;)

8 and HI : P = PI = J N(Bnv, l)g(dv) where On > 0 is a constant and G is a distribution of the mean v with compact support. The distribution G is chosen in such a way that, for some positive integer q depending on a, the first q moments of G match exactly with the corresponding moments of the standard normal distribution. The existence of such a distribution is given in the following lemma from Karlin and Studden (1966). Lemma 1 For any fixed positive integer q, there exist a B < oo and a symmetric dis- tribution G on [-B, B] such that G and the standard normal distribution have the same first q moments, i.e. JP, Lrn +a xjg(dx) = where cp denotes the density of the standard normal distribution. xjcp(x)dx, j = 1,2,..., q The moment matching property makes the testing between the two hypotheses L'difficult". The lower bound (13) then follows from a two-point argument with an appropriately chosen 8,. Technical details of the proof are given in Section 5. Remark 3: For a between 114 and 118, a much simpler proof can be given with a two- point mixture for PI which matches the mean and variance, but not the higher moments, of Po and PI. However, this simpler proof fails for smaller a. It appears to be necessary in general to match higher moments of Po and PI. Remark 4: Hall and Carroll (1989) gave the lower bound c max{n-4~l(1+2~)), n-2pl(1+2p))) for the minimax risk. This bound is larger than the lower bound given in our Theorem 2 and as we have noted it is incorrect. This is due to a miscalculation on appendix C of their paper. A key step in that proof is to find some d 2 0 such that In the above expression, Nl denotes a standard normal random variable. But in fact - exp( ). I-, exp(x/a) + exp(-x/2) JZ;;;~ 2d 8 This is an integral of an odd function which is identically 0 for all d.

9 4 Discussion Variance function estimation in regression is more typically based on the residuals from a preliminary estimator f of the mean function. Such estimators have the form where w,(x) are weight functions. A natural and common approach is to subtract in (14) an optimal estimator f of the mean function f (x). See, for example, Hall and Carroll (1989), Neumann (1994), Ruppert, Wand, Holst, and Hossjer (1997), and Fan and Yao (1998). When the unknown mean function is smooth, this approach often works well since the bias in f is negligible and V can be estimated as well as when f is identically zero. However, when the mean function is not smooth, using the residuals from an optimally smoothed f will lead to a sub-optimal estimator of V. For example, Hall and Carroll (1989) used a kernel estimator with optimal bandwidth for f^ and showed that the resulting variance estimator attains the rate of over f E Aa(Mf) and V E A~(MV). This rate is strictly slower than the minimax rate 2B P when & < or equivalently, a < -. Consider the example where V belongs to a regular parametric family, such as {V(x) = exp(ax + b) : a, b E R). As Hall and Carroll have noted this case is equivalent to the case of,f3 = ca in results like Theorems 1 and 2. Then the rate of convergence for this 4a estimator becomes nonparametric at n-2.l+l for a < i, while the optimal rate is the usual 1 parametric rate n-3 for all a 1 4 and is n-4a for 0 < a < 4. The main reason for the poor performance of such an estimator in the non-smooth setting is the "large" bias in f. An optimal estimator f of f balances the squared bias and variance. However, the bias and variance of f have significantly different effects on the estimation of V. The bias of f^ cannot be further reduced in the second stage smoothing of the squared residuals, while the variance of f can be incorporated easily. For f E Aa(Mf ), the maximum bias of an optimal estimator f^ is of order n-* B becomes the dominant factor in the risk of P when a < m. which To minimize the effect of the mean function in such a setting one needs to use an estimator f(xi) with minimal bias. Note that our approach is, in effect, using a very crude estimator f^ of f with f^(x,) = y,+l. Such an estimator has high variance and low bias. As we have seen in Section 2 the large variance of f does not pose a problem (in terms of

10 rates) for estimating V. Hence for estimating the variance function V an optimal f^ is the one with minimum possible bias, not the one with minimum mean squared error. (Here we should of course exclude the obvious, and not useful, unbiased estimator f"(xi) = yi). Another implication of our results is that the unknown mean function does not have any first-order effect for estimating V as long as f has more than 114 derivatives. When a > 114, the variance estimator is essentially adaptive over f E Aff(Mj) for all a > 114. In other words, if f is known to have more than 114 derivatives, the variance function V can be estimated with the same degree of first-order precision as if f is completely known. However, when a < 114, the rate of convergence for estimating V is entirely determined by the degree of smoothness of the mean function f. 5 Proofs 5.1 Upper Bound: Proof of Theorem 1 We shall only prove (8). Inequality (9) is a direct consequence of (8). Recall that 1 where 6, = f (xi)- f (xi+l), v,' = 4; (v(xi) + V(xi+1)) and Without loss of generality, suppose h = n-1/(1+2p). It is easy to see that for any x, E [0, I.], xi K:(x,) = 1, and when X, > (xi + xi+1)/2 + h or X, 5 (xi + ~ ~-~)/2 - h, K:(x,) equals to 0. Suppose Ic < c, we also have where K, (u) = K (u) when x, E (h, 1 - h); K, (u) = Kt (u) when x, = th for some t E [0, 11; and K,(u) = Kt(-u) when x, = 1 - th for some t E [O, 11.

11 For all f E Aa(Mf) and V E A~(Mv), the mean squared error of v at x, satisfies Suppose a 5 114, otherwise n-4a < n-2p/(1+2p) for any P. Since for any i, ldil = If (xi)- f (~i+~)l < Mf Ixi - xi+lla = Mfn-", we have Note that for any x, y E [O,l], Taylor's theorem yields x (x - u)lpj-l (LPl - Mv < 12-YIP. LPl Mv Ix- Y I P- LPJ du

12 and Since the kernel functions have vanishing moments, for j = 1,2, -., L,B], for some constant c' > 0. Similarly ~111' K,~(X,)(~ i+l - x*)j 5 dn-l. So, for some constant (? > 0 which does not depend on x,. Then we have

13 The last inequality is due to the fact 0 < h + < h + < 3h. On the other hand, and where p4 denotes the uniform bound for the fourth moments of the ei. Putting the four terms together we have, uniformly for all x, E [0, I], f E Aa(Mf) and v E AP(M~) for some constant Co > 0. This proves (8) Lower Bound: : Proof of Theorem 2 We shall only prove the lower bound for the pointwise squared error loss. The same proof with minor modifications immediately yields the lower bound under integrated squared error. Note that, to prove inequality (13), we only need to focus on the case where a < 114, otherwise n-2pl(1+2p) is always greater than np4" for sufficiently large n and then (13) follows directly from (12). For a given 0 < a < 114, there exists an integer q such that (q + 1)a > 1. For convenience we take q to be an odd integer. From lemma 1, there is a positive constant B < r x and a symmetric distribution G on [-B, B] such that G and N(0,l) have the

14 - same first q moments. Let ri, i = 1,..., n, be independent variables with the distribution G. Set On = $n-a, fa 0, &(x) = 1 + $2 and Vl(x) = 1. Let g(x) = 1-2nlxl for x E [-&-, &-I and 0 otherwise. Define the random function fl by Then it is easy to see that fl is in Aa(Mj) for all realizations of Ti. Moreover, f 1 (xi) = Onri are independent and identically distributed. Now consider testing the following hypotheses, where ~i are independent N(0,l) variables which are also independent of the ri's. Denote by Po and Pl the joint distributions of yi's under Ho and H1, respectively. Note that for any estimator p of V, where p(po, PI) is the Hellinger affinity between Po and PI. See, for example, Le Cam (1986). Let po and pl be the probability density function of Po and Pl with respect to the Lebesgue measure p, then p(po, PI) = J m d p. The minimax lower bound (13) follows immediately from the two-point bound (16) if we show that for any n, the Hellinger affinity p(po, PI) 2 C for some constant C > 0. (C may depend on q, but does not depend on n.) Note that under Ho, yi N N(0,l + 02) and its density do can be written as J p(t - uo~)g(~u). It is easy to see that p(po, PI) = (J&&'&d~)~, since the yz's are independent variables. Under HI, the density of yz is dl(t) Note that the Hellinger affinity is bounded below by the total variation affinity, Taylor expansion yields

15 where Hk(t) is the corresponding Hermite polynomial. And from the construction of distribution G, vzp(v)dv for i = 0,1,..., q. so, Suppose q + 1 = 2p for some integer p, it can be seen that and where (22 - I)!! A (22-1) x (2i - 3) x...3 x 1. So from (17),

16 and then For the Hermite polynomial H2ir we have For sufficiently large n, 9, < 112 and it then from the above inequality that and

17 Then from (18) where c is a constant that only depend on q. So Since cr(q + 1) 2 1, limn,,(l (16). 1 - ~n-"(q+l))~ 2 epc > 0 and the theorem then follows from Acknowledgment TTC would like to thank William Studden for discussions on the finite moment matching problem and for the reference Karlin and Studden (1966). References [I] Brown, L. D. and Levine, M. (2006). Variance estimation in nonparametric regression via the difference sequence method. Technical Report. [2] Fan, J. and Yao, Q. (1998). Efficient estimation of conditional variance functions in stochastic regression. Biometrika 85, [3] Hall, P. and Carroll, R. J. (1989). Variance function estimation in regression: the effect of estimating the mean. J. Roy. Statist. Soc. Ser. B 51, [4] Muller, H. G. and Stadtmuller, U. (1987). Estimation of heteroscedasticity in regression analysis. Ann. Statist. 15, [5] Muller, H. G. and Stadtmuller, U. (1993). On variance function estimation with quadratic forms. J. Stat. Planning and Inf. 35, [6] Gasser, T. and Muller, H. G. (1979). Kernel estimation of regression functions. In Smoothing Techniques for Curve Estimation, pp Berlin: Springer. (Lecture Notes in Mathematics No. 757). [7] Gasser, T. and Muller, H. G. (1984). Estimating regression functions and their derivatives by the kernel method. Scand. J. Statist. 11,

18 [8] Gasser, T., Miiller, H. G. and Mammitzsch, V. (1985). Kernels for nonparametric curve estimation. J. Roy. Statist. Soc. B 47, [9] Hall, P., Kay, J. and Titterington, D. M. (1990). Asymptotically optimal differencebased estimation of variance in nonparametric regression. Biometrika 77, [lo] Karlin, S. and Studden, W. J. (1966). Tchebycheff Systems: With Applications In Analysis And Statistics. Interscience, New York. [ll] Le Cam, L. (1986). Asymptotic Methods in Statistical Decision Theory. Springer- Verlag, New York Munk, A,, Bissantz, N., Wagner, T. and Freitag, G. (2005). On difference based variance estimation in nonparametric regression when the covariate is high dimensional. J. Roy. Statist. Soc. B 67, [13] Neumann, M. (1994). Fully data-driven nonparametric variance estimators. Statistics 25, [14] Parzen, E. (1962). On estimation of a probability density function and mode. Ann. Math. Statist. 33, [15] Rice, J. (1984). Bandwidth choice for nonparametric kernel regression. Ann. Statist. 12, [16] Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function. Ann. Math. Statist. 27, [17] Ruppert, D., Wand, M. P., Holst, U. and Hossjer, 0.(1997). Local polynomial variance function estimation. Technometrics 39, [18] Stone, C. J. (1980). Optimal rates of convergence for nonparametric estimators. Ann. Statist. 8, [19] von Neumann, J. (1941). Distribution of the ratio of the mean squared successive difference to the variance. Ann. Math. Statist. 12, [21:1] von Neumann, J. (1942). A further remark concerning the distribution of the ratio of the mean squared successive difference to the variance. Ann. Math. Statist. 13,