CONVERGENCE RATES OF COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION

Transcription

1 Statistica Sinica 16(2006), CONVERGENCE RATES OF COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION Yi Lin and Ming Yuan University of Wisconsin-Madison and Georgia Institute of Technoogy Abstract: Reguarization with radia basis functions is an effective method in many machine earning appications. In recent years casses of radia basis functions with compact port have been proposed in the approximation theory iterature and have become more and more popuar due to their computationa advantages. In this paper we study the statistica properties of the method of reguarization with compacty ported basis functions. We consider three popuar casses of compacty ported radia basis functions. In the setting of estimating a periodic function in a white noise probem, we show that reguarization with (periodized) compacty ported radia basis functions is rate optima and adapts to unknown smoothness up to an order reated to the radia basis function used. Due to resuts on equivaence of the white noise mode with many important modes incuding regression and density estimation, our resuts are expected to give insight on the performance of such methods in more genera settings than the white noise mode. Key words and phrases: method of reguarization, nonparametric estimation, radia basis functions, rate of convergence, reproducing kerne. 1. Introduction Radia basis functions (RBF s) are popuar toos in function approximation and have been used in many machine earning appications. Exampes incude RBF reguarization networks and, more recenty, port vector machines. See, for exampe, Girosi, Jones and Poggio (1993), Smoa, Schökopf and Müer (1998), Wahba (1999) and Evgeniou, Ponti and Poggio (2000). Traditiona radia basis functions have goba port. In recent years RBF s with compact port have been proposed in the approximation theory iterature (Wu (1995); Wendand (1995); Buhmann (1998)), and have become more and more popuar in function approximation and machine earning appications due to their computationa advantages. In this paper, we first present a brief overview of the commony used compacty ported radia basis functions, and then give some theoretica resuts on the asymptotic properties of the method of reguarization with compacty ported RBF s. The radia basis functions have the form Φ(x) = φ( x ) for vector x R d. Here φ is a univariate function defined on [0, ) and is the ordinary Eucidean

2 426 YI LIN AND MING YUAN norm on R d. To be appicabe in the method of reguarization, a radia basis function must satisfy that K(x, y) Φ(x y) is positive definite or conditionay positive definite. A function K(x, y): R d R d R is said to be positive definite if for any positive integer n, and any distinct x 1,..., x n R d, we have n j=1 k=1 n a j a k K(x j, x k ) > 0 (1.1) for any nonzero vector (a 1,..., a n ). It is said to be conditionay positive definite of an order m if (1.1) hods for any nonzero vector (a 1,..., a n ) satisfying n i=1 a ip(x i ) = 0 for a poynomias p of degree ess than m. Thus positive definiteness is the same as conditionay positive definiteness of zero order. Any positive definite or conditiona positive definite function K(, ) is associated with a reproducing kerne Hibert space of functions on R d. For an introduction to the (conditiona) positive definite functions and reproducing kerne Hibert spaces, the readers are referred to Wahba (1990). Schaback (1997) provided a survey of reproducing kerne Hibert spaces corresponding to radia basis functions. We say a radia basis function Φ (or the corresponding φ) is (conditionay) positive definite if K(x, y) = Φ(x y) = φ( x y ) is (conditionay) positive definite. For a positive definite radia basis function Φ, the squared norm in the associated reproducing kerne Hibert space H Φ can be written as J(f) = (2π) d/2 R f(ω) 2 / Φ(ω)dω for any function f H d Φ. Here f is the Fourier transform of f, f(ω) = (2π) d 2 f(x)e ixt ω dx, R d and, simiary, Φ is the the Fourier transform of Φ. For a conditiona radia basis function, J(f) is a squared semi-norm. Let f be a function of interest in a nonparametric function estimation probem. The method of reguarization with a radia basis function takes the form min f H Φ [L(f, data) + λj(f)], (1.2) where L is the empirica oss, often taken to be the negative og-ikeihood. The smoothing parameter λ contros the trade-off between minimizing the empirica oss and obtaining a smooth soution. For exampe, in nonparametric regression y j = f(x j ) + δ j, j = 1,..., n, (1.3) where x j R d, j = 1,..., n, are the regression inputs, y j s are the responses, and δ j s are independent Gaussian noises. In this case we may take L(f, data) = n j=1 (y j f(x j )) 2 in (1.2).

3 COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION 427 In many situations incuding regression and generaized regression, the representer theorem (Kimedorf and Wahba (1971)) for reguarization over reproducing kerne Hibert spaces guarantees that the soution to (1.2) over H Φ fas in the finite dimensiona space spanned by {Φ( x j ), j = 1,..., n}. For exampe, in the regression probem, when Φ is positive definite the soution to (1.2) has the form ˆf = n j=1 c jφ(x x j ), and the coefficient vector c = (c 1,..., c n ) can be soved by minimizing y Kc 2 + λc Kc. (1.4) Here, with a itte abuse of notation, K stands for the n by n matrix (Φ(x i x j )), and y stands for the vector (y 1,..., y n ). It is cear that the soution to (1.4) can be obtained by soving the inear system (K + λi)c = y. (1.5) Traditiona radia basis functions are gobay ported. Common exampes are given by φ(r) = r 2m og(r) (thin pate spine RBF), φ(r) = e ϱr2 /2 (Gaussian RBF), φ(r) = (c 2 +r 2 ) 1/2 (the mutiquadrics RBF), and φ(r) = (c 2 +r 2 ) 1/2 (the inverse mutiquadrics RBF). In recent years radia basis functions with compact port have been constructed. These RBF s are computationay appeaing and have become more and more popuar in practica appications. In Section 2 we give a brief introduction to them that focuses on aspects that are most reevant to statistica appications. In Sections 3 4 we present some resuts on the asymptotic properties of the method of reguarization with (periodized) compacty ported radia basis functions. The asymptotic properties wi be studied in the probem of estimating periodic functions in the nonparametric white noise mode: Y n (t) = t π f(u)du + n 1 2 B(t), t [ π, π], (1.6) where B(t) is a standard Brownian motion on [ π, π] and we observe Y n = (Y n (t), π t π). We consider the situation where the function f beongs to a certain Soboev eipsoid of periodic functions on [ π, π], H m (Q), with unknown m: H m (Q) = {f L 2 ( π, π) : f is 2π-periodic, π π [f(t)] 2 + [f (m) (t)] 2 dt Q}. (1.7) It is we known that spaces of periodic functions induce periodic reproducing kernes, see Schaback (1997) and Smoa, Schökopf and Müer (1998), for exampe. Therefore, for the estimation of periodic functions, it is appropriate to consider the periodic version of the radia basis functions. For any function

4 428 YI LIN AND MING YUAN Φ, the corresponding periodized function with period 2π is given by: Φ 0 (s) = k Zd Φ(s 2πk), where Z is the set of integers. Consideration of the white noise mode is motivated by the resuts on its equivaence to nonparametric regression (Brown and Low (1996)), density estimation (Nussbaum (1996)), spectra density estimation (Goubev and Nussbaum (1998)), and nonparametric generaized regression (Grama and Nussbaum (1997)). We choose to work with periodic function estimation because it aows a detaied and unified asymptotic anaysis. We beieve the resuts obtained in this paper aso give insights on the statistica properties of radia basis function reguarization in more genera situations. For simpicity, we work in the case of univariate white noise mode, but simiar resuts can be obtained in higher dimensiona situations. Our resuts in Section 3 show that for (1.6) with f H m (Q), the method of reguarization with periodized compacty ported radia basis functions achieves the optima rate of convergence, and that this is true whenever m does not exceed a certain number known for the specific compacty ported radia basis function used, and the tuning parameter λ is appropriatey chosen. In Section 4 we show that the smoothing parameter λ can be chosen adaptivey in our situation without any oss in the rate of convergence. Section 5 contains some discussion. The technica proofs are given in the Appendix. Throughout this paper, a n b n means that there exists 0 < c 1 < c 2 < such that c 1 < a n /b n < c 2, for a n = 1, 2, Radia Basis Functions with Compact Support In this section we briefy review those aspects of commony used, compacty ported radia basis functions that are most reevant to machine earning and statistica appications. The review is mosty based on materia in the papers by Wendand (1995), Schaback and Wendand (2000), and the book by Buhmann (2003). Interested readers are referred to them for further detais. There are severa popuar casses of radia basis functions with compact port. These casses of radia basis functions are constructed by finding compacty ported radia symmetric functions with positive (or nonnegative and not identicay zero) Fourier transforms. Let Φ(x) φ( x ) be a bounded integrabe radiay symmetric function from R d to R, with Fourier transform Φ. Then for any n Z, distinct x 1,..., x n R d, and nonzero vector (a 1,..., a n ), it is easy to show that, n j,k=1 a j a k φ( x j x k ) = (2π) d 2 R d n a j e ixt j ω 2 Φ(ω)dω. (2.1) j=1

5 COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION 429 Since the exponentias are inear independent on every open subset of R d, we see from (2.1) that if Φ is positive (or nonnegative and not identicay zero), then Φ is a positive definite radia basis function. In fact, this is a specia case of the famous Bochner s theorem that competey characterizes transation invariant positive definite functions through Fourier transforms Wendand functions A particuary interesting part of the work on radia basis functions with compact port is due to Wendand, initiated in part by Wu (1995). The Wendand functions are of the form { p(r) if 0 r 1, φ(r) = 0 if r > 1, (2.2) with a univariate poynomia p. They are ported in the unit ba, but a different port can easiy be achieved by scaing. Define an operator I by Ig(r) r g(t)tdt, and write φ (r) = (1 r) +. Then the Wendand functions are φ d,k (r) = I k φ d 2 +k+1(r), where I k stands for the I-operator appied k times, and s stands for the maximum integer ess than or equa to s. It is easy to see that φ d,k is compacty ported, and is a poynomia within its port. It has been shown (Wendand (1995)) that Φ d,k (x) φ d,k ( x ) has stricty positive Fourier transform except when d = 1 and k = 0, in which case the Fourier transform is nonnegative and not identica zero. Thus in a cases φ d,k (r) gives rise to a positive definite radia basis function. It was aso shown in Wendand (1995) that φ d,k (r) is 2k times continuousy differentiabe, and is of minima poynomia degree among a positive definite radia basis functions of the form (2.2) that is 2k times continuousy differentiabe. The expicit forms of Wendand functions can be computed directy from the definition or, aternativey, from the formua (Wendand (1995)): I k (1 r) + = with coefficients β 0,0 = 1, and β j,k+1 = k i=j 1 k i=0 β i,k r i (1 r) +2k i +, β i,k [i + 1] i j+1 ( + 2k i + 1) i j+2, 0 j k + 1,

6 430 YI LIN AND MING YUAN if the term for i = 1 and j = 0 is ignored. Here ( ) k and [ ] k are defined by [q] 1 = (1 + q) 1, (q) 0 = [q] 0 = 1, (q) k = q(q + 1) (q + k 1) and [q] k = q(q 1) (q k + 1). By direct cacuation we obtain the foowing exampes of Wendand functions: if = d/2 + k + 1, we have (1 r) + when k = 0, φ d,k (r) = (1 r) +1 + {( + 1)r + 1}/( + 1) 2 when k = 1, (1 r) +2 + {( + 1)( + 3)r2 + 3( + 2)r + 3}/( + 1) 4 when k = 2. (2.3) 2.2. Buhmann functions A different technique for generating compacty ported radia basis functions is due to Buhmann (1998, 2000). Let g(β) = (1 β µ ) ν +,with 0 < µ 1/2 and ν 1, and define φ d,γ,α (r) = r 2 (1 r2 β )γ β α g(β)dβ, and Φ d,γ,α (x) φ d,γ,α ( x ), where d is the dimension of x. It is easy to see that φ d,γ,α (r) is compacty ported. It is ported in the unit ba, but a different port can easiy be achieved by scaing. It has been shown (Buhmann (2003)) that 1. For d = 1, the Fourier transform of Φ d,γ,α (x) is everywhere positive if (i) γ 1, 1 < α γ/2, or (ii) γ 1/2, 1 < α min(1/2, γ 1/2); 2. For d > 1, the Fourier transform of Φ d,γ,α (x) is everywhere positive if γ (d 1)/2 and 1 < α [γ (d 1)/2]/2. Thus φ d,γ,α induces positive definite radia basis functions and Φ d,γ,α (x) C 1+ 2α (R d ), where s stands for the minimum integer greater than or equa to s. For exampe, if d = 1 and we take µ = 1/2, ν = 1, α = 1/2, and γ = 1, we get { 1 2 φ 1,1,1/2 (r) = r r3 r if 0 r 1, 0 if r > 1, This function is twice continuousy differentiabe. If d = 2 and we take µ = 1/2, ν = 1, α = 0, and γ = 2, we get { 4r 2 og r + r r3 + 4r if 0 r 1, φ 2,2,0 (r) = 0 if r > 1.

7 COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION 431 This function is once continuousy differentiabe Product functions A we-known property of positive definite functions is that the product of positive definite functions is positive definite. See, for exampe, Proposition 3.12 of Cristianini and Shawe-Tayor (2000). This suggests a strategy for constructing new compacty ported radia basis function based on known ones. If φ 1 (r) is a positive definite radia basis function and φ 2 (r) is a positive definite function radia basis function with compact port, φ(r) = φ 1 (r)φ 2 (r) is a positive definite radia basis function with compact port. This approach was proposed in Gaspari and Cohn (1999). As an exampe, we take φ 1 to be the Gaussian kerne e ϱr2 /2 and φ 2 to be a Wendand or Buhmann function. 3. Rates of Convergence in Soboev Spaces To study the convergence rates of the method of reguarization with periodized radia basis function in the white noise probem (1.6), we first transform the probem into a form that is particuary amenabe to anaysis. Let {ξ 0 (t) = (2π) 1/2, ξ 2 1 (t) = π 1/2 sin(t), and ξ 2 (t) = π 1/2 cos(t)} be the cassica trigonometric basis in L 2 ( π, π), and et θ = (f, ξ ) be the corresponding Fourier coefficients of f, where (f, φ) = π π f(t)ξ(t)dt denotes the usua inner product in L 2 ( π, π). By converting the functions into the corresponding sequences of Fourier coefficients, we can see that the white noise probem (1.6) is equivaent to the foowing Gaussian sequence mode: y = θ + ɛ, = 0, 1,..., (3.1) where ɛ s are independent N(0, 1/n) variabes. The condition that f beongs to H m (Q), the Soboev eipsoid of periodic functions defined in (1.7) with unknown m and Q, is equivaent to the condition that ρ θ 2 Q, (3.2) =0 where ρ 0 = 1, and ρ 2 1 = ρ 2 = 2m + 1. To study the method of reguarization (1.2) with the periodized radia basis function Φ 0 (s) = k Z Φ(s 2πk), we need the norm or seminorm of the reproducing kerne Hibert space corresponding to Φ 0. The casses of compacty ported radia basis functions we consider in this paper are positive definite, therefore we seek the norm in the reproducing kerne Hibert space. Proposition 1. Let Φ be a positive definite radia basis function with compact port, and Φ 0 be the periodization of Φ: Φ 0 (s) = k Z Φ(s 2πk). Then Φ 0

8 432 YI LIN AND MING YUAN is positive definite and the norm in the reproducing kerne Hibert space corresponding to Φ 0 is given by f 2 = β θ 2, (3.3) =0 where the θ s are the Fourier coefficients of f, and β 0 = (2π) 1/2 [ Φ(0)] 1, β 2 1 = β 2 = (2π) 1/2 [ Φ()] 1, = 1, 2,.... Therefore the method of reguarization with Φ 0 corresponds to min (y θ ) 2 + λ β θ 2 (3.4) =0 with β 2 1 = β 2 = (2π) 1/2 [ Φ()] 1. Theorem 1. Assume f H m (Q) with m 1 in the white noise mode (1.6). Consider the method of reguarization estimator ˆθ at (3.4) with β 2 1 = β 2 = (2π) 1/2 [ Φ()] If φ(r) = σ 1 φ d,k (r/σ) is a scaed Wendand radia basis function of compact port with scae σ, and k 1, m 2k+2, then when λ (n) n (2k+2)/(2m+1) we have =0 E(ˆθ θ ) 2 n 2m 2m+1. That is, the optima rate of convergence in the Soboev space is achieved if the smoothing parameter is appropriatey chosen. 2. If φ(r) = σ 1 φ d,γ,α (r/σ) is a scaed Buhmann radia basis function of compact port with scae σ, and m 3 + 2α, then when λ (n) n (3+2α)/(2m+1) we have E(ˆθ θ ) 2 n 2m 2m+1. That is, the optima rate of convergence in the Soboev space is achieved if the smoothing parameter is appropriatey chosen. 3. If φ(r) = G(r)φ 1 (r) where G(r) = e ϱr2 /2 is the Gaussian radia basis function and φ 1 is a scaed Wendand function (or scaed Buhmann function) given in part 1 (or 2), then when the conditions of part 1 (or 2) hod. E(ˆθ θ ) 2 n 2m 2m+1

9 COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION Adaptive Tuning The proper choice of the tuning parameters depends on the unknown smoothness order m of the estimand. In this section, we consider choosing the tuning parameters λ and σ adaptivey with Maows C p. We show that the tuning parameters chosen by the unbiased estimator of risk give rise to an estimator that has the same asymptotic risk as the estimator with the optima (theoretica) tuning parameter. Thus no asymptotic efficiency is ost due to not knowing m. Formay, we take a finite number of σ s, σ 1,..., σ J, and tune λ and σ jointy over λ and σ j {σ 1,..., σ J }. In our asymptotic considerations, a range of [0, 1] for λ suffices, since λ shoud go to zero. In practice we may use a sighty arger range. The tuning of σ in addition to λ is motivated by the common practice of tuning both λ and σ in the impementation of radia basis function reguarization. A our asymptotic resuts go through if we fix one σ and ony tune λ (this corresponds to J = 1 in our setting). However, with a finite sampe size it is desirabe to tune both λ and σ. In fact, it is cear that in the finite sampe situation, fixing σ at a vaue that is too sma compared to the distances between sampe points eads to poor estimation performance. The tuning is based on the C p criterion. Writing τ = (1 + λβ ) 1, our estimator is ˆθ = τ y. We can express the risk of our estimator as E(ˆθ θ ) 2 = 1 n τ 2 + (1 τ ) 2 θ 2. Now an unbiased estimator for θ 2 is y2 (1/n). Pugging in, we get that =0 [(τ 2 2τ )(y 2 1 n ) + 1 n τ 2 ] = [(τ 2 2τ )y n τ ] (4.1) =0 is an unbiased estimator of E(ˆθ θ ) 2 θ 2. We choose λ and σ that minimize (4.1), and use the corresponding estimator ˆθ. Li (1986, 1987) estabished the asymptotic optimaity of C p in many nonparametric function estimation methods, incuding the method of reguarization. Kneip (1994) studied the adaptive choice among ordered inear smoothers with the unbiased risk estimator. A famiy of ordered inear smoothers satisfy the condition that for any member ˆθ = τ y, = 0, 1,..., of the famiy, we have τ [0, 1], ; and for any two members of the famiy τ y and τ y, = 0, 1,...., we have either τ τ,, or τ τ,. Theorem 2. Consider (3.1) and the method of reguarization (3.4) with φ(r) defined as in Theorem 1(1), 1(2) or 1(3). Suppose λ and σ minimize (4.1) =0 =0

10 434 YI LIN AND MING YUAN over λ [0, 1] and σ {σ 1,..., σ J }, and ˆθ is the corresponding method of reguarization estimator. Then E(ˆθ θ ) 2 n 2m 2m+1. Therefore the adaptive method of reguarization with compacty ported radia basis function estimator ˆθ achieves the optima rate in H m (Q). In the case of product radia basis functions, one may want to tune ϱ k {ϱ 1,..., ϱ K } as we as λ [0, 1] and σ {σ 1,..., σ J }. The estimator obtained sti achieves the optima rate in H m (Q), as can be proved in simiar fashion. 5. Discussion Compacty ported radia basis functions have aso been used in spatia statistics to mode and approximate spatia covariance. Gneiting (2002) is a good source of compacty ported covariances. Furrer, Genton and Nychka (2005) proposed a method to taper the spatia covariance function to zero beyond a certain range, using an appropriate compacty ported radia radia basis function. This gives an approximation to the standard inear spatia predictor that is both accurate and computationay efficient. They show that their approximate taper-based method makes it possibe to anayze and fit very arge spatia data sets, and gives a inear predictor that is neary the same as the exact soution. Radia basis function reguarization has been shown to give exceent performance in practica appications. Direct impementation with traditiona gobay ported radia basis functions require a computation of order O(n 3 ). Compacty ported radia basis functions have computationa advantages. The resuts in this paper suggest that reguarization with compacty ported radia basis functions enjoy good statistica properties. Further study is needed to investigate these properties more generay. Another interesting research topic is how to efficienty make use of their computationa advantages. This is especiay important when anayzing arge datasets. When the compacty ported radia basis function is used in the method of reguarization, the kerne matrix K in (1.5), for exampe, is sparse. Sparse matrix computation can be used directy in the non-iterative evauation of the estimator in (1.5). This is the approach taken in Zhang, Genton and Liu (2004). Another approach to improving computation speed is to sove (1.5) with an iterative method, such as the conjugate gradient method. Due to the sparsity and positive definiteness of the inear system (1.5), the conjugate gradient method is expected to be very effective. There have been a number of effective proposas in the approximation theory iterature for

11 COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION 435 fast computation in function interpoation with compacty ported radia basis functions. See, for exampe, Schaback and Wendand (2000) and Foater and Iske (1996). It is promising that adapting these ideas for function approximation in noise free situations to function estimation in noisy situations can ead to fast and efficient machine earning and statistica methods based on compacty ported radia basis functions. Acknowedgement Lin s research was ported in part by NSF Grant DMS Appendix Proof of Proposition 1. Appying Lemma 14 of Wiiamson, Smoa and Schökopf (2001) (see aso Lin and Brown (2004)), we get Φ 0 (s t) = (2π) 1 2 Φ(0) + which can be rewritten as 2(2π) 1 2 Φ() cos[(s t)], =1 Φ 0 (s t) = (2π) 1 2 Φ(0)ξ0 (s)ξ 0 (t) + + (2π) 1 2 Φ()ξ2 1 (s)ξ 2 1 (t) =1 (2π) 1 2 Φ()ξ2 (s)ξ 2 (t), =1 where the ξ j form the cassica trigonometric basis in L 2 ( π, π). Then ξ, = 0, 1,..., are the eigenvectors of the reproducing kerne Φ 0 (s t), and the corresponding eigenvaues are β 1, where β0 1 = (2π) 1/2 Φ(0), β = β 1 2 = (2π) 1/2 Φ(), = 1, 2,..... Now appying Lemma of Wahba (1990), we get that for any function f in the reproducing kerne Hibert space induced by Φ 0, f 2 = =0 β (f, ξ ) 2, and the proposition is proved. Proof of Theorem 1(1). Since k 1, we have Φ d,k (ω) (1+ ω 2 ) (1+2k+d)/2 (Theorem 2.1 of Wendand (1998)). Therefore, for our scaed Wendand function Φ and d = 1, we have Φ(ω) = Φ d,k (σω) (1 + σ 2 ω 2 ) (k+1), and then β 2 1 = β 2 (1 + σ 2 2 ) k+1. Soving the minimization probem (3.4), we get the method of reguarization estimator ˆθ = (1 + λβ ) 1 y. We consider the variance and bias of ˆθ separatey.

12 436 YI LIN AND MING YUAN First, since λ n (2k+2)/(2m+1), we have, varˆθ = 1 (1 + λβ ) 2 2 n n For the bias part, we have (E ˆθ θ ) 2 = = = Q max 2 n [1 + λ(σ) 2k+2 ] 2 =0 0 = 2 n λ 1 2k+2 σ 1 [1 + λ(σx) 2k+2 ] 2 dx n 1 λ 1 2k+2 σ 1 n 2m =0 =0 (1 + λ 1 β 1 (1 + λ 1 β 1 0 ) 2 θ 2 [(1 + λ 1 β 1 ) 2 ρ 1 (1 + y 2k+2 ) 2 dy 2m+1. ) 2 ρ 1 (ρ θ 2 ) ] max 0 < [{1 + λ 1 (1 + σ 2 2 ) (k+1) } 2 (1 + 2m ) 1 ]. Here ρ 2 1 = ρ 2 = 1 + 2m are the coefficients in the condition (3.2) of the Soboev eipsoid H m (Q). Define B λ () = {1 + λ 1 (1 + σ 2 2 ) (k+1) } 2 (1 + 2m ). Then Now B λ (0) = (1 + λ 1 ) 2 and (E ˆθ θ ) 2 [min 0 B λ()] 1. min B λ() min { λ 1 (σ 2 2 ) (k+1) } 2 2m min x 1 {1 + λ 1 (σ 2 x 2 ) (k+1) } 2 x 2m = min x 1 {xm + λ 1 σ (2k+2) x m (2k+2) } 2. (A.1) If m = 2k + 2, then obviousy the ast expression is {1 + λ 1 σ (2k+2) } 2 λ 2 = λ 2m/(2k+2). When m < 2k+2, direct cacuation shows that the minimum in the ast expression is achieved at x = (2k + 2 m) 1/(2k+2) m 1/(2k+2) λ 1/(2k+2) σ 1, and the minimum is of order λ 2m/(2k+2). Since m 2k+2, we have min 0 B λ () λ 2m/(2k+2). From (A.1) and λ n (2k+2)/(2m+1), we get E (ˆθ θ ) 2 = varˆθ + (E ˆθ θ ) 2 n 2m 2m+1.

13 COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION 437 Proof of Theorem 1(2). We note that Φ d,γ,α (ω) (1 + ω 2 ) (1+α+d/2) (Buhmann (2003, p.158)). The rest of the proof is simiar to that of Theorem 1(1). Proof of Theorem 1(3). In the foowing, a = k +1 if φ 1 is Wendand function and a = α + 3/2 if φ 1 is Buhmann function. We have Φ(ω) = Φ 1 (s) G(ω s)ds (1 + σ 2 s 2 ) a e (s ω) 2ϱ ds E(1 + σ 2 Z 2 ω,ϱ ) a, where Z ω,ϱ is N(ω, ϱ). Since E(1 + σ 2 Z 2 ω,ϱ) a is a positive and continuous function in ω, when ω we have E(1 + σ 2 Z 2 ω,ϱ) a = ω 2a E[1/ω 2 + (σ + σz 0,ϱ/ω 2) 2 ] a ω 2a. We get that Φ(ω) E(1 + σ 2 Z 2 ω,ϱ) a (1 + ω 2 ) a. The rest of the proof is simiar to that of Theorem 1(1). Proof of Theorem 2. It is easy to check that, for any fixed σ {σ 1,..., σ J }, the method of reguarization estimators with varying λ form a famiy of ordered inear smoothers. Appying the resut in Kneip (1994) (see aso expression (15) in Cavaier, Goubev, Picard and Tsybakov (2002)) shows that there exist positive constants C 1 and C 2 such that, for any θ with θ 2 < + and any positive constant B, we have E(ˆθ θ ) 2 (1 + C 1 B 1 ) min{ E(ˆθ θ ) 2 } + n 1 C 2 B. (A.2) λ,σ j Therefore we have E(ˆθ θ ) 2 (1 + C 1 B 1 ) (1 + C 1 B 1 )) min λ,σ j min{ E(ˆθ θ ) 2 } + n 1 C 2 B λ,σ j 2 { E(ˆθ θ ) 2 } + n 1 C 2 B. Now take B = (og n) 1/3 and the concusion of the theorem then foows from Theorem 1. References Brown, L. D. and Low, M. G. (1996). Asymptotic equivaence of nonparametric regression and white noise. Ann. Statist. 24,

14 438 YI LIN AND MING YUAN Buhmann, M. D. (1998). Radia functions on compact port. Proc. Edinb. Math. Soc. (2) 41, Buhmann, M. D. (2000). A new cass of radia basis functions with compact port. Math. Comput. 70, Buhmann, M. D. (2003). Radia Basis Functions: Theory and Impementations. Cambridge University Press, Cambridge. Cavaier, L., Goubev, G. K., Picard, D. and Tsybakov, A. B. (2002). Orace inequaities for inverse probems. Ann. Statist. 30, Cristianini, N. and Shawe-Tayor, J. (2000). An Introduction to Support Vector Machines and Other Kerne-based Learning Methods. Cambridge University Press, Cambridge. Evgeniou, T., Ponti, M. and Poggio, T. (2000). Reguarization networks and port vector machines. Adv. Comput. Math. 13, Foater, M. S. and Iske, A. (1996). Mutistep scattered data interpoation using compacty ported radia basis functions. J. Comput. App. Math. 73, Furrer, R., Genton, M. G. and Nychka, D. (2005). Covariance tapering for interpoation of arge spatia datasets. Avaiabe at Gaspari, G. and Cohn, S. E. (1999). Construction of correation functions in two and three dimensions. Quart. J. Roy. Meteor. Soc. 125, Girosi, F., Jones, M. and Poggio, T. (1993). Priors, stabiizers and basis functions: from reguarization to radia, tensor and additive spines. M.I.T. Artificia Inteigence Laboratory Memo 1430, C.B.C.I. Paper 75. Gneiting, T. (2002). Compacty ported correation functions. J. Mutivariate Ana. 83, Goubev, G. and Nussbaum, M. (1998). Asymptotic equivaence of spectra density and regression estimation. Technica report, Weierstrass Institute for Appied Anaysis and Stochastics, Berin. Grama, I. and Nussbaum, M. (1997). Asymptotic equivaence for nonparametric generaized inear modes. Technica report, Weierstrass Institute for Appied Anaysis and Stochastics, Berin. Kneip, A. (1994). Ordered inear smoothers. Ann. Statist. 22, Kimedorf, G. S. and Wahba, G. (1971). Some resuts on Tchebycheffian spine functions. J. Math. Ana. App. 33, Li, K.-C. (1986). Asymptotic optimaity of C L and generaized cross-vaidation in ridge regression with appication to spine smoothing. Ann. Statist. 14, Li, K.-C. (1987). Asymptotic optimaity for C p, C L, cross-vaidation and generaized crossvaidation: discrete index set. Ann. Statist. 15, Lin, Y. and Brown, L. D. (2004). Statistica properties of the method of reguarization with periodic Gaussian reproducing kerne. Ann. Statist. 32, Nussbaum, M. (1996). Asymptotic equivaence of density estimation and Gaussian white noise. Ann. Statist. 24, Schaback, R. (1997). Native Hibert spaces for radia basis functions I. In Proceedings of Chamonix 1996 (Edited by A. Le Méhauté, C. Rabut and L. L. Schumaker), Schaback, R. and Wendand, H. (2001). Characterization and construction of radia basis functions. In Mutivariate Approximation and Appications (Edited by N. Dyn, D. Leviatan, D. Levin and A. Pinkus), Cambridge University Press, Cambridge. Smoa, A. J., Schökopf, B. and Müer, K. (1998). The connection between reguarization operators and port vector kernes. Neura Networks 11, Wahba, G. (1990). Spine Modes for Observationa Data. Society for Industria and Appied Mathematics. Phiadephia, Pennsyvania.

15 COMPACTLY SUPPORTED RADIAL BASIS FUNCTION REGULARIZATION 439 Wahba, G. (1999). Support vector machines, reproducing kerne Hibert spaces and the randomized GACV. In Advances in Kerne Methods - Support Vector Learning (Edited by Schökopf, Burges and Smoa), MIT Press. Wendand, H. (1995). Piecewise poynomia, positive definite and compacty ported radia functions of minima degree. Adv. Comput. Math. 4, Wendand, H. (1998). Error estimates for interpoation by compacty ported radia basis functions of minima degree. J. Approx. Theory 93, Wiiamson, R. C., Smoa, A. J. and Schökopf, B. (2001). Generaization performance of reguarization networks and port vector machines via entropy numbers of compact operators. IEEE Trans. Inform. Theory 47, Wu, Z. (1995) Mutivariate compacty ported positive definite radia functions. Adv. Comput. Math. 4, Zhang, H. H., Genton, M. G. and Liu, P. (2004). Compacty ported radia basis function kernes. Avaiabe at www4.stat.ncsu.edu/ hzhang/research.htm. Department of Statistics, University of Wisconsin-Madison, 1300 University Avenue, Madison, WI E-mai: yiin@stat.wisc.edu Schoo of Industria and Systems Engineering, Georgia Institute of Technoogy, 765 Ferst Drive, Atanta, GA E-mai: myuan@isye.gatech.edu (Received Apri 2005; accepted August 2005)