arxiv: v1 [math.st] 23 Feb 2012

Transcription

1 he Annals of Statistics 20, Vol. 39, No. 5, DOI: 0.24/-AOS898 c Institute of Matheatical Statistics, 20 arxiv: v [ath.s] 23 Feb 202 OPIMAL ESIMAION OF HE MEAN FUNCION BASED ON DISCREELY SAMPLED FUNCIONAL DAA: PHASE RANSIION By. ony Cai and Ming Yuan 2 University of Pennsylvania and Georgia Institute of echnology he proble of estiating the ean of rando functions based on discretely sapled data arises naturally in functional data analysis. In this paper, we study optial estiation of the ean function under both coon and independent designs. Miniax rates of convergence are established and easily ipleentable rate-optial estiators are introduced. he analysis reveals interesting and different phase transition phenoena in the two cases. Under the coon design, the sapling frequency solely deterines the optial rate of convergence when it is relatively sall and the sapling frequency has no effect on the optial rate when it is large. On the other hand, under the independent design, the optial rate of convergence is deterined jointly by the sapling frequency and the nuber of curves when the sapling frequency is relatively sall. When it is large, the sapling frequency has no effect on the optial rate. Another interesting contrast between the two settings is that soothing is necessary under the independent design, while, soewhat surprisingly, it is not essential under the coon design.. Introduction. Estiating the ean function based on discretely sapled noisy observations is one of the ost basic probles in functional data analysis. Much progress has been ade on developing estiation ethodologies. he two onographs by Rasay and Silveran (2002, 2005) provide coprehensive discussions on the ethods and applications. See also Ferraty and Vieu (2006). Let X( ) be a rando function defined on the unit interval =[0,] and X,...,X n beasapleof n independentcopies of X. hegoal is toestiate Received May 200; revised May 20. Supported in part by NSF FRG Grant DMS Supported in part by NSF Career Award DMS Key words and phrases. Functional data, ean function, iniax, rate of convergence, phase transition, reproducing kernel Hilbert space, soothing splines, Sobolev space. his is an electronic reprint of the original article published by the Institute of Matheatical Statistics in he Annals of Statistics, 20, Vol. 39, No. 5, his reprint differs fro the original in pagination and typographic detail.

2 2.. CAI AND M. YUAN theeanfunction g 0 ( ):=E(X( )) basedonnoisy observations frodiscrete locations on these curves: (.) Y ij =X i ( ij )+ε ij, j =,2,..., i and i=,2,...,n, where ij are sapling points, and ε ij are independent rando noise variables with Eε ij =0 and finite second oent Eε 2 ij =σ2 0 <+. he saple path of X is assued to be sooth in that it belongs to the usual Sobolev Hilbert spaces of order r alost surely, such that ( ) (.2) E [X (r) (t)] 2 dt <+. Such probles naturally arise in a variety of applications and are typical in functional data analysis [see, e.g., Rasay and Silveran (2005), Ferraty and Vieu (2006)]. Various ethods have been proposed. However, little is known about their theoretical properties. In the present paper, we study optial estiation of the ean function in two different settings. One is when the observations are sapled at the sae locations across curves, that is, j = 2j = = nj =: j for all j =,...,. We shall refer to this setting as coon design because the sapling locations are coon to all curves. Another setting is when the ij are independently sapled fro, which we shall refer to as independent design. We establish the optial rates of convergence for estiating the ean function in both settings. Our analysis reveals interesting and different phase transition phenoena in the two cases. Another interesting contrast between the two settings is that soothing is necessary under the independent design, while, soewhat surprisingly, it is not essential under the coon design. We reark that under the independent design, the nuber of sapling points oftenties varies fro curve to curve and ay even be rando itself. However, for ease of presentation and better illustration of siilarities and differences between the two types of designs, we shall assue an equal nuber of sapling points on each curve in the discussions given in this section. Earlier studies of nonparaetric estiation of the ean function g 0 fro a collection of discretely sapled curves can be traced back to at least Hart and Wehrly (986) and Rice and Silveran (99) in the case of coon design. In this setting, ignoring the teporal nature of { j : j }, the proble of estiating g 0 can be translated into estiating the ean vector (g 0 ( ),...,g 0 ( )), a typical proble in ultivariate analysis. Such notions are often quickly discarded because they essentially lead to estiating g 0 ( j ) by its saple ean Ȳ j = n (.3) Y ij, n i=

3 MEAN FUNCION ESIMAION 3 based on the standard Gauss Markov theory [see, e.g., Rice and Silveran (99)]. Note that E(Y ij )=g 0 ( ij ) and that the soothness of X iplies that g 0 is also sooth. It is therefore plausible to assue that soothing is essential for optial estiation of g 0. For exaple, a natural approach for estiating g 0 is to regress Y ij on ij nonparaetrically via kernel or spline soothing. Various ethods have been introduced along this vein [see, e.g., Rice and Silveran (99)]. However, not uch is known about their theoretical properties. It is noteworthy that this setting differs fro the usual nonparaetric soothing in that the observations fro the sae curve are highly correlated. Nonparaetric soothing with certain correlated errors has been previously studied by Hall and Hart (990), Wang (996) and Johnstone and Silveran (997), aong others. Interested readers are referred to Opsoer, Wang and Yang (200) for a recent survey of existing results. But neither of these earlier developents can be applied to account for the dependency induced by the functional nature in our setting. o coprehend the effectiveness of soothing in the current context, we establish iniax bounds on the convergence rate of the integrated squared error for estiating g 0. Under the coon design, it is shown that the iniax rate is of the order 2r +n where the two ters can be attributed to discretization and stochastic error, respectively. his rate is fundaentally different fro the usual nonparaetric rate of (n) 2r/(2r+) when observations are obtained at n distinct locations in order to recover an r ties differentiable function [see, e.g., Stone (982)]. he rate obtained here is jointly deterined by the sapling frequency and the nuber of curves n rather than the total nuber of observations n. A distinct feature of the rate is the phase transition which occurs when is of the order n /2r. When the functions are sparsely sapled, that is, =O(n /2r ), the optial rate is of the order 2r, solely deterined by the sapling frequency. On the other hand, when the sapling frequency is high, that is, n /2r, the optial rate reains /n regardless of. Moreover, our developent uncovers a surprising fact that interpolation of {( j,ȳ j):j =,...,}, that is, estiating g 0 ( j ) by Ȳ j, is rate optial. In other words, contrary to the conventional wisdo, soothing does not result in iproved convergence rates. In addition to the coon design, another popular sapling schee is the independent design where the ij are independently sapled fro. A natural approach is to sooth observations fro each curve separately and then average over all soothed estiates. However, the success of this two-step procedure hinges upon the availability of a reasonable estiate for each individual curve. In contrast to the case of coon design, we show that under the independent design, the iniax rate for

4 4.. CAI AND M. YUAN estiating g 0 is (n) 2r/(2r+) +n, which can be attained by soothing {( ij,y ij ): i n, j } altogether. his iplies that in the extree case of =, the optial rate of estiating g 0 is n 2r/(2r+), which also suggests the sub-optiality of the aforeentioned two-step procedure because it is ipossible to sooth a curve with only a single observation. Siilar to the coon design, there is a phase transition phenoenon in the optial rate of convergence with a boundary at =n /2r. When the sapling frequency is sall, that is, =O(n /2r ), the optial rate is of the order (n) 2r/(2r+) which depends jointly on the values of both and n. In the case of high sapling frequency with n /2r, the optial rate is always /n and does not depend on. It is interesting to copare the iniax rates of convergence in the two settings. he phase transition boundary for both designs occurs at the sae value, =n /2r. When is above the boundary, that is, n /2r, there is no difference between the coon and independent designs, and both have the optial rate of n. When is below the boundary, that is, n /2r, the independent design is always superior to the coon design in that it offers a faster rate of convergence. Our results connect with several observations ade earlier in the literature on longitudinal and functional data analysis. Many longitudinal studies follow the independent design, and the nuber of sapling points on each curve is typically sall. In such settings, it is widely recognized that one needs to pool the data to obtain good estiates, and the two-step procedure of averaging the sooth curves ay be suboptial. Our analysis here provides a rigorous justification for such epirical observations by pinpointing to what extent the two-step procedure is suboptial. he phase transition observed here also relates to the earlier work by Hall, Müller and Wang (2006) on estiating eigenfunctions of the covariance kernel when the nuber of sapling points is either fixed or of larger than n /4+δ for soe δ>0. It was shown that the eigenfunctions can be estiated at the rate of n 4/5 in the forer case and /n in the latter. We show here that estiating the ean function has siilar behavior. Furtherore, we characterize the exact nature of such transition behavior as the sapling frequency changes. he rest of the paper is organized as follows. In Section 2 the optial rate of convergence under the coon design is established. We first derive a iniax lower bound and then show that the lower bound is in fact rate sharp. his is accoplished by constructing a rate-optial soothing splines estiator. he iniax upper bound is obtained separately for the coon fixed design and coon rando design. Section 3 considers the independent design and establishes the optial rate of convergence in this case. he rate-optial estiators are easily ipleentable. Nuerical studies are carried out in Section 4 to deonstrate the theoretical results. Section 5 discusses connections and differences of our results with other related work. All proofs are relegated to Section 6.

5 MEAN FUNCION ESIMAION 5 2. Optial rate of convergence under coon design. In this section we consider the coon design where each curve is observed at the sae set of locations { j : j }. We first derive a iniax lower bound and then show that this lower bound is sharp by constructing a soothing splines estiator that attains the sae rate of convergence as the lower bound. 2.. Miniax lower bound. Let P(r;M 0 ) be the collection of probability easures for a rando function X such that its saple path is r ties differentiable alost surely and (2.) E [X (r) (t)] 2 dt M 0 for soe constant M 0 > 0. Our first ain result establishes the iniax lower bound for estiating the ean function over P(r;M 0 ) under the coon design. heore 2.. Suppose the sapling locations are coon in odel (.). hen there exists a constant d > 0 depending only on M 0 and the variance σ 2 0 of easureent error ε ij such that for any estiate g based on observations {( j,y ij ): i n, j }, (2.2) li sup n sup L(X) P(r;M 0 ) P( g g 0 2 L 2 >d( 2r +n ))>0. he lower bound established in heore 2. holds true for both coon fixed design where j s are deterinistic, and coon rando design where j s are also rando. he ter 2r in the lower bound is due to the deterinistic approxiation error, and the ter n is attributed to the stochastic error. It is clear that neither can be further iproved. o see this, first consider the situation where there is no stochastic variation and the ean function g 0 is observed exactly at the points j, j =,...,. It is well known [see, e.g., DeVore and Lorentz (993)] that due to discretization, it is not possible to recover g 0 at a rate faster than 2r for all g 0 such that [g (r) 0 ]2 M 0. On the other hand, the second ter n is inevitable since the ean function g 0 cannot be estiated at a faster rate even if the whole rando functions X,...,X n are observed copletely. We shall show later in this section that the rate given in the lower bound is optial in that it is attainable by a soothing splines estiator. It is interesting to notice the phase transition phenoenon in the iniax bound. When the sapling frequency is large, it has no effect on the rate of convergence, and g 0 can be estiated at the rate of /n, the best possible rate when the whole functions were observed. More surprisingly, such saturation occurs when is rather sall, that is, of the order n /2r. On the

6 6.. CAI AND M. YUAN other hand, when the functions are sparsely sapled, that is, =O(n /2r ), the rate is deterined only by the sapling frequency. Moreover, the rate 2r is in fact also the optial interpolation rate. In other words, when the functions are sparsely sapled, the ean function g 0 can be estiated as well as if it is observed directly without noise. he rate is to be contrasted with the usual nonparaetric regression with n observations at arbitrary locations. In such a setting, it is well known [see, e.g., sybakov (2009)] that the optial rate for estiating g 0 is (n) 2r/(2r+), and typically stochastic error and approxiation error are of the sae order to balance the bias-variance trade-off Miniax upper bound: Soothing splines estiate. We now consider the upper bound for the iniax risk and construct specific rate optial estiators under the coon design. hese upper bounds show that the rate of convergence given in the lower bound established in heore 2. is sharp. More specifically, it is shown that a soothing splines estiator attains the optial rate of convergence over the paraeter space P(r;M 0 ). We shall consider a soothing splines type of estiate suggested by Rice and Silveran (99). Observe that f [f (r) ] 2 is a squared sei-nor and therefore convex. By Jensen s inequality, (2.3) [g (r) 0 (t)]2 dt E [X (r) (t)] 2 dt<, which iplies that g 0 belongs to the rth order Sobolev Hilbert space, W r 2 ([0,])={g:[0,] R g,g(),...,g (r ) are absolutely continuous and g (r) L 2 ([0,])}. aking this into account, the following soothing splines estiate can be eployed to estiate g 0 : { n } (2.4) ĝ λ =argin (Y ij g( j )) 2 +λ [g (r) (t)] 2 dt, g W2 r n i= where λ>0 is a tuning paraeter that balances the fidelity to the data and the soothness of the estiate. Siilarly to the soothing splines for the usual nonparaetric regression, ĝ λ can be conveniently coputed, although the iniization is taken over an infinitely-diensional functional space. First observe that ĝ λ can be equivalently rewritten as (2.5) ĝ λ =argin g W r 2 { (Ȳ j g( j )) 2 +λ [g (r) (t)] 2 dt }.

7 MEAN FUNCION ESIMAION 7 Appealing to the so-called representer theore [see, e.g., Wahba (990)], the solution of the iniization proble can be expressed as (2.6) r ĝ λ (t)= d k t k + c i K(t, j ) k=0 for soe coefficients d 0,...,d r,c,...,c, where (2.7) K(s,t)= (r!) 2B r(s)b r (t) (2r)! B 2r( s t ), where B ( ) is the th Bernoulli polynoial. Plugging (2.6) back into (2.5), the coefficients and subsequently ĝ λ can be solved in a straightforward way. his observation akes the soothing splines procedure easily ipleentable. he readers are referred to Wahba (990) for further details. Despite the siilarity between ĝ λ and the soothing splines estiate in the usual nonparaetric regression, they have very different asyptotic properties. It is shown in the following that ĝ λ achieves the lower bound established in heore 2.. he analyses for the coon fixed design and the coon rando design are siilar, and we shall focus on the fixed design where the coon sapling locations,..., are deterinistic. In this case, we assue without loss of generality that 2. he following theore shows that the lower bound established in heore 2. is attained by the soothing splines estiate ĝ λ. (2.8) heore 2.2. Consider the coon fixed design and assue that ax j+ j C 0 0 j for soe constant C 0 >0 where we follow the convention that 0 =0 and + =. hen (2.9) li lisup D n sup L(X) P(r;M 0 ) for any λ=o( 2r +n ). P( ĝ λ g 0 2 L 2 >D( 2r +n ))=0 ogether with heore 2., heore 2.2 shows that ĝ λ is iniax rate optialifthetuningparaeterλissettobeoftheordero( 2r +n ).We note the necessity of the condition given by (2.8). It is clearly satisfied when the design is equidistant, that is, j =2j/(2+). he condition ensures that the rando functions are observed on a sufficiently regular grid. It is of conceptual iportance to copare the rate of ĝ λ with those generally achieved in the usual nonparaetric regression setting. Defined by (2.4),

8 8.. CAI AND M. YUAN ĝ λ essentially regresses Y ij on j. Siilarly to the usual nonparaetric regression, the validity of the estiate is driven by E(Y ij j ) = g 0 ( j ). he difference, however, is that Y i,...,y i are highly correlated because they are observed fro the sae rando function X i ( ). When all the Y ij s are independently sapled at j s, it can be derived that the optial rate for estiating g 0 is 2r +(n) 2r/(2r+). As we show here, the dependency induced by the functional nature of our proble leads to the different rate 2r +n. A distinct feature of the behavior of ĝ λ is in the choice of the tuning paraeter λ. uning paraeter selection plays a paraount role in the usual nonparaetric regression, as it balances the the tradeoff between bias and variance. Optial choice of λ is of the order (n) 2r/(2r+) in the usual nonparaetric regression. In contrast, in our setting, ore flexibility is allowed in the choice of the tuning paraeter in that ĝ λ is rate optial so long as λ is sufficiently sall. In particular, taking λ 0 +, ĝ λ reduces to the splines interpolation, that is, the solution to (2.0) in g W r 2 [g (r) (t)] 2 subject to g( j )=Ȳ j, j =,...,. his aounts to, in particular, estiating g 0 ( j ) by Ȳ j. In other words, there is no benefit fro soothing in ters of the convergence rate. However, as we will see in Section 4, soothing can lead to iproved finite saple perforance. Reark. More general stateents can also be ade without the condition on the spacing of sapling points. More specifically, denote by R(,..., )=ax j+ j j the discretization resolution. Using the sae arguent, one can show that the optial convergence rate in the iniax sense is R 2r +n and ĝ λ is rate optial so long as λ=o(r 2r +n ). Reark. Although we have focused here on the case when the sapling points are deterinistic, a siilar stateent can also be ade for the setting where the sapling points are rando. In particular, assuing that j are independent and identically distributed with a density function η such that inf t η(t) c 0 >0andg 0 W r,itcanbeshownthatthesoothingsplines estiator ĝ λ satisfies (2.) li lisup D n sup L(X) P(r;M 0 ) P( ĝ λ g 0 2 L 2 >D( 2r +n ))=0 for any λ=o( 2r +n ). In other words, ĝ λ reains rate optial.

9 MEAN FUNCION ESIMAION 9 3. Optial rate of convergence under independent design. In any applications, the rando functions X i are not observed at coon locations. Instead, each curve is discretely observed at a different set of points [see, e.g., Jaes and Hastie (200), Rice and Wu (200), Diggle et al. (2002), Yao, Müller and Wang (2005)]. In these settings, it is ore appropriate to odelthesaplingpoints ij asindependentlysapledfroacoondistribution. In this section we shall consider optial estiation of the ean function under the independent design. Interestingly, the behavior of the estiation proble is drastically different between the coon design and the independent design. o keep our treatent general, we allow the nuber of sapling points to vary. Let be the haronic ean of,..., n, that is, ( ) n :=. n i= i Denote by M() the collection of sapling frequencies (,..., n ) whose haronic ean is. In parallel to heore 2., we have the following iniax lower bound for estiating g 0 under the independent design. heore 3.. Suppose ij are independent and identically distributed with a density function η such that inf t η(t) c 0 >0. hen there exists a constant d>0 depending only on M 0 and σ0 2 such that for any estiate g based on observations {( ij,y ij ): i n, j }, (3.) li sup n sup L(X) P(r;M 0 ) (,..., n) M() P( g g 0 2 L 2 >d((n) 2r/(2r+) +n ))>0. he iniax lower bound given in heore 3. can also be achieved using the soothing splines type of estiate. o account for the different sapling frequency for different curves, we consider the following estiate of g 0 : (3.2) ĝ λ =argin g W r 2 { n n i i= i (Y ij g( ij )) 2 +λ [g (r) (t)] 2 dt heore 3.2. Under the conditions of heore 3., if λ (n) 2r/(2r+), then the soothing splines estiator ĝ λ satisfies (3.3) li lisup D n =0. sup L(X) P(r;M 0 ) (,..., n) M() In other words, ĝ λ is rate optial. P( ĝ g 0 2 L 2 >D((n) 2r/(2r+) +n )) }.

10 0.. CAI AND M. YUAN heores 3. and 3.2 deonstrate both siilarities and significant differences between the two types of designs in ters of the convergence rate. For either the coon design or the independent design, the sapling frequency only plays a role in deterining the convergence rate when the functions are sparsely sapled, that is, =O(n /2r ). But how the sapling frequency affects the convergence rate when each curve is sparsely sapled differs between the two designs. For the independent design, the total nuber of observations n, whereas for the coon design alone, deterines the iniax rate. It is also noteworthy that when =O(n /2r ), the optial rate under the independent design, (n) 2r/(2r+), is the sae as if all the observations are independently observed. In other words, the dependency aong Y i,...,y i does not affect the convergence rate in this case. Reark. We ephasize that heores 3. and 3.2 apply to both deterinistic and rando sapling frequencies. In particular for rando sapling frequencies, together with the law of large nubers, the sae iniax bound holds when we replace the haronic ean by ( n E(/ i )), n i= when assuing that i s are independent. 3.. Coparison with two-stage estiate. A popular strategy to handle discretely sapled functional data in practice is a two-stage procedure. In the first step, nonparaetric regression is run for data fro each curve to obtain estiate X i of X i, i=,2,...,n. For exaple, they can be obtained by soothing splines (3.4) X i,λ =argin f W r 2 { (Y ij g( ij )) 2 +λ [f (r) (t)] 2 dt Any subsequent inference can be carried out using the X i,λ as if they were the original true rando functions. In particular, the ean function g 0 can be estiated by the siple average (3.5) g λ = n n i= X i,λ. Although forulated differently, it is worth pointing out that this procedure is equivalent to the soothing splines estiate ĝ λ underthe coon design. }.

11 MEAN FUNCION ESIMAION Proposition 3.3. Under the coon design, that is, ij = j for i n and j. he estiate g λ fro the two-stage procedure is equivalent to the soothing splines estiate ĝ λ : ĝ λ = g λ. In light of heores 2. and 2.2, the two-step procedure is also rate optial under the coon design. But it is of great practical iportance to note that in order to achieve the optiality, it is critical that in the first step we undersooth each curve by using a sufficiently sall tuning paraeter. Under independent design, however, the equivalence no long holds. he success of the two-step estiate g λ depends upon getting a good estiate of each curve, which is not possible when is very sall. In the extree case of =, the procedure is no longer applicable, but heore 3.2 indicates that soothing splines estiate ĝ λ can still achieve the optial convergence rate of n 2r/(2r+). he readers are also referred to Hall, Müller and Wang (2006) for discussions on the pros and cons of siilar two-step procedures in the context of estiating the functional principal coponents. 4. Nuerical experients. he soothing splines estiators are easy to ipleent. o deonstrate the practical iplications of our theoretical results, we carried out a set of siulation studies. he true ean function g 0 is fixed as (4.) 50 g 0 = 4( ) k+ k 2 φ k, k= where φ (t)= and φ k+ (t)= 2cos(kπt) for k. he rando function X was generated as (4.2) X =g k= ζ k Z k φ k, where Z k are independently sapled fro the unifor distribution on [ 3, 3], and ζ k are deterinistic. It is not hard to see that ζk 2 are the eigenvalues of the covariance function of X and therefore deterine the soothness of a saple curve. In particular, we take ζ k =( ) k+ k./2. It is clear that the saple path of X belongs to the second order Sobolev space (r=2). We begin with a set of siulations designed to deonstrate the effect of interpolation and soothing under coon design. A data set of fifty curves were first siulated according to the aforeentioned schee. For each curve, ten noisy observations were taken at equidistant locations on each curve following odel (.) with σ0 2 =0.52. he observations, together with g 0 (grey line), are given in the right panel of Figure. Soothing splines estiate ĝ λ

12 2.. CAI AND M. YUAN is also coputed with a variety of values for λ. he integrated squared error, ĝ λ g 0 L2,as a functionof thetuningparaeter λ is given intheleft panel. For λ saller than 0., the soothing splines estiate essentially reduces to the spline interpolation. o contrast the effect of interpolation and soothing, the right panel also includes the interpolation estiate (solid black line) and ĝ λ (red dashed line) with the tuning paraeter chosen to iniize the integrated squared error. We observe fro the figure that soothing does lead to slightly iproved finite saple perforance although it does not affect the convergence rate as shown in Section 2. he next nuerical experient intends to deonstrate the effect of saple size n, sapling frequency as well as design. o this end, we siulated n curves, and fro each curve, discrete observations were taken following odel (.) with σ 2 0 =0.52. he sapling locations are either fixed at j =(2j)/(2 +), j =,...,, for coon design or randoly sapled fro the unifor distribution on [0,]. he soothing splines estiate ĝ λ for each siulated data set, and the tuning paraeter is set to yield the sallest integrated squared error and therefore reflect the best perforance Fig.. Effect of soothing under coon design: for a typical data set with fifty curves, ten observations were taken on each curve. he observations and g 0 (solid grey line) are given in the right panel together with the spline interpolation estiate (solid black line) and soothing splines estiate (red dashed line) with the tuning paraeter chosen to yield the sallest integrated squared error. he left panel gives the integrated squared error of the soothing splines estiate as a function of the tuning paraeter. It is noted that the soothing splines estiate essentially reduces to the spline interpolation for λ saller than 0..

13 MEAN FUNCION ESIMAION 3 of the estiating procedure for each data set. We repeat the experient with varying cobinations of n=25,50 or 200, =,5,0 or 50. For the coon design, we restrict to =0 or 50 to give ore eaningful coparison. he true function g 0 as well as its estiates obtained in each of the settings are given in Figure 2. Figure 2 agrees pretty well with our theoretical results. For instance, increasing either or n leads to iproved estiates, whereas such iproveent is ore visible for sall values of. Moreover, for the sae value of and n, independent designs tend to yield better estiates. o further contrast the two types of designs and the effect of sapling frequency on estiating g 0, we now fix the nuber of curves at n=00. For the coon design, we consider =0,20,50 or 00. For the independent design, we let =,5,0,20,50 or 00. For each cobination of (n,), two hundred data sets were siulated following the sae echanis as before. Figure 3 gives the estiation error averaged over the one hundred data sets for each cobination of (n, ). It clearly shows that independent design is preferable over coon design when is sall; and the two types of designs are siilar when is large. Both phenoena are in agreeent with our theoretical results developed in the earlier sections. Fig. 2. Effect of, n and type of design on estiating g 0: soothing splines estiates obtained under various cobinations are plotted together with g 0.

14 4.. CAI AND M. YUAN Fig. 3. Effect of design type and sapling frequency on estiating g 0: the black solid line and circles correspond to coon design whereas the red dashed lines and circles correspond to independent design. he error bars correspond to the average ± one standard errors based on two hundred repetitions. Note that both axes are in log scale to yield better coparison. 5. Discussions. We have established the optial rates of convergence for estiating the ean function under both the coon design and independent design. he results reveal several significant differences in the behavior of the iniax estiation proble between the two designs. hese revelations have iportant theoretical and practical iplications. In particular, for sparsely sapled functions, the independent design leads to a faster rate of convergence when copared to the coon design and thus should be preferred in practice. he optial rates of convergence for estiating the ean function based on discretely sapled rando functions behave in a fundaentally different way fro the iniax rate of convergence in the conventional nonparaetric regression probles. he optial rates in the ean function estiation are jointly deterined by the sapling frequency and the nuber of curves n rather than the total nuber of observations n. he observation that one can estiate the ean function as well as if the thewholecurvesareavailable when n /2r bearssoesiilarity to soe recent findings on estiating the covariance kernel and its eigenfunction under independent design. Assuing that X is twice differentiable (i.e., r = 2), Hall, Müller and Wang (2006) showed that when n /4+δ for soe δ>0, the covariance kernel and its eigenfunctions can be estiated at

15 MEAN FUNCION ESIMAION 5 the rate of /n when using a two-step procedure. More recently, the cutoff point is further iproved to n /2r logn with general r by Cai and Yuan (200) using an alternative ethod. Intuitively one ay expect estiating the covariance kernel to be ore difficult than estiating the ean function, which suggests that these results ay not be iproved uch further for estiating the covariance kernel or its eigenfunctions. We have also shown that the particular soothing splines type of estiate discussed earlier by Rice and Silveran (99) attains the optial convergence rates under both designs with appropriate tuning. he soothing splines estiator is well suited for nonparaetric estiation over Sobolev spaces. We note, however, other nonparaetric techniques such as kernel or local polynoial estiators can also be used. We expect that kernel soothing or other ethods with proper choice of the tuning paraeters can also achieve the optial rate of convergence. Further study in this direction is beyond the scope of the current paper, and we leave it for future research. Finally, we ephasize that although we have focused on the univariate Sobolev space for siplicity, the phenoena observed and techniques developed apply to ore general functional spaces. Consider, for exaple, the ultivariate setting where =[0,] d. Following the sae arguents, it can be shown that the iniax rate for estiating an r-ties differentiable function is 2r/d +n under the coon design and (n) 2r/(2r+d) +n under the independent design. he phase transition phenoena thus reain in the ultidiensional setting under both designs with a transition boundary of =n d/2r. 6. Proofs. Proof of heore 2.. Let D be the collection all easurable functions of {( ij,y ij ): i n, j }. First note that it is straightforward to show that li sup n inf sup g D L(X) P(r;M 0 ) P( g g 0 2 L 2 >dn )>0 by considering X as an unknown constant function where the proble essentially becoes estiating the ean fro n i.i.d. observations, and /n is known as the optial rate. It now suffices to show that li sup n inf sup g D L(X) P(r;M 0 ) P( g g 0 2 L 2 >d 2r )>0. Let ϕ,...,ϕ 2 be 2 functions fro W2 r with distinct support, that is, ) ( tk ϕ k ( )=h r K, k=,...,2, h

16 6.. CAI AND M. YUAN where h = /(2), t k = (k )/2 + /4, and K:R [0, ) is an r ties differentiable function with support [ /2, /2]. See sybakov (2009) for explicit construction of such functions. For each b=(b,...,b 2 ) {0,} 2, define It is clear that g b ( )= 2 k= b k ϕ k ( ). g b g b 2 L in 2 H(b,b ) H(b,b = ϕ k 2 L ) 2 =(2) (2r+) K 2 L 2. he clai then follows fro an application of Assouad s lea [Assouad (983)]. Proof of heore 2.2. It is well known [see, e.g., Green and Silveran (994)] that ĝ λ can be characterized as the solution to the following: [g (r) (t)] 2 dt subject to g( j )=ĝ λ ( j ), j =,...,. Write in g W r 2 δ j =ĝ λ ( ij ) g 0 ( ij ), and let h be the linear interpolation of {( j,δ j ): j }, that is, δ, 0 t, j+ t t j (6.) h(t)= δ j +δ j+, j t j+, j+ j j+ j δ, t. hen ĝ λ =Q (g 0 +h) where Q be the operator associated with the rth order spline interpolation, that is, Q (f) is the solution to [g (r) (t)] 2 dt subject to g( j )=f( j ), j =,...,. in g W r 2 Recall that Q is a linear operator in that Q (f +f 2 )=Q (f )+Q (f 2 ) [see, e.g., DeVore and Lorentz (993)]. herefore, ĝ λ =Q (g 0 )+Q (h). By the triangular inequality, (6.2) ĝ g 0 L2 Q (g 0 ) g 0 L2 + Q (h) L2. he first ter on the right-hand side represents the approxiation error of spline interpolation for g 0, and it is well known that it can be bounded by

17 [see, e.g., DeVore and Lorentz (993)] (6.3) MEAN FUNCION ESIMAION 7 Q (g 0 ) g 0 2 L 2 ( c 0 ax j+ j 2r) [g (r) 0 j 0 (t)]2 dt c 0 M 0 2r. Hereafter, we shall use c 0 >0 as a generic constant which ay take different values at different appearance. It now reains to bound Q (h) L2. We appeal to the relationship between spline interpolation and the best local polynoial approxiation. Let I j =[ j r+, j+r ] with the convention that j =0 for j < and j = for j >. Denote by P j the best approxiation error that can be achieved on I j by a polynoial of order less that r, that is, [ r 2 (6.4) P j (f)= in a k t f(t)] k dt. a k :k<r I j k=0 It can be shown [see, e.g., heore 4.5 on page 47 of DeVore and Lorentz (993)] that hen ( f Q (f) 2 L 2 c 0 P j (f) ). 2 ( ) /2. Q (h) L2 h L2 +c 0 P j (f) 2 ogether with the fact that P j (h) 2 h(t) 2 dt, I j we have Q (h) 2 L 2 c 0 h 2 L 2 c 0 δj( 2 j+ j ) c 0 =c 0 [ĝ λ ( j ) g 0 ( j )] 2 δ 2 j

18 8.. CAI AND M. YUAN c 0 ([Ȳ j ĝ λ ( j )] 2 +[Ȳ j g 0 ( j )] 2 ). Observe that ( E [Ȳ j g 0 ( j )] )=c 2 0 σ0n 2. It suffices to show that [Ȳ j ĝ λ ( j )] 2 =O p ( 2r +n ). o this end, note that by the definition of ĝ λ, (Ȳ j ĝ λ ( j )) 2 (Ȳ j ĝ λ ( j )) 2 +λ (Ȳ j g 0 ( j )) 2 +λ O p ( 2r +n ), because λ=o( 2r +n ). he proof is now coplete. [ĝ (r) λ (t)]2 dt [g (r) 0 (t)]2 dt Proof of heore 3.. Note that any lower bound for a specific case yields iediately a lower bound for the general case. It therefore suffices to consider the case when X is a Gaussian process and = 2 = = n =:. Denote by N =c(n) /(2r+) where c>0 is a constant to be specified later. Let b=(b,...,b N ) {0,} N be a binary sequence, and write g b ( )=M /2 0 π r 2N k=n+ N /2 k r b k N ϕ k ( ), where ϕ k (t)= 2cos(πkt). It is not hard to see that [g (r) b (t)] 2 dt=m 0 π 2r 2N k N+ =M 0 N 2N k=n+ (πk) 2r (N /2 k r b k N ) 2 b k N M 0.

19 Furtherore, MEAN FUNCION ESIMAION 9 g b g b 2 L 2 =M 0 π 2r N 2N k=n+ M 0 π 2r (2N) (2r+) 2N =c 0 N (2r+) H(b,b ) k 2r (b k N b k N )2 k=n+ (b k b k )2 for soe constant c 0 >0. By the Varshaov Gilbert bound [see, e.g., sybakov (2009)], there exists a collection of binary sequences {b (),...,b (M) } {0,} N such that M 2 N/8, and hen H(b (j),b (k) ) N/8 j <k M. g b (j) g b (k) L2 c 0 N r. Assue that X is a Gaussian process with ean g b, follows a unifor distribution on and the easureent error ε N(0,σ0 2 ). Conditional on { ij :j =,...,}, Z i =(Z i,...,z i ) follows a ultivariate noral distribution with ean µ b =(g b ( i ),...,g b ( i )) and covariance atrix Σ()=(C 0 ( ij, ik )) j,k +σ0 2 I. herefore,thekullback Leibler distance fro probability easure Π to Π gb (j) g can be bounded by b (k) KL(Π gb (j) Π g b (k) )=ne [(µ gb (j) µ g b (k) ) Σ ()(µ gb (j) µ g b (k) )] nσ 2 0 E µ gb (j) µ g b (k) 2 =nσ 2 0 g b (j) g b (k) 2 L 2 c nσ 2 0 N 2r. An application of Fano s lea now yields ax E g g g j M b (j) b (j) L 2 c 0 N ( r log(c nσ0 2 ) N 2r )+log2 logm (n) r/(2r+) with an appropriate choice of c, for any estiate g. his in turn iplies that li sup n inf sup g D L(X) P(r;M 0 ) P( g g 0 2 L 2 >d(n) 2r/(2r+) )>0. he proof can then be copleted by considering X as an unknown constant function.

20 20.. CAI AND M. YUAN Proof of heore 3.2. For brevity, in what follows, we treat the sapling frequencies,..., n as deterinistic. All the arguents, however, also apply to the situation when they are rando by treating all the expectations and probabilities as conditional on,..., n. Siilarly, we shall also assue that j s follow unifor distribution. he arguent can be easily applied to handle ore general distributions. It is well known that W2 r, endowed with the nor (6.5) f 2 W2 r = f 2 + (f (r) ) 2, fors a reproducing kernel Hilbert space [Aronszajn (950)]. Let H 0 be the collection of all polynoials of order less than r and H be its orthogonal copleent in W2 r. Let {φ k: k r} be a set of orthonoral basis functions of H 0, and {φ k :k >r} an orthonoral basis of H such that any f W2 r adits the representation f = ν f ν φ ν. Furtherore, f 2 L 2 = ν f 2 ν and f 2 W2 r = (+ρ ν )fν, 2 ν where ρ = =ρ r =+ and ρ ν ν 2r. Recall that { ĝ=argin l n (g)+λ [g (r) ] }, 2 g H(K) where l n (g)= n n i i= i (Y ij g( ij )) 2. For brevity, we shall abbreviate the subscript of ĝ hereafter when no confusion occurs. Write ( ) n i l (g)=e [Y ij g( ij )] 2 n i= i =E([Y g 0 ( )] 2 )+ [g(s) g 0 (s)] 2 ds. Let { ḡ=argin l (g)+λ g H(K) [g (r) ] 2 }.

21 MEAN FUNCION ESIMAION 2 Denote l n,λ (g)=l n (g)+λ [g (r) ] 2 ; l,λ (g)=l (g)+λ [g (r) ] 2. Let g=ḡ 2 G λ Dl n,λ(ḡ), where G λ =(/2)D 2 l,λ (ḡ) and D stands for the Fréchet derivative. It is clear that ĝ g 0 =(ḡ g 0 )+(ĝ g)+( g ḡ). We proceed by bounding the three ters on the right-hand side separately. In particular, it can be shown that (6.6) ḡ g 0 2 L 2 c 0 λ and (6.7) Furtherore, if then (6.8) herefore, aking yields [g (r) 0 ]2 g ḡ 2 L 2 =O p (n +(n) λ /(2r) ). nλ /(2r), ĝ g 2 L 2 =o p (n +(n) λ /(2r) ). ĝ g 0 2 L 2 =O p (λ+n +(n) λ /(2r) ). λ (n) 2r/(2r+) ĝ g 0 2 L 2 =O p (n +(n) 2r/(2r+) ). We now set to establish bounds (6.6) (6.8). For brevity, we shall assue in what follows that all expectations are taken conditionally on,..., n unless otherwise indicated. Define (6.9) where 0 α. g 2 α = (+ρ ν )α gν 2, ν

22 22.. CAI AND M. YUAN We begin with ḡ g 0. Write (6.0) g 0 ( )= k a k φ k ( ), g( )= k b k φ k ( ). hen (6.) l (g)=e([y g 0 ( )] 2 )+ k (b k a k ) 2. It is not hard to see (6.2) Hence, bk := ḡ,φ k L2 =argin{(b k a k ) 2 +λρ k b2 k }= ḡ g 0 2 α = (+ρ k )α ( b k a k ) 2 k = ( λρ (+ρ ) 2 k )α k +λρ a 2 k k k c 0 λ 2 ρ (+α) k sup k (+λρ k )2 c 0 λ α [g (r) 0 ]2. k= ρ k a2 k a k +λρ k Next, we consider g ḡ. Notice that Dl n,λ (ḡ)=dl n,λ (ḡ) Dl,λ (ḡ)= Dl n (ḡ) Dl (ḡ). herefore E[Dl n,λ (ḡ)f] 2 =E[Dl n (ḡ)f Dl (ḡ)f] 2 [ = 4 n i ] n 2 Var ([Y ij ḡ( ij )]f( ij )). Note that [ i ] Var ([Y ij ḡ( ij )]f( ij )) =Var =Var [ E [ i ( i 2 i= i )] [ [Y ij ḡ( ij )]f( ij ) +E Var ] [ ([g 0 ( ij ) ḡ( ij )]f( ij )) +E Var )] Y ij f( ij ) ( i ( i )] Y ij f( ij ) i,..

23 MEAN FUNCION ESIMAION 23 he first ter on the rightost-hand side can be bounded by [ i ] Var ([g 0 ( ij ) ḡ( ij )]f( ij )) = i Var([g 0 ( i ) ḡ( i )]f( i )) i E([g 0 ( i ) ḡ( i )]f( i )) 2 = i ([g 0 (t) ḡ(t)]f(t)) 2 dt i [g 0 (t) ḡ(t)] 2 dt f 2 (t)dt, where the last inequality follows fro the Cauchy Schwarz inequality. ogether with (6.6), we get [ i ] (6.3) Var ([g 0 ( ij ) ḡ( ij )]f( ij )) c 0 i f 2 L 2 λ. We now set out to copute the the second ter. First observe that ( i ) Var Y ij f( ij ) = i j,k= f( ij )f( ik )(C 0 ( ij, ik )+σ 2 0δ jk ), where δ jk is Kronecker s delta. herefore, [ ( ) ] i E Var Y j f( j ) = i ( i ) + i σ 2 0 f 2 L 2 + i Suing up, we have ( n (6.4) E[Dl n,λ (ḡ)φ k ] 2 c 0 n 2 where (6.5) f(s)c 0 (s,t)f(t)dsdt i= i f 2 (s)c(s,s)ds. ) + 4c k n, c k = φ k (s)c 0 (s,t)φ k (t)dsdt.

24 24.. CAI AND M. YUAN herefore, E g ḡ 2 α =E 2 G λ Dl n,λ(ḡ) 2 α = [ ] 4 E (+ρ k )α (+λρ k ) 2 (Dl n,λ (ḡ)φ k ) 2 k ( n ) c 0 n 2 (+ρ k )α (+λρ k ) 2 i= i k + (+ρ n k Observe that (+ρ and (+ρ hus, k k k )α (+λρ k )α (+λρ k )α (+λρ k ) 2 c k. k ) 2 c 0 λ α /(2r) k ) 2 c k (+ρ k )c k =E X 2 W r <. 2 k [ ( n ) ] E g ḡ 2 α c 0 n 2 λ α /(2r) +. n i= i It reains to bound ĝ g. It can be easily verified that (6.6) hen ĝ g= 2 G λ [D2 l (ḡ)(ĝ ḡ) D 2 l n (ḡ)(ĝ ḡ)]. ĝ g 2 α = k (+ρ [ n k )α (+λρ k ) 2 n Clearly, (ĝ ḡ)φ k H(K). Write i i= i (ĝ( ij ) ḡ( ij ))φ k ( ij ) 2 (ĝ(s) ḡ(s))φ k (s)ds]. (6.7) (ĝ ḡ)φ k = j h j φ j.

25 MEAN FUNCION ESIMAION 25 hen, by the Cauchy Schwarz inequality, [ n i (ĝ( ij ) ḡ( ij ))φ k ( ij ) n i= i = [ k h k ( n n i i= i [ ] (+ρ k ) γ h 2 k k [ (+ρ k k ) γ ĝ ḡ 2 γ (+ρ [ (+ρ k k )γ k ) γ ( n ( n (ĝ(s) ḡ(s))φ k (s)ds )] 2 φ k ( ij ) φ k (s)ds n i i= i n i i= i ] 2 ) 2 ] φ k ( ij ) φ k (s)ds ) 2 ] φ k ( ij ) φ k (s)ds for any 0 γ, where in the last inequality, we used the fact that (6.8) (ĝ ḡ)φ k γ ĝ ḡ γ φ k γ = ĝ ḡ γ (+ρ k )γ/2. Following a siilar calculation as before, it can be shown that [ ( E (+ρ k ) γ n i ) 2 ] φ k ( ij ) φ k (s)ds n k i= i ( n ) n 2 (+ρ k ) γ, i= i k which is finite whenever γ >/2r. Recall that is the haronic ean of,..., n. herefore, ( ) (6.9) ĝ g 2 α O p nλ α+γ+/(2r) ĝ ḡ 2 γ. If α>/2r, then taking γ =α yields ( ) ĝ g 2 (6.20) α =O p ĝ ḡ 2 α =o p ( ĝ ḡ α ), assuing that (6.2) nλ 2α+/(2r) nλ 2α+/(2r).

26 26.. CAI AND M. YUAN ogether with the triangular inequality (6.22) g ḡ α ĝ ḡ α ĝ g α =( o p ()) ĝ ḡ α. herefore, (6.23) ĝ ḡ 2 α =O p ( g ḡ 2 α)=o p (n +(n) λ α /(2r) ). ogether with (6.9), ( ) ĝ g 2 L 2 =O p nλ α+/(2r)(n +(n) λ α /(2r) ) =o p (n λ α +(n) λ /(2r) ). We conclude by noting that in the case when,..., n are rando, can also be replaced with the expectation of the haronic ean thanks to the law of large nubers. Proof of Proposition 3.3. Let Q,λ be the soothing spline operator, that is, Q,λ (f,...,f ) is the solution to It is clear that and in g W r 2 { (f j g( j )) 2 +λ ĝ λ =Q,λ (Ȳ,Ȳ 2,...,Ȳ ) X i,λ =Q,λ (Y i,y i2,...,y i ). [g (r) (t)] 2 dt Because Q,λ is a linear operator [see, e.g., Wahba (990)], we have g λ = n i= X i,λ = n } n Q,λ (Y i,y i2,...,y i ) i= =Q,λ (Ȳ,Ȳ 2,...,Ȳ )=ĝ λ. Acknowledgents. We thank an Associate Editor and two referees for their constructive coents which have helped to iprove the presentation of the paper..

27 MEAN FUNCION ESIMAION 27 REFERENCES Aronszajn, N. (950). heory of reproducing kernels. rans. Aer. Math. Soc MR Assouad, P. (983). Deux rearques sur l estiation. C. R. Acad. Sci. Paris Sér. I Math MR Cai,. and Yuan, M. (200). Nonparaetric covariance function estiation for functional and longitudinal data. echnical report, Georgia Institute of echnology, Atlanta, GA. DeVore, R. A. and Lorentz, G. G. (993). Constructive Approxiation. Grundlehren der Matheatischen Wissenschaften [Fundaental Principles of Matheatical Sciences] 303. Springer, Berlin. MR26635 Diggle, P., Heagerty, P., Liang, K. and Zeger, S. (2002). Analysis of Longitudinal Data, 2nd ed. Oxford Univ. Press, Oxford. MR Ferraty, F. and Vieu, P. (2006). Nonparaetric Functional Data Analysis: heory and Practice. Springer, New York. MR Green, P. J. and Silveran, B. W. (994). Nonparaetric Regression and Generalized Linear Models: A Roughness Penalty Approach. Monographs on Statistics and Applied Probability 58. Chapan and Hall, London. MR27002 Hall, P. and Hart, J. D. (990). Nonparaetric regression with long-range dependence. Stochastic Process. Appl MR Hall, P., Müller, H.-G. and Wang, J.-L. (2006). Properties of principal coponent ethods for functional and longitudinal data analysis. Ann. Statist MR Hart, J. D. and Wehrly,. E. (986). Kernel regression estiation using repeated easureents data. J. Aer. Statist. Assoc MR Jaes, G. M. and Hastie,. J. (200). Functional linear discriinant analysis for irregularly sapled curves. J. R. Stat. Soc. Ser. B Stat. Methodol MR85840 Johnstone, I. M. and Silveran, B. W. (997). Wavelet threshold estiators for data with correlated noise. J. Roy. Statist. Soc. Ser. B MR Opsoer, J., Wang, Y. and Yang, Y. (200). Nonparaetric regression with correlated errors. Statist. Sci MR86070 Rasay, J. O. and Silveran, B. W.(2002). Applied Functional Data Analysis: Methods and Case Studies. Springer, New York. MR90407 Rasay, J. O. and Silveran, B. W. (2005). Functional Data Analysis, 2nd ed. Springer, New York. MR Rice, J. A. and Silveran, B. W. (99). Estiating the ean and covariance structure nonparaetrically when the data are curves. J. Roy. Statist. Soc. Ser. B MR Rice, J. A. and Wu, C. O. (200). Nonparaetric ixed effects odels for unequally sapled noisy curves. Bioetrics MR83334 Stone, C. J. (982). Optial global rates of convergence for nonparaetric regression. Ann. Statist MR sybakov, A. B. (2009). Introduction to Nonparaetric Estiation. Springer, New York. MR Wahba, G. (990). Spline Models for Observational Data. CBMS-NSF Regional Conference Series in Applied Matheatics 59. SIAM, Philadelphia, PA. MR Wang, Y. (996). Function estiation via wavelet shrinkage for long-eory data. Ann. Statist MR Yao, F., Müller, H.-G. and Wang, J.-L. (2005). Functional data analysis for sparse longitudinal data. J. Aer. Statist. Assoc MR26056

28 28.. CAI AND M. YUAN Departent of Statistics he Wharton School University of Pennsylvania Philadelphia, Pennsylvania 904 USA E-ail: School of Industrial and Systes Engineering Georgia Institute of echnology Atlanta, Georgia USA E-ail: