Boosting local quasi-likelihood estimators

Transcription

1 Ann Inst Stat Mat (00) 6:5 48 DOI 0.007/s Boosting local quasi-likeliood estimators Masao Ueki Kaoru Fueda Received: Marc 007 / Revised: 8 February 008 / Publised online: 5 April 008 Te Institute of Statistical Matematics, Tokyo 008 Abstract For likeliood-based regression contets, including generalized linear models, tis paper presents a boosting algoritm for local constant quasi-likeliood estimators. Its advantages are te following: (a) te one-boosted estimator reduces bias in local constant quasi-likeliood estimators witout increasing te order of te variance, (b) te boosting algoritm requires only one-dimensional maimization at eac boosting step and (c) te resulting estimators can be written eplicitly and simply in some practical cases. Keywords Bias reduction L Boosting Generalized linear models Kernel regression Local quasi-likeliood Nadaraya Watson estimator Introduction Tis paper deals wit likeliood-based regression problems for wic generalized linear models are typically used. However, te effectiveness of generalized linear models are limited because of teir restricted fleibility. In te case, it is better to use some nonparametric approac suc as kernel regression (Wand and Jones 995; Fan and Gijbels 996). Fan et al. (995) etended te local constant and local polynomial regression estimators to quasi-likeliood metods, wic is an etension of generalized linear models (see Sect..). Loader (999) recommends te local quadratic fit tat as te bias of O( 4 ) and te variance of O{(n) }, were is te M. Ueki (B) K. Fueda Graduate Scool of Environmental Science, Okayama University, Naka --, Tsusima, Okayama , Japan ueki@ems.okayama-u.ac.jp K. Fueda fueda@ems.okayama-u.ac.jp

2 6 M. Ueki, K. Fueda bandwidt, from a practical viewpoint. However, te local polynomial regression estimators require etensive computations because tey rely on numerical maimization at eac evaluated point. Fan (999) overcomes tat problem by introducing one-step local quasi-likeliood estimators, altoug some efforts at implementation are needed. If one uses te local constant fit, suc difficulties in bot computation and implementation do not occur because it can be written eplicitly and simply. However, te bias of te local constant fit is O( ) wic is often not negligible, wile te variance is O{(n) }. We consequently take a course of not using local polynomials but applying a boosting algoritm to te local constant fit to reduce te order of te bias, were te boosting is a recently investigated statistical metodology (Scapire 990; Freund 995; Freund and Scapire 996; Friedman 00; Bülmann and Yu 00; Marzio and Taylor 004a).Marzio and Taylor (004b) proposed te boosting for Nadaraya Watson estimator, wic is te local constant fit in Gaussian model, were te algoritm tey applied is te L Boosting of Friedman (00) and Bülmann and Yu (00). Te bias of te Nadaraya Watson estimator is O( ) and teir one-boosted estimator reduces te bias to O( 4 ). Tis type of bias reduction is eamined by many autors (Jones et al. 995; Coi and Hall 998; Marzio and Taylor 004a). Te advantages of our algoritm are te following: (a) te one-boosted estimator reduces te bias of O( ) to O( 4 ) witout increasing te order of te variance, (b) our algoritm requires only one-dimensional maimization at eac boosting step wile te local polynomials need multi-dimensional maimization and (c) te resulting estimators can be written eplicitly and simply in some practical cases. Our approac is also te simplest among te bias reduction tecniques. Boosting local constant quasi-likeliood estimators. Local constant quasi-likeliood estimators Tis section describes local constant quasi-likeliood estimators. Let (X, Y ),..., (X n, Y n ) be a set of independent random pairs were, for eac i, Y i is a scalar response variable and X i is an R d -valued vector of covariates aving density f wit support supp( f ) R d.let(x, Y ) denote a generic member of te sample, and let m() = E(Y X = ). Wen te range of m() is restricted on an interval I of R in likeliood based problems, wit generalized linear models, suc as Bernoulli, Poisson and gamma, te estimation is suitable for η() = g{m()} instead of for m() were g is a one-to-one function from I to R, called link function. Te quasi-likeliood metod is an etension of te generalized linear models. Te former requires only te specification of a relationsip between te mean and variance of Y ; it is useful even if te likeliood function is not available. Te former metod maimizes te quasi-likeliood function Q{m(), y} instead of te log-likeliood function. Tis paper eplains only te case in wic te conditional variance is modeled as var(y X = ) = V {m()} for some known positive function V, and te corresponding quasi-likeliood function Q(m, y) satisfies y m Q(m, y) = m V (m). ()

3 Boosting local quasi-likeliood estimators 7 Te quasi-score () possesses properties tat resemble tose of te usual likeliood score function: one of te properties is tat it satisfies te first two moment conditions of Bartlett s identities (Fan and Gijbels 996). Te likeliood score of one-parameter eponential family is a special case of ()(Fan et al. 995). For simplicity, we deal wit scalar covariates X,...,X n. Te local constant quasilikeliood estimator for m() can be written eplicitly as ˆm 0 (; ) = g {ˆη 0 (; )} = ni= K (X i )Y i ni= K (X i ), () wic is given by maimizing n i= Q{g (η), Y i }K (X i ) wit respect to η, were K (z) = K (z/)/, K (z) is a symmetric unimodal probability density called kernel function, and > 0 is a parameter called bandwidt, wic controls te etent of smooting. Te estimator () is simple, but it performs poorly. In te net section, we strengten () using boosting.. Te boosting algoritm In L boosting, a simple base estimator, called weak learner, is used iteratively in least-squares fitting wit stage-wise updating of current residuals. In tis section, before proposing te boosting local quasi-likeliood estimators, we describe te L boosting algoritm proposed by Marzio and Taylor (004b), were te weak learner is te Nadaraya-Watson estimator, wic corresponds to () wen te link function g is te identity. Te algoritm is given as follows. Algoritm Step (initialization) Let ˆm 0 be te Nadaraya-Watson estimator wit a previously cosen > 0. Step (iteration) Repeat for b = 0,...,B, (i) (ii) Compute n estimates ˆm b (X i ),i =,...,n. Update ˆm b+ () = ˆm b ()+ˆδ(), were ˆδ() is te Nadaraya-Watson estimator in wic te response variables Y i are replaced by te current residuals U i = Y i ˆm b (X i ), i.e., ˆδ() = ni= K (X i )U i ni= K (X i ). () Least-squares fitting can be viewed as an optimization in te Gaussian regression model. Tis consideration enables us to generalize te L boosting to tat in a quasilikeliood framework. Here, we must take into account tat additivity in step (ii) does not necessarily old in tis framework. To acieve te generalization, we rewrite ()as ˆδ() = argma δ [Y i {ˆm b (X i ) + δ}] K (X i ). (4) i=

4 8 M. Ueki, K. Fueda Based on te form of(4), we generalize Algoritm for local constant quasi-likeliood estimators () asfollows. Algoritm Step (initialization) Let ˆη 0 be () wit a previously cosen > 0. Step (iteration) Repeat for b = 0,...,B, (i) Compute n estimates ˆη b (X i ),i =,...,n. (ii) Update ˆη b+ () =ˆη b () + ˆδ(), were ˆδ() = argma δ R Q[g {ˆη b (X i ) + δ}, Y i ]K (X i ). (5) i= We can obtain te estimator for m() by ˆm b+ () = g {ˆη b+ ()}. Note tat ˆδ() is added in η s space for range preservation, and (5) requires scalar maimization only, even for multiple covariates. Furtermore, some cases eist for wic te resulting estimator can be written eplicitly and simply as follows. Eample (Gaussian model wit identity link) Tis eample corresponds to Algoritm. Te quasi-likeliood function ten coincides wit te usual log-likeliood function of te Gaussian distribution wit mean m and variance unity: Q(m, y) = {(y m) +log(π)}/; V (m) =. Te link function g is te identity, η = g(m) = m. Attebt stage, ˆm b+ () = ˆm b () + ˆδ(), were ˆδ() is given in (). Eample (Poisson model wit log link) Te link function g is log link, η = g(m) = log m. Te quasi-likeliood function ten coincides wit te usual log-likeliood function of te Poisson distribution wit mean m: Q(m, y) = m + y log m + log y!; V (m) = m. Attebt stage, ˆη b+ () =ˆη b () + ˆδ(), were ep{ ˆδ()} = ni= K (X i )Y i ni= K (X i ) ep{ˆη b (X i )}. Eample (gamma model wit log link) Te link function g is log link, η = g(m) = log m. Te quasi-likeliood function ten coincides wit te usual log-likeliood function of te gamma density wit mean m and sape parameter α: Q(m, y) = αy/m α log m + (α ) log y + α log α log Ɣ(α), were Ɣ( ) is te gamma function; V (m) = m /α. Attebt stage, ˆη b+ () =ˆη b () + ˆδ(), were ep{ ˆδ()} = ni= K (X i ) ep{ ˆη b (X i )}Y i ni=. K (X i ). Empasizing te updating term Primarily, boosting can be regarded as a sequential greedy optimization of additive models, wic is typical in L boosting. Te updating term in eac step of boosting can be regarded as an estimation using iteratively reweigted data. From tis viewpoint, we specifically eamine te updating term defined in (5).

5 Boosting local quasi-likeliood estimators 9 Second-order Taylor approimation in (5) yields tat ( ) Xi l n (δ) = K Q[g {ˆη b (X i ) + δ, Y i }] i= ( )( Xi K Q[g {ˆη b (X i )}, Y i ]+δq {ˆη b (X i ), Y i }+ ) δ q {ˆη b (X i ), Y i }, i= were q i are defined in te Appendi. Terefore, te updating term is approimated as te following. ( ) ni= K Xi q {ˆη b (X i ), Y i } ˆδ() ( ) n i= K Xi q {ˆη b (X i ), Y i } ni= K (X i )[g {ˆm b (X i )}V {ˆm b (X i )}] {Y i ˆm b (X i )} = n. (6) i= K (X i )q {ˆη b (X i ), Y i } Using (6), te updating term ˆδ() can be interpreted approimately as te reweigted version of te kernel regressor (5), were te response variables Y i are replaced by te current residuals as in () in Algoritm. Tis consideration describes a transparent relationsip between te proposed algoritm and L boosting. Bias reduction property In te following teorem, we state te bias reduction property, i.e., one-boosted estimators ˆη (; ) reduce te bias of O( ) in local constant quasi-likeliood estimators, wic is often not negligible, to O( 4 ). Teorem Suppose tat te conditions presented in te Appendi old. If 0 and n as n, te estimator after one-boosting iteration, ˆη (; ), as bias of O( 4 ) and variance of var{ˆη (; )} =var(y X = ) g {m()} nf () T K (z)dz + o{(n) }, were T K (z) = K (z) K K (z) is te fourt order kernel in Jones et al. (995, Teorem ). In addition, ˆm (; ) = g {ˆη (; )} as te bias of O( 4 ) and te variance of var{ ˆm (; )} =var{ˆη (; )}/g {m()} + o{(n) }. Te proof is given in te Appendi. According to Fan et al. (995), te bias and variance of te local constant quasi-likeliood estimators, i.e., te non-boosted estimators, are O( ) and O{(n) }, respectively, wic in turn implies tat bias reduction is acieved witout increasing te order of te variance.

6 40 M. Ueki, K. Fueda 4 Numerical illustrations Tis section provides some numerical illustrations in Poisson (eample ) and eponential (Eample for α = ) models. We use te Epanecnikov kernel K (u) = 4 ( u ) { <u<}, were { } is te indicator function. Te eamined conditional means are m p () = ep{cos(π )}, m p () = arcsin +, for Poisson, m e () = 8ep( ), m e () = ( + ) / + 4, for eponential, and te design density f () is te uniform density on [, ]. To measure te performance of resulting estimator ˆm(), we use te square root of average square [ errors, RASE = j= {m( j) ˆm( j )} ] /,forj = + ( j )/99, j =,...,00, at wic te function m() is estimated. Te sample size n is 00 trougout. To sow ow te proposed algoritm works, we demonstrate te beaviors for one random sample in Fig. (Poisson) and Fig. (eponential) were ˆm b () are plotted for b = 0 (das), (dot), (dot das) and (long das), togeter wit te true curve (solid). Te bandwidt used in te left and rigt panels are optimal, respectively, for b = 0 and b =, wic are founded numerically wit respect to te RASE. It seems tat ˆm () in te rigt panels fit more appropriately to te true curves tan te ˆm 0 () in te left panels: te optimal boosted estimators are better tan te optimal non-boosted ones. (a) (c) (b) (d) Fig. in a Poisson case: for m p (), a =0.7, b =0.4; for m p (), c =0.96, d =.67

7 Boosting local quasi-likeliood estimators 4 (a) (c) (b) (d) Fig. in an eponential case: for m e (), a = 0.54, b = 0.99; for m e (), (c) = 0.97, d =.6 We eamine te boosting for various, and repeat te procedures 500 times to illustrate te efficiency. Figure sows te average RASEs against, were te plotted numbers 0 indicate te corresponding boosting iterations (te 0 corresponds to non-boosted estimator, i.e., local constant quasi-likeliood estimator). All figures suggest tat boosting works well for appropriate because eac minimum RASE of te boosted estimate is smaller tan tat of non-boosted estimate. Note tat wic minimizes te RASE tends to increase as te number of boosting iterations grows. Tis penomenon is identical to tat observed in Marzio and Taylor (004a) for boosting kernel density estimators. Terefore, we recommend to take somewat larger tan te optimal one for non-boosted estimators as te strategy to select. Net, we verify te implication in Teorem related to te mean squared error (MSE). Table compares teoretical MSE epressions given in Teorem and simulated true MSEs in 000 eperiments, were bot te non-boosted and one-boosted estimators, ˆm 0 () and ˆm () are evaluated at tree points = 0., 0, 0.6. Te results sow tat te asymptotic MSE epressions given in Teorem approimate te true MSEs well. In practice, te bandwidt must be estimated from te data. One way of coosing is to use likeliood based cross-validation. Te metod is useful wen te form of Q(m, y) is known, as in generalized linear models. Te bandwidt ĥ selected by te cross-validation is te maimizing Q{ ˆm i (X i ), Y i }, i=

8 4 M. Ueki, K. Fueda (a) RASE (b) RASE (c) RASE (d) RASE Fig. Average RASE plots: a for m p (), b for m p (), c for m e () and d for m e (). Te plotted numbers 0 indicate te corresponding boosting iterations Table Teoretical MSE epressions and simulated MSEs for non-boosted and one-boosted estimators, ˆm 0 () and ˆm () T/S B m p ( = 0.) m p ( =.) m e ( = 0.8) m e ( =.7) T , 0.68, , 0.040, ,.0, , 0.9, 0.44 S , 0.94, , 0.05, ,.65, , 0.49, 0.98 T 0.06, 0., , 0.08, ,.6, , 0.49, 0.6 S 0.08, 0.66, , 0.05, ,., , 0.49, 0.90 In te table, eac MSE evaluated at = 0., 0, 0.6 is described in te order corresponding to tat of. T and S denote Teoretical and Simulated values, respectively. B means te boosting iteration number were ˆm i ( ) corresponds to te version of ˆm( ) tat is constructed by eliminating it data (X i, Y i ). Table sows te average RASEs in 500 simulation eperiments for respective boosting iteration numbers 0, wit bandwidts selected using crossvalidation. Te bandwidts selected ere are cosen among finite candidates, wic consist of 50 equi-spaced points on te intervals given in te second line of Table. Tese intervals are determined empirically according to te variability of ĥ. By te results in Table, we ascertained tat te boosted estimation, at least once, works better tan non-boosted estimation, even if te bandwidts are estimated using crossvalidation. Consequently, it is wortwile to apply te boosting algoritm in practical situations.

9 Boosting local quasi-likeliood estimators 4 Table Average RASE for eac boosting number, wit bandwidt selected by cross-validation B m p m p m e m e [0.,.5] [0.,] [0.,.] [0.,4.5] B means te boosting iteration number. Te candidate bandwidts consist of 50 equi-spaced points on te intervals given in te second line of te table 5 Concluding remarks We propose a boosting algoritm for local constant quasi-likeliood estimators tat provides bias reduction. Te metod is valid in bot computation and implementation. Tere are still some issues. Te first is te selection of. A reasonable solution in generalized linear models is to use likeliood-based cross-validation. In some cases in wic te resulting estimators are given eplicitly, te required computations for crossvalidation are few. However, in oter cases in wic numerical maimizations are required, including logistic regression, te required computations could be epensive. For tis reason, better selection criteria are needed. Te second is ow to stop te boosting iteration. In our eaminations, te two-boosted and tree-boosted estimators work better tan te one-boosted estimators. However, as Bülmann and Yu (00) pointed out, many boosting iterations cause overfitting. To avoid tis, we ave to stop te iteration based on a stopping rule suc as cross-validation. Te tird is to analyze te two-boosted and more-boosted estimators because we ave only justified te one-boosted estimators in tis paper. Acknowledgments paper considerably. Te autors would like to tank te referees for elpful suggestions tat improve te Appendi: Proof of Teorem Preliminary Let q i (η, y) = ( i / η i )Q{g (η), y} for i =,. Since Q satisfies (), q i is linear in y for fied, q {η(), m()} =0 and q {η(), m()} = ρ(), were ρ() =[g {m()} V {m()}].alsoletσ () = var(y X = ). We present te conditions: (i) Te function q (η, y) < 0forη R and y in te range of te response variable; (ii) Te functions f (4),η (4),σ, V and g (4) are continuous; (iii) For eac supp( f ), ρ(), σ () and g {m()} are nonzero; (iv) Te kernel K is a symmetric probability density wit support [, ]; (v) is an interior point of supp( f ). Furtermore, we assume tat n /9, wic is te optimal rate tat minimizes te asymptotic MSE of order O{ 8 + (n) }. See te argument below Teorem of Jones et al. (995). We also write ( fρ)() = f ()ρ() and (mf)() = m() f ().

10 44 M. Ueki, K. Fueda Let δ = an δ, ˆη i() =ˆη i (; ) for i = 0,, ˆm 0 () = ˆm 0 (; ) and l n (δ ) = ( Xi K i= ) ( Q[g {ˆη 0 (X i ) + a n δ }, Y i ] Q[g {ˆη 0 (X i )}, Y i ]), were a n = (n) /. Condition (i) implies tat l n is concave in δ.letˆδ be te maimizer of l n (δ ), ten ˆδ = ( fρ)() W n + o p () were W n = a n ( Xi K i= ) q {ˆη 0 (X i ), Y i }. (7) Te derivation of (7) is as follows. Using Taylor epansion, l n (δ ) = W n δ + A nδ + a n 6 ( Xi K i= ) q (η i, Y i )δ, (8) were η i is between ˆη 0 (X i ) and ˆη 0 (X i ) + a n δ, and A n = an ni= K q {ˆη 0 (X i ), Y i }.By ˆη 0 () = η() + o p (), ( ) Xi ( E(A n ) = X E[K ) q {η(x ), m(x )}] + o() = ( fρ)() + o() and var(a n ) = O(a n );usinga n = E(A n ) + O p {var(a n ) / },weavea n = ( fρ)() + o p (). A similar argument in Fan and Gijbels (996, p. ) sows tat te last term in (8) is bounded by O p (a n ). Terefore, l n (δ ) = W n δ ( fρ)()δ + o p (). Using te quadratic approimation lemma (Fan and Gijbels 996, p. 0), we obtain (7). Bias First, we derive te bias. Let µ = z K (z)dz.using n ni= K (X i ) = f () + µ f () + O p ( 4 ) by conditions (ii), (iv) and (n) / n 4/9 4, it follows from () tat g {ˆη 0 ()} = ˆm 0 () = nf() j= { K (X j )Y j f } () µ + O p ( 4 ). f () (9)

11 Boosting local quasi-likeliood estimators 45 Using q {η(), y} =g {m()}ρ()[y g {η()}],(7) is rewritten as ˆδ = a n K (X i )ρ(x i )g {m(x i )} ( fρ)() i= { Y i K (X j X i )Y j f } (X i ) nf(x i ) µ + O p (an f (X j =i i ) 4 ). (0) In addition, using conditions (ii), (iv) and K (v u)(mf)(v)dv = (mf)(u) + µ (mf) (u) + O( 4 ), E(ˆδ ) = a nn ( fρ)(u)k (u )g {m(u)} ( fρ)() [ m(u) { K (v u)(mf)(v)dv f }] (u) f (u) µ f (u) du + O(an 4 ) = a n ( fρ)(u)k (u )g {m(u)} µ ( fρ)() { (mf) } (u) (mf )(u) du + O(an f (u) f (u) 4 ) = an g {m()} { f () µ (mf) () (mf )() } + O(an 4 ). () According to Fan et al. (995), te bias of ˆη 0 () is given as E{ˆη 0 ()} η() = g {m()} f () µ {(mf) () (mf )()}+O( 4 ). () Combining () and (), we can sow tat te bias of ˆη () =ˆη 0 ()+ ˆδ() is O( 4 ). Variance Secondly, we derive te variance. Define ˆη i () = a n [ˆη i() E{ˆη i () X}] for i = 0,, were E{ X} is te conditional epectation under given X,...,X n. Ten, it olds tat var{ˆη ()} =an E{ˆη () }+E ( [E{ˆη () X} η()] ) [E{ˆη ()} η()], were te tird term, te squared bias, is {O( 4 )}. To calculate te second term, we first note, using te Taylor epansion for (9), tat ˆη 0 () η() = g {m()} nf() j= { [K (X j )Y j f } ] () µ (mf)() +O p ( 4 ). f () ()

12 46 M. Ueki, K. Fueda From (0) and (), E{ˆη () X} η() = E{ˆη 0 () η() X}+E{ˆδ() X} =D + O p ( 4 ), (4) were D = n ni= R i + n ni = j S ij, R i = R(X i ), S ij = S(X i, X j ), R(X i ) = g {m()} f () [ { K (X i )m(x i ) f } ] () µ (mf)() f () + ( fρ)() K (X i )ρ(x i )g {m(x i )}m(x i ), S(X i, X j ) = ( fρ)() K (X i ) ρ(x i)g {m(x i )} K (X j X i )m(x j ) f (X i ) { f } (X i ) µ. f (X i ) Note tat E(D) equals te bias of ˆη () wit error O( 4 ), i.e., E(D) = O( 4 ). Observing tat E (R ) and E (S ) are of order O(), E(D ) = E n R i R j + n R i S jk + n 4 S ij S kl i, j i j =k i = j k =l ={E (R )} + E (R )E (S ) +{E (S )} + O(n ) ={E (R + S )} + O(n ) ={E(D)} + O(n ) = O( 8 ), were E and E represent epectations wit respect to X and (X, X ), respectively. Terefore, te second term is also of order O( 8 ). It is, after all, sufficient to calculate E{ˆη () }.By() and (na n ) = a n, ˆη 0 () = a n g {m()} f () K (X j )Ỹ j + O p ( ), (5) j= in wic Ỹ i = Y i m(x i ). On te oter and, defining G r,n = a n n i= K (X i ) {Ỹi nf(x } i ) j =i K (X j X i )Ỹ j (X i ) r for r =0, and ξ()=ρ()g {m()}, it follows from Taylor epanding ξ(x i ) around in (0), wit condition (ii), tat ˆδ E(ˆδ X) = ( fρ)() {G 0,nξ() + G,n ξ ()}+O p ( ) = g {m()} G 0,n + o p (). (6) f ()

13 Boosting local quasi-likeliood estimators 47 Te second equality follows from G,n = o p (), wic we sow in wat follows. Observing tat E(Ỹ i Ỹ j ) = 0ifi = j and = σ (w) f (w)dw oterwise, we ave (a n ) E(G r,n ) [ = E i,k= + n f (X i ) f (X k ) { K (X i )K (X k ) Ỹ i Ỹ k j =i,l =k = I I + I + o(i I + I ), nf(x i ) K (X j X i )Ỹ j Ỹ k j =i K (X j X i )K (X l X k )Ỹ j Ỹ l } (X i ) r (X k ) r ] were I = n I = n I = n K (w )( f σ )(w)(w ) r dw, K (w ) K (u )K (u w)( f σ )(u)(u ) r du(w ) r dw, K (u ) K (v ) K (w u)k (w v)( f σ )(w) dw(u ) r (v ) r dudv. Ten, I = n r z r K (z)( f σ )( + z)dz = O(an r ) and I = n r = n r K (z) K (z) ( u K ) ( ) u K z ( f σ )(u) ( ) u r duz r dz K (s)k (s z)( f σ )( + s)s r dsz r dz = O(a n r ). Similarly, I = n r 4 = n r K (s) = O(a n r ). K (s) K (t) K (t) ( w K ) ( ) w s K t ( f σ )(w)dw sdstdt K (z s)k (z t)( f σ )( + z)dzsdstdt Tus, we deduce tat E(Gr,n ) = O(r ). Noting tat E(G r,n ) = 0, var(g r,n ) = E(Gr,n ). Tis implies tat G r,n = O p ( r ) = O p ( r ), in particular, G,n = o p (), tereby yielding (6).

14 48 M. Ueki, K. Fueda Combining (5) and (6), we can write ˆη () = g {m()} Z n + o p (), (7) f () were Z n = a n { n i= K (X i ) Ỹ i nf(x i ) j =i K (X j X i )Ỹ j }; consequently, E{ˆη [ ] () }= g {m()} f () E(Z n ) + o(). Te same arguments as in deriving (6) apply to te calculation of E(Zn ), wic derives te variance. Bias and variance of ˆm () Te assertion regarding te estimator for m() is straigtforwardly obtained by noting ˆm () = g {ˆη ()} and using te same process as tat of te proof of Teorem in Fan et al. (995). References Bülmann, P., Yu, B. (00). Boosting wit te L loss: regression and classification. Journal of te American Statistical Association, 98, 4 9. Coi, E., Hall, P. (998). On bias reduction in local linear smooting. Biometrika, 85, 45. Fan, J. (999). One-step local quasi-likeliood estimation. Journal of te Royal Statistical Society, Ser. B, 6, Fan, J., Gijbels, I. (996). Local polynomial modelling and its applications. London: Capman and Hall. Fan, J., Heckman, N. E., Wand, M. P. (995). Local polynomial kernel regression for generalized linear models and quasi-likeliood functions. Journal of te American Statistical Association, 90, Freund, Y. (995). Boosting a weak learning algoritm by majority. Information and Computation,, Freund, Y., Scapire, R. E. (996). Eperiments wit a new boosting algoritm. In: Saitta, L. (Ed.) Macine Learning: Proceedings of te Tirteent International Conference, (pp ). San Francisco: Morgan Kauffman. Friedman, J. (00). Greedy function approimation: a gradient boosting macine. Te Annals of Statistics, 9, 89. Jones, M. C., Linton, O. and Nielsen, J. (995). A simple bias reduction metod for density estimation. Biometrika, 8, 7 8. Loader, C. R. (999). Local regression and likeliood. New York: Springer. Marzio, M. D., Taylor, C. C. (004a). Boosting kernel density estimates: A bias reduction tecnique? Biometrika, 9, 6. Marzio, M. D., Taylor, C. C. (004b). Multistep kernel regression smooting by boosting. leeds.ac.uk/~carles/boostreg.pdf, unpublised manuscript. Scapire, R. E. (990). Te strengt of weak learnability. Macine Learning, 5,. Wand, M. P., Jones, M. C. (995). Kernel smooting. London: Capman and Hall.