Nonparametric regression: minimax upper and lower bounds

Size: px

Start display at page:

Download "Nonparametric regression: minimax upper and lower bounds"

Amy Richard
5 years ago
Views:

1 Capter 4 Noparametric regressio: miimax upper ad lower bouds 4. Itroductio We cosider oe of te two te most classical o-parametric problems i tis example: estimatig a regressio fuctio o a subset of te real lie (te most classical problem beig estimatio of a desity). I o-parametric regressio, we assume tere is a ukow fuctio f : R R, were f belogs to a pre-determied class of fuctios F; usually tis class is parameterized by some type of smootess guaratee. To make our problems cocrete, we will assume tat te ukow fuctio f is L-Lipscitz ad defied o [0,]. Let F deote tis class. (For a fuller tecical itroductio ito oparametric estimatio, see te book by Tsybakov [].) Figure 4.. Observatios i a o-parametric regressio problem, wit fuctio f plotted. (Here f(x) = si(x+cos (3x)).) 4

2 Staford Statistics 3/Electrical Egieerig 377 Jo Duci I te stadard o-parametric regressio problem, we obtai observatios of te form Y i = f(x i )+ε i (4..) were ε i are idepedet, mea zero coditioal o X i, ad E[ε i ] σ. See Figure 4. for a example. We also assume tat we fix te locatios of te X i as X i = i/ [0,], tat is, te X i are evely spaced i [0,]. Give observatios Y i, we ask two questios: () ow ca we estimate f? ad () wat are te optimal rates at wic it is possible to estimate f? 4. Kerel estimates of te fuctio A atural strategy is to place small bumps aroud te observed poits, ad estimate f i a eigborood of a poit x by weigted averages of te Y values for oter poits ear x. We ow formalize a strategy for doig tis. Suppose we ave a kerel fuctio K : R R +, wic is cotiuous, ot idetically zero, as support supp K = [, ], ad satisfies te tecical coditio λ 0 supk(x) if K(x), (4..) x x / were λ 0 > 0 (tis says te kerel as some widt to it). A atural example is te tet fuctio give by K tet (x) = [ x ] +, wic satisfies iequality (4..) wit λ 0 = /. See Fig. 4. for two examples, oe te tet fuctio ad te oter te fuctio ( K(x) = { x < }exp (x ) wic is ifiitely differetiable ad supported o [, ]. ) exp ( (x+) ), Figure 4.: Left: tet kerel. Rigt: ifiitely differetiable compactly supported kerel. Now we cosider a atural estimator of te fuctio f based o observatios (4..) kow as te Nadaraya-Watso estimator. Fix a badwidt, wic we will see later smoots te estimated fuctios f. For all x, defie weigts W i (x) := ( ) Xi x K ( K Xj x 4 )

3 Staford Statistics 3/Electrical Egieerig 377 Jo Duci ad defie te estimated fuctio f (x) := Y i W i (x). Te ituitio ere is tat we ave a locally weigted regressio fuctio, were poits X i i te eigborood of x are give iger weigt ta furter poits. Usig tis fuctio f as our estimator, it is possible to provide a guaratee o te bias ad variace of te estimated fuctio at eac poit x [0,]. Propositio 4.. Let te observatio model (4..) old ad assume coditio (4..). I additio assume te badwidt is suitably large tat / ad tat te X i are evely spaced o [0,]. Te for ay x [0,], we ave E[ f (x)] f(x) L ad Var( f (x)) σ λ 0. Proof To boud te bias, we ote tat (coditioig implicitly o X i ) E[ f (x)] = E[Y i W i (x)] = E[f(X i )W i (x)+ε i W i (x)] = f(x i )W i (x). Tus we ave tat te bias is bouded as E[ f (x)] f(x) f(x i ) f(x) W i (x) i: X i x To boud te variace, we claim tat f(x i ) f(x) W i (x) L W i (x) mi W i (x) = L. { } λ 0,. (4..) Ideed, we ave tat W i (x) = ( Xi x ) K ( K Xj x ) = K ( ) Xi x j: X j x / K ( Xj x ) ( ) K Xi x λ 0 sup x K(x) {j : X j x /}, ad because tere are at least / idices satisfyig X j x, we obtai te claim (4..). Usig te claim, we ave [( ) ] Var( f (x)) = E (Y i f(x i ))W i (x) [( = E = ) ] ε i W i (x) W i (x) E[ε i] σ W i (x). 43

4 Staford Statistics 3/Electrical Egieerig 377 Jo Duci Notig tat W i (x) /λ 0 ad W i(x) =, we ave completig te proof. σ W i (x) σ maxw i (x) W i (x) i }{{} = σ λ 0, Wit te propositio i place, we ca te provide a teorem boudig te worst case poitwise mea squared error for estimatio of a fuctio f F. Teorem 4.. Uder te coditios of Propositio 4., coose = (σ /L λ 0 ) /3 /3. Te tere exists a uiversal (umerical) costat C < suc tat for ay f F, sup E[( f (x) f(x)) ] C x [0,] ( Lσ Proof Usig Propositio 4., we ave for ay x [0,] tat E[( f (x) f(x)) ] = λ 0 ) /3 3. ( ) E[ f (x)] f(x) +E[( f (x) E[ f (x)]) ] σ λ 0 +L. Coosig to balace te above bias/variace tradeoff, we obtai te toerem. By itegratig te result i Teorem 4. over te iterval [0, ], we immediately obtai te followig corollary. Corollary 4.3. Uder te coditios of Teorem 4., if we use te tet kerel K tet, we ave were C is a uiversal costat. supe f [ f f ] C f F ( ) Lσ /3, I Propositio 4., it is possible to sow tat a more clever coice of kerels oes tat are ot always positive ca attai bias E[ f (x)] f(x) = O( β ) if f as Lipscitz (β )t derivative. I tis case, we immediately obtai tat te rate ca be improved to supe[( f (x) f(x)) ] C β+, x ad every additioal degree of smootess gives a correspodig improvemet i covergece rate. We also remark tat rates of tis form, wic are muc larger ta, are caracteristic of oparametric problems; essetially, we must adaptively coose a dimesio tat balaces te sample size, so tat rates of / are difficult or impossible to acieve. 44

5 Staford Statistics 3/Electrical Egieerig 377 Jo Duci 4.3 Miimax lower bouds o estimatio wit Assouad s metod Now we ca ask weter te results we ave give are i fact sarp; do tere exist estimators attaiig a faster rate of covergece ta our kerel-based (locally weigted) estimator? Usig Assouad s metod, we sow tat, i fact, tese results are all tigt. I particular, we prove te followig result o miimax estimatio of a regressio fuctio f F, were F cosists of - Lipscitz fuctios defied o [0,], i te error, tat is, f g = 0 (f(t) g(t)) dt. Teorem 4.4. Let te observatio poits X i be spaced evely o [0,], ad assume te observatio model (4..). Te tere exists a uiversal costat c > 0 suc tat M (F, ) := if sup f f F ] ( σ E f [ f f c Deferrig te proof of te teorem temporarily, we make a few remarks. It is i fact possible to sow usig a completely idetical tecique tat if F β deotes te class of fuctios wit β derivatives, were te (β )t derivative is Lipscitz, te M (F β, ) c ( σ ) β β+. So for ay smootess class, we ca ever acieve te parametric σ / rate, but we ca come arbitrarily close. As aoter remark, wic we do ot prove, i dimesios d, te miimax rate for estimatio of fuctios f wit Lipscitz (β )t derivative scales as M (F β, ) c ( σ ) β β+d. Tis result ca, similarly, be proved usig a variat of Assouad s metod; see, for example, te book of Györfi et al. [, Capter 3], wic is available olie. Tis is a strikig example of te curse of dimesioality: te pealty for icreasig dimesio results i worse rates of covergece. For example, suppose tat β =. I dimesio, we require 90 (.05) 3/ observatios to acieve accuracy.05 i estimatio of f, wile we require 8000 = (.05) (+d)/ eve we te dimesio d = 4, ad observatios eve i 0 dimesios, wic is a relatively small problem. Tat is, te problem is made expoetially more difficult by dimesio icreases. We ow tur to provig Teorem 4.4. To establis te result, we sow ow to costruct a family of problems idexed by biary vectors v {,} k so tat our estimatio problem satisfies te separatio (3..), te we sow tat iformatio based o observig oisy versios of te fuctios we ave defied is small. We te coose k to make our resultig lower boud as ig as possible. Costructio of a separated family of fuctios To costruct our separatio i Hammig metric, asrequiredbyeq.(3..), fixsomek N; wewillcoosek later. Tisapproacissomewat differet from our stadard approac of usig a fixed dimesioality ad scalig te separatio directly; i o-parametric problems, we scale te dimesio itself to adjust te difficulty of te estimatio problem. Defie te fuctio g(x) = [/ x / ] +, so tat g is -Lipscitz ad is 0 outside of te iterval [0,]. Te for ay v {,} k, defie te bump fuctios g j (x) := ( (k k g x j )) k 45 ad f v (x) := ) 3. v j g j (x),

6 Staford Statistics 3/Electrical Egieerig 377 Jo Duci wic we see is -Lipscitz. Now, cosider ay fuctio f : [0,] R, ad let be sortad for te itervals = [(j )/k,j/k] for j =,...,k. We must fid a mappig idetifyig a fuctio f wit poits i te ypercube {,} k. To tat ed, we may defie a vector v(f) {,} k by v j (f) = argmi (f(t) sg j (t)) dt. s {,} We claim tat for ay fuctio f, ( (f(t) f v (t)) dt ) ( { vj (f) v j } f v (t) dt ). (4.3.) Ideed, o te set, we ave v j g j (t) = f v (t), ad tus g j (t) dt = f v (t) dt. Te by te triagle iequality, we ave ( ) ( ) { v j (f) v j } g j (t) dt = (( v j (f) v j )g j (t)) dt ( (f(t) v j g j (t)) dt ( (f(t) f v (t)) dt ) + ( ), (f(t) v j (f)g j (t)) dt by defiitio of te sig v j (f). Wit te defiitio of v ad iequality (4.3.), we see tat for ay vector v {,} k, we ave f f v = I particular, we kow tat f v (t) dt = k (f(t) f v (t)) dt /k 0 g(kt) dt = k 3 { v j (f) v j } f v (t) dt. 0 g(u) du c k 3, were c is a umerical costat. I particular, we ave te desired separatio f f v c k 3 ) { v j (f) v j }. (4.3.) Boudig te biary testig error Let P v deote te distributio of te observatios Y i = f v (X i ) + ε i we f v is te true regressio fuctio. Te iequality (4.3.) implies via Assouad s lemma tat M (F, ) c k 3 [ P +j P j ] TV. (4.3.3) Now, we use covexity ad Pisker s iequality to ote tat P +j P j TV max P v v,+j Pv, j 46 TV max v D ( kl P v,+j Pv, j ).

7 Staford Statistics 3/Electrical Egieerig 377 Jo Duci For ay two fuctios f v ad f v, we ave tat te observatios Y i are idepedet ad ormal wit meas f v (X i ) or f v (X i ), respectively. Tus D kl (Pv Pv ) = ( D kl N(fv (X i ),σ ) N(f v (X i ),σ ) ) = σ (f v(x i ) f v (X i )). (4.3.4) Now we must sow tat te expressio (4.3.4) scales more slowly ta, wic we will see must be te case as weever d am (v,v ). Ituitively, most of te observatios ave te same distributio by our costructio of te f v as bump fuctios; let us make tis rigorous. We may assume witout loss of geerality tat v j = v j for j >. As te X i = i/, we tus ave tat oly X i for i ear ca ave o-zero values i te tesorizatio (4.3.4). I particular, f v (i/) = f v (i/) for all i s.t. i k, i.e. i k. Rewritig expressio (4.3.4), te, ad otig tat f v (x) [ /k,/k] for all x by costructio, we ave /k σ (f v(x i ) f v (X i )) σ (f v(x i ) f v (X i )) σ k Combiig tis wit iequality (4.3.4) ad te miimax boud (4.3.3), we obtai k = k 3 σ. so P +j P j TV M (F, ) c k 3 k 3 σ, [ ] k 3 σ. Coosig k for optimal tradeoffs Now we simply coose k; i particular, settig ( ) /3 k = σ te k 3 σ /4 =, ad we arrive at M (F, ) c k 3 = c k c were c > 0 is a uiversal costat. Teorem 4.4 is proved. ( ) σ /3, 47

8 Bibliograpy [] L. Györfi, M. Koler, A. Krzyżak, ad H. Walk. A Distributio-Free Teory of Noparametric Regressio. Spriger, 00. [] A. B. Tsybakov. Itroductio to Noparametric Estimatio. Spriger,

5.1 A mutual information bound based on metric entropy

5.1 A mutual information bound based on metric entropy Chapter 5 Global Fao Method I this chapter, we exted the techiques of Chapter 2.4 o Fao s method the local Fao method) to a more global costructio. I particular, we show that, rather tha costructig a local