Notes On Nonparametric Density Estimation. James L. Powell Department of Economics University of California, Berkeley

Transcription

1 Notes O Noarametric Desity Estimatio James L. Powell Deartmet of Ecoomics Uiversity of Califoria, Berkeley Uivariate Desity Estimatio via Numerical Derivatives Cosider te roblem of estimatig te desity fuctio f(x) of a scalar, cotiuously-distributed i.i.d. sequece x i at a articular oit x: If te desity f is i a kow arametric family (e.g., Gaussia), estimatio of te desity reduces to estimatio of te ite-dimesioal arameters tat caracterize tat articular desity i te arametric family. Witout a arametric assumtio, toug, estimatio of te desity f over all oits i its suort would ivolve estimatio of a i ite umber of arameters, kow i statistics as a oarametric estimatio roblem (toug i ite-arametric estimatio migt be a more accurate title). Sice te desity fuctio f(x) is te derivative of te cumulative distributio fuctio F (x) Prfx i xg ad sice te emirical c.d.f. ^F (x) X fx i xg i is te atural oarametric of te c.d.f., it seems atural to base estimatio of f o te emirical c.d.f. However, wile ^F is -cosistet ad asymtotically ormal, it would be clearly osesical to estimate f by di eretiatig ^F sice its derivative is eiter zero or ude ed. Usig te de itio of te desity f as te (rigt-) derivative of te c.d.f., F (x + ) F (x) f(x) lim 0 we migt estimate te desity f by a corresodig di erece ratio of ^F : ^f(x) ^F (x + ) ^F (x) X fx < x i x + g i were te erturbatio, also kow as a badwidt or widow widt ) is ositive but small, deedig uo te samle size (i.e., ):

2 To sow MSE cosistecy of ^f it su ces to coose te badwidt sequece so tat te mea bias ad variace of ^f bot ted to zero as te samle size icreases. Sice te emirical c.d.f. ^F is a ubiased estimator of F tat is, E[ ^F (x)] F (x) te bias of ^f is evidetly E[ ^f(x)] f(x) 0 F (x + ) F (x) f(x) if Te variace of ^f(x) is wic will ted to zero if V ( ^f(x)) V 0 as : X fx < x i x + g i V (fx < x i x + g) F (x + ) F (x) ( (F (x + ) F (x))) f(x) + O( ) as : Tus, if te badwidt sequece teds to zero as teds to i ity, but at a slower rate ta te MSE of ^f will coverge to zero, esurig its (weak) cosistecy. To arrow dow te coice of badwidt sequece we migt wat to coose it to mamize te rate of covergece of te MSE of ^f to zero. To do tis, we eed a exlicit exressio for its bias as a fuctio of : Assumig F (x+) is smoot eoug to admit a secod-order Taylor s series exasio aroud 0, F (x + ) F (x) + f(x) + f 0 (x) + o( ) te MSE of ^f ca be exressed as MSE( ^f(x) ) E( ^f(x)) f(x)i + V ( ^f(x)) f 0 (x) + o( ) + f(x) + O( ): Because te squared bias is directly related to wile te variace is iversely related to te fastest covergece of te MSE to zero occurs we te squared bias ad variace coverge to zero at te same

3 seed. (If tey coverge at di eret seeds, te slower seed domiates.) Tus te otimal badwidt sequece will satisfy so O ( ) O O 3 will give te fastest rate of covergece of te MSE to zero, MSE( ^f(x) 3 ) O : Tis rate is slower ta te usual arametric rate of covergece of te MSE (if it ests) to zero, wic is O( ) for, e.g., te samle mea. Altoug te otimal rate of covergece of to zero does ot deed uo f itself, te otimal level would require kowledge of f(x): Assumig te badwidt sequece is of te form c 3 we ca coose c to miimize te limitig ormalized MSE wic is miimized i c at So te otimal badwidt sequece lim 3 MSE( ^f(x) ) f 0 (x) c c (f 0 (x)) 3 : f(x) f(x) (f 0 (x)) 3 + f(x) c tat miimizes te leadig terms i te MSE formula is ifeasible, sice it requires kowledge of te desity f(x) itself ad its rst derivative. Toug it ca be sow (uder additioal coditios) tat cosistet estimatio of te costat c will ot a ect te rate of covergece of ^f etc., oarametric estimatio of f 0 (x) caot be based uo ^f(x) directly (sice its derivative is zero werever it is de ed), but would require a similar umerical derivate aroac, wic would etail its ow badwidt coice roblem. A alterative is to stadardize te coice of te costat term c for some kow arametric desity fuctio for examle, if f(x) ( x ) were is te stadard ormal desity, te x c 3 (x) 3

4 ad estimatio of te otimal costat would reduce to estimatio of te stadard deviatio of te x i distributio. Istead of coosig te badwidt for a seci c value of x we migt wat to coose it to miimize a global MSE criterio over all ossible x values, suc as te itegrated mea-squared error criterio IMSE( ^f ) MSE( ^f(x) )dx f 0 (x) dx + + o( ) + O( ): Te same calculatios as for te oitwise otimal badwidt yield te otimal IMSE badwidt + to be + R (f 0 (x)) dx 3 wic still deeds uo te derivative of f: (A similar exressio olds for te badwidt miimizig a average MSE criterio, were \dx" is relaced by \f(x)dx" i te formula for te IMSE.) For te Gaussia desity f(x) ( x ) te otimal IMSE badwidt would be + R ( :4 3 x(x)) dx 3 so stadardizig te badwidt coice to be otimal for ormal desities reduces te badwidt coice roblem to estimatio of te stadard deviatio : Multivariate Kerel Desity Estimatio Te umerical derivative estimator of te uivariate desity f(x) above is a secial case of a geeral class of oarametric desity estimators called kerel desity estimators. Now suosig x i R we ca tik of smootig out te emirical c.d.f. ^F (x) for by relacig it wit a covolutio of ^F ad te distributio of a ideedet, cotiuously-distributed oise term " were is a small ositive badwidt as above ad " as a kow desity fuctio K(") (also kow as te kerel fuctio). For a articular realized value of x i te desity fuctio for X i x i + " would be f Xi (x) x K by te usual cage-of-variables formula for multivariate desities. (Here te absolute value of te determiat of te Jacobia d"dx 0 i is :) Tus te kerel desity estimator ^f(x) is te average of f Xi over 4

5 te observed values of x i i te samle, ^f(x) X i x K : As log as R K(u)du te desity estimator ^f(x) will also itegrate to oe over x as be ttig a desity fuctio. Te umerical derivative estimator discussed above is a secial case wit ad K(u) f u < 0g te desity fuctio for a Uiform( 0) radom variable. Te MSE calculatios are straigtforward extesios of tose for te umerical derivative estimator. Te exectatio of ^f is " E[ ^f(x)] X # x E K i x E K x z K f(z)dz: Makig te cage-of-variables u u(z) (x E[ ^f(x)] z) (wit du dz), K(u)f(x u)du wic clearly teds to f(x) as 0 if f is cotiuous ad bouded above (by domiated covergece), as log as R K(u)du : (Te last coditio imlies R jk(u)j du <, a coditio tat will be more relevat later, we te oegativity restrictio o K(u) is relaxed). Assumig tat f(x) is smoot eoug for f(x u) to admit a secod-order Taylor s exasio aroud 0 f(x u) 0 (u) + 0 (u) + o( 0 u tr 0 uu0 + o( ) te bias of ^f ca be exressed as E[ 0 uk(u)du tr 0 uu 0 K(u)du + o( ): If te mea of te kerel, R uk(u)du is ozero (as wit te oe-sided umerical derivative estimator above), te te bias is O() owever, if te kerel fuctio K(u) is cose to be symmetric about zero, or, more geerally, if uk(u)du 0 (*) 5

6 te te bias is E[ ^f(x)] f(x) O( ): Similarly, te variace of ^f(x) ca be calculated as V ar( ^f(x)) X x V ar K i x V ar K x E K x E K x z K f(z)dz [K (u)] f(x u)du (E[ ^f(x)]) f(x) [K (u)] du + o : (E[ ^f(x)]) (Te secod equality exloits te fact tat x i is i:i:d: ad te ft makes te same cage-of-variables as for te bias formula.) So te bias of ^f(x) is O( ) uder coditio (*), ad its variace is O(( ) ) te otimal badwidt sequece wic equates te rate of covergece of te squared bias ad variace to zero, tus satis es so ad te MSE evaluated at is O(( ) 4 ) O ( ) O MSE( ^f(x) ) O (+4) 4(+4) : Note tat, as icreases i.e., te umber of comoets i x i, umber of argumets of f(x) icreases te best rate of covergece of te MSE declies, a eomeo refereced by te catc rase, te curse of dimesioality. Te Asymtotic Distributio of te Kerel Desity Estimator Te kerel desity estimator ^f(x) ca be rewritte as a samle average of ideedet, ideticallydistributed radom variables ^f(x) X z i i 6

7 were z i ^f(x) K x : Here te secod subscrit i z i deotes its deedece o te samle size troug te badwidt term : Suc doubly-subscrited radom variables, were te rage of te rst subscrit is bouded above by te secod, are kow as triagular arrays, ad classical limit teorems must be modi ed to accout for te cagig distributio of te observatios as te samle size icreases. To demostrate asymtotic ormality of ^f(x) a coveiet cetral limit teorem is Liauov s Cetral Limit Teorem for Triagular Arrays. It statest tat, if te scalar radom variable z i is ideedetly (but ot ecessarily idetically) distributed wit variace V ar(z i ) i ad r-t absolute cetral momet E[jz i E(z i )j r ] i < for some r > ad if ( P i i) r P i i 0 as 0 (kow as te Liauov coditio), te is asymtotically ormal, z X i z i z E[z ] V ar(z ) d N (0 ): Alyig tis teorem to ^f(x) we obtai te variace of te z i terms as i V ar(z i ) x V ar K f(x) [K (u)] du + o from our earlier calculatios settig r 3 we get a uer boud for te tird cetral momet of z i as i E[jz i E(z i )j 3 ] 8E[jz i j 3 ] " # x 8E K 3 8f(x) jk (u)j 3 du + o 7

8 were te iequality uses te exasio E[jz i E(z i )j 3 ] E[(jz i j + je(z i )j) 3 ] E[jz i j 3 ] + 3E[jz i j ] je(z i )j + 3E[jz i j] je(z i )j + (E(z i )) 3 ad te last equality makes te same cage-of-variables calculatios as for te variace of z i : Tus, for tis roblem te Liauov coditio is ( P i i) r 8f(x) R jk (u)j 3 3 du + o P i i f(x) R [K (u)] du + o O 3 3 O O(( ) 6 ) 0 if te same coditio imosed to esure tat V ar( ^f(x)) 0 as : Uder tis coditio, te, ^f(x) E[ q ^f(x)] V ar( ^f(x)) d N (0 ) or, subtitutig te exressio for V ar( ^f(x)) O(( ) ) ad collectig terms, ( ^f(x) E[ ^f(x)]) d N (0 f(x) [K (u)] du): Tis is almost, but ot quite, i te form of te usual exressio for a asymtotic distributio of a estimator te di erece is tat te exectatio of te estimator E[ ^f(x)] rater ta te true value f(x) is subtracted from ^f(x) i tis exressio. Writig ( ^f(x) f(x)) ( ^f(x) E[ ^f(x)]) + (E[ ^f(x)] f(x)) te asymtotic ormality of ^f(x) aroud f(x) will require te secod, bias term to coverge to a costat. Isertig te exressio for te bias, (E[ ^f(x)] f(x)) tr O( uu 0 K(u)du + o( ) 8

9 If te badwidt takes te form c (+4) so tat it coverges to zero at te otimal rate, te (E[ ^f(x)] f(x)) f(x) 0 (x) uu 0 K(u)du ad ( ^f(x) f(x)) d N ((x) f(x) [K (u)] du): Here te aromatig ormal distributio for ^f(x) would be cetered at f(x) + (x) ot f(x) ad costructio of te usual co dece regios or test statistics would be comlicated by te fact tat (x) deeds uo te ukow secod derivative of f(x): te If te badwidt teds to zero faster ta te otimal rate, i.e., o (+4) (E[ ^f(x)] f(x)) 0 ad te bias term vaises from te asymtotic distributio, ( ^f(x) f(x)) d N (0 f(x) [K (u)] du): Ofte i ractice tis sort of udersmootig wic imlies te bias of f(x) is egligible relative to te variace is assumed, ad co dece itervals of te form " s s f(x) f(x) :96 ^f(x) [K (u)] du f(x) + :96 ^f(x) [K (u)] du # are reorted, toug it is best to view te claimed 95% asymtotic coverage rate wit some sketicism. Note tat if te badwidt teds to zero slower ta te otimal rate, e.g., o > + 4 te te bias of ^f(x) domiates its stadard deviatio, ad te ormalized di erece ( ^f(x) f(x)) diverges. 9

10 Rescalig for Multivariate Kerels Derivatives of te Kerel Desity ad Regressio Estimators Higer-Order (Bias-Reducig) Kerels 0