Nonparametric Conditional Density Estimation

Size: px

Start display at page:

Download "Nonparametric Conditional Density Estimation"

Morgan Snow
5 years ago
Views:

1 Noparametric Coditioal Desity Estimatio Bruce E Hase Uiversity of Wiscosi wwwsscwiscedu/~bhase November 004 Prelimiary ad Icomplete Research supported by the Natioal Sciece Foudatio Departmet of Ecoomics, 1180 Observatory Drive, Uiversity of Wiscosi, Madiso, WI

2 1 Itroductio Coditioal desity fuctios are a useful way to display ucertaity This paper ivestigates oparametric kerel methods for their estimatio The stadard estimator is the ratio of the joit desity estimate to the margial desity estimate Our proposal is to istead use a two-step estimator, where the first step cosists of estimatio of the coditioal mea, ad the secod step cosists of estimatig the coditioal desity of the regressio error If most of the depedece is captured by the coditioal mea, the secod step will require less smoothig, thereby reducig estimatio variace Coditioal desity estimatio was itroduced by Roseblatt (1969) A bias correctio was proposed by Hydma, Bashtayk ad Gruwald (1996) Fa, Yao ad Tog (1996) proposed a direct estimator based o local polyomial estimatio; see also Sectio 65 of Fa ad Yao (003) Badwidth selectio rules have bee proposed by Bashtayk ad Hydma (001), Fa ad Yim (004), ad Hall, Racie ad Li (004) The related problem of coditioal distributio estimatio is examied i Hall, Wolff ad Yao (1999) Other papers have used coditioal desity estimates as a iput to other problems, icludig Robiso (1991), Tjostheim (1994), Poloik ad Yao (000) ad Hydma ad Yao (00) Our two-step coditioal desity estimator is partially motivated by the two-step coditioal variace estimator of Fa ad Yao (1998) They showed that two-step estimatio is asymptotically efficiet sice the first-step coditioal mea estimate does ot affect the asymptotic distributio of the secodstep variace estimator We show here that this property also applies to coditioal desity estimatio Ouraalysisiscofied to the case of a real-valued coditioig variable The geeralizatio to the case of vector-valued coditioig variables should be straightforward, so log as the coditioig set for the coditioal mea ad coditioal desity are idetical However, if the coditioal desity of the regressio error has a reduced coditioig set relative to the coditioal mea, the aalysis chages (For example, if the coditioal mea has two variables ad the coditioal error desity oly oe) I this case the secod-step estimator may ot be asymptotically idepedet of the first-step More importatly, it appears that the two-step estimator may achieve a improved covergece rate relative to the covetioal direct estimator This aalysis is more ivolved ad remais to be completed Our two-step estimator could also be geeralized to three steps, where a itermediate step estimates the coditioal variace We expect the qualitative aalysis to be similar, ad cojecture that there will be further improvemets i estimatio efficiecy This work remais to be completed Furthermore, our discussio is based o local average estimates Alteratively, the mea, variace, or desity ca be estimated usig local liear estimators This should be explored, as local liear estimators have better bias properties tha local averages (ad thus have improved efficiecy) whe there is o-trivial depedece Other tha chages i the bias expressios, however, we expect that o importat chages will arise i the theory Agai, this work remais to be completed The orgaizatio of the remaider of the paper is as follows Sectio itroduces the framework, Sectio 3 the oe-step estimator, Sectio 4 the ew two-step estimator, ad Sectio 5 compares their asymptotic biases Sectio 6 discusses cross-validatio for badwidth selectio Sectio 7 presets simulatio evidece, ad Sectio 8 a applicatio to US GDP Proofs are preseted i the Appedix 1

3 Framework The observables {Y i,x i } i R R are strictly statioary ad strog mixig Let f(y, x) ad f (y x) deote the joit ad coditioal desity fuctios, ad let f(x) deote the margial desity of X i The goal is estimatio of f (y x) Our estimators will be based o kerel regressio Let K(x) :R R deote a bouded symmetric kerel fuctio ad set σ K = R R u K(u)du ad R(K) = R R K(u) du For a badwidth h let K h (u) = h 1 K (u/h) Defie the derivatives 3 Oe-Step Estimator f (r) f (r) (x) = r x r f (x) (s) (y x) = r+s y r f (y x) xs Let h 1 ad h be badwidths Stadard kerel estimators of f(y, x), f(x) ad f (y x) are f(y, x) = 1 f(x) = 1 K h (x X i ) K h1 (y Y i ) K h (x X i ) ad f (y x) = f(y, x) f(x) = K h (x X i ) K h1 (y Y i ) K h (x X i ) Asymptotic approximatios show that it it optimal for estimatio of f (y x) to set h 1 = c 1 1/6 ad h = c 1/6 for c 1 > 0 ad c > 0 Uder stadard regularity coditios the coditioal desity estimator has the asymptotic distributio 1/3 f (y x) f (y x) d N θ 1,σ 1 where ad θ 1 = σ K c c 1 c 1f () (y x)+c f () (y x)+c f (1) (y x) f (1) (x) σ = R(K) f (y x) c 1 c f (x) Observe that the rate of covergece is O 1/3, the same as for bivariate desity estimatio It slower tha the O /5 rate obtaied for uivariate desity estimatio ad bivariate regressio

4 4 Two-Step Estimator Defie the coditioal mea m(x) =E (Y i X i = x) so that Y i = m (X i )+e i ad e i is a regressio error Lettig g (e x) deote the coditioal desity of e i give X i = x, we have the equivalece f (y x) =g (y m(x) x) From this equatio we ca see that a alterative method for estimatio of f is through estimatio of g ad m Let b 0, ad b be badwidths The Nadaraya-Watso estimator of m(x) is ˆm (x) = K b0 (x X i ) Y i K b0 (x X i ) with residuals ê i = Y i ˆm (X i ) A secod-stage estimator of g is ĝ (e x) = K b (x X i ) K b1 (e ê i ) K b (x X i ) Together we obtai the two-step estimator ˆf (y x) = ĝ (y ˆm(x) x) K b (x X i ) K b1 (y ˆm(x) ê i ) Assume that b 0 = a 0 1/5, = a 1 1/6 ad b = a 1/6 = K b (x X i ) Theorem 1 1/3 ˆf (y x) f (y x) θ d N θ,σ 3

5 where with e = y m(x) ad θ = σ a a 1 a 1g () (e x)+a g () (e x)+a g (1) (e x) f (1) (x) σ = R(K) f (y x) a 1 a f (x) This result states that the asymptotic distributio of the two-step estimator is uaffected by the first estimatio step The badwidth b 0 does ot eter the first-order approximatio, ad the distributio isthesameaswhethemeam(x) ad errors e i are kow without estimatio This occurs because the coditioal mea estimator ˆm(x) coverges at the faster rate of O /5 5 Bias Compariso Note that the scaled ˆf ad f have differig biases We ca compare the latter by observig that f () (y x) =g () (e x) f (1) (y x) =g (1) (e x) g (1) (e x) m (1) (x) f () (y x) =g () (e x) g (1) (e x) m () (x)+g () (e x) m (1) (x) Therefore µ θ 1 = σ (K) c 1 g c 1 c () (e x) g (1) (e x) m () (x)+g () (e x) m (1) (x) + σ (K) c 1 c c g () (e x)+g (1) (e x) g (1) (e x) m (1) (x) f (1) (x) Uless m (1) (x) =0,θ 1 has more compoets tha θ, ad will typically be larger (for equal badwidths) Thus ˆf has lower bias tha f, eablig the selectio of a larger badwidth scale a for ˆf tha b for f, reducig variace ad mea-squared-error 6 Badwidth Selectio Fa ad Yim (004) ad Hall, Racie ad Li (004) have proposed a cross-validatio method appropriate for oparametric coditioal desity estimators I this sectio we describe this method ad its applicatio to our estimators For a estimator f (y x) of f (y x) defie the itegrated squared error I = = Z Z f (y x) f (y x) f(x)dydx Z Z Z Z Z Z f (y x) f(x)dydx f (y x) f (y x) f(x)dydx + = I 1 I + I 3 f (y x) f(x)dydx 4

6 Note that I 3 does ot deped o the badwidths ad is thus irrelvat Ideally, we would like to pick the badwidths to miimize I, but this is ifeasible as the fuctio I is ukow Cross-validatio replaces it with a estimate based o the leave-oe-out priciple Let f i (y x) deote the estimator f (y X) with observatio i omitted The cross-validatio estimators of I 1 ad I are Î 1 = 1 Î = 1 We the defie the cross-validatio fuctio as Z f i (y X i ) dy f i (Y i X i ) Î = Î1 Î The cross-validated badwidths are those which joitly miimize Î For the oe-step estimator these equal compoets equal Î = 1 X K h (X i X j ) K h1 (Y i Y j ) j6=i X K h (X i X j ) j6=i ad Î 1 = 1 = 1 P P j6=i k6=i K h (X i X j ) K h (X i X k ) R K h1 (y Y j ) K h1 (y Y k ) dy P j6=i K h (X i X j ) P P j6=i k6=i K h (X i X j ) K h (X i X k ) K h 1 (Y k Y j ) P, j6=i K h (X i X j ) the secod equality whe K(u) =φ(u), the Gaussia kerel For the two-step estimator we first defie the leave-oe-out Nadaraya-Watso regressio estimator ˆm i (x) ad leave-oe-out residual ê i = Y i ˆm i (X i ) The the estimators Î1 ad Î take the form Î 1 = 1 P j6=i Pk6=i K b (X i X j ) K b (X i X k ) K ê k ê j P j6=i K b (X i X j ) ad Î = 1 X K b (X i X j ) K b1 ê i ê j j6=i X K b (X i X j ) j6=i 5

7 These deped o the badwidth b 0 through the residual ê i Alteratively for the two-step estimator the badwidths may be selected separately for each step Specifically, the badwidth b 0 may be selected by least-squares cross-validatio, ad the (,b ) by usig the method for the oe-step estimator, with the Nadaraya-Watso residual ê i replacig Y i 7 Simulatio Evidece The performace of the oparametric estimators were compared i a simple stochastic settig The data are geerated by the process x i N(0, 1) y i x i N µ β 1 x i, 1+β x i 1+β 1000 samples of size = 100 were geerated We vary β 1 amog 01, 1, ad, ad β amog 01 ad 1 O each sample, the oe-step estimator f (y x) ad two-step estimator ˆf (y x) were calculated, usig a Gaussia kerel We measure accuracy by mea itegrated squared error I( f) = 100 E Z Z f (y x) f (y x) f(x)dydx where the itegrals are approximated by a grid o (y, x) The estimators deped critically o the badwidths h =(h 1,h ) ad b =(b 0,,b ) For our first compariso, we use the ifeasible oracle badwidth This is the badwidth which miimizes the fiite sample MISE This eables a compariso of the estimatio methods free of depedece o badwidth selectio methods For the two estimators Table 1 reports the MISE ad the oracle badwidths The results are as expected For the case of small coditioal mea effect (β 1 =01), the the two estimators perform similarly i terms of MISE However, if the coditioal mea effect is o-trivial, the the two-step estimator ˆf has much smaller MISE The reductio i MISE is as much as 50% Table 1 Mea Itegrated Squared Error Usig Oracle Badwidth = 100 β 1 β I( f) I( ˆf) h 1 h b 0 b

8 For our secod compariso, we use data-depedet badwidths For the oe-step estimator f we use the cross-validated badwidth For the two-step estimator ˆf we use sequetial badwidths The badwidth ˆb 0 is selected by least-squares cross-validatio for the mea, ad (ˆ, ˆb ) are selected by coditioal desity cross-validatio usig the estimated residuals Table reports the MISE for the two estimators It also reports the media data-depedet badwidths The qualitative results are similar to those for the optimal badwidths, with the otable chage that the improvemet of the two-step estimator relative to the oe-step estimator has bee reduced For the cases with small coditioal mea effect (β 1 =01) the MISE is eve somewhat higher for ˆf tha for f, but i the other cases ˆf has much lower MISE This suggests that further ivestigatio ito badwidth selectio may yield further improvemets Table Mea Itegrated Squared Error Usig Data-Depedet Badwidths = 100 β 1 β I( f) I( ˆf) ĥ 1 ĥ ˆb0 ˆb1 ˆb Applicatio We illustrate the method with a time-series applicatio Let Y t deote US quarterly real GDP ad let y t = 100(l(Y t ) l(y t 1 )) deotes its growth rate We are iterested i estimatio of the oe-step ahead coditioal desity f(y t y t 1 ) Due to strog evidece of a shift i variace i the early 1980s, we use the sample period 1983:1-004:3 which results i a small sample First, for a baselie we take the liear Gaussia model, for which least-squares yield the estimate ˆf 0 (y t y t 1 )=φ 05 (y t 5 4y t 1 ) Secod, we estimate f(y t y t 1 ) usig the oe-step estimator with cross-validated badwidth, ad let this estimate be deoted as ˆf 1 (y t y t 1 ) The cross-validated badwidths are h 1 = 6 ad h = 0 Third, we estimate the coditioal desity usig the two-step estimator with sequetial crossvalidated badwidth, ad deote this estimator as ˆf (y t y t 1 ) The cross-validated badwidths are h 0 = 15, h 1 = 1 ad h = 59 Thelattervalueofh meas that cross-validatio elimiates the 7

9 coditioal smoothig i the secod step, so the estimated coditioal desity oly depeds o y t 1 through the estimated coditioal mea This is ot surprisig due to our applicatio to a small sample This also highlights a importat distictio betwee the oe-step ad two-step estimators, as the former does ot have this flexibility Figures 1 through 4 display the three desity estimates as a fuctio of y t for four fixed values of y t 1 I geeral, the three estimators differ from oe aother I particular, the iefficiet oe-step estimator appears to be mis-cetered ad over-dispersed i Figure 1 (y t 1 = ), adiallcaseshasa thicker right tail tha the two-step estimator 8

10 Refereces [1] Bashtayk, DM ad Rob J Hydma (001): Badwidth selectio for kerel coditioal desity estimatio, Computatioal Statistics ad Data Aalysis, 36, [] Fa, Jiaqig ad Qiwei Yao (1998): Efficiet estimatio of coditioal variace fuctios i stochastic regressio, Biometrika, 85, [3] Fa, Jiaqig ad Qiwei Yao (003): Noliear Time Series: Noparametric ad Parametric Methods New York: Spriger-Verlag [4] Fa, Jiaqig, Qiwei Yao, ad Howell Tog (1996): Estimatio of coditioal desities ad sesitivity measures i oliear dyamical systems, Biometrika, 83, [5] Fa, Jiaqig ad Tsz Ho Yim (004): A cross-validatio method for estimatig coditioal desities, Biometrika, forthcomig [6] Hall, Peter, Jeff Racie ad Qi Li (004): Cross-validatio ad the estimatio of coditioal probability desities, workig paper [7] Hall, Peter, RCL Wolff ad Qiwei Yao (1999): Methods for estimatig a coditioal distributio fuctio, Joural of the America Statistical Associatio, 94, [8] Hase, Bruce E (004): Uiform Covergece Rates for Kerel Estimatio, workig paper [9] Hydma, Rob J ad Qiwei Yao (00): Noparametric estimatio ad symmetry tests for coditioal desity fuctios, Noparametric Statistics, 14, [10] Hydma, Rob J, DM Bashtayk ad GK Gruwald (1996): Estimatig ad visualizig coditioal desities, Joural of Computatioal ad Graphical Statistics, 5, [11] Poloik, W ad Qiwei Yao (000): Coditioal miimum volume preditive regios for stochastic processes, Joural of the America Statistical Associatio, 95, [1] Robiso, Peter M (1991): Cosistet oparametric etropy-based testig, Review of Ecoomic Studies, 58, [13] Roseblatt, M (1969): Coditioal probability desity ad regressio estimates, i Multivariate Aalysis II, Ed PR Krishaiah, pp 5-31 New York: Academic Press [14] Tjostheim, D (1994): No-liear time series: A selective review, Scadiavia Joural of Statistics, 1,

11 9 Appedix The proofs cotaied here are icomplete sketches, ad omit regularity coditios We first state a result from Hase (004) Lemma 1 Let Uder regularity coditios Ĝ(x, z) = 1 h 1 h sup x R,z R µ µ x Xi z Zi ψ (Y i,x i,z i ) G 1 G h 1 h Ã µ! log 1/ Ĝ(x, z) EĜ(x, z) = Op h 1 h Lemma Uiformly for f(x) δ =(log) 1/, if b 0 = c 0 1/5, ˆm(x) m(x) =f(x) 1 1 K b0 (x X i ) e i b 0σ (K)f(x) 1 f (1) (x)m (1) (x)+o p (log ) 3/5 Proof We start with the decompositio ˆm(x) m(x) = f(x) 1 1 K b0 (x X i ) e i (1) +f(x) 1 1 K b0 (x X i )(m(x i ) m(x)) Ã! K b0 (x X i ) f(x) 1 1 K b0 (x X i )(e i + m (X i ) m(x)) We ow examie the terms o the right-had-side First, it is well kow that EK b0 (x X i )=f(x)+o(b 0)=f(x)+O /5 Combied with Lemma 1, uiformly i x R 1 By a Taylor expasio, uiformly for f(x) δ, Ã 1 Ã µlog! 1/ K b0 (x X i ) = EK b0 (x X i )+O p b 0 /5 = f(x)+o p (log ) 1/! 1 /5 K b0 (x X i ) = f(x) 1 + O p δ (log ) 1/ 10

12 Secod, ote Z E (K b0 (x X i )(x X i )) = K b0 (x u)(x u) f(u)du RZ = b 0 K (u) uf(x b 0 u)du R = b 0f (1) (x)σ (K)+O(b 4 0) The by a Taylor expasio ad Lemma 1 1 K b0 (x X i )(m(x i ) m(x)) ' 1 uiformly i x Similarly, by Lemma 1 Thus (1) equal 1 K b0 (x X i )(x X i ) m (1) (x) = b 0σ (K)f (1) (x)m (1) (x)+o p (log ) 1/ 3/5 K b0 (x X i ) e i = O p (log ) 1/ /5 ˆm(x) m(x) = f(x) 1 1 K b0 (x X i ) e i b 0σ (K)f(x) 1 f (1) (x)m (1) (x) 3/5 4/5 +f(x) 1 O p (log ) 1/ + O p δ (log ) as claimed = f(x) 1 1 K b0 (x X i ) e i b 0σ (K)f(x) 1 f (1) (x)m (1) (x)+o p (log ) 3/5 Defie Lemma 3 For b = c 1/6 ĝ (e x) = K b (x X i ) K b1 (e e i ) K b (x X i ) ĝ (e x) ĝ (e x) =O p ((log ) 1/ /5 ) 11

13 Proof Observethat ĝ (e x) ĝ (e x) = B 1 A = 1 B = 1 A K b (x X i )(K b1 (e ê i ) K b1 (e e i )) K b (x X i ) Sice EK b (x X i )=f(x)+o(b )=f(x)+o 1/3 the usig Lemma 1, uiformly i x R B = f(x)+o 1/3 ad by a Taylor expasio, uiformly for f(x) δ, B 1 f(x) 1 = O p Ã µlog! 1/ + O p = f(x)+o p 1/3 b 1/3 δ Next, to decompose A, first observe that by a Taylor expasio K b1 (e ê i ) K b1 (e e i ) ' K (1) (e e i )(e i ê i ) Secod, by Lemma, uiformly i i ˆm (X i ) m (X i )=f (X i ) 1 1 b 0 µ Xi X j K j=1 b 0 = 1 µ e b K (1) ei 1 (ˆm (X i ) m (X i )) e j b 0σ (K)f(X i ) 1 f (1) (X i )m (1) (X i )+O p (log ) 3/5 1

14 Together A ' 1 K b (x X i ) 1 µ e b K (1) ei 1 f (X i ) 1 1 µ Xi X j K e j b 0σ (K)f(X i ) 1 f (1) (X i )m (1) (X i )+O p (log 3/5 ) b 0 b j=1 0 1 X µ x Xi = b 0 b 1 b + K(0) b 0 b 1 b 1 i6=j K b µ x Xi K b b 0 σ (K) µ x Xi b 1 b K b + 1 µ x Xi b 1 b K b = A 1 + A + A 3 + A 4 K (1) µ e ei K (1) µ e ei K (1) µ e ei K (1) µ e ei K µ Xi X j b 0 f (X i ) 1 e i f (X i ) 1 e j f(x i ) 1 f (1) (X i )m (1) (X i ) O p (log ) 3/5 say We ow examie the four terms o the right-had-side, i reverse First, observe that E 1 µ b 1 b K µ x Xi b Thus usig Lemma 1 A 4 = µ e K (1) ei Ã g (1) (e x) f(x)+o = O p (log ) 3/5 Z Z µ µ 1 x u e v = b 1 b K K (1) g (v u) f(u)dvdu b = 1 Z Z K (u) K (1) (v) g (e v x b u) f (x b u) dvdu Z = K (1) (v) vg (1) (e x) f(x)dv + O b 1 + O b = g (1) (e x) f(x)+o 1/3 1/3 Ã µ + 1!! log 1/ O p O p (log 3/5 ) b 13

15 Secod, µ µ E b 0 x Xi e b 1 b K K (1) ei f(x i ) 1 f (1) (X i )m (1) (X i ) b Z Z µ µ = b 0 x u e v b 1 b K K (1) f (1) (u)m (1) (u)g (v u) dvdu b = b 0f (1) (x)m (1) (x)g (1) (e x)+o /3 = O( /5 ) so by Lemma 1 Third, similarly, Thus ad A 3 = O( /5 )+ b 0 O p E 1 b 0 b 1 b K µ x Xi b Ã µ! log 1/ = O p ( /5 ) b µ µ e K (1) ei f (X i ) 1 1 e i = O b 0 µ 1 EA = O = O 5/6 b 0 A = O 5/6 Ã µ + 1! log 1/ O p = O 5/6 b 0 b Fially, we tur to A 1 Note that EA 1 =0 A tedious argumet [to be completed] bouds E(A 1 ) Together, we have A = O p ( /5 ) ad hece ĝ (e x) ĝ (e x) = f(x) 1 + O p δ 1/3 O p ( /5 ) = O p ((log ) 1/ /5 ) ProofofTheorem1 By Lemma 3, a Taylor expasio ˆf (y x) = ĝ (y ˆm(x) x) = ĝ (y ˆm(x) x)+o p ((log ) 1/ /5 ) By a Taylor expasio ĝ (y ˆm(x) x) ĝ (y m(x) x) sup (e x) ˆm(x) m(x) + O p((log ) 1/ /5 ) e,x = O p ((log ) 1/ /5 ) 14

16 Hece ˆf (y x) =ĝ (e x)+o p ((log ) 1/ /5 ) with e = y m(x) ad therefore 1/3 ˆf (y x) f (y x) = 1/3 (ĝ (e x) g (e x) θ )+O p ((log ) 1/ 1/3 /5 ) d N θ,σ as the asymptotic distributio of ĝ is well kow 15

Study the bias (due to the nite dimensional approximation) and variance of the estimators

Study the bias (due to the nite dimensional approximation) and variance of the estimators 2 Series Methods 2. Geeral Approach A model has parameters (; ) where is ite-dimesioal ad is oparametric. (Sometimes, there is o :) We will focus o regressio. The fuctio is approximated by a series a ite