Tying the straps tighter for generalized linear models

Transcription

1 Tying the straps tighter for generalized linear models Ya'acov Ritov Weining Wang Wolfgang K. Härdle Ladislaus von Bortkiewicz Chair of Statistics C.A.S.E. - Center for Applied Statistics and Economics Humboldt-Universität zu Berlin Department of Statistics The Hebrew University of Jerusalem

2 Motivation 1-1 Motivation Bootstrap improves nite sample performance Regression Problem: resample from residuals For nonlinear functionals (e.g. sup), little improvement def l n (x) l(x) = sup x l n (x) l(x), where l(x) is a mean or quantile curve, and l n (x) is a quantile or mean smoother

3 Motivation 1-2 Challenges L( l h - l ) (log n) -1 exp{-exp(-x)} Gumbel, Emil J. 43 Emil Julius Gumbel BBI (log n) -1 L * ( l * h - l g )

4 Motivation 1-3 Song et. al. (2012) 90% Quantile of Y Univariate X Figure 1: The real 0.9 quantile curve, 0.9 quantile estimate with corresponding 95% uniform condence band from asymptotic theory and con- dence band from bootstrapping.

5 Motivation 1-4 Theorem (Song et. al. (2010)) An approximate (1 α) 100% condence band over [0, 1] is l h (t) ± (nh) 1/2 {p(1 p)/ˆf X (t)} 1/2ˆf 1 {l h (t) t} {d n + c(α)(2δ log n) 1/2 } {λ(k)} 1/2, (1) where c(α) = log 2 log log(1 α) and ˆf X (t), ˆf {l h (t) t} are consistent estimates for f X (t), f {l(t) t}. n Cov. Prob (0.642) (0.742) (0.862) Table 1: Simulated coverage probabilities based on asymptotics ( bootstrap methods).

6 Motivation 1-5 Bootstrap Improvement Bootstrap improvements for the quantile smoother have been shown Quantile estimator is of Bounded Inuence M- smoothers or a general QMLE may be expected to have same performance? How to handle high dimensional objects?

7 Motivation 1-6 Opportunities Extend this to x R d and improve band precision? Hall (1991): bootstrap improvement Hahn (1995): consistency of bootstraping cdf Horowitz (1998): bootstrap (pointwise) for median Additive Models: Horowitz (2001), Horowitz and Lee (2005) and Stone(1985)

8 How to tie the straps tighter?

9 Outline 1. Motivation 2. Bootstrap Condence Bands 3. Bootstrap Condence Bands for Additive Models 4. Monte Carlo Study 5. Applications

10 Bootstrap condence bands 2-1 Regression with a general loss function {(X i, Y i )} n i=1 i.i.d. rv's, x J = (a, b) Suppose Y i = l(x i ) + ε i, ε i F ε Xi ( ). Both l & F are smooth. Suppose l(x) = arg min θ E (Y X =x) ρ(y θ), (2) where ρ(.) is a loss function of Hampel/Huber type or more generally a negative (quasi) log likelihood.

11 Bootstrap condence bands 2-2 Robust Statistics For ρ(.) is a trimmed mean (Tukey's biweight), { x ρ(x) = 2, x k, k 2, x > k (3) or a form of Winsorized mean (Huber): { ρ(x) = x 2 /2, x k, k 2 /2 + k x, x > k. }. (4)

12 Bootstrap condence bands 2-3 The Bootstrap Couple {U i } n i=1 : i.i.d. uniform U[0, 1] rv's Bootstrap sample Y i = l g (X i ) + ˆF 1 (ε i X i ) (U i), i = 1,..., n Couple with the true conditional distribution: Y # i = l(x i ) + F 1 (ε i X i ) (U i), i = 1,..., n. Given X 1,..., X n : Y 1,..., Y n and Y #,..., 1 Y # n distributed. are equally

13 Bootstrap condence bands 2-4 Bootstrapping Approximation Rate Theorem If assumptions (A1)(A3) hold, then sup l x J h (x) l g(x) l h (x) + l(x) = O p (δ n ) = O p (h 2 Γ n ), with Γ n a slowly varying sequence (a sequence a n is slowly varying if n α a n 0 for any α > 0). Bootstrap improves the rate of convergence

14 Bootstrap condence bands 2-5 Sketch of the Proof The basic elements: smoothness of F Y X ( ) and bounded inuence of ρ(.) max ε i ε i = O p (h 2 Γ n ) i max ψ(ε ) ψ(ε ) = O p (h 2 Γ n ) i max l g (X i ) l g (X j ) {l(x i ) l(x j )} X i X j ch = O p (h 2 Γ n ) ˆF 1 (u X i ) F 1 (u X i ) h 2 Γ n, u B. E ˆFε x=xi ψ(ε i ) = 0 = E Fε x=xi ψ(ε ) (5)

15 Bootstrap condence bands 2-6 G n (θ, X i ) G n(θ, X i ) T n,1 (X i ) n def = n 1 W h,j (X i )[ψ{ε j θ + ˆl g (X j )}] j=1 = n 1 n j=1 W h,j (X i ){ψ(y j θ)} n def = n 1 W h,j (X i )[ψ{ε j θ + l(x j)}] j=1 = n 1 n j=1 W h,j (X i ){ψ(y j θ)} def = G n {ˆl g (.), X i } Gn{l(.), X i }

16 Bootstrap condence bands 2-7 E F (ε Xi )T n,1 (X i ) = 0 n Var (ε Xi )T n,1 (X i ) = n 2 W 2 h,j (X i) Var{ψ(ε j ) ψ(ε i )} j=1 = O p (n 1 h 3 ) ˆl h,g (X i) ˆl g (X i ) = G n {ˆl g (X i ), X i } G n {ˆl h,g (X i), X i } + O p(h 2 ), (6) ˆlh (X i ) l(x i ) = G n{l(x i ), X i } G n {ˆl h (X i ), X i } + O p(h 2 ). (7)

17 Bootstrap condence bands 2-8 Why Oversmoothing? Take care of bias with tuning parameter: g Härdle and Marron (1991), let b h (x) def = E l # h (x) l(x) ˆb h,g (x) def = E l h (x) l g(x) [ } 2 ] Investigate E {ˆbh,g (x) b h (x) X1,..., X n. How fast it converges to 0?

18 Bootstrap condence bands 2-9 Oversmoothing Theorem Under some assumptions, for any x J [ } 2 ] E {ˆbh,g (x) b h (x) X1,..., X n h 4 {O p (g 4 ) + O p (n 1 g 5 )} To minimize RHS, g = O(n 1/9 ), g h, where h = O(n 1/5 )

19 Bootstrap Condence Bands in Additive Models 3-1 The Multivariate Case m(x i ) = d m j (x i,j ), (8) j=1 Estimate the additive model via a basis function approach: m j (x i,j ) L j +1 l=1 a l,j g l (x i,j ), ψ(x i,j )s are B-splines, e.g. linear B-splines. Hx l + 1 (l 1)H 1 x lh 1 ψ l (x) = l + 1 Hx lh 1 x (l + 1)H 1 0 otherwise

20 Bootstrap Condence Bands in Additive Models 3-2 Bootstrap Couple Quantile Z i = { 1 with prob τ 1 with prob 1 τ, where τ = 1/2 is for symmetric error distribution. The bootstrap couple ε (the bootstrap residuals) and ε (the theoretical couple) are: ε i = Z i ˆε i (9) ε i = Z i η i (10)

21 Bootstrap Condence Bands in Additive Models 3-3 Bootstrap F i,s (t) def = P( ε i t sε i > 0), i = 1,..., n, s {1, 1}, (11) def η i = F 1 i,z i {F i,sgn(εi )( ε i )}, i = 1,..., n. (12) Recall F Y X =xi {l(x i )} = τ and F ε X =xi (0) = τ. F i,sgn(εi )( ε i ) standard uniform F i,+1 (t) = F i(t) 1 + τ τ, F i, 1 (t) = 1 τ F i( t). 1 τ L(ε i ) = L(ε i). (13) Go to details

22 Bootstrap Condence Bands in Additive Models 3-4 Uniform Consistency Theorem Let assumptions A.1- A.9 be fullled, then sup ( ˆm j m j )(x) {( ˆm j ˆm j )(x)} = Op(h 2 Γ n ), x Go to conditions

23 Monte Carlo study 4-1 How to Bootstrap? Simulate {X i, Y i } n i=1, f (x, y) = g{y sin(πx)}1(x [0, 1]) (14) g(u) = 9ϕ(u)/10 + ϕ(u/9)/90 (15) Compute the smoother (Tukey biweight) ˆl h (x), ˆε i = Y i ˆl h (X i ) Compute the conditional edf: n i=1 ˆF ε Xi =x(t) = K h(x X i )1(ˆε i t) n i=1 K h(x X i ) with Gaussian kernel h = K h (u) = ( 2π) 1 exp{ u 2 /2h}/h,

24 Monte Carlo study 4-2 For each i = 1,..., n, generate random variable ε ˆF (t x), i = 1,..., n, where n is the bootstrap sample size: Y i,i = ˆl g (X i ) + ε i,i, with g = 0.2. For each sample X i, Y i, compute ˆl h (.) and the random variable d i def = sup { ˆf X (x) E y x {ψ (ε x J i )}/ E y x {ψ 2 (ε i )} ˆl h (x) ˆl g (x) } Calculate the 1 α quantile dα of d 1,..., d n. Construct the bootstrap uniform band centered around m h (x) ˆlh (x) ± [ ˆf X (x) E y x {ψ (ε i )}/ E y x {ψ 2 (ε i )}] 1 dα

25 Monte Carlo study Figure 2: Plot of true curve (grey), BI estimation and bands (blue), local polynomial estimation (black), bootstrap band (red)

26 Monte Carlo study 4-4 n Cov. Prob. Area (0.98) 1.23(2.51) (0.98) 0.89(1.95) (0.96) 0.78(1.32) Table 2: Simulated coverage probabilities areas of nominal asymptotic (bootstrap) 95% condence bands with 100 repetition.

27 Monte Carlo study 4-5 Bootstrap for Additive Models Simulate {X i, Y i } n i=1. The variable X i = (x 1i, x 2i, x 3i, x 4i ) U( 2.5, 2.5), m 1 (x 1 ) = sin(πx 1 ), m 2 (x 2 ) = Φ(3x 2 ), m 3 (x 3 ) = x 3 3, m 4 (x 4 ) = x 4 4, and ε i is as in (15). Compute ˆm 1 (x 1 ), ˆm 2 (x 2 ), ˆm 3 (x 3 ), ˆm 4 (x 4 ) and ˆε i = Y i 4 j=1 ˆm l(x i,j ). For each i = 1,..., n, generate random variable ε i, i = 1,..., n as in (9), where n is the bootstrap sample size: Y i,i = 4 ˆm l (x i,j ) + ε i,i. j=1

28 Monte Carlo study 4-6 For each sample (x 1i, x 2i, x 3i, x 4i, Y i ), compute m j (.) and the random variable d i def = sup { ˆfxj (x j) E y x {ψ (ε x J i )}/ E y x {ψ 2 (ε i )} ˆm j (x) ˆm j (x) } Calculate the 1 α quantile d α of d 1,..., d n. Construct the bootstrap uniform band centered around ˆm j (x j ) ˆm j (x j ) ± [ ˆfxj (x j) E {ψ y xj (ε i )}/ E y xj {ψ 2 (ε i )}] 1 dα

29 Monte Carlo study Figure 3: Plot of true curve (dark blue), robust estimation and bands (cyran), bootstrap band (red dotted)

30 Monte Carlo study 4-8 n Cov. Prob. Area , 0.98, 0.83, , 5.37, 5.44, , 0.95, 0.93, , 4.74, 4.54, , 0.95, 0.96, , 3.63, 3.76, 3.70 Table 3: Simulated coverage probabilities areas of nominal asymptotic (bootstrap) 95% condence bands with 100 repetition, 100 bootstrap sample. n Cov. Prob. Area , 0.94, 0.85, , 5.07, 5.04, , 0.95, 0.86, , 3.84, 3.85, , 0.90, 0.92, , 3.25, 3.11, 3.03 Table 4: Simulated coverage probabilities areas of nominal asymptotic (bootstrap) 90% condence bands with 100 repetition, 100 bootstrap sample.

31 Applications 5-1 Firm expenses analysis Joel L. Horowitz and Sokbae Lee (2005) Whether a concentrated shareholding is associated with lower expenditure on activities The dependent variable Y : general sales and administrative expenses deated by sales (denoted by MH5) The covariates: ownership concentration (denoted by TOPTEN, cumulative shareholding by the largest ten shareholders), rm characteristics: the log of assets, rm age, and leverage (the ratio of debt to debt plus equity) n = 185 MH5 = β 0 + m 1 (TOPTEN) + m 2 {log(assets)} +m 3 (Age) + m 4 (Leverage) + error

32 Applications Figure 4: Robust estimation (blue), bootstrap band (red dotted), left up: Log(Asset), right up: Leverage, left below: Age, right below: Leverage.

33 Applications 5-3 The impact on stock market (Oil, currency, bond, real estate) aect the stock market. ( /Main.jsp;jsessionid=009E36E74DFA15C80B74EE0BDAEB5746) The X variables are: the crude oil price, EUR- USD exchange rate, the 10 year treasury constant maturity ination index %, and the y variable is S&P 500 index returns , n = 170

34 Applications Figure 5: Robust estimation (blue), bootstrap band (red dotted), y S&P index, left up: exchange rates EUR-USD, right up: crude oil price, left below: ination index, right below: real estate price.

35 Applications Figure 6: Robust estimation (blue), bootstrap band (red dotted), y S&P index return, left up: exchange rates EUR-USD, right up: crude oil price, left below: ination index, right below: real estate price.

36 Applications 5-6 Exchange rate (EUR-USD): (< 1.27) negatively correlated with the stock indices, (> 1.43) a positive correlation follows Oil prices: negative impact Ination index: the ination rate high, interest rates typically high; A negative correlation >0.7 The stock indices raise when the real estate prices gets higher...

37 Tying the straps tighter for generalized linear models Ya'acov Ritov Weining Wang Wolfgang K. Härdle Ladislaus von Bortkiewicz Chair of Statistics C.A.S.E. - Center for Applied Statistics and Economics Humboldt-Universität zu Berlin Department of Statistics The Hebrew University of Jerusalem

38 Applications 5-8 References A. Belloni and V. Chernozhukov Intersection bounds: estimation and inference CEMMAP Working Paper, 10/09, M. Buchinsky Quantile regression, Box-Cox transformation model, and the U.S. wage structure, Journal of Econometrics, 65: , V. Chernozhukov, S. Lee and A. M. Rosen L 1 -Penalized quantile regression in high-dimensional sparse models CEMMAP Working Paper, 19/09, 2009.

39 Applications 5-9 References J. Franke and P. Mwita Nonparametric Estimates for conditional quantiles of time series Report in Wirtschaftsmathematik 87, University of Kaiserslautern, P. Hall On convergence rates of suprema Probab. Th. Rel. Fields, 89: , Jinyong Hahn Bootstrapping Quantile Regression Estimators Econometric Theory, 11(1): , 1995.

40 Applications 5-10 References W. Härdle and J. S. Marron Bootstrap simultaneous error bars for nonparametric regression Ann. Statist., 19: W. Härdle and S. Song Condence bands in quantile regression Econometric Theory, forthcoming, J. Horowitz Bootstrap methods for median regression models Econometrica, 66: , 1998.

41 Applications 5-11 References K. Jeong, W. Härdle and S. Song A Consistent Nonparametric Test for Causality in Quantile Revise and resubmit to Econometric Theory, C. Juhn, K. M. Murphy and B. Pierce Wage Inequality and the Rise in Returns to Skill Journal of Political Economy, 101(3): , R. Koenker and K. F. Hallock Quantile regression Journal of Econometric Perspectives, 15(4): , 2001.

42 Applications 5-12 References J. Komlós, P. Major and G. Tusnády An Approximation of Partial Sums of Independent RV'-s, and the Sample DF. I Z. Wahrscheinlichkeitstheorie verw., 32: , K. Yu, Z. Lu and J. Stander Quantile regression: applications and current research areas J. Roy. Statistical Society: Series D (The Statistician), 52: , Wolfgang Härdle and Joel Horowitz and Jens-Peter Kreiss Bootstrap Method for Time Series International Statistical Review / Revue Internationale de Statistique, 71(2): , 2003.

43 Applications 5-13 References Joel L. Horowitz and Sokbae Lee Nonparametric Estimation of an Additive Quantile Regression Model Journal of the American Statistical Association, 100: , Joel L. Horowitz Nonparametric Estimation of a Generalized Additive Model With an Unknown Link Function Econometrica, 69: , Joel Horowitz and Jussi Klemelä and Enno Mammen Optimal estimation in additive regression models Bernoulli, 12(2): , 2006.

44 Applications 5-14 References Charles J. Stone Additive Regression and Other Nonparametric Models The Annals of Statistics, 13: , 1985.

45 Appendix 6-1 Appendix - Assumptions A.1 ψ(.) = ρ (.) being a.s. dierentiable and Lipschitz continuous: µ 1, µ 2 B ψ(µ 1 ) ψ(µ 2 ) < C µ 1 µ 2, and we assume that M > 0 s.t.ψ(µ) M. A.2 The support of X is [0, 1] d. The conditional density f (y X =x)(.) is bounded from below > C 1 > inf t f (y X =x) (t) = c 1 > 0. A.3 The kernel function K(.) is a product kernel composed from one dimension kernel with bandwidth h = h n : K h (s) = Π d j=1 K(s j/h)/h, s = (s 1,..., s d ) R d. (16)

46 Appendix 7-1 Appendix - Assumptions A.4 The bandwidth satises h n 1/(4+d). Let g be another bandwidth sequence g >> h. (Work our later the speed). Let Γ n be a slowly increasing sequence in the sense that n α a n 0 for any α > 0. A.5 Assume sup x B ˆl g (x) l (x) = O p (1), and sup x B ˆl (x) l (x) = O p (h 2 Γ n ). A.6 There is an α > 0 such that d E X { m j (X j )} 2 αmax j E Xj {mj 2 (X j )} i=1 E Xj {m j (X j )} = 0, m j ( ) L 2 (X j ).

47 Appendix 7-2 A.7 E Xj {ψ 2 l (x i,j)} = 1 for any i 1,..., n and j 1,..., d. Φ l (X j ) C 3 /L, a.s., where Φ l (X j )),..., φ 2 l (x n,j)}. A.8 The inverse link function b satises the following: b C(R), b (θ) > 0,θ R while for a compact interval Θ whose interior contains m([0, 1] d ), C b > max θ Θ b (θ) > min θ Θ b (θ) > c b for C b > c b > 0 A.9 The number of regressors p = dl + 1 (more precisely L = L j ) with L n 1/5. Go back

48 Appendix 7-3 P(ε i < t) = τ P[F 1 i,+1 {F i, ð (ε i )( ε i )} < t] + 1 τ = τ P{F i, ð (εi )( ε i ) < F i,+1 (t)} + 1 τ = τ P{ε i < 0, F i, 1 ( ε i ) < F i,+1 (t)} + τ P{ε i > 0, F i,+1 (ε i ) < F i,+1 (t)} + 1 τ = τ P{ε i < 0, 1 τ F i(ε i ) < F i(t) 1 + τ } 1 τ τ + τ P(0 < ε i < t) + 1 τ = τ P[1 τ > F i (ε i ) > 1 τ {1 F i (t)}] τ + τ P(0 < ε i < t) + 1 τ = τ[1 1 τ {1 F i (t)} τ] + τ{f i (t) 1 + τ} + 1 τ τ = F i (t). Go back