Additive Isotonic Regression

Transcription

1 Additive Isotonic Regression Enno Mammen and Kyusang Yu 11. July 2006

2 INTRODUCTION: We have i.i.d. random vectors (Y 1, X 1 ),..., (Y n, X n ) with X i = (X1 i,..., X d i ) and we consider the additive isotonic regression model E(Y i X i 1,..., X i d ) = c + m 1(X i 1) + + m d (X i d ) where m j ( ) s are monotone functions. AIM: Develop asymptotic theory for the least squares estimator Show oracle property for the least squares estimator

3 WHY IS IT INTERESTING TO LOOK AT ADDITIVE ISOTONIC REGRESSION? Useful model for some applications Main motivation: statistical theory for models with several nonparametric components (plus additional finite-dimensional parameters) GENERAL THEME: Optimal estimation in a semiparametric model with m 1,..., m D nonparametric components of the model, θ 1,..., θ K parametric components of the model.

4 TYPES OF RESULTS (A) General theory for optimal estimation of the parametric components: Rate is of order n 1/2. Variance depends on geometry of score functions, see e.g. Bickel, Klaassen, Ritov, Wellner (1993). (B) Nonparametric components can be estimated with rates depending on entropy measures of the corresponding function classes (e.g. smoothness classes). Rates do not depend on the number of nonparametric components. Results available for a wide class of models, see e.g. Stone (1985,1986), work of S. van de Geer, L. Birge, P. Massart,...

5 (C) Can a nonparametric component, m 1 say, be estimated with the same asymptotic accuracy as if the other components m 2,..., m D, θ 1,..., θ K would be known? MORE PRECISELY: Consider an estimator m OR 1 (e.g. kernel estimator, smoothing spline, orthogonal series estimator,...) in the oracle model where m 2,..., m D, θ 1,..., θ K are known. THEN: does there exist an estimator m 1 in the "true"model (where all m 1,..., m D, θ 1,..., θ K are unknown) such that m 1 m OR 1 is of lower order? ORACLE PROPERTY, m OR 1 ORACLE ESTIMATOR

6 Result of type (C) (Oracle property) has been shown for local linear smooth backfitting for Additive Models: ( ) Y = m 1 (X 1 ) m d (X d ) + ε, X 1,..., X d (one-dimensional) covariables, m 1,..., m d unknown functions, ε unobserved error variable with E[ε X ] = 0 in Mammen, Linton and Nielsen (1999) for a two-step kernel smoother (Nadaraya-Watson + Local Linear) in Horowitz + M. (2005a,b) for Generalized Additive Models: Y = F [m 1 (X 1 ) m d (X d )] + ε, with F link function that is known (Horowitz + M., 2005a) or unknown (Horowitz + M., 2005b)

7 MORE GENERAL RESULT FOR ADDITIVE MODELS (Horowitz, Klemela + M., 2005) To each smoothing estimator (kernel smoother, local linear, orthogonal series, wavelet,...) m OR 1 in the oracle model Y = m 1 (X 1 ) + ε there exists an estimator m 1 in model (*) with m 1 m OR 1 is of lower order.

8 Why does the oracle property hold? Three motivations: FIRST MOTIVATION: Two step procedure of Horowitz, Klemela + M. (2005): Step 1: Construct consistent undersmoothed estimator m 1 of m 1 Step 2: Apply smoothing (kernel, spline, orthogonal series,..., respectively) to m 1 (X i 1 ), (i = 1,..., n) (instead of Y i). This works because: bias terms of m 1 are asymptotically negligible local averages with bandwidth h of local averages with bandwidth g behave in first order as local averages with bandwidth h if h/g

9 SECOND MOTIVATION: Local linear smooth backfitting of M., Linton and Nielsen (1999): Smooth backfitting estimator m j solves m j (x j ) = m j (x j ) m 0 m l (u l ) p Xl X j (u l x j )du l l j with m j estimator of the marginal expectation mj (x j) = E[Y X j = x j ] and p Xl X j estimator of the conditional density p Xl X j. The stochastic part of the last term on the right hand side is of lower order because it is an average of local averages. THIRD MOTIVATION Implicit Cross Validation: Local smoothing takes place in neighborhoods that are only local with respect to one coordinate.

10 For many smoothing methods the oracle property does not hold (in the additive model): classical backfitting smoothing splines, penalized least squares orthogonal series estimators: regression splines, wavelets,... For these estimators the bias term of m 1 depends on m 2,..., m d.

11 TODAY: A result of type (C) (Oracle property) for Additive Models with monotone components m 1,...,m d. COMPARE: The least squares estimator m 1,..., m d, and ĉ that minimizes n (Y i c µ 1 (X1) i µ d (Xd i ))2 i=1 over constants c and all monotone increasing functions µ j with µ j (u j )du j = 0.

12 WITH: The least squares estimator m OR 1, and ĉ OR that minimizes n (Y OR,i c µ 1 (X1)) i 2 = i=1 n (ε i + m(x1) i c µ 1 (X1)) i 2 i=1 over constants c and all monotone increasing functions µ 1 with µ 1 (u 1 )du 1 = 0.

13 RESULT: m 1 (x 1 ) m OR 1 (x 1 ) = o P (n 1/3 ). The difference is of lower order because m OR 1 m 1 is of order O P (n 1/3 ). Important assumption: X has density.

14 COROLLARY: n 1/3 [2p 1 (x 1 )] 1/3 σ 1 (x 1 ) 2/3 m 1 (x 1) 2/3 [ m 1(x 1 ) m 1 (x 1 )] converges in distribution to the slope of the greatest convex minorant of W (t) + t 2, where W is a two-sided Brownian motion. Here, p 1 density of X 1, σ 2 1 (x 1) conditional variance of ε i given X i 1 = x 1.

15 WHY DOES THE RESULT HOLD: Problem: Want to understand the estimator m 1, that minimizes n (Ŷ i c µ 1 (X1)) i 2 i=1 over monotone µ 1 with Ŷ i = Y i m 2 (X i 2 )... m d(x i d ). This cannot be done without studying m 2,..., m d. Approach: Use tools from empirical process theory and representations of monotone least squares estimators.

16 Representations of monotone least squares estimators: m 1 (x 1 ) = max 0 u x 1 m OR 1 (x 1 ) = max 0 u x 1 min x 1 v 1 min x 1 v 1 t i:u X1 i v Ŷ i #{i : u X1 i v}, t i:u X1 i v Y OR,i #{i : u X1 i v} Here, #A denotes the number of elements of a set A and Y OR,i = ε i + m 1 (X1), i Ŷ i = Y i m 2 (X2) i... m d (Xd i ) = Y OR,i [ m 2 (X2) i m 2 (X2)] i... [ m d (Xd i ) m d(xd i )].

17 These formulas say: m 1 (x 1 ) is a local average of Ŷ i = Y OR,i [ m 2 (X i 2 ) m 2(X i 2 )]... [ m d(x i d ) m d(x i d )]. m OR 1 (x 1 ) is a local average of Y OR,i. One can study the difference terms with the help of empirical process theory: (*) local averages of m l (Xl i) m l(xl i) = [ m l (u l ) m l (u l )]p Xl X 1 (u l x l )du l + o P (n 1/3 ) This is a constant. Thus m 1 solves the following integral equation: m 1 (x 1 ) = m 1 OR (x 1 ) [ m l (u l ) m l (u l )]p Xl X 1 (u l x l )du l +o P (n 1/3 ) l j

18 One can show that m l OR solves the integral equation because of [ m l OR (u l ) m l (u l )]p Xl X 1 (u l x l )du l = o P (n 1/3 ). l j This implies the claim of the result: m l (x 1 ) = m OR l (x 1 ) + o P (n 1/3 ).

19 SIMULATIONS: Model: Y = m 1 (X 1 ) + m 2 (X 2 ) + ɛ, where, (X 1, X 2 ) has truncated bivariate normal distribution and ɛ N(0, ). Furthermore, m 1 (x) = x for x > 0.5; 0.5 for 0 x 0.5; 0.5 for 0.5 x < 0 and m 2 (x) = sin(πx/2).

20 m 1 m 2 n ρ Backfitting Oracle B/O Backfitting Oracle B/

21 SUMMARY AND OUTLOOK. In the talk I have discussed a model with several nonparametric components and I have shown that the first order asymptotics of a nonparametric least squares estimator for one component does not depend on the presence of the additional components. In general, it is not clear when procedures achieve this oracle property. more complicated models, components with different entropy rates for the considered model: numerics, identifiability,... not discussed in talk

22