Single Index Quantile Regression for Heteroscedastic Data

Size: px

Start display at page:

Download "Single Index Quantile Regression for Heteroscedastic Data"

Angelina Underwood
5 years ago
Views:

1 Single Index Quantile Regression for Heteroscedastic Data E. Christou M. G. Akritas Department of Statistics The Pennsylvania State University JSM, 2015 E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

2 Outline 1 Introduction Motivation Possible Models 2 Single Index Quantile Regression The Proposed Estimator Main Results 3 Numerical Studies Example 1 Example 2 4 Conclusions 5 Future Work E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

3 Outline 1 Introduction Motivation Possible Models 2 Single Index Quantile Regression The Proposed Estimator Main Results 3 Numerical Studies Example 1 Example 2 4 Conclusions 5 Future Work E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

4 Introduction Motivation Least Squares Regression: Models the relationship between a d-dimensional vector of covariates X and the conditional mean of the response Y given X = x. Least Squares Estimator: 1 provides only a single summary measure for the conditional distribution of the response given the covariates. 2 sensitive to outliers and can provide a very poor estimator in many non-gaussian and especially long-tailed distributions. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

5 Introduction Motivation Quantile Regression (QR): Models the relationship between a d-dimensional vector of covariates X and the τ th conditional quantile of the response Y given X = x, Q τ (Y x). Quantile Regression: 1 provides a more complete picture for the conditional distribution of the response given the covariates. 2 useful for modeling data with heterogeneous conditional distributions, especially data where extremes are important. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

6 Outline 1 Introduction Motivation Possible Models 2 Single Index Quantile Regression The Proposed Estimator Main Results 3 Numerical Studies Example 1 Example 2 4 Conclusions 5 Future Work E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

7 Introduction Possible Models Linear QR: Q τ (Y x) = β x, for a d-dimensional vector of unknown parameters β. Disadvantage: linear assumption not always valid. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

8 Introduction Possible Models Linear QR: Q τ (Y x) = β x, for a d-dimensional vector of unknown parameters β. Disadvantage: linear assumption not always valid. Nonparametric QR: Q τ (Y x) = g(x), for an unspecified function g : R d R. Disadvantage: meets the data sparseness problem for high dimensional data. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

9 Introduction Possible Models Linear QR: Q τ (Y x) = β x, for a d-dimensional vector of unknown parameters β. Disadvantage: linear assumption not always valid. Nonparametric QR: Q τ (Y x) = g(x), for an unspecified function g : R d R. Disadvantage: meets the data sparseness problem for high dimensional data. Single-Index QR: Q τ (Y x) = g(β x β), depends on x only through the single linear combination β x. Advantage: reduces the dimension and maintain some nonparametric flexibility. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

10 Outline 1 Introduction Motivation Possible Models 2 Single Index Quantile Regression The Proposed Estimator Main Results 3 Numerical Studies Example 1 Example 2 4 Conclusions 5 Future Work E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

11 The Proposed Estimator The Model Let {Y i, X i } n i=1 be independent and identically distributed (iid) observations that satisfy Y i = Q τ (Y X i ) + ɛ i, where Q τ (Y x) = Q τ (Y β 1x) and the error term satisfies Q τ (ɛ i x) = 0. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

12 The Proposed Estimator The Model Let {Y i, X i } n i=1 be independent and identically distributed (iid) observations that satisfy Y i = Q τ (Y X i ) + ɛ i, where Q τ (Y x) = Q τ (Y β 1x) and the error term satisfies Q τ (ɛ i x) = 0. For identifiability, we impose certain conditions on β 1 : the most common one: β 1 = 1 with its first coordinate positive. the one adopted here: β 1 = (1, β ). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

13 The Proposed Estimator The Model The true parametric vector β satisfies β = arg min b E ( ρ τ (Y Q τ (Y b 1X)) ), where b 1 = (1, b ) and ρ τ (u) = (τ I (u < 0))u. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

14 The Proposed Estimator Iterative algorithms As in the single index mean regression (SIMR) problem, the unknown Q τ (Y b 1X) must be replaced with an estimator. Unlike the SIMR problem, however, there is no closed form expression for the estimator of Q τ (Y b 1 X i), and this has led to iterative algorithms for estimating β (Wu, Yu, and Yu, 2010; Kong and Xia, 2012). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

15 The Proposed Estimator The New Approach For any given b R d 1, define the function g(u b) : R R as where b 1 = (1, b ). g(u b) = E(Q τ (Y X) b 1X = u), E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

16 The Proposed Estimator The New Approach For any given b R d 1, define the function g(u b) : R R as where b 1 = (1, b ). Noting that β also satisfies where b 1 = (1, b ). g(u b) = E(Q τ (Y X) b 1X = u), g(β 1X β) = Q τ (Y X), β = arg min b E ( ρ τ (Y g(b 1X b)) ), (1) E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

17 The Proposed Estimator The New Approach The sample level version of (1) consists of minimizing S n (τ, b) = n ρ τ (Y i g(b 1X i b)). i=1 E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

18 The Proposed Estimator The New Approach The sample level version of (1) consists of minimizing S n (τ, b) = n ρ τ (Y i g(b 1X i b)). i=1 Again, g( b) is unknown but it can be estimated, in a non-iterative fashion, by first obtaining estimators Q τ (Y X i ), for i = 1,..., n, and forming the Nadaraya-Watson-type estimator ĝ NW ˆQ (t b) = n i=1 ( ) t b Q τ (Y X i )K 1 X i 2 h 2 ( ) n k=1 K t b, (2) 1 X k 2 h 2 where K 2 ( ) is a univariate kernel function and h 2 is a bandwidth. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

19 The Proposed Estimator The New Approach For the estimator Q τ (Y X i ), we use the local linear conditional quantile estimator (Guerre and Sabbah, 2012). Specifically, for a multivariate kernel function K 1 (x) = K 1 (x 1,..., x d ) and a univariate bandwidth h 1, let L n ((α 0, α 1 ); τ, h 1, x) = 1 nh d 1 n i=1 ( ) ρ τ (Y i α 0 α Xi x 1(X i x))k 1 h 1 and define Q τ (Y x) as α 0 (τ; h 1, x), where α 0 (τ; h 1, x) is defined through ( α 0 (τ; h 1, x), α 1 (τ; h 1, x)) = arg min (α 0,α 1 ) L n((α 0, α 1 ); τ, h 1, x). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

20 The Proposed Estimator The proposed estimator is obtained by β = arg min b Θ n ( ) ρ τ Y i ĝ NW (b 1X i b), (3) i=1 where Θ R d 1 is a compact set, β is in the interior of Θ. ˆQ E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

21 Outline 1 Introduction Motivation Possible Models 2 Single Index Quantile Regression The Proposed Estimator Main Results 3 Numerical Studies Example 1 Example 2 4 Conclusions 5 Future Work E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

22 Main Results Proposition 1 Let ĝ NW (t b) be as defined in (2).Under some regularity conditions, we ˆQ have sup ĝ NW Q (t b) g(t b) ( = O p a n + a n + h2) 2, b Θ,t T b where a n = (log n/n) s/(2s+d) and a n = (log n/(nh 2 )) 1/2. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

23 Main Results Proposition 1 Let ĝ NW (t b) be as defined in (2).Under some regularity conditions, we ˆQ have sup ĝ NW Q (t b) g(t b) ( = O p a n + a n + h2) 2, b Θ,t T b where a n = (log n/n) s/(2s+d) and a n = (log n/(nh 2 )) 1/2. Proposition 2 Let β be as defined in (3). Under regularity conditions, β is n-consistent estimator of β. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

24 Main Results Theorem Let β be as defined in (3). Under regularity conditions, n( β β) d N ( 0, τ(1 τ)v 1 ΣV 1), where V = E ( (g (β 1X β)) 2 (X 1 E(X 1 β 1X))(X 1 E(X 1 β 1X)) f ɛ X (0 X) ), and Σ = E ( (g (β 1X β)) 2 (X 1 E(X 1 β 1X))(X 1 E(X 1 β 1X)) ). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

25 Outline 1 Introduction Motivation Possible Models 2 Single Index Quantile Regression The Proposed Estimator Main Results 3 Numerical Studies Example 1 Example 2 4 Conclusions 5 Future Work E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

26 Numerical Studies Notation NWQR - the proposed estimator. WYY - the estimator resulting from the iterative algorithm proposed by Wu et al. (2010). WYY-1 - the estimator resulting from dividing the estimator of Wu et al. (2010) by its first component. WYY-2 - the estimator resulting from using the proposed constraint in conjunction with the iterative algorithm of Wu et al. (2010). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

27 Numerical Studies Example 1 Consider the model Y = exp(β 1X) + ɛ, (4) where X = (X 1, X 2 ), X i N(0, 1) are iid, β 1 = (1, 2) and the residual ɛ follows a standard normal distribution. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

28 Numerical Studies Example 1 Consider the model Y = exp(β 1X) + ɛ, (4) where X = (X 1, X 2 ), X i N(0, 1) are iid, β 1 = (1, 2) and the residual ɛ follows a standard normal distribution. We fit the single index quantile regression model for five different quantile levels, τ = 0.1, 0.25, 0.5, 0.75, 0.9. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

29 Numerical Studies Example 1 Consider the model Y = exp(β 1X) + ɛ, (4) where X = (X 1, X 2 ), X i N(0, 1) are iid, β 1 = (1, 2) and the residual ɛ follows a standard normal distribution. We fit the single index quantile regression model for five different quantile levels, τ = 0.1, 0.25, 0.5, 0.75, 0.9. We use sample size of n = 400 and perform N = 100 replications. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

30 Numerical Studies Example 1 For comparison of the above estimators, consider the mean square error, R( β), and the mean check based absolute residuals, R τ ( β 1 ), which are defined as R( β) = 1 N N ( β i 2) 2, R τ ( β 1 ) = 1 n i=1 n ρ τ (Y i i=1 Q LL τ (Y β 1X i )), (5) where Q LL τ (Y β 1x) is the local linear conditional quantile estimator on the univariate variable β 1X i. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

31 Table 1: mean values and standard errors (in parenthesis), R( β) and R τ ( β 1 ) for Model (4). 95% coverage probability for NWQR, WYY-2 and the second estimated coefficient of Wu et al. (2010). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32 Numerical Studies Example 1 τ NWQR (0.0603) (0.0477) (0.0481) (0.0683) (0.3905) β WYY-1 (0.2521) (0.2252) (3.8756) (0.3246) (0.5906) WYY (0.1593) (0.1307) (0.3937) (0.6735) (0.7240) NWQR R( β) WYY WYY NWQR R τ ( β 1 ) WYY WYY coverage NWQR prob. WYY WYY

32 Outline 1 Introduction Motivation Possible Models 2 Single Index Quantile Regression The Proposed Estimator Main Results 3 Numerical Studies Example 1 Example 2 4 Conclusions 5 Future Work E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

33 Numerical Studies Boston Housing Data 506 observations. 14 variables, response: medv, the median value of owner-occupied homes in $1000 s, predictors: measurements on the 506 census tracts in suburban Boston from the 1970 census. available in the MASS library in R. consider the four covariates: RM, TAX, PTRATIO, LSTAT (Opsomer and Ruppert 1998; Yu and Lu 2004; Wu et al. 2010). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

34 Numerical Studies Boston Housing Data center the dependent variable. take logarithmic transformations on TAX and LSTAT. standardize RM, log(tax), PTRATIO and log(lstat) and denote them by X 1, X 2, X 3 and X 4. consider the single index quantile regression models: Q τ (medv X) = g(x 1 + β 2 X 2 + β 3 X 3 + β 4 X 4 β), (6) for τ = 0.1, 0.25, 0.5, 0.75, 0.9. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

35 Numerical Studies Boston Housing Data τ RM log(tax) PTRATIO log(lstat) (0.2431) (0.3521) (0.4038) (0.2325) (0.4518) (0.4861) (0.3286) (0.6176) (0.6765) (0.3585) (0.6414) (0.7184) (0.0452) (0.0995) (0.0936) Table 2: Parametric vector estimates and standard errors (in parenthesis) for Boston housing data for Models (6) with five different quantile levels. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

36 Numerical Studies Boston Housing Data τ WYY NWQR Table 3: Comparison of average check absolute residuals R τ ( β 1 ) defined in (5). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

37 Conclusions conditional quantiles provide a more complete picture of the conditional distribution and share nice properties such as robustness. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

38 Conclusions conditional quantiles provide a more complete picture of the conditional distribution and share nice properties such as robustness. consider a single index model to reduce the dimensionality and maintain some nonparametric flexibility. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

39 Conclusions conditional quantiles provide a more complete picture of the conditional distribution and share nice properties such as robustness. consider a single index model to reduce the dimensionality and maintain some nonparametric flexibility. iterative algorithms for estimating the parametric vector; convergence issues. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

40 Conclusions conditional quantiles provide a more complete picture of the conditional distribution and share nice properties such as robustness. consider a single index model to reduce the dimensionality and maintain some nonparametric flexibility. iterative algorithms for estimating the parametric vector; convergence issues. proposed a method that directly estimates the parametric component non-iteratively and have derived its asymptotic distribution under heteroscedastic errors. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

41 Conclusions conditional quantiles provide a more complete picture of the conditional distribution and share nice properties such as robustness. consider a single index model to reduce the dimensionality and maintain some nonparametric flexibility. iterative algorithms for estimating the parametric vector; convergence issues. proposed a method that directly estimates the parametric component non-iteratively and have derived its asymptotic distribution under heteroscedastic errors. make use of a parametrization that corresponds to the asymptotic theory derived. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

42 Future Work for high dimensional data, it is possible to improve the local linear conditional quantile estimators Q τ (Y X i ). E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

43 Future Work for high dimensional data, it is possible to improve the local linear conditional quantile estimators Q τ (Y X i ). variable selection for estimating the local linear conditional quantile estimators. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

44 Future Work for high dimensional data, it is possible to improve the local linear conditional quantile estimators Q τ (Y X i ). variable selection for estimating the local linear conditional quantile estimators. penalty term for estimating the parametric component β. E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

45 References I E. Guerre and C. Sabbah. Uniform bias study and Bahadure representation for local polynomial estimators of the conditional quantile function. Econometric Theory, 28(01):87 129, E. Kong and Y. Xia. A single-index quantile regression model and its estimation. Econometric Theory, 28: , J. D. Opsomer and D. Ruppert. A fully automated bandwidth selection for additive regression model. Journal of the American Statistical Association, 93: , T. Z. Wu, K. Yu and Y. Yu. Single index quantile regression. Journal of Multivariate Analysis, 101(7): , E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

46 References II K. Yu and Z. Lu. Local linear additive quantile regression. Scandinavian Journal of Statistics, 31: , E. Christou, M. G. Akritas (PSU) SIQR JSM, / 32

Single Index Quantile Regression for Heteroscedastic Data

Single Index Quantile Regression for Heteroscedastic Data E. Christou M. G. Akritas Department of Statistics The Pennsylvania State University SMAC, November 6, 2015 E. Christou, M. G. Akritas (PSU) SIQR