Bayesian nonparametric modeling approaches for quantile regression

Transcription

1 Bayesian nnparametric mdeling appraches fr quantile regressin Athanasis Kttas Department f Applied Mathematics and Statistics University f Califrnia, Santa Cruz Department f Statistics Athens University f Ecnmics and Business December 19, 2007

2 Outline Outline 1. Intrductin and mtivatin 2. Dirichlet prcess prirs and Dirichlet prcess mixtures 3. Bayesian semiparametric quantile regressin 4. Fully nnparametric inference fr quantile regressin Bayesian nnparametric mdeling appraches fr quantile regressin 1/28

3 Intrductin and mtivatin 1. Intrductin and mtivatin In regressin settings, the cvariates may have effect nt nly n the center f the respnse distributin but als n its shape Quantile regressin quantifies relatinship beteen a set f quantiles f respnse distributin and cvariates, and thus, prvides a mre cmplete explanatin f the respnse distributin in terms f available cvariates Applicatins: ecnmetrics, medicine, scial sciences, educatinal studies... Objective is t develp mdeling fr quantile regressin that: relaxes, as much as pssible, parametric assumptins enables full and exact inference fr the quantile regressin functin and any ther feature f the respnse distributin that might be f interest Bayesian nnparametric mdeling appraches fr quantile regressin 2/28

4 Intrductin and mtivatin The area f Bayesian nnparametrics prvides the framerk fr such mdeling instead f specifying unknn functins and distributins up t a (small) number f parameters, treat them as the randm mdel parameters nnparametric prirs supprt the underlying spaces f randm functins/distributins resulting in flexible inferences given the data T mdeling appraches t quantile regressin: a semiparametric mdel here the errr distributin is assigned a nnparametric prir and the regressin functin is mdeled parametrically a fully nnparametric apprach here the jint distributin f the respnse and the cvariates is mdeled ith a mixture mdel, ith psterir inference fr quantile curves emerging thrugh the cnditinal distributin f the respnse given the cvariates Bth appraches utilize Dirichlet prcess mixtures, a flexible class f nnparametric mixture mdels Bayesian nnparametric mdeling appraches fr quantile regressin 3/28

5 Dirichlet prcess prirs and Dirichlet prcess mixtures 2. Dirichlet prcess prirs and Dirichlet prcess mixtures The Dirichlet prcess (DP) (Fergusn, 1973) is a randm prbability measure n distributins characterized by t parameters: a base distributin G 0 (the center f the prcess) and a (precisin) parameter α > 0 DP cnstructive definitin (Sethuraman, 1994) let {z s, s = 1, 2,...} and {φ j, j = 1, 2,...} be independent sequences f randm variables, ith z s i.i.d. Beta(1, α), and φ j i.i.d. G 0 define ω 1 = z 1, ω j = z j j 1 s=1 (1 z s), j 2 (stick-breaking cnstructin) then, a realizatin G frm DP(α, G 0 ) is (almst surely) f the frm G( ) = ω j δ φj ( ) j=1 i.e., a discrete distributin that can be represented as a cuntable mixture f pint masses Bayesian nnparametric mdeling appraches fr quantile regressin 4/28

6 Dirichlet prcess prirs and Dirichlet prcess mixtures P(X<x) x x DP ith G 0 = N(0, 1) and α = 20. In the left panel, the spiked lines are lcated at 1000 sampled values f x dran frm N(0, 1) ith heights given by the eights, ω l, calculated using the stick-breaking algrithm (a truncated versin s that the eights sum t 1). These spikes are then summed frm left t right t generate ne cdf sample path frm the DP. The right panel shs 8 such sample paths indicated by the lighter jagged lines. The heavy smth line indicates the N(0, 1) cdf. Bayesian nnparametric mdeling appraches fr quantile regressin 5/28

7 Dirichlet prcess prirs and Dirichlet prcess mixtures Dirichlet prcess mixture mdel: fr a parametric family f distributins K( ; θ), θ Θ R q, define F ( ; G) = K( ; θ)dg(θ), G DP(α, G 0 ) DP mixture prir can mdel bth discrete and cntinuus distributins (e.g., K( ; θ) might be Pissn, binmial, nrmal, gamma, multivariate nrmal,...) Hierarchical mdel: fr y 1,..., y n i.i.d., given G, frm F ( ; G), y i θ i θ i G ind. i.i.d. K( ; θ i ), i = 1,..., n G, i = 1,..., n G DP(α, G 0 ) typically, hyperprirs n α and/r the parameters ψ f G 0 G 0 (ψ) are added Psterir simulatin methds (mainly MCMC) fr p(θ, α, ψ, G data) Bayesian nnparametric mdeling appraches fr quantile regressin 6/28

8 Bayesian semiparametric quantile regressin 3. Bayesian semiparametric quantile regressin (jint rk ith Milvan Krnjajić) Respnse bservatins y i, ith cvariate vectrs x i, i = 1,..., n Additive quantile regressin frmulatin: y i = x iβ ε i ε i i.i.d. frm an errr distributin ith p-th quantile equal t 0 Parametric mdeling: specifies parametrically the errr distributin e.g., asymmetric Laplace distributin (mdel M 0 ): ith 0 kal p ε i iid k AL p (ε; σ) = σ 1 p(1 p) exp{ σ 1 ε(p 1 (σ 1 ε<0) )} (ε; σ)dε = p Limitatin: ne parameter p determines bth quantile and skeness (p > 0.5 left skeed, p = 0.5 symmetric, p < 0.5 right skeed) fr example, the errr distributin is symmetric in the median regressin case Bayesian nnparametric mdeling appraches fr quantile regressin 7/28

9 Bayesian semiparametric quantile regressin Objective: develp flexible nnparametric prir mdels fr the randm errr density f p ( ) DP mixture mdels fr the quantile regressin errr density Mdel M 1 : general scale mixture f asymmetric Laplace densities fp 1 (ε; G) = kp AL (ε; σ)dg(σ), G DP(α, G 0 ) captures mre flexible tail behavir (mixing preserves quantiles, 0 f 1 p (ε; G)dε = p) M 1 extends M 0 ith regard t tail behavir, but the skeness f the mixture f 1 p ( ; G) suffers the same limitatin as the kernel k AL p ( ; σ) Bayesian nnparametric mdeling appraches fr quantile regressin 8/28

10 Bayesian semiparametric quantile regressin A key result that alls a mre flexible mdel frmulatin is a representatin therem fr nn-increasing densities n R : Fr any nn-increasing density f( ) n R there exists a distributin functin G, ith supprt n R, such that f(t; G) = θ 1 1 [0,θ) (t)dg(θ) This result leads t a mixture representatin fr any unimdal density n the real line ith p-th quantile (and mde) equal t zer, k p (ε; σ 1, σ 2 )dg 1 (σ 1 )dg 2 (σ 2 ) ith G 1 and G 2 supprted by R, and k p (ε; σ 1, σ 2 ) = p σ 1 1 ( σ1,0)(ε) (1 p) σ 2 1 [0,σ2 )(ε), ith 0 < p < 1, σ r > 0, r = 1, 2 Bayesian nnparametric mdeling appraches fr quantile regressin 9/28

11 Bayesian semiparametric quantile regressin Assuming independent DP prirs fr G 1 and G 2, e btain mdel M 2 : fp 2 (ε; G 1, G 2 ) = k p (ε; σ 1, σ 2 )dg 1 (σ 1 )dg 2 (σ 2 ), G r DP(α r, G r0 ), r = 1, 2 mdel M 2 can capture general frms f skeness and tail behavir The full hierarchical mdel M 2 : y i β, σ 1i, σ 2i σ ri G r ind iid k p (y i x iβ; σ 1i, σ 2i ), i = 1,..., n G r, r = 1, 2, i = 1,..., n G r α r, d r DP(α r, G r0 = IGamma(c r, d r )), r = 1, 2 Psterir inference under all mdels is btained using standard MCMC methds fr DP mixture mdels (censring can als be handled) Bayesian nnparametric mdeling appraches fr quantile regressin 10/28

12 Bayesian semiparametric quantile regressin Data Illustratins Simulated data (n = 250 in each case) frm distributins ith a specific quantile fixed at 0 (n cvariates) and ith varying shapes three standard Laplace distributins (σ = 1) fr three values f p (p = 0.5, 0.9, and 0.1) a standard nrmal distributin, and t mixtures f nrmals, ne ith 0.6-th quantile at zer and anther ith median zer (the cmpnents fr bth nrmal mixtures are chsen s that the resulting mixture densities are right skeed ith nn-standard tail behavir) Small cell lung cancer data: survival times in days fr 121 patients ith small cell lung cancer; 23 survival times are right censred each patient as randmly assigned t ne f t treatments A and B, achieving 62 and 59 patients, respectively (treatment indicatr is the cvariate) Bayesian nnparametric mdeling appraches fr quantile regressin 11/28

13 Bayesian semiparametric quantile regressin Simulatin study. Prir and psterir predictive densities (dtted and dashed lines) under mdel M 2. The slid lines dente the true densities; the histgrams f the data are als included. Bayesian nnparametric mdeling appraches fr quantile regressin 12/28

14 Bayesian semiparametric quantile regressin Time in days Time in days Time in days Time in days Time in days Time in days Small cell lung cancer data. Mdel M 2 psterir predictive densities and survival functins, and psterirs fr 25th, 50th, 75th and 90th percentile survival times fr treatment A and B (slid and dashed lines). Bayesian nnparametric mdeling appraches fr quantile regressin 13/28

15 Bayesian semiparametric quantile regressin Time in days Small cell lung cancer data. The tp panel displays psterir predictive densities fr treatment A under mdel M 0 (slid line), mdel M 2 (dashed line), and a parametric Weibull mdel (dtted line). The bttm panels include CPO plts fr the uncensred (left panel) and censred data (right panel). The dente CPO values under mdel M 0, under mdel M 2, and under the Weibull mdel. Bayesian nnparametric mdeling appraches fr quantile regressin 14/28

16 Bayesian semiparametric quantile regressin Quantile regressin ith dependent errr densities Mtivatin: Under the previus setting, the distributin f ε i is the same fr all x i, and thus, the distributin f y i changes ith x i nly thrugh the p-th quantile x iβ T mdel nnparametrically errr distributins that change ith cvariates, e need a prir mdel fr f p,x ( ) = {f p,x ( ) : x X}, here X is the cvariate space, and fr each x, 0 f p,x(ε)dε = p Fr example, under mdel M 2, t all f 2 p (ε; G 1, G 2 ) t change ith x, the mixing distributins G 1, G 2 need t change ith x fr r = 1, 2, e need t replace G r ith a stchastic prcess G r,x ver X Bayesian nnparametric mdeling appraches fr quantile regressin 15/28

17 Bayesian semiparametric quantile regressin Dependent DP (DDP) prirs (MacEachern, 1999, 2000) can be used fr G r,x, r = 1, 2 Briefly, the idea is t use the cnstructive definitin f the DP here n the pint masses are i.i.d. realizatins frm a base stchastic prcess (say a Gaussian prcess rking ith lg(σ ri )), retaining the same (cmmn α) stick-breaking cnstructin fr the eights A key advantage f the DDP mdel is its flexibility in capturing different shapes fr different cvariate values (bth bserved and unbserved cvariate values) Several ther recent cnstructins and extensins f the DDP framerk: ANOVA DDP, spatial DP, hierarchical DP, rdered DP, nested DP, lcal DP... Bayesian nnparametric mdeling appraches fr quantile regressin 16/28

18 Bayesian semiparametric quantile regressin X = X = X = X = X = X = X = Simulatin experiment fr the DDP quantile regressin mdel. Psterir predictive densities (dashed lines) at five bserved cvariate values, verlaid n histgrams f the crrespnding respnse bservatins, and at t ne cvariate values, x = 10 and x = 95, verlaid n crrespnding true densities (slid lines). Bayesian nnparametric mdeling appraches fr quantile regressin 17/28

19 Bayesian semiparametric quantile regressin dse = dse = dse = dse = dse = dse = dse = dse = Cmet assay data. Psterir predictive densites under the DDP mdel (dashed lines) at the five bserved dse values, verlaid n histgrams f the crrespnding respnses, and at 3 ne dses (10, 40, and 95). Bayesian nnparametric mdeling appraches fr quantile regressin 18/28

20 Fully nnparametric inference fr quantile regressin 4. Fully nnparametric inference fr quantile regressin (jint rk ith Matt Taddy) Semiparametric additive quantile regressin framerk: enables readily interpretable inference by separating quantile regressin functin frm errr distributin prpsed Bayesian semiparametric mdel yields flexible inference fr unimdal errr densities ith parametric quantile regressin functins A pssible extensin: add nnparametric prir mdels fr the quantile regressin functins h m in the additive setting: y i = M m=1 h m(x mi ) ε i current rk studies the utility and feasibility f Gaussian prcess prirs fr the h m Bayesian nnparametric mdeling appraches fr quantile regressin 19/28

21 Fully nnparametric inference fr quantile regressin Alternative mdel-based nnparametric apprach: mdel jint density f(y, x) f the respnse y and the M-variate vectr f (cntinuus) cvariates x ith a DP mixture f nrmals: f(y, x) f(y, x; G) = N M1 (y, x; µ, Σ)dG(µ, Σ), G DP(α, G 0 ) ith G 0 (µ, Σ) = N M1 (µ; m, V ) IWish(Σ; ν, S) Fr any grid f values (y 0, x 0 ), btain psterir samples fr: jint density f(y 0, x 0 ; G), and marginal density f(x 0 ; G) cnditinal density f(y 0 x 0 ; G) and cnditinal cdf F (y 0 x 0 ; G) cnditinal quantile regressin q p (x 0 ; G), fr any 0 < p < 1 Bayesian nnparametric mdeling appraches fr quantile regressin 20/28

22 Fully nnparametric inference fr quantile regressin Apprach t inference requires mre general psterir simulatin methds, hich include sampling frm the psterir f G (mre demanding cmputatinally than the semiparametric mdel) Key features: mdeling framerk enables simultaneus inference fr mre than ne quantile regressin mdel alls flexible respnse distributins and nn-linear quantile regressin relatinships Bayesian nnparametric mdeling appraches fr quantile regressin 21/28

23 Fully nnparametric inference fr quantile regressin Data Example Mral hazard data n the relatinship beteen sharehlder cncentratin and several indices fr managerial mral hazard in the frm f expenditure ith scpe fr private benefit (Yafeh & Yshua, 2003) data set includes a variety f variables describing 185 Japanese industrial chemical firms listed n the Tky stck exchange respnse y: index MH5, cnsisting f general sales and administrative expenses deflated by sales fur-dimensinal cvariate vectr x: Leverage (rati f debt t ttal assets); lg(assets); Age f the firm; and TOPTEN (the percent f nership held by the ten largest sharehlders) Bayesian nnparametric mdeling appraches fr quantile regressin 22/28

24 Fully nnparametric inference fr quantile regressin Marginal Average Medians ith 90% CI Mral Hazard Mral Hazard TOPTEN Leverage Mral Hazard Mral Hazard Age Lg(Assets) Psterir mean and 90% interval estimates fr median regressin fr M H5 cnditinal n each individual cvariate. Data scatterplts are shn in grey. Bayesian nnparametric mdeling appraches fr quantile regressin 23/28

25 Fully nnparametric inference fr quantile regressin Marginal Average 90th Percentiles ith 90% CI Mral Hazard Mral Hazard TOPTEN Leverage Mral Hazard Mral Hazard Age Lg(Assets) Psterir mean and 90% interval estimates fr 90th percentile regressin fr M H5 cnditinal n each individual cvariate. Data scatterplts are shn in grey. Bayesian nnparametric mdeling appraches fr quantile regressin 24/28

26 Fully nnparametric inference fr quantile regressin Leverage TOPTEN Psterir estimates f median surfaces (left clumn) and 90th percentile surfaces (right clumn) fr M H5 cnditinal n each Leverage and TOPTEN. The psterir mean is shn n the tp r and the psterir interquartile range n the bttm. Bayesian nnparametric mdeling appraches fr quantile regressin 25/28

27 Fully nnparametric inference fr quantile regressin Cnditinal density fr MH MH5 Psterir mean and 90% interval estimates fr respnse densities f(y x 0 ; G) cnditinal n fur cmbinatins f values x 0 fr the cvariate vectr (TOPTEN, Leverage, Age, lg(assets)) (clckise frm tp left, x 0 = (40, 0.3, 55, 11), (35, 0.6, 55, 11), (40, 0.3, 70, 13), and (70, 0.8, 55, 11)) Bayesian nnparametric mdeling appraches fr quantile regressin 26/28

28 Fully nnparametric inference fr quantile regressin Mdel elabratins: extensins t incrprate bth categrical and cntinuus cvariates thrugh mixed discrete-cntinuus kernels fr the DP mixture mdel mdeling fr partially bserved respnses (and/r cvariates): quantile regressin fr survival analysis data ith censring; fully nnparametric Tbit quantile regressin fr ecnmetrics data General framerk ith ptentially imprtant applicatins beynd quantile regressin: nnparametric sitching regressin mdeling mdeling and inference fr marked spatial Pissn prcesses sensitivity analysis and inversin f cmputer mdel experiments Bayesian nnparametric mdeling appraches fr quantile regressin 27/28

29 Cntact inf: eb: thans UCSC Department f Applied Math and Statistics:.ams.ucsc.edu Technical Reprts series: E Y X A P I Σ T Ω!!! Bayesian nnparametric mdeling appraches fr quantile regressin 28/28