Discussion on Regularized Regression for Categorical Data (Tutz and Gertheiss)

Transcription

1 Discussin n Regularized Regressin fr Categrical Data (Tutz and Gertheiss) Peter Bühlmann, Ruben Dezeure Seminar fr Statistics, Department f Mathematics, ETH Zürich, Switzerland Address fr crrespndence: Peter Bühlmann, Seminar fr Statistics, Department f Mathematics, HG G 17, Rämistrasse 101, CH-8092 Zürich, Switzerland. buhlmann@stat.math.ethz.ch. Phne: (+41) Fax: (+41) Key wrds: Categrical data; Cmbining different penalizatin types; Cnfidence intervals; High-dimensinal generalized regressin; Ordinal categrical predictrs; P- values; Uncertainty quantificatin

2 2 Peter Bühlmann and Ruben Dezeure We cngratulate Gerhard Tutz and Jan Gertheiss, referred in the sequel as TG, fr an interesting and inspiring paper n the imprtant tpic f regressin fr categrical data. Much has happened ver the 15 years, including earlier cntributins frm the authrs: the current verview and expsitry paper is a mst welcme additin t the literature. 1 Sme further thughts n the paper TG have written a master piece n mdern regressin with categrical cvariables. Special attentin is given t the case with rdinal categrical predictrs: TG explain a variety f pssibilities t build in additinal infrmatin in terms f sparsity, smthness and clusteredness. Frm a fundamental perspective, the case with rdinal categries is cmplex and difficult. Using a vague analgy t nnparametric curve estimatin, the issue here is a kind f varying regularizatin prblem t adapt well t the rugh and flat parts f an underlying true functin (besides the rle f selectin). Clever regularizatin r penalizatin is certainly a pwerful vehicle fr btaining accurate pint estimates f the underlying generalized regressin cefficients, r fr mdel parameters beynd regressin as discussed als in TG in Sectins 5 and Different penalizatin types fr different predictrs A natural questin in practice will be the chice f the penalizatin type which might be different fr different cvariables, and which f the categrical variables shuld be subject t a penalty term. Regarding the secnd questin, a sparsity-type penalizatin shuld autmatically select the variables t be strngly clustered, strngly smthed r threshlded t zer and thus, such sparsity inducing schemes ften wrk well fr the purpse f selectin (n the level f individual r grups f parameters). Crssvalidatin is a natural candidate fr addressing the first questin (aside frm its use t chse the amunt f regularizatin) but it might becme cmputatinally very cumbersme t run it a large multitude f times fr evaluating many cmbinatins. Here, a cmbinatin means that sme subset f rdinal cvariables are subject t a clustering penalty (Sectin in TG), anther subset f rdinal cvariables subject t a smthing penalty (Sectin in TG), and anther subset subject t a sparsity nly penalty (Sectin in TG). We will illustrate in Sectin 2 an explratry way, by inspecting cnfidence intervals, frm which ne might extract sme reasnable infrmatin fr a cmbinatin f the qualitatively different penalty terms acting n different cvariables.

3 Discussin Tutz Gertheiss Tree methds and Randm Frests Sectin 7.3 in TG mentins tree based appraches. They have the advantage that they wrk in a natural way fr mixed variables, where sme f the cvariables are cntinuus, sme categrical with nminal and rdinal categries. Appraches based n penalizatin becme intrinsically mre cmplicated with mixed data, requiring a careful chice f varius penalty terms and their crrespnding regularizatin parameters (see als abve regarding the difficulty t chse different penalizatin types). If the task wuld be predictin nly, Randm Frests (Breiman, 2001) is ften a very pwerful methd which (essentially) des nt require t chse any tuning parameter(s): reasns fr its success in general include that the algrithm als takes interactins between variables int accunt and that it can deal in a scale-free manner with mixed variables. We wnder hw well it wuld perfrm fr predictin in the dataset n Spending fr Fd analyzed in TG. Based n adaptatins f Randm Frests, we have btained in ther wrks sme interesting results fr predictive survival mdeling (Hthrn et al., 2006), fr predictive imputatin f missing data (Stekhven and Bühlmann, 2012), but als fr variable imprtance in mixed undirected graphical mdels (Fellinghauer et al., 2013). The disadvantage f Randm Frests technlgy is that it des nt yield statistical pint estimatin r inferential statements in a well-defined statistical mdel; and ntins f measuring variable imprtance, as advcated in Breiman (2001) r its mdified versin in (Strbl et al., 2008), are still very algrithmic with n statistical guarantees in terms f p-values r cnfidence intervals. 2 Sme utlk: cnfidence intervals and p-values We wuld like t utline here the additinal perspective f uncertainty quantificatin. Recently, sme prgress has been made t cnstruct cnfidence intervals and statistical hypthesis tests in ptentially high-dimensinal generalized regressin mdels (Bühlmann, 2013; Zhang and Zhang, 2014; van de Geer et al., 2014; Javanmard and Mntanari, 2014; Dezeure et al., 2015; van de Geer and Stucky, 2015). The key idea is t de-bias r de-sparsify the sparse Lass estimatr (Tibshirani, 1996). Starting frm the Lass, the de-sparsifying peratin leads t a nn-sparse but regular estimatr whse lw-dimensinal cmpnents have an asympttic Gaussian distributin. Fr example, cnsider a linear mdel Y = Xβ 0 + ε, with fixed design n p matrix X and a Gaussian errr term ε with i.i.d. cmpnents having mean zer and variance σ 2 ε (where p dentes here the number f parameters). The situatin might be very high-dimensinal with p n but the parameter β 0 is assumed t be sparse. Assuming such sparsity cnditins and a restricted eigenvalue

4 4 Peter Bühlmann and Ruben Dezeure assumptin n X (Bühlmann and van de Geer, 2011, cf.), we have: n( ˆβ β 0 ) = W +, W N p (0, σ 2 εω), max j = P (1) (p n ), j=1,...,p where Ω is a knwn cvariance matrix depending n the design matrix X (in analgy t rdinary least squares where Ω wuld be equal t (X T X/n) 1 ). The remarkable feature f this result is that: (i) the de-sparsified estimatr has 1/ n cnvergence rate, despite the very high-dimensinal setting and unlike fr estimating the regressin functin where the cnvergence rate is nly lg(p)/n (Bühlmann and van de Geer, 2011, cf.); and (ii) that the limiting distributin is knwn up t the unknwn nise variance which can be estimated als in the high-dimensinal setting. Thus, we can cnstruct cnfidence intervals fr single parameters β 0 j, and we can test statistical hyptheses fr grups f parameters. Regarding the latter, a grup hyptheses is f the frm H 0,G ; β 0 j = 0 fr all j G, where G {1,... p}. If G is very large, we have t rely n the test-statistic max j G ˆβ j /s.e.( ˆβ j ), and if G is small (say G 10), we can als use the sum-statistic ˆβ j G j : the crrespnding asympttic limiting distributins can be calculated r efficiently simulated in case f the max-type statistics. In additin, multiple testing crrectin can be dne efficiently since the cvariance structure, prprtinal t Ω, is knwn. All f this (but nt the sum-type statistics) is implemented in the R-package hdi fr generalized linear mdels (Meier et al., 2014; Dezeure et al., 2015). The methdlgy utlined abve can be used fr categrical (nminal) predictr variables, at least in the simple frm. In the ntatin f TG, cnsider the mdel frmulatin (1) fr a linear mdel as in (3): the regressin parameters fr the jth cvariable are β jr, r = 1,..., k j (using the cnstraint frm TG with β j0 = 0), fr j = 1,..., p. The R-package hdi then leads t cnfidence intervals fr the underlying true parameters βjr, 0 r = 1,... k j, j = 1,... p. We can als test whether the j categrical cvariable has a significant effect n the respnse Y by cnsidering the null-hypthesis H 0,Gj : β jr = 0 fr all r = 1,..., k j. Testing this hypthesis is related t the grup-wise selectin discussed in Sectin in TG, but nw with a statistical uncertainty measure in terms f a p-value instead f a pint estimatr nly. We illustrate the cnfidence intervals f the single regressin parameters fr a simulated dataset. The suggested apprach allws als t infrmally infer whether the cefficients frm rdered categrical cvariables are similar fr neighbring levels within the same categrical variable, by inspecting the cnfidence intervals. The results are displayed in Figure 1. If the cnfidence intervals wuld heavily verlap fr the cefficients frm neighbring levels within rdinal categrical variables, clustering f the

5 Discussin Tutz Gertheiss 5 crrespnding categries wuld be natural; if the verlap f neighbring cnfidence intervals is less prnunced, smthing f the levels seems reasnable; and if the cnfidence intervals wuld nt verlap at all, there shuld be n penalty r regularizatin fr the levels within the categrical variable. In particular, with such a visual inspectin f cnfidence intervals, ne culd determine fr which f the rdinal categrical variables sme smthing r clustering f categries wuld be expected t be beneficial. (Trying all cmbinatins and ptimizing a crss-validatin scre culd becme quickly cmputatinally infeasible). We infer frm Figure 1 that there are 95% CI: S0 cverage = 0.481, S0c cverage = Truth * * 3.5* * 15.9* * 31.8* 31.9* * significant fr hypthesis testing fr significance level 0.05 after BH crrectin Other rejected hyptheses = Nne Figure 1: Cnfidence intervals fr the nn-zer regressin cefficients based n the de-sparsified Lass fr high-dimensinal inference with R-package hdi. The underlying data is a single realizatin frm mdel (M) described belw with sample size n = 800 and 100 categrical variables with 581 parameters in ttal. Shwn are the cnfidence intervals fr the parameters frm the active set S 0 = {1, 3, 15, 31} f the categrical variables. The cmplement (S 0 ) c = {1,..., 100} \ S 0 encdes the categrical variables which have n effect n the respnse. The true parameters were cvered by the nminal 95% cnfidence intervals in 48.1% f the 27 intervals crrespnding t the active variables in S 0 (ften nly slightly missing t cver the true parameter), and in 98.4% f the 554 intervals crrespnding t the nn-active variables in (S 0 ) c. When using a Bnferrni-Hlm crrectin fr cntrlling the familywise errr rate in multiple testing, we detect all the fur categrical variables frm S 0 and n false psitive finding is made (i.e., n significant variable frm (S 0 ) c ).

6 6 Peter Bühlmann and Ruben Dezeure fur significant categrical variables, namely variables 1, 3, 15 and 31; and these are indeed the nly true variables having an effect. The plt als suggests the fllwing scenaris: variable 1 has tw clusters f categrical values (which is true), variable 3 has unstructured categrical values (which is true), variable 15 has tw clusters f categrical values (which is true) and that there is an indicatin that variable 31 has a smth structure fr its categrical values (which is true). Thus, the plt in Figure 1 is indeed rather infrmative abut the true underlying structure. Mre statements can be made: the significance f the categrical variables when adjusted fr cntrlling the familywise errr rate in multiple testing, using the Bnferrni-Hlm prcedure, leads t the fur significant variables (as mentined already) and t n false psitive findings; withut the multiplicity adjustment, 9 ut f 554 cnfidence intervals fr cefficients whse true values are zer d nt cver the true value zer and hence wuld lead t false psitives. The mdel underlying the ne simulated dataset f sample size n = 800 with 100 categrical variables is cnstructed as fllws. (M) Cnsider X N 100 (0, Σ) with Teplitz matrix Σ ij = 0.9 i j. Generate n i.i.d. samples frm N 100 (0, Σ). The active set f categrical variables with nn-zer effect fr the respnse is S 0 = {1, 3, 15, 31}. We categrize each variable X (j) (j = 1,..., 100) accrding t the empirical quantile (ver the n = 800 realizatins) f X (j) with U j categries: U 1 = 5, U 3 = 6, U j = 10 (j = 15, 31), all ther U j s frm Unifrm frm {2,..., 12}. This leads t a dummy encding design matrix X with 100 j=1 (U j 1) clumns. Cnsider the regressin cefficients: clustered categrical values: β 1,1 = β 1,2 = β 1,3 = 1, β 1,4 = 2, β 15,1 = β 15,2 =... = β 15,7 = 1, β 15,8 = β 15,9 = 2, smthed categrical values: β 31,1 = 1, β 31,2 = 1.25, β 31,3 = 1.5,..., β 31,9 = , unstructured cefficients: β 3,1 = 1, β 3,2 = 1, β 3,3 = 0, β 3,4 = 2, β 3,5 = 2. Finally, take the errr term as n = 800 realizatins frm ε N (0, 1). We have nt explred hw t btain cnfidence statements based n regularizatin fr smthing r clustering the categry levels within an rdinal categrical variable (whse imprtance has been presented in TG fr pint estimatin). Fllwing the same idea as fr the de-sparsified estimatr utlined abve (see als van de Geer et al. (2014)), ne might be able t gain statistical pwer by cnstructing cnfidence statements based n a versin f a regularized estimatr which encurages nt nly sparsity (as the Lass) but als t smth r cluster rdinal categries. T ur knwledge, this has nt been pursued s far.

7 Discussin Tutz Gertheiss 7 3 Cnclusins Tutz and Gertheiss have prvided a very insightful and careful verview f methdlgy develped in the last years fr regularized regressin fr categrical data. Their fcus is n pint estimatin, which is the first step in statistical inference. We have illustrated that recent techniques fr cnstructing cnfidence statements in high-dimensinal generalized regressin have the ptential t be useful als fr regressin with categrical data. Fr a simulated dataset with sample size n = 800, 100 categrical variables and 581 parameters in ttal, we btain very reasnable results even withut smthing r clustering categrical values: this can serve as a step fr inferring which categrical variables (r parts f their levels) shuld be subject t smthing r clustering as described in TG. Obtaining mre pwerful cnfidence statements by using als smthing r clustering f rdinal categries might be pssible: achieving this, particularly in the high-dimensinal setting, is an pen prblem. References Breiman, L. (2001). Randm Frests. Machine Learning, 45, Bühlmann, P. (2013). Statistical significance in high-dimensinal linear mdels. Bernulli, 19, Bühlmann, P. and van de Geer, S. (2011). Statistics fr High-Dimensinal Data: Methds, Thery and Applicatins. Springer. Dezeure, R., Bühlmann, P., Meier, L., and Meinshausen, N. (2015). High-dimensinal inference: cnfidence intervals, p-values and R-sftware hdi. Statistical Science, 30, Fellinghauer, B., Bühlmann, P., Ryffel, M., vn Rhein, M., and Reinhardt, J. (2013). Stable graphical mdel estimatin with Randm Frests fr discrete, cntinuus, and mixed variables. Cmputatinal Statistics & Data Analysis, 64, Hthrn, T., Bühlmann, P., Dudit, S., Mlinar, A., and van der Laan, M. (2006). Survival ensembles. Bistatistics, 7, Javanmard, A. and Mntanari, A. (2014). Cnfidence intervals and hypthesis testing fr high-dimensinal regressin. Jurnal f Machine Learning Research, 15, Meier, L., Meinshausen, N., and Dezeure, R. (2014). hdi: High-Dimensinal Inference. URL R package versin Stekhven, D. and Bühlmann, P. (2012). Missfrest - nnparametric missing value imputatin fr mixed-type data. Biinfrmatics, 28,

8 8 Peter Bühlmann and Ruben Dezeure Strbl, C., Bulesteix, A.-L., Kneib, T., Augustin, T., and Zeileis, A. (2008). Cnditinal variable imprtance fr Randm Frests. BMC Biinfrmatics, 9:307. Tibshirani, R. (1996). Regressin shrinkage and selectin via the Lass. Jurnal f the Ryal Statistical Sciety, Series B, 58, van de Geer, S. and Stucky, B. (2015). χ 2 -cnfidence sets in high-dimensinal regressin. T appear in the Abel Sympsium 2014 bk; Preprint arxiv: van de Geer, S., Bühlmann, P., Ritv, Y., and Dezeure, R. (2014). On asympttically ptimal cnfidence regins and tests fr high-dimensinal mdels. Annals f Statistics, 42, Zhang, C.-H. and Zhang, S. (2014). Cnfidence intervals fr lw dimensinal parameters in high dimensinal linear mdels. Jurnal f the Ryal Statistical Sciety, Series B, 76,