A Note on Adaptive Group Lasso

Transcription

1 A Note o Adaptive Group Lasso Hasheg Wag ad Chelei Leg Pekig Uiversity & Natioal Uiversity of Sigapore July 7, Abstract Group lasso is a atural extesio of lasso ad selects variables i a grouped maer. However, group lasso suffers from estimatio iefficiecy ad selectio icosistecy. To remedy these problems, we propose the adaptive group lasso method. We show theoretically that the ew method is able to idetify the true model cosistetly, ad the resultig estimator ca be as efficiet as oracle. Numerical studies cofirmed our theoretical fidigs. KEY WORDS: Adaptive Group Lasso; Adaptive Lasso; Group Lasso; Lasso Itroductio Sice its first proposal by Tibshirai (996), the least absolute shrikage ad selectio operator (lasso) has geerated much iterest i statistical literature (Fu, 998; Kight ad Fu, 2000; Fa ad Li, 200; Efro et al., 2004). The key stregth of lasso lies i its ability to do simultaeous parameter estimatio ad variable selectio. However, recet research suggests that the traditioal lasso estimator may ot be fully efficiet (Fa ad Li, 200), ad its model selectio result could be icosistet (Leg et al., 2006; Yua ad Li, 2007; Zou, 2006). The major reaso accoutig for such a deficiecy is that lasso applies the same amout of shrikage for each regressio coefficiet. As a simple solutio, Zou (2006) modified the lasso pealty so that differet amouts of shrikage are allowed for differet regressio coefficiets. Such a modified lasso method was referred to as adaptive lasso (Zou, 2006, alasso). It has bee show theoretically that the alasso estimator is able to idetify the true model cosistetly, ad the resultig estimator is as efficiet as oracle. Similar methods were also developed for Cox s proportioal hazard model (Zhag ad Lu, 2007), least absolute deviatio regressio (Wag et al., 2007a), ad liear regressio with autoregressive residuals (Wag et al., 2007b). Both the origial lasso ad adaptive lasso were desiged to select variables idividually. However, there are situatios where it is desirable to choose predictive variables i a grouped maer. The multifactor aalysis-of-variace model is a typical example. To this ed, Yua & Li (2006) developed the group lasso (glasso) method, which pealizes the grouped coefficiets i a similar maer to lasso. Hece, it is expected that glasso i Yua & Li (2006) suffers from the estimatio iefficiecy ad selectio icosistecy i the same way as lasso. As a remedy, we propose the

2 adaptive group lasso (aglasso) method. It is similar to adaptive lasso but has the capability to select variables i a grouped maer. We show theoretically that the proposed aglasso estimator is able to idetify the true model cosistetly, ad the resultig estimator is as efficiet as oracle. Numerical studies cofirmed our theoretical fidigs. The rest of the article is orgaized as follows. The aglasso method is proposed i the ext sectio, ad its theoretical properties are established i Sectio 3. Simulatio results are reported i Sectio 4 ad oe real dataset is aalyzed i Sectio 5. The, the article cocludes with a short discussio i Sectio 6. All techical details are preseted i the Appedix. 2 Adaptive Group Lasso 2. Model ad Notatios Let (x, y ),, (x, y ) be a total of idepedet ad idetically distributed radom vectors, where y i R is the respose of iterest ad x i R d is the associated d-dimesioal predictor. Furthermore, it is assumed that x i ca be grouped ito p factors as x i = (x i,, x ip ), where x ij = (x ij,, x ijdj ) R d j is a group of d j variables. I such a situatio, it is practically more meaigful to idetify importat factors istead of idividual variables (Yua ad Li, 2006). I order to model the depedece relatioship betwee the respose y i ad x i, the followig typical liear regressio model is assumed y i = x ijβ j + e i = x i β + e i, where β j = (β j,, β jdj ) R d j is the regressio coefficiet vector associated with the jth factor ad β is defied to be β = (β,, β p ). Without loss of geerality, we assume that oly the first p 0 p factors are relevat (i.e, β j = 0 for j p 0 ad β j = 0 for j > p 0 ). 2.2 The aglasso Estimator For simultaeous parameter estimatio ad factor selectio, Yua ad Li (2006) proposed the followig pealized least squares type objective fuctio with the group lasso (glasso) pealty i= ( y i 2 ) 2 x ijβ j + λ β j, where stads for the typical L 2 orm. Note that if the umber of variables cotaied i each factor is ideed oe (i.e., d j = ), the above glasso objective fuctio reduces to the usual 2

3 lasso. However, if there do exist some factors cotaiig more tha oe variable, the above glasso estimator has the capability to select those variables i a grouped maer. As oe ca be see, glasso pealizes each factor i a very similar maer as the usual lasso. I other words, same tuig parameter λ is used for each factor without assessig their relative importace. I a typical liear regressio settig, it has bee show that such a excessive pealty applied to the relevat variables ca degrade the estimatio efficiecy (Fa ad Li, 200) ad affect the selectio cosistecy (Leg et al., 2006; Yua ad Li, 2007; Zou, 2006). Therefore, we ca reasoably expect that glasso suffers the same drawback. To overcome such a limitatio, we borrow the adaptive lasso idea ad propose the followig adaptive group lasso (aglasso) Q(β) = i= ( y i 2 ) 2 x ijβ j + λ j β j. (2.) The, miimizig the above objective fuctio produces the aglasso estimator ˆβ. As ca be see, the key differece betwee the aglasso ad glasso is that the aglasso allows for differet tuig parameters used for differet factors. Such a flexibility i tur produces differet amouts of shrikage for differet factors. Ituitively, if a relatively larger amout of shrikage is applied to the zero coefficiets ad a relatively smaller amout is used for the ozero coefficiets, a estimator with a better efficiecy ca be obtaied. For practical implemetatio, oe usually does ot kow which factor is importat ad which oe is ot. However, without such prior kowledge, simple estimators of λ j ca be obtaied i a similar maer to Zou (2006). With those estimated tuig parameters, we are able to show theoretically that the proposed aglasso estimator ca ideed idetify the true model cosistetly ad the resultig estimator is as efficiet as oracle. 2.3 Tuig Parameter Selectio For practical implemetatio, oe has to decide the values of the tuig parameters (i.e, λ j ). Traditioally, cross-validatio (CV) or geeralized cross-validatio (GCV) have bee widely used. However, those computatioally itesive methods ca hardly be useful for aglasso, simply because there are too may tuig parameters. As a simple solutio (Zou, 2006; Wag et al., 2007b; Zhag ad Lu, 2007), we cosider λ j = λ β j γ, (2.2) where β = ( β,, β p ) is the upealized least squares estimator ad γ > 0 is some prespecified positive umber. For example, γ = is used for our simulatio study ad real data aalysis. The, the origially p-dimesioal tuig parameter selectio problem for (λ,, λ p ) reduces to a uivariate problem for λ oly. Thereafter, ay appropriate selectio method ca be 3

4 used. I our umerical studies, the followig selectio criteria were cosidered: Y X ˆβ 2 Cp = σ df Y X GCV = ˆβ 2 ( df) 2 (2.3) ( ) 2 AIC = log Y X ˆβ + 2df/ ( ) 2 BIC = log Y X ˆβ + log df/. Note that df is the associated degrees of freedom as defied i Yua ad Li (2006), give by df = { I ˆβ } j > 0 + ˆβ j β j (d j ), where σ 2 = Y X β 2 /( df) is the usual variace estimator associated with β. 3 Theoretical Properties The mai theoretical properties of the proposed aglasso estimator are established i this sectio. For the purpose of easy discussio, we defie a = max{λ j, j p 0 } ad b = mi{λ j, j > p 0 }. Theorem. (Estimatio Cosistecy) If a p 0, the ˆβ β = O p ( /2 ). Note that p deotes covergece i probability. From Theorem we kow that, as log as the maximal amout of the shrikage applied to the relevat variables is sufficietly small, -cosistecy is assured. Next, we establish the cosistecy of the aglasso estimator as a variable selectio method. To facilitate discussio, some otatios eed to be defied. Let β a = (β,, β p 0 ) to be the vector cotaiig all the relevat factors, ad let β b = (β p 0 +,, β p ) be the vector cotaiig all the irrelevat factors. Furthermore, let ˆβ a ad ˆβ b be their associated aglasso estimators. If oe kows the true model, the oracle estimator ca be obtaied, which is deoted by β a. Stadard liear model theory implies that ( β a β a ) d N(0, Σ a ) where Σ a is the d 0 = p 0 d j dimesioal covariace matrix of the fisrt p 0 relevat factors. Theorem 2. (Selectio Cosistecy) If a p 0 ad b p, the P ( ˆβ b = 0). Accordig to Theorem 2, we kow that, with probability tedig to oe, all the zero coefficiets must be estimated exactly as 0. O the other had, by Theorem, we kow that the estimates for the ozero coefficiets must be cosistet. Such a cosistecy implies that, with probability tedig to oe, all the relevat variables must be idetified with o-zero coefficiets. Both Theorem ad Theorem 2 imply that aglasso does have the ability to idetify the true model cosistetly. 4

5 Theorem 3. (Oracle Property) If a 0 ad b, the ( ˆβ a β a ) d N(0, Σ a ). Note that d deotes covergece i distributio. By Theorem 3, we kow that, as log as the coditios a 0 ad b are satisfied, the resultig estimator is as efficiet as oracle. O the other had, oe ca verify easily that, as log as /2 λ 0 ad (+γ)/2 λ, the theorem coditios a 0 ad b are satisfied. 4 Simulatio Study Simulatio studies were coducted to evaluate the fiite sample performace of aglasso. For compariso purpose, the performace of both alasso ad glasso were also evaluated. For each simulated dataset, various selectio criteria defied i (2.3) were tested. All the examples reported i this sectio were obtaied from Yua ad Li (2006). Example. I this example, 5 latet variables Z,..., Z 5 were geerated accordig to a zero mea multivariate ormal distributio, whose covariace betwee Z i ad Z j was fixed to be 0.5 i j. Subsequetly, Z i was trichotomized as 0,, or 2 if it is smaller tha Φ (/3), larger tha Φ (2/3), or i betwee. The, respose Y is geerated from Y =.2I(Z = 0) +.8I(Z = ) + 0.5I(Z 3 = 0) + I(Z 3 = ) + I(Z 5 = 0) + I(Z 5 = ) + ɛ, where I( ) is the idicator fuctio ad the residual ɛ was ormally distributed with mea 0 ad stadard deviatio σ. For a relatively complete evaluatio, various sample sizes (i.e., = 50, 00, 50, 200, 250) ad various oise levels (i.e., σ = 0.5,.0, 2.0) were tested. For each parameter settig, 200 datasets were simulated ad the media relative model error (MRME) was summarized (Fa ad Li, 200). For each selectio method, the percetage of the 200 simulated datasets, at which the true model is correctly idetified, was computed. Lastly, the average model size (i.e., the umber of factors) were compared. Due to the fact that the simulatio results for GCV, Cp, ad AIC are very similar, oly the results for Cp ad BIC are preseted i Figure ad 2, respectively. We fid that aglasso clearly stads out to be the best estimator for every performace measure, almost every sample size, ad every selectio criterio. Example 2. I this example, we geerated 20 covariates X,..., X 20 i the same fashio as i Example. However, oly the last 0 covariates X,..., X 20 were trichotomized i the same maer as described i the first example. The true regressio model was fixed to be Y = X 3 + X X X 6 X X I(X = 0) + I(X = ) + ɛ, where ɛ N(0, σ 2 ). The, the three competig methods (i.e., alasso, glasso, aglasso) were compared at differet sample sizes (i.e., = 00, 200, 300, 400) ad differet oise levels (i.e., 5

6 σ = 0.5,.0, 2.0). The results were summarized i Figure 3 ad Figure 4. As oe ca see, the results are very similar to that of Example. Table : Model Selectio Compariso for Teachig Evaluatio Data No. of Outsample MSE Selectio Factors Selected ( 0 ) Method aglasso alasso glasso aglasso alasso glasso Cp GCV AIC BIC The Teachig Evaluatio Data I order to demostrate the usefuless of aglasso i real situatio, we preset i this sectio oe real example. The data is about the teachig evaluatio scores collected from a total of 340 courses taught i Pekig Uiversity. For each observatio, the respose of iterest is the teachig evaluatio score for oe particular course, taught i Pekig Uiversity durig the period of There is oly cotiuous predictor, which is the log-trasformed class size (i.e., how may studets erolled i the class). Due to the suspicio of some oliear relatioship, a third-order polyomial is used to fully characterize the class size effect. I additio to that, there are 5 differet categorical variables, which are suspected to have explaatory power for the respose. They are, respectively, the istructor s title (assistat professor, associate professor, ad full professor), the istructor s geder (male or female), the studet type (MBA, Udergraduate, ad Gradate), the semester (Sprig or Fall), ad the year (2002, 2003, ad 2004). For a fair evaluatio, we radomly split the 340 observatios ito two parts. Oe part cotais a total of 300 observatios, which are used to build the model. The other part cotais the remaiig 40 observatios, which are used to evaluate the outsample forecastig error. For a reliable compariso, we repeated such a procedure a total of 00 times with the key fidigs reported i Table. As oe ca see, regardless of which selectio method was used, the optimal model selected by aglasso cosistetly demostrated the smallest average model size ad the best predictio accuracy. 6 Discussio I this article, we propose the aglasso method for adaptive grouped variable selectio. Both our simulatio ad real data experiece suggest that aglasso ca outperform alasso ad glasso 6

7 substatially. Our prelimiary results are rather ecouragig ad the extesio to geeralized liear models should be straightforward. Appedix Proof of Theorem : Note that the aglasso objective fuctio Q(β) is a strictly covex fuctio. Hece, as log as we ca show that there is a local miimizer of (2.), which is cosistet, the by the global covexity of (2.), oe kows immediately that such a local miimizer must be ˆβ. Hece, the -cosistecy of ˆβ is established. Followig a similar idea i Fa ad Li (200), the existece of a -cosistet local miimizer is implied by that fact that for ay ɛ > 0, there is a sufficietly large costat C, such that lim if P { if Q(β + /2 u) > Q(β) u R d : u =C } > ɛ. (A.) For simplicity, defie the respose vector as Y = (y,, y ) ad the desig matrix as X = (x,, x ). It follows the that Q(β + /2 u) Q(β) = 2 Y X(β + /2 u) 2 + λ j β j + /2 u 2 Y Xβ 2 = ( ) ( ) 2 u X X u u X (Y Xβ) + = ( ) ( ) 2 u X X u u X (Y Xβ) + ( ) ( ) 2 u X X u u X (Y Xβ) λ j β j λ j β j + /2 u λ j β j p 0 λ j β j + /2 u λ j β j (A.2) p 0 + λ j ( β j + /2 u β j ) ( ) ( ) 2 u X X u u X (Y Xβ) p 0 ( a ) u, (A.3) where the equality (A.2) holds because β j = 0 for ay j > p 0 accordig to the model assumptio. Furthermore, accordig to the theorem s coditio, we kow that a = o p (), hece, the third term i (A.3) is o p (). O the other had, the first term coverges i probability to u cov(x)u, which is a quadratic fuctio i u. Last, the secod term i (A.3) is liear i u with a O p () coefficiet. Therefore, whe C is sufficietly large, the first term domiates the other two terms with a arbitrary large probability. This implies (A.) ad completes the proof. Proof of Theorem 2: Without loss of geerality, we show i detail that P ( ˆβ p = 0). 7

8 The, the same argumet ca be used to show that P ( ˆβ j = 0) for ay p 0 < j < p, which implies immediately that P ( ˆβ b = 0). For a better discussio, we defie X p be a (d d p ) matrix with the ith row give by (x i,, x i(p ) ), the desig matrix without the pth factor. Similarly, we defie X p to be the d p desig matrix with the ith row give by x ip. Next, we defie β p = (β,, β p ) ad let ˆβ p be its associated estimator. Note that if ˆβ p 0, the the pealty fuctio ˆβ p becomes a differetiable fuctio with respect to its compoets. Therefore, ˆβ p must be the solutio of the followig ormal equatio 0 = ) X p (Y X p ˆβ p X p ˆβp + ˆβp λ p ˆβ p = ( ) ( ) X p Y Xβ + (β p ˆβ ) p ( + X p X p X p X p ) (β p ˆβ ) p + λ p ˆβp ˆβ p, (A.4) where the first term i (A.4) is of the order O p (), ad the secod ad the third termd are also of the same order because β p ˆβ p = O p ( /2 ) ad β p ˆβ p = O p ( /2 ) accordig to Theorem. Next ote that if ˆβ p 0, the there must exist a k such that ˆβ pk = max{ ˆβ pk : k d p }. Without loss of geerality we ca assume that k =, the we must have ˆβ p / ˆβ p / d p > 0. I additio to that, ote that λ p b. Therefore, we kow that λ p ˆβpk / ˆβ j domiates the first three terms i (A.4) with probability tedig to oe. This simply meas that (A.4) caot be true as log as the sample size is sufficietly large. As a result, we ca coclude that with probability tedig to oe, the estimate ˆβ p must be i a positio where ˆβ p is ot differetiable. Hece, ˆβ p has to be exactly 0. This completes the proof. Proof of Theorem 3: Based o the results of Theorem ad Theorem 2, we kow that, with probability tedig to oe, we must have ˆβ j 0 for j p 0 ad ˆβ j = 0 for j > p 0. The, we kow that, with probability tedig to oe, ˆβ a must be the solutio of the followig ormal equatio X a (Y X a ˆβa ) + D( ˆβ a ) = 0, where D( ˆβ a ) = (λ ˆβ / ˆβ,, λ p0 ˆβ p0 / ˆβ p0 ). It follows the ( ) ( ( ˆβa β a ) = X a X a Xa (Y X a β a ) + ) D( ˆβ a ). (A.5) Due to the fact that λ j a p 0 for ay j p 0 ad ˆβ jk / ˆβ j < for ay k d j, we kow that D( ˆβ a ) = o p ( /2 ). Therefore, (A.5) ca be further writte as ( ) ( ) ( ˆβa β a ) = X a X a Xa (Y X a β a ) + o p () d N(0, Σ a ). 8

9 The theorem s coclusio follows ad this completes the proof. Refereces Efro, B., Hastie, T., Johstoe, I., ad Tibshirai, R. (2004), Least agle regressio, The Aals of Statistics, 32, Fa, J. ad Li, R. (200), Variable selectio via ococave pealized likelihood ad its oracle properties, Joural of the America Statistical Associatio, 96, Fu, W. J. (998), Pealized regressio: the bridge versus the LASSO, Joural of Computatioal ad Graphical Statistics, 7, Kight, K. ad Fu, W. (2000), Asymptotics for lasso-type estimators, The Aals of Statistics, 28, Leg, C., Li, Y., ad Wahba, G. (2006), A ote o lasso ad related procedures i model selectio, Statistica Siica, 6, Tibshirai, R. J. (996), Regressio shrikage ad selectio via the LASSO, Joural of the Royal Statistical Society, Series B, 58, Wag, H., Li, G., ad Jiag, G. (2007a), Robust regressio shrikage ad cosistet variable selectio via the LAD-LASSO, Joural of Busiess ad Ecoomics Statistics, To Appear. Wag, H., Li, G., ad Tsai, C. L. (2007b), Regressio coefficiet ad autoregressive order shrikage ad selectio via lasso, Joural of Royal Statistical Society, Series B, To Appear. Yua, M. ad Li, Y. (2006), Model selectio ad estimatio i regressio with grouped variables, Joural of the Royal Statistical Society, Series B, 68, (2007), O the oegative garrote estimator, Joural of the Royal Statistical Society, Series B, To appear. Zhag, H. H. ad Lu, W. (2007), Adaptive lasso for Cox s proportioal hazard model, Biometrika, To appear. Zou, H. (2006), The adaptive lasso ad its oracle properties, Joural of the America Statistical Associatio, 0,

10 Figure : Model with Cp as the Selectio Criterio σ =, MRME (%) σ = 0.5, MRME (%) aglasso alasso glasso

11 Figure 2: Model with BIC as the Selectio Criterio σ =, MRME (%) σ = 0.5, MRME (%) aglasso alasso glasso

12 Figure 3: Model 2 with Cp as the Selectio Criterio σ = 2, MRME (%) σ =, MRME (%) aglasso alasso glasso

13 Figure 4: Model 2 with BIC as the Selectio Criterio σ = 2, MRME (%) σ =, MRME (%) aglasso alasso glasso