Web Appendix for Joint Variable Selection for Fixed and Random Effects in Linear Mixed-Effects Models

Transcription

1 Web Appendix for Joint Variable Selection for Fixed and Rando Effects in Linear Mixed-Effects Models Howard D. Bondell, Arun Krishna, and Sujit K. Ghosh APPENDIX A A. Regularity Conditions Assue that the data {(X i, Z i, y i ) ; i =,...,} is a rando saple fro a linear ixed-effects odel (2.2) with probability density f(y i X i, Z i, φ) where φ = (β, d, γ ) is a k vector of unknown paraeters. Let L i (φ) = log(f(y i X i, Z i, φ)) denote the contribution of observation i to the log-likelihood function, and is given by L i (φ) = 2 log V i 2 (y i X i β) (V i ) (y i X i β), (A.) where V i = σ 2 (Z i DΓΓ DZ i + I n i ). Let L(φ) = i= L i(φ) and Q(φ) denote the log-likelihood and the penalized log-likelihood as given in (2.4) and (4.), respectively. To present the proof of the theores the following regularity conditions are iposed: (i) Each cluster size n i K, for soe K < and i =,...,. (ii) Let v i = I{Z i is full rank}, where I{A} denotes the indictor function of the event A. Assue that i= v i/ c, for soe 0 < c. (iii) The Fisher inforation atrix I(φ 0 ) knowing φ 20 = 0 is finite and positive definite. (iv) There exists a subset Θ of R k, containing the true paraeter φ 0 such that L i (φ) given in (A.) adits all third order derivatives. Specifically, for φ j = β j and (φ l, φ ) = {(d l, γ ), (d l, γ ), (γ l, γ )}, there exists a function M jl (y i, X i, Z i ) such that 3 L i (φ) β j φ l φ = X V i ij (y φ l φ i X i β) < M jl(y i, X i, Z i ), for all φ Θ, and E φ0 [M jl (y i, X i, Z i )] <. For (φ j, φ l ) = (β j, β l ) and φ is either d or γ there exists a function N jl (y, X, Z) such that 3 β jβ l d L i (φ) = X ij(v i S i V i )X il 3 β jβ l γ L i (φ) = X ij (V i T i V i )X il < N jl(y i, X i, Z i ),

2 for all φ Θ, and E φ0 [N jl (y i, X i, Z i )] <. Here S i and T i denote the partial derivatives of V i with respect to d and γ, respectively, and are given by { } { } S i = Z i (DΓΓ D) Z d i, T i = Z i D (ΓΓ ) DZ dγ i. (A.2) For φ j = d j and (φ l, φ ) = {(d l, d ), (d l, γ ), (γ l, γ )}, 3 L i (φ) d j φ l φ < P jl(y i, X i, Z i ), for all φ Θ, and E φ0 [P jl (y i, X i, Z i )] <. Although it ust be that d j 0 for all j, we allow the estiates to fall outside the boundary of the paraeter space by using the axiu likelihood (ML) ethod as opposed to the REML. Note that condition (i) can be relaxed to allow the cluster sized to also increase without bound. However, this can lead to a faster convergence rate for the fixed effects than that for the rando effects (see, for exaple, Nie, 2007). Appropriate odifications to the theory presented here is then possible, but beyond the scope of the paper. Condition (ii) is a sufficient condition that allows for full inforation regarding each rando effect to grow at order, that will typically hold in practice. However, less strict conditions can be derived. A.2 Proof of Theore Proof. Consider the penalized log-likelihood given in (4.) in a neighborhood of the true value φ 0. Let α = /2 with u 0, and φ = φ 0 + α u. Fixing φ 2 = 0 we show that for a sall enough ǫ > 0 there exists a large constant C such that for sufficiently large, P sup Q φ 0 + α u < Q φ 0 u =C 0 0 ǫ. Note that D (u) Q(φ ) Q(φ 0 ) s = {L(φ 0 + α u) L(φ 0 )} λ w j ( φ j0 + α u j φ j0 ). j= Using a Taylor series expansion we have D (u) = α ( L(φ 0 )) u + 2 u [ 2 L(φ 0 )]uα 2 λ s w j sgn(φ j0 )α u j j= = ( L(φ 0 )) u + 2 u [ 2 L(φ 0 )]u λ s j= w j sgn(φ j0 )u j, (A.3) 2

3 where L(φ 0 ), 2 L(φ 0 ) denote the vector and atrix of the first and second order partial derivatives of L(φ ), respectively, evaluated at φ 0. Fro regularity condition (iv) it follows that ( ) L 6 3/2 i (φ) 0, as, φ j φ l φ φ =φ 0 i= hence the reainder ter vanishes. For L(φ 0 ) the j th partial derivative for each corresponding β, d and γ satisfies { } [ E β j L(φ ) = E { } [ E d j L(φ ) = E { } [ E γ j L(φ ) = E X ()jṽ 2 2 ] () (y X ()β ) [Tr(Ṽ () S j ()) + (y X () β ) (Ṽ () S j ()Ṽ ())(y X () β )] [ Tr(Ṽ () T j () ) + (y X ()β ) (Ṽ () T j ()Ṽ () )(y X ()β )] ] ] = 0, φ =φ 0 where X ()j corresponds to the j th colun of stacked atrix X (), and S j () and T j () diagonal atrices of the partial derivatives of Ṽ () and are given by { } { } S j () = Z () ( d D D Γ Γ ) Z () and T j () = Z () D ( j γ Γ Γ ) j Fro standard arguents we have X ()jṽ () (y X ()β ) 2 [Tr(Ṽ S j () ()) + (y X () β ) (Ṽ S j () ()Ṽ ())(y X () β )] 2 [ Tr(Ṽ T j () () ) + (y X ()β ) (Ṽ T j () ()Ṽ () )(y X ()β )] For 2 L(φ ) we have 2 L(φ 0 ) p I(φ 0 ), D Z (). are block = O p (). (A.4) φ =φ 0 (A.5) where I(φ 0 ) is the Fisher inforation evaluated at φ 0. Using (A.4) and (A.5) the expansion in (A.3) becoes D (u) = O p ()u 2 u {I(φ 0 ) + o p ()}u λ s j= w j sgn(φ j0 )u j. Since I(φ 0 ) is finite and positive definite (condition i), hence choosing a sufficiently large C, the second ter doinates the first ter uniforly in u = C. For the penalty ter, if λ / 0 as, and since w j = /ˆφ j /φ j, it follows that λ s j= w j sgn(φ j0 )u j p 0, and thus is also doinated by the second ter. Hence by choosing a sufficiently large C there exists a local axiu in the ball {(φ 0 + α u, 0) : u C} with probability with ǫ, and hence there exists a local axiizer ˆφ = (ˆφ, 0) of φ 0 = (φ 0, 0) such that ˆφ φ 0 = O p ( /2 ). 3

4 A.3 Proof of Theore 2 Let φ = (β, d, γ ) denote the k vector of unknown paraeters, where k = k β + k d + k γ, the su of the lengths corresponding to each paraeter. Let φ 2 = (β 2, d 2, γ 2) be a vector of length k 2 = k s, corresponding to the true zero values, where k 2 = k β2 + k d2 + k γ2. Proof. It is sufficient to show that with probability tending to as, for any φ satisfying φ φ 0 M /2 and for soe sall ǫ = M /2 and for each j = (s + ),...,(k β2 + k d2 ), we have that Note that Q(φ) < 0 φ j for 0 < φ j < ǫ, Q(φ) > 0 φ j for ǫ < φ j < 0. (A.6) Q(φ) = L(φ) λ w j sgn(φ j ). φ j φ j To show (A.6) consider the Taylor series expansion about L(φ)/φ j, we have φ j Q(φ) = φ j L(φ 0 ) + 2 k k l= k i= l= = φ j φ l L(φ 0 )(φ l φ l0 ) 3 φ j φ l φ L i (φ )(φ l φ l0 )(φ φ 0 ) λ w j sgn(φ j ), (A.7) where φ lies between φ and φ 0. Again the first order partial derivative for j th ter for each β and d are given by β j L(φ 0 ) = X jv 0 (y Xβ 0) = O p (), d j L(φ 0 ) = 0. where X j corresponds to the j th colun of the stacked atrix X. The second order derivatives in (A.7) follows ( ) 2 L(φ) E( 2 L(φ)), φ=φ φ=φ 0 0 where E( 2 L(φ)) is given as E ( 2 L(φ) ) = E L ββ L βd L βγ L βd L dd L dγ L βγ L dγ L γγ, 4

5 where E(L ββ ) = X Ṽ X, and E(L βd ), E(L βγ ) has j th colun } E {L βd = j E[X j (Ṽ SjṼ )(y Xβ)] } E {L βγ = j E[X j (Ṽ T jṽ )(y Xβ)] = 0, φ=φ 0 where S j and T j are block diagonal atrices of S i and T i given in (A.2). The expectation for the second order partial derivatives for d and γ has (j, l) th ter E { L dd }jl = Tr(Ṽ SjṼ Sl ) E { Lγγ } = Tr(Ṽ T jṽ l T jl ) } E {L dγ = Tr(Ṽ SjṼ l T ), (A.8) jl for j = s+,...,(k β +k d ), it can be shown that S j or T j when evaluated at φ j = 0 are zero atrices and the set of equations given in (A.8) siplifies to zero. First, consider φ j = β j the expansion given in (A.7) yields ( ) Q(φ) = k β k O p( /2 ) {X β j j V 0 X d k γ l + o p()}(β l β l0 ) o p()(d l d l0 ) o p()(γ l γ l0 ) l= l=k β + l=k d + k β + k d i= l= =k β + X ij (V i S i V i )X il(β l β l0 )(d d 0 ) + k β + k d k d V i X ij (y 2 d i= l=k β + l=k β + l d i X i β )(d l d l0 )(d d 0 ) k γ i= l= =k d + X ij (V i T i V i )X il (β l β l0 )(γ γ 0 ) + k γ k γ V i X ij (y 2 γ i= l=k d + =k d + l γ i X i β )(γ l γ l0 )(γ γ 0 ) + k d k γ i= l=k β + =k d + (y d l γ i X i β )(d l d l0 )(γ γ 0 ) λ w j sgn(β j ), V i X ij (A.9) where φ φ 0 φ φ 0. Since we are considering φ φ 0 Mn /2, (A.9) gives w j Q(φ) = λ sgn(β j ) + O p (). β j Since for β j0 = 0, we have w j / = β j = O p (), and λ, the sign of the derivative is copletely deterined by that of β j. 5

6 Now consider φ j = d j. The Taylor series expansion in (A.7) gives ( ) Q(φ) = k β k d k γ 0 o p()(β l β l0 ) o p()(d l d l0 ) o p()(γ l γ l0 ) d j l= l=k β + l=k d + + k d k d L i (d j ) (d l d l0 )(d d 0 ) + k γ k γ L i (d j ) (γ l γ l0 )(γ γ 0 ) 2 d i= l=k β + =k β + l d 2 γ i= l=k d + =k d + l γ where k β + k β i= l= = k β + k γ i= l= =k d + X il (V i Sj i V k i )X d k γ il(β l β l0 )(β dβ 0 ) + i= l=k β + =k d + k β k d L i (d j ) (β l β l0 )(d d 0 ), β i= l= =k β + l d L i (d j ) β l γ (β l β l0 )(γ γ 0 ) + L i (d j ) d l γ (d l d l0 )(γ γ 0 ) L i (d j ) = L i (φ d ) = [ Tr(V i j 2 Sj i ) + (y i X i β ) (V i Sj i V i )(y i X i β )], and φ lies between φ and φ 0. As above, (A.0) siplifies to w Q(φ) = λ j sgn(d j ). d j (A.0) Hence, for d j0 = 0, as w j / = d j = O p (), the sign of the derivative is again copletely deterined by that of d j. This copletes the proof. A.4 Proof of Theore 3 Proof. We have fro Theore shown that there exists a ˆφ that is a local axiizer of Q(φ ) such that ˆφ φ 0 = O p ( /2 ), and satisfies the set of penalized likelihood equations Q(φ) φ ˆφ = φ=(,0) L(φ) φ ˆφ λ h(ˆφ ) = 0, φ=(,0) where h(ˆφ ) = ( w sgn(ˆφ ),..., w s sgn(ˆφ s )) an s vector where w j = 0 for φ j = γ j. Using the Taylor series expansion and ultiplying throughout by /, we have L(φ 0) {I(φ 0 ) + o p ()}(ˆφ φ 0 ) λ h(φ 0) = 0 { } (ˆφ φ 0 )I(φ 0 ) + λ h(φ 0) = L(φ 0 ). Since E { L(φ )} = 0 as in the proof of Theore, it follows fro the ultivariate central theore that L(φ 0 ) d N (0, I(φ 0 )), 6

7 where I(φ 0 ) is as given in the proof of Theore. Therefore This copletes the proof. I(φ0 ){(ˆφ φ 0 ) + λ I(φ 0) h(φ 0 )} d N{0, I(φ 0 )}. 7

8 APPENDIX B B. Further coputational details on the EM algorith Oitting ters that do not involve φ, we ay rewrite the expression in (3.3) as β d X X X ZDiag( Γb)( q I ) β ( q I ) Diag( Γb)Z X ( q I ) Diag( Γb)Z ZDiag( Γb)( q I ) d [ ] 2 y X ZDiag( Γb)( q I ) β p β j q + λ d β j + d j d j (B.) j= j= After soe atrix anipulation, part of the lower right block of the atrix in the quadratic for above can be written as Diag( Γb)Z ZDiag( Γb) = W ΓDiag(b) Diag(b) Γ, (B.2) where represents the Hadaard (eleent by eleent) product operator, represents an q vector of ones, and W = Z Z is a syetric block diagonal atrix. Coputing the outer product Diag(b) Diag(b), the expression given in (B.2) further siplifies to W Γbb Γ. Using this siplification, and taking the conditional expectation of (B.) yields a penalized quadratic objective function for (β,d) as g(β,d φ (ω) ) = β d X X X ZDiag( Γˆb (ω) )( q I ) ( q I ) Diag( Γˆb (ω) )Z X ( q I ) (W ) ΓĜ(ω) Γ ( q I ) [ ] 2 y X ZDiag( Γˆb(ω) )(q I ) β { p + λ j= d β d β j β j + q j= } d j d, j where Ĝ(ω) = E(bb ) = U (ω) + ˆb (ω)ˆb (ω), and U (ω) and ˆb (ω) are as given in (3.4). For a fixed γ, we now iniize the objective function (B.3) to obtain the updated estiates for (β,d). 8

9 The next step is, for a fixed (β,d), to obtain a closed for expression for the estiate of γ, the vector that relates to the correlation between the rando effect paraeters. Note that, if γ = 0, then the rando effects are utually independent with the rando effect covariance atrix Ψ being reduced to a siple diagonal for. Furtherore, if d l = 0 then γ lr = 0 for r = l +,...,q. Hence, the eleents of Γ and d are functionally related. Fixing (β,d) at its ost recent update, we first rewrite y Z D Γb Xβ 2 in a quadratic for for γ and then copute its conditional expectation. After a bit of atrix anipulation, and oitting ters not involving γ, we have that the objective function for γ is given by { g(γ φ (ω) ) = γ P (ω) γ 2 (y Xβ) R (ω) T (ω)} γ, (B.3) where P (ω) = E b y,φ (ω) {A A}, T (ω) = E b y,φ (ω)(a Z Db), and R (ω) = E b y,φ (ω)(a). Where A = [A,...,A ] represents a stacked atrix of A i, with each A i an n i q(q )/2 atrix, whose eleents in each row are given by A ij = (b il d r z ijr : l =,...,(q ),r = l +,...,q). Here, A ij denotes the j th row of the i th atrix, which contains q(q )/2 eleents. The appropriate iniizer of (B.3) is then given by γ = (P (ω)) { R (ω) (y Xβ) T (ω)}, (B.4) where A denotes the Moore-Penrose generalized inverse of A. Note that the atrices P (ω), T (ω) and R (ω) only involve first and second oents, i.e. ˆb (ω) and U (ω). The optiization proble is now solved by iniizing the quadratic for (B.3), along with explicit solution (B.4) iteratively to coplete the M-step. The final penalized likelihood estiates, ˆφ = (ˆβ, ˆd, ˆγ ) are obtained by successive EM steps. B.2 Coputation of the M-step Recently, Efron, Hastie, Johnstone and Tibshirani (2004) proposed the LARS (Least Angle Regression) algorith, and showed that it can be used to obtain the entire solution path for LASSO estiates, while being coputationally efficient. Zou (2006) showed that with inor changes to the design atrix, the LARS algorith can be ipleented to obtain the estiates for the regression coefficients under the adaptive LASSO penalty. Although we 9

10 can use the LARS algorith to iniize the penalized quadratic for in our M-step via a pseudo design atrix, it is not as advantageous to obtain the entire solution path here. This is due to the fact that the design atrix changes with every iterative step of the EM algorith. Hence, we propose the use of a standard quadratic prograing technique to obtain the penalized likelihood estiates for our paraeters at each iterative step. Given φ = φ (ω) and a tuning paraeter λ, we write β = β + β with both β + and β being non-negative, and only one is non-zero, and β = β + + β, the optiization proble for (β,d) given in (B.3) is equivalent to iniize [ 2 (y β + β d X ZDiag( Γˆb (ω) )( q I ) X X X ZDiag( Γˆb (ω) )( q I ) ( q I ) Diag( Γˆb (ω) )Z X ( q I ) (W ΓĜ(ω) Γ ) ( q I ) ] [ ]) + λ β,, β, p β,, β, d,, dq p subject to β + β d β + β d β + 0, β 0, d 0. (B.5) where the atrix X = [X X]. The iniization with respect to the expanded paraeter (β +,β,d) is now a direct quadratic prograing proble with 2p + q total paraeters and 2p + q total linear constraints. 0

11 APPENDIX C C. Brief Description of CASTNet Data A coplete description of the data can be found on the EPA website: The data used here is a subset of the coplete data and consists of 826 observation fro 5 relevant sites fro 2000 to 2004 across the eastern United States. The ap shown in Figure arks the relevant sites we have used for this analysis. Here is a list describing briefly our response variable and the 6 predictors. Y LOG(TNO) 3, Log of Total Nitrate Concentration (µol/ 3 ) x SO 4, Sulphate Concentration (µol/ 3 ) x 2 NH 4, Aonia Concentration (µol/ 3 ) x 3 O 3, Maxiu Ozone (ppb, parts per billion) x 4 T, Average Teperature ( C) x 5 T d, Average Due Point Teperature ( C) x 6 RH, Average Relative Huidity (%) x 7 SR, Average Solar Radiation (W/ 2 ) x 8 x 9 WS, Average Wind Speed (/sec) P, Total Precipitation (/onth) l(t) Tie of easureent in onths (,...,60) fro s j (t) c j (t) Sin( 2πjt 2 ) where j =,2,3 Cos( 2πjt 2 ) where j =,2,3 Figure 3 plots the predicted ean overlayed with the observed values of log(tno 3 ) using our penalized likelihood estiates for the fixed effects, for 4 specific sites of interest. Though it sees to fit well, we see that in soe sites it tends to underestiate (DCP 4) while overestiating (PNF 26) in others. This further reiterates that the rando effects are iportant to account for heterogeneity between the sites. We can also see fro Figure 2 that a single cycle (s (t),c (t)) sees to be sufficient to describe the seasonal trend.

12 ANA5 DCP4 MKG3 CDR9 VPI20 CTH0 PSU06 BEL6 SHN48 CAD50 ESP27 PNF26 COW37 GAS53 CND25 Figure : The location of the 5 Sites that were used for our analysis. The represents the 4 sites used for the overlay plots in Figure 3.

13 Log(TNO_3) Month Figure 2: Site (individual) profile plot to assess the seasonal trend in Nitrate concentration over each 2 onth period, for the CASTNet dataset. 3

14 DCP 4 CND 25 log(tno_3) log(tno_3) Months Months GAS 53 PNF 26 log(tno_3) log(tno_3) Months Months Figure 3: Plot of the observed LOG(TNO) 3 concentration represented by the solid line, overlayed with the fitted values for the fixed effects odel selected using our proposed ethod for 4 centers, for the CASTNet dataset. 4