Saisica Siica: Suppleme REGULARIZING LASSO: A CONSISTENT VARIABLE SELECTION METHOD Quefeg Li ad Ju Shao Uiversiy of Wiscosi, Madiso, Eas Chia Normal Uiversiy ad Uiversiy of Wiscosi, Madiso Supplemeary Maerial The supplemeary maerial is orgaized as he follows. I Secio S, we provide addiioal iformaio regardig he simulaio sudies i Secio 5.. I Secio S, we give lemmas for esablishig asympoic resuls whe he precisio mari is sparse. The proofs of all lemmas ad heorems are give i Secio S3. S Addiioal Tables of Simulaio Resuls Table provides he average compuaioal ime i miues for he eigh mehods uder he simulaio seigs. SIS clearly requires he leas compuaioal effor, whereas as well as Scou require much loger compuaioal ime. Bu all mehods ecep CLIME ca be compued uder a reasoable amou of ime for p = 5000 ad = 00. CLIME akes much loger because of iverig a mari of 5000 dimesio. However, 790.8 miues of compuaio may sill be accepable. I a urepored simulaio wih p = 000 ad he same oher seigs, he average compuaioal ime for CLIME is 46.7 miues. Fially, because he esimaio of Σ is a impora sep i, we provide he average Frobeius orms of esimaed Σ ad Ω i he simulaio i Table. I is clear ha S is o a good esimaor of Σ i erms of he Frobeius orm, ˆΣ by hresholdig or ˆΩ by CLIME is a good esimaor whe Σ or Ω is sparse bu o so good whe Σ or Ω is o sparse. S Lemmas Lemma S. Assume codiios C-C ad C3, for ay λ 0, here eis posiive cosas C 4, C 5, C 6 such ha β β > 8 ep C 4 [/s h r q ] q + 4p ep C 5 /s h + 8p ep C6 for ay 0 < < 8M h s h.
S Table : Average compuaioal ime of various mehods i miues p = 5000 ad = 00 Model Model Model 3 Model 4 AT 6.6. 4. 3.6 CLIME 790.8 700. 65.8 758.6 GLASSO.0.7 7.6 8. Scou, 9.6.5 5.8 6.3 LASSO.5 0.6..4 LASSO+T.8 0.8..9 SLSE+T 4..5.3.0 SIS 0.6 0.4 0.6 0.5 Table : The average Frobeius orms of esimaed Σ ad Ω Model Model Model 3 Model 4 S Σ F S = X X/ 68.7 877.5 495.43 570. ˆΣ Σ F AT 50.88 40.7 385.38 40.7 ˆΩ Ω F CLIME 4.0 73.0 49.99 4.03 Lemma S. Assume codiios C, C ad C3, for ay λ 0, here eis posiive cosas C, C, C 3 ad C 4 such ha β [ ] q β > C ep C [/r q s h ] + l τ for ay 0 < < 8M h s h. + C 3 p s l h l l + C 4 p l l, S3 roofs roof of Lemma. From, ˆβ Mj β Mj = = k= ij µ + [ β k ik + σ i ɛ i k= ] ij ik E j k β k + E j k β k k= σ i ij ɛ i + µ ij. The, ˆβ Mj β Mj > ij ik E j k β k > 3 k= S3. + σ i ij ɛ i > + µ 3 ij > 3
S3 Uder codiio C, applyig Lemma of Cai ad Liu 0 o ± ij ik E ij ik gives ha here eis D > 0 ad D > 0 such ha ma jk ij ik E j k > ep D, for all 0 < D. The, uder codiio C, i follows by Boferroi iequaliy ha here eis C > 0 ad C 3 > 0 such ha, for ay j p, ij ik E j k β k > 3 k= M h s h ma ij ik E ij ik k > 3 S3. p ma jk ij ik E j k > 3M h s h p ep C /s h, for all 0 < C 3 s h. Similarly, ma j p ma j p σ i ij ɛ i > ep C, 3 µ ij > ep C, 3 for some C > 0. Therefore, ˆβ Mj β Mj > p ep C /s h + 4 ep C. S3.3 S3.4 S3.5 The, Lemma follows by ˆβ M β M > j= ˆβ Mj β Mj >. roof of Lemma. By Karush-Kuh-Tucker codiios, he soluio β o 8 saisfies ha ˆΣ β ˆβ M = λ Z, S3.6 where Z has he form of, if β j > 0; Z =, if β j < 0; S3.7 [, ], if β j = 0.
S4 Simple algebra from S3.6 yields β β = Σ [ ˆβ M β M λ Z ˆΣ Σβ ˆΣ Σ β β]. S3.8 Hece, β β v p ˆβ M β M + λ Z + ˆΣ Σβ + ˆΣ Σ β β. Equivalely, v p β β ˆΣ Σ β β ˆβ M β M + λ Z + ˆΣ Σβ The, by ˆΣ Σ β β ˆΣ Σ β β, i holds ha β β > = β β >, ˆΣ Σ v p ˆβ M β M + λ Z + ˆΣ Σβ > + + ˆΣ Σ > v p ˆβ M β M > ˆΣ Σ > v p := I + II + III + IV + λ Z > + β β >, ˆΣ Σ > v p By Lemma, here eis posiive cosas C 4 ad C 5 such ha v p, ˆΣ Σ v p + ˆΣ Σβ > I p ep C 4 /s h v p + 4p ep C 5 /v p. S3.9 By he choice of λ, II = 0, whe is sufficiely large. For III, uder codiio C, ˆΣ Σβ ˆΣ Σ β M ˆΣ Σ. The, i follows from Theorem i of Cai ad Liu 0 ha, III + IV ˆΣ Σ > C 6 / p δ r q v p / / q, S3.0 6Mv p for some C 6 > 0. This complees he proof of he lemma.
S5 roof of Theorem. Noe ha M β = M β, ad M β,a M β = j: βj >a } j j: βj >a } j β j > a β β > a [ O ep C 7 log α α3 η + /p δ log / q α α3 η ] / q by Lemma ad he choice of a ad. Similarly, M β M β,/a = j: βj /a } j { j: βj /a β } j β j > a β β > a [ = O ep C 7 log α α3 η + /p δ log / q α α3 η ] / q. This complees he proof of he firs par of Theorem. I paricular, if we choose h = 0, M β M β = j Mβ } j j Mβ { β j β } j β j { j Mβ β } j β j / S3. = O [ep C 8 α α3 η + /p δ / q α α3 η/ q], sice uder C5, by he choice of, mi j Mβ β j > 3 for large eough. O he oher had, M β M β = j Mβ } j > = j Mβ } j β j > S3. = O [ep C 8 α α3 η + /p δ / q α α3 η/ q]. S3. ad S3. ogeher prove he heorem.
S6 roof of Lemma 3. The proof is aalogous o ha of Lemma. C, i holds ha Uder codiio E ij ik E j k l l [ E ij ik l + E ij ik l] = O. The, by Chebyshev Iequaliy ad Theorem i While 960. ma ij ik E j k jk > l E ij ik E j k Therefore, by replacig S3. wih ij ik E j k β k > 3 k= p ma jk ij ik E j k > 3M h s h C 9 ps l h l l, for some C 9 > 0 ad replacig S3.3 ad S3.4 wih ma σ i ij ɛ i > C 0 j p 3 l l, ma µ j p ij > C 0 3 l l, for some C 0 > 0, he res of proof follows from S3.. roof of Lemma 4. From Lemma 3, i holds ha ˆβ M β M > C p s l h vp l l l + C pvp l l l, for some C > 0 ad C > 0. l = O l l. S3.3 Uder codiio C3, i follows from Theorem ii of Cai ad Liu 0 ha, here eiss C 3 > 0 ha ˆΣ Σ > C 3 / p δ r q v p / / q + l τ. S3.4 6Mv p Replacig S3.9 ad S3.0 wih S3.3 ad S3.4 ad observig ha by he choice of λ, λ Z > = 0, whe is sufficiely large, he res of he proof resembles he proof of Lemma. roof of Theorem. The proof is he same as he proof of Theorem by replacig resuls i Lemma wih resuls i Lemma 4.
S7 roof of Lemma S. From S3.6, β = ˆΩ ˆβ M λ ˆΩZ. Recall ha, β = Ωβ M. Hece, β β = ˆΩ ˆβ M Ωβ M λ ˆΩZ. The, β β > ˆΩ ˆβ M Ωβ M > / + λ ˆΩZ > /. S3.5 Sice i holds ha ˆΩ ˆβ M Ωβ M = ˆΩ Ω ˆβ M + Ω ˆβ M β M, ˆΩ ˆβ M Ωβ M > / ˆΩ Ω ˆβ M > /4 + Ω ˆβ M β M > /4. S3.6 The firs iem i S3.6 is bouded by ˆΩ Ω ˆβ M > /4 ˆΩ Ω ˆβ M > /4 ˆβ M β M /8 + ˆβ M β M > /8 ˆΩ Ω β M > /8 ˆβ M β M /8 + + ˆΩ Ω ˆβ M β M > /8 ˆβ M β M /8 ˆβ M β M > /8 ˆΩ Ω > /[8M h s h ] + ˆΩ Ω > + ˆβ M β M > /8. For he secod iem i S3.6, i follows from he assumpio Ω M ha Ω ˆβ M β M > /4 Ω ˆβ M β M > /4 ˆβ M β M > /[4M]. Wihou loss of geeraliy, assume M. The, for ay 0 < < 8M h s h, β β > ˆΩ Ω > /[8M h s h ] + ˆβ M β M > /[4M], S3.7 sice λ ˆΩZ λ ˆΩ Z λ ˆΩ λ Ω M λ. By he choice of λ, whe is sufficiely large, λ ˆΩZ > / = 0. Uder C, i follows by Theorem a of Cai, Liu, ad Luo 0 ha ˆΩ Ω > /[8M h s h ] 4 ep C 4 [/s h r q ] / q, S3.8
S8 for some C 4 > 0. From Lemma, i holds ha ˆβ M β M > /4M p ep C 5 /s h + 4p ep C6. S3.9 S3.7, S3.8 ogeher wih S3.9 proves he lemma. roof of Lemma S. Uder codiios C ad C3, by Theorem ii of Cai, Liu, ad Luo 0. ˆΩ Ω > 8M h C [ ] q ep C [/s h r q ] + l τ. s h From Lemma 3, i holds ha ˆβ M β M > /4M C 3 p s l h l l + C 4 p l l. The res of proof follows by S3.7. roof of Theorem 3 ad Theorem 4. By usig resuls i Lemma S ad Lemma S, he proof is he same as ha of Theorem. roof of 5. Decompose Σ as Is0 Σ Σ =, Σ I p s0 where Σ = B 0. S S = S S I s 0 + S Σ I s0 + Σ. S3.0 I is well kow e.g. see Bickel ad Levia 008 ha for ormally disribued covariaes, ma s ij ρ ij = O log p, i,j p where s ij is he i, jh eleme of S. The, S Σ = ma s 0 s 0<i p j= Hece, S Σ I s0 = O s 0 log p. Moreover, S I s 0 = S I s 0 S. The, s ij ρ ij = O s 0 log p. S I s 0 S I s0 S + S I s 0 I s0 S.
S9 Hece, S I s 0 I s0 S / I s0 S = O s 0 log p. The, S S I s 0 S S I s 0 Σ + S Σ S I s 0 = O s 0 + s 0 log p O s 0 log p = O s 0 log p Sice Σ B + γ, uder he assumpios ha log p 0 ad s 0 is fied, i follows from S3.0 ha S S + γ. Refereces Bickel,. J. ad Levia, E. 008. Covariace regularizaio by hresholdig. The Aals of Saisics 36, 577 604. Cai, T. ad Liu, W. 0. Adapive hresholdig for sparse covariace mari esimaio. Joural of he America Saisical Associaio 06, 67 684. Cai, T., Liu, W., ad Luo, X. 0. A cosraied l miimizaio approach o sparse precisio mari esimaio. Joural of he America Saisical Associaio 06, 594 607. While,. 960. Bouds for he momes of liear ad quadraic forms i idepede variables. Theory of robabiliy ad is Applicaios 5, 30.