Regularity conditions

Transcription

1 Statistica Siica: Supplemet MEASUREMENT ERROR IN LASSO: IMPACT AND LIKELIHOOD BIAS CORRECTION Øystei Sørese, Aroldo Frigessi ad Mage Thorese Uiversity of Oslo Supplemetary Material This ote cotais proofs of theorems ad propositios stated i the paper S Regularity coditios We assume xed true covariates which satisfy /X X = C xx Σ xx, as, / max i x ix i, as, S S where Σ xx is a positive deite matrix The radom measuremet errors are assumed ormally distributed with mea zero ad covariace Σ uu It follows Aderso 3, Th 344 that the limitig distributio of / C uu Σ uu is ormal with mea ad covariaces Σ uu ik Σ uu jl + Σ uu il Σ uu jk, where Σ uu ik is the i, kth elemet of Σ uu ad i, j, k, l {,, p} Now, C uu Σ uu, as, / max i u iu i, as, S3 S4 hold with probability It follows from S-S4 that with probability, C ww Σ ww ad / max i w i w i, as The limitig distributio of / C ww Σ ww has mea zero ad ite covariaces Regularity coditios like these have also bee assumed by, eg, Kight ad Fu ad Zhao ad Yu 6

2 S S Ø Sørese, A Frigessi, M Thorese Karush-Kuh-Tucker Coditios We itroduce the ew coeciet γ = β β, which yields the aive lasso o the form ˆγ = arg mi ɛ Wγ + γ C ww γ + γ C wu β + λ γ + β γ, S where we have removed all terms which are costat i γ Takig derivatives, we arrive at the Karush-Kuh-Tucker KKT coditios for the aive Lasso Lemma ˆγ = ˆβ β is a solutio to S if ad oly if /ɛ W + C ww ˆγ + C wu = λˆτ, where ˆτ R p satises ˆτ ad ˆτ j = sig ˆβj for j such that ˆβ j The same chage of variables for the corrected lasso yields { ˆγ = arg mi γ : γ+β R ɛ Wγ + γ C ww Σ uu γ S + γ C wu Σ uu β + λ γ + β } Due to the additioal costrait γ + β R added because of o-covexity, the KKT coditios ca oly characterize critical poits i the iterior of this domai A critical poit o the boudary may ot have a zero subgradiet Uder the assumptios of Loh ad Waiwright, for sucietly large, all local optima lie i a small l -ball aroud β We assume that R is chose large eough such that γ + β < R for all these optima, while R is small eough to avoid the trivial solutios for which oe or more compoet of ˆγ is ± Lemma Assume ˆγ = ˆβ β is a critical poit of S If ˆγ + β < R, the /ɛ W + C ww Σ uu ˆγ + C wu Σ uu = λˆτ, where ˆτ R p is as deed i Lemma S3 Proof of Propositio By deitio Bühlma ad va de Geer, / y Wˆβ + λ ˆβ / y Wβ + λ β, ad after reorgaizig terms, / W ˆβ β + λ ˆβ / ɛ Uβ W ˆβ β + λ β S3 Uder S3, / ɛ Uβ W ˆβ β / ɛ Uβ W ˆβ β λ ˆβ β,

3 Measuremet Error i Lasso S3 which iserted ito S3 yields / W ˆβ β + λ ˆβ λ ˆβ β + λ β Now use, ˆβ β S ˆβ S β S + ˆβ S c, ˆβ β = ˆβ S β S + ˆβ S c, ad λ λ, to obtai / W ˆβ β + λ ˆβ S c 3λ ˆβ S β S S3 S33 Iequality S33 shows that ˆβ S c 3 ˆβ S β S That is, the vector ˆβ β is amog the vectors to which the compatibility coditio applies, for the idex set S Next, use S3 agai i S33 to obtai / W ˆβ β + λ ˆβ β 4λ ˆβ S β S S34 Uder the compatibility coditio o S, ˆβ S β S s / φ W ˆβ / β Usig this ad the iequality 4uv 4u + v i S34, we arrive at W ˆβ β + λ ˆβ β S 4λ s /φ S4 Proof of Propositio This proof goes alog the lies of the proof of Theorem i Kight ad Fu, but with the additio of measuremet error We start with the aive Lasso after reparametrizatio, ad let L γ = γ W ɛ + γ C ww γ + γ C wu β + λ γ + β S45 Note that / W/ ɛ d N, 4/σ Σ ww ; the rst term i S45 coverges i distributio to a ormally distributed quatity whose variace goes to zero as /, which is equivalet to covergece i probability to zero Combiig this result with the assumptio that λ as, yields L γ p Lγ = γ Σ ww γ + γ Σ uu β Sice L γ is covex, it follows that argmi{l γ} p argmi{lγ} Kight ad Fu γ γ The miimum of Lγ is easily foud, ad accordigly, ˆγ p Σ wwσ uu β The result follows immediately

4 S4 Ø Sørese, A Frigessi, M Thorese S5 Proof of Theorem We follow the structure of the proof by Zhao ad Yu 6, who proved the correspodig result i the absece of measuremet error Cosider the aive lasso, ad ote that { sig β S ˆγS > β } S {sig = sig ˆβS β } S ad ˆγ S c = ˆβ S c = Thus, by the KKT coditios for the aive lasso Lemma, if a solutio ˆγ exists, ad W S ɛ + C ww S, S ˆγ S + C wu S, S β S = λ sig β S, S5 ˆγ S < β S, S5 Sc W ɛ + C ww S c, S ˆγ S + C wu S, c S β S λ, S53 the sigˆβ S = sigβ S ad sigˆβ S c = Evet A implies the existece of ˆγ S < β S such that Z ɛ Z β S = ˆγ S λ C ww S, S sig β S Buth the there must also exist ˆγ S < β S such that Z ɛ Z β S = ˆγ S λ C ww S, S sig β S, which essetially meas choosig the appropriate sigs of the elemets of ˆγ S Multiplyig through by C ww S, S ad reorgaizig terms, we get S5 Thus, A esures that S5 ad S5 are satised Next, addig ad subtractig C ww S, c S ˆγ S to the left-had side of evet B ad the usig the triagle iequaltity, yields W S c ɛ + + C ww S c, S ˆγ S + C wu S, c S β S C wws, c S C ww S, S W S ɛ + C ww S c, S C ww S, S C wu S, S β S + C ww S c, S ˆγ S λ θ The secod term o the left-had side of this expressio is the left-had side of S5 multiplied by C ww S, c S C ww S, S It ca thus be replaced by the right-had side of S5 multiplied by this factor This yields W S c ɛ + + C ww S c, S ˆγ S + C wu S, c S β S

5 Measuremet Error i Lasso S5 λ C wws, c S C ww S, S sig β S λ θ, which implies, due to the IC-ME, W S c ɛ + C ww S c, S ˆγ S + C wu S, c S β S λ This is ideed S53 Altogether, A implies S5 ad S5, while B A implies S53 For the asymptotic result, dee the vectors We have z = C ww S, S W S ɛ, a = β S Cww S, S C wu S, S β S, b = C ww S, S sig β S, ζ = C ww S, c S C ww S, S W S W S c ɛ, f = C ww S c, S C ww S, S C wu S, S C wu S c, S β S P A B P A c + P B c s P z j a j λ p s b j + P ζ j f j λ θ It is clear that z d N, σ C ww S, S, as Hece, there exists a ite costat k such that Ez j < k for j =,, s Next, we have by assumptio a β S Σ ww S, S Σ uu S, S β S, as, b Σ ww S, S sig β S, as Now usig the assumptio λ = o, we get P A c s + o zj P k < k a j + o s = o exp c, Φ s a j + o where we used the boud for the Gaussia tail probability Φt < t exp /t S54

6 S6 Ø Sørese, A Frigessi, M Thorese Next, we ote that ζ d N, σ Σ ww S c, S c Σ ww S c, S Σ ww S, S Σ ww S, S c, as Next, we cosider f, ad ote that the limitig distributio of C wu = C uu +C xu as is ormal with mea Σ wu = Σ uu ad ite variaces Aderso 3, Th 344 I additio, C ww Σ ww as Thus, applyig Slutsky's theorem to the product of the matrices, the limitig distributio of Cww S c, S C ww S, S C wu S, S C wu S c, S as has mea Σww S c, S Σ ww S, S Σ wu S, S Σ wu S c, S = ad ite variaces The latter term equals zero by the MEC Now f = Cww S c, S C ww S, S C wu S, S C wu S c, S β S is a vector i R p s whose elemets are liear combiatios of variables whose limitig distributios as are ormal with mea zero ad ite variaces Accordigly, the limitig distributio of f as is ormal with mea zero ad ite variaces So agai there exists a ite costat k such that Eζ j f j < k for j =,, p s Thus, whe λ c/ for c [,, we have P B c p s + o ζj f j P < λ θ k k p s = o exp c It follows that P A B = oexp c λ Φ θ k S6 Proof of Propositio 3 We cosider { ow the Lasso with ɛ = I this case, y = Xβ, ad the Lasso becomes ˆβ = argmi Wβ Xβ } + λ β We follow the proof of Bühlma ad va de β Geer, Th 7, but also take measuremet error ito accout Part The KKT coditios take the form C ww S, S ˆβ S β S + C ww S, S c ˆβ S c + C wu S, S β S = λˆτ S S6

7 Measuremet Error i Lasso S7 C ww S c, S ˆβ S β S + C ww S c, S c ˆβ S c + C wu S c, S β S = λˆτ S c, S6 where ˆτ = ˆτ S, ˆτ S c has the properties ˆτ ad ˆτ j = sig ˆβ j if β j We multiply S6 by ˆβ S C wws, c S c C ww S, S ad S6 by ˆβ S, ad the subtract c the rst from the secod, to get ˆβ S c Cww S, c S c C ww S, c S C ww S, S C ww S, S c ˆβS + c ˆβ S c Cwu S, c S C ww S, c S C ww S, S C wu S, S β S = λ ˆβ S C wws, c S c C ww S, S ˆτ S ˆβ S ˆτ c S c S63 The matrix term withi the paratheses i the leftmost term is positive semideite, sice it is the Schur complemet of the positive semideite matrix C ww, i which the part C ww S, S is positive deite, sice s < Next, the term withi the paratheses o the right-had side is ˆβ S C ww S, c S C ww S, S ˆτ S ˆβ S C ww S, c S C ww S, S ˆτ S ˆβS c The last iequality follows from the IC-ME, ad is strict wheever ˆβ S c Fially, the secod term o the left-had side of S63 is zero by assumptio Thus, if ˆβ S c, the left-had side of S63 must be egative, which is a cotradictio We thus coclude that ˆβ S c =, ad the KKT coditios S6 ad S6 reduce to C ww S, S ˆβS β S + C wu S, S β S = λˆτ S S64 C ww S, c S ˆβS β S + C wu S, c S β S = λˆτ S c, S65 From S64 we get ˆβ S β λ S = C S S, S ˆτ S + C ww S, S C wu S, S β S λ sup Cww S, S τ S + Cww S, S C wu S, S β S τ S66 Now, if j S det ad ˆβ j =, the ˆβ j βj = β j λ > + sup τ Cww S, S τ S + v j, where v = v,, v p = C ww S, S C wu S, S β S, cotradictig S66 Thus, ˆβ j for j S det

8 S8 Ø Sørese, A Frigessi, M Thorese Part We start by assumig sigˆβ = sigβ Thus, the KKT coditios are S64 ad S65 From S64 we get ˆβ S β S = λ C wws, S ˆτ S C ww S, S C wu S, S β S Isertig this ito S65 yields C ww S, c S C ww S, S ˆτ S + Cww S c λ, S C ww S, S C wu S, S C wu S, c S β S = ˆτ S c, ad the ecessary coditio stated i Propositio 3 follows by deitio S7 Proof of Theorem Startig from the KKT coditios of Lemma, we will redo the steps of the proof of Theorem, but with the isertio of extra terms represetig the correctio for measuremet error The corrected lasso is ot i geeral covex, ad our aalysis will thus cocer ay critical poit ˆγ = ˆβ β i the iterior of the feasible set {γ : γ + β < R} If ˆγ exists, ad W S ɛ + C ww S, S Σ uu S, S ˆγ S + Cwu S, S Σ uu S, S β S = λ sig β S, ˆγ S < β S, W S c ɛ + C ww S c, S Σ uu S, c S ˆγ S + Cwu S c, S Σ uu S c, S β S λ, S7 S7 S73 the sigˆβ S = sigβ S ad sigˆβ S = Evet A i Theorem implies the existece of ˆγ S < β S such that Z6 Z 7 β S = ˆγ S λ C ww S, S Σ uu S, S sig β S But the there must exist ˆγ S < β S such that Z 6 Z 7 β S = ˆγ S λ C ww S, S Σ uu S, S sig β S

9 Measuremet Error i Lasso S9 Multiplyig through by C ww S, S Σ uu S, S ad reorgaizig terms, we get S7 Thus, A esures that S7 ad S7 are satised Next, addig ad subtractig Cww S, c S Σ uu S, c S ˆγ S to the left-had side of evet B ad the usig the triagle iequality, yields Sc W ɛ + C wu S c, S Σ uu S, c S ˆγ S + C wu S, c S Σ uu S, c S β S C ww S, c S Σ uu S, c S C ww S, S Σ uu S, S W S + Cww S, c S Σ uu S, c S S ww S, S Σ uu S, S C wu S, S Σ uu S, S β S + S ww S c, S Σ uu S c, S ˆγ S λ θ The secod term o the left-had side of this expressio is the left-had side of S7 multiplied by C ww S, c S Σ uu S, c S C ww S, S Σ uu S, S It ca thus be replaced by the right-had side of S7 multiplied by this factor This yields Sc W ɛ + C wu S c, S Σ uu S, c S ˆγ S + C wu S, c S Σ uu S, c S β S λ λ C ww S c, S Σ uu S c, S C ww S, S Σ uu S, S sig β S θ, which implies, due to the IC-CL, W S c ɛ + C wu S c, S Σ uu S, c S ˆγ S + C wu S c, S Σ uu S c, S β S λ This is ideed S73 Altogether, A implies S7 ad S7, while B A implies S73 For the asymptotic result, dee the vectors z = C ww S, S Σ uu S, S W S ɛ, a = β S C ww S, S Σ uu S, S C wu S, S Σ uu S, S β S, b = C ww S, S Σ uu S, S sig β S, ζ = C ww S, c S Σ uu S, c S C ww S, S Σ uu S, S W S W S c ɛ,

10 S Ø Sørese, A Frigessi, M Thorese f = We have C ww S c, S Σ uu S c, S C ww S, S Σ uu S, S C wu S, S Σ uu S, S C wu S c, S Σ uu S c, S P A B P A c + P B c s P z j a j λ p s b j + P It is clear that z d N β S ζ j f j λ θ, σ Σ xx S, S Σ ww S, S Σ xx S, S, as Hece, there exists a ite costat k such that Ez j < k for j =,, s Next, we have by assumptio a β S, as Now usig the assumptio λ = o, we get P A c s + o zj P k < k a j + o s = o exp c, where we used the boud S54 Next, we ote that Φ s a j + o ζ d N, σ Σ xx S c, S c Σ xx S c, S Σ xx S, S Σ xx S, S c, as Next, we cosider f, ad ote that the limitig distributio of C wu Σ uu, as, is ormal with mea ad ite variaces Aderso 3, Th 344 I additio, C ww Σ uu Σ xx, as Thus, applyig Slutsky's theorem to the product of the matrices, the limitig distributio of C ww S c, S Σ uu S c, S C ww S, S Σ uu S, S C wu S, S Σ uu S, S C wu S, c S Σ uu S, c S, as, is ormal with mea ad ite variaces Now, f = C ww S c, S Σ uu S c, S C ww S, S Σ uu S, S C wu S, S Σ uu S, S C wu S c, S Σ uu S c, S β S

11 Measuremet Error i Lasso S is a vector i R p s whose elemets are liear combiatios of variables whose limitig distributios as are ormal with mea zero ad ite variaces Accordigly, the limitig distributio of f as is ormal with mea zero ad ite variaces So agai there exists a ite costat k such that Eζ j f j < k for j =,, p s Thus, whe λ c/ for c [,, we have P B c p s + o ζj f j P < λ θ k k p s = o exp c It follows that P A B = oexp c λ Φ θ k Additioal Refereces Aderso, T W 3 A Itroductio to Multivariate Statistical Aalysis, Third Editio Joh Wiley ad Sos, Hoboke