SUPPLEMENTARY MATERIAL TO REGULARIZATION AND THE SMALL-BALL METHOD I: SPARSE RECOVERY

Transcription

1 arxiv: arxiv: SUPPLEMETARY MATERIAL TO REGULARIZATIO AD THE SMALL-BALL METHOD I: SPARSE RECOVERY By Guillaume Lecué, and Shahar Mendelson, CREST, CRS, Université Paris Saclay. Technion I.I.T. and Mathematical Sciences Institute, AU. An inspection of Theorem 3. reveals no mention of an isotropicity assumption. There is no choice of a Euclidean structure, and in fact, the statement itself is not even finite dimensional. All that isotropicity has been used for was to bound the complexity function r and the sparsity function in the three applications the LASSO in Theorem.4, SLOPE in Theorem.6 and the trace norm regularization in Theorem.7. We may apply Theorem 3. to situations that do not involve an isotropic vector and here we give an example of how this may be done. To simplify our presentation we will only consider l and SLOPE regularization, which may both be written as Ψt = d β j t j, j= where β β d > 0 and t t d 0 is the nondecreasing rearrangement of t j. As mentioned previously, the LASSO case is recovered for β = = β d = and the SLOPE norm is obtained for β j = C loged/j for some constant C. We also denote by B Ψ resp. S Ψ the unit ball resp. sphere associated with the Ψ-norm. Let Σ R d d be the covariance matrix of X and set D = {x R d : Σ / x } to be the corresponding ellipsoid. aturally, if X is not isotropic than Σ is not the identity matrix. Supported by Labex ECODEC AR - -LABEX-0047 and by the Chaire Economie et Gestion des ouvelles Données, under the auspices of Institut Louis Bachelier, Havas-Media and Paris-Dauphine. Supported by the Israel Science Foundation.

2 G. LECUÉ AD S. MEDELSO In order to apply Theorem 3., we need to bound from above the expectation of the remum of the Gaussian process indexed by ρb Ψ rd: 0. l ρb Ψ rd = E Σ / G, w where G is a standard Gaussian vector in R d. We also need to solve the sparsity equation that is, find ρ > 0 for which ρ 4ρ /5 where, for every ρ > 0, ρ = inf h, g h ρs Ψ rd g Γ t ρ and Γ t ρ is the collection of all subgradients of Ψ of vectors in t + ρ/0b Ψ. We will show that the same results that have been obtained for the LASSO and SLOPE in Theorem.4 and Theorem.6 actually hold under the following assumption. Assumption 0.. Let j {,..., d} and denote by Σ / j Σ /. Let s {,..., d} and set B s = s j= β j/ j.. There exists σ > 0 such that for all j {,..., d}, Σ / j. For all x 0B s S Ψ D, Σ / x J s x J. the j-th row of σ. SLOPE. We first control the Gaussian mean width in 0. when Ψ is the SLOPE norm. Lemma 0.. Set β j = C loged/j and let Σ be a d d symmetric nonnegative matrix for which max j σ. If D = {x R d : Σ / x Σ / } then E E Proof. ote that j Σ / G, w min { ρ C 3 6σ + r 8 π, r } d Σ / G, w re G, Σ / w / = re G r E G r d. w D

3 REGULARIZATIO AD THE SMALL-BALL METHOD 3 ext, let H : R d R be defined by Hu = Σ / u, w : w ρb Ψ rd and recall that G is the standard Gaussian vector in R d. It is straightforward to verify that HG Σ / G, w ρ w ρb Ψ C max ξ j j d loged/j where we set ξ j d j= = Σ/ G and ξ j d j= is the non-increasing rearrangement of ξ j d j=. Observe that for u, v Rd, Hu Hv Σ / u v, w r u v, w = r u v, w B d implying that H is a Lipschitz function with constant r; thus, it follows from p. in Chapter of [] that π 0. EHG MedHG + r, where MedHG is the median of HG. Hence, to obtain the claimed bound on E w ρbψ Σ / G, w it suffices to establish a suitable upper estimate on the median of max j d ξ j loged/j. With that in mind, let ξ,..., ξ be mean-zero Gaussian variables and assume that for every j =,..., d, 0.3 E expξ j /L e for some L > 0. ote that in our case, ξ,..., ξ satisfying 0.3 for L = 3σ/8. By Jensen s inequality, E exp hence, j j k= 0.4 P r ξ k L j j k= j ξ k L j E exp k= log ed/j ξ k L j d k= ξ E exp k L ed j ; exp log ed/j = j ed. Let q 0 be the integer that satisfies q d < q+. It follows from 0.4 that with probability at least q l ed = q > ed, l=0

4 4 G. LECUÉ AD S. MEDELSO for every l = 0,, q, ξ l l l ξ j L loged/ l. max j d j= Moreover, for l j < l+, we have ξ j ξ and loged/ l loged/j; l also for q j d, we have ξ j ξ and loged/ l 3 loged/j. q Therefore, ξ j 0.5 P r 6L > loged/j, proving the requested bound on MedHG. Observe that up to constant σ, we actually recover the same result as in Equation 5.; therefore, one may choose the same complexity function r as in the proof of Theorem.6. Let us turn to a lower bound on the sparsity function. Lemma 0.. There exists an absolute constant 0 < c < 80 for which the following holds. Let s {,..., d} and set B s = β j/ j. Assume that for every x 80B s S ψ D one has Σ / x / J s x J. Let ρ > 0 and assume further that there is a s-sparse vector in t + ρ/0b Ψ. If 80B s ρ/rρ then ρ = inf h ρs Ψ rρd g Γ t ρ 4ρ h, g 5. Proof. Let h ρs Ψ rρd and denote by h j the non-increasing rearrangement of h j. It follows from the proof of Lemma 4. that 7ρ h, g g Γ t ρ 0 β j h j. Let h,s be the s-sparse vector with coordinates given by h j for j s and 0 otherwise. We have h rρ ρ S Ψ D S Ψ D ρ 80B s

5 REGULARIZATIO AD THE SMALL-BALL METHOD 5 implying that Σ / h h,s. Furthermore, since h j h / j for every j s, we have β j h h j,s Σ B s B / s h B s rρ. Hence, if ρ 80rρB s then ρ 4ρ/5. We thus recover the same condition as in Lemma 4.3, implying that Theorem.6 actually holds under the weaker Assumption 0.: let X be an L-subgaussian random vector whose covariance matrix satisfies Assumption 0.. The SLOPE estimator with regularization parameter λ ξ Lq / satisfies, with probability at least δ exp c 0 L 8, Ψˆt t s ed c 3 ξ Lq log and Σ / ˆt t s c 3 ξ s ed L q log s when c 4 s loged/s and when there is a s-sparse vector close enough to t. The LASSO. Here, for every j d, β j = ; B Ψ = B d ; and B s = s j= / j s. Lemma 0.3. Let Σ be a d d symmetric nonnegative matrix for which Σ / max j j σ. If D = {x R d : Σ / x } then every ρ > 0 and r > 0, E Σ / G, w { min r d, ρσ } loged. w ρb d rd Proof. ote that E Σ / G, w re G, Σ / w / = re G r E G r d. w ρb d rd w D ext, set ξ j d j= = Σ/ G and let ξ j d j= be the non-increasing rearrangement of ξ j. Therefore, ξ,..., ξ d are mean-zero Gaussian variables and satisfy E expξj /L e for L = 3σ/8. It is evident that ξ E ξ d L log E exp ξ L log j E exp L log ed, and therefore, E Σ / G, w E w ρb d rd w ρb d j= Σ / G, w ρe ξ / 3ρσL log ed. 8

6 6 G. LECUÉ AD S. MEDELSO Lemma 0.3 leads to a slightly different result than in the isotropic case Lemma 5.3, and as a consequence, r has to be slightly modified. A straightforward computation shows that rmρ ξ Lq d loged L,q,δ min, ρσ ξ L q and r Qρ L { ρ σ 0 if L d log cld otherwise. and still rρ = max{r M ρ, r Q ρ}. Finally, let us prove the sparsity condition. Lemma 0.4. There exists an absolute constant 0 < c < 80 for which the following holds. Let s {,..., d} and set B s = / j. Assume that for every x 0B s Sl d D one has Σ / x / J s x J. Let ρ > 0 and assume further that there is a s-sparse vector in t + ρ/0b d. If 0B s ρ/rρ then ρ = inf h ρsl d rρd g Γ t ρ 4ρ h, g 5. Proof. Let h ρsl d rρd and denote by h j the non-increasing rearrangement of h j. It follows from the proof of Lemma 4. that g Γ t ρ h, g ρ h j. Let h,s be the s-sparse vector with coordinates given by h j for j s and 0 otherwise. Observe that h rρ ρ Sld D Sl d ρ D, 40B s and therefore Σ / h h,s. It follows that h j s h h Σ B s B / s h B s rρ, and in particular, if ρ 0rρB s then ρ 4ρ/5.

7 REGULARIZATIO AD THE SMALL-BALL METHOD 7 Using the estimate on r and Lemma 0.4, it is evident that when L,q,δ sσ loged, one has ρ 4ρ /5 for loged ρ L,q,δ ξ Lq s and if there is a s-sparse vector in t + ρ /0B d. Finally, one may choose the regularization parameter by setting λ r ρ ρ loged L,q,δ ξ Lq s. It follows that if X is an L-subgaussian random vector that satisfies Assumption 0. then with probability larger than δ exp c 0 L 8, ˆt t loged ρ = c δ ξ Lq s and Σ / ˆt t s loged rρ = c L, δ ξ Lq. References. [] Michel Ledoux. The concentration of measure phenomenon, volume 89 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, RI, 00. ESAE, 3, avenue Pierre Larousse, 945 MALAKOFF. France. Department of Mathematics, Technion, I.I.T., Haifa, Israel