OWL to the rescue of LASSO

Transcription

1 OWL to the rescue of LASSO IISc IBM day 2018 Joint Work R. Sankaran and Francis Bach AISTATS 17 Chiranjib Bhattacharyya Professor, Department of Computer Science and Automation Indian Institute of Science, Bangalore March 7, 2018

2 Identifying groups of strongly correlated variables through Smoothed Ordered Weighted L 1 -norms

3 LASSO: The method of choice for feature selection Ordered Weighted L1(OWL) norm and Submodular penalties Ω S : Smoothed OWL (SOWL)

4 LASSO: The method of choice for feature selection

5 Linear Regression in High dimension Question? Let x R d, y R and y = w x + ɛ ɛ N(0, σ 2 ) From D = {(x i, y i ) i [n], x i R d, y i R} can we find w X R n d, y R n X = x 1 x 2. x n, Y Rn

6 Linear Regression in High dimension Least Squares Linear Regression X R n d, y R d : w LS = argmin w R d 1 2 Xw Y 2 2 Assumptions Labels centered : n j=1 y j = 0. Features normalized : x i R d, x i 2 = 1, x i 1 d = 0.

7 Linear Regression in High dimension Least Squares Linear Regression w LS = ( X X ) 1 X Y and E(w LS ) = w Variance of Predictive error: 1E( X (w n LS w ) 2 ) = σ 2 d n

8 Linear Regression in High dimension rank(x ) = d w LS is unique Poor predictive performance, d is close to n

9 Linear Regression in High dimension rank(x ) = d rank(x ) < d w LS is unique Poor predictive performance, d is close to n d > n w LS is not unique.

10 Regularized Linear Regression Regularized Linear Regression X R n d, y R d, Ω : R d R: 1 min w R d 2 Xw y 2 2 s.t. Ω(w) t Regularizer: Ω(w) non-negative Convex function, typically a norm. Possibly non-differentiable.

11 Lasso regression[tibshirani, 1994] Ridge: Ω(w) = w 2 2 Does not promote sparsity Closed form solution Lasso: Ω(w) = w 1 Encourages sparse solutions. Solve convex optimization problem

12 Regularized Linear Regression Regularized Linear Regression X R n d, y R d, Ω : R d R: 1 min w R d 2 Xw y λω(w) Equivalent to the constraint version Unconstrained

13 Lasso: Properties at a glance Computational Proximal methods : IST, FISTA, Chambolle-Pock [Chambolle and Pock, 2011, Beck and Teboulle, 2009]. Convergence rate : O(1/T 2 ) in T iterations. Assumption: availability of proximal operator of Ω (Easy for l 1 ).

14 Lasso: Properties at a glance Computational Proximal methods : IST, FISTA, Chambolle-Pock [Chambolle and Pock, 2011, Beck and Teboulle, 2009]. Convergence rate : O(1/T 2 ) in T iterations. Assumption: availability of proximal operator of Ω (Easy for l 1 ). Statistical properties[wainwright, 2009] Support recovery (Will it recover the true support?). Sample complexity (How many samples needed?). Prediction error (What is the expected error in prediction?).

15 Lasso Model Recovery[Wainwright, 2009, Theorem 1] Setup Support of w Lasso y = Xw + ɛ, ɛ i N (0, σ 2 ), X R n d be S = {j w j 0 j [d]} min w R d 1 2 Xw y λ w 1

16 Lasso Model Recovery[Wainwright, 2009, Theorem 1] Conditions 1. ( 1 X S c X S XS S) X 1 γ, Incoherence with γ (0, 1], ( ) 1 Λ min n X S X S C min 2σ 2 log d λ > λ 0 2 γ n 1 Define M = max i j M ij

17 Lasso Model Recovery[Wainwright, 2009, Theorem 1] Conditions 1. ( 1 X S c X S XS S) X 1 γ, Incoherence with γ (0, 1], ( ) 1 Λ min n X S X S C min 2σ 2 log d λ > λ 0 2 γ n W.h.p, the following holds: ( 1 ŵ S ws (X λ S S/n) X + 4σ/ ) C min 1 Define M = max i j M ij

18 Lasso Model Recovery: Special cases Case: XS X c S = 0. The incoherence condition trivially holds, and γ = 1. The threshold λ 0 is lesser The recovery error is lesser. Case: XS X S = I. C min = 1/n, the largest possible for a given n. Larger C min lesser recovery error.

19 Lasso Model Recovery: Special cases Case: X S c X S = 0. The incoherence condition trivially holds, and γ = 1. The threshold λ 0 is lesser The recovery error is lesser. Case: X S X S = I. C min = 1/n, the largest possible for a given n. Larger C min lesser recovery error. When does Lasso work well? Lasso prefers low correlation between support and non-support columns. Low correlation of columns within support lead to better recovery.

20 Lasso Model Recovery: Implications Setting: Strongly correlated columns in X. Correlation between feature i and feature j ρ ij x i x j Large correlation between X S and X S c Large correlation within X S C min is small. γ is small.

21 Lasso Model Recovery: Implications Setting: Strongly correlated columns in X. Correlation between feature i and feature j ρ ij x i x j Large correlation between X S and X S c Large correlation within X S C min is small. γ is small. The r.h.s. of the bound is large. (loose bound). Hence w.h.p., lasso fails in model recovery. In other words: Lasso solutions differ with the solver used. Solution is not unique typically. The prediction error may not be as worse though [Hebiri and Lederer, 2013].

22 Lasso Model Recovery: Implications Setting: Strongly correlated columns in X. Correlation between feature i and feature j ρ ij x i x j Large correlation between X S and X S c Large correlation within X S C min is small. γ is small. The r.h.s. of the bound is large. (loose bound). Hence w.h.p., lasso fails in model recovery. In other words: Lasso solutions differ with the solver used. Solution is not unique typically. The prediction error may not be as worse though [Hebiri and Lederer, 2013]. Requirements. Need consistent estimates independent of the solver. Preferably select all the correlated variables as a group.

23 Illustration: Lasso under correlation[zeng and Figueiredo, 2015] Setting: strongly correlated features. {1,..., d} = G 1 G k, G m G l =, l m ρ ij x i x j very high ( 1) for pairs i, j G m.

24 Illustration: Lasso under correlation[zeng and Figueiredo, 2015] Setting: strongly correlated features. {1,..., d} = G 1 G k, G m G l =, l m ρ ij x i x j very high ( 1) for pairs i, j G m. Toy Example d = 40, k = 4. G 1 = [1 : 10], G 2 = [11 : 20], G 3 = [21 : 30], G 4 = [31 : 40]. Lasso: Ω(w) = w 1. Sparse recovery. Arbitrarily selects the variables within a group Figure: Original signal Figure: Recovered signal.

25 Possible solutions: 2-stage procedures Cluster Group Lasso [Buhlmann et al., 2013] Identify strongly correlated groups G = {G 1,..., G k } Canonical Correlation. Group selection. Ω(w) = k j=1 α j w Gj 2. Select all or no variables from each group.

26 Possible solutions: 2-stage procedures Cluster Group Lasso [Buhlmann et al., 2013] Identify strongly correlated groups G = {G 1,..., G k } Canonical Correlation. Group selection. Ω(w) = k j=1 α j w Gj 2. Select all or no variables from each group. Goal: Learn G and w simultaneously?

27 Ordered Weighted l 1 (OWL) norms OSCAR 2 [Bondell and Reich, 2008] d Ω O (w) = c i w (i) i=1 c i = c 0 + (d i)µ, c 0, µ, c d > 0. 2 Notation: w (i) : i th largest in w.

28 Ordered Weighted l 1 (OWL) norms OSCAR 2 [Bondell and Reich, 2008] d Ω O (w) = c i w (i) i=1 c i = c 0 + (d i)µ, c 0, µ, c d > 0. OWL [Figueiredo and Nowak, 2016]: c 1 c d 0. 2 Notation: w (i) : i th largest in w.

29 Ordered Weighted l 1 (OWL) norms OSCAR 2 [Bondell and Reich, 2008] d Ω O (w) = c i w (i) i=1 c i = c 0 + (d i)µ, c 0, µ, c d > Figure: Recovered: OWL OWL [Figueiredo and Nowak, 2016]: c 1 c d 0. 2 Notation: w (i) : i th largest in w.

30 Oscar: Sparsity Illustrated Figure: Examples of solutions:[bondell and Reich, 2008] Solutions encouraged towards vertices. Encourages blockwise constant solutions. See [Bondell and Reich, 2008].

31 OWL-Properties Grouping covariates [Bondell and Reich, 2008, Theorem 1], [Figueiredo and Nowak, 2016, Theorem 1] w i = w j, if λ λ 0 ij.

32 OWL-Properties Grouping covariates [Bondell and Reich, 2008, Theorem 1], [Figueiredo and Nowak, 2016, Theorem 1] w i = w j, if λ λ 0 ij. λ 0 ij 1 ρ 2 ij. Strongly correlated pairs grouped early in the regularization path. Groups: G j = {i w i = α j }.

33 OWL-Issues Figure: True model Figure: Recovered: OWL Bias for piecewise constant ŵ Easily understood through the norm balls. Requires more samples to consistent estimation.

34 OWL-Issues Figure: True model Figure: Recovered: OWL Bias for piecewise constant ŵ Easily understood through the norm balls. Requires more samples to consistent estimation. Lack of interpretations for choosing c

35 Ordered Weighted L1(OWL) norm and Submodular penalties

36 Preliminaries: Penalties on the Support Goal Encourage w to have desired support structure. supp(w) = {i w i 0}

37 Preliminaries: Penalties on the Support Goal Encourage w to have desired support structure. supp(w) = {i w i 0} Idea Penalty on support [Obozinski and Bach, 2012]: pen(w) = F (supp(w)) + w p p, p [1, ].

38 Preliminaries: Penalties on the Support Goal Encourage w to have desired support structure. supp(w) = {i w i 0} Idea Penalty on support [Obozinski and Bach, 2012]: pen(w) = F (supp(w)) + w p p, p [1, ]. Relaxation(pen(w)): Ω F p (w) Tightest positively homogenous, convex lower bound.

39 Preliminaries: Penalties on the Support Goal Encourage w to have desired support structure. supp(w) = {i w i 0} Idea Penalty on support [Obozinski and Bach, 2012]: pen(w) = F (supp(w)) + w p p, p [1, ]. Relaxation(pen(w)): Ω F p (w) Tightest positively homogenous, convex lower bound. Example: F (supp(w)) = supp(w)

40 Preliminaries: Penalties on the Support Goal Encourage w to have desired support structure. supp(w) = {i w i 0} Idea Penalty on support [Obozinski and Bach, 2012]: pen(w) = F (supp(w)) + w p p, p [1, ]. Relaxation(pen(w)): Ω F p (w) Tightest positively homogenous, convex lower bound. Example: F (supp(w)) = supp(w) Ω F p (w) = w 1. Familiar! Message: The cardinality function always relaxes to the l 1 norm.

41 Nondecreasing Submodular Penalties on Cardinality Assumptions. Denote F F, if F : A {1,..., d} R is: 1. Submodular [Bach, 2011]. A B, F (A {k}) F (A) F (B {k}) F (B). Lovász extension: f : R d R. (Convex extension of F to R d ). 2. Cardinality based. F (A) = g( A ) (Invariant to permutations). 3. Non Decreasing. g(0) = 0, g(x) g(x 1). Implication: F F F completely specified through g. Example: Let V = {1,..., d}, define F (A) = A V \ A. Then f (w) = i<j w i w j.

42 Ω F and Lovász extension Result: Case p = [Bach, 2010]: Ω F (w) = f ( w ) The l relaxation coincides with the Lovász extension in the positive orthant. To work with Ω F, may use existing results of submodular function minimization. Ω F p not known in closed form for p <.

43 Equivalence of OWL and Lovász extensions: Statement Proposition [Sankaran et al., 2017]: F F, Ω F (w) Ω O (w) 1. Given F (A) = f ( A ), Ω F (w) = Ω O (w), with c i = f (i) f (i 1). 2. Given c 1... c d 0, Ω O (w) = Ω F (w) with f (i) = c c i. Interpretations Gives alternate interpretations for OWL.

44 Equivalence of OWL and Lovász extensions: Statement Proposition [Sankaran et al., 2017]: F F, Ω F (w) Ω O (w) 1. Given F (A) = f ( A ), Ω F (w) = Ω O (w), with c i = f (i) f (i 1). 2. Given c 1... c d 0, Ω O (w) = Ω F (w) with f (i) = c c i. Interpretations Gives alternate interpretations for OWL. Ω F has undesired extreme points [Bach, 2011]. Explains piecewise constant solutions of OWL.

45 Equivalence of OWL and Lovász extensions: Statement Proposition [Sankaran et al., 2017]: F F, Ω F (w) Ω O (w) 1. Given F (A) = f ( A ), Ω F (w) = Ω O (w), with c i = f (i) f (i 1). 2. Given c 1... c d 0, Ω O (w) = Ω F (w) with f (i) = c c i. Interpretations Gives alternate interpretations for OWL. Ω F has undesired extreme points [Bach, 2011]. Explains piecewise constant solutions of OWL. Motivates Ω F p (w) for p <.

46 Ω S : Smoothed OWL (SOWL)

47 SOWL: Definition Smoothed OWL Ω S (w) := Ω F 2 (w).

48 SOWL: Definition Smoothed OWL Ω S (w) := Ω F 2 (w). Variational form for Ω F 2 [Obozinski and Bach, 2012]. ( d 1 Ω S (w) = min η R d 2 + i=1 w 2 i η i + f (η) ).

49 SOWL: Definition Smoothed OWL Ω S (w) := Ω F 2 (w). Variational form for Ω F 2 [Obozinski and Bach, 2012]. ( d 1 Ω S (w) = min η R d 2 + i=1 w 2 i η i + f (η) ). Use OWL equivalance: f ( η ) = Ω F (η) = d i=1 c i η (i), Ω S (w) = min η R d + ( w 2 i 1 d 2 η i i=1 }{{} Ψ(w,η) ) + c i η (i). (SOWL)

50 OWL vs SOWL Case: c = 1 d. Ω S (w) = w 1. Ω O (w) = w 1. Case: c = [1, 0,..., 0] }{{}. d 1 Ω S (w) = w 2. Ω O (w) = w.

51 OWL vs SOWL Case: c = 1 d. Ω S (w) = w 1. Ω O (w) = w 1. Norm Balls Case: c = [1, 0,..., 0] }{{}. d 1 Ω S (w) = w 2. Ω O (w) = w : OWL : SOWL Figure: Norm balls for OWL, SOWL, for different values of c

52 Group Lasso and Ω S SOWL objective (eliminating η): Ω S (w) = k w Gj j=1 i G j c i. Denote η w : denote the optimal η, given w.

53 Group Lasso and Ω S SOWL objective (eliminating η): Ω S (w) = k w Gj j=1 i G j c i. Denote η w : denote the optimal η, given w. Key differences: Groups defined through η w = [δ 1,..., δ }{{} 1,..., δ k,..., δ k ]. }{{} G 1 G k Influenced by the choice of c.

54 Open Questions: 1. Does Ω S promotes grouping of correlated variables as Ω O? Are there any benefits over ΩO?

55 Open Questions: 1. Does Ω S promotes grouping of correlated variables as Ω O? Are there any benefits over ΩO? 2. Is using Ω S computationally feasible?

56 Open Questions: 1. Does Ω S promotes grouping of correlated variables as Ω O? Are there any benefits over ΩO? 2. Is using Ω S computationally feasible? 3. Theoretical properties of Ω S vs Group Lasso?

57 Grouping property Ω S : Statement Learning Problem: LS-SOWL min w R d,η R d + ( w 2 i ) + c i η (i). 1 2n Xw y λ d 2 η i i=1 }{{} Γ (λ) (w,η)

58 Grouping property Ω S : Statement Learning Problem: LS-SOWL min w R d,η R d + ( w 2 i ) + c i η (i). 1 2n Xw y λ d 2 η i i=1 }{{} Γ (λ) (w,η) Theorem: [Sankaran et al., 2017] Define the following: ( ŵ (λ), ˆη (λ)) = argmin w,η Γ (λ) (w, η). ρ ij = x i x j. c = min i c i c i+1.

59 Grouping property Ω S : Statement Learning Problem: LS-SOWL min w R d,η R d + ( w 2 i ) + c i η (i). 1 2n Xw y λ d 2 η i i=1 }{{} Γ (λ) (w,η) Theorem: [Sankaran et al., 2017] Define the following: ( ŵ (λ), ˆη (λ)) = argmin w,η Γ (λ) (w, η). ρ ij = x i x j. c = min i c i c i+1. There exists 0 λ 0 y 2 c (4 4ρ 2 ij ) 1 4, such that λ > λ 0, ˆη (λ) i = ˆη (λ) j

60 Grouping property Ω S : Interpretation 1. Variables η i, η j grouped if ρ ij 1 (Even for small λ). Similar to Figueiredo and Nowak [2016, Theorem 1], which is for absolute values of w.

61 Grouping property Ω S : Interpretation 1. Variables η i, η j grouped if ρ ij 1 (Even for small λ). Similar to Figueiredo and Nowak [2016, Theorem 1], which is for absolute values of w. 2. SOWL differentiates grouping variable η and model variable w.

62 Grouping property Ω S : Interpretation 1. Variables η i, η j grouped if ρ ij 1 (Even for small λ). Similar to Figueiredo and Nowak [2016, Theorem 1], which is for absolute values of w. 2. SOWL differentiates grouping variable η and model variable w. 3. Allows model variance within group. ŵ (λ) i ŵ (λ) j as long as c has distinct values.

63 Illustration: Group Discovery using SOWL Aim: Illustrate group discovery of SOWL. Consider z R d, Compute prox λωs (z). Study the regularization path.

64 Illustration: Group Discovery using SOWL Aim: Illustrate group discovery of SOWL. Consider z R d, Compute prox λωs (z). Study the regularization path Figure: Original signal

65 Illustration: Group Discovery using SOWL Aim: Illustrate group discovery of SOWL. Consider z R d, Compute prox λωs (z). Study the regularization path Figure: Original signal Lasso w OWL w 4 2 Early group discovery. Model variation SOWL w SOWL η Figure: x-axis : λ, y-axis: ŵ.

66 Proximal Methods: A brief overview Proximal operator prox Ω (z) = argmin w 1 2 w z Ω(w) Easy to evaluate for many simple norms. prox λl1 (z) = sign(z) ( z λ) +. Generalization of Projected Gradient Descent

67 FISTA [Beck and Teboulle, 2009] Initialization t (1) = 1, w (1) = x (1) = 0. Steps : k > 1 w (k) ( = prox Ω w (k 1) 1 L f ( w (k 1) ) ). ( t (k) = ( ) t (k 1)) 2 /2. w (k) = w (k) + ( t (k 1) 1 t (k) ) (w (k) w (k 1)). Guarantee Convergence rate O(1/T 2 ). No additional assumptions than IST. Known to be optimal for this class of minimization problems.

68 Computing prox ΩS Problem: prox λω (z) = argmin w 1 2 w z λω(w). w (λ) = prox λωs (z), η (λ) w = argmin η Ψ(w (λ), η).

69 Computing prox ΩS Problem: prox λω (z) = argmin w 1 2 w z λω(w). w (λ) = prox λωs (z), η (λ) w = argmin η Ψ(w (λ), η). Key idea: Ordering of η (λ) w remains same for all λ. 3 3 η log(λ) η λ

70 Computing prox ΩS Problem: prox λω (z) = argmin w 1 2 w z λω(w). w (λ) = prox λωs (z), η (λ) w = argmin η Ψ(w (λ), η). Key idea: Ordering of η (λ) w remains same for all λ. 3 3 η2 1 η log(λ) η w (λ) = (η z λ) λ Same complexity as computing the norm Ω S (O(d log d)). True for all cardinality based F.

71 Random Design Problem setting: LS-SOWL. True model: y = Xw + ε, ( X ) i N (µ, Σ), ε i N (0, σ 2 ). Notation: J = {i wi 0}, η w = [δ1,..., δ1,..., δk }{{},..., δ k]. }{{} G 1 G k 3 D: Diagonal matrix, defined using groups G1,..., G k, and w

72 Random Design Problem setting: LS-SOWL. True model: y = Xw + ε, ( X ) i N (µ, Σ), ε i N (0, σ 2 ). Notation: J = {i wi 0}, η w = [δ1,..., δ1,..., δk }{{},..., δ k]. }{{} G 1 G Assumptions: 3 k Σ J,J is invertible, λ 0, and λ n. 3 D: Diagonal matrix, defined using groups G1,..., G k, and w

73 Random Design Problem setting: LS-SOWL. True model: y = Xw + ε, ( X ) i N (µ, Σ), ε i N (0, σ 2 ). Notation: J = {i wi 0}, η w = [δ1,..., δ1,..., δk }{{},..., δ k]. }{{} G 1 G Assumptions: 3 k Σ J,J is invertible, λ 0, and λ n. Irrepresentability conditions 1. δ k = 0 if J c. 2. Σ J c,j (Σ J,J ) 1 DwJ 2 β < 1. 3 D: Diagonal matrix, defined using groups G1,..., G k, and w

74 Random Design Problem setting: LS-SOWL. True model: y = Xw + ε, ( X ) i N (µ, Σ), ε i N (0, σ 2 ). Notation: J = {i wi 0}, η w = [δ1,..., δ1,..., δk }{{},..., δ k]. }{{} G 1 G Assumptions: 3 k Σ J,J is invertible, λ 0, and λ n. Irrepresentability conditions 1. δ k = 0 if J c. 2. Σ J c,j (Σ J,J ) 1 DwJ 2 β < 1. Result[Sankaran et al., 2017] 1. ŵ p w. 2. P( ˆ J = J ) 1. 3 D: Diagonal matrix, defined using groups G1,..., G k, and w

75 Random Design Problem setting: LS-SOWL. True model: y = Xw + ε, ( X ) i N (µ, Σ), ε i N (0, σ 2 ). Notation: J = {i wi 0}, η w = [δ1,..., δ1,..., δk }{{},..., δ k]. }{{} G 1 G Assumptions: 3 k Σ J,J is invertible, λ 0, and λ n. Irrepresentability conditions 1. δ k = 0 if J c. 2. Σ J c,j (Σ J,J ) 1 DwJ 2 β < 1. Result[Sankaran et al., 2017] 1. ŵ p w. 2. P( ˆ J = J ) 1. Similar to Group Lasso [Bach, 2008, Theorem 2]. Learns the weights, without explicit groups information. 3 D: Diagonal matrix, defined using groups G1,..., G k, and w

76 Quantitative Simulation: Predictive Accuracy Aim: Learn ŵ using LS-SOWL, evaluate prediction error. 4 The experiments followed the setup of Bondell and Reich [2008]

77 Quantitative Simulation: Predictive Accuracy Aim: Learn ŵ using LS-SOWL, evaluate prediction error. Generate samples: x N (0, Σ), ɛ N (0, σ 2 ), y = x w + ɛ. 4 The experiments followed the setup of Bondell and Reich [2008]

78 Quantitative Simulation: Predictive Accuracy Aim: Learn ŵ using LS-SOWL, evaluate prediction error. Generate samples: x N (0, Σ), ɛ N (0, σ 2 ), y = x w + ɛ. Metric: E[ x (w ŵ) 2 ] = (w ŵ) Σ(w ŵ). 4 The experiments followed the setup of Bondell and Reich [2008]

79 Quantitative Simulation: Predictive Accuracy Aim: Learn ŵ using LS-SOWL, evaluate prediction error. Generate samples: x N (0, Σ), ɛ N (0, σ 2 ), y = x w + ɛ. Metric: E[ x (w ŵ) 2 ] = (w ŵ) Σ(w ŵ). Data: 4 w = [0 10, 2 10, 0 10, 2 10 ]. n = 100, σ = 15 and Σ i,j = 0.5 if i j and 1 if i = j. 4 The experiments followed the setup of Bondell and Reich [2008]

80 Quantitative Simulation: Predictive Accuracy Aim: Learn ŵ using LS-SOWL, evaluate prediction error. Generate samples: x N (0, Σ), ɛ N (0, σ 2 ), y = x w + ɛ. Metric: E[ x (w ŵ) 2 ] = (w ŵ) Σ(w ŵ). Data: 4 w = [0 10, 2 10, 0 10, 2 10 ]. n = 100, σ = 15 and Σ i,j = 0.5 if i j and 1 if i = j. Models with group variance: Measure E[ x ( w ŵ) 2 ]. w = w + ɛ, ɛ U[ τ, τ], τ = 0, 0.2, The experiments followed the setup of Bondell and Reich [2008]

81 Predictive accuracy results Algorithm Med. MSE MSE (10th Perc). MSE (90th Perc) LASSO 46.1 / 45.2 / / 32.7 / / 61.5 / 61.4 OWL 27.6 / 27.0 / / 19.2 / / 40.4 / 39.2 El. Net 30.8 / 30.7 / / 22.6 / / 43.0 / 41.4 Ω S 23.9 / 23.3 / / 16.8 / / 35.4 / 33.2 Table: Each column has numbers for τ = 0, 0.2, 0.4.

82 Summary 1. Proposed a new family of norms Ω S. 2. Properties: Equivalent to OWL in group identification. Efficient computational tools Equivalences to Group Lasso. 3. Illustrations on performance through simulations.

83 Questions?

84 Thank you!!!

85 References I F. Bach. Consistency of the group lasso and multiple kernel learning. Journal of Machine Learning Research, 9: , F. Bach. Structured sparsity-inducing norms through submodular functions. In Neural Information Processing Systems, F. Bach. Learning with submodular functions: A convex optimization perspective. Foundations and Trends in Machine Learning, 6-2-3: , A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1): , March H. D. Bondell and B. J. Reich. Simultaneous regression shrinkage, variable selection, and supervised clustering of predictors with oscar. Biometrics, 64: , 2008.

86 References II P. Buhlmann, P. Rütimann, Sara van de Geer, and Cun-Hui Zhang. Correlated variables in regression : Clustering and sparse estimation. Journal of Statistical Planning and Inference, 43: , A. Chambolle and T. Pock. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of mathematical imaging and vision, 40(1): , May M. A. T. Figueiredo and R. D. Nowak. Ordered weighted l 1 regularized regression with strongly correlated covariates: Theoretical aspects. In International Conference on Artificial Intelligence and Statistics (AISTATS), Mohamed Hebiri and Johannes Lederer. How correlations influence lasso prediction. IEEE Transactions on Information Theory, 59 (3): , G. Obozinski and F. Bach. Convex relaxation for combinatorial penalties. Technical Report , HAL, 2012.

87 References III R. Sankaran, F. Bach, and C. Bhattacharya. Identifying groups of strongly correlated variables through smoothed ordered weighted l 1-norms. In Artificial Intelligence and Statistics, pages , R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58: , M.J. Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using l 1 -constrained quadratic programming (lasso). IEEE Transactions On Information Theory, 55(5): , X. Zeng and M. Figueiredo. The ordered weighted l 1 norm: Atomic formulation, projections, and algorithms. ArXiv preprint: v5, 2015.