Structured sparsity-inducing norms through submodular functions

Transcription

1 Structured sparsity-inducing norms through submodular functions Francis Bach INRIA - Ecole Normale Supérieure, Paris, France Joint work with R. Jenatton, J. Mairal, G. Obozinski IMA - September 2011

2 Outline Introduction: Sparse methods for machine learning Need for structured sparsity: Going beyond the l 1 -norm Classical approaches to structured sparsity Linear combinations of l q -norms Submodular functions Structured sparsity through submodular functions Relaxation of the penalization of supports Unified algorithms and analysis Extensions Shaping level sets through symmetric submodular functions l q -norm relaxation of combinatorial penalties

3 Sparsity in supervised machine learning Observed data (x i,y i ) R p R, i = 1,...,n Response vector y = (y 1,...,y n ) R n Design matrix X = (x 1,...,x n ) R n p Regularized empirical risk minimization: 1 min w R p n n i=1 l(y i,w x i )+λω(w) = min w R p L(y,Xw)+λΩ(w) Norm Ω to promote sparsity square loss + l 1 -norm basis pursuit in signal processing (Chen et al., 2001), Lasso in statistics/machine learning(tibshirani, 1996) Proxy for interpretability Allow high-dimensional inference: log p = O(n)

4 Sparsity in unsupervised machine learning Multiple responses/signals y = (y 1,...,y k ) R n k min X=(x 1,...,x p ) min w 1,...,w k R p k j=1 { } L(y j,xw j )+λω(w j )

5 Sparsity in unsupervised machine learning Multiple responses/signals y = (y 1,...,y k ) R n k min X=(x 1,...,x p ) min w 1,...,w k R p k j=1 { } L(y j,xw j )+λω(w j ) Only responses are observed Dictionary learning Learn X = (x 1,...,x p ) R n p such that j, x j 2 1 min X=(x 1,...,x p ) min w 1,...,w k R p k j=1 { } L(y j,xw j )+λω(w j ) Olshausen and Field (1997); Elad and Aharon (2006); Mairal et al. (2009a) sparse PCA: replace x j 2 1 by Θ(x j ) 1

6 Sparsity in signal processing Multiple responses/signals x = (x 1,...,x k ) R n k min D=(d 1,...,d p ) min α 1,...,α k R p k j=1 { } L(x j,dα j )+λω(α j ) Only responses are observed Dictionary learning Learn D = (d 1,...,d p ) R n p such that j, d j 2 1 min D=(d 1,...,d p ) min α 1,...,α k R p k j=1 { } L(x j,dα j )+λω(α j ) Olshausen and Field (1997); Elad and Aharon (2006); Mairal et al. (2009a) sparse PCA: replace d j 2 1 by Θ(d j ) 1

7 Interpretability Why structured sparsity? Structured dictionary elements (Jenatton et al., 2009b) Dictionary elements organized in a tree or a grid (Kavukcuoglu et al., 2009; Jenatton et al., 2010; Mairal et al., 2010)

8 Structured sparse PCA (Jenatton et al., 2009b) raw data sparse PCA Unstructed sparse PCA many zeros do not lead to better interpretability

9 Structured sparse PCA (Jenatton et al., 2009b) raw data sparse PCA Unstructed sparse PCA many zeros do not lead to better interpretability

10 Structured sparse PCA (Jenatton et al., 2009b) raw data Structured sparse PCA Enforce selection of convex nonzero patterns robustness to occlusion in face identification

11 Structured sparse PCA (Jenatton et al., 2009b) raw data Structured sparse PCA Enforce selection of convex nonzero patterns robustness to occlusion in face identification

12 Interpretability Why structured sparsity? Structured dictionary elements (Jenatton et al., 2009b) Dictionary elements organized in a tree or a grid (Kavukcuoglu et al., 2009; Jenatton et al., 2010; Mairal et al., 2010)

13 Modelling of text corpora (Jenatton et al., 2010)

14 Interpretability Why structured sparsity? Structured dictionary elements (Jenatton et al., 2009b) Dictionary elements organized in a tree or a grid (Kavukcuoglu et al., 2009; Jenatton et al., 2010; Mairal et al., 2010)

15 Interpretability Why structured sparsity? Structured dictionary elements (Jenatton et al., 2009b) Dictionary elements organized in a tree or a grid (Kavukcuoglu et al., 2009; Jenatton et al., 2010; Mairal et al., 2010) Stability and identifiability Optimization problem min w R p L(y,Xw)+λ w 1 is unstable Codes w j often used in later processing (Mairal et al., 2009c) Prediction or estimation performance When prior knowledge matches data (Haupt and Nowak, 2006; Baraniuk et al., 2008; Jenatton et al., 2009a; Huang et al., 2009) Numerical efficiency Non-linear variable selection with 2 p subsets (Bach, 2008)

16 Classical approaches to structured sparsity Many application domains Computer vision (Cevher et al., 2008; Mairal et al., 2009b) Neuro-imaging (Gramfort and Kowalski, 2009; Jenatton et al., 2011) Bio-informatics (Rapaport et al., 2008; Kim and Xing, 2010) Non-convex approaches Haupt and Nowak (2006); Baraniuk et al. (2008); Huang et al. (2009) Convex approaches Design of sparsity-inducing norms

17 Outline Introduction: Sparse methods for machine learning Need for structured sparsity: Going beyond the l 1 -norm Classical approaches to structured sparsity Linear combinations of l q -norms Submodular functions Structured sparsity through submodular functions Relaxation of the penalization of supports Unified algorithms and analysis Extensions Shaping level sets through symmetric submodular functions l q -norm relaxation of combinatorial penalties

18 Sparsity-inducing norms Popular choice for Ω The l 1 -l 2 norm, G H w G 2 = G H ( j G wj 2 ) 1/2 with H a partition of {1,...,p} The l 1 -l 2 norm sets to zero groups of non-overlapping variables (as opposed to single variables for the l 1 -norm) For the square loss, group Lasso (Yuan and Lin, 2006) G1 G 2 G 3

19 Unit norm balls Geometric interpretation w 2 w 1 w 2 1 +w w 3

20 Sparsity-inducing norms Popular choice for Ω The l 1 -l 2 norm, G H w G 2 = G H ( j G wj 2 ) 1/2 with H a partition of {1,...,p} The l 1 -l 2 norm sets to zero groups of non-overlapping variables (as opposed to single variables for the l 1 -norm) For the square loss, group Lasso (Yuan and Lin, 2006) G1 G 2 G 3 However, the l 1 -l 2 norm encodes fixed/static prior information, requires to know in advance how to group the variables What happens if the set of groups H is not a partition anymore?

21 Structured sparsity with overlapping groups (Jenatton, Audibert, and Bach, 2009a) When penalizing by the l 1 -l 2 norm, ( wj 2 ) 1/2 G 2 = G H w G H j G The l 1 norm induces sparsity at the group level: Some w G s are set to zero Inside the groups, the l 2 norm does not promote sparsity G 1 G 2 G 3

22 Structured sparsity with overlapping groups (Jenatton, Audibert, and Bach, 2009a) When penalizing by the l 1 -l 2 norm, ( wj 2 ) 1/2 G 2 = G H w G H j G The l 1 norm induces sparsity at the group level: Some w G s are set to zero Inside the groups, the l 2 norm does not promote sparsity G 1 G 2 G 3 The zero pattern of w is given by {j, w j = 0} = G for some H H G H Zero patterns are unions of groups

23 Examples of set of groups H Selection of contiguous patterns on a sequence, p = 6 H is the set of blue groups Any union of blue groups set to zero leads to the selection of a contiguous pattern

24 Examples of set of groups H Selection of rectangles on a 2-D grids, p = 25 H is the set of blue/green groups (with their not displayed complements) Any union of blue/green groups set to zero leads to the selection of a rectangle

25 Examples of set of groups H Selection of diamond-shaped patterns on a 2-D grids, p = 25. It is possible to extend such settings to 3-D space, or more complex topologies

26 Relationship between H and Zero Patterns (Jenatton, Audibert, and Bach, 2009a) H Zero patterns: by generating the union-closure of H Zero patterns H: Design groups H from any union-closed set of zero patterns Design groups H from any intersection-closed set of non-zero patterns

27 Unit norm balls Geometric interpretation w 1 w 2 1 +w w 3 w 2 + w 1 + w 2

28 Optimization for sparsity-inducing norms (see Bach, Jenatton, Mairal, and Obozinski, 2011) Gradient descent as a proximal method (differentiable functions) w t+1 = arg min w R pj(w t)+(w w t ) J(w t )+ L 2 w w t 2 2 w t+1 = w t 1 L J(w t)

29 Optimization for sparsity-inducing norms (see Bach, Jenatton, Mairal, and Obozinski, 2011) Gradient descent as a proximal method (differentiable functions) w t+1 = arg min w R pj(w t)+(w w t ) J(w t )+ L 2 w w t 2 2 w t+1 = w t 1 L J(w t) Problems of the form: min w R pl(w)+λω(w) w t+1 = arg min w R pl(w t)+(w w t ) L(w t )+λω(w)+ L 2 w w t 2 2 Ω(w) = w 1 Thresholded gradient descent Similar convergence rates than smooth optimization Acceleration methods (Nesterov, 2007; Beck and Teboulle, 2009)

30 Sparse Structured PCA (Jenatton, Obozinski, and Bach, 2009b) Learning sparse and structured dictionary elements: 1 min W R k n,x R p k n n i=1 y i Xw i 2 2+λ p Ω(x j ) s.t. i, w i 2 1 j=1

31 Application to face databases (1/3) raw data (unstructured) NMF NMF obtains partially local features

32 Application to face databases (2/3) (unstructured) sparse PCA Structured sparse PCA Enforce selection of convex nonzero patterns robustness to occlusion

33 Application to face databases (2/3) (unstructured) sparse PCA Structured sparse PCA Enforce selection of convex nonzero patterns robustness to occlusion

34 Application to face databases (3/3) Quantitative performance evaluation on classification task % Correct classification raw data PCA NMF SPCA shared SPCA SSPCA shared SSPCA Dictionary size

35 Dictionary learning vs. sparse structured PCA Exchange roles of X and w Sparse structured PCA (structured dictionary elements): 1 min W R k n,x R p k n n i=1 y i Xw i 2 2 +λ k j=1 Ω(x j ) s.t. i, w i 2 1. Dictionary learning with structured sparsity for codes w: 1 min W R k n,x R p k n n i=1 y i Xw i 2 2+λΩ(w i ) s.t. j, x j 2 1. Optimization: proximal methods Requires solving many times min w R p 1 2 y w 2 2+λΩ(w) Modularity of implementation if proximal step is efficient (Jenatton et al., 2010; Mairal et al., 2010)

36 Hierarchical dictionary learning (Jenatton, Mairal, Obozinski, and Bach, 2010) Structure on codes w (not on dictionary X) Hierarchical penalization: Ω(w) = G H w G 2 where groups G in H are equal to set of descendants of some nodes in a tree Variable selected after its ancestors (Zhao et al., 2009; Bach, 2008)

37 Hierarchical dictionary learning Modelling of text corpora Each document is modelled through word counts Low-rank matrix factorization of word-document matrix Probabilistic topic models (Blei et al., 2003) Similar structures based on non parametric Bayesian methods (Blei et al., 2004) Can we achieve similar performance with simple matrix factorization formulation?

38 Modelling of text corpora - Dictionary tree

39 Application to background subtraction (Mairal, Jenatton, Obozinski, and Bach, 2010) Input l 1 -norm Structured norm

40 Application to background subtraction (Mairal, Jenatton, Obozinski, and Bach, 2010) Background l 1 -norm Structured norm

41 Topographic dictionaries (Mairal, Jenatton, Obozinski, and Bach, 2010)

42 Application to neuro-imaging Structured sparsity for fmri (Jenatton et al., 2011) Brain reading : prediction of (seen) object size Multi-scale activity levels through hierarchical penalization

43 Application to neuro-imaging Structured sparsity for fmri (Jenatton et al., 2011) Brain reading : prediction of (seen) object size Multi-scale activity levels through hierarchical penalization

44 Application to neuro-imaging Structured sparsity for fmri (Jenatton et al., 2011) Brain reading : prediction of (seen) object size Multi-scale activity levels through hierarchical penalization

45 Structured sparse PCA on resting state activity (Varoquaux, Jenatton, Gramfort, Obozinski, Thirion, and Bach, 2010)

46 An alternative approach: Latent group Lasso (Jacob, Obozinski, and Vert, 2009) Ω latent (w) = min d A v A q (v A R p ) A G A G s.t. v A i = 0 for i / A, w = A Gv A. Supports are typically unions of subcollections of the sets defining the norm See analysis by Rao et al. (2011) in the context of signal processing

47 Latent group Lasso (Jacob et al., 2009) Application to Bioinformatics Metastasis prediction from microarray data Biological pathways Dedicated sparsity-inducing norm for better interpretability and prediction

48 Outline Introduction: Sparse methods for machine learning Need for structured sparsity: Going beyond the l 1 -norm Classical approaches to structured sparsity Linear combinations of l q -norms Submodular functions Structured sparsity through submodular functions Relaxation of the penalization of supports Unified algorithms and analysis Extensions Shaping level sets through symmetric submodular functions l q -norm relaxation of combinatorial penalties

49 l 1 -norm = convex envelope of cardinality of support Let w R p. Let V = {1,...,p} and Supp(w) = {j V, w j 0} Cardinality of support: w 0 = Card(Supp(w)) Convex envelope = largest convex lower bound (see, e.g., Boyd and Vandenberghe, 2004) w 0 w l 1 -norm = convex envelope of l 0 -quasi-norm on the l -ball [ 1,1] p

50 Convex envelopes of general functions of the support (Bach, 2010) Let F : 2 V R be a set-function Assume F is non-decreasing (i.e., A B F(A) F(B)) Explicit prior knowledge on supports (Haupt and Nowak, 2006; Baraniuk et al., 2008; Huang et al., 2009) Define Θ(w) = F(Supp(w)): How to get its convex envelope? 1. Possible if F is also submodular 2. Allows unified theory and algorithm 3. Provides new regularizers

51 Submodular functions (Fujishige, 2005; Bach, 2010b) F : 2 V R is submodular if and only if A,B V, F(A)+F(B) F(A B)+F(A B) k V, A F(A {k}) F(A) is non-increasing

52 Submodular functions (Fujishige, 2005; Bach, 2010b) F : 2 V R is submodular if and only if A,B V, F(A)+F(B) F(A B)+F(A B) k V, A F(A {k}) F(A) is non-increasing Intuition 1: defined like concave functions ( diminishing returns ) Example: F : A g(card(a)) is submodular if g is concave

53 Submodular functions (Fujishige, 2005; Bach, 2010b) F : 2 V R is submodular if and only if A,B V, F(A)+F(B) F(A B)+F(A B) k V, A F(A {k}) F(A) is non-increasing Intuition 1: defined like concave functions ( diminishing returns ) Example: F : A g(card(a)) is submodular if g is concave Intuition 2: behave like convex functions Polynomial-time minimization, conjugacy theory

54 Submodular functions (Fujishige, 2005; Bach, 2010b) F : 2 V R is submodular if and only if A,B V, F(A)+F(B) F(A B)+F(A B) k V, A F(A {k}) F(A) is non-increasing Intuition 1: defined like concave functions ( diminishing returns ) Example: F : A g(card(a)) is submodular if g is concave Intuition 2: behave like convex functions Polynomial-time minimization, conjugacy theory Used in several areas of signal processing and machine learning Total variation/graph cuts (Chambolle, 2005; Boykov et al., 2001) Optimal design (Krause and Guestrin, 2005)

55 Submodular functions - Examples Concave functions of the cardinality: g( A ) Cuts Entropies H((X k ) k A ) from p random variables X 1,...,X p Gaussian variables H((X k ) k A ) logdetσ AA Functions of eigenvalues of sub-matrices Network flows Efficient representation for set covers Rank functions of matroids

56 Submodular functions - Lovász extension Subsets may be identified with elements of {0,1} p Given any set-function F and w such that w j1 w jp, define: f(w) = p k=1 w jk [F({j 1,...,j k }) F({j 1,...,j k 1 })] If w = 1 A, f(w) = F(A) extension from {0,1} p to R p f is piecewise affine and positively homogeneous F is submodular if and only if f is convex (Lovász, 1982) Minimizing f(w) on w [0,1] p equivalent to minimizing F on 2 V

57 Submodular functions - Submodular polyhedron Submodular polyhedron: P(F) = {s R p, A V, s(a) F(A)} s 2 s 3 B(F) P(F) B(F) s 1 P(F) s 2 s 1

58 Submodular functions - Submodular polyhedron Submodular polyhedron: P(F) = {s R p, A V, s(a) F(A)} Link with Lovász extension (Lovász, 1982): if w R p +, then max s P(F) w s = f(w) Maximizer obtained by greedy algorithm: Sort the components of w, as w j1 w jp Set s jk = F({j 1,...,j k }) F({j 1,...,j k 1 }) Other operations on submodular polyhedron (Bach, 2010b)

59 Submodular functions - Optimization Submodular function minimization in O(p 6 ) Schrijver (2000); Iwata et al. (2001); Orlin (2009) Efficient active set algorithm with no complexity bound Based on the efficient computability of the support function Fujishige and Isotani (2011); Wolfe (1976) Submodular function maximization is NP-hard 1/4-approximation with random subset, 1/2 with local search (Feige et al., 2007) (1 1/e)-approximation for maximizing a monotonic submodular function with a cardinality constraint (Nemhauser et al., 1978) Special cases with faster algorithms: cuts, flows

60 Submodular functions and structured sparsity Let F : 2 V R be a non-decreasing submodular set-function Proposition: the convex envelope of Θ : w F(Supp(w)) on the l -ball is Ω : w f( w ) where f is the Lovász extension of F

61 Submodular functions and structured sparsity Let F : 2 V R be a non-decreasing submodular set-function Proposition: the convex envelope of Θ : w F(Supp(w)) on the l -ball is Ω : w f( w ) where f is the Lovász extension of F Sparsity-inducing properties: Ω is a polyhedral norm (0,1)/F({2}) (1,1)/F({1,2}) (1,0)/F({1}) A if stable if for all B A, B A F(B) > F(A) With probability one, stable sets are the only allowed active sets

62 Polyhedral unit balls w 3 w 2 w 1 F(A) = A Ω(w) = w 1 F(A) = min{ A,1} Ω(w) = w F(A) = A 1/2 all possible extreme points F(A) = 1 {A {1} } +1 {A {2,3} } Ω(w) = w 1 + w {2,3} F(A) = 1 {A {1,2,3} } +1 {A {2,3} } +1 {A {3} } Ω(w) = w + w {2,3} + w 3

63 Submodular functions and structured sparsity Examples From Ω(w) to F(A): provides new insights into existing norms Grouped norms with overlapping groups (Jenatton et al., 2009a) Ω(w) = G H w G l 1 -l norm sparsity at the group level Some w G s are set to zero for some groups G ( Supp(w) ) c = G H G for some H H

64 Submodular functions and structured sparsity Examples From Ω(w) to F(A): provides new insights into existing norms Grouped norms with overlapping groups (Jenatton et al., 2009a) Ω(w) = G H w G F(A) = Card ( {G H, G A } ) l 1 -l norm sparsity at the group level Some w G s are set to zero for some groups G ( Supp(w) ) c = G H G for some H H Justification not only limited to allowed sparsity patterns

65 Selection of contiguous patterns in a sequence Selection of contiguous patterns in a sequence H is the set of blue groups: any union of blue groups set to zero leads to the selection of a contiguous pattern

66 Selection of contiguous patterns in a sequence Selection of contiguous patterns in a sequence H is the set of blue groups: any union of blue groups set to zero leads to the selection of a contiguous pattern G H w G F(A) = p 2+Range(A) if A Jump from 0 to p 1: tends to include all variables simultaneously Add ν A to smooth the kink: all sparsity patterns are possible Contiguous patterns are favored (and not forced)

67 Extensions of norms with overlapping groups Selection of rectangles (at any position) in a 2-D grids Hierarchies

68 Submodular functions and structured sparsity Examples From Ω(w) to F(A): provides new insights into existing norms Grouped norms with overlapping groups (Jenatton et al., 2009a) Ω(w) = w G F(A) = Card ( {G H, G A } ) G H Justification not only limited to allowed sparsity patterns

69 Submodular functions and structured sparsity Examples From Ω(w) to F(A): provides new insights into existing norms Grouped norms with overlapping groups (Jenatton et al., 2009a) Ω(w) = w G F(A) = Card ( {G H, G A } ) G H Justification not only limited to allowed sparsity patterns From F(A) to Ω(w): provides new sparsity-inducing norms F(A) = g(card(a)) Ω is a combination of order statistics Non-factorial priors for supervised learning: Ω depends on the eigenvalues of X A X A and not simply on the cardinality of A

70 Non-factorial priors for supervised learning Joint variable selection and regularization. Given support A V, 1 min w A R A 2n y X Aw A 2 2+ λ 2 w A 2 2 Minimizing with respect to A will always lead to A = V Information/model selection criterion F(A) min min 1 A V w A R A 2n y X Aw A λ 2 w A 2 2 +F(A) min w R p 1 2n y Xw 2 2+ λ 2 w 2 2+F(Supp(w))

71 Non-factorial priors for supervised learning Selection of subset A from design X R n p with l 2 -penalization Frequentist analysis (Mallow s C L ): trx A X A(X A X A+λI) 1 Not submodular Bayesian analysis (marginal likelihood): logdet(x A X A+λI) Submodular (also true for tr(x A X A) 1/2 ) p n k submod. l 2 vs. submod. l 1 vs. submod. greedy vs. submod ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± 3.5

72 Unified optimization algorithms Polyhedral norm with O(3 p ) faces and extreme points Not suitable to linear programming toolboxes Subgradient (w Ω(w) non-differentiable) subgradient may be obtained in polynomial time too slow

73 Unified optimization algorithms Polyhedral norm with O(3 p ) faces and extreme points Not suitable to linear programming toolboxes Subgradient (w Ω(w) non-differentiable) subgradient may be obtained in polynomial time too slow Proximal methods (e.g., Beck and Teboulle, 2009) min w R pl(y,xw)+λω(w): differentiable + non-differentiable Efficient when (P) : min w R p 1 2 w v 2 2+λΩ(w) is easy Proposition: (P) is equivalent to min A V λf(a) j A v j with minimum-norm-point algorithm Possible complexity bound O(p 6 ), but empirically O(p 2 ) (or more) Faster algorithm for special case (Mairal et al., 2010)

74 Proximal methods for Lovász extensions Proposition (Chambolle and Darbon, 2009): let w be the solution of min w R p 1 2 w v 2 2+λf(w). Then the solutions of min λf(a)+ (α v j ) A V j A are the sets A α such that {w > α} A α {w α} Parametric submodular function optimization General decomposition strategy for f( w ) and f(w) (Groenevelt, 1991) Efficient only when submodular minimization is efficient Otherwise, minimum-norm-point algorithm (a.k.a. Frank Wolfe) is preferable

75 Comparison of optimization algorithms Synthetic example with p = 1000 and F(A) = A 1/2 ISTA: proximal method FISTA: accelerated variant (Beck and Teboulle, 2009) f(w) min(f) fista ista subgradient time (seconds)

76 Comparison of optimization algorithms (Mairal, Jenatton, Obozinski, and Bach, 2010) Small scale Specific norms which can be implemented through network flows log(primal Optimum) n=100, p=1000, one dimensional DCT ProxFlox SG ADMM Lin ADMM QP CP log(seconds)

77 Comparison of optimization algorithms (Mairal, Jenatton, Obozinski, and Bach, 2010) Large scale Specific norms which can be implemented through network flows 2 n=1024, p=10000, one dimensional DCT n=1024, p=100000, one dimensional DCT 2 log(primal Optimum) ProxFlox SG 8 ADMM Lin ADMM CP log(seconds) 4 log(primal Optimum) ProxFlox 8 SG ADMM Lin ADMM log(seconds) 4

78 Unified theoretical analysis Decomposability Key to theoretical analysis (Negahban et al., 2009) Property: w R p, and J V, if min j J w j max j J c w j, then Ω(w) = Ω J (w J )+Ω J (w J c) Support recovery Extension of known sufficient condition (Zhao and Yu, 2006; Negahban and Wainwright, 2008) High-dimensional inference Extension of known sufficient condition (Bickel et al., 2009) Matches with analysis of Negahban et al. (2009) for common cases

79 Support recovery - min w R p Notation 1 2n y Xw 2 2+λΩ(w) ρ(j) = min B J c F(B J) F(J) F(B) (0,1] (for J stable) c(j) = sup w R pω J (w J )/ w J 2 J 1/2 max k V F({k}) Proposition Assume y = Xw +σε, with ε N(0,I) J = smallest stable set containing the support of w Assume ν = min j,w j 0 w j > 0 Let Q = 1 n X X R p p. Assume κ = λ min (Q JJ ) > 0 Assume that for η > 0, (Ω J ) [(Ω J (Q 1 JJ Q Jj)) j J c] 1 η If λ κν 2c(J), ŵ has support equal to J, with probability larger than 1 3P ( Ω (z) > ληρ(j) n) 2σ z is a multivariate normal with covariance matrix Q

80 Consistency - min w R p Proposition 1 2n y Xw 2 2+λΩ(w) Assume y = Xw +σε, with ε N(0,I) J = smallest stable set containing the support of w Let Q = 1 n X X R p p. Assume that s.t. Ω J ( J c) 3Ω J ( J ), Q κ J 2 2 Then Ω(ŵ w ) 24c(J)2 λ κρ(j) 2 and 1 n Xŵ Xw c(J)2 λ 2 κρ(j) 2 with probability larger than 1 P ( Ω (z) > λρ(j) n 2σ z is a multivariate normal with covariance matrix Q Concentration inequality (z normal with covariance matrix Q): T set of stable inseparable sets Then P(Ω (z) > t) A T 2 A exp ( t2 F(A) 2 /2) 1 Q AA 1 )

81 Outline Introduction: Sparse methods for machine learning Need for structured sparsity: Going beyond the l 1 -norm Classical approaches to structured sparsity Linear combinations of l q -norms Submodular functions Structured sparsity through submodular functions Relaxation of the penalization of supports Unified algorithms and analysis Extensions Shaping level sets through symmetric submodular functions l q -norm relaxation of combinatorial penalties

82 Symmetric submodular functions (Bach, 2010a) Let F : 2 V R be a symmetric submodular set-function Proposition: The Lovász extension f(w) is the convex envelope of the function w max α R F({w α}) on the set [0,1] p +R1 V = {w R p, max k V w k min k V w k 1}.

83 Symmetric submodular functions (Bach, 2010a) Let F : 2 V R be a symmetric submodular set-function Proposition: The Lovász extension f(w) is the convex envelope of the function w max α R F({w α}) on the set [0,1] p +R1 V = {w R p, max k V w k min k V w k 1}. w > w >w 1 3 w =w 2 3 w > w >w 2 3 (1,0,1)/F({1,3}) (1,0,0)/F({1}) 1 2 w 1> w 2>w3 w =w 1 2 (0,0,1)/F({3}) w > w >w w > w >w 2 (1,1,0)/F({1,2}) 2 1 (0,1,1)/F({2,3}) 1 w 2> w 3>w1 (0,1,0)/F({2}) 3 3 w =w 1 3 (1,0,0) (1,0,1)/2 (0,0,1) (0,1,1) (0,1,0)/2 (1,1,0)

84 Symmetric submodular functions - Examples From Ω(w) to F(A): provides new insights into existing norms Cuts - total variation F(A) = d(k,j) f(w) = d(k,j)(w k w j ) + k A,j V \A k,j V NB: graph may be directed

85 Symmetric submodular functions - Examples From F(A) to Ω(w): provides new sparsity-inducing norms F(A) = g(card(a)) priors on the size and numbers of clusters weights 0 5 weights 0 5 weights λ 0.03 A (p A ) λ 3 1 A (0,p) λ 0.4 max{ A,p A } Convex formulations for clustering (Hocking, Joulin, Bach, and Vert, 2011)

86 Symmetric submodular functions - Examples From F(A) to Ω(w): provides new sparsity-inducing norms Regular functions (Boykov et al., 2001; Chambolle and Darbon, 2009) F(A)= min d(k, j)+λ A B W B W k B, j W\B V weights 0 weights 0 weights

87 l q -relaxation of combinatorial penalties (Obozinski and Bach, 2011) Main result of Bach (2010): f( w ) is the convex envelope of F(Supp(w)) on [ 1,1] p Problems: Limited to submodular functions Limited to l -relaxation: undesired artefacts (0,1)/F({2}) (1,1)/F({1,2}) (1,0)/F({1})

88 From l to l 2 Variational formulations for subquadratic norms (Bach et al., 2011) Ω(w) = min η R p p j=1 wj g(η) = min η j 2 η H p j=1 where g is a convex homogeneous and H = {η,g(η) 1} Often used for computational reasons (Lasso, group Lasso) May also be used to define a norm (Micchelli et al., 2011) w 2 j η j

89 From l to l 2 Variational formulations for subquadratic norms (Bach et al., 2011) Ω(w) = min η R p p j=1 wj g(η) = min η j 2 η H p j=1 where g is a convex homogeneous and H = {η,g(η) 1} Often used for computational reasons (Lasso, group Lasso) May also be used to define a norm (Micchelli et al., 2011) If F is a nondecreasing submodular function with Lovász extension f 1 p wj 2 Define Ω 2 (w) = min + 1 η R p 2 η + j=1 j 2 f(η) Is it the convex relaxation of some natural function? w 2 j η j

90 l q -relaxation of submodular penalties (Obozinski and Bach, 2011) F a nondecreasing submodular function with Lovász extension f Define Ω q (w) = min η R p + 1 q i V w i q η q 1 i + 1 r f(η) with 1 q + 1 r = 1. Proposition 1: Ω q is the convex envelope of w F(Supp(w)) w q Proposition 2: Ω q is the homogeneous convex envelope of w 1 r F(Supp(w))+ 1 q w q q Jointly penalizing and regularizing Special cases q = 1, q = 2 and q =

91 Some simple examples F Ω q A w 1 If H is a partition of V: 1 {A } w q B H 1 {A B } B H w B q Recover results of Bach (2010) when q = and F submodular However when H is not a partition and q <, Ω q is not in general an l 1 /l q -norm! F does not need to be submodular New norms

92 l q -relaxation of combinatorial penalties (Obozinski and Bach, 2011) F any strictly positive set-function (with potentially infinite values) Jointly penalizing and regularizing. Two formulations: homogeneous convex envelope of w F(Supp(w))+ w q q convex envelope of w F(Supp(w)) w q Proposition: These envelopes are equal to a constant times a norm Ω F q = Ω q defined through its dual norm itsdualnormisequalto (Ω q ) (s) = max A V Three-line proof s A r F(A) 1/r, with 1 q +1 r = 1

93 l q -relaxation of combinatorial penalties Proof Denote Θ(w) = w q F(Supp(w)) 1/r, and compute its Fenchel conjugate: Θ (s) = max w R pw s w q F(Supp(w)) 1/r = max A V max As A w A q F(A) 1/r w A (R ) Aw = max A V ι { s A r F(A) 1/r } = ι {Ω q (s) 1}, where ι {s S} is the indicator of the set S Consequence: If F is submodular and q = +, Ω(w) = f( w )

94 How tight is the relaxation? What information of F is kept after the relaxation? When F is submodular and q = the Lovász extension f = Ω is said to extend F because Ω F (1 A ) = f(1 A ) = F(A) In general we can still consider the function : G(A) = Ω F (1 A ) Do we have G(A) = F(A)? How is G related to F? What is the norm Ω G which is associated with G?

95 Lower combinatorial envelope Given a function F : 2 V R, define its lower combinatorial envelope as the function G given by G(A) = max s P(F) s(a) with P(F) = {s R p, A V, s(a) F(A)}. Lemma 1 (Idempotence) P(F) = P(G) G is its own lower combinatorial envelope For all q 1, Ω F q = Ω G q Lemma 2 (Extension property) Ω F (1 A ) = max 1 As = max (Ω F ) (s) 1 s P(F) s 1 A = G(A)

96 l q -relaxation of combinatorial penalties A large latent group Lasso Goal: Primal formulation for (Ω q ) (s) = max A V s A r F(A) 1/r Define V = {v = (v A ) A V ( R V) 2 V s.t. Supp(v A ) A} Proposition (Jacob et al., 2009; Obozinski and Bach, 2011): Ω q (w) = min F(A) 1 r v A q s.t. w = v A, v V A V A V

97 Block coding (Huang et al., 2009) F : 2 V R + {+ } a positive set function Define the weighted set cover problem G(A) = min F(B i ) s.t. A i I i IB i I = min (δ A ) A V {0,1} 2V + A V F(A)δ A, s.t. A V δ A 1 A 1 B

98 A principled relaxation for block-coding Define the weighted set cover problem G(A) = min (δ A ) A V {0,1} 2V + A V F(A)δ A, s.t. A V δ A 1 A 1 B G the lower combinatorial envelope of F Proposition: G is the value of the min fractional weighted set cover G(A) = min (δ A ) A V [0,1] 2V + A V F(A)δ A, s.t. A V δ A 1 A 1 B and we have G(A) G(A) F(A)

99 Examples Examples from submodular functions New norms for the overlapping group lasso (Obozinski and Bach, 2011) with fewer artefacts Some functions are not attainable Many set-functions lead to G(A) = A and the l 1 -norm Example: F(A) = A 1 A A Convexification is not always useful

100 Conclusion Structured sparsity for machine learning and statistics Many applications (image, audio, text, etc.) May be achieved through structured sparsity-inducing norms Link with submodular functions: unified analysis and algorithms

101 Conclusion Structured sparsity for machine learning and statistics Many applications (image, audio, text, etc.) May be achieved through structured sparsity-inducing norms Link with submodular functions: unified analysis and algorithms On-going work on structured sparsity Norm design beyond submodular functions Instance of general framework of Chandrasekaran et al. (2010) Links with greedy methods (Haupt and Nowak, 2006; Baraniuk et al., 2008; Huang et al., 2009) Links between norm Ω, support Supp(w), and design X (see, e.g., Grave, Obozinski, and Bach, 2011) Achieving log p = O(n) algorithmically (Bach, 2008)

102 References F. Bach. Exploring large feature spaces with hierarchical multiple kernel learning. In Advances in Neural Information Processing Systems, F. Bach. Structured sparsity-inducing norms through submodular functions. In NIPS, F. Bach. Shaping level sets with submodular functions. Technical Report , HAL, 2010a. F. Bach. Convex analysis and optimization with submodular functions: a tutorial. Technical Report , HAL, 2010b. F. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penalties. Technical Report , HAL, R. G. Baraniuk, V. Cevher, M. F. Duarte, and C. Hegde. Model-based compressive sensing. Technical report, arxiv: , A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, 2(1): , P. Bickel, Y. Ritov, and A. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. Annals of Statistics, 37(4): , D. Blei, A. Ng, and M. Jordan. Latent dirichlet allocation. The Journal of Machine Learning Research, 3: , January D. Blei, T.L. Griffiths, M.I. Jordan, and J.B. Tenenbaum. Hierarchical topic models and the nested Chinese restaurant process. Advances in neural information processing systems, 16:106, S. P. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.

103 Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. IEEE Trans. PAMI, 23(11): , V. Cevher, M. F. Duarte, C. Hegde, and R. G. Baraniuk. Sparse signal recovery using markov random fields. In Advances in Neural Information Processing Systems, A. Chambolle. Total variation minimization and a class of binary MRF models. In Energy Minimization Methods in Computer Vision and Pattern Recognition, pages Springer, A. Chambolle and J. Darbon. On total variation minimization and surface evolution using parametric maximum flows. International Journal of Computer Vision, 84(3): , V. Chandrasekaran, B. Recht, P.A. Parrilo, and A.S. Willsky. The convex geometry of linear inverse problems. Arxiv preprint arxiv: , S. S. Chen, D. L. Donoho, and M. A. Saunders. Atomic decomposition by basis pursuit. SIAM Review, 43(1): , M. Elad and M. Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15(12): , U. Feige, V.S. Mirrokni, and J. Vondrak. Maximizing non-monotone submodular functions. In Proc. Symposium on Foundations of Computer Science, pages IEEE Computer Society, S. Fujishige. Submodular Functions and Optimization. Elsevier, S. Fujishige and S. Isotani. A submodular function minimization algorithm based on the minimum-norm base. Pacific Journal of Optimization, 7:3 17, A. Gramfort and M. Kowalski. Improving M/EEG source localization with an inter-condition sparse prior. In IEEE International Symposium on Biomedical Imaging, 2009.

104 E. Grave, G. Obozinski, and F. Bach. Trace lasso: a trace norm regularization for correlated designs. Arxiv preprint arxiv: , H. Groenevelt. Two algorithms for maximizing a separable concave function over a polymatroid feasible region. European Journal of Operational Research, 54(2): , J. Haupt and R. Nowak. Signal reconstruction from noisy random projections. IEEE Transactions on Information Theory, 52(9): , T. Hocking, A. Joulin, F. Bach, and J.-P. Vert. Clusterpath: an algorithm for clustering using convex fusion penalties. In Proc. ICML, J. Huang, T. Zhang, and D. Metaxas. Learning with structured sparsity. In Proceedings of the 26th International Conference on Machine Learning (ICML), S. Iwata, L. Fleischer, and S. Fujishige. A combinatorial strongly polynomial algorithm for minimizing submodular functions. Journal of the ACM, 48(4): , L. Jacob, G. Obozinski, and J.-P. Vert. Group Lasso with overlaps and graph Lasso. In Proceedings of the 26th International Conference on Machine Learning (ICML), R. Jenatton, J.Y. Audibert, and F. Bach. Structured variable selection with sparsity-inducing norms. Technical report, arxiv: , 2009a. R. Jenatton, G. Obozinski, and F. Bach. Structured sparse principal component analysis. Technical report, arxiv: , 2009b. R. Jenatton, J. Mairal, G. Obozinski, and F. Bach. Proximal methods for sparse hierarchical dictionary learning. In Submitted to ICML, R. Jenatton, A. Gramfort, V. Michel, G. Obozinski, E. Eger, F. Bach, and B. Thirion. Multi-scale

105 mining of fmri data with hierarchical structured sparsity. Technical report, Preprint arxiv: , In submission to SIAM Journal on Imaging Sciences. K. Kavukcuoglu, M. Ranzato, R. Fergus, and Y. LeCun. Learning invariant features through topographic filter maps. In Proceedings of CVPR, S. Kim and E. P. Xing. Tree-guided group Lasso for multi-task regression with structured sparsity. In Proceedings of the International Conference on Machine Learning (ICML), A. Krause and C. Guestrin. Near-optimal nonmyopic value of information in graphical models. In Proc. UAI, L. Lovász. Submodular functions and convexity. Mathematical programming: the state of the art, Bonn, pages , J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online learning for matrix factorization and sparse coding. Technical report, arxiv: , 2009a. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Non-local sparse models for image restoration. In Computer Vision, 2009 IEEE 12th International Conference on, pages IEEE, 2009b. J. Mairal, F. Bach, J. Ponce, G. Sapiro, and A. Zisserman. Supervised dictionary learning. Advances in Neural Information Processing Systems (NIPS), 21, 2009c. J. Mairal, R. Jenatton, G. Obozinski, and F. Bach. Network flow algorithms for structured sparsity. In NIPS, C.A. Micchelli, J.M. Morales, and M. Pontil. Regularizers for structured sparsity. Arxiv preprint arxiv: , 2011.

106 S. Negahban and M. J. Wainwright. Joint support recovery under high-dimensional scaling: Benefits and perils of l 1 -l -regularization. In Adv. NIPS, S. Negahban, P. Ravikumar, M. J. Wainwright, and B. Yu. A unified framework for high-dimensional analysis of M-estimators with decomposable regularizers G.L. Nemhauser, L.A. Wolsey, and M.L. Fisher. An analysis of approximations for maximizing submodular set functions i. Mathematical Programming, 14(1): , Y. Nesterov. Gradient methods for minimizing composite objective function. Center for Operations Research and Econometrics (CORE), Catholic University of Louvain, Tech. Rep, 76, G. Obozinski and F. Bach. Convex relaxation of combinatorial penalties. Technical report, HAL, B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision Research, 37: , J.B. Orlin. A faster strongly polynomial time algorithm for submodular function minimization. Mathematical Programming, 118(2): , N. Rao, B. Recht, and R. Nowak. Tight measurement bounds for exact recovery of structured sparse signals. Arxiv preprint arxiv: , F. Rapaport, E. Barillot, and J.-P. Vert. Classification of arraycgh data using fused SVM. Bioinformatics, 24(13):i375 i382, Jul A. Schrijver. A combinatorial algorithm minimizing submodular functions in strongly polynomial time. Journal of Combinatorial Theory, Series B, 80(2): , R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of The Royal Statistical Society Series B, 58(1): , 1996.

107 G. Varoquaux, R. Jenatton, A. Gramfort, G. Obozinski, B. Thirion, and F. Bach. Sparse structured dictionary learning for brain resting-state activity modeling. In NIPS Workshop on Practical Applications of Sparse Modeling: Open Issues and New Directions, P. Wolfe. Finding the nearest point in a polytope. Math. Progr., 11(1): , M. Yuan and Y. Lin. Model selection and estimation in regression with grouped variables. Journal of The Royal Statistical Society Series B, 68(1):49 67, P. Zhao and B. Yu. On model selection consistency of Lasso. Journal of Machine Learning Research, 7: , P. Zhao, G. Rocha, and B. Yu. Grouped and hierarchical model selection through composite absolute penalties. Annals of Statistics, 37(6A): , 2009.