Projection onto A Nonnegative Max-Heap

Transcription

1 Projection onto A Nonnegatie Max-Heap Jun Liu Arizona State Uniersity Tempe, AZ 85287, USA j.iu@asu.edu Liang Sun Arizona State Uniersity Tempe, AZ 85287, USA sun.iang@asu.edu Jieping Ye Arizona State Uniersity Tempe, AZ 85287, USA jieping.ye@asu.edu Abstract We consider the probem of computing the Eucidean projection of a ector of ength p onto a non-negatie max-heap an ordered tree where the aues of the nodes are a nonnegatie and the aue of any parent node is no ess than the aue(s) of its chid node(s). This Eucidean projection pays a buiding bock roe in the optimization probem with a non-negatie maxheap constraint. Such a constraint is desirabe when the features foow an ordered tree structure, that is, a gien feature is seected for the gien regression/cassification task ony if its parent node is seected. In this paper, we show that such Eucidean projection probem admits an anaytica soution and we deeop a top-down agorithm where the key operation is to find the so-caed maxima root-tree of the subtree rooted at each node. A naie approach for finding the maxima root-tree is to enumerate a the possibe root-trees, which, howeer, does not scae we. We reea seera important properties of the maxima root-tree, based on which we design a bottom-up agorithm with merge for efficienty finding the maxima roottree. The proposed agorithm has a (worst-case) inear time compexity for a sequentia ist, and O(p 2 ) for a genera tree. We report simuation resuts showing the effectieness of the max-heap for regression with an ordered tree structure. Empirica resuts show that the proposed agorithm has an expected inear time compexity for many specia cases incuding a sequentia ist, a fu binary tree, and a tree with depth. Introduction In many regression/cassification probems, the features exhibit certain hierarchica or structura reationships, the usage of which can yied an interpretabe mode with improed regression/cassification performance [25]. Recenty, there hae been increasing interests on structured sparisty with arious approaches for incorporating structures; see [7, 8, 9, 7, 24, 25] and references therein. In this paper, we consider an ordered tree structure: a gien feature is seected for the gien regression/cassification task ony if its parent node is seected. To incorporate such ordered tree structure, we assume that the mode parameter x R p foows the non-negatie max-heap structure : P = {x,x i x j (x i,x j ) E t }, () where T t = (V t,e t ) is a target tree with V t = {x,x 2,...,x p } containing a the nodes and E t a the edges. The constraint set P impies that if x i is the parent node of a chid node x j then the aue of x i is no ess than the aue of x j. In other words, if a parent node x i is, then any of its chid nodes x j is aso. Figure iustrates three specia tree structures: ) a fu binary tree, 2) a sequentia ist, and 3) a tree with depth. To dea with the negatie mode parameters, one can make use of the technique empoyed in [24], which soes the scaed ersion of the east square estimate.

2 x x x2 x3 x x2 x3 x4 x5 x6 x7 x2 x3 x4 x5 x6 x7 x4 x5 x6 x7 (a) (b) (c) Figure : Iustration of a non-negatie max-heap depicted in(). Pots(a), (b), and(c) correspond to a fu binary tree, a sequentia ist, and a tree with depth, respectiey. The set P defined in () induces the so-caed heredity principe [3, 6, 8, 24], which has been proen effectie for high-dimensiona ariabe seection. In a recent study [2], Li et a. conducted a meta-anaysis of 3 data sets from pubished factoria experiments and concuded that an oerwheming majority of these rea studies conform with the heredity principes. The ordered tree structure is a specia case of the non-negatie garrote discussed in [24] when the hierarchica reationship is depicted by a tree. Therefore, the asymptotic properties estabished in [24] are appicabe to the ordered tree structrue. Seera reated approaches that can incorporate the ordered tree structure incude the Wedge approach [7] and the hierarchica group Lasso [25]. The Wedge approach incorporates such ordering information p i= (x2 i t i +t i ), with by designing a penaty for the mode parameter x as Ω(x P) = inf t P 2 tree being a sequentia ist. By imposing the mixed - 2 norm on each group formed by the nodes in the subtree of a parent node, the hierarchica group Lasso is abe to incorporate such ordering information. The hierarchica group Lasso has been appied for muti-task earning in [] with a tree structure, and the efficient computation was discussed in [, 5]. Compared to Wedge and hierarchica group Lasso, the max-heap in () incorporates such ordering information in a direct way, and our simuation resuts show that the max-heap can achiee ower reconstruction error than both approaches. In estimating the mode parameter satisfying the ordered tree structure, one needs to soe the foowing constrained optimization probem: minf(x) (2) x P for some conex function f( ). The probem(2) can be soed ia many approaches incuding subgradient descent, cutting pane method, gradient descent, acceerated gradient descent, etc [9, 2]. In appying these approaches, a key buiding bock is the so-caed Eucidean projection of a ector onto the conex set P: π P () = argmin x P 2 x 2 2, (3) which ensures that the soution beongs to the constraint set P. For some specia set P (e.g., hyperpane, hafspace, and rectange), the Eucidean projection admits a simpe anaytica soution, see [2]. In the iterature, researchers hae deeoped efficient Eucidean projection agorithms for the -ba [5, 4], the / 2 -ba [], and the poyhedra [4, 22]. When P is induced by a sequentia ist, a inear time agorithm was recenty proposed in [26]. Without the non-negatie constraints, probem(3) is the so-caed isotonic regression probem[6, 2]. Our major technica contribution in this paper is the efficient computation of (3) for the set P defined in (). In Section 2, we show that the Eucidean projection admits an anaytica soution, and we deeop a top-down agorithm where the key operation is to find the so-caed maxima root-tree of the subtree rooted at each node. In Section 3, we design a bottom-up agorithm with merge for efficienty finding the maxima root-tree by using its properties. We proide empirica resuts for the proposed agorithm in Section 4, and concude this paper in Section 5. 2 Atda: A Top-Down Agorithm In this section, we deeop an agorithm in a top-down manner caed Atda for soing (3). With the target tree T t = (V t,e t ), we construct the input tree T = (V,E) with the input ector, where V = {, 2,..., p } and E = {( i, j ) (x i,x j ) E t }. For the conenience of presenting our proposed agorithm, we begin with seera definitions. We aso proide some exampes for eaborating the definitions in the suppementary fie A.. 2

3 Definition. For a non-empty tree T = (V,E), we define its root-tree as any non-empty tree T = (Ṽ,Ẽ) that satisfies: ) Ṽ V, 2) Ẽ E, and 3) T shares the same root as T. Definition 2. For a non-empty tree T = (V,E), we define R(T) as the root-tree set containing a its root-trees. Definition 3. For a non-empty tree T = (V,E), we define ( m(t) = max ) i V i,, (4) V which equas the mean of a the nodes in T if such mean is non-negatie, and otherwise. Definition 4. For a non-empty tree T = (V,E), we define its maxima root-tree as: where M max (T) = arg max T=(Ṽ,Ẽ): T R(T),m( T)=m max(t) Ṽ, (5) m max (T) = max m( T) (6) T R(T) is the maxima aue of a the root-trees of the tree T. Note that if two root-trees share the same maxima aue, (5) seects the one with the argest tree size. When T = (Ṽ,Ẽ) is a part of a arger tree T = (V,E), i.e., Ṽ V and Ẽ E, we can treat T as a super-node of the tree T with aue m( T). Thus, we hae the foowing definition of a super-tree (note that a super-tree proides a disjoint partition of the gien tree): Definition 5. For a non-empty tree T = (V,E), we define its super-tree as S = (V S,E S ), which satisfies: ) each node in V S = {T,T 2,...,T n } is a non-empty tree with T i = (V i,e i ), 2) V i V and E i E, 3) V i Vj =,i j and V = n i= V i, and 4) (T i,t j ) E S if and ony if there exists a node in T j whose parent node is in T i. 2. Proposed Agorithm We present the pseudo code for soing (3) in Agorithm. The key idea of the proposed agorithm is that, in the i-th ca, we find T i = M max (T), the maxima root-tree of T, set x corresponding to the nodes of T i to m i = m max (T) = m(t i ), remoe T i from the tree T, and appy Atda to the resuting trees one by one recursiey. Agorithm A Top-Down Agorithm: Atda Input: the tree structure T = (V,E), i Output: x R p : Set i = i+ 2: Find the maxima root-tree of T, denoted by T i = (V i,e i ), and set m i = m(t i ) 3: if m i > then 4: Set x j = m i, j V i 5: Remoe the root-tree T i from T, denote the resuting trees as T, T 2,..., T ri, and appy Atda( T j,i), j =,2,...,r i 6: ese 7: Set x j = m i, j V i 8: end if 2.2 Iustration & Justification For a better iustration and justification of the proposed agorithm, we proide the anaysis of Atda for a specia case the sequentia ist in the suppementary fie A.2. Let us anayze Agorithm for the genera tree. Figure 2 iustrates soing (3) ia Agorithm for a tree with depth 3. Pot (a) shows a target tree T t, and pot (b) denotes the input tree T. The dashed frame of pot (b) shows M max (T), the maxima root-tree of T, and 3

4 x x2 x3 x x5 x6 x7 x8 x9 x x x2 x3 x4 x5 (a) (b) (c) (f) (e) Figure 2: Iustration of Agorithm for soing (3) for a tree with depth 3. Pot (a) shows the target tree T t, and pots (b-e) iustrate Atda. Specificay, pot (b) denotes the input tree T, with the dashed frame dispaying its maxima root-tree; pot (c) depicts the resuting trees after remoing the maxima root-tree in pot (b); pot (d) shows the resuting super-tree (we treat each tree encosed by the dashed frame as a super-node) of the agorithm; pot (e) gies the soution x R 5 ; and the edges of pot (f) show the dua ariabes, from which we can aso obtain the optima soution x (refer to the proof of Theorem ). (d) we hae M max (T) = 3. Thus, we set the corresponding entries of x to 3. Pot (c) depicts the resuting trees after remoing the maxima root-tree in pot (b), and pot (d) shows the generated maxima root-trees (encosed by dashed frame) by the agorithm. When treating each generated maxima root-tree as a super-node with the aue defined in Definition 3, pot (d) is a super-tree of the input tree T. In addition, the super-tree is a max-heap, i.e., the aue of the parent node is no ess than the aues of its chid nodes. Pot (e) gies the soution x R 5. The edges of pot (f) correspond to the aues of the dua ariabes, from which we can aso obtain the optima soution x R 5. Finay, we can obsere that the non-zero entries of x constitute a cut of the origina tree. We erify the correctness of Agorithm for the genera tree in the foowing theorem. We make use of the KKT conditions and ariationa inequaity [2] in the proof. Theorem. x = Atda(T, ) proides the unique optima soution to (3). Proof: As the objectie function of (3) is stricty conex and the constraints are affine, it admits a unique soution. After running Agorithm, we obtain the sequences {T i } k i= and {m i } k i=, where k satisfies k p. It is easy to erify that the trees T i,i =,2,...,k constitute a disjoint partition of the input tree T. With the sequences {T i } k i= and {m i} k i=, we can construct a super-tree of the input tree T as foows: ) we treat T i as a super-node with aue m i, and 2) we put an edge between T i and T j if there is an edge between the nodes of T i and T j in the input tree T. With Agorithm, we can erify that the resuting super-tree has the property that the aue of the parent node is no ess than its chid nodes. Therefore, x = Atda(T,) satisfies x P. Let x and denote a subset of x and corresponding to the indices appearing in the subtree T, respectiey. Denote P = {x : x,x i x j,( i, j ) E }, I = { : m > },I 2 = { : m = }. Our proof is based on the foowing inequaity: min x P 2 x 2 2 min x I P 2 x min x I P 2 x 2 2, (7) 2 which hods as the eft hand side has the additiona inequaity constraints compared to the right hand side. Our methodoogy is to show that x = Atda(T,) proides the optima soution to the right hand side of (7), i.e., x = arg min x P 2 x 2 2, I, (8) x = arg min x P 2 x 2 2, I 2, (9) 4

5 which, together with the fact 2 x 2 2 min x P 2 x 2 2, x P ead to our main argument. Next, we proe (8) by the KKT conditions, and proe (9) by the ariationa inequaity [2]. Firsty, I, we introduce the dua ariabe y ij for the edge ( i, j ) E, and y ii if i L, where L contains a the eaf nodes of the tree T. Denote the root of T by r. For a i V, i r, we denote its parent node by ji, and for the root r, we denote j r = r. We et C i = {j j is a chid node of i in the tree T }. R i = {j j is in the subtree of T rooted at i }. To proe (8), we erify that the prima ariabe x = Atda(T,) and the dua ariabe ỹ satisfy the foowing KKT conditions: ( i, j ) E, x i x j () ( i, j ) E,( x i x j )ỹ ij = () i L,ỹ ii x i = (2) i V, x i i ỹ ij +ỹ jii = (3) j C i ( i, j ) E,ỹ ij (4) i L,ỹ ii, (5) where ỹ jr r = (Note that ỹ jr r is a dua ariabe, and it is introduced for the simpicity of presenting (2)), and the dua ariabe ỹ is set as: ỹ ii =, i L, (6) ỹ jii = i m + ỹ ij, i V. (7) According to Agorithm, x i = m >, i V, I. Thus, we hae ()-(2) and (5). It foows from (7) that (3) hods. According to (6) and (7), we hae ỹ jii = j Ri m, i V, (8) j R i j C i where Ri denotes the number of eements in R i, the subtree of T rooted at i. From the nature of the maxima root-tree T, I, we hae j R i j Ri m. Otherwise, if j R i j < Ri m, we can construct from T a new root-tree T by remoing the subtree of T rooted at i, so that T achiees a arger aue than T. This contradicts with the argument that T, I is the maxima root-tree of the working tree T. Therefore, it foows from (8) that (4) hods. Secondy, we proe (9) by erifying the foowing optimaity condition: x x, x, x P, I 2, (9) whichistheso-caedariationainequaityconditionfor x beingtheoptimasoutionto(9). According to Agorithm, if I 2, we hae x i =, i V. Thus, (9) is equiaent to x,, x P, I 2. (2) For a gien x P, if x i =, i V, (2) naturay hods. Next, we consider x. Denote by x the minima nonzero eement in x, and T = (V,E ) a tree constructed by remoing the nodes corresponding to the indices in the set {i : x i =, i V } from T. It is cear that T shares the same root as T. It foows from Agorithm that i: i V i. Thus, we hae x, = x i + (x i x ) i (x i x ) i. i: i V i: i V 5 i: i V

6 If x i = x, i V, we arrie at (2). Otherwise, we set r = 2; denote by x r the minima nonzero eement in the set {x i r j= x j : i V r }, and T r = (V r,er ) a subtree of T r by remoing those nodes with the indices in the set {i : x i r j= x j =, i V r }. It is cear that T r shares the same root as T r and T as we, so that it foows from Agorithm that i: i V r i. Therefore, we hae r (x i x j) i = x r i + r (x i x j) i r (x i x j) i. (2) i: i V r j= i: i V r i: i V r j= i: i V r Repeating the aboe process unti V r is empty, we can erify that (2) hods. For a better understanding of the proof, we make use of the edges of Figure 2 (f) to show the dua ariabes, where the edge connecting i and j corresponds to the dua ariabe ỹ ij, and the edge starting from the eaf node i corresponds to the dua ariabe ỹ ii. With the dua ariabes, we can compute x ia (3). We note that, for the maxima root-tree with a positie aue, the associated dua ariabes are unique, but for the maxima root-tree with zero aue, the associated dua ariabes may not be unique. For exampe, in Figure 2 (f), we set ỹ ii = for i = 2, ỹ ii = for i = 3, ỹ ij = 2 for i = 6,j = 2, and ỹ ij = 2 for i = 6,j = 3. It is easy to check that the dua ariabes can aso be set as foows: ỹ ii = for i = 2, ỹ ii = for i = 3, ỹ ij = for i = 6,j = 2, and ỹ ij = 3 for i = 6,j = 3. 3 Finding the Maxima Root-Tree A key operation of Agorithm is to find the maxima root-tree used in Step 2. A naie approach for finding the maxima root-tree of a tree T is to enumerate a possibe roottrees in the root-tree set R(T), and identify the maxima root-tree ia (5). We ca such an approach Anae, which stands for a naie agorithm with enumeration. Athough Anae is simpe to describe, it has a ery high time compexity (see the anaysis gien in suppementary fie A.3). To this end, we deeop Abuam (A Bottom-Up Agorithm with Merge). The underying idea is to make use of the specia structure of the maxima root-tree defined in (5) for aoiding the enumeration of a possibe root-trees. We begin the discussion with some key properties of the maxima root-tree, and the proof is gien in the suppementary fie A.4. Lemma. For a non-empty tree T = (V,E), denote its maxima root-tree as T max = (V max,e max ). Let T = ( Ṽ,Ẽ) be a root-tree of T max. Assume that there are n nodes i,..., in, which satisfy: ) ij / Ṽ, 2) i j V, and 3) the parent node of ij is in Ṽ. If n, we denote the subtree of T rooted at ij as T j = (V j,e j ),j =,2,...,n, T j max = (V j max,e j max) as the maxima root-trees of T j, and m = max j=,2,...,n m(t j max). Then, the foowings hod: () If n =, then T max = T = T; (2) If n, m( T) =, and m =, then T max = T; (3) If n, m( T) >, and m( T) > m, then T max = T; (4) If n, m( T) >, and m( T) m, then V j max V max, E j max E max and ( i, ij ) E max, j : m(t j max) = m; and (5) If n, m( T) =, and m >, then V j max V max, E j max E max and ( i, ij ) E max, j : m(t j max) = m. For the conenience of presenting our proposed agorithm, we define the operation merge as foows: Definition 6. Let T = (V,E) be a non-empty tree, and T = (V,E ) and T 2 = (V 2,E 2 ) be two trees that satisfy: ) they are composed of a subset of the nodes and edges of T, i.e., V V, V 2 V, E E, and E 2 E; 2) they do not oerap, i.e., V V 2 =, and E E 2 = ; and 3) in the tree T, i2, the root node of T 2 is a chid of i, a eaf node of T. We define the operation merge as T = merge(t,t 2,T), where T = (Ṽ,Ẽ) with V = V V2 and E = E E2 {(i, i2 )}. Next, we make use of Lemma to efficienty compute the maxima root-tree, and present the pseudo code for Abuam in Agorithm 2. We proide the iustration of the proposed agorithm and the anaysis of its computationa cost in the suppementary fie A.5 and A.6, respectiey. j= 6

7 Agorithm 2 A Bottom-Up Agorithm with Merge: Abuam Input: the input tree T = (V,E) Output: the maxima root-tree T max = (V max,e max ) : Set T = (V,E ), where V = {x i } and E = 2: if i does not hae a chid node in T then 3: Set T max = T, return 4: end if 5: whie do 6: Set m =, denote by i,..., in the n nodes that satisfy: ) ij / V, 2) ij V, and 3) the parent node of ij is in V, and denote by T j = (V j,e j ),j =,2,...,n the subtree of T rooted at ij. 7: if n = then 8: Set T max = T = T, return 9: end if : for j = to n do : Set T j max = Abuam(T j ), and m = max(m(t j max), m) 2: end for 3: if m(t ) = m = then 4: Set T max = T, return 5: ese if m( T) > and m( T) > m then 6: Set T max = T, return 7: ese 8: Set T =merge(t, T j max, T), j : m(t j max) = m 9: end if 2: end whie Making use of the fact that T is aways a aid root-tree of T max, the maxima root-tree of T, we can easiy proe the foowing resut using Lemma. Theorem 2. T max returned by Agorithm 2 is the maxima root-tree of the input tree T. 4 Numerica Simuations Effectieness of the Max-Heap Structure We test the effectieness of the max-heap structure for inear regression b = Ax, foowing the same experimenta setting as in [7]. Specificay, the eements of A R n p are generated i.i.d. from the Gaussian distribution with zero mean and standard deriation and the coumns of A are then normaized to hae unit ength. The regression ector x has p = 27 nonincreasing eements, where the first eements are set as x i = i,i =,2,..., and the rest are zeros. We compared with the foowing three approaches: Lasso [23], Group Lasso [25], and Wedge [7]. Lasso makes no use of such ordering, whie Wedge incorporates the structure by using an auxiiary ordered ariabe. For Group Lasso and Max-Heap, we try binary-tree grouping and ist-tree grouping, where the associated trees are a fu binary tree and a sequentia ist, respectiey. The regression ector is put on the tree so that, the coser the node to the root, the arger the eement is paced. In Group Lasso, the nodes appearing in the same subtree form a group. For the compared approaches, we use the impementations proided in [7] 2 ; and for Max-Heap, we soe (2) with f(x) = 2 Ax b 2 2+ρ x for some sma ρ = r A T b (we set r = 4, and 8 for the binary-tree grouping and ist-tree grouping, respectiey) and appy the acceerated gradient descent[9] approach with our proposed Eucidean projection. We compute the aerage mode error x x 2 oer 5 independent runs, and report the resutswithaaryingnumberofsampesizeninfigure3(a)&(b). Asexpected, GL-binary, MH-binary, Wedge, GL-ist and MH-ist outperform Lasso which does not incorporate such ordering information. MH-binary performs better than GL-binary, and MH-ist performs better than Wedge and GL-ist, due to the direct usage of such ordering information. In addition, the ist-tree grouping performs better than the binary-tree grouping, as it makes better usage of such ordering information

8 Lasso GL binary MH binary Gaussian Distribution for sequentia ist fu binary tree tree of depth Gaussian Distribution, Fu Binary Tree Mode error Computationa Time Computationa Time 2 3 d= d=2 d=4 d=8 d=8 d=2 Mode error Sampe size p Random Initiaization of (a) (c) (e) Wedge GL ist MH ist Computationa Time sequentia ist fu binary tree tree of depth Uniform Distribution for Computationa Time 2 3 Uniform Distribution, Fu Binary Tree d= d=2 d=4 d=8 d=8 d= Sampe size p Random Initiaization of (b) (d) (f) Figure 3: Simuation resuts. In pots (a) and (b), we show the aerage mode error x x 2 oer 5 independet runs by different approaches with the fu binary-tree ordering and the ist-tree ordering. In pots (c) and (d), we report the computationa time (in seconds) of the proposed Atda (aeraged oer runs) with different randomy initiaized input. In pots (e) and (f), we show the computationa time of Atda oer runs. Efficiency of the Proposed Projection We test the efficiency of the proposed Atda approach for soing the Eucidean projection onto the non-negatie max-heap, equipped with our proposed Abuam approach for finding the maxima root-trees. In the experiments, we make use of the three tree structures as depicted in Figure, and try two different distributions: ) Gaussian distribution with zero mean and standard deriation and 2) uniform distribution in [, ] for randomy and independenty generating the entries of the input R p. In Figure 3 (c) & (d), we report the aerage computationa time (in seconds) oer runs under different aues of p = 2 d+, where d =,2,...,2. We can obsere that, the proposed agorithm scaes ineary with the size of p. In Figure 3 (e) & (f), we report the computationa time of Atda oer runs when the ordered tree structure is a fu binary tree. The resuts show that the computationa time of the proposed agorithm is reatiey stabe for different runs, especiay for arger d or p. Note that, the source codes for our proposed agorithm hae been incuded in the SLEP package [3]. 5 Concusion In this paper, we hae deeoped an efficient agorithm for the computation of the Eucidean projection onto a non-negatie max-heap. The proposed agorithm has a (worst-case) inear time compexity for a sequentia ist, and O(p 2 ) for a genera tree. Empirica resuts show that: ) the proposed approach deas with the ordering information better than existing approaches, and 2) the proposed agorithm has an expected inear time compexity for the sequentia ist, the fu binary tree, and the tree of depth. It wi be interesting to expore whether the proposed Abuam has a worst case inear (or inearithmic) time compexity for the binary tree. We pan to appy the proposed agorithms to rea-word appications with an ordered tree structure. We aso pan to extend our proposed approaches to the genera hierarchica structure. Acknowedgments This work was supported by NSF IIS-8255, IIS , MCB-267, CCF-2577, NGA HM , and NSFC 69535,

9 References [] E. Berg, M. Schmidt, M. P. Friedander, and K. Murphy. Group sparsity ia inear-time projection. Tech. Rep. TR-28-9, Department of Computer Science, Uniersity of British Coumbia, Vancouer, Juy 28. [2] S. Boyd and L. Vandenberghe. Conex Optimization. Cambridge Uniersity Press, 24. [3] N. Choi, W. Li, and J. Zhu. Variabe seection with the strong heredity constraint and its orace property. Journa of the American Statistica Association, 5: , 2. [4] Z. Dostá. Box constrained quadratic programming with proportioning and projections. SIAM Journa on Optimization, 7(3):87 887, 997. [5] J. Duchi, S. Shae-Shwartz, Y. Singer, and C. Tushar. Efficient projection onto the -ba for earning in high dimensions. In Internationa Conference on Machine Learning, 28. [6] M. Hamada and C. Wu. Anaysis of designed experiments with compex aiasing. Journa of Quaity Technoogy, 24:3 37, 992. [7] J. Huang, T. Zhang, and D. Metaxas. Learning with structured sparsity. In Internationa Conference on Machine Learning. 29. [8] L. Jacob, G. Obozinski, and J. Vert. Group asso with oerap and graph asso. In Internationa Conference on Machine Learning, 29. [9] R. Jenatton, J.-Y. Audibert, and F. Bach. Structured ariabe seection with sparsity-inducing norms. Technica report, arxi: , 29. [] R. Jenatton, J. Maira, G. Obozinski, and F. Bach. Proxima methods for sparse hierarchica dictionary earning. In Internationa Conference on Machine Learning, 2. [] S. Kim and E. P. Xing. Tree-guided group asso for muti-task regression with structured sparsity. In Internationa Conference on Machine Learning, 2. [2] X. Li, N. Sundarsanam, and D. Frey. Reguarities in data from factoria experiments. Compexity, :32 45, 26. [3] J. Liu, S. Ji, and J. Ye. SLEP: Sparse Learning with Efficient Projections. Arizona State Uniersity, 29. [4] J. Liu and J. Ye. Efficient Eucidean projections in inear time. In Internationa Conference on Machine Learning, 29. [5] J. Liu and J. Ye. Moreau-yosida reguarization for grouped tree structure earning. In Adances in Neura Information Processing Systems, 2. [6] R. Luss, S. Rosset, and M. Shahar. Decomposing isotonic regression for efficienty soing arge probems. In Adances in Neura Information Processing Systems, 2. [7] C. Micchei, J. Moraes, and M. Ponti. A famiy of penaty functions for structured sparsity. In Adances in Neura Information Processing Systems 23, pages [8] J. Neder. The seection of terms in response-surface modes how strong is the weak-heredity principe? Annas of Appied Statistics, 52:35 38, 998. [9] A. Nemiroski. Efficient methods in conex programming. Lecture Notes, 994. [2] Y. Nestero. Introductory Lectures on Conex Optimization: A Basic Course. Kuwer Academic Pubishers, 24. [2] P. M. Pardaos and G. Xue. Agorithms for a cass of isotonic regression probems. Agorithmica, 23:2 222, 999. [22] S. Shae-Shwartz and Y. Singer. Efficient earning of abe ranking by soft projections onto poyhedra. Journa of Machine Learning Research, 7: , 26. [23] R. Tibshirani. Regression shrinkage and seection ia the asso. Journa of the Roya Statistica Society Series B, 58(): , 996. [24] M. Yuan, V. R. Joseph, and H. Zou. Structured ariabe seection and estimation. Annas of Appied Statistics, 3: , 29. [25] P. Zhao, G. Rocha, and B. Yu. The composite absoute penaties famiy for grouped and hierarchica ariabe seection. Annas of Statistics, 37(6A): , 29. [26] L.W. Zhong and J.T. Kwok. Efficient sparse modeing with automatic feature grouping. In Internationa Conference on Machine Learning, 2. 9