On the Complexity of the Weighted Fused Lasso

Transcription

1 ON THE COMPLEXITY OF THE WEIGHTED FUSED LASSO On the Compleity of the Weighted Fused Lasso José Bento Ralph Furmaniak Surjyendu Ray Abstrat The solution path of the D fused lasso for an n- dimensional input is pieewise linear with at most O(n) segments [], [2]. However, eisting proofs of this bound do not hold for the weighted fused lasso. At the same time, results for the generalized lasso, of whih the weighted fused lasso is a speial ase, allow more than Ω(3 n ) segments [3]. In this paper, we prove that the number of segments in the solution path of the weighted fused lasso is O(n 2 ), and that, for some instanes, it is Ω(n 2 ). We also give a new, very simple, proof of the O(n) bound for the fused lasso. Inde Terms Filter, Lasso, Projetion, Proimal Operator, Sum of Absolute Differenes, Total Variation, Weights I. INTRODUCTION The generalized lasso solves minimize R n 2 y A γ D, () where γ 0, D R p n, A R m n and y R m. A speial ase of this problem, whih is important for signal proessing (see [4] and referenes therein for appliations), is minimize R n 2 n ( t y t ) 2 + γ α t t+ t. (2) t= t= We all (2) the weighted -D fused lasso (WFL) with input y and weights α t 0 t, and we distinguish it from the well studied speial ase when α t = t, whih is known as the -D fused lasso (FL). There are effiient algorithms to solve FL for a fied γ. Diret algorithms, algorithms that terminate with an eat solution, inlude the Taut String algorithm, [5], and the algorithm of [6], based on dynami programing, both with a worst ase ompleity of O(n); and the algorithm of [7], whih is very fast in pratie, but has O(n 2 ) worst ase ompleity. This algorithm has reently been improved to finish in O(n) steps, [8]. There are also algorithms that an deal with WFL, for fied γ, in O(n 2 ) iterations, [9], and in O(n) iterations, [0]. Iterative algorithms, mostly first-order fied-point methods, inlude [] [9]. Some of these are based on the ADMM method, known to ahieve the fastest possible onvergene rate among all first order methods, [20], [2]. However, in many appliations, when preision is ruial, or when implementing a termination proedure has a non-negligible omputational ost, diret algorithms are preferred. Frequently, we are not just interested in solving (2) for a single γ. Let (γ) be the unique solution of (2). There are degenerate ases where () does not have a unique solution, [22]. However, our problem has a unique solution beause (2) is stritly onve. An important problem is haraterizing the set { (γ) : γ 0}, known as the solution path of (2). This might be neessary, for eample, to effiiently tune the FL, i.e., find the value of γ that gives the best result in a given as a funtion of γ If α = /50, α 2 = /2, α 3 = /2, y = 0, y 2 = /2, y 3 = /2 and y 4 = /2 then = 2 when γ [ 25, ] [, ) and otherwise. At γ = 25, variables and 2 un-fuse and 23 at γ { 25, 25 } they fuse Fig.. Eample showing that in WFL, variables an fuse and un-fuse. Hene, eisting proofs that T = O(n) do not hold for WFL. appliation, [23], [24]. Another eample is when we want to use a WFL-path-solver to find the unique solution ( γ) of n min ( t y t ) 2 subjet to α t t+ t γ. (3) R n 2 t= t= One approah is to see, f. [25], that ( γ) = (γ) if γ = ma y i i ( γ) or inversely if γ = (γ). (4) i We an then use the path { (γ)}, and (4), to find whih γ we should use in our solver to get ( γ). Relations (4) show that finding the solution paths of (3) and of (2) is equivalent. Charaterizing { (γ)} is possible beause (γ) is a ontinuous pieewise linear funtion of γ, with a finite number T of different linear segments, a result that follows diretly from the KKT onditions, [2]. Therefore, to haraterize (γ), we only need to find the ritial values {γ i } T i= at whih (γ) hanges linear segment, and the value of (γ) at these γ s. All effiient eisting algorithms that find the solution path { (γ)} in a finite number of steps are essentially homotopy algorithms. These start with (γ = 0) = y, and sequentially ompute (γ i+ ) from (γ i ). One eample is the algorithm of [2], that for FL has a ompleity of O(n log 2 n), with a speial heap implementation. The best method is the primal path algorithm of [], with O(n log n) ompleity. To understand the ompleity of finding { (γ)}, we have to bound T. For FL, [26] proves that T = O(n), and [2], [27] give a different proof of the same fat. These proofs idea is as follows. The non-differentiable penalty t t+ t in (2) implies that we have a ritial point whenever, as γ inreases, for some t, the term t+ t, goes from non-zero to zero, or vie-versa. When this happens, we say that t+ and t fuse, or, onversely, that they un-fuse. One then proves that, as γ inreases, variables never un-fuse. Hene, there are at most n fusing events, and thus T n. Unfortunately, eisting proofs do not etend to WFS. Figure provides an eample where variables fuse and un-fuse. Hene, to bound T, we are left with bounds for the generalized lasso whih, in a worst ase senario, an be Ω(3 n ) [3]. Our main ontribution is to show that T = O(n 2 ), and that, in a worst ase senario, T = Ω(n 2 ).

2 ON THE COMPLEXITY OF THE WEIGHTED FUSED LASSO 2 II. MAIN RESULTS We start by reformulating WFS, as stated in Theorem. In this theorem, and throughout the paper, we use the notation [n] = {,..., n}. Theorem. Let ỹ i = n t=i y t for i [n+] and ỹ n+ = 0. Let α i+ = α i for i [n ] and α = α n+ = 0. Let w (γ) be the unique minimizer of min w R n (w i+ w i ) 2 s.t. w i ỹ i γ α i, i [n + ] (5) i= We have that t (γ) = w t+(γ) w t (γ) for all t [n]. See [9], e.g., for a proof of Theorem. The Supplementary Material inludes another proof, based on the Moreau identity. Theorem allows us to study the number of linear segments in by studying the number of linear segments in w. Indeed, sine by Theorem we have that t (γ) = w t+(γ) w t (γ), it follows that (γ) only hanges linear segment if w (γ) hanges linear segment. Hene, if there are at most T different linear segments in w (γ), there are at most T different linear segments in (γ). Similarly, if there are at least T linear segments in w (γ), and if, for eah γ, no two onseutive omponents of w hange linear segment, then there are at least T linear segments in (γ). We an make a few simple observations about w (γ), defined in Theorem, whih we use to get bounds on T. First we introdue some notation. Figure 2 illustrates its use. We refer to wi as the ith variable or ith point. We refer to [ỹ i α i γ, ỹ i + α i γ] as the interval assoiated to the ith point. We say that the ith point is touhing its left or right boundary if wi = ỹ i α i γ or if wi = ỹ i + α i γ respetively. A point that is not touhing either side of the boundary of its interval is alled free. Otherwise, it is alled bounded. We define F (γ) = {i : wi (γ) ỹ i < α i γ} and B(γ) = [n + ]\F. In words, a point is in F if and only if it is free. A point is in B if and only if it is bounded. We define s i (γ) so that w i (γ) = ỹ i + s i (γ) α i γ. (6) In partiular, s i (γ) is (resp. -) if the point touhes its right (resp. left) boundary. For i = and i = n +, for whih the left and right boundaries are the same, s i may be hosen arbitrarily. For the ith point, we define i (γ) = ma{j B(γ) : j < i} and let i (γ) = min{j B(γ) : i < j}. In words, i and i are the pair of indies of bounded points, smaller and larger than i respetively, that are losest to i. Whenever lear from the ontet, we omit the dependeny in γ in our epressions. We now list our observations. ) Sine w (γ) is a ontinuous funtion of γ (see e.g. [2]), if a point i has α i > 0, then it annot touh its left (right) boundary for γ 2 and touh its right (left) boundary for 0 < γ < γ 2 without being free for at least one γ (γ, γ 2 ). This implies that if B and F do not hange for γ [γ, γ 2 ], then s i is onstant in γ [γ, γ 2 ]. [ỹ 7 7, ỹ ] B = {, 4, 6, 8, 9, 0} F = {2, 3, 5, 7} w w9 s 8 / 8 = =6 s 4 = 6 / =4 8. =9 6. =8 Fig. 2. Illustration of the notation used. Square horizontal brakets represent intervals on the real-line inside whih eah variable must be. Solid irles represent variables touhing their intervals boundary. B is the set of bounded variables, and F is the set of free variables. i and i identify losest bounded variables to the left and right of i. In this piture n = 9, so α = α n+ = 0. 2) If i F, then w i is determined by the position of the i th and i th points, namely, w i = w i f i + w i f i, (7) where f i = i i i i and f i = f i. Equation (7) follows from the fat that, for any i < j < i, wj omes from the i solution of the problem min {wj} j=i (w j+ w j ) 2 subjet to w i = wi and w i = w i. Its solution is that the points {wj } must divide the interval [w i, w i ] in i i equal parts, hene (7). Note that, if for all γ [γ, γ 2 ], F does not hange then, by observation, i, i, s i and s i are onstant for all γ [γ, γ 2 ], and (7) beomes a linear funtion of γ given by w i (γ) = (ỹ i f i + ỹ i fi ) + γ( α i s i f i + α i s i fi ). (8) This also implies that a hange in linear segment, and hene a ritial point, only ours when F and B hange. 3) We give a neessary ondition for γ to be a ritial point, at whih wi transitions from being free to bounded as we inrease γ. Sine w is ontinuous and pieewise linear with a finite number of linear segments (see [2]), we know that F is onstant in a small enough interval of the form I = (γ, γ ). We an ompute wi (γ ) both from (6) and taking the limit of (8) for γ I and γ γ to onlude that for γ = γ (ỹ i f i + ỹ i fi ) + γ( α i s i f i + α i s i fi ) = ỹ i + α i s i γ, (9) where s i is evaluated at γ = γ and i, i, s i and s i are evaluated at any γ I. Furthermore, if s i (γ ) = +, then, for γ I, the left hand side, l.h.s., of (9) is stritly smaller than the right hand side, r.h.s, and, as γ inreases to γ, the l.h.s must inrease until it is equal to the r.h.s. Hene, we onlude that the following relation holds between their rates of growth α i s i f i + α i s i fi > α i s i. Similarly, if s i (γ ) =, then α i s i f i + α i s i fi < α i s i. We an thus write that, if the ith point transitions from free to bounded, then α i s i s i f i + α i s i s i fi > α i. (0) Our first result is a new very simple proof, when ompared to [2], [26], [27], of the known fat that, for FL, T = O(n). Our seond and third result are new altogether. Theorem 2. FL has at most O(n) different linear segments. Proof. It is enough to prove that w has at most O(n) different linear segments. For FL, we have α i+ = for all i [n ]. Therefore, if w i is free and, as γ inreases, it beomes bounded, then, by (0), we must have that < s i s i f i +

3 ON THE COMPLEXITY OF THE WEIGHTED FUSED LASSO 3 s i s i fi. This implies that < s i s i f i +s i s i fi f i + f i =, whih is a ontradition. Therefore, a point i never goes from F to B. Sine a ritial point in w (γ) only ours when F hanges (by observation 2), and sine F an only hange by the addition of n variables at most, we have T = O(n). Theorem 3. WFL has O(n 2 ) different linear segments. Proof. At any ritial value γ at whih B hanges, B might hange by multiple points. Some are added, others are removed. Let i(γ, r) be the rth element hanging in B at γ, and let i (γ, r) and i (γ, r) be indies of bounded points, larger and smaller than i(γ, r) respetively, that are losest to i(γ, r). Let i < j and onsider the set S ij = {γ : (i (γ, r), i (γ, r)) = (i, j) for some r}, i.e., the set of ritial values at whih some point that is being added or removed from B has as its losest bounded points to the left and to the right the points i and j. It suffies to prove that S ij 8, sine then total number of ritial points is equal to i<j S ij 8 ( n 2) = O(n 2 ). Let γ S ij. Let i < k < j. By equation (7), we know that wk (γ ) an be epressed as a linear funtion of γ. Hene, together with the fat that we must have wk ỹ k +γ [ α k, α k ], we know that γ [a k, b k ], for some a k 0 and b k, whih depend on s i and s j, and the parameters α, ỹ of the problem, where b k might be. In words, for wk (γ ) to be feasible, γ must be in some interval [a k, b k ]. This implies that γ i<k<j [a k, b k ] = [A ij, B ij ], for some A i,j 0 and B i,j, whih depend on s i and s j, and the parameters α, ỹ of the problem, where B i,j might be. We argue that it must be that γ is equal to either A ij or B ij. Indeed, sine γ is a ritial point, if k = i(γ, r) for some r, then wk must be at the boundary of ỹ k + γ [ α k, α k ]. This implies that γ is either a k or b k, and hene that γ is at the boundary of i<k <j[a k, b k ]. There are 4 hoies for the pair (s i, s j ), and, for eah of these hoies, γ an be either A ij or B ij. Hene, for any pair (i, j), there are at most 8 possible values for γ. Thus S ij 8. Theorem 4. There eists α and y suh that WFL has Ω(n 2 ) different linear segments. One eample is to hose α and y suh that α and ỹ satisfy α i = (i ) 2, i [n], α n+ = 0, () ỹ i = ( ) i q i, i [n], ỹ n+ = 0, (2) q =, q 2 =2, q i+2 =2q i+ q i +2g i+2 +, i [n 2], (3) g 3 = /3, g i+3 = 2g i+2 +. (4) Remark 5. This, together with Thrm. 3, implies that there are eamples where variables fuse Θ(n 2 ) times and un-fuse Θ(n 2 ) times. Furthermore, its proof implies that the differene between the number of fuse and un-fuse events is O(n). What is the idea behind Theorem (4)? The fat α i grows super-linearly with i, allows us to have a value of γ around whih the ith interval is responsible for driving the behavior of w (γ). Let us all this the ith epoh. The fat that ỹ i osillates and diverges eponentially fast with i, drives points between epohs to alternate between being bounded at their right or left boundary, aording to whether ỹ i is very large and negative or very large and positive. Sine there are n epohs, and sine, in general, from one epoh to the net, Ω(n) points hange from their right to left boundary (or vie versa), we have Ω(n 2 ) fuse and un-fuse events. Proof. We are going to produe a set of suffiient onditions that guarantee that the number of ritial points in w is Ω(n 2 ). It is then an algebra eerise, whih we omit, to hek that these onditions are satisfied by our hoie above. Finally, we prove why these same onditions imply that no two onseutive omponents of w hange their linear segment at the same time, and hene why also has at least Ω(n 2 ) different linear segments. In partiular, our onditions will imply the eistene of α 2,..., α n 0 and ỹ,... ỹ n, suh that there eits a sequene of ritial points 0 < γ 3 < γ 4 < < γ n suh that the following two senarios hold true: ) for γ = γ 2k, 3 2k n, every point wi with i {2,..., 2k } is touhing its right boundary; 2) for γ = γ 2k+, 3 2k + n, every point wi with i {2,..., 2k} is touhing its left boundary; Both senarios imply that, for γ [γ r, γ r+ ), r {3,..., n }, every point wi, i {2,..., r }, hanges from touhing one side of its boundary to the other side of its boundary. Note that sine α 2,... α n 0, if γ > 0, a point annot be simultaneously touhing its left and right boundary. Hene, w (γ) has at least r 2 ritial points in γ [γ r, γ r+ ). Hene, for γ [γ 2, γ n ], there are at least + + n = n(n )/2 (5) ritial points. Let us be more speifi about the two senarios. In Senario, in addition to what we have already desribed, for γ = γ r and r = 2k, we also want the intervals [ỹ α γ, ỹ + α γ],..., [ỹ r α r γ, ỹ r + α r γ] to be nested, with the intervals for large i ontaining the intervals for small i. We also want the left boundary of the interval [ỹ r α r γ, ỹ r + α r γ] to be larger than the right most limit of all the intervals involving indies smaller than r. The following piture illustrates these onditions. ỹ 2 ± 2 ỹ ± ỹ 2k ± 2k 2k ỹ 2k ± 2k 2k 2k 2k (...) w w 2 w 2k In Senario 2, in addition to what we have already desribed, for γ = γ r and r = 2k +, we want the same onditions as in Senario but now with left and right reversed. The following piture illustrates these onditions. ỹ 2k+ ± 2k+ 2k+ ỹ 2k ± 2k 2k+ (...) w2k w2 w ỹ 2 ± 2 2k+ ỹ ± 2k+

4 ON THE COMPLEXITY OF THE WEIGHTED FUSED LASSO 4 Note that in both senarios, we do not are where the points r +,..., n are. As we eplain net, a set of suffiient onditions for Senario (Senario 2) to hold is that, for all r {3,..., n}, ỹ i+ ỹ i < ( α i+ α i )γ r, i [r 2], (6) ỹ i+2 2ỹ i+ +ỹ i <( α i+2 2 α i+ + α i )γ r, i [r 3], (7) ( ) r (ỹ r 2ỹ r + ỹ r 2 )>( α r + 2 α r α r 2 )γ r. (8) Condition (6) diretly implies that the first r intervals are nested. Condition (7) implies that if the (i + 2)th and ith points are touhing their right (left) boundary, then the (i + )th point must be touhing its right (left) boundary. This an be seen by solving the simple quadrati problem min w (w i+2 w i+ ) 2 + (w i+ w i ) 2 subjet to w i+2 and w i+ behind at their boundaries, and w i+ being inside its interval. Condition (8) implies that, no matter where w r is, even if w r is touhing its left (right) boundary, the (r )th point must touh its right (left) boundary. This an also be seen by solving a simple quadrati problem involving three variables. These three onditions together imply that the first r points are touhing their right (left) boundary. As we eplain net, these onditions are in turn implied by the following set of onditions. These also imply that γ r is inreasing. α i+2 2 α i+ + α i > 0, i [n 2], (9) α i+ α i > 0, i [n ], (20) γ 3 > q 2 q, (2) α 2 { α q i+2 2q i+ +q i γ i+3 >ma γ i+2,, q } i+2 q i+, (22) α i+2 2 α i+ + α i α i+2 α i+ i [n 3], q 2 > q, (23) q i+2 >2q i+ q i +( α i+2 +2 α i+ α i )γ i+2, i [n 2], (24) ỹ i = ( ) i q i, i [n]. (25) Conditions (20), (2) and (23) imply that γ 3 > 0. Condition (22) further implies that 0 < γ 3 < < γ n. Condition (24) and (25) imply ondition (8). Sine α > 0, ondition (9) implies that α i+2 +2 α i+ α i > 0 for all i [n 2] and thus, sine γ i+2 > 0, ondition (23) together with (24) imply that q i+2 2q i+ +q i > 0 for all i [n 2] and that q i+ q i > 0 for all i [n ]. Therefore, ỹ i+2 2ỹ i+ + ỹ i = q i+2 2q i+ +q i for all i [n 2] and ỹ i+ ỹ i = q i+ q i for all i [n ]. These two equations, together with onditions (9), (20), (2) and (22) imply that (6) holds for all i [n ], if we replae γ r by γ i+2. These also imply that (7) holds for all i [n 2], if we replae γ r by γ i+3 there. But sine γ i is inreasing, we have that (6) and (7) hold true as speified. We finally argue by ontradition why onditions (9)-(25) also imply that no two onseutive omponents of w stop touhing, or start touhing, their boundary at the same γ. We prove this only for values of γ that are in some of the intervals [γ r, γ r+ ) that ontribute to our n 2 estimate in (5). Assume that both wi and w i+ have a ritial point at γ = γ. Assume also that γ [γ r, γ r+ ) for some r > i +. Note that γ must satisfy this ondition if it is to ontribute for our n 2 estimate in (5). It must be that wi w i = wi w i+ = w i+ w i+2. At the same time, and as we saw in the previous paragraph, onditions (9)-(25) imply that ỹ i+ 2ỹ i +ỹ i < ( α i+ 2 α i + α i )γ r ( α i+ 2 α i + α i )γ, whih in turn implies that wi w i < w i+ w i, whih is a ontradition. III. NUMERICAL EXPERIMENTS Figure 3-(left) shows a numerial omputation of the number of ritial points as a funtion of n for the eample ()-(4). As Theorem 4 predits, the number of ritial points grows quadratially with n. One an also observe that differene between fuse and un-fuse events is O(n). For Figure 3-(right), we generated 00 random sets of α and y, and, for eah size n, we show on the y-ais the average number of fuse events and un-fuse events observed over these 00 runs. Eah α i was sampled from a uniform distribution in [0, ], independently aross α s, and eah y i was sampled from a N (0, 0), independently aross y s. Although Theorem 4 tells us that we an observe Ω(n 2 ) fuse and un-fuse events, in our random instanes, un-fuse events are rare and both types of events seem to grow linearly with n. Critial points an be omputed using, for eample, [2]. In Supplementary Material we give simple algorithm whih we use to ompute the ritial points based on Theorem. # ritial points all events fuse events # ritial points un-fuse events Input size squared, n fuse events 20 un-fuse events Input size, n Fig. 3. (Left) Number of ritial points as funtion of n 2 for the eample of Theorem 4; (Right) Number of fuse and un-fuse events in random instanes of WFL, as a funtion of n. IV. FUTURE WORK The weighted fused lasso on an input of size n is substantially different from the equal-weights fused lasso: two onseutive omponents an both beome equal ( fuse ) and beome unequal ( un-fuse ) multiple times along the solution path. We have shown that there are instanes with Ω(n 2 ) of these fuse / un-fuse events, and that no instane an have more than O(n 2 ) events. We also gave a very simple proof of why, in the equal-weights fused lasso, there are O(n) events. Future work should inlude finding onditions for the weights α and input y under whih a O(n) bound holds, and onditions for whih the number of fuse and un-fuse events are substantially different. It would then be useful to ompute how likely it is for these onditions to be satisfied, under different stohasti models for the input. ACKNOWLEDGMENT We would like to thank Piotr Suwara for his important help in some of the proofs. This work was partially supported by the grants NSF-IIS-7429 and NIH-U0AI24302.

5 ON THE COMPLEXITY OF THE WEIGHTED FUSED LASSO 5 REFERENCES [] H. Hoefling, A path algorithm for the fused lasso signal approimator, Journal of Computational and Graphial Statistis, vol. 9, no. 4, pp , 200. [2] R. J. Tibshirani, The solution path of the generalized lasso. Stanford University, 20. [3] J. Mairal and B. Yu, Compleity analysis of the lasso regularization path, arxiv preprint arxiv: , 202. [4] A. Chambolle, V. Duval, G. Peyré, and C. Poon, Geometri properties of solutions to the total variation denoising problem, Inverse Problems, vol. 33, no., p , 206. [5] P. L. Davies and A. Kova, Loal etremes, runs, strings and multiresolution, Annals of Statistis, pp. 48, 200. [6] N. A. Johnson, A dynami programming algorithm for the fused lasso and l 0-segmentation, Journal of Computational and Graphial Statistis, vol. 22, no. 2, pp , 203. [7] L. Condat, A diret algorithm for -d total variation denoising, IEEE Signal Proessing Letters, vol. 20, no., pp , 203. [8] Total variation denoising of -d signals ode. grenoble-inp.fr/ laurent.ondat/download/condat TV D v2.. Aessed: [9] A. Barbero and S. Sra, Modular proimal optimization for multidimensional total-variation regularization, arxiv preprint arxiv:4.0589, 204. [0] L. Dümbgen, A. Kova, et al., Etensions of smoothing via taut strings, Eletroni Journal of Statistis, vol. 3, pp. 4 75, [] A. Bek and M. Teboulle, Fast gradient-based algorithms for onstrained total variation image denoising and deblurring problems, IEEE Transations on Image Proessing, vol. 8, no., pp , [2] A. Chambolle and T. Pok, A first-order primal-dual algorithm for onve problems with appliations to imaging, Journal of mathematial imaging and vision, vol. 40, no., pp , 20. [3] B. Wahlberg, S. Boyd, M. Annergren, and Y. Wang, An admm algorithm for a lass of total variation regularized estimation problems, IFAC Proeedings Volumes, vol. 45, no. 6, pp , 202. [4] S. Bonettini and V. Ruggiero, On the onvergene of primal dual hybrid gradient algorithms for total variation image restoration, Journal of Mathematial Imaging and Vision, vol. 44, no. 3, pp , 202. [5] G.-B. Ye and X. Xie, Split bregman method for large sale fused lasso, Computational Statistis & Data Analysis, vol. 55, no. 4, pp , 20. [6] A. Barbero and S. Sra, Fast newton-type methods for total variation regularization, in Proeedings of the 28th International Conferene on Mahine Learning (ICML-), pp , Citeseer, 20. [7] L. Condat, A primal dual splitting method for onve optimization involving lipshitzian, proimable and linear omposite terms, Journal of Optimization Theory and Appliations, vol. 58, no. 2, pp , 203. [8] A. Ramdas and R. J. Tibshirani, Fast and fleible admm algorithms for trend filtering, Journal of Computational and Graphial Statistis, vol. 25, no. 3, pp , 206. [9] Z.-F. Pang and Y. Duan, Primal-dual method to the minimized surfae regularization for image restoration, arxiv preprint arxiv: , 206. [20] G. França and J. Bento, An epliit rate bound for over-relaed admm, in Information Theory (ISIT), 206 IEEE International Symposium on, pp , IEEE, 206. [2] G. França and J. Bento, How is distributed admm affeted by network topology?, arxiv preprint arxiv: , 207. [22] R. J. Tibshirani et al., The lasso problem and uniqueness, Eletroni Journal of Statistis, vol. 7, pp , 203. [23] Y. Dong, M. Hintermüller, and M. M. Rinon-Camaho, Automated regularization parameter seletion in multi-sale total variation models for image restoration, Journal of Mathematial Imaging and Vision, vol. 40, no., pp , 20. [24] Y.-W. Wen and R. H. Chan, Parameter seletion for total-variationbased image restoration using disrepany priniple, IEEE Transations on Image Proessing, vol. 2, no. 4, pp , 202. [25] I. Loris, On the performane of algorithms for the minimization of l-penalized funtionals, Inverse Problems, vol. 25, no. 3, p , [26] J. Friedman, T. Hastie, H. Höfling, R. Tibshirani, et al., Pathwise oordinate optimization, The Annals of Applied Statistis, vol., no. 2, pp , [27] R. J. Tibshirani and J. Taylor, Supplement: Proofs and tehnial details for the solution path of the generalized lasso, [28] P. R. Halmos, Measure theory, vol. 8. Springer, 203. [29] R. K. Sundaram, A first ourse in optimization theory. Cambridge university press, 996.

6 ON THE COMPLEXITY OF THE WEIGHTED FUSED LASSO 6 Supplementary Material to On the Compleity of the Weighted fused Lasso V. PROOF OF THEOREM We begin by reviewing Moreau s identity. Let f() be a losed proper onve funtion and let F (y) = arg min f() + 2 y 2 2 (26) be its proimal operator. Let be the Fenhel dual of f and let ˆf() = sup z, f(z) (27) z ˆF (y) = arg min ˆf() + 2 y 2 2 (28) be its proimal operator. Moreau s identity is F (y) = y ˆF (y). (29) Proof of Theorem. If f() = n t= γα t t+ t, then (γ) = F (y). Therefore, Theorem amounts to an optimization problem to ompute F (y), from whih we an ompute F, and hene, using (29). We start by making a hange of variables in f () = sup z t= n z t t α t z t+ z t. (30) t= Let h 0 = z and h t = z t+ z t, for t [n ]. This implies that z t+ = t i=0 h i and thus that n t= z t t = n t t= i=0 h i t = n t=0 h t( n i=t+ i) = n t=0 h tu t, where we have defined u t = n i=t+ i Therefore, we an rewrite (30) as n n f () = sup h t u t γα t h t, (3) h t=0 t=0 where we have etended α suh that α 0 = 0. Problem (3) breaks down into n independent one-dimensional problems of the form sup h t u t γα t h t, (32) h t that have solution 0 if u t [ γα t, γα t ], and otherwise. Therefore, ˆf() = if ut = n i=t+ i / [ γα t, γα t ] for some t = 0,, n, and ˆf() = 0 otherwise. We an now write F (y) = y ˆF (y) = y arg min ( t y t ) 2 (33) t= subjet to i [ γα t, γα t ], t = 0,, n. i=t+ Finally, we make the following hange of variable in (33): w t = n i=t ( i y i ) for all t [n + ], where we define w n+ = 0. This implies that i = y i w i+ + w i for i [n], and hene that i = F i (y) = y i ˆF i (y) = y i (y i w i+ + w i ) (34) = w i+ w i (35) for all i [n], where w = arg min w (w t+ w t ) 2 (36) t= subjet to w t+ i=t+ y i [ γα t, γα t ], t = 0,, n. VI. SIMPLE ALGORITHM TO COMPUTE CRITICAL POINTS Before we introdue the algorithm, we need a fourth observation, in addition to the three observations already made in the main tet. For this observation to be valid, we assume that the α s are not in some set of measure zero on the spae of all possible α s. The following tehnial results, whose proofs are standard, e.g. [28], [29], will be used to etend these observations to any set of α s. We inlude a self ontained proof of these lemmas in Setion VII. Lemma 6. Let X R k, where X has zero Lebesgue measure. For any ɛ > 0, there eists a point y / X suh that y < ɛ. Lemma 7. The funtion (γ, α) is ontinuous at every point of the domain (γ, α) 0. Remark 8. Sine wt+(γ, α) = ỹ + t i= i (γ, α), this also proves that w (γ, α) is ontinuous at every point of its domain. 4) We an assume, without loss of generality, that at most one omponent of w transitions from free to bounded, or vie versa, at eah γ. This follows from the fat that, for two indies i and j (or more) to satisfy (9) for the same γ, the values of α i, α j, α i, α i, α j and α i must belong to some set ζ of measure zero (in the spae of possible α s). Using Lemma 6 and Lemma 7, we an then etend our arguments made outside ζ to this set as well. Our four observations allows us to desribe a simple diret algorithm to ompute the path w (γ). Its interpretation is simple: start with γ = 0. Inrease γ and, as the intervals ỹ i + γ[ α i, α i ] grow larger, keep trak of whih points are touhing either limit of its interval. For eah interval of values of γ for whih B is fied, we an use (8) to ompute how eah point moves, figure out whih net point that will beome free, or bounded, and hene ompute the net B. The following algorithm desribes how to ompute {γ i } T i=, the ritial points. However, it an be modified to ompute {w (γ i )} T i= with only a multiplying fator slow down in the run time. ) Start with γ = 0, F = {} and iteration number r =. All points are bounded. 2) At iteration r, use (9) to find γ s at whih an i / F might enter F or at whih an i F might leave F. Store these values as γ (),γ (2),... and γ (), γ (2),... respetively, where the upper inde (i) in γ (i) and γ (i) refers to the inde of the point that might enter or leave F. If point i annot leave F in round r, then γ (i) F in round r, then γ (i) =. =. If point i annot enter

7 ON THE COMPLEXITY OF THE WEIGHTED FUSED LASSO 7 These values are not all ritial points on w (γ), but all ritial points satisfy (9), and, in partiular, the smallest value produed orresponds to the net ritial point. 3) Append or remove from F the inde i = min{ min i:γ (i) >γ r γ (i), min i:γ (i) >γ r γ (i) }. (37) Set γ r+ = γ (i) or γ r+ = γ (i). Set γ (i) = or γ (i) =, the hoie depending on whether there is a point leaving, or entering F. We do this last assignment to avoid a yle of removing and adding the same point to F, ad infinitum, for the same value of γ, without the algorithm progressing. 4) Update γ (j) or γ (j) for all j {i,..., i, i+,..., i }. All the other previously omputed γ (j) and γ (j) will be the same. For later purposes, let us all by k r the number of values updated in this step. 5) Terminate if B = {, n + }, otherwise go bak to step 2. In the above algorithm, it is onvenient to represent B as a linked list suh that we an aess its elements in the order of their indies. We do not epliitly represent F. An element is in F if it is not in the linked list B. This representation for B also allows us to loop over the elements in F in order of their indies, by looping over the elements of B and, for any two onseutive elements in B, say a < b, looping over all indies a +,..., b, whih we know must be in F. Given a point i, either in B or in F, this allows us to determine, in O() steps, the indies i and i. We an also add and remove points from B in O() steps while keeping the list ordered. If we use a binary minimum heap to keep trak of the minimum value of aross {γ (j) }, we pay a omputational ost of O(log n) eah time we update γ (j) or γ (j) for some j. Therefore, the ompleity of this algorithm is (( T ) ) O k r log n (38) r= } and {γ (j) where, we reall, T is the number of ritial points in the path {w (γ)}, and n is the number of omponents in the input y. In the partiular ase of FL, where α i = i, we have T n (See e.g. Theorem 2) and points only enter F, they never leave F. Therefore, we only need to use (9) for points in B and thus k r = 2 r. This leads us to O(n log n), just like in []. defined as the point in the onve polytope P(α) = {z R n : ỹ i i r= z r γ α i for all i [n + ]} that is losest to the origin. Seond we show that z is ontinuous in α. To establish the first step, just make the hange of variable z = w, z i+ = w i+ w i, i > in (5). To establish the seond step, we first make three observations. (Obs. ) If α α < δ, then for any z P(α ), there eists z P(α) suh that z z < ɛ (δ), where ɛ (δ) onverges to zero as δ onverges to zero. (Obs. 2) If α α < δ, then z (α ) < z (α) + ɛ 2 (δ), where ɛ 2 (δ) onverges to zero as δ onverges to zero. (Obs. 3) If z P(α) is suh that z < z + δ, then z z < ɛ 3 (δ), where ɛ 3 (δ) onverges to zero as δ onverges to zero. Obs. follows beause the faes of the polytope hange ontinuously with α. Obs. 2 follows from Obs., sine, if z P(α ) is the loset point to z (α), then, by Obs., z (α) > z ɛ (δ) z (α ) ɛ (δ). Obs. 3 is trivial when the origin is in the interior of P(α), so we fous on the other ase. Let d = sup z z be suh that z P, and z < z + δ, and let d = sup z z be suh that z H, and z < z +δ, where H is the half-plane defined by z z z 2. Sine P(α) is onve, P(α) H and thus d d < ( z + δ) 2 z 2 = δ 2 z + δ, where the last inequality follows by simple geometry. Therefore, z z < δ 2 z + δ, whih onverges to zero as δ onverges to zero. Now let α α < δ and let q P(α) be the loset point to z (α ). By Obs., z (α ) q < ɛ (δ) and thus q < z (α ) +ɛ (δ). By Obs. 2, q < z (α) +ɛ 2 (δ)+ ɛ (δ). By Obs. 3, q z (α) < ɛ 3 (ɛ 2 (δ) + ɛ (δ)). We an finally write, z (α ) z (α) = z (α ) q + q z (α) z (α ) q + q z (α) ɛ (δ) + ɛ 3 (ɛ 2 (δ) + ɛ (δ)). This proves ontinuity sine the right hand side onverges to zero as δ onverges to zero. VII. PROOF OF TECHNICAL LEMMA 6 AND LEMMA 7 Proof of Lemma 6. The proof follows by ontradition. Assume that there eists ɛ > 0 suh that, for all y with y < ɛ, we have y X. X ontains a ball of size ɛ/2, and thus has non-zero measure. Proof of Lemma 7. Sine (γα,..., γα n+ ) is ontinuous as a funtion of (γ, α), it follows that, to prove that is ontinuous as a funtion of (γ, α), we only need to prove that is ontinuous as a funtion of α. We now assume γ fied, and for simpliity write (γ) as or (α). The same goes for all the variables introdued below. The proof proeeds in two steps. First we show that an be obtained from a linear transformation of a point z