Statistical machine learning and convex optimization 06 Lecture February 5th Lecturer: Francis Bach Scribe: Guillaume Maillard, Nicolas Brosse This lecture deals with classical methods for convex optimization. For a convex function, a local minimum is a global minimum and the uniqueness is assured in the case of strict convexity. In the sequel, g is a convex function on R d. The aim is to find one (or the) minimum θ R d and the value of the function at this minimum g(θ ). Some key additional properties will be assumed for g : Lipschitz continuity θ R d, θ D g (θ) B. Smoothness (θ, θ ) ( R d), g (θ ) g (θ ) L θ θ. Strong convexity (θ, θ ) ( R d),g(θ ) g(θ )+g (θ ) (θ θ )+ µ θ θ. We refer to Lecture of this course for additional information on these properties. We point out key references: [], [].. Smooth optimization If g : R d R is a convex L-smooth function, we remind that for all θ,η R d : g (θ) g (η) L θ η. Besides, if g is twice differentiable, it is equivalent to: 0 g (θ) LI. Proposition. (Properties of smooth convex functions) Let g : R d R be a convex L-smooth function. Then, we have the following inequalities:. Quadratic upper-bound: 0 g(θ) g(η) g (η) (θ η) L θ η. Co-coercivity: L g (θ) g (η) [ g (θ) g (η) ] (θ η) 3. Lower bound: g(θ) g(η)+g (η) (θ η)+ L g (θ) g (η) 4. Distance to optimum: g(θ) g(θ ) g (θ) (θ θ ) 5. If g is also µ-strongly convex, then g(θ) g(η)+g (η) (θ η)+ µ g (θ) g (η) -
Lecture February 5th 06 6. If g is also µ-strongly convex, another distance to optimum: g(θ) g(θ ) µ g (θ) Proof () is a simple application of Taylor expansion with integral remainder. (3) We define: h(θ) = g(θ) θ g (η) which is convex with a global minimum at η. Then: h(η) h(θ L h (θ)) h(θ)+h (θ) ( L h (θ)))+ L L h (θ)) h(θ) L h (θ) Thus g(η) η g (η) g(θ) θ g (η) L g (θ) g (η) () Apply (3) twice for (η,θ) and (θ,η), and sum to get 0 [g (η) g (θ)] (θ η)+ L g (θ) g (η) (4) is immediate from the definition of convex function. (5) We define h(θ) = g(θ) θ g (η) which is convex with a global minimum at η. h(η) = minh(θ) minh(θ)+h (θ) (ζ θ)+ µ ζ θ θ ζ The min is attained for: ζ θ = µ h (θ) This leads to h(η) h(θ) µ h (θ) Hence, g(η) η g (η) g(θ) θ g (η) µ g (η) (θ) (6) Put η = θ, where θ is a global minimizer of g in (5). Note.. The proofs are simple with second-order derivatives. For (5) and (6), there is no need of smoothness assumption... Smooth gradient descent Proposition. (Smooth gradient descent) Let g be a L-smooth convex function, and θ be one (or the) minimum of g. For the following algorithm, we have the bounds:. if g is µ-strongly convex, θ t = θ t L g (θ t ) g(θ t ) g(θ ) ( µ/l) t[ g(θ 0 ) g(θ ) ] -
Lecture February 5th 06 Figure.. Level lines of convex functions. On the left (large µ), best case for gradient descent. On the right (small µ), the algorithm tends to do small steps and is thus slower.. if g is only L-smooth, g(θ t ) g(θ ) L θ 0 θ. t+4 Note.. (step-size) In that case, the step-size (/L) is constant. A line search is also possible (see, e.g., https:// en. wikipedia. org/ wiki/ Line_ search ), which consists of optimizing the step at each iteration in order to find the minimum of g along the direction g (θ t ). However, it can be time-consuming and should be carefully used (in particular, no need to be very precise in each line search). (robustness) For the same algorithm, we obtain different bounds depending on the assumptions that we make on g. It is important to notice that the algorithm is identical in all cases and adapts to the difficulty of the situation. (lower bounds) It is not the best possible convergence rates after O(d) iterations. Simple direct proof for quadratic convex functions In order to illustrate these results, we study the case of quadratic convex functions: g(θ) = θ Hθ c θ, where H is a symmetric positive semi-definite matrix. We denote by µ andlrespectively the smallest and the largest eigenvalues of H. The global optimum is easily derived: θ = H c (or H c where H is the pseudo-inverse of H). Let us compute explicitly the gradient descent iteration: θ t = θ t L (Hθ c) = θ t L (Hθ Hθ ) θ t θ = (I L H)(θ t θ ) = (I L H)t (θ 0 θ ) -3
Lecture February 5th 06 Case : We assume strong convexity of g. µ is therefore positive (µ > 0) and the eigenvalues of (I L H)t are in [0,( µ L )t ]. We obtain an exponential decrease towards 0 and we deduce convergence of iterates: θ t θ ( µ/l) t θ 0 θ. To deal with function values g(θ t ) g(θ ), we notice that: g(θ t ) g(θ ) = (θ θ ) H(θ θ ), using Taylor expansion at the order, or the fact θ = H c. We thus get: g(θ t ) g(θ ) ( µ/l) t[ g(θ 0 ) g(θ ) ]. Case : g is only L-smooth convex, that is, µ equals 0, and the eigenvalues of (I L H)t are in [0,]. We do not have convergence of iterates: θ t θ θ 0 θ, but we still have: g(θ t ) g(θ ) = (θ θ ) H(θ θ ), from which we deduce: g(θ t ) g(θ ) max v [0,L] v( v/l)t θ 0 θ by a decomposition along the eigenspaces. For e [0,L] and t N, we have: v( v/l) t L [ ] tv t L exp( tv L ) C L t, and we get (up to a constant): g(θ t ) g(θ ) L t θ 0 θ. Proof () One iteration of the algorithm consists of: θ t = θ t γg (θ t ) with γ = /L. Let us compute the function value g(θ t ): g(θ t ) = g [ θ t γg (θ t ) ] g(θ t )+g (θ t ) [ γg (θ t ) ] + L γg (θ t ) = g(θ t ) γ( γl/) g (θ t ) = g(θ t ) L g (θ t ) if γ = /L, g(θ t ) µ L[ g(θt ) g(θ ) ] using strongly-convex distance to optimum Thus, g(θ t ) g(θ ) ( µ/l) t[ g(θ 0 ) g(θ ) ] -4
Lecture February 5th 06 We may also get [][p.70, thm..5]: θ t θ ( γµl ) t θ 0 θ µ+l as soon as γ µ+l () One iteration of the algorithm consists of: θ t = θ t γg (θ t ) with γ = /L. First, we bound the iterates: θ t θ = θ t θ γg (θ t ) = θ t θ +γ g (θ t ) γ(θ t θ ) g (θ t ) θ t θ +γ g (θ t ) γ L g (θ t ) using co-coercivity = θ t θ γ(/l γ) g (θ t ) θ t θ if γ /L θ 0 θ Using one equality in the proof of (), we get: g(θ t ) g(θ t ) L g (θ t ) By definition of convexity and Cauchy-Schwarz, we have: g(θ t ) g(θ ) g (θ t ) (θ t θ ) g (θ t ) θ t θ And finally, putting things together: g(θ t ) g(θ ) g(θ t ) g(θ ) L θ 0 θ [ g(θt ) g(θ ) ] We then have our Lyapunov function, a function decreasing when the number of iterates increases. Let denote by α the quantity: and by k : We then have: α = L θ 0 θ, k = g(θ k ) g(θ ). k k α k with 0 k = g(θ k ) g(θ ) L θ k θ α k by dividing by k k k k k α because ( k ) is non-increasing k k αt by summing from k = to t 0 t 0 t by inverting +αt 0 L θ 0 θ t+4 since 0 L θ 0 θ and α = -5 L θ 0 θ
Lecture February 5th 06.. Accelerated gradient methods A natural question arising from the results of this proposition is : can we do better? Let us state a lower bound using first-order methods. Definition.3 (First-order method) An algorithm using first-order method is any iterative algorithm that selects θ t in θ 0 +span(f (θ 0 ),...,f (θ t )) Theorem.4 (Lower-bound gradient descent) For every integer k (d )/ and every θ 0, there exist convex L-smooth functions with a global minimizer θ such that for any first-order method, g(θ t ) g(θ ) 3 L θ 0 θ 3 (t+) Note..3 k (d )/ is a strong assumption. It means that the number of iterations should not be higher than half the dimension. O(/t) rate for gradient method may not be optimal! Proof [sketch] We refer to [][p.58, section..] for the complete proof. We let the reader check the following facts as an exercise: Define the quadratic function Fact : g t is L-smooth g t (θ) = L t (θ 8[ ) + (θ i θ i+ ) +(θ t ) θ ] Fact : minimizer supported by first t coordinates (closed form) i= Fact 3: any first-order method starting from zero will be supported in the first k coordinates after iteration k Fact 4: the minimum over this support in {,...,k} may be computed in closed form Given iteration k, take g = g k+ and compute lower-bound on g(θ k) g(θ ) θ 0 θ. We now turn to methods susceptible to reach the lower bound stated in theorem.4. They are called accelerated gradient methods [3]. -6
Lecture February 5th 06 Theorem.5 (Accelerated gradient descent) Let g be a convex function with L-Lipschitzcont. gradient, and minimum attained at θ. For the following algorithm, we have the bound: θ t = η t L g (η t ) η t = θ t + t t+ (θ t θ t ) g(θ t ) g(θ ) L θ 0 θ (t+) Note..4 For a ten-line proof [4] This is not improvable (i.e., the scaling in t cannot be less than O(/t ) over all similar problems). Theorem.6 (Accelerated gradient descent) Let g be a µ-strongly convex function with L-Lipschitz-cont. gradient and minimum attained at θ. For the following algorithm, θ t = η t L g (η t ) η t = θ t + µ/l + µ/l (θ t θ t ) we have the bound : g(θ t ) f(θ ) L θ 0 θ ( µ/l) t Note..5 Ten-line proof [4] Not improvable Relationship with conjugate gradient for quadratic functions: a similar bound holds. We need to know µ and L for implementation of the algorithm! Often difficult...3 Optimization for sparsity-inducing norms For more information on the subject, see [5]. We use gradient descent as a proximal method, a way of approximating non-smooth functions by differentiable ones. Let us introduce the following minimisation problem: θ t+ = argmin θ R d f(θ t )+(θ θ t ) f(θ t )+ L θ θ t Since it is a quadratic function in θ, the problem is equivalent to: θ t+ = θ t L f(θ t) -7
Lecture February 5th 06 More generally, we want to tackle problems of the form: min θ R d f(θ)+µω(θ) where Ω(θ) is here a sparsity-inducing norm. We seek to minimise the number of non-zero terms in θ. The problem is then formulated as: θ t+ = argmin θ R d f(θ t )+(θ θ t ) f(θ t )+µω(θ)+ L θ θ t If Ω(θ) = θ Thresholded gradient descent In that case, convergence rates are similar than smooth optimization. For acceleration methods, look at [6, 7]. In the following, we illustrate the method by focusing on the softthresholding for the l -norm with an example in dimension. Example: quadratic problem in D, i.e. min x R x xy +λ x It is a piecewise quadratic function with a kink at zero, with derivatives at0+: g + = λ y and at 0 : g = λ y. Figure.. Representation of the quadratic function in D (x = 0 and x 0) x = 0 is the solution iff g + 0 and g 0 (i.e., y λ) x 0 is the solution iff g + 0 (i.e., y λ) x = y λ x 0 is the solution iff g 0 (i.e., y λ) x = y +λ In conclusion, the solution is x = sign(y)( y λ) + = soft thresholding -8
Lecture February 5th 06 x x*(y) λ λ y..4 Newton method Figure.3. Representation of soft thresholding in D Given θ t, we seek to minimize the second-order Taylor expansion : g(θ) = g(θ t )+g (θ t ) (θ θ t )+ (θ θ t ) g (θ t ) (θ θ t ) It is a quadratic function in θ and we easily derive the minimum: θ t = θ t g (θ t ) g (θ t ) The iteration is expensive, and the running-time complexity O(d 3 ) in general. We have a property of quadratic convergence: if θ t θ is small enough, for some constant C, we have (C θ t θ ) = (C θ t θ ) See for example [8].. Non-smooth optimization.. Counter-example: steepest descent for nonsmooth objectives There is a real difference between the class of smooth problems (with bounded smoothness constant) and the class of all convex problems: methods that converged with a controlled rate in the first setting may now even fail to be consistent. For example, consider the following function: { 5(9θ g(θ,θ ) = +6θ ) θ > θ (9θ +6 θ ) θ θ which is shown in Figure.4. For any value of the stepsize, the fixed stepsize (sub)gradient descent scheme gets stuck at zero, which is clearly not a minimum! New algorithms are needed. -9
Lecture February 5th 06 5 0 5 5 0 5 Figure.4. Level lines of g. Color indicates function value... Subgradient method New setting: - Convex continuous function g - Bounded domain C = {θ : θ D} although as the proof shows, any compact convex domain with diameter D will yield the same result. - g reaches its minimum on C at θ - g is Lipschitz-continuous on C with constant B Note.. : In fact, any globally defined, real-valued convex function is locally Lipschitz, so we are dealing with the general convex case. Moreover, an upper bound for the Lipschitz constant (for a given value of R) may be calculated with a finite number of function evaluations. The basic idea is still that of gradient descent, with three important changes: - Obviously, we will have to replace the gradient with (any possible) subgradient. - The stepsize will have to decrease with the number of iterates. - Since even then, iterates may fail to converge, the algorithm does not return the last iterate, but computes an average instead. Subgradient method: algorithm Let Π D be the orthogonal projection on C. Then the subgradient algorithm, when run for T iterations, is defined in Algorithm. For an easier proof that shows where the value for the stepsize comes from, we also consider a slightly different algorithm, in which the stepsize is constant over a single run of the algorithm (Algorithm ). This is known as the finite-horizon setting. -0
Lecture February 5th 06 Algorithm minimize g over C = {θ : θ D} Require: ε > 0, B 0 and D 0, θ 0 C Ensure: g(ˆθ) g(θ ) ε T 8D B ε for t = to T do ) D θ[t] Π C (θ[t ] B t g (θ[t ]) end for ˆθ T T t= θ[t] return ˆθ Algorithm minimize g over C = {θ : θ D} Require: ε > 0, B 0 and D 0, θ 0 C Ensure: g(ˆθ c ) g(θ ) ε T 4D B ε for t = to T do ( θ[t] Π C θ[t ] D ) B T g (θ[t ]) end for ˆθ T T t= θ[t] return ˆθ -
Lecture February 5th 06 The advantage of the first algorithm over the simpler version is that it can be run in an online fashion (often referred to as an anytime algorithm), i.e. without knowing in advance the number of iterations or the required precision. Theorem.7 g(ˆθ) g(θ ) DB T Proof Π C is -Lipschitz since it is a projection. Therefore for any θ Let γ t denote the stepsize. Then Π C (θ) θ = Π C (θ) Π C (θ ) (θ C) θ θ θ t θ θ t γ t g (θ t ) θ (by the preceding inequality) (.) θ t θ γ t(θ t θ ) g (θ t )+γt g (θ t (.) Thus we get: θ t θ +B γ t γ t(θ t θ ) g (θ t )( g (θ) B) (.3) θ t θ +B γ t γ t [g(θ t ) g(θ )](property of subgradients) (.4) g(θ t ) g(θ ) B γ t + γ t [ θ t θ θ t θ ]. We then average over t T Here there is a difference between the two versions of the algorithm: - If γ t = γ = constant then we get a telescoping sum: T T t=0 g(θ t ) g(θ ) B γ + γt [ θ 0 θ θ T θ ] B γ + γt θ 0 θ B γ + D γt. We can then optimize the upper bound in the parameter γ: γ = D suggests the right dependency for the online version. B T gives DB T which -
Lecture February 5th 06 - In the variable γ case we use summation by parts: T t= [g(θ t ) g(θ )] B B B = B T γ t + t= T t= T T γ t + t= t= γ t [ θ t θ θ t θ ] θ t θ [ γ t+ γ t ]+ θ 0 θ γ θ T θ γ T T T γ t + 4D [ ]+ 4D γ t+ γ t γ t= T t= t= γ t + 4D γ T. Now we set γ t = xd B t T t= to get: [g(θ t ) g(θ )] B xd B = xbd xbd T t= T t= T 0 + 4D B T t xd t + BD T x dt+ BD T t x xbd T + BD T x = (x+ x )BD T. (because d dt t = t ) which is optimized when the two terms are equal, that is x =, which gives: T T t= g(θ t ) g(θ ) BD T. Finally, the convexity inequality: ( ) T g θ t T T t=0 T g(θ t ) t=0 gives the desired result in both cases. Subgradient descent with strong convexity When the non-smooth convex function is strongly convex with known parameter µ, it is possible to improve the rate of convergence -3
Lecture February 5th 06 Algorithm 3 minimize g over {x : x D} Require: µ > 0, D 0, θ 0 : θ 0 D Ensure: ˆθ t = t (t+)(t+) 0 (u + )θ u where a subscript i indicates value of the variable at iteration i. θ θ 0 ˆθ θ 0 for t do ) θ Π D (θ µ(t+) g (θ) ˆθ t+ˆθ+ t θ t+ end for return ˆθ to O( ). The algorithm and the proof of its convergence properties are similar in spirit to µt the previous one. Since here the algorithm may be used in cases where B is unknown we present it as an iterative scheme (Algorithm 3). Theorem.8 Let g be B-Lipschitz and µ-strongly convex over the set {θ : θ D}. Then for ˆθ defined by the above algorithm, g(ˆθ t ) g(θ ) B µ(t+). Note.. : Somewhat surprisingly, D does not appear explicitly in the upper bound. This is because the conditions that g be B-Lipschitz and µ-strongly convex already impose a limit on the size of g s domain of definition. Proof The proof uses the same ideas as for the general case: the beginning is the same. At (.4), the assumption of strong-convexity permits a better bound: which leads to: θ t θ θ t θ +B γ t γ t[g(θ t ) g(θ )]+ µ θ t θ, γ t [g(θ t ) g(θ )] ( γ t µ) θ t θ +B γ t θ t θ g(θ t ) g(θ ) ( γ t µ) θ t θ + B γ t γ t θ t θ B µ(t+) + µ [t ] θ t θ µ [t+ ] θ t θ (here γ t = µ(t+) ). -4
Lecture February 5th 06 Here as before the idea is to average the iterates, taking advantage of convexity on the left hand side and of telescoping terms on the right: g(ˆθ t ) g(θ ) (t+)(t+) (t+)(t+) (t+)(t+) B µ(t+). t+ u[g(θ u ) g(θ )] u= { } B t+ u µ u+ + µ t+ [u(u ) θ u θ 4 (u+)u θ u θ ] u= u= { B (t+) + µ } µ 4 [0 (t+)(t+) θ t+ θ ]..3 Ellipsoid method Idea The basic idea of the ellipsoid method is that of dichotomy: by recursively splitting a set along hyperplanes given by subgradients (so that all minimizers are one side of the hyperplane), it is hoped that the diameter of the set might decrease to zero exponentially fast, as in the one dimensional dichotomy method. To get a good rate of convergence, a natural idea is to split the set at its geometric center. However, calculating the geometric center of a set is in general a difficult problem. This is where knowing few geometric properties of ellipsoids is useful: Definition.9 Ellipsoid: Given a point x R d and a symmetric positive definite matrix P, the ellipsoid E( x,p) is the set: {x R d (x x) P (x x) } The point x is the geometric center of E( x,p) and the matrix P describes the half-axes lengths (eigenvalues) and directions (eigenvectors). Proposition.0 Given an ellispoid E( x,p) and a hyperplane H = {g (x x) 0} containing x, there is a minimum volume ellipsoid E + ( x +,P + ) containing {x E( x,p) g (x x) 0}, with x + = x Pg (.5) d+ g,p P + = d d (P Pgg P ) (.6) d+ g,p -5
Lecture February 5th 06 where g,p is the intrinsic P-norm of g, g Pg. Its volume decreases by a constant factor relative to the original ellipsoid: vol d (E + ( x +,P + )) e d vold (E(x,P)) E 0 E E E Figure.5. cuts and minimal ellipsoids This suggests the iterative scheme described in Algorithm 4. Algorithm 4 minimize g over {x : x D} Require: D 0, x 0 : x 0 D x[0] x 0 P D I d h 0 for t do h g ( x[t ]) x[t] x[t ] Ph P d d (P d+ d+ h,p Phh P) h,p end for t argmin 0 t stoptime g( x[t]) return x t Proposition.0 is not quite enough to conclude to a convergence rate, because even if the volume decreases exponentially, it seems hard to ensure that the diameter does the same, as the ellipsoid may be very excentric. However, exponential decrease remains true for approximate minimization. The following proof is taken from Nesterov s book []. Definition. The following quantity will be useful: v g (x) = g (x) (g (x)) (x x ) -6
Lecture February 5th 06 Lemma. If g is B-Lipschitz, then g(x) g(x ) Bv g (x) More explicitly, g(x) g(x ) B g (x) (g (x)) (x x ) Proof Let y = x +v g (x) g (x) g (x) Clearly y x v g (x) By convexity, g(y) g(x)+(g (x)) (y x) But by definition, In the end: (g (x)) (x y) = (g (x)) (x x ) v g (x) g (x) = 0 g(x) g(x ) g(y) g(x ) B y x Bv g (x) The main point of introducing this new quantity is that it can be bounded by the volume: Proposition.3 Let Q be a convex set in R d with finite diameter D and nonempty interior. Let x,...,x n R d be points and (E i ) i=,..,n be sets such that Assume that x E 0 = Q. Then: {x Q i {,...,n},(g (x i )) (x i x) 0} E i min v g(x i ) D i n [ ] vold E d i vol d Q Proof we introduce some notation: let v n = min i nv g (x i ) and α = v n D. ( α)x +αq ( α)x +αb(x,d) = B(x,v n) By convexity, we also have that By definition of v n, ( α)x +αq = [( α)x +αq] Q B(x,v n ) Q x x v n i {,...,n}, g (x i (g (x i )) (x i x) = g (x i (g (x i )) (x i x )+ v n x x 0 x E i g (x i (g (x i )) (x x) -7
Lecture February 5th 06 Therefore: α d vol d (Q) vol d (( α)x +αq) vol d (E i ). Combining all three inequalities, we obtain the convergence rate of the ellipsoid method: Theorem.4 Let g be convex and B-Lipschitz on B(0,D). Let x n be the value returned by Algorithm 4 when stopped at iteration n. Then g( x n ) g(x ) BDe n d. -8
Lecture February 5th 06..4 Application to Machine Learning After having described many different methods, we now review their applicability to machine learning. Setting As seen in Lecture, in machine learning, more specifically in empirical risk minimization with linear predictors, the objective function is of the form: ˆf(θ) = n n l(y i,φ(x i ) θ) i= Assuming bounded features: Φ(x) R and that the loss is G-Lipschitz on {θ θ RD}, we have shown that ˆf is GR-Lipschitz on {θ θ D}. Which methods can be used? Reasonable hypothesis in this setting - Smoothness: ˆf may or may not be smooth, depending on l (notably the hinge loss is unsmooth, but the ridge loss is smooth). If smooth, the constant is essentially the same as for l, times R, by linearity of differenciation. - Strong convexity: In typical high-dimensional settings, ˆf will rarely be strongly convex for a useful value of µ (in particular it cannot be if n < d). - Dimensional complexity: High-dimensionality makes the ellipsoid method useless, despite its geometric convergence. This is because the rate is O(e n d ) so O(d ) iterates are needed. Each iterate requires multiplying a vector by a matrix, so O(d ) cost is encountered at each iteration, for a total, prohibitive dependency of O(d 4 ) in the dimension. By contrast, gradient and subgradient methods converge at a rate independent of the dimension. The only dimension-dependent cost is due to arithmetic operations, and is linear in d. Thus, a gradient or subgradient method should be used, if possible one that does not need µ. -9
Lecture February 5th 06 The case of subgradient descent : choosing the number of steps In the worst, non-smooth case, the convergence rate is O( t ). This is not a problem since we must remember that there is a statistical error: It is f(θ) = E[ˆf(θ)] and not ˆf(θ) that we wish to minimize and we know from the previous lesson that with probability greater than δ: min θ Θ ] ˆf(θ) minf(θ) max ˆf(θ) f(θ) GRD [+ log( δ ) θ Θ θ Θ n Therefore, with t = n iterations of subgradient descent we get an optimization error of same order as the statistical error: ˆf(ˆθ) min η Θ ˆf(η) GRD n. There is not much to gain by iterating for longer (only a factor of at best). The overall complexity is O(n d): it depends only linearly on the dimension. -0
Bibliography [] Y. Nesterov. Introductory lectures on convex optimization: a basic course. Kluwer Academic Publishers, 004. [] S. Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends R in Machine Learning, 8(3-4):3 357, 05. URL http://dx.doi.org/0. 56/00000050. [3] Y. Nesterov. A method for solving a convex programming problem with rate of convergence O(/k ). Soviet Math. Doklady, 69(3):543 547, 983. [4] M. Schmidt, N. Le Roux, and F. Bach. Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization. Advances in Neural Information Processing Systems (NIPS), 0. [5] Francis Bach, Rodolphe Jenatton, Julien Mairal, and Guillaume Obozinski. Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning, 4(): 06, 0. [6] Y. Nesterov. Gradient methods for minimizing composite objective function. Center for Operations Research and Econometrics (CORE), Catholic University of Louvain, Tech. Rep, 76, 007. [7] A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences, ():83 0, 009. [8] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 003.