Mathematics of Data: From Theory to Computation

Transcription

1 Mathematics of Data: From Theory to Computation Prof. Volkan Cevher Lecture 13: Disciplined convex optimization Laboratory for Information and Inference Systems (LIONS) École Polytechnique Fédérale de Lausanne (EPFL) EE-556 (Fall 2015)

2 License Information for Mathematics of Data Slides This work is released under a Creative Commons License with the following terms: Attribution The licensor permits others to copy, distribute, display, and perform the work. In return, licensees must give the original authors credit. Non-Commercial The licensor permits others to copy, distribute, display, and perform the work. In return, licensees may not use the work for commercial purposes unless they get the licensor s permission. Share Alike The licensor permits others to distribute derivative works only under a license identical to the one that governs the licensor s work. Full Text of the License Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 2/ 39

3 Recommended readings A. Ben-Tal and A. Nemirovski, Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications, A. Nemirovski, Introduction to linear optimization, F. Alizadeh and D. Goldfarb, Second-order cone programming, Math. Program., Ser. B, L. Vandenberghe and S. Boyd, Semidefinite programming, SIAM Rev., A. Nemirovski, Interior point polynomial time methods in convex programming, Mathematics of Data Prof. Volkan Cevher, Slide 3/ 39

4 Motivation Example (Convex Problem) min F (x) x R p s.t. g i (x) 0, i = 1,..., s A j x b j = 0, j = 1,..., t x X X is a set such that the set of solutions is a nonempty set g i (x) are convex for i = 0,..., s Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 4/ 39

5 Motivation Example (Convex Problem) min F (x) x R p s.t. g i (x) 0, i = 1,..., s A j x b j = 0, j = 1,..., t x X X is a set such that the set of solutions is a nonempty set g i (x) are convex for i = 0,..., s Approach 1 - Previous lectures Design special purpose software Increased convergence speed Non-reusable Hard to design Solid background needed Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 4/ 39

6 Motivation Example (Convex Problem) min F (x) x R p s.t. g i (x) 0, i = 1,..., s A j x b j = 0, j = 1,..., t x X X is a set such that the set of solutions is a nonempty set g i (x) are convex for i = 0,..., s Approach 1 - Previous lectures Design special purpose software Increased convergence speed Non-reusable Hard to design Solid background needed Approach 2 - This lecture Structured convex forms Less efficient per particular instance Readily available software Optimized solvers Minimal expertise required Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 4/ 39

7 Good news "One size fits all" LP QP QCQP SOCP SDP Good news: we need one solver! Today - Disciplined Convex Programming (DCP) 1. DCP Linear programming (LP) Quadratic programming (QP) Quadratically constrained quadratic programming (QCQP) Second order conic programming (SOCP) Semidefinite programming (SDP) 2. Methods Simplex method Interior point methods Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 5/ 39

8 Linear Programming (LP) A linear program (LP) is the problem of minimizing a linear function subject to finitely many linear equality and inequality constraints. Definition (LP in the canonical form) An LP in the canonical form is given by { } opt = min c T x : x R p, Ax b, x for some c R p, A R n p, and b R n. Any LP can be converted to an equivalent one in the canonical form. A linear equality constraint Bx = d is equivalent to two linear inequality constraints Bx d and Bx d, and can be written as [ ] [ ] B d x. B d Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 6/ 39

9 Application 1: Basis pursuit Example (Basis pursuit [4]) Recall the Gaussian linear model b = Ax + w, and assume that x R p is sparse. The basis pursuit estimator for x is given by for some A R n p and b R p. ˆx arg min x { x 1 : x R p, Ax = b }, We have used methods for constrained minimization in Lectures 11 and 12 to solve it. LP formulation The optimization problem is equivalent to { } min 1 T (x + x ) : x +, x 0, A(x + x ) = b, x +,x which is an LP, where 1 := (1, 1,..., 1) R p [4]. Another equivalent LP formulation is given by [11] { } min 1 T u : u 0, u x u, Ax = b, x,u where u denotes the contour of x. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 7/ 39

10 Application 2: Dantzig selector Example (Dantzig selector [3]) Recall the Gaussian linear model b = Ax + w, and assume that x R p is sparse. It is shown in [2] that the Dantzig selector defined as ˆx arg min x { x 1 : x R p, A T (b Ax) λ}, for some properly chosen λ > 0 behaves similarly to the Lasso, and hence can be used to estimate x. LP formulation The optimization problem is equivalent to { } min 1 T u : u 0, u x u, λ1 A T (b Ax) λ1, x,u which is an LP, where we used the contour trick as in the previous slide. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 8/ 39

11 Application 3: Maximum flow Example (Maximum flow) Let G = (V, E) be a directed graph, where V denotes the set of nodes, and E {(u, v) : u, v V} denotes the directed edges. Let s, t E. The maximum flow problem seeks to find the flow f u,v for all (u, v) V that maximizes the sum flow from s to t subject to capacity constraint: f u,v c u,v for all (u, v) E, where c u,v are given capacity constraints, and flow conservation: u:(u,v) E fu,v = fv,w for all v V \ {s, t}. w:(v,w) E LP formulation The maximum flow problem is equivalent to { max x s,v : x u,v 0 for all (u, v) E, x u,v v:(s,v) E } capacity constraint & flow conservation. Note that this is an LP. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 9/ 39

12 The dual problem of an LP Recall an LP in the canonical form is given by { } opt = min c T x : x R p, Ax b, x for some c R p, A R n p, and b R n. Definition (The dual problem) The corresponding dual problem is given by { } opt = min b T λ : λ R n, λ 0, A T λ = c. λ Intuition The primal problem is equivalent to maximizing c T x. Let λ R n satisfying λ 0 and A T λ = c. Then c T x = (A T λ) T x = λ T (Ax) λ T b. Therefore, the dual problem minimizes an upper bound of the original (primal) problem. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 10/ 39

13 LP duality theorem Theorem (Weak and strong LP duality) Consider an LP and the corresponding dual problem. Then Symmetry: The dual problem of the dual problem is equivalent to the primal problem. Weak duality: For any pair of feasible points (x, λ), we have where G is called the duality gap. G(x, λ) := b T λ ( c) T x 0, Strong duality: If the primal problem has a finite optimal value, so does the dual problem, and opt = opt. Application of weak duality If the optimal objective value of the primal problem is, then the dual problem is not feasible. If the optimal objective value of the dual problem is, then the primal problem is not feasible. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 11/ 39

14 Application of strong duality: the max-flow min-cut theorem Let G = (V, E) be a directed graph, where V denotes the set of nodes, and E {(u, v) : u, v V} denotes the directed edges. Let c u,v be given capacity constraints for each (u, v) E. Let s, t E. Example (Minimum cut) The minimum cut problem seeks to find a partition S, T of V that minimizes the cut capacity cu,v, subject to s S, and t T. (u,v) E:u S,v T The minimum cut capacity poses a bottleneck of the maximum flow from s to t. Theorem (Max-flow min-cut theorem [5]) The maximum sum flow from s to t equals the minimum cut capacity between s and t. Sketch of the proof [18]. The minimum cut problem is the dual of the maximum flow problem. Apply strong duality. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 12/ 39

15 Geometry of an LP Definition (Extreme point) A point x in a convex set X is an extreme point, if there does not exist α (0, 1) such that x = αu + (1 α)v for some u, v X. Theorem (Krein-Milman theorem [8]) A non-empty bounded convex set is the convex hull of the set of all its extreme points. Proposition If the feasible set of an LP does not contain a line, then one of the extreme points of the feasible set is a minimizer. Proof. By the Krein-Milman theorem, any point x in the feasible set can be written as m x = i=1 α m iv i, α i 0, i=1 α i = 1, where v 1,..., v m denote the extreme points. Then for any linear objective function f(x) := c T x for some vector c, we have f(x) = m i=1 α if(v i ) max i f(v i ). Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 13/ 39

16 Geometry of an LP contd. Definition (Polyhedron) A non-empty set X R p is polyhedral, or a polyhedron, if X = {x : x R p, Ax b}, for some A R n p and b R n. The feasible set of an LP of the canonical form is polyhedral. Proposition ([11]) A point x in a polyhedron X := {x : x R p, Ax b} is an extreme point, if and only if it is the unique solution of A I v = b I for some I {1,..., p}. Corollary For any polyhedron, the number of extreme points is finite. Proof. The number of systems of linear equations of the form A I v = b I is finite. Hence we only need to compare the function values on a finite number of points. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 14/ 39

17 Simplex methods Consider an LP of the canonical form. Assume its feasible set does not contain a line. Definition (Face and improving edge) A face of the feasible set is a subset of the feasible set, for which there exists a non-empty I {1,..., n} such that all of its elements satisfy A I x = b I. An improving edge is a one-dimensional face of the feasible set, along which the objective value decreases. Prototype of simplex methods A typical simplex method 1. v an extreme point of the feasible set 2. While there is an improving edge e involving v 3. Output v. v the othe end of e The rule of finding an improving edge in Step 2 is called a pivot rule. The complexity of simplex methods depends on the design of the pivot rule, which determines the number of iterations. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 15/ 39

18 Complexity of simplex methods A long-standing open problem Analyses imply the number of iterations for simplex methods cannot be polynomial in the worst case [1]. Empirical performance (on non-pathological cases) yields O(n). Is there a pivot rule for the simplex algorithm that yields a polynomial number of iterations? (See Problem 9 of Smale s Mathematical Problems for the Next Century [15]) Partial answers The smallest number of iterations can be upper-bounded by O(p log n ) [6]. Simplex methods can have polynomial expected number of iterations for random A, b, and c, while these results are not practical [16]. Smoothed analysis: For any LP of the canonical form, for which A and b are perturbed by a small random noise, a simplex method has polynomial expected number of iterations [16]. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 16/ 39

19 Quadratically constrained quadratic programming (QCQP) Definition (Quadratic program (QP)) A QP is an optimization problem of the form opt = min x { x T Px + 2q T x + r : x R p, Ax b }, for some A R n p, b R n, P R p p, q R p, and r R. A QP is a convex optimization problem if P 0. Definition (Quadratically Constrained Quadratic Program (QCQP)) A QCQP is an optimization problem of the form opt = min x { x T P 0 x + 2q T 0 x + r 0 : for some P i R p p, q i R p, and r i R. x R p, x T P i x + 2q T i x + r i 0 for all i = 1,..., m }, A QCQP is a convex optimization problem if P i 0 for all i. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 17/ 39

20 Application 1: Portfolio optimization Example (Markowitz portfolio optimization (Nobel Prize) [9]) Given a collection of n possible investments with return rates r 1,..., r n, modeled as RVs with mean E[r i ] = µ i and variance σ i = E[(r i µ i ) 2 ], the goal is to maximize the return of a portfolio represented by ratio of available capital invested x i in each of n them. The return of the portofolio is R = x i r i and E[R] = x T µ, i=1 E[(R E[R]) 2 ] = x T Gx, where G i,j = ρ i,j σ i σ j and ρ i,j is the corelation between investment return i and j. The convex optimization formulation of this problem is: where κ is a parameter for the "risk". min xt µ κx T Gx x R n n s.t. x i = 1 i=1 x 0, Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 18/ 39

21 Application 2: Sequential Quadratic Programming Definition (Sequential Quadratic Programming) To solve a given convex program min x D F (x) s.t. g i (x) 0, i = 1,..., s h j (x) = 0, j = 1,..., t we solve a series of QPs min x D F (x) + F (xk )(x x k ) (x xk ) T 2 F (x k )(x x k ) s.t. g i (x) + g i (x k )(x x k ) 0, i = 1,..., s h j (x) + h j (x k )(x x k ) = 0, j = 1,..., t Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 19/ 39

22 Second-order cone programming (SOCP) Definition (Second-order cone (Lorentz { cone)) } A second-order cone is a set of the form L = (x, t) : x R p, x 2 t R p+1. Definition (Partial ordering induced by a second-order cone) Let L be a second-order cone. The partial ordering induced by L is defined as x L y if and only if y x L p+1. Especially, (x, t) L 0 if and only if x 2 t. Definition (Second-order cone program (SOCP)) An SOCP is an optimization problem of the form { m } m opt = min c T i x i : x i R p, A i x i = b, x i L 0, x 1,...,x m i=1 i=1 for some A i R n p, b R n, and c i R p. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 20/ 39

23 Illustration of a second-order cone Definition (Second-order cone (Lorentz cone)) A second-order cone is a set of the form L p+1 = { (x, t) : x R p, x 2 t }. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 21/ 39

24 Application: Basis pursuit denoising Example (Basis pursuit denoising) Recall that the basis pursuit denoising estimator (Lecture?) is given by ˆx arg min x { x 1 : x R p, b Ax 2 σ }, for some A R n p, b R n, and σ > 0. We could use methods from Lectures 11 and 12 to solve it. SOCP formulation The optimization problem is equivalent to { 1 T (x + + x ) : y = b A(x + x ), z = σ, min x +,x,y,z (z, y) L 0, ((x + ) i, 0) L 0, ((x ) i, 0) L 0 for all i }, where (x + ) i and (x ) i denote the i-th element of x + and x, respectively. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 22/ 39

25 Semidefinite programming (SDP) Definition (Semidefinite program (SDP)) A semidefinite program is an optimization problem of the form { p } opt = min x c T x : x R p, F 0 + x i F i 0, i=1 for some c R p and symmetric matrices F 0,..., F p R m m. Reminder The eigenvalue decomposition of a square matrix, A R n n, is given by: A = XΛX 1 A 0 if all its eigenvalues are nonnegative i.e. λ min (A) 0. Similarly, A 0 if all its eigenvalues are nonnegative i.e. λ min (A) > 0. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 23/ 39

26 Examples: LP and maximum eigenvalue minimization Example (LP as SDP) The LP in the canonical form is equivalent to the SDP opt = min x opt = min x { c T x : x R p, Ax b } { c T x : x R p, diag(b Ax) 0 }. Example (Maximum eigenvalue minimization [17]) p Define A(x) := A 0 + i=1 x ia i for symmetric matrices A 0,..., A p. The problem of minimizing the maximum eigenvalue of A(x), is equivalent to the SDP opt = min x {λ max(a(x)) : x R p }, opt = min {t : t R, x R p, ti A(x) 0}. t,x Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 24/ 39

27 Schur complement Definition (Schur complement) Consider a symmetric matrix X R n n given by ( A B X = B T C for some symmetric matrix A R m m. If A is non-singular, then is called the Schur complement of A in X. ), S := C B T A 1 B Useful properties: det(x) = det(a)det(s) X 0 A 0 and S 0 If A 0, then X 0 S 0 Example: ( (Ax + b) T (Ax + b) c T x d 0 I (Ax + b) T Ax + b c T x + d ) 0 Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 25/ 39

28 QCQP as SDP Definition (Quadratically Constrained Quadratic Program (QCQP)) A QCQP is an optimization problem of the form opt = min x { x T P 0 x + 2q T 0 x + r 0 : for some P i R p p, q i R p, and r i R. SDP formulation x R p, x T P i x + 2q T i x + r i 0 for all i = 1,..., m }, Assume that P i 0, and hence can be decomposed as P i = M T i M i for all i. The QCQP is equivalent to the SDP given by { opt = min x,t [ t : x R p I M 0 x, t R, (M 0 x) T 2q T x r 0 + t [ ] } I M i x (M i x) T 2q T 0. x r i ] 0, Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 26/ 39

29 SOCP as SDP Definition (Second-order cone program (SOCP)) An SOCP is an optimization problem of the form { m } m opt = min c T i x i : x i R p, A i x i = b, x i L 0, x 1,...,x m i=1 i=1 for some A i R n p, b R n, and c i R p. SDP formulation Write each x i as x i = (v i, t i ) T for all i. The constraint x i L 0 is equivalent to [ ] ti I v i v T 0, i t i for all i. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 27/ 39

30 Applications: Matrix completion Example (Matrix completion) Let X R p p be an unknown low-rank matrix, ( and we want to estimate X given a linear operator A : R p p R n and b := A X ) R n. An estimator is given by { ˆX } arg min X : X R p p, A (X) = b. X We could use methods from Lectures 11 and 12 to solve it. SDP formulation [13] The optimization problem is equivalent to the SDP given by min X,Y,Z { 1 2 [Tr(Y) + Tr(Z)] : X, Y, Z Rp p, A (X) = b, [ Y X X T Z ] } 0. The proof in [13] uses the duality of SDP. We show another proof in the next slide. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 28/ 39

31 Applications: Matrix completion contd. Proposition For any matrix X R p p, we have X = min Y,Z { 1 2 [Tr(Y) + Tr(Z)] : Y, Z Rp p, [ Y X X T Z ] } 0. Proof. Consider the SVD of X, X = UΣV. If [ Y X X T Z ] 0, then we have [ U T V T ] [ ] [ Y X U X T Z V ] = U T YU + V T ZV 2Σ 0, and hence Tr(U T YU + V T ZV 2Σ) = Tr(Y) + Tr(Z) 2 X 0, and the minimum value of (1/2) [Tr(Y) + Tr(Z)] is X. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 29/ 39

32 From simplex methods to interior point methods (IPM) Observation Simplex methods scans the extreme points of the feasible set, and this is why the number of iterations can be sub-exponential in the problem dimensions. (Although empirical performance is much better.) Interior point method (IPM) [10, 12] The key idea of the Interior Point Methods (IPM) is, as the name suggests, to keep the iterates in the interior of the feasible set, and progress toward the optimum. Mathematics of Data Prof. Volkan Cevher, Slide 30/ 39

33 A brief history of IPM A brief history [10] N. Karmarkar proposed the first IPM for LP in 1984 [7]. This is the first algorithm for LP that has both theoretical polynomial time guarantee and good empirical performance. J. Renegar proposed the first path-following IPM for LP in 1986 [14]. This establishes the foundation of the current version of IPMs. Y. Nesterov extended the path-following idea to general constrained convex optimization problems in 1988 [12]. This is achieved by the notion of self-concordant barriers. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 31/ 39

34 The path-following IPM Let G be a closed bounded set in R p. Consider the convex program { } opt = min c T x : x G. x Definition (Barrier function) A barrier function of G is a smooth convex function F : int(g) R such that lim t F (x t) = for any sequence {x t : t N} converging to the boundary of G, and 2 F (x) 0 for all x int(g). Idea of a path-following IPM Consider a family of optimization problems: { x (t) = arg min tc T x + F (x) : x R p}, x where F is a barrier function of G. We call x (t) the path. For every t > 0, x (t) uniquely exists in G, and hence is always feasible. For any sequence t i, we have x (t i ) opt. For any sequence x i such that x i x (t i ) 0, we have x i opt. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 32/ 39

35 An example of the path-following IPM Consider an SDP given by opt = min x { c T x : x R p, F 0 + } p x i F i 0, i=1 for some c R p and symmetric matrices F 0,..., F p R m m. Choice of the barrier function ( ) p The function F (x) := ln det F 0 + i=1 x if i is a (self-concordant) barrier function of the feasible set. Obviously, F (x) < + if and only if x is in the feasible set of the SDP. Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 33/ 39

36 An example of the path-following IPM contd. Define F t(x) := tc T x + F (x), and λ(f t, x) := [ F t(x)] T [ 2 F (x)] 1 [ F t(x)]. It can be proved that x is close to x (t) if λ(f t, x) is small. Basic path-following scheme [10] Basic path-following scheme Let T N, and γ, κ > Set t 0, x 0 such that λ(t 0, x 0 ) κ. 2. For i = 1,..., T ( t i 1 + γ ) t i 1 m Find x i such that λ(f ti, x i ) κ. 3. Output x T. Theorem ([10]) The output of the basic path-following scheme satisfies c T x T opt = O(e T ). Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 34/ 39

37 Available solvers Warning Kids, do not try this at home! The following solvers have been designed by trained professionals. Solvers A list of solvers (commercial, academic free license and free/open source) categorized by application are available at Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 35/ 39

38 References I [1] Nina Amenta and Günter M. Ziegler. Deformed products and maximal shadows of polytopes. In Advances in Discrete and Computational Geometry, pages AMS, [2] Peter Bickel, Ya acov Ritov, and Alexandre B. Tsybakov. Simultaneous analysis of Lasso and Dantzig selector. Ann. Stat., 37(4): , [3] Emmanuel Candes and Terence Tao. The Dantzig selector: Statistical estimation when p is much larger than n. Ann. Stat., 35(6): , [4] Scott Shaobing Chen, David L. Donoho, and Michael A. Saunders. Atomic decomposition by basis pursuit. SIAM J. Sci. Comput., 20(1):33 61, [5] P. Elias, A. Feinstein, and C. E. Shannon. A note on the maximum flow through a network. IRE Trans. Inf. Theory, 2(4): , [6] Gil Kalai and Daniel J. Kleitman. A quasi-polynomial bound for the diameter of graphs of polyhedra. Bull. Amer. Math. Soc., 26(2): , Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 36/ 39

39 References II [7] N. Karmarkar. A new polynomial-time algorithm for linear programming. In Proc. 16th Annu. ACM Symp. Theory of Computing, pages , [8] M. Krein and D. Milman. On extreme points of regular convex sets. Stud. Math., 9(1): , [9] Harry Markowtiz. Portfolio selection. J. Finance, 7(1):77 91, [10] Arkadi Nemirovski. Interior point polynomial time methods in convex programming [11] Arkadi Nemirovski. Introduction to linear optimization [12] Yurii Nesterov and Arkadii Nemirovskii. Interior-Point Polynomial Algorithms in Convex Programming. SIAM, Philadelphia, PA, Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 37/ 39

40 References III [13] Benjamin Recht, Maryam Fazel, and Pablo A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev., 52(3): , [14] James Renegar. A polynomial-time algorithm, based on Newton s method, for linear programming. Math. Program., 40:59 93, [15] Steve Smale. Mathematical problems for the next century. Math. Intell., 20(2):7 15, [16] Daniel A. Spielman and Shang-Hua Teng. Smoothed analysis of algorithms: Why the simplex algorithm usually takes polynomial time. J. ACM, 51(3): , [17] Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. SIAM Rev., 38(1):49 95, Mathematics of Data Prof. Volkan Cevher, volkan.cevher@epfl.ch Slide 38/ 39

41 References IV [18] Vijay V. Vazirani. Approximation Algorithms, chapter 12, pages Springer, Berlin, Mathematics of Data Prof. Volkan Cevher, Slide 39/ 39