Sparse Optimization Lecture: Dual Certificate in l 1 Minimization
|
|
- Georgiana Randall
- 5 years ago
- Views:
Transcription
1 Sparse Optimization Lecture: Dual Certificate in l 1 Minimization Instructor: Wotao Yin July 2013 Note scriber: Zheng Sun Those who complete this lecture will know what is a dual certificate for l 1 minimization a strictly complementary dual certificate guarantees exact recovery it also guarantees stable recovery 1 / 24
2 What is covered A review of dual certificate in the context of conic programming A condition that guarantees recovering a set of sparse vectors (whose entries have the same signs), not for all k-sparse vectors The condition depends on sign(x o ), but not x o itself or b The condition is sufficient and necessary It also guarantees robust recovery against measurement errors The condition can be numerically verified (in polynomial time) The underlying techniques are Lagrange duality, strict complementarity, and LP strong duality. Results in this lecture are drawn from various papers. For references, see: H. Zhang, M. Yan, and W. Yin, One condition for all: solution uniqueness and robustness of l 1-synthesis and l 1-analysis minimizations 2 / 24
3 Lagrange dual for conic programs Let K i be a first-orthant, second-order, or semi-definite cone. It is self-dual. (Suppose a, b K i. Then, a T b 0. If a T b = 0, either a = 0 or b = 0.) Primal: min c T x s.t. Ax = b, x i K i i. Lagrangian relaxation: L(x; s) = c T x + s T (Ax b) Dual function: d(s) = min{l(x; s) : x i K i i} = b T s ι {(A T s+c) x i K i i} Dual problem: min s d(s) min s b T s s.t. (A T s + c) i K i i One problem might be simpler to solve than the other; solving one might help solve the other. 3 / 24
4 Dual certificate Given that x is primal feasible, i.e., obeying Ax = b, x i K i i. Question: is x optimal? Answer: One does not need to compare x to all other feasible x. A dual vector y will certify the optimality of x. 4 / 24
5 Dual certificate Theorem Suppose x is feasible (i.e., Ax = b, x i K i i). If s obeys 1. vanished duality gap: b T s = c T x, and 2. dual feasibility: (A T s + c) i K i, then x is primal optimal. Pick any primal feasible x (i.e., Ax = b, x i K i i), we have and thus due to Ax = b, (c + A T s ) T x = i (c + A T s ) T i x i 0 }{{}}{{} K i K i c T x = (c + A T s ) T x (A T s ) T x (A T s ) T x = b T s = c T x. Therefore, x is optimal. Corollary: (c + A T s ) T x = 0 and (c + A T s ) T i x i = 0, i. Bottom line: dual vector y = A T s certifies the optimality of x. 5 / 24
6 Dual certificate A related claim: Theorem If any primal feasible x and dual feasible s have no duality gap, then x is primal optimal and s is dual optimal. Reason: the primal objective value of any primal feasible x the dual objective value of any dual feasible s. Therefore, assuming both primal and dual feasibilities, a pair of primal/dual objectives must be optimal. 6 / 24
7 Complementarity and strict complementarity From and we get (c + A T s ) T i x i = (c + A T s ) T x = c T x + b T s = 0 i (c + A T s ) T i }{{} K i xi }{{} K i 0, i. (c + A T s ) T i x i = 0, i. Hence, at least one of (c + A T s ) T i and x i is 0 (but they can be both zero.) If exactly one of (c + A T s ) T i and x i is zero (the other is nonzero), then they are strictly complementary. Certifying the uniqueness of x requires a strictly complementary s. 7 / 24
8 l 1 duality and dual certificate Primal: Dual: min x 1 s.t. Ax = b (1) max b T s s.t. A T s 1 Given a feasible x, if s obeys 1. A T s 1, and 2. x 1 b T s = 0, then y = A T s certifies the optimality of x. Any primal optimal x must satisfy x 1 b T s = 0. 8 / 24
9 l 1 duality and complementarity a 1 = ab b. If ab = b, then 1. a < 1 b = 0 2. a = 1 b 0 3. a = 1 b 0 From A T s 1, we get x 1 = b T s = (A T s ) T x x 1 and (A T s ) i x i = x i, i. Therefore, 1. if (A T s ) i < 1, then x i = 0 2. if (A T s ) i = 1, then x i 0 3. if (A T s ) i = 1, then x i 0 Strict complementarity holds if for each i, 1 (A T s ) i or x i is zero but not both. 9 / 24
10 Uniqueness of x Suppose x is a solution to the basis pursuit model. Question: Is it the unique solution? Define I := supp(x ) = {i : x i 0} and J = I c. If s is a dual certificate and (A T s ) J < 1, x J = 0 for all optimal x. For i I, (A T s ) i = ±1 cannot determine x? i = 0 for optimal x. It is possible that (A T s ) i = ±1 yet x i = 0 (this is called degenerate.) On the other hand, if A Ix I = b has a unique solution, denoted by x I, then since x J = 0 is unique, x = [x I; x J] = [x I; 0] is the unique solution to the basis pursuit model. A Ix I = b has a unique solution provided that A I has independent columns, or equivalently, ker(a I) = {0}. 10 / 24
11 Optimality and uniqueness Condition For a given x, the index sets I = supp( x) and J = I c satisfy 1. ker(a I) = {0} 2. there exists y such that y R(A T ), y I = sign( x I), and y J < 1. Comments: part 1 guarantees unique x I as the solution to A Ix I = b part 2 guarantees x J = 0 y R(A T ) means y = A T s for some s the condition involves I and sign( x I), not the values of x I or b; but different I and sign( x I) require a different condition RIP guarantees the condition hold for all small I and arbitrary signs the condition is easy to verify 11 / 24
12 Optimality and uniqueness Theorem Suppose x obeys A x = b and the above Condition, then x is the unique solution to min{ x 1 : Ax = b}. In fact, the converse is also true, namely, the Condition is also necessary. 12 / 24
13 Uniqueness of x Part 1 ker(a I) = {0} is necessary. Lemma If 0 h ker(a I), then all x α = x + α[h; 0] for small α is optimal. Proof. x α is feasible since Ax α = Ax = b. We know x α 1 x 1, but for small α around 0, we also have x α 1 = x I + αh 1 = (A T s ) T I (x I + αh) = x 1 + α(a T s ) T I h. Hence, (A T s ) T I h = 0 and thus x α 1 = x 1. So, x α is also optimal. 13 / 24
14 Necessity Is part 2 necessary? Introduce min y y J s.t. y R(A T ), y I = sign( x I). (2) If the optimal objective value < 1, then there exists y obeying part 2, so part 2 is also necessary. We shall translate (2) and rewrite y R(A T ). 14 / 24
15 Necessity Define a = [sign( x I); 0] and basis Q of Null(A). If a R(A T ), set y = a. done. Otherwise, let y = a + z. Then y R(A T ) Q T y = 0 Q T z = Q T a y I = sign( x I) = a I z I = 0 a J = 0 y J = z J Equivalent problem: min z z J s.t. Q T z = Q T a, z I = 0. (3) If the optimal objective value < 1, then part 2 is necessary. 15 / 24
16 Necessity Theorem (LP strong duality) If a linear program has a finite solution, its Lagrange dual has a finite solution. The two solutions achieve the same primal and dual optimal objective. Problem (3) is feasible and has a finite objective value. The dual of (3) is max p (QT a) T p s.t. (Qp) J 1 1. If its optimal objective value < 1, then part 2 is necessary. 16 / 24
17 Lemma Necessity If x is unique, then the optimal objective of the following primal-dual problems is strictly less than 1. min z J s.t. Q T z = Q T a, z I = 0. z max p (QT a) T p s.t. (Qp) J 1 1. Proof. Define a = sign(x ). Uniqueness of x = for h ker(a) \ {0}, we have x 1 < x + h 1 = a T I h I < h J 1 Therefore, if p = 0, then z J = (Q T a) T p = 0. if p 0, then h := Qp ker(a) \ {0} obeys z J = (Q T a) T p = a T I h I < h J 1 (Qp) J 1 1. In both cases, the optimal objective value < / 24
18 Theorem Suppose x obeys A x = b. Then, x is the unique solution to min{ x 1 : Ax = b} if and only if the Condition holds. Comments: the uniqueness requires strong duality result for problems involving z J strong duality does not hold for all convex programs strong duality does hold for convex polyhedral functions f(z J), as well as those with constraint qualifications (e.g., the Slater condition) indeed, the theorem generalizes to analysis l 1 minimization: Ψ T x 1 does it generalize to x Gi 2 or X? the key is strong duality for 2 and also, the theorem generalizes to the noisy l 1 models (next part...) 18 / 24
19 Noisy measurements Suppose b is contaminated by noise: b = Ax + w Appropriate models to recover a sparse x include min λ x Ax b 2 2 (4) min x 1 s.t. Ax b 2 δ (5) Theorem Suppose x is a solution to either (4) or (5). Then, x is the unique solution if and only if the Condition holds for x. Key intuition: reduce (4) to (1) with a specific b. Let ˆx be any solution to (4) and b := Aˆx. All solutions to (4) are solutions to min x 1 s.t. Ax = b. The same applies to (5). Recall that the Condition does not involve b. 19 / 24
20 Stable recovery Assumptions: x and y satisfy the Condition. x is the original signal. b = A x + w, where w 2 δ x is the solution to min x 1 s.t. Ax b 2 δ. Goal: obtain a bound x x 2 Cδ. Constant C shall be independent of δ. 20 / 24
21 Stable recovery Lemma Define I = supp( x) and J = I c. x x 1 C 3δ + C 4 x J 1, where C 3 = 2 I r(i) and C 4 = A I r(i) + 1. Proof. x x 1 = x I x I 1 + x J 1 x I x I 1 I x I x I 2 I r(i) A I(x I x I) 2, where r(i) := sup supp(u)=i,u 0 (r(i) is related to one side of the RIP bound) introduce ˆx = [x I; 0]. u Au A I(x I x I) 2 = A(ˆx x) 2 A(ˆx x ) 2 + A(x x) 2 }{{} 2δ A(ˆx x ) 2 A ˆx x 2 A ˆx x 1 = A x J 1 21 / 24
22 Stable recovery Recall in the Condition, y I = sign( x) and y J < 1 x I 1 y I, x I x J 1 (1 y J ) 1 ( x J 1 y J, x ) Therefore, x J 1 (1 y J ) 1 ( x 1 y, x ) = (1 y J ) 1 d y(x, x), where d y(x, x) = x 1 x 1 y, x x is the Bregman distance induced by 1. Recall in the Condition, y R(A T ) so y = A T β for some vector β. d y(x, x) 2 β 2δ. Lemma Under the above assumptions, x J 1 2(1 y J ) 1 β 2δ. 22 / 24
23 Stable recovery Theorem Assumptions: x and y satisfy the Condition. x is the original signal. y = A T β. b = A x + w, where w 2 δ x is the solution to min x 1 s.t. Ax b 2 δ. Conclusion: where x x 1 Cδ, C = 2 I r(i) + 2 β 2( A I r(i) + 1) 1 y J Comment: a similar bound can be obtained for min λ x Ax b 2 2 with a condition on λ. 23 / 24
24 Generalization All the previous results (exact and stable recovery) generalize to the following models: min Ψ T x 1 s.t. Ax = b min λ Ψ T x Ax b 2 2 min Ψ T x 1 s.t. Ax b 2 δ Assume that A and Ψ each has independent rows, the update conditions are Condition For a given x, the index sets I = supp(ψ T x) and J = I c satisfy 1. ker(ψ T J ) ker(a I) = {0} 2. there exists y such that Ψy R(A T ), y I = sign(ψ T I x), and y J < / 24
I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationOne condition for all: solution uniqueness and robustness of l 1 -synthesis and l 1 -analysis minimizations
One condition for all: solution uniqueness and robustness of l 1 -synthesis and l 1 -analysis minimizations Hui Zhang Ming Yan Wotao Yin June 9, 2013 The l 1-synthesis and l 1-analysis models recover structured
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationConvex Optimization and SVM
Convex Optimization and SVM Problem 0. Cf lecture notes pages 12 to 18. Problem 1. (i) A slab is an intersection of two half spaces, hence convex. (ii) A wedge is an intersection of two half spaces, hence
More informationECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis
ECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis Lecture 3: Sparse signal recovery: A RIPless analysis of l 1 minimization Yuejie Chi The Ohio State University Page 1 Outline
More informationConic Linear Optimization and its Dual. yyye
Conic Linear Optimization and Appl. MS&E314 Lecture Note #04 1 Conic Linear Optimization and its Dual Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.
More informationLinear and Combinatorial Optimization
Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality
More informationNecessary and Sufficient Conditions of Solution Uniqueness in 1-Norm Minimization
Noname manuscript No. (will be inserted by the editor) Necessary and Sufficient Conditions of Solution Uniqueness in 1-Norm Minimization Hui Zhang Wotao Yin Lizhi Cheng Received: / Accepted: Abstract This
More informationLecture Note 5: Semidefinite Programming for Stability Analysis
ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State
More informationNecessary and sufficient conditions of solution uniqueness in l 1 minimization
1 Necessary and sufficient conditions of solution uniqueness in l 1 minimization Hui Zhang, Wotao Yin, and Lizhi Cheng arxiv:1209.0652v2 [cs.it] 18 Sep 2012 Abstract This paper shows that the solutions
More informationLecture: Introduction to Compressed Sensing Sparse Recovery Guarantees
Lecture: Introduction to Compressed Sensing Sparse Recovery Guarantees http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html Acknowledgement: this slides is based on Prof. Emmanuel Candes and Prof. Wotao Yin
More informationU.C. Berkeley CS294: Beyond Worst-Case Analysis Handout 12 Luca Trevisan October 3, 2017
U.C. Berkeley CS94: Beyond Worst-Case Analysis Handout 1 Luca Trevisan October 3, 017 Scribed by Maxim Rabinovich Lecture 1 In which we begin to prove that the SDP relaxation exactly recovers communities
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationLP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra
LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality
More informationConstrained optimization
Constrained optimization DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Compressed sensing Convex constrained
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationSparse Optimization Lecture: Dual Methods, Part I
Sparse Optimization Lecture: Dual Methods, Part I Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know dual (sub)gradient iteration augmented l 1 iteration
More informationSummer School: Semidefinite Optimization
Summer School: Semidefinite Optimization Christine Bachoc Université Bordeaux I, IMB Research Training Group Experimental and Constructive Algebra Haus Karrenberg, Sept. 3 - Sept. 7, 2012 Duality Theory
More informationLagrangian Duality and Convex Optimization
Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?
More information15. Conic optimization
L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone
More informationKarush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725
Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =
More information6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC
6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC 2003 2003.09.02.10 6. The Positivstellensatz Basic semialgebraic sets Semialgebraic sets Tarski-Seidenberg and quantifier elimination Feasibility
More informationLagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationLECTURE 10 LECTURE OUTLINE
LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationLecture 14: Optimality Conditions for Conic Problems
EE 227A: Conve Optimization and Applications March 6, 2012 Lecture 14: Optimality Conditions for Conic Problems Lecturer: Laurent El Ghaoui Reading assignment: 5.5 of BV. 14.1 Optimality for Conic Problems
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More informationIntroduction to Compressed Sensing
Introduction to Compressed Sensing Alejandro Parada, Gonzalo Arce University of Delaware August 25, 2016 Motivation: Classical Sampling 1 Motivation: Classical Sampling Issues Some applications Radar Spectral
More informationOn the local stability of semidefinite relaxations
On the local stability of semidefinite relaxations Diego Cifuentes Department of Mathematics Massachusetts Institute of Technology Joint work with Sameer Agarwal (Google), Pablo Parrilo (MIT), Rekha Thomas
More informationMath 273a: Optimization Subgradients of convex functions
Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 42 Subgradients Assumptions
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationSolving Dual Problems
Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
More informationSparsity and Compressed Sensing
Sparsity and Compressed Sensing Jalal Fadili Normandie Université-ENSICAEN, GREYC Mathematical coffees 2017 Recap: linear inverse problems Dictionary Sensing Sensing Sensing = y m 1 H m n y y 2 R m H A
More informationLecture 5. Theorems of Alternatives and Self-Dual Embedding
IE 8534 1 Lecture 5. Theorems of Alternatives and Self-Dual Embedding IE 8534 2 A system of linear equations may not have a solution. It is well known that either Ax = c has a solution, or A T y = 0, c
More informationLecture 7: Convex Optimizations
Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1
More informationDuality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
More informationIntroduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationAgenda. 1 Duality for LP. 2 Theorem of alternatives. 3 Conic Duality. 4 Dual cones. 5 Geometric view of cone programs. 6 Conic duality theorem
Agenda 1 Duality for LP 2 Theorem of alternatives 3 Conic Duality 4 Dual cones 5 Geometric view of cone programs 6 Conic duality theorem 7 Examples Lower bounds on LPs By eliminating variables (if needed)
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationPractice Exam 1: Continuous Optimisation
Practice Exam : Continuous Optimisation. Let f : R m R be a convex function and let A R m n, b R m be given. Show that the function g(x) := f(ax + b) is a convex function of x on R n. Suppose that f is
More informationEE364a Review Session 5
EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary
More information1 Review Session. 1.1 Lecture 2
1 Review Session Note: The following lists give an overview of the material that was covered in the lectures and sections. Your TF will go through these lists. If anything is unclear or you have questions
More informationThe fundamental theorem of linear programming
The fundamental theorem of linear programming Michael Tehranchi June 8, 2017 This note supplements the lecture notes of Optimisation The statement of the fundamental theorem of linear programming and the
More informationLecture 8 Plus properties, merit functions and gap functions. September 28, 2008
Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plus-properties and F-uniqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FP-I book:
More informationLeast Sparsity of p-norm based Optimization Problems with p > 1
Least Sparsity of p-norm based Optimization Problems with p > Jinglai Shen and Seyedahmad Mousavi Original version: July, 07; Revision: February, 08 Abstract Motivated by l p -optimization arising from
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationCompressed Sensing and Sparse Recovery
ELE 538B: Sparsity, Structure and Inference Compressed Sensing and Sparse Recovery Yuxin Chen Princeton University, Spring 217 Outline Restricted isometry property (RIP) A RIPless theory Compressed sensing
More informationSCRIBERS: SOROOSH SHAFIEEZADEH-ABADEH, MICHAËL DEFFERRARD
EE-731: ADVANCED TOPICS IN DATA SCIENCES LABORATORY FOR INFORMATION AND INFERENCE SYSTEMS SPRING 2016 INSTRUCTOR: VOLKAN CEVHER SCRIBERS: SOROOSH SHAFIEEZADEH-ABADEH, MICHAËL DEFFERRARD STRUCTURED SPARSITY
More informationConvex Optimization. Lecture 12 - Equality Constrained Optimization. Instructor: Yuanzhang Xiao. Fall University of Hawaii at Manoa
Convex Optimization Lecture 12 - Equality Constrained Optimization Instructor: Yuanzhang Xiao University of Hawaii at Manoa Fall 2017 1 / 19 Today s Lecture 1 Basic Concepts 2 for Equality Constrained
More informationSTAT 200C: High-dimensional Statistics
STAT 200C: High-dimensional Statistics Arash A. Amini May 30, 2018 1 / 57 Table of Contents 1 Sparse linear models Basis Pursuit and restricted null space property Sufficient conditions for RNS 2 / 57
More informationAsteroide Santana, Santanu S. Dey. December 4, School of Industrial and Systems Engineering, Georgia Institute of Technology
for Some for Asteroide Santana, Santanu S. Dey School of Industrial Systems Engineering, Georgia Institute of Technology December 4, 2016 1 / 38 1 1.1 Conic integer programs for Conic integer programs
More informationRecent Developments in Compressed Sensing
Recent Developments in Compressed Sensing M. Vidyasagar Distinguished Professor, IIT Hyderabad m.vidyasagar@iith.ac.in, www.iith.ac.in/ m vidyasagar/ ISL Seminar, Stanford University, 19 April 2018 Outline
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationAdditional Homework Problems
Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.
More informationMidterm Review. Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.
Midterm Review Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye (LY, Chapter 1-4, Appendices) 1 Separating hyperplane
More informationSparse Optimization Lecture: Basic Sparse Optimization Models
Sparse Optimization Lecture: Basic Sparse Optimization Models Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know basic l 1, l 2,1, and nuclear-norm
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationLecture: Introduction to LP, SDP and SOCP
Lecture: Introduction to LP, SDP and SOCP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2015.html wenzw@pku.edu.cn Acknowledgement:
More informationChapter 1. Preliminaries
Introduction This dissertation is a reading of chapter 4 in part I of the book : Integer and Combinatorial Optimization by George L. Nemhauser & Laurence A. Wolsey. The chapter elaborates links between
More informationThe Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem:
HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationOptimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers
Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex
More informationLecture 16: October 22
0-725/36-725: Conve Optimization Fall 208 Lecturer: Ryan Tibshirani Lecture 6: October 22 Scribes: Nic Dalmasso, Alan Mishler, Benja LeRoy Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationA Review of Linear Programming
A Review of Linear Programming Instructor: Farid Alizadeh IEOR 4600y Spring 2001 February 14, 2001 1 Overview In this note we review the basic properties of linear programming including the primal simplex
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More information4. Algebra and Duality
4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone
More informationLecture 10: Linear programming duality and sensitivity 0-0
Lecture 10: Linear programming duality and sensitivity 0-0 The canonical primal dual pair 1 A R m n, b R m, and c R n maximize z = c T x (1) subject to Ax b, x 0 n and minimize w = b T y (2) subject to
More informationExample: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma
4-1 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid
More informationThe Karush-Kuhn-Tucker (KKT) conditions
The Karush-Kuhn-Tucker (KKT) conditions In this section, we will give a set of sufficient (and at most times necessary) conditions for a x to be the solution of a given convex optimization problem. These
More informationFinite Dimensional Optimization Part III: Convex Optimization 1
John Nachbar Washington University March 21, 2017 Finite Dimensional Optimization Part III: Convex Optimization 1 1 Saddle points and KKT. These notes cover another important approach to optimization,
More informationComputational Optimization. Constrained Optimization Part 2
Computational Optimization Constrained Optimization Part Optimality Conditions Unconstrained Case X* is global min Conve f X* is local min SOSC f ( *) = SONC Easiest Problem Linear equality constraints
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationMaking Flippy Floppy
Making Flippy Floppy James V. Burke UW Mathematics jvburke@uw.edu Aleksandr Y. Aravkin IBM, T.J.Watson Research sasha.aravkin@gmail.com Michael P. Friedlander UBC Computer Science mpf@cs.ubc.ca Vietnam
More information10701 Recitation 5 Duality and SVM. Ahmed Hefny
10701 Recitation 5 Duality and SVM Ahmed Hefny Outline Langrangian and Duality The Lagrangian Duality Eamples Support Vector Machines Primal Formulation Dual Formulation Soft Margin and Hinge Loss Lagrangian
More informationGlobal Optimization of Polynomials
Semidefinite Programming Lecture 9 OR 637 Spring 2008 April 9, 2008 Scribe: Dennis Leventhal Global Optimization of Polynomials Recall we were considering the problem min z R n p(z) where p(z) is a degree
More information1 Regression with High Dimensional Data
6.883 Learning with Combinatorial Structure ote for Lecture 11 Instructor: Prof. Stefanie Jegelka Scribe: Xuhong Zhang 1 Regression with High Dimensional Data Consider the following regression problem:
More informationMachine Learning. Lecture 6: Support Vector Machine. Feng Li.
Machine Learning Lecture 6: Support Vector Machine Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2018 Warm Up 2 / 80 Warm Up (Contd.)
More informationDuality of LPs and Applications
Lecture 6 Duality of LPs and Applications Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will
More informationWeek 3 Linear programming duality
Week 3 Linear programming duality This week we cover the fascinating topic of linear programming duality. We will learn that every minimization program has associated a maximization program that has the
More informationLecture Notes 9: Constrained Optimization
Optimization-based data analysis Fall 017 Lecture Notes 9: Constrained Optimization 1 Compressed sensing 1.1 Underdetermined linear inverse problems Linear inverse problems model measurements of the form
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming
E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program
More informationAdvanced Mathematical Programming IE417. Lecture 24. Dr. Ted Ralphs
Advanced Mathematical Programming IE417 Lecture 24 Dr. Ted Ralphs IE417 Lecture 24 1 Reading for This Lecture Sections 11.2-11.2 IE417 Lecture 24 2 The Linear Complementarity Problem Given M R p p and
More informationExample Problem. Linear Program (standard form) CSCI5654 (Linear Programming, Fall 2013) Lecture-7. Duality
CSCI5654 (Linear Programming, Fall 013) Lecture-7 Duality Lecture 7 Slide# 1 Lecture 7 Slide# Linear Program (standard form) Example Problem maximize c 1 x 1 + + c n x n s.t. a j1 x 1 + + a jn x n b j
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationDuality in Linear Programming
Duality in Linear Programming Gary D. Knott Civilized Software Inc. 1219 Heritage Park Circle Silver Spring MD 296 phone:31-962-3711 email:knott@civilized.com URL:www.civilized.com May 1, 213.1 Duality
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationConvex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationTrust Region Problems with Linear Inequality Constraints: Exact SDP Relaxation, Global Optimality and Robust Optimization
Trust Region Problems with Linear Inequality Constraints: Exact SDP Relaxation, Global Optimality and Robust Optimization V. Jeyakumar and G. Y. Li Revised Version: September 11, 2013 Abstract The trust-region
More information