Sparse Optimization Lecture: Basic Sparse Optimization Models
|
|
- Eugenia Atkinson
- 6 years ago
- Views:
Transcription
1 Sparse Optimization Lecture: Basic Sparse Optimization Models Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know basic l 1, l 2,1, and nuclear-norm models some applications of these models how to reformulate them into standard conic programs which conic programming solvers to use 1 / 33
2 Eamples of Sparse Optimization Applications See online seminar at piazza.com 2 / 33
3 Basis pursuit min{ 1 : A = b} find least l 1-norm point on the affine plane { : A = b} tends to return a sparse point (sometimes, the sparsest) l 1 ball 3 / 33
4 Basis pursuit min{ 1 : A = b} find least l 1-norm point on the affine plane { : A = b} tends to return a sparse point (sometimes, the sparsest) l 1 ball touches the affine plane 4 / 33
5 Basis pursuit denoising, LASSO min { A b 2 : 1 τ}, (1a) min 1 + µ 2 A b 2 2, (1b) min { 1 : A b 2 σ}. (1c) all models allow A b 5 / 33
6 Basis pursuit denoising, LASSO min { A b 2 : 1 τ}, (2a) min 1 + µ 2 A b 2 2, (2b) min { 1 : A b 2 σ}. (2c) 2 is most common for error but can be generalized to loss function L (2a) seeks for a least-squares solution with bounded sparsity (2b) is known as LASSO (least absolute shrinkage and selection operator). it seeks for a balance between sparsity and fitting (2c) is referred to as BPDN (basis pursuit denoising), seeking for a sparse solution from tube-like set { : A b 2 σ} they are equivalent (see later slides) in terms of regression, they select a (sparse) set of features (i.e., columns of A) to linearly epress the observation b 6 / 33
7 Sparse under basis Ψ / l 1 -synthesis model min s { s 1 : AΨs = b} (3) signal is sparsely synthesized by atoms from Ψ, so vector s is sparse Ψ is referred to as the dictionary commonly used dictionaries include both analytic and trained ones analytic eamples: Id, DCT, wavelets, curvelets, gabor, etc., also their combinations; they have analytic properties, often easy to compute (for eample, multiplying a vector takes O(n log n) instead of O(n 2 )) Ψ can also be numerically learned from training data or partial signal they can be orthogonal, frame, or general 7 / 33
8 Sparse under basis Ψ / l 1 -synthesis model If Ψ is orthogonal, problem (3) is equivalent to by change of variable = Ψs, equivalently s = Ψ. Related models for noise and approimate sparsity: min { Ψ 1 : A = b} (4) min{ A b 2 : Ψ 1 τ}, min Ψ 1 + µ 2 A b 2 2, min{ Ψ 1 : A b 2 σ}. 8 / 33
9 Sparse after transform / l 1 -analysis model min { Ψ 1 : A = b} (5) Signal becomes sparse under the transform Ψ (may not be orthogonal) Eamples of Ψ: DCT, wavelets, curvelets, ridgelets,... tight frames, Gabor,... (weighted) total variation When Ψ is not orthogonal, the analysis is more difficult 9 / 33
10 Eample: sparsify an image 10 / 33
11 (a) DCT coefficients (b) Haar wavelets (c) Local variation sorted coefficient magnitudes 10 5 (d) DCT coeff s decay sorted coefficient magnitudes 10 5 (e) Haar wavelets sorted coefficient magnitudes 10 5 (f) Local variation Figure: the DCT and wavelet coefficients are scaled for better visibility. 11 / 33
12 Questions 1. Can we trust these models to return intended sparse solutions? 2. When will the solution be unique? 3. Will the solution be robust to noise in b? 4. Are constrained and unconstrained models equivalent? in what sense? Questions 1 4 will be addressed in net lecture. 5. How to choose parameters? τ (sparsity), µ (weight), and σ (noise level) have different meanings applications determine which one is easier to set generality: use a test data set, then scale parameters for real data cross validation: reserve a subset of data to test the solution 12 / 33
13 Joint/group sparsity Joint sparse recovery model: min{ X 2,1 : A(X) = b} (6) X where m X 2,1 := [ i1 i,2 in] 2. i=1 l 2-norm is applied to each row of X l 2,1-norm ball has sharp boundaries across different rows, which tend to be touched by {X : A(X) = b}, so the solution tends to be row-sparse also X p,q for 1 < p, affects magnitudes of entries on the same row comple-valued signals are a special case 13 / 33
14 Joint/group sparsity Decompose {1,..., n} = G 1 G 2 G S. non-overlapping groups: G i G j =, i j. otherwise, groups may overlap (modeling many interesting structures). Group-sparse recovery model: where min { G,2,1 : A = b} (7) G,2,1 = S w s Gs 2. s=1 14 / 33
15 Auiliary constraints Auiliary constraints introduce additional structures of the underlying signal into its recovery, which sometimes significantly improve recovery quality nonnegativity: 0 bound (bo) constraints: l u general inequalities: Q q They can be very effective in practice. They also generate corners. 15 / 33
16 Reduce to conic programs Sparse optimization often has nonsmooth objectives. Classic conic programming solvers do not handle nonsmooth functions. Basic idea: model nonsmoothness by inequality constraints. Eample: for given, we have Therefore, 1 = min 1, 2 {1 T ( 1 + 2) : 1 2 =, 1 0, 2 0}. (8) min{ 1 : A = b} reduces to a linear program (LP) min 1 + µ 2 A b 2 2 reduces to a bound constrained quadratic program (QP) min { A b 2 : 1 τ} reduces to a bound constrained QP min { 1 : A b 2 σ} reduces to a second-order cone program (SOCP) 16 / 33
17 Basic form: Conic programming min{c T : F + g K 0, A = b.} a K b stands for a b K, which is a conve, closed, pointed cone. Eamples: first orthant (cone): R n + = { R n : 0}. norm cone (2nd order cone): Q = {(, t) : t} polyhedral cone: P = { : A 0} positive semidefinite cone: S + = {X : X 0, X T = X} Eample: { (, y, z) : [ y ] } y S + z z y / 33
18 Model Linear program min{c T : A = b, K 0} where K is the nonnegative cone (first orthant). K 0 0. Algorithms the Simple method (move between vertices) interior-point methods (IPMs) (move inside the polyhedron) decomposition approaches (divide and conquer) In primal IPM, 0 is replaced by its logarithmic barrier: ψ(y) = i log(y i) log-barrier formulation: min{c T (1/t) i log( i) : A = b} 18 / 33
19 Second-order cone program Model min{c T : A = b, K 0} where K = K 1 K K; each K k is the second-order cone { } K k = y R n k : y nk y y2 n k 1. IPM is the standard solver (though other options also eist) Log-barrier of K k : ψ(y) = log ( y 2 n k (y y nk 1) ) 19 / 33
20 Semi-definite program Model min{c X : A(X) = b, X K 0} where K = K 1 K K; each K k = S n k +. IPM is the standard solver (though other options also eist) Log-barrier of S n k + (still a concave function): ψ(y) = log det(y). 20 / 33
21 (from Boyd & Vandenberghe, Conve Optimization) properties (without proof): for y K 0, ψ(y) K 0, y T ψ(y) = θ nonnegative orthant R n +: ψ(y) = n i=1 log y i ψ(y) = (1/y 1,..., 1/y n ), y T ψ(y) = n positive semidefinite cone S n +: ψ(y ) = log det Y ψ(y ) = Y 1, tr(y ψ(y )) = n second-order cone K = {y R n+1 (y yn) 2 1/2 y n+1 }: y 1 2 ψ(y) = yn+1 2 y2 1. y2 n y n, yt ψ(y) = 2 y n+1 21 / 33
22 (from Boyd & Vandenberghe, Conve Optimization) Central path for t > 0, define (t) as the solution of minimize subject to tf 0 () + φ() A = b (for now, assume (t) eists and is unique for each t > 0) central path is { (t) t > 0} eample: central path for an LP minimize c T subject to a T i b i, i = 1,..., 6 hyperplane c T = c T (t) is tangent to level curve of φ through (t) c (10) 22 / 33
23 Log-barrier formulation: min{tf 0() + φ() : A = b} Compleity of log-barrier interior-point method: log(( i k θi)/(εt(0) )) log µ 23 / 33
24 Primal-dual interior-point methods more efficient than barrier method when high accuracy is needed update primal and dual variables at each iteration; no distinction between inner and outer iterations often ehibit superlinear asymptotic convergence search directions can be interpreted as Newton directions for modified KKT conditions can start at infeasible points cost per iteration same as barrier method 24 / 33
25 Model l 1 minimization by interior-point method min { 1 : A b 2 σ} min min {1 T ( 1 + 1, 2) : 1 0, 2 0, 1 2 =, A b 2 σ} 2 min 1, 2 {1 T ( 1 + 2) : 1 0, 2 0, A( 1 2) b 2 σ} min {1 T T 2 : A 1 A 2 + y = b, z = σ, ( 1, 1, 2, z, y) K 0} 2 where ( 1, 2, z, y) K 0 means 1, 2 R n +, (t, y) Q m+1. Solver: Mosek, SDPT3, Gurobi. Also, modeling language CVX and YALMIP. 25 / 33
26 Nuclear-norm minimization by interior-point method If we can model as an SDP... (how? see net slide)... then, we can also model min X { X : A(X) b F σ} min X { A(X) b F : X τ} min X µ X A(X) b 2 F min{ X : A(X) = b} (9) X as well as problems involving tr(x) and spectral norm X. X α αi X / 33
27 Sparse calculus for l 1 inspect to get some ideas: y, z 0 and yz = 1 2 (y + z) yz. moreover, 1 2 (y + z) = yz = if y = z =. observe y, z 0 and yz [ y ] 0. z So, [ y ] z 0 = 1 (y + z). 2 we attain 1 (y + z) = if y = z =. 2 Therefore, given, we have = min M { 1 2 tr(m) : [ ] } 0 1 M =, M = M T, M / 33
28 Generalization to nuclear norm Consider X R m n (w.o.l.g., assume m n) and let s try imposing [ ] Y X 0 X T Z Diagonalize X = UΣV T, Σ = diag(σ 1,..., σ m), X = i σi. [U T, V T ] [ Y X T ] [ ] X U = U T YU + V T ZV U T XV V T X T U Z V = U T YU + V T ZV 2Σ 0. So, tr(uyu T + VZV T 2Σ) = tr(y) + tr(z) 2 X 0. To attain =, we can let Y = UΣU T and Z = VΣ n nv T. 28 / 33
29 Therefore, X = min Y,Z = min M { [ ] } 1 Y X (tr(y) + tr(z)) : 0 (10) 2 X T Z { [ ] } 1 2 tr(m) : 0 I M = X, M = M T, M 0. (11) 0 0 Eercise: epress the following problems as SDPs min X { X : A(X) = b} min X µ X + 1 A(X) b F 2 min L,S { L + λ S 1 : A(L + S) = b} 29 / 33
30 Practice of interior-point methods (IPMs) LP, SOCP, SDP are well known and have reliable (commercial, off-the-shelf) solvers Yet, the most reliable solvers cannot handle large-scale problems (e.g., images, video, manifold learning, distributed stuff,...) Eample: to recover a still image, there can be 10M variables and 1M constraints. Even worse, the constraint coefficients are dense. Result: Out of memory. Simple and active-set methods: matri containing A must be inverted or factorized to compute the net point (unless A is very sparse). IPMs approimately solve a Newton system and thus also factorize a matri involving A. Even large and dense matrices can be handled, for sparse optimization, one should take advantages of the solution sparsity. Some compressive sensing problems have A with structures friendly for operations like A and A T y. 30 / 33
31 Practice of interior-point methods (IPMs) The Simple, active-set, and IPMs have reliable solvers; good to be the benchmark. They have nice interfaces (including CVX and YALMIP, which save you time.) CVX and YALMIP are not solvers; they translate problems and then call solvers; see and They can return highly accurate solutions; some first-order algorithms (coming later in this course) do not always. There are other remedies; see net slide. 31 / 33
32 Papers of large-scale SDPs Low-rank factorizations: S. Burer and R. D. C. Monteiro, A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization, Math. Program., 95: , LMaFit, First-order methods for conic programming: Z. Wen, D. Goldfarb, and W. Yin. Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Program. Comput., 2(3-4): , Matri-free IPMs: K. Fountoulakis, J. Gondzio, P. Zhlobich. Matri-free interior point method for compressed sensing problems, / 33
33 Subgradient methods Sparse optimization is typically nonsmooth, so it is natural to consider subgradient methods. apply subgradient descent to, say, min 1 + µ A 2 b 2 2. apply projected subgradient descent to, say, min { 1 : A = b}. Good: subgradients are easy to derive, methods are simple to implement. Bad: convergence requires carefully chosen step sizes (classical ones require diminishing step sizes). Convergence rate is weak on paper (and in practice, too?) Further readings: / 33
12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationConvex Optimization and l 1 -minimization
Convex Optimization and l 1 -minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l
More informationConvex Optimization Problems. Prof. Daniel P. Palomar
Conve Optimization Problems Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST) MAFS6010R- Portfolio Optimization with R MSc in Financial Mathematics Fall 2018-19, HKUST,
More informationLecture 4: September 12
10-725/36-725: Conve Optimization Fall 2016 Lecture 4: September 12 Lecturer: Ryan Tibshirani Scribes: Jay Hennig, Yifeng Tao, Sriram Vasudevan Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 6. Suvrit Sra. (Conic optimization) 07 Feb, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 6 (Conic optimization) 07 Feb, 2013 Suvrit Sra Organizational Info Quiz coming up on 19th Feb. Project teams by 19th Feb Good if you can mix your research
More informationLecture 20: November 1st
10-725: Optimization Fall 2012 Lecture 20: November 1st Lecturer: Geoff Gordon Scribes: Xiaolong Shen, Alex Beutel Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not
More informationIntroduction to Alternating Direction Method of Multipliers
Introduction to Alternating Direction Method of Multipliers Yale Chang Machine Learning Group Meeting September 29, 2016 Yale Chang (Machine Learning Group Meeting) Introduction to Alternating Direction
More informationCourse Outline. FRTN10 Multivariable Control, Lecture 13. General idea for Lectures Lecture 13 Outline. Example 1 (Doyle Stein, 1979)
Course Outline FRTN Multivariable Control, Lecture Automatic Control LTH, 6 L-L Specifications, models and loop-shaping by hand L6-L8 Limitations on achievable performance L9-L Controller optimization:
More informationLecture 7: Convex Optimizations
Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1
More informationLecture 14: Optimality Conditions for Conic Problems
EE 227A: Conve Optimization and Applications March 6, 2012 Lecture 14: Optimality Conditions for Conic Problems Lecturer: Laurent El Ghaoui Reading assignment: 5.5 of BV. 14.1 Optimality for Conic Problems
More informationHomework 4. Convex Optimization /36-725
Homework 4 Convex Optimization 10-725/36-725 Due Friday November 4 at 5:30pm submitted to Christoph Dann in Gates 8013 (Remember to a submit separate writeup for each problem, with your name at the top)
More informationSparse Optimization Lecture: Dual Certificate in l 1 Minimization
Sparse Optimization Lecture: Dual Certificate in l 1 Minimization Instructor: Wotao Yin July 2013 Note scriber: Zheng Sun Those who complete this lecture will know what is a dual certificate for l 1 minimization
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications
ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March
More informationDuality revisited. Javier Peña Convex Optimization /36-725
Duality revisited Javier Peña Conve Optimization 10-725/36-725 1 Last time: barrier method Main idea: approimate the problem f() + I C () with the barrier problem f() + 1 t φ() tf() + φ() where t > 0 and
More informationSemidefinite Programming Basics and Applications
Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent
More informationComputational Optimization. Constrained Optimization Part 2
Computational Optimization Constrained Optimization Part Optimality Conditions Unconstrained Case X* is global min Conve f X* is local min SOSC f ( *) = SONC Easiest Problem Linear equality constraints
More informationConstrained optimization
Constrained optimization DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Compressed sensing Convex constrained
More informationIII. Applications in convex optimization
III. Applications in convex optimization nonsymmetric interior-point methods partial separability and decomposition partial separability first order methods interior-point methods Conic linear optimization
More informationLecture 7: Weak Duality
EE 227A: Conve Optimization and Applications February 7, 2012 Lecture 7: Weak Duality Lecturer: Laurent El Ghaoui 7.1 Lagrange Dual problem 7.1.1 Primal problem In this section, we consider a possibly
More informationProximal Methods for Optimization with Spasity-inducing Norms
Proximal Methods for Optimization with Spasity-inducing Norms Group Learning Presentation Xiaowei Zhou Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology
More informationProjection methods to solve SDP
Projection methods to solve SDP Franz Rendl http://www.math.uni-klu.ac.at Alpen-Adria-Universität Klagenfurt Austria F. Rendl, Oberwolfach Seminar, May 2010 p.1/32 Overview Augmented Primal-Dual Method
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko
More informationConvex Optimization Overview (cnt d)
Conve Optimization Overview (cnt d) Chuong B. Do November 29, 2009 During last week s section, we began our study of conve optimization, the study of mathematical optimization problems of the form, minimize
More informationLecture 16: October 22
0-725/36-725: Conve Optimization Fall 208 Lecturer: Ryan Tibshirani Lecture 6: October 22 Scribes: Nic Dalmasso, Alan Mishler, Benja LeRoy Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationLecture 23: November 19
10-725/36-725: Conve Optimization Fall 2018 Lecturer: Ryan Tibshirani Lecture 23: November 19 Scribes: Charvi Rastogi, George Stoica, Shuo Li Charvi Rastogi: 23.1-23.4.2, George Stoica: 23.4.3-23.8, Shuo
More informationThe Q-parametrization (Youla) Lecture 13: Synthesis by Convex Optimization. Lecture 13: Synthesis by Convex Optimization. Example: Spring-mass System
The Q-parametrization (Youla) Lecture 3: Synthesis by Convex Optimization controlled variables z Plant distubances w Example: Spring-mass system measurements y Controller control inputs u Idea for lecture
More informationSparsity and Compressed Sensing
Sparsity and Compressed Sensing Jalal Fadili Normandie Université-ENSICAEN, GREYC Mathematical coffees 2017 Recap: linear inverse problems Dictionary Sensing Sensing Sensing = y m 1 H m n y y 2 R m H A
More informationSparse linear models
Sparse linear models Optimization-Based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_spring16 Carlos Fernandez-Granda 2/22/2016 Introduction Linear transforms Frequency representation Short-time
More information10-725/36-725: Convex Optimization Spring Lecture 21: April 6
10-725/36-725: Conve Optimization Spring 2015 Lecturer: Ryan Tibshirani Lecture 21: April 6 Scribes: Chiqun Zhang, Hanqi Cheng, Waleed Ammar Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationFRTN10 Multivariable Control, Lecture 13. Course outline. The Q-parametrization (Youla) Example: Spring-mass System
FRTN Multivariable Control, Lecture 3 Anders Robertsson Automatic Control LTH, Lund University Course outline The Q-parametrization (Youla) L-L5 Purpose, models and loop-shaping by hand L6-L8 Limitations
More informationUses of duality. Geoff Gordon & Ryan Tibshirani Optimization /
Uses of duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Remember conjugate functions Given f : R n R, the function is called its conjugate f (y) = max x R n yt x f(x) Conjugates appear
More informationMath 273a: Optimization Subgradient Methods
Math 273a: Optimization Subgradient Methods Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com Nonsmooth convex function Recall: For ˉx R n, f(ˉx) := {g R
More informationSecond-order cone programming
Outline Second-order cone programming, PhD Lehigh University Department of Industrial and Systems Engineering February 10, 2009 Outline 1 Basic properties Spectral decomposition The cone of squares The
More informationGeometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as
Chapter 8 Geometric problems 8.1 Projection on a set The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as dist(x 0,C) = inf{ x 0 x x C}. The infimum here is always achieved.
More information15. Conic optimization
L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationFast ADMM for Sum of Squares Programs Using Partial Orthogonality
Fast ADMM for Sum of Squares Programs Using Partial Orthogonality Antonis Papachristodoulou Department of Engineering Science University of Oxford www.eng.ox.ac.uk/control/sysos antonis@eng.ox.ac.uk with
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationLecture: Convex Optimization Problems
1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt-2015-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/36 optimization
More informationLecture 14: Newton s Method
10-725/36-725: Conve Optimization Fall 2016 Lecturer: Javier Pena Lecture 14: Newton s ethod Scribes: Varun Joshi, Xuan Li Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationL. Vandenberghe EE236C (Spring 2016) 18. Symmetric cones. definition. spectral decomposition. quadratic representation. log-det barrier 18-1
L. Vandenberghe EE236C (Spring 2016) 18. Symmetric cones definition spectral decomposition quadratic representation log-det barrier 18-1 Introduction This lecture: theoretical properties of the following
More informationMath 273a: Optimization Lagrange Duality
Math 273a: Optimization Lagrange Duality Instructor: Wotao Yin Department of Mathematics, UCLA Winter 2015 online discussions on piazza.com Gradient descent / forward Euler assume function f is proper
More informationCS295: Convex Optimization. Xiaohui Xie Department of Computer Science University of California, Irvine
CS295: Convex Optimization Xiaohui Xie Department of Computer Science University of California, Irvine Course information Prerequisites: multivariate calculus and linear algebra Textbook: Convex Optimization
More information1 Computing with constraints
Notes for 2017-04-26 1 Computing with constraints Recall that our basic problem is minimize φ(x) s.t. x Ω where the feasible set Ω is defined by equality and inequality conditions Ω = {x R n : c i (x)
More informationMathematics of Data: From Theory to Computation
Mathematics of Data: From Theory to Computation Prof. Volkan Cevher volkan.cevher@epfl.ch Lecture 13: Disciplined convex optimization Laboratory for Information and Inference Systems (LIONS) École Polytechnique
More informationLecture Notes 9: Constrained Optimization
Optimization-based data analysis Fall 017 Lecture Notes 9: Constrained Optimization 1 Compressed sensing 1.1 Underdetermined linear inverse problems Linear inverse problems model measurements of the form
More informationAdvances in Convex Optimization: Theory, Algorithms, and Applications
Advances in Convex Optimization: Theory, Algorithms, and Applications Stephen Boyd Electrical Engineering Department Stanford University (joint work with Lieven Vandenberghe, UCLA) ISIT 02 ISIT 02 Lausanne
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationOptimization for Compressed Sensing
Optimization for Compressed Sensing Robert J. Vanderbei 2014 March 21 Dept. of Industrial & Systems Engineering University of Florida http://www.princeton.edu/ rvdb Lasso Regression The problem is to solve
More informationSubgradient Method. Ryan Tibshirani Convex Optimization
Subgradient Method Ryan Tibshirani Convex Optimization 10-725 Consider the problem Last last time: gradient descent min x f(x) for f convex and differentiable, dom(f) = R n. Gradient descent: choose initial
More informationInterior Point Methods: Second-Order Cone Programming and Semidefinite Programming
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationHomework 5. Convex Optimization /36-725
Homework 5 Convex Optimization 10-725/36-725 Due Tuesday November 22 at 5:30pm submitted to Christoph Dann in Gates 8013 (Remember to a submit separate writeup for each problem, with your name at the top)
More informationLecture 4: January 26
10-725/36-725: Conve Optimization Spring 2015 Lecturer: Javier Pena Lecture 4: January 26 Scribes: Vipul Singh, Shinjini Kundu, Chia-Yin Tsai Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationAsynchronous Algorithms for Conic Programs, including Optimal, Infeasible, and Unbounded Ones
Asynchronous Algorithms for Conic Programs, including Optimal, Infeasible, and Unbounded Ones Wotao Yin joint: Fei Feng, Robert Hannah, Yanli Liu, Ernest Ryu (UCLA, Math) DIMACS: Distributed Optimization,
More informationABSTRACT. Recovering Data with Group Sparsity by Alternating Direction Methods. Wei Deng
ABSTRACT Recovering Data with Group Sparsity by Alternating Direction Methods by Wei Deng Group sparsity reveals underlying sparsity patterns and contains rich structural information in data. Hence, exploiting
More informationCoordinate Update Algorithm Short Course Proximal Operators and Algorithms
Coordinate Update Algorithm Short Course Proximal Operators and Algorithms Instructor: Wotao Yin (UCLA Math) Summer 2016 1 / 36 Why proximal? Newton s method: for C 2 -smooth, unconstrained problems allow
More informationarxiv: v1 [math.oc] 16 Nov 2015
Conve programming with fast proimal and linear operators arxiv:1511.04815v1 [math.oc] 16 Nov 2015 Matt Wytock, Po-Wei Wang, J. Zico Kolter November 17, 2015 Abstract We present Epsilon, a system for general
More informationSparse Optimization Lecture: Dual Methods, Part I
Sparse Optimization Lecture: Dual Methods, Part I Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know dual (sub)gradient iteration augmented l 1 iteration
More informationLecture 24: August 28
10-725: Optimization Fall 2012 Lecture 24: August 28 Lecturer: Geoff Gordon/Ryan Tibshirani Scribes: Jiaji Zhou,Tinghui Zhou,Kawa Cheung Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationEE Applications of Convex Optimization in Signal Processing and Communications Dr. Andre Tkacenko, JPL Third Term
EE 150 - Applications of Convex Optimization in Signal Processing and Communications Dr. Andre Tkacenko JPL Third Term 2011-2012 Due on Thursday May 3 in class. Homework Set #4 1. (10 points) (Adapted
More information10725/36725 Optimization Homework 4
10725/36725 Optimization Homework 4 Due November 27, 2012 at beginning of class Instructions: There are four questions in this assignment. Please submit your homework as (up to) 4 separate sets of pages
More informationTutorial on Convex Optimization: Part II
Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation
More informationA CONIC DANTZIG-WOLFE DECOMPOSITION APPROACH FOR LARGE SCALE SEMIDEFINITE PROGRAMMING
A CONIC DANTZIG-WOLFE DECOMPOSITION APPROACH FOR LARGE SCALE SEMIDEFINITE PROGRAMMING Kartik Krishnan Advanced Optimization Laboratory McMaster University Joint work with Gema Plaza Martinez and Tamás
More informationLecture Notes 5: Multiresolution Analysis
Optimization-based data analysis Fall 2017 Lecture Notes 5: Multiresolution Analysis 1 Frames A frame is a generalization of an orthonormal basis. The inner products between the vectors in a frame and
More informationLecture: Examples of LP, SOCP and SDP
1/34 Lecture: Examples of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:
More information9. Interpretations, Lifting, SOS and Moments
9-1 Interpretations, Lifting, SOS and Moments P. Parrilo and S. Lall, CDC 2003 2003.12.07.04 9. Interpretations, Lifting, SOS and Moments Polynomial nonnegativity Sum of squares (SOS) decomposition Eample
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence
More informationLecture Note 5: Semidefinite Programming for Stability Analysis
ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State
More informationThe convex algebraic geometry of rank minimization
The convex algebraic geometry of rank minimization Pablo A. Parrilo Laboratory for Information and Decision Systems Massachusetts Institute of Technology International Symposium on Mathematical Programming
More informationSparsity Matters. Robert J. Vanderbei September 20. IDA: Center for Communications Research Princeton NJ.
Sparsity Matters Robert J. Vanderbei 2017 September 20 http://www.princeton.edu/ rvdb IDA: Center for Communications Research Princeton NJ The simplex method is 200 times faster... The simplex method is
More informationGauge optimization and duality
1 / 54 Gauge optimization and duality Junfeng Yang Department of Mathematics Nanjing University Joint with Shiqian Ma, CUHK September, 2015 2 / 54 Outline Introduction Duality Lagrange duality Fenchel
More informationMath 273a: Optimization Overview of First-Order Optimization Algorithms
Math 273a: Optimization Overview of First-Order Optimization Algorithms Wotao Yin Department of Mathematics, UCLA online discussions on piazza.com 1 / 9 Typical flow of numerical optimization Optimization
More information4. Convex optimization problems
Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization
More informationFast Algorithms for SDPs derived from the Kalman-Yakubovich-Popov Lemma
Fast Algorithms for SDPs derived from the Kalman-Yakubovich-Popov Lemma Venkataramanan (Ragu) Balakrishnan School of ECE, Purdue University 8 September 2003 European Union RTN Summer School on Multi-Agent
More information10-725/ Optimization Midterm Exam
10-725/36-725 Optimization Midterm Exam November 6, 2012 NAME: ANDREW ID: Instructions: This exam is 1hr 20mins long Except for a single two-sided sheet of notes, no other material or discussion is permitted
More informationAgenda. 1 Cone programming. 2 Convex cones. 3 Generalized inequalities. 4 Linear programming (LP) 5 Second-order cone programming (SOCP)
Agenda 1 Cone programming 2 Convex cones 3 Generalized inequalities 4 Linear programming (LP) 5 Second-order cone programming (SOCP) 6 Semidefinite programming (SDP) 7 Examples Optimization problem in
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 28. Suvrit Sra. (Algebra + Optimization) 02 May, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 28 (Algebra + Optimization) 02 May, 2013 Suvrit Sra Admin Poster presentation on 10th May mandatory HW, Midterm, Quiz to be reweighted Project final report
More informationResearch overview. Seminar September 4, Lehigh University Department of Industrial & Systems Engineering. Research overview.
Research overview Lehigh University Department of Industrial & Systems Engineering COR@L Seminar September 4, 2008 1 Duality without regularity condition Duality in non-exact arithmetic 2 interior point
More informationCompressive Sensing (CS)
Compressive Sensing (CS) Luminita Vese & Ming Yan lvese@math.ucla.edu yanm@math.ucla.edu Department of Mathematics University of California, Los Angeles The UCLA Advanced Neuroimaging Summer Program (2014)
More informationLecture: Algorithms for LP, SOCP and SDP
1/53 Lecture: Algorithms for LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:
More informationPrimal-Dual Geometry of Level Sets and their Explanatory Value of the Practical Performance of Interior-Point Methods for Conic Optimization
Primal-Dual Geometry of Level Sets and their Explanatory Value of the Practical Performance of Interior-Point Methods for Conic Optimization Robert M. Freund M.I.T. June, 2010 from papers in SIOPT, Mathematics
More informationMath 273a: Optimization Subgradients of convex functions
Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 42 Subgradients Assumptions
More informationLow Complexity Regularization 1 / 33
Low Complexity Regularization 1 / 33 Low-dimensional signal models Information level: pixels large wavelet coefficients (blue = 0) sparse signals low-rank matrices nonlinear models Sparse representations
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationMathematical Optimization Models and Applications
Mathematical Optimization Models and Applications Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 1, 2.1-2,
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationOptimization Algorithms for Compressed Sensing
Optimization Algorithms for Compressed Sensing Stephen Wright University of Wisconsin-Madison SIAM Gator Student Conference, Gainesville, March 2009 Stephen Wright (UW-Madison) Optimization and Compressed
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationLMI MODELLING 4. CONVEX LMI MODELLING. Didier HENRION. LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ. Universidad de Valladolid, SP March 2009
LMI MODELLING 4. CONVEX LMI MODELLING Didier HENRION LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ Universidad de Valladolid, SP March 2009 Minors A minor of a matrix F is the determinant of a submatrix
More informationModule 04 Optimization Problems KKT Conditions & Solvers
Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September
More informationDuality Uses and Correspondences. Ryan Tibshirani Convex Optimization
Duality Uses and Correspondences Ryan Tibshirani Conve Optimization 10-725 Recall that for the problem Last time: KKT conditions subject to f() h i () 0, i = 1,... m l j () = 0, j = 1,... r the KKT conditions
More informationLecture Notes on Support Vector Machine
Lecture Notes on Support Vector Machine Feng Li fli@sdu.edu.cn Shandong University, China 1 Hyperplane and Margin In a n-dimensional space, a hyper plane is defined by ω T x + b = 0 (1) where ω R n is
More informationPrimal/Dual Decomposition Methods
Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients
More informationLecture: Introduction to LP, SDP and SOCP
Lecture: Introduction to LP, SDP and SOCP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2015.html wenzw@pku.edu.cn Acknowledgement:
More information