1. Introduction I course logistics I mathematical optimization I least-squares and linear programming I convex optimization I example I course goals and topics I nonlinear optimization I brief history of convex optimization IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 1 IOE 611/Math 663: Nonlinear Programming I Lectures: Tue and Thu, 12:00 1:20 PM, 1005 Dow I Instructors: Marina A. Epelman and Yiling Zhang (GSI); O ce hours: as announced on Canvas, and by appointment I Textbook: Convex Optimization by Stephen Boyd and Lieven Vandenberghe I Unrestricted resources (excluding assignments, exams, solutions, and grades) and calendar are at www-personal.umich.edu/~mepelman/teaching/ioe611/ Everything else, plus announcements, on Canvas I Required background: Working knowledge of linear algebra and analysis (review Appendix A of the book). Also, a level of mathematical maturity and willingness to program in Matlab. Exposure to numerical computing, optimization, and its application fields is helpful but not required. Please read syllabus for complete course policies IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 2
Course Logistics I Lectures: I Slides posted prior to lectures I In class, combination of slides (bring a copy, preferably one you can write on) and board writing (take notes!, photographing the board is strongly discouraged) I Problem sets weekly or so, include some computer modeling and programming I Not graded, discussed in weekly review sessions (times TBD via online survey) I Crucial in preparation for... I Four takehome exams (20%, 20%, 30%, 30%) I Written solutions must be typed, L A TEX highly recommended I Computer code, when appropriate, uploaded to Canvas I For each exam, choose 60 consecutive hours within pre-announced week-long period IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 3 Mathematical optimization (mathematical) optimization problem minimize f 0 (x) subject to f i (x) apple b i, i =1,...,m I x =(x 1,...,x n ): optimization variables (or decision variables) I f 0 : R n! R: objective function I f i : R n! R, i =1,...,m: constraint functions optimal solution x? has smallest value of f 0 among all vectors that are feasible, i.e., satisfy the constraints IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 4
Examples portfolio optimization: looking for a best way to invest in a set of n assets I variables: amounts invested in di erent assets I constraints: budget, max./min. investment per asset, minimum return I objective: risk device sizing in electronic circuits: choosing the width and length of devices in a circuit I variables: device widths and lengths I constraints: manufacturing limits, timing requirements, maximum area I objective: power consumption data fitting: from a family of potential models, find a model that best fits observed data and prior information I variables: model parameters I constraints: prior information, parameter limits I objective: measure of misfit or prediction error IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 5 Solving optimization problems solution method I A solution method for a class of problems is an analytical solution or an algorithm that computes the solution (to a given accuracy), given an instance of a problem from the class. general optimization problem I very di cult to solve I methods involve some compromise, e.g., very long computation time, or not always finding the solution exceptions: certain problem classes can be solved e reliably I least-squares problems I linear programming problems I convex optimization problems ciently and IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 6
Least-squares minimize kax bk 2 2 A 2 R k n, k n, rank(a) =n solving least-squares problems I analytical solution: x? =(A T A) 1 A T b I reliable and e cient algorithms and software I computation time proportional to n 2 k;lessifstructured I a mature technology using least-squares I least-squares problems are easy to recognize I a few standard techniques increase flexibility (e.g., including weights, adding regularization terms) IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 7 Linear programming minimize c T x subject to ai T x apple b i, i =1,...,m solving linear programs I no analytical formula for solution I reliable and e cient algorithms and software I computation time in practice proportional to n 2 m if m n; less with structure I a mature technology using linear programming I not as easy to recognize as least-squares problems I a few standard tricks used to convert problems into linear programs (e.g., problems involving `1- or`1-norms, convex piecewise-linear functions) IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 8
Convex optimization problem minimize f 0 (x) subject to f i (x) apple b i, i =1,...,m I objective and constraint functions are convex: for i =0, 1,...,m, f i ( x + y) apple f i (x)+ f i (y) for any x, y 2 R n and any 0, 0 such that + =1 I includes least-squares problems and linear programs as special cases IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 9 Convex optimization problem solving convex optimization problems I no analytical solution I reliable and e cient algorithms I computation time (roughly) proportional to max{n 3, n 2 m, F }, where F is cost of evaluating f i s and their first and second derivatives I almost a technology using convex optimization I often di cult to recognize I many tricks for transforming problems into convex form I surprisingly many problems can be solved via convex optimization IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 10
Example m lamps illuminating n (small, flat) patches s lamp power p j θ kj r kj illumination I k intensity I k at patch k depends linearly on lamp powers p j : mx 2 I k = a kj p j, a kj = rkj max{cos kj, 0} j=1 problem: achieve desired illumination I des with bounded lamp powers minimize max k=1,...,n log I k log I des = f 0 (I ) subject to I k = P m j=1 a kjp j, k =1,...,n 0 apple p j apple p max, j =1,...,m IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 11 How to solve? (using techniques we know, or ad hoc approaches) 1. use uniform power: p j = p, varyp 2. use least-squares: minimize P n k=1 (I k I des ) 2 round p j if p j > p max or p j < 0 3. use weighted least-squares: P minimize n k=1 (I k I des ) 2 + P m j=1 w j(p j p max /2) 2 iteratively adjust weights w j until 0 apple p j apple p max 4. use linear programming: minimize max k=1,...,n I k I des subject to 0 apple p j apple p max, j =1,...,m which can be solved via linear programming of course these are approximate (suboptimal) solutions IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 12
5. use convex optimization: problem is equivalent to minimize f0 (I )=max k=1,...,n h(i k /I des ) subject to I k = P m j=1 a kjp j, k =1,...,n 0 apple p j apple p max, j =1,...,m with h(u) =max{u, 1/u} } 5 4 h(u) 3 2 nts 1 0 0 1 2 3 4 u f 0 is convex because maximum of convex functions is convex exact solution obtained with e ort modest factor least-squares e ort IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 13 additional constraints: does adding (1) or (2) below complicate the problem? 1. no more than half of total power is in any 10 lamps 2. no more than half of the lamps are on (p j > 0) I answer: I moral: (untrained) intuition doesn t always work; without the proper background very easy problems can appear quite similar to very di cult problems IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 14
Course goals and topics goals 1. recognize/formulate problems (such as the illumination problem) as convex optimization problems 2. develop code for problems of moderate size (e.g., 1000 lamps, 5000 patches) 3. characterize optimal solution (optimal power distribution), give limits of performance, etc. topics 1. convex sets, functions, optimization problems 2. examples and applications 3. algorithms IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 15 Nonlinear optimization traditional techniques for general nonconvex problems involve compromises local optimization methods (nonlinear programming) I find a point that minimizes f 0 among feasible points near it, useful if goal is to find a good point I fast, can handle large problems I require initial guess I provide no information about distance to (global) optimum global optimization methods I find the (global) solution I worst-case complexity grows exponentially with problem size these algorithms are often based on solving convex subproblems IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 16
Brief history of convex optimization theory (convex analysis): 1900 1970 algorithms I 1947: simplex algorithm for linear programming (Dantzig) I 1960s: early interior-point methods (Fiacco & McCormick, Dikin,... ) I 1970s: ellipsoid method and other subgradient methods I 1980s: polynomial-time interior-point methods for linear programming (Karmarkar 1984) I late 1980s now: polynomial-time interior-point methods for nonlinear convex optimization (Nesterov & Nemirovski 1994) I mid 1990s now: accelerated first order methods for large-scale and non-smooth convex optimization (Nesterov 2005) applications I before 1990: mostly in operations research; few in engineering I since 1990: many new applications in engineering (control, signal processing, communications, circuit design,... ); new problem classes (semidefinite and second-order cone programming, robust optimization) IOE 611: Nonlinear Programming, Fall 2018 1. Introduction Page 1 17