Lecture Note 1: Introduction to optimization Xiaoqun Zhang Shanghai Jiao Tong University Last updated: September 23, 2017
1.1 Introduction 1. Optimization is an important tool in daily life, business and engineer: Running a business: to maximize profit, minimize loss, maximize efficiency, or minimize risk. Design: minimize the weight of a bridge, and maximize the strength, within the design constrains; airplane engineering such as shape design and material selection; packing transistors in a computer chip in a functional way. Planning: select a flight route to minimize time or fuel consumption of an airplane Supermarket pricing and logistic 2. Formal definition: to minimize (or maximize) a real function by deciding the values of free variables from within an allowed set. 3. Status of optimization: in last few decades, astonishing improvements in computer hardware and software motivated optimization modeling, algorithm designs, and implementation. Solving certain optimization problems has become standard techniques and everyday practice in businesses, science and engineering. It is now possible to solve certain optimization problems with thousands, millions and even thousands of millions of variables; optimization (along with statistics) has been the foundation of machine learning and big-data analytic. 4. Ingredients of successful optimization: modeling (turn a problem into one of the typical optimization formulations); algorithms an (iterative) procedure that leads you toward a solution (most optimization problems do not have a closed-form solution); software and hardware implementation: realize the algorithms and return numerical solutions. 5. Optimization formulation: mininize x (maximize) f(x) subject to x C minimize is often abbreviated as min ; decision variable is typically stated under minimize, unless obvious; subject to is often shorten as s.t. ; in linear and nonlinear optimization, feasible set C is represented by h i (x) = b i, i E(equality constraints) g j (x) b j, j I(inequality constrains) 2
6. First examples: An office furniture manufacturer wants to maximize the daily profit. Decide the daily number of desks and chairs that should be manufactured such that the profit is maximum. The required material and profit of manufacturer desks and chair is listed as followed: Each piece Required material Required labor Profit desk 10 5 20 chair 6 2 10 Total available 240 100 Problem: let the daily manufactured number of desks and chairs be x 1, x 2, solve max x 1,x 2 20x 1 +10x 2 s.t. 10x 1 +6x 2 240 5x 1 +2x 2 100 x 1,x 2 0 Find two nonnegative numbers whosesumis up to6sothattheir product is a maximum. ( find the largest area of a rectangular region provided that its perimeter is no greater than 12). Problem: let the two numbers be x,y, solve max x,y xy s.t. x+y = 6 x,y 0 Find a line that best fit three given points (x 1,y 1 ) = (2,1),(x 2,y 2 ) = (3,6) and (x 3,y 3 ) = (5,4). Problem: let the line equation be y = ax+b, in the following least square sense: min a,b 3 (ax i +b y i ) 2 = (2a+b 1) 2 +(3a+b 6) 2 +(5a+b 4) 2 i=1 Traveling sales man problem: A salesman needs to visit a number (n) of cities, denoted by city 1,2,,n. The distances between cities are known, i.e. d ij denotes the distance between city i and city j. The salesman wants to travel each city once in turn and in the meantime 3
minimize the total travel distance. What order should be used? Problem: denote the order of travelling by (i 1,,i n ), model n 1 min d ik,i k+1 k=1 s.t. (i 1,i 2,,i n ) is a permutation of (1,2,,n) 7. Classification of optimization problems 8. Convex programming: Continuous vs Discrete Constrained vs Unconstrained Global vs Local Stochastic vs Deterministic Convex vs Nonconvex minf(x) s.t. X C both f and C are convex. Special cases: linear programming minc T x s.t. Ax b;bx = d; least square problems (quadratic program): minx T Qx+c T x s.t. Ax b;bx = d; 9. Global vs local solution: solutions means optimal solution. Global solution x : f(x ) f(x) for all x C. Local solutions x : δ > 0 such that f(x ) f(x) for all x C and x x δ. A (global or local) solution x is unique if holds strictly as <. In general, it is difficult to tell if a local solution is global because algorithms can only check nearby points and have not clue of behaviors father way. Hence a solution may refer to a local solution. A local solution to a convexprogram if globally optimal. A LP is convex. A stationary point (where the derivatives is zero is also known as a solution, but it can be a maximization, minimization 10. Nonlinear program: min f(x) = (x 1 +x 2 ) 2 s.t.x 1 x 2 0 2 x 1 1 2 x 2 1 Decision variables, feasible set, objective, (box) constraints,global minimizer (x = ( 2, 2)) vs local minimizer (x = (1,1)). See the figure 10. 4
3 2 A 1 H D x (Local solution 4 3 F 2 1 E I 1 2 1 B 2 x g G (Global solution) C 3 4 5 Figure 1.1: Global solution vs. local solution 11. Optimization algorithms requirement Robustness: perform well on a wide variety of problem Efficiency: reasonable computer time or storage Accuracy: precision, not being sensitive to norms Using convex optimization problem: often difficult to recognize, many tricks for transforming problems into convex form, surprisingly many problems can be solved via convex optimization. 12. This course focuses on finding local solutions and, for convex programs, global solutions. Asking for global solutions is computationally intractable, in general. Many useful problems are convex, that is, a local solution is global In some applications, a local solution is an improvement from an existing point. Local solutions are OK. 13. We do not cover discrete optimization: Discrete variables often take binary or integer values, or values from a discrete set. Those problems are called discrete optimization. 5