In an optimization problem, the objective is to optimize (maximize or minimize) some function f. This function f is called the objective function.

Size: px

Start display at page:

Download "In an optimization problem, the objective is to optimize (maximize or minimize) some function f. This function f is called the objective function."

Leo Carr
5 years ago
Views:

1 Optimization In an optimization problem, the objective is to optimize (maximize or minimize) some function f. This function f is called the objective function. For example, an objective function f to be maximized may be the revenue in a production of TV sets, the yield per minute in a chemical process, the hourly number of customers served in some office, the hardness of steel, or the tensile strength of a rope. Similarly, we may want to minimize f if f is the cost per unit of producing certain cameras, the operating cost of some power plant, the daily loss of heat in a heating system, the idling time of some lathe, or the time needed to produce a fender. EE202 ~ Page 48 / Part 2

2 Unconstrained Optimization: An Example Cour rtesy of Professor Wu-Shen ng Lu of University of Victo oria, Can nada EE202 ~ Page 49 / Part 2

3 Unconstrained Optimization: An Example (cont.) Cour rtesy of Professor Wu-Shen ng Lu of University of Victo oria, Can nada EE202 ~ Page 50 / Part 2

4 Unconstrained Optimization: An Example (cont.) Cour rtesy of Professor Wu-Shen ng Lu of University of Victo oria, Can nada EE202 ~ Page 5 / Part 2

5 Unconstrained Optimization: An Example (cont.) EE202 ~ Page 52 / Part 2

6 Constrained Optimization: An Example Courtesy of Pro ofessor Wu-Sheng Lu of University of Victor ria, Cana ada EE202 ~ Page 53 / Part 2

7 Background In most optimization problems the objective function f depends on several variables x,, x. n There are called control variables because we can control them, e.g., ) The yield of a chemical process may depend on pressure x and temperature x 2. 2) The efficiency of a certain air conditioning system may depend on temperature x, air pressure x 2, moisture content x 3, crosssectional area of outlet x 4, etc. EE202 ~ Page 54 / Part 2

8 Background Optimization theory develops methods for optimal choices of x, x 2,, x n, which maximize (or minimize) the objective function f, that is methods for finding optimal values of x, x 2,, x n. In many problems the choice of values of x, x 2,, x n is not entirely free but is subject to some constraints, that is, additional conditions arising from the nature of the problem and the variables. For example, if x is production cost, then x 0 and there are many other variables (time, weight, distance travelled by a salesman, etc.) that can take nonnegative values only. Constraints can also have the form of equations (instead of inequalities). EE202 ~ Page 55 / Part 2

9 Unconstrained Optimization Let us first consider unconstrained optimization in the case of a real-valued function f (x, x 2,, x n ). We also write x = (x, x 2,, x n )' and f (x), for convenience. f has a minimum at point x = X 0 in a region R if f (x) f (X 0 ) for x in R. Similarly, f has a maximum at X 0 if f (x) f (X 0 ) for all x in R. Minima and maxima are called extrema. f is said to have a local minimum at X 0 if f (x) f (X 0 ) for all x in a neighbourhood of X 0, for all x satisfying where r > 0 is sufficiently small. x X r 0 EE202 ~ Page 56 / Part 2

10 Example: Minimizing a Function of a Single Variable Golden Section Search EE202 ~ Page 57 / Part 2

11 Unconstrained Optimization Question: How can one reach the valley bottom (minima) fastest? Answer: Take the path in the most downhill direction. Mathematically, this is the direction of the negative gradient. EE202 ~ Page 58 / Part 2

12 Unconstrained Optimization (cont.) If f is differentiable and has an extremum at a point X 0, then the partial derivatives f / x must be zero at X 0. Thus its gradient,, f / xn f x f ( X0 ) 0. () f x n x=x0 A point X 0 at which f ( X ) 0 is called a stationary point (valley bottom) of f. 0 Condition () is necessary for an extremum of f at X 0 in the interior of R, but is not sufficient. In practice, solving () will often be difficult. For this reason, one generally prefers solution by iteration, by search processes that start at some point and move stepwise to points at which f is smaller (if a minimum of f is wanted) or bigger (in the case of maximum). EE202 ~ Page 59 / Part 2

13 Cauchy s Method Cauchy s method of steepest descent or gradient method is one such popular method. However, convergence can sometimes be slow. Given a multivariable function f (x), we examine its Taylor expansion (obtained from Taylor series) f ( x t x ) f ( x ) t f ( x ) x, Thus, we can expect a decrease in value of f if we set the step direction x to be T x x f x x, x, f ( x) x n x n f x n x 2 f ( x) xf ( x) and the step size t 0. This is because T f ( x t f( x)) f( x) t f ( ) f ( ) x x f ( x) EE202 ~ Page 60 / Part 2 0 x x+tδx x f ( xtx )

14 Cauchy s Method (cont.) T Why f( x) f( x) 0? Noting that f x f x T f f f( x) f( x) x xn f xn f xn and f x 2 2 T f f f f f( x) f( x) 0 x x n x xn f x n thus, we have T f ( x t f( x )) f( x ) t f( ) f( ) x x f( x ) (2) T EE202 ~ Page 6 / Part 2

15 Basic Idea of Cauchy s Method Now suppose that we start from an initial point, say x 0, and an appropriate step size t 0 and let x = x 0 t f f (x( 0 ). From (2), we have T f( x ) f( x 0 tf( x 0)) f( x 0) t f( 0) f( 0) x x f( x 0) Let us move on from x by defining x 2 = x t 2 f (x ) with an appropriately chosen step size t 2. Again from (2), we have T f( x 2) f( x t2f( x )) f( x ) t 2 f( ) f( ) x x f( x ) f( x 0) By repeating this process, we obtain an iterative sequence or scheme: x x tf( x ) (3) i i i i such that the resulting sequence has the following property: f( x ) f( x ) f( x ) f( x ) f( X ) 0 i i 0 The sequence generated by (3) would converge to X 0 as i tends to. EE202 ~ Page 62 / Part 2

16 Cauchy s Method Selection of Step Size The issue on how to select an appropriate step size is quite complicated. In principle, we can fix the step size t to be some appropriate small positive scalar, say t *, and carry on the iteration: * i i t f i x x ( x ) (4) Such an iteration might not be efficient, but surely works. Alternatively, we can also determine the step size t i and the corresponding point x x tf( x ) i i i i at which the function f (x i ) is the smallest (a bit closer to the minima ) among all the possible choices of step size t. We illustrate this in an example EE202 ~ Page 63 / Part 2

17 An Illustrative Example x f ( x ) x 3x f (x) x 2 minima x 2 x Oops, overshoot x EE202 ~ Page 64 / Part 2

18 Example 2 2 Determine a minimum of f ( x) x 3 x, starting from x0 6 3 using the 2 method of steepest descent. Clearly, inspection shows that f (x) has a minimum at 0. Knowing the solution gives us a better feeling of how the method works. We first obtain the gradient and the iteration scheme f x 2x f ( x) f x 6x 2 2 x x, i 2 x, i ( 2 t) x, i i() t x i tf ( x i) t x2, i 6 x2, i ( 6t) x (5) 2, i Note that x i is depended on the choice of the step size t, and x,i and x 2,i are the values of x and x 2 corresponding to x i. We are now to select a step size t i such that the resulting f (x i ) is the smallest among all the choices of t. EE202 ~ Page 65 / Part 2

19 From (5), we have Optimal Selection of Step Size ( 2 tx ) f t f t x t x ( 6 tx ) 2, i, i x i () ( 2), i 3( 6) 2, i To find the small value of f above with respect to t, we take d d 2 2 d 2 2 f xi() t ( 2) t x, i 3( 6) t x2, i dt dt dt 2 2, i tx 2, i 2 ( 2 tx ) ( 2) 23( 6 ) ( 6) 2 2, i tx2, i 4( 2 tx ) 36( 6 ) x, i x 2, i t x, i x 2, i f ( x ) x 3x t i x 2 2, i 9 x2, i 2 2, i x 2, i 2x 54 the optimal choice of step size EE202 ~ Page 66 / Part 2

20 6 Starting from x we compute the values for x and x as listed in the, table below and plotted in the figures on the next page. Step i x i x i x 2 i t i+ Step Size,i 2,i t i x 2 2, i 9 x 2, i 2 2, i x2, i 2x x, i f ( x i ) 6x2, i fast x x tf( x ) i i i i steepest.m EE202 ~ Page 67 / Part 2

21 x 2 x 0 2D View x 2 x x x 0 f (x) x 2 3D View x 2 x x EE202 ~ Page 68 / Part 2

22 Example (cont.) Solving it with a fixed step size Step i x i x i x 2 i Size Fixed Step Size t * 2x, i,i 2,i f ( x i ) 6x2, i * t * : : : : : slow x x t f( x ) i i i The drawback of the method with a fixed step size is: it is generally slow in converging to the solution. steepest2.m EE202 ~ Page 69 / Part 2

23 Summary of Unconstrained Optimization (Cauchy s Method) Given a real valued nonlinear function f (x) = f (x,, x n ) and an initial point x 0, the problem is to find a solution X 0 such that f (X 0 ) is optimal (either minimum or maximum) Compute the gradient of f (x), i.e., f (x) Design and perform an iterative e scheme: e x i = x i t i f (x i ) If the iterative i scheme converges and stops at a certain step, say k,, then X 0 x k and the minimum value of f (x) is then approximately given by f (x k ), i.e., f (X 0 ) f (x k ) EE202 ~ Page 70 / Part 2

24 Linear Programming Linear programming (or linear optimization) consists of methods for solving optimization problems with constraints in which the objective function f is a linear function of the control variables x, x 2,, x n. Problems of this type arise in production, distribution of goods economics, and approximation theory. EE202 ~ Page 7 / Part 2

25 Example A Container Production Optimization Problem: Suppose that in producing two types of containers K and L one uses two machines M and M 2. To produce a container K, one needs M two minutes 2 and M 2 four minutes. Similarly, L occupies M eight minutes and M 2 four minutes. The net profit for a container K is $29 and for L it is $45. Determine the production plan that maximizes the net profit. EE202 ~ Page 72 / Part 2

26 Example (cont.) Problem Formulation: If we produce x containers K and x 2 containers L per hour, the profit per hour is f ( xx, ) 29x 45 x. 2 2 The constraints are 2x 8x 60 (resulting from machine M ) 2 4x 4x 60 (resulting from machine M ) 2 2 x x EE202 ~ Page 73 / Part 2

27 Figure 2 shows these constraints (x, x 2 ) must lie in the first quadrant and below or on the straight line 2 x + 8 x 2 = 60 as well as below or on the line 4 x + 4 x 2 = 60. Thus, (x, x 2 ) is restricted to the quadrangle OABC. We have to find (x, x 2 ) in OABC such that f (x, x 2 ) is maximum. Now f (x,x 2 ) = 0 gives x 2 = (29/45) x (see Fig. 2). The lines f (x, x 2 ) = constant are parallel to that line. We see that B, that is, x = 0 and x 2 = 5, gives the optimum f (0, 5) = 55. (M 2 ) x 2 Hence the answer is that the optimal production Figure 2 plan that maximizes the C B profit is achieved by producing containers K and L in the ratio 2:, the O 0 A (M ) maximum profit being f = 55 f = 0 $55 per hour. f = 200 x EE202 ~ Page 74 / Part 2

28 Simplex Method In practice, linear programming problems might contain many more variables than the two variables considered in the previous example. e A computational o a method is then required ed to solve the problem. One such technique is the Simplex Method. We illustrate such a method through examples EE202 ~ Page 75 / Part 2

29 Example Maximize f x 4x 2 subject to constraints x 2 x 6 2 5x 4 x 40 2 x, x 0 2 Idea: Introduce slack variables w, w. The constraints then become 2 x 2x w 6 2 5x 4x w x, x, w, w and the objective function f x 4 x 0 2 Solution: Simplex table (to be constructed on the next page) EE202 ~ Page 76 / Part 2

30 Formulation of Simplex Method The problem is equivalent to finding one of the solutions of the following set of linear equations: 0 0 f x 2x w 0w f 5x 4x 0w w f x 4x 0w 0w subject to the constraints, x, x 2, w, w 2 0, such that f is maximum. From what we have learnt in linear algebra, we know solutions to the above linear equations remain unchanged with the following 3 basic operations (B.O. B.O.): BO:Interchange B.O.: of two equations B.O.2: Multiplication of an equation by a nonzero constant BO3:Addition B.O.3: of a multiple of one equation to another equation EE202 ~ Page 77 / Part 2

31 Key Idea of Simplex Method The key idea of Simplex Method is to carry out a series of these basic operations (mainly B.O.2 and B.O.3) on the equations to transfer them into an certain form for which the desired solution (the solution corresponding to the maximum f can be easily observed and deduced). We will illustrate this idea through specific examples EE202 ~ Page 78 / Part 2

32 Simplex Table: Similar to the Matrix Form of Linear Equations Problem variables Slack variables Constants Basis x x 2 w w 2 b check w w f f x 2x w 0w f 5 x 4 x 0 w w f x 4x 0w 0w unity matrix EE202 ~ Page 79 / Part 2

33 Simplex Method (cont.) The check column on the right hand side is included to provide a check on the numerical calculations as we develop the simplex. For each row, total up the entries in that row and enter the sum in the check column. We are to perform a series of basic operations (B.O.2 and B.O.3) such that the coefficients associated with the objective function are all nonnegative. For the given example, it will be seen soon that the objective function can be transformed into: f 0x0 x2 w w2 24 f 24 w w2, w, w Clearly, the maximum of f is 24 (by setting w = w 2 = 0). EE202 ~ Page 80 / Part 2

34 EE202 ~ Page 8 / Part 2

35 EE202 ~ Page 82 / Part 2

36 EE202 ~ Page 83 / Part 2

37 f 0 x0 x2 w w2 24 f 24 w w2, w, w EE202 ~ Page 84 / Part 2

38 Simplex Method (re-cap) EE202 ~ Page 85 / Part 2

39 Com mplete e Simp plex Ta able Basis x x 2 w w 2 b check w w f w w f x 2 w w f x w f x 2 0 x w Simplex_Ex_ EE202 ~ Page 86 / Part 2 f f max 24 with x 4, x2 5. f x x 24 with 4, 5 max 2

40 Container Production Optimization Problem (re-visit) The Problem: Suppose that in producing two types of containers K and L one uses two machines M and M 2. To produce a container K, one needs M two minutes and M 2 four minutes. Similarly, L occupies M eight minutes and M 2 four minutes. The net profit for a container K is $29 and for L it is $45. Determine the production plan that maximizes the net profit per hour. Formulation: If we produce x containers K and x 2 containers L per hour, the profit is f ( x, x ) 29x 45x subject to the constraints: 2 2 2x 8x x 4x 60 2 x 0, x x 8 x w x 4x w f 29x 45x 0 2 x, x2, w, w2 0 EE202 ~ Page 87 / Part 2

41 Sim mplex Metho od x x 2 w w2 basis b check f x x w 2 w2 basis b check x, x2, w, w f x x 2 w w 2 basis b check f EE202 ~ Page 88 / Part 2

42 Sim mplex Metho od x x 2 w w 2 basis b check f x x 2 w w 2 basis b check x, x2, w, w2 0 f x x 2 w w 2 basis b check x x x Simplex_Ex_2 f f 2.666w 5.97w 55 f 2.666w 5.97w 55 f max EE202 ~ Page 89 / Part 2

43 Simplex Method for More Variables Many problems in real life involve more than just two variables. However, the method of computation and the key idea remain basically the same. It is an iterative process which is repeated until the index row contains no negative entry, at which h point the optimal value of the objective function is attained. We illustrate the process for solving optimization problems for more variables thru the following examples EE202 ~ Page 90 / Part 2

44 Example of 3 Variables EE202 ~ Page 9 / Part 2

45 Basis x x 2 x 3 w w 2 w 3 b check w w w f B.O.2. w w w f B.O.3. x 2 w w w f EE202 ~ Page 92 / Part 2

46 Basis x x 2 x 3 w w 2 w 3 b check x w B.O.2. w f x w B.O.3. 9 x 3 w f fmax 60 with x 0, x2 4, x3 9. Simplex_Ex_ f x w w3 60 f 60 x w w3, x, w, w EE202 ~ Page 93 / Part 2

47 Example with Constraint x x Introduction ti of slack variables w, w 2, w 3 gives 2x x2 w 50 This innovative procedure was 4x 3x2 w2 350 suggested by Yin Mingbao, a student xx2 w3 80 taking EE202 in Semester of x, x, w, w, w 0 Academic Year 2009/ EE202 ~ Page 94 / Part 2

48 Com mplete Simpl lex Ta able basis x x 2 w w 2 w 3 b check w w w f w w w f x w B.O.3. w w f x w w w f EE202 ~ Page 95 / Part 2

49 Com mplete Simpl lex Tab ble (co ont.) basis x x 2 w w 2 w 3 b check x w w x 2 w f x w 3 w x f x w x f B.O.3. f 550 with x 50, x 50, w 20 Simplex_Ex_4 max 2 3 f 2.5w 0.5w 550 f w 0.5 w, w, w EE202 ~ Page 96 / Part 2

50 Another Example with Constraint 3x 5x 25 2 Introduction of slack variables w, w 2 2, w 3 gives 2x3x2 w 20 x We are to solve this problem once x2 w2 45 3x 5x2 w3 25 again using the procedure suggested by Yin Mingbao. x, x, w, w, w EE202 ~ Page 97 / Part 2

51 Simp plex Ta able basis x x 2 w w 2 w 3 b check w w w f w w w f w x w w B.O.3. f f 4x 8w 360 f 360 4x 8 w, x, w 0 f 360? max 8x 3w w 60 w This is a contradiction! EE202 ~ Page 98 / Part 2

52 What to do next? The remaining control variable. The row has problem basis x x 2 w w 2 w 3 b check w x w f Pivot: The intersection of the row containing w 3 and the control variable x 2 to o get rid of w 3 in the basis and obtain an explicit solution for x 2. EE202 ~ Page 99 / Part 2

53 Simple ex Tab ble (co nt.) basis x x 2 w w 2 w 3 b check w x w /8 / f w 0 0 9/8 / x 0 0 5/8 / x /8 / f B.O.3. f 280 with x 25, x 20, w 0 max 2 f 6.5w 0.5w 280 f w 0.5 w, w, w Simplex_Ex_5 EE202 ~ Page 200 / Part 2

54 Summary of Constrained Optimization (Simplex Method) Given a set of linear equations and/or inequalities (constraints), the problem is to determine a solution such that a certain objective function is maximized Construct a Simplex table with slack variables Perform iteratively eyase series eso of basic bascope operations ato s( (B.O.2 & B.O.3) O3) The maximum value of the objective function can be determined d when all its resulting coefficients are nonnegative EE202 ~ Page 20 / Part 2

Optimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30

Optimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained