Outline. Basic Concepts in Optimization Part I. Illustration of a (Strict) Local Minimum, x. Local Optima. Neighborhood.

Transcription

1 Outline Basic Concepts in Optimization Part I Local and Global Optima Benoît Chachuat <benoit@mcmaster.ca> McMaster University Department of Chemical Engineering ChE G: Optimization in Chemical Engineering Numerical Methods: Improving earch Notions of Conveity Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Local Optima Neighborhood The neighborhood N δ ( ) of a point consists of all nearby points; that is, all points within a small distance δ > of : Illustration of a (trict) Local Minimum, f ( ) < f (), N δ ( ) \ { } N δ ( ) = { : < δ} Local Optimum f () A point is a [strict] local minimum for the function f : IR n IR on the set if it is feasible ( ) and if sufficiently small neighborhoods surrounding it contain no points that are both feasible and [strictly] lower in objective value: δ > : f ( ) f (), N δ ( ) [ δ > : f ( ) < f (), N δ ( ) \ { } ] f ( ) δ N δ ( ) Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G /

2 Global Optima Illustration of a (trict) Global Minimum, Global Optimum A point is a [strict] global minimum for the function f : IR n IR on the set if it is feasible ( ) and if no other feasible solution has [strictly] lower objective value: f ( ) < f (), \ { } f ( ) f (), f () [ f ( ) < f (), \ { } ] Remarks: Global minima are always local minima Local minima may not be global minima f ( ) Analog definitions hold for local/global optima to maimize problems Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 6 / Global vs. Local Optima Class Eercise: Identify the various types of minima and maima for f on = [ min, ma ] f () How to Find Optima? Review: Three Methods for Optimization Graphical olutions Great display + see multiple optima But impractical for nearly all practical problems Analytical olutions (e.g., Newton, Euler, etc.) Eact solution + easy analysis for changes in (uncertain) parameters But not possible for most practical problems Numerical olutions The only practical method for comple models! But only guarantees local optima + challenges in finding effects of (uncertain) parameters min ma Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 7 / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 8 /

3 Numerical Optimization: The Dilemma! Consider the optimization problem: min f (, ) =, f (,) ( ) + ( ) +. + ( ) + ( ) Numerical Optimization: The Dilemma! Typically, only some local information is know about the objective function typically at a current point = (, )! Question: Which move do I make net? f (,) current point Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 9 / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 9 / Numerical Optimization: The Basic Approach Improving earch Improving search methods are numerical algorithms that begin at a feasible solution to a given optimization model, and advance along a search path of feasible points with ever-improving function value f (,) Direction-tep Paradigm At the current point (k), how do I decide: the direction of change the magnitude of change whether further improvement is possible? The Basic Equation Improving search advances from current point (k) to new point (k+) as: (k+) (k) (k+) (k+) =. = (k) (k) + α =. + α. n (k+) n (k) n where: defines a move direction of solution change at (k) ( = ) α > determines a move magnitude, how far to pursue this direction Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G /

4 Direction of Change, Improving Directions Vector IR n is an improving direction at current point (k) if the objective function value at (k) + α is superior to that of (k), for all α > sufficiently small (maimize problem) ᾱ > : f ( (k) + α ) > f ( (k) ), α (,ᾱ] Direction of Change, Improving Directions Vector IR n is an improving direction at current point (k) if the objective function value at (k) + α is superior to that of (k), for all α > sufficiently small (maimize problem) ᾱ > : f ( (k) + α ) > f ( (k) ), α (,ᾱ]. current point f (,).8.6.., improving direction (k+) (k) set of improving directions at (k+) set of improving directions at (k) Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Direction of Change, (cont d) Feasible Directions Vector IR n is an feasible direction at current point (k) if point (k) + α violates no model constraint for all α > sufficiently small ᾱ > : (k) + α, α (,ᾱ] Optimality Criterion Necessary Condition of Optimality (NCO) No optimization model solution at which an improving feasible direction is available can be a local optimum set of feasible directions at set of feasible directions at (k) (k) set of improving directions at Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G /

5 Continuous Improving earch Algorithm tep : Initialization. Choose any starting feasible point () and let inde k. tep : Move Direction. If no improving feasible direction eists at current point (k), stop. Otherwise, construct an improving feasible direction at (k) as (k+). tep : tep ize. If there is no limit on step sizes for which direction (k+) continues to both improve the objective function and retain feasibility, stop The model is unbounded. Otherwise, choose the largest step size α (k+). tep : Update. (k+) (k) + α (k+) (k+) Increment inde k k + and return to step. A Word of Caution! Caution: A point at which no improving feasible direction is available may not be a local optimum! set of feasible directions at Remarks: This basic algorithm may terminate at a suboptimal point Moreover, it does not distinguish between local and global optima set of improving directions at Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 6 / Finding out Optima! Class Eercise: Determine whether each of the following points is apparently a local/global minimum? a local/global maimum? neither? Conve ets A set IR n is said to be conve if every point on the line connecting any two points,y in is itself in, γ + ( γ)y, γ (,) y 6 Nonconve et: ome points on the line connecting,y do not lie in Nonconnected sets are nonconve; e.g., the discrete set {,,,...} 6 y 6 7 Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 7 / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 8 /

6 Conve and Concave Functions Conve Functions A function f : IR, defined on a conve set IR n, is said to be conve on if the line segment connecting f () and f (y) at any two points,y lies above the function between and y, f (γ + ( γ)y) γf () + ( γ)f (y), γ (,) Conve and Concave Functions (cont d) Case of a strictly conve function on the conve set f () γf ( ) + ( γ)f ( ) Case of a nonconve function on, yet conve on the conve set f () trict conveity: f (γ + ( γ)y) < γf () + ( γ)f (y),,y, γ (,) Concave Functions f is said to be [strictly] concave on if ( f ) is [strictly] conve on, f (γ + ( γ)y) [>]γf () + ( γ)f (y),,y, γ (,) f (γ + ( γ) ) γ + ( γ) Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G 9 / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / ets Defined by Constraints Define the set = { IR n : g() }, with g a conve function on IR n. Then, is a conve set Why? Consider any two points,y. By the conveity of g, g(γ + ( γ)y) γg() + ( γ)g(y), γ (,) ince g() and g(y), g() + ( γ)g(y), γ (,) g() = Therefore, γ + ( γ)y for every γ (, ); i.e., is conve Class Eercise: Give a condition on g for the following set to be conve: = { IR n : g() } ets Defined by Constraints (cont d) What is the condition on h for the following set to be conve: = { IR n : h() = } The set is conve if and only if h is affine Conve ets Defined by Constraints Consider the set points not in y h() = = { IR n : g (),...,g m (),h () =,...,h p () = } Then, is conve if: g,...,g m are conve on IR n h,...,h p are affine Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G / Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G /

7 Conveity and Global Optimality Consider the constrained program: ma f () s.t. g j (), j =,...,m h j () =, j =,...,p If f and g,...,g m are conve on IR n, and h,...,h p are affine, then this program is said to be a conve program ufficient Condition for Global Optimality A [strict] local minimum to a conve program is also a [strict] global minimum On the other hand, a nonconve program may or may not have local optima that are not global optima Benoît Chachuat (McMaster University) Basic Concepts in Optimization Part I G /