Part 2: NLP Constrained Optimization

Size: px

Start display at page:

Download "Part 2: NLP Constrained Optimization"

Dominick Farmer
6 years ago
Views:

1 Part 2: NLP Constrained Optimization James G. Shanahan 2 Independent Consultant and Lecturer UC Santa Cruz James_DOT_Shanahan_AT_gmail_DOT_com WIFI: SSID Student USERname ucsc-guest Password EnrollNow! TIM Introduction to Optimization Theory and Applications Thursday, March 14, 2013 Lecture 14 University of California, Santa Cruz TIM Introduction to Optimization to Optimization Theory Theory and Applications, and Applications, Winter 2013 Winter 2013 James 2013 G. James Shanahan G. Shanahan 1

2 Final Eam Take home final Eam, due on March 24, 2013, You will receive it on March 17, 2013 Happy Saint Patrick s Day! 2

3 Outline Lecture 10 Multivariable unconstrained Linear regression, Logistic Regression KKT conditions for Constrained Optimization Lagrangian, KKT conditions Quadratic Programming Modified Simple Algorithm Conve Programming Frank-Wolfe Algorithm, Penalty or barrier function e.g., SUMT Noncove Programming Course review 3

4 Newton s Method in Optimization This iterative scheme can be generalized to several dimensions by replacing the derivative with the gradient, f X, and the reciprocal of the second derivative with the inverse of the Hessian matri,. One obtains the Newtoniterative scheme i i+ 1 f ' i = + For one variable i f '' X i+ 1 = = i X d2 f d i i 1 f ' i in matri form [ ] i 1 i f '' X f ' X For multivariable 4

5 Newton s Method in Optimization As we have seen above, Newton's method is used to find the roots of equations in one or more dimensions. It can also be used to find local maima and local minima of functions, as these etrema are the roots of the derivative function i.e., f. We shall define a series of -s, starting from an initial guess 0, s.t. the series converges towards * which satisfies f' * = 0. This * will also be an etremum, i.e. stationary point, of f Thus, provided that f is a twice-differentiable function and the initial guess is chosen close enough to *, the sequence n defined as follows will converge on *: i [ ] i+1 f ' i i i 1 i = + = f '' f ' Iteration function i f '' 5

6 Gradient: Nabla is the symbol. the gradient at a specific point = is the vector whose elements are the respective partial derivatives evaluated at X=, so that f = df/d1, df/d2,. df/dn The significance of the gradient is that the infinitesimal change in that maimizes the rate at which f increases is the change that is proportional to f. 6

7 Gradient. 7

Significance of the Gradient Vector The gradient vector, f,y, gives the direction of fastest increase of f, y assuming a two-variable function here.

8 Significance of the Gradient Vector The gradient vector, f,y, gives the direction of fastest increase of f, y assuming a two-variable function here. [Newton-Raphson] The gradient vector, f,y, is orthogonal to the contour lines Imagine climbing an upside-down bowl from below, where I can move in any <, y> direction NOTE I cant move in z;, and y are independent variables. If I follow the level curve f,y=k then I make no progress to the summit or bottom but if I move perpendicular to the level curve then I make the quickest progress to the summit of the bowl. 8

9 F = where F I. E. F = F = F = F * + F = * 1 [ ] T F, F, Fn, [ F ', F ', F ', ] T F, T * +... is the gradient of F evaluated at * = * F,... n n F T MultiVariate Taylor i+ 1 = i i g i g' Iteration function where g = f' i+ 1 = i i f ' i f '' Iteration function for finding roots of f' 9

10 10 MultiVariate Taylor = = + + = = = = = evaluated at * F is the Hessian of and evaluated at * F isthe gradient of * * 2 1 * * * 2 * * 2 * F F F F F F F F F F and F F F F Here F F where F F F F n n n n n T n T T for findi '' ' g where ' 1 1 function Iteration f f function Iteration g g i i i i i i i i = = + +

Multivariate Newton s Method Find the roots of an equation or system of equations Calculating gradient and Hessian not very timeconsuming but calculating the inverse of

11 Multivariate Newton s Method Find the roots of an equation or system of equations Calculating gradient and Hessian not very timeconsuming but calculating the inverse of H is! In R, have a look at?optim #method=bfgs [ 29/lecture-29.pdf] [Hand, Manilla, Smith, Data Mining, Section 8.3] 11

12 Tangent Approimations 12

13 A plane from a point and orthogonal vector Although a line in space is determined by a point and a direction, a plane in space is more difficult to describe. A single vector parallel to a plane is not enough to convey the direction of the plane, but a vector perpendicular to the plane does completely specify its direction. Thus, a plane in space is determined by a point in the plane and a vector that is orthogonal to the plane. This orthogonal vector is called a normal vector. 13

14 Tangent Planes and Linear Approimation Just as we can visualize the line tangent to a curve at a point in 2-space, in 3-space we can picture the plane tangent to a surface at a point. Consider the surface given by z=f,y. Let 0, y0, z0 be any point on this surface. If f,y is differentiable at 0, y0, then the surface has a tangent plane at 0, y0, z0. The equation of the tangent plane at 0,y0, z0 is given by: Tangent Plane Tangent Line z z0= f 0,y0* 0+f y 0,y0 *y y0 similar form: y y0= f 0 * 0 where f 0,y0 is the partial derivative of f WRT calculated at 0,y0; similarly for f y 0,y0 14

15 f F where F I. E. F a F = F = = * = 1 [ ] T F, F, Fn, [ F ', F ', F ', ] T isthe gradient of 1 1 F, = * F,... n F evaluated at * n F Notation = f a + f ' a 1D Linear Approimation T = F * + F * T Multi Variable Linear Appro. T F = F * + F * *

16 z z0= f 0,y0 0+f y 0,y0y y0 Tangent Plane Function of variables Calculate gradient vector by evaluating partial derivatives at the tangential point Gradient vector at 1, 1 is 4, 2; f 1,1 = 4,2 f1,1=3 Tangent plane at 1, 1, 3 with gradient 4,2 [Adapted from Multivariable Calculus: Concepts and Contets, James Stewart] 16

17 Tangent Plane Eample 1, 1 4,2 Gradient vector 17

18 Tangent Plane to Ellipoid Eample 18

19 Approimate Δy with dy via the tangent Difference in f; i.e., second term in Linear Taylor Epansion f = f a + f ' a a Actual y dy Linear Approimation Actual approimated by Predicted I.e., Δy~dy dy is the predicted difference in f given the linear approimation Can change Δ as much as we like but the bigger the Δ the bigger the gap between the tangent approimation and the actual function 19 and dy and Δy

20 20 Linear and Quadratic Approimations Approimate fx for X around point a by the tangent at a point a, fa Taylor Series eplores different approimations of fx; the above tangential form is linear approimation General Form of a Taylor Series n n a f a f a a a a f a f f n... '! 2 2! '' = ' a a f a f f + = k k k a k a f f! 0 = = More compactly f'a slope fa a, AT ' = + = + = = = a a f a f f m y f m y f m y y

21 21 Total Differential, dz, for z=f, y in 2D Estimated change in z using total differential Total Differential in 2D estimated change in z=f using a linear approimation This corresponds to the second term the linear term in Taylors epansion ' a a f a f f dz a f f + = + = Total Change in f,y a a f a f f dz a f f + +

22 Total Differential in 2D estimated change in z First-order Taylor Series a+, b+ y, La+, a + Where L is the linear approimation of f, around the point a,b 22

23 Total Differential in 2D estimated change in z The total derivative estimates how much z changes estimated based on the tangential plane approimation Any f,y can be approimated fa,b +total differential for any,y close to a, b 23

24 Total Differential in 2D: An Eample change in height of f change in height of f when I travel in 0.05, Total Differential Estimated z difference between f, y and fa, b z=f Tanga,b,y-fa,b Actual z difference i.e., z=f,y-fa,b 24

25 Total Differential and Directional Derivative The total derivative tells us how much z changes only estimated as it is based on the tangential plane approimation when we travel in a particular direction u, i.e.,d u f,y= f,y u Where u=-a, y-b Any f,y can be approimated by fa,b +total differential 25

26 Directional Derivative== Total Deriv. == Change in f Change in f when we travel in direction v We can approimate the change in z i.e., f f+v using D v f,y= f,y v D A f,y= f,y -A1,y-A2 26

27 Necessary and Sufficient Conditions for Optimality.. 27

28 KKT Conditions.. 28

29 Problem needs to be concave.. 29

30 .. Eample 30

31 .. Eample 31

32 Shadow prices/dual variables u i.. 32

33 Quadratic Programming. 33

34 Eample.. 34

35 Concave Quadratic Programming.. 35

36 KKT Conditions of QP.. 36

37 .. 37

38 Complementarity Constraint.. 38

39 .. 39

40 From Quadratic to LP via KKT conditions.. 40

41 Find a feasible solution to these constraints.. 41

42 Modified Simple Algorithm Find a feasible solution to this: 42

43 .. 43

44 Restricted Entry Rule.. 44

45 Eample.. 45

46 Simple Tableau.. 46

47 Conve Programming.. 3 families of approaches Sequential Approimation Algorithms Sequential Unconstrained Minimization Techniques SUMT Generalized reduced gradient 47

48 Sequential Approimation Algorithms Sequential-approimation algorithms includes linear approimation and quadratic approimation methods These algorithms replace the nonlinear objective function by a succession of linear or quadratic approimations. For linearly constrained optimization problems, these approimations allow repeated application of linear or quadratic programming algorithms. This work is accompanied by other analysis that yields a sequence of solutions that converges to an optimal solution for the original problem. Can be etended to problems with nonlinear constraint functions by the use of appropriate linear approimations. 48

49 Frank Wolfe Algorithm As one eample of a sequential-approimation algorithm, consider the Frank-Wolfe algorithm for the case of linearly constrained conve programming With constraints: A b and 0 in matri form. This procedure is particularly straightforward; it combines linear approimations of the objective function enabling us to use the simple method with the onedimensional search procedure of 49

50 Frank Wolfe Algorithm The Frank Wolfe algorithm is a simple iterative firstorder optimization algorithm for constrained conve optimization quadratic programming problems. Also known as the conditional gradient method, reduced gradient algorithm and the conve combination algorithm, the method was originally proposed by Marguerite Frank and Philip Wolfe in In each iteration, the Frank Wolfe algorithm considers a linear approimation of the objective function, and moves slightly towards a minimizer of this linear function taken over the same domain. 50

51 Introduction The basic Idea: 1. Use linear functions to approimate both the objective function as well as the constraints Linearization. 2. Employ LP algorithms to solve this new linear program. 51

52 Introduction Linearization can be achieved in two ways: Any non-linear function can be approimated in the vicinity of a point by using Taylor s epansion, is called the linearization point. Using piecewise linear approimations and then applying a modified simple algorithm separable programming. 52

53 Direct Use of Successive Linear Programs Using Taylor s epansion linearize all problem functions at some selected estimate of the solution. Result is an LP. is called the linearization point. With some additional precautions the LP solution ought to be an improvement over the linearization point. There are two cases to be considered: 1. Linearly constrained NLP case: 2. General NLP case: 53 53

54 Linearly constrained NLP case The linearly constrained NLP problem that of: is a nonlinear objective function. Feasible region is a polyhedron, however optimal solution can lie anywhere within the feasible region. 54

Linearly Constrained NLP case Using Taylor s approimation around the linearization point and ignoring the second and higher order terms we obtain the linear approimation of around the

55 Linearly Constrained NLP case Using Taylor s approimation around the linearization point and ignoring the second and higher order terms we obtain the linear approimation of around the point. So the linearized version becomes: The Solution of the linearized version is. How close is to the solution to the original NLP? By virtue of minimization it must be true that 55

56 56 Linear and Quadratic Approimations Using Taylor s approimation around the linearization point 0 =a n n a f a f a a a a f a f f n... '! 2 2! '' = k k k a k a f f! 0 = = More compactly

57 Total Differential in 2D estimated change in z First-order Taylor Series a+, b+ y, La+, a + Where L is the linear approimation of f, around the point a,b 57

58 .. 58

59 FW Algo.. 59

Linearly Constrained NLP case Using a bit of algebra leads us

Previously studied that a descent direction can lead to an

procedure. All points between and are feasible.

60 Linearly Constrained NLP case Using a bit of algebra leads us to the result: So the vector is a descent direction. Previously studied that a descent direction can lead to an improved point only if it is coupled with a step adjustment procedure. All points between and are feasible. Moreover since is a corner point, any point beyond it on the line are outside the feasible region. So, to improve upon, a line search method is employed in the line segment: Minimizing will find a point such that 60

61 Linearly Constrained NLP case will not in general be the optimal solution but it will serve as a linearization point for the net approimating LP. The tet book presents the Frank-Wolfe Algorithm that employs this sequence of alternating LP s and line searches. 61

62 FW Algo.. 62

63 FW Eample.. 63

64 .. 64

65 Iteration

66 .. 66

67 Iteration

68 Optimal Solution.. 68

69 .. 69

70 Frank-Wolfe Algorithm Eecution: Eample 8.1 Page Number

71 Frank-Wolfe Algorithm Eecution: Eample 8.1 Page Number

Frank-Wolfe Algorithm Frank-Wolfe algorithm converges to a Kuhn-Tucker point from any feasible starting point. No analysis for rate of convergence.

72 Frank-Wolfe Algorithm Frank-Wolfe algorithm converges to a Kuhn-Tucker point from any feasible starting point. No analysis for rate of convergence. However, if is conve we can obtain estimates on how much remaining improvements can be achieved. If is conve and it is linearized at a point, for all Hence after each cycle the difference gives an estimate of the improvement. 72

73 Conclusions: Frank-Wolfe Algorithm In conclusion, we emphasize that the Frank-Wolfe algorithm is just one eample of sequential-approimation algorithms. Many of these algorithms use quadratic instead of linear approimations at each iteration because quadratic approimations provide a considerably closer fit to the original problem and thus enable the sequence of solutions to converge considerably more rapidly toward an optimal solution than was the case in the linear FW algo. For this reason, even though sequential linear approimation methods such as the Frank-Wolfe algorithm are relatively straightforward to use, sequential quadratic approimation methods now are generally preferred in actual applications. Popular among these are the quasi - Newton or variable metric methods, which compute a quadratic approimation to the curvature of a nonlinear function without eplicitly calculating second partial derivatives. For further information about conve programming algorithms, see Selected References 4 and 6. 73

74 Conclusions: Frank-Wolfe Algorithm In conclusion, we emphasize that the Frank-Wolfe algorithm is just one eample of sequential-approimation algorithms. Many of these algorithms use quadratic instead of linear approimations at each iteration because quadratic approimations provide a considerably closer fit to the original problem and thus enable the sequence of solutions to converge considerably more rapidly toward an optimal solution than was the case in Fig b H&L Book. For this reason, even though sequential linear approimation methods such as the Frank-Wolfe algorithm are relatively straightforward to use, sequential quadratic approimation methods1 now are generally preferred in actual applications. Popular among these are the quasi -Newton or variable metric methods, which compute a quadratic approimation to the curvature of a nonlinear function without eplicitly calculating second partial derivatives. For linearly constrained optimization problems, this nonlinear function is just the objective function; whereas with nonlinear constraints, it is the Lagrangian function described in Appendi 3. Some quasi-newton algorithms do not even eplicitly form and solve an approimating quadratic programming problem at each iteration, but instead incorporate some of the basic ingredients of gradient algorithms. 74

75 Linearly Constrained NLP case: Frank-Wolfe Algorithm 75

76 MULTI VARIABLE OPTIMIZATION PROCEDURES Introduction Mutivariable Search Methods Overview Unconstrained Multivariable Search Methods Quasi-Newton Methods Conjugate Gradient and Direction Methods Logical Methods Constrained Multivariable Search Methods Successive Linear Programming Successive Quadratic Programming Generalized Reduced Gradient Method Penalty, Barrier and Augmented Lagrangian Functions Other Multivariable Constrained Search Methods Comparison of Constrained Multivariable Search Methods Stochastic Approimation Procedures Closure FORTRAN Program for BFGS Search of an Unconstrained Function References Problems 76

77 Sequential Unconstrained Minimization Techniques SUMT. 77

78 Penalty and Barrier Methods General classical constrained minimization problem minimize f subject to g 0 h = 0 Penalty methods are motivated by the desire to use unconstrained optimization techniques to solve constrained problems. This is achieved by either adding a penalty for infeasibility and forcing the solution to feasibility and subsequent optimum, or adding a barrier to ensure that a feasible solution never becomes infeasible. 78

79 Penalty Methods Penalty methods use a mathematical function that will increase the objective for any given constrained violation. General transformation of constrained problem into an unconstrained problem: min T = f + r k P where f is the objective function of the constrained problem r k is a scalar denoted as the penalty or controlling parameter P is a function which imposes penalities for infeasibility note that P is controlled by r k T is the pseudo transformed objective Two approaches eist in the choice of transformation algorithms: 1 Sequential penalty transformations 2 Eact penalty transformations 79

80 Sequential Penalty Transformations Sequential Penalty Transformations are the oldest penalty methods. Also known as Sequential Unconstrained Minimization Techniques SUMT based upon the work of Fiacco and McCormick, Consider the following frequently used general purpose penalty function: m T = y + r k { ma[0, g i ] 2 + [h j ] 2 } i=1 l j=1 Sequential transformations work as follows: Choose P and a sequence of r k such that when k goes to infinity, the solution found. * is For eample, for k = 1, r 1 = 1 and we solve the problem. Then, for the second iteration r k is increased by a factor 10 and the problem is resolved starting from the previous solution. Note that an increasing value of r k will increase the effect of the penalties on T. The process is terminated when no improvement in T is found and all constraints are satisfied. 80

81 Two Classes of Sequential Methods Two major classes eist in sequential methods: 1 First class uses a sequence of infeasible points and feasibility is obtained only at the optimum. These are referred to as penalty function or eterior-point penalty function methods. 2 Second class is characterized by the property of preserving feasibility at all times. These are referred to as barrier function methods or interiorpoint penalty function methods. General barrier function transformation T = y + r k B where B is a barrier function and r k the penalty parameter which is supposed to go to zero when k approaches infinity. Typical Barrier functions are the inverse or logarithmic, that is: m B = i=1 g i -1 m or B = ln[ g i ] i=1 81

82 SUMT Algo. 82

83 SUMT Summary.. 83

84 What to Choose? Some prefer barrier methods because even if they do not converge, you will still have a feasible solution. Others prefer penalty function methods because You are less likely to be stuck in a feasible pocket with a local minimum. Penalty methods are more robust because in practice you may often have an infeasible starting point. However, penalty functions typically require more function evaluations. Choice becomes simple if you have equality constraints. Why? 84

85 SUMT Closing Remarks Typically, you will encounter sequential approaches. Various penalty functions P eist in the literature. Various approaches to selecting the penalty parameter sequence eist. Simplest is to keep it constant during all iterations. Always ensure that penalty does not dominate the objective function during initial iterations of eterior point method. 85

86 Penalty, Barrier Methods These methods convert the constrained optimization problem into an unconstrained one. The idea is to modify the economic model by adding the constraints in such a manner to have the optimum be located and the constraints be satisfied. There are several forms for the function of the constraints that can be used. These create a penalty to the economic model if the constraints are not satisfied or form a barrier to force the constraints to be satisfied, as the unconstrained search method moves from the starting point to the optimum. This approach is related to the method of Lagrange multipliers which is a procedure that modifies the economic model with the constraint equations to have an unconstrained problem. Also, the Lagrangian function can be used with an unconstrained search technique to locate the optimum and satisfy the constraints. In addition, the augmented Lagrangian function combines a penalty function with the Lagrangian function to alleviate computational difficulties associated with boundaries formed by equality constraints when the Lagrangian function is used alone. 86

87 SUMT Eample.. 87

88 Iterations in SUMT Eample.. Homework b 88

89 See Book website for more worked eamples.. 89

90 Generalized Reduced Gradient GRG 90

91 Nonconve Programming Convert into subproblems Find local optimal Evolutionary Computing e.g., genetic algorithms Run EC, and then use gradient descent 91

92 Outline Lecture 10 Multivariable unconstrained Multivariable unconstrained optimization Linear regression, Logistic Regression KKT conditions for Constrained Optimization Lagrangian, KKT conditions Quadratic Programming Modified Simple Algorithm Conve Programming Frank-Wolfe Algorithm, Penalty or barrier function e.g., SUMT Noncove Programming Course review 92

93 Course Review Course Reviw 93

94 End of Lecture 94

5 Handling Constraints

5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest